This lab is structured as six interconnected modules, each building upon the previous to create a comprehensive SSH skill set. You can complete modules in order for the full experience, or jump to specific modules based on your current needs. Each module includes theoretical foundations before hands-on exercises, ensuring you understand not just how to do something, but why it works and when to apply it. Plan for 5-7 hours total, or tackle individual modules in 45-90 minute sessions.
This intermediate lab transforms you from someone who can use SSH keys into someone who can architect SSH-based access systems. Building on the foundations from LAB 3, you will implement a complete home lab environment featuring multiple SSH identities for work/personal separation, GitHub integration for passwordless Git operations, ssh-agent mastery for secure key caching, server hardening to eliminate password authentication, SSH tunneling for secure access to remote services, and SSH certificates following enterprise patterns used by companies like Facebook and Netflix. By the end, you'll have skills that directly translate to DevOps, Site Reliability Engineering, and Security Engineering roles.
This lab assumes you have completed LAB 3: SSH Key-Based Authentication and understand the fundamentals of public/private key pairs, basic SSH connections, and file permissions. If you're starting fresh, complete LAB 3 first—it takes 2-3 hours and provides essential foundations.
By completing all six modules, you will have:
You've been promoted to Senior Systems Administrator at CloudScale Inc. With the promotion comes new responsibilities: you now manage SSH access for the entire engineering team (50+ developers), ensure compliance with security audits, and architect solutions that scale. Your manager says:
"We need to eliminate password authentication across all systems, implement proper key management for the team, set up SSH certificates so we're not managing thousands of authorized_keys entries, and document SSH tunneling procedures for developers who need database access from home."
This lab teaches exactly those skills. After completion, you'll be able to:
Design multi-key systems that separate personal, work, and service identities. Foundation for IAM roles and credential management.
SSH + Git integration used daily by millions of developers. Essential for DevOps and platform engineering.
ssh-agent patterns apply to all credential caching: AWS STS, Kubernetes tokens, Vault leases.
Server hardening skills transfer to all Linux services, containers, and cloud instances.
SSH tunneling is the basis for secure access patterns, VPN alternatives, and service mesh concepts.
SSH certificates teach PKI fundamentals that apply to TLS, code signing, and enterprise CA systems.
This lab is designed for a home lab environment using your local machine and a Linux virtual machine. No cloud accounts are required (though Module 2 uses a free GitHub account). The entire lab can be completed offline except for GitHub integration. Budget 5-7 hours for all modules, or complete individual modules in separate sessions.
| Component | Requirement | Purpose |
|---|---|---|
| Local Machine | Windows 10+, macOS 10.14+, or Linux | SSH client, key generation, agent |
| Ubuntu VM | Ubuntu 22.04 LTS, 2GB RAM, 20GB disk | SSH server for hardening and tunneling |
| GitHub Account | Free account at github.com | Module 2 SSH integration |
| LAB 3 Completion | Basic SSH key authentication working | Foundation for all modules |
Set up your Ubuntu VM using either:
Code blocks are tagged with badges indicating where to run commands:
This diagram shows the complete environment you'll build across all six modules. Each colored box represents a component you'll configure, with connections showing how they interact. By the end of this lab, you'll have a fully functional SSH ecosystem demonstrating enterprise-grade practices.
Module 1
Personal + Work + GitHub
~/.ssh/config routing
Module 2
Passwordless Git
SSH URL format
Module 3
Key caching
Timeouts & confirmation
Module 4
No passwords
Rate limiting & logging
Module 5
Local/Remote/Dynamic
SOCKS proxy
Module 6
User CA
No authorized_keys
Learn to create and manage multiple SSH identities for different purposes—personal projects, work systems, and service accounts.
Using a single SSH key for everything is like using one password for all your accounts—convenient but risky. If that key is compromised, every system trusting it is vulnerable. In enterprise environments, security policies often require separate keys for different security domains. This module teaches you to create purpose-specific keys and configure SSH to automatically select the correct key based on the destination host.
When you run ssh user@host, SSH tries keys in a specific order: keys offered by ssh-agent, then default key files (~/.ssh/id_ed25519, ~/.ssh/id_rsa, etc.), then keys specified in the config file. By creating separate keys and using SSH config to map them to hosts, you gain:
Common Identity Patterns:
First, let's create a dedicated key for personal projects. We'll use a custom filename and a comment that clearly identifies its purpose. This key will be used for personal servers, side projects, and non-work GitHub repositories.
Local MachineExplanation of flags:
-t ed25519 — Use the Ed25519 algorithm (modern, secure, fast)-f ~/.ssh/id_ed25519_personal — Custom filename instead of default-C "..." — Comment including hostname and year for identificationWhen prompted for a passphrase, enter a strong passphrase. For personal keys, a memorable sentence works well: "My first car was a 1997 Honda!"
Now create a separate key for work-related systems. This key should have a different (ideally stronger) passphrase than your personal key. In a real work environment, you might store this passphrase in your company's password manager.
Local MachineGitHub recommends using a dedicated SSH key for their service. This allows you to easily revoke GitHub access without affecting other systems, and provides a clear audit trail of Git operations.
Local Machine
Now we'll configure SSH to automatically select the appropriate key based on the destination. Create or edit your SSH config file to map hosts to keys. This is the magic that makes multi-key setups seamless—you never have to specify -i manually.
Key Configuration Options Explained:
IdentitiesOnly yes — Only use keys explicitly specified, don't try all keysAddKeysToAgent yes — Automatically add keys to ssh-agent after first useHost *.work.example.com work-* — Pattern matching for multiple hostsYou now have three separate SSH identities with automatic key selection. Run ls -la ~/.ssh/ to verify your new keys. In Module 2, we'll register the GitHub key with your GitHub account.
Configure GitHub to authenticate via SSH keys, eliminating password prompts and personal access tokens for Git operations.
Every professional developer uses SSH for GitHub authentication. It's more secure than HTTPS with passwords (no credentials transmitted), more convenient than personal access tokens (no token management), and provides clear audit trails. After this module, you'll push and pull code with zero password prompts—the way millions of developers work daily.
When you clone a repository using SSH (git@github.com:user/repo.git), Git initiates an SSH connection to github.com on port 22 as user git. GitHub doesn't have individual Unix users—instead, it identifies you by which SSH public key you present. Each key you upload to your GitHub account is associated with your profile. When you connect:
SSH vs. HTTPS for Git:
| Factor | SSH | HTTPS |
|---|---|---|
| Authentication | Key-based (no password) | Token or password |
| Setup | One-time key upload | Credential helper or prompts |
| Security | Key never transmitted | Token transmitted (encrypted) |
| Firewalls | Port 22 (sometimes blocked) | Port 443 (rarely blocked) |
First, display your GitHub-specific public key so you can copy it to your clipboard. Remember, this is the public key (.pub file)—safe to share.
Local Machine
Select and copy the entire output. It should start with ssh-ed25519 and end with your comment.
Navigate to GitHub's SSH key settings and add your public key. This registers your key with your account so GitHub can identify you.
GitHub Web InterfaceVerify your SSH connection to GitHub works. GitHub provides a special endpoint that confirms your identity without accessing any repository. The first connection will ask you to verify GitHub's host key—this is expected and safe to accept.
Local Machine
Expected output: Hi [username]! You've successfully authenticated, but GitHub does not provide shell access.
If you see "Permission denied (publickey)", check that your SSH config correctly specifies the GitHub key and that you uploaded the correct public key.
Test the full workflow by cloning a repository using SSH. You can use any public repository. Notice the URL format: git@github.com:owner/repo.git instead of https://github.com/owner/repo.git.
The git remote -v output should show SSH URLs (starting with git@), not HTTPS URLs.
If you have existing repositories using HTTPS, convert them to SSH by changing the remote URL. This is a one-time change per repository.
Local MachineYour GitHub account now recognizes your SSH key. All future git clone, git push, and git pull operations using SSH URLs will authenticate automatically. No more password prompts!
Learn to cache decrypted keys in memory, configure automatic timeouts, and require confirmation for sensitive operations.
If you set passphrases on your SSH keys (as you should), typing that passphrase for every SSH connection quickly becomes tedious. The ssh-agent solves this by securely caching your decrypted keys in memory. You type your passphrase once when adding the key, and the agent handles authentication for subsequent connections. This module teaches you to use ssh-agent effectively while maintaining security through timeouts and confirmation prompts.
The ssh-agent is a background process that holds your private keys in memory after you unlock them with your passphrase. When SSH needs to authenticate:
Key Security Concepts:
Agent Forwarding Warning: The -A flag (agent forwarding) allows remote servers to use your agent. This is convenient but dangerous—a compromised server can use your agent to authenticate elsewhere. We'll cover safer alternatives.
Most modern systems start ssh-agent automatically, but let's verify it's running and understand how to start it manually if needed.
Local Machine
If $SSH_AUTH_SOCK shows a path (like /tmp/ssh-xxx/agent.xxx), the agent is running. If ssh-add -l shows "The agent has no identities," no keys are loaded yet—that's expected.
Add your keys to the agent. You'll be prompted for each key's passphrase. After entering it once, you won't need to enter it again until the key expires or is removed.
Local Machine
For better security, add keys with automatic expiration. This limits the window of vulnerability if your machine is compromised while you're away. The -t flag sets the lifetime in seconds.
Recommended Lifetimes:
For high-security keys, require confirmation before each use. This prevents automated processes (including malware) from silently using your keys. The -c flag enables confirmation mode.
Confirmation mode is recommended for keys that access:
On macOS, you can configure SSH to automatically add keys to the agent and use the system keychain for passphrase storage. On Linux, we'll use the AddKeysToAgent directive to add keys on first use.
Here's a reference of common ssh-agent and ssh-add commands you'll use regularly:
Local MachineYou now understand ssh-agent and can configure keys with lifetimes and confirmation requirements. Your daily workflow will be: unlock keys once at login, use them seamlessly all day, and have them auto-expire for security.
Transform a default SSH installation into a hardened, audit-ready configuration following industry best practices.
A default SSH server installation prioritizes convenience over security—password authentication enabled, root login allowed, no rate limiting. In production environments, this is unacceptable. This module transforms your Ubuntu VM into a hardened SSH bastion following CIS benchmarks, Mozilla security guidelines, and real-world enterprise standards. You'll disable password authentication, implement rate limiting, configure audit logging, and create a server that would pass a security review.
SSH hardening follows the principle of defense in depth—multiple security layers so that if one fails, others still protect the system:
Industry Standards We'll Follow:
Before disabling password authentication, verify key-based authentication works. Test in a second terminal before closing your current session. Always maintain a backup access method (VM console, recovery boot, etc.).
Before hardening, ensure you can access the VM via SSH key. From your local machine, copy your personal key to the VM. If you completed LAB 3, this may already be done.
Local MachineAlways backup configuration files before making changes. This allows quick recovery if something goes wrong.
SSH Server (Ubuntu VM)Create a new, hardened SSH server configuration file. This configuration follows industry best practices and will be explained line by line.
SSH Server (Ubuntu VM)A login banner provides legal notice and may deter casual attackers. Many compliance frameworks require this. The banner is displayed before authentication.
SSH Server (Ubuntu VM)Before applying, validate the configuration syntax. A syntax error could prevent SSH from starting, locking you out. Then apply the configuration by restarting the SSH service.
SSH Server (Ubuntu VM)Open a NEW terminal and test SSH key authentication before closing your current session:
ssh -i ~/.ssh/id_ed25519_personal labuser@vm-ip
Fail2ban monitors logs for failed authentication attempts and temporarily bans IP addresses that show malicious behavior. This prevents brute force attacks even if password authentication is re-enabled accidentally.
SSH Server (Ubuntu VM)Test that password authentication is truly disabled. This attempt should fail immediately without even prompting for a password.
Local MachineCheck the SSH logs to see authentication attempts and verify logging is working. These logs are essential for security monitoring and incident investigation.
SSH Server (Ubuntu VM)Your SSH server is now hardened to production standards. Password authentication is disabled, root login is prohibited, secure ciphers are enforced, and fail2ban protects against brute force attacks. This configuration would pass most security audits.
Master local, remote, and dynamic port forwarding to securely access services through encrypted SSH connections.
SSH tunneling (port forwarding) is one of SSH's most powerful features, allowing you to securely access services that aren't directly exposed to the internet. Need to access a database that only accepts connections from localhost? SSH tunnel. Want to browse the web through a remote server? SSH SOCKS proxy. This module teaches all three types of tunneling with practical scenarios you'll encounter in real work.
SSH provides three types of port forwarding, each solving different problems:
Forwards a port on your local machine to a destination through the SSH server. Traffic flows: Local → SSH Server → Destination
Use case: Access a remote database that only accepts localhost connections
Forwards a port on the remote server to a destination through your local machine. Traffic flows: Remote → SSH Tunnel → Local → Destination
Use case: Expose a local development server to the internet temporarily
Creates a SOCKS proxy on your local machine that routes all traffic through the SSH server. Traffic flows: Local App → SOCKS → SSH Server → Internet
Use case: Browse the web as if you were at the remote server's location
In Module 4, we disabled TCP forwarding for security. To practice tunneling, temporarily enable it on your VM:
Temporarily enable TCP forwarding on your hardened server. In production, you would enable this selectively per-user or create a separate SSH configuration for jump hosts.
SSH Server (Ubuntu VM)Create a simple web server on the VM that only listens on localhost. This simulates a service (like a database admin panel) that isn't exposed to the network.
SSH Server (Ubuntu VM)This server only listens on localhost:8080
" > ~/webtest/index.html # Start a simple HTTP server (Python 3) bound to localhost only cd ~/webtest python3 -m http.server 8080 --bind 127.0.0.1 & # Verify it's running and only on localhost ss -tlnp | grep 8080The web server is now running on port 8080, but it only accepts connections from 127.0.0.1 (localhost on the VM). You cannot access it directly from your local machine.
Use local port forwarding to access the VM's localhost-only web server from your local machine. This forwards your local port 9080 to the VM's localhost:8080.
Local Machinelocalhost:9080
encrypted
receives request
127.0.0.1:8080
Create a SOCKS proxy that routes all traffic through the SSH server. This is useful for browsing the web as if you were at the server's location, or for accessing multiple services behind a firewall.
Local MachineRemote port forwarding exposes a service on your local machine through the SSH server. This is useful for sharing a local development server with a colleague or webhook testing.
Local MachineFor tunnels you use frequently, configure them in ~/.ssh/config so you can establish them with a simple command.
Local MachineWhen you're done with tunnels, clean up the background SSH processes.
Local MachineYou now understand all three types of SSH tunneling. These techniques are used daily by developers and administrators to securely access databases, internal services, and test webhooks.
Implement an SSH Certificate Authority to issue time-limited user certificates—the same pattern used by major tech companies.
Managing SSH keys at scale is challenging: authorized_keys files on hundreds of servers, key rotation across thousands of users, no built-in expiration. SSH certificates solve these problems elegantly. Instead of copying public keys to every server, you create a Certificate Authority (CA) that signs user keys. Servers trust the CA, so any user with a valid certificate can authenticate. Facebook, Netflix, and many other large companies use this pattern. This module teaches you to set up your own SSH CA.
Understanding the difference between traditional keys and certificates:
authorized_keysAn SSH certificate contains:
First, create a dedicated key pair for your SSH Certificate Authority. This CA key will sign user certificates. Protect this key carefully—anyone with access to it can create certificates trusted by all servers.
Local MachineIn production, the CA private key should be stored in a Hardware Security Module (HSM) or air-gapped system. For this lab, we're keeping it on your local machine, but treat it as highly sensitive.
Install the CA public key on your server and configure SSHD to trust certificates signed by this CA. This is a one-time setup per server.
SSH Server (Ubuntu VM)Now the exciting part—sign your existing public key to create a certificate. This certificate proves your key was approved by the CA and specifies which username(s) you can authenticate as.
CA Server (Local Machine)Understanding the signing options:
-s ~/.ssh/ca/user_ca — Use this CA key to sign-I "labuser@myworkstation" — Key ID for audit logs-n labuser — This cert can authenticate as "labuser"-V +1d — Valid for 1 day (production might use hours)Before using the certificate, let's examine its contents to understand what we created. The certificate contains metadata about validity, principals, and restrictions.
Local MachineKey fields in the output:
Now test authentication using your certificate. SSH will automatically use the certificate file if it exists alongside your private key. You can also explicitly specify it.
Local MachineCertificates can include restrictions that limit what the user can do. This is useful for service accounts or limited-access scenarios.
CA Server (Local Machine)Available restrictions:
no-port-forwarding — Disable all port forwardingno-pty — Prevent getting a terminal (useful for rsync/scp only access)no-agent-forwarding — Disable agent forwardingsource-address=CIDR — Only allow connections from specific IPsforce-command=CMD — Force a specific command on connectCertificates automatically expire after their validity period. When expired, the user must request a new certificate from the CA. This is a feature, not a bug—it ensures regular re-authorization.
Local MachineWith certificate-based authentication working, you can remove traditional authorized_keys entries. The server now authenticates based on the CA signature, not individual public keys.
SSH Server (Ubuntu VM)You've implemented SSH certificate-based authentication! This enterprise pattern eliminates authorized_keys management, provides automatic expiration, and enables centralized access control. You now understand how companies like Facebook and Netflix manage SSH access at scale.
This lab covers advanced SSH techniques that introduce new attack surfaces. Understanding these vulnerabilities and their mitigations is essential for secure implementation. This section covers risks specific to the technologies in this lab: multi-key management, agent forwarding, server hardening gaps, tunneling abuse, and certificate authority compromise.
If the CA private key is stolen, attackers can issue certificates for any user, bypassing all authentication on every server trusting that CA.
An attacker gains access to the system storing the CA key. They issue themselves a certificate for "root" on all servers. Every server trusting the CA accepts their certificate without question.
A malicious process on your local machine could connect to your ssh-agent socket and use your cached keys to authenticate to remote systems.
You download a compromised npm package that runs malicious code. The malware detects your ssh-agent socket, uses it to authenticate to GitHub, and pushes malicious code to your repositories.
SSH tunnels can be used by attackers for command-and-control channels or to pivot to internal networks.
An attacker compromises a server with SSH access. They create a reverse tunnel to their C2 server, establishing persistent access that bypasses firewalls and looks like normal SSH traffic.
With multiple keys, users may accidentally use the wrong key, potentially exposing work credentials to personal systems or vice versa.
A user's SSH config doesn't correctly route keys. They accidentally use their personal key for a work server, creating an unauthorized authentication path. Or worse, they upload a work key to a public GitHub account.
| Setting | Lab Value | Production Value |
|---|---|---|
| CA Key Storage | Local filesystem | HSM or air-gapped system |
| Certificate Validity | 1 day (learning) | 4-8 hours maximum |
| AllowTcpForwarding | Enabled (for Module 5) | Disabled or per-user only |
| SSH Agent Confirmation | Optional | Required for production keys |
| Key Lifetime in Agent | Unlimited (convenience) | 4-8 hours |
Advanced SSH configurations have more failure points than basic setups. This section covers troubleshooting techniques for the specific technologies in this lab.
Root Cause: Certificate not being used, or principals don't match.
Diagnostic Steps:
ls ~/.ssh/*-cert.pubssh-keygen -L -f ~/.ssh/key-cert.pub | grep Validssh-keygen -L -f ~/.ssh/key-cert.pub | grep Principalsgrep TrustedUserCAKeys /etc/ssh/sshd_config*ssh -vvv user@server and look for certificate messagesRoot Cause: TCP forwarding disabled or service not listening.
Diagnostic Steps:
sshd -T | grep allowtcpforwardingss -tlnp | grep PORTcurl localhost:PORT on the serverlsof -i :LOCALPORTRoot Cause: SSH config not routing correctly or agent offering wrong key first.
Diagnostic Steps:
ssh -v user@host 2>&1 | grep "Offering"ssh -G hostname | grep identityfilessh-add -lCongratulations on completing this comprehensive lab! You've progressed from basic SSH usage to implementing enterprise-grade patterns. Here's a summary of the skills you've mastered.
Continue your SSH mastery with these carefully selected resources covering advanced topics, enterprise implementations, and emerging best practices.