the_information_nexus/tech_docs/linux/ssh_best_practices.md

SSH Key Management Best Practices: A Comprehensive Guide

Introduction

Secure Shell (SSH) key-based authentication represents a cornerstone of modern system administration and secure remote access. Unlike password-based authentication, SSH keys provide cryptographically strong authentication that is resistant to brute force attacks and eliminates the transmission of credentials over the network. This document provides comprehensive guidance on implementing and maintaining robust SSH key management practices that balance security, usability, and operational efficiency.

1. Key Storage and Permissions

Understanding SSH Directory Structure

The SSH client and server rely on a well-defined directory structure with specific permission requirements. These permissions are not merely recommendations—SSH will refuse to operate if permissions are too permissive, as this could indicate a security compromise.

Private Key Storage and Protection

Private keys represent the most sensitive component of SSH key authentication and require the highest level of protection:

• Location: Store private keys in the user's ~/.ssh directory, which should have permissions set to 700 (readable, writable, and executable by owner only)
• File Permissions: Private key files must have permissions set to 400 (read-only for owner) or 600 (read-write for owner). More permissive settings will cause SSH to reject the key
• Naming Conventions: Use descriptive names that indicate the key's purpose, such as id_rsa_production_server or id_ed25519_github_2024

# Secure the SSH directory
chmod 700 ~/.ssh

# Secure private keys
chmod 400 ~/.ssh/id_rsa
chmod 400 ~/.ssh/id_ed25519

# Verify permissions
ls -la ~/.ssh/

Public Key Deployment

Public keys, while less sensitive than private keys, still require proper handling:

• Server-side Storage: Deploy to ~/.ssh/authorized_keys on target systems
• File Permissions: The authorized_keys file should have permissions 600 or 644
• Directory Permissions: The target user's home directory should not be world-writable, and the .ssh directory should have 700 permissions

# On the target server
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

File System Considerations

• Encryption at Rest: Consider storing SSH keys on encrypted file systems or using OS-level encryption features
• Backup Strategy: Include SSH keys in backup procedures, but ensure backups are encrypted and access-controlled
• Network File Systems: Avoid storing SSH keys on network-mounted file systems where possible, as this introduces additional attack vectors

2. Key Security Enhancements

Passphrase Protection

Passphrases serve as the final line of defense if private keys are compromised. A strong passphrase effectively encrypts the private key using symmetric encryption, rendering it useless without the passphrase.

Passphrase Best Practices:

• Complexity Requirements: Use passphrases of at least 20 characters combining multiple words, numbers, and symbols
• Avoid Dictionary Words: Employ techniques like Diceware for generating strong, memorable passphrases
• Unique Passphrases: Each key should have a unique passphrase to prevent cascading compromises
• Passphrase Managers: Consider using dedicated passphrase managers for complex environments

# Generate a key with strong passphrase protection
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_production -C "production-server-access-2024"

Key Rotation Strategies

Regular key rotation is essential for maintaining long-term security posture and limiting the impact of potential compromises.

Rotation Planning:

• Risk-Based Intervals: High-privilege keys should be rotated every 90 days, while standard user keys can follow 6-12 month cycles
• Event-Driven Rotation: Immediately rotate keys when personnel leave, security incidents occur, or systems are compromised
• Automated Rotation: Implement automated rotation for service accounts and high-volume environments using configuration management tools

Rotation Process:

1. Generate new key pairs with appropriate metadata
2. Deploy new public keys to all target systems
3. Test connectivity with new keys
4. Remove old public keys from authorized_keys files
5. Securely delete old private keys
6. Update documentation and key inventories

3. Configuration and Usage Restrictions

SSH Protocol Configuration

Modern SSH implementations should exclusively use SSH Protocol 2, which provides significant security improvements over the deprecated SSH Protocol 1.

Server Configuration (/etc/ssh/sshd_config):

# Force SSH Protocol 2 (default in modern versions)
Protocol 2

# Disable SSH Protocol 1 legacy support
# Note: Most modern SSH versions don't support Protocol 1 by default

Client Configuration (~/.ssh/config):

# Global default settings
Host *
    Protocol 2
    HashKnownHosts yes
    PasswordAuthentication no
    ChallengeResponseAuthentication no
    PubkeyAuthentication yes

Authorized Keys Restrictions

The authorized_keys file supports powerful restriction options that can significantly limit the scope of access granted by each key.

Command Restrictions:

# Restrict key to specific command execution
command="/usr/local/bin/backup-script.sh" ssh-rsa AAAAB3NzaC1yc2E...

# Allow only rsync for backup purposes
command="rsync --server --daemon ." ssh-rsa AAAAB3NzaC1yc2E...

Network Restrictions:

# Restrict access to specific IP addresses
from="192.168.1.100,10.0.0.0/8" ssh-rsa AAAAB3NzaC1yc2E...

# Combine multiple restrictions
from="192.168.1.100",command="/usr/bin/git-shell" ssh-rsa AAAAB3NzaC1yc2E...

Additional Restriction Options:

• no-port-forwarding: Disable SSH port forwarding
• no-X11-forwarding: Disable X11 forwarding
• no-agent-forwarding: Disable SSH agent forwarding
• no-pty: Disable pseudo-terminal allocation
• cert-authority: Specify certificate authorities for certificate-based authentication

4. Advanced Security Practices

SSH Agent Management

SSH agents provide secure, in-memory storage of decrypted private keys, eliminating the need to repeatedly enter passphrases while maintaining security.

Agent Security Considerations:

• Agent Lifetime: Configure automatic agent termination after periods of inactivity
• Agent Forwarding: Use agent forwarding judiciously and only over trusted connections
• Agent Locking: Implement screen locking that also locks SSH agents

# Start SSH agent with lifetime limitation
ssh-agent -t 3600  # Agent expires after 1 hour

# Add key with confirmation requirement
ssh-add -c ~/.ssh/id_ed25519

# List loaded keys
ssh-add -l

# Remove specific keys
ssh-add -d ~/.ssh/id_ed25519

# Remove all keys
ssh-add -D

Advanced Agent Configuration:

# In ~/.bashrc or ~/.zshrc
export SSH_AUTH_SOCK="$XDG_RUNTIME_DIR/ssh-agent.socket"

# Automatic agent startup and key loading
if ! pgrep -u "$USER" ssh-agent > /dev/null; then
    ssh-agent -t 1h > "$XDG_RUNTIME_DIR/ssh-agent.env"
fi
if [[ ! "$SSH_AUTH_SOCK" ]]; then
    source "$XDG_RUNTIME_DIR/ssh-agent.env" >/dev/null
fi

Certificate-Based Authentication

SSH certificates provide centralized key management and enhanced security features beyond traditional public key authentication.

Certificate Advantages:

• Centralized Management: Single certificate authority for all SSH access
• Time-Limited Access: Built-in expiration and validity periods
• Principal Restrictions: Fine-grained control over user and host identities
• Revocation Support: Immediate revocation capabilities without key redistribution

Multi-Factor Authentication Integration

Combining SSH keys with additional authentication factors provides defense in depth:

• Hardware Tokens: PKCS#11 integration with hardware security modules
• TOTP Integration: Time-based one-time passwords as additional factors
• PAM Integration: Leverage existing authentication infrastructure

5. Monitoring and Auditing

Comprehensive Logging Strategy

Effective SSH security requires comprehensive logging and monitoring of authentication events and key usage patterns.

Server-Side Logging Configuration:

# Enhanced logging in /etc/ssh/sshd_config
LogLevel VERBOSE
SyslogFacility AUTHPRIV

# Log successful authentications
LogLevel INFO

Key Audit Procedures:

• Regular Access Reviews: Quarterly reviews of all authorized_keys files
• Unused Key Detection: Identify and remove keys that haven't been used recently
• Anomaly Detection: Monitor for unusual access patterns or geographic anomalies
• Compliance Reporting: Generate regular reports for security and compliance teams

Automated Monitoring Scripts:

#!/bin/bash
# SSH key audit script
find /home -name "authorized_keys" -exec ls -la {} \; 2>/dev/null | \
  awk '{print $9, $6, $7, $8}' | sort

Incident Response Procedures

Develop and maintain incident response procedures specific to SSH key compromises:

1. Immediate Response: Disable compromised keys across all systems
2. Impact Assessment: Determine scope of potential unauthorized access
3. Forensic Analysis: Analyze logs for evidence of compromise
4. Recovery Actions: Generate new keys and restore secure access
5. Post-Incident Review: Update procedures based on lessons learned

6. Implementation and Compliance

Organizational Policy Development

Develop comprehensive SSH key management policies that address:

• Key Generation Standards: Approved algorithms, key lengths, and tools
• Approval Processes: Workflows for key generation and deployment requests
• Access Control Matrix: Define who can access what systems using SSH keys
• Documentation Requirements: Mandatory record-keeping for all SSH keys

Compliance Considerations

Different regulatory frameworks impose specific requirements on SSH key management:

PCI DSS Requirements:

• Strong cryptography and key management procedures
• Regular key rotation schedules
• Secure key storage and transmission

SOX Compliance:

• Segregation of duties in key management processes
• Audit trails for all key-related activities
• Change management procedures for SSH access

NIST Guidelines:

• Implementation of NIST SP 800-57 key management recommendations
• Use of FIPS 140-2 validated cryptographic modules where required

Automation and Tooling

Implement automation tools to reduce manual effort and improve consistency:

Configuration Management:

• Use tools like Ansible, Puppet, or Chef for automated key deployment
• Implement infrastructure as code for SSH configurations
• Automate compliance checking and remediation

Key Management Platforms:

• Consider enterprise SSH key management solutions
• Implement centralized key generation and distribution
• Automate key lifecycle management processes

7. Key Algorithm Selection and Modern Cryptography

Algorithm Recommendations

The choice of cryptographic algorithm significantly impacts both security and performance:

Ed25519 (Recommended):

• Security: Based on elliptic curve cryptography with excellent security properties
• Performance: Fast key generation, signing, and verification
• Key Size: Compact 256-bit keys reduce storage and transmission overhead
• Support: Supported in OpenSSH 6.5+ and most modern SSH implementations

ssh-keygen -t ed25519 -C "user@hostname-$(date +%Y%m%d)"

RSA Considerations:

• Minimum Key Size: Use 3072-bit keys minimum (4096-bit preferred for high-security environments)
• Legacy Compatibility: Better support in older systems
• Performance: Slower than Ed25519, especially for large key sizes

ECDSA Caveats:

• Curve Selection: Only use NIST P-256, P-384, or P-521 curves
• Implementation Risks: Vulnerable to poor random number generation
• Generally Discouraged: Ed25519 provides better security guarantees

Future-Proofing Considerations

• Post-Quantum Cryptography: Monitor developments in quantum-resistant algorithms
• Algorithm Deprecation: Plan for eventual phase-out of older algorithms
• Hybrid Approaches: Consider dual-algorithm deployments during transition periods

Conclusion

Effective SSH key management requires a comprehensive approach that balances security, usability, and operational efficiency. The practices outlined in this document provide a foundation for robust SSH security, but organizations must adapt these recommendations to their specific threat models, compliance requirements, and operational constraints.

Regular review and updates of SSH key management strategies are essential as the threat landscape evolves and new technologies emerge. Organizations should establish clear policies, implement appropriate tooling, and maintain ongoing vigilance to ensure their SSH infrastructure remains secure and compliant.

The investment in proper SSH key management pays dividends through reduced security incidents, improved operational efficiency, and enhanced compliance posture. By implementing these best practices systematically and maintaining them consistently, organizations can achieve both strong security and operational excellence in their SSH implementations.