SSH Server - Deployment & Configuration

SSH
Published

March 14, 2010

Modified

November 21, 2024

Secure Shell Protocol (SSH)…(replaced telnet, rlogin & rexec)

Swiss army knife for sys-admins…

Client/server architecture for remote login:

OpenSSH…most widely used implementation of SSH…

sshd

…listens for incoming connections acting as a server for the SSH protocol.

  • …handles…
    • …authentication and encryption
    • …terminal connections, file transfer and tunnels
  • …typically the sshd process is started on system boot
  • …usually located in /usr/sbin/sshd and executed as root
  • …each connections becomes a child instance of sshd
  • …the master server can be restarted without interference

Configuration

Configuration in /etc/ssh/sshd_configman sshd

  • …contains keyword-argument pairs …one per line
  • …lines starting with # considered comments

Dropping client connections…

  • TCPKeepAlive ..defaults to yes
    • network down or death of a client properly noticed
    • …if not sent sessions may hang indefinitely on the server
  • ClientAliveInterval
    • …time (in seconds) the server waits before sending a null packet
    • …this should keep the client connection alive
    • …defaults to 0 …connections drop if idle
  • ClientAliveCountMax
    • …time clients are allowed to stay unresponsive before disconnect
    • …defaults to 3 …three times ClientAliveInterval

Maintenance

Before shutdown of an sshd master server…

  • …make sure to provide a grace period for users to close active SSH connections
  • …login to the nodes and run following commands
# send a message to all users with an existing login
cat > /tmp/wall.sh <<EOF
while true
do
  echo -e "Service will be interrupted at 18:00.\nConnect to..." | wall
  sleep 900 # every 15 min
done
EOF
chmod +x /tmp/wall.sh
nohup /tmp/wall.sh &
disown

# schedule the shutdown
shutdown -h 18:00 --no-wall

# disable user login
echo "System maintenance in progress" | sudo tee /run/nologin

Debugging

Diagnosing of connection problems running sshd in fore-ground…

$(which sshd) -dt           # check validity of the configuration file and keys
$(which sshd) -d            # run the server in debug mode
$(which sshd) -d -p 2222    # on a different port

# enable debuggin in the configuration file
echo 'LogLevel DEBUG3' >> /etc/ssh/sshd_config && systemctl restart sshd
  • LogLevel specifies the level of verbosity for logging messages:
    • …defaults to INFO
    • …possible values: QUIET, FATAL, ERROR, INFO, VERBOSE, DEBUG, DEBUG2 and DEBUG3
  • …unprivileged, completely decoupled from the default system sshd
cd $(mktemp -d /tmp/sshd.XXXXXX)
ssh-keygen -t rsa -N '' -f ssh_host_rsa_key
cat > sshd_config <<EOF
UsePrivilegeSeparation no
HostKey $PWD/ssh_host_rsa_key
Port 2022
LogLevel DEBUG3
PidFile $PWD/sshd.pid
EOF
$(which sshd) -f sshd_config -dD

Add the -vvv options to a client for verbose output…

  • …does it establish a TCP connection
    • …check DNS resolution
    • …check routing
    • …check if the server is listening (by default on port 22)
  • …check the user name in the server logs
  • …check for successful encryption negotiation SSH2_MSG_SERVICE_ACCEPT received
  • …look for the authentication methods …typically public keys
    • …check permission to ~/.ssh/authorized_keys
    • Authentication succeeded …check login shell …shell profiles like .bashrc

Host Fingerprints

  • fingerprint are based on the host public key…
    • … Used for identification/verification of the host:
/etc/ssh/ssh_host_*                     # host key pairs
~/.ssh/known_hosts                      # accepted host fingerprints on the client
ssh-keygen -H -F <host>                 # search for a host in known_hosts
ssh-keygen -l -f <pubkey>               # get the fingerprint of a public key
ssh-keygen -lv -f <pubkey>              # ^^ include identicon
ssh -o VisualHostKey=yes ...            # show identicon at login
# add fingerprint of a server host key to known hosts
ssh-keyscan -H <hostname> >> ~/.ssh/known_hosts
# genreate fingerprint DNS records on a host
ssh-keygen -r `hostname`
  • first client connection includes server host key discovery
    • fingerprint presented to the user for verification
    • accepting a fingerprint adds it to the ~/.ssh/known_hosts known host file
    • fingerprint are associated to a specific hostname or IP address
    • breaking this association results in a security warning
  • fingerprint verification methods:
    • fingerprints are checked by users, e.g. comparison to public web-site
    • SSHFP (RFC4255) publishes fingerprints as DNS records
  • common anti pattern is “trust on first use”
    • assumes that a host is trusted at the time of the first connection
    • conditions users to always ignore the corresponding warning message
  • client automatically checks the fingerprint each login, and warns eventually
  • differing fingerprints (changed host key) indicate a potential man-in-the-middle attack
  • visual fingerprint, identicons (generated icons, to recognize or distinguish textual information)

RPort

…as an addition (alternative) to Jump hosts and SSH based VPN tunnels.

Rport acts as server and client establishing permanent or on-demand secure tunnels to devices inside protected intranets behind a firewall. … Any machine with the RPort client installed can act as a bridge, creating tunnels to any other IP address or host. This way you can easily manage routers, printer, switches or NAS systems inside remote networks. No VPN needed.

  • Server (REST API) (implemented in Go (single binary)) [^rserv]
    • Based on encrypted reverse tunnels…
    • Does not store remote login credentials…
    • Supports SSH, HTTP, VNC, RDP
  • Clients supported on Windows, Mac OS X, Linux/Unix
  • Web-interface and command-line rportcli
  • Two factor authentication (second factor over email)
  • Protocol agnostic (over TCP)…may be used for IPMI/BMC connections…

SSH through HTTP…

Clients establish their connection via HTTP. The use of HTTP proxies is supported. Within the HTTP connection, an SSH connection is established from the client to the server. Thus, the entire communication is encrypted. Proxies must allow HTTP CONNECT. Consequently, encryption happens at the application level and not at the transport level.

The client must know the fingerprint of the server before the connection is established. If the fingerprint does not match the server, the client refuses the connection. This prevents a possible man-in-the-middle attack. The statically compiled client has all SSH libraries on board. It does not access SSH program files in the operating system. [^rtech]

References

SSHGuard

SSHGuard1 …protects from brute-force attacks (similar to Fail2Ban)

  • …written in C …monitors & interprets log files
  • …integrates with firewall like: UFW, firewalld, iptables, pf, IPFW, etc

RPM package sshgaurd in EPEL:

dnf install epel-release ; dnf update
dnf install sshguard
systemctl enable --now sshguard.service
  • Configuration file /etc/sshguard.conf
  • Logs to /var/log/auth.log

SSH Load-Balancing

HAProxy…

Alternatives…

Protocol

SSH authentication protocol…over the SSH transport layer protocol…

  • Protocol packets have message numbers
  • …byte value that describes the payload of a packet
  • …show package numbers with layer 3 debugging ssh -vvv
  • Range 1 to 255
    • 1-49 transport protocol
    • 50-79 user authentication protocol
    • 80-127 connection protocol
    • 128-255 client local…

Related RFCs…

Transport

Package numbers…

  • 1 …SSH_MSG_DISCONNECT …rfc4253#section-11.1
  • 4 …SSH_MSG_DEBUG
  • 5 (client) …SSH_MSG_SERVICE_REQUEST
  • 6 …SSH_MSG_SERVICE_ACCEPT

Key exchange…

  • …negotiate algorithm for communication
  • Package numbers…
    • 20 …SSH_MSG_KEXINIT …exchange begins
    • 21 …SSH_MSG_NEWKEYS …exchange ends
    • 30-49…used for key exchange packets (depending on algorithm)
debug3: send packet: type 20                   # SSH_MSG_KEXINIT
debug3: receive packet: type 20
...
debug2: local client KEXINIT proposal
debug2: KEX algorithms: ...
debug2: host key algorithms: ...
...
debug2: peer server KEXINIT proposal
...
debug1: kex: algorithm: curve25519-sha256 ...
debug1: kex: host key algorithm: ssh-ed25519 ...
...
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
...
debug3: send packet: type 21                   # SSH_MSG_NEWKEYS

Authentication

Package numbers…

  • 50 (client) …SSH_MSG_USERAUTH_REQUEST
  • 51 …SSH_MSG_USERAUTH_FAILURE
  • 52 …SSH_MSG_USERAUTH_SUCCESS
  • 53 …SSH_MSG_USERAUTH_BANNER
  • 60
    • method “publickey”… SSH_MSG_USERAUTH_PK_OK
    • method “password”… SSH_MSG_USERAUTH_PASSWD_CHANGEREQ

Successful login with public key…

debug1: Next authentication method: publickey 
debug1: Offering public key: ...
debug3: send packet: type 50                   # SSH_MSG_USERAUTH_REQUEST
...
debug3: receive packet: type 60                # SSH_MSG_USERAUTH_PK_OK
debug1: Server accepts key: ...
debug3: sign_and_send_pubkey: ...
debug3: sign_and_send_pubkey: signing using... 
debug3: send packet: type 50
debug3: receive packet: type 52                # SSH_MSG_USERAUTH_SUCCESS
Authenticated to ... using "publickey".

Connection

Package numbers…

  • 80… SSH_MSG_GLOBAL_REQUEST
  • 81… SSH_MSG_REQUEST_SUCCESS
  • 82… SSH_MSG_REQUEST_FAILURE
  • 90… SSH_MSG_CHANNEL_OPEN
  • 91… SSH_MSG_CHANNEL_OPEN_CONFIRMATION
  • 92… SSH_MSG_CHANNEL_OPEN_FAILURE
  • 93… SSH_MSG_CHANNEL_WINDOW_ADJUST
  • 94… SSH_MSG_CHANNEL_DATA
  • 95… SSH_MSG_CHANNEL_EXTENDED_DATA
  • 96… SSH_MSG_CHANNEL_EOF
  • 97… SSH_MSG_CHANNEL_CLOSE
  • 98… SSH_MSG_CHANNEL_REQUEST
  • 99… SSH_MSG_CHANNEL_SUCCESS
  • 100… SSH_MSG_CHANNEL_FAILURE

After authentication…establish a communication channel…

debug1: channel 0: new [client-session]
...
debug3: send packet: type 90               # SSH_MSG_CHANNEL_OPEN
...
debug3: send packet: type 80
debug1: Entering interactive session.
debug3: receive packet: type 80
... # host key verification...
debug3: receive packet: type 91            # SSH_MSG_CHANNEL_OPEN_CONFIRMATION
... # establish communication channel...
debug3: send packet: type 98               # SSH_MSG_CHANNEL_REQUEST
... # setup environment...
debug3: receive packet: type 99            # SSH_MSG_CHANNEL_SUCCESS
... # start shell...