SSH Server - Deployment & Configuration

SSH
Published

March 14, 2010

Modified

November 11, 2025

Secure Shell Protocol (SSH)…(replaced telnet, rlogin & rexec)

Swiss army knife for sys-admins…

Client/server architecture for remote login:

OpenSSH…most widely used implementation of SSH…

sshd

…listens for incoming connections acting as a server for the SSH protocol.

  • …handles…
    • …authentication and encryption
    • …terminal connections, file transfer and tunnels
  • …typically the sshd process is started on system boot
  • …usually located in /usr/sbin/sshd and executed as root
  • …each connections becomes a child instance of sshd
  • …the master server can be restarted without interference

Configuration

Configuration in /etc/ssh/sshd_configman sshd

  • …contains keyword-argument pairs …one per line
  • …lines starting with # considered comments

Dropping client connections…

  • TCPKeepAlive ..defaults to yes
    • network down or death of a client properly noticed
    • …if not sent sessions may hang indefinitely on the server
  • ClientAliveInterval
    • …time (in seconds) the server waits before sending a null packet
    • …this should keep the client connection alive
    • …defaults to 0 …connections drop if idle
  • ClientAliveCountMax
    • …time clients are allowed to stay unresponsive before disconnect
    • …defaults to 3 …three times ClientAliveInterval

Maintenance

Before shutdown of an sshd master server…

  • …make sure to provide a grace period for users to close active SSH connections
  • …login to the nodes and run following commands
# send a message to all users with an existing login
cat > /tmp/wall.sh <<EOF
while true
do
  echo -e "Service will be interrupted at 18:00.\nConnect to..." | wall
  sleep 900 # every 15 min
done
EOF
chmod +x /tmp/wall.sh
nohup /tmp/wall.sh &
disown

# schedule the shutdown
shutdown -h 18:00 --no-wall

# disable user login
echo "System maintenance in progress" | sudo tee /run/nologin

Debugging

Diagnosing of connection problems running sshd in fore-ground…

$(which sshd) -dt           # check validity of the configuration file and keys
$(which sshd) -d            # run the server in debug mode
$(which sshd) -d -p 2222    # on a different port

# enable debuggin in the configuration file
echo 'LogLevel DEBUG3' >> /etc/ssh/sshd_config && systemctl restart sshd
  • LogLevel specifies the level of verbosity for logging messages:
    • …defaults to INFO
    • …possible values: QUIET, FATAL, ERROR, INFO, VERBOSE, DEBUG, DEBUG2 and DEBUG3
  • …unprivileged, completely decoupled from the default system sshd
cd $(mktemp -d /tmp/sshd.XXXXXX)
ssh-keygen -t rsa -N '' -f ssh_host_rsa_key
cat > sshd_config <<EOF
UsePrivilegeSeparation no
HostKey $PWD/ssh_host_rsa_key
Port 2022
LogLevel DEBUG3
PidFile $PWD/sshd.pid
EOF
$(which sshd) -f sshd_config -dD

Add the -vvv options to a client for verbose output…

  • …does it establish a TCP connection
    • …check DNS resolution
    • …check routing
    • …check if the server is listening (by default on port 22)
  • …check the user name in the server logs
  • …check for successful encryption negotiation SSH2_MSG_SERVICE_ACCEPT received
  • …look for the authentication methods …typically public keys
    • …check permission to ~/.ssh/authorized_keys
    • Authentication succeeded …check login shell …shell profiles like .bashrc

Host Fingerprints

SSH fingerprint — Uniquely identifies a server

  • Fingerprint based on the host keys /etc/ssh/ssh_host_*
  • Used to verify the identity of a server you’re connecting to
  • Prevents MITM “man-in-the-middle” attacks

Fingerprint Discovery

First connection includes server host key discovery

  • The fingerprint is presented to the user for verification
    • Identicons …generated recognizable icons of identification
    • ssh -o VisualHostKey=yes shows identicon at login
  • Fingerprints are associated to a specific host name or IP address
  • TOFU — “Trust On First Use” (anti-pattern)
    • Describes the situation of “blindly” trusting a fingerprint…
    • Risky …enables to being compromised during first connection
    • Conditions users to always ignore corresponding warning message

Accepting a fingerprint saves it to the ~/.ssh/known_hosts

  • SSH checks the fingerprint for each reconnect
  • Non-matching fingerprint indicates a potential problem
  • No (automatic) ongoing check to verify the server’s identity (beyond that first time)

Out-of-band Verification

Recommended to verify the fingerprint out-of-band

Physical Access or Secure Communication

  • Login to the server via a console (direct attached or KVM)
  • Login to the server via BMC interface to access a console
# show the fingerprint of a specific host public key
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
# …include identicon
ssh-keygen -lv -f /etc/ssh/ssh_host_ed25519_key.pub

Automate verification …to eliminate human error:

  • SSHFP (RFC42551), look up a fingerprint in the DNS
  • Storing SSH fingerprints in an LDAP schema extension
# generate fingerprint DNS records on a host
ssh-keygen -r localhost

Manage Fingerprints

# search for a host in known_hosts
ssh-keygen -H -F <host>

# add fingerprint of a server host key to known hosts
ssh-keyscan -H $hostname >> ~/.ssh/known_hosts

RPort

…as an addition (alternative) to Jump hosts and SSH based VPN tunnels.

Rport acts as server and client establishing permanent or on-demand secure tunnels to devices inside protected intranets behind a firewall. … Any machine with the RPort client installed can act as a bridge, creating tunnels to any other IP address or host. This way you can easily manage routers, printer, switches or NAS systems inside remote networks. No VPN needed.

  • Server (REST API) (implemented in Go (single binary)) [^rserv]
    • Based on encrypted reverse tunnels…
    • Does not store remote login credentials…
    • Supports SSH, HTTP, VNC, RDP
  • Clients supported on Windows, Mac OS X, Linux/Unix
  • Web-interface and command-line rportcli
  • Two factor authentication (second factor over email)
  • Protocol agnostic (over TCP)…may be used for IPMI/BMC connections…

SSH through HTTP…

Clients establish their connection via HTTP. The use of HTTP proxies is supported. Within the HTTP connection, an SSH connection is established from the client to the server. Thus, the entire communication is encrypted. Proxies must allow HTTP CONNECT. Consequently, encryption happens at the application level and not at the transport level.

The client must know the fingerprint of the server before the connection is established. If the fingerprint does not match the server, the client refuses the connection. This prevents a possible man-in-the-middle attack. The statically compiled client has all SSH libraries on board. It does not access SSH program files in the operating system. [^rtech]

References

SSHGuard

SSHGuard2 …protects from brute-force attacks (similar to Fail2Ban)

  • …written in C …monitors & interprets log files
  • …integrates with firewall like: UFW, firewalld, iptables, pf, IPFW, etc

RPM package sshgaurd in EPEL:

dnf install epel-release ; dnf update
dnf install sshguard
systemctl enable --now sshguard.service
  • Configuration file /etc/sshguard.conf
  • Logs to /var/log/auth.log

SSH Load-Balancing

HAProxy…

Alternatives…

Protocol

SSH authentication protocol…over the SSH transport layer protocol…

  • Protocol packets have message numbers
  • …byte value that describes the payload of a packet
  • …show package numbers with layer 3 debugging ssh -vvv
  • Range 1 to 255
    • 1-49 transport protocol
    • 50-79 user authentication protocol
    • 80-127 connection protocol
    • 128-255 client local…

Related RFCs…

Transport

Package numbers…

  • 1 …SSH_MSG_DISCONNECT …rfc4253#section-11.1
  • 4 …SSH_MSG_DEBUG
  • 5 (client) …SSH_MSG_SERVICE_REQUEST
  • 6 …SSH_MSG_SERVICE_ACCEPT

Key exchange…

  • …negotiate algorithm for communication
  • Package numbers…
    • 20 …SSH_MSG_KEXINIT …exchange begins
    • 21 …SSH_MSG_NEWKEYS …exchange ends
    • 30-49…used for key exchange packets (depending on algorithm)
debug3: send packet: type 20                   # SSH_MSG_KEXINIT
debug3: receive packet: type 20
...
debug2: local client KEXINIT proposal
debug2: KEX algorithms: ...
debug2: host key algorithms: ...
...
debug2: peer server KEXINIT proposal
...
debug1: kex: algorithm: curve25519-sha256 ...
debug1: kex: host key algorithm: ssh-ed25519 ...
...
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
...
debug3: send packet: type 21                   # SSH_MSG_NEWKEYS

Authentication

Package numbers…

  • 50 (client) …SSH_MSG_USERAUTH_REQUEST
  • 51 …SSH_MSG_USERAUTH_FAILURE
  • 52 …SSH_MSG_USERAUTH_SUCCESS
  • 53 …SSH_MSG_USERAUTH_BANNER
  • 60
    • method “publickey”… SSH_MSG_USERAUTH_PK_OK
    • method “password”… SSH_MSG_USERAUTH_PASSWD_CHANGEREQ

Successful login with public key…

debug1: Next authentication method: publickey 
debug1: Offering public key: ...
debug3: send packet: type 50                   # SSH_MSG_USERAUTH_REQUEST
...
debug3: receive packet: type 60                # SSH_MSG_USERAUTH_PK_OK
debug1: Server accepts key: ...
debug3: sign_and_send_pubkey: ...
debug3: sign_and_send_pubkey: signing using... 
debug3: send packet: type 50
debug3: receive packet: type 52                # SSH_MSG_USERAUTH_SUCCESS
Authenticated to ... using "publickey".

Connection

Package numbers…

  • 80… SSH_MSG_GLOBAL_REQUEST
  • 81… SSH_MSG_REQUEST_SUCCESS
  • 82… SSH_MSG_REQUEST_FAILURE
  • 90… SSH_MSG_CHANNEL_OPEN
  • 91… SSH_MSG_CHANNEL_OPEN_CONFIRMATION
  • 92… SSH_MSG_CHANNEL_OPEN_FAILURE
  • 93… SSH_MSG_CHANNEL_WINDOW_ADJUST
  • 94… SSH_MSG_CHANNEL_DATA
  • 95… SSH_MSG_CHANNEL_EXTENDED_DATA
  • 96… SSH_MSG_CHANNEL_EOF
  • 97… SSH_MSG_CHANNEL_CLOSE
  • 98… SSH_MSG_CHANNEL_REQUEST
  • 99… SSH_MSG_CHANNEL_SUCCESS
  • 100… SSH_MSG_CHANNEL_FAILURE

After authentication…establish a communication channel…

debug1: channel 0: new [client-session]
...
debug3: send packet: type 90               # SSH_MSG_CHANNEL_OPEN
...
debug3: send packet: type 80
debug1: Entering interactive session.
debug3: receive packet: type 80
... # host key verification...
debug3: receive packet: type 91            # SSH_MSG_CHANNEL_OPEN_CONFIRMATION
... # establish communication channel...
debug3: send packet: type 98               # SSH_MSG_CHANNEL_REQUEST
... # setup environment...
debug3: receive packet: type 99            # SSH_MSG_CHANNEL_SUCCESS
... # start shell...