SSH Server - Deployment & Configuration
Secure Shell Protocol (SSH)…(replaced telnet
, rlogin
& rexec
)
- …cryptographic network protocol
- …hierarchical layered protocol suite…
- …transport…server authentication, confidentiality, and integrity
- …user authentication…
- …connection…multiplexes the encrypted tunnel…into communication channels
- Versions…
SSH-1
…deprecated in 2006SSH-2
…adopted as IETF standard…incompatible with SSH-1
Swiss army knife for sys-admins…
Client/server architecture for remote login:
sshd
(SSH daemon) accepts incoming connections to its host computerssh
(SSH client) connects tosshd
on a remote server- transparently encrypted communication (optionally compressed)
- public key authentication (replacing password authentication)
- Authentication of the server (prevent man-in-the-middle attacks)
- file transfer, the SSH protocol family includes two file transfer protocols
- Port forwarding for TCP sessions over an SSH connection
OpenSSH…most widely used implementation of SSH…
- Project – https://www.openssh.com – https://github.com/openssh
- Release Notes – https://www.openssh.com/releasenotes.html
- Security – https://www.openssh.com/security.html
sshd
…listens for incoming connections acting as a server for the SSH protocol.
- …handles…
- …authentication and encryption
- …terminal connections, file transfer and tunnels
- …typically the
sshd
process is started on system boot - …usually located in
/usr/sbin/sshd
and executed asroot
- …each connections becomes a child instance of
sshd
- …the master server can be restarted without interference
Configuration
Configuration in /etc/ssh/sshd_config
…man sshd
…
- …contains keyword-argument pairs …one per line
- …lines starting with
#
considered comments
Dropping client connections…
TCPKeepAlive
..defaults toyes
- …network down or death of a client properly noticed
- …if not sent sessions may hang indefinitely on the server
ClientAliveInterval
- …time (in seconds) the server waits before sending a null packet
- …this should keep the client connection alive
- …defaults to
0
…connections drop if idle
ClientAliveCountMax
- …time clients are allowed to stay unresponsive before disconnect
- …defaults to
3
…three timesClientAliveInterval
Maintenance
Before shutdown of an sshd
master server…
- …make sure to provide a grace period for users to close active SSH connections
- …login to the nodes and run following commands
# send a message to all users with an existing login
cat > /tmp/wall.sh <<EOF
while true
do
echo -e "Service will be interrupted at 18:00.\nConnect to..." | wall
sleep 900 # every 15 min
done
EOF
chmod +x /tmp/wall.sh
nohup /tmp/wall.sh &
disown
# schedule the shutdown
shutdown -h 18:00 --no-wall
# disable user login
echo "System maintenance in progress" | sudo tee /run/nologin
Debugging
Diagnosing of connection problems running sshd
in fore-ground…
$(which sshd) -dt # check validity of the configuration file and keys
$(which sshd) -d # run the server in debug mode
$(which sshd) -d -p 2222 # on a different port
# enable debuggin in the configuration file
echo 'LogLevel DEBUG3' >> /etc/ssh/sshd_config && systemctl restart sshd
- …
LogLevel
specifies the level of verbosity for logging messages:- …defaults to INFO
- …possible values: QUIET, FATAL, ERROR, INFO, VERBOSE, DEBUG, DEBUG2 and DEBUG3
- …unprivileged, completely decoupled from the default system
sshd
…
cd $(mktemp -d /tmp/sshd.XXXXXX)
ssh-keygen -t rsa -N '' -f ssh_host_rsa_key
cat > sshd_config <<EOF
UsePrivilegeSeparation no
HostKey $PWD/ssh_host_rsa_key
Port 2022
LogLevel DEBUG3
PidFile $PWD/sshd.pid
EOF
$(which sshd) -f sshd_config -dD
Add the -vvv
options to a client for verbose output…
- …does it establish a TCP connection
- …check DNS resolution
- …check routing
- …check if the server is listening (by default on port 22)
- …check the user name in the server logs
- …check for successful encryption negotiation
SSH2_MSG_SERVICE_ACCEPT
received - …look for the authentication methods …typically public keys
- …check permission to
~/.ssh/authorized_keys
- …
Authentication succeeded
…check login shell …shell profiles like.bashrc
- …check permission to
Host Fingerprints
- fingerprint are based on the host public key…
- … Used for identification/verification of the host:
/etc/ssh/ssh_host_* # host key pairs
~/.ssh/known_hosts # accepted host fingerprints on the client
ssh-keygen -H -F <host> # search for a host in known_hosts
ssh-keygen -l -f <pubkey> # get the fingerprint of a public key
ssh-keygen -lv -f <pubkey> # ^^ include identicon
ssh -o VisualHostKey=yes ... # show identicon at login
# add fingerprint of a server host key to known hosts
ssh-keyscan -H <hostname> >> ~/.ssh/known_hosts
# genreate fingerprint DNS records on a host
ssh-keygen -r `hostname`
- first client connection includes server host key discovery
- fingerprint presented to the user for verification
- accepting a fingerprint adds it to the
~/.ssh/known_hosts
known host file - fingerprint are associated to a specific hostname or IP address
- breaking this association results in a security warning
- fingerprint verification methods:
- fingerprints are checked by users, e.g. comparison to public web-site
- SSHFP (RFC4255) publishes fingerprints as DNS records
- common anti pattern is “trust on first use”
- assumes that a host is trusted at the time of the first connection
- conditions users to always ignore the corresponding warning message
- client automatically checks the fingerprint each login, and warns eventually
- differing fingerprints (changed host key) indicate a potential man-in-the-middle attack
- visual fingerprint, identicons (generated icons, to recognize or distinguish textual information)
RPort
…as an addition (alternative) to Jump hosts and SSH based VPN tunnels.
Rport acts as server and client establishing permanent or on-demand secure tunnels to devices inside protected intranets behind a firewall. … Any machine with the RPort client installed can act as a bridge, creating tunnels to any other IP address or host. This way you can easily manage routers, printer, switches or NAS systems inside remote networks. No VPN needed.
- Server (REST API) (implemented in Go (single binary)) [^rserv]
- Based on encrypted reverse tunnels…
- Does not store remote login credentials…
- Supports SSH, HTTP, VNC, RDP
- Clients supported on Windows, Mac OS X, Linux/Unix
- Web-interface and command-line
rportcli
… - Two factor authentication (second factor over email)
- Protocol agnostic (over TCP)…may be used for IPMI/BMC connections…
SSH through HTTP…
Clients establish their connection via HTTP. The use of HTTP proxies is supported. Within the HTTP connection, an SSH connection is established from the client to the server. Thus, the entire communication is encrypted. Proxies must allow HTTP CONNECT. Consequently, encryption happens at the application level and not at the transport level.
The client must know the fingerprint of the server before the connection is established. If the fingerprint does not match the server, the client refuses the connection. This prevents a possible man-in-the-middle attack. The statically compiled client has all SSH libraries on board. It does not access SSH program files in the operating system. [^rtech]
References
- [^rport] Project Page and Source Code
- RPort remote access and remote management, FOSDEM’22
- Komfortable und sichere Fernwartung für alle Betriebssysteme OSS, FrOSCon’21
- [^rtech] RPort Technology Explained
- Rport Full feature list
- [^rserv] Rport Install on-premises
SSHGuard
SSHGuard1 …protects from brute-force attacks (similar to Fail2Ban)
- …written in C …monitors & interprets log files
- …integrates with firewall like: UFW, firewalld, iptables, pf, IPFW, etc
RPM package sshgaurd
in EPEL:
dnf install epel-release ; dnf update
dnf install sshguard
systemctl enable --now sshguard.service
- Configuration file
/etc/sshguard.conf
- Logs to
/var/log/auth.log
SSH Load-Balancing
HAProxy…
- Load Balancing SSH
- Proxy and port forwarding
- Load-Balancing ssh with HAProxy
- Route SSH Connections with HAProxy
Alternatives…
- SSH Piper reverse proxy for sshd
- Ballast - Lightweight SSH load balancer
- SSHIM - SSH Interruption Mitigator
Protocol
SSH authentication protocol…over the SSH transport layer protocol…
- Protocol packets have message numbers…
- …byte value that describes the payload of a packet
- …show package numbers with layer 3 debugging
ssh -vvv
- Range 1 to 255
1-49
transport protocol50-79
user authentication protocol80-127
connection protocol128-255
client local…
Related RFCs…
- The Secure Shell (SSH) Protocol Architecture
- The Secure Shell (SSH) Transport Layer Protocol
- The Secure Shell (SSH) Authentication Protocol
- The Secure Shell (SSH) Connection Protocol
- The Secure Shell (SSH) Protocol Assigned Numbers
Transport
Package numbers…
1
…SSH_MSG_DISCONNECT …rfc4253#section-11.14
…SSH_MSG_DEBUG5
(client) …SSH_MSG_SERVICE_REQUEST6
…SSH_MSG_SERVICE_ACCEPT
Key exchange…
- …negotiate algorithm for communication
- Package numbers…
20
…SSH_MSG_KEXINIT …exchange begins21
…SSH_MSG_NEWKEYS …exchange ends30-49
…used for key exchange packets (depending on algorithm)
debug3: send packet: type 20 # SSH_MSG_KEXINIT
debug3: receive packet: type 20
...
debug2: local client KEXINIT proposal
debug2: KEX algorithms: ...
debug2: host key algorithms: ...
...
debug2: peer server KEXINIT proposal
...
debug1: kex: algorithm: curve25519-sha256 ...
debug1: kex: host key algorithm: ssh-ed25519 ...
...
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
...
debug3: send packet: type 21 # SSH_MSG_NEWKEYS
Authentication
Package numbers…
50
(client) …SSH_MSG_USERAUTH_REQUEST51
…SSH_MSG_USERAUTH_FAILURE52
…SSH_MSG_USERAUTH_SUCCESS53
…SSH_MSG_USERAUTH_BANNER60
…- method “publickey”… SSH_MSG_USERAUTH_PK_OK
- method “password”… SSH_MSG_USERAUTH_PASSWD_CHANGEREQ
Successful login with public key…
debug1: Next authentication method: publickey
debug1: Offering public key: ...
debug3: send packet: type 50 # SSH_MSG_USERAUTH_REQUEST
...
debug3: receive packet: type 60 # SSH_MSG_USERAUTH_PK_OK
debug1: Server accepts key: ...
debug3: sign_and_send_pubkey: ...
debug3: sign_and_send_pubkey: signing using...
debug3: send packet: type 50
debug3: receive packet: type 52 # SSH_MSG_USERAUTH_SUCCESS
Authenticated to ... using "publickey".
Connection
Package numbers…
80
… SSH_MSG_GLOBAL_REQUEST81
… SSH_MSG_REQUEST_SUCCESS82
… SSH_MSG_REQUEST_FAILURE90
… SSH_MSG_CHANNEL_OPEN91
… SSH_MSG_CHANNEL_OPEN_CONFIRMATION92
… SSH_MSG_CHANNEL_OPEN_FAILURE93
… SSH_MSG_CHANNEL_WINDOW_ADJUST94
… SSH_MSG_CHANNEL_DATA95
… SSH_MSG_CHANNEL_EXTENDED_DATA96
… SSH_MSG_CHANNEL_EOF97
… SSH_MSG_CHANNEL_CLOSE98
… SSH_MSG_CHANNEL_REQUEST99
… SSH_MSG_CHANNEL_SUCCESS100
… SSH_MSG_CHANNEL_FAILURE
After authentication…establish a communication channel…
debug1: channel 0: new [client-session]
...
debug3: send packet: type 90 # SSH_MSG_CHANNEL_OPEN
...
debug3: send packet: type 80
debug1: Entering interactive session.
debug3: receive packet: type 80
... # host key verification...
debug3: receive packet: type 91 # SSH_MSG_CHANNEL_OPEN_CONFIRMATION
... # establish communication channel...
debug3: send packet: type 98 # SSH_MSG_CHANNEL_REQUEST
... # setup environment...
debug3: receive packet: type 99 # SSH_MSG_CHANNEL_SUCCESS
... # start shell...