SSH Client — Configuratrion & Usage

Reference

SSH

Published

March 14, 2010

Modified

March 26, 2025

Most commonly used list of files and commands…

~/.ssh/config             # client configuration
~/.ssh/authorized_keys    # authorized keys on a server
ssh                       # login to a remote server
scp                       # copy files from/to a remote server
rsync                     # ^^ (delta sync.)
ssh-keygen                # generate an ssh public/private key pair
ssh-copy-id               # copy a public key to a server
ssh-agent                 # daemon temporarily storing the private key
ssh-add                   # add a private key to an SSH agent
ssh-keyscan               # manage host fingerprints
sshuttle                  # private network tunnel over SSH
sshfs                     # mount remote file-systems over SSH

`ssh`

Establish a remote terminal session to a server <user>@<server>:

<user> is a login account name on the remote server, i.e. “jdow”
<server> is an IP address or hostname of the server to connect to, i.e. “pool.devops.test”
…ask you to accept the remote computer into the list of known hosts (only at first login)
Make sure fingerprint is trustworthy before accepting
Type in your password before you can login
Use exit to logout from the remote computer and end the SSH session
Close hanging SSH connections using an escape sequence ~..

# ↵ indictates pressing Enter
>>> ssh jdow@pool.devops.test↵
The authenticity of host 'pool.devops.test' (10.10.10.10)' can't be established.
RSA key fingerprint is 96:15:0e:e7:70:09:60:9a:c4:f6:89:05:be:ed:be:c6.
Are you sure you want to continue connecting (yes/no)? yes↵
Warning: Permanently added 'pool.devops.test' (RSA) to the list of known hosts.
jdow@pool.devops.test's password:↵
jdow@node1: exit↵

Known hosts prevents attacks based on subverting the naming services:

Client and server provide an identity to each other upon connection
The SSH server has a secret unique ID, called a host key, to identify itself to clients
Host key gets stored on the client in the ~/.ssh/known_hosts file
Each subsequent connection is authenticated using the server public key

Disable known-hosts….

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ...
scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ...

Force password authentication….

Depends on PasswordAuthentication yes in sshd_config…
…and PermitRootLogin yes if required

# force password login...ignore public keys....
ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no ...

`scp`

scp moves files across SSH connections:

source and/or destination can be located on a remote computer
option -r copies recursively all file in the directory tree

# copy files to a remote server
scp -r /local/path/ jdow@pool.devops.test:remote/path/

# copy files from a remote server
scp -r jdow@pool.devops.test:remote/file/ /local/path

# transfer a signification amount of data by increasing
# the speed with an alternative encryption method
scp -c blowfish ...

# less secure but faster...
ssh -C4c arcfour,blowfish-cbc ...

`~/.ssh/config`

Client OpenSSH configuration in ~/.ssh/config…

…sections separated by Host specifications
- …contains keyword-argument pairs…one per line
- …arguments optionally enclosed in double quotes "
… matched Host usually given on the command line
…first obtained value will be used
…lines starting #…comments

Example

# specific nodes 
Host lxdev* lxabc?
  User alice
  HostKeyAlgorithms +ssh-rsa

# specific domain
Host *.domain.org
  User bob
  IdentityFile ~/.ssh/%u/id_ed25519
  AddKeysToAgent yes

# global defaults
Host *
  GSSAPIAuthentication no
  ForwardAgent no

Host restricts the following…

…to node names matching a pattern…
- * matches zero or more characters
- ? matches exactly one character
- ! negates a match
- multiple patterns separated by space
single *…used for global defaults

Keyword-argument pairs…

Hostname specifies the real host name
User specify node login account
…more in the following sections…

`ssh-keygen` & `ssh-copy-id`

SSH keys (asymmetric cryptography)

Provides cryptographic strength that even extremely long passwords can not offer
Grants access to servers; allows automated, password-less login
Users can self-provision a key pair (authentication credential)

Each SSH key pair includes two keys:

Public key that is copied to the SSH server(s)
Private key that remains (only) with the user (aka. identity key)

ssh-keygen generates a public/private key pair:

~/.ssh/id_* default path to private key
~/.ssh/id_*.pub default path to user identities (public key)
~/.ssh/authorized_keys lists identities allowed to login

Key type: ed25519 (recommended, RSA is deprecated)

Based on elliptic curve discrete logarithm problem
Variant of the ECDSA algorithm, improved random number generator

ssh-keygen -q -t ed25519 -f ~/.ssh/id_ed25519

It is extremely important that the privacy of the private key is guarded carefully!

Encrypting the private key with a passphrase
Users require the passphrase to use/decrypt the private key
Handling of passphrases can be automated with an ssh-agent

Change password of private key

ssh-keygen -p -f ~/.ssh/id_ed25519

ssh-copy-id deploys and identity key to ~/.ssh/authorized_keys on a login node

# copy the public key to the GSI login nodes
ssh-copy-id -i ~/.ssh/id_ed25519.pub $USER@lxpool.gsi.de

`ssh-agent`

ssh-agent stores private keys used for SSH authentication…

Allows to login to a node hosts while keeping private keys local…
- …forward the authentication requests back to the ssh-agent on localhost
- …support multiple hops in a login chain

…ssh-agent prints environment variables to standard output

eval loads environment variables emitted by the agent
Many Linux distributions automatically start an SSH agent on login

# start ssh-agent...add variables to the environment
>>> eval $(ssh-agent)
Agent pid 2157
# add an authentication Identity...
>>> ssh-add ~/.ssh/id_rsa
Enter passphrase for /home/vpenso/.ssh/id_rsa:
Identity added: /home/vpenso/.ssh/id_rsa (/home/vpenso/.ssh/id_rsa)

Environment variables used to locate the agent socket…

$SSH_AUTH_SOCK …location of the unix socket
$SSH_CONNECTION…source IP and port of the previous host…as well as the local IP and port

Forwarding

ssh -A enables agent forwarding to a login node…

Exposes authentication keys to the node… (requires trustworthy remote nodes)

…attacker with control of the server (i.e. root access) can communicate with your agent
Defaulting to always forwarding the agent is strongly discouraged
…proxy jumps are a safer alternative to agent forwarding

Configure agent forwarding in ~/.ssh/config…

Host ...
    # automatically add ssh-agent forwarding for a connection
    ForwardAgent yes

`ssh-add`

Added public-key (identities) to an running agent…

# add default identity
ssh-add
# add a specific identity
ssh-add ~/.ssh/id_ed25519
# ...ask if using the key... lock the key after 30 minutes
ssh-add -c -t 1800 ~/.ssh/id_ed25519
# remove a specific identity
ssh-add -d ~/.ssh/id_ed25519

Options to harden identity forwarding…
- -c require confirmation every time the agent uses an identity
- -t $seconds lock identity in the agent after a configurable amount of time
Options to remove identities
- -d removes identities from the agent
- -D deletes all identities from the agent

Configure to use an identity on login in ~/.ssh/config…

Host ...
    # specify a file to read an authentication identity, % syntax for specific tokens
    IdentityFile ~/.ssh/%r/id_ed25519
    # automatically added to a running ssh-agent
    AddKeysToAgent yes

Commonly used tokes for the IdentityFile path…

%r remote username
%h remote hostname

Stored Identities

ssh-add options to list identities currently represented by the agent…

-l …lists fingerprints
-L …lists public key parameters

Following ssh-added script prints a list or identities in the ssh-agent including path…

>>> cat ~/bin/ssh-added
#!/usr/bin/env sh
while read -r line; do
        # loop over all keys in the key chain
        for file in $(find -L ~/.ssh -maxdepth 5 -type f -name 'id_*' -perm 600)
        do
            printf "%s %s\n" "$(ssh-keygen -lf "$file" | awk '{$1=""}1')" "$file";
        # print if the key is loaded in ssh-agent
        done \
                | column -t \
                | tr -s ' ' \
                | grep --color=auto "$line" \
                || echo "$line"
done < <(ssh-add -l | awk '{print $2}')

`ssh-agent-session`

Typically users have multiple parallel shells…

…convenient to connect with a single ssh-agent
ssh-agent-session script provides this functionality
- ~/.ssh/agent-session stores agent connection information
- ssh-agent-stop to remove all identities and shutdown

# typically store in your home-directory
>>> cat ~/bin/ssh-agent-session
# ...see print below...

# start the ssh-agent (if not running)
>>> source ~/bin/ssh-agent-session
ssh-agent started

# if agent is already running
>>> source ssh-agent-session
ssh-agent running with process ID 19264

# Source this script within the your shell profile...
# ...for example in Bash
echo "source ~/bin/ssh-agent-session"  >> ~/.bashrc

Print of the ssh-agent-session for reference:

# does not include a she-bang line for an interpreter...

if [ ! -d ~/.ssh ]
then
  mkdir $HOME/.ssh
fi

# SSH agent configuration is saved to file:
export SSH_ENV=$HOME/.ssh/agent-session

# check if an SSH agent is running
function __ssh_agent_running() {
  if [ -f $SSH_ENV ]; then
    source $SSH_ENV
    ps $SSH_AGENT_PID > /dev/null
    return $?
  else
    return 1
  fi
}

# If an SSH agent is running attach new shells
function __ssh_agent_load() {
  if __ssh_agent_running
  then
    source $SSH_ENV
  fi
}


# remove any trace of existing SSH agents
function __ssh_agent_stop() {
  if __ssh_agent_running 
  then
    ssh-add -D
    ssh-agent -k > /dev/null 2>&1
    rm $SSH_ENV
  fi
  # just to be sure!
  killall -u $USER ssh-agent
  # removes un-managed SSH agents too!
}

alias ssh-agent-stop=__ssh_agent_stop

function __ssh_agent_start() {
  if ! __ssh_agent_running 
  then
    __ssh_agent_stop # be absolutely sure!
    # kills also agents spawned by other entities
    ssh-agent | sed 's/^echo/#echo/' > $SSH_ENV
    # restrict permissions
    chmod 600 $SSH_ENV
    source $SSH_ENV
    echo "ssh-agent started, session in $SSH_ENV"
  fi
}

# start SSH agent if missing
__ssh_agent_start
# load existing sessions by default
command=${1:-"load"}
case "$command" in
  load)
    __ssh_agent_load
    ;;
  stop)
    __ssh_agent_stop
    echo "Killed"
    ;;
  *)
    echo "Usage: $(basename $0) [load|stop]"
    ;;
esac

Port Forwarding

Connect over a jump proxy $bastion…

…option -A …forward ssh-agent (to authenticate to the gateway node)
…to a gateway node $gateway (to access a network behind the bastion node)
…forward the HTTPS port 443 from the target $server
…option -f …background
…option -N …do not execute a remote command

ssh -J $bastion -A -L localhost:8080:$server:443 $gateway -fN

boring¹ is a simple command-line SSH tunnel manager.

`rsync`

Advanced alternative for the cp and scp commands:

Synchronizes data (files/directories) between a source and destination path
Either the source or the destination can be on another machine but not both
Special algorithm to minimize the network traffic

rsync -r <src> <dst>                 # copy source directory into destination
rsync -r <src>/ <dst>                # copy content of source directory into destination

# remote locations can be specified with a host-colon syntax
rsync <src> <host>:<dst>             # copy to a remote node
rsync <host>:<src> <dst>             # copy from a remote node

Special treatment to source directories with a trailing slash /…
…copies the contents of the source directory to the destination

-e, --rsh (remote shell) option to overwrite the remote shell connection…

# SSH option to connect with a non-default port
rsync -e 'ssh -p 2022' ...

# multiple options including variable substitution
rsync -e "ssh -o ProxyJump=${USER}@example.domain -o PubkeyAcceptedKeyTypes=+ssh-rsa"...

Show changes without modification to the destination…

-n, --dry-run perform a trial run with no changes made
-i, --itemize-changes output a change-summary for all updates

Output format for itemize changes:

YXcstpoguax  path/to/file
|||||||||||
`----------- the type of update being done::
 ||||||||||   <: file is being transferred to the remote host (sent).
 ||||||||||   >: file is being transferred to the local host (received).
 ||||||||||   c: local change/creation for the item, such as:
 ||||||||||      - the creation of a directory
 ||||||||||      - the changing of a symlink,
 ||||||||||      - etc.
 ||||||||||   h: the item is a hard link to another item (requires --hard-links).
 ||||||||||   .: not being updated (...might have attributes that are being modified).
 ||||||||||   *: rest of the itemized-output area contains a message (e.g. "deleting").
 ||||||||||
 `---------- the file type:
  |||||||||   f for a file,
  |||||||||   d for a directory,
  |||||||||   L for a symlink,
  |||||||||   D for a device,
  |||||||||   S for a special file (e.g. named sockets and fifos).
  |||||||||
  `--------- c: different checksum (for regular files)
   ||||||||     changed value (for symlink, device, and special file)
   `-------- s: Size is different
    `------- t: Modification time is different
     `------ p: Permission are different
      `----- o: Owner is different
       `---- g: Group is different
        `--- u: The u slot is reserved for future use.
         `-- a: The ACL information changed

File Comparison

Quick check algorithm by default…

…looks for files that have changed in size or in last−modified time.
Modification options…
- --ignore-existing skip updating files that exist on receiver
- --size-only skip files that match in size, even if the timestamps differ
- -I, --ignore-times checksum every file, even if the timestamps and file sizes match

Use 128−bit checksum for each file that has a matching size…

…lot of disk I/O reading all the data
-c, --checksum checksum instead of “quick check”
--checksum-choice=auto overrides the checksum algorithm, “md4”, “md5”, and “none”

Transfer Options

Option -a archive mode equals -rlptgoD (not ACLs, hard links or extended attributes such as capabilities):

-r, --recursive recursive into directories
-l, --links copy symlinks as symlinks
-p, --perms preserve permissions
-t, --times preserve modification times
-g, --group preserve group
-o, --owner preserve owner (super-user only)
-D same as --devices --specials
--devices preserve device files (super-user only)
--specials preserve special files

Option -hP human readable progress/speed indicator:

-h, --human-readable numbers in a human-readable format
--progress shows progress during transfer
--partial keep partially transferred files, enables to resume interrupted file transfer
-P same as --partial --progress

Remove extraneous files from the destination with option --del.

--del alias for --delete-during
--delete-during receiver deletes during the transfer

Include & Exclude

Exclude files/directories with option --exlude:

Support multiple uses of the --exclude (and --include) option
Alternatively use -files-from=- to pre-filter a list of files…

# exlcude all images...
rsync -a --exclude '*.jpg*' ...

# pass a list of files vis STDIN
find $PWD -name "*.jpg" -printf %P\\0\\n | rsync --files-from=- ...

Pattern matching…

* matches any non-empty path component (it stops at slashes)
** to match anything, including slashes
? matches any character except a slash (/).
[ introduces a character class, such as [a-z] or [[:alpha:]].
In a wild-card pattern, a backslash \ can be used to escape…
…matched literally when no wild-cards are present

`sshuttle`

Sshuttle² is part transparent proxy, part VPN (Virtual Private Network), and part SSH:

No need to install anything on the remote server…

…based on remote login over SSH (requires an account on the remote node)
…the remote node needs Python installed (mostly installed by default on Linux)
User requires root access on localhost (aka Sudo)…
- …to modify your system firewall
- …to tunnel all traffic through a remote SSH connection

Connect to a remote network…

# First the `sudo` password is prompted …second prompt for the SSH login password
sshuttle --dns --remote [user@]host[:port] --daemon --pidfile=/tmp/sshuttle.pid 0/0

# disconnect
kill $(cat /tmp/sshuttle.pid)

Option --dns to enable DNS tunneling (use DNS service from the remote node)
It is possible to start multiple connections to different networks
…furthermore connections can be nested

Configuration Files

Path to a configuration file preceded by @ character:

# create a configuration file in your home-directory
mkdir ~/.sshuttle
cat > ~/.sshuttle/example.conf <<EOF
value
--option1
value1
--option2
value2
EOF

# pass the configuration path as argument
sshuttle @~/.sshuttle/example.conf

…arguments read from a file must be one per line
…arguments to the commands take precedents over the configuration file

Connection Management

sshuttles³ is a simple wrapper-script to manage multiple connections…

…reading configuration files from a directory (~/.sshuttle by default)
…writes PID files to /tmp/$USER-sshuttle*.pid …one per connections

# list the configuration filenames (no path, no suffix) 
sshuttles list

# start a connection by specifying the name
sshuttles start $config_name

# stop a specific connection
sshuttles stop $config_name

Why? …many complex connection configurations to many different targets, for example:

cat > ~/.sshuttle/node_example_org.conf <<EOF
# target networks
172.100.0.0/16
10.0.0.0/8
--daemon
--dns
--disable-ipv6
# node used for SSH login
--remote
node.example.org
# exclude following network segments from tunneling
--exclude
172.100.60.0/24
--exclude
172.100.3.0/24
--pidfile
/tmp/sshuttle-node_example_org.pid
EOF

`sshfs`

# install on CentOS (requries EPEL)
yum install -y fuse-sshfs
# install on Debian
apt install -y sshfs
# install on Arch
pacman -S sshfs

Mount a directory from a remote host

# mount a remote directory
sshfs -C -o reconnect,auto_cache,follow_symlinks [user@]host[:port] /mnt/path 
# unmount the remote directory
fusermount -u /mnt/path

Alternatives…

BASSHFS - Bash-Accessible SSH File System
- https://pkolano.github.io/projects/basshfs.html
- https://github.com/pkolano/basshfs
sshocker
- https://github.com/lima-vm/sshocker

Expect

expect ⁴ automates interactive applications such as SSH…

sudo dnf install -y expect

…for example to automate the password prompt during SSH login:

expect-ssh

#!/usr/bin/expect

set timeout 20
set password "$env(USER_PASSWORD)"
set user "$env(USER_NAME)"
set node [lindex $argv 0]
set cmd [lrange $argv 1 end]
set ssh_options "-o PreferredAuthentications=password -o PubkeyAuthentication=no"

eval spawn ssh $ssh_options $user@$node $cmd

expect "assword:"   # matches both 'Password' and 'password'
send -- "$password\r";
interact

Script	Description
`expect-ssh`	Execute a command over SSH on target
`expect-scp`	Copy a file over SSH to/from target
`expect-ssh-sudo`	Execute a command over SSH on target, command requires a subsequent password prompt (for example `sudo`)
`expect-ssh-setpwd`	Execute a command over SSH on target, command requires a subsequent password prompt, and sets and confirms a new password

The scripts require following environment variables to be set:

Environment Variable	Description
`EXPECT_LOGIN_PASSWORD`	Login password on the target, password used for subsequent prompts (for example `sudo`) as well
`EXPECT_LOGIN_USER`	Login user name on the target
`EXPECT_SET_PASSWORD`	Password to be set & confirmed (for example during user creating)

Prepare the environment:

# handy aliases
alias essh=expect-ssh
alias escp=expect-scp

# set require environment variables
export EXPECT_LOGIN_USER=$USER
export EXPECT_LOGIN_PASSWORD=secret123

# usage examples
expect-ssh $fqdn ls -l /tmp
expect-scp /bin/bash $fqdn:/tmp

EXPECT_SET_PASSWORD=secret321
expect-ssh-setpwd $fqdn passwd

`rclone`

Rclone manages files on remote storage…

…preserves timestamps …verifies checksums
…transfers over limited bandwidth …intermittent connections
rclone mount …as a file system with FUSE

# configure the SFTP connection
cat >> ~/.config/rclone/rclone.conf <<EOF
[example.org]
type = sftp
host = login.example.org
EOF

# mount a path from the remote storage using SFTP
rclone mount example.org:path/to/ ~/example.org

`opkssh`

OpenPubkey SSH⁵

…adds user public keys to OpenID Connect (OIDC)⁶
…protocol builds on the OpenPubkey⁷ …uses SSH Certificates extensions

Footnotes

boring SSH tunnel manager, GitHub
https://github.com/alebeck/boring ↩︎
Sshuttle Project, GitHub
https://github.com/sshuttle/sshuttle
https://sshuttle.readthedocs.io ↩︎
sshuttles Script, GitHub
https://github.com/vpenso/scripts/blob/master/bin/sshuttles ↩︎
expect Documentation
https://core.tcl-lang.org/expect/index ↩︎
OpenPubkey SSH, GitHub
https://github.com/openpubkey/opkssh ↩︎
Open-sourcing OpenPubkey SSH (OPKSSH): integrating single sign-on with SSH, Cloudflare
https://blog.cloudflare.com/open-sourcing-openpubkey-ssh-opkssh-integrating-single-sign-on-with-ssh/↩︎
OpenPubkey, GitHub
https://github.com/openpubkey/openpubkey ↩︎

--- title: 'SSH Client — Configuratrion & Usage' categories: - Reference - SSH date: 2010/03/14 date-modified: 2025/03/26 toc-expand: 4 --- Most commonly used list of files and commands... ```bash ~/.ssh/config # client configuration ~/.ssh/authorized_keys # authorized keys on a server ssh # login to a remote server scp # copy files from/to a remote server rsync # ^^ (delta sync.) ssh-keygen # generate an ssh public/private key pair ssh-copy-id # copy a public key to a server ssh-agent # daemon temporarily storing the private key ssh-add # add a private key to an SSH agent ssh-keyscan # manage host fingerprints sshuttle # private network tunnel over SSH sshfs # mount remote file-systems over SSH ``` ## `ssh` Establish a **remote terminal session** to a server `<user>@<server>`: - `<user>` is a login account name on the remote server, i.e. "jdow" - `<server>` is an IP address or hostname of the server to connect to, i.e. "pool.devops.test" - ...ask you to accept the remote computer into the list of known hosts (only at first login) - Make sure fingerprint is trustworthy before accepting - Type in your password before you can login - Use `exit` to logout from the remote computer and end the SSH session - Close hanging SSH connections using an **escape sequence `~.`**. ```bash # ↵ indictates pressing Enter >>> ssh jdow@pool.devops.test↵ The authenticity of host 'pool.devops.test' (10.10.10.10)' can't be established. RSA key fingerprint is 96:15:0e:e7:70:09:60:9a:c4:f6:89:05:be:ed:be:c6. Are you sure you want to continue connecting (yes/no)? yes↵ Warning: Permanently added 'pool.devops.test' (RSA) to the list of known hosts. jdow@pool.devops.test's password:↵ jdow@node1: exit↵ ``` **Known hosts** prevents attacks based on subverting the naming services: - Client and server provide an identity to each other upon connection - The SSH server has a secret unique ID, called a **host key**, to identify itself to clients - Host key gets stored on the client in the `~/.ssh/known_hosts` file - Each subsequent connection is authenticated using the server public key Disable known-hosts.... ```bash ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ... scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ... ``` **Force password** authentication.... - Depends on `PasswordAuthentication yes` in `sshd_config`... - ...and `PermitRootLogin yes` if required ```bash # force password login...ignore public keys.... ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no ... ``` ## `scp` **`scp` moves files** across SSH connections: - source and/or destination can be located on a remote computer - option **`-r` copies recursively** all file in the directory tree ```bash # copy files to a remote server scp -r /local/path/ jdow@pool.devops.test:remote/path/ # copy files from a remote server scp -r jdow@pool.devops.test:remote/file/ /local/path # transfer a signification amount of data by increasing # the speed with an alternative encryption method scp -c blowfish ... # less secure but faster... ssh -C4c arcfour,blowfish-cbc ... ``` ## `~/.ssh/config` Client OpenSSH configuration in `~/.ssh/config`... - ...**sections separated by `Host`** specifications - ...**contains keyword-argument pairs**...one per line - ...arguments optionally enclosed in double quotes `"` - ... matched `Host` usually given on the command line - ...first obtained value will be used - ...lines starting `#`...comments Example ```sh # specific nodes Host lxdev* lxabc? User alice HostKeyAlgorithms +ssh-rsa # specific domain Host *.domain.org User bob IdentityFile ~/.ssh/%u/id_ed25519 AddKeysToAgent yes # global defaults Host * GSSAPIAuthentication no ForwardAgent no ``` `Host` restricts the following... - ...to node names matching a pattern... - `*` matches zero or more characters - `?` matches exactly one character - `!` negates a match - multiple patterns separated by space - single `*`...used for global defaults Keyword-argument pairs... - `Hostname` specifies the real host name - `User` specify node login account - ...more in the following sections... ## `ssh-keygen` & `ssh-copy-id` SSH keys (asymmetric cryptography) - Provides cryptographic strength that even extremely long passwords can not offer - Grants access to servers; allows automated, password-less login - Users can **self-provision a key pair** (authentication credential) Each SSH key pair includes two keys: - **Public key** that is copied to the SSH server(s) - **Private key** that remains (only) with the user (aka. identity key) `ssh-keygen` generates a public/private key pair: - `~/.ssh/id_*` default path to private key - `~/.ssh/id_*.pub` default path to user identities (public key) - `~/.ssh/authorized_keys` lists identities allowed to login Key type: **ed25519** (recommended, RSA is deprecated) - Based on elliptic curve discrete logarithm problem - Variant of the ECDSA algorithm, improved random number generator ```bash ssh-keygen -q -t ed25519 -f ~/.ssh/id_ed25519 ``` **It is extremely important that the privacy of the private key is guarded carefully!** - **Encrypting the private key** with a passphrase - Users require the passphrase to use/decrypt the private key - Handling of passphrases can be automated with an `ssh-agent` Change password of private key ```bash ssh-keygen -p -f ~/.ssh/id_ed25519 ``` `ssh-copy-id` deploys and identity key to `~/.ssh/authorized_keys` on a login node ```bash # copy the public key to the GSI login nodes ssh-copy-id -i ~/.ssh/id_ed25519.pub $USER@lxpool.gsi.de ``` ## `ssh-agent` `ssh-agent` stores private keys used for SSH authentication... - Allows to login to a node hosts while keeping private keys local... - ...forward the authentication requests back to the ssh-agent on localhost - ...support multiple hops in a login chain ...`ssh-agent` prints environment variables to standard output - `eval` loads environment variables emitted by the agent - Many Linux distributions automatically start an SSH agent on login ```sh # start ssh-agent...add variables to the environment >>> eval $(ssh-agent) Agent pid 2157 # add an authentication Identity... >>> ssh-add ~/.ssh/id_rsa Enter passphrase for /home/vpenso/.ssh/id_rsa: Identity added: /home/vpenso/.ssh/id_rsa (/home/vpenso/.ssh/id_rsa) ``` Environment variables used to locate the agent socket... - `$SSH_AUTH_SOCK` ...location of the unix socket - `$SSH_CONNECTION`...source IP and port of the previous host...as well as the local IP and port ### Forwarding `ssh -A` enables agent forwarding to a login node... **Exposes authentication keys to the node...** (requires trustworthy remote nodes) - ...attacker with control of the server (i.e. root access) can communicate with your agent - Defaulting to always forwarding the agent is strongly discouraged - ...**proxy jumps are a safer alternative to agent forwarding** Configure agent forwarding in `~/.ssh/config`... ```sh Host ... # automatically add ssh-agent forwarding for a connection ForwardAgent yes ``` ### `ssh-add` Added public-key (identities) to an running agent... ```sh # add default identity ssh-add # add a specific identity ssh-add ~/.ssh/id_ed25519 # ...ask if using the key... lock the key after 30 minutes ssh-add -c -t 1800 ~/.ssh/id_ed25519 # remove a specific identity ssh-add -d ~/.ssh/id_ed25519 ``` - Options to harden identity forwarding... - `-c` require confirmation every time the agent uses an identity - `-t $seconds` lock identity in the agent after a configurable amount of time - Options to remove identities - `-d` removes identities from the agent - `-D` deletes all identities from the agent Configure to use an identity on login in `~/.ssh/config`... ```sh Host ... # specify a file to read an authentication identity, % syntax for specific tokens IdentityFile ~/.ssh/%r/id_ed25519 # automatically added to a running ssh-agent AddKeysToAgent yes ``` Commonly used tokes for the `IdentityFile` path... - `%r` remote username - `%h` remote hostname ### Stored Identities `ssh-add` options to list identities currently represented by the agent... - `-l` ...lists fingerprints - `-L` ...lists public key parameters Following `ssh-added` script prints a list or identities in the ssh-agent including path... ```sh >>> cat ~/bin/ssh-added #!/usr/bin/env sh while read -r line; do # loop over all keys in the key chain for file in $(find -L ~/.ssh -maxdepth 5 -type f -name 'id_*' -perm 600) do printf "%s %s\n" "$(ssh-keygen -lf "$file" | awk '{$1=""}1')" "$file"; # print if the key is loaded in ssh-agent done \ | column -t \ | tr -s ' ' \ | grep --color=auto "$line" \ || echo "$line" done < <(ssh-add -l | awk '{print $2}') ``` ### `ssh-agent-session` Typically users have multiple parallel shells... - ...convenient to connect with a single `ssh-agent` - `ssh-agent-session` script provides this functionality - `~/.ssh/agent-session` stores agent connection information - `ssh-agent-stop` to remove all identities and shutdown ```sh # typically store in your home-directory >>> cat ~/bin/ssh-agent-session # ...see print below... # start the ssh-agent (if not running) >>> source ~/bin/ssh-agent-session ssh-agent started # if agent is already running >>> source ssh-agent-session ssh-agent running with process ID 19264 # Source this script within the your shell profile... # ...for example in Bash echo "source ~/bin/ssh-agent-session" >> ~/.bashrc ``` Print of the `ssh-agent-session` for reference: ```sh # does not include a she-bang line for an interpreter... if [ ! -d ~/.ssh ] then mkdir $HOME/.ssh fi # SSH agent configuration is saved to file: export SSH_ENV=$HOME/.ssh/agent-session # check if an SSH agent is running function __ssh_agent_running() { if [ -f $SSH_ENV ]; then source $SSH_ENV ps $SSH_AGENT_PID > /dev/null return $? else return 1 fi } # If an SSH agent is running attach new shells function __ssh_agent_load() { if __ssh_agent_running then source $SSH_ENV fi } # remove any trace of existing SSH agents function __ssh_agent_stop() { if __ssh_agent_running then ssh-add -D ssh-agent -k > /dev/null 2>&1 rm $SSH_ENV fi # just to be sure! killall -u $USER ssh-agent # removes un-managed SSH agents too! } alias ssh-agent-stop=__ssh_agent_stop function __ssh_agent_start() { if ! __ssh_agent_running then __ssh_agent_stop # be absolutely sure! # kills also agents spawned by other entities ssh-agent | sed 's/^echo/#echo/' > $SSH_ENV # restrict permissions chmod 600 $SSH_ENV source $SSH_ENV echo "ssh-agent started, session in $SSH_ENV" fi } # start SSH agent if missing __ssh_agent_start # load existing sessions by default command=${1:-"load"} case "$command" in load) __ssh_agent_load ;; stop) __ssh_agent_stop echo "Killed" ;; *) echo "Usage: $(basename $0) [load|stop]" ;; esac ``` ## Port Forwarding Connect over a jump proxy `$bastion`... - ...option `-A` ...forward `ssh-agent` (to authenticate to the gateway node) - ...to a gateway node `$gateway` (to access a network behind the bastion node) - ...forward the HTTPS port 443 from the target `$server` - ...option `-f` ...background - ...option `-N` ...do not execute a remote command ```bash ssh -J $bastion -A -L localhost:8080:$server:443 $gateway -fN ``` `boring`[^0c27b] is a simple command-line SSH tunnel manager. [^0c27b]: `boring` SSH tunnel manager, GitHub <https://github.com/alebeck/boring> # `rsync` Advanced alternative for the `cp` and `scp` commands: * Synchronizes data (files/directories) between a source and destination path * Either the source or the destination can be on another machine but not both * Special algorithm to minimize the network traffic ```bash rsync -r <src> <dst> # copy source directory into destination rsync -r <src>/ <dst> # copy content of source directory into destination # remote locations can be specified with a host-colon syntax rsync <src> <host>:<dst> # copy to a remote node rsync <host>:<src> <dst> # copy from a remote node ``` - Special treatment to source directories with a **trailing slash `/`**... - ...copies the contents of the source directory to the destination `-e`, `--rsh` (remote shell) option to **overwrite the remote shell** connection... ```bash # SSH option to connect with a non-default port rsync -e 'ssh -p 2022' ... # multiple options including variable substitution rsync -e "ssh -o ProxyJump=${USER}@example.domain -o PubkeyAcceptedKeyTypes=+ssh-rsa"... ``` Show **changes without modification** to the destination... - `-n`, `--dry-run` perform a trial run with no changes made - `-i`, `--itemize-changes` output a change-summary for all updates Output format for itemize changes: ```txt YXcstpoguax path/to/file ||||||||||| `----------- the type of update being done:: |||||||||| <: file is being transferred to the remote host (sent). |||||||||| >: file is being transferred to the local host (received). |||||||||| c: local change/creation for the item, such as: |||||||||| - the creation of a directory |||||||||| - the changing of a symlink, |||||||||| - etc. |||||||||| h: the item is a hard link to another item (requires --hard-links). |||||||||| .: not being updated (...might have attributes that are being modified). |||||||||| *: rest of the itemized-output area contains a message (e.g. "deleting"). |||||||||| `---------- the file type: ||||||||| f for a file, ||||||||| d for a directory, ||||||||| L for a symlink, ||||||||| D for a device, ||||||||| S for a special file (e.g. named sockets and fifos). ||||||||| `--------- c: different checksum (for regular files) |||||||| changed value (for symlink, device, and special file) `-------- s: Size is different `------- t: Modification time is different `------ p: Permission are different `----- o: Owner is different `---- g: Group is different `--- u: The u slot is reserved for future use. `-- a: The ACL information changed ``` ## File Comparison **Quick check algorithm** by default... - ...looks for files that have changed in size or in last−modified time. - Modification options... - `--ignore-existing` skip updating files that exist on receiver - `--size-only` skip files that match in size, even if the timestamps differ - `-I`, `--ignore-times` checksum every file, even if the timestamps and file sizes match Use **128−bit checksum** for each file that has a matching size... - ...lot of disk I/O reading all the data - `-c`, `--checksum` checksum instead of "quick check" - `--checksum-choice=auto` overrides the checksum algorithm, "md4", "md5", and "none" ## Transfer Options Option `-a` **archive mode** equals `-rlptgoD` (not ACLs, hard links or extended attributes such as capabilities): - `-r`, `--recursive` recursive into directories - `-l`, `--links` copy symlinks as symlinks - `-p`, `--perms` preserve permissions - `-t`, `--times` preserve modification times - `-g`, `--group` preserve group - `-o`, `--owner` preserve owner (super-user only) - `-D` same as `--devices --specials` - `--devices` preserve device files (super-user only) - `--specials` preserve special files Option `-hP` **human readable progress**/speed indicator: - `-h`, `--human-readable` numbers in a human-readable format - `--progress` shows progress during transfer - `--partial` keep partially transferred files, enables to resume interrupted file transfer - `-P` same as `--partial --progress` **Remove extraneous files** from the destination with option `--del`. - `--del` alias for `--delete-during` - `--delete-during` receiver deletes during the transfer ## Include & Exclude Exclude files/directories with option `--exlude`: - Support multiple uses of the `--exclude` (and `--include`) option - Alternatively use `-files-from=-` to pre-filter a list of files... ```sh # exlcude all images... rsync -a --exclude '*.jpg*' ... # pass a list of files vis STDIN find $PWD -name "*.jpg" -printf %P\\0\\n | rsync --files-from=- ... ``` Pattern matching... - `*` matches any non-empty path component (it stops at slashes) - `**` to match anything, including slashes - `?` matches any character except a slash (`/`). - `[` introduces a character class, such as `[a-z]` or `[[:alpha:]]`. - In a wild-card pattern, a backslash `\` can be used to escape... - ...matched literally when no wild-cards are present [rsync]: https://rsync.samba.org/ # `sshuttle` Sshuttle[^LmpWI] is part **transparent proxy**, part **VPN** (Virtual Private Network), and part SSH: [^LmpWI]: Sshuttle Project, GitHub <https://github.com/sshuttle/sshuttle> <https://sshuttle.readthedocs.io> No need to install anything on the remote server… - …based on remote login over SSH (requires an account on the remote node) - …the **remote node needs Python installed** (mostly installed by default on Linux) - User requires **root access on `localhost`** (aka Sudo)… - …to modify your system firewall - …to tunnel all traffic through a remote SSH connection Connect to a remote network… ```bash # First the `sudo` password is prompted …second prompt for the SSH login password sshuttle --dns --remote [user@]host[:port] --daemon --pidfile=/tmp/sshuttle.pid 0/0 # disconnect kill $(cat /tmp/sshuttle.pid) ``` - Option `--dns` to enable **DNS tunneling** (use DNS service from the remote node) - It is possible to start multiple connections to different networks …furthermore connections can be nested ## Configuration Files Path to a configuration file **preceded by `@` character**: ```bash # create a configuration file in your home-directory mkdir ~/.sshuttle cat > ~/.sshuttle/example.conf <<EOF value --option1 value1 --option2 value2 EOF # pass the configuration path as argument sshuttle @~/.sshuttle/example.conf ``` - …**arguments read from a file must be one per line** - …arguments to the commands take precedents over the configuration file ## Connection Management `sshuttles`[^793T0] is a simple wrapper-script to manage multiple connections… - …reading configuration files from a directory (`~/.sshuttle` by default) - …writes PID files to `/tmp/$USER-sshuttle*.pid` …one per connections [^793T0]: `sshuttles` Script, GitHub <https://github.com/vpenso/scripts/blob/master/bin/sshuttles> ```bash # list the configuration filenames (no path, no suffix) sshuttles list # start a connection by specifying the name sshuttles start $config_name # stop a specific connection sshuttles stop $config_name ``` Why? …many complex connection configurations to many different targets, for example: ```bash cat > ~/.sshuttle/node_example_org.conf <<EOF # target networks 172.100.0.0/16 10.0.0.0/8 --daemon --dns --disable-ipv6 # node used for SSH login --remote node.example.org # exclude following network segments from tunneling --exclude 172.100.60.0/24 --exclude 172.100.3.0/24 --pidfile /tmp/sshuttle-node_example_org.pid EOF ``` # `sshfs` ```bash # install on CentOS (requries EPEL) yum install -y fuse-sshfs # install on Debian apt install -y sshfs # install on Arch pacman -S sshfs ``` Mount a directory from a remote host ```bash # mount a remote directory sshfs -C -o reconnect,auto_cache,follow_symlinks [user@]host[:port] /mnt/path # unmount the remote directory fusermount -u /mnt/path ``` Alternatives... - BASSHFS - Bash-Accessible SSH File System - <https://pkolano.github.io/projects/basshfs.html> - <https://github.com/pkolano/basshfs> - `sshocker` - <https://github.com/lima-vm/sshocker> # Expect `expect` [^NFwN6] automates interactive applications such as SSH… [^NFwN6]: `expect` Documentation <https://core.tcl-lang.org/expect/index> ```bash sudo dnf install -y expect ``` …for example to automate the password prompt during SSH login: ```{.bash filename=expect-ssh} #!/usr/bin/expect set timeout 20 set password "$env(USER_PASSWORD)" set user "$env(USER_NAME)" set node [lindex $argv 0] set cmd [lrange $argv 1 end] set ssh_options "-o PreferredAuthentications=password -o PubkeyAuthentication=no" eval spawn ssh $ssh_options $user@$node $cmd expect "assword:" # matches both 'Password' and 'password' send -- "$password\r"; interact ``` Script | Description -------|------------- [`expect-ssh`][SN2Ni] | Execute a command over SSH on target [`expect-scp`][IxK45] | Copy a file over SSH to/from target [`expect-ssh-sudo`][pKFxP] | Execute a command over SSH on target, command requires a subsequent password prompt (for example `sudo`) [`expect-ssh-setpwd`][Qngh5] | Execute a command over SSH on target, command requires a subsequent password prompt, and sets and confirms a new password [SN2Ni]: https://github.com/vpenso/scripts/blob/master/bin/expect-ssh [IxK45]: https://github.com/vpenso/scripts/blob/master/bin/expect-ssh [pKFxP]: https://github.com/vpenso/scripts/blob/master/bin/expect-ssh-sudo [Qngh5]: https://github.com/vpenso/scripts/blob/master/bin/expect-ssh-setpwd The scripts require following environment variables to be set: Environment Variable | Description -|---------------- `EXPECT_LOGIN_PASSWORD` | Login password on the target, password used for subsequent prompts (for example `sudo`) as well `EXPECT_LOGIN_USER` | Login user name on the target `EXPECT_SET_PASSWORD` | Password to be set & confirmed (for example during user creating) Prepare the environment: ```bash # handy aliases alias essh=expect-ssh alias escp=expect-scp # set require environment variables export EXPECT_LOGIN_USER=$USER export EXPECT_LOGIN_PASSWORD=secret123 # usage examples expect-ssh $fqdn ls -l /tmp expect-scp /bin/bash $fqdn:/tmp EXPECT_SET_PASSWORD=secret321 expect-ssh-setpwd $fqdn passwd ``` # `rclone` [Rclone](https://rclone.org/) manages files on remote storage... - ...preserves timestamps ...verifies checksums - ...transfers over limited bandwidth ...intermittent connections - `rclone mount` ...as a file system with FUSE ```sh # configure the SFTP connection cat >> ~/.config/rclone/rclone.conf <<EOF [example.org] type = sftp host = login.example.org EOF # mount a path from the remote storage using SFTP rclone mount example.org:path/to/ ~/example.org ``` # `opkssh` OpenPubkey SSH[^WZPom] - …adds user public keys to OpenID Connect (OIDC)[^AfJYM] - …protocol builds on the OpenPubkey[^LA0dK] …uses SSH Certificates extensions [^WZPom]: OpenPubkey SSH, GitHub <https://github.com/openpubkey/opkssh> [^LA0dK]: OpenPubkey, GitHub <https://github.com/openpubkey/openpubkey> [^AfJYM]: Open-sourcing OpenPubkey SSH (OPKSSH): integrating single sign-on with SSH, Cloudflare <https://blog.cloudflare.com/open-sourcing-openpubkey-ssh-opkssh-integrating-single-sign-on-with-ssh/>

ssh

scp

~/.ssh/config

ssh-keygen & ssh-copy-id

ssh-agent