Command-line Shell Quick Reference

Terminal

Published

July 8, 2010

Modified

November 20, 2024

Why, learn it?

long lasting knowledge unlike many other computer skills
powerful expressive way of communicating with a computer
complete (technically advanced) control over the system components
facilitates automation for time consuming repetitive tasks
diversity of tooling and interfaces (makes difficult tasks possible)
improves productivity with increased proficiency

Note: often the terms shell, terminal and console are used as synonym

Shells are a user interface to system functions and other applications

Shells fall into one of two categories – command-line and graphical
Terminal emulator, wrapper interfacing a shell with the OS, i.e. xterm
Interface provided by a command-line interpreter CLI eg. bash
Text user interface TUI runs in a terminal, i.e. vim (text editor)

References…

Interpreter

The prompt indicates readiness to accept command input by the user

Literally prompts the user to take action
Customizable with environment variables
May include information like user, hostname, working directory, etc.

Commands are a large vocabulary of instructions and queries:

Type	Description
Internal	Recognized and processed by the command line interpreter itself
Included	Separate executable file part of the operating system
External	Executables added by other parties for specific applications

Anatomy of a command:

# list files & directories in your home-directory
ls -lh $HOME
├┘ ├─┘ └───┴───────────────────── argument
│  └───────────────────────────── options
└──────────────────────────────── command

Parameters are required and/or optional arguments/options

Arguments are positional inputs to the command
Options (flags/switches) modify the operation of a command

Delimiters - white space characters and the end-of-line (EOF) delimiter

`echo` & `cat`

echo displays lines of text

Accepts a list of strings as arguments
Quotes: preserve white-space in text using '...'
Escapes: -e enables interpretation of backslash escape sequences
- \t horizontal tab, \n new line
- -n do not output the trailing newline

echo some simple text                # some simple text
echo text 'can be   quoted'          # quoted text with white-spaces
echo 'quote text to show "quotes"'   # quote text to show "quotes"
echo -e 'text with a\ttab'           # text with a  tab

cat prints file content to standard output:

Special characters: -A show ^I tab and $ end of line
Line numbers: -n number each output line

cat /etc/hostname      # show file containing your local host name
echo "\tone\011two\040three\0" | cat -A # ^Ione^Itwo three^@$

Shell Modes

Two modes of operations:

Interactive
- Reads user input at the command prompt, enable users to enter/execute commands
- Shells commonly used in interactive mode started by default upon launch of a terminal
Non-interactive
- A shell executing a script (no human interaction) is always non-interactive
- Scripts must be executable chmod +x <script>

`man` & `help`

Many ways to find information…

Most programs (commands) provide manual pages (manuals):
- Detailed documentation, usually with references to other documents
- man uses a terminal pager program, i.e. more or less to display
- info, official documentation format textinfo of the GNU project
Convention…options -h, --help with a usage summery

In general documentation is accessible with following commands:

man <command>                  # man page for a given command
info <command>                 # documentation of the GNU project
whatis <command>               # display one-line manual page descriptions
apropos <keyword>              # search for commands by keyword
## bash builtin help
help                           # list of builtin commands
help -m <command>              # help for a specific builtin command

Unix Philosophy

Set of software development concepts: modularity and reusability

Best practices to designed a program for composability

Write programs that do one thing and do it well
Write programs to work together (be composeable)
Write programs to handle text streams (a universal interface)

Filter program, reads from standard input, writes to the standard output

Compose multiple programs into a chain with pipelines |

# following one-liner simulates the rolling of a traditional die with 6 faces
>>> seq 1 6 | shuf | head -n 1
3

the example above composes three programs into a pipeline:
- seq 1 6 print integers numbers from 1 to 6
- shuf randomly permutes lines from an input text stream
- head -n 1 extracts the 1st line from an input text stream

`ls`, `cd`, `z`

Bare minimum to know about the file-system tree, files and directories…

~                        # abbr. for the home directory (like $HOME)
/                        # root directory
/<dir>[/<dir>[/<file>]]  # absolute path, starts with / for the root directory
.                        # current directory
..                       # parent directory
<dir>[/<dir>/]           # relative sub-directory
# commands...
ls $path                 # list contents of a directory
pwd                      # print working directory
cd $path                 # change to specified directory
cd                       # change to home directory of the login user
cd -                     # change to previous directory
touch $path              # create an empty file with specifed path and name
rm $path                 # delete a file (permanently)
mkdir $path              # create an (new) empty directory in the tree
rmdir $path              # remove a directory if it is empty
cp $spath $dpath         # copy a file
mv $spath $dpath         # move a file/directory
file $path               # determine file type
cat $path                # print file content
ln -s $path $link        # create a symbolic link

Zoxide ¹ is a smarter cd command

# install required packages
sudo dnf install -y fzf zoxide

# load Zoxide int your environment
eval "$(zoxide init zsh)"

Use… zoxide --help

# work like regular cd
z $path   # highest ranking match
zi $path  # interactive selection

# list the database with scoring
zoxide query -ls

Cursor & Key Bindings

text cursor (aka caret) indicates the current position for user interaction
- typically underscore, rectangle or vertical line, flashing or steady
- indicates the insertion point in text mode
- moved by pressing arrows, page up/down, home/end key, etc.
key bindings tie key sequences to specific functions, cf. man readline

Key	Description
tab	command line completion
ctrl-r	search command history
ctrl-c	break current execution
ctrl-a	cursor to beginning of line
ctrl-e	cursor to end of line
ctrl-l	clear screen

aka keyboard shortcuts, key-combinations to invoke an action

`chmod` & `chown`

ls -l lists files & directories with ownership and permission

>>> ls -ld /tmp              # ownership and permissions of /tmp
 drwxrwxrwt 16 root root 4.0K Nov 15 07:44 /tmp/
  ├───────┘    ├──┘ ├──┘
  │            │    └─────────── group
  │            └──────────────── owner
  └───────────────────────────── permissions

Alternatively use the stat -c <format> command:

>>> stat -c '%A %a %U %u %G %g' /tmp
drwxrwxrwt 1777 root 0 root 0
 ├───────┘ ├──┘ ├──┘ │ ├──┘ └─ gid
 │         │    │    │ └────── group
 │         │    │    └──────── uid
 │         │    └───────────── user
 │         └────────────────── permissions octal
 └──────────────────────────── permissions human readable

The human readable permission consists of three permission triads:

…configured with a selection of three characters from [r-][w-][x-st]
The setuid/gid bits allow users to run an executable with the permissions the owner or group
The Sticky bit only owner and root can rename/delete files/directories

r             read
w             write
x             execute
-             no read/write/execute
s             siduid/setgid
t             sticky bit

Change permissions with the chmod command. The format of <mode> is a combination of 3 fields:

Effected triad u user, g group, a all or o others
+ add permissions, - remove permissions
Permissions, a combination of [rwxsStT]

id [<user>]                     # show user/group ID
chgrp <group> <path>            # change group ownership
chown <user>[:<group>] <path>   # change user/group
chmod <mode> <path>             # change permissions
umask <mode>                    # set default permissions
umask -S                        # displays mask in simbolic notation
getfacl                         # show ACL permissions
setfacl                         # modify ACL permissions

Examples:

chmod +r ...              # read for everyone
chmod ug+rx ...           # add read/execute for user/group
chmod a-r ...             # remove read access for everyone
chmod ugo-rwx             # remove all permissions
chmod u=rw,g=r,o= ...     # user read/write, group read, no access for others
chmod -R u+w,go-w ...     # set permissions recursive

Octal notation, think of rwx an binary variables r*2^2 + w*2^1 + x*2^0

7  rwx  111
6  rw-  110
5  r-x  101
4  r--  100
3  -wx  011
2  -w-  010
1  --x  001
0  ---  000

Examples

chmod 1755 ...            # sticky bit
chmod 4755                # setuid
chmod 2755                # setgid

`history`

The shell maintains a list of recently executed commands, so called events.

history       # show command history
fc            # invoke editor to edit last command
fc n          # edit command n in history

History event designators:

!             # start to reference history event
!!            # previous command
!*            # last argument list
!n            # command n in history
!-n           # the n preceding commands
!s            # most recent command starting with string s
!?s           # most recent command containing string s

Event modifier appended to an event, e.g. !3:* (use multiple modifiers like !!:-4:p:

:              # modifier prefix
:n             # nth word (word 0 is the command)
:^             # first word
:$             # last word
:-             # all words until the last
:-n            # all words including the nth
:m-n           # words from m to n
:n*            # wall words from n to the last
:*             # all words except the command
:p             # print, but not execute
:r             # remove the filename extension
:e             # remove all but the file name extension
:h             # remove last part from a path
:s/P/S/        # replace first match of pattern P with string S
:gs/P/S/       # like above, but replace all matches

Quick substitute ^P^S^ similar to !!:s/P/S/.

setopt kshoptionprint && setopt | grep hist      # show Zsh history settings
printenv | grep HIST                             # ^^ show environment variables

`hstr`

HSTR (HiSToRy)…

…shell history suggest box
…easier and more efficient than Ctrl-r
…manage command history…
- …remove commands
- …bookmark favorite commands
- …blacklist commands

Configuration in the shell profile…

alias '?'=hstr
setopt histignorespace
export HSTR_CONFIG=hicolor,help-on-opposite-side
export HSTR_PROMPT="> "

`alias`

alias       # defines a new alias command
unalias     # unset an alias command

aliases for the ps and pstree commands for process listings

# List of all processes including their hierarchy
alias psa='ps -AfH'
# Comprehensive inspection of the process tree
alias pstree='pstree -lpu'
# Print a sorted list of all processes by CPU utilization
alias pscpu="ps -A -o ruser,pcpu,time,state,args --sort pcpu"
# Continuously updated list of all running processes
alias prun="watch -n1 ps r -AL -o stat,pid,user,psr,pcpu,pmem,args"
# include cgroup
alias psc='ps xawf -eo pid,user,cgroup,args'

aliases for the ssh command

alias sr='ssh -l root'
alias sA='ssh -A'
alias sl='ssh-add -l'
alias ssh-no-checks='ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
ssh-tmux() {/usr/bin/ssh -t $@ "tmux attach || tmux new";}

`su` & `sudo`

su command stands for “switch user”, and requires the password of the account to be used.

su                         # switch to root user
su <user>                  # switch to a specific user
su - ...                   # invoke a login shell for another user
su -p ...                  # preserve caller environment variables
su - -c '<command>' <user> # execute a command as different user
gksu -u <user> <app>       # start a GUI application

Example

# start X applications as another user
xhost +                  # grant users access to your display
su - <user>              # switch user sourcing profiles

# after login as another user
export DISPLAY=:0.0      # export DISPLAY environment variable

sudo allows to execute a command on behalf of another users without requiring the password.

/etc/sudoers               # main configuration file for policies
/etc/sudoers.d/*           # additional configuration files    
sudo -ll                   # show user sudo configuration
sudo -lU <user>            # show configuration for another user
sudo su - <user>           # invoke a login shell as user
sudo -i <user>             # ^^
sudo -s <user>             # executes $SHELL as user
sudo -E <command>          # preserver environment variables (no shell functions)

Path variables are not inherited with option --preserve-env, therefore set them explicitly

sudo -E PATH=$HOME/bin:$PATH -- env | grep PATH

Run a shell with option --shell to execute multiple commands:

sudo -sE PATH=$HOME/bin:$PATH -- <<EOF
        id
        pwd
        env | grep PATH
EOF

Write content to a file:

sudo tee -a /etc/motd -- <<EOF
Append message of the day
EOF

Configuration

Basic policy:

# comment a policy
<user> <host>[,<host>,...] = (<user>[,<user>,...]:<group>[.<group>,...]) [NOPASSWD:] <command>[,<command>,...]

Aliases:

# list of one or more hostnames, IP addresses, network numbers or netgroups
Host_Alias <NAME> = <ip>,[<ip>,...][,<hostname>[,<hostname>,...]
Host_Alias <NETWORK> = <ip>/<mask>
# alias is list of one or more users, groups, uids etc.
User_Alias <USERS> = <user>[,...][,%<group>,...], 
# list of commandnames, files or directories
Cmnd_Alias <COMMAND> = <command][,<command>,...]

Examples

vpenso ALL=(ALL) ALL       # user vpenso gains root priviliges
%wheel ALL=(ALL) ALL       # group wheel gains root priviliges
# user vpenso can execute apt as root
vpenso ALL = (root) /usr/bin/apt
# host shutdown
Cmnd_Alias POWER = /bin/systemctl poweroff
# allow password changes with the exception of root
Cmnd_Alias PASSWD = /usr/bin/passwd [A-z]*, !/usr/bin/passwd root
# vpenso can execute defined by command aliases
vpenso localhost = (root) NOPASSWD: POWER,PASSWD

Files & Directories

Typically data stored in a file-systems is organized within a tree structure.

This tree structure consists of parent and child directories in a hierarchical order.
A directory (folder) contains files and further “child” directories.
A file contains data stored in the file-system.
The root directory is the top-most directory in the tree.
The current working directory is the location of the user while navigating in the directory tree.
A sub directory (aka child directory) is hierarchically below the working director.
Each user has a home directory (login directory) for personal data.

Following shows the hierarchical listing of the home directory of a use “jdow”:

/home/jdow
├── /home/jdow/bin
├── /home/jdow/docs
│   ├── /home/jdow/docs/manual.pdf
│   └── /home/jdow/docs/readme.md
├── /home/jdow/music
│   ├── /home/jdow/music/song.mp3
│   └── /home/jdow/music/sound.mp3
└── /home/jdow/var

Navigate the directory tree:

pwd                  # print working directory
cd <path>            # change to specified directory
cd                   # no path argument changes to the home directory of the login user
cd -                 # change to previous directory
pushd                # push directory to stack
popd                 # remove directory from stack
dirs                 # list directory stack
cd ~<n>              # change to nth directory from stack
touch <path>         # create an empty file with specified path and name

Manipulating files & directories:

rm <path>            # delete a file (permanently)
rm -r <path>         # remove a directory and all its content (recursive decent)
mkdir <path>         # create an (new) empty directory in the tree
mkdir -p <path>      # ^^ recursive create of a new directory (missing parents included)
rmdir <path>         # remove a directory if it is empty
cp <path> <path>     # copy a file
cp -R <path> <path>  # copy a directory
mv <path> <path>     # move a file/directory
rename <path>        # rename multiple files according to pattern
ln -s <path> <link>  # create a symbolic link

Work with files:

file              # determine file type
stat              # display file status
cat               # print file content
head              # show the beginning of a file
tail              # show the end of a file
less              # pager for files
pg                # ^^
more              # ^^
wc                # count line/chars in a file
touch             # create empty file, change timestamp 
nl                # number lines of a file

bat replacement for cat

Path

A path specifies a unique location in the directory tree:

A path is separated by slashes / like /home/jdoe/docs.
An absolute path starts with a slash, hence specifies a “full” path to a location in the tree.
A relative path specifies a path in relation to the current working directory.

Paths are constructed with the following notation:

~                        # abbr. for the home directory (like $HOME)
~+                       # current directory (like $PWD)
~-                       # previous working directory
/                        # root directory
### absolute path
/<dir>[/<dir>[/<file>]]  # starts with / for the root directory
### relative path
.                        # current directory
..                       # parent directory
../..[/..]               # parent of the parent directory
<dir>/[<dir>[/<file>]]   # sub/child directory/file
../<dir>[/<dir>/]        # relative to parent directory
~/<dir>[/<dir>/]         # relative to the (login user) home directory
~<user>/<dir>[/<dir>/]   # relative to a specific user home directory 
./.<path>                # hide a path by prefixing it with a dot

`ls` Aliases & Colors

ls (list) show the contents of a directory-tree:

alias ls='ls -Fh --color=always'   # default to classify, human readable sizes and colors
alias lS='ls -lS'                  # displays file size in order
alias l='ls -1'                    # only names, one per line
alias ll='ls -l'                   # long format
alias l.='ls -lA -d .*'            # only hidden files

dotfile treated as hidden, i.e. ~/.profile:

File/directory name preceded by a dot .
Not displayed by default in a directory listing

The classify option -F appends symbols to filenames:

*       executable
/       directory
@       link
=       socket
>       door (IPC)   
|       named pipe

The LS_COLORS variable is used to define colors for the ls command

Use the dircolors program for a more complex configuration:
lsd the next gen ls command

# a simple file with a color configuration
cat > ~/.dircolors <<EOF
TERM screen-256color
RESET                 0
FILE                  00;38;5;241
DIR                   00;38;5;244
LINK                  00;38;5;239
EOF
# load the configuration on shell init
echo 'eval "$(dircolors -b ~/.dircolors)"' >> ~/.profile

`tree` & `eza`

tree lists directories in a tree like format:

alias td='tree -d'                 # list only dirs 
alias t2='tree -L 2'               # max recursive depth of 2 levels
alias tu='tree -pfughF --du'       # permissions, user, group, sizes

eza ² is a modern replacement for ls

export EXA_ICON_SPACING=2
local opts="--icons --color=never"
alias ls="eza -F $opts"
alias l="eza -1 $opts"
alias ll="eza -alF $opts"
alias lt="eza --tree --level=2 $opts"

Naming Conventions

Naming files/directories:

File names typically use A–Za–z0–9._- alphanumeric characters (mostly lower case), underscores, hyphens, periods
- or _ separates logical words, e.g. this_is_a_file.txt (note that CamelCase is normally not used)
Executables (including shell scripts) usually never have any type of extension
Files with a specific format use dot . to separate the (file-type) extension, e.g. image.jpg
- Identifier specified as a suffix to the name
- Indicates a characteristic of the file contents or its intended use
Capitalized file names are used for high-lighting e.g. README.md
Following characters must be quoted if they are to represent themselves:
- |&;<>()$\"' as well as ␣⇥↵ space, tab and newline
- *?[#~=% depending on conditions
- Quoting mechanisms are the escape character \, single-quotes (literal value) and double-quotes ($ variable expansion)

# examples on escaping files names
>>> touch 'foo?bar' foo\ bar foo%bar foo~bar "foo*bar" foo\$\ bar
# list file names with special characters
>>> ls | grep -E '\s|\*|\?|\$|%'
foo bar
foo?bar
foo$ bar
foo*bar
foo%bar

`mktemp` Temporary Data

Create a temporary file in /tmp…

…and print its name
- …-d to create a directory instead
- …-p $path change the path

# create an empty file in /tmp
>>> mktemp       
/tmp/tmp.CgwZK2lgbN

# create an empty directory
>>> mktemp -d     
/tmp/tmp.73Akad37rK

# adjust the path
>>> mktemp -p /var/tmp            
/var/tmp/tmp.XpDoIxU3ON

Optional first argument defines a template for the name…

…defaults to /tmp/tmp.XXXXXXXXXX
…must contain at least 3 consecutive X in last component

# ..custom prefix in the name...
>>> mktemp -d /tmp/some-prefix.XXXXXX
/tmp/some-prefix.AfHubr

Input/Output & Pipes

`|` Pipes

A composable program architecture allows to chain multiple programs.

A key character is | (pipe):

a | b   # run a and b, send output of a as input to b, print output of b
a | b | c ...  # chained pipelines can build complex process structures

Programs (processes) have three standard data streams:

STDIN - Input read by a program
STDOUT - Output generated by a program
STDERR - 2nd output for errors messages & debug diagnostics

“Redirection” uses data streams to compose interprocess communication

Why is this useful?

Print the host IP configuration with the ip command, and limit its output to the first five lines with the head command:

» ip addr | head -n 5 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN ...
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host

Select a subset of the output matching a pattern with the grep command:

» ip addr | grep 'inet '                 
    inet 127.0.0.1/8 scope host lo
    inet 140.181.84.103/18 brd 140.181.127.255 scope global dynamic eno1
    inet 10.1.1.1/24 brd 10.1.1.255 scope global nbr0

Limit the output to specific columns using awk (text-processor):

» ip addr | grep 'inet ' | awk '{print $2 " " $NF}'
127.0.0.1/8 lo
140.181.84.103/18 eno1
10.1.1.1/24 nbr0

Standard Input/Output

STDIO data streams automatically opened at program start:

STDIN - Usually keyboard input, or redirect from file(s)/program(s)
STDOUT - Output to console (display) and/or redirected to file(s)/program(s)
STDERR - Also sent to console or redirected to a log file

I/O streams identified by an integer file descriptor (file handle):

Integer	Name	Stream
0	Standard input	`STDIN`
1	Standard output	`STDOUT`
2	Standard error	`STDERR`

Redirect streams between programs and files:

a | b   # STDOUT of program a to STDIN of process b
a > f   # STDOUT of program a to file f (same as a 1> f)
a >> f  # append STDOUT of program a to file f
a < f   # input content of file f to STDIN of program a (same as a 0< f)

`>` Redirection

Shells provide a versatile way to compose programs:

> f                   # create an empty file f from empty STDIN (:> f)
a 1>&-                # close STDOUT of program a
a 2> f                # write STDERR of program a to file f
a > f 2>&1            # write STDOUT & STDERR of program a to file f
a &> f                # ...same as above, in a short notation
# redirect content of file f to STDIN of program a, and...
a < f > g             #...write STDOUT or program a to file g  
{a ; b} > f           # redirect STDOUT of programs a and b to file f
a |& b                # pipe STDOUT & STDERR of program a to STDIN of program b
a | tee f             # redirect STDOUT of program a to console and file f
a |:                  # redirect STDOUT & STDERR to pipeline sink (&>/dev/null)

Redirect to/from a sub-shell:

a <(b)                # redirect STDOUT of program b to STDIN of program a
a >(b)                # redirect STDOUT of program a to STDIN of program b
(a ; b) > f           # redirect STDOUT of programs a and b to file f
# redirect STDOUT of program a to STDIN of program b and c...
a >(b > f) >(c > g)   #...both programs a and b write to files f and g

File Descriptors

Simple example script with permanent output redirection:

#!/usr/bin/env bash
# write STDOUT to file (temporary redirect for a single command)
echo one 1>/tmp/file
# permanent redirect (append) STDOUT to file from here one
exec 1>>/tmp/file
echo two
echo three

The exec program is used it opening file descriptors:

exec 3< f          # open file f for reading on file descriptor  3
exec 8<> f         # open file f for reading and writing on file descriptor 8
exec 6>> f         # open file f for appending on file descriptor 6
exec 5<&0          # copy read file descriptor 0 onto file descriptor 5
exec 7>&4          # copy write file descriptor 4 onto 7
exec 3<&-          # close the read file descriptor 3
exec 6>&-          # close the write file descriptor 6

Preserve references to STDOUT and STDIN file descriptors:

exec 3>&1 ; exec 4>&2         # preserve
# ...run code here...
exec 2>&4 ; exec 1>&3         # reset

Process & Filter

Command-line provides a versatile collection tools to print & filter text

echo                    # display text
cat                     # concatenate, print files
grep                    # print lines matching a pattern
sort                    # sort lines of a text file
uniq                    # report or omit repeated lines
cut                     # remove sections from each line of files
tr                      # translate or delete characters
sed                     # stream editor for filtering and transforming text
awk                     # pattern scanning and processing language
fmt                     # text formatter
paste                   # merge lines of files
split                   # split files into pieces
tac                     # concatenate and print files in reverse

Some examples for conventions to describe characters

Escape sequence: \[abfnrtv] starting with a backslash
Hex code: \x followed by 1 or 2 digits
Unicode: \u 16bit, U 32bit followed by 1 to 4 digits

echo "\x48allo"                      # Hallo
echo "\u00A9"                        # ©

`grep`

Search for patterns in text…

grep will read lines from a text that has a match for a specific pattern
useful to find lines with a specific pattern in a large body of text, eg.:
- look for a process in a list of processes
- spot check a large number of files for occurrence of a pattern
- exclude some text from a large body of text
variants of the grep command:

agrep         # approximately matching a pattern
egrep         # extended regex
rgrep         # recursive grep
zgrep         # search compressed files

options
- -i ignore case, -v inverse (lines that do not match pattern)
- -n line numbers, -c count matching lines
- -A<n> lines after match, -B<n> lines before match
- -o print only the matched expression (not the whole line)

A very important command to master…

# case insensitive search
>>> grep -i name /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
# use extended regex to match words
>>> grep -E -w 'bin|sys|adm' /etc/group
bin:x:2:
sys:x:3:
adm:x:4:
# match a regex in a file
>>> grep '^s[y,s].\:' /etc/group
sys:x:3:
ssh:x:112:
# match CIDR IP addresses from STDIN
>>> ip a | egrep -o '([0-9]{1,3}[\.]){3}[0-9]{1,3}\/[0-9]{1,3}'
127.0.0.1/8
140.181.84.103/18
# combine the find and grep commands 
>>> find $HOME/docs -iname "*.md" -exec grep '^# [A-Z].' {} +
...
.../docs/tools/ssh/host-fingerprints.md:# SSH Host Fingerprints
.../docs/tools/ssh/certs/host.md:# OpenSSH Host Certificates
.../docs/tools/ssh/certs/host.md:# SSH client connection configuration
...

`cut`

Cutting out sections from text…

cut: print selected parts of a line (fields) split by a delimiter
- field delimiter character: specified with -d (defaults to a TAB)
- fields separated by a single occurrence of the field delimiter character
- fields: -f lists the field numbers to print

text="A simple sentence! Nothing special to see..."
echo "$text" | cut -d' ' -f1-3,5    # A simple sentence! special
echo "$text" | cut -d! -f2-         # Nothing special to see...
echo "$text" | cut -d'.' -f1        # A simple sentence! Nothing special to see

byte position: -b cuts each line specified by a list of byte numbers
- always works in terms of single-byte characters
- character-list: -c does not support Unicode at the moment

echo "abcde12345" | cut -b3,7-      # c2345
echo "abcde12345" | cut -c-3        # abc
# unicode characters are 2 bytes long
echo "αβγδεζ" | cut -b 3-4,9-10     # βε

`sort`

Sort lines of text files…

Default …compares entire line’s …decides how to sort it…
Options for simple sort customization…
- …numerically or alphabetically
- …ascending or descending order

Option -k $keydef is used to select specific fields for comparison with following syntax:

F[.C][OPTS][,F[.C][OPTS]]

…F is a field number (or column)
- -k2 …second field to end of line
- -k3,7 …third to seventh field
…C a character position
…OPTS single-letter ordering options…[bdfgiMhnRrV]
- -k3n …third field to end of line …numerical sort
- -k2.3d …second field, third character …dictionary order

Multiple -k options will be sorted in order of the argument list…

cat <<EOF | sort -k2,2n -k3,3n -k1,1
cba 321 111
bac 123 111
bac 123 000
bac 213 000
bac 213 111
abc 213 111
bac 321 000
abc 123 111
EOF

`fzf`

fzf is similar to grep… https://github.com/junegunn/fzf

# ...on RPM based distributions
sudo dnf -y install fzf fd-find

…more suitable for interactive searches
- …user select match after initial filtering
- …single- or multi-text selection
…fuzzy finding capabilities…
- …approximate searches
- …any part of the match in any order
…pseudo user interfaces for command-line programs

`regex`

Regular expression (regex), defines a search pattern

([A-Za-z0-9-]+)                             # Letters, numbers and hyphens
(\w+@[a-zA-Z_]+?\.[a-zA-Z]{2,6})            # email address
(\<(/?[^\>]+)\>)                            # HTML tag
([0-9]{1,3}[\.]){3}[0-9]{1,3}\/[0-9]{1,3}   # CIDR IP addresses

ranges
- . any character (except newline)
- (a|e) a or e ((...) groups)
- [a-z0-9] range, lowercase letter or any digit
- [^b-d] invert of range b to d
quantifiers
- * 0 or more, ? 0 or 1, + 1 or more
- {N} N (num.), {N,} N or more, {,N} N or less, {N,M} between N & M
anchors: ^ start of line, $ end of line
escape: characters ^$()<>[{\|.+*? using \ (back-slash) as prefix

>>> regex '^[f,b].[o]' foo bar # test with the `regex` command
regex: ^[f,b].[o]\n
foo matches
bar does not match

CSV, JSON & YAML

Miller ³ …reformat data …formats CSV, TSV, JSON, and JSON

`jc`

jc ⁴ converts output of other commands to JSON

# input from STDIN...
df | jc --df -p
cat /etc/group | jc --group -p -r
echo "A,B,C\n1,2,3" | jc --csv
# ...command of file as argument
jc -p free
jc -p /proc/meminfo

`jq`

jq ⁵ filters a stream of JSON data…

…extract fields of an object
…conversion of numbers to strings

# object’s keys as an array
echo '{"a":1,"b":2}' | jq 'keys'
# get a named property
echo '{"a":1,"b":2}' | jq '.a'
echo '{"a":{"b":{"c":1}}}' | jq '.a.b.c'
# element by index
echo '[0,1,2,3,4,5]' | jq '.[3]'
echo '[{"a":1,"b":2},{"c":3,"d":4}]' | jq '.[1].d'
# slice an array
echo '[0,1,2,3,4,5]' | jq '.[2:4]'
echo '[0,1,2,3,4,5]' | jq '.[3:]'
# number of array elements
echo '[0,1,2,3,4,5]' | jq 'length'
# array of unique values
echo '[1,2,2,3]' | jq 'unique'
# flatten nested arrays
echo '[1,2,[3,4]]' | jq 'flatten'

`jless`

jless ⁶ is a command-line JSON viewer…

…supports Vi style key-binding
Collapse siblings with c/C and expand with e/E
…toggle collapse state with Space

# support for YAML output
alias yless="jless --yaml"

Search & Replace

`find`

Search files based on criteria…

find $HOME -name "README*" -exec wc -l {} +
           ├─────────────┘ ├──────────────┘
           │               └───────────────── action (optinal)
           └───────────────────────────────── criteria (optinal)

path: may have multiple paths, eg. find /usr /opt -name "*.so"
criteria
- -name "abc[a-z][0-9]" using a regular expressions
- -type (f,d,l) file, directory, sym. link
- -user <uname>, -group <gname>, -perm (ugo+/-rwx)
- -mmin [+-]n modified min. ago, -mtime days ago, -amin access time
- criteria may be combined with logical and -a and or -o
action
- -print: default action, display
- -ls: run ls -lids command on each resulting file
- -exec <cmd>: execute command
- -ok <cmd>: exec command after user confirms

Examples…

find ... -exec ... {} +            # pass file(s) as argument
find ... -exec ... {} \;           # use a subprocess to execute a command
find ... -print | xargs -r ...     # execute a command against the found files
find ... -print0 | xargs -r -0 ... # files separated using null byte to support
                                   # special chars, spaces, etc.

placeholder {} for file name(s)
; end of the command to exec, might need to be escaped (with a \)
+ append a list of file to the command (on call)

# text files in current directory, case insensitive
find . -type f -iname "*.txt"
# change permissions on directories (single call of chmod)
find . -type d -exec chmod 770 {} +
# files bigger then 512MB
find . -type f -size +512M -print
# delete all files that were not accessed in a year
find $HOME -type f -atime+365 -exec rm {} +
# embedded bash script consuming the list of files
find . -type d -exec bash -c 'for f; do echo "$f" ; done' _ {} +
# all files that have either .c or .h extension
find . \( -name "*.c" -o -name "*.h" \)

`paste`

Display to inputs side by side

>>> paste <(echo 'a\nb\nc') <(echo '1\n2\n3')       
a   1
b   2
c   3
# custom a delimiter 
>>> paste -d'-' <(echo 'a\nb\nc') <(echo '1\n2\n3')
a-1
b-2
c-3
# three inputs with custom delimiter
>>> paste -d'%|' <(echo 'a\nb\nc') <(echo '1\n2\n3') <(echo 'x\ny\nz')
a%1|x
b%2|y
c%3|z

Serialize multiple lines

# tab as default delimiter
>>> echo 'a\nb\nc' | paste -s
a   b   c
# custom delimiter
>>> echo 'a\nb\nc' | paste -s -d'-'
a-b-c

`sed`

Use sed (stream editor) by piping input or reading it from a file:

cat f | sed 'c'             # input from stdin, apply command c
sed 'c' f                   # input from file f apply command c
sed -i 'c' f                # modify file f in place with command c
sed -f f ...                # read commands from file f
sed 'c1;c2' ...             # multiple commands colon sep.
sed -e 'c1' -e 'c2' ...     # ^^ alternative

Stand-alone script spawns a shell calling sed, and printing output to standard out with $*.

sed 'command' $*

Call sed directly with a shebang like #!/bin/sed -f

Basic command format (omit address to process all input lines)

[address[,address]] instruction [arguments]

Use ; to separate multiple commands. Available instructions:

a\        append
c\        change
d         delete
i\        insert
n         next
N         next without write
p         print
q         quit
s/ / /    substitute (delimiter `/`)
w file    write

Substitution

Substitution commands can use an alternative separator e.g. sed "s'P'S'g" or sed "s|P|S|g". Examples for substitutions:

s/P/S/             # substitute first match of pattern P with S
s/P/S/g            # substitute all matches of pattern P with S
/M/s/P/S/g         # like above but only lines matching pattern M
s/^[ \t]*//        # delete leading white spaces
s/[ \t]*$//        # delete trailing white spaces
s/^/ /             # insert leading space 
N,M s/P/S/         # substitute only in lines between N and M
N,$ s/P/S/         # substitute after line N

Pattern flags are /g global, /I ignore case. & refers to pattern found, use \ to escape.

Use with option sed -n 'E' to suppress unselected input when using the print command:

/P/p              # print lines matching pattern P 
/P/!p             # print lines not matching pattern P
N,Mp              # print lines with numbers from N to M
/P1/,/P2/p        # print lines between pattern P1 and P2
/P/{p;q;}         # print first line matching pattern P

White Spaces

crush removes duplicate blank lines:

#!/bin/sed -f
/^[[:blank:]]*$/d

>>> echo "\n\na\nb\n\nc\n" | crush
a
b
c

pushin reduces multiple spaces to one

#!/bin/sed -f
s/[ ][  ]*/ /g

>>> echo "a   b      c" | pushin
a b c

chop removes trailing spaces

#!/bin/sed -f
s/[[:blank:]]*$//

>>> echo "a   b      c  " | chop | cat -A
a   b      c$

Scripting

A series of commands stored in a plain text file…

Why Learn to Program?

To understand our modern world
To use computers better
As a job skill
To use an important new form of literacy
As a medium in which to learn problem-solving
To study and understand processes
To have a new way to learn art, music, science, and mathematics

Why shell scripting?

Create custom tools & utilities specific to your work
Automate repressive tasks to free up time
Codify complex administrative task as reference for co-workers
Create simple applications for customers

Compared to “real” programming languages (like C/C++)

Easy to use
Quick start…interactive debugging
Fast development…very expressive
Executable on nearly all UNIX like systems
…compatibility problems between shell variants
…difficult to scale in scope (>1000 lines)

Each shell script consists of…

Keywords: such as if..else, do..while
Commands: such as pwd, echo, grep, find
Variables: storing data referenced by a name
Control flow: statements expressing program behaviour
Functions: to reference script segments by name

Interpreter Directive

Executing a shell script:

/bin/bash /path/to/script  # interpreter followed by the path to a script
bash /path/to/script       # assumes bash is in $PATH
/path/to/script            # (recommended) interpreted defined by shebang line

Interpreter directive: #! (shebang)

Indicates the interpreter used to execute the file
Must be in the first line of a script
Interpreter is the full path to a binary file eg. /bin/bash
Requires the executable bit chmod +x
Some examples…
- #!/bin/bash use Bash as interpreter
- #!/usr/bin/python execute using the python command
- #!/usr/bin/env ruby use env to find the ruby command and execute

Obligatory “Hello World” Bash script:

#!/bin/bash
echo "Hello World"

Variables

Variable name: symbolic placeholder for a value (data)

Convention to use lower_case_name (underscore as delimiter)
Opposed to constants and environment variables UPPER_CASE_NAME

Assignment: declare a variable using the = operator

date=2019/11/03    # no spaces!
temp=21.5C
name=alice
empty=             # assign an empty (null) value
# errors...
name= alice        # declares an empty variable "name", run alice command
name =alice        # tries to run a command "name" with one argument
name = alice       # tries to run a command "name" with two arguments

Reference: precede variable name with $ to access the value

echo $date $temp   # reference variables
echo name          # just text, not a variable reference

Build-in

Automatically set and heavily used inside shell scripts

Access command-line parameter in a shell script

$1 $2 $3...$n  # direct access to parameters (position arguments)
$@             # all parameters
"$@"           # each parameter in double-quotes ("$1" "$2"...)
"$*"           # all parameters in double-quotes ("$1 $2...")
$#             # number of parameters

Shell script process and run-time information

$$             # current shell process ID
$0             # name of executed script
$_             # absolute path to executed script
$?             # most recent exit status
$!             # most recent process ID

Environment Variables

Pre-set environment variables (exported to all child processes)

$PATH          # search path for executables (; separated)
$SHELL         # current interpreter
$PWD           # current working directory
$HOME          # user home directory

Example script to read (input) parameters using build-in variables

cat > /tmp/params <<'EOF'
#!/bin/bash
echo $#: "$@"
echo $1 $5
EOF
chmod +x /tmp/params
/tmp/params 1 a 3 b 6 c

$PATH represents the command search path

: colon-separated list of directories that are searched (left to right)
Standard system directories such as /bin, /usr/bin
Third-party or locally-installed software such as /usr/local/bin
Append personally-chosen directories of your scripts

# list the search path (one directory per line)
echo $PATH | tr ':' '\n'
# append our personal scripts directory (in the current shell)
PATH=$PATH:$HOME/bin
# make a shell variable available as an environment variable
export PATH

Expansion

Variable expansion with the character $

Weak quoting " (double quotes) expands arguments
Strong quoting ' (single quotes) does not expands arguments
Prevent expansion with the escape character \

's'              # preserves literal value of characters in string s
"s $V"           # preserves literal value of characters in string s except $ \ ,
\                # escape character (preserves the literal value)
"s \$V"          # using an escape sequenze to prevent variable expansion
"${V}s"          # name to be expanded in braces
$'s'             # expands string s with backslash-escaped characters replaced

`<<EOF` Here-Documents

Block of code or text which can be redirected…

Avoid substitution and expansion by quoting the tag (here EOF)
Using <<- suppresses leading tabs (indent closing tag with tabs

Write to a file,,,

cat > /path/to/file <<'EOF'
  # ...content
EOF

Assign to a variable…

var=$(cat <<EOF
  # ...content
EOF
)

Pipe to a command…

cat <<EOF | sort
2
1
3
EOF

Commands

Statement separator ;

Grouping:

{ … } groups commands purely for syntactic purposes (no sub-shell)
( … ) create a sub-shell and wait for it to terminate

Logic operators: and &&, or ||

c1; c2         # execute command c1 before c2
{c1; c2}       # execute commands in current shell environment
(c1; c2)       # execute the commands inside a sub-shell environment
c1 && c2       # if command c1 runs successful execute c2
c1 || c2       # if command c1 runs not successful execute c2
$(c)           # execute command c in sub-shell (alternative `c`)
c1 $(c2)       # command c1 uses output of c2 as parameters
c &            # execute command c in background

Command substitution: $(…) creates a sub-shell…

..with its standard output set to a pipe…
..collects the output in the parent and expands to that output..

Background: … & sub-shell, does not wait for it to terminate

List of build in commands to know…

set -v         # print shell input lines as they are read
set -x         # print commands and their arguments when executed
set -f         # disable globbing
source f       # execute file f in current shell environment
eval `c`       # execute command c in sub-shell and evaluate
shift          # remove leading positional parameter
jobs           # list active jobs associated with current shell
disown         # detach jobs from current shell
nohup          # continue job after logout
bg             # move job to background 
fg j           # resume job j to foreground
stop j         # stops background job j
trap c s       # execute command c when catching signal s
rehash         # re-index executables in $PATH (Zsh)
reset          # reset terminal

Statements

`if...then...else`

if statements are to most used control structure

Allows for branching in the code execution path
Closing fi is necessary, elif and else optional
Always include spaces around [[...]] the conditional operator

# evaluate the number of command-line parameters
if [[ $# -eq 1 ]] ; then
        # one argument 
elif [[ $# -ge 2 ]] ; then
        # two or more arguments
else
        # no arguments
fi

Evaluate a mathematical expression with ((...))

if (( $1 + 1 > 2 )); then
        #...
fi

Evaluate the existence of a command in the environment $PATH

if command -v <cmd> >/dev/null ; then
    ...
fi

Execute a command and use its exit status as condition to an if statement:

Every program returns an exit status as an integer number
0 (zero) means success, 1-255 non-zero means failure

if <cmd> ; then
    ...
else
    ...
fi

Exit Status	Description
1	catchall for general errors
126	command invoked cannot execute
127	command not found

`case`

case statement matches values against a variable
- argument to case is expanded and matched against patterns
- in case of a match run commands up to ;;
- patterns are not regular expressions but shell patterns (aka globs)

case "$1" in
        1)
                echo $1
                ;;
        abc)
                echo def
                ;;
        2|3|4)
                echo 2,3,4
                ;;
        [a-z]*)
                echo a-z
                ;;
        *)
                echo no case
esac

`for..do..done`

iterator: loop over a sequentially changed variable

for (( i = 0; i < 10; i++ ))
do
        echo $i
done
# two variables at once
for (( i = 0, j = 0; i < 10; i++, j = i * i ))
do
        echo $i $j
done

examples for for..in..

# brace expansion
for i in {1..10}; do 
    echo $i
done

# loop an array 
arr=(a b c d e f)
for i in "${arr[@]}";do
    echo $i
done

`while...do...done`

loop until a condition is true

# arithmetic
i=0
while (( $i < 10 )) ; do
        echo $i
        ((i++))
done
# comparison 
i=0
while [ $i -lt 10 ] ; do
        echo $i
        i=$[$i+1]
done

build an (infinite) endless loop…

while true ; do
    ...
done

Expressions

Conditional expressions:

&& (and), || (or) have no precedence (left-associative)
Bit faster the if...then

!e             # true if expression e is false
e1 && e2       # true if both expressions e1 and e2 are true (alternative e1 -a e2)
e1 || e2       # true if either expression e1 or e2 is true (alternative e1 -o e2)
[[e]]          # returns bool after evaluation of the conditional expression e
:              # null-statement (returns 0)
$[e]           # evaluate integer expression
((e))          # expand and evaluate of integer expression e
$((e))         # expand, evaluate and substitute integer expression e

Brace Expansion

Brace expansion

{a,b,c}                             # brace expansion
{a..z}                              # extened brace expansion, ranges
name.{foo,bar}                      # typical for files
name{,.foo}                         # includes name
{1..20}                             # digits
{01..20}                            # with leading zero
{20..1}                             # reverse
{0..10..2}                          # use increments
{a..d}{1..3}                        # combine braces

Operators

Typical use of the conditional operator [[...]]:

# string compare
s1 = s2         # true if strings s1 equals s2
s1 != s2        # true if strings s1 not equal s2
-n s            # true if string s is not empty
-z s            # true if string s is empty
# file conditions
-c f            # true if f is a special character file
-d p            # true if p is the path to an existing directory
-e f            # true if f is an existing file
-f f            # true if f is an existing file but not an directory
-G f            # true if f exists and is owned by effective group-id
-g f            # true if f exists and is set-group-id.
-k f            # true if f has sticky bit
-L f            # true if f is a symbolic link
-O f            # true if f exists and is owned by the effective user-id.
-r f            # true if f is readable
-S f            # true if f is socket
-s f            # true if f nonzero size
-u f            # true if f set-user-id bit is set
-w f            # true if f is writable
-x f            # true if f is executable
# numerical comparison
n1 -lt n2       # true if integer n1 less then n2
n1 -gt n2       # true if n1 greater then n2
n1 -le n2       # true if n1 less or equal n2
n1 -ge n2       # true if n1 greater or equal n2
n1 -eq n2       # true if n1 equals n2
n1 -ne n2       # true if n1 not equal n2

Empty vs Unset String

Distinguish between empty and unset strings:

case ${var+x$var} in
    (x) echo empty;;
    ("") echo unset;;
    (x*[![:blank:]]*) echo non-blank;;
    (*) echo blank
esac

`exit`

each shell command returns an exit code when it terminates
- 0 (zero) means success, 1-255 non-zero means failure
- special variable $?: exit status of the last executed command

# successfull execution of the date command
>>> date |: ; echo $?
0
# list a non-existing directory... returns a non zero exit status
>>> ls /foobar ; echo $?
ls: cannot access '/foobar': No such file or directory
2

if branching based on an exit status of a command (substitution)

…
        passwords=$(ccrypt -c $PASSWORD_FILE)
        # check if the password-name is present in the password-file
        if $(echo "$passwords"| grep "^$2" >/dev/null) ; then
                echo "$passwords" | grep "^$2" | cut -d',' -f2
        else
                echo "Password name '$2' does not exist"
                exit 1
        fi
…

exit command stops execution and returns an exit status

Functions

abstract a segment of program code with a function name name()

echo_n() { echo -n "$@" }
├────┘   ├──────────────┘
│        └────────────────── function name
└─────────────────────────── function body

# function call with arguments
echo_n 1 2 a b

function body defined by all code in between { … } (curly brackets)
function exit status: specified with return value of 0 to 255

return_one() { 
        echo ${1:-one} # print first argument or 'one'
        return 1       # return failure
}
if return_one # evaluates exit status
then
        echo success
else
        echo failure $(return_one 1)
fi

`source`

source executed a file in the current shell (abbreviated to .)

>>> echo '#!/bin/bash\necho $USER $name' | tee name.sh
#!/bin/bash
echo $USER $name
# creates a new (sub)shell
>>> name=vic ; bash name.sh
vpenso
# work in the current shell
>>> name=vic ; source name.sh 
vpenso vic

typically used modify environment variables by sourcing a file

>>> cat ~/.bashrc
if ! [ -z "$(ls -A ~/.bashrc.d)" ] ; then # if the directory is not empty
    for file in `\ls ~/.bashrc.d/*` ; do
        source $file       # execute files in current shell
    done
fi
# configure the shell history, cf. bash manual
>>> cat ~/.bashrc.d/history.sh
export HISTSIZE=100000                       # set history size
export SAVEHIST=$HISTSIZE                    # save history after logout
export HISTCONTROL=ignoreboth                # ignore leading spaces, duplicates

~/bashrc: customize your interactive shell session

Processes

A process is the execution of a program

…a command can only generate a process
…an application can run multiple processes for different tasks
…each process is assigned a unique PID (process identification number)

ps            # (process status) snapshot of all running processes
pstree        # display a tree of processes
pgrep         # search process ID by criteria
kill          # send a signal to a process
pkill         # search a process and send a signal
killall       # send a signal to a process by name
nice          # run a program with modified scheduling priority
renice        # alter priority of running processes
top           # dynamic real-time view of a running system

Starting a program from an interactive shell (within a terminal)…

…creates a process …inherits stdin, stdout, and stderr
…şhell blocked until process terminates …SIGHUP to shell propagates to process
Run a program without blocking the shell by appending an ampersand &
- …creates a process …inherits stdin, stdout, and stderr
- …added to the list of background jobs the shell manages
- …SIGHUP to shell propagates to process

<cmd> &         # start process in background of the current shell
nohup <cmd>     # detach a process from the current terminal
jobs            # processes started in the context of the current shell 
bg %<job_id>    # send send a process to the background
fg %<job_id>    # pull a process to foreground
<cmd> & disown   # detach from current shell

`disown` & `nohup`

# decouple from a shell
sleep 300 & disown

# decouple from terminal and shell
nohup sleep 300 & disown

disown removes the process from the shell’s job control…

…still connected to the terminal (for example pty for SSH logins)
…SIGHUP to shell does not propagates to process

nohup disconnects the process from the terminal

…redirects redirects stderr to stdout and stdout to $HOME/nohup.out
…may still receive signals from the origin shell (parent process)

`kill` & `trap`

Signals: one-way notifications for process

Used to manipulate the state of a processes
User typically use signals to terminate a process

kill command sends a signal to a process

Read about the available signals with man 7 signal
Get a short signal listing with kill -L

# shell buildin command behaves different to the external program
>>> type -a kill
kill is a shell builtin
kill is /bin/kill
>>> /bin/kill -L
 1 HUP      2 INT      3 QUIT     4 ILL      5 TRAP     6 ABRT     7 BUS
 8 FPE      9 KILL    10 USR1    11 SEGV    12 USR2    13 PIPE    14 ALRM
15 TERM    16 STKFLT  17 CHLD    18 CONT    19 STOP    20 TSTP    21 TTIN
22 TTOU    23 URG     24 XCPU    25 XFSZ    26 VTALRM  27 PROF    28 WINCH
29 POLL    30 PWR     31 SYS

Signals have a number value 9 and a name KILL

kill -9 <pid>        # kill a program using a signal number
kill -KILL <pid>     # similar using the signal name

Exit and error in a sub-shell, simple cases:

(c)                # execute command c in sub-shell, ignore errors
(c) || exit $?     # same as above but propagate signal code

Catching Signals

Execute commands COMMAND on signal SIG

trap 'COMMAND' SIG [SIG,..]

Trap breaking child processes

set -m
# rescue function executed by trap
ensure() {
        echo "Clean up after child process died with signal $?."
}
# catch errors
trap ensure ERR
# execute child process
segfaulter &    
# wait for the child process to finish
wait $!

`zsh` Shell

`bindkey`

…no argument …print all keybindings
-l …list of existing keymap names
-M <KEYMAP> …list all the bindings in a given keymap
- …Vi mode …keymaps viins and vicmd
Documentation…
- …manual page …man 1 zshzle
- …zle -al …list all registered line editor commands

Examples

# bind Ctrl-l ro run the `ls` command
bindkey -s "^L" 'ls^M'

Plugins

`zplug`

Zsh Plugin Manager…

Add following to ~/.zshrc:

export ZPLUG_HOME=~/.zplug

test -d $ZPLUG_HOME \
        || git clone https://github.com/zplug/zplug $ZPLUG_HOME

source $ZPLUG_HOME/init.zsh

zplug "zsh-users/zsh-completions"
# ...add more plugins

if ! zplug check; then
    zplug install
fi
zplug load

Terminals

Text terminals….

…serial computer interface for text entry and display
…historically…
- …physical terminals …like the IBM 3270, VT100
- …keyboard/screen for display and input of data on a remote computer
…terminal emulators …called tty
- …mimic a hardware (video) terminal in software
- …provide interactive access to programs that run in the command line
- VTE (virtual terminal emulator)…
  - …emulates a text terminal inside a graphical user interface (GUI)
  - …widgets that implement a fully functional terminal emulator

/dev/tty[0-9]               # teletypes, cf. man `tty`
/dev/pts/[0-9]              # pseudo terminals, cf man `pty`
tty                         # show assoc. pseudo terminal
stty -a                     # show terminal settings
getty                       # program watching a physical terminal (tty) port
ps -a                       # list processes with attached terminals
terminfo 
termcap                     # terminal capability data base
tput
tget
reset                       # init terminal

A system console is a special terminal…

On modern computers a directly connected monitor/keyboard
Receives messages from the OS regarding booting/shutdown progress
Linux support virtual consoles to provide several text terminals
Access with a key combination including function keys (e.g. Ctrl+Alt+F2)

Terminology…

Control characters display control codes like line-feed, backspace, etc.
Escape sequences…
- …series of characters that give commands to the terminal
- …user for cursor movement, colors, etc.
- …consists of the ESC control character followed by a sequence of characters
…cut & paste…
- …note that Ctrl-c breaks current execution in shell
- Ctrl-Shift-c and Ctrl-Shift-v for cut & paste

`gnome-console`

GNOME Console

…simple user-friendly terminal emulator
…minimalistic designed for Mobile devices as part of Phosh

`gnome-terminal`

GNOME Terminal…

…installed by default with GNOME
Options…
- --window …open a new window
- --title …set title of tab/window
- --tab …open a new tab in window
- --command …first command to execute…
  - …can be a specific shell Zsh
  - …or a terminal multiplexer like Tmux
Shortcuts …preference menu …shortcuts tab

# open two tabs called local and remote ...local tab starts Tmux
gnome-terminal --window --title remote --tab --title local --command tmux
# typically mapped to a keyboard shortcut like F2 or Alt+9

`tabby`

References…
- https://tabby.sh
- https://github.com/Eugeny/tabby

Encoding

System of rules to convert information (letters, words, sound, images) into another form or representation for data transmissions/storage…

character encodings are representations of textual data
decoding is the reverse process, converting code symbols back
ASCII: single byte encoding only using the bottom 7 bits (man ascii)
- first 32 codes (0–31 decimal) for control characters
- letters A-Za-z, digits 0-9, punctuation !#$%&'()*+,-. etc.
UTF-8 general-purpose way of representing Unicode characters
- encoded as sequence of 1-4 bytes (variable width character encoding)
- designed for backward compatibility with ASCII

locale                   # current language and encoding settings           
localectl                # ^^with systemd tools
file -bi <file>          # show file encoding
# change encoding from ASCII to UTF-8
iconv -f utf-8 -t ascii -o <out_file> <in_file>
# change encoding from UTF-8 charset to ASCII, omit invalid characters
iconv -c -f utf-8 -t ascii -o <out_file> <in_file>

ASCII & Unicode

ASCII], single byte encoding only using the bottom 7 bits

ASCII order (ASCIIbetical)
- Uppercase come before lowercase letters
- Digits and many punctuation marks come before letters
First 32 codes (0–31 decimal) for control characters
Many 8-bit encodings which are supersets of ASCII

UTF-8, general-purpose way of representing Unicode characters

Character encoded as sequence of 1-4 bytes (variable width character encoding)
Designed for backward compatibility with ASCII (valid ASCII text is valid UTF-8-encoded Unicode)

Notations & Escaping

Conventions to describe characters

Octal digit characters (01234567), \<octal> 1,2 or 3
Escape sequence, starting with a backslash \[abfnrtv]
Caret notation, ^ followed by a single character (usually a capital letter)

printf `\O`                         # print octal
printf '\xH'                        # print hex H
printf "\\$(printf %o D)"           # print decimal d

Non-Printable Characters

Non-printable (white-space) characters:

space (blank, word divider)
backspace (BS), \b, ^H
tab (horizontal tab (HT), \t, \011, ^I
newline (line feed (LF)), \n, \012, ^J
null (NUL) \0, ^@
escape (ESC) \e, ^[

cat show non-printable characters with -A (equivalent to -vET)

>>> s="\tone\n\011two\040three\0"
>>> echo "$s" | cat -A
^Ione$
^Itwo three^@$

^ and M- notation (100-137 ascii chars), for LF $

od show non-printable chars with backslash escapes:

>>> echo "$s" | od -c
0000000  \t   o   n   e  \n  \t   t   w   o       t   h   r   e   e  \0
0000020  \n
0000021

Footnotes

Zoxide Source Code, GitHub
https://github.com/ajeetdsouza/zoxide ↩︎
eze Source Code, GitHub
https://eza.rocks
https://github.com/eza-community/eza ↩︎
Miller
https://miller.readthedocs.io
https://github.com/johnkerl/miller ↩︎
jc JSON Convert
https://kellyjonbrazil.github.io/jc/
https://github.com/kellyjonbrazil/jc ↩︎
jq JSON Processor
https://stedolan.github.io/jq/manual/
https://github.com/stedolan/jq ↩︎
jless Command-Line JSON Viewer
https://jless.io/user-guide
https://github.com/PaulJuliusMartinez/jless ↩︎

--- title: Command-line Shell Quick Reference categories: - Terminal date: 2010/07/08 date-modified: 2024/11/20 toc-expand: 4 --- ## Why, learn it? - long **lasting knowledge** unlike many other computer skills - powerful **expressive** way of communicating with a computer - complete (technically advanced) **control** over the system components - facilitates **automation** for time consuming repetitive tasks - **diversity** of tooling and interfaces (makes difficult tasks possible) - improves **productivity** with increased proficiency Note: often the terms **shell**, **terminal** and **console** are used as synonym Shells are a **user interface** to system functions and other applications - Shells fall into one of two categories – command-line and graphical - **Terminal** emulator, wrapper interfacing a shell with the OS, i.e. `xterm` - Interface provided by a command-line interpreter **CLI** eg. `bash` - Text user interface **TUI** runs in a terminal, i.e. `vim` (text editor) References... * [A Guide to the Terminal, Console, and Shell](https://thevaluable.dev/guide-terminal-shell-console) * [What happens when you open a terminal and enter ‘ls’](https://www.warp.dev/blog/what-happens-when-you-open-a-terminal-and-enter-ls) ## Interpreter The **prompt** indicates readiness to accept command input by the user - Literally prompts the user to take action - Customizable with environment variables - May include information like user, hostname, working directory, etc. **Commands** are a large vocabulary of instructions and queries: Type | Description ---------|----------- Internal | Recognized and processed by the command line interpreter itself Included | Separate executable file part of the operating system External | Executables added by other parties for specific applications Anatomy of a command: ```bash # list files & directories in your home-directory ls -lh $HOME ├┘ ├─┘ └───┴───────────────────── argument │ └───────────────────────────── options └──────────────────────────────── command ``` **Parameters** are required and/or optional arguments/options - **Arguments** are positional inputs to the command - **Options** (flags/switches) modify the operation of a command **Delimiters** - _white space_ characters and the end-of-line (EOF) delimiter ### `echo` & `cat` `echo` displays lines of text - Accepts a list of **strings** as arguments - **Quotes**: preserve white-space in text using `'...'` - **Escapes**: `-e` enables interpretation of backslash escape sequences - `\t` horizontal tab, `\n` new line - `-n` do not output the trailing newline ```bash echo some simple text # some simple text echo text 'can be quoted' # quoted text with white-spaces echo 'quote text to show "quotes"' # quote text to show "quotes" echo -e 'text with a\ttab' # text with a tab ``` `cat` prints file content to standard output: - Special characters: `-A` show `^I` tab and `$` end of line - **Line numbers**: `-n` number each output line ```bash cat /etc/hostname # show file containing your local host name echo "\tone\011two\040three\0" | cat -A # ^Ione^Itwo three^@$ ``` ### Shell Modes Two modes of operations: * **Interactive** - Reads user input at the command prompt, enable users to enter/execute commands - Shells commonly used in interactive mode started by default upon launch of a terminal * **Non-interactive** - A shell executing a script (no human interaction) is always non-interactive - Scripts must be executable `chmod +x <script>` ## `man` & `help` Many ways to find information... * Most programs (commands) provide **manual** pages (manuals): - Detailed documentation, usually with references to other documents - `man` uses a terminal **pager** program, i.e. `more` or `less` to display - `info`, official documentation format textinfo of the GNU project * Convention...options `-h`, `--help` with a usage summery In general documentation is accessible with following commands: ```bash man <command> # man page for a given command info <command> # documentation of the GNU project whatis <command> # display one-line manual page descriptions apropos <keyword> # search for commands by keyword ## bash builtin help help # list of builtin commands help -m <command> # help for a specific builtin command ``` ## Unix Philosophy Set of software development concepts: **modularity** and **reusability** Best practices to designed a program for **composability** - Write programs that do one thing and do it well - Write programs to work together (be composeable) - Write programs to handle text streams (a universal interface) **Filter** program, reads from standard input, writes to the standard output Compose multiple programs into a chain with **pipelines** `|` ```bash # following one-liner simulates the rolling of a traditional die with 6 faces >>> seq 1 6 | shuf | head -n 1 3 ``` * the example above composes three programs into a pipeline: - `seq 1 6` print integers numbers from 1 to 6 - `shuf` randomly permutes lines from an input text stream - `head -n 1` extracts the 1st line from an input text stream ## `ls`, `cd`, `z` Bare minimum to know about the file-system tree, files and directories... ```bash ~ # abbr. for the home directory (like $HOME) / # root directory /<dir>[/<dir>[/<file>]] # absolute path, starts with / for the root directory . # current directory .. # parent directory <dir>[/<dir>/] # relative sub-directory # commands... ls $path # list contents of a directory pwd # print working directory cd $path # change to specified directory cd # change to home directory of the login user cd - # change to previous directory touch $path # create an empty file with specifed path and name rm $path # delete a file (permanently) mkdir $path # create an (new) empty directory in the tree rmdir $path # remove a directory if it is empty cp $spath $dpath # copy a file mv $spath $dpath # move a file/directory file $path # determine file type cat $path # print file content ln -s $path $link # create a symbolic link ``` Zoxide [^FNVI4] is a smarter `cd` command ```bash # install required packages sudo dnf install -y fzf zoxide # load Zoxide int your environment eval "$(zoxide init zsh)" ``` Use… `zoxide --help` ```bash # work like regular cd z $path # highest ranking match zi $path # interactive selection # list the database with scoring zoxide query -ls ``` [^FNVI4]: Zoxide Source Code, GitHub <https://github.com/ajeetdsouza/zoxide> ## Cursor & Key Bindings * text **cursor** (aka caret) indicates the current position for user interaction - typically underscore, rectangle or vertical line, flashing or steady - indicates the **insertion point** in text mode - moved by pressing arrows, page up/down, home/end key, etc. * **key bindings** tie key sequences to specific functions, cf. `man readline` Key | Description ---------|----------------------------- tab | command line completion ctrl-r | search command history ctrl-c | break current execution ctrl-a | cursor to beginning of line ctrl-e | cursor to end of line ctrl-l | clear screen * aka **keyboard shortcuts**, key-combinations to invoke an action ## `chmod` & `chown` `ls -l` lists files & directories with **ownership** and **permission** ```sh >>> ls -ld /tmp # ownership and permissions of /tmp drwxrwxrwt 16 root root 4.0K Nov 15 07:44 /tmp/ ├───────┘ ├──┘ ├──┘ │ │ └─────────── group │ └──────────────── owner └───────────────────────────── permissions ``` Alternatively use the `stat -c <format>` command: ```sh >>> stat -c '%A %a %U %u %G %g' /tmp drwxrwxrwt 1777 root 0 root 0 ├───────┘ ├──┘ ├──┘ │ ├──┘ └─ gid │ │ │ │ └────── group │ │ │ └──────── uid │ │ └───────────── user │ └────────────────── permissions octal └──────────────────────────── permissions human readable ``` The human readable permission consists of three **permission triads**: - ...configured with a selection of three characters from `[r-][w-][x-st]` - The **setuid/gid** bits allow users to run an executable with the permissions the owner or group - The **Sticky** bit only owner and root can rename/delete files/directories ``` r read w write x execute - no read/write/execute s siduid/setgid t sticky bit ``` Change permissions with the `chmod` command. The format of `<mode>` is a combination of 3 fields: * Effected triad `u` user, `g` group, `a` all or `o` others * `+` add permissions, `-` remove permissions * Permissions, a combination of `[rwxsStT]` ```bash id [<user>] # show user/group ID chgrp <group> <path> # change group ownership chown <user>[:<group>] <path> # change user/group chmod <mode> <path> # change permissions umask <mode> # set default permissions umask -S # displays mask in simbolic notation getfacl # show ACL permissions setfacl # modify ACL permissions ``` Examples: ```bash chmod +r ... # read for everyone chmod ug+rx ... # add read/execute for user/group chmod a-r ... # remove read access for everyone chmod ugo-rwx # remove all permissions chmod u=rw,g=r,o= ... # user read/write, group read, no access for others chmod -R u+w,go-w ... # set permissions recursive ``` Octal notation, think of rwx an binary variables `r*2^2 + w*2^1 + x*2^0` ```txt 7 rwx 111 6 rw- 110 5 r-x 101 4 r-- 100 3 -wx 011 2 -w- 010 1 --x 001 0 --- 000 ``` Examples ```bash chmod 1755 ... # sticky bit chmod 4755 # setuid chmod 2755 # setgid ``` ## `history` The shell maintains a list of **recently executed commands**, so called events. ```bash history # show command history fc # invoke editor to edit last command fc n # edit command n in history ``` History event designators: ```bash ! # start to reference history event !! # previous command !* # last argument list !n # command n in history !-n # the n preceding commands !s # most recent command starting with string s !?s # most recent command containing string s ``` Event modifier appended to an event, e.g. `!3:*` (use multiple modifiers like `!!:-4:p`: ```bash : # modifier prefix :n # nth word (word 0 is the command) :^ # first word :$ # last word :- # all words until the last :-n # all words including the nth :m-n # words from m to n :n* # wall words from n to the last :* # all words except the command :p # print, but not execute :r # remove the filename extension :e # remove all but the file name extension :h # remove last part from a path :s/P/S/ # replace first match of pattern P with string S :gs/P/S/ # like above, but replace all matches ``` Quick substitute `^P^S^` similar to `!!:s/P/S/`. ```bash setopt kshoptionprint && setopt | grep hist # show Zsh history settings printenv | grep HIST # ^^ show environment variables ``` #### `hstr` [HSTR (HiSToRy)](https://github.com/dvorka/hstr)... * ...shell history suggest box * ...easier and more efficient than **Ctrl-r** * ...manage command history... * ...remove commands * ...bookmark favorite commands * ...blacklist commands Configuration in the shell profile... ```sh alias '?'=hstr setopt histignorespace export HSTR_CONFIG=hicolor,help-on-opposite-side export HSTR_PROMPT="> " ``` ## `alias` ```bash alias # defines a new alias command unalias # unset an alias command ``` * aliases for the `ps` and `pstree` commands for process listings ```bash # List of all processes including their hierarchy alias psa='ps -AfH' # Comprehensive inspection of the process tree alias pstree='pstree -lpu' # Print a sorted list of all processes by CPU utilization alias pscpu="ps -A -o ruser,pcpu,time,state,args --sort pcpu" # Continuously updated list of all running processes alias prun="watch -n1 ps r -AL -o stat,pid,user,psr,pcpu,pmem,args" # include cgroup alias psc='ps xawf -eo pid,user,cgroup,args' ``` * aliases for the `ssh` command ```bash alias sr='ssh -l root' alias sA='ssh -A' alias sl='ssh-add -l' alias ssh-no-checks='ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' ssh-tmux() {/usr/bin/ssh -t $@ "tmux attach || tmux new";} ``` ## `su` & `sudo` `su` command stands for "switch user", and requires the password of the account to be used. ```sh su # switch to root user su <user> # switch to a specific user su - ... # invoke a login shell for another user su -p ... # preserve caller environment variables su - -c '<command>' <user> # execute a command as different user gksu -u <user> <app> # start a GUI application ``` Example ```sh # start X applications as another user xhost + # grant users access to your display su - <user> # switch user sourcing profiles # after login as another user export DISPLAY=:0.0 # export DISPLAY environment variable ``` `sudo` allows to execute a command on behalf of another users without requiring the password. ```sh /etc/sudoers # main configuration file for policies /etc/sudoers.d/* # additional configuration files sudo -ll # show user sudo configuration sudo -lU <user> # show configuration for another user sudo su - <user> # invoke a login shell as user sudo -i <user> # ^^ sudo -s <user> # executes $SHELL as user sudo -E <command> # preserver environment variables (no shell functions) ``` Path variables are not inherited with option `--preserve-env`, therefore set them explicitly ```sh sudo -E PATH=$HOME/bin:$PATH -- env | grep PATH ``` Run a shell with option `--shell` to execute multiple commands: ```sh sudo -sE PATH=$HOME/bin:$PATH -- <<EOF id pwd env | grep PATH EOF ``` Write content to a file: ```sh sudo tee -a /etc/motd -- <<EOF Append message of the day EOF ``` ### Configuration Basic policy: ```bash # comment a policy <user> <host>[,<host>,...] = (<user>[,<user>,...]:<group>[.<group>,...]) [NOPASSWD:] <command>[,<command>,...] ``` Aliases: ```bash # list of one or more hostnames, IP addresses, network numbers or netgroups Host_Alias <NAME> = <ip>,[<ip>,...][,<hostname>[,<hostname>,...] Host_Alias <NETWORK> = <ip>/<mask> # alias is list of one or more users, groups, uids etc. User_Alias <USERS> = <user>[,...][,%<group>,...], # list of commandnames, files or directories Cmnd_Alias <COMMAND> = <command][,<command>,...] ``` ### Examples ```bash vpenso ALL=(ALL) ALL # user vpenso gains root priviliges %wheel ALL=(ALL) ALL # group wheel gains root priviliges # user vpenso can execute apt as root vpenso ALL = (root) /usr/bin/apt # host shutdown Cmnd_Alias POWER = /bin/systemctl poweroff # allow password changes with the exception of root Cmnd_Alias PASSWD = /usr/bin/passwd [A-z]*, !/usr/bin/passwd root # vpenso can execute defined by command aliases vpenso localhost = (root) NOPASSWD: POWER,PASSWD ``` # Files & Directories Typically data stored in a file-systems is organized within a **tree structure**. * This tree structure consists of parent and child directories in a hierarchical order. * A **directory** (folder) contains files and further "child" directories. * A **file** contains data stored in the file-system. * The **root directory** is the top-most directory in the tree. * The current **working directory** is the location of the user while navigating in the directory tree. * A **sub directory** (aka child directory) is hierarchically below the working director. * Each user has a **home directory** (login directory) for personal data. Following shows the hierarchical listing of the home directory of a use "jdow": ```bash /home/jdow ├── /home/jdow/bin ├── /home/jdow/docs │ ├── /home/jdow/docs/manual.pdf │ └── /home/jdow/docs/readme.md ├── /home/jdow/music │ ├── /home/jdow/music/song.mp3 │ └── /home/jdow/music/sound.mp3 └── /home/jdow/var ``` Navigate the directory tree: ```bash pwd # print working directory cd <path> # change to specified directory cd # no path argument changes to the home directory of the login user cd - # change to previous directory pushd # push directory to stack popd # remove directory from stack dirs # list directory stack cd ~<n> # change to nth directory from stack touch <path> # create an empty file with specified path and name ``` Manipulating files & directories: ```bash rm <path> # delete a file (permanently) rm -r <path> # remove a directory and all its content (recursive decent) mkdir <path> # create an (new) empty directory in the tree mkdir -p <path> # ^^ recursive create of a new directory (missing parents included) rmdir <path> # remove a directory if it is empty cp <path> <path> # copy a file cp -R <path> <path> # copy a directory mv <path> <path> # move a file/directory rename <path> # rename multiple files according to pattern ln -s <path> <link> # create a symbolic link ``` Work with files: ```bash file # determine file type stat # display file status cat # print file content head # show the beginning of a file tail # show the end of a file less # pager for files pg # ^^ more # ^^ wc # count line/chars in a file touch # create empty file, change timestamp nl # number lines of a file ``` [bat](https://github.com/sharkdp/bat) replacement for `cat` ## Path A **path** specifies a unique location in the directory tree: * A path is separated by **slashes `/`** like `/home/jdoe/docs`. * An **absolute path** starts with a slash, hence specifies a "full" path to a location in the tree. * A **relative path** specifies a path in relation to the current working directory. Paths are constructed with the following notation: ```bash ~ # abbr. for the home directory (like $HOME) ~+ # current directory (like $PWD) ~- # previous working directory / # root directory ### absolute path /<dir>[/<dir>[/<file>]] # starts with / for the root directory ### relative path . # current directory .. # parent directory ../..[/..] # parent of the parent directory <dir>/[<dir>[/<file>]] # sub/child directory/file ../<dir>[/<dir>/] # relative to parent directory ~/<dir>[/<dir>/] # relative to the (login user) home directory ~<user>/<dir>[/<dir>/] # relative to a specific user home directory ./.<path> # hide a path by prefixing it with a dot ``` ## `ls` Aliases & Colors **`ls`** (list) show the contents of a directory-tree: ```bash alias ls='ls -Fh --color=always' # default to classify, human readable sizes and colors alias lS='ls -lS' # displays file size in order alias l='ls -1' # only names, one per line alias ll='ls -l' # long format alias l.='ls -lA -d .*' # only hidden files ``` **dotfile** treated as hidden, i.e. `~/.profile`: * File/directory name preceded by a dot `.` * Not displayed by default in a directory listing The **classify** option `-F` appends symbols to filenames: ```txt * executable / directory @ link = socket > door (IPC) | named pipe ``` The `LS_COLORS` variable is used to define colors for the `ls` command - Use the **`dircolors`** program for a more complex configuration: - [lsd](https://github.com/Peltoche/lsd) the next gen `ls` command ```bash # a simple file with a color configuration cat > ~/.dircolors <<EOF TERM screen-256color RESET 0 FILE 00;38;5;241 DIR 00;38;5;244 LINK 00;38;5;239 EOF # load the configuration on shell init echo 'eval "$(dircolors -b ~/.dircolors)"' >> ~/.profile ``` ## `tree` & `eza` **`tree`** lists directories in a tree like format: ```bash alias td='tree -d' # list only dirs alias t2='tree -L 2' # max recursive depth of 2 levels alias tu='tree -pfughF --du' # permissions, user, group, sizes ``` `eza` [^rsuvt] is a modern replacement for `ls` [^rsuvt]: `eze` Source Code, GitHub <https://eza.rocks> <https://github.com/eza-community/eza> ```bash export EXA_ICON_SPACING=2 local opts="--icons --color=never" alias ls="eza -F $opts" alias l="eza -1 $opts" alias ll="eza -alF $opts" alias lt="eza --tree --level=2 $opts" ``` ## Naming Conventions Naming files/directories: * File names typically use `A–Za–z0–9._-` **alphanumeric characters** (mostly lower case), underscores, hyphens, periods * `-` or `_` separates logical words, e.g. `this_is_a_file.txt` (note that `CamelCase` is normally not used) * Executables (including shell scripts) usually never have any type of extension * Files with a specific format use dot **`.` to separate the** (file-type) **extension**, e.g. `image.jpg` - Identifier specified as a suffix to the name - Indicates a characteristic of the file contents or its intended use * Capitalized file names are used for high-lighting e.g. `README.md` * Following characters must be quoted if they are to represent themselves: - `|&;<>()$\"'` as well as `␣⇥↵` space, tab and newline - `*?[#~=%` depending on conditions - Quoting mechanisms are the escape character `\`, single-quotes (literal value) and double-quotes (`$` variable expansion) ```bash # examples on escaping files names >>> touch 'foo?bar' foo\ bar foo%bar foo~bar "foo*bar" foo\$\ bar # list file names with special characters >>> ls | grep -E '\s|\*|\?|\$|%' foo bar foo?bar foo$ bar foo*bar foo%bar ``` ## `mktemp` Temporary Data Create a temporary file in `/tmp`... * ...and print its name * ...**`-d` to create a directory** instead * ...`-p $path` change the path ```sh # create an empty file in /tmp >>> mktemp /tmp/tmp.CgwZK2lgbN # create an empty directory >>> mktemp -d /tmp/tmp.73Akad37rK # adjust the path >>> mktemp -p /var/tmp /var/tmp/tmp.XpDoIxU3ON ``` Optional first argument defines a **template for the name**... * ...defaults to `/tmp/tmp.XXXXXXXXXX` * ...must contain at least 3 consecutive `X` in last component ```sh # ..custom prefix in the name... >>> mktemp -d /tmp/some-prefix.XXXXXX /tmp/some-prefix.AfHubr ``` # Input/Output & Pipes ## `|` Pipes A composable program architecture allows to **chain multiple programs**. A key character is **`|` (pipe)**: ```bash a | b # run a and b, send output of a as input to b, print output of b a | b | c ... # chained pipelines can build complex process structures ``` Programs (processes) have three **standard data streams**: 1. `STDIN` - **Input** read by a program 2. `STDOUT` - **Output** generated by a program 3. `STDERR` - 2nd output for **errors** messages & **debug** diagnostics "[Redirection][redir]" uses data streams to compose **interprocess communication** [unp]: https://en.wikipedia.org/wiki/Unix_philosophy [redir]: https://en.wikipedia.org/wiki/Redirection_(computing) **Why is this useful?** Print the host IP configuration with the `ip` command, and limit its output to the first five lines with the `head` command: ```bash » ip addr | head -n 5 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN ... link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host ``` Select a subset of the output matching a pattern with the `grep` command: ```bash » ip addr | grep 'inet ' inet 127.0.0.1/8 scope host lo inet 140.181.84.103/18 brd 140.181.127.255 scope global dynamic eno1 inet 10.1.1.1/24 brd 10.1.1.255 scope global nbr0 ``` Limit the output to specific columns using `awk` (text-processor): ```bash » ip addr | grep 'inet ' | awk '{print $2 " " $NF}' 127.0.0.1/8 lo 140.181.84.103/18 eno1 10.1.1.1/24 nbr0 ``` ## Standard Input/Output STDIO data streams **automatically opened** at program start: * `STDIN` - Usually **keyboard** input, or redirect from file(s)/program(s) * `STDOUT` - Output to **console** (display) and/or redirected to file(s)/program(s) * `STDERR` - Also sent to console or redirected to a **log file** I/O streams identified by an **integer [file descriptor][fld]** (file handle): Integer | Name | Stream -----------|-----------------|----------------------- 0 | Standard input | `STDIN` 1 | Standard output | `STDOUT` 2 | Standard error | `STDERR` Redirect streams between programs and files: ```bash a | b # STDOUT of program a to STDIN of process b a > f # STDOUT of program a to file f (same as a 1> f) a >> f # append STDOUT of program a to file f a < f # input content of file f to STDIN of program a (same as a 0< f) ``` [fld]: https://en.wikipedia.org/wiki/File_descriptor ## `>` Redirection Shells provide a **versatile way to compose programs**: ```bash > f # create an empty file f from empty STDIN (:> f) a 1>&- # close STDOUT of program a a 2> f # write STDERR of program a to file f a > f 2>&1 # write STDOUT & STDERR of program a to file f a &> f # ...same as above, in a short notation # redirect content of file f to STDIN of program a, and... a < f > g #...write STDOUT or program a to file g {a ; b} > f # redirect STDOUT of programs a and b to file f a |& b # pipe STDOUT & STDERR of program a to STDIN of program b a | tee f # redirect STDOUT of program a to console and file f a |: # redirect STDOUT & STDERR to pipeline sink (&>/dev/null) ``` Redirect **to/from a sub-shell**: ```bash a <(b) # redirect STDOUT of program b to STDIN of program a a >(b) # redirect STDOUT of program a to STDIN of program b (a ; b) > f # redirect STDOUT of programs a and b to file f # redirect STDOUT of program a to STDIN of program b and c... a >(b > f) >(c > g) #...both programs a and b write to files f and g ``` ## File Descriptors Simple example script with **permanent output redirection**: ```bash #!/usr/bin/env bash # write STDOUT to file (temporary redirect for a single command) echo one 1>/tmp/file # permanent redirect (append) STDOUT to file from here one exec 1>>/tmp/file echo two echo three ``` The `exec` program is used it **opening file descriptors**: ```bash exec 3< f # open file f for reading on file descriptor 3 exec 8<> f # open file f for reading and writing on file descriptor 8 exec 6>> f # open file f for appending on file descriptor 6 exec 5<&0 # copy read file descriptor 0 onto file descriptor 5 exec 7>&4 # copy write file descriptor 4 onto 7 exec 3<&- # close the read file descriptor 3 exec 6>&- # close the write file descriptor 6 ``` **Preserve references to `STDOUT` and `STDIN`** file descriptors: ```bash exec 3>&1 ; exec 4>&2 # preserve # ...run code here... exec 2>&4 ; exec 1>&3 # reset ``` # Process & Filter Command-line provides a versatile collection tools to **print** & **filter** text ```bash echo # display text cat # concatenate, print files grep # print lines matching a pattern sort # sort lines of a text file uniq # report or omit repeated lines cut # remove sections from each line of files tr # translate or delete characters sed # stream editor for filtering and transforming text awk # pattern scanning and processing language fmt # text formatter paste # merge lines of files split # split files into pieces tac # concatenate and print files in reverse ``` Some examples for conventions to describe characters - Escape sequence: `\[abfnrtv]` starting with a backslash - Hex code: `\x` followed by 1 or 2 digits - Unicode: `\u` 16bit, `U` 32bit followed by 1 to 4 digits ```bash echo "\x48allo" # Hallo echo "\u00A9" # © ``` ## `grep` Search for patterns in text... * `grep` will read **lines** from a text that has a match for a specific **pattern** * useful to find lines with a specific pattern in a large body of text, eg.: - look for a process in a list of processes - spot check a large number of files for occurrence of a pattern - exclude some text from a large body of text * **variants** of the `grep` command: ```bash agrep # approximately matching a pattern egrep # extended regex rgrep # recursive grep zgrep # search compressed files ``` * **options** - `-i` ignore case, `-v` inverse (lines that do not match pattern) - `-n` line numbers, `-c` count matching lines - `-A<n>` lines after match, `-B<n>` lines before match - `-o` print only the matched expression (not the whole line) A very important command to master... ```bash # case insensitive search >>> grep -i name /etc/os-release PRETTY_NAME="Debian GNU/Linux 9 (stretch)" NAME="Debian GNU/Linux" # use extended regex to match words >>> grep -E -w 'bin|sys|adm' /etc/group bin:x:2: sys:x:3: adm:x:4: # match a regex in a file >>> grep '^s[y,s].\:' /etc/group sys:x:3: ssh:x:112: # match CIDR IP addresses from STDIN >>> ip a | egrep -o '([0-9]{1,3}[\.]){3}[0-9]{1,3}\/[0-9]{1,3}' 127.0.0.1/8 140.181.84.103/18 # combine the find and grep commands >>> find $HOME/docs -iname "*.md" -exec grep '^# [A-Z].' {} + ... .../docs/tools/ssh/host-fingerprints.md:# SSH Host Fingerprints .../docs/tools/ssh/certs/host.md:# OpenSSH Host Certificates .../docs/tools/ssh/certs/host.md:# SSH client connection configuration ... ``` ## `cut` Cutting out sections from text... * `cut`: print selected parts of a line (fields) split by a delimiter - field **delimiter** character: specified with `-d` (defaults to a TAB) - fields separated by a single occurrence of the field delimiter character - **fields**: `-f` lists the field numbers to print ```bash text="A simple sentence! Nothing special to see..." echo "$text" | cut -d' ' -f1-3,5 # A simple sentence! special echo "$text" | cut -d! -f2- # Nothing special to see... echo "$text" | cut -d'.' -f1 # A simple sentence! Nothing special to see ``` * **byte position**: `-b` cuts each line specified by a list of byte numbers - always works in terms of single-byte characters - **character-list**: `-c` does not support Unicode at the moment ```bash echo "abcde12345" | cut -b3,7- # c2345 echo "abcde12345" | cut -c-3 # abc # unicode characters are 2 bytes long echo "αβγδεζ" | cut -b 3-4,9-10 # βε ``` ## `sort` Sort lines of text files... * Default ...compares entire line's ...decides how to sort it... * Options for simple sort customization... * ...numerically or alphabetically * ...ascending or descending order Option `-k $keydef` is used to select **specific fields for comparison** with following syntax: ```txt F[.C][OPTS][,F[.C][OPTS]] ``` * ...`F` is a field number (or column) * `-k2` ...second field to end of line * `-k3,7` ...third to seventh field * ...`C` a character position * ...`OPTS` single-letter ordering options...`[bdfgiMhnRrV]` * `-k3n` ...third field to end of line ...numerical sort * `-k2.3d` ...second field, third character ...dictionary order Multiple `-k` options will be sorted in order of the argument list... ```sh cat <<EOF | sort -k2,2n -k3,3n -k1,1 cba 321 111 bac 123 111 bac 123 000 bac 213 000 bac 213 111 abc 213 111 bac 321 000 abc 123 111 EOF ``` ## `fzf` `fzf` is similar to `grep`... <https://github.com/junegunn/fzf> ```sh # ...on RPM based distributions sudo dnf -y install fzf fd-find ``` * ...more suitable for interactive searches * ...user select match after initial filtering * ...single- or multi-text selection * ...fuzzy finding capabilities... * ...approximate searches * ...any part of the match in any order * ...pseudo user interfaces for command-line programs ## `regex` Regular **expression** (regex), defines a search **pattern** ```sh ([A-Za-z0-9-]+) # Letters, numbers and hyphens (\w+@[a-zA-Z_]+?\.[a-zA-Z]{2,6}) # email address (\<(/?[^\>]+)\>) # HTML tag ([0-9]{1,3}[\.]){3}[0-9]{1,3}\/[0-9]{1,3} # CIDR IP addresses ``` * **ranges** - `.` any character (except newline) - `(a|e)` a or e (`(...)` groups) - `[a-z0-9]` range, lowercase letter or any digit - `[^b-d]` invert of range b to d * **quantifiers** - `*` 0 or more, `?` 0 or 1, `+` 1 or more - `{N}` N (num.), `{N,}` N or more, `{,N}` N or less, `{N,M}` between N & M * **anchors**: `^` start of line, `$` end of line * **escape**: characters `^$()<>[{\|.+*?` using `\` (back-slash) as prefix ```sh >>> regex '^[f,b].[o]' foo bar # test with the `regex` command regex: ^[f,b].[o]\n foo matches bar does not match ``` ## CSV, JSON & YAML Miller [^mkF9p] ...reformat data ...formats CSV, TSV, JSON, and JSON [^mkF9p]: Miller <https://miller.readthedocs.io> <https://github.com/johnkerl/miller> ### `jc` `jc` [^fByRn] converts output of other commands to JSON [^fByRn]: `jc` JSON Convert <https://kellyjonbrazil.github.io/jc/> <https://github.com/kellyjonbrazil/jc> ```sh # input from STDIN... df | jc --df -p cat /etc/group | jc --group -p -r echo "A,B,C\n1,2,3" | jc --csv # ...command of file as argument jc -p free jc -p /proc/meminfo ``` ### `jq` `jq` [^AUTr7] filters a stream of JSON data... [^AUTr7]: `jq` JSON Processor <https://stedolan.github.io/jq/manual/> <https://github.com/stedolan/jq> - …extract fields of an object - …conversion of numbers to strings ```sh # object’s keys as an array echo '{"a":1,"b":2}' | jq 'keys' # get a named property echo '{"a":1,"b":2}' | jq '.a' echo '{"a":{"b":{"c":1}}}' | jq '.a.b.c' # element by index echo '[0,1,2,3,4,5]' | jq '.[3]' echo '[{"a":1,"b":2},{"c":3,"d":4}]' | jq '.[1].d' # slice an array echo '[0,1,2,3,4,5]' | jq '.[2:4]' echo '[0,1,2,3,4,5]' | jq '.[3:]' # number of array elements echo '[0,1,2,3,4,5]' | jq 'length' # array of unique values echo '[1,2,2,3]' | jq 'unique' # flatten nested arrays echo '[1,2,[3,4]]' | jq 'flatten' ``` ### `jless` `jless` [^lAHYL] is a command-line JSON viewer… [^lAHYL]: `jless` Command-Line JSON Viewer <https://jless.io/user-guide> <https://github.com/PaulJuliusMartinez/jless> - …supports Vi style key-binding - Collapse siblings with `c`/`C` and expand with `e`/`E` - …toggle collapse state with `Space` ```sh # support for YAML output alias yless="jless --yaml" ``` # Search & Replace ## `find` Search files based on criteria... ```bash find $HOME -name "README*" -exec wc -l {} + ├─────────────┘ ├──────────────┘ │ └───────────────── action (optinal) └───────────────────────────────── criteria (optinal) ``` * **path**: may have multiple paths, eg. `find /usr /opt -name "*.so"` * **criteria** - `-name "abc[a-z][0-9]"` using a regular expressions - `-type (f,d,l)` file, directory, sym. link - `-user <uname>`, `-group <gname>`, `-perm (ugo+/-rwx)` - `-mmin [+-]n` modified min. ago, `-mtime` days ago, `-amin` access time - criteria may be combined with logical and `-a` and or `-o` * **action** - `-print`: default action, display - `-ls`: run `ls -lids` command on each resulting file - `-exec <cmd>`: execute command - `-ok <cmd>`: exec command after user confirms Examples... ```bash find ... -exec ... {} + # pass file(s) as argument find ... -exec ... {} \; # use a subprocess to execute a command find ... -print | xargs -r ... # execute a command against the found files find ... -print0 | xargs -r -0 ... # files separated using null byte to support # special chars, spaces, etc. ``` * **placeholder** `{}` for file name(s) * `;` end of the command to exec, might need to be escaped (with a `\`) * `+` append a list of file to the command (on call) ```bash # text files in current directory, case insensitive find . -type f -iname "*.txt" # change permissions on directories (single call of chmod) find . -type d -exec chmod 770 {} + # files bigger then 512MB find . -type f -size +512M -print # delete all files that were not accessed in a year find $HOME -type f -atime+365 -exec rm {} + # embedded bash script consuming the list of files find . -type d -exec bash -c 'for f; do echo "$f" ; done' _ {} + # all files that have either .c or .h extension find . $ -name "*.c" -o -name "*.h" $ ``` ## `paste` Display to inputs side by side ```bash >>> paste <(echo 'a\nb\nc') <(echo '1\n2\n3') a 1 b 2 c 3 # custom a delimiter >>> paste -d'-' <(echo 'a\nb\nc') <(echo '1\n2\n3') a-1 b-2 c-3 # three inputs with custom delimiter >>> paste -d'%|' <(echo 'a\nb\nc') <(echo '1\n2\n3') <(echo 'x\ny\nz') a%1|x b%2|y c%3|z ``` Serialize multiple lines ```bash # tab as default delimiter >>> echo 'a\nb\nc' | paste -s a b c # custom delimiter >>> echo 'a\nb\nc' | paste -s -d'-' a-b-c ``` ## `sed` Use `sed` (stream editor) by piping input or reading it from a file: ```bash cat f | sed 'c' # input from stdin, apply command c sed 'c' f # input from file f apply command c sed -i 'c' f # modify file f in place with command c sed -f f ... # read commands from file f sed 'c1;c2' ... # multiple commands colon sep. sed -e 'c1' -e 'c2' ... # ^^ alternative ``` Stand-alone script spawns a shell calling sed, and printing output to standard out with `$*`. ```sh sed 'command' $* ``` Call sed directly with a shebang like `#!/bin/sed -f` Basic _command_ format (omit address to process all input lines) ```txt [address[,address]] instruction [arguments] ``` Use `;` to separate multiple commands. Available instructions: ```txt a\ append c\ change d delete i\ insert n next N next without write p print q quit s/ / / substitute (delimiter `/`) w file write ``` ### Substitution Substitution commands can use an alternative separator e.g. `sed "s'P'S'g"` or `sed "s|P|S|g"`. Examples for substitutions: ```sh s/P/S/ # substitute first match of pattern P with S s/P/S/g # substitute all matches of pattern P with S /M/s/P/S/g # like above but only lines matching pattern M s/^[ \t]*// # delete leading white spaces s/[ \t]*$// # delete trailing white spaces s/^/ / # insert leading space N,M s/P/S/ # substitute only in lines between N and M N,$ s/P/S/ # substitute after line N ``` Pattern flags are `/g` global, `/I` ignore case. `&` refers to pattern found, use `\` to escape. Use with option `sed -n 'E'` to suppress unselected input when using the print command: ```sh /P/p # print lines matching pattern P /P/!p # print lines not matching pattern P N,Mp # print lines with numbers from N to M /P1/,/P2/p # print lines between pattern P1 and P2 /P/{p;q;} # print first line matching pattern P ``` ### White Spaces `crush` removes duplicate blank lines: ```sed #!/bin/sed -f /^[[:blank:]]*$/d ``` ```bash >>> echo "\n\na\nb\n\nc\n" | crush a b c ``` `pushin` reduces multiple spaces to one ```sed #!/bin/sed -f s/[ ][ ]*/ /g ``` ``` >>> echo "a b c" | pushin a b c ``` `chop` removes trailing spaces ```sed #!/bin/sed -f s/[[:blank:]]*$// ``` ```bash >>> echo "a b c " | chop | cat -A a b c$ ``` # Scripting A series of commands stored in a plain text file... **Why Learn to Program?** 1. To understand our modern world 2. To use computers better 3. As a job skill 4. To use an important new form of literacy 5. As a medium in which to learn problem-solving 6. To study and understand processes 7. To have a new way to learn art, music, science, and mathematics **Why shell scripting?** - Create custom tools & utilities specific to your work - Automate repressive tasks to free up time - Codify complex administrative task as reference for co-workers - Create simple applications for customers Compared to "real" programming languages (like C/C++) - Easy to use - Quick start...interactive debugging - Fast development...very expressive - Executable on nearly all UNIX like systems - ...compatibility problems between shell variants - ...difficult to scale in scope (>1000 lines) Each shell script consists of... - **Keywords**: such as `if..else`, `do..while` - **Commands**: such as `pwd`, `echo`, `grep`, `find` - **Variables**: storing data referenced by a name - **Control flow**: statements expressing program behaviour - **Functions**: to reference script segments by name ## Interpreter Directive Executing a shell script: ```bash /bin/bash /path/to/script # interpreter followed by the path to a script bash /path/to/script # assumes bash is in $PATH /path/to/script # (recommended) interpreted defined by shebang line ``` **Interpreter directive**: `#!` (shebang) - Indicates the interpreter used to execute the file - Must be in the first line of a script - Interpreter is the full path to a binary file eg. `/bin/bash` - Requires the executable bit `chmod +x` - Some examples... - `#!/bin/bash` use Bash as interpreter - `#!/usr/bin/python` execute using the `python` command - `#!/usr/bin/env ruby` use `env` to find the ruby command and execute Obligatory "Hello World" Bash script: ```bash #!/bin/bash echo "Hello World" ``` ## Variables Variable **name**: symbolic placeholder for a value (data) - Convention to use `lower_case_name` (underscore as delimiter) - Opposed to constants and environment variables `UPPER_CASE_NAME` **Assignment**: declare a variable using the `=` operator ```bash date=2019/11/03 # no spaces! temp=21.5C name=alice empty= # assign an empty (null) value # errors... name= alice # declares an empty variable "name", run alice command name =alice # tries to run a command "name" with one argument name = alice # tries to run a command "name" with two arguments ``` **Reference**: precede variable name with `$` to access the value ```bash echo $date $temp # reference variables echo name # just text, not a variable reference ``` ### Build-in **Automatically set** and heavily used inside shell scripts Access **command-line parameter** in a shell script ```bash $1 $2 $3...$n # direct access to parameters (position arguments) $@ # all parameters "$@" # each parameter in double-quotes ("$1" "$2"...) "$*" # all parameters in double-quotes ("$1 $2...") $# # number of parameters ``` Shell script process and **run-time information** ```bash $$ # current shell process ID $0 # name of executed script $_ # absolute path to executed script $? # most recent exit status $! # most recent process ID ``` ### Environment Variables Pre-set **environment variables** (exported to all child processes) ```bash $PATH # search path for executables (; separated) $SHELL # current interpreter $PWD # current working directory $HOME # user home directory ``` Example script to read (input) parameters using build-in variables ```bash cat > /tmp/params <<'EOF' #!/bin/bash echo $#: "$@" echo $1 $5 EOF chmod +x /tmp/params /tmp/params 1 a 3 b 6 c ``` `$PATH` represents the command **search path** - `:` colon-separated list of directories that are searched (left to right) - Standard system directories such as `/bin`, `/usr/bin` - Third-party or locally-installed software such as `/usr/local/bin` - Append personally-chosen directories of your scripts ```bash # list the search path (one directory per line) echo $PATH | tr ':' '\n' # append our personal scripts directory (in the current shell) PATH=$PATH:$HOME/bin # make a shell variable available as an environment variable export PATH ``` ### Expansion Variable **expansion** with the character `$` - **Weak** quoting `"` (double quotes) expands arguments - **Strong** quoting `'` (single quotes) does not expands arguments - Prevent expansion with the **escape character** `\` ```bash 's' # preserves literal value of characters in string s "s $V" # preserves literal value of characters in string s except $ \ , \ # escape character (preserves the literal value) "s \$V" # using an escape sequenze to prevent variable expansion "${V}s" # name to be expanded in braces $'s' # expands string s with backslash-escaped characters replaced ``` ### `<<EOF` Here-Documents Block of code or text which can be redirected... - Avoid substitution and expansion by quoting the tag (here `EOF`) - Using `<<-` suppresses leading tabs (indent closing tag with tabs Write to a file,,, ```sh cat > /path/to/file <<'EOF' # ...content EOF ``` Assign to a variable... ```sh var=$(cat <<EOF # ...content EOF ) ``` Pipe to a command... ```sh cat <<EOF | sort 2 1 3 EOF ``` ## Commands **Statement separator** `;` **Grouping**: - `{ … }` groups commands purely for syntactic purposes (no sub-shell) - `( … )` create a sub-shell and wait for it to terminate **Logic operators**: and `&&`, or `||` ```bash c1; c2 # execute command c1 before c2 {c1; c2} # execute commands in current shell environment (c1; c2) # execute the commands inside a sub-shell environment c1 && c2 # if command c1 runs successful execute c2 c1 || c2 # if command c1 runs not successful execute c2 $(c) # execute command c in sub-shell (alternative `c`) c1 $(c2) # command c1 uses output of c2 as parameters c & # execute command c in background ``` **Command substitution**: `$(…)` creates a sub-shell... - ..with its standard output set to a pipe... - ..collects the output in the parent and expands to that output.. **Background**: `… &` sub-shell, does not wait for it to terminate List of build in commands to know... ```bash set -v # print shell input lines as they are read set -x # print commands and their arguments when executed set -f # disable globbing source f # execute file f in current shell environment eval `c` # execute command c in sub-shell and evaluate shift # remove leading positional parameter jobs # list active jobs associated with current shell disown # detach jobs from current shell nohup # continue job after logout bg # move job to background fg j # resume job j to foreground stop j # stops background job j trap c s # execute command c when catching signal s rehash # re-index executables in $PATH (Zsh) reset # reset terminal ``` ## Statements ### `if...then...else` `if` statements are to most used **control structure** - Allows for **branching** in the code execution path - Closing `fi` is necessary, `elif` and `else` optional - Always include spaces around `[[...]]` the **conditional operator** ```bash # evaluate the number of command-line parameters if [[ $# -eq 1 ]] ; then # one argument elif [[ $# -ge 2 ]] ; then # two or more arguments else # no arguments fi ``` Evaluate a **mathematical expression** with `((...))` ```bash if (( $1 + 1 > 2 )); then #... fi ``` Evaluate the **existence of a command** in the environment `$PATH` ```bash if command -v <cmd> >/dev/null ; then ... fi ``` Execute a command and use its **exit status** as condition to an if statement: - Every program returns an exit status as an integer number - `0` (zero) means success, `1-255` non-zero means failure ```bash if <cmd> ; then ... else ... fi ``` Exit Status | Description -------------|---------------------------- 1 | catchall for general errors 126 | command invoked cannot execute 127 | command not found ### `case` * `case` statement **matches values against a variable** - argument to case is expanded and matched against patterns - in case of a match run commands up to `;;` - patterns are not regular expressions but **shell patterns** (aka globs) ```bash case "$1" in 1) echo $1 ;; abc) echo def ;; 2|3|4) echo 2,3,4 ;; [a-z]*) echo a-z ;; *) echo no case esac ``` ### `for..do..done` * **iterator**: loop over a sequentially changed variable ```bash for (( i = 0; i < 10; i++ )) do echo $i done # two variables at once for (( i = 0, j = 0; i < 10; i++, j = i * i )) do echo $i $j done ``` * examples for `for..in..` ```bash # brace expansion for i in {1..10}; do echo $i done # loop an array arr=(a b c d e f) for i in "${arr[@]}";do echo $i done ``` ### `while...do...done` * loop **until a condition is `true`** ```bash # arithmetic i=0 while (( $i < 10 )) ; do echo $i ((i++)) done # comparison i=0 while [ $i -lt 10 ] ; do echo $i i=$[$i+1] done ``` * build an (infinite) **endless loop**... ```bash while true ; do ... done ``` ## Expressions Conditional expressions: * `&&` (and), `||` (or) have no precedence (left-associative) * Bit faster the `if...then` ```bash !e # true if expression e is false e1 && e2 # true if both expressions e1 and e2 are true (alternative e1 -a e2) e1 || e2 # true if either expression e1 or e2 is true (alternative e1 -o e2) [[e]] # returns bool after evaluation of the conditional expression e : # null-statement (returns 0) $[e] # evaluate integer expression ((e)) # expand and evaluate of integer expression e $((e)) # expand, evaluate and substitute integer expression e ``` ### Brace Expansion Brace expansion ```bash {a,b,c} # brace expansion {a..z} # extened brace expansion, ranges name.{foo,bar} # typical for files name{,.foo} # includes name {1..20} # digits {01..20} # with leading zero {20..1} # reverse {0..10..2} # use increments {a..d}{1..3} # combine braces ``` ### Operators Typical use of the conditional operator `[[...]]`: ```bash # string compare s1 = s2 # true if strings s1 equals s2 s1 != s2 # true if strings s1 not equal s2 -n s # true if string s is not empty -z s # true if string s is empty # file conditions -c f # true if f is a special character file -d p # true if p is the path to an existing directory -e f # true if f is an existing file -f f # true if f is an existing file but not an directory -G f # true if f exists and is owned by effective group-id -g f # true if f exists and is set-group-id. -k f # true if f has sticky bit -L f # true if f is a symbolic link -O f # true if f exists and is owned by the effective user-id. -r f # true if f is readable -S f # true if f is socket -s f # true if f nonzero size -u f # true if f set-user-id bit is set -w f # true if f is writable -x f # true if f is executable # numerical comparison n1 -lt n2 # true if integer n1 less then n2 n1 -gt n2 # true if n1 greater then n2 n1 -le n2 # true if n1 less or equal n2 n1 -ge n2 # true if n1 greater or equal n2 n1 -eq n2 # true if n1 equals n2 n1 -ne n2 # true if n1 not equal n2 ``` ### Empty vs Unset String Distinguish between empty and unset strings: ```bash case ${var+x$var} in (x) echo empty;; ("") echo unset;; (x*[![:blank:]]*) echo non-blank;; (*) echo blank esac ``` ## `exit` * each shell command returns an exit code when it terminates - `0` (zero) means success, `1-255` non-zero means failure - special variable `$?`: exit status of the last executed command ```bash # successfull execution of the date command >>> date |: ; echo $? 0 # list a non-existing directory... returns a non zero exit status >>> ls /foobar ; echo $? ls: cannot access '/foobar': No such file or directory 2 ``` * `if` branching based on an exit status of a command (substitution) ```bash … passwords=$(ccrypt -c $PASSWORD_FILE) # check if the password-name is present in the password-file if $(echo "$passwords"| grep "^$2" >/dev/null) ; then echo "$passwords" | grep "^$2" | cut -d',' -f2 else echo "Password name '$2' does not exist" exit 1 fi … ``` * `exit` command stops execution and returns an exit status ## Functions * abstract a segment of program code with a **function name** `name()` ```bash echo_n() { echo -n "$@" } ├────┘ ├──────────────┘ │ └────────────────── function name └─────────────────────────── function body # function call with arguments echo_n 1 2 a b ``` * **function body** defined by all code in between `{ … }` (curly brackets) * function **exit status**: specified with `return` value of 0 to 255 ```bash return_one() { echo ${1:-one} # print first argument or 'one' return 1 # return failure } if return_one # evaluates exit status then echo success else echo failure $(return_one 1) fi ``` ## `source` * `source` executed a file in the **current shell** (abbreviated to `.`) ```bash >>> echo '#!/bin/bash\necho $USER $name' | tee name.sh #!/bin/bash echo $USER $name # creates a new (sub)shell >>> name=vic ; bash name.sh vpenso # work in the current shell >>> name=vic ; source name.sh vpenso vic ``` * typically used modify environment variables by sourcing a file ```bash >>> cat ~/.bashrc if ! [ -z "$(ls -A ~/.bashrc.d)" ] ; then # if the directory is not empty for file in `\ls ~/.bashrc.d/*` ; do source $file # execute files in current shell done fi # configure the shell history, cf. bash manual >>> cat ~/.bashrc.d/history.sh export HISTSIZE=100000 # set history size export SAVEHIST=$HISTSIZE # save history after logout export HISTCONTROL=ignoreboth # ignore leading spaces, duplicates ``` * `~/bashrc`: customize your interactive shell session ## Processes A process is the **execution of a program** - …a command can only generate a process - …an application can run multiple processes for different tasks - …each process is assigned a unique **PID** (process identification number) ```bash ps # (process status) snapshot of all running processes pstree # display a tree of processes pgrep # search process ID by criteria kill # send a signal to a process pkill # search a process and send a signal killall # send a signal to a process by name nice # run a program with modified scheduling priority renice # alter priority of running processes top # dynamic real-time view of a running system ``` Starting a program from an interactive shell (within a terminal)… - …creates a process …inherits `stdin`, `stdout`, and `stderr` - …şhell blocked until process terminates …`SIGHUP` to shell propagates to process - Run a program **without blocking the shell** by appending an ampersand `&` - …creates a process …inherits `stdin`, `stdout`, and `stderr` - …added to the list of background jobs the shell manages - …`SIGHUP` to shell propagates to process ```bash <cmd> & # start process in background of the current shell nohup <cmd> # detach a process from the current terminal jobs # processes started in the context of the current shell bg %<job_id> # send send a process to the background fg %<job_id> # pull a process to foreground <cmd> & disown # detach from current shell ``` ### `disown` & `nohup` ```bash # decouple from a shell sleep 300 & disown # decouple from terminal and shell nohup sleep 300 & disown ``` `disown` **removes the process from the shell's job control**… - …still connected to the terminal (for example `pty` for SSH logins) - …`SIGHUP` to shell does not propagates to process `nohup` **disconnects the process from the terminal** - …redirects redirects `stderr` to `stdout` and `stdout` to `$HOME/nohup.out` - …may still receive signals from the origin shell (parent process) ### `kill` & `trap` **Signals**: one-way notifications for process - Used to manipulate the state of a processes - User typically use signals to **terminate a process** `kill` command sends a signal to a process - Read about the available signals with `man 7 signal` - Get a short signal listing with `kill -L` ```bash # shell buildin command behaves different to the external program >>> type -a kill kill is a shell builtin kill is /bin/kill >>> /bin/kill -L 1 HUP 2 INT 3 QUIT 4 ILL 5 TRAP 6 ABRT 7 BUS 8 FPE 9 KILL 10 USR1 11 SEGV 12 USR2 13 PIPE 14 ALRM 15 TERM 16 STKFLT 17 CHLD 18 CONT 19 STOP 20 TSTP 21 TTIN 22 TTOU 23 URG 24 XCPU 25 XFSZ 26 VTALRM 27 PROF 28 WINCH 29 POLL 30 PWR 31 SYS ``` Signals have a **number value** `9` and a **name** `KILL` ```bash kill -9 <pid> # kill a program using a signal number kill -KILL <pid> # similar using the signal name ``` Exit and error in a sub-shell, simple cases: ```bash (c) # execute command c in sub-shell, ignore errors (c) || exit $? # same as above but propagate signal code ``` #### Catching Signals Execute commands COMMAND on signal `SIG` ```sh trap 'COMMAND' SIG [SIG,..] ``` Trap breaking child processes ```sh set -m # rescue function executed by trap ensure() { echo "Clean up after child process died with signal $?." } # catch errors trap ensure ERR # execute child process segfaulter & # wait for the child process to finish wait $! ``` # `zsh` Shell ## `bindkey` * ...no argument ...print all keybindings * `-l` ...list of existing keymap names * `-M <KEYMAP>` ...list all the bindings in a given keymap * ...Vi mode ...keymaps `viins` and `vicmd` * Documentation... * ...manual page ...`man 1 zshzle` * ...`zle -al` ...list all registered line editor commands Examples ```sh # bind Ctrl-l ro run the `ls` command bindkey -s "^L" 'ls^M' ``` ## Plugins ### `zplug` [Zsh Plugin Manager](https://github.com/zplug/zplug)... Add following to `~/.zshrc`: ```sh export ZPLUG_HOME=~/.zplug test -d $ZPLUG_HOME \ || git clone https://github.com/zplug/zplug $ZPLUG_HOME source $ZPLUG_HOME/init.zsh zplug "zsh-users/zsh-completions" # ...add more plugins if ! zplug check; then zplug install fi zplug load ``` # Terminals Text terminals.... * ...serial computer interface for text entry and display * ...historically... * ...physical terminals ...like the IBM 3270, VT100 * ...keyboard/screen for display and input of data on a remote computer * ...**terminal emulators** ...called `tty` * ...mimic a hardware (video) terminal in software * ...provide interactive access to programs that run in the **command line** * VTE (virtual terminal emulator)... * ...emulates a text terminal inside a graphical user interface (GUI) * ...widgets that implement a fully functional terminal emulator ```bash /dev/tty[0-9] # teletypes, cf. man `tty` /dev/pts/[0-9] # pseudo terminals, cf man `pty` tty # show assoc. pseudo terminal stty -a # show terminal settings getty # program watching a physical terminal (tty) port ps -a # list processes with attached terminals terminfo termcap # terminal capability data base tput tget reset # init terminal ``` A system **console** is a special terminal... * On modern computers a directly connected monitor/keyboard * Receives messages from the OS regarding booting/shutdown progress * Linux support **virtual consoles** to provide several text terminals * Access with a key combination including function keys (e.g. Ctrl+Alt+F2) Terminology... * Control characters display control codes like line-feed, backspace, etc. * Escape sequences... * ...series of characters that give commands to the terminal * ...user for cursor movement, colors, etc. * ...consists of the `ESC` control character followed by a sequence of characters * ...**cut & paste**... * ...note that Ctrl-c breaks current execution in shell * **Ctrl-Shift-c** and **Ctrl-Shift-v** for cut & paste #### `gnome-console` [GNOME Console](https://gitlab.gnome.org/GNOME/console) * ...simple user-friendly terminal emulator * ...minimalistic designed for Mobile devices as part of [Phosh](https://gitlab.gnome.org/World/Phosh) #### `gnome-terminal` [GNOME Terminal](https://help.gnome.org/users/gnome-terminal/stable/)... * ...installed by default with GNOME * Options... * `--window` ...open a new window * `--title` ...set title of tab/window * `--tab` ...open a new tab in window * `--command` ...first command to execute... * ...can be a specific shell Zsh * ...or a terminal multiplexer like Tmux * Shortcuts ..._preference_ menu ..._shortcuts_ tab ```sh # open two tabs called local and remote ...local tab starts Tmux gnome-terminal --window --title remote --tab --title local --command tmux # typically mapped to a keyboard shortcut like F2 or Alt+9 ``` #### `tabby` * References... - <https://tabby.sh> - <https://github.com/Eugeny/tabby> # Encoding System of rules to **convert information** (letters, words, sound, images) into another form or representation for data transmissions/storage... * **character encodings** are representations of textual data * **decoding** is the reverse process, converting code symbols back * **ASCII**: single byte encoding only using the bottom 7 bits (`man ascii`) - first 32 codes (0–31 decimal) for **control characters** - letters `A-Za-z`, digits `0-9`, punctuation `!#$%&'()*+,-.` etc. * UTF-8 general-purpose way of representing **Unicode** characters - encoded as sequence of 1-4 bytes (variable width character encoding) - designed for backward compatibility with ASCII ```sh locale # current language and encoding settings localectl # ^^with systemd tools file -bi <file> # show file encoding # change encoding from ASCII to UTF-8 iconv -f utf-8 -t ascii -o <out_file> <in_file> # change encoding from UTF-8 charset to ASCII, omit invalid characters iconv -c -f utf-8 -t ascii -o <out_file> <in_file> ``` ## ASCII & Unicode **ASCII]**, single byte encoding only using the bottom **7 bits** * ASCII order (ASCIIbetical) - Uppercase come before lowercase letters - Digits and many punctuation marks come before letters * First 32 codes (0–31 decimal) for **control characters** * Many 8-bit encodings which are supersets of ASCII **UTF-8**, general-purpose way of representing **Unicode** characters * Character encoded as sequence of 1-4 bytes (variable width character encoding) * Designed for backward **compatibility with ASCII** (valid ASCII text is valid UTF-8-encoded Unicode) ## Notations & Escaping Conventions to describe characters * **Octal** digit characters (01234567), `\<octal>` 1,2 or 3 * **Escape sequence**, starting with a backslash `\[abfnrtv]` * **Caret notation**, `^` followed by a single character (usually a capital letter) ```bash printf `\O` # print octal printf '\xH' # print hex H printf "\\$(printf %o D)" # print decimal d ``` ## Non-Printable Characters Non-printable (white-space) characters: * **space** (blank, word divider) * backspace (BS), `\b`, `^H` * **tab** (horizontal tab (HT), `\t`, `\011`, `^I` * **newline** (line feed (LF)), `\n`, `\012`, `^J` * null (NUL) `\0`, `^@` * escape (ESC) `\e`, `^[` `cat` show non-printable characters with `-A` (equivalent to `-vET`) ```bash >>> s="\tone\n\011two\040three\0" >>> echo "$s" | cat -A ^Ione$ ^Itwo three^@$ ``` `^` and `M-` notation (100-137 ascii chars), for LF `$` `od` show non-printable chars with backslash escapes: ```bash >>> echo "$s" | od -c 0000000 \t o n e \n \t t w o t h r e e \0 0000020 \n 0000021 ```

Why, learn it?

Interpreter

echo & cat

Shell Modes

man & help

Unix Philosophy

ls, cd, z

Cursor & Key Bindings

chmod & chown

history

hstr

alias

su & sudo

Configuration

Examples

Files & Directories

Path

ls Aliases & Colors

tree & eza

Naming Conventions

mktemp Temporary Data

Input/Output & Pipes

| Pipes

Standard Input/Output

> Redirection

File Descriptors

Process & Filter

grep

cut

sort

fzf

regex

CSV, JSON & YAML

jc

jq

jless

Search & Replace

find

paste

sed

Substitution

White Spaces

Scripting

Interpreter Directive

Variables

Build-in

Environment Variables

Expansion

<<EOF Here-Documents

Commands

Statements

if...then...else

case

for..do..done

while...do...done

Expressions

Brace Expansion

Operators

Empty vs Unset String

exit

Functions

source

Processes

disown & nohup

kill & trap

Catching Signals

zsh Shell

bindkey

Plugins

zplug

Terminals

gnome-console

gnome-terminal

tabby

Encoding

ASCII & Unicode

Notations & Escaping

Non-Printable Characters

Footnotes

`echo` & `cat`

`man` & `help`

`ls`, `cd`, `z`

`chmod` & `chown`

`history`

`hstr`

`alias`

`su` & `sudo`

`ls` Aliases & Colors

`tree` & `eza`

`mktemp` Temporary Data

`|` Pipes

`>` Redirection

`grep`

`cut`

`sort`

`fzf`

`regex`

`jc`

`jq`

`jless`

`find`

`paste`

`sed`

`<<EOF` Here-Documents

`if...then...else`

`case`

`for..do..done`

`while...do...done`

`exit`

`source`

`disown` & `nohup`

`kill` & `trap`

`zsh` Shell

`bindkey`

`zplug`

`gnome-console`

`gnome-terminal`

`tabby`