Shell Command-Line Basics

Terminal
Published

July 8, 2010

Modified

March 13, 2024

Command-line

Why, learn it?

  • long lasting knowledge unlike many other computer skills
  • powerful expressive way of communicating with a computer
  • complete (technically advanced) control over the system components
  • facilitates automation for time consuming repetitive tasks
  • diversity of tooling and interfaces (makes difficult tasks possible)
  • improves productivity with increased proficiency

Note: often the terms shell, terminal and console are used as synonym

Shells are a user interface to system functions and other applications

  • Shells fall into one of two categories – command-line and graphical
  • Terminal emulator, wrapper interfacing a shell with the OS, i.e. xterm
  • Interface provided by a command-line interpreter CLI eg. bash
  • Text user interface TUI runs in a terminal, i.e. vim (text editor)

References…

Interpreter

The prompt indicates readiness to accept command input by the user

  • Literally prompts the user to take action
  • Customizable with environment variables
  • May include information like user, hostname, working directory, etc.

Commands are a large vocabulary of instructions and queries:

Type Description
Internal Recognized and processed by the command line interpreter itself
Included Separate executable file part of the operating system
External Executables added by other parties for specific applications

Anatomy of a command:

# list files & directories in your home-directory
ls -lh $HOME
├┘ ├─┘ └───┴───────────────────── argument
│  └───────────────────────────── options
└──────────────────────────────── command

Parameters are required and/or optional arguments/options

  • Arguments are positional inputs to the command
  • Options (flags/switches) modify the operation of a command

Delimiters - white space characters and the end-of-line (EOF) delimiter

echo & cat

echo displays lines of text

  • Accepts a list of strings as arguments
  • Quotes: preserve white-space in text using '...'
  • Escapes: -e enables interpretation of backslash escape sequences
    • \t horizontal tab, \n new line
    • -n do not output the trailing newline
echo some simple text                # some simple text
echo text 'can be   quoted'          # quoted text with white-spaces
echo 'quote text to show "quotes"'   # quote text to show "quotes"
echo -e 'text with a\ttab'           # text with a  tab

cat prints file content to standard output:

  • Special characters: -A show ^I tab and $ end of line
  • Line numbers: -n number each output line
cat /etc/hostname      # show file containing your local host name
echo "\tone\011two\040three\0" | cat -A # ^Ione^Itwo three^@$

Shell Modes

Two modes of operations:

  • Interactive
    • Reads user input at the command prompt, enable users to enter/execute commands
    • Shells commonly used in interactive mode started by default upon launch of a terminal
  • Non-interactive
    • A shell executing a script (no human interaction) is always non-interactive
    • Scripts must be executable chmod +x <script>

man & help

Many ways to find information…

  • Most programs (commands) provide manual pages (manuals):
    • Detailed documentation, usually with references to other documents
    • man uses a terminal pager program, i.e. more or less to display
    • info, official documentation format textinfo of the GNU project
  • Convention…options -h, --help with a usage summery

In general documentation is accessible with following commands:

man <command>                  # man page for a given command
info <command>                 # documentation of the GNU project
whatis <command>               # display one-line manual page descriptions
apropos <keyword>              # search for commands by keyword
## bash builtin help
help                           # list of builtin commands
help -m <command>              # help for a specific builtin command

Unix Philosophy

Set of software development concepts: modularity and reusability

Best practices to designed a program for composability

  • Write programs that do one thing and do it well
  • Write programs to work together (be composeable)
  • Write programs to handle text streams (a universal interface)

Filter program, reads from standard input, writes to the standard output

Compose multiple programs into a chain with pipelines |

# following one-liner simulates the rolling of a traditional die with 6 faces
>>> seq 1 6 | shuf | head -n 1
3
  • the example above composes three programs into a pipeline:
    • seq 1 6 print integers numbers from 1 to 6
    • shuf randomly permutes lines from an input text stream
    • head -n 1 extracts the 1st line from an input text stream

ls, cd, etc

Bare minimum to know about the file-system tree, files and directories…

~                        # abbr. for the home directory (like $HOME)
/                        # root directory
/<dir>[/<dir>[/<file>]]  # absolute path, starts with / for the root directory
.                        # current directory
..                       # parent directory
<dir>[/<dir>/]           # relative sub-directory
# commands...
ls $path                 # list contents of a directory
pwd                      # print working directory
cd $path                 # change to specified directory
cd                       # change to home directory of the login user
cd -                     # change to previous directory
touch $path              # create an empty file with specifed path and name
rm $path                 # delete a file (permanently)
mkdir $path              # create an (new) empty directory in the tree
rmdir $path              # remove a directory if it is empty
cp $spath $dpath         # copy a file
mv $spath $dpath         # move a file/directory
file $path               # determine file type
cat $path                # print file content
ln -s $path $link        # create a symbolic link

z Zoxide

Zoxide 1 is a smarter cd command

# install required packages
sudo dnf install -y fzf zoxide

# load Zoxide int your environment
eval "$(zoxide init zsh)"

Use… zoxide --help

# work like regular cd
z $path   # highest ranking match
zi $path  # interactive selection

# list the database with scoring
zoxide query -ls

Cursor & Key Bindings

  • text cursor (aka caret) indicates the current position for user interaction
    • typically underscore, rectangle or vertical line, flashing or steady
    • indicates the insertion point in text mode
    • moved by pressing arrows, page up/down, home/end key, etc.
  • key bindings tie key sequences to specific functions, cf. man readline
Key Description
tab command line completion
ctrl-r search command history
ctrl-c break current execution
ctrl-a cursor to beginning of line
ctrl-e cursor to end of line
ctrl-l clear screen
  • aka keyboard shortcuts, key-combinations to invoke an action

chmod & chown

ls -l lists files & directories with ownership and permission

>>> ls -ld /tmp              # ownership and permissions of /tmp
 drwxrwxrwt 16 root root 4.0K Nov 15 07:44 /tmp/
  ├───────┘    ├──┘ ├──┘
  │            │    └─────────── group
  │            └──────────────── owner
  └───────────────────────────── permissions

Alternatively use the stat -c <format> command:

>>> stat -c '%A %a %U %u %G %g' /tmp
drwxrwxrwt 1777 root 0 root 0
 ├───────┘ ├──┘ ├──┘ │ ├──┘ └─ gid
 │         │    │    │ └────── group
 │         │    │    └──────── uid
 │         │    └───────────── user
 │         └────────────────── permissions octal
 └──────────────────────────── permissions human readable

The human readable permission consists of three permission triads:

  • …configured with a selection of three characters from [r-][w-][x-st]
  • The setuid/gid bits allow users to run an executable with the permissions the owner or group
  • The Sticky bit only owner and root can rename/delete files/directories
r             read
w             write
x             execute
-             no read/write/execute
s             siduid/setgid
t             sticky bit

Change permissions with the chmod command. The format of <mode> is a combination of 3 fields:

  • Effected triad u user, g group, a all or o others
  • + add permissions, - remove permissions
  • Permissions, a combination of [rwxsStT]
id [<user>]                     # show user/group ID
chgrp <group> <path>            # change group ownership
chown <user>[:<group>] <path>   # change user/group
chmod <mode> <path>             # change permissions
umask <mode>                    # set default permissions
umask -S                        # displays mask in simbolic notation
getfacl                         # show ACL permissions
setfacl                         # modify ACL permissions

Examples:

chmod +r ...              # read for everyone
chmod ug+rx ...           # add read/execute for user/group
chmod a-r ...             # remove read access for everyone
chmod ugo-rwx             # remove all permissions
chmod u=rw,g=r,o= ...     # user read/write, group read, no access for others
chmod -R u+w,go-w ...     # set permissions recursive

Octal notation, think of rwx an binary variables r*2^2 + w*2^1 + x*2^0

7  rwx  111
6  rw-  110
5  r-x  101
4  r--  100
3  -wx  011
2  -w-  010
1  --x  001
0  ---  000

Examples

chmod 1755 ...            # sticky bit
chmod 4755                # setuid
chmod 2755                # setgid

history

The shell maintains a list of recently executed commands, so called events.

history       # show command history
fc            # invoke editor to edit last command
fc n          # edit command n in history

History event designators:

!             # start to reference history event
!!            # previous command
!*            # last argument list
!n            # command n in history
!-n           # the n preceding commands
!s            # most recent command starting with string s
!?s           # most recent command containing string s

Event modifier appended to an event, e.g. !3:* (use multiple modifiers like !!:-4:p:

:              # modifier prefix
:n             # nth word (word 0 is the command)
:^             # first word
:$             # last word
:-             # all words until the last
:-n            # all words including the nth
:m-n           # words from m to n
:n*            # wall words from n to the last
:*             # all words except the command
:p             # print, but not execute
:r             # remove the filename extension
:e             # remove all but the file name extension
:h             # remove last part from a path
:s/P/S/        # replace first match of pattern P with string S
:gs/P/S/       # like above, but replace all matches

Quick substitute ^P^S^ similar to !!:s/P/S/.

setopt kshoptionprint && setopt | grep hist      # show Zsh history settings
printenv | grep HIST                             # ^^ show environment variables

hstr

HSTR (HiSToRy)…

  • …shell history suggest box
  • …easier and more efficient than Ctrl-r
  • …manage command history…
    • …remove commands
    • …bookmark favorite commands
    • …blacklist commands

Configuration in the shell profile…

alias '?'=hstr
setopt histignorespace
export HSTR_CONFIG=hicolor,help-on-opposite-side
export HSTR_PROMPT="> "

alias

alias       # defines a new alias command
unalias     # unset an alias command
  • aliases for the ps and pstree commands for process listings
# List of all processes including their hierarchy
alias psa='ps -AfH'
# Comprehensive inspection of the process tree
alias pstree='pstree -lpu'
# Print a sorted list of all processes by CPU utilization
alias pscpu="ps -A -o ruser,pcpu,time,state,args --sort pcpu"
# Continuously updated list of all running processes
alias prun="watch -n1 ps r -AL -o stat,pid,user,psr,pcpu,pmem,args"
# include cgroup
alias psc='ps xawf -eo pid,user,cgroup,args'
  • aliases for the ssh command
alias sr='ssh -l root'
alias sA='ssh -A'
alias sl='ssh-add -l'
alias ssh-no-checks='ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
ssh-tmux() {/usr/bin/ssh -t $@ "tmux attach || tmux new";}

su & sudo

su command stands for “switch user”, and requires the password of the account to be used.

su                         # switch to root user
su <user>                  # switch to a specific user
su - ...                   # invoke a login shell for another user
su -p ...                  # preserve caller environment variables
su - -c '<command>' <user> # execute a command as different user
gksu -u <user> <app>       # start a GUI application

Example

# start X applications as another user
xhost +                  # grant users access to your display
su - <user>              # switch user sourcing profiles

# after login as another user
export DISPLAY=:0.0      # export DISPLAY environment variable

sudo allows to execute a command on behalf of another users without requiring the password.

/etc/sudoers               # main configuration file for policies
/etc/sudoers.d/*           # additional configuration files    
sudo -ll                   # show user sudo configuration
sudo -lU <user>            # show configuration for another user
sudo su - <user>           # invoke a login shell as user
sudo -i <user>             # ^^
sudo -s <user>             # executes $SHELL as user
sudo -E <command>          # preserver environment variables (no shell functions)

Path variables are not inherited with option --preserve-env, therefore set them explicitly

sudo -E PATH=$HOME/bin:$PATH -- env | grep PATH

Run a shell with option --shell to execute multiple commands:

sudo -sE PATH=$HOME/bin:$PATH -- <<EOF
        id
        pwd
        env | grep PATH
EOF

Write content to a file:

sudo tee -a /etc/motd -- <<EOF
Append message of the day
EOF

Configuration

Basic policy:

# comment a policy
<user> <host>[,<host>,...] = (<user>[,<user>,...]:<group>[.<group>,...]) [NOPASSWD:] <command>[,<command>,...]

Aliases:

# list of one or more hostnames, IP addresses, network numbers or netgroups
Host_Alias <NAME> = <ip>,[<ip>,...][,<hostname>[,<hostname>,...]
Host_Alias <NETWORK> = <ip>/<mask>
# alias is list of one or more users, groups, uids etc.
User_Alias <USERS> = <user>[,...][,%<group>,...], 
# list of commandnames, files or directories
Cmnd_Alias <COMMAND> = <command][,<command>,...]

Examples

vpenso ALL=(ALL) ALL       # user vpenso gains root priviliges
%wheel ALL=(ALL) ALL       # group wheel gains root priviliges
# user vpenso can execute apt as root
vpenso ALL = (root) /usr/bin/apt
# host shutdown
Cmnd_Alias POWER = /bin/systemctl poweroff
# allow password changes with the exception of root
Cmnd_Alias PASSWD = /usr/bin/passwd [A-z]*, !/usr/bin/passwd root
# vpenso can execute defined by command aliases
vpenso localhost = (root) NOPASSWD: POWER,PASSWD

Files & Directories

Typically data stored in a file-systems is organized within a tree structure.

  • This tree structure consists of parent and child directories in a hierarchical order.
  • A directory (folder) contains files and further “child” directories.
  • A file contains data stored in the file-system.
  • The root directory is the top-most directory in the tree.
  • The current working directory is the location of the user while navigating in the directory tree.
  • A sub directory (aka child directory) is hierarchically below the working director.
  • Each user has a home directory (login directory) for personal data.

Following shows the hierarchical listing of the home directory of a use “jdow”:

/home/jdow
├── /home/jdow/bin
├── /home/jdow/docs
│   ├── /home/jdow/docs/manual.pdf
│   └── /home/jdow/docs/readme.md
├── /home/jdow/music
│   ├── /home/jdow/music/song.mp3
│   └── /home/jdow/music/sound.mp3
└── /home/jdow/var

Navigate the directory tree:

pwd                  # print working directory
cd <path>            # change to specified directory
cd                   # no path argument changes to the home directory of the login user
cd -                 # change to previous directory
pushd                # push directory to stack
popd                 # remove directory from stack
dirs                 # list directory stack
cd ~<n>              # change to nth directory from stack
touch <path>         # create an empty file with specified path and name

Manipulating files & directories:

rm <path>            # delete a file (permanently)
rm -r <path>         # remove a directory and all its content (recursive decent)
mkdir <path>         # create an (new) empty directory in the tree
mkdir -p <path>      # ^^ recursive create of a new directory (missing parents included)
rmdir <path>         # remove a directory if it is empty
cp <path> <path>     # copy a file
cp -R <path> <path>  # copy a directory
mv <path> <path>     # move a file/directory
rename <path>        # rename multiple files according to pattern
ln -s <path> <link>  # create a symbolic link

Work with files:

file              # determine file type
stat              # display file status
cat               # print file content
head              # show the beginning of a file
tail              # show the end of a file
less              # pager for files
pg                # ^^
more              # ^^
wc                # count line/chars in a file
touch             # create empty file, change timestamp 
nl                # number lines of a file

bat replacement for cat

Path

A path specifies a unique location in the directory tree:

  • A path is separated by slashes / like /home/jdoe/docs.
  • An absolute path starts with a slash, hence specifies a “full” path to a location in the tree.
  • A relative path specifies a path in relation to the current working directory.

Paths are constructed with the following notation:

~                        # abbr. for the home directory (like $HOME)
~+                       # current directory (like $PWD)
~-                       # previous working directory
/                        # root directory
### absolute path
/<dir>[/<dir>[/<file>]]  # starts with / for the root directory
### relative path
.                        # current directory
..                       # parent directory
../..[/..]               # parent of the parent directory
<dir>/[<dir>[/<file>]]   # sub/child directory/file
../<dir>[/<dir>/]        # relative to parent directory
~/<dir>[/<dir>/]         # relative to the (login user) home directory
~<user>/<dir>[/<dir>/]   # relative to a specific user home directory 
./.<path>                # hide a path by prefixing it with a dot

ls Aliases & Colors

ls (list) show the contents of a directory-tree:

alias ls='ls -Fh --color=always'   # default to classify, human readable sizes and colors
alias lS='ls -lS'                  # displays file size in order
alias l='ls -1'                    # only names, one per line
alias ll='ls -l'                   # long format
alias l.='ls -lA -d .*'            # only hidden files

dotfile treated as hidden, i.e. ~/.profile:

  • File/directory name preceded by a dot .
  • Not displayed by default in a directory listing

The classify option -F appends symbols to filenames:

*       executable
/       directory
@       link
=       socket
>       door (IPC)   
|       named pipe

The LS_COLORS variable is used to define colors for the ls command

  • Use the dircolors program for a more complex configuration:
  • lsd the next gen ls command
# a simple file with a color configuration
cat > ~/.dircolors <<EOF
TERM screen-256color
RESET                 0
FILE                  00;38;5;241
DIR                   00;38;5;244
LINK                  00;38;5;239
EOF
# load the configuration on shell init
echo 'eval "$(dircolors -b ~/.dircolors)"' >> ~/.profile

tree & eza

tree lists directories in a tree like format:

alias td='tree -d'                 # list only dirs 
alias t2='tree -L 2'               # max recursive depth of 2 levels
alias tu='tree -pfughF --du'       # permissions, user, group, sizes

eza 2 is a modern replacement for ls

export EXA_ICON_SPACING=2
local opts="--icons --color=never"
alias ls="eza -F $opts"
alias l="eza -1 $opts"
alias ll="eza -alF $opts"
alias lt="eza --tree --level=2 $opts"

Naming Conventions

Naming files/directories:

  • File names typically use A–Za–z0–9._- alphanumeric characters (mostly lower case), underscores, hyphens, periods
  • - or _ separates logical words, e.g. this_is_a_file.txt (note that CamelCase is normally not used)
  • Executables (including shell scripts) usually never have any type of extension
  • Files with a specific format use dot . to separate the (file-type) extension, e.g. image.jpg
    • Identifier specified as a suffix to the name
    • Indicates a characteristic of the file contents or its intended use
  • Capitalized file names are used for high-lighting e.g. README.md
  • Following characters must be quoted if they are to represent themselves:
    • |&;<>()$\"' as well as ␣⇥↵ space, tab and newline
    • *?[#~=% depending on conditions
    • Quoting mechanisms are the escape character \, single-quotes (literal value) and double-quotes ($ variable expansion)
# examples on escaping files names
>>> touch 'foo?bar' foo\ bar foo%bar foo~bar "foo*bar" foo\$\ bar
# list file names with special characters
>>> ls | grep -E '\s|\*|\?|\$|%'
foo bar
foo?bar
foo$ bar
foo*bar
foo%bar

mktemp Temporary Data

Create a temporary file in /tmp…

  • …and print its name
    • …-d to create a directory instead
    • …-p $path change the path
# create an empty file in /tmp
>>> mktemp       
/tmp/tmp.CgwZK2lgbN

# create an empty directory
>>> mktemp -d     
/tmp/tmp.73Akad37rK

# adjust the path
>>> mktemp -p /var/tmp            
/var/tmp/tmp.XpDoIxU3ON

Optional first argument defines a template for the name…

  • …defaults to /tmp/tmp.XXXXXXXXXX
  • …must contain at least 3 consecutive X in last component
# ..custom prefix in the name...
>>> mktemp -d /tmp/some-prefix.XXXXXX
/tmp/some-prefix.AfHubr

Input/Output & Pipes

| Pipes

A composable program architecture allows to chain multiple programs.

A key character is | (pipe):

a | b   # run a and b, send output of a as input to b, print output of b
a | b | c ...  # chained pipelines can build complex process structures

Programs (processes) have three standard data streams:

  1. STDIN - Input read by a program
  2. STDOUT - Output generated by a program
  3. STDERR - 2nd output for errors messages & debug diagnostics

“Redirection” uses data streams to compose interprocess communication

Why is this useful?

Print the host IP configuration with the ip command, and limit its output to the first five lines with the head command:

» ip addr | head -n 5 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN ...
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host

Select a subset of the output matching a pattern with the grep command:

» ip addr | grep 'inet '                 
    inet 127.0.0.1/8 scope host lo
    inet 140.181.84.103/18 brd 140.181.127.255 scope global dynamic eno1
    inet 10.1.1.1/24 brd 10.1.1.255 scope global nbr0

Limit the output to specific columns using awk (text-processor):

» ip addr | grep 'inet ' | awk '{print $2 " " $NF}'
127.0.0.1/8 lo
140.181.84.103/18 eno1
10.1.1.1/24 nbr0

Standard Input/Output

STDIO data streams automatically opened at program start:

  • STDIN - Usually keyboard input, or redirect from file(s)/program(s)
  • STDOUT - Output to console (display) and/or redirected to file(s)/program(s)
  • STDERR - Also sent to console or redirected to a log file

I/O streams identified by an integer file descriptor (file handle):

Integer Name Stream
0 Standard input STDIN
1 Standard output STDOUT
2 Standard error STDERR

Redirect streams between programs and files:

a | b   # STDOUT of program a to STDIN of process b
a > f   # STDOUT of program a to file f (same as a 1> f)
a >> f  # append STDOUT of program a to file f
a < f   # input content of file f to STDIN of program a (same as a 0< f)

> Redirection

Shells provide a versatile way to compose programs:

> f                   # create an empty file f from empty STDIN (:> f)
a 1>&-                # close STDOUT of program a
a 2> f                # write STDERR of program a to file f
a > f 2>&1            # write STDOUT & STDERR of program a to file f
a &> f                # ...same as above, in a short notation
# redirect content of file f to STDIN of program a, and...
a < f > g             #...write STDOUT or program a to file g  
{a ; b} > f           # redirect STDOUT of programs a and b to file f
a |& b                # pipe STDOUT & STDERR of program a to STDIN of program b
a | tee f             # redirect STDOUT of program a to console and file f
a |:                  # redirect STDOUT & STDERR to pipeline sink (&>/dev/null)

Redirect to/from a sub-shell:

a <(b)                # redirect STDOUT of program b to STDIN of program a
a >(b)                # redirect STDOUT of program a to STDIN of program b
(a ; b) > f           # redirect STDOUT of programs a and b to file f
# redirect STDOUT of program a to STDIN of program b and c...
a >(b > f) >(c > g)   #...both programs a and b write to files f and g

File Descriptors

Simple example script with permanent output redirection:

#!/usr/bin/env bash
# write STDOUT to file (temporary redirect for a single command)
echo one 1>/tmp/file
# permanent redirect (append) STDOUT to file from here one
exec 1>>/tmp/file
echo two
echo three

The exec program is used it opening file descriptors:

exec 3< f          # open file f for reading on file descriptor  3
exec 8<> f         # open file f for reading and writing on file descriptor 8
exec 6>> f         # open file f for appending on file descriptor 6
exec 5<&0          # copy read file descriptor 0 onto file descriptor 5
exec 7>&4          # copy write file descriptor 4 onto 7
exec 3<&-          # close the read file descriptor 3
exec 6>&-          # close the write file descriptor 6

Preserve references to STDOUT and STDIN file descriptors:

exec 3>&1 ; exec 4>&2         # preserve
# ...run code here...
exec 2>&4 ; exec 1>&3         # reset

Process & Filter

Command-line provides a versatile collection tools to print & filter text

echo                    # display text
cat                     # concatenate, print files
grep                    # print lines matching a pattern
sort                    # sort lines of a text file
uniq                    # report or omit repeated lines
cut                     # remove sections from each line of files
tr                      # translate or delete characters
sed                     # stream editor for filtering and transforming text
awk                     # pattern scanning and processing language
fmt                     # text formatter
paste                   # merge lines of files
split                   # split files into pieces
tac                     # concatenate and print files in reverse

Some examples for conventions to describe characters

  • Escape sequence: \[abfnrtv] starting with a backslash
  • Hex code: \x followed by 1 or 2 digits
  • Unicode: \u 16bit, U 32bit followed by 1 to 4 digits
echo "\x48allo"                      # Hallo
echo "\u00A9"                        # ©

grep

Search for patterns in text…

  • grep will read lines from a text that has a match for a specific pattern
  • useful to find lines with a specific pattern in a large body of text, eg.:
    • look for a process in a list of processes
    • spot check a large number of files for occurrence of a pattern
    • exclude some text from a large body of text
  • variants of the grep command:
agrep         # approximately matching a pattern
egrep         # extended regex
rgrep         # recursive grep
zgrep         # search compressed files
  • options
    • -i ignore case, -v inverse (lines that do not match pattern)
    • -n line numbers, -c count matching lines
    • -A<n> lines after match, -B<n> lines before match
    • -o print only the matched expression (not the whole line)

A very important command to master…

# case insensitive search
>>> grep -i name /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
# use extended regex to match words
>>> grep -E -w 'bin|sys|adm' /etc/group
bin:x:2:
sys:x:3:
adm:x:4:
# match a regex in a file
>>> grep '^s[y,s].\:' /etc/group
sys:x:3:
ssh:x:112:
# match CIDR IP addresses from STDIN
>>> ip a | egrep -o '([0-9]{1,3}[\.]){3}[0-9]{1,3}\/[0-9]{1,3}'
127.0.0.1/8
140.181.84.103/18
# combine the find and grep commands 
>>> find $HOME/docs -iname "*.md" -exec grep '^# [A-Z].' {} +
...
.../docs/tools/ssh/host-fingerprints.md:# SSH Host Fingerprints
.../docs/tools/ssh/certs/host.md:# OpenSSH Host Certificates
.../docs/tools/ssh/certs/host.md:# SSH client connection configuration
...

cut

Cutting out sections from text…

  • cut: print selected parts of a line (fields) split by a delimiter
    • field delimiter character: specified with -d (defaults to a TAB)
    • fields separated by a single occurrence of the field delimiter character
    • fields: -f lists the field numbers to print
text="A simple sentence! Nothing special to see..."
echo "$text" | cut -d' ' -f1-3,5    # A simple sentence! special
echo "$text" | cut -d! -f2-         # Nothing special to see...
echo "$text" | cut -d'.' -f1        # A simple sentence! Nothing special to see
  • byte position: -b cuts each line specified by a list of byte numbers
    • always works in terms of single-byte characters
    • character-list: -c does not support Unicode at the moment
echo "abcde12345" | cut -b3,7-      # c2345
echo "abcde12345" | cut -c-3        # abc
# unicode characters are 2 bytes long
echo "αβγδεζ" | cut -b 3-4,9-10     # βε

sort

Sort lines of text files…

  • Default …compares entire line’s …decides how to sort it…
  • Options for simple sort customization…
    • …numerically or alphabetically
    • …ascending or descending order

Option -k $keydef is used to select specific fields for comparison with following syntax:

F[.C][OPTS][,F[.C][OPTS]]
  • …F is a field number (or column)
    • -k2 …second field to end of line
    • -k3,7 …third to seventh field
  • …C a character position
  • …OPTS single-letter ordering options…[bdfgiMhnRrV]
    • -k3n …third field to end of line …numerical sort
    • -k2.3d …second field, third character …dictionary order

Multiple -k options will be sorted in order of the argument list…

cat <<EOF | sort -k2,2n -k3,3n -k1,1
cba 321 111
bac 123 111
bac 123 000
bac 213 000
bac 213 111
abc 213 111
bac 321 000
abc 123 111
EOF

fzf

fzf is similar to grep… https://github.com/junegunn/fzf

# ...on RPM based distributions
sudo dnf -y install fzf fd-find
  • …more suitable for interactive searches
    • …user select match after initial filtering
    • …single- or multi-text selection
  • …fuzzy finding capabilities…
    • …approximate searches
    • …any part of the match in any order
  • …pseudo user interfaces for command-line programs

regex

Regular expression (regex), defines a search pattern

([A-Za-z0-9-]+)                             # Letters, numbers and hyphens
(\w+@[a-zA-Z_]+?\.[a-zA-Z]{2,6})            # email address
(\<(/?[^\>]+)\>)                            # HTML tag
([0-9]{1,3}[\.]){3}[0-9]{1,3}\/[0-9]{1,3}   # CIDR IP addresses
  • ranges
    • . any character (except newline)
    • (a|e) a or e ((...) groups)
    • [a-z0-9] range, lowercase letter or any digit
    • [^b-d] invert of range b to d
  • quantifiers
    • * 0 or more, ? 0 or 1, + 1 or more
    • {N} N (num.), {N,} N or more, {,N} N or less, {N,M} between N & M
  • anchors: ^ start of line, $ end of line
  • escape: characters ^$()<>[{\|.+*? using \ (back-slash) as prefix
>>> regex '^[f,b].[o]' foo bar # test with the `regex` command
regex: ^[f,b].[o]\n
foo matches
bar does not match

CSV, JSON & YAML

Miller 3 …reformat data …formats CSV, TSV, JSON, and JSON

jc

jc 4 converts output of other commands to JSON

# input from STDIN...
df | jc --df -p
cat /etc/group | jc --group -p -r
echo "A,B,C\n1,2,3" | jc --csv
# ...command of file as argument
jc -p free
jc -p /proc/meminfo

jq

jq 5 filters a stream of JSON data…

  • …extract fields of an object
  • …conversion of numbers to strings
# object’s keys as an array
echo '{"a":1,"b":2}' | jq 'keys'
# get a named property
echo '{"a":1,"b":2}' | jq '.a'
echo '{"a":{"b":{"c":1}}}' | jq '.a.b.c'
# element by index
echo '[0,1,2,3,4,5]' | jq '.[3]'
echo '[{"a":1,"b":2},{"c":3,"d":4}]' | jq '.[1].d'
# slice an array
echo '[0,1,2,3,4,5]' | jq '.[2:4]'
echo '[0,1,2,3,4,5]' | jq '.[3:]'
# number of array elements
echo '[0,1,2,3,4,5]' | jq 'length'
# array of unique values
echo '[1,2,2,3]' | jq 'unique'
# flatten nested arrays
echo '[1,2,[3,4]]' | jq 'flatten'

jless

jless 6 is a command-line JSON viewer…

  • …supports Vi style key-binding
  • Collapse siblings with c/C and expand with e/E
  • …toggle collapse state with Space
# support for YAML output
alias yless="jless --yaml"

Search & Replace

find

Search files based on criteria…

find $HOME -name "README*" -exec wc -l {} +
           ├─────────────┘ ├──────────────┘
           │               └───────────────── action (optinal)
           └───────────────────────────────── criteria (optinal)
  • path: may have multiple paths, eg. find /usr /opt -name "*.so"
  • criteria
    • -name "abc[a-z][0-9]" using a regular expressions
    • -type (f,d,l) file, directory, sym. link
    • -user <uname>, -group <gname>, -perm (ugo+/-rwx)
    • -mmin [+-]n modified min. ago, -mtime days ago, -amin access time
    • criteria may be combined with logical and -a and or -o
  • action
    • -print: default action, display
    • -ls: run ls -lids command on each resulting file
    • -exec <cmd>: execute command
    • -ok <cmd>: exec command after user confirms

Examples…

find ... -exec ... {} +            # pass file(s) as argument
find ... -exec ... {} \;           # use a subprocess to execute a command
find ... -print | xargs -r ...     # execute a command against the found files
find ... -print0 | xargs -r -0 ... # files separated using null byte to support
                                   # special chars, spaces, etc.
  • placeholder {} for file name(s)
  • ; end of the command to exec, might need to be escaped (with a \)
  • + append a list of file to the command (on call)
# text files in current directory, case insensitive
find . -type f -iname "*.txt"
# change permissions on directories (single call of chmod)
find . -type d -exec chmod 770 {} +
# files bigger then 512MB
find . -type f -size +512M -print
# delete all files that were not accessed in a year
find $HOME -type f -atime+365 -exec rm {} +
# embedded bash script consuming the list of files
find . -type d -exec bash -c 'for f; do echo "$f" ; done' _ {} +
# all files that have either .c or .h extension
find . \( -name "*.c" -o -name "*.h" \)

paste

Display to inputs side by side

>>> paste <(echo 'a\nb\nc') <(echo '1\n2\n3')       
a   1
b   2
c   3
# custom a delimiter 
>>> paste -d'-' <(echo 'a\nb\nc') <(echo '1\n2\n3')
a-1
b-2
c-3
# three inputs with custom delimiter
>>> paste -d'%|' <(echo 'a\nb\nc') <(echo '1\n2\n3') <(echo 'x\ny\nz')
a%1|x
b%2|y
c%3|z

Serialize multiple lines

# tab as default delimiter
>>> echo 'a\nb\nc' | paste -s
a   b   c
# custom delimiter
>>> echo 'a\nb\nc' | paste -s -d'-'
a-b-c

sed

Use sed (stream editor) by piping input or reading it from a file:

cat f | sed 'c'             # input from stdin, apply command c
sed 'c' f                   # input from file f apply command c
sed -i 'c' f                # modify file f in place with command c
sed -f f ...                # read commands from file f
sed 'c1;c2' ...             # multiple commands colon sep.
sed -e 'c1' -e 'c2' ...     # ^^ alternative

Stand-alone script spawns a shell calling sed, and printing output to standard out with $*.

sed 'command' $*

Call sed directly with a shebang like #!/bin/sed -f

Basic command format (omit address to process all input lines)

[address[,address]] instruction [arguments]

Use ; to separate multiple commands. Available instructions:

a\        append
c\        change
d         delete
i\        insert
n         next
N         next without write
p         print
q         quit
s/ / /    substitute (delimiter `/`)
w file    write

Substitution

Substitution commands can use an alternative separator e.g.  sed "s'P'S'g" or sed "s|P|S|g". Examples for substitutions:

s/P/S/             # substitute first match of pattern P with S
s/P/S/g            # substitute all matches of pattern P with S
/M/s/P/S/g         # like above but only lines matching pattern M
s/^[ \t]*//        # delete leading white spaces
s/[ \t]*$//        # delete trailing white spaces
s/^/ /             # insert leading space 
N,M s/P/S/         # substitute only in lines between N and M
N,$ s/P/S/         # substitute after line N

Pattern flags are /g global, /I ignore case. & refers to pattern found, use \ to escape.

Use with option sed -n 'E' to suppress unselected input when using the print command:

/P/p              # print lines matching pattern P 
/P/!p             # print lines not matching pattern P
N,Mp              # print lines with numbers from N to M
/P1/,/P2/p        # print lines between pattern P1 and P2
/P/{p;q;}         # print first line matching pattern P

White Spaces

crush removes duplicate blank lines:

#!/bin/sed -f
/^[[:blank:]]*$/d
>>> echo "\n\na\nb\n\nc\n" | crush
a
b
c

pushin reduces multiple spaces to one

#!/bin/sed -f
s/[ ][  ]*/ /g
>>> echo "a   b      c" | pushin
a b c

chop removes trailing spaces

#!/bin/sed -f
s/[[:blank:]]*$//
>>> echo "a   b      c  " | chop | cat -A
a   b      c$

Scripting

A series of commands stored in a plain text file…

Why Learn to Program?

  1. To understand our modern world
  2. To use computers better
  3. As a job skill
  4. To use an important new form of literacy
  5. As a medium in which to learn problem-solving
  6. To study and understand processes
  7. To have a new way to learn art, music, science, and mathematics

Why shell scripting?

  • Create custom tools & utilities specific to your work
  • Automate repressive tasks to free up time
  • Codify complex administrative task as reference for co-workers
  • Create simple applications for customers

Compared to “real” programming languages (like C/C++)

  • Easy to use
  • Quick start…interactive debugging
  • Fast development…very expressive
  • Executable on nearly all UNIX like systems
  • …compatibility problems between shell variants
  • …difficult to scale in scope (>1000 lines)

Each shell script consists of…

  • Keywords: such as if..else, do..while
  • Commands: such as pwd, echo, grep, find
  • Variables: storing data referenced by a name
  • Control flow: statements expressing program behaviour
  • Functions: to reference script segments by name

Interpreter Directive

Executing a shell script:

/bin/bash /path/to/script  # interpreter followed by the path to a script
bash /path/to/script       # assumes bash is in $PATH
/path/to/script            # (recommended) interpreted defined by shebang line

Interpreter directive: #! (shebang)

  • Indicates the interpreter used to execute the file
  • Must be in the first line of a script
  • Interpreter is the full path to a binary file eg. /bin/bash
  • Requires the executable bit chmod +x
  • Some examples…
    • #!/bin/bash use Bash as interpreter
    • #!/usr/bin/python execute using the python command
    • #!/usr/bin/env ruby use env to find the ruby command and execute

Obligatory “Hello World” Bash script:

#!/bin/bash
echo "Hello World"

Variables

Variable name: symbolic placeholder for a value (data)

  • Convention to use lower_case_name (underscore as delimiter)
  • Opposed to constants and environment variables UPPER_CASE_NAME

Assignment: declare a variable using the = operator

date=2019/11/03    # no spaces!
temp=21.5C
name=alice
empty=             # assign an empty (null) value
# errors...
name= alice        # declares an empty variable "name", run alice command
name =alice        # tries to run a command "name" with one argument
name = alice       # tries to run a command "name" with two arguments

Reference: precede variable name with $ to access the value

echo $date $temp   # reference variables
echo name          # just text, not a variable reference

Build-in

Automatically set and heavily used inside shell scripts

Access command-line parameter in a shell script

$1 $2 $3...$n  # direct access to parameters (position arguments)
$@             # all parameters
"$@"           # each parameter in double-quotes ("$1" "$2"...)
"$*"           # all parameters in double-quotes ("$1 $2...")
$#             # number of parameters

Shell script process and run-time information

$$             # current shell process ID
$0             # name of executed script
$_             # absolute path to executed script
$?             # most recent exit status
$!             # most recent process ID

Environment Variables

Pre-set environment variables (exported to all child processes)

$PATH          # search path for executables (; separated)
$SHELL         # current interpreter
$PWD           # current working directory
$HOME          # user home directory

Example script to read (input) parameters using build-in variables

cat > /tmp/params <<'EOF'
#!/bin/bash
echo $#: "$@"
echo $1 $5
EOF
chmod +x /tmp/params
/tmp/params 1 a 3 b 6 c

$PATH represents the command search path

  • : colon-separated list of directories that are searched (left to right)
  • Standard system directories such as /bin, /usr/bin
  • Third-party or locally-installed software such as /usr/local/bin
  • Append personally-chosen directories of your scripts
# list the search path (one directory per line)
echo $PATH | tr ':' '\n'
# append our personal scripts directory (in the current shell)
PATH=$PATH:$HOME/bin
# make a shell variable available as an environment variable
export PATH

Expansion

Variable expansion with the character $

  • Weak quoting " (double quotes) expands arguments
  • Strong quoting ' (single quotes) does not expands arguments
  • Prevent expansion with the escape character \
's'              # preserves literal value of characters in string s
"s $V"           # preserves literal value of characters in string s except $ \ ,
\                # escape character (preserves the literal value)
"s \$V"          # using an escape sequenze to prevent variable expansion
"${V}s"          # name to be expanded in braces
$'s'             # expands string s with backslash-escaped characters replaced

<<EOF Here-Documents

Block of code or text which can be redirected…

  • Avoid substitution and expansion by quoting the tag (here EOF)
  • Using <<- suppresses leading tabs (indent closing tag with tabs

Write to a file,,,

cat > /path/to/file <<'EOF'
  # ...content
EOF

Assign to a variable…

var=$(cat <<EOF
  # ...content
EOF
)

Pipe to a command…

cat <<EOF | sort
2
1
3
EOF

Commands

Statement separator ;

Grouping:

  • { … } groups commands purely for syntactic purposes (no sub-shell)
  • ( … ) create a sub-shell and wait for it to terminate

Logic operators: and &&, or ||

c1; c2         # execute command c1 before c2
{c1; c2}       # execute commands in current shell environment
(c1; c2)       # execute the commands inside a sub-shell environment
c1 && c2       # if command c1 runs successful execute c2
c1 || c2       # if command c1 runs not successful execute c2
$(c)           # execute command c in sub-shell (alternative `c`)
c1 $(c2)       # command c1 uses output of c2 as parameters
c &            # execute command c in background

Command substitution: $(…) creates a sub-shell…

  • ..with its standard output set to a pipe…
  • ..collects the output in the parent and expands to that output..

Background: … & sub-shell, does not wait for it to terminate

List of build in commands to know…

set -v         # print shell input lines as they are read
set -x         # print commands and their arguments when executed
set -f         # disable globbing
source f       # execute file f in current shell environment
eval `c`       # execute command c in sub-shell and evaluate
shift          # remove leading positional parameter
jobs           # list active jobs associated with current shell
disown         # detach jobs from current shell
nohup          # continue job after logout
bg             # move job to background 
fg j           # resume job j to foreground
stop j         # stops background job j
trap c s       # execute command c when catching signal s
rehash         # re-index executables in $PATH (Zsh)
reset          # reset terminal

Statements

if...then...else

if statements are to most used control structure

  • Allows for branching in the code execution path
  • Closing fi is necessary, elif and else optional
  • Always include spaces around [[...]] the conditional operator
# evaluate the number of command-line parameters
if [[ $# -eq 1 ]] ; then
        # one argument 
elif [[ $# -ge 2 ]] ; then
        # two or more arguments
else
        # no arguments
fi

Evaluate a mathematical expression with ((...))

if (( $1 + 1 > 2 )); then
        #...
fi

Evaluate the existence of a command in the environment $PATH

if command -v <cmd> >/dev/null ; then
    ...
fi

Execute a command and use its exit status as condition to an if statement:

  • Every program returns an exit status as an integer number
  • 0 (zero) means success, 1-255 non-zero means failure
if <cmd> ; then
    ...
else
    ...
fi
Exit Status Description
1 catchall for general errors
126 command invoked cannot execute
127 command not found

case

  • case statement matches values against a variable
    • argument to case is expanded and matched against patterns
    • in case of a match run commands up to ;;
    • patterns are not regular expressions but shell patterns (aka globs)
case "$1" in
        1)
                echo $1
                ;;
        abc)
                echo def
                ;;
        2|3|4)
                echo 2,3,4
                ;;
        [a-z]*)
                echo a-z
                ;;
        *)
                echo no case
esac

for..do..done

  • iterator: loop over a sequentially changed variable
for (( i = 0; i < 10; i++ ))
do
        echo $i
done
# two variables at once
for (( i = 0, j = 0; i < 10; i++, j = i * i ))
do
        echo $i $j
done
  • examples for for..in..
# brace expansion
for i in {1..10}; do 
    echo $i
done

# loop an array 
arr=(a b c d e f)
for i in "${arr[@]}";do
    echo $i
done

while...do...done

  • loop until a condition is true
# arithmetic
i=0
while (( $i < 10 )) ; do
        echo $i
        ((i++))
done
# comparison 
i=0
while [ $i -lt 10 ] ; do
        echo $i
        i=$[$i+1]
done
  • build an (infinite) endless loop…
while true ; do
    ...
done

Expressions

Conditional expressions:

  • && (and), || (or) have no precedence (left-associative)
  • Bit faster the if...then
!e             # true if expression e is false
e1 && e2       # true if both expressions e1 and e2 are true (alternative e1 -a e2)
e1 || e2       # true if either expression e1 or e2 is true (alternative e1 -o e2)
[[e]]          # returns bool after evaluation of the conditional expression e
:              # null-statement (returns 0)
$[e]           # evaluate integer expression
((e))          # expand and evaluate of integer expression e
$((e))         # expand, evaluate and substitute integer expression e

Brace Expansion

Brace expansion

{a,b,c}                             # brace expansion
{a..z}                              # extened brace expansion, ranges
name.{foo,bar}                      # typical for files
name{,.foo}                         # includes name
{1..20}                             # digits
{01..20}                            # with leading zero
{20..1}                             # reverse
{0..10..2}                          # use increments
{a..d}{1..3}                        # combine braces

Operators

Typical use of the conditional operator [[...]]:

# string compare
s1 = s2         # true if strings s1 equals s2
s1 != s2        # true if strings s1 not equal s2
-n s            # true if string s is not empty
-z s            # true if string s is empty
# file conditions
-c f            # true if f is a special character file
-d p            # true if p is the path to an existing directory
-e f            # true if f is an existing file
-f f            # true if f is an existing file but not an directory
-G f            # true if f exists and is owned by effective group-id
-g f            # true if f exists and is set-group-id.
-k f            # true if f has sticky bit
-L f            # true if f is a symbolic link
-O f            # true if f exists and is owned by the effective user-id.
-r f            # true if f is readable
-S f            # true if f is socket
-s f            # true if f nonzero size
-u f            # true if f set-user-id bit is set
-w f            # true if f is writable
-x f            # true if f is executable
# numerical comparison
n1 -lt n2       # true if integer n1 less then n2
n1 -gt n2       # true if n1 greater then n2
n1 -le n2       # true if n1 less or equal n2
n1 -ge n2       # true if n1 greater or equal n2
n1 -eq n2       # true if n1 equals n2
n1 -ne n2       # true if n1 not equal n2

Empty vs Unset String

Distinguish between empty and unset strings:

case ${var+x$var} in
    (x) echo empty;;
    ("") echo unset;;
    (x*[![:blank:]]*) echo non-blank;;
    (*) echo blank
esac

exit

  • each shell command returns an exit code when it terminates
    • 0 (zero) means success, 1-255 non-zero means failure
    • special variable $?: exit status of the last executed command
# successfull execution of the date command
>>> date |: ; echo $?
0
# list a non-existing directory... returns a non zero exit status
>>> ls /foobar ; echo $?
ls: cannot access '/foobar': No such file or directory
2
  • if branching based on an exit status of a command (substitution)
…
        passwords=$(ccrypt -c $PASSWORD_FILE)
        # check if the password-name is present in the password-file
        if $(echo "$passwords"| grep "^$2" >/dev/null) ; then
                echo "$passwords" | grep "^$2" | cut -d',' -f2
        else
                echo "Password name '$2' does not exist"
                exit 1
        fi
…
  • exit command stops execution and returns an exit status

Functions

  • abstract a segment of program code with a function name name()
echo_n() { echo -n "$@" }
├────┘   ├──────────────┘
│        └────────────────── function name
└─────────────────────────── function body

# function call with arguments
echo_n 1 2 a b
  • function body defined by all code in between { … } (curly brackets)
  • function exit status: specified with return value of 0 to 255
return_one() { 
        echo ${1:-one} # print first argument or 'one'
        return 1       # return failure
}
if return_one # evaluates exit status
then
        echo success
else
        echo failure $(return_one 1)
fi

source

  • source executed a file in the current shell (abbreviated to .)
>>> echo '#!/bin/bash\necho $USER $name' | tee name.sh
#!/bin/bash
echo $USER $name
# creates a new (sub)shell
>>> name=vic ; bash name.sh
vpenso
# work in the current shell
>>> name=vic ; source name.sh 
vpenso vic
  • typically used modify environment variables by sourcing a file
>>> cat ~/.bashrc
if ! [ -z "$(ls -A ~/.bashrc.d)" ] ; then # if the directory is not empty
    for file in `\ls ~/.bashrc.d/*` ; do
        source $file       # execute files in current shell
    done
fi
# configure the shell history, cf. bash manual
>>> cat ~/.bashrc.d/history.sh
export HISTSIZE=100000                       # set history size
export SAVEHIST=$HISTSIZE                    # save history after logout
export HISTCONTROL=ignoreboth                # ignore leading spaces, duplicates
  • ~/bashrc: customize your interactive shell session

Processes

  • a process is the execution of a program
    • a command can only generate a process
    • an application can run multiple processes for different tasks
  • each process is assigned a unique PID (process identification number)
  • process can be initiated as a foreground or background process
    • by default a command starts as foreground processes…
    • …users have to wait until it is finished before continuing
    • run a command as background process by adding an ampersand … &
<cmd> &       # start process in background of the current shell
nohup <cmd> & # detach a process from the current shell
jobs          # processes started in the context of the current shell 
bg %<job_id>  # send send a process to the background
fg %<job_id>  # pull a process to foreground
disown        # detach all processes from current shell
ps            # (process statuses) snapshot of all running processes
pstree        # display a tree of processes
pgrep         # search process ID by criteria
kill          # send a signal to a process
pkill         # search a process and send a signal
killall       # send a signal to a process by name
nice          # run a program with modified scheduling priority
renice        # alter priority of running processes
top           # dynamic real-time view of a running system

kill & trap

Signals: one-way notifications for process

  • Used to manipulate the state of a processes
  • User typically use signals to terminate a process

kill command sends a signal to a process

  • Read about the available signals with man 7 signal
  • Get a short signal listing with kill -L
# shell buildin command behaves different to the external program
>>> type -a kill
kill is a shell builtin
kill is /bin/kill
>>> /bin/kill -L
 1 HUP      2 INT      3 QUIT     4 ILL      5 TRAP     6 ABRT     7 BUS
 8 FPE      9 KILL    10 USR1    11 SEGV    12 USR2    13 PIPE    14 ALRM
15 TERM    16 STKFLT  17 CHLD    18 CONT    19 STOP    20 TSTP    21 TTIN
22 TTOU    23 URG     24 XCPU    25 XFSZ    26 VTALRM  27 PROF    28 WINCH
29 POLL    30 PWR     31 SYS

Signals have a number value 9 and a name KILL

kill -9 <pid>        # kill a program using a signal number
kill -KILL <pid>     # similar using the signal name

Exit and error in a sub-shell, simple cases:

(c)                # execute command c in sub-shell, ignore errors
(c) || exit $?     # same as above but propagate signal code

Catching Signals

Execute commands COMMAND on signal SIG

trap 'COMMAND' SIG [SIG,..]

Trap breaking child processes

set -m
# rescue function executed by trap
ensure() {
        echo "Clean up after child process died with signal $?."
}
# catch errors
trap ensure ERR
# execute child process
segfaulter &    
# wait for the child process to finish
wait $!

zsh Shell

bindkey

  • …no argument …print all keybindings
  • -l …list of existing keymap names
  • -M <KEYMAP> …list all the bindings in a given keymap
    • …Vi mode …keymaps viins and vicmd
  • Documentation…
    • …manual page …man 1 zshzle
    • …zle -al …list all registered line editor commands

Examples

# bind Ctrl-l ro run the `ls` command
bindkey -s "^L" 'ls^M'

Plugins

zplug

Zsh Plugin Manager…

Add following to ~/.zshrc:

export ZPLUG_HOME=~/.zplug

test -d $ZPLUG_HOME \
        || git clone https://github.com/zplug/zplug $ZPLUG_HOME

source $ZPLUG_HOME/init.zsh

zplug "zsh-users/zsh-completions"
# ...add more plugins

if ! zplug check; then
    zplug install
fi
zplug load

Terminals

Text terminals….

  • …serial computer interface for text entry and display
  • …historically…
    • …physical terminals …like the IBM 3270, VT100
    • …keyboard/screen for display and input of data on a remote computer
  • …terminal emulators …called tty
    • …mimic a hardware (video) terminal in software
    • …provide interactive access to programs that run in the command line
    • VTE (virtual terminal emulator)…
      • …emulates a text terminal inside a graphical user interface (GUI)
      • …widgets that implement a fully functional terminal emulator
/dev/tty[0-9]               # teletypes, cf. man `tty`
/dev/pts/[0-9]              # pseudo terminals, cf man `pty`
tty                         # show assoc. pseudo terminal
stty -a                     # show terminal settings
getty                       # program watching a physical terminal (tty) port
ps -a                       # list processes with attached terminals
terminfo 
termcap                     # terminal capability data base
tput
tget
reset                       # init terminal

A system console is a special terminal…

  • On modern computers a directly connected monitor/keyboard
  • Receives messages from the OS regarding booting/shutdown progress
  • Linux support virtual consoles to provide several text terminals
  • Access with a key combination including function keys (e.g. Ctrl+Alt+F2)

Terminology…

  • Control characters display control codes like line-feed, backspace, etc.
  • Escape sequences…
    • …series of characters that give commands to the terminal
    • …user for cursor movement, colors, etc.
    • …consists of the ESC control character followed by a sequence of characters
  • …cut & paste…
    • …note that Ctrl-c breaks current execution in shell
    • Ctrl-Shift-c and Ctrl-Shift-v for cut & paste

gnome-console

GNOME Console

  • …simple user-friendly terminal emulator
  • …minimalistic designed for Mobile devices as part of Phosh

gnome-terminal

GNOME Terminal…

  • …installed by default with GNOME
  • Options…
    • --window …open a new window
    • --title …set title of tab/window
    • --tab …open a new tab in window
    • --command …first command to execute…
      • …can be a specific shell Zsh
      • …or a terminal multiplexer like Tmux
  • Shortcuts …preference menu …shortcuts tab
# open two tabs called local and remote ...local tab starts Tmux
gnome-terminal --window --title remote --tab --title local --command tmux
# typically mapped to a keyboard shortcut like F2 or Alt+9

tabby

Encoding

System of rules to convert information (letters, words, sound, images) into another form or representation for data transmissions/storage…

  • character encodings are representations of textual data
  • decoding is the reverse process, converting code symbols back
  • ASCII: single byte encoding only using the bottom 7 bits (man ascii)
    • first 32 codes (0–31 decimal) for control characters
    • letters A-Za-z, digits 0-9, punctuation !#$%&'()*+,-. etc.
  • UTF-8 general-purpose way of representing Unicode characters
    • encoded as sequence of 1-4 bytes (variable width character encoding)
    • designed for backward compatibility with ASCII
locale                   # current language and encoding settings           
localectl                # ^^with systemd tools
file -bi <file>          # show file encoding
# change encoding from ASCII to UTF-8
iconv -f utf-8 -t ascii -o <out_file> <in_file>
# change encoding from UTF-8 charset to ASCII, omit invalid characters
iconv -c -f utf-8 -t ascii -o <out_file> <in_file>

ASCII & Unicode

ASCII], single byte encoding only using the bottom 7 bits

  • ASCII order (ASCIIbetical)
    • Uppercase come before lowercase letters
    • Digits and many punctuation marks come before letters
  • First 32 codes (0–31 decimal) for control characters
  • Many 8-bit encodings which are supersets of ASCII

UTF-8, general-purpose way of representing Unicode characters

  • Character encoded as sequence of 1-4 bytes (variable width character encoding)
  • Designed for backward compatibility with ASCII (valid ASCII text is valid UTF-8-encoded Unicode)

Notations & Escaping

Conventions to describe characters

  • Octal digit characters (01234567), \<octal> 1,2 or 3
  • Escape sequence, starting with a backslash \[abfnrtv]
  • Caret notation, ^ followed by a single character (usually a capital letter)
printf `\O`                         # print octal
printf '\xH'                        # print hex H
printf "\\$(printf %o D)"           # print decimal d

Non-Printable Characters

Non-printable (white-space) characters:

  • space (blank, word divider)
  • backspace (BS), \b, ^H
  • tab (horizontal tab (HT), \t, \011, ^I
  • newline (line feed (LF)), \n, \012, ^J
  • null (NUL) \0, ^@
  • escape (ESC) \e, ^[

cat show non-printable characters with -A (equivalent to -vET)

>>> s="\tone\n\011two\040three\0"
>>> echo "$s" | cat -A
^Ione$
^Itwo three^@$

^ and M- notation (100-137 ascii chars), for LF $

od show non-printable chars with backslash escapes:

>>> echo "$s" | od -c
0000000  \t   o   n   e  \n  \t   t   w   o       t   h   r   e   e  \0
0000020  \n
0000021