Systemd Configuration

Linux

Systemd

Published

January 24, 2016

Modified

October 4, 2022

Initialize user-space…

…first process executed in users space (PID 1)
…provides dependency management between units

Each Unit is describes with a configuration file…

…suffix determines the unit type unit_name.type_extension
target…
- …group of units (like run levels)
- …may inherit services from another target
scope are groups of processes registered using the runtime APIs (e.g. containers).
slice…
- …hierarchies of services/scopes (contain processes)
- …inherit resource limits to descendants
- …-.slice root slice…system.slice default slice
Every process is spawned in a control group named after its service

systemd --version                            # show systemd version
/etc/os-release                              # platform information
/etc/machine-id                              # unique machine identifier
man systemd.index                            # overview documentation
{/etc,/run,/lib}/systemd/system/             # unit configuration files
LESS="-p SIGNALS" man -P less systemd        # list of supported signals
LESS="-p KERNEL" man -P less systemd         # kernel command line options for boot
SYSTEMD_LESS=FRXMK                           # export to wrap lines to screen width

Usage

`systemctl`

systemctl                                    # list all units with state
systemctl -t <unit_type>                     # list units with a given type, e.g. "service"
systemctl list-unit-files -t service         # list unit files for a given unit type 
systemctl reenable <unit>                    # reconfigure existing unit
systemctl --failed --all                     # list units in failed state
systemctl help <unit>
systemctl status [<unit>]                    # show state of a unit, e.g. ssh.service
systemctl is-enabled <unit>                  # check if a unit will bi started during init
systemctl is-active <unit>
systemctl list-dependencies <unit>           # show unit dependencies
systemctl start|stop|restart|reload <unit>   # state management of units
systemctl -t target                          # list available targets
systemctl get-default                        # show current default target at boot
systemctl set-default -f <target>            # set a new default target
systemctl show -p Wants <target>             # show dependencies to a target
systemctl isolate <target>                   # switch to another target
systemctl kill <service>                     # send TERM to service
systemctl kill -s <sig> <service>            # send a given signal to servive, e.g. HUB  
systemctl reboot|poweroff|suspend|hibernate  # power management
systemctl list-jobs                          # show pending jobs
systemctl daemon-reload                      # Re-read configuration files
systemctl daemon-reexec                      # re-execute systemd
systemd-delta --type=extended                # Identify configuration which override others

`cat`, `show` & `edit`

cat show backing files of one or more units

>>> systemctl cat dbus.service              
# /usr/lib/systemd/system/dbus-broker.service
[Unit]
Description=D-Bus System Message Bus
Documentation=man:dbus-broker-launch(1)
DefaultDependencies=false
...

show print properties of one or more units

…expose runtime state in addition to configuration
…used whenever computer-parsable output is required
…--property= to select specific properties
…empty properties are suppressed…--all

edit unit configuration…

…default to drop-in snippet…overwrite.conf
-l/--full work on the original unit
…after editing…systemd configuration is reloaded
--runtime changes will be made temporarily in /run/
- …only if unit is in /usr/lib/systemd/system
- …otherwise /etc/systemd/system take precedents

`localectl`

Localization…sets configuration for native language support…

Output of programs translated into the native language
Classification of characters into letters, digits and other classes
Correct alphabetical sorting order for the country
Appropriate default paper size
Correct formatting of monetary, time, and date values

Names are typically of the form language[_territory][.codeset][@modifier]

# from /etc/locale.conf 
LANG=en_US.UTF-8
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_MONETARY=de_DE.UTF-8
LC_PAPER=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8

/etc/locale.conf holds the system-wide locale settings

LANG…default local…used for all the LC_* variables not explicitly set
LC_NUMERIC…formatting rules used for nonmonetary numeric values
LC_TIME…formatting used for date and time values
…more details in man 7 locale

localectl queries and changes the system locale and keyboard layout settings

localectl                           # show language configuration
localectl list-locales              # list vailable keys configuration
locale -a
localectl set-locale LANG="en_US.UTF-8" LC_CTYPE="en_US"
localectl list-keymaps              # list all available keyboard layouts
localectl set-keymap us,de          # persistent configuration
# get messages from the service log
journalctl -u systemd-localed

Instead of manually editing X configuration files:

localectl --no-convert set-x11-keymap us,de pc104,dvorak grp:alt_shift_toggle
# writes /etc/X11/xorg.conf.d/00-keyboard.conf

`systemd-nspawn`

apt -y install systemd-container                 # install container support 
man systemd.nspaw                                # container settings documentation
{/etc,/run}/systemd/nspawn/*.nspawn              # nspawn container settings files
findmnt /var/lib/machines                        # container images & container settings
systemd-nspawn -D <rootfs>                       # chroot to container path
               -i <image>                        # chroot to a container image
               -b -D <rootfs>                    # boot container in path 
machinectl pull-raw --verify=no <url>            # download container archive 
           import-tar $archive                   # import rootfs from an archive
           list-images                           # list container images
           image-status <image>                  # status information about container image
           show-image <image>                    # properties of container image
           clone <src_image> <dst_image>         # clone a container image
           remove <image>                        # delete a continer image
           start <image>                         # start container
           list                                  # list running containers
           status <image>                        # process tree in container
           shell <user>@<image> /bin/bash        # start a shell in the container
           poweroff <image>                      # shutdown container
journalctl -M <image>                            # show log of container

Unit skeleton for a container in /etc/systemd/system/${name}.service

[Unit]
Description= # name of the container

[Service]
ExecStart=/usr/bin/systemd-nspawn -bD <rootfs>  # command to start the container

Use a virtual ethernet interfaces:

echo -e "[Link]\nName=host0" > /etc/systemd/network/10-host0.link
echo -e "[Match]\nName=host0\n[Network]\nDHCP=yes" > /etc/systemd/network/11-host0.network
systemctl enable systemd-networkd                          # prepare the network configuration
## -- use a virtual network with a container -- ##
systemd-nspawn --network-veth --network-bridge=nbr0 ...

`systemd-analyze`

systemd-analyze                              # time the system required during last booting
systemd-analyze blame                        # ^^ time by by unit
systemd-analyze critical-chain
systemd-analyze verify $path.service         # validate a systemd unit

Boot

Systemd kernel command-line arguments [1]:

console=ttyS1,38400 console=tty0         # configure the serial console [2]
systemd.log_level=debug                  # enable debugging output
systemd.log_target=console               # send debugging output to serial console
systemd.journald.forward_to_console=1    # configure journald to forward to the console
systemd.debug-shell=1                    # enable shell access early in the startup process to fall back on and diagnose systemd
systemd.confirm_spawn=1                  # asks for confirmation when spawning processes
systemd.unit=rescue.target               # boot directly into rescue target (if problem occurs somewhere after the basic system is brought up)
systemd.unit=emergency.target            # boot directly into emergency shell
init=/bin/sh                             # boot directly into a shell (no systemd as PID 1)

References…

Systemd - Diagnosing Boot Problems
- https://freedesktop.org/wiki/Software/systemd/Debugging/
Linux Serial Console
- https://www.kernel.org/doc/html/v4.14/admin-guide/serial-console.html
Systemd - Kernel Command Line
- https://www.freedesktop.org/software/systemd/man/kernel-command-line.html

`systemd-logind`

systemctl status systemd-logind.service         # state of the login manager
man logind.conf                                 # login manager configuration

pam_systemd registers user sessions with the systemd login manager

apt install libpam-systemd                      # install PAM support for systemd
grep pam_systemd.so /etc/pam.d/*                # PAM configuration for systemd
systemctl status dbus.service                   # DBus is required by pam_systemd.so

All user sessions session-<id>.scope of belong to a slice user-<uid>.slice below the user.slice

loginctl [list-users]                           # list users
loginctl user-status <user>                     # show run-time information of user
loginctl terminate-user <user>                  # terminate all user sessions
loginctl list-seats                             # list seats
loginctl seat-status <seat>                     # list devices associated to seat
ls ~<user>/.config/systemd/user                 # unit files for a given user
loginctl enable-linger <user>                   # make user sessions (boot) persistant

Systemd Units

`*.service` Units

References…

…man systemd.service & man systemd.exec
…man systemd.unit applies as well

Skeleton for a custom service unit /etc/systemd/system/*.service to execute a program once:

[Unit]
Description= # text

[Service]
Type=oneshot
ExecStart= # absolute path to program, arguments

Service Action

Must include a service action…

[Service]
Type=....
ExecStart=...

Type…process start-up type…

simple…expected that the process configured with ExecStart= is the main process
exec…consider the unit started immediately after the main service binary has been executed
forking…process configured with ExecStart= will call fork() as part of its start-up
- …PIDFile= read the PID of the main process of the service from this file after start-up of the service
oneshot…consider the unit up after the main process exits…

ExecStart…executed when this service is started…

First argument is the service executable…
- …according to command lines rules
- …absolute path recommended…avoid ambiguity
- …non absolute path…fixed standard path {/usr,/usr/local}/{bin,sbin}
- …systemd-path search-binaries-default to print standard path

Environment

Environment variables may be defined…

Environment=
- …space separated list of variable assignments
- …quote whole assignment for values containing spaces or the equals sign
- …option may be specified more than once
EnvironmentFile=
- …reads the environment variables from a text file
  - …absolute filename or wildcard expression
  - …- prefix…ignores non existing files
  - …read shortly before the process is executed
- …contents…
  - …override settings made with Environment=
  - …newline-separated variable assignments (# for comments)
  - …line ending \…discards newline to continue in the following line

Example…

# script to show the environment
cat > /usr/local/sbin/env-echo <<'EOF'
#!/usr/bin/env sh
while true ; do echo ARGS $@ ; env ; sleep 10 ; done
EOF
chmod +x /usr/local/sbin/env-echo
# environment file...
cat > /etc/default/env-echo <<'EOF'
VAR2=val2
VAR4=val5 val6 val7
EOF
# systemd unit
cat > /etc/systemd/system/env-echo.service <<'EOF'
[Unit]
Description=Echo Environment Variables
[Service]
Type=simple
Environment=VAR1=val1 "VAR3=val3 val4"
EnvironmentFile=/etc/default/env-echo
ExecStart=env-echo $VAR1 $VAR2 $VAR3 $VAR4
[Install]
WantedBy=default.target
EOF
systemctl daemon-reload && systemctl restart env-echo
# inpsect the service environment...
journalctl -xeu env-echo | grep -e ARGS -e VAR

`*.timer` Units

Timers .timer control services or events (similar to Cron) supporting calender/monotonic time

man systemd.timer
systemctl list-timers --all                     # list all timers, including inactive
systemd-run --on-active=30 <command>            # transient .timer unit executes a command
systemd-run --on-active=<time> --unit <unit>    # transient .timer unit executes unit
apt install systemd-cron                        # systemd units to run cron scripts

Cleaning Temporary Directories

man 5 tmpfiles.d                                # documentation manual
{/etc,/run,/usr/lib}/tmpfiles.d/*.conf          # configuraion path
systemctl status systemd-tmpfiles-clean.timer   # timer state

`*.mount` Units

Systemd Mount Unit Configuration
https://www.freedesktop.org/software/systemd/man/systemd.mount.html

Systemd Special Units
https://www.freedesktop.org/software/systemd/man/systemd.special.html

/etc/fstab                                      # translated by systemd-fstab-generator into units
/etc/systemd/system/*.mount                     # mount units
systemctl --all -t mount                        # show mounts

Mount units must be named after the mount point directories they control, cf systemd-escape.

Unit skeleton for a local file-system:

[Unit]
Description= # comment

[Mount]
What= # partition name, path or UUID to mount
Where= # path to a mount point
Type= # file system type (e.g. ext4)
Options=defaults

[Install]
WantedBy=local-fs.target

Unit skeleton for an NFS mount:

[Unit]
Description= # comment
Wants=network-online.target
After=network-online.target

[Mount]
What= # path uuid, e.g. nfs.devops.test:/srv/nfs/devops
Where= # path to a mount point
Type=nfs4
Options=defaults
TimeoutSec=10s

[Install]
WantedBy=network-online.target

Use tmpfs to mount /tmp:

>>> cp /usr/share/systemd/tmp.mount /etc/systemd/system/tmp.mount
>>> systemctl enable tmp.mount && systemctl start tmp.mount

Resource Limits

Enforce resource policies (CPU, RAM, I/O) for units with Linux Control Groups (cgroups)

man systemd.resource-control                      # documentation about resource limits
man systemd-system.conf                           # system wide defaults
systemd-cgtop                                     # enumerate all cgroups of the system including resource usage
systemd-cgls                                      # hierarchy of slices, scopes, and units

Limits are defined in the service section:

[Service]
ControlGroupAttribute=               # e.g. memory.swappiness 70
CPUShares=<weight>                   # relative shares, e.g. 1024 (default)
CPUQuota=<perc>                      # time in percent, e.g. 20%
MemoryLimit=<bytes>                  # allocation limit, e.g. 1G
MemorySoftLimit=<bytes>              # suffix K, M, G, T
BlockIOWeight=<target> <weight>      # I/O bandwidth between 10 and 1000

Network

`hostnamectl`

hostnamectl status                  # view hostname
hostnamectl set-hostname <fqdn>     # set hostname
hostnamectl set-hostname ""         # clear hostname
/etc/hostname                       # hostname configuration file
/etc/hosts                          # may contain FQDN of host (to be resolved without DNS)
dnsdomainname                       # display DNS domain name
domainname                          # (aka. nisdomainname) display NIS domain name
hostname                            # print hostname returned by the gethostname(2) function
hostname -s                         # print hostname cut at the first dot
hostname -f                         # print host FQDN
hostname -d                         # print host domain name

`systemd-networkd`

Systemd service that manages network configurations:

Detects and configures network devices as they appear

ls -l {/etc,/run,/lib}/systemd/network/*.network # network configuration 
networkctl list                                  # list network connections
networkctl status                                # show IP addresses for interfaces

Persistently enable the service:

# eventually prevent NetworkManager [1] from controlling the network interface:
systemctl disable --now NetworkManager
systemctl mask NetworkManager
# enable systemd-networkd
systemctl enable --now systemd-networkd

Debugging:

# execute with debugging in foreground
SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd   
# Permanent by drop-in configuration
mkdir /etc/systemd/system/systemd-networkd.service.d/
echo -e "[Service]\nEnvironment=SYSTEMD_LOG_LEVEL=debug" \
        > /etc/systemd/system/systemd-networkd.service.d/10-debug.conf 
systemctl daemon-reload && systemctl restart systemd-networkd

Predictable interface names

Prefix en Ethernet wl WLAN with following types:

o<index>                                                         on-board device index number
s<slot>[f<function>][d<dev_id>]                                  hotplug slot index number
x<MAC>                                                           MAC address
p<bus>s<slot>[f<function>][d<dev_id>]                            PCI geographical location
p<bus>s<slot>[f<function>][u<port>][..][c<config>][i<interface>] USB port number chain

Cf. /lib/udev/rules.d/80-net-setup-link.rules, cf. systemd.link

udevadm info -e | grep -A 9 ^P.*en[sop]          # dump udev database and grep for ethernet
udevadm test /sys/class/net/* 2>&- | grep ID_NET_NAME_
## -- .link file are used to rename an interface -- ##
ls -l {/etc,/run,/lib}/systemd/network/*.link    # list link configuration files
ip -o -4 link                                    # show link state

Unit File

Configure the network, cf. systemd.network

Skeleton for a dynamic IP-address

[Match]
Name=                              # device name (e.g en*)

[Network]
DHCP={yes,no,ipv4,ipv6}            # enable DHCP

Skeleton for a static IP-address

[Match]
Name=              # device name (e.g en*)

[Network]
Address=           # IP address, CIDR notation
Gateway=           # IP address
DNS=               # is a DNS server address (multiples possibel)
Domains=           # a list of the domains used for DNS host name resolution

Wired & Wireless in Parallel

cat <<EOF | sudo tee /etc/systemd/network/20-wired.network
[Match]
Name=en*

[Network]
DHCP=ipv4

[DHCP]
RouteMetric=10
EOF 
cat <<EOF | sudo tee /etc/systemd/network/25-wireless.network
[Match]
Name=wl*

[Network]
DHCP=ipv4

[DHCP]
RouteMetric=20
EOF

Note that the wireless interface needs to be connected by another service like iwd.

systemd-resolved service is required if DNS entries are specified in .network files

`systemd-resolved`

Provides network name resolution…

…supports DNSSEC and DNS over TLS
…glibc NSS interface plugin nss-resolve

systemctl enable --now systemd-resolved # start the service daemon
systemd-resolve --status
/etc/systemd/resolved.conf              # conf. file
/etc/systemd/resolved.conf.d/*.conf     # drop-in conf. files

Fallback DNS addresses ensure DNS resolution in case no config is present

Provides a local DNS stub listener on IP address 127.0.0.53 on the local loopback interface:

rm /etc/resolv.conf
# redirect glibc NSS conf. file to local stub DNS resolver 
ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
# operations mode detected automatically depending on existence of this link

Configuration

Temporarily change the DNS server configuration:

LINK specifies the network interface, get the name from resolvectl status or ip addr
SERVER specifies the IP address of a DNS server i.e. 1.1.1.1

systemd-resolve -i LINK --set-dns=SERVER
# or
sudo resolvectl dns LINK SERVER

Configure a custom list of DNS resolvers, and enable DNSSEC

mkdir /etc/systemd/resolved.conf.d
# use a drop in .conf file for the Quad9 primary DNS resolvers 
cat > /etc/systemd/resolved.conf.d/dns.conf <<EOF
[Resolve]
DNS=9.9.9.9 149.112.112.112
EOF
# if supported by the DNS resolvers, enforce DNSSEC validation
cat > /etc/systemd/resolved.conf.d/dnssec.conf <<EOF
[Resolve]
DNSSEC=true
EOF
# if supported by the DNS resolver, attempt to use DNS over TLS
cat > /etc/systemd/resolved.conf.d/dot.conf <<EOF
[Resolve]
DNSOverTLS=opportunistic
EOF

systemd-resolved currently only supports opportunistic DNS over TLS resolution

Resolver tries resolution using DoT before fall back to traditional DNS (allowing for downgrade attacks)
Eventually another option will be added strict to prevent fallback

DNS server certificates are not checked making systemd-resolved vulnerable to man-in-the-middle attacks

`resolvectl`

resolvectl status                       # check service status
resolvectl query <ip|name>              # query DNS records
resolvectl statistics                   # show DNS cache stats

Enable debugging and follow the logs:

resolvectl log-level debug             
journalctl -fu systemd-resolved
# disable debugging
resolvectl log-level info

`systemd-timesyncd`

Types of clocks:

RTC (Real-Time Clock) aka hardware clock runs on battery when the machine is powered off (unplugged)
- Uses local time (in current time zone)
- Or UTC (Coordinated Universal Time) with an time zone dependent offset
SC (Software Clock) aka system clock
- Maintained by the OS kernel during run-time driven by a timer interrupt
- Initialized on boot using RTC as reference.

timedatectl                                 # show time and time zone configuration
timedatectl set-time YYYY-MM-DD             # change the current date
timedatectl set-time HH:MM:SS               # change the current time
timedatectl list-timezones                  # list available time zones
timedatectl set-timezone <zone>             # set a given time zone, e.g. Europe/Berlin
grep ^Servers /etc/systemd/timesyncd.conf   # list time servers 
timedatectl set-ntp true                    # enable NTP
timedatectl set-local-rtc 0                 # RTC in UTC mode
timedatectl set-local-rtc 1                 # RTC in locel time mode
systemctl start systemd-timesyncd           # start the time sync daemon 
systemctl enable systemd-timesyncd          # make the time sync daemon boot persistant

Low-level administration tool

locale                                      # print locale settings
hwclock                                     # manipulate the hardware clock
/etc/adjtime                                # state file for hwclock
/usr/share/zoneinfo                         # time zone database

--- title: Systemd Configuration categories: - Linux - Systemd date: 2016/01/24 date-modified: 2022/10/04 toc-expand: 3 --- Initialize user-space... - ...first process executed in users space (PID 1) - ...provides dependency management between **units** Each Unit is describes with a configuration file... - ...suffix determines the unit type `unit_name.type_extension` - `target`... - ...group of units (like run levels) - ...may inherit services from another target - `scope` are groups of processes registered using the runtime APIs (e.g. containers). - `slice`... - ...hierarchies of services/scopes (contain processes) - ...inherit resource limits to descendants - ...`-.slice` root slice...`system.slice` default slice - Every process is spawned in a control group named after its service ```bash systemd --version # show systemd version /etc/os-release # platform information /etc/machine-id # unique machine identifier man systemd.index # overview documentation {/etc,/run,/lib}/systemd/system/ # unit configuration files LESS="-p SIGNALS" man -P less systemd # list of supported signals LESS="-p KERNEL" man -P less systemd # kernel command line options for boot SYSTEMD_LESS=FRXMK # export to wrap lines to screen width ``` # Usage ## `systemctl` ```bash systemctl # list all units with state systemctl -t <unit_type> # list units with a given type, e.g. "service" systemctl list-unit-files -t service # list unit files for a given unit type systemctl reenable <unit> # reconfigure existing unit systemctl --failed --all # list units in failed state systemctl help <unit> systemctl status [<unit>] # show state of a unit, e.g. ssh.service systemctl is-enabled <unit> # check if a unit will bi started during init systemctl is-active <unit> systemctl list-dependencies <unit> # show unit dependencies systemctl start|stop|restart|reload <unit> # state management of units systemctl -t target # list available targets systemctl get-default # show current default target at boot systemctl set-default -f <target> # set a new default target systemctl show -p Wants <target> # show dependencies to a target systemctl isolate <target> # switch to another target systemctl kill <service> # send TERM to service systemctl kill -s <sig> <service> # send a given signal to servive, e.g. HUB systemctl reboot|poweroff|suspend|hibernate # power management systemctl list-jobs # show pending jobs systemctl daemon-reload # Re-read configuration files systemctl daemon-reexec # re-execute systemd systemd-delta --type=extended # Identify configuration which override others ``` ### `cat`, `show` & `edit` `cat` show backing files of one or more units ```sh >>> systemctl cat dbus.service # /usr/lib/systemd/system/dbus-broker.service [Unit] Description=D-Bus System Message Bus Documentation=man:dbus-broker-launch(1) DefaultDependencies=false ... ``` `show` print properties of one or more units - ...expose runtime state in addition to configuration - ...used whenever computer-parsable output is required - ...`--property=` to select specific properties - ...empty properties are suppressed...`--all` `edit` unit configuration... - ...default to drop-in snippet...`overwrite.conf` - `-l`/`--full` work on the original unit - ...after editing...systemd configuration is reloaded - `--runtime` changes will be made temporarily in `/run/` - ...only if unit is in `/usr/lib/systemd/system` - ...otherwise `/etc/systemd/system` take precedents ## `localectl` Localization...sets configuration for native language support... - Output of programs translated into the native language - Classification of characters into letters, digits and other classes - Correct alphabetical sorting order for the country - Appropriate default paper size - Correct formatting of monetary, time, and date values Names are typically of the form `language[_territory][.codeset][@modifier]` ```sh # from /etc/locale.conf LANG=en_US.UTF-8 LC_NUMERIC=de_DE.UTF-8 LC_TIME=de_DE.UTF-8 LC_MONETARY=de_DE.UTF-8 LC_PAPER=de_DE.UTF-8 LC_MEASUREMENT=de_DE.UTF-8 ``` `/etc/locale.conf` holds the system-wide locale settings - `LANG`...default local...used for all the `LC_*` variables not explicitly set - `LC_NUMERIC`...formatting rules used for nonmonetary numeric values - `LC_TIME`...formatting used for date and time values - ...more details in `man 7 locale` `localectl` queries and changes the system locale and keyboard layout settings ```bash localectl # show language configuration localectl list-locales # list vailable keys configuration locale -a localectl set-locale LANG="en_US.UTF-8" LC_CTYPE="en_US" localectl list-keymaps # list all available keyboard layouts localectl set-keymap us,de # persistent configuration # get messages from the service log journalctl -u systemd-localed ``` Instead of manually editing X configuration files: ```bash localectl --no-convert set-x11-keymap us,de pc104,dvorak grp:alt_shift_toggle # writes /etc/X11/xorg.conf.d/00-keyboard.conf ``` ## `systemd-nspawn` ```bash apt -y install systemd-container # install container support man systemd.nspaw # container settings documentation {/etc,/run}/systemd/nspawn/*.nspawn # nspawn container settings files findmnt /var/lib/machines # container images & container settings systemd-nspawn -D <rootfs> # chroot to container path -i <image> # chroot to a container image -b -D <rootfs> # boot container in path machinectl pull-raw --verify=no <url> # download container archive import-tar $archive # import rootfs from an archive list-images # list container images image-status <image> # status information about container image show-image <image> # properties of container image clone <src_image> <dst_image> # clone a container image remove <image> # delete a continer image start <image> # start container list # list running containers status <image> # process tree in container shell <user>@<image> /bin/bash # start a shell in the container poweroff <image> # shutdown container journalctl -M <image> # show log of container ``` Unit skeleton for a container in `/etc/systemd/system/${name}.service` ```ini [Unit] Description= # name of the container [Service] ExecStart=/usr/bin/systemd-nspawn -bD <rootfs> # command to start the container ``` Use a virtual ethernet interfaces: ```bash echo -e "[Link]\nName=host0" > /etc/systemd/network/10-host0.link echo -e "[Match]\nName=host0\n[Network]\nDHCP=yes" > /etc/systemd/network/11-host0.network systemctl enable systemd-networkd # prepare the network configuration ## -- use a virtual network with a container -- ## systemd-nspawn --network-veth --network-bridge=nbr0 ... ``` ## `systemd-analyze` ```sh systemd-analyze # time the system required during last booting systemd-analyze blame # ^^ time by by unit systemd-analyze critical-chain systemd-analyze verify $path.service # validate a systemd unit ``` # Boot Systemd kernel command-line arguments [1]: ```bash console=ttyS1,38400 console=tty0 # configure the serial console [2] systemd.log_level=debug # enable debugging output systemd.log_target=console # send debugging output to serial console systemd.journald.forward_to_console=1 # configure journald to forward to the console systemd.debug-shell=1 # enable shell access early in the startup process to fall back on and diagnose systemd systemd.confirm_spawn=1 # asks for confirmation when spawning processes systemd.unit=rescue.target # boot directly into rescue target (if problem occurs somewhere after the basic system is brought up) systemd.unit=emergency.target # boot directly into emergency shell init=/bin/sh # boot directly into a shell (no systemd as PID 1) ``` References... - Systemd - Diagnosing Boot Problems - <https://freedesktop.org/wiki/Software/systemd/Debugging/> - Linux Serial Console - <https://www.kernel.org/doc/html/v4.14/admin-guide/serial-console.html> - Systemd - Kernel Command Line - <https://www.freedesktop.org/software/systemd/man/kernel-command-line.html> ## `systemd-logind` ```bash systemctl status systemd-logind.service # state of the login manager man logind.conf # login manager configuration ``` `pam_systemd` registers user sessions with the systemd login manager ```bash apt install libpam-systemd # install PAM support for systemd grep pam_systemd.so /etc/pam.d/* # PAM configuration for systemd systemctl status dbus.service # DBus is required by pam_systemd.so ``` All user sessions `session-<id>.scope` of belong to a slice `user-<uid>.slice` below the `user.slice` ```bash loginctl [list-users] # list users loginctl user-status <user> # show run-time information of user loginctl terminate-user <user> # terminate all user sessions loginctl list-seats # list seats loginctl seat-status <seat> # list devices associated to seat ls ~<user>/.config/systemd/user # unit files for a given user loginctl enable-linger <user> # make user sessions (boot) persistant ``` # Systemd Units ## `*.service` Units References... - ...[`man systemd.service`](https://www.freedesktop.org/software/systemd/man/systemd.service.html) & [`man systemd.exec`](https://www.freedesktop.org/software/systemd/man/systemd.exec.html) - ...[`man systemd.unit`](https://www.freedesktop.org/software/systemd/man/systemd.unit.html) applies as well Skeleton for a custom service unit `/etc/systemd/system/*.service` to execute a program once: ```ini [Unit] Description= # text [Service] Type=oneshot ExecStart= # absolute path to program, arguments ``` ### Service Action Must include a service action... ```ini [Service] Type=.... ExecStart=... ``` `Type`...process start-up type... - `simple`...expected that the process configured with `ExecStart=` is the main process - `exec`...consider the unit started immediately after the main service binary has been executed - `forking`...process configured with `ExecStart=` will call `fork()` as part of its start-up - ...`PIDFile=` read the PID of the main process of the service from this file after start-up of the service - `oneshot`...consider the unit up after the main process exits... `ExecStart`...executed when this service is started... - First argument is the service executable... - ...according to [command lines](https://www.freedesktop.org/software/systemd/man/systemd.service.html#Command%20lines) rules - ...absolute path recommended...avoid ambiguity - ...non absolute path...fixed standard path `{/usr,/usr/local}/{bin,sbin}` - ...`systemd-path search-binaries-default` to print standard path ### Environment Environment variables may be defined... - [`Environment=`](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Environment=) - ...space separated list of variable assignments - ...quote whole assignment for values containing spaces or the equals sign - ...option may be specified more than once - [`EnvironmentFile=`](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#EnvironmentFile=) - ...reads the environment variables from a text file - ...absolute filename or wildcard expression - ...`-` prefix...ignores non existing files - ...read shortly before the process is executed - ...contents... - ...override settings made with `Environment=` - ...newline-separated variable assignments (`#` for comments) - ...line ending `\`...discards newline to continue in the following line Example... ```sh # script to show the environment cat > /usr/local/sbin/env-echo <<'EOF' #!/usr/bin/env sh while true ; do echo ARGS $@ ; env ; sleep 10 ; done EOF chmod +x /usr/local/sbin/env-echo # environment file... cat > /etc/default/env-echo <<'EOF' VAR2=val2 VAR4=val5 val6 val7 EOF # systemd unit cat > /etc/systemd/system/env-echo.service <<'EOF' [Unit] Description=Echo Environment Variables [Service] Type=simple Environment=VAR1=val1 "VAR3=val3 val4" EnvironmentFile=/etc/default/env-echo ExecStart=env-echo $VAR1 $VAR2 $VAR3 $VAR4 [Install] WantedBy=default.target EOF systemctl daemon-reload && systemctl restart env-echo # inpsect the service environment... journalctl -xeu env-echo | grep -e ARGS -e VAR ``` ## `*.timer` Units Timers `.timer` control services or events (similar to Cron) supporting calender/monotonic time ```bash man systemd.timer systemctl list-timers --all # list all timers, including inactive systemd-run --on-active=30 <command> # transient .timer unit executes a command systemd-run --on-active=<time> --unit <unit> # transient .timer unit executes unit apt install systemd-cron # systemd units to run cron scripts ``` Cleaning Temporary Directories ```bash man 5 tmpfiles.d # documentation manual {/etc,/run,/usr/lib}/tmpfiles.d/*.conf # configuraion path systemctl status systemd-tmpfiles-clean.timer # timer state ``` ## `*.mount` Units Systemd Mount Unit Configuration https://www.freedesktop.org/software/systemd/man/systemd.mount.html Systemd Special Units https://www.freedesktop.org/software/systemd/man/systemd.special.html ```bash /etc/fstab # translated by systemd-fstab-generator into units /etc/systemd/system/*.mount # mount units systemctl --all -t mount # show mounts ``` Mount units must be named after the mount point directories they control, cf `systemd-escape`. Unit skeleton for a local file-system: ```ini [Unit] Description= # comment [Mount] What= # partition name, path or UUID to mount Where= # path to a mount point Type= # file system type (e.g. ext4) Options=defaults [Install] WantedBy=local-fs.target ``` Unit skeleton for an NFS mount: ```ini [Unit] Description= # comment Wants=network-online.target After=network-online.target [Mount] What= # path uuid, e.g. nfs.devops.test:/srv/nfs/devops Where= # path to a mount point Type=nfs4 Options=defaults TimeoutSec=10s [Install] WantedBy=network-online.target ``` Use tmpfs to mount /tmp: ```bash >>> cp /usr/share/systemd/tmp.mount /etc/systemd/system/tmp.mount >>> systemctl enable tmp.mount && systemctl start tmp.mount ``` # Resource Limits Enforce resource policies (CPU, RAM, I/O) for units with Linux Control Groups (cgroups) ```bash man systemd.resource-control # documentation about resource limits man systemd-system.conf # system wide defaults systemd-cgtop # enumerate all cgroups of the system including resource usage systemd-cgls # hierarchy of slices, scopes, and units ``` Limits are defined in the service section: ```ini [Service] ControlGroupAttribute= # e.g. memory.swappiness 70 CPUShares=<weight> # relative shares, e.g. 1024 (default) CPUQuota=<perc> # time in percent, e.g. 20% MemoryLimit=<bytes> # allocation limit, e.g. 1G MemorySoftLimit=<bytes> # suffix K, M, G, T BlockIOWeight=<target> <weight> # I/O bandwidth between 10 and 1000 ``` # Network ### `hostnamectl` ```bash hostnamectl status # view hostname hostnamectl set-hostname <fqdn> # set hostname hostnamectl set-hostname "" # clear hostname /etc/hostname # hostname configuration file /etc/hosts # may contain FQDN of host (to be resolved without DNS) dnsdomainname # display DNS domain name domainname # (aka. nisdomainname) display NIS domain name hostname # print hostname returned by the gethostname(2) function hostname -s # print hostname cut at the first dot hostname -f # print host FQDN hostname -d # print host domain name ``` ## `systemd-networkd` Systemd service that manages network configurations: * Detects and configures network devices as they appear ```bash ls -l {/etc,/run,/lib}/systemd/network/*.network # network configuration networkctl list # list network connections networkctl status # show IP addresses for interfaces ``` Persistently enable the service: ```bash # eventually prevent NetworkManager [1] from controlling the network interface: systemctl disable --now NetworkManager systemctl mask NetworkManager # enable systemd-networkd systemctl enable --now systemd-networkd ``` Debugging: ```bash # execute with debugging in foreground SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd # Permanent by drop-in configuration mkdir /etc/systemd/system/systemd-networkd.service.d/ echo -e "[Service]\nEnvironment=SYSTEMD_LOG_LEVEL=debug" \ > /etc/systemd/system/systemd-networkd.service.d/10-debug.conf systemctl daemon-reload && systemctl restart systemd-networkd ``` ### Predictable interface names Prefix `en` Ethernet `wl` WLAN with following types: ```txt o<index> on-board device index number s<slot>[f<function>][d<dev_id>] hotplug slot index number x<MAC> MAC address p<bus>s<slot>[f<function>][d<dev_id>] PCI geographical location p<bus>s<slot>[f<function>][u<port>][..][c<config>][i<interface>] USB port number chain ``` Cf. `/lib/udev/rules.d/80-net-setup-link.rules`, cf. `systemd.link` ```bash udevadm info -e | grep -A 9 ^P.*en[sop] # dump udev database and grep for ethernet udevadm test /sys/class/net/* 2>&- | grep ID_NET_NAME_ ## -- .link file are used to rename an interface -- ## ls -l {/etc,/run,/lib}/systemd/network/*.link # list link configuration files ip -o -4 link # show link state ``` ### Unit File Configure the network, cf. `systemd.network` Skeleton for a **dynamic IP-address** ```ini [Match] Name= # device name (e.g en*) [Network] DHCP={yes,no,ipv4,ipv6} # enable DHCP ``` Skeleton for a **static IP-address** ```ini [Match] Name= # device name (e.g en*) [Network] Address= # IP address, CIDR notation Gateway= # IP address DNS= # is a DNS server address (multiples possibel) Domains= # a list of the domains used for DNS host name resolution ``` ### Wired & Wireless in Parallel ```bash cat <<EOF | sudo tee /etc/systemd/network/20-wired.network [Match] Name=en* [Network] DHCP=ipv4 [DHCP] RouteMetric=10 EOF cat <<EOF | sudo tee /etc/systemd/network/25-wireless.network [Match] Name=wl* [Network] DHCP=ipv4 [DHCP] RouteMetric=20 EOF ``` Note that the wireless interface needs to be connected by another service like `iwd`. `systemd-resolved` service is required if DNS entries are specified in `.network` files ## `systemd-resolved` Provides network name resolution... * ...supports DNSSEC and DNS over TLS * ...`glibc` NSS interface plugin `nss-resolve` ```bash systemctl enable --now systemd-resolved # start the service daemon systemd-resolve --status /etc/systemd/resolved.conf # conf. file /etc/systemd/resolved.conf.d/*.conf # drop-in conf. files ``` Fallback DNS addresses ensure DNS resolution in case no config is present Provides a **local DNS stub** listener on IP address 127.0.0.53 on the local loopback interface: ```bash rm /etc/resolv.conf # redirect glibc NSS conf. file to local stub DNS resolver ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf # operations mode detected automatically depending on existence of this link ``` ### Configuration Temporarily change the DNS server configuration: * `LINK` specifies the network interface, get the name from `resolvectl status` or `ip addr` * `SERVER` specifies the IP address of a DNS server i.e. `1.1.1.1` ```bash systemd-resolve -i LINK --set-dns=SERVER # or sudo resolvectl dns LINK SERVER ``` Configure a custom list of DNS resolvers, and enable DNSSEC ```bash mkdir /etc/systemd/resolved.conf.d # use a drop in .conf file for the Quad9 primary DNS resolvers cat > /etc/systemd/resolved.conf.d/dns.conf <<EOF [Resolve] DNS=9.9.9.9 149.112.112.112 EOF # if supported by the DNS resolvers, enforce DNSSEC validation cat > /etc/systemd/resolved.conf.d/dnssec.conf <<EOF [Resolve] DNSSEC=true EOF # if supported by the DNS resolver, attempt to use DNS over TLS cat > /etc/systemd/resolved.conf.d/dot.conf <<EOF [Resolve] DNSOverTLS=opportunistic EOF ``` systemd-resolved currently only supports `opportunistic` DNS over TLS resolution - Resolver tries resolution using DoT before fall back to traditional DNS (allowing for downgrade attacks) - Eventually another option will be added `strict` to prevent fallback DNS server certificates are not checked making systemd-resolved vulnerable to man-in-the-middle attacks ### `resolvectl` ```bash resolvectl status # check service status resolvectl query <ip|name> # query DNS records resolvectl statistics # show DNS cache stats ``` Enable debugging and follow the logs: ```bash resolvectl log-level debug journalctl -fu systemd-resolved # disable debugging resolvectl log-level info ``` ## `systemd-timesyncd` Types of clocks: * **RTC** (Real-Time Clock) aka hardware clock runs on battery when the machine is powered off (unplugged) - Uses _local time_ (in current time zone) - Or **UTC** (Coordinated Universal Time) with an time zone dependent offset * **SC** (Software Clock) aka system clock - Maintained by the OS kernel during run-time driven by a timer interrupt - Initialized on boot using RTC as reference. ```bash timedatectl # show time and time zone configuration timedatectl set-time YYYY-MM-DD # change the current date timedatectl set-time HH:MM:SS # change the current time timedatectl list-timezones # list available time zones timedatectl set-timezone <zone> # set a given time zone, e.g. Europe/Berlin grep ^Servers /etc/systemd/timesyncd.conf # list time servers timedatectl set-ntp true # enable NTP timedatectl set-local-rtc 0 # RTC in UTC mode timedatectl set-local-rtc 1 # RTC in locel time mode systemctl start systemd-timesyncd # start the time sync daemon systemctl enable systemd-timesyncd # make the time sync daemon boot persistant ``` Low-level administration tool ```bash locale # print locale settings hwclock # manipulate the hardware clock /etc/adjtime # state file for hwclock /usr/share/zoneinfo # time zone database ```

Usage

systemctl

cat, show & edit

localectl

systemd-nspawn

systemd-analyze