Log-file Management

Linux

Systemd

Published

June 18, 2024

Modified

July 30, 2024

Why collect log-files?

Troubleshooting
- …identify & diagnose operative issues
- …for example: connection issues, application errors, system crashes
- …helps to improve system configuration …optimize performance & efficiency
- …use errors & warnings to identify irregular behaviour
Reliability
- …insights into system activity (…helps for capacity planing)
- …identify recurring issues …enable proactive measures
Security …detect & alert potential security threats
- …for example unauthorized access attempts …suspicious activity
- …monitor unintentional configuration changes
- …monitor security updates …unpatched vulnerabilities
Compliance …regulatory requirements …industry standards

Troubleshooting

Analyze log-files to identify the root cause of a problem

What functionality is not working?
- …identify associated warning & error messages from the logs
- …extract messages with simple CLI tools like grep, less, tail
Correlate log files from different sources…
- …identify patterns and relationships between events
- …repeated errors & warning …consider time stamps

# continuously print logs
less +F /var/log/messages
more -f /var/log/messages
tail -f /var/log/messages

Common log files for troubleshooting

/var/log/{syslog,messages,kern.log}
- …system wide logs …hardware issues
/var/log/{apache,myswl,nginx} and other
- …service & application specific logs
- …typically user-facing …referenced by trouble tickets

Security

Common security-related logs:

/var/log/{syslog,messages}
- …system-wide logs including security-related event
- …system crashes, network connectivity, hardware error
- …very relevant to forensics
/var/log/{auth.log,security}
- …authentication successes and failures
- …changes to user accounts and permissions
/var/log/ssh remote authentication and login attempts
/var/log/audit.log

Monitoring logs is a critical component of a robust security strategy

…used with IDS (Intrusion Detection Systems)
- …detect anomalies …suspicious activity
- …identify patterns of behavior
- …response (for example blocking traffic from a suspicious IPs)
…enable real-time threat detection and response

Regulations …laws, standards, and guidelines to…

…ensure confidentiality, integrity, availability of sensitive data
…common measures to protect sensitive information from
- …unauthorized access and use
- …disclosure and modification (espionage)
…encryption (ransom-ware) or destruction

Logging Types

Types of logging mechanisms:

Kernel …errors and warning from the Linux kernel
- …via a kernel ring buffer (…available beginning of boot)
- …use dmesg to print kernel-logs
User …processes services in user-space
- …typically based on the Syslog protocol

sudo dmesg
sudo dmesg –facility=user
sudo dmesg | grep -i -E 'error|warn|failed'
sudo dmesg | less

Methods how a service can write a log file:

…directly into a dedicated log-file
syslog daemon …forward messages to a log-service
journald …Systemd integrated log-service

Usage

multitail ¹ displays multiple log files simulations in terminal
logwatch ² …identify log-patterns and automate response

Compressed Logs

Reading compressed logs…

zcat …cat to view compressed file
zgrep …grep to search inside the compressed files
zless …pager for compressed files
zdiff …comparison of compressed files

Create Messages

Send a log message:

logger …simple log client
systemd-cat …send messages to the systemd journal
wall …send messages to all logged-in users

# send an emergency message from the auth facility 
logger -p auth.emerg "Somebody tried to connect to the system"

# send message to journal via stdin
echo "This is a message to journald" | systemd-cat

# send the output of the `ls` command to the journal
systemd-cat ls -l

Syslog

Standard to format and transmit log messages

Massages contain following elements:
- Priority …number indicating severity of the message
- Header …timestamp, hostname, and process ID
- Message …actual log message

Syslog daemons …collect, process, and forward log messages

syslogd (1980)
syslog-ng (1998)
- https://www.syslog-ng.com
- https://github.com/syslog-ng/syslog-ng
rsyslog (2004)
- https://www.rsyslog.com
- https://github.com/rsyslog/rsyslog
systemd-journald …incorporates syslog-like functionality

Architecture

Client-server architecture …terminology:

Originator …aka syslog-client …sends messages over network
Relay …forward messages over the network …can transform messages
Collector …syslog-server …stores logs from clients

Logging on a local Linux system…

Applications & services (Originators)…
- …embed syslog or journald librariessend
- …send logs to local collector
Typically no rely on localhost
Collectors rsyslog and the journald daemon

Levels

Value	Severity	Keyword
0	Emergency	`emerg`
1	Alert	`alert`
2	Critical	`crit`
3	Error	`err`
4	Warning	`warning`
5	Notic	`notice`
6	Informational	`info`
7	Debug	`debug`

`systemd-journald`

Data stored to /run/systemd/journal/ & /var/log/journal

Systemd service that provides…

Structure
- …enforced structure for indexed log files (called journals)
- …simplifies filtering (for example by priority, time-frame)
Indexing
- …uses a binary storage for logs
- …stores journal in a secure manner
- …lookups are faster then plain text
Access control
- …storage files are split by user (with correct permissions)
- …users have access to their logs
Automatic log rotation
- …journal files are automatically rotated if they grow above certain limits

Usage

journalctl used to read the journal binary log files

…all entries, from oldest to newest
…uses a pager …defaults to less …or --no-pager …or SYSTEMD_PAGER=cat

Most common usage:

# …continually print new messages added to log …aka follow
journalctl -f 

journalctl -k                         # kernel ring buffer
journalctl -f [...]                   # tail the log file
journalctl -o verbose -n
journalctl -f -l SYSLOG_FACILITY=10
journalctl --vacuum-time=2weeks       # clean journal files
journalctl -p err..alert              # priority

# …by time constrains
journalctl --since=yesterday
journalctl --since=00:00 --until=9:30
journalctl --since -30min

Journal Fields

Filter messages by journal fields

…details in man systemd.journal-fields
…option -F used to show available values for a given journal field

# …list for which user IDs logs exist
journalctl -F _UID | xargs id ${}

# …by specific user or group
journalctl _UID=1234
journalctl _GID=4321

# …by process ID or executable path
journalctl _PID=123456
journalctl _EXE=/bin/bash

System Units

Messages for the specified systemd unit…

-u, --unit=UNIT|PATTERN
- …pattern matches unit names found in the journal
- …match pattern that includes UNIT=name.service

# list available units…
journalctl -F UNIT | sort | less

# …by unit
journalctl -u nginx.service
journalctl -u sssd.service --since today
journalctl -u ldap.service -u sssd.service

# …match all systemd units
journalctl -u 'systemd*'

# …or
journalctl _SYSTEMD_UNIT=sshd.service

Configuration

Default configuration in /etc/systemd/journald.conf

# …after changes to the configuration
systemctl reload systemd-journald.service

Persistent Storage

Read logs from previous boots…

# limit messages to recent boot
journalctl -b

# List the boot IDs
journalctl --list-boot # …includes time-frame of log accumulation

# …used for reference with option `-b` 
journalctl -b -1

# storage used for logs
journalctl --disk-usage

# shrink your journal by indicating a size
journalctl --vacuum-size=1G

# …keep entries from the last year
journalctl --vacuum-time=1years

Different types of storage…

volatile …stores only in memory …below /run/log/journal
persistant …stores on local device …below /var/log/journal
none …turns off all storage

/etc/systemd/journald.conf.d/storage.conf

[Journal]
Storage=persistent

Enable this change without reboot…

systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald

Specifies the maximum disk space that can be used by the journal

/etc/systemd/journald.conf.d/limits.conf

[Journal]
SystemMaxUse=5G

Forward Messages

Forward messages to a central log server with syslog-ng

/etc/systemd/journald.conf.d/forward.conf

[Journal]
ForwardToSyslog=yes

Centralized Logging

dnf install -y systemd-journal-remote

Multiple ways of centralizing journald logs…

…forwards entries to a local syslog.socket
…rsyslog daemon reads from the local journal
systemd-journal-remote.service
- …to accumulate logs on a single server
- …work in “pull” or “push” mode
systemd-journal-upload.service
- …send logs to a collection service
- …for example Elasticsearch
systemd-journal-gatewayd.service
- …serves journal events over the network
- …client connect using HTTP …port 19531 (by default)

`systemd-journal-gatewayd`

HTTP server for journal events…

systemctl enable --now systemd-journal-gatewayd

# check connection
curl http://localhost:19531/entries?boot

Read from systemd-journal-gatewayd…

# pull journal from another node
export PATH=$PATH:/usr/lib/systemd
systemd-journal-remote --output=/tmp/node.journal --split-mode=none \
                       --url=http://$node:19531

# …read the journal
journalctl --file /tmp/node.journal

`systemd-journal-remote`

Receive journal messages over the network…

…“active” …requests and pulls the data
- …can read more than one stream at a time
- …output interleaved …each source one stream
…“passive” …waits for a connection …receives massages pushed

Configuration default and drop-ins:

/etc/systemd/journal-remote.conf 
/etc/systemd/journal-remote.conf.d/*.conf

Certificate configuration…

/etc/ssl/private/journal-remote.pem …key file
/etc/ssl/certs/journal-remote.pem …certificate file
/etc/ssl/ca/trusted.pem …CA certificate file

systemctl enable --now systemd-journal-remote.socket systemd-journal-remote.service

Log Rotation

Automated process to archive and compress log-files…

…renames existing log files appending a timestamp
…creates a new log-file to the application
…compress rotated log files to save disk user-space
…deletes old log-files according to a rule

Typically run as a cron job …configuration in:

/etc/logrotate.conf
/etc/logrotate.d/

Rsyslog

Rsyslog ³ …logging and event processing tool-set …highly customizable in the details of this message flow

Messages enter rsyslog with the help of input modules ⁴…
…passed to a set of conditionally applied rules ⁵…
…when rules match message is transferred to an action (outputs)
…action write to a file, database or forwards it to a remote host

# install documentation
dnf install rsyslog-doc
$BROWSER /usr/share/doc/rsyslog/html/index.html

/etc/rsyslog.conf default configuration…

…consists of statements …directives processed from the top until match
…recommended to use RainerScript type statements when
- …configuration parameters are required …for example actions
- more elaborate control-of-flow is required …nesting
Extend configuration in /etc/rsyslog.d/ …included by default
rsyslogd -N 1 …to validate syntax of configuration files

Filters

A rule consists of a filter ⁶ and an action list …filters provide yes/no decisions and thus control-of-flow capability

Facility/priority-based filters:

…<facility>.<priority> (facility and priority separated by a dot)
…facility …subsystem that produces a specific syslog message
…priority (or higher) …priority of a syslog message
- …preceding = only messages with specified priority will be selected
- …preceding ! selects messages except those with the defined priority
- none …for facilities with no given priorities
* …define all facilities or priorities
, …define multiple facilities and priorities
; …combine multiple selectors

kern.*              # all kernel messages
mail.crit           # all critical mail syslog messages
cron.!info,!debug   # cron syslog messages except info or debug priority
*.err;mail.debug    # all error message, and debug from mail

Remote Logs

Example:

cat > /etc/rsyslog.d/10-remotelog.conf <<EOF
auth,authpriv.* @$SERVER_IP_ADDRESS:514
EOF
# syntax check file
rsyslogd -N 1 -f /etc/rsyslog.d/10-remotelog.conf
systemctl restart rsyslog && systemctl status rsyslog

Messages can be sent using @ UDP (default) or @@ TCP

auth.*      @$SERVER:$PORT            # UDP
*.err       @@$SERVER:$PORT           # TCP

More advanced configuration with the omfwd ⁷ module:

auth.* action(type="omfwd"
           action.resumeRetryCount="-1"
           target="example.com" port="514" protocol="tcp")

References

[M23PT] Guide to Computer Security Log Management, NIST
https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-92.pdf

Footnotes

MultiTail
https://www.vanheusden.com/multitail
https://github.com/folkertvanheusden/multitail ↩︎
Logwatch
https://github.com/gjalves/logwatch ↩︎
Rsyslog Documentation
https://www.rsyslog.com/doc
https://wiki.gentoo.org/wiki/Rsyslog ↩︎
Modules, Rsyslog Documentation
https://rsyslog.readthedocs.io/en/latest/configuration/modules/↩︎
Rulesets in rsyslog, Rsyslog Documentation
https://rsyslog.readthedocs.io/en/latest/concepts/multi_ruleset.html ↩︎
Filter Conditions, Rsyslog Documentation
https://rsyslog.readthedocs.io/en/latest/configuration/filters.html ↩︎
omfwd Forwarding Output Module
https://www.rsyslog.com/doc/configuration/modules/omfwd.html ↩︎

--- title: Log-file Management categories: - Linux - Systemd date: 2024/06/18 date-modified: 2024/07/30 toc-depth: 4 toc-expand: 4 --- ## Why collect log-files? - **Troubleshooting** - …identify & diagnose operative issues - …for example: connection issues, application errors, system crashes - …helps to improve system configuration …optimize performance & efficiency - …use errors & warnings to identify irregular behaviour - **Reliability** - …insights into system activity (…helps for capacity planing) - …identify recurring issues …enable proactive measures - **Security** …detect & alert potential security threats - …for example unauthorized access attempts …suspicious activity - …monitor unintentional configuration changes - …monitor security updates …unpatched vulnerabilities - **Compliance** …regulatory requirements …industry standards ### Troubleshooting Analyze log-files to identify the root cause of a problem - What functionality is not working? - …identify associated warning & error messages from the logs - …extract messages with simple CLI tools like `grep`, `less`, `tail` - Correlate log files from different sources… - …identify patterns and relationships between events - …repeated errors & warning …consider time stamps ```bash # continuously print logs less +F /var/log/messages more -f /var/log/messages tail -f /var/log/messages ``` Common log files for troubleshooting - `/var/log/{syslog,messages,kern.log}` - …system wide logs …hardware issues - `/var/log/{apache,myswl,nginx}` and other - …service & application specific logs - …typically user-facing …referenced by trouble tickets ### Security Common security-related logs: - `/var/log/{syslog,messages}` - …system-wide logs including security-related event - …system crashes, network connectivity, hardware error - …very relevant to forensics - `/var/log/{auth.log,security}` - …authentication successes and failures - …changes to user accounts and permissions - `/var/log/ssh` remote authentication and login attempts - `/var/log/audit.log` Monitoring logs is a critical component of a robust security strategy - …used with **IDS** (Intrusion Detection Systems) - …detect anomalies …suspicious activity - …identify patterns of behavior - …response (for example blocking traffic from a suspicious IPs) - …enable real-time threat detection and response Regulations …laws, standards, and guidelines to… - …ensure confidentiality, integrity, availability of sensitive data - …common measures to protect sensitive information from - …unauthorized access and use - …disclosure and modification (espionage) - …encryption (ransom-ware) or destruction ## Logging Types Types of logging mechanisms: - **Kernel** …errors and warning from the Linux kernel - …via a kernel ring buffer (…available beginning of boot) - …use `dmesg` to print kernel-logs - **User** …processes services in user-space - …typically based on the Syslog protocol ```bash sudo dmesg sudo dmesg –facility=user sudo dmesg | grep -i -E 'error|warn|failed' sudo dmesg | less ``` Methods how a service can write a log file: - …directly into a dedicated log-file - `syslog` daemon …forward messages to a log-service - `journald` …Systemd integrated log-service ## Usage - `multitail` [^HhkUe] displays multiple log files simulations in terminal - `logwatch` [^QDGVr] …identify log-patterns and automate response [^HhkUe]: MultiTail <https://www.vanheusden.com/multitail> <https://github.com/folkertvanheusden/multitail> [^QDGVr]: Logwatch <https://github.com/gjalves/logwatch> ### Compressed Logs Reading compressed logs… - `zcat` …cat to view compressed file - `zgrep` …grep to search inside the compressed files - `zless` …pager for compressed files - `zdiff` …comparison of compressed files ### Create Messages Send a log message: - `logger` …simple log client - `systemd-cat` …send messages to the systemd journal - `wall` …send messages to all logged-in users ```bash # send an emergency message from the auth facility logger -p auth.emerg "Somebody tried to connect to the system" # send message to journal via stdin echo "This is a message to journald" | systemd-cat # send the output of the `ls` command to the journal systemd-cat ls -l ``` ## Syslog Standard to format and transmit log messages - Massages contain following elements: - **Priority** …number indicating severity of the message - **Header** …timestamp, hostname, and process ID - **Message** …actual log message Syslog daemons …collect, process, and forward log messages - `syslogd` (1980) - `syslog-ng` (1998) - <https://www.syslog-ng.com> - <https://github.com/syslog-ng/syslog-ng> - `rsyslog` (2004) - <https://www.rsyslog.com> - <https://github.com/rsyslog/rsyslog> - `systemd-journald` …incorporates syslog-like functionality ### Architecture Client-server architecture …terminology: - **Originator** …aka `syslog`-client …sends messages over network - **Relay** …forward messages over the network …can transform messages - **Collector** …`syslog`-server …stores logs from clients Logging on a local Linux system… - Applications & services (Originators)… - …embed syslog or journald librariessend - …send logs to local collector - Typically no rely on localhost - Collectors `rsyslog` and the `journald` daemon ### Levels Value | Severity | Keyword ------|---------------|-------- 0 | Emergency | `emerg` 1 | Alert | `alert` 2 | Critical | `crit` 3 | Error | `err` 4 | Warning | `warning` 5 | Notic | `notice` 6 | Informational | `info` 7 | Debug | `debug` ## `systemd-journald` **Data stored to `/run/systemd/journal/` & `/var/log/journal`** Systemd service that provides… - **Structure** - …enforced structure for indexed log files (called journals) - …simplifies filtering (for example by priority, time-frame) - **Indexing** - …uses a binary storage for logs - …stores journal in a secure manner - …lookups are faster then plain text - **Access control** - …storage files are split by user (with correct permissions) - …users have access to their logs - **Automatic log rotation** - …journal files are automatically rotated if they grow above certain limits ### Usage **`journalctl` used to read the journal binary log files** - …all entries, from oldest to newest - …uses a pager …defaults to `less` …or `--no-pager` …or `SYSTEMD_PAGER=cat ` Most common usage: ```bash # …continually print new messages added to log …aka follow journalctl -f journalctl -k # kernel ring buffer journalctl -f [...] # tail the log file journalctl -o verbose -n journalctl -f -l SYSLOG_FACILITY=10 journalctl --vacuum-time=2weeks # clean journal files journalctl -p err..alert # priority ``` ```bash # …by time constrains journalctl --since=yesterday journalctl --since=00:00 --until=9:30 journalctl --since -30min ``` #### Journal Fields Filter messages by **journal fields** - …details in `man systemd.journal-fields` - …option `-F` used to show available values for a given journal field ```bash # …list for which user IDs logs exist journalctl -F _UID | xargs id ${} # …by specific user or group journalctl _UID=1234 journalctl _GID=4321 # …by process ID or executable path journalctl _PID=123456 journalctl _EXE=/bin/bash ``` #### System Units Messages for the specified systemd unit… - `-u, --unit=UNIT|PATTERN` - …pattern matches unit names found in the journal - …match pattern that includes `UNIT=name.service` ```bash # list available units… journalctl -F UNIT | sort | less # …by unit journalctl -u nginx.service journalctl -u sssd.service --since today journalctl -u ldap.service -u sssd.service # …match all systemd units journalctl -u 'systemd*' # …or journalctl _SYSTEMD_UNIT=sshd.service ``` ### Configuration **Default configuration in `/etc/systemd/journald.conf`** ```bash # …after changes to the configuration systemctl reload systemd-journald.service ``` #### Persistent Storage Read logs from previous boots… ```bash # limit messages to recent boot journalctl -b # List the boot IDs journalctl --list-boot # …includes time-frame of log accumulation # …used for reference with option `-b` journalctl -b -1 # storage used for logs journalctl --disk-usage # shrink your journal by indicating a size journalctl --vacuum-size=1G # …keep entries from the last year journalctl --vacuum-time=1years ``` Different types of storage… - `volatile` …stores only in memory …below `/run/log/journal` - `persistant` …stores on local device …below `/var/log/journal` - `none` …turns off all storage ```{.ini filename=/etc/systemd/journald.conf.d/storage.conf} [Journal] Storage=persistent ``` Enable this change without reboot... ```bash systemd-tmpfiles --create --prefix /var/log/journal systemctl restart systemd-journald ``` Specifies the maximum disk space that can be used by the journal ```{.ini filename=/etc/systemd/journald.conf.d/limits.conf} [Journal] SystemMaxUse=5G ``` #### Forward Messages Forward messages to a central log server with `syslog-ng` ```{.ini filename=/etc/systemd/journald.conf.d/forward.conf } [Journal] ForwardToSyslog=yes ``` ### Centralized Logging ```bash dnf install -y systemd-journal-remote ``` Multiple ways of centralizing `journald` logs… - …forwards entries to a local `syslog.socket` - …`rsyslog` daemon reads from the local journal - `systemd-journal-remote.service` - …to accumulate logs on a single server - …work in “pull” or “push” mode - `systemd-journal-upload.service` - …send logs to a collection service - …for example Elasticsearch - `systemd-journal-gatewayd.service` - …serves journal events over the network - …client connect using HTTP …port 19531 (by default) #### `systemd-journal-gatewayd` HTTP server for journal events… ```bash systemctl enable --now systemd-journal-gatewayd # check connection curl http://localhost:19531/entries?boot ``` Read from `systemd-journal-gatewayd`… ```bash # pull journal from another node export PATH=$PATH:/usr/lib/systemd systemd-journal-remote --output=/tmp/node.journal --split-mode=none \ --url=http://$node:19531 # …read the journal journalctl --file /tmp/node.journal ``` #### `systemd-journal-remote` Receive journal messages over the network… - …"active" …requests and pulls the data - …can read more than one stream at a time - …output interleaved …each source one stream - …"passive" …waits for a connection …receives massages pushed Configuration default and drop-ins: ```bash /etc/systemd/journal-remote.conf /etc/systemd/journal-remote.conf.d/*.conf ``` Certificate configuration… - `/etc/ssl/private/journal-remote.pem` …key file - `/etc/ssl/certs/journal-remote.pem` …certificate file - `/etc/ssl/ca/trusted.pem` …CA certificate file ```bash systemctl enable --now systemd-journal-remote.socket systemd-journal-remote.service ``` ## Log Rotation Automated process to archive and compress log-files… - …renames existing log files appending a timestamp - …creates a new log-file to the application - …compress rotated log files to save disk user-space - …deletes old log-files according to a rule Typically run as a cron job …configuration in: ```bash /etc/logrotate.conf /etc/logrotate.d/ ``` ## Rsyslog Rsyslog [^kr03a] …logging and event processing tool-set …highly customizable in the details of this message flow [^kr03a]: Rsyslog Documentation <https://www.rsyslog.com/doc> <https://wiki.gentoo.org/wiki/Rsyslog> - Messages enter rsyslog with the help of **input modules** [^Tu0DF]… - …passed to a set of conditionally applied **rules** [^OKsE8]… - …when rules match message is transferred to an **action** (outputs) - …action write to a file, database or forwards it to a remote host [^Tu0DF]: Modules, Rsyslog Documentation <https://rsyslog.readthedocs.io/en/latest/configuration/modules/> [^OKsE8]: Rulesets in rsyslog, Rsyslog Documentation <https://rsyslog.readthedocs.io/en/latest/concepts/multi_ruleset.html> [^mtPxy]: Filter Conditions, Rsyslog Documentation <https://rsyslog.readthedocs.io/en/latest/configuration/filters.html> ```bash # install documentation dnf install rsyslog-doc $BROWSER /usr/share/doc/rsyslog/html/index.html ``` **`/etc/rsyslog.conf` default configuration**… - …consists of statements …directives processed from the top until match - …recommended to use RainerScript type statements when - …configuration parameters are required …for example actions - more elaborate control-of-flow is required …nesting - **Extend configuration in `/etc/rsyslog.d/`** …included by default - `rsyslogd -N 1` …to validate syntax of configuration files ### Filters A rule consists of a **filter** [^mtPxy] and an action list …filters provide yes/no decisions and thus control-of-flow capability Facility/priority-based filters: - …`<facility>.<priority>` (_facility_ and _priority_ separated by a dot) - …facility …subsystem that produces a specific syslog message - …priority (or higher) …priority of a syslog message - …preceding `=` only messages with specified priority will be selected - …preceding `!` selects messages except those with the defined priority - `none` …for facilities with no given priorities - `*` …define all facilities or priorities - `,` …define multiple facilities and priorities - `;` …combine multiple selectors ```bash kern.* # all kernel messages mail.crit # all critical mail syslog messages cron.!info,!debug # cron syslog messages except info or debug priority *.err;mail.debug # all error message, and debug from mail ``` ### Remote Logs Example: ```bash cat > /etc/rsyslog.d/10-remotelog.conf <<EOF auth,authpriv.* @$SERVER_IP_ADDRESS:514 EOF # syntax check file rsyslogd -N 1 -f /etc/rsyslog.d/10-remotelog.conf systemctl restart rsyslog && systemctl status rsyslog ``` Messages can be sent using `@` UDP (default) or `@@` TCP ```bash auth.* @$SERVER:$PORT # UDP *.err @@$SERVER:$PORT # TCP ``` More advanced configuration with the `omfwd` [^HYLIs] module: [^HYLIs]: `omfwd` Forwarding Output Module <https://www.rsyslog.com/doc/configuration/modules/omfwd.html> ```ruby auth.* action(type="omfwd" action.resumeRetryCount="-1" target="example.com" port="514" protocol="tcp") ``` ## References [M23PT] Guide to Computer Security Log Management, NIST <https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-92.pdf>