Netdata Serverless Monitoring on Localhost

Monitor
Containers
Published

January 19, 2016

Modified

May 19, 2023

Noteworthy features…

References…

Installation

After installation access the agent dashboard at http://localhost:19999

# deployment on Fedora
sudo dnf -y install netdata # lm_sensors
sudo systemctl enable --now netdata.service

Netdata Docker Container…

# using Podman as container engine
podman run -d --name=netdata \
  -p 19999:19999 \
  -v netdataconfig:/etc/netdata \
  -v netdatalib:/var/lib/netdata \
  -v netdatacache:/var/cache/netdata \
  -v /etc/passwd:/host/etc/passwd:ro \
  -v /etc/group:/host/etc/group:ro \
  -v /proc:/host/proc:ro \
  -v /sys:/host/sys:ro \
  -v /etc/os-release:/host/etc/os-release:ro \
  --restart unless-stopped \
  --cap-add SYS_PTRACE \
  --security-opt apparmor=unconfined \
  docker.io/netdata/netdata

Configuration

  • In /etc/netdata/ by default…
    • netdata.conf main configuration
    • conf.d/**./*.conf stock configuration (do not modify)
  • Documentation

edit-config is a shell script used for editing configuration files.

# on Fedora (uses $EDITOR)
sudo -E /usr/libexec/netdata/edit-config netdata.conf
sudo systemctl restart netdata.service
# read the configuration from the local netdata daemon...
curl localhost:19999/netdata.conf | less

Start the netdata daemon for debugging

sudo systemctl stop netdata.service
# start in fore-ground...
sudo netdata -D
# Ctrl-C to exit

Log Files

# list errors executing `netdata`...
grep ERROR /var/log/netdata/error.log

Log files…

  • error.log
    • …stderr of the netdata daemon
    • Labels…
      • ERROR…might disable a part of netdata, auto-detection of data collection fail
      • FATAL…prevents a program from running
  • access.log…logs web requests
  • debug.log…if debugging/tracing is also enabled

Anonymous Statistics

Collects anonymous usage information…

# opt-out from sending anonymous statistics
sudo touch /etc/netdata/.opt-out-from-anonymous-statistics

Netdata Anonymous statistics Opt-Out
https://learn.netdata.cloud/docs/agent/anonymous-statistics#opt-out

Access Control

Allows access to the dashboard from local networks…

allow dashboard from = localhost 10.* 192.168.*
# bind to = *

Check the port state with ss…

ss -tupanl | grep 19999

Data Collection & Retention

  • Recently-collected metrics are stored in RAM…
  • …compressed and “spilled” to disk for long-term storage
    • page cache size RAM (in MiB) the database engine uses to cache and index recent metrics
    • dbengine multihost disk space maximum disk space (again, in MiB) the database engine uses to store historical, compressed metrics
# netdata.conf
[global]
    page cache size = 32
    dbengine multihost disk space = 256

Netdata Long-term Metrics Storage
https://learn.netdata.cloud/docs/store/change-metrics-storage
https://learn.netdata.cloud/guides/step-by-step/step-09

Optimize Netdata Performance
https://learn.netdata.cloud/guides/configure/performance

Collectors

Catch-all term for a process that gathers metrics from an endpoint

Plugins…

  • Internal plugins written in C…threads within the Netdata daemon
  • External plugins…independent processes…
    • …communicate via a pipe with the Netdata daemon
    • Orchestrator…
      • …written in Bash, Python or Go
      • …support additional collector modules
  • …/usr/libexec/netdata
  • …by default most plugins are enabled

Configuration…

Battery Monitoring

Power supply configuration in netdata.conf

https://learn.netdata.cloud/docs/agent/collectors/proc.plugin#linux-power-supply

[plugin:proc:/sys/class/power_supply]
  battery capacity = yes
  battery charge = no
  battery energy = no
  power supply voltage = yes
  # keep files open = auto
  # directory to monitor = /sys/class/power_supply

LM Sensors

# enabled by default
grep sensors /etc/netdata/conf.d/python.d.conf

FreeIPMI

Plugin to monitor IPMI sensors and events…

# get help text
/usr/libexec/netdata/plugins.d/freeipmi.plugin -h
# run the plugin in debug mode
sudo su -s /bin/sh netdata
/usr/libexec/netdata/plugins.d/freeipmi.plugin 5 debug

netdata.conf

[plugins]
    freeipmi = yes

[plugin:freeipmi]
    update every = 5                     # update interval in seconds
    command options = ignore 1,2,3,4,... # ignoring specific sensors

SMART Attributes

Configure smartd.service…-A option to write smartd attribute information to files

# /etc/sysconfig/smartmontools 
smartd_opts="-A /var/log/smartd/ -i 600"

Add log rotation and restart…

cat > /etc/logrotate.d/smartd <<EOF
/var/log/smartd/*.csv {
    monthly
    rotate 12
    missingok
    notifempty
    sharedscripts
    postrotate
        killall -HUP smartd
    endscript
}
EOF
mkdir -p /var/log/smartd && systemctl restart smartd

Plugins

Plugin orchestrators run most collectors…

  • go.d.plugin…data collection modules written in go
  • python.d.plugin…data collection modules written in python v2/v3
  • charts.d.plugin…data collection modules written in bash v4+

charts.d.plugin

Data collection modules written in Bash…

  • /etc/netdata/charts.d.conf…orchestrator configuration file
  • …looks for the plugin scripts in…
    • /usr/lib/netdata/charts.d by default
    • /usr/libexec/netdata/charts.d with netdata*.rpm packages
  • …scripts should have the filename suffix: .chart.sh
  • …module configuration files…
    • …in directory /etc/netdata/charts.d/
    • …same name as the script with suffix .conf

Implementation…

Run the example.chart.sh plugin…

# edit the configuration file...
# ...make sure example=force is uncommented
sudo -E /usr/libexec/netdata/edit-config charts.d.conf
# create a simple configuration for the example plugin
cat << EOF | sudo tee -a /etc/netdata/charts.d/example.conf
example_magic_number=12345
EOF
# become user netdata...run the example plugin in debug mode
sudo su -s /bin/sh netdata -c '
      /usr/libexec/netdata/plugins.d/charts.d.plugin debug 1 example
'

upower.charts.sh

Add script demonstrates the use upower to monitor the energy rate on the battery

  • …path to script /usr/libexec/netdata/charts.d/upower.chart.sh
  • …add upower=yes to /etc/netdata/charts.d.conf
  • …restart `netdata.service
# _update_every is a special variable - it holds the number of seconds
# between the calls of the _update() function
upower_update_every=

# the priority is used to sort the charts on the dashboard
# 1 = the first chart
upower_priority=150000

# global variables to store our collected data
upower_energy_rate= 

# path to the battery configuration
upower_battery=

upower_get() {

        upower_energy_rate=$(
                upower -i $upower_battery \
                        | grep 'energy-rate:' \
                        | tr -s ' ' \
                        | cut -d: -f2 \
                        | cut -d' ' -f2
        )
        
        # this should return:
        #  - 0 to send the data to netdata
        #  - 1 to report a failure to collect the data

        return 0
}

# returns 0 or 1 depending on whether the module is able to run or not
upower_check() {
        # check if the upower command is installed
        require_cmd upower || return 1
        # identify the path to the battery configuration
        upower_battery=$(upower -e | grep 'BAT')
        return 0
}

# _create is called once, to create the charts
upower_create() {

        cat << EOF
CHART upower.energy_rate '' "Rate" "Watt" 'energy' upower line $upower_priority $upower_update_every '' '' 'Upower'
DIMENSION energy_rate 'Energy Rate' absolute 1 1
EOF

        return 0
}

# _update is called continuously, to collect the values
upower_update() {

        # the first argument to this function is the microseconds since last update
        # pass this parameter to the BEGIN statement (see below).
        upower_get || return 1

        # write the result of the work.
        cat << VALUESEOF
BEGIN upower.energy_rate $1
SET energy_rate = $upower_energy_rate 
END
VALUESEOF

        return 0
}

Testing & Development

Following examples use vagrant for testing and debugging…

Vagrant + Package

Install Netdata from Fedora EPEL repository…

cd $(mktemp -d)
cat > Vagrantfile <<EOF
Vagrant.configure("2") do |config|
  config.vm.define "netdata" 
  config.vm.box = "almalinux/8"
  config.vm.provider :libvirt do |libvirt|
    libvirt.memory = 2048
  end
  config.vm.provision "shell" do |s|
    s.inline = %q[
      dnf install -y epel-release
      dnf install -y netdata
      systemctl enable --now netdata.service
    ]
    s.privileged = true
  end
  config.vm.network "forwarded_port", guest: 19999, host: 29999
end
EOF
vagrant up
$BROWSER http://localhost:29999

Vagrant + Container

Install Netdata from a container…

cd $(mktemp -d)
cat > Vagrantfile <<EOF
Vagrant.configure("2") do |config|
  config.vm.define "netdata-container" 
  config.vm.box = "fedora/36-cloud-base"
  config.vm.provider :libvirt do |libvirt|
    libvirt.memory = 2048
  end
  config.vm.provision "shell" do |s|
    s.inline = "dnf install -y podman"
    s.privileged = true
  end
  config.vm.network "forwarded_port", guest: 19999, host: 29999
end
EOF
vagrant up
# deploy the Netdata container...
vagrant ssh -- \
  podman run -d --name=netdata \
    -p 19999:19999 \
    -v netdataconfig:/etc/netdata \
    -v netdatalib:/var/lib/netdata \
    -v netdatacache:/var/cache/netdata \
    -v /etc/passwd:/host/etc/passwd:ro \
    -v /etc/group:/host/etc/group:ro \
    -v /proc:/host/proc:ro \
    -v /sys:/host/sys:ro \
    -v /etc/os-release:/host/etc/os-release:ro \
    --restart unless-stopped \
    --cap-add SYS_PTRACE \
    --security-opt apparmor=unconfined \
    docker.io/netdata/netdata
# open the web-interface with the default web-browser
$BROWSER http://localhost:29999

Alter the Netdata configuration in the container…

# login to the virtual machine
vagrant ssh
# get the current configuration..
curl localhost:19999/netdata.conf -o netdata.conf
# ...modify the configuration according to your needs...
# ...copy the configuration into the container
podman cp netdata.conf netdata:/etc/netdata
# restart the service
podman restart netdata