Grafana Monitoring Dashboard

Monitor

Published

January 19, 2016

Modified

May 3, 2024

Grafan ¹ is an open source dashboard to present monitoring metrics:

Multiple time series data storage solutions (e.g. InfluxDB, Promehteus, OpenTSDB) can be used in combination by connecting them as data sources.
Organizations provide a method for multi-tenancy, and group users, data-sources, and the interfaces into distinct entities.
Users have accounts with different levels of privileges, and can authenticate with internal and external (LDAP, OAuth, GitHub, Google) methods.
Dashboards are build by users to visualize monitoring data in a customized way. Dashboard are a combination of panels showing graphs, tables and other data based on configurable queries on the data sources.
The basic visualization block are panels. They provide a query editor to source the data, and various styling and formatting options to model the data visualization.
The dashboard layouts use rows as logical dividers to arrange panels on the view horizontally and vertically.

Installation

# Open source Grafana
podman run -d -p 3000:3000 --name=grafana \
           -v grafana-storage:/var/lib/grafana \
           grafana/grafana-oss

Install Grafana using the official container container:

Defaults to HTTP port 3000, with default user/password admin.
Grafana uses an embedded SQLite version 3 database to store configuration…

Environment Variables

Set an environment variable to run in debug mode:

podman run -d -p 3000:3000 --name=grafana \
           -v grafana-storage:/var/lib/grafana \
           -e "GF_LOG_LEVEL=debug" \
           grafana/grafana-oss

…override existing options …naming convention GF_<SectionName>_<KeyName>
Reference the grafana.ini…
- …section name text within the brackets
- …all text uppercase …replace .- with underscore

Configuration File

Use a configuration file…

mkdir /etc/grafana /var/lib/grafana

# …dummy configuration file example
cat > /etc/grafana/grafana.ini <<EOF
# default section
instance_name = my-instance

[security]
admin_user = admin
admin_password = secret123 

[paths]
data = /srv/grafana
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning
EOF

# …mount the configuration into the container
podman run -d -p 3000:3000 --name=grafana \
           -v /etc/grafana/:/etc/grafana:ro \
           -v /srv/grafana:/srv/grafana:rw,U,Z
           -v /var/lib/grafana:/var/lib/grafana:rw,U,Z \
           grafana/grafana-oss:10.2.6

Systemd Unit

Create a systemd service unit for the container:

cd /etc/systemd/system

# …generate the service unit file …name is prefixed with `container-`
podman generate systemd --files --name grafana

# …make sure to start Grafana on boot
systemctl enable --now container-grafana.service

Configuration

File	Description
`/etc/grafana/grafana.ini`	Local custom configuration file

Configure Grafana ² using a custom configuration file:

…default settings in $WORKING_DIR/conf/defaults.ini
…default path overwritten with option --config
…; semicolons for comments
Structured into sections …for example [server] …notable sections:
- paths …data store, log file & provisioning
- server …ports, certificates, HTTPS, routes
- database …for users, dashboards, etc
- security …admins login, HTTP security, etc
- users …sign-up, themes, language, etc
- auth.* …external identity providers
- log.* …all about access/error logs
- …and more

Provisioning

Provisioning system that uses configuration files ³:

Configure where to find the provisioning directory with paths.provisioning
Add YAML configuration files to …updated during (re-)start

Assumes following directory structure:

Path	Description
`provisioning/datasources/`	…each file can contain a list of `datasources` ⁴
`provisioning/plugins/`	…manage plugin applications
`provisioning/dashboards/`	…manage dashboards
`provisioning/notifiers`	…alerting resources

Data Sources

Create a data source manifest:

mkdir -p /etc/grafana/provisioning/datasources
cat > /etc/grafana/provisioning/datasources/promehteus.yml <<EOF
datasources:
- name: prometheus
  type: prometheus
  url: http://localhost:9090
EOF

Dashboards

Create a dashboard manifest ⁵ & dashboard definition ⁶:

mkdir -p /etc/grafana/provisioning/dashboards
cat > /etc/grafana/provisioning/dashboards/node-exporter.yml <<EOF
apiVersion: 1
providers:
  - name: Node-Exporter
    folder: Node-Exporter
    type: file
    disableDeletion: false
    allowUiUpdates: true
    options:
      path: /var/lib/grafana/dashboards/node-exporter
EOF

mkdir -p /var/lib/grafana/dashboards/node-exporter
# download a dashboard definition for the Node-Exporter
wget -O /var/lib/grafana/dashboards/node-exporter/dashboard.json \
  https://raw.githubusercontent.com/rfmoz/grafana-dashboards/master/prometheus/node-exporter-full.json

…regular checks for changes to the dashboard definitions (defaults 10s)
…community references dashboards on a search page ⁷
allowUiUpdates …allows to make changes to the dashboard
- …not saved back to the provisioning source!
- …changes persist in the Grafana database
- …changes overwritten by updated to the provisioning source
- …use export to preserver the dashboard
disableDeletion …deletes dashboards from the database if provisioning source removed

Grafana CLI

# execute Grafana CLI in a container
podman exec -it grafana grafana cli --help

Grafana CLI ⁸ …bundled with Grafana server

# Reset the admin password…
grafana cli admin reset-admin-password secret123

…some basic administrative functions
…primarily used to manage plugins

Upgrades

Upgrade Grafana ⁹ …to any of the supported releases:

…upgrades are backward compatible between supported versions
Breaking changes ¹⁰ …require modifications on the user side

Footnotes

Grafana Project
https://grafana.com/oss
https://grafana.com/docs/grafana/latest
https://github.com/grafana/grafana ↩︎
Grafana Configuration, Grafana Documentation
https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana ↩︎
Provision Grafana, Grafana Documentation
https://grafana.com/docs/grafana/latest/administration/provisioning/↩︎
Built-in core data sources, Grafana Documentation
https://grafana.com/docs/grafana/latest/datasources/#built-in-core-data-sources ↩︎
Provisioning Dashboards, Grafana Documentation
https://grafana.com/docs/grafana/latest/administration/provisioning/#dashboards ↩︎
Node Exporter Dashboard, Grafana
https://grafana.com/grafana/dashboards/1860-node-exporter-full ↩︎
Dashboard Search, GrafanaLabs
https://grafana.com/grafana/dashboards ↩︎
Grafana CLI, Grafana Documentation
https://grafana.com/docs/grafana/latest/cli ↩︎
Upgrade Grafana, Grafana Documentation
https://grafana.com/docs/grafana/latest/upgrade-guide ↩︎
Breaking changes in Grafana, Grafana Documentation
https://grafana.com/docs/grafana/latest/breaking-changes ↩︎

--- title: Grafana Monitoring Dashboard categories: - Monitor date: 2016/01/19 date-modified: 2024/05/03 toc-expand: 3 --- Grafan [^uZm6R] is an open source dashboard to present monitoring metrics: [^uZm6R]: Grafana Project <https://grafana.com/oss> <https://grafana.com/docs/grafana/latest> <https://github.com/grafana/grafana> - Multiple time series data storage solutions (e.g. InfluxDB, Promehteus, OpenTSDB) can be used in combination by connecting them as **data sources**. - **Organizations** provide a method for multi-tenancy, and group users, data-sources, and the interfaces into distinct entities. - **Users** have accounts with different levels of privileges, and can authenticate with internal and external (LDAP, OAuth, GitHub, Google) methods. - **Dashboards** are build by users to visualize monitoring data in a customized way. Dashboard are a combination of panels showing graphs, tables and other data based on configurable queries on the data sources. - The basic visualization block are **panels**. They provide a **query editor** to source the data, and various styling and formatting options to model the data visualization. - The dashboard layouts use **rows** as logical dividers to arrange panels on the view horizontally and vertically. ## Installation ```bash # Open source Grafana podman run -d -p 3000:3000 --name=grafana \ -v grafana-storage:/var/lib/grafana \ grafana/grafana-oss ``` Install Grafana using the official container container: - Defaults to HTTP port `3000`, with default user/password `admin`. - Grafana uses an embedded SQLite version 3 database to store configuration… ### Environment Variables Set an environment variable to run in debug mode: ```bash podman run -d -p 3000:3000 --name=grafana \ -v grafana-storage:/var/lib/grafana \ -e "GF_LOG_LEVEL=debug" \ grafana/grafana-oss ``` - …override existing options …naming convention `GF_<SectionName>_<KeyName>` - Reference the `grafana.ini`… - …section name text within the brackets - …all text uppercase …replace `.-` with underscore ### Configuration File Use a configuration file… ```bash mkdir /etc/grafana /var/lib/grafana # …dummy configuration file example cat > /etc/grafana/grafana.ini <<EOF # default section instance_name = my-instance [security] admin_user = admin admin_password = secret123 [paths] data = /srv/grafana logs = /var/log/grafana plugins = /var/lib/grafana/plugins provisioning = /etc/grafana/provisioning EOF # …mount the configuration into the container podman run -d -p 3000:3000 --name=grafana \ -v /etc/grafana/:/etc/grafana:ro \ -v /srv/grafana:/srv/grafana:rw,U,Z -v /var/lib/grafana:/var/lib/grafana:rw,U,Z \ grafana/grafana-oss:10.2.6 ``` ### Systemd Unit Create a systemd service unit for the container: ```bash cd /etc/systemd/system # …generate the service unit file …name is prefixed with `container-` podman generate systemd --files --name grafana # …make sure to start Grafana on boot systemctl enable --now container-grafana.service ``` ## Configuration File | Description -----|-------------- `/etc/grafana/grafana.ini` | Local custom configuration file Configure Grafana [^vlDEP] using a custom configuration file: [^vlDEP]: Grafana Configuration, Grafana Documentation <https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana> - …default settings in `$WORKING_DIR/conf/defaults.ini` - …default path overwritten with option `--config` - …`;` semicolons for comments - Structured into sections …for example `[server]` …notable sections: - `paths` …data store, log file & provisioning - `server` …ports, certificates, HTTPS, routes - `database` …for users, dashboards, etc - `security` …admins login, HTTP security, etc - `users` …sign-up, themes, language, etc - `auth.*` …external identity providers - `log.*` …all about access/error logs - …and more ## Provisioning Provisioning system that uses configuration files [^7dhLo]: [^7dhLo]: Provision Grafana, Grafana Documentation <https://grafana.com/docs/grafana/latest/administration/provisioning/> - Configure where to find the provisioning directory with `paths.provisioning` - Add YAML configuration files to …updated during (re-)start Assumes following directory structure: Path | Description -----|------------- `provisioning/datasources/` | …each file can contain a list of `datasources` [^1bjdD] `provisioning/plugins/` | …manage plugin applications `provisioning/dashboards/` | …manage dashboards `provisioning/notifiers` | …alerting resources [^1bjdD]: Built-in core data sources, Grafana Documentation <https://grafana.com/docs/grafana/latest/datasources/#built-in-core-data-sources> ### Data Sources Create a data source manifest: ```bash mkdir -p /etc/grafana/provisioning/datasources cat > /etc/grafana/provisioning/datasources/promehteus.yml <<EOF datasources: - name: prometheus type: prometheus url: http://localhost:9090 EOF ``` ### Dashboards Create a dashboard manifest [^pDVTB] & dashboard definition [^4bKeQ]: [^4bKeQ]: Node Exporter Dashboard, Grafana <https://grafana.com/grafana/dashboards/1860-node-exporter-full> [^pDVTB]: Provisioning Dashboards, Grafana Documentation <https://grafana.com/docs/grafana/latest/administration/provisioning/#dashboards> ```bash mkdir -p /etc/grafana/provisioning/dashboards cat > /etc/grafana/provisioning/dashboards/node-exporter.yml <<EOF apiVersion: 1 providers: - name: Node-Exporter folder: Node-Exporter type: file disableDeletion: false allowUiUpdates: true options: path: /var/lib/grafana/dashboards/node-exporter EOF mkdir -p /var/lib/grafana/dashboards/node-exporter # download a dashboard definition for the Node-Exporter wget -O /var/lib/grafana/dashboards/node-exporter/dashboard.json \ https://raw.githubusercontent.com/rfmoz/grafana-dashboards/master/prometheus/node-exporter-full.json ``` - …regular checks for changes to the dashboard definitions (defaults 10s) - …community references dashboards on a search page [^oCVqD] - `allowUiUpdates` …allows to make changes to the dashboard - …not saved back to the provisioning source! - …changes persist in the Grafana database - …changes overwritten by updated to the provisioning source - …use export to preserver the dashboard - `disableDeletion` …deletes dashboards from the database if provisioning source removed [^oCVqD]: Dashboard Search, GrafanaLabs <https://grafana.com/grafana/dashboards> ## Grafana CLI ```bash # execute Grafana CLI in a container podman exec -it grafana grafana cli --help ``` Grafana CLI [^urxK3] …bundled with Grafana server [^urxK3]: Grafana CLI, Grafana Documentation <https://grafana.com/docs/grafana/latest/cli> ```bash # Reset the admin password… grafana cli admin reset-admin-password secret123 ``` - …some basic administrative functions - …primarily used to manage plugins ## Upgrades Upgrade Grafana [^JuGrb] …to any of the supported releases: - …upgrades are backward compatible between supported versions - Breaking changes [^JuG1b] …require modifications on the user side [^JuGrb]: Upgrade Grafana, Grafana Documentation <https://grafana.com/docs/grafana/latest/upgrade-guide> [^JuG1b]: Breaking changes in Grafana, Grafana Documentation <https://grafana.com/docs/grafana/latest/breaking-changes>