NFS Storage on the Network

Quick Reference

Linux

Storage

Published

May 14, 2024

Modified

November 29, 2024

NFS (Network File System) …network-mounted file-sharing system

…client/server protocol based on RPC (Remote Procedure Call) services
…mounted in a directory and appear as a local file system
…recommended to use time-synchronization for the system clock

# install packages
sudo dnf install nfs-utils

Server

# start the server
sudo systemctl enable --now nfs-server rpcbind

# open the firewall
sudo firewall-cmd --add-service={nfs,nfs3,mountd,rpc-bind} --permanent 
sudo firewall-cmd --reload

`/etc/nfs.conf`

Global configuration options in /etc/nfs.conf

…configuration in /etc/nfs.conf.d/*.conf overwrite
…file format supports multiple sections
- …introduced by a line containing the section for example [nfsd]
- …configuration by key=value, for example threads=32

mkdir /etc/nfs.conf.d
cat > /etc/nfs.conf.d/threads.conf <<EOF
[nfsd]
threads=32
EOF

`/etc/exports`

Or /etc/exports.d/*.exports…

…controls which file systems are exported to remote hosts
…comments use hash mark # …line-wrap with \

One NFS share per line:

/exported/directory 192.168.*(rw,async) node*.example.org(ro)

/path …absolute path to the shared directory
Followed by a list of authorized hosts…
- FQDN or IP address …network addresses …* for everybody
- …multiple hosts separated by space
Options
- ro read-only (default) …remote can not change data
- rw read-write
- sync waits for write requests to be finished before response
- async asynchronous writes
- root_squash prevents root users connected remotely from having root privileges

# modifications to exports requires a re-export to take effect
exportfs -arv

# view current loaded exports
exportfs -v

Client

Mount an NFS share:

# list available shares on a server
>>> showmount -e nfs01
Export list for nfs01:
/srv/nfs4 *.example.org

# mount a shared-directory from a server
>>> mount -t nfs4 nfs01:/srv/nfs /mnt/nfs

# remove an NFS share
>>> umount /mnt/nfs

Systemd

Example of using a Systemd unit to mount storage from an NFS server:

cat > /etc/systemd/system/scratch.mount <<EOF
[Unit]
Description=Mount NFS scratch storage
Wants=network-online.target
After=network-online.target

[Mount]
What=10.20.12.3:/srv/nfs4
Where=/scratch
Type=nfs
Options=rw,nodev,hard,vers=4.2,proto=rdma,port=20049
TimeoutSec=10s
systemd
[Install]
WantedBy=multi-user.target
WantedBy=remote-fs.target
EOF
systemctl daemon-reload
systemctl start scratch.mount

Autofs

autofs¹ mounts & unmounts file systems automatically (on-demand)

Map files configure individual on-demand mount points
- /etc/auto.master (master map) …avoid to edit this file
- …additional maps in /etc/auto.master.d/*.autofs
- …full description in man 5 auto.master
automount manages autofs mount points
- Starts by reading /etc/auto.master
- …creates the directories if they do not exist

Install using the autofs RPM package², start the service with Systemd:

dnf install -y autofs
systemctl enable --now autofs

Configure a custom autofs map to mount an NFS file-systems to /scratch:

mkdir /etc/auto.master.d

# reference a map in the local file-system and provide common mount options
cat > /etc/auto.master.d/scratch.autofs <<EOF
/scratch file,sun:/etc/auto.scratch hard,vers=4.2,proto=rdma,port=20049
EOF

# list of mount point as sub-directory to /scratch...
# ...and the corresponding NFS server path 
cat > /etc/auto.scratch <<EOF
foo nfs1.example.org:/srv/nfs4/foo
bar nfs2.example.org:/srv/nfs4/bar
EOF

# read the new map configuration
systemctl reload autofs

List information about all configured maps with autmount -m

Check the log-information with journalctl -u autofs
/etc/autofs.conf …default configuration in …see man 5 autofs.conf
- …location of the master map file …default timeout to unmount

# check the configuration file
grep -v '^#' /etc/autofs.conf

Stop autofs.service for debugging:

# running the daemon in foreground and with verbose mode
automount -f -v

`nfsiostat`

Reads from /proc/self/mountstats…

# NFS I/O statistics for a given mount point
nfsiostat /scratch/hpc 1

Statistic	Description
`ops/s`	number of operations per second
`rpc bklog`	length of the backlog queue
`kB/s`	number of kB written/read per second
`kB/op`	number of kB written/read per each operation
`retrans`	number of retransmissions
`avg RTT (ms)`	duration …client’s kernel sends RPC until it receives a reply
`avg exe (ms)`	duration …NFS request until completion (includes RTT)

`mountstats`

mountstats displays various NFS client per-mount statistics…

…for each NFS operation:
- Number of times operation has been called and number of errors
- Bytes send/received per operation
- RTT Round-Trip Time in milliseconds
- Total execution time in milliseconds
Complete list of NFS operations³ in RFC7530

# statistics for a specific mount point
mountstats /scratch/hpc | less

Example output:

#...
LOOKUP:
        256 ops (0%)    254 errors (99%)
        avg bytes sent per op: 215      avg bytes received per op: 97
        backlog wait: 1.835938  RTT: 1.429688   total execute time: 3.292969 (milliseconds)
#...
OPEN:
        126 ops (0%)    26 errors (20%)
        avg bytes sent per op: 321      avg bytes received per op: 320
        backlog wait: 0.047619  RTT: 1.468254   total execute time: 1.555556 (milliseconds)
#...

Footnotes

Aut0mount, Linux Kernel Documentation
https://docs.kernel.org/filesystems/automount-support.html
https://www.kernel.org/doc/html/latest/filesystems/autofs.html ↩︎
autofs Fedora Package
https://src.fedoraproject.org/rpms/autofs ↩︎
RFC7530, NFSv4 Operations
https://www.rfc-editor.org/rfc/rfc7530#page-214 ↩︎

--- title: NFS Storage on the Network subtitle: Quick Reference categories: - Linux - Storage date: 2024/05/14 date-modified: 2024/11/29 toc-expand: 3 --- NFS (Network File System) …network-mounted file-sharing system - …client/server protocol based on **RPC** (Remote Procedure Call) services - …mounted in a directory and appear as a local file system - …recommended to use time-synchronization for the system clock ```bash # install packages sudo dnf install nfs-utils ``` # Server ```bash # start the server sudo systemctl enable --now nfs-server rpcbind # open the firewall sudo firewall-cmd --add-service={nfs,nfs3,mountd,rpc-bind} --permanent sudo firewall-cmd --reload ``` ## `/etc/nfs.conf` Global configuration options in `/etc/nfs.conf` - ...configuration in `/etc/nfs.conf.d/*.conf` overwrite - ...file format supports multiple sections - ...introduced by a line containing the section for example `[nfsd]` - ...configuration by `key=value`, for example `threads=32` ```bash mkdir /etc/nfs.conf.d cat > /etc/nfs.conf.d/threads.conf <<EOF [nfsd] threads=32 EOF ``` ## `/etc/exports` Or `/etc/exports.d/*.exports`… - …controls which file systems are exported to remote hosts - …comments use hash mark `#` …line-wrap with `\` One **NFS share** per line: ```txt /exported/directory 192.168.*(rw,async) node*.example.org(ro) ``` - `/path` …absolute path to the shared directory - Followed by a list of authorized hosts… - FQDN or IP address …network addresses …`*` for everybody - …multiple hosts separated by space - Options - `ro` read-only (default) …remote can not change data - `rw` read-write - `sync` waits for write requests to be finished before response - `async` asynchronous writes - `root_squash` prevents root users connected remotely from having root privileges ```bash # modifications to exports requires a re-export to take effect exportfs -arv # view current loaded exports exportfs -v ``` # Client Mount an NFS share: ```bash # list available shares on a server >>> showmount -e nfs01 Export list for nfs01: /srv/nfs4 *.example.org # mount a shared-directory from a server >>> mount -t nfs4 nfs01:/srv/nfs /mnt/nfs # remove an NFS share >>> umount /mnt/nfs ``` ## Systemd Example of using a Systemd unit to mount storage from an NFS server: ```bash cat > /etc/systemd/system/scratch.mount <<EOF [Unit] Description=Mount NFS scratch storage Wants=network-online.target After=network-online.target [Mount] What=10.20.12.3:/srv/nfs4 Where=/scratch Type=nfs Options=rw,nodev,hard,vers=4.2,proto=rdma,port=20049 TimeoutSec=10s systemd [Install] WantedBy=multi-user.target WantedBy=remote-fs.target EOF systemctl daemon-reload systemctl start scratch.mount ``` ## Autofs **`autofs`[^w9ggM] mounts & unmounts file systems automatically (on-demand)** [^w9ggM]: Aut0mount, Linux Kernel Documentation <https://docs.kernel.org/filesystems/automount-support.html> <https://www.kernel.org/doc/html/latest/filesystems/autofs.html> - Map files configure individual on-demand mount points - `/etc/auto.master` (master map) …avoid to edit this file - …additional maps in `/etc/auto.master.d/*.autofs` - …full description in `man 5 auto.master` - `automount` manages `autofs` mount points - Starts by reading `/etc/auto.master` - …creates the directories if they do not exist Install using the `autofs` RPM package[^2OeaD], start the service with Systemd: [^2OeaD]: `autofs` Fedora Package <https://src.fedoraproject.org/rpms/autofs> ```bash dnf install -y autofs systemctl enable --now autofs ``` Configure a custom autofs map to mount an NFS file-systems to `/scratch`: ```bash mkdir /etc/auto.master.d # reference a map in the local file-system and provide common mount options cat > /etc/auto.master.d/scratch.autofs <<EOF /scratch file,sun:/etc/auto.scratch hard,vers=4.2,proto=rdma,port=20049 EOF # list of mount point as sub-directory to /scratch... # ...and the corresponding NFS server path cat > /etc/auto.scratch <<EOF foo nfs1.example.org:/srv/nfs4/foo bar nfs2.example.org:/srv/nfs4/bar EOF # read the new map configuration systemctl reload autofs ``` **List information about all configured maps with `autmount -m`** - Check the log-information with `journalctl -u autofs` - `/etc/autofs.conf` …default configuration in …see `man 5 autofs.conf` - …location of the master map file …default timeout to unmount ```bash # check the configuration file grep -v '^#' /etc/autofs.conf ``` Stop `autofs.service` for debugging: ```bash # running the daemon in foreground and with verbose mode automount -f -v ``` ## `nfsiostat` Reads from `/proc/self/mountstats`… ```bash # NFS I/O statistics for a given mount point nfsiostat /scratch/hpc 1 ``` Statistic | Description -|-------------------------------------------------- `ops/s` | number of operations per second `rpc bklog` | length of the backlog queue `kB/s` | number of kB written/read per second `kB/op` | number of kB written/read per each operation `retrans` | number of retransmissions `avg RTT (ms)` | duration …client's kernel sends RPC until it receives a reply `avg exe (ms)` | duration …NFS request until completion (includes RTT) ## `mountstats` `mountstats` displays various NFS client per-mount statistics… - …for each NFS operation: - Number of times operation has been called and number of errors - Bytes send/received per operation - `RTT` Round-Trip Time in milliseconds - Total execution time in milliseconds - Complete list of NFS operations[^ZfLRd] in RFC7530 [^ZfLRd]: RFC7530, NFSv4 Operations <https://www.rfc-editor.org/rfc/rfc7530#page-214> ```bash # statistics for a specific mount point mountstats /scratch/hpc | less ``` Example output: ```bash #... LOOKUP: 256 ops (0%) 254 errors (99%) avg bytes sent per op: 215 avg bytes received per op: 97 backlog wait: 1.835938 RTT: 1.429688 total execute time: 3.292969 (milliseconds) #... OPEN: 126 ops (0%) 26 errors (20%) avg bytes sent per op: 321 avg bytes received per op: 320 backlog wait: 0.047619 RTT: 1.468254 total execute time: 1.555556 (milliseconds) #... ```