Slurm — Authentication Service

Overview

Authentication mechanisms¹ — Verify legitimacy & integrity of RPC requests

Mechanism	AuthType	Description
MUNGE	auth/munge	External munged service to authenticate
Slurm	auth/slurm	Slurm internal authentication plugin
JWT		Authentication using JSON Web Tokens (JWT)2

MUNGE

MUNGE (MUNGE Uid ‘N’ Gid Emporium)…

• …authentication service for creating and validating credentials
• …highly scalable…designed for HPC clusters
• …hosts form a security realm…defined by a shared cryptographic key
  • …clients create/validate credentials without root privileges
  • …process to authenticate the UID/GID of another local/remote process

References…

• GitHub Repository
  • https://dun.github.io/munge
  • https://github.com/dun/munge/releases
• Wiki - Installation Guide & Man-Pages
  • https://github.com/dun/munge/wiki/Installation-Guide
• Fedora EPEL Package
  • https://src.fedoraproject.org/rpms/munge
• RPM SPEC & Systemd Service Unit
  • https://github.com/dun/munge/blob/master/munge.spec.in
  • https://github.com/dun/munge/blob/master/src/etc/munge.systemd.service.in

munged.service

Create a munge.service Systemd unit…

cat > /etc/systemd/system/munge.service <<'EOF'
[Unit]
Description=munged authentication service
[Service]
Type=forking
User=munge
Group=munge
RuntimeDirectory=munge
RuntimeDirectoryMode=0755
EnvironmentFile=-/etc/default/munge
ExecStart=munged $MUNGE_OPTIONS
[Install]
WantedBy=default.target
EOF
# load new unit file...restart...check state
systemctl daemon-reload && systemctl restart munge
systemctl status --full --lines=50 munge

Shared Secret

….requires a shared secret on all nodes…

• …stored in /etc/munge/munge.key
• …typically distributed by configuration management

# create the configuration directory (if missing)
mkdir /etc/munge \
      && chown munge:munge /etc/munge \
      && chmod 0700 /etc/munge
# create a shared secret
sudo -u munge /usr/sbin/mungekey -v
# ...or...
dd if=/dev/urandom of=/etc/munge/munge.key bs=1 count=1024
# ...adjust permissions
chown munge:munge /etc/munge/munge.key
chmod 0600 /etc/munge/munge.key

Verify functionality…

# encode credential
munge -n
# enode & decode credential 
munge -n | unmunge
munge -n -t 10 | ssh $node unmunge
ssh $node munge -n -t 10 | unmunge

Configuration

Fore-ground daemon for debugging…

# overwrite nologin and run munged in foreground
su -s /bin/bash munge -c "$MUNGE_PATH/sbin/munged $OPTIONS -Fv"
# ...with logging to a file /var/log/... don't foreground
su -s /bin/bash munge -c "$MUNGE_PATH/sbin/munged $OPTIONS"
ps -p $(pgrep munged) -fH ; pkill munged

Use a dedicated user…

groupadd -r munge
useradd -c "MUNGE authentication service" \
        -d "/var/lib/munge" \
        -g munge \
        -s /sbin/nologin \
        -r munge

munged enforces files permissions…

# PRNG Seed...
mkdir -p /var/lib/munge \
       && chown munge:munge /var/lib/munge \
       && chmod 0700 /var/lib/munge
# socket configuration...
cat > /usr/lib/tmpfiles.d/munge.conf <<EOF
d /run/munge 0755 munge munge -
EOF
systemd-tmpfiles --create /usr/lib/tmpfiles.d/munge.conf
# log-files...
mkdir /var/log/munge \
      && chown munge:munge /var/log/munge \
      && chmod 0700 /var/log/munge

Non-Default Paths

Path to the socket is set by a compile time option…

• …MUNGE Installation Guide
• …specifically --runstatedir=/run

# options to configure the build
./configure --sysconfdir=/etc --localstatedir=/var --runstatedir=/run

Custom paths to all files…

munged \
      --seed-file=/var/lib/munge/munge.seed \
      --key-file=/etc/munge/munge.key \
      --pid-file=/run/munge/munged.pid \
      --socket /run/munge/munge.socket.2 \
      --log-file=/var/log/munge/munged.log

Slurm is configured with --with-munge=$path…

slurmctld & slurmdbd require socket= for non-default path…

>>> grep Auth /etc/slurm/slurm*
/etc/slurm/slurm.conf:AuthType=auth/munge
/etc/slurm/slurm.conf:AuthInfo=socket=/run/munge/munge.socket.2
/etc/slurm/slurmdbd.conf:AuthType=auth/munge
/etc/slurm/slurmdbd.conf:AuthInfo=socket=/run/munge/munge.socket.2

It is likely that Slurm uses the MUNGE API man 3 munge…

• …with a context format described in man 3 munge_ctx
• …which support MUNGE_OPT_SOCKET
• …depends on propagation of the socket location by the caller
• …seams that sacct* commands are not support a non-default socket

Thread Count

Munge daemon runs with two threads by default…

• …higher thread count can improve its throughput.
• …start with munged --num-threads 10
• …thread count to 10 will not have negative effects

Slurm

Configure the Slurm authentication type:

slurm.conf

AuthType=auth/slurm
CryptoType=crypto/slurm

Shared Key

All hosts in the cluster must have a shared cryptographic key

# Generate a shared key from random data
dd if=/dev/random of=/etc/slurm/slurm.key bs=1024 count=1
chown slurm:slurm /etc/slurm/slurm.key
chmod 600 /etc/slurm/slurm.key

From the slurmctld logs:

# key missing
fatal: auth/slurm: cannot stat '/etc/slurm/slurm.key': No such file or directory

# key loaded
debug:  auth/slurm: init: running as daemon
debug:  auth/slurm: init_internal: loading key: `/etc/slurm/slurm.key`

Footnotes

1. Authentication Plugins, SchedMD Documentation
   https://slurm.schedmd.com/authentication.html

2. JSON Web Tokens (JWT) Authentication, SchedMD Documentation
   https://slurm.schedmd.com/jwt.html