Slurm - Accounts & Associations

HPC

Published

May 21, 2019

Modified

October 13, 2023

Enforce user account associations in slurm.conf to prevents users without an account association to interact with slurmctdl

>>> scontrol show config | grep AccountingStorageEnforce
AccountingStorageEnforce = associations,[...]

User in Slurm can have multiple of the following roles:

Role	Description
User	(Linux) Account able to request resources
Coordinator	Representative managing a single account
Operator	Creates and manages accounts in general
Admin	Owns all available privileges (like root)

`sacctmgr`

…used to configure accounts:

Account records are maintained based on associations
Associations are a combination of: cluster, (partition,) account, user
Associations have a fair-share allocation and group limits
Account names must be unique, can’t be repeated inside the hierarchy
Accounts inherit limits assigned to a parent association

sacctmgr -i add cluster $name             # add a new cluster to the database
sacctmgr show problem                     # cheek config consistency
sacctmgr list cluster                     # list all cluster in the database
scontrol show config | grep ClusterName   # show the name of the cluster
sacctmgr list account                      # list accounts
# create a new account
sacctmgr add account $name description="$text" organization=$org
# alter the account configuration
sacctmgr modify account name=$name set organization=$org
# remove an account
sacctmgr delete account $name
# accounts may be arranged in a hierarchical tree-like structure
# limits are inherited
sacctmgr add account $name parent=$parent description="$text" organization=$org

Users

List users associated to an account…

# print a list of all user associations
sacctmgr show user withassoc format=user,account,defaultaccount

# ...a specific user
sacctmgr show user withassoc format=user,account,defaultaccount where user=$USER

# ...parsable output example...
sacctmgr --noheader --parsable2 show user withassoc format=user,account,defaultaccount \
      | cut -d'|' -f1- --output-delimiter=" " \
      | column --table --table-columns "User,Account,Default"

Associate users to an account…

# ...user not assoicated to any account yet...
sacctmgr create user name=$user account=$account defaultaccount=$default

# ...assoicate a user to another account...
sacctmgr add user account=$account names=$users

# ...modify the default account of a user...
sacctmgr modify user name=$user set defaultaccount=$account

Default Account

Create a default account …very limited amount of available resources:

>>> sacctmgr -s list account where name=default \
        format=account,grpsubmit,grpjob,grpnode,grpcpus
   Account GrpSubmit GrpJobs GrpNodes  GrpCPUs 
---------- --------- ------- -------- -------- 
   default      1000     100       25      500

Associate user to the default account under following circumstance…

…no account is available to a user (yet)
…user dose not know its account coordinator (yet)
…user wants to preview access

None Account

The account none has no associated resources:

>>> sacctmgr -s show account where account=none \
        format=account,grpsubmit,grpjob,grpnode,grpcpus
   Account GrpSubmit GrpJobs GrpNodes  GrpCPUs 
---------- --------- ------- -------- -------- 
      none         0       0        0

Enforce the --account option to be specified…

…for the salloc, srun, and sbatch commands
…constrain user with multiple associated accounts to specifically select one
- …use it as defaultaccount=none for users which should be force to specify the option
- …note that beforehand a users needs to be associated to the “none” account

Following error will be printed:

Unable to allocate resources: Invalid account or account/partition...

Coordinators

List account coordinators:

# print a list of all coordinators for a given account
sacctmgr list account withcoordinator where account=$account
# list coordinators alphabetically
sacctmgr list user withcoordinator format=user,coordinator -P \
        | tr '|' ' ' | awk 'NF==2'

Promote a user to be coordinator for a given account:

# add coordinator(s) to account
sacctmgr add coordinator account=$account names=$user,...

Administrators

Admin users can operate the accounting database, and alter anything on an instance of slurmctld as if root.

# list all user with admin privileges
sacctmgr show user withassoc format=user,adminlevel where adminlevel=admin
# grant admin privileges to a user
sacctmgr modify user where user=$user set adminlevel=admin

Options for adminlevel=

Level	Description
`none`	Regular user, no special privileges.
`operator`	Can add, modify, and remove any database object (user, account, etc), and add other operators.
`admin`	Same level of privileges as an operator in the database. Can also alter anything on `slurmctld` as if root.

Migration

sacctmgr supports retrieving the association data including limits

…does not include the accounting data like counters for consumed resources
…to preserve accounting data it is required to backup the accounting database

# find more detailed information in the man page
LESS="-p FLAT FILE DUMP AND LOAD" man -P less sacctmgr

# print a list of the clusters
sacctmgr list cluster format=cluster,controlhost,controlport,rpc --noheader

Each cluster requires to be dumped into a dedicated file

# ...retrieve association configuration form a cluster
sacctmgr dump cluster=$NAME file=$NAME.cfg

# ...configure the associations for a cluster from a file
sacctmgs load cluster=$NAME file=$NAME.cfg

…dump <CLUSTER> [File=<FILENAME>]…
- …dump database information of the specified cluster to flat file
…load <FILE> [<SPECS>]…
- …read in the file to update the database
- <SPECS>…
  - …cluster= will override the cluster name given in the file
  - …clean …remove configuration …and replace with file
- Without clean …additions & modifications …no deletions

--- title: Slurm - Accounts & Associations categories: - HPC date: 2019/05/21 date-modified: 2023/10/13 toc-expand: 3 --- Enforce user account associations in `slurm.conf` to prevents users without an account association to interact with `slurmctdl` ```bash >>> scontrol show config | grep AccountingStorageEnforce AccountingStorageEnforce = associations,[...] ``` User in Slurm can have multiple of the following roles: Role | Description ------------|------------- User | (Linux) Account able to request resources Coordinator | Representative managing a single account Operator | Creates and manages accounts in general Admin | Owns all available privileges (like root) ## `sacctmgr` ...used to configure accounts: * Account records are maintained based on **associations** * Associations are a combination of: cluster, (partition,) account, user * Associations have a **fair-share allocation** and **group limits** * Account names must be unique, can't be repeated inside the hierarchy * Accounts inherit limits assigned to a parent association ```bash sacctmgr -i add cluster $name # add a new cluster to the database sacctmgr show problem # cheek config consistency sacctmgr list cluster # list all cluster in the database scontrol show config | grep ClusterName # show the name of the cluster sacctmgr list account # list accounts # create a new account sacctmgr add account $name description="$text" organization=$org # alter the account configuration sacctmgr modify account name=$name set organization=$org # remove an account sacctmgr delete account $name # accounts may be arranged in a hierarchical tree-like structure # limits are inherited sacctmgr add account $name parent=$parent description="$text" organization=$org ``` ## Users List users associated to an account... ```sh # print a list of all user associations sacctmgr show user withassoc format=user,account,defaultaccount # ...a specific user sacctmgr show user withassoc format=user,account,defaultaccount where user=$USER # ...parsable output example... sacctmgr --noheader --parsable2 show user withassoc format=user,account,defaultaccount \ | cut -d'|' -f1- --output-delimiter=" " \ | column --table --table-columns "User,Account,Default" ``` Associate users to an account... ```sh # ...user not assoicated to any account yet... sacctmgr create user name=$user account=$account defaultaccount=$default # ...assoicate a user to another account... sacctmgr add user account=$account names=$users # ...modify the default account of a user... sacctmgr modify user name=$user set defaultaccount=$account ``` ### Default Account Create a default account ...very limited amount of available resources: ```bash >>> sacctmgr -s list account where name=default \ format=account,grpsubmit,grpjob,grpnode,grpcpus Account GrpSubmit GrpJobs GrpNodes GrpCPUs ---------- --------- ------- -------- -------- default 1000 100 25 500 ``` Associate user to the `default` account under following circumstance... * ...no account is available to a user (yet) * ...user dose not know its account coordinator (yet) * ...user wants to preview access ### None Account The account `none` has no associated resources: ```bash >>> sacctmgr -s show account where account=none \ format=account,grpsubmit,grpjob,grpnode,grpcpus Account GrpSubmit GrpJobs GrpNodes GrpCPUs ---------- --------- ------- -------- -------- none 0 0 0 ``` Enforce the `--account` option to be specified... * ...for the `salloc`, `srun`, and `sbatch` commands * ...constrain user with multiple associated accounts to specifically select one * ...use it as `defaultaccount=none` for users which should be force to specify the option * ...note that beforehand a users needs to be associated to the "none" account Following error will be printed: ``` Unable to allocate resources: Invalid account or account/partition... ``` ## Coordinators List account coordinators: ```bash # print a list of all coordinators for a given account sacctmgr list account withcoordinator where account=$account # list coordinators alphabetically sacctmgr list user withcoordinator format=user,coordinator -P \ | tr '|' ' ' | awk 'NF==2' ``` **Promote a user to be coordinator** for a given account: ```bash # add coordinator(s) to account sacctmgr add coordinator account=$account names=$user,... ``` ## Administrators Admin users can operate the accounting database, and alter anything on an instance of `slurmctld` as if root. ```bash # list all user with admin privileges sacctmgr show user withassoc format=user,adminlevel where adminlevel=admin # grant admin privileges to a user sacctmgr modify user where user=$user set adminlevel=admin ``` Options for `adminlevel=` Level | Description -----------|-------------- `none` | Regular user, no special privileges. `operator` | Can add, modify, and remove any database object (user, account, etc), and add other operators. `admin` | Same level of privileges as an operator in the database. Can also alter anything on `slurmctld` as if root. ## Migration `sacctmgr` supports **retrieving the association data including limits** * ...does not include the accounting data like counters for consumed resources * ...to preserve accounting data it is required to backup the accounting database ```bash # find more detailed information in the man page LESS="-p FLAT FILE DUMP AND LOAD" man -P less sacctmgr # print a list of the clusters sacctmgr list cluster format=cluster,controlhost,controlport,rpc --noheader ``` Each cluster requires to be dumped into a dedicated file ```bash # ...retrieve association configuration form a cluster sacctmgr dump cluster=$NAME file=$NAME.cfg # ...configure the associations for a cluster from a file sacctmgs load cluster=$NAME file=$NAME.cfg ``` * ...`dump <CLUSTER> [File=<FILENAME>]`... * ...dump database information of the specified cluster to flat file * ...`load <FILE> [<SPECS>]`... * ...read in the file to update the database * `<SPECS>`... * ...`cluster=` will override the cluster name given in the file * ...`clean` ...remove configuration ...and replace with file * Without `clean` ...additions & modifications ...no deletions