Slurm User Interface

HPC

Published

October 31, 2024

Modified

October 31, 2024

Recurring Jobs

scrontab schedules recurring jobs on the cluster. It provides a cluster based equivalent to crontab (short for “cron table”), a system that specifies scheduled tasks to be run by the cron daemon¹ on Unix-like systems. scrontab is used to configure Slurm to execute commands at specified intervals, allowing users to automate repetitive tasks.

All users can have their own scrontab file, allowing for personalized job scheduling without interfering with other users. Users can define jobs directly in the scrontab file, specifying the command to run, the schedule, and any Slurm options (like resource requests).

Format

The scrontab configuration format works similar to the traditional cron format, allowing users to specify when and how often jobs should be executed. The configuration can have several crontab entries (jobs).

# create a simple example for scrontab
>>> cat > sleep.scrontab <<EOF
#SCRON --time=00:02:00
#SCRON --job-name=sleep-scrontab
#SCRON --chdir=/lustre/hpc/vpenso
#SCRON --output=sleep-scrontab-%j.log
#SCRON --open-mode=append
*/10 * * * * date && sleep 30
EOF

# install a new scrontab from a file
>>> scrontab sleep.scrontab

# check the queue
>>> squeue --me -O Jobid,EligibleTime,Name,State
JOBID               ELIGIBLE_TIME       NAME                STATE               
14938318            2024-10-31T10:20:00 sleep-scrontab      PENDING

Time Fields

The first five fields specify the schedule for the job, and they represent from left to right:

Field	Description
Minute (0-59)	The minute of the hour when the job should be scheduled
Hour (0-23)	The hour of the day when the job should be scheduled
Day of the Month (1-31)	The specific day of the month when the job should run
Month (1-12)	The month when the job should run
Day of the Week (0-7)	The day of the week when the job should run (0 and 7 both represent Sunday).

Special characters are sued to define more complex schedules:

Character	Description
Asterisk (`*`)	Represents “every” unit of time. For example, an asterisk in the minute field means the job will run every minute.
Comma (`,`)	Used to specify multiple values. For example, `1,15` in the minute field means the job will run at the 1st and 15th minute of the hour.
Dash (`-`)	Specifies a range of values. For example, `1-5` in the day of the week field means the job will run from Monday to Friday.
Slash (`/`)	Specifies increments. For example, `*/5` in the minute field means the job will run every 5 minutes.

Some users may find it convenient to us a web-site based crontab generator² to prepare a custom configuration.

Shortcuts

Shortcuts to specify some common time intervals

Shortcut	Description
`@annually`	Job will become eligible at 00:00 Jan 01 each year
`@monthly`	Job will become eligible at 00:00 on the first day of each month
`@weekly`	Job will become eligible at 00:00 Sunday of each week
`@daily`	Job will become eligible at 00:00 each day
`@hourly`	Job will become eligible at the first minute of each hour.

Meta-Commands

Lines starting with #SCRON allow users to set Slurm options for the single following crontab entry. This means each crontab entry needs its own list of #SCRON meta-commands, for example:

#SCRON --job-name=sleep-scrontab
#SCRON --chdir /lustre/hpc/vpenso
@daily path/to/sleep.sh > sleep-$(date +%Y%m%dT%H%M).log

Options include most of those available to the sbatch command (make sure to read the manual pages for more details). In order to write output of a recurring job into a single file use following option:

Options	Description
`--open-mode`	Appends output to an existing log-file (instead of overwrite)

#SCRON --job-name=sleep-scrontab
#SCRON --chdir /lustre/hpc/vpenso
#SCRON --output=sleep-scrontab-%j.log
#SCRON --open-mode=append
0 8 * * * path/to/sleep.sh

Usage

Users can configure their scrontab in multiple ways:

# modify the configuration with your preferred text-edotr
1EDITOR=vim scrontab -e
# read the configuration from a file
2scrontab path/to/file

# print the configuration
3scrontab -l

# clear the configuration
4scrontab -r

1: Modify the configuration with an text-editor using option -e.
2: Apply a configuration by passing a file as argument.
3: Option -l print the configuration to the terminal
4: Option -r removes the entire configuration (jobs continue to run, but won’t longer recur).

Jobs have the same Job ID for every run (until the next time the configuration is modified).

# list jobs with 
1squeue --me -O Jobid,EligibleTime,Name,State

# list all recurring jobs in the past
2sacct --duplicates --jobs $job_id

# skip next run
3scontrol requeue $job_id

# disable a cron job
4scancel --cron $job_id

1: List when cronjobs will be eligible for next execution. Note that jobs are not guaranteed to execute at the preferred time.
2: List all recurring executions of the cronjob from the accounting.
3: Skip next execution of a cronjob with scontrol and reschedule the job to the upcoming available time.
4: Request to cancel a job submitted by crontab with scancel. The job in the crontab will be preceded by the comment #DISABLED

Footnotes

cron, Wikipedia
https://en.wikipedia.org/wiki/Cron ↩︎
Crontab Generator
https://crontab-generator.org ↩︎

--- title: Slurm User Interface categories: - HPC date: 2024/10/31 date-modified: 2024/10/31 --- # Recurring Jobs `scrontab` schedules recurring jobs on the cluster. It provides a cluster based equivalent to `crontab` (short for "cron table"), a system that specifies scheduled tasks to be run by the cron daemon[^vlMHW] on Unix-like systems. `scrontab` is used to configure Slurm to **execute commands at specified intervals, allowing users to automate repetitive tasks**. [^vlMHW]: `cron`, Wikipedia <https://en.wikipedia.org/wiki/Cron> All users can have their own `scrontab` file, allowing for personalized job scheduling without interfering with other users. Users can define jobs directly in the `scrontab` file, specifying the command to run, the schedule, and any Slurm options (like resource requests). ## Format The `scrontab` configuration format works similar to the traditional cron format, allowing users to specify when and how often jobs should be executed. The configuration can have several crontab entries (jobs). ```bash # create a simple example for scrontab >>> cat > sleep.scrontab <<EOF #SCRON --time=00:02:00 #SCRON --job-name=sleep-scrontab #SCRON --chdir=/lustre/hpc/vpenso #SCRON --output=sleep-scrontab-%j.log #SCRON --open-mode=append */10 * * * * date && sleep 30 EOF # install a new scrontab from a file >>> scrontab sleep.scrontab # check the queue >>> squeue --me -O Jobid,EligibleTime,Name,State JOBID ELIGIBLE_TIME NAME STATE 14938318 2024-10-31T10:20:00 sleep-scrontab PENDING ``` ### Time Fields The first five fields specify the schedule for the job, and they represent from left to right: Field | Description ------------|---------------------------------- Minute (0-59) | The minute of the hour when the job should be scheduled Hour (0-23) | The hour of the day when the job should be scheduled Day of the Month (1-31) | The specific day of the month when the job should run Month (1-12) | The month when the job should run Day of the Week (0-7) | The day of the week when the job should run (0 and 7 both represent Sunday). Special characters are sued to define more complex schedules: Character | Description ------------|---------------------------------- Asterisk (`*`) | Represents "every" unit of time. For example, an asterisk in the minute field means the job will run every minute. Comma (`,`) | Used to specify multiple values. For example, `1,15` in the minute field means the job will run at the 1st and 15th minute of the hour. Dash (`-`) | Specifies a range of values. For example, `1-5` in the day of the week field means the job will run from Monday to Friday. Slash (`/`) | Specifies increments. For example, `*/5` in the minute field means the job will run every 5 minutes. Some users may find it convenient to us a web-site based `crontab` generator[^THlUD] to prepare a custom configuration. [^THlUD]: Crontab Generator <https://crontab-generator.org> ### Shortcuts Shortcuts to specify some common time intervals Shortcut | Description ------------|---------------------------------- `@annually` | Job will become eligible at 00:00 Jan 01 each year `@monthly` | Job will become eligible at 00:00 on the first day of each month `@weekly` | Job will become eligible at 00:00 Sunday of each week `@daily` | Job will become eligible at 00:00 each day `@hourly` | Job will become eligible at the first minute of each hour. ### Meta-Commands Lines starting with `#SCRON` allow users to set Slurm options for the single following `crontab` entry. This means each `crontab` entry needs its own list of `#SCRON` meta-commands, for example: ```bash #SCRON --job-name=sleep-scrontab #SCRON --chdir /lustre/hpc/vpenso @daily path/to/sleep.sh > sleep-$(date +%Y%m%dT%H%M).log ``` Options include most of those available to the `sbatch` command (make sure to read the manual pages for more details). In order to write output of a recurring job into a single file use following option: Options | Description ------------|---------------------------------- `--open-mode` | Appends output to an existing log-file (instead of overwrite) ```bash #SCRON --job-name=sleep-scrontab #SCRON --chdir /lustre/hpc/vpenso #SCRON --output=sleep-scrontab-%j.log #SCRON --open-mode=append 0 8 * * * path/to/sleep.sh ``` ## Usage Users can configure their `scrontab` in multiple ways: ```bash # modify the configuration with your preferred text-edotr EDITOR=vim scrontab -e #<1> # read the configuration from a file scrontab path/to/file #<2> # print the configuration scrontab -l #<3> # clear the configuration scrontab -r #<4> ``` 1. Modify the configuration with an text-editor using option `-e`. 2. Apply a configuration by passing a file as argument. 3. Option `-l` print the configuration to the terminal 4. Option `-r` removes the entire configuration (jobs continue to run, but won't longer recur). **Jobs have the same Job ID for every run** (until the next time the configuration is modified). ```bash # list jobs with squeue --me -O Jobid,EligibleTime,Name,State #<1> # list all recurring jobs in the past sacct --duplicates --jobs $job_id #<2> # skip next run scontrol requeue $job_id #<3> # disable a cron job scancel --cron $job_id #<4> ``` 1. List when cronjobs will be eligible for next execution. Note that jobs are not guaranteed to execute at the preferred time. 2. List all recurring executions of the cronjob from the accounting. 3. Skip next execution of a cronjob with `scontrol` and reschedule the job to the upcoming available time. 4. Request to cancel a job submitted by crontab with `scancel`. The job in the crontab will be preceded by the comment `#DISABLED`