2. Jobs howto
2.1. General Recommendations
In this guide, we provide information about how to submit your jobs to Slurm, how to monitor them and how to request various resources.
In Job examples guide we show specific in-depth examples how to run array jobs, request GPUs, CPUs, and memory for a variety of different job types and common applications.
- Get the most from your allocation
Check your past jobs for how many CPU and memory resources were used. Don’t request more than needed.
Prioritize your own work: testing and low-priority jobs can be free jobs, others should be allocated.
Understand that free comes with no guarantees: your free job can be killed at anytime.
- Quota Enforcement
When users exceed their disk space or CPUs quota allocations the following will happen:
users will not be able to submit new jobs
running jobs will fail
Important
Please check the available disk space and CPU hours in your Slurm account regularly. Delete or archive data as needed.
- Inherited environment
All Slurm batch and interactive jobs will inherit all environment variables that were set in your login shell (on the login node where you are executing job submission).
By default, there should be none, unless one changes the environment via setting environment variables either on command line or in $HOME/.bashrc file or is using conda.
Check your $HOME/.bashrc and make sure that there are no environment variables set that can cause problems in the
sbatchorsruncommands. Follow the Bash init files guide.Load modules either in the Slurm submit script or on an interactive node after you executed the
sruncommand and not in the login shell before the job submission.If you are using conda please review Install with conda/mamba guide that explains how to cleanly separate conda-set environment from your login environment.
2.2. Batch Job
sbatch
A batch job is run at some time in the future by the scheduler.
Submitting batch jobs to Slurm is done with the sbatch command
and the job description is provided by the submit script.
- When to use:
when your job runs for hours or days.
- How to submit:
[user@login-x:~]$ sbatch your-submit-script.sub
- Example job:
Login on HPC3 and using your favorite editor create simple.sub file using the following contents as a template:
#!/bin/bash #SBATCH --job-name=test ## Name of the job. #SBATCH -A panteater_lab ## CHANGE account to charge #SBATCH -p free ## partition name #SBATCH --nodes=1 ## (-N) number of nodes to use #SBATCH --ntasks=1 ## (-n) number of tasks to launch #SBATCH --cpus-per-task=1 ## number of cores the job needs #SBATCH --error=slurm-%J.err ## error log file #SBATCH --output=slurm-%J.out ## output log file # Run command hostname and save output to the file out.txt hostname > out.txt
Edit the Slurm account to charge for the job to either your personal account or lab account. Your personal account is the same as your UCInetID.
Submit the job:
[user@login-x:~]$ sbatch simple.sub Submitted batch job 21877983
When the job has been submitted, Slurm returns a job ID (here 21877983) that will be used to reference the job in Slurm user log files and Slurm job reports. After the job is finished, there will be 3 files created by the job:
- slurm-21877983.err:
Slurm job error log file
- slurm-21877983.out:
Slurm job output log file
- out.txt:
Output file created by a specific command that was run in the job.
Note
Slurm error and log files are extremely useful especially to track progress and issues with the jobs.
2.3. Interactive job
srun
The command srun is used to submit an interactive job which runs in a shell terminal.
The job uses your console for for standard input/output/error.
Important
All interactive jobs are run on a single node, they can not be run on multiple nodes.
Interactive jobs are submitted for execution but do not bypass scheduler priority. If your job cannot run immediately, you will wait until Slurm can schedule your request.
- When to use:
when you want to test a short computation, compile software, or run an interactive Python or R session.
- When NOT to use:
when your job runs for many hours or days. Use
sbatchinstead.
The main difference between srun and sbatch:
Srun
Sbatch
Interactive and blocking
Batch processing and non-blocking
You type commands interactively
Your commands run unattended
Can be used to create job steps in submit scripts
Can do everything
sruncan and more.
- How to submit:
Get an interactive node
While on a login node, run the
sruncommand. When it is executed, the scheduler allocates available resource and starts an interactive shell on the available node. Your shell prompt will indicate a new hostname:[user@login-x:~]$ srun -c 2 -p free --pty /bin/bash -i srun: job 32654143 queued and waiting for resources srun: job 32654143 has been allocated resources [user@hpc3-y-z:~]$
Execute your interactive commands
[user@hpc3-y-z:~]$ module load python/3.10.2 [user@hpc3-y-z:~]$ myProgRun.py -arg1 someDir/ -d outputDir/ -f file.nii -scale > out
Once done with your work simply type at the prompt:
[user@hpc3-y-z:~]$ exit
- Example jobs:
The following is a list of examples of different requests for interactive jobs. The –pty /bin/bash -i options must be the last on a command line and should not be separated by other options:
[user@login-x:~]$ srun -A PI_LAB --pty /bin/bash -i # 1 [user@login-x:~]$ srun -p free --pty /bin/bash -i # 2 [user@login-x:~]$ srun --mem=8G -p free --pty /bin/bash -i # 3 [user@login-x:~]$ srun -c 4 --time=10:00:00 -N 1 --pty /bin/bash -i # 4 [user@login-x:~]$ srun -p free-gpu --gres=gpu:V100:1 --pty /bin/bash -i # 5 [user@login-x:~]$ srun -p free --x11 --pty /bin/bash -i* # 6
Use standard partition and charge to the PI_LAB account
Use free partition (where it may be killed at any time)
Use free partition and ask for 8GB of memory per job (ONLY when you truly need it)
Use standard partition and ask for 4 CPUs for 10 hrs
Use free-gpu partition and ask for one V100 GPU. Do not ask for more than 1 GPU! For many jobs the GPU specification can be simply –gres=gpu:1 (GPU type makes no difference for the job).
- Start an interactive session with Xforward enabled (option –x11) for GUI jobs.Note, a user should have logged on HPC3 with ssh Xforward enabled see Ssh and Xforward before running this
sruncommand.
2.4. Attach to a job
srun --pty --jobid
Attention
The ssh access to compute nodes is turned off
Users will need to use a job ID of a running job and to attach to it if they want to run simple jobs verification commands on the node where their job is running.
Once attached to a job, the user will be put on the node where the job is running and will run inside the cgroup (CPU, RAM etc.) of the running job. This means the user:
will be able to execute simple commands such as ls, top, ps.
will not be able to start new processes that use resources outside of what is specified in jobid. Any command will use computing resources, and will add to the usage of the job.
needs to type
exitafter executing desired verification commands in order to stop attachment to the job. The original job will be still running.
- Find jobid and attach to it:
[user@login-x:~]$ squeue -u panteater JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 3559123 free Tst41 panteater R 17:12:33 5 hpc3-14-02 3559124 free Tst42 panteater R 17:13:33 7 hpc3-14-17,hpc3-15-[05-08] [user@login-x:~]$ srun --pty --jobid 3559123 --overlap /bin/bash [user@hpc3-14-02:~]$
Execute your commands at the prompt and exit:
[user@hpc3-14-02:~]$ top [user@hpc3-14-02:~]$ exit [user@login-x:~]$
- Attach to a specific node using -w switch (for multi-node jobs):
By default, the user will be put on the first node listed in
squeueoutput if running on multi-node. To attach to a specific node:[user@login-x:~]$ srun --pty --jobid 3559124 --overlap -w hpc3-15-08 /bin/bash [user@hpc3-15-08:~]$
- Run command while attaching to the running job:
Most often users just need to see the processes of the job, etc. Such commands can be run directly. For example, to run
top:[user@login-x:~]$ srun --pty --overlap --jobid $JOBID top
2.5. Requesting Resources
2.5.1. Nodes
Very few applications that are compiled to run with Open MPI or MPICH need to use multiple nodes. Most jobs on HPC3 including all interactive jobs are single node jobs and must be run on a single node. If a single node job is submitted to multiple nodes it will either:
fail
misuse the resources. You will be charged for reserved and unused resources.
- How to request a single node:
Users should explicitly ask for 1 node. This is important to let SLURM know that all your processes should be on a single node and not spread over multiple nodes. In your submit script use:
#SBATCH --nodes=1 ## (-N) use 1 node- How to request multiple nodes:
Your submit script need to include desired number of nodes, for example:
#SBATCH --nodes=2 ## (-N) use 2 nodes
2.5.2. Features/Constraints
HPC3 has a heterogeneous hardware with several different CPU types. You can request that a job only runs on nodes with certain features which is done via a use of constraints.
To request a feature, you must add to your submit script:
#SBATCH --constraint=feature_name
where feature_name is one of the defined features specific to the cluster (or one of the standard features described in the Slurm sbatch guide). Multiple features can be requested and are separated by commas.
We defined the following for node selection:
HPC3-Specific Features
Feature name |
Node Description (processor/storage) |
Node count |
Cores min/mod/max |
|---|---|---|---|
intel |
any Intel node including HPC legacy |
compute: 215 GPU: 38 |
24 / 40 / 80 32 / 40 / 64 |
avx512 |
Intel AVX512 |
compute: 191 GPU: 38 |
28 / 40 / 48 32 / 40 / 64 |
epyc or amd |
any AMD EPYC |
18 |
40 / 64 / 64 |
epyc7551 |
AMD EPYC 7551 |
1 |
40 / 64 / 64 |
epyc7601 |
AMD EPYC 7601 |
17 |
64 / 64 / 64 |
nvme or fastscratch |
Intel AVX512 with /tmp on NVMe disk |
86 |
32 / 48 / 80 |
mlx5_ib |
Updated Infiniband firmware |
243 |
36 / 40 / 80 |
mlx4_ib |
Older Infiniband firmware |
9 |
24 / 40 / 64 |
To request nodes with updated InfiniBand firmware for your MPI-based jobs:
#SBATCH --constraint=mlx5_ib
To request nodes with a large local scratch storage:
#SBATCH --constraint=nvme or #SBATCH --constraint=fastscratchSee Scratch storage for details.
To request nodes with CPUs capable of AVX512 instructions:
#SBATCH --constraint=avx512
To request Intel nodes with CPUs capable of AVX512 instructions:
#SBATCH --constraint=intel,avx512
2.5.3. Scratch storage
Scratch storage is local to each compute node and is the fastest disk access for reading and writing the input/output job files.
Scratch storage is created for each job automatically as /tmp/UCInetID/jobid/ when the job starts on a compute node. Slurm knows this location and is referring to it via an environment variable $TMPDIR. Users don’t need to create $TMPDIR but simply need to use it in their submit scripts.
For example, a user panteater who has 2 running jobs:
[user@login-x:~]$ squeue -u panteater squeue JOBID PARTITION NAME USER ACCOUNT ST TIME CPUS NODE NODELIST(REASON) 20960254 standard test-001 panteater PI_lab R 1:41:12 25 1 hpc3-15-08 20889321 standard test-008 panteater PI_lab R 17:24:10 20 1 hpc3-15-08will have the following directories created by Slurm on hpc3-15-08
/tmp/panteater/20960254 /tmp/panteater/20889321Note
While the directory is created automatically, it is a user responsibility to copy:
input files to $TMPDIR before doing computational commands
the final results from $TMPDIR to user area before the job ends.
These copy commands need to be in the Slurm submit script.
Slurm doesn’t have any default amount of scratch space defined per job and that may be fine for most, but not all. The problem of having enough local scratch arises when nodes are shared by multiple jobs and users. One job can cause the other jobs running on the same node to fail, so please be considerate of your colleagues by requesting storage for your jobs as follows:
- Your job creates a few Gb of temporary data directly in $TMPDIR
and handles the automatic creation and deletion of these temp files. Many Python, Perl, R, Java programs and 3rd party commercial software will write to $TMPDIR which is the default for many applications.
- Your action:
You don’t need to do anything special. Do not reset $TMPDIR.
- Your job creates a few Gb of output in the current directory
where you run the job and does many frequent small file reads or writes (a few Kb every few minutes).
- Your action:
You will need to use a scratch storage where you bring your job data, write temp files and then copy the final output files back when the job is done.
Attention
In this scenario, Slurm job is run in $TMPDIR which is much faster for the disk I/O, then the program output is copied back as a big write which is much more efficient compare to many small writes.
The reason is parallel filesystem (CRSP or DFS) is not suitable for small writes and reads and such operations need to be off-loaded to the local scratch area on the node where the job is executed. Otherwise you create an I/O problem not just for yourself but for many others who use the same filesystem.
The following partial submit script shows how to use $TMPDIR for such jobs:
<the rest of submit script is omitted> #SBATCH --tmp=20G # requesting 20 GB (1 GB = 1,024 MB) local scratch # explicitly copy input files from DFS/CRSP to $TMPDIR # note, $TMPDIR is already created for your job by SLURM cd $TMPDIR cp /pub/myacount/path/to/my/jobs/data/*dbfiles $TMPDIR # create a directory for the application output mkdir -p $TMPDIR/output # your job commands, this is just one possible example # output from application goes to $TMPDIR/output/ mapp -tf 45 -o $TMPDIR/output # program output directory is specified via -o flag mapp2 > $TMPDIR/output/mapp.out # program output in a specific file # explicitly copy output files from $TMPDIR to DFS/CRSP cp $TMPDIR/output/* /pub/myaccount/myDesiredDir/
- Your job creates many Gbs of temporary data (order of ~100Gb)
- Your action:
You will need to submit your job to a node with a lot of local scratch storage where you bring your job data, write temp files, and then copy the final output files back when the job is done.
In your submit script define how much scratch space your job needs (you may need to figure it out by trial test run) and request the nodes that have fast local scratch area via the following Slurm directives:
#SBATCH --tmp=180G # requesting 180 GB (1 GB = 1,024 MB) local scratch #SBATCH --constraint=fastscratch # requesting nodes with a lot of space in /tmp
Follow the above submit script example to:
at job start explicitly copy input files from DFS/CRSP to $TMPDIR
at job end explicitly copy output files from $TMPDIR to DFS/CRSP
2.5.4. Memory
There are nodes with different memory footprints. Slurm uses Linux cgroups to enforce that applications do not use more memory/cores than they have been allocated.
Slurm has default and max settings for a memory allocation per core for each partition. Please see all partitions settings in Available CPU partitions.
- default settings:
Are used when a job submission script does not specify different memory allocation, and for most jobs this is sufficient.
- max settings:
Are used when a job requires more memory. Job memory specifications can not exceed the partition’s max setting. If a job specifies a memory per CPU limit that exceeds the system limit, the job’s count of CPUs per task will automatically be increased. This may result in the job failing due to CPU count limits.
Note
Please do not override the memory defaults unless your particular job really requires it. Analysis of more than 3 Million jobs on HPC3 indicated that more than 98% of jobs fit within the defaults. With slightly smaller memory footprints, the scheduler has MORE choices as to where to place jobs on the cluster, so your job has a better change to start sooner.
- How to request more memory:
You should specify the memory needs via one of the two mutually exclusive directives (one or another but not both):
Scenario 1:–mem=X<size> ask for more total memory for the jobScenario 2:–mem-per-cpu=X<size> ask for max memory per core and if this is not enoughrequest more cores.where X is an integer and <size> of an optional size specification (M - megabytes, G - gigabytes, T - terabytes). A default is in megabytes.
You will be charged more for more cores, but you use a larger fraction of the node. The same directives formats are used in Slurm submit scripts and for interactive jobs in any partition.
Examples of memory requests:
Ask for the total job memory in submit script
#SBATCH --mem=500 # requesting 500MB memory for the job #SBATCH --mem=4G # requesting 4GB (1GB = 1,024MB) for the jobAsk for the memory per CPU in submit script
#SBATCH --mem-per-cpu=5000 # requesting 5000MB memory per CPU #SBATCH --mem-per-cpu=2G # requesting 2GB memory per CPUAsk for 180GB for job in standard partition:
#SBATCH --partition=standard #SBATCH --mem-per-cpu=6G # requesting max memory per CPU #SBATCH --ntasks=30 # requesting 30 CPUsAsk for max memory per CPU and a number of CPUs to make up needed total memory for job as 30 x 6Gb = 180Gb
Use
srunand request 2 CPUs with a default or max memory[user@login-x:~]$ srun -p free --nodes=1 --ntasks=2 --pty /bin/bash -i [user@login-x:~]$ srun -p free --nodes=1 --ntasks=2 --mem-per-cpu=18G --pty /bin/bash -i [user@login-x:~]$ srun -p free --nodes=1 --ntasks=2 --mem=36G --pty /bin/bash -iThe first job will have a total memory 2 x 3Gb = 6GbThe second and third job each will have a total memory 2 x 18Gb = 36GbUse
srunand request 4 CPUs and 10GB memory per CPU,[user@login-x:~]$ srun -p free --nodes=1 --ntasks=4 --mem-per-cpu=10G --pty /bin/bash -itotal memory for job is 4 x 10Gb = 40Gb
Note
For information how to get an access to higher memory partitions please see Higher Memory
2.5.5. Runtime
Slurm has default and max settings for a runtime for each partition. Please see all partitions settings in Available CPU partitions.
Important
All interactive jobs submitted with srun command and
all batch jobs submitted with sbatch command
have time limits whether you explicitly set them or not.
- default settings:
are used when a job submission script or
sruncommand do not specify runtime, and for most jobs this is sufficient.- max settings:
specify the longest time a job can run in a given partition. Job time specifications can not exceed the partition’s max setting.
When a job requires longer run time than a default it needs to be specified using time directive –time=format (or the equivalent short notation -t format.
- Acceptable time formats:
minutes
days-hours
minutes:seconds
days-hours:minutes
hours:minutes:seconds
days-hours:minutes:seconds
For example, for Slurm script:
#SBATCH --time=5 # 5 minutes
#SBATCH -t 36:30:00 # 36 hrs and 30 min
#SBATCH -t 7-00:00:00 # 7 days
Similarly, for srun command:
srun --time=10 <other arguments> # 10 minutes
srun -t 15:00:00 <other arguments> # 15 hours
srun -t 5-00:00:00 <other arguments> # 5 days
- Runtime extension
If your job and was submitted for the max default time and you realize it will not finish by the specified runtime limit you can ask for a runtime extension. Please see change job time limit.
Note, there is no runtime extension for free jobs.
2.5.6. Mail notification
To receive email notification on the status of jobs, include the following lines in your submit scripts and make the appropriate modifications to the second line:
#SBATCH --mail-type=fail,end
#SBATCH --mail-user=user@domain.com
The first line specifies the event type for which a user requests an email (here failure and end events), the
second specifies a valid email address. We suggest to use a very few event
types especially if you submit hundreds of jobs. For more info, see output of man sbatch command.
Attention
Do not use mail event type ALL,BEGIN.
Do not enable email notification if you submit hundreds of jobs. Sending an email for each job overloads Postfix server.
Make sure to use your actual UCI-issued email address. While Slurm sends emails to any email address, system administrators will use UCInetID@uci.edu if they need to contact you about a job.
2.6. Monitoring
2.6.1. Status
squeuescontrol show job- To check the status of your job in the queue:
[user@login-x:~]$ squeue -u panteater JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 22877983 standard test panteater R 0:03 1 hpc3-17-11
Attention
Avoid using commandwatchto query the Slurm queue in a continuous loop as inwatch -d squeue <...some arguments...>This frequent querying of Slurm queue creates an unnecessary overhead and affects many users. Instead, check your job output and use Mail notification for the job end.
- To get detailed info about the job:
[user@login-x:~]$ scontrol show job 22877983
The output will contain a list of key=value pairs that provide job information.
2.6.2. Account balance
sbankzotledgerIn order to run jobs on HPC3, a user must have available CPU hours.
The
sbankis short for Slurm bank It is used to display the balance of used and available hours to the user for a given account (defaults to the current user).- Display the account balance for specific account:
[user@login-x:~]$ sbank balance statement -a panteater User Usage | Account Usage | Account Limit Available (CPU hrs) ---------- ------- + ----------- ------- + ------------- --------- panteater* 58 | PANTEATER 58 | 1,000 942
- Display the account balances for specific user:
[user@login-x:~]$ sbank balance statement -u panteater User Usage | Account Usage | Account Limit Available (CPU hrs) ---------- ------ + ----------- -------- + ------------- --------- panteater* 58 | PANTEATER 58 | 1,000 942 panteater* 6,898 | PI_LAB 6,898 | 100,000 93,102 panteater* 84 | PANTEATER_LAB_GPU 84 | 33,000 32,916
Note
An hour of a GPU requires at least 2 CPU cores. Hence, the minimum charge for a single GPU is (32 + 2) = 34 SUs/hour.
The
zotledgeris a cluster-specific tool to print a ledger of jobs based on specified arguments. To find all available arguments for this command usezotledger -h.Default is to print jobs of the current user for the last 30 days:
[user@login-x:~]$ zotledger -u panteater DATE USER ACCOUNT PARTITION JOBID JOBNAME ARRAYLEN CPUS WALLHOURS SUs 2021-07-21 panteater panteater standard 1740043 srun - 1 0.00 0.00 2021-07-21 panteater panteater standard 1740054 bash - 1 0.00 0.00 2021-08-03 panteater lab021 standard 1406123 srun - 1 0.05 0.05 2021-08-03 panteater lab021 standard 1406130 srun - 4 0.01 0.02 2021-08-03 panteater lab021 standard 1406131 srun - 4 0.01 0.02 TOTALS - - - - - - - 0.07 0.09
2.6.3. Efficiency
sacctseffseff-arraysstatThese are commands that provide info about resources consumed by the job.
- use for running jobs:
sstat- use for completed jobs:
sacct,seff,seff-array
All commands need to use a valid jobid.
The sstat displays various running job and job steps resource utilization information.
For example, to print out a job’s average CPU time use (avecpu), average number of bytes written by all tasks (AveDiskWrite), average number of bytes read by all tasks (AveDiskRead), as well as the total number of tasks (ntasks) execute:
[user@login-x:~]$ sstat -j 125610 --format=jobid,avecpu,aveDiskWrite,AveDiskRead,ntasks JobID AveCPU AveDiskWrite AveDiskRead NTasks ------------ ---------- ------------ ------------ -------- 125610.batch 10-18:11:+ 139983973691 153840335902 1
The sacct command can be used to see accounting data for all jobs and job steps and other useful info such how long job waited in the queue.
Find accounting info about a specific job:
[user@login-x:~]$ sacct -j 43223 JobID JobName Partition Account AllocCPUS State ExitCode ------------ -------- ---------- ------------ ---------- ---------- -------- 36811_374 array standard panteater_l+ 1 COMPLETED 0:0
The command uses a default output format.
- Find detailed accounting info a job using specific format:
[user@login-x:~]$ export SACCT_FORMAT="JobID,JobName,Partition,Elapsed,State,MaxRSS,AllocTRES%32" [user@login-x:~]$ sacct -j 600 JobID JobName Partition Elapsed State MaxRSS AllocTRES ---------- ------- -------- -------- --------- ------- -------------------------------- 600 all1 free-gpu 03:14:42 COMPLETED billing=2,cpu=2,gres/gpu=1,mem=+ 600.batch batch 03:14:42 COMPLETED 553856K cpu=2,mem=6000M,node=1 600.extern extern 03:14:42 COMPLETED 0 billing=2,cpu=2,gres/gpu=1,mem=+
MaxRSS: shows your job memory usage.
AllocTRES: is trackable resources, these are the resources allocated to the job after the job started running. The %32 is a format specification to reserve 32 characters for this option in the output. Format specification can be used for any option.
- Find how long your jobs were queued (column Planned) before they started running:
[user@login-x:~]$ export SACCT_FORMAT='JobID%20,Submit,Start,Elapsed,Planned' [user@login-x:~]$ sacct -j 30054126,30072212,30072182 -X JobID Submit Start Elapsed Planned ------------ ------------------- ------------------- ---------- ---------- 30054126 2024-07-14T11:17:00 2024-07-14T17:03:08 00:22:09 05:46:08 30072182 2024-07-14T20:29:30 2024-07-14T20:31:16 00:05:20 00:01:46 30072212 2024-07-14T20:44:14 2024-07-14T20:44:26 00:05:58 00:00:12
Note
Other useful options in SACCT_FORMAT are User, NodeList, ExitCode. To see all available options, run
man sacctcommand.The
seffSlurm efficiency script is used to find useful information about the job including the memory and CPU use and efficiency. Note,seffdoesn’t produce accurate results for multi-node jobs. Use this command for single node jobs.[user@login-x:~]$ seff -j 423438 Job ID: 423438 Cluster: hpc3 User/Group: panteater/panteater State: COMPLETED (exit code 0) Nodes: 1 Cores per node: 8 CPU Utilized: 00:37:34 CPU Efficiency: 12.21% of 05:07:36 core-walltime Job Wall-clock time: 00:38:27 Memory Utilized: 2.90 MB Memory Efficiency: 0.01% of 24.00 GB
Important info is on CPU and Memory lines.
- CPU efficiency:
at 12.21% the job used only a small portion of requested 8 CPUs
- Memory efficiency:
at 0.011% the job used only a fraction of requested 24GB of memory
The user should fix the job submit script and ask for less memory per CPU and for fewer CPUs.
The
seff-arraycommand is an extension ofseffto use for array jobs. It provides CPU and memory efficiency and wall-clock time for the array tasks and calculates their Min, Max, Mean and Standard Deviation.[user@login-x:~]$ seff-array 123456 Job information: ----------------------------- ID: 123456 Name: DataUpdater Cluster: hpc3 User/Group: panteater/panteater Requested CPUs: 1 cores on 1 node(s) Requested Memory: 3G Requested Time: 12-00:00:00 Job tasks status: ----------------------------- CANCELLED by 63168: 1 COMPLETED: 148 Finished job tasks stats: ----------------------------- Excluding pending/running/cancelled jobs tasks. StDev is calculated as 'population standard deviation'. -------------------------------------------------------------------------- Measure Min Max Mean StDev -------------------------------------------------------------------------- CPU efficiency (%) 0.67 24.20 1.18 1.91 Memory efficiency (%) 0.14 0.64 0.61 0.04 Wall-clock (d-hh:mm:ss) 00:01:27 2-04:06:32 1-013:12:16 10:06:39 --------------------------------------------------------------------------
2.7. Pending
Jobs submitted to Slurm will start up as soon as the scheduler can find an appropriate resource depending on the availability of the nodes, job priority and job requests.
Lack of resources or insufficient account balance (status reason is AssocGrpCPUMinutesLimit or AssocGrpBillingMinutes) are the most common reasons that prevent a job from starting.
RCIC does not generally put limits in place unless we see excess, unreasonable impact to shared resources (often, file systems), or other fairness issues.
Important
The balance in the account must have enough core hours to cover the job request.
This applies to all jobs submitted with
sbatchorsrun.This applies to all partitions, including free. While your job will not be charged when submitted to a free partition, there must be a sufficient balance for Slurm to begin your job.
When a job is in PD (pending) status you need to determine why.
2.7.1. Pending Job Reasons
While lack of resources or insufficient account balance are common reasons that prevent a job from starting, there are other possibilities. A job may be waiting for more than one reason.
To see all available job pending reasons and their definitions in the
Slurm Job Reason Codes guide
or see output of man squeue command in the JOB REASON CODES section.
- How to find job pending reason:
Run the
squeuecommand (with your login account after -u):[user@login-x:~]$ squeue -t PD -u peat JOBID PARTITION NAME USER ACCOUNT ST TIME CPUS NODE NODELIST(REASON) 92005 standard watA peat p_lab PD 0:00 1 1 (ReqNodeNotAvail,Reserved for maintenance) 92008 standard watA peat p_lab PD 0:00 1 1 (ReqNodeNotAvail,Reserved for maintenance) 92011 standard watA peat p_lab PD 0:00 1 1 (ReqNodeNotAvail,Reserved for maintenance) 95475 free-gpu 7sMD peat p_lab PD 0:00 2 1 (QOSMaxJobsPerUserLimit) 95476 free-gpu 7sMD peat p_lab PD 0:00 2 1 (QOSMaxJobsPerUserLimit)
- Most common reasons for job pending state and their explanations:
- Pending job due to AssocGrpCPUMinutesLimit
Same as AssocGrpBillingMinutes.
- Pending job due to AssocGrpBillingMinutes
This means insufficient funds are available to run the job to completion.
Slurm calculates MAX Time a job might consume as Number cores x Number hours from the cores and hours requested for the job. If the calculated value is less than the available account balance it means there is not enough balance to run the job.
Slurm will not start a new job if left MAX Time of current jobs plus MAX Time of queued jobs would cause the account to go negative.
Note
A user needs to check if there are any other jobs already running in the specified account and compute what is the time already requested and allocated by Slurm to all jobs on this account.
- Job in personal account
Check your jobs status
[user@login-x:~]$ squeue -u panteater JOBID PARTITION NAME USER ACCOUNT ST TIME NODES NODELIST(REASON) 6961 standard tst1 panteater panteater PD 0:00 1 (AssocGrpBillingMinutes) 6962 standard tst2 panteater panteater PD 0:00 1 (AssocGrpBillingMinutes)
The reason AssocGrpBillingMinutes is given for the personal account where the job was submitted.
Check your Slurm account balance
[user@login-x:~]$ sbank balance statement -u panteater User Usage | Account Usage | Account Limit Available (CPU hrs) ---------- ------ + ----------- ------- + ------------- --------- panteater* 58 | PANTEATER 58 | 1,000 942 panteater* 6,898 | PI_LAB 6,898 | 100,000 93,102
The account has 942 hours.
Check your job requirements
You can use
scontrol show job <jobid>or a command below[user@login-x:~]$ squeue -o "%i %u %j %C %T %L %R" -p standard -t PD -u panteater JOBID USER NAME CPUS STATE TIME_LEFT NODELIST(REASON) 66961 panteater tst1 16 PENDING 3-00:00:00 (AssocGrpBillingMinutes) 66962 panteater tst2 16 PENDING 3-00:00:00 (AssocGrpBillingMinutes)
Each jobs asks for 16 CPUs to run for 3 days which is \(16 * 24 * 3 = 1152\) core-hours, and it is more than 942 hours in the account balance.
Attention
These two jobs will never be scheduled to run and need to be canceled
- Job in LAB account
Important
A lab account has a combined single balance limit for all members of the lab.
Check your jobs status
[user@login-x:~]$ squeue -u panteater -t PD JOBID PARTITION NAME USER ACCOUNT ST TIME CPUS NODE NODELIST(REASON) 12341501 standard myjob_98 panteater PI_lab PD 0:00 1 1 (AssocGrpBillingMinutes) 12341502 standard myjob_99 panteater PI_lab PD 0:00 1 1 (AssocGrpBillingMinutes)
Check your Slurm lab account balance
[user@login-x:~]$ sbank balance statement -a PI_LAB User Usage | Account Usage | Account Limit Available (CPU hrs) ---------- ------- + ----------- -----+ ------------- --------- panteater1 0 | PI_LAB 75,800 | 225,000 67,300 panteater2 50,264 | PI_LAB 75,800 | 225,000 67,300 panteater* 25,301 | PI_LAB 75,800 | 225,000 67,300
Check your job requirements
[user@login-x:~]$ scontrol show job 12341501 JobI4=12341501 JobName=myjob_98 UserId=panteater(1234567) GroupId=panteater(1234567) MCS_label=N/A Priority=299 Nice=0 Account=PI_lab QOS=normal JobState=PENDING Reason=AssocGrpBillingMinutes Dependency=(null) Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=14-00:00:00 TimeMin=N/A SubmitTime=2023-01-18T16:36:06 EligibleTime=2023-01-18T16:36:06 AccrueTime=2023-01-18T16:36:06 StartTime=Unknown EndTime=Unknown Deadline=N/A NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=1,mem=6G,node=1,billing=1 <output cut>
There is a similar output is for the second job. Note the TimeLimit which is the time run requested for the job. For each of two pending jobs the resource request is: \(1 CPU * 14 days * 24 hrs = 336 hrs\)
Check ALL the running jobs for your lab account
[user@login-x:~]$ squeue -t R -A PI_lab -o "%.10i %.9P %.8j %.8u %.16a %.2t %.6C %l %L" JOBID PARTITION NAME USER ACCOUNT ST CPUS TIME_LIMIT TIME_LEFT 12341046 standard myjob_39 panteater PI_lab R 1 14-00:00:00 13-23:00:22 12341047 standard myjob_40 panteater PI_lab R 1 14-00:00:00 13-23:00:22 12341048 standard myjob_41 panteater PI_lab R 1 14-00:00:00 13-23:00:22 < total 200 running jobs >
Each of 200 running jobs in the account has run for about 1hr out of allocated 14 days. Total max time Slurm has allocated for these running jobs is\(1 CPU * 200 jobs * 14 days * 24 hrs = 67200 hrs\)There are about 200 hrs already used, (each job already run for ~1 hr), so remaining needed balance is 67100 hrs. Your 2 pending jobs require\(1 CPU * 14 days * 24 hrs * 2 jobs = 672 hrs\).Slurm is computing that if all current jobs ran to their MAX times and if the next job were to run MAX time your account would end up negative:\(67300 - 67100 - 672 = -472 hrs\).Therefore Slurm puts these new jobs on hold. These 2 jobs will start running once some of the remaining running jobs completed and the account balance is sufficient.
Important
It is important to correctly estimate time needed for the job, and not ask for more resources (time, cpu, memory) than needed.
- Pending job due to Dependency
Job has a user-defined dependency on a running job and cannot start until this running job has completed.
- Pending due to DependencyNeverSatisfied
Job has a user-defined dependency that failed. Job will never run and needs to be canceled.
- Pending due to JobArrayTaskLimit
The user has reached job array’s limit on the number of simultaneously running tasks.
- Pending due to Priority
Slurm scheduler is temporarily holding the job in pending state because other queued jobs have a higher priority.
- Pending due to MaxGRESPerAccount
The job’s GRES request exceeds the per-Account limit. We have limits how many GPUs can be allocated to a single user.
- Pending due to QOSMaxJobsPerUserLimit
The user is already running the maximum number of jobs allowed by the particular partition.
- Pending job due to ReqNodeNotAvail, Reserved for maintenance
This means if your job is started now it will not complete by the time the scheduled maintenance starts. Slurm is holding your job because no jobs can run during the maintenance period. You either have to wait or you need to change your job requirements. See fix pending jobs. Job will not start until maintenance has been completed.
- Pending job due to Resources
This means the requested resource configuration is not currently available. If a job requests a resource combination that physically does not exist, the job will remain in this state forever.
2.7.2. Fix Pending Job
Fixes apply for batch jobs submitted with
sbatchor for interactive jobs submitted withsrun.
- Cancel the job
For many of the job pending reasons the first step is to cancel your job:
[user@login-x:~]$ scancel <jobid>Fix pending job due to
- AssocGrpBillingMinuteso or AssocGrpCPUMinutesLimit
First, cancel existing pending job (it will never run). Next, resubmit the job so that the requested execution hours can be covered by your bank account balance. Check and update the following in your submit script:
If your job was run from personal account
SBATCH -A - use a different Slurm account (lab) where you have enough balance
Lower requirements of your job so that requested resources will be no more than core hours available in your account. This may mean to use:
SBATCH –ntasks or #SBATCH –cpus-per-task - fewer CPUs
SBATCH –ntasks and #SBATCH –mem-per-cpu - fewer CPUs, increase memory per CPU
SBATCH –mem or #SBATCH –mem-per-cpu - less memory
SBATCH –time - set a time limit shorter than the default runtime
- ReqNodeNotAvail, Reserved for maintenance
You need to cancel your job and re-submit it job with a shorter time limit that will end BEFORE the maintenance begins.
If you did not specify time limit, the default time setting is in effect. Please see Available CPU partitions for partitions default and max settings and Job examples for additional info.
To find out the reservation details use:
[user@login-x:~]$ scontrol show reservation ReservationName=RCIC: HPC3 scheduled maintenance StartTime=2024-03-27T08:00:00 EndTime=2024-03-28T08:00:00 Duration=1-00:00:00 Nodes=hpc3-14-[00-31],... NodeCnt=228 CoreCnt=9936 Features=(null) PartitionName=(null) Flags=MAINT,IGNORE_JOBS,SPEC_NODES,ALL_NODES TRES=cpu=9936 Users=root,... Groups=(null) Accounts=(null) Licenses=(null) State=INACTIVE BurstBuffer=(null) Watts=n/a MaxStartDelay=(null)The first output line includes the maintenance start time, end time and duration.
Based on the info about the reservation time and the current day/time you can estimate what time limit SBATCH –time should be specified for your job in order for it to finish before the maintenance starts.
If your job truly needs requested time limit, nothing can be done until the maintenance is over. Cancel your job and resubmit after the maintenance.
- Resources
Check Slurm estimate for the job start time:
[user@login-x:~]$ squeue --start -j 32511 JOBID PARTITION NAME USER ST START_TIME NODES SCHEDNODES NODELIST(REASON) 32511 free GEN panteater PD 2024-08-15T13:36:57 1 hpc3-14-00 (Resources)The estimated time start is listed under START_TIME.You either have to wait or you need to cancel your job, change its requirements and resubmit.- JobArrayTaskLimit
You need to wait till some already running tasks complete. Scheduler will automatically start the next task.
- MaxGRESPerAccount
You need to wait till some of your already running jobs requesting GPUs complete. There is a limit how many GPUs can be used by a single user or a single GPU account.
- Dependency
You need to wait till the job dependency on another already running job is fulfilled.
- DependencyNeverSatisfied
You need to cancel your job because its dependency job was canceled or failed. This job will never run and only takes scheduler resources.
- QOSMaxJobsPerUserLimit
You need to wait till some of your already running jobs in this partition complete.
- Priority
The scheduler will automatically start your job when the higher priorities jobs are run and the resources become available.
2.8. Modification
scontrol
This command can be used to make some changes to jobs that are still waiting to run.
If changes need to be made for a running job, it may be better to kill the job and restart it after making the necessary changes.
- Change QOS:
By default, jobs are set to run with qos=normal. Users rarely need to change QOS.
[user@login-x:~]$ scontrol update jobid=<jobid> qos=[low|normal|high]
- Change job time limit:
Only the Slurm administrator can increase job’s time limit. If your job is already running and you have established that it will not finish by its current time limit you can submit a ticket indicating:
your JOBID
your desired time extension
Note, we need to receive your request before your job’s current end time and your bank account must have sufficient funds to cover the desired time extension.
2.9. Cancel/Hold/Release
scancelscontrolThe following commands can be used to:
- Cancel a specific job:
[user@login-x:~]$ scancel <jobid>
- Cancel all jobs owned by a user:
This only applies to jobs that are associated with your accounts
[user@login-x:~]$ scancel -u <username>
- Prevent a pending job from starting:
[user@login-x:~]$ scontrol hold <jobid>
- Release held jobs to run:
[user@login-x:~]$ scontrol release <jobid>