Slurm uses the term partition to signify a batch queue of resources. HPC3 has different kinds of hardware, memory footprints, and nodes with GPUs. Jobs running in some queues will charge core-hours (or GPU-hours) to the account.

Memory is allocated per-CPU core. When you request more cores, your job is allocated more memory.

¹ You must be added to a specific group to access the highmem / hugemem/ maxmem partitions. If you are not a member of these groups then you will not be able to submit jobs to these partitions and sinfo will not show these partitions.

² The maxmem partition is a single 1.5 TB node and that is reserved for those rare applications that really require that much memory. You can only be allocated the entire node. No free jobs run in this partition.

³ You must have a gpu account and you must specify it in order to submit to the gpu/gpu-debug partitions. This is because of differential charging. GPU accounts are not automatically given to everyone, your faculty adviser can request a GPU lab account.

⁴ Anyone can run jobs in free-gpu partition without special account.

Every HPC3 user is granted 1,000 free CPU hours as a startup allowance. This allocation is there for you to become familiar with HPC3, Slurm scheduler, and accounting.

Each CPU core-hour (allocation unit) is charged to the account you specify (or your default account which is your user account). Similarly, each GPU hour is charged to a gpu-enabled account at the rate of 32/GPU-hour plus the core-hour charge of the cpu cores the job utilizes (at least one is required). A GPU-enabled account can only be used on GPU-nodes.

Most jobs ran on HPC3 are charged to a lab account because most HPC3 users are part of at least one research lab. If a user or a research lab runs out of CPU hours, more CPU hours can be purchased via recharge.

Any UCI Professor can request an HPC3 lab account on behalf of their research group and add any number of researchers/students to this account. Based upon the number of requests and the number of nodes that have been purchased by RCIC, this number will vary. The aspirational goal is faculty who request an account will be granted 200,000 CPU hours per fiscal year.

You may request your Slurm lab account by emailing hpc-request@uci.edu. In the email, please specify

We provide the numerous EXAMPLES that show in depth how to run array jobs, request GPUs, CPUs, and memory for a variety of different job types and common applications. The scripts can be downloaded from this directory.

There are a few methods to submit your jobs to Slurm: batch, interactive, and running jobs immediately. The sections below show a few common submission details.

A batch job is run at sometime in the future by the scheduler. Submitting batch jobs to Slurm is accomplished through the sbatch command and the job description is provided by the submit script.

An example job description is shown below:

simplejob.sub

To submit a job on HPC3, login and using your favorite editor create simplejob.sub file with the contents shown above.

To submit the job do:

When the job has been submitted, Slurm returns a job ID that will be used to reference the job in Slurm user log files and Slurm job reports. After the job is finished look at the file out.txt to see the name of the compute node that ran the job.

To check the status of your job in the queue:

To get detailed info about the job:

The output will contain a list of key=value pairs that provide job information.

This frequent querying of Slurm queue creates an unnecessary overhead and affects many users. Instead, check your job output and use mail notification for the job end or use the squeue command when you want to see an update.

The command srun is used to run a job immediately and uses your console for stdin / stdout / stderr (standard input/output/error) instead of redirecting them to a file.

Srun submits jobs for immediate execution but it does not bypass scheduler priority. If your job cannot run immediately, you will wait until Slurm can schedule your request.

The main difference between srun and sbatch:

To request an interactive shell, use the salloc command. For example, a user panteater can use one of the following to submit a job to a standard partition:

A simpler way to run an interactive is to use srun. For example:

Once the salloc or srun command is executed, the scheduler allocates available resource and starts an interactive shell on the chosen node. Your shell prompt will indicate a new hostname.

Once done with your work simply type at the prompt:

To run an interactive session for GUI jobs a user must login with Xforwarding enabled in ssh, see Reference guide and then use the --x11 to enable Xforwarding in srun command.

The ssh access to compute nodes is turned off

Users will need to use Slurm commands to attach to running jobs if they want to run simple jobs verification commands on the node where their job is running.

To receive email notification on the status of jobs, include the following lines in your submit scripts and make the appropriate modifications to the second line:

The first line specifies the event type for which a user requests an email (failure/exception events), the second specifies a valid email address. We suggest to use a very few event types especially if you submit hundreds of jobs. See output of man sbatch command for more info.

HPC3 is heterogeneous hardware with several different CPU types available. In Slurm you can request that a job only run on nodes with certain "features". We add features to assist users who have discussed their specific needs. Using a constraint is straightforward in Slurm. For example, to run on only nodes that support the AVX512 instruction set, one would add to the submission:

We have defined the following features for node selection:

We have a clsuter-specifc tool to print a ledger of jobs based on specified arguments. Default is to print jobs of the current user for the last 30 days:

To find all available arguments use:

sacct can be used to see accounting data for all jobs and job steps. An example below shows how to use job ID for the command:

The above command uses a default output format. A more useful example will set a specific format for sacct that provides extra information:

Slurm efficiency script seff can be used after the job completes, to find useful information about the job including the memory and CPU use and efficiency.

sstat displays various running job and job steps resource utilization information. For example, to print out a job’s average CPU time use (avecpu), average number of bytes written by all tasks (AveDiskWrite), average number of bytes read by all tasks (AveDiskRead), as well as the total number of tasks (ntasks) execute:

sbank is short for "Slurm Bank". Sbank is used to display HPC3 user account information. In order to run jobs on HPC3, a user must have available CPU hours. To check how many CPU hours are available in your personal account, run the command with your account name:

To check how many CPU hours are available in all accounts that you have access to and how much you used:

Once you submit your job it should start running depending on the availability of the nodes, job priority and job resources. However, sometimes job is in PD (pending) status for a long time. Here is how to determine why.