Tasks for Account Coordinators

1. Account Access Control

PIs have control over who can charge to their account and how much they can charge

Slurm uses account coordinators as users who can directly control accounts. We highly encourage PIs to assign additional coordinators (typically postdocs, grad students) to manage access to their Slurm lab accounts.

The following common tasks can be run by account coordinators.

1.1. Add/Remove Users

  1. To add a user to your lab’s Slurm account do:

    [user@login-x:~]$ sacctmgr add user <username> account=<account_name>
  2. To remove a user from your lab’s Slurm charge account do:

    [user@login-x:~]$ sacctmgr remove user where user=<username> account=<account_name>

1.2. Limit Users

  1. Check current limits

    1. by account:

      [user@login-x:~]$ sacctmgr list assoc account=panteater_lab
    2. by user:

      [user@login-x:~]$ sacctmgr list assoc user=<username>
  2. Update limits

    1. limit user’s charge limit on group account to 100k hours (6M minutes)

      [user@login-x:~]$ sacctmgr modify user where user=<username> account=<account> \
                        set GrpTRESMins=cpu=6000000
    2. limit user’s CPU time on running jobs

      [user@login-x:~]$ sacctmgr modify user <username> set GrpCPURunMins=10000
    3. limit a specific user to have no more than 20 jobs in the system:

      [user@login-x:~]$ sacctmgr modify user where account=<account> \
                        name=<username> set maxjobs=20
    4. limit number of cores per user to 40 CPUs at a time:

      [user@login-x:~]$ sacctmgr modify user <username> set GrpCPUs=40

      To clear a limit, set its value to -1.

1.3. Manage User Jobs

  1. List jobs

    1. by charge account:

      [user@login-x:~]$ squeue -A <panteater_lab>
    2. by user:

      [user@login-x:~]$ squeue -u <username>
  2. Cancel jobs

    1. by job id:

      [user@login-x:~]$ scancel <jobid>
    2. all jobs for a specific user:

      [user@login-x:~]$ scancel -u <username>
    3. all running jobs for a specific user:

      [user@login-x:~]$ scancel -u <username> --state=running
    4. all pending jobs for a specific user:

      [user@login-x:~]$ scancel -u <username> --state=pending
    5. all jobs in the account:

      [user@login-x:~]$ scancel -A <slurm_account>
  3. Hold jobs

    1. by jobid:

      [user@login-x:~]$ scontrol hold <jobid>
    2. hold the job but allow its owner to release it:

      [user@login-x:~]$ scontrol uhold <jobid>
    3. remove the job hold:

      [user@login-x:~]$ scontrol release <jobid>

1.4. Modify User Priority/Fairshare

  1. Display historical fairshare usage for account:

    [user@login-x:~]$ sshare -a -l -A panteater_lab
  2. Display pending jobs sorted by priority:

    [user@login-x:~]$ squeue -S '-p' --state=pending | less