The Office of Research Computing treats each faculty member as the head of a research group. Each research group has a Slurm "account" that all members of the research group are a part of, including the faculty member. Each faculty member is set as an "account coordinator" for that account.

A Slurm account coordinator is a user who can modify settings for users in that Slurm account. The coordinator can also hold and cancel jobs submitted by other users in the account.

Manage user jobs

List jobs

squeue -A prof73   # by account
squeue -u fav_student   # by user

Cancel jobs

scancel 1234  # by job ID
scancel -u peon1   # kill all jobs by user
scancel -u peon1 --state=running  # kill running jobs by user
scancel -u peon1 --state=pending  # kill pending jobs by user
scancel -A prof73   # kill all jobs in the account

Hold/release jobs

scontrol hold 1234   # by job ID
scontrol release 1234   # remove the hold
scontrol uhold 1234   # hold job 1234 but allow the job's owner to release it

Per-user limits

The Easy Way

We created a web-based tool to show current limits. The page generates a list of commands that you can run to modify the limits. It is available under My Account->Slurm Limits.

Check current limits

sacctmgr list assoc account=prof73
sacctmgr list assoc user=peon1 format=account,user,grpcpurunmins
sacctmgr list assoc user=fav_student

Limit on the cputime of a user's running jobs

This can be a little complicated to understand, so it deserves a blog post of its own. It's likely worth it, so please read that and play with this GrpCPURunMins simulator.

This limit can help you if you want to limit a user's usage but not set a hard cap on the number of cores. This limit helps to stagger the start times of a user's jobs to that the start and, consequently, end times of the jobs are staggered.

If you're not scared off, this is how you set it:

sacctmgr modify user peon1 set GrpCPURunMins=172800 #try the simulator with: 72 hours, 1 core, and 172800. Example graph below

Limit on the number of processor cores per user

You can limit the amount of processors that a particular user in your group can have allocated at any moment in time. There are several other parameters you can set, but we recommend opening a support ticket to ask about those settings before trying them out.

sacctmgr modify user peon1 set GrpCPUs=64 #peon1 can only have 64 CPUs allocated at a time

The GrpCPUs parameter should not be confused with MaxCPUs, a parameter that controls the maximum cores per job

Clear a limit

Set a value to -1 to clear it.

Commands to check current settings/status

Account limits and fairshare settings

sacctmgr list assoc account=prof73

Fairshare and historical usage information for the account

Most of the fields in the command shown below do factor in the effects of "half-life decay" such that older usage information slowly disappears from the calculation.

sshare -a -l -A prof73

Pending jobs sorted by priority

squeue -o '%.7i %.9Q %.9P %.8j %.8u %.8T %.10M %.11l %.8D %.5C %R' -S '-p' --state=pending | less

Adjust the priority of individual users

Job priority

Job priority is made up of several components, also known as factors. Job size, queue time, and fairshare are some of the factors available in Slurm. Each factor has an integer weight that is multiplied by a decimal value (0.0 to 1.0) representing what proportion of the weight it should receive. The results for each factor are summed up to calculate the job priority.

For example, let's say that the weights are JobSize=100, Fairshare=1000. You submit a job that, based on the number of cores requested, received a value 0.2 and your fairshare factor is 0.8. Your job's priority will be 100*0.2 + 1000*0.8 = 820.

Fairshare Factor

The Office of Research Computing uses the Fair Tree plugin to calculate fairshare. This plugin was authored by the Office of Research Computing specifically for our needs, has been contributed to Slurm for general use, and is now the standard fairshare implementation. Please see the documentation for an explanation of how the algorithm works or the original presentation if you want a lot more information.

Fairshare factor is calculated based on two key components: usage and shares. Fairshare is essentially just Shares / Usage.

Usage is based on the cputime allocated to prior jobs. The usage decays over time such that recent usage is weighted more heavily.

Shares are similar to a slice of a pie and represent the amount of the system that the scheduler should try to dedicate to a particular user. Within an account, the size of the "pie" is the sum of the shares allocated to each of its users. Each user's slice of the pie is shares_self / sum(shares_self_plus_siblings). The Office of Research Computing treats each account as equals; each account has the same Shares value as other accounts. Within each account users can have different amounts of shares, determined at the whim of the account coordinator.

As usage increases, fairshare factor decreases. As Shares increases relative to sibling users, fairshare factor increases relative to sibling users.

Fairshare Calculations

Within the account, the fairshare calculations are referred to as Level Fairshare and are visible using sshare -laA youraccount. The equation is:

Level FS = Norm Shares / Effectv Usage

where:

Norm Shares = Raw Shares / sum(self + siblings' Raw Shares)
Effectv Usage = Raw Usage / account's Raw Usage

Raw Shares is the Shares value mentioned above. From sacctmgr, it is the "fairshare" or "share" field (synonyms for each other).

Modifying user fairshare

The Easy Way

We created a web-based tool to show current fairshare settings. The page generates a list of commands that you can run to modify shares. It is available under My Account->Slurm Shares.

Commands to set Shares

All new users are created with fairshare=1024. Account coordinators can adjust the fairshare shares of users in their account. This number is relative to other users in that account. (Note: You don't have to stick with powers of 2.)

sacctmgr modify user peon1 set fairshare=128 #set to be 1/8 of future new users in the account
sacctmgr modify user prof73 set fairshare=8192 #set to be 8x new user fairshare
sacctmgr modify user fav_student set fairshare=4096 #set to be 4x new user fairshare but 1/2 of prof73

Note: Fairshare is only calculated every few minutes; changes in job priority won't appear immediately.

This makes it so that in a battle of priorities (ignoring the effect of historical usage) the users would have fairshare shares in the following order (descending size):

• prof73
• fav_student
• new users who don't have their fairshare manually adjusted
• peon1

Historical usage will affect the ordering of the users when it comes to fairshare factor. If fav_student's usage accounted for 95% of the account's usage within the last few weeks, fav_student will receive a lower fairshare priority than new users but will likely still be higher than peon1. Depending on the circumstances, peon1 may never get to run if other users from the same account have pending jobs. With only those three users existing, peon1 has a normalized shares value of 128 / (128+8192+4096) = 0.01 compared to fav_student's 0.33.

Also note that fairshare only matters if there are multiple users in the queue. If resources are available, users can run regardless of their overall job priority.

QOS Access

Some faculty own dedicated hardware and have a QOS set up to manage access. If this applies to you, this is how you add and remove people's access to the QOS.

Grant access to just one user:

sacctmgr modify user someusername2 set qos+=myspecialqos

Revoke access from just one user:

sacctmgr modify user someusername2 set qos-=myspecialqos

List access for all users in the account, including the account itself:

sacctmgr list assoc account=someprof4 format=account,user,qos%100

The list contains a blank username that refers to the account itself. If a QOS is listed for that account, all users in the account will have access to the QOS by default unless access is revoked for a particular user. Note that there a number of QOSs that are available for various things that manage access to other hardware and provide other features. This is normal and should not be adjusted.

The Office of Research Computing can also set it so that all users in your account have access to the QOS. Open a support ticket if this is what you would like to do.

Web tools to manage Slurm users

In case you missed the links in all the text above:

• Slurm Shares
• Slurm Limits
• GrpCPURunMins simulator