Slurm Account Coordinator
The Office of Research Computing treats each faculty member as the head of a research group. Each research group has a Slurm "account" that all members of the research group are a part of, including the faculty member. Each faculty member is set as an "account coordinator" for that account.
A Slurm account coordinator is a user who can modify settings for users in that Slurm account. The coordinator can also hold and cancel jobs submitted by other users in the account.
Manage user jobs
squeue -A prof73 # by account squeue -u fav_student # by user
scancel 1234 # by job ID scancel -u peon1 # kill all jobs by user scancel -u peon1 --state=running # kill running jobs by user scancel -u peon1 --state=pending # kill pending jobs by user scancel -A prof73 # kill all jobs in the account
scontrol hold 1234 # by job ID scontrol release 1234 # remove the hold scontrol uhold 1234 # hold job 1234 but allow the job's owner to release it
The Easy Way
We created a web-based tool to show current limits. The page generates a list of commands that you can run to modify the limits. It is available under My Account->Slurm Limits.
Check current limits
sacctmgr list assoc account=prof73 sacctmgr list assoc user=peon1 format=account,user,grpcpurunmins sacctmgr list assoc user=fav_student
Limit on the cputime of a user's running jobs
This limit can help you if you want to limit a user's usage but not set a hard cap on the number of cores. This limit helps to stagger the start times of a user's jobs to that the start and, consequently, end times of the jobs are staggered.
If you're not scared off, this is how you set it:
sacctmgr modify user peon1 set GrpCPURunMins=172800 #try the simulator with: 72 hours, 1 core, and 172800. Example graph below
Limit on the number of processor cores per user
You can limit the amount of processors that a particular user in your group can have allocated at any moment in time. There are several other parameters you can set, but we recommend opening a support ticket to ask about those settings before trying them out.
sacctmgr modify user peon1 set GrpCPUs=64 #peon1 can only have 64 CPUs allocated at a time
The GrpCPUs parameter should not be confused with MaxCPUs, a parameter that controls the maximum cores per job
Clear a limit
Set a value to -1 to clear it.
Commands to check current settings/status
Account limits and fairshare settings
sacctmgr list assoc account=prof73
Fairshare and historical usage information for the account
Most of the fields in the command shown below do factor in the effects of "half-life decay" such that older usage information slowly disappears from the calculation.
sshare -a -l -A prof73
Pending jobs sorted by priority
squeue -o '%.7i %.9Q %.9P %.8j %.8u %.8T %.10M %.11l %.8D %.5C %R' -S '-p' --state=pending | less
Adjust the priority of individual users
Job priority is made up of several components, also known as factors. Job size, queue time, and fairshare are some of the factors available in Slurm. Each factor has an integer weight that is multiplied by a decimal value (0.0 to 1.0) representing what proportion of the weight it should receive. The results for each factor are summed up to calculate the job priority.
For example, let's say that the weights are JobSize=100, Fairshare=1000. You submit a job that, based on the number of cores requested, received a value 0.2 and your fairshare factor is 0.8. Your job's priority will be 100*0.2 + 1000*0.8 = 820.
The Office of Research Computing uses the Fair Tree plugin to calculate fairshare. This plugin was authored by the Office of Research Computing specifically for our needs, has been contributed to Slurm for general use, and is now the standard fairshare implementation. Please see the documentation for an explanation of how the algorithm works or the original presentation if you want a lot more information.
Fairshare factor is calculated based on two key components: usage and shares. Fairshare is essentially just Shares / Usage.
Usage is based on the cputime allocated to prior jobs. The usage decays over time such that recent usage is weighted more heavily.
Shares are similar to a slice of a pie and represent the amount of the system that the scheduler should try to dedicate to a particular user. Within an account, the size of the "pie" is the sum of the shares allocated to each of its users. Each user's slice of the pie is shares_self / sum(shares_self_plus_siblings). The Office of Research Computing treats each account as equals; each account has the same Shares value as other accounts. Within each account users can have different amounts of shares, determined at the whim of the account coordinator.
As usage increases, fairshare factor decreases. As Shares increases relative to sibling users, fairshare factor increases relative to sibling users.
Within the account, the fairshare calculations are referred to as Level Fairshare and are visible using sshare -laA youraccount. The equation is:
Level FS = Norm Shares / Effectv Usage
Norm Shares = Raw Shares / sum(self + siblings' Raw Shares) Effectv Usage = Raw Usage / account's Raw Usage
Raw Shares is the Shares value mentioned above. From sacctmgr, it is the "fairshare" or "share" field (synonyms for each other).
Modifying user fairshare
The Easy Way
We created a web-based tool to show current fairshare settings. The page generates a list of commands that you can run to modify shares. It is available under My Account->Slurm Shares.
Commands to set Shares
All new users are created with fairshare=1024. Account coordinators can adjust the fairshare shares of users in their account. This number is relative to other users in that account. (Note: You don't have to stick with powers of 2.)
sacctmgr modify user peon1 set fairshare=128 #set to be 1/8 of future new users in the account sacctmgr modify user prof73 set fairshare=8192 #set to be 8x new user fairshare sacctmgr modify user fav_student set fairshare=4096 #set to be 4x new user fairshare but 1/2 of prof73
Note: Fairshare is only calculated every few minutes; changes in job priority won't appear immediately.
This makes it so that in a battle of priorities (ignoring the effect of historical usage) the users would have fairshare shares in the following order (descending size):
- new users who don't have their fairshare manually adjusted
Historical usage will affect the ordering of the users when it comes to fairshare factor. If fav_student's usage accounted for 95% of the account's usage within the last few weeks, fav_student will receive a lower fairshare priority than new users but will likely still be higher than peon1. Depending on the circumstances, peon1 may never get to run if other users from the same account have pending jobs. With only those three users existing, peon1 has a normalized shares value of 128 / (128+8192+4096) = 0.01 compared to fav_student's 0.33.
Also note that fairshare only matters if there are multiple users in the queue. If resources are available, users can run regardless of their overall job priority.
Some faculty own dedicated hardware and have a QOS set up to manage access. If this applies to you, this is how you add and remove people's access to the QOS.
Grant access to just one user:
sacctmgr modify user someusername2 set qos+=myspecialqos
Revoke access from just one user:
sacctmgr modify user someusername2 set qos-=myspecialqos
List access for all users in the account, including the account itself:
sacctmgr list assoc account=someprof4 format=account,user,qos%100
The list contains a blank username that refers to the account itself. If a QOS is listed for that account, all users in the account will have access to the QOS by default unless access is revoked for a particular user. Note that there a number of QOSs that are available for various things that manage access to other hardware and provide other features. This is normal and should not be adjusted.
The Office of Research Computing can also set it so that all users in your account have access to the QOS. Open a support ticket if this is what you would like to do.
Web tools to manage Slurm users
In case you missed the links in all the text above: