Skip to main content

Running Jobs

How to use Slurm to submit a job

note

Please note that the job submission procedure is evolving and the directions below may change over time. Please reach out if you run into any issues.

Slurm Configuration​

Version: v4 - enabled 2026-10-14​

Guiding principles​

  • Iterative This will be versioned and improved over time
  • Maximized Utilization We aim for maximum cluster utilization and high-impact outcomes, guided by the governance committee
  • Minimal Friction We want to minimize wait times and avoid unnecessary barriers
  • Feedback-Driven We know we won't get it right the first time—we need your input
  • User Experience We want everyone to have a good experience, though we understand not everyone will agree with every configuration choice

Current configuration​

FairShare Scheduling​

Scheduling priority is governed by your FairShare score, calculated by Slurm based on:

  • QoS levels
  • Resources requested
  • Time requested
  • Past utilization
  • Decay factors

Ledger System​

As part of our ongoing efforts to enable equitable access to VISION, we have implemented a ledger system. The ledger system tracks usage over time and adjusts the FairShare cap or maximum score based on historical usage proportionate to maximum system availability. The FairShare score will not fall below half of the base value. When a user is onboarded, their FairShare value is set to 1000. The ledger calculates usage from 2026-10-14 onwards.

Project Tiers​

Tier One​

  • Startup tier - grants access only to a dedicated startup QoS level
  • Limited to a max of 40 GPU's across all running jobs
  • Limited to 970 total GPU hours across all jobs. Once this limit is hit (or 6 months after project creation), your FairShare drops to 0 and you will be unable to submit any new jobs. If you want to continue, you will need to talk with the Allocation Committee to be moved to Tier Two.

Tier Two​

  • Eligibilty for Tier Two requires proven ability to scale to the 40 GPU max of Tier One
  • Normal operating tier - grants access to currently defined QoS levels except heroic and startup
  • Limited to a max of 240 GPU's per job
  • No maximum limit on total GPU hours.
  • All projects will be reviewed by the Allocation Committee every 6 months beginning from October 1, 2026. Review will lead to either the continuation of the project through the end date or reassigning the project to Tier One.

Tier Three​

  • Heroic job eligibility tier - grants access to currently defined QoS levels and upon approval, access to the heroic QoS
  • Running a heroic job will impact your FairShare since you will consume a large number of GPU hours at once. This will have the effect of deprioritizing your future jobs for some period of time.
  • Limited to a max of 720 GPU's per job
  • All heroic jobs will be reviewed and approved by the Allocation Committee
  • Eligibility will require proving the ability to actually scale beyond the 240 GPU max for Tier Two
  • Heroic jobs will be run immediately after the maintenance window each month

Understanding how your FairShare value is calculated​

We have developed a utility that will show you how your FairShare score is being calculated. Please run the command my-fairshare once logged into a VISION login node to get detailed information on how your score is calculated. See below for sample redacted output.

$ my-fairshare --brief

================================================================================
 How your fair-share priority is computed
 user <redacted-user>   project <redacted-project>   <date>
================================================================================

You belong to 2 projects: <redacted-project-1> <redacted-project-2>
Showing <redacted-project-1>. Use -A <project> to see another.

Your FairShare factor is 0.450512.
It earns you 4505 of 10000 possible fair-share priority points.
You are ranked 132 of 293 users, where 293 is the highest priority
and 1 is the lowest.

Run without --brief to see the full derivation.
note

Run the script with no flags after it (i.e. my-fairshare) to see detailed output

QoS Recommendations​

QoSQoS DescriptionDefault
highHigher initial priority, faster FairShare decayNo
standardBalanced (recommended)Yes
lowBoosts FairShare by slowing decayNo
scavengerLowest initial priority, greatest FairShare boost (slowest decay)No
startupEnables access to 40 GPUs - users in Tier One only, same as standard priorityNo
heroicEnables access to 720 GPUs - users in Tier Three only, same as standard FairShare decay, requires additional approvalNo

Key Limits​

note

The max resources was changed from 720 GPUs to 240 GPUs on 2026-07-24

  • Max wall time: 48 hours (all QoS levels)
  • Max resources
    • Tier One: 40 GPUS (startup QoS level)
    • Tier Two: 240 GPUs (high, standard, low and scavenger QoS levels)
    • Tier Three: 720 GPUs (high, standard, low and scavenger and, if approved, heroic QoS levels)
  • Time field: Required on all submissions
  • Project/Account: Specify if you belong to multiple projects

Slurm Submission Flags​

SpecificationOptionExampleExample-PurposeRequired
Wall Clock Limit--time=[hh:mm:ss]--time=05:00:00Set wall clock limit to 5 hours 00 minYes
Job Name--job-name=[SomeText]--job-name=myJobSet the job name to "myJob"No
Quality of Service--qos=[QoS name]--qos=standardChoose the “standard” Qos - select from values in above tableNo
Total nodes--nodes=[#]--nodes=1Request 1 nodeNo
Total Task Count--ntasks=[#]--ntasks=2Request 2 tasks totalNo
CPUs per task 1--cpus-per-task=[#]--cpus-per-task=4Request 4 CPUs per taskNo
Total GPUs per node--gres=gpu:[#]--gres=gpu:4Request 4 GPUs per nodeNo
Total GPUs per task--gpus-per-task=[#]--gpus-per-task=2Request 2 GPUs per taskNo
Total GPUs for job--gpus=[#]--gpus=10Request 10 GPUs across the jobNo
Tasks per Node--ntasks-per-node=[#]--ntasks-per-node=48Request exactly (or max) of 48 tasks per nodeNo
Memory Per Node 2--mem=value[K|M|G|T]--mem=360GRequest 360 GB per nodeNo
Memory Per CPU 2--mempercpu=value[K|M|G|T]--mem=8292MRequest 8,292 MB per CPUNo
Combined stdout/stderr--output=[OutputName].%j--output=myJobOut.%jCollect stdout/err in myJobOut.[JobID]No

Examples​

  • Job request for 1 node, 1 CPU, 1 GPU, and 10 minutes of runtime.

Create a script called job_hello_world.job:

#!/bin/bash  
#SBATCH --job-name=hello_world  
#SBATCH --nodes=1  
#SBATCH --ntasks=1  
#SBATCH --output=hello_%j.out  
#SBATCH --error=hello_%j.err  
#SBATCH --time=00:10:00  
#SBATCH --qos=standard  
#SBATCH --gres=gpu:1

srun sh -c 'echo "hello world ($(hostname)) ($XDG_RUNTIME_DIR) ($XDG_SESSION_ID) ($XDG_SESSION_TYPE) ($XDG_SESSION_CLASS)" | tee /scratch/user/$USER/hello_world_$(hostname)'

Run the job with:

sbatch ./job_hello_world.job
  • Job request for 100 GPUs, 150 GBs of RAM per node, 500 CPUs and 30 hours of runtime. There will be 500 instances of the srun command, each with 1 CPU.

Create a script called job_hello_world.job:

#!/bin/bash  
#SBATCH --job-name=hello_world  
#SBATCH --gpus=100  
#SBATCH --mem=150G  
#SBATCH --ntasks=500  
#SBATCH --output=hello_%j.out  
#SBATCH --error=hello_%j.err  
#SBATCH --time=30:00:00  
#SBATCH --qos=standard

srun sh -c 'echo "hello world ($(hostname)) ($XDG_RUNTIME_DIR) ($XDG_SESSION_ID) ($XDG_SESSION_TYPE) ($XDG_SESSION_CLASS)" | tee /scratch/user/$USER/hello_world_$(hostname)'

Run the job with:

sbatch ./job_hello_world.job
  • Job request for 100 GPUs, 150 GBs of RAM per node, 500 CPUs and 30 hours of runtime. There will be 500 instances of the srun command each with 4 CPUs.

Create a script called job_hello_world.job:

#!/bin/bash  
#SBATCH --job-name=hello_world  
#SBATCH --gpus=100  
#SBATCH --mem=150G  
#SBATCH --ntasks=500  
#SBATCH --cpus-per-task=4  
#SBATCH --output=hello_%j.out  
#SBATCH --error=hello_%j.err  
#SBATCH --time=30:00:00  
#SBATCH --qos=standard

srun sh -c 'echo "hello world ($(hostname)) ($XDG_RUNTIME_DIR) ($XDG_SESSION_ID) ($XDG_SESSION_TYPE) ($XDG_SESSION_CLASS)" | tee /scratch/user/$USER/hello_world_$(hostname)'

Run the job with:

sbatch ./job_hello_world.job

Job Submission and Monitoring​

The commands below are commonly used to submit and monitor jobs, as well as check resource usage.
In the examples below, >$ indicates your shell prompt.

SBATCH​

Once you have a job script, submit the job to the compute nodes using the sbatch command:
>$ sbatch jobfile.sh
Submitted batch job 123456

Notes:

  • This is different than running the job with ./jobfile.sh !
    • Using sbatch sends your job to the compute nodes, where the resources are.
    • Using ./ runs it locally, on the small login nodes, where it may not succeed.
  • You can specify Slurm directives when you submit the job on the command line.
    • This is an alternative to the #SBATCH lines in the job script itself.
    • For example: >$ sbatch --ntasks=1 jobfile.sh
  • The suffix of your job file can be anything; it does not have to be .sh. You might use .sh to indicate it's a shell script or something like .slrm to indicate it's intended to be a Slurm job.
  • If the job is successfully submitted, sbatch will return a job ID (as in the example above) that you can use to track the job.

SQUEUE​

To monitor your job once it's submitted, you can use the command squeue --me command to track it. This will return a list of your jobs with details about where they are running, time limits, etc.

>$ squeue --me  
JOBID   PARTITION  NAME     USER        ST  TIME   NODES   NODELIST(REASON)
123456     def     job_name user_name   R   1:34   1       dgx019

(If you don't provide the --me, you will see information for all users' jobs, which is likely not what you need.)

SCANCEL​

Should you need to cancel a job, you can pass the job ID to the scancel command:
>$ scancel 123456
Do this to free up resources if you realize you will not need the output of your job for whatever reason.

SACCT​

Show job history, including job IDs, completion status, resources, etc. Defaults to information from only the current day, but you can specify start and end times with the -S and -E flags.

>$ sacct -S 2026-03-01 -E 2026-04-01
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
87079        hello_wor+        def prj-02-vi+          2  COMPLETED      0:0
87079.batch       batch            prj-02-vi+          2  COMPLETED      0:0
87079.extern     extern            prj-02-vi+          2  COMPLETED      0:0
87079.0              sh            prj-02-vi+          2  COMPLETED      0:0
87158        hello_wor+        def prj-02-vi+          2  COMPLETED      0:0
87158.batch       batch            prj-02-vi+          2  COMPLETED      0:0
87158.extern     extern            prj-02-vi+          2  COMPLETED      0:0
87158.0              sh            prj-02-vi+          2  COMPLETED      0:0

Glossary​

FairShare Scheduling:
Scheduling priority is governed by a user’s FairShare score. The FairShare score is calculated by Slurm based on QoS levels, resources requested, time requested, past utilization, decay and other factors.
Using the “high” QoS will initially give you a higher priority but will cause a faster decay of your FairShare score. This will have the effect of delaying the start of future jobs. It is generally recommended to use the “standard” QoS. However, if your job can wait a bit, use of the “low” and “scavenger” QoS will boost your FairShare score by slowing its decay. This will enable future jobs to be scheduled more quickly.

Project: The basic unit of system allocation

Slurm Account: There is a 1 to 1 correspondence between a VISION project and a Slurm account

Slurm Association: Assignment to project

  • All members of the project will be associated with the relevant Slurm account

Slurm Quality of Service (QoS): The requested priority of a job submission

Changelog​

  • 2026-10-14 - Enable v4 of Slurm configuration - Update ledger system by resetting initial calculation date to October 14, 2026 and implement Tier system
  • 2026-09-09 - Enable v3 of Slurm configuration - implement ledger system and adjust initial FairShare value for new users
  • 2026-07-31 - Scheduler parameter values were changed
  • 2026-07-24 - The max resources was changed from 720 GPUs to 240 GPUs
  • 2026-04-21 - Enable v2 of Slurm configuration

Footnotes​

  1. Default value is set to 14 CPU Cores per GPU - if not specified. ↩

  2. Default value is set to 8,292M per CPU. ↩ ↩2