Running SLURM Array Jobs

Running Multiple Scripts with an Array Job

It is possible to run array jobs with your SLURM script. In the script, you would set the number of instances you want to run at the same time with:

#!/bin/bash 

#SBATCH --partition=normal                 # will run on any cpus in the 'normal' partition 

#SBATCH --job-name=python-job 



## Separate output and error messages into 2 files. 

## NOTE: %u=userID, %x=jobName, %N=nodeID, %j=jobID 

#SBATCH --output=/scratch/%u/%x-%N-%j.out  # Output file 

#SBATCH --error=/scratch/%u/%x-%N-%j.err   # Error file 



#SBATCH --nodes=1 

#SBATCH --cpus-per-task=12                   # up to 48 per node 

#SBATCH --mem-per-cpu=3500M                 # memory per CORE; maximum is 180GB per node 



#SBATCH --time=0-01:00:00                   # set to 1hr; please choose carefully 



## Specify number of instances you want to run: 

#SBATCH --array=1-5 





module load gnu10 

module load python                          # load the recommended Python version 





## Execute pytthon scripts based on array instance 

python myscript_${SLURM_ARRAY_TASK_ID}.py                          # execute your Python script 

You can name your python script based on the array numbers:

myscript_1.py 

myscript_2.py 

myscript_3.py 

myscript_4.py 

myscript_5.py 

and execute them in that way in the SLURM script:

python myscript_${SLURM_ARRAY_TASK_ID}.py 

The python scripts would each contain the following:

#!/usr/bin/python 



# import sys/os library (needed for accepted command line args) 

import sys 

import os 



# print task number 

print('Hello! I am a task number: ', os.environ.get(‘SLURM_ARRAY_TASK_ID’)) 

Running a Single Script with Array Jobs

Alternatively, you can have a single python script and use the environment variable ${SLURM_ARRAY_TASK_ID} to change a specific parameter in your calculation, e.g. in the script below:

#!/usr/bin/python 



# import sys library (needed for accepted command line args) 

import sys 



# print task number 

print('Hello! I am a task number: ', sys.argv[1]) 

And then simply execute the python script in your SLURM script and 5 instances will be started:

python myscript.py ${SLURM_ARRAY_TASK_ID}