Running R on Argo
Running Serial R Jobs
To use the optimized version of R compiled with OpenBLAS, you have to load the module R. Currently the most up-to-date R version is 3.5.1. Make sure you check all the available R modules installed on the cluster before submitting your script. Ensure that you specify the R module version you want to use.
If you type module show R/3.5.1, you will see the following:
-------------------------------------------------------------------
/cm/shared/modulefiles/R/3.5.1:
module-whatis Adds R with openblas library to your environment.
module load openblas/0.2.20
prepend-path PATH /cm/shared/apps/R/3.5.1/bin
prepend-path LD_LIBRARY_PATH /cm/shared/apps/R/3.5.1/lib64
-------------------------------------------------------------------
You can submit batch R jobs with a Slurm submission script. At the end
of your Slurm script, you can run your R script with the following
command: RScript [options].R. To find what the
options can be passed to RScript type R --help after you have loaded
the R module. You need to load the R module explicitly inside your Slurm
job submission file. NOTE: R uses ".RData" file in your current
directory to load/save workspace every time it starts/finishes a
session. This can significantly slow down execution of your job
depending on the size of the ".RData" file.
It is advisable to use the following options
"--no-restore --quiet --no-save" when starting a job. The option
"--quiet" suppresses the start up messages, "--no-restore" directs R
not to restore anything from ".RData", and "--no-save" ensures that
the workspace is not saved to ".RData" at the end of the R session.
Given below is a sample Slurm job submission script for a R job:
#!/bin/sh
## Specify the name for your job, this is the job name by which Slurm will
## refer to your job. This can be different from the name of your executable
## or the name of your script file
#SBATCH --job-name My_R_Job
## General partitions: all-LoPri, all-HiPri, bigmem-LoPri, bigmem-HiPri, gpuq
## all-* Will run jobs on (almost) any node available
## bigmem-* Will run jobs only on nodes with 512GB memory
## *-HiPri Will run jobs for up to 12 hours
## *-LoPri Will run jobs for up to 5 days
## gpuq Will run jobs only on nodes with GPUs (40, 50, 55, 56)
## Restricted partitions: CDS_q, CS_q, STATS_q, HH_q, GA_q, ES_q, COS_q
## Provide high priority access for contributors
#SBATCH --partition=all-HiPri
## Deal with output and errors. Separate into 2 files (not the default).
## May help to put your result files in a directory: e.g. /scratch/%u/logs/...
## NOTE: %u=userID, %x=jobName, %N=nodeID, %j=jobID, %A=arrayID, %a=arrayTaskID
#SBATCH --output=/scratch/%u/%x-%N-%j.out # Output file
#SBATCH --error=/scratch/%u/%x-%N-%j.err # Error file
#SBATCH --mail-type=BEGIN,END,FAIL # ALL,NONE,BEGIN,END,FAIL,REQUEUE,..
#SBATCH --mail-user=<GMUnetID>@gmu.edu # Put your GMU email address here
## Specifying an upper limit on needed resources will improve your scheduling
## priority, but if you exceed these values, your job will be terminated.
## Check your "Job Ended" emails for actual resource usage info.
#SBATCH --mem=1M # Total memory needed for your job (suffixes: K,M,G,T)
#SBATCH --time=0-00:02 # Total time needed for your job: Days-Hours:Minutes
## Load the relevant modules needed for the job
module load R/3.5.1
## Start the job
Rscript --no-restore --quiet --no-save RHello.R
#!/usr/bin/env Rscript
## Just output the text "Hello, world!"
cat("Hello, world!\n")
Just note that the values for --mem and --time are set to be
extremely low. You will want to increase these when you run your own
programs.
Running Multi-threaded R Jobs
On the ARGO cluster, many of the R modules (including R/3.5.0 and
R/3.5.1) are configured to use the OpenBLAS library. This means that R
can take advantage of additional CPUs if they are available when
performing linear algebra operations. To request additional CPUs for
your job you can use the --cpus-per-task Slurm parameter. You should
also make sure that the environment variable OPENBLAS_NUM_THREADS is
set to an appropriate value greater than or equal to the number of cores
requested.
Add the following to your Slurm submission script, replacing
#SBATCH --cpus-per-task <C> # Request extra CPUs for threads
export OPENBLAS_NUM_THREADS=<C>
Be aware that
There are some R packages that take advantage of the R parallel library to make use of additional CPUs cores e.g. parallelDist, Rdsm and RcppThread.
A common R programming practice with the parallel library is to set the number of cores for the parallel cluster using the parallel::detectCores() routine e.g.:
mc.cores = parallel::detectCores()
This can cause problems when running with SLURM, as SLURM will restrict your job to the cores requested but detectCores() will return the total number of cores on the node. Unless you are requesting a full node this will overload the cores in your job and may severly impact performance. The best practice for using the parallel library is to set mc.cores to the number of cores assigned by SLURM. This can be done by adding the code below to your R script:
# set the number of parallel worker according to SLURM
nworkers <- Sys.getenv("SLURM_CPUS_PER_TASK")
message("I have ", nworkers, " cores available")
Running Parallel R Jobs Using Rmpi
NOTE: Rmpi has been installed and tested for every version of R except R/3.4.1. All of them were compiled using the openmpi/3.1.3 module as a dependency.
Rmpi works with OpenMPI as a wrapper to spawn slave processes through your R script file. Hence OpenMPI needs to be loaded before using Rmpi. Detailed information about Rmpi can be found in Rmpi documentation.
Below is a sample job submission script that shows how to submit Rmpi jobs:
#!/bin/sh
## Specify the name for your job, this is the job name by which Slurm will
## refer to your job. This can be different from the name of your executable
## or the name of your script file.
#SBATCH --job-name=RmpiHello
## General partitions: all-LoPri, all-HiPri, bigmem-LoPri, bigmem-HiPri, gpuq
## all-* Will run jobs on (almost) any node available
## bigmem-* Will run jobs only on nodes with 512GB memory
## *-HiPri Will run jobs for up to 12 hours
## *-LoPri Will run jobs for up to 5 days
## gpuq Will run jobs only on nodes with GPUs (40, 50, 55, 56)
## Restricted partitions: CDS_q, CS_q, STATS_q, HH_q, GA_q, ES_q, COS_q
## Provide high priority access for contributors
#SBATCH --partition=all-HiPri
## Deal with output and errors. Separate into 2 files (not the default).
## May help to put your result files in a directory: e.g. /scratch/%u/logs/...
## NOTE: %u=userID, %x=jobName, %N=nodeID, %j=jobID, %A=arrayID, %a=arrayTaskID
#SBATCH --output=/scratch/%u/%x-%N-%j.out # Output file
#SBATCH --error=/scratch/%u/%x-%N-%j.err # Error file
#SBATCH --mail-type=BEGIN,END,FAIL # ALL,NONE,BEGIN,END,FAIL,REQUEUE,..
#SBATCH --mail-user=<GMUnetID>@gmu.edu # Put your GMU email address here
## You can improve your scheduling priority by specifying upper limits on
## needed resources, but jobs that exceed these values will be terminated.
## Check your "Job Ended" emails for actual resource usage info as a guide.
#SBATCH --mem=1M # Total memory needed per task (units: K,M,G,T)
#SBATCH --time=0-00:02 # Total time needed for job: Days-Hours:Minutes
## ----- Parallel Processes -----
## Some libraries (MPI) implement parallelism using processes that communicate.
## This allows tasks to run on any set of cores in the cluster. Programs can
## use this approach in combination with threads (if designed to).
#SBATCH --ntasks <T> # Number of processes you plan to launch
# Optional parameters. Uncomment (remove one leading '#') to use.
##SBATCH --nodes <N> # If you want some control over how tasks are
# distributed on nodes. <T> >= <N>
##SBATCH --ntasks-per-node <Z> # If you want more control over how tasks are
# distributed on nodes. <T> = <N> * <Z>
## Load the R module which also loads the OpenBLAS module
module load R/3.5.1
## To use Rmpi, you need to load the openmpi module
module load openmpi/3.1.3
# R still wants to write files to our current directory, despite using
# "--no-restore --quiet --no-save" below, so move someplace writable.
ORIG_DIR=$PWD
cd $SCRATCH
echo "Calling mpirun now!!!"
## Use "--no-restore --quiet --no-save" to be as quiet as possible.
## Note: You do not spawn the parallel processes directly through mpirun, but
## instead from inside your R script, hence parameter -np is set to 1.
mpirun -np 1 Rscript --no-restore --quiet --no-save $ORIG_DIR/RmpiHello.R
Below is a parallel hello world Rmpi program that can be used to test
the above script. Be sure to replace the
## RmpiHello.R
## Load the R MPI package if it is not already loaded.
if (!is.loaded("mpi_initialize"))
{
library("Rmpi")
}
## Specify how many slave processes will be spawned.
## This must be 1 less than the number of tasks requested (master uses 1).
mpi.spawn.Rslaves(nslaves=<T-1>) # Change this to match your Slurm script
## In case R exits unexpectedly, automatically clean up
## resources taken up by Rmpi (slaves, memory, etc...)
.Last <- function()
{
if (is.loaded("mpi_initialize"))
{
if (mpi.comm.size(1) > 0)
{
print("Please use mpi.close.Rslaves() to close slaves.")
mpi.close.Rslaves()
}
print("Please use mpi.quit() to quit R")
.Call("mpi_finalize")
}
}
## Tell all slaves to return a message identifying themselves
mpi.remote.exec(paste("Hello, World from process ",mpi.comm.rank(),"of",mpi.comm.size()))
## Tell all slaves to close down, and exit the program
mpi.close.Rslaves()
mpi.quit()
Note. Number of slaves requested must be 1 less than the number of tasks requested as shown in the above scripts.
Installing Additional R-Packages
If you want to add additional packages to the R module (user packages
are added in such a way that they will only be available to you when you
load R) you need to run the following command from inside R terminal. By
default R will install these packages to the default user package
directory :~/R/x86_64-unknown-linux-gnu-library/VERSION/. Where,
VERSION is the R version you are using. The first time you install a
package you will be asked to select whether to use the default location
or not. Select yes to proceed.
$ module load R/<version>
$ R
> install.packages("package_name")
You may see a prompt mentioning which server to download the package from. Select a server of your choice and proceed.
To load the said package inside your R script use:
library("package_name")
To install an R package to a specific location you need to specify the
location in the install.packages command:
> install.packages("package_name", lib="/custom/path/to/R-packages/")
If you don't specify a lib= parameter then R will ask you if you want
to use a default path in your home directory. This is probably a better
choice so that you don't have to remember where you put your packages a
month or a year from now.
> install.packages("package_name")
If you receive compile errors while installing a package, then you may need to load a newer version of the gcc compiler. First exit R, then type:
$ module avail gcc
----------------------------------------------- /cm/shared/modulefiles -----------------------------------------------
gcc/4.8.4 gcc/5.2.0(default) gcc/6.3.0 gcc/7.1.0 gcc/7.3.1 gcc/8.3.1
$ module load gcc/8.3.1
The latest compiler is not necessarily the best, so you may need to go through this process a couple of times, selecting different compilers until you get it to work. Each time you will need to restart R, and try installing the package again. If you still cannot get it to work, send a request for help.
To load this package use:
> library("package_name", lib.loc="/custom/path/to/R-packages/")
> library("package_name")