Accessing gpus

The ALICE HPC cluster provides access to 4 gpu nodes, each with 64 intel icelake cpu cores, 2 NVidia A100-80 gpus and 512GB of memory. To access the gpu nodes, you need to be a member of an HPC project.

As the older ALICE2 cluster is retired, we also plan to move it's GPU nodes (which have 2 V100 gpus) to the new ALICE cluster.

Simple CUDA code

To make use of the gpus, you must specify the number and type of gpus that your code requires - the following example job script requests 4 cpu cores, 40 GB of memory and 1 pascal (P100) gpu:

#!/bin/bash
#
# Example SLURM GPU job script for ALICE3

#SBATCH --job-name=tst_gpu
#SBATCH --account=MY_HPC_PROJECT
#SBATCH --tasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=40G
#SBATCH --gres=gpu:pascal:1
#SBATCH --time=01:00:00
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --mail-user=MY_EMAIL@le.ac.uk
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err
#SBATCH --export=None

# load gcc, cuda and cudnn modules
module load gcc/12.3.0
module load cuda12.1/toolkit
module load cudnn8.9-cuda12.1

# Change directory to the directory from which the job was submitted:
cd $SLURM_SUBMIT_DIR

# Run your code here - it should detect the gpu(s) that have been assigned.
./your_cuda_code

You can request up to 2 gpus per node. We recommend you request at least as much memory for your job as is available on the gpus (80GB per gpu for the Ampere A100-80 gpus,16GB for the Pascal P100 gpus). Please avoid requesting more that 32 cpu cores and 250GB memory per GPU if possible (14 cores and 64GB per GPU on Pascal nodes). If you need to request more, please consider using both GPUs on the node if your code supports this.

GPU Architecture

ALICE has a number of Ampere A100-80 and Pascal P100-16 GPU nodes, you can choose which architecture you wish for your job using the --gres parameter.

For ampere:

--gres=gpu:ampere:N

For pascal:

--gres=gpu:pascal:N

replacing N with the number of GPU nodes to request. Ampere is the more modern architecture, with greater performance and more memory per GPU, however wait times for the Ampere GPUs tend to be long, so please consider using the older Pascal GPUs if your code supports them.

gpu-devel jobs

The gpu devel partition is provided for running short (<2 hour) developement or testing jobs on the gpus. It is limited to a single job, using up to 2 gpus on a single node, per user.

The gpu-devel queue must be explicitly requested, by adding :

#SBATCH --partition=gpu-devel

to your job script or srun command line. For example:

#!/bin/bash
#
# Example SLURM GPU job script for ALICE3

#SBATCH --job-name=tst_gpu
#SBATCH --account=MY_HPC_PROJECT
#SBATCH --partition=gpu-devel
#SBATCH --tasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=128G
#SBATCH --gres=gpu:ampere:1
#SBATCH --time=01:00:00
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --mail-user=MY_EMAIL@le.ac.uk
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err
#SBATCH --export=None

# load gcc, cuda and cudnn modules
module load gcc/12.3.0
module load cuda12.1/toolkit
module load cudnn8.9-cuda12.1

# Change directory to the directory from which the job was submitted:
cd $SLURM_SUBMIT_DIR

# Run your code here - it should detect the gpu(s) that have been assigned.
./your_cuda_code

Or, for an interactive job:

srun --partition=gpu-devel --account=MY_HPC_PROJECT --cpus-per-task=4 \
     --time=01:00:00 --mem=128G --gres=gpu:ampere:1 --pty /bin/bash

You can also request interactive devel jobs on the Pascal nodes run on nodes which are dedicated to running devel jobs during working hours using a job reservation:

srun --partition=gpu-devel --reservation=gpu-devel --account=MY_HPC_PROJECT \
     --cpus-per-task=4 --time=01:00:00 --mem=128G --gres=gpu:pascal:1 \
     --pty /bin/bash

Note - this facility is currently only avaiable for the Pascal nodes. We aim to extend this to Ampere nodes at the December service day.

Interactive jobs

Long (>24 hour) interactive jobs on the GPU nodes are discouraged as these often waste valuable GPU resources. From 16th December only a single interactive job per user will be permitted to run at a time.

Hybrid GPU/mpi job

GPU codes are able to use multiple gpus on multiple nodes, usually making use of mpi to launch tasks on multiple nodes, each of which has one or more gpus. We generally recomend not doing this on ALICE as our gpu nodes do not support NVLink and so the communication overhead, even over infiniband, tends to be very high. If you have a code which supports this and wish to try, please contact RCS support to discuss this with us.