Clusters

This page shows how to run SIMPLE under Slurm and which job scripts in the repository fit which machine. One SIMPLE process traces all markers of a run with OpenMP on one node, or on one GPU; there is no MPI decomposition. Scale out by submitting independent jobs (equilibria, seeds, batch_size batches), each in its own run directory, because output file names are fixed.

Job pattern

#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=<cores>
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export OMP_PLACES=cores OMP_PROC_BIND=spread OPENBLAS_NUM_THREADS=1
srun --cpu-bind=cores /path/to/SIMPLE/build/simple.x simple.in

Submit from a run directory that contains simple.in and the files it references. Build on the target architecture; CONFIG=Fast tunes for the build host.

GPU jobs

GPU runs use the native CUDA backend (gpu_backend = 'cuda-native', GPU tracing), one process per GPU. Build with -DSIMPLE_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=<arch>:

System

GPU

CMAKE_CUDA_ARCHITECTURES

aCluster

NVIDIA Tesla T4

75

Perlmutter, CINECA Leonardo

NVIDIA A100

80

CINECA Pitagora (Booster)

NVIDIA H100

90

sCluster

NVIDIA RTX PRO 6000 Blackwell Max-Q

120

On Perlmutter the GNU environment suffices for the Fortran host code:

module load PrgEnv-gnu cudatoolkit/12.9 craype-accel-nvidia80 cray-hdf5 cray-netcdf
cmake -S . -B build-cuda -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn \
  -DSIMPLE_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=80
cmake --build build-cuda -j

Request the GPU with --gpus-per-task=1 (Perlmutter) or --gres=gpu:1 (aCluster, sCluster).

Job scripts

  • RUN/{Perlmutter,aCluster,sCluster}/submit_gpu.slurm: one GPU; set SIMPLE_BUILD (default $HOME/code/SIMPLE/build-cuda) and SIMPLE_INPUT.

  • examples/hpc/cineca/, examples/hpc/pitagora/: Leonardo smoke test and untested Pitagora templates (CINECA Leonardo and Pitagora).