CINECA Leonardo and Pitagora¶

This page builds, tests and runs SIMPLE on the CINECA machines Leonardo and Pitagora. Tested on Leonardo (CPU, GCC 12.2, OpenMP) at SIMPLE ffd7f4a: build, the test selection and the smoke run below pass, also as a batch job. The CUDA build compiles but was not run on a GPU. The Pitagora recipe has not been run.

Access¶

Follow CINECA’s access and 2FA instructions: enrol an OTP authenticator, install the Smallstep step client and bootstrap its certificate authority with the published URL and fingerprint. Then, with an SSH agent running:

step ssh login '<registered-email>' --provisioner cineca-hpc
ssh <username>@login.leonardo.cineca.it

Certificates last 12 hours. Cloning SIMPLE from GitHub over HTTPS works on the login nodes; CMake fetches libneo, Fortnum and Fortplot the same way.

Leonardo: build¶

examples/hpc/cineca/leonardo-gcc-env.sh, sourced from the SIMPLE root, loads the module stack and limits threads for the shared login node:

module purge
module load profile/base gcc/12.2.0 cmake/3.27.9 ninja/1.11.1 \
    openblas/0.3.26--gcc--12.2.0 python/3.11.7 \
    netcdf-c/4.9.2--gcc--12.2.0-spack0.22
export CMAKE_BUILD_PARALLEL_LEVEL=4
export OMP_NUM_THREADS=4 OMP_THREAD_LIMIT=4
export OPENBLAS_NUM_THREADS=1 CTEST_PARALLEL_LEVEL=1
export TMPDIR="$PWD/build/tmp"; mkdir -p "$TMPDIR"

MPI, NetCDF-Fortran, FFTW, GSL and MKL are not needed: NetCDF-C (which loads serial HDF5) serves SIMPLE’s C test-data generator, and Fortran NetCDF access goes through Fortio, fetched by libneo. Python 3 is needed for the test drivers even with the Python interface off.

git clone https://github.com/itpplasma/SIMPLE.git && cd SIMPLE
source examples/hpc/cineca/leonardo-gcc-env.sh
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
    -DENABLE_OPENMP=ON -DSIMPLE_DETERMINISTIC_FP=ON \
    -DENABLE_PYTHON_INTERFACE=OFF -DSIMPLE_ENABLE_PYTHON_TOOLS=OFF \
    -DLIBNEO_REF=f6090e8150c2f5b1c0afde09baadf79b50b1ffe7
cmake --build build -j 4

Use Release, not Fast, which tunes for the build host. Without LIBNEO_REF CMake takes the libneo branch matching SIMPLE’s branch name, or main; the value above is the tested pin. The clean build took 107 s.

Leonardo: test and run¶

On the login node, run a short selection of the fast suite (11 tests, 13 s), keeping the libneo pin in FLAGS:

make test-fast CONFIG=Release \
    FLAGS='-DENABLE_PYTHON_INTERFACE=OFF -DSIMPLE_ENABLE_PYTHON_TOOLS=OFF -DENABLE_OPENMP=ON -DLIBNEO_REF=f6090e8150c2f5b1c0afde09baadf79b50b1ffe7' \
    TEST='^(test_config_reference|test_finite_check|test_profiles|test_deterministic_seed|test_orbit_invariants|test_stencil_utils|test_orbit_symplectic_base|test_newton_solver_status|test_spectre_fo_spline_tok2vol|test_spectre_fo_spline_ncsx_freeboundary|test_diag_albert)$'

The full fast suite contains integration cases with 900 s timeouts; run it in a batch job, not on the login node.

The smoke run traces eight markers for \(10^{-5}\) s in the QA equilibrium that the test configuration downloads to test/test_data/wout.nc:

mkdir -p run/cineca-smoke
cp examples/hpc/cineca/smoke.in run/cineca-smoke/simple.in
cp test/test_data/wout.nc run/cineca-smoke/wout.nc
(cd run/cineca-smoke && ../../build/simple.x simple.in)

Expected: exit status 0 after about 10 s, 8 rows in times_lost.dat and orbit_exit_code.dat, 20 rows in confined_fraction.dat, all finite, final time 1e-5; exit codes four 0 (traced) and four 3 (skipped as strongly passing, default contr_pp).

As a batch job on the budget-free serial partition (no account needed; one task, four cores, 4 GB, three minutes), submitted from the SIMPLE root:

sbatch examples/hpc/cineca/leonardo-serial.sbatch
sacct -j <job-id> --format=JobID,State,ExitCode,Elapsed

Production jobs follow Clusters; partitions and limits are in Leonardo’s Slurm guide.

Leonardo: CUDA¶

Run GPU jobs only in an allocation with a GPU, never on login or serial nodes. Configuration and valid inputs: GPU tracing. The A100 build uses the CPU stack plus CUDA 12.3:

module load cuda/12.3
cmake -S . -B build-cuda -G Ninja -DCMAKE_BUILD_TYPE=Release \
    -DSIMPLE_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=80 \
    -DENABLE_PYTHON_INTERFACE=OFF -DSIMPLE_ENABLE_PYTHON_TOOLS=OFF \
    -DLIBNEO_REF=f6090e8150c2f5b1c0afde09baadf79b50b1ffe7
cmake --build build-cuda -j 4 --target \
    simple.x test_cuda_native_rk54.x test_cuda_native_bridge.x
cuobjdump --list-elf build-cuda/libsimple_cuda_native_rk54.so   # expect sm_80

The device code lives in the shared library libsimple_cuda_native_rk54.so. Validate with make test ... TEST='test_cuda_native_(rk54|bridge)' inside a GPU job before production.

NVHPC 24.5 (nvfortran) does not build SIMPLE: Fortnum’s src/verified/fortnum_interval_qp.f90 declares real(real128), for which NVFORTRAN returns kind -3. Use GNU Fortran with nvcc.

Leonardo: pitfalls¶

netcdf.h not found in test/tests/generate_test_boozer_chartmap.c: load the netcdf-c module.
module load git/2.38.1 together with the OpenMPI stack fails (its openssh dependency loads an older Kerberos). Use /usr/bin/git after module purge.

Pitagora (untested)¶

Consult the current Pitagora guide. Use login-dcgp.pitagora.cineca.it for the AMD Turin CPU partition and login-boost.pitagora.cineca.it for the Intel/H100 Booster. Leonardo’s module names are not known to exist there: find GCC, CMake, Ninja, Python, OpenBLAS and NetCDF-C with modmap -m <name>, write the loads into an environment script like leonardo-gcc-env.sh, export its absolute path as SIMPLE_ENV, and repeat the Leonardo build, test selection and smoke run on the destination partition. For the H100 use -DCMAKE_CUDA_ARCHITECTURES=90.

Partition	Nodes	Max. nodes/job	Max. wall time
`dcgp_fua_dbg` / `dcgp_fua_prod`	2 × AMD EPYC 9745, 256 cores, 768 GiB	2 / 64	30 min / 24 h
`boost_fua_dbg` / `boost_fua_prod`	64 cores, 4 × H100, 512 GiB	2 / 16	30 min / 24 h
`ptgr_all_serial`	CPU only, budget-free, max. 4 cores	—	4 h

One SIMPLE process uses one node or one GPU. Templates (replace the account before submission):

Template	Request	Submit from
`examples/hpc/cineca/pitagora-cpu.sbatch`	debug, 4 cores, 3 min	SIMPLE root, runs `run/cineca-smoke`
`examples/hpc/cineca/pitagora-gpu.sbatch`	debug, 1 GPU, 3 min	SIMPLE root, runs `run/cineca-gpu`
`examples/hpc/pitagora/dcgp.slurm`	debug, 256 cores, 30 min	run directory; `SIMPLE_EXE` = path to `simple.x`
`examples/hpc/pitagora/booster.slurm`	debug, 1 H100 + 16 cores, 30 min	run directory; `SIMPLE_EXE` = path to `build-cuda/simple.x`

For production, override on the command line, e.g. sbatch --partition=dcgp_fua_prod --time=24:00:00 examples/hpc/pitagora/dcgp.slurm. A GPU input sets gpu_mode = 'production', gpu_backend = 'cuda-native', gpu_method = 'dopri', gpu_num_devices = 1, isw_field_type = 2, with collisions, walls, classifiers and orbit output off. Native CUDA tracing needs a Boozer chartmap field; with a VMEC file it stops before tracing (native CUDA RK requires initialized compact Boozer tables). Compare seeded CPU and GPU results on the same case before production.