Testing¶
Tests are registered with CTest and run through make from the repository
root. Every change of behaviour comes with a test that fails without it.
Test targets¶
All targets except test-all first rebuild with deterministic floating
point (SIMPLE_DETERMINISTIC_FP=ON).
Command |
Runs |
|---|---|
|
all but labels |
|
all but |
|
as |
|
label |
|
label |
|
label |
|
every test, current build configuration |
TEST=<regex> selects tests by name, VERBOSE=1 shows all output and
CONFIG=Fast selects the build profile. The regex passes through make:
quote it and write $ as $$, e.g. TEST='^test_finite_check$$'.
Tests labelled python and some unlabelled wrappers need the packages from
./setup-venv.sh.
CI tiers and CPU allocation¶
Every pull request, including drafts and forks, runs Build and Test on
GitHub-hosted Ubuntu 24.04 using GCC 14, make test-fast CONFIG=Fast and the
GCC matrix’s ENABLE_PYTHON_INTERFACE=OFF / SIMPLE_ENABLE_PYTHON_TOOLS=OFF
build mode. Two PR shards partition CTest by odd/even test indices; CTest
fixtures add required producers to either shard. Each shard builds separately
and uses all runner cores and its own memory resource budget. Full queue and
dispatch runs remain unsharded. Standalone Python drivers still run;
Python-interface and frontend tests remain in the full queue build. CTest uses all
runner cores and the generated cgroup-aware memory resource spec. OpenMP
defaults to two threads for field setup and one for the FO stepper;
OpenBLAS uses one. The loss and SPECTRE orbit checks reserve two CTest
processors for their existing parallel particle/process work.
The merge queue runs the full suite on itp-slurm. Manual dispatch preserves
its runner choice (itp-slurm or hosted) and smoke/full selection. The
full-ci PR label adds compiler and VMEC++ checks, while Build and Test
continues to run the hosted fast tier. Use full dispatch or the merge queue
for full testing: the 13–15 GiB golden-record reservations exceed the usable
memory budget of the standard hosted runner. GCC 13 smoke remains in the
compiler matrix; coverage runs on sCluster with GCC 12 instead of hosted GCC 14.
Each full sCluster run uses one unchanged full Slurm allocation (192 GiB,
90 minutes), with two concurrent builds: the correctness gate runs fast/slow
with deterministic Fast and no coverage, plus a separate deterministic Release
regression tree matching the golden reference; reporting runs only the fast
tier with ENABLE_COVERAGE=ON. Coverage gets up to 16 CPUs (one quarter of
smaller allocations); the gate splits the remaining CPUs between fast/slow
and regression. For 94 CPUs this is 16 coverage, 39 fast/slow and 39 regression.
The three CTest memory caps sum to 75% of SCI_MEM_GB (24/60/60 GiB at
94 CPUs and 192 GiB), and each tree also uses the generated Slurm/cgroup-aware
resource spec. Python environments and editable installs are isolated.
Only the correctness gate’s exit status decides the check. Coverage setup,
test, package-install or gcovr failures are reporting failures: the job log
and Actions summary show their exit statuses and a warning, while the gate
still runs to completion. A successful report is uploaded even if some
instrumented tests failed; the summary identifies that incomplete test run.
Artifact upload is also non-blocking. Reports return through ci-out/ and
SCI_OUT under the infrastructure’s 64 MiB compressed cap, retaining the
fast-suite-coverage artifact’s root coverage.xml, coverage.html and
summary layout consumed by Pages. Oversized details fall back to an overview;
a still-oversized or failed report is omitted, with its failure recorded.
Incomplete XML/HTML artifacts are not uploaded.
The short FO tests keep all field-jet, derivative, energy and one-step map
checks, and compose 64 steps in the fast tier. Their _full siblings retain
the original 1,000-step compositions in slow. Both use the same strict
1e-7 tangent, 1e-5 finite-difference and 1e-8 correspondence bounds: these test
symplecticity/endpoint agreement for a fixed map, rather than a Monte Carlo
estimate, so shortening a composition does not require a larger tolerance.
The long energy and resolution runs retain every original step.
The loss and SPECTRE trajectory drivers also have slow
_full siblings with their original configurations. Fast counts are 8/32
for axis-loss, 16/64 for outer explicit losses, 1/8 for the explicit
near-axis smoke check, and 4/24 for SPECTRE trajectories. Trace
lengths and per-orbit bounds are unchanged: zero forbidden losses/escapes,
energy/trajectory error maxima and exact metadata identities do not acquire
a sampling tolerance. The explicit loss-count cross-check retains its
existing max(2, round(1.5*sqrt(reference_losses))) counting-error bound,
evaluated for the reduced count; its nonzero-reference-loss guard remains.
The stellarator stepper driver runs two periods (96 stored states) for every
scheme in fast, requiring zero step errors and finite states. Its _full
case retains the original 7,500-state plotting trace and output files.
Memory allocation on workstations and CI¶
Configure on the machine that will run the tests. CMake writes
build/ctest-resources.json with one mem_gb resource whose slots each
represent one GiB. Usable memory is the minimum of Linux MemAvailable,
cgroup memory headroom, and any positive Slurm memory allocation
(SLURM_MEM_PER_NODE or SLURM_MEM_PER_CPU * SLURM_CPUS_ON_NODE, in MiB).
For cgroup v2, CMake locates the job via /proc/self/cgroup, takes the
smallest finite memory.max from that cgroup up to the mount root, and
subtracts the job cgroup’s memory.current. The v1 fallback uses
memory.limit_in_bytes and memory.usage_in_bytes under the memory
controller, ignoring huge unlimited sentinels. Negative headroom becomes zero.
Missing cgroup files impose no constraint; missing MemAvailable falls back
to CMake’s available physical memory query.
The automatic budget is this usable memory, rounded down to GiB, less a
reserve of 25%, rounded up, or 4 GiB, whichever is larger. This leaves room
for the OS, other jobs and tests without reservations. Cap this snapshot on
a shared machine, and reconfigure when its available memory changes:
cmake -S . -B build -DSIMPLE_TEST_MEMORY_GB=24
ctest --test-dir build/test -j8 --output-on-failure
CMake records CTEST_RESOURCE_SPEC_FILE in every generated
CTestTestfile.cmake. Both ctest from build and
ctest --test-dir build/test therefore load the generated spec automatically.
Thus CI’s existing ctest -j16 also uses memory allocation without another
command-line option. A site-specific spec can override the default:
ctest --test-dir build/test -j16 --resource-spec-file /path/to/resources.json
For example, a 24 GiB budget has this format:
{"version": {"major": 1, "minor": 0},
"local": [{"mem_gb": [{"id": "ram", "slots": 24}]}]}
CTest’s RESOURCE_GROUPS alone does not limit parallelism without a
resource spec; tests still run. This is documented in the
CTest resource allocation manual
and verified with CTest 4.4.4. The automatic default above prevents that
unbounded fallback. A test requiring more slots than the budget is reported
as Not Run with insufficient resources; the budget is never raised to
force it to run. Memory reservations govern scheduling within one CTest
invocation, not OS memory limits or separate CTest invocations. Keep OpenMP
thread counts appropriate to the CPU allocation as well.
Reservations are peak RSS rounded up to whole GiB, measured one registered
test at a time with GNU /usr/bin/time -v. Except for the updated wall test,
these measurements ran inside
systemd-run --user --scope -p MemoryMax=16G. The workstation measurements
use GNU Fortran 16.2.1, the deterministic Fast build, four OpenMP threads
and one OpenBLAS thread; golden-record scripts retain their fixed two threads
and the wall test retains its fixed single thread.
The chartmap measurements use the current fixtures: pipeline shares one
VMEC setup with angular multiplier 1; R,Z keeps angular multiplier/order 5
and uses radial order 3 (the boundary remains an exact interpolation node).
Other derivative reservations retain their previous conservative bounds
from the larger grids. The wall test uses VMEC angular multiplier 1 for initialization; its coil
field, chartmap, explicit starts and numerical assertions are unchanged.
Test |
RSS (GiB) |
Slots |
|---|---|---|
test_splined_field_derivatives |
0.8202 |
1 |
test_bder_from_splines |
0.8191 |
1 |
test_magfie_refcoords_splines |
1.1990 |
2 |
test_hcurl_from_dhcov_splines |
0.8194 |
1 |
test_splined_kernel_bench |
12.6803 |
13 |
test_chartmap_pipeline |
0.4584 |
2 |
test_chartmap_rz_consistency |
6.5831 |
8 |
gc_ck_ncsx_vmec_vs_meiss |
11.9712 |
12 |
golden_record_albert_coils |
14.0856 |
15 |
golden_record_boozer_ncsx |
7.2261 |
8 |
golden_record_meiss_coils |
12.0610 |
13 |
test_collisional_stl_wall |
0.9169 |
1 |
generate_chartmap_ncsx_map2disc |
0.3585 |
1 |
generate_chartmap_ncsx_map2disc_wall |
0.3940 |
1 |
test_chartmap_ncsx_map2disc_wall_offset |
0.3940 |
1 |
test_chartmap_boundary_param |
0.3988 |
1 |
Re-measure when adding NCSX consumers or increasing fixture sizes.
Golden cases use flock (util-linux) only while preparing their shared
reference build and input files; their numerical runs share the memory budget.
Kinds of tests and labels¶
Kind (labels) |
Location: checks |
|---|---|
Unit ( |
|
Integration ( |
|
Python ( |
|
Golden record ( |
|
Add slow to tests that run longer than about a minute. Register Fortran
tests in test/tests/CMakeLists.txt and pytest files with add_pytest in
test/python/CMakeLists.txt.
Oracles and tolerances¶
The oracle is independent of the implementation: an analytic result, an identity, a roundtrip (
ref -> integ -> ref), a symmetry, a convergence rate or a second field representation. Rerunning the same code is not an oracle.A tolerance follows from a physical or numerical scale (
relerr, spline order and grid spacing, Monte Carlo error \(\sim 1/\sqrt{N}\)), never from the observed error.Stochastic results are compared as distributions or moments, not particle by particle.
Data available on only one machine must not be required; missing optional data skips the test.
Invariants to keep¶
Subsystem |
Must hold after a change |
|---|---|
Coordinate transforms |
|
VMEC, Boozer, chartmap |
same start surface, same short-time orbit within tolerance, statistically equal loss fraction |
Symplectic integrators |
bounded invariants, no drift beyond the reference envelope |
Collisions |
Maxwellian fixed point, analytic pitch and energy relaxation, seeded reproducibility |
Wall losses |
same wall hit in Cartesian and coordinate space |
NetCDF output |
unchanged meaning, units and frame of |
GPU and OpenMP |
CPU path unchanged when acceleration is off, short-run CPU/GPU agreement, long-run statistical agreement, no CPU slowdown |
Golden records¶
golden_record_<case> clones and builds main under build/, runs
test/golden_record/<case>/simple.in with both builds and compares the
outputs with relative tolerance 1.5e-7. The first run needs network
access and several minutes. Use make test-regression TEST=golden_record_:
it pins the reference libneo revision (libneo_golden in
cmake/dependency_versions.json) for both builds. Patches in
test/golden_record/reference_patches/ make old reference revisions
buildable and comparable. The stochastic collision case uses its own
acceptance rule (Stochastic collision golden case). Add a case when a new
field type, integrator or output path becomes part of the supported workflow.
A change of golden output is a scientific change and must be explained in the
pull request (Contributing to SIMPLE).
The bounded numerical verification report and its evidence table are in Numerical verification report.