Testing ======= Tests are registered with CTest and run through ``make`` from the repository root. Every change of behaviour comes with a test that fails without it. Test targets ------------ All targets except ``test-all`` first rebuild with deterministic floating point (``SIMPLE_DETERMINISTIC_FP=ON``). ========================= ================================================== Command Runs ========================= ================================================== ``make test-fast`` all but labels ``slow``, ``regression``, ``performance``; run before every push ``make test`` all but ``regression`` ``make test-nopy`` as ``make test`` without label ``python`` ``make test-slow`` label ``slow`` ``make test-smoke`` label ``smoke`` (sub-minute gate used by libneo) ``make test-regression`` label ``regression``: golden records vs ``main`` ``make test-all`` every test, current build configuration ========================= ================================================== ``TEST=`` selects tests by name, ``VERBOSE=1`` shows all output and ``CONFIG=Fast`` selects the build profile. The regex passes through ``make``: quote it and write ``$`` as ``$$``, e.g. ``TEST='^test_finite_check$$'``. Tests labelled ``python`` and some unlabelled wrappers need the packages from ``./setup-venv.sh``. CI tiers and CPU allocation --------------------------- Every pull request, including drafts and forks, runs ``Build and Test`` on GitHub-hosted Ubuntu 24.04 using GCC 14, ``make test-fast CONFIG=Fast`` and the GCC matrix's ``ENABLE_PYTHON_INTERFACE=OFF`` / ``SIMPLE_ENABLE_PYTHON_TOOLS=OFF`` build mode. Two PR shards partition CTest by odd/even test indices; CTest fixtures add required producers to either shard. Each shard builds separately and uses all runner cores and its own memory resource budget. Full queue and dispatch runs remain unsharded. Standalone Python drivers still run; Python-interface and frontend tests remain in the full queue build. CTest uses all runner cores and the generated cgroup-aware memory resource spec. OpenMP defaults to two threads for field setup and one for the FO stepper; OpenBLAS uses one. The loss and SPECTRE orbit checks reserve two CTest processors for their existing parallel particle/process work. The merge queue runs the full suite on ``itp-slurm``. Manual dispatch preserves its runner choice (``itp-slurm`` or ``hosted``) and smoke/full selection. The ``full-ci`` PR label adds compiler and VMEC++ checks, while ``Build and Test`` continues to run the hosted fast tier. Use full dispatch or the merge queue for full testing: the 13--15 GiB golden-record reservations exceed the usable memory budget of the standard hosted runner. GCC 13 smoke remains in the compiler matrix; coverage runs on sCluster with GCC 12 instead of hosted GCC 14. Each full sCluster run uses one unchanged ``full`` Slurm allocation (192 GiB, 90 minutes), with two concurrent builds: the correctness gate runs fast/slow with deterministic Fast and no coverage, plus a separate deterministic Release regression tree matching the golden reference; reporting runs only the fast tier with ``ENABLE_COVERAGE=ON``. Coverage gets up to 16 CPUs (one quarter of smaller allocations); the gate splits the remaining CPUs between fast/slow and regression. For 94 CPUs this is 16 coverage, 39 fast/slow and 39 regression. The three CTest memory caps sum to 75% of ``SCI_MEM_GB`` (24/60/60 GiB at 94 CPUs and 192 GiB), and each tree also uses the generated Slurm/cgroup-aware resource spec. Python environments and editable installs are isolated. Only the correctness gate's exit status decides the check. Coverage setup, test, package-install or gcovr failures are reporting failures: the job log and Actions summary show their exit statuses and a warning, while the gate still runs to completion. A successful report is uploaded even if some instrumented tests failed; the summary identifies that incomplete test run. Artifact upload is also non-blocking. Reports return through ``ci-out/`` and ``SCI_OUT`` under the infrastructure's 64 MiB compressed cap, retaining the ``fast-suite-coverage`` artifact's root ``coverage.xml``, ``coverage.html`` and summary layout consumed by Pages. Oversized details fall back to an overview; a still-oversized or failed report is omitted, with its failure recorded. Incomplete XML/HTML artifacts are not uploaded. The short FO tests keep all field-jet, derivative, energy and one-step map checks, and compose 64 steps in the fast tier. Their ``_full`` siblings retain the original 1,000-step compositions in ``slow``. Both use the same strict 1e-7 tangent, 1e-5 finite-difference and 1e-8 correspondence bounds: these test symplecticity/endpoint agreement for a fixed map, rather than a Monte Carlo estimate, so shortening a composition does not require a larger tolerance. The long energy and resolution runs retain every original step. The loss and SPECTRE trajectory drivers also have slow ``_full`` siblings with their original configurations. Fast counts are 8/32 for axis-loss, 16/64 for outer explicit losses, 1/8 for the explicit near-axis smoke check, and 4/24 for SPECTRE trajectories. Trace lengths and per-orbit bounds are unchanged: zero forbidden losses/escapes, energy/trajectory error maxima and exact metadata identities do not acquire a sampling tolerance. The explicit loss-count cross-check retains its existing ``max(2, round(1.5*sqrt(reference_losses)))`` counting-error bound, evaluated for the reduced count; its nonzero-reference-loss guard remains. The stellarator stepper driver runs two periods (96 stored states) for every scheme in fast, requiring zero step errors and finite states. Its ``_full`` case retains the original 7,500-state plotting trace and output files. Memory allocation on workstations and CI ---------------------------------------- Configure on the machine that will run the tests. CMake writes ``build/ctest-resources.json`` with one ``mem_gb`` resource whose slots each represent one GiB. Usable memory is the minimum of Linux ``MemAvailable``, cgroup memory headroom, and any positive Slurm memory allocation (``SLURM_MEM_PER_NODE`` or ``SLURM_MEM_PER_CPU * SLURM_CPUS_ON_NODE``, in MiB). For cgroup v2, CMake locates the job via ``/proc/self/cgroup``, takes the smallest finite ``memory.max`` from that cgroup up to the mount root, and subtracts the job cgroup's ``memory.current``. The v1 fallback uses ``memory.limit_in_bytes`` and ``memory.usage_in_bytes`` under the memory controller, ignoring huge unlimited sentinels. Negative headroom becomes zero. Missing cgroup files impose no constraint; missing ``MemAvailable`` falls back to CMake's available physical memory query. The automatic budget is this usable memory, rounded down to GiB, less a reserve of 25%, rounded up, or 4 GiB, whichever is larger. This leaves room for the OS, other jobs and tests without reservations. Cap this snapshot on a shared machine, and reconfigure when its available memory changes:: cmake -S . -B build -DSIMPLE_TEST_MEMORY_GB=24 ctest --test-dir build/test -j8 --output-on-failure CMake records ``CTEST_RESOURCE_SPEC_FILE`` in every generated ``CTestTestfile.cmake``. Both ``ctest`` from ``build`` and ``ctest --test-dir build/test`` therefore load the generated spec automatically. Thus CI's existing ``ctest -j16`` also uses memory allocation without another command-line option. A site-specific spec can override the default:: ctest --test-dir build/test -j16 --resource-spec-file /path/to/resources.json For example, a 24 GiB budget has this format:: {"version": {"major": 1, "minor": 0}, "local": [{"mem_gb": [{"id": "ram", "slots": 24}]}]} CTest's ``RESOURCE_GROUPS`` alone does **not** limit parallelism without a resource spec; tests still run. This is documented in the `CTest resource allocation manual `_ and verified with CTest 4.4.4. The automatic default above prevents that unbounded fallback. A test requiring more slots than the budget is reported as ``Not Run`` with insufficient resources; the budget is never raised to force it to run. Memory reservations govern scheduling within one CTest invocation, not OS memory limits or separate CTest invocations. Keep OpenMP thread counts appropriate to the CPU allocation as well. Reservations are peak RSS rounded up to whole GiB, measured one registered test at a time with GNU ``/usr/bin/time -v``. Except for the updated wall test, these measurements ran inside ``systemd-run --user --scope -p MemoryMax=16G``. The workstation measurements use GNU Fortran 16.2.1, the deterministic ``Fast`` build, four OpenMP threads and one OpenBLAS thread; golden-record scripts retain their fixed two threads and the wall test retains its fixed single thread. The chartmap measurements use the current fixtures: pipeline shares one VMEC setup with angular multiplier 1; R,Z keeps angular multiplier/order 5 and uses radial order 3 (the boundary remains an exact interpolation node). Other derivative reservations retain their previous conservative bounds from the larger grids. The wall test uses VMEC angular multiplier 1 for initialization; its coil field, chartmap, explicit starts and numerical assertions are unchanged. ============================================== ========== ===== Test RSS (GiB) Slots ============================================== ========== ===== test_splined_field_derivatives 0.8202 1 test_bder_from_splines 0.8191 1 test_magfie_refcoords_splines 1.1990 2 test_hcurl_from_dhcov_splines 0.8194 1 test_splined_kernel_bench 12.6803 13 test_chartmap_pipeline 0.4584 2 test_chartmap_rz_consistency 6.5831 8 gc_ck_ncsx_vmec_vs_meiss 11.9712 12 golden_record_albert_coils 14.0856 15 golden_record_boozer_ncsx 7.2261 8 golden_record_meiss_coils 12.0610 13 test_collisional_stl_wall 0.9169 1 generate_chartmap_ncsx_map2disc 0.3585 1 generate_chartmap_ncsx_map2disc_wall 0.3940 1 test_chartmap_ncsx_map2disc_wall_offset 0.3940 1 test_chartmap_boundary_param 0.3988 1 ============================================== ========== ===== Re-measure when adding NCSX consumers or increasing fixture sizes. Golden cases use ``flock`` (util-linux) only while preparing their shared reference build and input files; their numerical runs share the memory budget. Kinds of tests and labels ------------------------- ========================================= ===================================== Kind (labels) Location: checks ========================================= ===================================== Unit (``unit``) ``test/tests/test_*.f90``: one procedure against an independent oracle Integration (``integration``, ``system``) ``test/tests/``: several modules or ``simple.x`` on a small case; physical invariants Python (``python``) ``test/python/test_*.py`` (pytest): ``pysimple``, ``tools/``, agreement of ``pysimple`` and ``simple.x`` Golden record (``regression``) ``test/golden_record//``: end-to-end output against ``main`` ========================================= ===================================== Add ``slow`` to tests that run longer than about a minute. Register Fortran tests in ``test/tests/CMakeLists.txt`` and pytest files with ``add_pytest`` in ``test/python/CMakeLists.txt``. Oracles and tolerances ---------------------- - The oracle is independent of the implementation: an analytic result, an identity, a roundtrip (``ref -> integ -> ref``), a symmetry, a convergence rate or a second field representation. Rerunning the same code is not an oracle. - A tolerance follows from a physical or numerical scale (``relerr``, spline order and grid spacing, Monte Carlo error :math:`\sim 1/\sqrt{N}`), never from the observed error. - Stochastic results are compared as distributions or moments, not particle by particle. - Data available on only one machine must not be required; missing optional data skips the test. Invariants to keep ------------------ ======================= =================================================== Subsystem Must hold after a change ======================= =================================================== Coordinate transforms ``ref -> integ -> ref`` roundtrip, periodicity, explicit :math:`s`, :math:`\sqrt{s}` or :math:`\rho` at every interface VMEC, Boozer, chartmap same start surface, same short-time orbit within tolerance, statistically equal loss fraction Symplectic integrators bounded invariants, no drift beyond the reference envelope Collisions Maxwellian fixed point, analytic pitch and energy relaxation, seeded reproducibility Wall losses same wall hit in Cartesian and coordinate space NetCDF output unchanged meaning, units and frame of ``zend``, ``xend_cart``, ``wall_hit_cart`` GPU and OpenMP CPU path unchanged when acceleration is off, short-run CPU/GPU agreement, long-run statistical agreement, no CPU slowdown ======================= =================================================== Golden records -------------- ``golden_record_`` clones and builds ``main`` under ``build/``, runs ``test/golden_record//simple.in`` with both builds and compares the outputs with relative tolerance ``1.5e-7``. The first run needs network access and several minutes. Use ``make test-regression TEST=golden_record_``: it pins the reference libneo revision (``libneo_golden`` in ``cmake/dependency_versions.json``) for both builds. Patches in ``test/golden_record/reference_patches/`` make old reference revisions buildable and comparable. The stochastic ``collision`` case uses its own acceptance rule (:doc:`/collision-golden-contract`). Add a case when a new field type, integrator or output path becomes part of the supported workflow. A change of golden output is a scientific change and must be explained in the pull request (:doc:`/contributing`). The bounded numerical verification report and its evidence table are in :doc:`/verification`.