How it works

A benchmark is only useful if its answers are right and its results can be repeated. Here is how this one makes sure of both.

01 — Same inputs

Every model gets identical terrain, flows and settings. Each file is fixed by a SHA-256 fingerprint, so nobody can quietly change it, and large files are fetched and checked against that fingerprint.

02 — A known right answer

Exact mathematical solutions (dam breaks, backwater, water hammer), laboratory and field measurements, or — for compatibility — the output of a pinned SWMM or EPANET release.

03 — A result anyone can check

Every number measured, as a percent difference from the right answer, with the run time, the machine and the engine's own report. Score cards are hashed and signed (with the suite's public development key: the signature shows a card has not changed, not who wrote it); every one shown here is re-run by CI on each change.

How results are scored and ranked

Measurements, not verdicts

Each check's error is given as a percentage of the quantity it is an error in (a depth error over the exact mean depth, a front error over the distance the front travels; mass errors are already percentages). A result's score is its largest percent difference over the test's accuracy checks; there is no limit and no pass or fail. Models are listed in that order on every test; results within 10 % of each other share a rank. Overall rankings count head-to-head wins on the tests two models share, so a model is never compared on tests it did not run.

Getting better on finer grids

Each test is run on up to four grids and scored on the finest. Every card lists the error on each grid and the rate it shrinks at (the "observed order", as in ASME V&V 20), beside the scheme's design order. The rate is reported, not counted in the score.

Each check has a reason

Every check is written down with why it was chosen and what its percentage is of (the exact mean depth, the distance a front travels). Mass errors, timing errors (in seconds) and NSEs are shown in their own columns, not folded into the score. Open any test's technical details to read them.

Production software, run fairly

Production models are installed from their official sources by automated jobs and run with their defaults. The inputs, the run length, the boundaries and the output format are set for every model; any other setting the suite changes (a time-step limit, a minimum surface area, an iteration tolerance) is named on each result under "Settings". When a model fails, its card says so and why, and the run can be reproduced from the repository.

HEC-RAS 7.0 (and 6.6)

Installed silently on a Windows runner from HEC's release (installer fingerprint pinned), projects written as plain text, run headless with ras-commander. 6.6 is kept as the previous release.

TELEMAC-2D 9.1

openTELEMAC from conda-forge, finite-volume settings by default.

ANUGA, GeoClaw, LISFLOOD-FP

Built from their official releases weekly; LISFLOOD-FP's three CPU solvers are scored separately.

SWMM and EPANET

Pinned releases (SWMM 5.2.4, EPANET 2.2) define the compatibility answers; a second build of each checks the answer does not depend on the compiler.

How tests grow up

Draft · 8

Defined, but waiting on data or a verified answer.

Candidate · 31

Inputs, answer and limits published, and at least one model scored.

Frozen · 0

Reviewed, run by independent models, and fixed: a change to a frozen test makes a new version.

39 tests in all. The rules for each step are in the method.

Go deeper

Method