Reports

Score cards

Download every score card (.zip). Check one with benchmark1d2d verify-card path/to/card.json.

Second build of the reference engine

The official EPA SWMM 5.2.4 Windows binaries run on each SWMM catalog input and are scored against the reference built from the same source in swmm-toolkit. Agreement shows a parity reference does not depend on the compiler.

Audit log

End-to-end reviews found the following defects. Each is fixed, covered by tests, and recorded here because earlier results published from this repository were affected.

Second review, 14 October 2026

1. Six exact-solution cases had never been scored, and their specifications could not be met

   Found: Darcy–Weisbach, Todini network, Ogata–Banks tracer, Bélanger jump, MacDonald 2D and oblique jump had references but no runs. The pipe cases asked for grid convergence of a steady closed-form solution, and their "1 %" tolerances were compared against absolute errors in pascals.

   Now: Each case is re-specified for its physics: relative norms when a tolerance is in %, a head-loss metric, element-keyed network references, and "convergence not applicable" where nothing is discretised. EPA EPANET 2.2 passes both pipe cases (head loss 0.60 % high, the Swamee–Jain offset theory predicts); new demo solvers pass the tracer (order 1.67), the jump (1.00), MacDonald 2D (1.02) and the oblique jump (0.91).

2. The MacDonald 2D reference was not a solution of the shallow-water equations

   Found: It held the free surface flat while water flowed at 0.2 m²/s without friction, which momentum forbids; the module's docstring admitted that a solver copying the field would pass.

   Now: Rebuilt by MacDonald's method: A1F-001's profile on a 2D grid with Manning friction and a bed constructed so the steady equations hold exactly (momentum residual tested). Case version 2.0.0.

3. The oblique-jump reference used the wrong jump relation and the wrong side

   Found: It solved the gas-dynamics θ–β–M relation (jump at 34.0° instead of the shallow-water 39.6° for Fr₁ = 2.5, θ = 15°) and labelled the region between wall and jump as upstream.

   Now: Rewritten from the exact shallow-water Rankine–Hugoniot conditions, with tests for mass, normal momentum and tangential velocity across the jump. Case version 2.0.0.

4. The Joukowsky "continuity" check measured front smearing, not mass

   Found: Mean |u| where p exceeds half the rise grows with the smeared zone, so accurate first-order schemes failed (1.87 %) while a non-conserving sharp scheme could pass.

   Now: Replaced by an exact storage balance: the packed volume ∫Δp dx must equal ρa²V₀t. The method-of-characteristics demo conserves it to round-off. Case version 3.0.0.

5. Two builds of SWMM 5.2.4 disagree on the control-rule case

   Found: The official win64 binaries and the swmm-toolkit build match EXTRAN Example 1 exactly, but on B-RTC-001 differ by −3.7 % peak depth and move a pump-cycle peak by 92,400 s.

   Now: Reported every run by the cross-build job; B-RTC-001 cannot be frozen until its tolerances or timing metric are revised (decision recorded in the case README).

6. The PR corpus tier tested one collection only

   Found: All 38 PR-tier models were legacy EXTRAN examples, on which SWMM 5.1.15 and 5.2.4 give bit-identical output, so the gate could not see most engine changes.

   Now: The PR tier is now 40 models picked to cover feature tags (9 collections, ≥ 5 control-rule models). A corpus parity runner scores any SWMM build against 5.2.4 across a tier; a self-check requires exactly zero against itself on every PR run.

7. Nothing said how a case becomes frozen

   Found: Only frozen cases count, but there was no rule for getting there, so no case could ever be frozen on evidence.

   Now: Eight promotion rules, seven computed from the manifest and score cards; every case page shows its checklist and the catalog lists what each case still needs.

First review, 13 October 2026

1. Track B parity compared results with themselves

   Found: Every non-analytic case (all of Track B, every measured case) was scored by comparing the solver's output with itself, so any output passed and earned a B-1 % claim.

   Now: Parity now compares per element against the pinned engine's reference results; a case without its reference is not scored rather than passed. Regression tests cover pass, perturbed-fail and missing-reference.

2. Error norms inflated every observed order by one

   Found: L1 was computed as mean(|e|) × dx, adding a power of dx; reported orders were +1 (L1) and +0.5 (L2) too high.

   Now: Norms are domain-normalised. True orders: Ritter 0.95, Stoker 0.95, MacDonald 1.99, Joukowsky 0.49.

3. Two of four 1D 'passes' never solved anything

   Found: The demo MacDonald and Joukowsky solvers returned the analytical solution; zero error was then reported as order 1.0 and passed.

   Now: Real solvers: Heun integration of the gradually-varied-flow equation and a method-of-characteristics water-hammer solver. Convergence now refuses round-off errors on every mesh.

4. The Thacker reference violated mass conservation

   Found: The bowl solution used ω = √(8gh₀)/L and omitted a cos² term; continuity residual was 83 % of ∂h/∂t, so every correct solver would fail.

   Now: Corrected solution (ω = √(2gh₀)/L) with pointwise PDE-residual tests. A real 2D finite-volume solver now passes with order 0.91.

5. The Joukowsky convergence target was unreachable

   Found: A minimum order of 0.8 cannot be met by any first-order fixed-grid scheme on a linear discontinuity (theory gives 0.5).

   Now: Target set from theory to 0.5 / 0.4 minimum; case version bumped to 2.0.0 because scores are not comparable.

6. EA Tests 1, 2, 4, 5 were filed as laboratory validation

   Found: These are code inter-comparison tests with no measured truth; inputs were empty placeholders.

   Now: Reclassified as A2C code inter-comparison with real inputs from the official archive (by hash) and objective expected states: settled pond level and mass conservation.

7. Placeholder and mis-licensed inputs

   Found: Most inputs were 0-byte files; Weber, Bergant and Rubinato data were labelled Unlicense; Bellinge rain-gauge data is CC BY-NC.

   Now: Real case definitions or explicit pending inputs that block promotion; journal-only data marked TBD-confirm; the CC BY-NC item is excluded.

8. Two Track B cases did not test what they claimed

   Found: B-EXTRAN-001 was a 2-conduit synthetic network presented as EXTRAN Example 1; in B-RTC-001 the rule's condition never changed, so no control switch was ever exercised.

   Now: B-EXTRAN-001 now runs the real EXTRAN Example 1; B-RTC-001 is a pump-station network whose rules switch one pump 74 times in 36 h.

9. Gauges on cell faces mixed two cells

   Found: A probe exactly between cells took depth from one cell and bed from another, producing ±4 cm level errors in a flat pond.

   Now: Ties resolve to a single cell; regression test added.

10. The PR tier contained models that cannot serve as references

    Found: 11 of its 49 corpus models were unfit: 4 do not run on SWMM 5.2.4 (input errors) and 7 run with continuity errors of 6.6 % to 100 %. The PR gate could never have passed, and agreeing with a non-conserving reference proves nothing.

    Now: A full census runs every corpus model on SWMM 5.2.4; tiers admit only models that run cleanly with |continuity| ≤ 5 %. The PR tier is now 38 corpus models (7 with control rules) plus the 3 catalog cases.

11. Signatures implied authorship

    Found: All published cards were signed with development keys hard-coded in the repository, and the documented verify command checked manifests, not cards.

    Now: benchmark1d2d verify-card checks the content hash and says when a card carries a public development key; the site no longer presents those signatures as proof of origin.

12. Every pass was labelled A-V

    Found: Validation and parity passes received verification or fixed B-1 % claims.

    Now: Claims follow the case class (A-V, A-C, A-L, A-F) and the gating tolerance (B-x %).