Do pumps switch on and off at the same moments as in SWMM 5.2.4?
Every engine runs this EPA SWMM 5 input file unchanged. Open it in EPA SWMM 5, or in any engine that reads the format, and you are running exactly what was scored.
On the map: click any element to see its line in the file and every engine's result there over time, or colour the network by how far each engine is from the reference.
Each step is a rule firing. A candidate that evaluates the rules at different moments moves these steps, which shows up in pumped volume and peak timing.
A 90-node sewer with two pump stations whose pumps are switched by control rules on wet-well level; one pump switches 74 times in 36 hours.
Control rules decide overflows and energy use; a rule evaluated at a different moment changes both.
In rank order: by each model's percent difference from the right answer, the largest over the test's accuracy checks (lower is better; results within 10 % of each other share a rank). Mass error, timing and NSE are shown beside it, not folded in. Each model runs the same inputs.
| Rank | Model | Difference | Largest difference in | Timing | Run time |
|---|---|---|---|---|---|
| 1 | SWMM 5.1.15 previous release | <0.01 % | Peak depths | 0 s | – |
| 2 | OpenSWMM 6 vs SWMM 5.2.4 | 51 % | Peak depths | 180 s | – |
| 3 | Hydra urban drainage | 90 % | Peak depths | 101,460 s | – |
| 4 | OpenSWMM 6 finite volume | 3e4 % | Volumes | 101,220 s | – |
Settings: swmm5 (swmm-toolkit 0.10.0)
Scored with method 3.0 · Score card: EPA SWMM 5.1.15
The badge always shows the latest published result.
Settings: openswmm (v6 engine, SWMM binary output)
Pumps switch the same 74 times, but one node peaks 50 % higher than in SWMM 5.2.4, at a different time; worth a look by the OpenSWMM developers.
Scored with method 3.0 · Score card: OpenSWMM Engine 6.0.0a4.dev1
The badge always shows the latest published result.
Settings: hydra CLI 4.1.1 (uds engine, SWMM-format binary output)
Scored with method 3.0 · Score card: Hydra 17.0.0 urban drainage
The badge always shows the latest published result.
Settings: openswmm (v6 engine, FLOW_ROUTING FV, SWMM binary output)
Scored with method 3.0 · Score card: OpenSWMM Engine v6 finite volume (6.0.0a4.dev1)
The badge always shows the latest published result.
Candidate. To become frozen it needs: reference reproduced by a second build; maintainer review recorded.
| Check | Difference measured as a percentage of | Latest | Difference | Why this check |
|---|---|---|---|---|
| peak_depth relative_peak of h · in the score | already a percentage | – | Architecture parity band (±2 % peak) at every node, wet wells included. | |
| link_volume relative_volume of Q · in the score | already a percentage | – | Pumped and conveyed volume within ±2 % in every link: a rule that fires at the wrong level changes how much each pump moves. | |
| peak_timing timing_error of Q · timing, beside the score | not a percentage: reported in seconds | – | Four 30 s reporting steps; a rule evaluated at a different moment shifts pump-on times. |
Second build of the reference. The official EPA SWMM 5.2.4 official win64 binaries, run on this input and scored against the reference: fail (Peak depths 3.7 %; Volumes 1.2 %; Timing 92,400 s). The two builds of the same version disagree beyond this test's limits.
SWMM 5.2.4, run on the test's own input.
Only frozen tests have fixed inputs and answers. This list is computed from the manifest and the published score cards; the rules are in the method.
| File | Role | Format | SHA-256 | Get it |
|---|---|---|---|---|
inputs/network.inp | network | inp | 6756016d98e6… | Fetched by hash from source |
Two pump stations (wet wells PP_U433, PP_U435) are run by level rules, for example:
RULE U433_R2
IF NODE PP_U433 DEPTH >= 1.778
AND NODE PP_U433 DEPTH < 2.032
AND PUMP U433_Psan_1_et_2 STATUS = OFF
AND PUMP U433_Psan_1_et_2_et_3 STATUS = OFF
THEN PUMP U433_Psan_1_ou_2 STATUS = ON
On SWMM 5.2.4 pump U433_Psan_1_ou_2 switches 74 times in the 36 h run, so a candidate that evaluates rules late,
early or with different hysteresis shows up in pumped volumes and peak timing.
Source: HydroCouple openswmm.engine.benchmarks, epa/221-h-h-si-units-elements, commit 63eb8bc5b4e741c82195ed65c4d7d146b86da1cc (fetched by hash with
python scripts/fetch_inputs.py B-RTC-001). Reference: EPA SWMM 5.2.4.
The cross-build job (scripts/run_swmm_self_parity.py) runs the official EPA
SWMM 5.2.4 win64 binaries on this input and scores them against the reference
(produced by the same 5.2.4 source compiled in swmm-toolkit 0.17.0). Both
builds repeat bit-for-bit, but they disagree with each other beyond this case's
tolerances: −3.7 % peak depth at Out_U435, and the Ref_U435 flow peak moves
by 92,400 s because two near-equal pump-cycle peaks swap order. Depth-triggered
pump rules amplify last-bit floating-point differences between compilers.
Consequences: the cross-build check for this case is informative, not gating; and a candidate that fails B-RTC-001 by a similar margin is not necessarily wrong. Whether to widen these tolerances, switch the timing metric to one that is robust to near-equal peaks, or score against both builds is an open decision for the maintainers.
Test B-RTC-001_pump_station_rules · version 2.0.0 · data licence Unlicense