Measured in a lab or the field1D · mixedCandidate

Real sewer network, Odense

Does a real sewer model, run as published, reproduce the water levels measured in eight storms?

Test A1F-100
Pedersen (2021)
Version 1.0.0

Right answer: Laboratory or field measurements · Pedersen (2021)
Results: 3 models · closest 30 % off · see them ↓
Must: track the measured levels (Nash–Sutcliffe efficiency at least 0.5 for the typical meter and storm)

Sketch, not to scale.

The model file

The published Bellinge model with the dates of the 29 June 2012 storm set, as the suite ran it. It reads the rain-gauge file that comes with the model from DTU Data; the other seven storms differ only in their dates.

Download BellingeSWMM_v021_nopervious_2012-06-29.inp 1.2 MB Open it on the map →

On the map: click any element to see its line in the file and every engine's result there over time.

What happens

The Bellinge combined sewer system in Odense, Denmark (about 1,000 manholes and pipes, with basins, pumps and weirs) has a published SWMM model and years of level-meter records. Each model runs that SWMM model unchanged, apart from its dates, on eight storms from 2011 to 2019, and its water levels are compared with seven level meters.

Why it matters

Field data is the final test of a drainage model: real rainfall, real pipes, real measurements. The published model is uncalibrated, so this test judges the model and the engine together. A miss here can come from the model's set-up as much as from the engine.

A good model must

track the measured levels (Nash–Sutcliffe efficiency at least 0.5 for the typical meter and storm)
get the typical peak level within 0.1 m

Measurements and models

Water level at meter G71F68Y_LevelPS (node G71F68Y), storm of 2012-06-29

Measured level (blue) and each engine running the published model. The meter reads no lower than 11.18 m; the scorer reads the model the same way.

Point at the chart to read every model's value; click a name to hide or show it.

Water level at meter G71F05R_LevelBasin (node G71F05R), storm of 2012-06-29

Measured level (blue) and each engine running the published model. The meter reads no lower than 14.14 m; the scorer reads the model the same way.

Point at the chart to read every model's value; click a name to hide or show it.

Water level at meter G80F13P_LevelPS (node G80F13P), storm of 2012-06-29

Measured level (blue) and each engine running the published model. The meter reads no lower than 33.32 m; the scorer reads the model the same way.

Point at the chart to read every model's value; click a name to hide or show it.

How models did

In rank order: by each model's percent difference from the right answer, the largest over the test's accuracy checks (lower is better; results within 10 % of each other share a rank). Mass error, timing and NSE are shown beside it, not folded in. Each model runs the same inputs. Compare them in the results viewer →

Rank	Model	Difference	Largest difference in	NSE	Run time
1=	EPA SWMM 5.2.4 dynamic wave	30 %	Peak level	-0.488	–
1=	OpenSWMM 6 dynamic wave	30 %	Peak level	-0.488	–
1=	Hydra urban drainage	34 %	Peak level	-0.528	–

This is a validation of the published Bellinge model against its gauges, not a comparison of engines: the SWMM-family engines run the same model file and land within each other's build noise (two builds of EPA SWMM 5.2.4 from the same source differ by more than 5 cm in peak depth at 75 of the model's 1,020 nodes, as Bob Dickinson measured with his Bellinge Explorer, 17 October 2026). Field data is kept off the overall rating.

EPA SWMM 5.2.4

EPA SWMM 5.2.4 with dynamic-wave routing; on the channel tests, the channel is written as one conduit per grid cell

#1= 30 % off

Settings: swmm (the published Bellinge model, run dates set per event)

Fit to measurements

· NSE -0.488

skill score (1 is perfect)

Peak level

• 30 cm off

30 % of the median gauge's 0.97 m level rise in a storm

Scored with method 3.0 · Score card: EPA SWMM 5.2.4

Run it yourself

EPA SWMM 5.2.4 · runs on Linux · produced by scripts/run_bellinge.py in bellinge.yml.

git clone https://github.com/scott-jeffers/benchmark1d2d && cd benchmark1d2d
git checkout 71dc10fc07a9
python -m pip install -e ".[dev,engines]"
python scripts/bellinge.py fetch --out out/bellinge
python scripts/run_bellinge.py --model-dir out/bellinge/model/7_SWMM --engines swmm524

The model files the suite wrote for this run (.zip, 1,446 KB): the exact inputs the solver read (one folder per grid, where the test has several), with their SHA-256.

The model files below are the published model with each storm's dates set. The rain-gauge file they read (rg_bellinge_Jun2010_Aug2021.dat, 28 MB) is too large to include; the fetch step downloads it with the model from DTU Data.

The test itself (inputs, exact answer, checks): A1F-100.zip. Check the score card with benchmark1d2d verify-card: its signature by the suite's public development key shows the card has not changed since it was written, not who wrote it.

OpenSWMM 6

OpenSWMM Engine v6 (HydroCouple's SWMM 6, pre-release) with dynamic-wave routing

#1= 30 % off

Settings: swmm (the published Bellinge model, run dates set per event)

Fit to measurements

· NSE -0.488

skill score (1 is perfect)

Peak level

• 30 cm off

30 % of the median gauge's 0.97 m level rise in a storm

Scored with method 3.0 · Score card: OpenSWMM Engine v6 dynamic wave (6.0.0a4.dev1)

Run it yourself

OpenSWMM Engine v6 dynamic wave (6.0.0a4.dev1) · runs on Linux · produced by scripts/run_bellinge.py in bellinge.yml.

git clone https://github.com/scott-jeffers/benchmark1d2d && cd benchmark1d2d
git checkout 71dc10fc07a9
python -m pip install -e ".[dev,engines]"
python -m venv .venv-openswmm
.venv-openswmm/bin/python -m pip install "openswmm==6.0.0a4.dev1" numpy
python scripts/bellinge.py fetch --out out/bellinge
python scripts/run_bellinge.py --model-dir out/bellinge/model/7_SWMM --engines openswmm-dw --openswmm-python .venv-openswmm/bin/python

The model files the suite wrote for this run (.zip, 1,446 KB): the exact inputs the solver read (one folder per grid, where the test has several), with their SHA-256.

Hydra

Hydra 17 (NEER AI, Rust): its urban-drainage engine on the SWMM data model, run through its CLI with default settings

#1= 34 % off

Settings: swmm (the published Bellinge model, run dates set per event)

Fit to measurements

· NSE -0.528

skill score (1 is perfect)

Peak level

• 33 cm off

34 % of the median gauge's 0.97 m level rise in a storm

Scored with method 3.0 · Score card: Hydra urban drainage (17.0.0/4.1.1)

Run it yourself

Hydra urban drainage (17.0.0/4.1.1) · runs on Linux · produced by scripts/run_bellinge.py in bellinge.yml.

git clone https://github.com/scott-jeffers/benchmark1d2d && cd benchmark1d2d
git checkout 71dc10fc07a9
python -m pip install -e ".[dev,engines]"
HYDRA=$(python scripts/fetch_hydra.py .hydra)  # the pinned Hydra 17 CLI
python scripts/bellinge.py fetch --out out/bellinge
python scripts/run_bellinge.py --model-dir out/bellinge/model/7_SWMM --engines hydra-uds --hydra-exe "$HYDRA"

The model files the suite wrote for this run (.zip, 1,446 KB): the exact inputs the solver read (one folder per grid, where the test has several), with their SHA-256.

Not run yet: HEC-RAS 7.0, TELEMAC-2D, ANUGA, GeoClaw, LISFLOOD-FP, TUFLOW, MIKE 21, Iber, BASEMENT, Delft3D FM

Run it and add yours →

Status of this test

Candidate. To become frozen it needs: a published run passes; two independent codes pass; maintainer review recorded.

Download inputs (.zip)Cite this test View on GitHub

Technical details for reviewers Checks, reference, inputs, sources

Checks and why each limit was chosen

Check	Difference measured as a percentage of	Latest	Difference	Why this check
level_nse NSE of wse · skill score, beside the score	not a percentage: reported as the NSE (1 is perfect)	–		Set before any run: Moriasi et al. (2007) 'satisfactory' band, NSE ≥ 0.5, for the median gauge record (a meter on one storm). A record where the meter moved less than three times its accuracy is not scored for NSE.
peak_level peak_error of wse · in the score	of the median gauge's 0.97 m level rise in a storm	–		Set before any run: the ±0.1 m depth criterion for verifying sewer models against measurements (WaPUG 2002; CIWEM UDG 2017 Code of Practice), for the median gauge record.

Reference

Measured data at reference/levels.parquet; availability: available.

Promotion checklist

Only frozen tests have fixed inputs and answers. This list is computed from the manifest and the published score cards; the rules are in the method.

✓Data licence verifiedfor candidateCC-BY-4.0
✓All inputs availablefor candidateall committed or fetched by SHA-256
✓Reference (truth) availablefor candidatecommitted measured data
✓Every tolerance justifiedfor candidateeach gating criterion has a rationale
✗A published run passesfor frozenno passing score card published yet
✓Order of accuracy shownfor frozennot a verification case
✗Two independent codes passfor frozenpassing codes: none — needs a second, independently written solver or engine
✗Maintainer review recordedfor frozenno maintainer review recorded

Inputs

File	Role	Format	SHA-256	Get it
`inputs/7_SWMM.zip`	network	zip	`fe07d487925c…`	Fetched by hash from source
`inputs/2_cleaned_data.zip`	observations	zip	`0801435a0077…`	Fetched by hash from source

About this test

The real combined sewer system of Bellinge, Odense (Denmark). It covers 1.7 km² and has about 1,000 nodes and 1,000 links, with basins, pumps, weirs and control rules. Pedersen et al. (2021, ESSD 13, 4779–4798) published it with an EPA SWMM model and years of level-meter records.

Licence: read before using

DTU item	Content	Licence	Used
2	Sensor data	CC BY 4.0	the measured levels (`reference/levels.parquet`)
3a	Rain gauge data	CC BY-NC 4.0 (non-commercial)	no
3b	Radar and meteorological station	CC BY 4.0	no (not needed)
7	Models (SWMM, MIKE URBAN)	CC BY 4.0	the SWMM model and its rain-gauge file

The case uses only CC BY items so commercial vendors can run it.

The test

Model. BellingeSWMM_v021_nopervious.inp from item 7 (SHA-256 of the zip in the manifest), run unchanged except for its run dates. scripts/run_bellinge.py writes each event's dates and a one-minute report step into a copy. The model's own rain-gauge file, rg_bellinge_Jun2010_Aug2021.dat, drives every run.
Storms. There are eight. Each is the most intense 10-minute rainfall or one of the largest daily totals in the model's gauge series after the level meters start in August 2010, and the list includes the model's own event of 29 June 2012. Each run starts 12 hours before the storm's day as a warm-up, and its levels are scored over the next 48 hours.
Level meters. There are seven, each read at the model node it sits in.

Meter	Model node
G71F04R_Level2	G71F04R
G71F05R_LevelBasin	G71F05R
G71F05R_LevelInlet	G71F05R_skyl
G71F06R_LevelInlet	G71F06R
G71F68Y_LevelPS	G71F68Y
G73F010	G73F010
G80F13P_LevelPS	G80F13P

Not used: G71F04R Level1 (the same basin as Level2, with fewer records), G71F05R_position (a gate) and G71F68Yp1 (a pump). - Reference. reference/levels.parquet holds 138,154 one-minute levels, in m above DVR90, for each meter and storm with a whole record (90 % or more). Build it with scripts/bellinge.py extract and then scripts/bellinge.py reference. - Floor. A meter does not read below its own floor, taken as the lowest level it recorded in any of the storms. The scorer reads the model's level the same way, as max(model, floor).

Criteria

Both criteria were set before any run, from published standards rather than from a baseline. Both are judged on the median gauge record (one meter on one storm).

Level NSE ≥ 0.5. This is the "satisfactory" band of Moriasi et al. (2007). A record where the meter moved less than three times its accuracy (0.06 m) is not scored for NSE.
Peak level within ±0.1 m. This is the depth criterion for verifying sewer models against measurements (WaPUG 2002; CIWEM UDG 2017 Code of Practice).

The published model is uncalibrated (ESSD paper §4), so this case judges the model and the engine together. A failure here can come from the model's set-up as much as from the engine; the parity comparison on the same model (the Bellinge tier on the parity page) isolates the engine.

First results (October 2026)

EPA SWMM 5.2.4, OpenSWMM 6 (dynamic wave) and Hydra 17 all fail, by almost the same amounts. The median NSE is −0.49, −0.49 and −0.53, and the median peak error is 0.30, 0.30 and 0.33 m. The three engines agree with each other far better than any of them agrees with the meters, so the misses come from the published model, not from the engines.

Median over the eight storms, EPA SWMM 5.2.4:

Meter	NSE	Peak error
G71F68Y_LevelPS (pump sump)	0.81	−0.04 m
G73F010 (one storm)	0.45	−0.02 m
G71F06R_LevelInlet	−0.52	+0.05 m
G71F05R_LevelInlet	−0.83	−0.26 m
G71F04R_Level2 (basin)	−0.54	−1.21 m
G71F05R_LevelBasin (basin)	−0.41	−2.56 m
G80F13P_LevelPS (pump sump)	−1.79	+0.09 m

The two basins fill 1.2–2.6 m less than measured in every storm. The published model is the nopervious variant, which leaves out runoff from pervious surfaces, and that is the likeliest cause. Calibrating the model is outside this case, which runs the model as published.

History

1.0.0: first scored version. The rainfall is the model's own gauge file rather than the radar. Peak levels are judged in metres, not per cent.
0.1.0: draft.

Sources

Pedersen, A. N., Pedersen, J. W., Vigueras-Rodriguez, A., Brink-Kjær, A., Borup, M., Mikkelsen, P. S. (2021). The Bellinge data set: open data and models for community-wide urban drainage systems research. doi:10.5194/essd-13-4779-2021
Pedersen, A. N., Pedersen, J. W., Vigueras-Rodriguez, A., Brink-Kjær, A., Borup, M., Mikkelsen, P. S. (2021). Dataset for Bellinge: An urban drainage case study. doi:10.11583/DTU.c.5029124

Test A1F-100_bellinge_urban_drainage · version 1.0.0 · data licence CC-BY-4.0