OpenHydro Bench

Method

How tests are verified and validated, how limits are chosen, and the literature behind every test.

Track A — Verification (analytic)

Per ASME V&V 20 and Roache GCI:

  1. Run the solver on ≥ 3 systematically refined meshes (fixed Courant).
  2. Compute domain-normalised L1 / L2 / L∞ norms against the exact solution (L1 = mean |e|, so a scheme of order p gives L1 ∝ dx^p).
  3. Estimate the observed order of accuracy p from successive levels.
  4. Compute the Grid Convergence Index (GCI).
  5. Judge with scoring method 2: accurate if every gating criterion passes; converges if the order fitted through every grid has its whole 90 % range at or above min_observed_order, fails if the whole range is below it, and is inconclusive otherwise. The result passes when it is accurate; it is verified (and may carry a certification claim) when it also converges.

A solver that hits 1 % L1 on a fine grid but converges at order 0.4 fails. Errors at round-off on every mesh are rejected: a discretisation always leaves truncation error, so such results are the reference itself.

Every exact solution in references/ is tested pointwise against the governing equations, not only against its own initial state.

Track A — Code inter-comparison

Some widely used benchmarks (Environment Agency Tests 1–5) have neither an exact solution nor measurements; the original exercise compared packages with each other. These cases (A2C-) gate only on states that follow from the inputs: mass conservation in a closed domain, and levels fixed by hydrostatics (a pond settling at its sill crest). Published package results are context, never truth.

Track A — Validation (lab / field)

Metrics: NSE, KGE, peak error, arrival-time error, CSI (inundation). Thresholds may set include_measurement_uncertainty: true so a model is not failed for disagreeing with a gauge by less than the gauge's own error.

Track B — Industry parity

Compare against a pinned reference engine (EPA SWMM 5.2.4, EPANET 2.2; build and binary SHA-256 recorded). Metrics are computed per element (every node and link) and the worst element is reported: relative peak, conveyed volume, peak timing. Reference results are produced by scripts/make_reference.py and checked in CI — never hand-edited.

Only frozen cases count toward certification claims (see GOVERNANCE.md). A case moves up when it meets every rule for the next status. Rules 1–7 are computed from the manifest and the published score cards (benchmark1d2d.catalog.promotion), so each case page shows exactly what blocks it. Rule 8 is a person's sign-off.

draft → candidate

# Rule Why
1 Data licence verified (not TBD-confirm) We cannot redistribute or publish scores on data we have no right to use.
2 Every input committed, or fetchable by SHA-256 (nothing pending) Anyone must be able to run the case byte-for-byte.
3 A reference exists: analytic generator, objective expected states, committed measured data, or pinned-engine results Without truth there is nothing to score against; such a case is not scored, never passed.
4 Every gating criterion states why its tolerance is what it is A tolerance without a reason cannot be reviewed or defended.

candidate → frozen

# Rule Why
5 A published run passes the case Shows the case is passable as specified.
6 Verification cases: that run's observed order of accuracy meets the minimum (or the manifest explains why order does not apply) Pointwise agreement at one grid can be luck; convergence cannot.
7 Two independently written codes pass. For parity cases: a second build of the reference engine (the official EPA binaries, scored by the cross-build job) reproduces the reference A case that only one code passes may be tuned to that code; a parity reference that depends on the compiler is not a stable target.
8 A maintainer review is recorded in the manifest (review: {by, date, notes}) Someone other than the author has checked the physics, the reference and the tolerances.

A frozen case is never edited in place. Any change to its inputs, reference or tolerances bumps case_version, returns it to candidate, and is listed in the case README's history, because scores across versions are not comparable.

Where the catalog stands

Most candidates are blocked only by rules 7 and 8: each has one passing code (the demo solvers, EPA EPANET 2.2, or SWMM 5.1.15 for parity) and no review yet. Drafts are blocked by data licences or by measurements still to be obtained from their authors. B-RTC-001 additionally fails rule 7: the official SWMM 5.2.4 build and the swmm-toolkit build disagree on it beyond tolerance.

Scoring method 2.1 (16 October 2026) decides every verdict: a result passes when every check is within its limit, each check shows its share of the limit, and the convergence rate, fitted through every grid with its 90 % range, decides the separate verified label. See scoring method 2 and the page that shows every result both ways. The limits below are unchanged, except A1F-010's front (9.8 m since v3.0.0).

Every scoring criterion carries a rationale string. That string appears on the public test page. A threshold is never a bare number.

Principles

  1. Verification: thresholds are in physical units (metres, %, Pa) and are justified by mesh spacing, truncation error estimates, or published acceptance criteria. Convergence order is always required.
  2. Validation: thresholds absorb stated measurement uncertainty when include_measurement_uncertainty is true.
  3. Parity: thresholds are relative to the pinned reference engine's own round-off / continuity behaviour (typically ±0.5–2 % on peak and volume).
  4. Change control: editing a tolerance requires updating the rationale and a reviewer sign-off before the case can return to frozen.

Examples

Criterion Threshold Rationale
Ritter front position abs ≤ 2.5 m Two finest cells; exact front at 1375.8 m for h0=1, x0=1000, t=60, g=9.81
Global mass error abs ≤ 0.1 % Conservation property of the continuous equations
Track B peak depth abs ≤ 1 % Within SWMM 5.2.4 self-consistency band on EXTRAN cases
Joukowsky observed order ≥ 0.4 (design 0.5) A discontinuity in a linear wave equation: a p-th order fixed-grid scheme converges in L1 only as dx^(p/(p+1)), so 0.5 for first-order schemes
EA Test 2/4/5 mass abs ≤ 0.1 % Closed domain, dry start: stored volume must equal the inflow integral
EA Test 1 pond level abs ≤ 0.02 m Hydrostatics: the pond settles at the 10.25 m sill crest

Expectations come from theory, not from results

A tolerance or expected order is set from analysis of the problem (regularity of the solution, design order of a reasonable scheme), never tuned to a particular solver's output. When an expectation is found to be wrong, the case's major version is bumped and the reason is written into its description, because scores before and after are not comparable (example: A1P-020 v2.0.0).

Cross-build sensitivity (Track B)

Two builds of the same engine version are not guaranteed to agree. For EPA SWMM 5.2.4 the official win64 build and the swmm-toolkit 0.17.0 build agree exactly on EXTRAN Example 1, but differ on the control-rule case B-RTC-001 by −3.7 % peak depth and a 92,400 s peak-timing shift. A tolerance tighter than the spread between reference builds cannot separate a correct candidate from a wrong one, so B-RTC-001's ±2 % / ±120 s criteria are under review (see the case README).

ASCE author-date style. Entries marked † were checked against the live repository, publisher or agency page on 13 Oct 2026; all others are canonical references cited from standard literature and should pass an automated Crossref DOI check (Phase 0) before submission to ASCE/EPA.

A. Verification & validation methodology

  • ASME (2009, reaffirmed). V&V 20-2009: Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer. American Society of Mechanical Engineers, New York.
  • Oberkampf, W. L., and Roy, C. J. (2010). Verification and Validation in Scientific Computing. Cambridge University Press, Cambridge, UK. https://doi.org/10.1017/CBO9780511760396
  • Roache, P. J. (1994). "Perspective: A method for uniform reporting of grid refinement studies." J. Fluids Eng., 116(3), 405–413. https://doi.org/10.1115/1.2910291
  • Roache, P. J. (1998). Verification and Validation in Computational Science and Engineering. Hermosa Publishers, Albuquerque, NM.
  • Roy, C. J. (2005). "Review of code and solution verification procedures for computational simulation." J. Comput. Phys., 205(1), 131–156. https://doi.org/10.1016/j.jcp.2004.10.036

B. 1D pressurized flow (EPANET, transients, water quality)

  • Bergant, A., and Simpson, A. R. (1999). "Pipeline column separation flow regimes." J. Hydraul. Eng., 125(8), 835–848. https://doi.org/10.1061/(ASCE)0733-9429(1999)125:8(835)
  • Bergant, A., Simpson, A. R., and Tijsseling, A. S. (2006). "Water hammer with column separation: A historical review." J. Fluids Struct., 22(2), 135–171. https://doi.org/10.1016/j.jfluidstructs.2005.08.008
  • Colebrook, C. F. (1939). "Turbulent flow in pipes, with particular reference to the transition region between the smooth and rough pipe laws." J. Inst. Civ. Eng., 11(4), 133–156. https://doi.org/10.1680/ijoti.1939.13150
  • Cross, H. (1936). Analysis of Flow in Networks of Conduits or Conductors. Bulletin No. 286, Engineering Experiment Station, University of Illinois, Urbana, IL.
  • Ghidaoui, M. S., Zhao, M., McInnis, D. A., and Axworthy, D. H. (2005). "A review of water hammer theory and practice." Appl. Mech. Rev., 58(1), 49–76. https://doi.org/10.1115/1.1828050
  • Joukowsky, N. (1900). "Über den hydraulischen Stoss in Wasserleitungsröhren." Mémoires de l'Académie Impériale des Sciences de St.-Pétersbourg, Series 8, 9(5). English translation: Simin, O. (1904). "Water hammer." Proc. Am. Water Works Assoc., 24, 341–424.
  • Moody, L. F. (1944). "Friction factors for pipe flow." Trans. ASME, 66(8), 671–684.
  • Ogata, A., and Banks, R. B. (1961). A Solution of the Differential Equation of Longitudinal Dispersion in Porous Media. Professional Paper 411-A, U.S. Geological Survey, Washington, DC.
  • Rossman, L. A., and Boulos, P. F. (1996). "Numerical methods for modeling water quality in distribution systems: A comparison." J. Water Resour. Plann. Manage., 122(2), 137–146. https://doi.org/10.1061/(ASCE)0733-9496(1996)122:2(137)
  • Rossman, L. A., Woo, H., Tryby, M., Shang, F., Janke, R., and Haxton, T. (2020). EPANET 2.2 User Manual. EPA/600/R-20/133, U.S. EPA, Cincinnati, OH.
  • Taylor, G. I. (1954). "The dispersion of matter in turbulent flow through a pipe." Proc. R. Soc. Lond. A, 223(1155), 446–468. https://doi.org/10.1098/rspa.1954.0130
  • Todini, E., and Pilati, S. (1988). "A gradient algorithm for the analysis of pipe networks." Computer Applications in Water Supply, Vol. 1, B. Coulbeck and C. H. Orr, eds., Research Studies Press, Letchworth, UK, 1–20.
  • Wylie, E. B., and Streeter, V. L. (1993). Fluid Transients in Systems. Prentice Hall, Englewood Cliffs, NJ.

C. 1D free-surface flow (Saint-Venant, SWMM, structures)

  • Bos, M. G., ed. (1989). Discharge Measurement Structures, 3rd ed. ILRI Publication 20, International Institute for Land Reclamation and Improvement, Wageningen, Netherlands.
  • Chanson, H. (2004). The Hydraulics of Open Channel Flow: An Introduction, 2nd ed. Elsevier Butterworth-Heinemann, Oxford, UK.
  • Chow, V. T. (1959). Open-Channel Hydraulics. McGraw-Hill, New York.
  • de Saint-Venant, A. J. C. B. (1871). "Théorie du mouvement non permanent des eaux, avec application aux crues des rivières et à l'introduction des marées dans leur lit." Comptes Rendus de l'Académie des Sciences, 73, 147–154 and 237–240.
  • Hager, W. H. (1992). Energy Dissipators and Hydraulic Jump. Kluwer Academic, Dordrecht, Netherlands.
  • Henderson, F. M. (1966). Open Channel Flow. Macmillan, New York.
  • MacDonald, I., Baines, M. J., Nichols, N. K., and Samuels, P. G. (1997). "Analytic benchmark solutions for open-channel flows." J. Hydraul. Eng., 123(11), 1041–1045. https://doi.org/10.1061/(ASCE)0733-9429(1997)123:11(1041)
  • Ritter, A. (1892). "Die Fortpflanzung der Wasserwellen." Zeitschrift des Vereines Deutscher Ingenieure, 36(33), 947–954.
  • Roesner, L. A., Aldrich, J. A., and Dickinson, R. E. (1988). Storm Water Management Model User's Manual, Version 4: EXTRAN Addendum. EPA/600/3-88/001b, U.S. EPA, Athens, GA.
  • Rossman, L. A. (2017). Storm Water Management Model Reference Manual, Volume II – Hydraulics. EPA/600/R-17/111, U.S. EPA, Cincinnati, OH.
  • Rossman, L. A., and Huber, W. C. (2016). Storm Water Management Model Reference Manual, Volume I – Hydrology (Revised). EPA/600/R-15/162A, U.S. EPA, Cincinnati, OH.
  • Rossman, L. A., and Simon, M. A. (2022). Storm Water Management Model User's Manual, Version 5.2. EPA/600/R-22/030, U.S. EPA, Cincinnati, OH.
  • Stoker, J. J. (1957). Water Waves: The Mathematical Theory with Applications. Interscience, New York.
  • U.S. EPA (2023). † SWMM 5.2.4 (Build 5.2.4, 15 July 2023): Engine update history. https://www.epa.gov/system/files/other-files/2023-08/epaswmm5_updates.txt
  • Villemonte, J. R. (1947). "Submerged weir discharge studies." Eng. News-Record, 139(26), 866–869.
  • Weber, L. J., Schumate, E. D., and Mawer, N. (2001). "Experiments on flow at a 90° open-channel junction." J. Hydraul. Eng., 127(5), 340–350. https://doi.org/10.1061/(ASCE)0733-9429(2001)127:5(340)

D. 2D shallow water: analytical verification

  • Alcrudo, F., and Garcia-Navarro, P. (1993). "A high-resolution Godunov-type scheme in finite volumes for the 2D shallow-water equations." Int. J. Numer. Methods Fluids, 16(6), 489–505. https://doi.org/10.1002/fld.1650160604
  • Carrier, G. F., and Greenspan, H. P. (1958). "Water waves of finite amplitude on a sloping beach." J. Fluid Mech., 4(1), 97–109. https://doi.org/10.1017/S0022112058000331
  • Delestre, O., Lucas, C., Ksinant, P.-A., Darboux, F., Laguerre, C., Vo, T. N. T., James, F., and Cordier, S. (2013). "SWASHES: A compilation of shallow water analytic solutions for hydraulic and environmental studies." Int. J. Numer. Methods Fluids, 72(3), 269–300. https://doi.org/10.1002/fld.3741
  • Ippen, A. T. (1951). "Mechanics of supercritical flow." Trans. ASCE, 116, 268–295.
  • MacDonald, I. (1996). Analysis and Computation of Steady Open Channel Flow. PhD thesis, University of Reading, UK.
  • Sampson, J., Easton, A., and Singh, M. (2006). "Moving boundary shallow water flow above parabolic bottom topography." ANZIAM J., 47 (EMAC2005), C373–C387.
  • Synolakis, C. E. (1987). "The runup of solitary waves." J. Fluid Mech., 185, 523–545. https://doi.org/10.1017/S002211208700329X
  • Thacker, W. C. (1981). "Some exact solutions to the nonlinear shallow-water wave equations." J. Fluid Mech., 107, 499–508. https://doi.org/10.1017/S0022112081001882

E. 2D laboratory and field validation

  • Goutal, N. (1999). "The Malpasset dam failure: An overview and test case definition." Proc. 4th CADAM Workshop, Zaragoza, Spain.
  • Hervouet, J.-M., and Petitjean, A. (1999). "Malpasset dam-break revisited with two-dimensional computations." J. Hydraul. Res., 37(6), 777–788. https://doi.org/10.1080/00221689909498511
  • Morris, M. W. (2000). CADAM: Concerted Action on Dambreak Modelling – Final Report. Report SR 571, HR Wallingford, UK.
  • Néelz, S., and Pender, G. (2009). † Desktop Review of 2D Hydraulic Modelling Packages. Science Report SC080035/SR, Environment Agency, Bristol, UK. https://assets.publishing.service.gov.uk/media/5a7ca8c0e5274a2f304ef487/scho0709bqse-e-e.pdf
  • Néelz, S., and Pender, G. (2010). † Benchmarking of 2D Hydraulic Modelling Packages. Report SC080035/SR2, Environment Agency, Bristol, UK. https://assets.publishing.service.gov.uk/media/5a75765c40f0b6360e4744dd/scho0510bsno-e-e.pdf
  • Néelz, S., and Pender, G. (2013). † Benchmarking the Latest Generation of 2D Hydraulic Modelling Packages. Report SC120002, Environment Agency, Bristol, UK. https://www.gov.uk/flood-and-coastal-erosion-risk-management-research-reports/2d-benchmarking-evaluating-the-latest-generation-of-the-hydraulic-models-for-fcrm
  • Soares-Frazão, S. (2007). "Experiments of dam-break wave over a triangular bottom sill." J. Hydraul. Res., 45(sup1), 19–26. https://doi.org/10.1080/00221686.2007.9521829
  • Soares-Frazão, S., and Testa, G. (1999). "The Toce River test case: Numerical results analysis." Proc. 3rd CADAM Workshop, Milan, Italy.
  • Soares-Frazão, S., and Zech, Y. (2008). "Dam-break flow through an idealised city." J. Hydraul. Res., 46(5), 648–658. https://doi.org/10.3826/jhr.2008.3164
  • Testa, G., Zuccalà, D., Alcrudo, F., Mulet, J., and Soares-Frazão, S. (2007). "Flash flood flow experiment in a simplified urban district." J. Hydraul. Res., 45(sup1), 37–44. https://doi.org/10.1080/00221686.2007.9521831
  • Valiani, A., Caleffi, V., and Zanni, A. (2002). "Case study: Malpasset dam-break simulation using a two-dimensional finite volume method." J. Hydraul. Eng., 128(5), 460–472. https://doi.org/10.1061/(ASCE)0733-9429(2002)128:5(460)

F. 1D/2D coupling

  • Chen, A. S., Djordjević, S., Leandro, J., and Savić, D. A. (2007). "The urban inundation model with bidirectional flow interaction between 2D overland surface and 1D sewer networks." Proc. NOVATECH 2007, Lyon, France.
  • Djordjević, S., Prodanović, D., Maksimović, Č., Ivetić, M., and Savić, D. (2005). "SIPSON – Simulation of interaction between pipe flow and surface overland flow in networks." Water Sci. Technol., 52(5), 275–283. https://doi.org/10.2166/wst.2005.0143
  • Leandro, J., Chen, A. S., Djordjević, S., and Savić, D. A. (2009). "Comparison of 1D/1D and 1D/2D coupled (sewer/surface) hydraulic models for urban flood simulation." J. Hydraul. Eng., 135(6), 495–504. https://doi.org/10.1061/(ASCE)HY.1943-7900.0000037
  • Rubinato, M., Martins, R., Kesserwani, G., Leandro, J., Djordjević, S., and Shucksmith, J. (2017). "Experimental calibration and validation of sewer/surface flow exchange equations in steady and unsteady flow conditions." J. Hydrol., 552, 421–432. https://doi.org/10.1016/j.jhydrol.2017.06.024

G. Scoring metrics

  • Bates, P. D., and De Roo, A. P. J. (2000). "A simple raster-based model for flood inundation simulation." J. Hydrol., 236(1–2), 54–77. https://doi.org/10.1016/S0022-1694(00)00278-X
  • Gupta, H. V., Kling, H., Yilmaz, K. K., and Martinez, G. F. (2009). "Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling." J. Hydrol., 377(1–2), 80–91. https://doi.org/10.1016/j.jhydrol.2009.08.003
  • Moriasi, D. N., Arnold, J. G., Van Liew, M. W., Bingner, R. L., Harmel, R. D., and Veith, T. L. (2007). "Model evaluation guidelines for systematic quantification of accuracy in watershed simulations." Trans. ASABE, 50(3), 885–900. https://doi.org/10.13031/2013.23153
  • Nash, J. E., and Sutcliffe, J. V. (1970). "River flow forecasting through conceptual models part I – A discussion of principles." J. Hydrol., 10(3), 282–290. https://doi.org/10.1016/0022-1694(70)90255-6

H. Datasets and model corpora (provided sources)

  • Dickinson, R. E. (2026). † 1729-SWMM5-Models-2030 [Data set, GitHub repository]. ~3,800 SWMM5 .inp models with per-model summaries; Unlicense; CITATION.cff provided. https://github.com/SWMMBobSWMM6/1729-SWMM5-Models-2030
  • HydroCouple (2026). † openswmm.engine.benchmarks [Software and data set, GitHub repository]. Engine-agnostic SWMM benchmark harness, 1,396 tagged cases; Unlicense; under construction. https://github.com/HydroCouple/openswmm.engine.benchmarks
  • Pedersen, A. N., Pedersen, J. W., Vigueras-Rodriguez, A., Brink-Kjær, A., Borup, M., and Mikkelsen, P. S. (2021a). † Dataset for Bellinge: An urban drainage case study [Data collection]. Technical University of Denmark. https://doi.org/10.11583/DTU.c.5029124
  • Pedersen, A. N., Pedersen, J. W., Vigueras-Rodriguez, A., Brink-Kjær, A., Borup, M., and Mikkelsen, P. S. (2021b). † "The Bellinge data set: Open data and models for community-wide urban drainage systems research." Earth Syst. Sci. Data, 13, 4779–4798. https://doi.org/10.5194/essd-13-4779-2021
  • SWMMEnablement (2026). † 1729-SWMM5-Models [Data set, GitHub repository]. 1,729 OWA SWMM5 test models, compiled during the February 2026 Fixathon; Unlicense. https://github.com/SWMMEnablement/1729-SWMM5-Models

I. Data and software standards

  • Druskat, S., Spaaks, J. H., Chue Hong, N., Haines, R., Baker, J., et al. (2021). Citation File Format (CFF), Version 1.2.0. https://doi.org/10.5281/zenodo.5171937
  • Jagers, B., et al. (2016). UGRID Conventions, Version 1.0. https://ugrid-conventions.github.io/ugrid-conventions/
  • Open Geospatial Consortium (2019). OGC GeoTIFF Standard, Version 1.1. OGC 19-008r4.
  • Wilkinson, M. D., et al. (2016). "The FAIR Guiding Principles for scientific data management and stewardship." Sci. Data, 3, 160018. https://doi.org/10.1038/sdata.2016.18
  • Wright, A., Andrews, H., Hutton, B., and Dennis, G. (2022). JSON Schema: A Media Type for Describing JSON Documents (Draft 2020-12). https://json-schema.org/draft/2020-12/json-schema-core
  • Apache Software Foundation. Apache Arrow and Apache Parquet format specifications. https://arrow.apache.org/docs/format/Columnar.html ; https://parquet.apache.org/docs/file-format/