A New Performance Metric for Optimizing Model Calibrations

The modeling community for years has struggled with well calibrated models that still fail to achieve their primary objective: accurate prediction of overflows. The consequences of inaccurately predicting overflow volumes costs sewerage agencies millions of dollars in excess expenditures. Money is spent on building larger than needed facilities or correcting controls that are not sufficient. The Metropolitan Sewer District of Greater Cincinnati (MSDGC) recognized this issue and has developed a new performance metric that directly addresses the issue. This paper proposes a new metric that accurately and consistently predicts overflow volumes of CSOs and peak flows for conveyance solutions, something not at all guaranteed by current industry accepted goodness-of-fit (GOF) metrics. The reason is simple. GOF metrics must recognize the importance of and properly weight the middle portion of the storm response shape, or the total flow above some critical flow rate (Qc) such as the capacity of an underflow pipe or 3.5 times the dry weather flow. Previous metrics (WaPUG 2002) emphasized the total storm event volume and peak flow rate as the primary criteria to which numerical accuracies are calculated. Hydrograph shape was only addressed qualitatively, by stating that modeled and calibrated events should "closely follow each other in shape and in magnitude". Therefore "as a general guide", peak storm flow rate and total storm volume were given target ranges. The emphasis on peak flow rate and event volume often superseded qualitative fit in any regulations, software or procedures (MSDGC Regulation 2409). In updates to these metrics (CIWEM 2017), the Nash Sutcliffe Efficiency Coefficient (NSEC) was added as an additional metric depicting GOF. However, since the NSEC score can be heavily influenced by the tail of a storm, resulting in a failure of this metric to improve the critical overflow period of the storm. MSDGC has identified a new approach for optimizing model calibrations that specifically focus on critical portions of an event when overflows occur. The new approach uses a sub-event analysis approach and statistical metrics to optimize the GOF between model results and observed data. MSDGC's new approach specifically focuses on periods of wet weather events that have the greatest impact on peak flows and overflow volumes. The model calibrations developed through this new approach reduce the overall risk for MSDGC as it uses the models to size and construct wet weather improvements.