Managing a Million Data Points: A How-To Guide

HRSD, serving the Tidewater Area of Virginia, has been tasked to maintain strict data reliability of its flow, pressure, and rain gauge monitoring sites. Per the Environmental Protection Agency (EPA) Administrative Order (AO), HRSD was required to maintain 75% data reliability for each individual meter during each monthly monitoring period and 90% data reliability for each individual meter type during qualifying wet weather events during our Consent Decree monitoring period which occurred March 2010 – March 2011. To this day, HRSD continues to maintain these data reliability standards as part of our Management, Operations, and Maintenance (MOM) program. HRSD has deployed 173 pressure monitoring sites, 160 flow monitoring sites and 74 rain gauge sites. Approximately 247,000 data points must go through a daily Quality Control (QC) process to ensure data validity and reliability for the flow meters, pressure sensors, and rain gauges alone. The automation of data quality alerts through SQL statements has created a streamlined, efficient method to handle thousands of data points on a daily basis to identify system changes, system reaction, and instrument fouling.