Managing a Million Data Points: A How-To Guide

The Hampton Roads Sanitation District (HRSD), serving the Tidewater Area of Virginia, has been tasked to maintain strict data reliability of its flow, pressure, and rain gauge monitoring sites. Per the Environmental Protection Agency (EPA) Administrative Order (AO), HRSD is required to maintain 75% data reliability for each individual meter during each monthly monitoring period and 90% data reliability for all data for each type of monitoring sensor or meter during qualifying wet weather events. HRSD has deployed 122 pressure monitoring sites, 175 flow monitoring sites and 66 rain gauge sites. Approximately 87,000 data points must go through a daily QA/QC process to ensure data validity and reliability for the flow meters and pressure sensors alone. The automation of data quality alerts through SQL statements has created a streamlined, efficient method to handle thousands of data points on a daily basis to identify system changes, system reaction, and instrument fouling.