Unlocking the Power of Wastewater Data

Effective use of data in the wastewater industry ultimately yields optimized operations and sound business decisions. But reliable data driven tools are impossible to develop in the absence of a strong data governance system, powerful data integration and calculation tools, and a dynamic data delivery system. Data governance frameworks dictate the rules and standards applied to both fundamental and calculated data, from generation to deletion, and they ensure that the calculations and tools built around data provide trusted guidance and accuracy. The pillars of an effective data utilization program are governance, transparency, and calculation management. These concepts are being applied at Metro Water Recovery (Metro) in an effort led by the Data Analytics Group in the Technology and Innovation Department.

Data governance includes rules for data organization, ownership, access, and management. When changes to data are controlled and logged, definitions and metadata are clear, and access is simple, data users trust that the data is correct and that products built on it offer valuable insights. Metro is improving its data governance program by building a system to allow users to request changes to fundamental (i.e., raw) flow and process data (Figure 1). Requests will be routed to the relevant data owners for approval and then forwarded to the appropriate technical group to institute the change, which is then is logged in a database to capture the details of the modification.

Figure 1: Date Correction Workflow at Metro

Metro is unique in that both fundamental and calculated data are historized. Calculated data is created using the calculation engine built into PI Asset Framework (Figure 2) or through external systems such as Alteryx (Figure 3) or Python. Historizing calculated data allows for easy access to large, calculated datasets, which are often requested by internal and external project teams and can be time consuming to develop. Centralized calculations reduce risks associated with manual methodologies and errors, and workloads are reduced because calculations are automated. At Metro, hundreds of calculated data points are logged every day without the need for intervention by engineers or data analysts, although periodic checks are needed to ensure that the calculations are working properly.

Figure 2: Calculation Example from PI Asset Framework

Figure 3: Calculation Example from Alteryx

Transparency is imperative in centralized calculation systems. Data users must be able to easily access calculation methodologies, including exactly which data and operations are employed and at what frequency, so that they develop trust in the calculations. If trust is lost, users may prefer manual calculations to ensure their own complete understanding, leading to unmanaged copies of calculations that may differ from user to user.

At Metro, much of the historized calculated data is included on various operations reports. Each report contains links to documents on SharePoint that contain metadata about the values in the report including calculation definitions (Figure 4), standard assumptions and conversion factors, and explanations of any codes or notes. While these documents are manually generated during report creation and can be onerous to maintain, the Data Analytics Group at Metro continues to evaluate solutions to automate these processes and make calculation definitions accessible to all users. In addition, there are many other calculations that are only displayed on dashboards or only available on-demand for which calculation methodologies are only noted in comments included in the calculation itself, but that information may not be accessible to all data users. The Data Analytics Group at Metro continues to evaluate solutions to allow all data users to access calculation definitions and other metadata.

Figure 4: Calculation Definitions Document

Calculation management is needed to keep track of all the calculations that exist, which processes are used to calculate them, and how calculations are related to each other and to fundamental data. If changes are made to fundamental data, there may be cascading effects on calculated data streams. A calculation management system will identify the calculations that depend on the changed data so they can be updated. This web of interdependency can be highly complex and difficult to manage.

Effective data governance, calculation transparency, and calculation management are the cornerstones of an advanced data utilization program. At Metro Water Recovery, these principles are being applied to empower data-driven decision-making and operational efficiency. By historizing calculated data, implementing robust change management workflows, and promoting calculation transparency, Metro ensures that data users can access accurate, reliable, and actionable insights. While progress has been made, the journey to a fully integrated and automated system continues, driven by the commitment of the Data Analytics Group to refine processes, foster trust in data, and unlock the full potential of wastewater analytics and data utilization.