Inland Empire Utilities Agency SCADA Master Plan · Inland Empire Utilities Agency Facilities SCADA...

InlandEmpireUtilitiesAgency

FacilitiesSCADAMasterPlanProjectNo.EN11029

6075KimballAvenue Chino,CA91710 June2012

Inland Empire Utilities Agency Facilities SCADA System Master Plan June 2012

TOC-1

1 Executive Summary ................................................................................................................21.1 Strategies and Goals ...................................................................................................... 2

1.2 Approach .......................................................................................................................... 3

1.3 Program Summary ............................................................................................................ 4

2 Current State ...........................................................................................................................72.1 Approach .......................................................................................................................... 7

2.2 Current Systems ............................................................................................................... 7

2.3 Current State Assessment ............................................................................................... 10

3 Recommended Delivery Approach ......................................................................................273.1 Full Implementation ....................................................................................................... 27

3.2 Pilot Project .................................................................................................................... 27

3.3 Recommended Project Phasing ...................................................................................... 28

3.4 Mitigating Operational and Staff Impact ....................................................................... 30

4 Desired State ..........................................................................................................................314.1 Mission and Vision ......................................................................................................... 31

4.2 Business and Technical Requirements ........................................................................... 32

4.3 Master Plan Development .............................................................................................. 33

4.4 Program Phasing and Strategies ..................................................................................... 36

5 Glossary ..................................................................................................................................79

Appendix A - Current State Technical Memorandum

Appendix B - SCADA Requirements Technical Memorandum

Appendix C - SCADA Alternatives Assessment Technical Memorandum

Appendix D Desired State Block Diagrams

Appendix E Overall SCADA System Rockwell/Application Software

Appendix F - Program Schedule

Appendix G - Cost Estimates


2

1 Executive Summary

1.1 Strategies and Goals The primary goal of the Inland Empire Utilities Agencys Facilities SCADA Master Plan is to define and document a road map for the implementation of the technology, practices, and organization required to meet Agencys long-term vision for SCADA. The Agencys long-term, vision for their SCADA system is to build a fully integrated and uniform SCADA system that provides enterprise-wide control and the information necessary to optimize operations. The Agencys top-line strategies for achieving their SCADA vision are summarized below:

Enterprise Integration: Implement a fully integrated and uniform SCADA system to provide the Agency with the operational flexibility required to control any-facility from any-location and enable enterprise-wide, coordinated strategies for wastewater treatment, recycled water supply, and energy management.

Information Driven: Provide all stakeholders with direct access to quality operations and business data required to support regulatory compliance, operations decision support, process optimization, planning, asset management, and energy management.

Operations Improvement: Continually improve operations by: documenting operating procedures and workflow; establish key performance metrics and monitor performance; investigate, plan, and implement the highest-value opportunities for optimizing revenue, operating cost, and environmental impact.

SCADA Asset Management: Development of the organization, practices, and procedures required to effectively plan, govern, and manage the Agencys SCADA systems to achieve high stakeholder satisfaction and continue to meet the Agencys business requirements

The Agencys SCADA Master Plan has been programmed with intermediate goals that progressively move the Agency through increasing levels of SCADA maturity. Applying the strategies described above the SCADA Master Plan program is designed to achieve the Agencys SCADA vision within 7 years.

Goal 1 Reliable Operations: Build the SCADA infrastructure and SCADA asset management practices to support Operations in reliably meeting the Agencys primary mission.

Goal 2 Enhanced Operations: Provide the information technology, tools, and practices necessary to monitor and enhance operational performance.

Goal 3 Optimized Operations: Optimize operational performance by leveraging the Agencys process expertise and enterprise-wide, SCADA infrastructure to enhance recycled water revenues, reduce operating costs, and maximize the production of green power.

The goals and related phases are structured to logically sequence the SCADA upgrade to address the highest priority issues first and to comply with the Agencys budget and resource constraints. In addition, the master plan describes the scope, budget, resources and scheduling of the individual projects that comprise each of the phases.


3

Optimized Operations (2016 - 2021) Build ODMS Layer 2 Automated Data Validation

Enhanced Operations Equipment Condition Based Monitoring (2014 - 2018) Automated Workflow Management

Optimization Implementation Direct Control Level Upgrade Electronic O&M Manuals SCADA Governance Standard Div 17 Specifications Instrument Spec. and Calib. Proc. SCADA Lifecycle Management

Reliable Operations SCADA Standards Committee and (2012 - 2017) Comprehensive Standards

Knowledge Management Plan Security VA and Remediation Plan Electronic Document Management SCADA Network Upgrade Inventory, Collate, and Load SCADA Design, Procurement, and Documentation Build Software Lab Optimization Feasibility Studies Supervisory Control Level Upgrade Instrument Study and Upgrade HMI & PLC Procurement Agreements Operations and Maintenance KPIs Standard SCADA Project Delivery Methodologies SCADA Change Management Alarm-Event Standard Standard HMI and PLC Templates SCADA DR/BC Plan SCADA Organization and Support Plan Build ODMS Layer 1 ODMS Management and Data Validation Update Critical Operations Documentation

1.2 Approach Westin mapped its standard master planning methodology to the Agencys project delivery requirements (50% Submittal, 70% Submittal, 100% Submittal and Final Submittal). The 50% Submittal summarized the findings and recommendations of the Needs Assessment and provided a preliminary draft of the master plan document. The 70% Submittal represents the draft Master Plan and the 100% submission will constitute a further refinement of the plan.

The steps followed to develop the Agencys master plan were:

i. Review expectations and approach to establish the Project Charter.

ii. Document the Current State by reviewing existing documentation, site visits and staff interviews.

iii. Capture the Agencys Business Requirements and Vision for SCADA through facilitated stakeholder workshops.

iv. Research and analyze technology alternatives and best practices required to address the business requirements.

v. Review preliminary recommendations to reach consensus on projects and priorities required to achieve the desired state.

vi. Develop project scope statements, budget and implementation plan to bridge the gap between the current state and the desired state.


4

vii. Refine the plan for final acceptance and endorsement by key stakeholders.

1.3 Program Summary The Master Plan defines a series of inter-related projects to achieve the Agencys goals and vision for their SCADA system. These projects have been categorized into the following four (4) strategies.

SCADA Asset Management (0 4 years)

The projects included in this strategy are designed to develop the organization, practices, and procedures required to effectively plan and manage the Agencys SCADA system.

Develop a Procurement Policy for Programmable Logic Controller (PLC) and Human-Machine Interface (HMI) components. This policy will establish long-term, standardized pricing from the PLC and HMI vendors to assure competitive pricing for equipment purchase and support over the life of the SCADA Systems.

Establish a Change Management System for the SCADA System to ensure that all future additions and enhancements are completed in a coordinated, well-planned and accessible manner.

Define standard project delivery methodologies for SCADA System projects to ensure consistency of approach and to simplify project oversight.

Establish a SCADA Standards Committee and define the process for the continuous upgrade, expansion, and maintenance of the SCADA System Standards.

Develop a standard set of SCADA System specifications (CSI Division 17 or 40 9X XX) that can be provided to design consultants for future SCADA System projects.

Develop an Organization and Support Plan that provides for the most effective approach for continued development and support of the Facilities SCADA Systems using both internal and external resources.

Create a SCADA Planning Committee to review and approve all SCADA System additions, enhancements, and improvements.

Incorporate critical SCADA assets into the Agencys CMMS so that predictive and preventative maintenance practices can be applied to the SCADA System components.

Complete a SCADA Disaster Recovery and Business Continuity Plan to ensure that the Agency is well prepared for continuous operations in the event of a significant failure of the SCADA System

Develop standard set of instrumentation specifications and develop/update instrument calibration procedures to ensure that all critical process instrumentation is properly calibrated

Perform an independent SCADA vulnerability assessment to determine the effectiveness of the Agencys security measures

Enterprise SCADA (0 5 years)

The projects included in this strategy are designed to move the Agencys SCADA system from the current multi-vendor platform with islands of automation to a uniform SCADA system that is fully integrated across the enterprise.

Implement network modifications to comply with industry best practices and recommended security guidelines


5

Architect and design the new SCADA Systems to allow full control of any facility from any location within the system, based on operator rights (Enterprise SCADA)

Migrate Foxboro-based SCADA System at each plant to new single-vendor HMI and Allen-Bradley (A-B) PLCs using the following phased approach:

o Establish Top-End HMI Implement new HMI at each plant

Consolidate older Foxboro controllers to fewer and newer model (currently being completed by IEUA)

Interface new Foxboro controllers to new HMI

o Direct Connect Existing A-B PLCs Remove connection to FDSI

Directly connect PLCs to new HMI

o Replace Foxboro Remote I/O Convert Foxboro Remote I/O to A-B PLCs

Connect A-B PLCs to new HMI

o Replace Foxboro Control Processors Replace remaining Foxboro controllers with A-B PLCs

Connect A-B PLCs to new HMI

Information Driven (2 6 years)

The projects included in this strategy are designed to facilitate the Agency becoming an information-driven organization with unrestricted access to the quality data and documentation required to optimize operations, capital planning, and asset management.

Improve and expand electronic Operations and Maintenance manuals and context-sensitive help that are integrated with SCADA

Develop a Knowledge Management Plan to retain and share the Agencys institutional SCADA System knowledge

Improve and expand existing Electronic Document Management system (eDMS) for the management of engineering, maintenance, and operations documentation

Load the Agencys engineering, operations and maintenance documentation into the eDMS

Update critical Operations Documentation to accurately reflect the As-Is condition

Implement an Operations Data Management System (ODMS) that provides a consolidated historian and user-friendly reporting tools

Integrate the ODMS with business systems (financial and maintenance management) to support effective business decision making

Implement an Automated Data Validation and Resolution process

Determine Instrumentation Upgrades required to more effectively control, monitor, and optimize the Agencys processes


6

Operations Improvement (4 - 9 years)

The projects included in this phase of the program are designed optimize operations to enhance recycled water revenues, reduce operating costs, and maximize the production of green power.

Establish/refine/standardize performance metrics (Key Performance Indicators) for Operations and Maintenance. Subsequent SCADA System enhancements and improvements will be designed to measure and report these KPIs to operations and maintenance staff and Agency management in an intuitive fashion.

Monitor Key Performance Indicators (KPIs) for critical assets in the Agencys ODMS and push up to the Agencys CMMS

Conduct Optimization Feasibility Studies for the following areas of Utility operations:

o Operating Costs (non-labor) o Energy Coordination o Digester Gas Production o Supply-Demand Coordination

Model and automate the tracking of workflow for critical SCADA asset management procedures and related Operations and Maintenance procedures

Execute the optimization strategies for operating cost, energy, digester gas production, and supply-demand coordination based on the previous feasibility studies


7

2 Current State

2.1 Approach The Agencys existing Supervisory Control & Data Acquisition (SCADA) System network is comprised of a wide range of equipment that is located at various remote sites (mainly lift stations) and facilities throughout the Agencys service area. The Facilities SCADA Master Plan is intended to guide the future direction of the IEUA Facilities SCADA systems.

The first step of the project was to develop a description of the current state of the existing Facilities SCADA Systems. A combination of existing documentation review, site surveys, and stakeholder interviews were utilized to capture and validate the Current State of the Agencys Facilities SCADA Systems. The stakeholder interviews were structured to cover a full-scope of SCADA topics (Functional, Operations, Technical, Asset Management and Optimization). The results of the Current State assessment effort are captured in several documents incorporated in Appendix A Current State Technical Memorandum, Current State Technical Memorandum Appendix B - Current State SCADA System Block Diagrams, and Current State Technical Memorandum Appendix C Current State SCADA System Equipment Inventories.

2.2 Current Systems The IEUA Facilities SCADA Systems currently consist of seven (7) separate systems divided into Southern and Northern service areas. The Southern service area includes Regional Water Recycling Plant No. 5 (RP-5), Regional Water Recycling Plant No. 2 (RP-2), and Carbon Canyon SCADA systems with main SCADA servers located at RP-5. The Northern service area includes Regional Water Recycling Plant No. 1 (RP-1), Regional Water Recycling Plant No. 4 (RP-4), and Ground Water Recharge SCADA Systems with main SCADA Servers located at RP-1. Figure 1 presents a conceptual block diagram of these systems. Standalone Foxboro Distributed Control Systems (DCS) are installed at four (4) of the treatment facilities. Regional Plant 4, the Recycled Water and Groundwater Recharge Systems, and the Composting Facility utilize Rockwell Automation HMIs and PLCs. The following paragraphs provide an overview of each of these SCADA Systems.

2.2.1 Regional Water Recycling Plant No. 1 Regional Water Recycling Plant No. 1 (RP-1) is located in the city of Ontario. The plant SCADA system consists of four (4) separate systems: Tertiary, Liquid, Dewatering, and Ground Water Recharge (GWR). The Tertiary, Flow, and Solids SCADA systems each consist of a Foxboro DCS system as the main control system with remote I/O panels and Rockwell Automation PLCs to control plant sub-systems. The GWR SCADA system consists of Rockwell Automation HMI with PLC panels at remote GWR sites (see GWR section).

Refer to Current State Technical Memorandum Appendix B for a block diagram of the existing RP-1 SCADA System and Current State Technical Memorandum Appendix C for an equipment inventory.

2.2.2 Regional Water Recycling Plant No. 4 Regional Water Recycling Plant No. 4 (RP-4) is located in the city of Rancho Cucamonga. Rockwell Automations SCADA system is used as the main plant control system with Rockwell Automations remote I/O panels and PLCs to control sub-systems. This is the only IEUA facility that uses Rockwell Automation hardware and software as the primary plant control system.


Inland Empire Utilities Agency Facilities SCADA System Master Plan May 2012

8

Figure 1 SCADA Block Diagram


9

2.2.3 Regional Water Recycling Plant No. 5 Regional Water Recycling Plant No. 5 (RP-5) is located in the city of Chino adjacent to the Agencys Headquarters buildings. A Foxboro DCS system is used as the main plant control system at this plant with remote I/O panels and Rockwell Automations PLCs to control sub-systems. Each of three (3) remote I/O panels are located in Power Center 1, 2, and 3 buildings along with a local operator interface terminal (OIT) which functions as one of the plant control workstations.


2.2.4 Regional Water Recycling Plant No. 2 Regional Water Recycling Plant No. 2 (RP-2) is located in the city of Chino approximately 1.0 mile southeast of RP-5. A Foxboro DCS system is used as the main plant control system at this plant with remote I/O panels and Rockwell Automations PLCs to control sub-systems.


2.2.5 Carbon Canyon Water Recycling Facility The Carbon Canyon Water Recycling Facility (CCWRF) is located in the city of Chino approximately 1.5 miles northwest of RP-5. A Foxboro DCS system is used as the main plant control system at this plant with remote I/O panels and Rockwell Automations PLCs to control sub-systems.

Refer to Current State Technical Memorandum Appendix B for a block diagram of the existing Carbon Canyon SCADA System and Current State Technical Memorandum Appendix C for an equipment inventory.


10

2.3 Current State Assessment Strengths and weaknesses of the current SCADA system were analyzed based on the major functional components of the system, SCADA practices and procedures, and opportunities for optimization. The strengths and weaknesses of the current system are summarized in this section and benchmarked against industry best practices. A bulleted list of recommendations required to close the gap between the current state and best practices has been provided for each topic analyzed. For a more detailed reference, refer to Appendix A Current State Technical Memorandum.

Business Drivers

Industry Objectives Current State Regulatory Compliance: Wastewater utilities are required by law to meet certain discharge regulations. Violations of these permit requirements can result in costly fines. Therefore, it is imperative that SCADA System designs fully support the control processes required for effective wastewater treatment. Reduced Operating and Capital Costs: Public utilities are continually challenged to reduce operating and capital costs; to do more with less. The weakened economy has reduced revenue causing utilities to look to SCADA and automation for ways to lower power and chemical costs and to maintain or increase service levels without increasing staff. In addition, the dramatic slowdown in new construction has limited utilities sources of new capital and resulted in their need to more efficiently use their existing facilities. Risk Management: Public utilities are operated in the public eye and must place a high premium on the reduction of risk. All system operations must be designed to maximize the safety of IEUA staff and the public. In addition, systems must be designed to prevent wastewater discharges to the greatest extent possible.

Regulatory Compliance Strength: IEUA has a proven track record of consistently meeting discharge permit requirements. The existing instrumentation and SCADA Systems help to meet this objective. Weaknesses: Compliance reports currently require combining information from multiple systems and involve some labor intensive manual processes. Current systems do not support ad hoc reports. Reduced Operating and Capital Costs Strength: IEUA staff continually seeks better, lower costs methods for plant operation. Weakness: IEUA has not performed a formal evaluation process to determine what areas of operation can provide the most payback for additional optimization. The evaluation should lead to a plan for implementing the identified optimization techniques. Risk Management Strength: IEUA has an excellent safety record and a low incident rate of overflows. Weakness: None

Initial Recommendations Conduct an optimization study to determine the feasibility of optimization strategies and

which strategies will provide the best return on investment Implement a consolidated historian that captures and manages all data in a single location

to support automated compliance reporting. Define and measure performance metrics to understand how well the Agency is meeting

their goals and delivering on their mission.


11

Organization, Practices, and Procedures Industry Best Practices Current State Organization: Providing SCADA staff with the tools and skills to support the technologies deployed by the Utility is an ongoing effort. SCADA technology changes rapidly with major versions of software being released by vendors every couple of years and incremental versions being released several times each year. Keeping pace with this change requires a support plan that identifies the skills required to maintain the system, staff skills and skill gaps, and training or 3rd party support required to close the gaps. Change Management steering committee and process for prioritizing and resourcing SCADA enhancements and upgrades. Includes procedures for reviewing, implementing, and deploying system change. Effective change management helps to ensure that only those system changes that provide measurable value are undertaken and that ongoing changes do not jeopardize the ability to support those applications. Software Revision Control: SCADA software revision control is a key strategy for preventing process upset introduced when an incorrect version of code is downloaded to a controller. Preventive Maintenance: The SCADA system has few moving parts and is often overlooked for preventive maintenance (PM). A SCADA PM plan and procedures can prevent catastrophic failure of components such as hard-drives and power supplies. Dust accumulation is often the culprit in the over-heating or short-circuiting of components. SCADA Asset Management - SCADA is a critical asset that is comprised of components that have considerable lifecycle variability. A SCADA asset plan is necessary to manage this critical asset. The plan should incorporate SCADA assets into the Utilitys CMMS with sufficient granularity to manage the different component lifecycles; inventory, tag and assess the condition of SCADA assets; and include SCADA Asset renewal in Utilitys risk-based capital improvement plan.

Organization Strengths: The consensus of Operations staff is that the DCS group responds in a timely manner. IEUA has an extensive set of Standard Operation Procedures (SOPs) which provide detailed, step-by-step directions on how to perform typical operator actions associated with the SCADA Systems. Weaknesses: SCADA System planning, development, and maintenance are currently performed primarily by the DCS/SCADA Maintenance Group (DCS Group) with input from the Engineering and Operations Groups. The intended primary focus of the DCS Group is to provide ongoing support and maintenance of the deployed SCADA systems. However, they are frequently required to provide SCADA support for capital delivery projects, which is frequently pre-empted by their need to respond to SCADA maintenance issues.

Change Management/Software Revision Control Strength: There is a master directory for control system change management. In addition, to prevent old programs from being downloaded to PLCs the Agency has deployed Rockwell AssetCenter for revision control. SCADA Asset Management

Strengths: The Facilities SCADA Master Plan project is the first step toward developing a formal framework for the future SCADA System planning and development. The Master Plan will help IEUA anticipate and budget for SCADA System costs and to ensure the long-term viability of their SCADA Systems.

To support capital delivery and realize the O&M benefits of uniformity of design, the Westin is assisting the Agency in developing HMI and PLC standards as part of the current Master Planning project.

Weaknesses: There is currently not a stakeholder committee or process for planning


12

Organization, Practices, and Procedures Industry Best Practices Current State

future upgrades or on-going enhancements of the existing SCADA Systems.

Construction Management (CM) often has to interpret control systems needs for capital projects and without published standards this is very challenging for CM staff. The lack of standards directly impacts the level of effort for DCS staff since they must communicate the Agencys requirements to every design and implementation team.

Initial Recommendations Establish a SCADA Steering Committee to guide SCADA and prioritize SCADA

enhancements and capital delivery. Complete a SCADA system support plan to ensure that adequate resources are in place to

support the continuity of operations, system enhancements, and capital delivery. Inventory and tag all critical SCADA Assets and log these assets into the Agencys

CMMS. Establish preventive maintenance procedures and schedules for SCADA equipment. Model workflow associated with major SCADA work tasks. Develop comprehensive capital delivery and change management procedures.

Documentation

Industry Best Practices Current State Quality SCADA system documentation starts with the design process. A documented SCADA Project Delivery Methodology that details project phases, deliverables and deliverable content is the best method for ensuring that the Utility receives the design and as-built documentation it requires. It is important to clearly define as-built requirements by defining a Project Delivery Standard (for consultants) and documentation specification (for contractors). It is equally important to ensure that contracts and budgets are sufficient to complete as-built documentation. A common practice requires the Contractor to submit red-lined working drawings to the Consultant for field validation,

Strength: IEUA stores electronic copies of SCADA System design and as-built documentation on a centralized server. However, these documents are not available to everyone in the organization who needs them. Weakness: The current state of record documentation and document management is considered to be inadequate by all staff members. The lack of organization of electronic documents makes documentation difficult to locate. Westin observed the generally inadequate state of SCADA System documentation during execution of the Facilities SCADA Master Plan project. Weakness: P&IDs and Process Control Narratives which describe the functional


13

Documentation

Industry Best Practices Current State documentation with AutoCAD, and application of the design engineers professional seal. Utilities accumulate substantial as-built and record documentation as a result of ongoing capital project delivery. Document management policies and procedure are required to organize this quantity of documentation and enable quick retrieval of documents. The document management process can be further enhanced with the implementation of an electronic document management system. It is very common for enhancements to be implemented by the Utility post-construction. Source documentation should be updated to eliminate differences between the as-built condition and documentation. A formal Document Maintenance project should occur at a recurring interval, to ensure the accuracy of critical documentation.

behavior of the SCADA system are not available for many of the facilities. Weakness: The Agency does not have SCADA documentation standards for capital delivery. As a result, each project delivered has different documentation quality and content. Without defined requirements for SCADA documentation, system requirements are missed during the design phase (where they can be dealt with minimum cost) and often must be corrected during the construction or post-construction phases at a much greater cost. Weakness: There are currently no formal standards which define the content and format of SCADA documentation, there is a wide discrepancy in the completeness and quality of the documentation. Weakness: There is no structured change process to ensure that as-built documentation is kept current with post-commissioning field changes and Agency implemented system enhancements.

Initial Recommendations

Define SCADA documentation requirements as part of a more comprehensive SCADA Standards effort including documentation templates or examples.

Publish policies and procedures for document management. Include a recurring document maintenance project in the Agencys Capital Improvement

Plan.

Optimization

Industry Objectives Current State Continuous performance improvement is a philosophy of every utility. SCADA, and other information technologies, enable utility optimization by providing the data required for effective decision making and the ability to provide an automated execution of the decision. There are no established best

Energy Management: Strength: The Agency has installed interval metering on many of the electric loads and has integrated some of these meters with the SCADA system. IEUA is establishing a standard approach for


14

Optimization

Industry Objectives Current State practices for optimization strategies. Each opportunity for optimization must be evaluated on its value to the Utility. In general, the technologies employed in optimization strategies are proven. The risk in employing these technologies is typically the lack of application experience required to accurately predict the outcome of implementation. The recommended approach to mitigating that risk is to conduct a feasibility study to determine if the business case supports moving ahead with implementation. Energy optimization opportunities:

billing validation, equipment energy efficiency, optimal energy supply to minimize carbon footprint or reduce cost, energy off-peak control strategies.

Process optimization opportunities: strategies for limiting peaking factors, utilization of existing capacity to reduce process variability, additional sensing and predictive technologies.

Asset optimization: the determination of asset condition based on real-time monitoring and historical trending, the development of predictive maintenance KPIs (key performance indicators) to implement condition based PM.

electric metering. Weakness: Not all meters are integrated with SCADA. The Agency does not have an energy optimization plan. Process Optimization Weakness: The Agency has not completed an optimization assessment. Operations feels that there is inadequate instrumentation and automation to operate optimally. Asset Optimization Strength: The Agency is deploying SAP CMMS for maintenance management and planning. Weakness: The CMMS is not fully deployed and Asset Management practices are not well defined. Current system is inadequate and is labor intensive to use.

Initial Recommendations Complete the installation and SCADA integration of power meters on all major electrical

loads. Develop energy reports to summarize energy usage and pump efficiency. Investigate the feasibility of time-of-day pumping strategies to pump more in off-peak

hours. Automate energy billing validation. Complete feasibility studies to determine the value of optimization for energy, supply and

demand coordination, and treatment. Create Management dashboards to present summarized KPIs.


15

Standards

Industry Objectives Current State SCADA management is significantly enhanced if variability in system design and implementation is limited by the development of comprehensive SCADA standards. SCADA Standards are most effective when they include: design and implementation methodology, guidelines and standards for the compliant system; and supporting design templates and software modules. A comprehensive set of SCADA Standards will limit variability in the application of hardware and software technology without limiting the ability of the Consultant to engineer the system.

Strength: The Agency has developed a number of standard specification sections which define the functional requirements for SCADA System hardware and software components. These sections can be modified as necessary for use in specific bid packages. Strength: HMI Graphic Standards and PLC Programming Standards are being developed as part of the Facilities SCADA Master Plan project. In addition, the Master Plan will describe additional, formal standards which should be developed, as well as a process for keeping the standards current and up-to-date. Weakness: There are essentially no formal SCADA design and implementation standards for contractors, consultants, and integrators to follow. HMI and PLC programming methodology used by different contractors vary project to project. This is apparent in the current SCADA system throughout the Agency. For instance, the HMI screen layouts and color schemes which are configured for different platforms such as Foxboro DCS and Rockwell Automation are not consistent. This type of inconsistency in HMI and PLC configuration will continue if standards are not available.

Weakness: The lack of programming standards has resulted in a more significant maintenance effort for the DCS group due to the variability of code structure, behavior and documentation. The lack of standards often results in post-construction modifications to the system. These modifications negatively impact staff effort and operating budgets.

Initial Recommendations Establish a suite of comprehensive SCADA standards and a Standards Committee to

maintain the standards. Develop, publish and maintain software development standards and create a library of

standard HMI and PLC modules. Develop standard capital delivery methodologies for SCADA system design and software

development. Develop, publish and maintain documentation standards including documentation

templates.


16

HMI Hardware and Software Functionality

Industry Objectives Current State There are numerous Human Machine Interface (HMI) products from different vendors that provide very similar functionality. In addition to component price and functionality, it is important to select control system products based on the total ownership cost (TOC). TOC issues include: Utility staff training and experience; vendors installed base both in the water industry and in the Utilitys geographic area; vendors current and historical commitment to automation technologies; vendors upgrade practices and policies; and available vendor and local integrator support. Since every product family requires training, tools and support contracts, it is important to minimize technology proliferation by establishing and enforcing standards for hardware and software components. SCADA software standards are necessary to ensure uniformity in system behavior and look and feel. Without standards, the operational ease of use, level of system support, deployment, and operational risk are all negatively affected. Redundancy is a strategy employed for points of consolidation or for critical equipment to achieve desired service levels for system availability. For example, the HMI is the primary tool for Operator interaction with the process. As a point of system consolidation, failure of an HMI server can have a very negative effect on the continuity of operations. For this reason HMI server redundancy is considered a best practice. Secure remote access to provide O&M staff with ability to control and monitor the system from any Utility location or from any external location with communication access.

Strength: IEUA has standardized on two automation vendors (Rockwell Automation and Invensys Foxboro). Both vendors are well established and committed automation system vendors. Strength: DCS staff members are experienced and competent in all the installed technologies. The local Consultant and System Integrator community is experienced with the application of Rockwell Automation products. Strength: The Agency maintains high system availability by implementing redundancy for all critical SCADA components. This is a necessity since the Agency operates unattended for up to 16 hours per day. Strength: The Agency has implemented remote access to provide access to offsite O&M staff. Weakness: Support for Foxboros HMI configuration is only available from contracted Foxboro staff or trained IEUA staff. Outsourcing to local consultants and integrators in the area is difficult due to their limited experience with Foxboro systems. Weakness: The Agency has not performed a 3rd party vulnerability assessment to verify the security of their remote access. Weakness: The software product and approach to HMI graphics is different at each facility. For example, at some plants a green equipment symbol means ON and at others green is OFF. Screen navigation and layout is not intuitive and in many situations does not accurately represent actual equipment layout. The lack of standardization reduces operating staff efficiency, complicates cross-training efforts, and represents a potential operating risk. Weakness: The current system requires operators to use several different log-ins to access the entire system. This directly affects operational efficiency since staff must log-in several times to control and monitor the


17

HMI Hardware and Software Functionality

Industry Objectives Current State system. Weakness: The Agencys lack of standardization on a single HMI increases the effort to maintain the SCADA system and increases the variability in graphics. For example: FactoryTalk View, RSView32, PanelView Plus, and PanelView 550 each require separate software to configure.

Initial Recommendations Conduct a formal process for evaluating, selecting, and procuring a standard HMI

platform across all facilities. Negotiate a long-term agreement with an HMI vendor that provides the Agency with

stable pricing Perform a SCADA cyber vulnerability assessment to verify its level of security. Create HMI templates for the selected HMI to ensure uniform application of the

Standards. Implement a phased plan to upgrade all HMI components to a uniform standard.

PLC/RTU Hardware and Software Functionality

Industry Objectives Current State The use of industry standard programming languages (IEC-1131) and newer object oriented programming enables the creation of re-useable software graphics and control modules that reduce risk and lower the cost of system enhancement and expansion. Standardizing on a single vendor and product family has numerous advantages for a utility:

Re-useable design templates can be developed for specifications and drawings to reduce the cost and risk associated with system enhancement and expansion.

The quantity of spare parts that must be in-stock is reduced.

Support staff knowledge of the standard components is enhanced improving system recovery, maintenance, and the

Strength: The Agency has trained and skilled RSLogix programmers on staff that can make modifications to PLCs for maintenance and also develop programs on new systems. The Agency also uses local consultants and integrators since many of them have RSLogix programming knowledge and experience.

Strength: The Agency is in the process of transitioning to the newer and more user-friendly Foxboro I/A Series Configuration Component (IACC) software which provides object and template oriented programming environment.

Weakness: The Agency does not have long-term procurement agreements with their vendors and are susceptible to escalating support and component costs.


18

PLC/RTU Hardware and Software Functionality

Industry Objectives Current State efficiency of project delivery.

Utilities understand the necessity and benefits for standardization of control components but many have experience the challenges associated with locking into a single vendor and losing the cost benefits of competitive procurement. To address this issue many utilities are standardizing on control system vendors by through a competitive procurement process that establishes long-term (5 10 years), stable pricing for components, support, and training.

Weakness: The Agency relies on contracted Foxboro staff to make most ICC modifications due to high learning curve, limited trained IEUA staff, and limited or no expertise available from consultants and integrators in the area.

Weakness: Rockwell Automations controllers and I/O platform used at the treatment plants and remote sites are a mix three different product families: Logix family (ControlLogix and CompactLogix), SLC family (SLC504 and SLC505) and the Micro family (logix1000 and1100).

Initial Recommendations Conduct a formal process for evaluating, selecting, and procuring a standard

DCS/PLC/RTU platform. Negotiate a long-term agreement with the selected vendor that provides the Agency with

stable pricing Create a standard program structure and re-useable software modules. Implement a phased plan to upgrade all controllers to a uniform standard.

SCADA Network Communications Industry Best Practices Current State Low-cost, high-speed switching has resulted in Ethernet/IP becoming the industry de facto network standard for Business, SCADA, Smart Devices, and Instrumentation. Modern Ethernet/IP networks enable convergence of multiple networks on the same infrastructure without interference. Compliance with industry standards (IEEE-802.3x and TIA/EIA-568) and avoidance of vendor proprietary features. The SCADA network is the backbone of the SCADA system and must have a very high availability (99.995% to 99.999%). Network availability is enhanced with accurate and complete documentation and effective network management.

Strength: The Agency has standardized on Ethernet/IP for its intra-plant control networks (Plant Information Network) and has installed fiber optic cable for the backbone. The fiber backbone utilizes redundant fibers. Strength: The Agency has recently upgraded its inter-plant, wide area network to a private, unlicensed 5.8 GHz wireless network. In addition, the Agency has projects underway which will create a redundant wireless loop and upgrade to a higher data rate, licensed 18 GHz wireless network. Strength: The Agency has standardized to some degree on network components. In an effort to move towards standardizing communications to Ethernet, existing SLC504


19

SCADA Network Communications Industry Best Practices Current State For security and performance reasons all electronic network components should be intelligent, managed devices. To ensure uniformity of design, network standards should exist for the physical and logical network. Network topology should be either spoke-and-hub or a redundant ring. All points of significant consolidation, such as the network core, should be redundant. Servers and network core components should be housed in an environmentally controlled and physically secure location. Structured cabling should consist of multi-fiber optic cabling (single mode or hybrid) for the backbone and high-bandwidth copper cabling for terminations (CAT6 or CAT5e). Copper cable should be run within metal conduit or be shielded (ScTP).

controllers with legacy Data Highway Plus (DH+) communications are being replaced with Ethernet based SLC505 controllers. Strength: Servers and the network core are installed in a secure, environmentally controlled, computer room. Strength: The Agency has deployed Solarwinds for network management and monitoring. Weakness: There is no documented network topology. The network may be spoke-and-hub or a loop or a mix of both. Weakness: Network documentation is sparse or non-existent. Weakness: Legacy proprietary networks (AB ControlNet and Foxboro Nodebus) still exist within the control system. Weakness: The Agency does not have a standard fieldbus network for instrumentation and smart devices networking.

Initial Recommendations Establish a fieldbus standard/s for smart devices and instrumentation. Develop network physical and logical design standards including standard drawings for

documenting the physical network. Document existing network infrastructure. Create a long-term plan for compliance with industry best practices for network design. Transition legacy/proprietary networks to new standards.

Control Philosophy and Operations

Industry Best Practices Current State Complete instrumentation and control of key process equipment is necessary to realize the benefits of full automation and provide a platform for business optimization. SCADA systems are architected to achieve the availability and performance objectives of a mission critical system. In designing the system architecture, points of consolidation are typically candidates for redundancy as are

Strength: IEUA has implemented automatic, closed loop control for all major plant process areas at each of the treatment facilities. Control algorithms execute in Foxboro controllers or Rockwell Automation (Allen Bradley) PLCs, depending on the location. In general, the Operations staff considers these control strategies to be effective, and the expected mode of operation is for all control


20

Control Philosophy and Operations

Industry Best Practices Current State system components that represent significant risk to the agency in the event of failure. In addition, SCADA Systems are architected using a modular, open architecture that enables system expansion or component replacement with minimal impact to continuity of operations. A fully automated process combined with accurate and complete historical operations data are the foundational tools that enable Operations to optimize the process utilizing predictive and adaptive control strategies. Whether online or in an advisory mode, advanced control applications are proving to be beneficial in process and business optimization.

loops to be operated in automatic mode. Operators can make setpoint adjustments to modify system operation, and occasionally operate individual pieces of equipment in remote manual mode.

Strength: The Agency operates unattended (dark) for up to sixteen hours per day. To ensure the SCADA system achieves its system availability objective the Agency has designed the system with redundant controllers. Weakness: There is inadequate instrumentation to support process control decisions and additional automation is required to implement optimization strategies. Weakness: O&M Manuals are inadequate or not accessible. Staff requires additional process training to truly understand the operational goals and constraints of the Agencys process. Weakness: The current ControlLogix redundancy configuration does not failover quickly enough to be bumpless. IEUA has reviewed their configuration with Rockwell and is implementing recommended modifications to improve the failover performance.


Architect a fully-integrated and uniform Facilities SCADA system. Identify the level of instrumentation and process equipment automation required to

achieve full automation. Re-publish Operations and Maintenance (O&M) manuals to incorporate updated P&IDs

and Process Control Narratives. Create electronic O&M manuals from the new content and integrate with SCADA for

context-sensitive help. Provide staff with process control training using the re-published O&M manuals.

Videotape the training for re-use with new staff or a refresher for existing staff. Model workflow for Standard Operating Procedures using a product such as GE

Workflow to enable the automated management and trigger of SOPs. Videotape Agencys most experienced Operators performing SOPs for training of new

staff.


21

Alarm and Event Management Industry Best Practices Current State Adapting a published industry standard such as ISA-S18 and EEMUA 191 to develop the Utilitys Alarm Standard for alarm classification and handling is a best practice. Consistent presentation and visualization of alarms is critical for ensuring that Operators understand the meaning of alarms and the required response. Ensuring that alarms are meaningful and that Operators are not overwhelmed with nuisance alarms is accomplished by classifying alarms based on criticality and implementing smart alarming strategies. Condition-based alarming (smart alarming) adjusts alarm properties based on the mode of operation. For example: a low-flow alarm is not activated if the supply pump is not running. Develop a formal process to review new alarms. Over time, process changes will introduce new alarm requirements. If not addressed through a formalized process these new alarm requirements will accumulate resulting in an increasing number of nuisance alarms and an ineffective alarm system. Equipment condition alarms or equipment key performance indicators (e.g. high bearing temperature) are important for preventative maintenance but should not be classified as high priority alarms unless they represent a safety risk.

Strength: Current alarm configuration and management is meeting requirements of Facilities operations and provides effective alarm-event annunciation. Strength: The Agency implemented FoxRay software from LimeWare about a year ago. Foxray generates daily and monthly alarm summary report for priority 1 alarms. The Agency also uses alarm extractor in Foxboros WISE (Web Information System for Executives) to access alarm/event reports. Weakness: The Facilities SCADA Systems suffer from numerous nuisance alarms since almost every event has been configured as an alarm. In addition too many alarms are configured as high priority. A high volume of nuisance alarms creates the risk that critical alarms will be missed by Operations staff. Weakness: The Agency DCS group follows a defined alarm strategy. However, this strategy is not documented as an Alarm Standard. Weakness: The alarm summary screens between Foxboro DCS and Rockwell Automation systems use different color and display schemes. Foxboro system also does not automatically remove cleared alarms.

Initial Recommendations Document Agency Alarm strategy in an Alarm Standard. Institute an annual alarm remediation project to ensure proper alarm classification. Define Operations and Maintenance KPIs and configure as events. Create smart equipment HMI templates that provide Operators with meaningful

information about active alarms, interlocks and equipment KPIs.


22

Data Management and Reporting

Industry Best Practices Current State Historians are configured to capture all the data required for regulatory compliance, operator decision support, preventative maintenance, and planning. The data that must be captured includes data that cannot be re-created or is important for analyzing the sequence of events: I/O readings (AI, AO, DI and DO); alarms and events, set points, and process calculations used in process control. Data capture is fully automated to minimize data errors resulting from manual collection and entry of data. Data capture employs redundancy and store-and-forward techniques to mitigate data holes. For reporting data is derived from a single source to ensure consistent results or single version of the truth. Critical data used for regulatory compliance and revenue generation undergoes data validation process. End users have data access tools that provide a user-friendly, graphical interface that does not require specialized programming knowledge to retrieve data. Business Intelligence (data access) tools should be provided for the creation of scheduled reports, ad-hoc reports and dashboards. All data is stored in a least two, physically separate locations to support disaster recovery. Data is integrated with other business systems (maintenance, financial, laboratory) to provide an enterprise view of data and support effective decision making.

Strength: Most of the data required by the Facilities operating groups is captured by the historical databases. Strength: Data is captured by-exception as well as at intervals using remote data collectors with ability to store-and-forward data in the event of a network or historian failure Weakness: The data required by the operating groups for regulatory and operations reports are split between multiple historical databases. Some instances require Operations Staff to generate partial reports on each historical database and then manually merge the data into the finished report. This effort is very time consuming and introduces the possibility of transposition errors. Weakness: The tools available for accessing data and report generation are too programming intensive for Operation Staff to utilize. As a result, Operations Staff is dependent on the DCS group to retrieve and organize data required for reporting. The DCS group shifts and reprioritizes other maintenance effort to provide the necessary reporting support. Operations Staff is unable to leverage the available historical data for decision support due to the time delay between data need and data access. Weakness: Each system collects and manages the historical data at respective plants, but the separate historian servers are not linked, synchronized or consolidated. Report generation is done independently on Foxboro and RA systems. As a result, to obtain Agency wide operations reports, Agencys staff is required to run the reports on each system and then consolidate the data manually. Weakness: There are no dashboards or high-level summary reports that allow IEUA management to quickly review Key Performance Indicators.


23

Data Management and Reporting


Implement a consolidated historian that captures and manages all data in a single location.

Implement a replicated SCADA Historian in a protected area (DMZ) to enable secure data access by users not directly connected to the Control System Network.

Deploy a graphically-driven, user-friendly, business-intelligence tool. Train operations and engineering staff member to provide staff with the ability to create their own reports.

Create reports and procedure for validating critical data.

Cyber Security Industry Best Practices Current State The power industry is governed by the NERC CIP standards which specify the requirements for Cyber Security and annual vulnerability assessments to verify compliance. Although it has been suggested that the water industry will eventually have to follow NERC CIP, the current situation is that water organizations (DHS and EPA) have only published cyber security guidelines for the water industry. There are numerous SCADA Security standards and certifications managed by Standards Organizations (e.g. ISA, DHS, NERC, ISO). The security standard that the Agency follows is less important than adopting a standard because they are all very similar. NERC requires power utilities to complete annual vulnerability assessments (VA) to identify their security risks. There is no recommended frequency for water utilities but once every 2 years should be adequate. The development of written Security Policies and Procedures and staff training is necessary for the long-term security of the SCADA system. Security in-depth (i.e. multiple layers of locked doors) is considered a best practice and includes: physical security, port-security, AAA security (authentication, authorization, and accounting), intrusion detection, anti-virus protection, and firewalls.

Strength: The Agency has followed many best practices for SCADA security including: protecting critical systems components in locked enclosures; installation of a firewall between the SCADA network and other networks; encryption of wireless SCADA communications; and creation of a De-Militarized Zone (DMZ) as a staging area for remote access by external users. Strength: Wireless network traffic is encrypted using Advanced Encryption Services (AES) and the plant SCADA networks are protected behind firewalls. Weakness: The Agency does not have published security policies and procedures. Weakness: The Agency has not adopted a specific security standard to follow and has not published an IEUA Cyber Security Standard for the SCADA network. Weakness: The Agency has not performed a VA to identify potential weaknesses in the SCADA system.


24

Cyber Security Industry Best Practices Current State To enable unattended operations and to enhance the continuity of operations, operations, maintenance and SCADA staff members require secure remote access to the SCADA system. The recommended approach involves authenticating into the SCADA network through a central gateway that is managed by an AAA server and employs two-factor authentication and encrypted traffic. Initial Recommendations

Adopt a Security Standard to guide security design and implementation of the SCADA system.

Document Security Policies and Procedures and train staff. Perform a Cyber Security Vulnerability Assessment and Remediation project. Develop a Logical Network Design Standard that includes Cyber Security.

Disaster Recovery/Business Continuity

Industry Objectives Current State Regulatory compliance and the continuity of operations are highly dependent on a reliable SCADA system. A SCADA disaster recovery and business continuity plan addresses all aspects of SCADA that impact system availability and recovery. It is important that continuity of operations for the SCADA Systems be addressed in the Utilitys DR/BC Plan.

Strength: Although the Agency does not have set goals for system availability that have design and implemented a high reliable system with extensive redundancy. Weakness: IEUA does not currently have a Disaster Recovery/Business Continuity Plan that specifically considers the impact of SCADA System failures.

Initial Recommendations Establish and implement data back-up and recovery tools and procedures for disaster

recovery. Prepare a SCADA Disaster Recovery and Business Continuity plan.


25

Integration

Industry Objectives Current State O&M Performance Management: Define performance metrics that are clear indicators of how well the Agency is executing on its mission and goals. Calculating and displaying these key performance indicators in near real-time provides management with timely feedback on the effectiveness of their decisions and actions. In addition, trending or alarming on these KPIs bring problem areas to the attention of managers while there is still an opportunity to affect change. Enterprise Integration: integration of operations data with business systems such as financials (FIS), laboratory (LIMS), maintenance (CMMS) to support effective and timely business decisions. Equipment Condition Metrics: Define equipment performance metrics that are a measure of equipment condition. Trend these metrics and integrate with a utilitys CMMS to support a predictive maintenance strategy and to trigger maintenance activities.

Strength: The Agency has built the network and IT infrastructure to support future integration. Weakness: The applications and tools required for integration have not been deployed. Weakness: In general, information from the existing SCADA Systems is not integrated with other IEUA Business applications. Weakness: The Agency has implemented a financial system based on SAP. A Computerized Maintenance Management System (CMMS) component is currently being implemented. These systems have not been integrated with operations data from SCADA.

Weakness: In order to develop system-wide reports, information from each of the plant Historians must be gathered and manually transferred to Microsoft Excel spreadsheets. Information from other systems, such as LIMS, is then added.

Initial Recommendations Define Operations and Maintenance (O&M) key performance indicators (KPIs). Incorporate O&M KPIs into management dashboards. Determine data integration requirements, evaluate and implement preferred integration

strategy. Define Equipment key performance indicators that are asset condition metrics. Configure equipment KPIs and integrate with CMMS

Instrumentation

Industry Best Practices Current State Routine review of instrumentation against process requirements will determine: 1) if all process variables requiring measurement are instrumented; 2) do installed instruments have the right accuracy and resolution; 3) are instruments installed properly; 4) is there

Strength: Operations staff reports that the type and quantity of instrumentation in use at the treatment facilities is appropriate. Instrument calibration and maintenance are performed in a timely manner.


26

Instrumentation

Industry Best Practices Current State instrument redundancy for critical process variables; and 5) has the right instrument technology been employed. Smart instruments with digital communication interfaces (such as HART, Profibus, DeviceNet) are able to transmit considerable information about the performance of the instrument including calibration records and maintenance status. This capability adds a small, incremental cost to instruments for the potential value. SCADA design standards should address instrument application, preferred instrument vendors, and installation details. All instrumentation critical to regulatory compliance and safety should have documented calibration and maintenance procedures. Ideally, this effort would be scheduled through the utilitys computerized maintenance management system (CMMS) and procedures would be generated with the PM work order.

Weakness: The lack of Instrumentation and Control standards has resulted in the incorrect selection and installation of instruments. This requires Agency staff to correct the problem post-construction, which has a negative impact on Operations, Engineering, and DCS support and cost. Weakness: Operations staff members have indicated that there is inadequate instrumentation in the collection, treatment, and distribution systems to effectively monitor system performance. Process optimization would require additional instrumentation and automation.

Initial Recommendations Evaluate the need for additional or modified instrumentation and automation as part of

optimization feasibility studies for the plants, collection, and distribution system. Include instrument standards (including preferred vendors and products) in the SCADA

design standards. Select a standard digital communication interface for all future instruments and

incorporate into standards. For example Rockwell Automation provides Logix-family analog input modules that have an integrated HART interface and 4-20 mA.

Incorporate instrument upgrades, required for optimized operations, into planned capital projects.

Establish calibration procedures and schedules for critical instrumentation stored and managed within the CMMS.


27

3 Recommended Delivery Approach

3.1 Full Implementation The recommended approach for a project of this magnitude is that it be implemented in phases that are, in effect, smaller self-contained projects that are implemented in their entirety (design, construct, program and commission). The benefits to this multi-phased approach include:

1. Early Realization of Benefits: Smaller phases can be completed quicker bringing the benefits of the new system to the Agency earlier.

2. Shortened Schedule: Phased implementation provides clear separation of like effort, thus enabling phases to be implemented in parallel or staggered (rather than sequential) by multiple delivery teams resulting in a shortened schedule.

3. Minimize Operational Disruption: Phases are segmented and scheduled to meet the operational needs and priorities of the Agency. While a system is being deployed, there is some level of disruption to the operational routine. Expediting short, encapsulated phases mitigates this disruption.

4. Resource Leveling: The traditional design-all then construct-all approach can result in significant spikes in specialized resource requirements (e.g. designers or programmers) that are exacerbated on large projects. A multi-phased approach enables staff to move immediately from one phase to the next improving efficiency and continuity of resources over the entire project lifecycle.

5. Enhanced Quality and Value: Continuity of resources improves quality and uniformity (e.g. the same group of programmers moves from one phase to the next) over the entire project lifecycle. There is no need to constantly re-educate new project staff on system requirements, standards and lessons learned from phase to phase.

6. Flexibility in Delivery: It is not uncommon for new requirements or contracting conflicts to emerge during execution of multi-year projects. If the entire project is implemented in one phase, new requirements that affect the whole project can represent significant and expensive change orders. A phased approach enables new requirements to be incorporated in future phases with much less impact to schedule and cost.

3.2 Pilot Project Well defined, documented and proven SCADA Software Standards and Modules (Objects) are the cornerstone to implementing a system that is reliable, maintainable and consistent in its look-and-feel. Utilizing an integrated team, which is comprised of Agency staff and an experienced,


28

professional consulting firm, for standards development and validation not only ensures high-quality Standards, it also supports knowledge transfer and ensures the creation of an implementation Oversight Team with hands-on experience with the project Standards. Thus enabling the Oversight Team to mentor other implementation teams and manage their compliance with the Agency standards.

As a minimum, the SCADA Software Standards and Modules developed in this phase would cover the following: software delivery methodology and test standards, Process Control Narratives, program structure and object development, I/O and device objects (both HMI and PLC), intelligent alarming, equipment KPIs and historical data capture. The standard developed for Process Control Narratives (PCN) will provide a consistent presentation of the control strategy to support the conversion of existing control software. In addition, the data management plan will define data requirements for each defined asset class and will architect the necessary software hooks for integrating future operations optimization applications.

Applying proven standards will significantly reduce system delivery risk and optimize operator acceptance and satisfaction. It is recommended that the standards be validated on a single facility as a Pilot Project. A Pilot Project offers numerous benefits beyond validation of design and software standards. These benefits include:

Develop and validate re-useable Software Modules for common equipment and unit processes. When executed effectively, the risk and effort associated with implementing the following enterprise SCADA phases are significantly reduced.

Validate migration strategies and methods for transitioning from the old to the new SCADA system. Testing these strategies on a Pilot Project minimizes the possibility of widespread service disruption.

Identify any unanticipated issues and validate constructability.

Provide Operations staff with operating experience on the new SCADA system so that they can provide operational feedback on the system configuration early in the project. It also provides the Operations staff with a training tool to ease the transition on future phases.

Following completion of the Pilot Project, the SCADA Standards and Software Modules are revised to reflect the as-built state and then published.

To ensure project continuity, it is recommended that the Pilot Project delivery team be part of the implementation oversight and standard compliance team for the full enterprise implementation. As leaders of subsequent phases, they ensure that delivery teams understand and comply with standards and templates. Where feasible, it is also recommended that the Agencys technical and operations staff actively participate in the Pilot Project. This promotes team building and buy-in by all stakeholders and effective knowledge transfer to support the full-scale system as it is implemented.

3.3 Recommended Project Phasing The final implementation and migration strategies for the delivery of the Agencys Enterprise SCADA System will be developed during the Design and Pilot Project phases of the project with input from Operations, Maintenance, and Engineering. The following approach is a high-level framework designed to: maintain operational continuity; minimize delivery risk and staff impact; achieve the desired implementation duration and resource leveling; and achieve the end result of a system that is operator-friendly, maintainable, scalable, and has a uniform look-and-feel.


29

3.3.1 Supervisory Control Design-Build As Design-Build methodology is recommended for implementing the Supervisory Control layer of the Agencys Enterprise SCADA system. Design-Build offers the Agency the shortest possible project duration and potentially and lower project cost by eliminating the bid phase. In addition, uniformity of design and implementation will be enhanced by competitively selecting a single Consulting team to deliver the entire supervisory layer. The following is the recommend project phasing for the Supervisory Control Layer to achieve the desired goals (uniformity of system implementation, mitigation of operational impact, and optimized project schedule):

i. Phase 1 Architect and Design: Architect and design the Agencys Supervisory Control Layer for the Enterprise SCADA System. It is recommended that an Integrated Project Delivery team comprised of Agency staff, the selected Consultant, and Rockwell be created to deliver Phase 1 and 2 (Architect and Design, Pilot Project). An integrated team consisting of a professional SCADA Consultant, Agency staff, and the vendor will enhance collaboration and knowledge transfer to effectively translate the Agencys requirements into technology solutions. The vendors participation on this team will ensure that the system is architected in compliance with their recommended best practices for optimal performance and future developments.

ii. Phase 2 Supervisory Control Pilot Project: Complete the implementation of a Pilot Project for the Supervisory Control layer to: test and refine migration strategies; validate and enhance SCADA Standards based on Operator feedback; develop HMI objects for plant specific equipment and processes; optimize system performance; and resolve any outstanding integration issues. This effort will leverage the previous Standards development completed as part of this Master Plan and the implementation of the Northwest Recycled Water SCADA Upgrades for the 1630 W. Pump Station and Reservoir projects.

iii. Phase 3 - Supervisory Control Full Implementation: Complete the implementation of the Supervisory Control layer at all of the Agencys facilities based on a phased approach designed to mitigate the impact on operations and staff. The phased approach for the Supervisory Control layer will be finalized during the design phase with input from Operations and Maintenance, possible alternatives include:

Completing each plant in sequence and within each plant completing an entire process area before moving to the next process. For example: completing all of RP1 before moving to RP2 and within each plant completing all of Aeration before moving to the Secondary Clarifiers. The advantage of this approach is that it minimizes the duration that anyone plant is disrupted by the migration and the duration that a plant is operating with two systems (Rockwell and Foxboro).

Completing each process area in sequence across all the plants before moving to the next process area. The advantage of this approach is that it minimizes the duration that Operations will be required to operate the same process area across the plants on two systems. For example: completing all of the primary clarifiers for all of the plants in sequence will minimize the duration that the Agency is operating clarifiers on both Foxboro and Rockwell.

3.3.2 Direct Control Design-Bid-Build Phase 1 Pilot Project: The purpose of the Pilot Project is to validate and refine the migration strategies, create and validate design templates and standard specifications, further refine PLC


30

software standards, and resolve any unforeseen integration issues. The Direct Control Layer, Pilot Project will be implemented immediately following the implementation of the Supervisory Layer at all of the Recycled Water Plants. The delivery methodology will be a traditional Design-Bid-Build, whereby the selected Consultant designs the system and creates bid documents to competitively select a Contractor for the supply and installation of PLC based controls. It is recommended that the Consultant also provide the PLC programming for the Pilot Project to ensure the end product results in well documented, high-quality, reusable software templates.

Phase 2 Remaining Facilities: Following completion of the Pilot Project the remaining Direct Control layer will be rolled out in three (3) phases. The phasing strategy will be determined with input from Operations during the Pilot Project to minimize the impact on Operations and staff. One phasing strategy involves completing an entire plant within a phase and the other is to complete an entire process area across all the plants in a single phase. Each of the remaining phases will be completed as a distinct Design-Bid-Build process schedule to ensure that there are not over-lapping construction phase.

3.4 Mitigating Operational and Staff Impact The final migration plan will be developed during the design phase with input from all affected stakeholders. However, the following strategies are guidelines for developing migration that migration:

Migration Plan: Develop a detailed Migration Plan during the design phase logically sequences the migration to mitigate the impact on operations and staff. In addition the plan must address change management issues by analyzing and developing strategies for dealing with the what-ifs and system failures. A Stakeholder Workshop is the best format for working through these scenarios with stakeholders. Where possible, the Migration Plan will include a strategy for backing-out of the migration if it goes poorly or if backing-out is not possible incorporate adequate contingency measures.

Existing System: Maintain the existing Supervisory Control Systems until the new systems is fully deployed and tested. This provides a familiar platform for Operations until they are adequately trained on the system and also provides a stable platform in the event of migration problems.

System Staging: The more pre-deployment staging is representative of the in-situ environment and the more thorough the testing the more likely the migration will be bumpless. It is recommended that the both the Agency and selected Consultant build a Software Lab to stage and test the software prior to deployment.

Pilot Project: Implementing a smaller subset of the total system as a pilot (or test) project enables the delivery team to prove out concepts, validate standards, and identify unforeseen integration issues. In addition to lowering the risk and optimizing delivery for the larger system, Pilot Projects also provide Operations and SCADA Support staff with the opportunity to gain experience with the new system early in the upgrade process.

Structured Methodology: The root cause of most system deployment and stakeholder satisfaction problems stem from the failure of the implementation team to adhere to the stages and quality control gates of a structured software methodology. The right time to determine that the process control strategy has not been interpreted properly or that a failure mode was not considered is early in the development process not at the deployment stage.

Acceptance Testing: Thorough bench, factory and site acceptance testing are the quality


31

control gates for validating that system has been programmed in compliance with standards and meets system functional requirements. Testing the developed software with a device/process simulator, configured to emulate the real-world, is an effective method for minimizing the time spent commissioning the system after deployment.

Revision Control: Acceptance testing almost always identifies changes required to the tested software. In addition to logging the required changes in an Issues Log it is important that software be checked into a revision control system immediately prior to factory acceptance testing and that all future changes be made to the revision controlled software so that changes can be audited.

Project Phasing: Breaking up the project into phases which are smaller encapsulated projects that can be started and completed quicker than the whole reduces implementation risk and operational disruption. Phases are segmented and scheduled to meet the operational needs and priorities of the Agency.

Operations Training: Training Operators and Support staff prior to software deployment enables the Agencys staff to assist in the changeover and gain valuable operating experience with the new system prior to commissioning.

4 Desired State

4.1 Mission and Vision The primary mission of Inland Empire Utilities Agency (Agency) is the production of high quality, recycled water through the collection, transmission, and treatment of wastewater that meets or exceeds California Department of Health Title 17 and Title 22.

The full scope of the Agencys mission incorporates secondary services as stated by the Agency in their mission statement below:

The mission of the Agency is to supply imported and recycled water; collect, treat, and dispose of wastewater; and provide other utility-related (renewable electrical energy, compost) services to the communities it serves. The Agency strives to provide these services in a regionally planned, managed, and cost-effective manner.

Inland Empire Utilities Agency (IEUA), at our wastewater facilities, is focused on producing three key products:

Recycled water to drought-proof our service area,

Compost to ensure healthy soils, and

Renewable energy through methane gas and solar generation

SCADA Vision:

The Agencys long-term, vision for their SCADA system is to build a fully integrated and uniform SCADA system that provides enterprise-wide control and the information necessary to optimize operations. The Agencys vision for SCADA supports the delivery of their primary mission in a regionally planned, managed, and cost-effective manner. The following statements summarize Agencys top-line strategies for achieving their SCADA vision:

Enterprise Integration: Implement a fully integrated and uniform SCADA system to provide the Agency with the operational flexibility required to control any-facility from


32

any-location and enable enterprise-wide, coordinated strategies for wastewater treatment, recycled water supply, and energy management.

Information Driven: Provide all stakeholders with direct access to the quality operations and business data required to support regulatory compliance, operations decision support, process optimization, planning, asset management, and energy management.

Operations Improvement: Leverage the Agencys process expertise and enterprise-wide, SCADA infrastructure to implement optimal control strategies to reduce the cost of operations, improve revenue, and water quality

SCADA Mission:

The Agency relies heavily upon SCADA for the optimal delivery of their primary mission making SCADA the Agencys most critical cyber asset. The planning, design, and management of the Agencys SCADA System must address the following requirements:

Provide the infrastructure, tools, and information to enable Operations to maintain regulatory compliance and meet their levels of service in an efficient manner.

Plan and manage the growth, enhancement and maintenance of the Agencys SCADA system assets to achieve the SCADA performance required by Operations.

Provide Agency stakeholders (executive management, operations, maintenance, planning, and engineering) with user-friendly access to operations data.

4.2 Business and Technical Requirements As a result of site visits, staff interviews, the Stakeholder and Vision Workshops, the following high priority, business and technical requirements were identified by Agency SCADA stakeholders:

Enterprise SCADA System: The design and construction of a highly-reliable, scalable, enterprise SCADA system that enables the control of any-facility from any-location, unattended operation, and supports system-wide, wastewater treatment and recycled water supply-demand management.

Migration to a Uniform SCADA System: Plan and build a migration to a SCADA system that is uniform in its design, configuration, documentation, and components.

Maintain and Optimize SCADA Assets: Maintain, integrate, and optimize SCADA assets to maximize their lifecycle, meet business needs, and mitigate disruptions to operational continuity.

Minimize Project Implementation Risk: The development of a comprehensive suite of SCADA Standards to minimize system delivery risk, reduce system variability, enhance system quality, and ensure that Agency SCADA staff members have the tools to support the system post-commissioning.

Highly Functional and User-Friendly Information Solution: Design and build an Operations Data Management System that consolidates data from all facilities and provides a user-friendly, graphically driven, user-interface for data access by stakeholders including: executive management, operations, maintenance, engineering, and planning.

Reliable Operations: Institute the technology, organization, practices and procedures to


33

meet the SCADA performance required for operations.

Optimize Operations: Implement the technologies, organization and practices that enable optimized operations including increased instrumentation, automation, and enterprise integration.

Long-term Stable Pricing for SCADA Components and Support: Negotiate p

Inland Empire Utilities Agency SCADA Master Plan · Inland Empire Utilities Agency Facilities SCADA...

Documents

Transcript of Inland Empire Utilities Agency SCADA Master Plan · Inland Empire Utilities Agency Facilities SCADA...