NERC ID Name NCR ID Actions DL...2020/05/28 · Until March 2017, the entity utilized a...

ReliabilityFirst Corporation (ReliabilityFirst) FFT O&P

NERC Violation ID Reliability Standard Req. Entity Name NCR ID Noncompliance Start Date Noncompliance End Date Method of Discovery

Future Expected Mitigation Completion Date

RFC2019021684 BAL‐002‐2(i) R2 PJM Interconnection, LLC NCR00879 1/1/2018 2/5/2019 Self‐Report Completed Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible, or confirmed noncompliance.)

On June 11, 2019, the entity submitted a Self‐Report stating that, as a Balancing Authority, it was in noncompliance with BAL‐002‐2(i) R2. On January 25, 2019, entity personnel observed a thermal trend on the sole line outlet for Lackawanna Energy Center (Lackawanna), which is a combined‐cycle power plant. Lackawanna had recently completed relay upgrades and began operating continuously above 1370 MW for the first time. Entity personnel recognized the large amount of MW on the line (i.e., around 1470 MW) and crosschecked to see if Lackawanna was identified as the Most Severe Single Contingency. It was not. The entity had identified a separate generating resource with an output of 1350 MW as its Most Severe Single Contingency. The entity investigated the issue and discovered that it had not accounted for Lackawanna as a single contingency. Rather, it accounted for individual units at Lackawanna as separate contingencies (i.e., two combustion units and one steam unit). But, since the entire plant connected to the Bulk Electric System (BES) through a single 230kv line, the sum of all the units needed to be accounted for as a single contingency (i.e., if the 230kv path were to trip or otherwise become unavailable, the entire combined‐cycle plant would be lost).

The entity conducted an extent of condition review and determined that this particular combined‐cycle plant (i.e., Lackawanna) should have been accounted for as the Most Severe Single Contingency on 23 occasions, but it was not. Other combined‐cycle plants with single connection points to the BES were similarly accounted for incorrectly. Only one of the additional combined‐cycle plants, Hanging Rock Energy Facility (Hanging Rock), should have been accounted for as the Most Severe Single Contingency on 195 occasions. In summary, the entity failed to correctly identify the Most Severe Single Contingency on 218 occasions.

The root cause of this noncompliance was a lack of instruction and guidance on how to account for combined‐cycle plants with multiple units that could be dispatched independently. And, the issue repeated itself and persisted due to a lack of adequate preventative and detective controls.

This noncompliance involves the management practice of grid operations. Entities should strive to maintain situational awareness of operations by implementing well‐defined and executable processes and procedures and combining appropriately skilled and trained staff with appropriate work tools. A failure to maintain situational awareness negatively impacts an entity’s ability to plan for, and effectively respond to, system events. This noncompliance also involves the management practice of measurement and analysis. Entities should strive to implement processes, procedures, internal controls, and technology to ensure the accurate collection and analysis of data. Then, the data and analysis can be relied upon to make informed decisions.

This noncompliance started on January 1, 2018, which was the effective date of BAL‐002‐2(i) R2, and ended on February 5, 2019, when the entity correctly accounted for combined‐cycle plants in its footprint with single connection points to the BES.

Risk Assessment This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system (BPS) based on the following factors. A violation of BAL‐002‐2(i) R2 has the potential to affect the reliable operation of the BPS by increasing the likelihood that a responsible entity will not have sufficient contingency reserves available to meet Firm Demand and export obligations after a Most Severe Single Contingency event (i.e., the greatest loss of resource output measured in MW). A sizeable loss of generation, and the corresponding imbalance in generation and demand, could lead to abrupt decreases in frequency, cascading outages, and complete frequency collapse. The risk was not minimal in this case because of the duration of the noncompliance, the number of instances of noncompliance, and the apparent lack of adequate preventative or detective controls. However, the risk was not serious or substantial based upon the following facts. In every instance, despite not identifying the correct Most Severe Single Contingency, the entity’s BAL‐002 reserves exceeded the required amount. This occurred for the following reasons. First, during every instance, the entity identified the second largest single contingency, which had a similar generating output as the correct Most Severe Single Contingency. In fact, on any given day in the entity’s footprint, several generating units/plants are operating with generating outputs that are in the same range as the generating outputs of Lackawanna and Hanging Rock. This is important because it means that as long as the entity made preparations to have a Contingency Reserve equal to, or greater than, any of these units/plants, its Contingency Reserve would never be grossly inadequate to address an event associated with a loss of output from Lackawanna or Hanging Rock. Second, as part of its program, the entity adds 190 MW to its identified largest contingency when procuring reserves in real time. In addition to the foregoing, the entity had 30‐minute reserves equal to 150% of its identified largest contingency, which further minimized the potential harm associated with this noncompliance. Lastly, balancing and frequency control for a rapid reduction of system generation occur over a continuum of time using different resources (e.g., generator governors, reserve deployment, under‐frequency relays that interrupt pre‐defined load, and load shedding programs). In summary, even though the entity failed to identify the true Most Severe Single Contingency in each instance, it maintained enough reserves (and additional mechanisms and countermeasures were in place) to address such contingencies. It is also worth noting that even though the entity lacked adequate preventative and detective controls to address this noncompliance, the issue was identified, and ultimately resolved, because of the vigilance and questioning attitude of entity personnel. No harm is known to have occurred.

ReliabilityFirst considered the entity’s compliance history and determined there were no relevant instances of noncompliance. Mitigation To mitigate this noncompliance, the entity:

1) developed a document to provide guidance on how to correctly sum the total plant output of multiple units associated with one combined‐cycle plant, where each unit may be dispatchedindependently; and2) reviewed similar combined‐cycle plants in the entity’s Energy Management System and updated the calculation of the total plant outputs.

A-1 Public Non-CIP – Find, Fix, Track, and Report Consolidated Spreadsheet

Last Updated 05/28/2020 1




RFC2019021684 BAL‐002‐2(i) R2 PJM Interconnection, LLC NCR00879 1/1/2018 2/5/2019 Self‐Report Completed ReliabilityFirst has verified the completion of all mitigation activity.






RFC2019022104 PRC‐005‐6 R3 Public Service Electric & Gas Company NCR00896 8/1/2017 2/16/2018 Self‐Report Completed Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible, or confirmed noncompliance.)

On August 16, 2019, the entity submitted a Self‐Report stating that, as a Distribution Provider and Transmission Owner, it was in noncompliance with PRC‐005‐6 R3.

The maximum maintenance intervals for communications systems are provided in Table 1‐2 of PRC‐005‐6. Verification of communications systems that have continuous monitoring or periodic automated testing have a maximum maintenance interval of 12 calendar years. However, communications systems that do not have continuous monitoring or periodic automated testing must have their functionality verified and documented every four (4) calendar months. The entity in question owns a total of 31 power line carriers that have component attributes that are subject to either the monitored or unmonitored maintenance intervals of PRC‐005‐6 Table 1‐2. Until March 2017, the entity utilized a programmable logic controller (PLC) to perform a full operational daily self‐check of one of the entity’s 500 kV lines. In addition to performing periodic automated testing, the PLC would send an alarm to a 24‐hour manned facility if a test was unsuccessful. Together, these traits created a communications system that had a maximum maintenance interval of 12 calendar years.

In March 2017, the entity permanently retired the remedial action scheme that operated the PLC. This rendered the continuous monitoring and periodic automated testing inoperable. As soon as the PLC was retired, the communications system became an unmonitored system for purposes of PRC‐005‐6, and the maximum maintenance interval for verifying functionality became four (4) calendar months. During the noncompliance, the entity performed visual inspections of all communications systems as part of weekly tours. While the tours did check to make sure that the equipment was operational and that no alarms were present, the tours did not include a step to explicitly verify and document the proper functioning of the communications system as required by PRC‐005‐6. The only verification efforts between August 1, 2017 and February 16, 2018 were the weekly visual inspections. The entity did not realize that the automatic testing feature of the PLC was inoperable until December, 2017. At that time, the entity began developing a new procedure to test and document the functional check on the line’s communication systems.

The root cause of this noncompliance, was a lack of understanding of the systems that the entity had in place, and their impact on PRC‐005‐6. Specifically, when the entity retired its remedial action scheme, the effects of that retirement and how it would impact the PLC were either not communicated or not understood, resulting in the noncompliance.

This noncompliance involves the management practices of grid maintenance and verification. Grid maintenance is involved because the entity it did not fully understand the impacts of retiring the remedial action scheme, and in turn, the PLC. Verification is involved because the entity did not have proper internal controls to identify the impact of retiring the remedial action scheme.

The violation began on August 1, 2017, four (4) calendar months after the communications system shifted in classification from monitored to unmonitored due to disabled PLC. The noncompliance ended on February 16, 2018, when the entity performed functionality testing required under PRC‐005‐6 for unmonitored communications systems.

Risk Assessment This noncompliance posed a moderate risk and did not pose a minimal or serious or substantial risk to the reliability of the bulk power system. The risk in this noncompliance is not minimal because of the following factors. First, the noncompliance involved a 500kV line, the failure of which has an elevated potential magnitude of harm. (The entity does not have any critical facilities as defined by the Reliability Coordinator. Additionally, the entity does not have an Interconnection Reliability Operating Limit (IROL). However, the affected line serves a Nuclear facility and connects to two other 500 kV stations.) Second, the entity did not verify the carrier’s functionality for 11 months, when it should have been verified every four (4) months. The risk in this noncompliance is not serious because of the following. The entity performed weekly visual inspections of all communications equipment onsite to check that it was functioning properly and that no alarms were present during the time of the noncompliance. ReliabilityFirst also notes that there is no indication that the carrier experienced a failure without the entity knowing during the period of noncompliance. Finally, no harm is known to have occurred.

The entity has relevant compliance history. However, ReliabilityFirst determined that the entity’s compliance history should not serve as a basis for applying a penalty because while the result of the prior noncompliance was arguably similar, the prior noncompliance arose from a different cause.

Mitigation To mitigate this noncompliance, the entity:

1) developed and implemented a written procedure for manual functional testing at Salem;2) reported all testing results for the Salem 5021 carrier for recording in CMMS maintenance database;3) added the Salem 5021 line to the entity NERC Compliance Group carrier checklist, which internally requires testing to be performed at a minimum of every three months, with an automatic reminderfor monthly follow up; and4) determined the scope of carriers which are both designated as “Continuously Monitored” and were subject at any time to a period of greater than four calendar months without performing functionaltesting. Once this scope was determined, verified that the Continuously Monitored function currently exists by reviewing the design of the component and performing a field walk down. If theContinuously Monitored function is disabled on any of the carriers within this scope, perform an additional review to determine if a potential non‐ compliance may have occurred in the past. Note that allentity carriers now have functional tests performed at a minimum of every three months as described in Milestone Activity 3.ReliabilityFirst has verified the completion of all mitigation activity.





SERC2018019333 PRC-023-4 R1 Ameren Services Company (Ameren) NCR01175 04/01/2017 01/09/2018 Self-Report Completed

Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible, or confirmed violation.)

On March 2, 2018, Ameren submitted a Self-Report stating that, as a Transmission Owner, it was in noncompliance with PRC-023-4 R1. Ameren reported six instances where the relay settings for its transmission lines did not meet PRC-023-4 R1, Criterion 1 for relay loadability. Under Criterion 1, the relay settings must not operate at or below 150 percent of the highest seasonal Facility Rating of a circuit.

On January 1, 2014, the Planning Coordinator (PC) notified Ameren that it designated 144 of Ameren’s line terminals, operating at 100kV to 200kV, as critical lines, which were required to be in compliance by April 1, 2017. Ameren determined that 9 of those terminals required relay upgrades, which were completed by the end of 2016, and 22 terminals required relay settings changes, which were assigned to several System Protection Engineers to calculate new settings. One of the Engineers calculated the relay settings for six terminals which resulted in a relay load limit (RLL) that was the most limiting component of the line terminal. The Engineer did not understand that the relay could not be the most limiting component and that relay settings were required to remove the relay as the most limiting component. Therefore, the incorrect settings for the six relays ranged from 78% to 99% of the equipment limit. The highest loading based on the correct setting was 62%.

The first incorrect relay was not discovered until November 9, 2017, when Ameren Transmission Operations performed a cursory review of flowgates after the change to winter ratings and noticed the RLL was limiting the line rating on a line. The five additional relays were discovered on November 24, 2017, when Ameren completed a review of the PRC-023 database and checking the database against the PC’s 100kV-200kV critical line list to determine the extent of the noncompliance. The review included two independent reviewers to verify accuracy. On January 9, 2018, Ameren changed the relay settings for the six relays to meet Criteria 1.

This noncompliance started on April 1, 2017, when Ameren was required to meet the PRC-023-4 R1, Criterion 1 requirement, and ended on January 9, 2018, when Ameren implemented the new relay settings.

The cause was management oversight, specifically, a misinterpretation of the standard and an inadequate internal control. Management failed to ensure that the Associate Engineer understood the requirements of PRC-023 and that the relay could not be the most limiting component, therefore, relay settings were required to remove the relay as the most limiting component. Additionally, Management failed to ensure independence of the internal control between the Engineer performing the work and the individual responsible for performing the crosscheck. The same Associate Engineer was responsible for calculating the new settings and crosschecking the revised settings. If an independent individual performed the crosscheck, it is likely the individual would have caught the mistake.

Risk Assessment This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system. Specifically, a Bulk Electric System (BES) relay could open a transmission terminal on load and interrupt BES flow. The Ameren PC identified these lines as critical based on an existing monitored flowgate. As Ameren monitored these flowgates, Ameren would have addressed any loading which would have resulted in flows exceeding the RLL through transmission system and/or generation adjustments. Based on hourly power flows recorded on these lines, Ameren did not approach the RLL limits. Also, Ameren's historical RLL calculation, which includes a 143% margin at 90% power factor, exceeds the highest seasonal thermal loading limit on the line. No harm is known to have occurred.

SERC considered Ameren’s compliance history in determining the disposition track. Ameren’s relevant prior noncompliance includes NERC Violation ID SERC2013012889. In SERC2013012889, unlike in the instant case, Ameren used an incorrect formula to program the settings. Thus, the cause and the mitigation of the 2013 issue would not have prevented the instant issue.

Mitigation To mitigate this noncompliance, Ameren:

1) retrained System Protection Engineers on the requirements of PRC-023;2) completed a risk assessment of PRC-023;3) developed a Change Management Process, which requires a cross check of the relay load limit between System protection and the Transmission Operations prior to restoring any line to ensure

that no relay limits a line or facility;4) performed an extent-of-condition to verify that all designated terminals from the Planning Coordinator meet or are scheduled to meet the requirements of R1 within the required time frame;5) performed an extent-of-condition to verify that all relay load limits are shown correctly in the Transmission Operations Facility Rating database and all limiting elements from that database are

shown correctly in the relay load limit database and verify all load limits;6) updated Ameren's PRC-023 process to add internal controls based on the output of the risk assessment;7) developed and implemented a formal review process to verify relay load limits prior to a setting being issued to the field, which was communicated to staff by assigned reading tasks and verified

by sign-offs on the process;

O&P



8) developed and implemented a closeout process to verify changes to PC Critical Lines have been resolved and cross-checked, which was communicated to staff by assigned reading tasks andverified by sign-offs on the process; and

9) developed and implemented a process to verify that all Relay Load Limits are being continuously updated and verified within the Facility Rating database, which was communicated to staff byassigned reading tasks and verified by sign-offs on the process.

O&P



NERC Violation ID Reliability Standard

Req. Entity Name NCR ID Noncompliance Start Date Noncompliance End Date Method of Discovery Future Expected Mitigation Completion Date

WECC2018020414 EOP-008-1 R6 Valley Electric Association, Inc (VEA) NCR05447 01/22/2016 10/25/2018 Self-Report Completed

Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible or confirmed violation.)

On September 14, 2018, VEA submitted a Self-Report stating that, as a Transmission Operator (TOP), it was in potential noncompliance with EOP-008-1 R6.

On July 28, 2018, VEA experienced a loss of control and monitoring functionality, that was caused by a flood of network packets that disrupted switch communications within VEA’s primary Control Center (PCC). Subsequently, VEA performed an internal investigation and discovered that its backup Control Center’s (BUCC) Supervisory Control and Data Acquisition (SCADA) system did not have sufficient independence from the PCC. Specifically, the BUCC did not have a direct communication channel to the substation network in which the field SCADA devices were located. Therefore, any SCADA traffic between the BUCC and the field devices had to first travel to the data center located at VEA’s main campus and then to the BUCC. Further investigation revealed that when VEA moved the BUCC to a new location on January 22, 2016, the design of the network did not identify that the new network configuration removed some of the segregation that had been built into the BUCC at its previous location. The root cause of the issue was attributed to VEA’s inadequate assessment of the change in the BUCC network design. As such, VEA discovered it did not have primary and backup functionality that did not depend on each other for the control center functionality required to maintain compliance with Reliability Standards, as required by EOP-008-1 R6.

This issue began on January 22, 2016, when VEA moved its BUCC location without identifying the removal of segregation in the new network configuration and ended on October 25, 2018, when VEA confirmed that the BUCC no longer depended on the PCC for functionality.

Risk Assessment This issue posed a moderate risk and did not pose a serious or substantial risk to the reliability of the Bulk Power System (BPS). In this instance, VEA failed to have primary and backup functionality for its SCADA system that it did not depend on each other for Control Center functionality required to maintain compliance with Reliability Standards, as required by EOP-008-1 R6.

Failure to have primary and backup functionality that did not depend on each other could have resulted in the BUCC being inoperable when needed. Additionally, had the PCC failed it could have resulted in VEA being unable to monitor its system footprint of 191 miles of 138 kV transmission lines and 164 miles of 230 kV transmission lines that is primarily used to serve VEA’s load of 130.5, which is especially important if an UVLS condition is met since 84 MW of VEA’s load is subject to Undervoltage Load Shedding (UVLS) to prevent voltage collapse.

VEA did not implement any preventative controls to prevent this noncompliance. However, VEA did have effective detective controls. VEA performed an event analysis review and detected the current issue. No harm is known to have occurred.

WECC determined that VEA’s compliance history with EOP-008-1 R6 should not serve as a basis for applying a penalty. The previous violation did not have the same root cause as the instant issue and is not indicative of a systemic or programmatic failure.

Mitigation To mitigate this issue, VEA has:

1) ordered and installed new firewall for the BUCC;2) compliance, SCADA, IT, and Network Groups met to review EOP-008-1 and discussed how to identify changes that could affect redundancy or independence, they implemented changes

without affecting existing redundancy and independence, and decided how to verify redundancy and independence after changes or on a scheduled basis; and3) created a workflow that has pre-install and post-install checklists to ensure that the PCC and BUCC redundancy and independence is not affected when changes take place.

WECC has verified the completion of all mitigation activity.

O&P





MRO2018020449 FAC-009-1 R1 Otter Tail Power Company (OTP) NCR01023 01/01/2008 08/22/2019 Self-Report Completed


On September 21, 2018, OTP submitted a Self-Report stating that as a Generator Owner and Transmission Owner, it was in noncompliance with FAC-008-3 R6. Because the start date of the noncompliance predates the enforceable date of FAC-008-3, the Standard and Requirement was corrected to FAC-009-1 R1.

OTP reported that it had ratings documentation errors for components of its Facilities that constituted as noncompliance with FAC-008-3 R6, as the Facility Ratings were not consistent with OTP’s associated Facility Ratings Methodology (FRM). A subset of these errors affected the most limiting element of a Facility, causing the overall Facility Ratings for a number of Facilities to be either higher or lower than the rating of the most limiting element.

The cause of the noncompliance was that OTP did not have sufficient controls in place to ensure that equipment and Facilities were rated according to its updated Facility Rating Methodology document. Additionally, OTP attributed some rating errors to deficient communications, both internal communication within OTP departments and/or communication with its neighbors.

The noncompliance is estimated to have begun on January 1, 2008, when OTP updated assumptions in its FRM that affected Facility Ratings but did not update the affected conductor ratings to reflect these change, and ended on August 22, 2019, when all the Facility Ratings were verified and corrected in the Entities Facility Ratings database, in addition to performing the required communications with the neighboring joint owners.

Risk Assessment This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system. OTP identified a number Facilities with Facility Ratings that were not consistent with its Facility Ratings methodology. The subset of those ratings errors that caused the overall Facility Rating to be either higher or lower than the rating of the most limiting element. Of the ratings errors that affected the overall Facility Rating, two discrepancies resulted in a reduction of the overall Facility’s Rating. These Facility Rating reductions were limited to between approximately 2.5% (for Winter Emergency rating) to 5.7% (for Summer Continuous rating). The remaining two errors resulted in an increase in the overall Facility Rating. OTP determined that in the instances with the minimum trip relay ratings errors, they had no impact on the overall Facility Ratings. Additionally, the duration of the noncompliance was over ten years, and OTP did not have sufficient detective controls in place to identify and correct the issue during this time. Two Facilities had an overall Facility Rating increase because of the noncompliance. The scope of the noncompliance was not limited to specific types of equipment or locations on OTP’s system. The noncompliance only affected 2% of OTP’s equipment. No harm is known to have occurred.

MRO considered OTP’s compliance history and determined it should not serve as a basis for applying a penalty. The current issue involved an incorrect sag limit in OTP’s database, which was not identified or addressed in the prior instance’s mitigation efforts (MRO2012011457).

Mitigation To mitigate this noncompliance, OTP:

1) updated conductor ratings in several facilities;2) added correct ratings in its database for several of it lines segments;3) corrected and communicated, internally and externally, operating temperature and new ratings and revised relay settings for Facilities with issues;4) corrected and verified minimal trip relays in Facility Ratings database;5) implemented process to ensure FRM updates are communicated to affected SME/department which includes:

• providing rating data directly to OTP’s ratings email inbox;• member of Delivery Planning department to attend weekly System Operations department meeting to capture information on necessary Facility Rating adjustments;• distributes Facility Ratings sheet to multiple departments in efforts of maintaining Compliance and to identify errors;

6) conducted annual refresher training of its Facility Ratings; and7) created new Facility Ratings database which included recalculating all ratings and confirming correct wind speed.

Midwest Reliability Organization (MRO) FFT O&P






RFC2019021148 TOP‐001‐4 R13 City of Lansing by its Board of Water and Light NCR00718 10/2/2018 10/26/2018 Self‐Report Completed

Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible, or confirmed noncompliance.)

On February 27, 2019, the entity submitted a Self‐Report stating that, as a Transmission Operator, it was in noncompliance with TOP‐001‐4 R13. Midcontinent Independent System Operator, Inc. (MISO) performs the entity’s Real‐time Assessments (RTAs), contingency analysis, and state estimation. As a backup method, when the Inter‐Control Center Communications Protocol (ICCP) connection is down between the entities or MISO is otherwise unable to perform the studies, the entity performs the studies internally. In each of the following instances, the ICCP connection between the entity and MISO failed for a period of time greater than 30 minutes, and the entity failed to execute its backup method of completing RTAs in a timely manner.

First, on October 2, 2018, at 1:55 a.m., a lightning strike caused two breakers to operate at a substation, thereby creating a small island on the City of Lansing system. Generation tripped offline to sustain the island. The entity’s primary Control Center is partially fed from one of the circuits that was part of the island. During the event, a system disturbance caused breakers in the Power Distribution Unit (PDU) to open and network switches to lose power in the Control Center. This caused a loss of several systems, including ICCP connections and the fiber network, which connects to approximately 55% of the entity’s substations. Complete system visibility was restored at 2:57 a.m.; ICCP connections with all entities except MISO were reestablished by 5:56 a.m.; and the ICCP connection with MISO was restored at 12:45 p.m. Upon the loss of the ICCP connection with MISO, entity personnel began the process of internally completing RTAs using PowerWorld. However, system functionality and connectivity issues delayed the completion of the internal studies. The first study results were completed at 5:00 a.m. (i.e., more than 30 minutes after the prior RTA). Despite its failure to complete RTAs, the entity’s energy management system (EMS) remained functional, and it had visibility to all supervisory control and data acquisition (SCADA) information, with the exception of a partial loss of visibility for approximately 30 minutes.

Second, on October 18, 2018, the entity experienced issues with the ICCP connection with MISO. The connection was completely lost at 12:40 p.m. when the entity attempted to failover to the backup ICCP host. Entity personnel began the process of internally completing RTAs but did not complete the first study until 2:35 p.m. (i.e., more than 30 minutes after the prior RTA). The entity and MISO continued to troubleshoot the connection while the entity was performing internal RTAs, and the ICCP connection was reestablished at 12:37 p.m. on October 19, 2018. Despite its failure to complete RTAs, the entity’s EMS remained functional, and it had visibility to all SCADA information.

Third, on October 19, 2018, as part of planned maintenance, the entity was replacing a core switch at the backup data center. The entity knew that it would lose access to the system to perform RTAs internally using PowerWorld during the replacement (i.e., it knew that its backup method of compliance with TOP‐001‐4 R13 would be unavailable). The outage window was expected to last from 5:00 p.m. to 10:00 p.m. At 7:46 p.m., the entity unexpectedly lost its ICCP connection with MISO, and it did not internally conduct RTAs for the reasons described herein. The connection with MISO wasrestored at 10:18 p.m., but the entity continued to experience connectivity issues until 9:08 a.m. the next morning. No additional instances of noncompliance occurred after 10:18 p.m. on October 19,2018, because the continued outages were all less than 30 minutes in duration, thereby allowing MISO to perform RTAs. Despite its failure to complete RTAs, the entity’s EMS remained functional, and ithad visibility to all SCADA information.

Fourth, on October 20, 2018, the entity experienced intermittent losses of ICCP connection with MISO due to a loss of power to the Control Center and subsequent frequency deviations, which were caused by a broken pole and subsequent line switching issues. Only one of the ICCP losses was over 30 minutes in duration (i.e., a loss from 10:00 p.m. to 10:56 p.m.), and the entity did not complete its backup internal RTA during that period. Despite its failure to complete RTAs, the entity’s EMS remained functional, and it had visibility to all SCADA information.

Fifth, on October 22, 2018, at 12:19 p.m., MISO informed the entity that it was not receiving data. The entity was still showing that the ICCP connection was online. It switched over the connection at 1:58 p.m. in an effort to fix the problem. At 2:16 p.m., MISO confirmed that it was receiving data. During the ICCP outage, the entity was internally performing RTAs, but the first one was not completed until1:11 p.m. (i.e., more than 30 minutes after the prior RTA). Despite its failure to complete RTAs, the entity’s EMS remained functional, and it had visibility to all SCADA information.

Sixth, on October 26, 2018, the entity lost its ICCP connection with MISO from 2:29 p.m. to 8:02 p.m. The outage occurred when a switch failed, leading to additional technical issues that also rendered the corporate network non‐operational. As a result of the foregoing, the ICCP connection with MISO was down (i.e., the entity lost its primary method of completing RTAs) and the server containing modeling information and the license to run PowerWorld was unavailable. Despite its failure to complete RTAs, the entity’s EMS remained functional, and it had visibility to all SCADA information.

The root causes of this noncompliance were inadequate planning and a deficient backup process for performing RTAs. The entity encountered a variety of issues that prevented it from executing its backup method in a timely manner. For example, during some of the instances described above, personnel were not available to complete the studies within 30 minutes, especially during the instances that occurred after business hours. As another example, the entity relied on access to a licensing server to complete the studies internally, and in some of the instances, the server was inaccessible.

This noncompliance involves the management practice of planning. The entity relied upon MISO to perform its RTAs. Although it planned for an event where MISO would be unable to complete this task, its plan was inadequate, as evidenced by the fact that its backup solution was ineffective on multiple occasions. Entities should engage in thoughtful and deliberate planning when creating and






RFC2019021148 TOP‐001‐4 R13 City of Lansing by its Board of Water and Light NCR00718 10/2/2018 10/26/2018 Self‐Report Completed

implementing processes and procedures to reduce the risk of harm to reliability and resilience of the Bulk Electric System. Here, as part of the development of its backup method of completing RTAs, the entity failed to identify and plan for foreseeable issues (e.g., needing to utilize the backup method after business hours).

This noncompliance involved six separate instances. a. The first instance started on October 2, 2018, at 2:26 a.m. after the entity failed to complete a RTA within a 30‐minute interval and ended on the same date at 5:00 a.m. when the next RTA was

completed.b. The second instance started on October 18, 2018, at 1:11 p.m. after the entity failed to complete a RTA within a 30‐minute interval and ended on the same date at 2:35 p.m. when the next RTA was

completed.c. The third instance started on October 19, 2018, at 8:17 p.m. after the entity failed to complete a RTA within a 30‐minute interval and ended on the same date at 10:18 p.m. when the next RTA was

completed.d. The fourth instance started on October 20, 2018, at 10:31 p.m. after the entity failed to complete a RTA within a 30‐minute interval and ended on the same date at 10:56 p.m. when the next RTA was

completed.e. The fifth instance started on October 22, 2018, at 12:50 p.m. after the entity failed to complete a RTA within a 30‐minute interval and ended on the same date at 1:11 p.m. when the next RTA was

completed.f. The sixth instance started on October 26, 2018, at 3:00 p.m. after the entity failed to complete a RTA within a 30‐minute interval and ended on the same date at 8:02 p.m. when the next RTA was

completed.Risk Assessment This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system. A failure to complete an RTA reduces an entity’s situational awareness

of existing and potential operating conditions, thereby increasing the likelihood of improper action or inaction by system operators leading to instability, uncontrolled separation, or cascading outages. The risk was not minimal in this case due to the number of instances and the demonstrated ineffectiveness of the backup method of performing RTAs. Further, the timing and duration of some of the instances increased the risk. The risk was not serious or substantial in this case because the entity maintained awareness of system conditions through its EMS and visibility of all of its SCADA information during all of the incidents described herein, with the exception of a partial loss of visibility for approximately 30 minutes during the October 2, 2018, event. Additionally, during the instances described herein, the entity maintained communication with MISO regarding system conditions and risks. Lastly, based upon the entity’s size and operations, any potential harm resulting from this violation would most likely have been limited and local in nature. No harm is known to have occurred.


1) set up a stand‐alone laptop in the control room to run Power World software;2) put the PowerWorld License onto the control room laptop;3) conducted testing and training of the control room computer to ensure proper functionality and trained members of the engineering team with the overall responsibility for conducting the studies;4) updated the supporting data for the PowerWorld software at regular planned intervals; and5) trained additional personnel to run the software from control room computer.

ReliabilityFirst has verified the completion of all mitigation activity.






RFC2019021708 PRC‐005‐2(i) R3 Scrubgrass Generating Company, LP NCR00912 10/1/2015 8/15/2019 Compliance Audit Completed Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible, or confirmed noncompliance.)

On June 11, 2019, ReliabilityFirst determined that the entity, as a Generator Owner, was in noncompliance with PRC‐005‐2(i) R3 identified during a Compliance Audit conducted from December 3, 2018 through December 10, 2018. During an audit, ReliabilityFirst reviewed the entity’s compliance with 13 Standards/Requirements and identified one issue. Specifically, ReliabilityFirst determined that the documentation for the 4 month and 18 month battery maintenance activities contained in Table 1‐4 for PRC‐005‐2(i) R3 was missing or incomplete for all samples. Some of the maintenance activities, such as inspecting electrolyte levels and unintentional grounds, were listed in the procedures, but the entity provided no documentation showing that they had been completed. Other required maintenance activities, such as verifying battery terminal connection resistance and verifying battery intercell or unit‐to‐unit connection resistance, were not documented in the procedures or in the maintenance reports.

The root cause of this noncompliance was the entity’s lack of understanding regarding what was required by the Standard. This root cause involves the management practices of workforce management, which includes providing training, education, and awareness to employees, and grid maintenance.

This noncompliance started on October 1, 2015, when the entity was required to comply with PRC‐005‐2(i) R3 and ended on August 15, 2019, when the entity completed its Mitigation Plan. Risk Assessment This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system (BPS) based on the following factors. The risk associated with failing to

perform required maintenance activities within the required timeframe is that the device could fail to operate as expected, which could reduce the reliability of the BPS. The risk is not minimal in this case considering the length of time that the issue persisted and the fact that ReliabilityFirst identified it at audit. The risk is not serious or substantial in this case based on the following factors. First, the batteries at issue are the third backup to the battery chargers and the AC inverter backup, which reduces the likelihood that any potential harm posed by failing to properly maintain and test them would actually occur. Second, the plant is an 85 MW net unit with a capacity factor less than 60% and one 115 kV interconnection, which reduces the potential impact of the issue. No harm is known to have occurred.


1) contracted a third party to review preventative maintenance (PM) and procedures and make recommendations to ensure proper testing and documentation is being completed; and2) updated PMs and procedures per third party recommendations, purchased any additional equipment needed to complete testing. The entity also trained all technicians on new PM and new

equipment used to complete testing.







RFC2018019779 PRC‐005‐1.1b R2 Seneca Generation, LLC NCR11446 2/13/2014 5/23/2018 Self‐Report Completed Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible, or confirmed noncompliance.)

On May 17, 2018, the entity submitted a Self‐Report stating that, as a Generator Owner, it was in noncompliance with PRC‐005‐1.1b R2.

In May, 2018, in preparation for an upcoming audit, entity staff reviewed protection system maintenance records and discovered that the plant’s previous operator had disorganized records, and a few missing records. Specifically, the entity failed to comply with its Protection System Maintenance Program (PSMP), in that it could not provide evidence that it performed all testing in compliance with its internal PSMP. The entity had three noncompliant DC Control Circuits, which were all electromechanical, which were not tested until May 23, 2018. The entity had no evidence to demonstrate prior testing. The entity PSMP required that electromechanical DC Control Circuits be tested every five (5) years. However, the only evidence of testing of the three DC Circuits affected is May 23, 2018, thereby producing a failure to adhere to the entity PSMP, and in turn, PRC‐005‐1.1b R2.

The root cause is that the transition between the prior owner and the current owner resulted in gaps in information and reporting, resulting in this noncompliance. The sale from FirstEnergy to the current owners of the entity and the transfer of the PSMP as well as supporting documentation was disorganized and ineffective, resulting in the three DC Control Circuits being missed.

This noncompliance involves the management practices of grid maintenance. Grid maintenance is involved because the entity failed to complete all necessary maintenance and testing on DC Circuits.

This noncompliance started on February 13, 2014, when the entity registered on the NERC Compliance Registry and ended on May 23, 2018, when the entity completed the requisite maintenance and testing.

Risk Assessment This noncompliance posed a minimal risk and did not pose a serious or substantial risk to the reliability of the bulk power system based on the following factors. The potential risk posed by failing to properly maintain and test protection system components is that they may fail to operate as expected. This risk is minimal in this instance because of the following factors. The entity completed all other maintenance activities on other equipment. For example, the entity kept current with its battery maintenance, lockout relays, current transformers, and potential transformers. Further, the plant installed all new relays in 2012 that are micro‐processor based, and consequentially all of the other NERC‐scoped DC Circuits owned by the entity were tested at the time. ReliabilityFirst also notes that during the time period of noncompliance, no misoperations or equipment failure occurred. No harm is known to have occurred.

ReliabilityFirst considered the entity’s compliance history and determined there were no relevant instances of noncompliance. ReliabilityFirst determined that as this noncompliance was discovered in preparation for an upcoming audit, and the duration exceeded four years, that this minimal risk noncompliance was appropriate for FFT treatment.


1) hired ABM, aProtection System Maintenance and Testing company to complete all maintenance items and bring the plant into full compliance with PRC‐005‐6. The entity reviewed the ABM reportbefore updating the control matrix in Milestone #2; and2) updated its protection system testing program with the corrective devices that are NERC related and removed devices that were not NERC related on its Matrix. The matrix is now up to date and hasbeen updated to include all testing completed by ABM. Additionally, the entity added a work management system where future maintenance items will be tracked.







RFC2019021372 EOP‐011‐1 R2 Southern Indiana Gas & Electric Company d/b/a Vectren Energy Delivery of Indiana, Inc.

NCR00917 1/30/2019 1/30/2019 Self‐Report Complete


On April 12, 2019, the entity submitted a Self‐Report stating that, as a Balancing Authority, it was in noncompliance with EOP‐011‐1 R2.

On January 30, 2019, the entity failed to implement procedures to mitigate Capacity and Energy Emergencies. Specifically, the entity did not have a sufficient process to mitigate emergencies that involve the use of Interruptible Load, curtailable Load and demand response.

During the early morning hours of January 30, 2019, MISO (the Reliability Coordinator (RC)) declared a Maximum Generation Emergency Event that escalated from a Step 1a to Step 2a/2b in approximately four hours. (In order to manage reliability in balance with pricing, MISO uses a process and system that utilizes Max Gen Alert Level Actions to manage resources dedicated to firm Load and maintaining Operating Reserves. These Max Gen Alert Level Actions are contained in the MISO Market Capacity Emergency SO‐P‐EOP‐00‐002 Rev: 6. The process has different “Event Steps” or tiers depending on the severity of the capacity emergency to protect firm Load and Operating Reserves. There are five separate tiers, Max Gen Emergency Event Step 1 Actions being the lowest severity, and Max Gen Emergency Event Step 5 being the most severe. Issuing “Operating Instructions” for firm Load shed is not provided for in the MISO Market Capacity Emergency manual until Max Gen Emergency Event Step 5b.) Additionally, the RC declared Conservative Operations during that same time period. All declarations and requests distributed by the RC were via email and the MISO Communication System (MCS). The Maximum Generation Emergency Event Steps and Conservative Operations declarations were attributed to lower than projected wind generation and higher than expected load due to colder than normal temperatures. However, MISO did not issue any Operating Instructions (as that term is defined by MISO) on January 30, 2019, related to the cold‐weather event to Vectren or any other entity.

When the RC escalated to Maximum Generation Emergency Event Step 2a/2b, Vectren was directed to: (a) Postpone non‐critical maintenance and testing on transmission and generation equipment; (b) Start off‐line designated resources for Available Maximum Emergency conditions; (c) Update and acknowledge Load Modifying Resources; (d) Submit Stage 2 Load Management Measures; and (e) Curtail load based on Load Modifying Resources.

The entity executed every directive above except for the final step – the entity did not request interruptible customers to curtail load per the RC’s scheduling instructions. The entity did not curtail as directed because the entity had an unclear process and plan for Capacity Emergencies and Energy Emergencies within its Balancing Authority Area. Specifically, the entity failed to implement its plan regarding curtailable load, in part because internal procedures regarding Generation and Capacity Emergencies were both too complex and decentralized with procedures spread amongst multiple documents and sections.

The root cause of this noncompliance was an unclear process and plan for Capacity Emergencies and Energy Emergencies. This noncompliance involves the management practices of grid operations and workforce management. Grid operations management is involved because the entity lacked the necessary procedural detail to effectively implement their Generation Emergency Operating Plan. Workforce management is involved because entity employees were not adequately trained on how to follow RC instructions.

This noncompliance started on January 30, 2019, at 9:00 am when the entity failed to effectively implement its Generation Emergency Operating Plan and curtail load. The noncompliance ended on January 30, 2019, at 9:52 am when the scheduled curtailment was terminated by MISO.

Risk Assessment This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system based on the following factors. The risk posed by the failure to implement the Generation Emergency Operating Plan and corresponding failure to curtail load is that the failure slightly exacerbated the load generation imbalances on the wider system. The risk was limited because the amount not curtailed was limited to 23.5 MW. The risk was reduced by the fact that the entity had 125 MW of generation available. Additionally, Maximum Generation Emergency Event 2a/2b, which was issued here, is only the second step of five Maximum Generation Emergency Event Steps (there are five steps to manage an emergency). Maximum Generation Emergency Event 2a/2b includes an increase to Emergency Pricing Tier Two (2) as well as the other responsibilities detailed above; Maximum Generation Emergency Events 3 and 4 are increasingly more aggressive approaches to address firm Load and Operating Reserves; and finally, Max Gen Event Step 5b introduces the use of Operating Instructions for firm Load shed. No harm is known to have occurred.


1) simplified and consolidated procedures into one document and section;2) included additional notifications, such as a phone call or text, to key stakeholders when MISO declares Maximum Generation Emergency Alerts, Warnings, or Events;3) trained personnel on updated procedural changes and new checklists; and4) worked with MISO in an attempt to receive more clear communications.





SERC2018019201 TOP-001-3 R13 Duke Energy Progress, LLC (DEP) NCR01298 11/14/2017 11/27/2018 Self-Report Completed


On February 19, 2018, Duke Energy Progress, LLC (DEP) submitted a Self-Report on behalf of Duke Energy Florida, LLC (DEF), pursuant to the Coordinated Oversight Program, stating that DEF, as a Transmission Operator (TOP), was in noncompliance with TOP-001-4 R13. DEF did not ensure performance of a Real-time Assessment at least once every 30 minutes (Instance 1).

On November 14, 2017, at approximately 6:36 PM, the Real-Time Contingency Analysis (RTCA) application of DEF stopped solving due to a failed case preparation sub-routine. A DEF System Operator (SO) discovered the failure at 9:09 PM, when the operator attempted to run an Online Network Sequence. Between 6:36 PM and 9:09 PM, operators did not receive any alarms indicating that RTCA was not running normally.

On November 14, 2017, at 9:13 PM, DEF notified its Reliability Coordinator (RC) and the Energy Management System (EMS) support of the failure. The RC had not identified any contingencies requiring action during the time of the failure and continued to actively monitor the DEF system.

On November 14, 2017, at 9:18 PM, DEF EMS support restarted the case preparation sub-routine in RTCA. RTCA successfully completed and produced a valid Real-Time Assessment that same day at 9:21 PM.

On February 8, 2019, DEP submitted a Scope Expansion reporting a noncompliance with TOP-001-4 R13 (Instance 2).

On November 27, 2018, DEP’s RTCA failed to include all appropriate contingencies after an EMS database update. Approximately 3,700 contingencies where ignored during the noncompliance, and while RTCA ran during that time, the missing contingencies rendered the results unusable as a valid Real-Time Assessment.

On November 27, 2018, at 11:26 AM, the noncompliance was discovered when a DEP System Operator (SO) noted that the contingencies where displayed in an abnormal color. The SO contacted EMS support believing it was a display issue. EMS support began investigating the abnormal color and found the underlying cause. EMS support restored the system to proper functionality that same day at 2:25 PM.,

Post-mortem analysis showed that the DEP RTCA excluded the contingencies due to a previously unknown data dependency within the EMS database.

These instances of noncompliance began on November 14, 2017, at 6:36 PM, when, in Instance 1, DEF RTCA failed, and ended on November 27, 2018, at 2:25 PM, when the missing contingencies became active again in Instance 2.

The cause of both instances of noncompliance was management oversight; specifically, inadequate internal controls. Management failed to verify that all necessary internal controls were implemented to ensure errors in the RTCA application are readily detectable. Within DEF, the internal controls failed to notify the SO that the RTCA had failed. Within DEP, the internal controls failed to ensure the database update completed successfully and failed to inform the SO of a significant change in the RTCA.

Risk Assessment This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system. The failure to monitor contingencies during Real Time Operations can lead to system overloads, breaker operation, and, in a worst case scenario, loss of load. However, SOs at both DEF and DEP consistently monitored the system during the events and found both instances of noncompliance through their monitoring. Once the SOs had identified each instance of noncompliance, DEF/DEP took appropriate actions to mitigate the risk and bring the systems back to normal operations within 15 minutes. No harm is known to have occurred.

SERC considered DEP’s TOP-001-3 R13 compliance history in determining the disposition track. DEP has one relevant prior noncompliance with TOP-001-3 R13 includes. SERC determined that the relevant compliance history should not serve as a basis for aggravating the penalty. The historic noncompliance was due to a failure to ensure the appropriate save case loaded into EMS while both instant cases of noncompliance happened because of system failures internal to the RTCA application with a lack of appropriate notification. The mitigation plan of the historic noncompliance could not have prevented either instance of the instant noncompliance.

Mitigation To mitigate the noncompliance, DEF/DEP:

1) informed the RC of the RTCA failure;2) resolved issues in RTCA to bring the application back to normal operation;3) deployed patching to address the cause of the RTCA failure;

A-1 Public Non-CIP – Find, Fix, Track, and Report Spreadsheet


O&PFFTSERC Reliability Corporation (SERC)



4) created alarming with high priority in RTCA;5) added warning within database tied to user changes;6) revised contingency automation process to eliminate previously unknown dependency on sequence of steps;7) revised the Operation Technology (OT) EMS database Update Prep procedure in all regions to include steps to check for abnormal situations and correct them before updates are pushed to

Production;8) trained appropriate OT personnel on the revisions made to the EMS database Update Prep procedure listed in Corrective Action #7, and the expectations;9) revised the OT EMS database Update procedure to include steps to:

a) ensure oversight role is in place within OT to oversee EMS database update from start to completion;b) open a telephone bridge line for immediate and continual communications between OT and all functional entities involved during EMS database updates; call should end when SOs Checklists

are completed;c) conduct roll call and perform Pre-Job brief;d) verify that results in Production match final accepted results as documented in Quality Assurance Development;e) check contingencies in an expedient manner to verify they are active in Production;f) put previously enabled server containing previous database into service upon completion of SOs EMS Checklists; andg) retain the procedure by attaching to Footprints ticket

10) trained appropriate OT personnel on the revisions made to the EMS database Update procedure listed in Corrective Action #5, and the expectations.11) revised the SOs EMS Checklist used in DEP and DEF to include steps to document policy statement to perform manual RTAs during EMS database updates and EMS host failovers; join telephone bridge

line for immediate and continual communications with OT and between all functional entities involved during EMS database updates (call should end when System Operations Checklists arecompleted); participate in Pre-Job Brief conducted by OT; perform manual RTAs before and during EMS database updates; and include guidance to check contingencies to verify they are active inProduction immediately after the failover;

12) trained SOs in DEP and DEF on the revisions to the SOs EMS Checklist listed in Corrective Action #11, and the expectations.13) revised the procedure, SORMC-TOP-320 Real-Time Monitoring and Assessment and Monitoring Applications, to include: expedite communication among Carolinas functional entities of any EMS

priority alarm indicating a problem with Real-Time Network or RTCA; and when acknowledging an EMS priority alarm, verify all functional entities using the consolidated EMS are aware; and14) trained DEP SOs on the revisions to the procedure listed in Corrective Action #13.

SERC Reliability Corporation (SERC) FFT O&P






RFC2019021406 COM‐002‐4 R3 Coldwater Board of Public Utilities NCR08012 7/1/2016 5/7/2019 Compliance Audit Completed Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible, or confirmed noncompliance.)

On April 22, 2019, ReliabilityFirst determined that the entity, as a Distribution Provider, was in noncompliance with COM‐002‐4 R3. The noncompliance was identified during a Compliance Audit conducted on April 11, 2019. Specifically, ReliabilityFirst concluded that entity operating personnel had not completed the initial training on receiving oral two‐party, person‐to‐person Operating Instructions prior to receiving Operating Instructions.

The root cause of this noncompliance was a failure to realize that COM‐002‐4 R3 was applicable to the entity. This noncompliance involves the management practices of grid operations, workforce management, and reliability quality management. A basic tenet of grid operations management is providing sufficient operator training in order to support reliable operation. Workforce management involves ensuring that employees have baseline competencies to perform their compliance responsibilities, and this can be achieved, in part, through effective training. Reliability quality management should include: (a) efforts to identify and fully understand Reliability Standards and Requirements; and (b) evaluations for quality assurance of the organization’s Bulk Electric System reliability and resilience activities.

This noncompliance started on July 1, 2016, when the entity was required to comply with COM‐002‐4 R3 and ended on May 7, 2019, when entity operating personnel completed required training. Risk Assessment This noncompliance posed a minimal risk and did not pose a serious or substantial risk to the reliability of the Bulk Power System (BPS) based on the following factors. Failing to train operators on

predefined communication protocols increased the likelihood of a miscommunication that could have led to action or inaction harmful to the reliability of the BPS. The risk in this case was minimized based upon the following facts. The entity’s all‐time peak system load was 83.7 MW, and the entity’s interconnections are limited to a common 138kV bus owned and operated by ITC Interconnection, LLC (ITC). ITC is responsible for switching on the 138kV system. Based upon the foregoing, any miscommunication regarding an Operating Instruction issued to the entity would have had a minimal impact on the overall reliability and resilience of the BPS. Further, the entity receives only a couple of Operating Instructions per year on average, thereby reducing the number of opportunities for a miscommunication that would impact the BPS. Lastly, the primary operator at the entity has over thirty years of industry experience and has utilized three‐part communication over the course of his career, thereby reducing the likelihood of a miscommunication. No harm is known to have occurred.


1) trained staff on oral two‐party, person‐to‐person Operating Instructions; and2) implemented a more rigorous orientation process for new staff that would be able to receive an oral Operating Instruction.


A-1 Public Non-CIP - Find, Fix, Track, and Report Consolidated Spreadsheet





RFC2019021337 PRC‐005‐6 R3 Forked River Power LLC NCR00376 4/1/2017 4/1/2019 Self‐Report Completed Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible, or confirmed noncompliance.)

On April 5, 2019, the entity submitted a Self‐Report stating that, as a Generator Owner, it was in noncompliance with PRC‐005‐6 R3. (On May 23, 2019, ReliabilityFirst discovered the same PRC‐005‐6 R3 during Compliance Audit conducted on May 20, 2019, because Self‐Report’s submitted during the 90‐day audit window are considered to be audit findings.)

The entity failed to complete 18‐month tests of battery terminal connection resistance and battery intercell or unit to unit connection resistance as required by PRC‐005‐6 Table 1‐4(a) from April 1, 2017, through April 19, 2019, for its two applicable Vented Lead Acid batteries. The entity did complete other 18 month tests including verifying battery continuity, verifying float voltage, and inspecting the condition of all cells where visible and additional battery tests including the 4‐month battery tests required under PRC‐005‐6. The reason the 18‐month tests of battery terminal connection resistance and battery intercell or unit to unit connection resistance tests were missed is that the entity failed to enter those tests into its work order system.

The root cause of this noncompliance was inadequate internal verification controls to assure that the 18‐month tests of battery terminal connection resistance and battery intercell or unit to unit connection resistance tests were entered into the entity’s work order system.

This noncompliance involves the management practices of verification and grid maintenance. Verification management is involved because the entity failed to assure that all PRC‐005‐6 compliance activities were included in related work orders. Grid maintenance management is involved because the noncompliance occurred in the entity’s Protection System Maintenance program.

This noncompliance started on April 1, 2017, when the entity was required to comply with PRC‐005‐6 R3 and ended on April 1, 2019, when the entity completed the 18‐month tests of battery terminal connection resistance and battery intercell or unit to unit connection resistance tests.

Risk Assessment This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system (BPS) based on the following factors. The risk associated with failing to perform required maintenance activities within the required timeframe is that the device could fail to operate as expected, which could reduce the reliability of the BPS. The risk here is not minimal because the noncompliance had a duration of two (2) years. The risk here is not serious because the batteries were otherwise maintained and checked on a regular basis. The entity performed 4‐month testing, and some activities required for 18‐month testing including verifying battery continuity, verifying float voltage, and inspecting the condition of all cells where visible. Additionally, the batteries involved were relatively new, installed in 2015, and 2016, respectively which decreases the likelihood of substandard performance. ReliabilityFirst also notes that the entity contributes 86 MW of generation to the system. No harm is known to have occurred.


1) verified “Battery terminal connection resistance” and “Battery intercell or unit‐to‐unit connection resistance”; and2) aligned MP2 system with tracking and verifying that all 4‐month, 18‐month, and 6‐year PRC‐005‐6 tests under Table 1‐4(a) have been completed. For each test, a work order will be provided to

relevant employees indicating that the test must be completed. The entity also provided appropriate personnel with training on NERC responsibilities including PRC‐005‐6 maintenance and testing byconducting annual PRC‐005 training for all plant personnel who are involved in Protection System maintenance and testing. This will assure that plant personnel are apprised of the most currentstandards versions as well as any announced changes to the standard and implementation dates, as well as to allow the plant personnel to ask clarifying questions related to performance ofmaintenance obligations, documentation and compliance.






SERC2016016372 PRC-005-1 R2 Entergy NCR01234 11/19/2007 12/01/2017 Self-Report Completed


On October 18, 2016, Entergy submitted a Self-Report stating that, as a Transmission Owner, it was in noncompliance with PRC-005-1 R2. Entergy failed to maintain a Protection System device within its required interval for the equipment at the 115 kV Dumas substation. On February 3, 2017, Entergy submitted an Expansion of Scope to the October 18, 2016 Self-Report stating that there was additional equipment at the 115 kV Dumas substation that had not received timely maintenance.

Regarding Instance 1, in 2008, Entergy suspended maintenance for a 115kV bus differential panel at the Dumas station in its Substation Work Management System (SWMS). The panel contained 10 Protection System devices: one microprocessor relay; four auxiliary trip relays; one lockout relay; four current transformers; and one potential transformer. Entergy issued no future work orders and was unable to provide the cause for the suspension of maintenance. As a result of the suspended maintenance, on December 31, 2013, Entergy missed the deadline to complete maintenance for the microprocessor relay; three auxiliary relays; and the lockout relay.

On December 2, 2015, Entergy completed a review of the Dumas station. The contractor performing the survey noted that the subject panel was located in the substation but was not included in the inventory. On October 24, 2016, Entergy completed required maintenance on the 115 kV Dumas station.

On November 28, 2017, Entergy submitted an additional Self-Report of PRC-005-1 R2 relating to the Adams Creek 230kV switchyard, which was designated NERC ID SERC2017018726. SERC determined that the noncompliance in SERC2017018726 was related to the instance in the original October 18, 2016 Self-Report. Therefore, SERC dismissed and consolidated SERC2017018726 with SERC2016016372.

In Instance 2, on February 10, 2010, Entergy suspended maintenance of the Adams Creek panel. The panel provides protection on the Adams Creek to Calpine 230 kV line. The panel included solid-state relays for protection of a short bus section and transmission line stub, which Entergy intentionally kept electrically isolated. Relay types included a distance relay; a current differential (86) relay; two associated auxiliary relays; and the lockout relays for the protection of the stub that remained energized. Entergy assigned a four-year maintenance interval for the panel devices. Entergy issued no future work orders and was unable to provide the cause for the suspension of maintenance. As a result of the suspended maintenance, on December 31, 2010, Entergy missed the first deadline to complete maintenance. On December 31, 2014, Entergy missed a second deadline to complete maintenance.

On July 7, 2017, an engineer was performing a data review of protective relay panels as part of an ongoing SWMS enhancement project and they discovered that regular maintenance was suspended in SWMS. Entergy performed the required maintenance on this date.

On December 28, 2017, Entergy submitted an Expansion of Scope to the November 28, 2017 Self-Report stating that Little Rock South substation had a missing interval of maintenance.

In Instance 3, on April 2, 2009, Entergy completed maintenance on the primary line protection relay at the Little Rock South station. The equipment has a four year requirement for maintenance. Entergy incorrectly scheduled the next maintenance for 2012 due to a mistake in the work order, but it was not performed since the required interval was not yet reached. On April 2, 2013, Entergy missed its maintenance deadline. On December 1, 2017, Entergy completed testing on the Little Rock South station.

On February 5, 2019, Entergy submitted another Self-Report of PRC-005-1 R2 relating to the Addis to Plaquemine 115kV line, which was designated NERC ID SERC2019021037. SERC determined that the noncompliance in SERC2019021037 involved similar maintenance failures to the instance in the original October 18, 2016 Self-Report. Therefore, SERC dismissed and consolidated SERC2019021037 with SERC2016016372.

Regarding Instance 4, in 2007, Entergy suspended maintenance on the tone set relay on the Addis to Plaquemine 115 kV line. The maintenance interval for the relay was set to 12 months. Entergy issued no future work orders and was unable to provide the cause for the suspension of maintenance. On November 19, 2007, Entergy missed the first deadline to complete maintenance. Entergy also missed the deadlines in 2008, 2009, 2010, and 2011. On June 23, 2012, Entergy instituted a revised Protection System Maintenance Program as part of the implementation plan for an upcoming NERC Standard. Entergy changed the maintenance interval from 12 months to 12 years. On January 10, 2019, Entergy discovered the missed maintenance as a part of an evaluation of maintenance records.

The duration includes multiple isolated instances and was not a continuous state of noncompliance. These instances of noncompliance started on November 19, 2007, when Entergy missed maintenance on the Addis to Plaquemine 115 kV tone set relay, and ended on December 1, 2017, when Entergy performed required maintenance on the Little Rock South station.

The cause for these four instances was management oversight. Management failed to verify implementation of adequate internal controls to ensure the closure of open work orders and the suspension of maintenance activities. For all instances, management failed to implement controls to identify the missed maintenance activities.



Risk Assessment These instances of noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system. Specifically, Entergy’s failure to perform maintenance on its Protection System devices could have caused improper operation of a relay. However, because of the setup of each system, it was unlikely that the operation would have caused a transmission line outage. For instance, the equipment at issue either had a backup system, which had been properly maintained or was unable to cause an outage to the transmission system. No harm is known to have occurred.

SERC considered Entergy’s PRC-005-1 R2 compliance history in determining the disposition track. Entergy has four relevant prior instances of noncompliance with PRC-005-1 R2. SERC determined that Entergy’s PRC-005-1 R2 compliance history should not serve as a basis for applying a penalty. The historical instances of noncompliance were primarily caused by Entergy misunderstanding its obligations, which led to missed deadlines through inaction. In contrast, the instant noncompliance was a direct result of Entergy actively switching maintenance tasks off in its tracking software by mistake. The historical Mitigation Plans focused on addressing issues due to inaction and therefore could not have prevented the instant noncompliance from occurring.

Mitigation To mitigate this noncompliance, Entergy:

1) unsuspended maintenance in SWMS and performed maintenance at the Dumas station;2) performed Panel DC Operational Test WR# 2105463 and Relay Calibration WR# 2105461 tasks at Adams Creek;3) tested Relays at Little Rock South;4) re-enabled the SWMS maintenance for the Addis tone set relay and the Plaquemine tone set relays;5) performed the channel integrity maintenance at Addis;6) issued Configuration Management (AM-CM-FAC-001) and Energization Notice (AM-CM-AD-001) procedures;7) updated AM-PC-AD-001, sections 1.2.1 and 4.2.1, to require approval when PRC-005 components are removed from maintenance;8) implemented user access level controls on the Override Reliability-Centered Maintenance (RCM) Rank option and limited access to Subject Matter Experts;9) updated AM-CM-AD-027 to provide guidance on selecting and configuring communication system schemes in SWMS;10) developed an exception report for SWMS work requests that flag suspect work requests;11) created a new monthly automated report to query SWMS for SERC applicable components without maintenance enabled;12) updated the WebTap training module - PRC-005 Substation Supervisor and Planner Scheduler Training - to align with the changes to AM-CM-AD-027 and reissued the WebTap training;13) implemented a change in SWMS to: expand the scope of RCM update script to look for changes in in-service date, application, availability, and a default of "RCM Low" for each new asset that is

created. This is currently set to "No Maintenance";14) developed requirements and guidance specific to the expectations of closing current open work to previously completed work requests;15) modified the Work Management Procedure to include guidance specific to the expectations of closing current open work to previous completed work requests based on information provided by AM

Transmission Protection; and16) created and issued training on work management procedure modifications.






RFC2019021379 VAR‐002‐4.1 R2 AEP Generation Resources Inc. NCR11401 2/15/2018 9/18/2018 Self‐Report Completed Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible, or confirmed noncompliance.)

On April 16, 2019, the entity submitted a Self‐Report stating that, as a Generator Operator, it was in noncompliance with VAR‐002‐4.1 R2.

On September 18, 2018, the entity performed an annual internal document review and discovered that the AEP Generation Dispatch desk procedure did not match the AEP Transmission Operator (TOP) VAR‐002 Generation instruction letter for communicating deviations from the voltage or Reactive Power schedules provided by the TOP. The desk procedure required the dispatcher to call only AEP (TOP) for a voltage schedule deviation. The instruction letter required the dispatcher to call AEP (TOP) and PJM (TOP) when making notifications about a voltage schedule deviation.

As a result of this discovery, the entity performed an extent of condition and discovered that from February 15, 2018 to September 18, 2018, there were 40 instances of the AEP Generation Dispatcher not communicating voltage schedule deviations to PJM as is required.

Prior to revising the AEP Generation Dispatch desk procedure on September 18, 2018, the desk procedure stated that the dispatcher should call the AEP TOP when deviating from the voltage schedule. However, the AEP TOP instruction letter effective February 6, 2018 for the Cardinal facility and February 8, 2018, for the Conesville facility stated that dispatch must notify AEP TOP and PJM TOP. Because the dispatchers followed the desk procedure (which was inaccurate), the entity was notifying only one of the two TOPs it was required to notify (it was notifying AEP, but not PJM) in the case of a deviation from the voltage schedule.

The root cause of this noncompliance was a failure to integrate a new TOP notification requirement into the existing entity procedure for dispatchers resulting in a failure to notify PJM of voltage schedule deviation on 40 separate instances.

This noncompliance involves the management practices of grid operations and workforce management. Grid operations management is involved because by failing to communicate voltage deviations to PJM, the entity was adversely impacting PJM’s ability to operate with comprehensive situational awareness. Workforce management is involved because the entity did not train its dispatchers on the new process requirements created by AEP TOP’s instruction letter.

This noncompliance started on February 15, 2018, when the entity was required to notify both TOPs of a voltage deviation, and failed to notify PJM. The noncompliance ended on September 18, 2018, when the entity updated its dispatch notification procedure.

Risk Assessment This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system (BPS) based on the following factors. The risk posed by this noncompliance is that the failure to notify the TOP of voltage deviations places the TOP in an informational disadvantage and adversely impacts situational awareness, potentially resulting in harm to the BPS. The risk is not minimal because of the 40 separate instances, some with durations lasting multiple hours. The risk is partially reduced because the AEP TOP was notified in all instances. (The AEP TOP is required to communicate with PJM for any reliability issues that may arise.) Further reducing the risk, the entity’s system voltage telemetry was readily available to PJM at all times. No harm is known to have occurred.


1) revised the Generation Dispatch desk procedure to align to the Transmission Operator (TOP) instruction for notification of voltage deviations;2) performed an extent of condition to investigate not notifying TOP as instructed. No gaps in communication were identified other than what was identified from this incident;3) changed the management process to address any changes to the awareness and readiness process of going live with any changes to VAR‐002 R2 operating procedures; and4) revised change management process for alignment to the current organization to update the awareness and readiness communication between the compliance organization and the business units

accountable for compliance roles to the NERC requirement. The flow of communication to provide feedback from business units to the NERC compliance organization providing readiness of updatedcompliance documents has been updated to have a primary central compliance group responsible for ensuring compliance readiness prior to going live with any changes to operation processes.

AEP as a GO/GOP/TOP has undergone a joint venture in continuous improvement of communication of VAR‐002 measures to enhance the real‐time system voltage data and statuses between Generation and Transmission for improved system performance and reliability of the Bulk Electric System from the latter part of 2017 through the end of 2018. Enhancements include more visibility and situational awareness into the real‐time system generator voltage and schedule to improve overall regional system voltages.







RFC2019021378 VAR‐002‐4.1 R2 American Electric Power Service Corporation as agent for etc. NCR00682 9/11/2018 9/19/2018 Self‐Report Completed


On April 15, 2019, the entity submitted a Self‐Report stating that, as a Generator Operator, it was in noncompliance with VAR‐002‐4.1 R2. On September 25, 2018, the entity discovered, as the result of an internal generation dispatch management review, that the generation dispatcher had not verbally communicated voltage deviations to the Transmission Operator (TOP) in two instances. In both instances, voltage deviated for more than 30 continuous minutes and the entity failed to adhere to the TOP voltage schedule letter guidance and provide verbal notification to both the TOP and PJM. The first instance occurred at Generator Amos Unit 2 on September 11, 2018. Voltage deviated from the scheduled tolerance at 05:53 AM, returned within the voltage schedule tolerance at 06:26 AM. The dispatcher missed a system voltage deviation alarm that alerts the dispatcher to make verbal notifications to the entity TOP and PJM. The plant operator also received a system voltage deviation alarm and likewise failed to make the required notification to generation dispatch. The second instance occurred at Mitchell Unit #1 on September 18‐19, 2018, when the entity failed to report to the entity TOP an exceedance deviation from the assigned voltage schedule. The dispatcher notified PJM as instructed but did not notify the entity TOP of the system voltage deviation. Voltage deviated from scheduled tolerance at 18:02 on September 18, 2018 and returned within voltage schedule tolerance at 15:12 on September 19, 2018. The root cause of this noncompliance was inadequate training as entity personnel were not effectively trained on the purpose of the system voltage deviation alarm and on the actions that need to be taken to respond to that alarm. That ineffective training resulted in the failure to notify the TOPs of voltage deviations from the assigned voltage schedule in excess of 30 continuous minutes. This noncompliance involves the management practices of grid operations and workforce management. Grid operations management is involved because by failing to verbally communicate voltage deviations to the TOPs, the entity was adversely impacting the TOP’s ability to operate with comprehensive situational awareness. Workforce management is involved because though the dispatchers were trained on verbal notifications, they were not trained adequately to assure that verbal notifications were executed during voltage deviations. This noncompliance started on September 11, 2018, when the first instance of the failure to notify the entity TOP of a voltage deviation occurred and ended on September 19, 2018, when the voltage was no longer deviating after the second failure to notify the TOP verbally.

Risk Assessment

This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system (BPS) based on the following factors. The risk posed by this noncompliance is that the failure to notify the TOP of voltage deviations places the TOP in an informational disadvantage and adversely impacts situational awareness, potentially resulting in harm to the BPS. The risk is partially reduced because both TOPs were utilizing Supervisory Control and Data Acquisition (SCADA), Inter‐Control Center Communications Protocol, and other alarms for monitoring and tracking generator system voltage, in place of verbal notification. (The additional verbal notification requirements were put in place in 2017 as the result of a recommendations following a 2015 regional audit. These serve as an additional layer of monitoring, beyond SCADA and alarms, rather than a single method of monitoring and communication.) Further reducing the risk is the short three minute duration of the first instance and the fact that the entity self‐identified both instances. The entity also self‐reported additional related instances which are being resolved under a separate NERC Violation ID. All of the instances, taken together, pose a moderate risk to the BPS. No harm is known to have occurred. ReliabilityFirst considered the entity’s compliance history and determined there were no relevant instances of noncompliance.

Mitigation

To mitigate this noncompliance, the entity: 1) reviewed its processes for gaps and awareness of desired processes to Generation Dispatchers. As part of the review, the roles and responsibilities of notification to entity Transmission Operator and

PJM for Generator system voltage deviations was discussed to affirm operators understood their role in the communication process. Generation NERC compliance group reviewed and verified the desk procedure was meeting Transmission Operator expectations for voltage deviation notification;

2) helped achieve preventative recurrence by periodic or as needed reviews of source documents and operator procedures for alignment; and 3) revised rebooting to maintain awareness of generator voltage monitoring during entity Generation Dispatch shift change. The restarting and rebooting of the non‐CIP computer that normally occurs

at the beginning of each shift every day has been revised to reboot only on Monday and Thursday on day shift to ensure availability of support staff if issues are encountered. In 2017, the entity developed a revised communication process to enhance the awareness of monitoring generator system voltage, and communication of voltage deviations. This revised process was a result of a 2015 regional audit recommendation to improve voltage schedule communication. The improved communication workflows for voltage deviations provide an additional layer of monitoring and communication, beyond the existing SCADA/alarms, utilizing Generation Dispatch for increased awareness. The entity as a GO/GOP/TOP has undergone a joint venture in continuous improvement of communication of VAR‐002 measures to enhance the real‐time system voltage data and statuses between Generation and Transmission for improved system performance and reliability of the Bulk






RFC2019021378 VAR‐002‐4.1 R2 American Electric Power Service Corporation as agent for etc. NCR00682 9/11/2018 9/19/2018 Self‐Report Completed

Electric System from the latter part of 2017 through the end of 2018. Enhancements include more visibility and situational awareness into the real‐time system generator voltage and schedule to improve overall regional system voltages.






RFC2019021566 PRC‐005‐1 R2 The Dayton Power and Light Company NCR00748 6/18/2007 4/30/2019 Self‐Report Completed Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible, or confirmed noncompliance.)

On May 14, 2019, the entity submitted a Self‐Report stating that, as a Transmission Owner, it was in noncompliance with PRC‐005‐1 R2. On April 29, 2019, during a review and update of the model used for short circuit analysis and protection engineering, the entity discovered that three 345 kV bus differential relays were missing from its asset management database, and consequently excluded from the entity’s routine maintenance schedule, which required these relays to be tested every 4 years. Upon investigation, the entity determined that these three relays were missed during its initial data migration from a paper‐based system to the asset management database in 2009/2010. That data migration included the transcription of information for approximately 8,000 total relays and over 11,000 associated paper test cards. The root cause of this noncompliance was an oversight during the initial data migration and subsequent failure to identify the oversight during the data validation process. This root cause involves the management practices of verification and validation. This noncompliance started on June 18, 2007, when the entity was required to comply with PRC‐005‐1 R2 and ended on April 30, 2019, when the entity completed the testing and maintenance activities on these three relays.

Risk Assessment

This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system (BPS) based on the following factors. The risk posed by failing to perform required maintenance activities within the required timeframe is that the devices could fail to operate as expected, which could reduce the reliability of the BPS. The risk is not minimal in this case because of the length of time that the issue persisted. The risk is not serious or substantial in this case based on the following factors. First, these three relays represent only .13% of the entity’s total inventory of PRC‐005 components and only .3% of its total inventory of PRC‐005 relays, which indicates that this was an isolated issue and not a systemic fall down. Second, overlap protection from the bank high side and connected breakers are set to operate before other facilities would have operated to isolate the bus at issue. ReliabilityFirst also notes that throughout the time period that this issue persisted: (a) no events occurred that called upon the bus differential to operate; and, (b) the testing and maintenance of these three relays identified no operational issues. No harm is known to have occurred. ReliabilityFirst considered the entity’s compliance history and determined there were no relevant instances of noncompliance.

Mitigation

To mitigate this noncompliance, the entity: 1) performed the requisite maintenance and testing, and added the relays to its asset management database to ensure future maintenance and testing is done on time; 2) identified Bulk Electric System substations where potential devices are located and field verification is required; 3) developed work packets for first set of field verifications. The entity also performed field inspection of substation relays for the first set of substations; 4) developed work packets for second set of field verifications. The entity also performed field inspection of substation relays for the second set of substations; and 5) summarized results of field inspection program, submit any pertinent information from the field inspection verification program to ReliabilityFirst.






RFC2019020967 EOP‐005‐2 R14 Seneca Generation, LLC NCR11446 2/13/2014 11/13/2018 Compliance Audit Completed Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible, or confirmed noncompliance.)

On January 10, 2019, ReliabilityFirst determined that the entity, as a Generator Operator, was in noncompliance with EOP‐005‐2 R14 identified during a Compliance Audit conducted from September 14, 2018 through December 4, 2018.

The ReliabilityFirst audit team determined that the entity did not have a specific process for starting its Blackstart Resources during an event (Blackstart Event Procedure) for the two entity Black Start units which are 229 MVA units. Rather, the entity had a Blackstart Test Procedure, and indicated that the steps to be taken in case of an event were included in the Blackstart Test Procedure, but not specifically noted in the Blackstart Test Procedure.

The root cause of this noncompliance was inadequate internal verification controls and lack of knowledge of EOP‐005‐2 R14 resulting in a failure to implement a distinct internal Blackstart Event Procedure.

This noncompliance involves the management practices of grid operations and verification. Grid operations management is involved because the entity failed to effectively define and implement operating procedures for the Blackstart units. Verification management is involved because the entity did not ensure that it had an actionable procedure for a Blackstart during a grid event.

The noncompliance began on February 13, 2014, the date the entity was required to comply with EOP‐005‐2 R14. The noncompliance ended on November 13, 2018, the date the entity implemented a new, distinct, Blackstart Event Procedure.

Risk Assessment This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system based on the following factors. The risk of not having procedures in place for starting a Blackstart Resource is that if a blackout or partial blackout occurred, the entity may not have been able to start the Blackstart Resources in a timely manner if required to do so. The risk is not minimal because of the five year duration, and inability to identify the noncompliance via internal controls. The risk here is not serious because while the entity did not have a Blackstart Event Procedure, it did have a Blackstart Test Procedure which included the steps to be taken in case of an event (but were not outlined as event specific steps). The entity’s Blackstart Test Procedure had been in place since 2011, and employees had been trained on the procedure in 2015, 2017, and 2018 which meant employees were likely familiar with the steps involved in the case of an event. ReliabilityFirst also notes that during the period of the noncompliance, the entity was not directed to use its Blackstart Resources at any time. No harm is known to have occurred.

The entity has relevant compliance history. However, ReliabilityFirst determined that the entity's compliance history does not warrant an elevated risk and should not serve as a basis for an aggravating penalty because the prior noncompliance is distinguishable as it involved different conduct (i.e. failure to train) and a different root cause.


1) created a Blackstart Event Training work order to ensure plant operating personnel receive Blackstart Event Training which includes the documented procedures for starting each Blackstart Resourceand energizing a bus. The operating personnel will also receive training on the PJM restoration plan including coordination with the Transmission Operator;

2) created a new Blackstart Start up Procedure for both Units 1 and 2 which includes documented procedures for starting each Blackstart Resource and energizing a bus; and3) trained all Operating Personnel on the Blackstart Start up Procedure for both Units 1 and 2 and the Restoration Plan.






SERC2019021062 MOD-027-1 R5 Tennessee Valley Authority (TVA) NCR01151 02/03/2016 01/03/2019 Self-Report Completed


On February 15, 2019, TVA submitted a Self-Report stating that, as a Transmission Planner (TP), it was in noncompliance with MOD-027-1 R5. TVA did not provide a written response to the Generator Owner (GO) within 90 days of the turbine/governor and load control or active power/frequency control system model verification that the model was usable or not useable.

On November 19, 2018, TVA determined that it failed to provide a written response to the GO, within 90 days of the data submission by the GO, for units in November 2015, April 2016, June 2016, May 2017, and August 2017. Additionally, in some cases, TVA did not determine if the model was or was not useable within 90 days of model data submission. Ultimately, TVA determined that all of these models were useable. The model data for the five periods involved 36 units, which represented 4,498 MVA of TVA’s 33,526 MVA of generation (13.4%).

As part of its mitigating activities, TVA completed an extent-of-condition review and identified five additional instances that included nine additional units. In total, the number of days that elapsed from the date of submission to the date TVA responded ranged from 94 days to 1,155 days.

This noncompliance started on February 3, 2016, when TVA failed to provide the written response to the GO, and ended on January 3, 2019, when TVA submitted the last overdue response to the GO.

The cause of the noncompliance was management oversight (A4B1). Management failed to verify that all necessary internal controls, e.g., tracking software and an assignment of a secondary delegate responsible for tracking model submissions, were implemented to ensure timely responses to the GO. The TVA TP used an email inbox for the receipt of model data from GOs and assigned a single subject-matter-expert (SME) to monitor the mailbox. The SME failed to follow-up with written notification after discussions with the GO regarding models. The SME also went on leave after receiving the notification and failed to test the models and respond to the GO after returning from leave. Additionally, when the SME received a batch of submissions, the SME used the date of the last submission of the batch as the date to respond by, instead of addressing all units from the submission in the response.

Risk Assessment This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system. Specifically, TVA’s failure to verify model usability, and notify the GO within 90 days that the models submitted are usable or not usable could have led to TVA using incorrect models that could have resulted in erroneous assessments or an incorrect corrective action plan. The noncompliance included 45 units and the response to the GO data submission ranged from 94 days to 1155 days. TVA was able to include the models that were slightly late in the applicable SERC data submittal, however, for those that had longer delays, TVA was unable to include those models in the applicable SERC data submittal. While TVA ultimately determined the model information was useable, if TVA determined the models were unusable, the GO would have been required to re-engage the contractor that had originally provided the verified models. This could have resulted in additional delays in obtaining a usable verified model for these units. In addition, TVA only identified the instant noncompliance when reviewing data for a previous GO MOD noncompliance. No harm is known to have occurred.

SERC considered TVA’s compliance history and determined that there were no relevant instances of noncompliance. Mitigation To mitigate this noncompliance, TVA:

1) sent replies to all outstanding model validations as required by the Standard;2) assigned a delegate, in addition to the SME, to assist in tracking the model submissions;3) implemented a new action tracking software. As models are received by the TP, they will be entered into the software manually, which will then track the appropriate due dates for the replies. E-mailsfrom the software are automatically sent to the SME each week, along with an additional e-mail as the due date for the reply approaches to both the SME and the assigned delegate;4) conducted an extent-of-condition (EOC) analysis to verify the results of the GO EOC analysis to be certain that all received models reports have been tested and the appropriate replies to the GO havebeen performed; and5) created a new Technical Procedure to document the process that the TP used to process and reply to incoming MOD-027 model submissions with enough details such that a new employee will be ableto track, process, and reply to a new MOD-027 model supplied by a GO. All impacted employees were notified and trained on the contents of the procedure.





TRE2017018733 PRC-019-2 R1 CPS Energy (CPS Energy1) NCR04037 07/01/2016 09/13/2017 Self-Report Completed


On November 29, 2017, CPS Energy1 submitted a Self-Report stating that, as a Generator Owner (GO), it was in noncompliance with PRC-019-2 R1. Specifically, CPS Energy1 failed to properly coordinate the voltage regulating system controls with the applicable equipment capabilities and settings of the applicable Protection System devices and functions for 40% of its applicable Facilities by July 1, 2016, and for 60% of its applicable Facilities by July 1, 2017, as required.

The Implementation Plan for PRC-019-2 required that 40% of an entity’s applicable Facilities be verified to meet the requirements of R1 by July 1, 2016, and that 60% of an entity’s applicable Facilities be verified by July 1, 2017. CPS Energy1 was late in meeting these requirements. The Implementation Plan also required that 80% of an entity’s applicable Facilities be verified by July 1, 2018.

On March 1, 2016, CPS Energy1’s Compliance department became aware of the PRC-019-2 R1 requirement. On July 24, 2017, which is after the July 1, 2016 implementation plan milestone, studies were completed and settings were coordinated on 40% of CPS Energy1’s applicable Facilities. On September 13, 2017, which is after the July 1, 2017 implementation plan milestone to coordinate settings on 60% of applicable Facilities and before the July 1, 2018 implementation plan milestone to coordinate settings on 80% of applicable Facilities, studies were completed and settings were coordinated on 60% of CPS Energy1’s applicable Facilities, ending the noncompliance.

The root cause of the noncompliance was twofold. First, CPS Energy1’s Compliance department did not fully monitor the development and implementation of PRC-019-2. As a result, CPS Energy1 did not become completely aware of the scope and breadth of PRC-019-2 R1 and the efforts needed to comply until the spring of 2016. Second, CPS Energy1 underestimated the amount of time and resources needed to perform the studies required by PRC-019-2 R1. CPS Energy1 attempted to perform the studies itself, but soon determined it lacked the necessary resources and would need to hire a contractor. CPS Energy1 was not initially aware of the cost and timeframe involved in hiring a contractor to gather and analyze data and create the coordination and evidence required for compliance with PRC-019-2 R1.

This noncompliance started on July 1, 2016, when PRC-019-2 became mandatory and enforceable, and ended on September 13, 2017, when CPS Energy1 completed the coordination of its voltage regulating system controls with the applicable equipment capabilities and settings of the applicable Protection System devices and functions for 60% of its applicable Facilities.

Risk Assessment This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system.

Specifically, the failure to coordinate the voltage regulating system controls with the applicable equipment capabilities and settings of the applicable Protection System devices and functions could lead to a generator failing to trip before the equipment sustained damage, or tripping for a system event that should not have caused the generator to trip at all. CPS Energy1’s applicable Facilities fall into three groups with varying degrees of potential impact on the grid. The first group of applicable Facilities is connected to the 138 kV Transmission system and is located in San Antonio, a large load center. The second group of generation is connected to the 345 kV Transmission system, which provides this group the ability to serve load beyond its local area, and is located in the same large load center. Transmission planners and operators expect both groups of applicable Facilities to be available to serve load and provide voltage support in that load center. The final group of applicable facilities is connected to the 345 kV Transmission system, which provides this group the ability to serve load beyond its local area. This group is located between two large load centers, San Antonio and Austin, and Transmission planners and operators expect this group to be available to serve load in both the large load centers. Of the applicable Facilities that were not timely compliant with the Reliability Standard, CPS Energy1 was an average of 263 days late achieving compliance, bringing the applicable Facilities into compliance 23 days through 388 days after the applicable implementation plan deadline. Thirteen applicable Facilities of 24 total (as of (November 29, 2017)) were not timely compliant and needed settings changes to achieve compliance. CPS Energy1 made the settings changes an average of 95 days after the date of the study. The 13 applicable Facilities have a combined real power rating of 1,315 MW, with an average rating of 101 MW and the largest applicable Facility rated at 323 MW.

CPS Energy1 did not experience any Misoperations or trips related to the lack of coordination, and when the study results indicated that settings needed to be changed, CPS Energy1 determined that the old settings had not impacted equipment performance.

No harm is known to have occurred.

Texas RE considered CPS Energy1’s compliance history and determined there were no relevant instances of noncompliance.



Mitigation Activity To mitigate this noncompliance, CPS Energy1: 1) completed studies and coordinated settings on 60% of its applicable Facilities; 2) began holding recurring meetings to track Reliability Standard development and GO compliance status; 3) reorganized the Compliance department to more effectively utilize internal resources; 4) hired a Power Generation Regulatory Compliance Manager for increased oversight of compliance with PRC-019-2 Requirements; and 5) implemented a compliance tracking tool that assigns compliance responsibility for each applicable NERC Reliability Standard and Requirement and tasks related to compliance. Texas RE has verified the completion of all mitigation activity.





TRE2018019912 PRC-008-0 R1 Kerrville Public Utility Board (KPUB) NCR10179 12/06/2007 03/31/2015 Compliance Audit

Completed


During a Compliance Audit conducted from April 17, 2018, through April 19, 2018, Texas RE determined that KPUB, as a Distribution Provider (DP) was in noncompliance with PRC-008-0 R1. Specifically, KPUB failed to have an Underfrequency Load Shedding (UFLS) equipment maintenance and testing program in place that included UFLS equipment identification, the schedule for UFLS equipment testing, and the schedule for UFLS maintenance, for all of KPUB’s UFLS equipment. During the compliance audit, KPUB was asked to provide a copy of its UFLS equipment maintenance and testing program. In response, KPUB provided its Protection System Maintenance Program (PSMP) and an interconnection agreement between it and another Entity. KPUB’s PSMP and interconnection agreement did not address UFLS equipment maintenance and testing scheduling as required by PRC-008-0 with regard to the bus potential transformers at issue. The root cause of this noncompliance was KPUB’s failure to adhere to its PSMP. KPUB failed to adequately inventory its equipment and subsequently follow the applicable requirements of PRC-008-0. This noncompliance started on December 6, 2007, when KPUB was registered, and ended on March 31, 2015, the last day PRC-008-0 R1 was applicable to the equipment at issue.

Risk Assessment

This noncompliance posed a minimal risk and did not pose a serious or substantial risk to the reliability of the bulk power system. Failure to identify all UFLS equipment and provide a schedule for maintenance and testing of all component types could potentially result in inconsistent testing and maintenance activity and/or documentation thereof, which could limit KPUB’s awareness of the working order of its UFLS equipment. The risk of this noncompliance was reduced by the fact that the other Entity that was party to the interconnection agreement with KPUB was regularly testing and maintaining the bus potential transformers in question during maintenance and testing of its own UFLS equipment at these facilities. The risk of this noncompliance was further reduced by the fact that the 2017 ERCOT UFLS survey shows that the UFLS relays associated with KPUB’s potential transformers would trip 15.81 MW of Load at the time of the survey. This is approximately 2.6% of its Transmission Operator’s (TOP’s) UFLS enabled Load (609.03 MW) at the time of the survey. No harm is known to have occurred. Texas RE considered KPUB’s compliance history and determined there were no relevant instances of noncompliance.

Mitigation To mitigate this noncompliance, KPUB: 1) performed maintenance activities in 2011, 2013, and 2015 that met the requirements of PRC-005-2, Table 3, which became effective April 1, 2015; and 2) updated its PSMP to include tracking of maintenance on all PRC-005-6 applicable Protection System Components (including its UFLS Components). Texas RE has verified the completion of all mitigation activity.





TRE2018019913 PRC-005-1 R2.1 Kerrville Public Utility Board (KPUB) NCR10179 09/01/2009 02/20/2018 Compliance Audit

Completed


During a Compliance Audit conducted from April 17, 2018, through April 19, 2018, Texas RE determined that KPUB, as a Distribution Provider (DP) was in noncompliance with PRC-005-1 R2.1. Specifically, KPUB failed to have evidence its Protection System devices were maintained and tested within the defined intervals of its Protection System Program (PSMP). KPUB’s PSMP in effect during the period of the noncompliance required KPUB to verify the presence and rotation of all voltage and currents associated with transformers on a 48-month maintenance interval. During the compliance audit, Texas RE determined that KPUB failed to conduct the required maintenance in a timely fashion for five of ten of KPUB’s transformers designed to detect faults on Bulk Electric System (BES) Elements. The due date for the required maintenance and testing for the Protection System Devices that are the subject of this noncompliance expired prior to KPUB’s transition of these devises to its PRC-005-2(i) PSMP, therefore, the longer maintenance period for relays under PRC-005-2(i) is not applicable to these devices during this time period. The root cause of this noncompliance was KPUB’s failure to adhere to its PSMP. More specifically, KPUB failed to monitor and track maintenance activities performed by third-party contractors and ensure that the required performance was completed in a manner, and within a timeframe, required by its PSMP. This noncompliance started on September 1, 2009, the due date for the earliest missed maintenance interval for one of its transformers, and ended on February 20, 2018, when KPUB completed testing on the last transformer that was the subject of overdue testing.

Risk Assessment

This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system. The failure to maintain transformers and associated protection systems in accordance with an Entity’s PSMP has the potential to affect the reliability of the bulk power system by increasing the risk that a protection system may fail and cause a misoperation. However, the risk regarding these instances of noncompliance was reduced because, while the periods for the actual maintenance exceeded the intervals prescribed within KPUB’s PRC-005-1 PSMP, they would not have exceeded the maximum 12-year maintenance intervals prescribed within PRC-005-6 for these transformers had KPUB transitioned this equipment to the new standard. KPUB is a municipal electric system that serves approximately 22,000 customers over an area of 146 square miles. No harm is known to have occurred. Texas RE considered KPUB’s compliance history and determined there were no relevant instances of noncompliance.

Mitigation To mitigate this noncompliance, KPUB: 1) completed the required testing on the applicable transformers; and 2) updated its PSMP to include tracking of maintenance on all PRC-005-6 applicable Protection System Components. Texas RE has verified the completion of all mitigation activity.






RFC2018020611 PRC‐005‐6 R3 CPV Maryland, LLC NCR11706 6/14/2017 10/10/2018 Self‐Report Completed Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible, or confirmed noncompliance.)

On October 22, 2018, the entity submitted a Self‐Report to ReliabilityFirst stating that, as a Generator Owner, it was not in compliance with PRC‐005‐6 R3.

On February 14, 2017, the entity brought a natural gas‐fired 2x1 combined cycle 745 MW electric generation facility (the Plant) into commercial operation. Prior to the start of commercial operations the entity engaged a third party operations and maintenance contractor (the Contractor) to operate the Plant and to perform services necessary to ensure that the entity was in compliance with its obligations relating to the Plant. (The entity was responsible for overseeing the Contractor’s NERC compliance program. The entity performed its oversight duty by participating in multiple meetings with the Contractor before, during, and after commissioning of the Plant; communicating with the Contractor personnel who were on site at the Plant regarding compliance; and communicating with the Contractor’s corporate NERC personnel regarding compliance of the Plant.)

During his time with the entity, the Contractor turned over the management team at the Plant significantly, including three different plant managers. On July 23, 2018, based on ineffective communication and a lack of responsiveness from the Contractor, the entity replaced the Contractor with a new operations and management contractor, the Second Contractor. Upon hiring the Second Contractor, the entity directed the Second Contractor to perform a comprehensive review of the entity’s compliance program as it relates to the NERC standards which apply to a Generator Owner/Generator Operator. The Second Contractor identified the following noncompliance.

The Plant has four (4) battery locations housing the Protection System Station DC supply Vented Lead‐Acid batteries: 1) the battery racks at the steam turbine generator power distribution center; 2) the combustion turbine generator Unit 11; 3) the combustion turbine generator Unit 12; and 4) the main balance of the Plant. The Vented Lead‐Acid batteries provide backup DC power to plant equipment in the event that main AC power is lost.

While the entity had a procedure in place that outlined the compliance requirements of PRC‐005‐6 R3 and required actions of the Contractor; the Contractor failed to perform them, or failed to document performance resulting in this noncompliance. Specifically, the Contractor was to perform maintenance on the Protection System Station DC supply Vented Lead‐Acid batteries required by PRC‐005‐6 R3 Table 1‐4. During the performance of the internal audit, the Second Contractor found no records to establish that the Contractor performed either four‐month or eighteen month maintenance on the Protection System Station DC supply Vented Lead‐Acid batteries required by PRC‐005‐6 R3 Table 1‐4.

The root cause of this noncompliance was that the entity did not have an adequate verification control to assure that the Contractor responsible for NERC compliance took the steps necessary to be fully compliant with PRC‐005‐6 R3 Table 1‐4.

This noncompliance involves the management practices of external interdependencies and verification. External interdependencies is involved because the noncompliance arose from the failure of a contractor and the entity’s inadequate oversight of that contractor. Verification management is involved because the entity failed to confirm that the Contractor was properly performing its NERC compliance functions.

The noncompliance began on June 14, 2017, the date the entity was required to comply with PRC‐005‐6 R3. The noncompliance ended on October 10, 2018, the date the entity had completed both the four month and eighteen month battery maintenance activities.

Risk Assessment This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system based on the following factors. The potential risk posed by this noncompliance is that unmaintained and untested batteries would fail and that failure could lead to local loss of load or transmission equipment at the substation. The risk is not minimal because the entity missed four different four month maintenance cycles. However, the risk is not serious because the entity operators performed basic daily checks of all batteries including voltage, amps, fan and visual inspection. Additionally, the entity performed battery bank testing during the commissioning phase in anticipation of coming online on February 14, 2017, which further reduces the risk of battery failure. ReliabilityFirst notes that there is no evidence of a loss of generation or downtime resulting from a battery related failure. No harm is known to have occurred.


1) conducted 4 calendar month battery maintenance with entity personnel;2) conducted the 18‐month battery maintenance activities;3) developed forms to be completed by personnel as they are completing the 4‐month and 18‐month battery maintenance activities;4) uploaded the required maintenance and testing activities into two different tracking software applications utilized at the Plant (Gensuite and Maximo). Gensuite will automatically notify Plant

Management of the 4‐month and 18‐month maintenance intervals, prior to the respective due dates. The Battery Maintenance Forms have been uploaded to Maximo to be generated as preventive






RFC2018020611 PRC‐005‐6 R3 CPV Maryland, LLC NCR11706 6/14/2017 10/10/2018 Self‐Report Completed maintenance attachments. The 4‐month forms will automatically generate every three calendar months and the 18‐month form will generate every 12 months, to be completed during the required maintenance interval by the entity.







RFC2018020612 PRC‐019‐2 R1 CPV Maryland, LLC NCR11706 2/14/2017 11/16/2018 Self‐Report Completed Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible, or confirmed noncompliance.)

On October 22, 2018, the entity submitted a Self‐Report to ReliabilityFirst stating that, as a Generator Owner, it was not in compliance with PRC‐019‐2 R1. On February 14, 2017, the entity brought a natural gas‐fired 2x1 combined cycle 745 MW electric generation facility (the Plant) into commercial operation. Prior to the start of commercial operations the entity engaged a third party operations and maintenance contractor (the Contractor) to operate the Plant and to perform services necessary to ensure that the entity was in compliance with its obligations relating to the Plant. (The entity was responsible for overseeing the Contractor’s NERC compliance program. The entity performed its oversight duty by participating in multiple meetings with the Contractor before, during, and after commissioning of the Plant; communicating with the Contractor personnel who were on site at the Plant regarding compliance; and communicating with the Contractor’s corporate NERC personnel regarding compliance of the Plant.) During his time with the entity, the Contractor turned over the management team at the Plant significantly, including three different plant managers. On July 23, 2018, based on ineffective communication and a lack of responsiveness from the Contractor, the entity replaced the Contractor with a new operations and management contractor (the Second Contractor). Upon hiring the Second Contractor, the entity directed the Second Contractor to perform a comprehensive review of the entity’s compliance program as it relates to the NERC standards which apply to a Generator Owner/Generator Operator. The Second Contractor identified the following noncompliance. Since the Plant became operational as of February 14, 2017, after the implementation of PRC‐019‐2 R1, the entity was required to have: (a) 40 percent of its generation units, synchronous condenser voltage regulating controls, limit functions, equipment capabilities, and Protection System settings coordinated and verified in accordance with PRC‐019‐2 Section G Attachments 1, 2, & 3 no later than 2/14/2017, (b) 60 percent of such facilities coordinated and verified by July 1, 2017; and (c) 80 percent of such facilities coordinated and verified by July 1, 2018. When the Second Contractor performed an internal review of the entity’s NERC Compliance Program in the summer of 2018, they did not find evidence or documentation that the entity had adhered to PRC‐019‐2 R1. The root cause of this noncompliance was that the entity did not have an adequate verification control to assure that the Contractor responsible for NERC compliance took the steps necessary to be fully compliant with PRC‐019‐2 R1. This noncompliance involves the management practices of external interdependencies and verification. External interdependencies is involved because the noncompliance arose from the failure of a contractor and the entity’s inadequate oversight of that contractor. Verification management is involved because the entity failed to confirm that the Contractor was properly performing its NERC compliance functions. The noncompliance began on February 14, 2017, the date the entity was required to comply with PRC‐019‐2 R1. The noncompliance ended on November 16, 2018, the date the entity completed the coordination and verification of facilities required by PRC‐019‐2 R1.

Risk Assessment

This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system based on the following factors. The risk posed by this instance of noncompliance is the discoordination of voltage controls which can result in a generator falsely tripping. The risk is not minimal because of the long duration of this noncompliance as the entity did not test any units until after the 80% deadline of July 1, 2018 (more than two years after the initial implementation date of July 1, 2016).However, the risk is not serious because the entity missed the 80% deadline by less than five months, and the entity identified the issue prior to the deadline to test 100% of its units (July 1, 2019) and was thereafter on schedule to meet the 100% deadline). The entity provides 750 MWs in generation with a gross capacity factor of 65.49%. ReliabilityFirst also notes that there is no record of a generator trip arising from incorrect relay or generator settings. Out of the 230 settings reviewed, 25 settings were changed (11%). Of the 25 settings that were changed, seven of those changes were greater than a 10% change. No harm is known to have occurred. ReliabilityFirst considered the entity’s compliance history and determined there were no relevant instances of noncompliance.

Mitigation

To mitigate this noncompliance, the entity: 1) engaged GE to provide a coordination report demonstrating that generating units, synchronous condenser voltage regulating controls, limit functions, equipment capabilities, and Protection System

settings are set per Section G Attachments 1, 2, and 3; 2) created recurring automated task reminders in Gensuite to remind the Plant Management to coordinate the voltage regulating system controls, limiters and protection functions with the applicable

equipment capabilities and settings of the applicable Protection System devices and functions at a maximum of every five calendar years. Gensuite will send an automated email to Plant Management as a reminder to complete this task several times prior to the due date;

3) implemented a documented PRC‐019 procedure that provides guidance for the entity to follow in order to ensure compliance with PRC‐019.






RFC2018020612 PRC‐019‐2 R1 CPV Maryland, LLC NCR11706 2/14/2017 11/16/2018 Self‐Report Completed ReliabilityFirst has verified the completion of all mitigation activity.






RFC2018020613 VAR‐002‐4 R2 CPV Maryland, LLC NCR11706 2/23/2017 10/9/2018 Self‐Report Completed Description of the Noncompliance (For purposes of this document, each noncompliance at issue is described as a “noncompliance,” regardless of its procedural posture and whether it was a possible, or confirmed noncompliance.)

On October 22, 2018, the entity submitted a Self‐Report to ReliabilityFirst stating that, as a Generator Operator, it was not in compliance with VAR‐002‐4 R2. On February 14, 2017, the entity brought a natural gas‐fired 2x1 combined cycle 745 MW electric generation facility (the Plant) into commercial operation. Prior to the start of commercial operations, the entity engaged a third party operations and maintenance contractor (the Contractor) to operate the Plant and to perform services necessary to ensure that the entity was in compliance with its obligations relating to the Plant. (The entity was responsible for overseeing the Contractor’s NERC compliance program. The entity performed its oversight duty by participating in multiple meetings with the Contractor before, during, and after commissioning of the Plant; communicating with the Contractor personnel who were on site at the Plant regarding compliance; and communicating with the Contractor’s corporate NERC personnel regarding compliance of the Plant.) During his time with the entity, the Contractor turned over the management team at the Plant significantly, including three different plant managers. On July 23, 2018, based on ineffective communication and a lack of responsiveness from the Contractor, the entity replaced the Contractor with a new operations and management contractor (the Second Contractor). Upon hiring the Second Contractor, the entity directed the Second Contractor to perform a comprehensive review of the entity’s compliance program as it relates to the NERC standards which apply to a Generator Owner/Generator Operator. The Second Contractor identified the following noncompliance. While the entity had a procedure in place that outlined the compliance requirements of VAR‐002‐4 and required actions of the Contractor; the Contractor failed to perform them, or failed to document performance resulting in this noncompliance. Specifically, the Contractor was to monitor voltage to ensure the Plant met the Transmission Operator’s (TOP) voltage schedule as required by VAR‐002‐4.1 R2. The entity operates within the Potomac Electric Power Company (PEPCO) zone of PJM and generates electricity at the 230 kV voltage level. The Second Contractor’s internal review included a review of previous correspondence between the TOP and the entity regarding voltage schedule specification. The TOP confirmed on September 28, 2018, that the entity must comply with the default PJM voltage of 235.0 KV +/‐4.0 kV. The Second Contractor then retrieved five minute operating data and examined every instance where the entity deviated outside the voltage scheduled for longer than 30 minutes without notifying the TOP. During the period starting February 23, 2017 and ending October 1, 2018, the entity deviated the required voltage schedule 125 times to varying degrees and all to the high side of 239 kV. Following is a table of the deviations:

Deviation Range Total Deviations (125) <0.5 kV 75 >0.51 kV <1.0 kV 23 >1.01 kV <1.5 kV 17 >1.51 kV <2.0 kV 6 >2.0 kV (2.06 kV, 2.55 kV, 2.46 kV, and 2.1 kV)

4

The root cause of this noncompliance was that the entity did not have an adequate verification control to assure that the Contractor responsible for NERC compliance took the steps necessary to be fully compliant with VAR‐002‐4. This noncompliance involves the management practices of external interdependencies and verification. External interdependencies is involved because the noncompliance arose from the failure of a contractor and the entity’s inadequate oversight of that contractor. Verification management is involved because the entity failed to confirm that the Contractor was properly performing its NERC compliance functions. The noncompliance began on February 23, 2017, the date the entity was required to comply with VAR‐002‐4 R2. The noncompliance ended on October 9, 2018, the date the entity completed its Mitigating Activities.

Risk Assessment

This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system (BPS) based on the following factors. The risk posed by this instance of noncompliance is allowing voltage schedule deviations at levels detrimental to the BPS’s voltage level without the TOP having knowledge, which could result in harm to the BPS. The risk is not minimal because of the extended duration and multiple instances. The risk is not serious because a majority of the voltage deviations were less than 0.5 kV outside of the voltage schedule, and all but 4 deviations were less than 2 kV outside of the voltage schedule. No harm is known to have occurred.






RFC2018020613 VAR‐002‐4 R2 CPV Maryland, LLC NCR11706 2/23/2017 10/9/2018 Self‐Report Completed ReliabilityFirst considered the entity’s compliance history and determined there were no relevant instances of noncompliance.

Mitigation

To mitigate this noncompliance, the entity: 1) trained all operations personnel on VAR‐002‐4.1 R2 compliance through a Learning Management System. The entity also trained operations personnel on how to adjust generator voltage on the Plant

load control page of the Distributed Control System; 2) laminated and taped to the control room operating board, the required voltage schedule and instructions on who to contact in the event the Plant is unable to maintain the requisite voltages schedule.

The entity also dedicated a Distributed Control System monitor to show operating personnel the Plant’s line voltage at all times, which includes a visual alarm that switches the screen to red when the line voltage reaches 232.0kV and 238.0kV; and

3) implemented internal controls to assist management with on‐going monitoring activities, as follows: (a) The control room shift turnover, which occurs every 12 hours, now includes the following two questions: (i) was the voltage schedule of 235kV (+/‐ 3kV) maintained during the shift?; and (ii) if not, was Pepco and PJM notified? (b) Plant Management will now complete a weekly Compliance Attestation after he/she reviews five minute voltage data for the prior week to determine if the Plant maintained the requisite voltage schedule during that week. The Compliance Attestation for weekly VAR‐002‐4.1 R2 review will be filed electronically at the Plant. If this review indicates the operating voltage was outside the schedule, Plant Management will confirm the appropriate notification was given to Pepco and PJM and was documented on the aforementioned shift turnover. (c) Plant Management is now required to complete a Compliance Attestation the month following each calendar quarter to summarize how the entity complied with the four applicable requirements of the VAR‐002 Standard. Gensuite will automatically notify Plant Management to complete this Attestation at the end of each quarter.






SERC2018019676 PRC-001-1.1(ii) R3 Columbia Energy LLC (Columbia) NCR11480 01/18/2017 10/03/2017 Self-Report Completed


On May 11, 2018, Columbia submitted a Self-Report stating that, as a Generator Operator, it was in noncompliance with PRC-001-1.1(ii) R3. Columbia did not coordinate protective system changes with its Transmission Operator (TOP) and Host Balancing Authority (BA). On January 18, 2017, Columbia made changes to relay settings to meet compliance requirements of PRC-024-2. On October 3, 2017, during an internal compliance review, Columbia could not locate evidence to demonstrate it coordinated the relay changes with the TOP and BA. Columbia contacted the TOP and BA to request evidence that the facility coordinated the changes, but the TOP and BA did not provide any documentation to demonstrate that the coordination occurred. Columbia made 24 relay setting changes for which it is had no evidence of coordination with the TOP and BA. This noncompliance started on January 18, 2017, when Columbia changed relay settings without coordinating with its TOP and BA, and ended on October 3, 2017, when Columbia coordinated the relay setting changes with the TOP and BA. The cause of this noncompliance was a lack of a compliance facilitation procedure that requires coordination with its TOP and BA for protective system changes. Columbia made the relay setting changes for PRC-024 compliance but failed to realize that those changes should have been coordinated with the TOP and BA.

Risk Assessment

This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system. Columbia’s failure to coordinate 24 relay changes for approximately nine months on a 749 MW facility could impact real-time operations. Specifically, it could create issues where facilities trip in unexpected ways or at unexpected times, result in inadequate protection for the interconnecting facility, and result in the BA and TOP not having an accurate depiction of its system. However, the relay setting changes that Columbia made only impacted its generators with respect to the trigger points as to when the units would trip in response to an event on the system. The relay setting changes results in the generators being less likely to trip for excursions. While using the previous settings, the TOP and BA would anticipate the generation trip sooner than what would occur with new settings implemented to meet compliance requirements of PRC-024-2. Additionally, the BA and TOP did not report any concerns with the relay setting changes made by Columbia after receiving notification. No harm is known to have occurred. SERC considered Columbia’s compliance history and determined that there were no relevant instances of noncompliance. Note: Columbia will be deregistered upon acceptance of this filing by FERC. SCE&G has purchased this facility.

Mitigation

To mitigate this noncompliance, Columbia: 1) sent an e-mail to its TOP and BA on October 3, 2017, describing the changes made to the relays to comply with PRC-024. The TOP and BA sent a response to Columbia on October 4, 2017 acknowledging receipt of the e-mail; and 2) implemented a compliance facilitation procedure. This procedure requires the performance of a monthly compliance assessment of the NERC Standards. The procedure explicitly states that new/changes to protection systems will be coordinated with TOP and BA, per PRC-001-1.1(ii). Training was completed on this procedure on August 10, 2017.





SERC2019021061 MOD-026-1 R6 Tennessee Valley Authority (TVA) NCR01151 02/03/2016 01/03/2019 Self- Report Completed


On February 15, 2019, TVA submitted a Self-Report stating that, as a Transmission Planner (TP), it was in noncompliance with MOD-026-1 R6. TVA did not provide a written response to the Generator Owner (GO) within 90 days of the verified excitation control system or plant volt/var control function model verification that the model was usable or not useable. On November 19, 2018, TVA determined that it failed to provide a written response to the GO, within 90 days of the data submission by the GO, for four units in November 2015, 12 units in April 2016, 20 units in June 2016, one unit in May 2017, and one unit in August 2017. Additionally, in some cases, TVA did not determine if the model was or was not useable within 90 days of model data submission. Ultimately, TVA determined that all of these models were useable. The model date for the five periods involved 38 units, which represented 4,590 MVA of TVA’s 33,526 MVA of generation (13.7%). As part of its mitigating activities, TVA completed an extent-of-condition review and identified eight additional instances that included 30 additional units. For these additional instances, the number of days that elapsed from the date of submission to the date TVA responded ranged from 91 days to 1,226 days. This noncompliance started on February 3, 2016, when TVA was required to provide the written response to the GO, and ended on January 3, 2019, when TVA submitted the last overdue response to the GO. The cause of the noncompliance was management oversight. Management failed to verify that all necessary internal controls, e.g., tracking software and an assignment of a secondary delegate responsible for tracking model submissions, were implemented to ensure timely responses to the GO. The TVA TP used an email inbox for the receipt of model data from GOs and assigned a single subject-matter-expert (SME) to monitor the mailbox. The SME failed to follow-up with written notification after discussions with the GO regarding models. The SME also went on leave after receiving the notification and failed to test the models and respond to the GO after returning from leave. Additionally, when the SME received a batch of submissions, the SME used the date of the last submission of the batch as the date to respond by, instead of addressing all units from the submission in the response.

Risk Assessment

This noncompliance posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system (BPS). TVA’s failure to notify the GO, within 90 days, that the models submitted are usable or not usable, could have led to the GO using incorrect models that could have resulted in erroneous assessments or as part of an incorrect corrective action plan. The noncompliance included 68 units. The response to the GO data submission ranged from 94 days to 1,226 days. TVA was able to include the models that were slightly late in the applicable SERC data submittal, however, for those that had longer delays, TVA was unable to include those models in the applicable SERC data submittal. Although TVA ultimately determined that the model information was useable, had TVA determined that the models were unusable, the GO would have been required to re-engage the contractor that had originally provided the verified models, which would have resulted in additional delays in obtaining a usable verified model for these units. In addition, TVA only identified the instant noncompliance when reviewing data for a previous GO MOD noncompliance. No harm is known to have occurred. SERC considered TVA’s compliance history and determined that there were no relevant instances of noncompliance.

Mitigation

To mitigate this noncompliance, TVA: 1) sent replies to all outstanding model validations as required by the Standard; 2) assigned a delegate in addition to the SME to assist in tracking the model submissions; 3) implemented a new action tracking software. As models are received by the TP, they will be entered into the system, which will then track the appropriate due dates for the replies. E-mails from the software are automatically sent to the SME each week, along with an additional e-mail as the due date for the reply approaches to both the SME and the assigned delegate; 4) conducted an extent-of-condition (EOC) review to verify the results of the GO EOC analysis to be certain that all received models reports have been tested and the appropriate replies to the GO have been performed; and 5) created a new Technical Procedure to document the process that the TP uses to process and reply to incoming MOD-026 model submissions with enough details such that a new employee will be able to track, process, and reply to a new MOD-026 model supplied by a GO. All impacted employees were notified and trained on the contents of the procedure.





TRE2017017837 PRC-005-1.1b R2 Sharyland Utilities, LLC, (SU) NCR04119 04/01/2015 06/13/2018 Self-Report Completed


On June 30, 2017, SU submitted a Self-Log stating that, as a Transmission Owner (TO), it was in noncompliance with PRC-005-6 R3. Specifically, SU reported that it did not maintain its Protection Systems included within the time-based maintenance program in accordance with the minimum maintenance activities and maximum maintenance intervals prescribed by the Standard. On January 9, 2018, SU provided evidence of additional instances of noncompliance with PRC-005-6 R3, as well as instances of noncompliance with predecessor Standard PRC-005-1.1b R2. After subsequent review, Texas RE determined that the instances of noncompliance posed a moderate risk to the reliability of the bulk power system. As a result, Texas RE removed this issue from the Entity’s Self-Log.

Texas RE determined that, consistent with SU’s June 30, 2017 Self-Report, SU failed to complete battery inspections at five of its seven substations within the maximum maintenance interval prescribed in Table 1-4(b) of PRC-005-6. The Standard requires that for Valve Regulated Lead-Acid (VRLA) station batteries that do not have internal ohmic value monitoring and alarming, an inspection of the condition of the batteries by measuring battery cell/unit internal ohmic values, must be conducted no less than every six calendar months. SU conducted the overdue inspections on April 11, 2017, 11 days after the applicable due date.

Texas RE determined that, consistent with SU’s January 9, 2018 submission of additional evidence, SU also failed to conduct protection system maintenance in accordance with PRC-005-1.1b R2, at its Railroad 138kV substation since the in-service date of the Facility, April 1, 2007. According to SU’s Protection System Maintenance Program (PSMP) in effect at the time, such testing and maintenance was due every eight years, which indicated that the due date for the Railroad Facility was April 1, 2015. SU completed the required testing and maintenance at the Railroad Facility on June 13, 2018, three years, two months, and 13 days after the testing and maintenance was due.

Texas RE determined that, consistent with SU’s January 9, 2018 submission of additional evidence, SU was also noncompliant with the Implementation Plan for PRC-005-6 R3 in that it failed to test at least 30% of its vented lead-acid (Table 1-4a), and 30% of its nickel-cadmium station batteries (Table 1-4c) by April 1, 2017. SU completed testing on 13 of its 42 (31%) vented lead-acid batteries, and 3 of 3 (100%) of its nickel-cadmium batteries, on November 8, 2017, 221 days after the applicable due date.

The root cause of this noncompliance was based on several factors. First, SU’s PSMP lacked precision. SU failed to draft its PSMP with sufficient detail as to properly identify the responsible staff members, and the particular testing responsibilities required by the Standard. Second, SU failed to shift responsibility for compliance to alternative staff members during staffing changes. SU failed to ensure that the testing required by the standard was completed notwithstanding the fact that the staff position responsible for this activity was vacant. Third, SU’s failure to conduct a proper extent-of-condition analysis when the noncompliance was first discovered resulted in the discovery of additional instances of noncompliance after SU’s initial Self-Report.

This noncompliance started on April 1, 2015, when SU failed to complete the required testing and maintenance at the Railroad substation, and ended on June 13, 2018, when SU completed the required testing and maintenance at the Railroad substation.

Risk Assessment This issue posed a moderate risk and did not pose a serious or substantial risk to the reliability of the bulk power system. PRC-005-6 is intended to ensure that proper testing and maintenance of Protection Systems occurs so that these devices remain in good working order. The risk to the bulk power system caused by SU’s noncompliance was reduced by the fact that SU's transmission substation batteries and other Protection Systems, including the Amarillo-area VRLA batteries, are monitored continuously via SCADA. Any failure could have been addressed immediately. SU has not experienced a transmission failure due to Protection System hardware failure. Also, for some of the substations at issue, Sharyland has a looped transmission system that connects all substations to multiple feed points in the ERCOT grid, helping to stabilize voltage as well as feed from different points during emergency situations. For the Railroad substation, there are two feeds from two separate substations. The failure or loss of one of these lines does not result in an overload condition, thus allowing the high-voltage DC line to continue to be rated for full import/export capability even during the loss of one line. No harm is known to have occurred.

Texas RE considered the SU’s compliance history and determined there were no relevant instances of noncompliance. Mitigation To mitigate this noncompliance, SU:

1) completed the required testing and maintenance activities;2) created the staff position of Transmission Maintenance Coordinator with the responsibility of assisting the Maintenance Supervisor with ensuring that testing and maintenance inspections are

carried out; and3) revised its PSMP so that it is more precise and identifies the specific staff members and alternates with the responsibilities to complete the tasks required by the Standard.

Texas RE has verified the completion of all mitigation activity.



NERC ID Name NCR ID Actions DL...2020/05/28 · Until March 2017, the entity utilized a...

Documents

Transcript of NERC ID Name NCR ID Actions DL...2020/05/28 · Until March 2017, the entity utilized a...