Reliability Assurance Inii iitiative Assurance Initiative/RAI B… · certs, self reports, and...

Reliability Assurance i i iInitiative

Board of Trustees Compliance CommitteeNovember 6, 2013

End-State Vision – Compliance

• Compliance regime shifts to using standard, risk‐based audit practices similar to other industriespractices similar to other industries

• Scope of audit based on standard approach to assessing entity’s risk to reliability

• Compliance focus shifts to assess strength of management controls relative to meeting standards

ll f l k l l• Process allows for lower risk violations to stay in compliance space

• Visibility of all violations maintained including self‐reported toVisibility of all violations maintained, including self reported, to allow for trend analysis

RELIABILITY | ACCOUNTABILITY2

End-State Vision – Enforcement

• Focus on noncompliance that poses a serious and substantial risk to the reliability of the bulk power system (BPS)risk to the reliability of the bulk power system (BPS) NERC and the Regional Entities exercise discretion whether to initiate an

enforcement action or to address issues outside of enforcement that do not pose a serious or substantial risknot pose a serious or substantial risk

• Recognize existing processes in place and encourage registered entities to continue to self‐identify, mitigate, and record noncompliance under the oversight of NERC and the Regional Entities Well‐controlled entities will log self‐identified issues that do not pose aWell controlled entities will log self identified issues that do not pose a

serious or substantial risk

NERC and the Regional Entities continue to maintain visibility regarding all noncompliance


all noncompliance

Four Key Areas of Focus for 2013

• The auditor handbook is currently under development and is targeted for completion by the end of 2013 The training and

• The auditor handbook is currently under development and is targeted for completion by the end of 2013 The training andAuditor Handbook targeted for completion by the end of 2013. The training and rollout efforts will occur in 2014.targeted for completion by the end of 2013. The training and rollout efforts will occur in 2014.

Auditor Handbook

• The prototypes and pilot programs are currently underway and• The prototypes and pilot programs are currently underway andThe prototypes and pilot programs are currently underway and will continue throughout 2013. By year‐end, the results and lessons learned will be evaluated and will serve as inputs into an ERO‐wide, risk‐based auditing approach.

The prototypes and pilot programs are currently underway and will continue throughout 2013. By year‐end, the results and lessons learned will be evaluated and will serve as inputs into an ERO‐wide, risk‐based auditing approach.

Prototypes and Pilot Programs

• Improvements to self‐reporting process, including process and communication improvements, are being implemented in 2013. Other improvements will be tested in a series of pilot programs extending into 2014.

• Improvements to self‐reporting process, including process and communication improvements, are being implemented in 2013. Other improvements will be tested in a series of pilot programs extending into 2014.

Improvements to Self‐Reporting

gg

• Enhancements to the FFT process, including a triage process will be implemented in 2013. Other improvements associated with greater exercise of enforcement discretion will be tested in a series

• Enhancements to the FFT process, including a triage process will be implemented in 2013. Other improvements associated with greater exercise of enforcement discretion will be tested in a seriesFFT Enhancements


greater exercise of enforcement discretion will be tested in a series of pilot programs extending into 2014. greater exercise of enforcement discretion will be tested in a series of pilot programs extending into 2014.

Timeline

Pl i Ph D l ImplementationPlanning Phase Develop the end game pilot strategy

2014

ImplementationExecute roll-out

across ERO.

QTR 2 QTR 3 QTR 4 QTR 1 QTR 2QTR 1 QTR 2 QTR 3 QTR 4

2013 2015Pilot Phase

Pilot the approach – audits, self certs, self reports, and

Early Adoption/Design Completion

I l t “ i k i ”

“exceptions” disposition.


Implement “quick wins”, complete design and file where

necessary

Industry Engagement Timeline

Industry Readiness Assessment Webinar

(Broad Use of C li

Industry Focus Group Activity

Pilot Evaluation and RAI Rollout Webinar Continuous

Communication, Training and Feedback During

Deployment

Compliance Approach

Auditor Handbook

Annual Implementation Plan

Webinarnce

Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4

Informational Webinar

Com

plia

2013 2014 2015

men

tE

nfor

cem

Industry Focus Group Activity

Presentation of Pilot Results via

workshop or webinar


Pilot evaluation criteria finalized

Includes risk assessment, identifying and testing of controls

RAI from Industry Perspective

Q3 Q3 Q3

2013 2014 2015 2016

• I m p r o v e d s e l f r e p o r t i n g /M R R E p r o c e s s e s

• F F T e n h a n c em e n t s

• L e s s e r r i s k v i o l a t i o n s d o n o t r e s u l t i n

f t t i

• E n d s t a t e e n f o r c e m e n t i m p l e m e n t a t i o nF F T e n h a n c em e n t s

i m p l e m e n t e d• S i m p l i f i e d AM L• A u d i t s c o p e

i n c o r p o r a t e s r i s k

e n f o r c e m e n t a c t i o n s• R i s k b a s e d a u d i t s

f o r P h a s e 1 E n t i t i e s• A u d i t s c o p e b a s e d

E R O i k

• T r a n s i t i o n t o r i s k b a s e d a u d i t s f o r r e m a i n i n g e n t i t i e s

i n c o r p o r a t e s r i s k c r i t e r i a

o n c omm o n E R O r i s k b a s e d me t h o d o l o g y


CMEP Posted

Reliability Assurance i i i liInitiative – Compliance

TrackTrack

Key Deliverables

• Auditor Manual and Handbook Defined audit activities Defined audit activities

Structured audit approach

• RAI Regional Pilots Test approaches to risk assessment, audit scoping and controls

assessment

Evaluation of pilotsEvaluation of pilots

• Common ERO‐wide methodologies for: Annual Planning for Compliance Activities (Annual Implementation Plan)

Entity risk assessment

Audit scoping approach

Management controls assessment


Management controls assessment

Auditor Handbook

Organization• Manual and Handbook

framework completed

• Format and selected sectionsFormat and selected sections presented at Auditor Workshop – September 18

• Target completion –Target completion December 2013

• Auditor training and rollout plan under developmentplan under development


RAI Regional Pilots

• By year‐end, results and lessons learned will serve as inputs into an ERO‐wide, risk‐based auditing approach.

f l• Expectations from pilots: Process methods for risk assessments

o Standardize risk‐evaluation criteria

Li k i k i ho Link risk to a common scoping approach

Process methods for testing of management controlso Establish common control assessment criteria

o Define testing methodology and documentation requirementso Define testing methodology and documentation requirements

Evaluation criteria to support best of class methods

• All methods/prototypes will be: E l d d i b f l d bl do Evaluated to determine best of class and most reasonable to adopt

o Reviewed and set as policy at the ERO EMG level

• Policy will be anchored in Compliance Auditor Manual and Handbook, as well as the annual implementation plan


well as the annual implementation plan

Pilot Development and Evaluation

Pilots are designed and will be evaluated to a selected set of criteria

RFC/AEP

END STATE DELIVERABLES

• Entity risk assessment

Common E l i

SERC/PowerSouth

NPCC/NYPA

MRO/ATC

assessment• Audit scoping

approach• Management

Evaluation Criteria

NPCC/NYPA

• Management controls assessment and testing

WECC/Confidential


RAI Regional Pilots

Region Partner(s) Risk Assessment Audit Scoping

Controls Assessmentp g

MRO ATC Established risk criteria and entity survey input

Predeterminedstandards

Detailed audit testing

RFC AEP Established risk criteria Determined by Detailed audit (via maturity model) and entity survey input

risk assessment results

testing

SERC PowerSouth* Established risk criteria and entity survey input

Determined by risk assessment

Detailed audit testingand entity survey input risk assessment

resultstesting

WECC Confidential Established risk criteria and entity survey input

Determined by risk assessment results


results

NPCC NYPA Established risk criteria and entity survey input

Determined by risk assessment results



*Second round of pilots

ERO Pilot Evaluating Criteria

Evaluation Criteria Criteria Explanation

Transparency for oversight purposes• Ability to document scope for oversight review (FERC

and NERC)• Change management

• Defined approach demonstrates repeatability and

Program design elements effectiveness

• Defined approach demonstrates repeatability and scalability

• Program identifies an entity’s key functional activities• Consideration of the 11 CMEP-IP risk factor elements

Alignment to the Reliability Standards• Are risks factors ranked only as they relate to specific

auditable standards and requirements?• Are other reliability risks considered?

Implementation requirements• Does program require additional regional resources?• Does complexity require complex IT platforms?


Impact on Registered Entities• Impact on different entity levels (large, medium, small) • Timely, reasonable implementation

The Differences

• Improved Bulk Electric System reliability due to enhanced focus on high reliability risks and controls that mitigate those riskshigh reliability risks and controls that mitigate those risks.

• Increased ERO Enterprise compliance and enforcement consistency.

• Tools and scoping appropriate to the entities risk and management practices.

R f d li bilit d ff ti t l• Resources focused on reliability and effective controls.• Improved Reliability Standards development and retirement as a result of informational feedback loop.result of informational feedback loop.

• Higher level of compliance program maturity.


The Differences

• Recognizing and rewarding registered entities that design and implement strong management control programs

• Process harmonization

M h i f li l• More comprehensive use of compliance tools

• Transition away from a comprehensive checklist‐style audit with burdensome administrative redundancy to a targeted, riskwith burdensome administrative redundancy to a targeted, risk approach

• Scalability of approach

• Distinctions in the application of compliance monitoring given the risk to reliability


Next Steps

• Continue quarterly industry forums Compliance pilot results Compliance pilot results

Principles for entity risk assessment and assessing internal controls

• Continue to review program milestones and schedule at BOTCC

• Complete compliance design elements by Q2 2014

• Filing on internal controls approach for CIP Version 5 targeted for Q2 2014


Reliability Assurance i i i fInitiative – Enforcement

TrackTrack

Issues Identified by Industry Focus Group

• Combination of long overall processing time forCombination of long overall processing time for minimal risk issues with lack of regular communication throughout enforcement process.

• Lack of information by registered entities on sufficiency of content and process for self‐reports

• Lack of centralized information collection (particularly for multi‐region registered entities)

• Remaining process inconsistencies being phased out


Overall Approach to Solutions

• Early identification and streamlined processing ofEarly identification and streamlined processing of lesser risk issues with appropriate visibility

• Increased exercise of discretion by NERC and yRegional Entities

• Incentives for registered entities to develop and g pmaintain the necessary processes to self‐identify and mitigate all noncompliance

• Pilots allow testing of concepts associated with end‐state vision in a controlled manner (limited scope of i d titi d i th il t h )


issues and entities during the pilot phase)

Key Activities in 2013

• User guide and improved communication

T i d ff f i i l i k i• Triage and off‐ramp for minimal risk issues

• Multi‐Region Registered Entity Process

• Improved intake and process flow• Improved intake and process flow

• Pilots Aggregation of minimal risk issues

Alternative path to enforcement


Short-term Solutions – User Guide and Improved Communication

• Additional guidance on process and content of self‐report: ERO enterprise user guide for self‐reports and mitigationp g p g

Point of contact at Regional Entity for additional guidance


Short-Term Solutions – Triage

• Early triage with off‐ramp for minimal risk issues Triage executed within 60 days on average Triage executed within 60 days on average

Possible outcomes from triage:o Enough information to support a finding of minimal risk

I d t t i f t ti d tifi ti i i d t NERC- Issue does not trigger an enforcement action and notification is issued to NERC and FERC (pilot), or

- Issue is processed as an FFT

o More information is required prior to determining the disposition, oro More information is required prior to determining the disposition, or

o Noncompliance needs to be enforced and processed as a SNOP or full NOP

Triage‐related metrics being developed


Short-Term Solutions – MRRE

• Multi‐Region Registered Entity Process Evaluation of current practices (e g assignment of lead CEA and Evaluation of current practices (e.g., assignment of lead CEA and

improved coordination among the Regional Entities involved)

Development and publication of a process that reflects the current best practices and any other necessary changespractices and any other necessary changes


Medium- to Long-Term Solutions –Process Flow

• Improved intake and process flow: Ability to log noncompliance information prior to self report Ability to log noncompliance information prior to self‐report

Ability to augment information

Ability to cross reference information already provided

Ability to store streamlined record for matters that do not trigger an enforcement action


Medium- to Long-Term Solutions –Process Flow


Medium- to Long-Term Solutions –Pilots

• Pilots Aggregation of minimal risk issues Aggregation of minimal risk issues

Alternative path to enforcement

• Timing First cycle – October 2013 to April 2014


Aggregation Pilot – Parameters

Minimal risk issues only

Selected Regional Entities and selected registered entities (see ScopeSelected Regional Entities and selected registered entities (see Scope, below)

Record maintained by registered entity during aggregation cycle, until system changes allow direct input of information into Regional Entitysystem changes allow direct input of information into Regional Entity system

Format and content of record is similar to FFT spreadsheet

P i di i f t d i b R i l E tit Periodic review of aggregated issues by Regional Entity.o First cycle began in October 2013; First evaluation of results will be in April 2014.

N tifi ti ill b t t NERC d FERC t th ti f i d t Notifications will be sent to NERC and FERC at the time of review and at disposition.


Aggregation Pilot – Scope

Regional Entity Registered Entity Scope (Minimal Risk issues))

FRCC Duke Energy Florida PRC-005, R2MRO Nebraska Public Power

DistrictAll standards

Alliant East and WestNPCC NYPA All standardsSERC Associated Electric All standardsSERC Associated Electric

Corporation Inc.All standards

Texas RE Lower Colorado River Authority

All standards –transmission registration y gonly

ReliabilityFirst AEP Standards mapped to process areas appraised


by RF

Alternative Path to Enforcement –Parameters

Minimal risk issues only (during pilot phase)

Pilot to consider initial findings from Compliance Audit pilots at MROPilot to consider initial findings from Compliance Audit pilots at MRO and SERC; as pilots progress, NERC and Regional Entities will identify a specific set of self‐reported issues for inclusion in the pilot

Notifications to NERC and FERC at the time of intake and dispositionNotifications to NERC and FERC at the time of intake and disposition

Format and content of record is similar to FFT spreadsheet (recorded in a spreadsheet during pilot phase and subsequently entered into Regional Entity portal)Regional Entity portal)

Records retained by Regional Entity for review by NERC and FERC

NERC to review sample issues disposed without an enforcement action

Begin in November 2013 with findings from audit pilots


Next Steps

• Continue to work with industry focus group to review deliverablesdeliverables

• Beginning in April 2014, NERC and the Regional Entities will review the results of the pilots and develop a strategy for expanding based on results Consider whether to continue to allow aggregation and whether to

expand concept beyond initial scope or issues and entities p p y p

• NERC expects to make a filing with FERC to review the results of the pilots and seek any changes to existing rules on or before th f th t f 2014the fourth quarter of 2014 Consider the appropriate scope for implementing the alternative path

to enforcement throughout the ERO enterprise


Please send any questionsPlease send any questions or comments to:[email protected]


Reliability Assurance Inii iitiative Assurance Initiative/RAI B… · certs, self reports, and...

Documents

Transcript of Reliability Assurance Inii iitiative Assurance Initiative/RAI B… · certs, self reports, and...