SPP RE Audit Processes and Sampling
25
March 16, 2016 Shon Austin Lead Compliance Specialist – CIP Mike Hughes Lead Compliance Engineer - O&P 1 SPP RE Audit Processes and Sampling
Transcript of SPP RE Audit Processes and Sampling
March 16, 2016
Shon AustinLead Compliance Specialist – CIP
Mike HughesLead Compliance Engineer - O&P
1
SPP RE Audit Processes and Sampling
Outline
• Inherent Risk Assessment (IRA)
• Internal Control Evaluation (ICE)
• Audit Packet
• RAT-STATS
• Pre-Audit
• Audit
• Audit Report
• Feedback Form
2
Inherent Risk Assessment (IRA) Questionnaire and RAT-STATS Sample
• Registered Functions
• Transmission and Generation Facilities
• Underfrequency Load Shedding
• System Network Information
• Control Centers
• Events
• Internal Compliance Program
• RAT-STATS Spreadsheet (first pass)
• SCADA Environment3
IRA Summary Report
• Summary of Risk Factors
– High
– Moderate
– Low
• Compliance Oversight Program
– Standards and Requirements
– Proposed Monitoring Tool for each Standard
Audit; Spot Check; Periodic Data Submittal; Self-Certification
• Offer to perform an Internal Control Evaluation (ICE)
4
Benefits of ICE
• Internal control consultation
• Scalable – Registered Entities pick/choose requirements
• Possible reduction in RAT-STATS sampling
• Possible shift from audit to Self-Certification
• Not an audit; non-binding recommendations
5
145
130 115 100
Start of the monitoring activity
Registered Entity will provide
documentation and SPP RE will evaluate the effectiveness of
the Internal Controls
The Registered Entity will provide
documentation and SPP RE will evaluate
the design of the Internal Control
Upon receiving the IRA Letter, the
Registered Entity will have 10 days to
request an ICE
IRA completed and approved at approx
165 days prior to monitoring activity
and the IRA Letter is sent to the
Registered Entity
90 Days40 Days25 Days10
Days
165 Days 90 Days155 Days 130 Days 0 Days
SPP RE will send the Registered Entity the
monitoring activity notification at 90 days as
stated in the Rules of Procedure
IRA and ICE Timeline
15 Days
180 Days
IRA started at approx 180 days
prior to monitoring
activity
ICE Summary Report
• Review Key Control Design
• Implementation Level of Key Controls
– Fully Implemented <- to -> Missing
– Monitoring Tool for each Standard
Audit; Spot Check; Periodic Data Submittal; Self-Certification
• Impact on Compliance Oversight Plan
– Reduced Fieldwork; Self-Certification
7
145
130 115 100
Start of the monitoring activity
Registered Entity will provide
documentation and SPP RE will evaluate the effectiveness of
the Internal Controls
The Registered Entity will provide
documentation and SPP RE will evaluate
the design of the Internal Control
Upon receiving the IRA Letter, the
Registered Entity will have 10 days to
request an ICE
IRA completed and approved at approx
165 days prior to monitoring activity
and the IRA Letter is sent to the
Registered Entity
90 Days40 Days25 Days10
Days
165 Days 90 Days155 Days 130 Days 0 Days
SPP RE will send the Registered Entity the
monitoring activity notification at 90 days as
stated in the Rules of Procedure
IRA and ICE Timeline
15 Days
180 Days
IRA started at approx 180 days
prior to monitoring
activity
Audit Packet
• Notification letter
• Audit team bios
• Monitoring scope
• Reliability standard audit worksheets (RSAWs)
• Draft agenda (Ops & Planning)
• RAT-STATS spreadsheet
• Webex to review audit packet
9
10
File Structure
RAT-STATS Spreadsheet – O&P
Sent with IRA questionnaire - first pass sampling:
• BES Substations
• BES Transmission Lines
• BES Generation
• Flowgates
11
RAT-STATS Spreadsheet – O&P
Sent with Audit Packet - second pass selected samples:
• Transmission Line FAC-003 Vegetation
• Transmission Line FAC-008 Facility Ratings
• Generation FAC-008 Facility Ratings
• TOP-002 Next-Day/Current Day Studies
12
RAT-STATS PRC-005 Example
• Questionnaire => First pass 22 BES substations
• SPP RE RAT-STATS random sample => 8 substations
• Second pass BES relays => 110 relays
• SPP RE RAT-STATS random sample => 29 relays
• Submit records for Communications, Batteries, Battery Charger, Sensing Devices (PTs & CTs), and DC Circuitry associated with the 29 identified relays
13
CIP RAT-STATS
14
CIP RAT-STATS Spreadsheet
• Spreadsheet is sent with Audit Packet
• Registered Entity submits “population” in spreadsheet, including but not limited to:
– Identified assets containing BES Cyber Systems (BES Assets)
– Cyber Assets (CA)
– BES Cyber Systems (BCS/BCS detail)
– Logical group of Cyber Asset(s) into one or more BES Cyber Systems (CABCS)
15
CIP RAT-STATS Spreadsheet
One week after receiving sampling data, SPP RE selects samples from the population
• EXAMPLE (CIP-004 R3 Part 3.5): For each sampled “Personnel” in Sample Set, provide a redacted copy of personnel risk assessment with only sufficient evidence to demonstrate:
1. Assessment date
2. Identity check was performed
3. Appropriate criminal history check was performed
• EXAMPLE (CIP-010 R1 Part 1.1): For each Cyber Asset selected in Sample Set, please provide the baseline configuration for this Cyber Asset.
• EXAMPLE (CIP-007 R4 Part 4.2) : For each Cyber Asset selected in Sample Set, provide evidence of actual alerts generated, if any, during the audit period.
16
CIP RAT-STATS Example
• Universe: Registered Entity submits 176 Personnel
• Sample: SPP RE selects 57
• Universe: Registered Entity submits 54 Cyber Assets
• Sample: SPP RE selects 37
• Universe: Registered Entity submits 2,026 generated alerts
• Sample: SPP RE selects 57
17
O&P Timeline for Submission of Evidence
• 60 Days before audit:
– Email stating agreement with Draft Audit Agenda– Verification of Recent Employment (potential conflicts
of interest)– First pass RAT-STATS spreadsheet data
• 45 Days before audit:
– First round of evidence– Responses to RSAWs– Attestation letter
• 15 Days before audit:
– Objection to audit team members
18
CIP Timeline for Submission of EvidenceAudit Start Date: 10/19/2016 D-nn* End of Week
IRA Start Date 4/22/2016 D-180 180
IRA Notice 5/7/2016 D-165 165
ICE Approval 5/17/2016 D-155 155
V5 Notice 6/6/2016 D-135 135
Initial Notice: 7/17/2016 D-90 94
Initial Evidence Request 7/17/2016 D-90 94
Pre-Sample Evidence Due: 7/31/2016 D-76 80
Sample Selection Request: 8/7/2016 D-69 73
Survey/Team Objections Due: 8/21/2016 D-56 59
RSAWs/Workbooks Due: 8/28/2016 D-48 52
Initial Evidence Due: 8/28/2016 D-48 52
Second Evidence Request: 9/4/2016 D-41 45
Second Evidence Due: 9/25/2016 D-20 24
First Day Onsite: 10/19/2016 D-0 019
* “D” is first day of audit
Pre-Audit
• Audit Team Review of Evidence
• Possible Subject Matter Expert (SME) Interviews
• Evidence Requests
• Periodic Status Reports
20
21
22
23
Audit
• Opening Presentation
• SME Interviews
– Control Center Visit (Transmission Operator)
• Evidence Requests
• Daily Status Reports
• Exit Presentation
• Audit Report Review
• Final Report(s)
• Feedback Survey Link
24
25
Shon Austin Mike HughesLead Compliance Specialist - CIP Lead Compliance Engineer - O&P501.614.3273 [email protected] [email protected]
