Post on 09-Sep-2020
1
Substation CIP Change Management Program
SPP CIP Users Group Dawn Berndt June 2014
2
Agenda Background Substation CIP Change Management Program Challenges Lessons Learned
3
Xcel Energy Background Serves 3.4 million electric customers in 8 states Three Operating Companies
Northern States Power (NSP) Public Service Company (PSCo) Southwestern Public Service (SPS)
NERC Compliance Regions MRO (NSP) WECC (PSCo) SPP (SPS)
4
CIP Change Control Substations
78 CIP substations with 1,100 Critical Cyber Assets 16,000 work orders per year at CIP substations More than 100 change control forms processed
each year Opportunity for change control errors is large
Change Control Program continues to evolve
5
CIP Change Control Key Players
CIP Consultant
Substation Field Engineering
Project Initiator
Engineering
Operations & Maintenance
Field Technician
6
CIP Substation Change Control
Pro
ject
/Wor
k In
tiatit
edC
IP S
ubst
atio
n / C
yber
Ass
ets
7 D
AYS
23 D
AYS
CIP Consultant
Project InitiatorsSFE -Substation Field Engineering
SED – Substation Engineering DesignSCE – Substation Commissioning
EngineeringSPE – System Protection Engineering
CO&M
YesNo
Store Project Spreadsheet in
ProjectWise
CIP Sub <AND>Cyber Assets
Impacted?
Record all affected Cyber
Assets in Spreadsheet
Update CIP ESP Diagram
Done
CIP Project Spreadsheet
Change Control not needed,
inform InitiatorProcess Done
Update CIP Asset Inventory
Issue Project Spreadsheet
to field
PROJECTS Small Capital or Maintenance Large Capital Small Special O&M Install or Upgrades
Normal or Emergency Maintenance
YesCIP Project
SpreadsheetRecord device information on Spreadsheet
Perform required configuration and testing
activities
Return data to CIP
Consultant
Substation Work
Is Substation Work Complete?<OR>
Has remote connectivity been established for the first time?
Yes 7 DAYS Start
Complete Project Spreadsheet
documentation
No
CIP Substation Change ControlP
roje
ct/W
ork
Intia
tited
CIP Consultant
Project InitiatorsSFE -Substation Field Engineering
SED – Substation Engineering DesignSCE – Substation Commissioning
EngineeringSPE – System Protection Engineering
Substation O&M
YesNo
CIP Sub <AND> Cyber Assets
Impacted?
CIP Project Spreadsheet
PROJECTS Small Capital or Maintenance Large Capital Small Special O&M Install or Upgrades
Normal or Emergency Maintenance
YesCIP Project
SpreadsheetRecord device information on Spreadsheet
Perform required configuration and testing
activities
7
CIP Substation Change Control
Pro
ject
/Wor
k In
tiatit
edC
IP S
ubst
atio
n / C
yber
Ass
ets
7 D
AYS
23 D
AYS
CIP Consultant
Project InitiatorsSFE -Substation Field Engineering
SED – Substation Engineering DesignSCE – Substation Commissioning
EngineeringSPE – System Protection Engineering
Substation O&M
YesNo
Store Project Spreadsheet in
ProjectWise
CIP Sub <AND> Cyber Assets
Impacted?
Record all affected Cyber
Assets in Spreadsheet
Update CIP ESP Diagram
Done
CIP Project Spreadsheet
Change Control not needed,
inform InitiatorProcess Done
Update CIP Asset Inventory
Issue Project Spreadsheet
to field
PROJECTS Small Capital or Maintenance Large Capital Small Special O&M Install or Upgrades
Normal or Emergency Maintenance
YesCIP Project
SpreadsheetRecord device information on Spreadsheet
Perform required configuration and testing
activities
Return data to CIP
Consultant
Substation Work
Is Substation Work Complete?<OR>
Has remote connectivity been established for the first time?
Yes 7 DAYS Start
Complete Project Spreadsheet
documentation
No
C
IP S
ubst
atio
n / C
yber
Ass
ets
YesNo
Record all affected Cyber
Assets in Spreadsheet
CIP Project Spreadsheet
Change Control not needed,
inform InitiatorProcess Done
Issue Project Spreadsheet
to field
YesCIP Project
SpreadsheetRecord device information on Spreadsheet
Perform required configuration and testing
activities
Substation Work
Is Substation Work Complete?<OR>
Has remote connectivity been established for the first time?
Yes 7 DAYS Start
No
8
CIP Substation Change Control
Pro
ject
/Wor
k In
tiatit
edC
IP S
ubst
atio
n / C
yber
Ass
ets
7 D
AYS
23 D
AYS
CIP Consultant
Project InitiatorsSFE -Substation Field Engineering
SED – Substation Engineering DesignSCE – Substation Commissioning
EngineeringSPE – System Protection Engineering
Substation O&M
YesNo
Store Project Spreadsheet in
ProjectWise
CIP Sub <AND> Cyber Assets
Impacted?
Record all affected Cyber
Assets in Spreadsheet
Update CIP ESP Diagram
Done
CIP Project Spreadsheet
Change Control not needed,
inform InitiatorProcess Done
Update CIP Asset Inventory
Issue Project Spreadsheet
to field
PROJECTS Small Capital or Maintenance Large Capital Small Special O&M Install or Upgrades
Normal or Emergency Maintenance
YesCIP Project
SpreadsheetRecord device information on Spreadsheet
Perform required configuration and testing
activities
Return data to CIP
Consultant
Substation Work
Is Substation Work Complete?<OR>
Has remote connectivity been established for the first time?
Yes 7 DAYS Start
Complete Project Spreadsheet
documentation
No
C
7
DA
YS23
DA
YS
Store Project Spreadsheet in
ProjectWise
Update CIP ESP Diagram
Done
Update CIP Asset Inventory
Return data to CIP
Consultant
Yes 7 DAYS Start
Complete Project Spreadsheet
documentation
9
10
11
12
13
14
15
16
CIP Change Control Challenges
Manual change control process Rely on personnel to initiate change control for CIP substations Multiple departments required to fill out information
Depend on human performance for documentation Time Accuracy Completeness
Large number of people had access to CIP Substations Almost 25% were external parties (other utilities, contractors) Paring this list down
Training and awareness for all affected personnel on correct procedure
17
CIP Change Control Lessons Learned
Workflow automation is crucial when dealing with many touch points in CIP Substations Identify process handoffs and consider controls
Importance of ongoing training and awareness Employee turnover Contractors/Consultants Face-to-face is valuable with field resources
Implementing compliance ‘controls’ can help prevent and detect issues
18
CIP Change Control Controls Implemented
Established controls to help identify substation work requiring change control upfront Automated biweekly report to review new work orders Established process for checking CIP status on new substation
design projects Implemented manual security testing of access points Conduct bi-weekly change control meetings in each Operating
Company with Engineering, Operations and CIP Compliance groups Quarterly monitoring program Implemented tagging of disconnected device cables Installed access point “CIP notice” labels
19
CIP Change Control Improved Field Awareness
20
Port plugs and locks to prevent inadvertent connections
Technician guide for connecting and making changes to cyber assets in the Electronic Security Perimeter
Restructuring process documentation Change control awareness posters in CIP substations
CIP Change Control Controls In Development
21
Implement additional controls Asset management solution (not Excel spreadsheets) Investigate automation solutions
Change control workflow Configuration management
Evaluate viability of additional testing facilities
CIP Change Control Preparation for CIP V5
22
Questions?