August 14, 2003 Blackout - NERC 14 2003... · Michigan 4:10:36 – 4:10:37 PM 38 Northern Ohio and...
Transcript of August 14, 2003 Blackout - NERC 14 2003... · Michigan 4:10:36 – 4:10:37 PM 38 Northern Ohio and...
1
August 14, 2003 Blackout
Gerry CauleyDirector – Reliability Performance
North American Electric Reliability Council
2
NERC Response
●First hoursWorking with reliability coordinators –assessing restoration efforts and identifying what had initially tripped
●Began organizing investigation effortOutage covered three NERC regionsEstablished a Steering Group of industry experts
●Became technical lead for the U.S.-Canada Power System Outage Task Force
2
3
Investigation Organization OverviewSteering Group
MAAC/ECAR/NPCC Coordinating Group
MAAC
ECAR
NPCC
MEN Study Group
Project Planning and Support
Sequence of Events
Data Requests and Management
Investigation Team Lead
System Modeling and Simulation Analysis
NERC & Regional Standards/Procedures
& Compliance
Transmission System Performance,
Protection, ControlMaintenance & Damage
Operations - Tools, SCADA/EMS
Communications Op Planning
System Planning, Design, & Studies
Root Cause AnalysisCooper Systems
Generator Performance, Protection, Controls
Maintenance & Damage
U.S – CanadaTask Force
Vegetation/ROW Management
Frequency/ACE
Restoration
Investigation Process Review
4
Transmission Map Key
ONTARIO
Transmission Lines
765 kV500 kV345 kV230 kV
Transmission Lines
765 kV500 kV345 kV230 kV
3
5
Footprints of Reliability Coordinators in Midwest
6
Warm But Not Unusual for August
4
7
August 14 Imports to Northeast (Except ISO-NE, and Maritimes) Compared to 6/1 to 8/13/2003
-8000
-6000
-4000
-2000
0
2000
4000
6000
0:00
2:00
4:00
6:00
8:00
10:00
12:00
14:00
16:00
18:00
20:00
22:00
Hour (EDT)
MW
Max ImportsMax ExportsAverage14-Aug
Impo
rtsEx
ports
16:00
8
Voltages Prior to 15:05 EDT August 14
5
9
Transmission Lines
765 kV500 kV345 kV230 kV
Transmission Lines
765 kV500 kV345 kV230 kV
411
399
689
8241240
2411
1518
629
1588
2753
702
18841865
Reactive Power Margins from V-Q Analysis
158
144
178
303
466468
1923
681
10702226
578
18161803
Reactive Power Margin @ 15:32
Reactive Power Margin @ 15:41
430
143
10
Avon 345kV
-1000
-900
-800
-700
-600
-500
-400
-300
-200
-100
0
100
200
300
400
500
0.7 0.8 0.9 1 1.1
Voltage (p.u.)
Rea
ctiv
e M
argi
n
15-05 Chamberlin-Harding
15-32 Hanna-Juniper
15-41 Star-S Canton
15-45-40 Canton Central-Tidd
15-59 West Akron 138kV Lines
6
11
East Lake 5 Trip: 1:31:34 PM
ONTARIO
2
1
ONTARIO
12
East Lake 5 Exciter Failure Causes Trip
7
13
Stuart Atlanta Trip: 2:02 PM
ONTARIO
14
MISO State Estimator and Reliability Analysis
●MISO state estimator and contingency analysis ineffective from 12:37 to 16:04
State estimator not solving due to missing information on lines out in Cinergy then DPLHuman error in not resetting SE automatic trigger
●Using Flowgate Monitoring tool to monitor conditions on previously identified critical flowgates
8
15
FirstEnergy Computer Failures● 14:14 Alarm logger fails and operators are not aware
No further alarms to FE operators● 14:20 Several remote consoles fail● 14:41 EMS server hosting alarm processor and other
functions fails to backup● 14:54 Backup server fails
EMS continues to function but with very degraded performance (59 second refresh)FE system data passed normally to others: MISO and AEPAGC function degraded and strip charts flat-lined
● 15:08 IT warm reboot of EMS appears to work but alarm process not tested and still in failed condition
● No contingency analysis of events during the day including loss of East Lake 5 and subsequent line trips
16
Star- S. Canton (2:27:15)
9
17
Chamberlin-Harding (3:05:41)
18
(3:05:41) Hanna-Juniper(3:32:03)
10
19
Hanna - Juniper Tree ContactInsufficient Clearance with Trees
20
Chamberlin-Harding Indication of Ground Fault Due to Tree Contact as Measured by DFR at Juniper
11
21
Effects of Ambient Conditions on Ratings
22
(3:05:41)(3:32:03)
Star- S. Canton (3:41:35)
12
23
Actual Loading on Critical Lines
0
400
800
1200
1600
12:00 13:00 14:00 15:00 16:00
Time - EDT
Flow
s (M
W)
Harding - ChamberlinHanna - JuniperStar - South CantonSammis - Star
East Lake 5 Trip
Harding - Chamberlin Line Trip
Hanna - Juniper Line Trip
Sammis - Star Line Trip
Star - South Canton Line Trip
24
Situation after Initial Trips 3:05:41 – 3:41:35
13
25
Actual Voltages Leading to Sammis-Star
250
270
290
310
330
350
370
15:00 16:00Time - EDT
Vol
tage
(kV)
Star
Hanna
Beaver
Perry
100% Voltage
95% Voltage
90% Voltage
Sammis - Star 345 kVLine Trip
Star - South Canton345 kV Line Trip
Hanna - Juniper345 kV line Trip
Harding - Chamberlain345 kV Line Trip
26
Phone Calls to FirstEnergy● FE received calls from MISO, AEP, and PJM
indicating problems on the FE system but did not recognize evolving emergency
14:32 AEP calls regarding trip and reclose of Star-S. Canton15:19 AEP calls again confirming Star-S. Canton trip and reclose15:35 Calls received about “spikes” seen on system15:36 MISO calls FE regarding contingency overload on Star-Juniper for loss of Hanna-Juniper15:45 FE tree trimming crew calls in regarding Hanna-Juniper flashover to a treePJM called MISO at 15:48 and FE at 15:56 regarding overloads on FE system
14
27
138 kV Lines Overload and Cascade Near Akron
Simulated 138 kV Line Loadings
0
20
40
60
80
100
120
140
160
180
200
Outages
% o
f Nor
mal
Rat
ings
(Am
ps)
Dale-W.Can138 kV
W.Ak-PVQ22 138 kV
Cham-W.Ak138 kV
E.Lima-N.Fin 138 kV
CantC Xfmr
W.Ak-PVQ21 138 kV
Babb-W.Ak138 kV
E.Lima-N.Lib 138 kV
Clov-Torrey138 kV
Dale-W
.Can
138 kV
W.A
k 138 kVB
kr Failure
E.Lima-N
.Fi n138 kV
CantC
Xfmr
W.A
k-PV Q21
138 kV
Babb-W
.Ak
138 kV
E.Lima-N
.Lib138 kV
Clov-Torrey
138 kV
Star-S.Cant
345 kV
Hanna-Jun
345 kV
Hard-C
hamb
345 kV
Cham
-W.A
k138 kV
28
15:05:41 EDT
15:32:03 EDT
15:41:35 EDT
15:51:41 EDT
16:05:55 EDT
0
20
40
60
80
100
120
140
% o
f Nor
mal
Rat
ings
Canton C
entral Transformer
Babb-W
.Akron 138 kV H
arding-C
hamberlin
Hanna-
Juniper
Star-S.Canton
Cloverdale-Torrey 138 kV
E.Lima-N
ew Liberty 138 kV
W.A
kron-Pleasant Valley 138 kV
E.Lima-N
.Finlay 138 kV
Cham
berlin-W.A
kron 138 kV
W.A
kron 138 kV Breaker
Dale-W
.Canton 138 kV
Sammis-Star
138 kV Cascade Contributes Furtherto Overload of Sammis-Star
15
29
Sammis-Star(4:05:57.5)
30
Sammis-Star Zone 3 Relay Operateson Steady State Overload
16
31
Last Major Path to Cleveland Blocked after Loss of Sammis-Star 4:05:57.5 PM
32
Blackout Root Cause Group 1FirstEnergy Lack of Situational Awareness● Did not ensure a reliable system after
contingencies occurred because it did not have an effective contingency analysis capability
● Did not have effective procedures to ensure operators were aware of the status of critical monitoring tools
● Did not have effective procedures to test monitoring tools after repairs
● Did not have additional high level monitoring tools after alarm system failed
17
33
Blackout Out Root Cause Group 2FirstEnergy Ineffective Vegetation Management
● Did not adequately manage ground clearance (tree clearance) in its transmission rights of way
34
Blackout Cause Group 3Reliability Coordinator Ineffective Diagnostics
●Reliability Coordinator (MISO for FE)State estimator failed due to a data error.Flowgate monitoring tool didn’t have real-time line information to detect growing overloadsOperators couldn’t easily link breaker status to line status to understand changing conditions.Did not declare emergency or take any action
●PJM & MISO ineffective procedures & wide grid visibility to coordinate problems affecting their common boundaries
18
35
345 kV Lines Trip Across Ohio to West4:08:59 - 4:09:07 PM
ONTARIO
36
Generation Trips 4:09:08 – 4:10:27 PM
ONTARIO
19
37
345 kV Transmission Cascade Moves North into Michigan 4:10:36 – 4:10:37 PM
38
Northern Ohio and Eastern Michigan Served Only from Ontario after 4:10:37.5 – 4:10:38.6 PM
20
39
Power Transfers Shift at 4:10:38.6 PM
40
NY to Ontario 345kV Line Flows at NiagaraProgressively Worsening Stability Conditions
New York to Ontario 345 kV Line Flow at Niagara(does not include 230 kV line flow)
0.00
200.00
400.00
600.00
800.00
1000.00
1200.00
1400.00
1600.00
1800.00
2000.00
16:0
5:43
16:0
5:49
16:0
5:54
16:0
6:00
16:0
6:05
16:0
6:11
16:0
6:16
16:0
6:22
16:0
6:27
16:0
6:33
16:0
6:38
16:0
6:44
16:0
6:49
16:0
6:55
16:0
8:55
16:0
9:00
16:0
9:06
16:0
9:11
16:0
9:17
16:0
9:23
16:0
9:28
16:0
9:33
16:0
9:39
16:0
9:45
16:0
9:50
16:0
9:56
16:1
0:01
16:1
0:20
16:1
0:26
16:1
0:31
16:1
0:37
16:1
0:42
16:1
0:48
MW
200.0
220.0
240.0
260.0
280.0
300.0
320.0
340.0
360.0
380.0
KV
Niagara KV
PA301&2 MW
Sammis-Star 345 trip
East Lima - Fostoria Central 345 trip
Thetford-Jewel, Hampton Pontiac,& Perry - Ashtabula 345kV lines trip
Argenta - Battle Creekdouble circuit 345 trip
21
41
Power Surge on PJM – NY Ties 4:10:39 PM
42
PJM – NY Separating 4:10:44 PM
22
43
Cleveland – Toledo Island 4:10:39 - 4:10:46 PMCleveland Blacks Out
44
Eastern Eastern Michigan (Detroit) UnstableVoltage and Frequency Collapse and Pole Slipping
Ontario – Michigan Interface Flow and Voltages Beginning 16:10:38
23
45
Severe Under Frequency Condition
46
View Into Detroit from Lambton
24
47
Northeast Completes Separation from Eastern Interconnection 4:10:43 – 4:10:45 PM
North of LakeSuperior
48
Power plants affected
1. 531 units shut down at 263 plants2. During the conclusion of the cascading
failure, generation tripped off in three general categories:1. Excitation system overload or
extreme low voltage – 35%2. Generator protection or control
system action – 34%3. Consequential result of the broken
transmission system – 31%3. Some prolonged out-of-step conditions
are evident4. To date, little damage has been
discovered as a result of the cascade
25
49
Frequency in Ontario and New York
50
Island Breaks Up: 4:10:46 – 4:13 PM
26
51
Areas Affected by the BlackoutService maintained
in some area
Some Local Load Interrupted
End of the Cascade
52
When the Cascade Was Over50 million people8 states and 2 provinces60-65,000 MW of loadinitially interrupted− Approximately 11% of Eastern Interconnection
Sammis – Star trip at 4:06 PM – Blackout essentially complete by 4:13 PMHigh speed cascading lasted approximately 12 secondsThousands of discrete events to evaluate− Time stamping - critical
27
53
Violations of NERC Reliability Standards
● FE did not return the system to safe operating state within 30 minutes (OP-2)
● FE did not notify others of impending emergency (OP-5)
● FE did not have effective monitoring capability (OP-5)
OperatingPolicies
● FE did not adequately train operating personnel for emergency response (OP-8)
● MISO did not notify others of impending emergency (OP-9)
54
Other Key Findings of Investigation●Compliance with
reliability rules requires objective measurements and firm actions to resolve violations
●NERC policies were not sufficiently specific regarding reliability coordinator and control area functions, responsibilities, authorities, tools
●Problems from prior wide-area blackouts are being repeated: trees, operator tools, training
28
55
Other Key Findings of Investigation● System planning and design
studies, operations planning, facilities ratings, and modeling data accuracy were ineffective preparations for 8/14 event
● Power system in northeastern Ohio was being operated with insufficient reactive margins to meet NERC criteria
● Protection and controls could be more effectively used to slow or minimize spread of cascade
56
Corrective Actions - FE
●Voltage criteria and reactive resources
●Operational preparedness and action plan
●Emergency response capabilities and preparedness
●Control center and operator training
Juniper 345kV
-1000
-900
-800
-700
-600
-500
-400
-300
-200
-100
0
100
200
300
400
500
0.7 0.8 0.9 1 1.1
Voltage PU
Reac
tive
Pow
er (M
VAR)
15-05 Chamberlin-Harding
15-32 Hanna-Juniper
15-41 Star-S Canton
15-45-40 Canton Central-Tidd
15-59 West Akron 138kV Lines
29
57
Corrective Actions – Reliability Coordinators
●PJMCommunications protocols and procedures
●MISOReliability toolsVisualization toolsOperator trainingCommunications protocols and proceduresOperating agreements
58
NERC Strategic Initiatives
● Performance reviews● Readiness audits● Vegetation-related
outage reporting● Recommendations
implementation tracking
30
59
NERC Technical Initiatives - 1● Operator and reliability
coordinator emergency response training
● Reactive power and voltage control
● Cascade mitigation● Reliability coordinator
and control area functions, authorities, and requirements
● Real-time operating tools● Restoration review
40 20 0 20 40 60
20
40
60
80
100
Sammis-Star 345kV @ Sammis 1606:03110
10−
Im ZBCx( )Im Z1l( )
7040− Re ZBCx( ) Re Z1l( ),
60
NERC Technical Initiatives - 2
● Time-synchronized measurements for disturbance analysis and operations
● Reevaluate system design, planning and operating criteria
● System modeling and data exchange standards