Power Grid Research at Pacific Northwest National Laboratory

Moe KhaleelLaboratory Fellow and Director

Computational Sciences and Mathematics DivisionSeptember 10, 2009

HPC User Forum

1

Overview

IntroductionProblems on the electrical gridNeed for HPC on the electrical grid

• Operations• Integration of renewables• Cyber security

• Need for HPC at vision level– State estimation– Multithreaded platforms for contingency analysis

• Looking forward

Transform the way the U.S. generates, transmits, distributes and uses electricity

• Current U.S. electricity infrastructure is inadequate for national energy priorities in the 21st century. Three main areas need to be addressed:

– Capacity– Grid management (wide area, real-time)– Vulnerability, resiliency, reliability

• The future state of the grid must be able to:– Substantially increase the integration of renewables – Reduce carbon emissions– Provide flexibility to enable electrification of transportation and reduce dependence on

oil imports (substitute electricity for oil)– Respond to increased demand

• Reality: The current grid infrastructure/operation is limited– Currently manage/engage grid at sub-optimal (service territory) level; can’t efficiently

move electrons across large enough spaces– No ability to see performance across grid (lacks transparency)– Inability to integrate renewables; need storage, ability to offset, transmission across

service territories where generated, load/dispatch renewables– System communication has been one-directional: supply to demand– Fragmented authority, control, market, function, regulation

• Need to build new functionality and infrastructure into the grid• Future grid must be transformed while maintaining reliability

and affordability (serving public good)3

Transform the way the U.S. generates, transmits, distributes and uses electricity

• Capacity: Transmission infrastructure cannot meet future load growth and large-scale renewable connectivity to grid

– Utilities not incentivized to build physical infrastructure– Difficult to site & permit new transmission infrastructure – Renewable resource physically isolated from high grid transmission infrastructure

• Grid Management: Unable to manage grid at national, interconnect scale

– Large scale models that allow examination and optimization of future national grid do not exist

– Integrated wide area models (variable renewable generation, energy storage, distributed generation, demand management) that describe real-time power flow and predict reliability do not exist

– Ability to see and understand the grid at interconnections scale are limited; wide area grid performance is not accessible, transparent so can’t optimize supply and demand across limited service areas

– Transparent real-time monitoring and operation currently not in place– Large-scale wind generation introduces significant variability– Large scale electric energy storage capability is limited – pumped hydro, flywheels,

electro-chemical systems connected to and supporting the power grid

• Vulnerability and Resiliency– Susceptible to cyber, other threats – can we prevent, respond to threats?

Resilient to catastrophic events – can we rapidly recover?

4

Power System Elements

Problems on the electrical grid

• Inadequacy of current control center functions– Slow, not able to keep up with the change of the grid– Static, no dynamic information for real-time operations

• Computational Issues with today’s grid operations– Real-time grid view: static– Not able to capture grid dynamics – Low computational efficiency: not keep up with system

changing – No use of high-performance computing architectures

Problems on the electrical grid

The need for HPC on the electrical grid

• Major HPC Architectures – Shared-memory architecture for extensive data sharing and

un-uniform data access– Distributed-memory architecture for less data sharing and

uniform data access– Hybrid architecture - re-configurable architecture: e.g. FPGA

+ shared-memory

• Performance of Parallel Algorithms/Programs– Speedup/Scalability– Reliability

The need for HPC on the electrical grid

• Parallel Computing is essential – Only explicitly parallelized algorithms can take advantage of

multi-core parallel computers

• Parallel Computing is “an art”– Parallelization approaches are problem-dependent – Computing implementation needs to consider “good match”

of computing architecture and the problems

Today’s Electrical Grid Operations Paradigm

SCADA

State Estimation

Contingency Analysis

Operator

Market Operation

~ minutes ~ minutes

Violations

Constrained solutions

~ seconds

Normal operations• Asset underutilization • Limited market opportunities • Lead to emergency operations

Emergency operations• Blackouts and cascading failures

slow static

~ hours

Off-line Transient/VoltageStability Analysis

Ratings & Limitsseasonal

Trends Impacting Control System Security

• Open Protocols– Open industry standard protocols are replacing vendor-

specific proprietary communication protocols• General Purpose Computing Equipment and Software

– Standardized computational platforms increasingly used to support control system applications

• Interconnected to Other Systems– Connections with enterprise networks

to obtain productivity improvementsand information sharing

• Reliance on External Communications– Increasing use of public telecommunication

systems, the Internet, and wireless for controlsystem communications

• Increased Capability of Field Equipment– “Smart” sensors and controls with enhanced

capability and functionality

The Emerging Cyber Threat

• Industry has long history of planning for and coping with natural disasters and other reliability events

– Through industry standard operating procedures, there is much effort expended to reduce likelihood of cascading outages leading to widespread blackouts

• Historically, cyber security focused on countering unstructured adversaries

– e.g., individuals, untargeted malicious software, human error

• Very little protection against structured adversaries intent on exploiting vulnerabilities to maximize consequences

– e.g., terrorist groups, organized crime, nation states– Insider threat remains very challenging, can be used as part of structured

threat vector

• New possibilities for widespread sustained outages resulting from cyber attack are now being contemplated

– But industry still not ready to cope with this threat

The need for HPC: State Estimation

• Power system State Estimation (PSE)– Given: power grid topological information, telemetry on line

flows, bus injections or bus voltages– Compute: a reliable estimate of the system state (bus

voltages), validate model structure and parameter values– Calculated using Weighted Least-Squares (WLS) method– WLS: minimize– Where r = z - h(x), and r is the residual vector, x is the

system state, z is a vector of measured quantities, h is a vector function, wi is the weight for residual ri and W is as diagonal matrix.

– This is a non-linear problem, which is solved using the Newton-Raphson iterative procedure

The need for HPC: State Estimation

• PSE– Every iteration of the method requires solving a large set of

sparse linear equations– Sparse matrices are derived from the topology of the power grid

being analyzed– The number of non-zeros per row varies greatly and the matrix is

badly conditioned– The set of linear equations can be solved using direct solvers

such as sparse LU factorization or iterative solvers such as sparse Conjugate Gradient (CG)

• PSE is a critical element of the software used by power grid control centers

– Under real-time constraints (< 10 seconds)– Commercial PSE solvers are not commonly parallel

Power System State Estimation

August 13, 2003Normal

August 14, 2003Blackout

Situational Awareness?

Source: NOAA/DMSP

Do we know what really happened?Could it be prevented?

Model Validation Need and Challenges

Reality

Model

Recorded system dynamics vs. simulation results: California and Oregon Intertie (COI) real power flow during the August 10, 1996 event

The need for HPC: Multithreaded Platforms for Contingency Analysis

• Growing class of scientific applications is becoming memory-bound

• Many scientific applications exhibit irregular memory access patterns

• CPU and memory technology trends indicate that the situation will not improve anytime soon

• Multithreaded architectures offer an appealing alternative for irregular applications

– Processors tolerate memory access latencies by switching execution context between multiple hardware threads

– Examples of such architectures are the Cray MTA-2 and XMT systems and the Sun Niagara

– Latency tolerance mechanisms should improve the performance of irregular, data-intensive applications with abundant fine-grained parallelism

Role of Contingency Analysis

• From “N-1” to “N-x”– To improve situational awareness

• From Balancing Authorities to a Wide Area

– Example: 35 BAs in west– Further require “N-x” CA– To better understand cascading

failures

• N-x Contingency AnalysisResult in a large number of cases. “N-5” 1020 cases for the west =~ 1020 seconds + lots of data

– Needs: better contingency selection and post-processing

Looking forward: PNNL’s vision of advanced contingency analysis

Industry Need:

Issue:

Need:

Graph Theory

Contingency Index Sorting

10Z cases 10Y cases 10X casesContingency

AnalysisData

Time

past now future

Pre

dic

tio

n

Trending

Visual Analytics& Graph Trending

Interactive Graphing

Multi-failure (N-x) massive contingency analysis

Massive number of cases

Smart case selection

Massive amount of data

Operator-oriented data processing

Our Solution:

Major Tasks:

ExpectedOutcomes:

• An advanced tool for contingency analysis to support the operation of the nation’s critical infrastructures, e.g. power grids

• Analysis capabilities through generic implementation of algorithms on Cray XMT ThreadStorm processors, applicable to other problems

Tool Suites for Advanced Power System Simulation

Parallel contingency analysis

Dynamic state estimation

Look-ahead dynamic simulation

Market Monitoring

Parallel state estimation

On-line voltage stability

SCADA Measurements

Phasor Measurements

Dyn. contingency analysis

Applicable to current grid operations Next-generation grid operation tools

Visualization

National Level: monitoring, planning, design Utility Level

Looking forward: PNNL’s long-term vision for electrical grid operations

SCADA & Phasor Dynamic

State Estimation

Dynamic CA,On-line TSA/VSA

Operator

Real-TimeOptimized Market

Operation

~ (sub)seconds~ (sub)seconds

< 1 hour

Violations

Constrained solutions

~ (sub)seconds

DynamicRatings & Limits

Normal operations• Optimized asset utilization • Enabled regular & ancillary markets• Predict emergencies

Emergency operations• Prevent/mitigate failures

fastdynamic

22

Use HPC to analyze sensor data for real-time grid monitoring, prediction, and operation: data-driven models, stochastic simulations, visual analytic tools, secure sensor network infrastructure, data provenance

Electricity Infrastructure Operational Center at PNNL

QUESTIONS?