Installing Modern Data Networks in Electrical · Availability for SCADA and Engineering Access...

Copyright © SEL 2009

“Selecting, Designing, and

Installing Modern Data

Networks in Electrical

Substations”

Gary W. Scheer, David J. Dolezilek

Schweitzer Engineering Laboratories, Inc.

Copyright © SEL 2009Copyright © SEL 2009

Scientific Comparison of

Communication Topologies“Selecting, Designing, and Installing Modern

Data Networks in Electrical Substations”

Gary W. Scheer, David J. Dolezilek

Schweitzer Engineering Laboratories, Inc.


The Specifications of IEC 61850 Part 3

“… pertain to the general requirements of

the communication network, with

emphasis on the quality requirements.

It also deals with guidelines for

environmental conditions and auxiliary

services ...”


IEC 61850-3 Section 4 Describes

SAS Quality Requirements

“This clause details the quality

requirements such as

reliability – availability – maintainability –

security – data integrity – and others

that apply to the communication systems

that are used for monitoring,

configuration, and control of processes

within the substation”


IEC 61850 Describes “Graceful

Degradation Principle”

“There should be no single point of failure that

will cause the substation to be inoperable.

Adequate local monitoring and control shall be

maintained.

A failure of any component should not result in

an undetected loss of functions nor multiple

and cascading component failures”


“A Fail-Safe Design Shall Be

Provided (i.e., Is Required) …

“… There shall be no single failure mode

that causes the SAS to initiate an

undesired control action, such as tripping

or closing a circuit breaker.

In addition, SAS failures shall not disable

any available local metering and local

control functions at the substation.”


System Quality Includes Repair Travel

Time; System Design Quality Does Not

MTTR: Mean Time to Detect and Repair Failure

MRT: Mean Repair Time (after self-test alarm;

vendor has no influence time to detect)

MTTF: Mean Time to Fail

MTBF: Mean Time Between Failures

(MTTR + MTTF)


Population of 300 devices

Three failures in one year

Device failure rate = 3/300

= 0.01 failures/year

Device MTBF = 300/3

= 100 years

Device Reliability Quantified by Failure

Rate and MTBF


Failure, Repair, and UnavailabilityMTTR = MRT Plus Detection & Travel Time

1

MTBF

qMTTR

MTBFMTTR

Failure Rate

Unavailability

MTTR

MTBF

MTTF


Unavailability and Availability

Unavailability: probability that device

cannot perform defined task(s)

Unavailability = MTTR ÷ MTBF

Availability = 1 – Unavailability


Example Availability Calculation for

Communications Processor

q = 16 • 10-6 Unavailability

1–q = 99.9998% Availability

day / hours 24 • / yeardays 365 • years335

hours48q


99% Availability Perspective

20,000 lost articles mail/hour

Unsafe drinking water 15 min/day

5,000 incorrect surgical operations/week

200,000 incorrect prescriptions/year


99% Availability Perspective

Two short or long landings

major airports/day

No electricity 7 hours/month


IEC 61850 Refers to These Quality

Measures Described in IEC 60870-4

Reliability class severity (R1, R2, or R3)

measured as MTBF

Availability class severity (A1, A2, or A3)

measured as % availability

Maintainability class severity (M1, M2,

M3, or M4) measured as MTTR and

(RT1, RT2, RT3, or RT4) as MRT


Fault Tree Method Uses Device Quality

Measures to Evaluate Systems

Pick top event – “failure of system to

meet specification”

OR gates combine events that can

singly cause failure

AND gates combine events that must

happen together to cause failure

Review for common causes of failure


Network Fault Tree Example

Unavailability of Integration or

Protection of Networked IEDs

Ethernet

Switch

Information

Processor

Ethernet

Interface

Relays


True Objective of IEC 61850 Part 3

Use data to engineer the best SAS

system to provide information about and

influence the state and history of a real-

world system

Acceptance criteria include

reliability – availability – maintainability –

security – data integrity – and others

We also add cost and performance


Networked IED SAS Topology

Design Requirements Summary


Ethernet Networked IED Systems

Require an Information Processor

Many functions handled

directly peer-to-peer

Unlikely that others will

be completely

distributed

Sequential controls

Synchronization

Load shedding

Voltage control

Information

Processor


DNP LAN/WAN, IEC 61850, Others,

Require an Information Processor in

Every System

Bandwidth concerns

More data

Need storage location

for bulk and historical

data

Need to communicate

data subset to SCADA

systems

Information

Processor


Integration With Networked IEDs

IEDs

Information Processing and

Communications

Engineering

Access SCADA

Master

Accurate

Time

Voltage Regulator

Control

Recloser

Control

Protective

Relays


Information Processor Functions

Act as upstream gateway to SCADA and other

enterprise applications

Perform channel diagnostics and visualize

parameters for troubleshooting

Support multimedia connections – copper,

fiber, and wireless

Perform protocol conversion between clients,

servers, information processors, and IEDs



Concentrate data extracted from IEDs

Filter out unnecessary data

Combine data from multiple IEDs into data sets

Support interleaved conversations

Segregate IEDs

Each IED receives and processes only the

information addressed to it

Frees IED from unnecessary processing



Prioritize important protection and

automation messages to assure rapid,

deterministic delivery

Store and forward all received message

traffic to eliminate message collisions and

assure consistent delivery of data


Serial or Other Information Processing

Solutions in a Single Box

Serial

Serial IED-to-IED Links

Ethernet

Master

Router

HMI

Relay RelayRelay

Information

Processor

Communications Processor

or Rugged Computer

Upstream Gateway

Channel Diagnostics

Multimedia

Protocol ConversionData Concentration

Rugged Construction

Interleaved Conversations

Segregation

PrioritizationStore and Forward


Ethernet Requires Installing and Setting

Two Boxes for Information Processing

Upstream Gateway

Channel DiagnosticsMultimedia

Protocol Conversion

Data Concentration

Rugged ConstructionInterleaved Conversations

Rugged ConstructionInterleaved Conversations

SegregationPrioritization

Store and Forward

Communications Processor or

Rugged Computer

Ethernet Switch

Serial

GOOSE

Ethernet

Router

Master

HMIInformation

Processor

relay relay relay

Ethernet Switch


Changes to Ethernet Since 2000

Message prioritization via IEEE

802.1p improves delivery of

important messages

Store and forward in switches

reduces data loss collisions

IEEE Standard 1613-2003

provides measure of reliability “Environmental and testing requirements for

communications networking devices in

substations”

Message segregation via IEEE 802.1q improves message delivery and processing efficiency


Alternative Information Processor

Architectures

Information Processing and Communications

Protective

Relays and

Other IEDs

Protective

Relays and

Other IEDs

Engineering

Access

SCADA

Master

Accurate

Time

Voltage Regulator

Control

Recloser

Control

Select

Serial Star Network OR Ethernet Star Network

Communications

Processor

Router

Ethernet Switch

Intelligent Server

Router


Topology Comparison Criteria

Reliability

Cost of equipment, installation, and

commissioning

Effective data transfer rates

Ease and cost to design, implement,

maintain, and expand


Topology Comparison Criteria

Flexibility to choose best IED for each

job, unconstrained by network

Performance of high-speed control

Ease and cost of using existing devices

and designs in new network


Three Popular Network Topologies

Serial SELSerial Hybrid

SEL, SCADA

Ethernet Hybrid

SEL, IEC 61850, SCADA

MIRRORED BITS

Router

HMI

Relay RelayRelay

Master

Information

Processor

MIRRORED BITS

Master

Router

HMI

Relay RelayRelay

Information

Processor

MIRRORED BITS and GOOSE

Router

Master

HMIInformation

Processor

Relay Relay Relay

Ethernet Switch


Serial Topology With SEL Protocols

Functionality

SEL

“SMART”

Serial

Information Processing

Capabilities10 / 10

IED Networking Capabilities 15 / 15

MIRRORED BITS

Master

Router

HMI

Relay RelayRelay

Information

Processor

Serial Ethernet


Serial Topology With Combination of

Protocols

FunctionalitySCADA

Serial

SMART and

SCADA Serial


Capabilities8 / 10 10 / 10

IED Networking


MIRRORED BITS

Master

Router

HMI

Relay RelayRelay

Information

Processor

Serial Ethernet


Ethernet Topology With Combination

of Protocols: SEL, SCADA, Both

FunctionalitySCADA DNP3/

LAN/WAN, 104

SMART

&

SCADA

Information

Processing


IED Networking



Router

Master

HMIInformation

Processor

Relay Relay Relay

Ethernet Switch

Serial Ethernet


Ethernet Topology With Combination

of Protocols: SEL, IEC 61850, Both

FunctionalityIEC

61850

SMART &

IEC 61850



IED Networking



Router

Master

HMIInformation

Processor

Relay Relay Relay

Ethernet Switch

Serial Ethernet


Example Substation One-Line


Compare 12-Relay Topologies

Serial SEL Serial Hybrid

SEL, SCADA

Ethernet Hybrid

SEL, IEC 61850, SCADA

RouterProtocol

Converter

HMI

Master

Information

Processor

relay relayrelayrelay relayrelayrelay relayrelayRelay RelayRelay

Master

Router

HMIInformation

Processor


Router

Master

HMIInformation

Processor

Ethernet Switch



Protective Relays for Example

Station

Relay Quantity

Line Distance 2

Line Current Differential 2

Transformer 2

Bus 2

Feeder 4

Total 12


Approximate Unavailabilities of

Substation-Grade Devices

Component Unavailability (multiply by 10-6)

IED Network Interface 4

Fiber-Optic Transceiver 9

Communications Processor 16

Protective Relay 37

Dual Power Supply Switch 52

Switch 96

Rugged Computer 110

Dual Power Supply Router 156


Serial Star Network Fault TreeNote: Multiply all

unavailabilities by 10-6

Unavailability SCADA or

Engineering Access Functions

444

Router Comm.

Proc.

Ethernet

Interface

Relays

137 16 412 x 37

601

Master

Router

HMIInformation

Processor



Switched LAN Fault Tree

Note: Multiply all

unavailabilities by 10-6

Unavailability SCADA or


444

Router Information

Processor

SwitchRelays

137 110 5712 x 37

Ethernet

Interface

12 x 4

48

796

Router

Master

HMIInformation

Processor

Ethernet Switch



Availability for SCADA and


Alternative Availability %Predicted Annual

Hours Out of Service

Serial Star

Communications

Processor

99.9401 5.2

Serial Star Rugged

Computer99.9315 6.0

Ethernet LAN With Dual

Power Supply Switch99.9214 6.9

Ethernet LAN With

Single Power Supply

Switch

99.9169 7.3


Compare Three Integrated Systems Reliability, Speed

SEL-2032 SerialSEL Protocols

SEL-3332 SerialSCADA & SEL Protocols

SEL-3351 EthernetIEC 61850 & SEL Protocols

System MTBF 23 yrs 16.7 yrs 11 yrs

Expected

Annual

Downtime

115 min 164 min 276 min

Peer-to-Peer

Speed

3 ms

MIRRORED BITS

3 ms

MIRRORED BITS

4 ms

SEL GOOSE

Based on 12 Relays


Compare Three Integrated Systems

Information Processors

SEL-2032

SerialSEL Protocols

SEL-3332 Serial SCADA & SEL Protocols

SEL-3351 EthernetIEC 61850 & SEL Protocols

IED Integration

Engineering Effort24 hrs 36 hrs 36 hrs

SEL Hardware /

OS Warranty10 yrs 10 yrs 10 yrs

Logic Processing

Speed

SELOGIC®

every 16 ms

Calculator every

1000 ms

Calculator every

1000 ms

Size of OS and

Associated Code2 MB 575 MB 1600 MB

Based on 12 Relays


Availability for Relay-to-Relay

Communications for Six Relays

Alternative Availability %Predicted Annual

Hours Out of Service

Serial Star

Communications

Processor

99.9781 1.9

Ethernet LAN 99.9660 3.0


Relay-to-Relay CommunicationsComplexity Drives Maintainability

Serial Point-to-PointSwitched Ethernet

LAN

Complexity: Lines of

Code (LOC)460 8020

Complexity: Message

Size for 1 Bit Change4 Bytes 200 Bytes


SELSerial SCADA

ProtocolIEC 61850

Peer-to-Peer

Configuration

Engineering Effort

4 Settings

MIRRORED

BITS

4 Settings

MIRRORED BITS

28 Settings

IEC

GOOSE

Peer-to-Peer Channel

MonitoringYes Yes

Custom

SELOGIC

Peer-to-Peer Payload

Repetition SecurityYes Yes

Custom

SELOGIC

Source of Peer-to-Peer

ProtocolSEL SEL

SISCO

Tamarack

Number of Installed

Systems15,000 >300 >10

Compare Three Integrated Systems Peer-to-Peer Messaging


Comparative Costs ($)

Serial Star

Comm. Proc

Serial Star

Rugged

Computer

Switched

Ethernet LAN

Initial Equipment 74,000 76,000 173,000

Ten-Year Repair 1,380 1,380 147,000

SCADA Gateway

Engineering Effort1,760 2,640 21,120*

IED Integration

Engineering Effort2,640 3,960 3,960

Total Protocol,

Mapping, Integration4,400 7,700 35,200

*Reflects mapping to protocol without matching data types. Will improve with

improvement in software drivers and tools

Installing Modern Data Networks in Electrical · Availability for SCADA and Engineering Access...

Documents

Transcript of Installing Modern Data Networks in Electrical · Availability for SCADA and Engineering Access...