1

The Communication Backbone for

Substation Automation and IEC61850

René MidenceRuggedCom Inc.

Vice President, Marketing

Phone: +1 905 266 11 39

Mobile: +1 647 504 58 15

E-mail: ReneMidence@ruggedcom.com

2

Agenda

• Background: Substations Past and Present

• Ethernet Refresh

• Switching Refresh

• Sources of Latency in a Switched Ethernet

• IEC 61850 Core Components, Architecture &

Communication Model

• IEEE 802.1Q VLANs

• User-Priority / Classes of Service / IEEE 802.1P

• Example

3

Substation Network - Past

4

Substation Network - Present

5

Typical Substation DiagramSub

Transmission

Line No. 1

Sub

Transmission

Line No. 2

Power

Transformers

Tie Breaker

Feeder Breakers

IED 4

TX 1

IED 1

LN 1 IED 3

TB 1

IED 2

LN 2

IED 5

TX 1

IED 6

FDR 1IED 7

FDR 2IED 8

FDR 3

IED 9

FDR 4IED 10

FDR 5IED 11

FDR 6

IED 12

TB 2

IED 14

BUS 2

IED 13

BUS 1

Wiring requirements:

• CTs – Total 22 x 4c cables– 2 x Transmission Lines – Distance

Protection

– 2 x HV Busbar Protection

– 2 x Bus Tie – Busbar Protection

– 2 x 2 Step Down Transformers

– 6 x Feeder Protection

– 6 x LV Busbar Protection

– 2 x LV Tie Breaker

• VTs – Total 8 x 2c cables; 4 x 4c Cables

– 2 Single Phase – Transmission Lines

– 2 x Three Phase HV Bus

– 2 x Three Phase LV Bus

– 6 x Single Phase – LV Feeders

• Circuit Breakers Control – Total 14 x

12c cables– 2 x Transmission Lines

• 1 x HV Bus Tie– 2 x HV Step Down Transformers

– 2 x LV Step Down Transformers

– 6 x LV Feeders

– 1 x LV Tie Breaker

• Power Transformer Control – 2 x 12c

Cables

• Communication cables between relays

and RTU or PLC – Total 14 cables Total: At least 50 multi-core cables from

the HV equipment to the Control Building

6

Substation Network Future

7

Configuring the LAN for IEC 61850

8

Ethernet Refresh

• Ethernet

– Initially: shared media (collisions)

– MAC (Media Access Control) protocol based-on CSMA/CD

(Carrier Sense Multiple Access with Collision Detection)

– Contention-based

• Set of devices that contend for access to a segment are

in the same collision domain

– Bus network

– Hub Network

9

Ethernet Refresh:MAC -Addressing

• Ethernet also defines framing of messages

– Multiple Access network: addresses required

• Destination / Source

• 3 ways of delivering messages:

– Unicast (one): 00-1B-5B-D1-4A-F5

– Multicast (more than 1): 01-80-C2-00-00-00

– Broadcast (more than 1 All): FF-FF-FF-FF-FF-FF

10

Where Are We?

• Communication between IEDs is necessary

• Ethernet has many advantages over serial

communications

– Shared medium means less predictability: collisions

11

Switches Help

• Switches helped by breaking-up collision domains

– Node connected directly to a switch, doesn‟t contend for access

to media

– No collisions, (more predictable)

12

Switching Refresher

• Switches

– Work at L2 (Data-Link Layer)

– Store and then forward received frames to the appropriate

interface

– Maintain a forwarding database (switching table) containing

• MAC Addresses (00-1B-5B-D1-4A-F5)

• Port numbers (Port 1)

• Additional information (VLAN ID)

– Can buffer traffic should the egress port be busy

13

Switching Refresher

– Optimize traffic flows by Learning: the location of nodes

dynamically.

• Specifically, they learn the Switch Address and associate it with the

ingress port

– If the address is unknown it gets flooded out all ports, except the

one it was received on

– If the address is known (in switch Data Base), it gets forwarded

to the appropriate port

• Also considered filtering, since only the port to which the destination

host is connected will receive this frame

14

Switching Refresher

• 1 Broadcast domain per switch (~ per VLAN)

– Broadcast frames are flooded out all ports (in the same VLAN)

– Multicast frames are flooded out all ports (in the same VLAN)

• Unicast frames:

– flooded (if the destination MAC address is not in the MAC Data

Base)

– forwarded to only one port (if the destination MAC address is in

the MAC Data Base)

15

Where Are We?

• Communication between IEDs is necessary

• Ethernet has many advantages over serial

communications

– Shared medium means less predictability: collisions

• Switches improve Ethernet performance by eliminating

collisions

16

Switched Ethernet

• For some applications, the network must guarantee a

certain maximum latency or the application may not

behave (or fail outright)

• Switched Ethernet Networks have many sources of

latency:

– Store & Forward, Switch Fabric, Wireline, Queuing

• All of these latencies (except for queuing) are

deterministic

– The effects of frame queuing can also be calculated providing

one knows the nature of all sources of traffic on the network

17

Who Cares?

• The benefits and drawbacks of Ethernet technology are

important

• IEC Chose Ethernet as the underlying technology for

61850

18

IEC 61850 Primer

• Culmination of:

– Comprehensive EPRI-Project UCA 2.0 &

– European Experience (IEC 60870-5-101, -103, -104)

• Framework for SS communication, high-level description

of SS automation

• Standards required:

– Interoperability & Integration: data representation, how devices

should look / behave to network applications

– Intuitive device/data modeling/naming: Hierarchical and

structured

– Fast and convenient communication

– Lower cost: install, configure, maintain, & also connect legacy

devices

19

Core Components

• Object Model (info available from primary equipment &

SS automation functions)

• Inter-IED Message/communication format

• Environmental robustness

• Configuration language

20

Architecture

• Process Bus: 61850-enabled IEDs get digitized power

grid condition data via MUs

• Station Bus: 61850-enabled IEDs communicate with e/o

using station busses

21

IEC61850 Substation ArchitectureIE

C61850

Pro

cess B

us

IEC

6185

0

Sta

tion B

us

22

IEC61850 Substation ArchitectureIE

C6185

0

Sta

tion B

us

23

Ethernet Packet

Station Bus

Status, Trips, Close,

GOOSE / GSSE

A, V, Hz, W, Wh,

Var, Varh, SOEIED Block Diagram

29

IEC61850 Substation ArchitectureIE

C61850

Pro

cess B

us

30

Ethernet Packet

Process Bus

Merging Unit Block Diagram

I/Os

Process Bus data is used by

other IEDs in the network to perform real time

protection & control mission

critical actions

31

Remarks

• Ethernet has been chosen as the common communications architecture in Substation automation

• The main goal is having interoperability between a variety of substation Intelligent Electronic Devices (IEDs)

• IEC 61850 is the International standard has as its underlying network technology - Ethernet.

• Requirements for Ethernet in the substation environment requires real-time and reliable performance

• Considerations when designing substation automation and networking:– Equipment designed to meet IEC and IEEE standards for

substation hardened equipment

– Type of cables to be used

– Network topology

– Network recovery i.e. rapid spanning tree

– Network security

32

Communication Model

• GOOSE messages

• Used for fast transmission of substation events, such as

commands, alarms, indications

– A GOOSE message sent by an IED can be received several

receivers

Examples:

• Tripping of switchgear

• Providing position status of interlocking

33

Connecting the Two(Ethernet and 61850)

• Details of 61850 irrelevant to switches

• Switches are Interested in the Ethernet part only

• 61850 Architects knew importance of predictable latency

inside the sub

• Also knew that Switched Ethernet unpredictable due to

queuing

34

Answer: Use VLANs

Question:

• If queuing / buffering is important to operation of

Ethernet > How to minimize any adverse effect?

Answer:

• Identify the relative priority of frames

• Increase available bandwidth, reduce cost, improve

security

– Using standard mechanism: IEEE (802.1P/Q > 802.1D-2004)

35

VLANs - Basic Facts

• VLAN: a logical group of ports – on a single device or across multiple switches

– Used construct separate “virtual” networks (independent domains that share cabling and switch infrastructure)

• IEEE 802.1Q defines how to carry multiple VLANs over the same physical link (between switches)

• Broadcasts are contained to a single VLAN

• Since devices in the same VLAN are in the same broadcast domain, they can communicate without a router (L3 device)

• A Layer 3 device is required to communicate between VLANs

36

Cost Effective Solution

37

Relay Meter

Red, Green, and

Blue VLANs are

separate networks

that share the

same

infrastructure.

SCADASurveillance

Billing

Trunk ports

carry traffic

from all

VLANs on

same fiber

MPEG

IEEE 802.1Q VLANs in the Substations

38

Frame formatTagged v. Untagged

Standard

FrameDest. Src. Length / Type Data

6 bytes 6 bytes 2 bytes Variable

Dest. Src. Length / Type Data

6 bytes 6 bytes 2 bytes Variable

TPID TCI

Priority CFI VID

2 bytes

3 bits 1 bit 12 bits

2 bytes

Tagged

Frame

39

Edge port vs. Trunk port

Red

Blue

Green

Green

Red

Blu

e

Green

Red

Blue

Trunk Link (Carries all 3 VLANs)

•Trunk:

• A VLAN „Trunk‟ carries traffic for multiple VLANs over

the same physical link

•A VLAN tag identifies the VLAN membership of the

frames on the trunk

•Trunk ports send tagged frames

•Edge:

•A VLAN „Edge‟ port carries traffic for a single VLAN

•Edge ports send untagged frames

40

IEEE 802.1P Prioritization

• Class of Service (CoS)

• High priority traffic gets to

bypass queue

• Multiple egress traffic

queues: higher priority

traffic sent first

• Reduces jitter and latency

for time-sensitive traffic

(voice / GOOSE)

• Shares tag header with

802.1Q VLAN

41

GOOSE, Priority, and VLANs

• Different traffic flows in the

61850 substation network

merit separate VLANs:

– Substation LAN management

(e.g. switches, routers,

modems, etc.)

– SCADA/Engineering Access

– GOOSE Messages

– Process bus (IEC 61850-9-2

sampled values)

– Synchrophasors

– Protection A vs. Protection B

– Video surveillance and access

control

GOOSE should have the highest Class of Service (CoS) as per IEEE 802.1P

GOOSE is faster than conventional I/O

42

802.1P/Q Summary

• VLANs

– Logical group of ports (Broadcast domain)

– Convey group membership via IEEE 802.1Q “Tag”

– Tag carries VLAN ID and User-priority (they work in tandem)

– Trunk, Tagged

– Edge, Untagged

Standard

FrameDest. Src. Length / Type Data

6 bytes 6 bytes 2 bytes Variable

Dest. Src. Length / Type Data

6 bytes 6 bytes 2 bytes Variable

TPID TCI

Priority CFI VID

2 bytes

3 bits 1 bit 12 bits

2 bytes

Tagged

Frame

43

Where Are We?

• Communication between IEDs is necessary

• Ethernet has many advantages over serial

communications

– Shared medium means less predictability: collisions

• Switches improve Ethernet performance by eliminating

collisions

• Queuing latency minimized with Classes of Service

(CoS/802.1P)

• CoS (priority) information carried in VLAN tag

44

IEC61850 Standard –Multicast Addressing

The multicast (destination) addresses use following structure:

– First 3 Bytes assigned by IEEE: 01-0C-CD.

– 4th Byte shall be: 01 for GOOSE, 02 for GSSE, and 04 for multicast

sampled values.

– Last 2 Bytes to be used as individual addresses assigned by the range

defined in Table B.1.

Source: IEC61850 Standard 61850-8-1

First edition 2004-05

Communication networks and systems in substations – Part 8-1: Specific Communication Service

Mapping (SCSM) – Mappings to MMS (ISO 9506-1 and ISO 9506-2) and to ISO/IEC 8802-3

45

Remember:

• 61850-enabled IEDs use messages to communicate

• GOOSE messages:

– Multicast delivery (flooded by switches)

– So, all nodes see message (regardless of whether it‟s required)

• Want to convey priority to switch

• Not necessarily set VID

• Priority Frame:

46

GOOSE and Network Performance

• Generic Object Oriented Substation Event (GOOSE):

– Very powerful IED to IED messaging mechanism

– Allows efficient real time exchange of any state or analog

parameter.

• IEC 61850-5 Type 1A “Trip” total transfer time at 3ms

• Transfer time = Application to Application and includes

IED times + Network delays.

• Worst case of total network delay is

– 100 µs at 100MBps links speeds

– 10 µs at 1Gbps

• Small networks or for less demanding applications the

delay introduced by switches is tolerable

47

GOOSE and Network Performance

• For larger networks of say twenty switches with 100

Mbps links, a delay of 2 ms could impact the protection

scheme

• GOOSE messages are priority tagged

– Highest Class of Service (CoS – IEEE 802.1P) in the network

– GOOSE frames are placed in the front of the store and forward

queue

– Frames already being sent are not interrupted

• Recommendation:

– Consider Gigabit links for networks running highly demanding

IEC 61850 GOOSE and Sampled Values applications

48

VLAN Advanced - Cost Effective Solution

Sub Transmission

Line No. 1

Sub Transmission

Line No. 2

Power

Transformers

Tie Breaker

Feeder Breakers

IED 4

TX 1

IED 1

LN 1 IED 3

TB 1

IED 2

LN 2

IED 5

TX 1

IED 6

FDR 1IED 7

FDR 2IED 8

FDR 3

IED 9

FDR 4IED 10

FDR 5IED 11

FDR 6

IED 12

TB 2

IED 14

BUS 2

IED 13

BUS 1 VLAN 10 – Transmission +

HV Bus Protection IEDs

VLAN 20 – Distribution + LV

Bust Protection IEDs

VLAN 30 – Bus Bar

Protection IEDs

VLAN 40 – Transformer

Protection IEDs

VLAN 50 – Video

Surveillance

VLAN 1 – Engineering /

Operations / SCADA

49

VLAN Advanced - Cost Effective Solution

Sub Transmission

Line No. 1

Sub Transmission

Line No. 2

Power

Transformers

Tie Breaker

Feeder Breakers

IED 4

TX 1

IED 1

LN 1 IED 3

TB 1

IED 2

LN 2

IED 5

TX 1

IED 6

FDR 1IED 7

FDR 2IED 8

FDR 3

IED 9

FDR 4IED 10

FDR 5IED 11

FDR 6

IED 12

TB 2

IED 14

BUS 2

IED 13

BUS 1

VLAN 10 – Transmission +

HV Bus Protection IEDs

VLAN 20 – Distribution + LV

Bust Protection IEDs

VLAN 30 – Bus Bar

Protection IEDs

VLAN 40 – Transformer

Protection IEDs

VLAN 50 – Video

Surveillance

VLAN 1 – Engineering /

Operations / SCADA

50

Example

IED1 – LN1

IED2 – LN2

IED3 – TB1

IED4 – TX1

IED5 – TX2

IED6 – FDR1

IED7 – FDR2

IED8 – FDR3

IED9 – FDR4

IED10 – FDR5

IED11 – FDR6

IED12 – TB2

IED13 – BUS1

IED14 – BUS2

SCADA

Engineering /

Operations

Video

Surveillance

VLAN 10 – Transmission +

HV Bus Protection IEDs

VLAN 20 – Distribution + LV

Bust Protection IEDs

VLAN 30 – Bus Bar

Protection IEDs

VLAN 40 – Transformer

Protection IEDs

VLAN 50 – Video

Surveillance

VLAN 1 – Engineering /

Operations / SCADA

51

IED1 – LN1

IED2 – LN2

IED3 – TB1

IED4 – TX1

IED5 – TX2

IED6 – FDR1

IED7 – FDR2

IED8 – FDR3

IED9 – FDR4

IED10 – FDR5

IED11 – FDR6

IED12 – TB2

IED13 – BUS1

IED14 – BUS2

SCADA

Engineering /

Operations

Video

Surveillance

Network BehaviourVLAN 10 – Transmission +

HV Bus Protection IEDs

VLAN 20 – Distribution + LV

Bust Protection IEDs

VLAN 30 – Bus Bar

Protection IEDs

VLAN 40 – Transformer

Protection IEDs

VLAN 50 – Video

Surveillance

VLAN 1 – Engineering /

Operations / SCADA

52

IED1 – LN1

IED2 – LN2

IED3 – TB1

IED4 – TX1

IED5 – TX2

IED6 – FDR1

IED7 – FDR2

IED8 – FDR3

IED9 – FDR4

IED10 – FDR5

IED11 – FDR6

IED12 – TB2

IED13 – BUS1

IED14 – BUS2

SCADA

Engineering /

Operations

Video

Surveillance

Network BehaviourVLAN 10 – Transmission +

HV Bus Protection IEDs

VLAN 20 – Distribution + LV

Bust Protection IEDs

VLAN 30 – Bus Bar

Protection IEDs

VLAN 40 – Transformer

Protection IEDs

VLAN 50 – Video

Surveillance

VLAN 1 – Engineering /

Operations / SCADA

53

Where Are We?

• Communication between IEDs is necessary

• Ethernet has many advantages over serial communications

– Shared medium means less predictability: collisions

• Switches improve Ethernet performance by eliminating collisions

• Queuing latency minimized with Classes of Service (CoS/802.1P)

• CoS (priority) information carried in VLAN tag

• VLAN configuration on a link must match (IED <> Switch)

• Trunk ports send traffic for multiple VLANs, each* identified by a

„tag‟

• Edge ports send traffic for a single VLAN, and typically send

untagged frames

54

Thank You!

Questions?

René MidenceRuggedCom Inc.

Vice President, Marketing

Phone: +1 905 266 11 39

Mobile: +1 647 504 58 15

E-mail: ReneMidence@ruggedcom.com

55

References

• IEC 61850 -Communication Networks and Systems in

Substations:An Overview of Computer Science, Jianqing

Zhangand Carl A. Gunter, Illinois Security Lab

• RuggedCom: The Communications Backbone for IEC

61850, IEC 61850 Seminar, derived from UCA Users

Group Meeting CIGRE 2006

• DistribuTECH 2009 - Setting your network for IEC61850

Traffic, Rene Midence