T28 - Design Considerations for Robust EtherNet/IP · PDF fileStructure, Hierarchy and...

Copyright © 2015 Rockwell Automation, Inc. All Rights Reserved.

PUBLIC

PUBLIC - 5058-CO900H

T28 - Design Considerations for Robust EtherNet/IP Networking

Copyright © 2015 Rockwell Automation, Inc. All Rights Reserved.PUBLIC

Abstract

Learn about top design considerations that are developed by Rockwell

Automation and our partners to help you design and deploy a more

scalable, robust, secure and future-ready EtherNet/IP network

infrastructure. A prior understanding of general Ethernet concepts is

recommended.

2

Copyright © 2015 Rockwell Automation, Inc. All Rights Reserved.PUBLIC 3

Why Is This Important?Design Considerations for Robust EtherNet/IP Networking

Scalable, robust, secure and future-ready infrastructure/architecture:

Application

Software

Network

Internet of Things, Internet of Everything


Industrial Network Design

Methodology

Single Industrial Network

Technology

Sizing and Selection of

EtherNet/IP Devices

Physical Infrastructure

Structure, Hierarchy and

Segmentation

Broadcast Domains

IP Addressing

Network Availability

Convergence-Ready Network

Solutions

Industrial Network Design MethodologyDesign Considerations for Robust EtherNet/IP Networking


Understand application and functional requirements

Devices to be connected – industrial and non-industrial

Data requirements for availability, integrity and confidentiality

Communication patterns, topology and resiliency requirements

Types of traffic – information, control, safety, time synchronization, drive control, voice, video

Develop a logical framework (roadmap)

Migrate from flat networks to structured and hardened networks

Define zones and segmentation, place applications and devices in the logical framework that is based on requirements

Develop a physical framework to align with and support the logical framework

Deploy a Holistic Defense-in-Depth Security Model

Reduce risk, simplify design, and speed deployment:

Use information technology (IT) standards

Follow industrial automation technology (IAT) standards

Use reference models and reference architectures

Industrial Network Design Methodology

5

Avoiding

Network Sprawl

MANAGE /

MONITOR

IMPLEMENT

AUDIT DESIGN/PLAN

ASSESS

Enabling OEM

Convergence-Ready

SolutionsBecause Network

Infrastructure Matters


CIP

IEC 61158

6

Single Industrial Network TechnologyOSI 7-Layer Reference Model

Application

Presentation

Session

Transport

Network

Data Link

Physical

Layer 7

Layer 6

Layer 5

Layer 4

Layer 3

Layer 2

Layer 1

Network Services to User App

Encryption/Other processing

Manage Multiple Applications

Reliable End-to-End Delivery

Error Correction

Packet Delivery, Routing

Framing of Data, Error Checking

Signal type to transmit bits,

pinouts, cable type

IETF

TCP/UDP

IETF IP

IEEE

802.3/802.11

TIA - 1005

Routers

Switches

Cabling/RF

Layer Name Layer No. Function Examples

Open Systems

Interconnection

What makes EtherNet/IP

industrial?

Physical Layer

Hardening

Infrastructure Device

Hardening

Common Application

Layer Protocol


Controller exchanges 36 bytes of I/O data with 10 I/O Adapters with a 1 ms Requested Packet Interval (RPI)

RPI = 1 ms

1,000 frames/second in each direction

Each I/O Adapter must be able to:

Consume 1,000 frames/second

Produce 1,000 frames/second

The Controller must be able to:

Consume 10,000 frames/second

Produce 10,000 frames/second

Design considerations

Size the Controller

Maximum # of Adapters (CIP Connections)

Minimum RPI (how fast)

Maximum I/O Data Size per RPI

Size the Adapters

Minimum RPI (how fast)

Maximum I/O Data Size per RPI

Physical Environment – for example, EMI Interference for Copper Media

Speed / Duplex (potential mismatch)

Network Infrastructure Latency and Jitter

Sizing and Selection of DevicesTheoretical EtherNet/IP Performance Example

This represents about 10% of the total network bandwidth


Physical Infrastructure

8

Design and implement a robust physical layer

Environment Classification - MICE

More than cable Connectors

Patch panels

Cable management

Noise mitigation

Grounding, Bonding and Shielding

Standard Physical Media Wired vs. Wireless

Copper vs. Fiber

UTP vs. STP

Single-mode vs. Multi-mode

SFP – LC vs. SC

Standard Topology Choices Switch-Level and Device-Level

Cable

Selection

ENET-WP007

ODVA Guide

Fiber Guide

ENET-TD003

http://literature.rockwellautomation.com/idc/groups/literature/documents/wp/enet-wp007_-en-p.pdf




http://www.odva.org/Portals/0/Library/Publications_Numbered/PUB00148R0_EtherNetIP_Media_Planning_and_Installation_Manual.pdf

http://www.odva.org/Portals/0/Library/Publications_Numbered/PUB00148R0_EtherNetIP_Media_Planning_and_Installation_Manual.pdf

http://literature.rockwellautomation.com/idc/groups/literature/documents/td/enet-td003_-en-e.pdf






Physical InfrastructureEnvironmental Focus – M.I.C.E.

Office IndustrialTIA 1005

Increased Environmental Severity

M.I.C.E. provides a method of categorizing the environmental classes for each plant Cell/Area Zone.

The MICE environmental classification is a measure of product robustness: Specified in ISO/IEC 24702

Part of TIA-1005 and ANSI/TIA-568-C.0 standards

This provides for determination of the level of “hardening” required for the network media, connectors, pathways, devices and enclosures.

Examples of rating: 1585 Industrial Ethernet Media: M3I3C3E3

M12: M3I3C3E3

RJ45: M1I1C2E2


Physical InfrastructureSelect best media for your needs

UTP vs. STP

Unshielded Twisted Pair (UTP) Shielded Twisted Pair (STP)

Costs less Excellent immunity from EMI and RFI noise

Installs faster Can locate cable close to source of noise

Smaller diameter, more flexible Well suited for more rigorous environments

CAT5e vs. CAT6a

CAT5e CAT6a

Costs Less Higher signal to noise ration; performance margins

Suitable for speeds of less than a Gbps Designed to deliver Gbps performance

Copper vs. Fiber

Copper Fiber

Termination and installation are faster Cost of fiber transceivers is higher

Less fragile Use when excessive EMI noise is present

Distances of less than 100 m Use when distance is a factor (over 100 m)

Multi-mode vs. Single-mode

Fiber

Multi-mode Single-mode

For distances of up to 550 m @ 1 Gbps and 2 km @ 100 Mbps

Longer distances (up to 40 km)

Lower-cost transceivers, connectors and installation High-bandwidth capabilities

Higher fiber cost, but lower total system cost Lower fiber cost, but higher total system cost


Structure, Hierarchy and Segmentation

11

Smaller modular building blocks to help

minimize network sprawl and build

scalable, robust and future-ready

network infrastructure

Smaller broadcast domains

(restrict broadcast traffic)

Smaller fault domains (for example, Layer

2 loops)

Smaller domains of trust (security)

Multiple techniques to create smaller network building blocks (Layer 2 domains) Structure and hierarchy

Logical model – geographical and functional organization of IACS devices

Campus network model - multi-tier switch model – Layer 2 and Layer 3

Logical framework

Segmentation Multiple network interface cards (NICs) – for

example, CIP bridge

Network Address Translation (NAT) appliance

Virtual Local Area Networks (VLANs)

VLANs with NAT

Integrated Services Router


Structure, Hierarchy and SegmentationLayer 2 Collision Domains

Fa1/1Fa1/2

Gi1/1 Gi1/1 Gi1/2 Fa1/1 Fa1/2Controller 1 Controller 3

Controller 2

Switch L2-1 Switch L2-2Switch L3-1

L3 - 10.10.10.5

L2 - 0000:BC10:1005

L3 - 10.10.20.5

L2 - 0000:BC10:2005

L3 - 10.10.10.6

L2 - 001D:9C10:1006

L3 - 10.10.10.1

L2 - E490.6919.5B44L3 - 10.10.20.1

L2 - E490.6919.5B41

L3 - 10.10.10.0/24

L2 - VLAN 10L3 - 10.10.20.0/24

L2 - VLAN 20


Structure, Hierarchy and SegmentationLayer 2 Broadcast Domains - Switch Hierarchy

Fa1/1Fa1/2

Gi1/1 Gi1/1 Gi1/2 Fa1/1 Fa1/2Controller 1 Controller 3

Controller 2

Switch L2-1 Switch L2-2Switch L3-1

L3 - 10.10.10.5

L2 - 0000:BC10:1005

L3 - 10.10.20.5

L2 - 0000:BC10:2005

L3 - 10.10.10.6

L2 - 001D:9C10:1006

L3 - 10.10.10.1

L2 - E490.6919.5B44L3 - 10.10.20.1

L2 - E490.6919.5B41

L3 - 10.10.10.0/24

L2 - VLAN 10L3 - 10.10.20.0/24

L2 - VLAN 20


Structure, Hierarchy and Segmentation

Structured and Hardened

IACS Network Infrastructure

Flat and Open

Industrial Automation and Control System

Network Infrastructure

Flat and Open

IACS Network Infrastructure


Structure, Hierarchy and SegmentationMultiple Network Interface Cards (NICs) - CIP Bridge

Benefits Clear network ownership demarcation line

Challenges Limited visibility to control network devices for

asset management Limited future-ready capability Smaller PACs may not support

Benefits Plant-wide information sharing for data collection

and asset management Future-ready

Challenges Blurred network ownership demarcation line

Isolated networks - two NICs for physical network segmentation

Converged networks – logical segmentation

Converged

Network

Shared Layer 2 Network

VLAN 102

Control Network

Levels 0-2

Plant Network

Level 3

Layer 2 Network

Layer 2 Network

Control Network

Levels 0-2

Plant Network

Level 3


Structure, Hierarchy and SegmentationVirtual LANs (VLANs)

Layer 2

Stratix 8300™

Ring

Stratix 5700™

Stratix 8000™

Plant-wide IACS

Machine #1OEM #1

Machine #2OEM #2

OWS

CompactLogix™5370 L3

1732E Slim ArmorBlock® I/O

1734Point I/O

ControlLogix®1756-EN2T

Plant-wide IACS

VLAN 40

IP Subnet 172.16.40.0/24

Large Layer 2 Broadcast Domain

Machine #1 (OEM #1)

VLAN 20

IP Subnet 10.20.20.0/24

VLAN 10

IP Subnet 10.10.10.0/24

Machine #2 (OEM #2)

VLAN 30

IP Subnet 192.168.30.0/24

VLAN 5

IP Subnet 192.168.1.0/24

Plant-wide IACS

VLAN 40

IP Subnet 172.16.40.0/24

VLAN10

Stratix 8300™

Ring

Stratix 5700™

Stratix 8000™

Plant-wide IACS

Machine #1OEM #1

Machine #2OEM #2

OWS

CompactLogix™5370 L3

1732E Slim ArmorBlock® I/O

1734Point I/O

ControlLogix®1756-EN2T

Layer 3

VLAN20

VLAN30

VLAN5

Smaller Layer 2 Broadcast Domains


Network Address Translation (NAT)is a service that can translate a Source IP address to another IP address within a packet

Can be a Layer 2 or Layer 3 device

Has two forms:

One to One (1:1) – Allows for the assignment of a unique outside IP address to a specific inside IP address

One to Many (1:n) – a.k.a. TCP/UDP Port Address Translations (PAT). Allows Multiple devices to share one “Outside” address

Structure, Hierarchy and SegmentationNetwork Address Translation

Inside

Outside Subnet

(ex. 10.0.0.x)

NAT Enabled Device

Inside Subnet

(ex. 192.168.1.x)

Many Outside IP addresses

(One per device wishing to be accessible from the Outside Subnet

Many Inside IP addresses

(One per connected device)


Structure, Hierarchy and SegmentationNetwork Address Translation

• Multiple Skids/Machines– Each Skid/Machine Aggregated by One

Stratix 5700™ Layer 2 NAT Switch

– Single VLAN Architecture

Inside Outside

192.168.1.10 10.10.10.10

Outside Inside

10.10.10.5 192.168.1.5

Inside to Outside

NAT Table

Outside to inside

NAT Table

HMI.11

I/O

Line Controller10.10.10.5

VFD.12

Industrial ZoneLevels 0-3

(Plant-wide Network)

Cell/Area Zone - Levels 0-2

IES-1

HMI.11

VFD.12

Controller192.168.1.10

InsideVLAN 2

192.168.1.0/24

InsideVLAN 2

192.168.1.0/24

OutsideVLAN 2

10.10.10.0/24

Controller192.168.1.10

IES-2 IES-3

I/O

.13 .14.13.14

Skid /Machine

#1

Skid /Machine

#2

Inside Outside

192.168.1.10 10.10.10.20

Outside Inside

10.10.10.5 192.168.1.5

Inside to Outside

NAT Table

Outside to inside

NAT Table

IES-2 Stratix 5700™ w/ NAT

IES-3 Stratix 5700™ w/ NAT

Stratix 5700™ w/NAT

IES-4


Structure, Hierarchy and SegmentationNo Segmentation (not recommended)

Enterprise-wideBusiness Systems Levels 4 & 5 – Data Center

Enterprise Zone

Level 3.5 - IDMZ

Cell/Area Zone #1

Subnet

10.17.10.0/24

Cell/Area Zone #2

Subnet

10.17.10.0/24Cell/Area Zone #3

Subnet 10.17.10.0/24

Plant-wideSite-wide

Operation Systems

• Plant LAN – VLAN17 - Layer 2 Domain

• Plant IP - Subnet 10.17.10.0/24, every

device requires a unique IP address

Physical or Virtualized Servers• Application Servers and Services Platform• Network Services – for example, DNS, AD, DHCP,

AAA• Remote Access Server (RAS)• Storage Array


(Plantwide Network)

Level 3 - Site Operations

Cell/Area ZonesLevels 0-2




Structure, Hierarchy and SegmentationMultiple NIC Segmentation

Line/Area

Controller


Enterprise Zone

Level 3.5 - IDMZ

Cell/Area Zone #1

Subnet 192.168.1.0/24

Cell/Area Zone #2

Subnet 192.168.1.0/24 Cell/Area Zone #3

Subnet 192.168.1.0/24

Plant-wideSite-wide

Operation Systems


• Plant IP - Subnet 10.17.10.0/24










Structure, Hierarchy and SegmentationNAT Appliance Segmentation


Enterprise Zone

Level 3.5 - IDMZ

Plant-wideSite-wide

Operation Systems

Cell/Area Zone #1

Subnet 192.168.1.0/24

Cell/Area Zone #2


Subnet 192.168.1.0/24


• Plant IP - Subnet 10.17.10.0/24










Structure, Hierarchy and SegmentationIntegrated Services Router Segmentation


Enterprise Zone

Level 3.5 - IDMZ

Plant-wideSite-wide

Operation Systems

Cell/Area Zone #1

Subnet

192.168.1.0/24

Cell/Area Zone #2


Subnet 192.168.1.0/24


• Plant IP - Subnet 10.17.10.0/24










Structure, Hierarchy and SegmentationVLAN Segmentation without NAT

Levels 4 & 5 – Data CenterEnterprise Zone

Level 3.5 - IDMZ

Cell/Area Zone #1

VLAN10

Subnet 10.10.10.0/24

Cell/Area Zone #2

VLAN20

Subnet

10.10.20.0/24

Cell/Area Zone #3

VLAN30

Subnet 10.10.30.0/24

Enterprise-wideBusiness Systems

Plant-wideSite-wide

Operation Systems


• Plant IP - Subnet 10.17.10.0/24, every

device requires a unique IP address










Structure, Hierarchy and SegmentationVLAN Segmentation with NAT

Levels 4 & 5 – Data CenterEnterprise Zone

Level 3.5 - IDMZ

Cell/Area Zone #1

VLAN10

Subnet 192.168.1.0/24

Cell/Area Zone #2

VLAN20

Subnet 192.168.1.0/24Cell/Area Zone #3

VLAN30

Subnet 192.168.1.0/24

Enterprise-wideBusiness Systems

Plant-wideSite-wide

Operation Systems


• Plant IP - Subnet 10.17.10.0/24










Redundant Ethernet Networks Independent LANs

Independent Paths

Beacon Protocol

Redundant Path Ethernet Network Common LAN

Redundant Paths

Resiliency Protocol

Network AvailabilityRedundant vs. Redundant Path


Network AvailabilityRedundant Path Topologies with Resiliency Protocols

Switch-level TopologiesRedundantStarFlex Links

HMI

CiscoCatalyst 2955

Cell/Area Zone

Cisco Catalyst3750 StackWiseSwitch Stack

Controllers,Drives, and Distributed I/O

Cell/Area Zone

HMI

Controller

RingResilient Ethernet Protocol (REP)

HMI

Cell/Area Zone

Controllers

Controllers, Drives, and Distributed I/O


Star/BusLinear

Cell/Area ZoneControllers, Drives, and Distributed I/O

HMI

Controllers


Device-level Topologies

Stratix 8300™

VFDDrive

HMI

I/O I/O

VFDDrive

HMII/O

I/O

Instrumentation

VFDDrive

Controller

ControllerServoDrive

Switch-level andDevice-level Topologies

Controllers,Drives, and Distributed I/O

Cell/Area Zone


Network AvailabilityRedundant Path Topologies with Resiliency Protocols

IES

IES

IES

IES

IES

SafetyController

Safety I/O

HMI

ServoDrive

I/O

Controller

VFDDrive

VFDDrive

HMI I/O

Controller

CIP

Class 1

Class 3

CIP Safety

CIP Sync

Integrated

Motion on

the

EtherNet/IP

network

CIP Class 1

CIP Class 3

IndustrialZone

RedundantControllers

VFDDrive

I/O

Instrumentation

IES

I/O

DLRDLR

REP


Convergence-ready Network SolutionsDesign and Implementation Considerations

Partner Solution(s)

for example,

Process SkidPlant-wide / Site-wide

Industrial

Automation Systems

Partner Solution(s)

for example,

MachinePlant-wide

Industrial

Automation Systems

Design and deployment considerations that a partner (for example, OEM, SI, Contractor)

has to take into account to achieve seamless integration of their solution (for example,

machine, skid) into their customers’ plant-wide/site-wide network infrastructure.Early, open and two-way

dialogue is critical!


Alignment with End User - Security Stance:

Business Practices

Corporate/Local Standards

Tolerance to Risk

Current Status of Network Infrastructure (End User and OEM)

Segmentation of Domains of Trust

Application Requirements

Industrial Security Policies

Physical access, port security, access control lists, application security, remote access (avoidance of back doors)

Alignment with industrial automation and control system (IACS) security standards such as ISA/IEC-62443 (formerly ISA99) and NIST 800-82


Early, open and two-way



Alignment with End User - Network Services: Use of a common industrial network technology that fully uses standard Ethernet and IP

networking technology as the multi-discipline industrial network infrastructure. Common network infrastructure devices – asset utilization

Future-ready - sustainability

IP addressing schema Who manages? End User (OT/IT) or OEM?

Address range (class), subnet, default gateway (routability)

Implementation conventions – static/dynamic, hardware/software configurable, NAT/DNS

Use of Network Services Switches - managed vs. unmanaged, industrial vs. COTS, system vs. component approach

Segmentation, data prioritization

Topologies - switch-level, device-level, hybrid

Availability – loop prevention, redundant path topologies with resiliency protocols

Time Synchronization Services IEEE 1588 Precision Time Protocol (PTP w/E2E) – first fault, SOE, Motion


Early, open and two-way



Websites Reference Architectures

Design and Implementation Guides ENET-TD001E-EN-P - Converged Plantwide Ethernet (CPwE) Baseline Document

ENET-TD005B-EN-P - Deploying the Resilient Ethernet Protocol (REP) in a Converged

Plantwide Ethernet Architecture

ENET-TD006A-EN-P - Deploying 802.11 Wireless LAN Technology within a Converged

Plantwide Ethernet Architecture

ENET-TD007A-EN-P - Deploying Network Address Translation within a Converged Plantwide

Ethernet Architecture

ENET-TD008A-EN-P - Deploying Identity Services within a Converged Plantwide Ethernet

Architecture

ENET-TD009A-EN-P - Securely Traversing IACS Data Across the Industrial Demilitarized Zone

ENET-TD010A-EN-P - Deploying A Resilient Converged Plantwide Ethernet Architecture

Additional MaterialCPwE Architectures - Cisco and Rockwell Automation

31

http://www.rockwellautomation.com/rockwellautomation/products-technologies/network-technology/architectures.page?

http://literature.rockwellautomation.com/idc/groups/literature/documents/td/enet-td001_-en-p.pdf







Application Guides ENET-TD003A-EN-E - Fiber-optic Infrastructure Application Guide (Panduit/Cisco/Rockwell Automation)

Whitepapers ENET-WP022B-EN-P - Top 10 Recommendations for Plant-wide EtherNet/IP Deployments

ENET-WP009A-EN-P - Achieving Secure Remote Access to plant-floor Applications and Data

ENET-WP031A-EN-P - Design Considerations for Securing Industrial Automation and Control System

Networks

ENET-WP033A-EN-P - Resilient Ethernet Protocol in a Converged Plantwide Ethernet (CPwE) Architecture

ENET-WP034A-EN-P - Deploying 802.11 Wireless LAN Technology within a Converged Plantwide Ethernet

Architecture

ENET-WP036A-EN-P - Deploying Network Address Translation within a Converged Plantwide Ethernet

Architecture

ENET-WP037A-EN-P - Deploying Identity Services within a Converged Plantwide Ethernet Architecture

ENET-WP038A-EN-P - Securely Traversing IACS Data Across the Industrial Demilitarized Zone

ENET-WP039A-EN-P - A Resilient Converged Plantwide Ethernet Architecture

Additional MaterialCPwE Architectures - Cisco and Rockwell Automation

32


http://literature.rockwellautomation.com/idc/groups/literature/documents/wp/enet-wp022_-en-e.pdf










Cisco Industrial Networking Specialist Training and Certification

E-learning modules (pre-learning courses)

Control Systems Fundamentals for Industrial Networking (ICINS)

Networking Fundamentals for Industrial Control Systems (INICS)

Classroom training

Managing Industrial Networks with Cisco Networking Technologies (IMINS)

Exam

200-401 IMINS

CCNA Industrial Training and

Certification

Classroom training

Managing Industrial Networks for

Manufacturing with Cisco Technologies

(IMINS2)

Exam

200-601 IMINS2

Industrial IP Advantage: e-Learning

CPwE Design Considerations and Best

Practices

Additional Material Training and Certifications

33

https://learningnetworkstore.cisco.com/industrial-networking/control-systems-fundamentals-for-icins-elt-icins-v1-0-021145?utm_source=Certs_industrial-networking&utm_content=ICINS&utm_campaign=CLNStore-Products

https://learningnetworkstore.cisco.com/internet-of-things-iot/networking-fundamentals-for-industrial-control-systems-inics-elt-inics-v1-0-021144?utm_source=Certs_industrial-networking&utm_content=INICS&utm_campaign=CLNStore-Products

https://learningnetwork.cisco.com/community/certifications/iot/industrial-networking/imins-exam/exam-topics/overview

https://learningnetwork.cisco.com/community/certifications/iot/industrial-networking/imins-exam

http://tools.cisco.com/GlobalLearningLocator/courseDetails.do?actionType=executeCourseDetail&courseID=6094

https://learningnetwork.cisco.com/community/certifications/ccna-industrial/imins2

http://iipa.shop.gpstrategies.com/


A ‘go-to’ resource for educational information

about industrial network communication and

using standard Internet Protocol (IP) for

industrial applications

Community of like-minded companies –

Cisco®, Panduit®, and Rockwell Automation®

Receive monthly e-newsletters with

articles and videos on the latest trends

e-Learning courses available on network

design topics

Additional Material Education

www.industrial-ip.org

34

http://www.industrial-ip.org/

T28 - Design Considerations for Robust EtherNet/IP · PDF fileStructure, Hierarchy and...

Documents

Transcript of T28 - Design Considerations for Robust EtherNet/IP · PDF fileStructure, Hierarchy and...