You certainly know that SONET/SDH is an aging technology and that if your

operational network is still based on it then you will need to transition to a packet

switched network. Just as certainly, you know that the Internet Protocol (IP) as used

in IT networks defines packet formats and routing protocols, but does not define a full

networking solution for OT networks. You have probably heard that Carrier Ethernet

(CE) and Multiprotocol Label Switching (MPLS) are alternative solutions, but you may

be unsure which is more suitable for operational networks. You probably have not

seen any recent comparisons of CE and MPLS, and would like to understand the

advantages and disadvantages of each. If all of the above hold, then read on. And if

you want to brush up on packet switched technologies (IP, MPLS, and Ethernet), we

have prepared everything you need to know in an appendix.

A Comparison of

Carrier Ethernet and MPLSfor Critical Infrastructure Operational Networks

White Paper

Critical infrastructure networks, whether circuit or packet

switched, need to utilize guaranteed networking. This is in

contrast with best effort networking which may be good enough

for browsing the Internet and sending emails, but is decidedly

not so for mission critical traffic, such as that of connectivity of

power utilities.

Best effort networking means that there are no guarantees

that information will actually reach its destination, only that the

network will do its best to deliver it. Guaranteed networking

means that that there are a set of service characteristics

that need to be guaranteed, such as availability, Committed

Information Rate (CIR), restoration time, propagation delay, and

information loss rate. These guarantees are enforced in service

provider networks via Service Level Agreements (SLAs) between

the service provider and its customers, and in self-built networks

by pursuing Service Level Objectives (SLOs).

Networks that are guaranteed to meet Service Level

Objectives do so by a combination of:

• Determinism

• Resilience

• Monitoring and diagnostics

• Traffic conditioning

• Security

• and in some cases: timing support

Let’s look at each of these attributes in turn.

Guaranteed Networking

2

White Paper

Circuit switched networks, such as TDM, PDH, and SONET/SDH,

are necessarily deterministic, by which we mean that the path

taken by the information from source to destination is constant

and well known. Even if a fault occurs and protection switching

kicks in, the protection path has been pre-configured, and the

jump from the working path to the protection path is minimally

intrusive. Furthermore, in circuit switched networks bits always

arrive at a known constant rate.

For packet switched networks determinism means that all

packets belonging to the same application are handled the same

way – i.e., follow the same path through the network and are

processed in the same way by the network elements traversed.

Packets needn’t be sent or arrive at a constant rate, so that the

statistical multiplexing advantage is maintained.

In best effort networks packet forwarding paths are stitched

together via the local decisions of routers that learn network

topology using distributed routing protocols. While these local decisions can be influenced by policy, they are mostly up to the whims of individual routers, and indeed change over time. Deterministic packet delivery is accomplished

by having packet forwarding paths configured by Network

Management Systems (NMS) or by Software Defined Networking

(SDN) controllers. These systems are centrally located, maintain

a full view of the topology and usage of the entire network, can

exploit sophisticated algorithms to attain optimal efficiency, and

can be directly managed by human operators.

Determinism

3

White Paper

Note that for non-deterministic networks it is not possible to

reserve resources for a particular service in any given router, since

we can’t be sure that the packets will traverse this particular

router. On the other hand, deterministic networks can be traffic

engineered, that is, resources such as link bandwidth and

network element processing power can be reserved, thus closely

emulating circuit switched networking. Although less celebrated,

we put determinism at the top of our list since it is the heart of guaranteed networking. It is meaningless to speak about Service

Level Objectives if each packet can behave differently from its

predecessor. Similarly, if packets do not arrive or do so tardily,

how could one diagnose the problem if the path taken by the

packets is unknown and in fact indeterminate? Moreover, security

can never be guaranteed if packets are free to traverse arbitrary

and possibly untrusted network elements.

So, how do the various packet switched technologies stack up

regarding determinism?

Routed IP networks are not deterministic, so although modern

applications produce IP packets, mission critical applications

should never rely solely on IP forwarding mechanisms. For the

same reason best effort MPLS (sometimes called IP/MPLS, and

by which we mean any routed – non-traffic engineered flavor

of MPLS) should be ruled out, as it merely expedites the

forwarding of nondeterministic IP.

Mission critical networks can benefit from the determinism of

an underlying Carrier Ethernet or MPLS-TE network. Both are

typically configured by Network Management Systems (or more

recently by SDN).

Determinism

White Paper

4

The most important service characteristic is path continuity,

which takes up or down values. Obviously, the service objective is for continuity to be up 100% of the time, but faults are inevitable, and

can be tolerated if they are rapidly detected and rectified; an

attribute called resilience. We will discuss fault detection in

the next section, here we will consider rectification aspects.

In best effort networks, faults are eventually detected and

processed because affected network elements send status

indications to other network elements, and thus modifying

their forwarding information databases. While these

communications are ongoing packets may be misdirected and

become lost, but eventual consistency implies that eventually

correct packet delivery will be restored.

In guaranteed networking, resilience is attained by carefully

planning protection switching mechanisms, and properly

implementing protection switching protocols. Typical

objectives are under 50 milliseconds from fault detection

to service restoration, and reversion to the original

configuration once slower maintenance operations have

cleared the fault.

So, how do the various packet switched technologies stack

up regarding resilience?

IP networks do not provide protection switching, rather rely

on reconvergence of routing protocols to eventually restore

correct packet delivery (and discard black holed packet in the

meantime based on the TTL field).

Resilience

White Paper

5

Carrier Ethernet provides both linear and ring protection

mechanisms, defined in ITU-T Recommendations G.8031 and

G.8032, respectively. These mechanisms have been widely

implemented and deployed, and are interoperable amongst

all vendors.

MPLS networks, including otherwise best effort ones, may

provide resilience using a mechanism known as Fast ReRoute

(FRR), as described in IETF RFC 4090. Unlike protection

switching, FRR utilizes local detours to achieve very fast

service restoration, at the expense of some reduction in

determinism. MPLS-TP defines linear protection mechanisms

similar to those of Carrier Ethernet. The IETF defined RFC

6378 and the ITU-T standardized Recommendation G.8131.

Unfortunately, these standards are based on conflicting

principles and are not interoperable, and neither has seen

significant adoption. Both IETF and ITU-T worked on ring

protection for MPLS-TP, with the IETF’s RFC 6974 describing

how to achieve ring protection using linear protection

mechanisms. The ITU-T work did not reach fruition.

Resilience

White Paper

6

Guaranteeing communications in a network depends on

continuously monitoring the service level objectives. Even if

planning is carefully carried out, without monitoring one could

never be sure that the objectives are being met. Furthermore, if

some objective is found to be inadequate or deteriorating, a set of

diagnostic tools is needed to find the root cause of the problem.

The monitoring and diagnostic toolset is collectively called

Operations, Administration, and Maintenance (OAM). In circuit

switched networks OAM is added to the bit streams as overhead,

while in packet switched networks special OAM packets are

typically employed. OAM isn’t fully dependable for nondeterministic

packet switched networks, since the results depend on the packet-

forwarding path, which may arbitrarily vary.

Basic continuity is always essential for communications

of any type to succeed, while other specifications (such

as available information rate and propagation delay)

may or may not be important for a given application.

Hence, we conventionally differentiate between Fault Monitoring (FM) – the OAM mechanisms that monitor continuity and Performance Measurement (PM) – the OAM mechanisms for all the other

specifications. FM OAM may be used to trigger

protection switching, and to measure availability – the

percentage of up-time, with numbers such as “five nines”,

i.e., 99.999% availability, being common (five nines translates

to down time of about five and a half minutes per year). PM

OAM is conventionally employed to collect statistics for SLA/

SLO compliance reports.

Monitoring and Diagnostics

White Paper

7

So, how do the various packet switched technologies stack

up regarding OAM?

Pure IP is nondeterministic, and thus can’t fully support

monitoring. A protocol known as Bidirectional Forwarding

Detection (BFD) was developed to test continuity between

adjacent routers, and was later extended to the multi-hop

case. An extensive set of PM tools were produced by the IETF

IPPM working group.

Carrier Ethernet has an extensive OAM toolset, based on

ITU-T Recommendation Y.1731, IEEE 802.1 CFM, and MEF

Implementation Agreements. Extensive interoperability

testing and wide deployment ensures that this toolset is

mature. Furthermore, commissioning testing defined in

Y.1564 allows a service to be fully tested before deployment.

MPLS-TP was specifically developed to achieve OAM

functionality equivalent to that of Carrier Ethernet, and

hence only MPLS-TP has significant OAM capabilities.

Unfortunately, once again the two competing standards

organizations produced two non-interoperable versions of

OAM. The extent of deployment of either of these standards

is unclear.

Monitoring and Diagnostics

White Paper

8

In circuit switched networks, information sources have constant

bit rates, while in packet switched networks information sources

are free to send information, or not, as they see fit. Obviously,

objectives can’t be guaranteed if information sources start

sending at much higher than expected rates, as the required

physical resources would not be available. In such cases, packets

need to be deliberately discarded.

Best effort packet switched networks are often designed to

ensure fairness, meaning that all packets have equal chance of

being delivered. Alternately, packets may be prioritized, so that

lower priority packets are preferentially discarded.

Guaranteed networking employs a more disciplined approach.

When a communications service is configured, it is allocated

a Committed Information Rate (CIR). This does not mean

that the service continually consumes that rate as in circuit

switched networks, merely that the network is planned so

that this rate can be guaranteed. In service provider networks the CIR is directly linked to payment, i.e., the customer pays more for higher CIR. If a source exceeds its committed

information rate, the network defends itself by employing

traffic conditioning, i.e., it limits the information rate to

comply with the CIR.

There are several types of conditioning in use. Simple

policing colors packets green (i.e., allows them into the

network) if the suitably averaged traffic rate does not

exceed the rate for which the customer paid, and colors

them red (and discards them) if the rate is excessive.

More sophisticated policers can color packets yellow and

conditionally allow them to enter the network.

Traffic Conditioning

White Paper

9

An alternative to policing is shaping, whereby packets are

queued when the instantaneous data rate exceeds the CIR,

but afterwards allowed in if the suitably averaged rate is

acceptable. This averaging is conventionally carried out using

a token bucket algorithm.

So, how do the various packet switched technologies stack

up regarding traffic conditioning?

IP packets have DSCP field for indicating packet priority. It is

up to the network administrator to decide if this feature is

utilized.

Carrier Ethernet supports priority-dependent dual-token-

bucket policing and shaping to achieve traffic conditioning

and to protect the network from unfair exploitation.

All flavors of MPLS have a 3-bit Traffic Class (TC) field to

influence queuing or to indicate congestion, but its use is

not enforced by any standardized SLAs.

Of course, since MPLS packets may be transported by

Ethernet, one could envision using Carrier Ethernet

mechanisms to perform the traffic conditioning. However,

this begs the question as to why MPLS is needed in the first

place if Carrier Ethernet is employed.

Traffic Conditioning

White Paper

10

By now, it is well known that network infrastructures can be

hacked in order to illicitly provide information or service, or to

deny information or service from legitimate users. The principal

objectives of network security mechanisms are authorization,

establishing security associations, authentication, and

confidentiality. Authorization means ensuring that a legitimate

user (but only a legitimate user) gains access to its information

or services. Authentication means verifying that a packet claiming

to be from a particular source is indeed from that source.

Confidentiality means denying access to information from those

who should not obtain access.

Sophisticated attackers can compromise both circuit switched and packet switched networks. Circuit switched networks are

often considered secure, but that is merely because their basic

technologies are less understood by the less sophisticated

hacker. Packet switched networks may suffer from many more

attack vectors than circuit switched ones, but have also received

much more attention from the security industry.

So, how do the various packet switched technologies stack up

regarding security?

The IP suite, which was originally designed without security in

mind, has been retrofitted with a number of security features.

IPsec is the fundamental network-layer mechanism, and offers

both user-to-user and tunneling modes. Internet Key Exchange

(IKE) handles authorization for IPsec. The IP suite also contains

security mechanisms at higher layers, including SSH, TLS, and

HTTPS. IP routing protocols have limited security features, but

may themselves be protected using the above.

Security

White Paper

11

Similar to IPsec Carrier Ethernet defines MACsec (802.1AE)

for authentication and confidentiality, and a mechanism for

authorization and establishing security associations called

802.1X. Ethernet management communications have their own

security mechanisms, such as those defined in SNMPv3.

MPLS was designed for core networks, under the assumption

that such networks are walled gardens, meaning that although

defense could be applied at the perimeter, no attacks were

envisioned from inside the network.

For this reason, MPLS packets, of any flavor, forgo source

identifiers. Without a source identifier there is no possibility

of authenticating a packet as coming from any given source.

Furthermore, a MPLS packet has no real destination address -

merely a short label. Hence it is relatively simple for an insider

to fabricate a packet that is technically legitimate. Finally, MPLS

defines no security mechanism of its own, so that confidentiality

can’t be enforced.

Furthermore, the MPLS control plane employs soft state

protocols. A soft-state protocol is one that requires periodic

keep-alive messages to maintain state. Thus by maliciously

discarding a few packets, an attacker can lead to massive

denial of service.

Of course, since MPLS is a thin layer sitting between

Ethernet and IP, an MPLS packet could avail itself of Ethernet

and/or IP security mechanisms. There are two problems

with this approach. First, according to modern thinking,

every network layer requires security mechanisms. Note

that although IPsec may be used, a denial of service attack

at the MPLS layer will still disrupt service. Second, although

multiple layers may be employed, the important question is

what functionality is deployed at each such layer. If a so-

called MPLS network uses mostly non-MPLS mechanisms,

the question is whether it is an MPLS network at all, and for

what purpose the MPLS label was inserted.

Security

White Paper

12

Many modern applications require some level of timing/synchronization

support, with the stringency of the requirements increasing over time.

For power utilities, teleprotection systems may require accuracies on

the order of microseconds, modern synchrophasors need to maintain

1 microsecond time accuracy even when they lose GPS reception, and

smart grid applications have been targeting 1 microsecond absolute

time accuracy as well. The time accuracies required by other critical

infrastructures, such as water distribution networks, railway and

maritime transportation networks and Air Traffic Control systems, are

becoming more stringent as well.

When communicating using synchronous circuit switched technologies,

the network elements themselves must maintain highly accurate

frequency lock in order to extract bits from the received bit-stream.

This inherent mechanism of frequency distribution can be used to

provide high accuracy frequency information to applications that

need it, or to maintain highly accurate time of day for applications

that require time or phase information. Transferring highly accurate frequency and time of day information over asynchronous packet switched networks is much more challenging.The Network Time Protocol used in IP networks can attain accuracies

of 1 millisecond on small networks, and tens of milliseconds on larger

ones. The Precision Time Protocol, known by its moniker IEEE 1588,

uses on-path support to attain much higher accuracies, typically sub-

microsecond.

So, how do the various packet switched technologies stack up

regarding timing?

As we have mentioned, IP supports NTP for low accuracy timing needs.

Timing Support

White Paper

13

Since Ethernet (unlike MPLS and IP) defines a physical layer,

and for high-rate point-to-point links this physical layer is

constant bit rate, all that is required to support distribution of

frequency to user applications is to lock the source frequency

to a suitable frequency reference and to ensure traceability of

delivered frequency to this source. This technique is known as

Synchronous Ethernet or SyncE.

Carrier Ethernet can avail itself of SyncE and IEEE 1588 to

distribute highly accurate frequency and time information to

applications that need it. No similar mechanisms are available

for MPLS (of any flavor), simply because MPLS does not provide

a physical layer, and MPLS standards do not define the requisite

on-path support elements.

Of course, when MPLS is transported over Ethernet, the

underlying Ethernet layer may be used for timing distribution,

as long as the MPLS mechanisms do not impede the required

functions. However, MPLS routers do not universally support

SyncE functions, and MPLS forwarding may decide to forward

packets along paths without on-path support, thus disrupting

the Ethernet timing functionalities. On the other hand, if the

Ethernet layer is given free rein in forwarding decisions, then, as

before, the MPLS layer serves no purpose.

Timing Support

White Paper

14

We have reviewed the requirements for guaranteed networking,

and the fundamental behavior of the various types of IP,

Ethernet, and MPLS networks. We inevitably reach the conclusion

that no version of IP is by itself satisfactory for mission critical

applications. On the other hand, Carrier Ethernet networks fulfil

all of our requirements, and MPLS can fulfil most of them if we

use the proper combination of flavors.

One needs to be careful regarding MPLS, as vendors and service

providers are often unclear as to whether they are deploying

best-effort routed MPLS or completely deterministic traffic

engineered MPLS. Even in the latter case, the MPLS-TP’s OAM and

resilience features are not widely deployed, and their absence

should be seen as a red flag for any mission critical network.

The most significant differences between Carrier Ethernet and

deterministic and monitored MPLS remain the timing and security

aspects. No form of MPLS can provide physical layer frequency

support (as in SyncE) and any time distribution protocol will be

either proprietary or at a non-MPLS layer (either below at the

Ethernet layer or above at the IP layer). Regarding security, MPLS

was and remains a protocol for unattackable walled-garden

networks. While mechanisms can be deployed at IP and Ethernet

layers to overcome some portion of the attack spectrum, a full

security solution for MPLS remains elusive.

We summarize our conclusions in the following table:

Wrap-Up

White Paper

Pure IP

Carrier Ethernet

Best Effort MPLS

MPLS-TE without TP

MPLS-TP without TE

TE+TP

Determinism Resilience Monitoring and diagnostics

Traffic conditioning Security Timing

support

Partial compliance Full compliance No compliance

The driving force behind the transition from circuit switched

networks to packet switched ones is statistical multiplexing.

Networks based on E1s or T1s or SONET/SDH reserve bandwidth

for the worst-case messaging rate, and transmit idle patterns

when there is no traffic to send. Thus, when PDH or SONET/

SDH multiplex together multiple information sources, each

source always occupies its maximum information rate. On

the other hand packet switched networks are not required to

reserve resources (bandwidth, switching capability, memory).

If there are no messages from one source of information,

then network resources are automatically available to carry

information from another source. This characteristic, known as

statistical multiplexing, makes providing service guarantees more

challenging.

The advantage of statistical multiplexing may not seem very

important for a conventional massively overprovisioned OT

network, but the savings are huge in most modern applications,

and hence packet switched network technologies have replaced

TDM-based networks except for in the lowest layers of high-

data-rate transport (such as Optical Transport Networks – OTN).

Hence, if you are going to build a new network, or completely

upgrade an existing one, you are almost certainly going to deploy

a packet switched network. However, you still have considerable

leeway, as there are three different packet switched network

technologies in common use, and each of these has several

flavors that need to be understood before deciding on which

suits your needs.

APPENDIX: Packet Switched Networks

White Paper

16

The first technology is pure IP which comes in two flavors:

IPv4 and IPv6. The second is Ethernet, which comes in three

flavors: basic Ethernet, Carrier Ethernet (CE; which itself has two

categories, namely CE-1.0 and CE-2.0), and Industrial Ethernet.

The third is MPLS (Multiprotocol Label Switching), which presently

comes in four flavors: best effort MPLS, MPLS for L3VPN services,

MPLS-TE, and MPLS-TP.

These three technologies are often used in conjunction, in fact,

a single packet can be simultaneously IP, MPLS and Ethernet. The

essential issue is which functionalities are performed by which

technology. For example, most modern applications utilize IP as

their basic packet format, but this IP packet may be encapsulated

in Ethernet to deliver it from its source to the first router, and

may then be encapsulated in MPLS to be transported from that

router to the next router and the next, and finally encapsulated in

Ethernet for the last hop to its destination.

The next three sections provide a brief overview of each of the

three packet switched network technologies.

IPIP, or Internet Protocol, is a large suite of protocols used by both

the public Internet and in many private networks. The original

version, called IP version 4 (IPv4), is constrained by having far

too few addresses for the public Internet (only about 4.3 billion

devices can be simultaneously connected) and is thus being

replaced by IP version 6 (IPv6).

The IP suite is not a complete protocol stack, starting above

the second layer of the OSI 7-layer model, and thus requires

physical layer and link layer protocols to transport it (for example

– Ethernet). On the other hand the IP suite is highly developed

in many ways, for example, it has optional delivery reliability

mechanisms (e.g., TCP), security features (IPsec, SSH, and TLS),

APPENDIX: Packet Switched Networks

White Paper

17

and support for many applications (e.g., web browsing, email,

and video streaming).

Anyone designing a critical infrastructure network should be

aware that IP is a nondeterministic best effort technology. For

this reason, a higher layer protocol (usually TCP) must be used to

retransmit packets that were lost along the way. This, of course,

introduces additional delay and delay variation.

Ethernet and Carrier Ethernet Ethernet is debatably the most successful packet switched

technology, with hundreds of millions of ports in use. Today

Ethernet handles data rates from 2 Mbps to 100 Gbps, travels

over DSL, dedicated copper wiring, fiber, and radio. Although

it started as a broadcast-domain technology with a specific

congestion avoidance mechanism, high-speed today’s Ethernet

based on full-duplex links connected by switches can support

arbitrary topologies.

Having started in local area networks operated by the customer,

basic Ethernet provides absolutely no service assurances and

is decidedly not carrier-grade. However, Ethernet has been

upgraded to “Carrier Ethernet” by adding mechanisms such as

network management, OAM and QoS, protection switching,

timing, and security. Carrier Ethernet is today a multi-billion dollar

market with huge-scale deployments by all major communications

service providers.

On the other hand, Ethernet has also been adapted to industrial

automation applications by supplementing it with ruggedized

connectors and extended temperature switches, as well as

protocols that provide determinism, resilience and real-time

control. Industrial Ethernet has also found use in power utilities

substations.

APPENDIX: Packet Switched Networks

White Paper

18

MPLSMPLS is a thin layer, most often residing between the Ethernet

and IP layers, originally designed to accelerate the forwarding

of IP packets in large networks. MPLS packets do not contain

addresses, but instead a stack of labels that serve to locally

direct forwarding.

MPLS now has four distinguishable flavors (differing in control

protocols):

1. Best effort MPLS (sometimes called IP/MPLS), as used in the

core of the Internet, is a best effort technology which relies

on the IP routing system, the MPLS layer serving solely to

accelerate forwarding;

2. MPLS for L3VPN services is a popular offering to deliver VPN

services to businesses by providing service isolation;

3. MPLS-TE is MPLS augmented with traffic engineering and

resource reservation, and is used when true service level

guarantees are required;

4. MPLS-TP supplements MPLS with network management, OAM

and QoS, and protection switching, in order to offer a drop-in

replacement for Carrier Ethernet. MPLS-TP forwarding may be

determined by IP routing or may be entirely configured by a

centralized management system. MPLS-TP functionality may

be combined with MPLS-TE.

APPENDIX: Packet Switched Networks

White Paper

Specifications are subject to change without prior notification. The RAD name, logo and logotype are registered trademarks of RAD Data Communications Ltd. ©2017 RAD Data Communications. All rights reserved. Version 6/17

Your Network’s Edge

For more information visit www.rad.com