Entering the Next Era in Internet2 Transport: Bandwidth and Latency Issues - Solved Today, and Solved Tomorrow

Stephen Smithstephen.smith@us.fujitsu.com

Product and Technology MarketingFujitsu Network Communications

April 2012

Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Problem Statement All of the communication’s pundits are projecting exponential growth in data

services. This likely applies to Academia as well general commercial growth

SONET is not sufficient to support higher bitrate wavelengths beyond 10G

If SONET is being capped, what is the next generation network? Is it a pure packet network? Is it a pure next generation TDM network like OTN? Is it a combination of both?

What are some of the strengths and weaknesses of these networks?

What is the most cost effective Network Solution that will scale for the future?

2Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Content + Mobility + Cloud = Big BandwidthBandwidth Predictions

It would take over 5 years to watch the amount of video that will cross global IP networks

every second in 2015.

Internet video is now 40 percent of consumer Internet traffic, and will reach 61 percent

by the end of 2015.

Globally, mobile data traffic will increase 26 times between 2010 and 2015.

The number of devices connected to IP networks will be twice as high as the global

population in 2015.Sources + Cisco VNI, 2011.

-

2,000

4,000

6,000

8,000

10,000

12,000

14,000

2008 2009 2010 2011 2012 2013 2014 2015 2016

Gbps

/ y

ear

Traditional Phone 3G Smart Phone 4G Smart Phone Aircard/Hostspots Tablets

Source : UBS 1Q11 – N. America Wireless Demand by device

3Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Requirements for the Network(On Campus and Transport between Campuses)

Full Transparency Options Minimum Latency Minimum Jitter 1:1 and Mesh Redundancy Full network visibility and remote trouble isolation

capabilities Maintaining SLAs across multiple domains Security (separation between customer and management

planes) Minimum First Cost, minimum Operational Costs Scalable from a 1X, 10X, and 100X Support of Legacy Services

4Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Today’s IP/MPLS Services VPLS Private IP Public IP L3 – VPN Telepresence

Likely has an OTN/Photonics layer underneath the routers that can be

utilized to expedite traffic

PublicIP

PublicIP

PIPPIP

vBNS+vBNS+

CPA -EVPLVPLS

CPA -EVPLVPLS

OTN

ROADM

OTN

ROADM

OTN

ROADM

OTN

ROADM

OTN

ROADM

5Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Effects of Latency and Delay Some services have very strict latency and delay requirements

VM Migration Financial Services – Stock trading Gaming Two Way Video applications Remote Health services Strict SLA (QoS) across multiple Domains General TCP throughput degradation with Latency

These applications can be severely hindered or even denied if latency and/or jitter become large Want a network where latency is deterministic and known under a standard

working condition and under a fault condition Latency Inflation (where latency can vary between 20 and >100ms within a

day’s time) can be problematic for some services

6Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Cloud Based Virtual Machine Migration

Being able to migrate Virtual Machines to optimize performance or minimize power usage without the customer realizing the move occurred

Requires Very low latency so that customer’s experience is unchanged with the migration

Virtual Machine A

Virtual Machine B

Migration

Synchronous ReplicationRound Trip Delay - Less than 10msJitter - less than 2.5ms

Source: IBM/Cisco SANMultiprotocol RoutingIBM Redbook SG24-7543-01

7Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Financial Transactions Low latency is a feature directly tied to the core business process of trading,

says Steve Kammerer, IPC VP. “And that means low latency is the priority.” “Missing the transaction by just a nano-second could cost the financial

institution money”, says Optimum Lightpath VP Glenn Calafati.

Chicago Stock Exchange

NY Stock Exchang

e

Los Angeles Stock

Exchange

Transport Network

Latency for TradingRound Trip Delay - Less than 10msLow latency is critically important in the options market, and in the coming years it will only become more so.  Latency is already being reduced at each stage of the trading process but at increments and levels of priority that vary by firm. Options pricing and analytics will be shaved from minutes to seconds, market data will be disseminated in single- rather than double-digit milliseconds, and trading opportunities will be identified and acted upon within microseconds. The timelines to reaching these goals, too, are constantly being shortened.

Source: http://www.tabbgroup.com/PublicationDetail.aspx?PublicationID=401&MenuID=14&ParentMenuID=2&PageID=9

8Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Gaming

Gamers in Dallas

Gamers in Austin

Transport Network

Gamers in Houston

Latency Can Kill: Precision and Deadline in Online Games, Mark Claypool, et.al

Avatar: First Person (Players shoot directly at each other)Round Trip (<100ms)Round Trip (60ms) - has shown to affect player's accuracy"An evaluation of Problems and Solution of Latency in Online Games", Gert Scholten, January 31, 2008.

9Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Two Way Video applications Latency

Two-way interactive communication is sensitive to delays in the network. 300ms of lag causes users to resort to one-at-a-time, walkie-talkie-style conferencing to

communicate.

Jitter Causes irregularities in the flow and delivery of data. Even 100ms of jitter causes conferencing quality to suffer

Source: Optimizing Video Performance Across the Distributed Enterprisesuffer, Blue Coat Whitepaper

Transport Network

Two Way Video (includes encoding/decoding/transport)One Way Delay<400ms with Echo suppressor<150ms (preferred) with Echo suppressor<80ms with Lip SynchronizationSource: ITU G.1010

10Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Remote Health services Tele-surgery

Delay in sensor feedback can distract the surgeon and cause serious safety hazard Varying latency significantly reduces the operators’ performance both with robotic

telesurgery and virtual reality (VR) applications (Thomson et al., 1999).Source: Extreme Telesurgery, Tamás Haidegger and Zoltán Benyó,

Budapest University of Technology and Economics, Hungary

Tele-diagnostic Interactive video communication requires low delay of 200 to 300 ms round-trip and an

average jitter that is not more than 30 ms Speech latency should be less than 200-300 ms and jitter must be limited to 50ms (Cisco

Systems, 2002; Sze et al., 2002; Hassan et al., 2005; Tobagi, 2005).QoS in Telemedicine, Phumzile Malindi, Walter Sisulu University, South

Africa

Telepresence (Remote Surgery (Video)One Way Delay < 120msSource: MEF, Implementation Agreement MEF 23.1, Carrier Ethernet Class of Service - Phase 2, January 2012.

11Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

TCP Throughput Degradation with Latency

1GE Client Port TCP Window size is

65536 Bytes Source:

http://www.babinszki.com/Networking/Max-Ethernet-and-TCP-Throughput.html0 100 200 300 400 500 600

0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

900,000

TCP Throughput

Round Trip Latency (ms)

TC

P T

hro

ug

hp

ut

(Kb

ps)

0 2 4 6 8 10 120

100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

900,000

TCP Throughput

Round Trip Latency (ms)

TC

P T

hro

ug

hp

ut

(Kb

ps)

20 40 60 80 100 120 140 160 180 200 2200

2,0004,0006,0008,000

10,00012,00014,00016,00018,00020,000

TCP Throughput

Round Trip Latency (ms)

TC

P T

hro

ug

hp

ut

(Kb

ps)

12Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Latency Fluctuation with MPLS-base TE

IP/MPLS utilizes Traffic Engineered (TE) based tunnels Most of these tunnels are dynamic in nature

Algorithms are dynamically run to optimize the tunnels for Shortest Path The tunnels can carry any traffic that is being demanded and can change the size of

their tunnels according to the bandwidth demand

This dynamic aspect causes variances in latency and jitter When the tunnels adjust to the bandwidth demands, they can incur radical latency

fluctuations which can cause large step functions in their latency (on the order of 50ms or more). This can occur at any time.

Source: Latency Inflation with MPLS-based Traffic Engineering,

Abhinav Pathak, Purdue University

13Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Jitter in General

As traffic traverses different tunnels, jitter is incurred: Anytime queuing occurs which happens at different speed

interfaces

To have a low jitter network, need to minimize the number of queues traversed or increase the latency with jitter buffers

14Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Local Aggregation with Packet and OTN

Non Ethernet

Private Line

Non Ethernet

Private Line

Non Ethernet

Private Line

Non-Ethernet based Private Line Traffic

OTNMux

Ethernet Links between Packet Devices

Campus Aggregation

Area

Building 1

Building 2

Building 3

Building 4

Building 5

Building 6

Building 7

Building 8

Engineering

Nursing

Administration

OTN Tunnels headed to different destinations

15Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

CAAOTN NE

CAAOTN NE

CAAOTN NE

CAAOTN NE

CAA

OTN NE

Transport with OTN/Photonics layer

CAAOTN NE

CAAOTN NE

CAA

OTN NE

Hospital Network

Off Campus Research Data

base

Internet PoP

Carrier Aggregation Area (CAA)

OTN NE

ROADM with OTN Switching Network

OTN

ROADM

OTN

ROADM

OTN

ROADM

OTN

ROADM

OTN

ROADM

16Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Advantages of Two ArchitecturesItem Pure IP/MPLS COE/OTNLatency AdvantageDelay Variation AdvantageAggregation (Highest Aggregated Pipes)

Advantage

Ability to backhaul non-Ethernet based traffic

Advantage

Transparency AdvantageRedundancy (Ability to switch within 50ms)

Advantage

Segmentation for purposes of Troubleshooting (Allows for non-intrusive loopbacks in all nodes of network)

Advantage

Security AdvantageCost (L1/L2 is more cost effective than L2.5/L3)

Advantage

Scalability (Ability to economically address growing market)

Advantage

Support of Legacy Services (Ability to transport SONET/SDH, FC, Etc.)

Advantage

17Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

ApplicationsItem MPLS COE/OTNVM Migration Bypass with COE/OTN due to low latency

requirementsFinancial Services (Stock Trading) Bypass with COE/OTN due to low latency

requirementsGaming Bypass with COE/OTN due to low latency

requirementsMulti-way Video Bypass with COE/OTN due to low latency

requirementsRemote Health Services Bypass with COE/OTN due to low latency

requirementsStrict SLA across Multiple Domains Use OTN to maintain SLA’s through third

party DomainAbility to offload OTT video traffic Bypass with COE/OTN due to high capacity

and scalability concernsNeed for high Throughput with TCP traffic Bypass with COE/OTN due to latency

concernsPublic IP / Private IP services No need to Bypass unless have strict latency

requirementsL3 – VPN / VPLS No need to Bypass unless have strict latency

requirementsTelepresence Bypass with COE/OTN due to low latency

requirements

18Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Summary

Universities today are utilizing an IP/MPLS network for campus services IP/MPLS networks have an OTN/Photonics layer underneath

Some services are difficult to transport over a Packet network Use the OTN/Photonics layer to transport these services

Video will be increase 6 fold (from 2010 to 2015), dominating Internet traffic This could cause scaling issues within the network Can use the OTN/Photonics layer to bypass the MPLS network for the OTT

video

A COE/OTN network will efficiently aggregate and transport traffic COE to sufficiently aggregate “same destined” traffic together OTN to transport to the core. Once at the core, further aggregate as needed

Exploit the lower layers as much as possible (Layer 0/1/2) to save power, capital, and operations costs

19Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

Recommendation

As campuses builds out their IP/MPLS network and move towards the Internet2, ensure that there is a COE/OTN/Photonics layer underneath to aggregate, bypass and expedite traffic, while providing the needed scale at the lowest costs

20Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.

21Copyright 2010 FUJITSU LIMITED 21Fujitsu Proprietary and Confidential All Rights Reserved, ©2012 Fujitsu Network Communications Inc.