QoS Enablers for Next-Gen Networks
Krishan Sabnani
Bell Labs
04/13/23
2
Today’s Data Networks
AccessRouterAccessRouter
3G CellularNetworks
RadioController
RadioController
AccessRouterAccessRouter
EnterpriseNetworks
HomeNetworks
MetroNetworks Packet-Based
Network
EdgeRouterEdge
Router
Edge RouterEdge Router
EdgeRouterEdge
Router
AccessRouterAccessRouter
Ad hocNetworks
Not QoS enabled, secure, manageable, and reliable
04/13/23
3
The Network Evolution
Networks were designed to carry voice traffic
Data traffic mostly overlaid on voice networks (using Modems)
Networks are designed to carry primarily data traffic
Voice traffic overlaid on data networks (e.g. VoIP)
Future networks should be designed primarily for efficient content distribution and content search/location
– Content distribution should not only be overlaid, but built in from ground up
Future networks should also be able to effectively carry best-effort data traffic and QoS-sensitive voice.
Yesterday… …Today...
…Tomorrow ...
Volume of data trafficexceeds voice traffic
Content trafficbecomesdominant
04/13/23
4
Impact of Video on Network Traffic
VoD/IPTV will grow by an order of magnitude over the next 5 years
– 2005: 90% best-effort data traffic
– 2010: 40% high-priority VoD traffic; 50% best-effort traffic
– Carrier-class links with capacities of 40G and 100G required in MAN
Broadcast IPTV and streaming VoD is high-priority traffic, close to TDM
– Very different from today’s best-effort bursty data traffic
Source: Lucent Bell Labs IPTV/VoD study, May 2006
04/13/23
5
Tomorrow’s Converged Network
EdgeRouterEdge
Router
AccessRouterAccessRouter
AccessRouterAccessRouter
3G CellularNetworks
RadioController
RadioController
AccessRouterAccessRouter
EnterpriseNetworks
HomeNetworks
Next-GenMetro
Networks
4G/Mesh
UserMobility
Traffic Type(Multimedia)
Qualityof Service
(e.g. for voice)
Network Intelligence
QoS-EnabledPacket Core Network
EdgeRouterEdge
Router
EdgeRouterEdge
Router
Services Enablement Layer
Always-OnAlways-OnGlobal RoamingGlobal Roaming
PersonalizationPersonalization
04/13/23
6
Enabling technologies
Future converged network will put significant demands on next generation network elements
– Carrier-grade and increased complexity demands
• QoS support, Scalability, Reliability, Security, Manageability
– SoftRouter
– Capacity demands
• Switching capacity > 100Tb/s will be required in a single switching element
– Optical Data Router
– High-speed wireless data access
• Simplify RAN to improve QoS
– Base Station Router
04/13/23
7
Motivation
Future converged network will put significant demands on next generation network elements
– Carrier-grade demands
• QoS support, Scalability, Reliability, Security, Manageability
– SoftRouter
– Capacity demands
• Switching capacity > 100Tb/s may be required in a single switching element
– Optical Data Router
– High-speed wireless data access
• All wireless services will converge to IP
– Base Station Router
04/13/23
8
Routers are becoming increasingly complex
Complexity is an IP “Middle-Age” problem!
– IP provides end-to-end datagram delivery service to protocols/applications
– IP can use any link-layer technology that delivers packets
Emerging applications are driving more functions into IP, expanding the “waist” of the IP hour glass
Router vendors incorporate all new IP functions into routers
Complexity is spread throughout the network
• Achieving network-wide objectives such as traffic engineering requires complex translation of global objectives to configuration information in numerous individual routers
• Misconfiguration or uncoordinated configuration can result in poor performance or even network instability
email WWW phone...
SMTP HTTP RTP...
TCP UDP…
IP
ethernet PPP…
CSMA async sonet...
copper fiber radio...
email WWW phone...
SMTP HTTP RTP...
TCP UDP…
IP
ethernet PPP…
CSMA async sonet...
copper fiber radio...
IPmobile mcast
IPsecdiff-serv
NAT
04/13/23
9
Solution: SoftRouter
Disaggregation of router hardware from software addresses this problem and has the potential for major additional advantages
Bell Labs has a research program that disaggregates router control and transport planes (called SoftRouter-based approach)
• Transport plane: packet forwarding element
• Control plane: control element server and feature server
• Control plane servers and transport plane communicate using standard protocols
• Approach similar to SoftSwitch-based disaggregation of class 5 switches
04/13/23
10
New Router Architecture: SoftRouter
3 key components of SoftRouter approach– Decoupling: Separate complex control plane processing from the transport
plane
– Servers: Implement control plane processing functions on dedicated external control plane servers
– Standard Interface: Define standard protocol for control plane servers to interface to the forwarding elements
Control plane
processing
Forwarding plane
processing
ProprietaryAPI
Standardprotocol
Current Router Model SoftRouter Model
Control Plane
Transport Plane
FeatureServer
PacketForwarding
Element
Control ElementServer
04/13/23
11
SoftRouter Network Architecture
Router
PacketForwarding
Element
ControlElementServer
Traditional Router-based Network
SoftRouter-basedNetwork
The SoftRouter approach separates and centralizes the software-intensive control element and feature servers from
hardware-centric transport and packet forwarding
FeatureServer
04/13/23
12
SoftRouter Benefits
1. Lower Costs
– Commoditized, standards-based hardware (lower capex)
– Dedicated control element servers imply fewer management points (lower opex)
2. New Features
– Network-based features to support new services more easily added using open APIs
– Incremental deployment made simpler through centralized management
3. Better Scalability
– Centralized control element servers easier to scale using well-established server scaling techniques
4. Enhanced Stability, Controllability, and Reliability
– Network instability problems due to BGP Route Reflectors are eliminated
– SoftRouter-based network can be designed to be more reliable than a traditional network
5. Increased Security
– Fewer control element servers easier to secure using perimeter defense systems, e.g., firewalls
04/13/23
13
Technical Challenges: Summary
Protocol aggregation: how would protocols like OSPF/BGP operate when a single protocol instantiation at the control element server manages multiple forwarding elements?
– OSPF: Preliminary results indicate 50ms failure recovery time is feasible when the SoftRouter network is managed by one or two primary CE/OSPF processes and the propagation delay from CE to its FEs is small
– BGP: Full I-BGP mesh can be maintained among the few control servers, eliminating network instability possible under the Route Reflector architecture
Network design: where do we place the control element servers and how do we determine which control element servers manage which forwarding elements?
– Algorithm based on recursive graph bisection appears to work reasonably well in identifying where to place the CEs and which set of FEs that each CEs manage
Bootstrapping paradox: the forwarding element needs updated forwarding tables in order to route packets to its control element server but only the control element server can update the forwarding tables
– We develop a new discovery protocol to break the above circularity
– This protocol allows each FE to bind to its best CE and provides simple routing capability between them
04/13/23
14
Motivation
Future converged network will put significant demands on next generation network elements
– Carrier-grade demands
• QoS support, Scalability, Reliability, Security, Manageability
– SoftRouter
– Capacity demands
• Switching capacity > 100Tb/s may be required in a single switching element
– Optical Data Router
– High-speed wireless data access
• All wireless services will converge to IP
– Base Station Router
04/13/23
15
Router Capacity over Time
Growth in Router Capacity
Internet traffic doubles every 12 months
Router Capacity doubles every 18 months
Routers will reach 100Tb/s in 2010
Single Rack <1Tb/s Multi-rack solutions >1Tb/s
e.g. Juniper TX640or Cisco XR12416
e.g. Juniper TX Matrix or Cisco CRS-1
Routers2x every
18 months
Internet2x every
12 months
Source: Nick McKeown, Stanford
Currently Third Generation of Packet Routers : 1Tb/s capacity in a Single Rack
04/13/23
16
Why do current packet routers not scale?
Power and heat dissipation limit density.
– Require multiple shelves and racks of Line Cards
– Increases distance between Line Cards and Switch Fabric
SwitchFabric
CentralScheduler
ToNetwork
Line Cards
PacketProcessing
Buffer
Optics
I/OPacket
Processing
Buffer
Optics
I/OPacket
Processing
Buffer
Optics
I/OPacket
Processing
Buffer
Optics
I/O
Centralized Switch Fabric
– Switch Fabric alone pushes power density and heat dissipation limits
– Massive wiring density between Line Cards and Switch Fabric
Centralized Control and Scheduling complexity grows nonlinearly
Centralizedfunctions
scale poorly
Massive Wiring
04/13/23
17
SwitchFabric
CentralScheduler
ToNetwork
Line Cards
PacketProcessing
Buffer
Optics
I/OPacket
Processing
Buffer
Optics
I/OPacket
Processing
Buffer
Optics
I/OPacket
Processing
Buffer
Optics
I/O
Bell Labs Solution: Distributed Optical Switch Fabric
Optical fibers eliminate massive wiring density and allow higher bandwidths
CentralScheduler
ToNetwork
Line Cards
PacketProcessing
Buffer
Optics OpticsPacket
Processing
Buffer
Optics OpticsPacket
Processing
Buffer
Optics OpticsPacket
Processing
Buffer
Optics
T-TX Optics AWGFiber
Central Switch fabric replaced by passive optical device that directs signals according to their wavelength
– Called Arrayed Waveguide Grating (AWG) Switching using Fast Wavelength Tunable Transmitters (T-TX)
distributed on each line card– Deployed hardware scales with deployed capacity– Passive Optical Backplane
Fixes HardwareScaling
Passive Optical Backplane
04/13/23
18
Bell Labs Solution: Distributed Scheduler
Eliminate central scheduler without sacrificing throughput
– Distribute traffic uniformly to Line Cards
– Technique called “load balancing”
ToNetwork
Line Cards
AWG
PacketProcessing
Buffer
Optics Optics
LocalScheduler
PacketProcessing
Buffer
Optics Optics
LocalScheduler
PacketProcessing
Buffer
Optics Optics
LocalScheduler
PacketProcessing
Buffer
Optics Optics
LocalScheduler
Each line card works like a small router with its own scheduler– Scales gracefully since complexity of scheduling is constant– Deployed hardware scales with deployed capacity
Combined with Distributed Optical Switch Fabric allows scaling to very large routers
RouterScales to >100Tb/s
Lucent Technologies – Proprietary
Use pursuant to company instruction
04/13/23
19
Optical Data Router
DARPA-funded program Linecards
– Highly integrated optical components• Denser integration of optical interfaces on line
card
• Data never converted to electronics for lower power
– Shallow Buffers• Demonstration of high throughput with buffers
20 packet deep
Scalable switch fabrics– Highly scalable nonblocking switch fabrics
• 5Tb/s using AWG plus tunable lasers and 100Gb/s Transmitter
– Use of load balancing to allow blocking switch fabrics to be used and simplifies scheduling
• Scalability to 256 Tb/s throughput
• Transparent optical packet router demonstrated
100Gb/s Packet
switching
Space SwitchSpace Switch Time Buffer
AWG3
drop
WC1
WC2
AWG1
WC3
WC4
AWG4
AWG2
TL4
TL3TL1
TL2
MZM EAM PD
PD
40GPG
40GCLK
40GBERT
NR
ZD
ata
RR
RR
TB
Sch
edul
e
FDLs
CSG
Optical Packet Router
Highly integrated photonic
chips
Effects of shallow buffers
04/13/23
20
The Optical Data (IRIS) Router in a Carrier Network
IRIS intermediate
node
IRIS intermediatenode
IRIS Router
Riverstone15008
IRISintermediate
node
Riverstone15008
IRISedgecard
IRISedgecard
10 GbE
10GbE
40 Gb/s optical packets
Riverstone15008
Riverstone15008
10GbE 10
GbE
04/13/23
21
Motivation
Future converged network will put significant demands on next generation network elements
– Carrier-grade demands
• QoS support, Scalability, Reliability, Security, Manageability
– SoftRouter
– Capacity demands
• Switching capacity > 100Tb/s may be required in a single switching element
– Optical Data Router
– High-speed wireless data access
• All wireless services will converge to IP
– Base Station Router
04/13/23
22
Current wireless networks are complex, involving many network elements, and result in high cost and high latency
Base Station Router terminates all air interface-specific functionality in the base station
Base Station Router: push intelligence to the edge
Collapsing Radio Access Network elements into the base station simplifies network and reduces latency
Pushing IP intelligence to the base station results in better Quality of Service support
Base Station
O
MobileSwitching
Center
Base Station
O
MobileRouter
O
RadioController
PacketBackhaul
Packetbackhaul
Telephone Network
Internetpacketdata
circuitvoice
Base Station Router
O
MobileRouter
04/13/23
23
BSR Motivation
BSR provides advantages in the following areas
reduces system OpEx by 75%
increases VoIP Capacity by 25%
reduces delay by 40%
O
OO
O
PSTN
IP
OO
OOOO
O
PSTNPSTN
IPIP
BSR
Collapse access, control, and gateway elements to a single IP Access Point
AccessControl
Gateway
04/13/23
24
Lucent Technologies' Base Station Router Receives CTIA Emerging Technology Award
Revolutionary Product Takes Top Honors for Most Innovative In-Building Solution
LAS VEGAS – Lucent Technologies (NYSE:LU) today announced that its Base Station Router (BSR) product was selected as the first place winner of a CTIA WIRELESS 2006 Wireless Emerging Technologies (E-tech) Award in the category of “Most Innovative In-Building Solution.” Award recipients were announced yesterday in a ceremony at the Las Vegas Convention Center during the CTIA WIRELESS trade show.
The Wireless E-tech Awards program is designed to give industry recognition and exposure to the best wireless products and services in the areas of Consumer, Enterprise and Network technology. Nearly 200 applications were submitted and reviewed by a panel of recognized members of the media, industry analysts and executives, as well as select show attendees. Products were judged on innovation, functionality, technological importance, implementation and overall “wow” factor.
04/13/23
25
State of QoS Deployment and Research
04/13/23
26
Typical Objections to QoS Deployment
QoS can be avoided by over-provisioning– May work for core network, not true for access network (difficult to add
capacity) and wireless network (limited air spectrum)
– Instantaneous congestion can still occur, disrupting flows that have strict QoS requirements
QoS is too expensive to implement– Range of QoS mechanisms available, some scale better than others
– Fine grain QoS at the edge and aggregate based QoS in the core
QoS is too complicated to manage– Results show that many benefits of traffic management can be achieved by
placing QoS-awareness in a small (5-10%) number of network elements
04/13/23
27
State of QoS Deployment
Core: MPLS, some with traffic engineering, is widely implemented in core networks
Access: Most access network elements have built-in QoS features such as policing. New services such as VoIP/IPTV will need better QoS support such as diff-serv.
Enterprise: Enterprises are increasingly deploying QoS-aware boxes at their edges. Main functions are application prioritization and acceleration.
Cellular networks: Limited bandwidth has made QoS Support an absolute must in air interfaces.
– Air interfaces use mechanisms to ensure fairness among mobiles.
– Radio access networks (RAN) differentiate among mobile flows.
04/13/23
28
EthernetAccess
DSL/CableAccess
TelephoneAccess
802.11Access
CircuitMobile Network
PacketMobile
Network
IAD
IP/MPLS IP/MPLS BackboneBackboneIP/MPLS IP/MPLS BackboneBackbone
Focus on Focus on Core networkCore network
Focus on Focus on Core networkCore network
Core Networks are implementing MPLS
MPLS label-switched paths
Bandwidth guaranteed
Strong traffic isolation and flexible routing
Allows traffic to be routed away from congestion points
Key tools for traffic engineering and policy-based routing
04/13/23
29
Access Networks are implementing Differentiated Services
Differentiated services: Connection less IP-QoS mechanism– Many traffic classes
• EF: Expedited Forwarding
• AF: Assured Forwarding (several classes)
• Best-Effort
– Priority scheduling among different classes• E.g., EF traffic given highest priority, current best-effort traffic carried as
lowest priority
– Admission control for each class• E.g., Overall levels of EF traffic in network kept low to avoid starvation
– Network traffic shaped at edge using leaky-bucket controllers
– Network routing can be determined by IP routing protocols\
Pros:– No per-flow or per-LSP state needed in network
Cons:– Difficult to provide bandwidth guarantees finer than aggregate level
04/13/23
30
Enterprises are implementing Application Acceleration Deploy deep packet inspection at enterprise edge routers to
– Classify packets by applications
– Provide prioritization among different application packets
– E.g., VoIP flows given priority over email traffic
Some large corporations implementing large-scale VPN or even private network to enable control over its internal routing
– E.g., Google
04/13/23
31
Wireless NetworksNeed More Than Diff-Serv and MPLS Air Interface: Time-varying bandwidth constrained channel
– Diff-serv and MPLS only differentiate between traffic classes
– Wireless scheduling needs to take into account the channel conditions for maximizing cell-capacity: fundamentally different from scheduling diff-serv and MPLS traffic on wireline links
– New scheduling mechanisms use information on channel condition and current queue-lengths to maximize throughput while maintaining QoS
Radio access network: Low bandwidth network transports control, real-time, streaming, and best-effort traffic– Requires QoS mechanisms but all traffic is encapsulated in tunnels
– Diff-serv and MPLS mechanisms can be used if different traffic classes are carried over different tunnels and not on the same tunnel
04/13/23
32
Provides minimum and maximum user throughput bounds
Provides “hogging prevention”
Used for inter-user throughput allocation. (Can be used for intra-user as well)
Proportional Fair with Minimum Rate (PFMR) Algorithm
0)( if
0)( if )(
otherwise 0
slot in served user if )()(
)()()()1(
)(
)(max
max
min
)(
nTR
nTRnX
ninDRCnr
nrnXnTnT
enR
nDRC
ii
iii
ii
iiii
nTa
i
ii
ii
miniRmax
iRir
0Research work of:Research work of:
Alexander Stolyar; Matthew Andrews; Lijun Qian (Bell Labs)Alexander Stolyar; Matthew Andrews; Lijun Qian (Bell Labs)Research work of:Research work of:
Alexander Stolyar; Matthew Andrews; Lijun Qian (Bell Labs)Alexander Stolyar; Matthew Andrews; Lijun Qian (Bell Labs)
Ti is a virtual “token queue” corresponding to each flow i
Tokens arrive in the token queue at the rate Xi which is Ri
min or Rimax per slot
If user i is served in slot n, then DRCi (n) tokens are removed from the queue
If in a time interval, the average service rate of flow i is less than Ri
min, the token queue size has a positive drift, so the chances of flow i being served in each time slot gradually increase
Ti is a virtual “token queue” corresponding to each flow i
Tokens arrive in the token queue at the rate Xi which is Ri
min or Rimax per slot
If user i is served in slot n, then DRCi (n) tokens are removed from the queue
If in a time interval, the average service rate of flow i is less than Ri
min, the token queue size has a positive drift, so the chances of flow i being served in each time slot gradually increase
04/13/23
33
Provides minimum and maximum user throughput bounds
Provides “hogging prevention”
Provides individual stream minimum throughput and latency control
Provides unified mechanism for inter-user and inter-flow (intra-user) resource allocation. This has its pluses (universality, higher overall throughput) and minuses (complexity)
Key to supporting VoIP in Ev-DO networks
PFMR with Latency Control (PFMR-LC) algorithm:PFMR with Latency Control (PFMR-LC) algorithm:
04/13/23
34
State of QoS Research
QoS research is as old as networking itself
– Lots of papers have appeared, most have analytical focus
Most skeptics of QoS are also in academic
– QoS mechanisms are being implemented and used, adoption is mostly driven by real-world requirements, not by elegance of the theory
Is more QoS research needed?
– How QoS mechanisms can be implemented in the networks of future?
– Inter-domain QoS peering
04/13/23
35
Net Neutrality
Should transport providers be allowed to differentiate traffic transported over their networks?
– Verizon, AT&T, Comcast, … : ability to offer service differentiation provides the incentive for large capital investment in building out new access networks
– Google, Yahoo, Microsoft … : service differentiation can be used by transport providers to favor their own traffic or traffic from their favored content aggregators, giving them an unfair advantage over 3rd party content providers, and thus slowing competition and innovation
My position:
– Transport providers should offer services with QoS
– Such services should however be universally available to all content aggregators
04/13/23
36
Conclusions
Network convergence is happening QoS is a necessity for next generation converged networks Bell Labs has several programs to produce key assets for creating next
generation converged networks– SoftRouter– Optical Data Router– Base Station Router
QoS is being deployed in service provider networks and enterprise networks
QoS research is still needed – Inter-domain QoS– Simpler implementation
Top Related