Sky Core Network
Positioning for Massive Video Demand
UKNOF - 17 Jan 2018
Tim Rossiter – Core Network Architect, Sky Network Services
Sky Core Network , UKNOF, 17Jan 2018 - 2
Positioning Sky Network for Massive Video Demand
• UK Broadband Usage and Pricing
• Sky Network Structure and Evolution
• 2017/18 Capacity Upgrade
• Cost Reduction by Content Delivery (CDN) Localisation
Sky Core Network , UKNOF, 17Jan 2018 - 3
Uk & International Historic Traffic Trends
UK 58%
90% in 2016
40% in 2017
YoY Growth
USA 20%
Source: Ofcom International & UK Communication Market Reports
Sky Core Network , UKNOF, 17Jan 2018 - 4
Volume - Sky Traffic Growth (10 Year History)
• Sky Experience Similar to UK
• Total Traffic x100 over 10 years => 58% YoY
• Per Sub traffic x25 => 38% YoY
Sky Core Network , UKNOF, 17Jan 2018 - 5
• Internet connectivity cost ~ £30 – £50 p.m (also inflation ~50% )
• Access Data rates increased x 100 in 12 years
• Cost per bit Reduces x 67% YoY, or 10 ->1 in 6 years.
• ISP per sub revenue unchanged, Access rate increased by x100
UK Broadband Price Trend
Since 2000
Sky Core Network , UKNOF, 17Jan 2018 - 6
P&T
P&T
Sky Network Structure (Simplified)
Super
CoreSuper
Core
P Router
NCS6k
PoP PE
Rem
CDN
CDN
Core
CDN
PoP PE
PoP
CDN
Exch
Exch
Exch
Exch
CPE
Exch Exchange
DSL or GEA
Sky Core Network , UKNOF, 17Jan 2018 - 7
Resilience and Provisioning Policy
Resilience – 1 + 1 or 1 : N
• Dual Rings, Dual feed to each PoP
• Dual P&T sites
• Multiple CDN (1:N )
• 2 or 3 Entry points to each Super-Core site
Dimensioning
• Serve peak traffic in the event of a single major failure
• Major failure: Loss of one entire site or fibre (up to 80 layer 3 paths)
Capacity Consequences
• Largest Router Capacity ≈ Traffic Served
( In+Out ) x Protection + Interconnect + Extras – Partial Fill
• 8 Tb Traffic Requires 8 Tb Router
Sky Core Network , UKNOF, 17Jan 2018 - 8
Core Router Evolution
2005 Initial Cisco CRS1
• Traffic < 50G, 40Gb per slot
=> 640 Gb chassis
2011 Upgrade to CRS3. Fabric, Line cards
• Traffic 350G, 140Gb per slot
=> 2240 Gb chassis (x3.5)
2014 - Replace with NCS6008
-> New Chassis, Fork Lift
• Traffic 2 Tb, 8Tb per chassis (x12.5)
2017/18 - Add NCS5516
-> New Chassis
• Largest NCS6008 almost full
• Traffic ~ 8 Tb, 55 Tb per chassis (x 90)
CRS1
CRS3
NCS6k
NCS6k B2B
NCS5k
• Retain NCS6008 in Back to Back Config, 16 Tb per chassis
Sky Core Network , UKNOF, 17Jan 2018 - 9
NCS6008 and NCS5516NCS6008
• 8 Slots, 1Tb per slot => 8 Tb node
• 100G or ( 10x10G) ports
• High Functionality
• Roadmap: B2B, 2Tb per slot => 16-32 Tb
NCS5516
• 16 Slots, 3.6Tb per slot => 55 Tb node
• 36 x ( 100G or (4 x 10G) ) ports/slot
• Low touch – MPLS P-Router, No B2B
Sky Core Network , UKNOF, 17Jan 2018 - 10
NCS6008 Back to Back
• 96 x 100G CXP SR10 Connectors => ~1000 Fibre Pairs!
• 2 x 40G Ethernet for processor links
• Shelves can be ~ 100m apart
Sky Core Network , UKNOF, 17Jan 2018 - 11
• CFP Original 100G
Bulky, Expensive
• CPAK,
Cisco Proprietary
• CXP SR10 (10x10G)
• QSFP-28 (4x25G)
– Standard SR4, LR4 + Cost Effective Variants
– Large Quantities (Data Centre Usage)
– SR-4 very low cost
CPAK
100G Optics – CFP, CPAK, QSFP
Sky Core Network , UKNOF, 17Jan 2018 - 12
Matching Broadband Cost Trend ?
• Port savings: 10G -> 100G, Router silicon, Moore’s Law, Reduced Functionality, LR->SR
• NCS5516 + low cost QSFP SR4 nearly matches BB price erosion, 10 : 1 in 6 years
Sky Core Network , UKNOF, 17Jan 2018 - 13
Migration Steps for NCS5516 in Super-core
• Different approach from previous NCS6008 ‘whole ring’ introduction
– Growth slower => Longer timescale for transitions
– No ‘10G -> 100G’ transition issues
– Super-core transmission fully 100G
– Not a ‘big bang’ migration
• Upgrade one node at a time (in multiple stages)
• Birmingham first
• Staged migration modelled to verify capacity for all failures
Sky Core Network , UKNOF, 17Jan 2018 - 14
P&T
P&T
Migration 1: Insert NCS65516
Super
Core
Birmingham
P Router
NCS6k
PoP PE
CDN
Core
CDN
PoP PE
Exch
Exch
Exch
Exch
CPE
NCS5k
Sky Core Network , UKNOF, 17Jan 2018 - 15
P&T
P&T
Migration 2a: Migrate some feeds and consumers
Super
Core
Super
Core
P Router
NCS6k
PoP PE
CDN
Core
CDN
PoP PE
Exch
Exch
Exch
Exch
CPE
NCS5k
Sky Core Network , UKNOF, 17Jan 2018 - 16
P&T
P&T
Migration 2b, 2c … : Migrate remaining feeds and consumers
Super
Core
Super
Core
P Router
NCS6k
PoP PE
CDN
Core
CDN
PoP PE
Exch
Exch
Exch
Exch
CPE
NCS5k
Sky Core Network , UKNOF, 17Jan 2018 - 17
P&T
P&T
Migration 3: Isolate Original NCS6008 & Decom
Super
Core
Super
Core
P Router
NCS6k
PoP PE
CDN
Core
CDN
PoP PE
Exch
Exch
Exch
Exch
CPE
NCS5k
Sky Core Network , UKNOF, 17Jan 2018 - 18
P&T
P&T
Migration 5: ‘Old’ NCS6k Makes B2B NC6k on Other Ring
Super
Core
P Router
NCS6k
NCS6008 B2B
• 16 Tb with 1Tb
cards
• SR4 CPAK
• Accepts 2Tb cards
(-> 32 Tb)
• QSFP/CPAK
Core
CDN
PoP PE
Exch
Exch
Exch
Exch
CPE
NCS5k
Migration ‘Waterfall’
pr2 Lon1 N6k B2B
Leeds N6k B2B
Birmingham N6k B2B
Lon2 N6k B2B
pr3 Lon1 NCS5500
Leeds NCS5500
Birmingham NCS5500
Lon2 NCS5500
Jul 17 Oct 17 Jan 18 Apr 18 Jul 18
• Red triangles: Slot exhaust time, Green cells: Quarter for upgrade
• NCS6k replaced by new NCS5k
• Released NCS6k chassis incorporated into NCS6k B2B configuration
• Staged operation, One upgrade per quarter
Sky Core Network , UKNOF, 17Jan 2018 - 20
Capacity Issue Sorted
• New NCS5516 Ring sufficient for 5+ years
• Remaining NCS6008 B2B exhaust earlier, Replace with NCS5516 ?
• New Cost point with NCS5516 silicon, low-touch and SR4 QSFP
• Router port price trend close to Broadband price Trends
• Any problems ?
Sky Core Network , UKNOF, 17Jan 2018 - 21
Transponder Price Trend
• Transponder price (per Gb) falling at half rate of router port cost ( 10:1 in 12 years)
• 10 Gb Easy (in retrospect) => On-Off Modulation, FEC
• 100G Very Complex => QAM Modulation, Coherent Detection ( 2 Detectors I&Q )
Dual Polarisation ( 2 lasers, 2 x 2 Detectors: H&V x I&Q ),
Ultra High Speed A/D Converters
Dispersion Compensation DSP, Stronger FEC
• Distance is becoming (relatively) more expensive
• Localise Traffic Sources
Sky Core Network , UKNOF, 17Jan 2018 - 22
How to achieve further cost reductions ?
• Most traffic originates from CDN (On-Net or Off-Net)
1. CDN Private Peering ( v.s. Transit and Public Peering)
2. On-Net CDN - 65% of traffic sourced On-Net
3. Localise CDN Source
• How much can be Localised to minimise carriage across network ?
• Practicalities Matter – Diminishing Returns, Long Tails
Sky Core Network , UKNOF, 17Jan 2018 - 23
P&TP&T
CDN Deployment Options – Off-Net via P&T
Super
CoreSuper
Core
P Router
PoP PE
OffNet
CDN
CDN Cache
PoP PE
Exch
Exch
Exch
CPE
OffNet
CDN
• At least 4 transmission links
(Including Protection)
• Peering Interconnects
• Multiple Router Transitions
Sky Core Network , UKNOF, 17Jan 2018 - 24
P&T
CDN Deployment – On-Net, Remote CDN
Super
Core
Rem
CDN
PoP PE
Exch
Exch
Exch
CPE
• Multiple transmission links
• Fewer Router Transitions
Sky Core Network , UKNOF, 17Jan 2018 - 25
CDN Deployment – On-Net, Super Core
Super
CoreSuper
Core
Core
CDN
PoP PE
Exch
Exch
Exch
CPE
• Minimal Core Links
• Traffic still on PoP Links
• SuperCore Constraints
– Space Limited
– Expensive
– Power restrictions
Sky Core Network , UKNOF, 17Jan 2018 - 26
CDN Deployment Options – PoP CDN
Super
CoreSuper
Core
PoP PE
PoP
CDN
Exch
Exch
Exch
CPE
Core
CDN
• No Primary Traffic on Core
or PoP links
• If local CDN fails, sufficient
capacity on PoP links for
central fall-back
• Requires per PoP mapping
to localise traffic source
Sky Core Network , UKNOF, 17Jan 2018 - 27
CDN Deployment – PoP CDN feeding remote PoP
Super
CoreSuper
Core
PoP PE
PoP
CDN
Exch
Exch
Exch
CPE
• Local CDN serves other PoPs
• Upstream ‘for free’
Sky Core Network , UKNOF, 17Jan 2018 - 28
CDN Deployment – Exchange, Set Top Box
Super CoreSuper Core
PoP PE
PoP
CDN
Exch
Exch
Exch
CPE
Exchange CDN
• Problematic
• Insufficient traffic
• Power/Space constraint
• BNG architecture
STB
‘Home Pre-Caching’ ?
• Pre-load by satellite side
channel
• Limited application
- For very popular material
Sky Core Network , UKNOF, 17Jan 2018 - 29
PoP CDN Early Deployment – Low Traffic
• 5 larger providers
(+ many more off-net)
• Representative PoP sizing ( actually 60 PoPs)
PoPs, A few large + Long tail
• Need > 10Gb to warrant PoP CDN => Green Cells
• Early Deployment: 9 Locations, 31% CDN Localised
• Other Locations ( < 10G localised) serve Local+Other PoPs.
Provider Fraction 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Prov1 21% 26 17 13 11 9 8 7 7 6 6 5 5 5 4 4 4 4 4 4 3
Prov2 17% 21 13 10 9 7 7 6 5 5 5 4 4 4 4 3 3 3 3 3 3
Prov3 13% 15 10 8 6 6 5 4 4 4 3 3 3 3 3 3 2 2 2 2 2
Prov4 6% 8 5 4 3 3 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1
Prov5 5% 6 4 3 3 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1
Sky Core Network , UKNOF, 17Jan 2018 - 30
CDN PoP – Higher Traffic Levels
• Further Growth: 4 x Traffic => 63 Deployments, 88% Localised
• 130% More Deployments, 50% More traffic localised
• Diminishing Returns
Provider Fraction 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Prov1 21% 103 67 52 43 37 33 29 27 24 23 21 20 19 18 17 16 15 15 14 14
Prov2 17% 82 54 41 34 30 26 23 21 20 18 17 16 15 14 13 13 12 12 11 11
Prov3 13% 62 40 31 26 22 20 18 16 15 14 13 12 11 11 10 10 9 9 8 8
Prov4 6% 31 20 16 13 11 10 9 8 7 7 6 6 6 5 5 5 5 4 4 4
Prov5 5% 24 16 12 10 9 8 7 6 6 5 5 5 4 4 4 4 4 3 3 3
Provider Fraction 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Prov1 21% 51 34 26 21 18 16 15 13 12 11 11 10 9 9 8 8 8 7 7 7
Prov2 17% 41 27 21 17 15 13 12 11 10 9 8 8 8 7 7 6 6 6 6 5
Prov3 13% 31 20 16 13 11 10 9 8 7 7 6 6 6 5 5 5 5 4 4 4
Prov4 6% 15 10 8 6 6 5 4 4 4 3 3 3 3 3 3 2 2 2 2 2
Prov5 5% 12 8 6 5 4 4 3 3 3 3 2 2 2 2 2 2 2 2 2 2
• Double Traffic => 27 CDN deployments, 58% of CDN traffic
Sky Core Network , UKNOF, 17Jan 2018 - 31
CDN Localisation History
• Easy to see Benefits of PoP CDN
• Not so easy to realise them
• 5 x CDN providers
• Each with own approach
DNS, HTTP Redirect, + Secret Sauce
• All or nothing
– Specify Local and Central Mapping
– Allow headroom for overflow
• BBPE Software Upgrades
– Rolled out to 100 BBPEs
– Separate BBPE features for IPv6
Provider
Localisation
Sky Specifies
Localisation
BBPE & IPv6
Enhancements
Sky Core Network , UKNOF, 17Jan 2018 - 32
IPv6 Deployment
• Dual Stack alongside IPv4 (eg. Not DS-Lite)
• Both LLU (PPP) and “Fibre” (IPoE) services eligible
• > 5 Million customers eligible and enabled (>95%)
• Support all Sky Hub models (1,2,3) and legacy Sky-supplied Sagem routers.
• > 90% uptake. (Remaining assumed to be user-disabled and/or 3rd party an or very
old routers without compatible IPv6 support)
• New customer default since April 2016
• /56 Prefix Delegation
• 2 year exercise 20% -> 90%
Sky Core Network , UKNOF, 17Jan 2018 - 33
Acknowlegements…
Rapid, well executed program through 2017
Made Possible by Great Team within Sky Networks
• Network Architecture & Planning - Forecast, Modelling, Migration Planning
• Network Development – Testing, Verification, Product Advice
• Network Implementation - Configuration, Installation
• Network Operations - Many overnight migrations
• Project Mgt - Complex Project, Juggling Resources
• Cisco - Early testing support, Expedited delivery
Storm Surfers
Sky Core Network
Positionined for Massive Video Demand
Thank you!
Top Related