Applying Hyper-scale Design Patterns to Routing

Applying Hyper-scaleDesign Patterns to Routing

Hannes Gredler, CTO RtBrick Inc.DEVNET-2064

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Who am I ?• CTO at RtBrick, Inc.• Past stint: Distinguished Engineer

with the “other router-vendor”• 18 Years working experience,

developing, deploying and supporting Routing Software

• Expertise• BGP, IS-IS, MPLS• 20+ Patents• 20+ Proposed Standards

http://www.arkko.com/tools/allstats/hannesgredler.html

• IETF WG co-chair (IS-IS)

DEVNET-2064 2




>2013 exposure to Data Center Networks & SRNew large-scale data-center network model emerging [draft-ietf-rtgwg-bgp-routing-large-dc] End-to-End Layer-3 routing

Fixes issues with L2 switching data plane.

Hierarchical Topology CLOS-based

Max 5-stages Use of aggregation at TORs

DEVNET-2064 3

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 4

Got a couple of inconvenient insights …

• Networks have become Anti-Moore• direct sourcing from OEM manufacturers in Taiwan

• Hardware is a Commodity• Cost per Bit dropping sharply (USD 400 / 100GBE port)• Boutique ASICs viable in 5 years from now ?

• Curated Software Release models approaching EOL• Modularization or Custom package selection desired (no-PIM, no RSVP, etc.)• Pay per-use

• Different model (node vs. system) for Resiliency• Open sourcing of components the new normal

• Integration of components becomes core competency

DEVNET-2064

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5DEVNET-2064

John GageSun Microsystems

1) “The network is the computer”


Hannes Gredler, 2015

2) “Is it possible to construct a router based on the web 2.0 mindset ?”


• Introduction• Multi-Level Architecture• Micro-services & APIs• Commoditization & Unit Economics• Resiliency, system coupling and state recovery• Open Source Development & Test• Conclusion

Agenda

DEVNET-2064


Multi-Level Architecture

8DEVNET-2064


Hyper-scale Multi-level Architecture


Forwarding node• Translates RIB Objects to local OS

representation• Tables• Routes• Nexthops

• Hardware Prefix Caching

• Aggregate FIB table• (filter specifics)

• Localize fwd table• VPNs

DEVNET-2064


Protocol I/O node• Schema driven protocol

serializer /de-serializer• Keep alive delegation /absorption• Terminal Communication point for

Sockets, stdio & file I/O• Pre-processing protocol stream

(filter BGP PA128)• Queuing machinery & Routing

Protocol update generation• Interface state handling

DEVNET-2064


Application (route computation) node• Schema driven Data Structure

Server

• Stores Applications Objects• Routes, Nexthops, Tables

• Triggered execution (Add, Chg, Del) of internal/external Application code• Python functions• C/C++ library calls• Executables vfork()

DEVNET-2064


Putting it all together


Micro-Services & APIs

14DEVNET-2064


Build a system of little components

• Micro-service architecture is like a UNIX pipeline model

• Small pieces of software, serving a unique purpose

• Easy transfer of state from one brick to next

SortFilterSource Filter Sink

curl http://192.168.1.1/bds/object | grep “Received-From:” | sort | grep “foo” > out

DEVNET-2064


REST/JSON based APIs

DEVNET-2064


bds://local/bgp.neighbor

bds://local/isis.adj

bds://local/isis.lsdb.l2

bds://217.160.181.216/bgp.rib-in

PUBSUB

Database centric / Distributed Data Store

DEVNET-2064


Open IPC format = BSON/JSON• Binary JSON for memory and I/O

efficiency

• JSON conversion on the fly possible

DEVNET-2064


Table replication & state flow within a system

DEVNET-2064


Commoditization & Unit Economics

20DEVNET-2064


Compute Strategy: Yahoo vs. Google

DEVNET-2064

Few Big vs. Many Small


• Economy of scale will ultimately render custom-ASICs obsolete• FY2016 systems shipping: 100GB, > 128K FIB entries• Disintegration happening

• soon to enter the Edge Router Business …

• For ease of integration makes no Hardware, no locality, no OS assumptions• Unbounded Configuration Possibilities:• Single Switch, Cluster of Switches, Co-located x86 Rack Servers ….• Large FIBs, Small FIBs, SW-based forwarders & Combos thereof

Commodity data plane = White-boxes

DEVNET-2064


• 16-32 CPU Cores, 64 GB RAM, Solid State Disks• app USD 3000

• Runs stock Ubuntu / Centos

• Linux Containers (LXC)• dependency management• Para-Virtualization

Commodity control plane = 1RU Rack Servers

DEVNET-2064


Resiliency, system coupling and state recovery

24DEVNET-2064


Hyper-scale Multi-level Architecture


Resiliency


Resiliency – snapshot DB to disk


Resiliency (2) – restart based on disk snapshot

DEVNET-2064


Open SourceDevelopment & Test

29DEVNET-2064


Open Development• Open Source

• 100 eyes better than 4 eyes, Network effects• Long term Maintenance

• Open Source means sharing of not just Code:• Code• Test• Build• Documentation


Open development (1)• Use what is usable

• No needs to re-invent Linux, event-loops, memory managers

• Kernel based networking stacks are not usable for a router• Debugging hard (GDB live attachment)• Experimental forwarding code with no fault-domains in your kernel, really ?• TCP snapshots / restart.

• In 2016Q1 we did not have a packet forwarding core• Cisco did release fd.io / VPP• User space DPDK design aligned with our (religious) believes• Most feature complete open-source L3 forwarder• Engineered for performance and maintainability

DEVNET-2064


Open development (2)• Kick-ass VPP crew

• Helped us to implement necessary core-features (indirect next-hop) within two weeks.• Good balance between Stability and feature velocity• Excellent Continuous Integration & Test Automation (untypical for FLOSS projects)


Open Development (3)VPP Internet stream generator

DEVNET-2064


Conclusion

34DEVNET-2064


In Conclusion• Network Equipment design got to be

• Distributed, Multi-level Architecture• Micro-service based• Running on Commodity Hardware• “System” Resilient• Open Development / Open Test

• Cisco Vector Packet Processing (VPP)• Best code in the industry (why is this free ?)• Good Code Governance• Establishment of an innovative ecosystem around VPP underway

DEVNET-2064


rtbrick demo at fd.io booth

36DEVNET-2064


Demo hosted at EC2 instance


• 39Million IPv4 / 3M IPv6 route entries

• BGP table snapshots from RIPE RIS server

• route-processing / update / restart performance• 20x compared to JUNOS | IOS-XR• Full-bringup time 180s• Resync time 26s

• Full fault domain isolation• Blast radius within a protocol-process of an address family

• Process restart• Preservation of TCP session• Fast, robust Re-sync of state• Everything versioned

Demo SCALE


Process restart using snapshots (3)

Normal appd restart resync time for 2.5M RIB entries / 28s

snapshot appd restart resync time for 2.5M RIB entries / 6s

Thank you


Applying Hyper-scale Design Patterns to Routing

Internet

Transcript of Applying Hyper-scale Design Patterns to Routing