Scaling NFV - OSS Boston · • Development environments CI/CD • Introduc7on of new func7onality...
35
Scaling NFV - Are containers the answer? Azhar Sayeed - [email protected] Doug Smith - [email protected]
Transcript of Scaling NFV - OSS Boston · • Development environments CI/CD • Introduc7on of new func7onality...
Scaling NFV - Are containers the answer?
Azhar Sayeed - [email protected] Doug Smith - [email protected]
2
Acknowledgements
Thisisaresultofmul7pleeffortsinRedHatonContainersandContainerNetworking.WewouldlikethankeveryonewhohelpedusputthisPOC,demoandpresenta7ontogether.
Abigthankyouto
• DanWilliams([email protected])andRashidKhan([email protected])forlistening,beingpa7entwithusandforbuildingaprototypethatisreallypowerful
• DanforwhippingupslidesandcodeinamaKerof8weeks.
• AjaySimha([email protected])forhisreviewandcontribu7onstothepresenta7onandtheworkhewasdoingwithDougSmithtobuildaPOC
• TomofumiHayashiforhisworkonkoko(ContainerConnector)basisforthedemo
3
Agenda
• Introduc7on• TelcoRequirementsforNFVscale• Containers-howcantheyhelp?• Scaleques7ons• Dotheysolvetheproblem?
• IssuesandChallenges• Demo• Summary
Virtualiza7onProgression
Baremetal
VirtualizedAppsin-VMs
VirtualizedAppsinContainers
Applica7onsandNetworkFunc7ons
ContainersinVMsandVMsinContainers?
NFV-Usecasesandscale
5
vCPE-Residen7al vCPE/SDWAN
vEPC/vIMS/VoLTE vGiLAN
Mobile
Wireline
Consumer Business
6
NFVUsecase-vCPE
EnterprisevCPE
VirtualizedCentralOfficeORDataCenter
Internet
Residen7alvCPEEnterprisevCPE
NFVOSDNControllerVNFM(s)VIM
Residen7alNID
Security&FirewallQualityofService(QoS)TrafficShapingDeviceManagement
Security&FirewallParentalControlQuotaManagementHomeAutoma7on
CPEvirtualiza-onisnotjustaboutcostreduc-onbutprovidingnewservicestocustomersatthepaceofinnova-onandScale
● FlexibilityofIPaddressassignment-PublicIP,PrivateIP,IPv4andIPv6etc-manyVNFsrequirenoNAT○ DHCPbasedaddressassignment
● Mul7pleInterfaceassignment-Rou7ng,Meteringetc● Mul7-TenancyandManagementofoverlays● PacketForwardingPerformancerequirements-Allworkloadsarenotequal
○ NICbonding○ NUMAaffinity-containerscheduling○ HugePageSupport○ CPUpinningorpar77oning○ Jumboframessupport
● HybridVNFs(containerandVMs)● MixedtopologiescontainersandVMs● Loadsharing● Elas7city-Orchestra7on
NFVrequirementsGenericNFVWorkloadRequirements
AMul7-dimensionalproblemforTelcos
8
Scalemetricsandfactors
• TotalnumberofSessions,subscribersscale• ServiceDensity-VMs,Apps• Throughputscale• Orchestra7onscale
• Numberofcomple7ons(Addsmovesanddeletes)• Managementandtroubleshoo7ngscale
• VisibilityandTraceabilityatscale• AuditTrailofTransac7ons
• DevelopmentenvironmentsCI/CD• Introduc7onofnewfunc7onality
Itisnotjustaboutscalebutalsothespeedofscale
Example:vCPEForResiden7alServices
9
Scalemetricsandfactors
• Footprint-Subscriberdensity• TypicalBNGRouterserves300KIPSessions-Halfrackdedicatedhardware• AddingQoSandotherbellsandwhistles=>150-200KIPSessions• Throughputpersubscriber
• 10Gbpsconnec7onscommon-butsimultaneoususersandsubscribersaverageto<0.5Gbpspersubscriber
• 50Kac7vesubscribers=>25x100Gbpssustainedthroughput• NumberofVMsperserver-VNFRequirementsonCPU,MemoryandIO• NumberofSubsperVMs• NumberofServers
• NumberofcoresneededtoservethatthroughputusingOVS+Accelera7onorVPPetc
HOWCANWESCALETHISTOEVENHIGHERDESNITIES
● Lowvirtualiza7onoverheadperVNF(applica7on)● Lowmemoryfootprint● Instantrestart7me● LowLatency-duetoasharedmemorymodel● Higherdensityperserver/socketthanVMs● Encapsula7onofmicroservices● Portability● Determinis7cpackaging● ReasonableIsola7oncanbeaccomplishedeasily
WhyContainers?Containers:Sonwarepackagingconceptthatincludeanapplica7onandallitsrun7medependencies
ComparingVMsandContainers
Hardware Hardware
HostOS HostOS
Hypervisor DockerEngine
OS OS
App1 App2
App2App1
VMStack ContainerStack
VirtualMachine
VMs● GuestOSisneededperVM● EachVirtualMachineisisolatedbythehypervisor● InterfaceandhardwareemulatedbytheHypervisor● Distribu7onofapp7edtoOS● OScommonlytunedtodeliverappperformance
Containers● Thereisnohypervisorinthecontainerstack● DockerEngineactsasthe“hypervisor”● Eachapplica7onrunsasaprocessinuserspace● Isola7onthroughcnames● Considered“lightweight”comparedtoVMs● Packetforwardingperformancedependenton
kernelstack● Orchestra7onviaKubernetes● Scale->10x
Libs Libs
VMsandContainers–TelcoEvolu7onviewpoint
12
HW
HOSTOS
HYPERVISOR
GuestOS GuestOS
Libs&Run7me Libs&Run7me
App App
HW
HOSTOS
Libs&Run7me
App App
HW
HOSTOS
HYPERVISOR
GuestOS GuestOS
Libs&Run7me Libs&Run7me
App AppAppApp
VM
Containers
ContainersinVM(TenantIsola7on)
VMs
HW
HOSTOS
Hypervisor
GuestOS
Libs&Run7me
App
AppApp
App
Containers&VMs
Libs&Run7me
Containers&NFV
13
● Usecontainersala“VM”● Leveragedockeriza7onofsomefunc7ons-suchasDHCP,IPAM,NAT,FWetc● Notreallysepara7ngcomponentswithinNetworkFunc7ons(alaMicroservices)as
thenetworkfunc7onsthemselvesarevirtualized● Intui7vetoapplyandhenceassumedeasytomakeithappen
Containers&NFVApplicability
● FlexibilityofIPaddressassignmenttocontainers-PublicIP,PrivateIP,IPv4andIPv6etc-manyVNFsrequirenoNAT
● Mul7pleInterfaceassignmenttoacontainer● Mul7-TenancyandManagementofoverlays● Performancerequirements-Allworkloadsarenotequal
○ NICbonding○ NUMAaffinity-scheduling○ HugePageSupport○ CPUpinningorpar77oning
● HybridVNFs(containerandVMs)● MixedtopologiescontainersandVMs● Loadsharingandscale
Revisi7ngtheNFVrequirementsWithContainers-Howdotheyfare?
???
?
● ControlplaneheavyVNFs○ Highsessioncountorcontroltraffic○ Lowdataforwarding○ Latencyandavailabilitysensi7vefornetworkconvergence○ Examples-signaling,subscriberpolicy,controlprotocols
● DataplaneheavyVNFs○ Requirelargememoryalloca7on○ Largefootprintapplica7ons(CPU,memory,I/O)○ Highforwardingraterequirements○ Highvolumeoftraffic○ Examples-PGW,ePDG,DPIetc
ContainersandNFV
Telcoprovideddefini7on
CandidatesforContaineriza7on
● SimpleVNF-vRouterwith2interfaces○ SimpleIGPandBGPConfigura7on○ Stockimages-Vyosdistribu7on○ MemoryneededtoruntheVMwithbasicalloca7ons-387MB
● Containers○ Sameconfigura7on○ StockContainerimage○ RunusingDocker○ Percontainer-34MB
● vCPUalloca7onsperVM○ CoreprocessingforDPDK○ 6-12coresforVNFslikevEPC,BNG○ 16-32GBofmemory
SizingNFVExample
Notes:Smallerconfigsresultinsmallercontainers-Only1BGPsessionandanIGPresultsin28MBpercontainer
6-10Xdensity
● Usenamespacestoisolatenetworkfunc7ons● Networknamespacesforcontainerstoseetheirresources● Kernelperformancebecomesimportant● Sonwareswitch-likemacvlan● AssignSR-IOVtonetworknamespace● UsingDPDKaccelera7on?
ForwardingperformancewithcontainersNFV=LineRatePerformanceRequirement
● Kubernetes-ScaleisProven-Openshin○ TodayoperateslargestofDCswithmillionsofcontainers○ EnterpriseITandOTT
● Scalingnumberofpodsandnodes○ Commontofind100Nodesand3000PodsforVNFdeployments
● Kolla-AnsibleplaybookswithDockerContainerstoprovideproduc7onreadycontainersforopenstackclouds
● NFVspecialrequirements○ ConstraintsonKubernetes/Openshin○ WhataboutOAMmanagement,Traceability,Performance,conformance,audittrail
ContainerOrchestra7onScaleofOrchestra7on
OPENSHIFT–PLATFORMFORCONTAINERS
KubernetesbasedOrchestraDon DockerContainerFormat
AtomicHost Networking Telemetry
Security Automa7on ClusteringStorage
● RunOpenshin/KubernetesonOpenstack● Kuryr● Magnum
● RunOpenstackservicesincontainers○ Kolla
OpenstackandContainersManagingcontainersinOpenstackEnvironments
22
• Containersattheremotesiteorcentraldatacenter• S7tchedtogetherforservicechaining–
• samehost–IPC• differenthosts-VLAN/VXLAN
• Portmappingarchitecturecanbemadetoworkhere• WillthisimpactNSHordynamicSFC?
SubscriberServiceChaining–withContainers
HFC/GPON
ServiceFunc7onChain
Demarca7onpoint
CloudBoundary
VxLAN
L2NID
AAAConfigura7onPolicy
Applica7onorContentCache
ParentalControl
QuotaManagement
Firewall&NATing
CPE
ⲙVNF
VxLANorIPSec
CentralOfficeorDC
HowdoesSFCworkwithcontainers?
vOLT
ProofofConcept
23
ContainersNFV(Needs/Requests)
● Mul7plenetworks/interfacesinacontainer○ Op7onalSDNformanagement○ PhysicalNICsandSR-IOVinterfaces○ Storagenetworks,legacyservers
● DPDK-enabledapplica7ons● FlexibleIPaddressing
○ Public/administrator-definedcontainerIPaddresses○ OverlappingIPnetworks&mul7-tenancy
● Flatarchitectureforline-rateprocessingandlowlatency○ Reducednumberofinterfacesbetweenwireandapplica7on
● Determinis7cCPUandmemoryresources○ Pincontainerizedfunc7onstospecificCPUsandNUMAnodes
● Coordina7ngVFsinwidelyseparatedpremises● IPv6support,especiallyinpubliccloud● Provideexis7ngorchestra7onandop7onalmicro-servicefeatures
○ Enablenewdistributedapplica7onarchitectures
vCPEServer
SFC
NIC NIC NIC
Customer Metro/GPON
NIDDemarcPoint
GPON
OperatorDatacenter
VFContainer
VFContainer
Management
ContainerNFVProof-of-ConceptUsingOpenShin
VFContainer
PoCdemonstratesacontainerizedmul7-VFvCPEonacustomerpremises,usingavRouterandvFirewallconnectedviasimpleServiceFunc7onChain,directlyconnectedtoboththecustomernetworkandtheprovidernetwork.
Addi7onalNICAddi7onalNIC
TonexthopinSFConthesamenode
ManagementSDN
NICorSR-IOV
ContainerNFVPoC:InsidetheContainer
vethSFCendpoint
SDNendpoint
AdministratordefinedIPaddress
AdministratordefinedIPaddress
OpDonalSDNprovidedIPaddressandmicro-services
VirtualFuncDon(vRouter,vFirewall,etc)
Addi7onalNIC
DPDK(op7onal) KernelNetworking
● Sta7cIPv6support● SimpleServiceFunc7onChains● Flexible,administratordefinedIPaddressing● NobridgingorSDNinpacketfast-paths● Mul7pleinterfacespercontainer(NIC,VLAN,SDN)● ContainerVFCPUaffinity
Features:
ContainerNFVPoC:NextSteps● DeterminefeasibilityofoverlappingIPnetworks● Inves7gateaddi7onalfeaturesandrequirements:
○ NUMAaffinity○ IPv6SLAACaddressing○ DynamicServiceFunc7onChains○ NetworkServiceHeaders○ Servicesupportforaddi7onalinterfaces
● ImplementsupportforSR-IOVinterfaces● Gathermorerequirementsanduse-cases● WorkwithupstreamKubernetescommunitytostandardizethesefeatures
DemoDougSmith&AjaySimha
28
HostA-OpenStack
192.168.2.100
quagga_aID:2.2.2.2
192.168.2.101
192.168.3.100
192.168.4.100
192.168.3.101 192.168.4.101
centos_aID:1.1.1.1
centos_bID:4.4.4.4
quagga_bID:3.3.3.3
in1
in2
mid1
out1
mid2 out2
HostB-OpenShinonAWS
vxlan vxlan
Legend
Host/Guest DockerContainer
Networkpath NetworkInterface
WAN
ContainerizedRouterDemo-overHybridCloud
OSPF
StaDc StaDc
30
Suppor7ngTechnologies
KoKo-Networknamespaceu7lityAvailableonGitHub
Ratchet-CNIPluginAvailableonGitHub
31
KoKofunc7onality
veth/vxlan veth/vxlan
KoKo
● ExposeSystemresourcestocontainers-suchasPCIdevices,NUMANodes,KernelModules,○ Highersecurityrisk-poten7allylargeraKacksurfacewhencomparedtoVMs○ Kernelop7miza7onsbecomecri7calforforwardingperformance-nooffload
techniques○ Inter-containercommunica7onusesIPCinsteadofEthernet/IP=>Advantageanda
disadvantage● Achievingmul7-tenancyisconsideredharderwrtcontainersthanVMs
○ Namespacesandcgroups● AddingOAMcapabili7esfornetworkfunc7onsincontainersincreasesthesizeofthemicro-
service-NewerOAMarchitecturesneededtodefineservicesformul7plecontainersversusreplica7ngforeachcontainers
● MayRequiresomeservicere-architec7ng● Interfacelimita7ons(OSissues)
ChallengeswithusingContainersforVNFs
● Highscaleneededfordifferentusecases○ IoT,vCPEResiden7aletc○ Millionsofsessionsmaptothousandsofservers
● ContainerscanprovidemuchhigherscalethanVMs(>10x)○ Smallerfootprint-whencomparedtoVMonOSP
● VNFsmayneedtobere-wriKentotakeadvantageofContainer/Micro-servicesmodels● Kubernetesscaleswellformassiveapplica7onfarms=>adopttoNFV● ContainernetworkingasubjectofmoreworkandinterestintheindustryforNFV
○ KuryrprovidesanetworkingmodelwithOpenstack● Na7venetworkingforcontainersusingKubernetes/Openshin
○ POCandCodeavailablefortes7ng-GoaltocommitupstreamaspartofKubernetesSDNenhancements
● ForNFVwithContainers-s7llmoreworkneeded○ DynamicServiceChains,NUMAAffinityetc
Summary
THANK YOU
INSERTDESIGNATOR,IFNEEDED35
Abstract
Containersarethebiggesthypetoday-InthelatestHeavyReadingSurvey(October2016),68.4%oftheresponderssaidtheywillusecontainersforNFV.Whilecontainersprovidehighscale,lowlatencyandalowstartup7me,however,noonereallyunderstandsthecompleteimpactofcontainersonhowitchangesthevirtualiza7onmodelforNFVandwhatimpactithasonthenetworkingandorchestra7onmodelforNFV.
Containersarewelldesignedforscaleoutapplica7onsbutforcontainerstoworkwithNFVweneedtheabilitytoassignpublicIPaddressestocontainers.Thatisnotsoeasyasitsounds.
Inthispresenta7on,wewilldiscusstheNFVarchitecturewithcontainersindetail.Inpar7cularwewilldiscusstopicslikeKuryr(ContainersandOpenstack),Containernetworking,Containerinstan7a7onwithOpenstack,Scale,Performance(LatencyandThroughput)anddatapathaccelera7onforcontainers.
WhatcanIexpecttolearn?
• ContainerandhowtheycanbeusedforNFV
• Limita7onsofcontainers
• Orchestra7onNFVwithVMsandContainersusingOpenstack
