Leveraging Containers and OpenStack
12
Leveraging Containers and OpenStack A Comprehensive Review Introduction Imagine that you are tasked to build an entire private cloud infrastructure from the ground up. You have a limited budget, a small but dedicated team, and are asked to pull off a miracle. A few years ago, you’d build an infrastructure with applications running in virtual machines, with some bare-metal machines for legacy applications. As infrastructure has evolved, virtual machines (VMs) enabled greater levels of efficiency and agility, but VMs alone don’t completely meet the needs of an agile approach to application deployment. They continue to serve as a foundation for running many applications, but increasingly, developers are looking toward the emerging trend of containers for leading-edge application development and deployment because containers offer increased levels of agility and efficiency. Container technologies like Docker and Kubernetes are becoming the leading standards for building containerized applications. They help free organizations from complexity that limits development agility. Containers, container infrastructure, and container deployment technologies have proven themselves to be very powerful abstractions that can be applied to a number of different use cases. Using something like Kubernetes, an organization can deliver a cloud that solely uses containers for application delivery. But a leading-edge private cloud isn’t just about containers, and containers aren’t appropriate for all workloads and use cases. Today, most private cloud infrastructures need to encompass bare-metal machines for managing infrastructure, virtual machines for legacy applications, and containers for newer applications. The ability to support, manage and orchestrate all three approaches is the key to operational efficiency. OpenStack is currently the best available option for building private clouds, with the ability to manage networking, storage and compute infrastructure, with support for virtual machines, bare-metal, and containers from one control plane. While Kubernetes is arguably the most popular container orchestrator and has changed application delivery, it depends on the availability of a solid cloud infrastructure, and OpenStack offers the most comprehensive open source infrastructure for hosting applications. OpenStack’s multi-tenant cloud infrastructure is a natural fit for Kubernetes, with several integration points, deployment solutions, and ability to federate across multiple clouds. In this paper, we’re going to explore how containers work within OpenStack, examine various use cases, and provide an overview of open source projects, from OpenStack and elsewhere, that help make containers a technology that’s easily adopted and utilized. I. A High Level View of Containers in OpenStack There are three primary scenarios where containers and OpenStack intersect. The first scenario, called infrastructure containers, allows operators to leverage containers in a way that improves cloud
Transcript of Leveraging Containers and OpenStack
LeveragingContainersandOpenStackAComprehensiveReviewIntroduction
Imaginethatyouaretaskedtobuildanentireprivatecloudinfrastructurefromthegroundup.Youhavealimitedbudget,asmallbutdedicatedteam,andareaskedtopulloffamiracle.
Afewyearsago,you’dbuildaninfrastructurewithapplicationsrunninginvirtualmachines,withsomebare-metalmachinesforlegacyapplications.Asinfrastructurehasevolved,virtualmachines(VMs)enabledgreaterlevelsofefficiencyandagility,butVMsalonedon’tcompletelymeettheneedsofanagileapproachtoapplicationdeployment.Theycontinuetoserveasafoundationforrunningmanyapplications,butincreasingly,developersarelookingtowardtheemergingtrendofcontainersforleading-edgeapplicationdevelopmentanddeploymentbecausecontainersofferincreasedlevelsofagilityandefficiency.
ContainertechnologieslikeDockerandKubernetesarebecomingtheleadingstandardsforbuildingcontainerizedapplications.Theyhelpfreeorganizationsfromcomplexitythatlimitsdevelopmentagility.Containers,containerinfrastructure,andcontainerdeploymenttechnologieshaveproventhemselvestobeverypowerfulabstractionsthatcanbeappliedtoanumberofdifferentusecases.UsingsomethinglikeKubernetes,anorganizationcandeliveracloudthatsolelyusescontainersforapplicationdelivery.
Butaleading-edgeprivatecloudisn’tjustaboutcontainers,andcontainersaren’tappropriateforallworkloadsandusecases.Today,mostprivatecloudinfrastructuresneedtoencompassbare-metalmachinesformanaginginfrastructure,virtualmachinesforlegacyapplications,andcontainersfornewerapplications.Theabilitytosupport,manageandorchestrateallthreeapproachesisthekeytooperationalefficiency.
OpenStackiscurrentlythebestavailableoptionforbuildingprivateclouds,withtheabilitytomanagenetworking,storageandcomputeinfrastructure,withsupportforvirtualmachines,bare-metal,andcontainersfromonecontrolplane.WhileKubernetesisarguablythemostpopularcontainerorchestratorandhaschangedapplicationdelivery,itdependsontheavailabilityofasolidcloudinfrastructure,andOpenStackoffersthemostcomprehensiveopensourceinfrastructureforhostingapplications.OpenStack’smulti-tenantcloudinfrastructureisanaturalfitforKubernetes,withseveralintegrationpoints,deploymentsolutions,andabilitytofederateacrossmultipleclouds.
Inthispaper,we’regoingtoexplorehowcontainersworkwithinOpenStack,examinevarioususecases,andprovideanoverviewofopensourceprojects,fromOpenStackandelsewhere,thathelpmakecontainersatechnologythat’seasilyadoptedandutilized.
I.AHighLevelViewofContainersinOpenStack
TherearethreeprimaryscenarioswherecontainersandOpenStackintersect.
Thefirstscenario,calledinfrastructurecontainers,allowsoperatorstoleveragecontainersinawaythatimprovescloud
infrastructuredeployment,management,andoperation.Inthisscenario,containersaresetuponabare-metalinfrastructure,andareallowedprivilegedaccesstohostresources.Thisaccessallowsthemtotakedirectadvantageofcompute,networking,andstorageresourcesthatcontainerruntimesaretypicallytryingtohidefromusers.Thecontainersisolatetheoftencomplexsetofdependenciesthateachapplicationdependson,whilestillallowingtheinfrastructureapplicationstodirectlymanageandmanipulatetheunderlyingsystemresources.Whenthetimecomestoupgradeanservice,theupgradecanbehandledwithoutchangesindependenciesdisruptingco-locatedservices.
ModernversionsofOpenStackhaveembracedthisinfrastructurecontainermodel,andit’snownormaltomanageanentirelifecycleofanOpenStackdeploymentwithacombinationoforchestrationtoolingandcontainerizedservices.Infrastructurecontainersenableoperatorstousecontainerorchestrationtechnologiestosolvemanyissues,particularlyaroundrapidlyiterating/upgradingexistingsoftwareincludingOpenStack.RunningOpenStackwithincontainershelpsoperatorstosolveDay2challenges,includingaddingnewcomponentsforservices,upgradingversionsofsoftwarequickly,andrapidlyrollingupdatesacrossmachinesanddatacenters.ThisapproachbringstheagilityofcontainerstotheproblemofOpenStackdeploymentandupgrades.
Thesecondscenarioisconcernedwithhostingcontainerizedapplicationframeworksoncloudinfrastructure.ThesecanincludeContainerOrchestrationEngines(COEs)likeDockerSwarmandKubernetes,orlighter-weightcontainer-focusedservicesandserverlessapplicationprogramminginterfaces(APIs).Whetheronbare-metalorVMs,theOpenStackcommunityhasworkedtoensurethatit’spossibletodelivercontainerizedapplicationsonasecure,tenant-isolatedcloudhost.ThisscenarioisfacilitatedbydriversthatallowprojectslikeKubernetestodirectlytakeadvantageofOpenStackAPIsforstorage,load-balancing,andidentity.ItalsoincludesAPIsforprovisioningmanagedKubernetesclustersandapplicationcontainersondemand.Withthesecapabilities,developmentteamscanwritenewcontainerizedapplicationsandquicklyprovisionKubernetesclustersonOpenStackclouds.It’sacompleteapplicationlifecyclesolutionthatgivesthemtheresourcesneededtodevelop,test,anddebugtheircode,withrobustautomationtodeploytheirapplicationsintoproduction.
Inthefinalscenario,weconsidertheinteractionsbetweenindependentOpenStackandCOEdeployments,andinthispaperparticularlyKubernetesclusters.ConsistencyandinteroperabilityofAPIsacrossbothOpenStackandKubernetesclustersistheprimarysourceofsuccessforthisscenario.Forexample,it’spossibleforKubernetestodirectlyattachtoOpenStackCinderhostedvolumes,useOpenStackKeystoneasanauthorizationandauthenticationbackend,orconnecttoOpenStackNeutronasanetworkoverlaywithOpenStackKuryr.Conversely,it’spossibleforanOpenStackcloudtosharethesamenetworkoverlayasaKubernetesclusterwithNeutrondriversforprojectslikeCalico.Thethirdscenarioislessfocusedonhowacloudserviceishosted(beitKubernetesorOpenStack),andmoreonhowindependentservicesinteract.
II.OpenStackContainerIntegrationPoints
DeployingOpenStackInfrastructureonContainers
Asnotedintheintroduction,thedeploymentandmanagementofOpenStackhaschangedsignificantlywiththeriseofcontainers,becausecontainersunlocknewapproachestomanaginginfrastructurecode.Previousmanagementstrategiesrequiredeitherthecreationandmaintenanceofheavyweightgoldenmachineimages,orusingbrittlestate-maintainingconfiguration-managementsystems.Eachapproachcomeswithcomplexitiesandrestrictions.Addingtothedegreeofdifficultyisthemanagementofacollectionofservicesthatallrequiretheirowndependenciesthatchangefromrelease-to-release.Withoutsomeformofapplicationisolation,solvingforthedependenciesbecomesdifficultifnotimpossible.
InfrastructurecontainersenablenewOpenStackdeploymentprojectstostrikeabalancebetweenthetwowhileelegantlysolvingthedependencyproblem.Usinglightweight,independent,self-contained,andtypicallystatelessapplicationcontainers,acloudoperatorgainstremendousflexibilitywhendeployingacomplexcontrolplane.Combinedwithacontainerruntimeandanorchestrationengine,infrastructurecontainersmakeitpossibletoquicklydeploy,maintain,andupgradecomplexandhighlyavailableinfrastructure.
InbuildinganOpenStackcluster,thereareseveraldimensionsforchoosingdeploymenttechnologies.AnoperatorcouldchooseLinuxContainers(LXC)orDockerfortheirbasecontainers,usepre-builtorcustom-builtapplicationcontainers,andselecteithertraditionalconfiguration-managementsystemsfororchestrationoramoremodernapproachlikeKubernetes.Table1summarizestheexistingOpenStackdeploymentprojectsandtheirunderlyingtechnologies.
Table1
Project
OpenStack-Ansible
Kolla-Ansible
Triple-O
OpenStack-Helm
ContainerType
LXC
Docker
Docker
Docker
SupportedContainers
OSA LXC Containers
Kolla Containers
Kolla Containers
Kolla Containers
Loci Containers
Project
Ansible
Ansible
Ansible
Kubernetes and Helm
UnderlyingeachofthesedeploymentsystemsaredifferentapproachestobuildingasetofcontainersfortheOpenStackcodeandsupportingservices.TheOpenStackAnsible(OSA)andKollaprojectsprovidetheirownproject-hostedbuildsystems,whileLOCIfocusesonbuildingprojectapplicationcontainers,withoutaspecificorchestrationsysteminmind.Atahighlevel,thedifferencesare:
1. OSAisuniqueinthatitreliesonlower-levelLXCcontainers,andhasacustombuildsystemforcreatingLXCapplicationcontainers.
2. TheKollabuildsystemproducesDockercontainers,oneforeachservice,alongwithsupportingcontainersforinitializingandmanaginganOpenStackdeployment.Kollacontainersarehighlyconfigurable,withachoiceofbaseoperatingsystem,sourceorpackageinstallations,andatemplateengineforevenfurthercustomization.
3. ThefinaloptionforbuildingOpenStackapplicationcontainersisLOCI.LOCIalsobuildsDockercontainers,anddeliversonecontainerforeachproject.LOCIisfocusedonproducingcompactandsecurecontainersquickly,forallcommondistributions,withtheexpectationthattheywillbeusedasafoundationtobuilduponbythedeploymentsystem.
Bare-MetalInfrastructure-OpenStackandSolvingtheBootstrapProblem
Atthefoundationofeverycloud,thereexistsadatacenterofbare-metalserversthathosttheinfrastructureservices.Even“serverlesscomputing”isrunningsoftwareonacloudonhardwareinadatacenter.TheproblemofhowtobootstraphardwareinfrastructureisacriticalproblemthatOpenStacksoftwareisuniquelyqualifiedtoaddressinawaythatgivescloud-likequalitiestobare-metalmanagement.
OpenStackIronicprovidesbare-metalasaservice.Asastandaloneserviceitcandiscoverbare-metalnodes,catalogtheminamanagementdatabase,andmanagetheentireserverlifecycleincludingenrolling,provisioning,maintenance,anddecommissioning.WhenusedasadrivertoOpenStackNovaandcombinedwiththefullsuiteofOpenStackservices,itdeliversapowerful,cloud-likeserviceformanagingyourentirebare-metalinfrastructure.
Thisraisesthequestion:HowdoesonebootstrapOpenStackservicestomanagebare-metalinfrastructure?Onetypicalsolutionistousethesamecontainer-basedinstallationtoolsasdescribedintheprevioussectionstocreateaseedinstallation.Thisseed,oftencalledan‘undercloud’,canbeusedtoentirelyautomatethemanagementofabare-metalclusterasifitwereavirtualizedcloud.
ThisopensupanopportunitytonotjustrunOpenStackvirtualizationonabare-metalcloud,buttoalsorunbare-metalKubernetes-onlyinstallationsthatcantakefulladvantageoftheidentity,storage,networking,andothercloudAPIsavailablethroughOpenStackservices.
DeliveringContainer-BasedApplicationsonOpenStack
Bothinfrastructurecontainersandbare-metalinfrastructureareimportant,butwhenmostpeoplethinkofcontainers,they’rethinkingofapplicationcontainers.Theisolation,encapsulation,andeaseofmaintenanceofferedbycontainersmakesthemanidealsolutionfordeliveringapplications.However,containersstillneedahostplatformtoservethemfrom,whetherbare-metal,publiccloud,orprivatecloud.
Kubernetesisaplatformfordeliveringapplications,andworksbestwithcloud-APIsthatcanautomatethedeliveryofcriticalinfrastructuresuchaspermanentstorage,load-balancers,networks,anddynamicallocationofcomputenodes.OpenStackdeliverscloudinfrastructure,whetherasanon-premprivatecloudorthroughanyoftheavailablepublicormanagedOpenStackclouds.
OpenStackwasoneofthefirstupstreamcloudprovidersforKubernetes,withanactiveteamofdevelopersmaintainingthe"Kubernetes/CloudProviderOpenStack"plugin.ThispluginallowsKubernetestotakeadvantageofCinderblockstorage,NeutronandOctaviaLoadBalancers,anddirectmanagementofcomputeresourceswithNova.UsingtheproviderisassimpleasdeployingthedrivertoyourKubernetesinstallation,settingaflagtoloadthedriver,andprovidingyourlocalusercloudcredentials.
ThereareanumberofsolutionsforinstallingKubernetesandotherapplicationframeworksontopofOpenStack.OneoftheeasiestwaystodelivercontainerframeworksistouseMagnum,anOpenStackprojectthatprovidesasimpleAPItodeployfullymanagedclustersbackedbyachoiceofseveralapplicationplatforms,includingKubernetes.It’sanexampleofaKubernetesdeploymentsystemthatreliesonOpenStackAPIsandcloudproviderplugin.Forexample,rightnowit’sbeingusedtomanageover200independentandfederatedKubernetesinstallationsonCERN’sOpenStackon-sitecloud,aswellasonpartnerclouds.Ifyoudon’thavetheMagnumAPIavailabletoyouinyourpreferredOpenStackcloud,youcanuseanyotherKubernetesinstallationtoolssuchasthekubeadm,KubernetesAnywhere,Cross-Cloud,orKubespray,toinstallandmanageyourKubernetesclusteronOpenStack.BecauseeachusesstandardKubernetes,it’seasytoenablethecloudproviderinterfacetotakeadvantageofstorageandloadbalancing.
Zun,anotherOpenStackproject,offersalighter-weightcontainerserviceAPIformanagingindividualcontainerswithouttheneedformanagingserversorclusters.AnOpenStack-hostedKubernetesclusteriselasticbecauseitcanbedynamicallyresizedbyaddingorremovingcloudresourcestotheclusterdirectlythroughtheNovaAPI.Alternatively,KubernetescanserveasacontainerbackendtoOpenStackZun,turningoverthemanagementofthepodinfrastructuretoZun.Itoffersalighter-weightandmulti-tenantcontainerserviceAPIforrunningcontainerswithouttheneedfordirectlycreatingservers.DirectintegrationwithNeutronandCinderareusedtoprovidenetworkingandvolumesforindividualcontainers.
Finally,theQinlingprojectoffers"FunctionasaService"thataimstoprovideaplatformtosupportserverlessfunctions,similartoLambda,AzureFunctions,orGoogleCloudFunctions.Itfurtherabstractsthemanagementofcontainers,andallowsuserstoacceleratedevelopmentwithanevent-driven,serverlesscomputeexperiencethatscalesondemand.QinlingsupportsdifferentcontainerorchestrationbackendslikeKubernetesandDockerswarm,avarietyofpopularfunctionpackagestoragebackendslikelocalstorageandOpenStackSwift.
KataContainers-SecureApplicationsthroughVirtualization
KataContainers,anewopensourceproject,isanovelimplementationofalightweightvirtualmachinethatseamlesslyintegrateswithinthecontainerecosystem.KataContainersareaslightandfastascontainersandintegratewiththecontainermanagementlayers–includingpopularorchestrationtoolssuchasDockerandKubernetes(k8s)–whilealsodeliveringthesecurityadvantagesofVMs.KataContainersadheretotheOpenContainerInitiative(OCI)standard,whichtheOpenStackFoundationisanactivememberof.KataContainersishostedattheOpenStackFoundation,butisaseparateprojectfromtheOpenStackprojectwithitsowngovernanceandcommunity.
Theindustryshifttocontainerspresentsuniquechallengesinsecuringuserworkloadswithinmulti-tenantenvironmentswithamixofbothtrustedanduntrustedworkloads.KataContainersuseshardware-backedisolationastheboundaryforeachcontainerorcollectionofcontainersinapod.Thisapproachaddressesthesecurityconcernsofasharedkernelinatraditionalcontainerarchitecture.
KataContainersisanexcellentfitforbothon-demand,event-baseddeploymentssuchascontinuousintegration/continuousdelivery,aswellaslongerrunningwebserverapplications.Kataalsoenablesaneasiertransitiontocontainersfromtraditionalvirtualizedenvironments,asitsupportslegacyguestkernelsanddevicepassthroughcapabilities.KataContainersdeliverenhancedsecurity,scalabilityandhigherresourceutilization,whileatthesametimeleadingtoanoverallsimplifiedstack.
Side-by-SideOpenStackandKubernetesIntegrations
Oneoftheprimarybenefitsofchoosingopensourceplatformsisinthestabilityofinterfacesacrossstandarddeploymentsofthoseplatforms.BoththeOpenStackFoundationandtheCloudNativeComputingFoundation(CNCF)maintaininteroperabilitystandardsforOpenStackcloudsandKubernetesclusters,guaranteeingthatlibraries,applications,anddriverswillworkacrossallplatformsregardlessofwheretheyaredeployed.Thiscreatesopportunitiesforside-by-sideintegrations,allowingbothOpenStackandKubernetestotakeadvantageoftheresourcesprovidedbytheother.
TheOpenStackSpecialInterestGroup(SIG-OpenStack)intheKubernetescommunitymaintainstheCloudProviderOpenStackplugin.InadditiontocloudproviderinterfaceforrunningKubernetesonOpenStack,italsomaintainsseveraldriversthatallowsKubernetestotakeadvantageofindividualOpenStackservices.Thesedriversinclude:
TwostandaloneCinderdrivers.AFlexVolumedriverusesanexec-basedmodeltointerfacewithdrivers,andaContainerStorageInterface(CSI)driverwhichusesastandardinterfaceforcontainerorchestrationsystemstoexposearbitrarystoragesystemstotheircontainerworkloads.Withsupportforover70storagedrivers,thesedriversmakeitpossibletointerfaceawealthofbattletestedproprietaryandopensourcestoragedevicesthroughasingleCinderAPI.Awebhook-basedauthenticationandauthorizationinterfacetoKeystone.Eachmode,authenticationandauthorization,canbeconfiguredindependentlyofoneanother.Thoughaworkinprogress,theinterfacesupportsasoft-multi-tenancythatbacksKubernetesRBACwithOpenStackKeystone.
BothOpenStackandKubernetessupporthighlydynamicnetworkingmodelsthatarebackedbyavarietyofdrivers.Becauseofthesestandardnetworkinterfaces,it’seasytobuildstandaloneOpenStackandKubernetesclusterswithstrongnetworkintegrations.WithinOpenStack,theKuryrprojectproducesaCommonNetworkInterface(CNI)driverthatdeliversNeutronnetworkingtoDockerandKubernetes.Ontheflipside,thereprojectslikeCalicoofferNeutrondrivers,providingdirectaccesstopopularKubernetesnetworkoverlaysthroughstandardNeutronAPIs.
III.CaseStudies
ManymembersoftheOpenStackcommunityarecontributingnewcodetovariousOpenStackprojectsrelevanttocontainers,evaluatingtheimplicationsandbenefitsofcontainers,andusingcontainersinproductiontosolvechallengesandunlocknewcapabilities.Thissectionhighlightssomeofthemostinterestingcasestudies.
AT&T
AT&T,oneofthelargesttelecommunicationscompaniesintheworld,leveragescontainertechnologytodeployandmanageOpenStackitself,relyingoninfrastructurecontainerstogeneratesimplicityandefficiency,withtheaimofbuildingtheir5GinfrastructureoncontainerizedOpenStack.
Toaccomplishtheirgoals,AT&TisusingtheOpenStack-HelmprojecttoorchestrateLOCI-basedOpenStackimagesacrossaKubernetescluster,alsoleveragingKubernetes,Docker,andthecoreOpenStackservices.They’realsousingBandit,Tempest,Patrole,andmanyotherOpenStackprojects.AT&TisalsocollaboratinginthecommunitytointroduceacollectionofundercloudprojectscalledAirship,whichwillprovisioncloudsfrombare-metaltoproduction-gradeKubernetesrunningOpenStackworkloads.
AT&Tisfindingthatcontainerizationallowsthemtoshifttraditionaldeployment-typeactivitiesfartotheleft,andtovalidatethemusingCI/CD.Kubernetesadditionallyprovidesmassivescalabilityandresiliency,aswellashookstoallowOpenStack-Helmtodeclarativelyconfigureoperationalbehavior,injectconfiguration,andaccomplishrollingupgradesandupdateswithoutimpactingtenantworkloads.
LeveragingcontainertechnologytodeployandmanageOpenStackshouldn’thavemuchobviousimpactontenants—withtheexceptionthattheywillhaveamorehighlyresilientplatform,andwillbeabletogetcloudfeaturesmore
frequentlyandwithminimalinterruption.AT&T’soperationsteamsnewexperiencewillshiftmoreoftheireffortstodefiningthedeclarativeconfigurationforasite,andtolettheKubernetes-orientedautomationcarryoutthedeploymentsthemselves.
AT&Taimstousethisarchitecturetopowerthevirtualnetworkfunctionsthatformthebackboneofitsconsumerandbusiness-focusedproductsandservices.TheinitialusecaseforAT&T’scontainerizedNetworkCloudwillbetheinitialdeploymentofVNFsfortheemerging5Gnetworking.OpenStackhasbeen,is,andwillbeanexcellentfitforAT&T’sVNF-focusedcloudusecases.ContainerizationissimplyanevolutionthatallowsAT&Ttodeploy,manage,andscaletheirOpenStackinfrastructureinamorereliable,rapid,zero-touchmanner.
Operationally,AT&Tisstilltestingthisapproachbuthascommittedtogetting5Gserviceintoproductionbeforetheendoftheyear.OpenStackandcontainertechnologywillformthebackboneofthisservice,whichisstrategicallyimportantforAT&T’smillionsofusers.Deployingtheir5GservicewilldemonstratetherelevanceofOpenStackandcontainersinamassivelydistributedproductionenvironment.
Cern
CERN,theEuropeanOrganizationforNuclearResearch,enablesphysicistsandengineerstoprobethefundamentalstructureoftheuniverse,usingtheworld’slargestandmostcomplexscientificinstrumentstostudythebasicconstituentsofmatter–thefundamentalparticles.TheCERNcloudprovidesphysicistswithcomputeresourcesforscientificcomputing,analyzingdatacomingfromtheLargeHadronColliderandotherexperiments.
CERNhasbeenrunningOpenStackinproductionsince2013andisnowprovidingservicesforvirtualmachines,bare-metalandcontainerswithinasinglecloud.Containersrunoneithervirtualmachinesorbare-metaldependingontheusecases,allprovisionedviaOpenStackMagnum.AselectionofdifferentcontainertechnologiesareavailableincludingKubernetes,DockerSwarmandDC/OS.
CERNiscurrentlyrunning250containerclustersprovisionedthroughMagnumontopofOpenStack.
CERN’sOpenStackcloudgivesusersself-serviceaccesstorequestaconfiguredcontainerenginewithacoupleofcommandsorviaawebGUI.Thisallowsrapidutilizationofthetechnologiesandcanscaleto1000sofnodesifneeded.BestpracticeconfigurationsareavailablewithbuiltinmonitoringandintegrationintoCERNstorageandauthenticationservices.
Runningthisresourcepoolefficiently,scalingitwithoutneedingextraoperationsmanpowerrequiresconsistentmanagementprocessesandtools.AddingcontainersviaMagnumontopofOpenStackenabledtheservicetousetheautomationpreviouslydeveloped,suchashardwarerepairprocessesandconsistentauthorisationmodelswhilesupportingrapidlyreallocationofresourcesdependingonuserneeds.
Asapubliclyfundedlaboratory,opensourcesolutionssuchasKubernetesandOpenStackprovideaframeworktocollaboratewithotherorganisationsandgivebacktothecommunities.CERNhasworkedwithanumberofvendorsthroughtheCERNopenlabframework,suchasRackspaceandHuawei,toprovidecloudsatscalewithfunctionalitieslikeMagnumandfederation.TheseexperiencesarealsosharedthroughOpenStackSpecialInterestGroups,withothersciencessuchastheSquareKilometerArray(SKA),publicpresentationssuchasKubeconEuropeandblogssuchastheOpenStackinProduction.
AtCERN,severalworkloadsrunwithincontainersprovisionedbyMagnum,theseinclude:
Reana/RecastThesetoolsprovideaframeworkforexecutingreusableworkflowsinHighEnergyPhysics.Containersoffertheabilitytopackagetheanalysissoftwareanddatainasingle,easilyshareableunitaswellaseasyscalingoutbothon-premisesandusingexternalresources.WorkisscheduledasKubernetesjobsbasedonYadageWorkflowssupportinganalysisanddatapreservationactivities.
SparkasaServiceRecently,KuberneteswasaddedasaresourcemanagerforSpark.SparkcanspawndriversandexecutorsaspodsandKubernetesisresponsiblefortheschedulingandlifecycle.AteamintheCERNITdepartmentisdevelopingaservicewhereuserscancreateKubernetesclustersondemandwithOpenStackMagnumanddeploySparkonKubernetes,providingalltherequiredintegrationswithCERN’sspecializedfilesystemsanddatasourcesinasecureway.UserswithafewcommandscaneffectivelycreateaSparkdeploymentwiththedesiredsize,onlyforthetimetheyneeditandwiththeoptiontoscaleupordowntheirdeploymentwhilerunning.
LHCexperimentdetectortriggersimulationforLHCupgradeTheLHCisduetobeupgradedtohigherluminosityduringthe2020swhichrequiressignificantenhancementsintheexperimenttriggerfarmswhichfilterthecollisions.LargescaleKubernetesclustershavebeencreatedtosimulatethedifferentapproachesfortheATLASexperimentandvalidatethedesign,resultinginsomefinetuningofKubernetesandOpenStackcomponents.
GitlabContinuousIntegrationRunnersGitlabenablesuserstobuildCI/CDjobsandexecutethemonsharedorprojectspecificrunners.CERNuserscanleveragetheCERNContainerServicetotestandbuildsoftware,buildandpublishcontainerimagesanddocumentationorsetcomplexpipelinesmanagingthefullapplicationlifecycle,includingautomateddeploymentsintodifferentenvironments.
FederatedKubernetescomputefarmswithexternalcloudsCERNusesfederationsofKubernetesclusterstosupportmulti-cloudoperations.Multipleclusterscanbeseamlesslyintegratedacrosscloudsofvaryingtechnologies,includingAWS,GCEandOpenStackcloudssuchasCERNandtheT-SystemsOpenTelekomCloudasdemonstratedatKubecon2018.
Integratingvirtualmachines,containerenginesandbare-metalunderasingleframeworkprovidesforeasyviewsonusageaccounting,ownershipandquota.ManilastoragedriversforKubernetesallowtransparentprovisioningoffileshares.ThissupportsboththeITdepartmentincapacityplanningandtheexperimentresourcecoordinatorsindefiningtheprioritiesfortheirworkinggroups.Resourcemanagementpoliciessuchasreassignmentorexpiryofresourcesondepartureofstaffarehandledinconsistentworkflows.
SKTelecom
SKTelecom(SKT),SouthKorea’slargesttelecommunicationsoperator,hasbeenexploringoptimizedapproachesfordeployingOpenStackonKuberneteswiththeaimofputtingcorebusinessfunctionsoncontainerizedOpenStackbytheendof2018.SKTleveragesKollaandOpenstack-Helm.withdeploymentsautomatedbyKubespray.SKTdevotesnearly100%ofit’sdevelopmenteffortstoOpenStack-Helm,andworkscloselywithAT&TtomakeOpenStack-Helmsuccessful.
SKThasalsoincorporatedothertoolsintotheirOpenStackonKubernetesefforts.Forlogging,monitoring,andalarms,theyareusingPrometheusandElasticsearch,Fluent-bit,andKibana,allofwhicharedefaultreferencetoolsintheOpenStack-Helmcommunity.SKTcombinesalloftheseintoasingleclosed-integratedsolutioncalledTACO:SKTAllContainerOpenStack.
SKTspecificallyemphasizesanautomatedcontinuousintegration/continuousdelivery(CI/CD)pipelinearoundcontainerizedOpenstackonKubernetes.SKT’sCIsystemconsistsofJenkins,Rally,Tempest,DockerRegistry,aswellasJiraandBitbucket.SKTalsodevelopedanopensourcetoolcalledCookiemonster,achaos-monkeylikeresiliencytesttoolforKubernetesdeploymentthatperformsresiliencytestsfortheirCIpipeline.
Witheverychange,SKTautomaticallybuildsandtestsboththeOpenStackcontainersandHelmcharts.Daily,theyautomaticallyinstallahighlyavailableOpenStackdeploymentwiththreecontrolnodesandtwocompute-nodes,run400testcasesfromTempestagainstittovalidatetheservices,andfinallyrunresiliencytestingwithCookiemonsterandRally.ThecompleteCIsystemisillustratedinthefollowingdiagram:
SKTautomatesitsdeploymentswithArmada,asub-projectofAirship,whichwasintroducedinthecommunityasanewopeninfrastructureprojectbyAT&T.SKTiscollaboratingincommunitytoprovideenhancementstotheprojectbasedontheirproductionuses.
Inpracticaluse,SKThasalreadyseenalargenumberofbenefitsfromdeployingOpenStackonKubernetesincluding:
SimpleandEasyInstallations.ClusterAuto-Healing.AnabilitytoupgradeandupdateOpenStackwithminimalimpacttorunningservices.Rapidadoptionofadvancedreleasemethodologies,includingblue-greendeployment,canaryreleases.CompleteautomatedmanagementofPythondependenciesthroughcontainerisolation.Securesecretandconfigurationmanagement.Fastandflexibleroll-outsofclusterupdates.
SKTisstilltestingtheapproach,butisactivelymovingtowardsrunningtheirOpenStack-Helmdeploymentsinproduction.Byendofthisyear,SKTwillhaveatleastthreeproductionclusters,withthefourthandlargestcomingonlinein2019.Theseusecasesinclude:
BigDataplatform(plannedtogoliveQ42018)Avirtualdesktopinfrastructureplatform(productionreadybyQ42018)AGeneralpurposeInternalPrivateCloud(plannedtogoliveQ32018)Atelconetworkinfrastructurebuiltonvirtualnetworkfunctions(plannedtoopensometimein2019)
SKTisalsotryingtoimproveautomationontelecominfrastructureoperationbyutilizingcontainerizedVNFsandleveragingcontainers’autohealingandfastscale-outfeatures.InordertoallowinteractionbetweenvirtualmachinebasedVNFsandcontainerizedVNFs,SimplifiedOverlayNetworkArchitecture(SONA),whichisavirtualnetworksolutionforOpenStack,willsupportcommunicationbetweenVMsandcontainers.SONAusestheKuryrprojectforintegrationofOpenStackandKubernetes,anditoptimizesnetworkperformanceusingsoftwaredefinednetworkingtechnologies.
Overall,SKTisfindingthatKuberneteshelpssolvemanyofthecomplexitiesofdeployingandoperatingOpenStack.SimplifyingOpenStackgivesthemapowerfulapproachtodeliveradvancedinfrastructureinnovationforthe5Gera.
FocusingeffortsonOpenstackonKubernetesdramaticallyincreasedtheirinternalcapabilitytodealwiththeevolvingshifttowardmicroservicesincontainersandbecomeacriticalinfrastructurefordeliveringArtificialIntelligence,InternetofThings,andMachineLearning.
Superfluidity
TheSuperfluidityprojectismadeupof18partnersfrom12Europeancountries.Itaimstoenhancetheabilitytoinstantiateserviceson-the-fly,runthemanywhereinthenetwork(core,aggregation,edge)andshiftthemtransparentlytodifferentlocations.SUPERFLUIDITYisaEuropeanResearchproject(Horizon2020)tryingtobuildthebasicinfrastructureblocksfor5Gnetworksbyleveragingandextendingwellknownopensourceprojects.SUPERFLUIDITYwillprovideaconvergedcloud-based5Gconceptthatwillenableinnovativeusecasesinthemobileedge,empowernewbusinessmodels,andreduceinvestmentandoperationalcosts.
Topursuethesegoals,theprojectconsortiumisshiftingawayfromlegacy,VM-basedapplicationstoCloudNativecontainerizedapplications.KuryrservesasabridgebetweenOpenStackvirtualmachines,andKubernetesandOpenShiftcontainerizedservices.
TheprojectmakesuseofManageIQasacentralnetworksfunctionvirtualizationorchestrator(NFVO),AnsibleforApplicationdeploymentandlifecyclemanagement,OpenStackservicesincludingHeat,Neutron,andOctavia,andKubernetesthroughOpenShiftforVMsandcontainersintegration.
ByleveragingAnsibleplaybooksexecutedfromtheManageIQappliance,SUPERFLUIDITYoffersacommonwaytodeployapplications.TheseapplicationsinturnusethecloudorchestrationfunctionalityprovidedbyOpenStackHeattemplatesandOpenShifttemplates.
Theconsortiumdeploys5Gcloudradioaccessnetworks(CRAN)andmobileedgecomputing(MEC)componentswithincontainers.Italsodeployshighthroughputapplicationslikevideostreamingontopofthedistributedinfrastructure.
ShiftingtowardacloudnativeapproachtoapplicationdeliveryallowsforrapidandresilientSUPERFLUIDITYinstallations.ItenablesasmoothtransitionfromVM-basedapplicationsandcomponentstocontainers,whileretainingtheversatilitytoenableVMsforsomespecificapplications.Examplesoftheseapplicationsarespecialsecurityprotectionsornetworkaccelerationrequiredbysingle-routeinput/outputvirtualization(SRIOV).
Inscaleperformancetesting,SUPERFLUIDITYwasabletolaunchapproximately1000podsatarateof22pods/second(withtimemeasuredfromcreationtorunning).ThisremarkableperformancewasachievedbyrunningOpenShiftonVMsmanagedbyOpenStack,withKuryractingasapodnetworkdrivertoavoiddouble-encapsulationperformancehits.
IV.Conclusion
Overthepastfewyears,ascontainershavebecomeanimportanttoolfordevelopersandorganizationsalike,OpenStackhasleverageditsmodulardesignandexpansivecommunitytointegratecontainertechnologiesatmanylevels.ThiscanbeseenbothbythevariousorganizationsbringingcontainersandOpenStackintoproduction,andthenumberofprojectsthatworkalongsidecontainerstodelivernewcapabilities.TheOpenStackFoundationiscommittedtoensuringthatemergingtechnologiescanbeincorporatedandutilizedwithinOpenStack,andcontainersareanimportantexampleofthatcommitment.
Tolearnmore,visittheContainersLandingPage,whereyoucanfindacopyofthisdocumentaswellaslinkstodozensofvideosfocusedontheintegrationsofOpenStackandcontainers.KubernetesSIG-OpenStackhasaSlackchannel,mailinglist,andweeklymeetingifyouengagedirectlywiththecommunitythat’sbuildingKubernetesandOpenStack
integrations.
V.OpenSourceProjectIndex
Airship
Airshipisacollectionofinteroperableandlooselycoupledopensourcetoolsthatprovideautomatedcloudprovisioningandmanagementinadeclarativeway,basedaroundKubernetesasanapplicationplatform.
Ansible
AnsibleisacommonlyusedorchestrationtoolusedtodeployandmanageOpenStackinstallations.
Cinder
OpenStackCinderoffersblockstorageasaservice,providingasingleAPIbackedbyoverseventydifferentpossiblestoragedrivers.
CloudProviderOpenStack
CloudProviderOpenStackistheimplementationoftheKubernetesCloudProviderinterface.ItallowsanOpenStack-hostedKubernetesclustertodirectlyaccessstorageandloadbalancerresourcesintheOpenStackcloud.
Calico
CalicoisanetworkoverlaywithdriversforbothKubernetesandOpenStackthatfeaturesL3-onlyrouting.
Cyborg
CyborgisanOpenStackprojectthatprovidesageneralmanagementframeworkforhardwareacceleratorsincludingFPGA,GPU,ASIC,andothers.Workisinprogresstosurfaceageneralhardwareinterfacetopods.
Docker
Dockerisanopensourcecontainervirtualizationframework,usedtohostcontainerizedapplications.
Helm
HelmistheofficialpackagemanagerforKubernetes.ApplicationdeploymentsaredescribedbyHelm-Charts,whichcanbeautomaticallydeployedandmanagedonaKubernetescluster.
Ironic
IronicistheOpenStackbare-metalservice.RunningeitherasastandaloneserviceorasadrivertoOpenStackNova,itcanmanagethecompletelife-cycleofbare-metalsystems,includingenrollment,provisioning,maintenance,anddecommissioning.
Loci
LOCIisanOpenStackprojecttobuildlightweight,OCIcompliantcontainersforOpenStackprojects.
LXC
LXCisalow-levelcontainervirtualizationinterfacethattakesadvantageofLinuxkernelnamespaceisolationandothertechnologiestocreateisolatedlinuxruntimes.
KataContainers
KataContainersisastandardimplementationoflightweightVirtualMachines(VMs)thatfeelandperformlikecontainers,butprovidetheworkloadisolationandsecurityadvantagesofVMs.
Keystone
KeystoneistheOpenStackIdentityservicethatprovidesmeansforauthenticatingandmanaginguseraccountsandroleinformationprimarilyfortheOpenStackcloudenvironment,butalsoasaplugintootherenvironments,includingKubernetes.
Kolla(Containers)
Kolla(Containers)isanOpenStackprojecttobuildcontainersforeachOpenStackservice.Itincludesasophisticatedbuildandtemplatingsystems,andiscapableofbuildingcontainersfrombothsourceandpackagesonavarietyofhost
operatingsystems.
KollaAnsible
KollaAnsibleisanOpenStackprojectthatusesAnsibletodeployandmaintainafullOpenStackinstallationusingKollacontainers.
Kubernetes
Kubernetesisacontainerorchestrationsystemthatdeliversrobustandhighly-availableapplicationsontopofcloud-infrastructure.
Kuryr
KuryrisanOpenStackprojectthatprovidesaNeutronnetworkoverlaytocontainerruntimes,includingDockerandKubernetes.Itaimstobethe“integrationbridge”forcontainerandVMnetworks.
Magnum
MagnumisanOpenStackprojectthatoffersmanagedcontainerplatformsasaservice,includingKubernetes,DockerSwarm,Mesos,andDC/OSplatforms.Itiscapableofcreatingtenantisolatedapplicationplatformsthroughasimpleuser-facingAPI.
Neutron
NeutronistheOpenStacksoftware-definednetworkingservice,offeringasingleAPItodeliverdynamicnetworkinfrastructurebackedbydozensofnetworkdrivers.
OpenStackAnsible
OpenStackAnsibleisaprojectforbuildingOpenStackservicesintoLXCcontainers,andfordeployingandmanagingOpenStackinstallationswithinthosecontainerizedservices.
OpenStackHelm
OpenStackHelmisanOpenStackprojectthatdeploysandmanagesthelifecycleofOpenStackandsupportinginfrastructureontopofKubernetes(egCephandMariaDB),deliveringproductionreadydeployments,forarangeofusecasesfromsmalledgedeploymentstolargecentraloffices.LeveragingtheHelmpackagemanagementsystem.OpenStackHelmhassupportforbothbaremetal(Ironic)andvirtual(Nova/KVM)workloadmanagement,andisimageagnosticsupportingbothLOCIandKollacontainers.
Qinling
QinlingisanOpenStackprojecttodeliverFunctionsasaService.Qinlingsupportsdifferentcontainerorchestrationplatforms,suchasKubernetesandDockerSwarm,aswellasdifferentfunctionpackagestoragebackendssuchaslocalfile-store,OpenStackSwift,andS3.
Triple-O
TripleOisaprojectaimedatinstalling,upgradingandoperatingOpenStackcloudsusingOpenStack’scloudservicesasthefoundation-buildingonNova,Ironic,Neutron,HeatandAnsibletoautomatecloudmanagement.
Zun
ZunistheOpenStackContainersservice.ItaimstoprovideanAPIserviceforrunningapplicationcontainerswithouttheneedtomanageserversorclusters.
VI.Authors
MembersoftheOpenStackSIG-KubernetesCommunity
JaesukAhn,SKTelecomChristianBerendt,BetacloudSolutionsGmbHAnneBertucio,OpenStackFoundationPeteBirley,AT&TChrisHoge,OpenStackFoundationLingxianKong,CatalystCloudHongbinLu,HuaweiDanielMellado,RedHat,Inc.AllisonPrice,OpenStackFoundationDavidRabel,B1SystemsGmbHSanghoShin,SKTelecom
DavanumSrinivas,HuaweiLuisTomás,RedHat,Inc.SamYaple,VerizonDigitalMediaServicesMikhailFedosin,RedHat,Inc.FlavioPercoco,RedHat,Inc.
Editor
BrianEWhitaker,ZettabyteContentLLC
