Post on 19-Feb-2022
LeveragingContainersandOpenStackAComprehensiveReviewIntroduction
Imaginethatyouaretaskedtobuildanentireprivatecloudinfrastructurefromthegroundup.Youhavealimitedbudget,asmallbutdedicatedteam,andareaskedtopulloffamiracle.
Afewyearsago,you’dbuildaninfrastructurewithapplicationsrunninginvirtualmachines,withsomebare-metalmachinesforlegacyapplications.Asinfrastructurehasevolved,virtualmachines(VMs)enabledgreaterlevelsofefficiencyandagility,butVMsalonedon’tcompletelymeettheneedsofanagileapproachtoapplicationdeployment.Theycontinuetoserveasafoundationforrunningmanyapplications,butincreasingly,developersarelookingtowardtheemergingtrendofcontainersforleading-edgeapplicationdevelopmentanddeploymentbecausecontainersofferincreasedlevelsofagilityandefficiency.
ContainertechnologieslikeDockerandKubernetesarebecomingtheleadingstandardsforbuildingcontainerizedapplications.Theyhelpfreeorganizationsfromcomplexitythatlimitsdevelopmentagility.Containers,containerinfrastructure,andcontainerdeploymenttechnologieshaveproventhemselvestobeverypowerfulabstractionsthatcanbeappliedtoanumberofdifferentusecases.UsingsomethinglikeKubernetes,anorganizationcandeliveracloudthatsolelyusescontainersforapplicationdelivery.
Butaleading-edgeprivatecloudisn’tjustaboutcontainers,andcontainersaren’tappropriateforallworkloadsandusecases.Today,mostprivatecloudinfrastructuresneedtoencompassbare-metalmachinesformanaginginfrastructure,virtualmachinesforlegacyapplications,andcontainersfornewerapplications.Theabilitytosupport,manageandorchestrateallthreeapproachesisthekeytooperationalefficiency.
OpenStackiscurrentlythebestavailableoptionforbuildingprivateclouds,withtheabilitytomanagenetworking,storageandcomputeinfrastructure,withsupportforvirtualmachines,bare-metal,andcontainersfromonecontrolplane.WhileKubernetesisarguablythemostpopularcontainerorchestratorandhaschangedapplicationdelivery,itdependsontheavailabilityofasolidcloudinfrastructure,andOpenStackoffersthemostcomprehensiveopensourceinfrastructureforhostingapplications.OpenStack’smulti-tenantcloudinfrastructureisanaturalfitforKubernetes,withseveralintegrationpoints,deploymentsolutions,andabilitytofederateacrossmultipleclouds.
Inthispaper,we’regoingtoexplorehowcontainersworkwithinOpenStack,examinevarioususecases,andprovideanoverviewofopensourceprojects,fromOpenStackandelsewhere,thathelpmakecontainersatechnologythat’seasilyadoptedandutilized.
I.AHighLevelViewofContainersinOpenStack
TherearethreeprimaryscenarioswherecontainersandOpenStackintersect.
Thefirstscenario,calledinfrastructurecontainers,allowsoperatorstoleveragecontainersinawaythatimprovescloud
infrastructuredeployment,management,andoperation.Inthisscenario,containersaresetuponabare-metalinfrastructure,andareallowedprivilegedaccesstohostresources.Thisaccessallowsthemtotakedirectadvantageofcompute,networking,andstorageresourcesthatcontainerruntimesaretypicallytryingtohidefromusers.Thecontainersisolatetheoftencomplexsetofdependenciesthateachapplicationdependson,whilestillallowingtheinfrastructureapplicationstodirectlymanageandmanipulatetheunderlyingsystemresources.Whenthetimecomestoupgradeanservice,theupgradecanbehandledwithoutchangesindependenciesdisruptingco-locatedservices.
ModernversionsofOpenStackhaveembracedthisinfrastructurecontainermodel,andit’snownormaltomanageanentirelifecycleofanOpenStackdeploymentwithacombinationoforchestrationtoolingandcontainerizedservices.Infrastructurecontainersenableoperatorstousecontainerorchestrationtechnologiestosolvemanyissues,particularlyaroundrapidlyiterating/upgradingexistingsoftwareincludingOpenStack.RunningOpenStackwithincontainershelpsoperatorstosolveDay2challenges,includingaddingnewcomponentsforservices,upgradingversionsofsoftwarequickly,andrapidlyrollingupdatesacrossmachinesanddatacenters.ThisapproachbringstheagilityofcontainerstotheproblemofOpenStackdeploymentandupgrades.
Thesecondscenarioisconcernedwithhostingcontainerizedapplicationframeworksoncloudinfrastructure.ThesecanincludeContainerOrchestrationEngines(COEs)likeDockerSwarmandKubernetes,orlighter-weightcontainer-focusedservicesandserverlessapplicationprogramminginterfaces(APIs).Whetheronbare-metalorVMs,theOpenStackcommunityhasworkedtoensurethatit’spossibletodelivercontainerizedapplicationsonasecure,tenant-isolatedcloudhost.ThisscenarioisfacilitatedbydriversthatallowprojectslikeKubernetestodirectlytakeadvantageofOpenStackAPIsforstorage,load-balancing,andidentity.ItalsoincludesAPIsforprovisioningmanagedKubernetesclustersandapplicationcontainersondemand.Withthesecapabilities,developmentteamscanwritenewcontainerizedapplicationsandquicklyprovisionKubernetesclustersonOpenStackclouds.It’sacompleteapplicationlifecyclesolutionthatgivesthemtheresourcesneededtodevelop,test,anddebugtheircode,withrobustautomationtodeploytheirapplicationsintoproduction.
Inthefinalscenario,weconsidertheinteractionsbetweenindependentOpenStackandCOEdeployments,andinthispaperparticularlyKubernetesclusters.ConsistencyandinteroperabilityofAPIsacrossbothOpenStackandKubernetesclustersistheprimarysourceofsuccessforthisscenario.Forexample,it’spossibleforKubernetestodirectlyattachtoOpenStackCinderhostedvolumes,useOpenStackKeystoneasanauthorizationandauthenticationbackend,orconnecttoOpenStackNeutronasanetworkoverlaywithOpenStackKuryr.Conversely,it’spossibleforanOpenStackcloudtosharethesamenetworkoverlayasaKubernetesclusterwithNeutrondriversforprojectslikeCalico.Thethirdscenarioislessfocusedonhowacloudserviceishosted(beitKubernetesorOpenStack),andmoreonhowindependentservicesinteract.
II.OpenStackContainerIntegrationPoints
DeployingOpenStackInfrastructureonContainers
Asnotedintheintroduction,thedeploymentandmanagementofOpenStackhaschangedsignificantlywiththeriseofcontainers,becausecontainersunlocknewapproachestomanaginginfrastructurecode.Previousmanagementstrategiesrequiredeitherthecreationandmaintenanceofheavyweightgoldenmachineimages,orusingbrittlestate-maintainingconfiguration-managementsystems.Eachapproachcomeswithcomplexitiesandrestrictions.Addingtothedegreeofdifficultyisthemanagementofacollectionofservicesthatallrequiretheirowndependenciesthatchangefromrelease-to-release.Withoutsomeformofapplicationisolation,solvingforthedependenciesbecomesdifficultifnotimpossible.
InfrastructurecontainersenablenewOpenStackdeploymentprojectstostrikeabalancebetweenthetwowhileelegantlysolvingthedependencyproblem.Usinglightweight,independent,self-contained,andtypicallystatelessapplicationcontainers,acloudoperatorgainstremendousflexibilitywhendeployingacomplexcontrolplane.Combinedwithacontainerruntimeandanorchestrationengine,infrastructurecontainersmakeitpossibletoquicklydeploy,maintain,andupgradecomplexandhighlyavailableinfrastructure.
InbuildinganOpenStackcluster,thereareseveraldimensionsforchoosingdeploymenttechnologies.AnoperatorcouldchooseLinuxContainers(LXC)orDockerfortheirbasecontainers,usepre-builtorcustom-builtapplicationcontainers,andselecteithertraditionalconfiguration-managementsystemsfororchestrationoramoremodernapproachlikeKubernetes.Table1summarizestheexistingOpenStackdeploymentprojectsandtheirunderlyingtechnologies.
Table1
Project
OpenStack-Ansible
Kolla-Ansible
Triple-O
OpenStack-Helm
ContainerType
LXC
Docker
Docker
Docker
SupportedContainers
OSA LXC Containers
Kolla Containers
Kolla Containers
Kolla Containers
Loci Containers
Project
Ansible
Ansible
Ansible
Kubernetes and Helm
UnderlyingeachofthesedeploymentsystemsaredifferentapproachestobuildingasetofcontainersfortheOpenStackcodeandsupportingservices.TheOpenStackAnsible(OSA)andKollaprojectsprovidetheirownproject-hostedbuildsystems,whileLOCIfocusesonbuildingprojectapplicationcontainers,withoutaspecificorchestrationsysteminmind.Atahighlevel,thedifferencesare:
1. OSAisuniqueinthatitreliesonlower-levelLXCcontainers,andhasacustombuildsystemforcreatingLXCapplicationcontainers.
2. TheKollabuildsystemproducesDockercontainers,oneforeachservice,alongwithsupportingcontainersforinitializingandmanaginganOpenStackdeployment.Kollacontainersarehighlyconfigurable,withachoiceofbaseoperatingsystem,sourceorpackageinstallations,andatemplateengineforevenfurthercustomization.
3. ThefinaloptionforbuildingOpenStackapplicationcontainersisLOCI.LOCIalsobuildsDockercontainers,anddeliversonecontainerforeachproject.LOCIisfocusedonproducingcompactandsecurecontainersquickly,forallcommondistributions,withtheexpectationthattheywillbeusedasafoundationtobuilduponbythedeploymentsystem.
Bare-MetalInfrastructure-OpenStackandSolvingtheBootstrapProblem
Atthefoundationofeverycloud,thereexistsadatacenterofbare-metalserversthathosttheinfrastructureservices.Even“serverlesscomputing”isrunningsoftwareonacloudonhardwareinadatacenter.TheproblemofhowtobootstraphardwareinfrastructureisacriticalproblemthatOpenStacksoftwareisuniquelyqualifiedtoaddressinawaythatgivescloud-likequalitiestobare-metalmanagement.
OpenStackIronicprovidesbare-metalasaservice.Asastandaloneserviceitcandiscoverbare-metalnodes,catalogtheminamanagementdatabase,andmanagetheentireserverlifecycleincludingenrolling,provisioning,maintenance,anddecommissioning.WhenusedasadrivertoOpenStackNovaandcombinedwiththefullsuiteofOpenStackservices,itdeliversapowerful,cloud-likeserviceformanagingyourentirebare-metalinfrastructure.
Thisraisesthequestion:HowdoesonebootstrapOpenStackservicestomanagebare-metalinfrastructure?Onetypicalsolutionistousethesamecontainer-basedinstallationtoolsasdescribedintheprevioussectionstocreateaseedinstallation.Thisseed,oftencalledan‘undercloud’,canbeusedtoentirelyautomatethemanagementofabare-metalclusterasifitwereavirtualizedcloud.
ThisopensupanopportunitytonotjustrunOpenStackvirtualizationonabare-metalcloud,buttoalsorunbare-metalKubernetes-onlyinstallationsthatcantakefulladvantageoftheidentity,storage,networking,andothercloudAPIsavailablethroughOpenStackservices.
DeliveringContainer-BasedApplicationsonOpenStack
Bothinfrastructurecontainersandbare-metalinfrastructureareimportant,butwhenmostpeoplethinkofcontainers,they’rethinkingofapplicationcontainers.Theisolation,encapsulation,andeaseofmaintenanceofferedbycontainersmakesthemanidealsolutionfordeliveringapplications.However,containersstillneedahostplatformtoservethemfrom,whetherbare-metal,publiccloud,orprivatecloud.
Kubernetesisaplatformfordeliveringapplications,andworksbestwithcloud-APIsthatcanautomatethedeliveryofcriticalinfrastructuresuchaspermanentstorage,load-balancers,networks,anddynamicallocationofcomputenodes.OpenStackdeliverscloudinfrastructure,whetherasanon-premprivatecloudorthroughanyoftheavailablepublicormanagedOpenStackclouds.
OpenStackwasoneofthefirstupstreamcloudprovidersforKubernetes,withanactiveteamofdevelopersmaintainingthe"Kubernetes/CloudProviderOpenStack"plugin.ThispluginallowsKubernetestotakeadvantageofCinderblockstorage,NeutronandOctaviaLoadBalancers,anddirectmanagementofcomputeresourceswithNova.UsingtheproviderisassimpleasdeployingthedrivertoyourKubernetesinstallation,settingaflagtoloadthedriver,andprovidingyourlocalusercloudcredentials.
ThereareanumberofsolutionsforinstallingKubernetesandotherapplicationframeworksontopofOpenStack.OneoftheeasiestwaystodelivercontainerframeworksistouseMagnum,anOpenStackprojectthatprovidesasimpleAPItodeployfullymanagedclustersbackedbyachoiceofseveralapplicationplatforms,includingKubernetes.It’sanexampleofaKubernetesdeploymentsystemthatreliesonOpenStackAPIsandcloudproviderplugin.Forexample,rightnowit’sbeingusedtomanageover200independentandfederatedKubernetesinstallationsonCERN’sOpenStackon-sitecloud,aswellasonpartnerclouds.Ifyoudon’thavetheMagnumAPIavailabletoyouinyourpreferredOpenStackcloud,youcanuseanyotherKubernetesinstallationtoolssuchasthekubeadm,KubernetesAnywhere,Cross-Cloud,orKubespray,toinstallandmanageyourKubernetesclusteronOpenStack.BecauseeachusesstandardKubernetes,it’seasytoenablethecloudproviderinterfacetotakeadvantageofstorageandloadbalancing.
Zun,anotherOpenStackproject,offersalighter-weightcontainerserviceAPIformanagingindividualcontainerswithouttheneedformanagingserversorclusters.AnOpenStack-hostedKubernetesclusteriselasticbecauseitcanbedynamicallyresizedbyaddingorremovingcloudresourcestotheclusterdirectlythroughtheNovaAPI.Alternatively,KubernetescanserveasacontainerbackendtoOpenStackZun,turningoverthemanagementofthepodinfrastructuretoZun.Itoffersalighter-weightandmulti-tenantcontainerserviceAPIforrunningcontainerswithouttheneedfordirectlycreatingservers.DirectintegrationwithNeutronandCinderareusedtoprovidenetworkingandvolumesforindividualcontainers.
Finally,theQinlingprojectoffers"FunctionasaService"thataimstoprovideaplatformtosupportserverlessfunctions,similartoLambda,AzureFunctions,orGoogleCloudFunctions.Itfurtherabstractsthemanagementofcontainers,andallowsuserstoacceleratedevelopmentwithanevent-driven,serverlesscomputeexperiencethatscalesondemand.QinlingsupportsdifferentcontainerorchestrationbackendslikeKubernetesandDockerswarm,avarietyofpopularfunctionpackagestoragebackendslikelocalstorageandOpenStackSwift.
KataContainers-SecureApplicationsthroughVirtualization
KataContainers,anewopensourceproject,isanovelimplementationofalightweightvirtualmachinethatseamlesslyintegrateswithinthecontainerecosystem.KataContainersareaslightandfastascontainersandintegratewiththecontainermanagementlayers–includingpopularorchestrationtoolssuchasDockerandKubernetes(k8s)–whilealsodeliveringthesecurityadvantagesofVMs.KataContainersadheretotheOpenContainerInitiative(OCI)standard,whichtheOpenStackFoundationisanactivememberof.KataContainersishostedattheOpenStackFoundation,butisaseparateprojectfromtheOpenStackprojectwithitsowngovernanceandcommunity.
Theindustryshifttocontainerspresentsuniquechallengesinsecuringuserworkloadswithinmulti-tenantenvironmentswithamixofbothtrustedanduntrustedworkloads.KataContainersuseshardware-backedisolationastheboundaryforeachcontainerorcollectionofcontainersinapod.Thisapproachaddressesthesecurityconcernsofasharedkernelinatraditionalcontainerarchitecture.
KataContainersisanexcellentfitforbothon-demand,event-baseddeploymentssuchascontinuousintegration/continuousdelivery,aswellaslongerrunningwebserverapplications.Kataalsoenablesaneasiertransitiontocontainersfromtraditionalvirtualizedenvironments,asitsupportslegacyguestkernelsanddevicepassthroughcapabilities.KataContainersdeliverenhancedsecurity,scalabilityandhigherresourceutilization,whileatthesametimeleadingtoanoverallsimplifiedstack.
Side-by-SideOpenStackandKubernetesIntegrations
Oneoftheprimarybenefitsofchoosingopensourceplatformsisinthestabilityofinterfacesacrossstandarddeploymentsofthoseplatforms.BoththeOpenStackFoundationandtheCloudNativeComputingFoundation(CNCF)maintaininteroperabilitystandardsforOpenStackcloudsandKubernetesclusters,guaranteeingthatlibraries,applications,anddriverswillworkacrossallplatformsregardlessofwheretheyaredeployed.Thiscreatesopportunitiesforside-by-sideintegrations,allowingbothOpenStackandKubernetestotakeadvantageoftheresourcesprovidedbytheother.
TheOpenStackSpecialInterestGroup(SIG-OpenStack)intheKubernetescommunitymaintainstheCloudProviderOpenStackplugin.InadditiontocloudproviderinterfaceforrunningKubernetesonOpenStack,italsomaintainsseveraldriversthatallowsKubernetestotakeadvantageofindividualOpenStackservices.Thesedriversinclude:
TwostandaloneCinderdrivers.AFlexVolumedriverusesanexec-basedmodeltointerfacewithdrivers,andaContainerStorageInterface(CSI)driverwhichusesastandardinterfaceforcontainerorchestrationsystemstoexposearbitrarystoragesystemstotheircontainerworkloads.Withsupportforover70storagedrivers,thesedriversmakeitpossibletointerfaceawealthofbattletestedproprietaryandopensourcestoragedevicesthroughasingleCinderAPI.Awebhook-basedauthenticationandauthorizationinterfacetoKeystone.Eachmode,authenticationandauthorization,canbeconfiguredindependentlyofoneanother.Thoughaworkinprogress,theinterfacesupportsasoft-multi-tenancythatbacksKubernetesRBACwithOpenStackKeystone.
BothOpenStackandKubernetessupporthighlydynamicnetworkingmodelsthatarebackedbyavarietyofdrivers.Becauseofthesestandardnetworkinterfaces,it’seasytobuildstandaloneOpenStackandKubernetesclusterswithstrongnetworkintegrations.WithinOpenStack,theKuryrprojectproducesaCommonNetworkInterface(CNI)driverthatdeliversNeutronnetworkingtoDockerandKubernetes.Ontheflipside,thereprojectslikeCalicoofferNeutrondrivers,providingdirectaccesstopopularKubernetesnetworkoverlaysthroughstandardNeutronAPIs.
III.CaseStudies
ManymembersoftheOpenStackcommunityarecontributingnewcodetovariousOpenStackprojectsrelevanttocontainers,evaluatingtheimplicationsandbenefitsofcontainers,andusingcontainersinproductiontosolvechallengesandunlocknewcapabilities.Thissectionhighlightssomeofthemostinterestingcasestudies.
AT&T
AT&T,oneofthelargesttelecommunicationscompaniesintheworld,leveragescontainertechnologytodeployandmanageOpenStackitself,relyingoninfrastructurecontainerstogeneratesimplicityandefficiency,withtheaimofbuildingtheir5GinfrastructureoncontainerizedOpenStack.
Toaccomplishtheirgoals,AT&TisusingtheOpenStack-HelmprojecttoorchestrateLOCI-basedOpenStackimagesacrossaKubernetescluster,alsoleveragingKubernetes,Docker,andthecoreOpenStackservices.They’realsousingBandit,Tempest,Patrole,andmanyotherOpenStackprojects.AT&TisalsocollaboratinginthecommunitytointroduceacollectionofundercloudprojectscalledAirship,whichwillprovisioncloudsfrombare-metaltoproduction-gradeKubernetesrunningOpenStackworkloads.
AT&Tisfindingthatcontainerizationallowsthemtoshifttraditionaldeployment-typeactivitiesfartotheleft,andtovalidatethemusingCI/CD.Kubernetesadditionallyprovidesmassivescalabilityandresiliency,aswellashookstoallowOpenStack-Helmtodeclarativelyconfigureoperationalbehavior,injectconfiguration,andaccomplishrollingupgradesandupdateswithoutimpactingtenantworkloads.
LeveragingcontainertechnologytodeployandmanageOpenStackshouldn’thavemuchobviousimpactontenants—withtheexceptionthattheywillhaveamorehighlyresilientplatform,andwillbeabletogetcloudfeaturesmore
frequentlyandwithminimalinterruption.AT&T’soperationsteamsnewexperiencewillshiftmoreoftheireffortstodefiningthedeclarativeconfigurationforasite,andtolettheKubernetes-orientedautomationcarryoutthedeploymentsthemselves.
AT&Taimstousethisarchitecturetopowerthevirtualnetworkfunctionsthatformthebackboneofitsconsumerandbusiness-focusedproductsandservices.TheinitialusecaseforAT&T’scontainerizedNetworkCloudwillbetheinitialdeploymentofVNFsfortheemerging5Gnetworking.OpenStackhasbeen,is,andwillbeanexcellentfitforAT&T’sVNF-focusedcloudusecases.ContainerizationissimplyanevolutionthatallowsAT&Ttodeploy,manage,andscaletheirOpenStackinfrastructureinamorereliable,rapid,zero-touchmanner.
Operationally,AT&Tisstilltestingthisapproachbuthascommittedtogetting5Gserviceintoproductionbeforetheendoftheyear.OpenStackandcontainertechnologywillformthebackboneofthisservice,whichisstrategicallyimportantforAT&T’smillionsofusers.Deployingtheir5GservicewilldemonstratetherelevanceofOpenStackandcontainersinamassivelydistributedproductionenvironment.
Cern
CERN,theEuropeanOrganizationforNuclearResearch,enablesphysicistsandengineerstoprobethefundamentalstructureoftheuniverse,usingtheworld’slargestandmostcomplexscientificinstrumentstostudythebasicconstituentsofmatter–thefundamentalparticles.TheCERNcloudprovidesphysicistswithcomputeresourcesforscientificcomputing,analyzingdatacomingfromtheLargeHadronColliderandotherexperiments.
CERNhasbeenrunningOpenStackinproductionsince2013andisnowprovidingservicesforvirtualmachines,bare-metalandcontainerswithinasinglecloud.Containersrunoneithervirtualmachinesorbare-metaldependingontheusecases,allprovisionedviaOpenStackMagnum.AselectionofdifferentcontainertechnologiesareavailableincludingKubernetes,DockerSwarmandDC/OS.
CERNiscurrentlyrunning250containerclustersprovisionedthroughMagnumontopofOpenStack.
CERN’sOpenStackcloudgivesusersself-serviceaccesstorequestaconfiguredcontainerenginewithacoupleofcommandsorviaawebGUI.Thisallowsrapidutilizationofthetechnologiesandcanscaleto1000sofnodesifneeded.BestpracticeconfigurationsareavailablewithbuiltinmonitoringandintegrationintoCERNstorageandauthenticationservices.
Runningthisresourcepoolefficiently,scalingitwithoutneedingextraoperationsmanpowerrequiresconsistentmanagementprocessesandtools.AddingcontainersviaMagnumontopofOpenStackenabledtheservicetousetheautomationpreviouslydeveloped,suchashardwarerepairprocessesandconsistentauthorisationmodelswhilesupportingrapidlyreallocationofresourcesdependingonuserneeds.
Asapubliclyfundedlaboratory,opensourcesolutionssuchasKubernetesandOpenStackprovideaframeworktocollaboratewithotherorganisationsandgivebacktothecommunities.CERNhasworkedwithanumberofvendorsthroughtheCERNopenlabframework,suchasRackspaceandHuawei,toprovidecloudsatscalewithfunctionalitieslikeMagnumandfederation.TheseexperiencesarealsosharedthroughOpenStackSpecialInterestGroups,withothersciencessuchastheSquareKilometerArray(SKA),publicpresentationssuchasKubeconEuropeandblogssuchastheOpenStackinProduction.
AtCERN,severalworkloadsrunwithincontainersprovisionedbyMagnum,theseinclude:
Reana/RecastThesetoolsprovideaframeworkforexecutingreusableworkflowsinHighEnergyPhysics.Containersoffertheabilitytopackagetheanalysissoftwareanddatainasingle,easilyshareableunitaswellaseasyscalingoutbothon-premisesandusingexternalresources.WorkisscheduledasKubernetesjobsbasedonYadageWorkflowssupportinganalysisanddatapreservationactivities.
SparkasaServiceRecently,KuberneteswasaddedasaresourcemanagerforSpark.SparkcanspawndriversandexecutorsaspodsandKubernetesisresponsiblefortheschedulingandlifecycle.AteamintheCERNITdepartmentisdevelopingaservicewhereuserscancreateKubernetesclustersondemandwithOpenStackMagnumanddeploySparkonKubernetes,providingalltherequiredintegrationswithCERN’sspecializedfilesystemsanddatasourcesinasecureway.UserswithafewcommandscaneffectivelycreateaSparkdeploymentwiththedesiredsize,onlyforthetimetheyneeditandwiththeoptiontoscaleupordowntheirdeploymentwhilerunning.
LHCexperimentdetectortriggersimulationforLHCupgradeTheLHCisduetobeupgradedtohigherluminosityduringthe2020swhichrequiressignificantenhancementsintheexperimenttriggerfarmswhichfilterthecollisions.LargescaleKubernetesclustershavebeencreatedtosimulatethedifferentapproachesfortheATLASexperimentandvalidatethedesign,resultinginsomefinetuningofKubernetesandOpenStackcomponents.
GitlabContinuousIntegrationRunnersGitlabenablesuserstobuildCI/CDjobsandexecutethemonsharedorprojectspecificrunners.CERNuserscanleveragetheCERNContainerServicetotestandbuildsoftware,buildandpublishcontainerimagesanddocumentationorsetcomplexpipelinesmanagingthefullapplicationlifecycle,includingautomateddeploymentsintodifferentenvironments.
FederatedKubernetescomputefarmswithexternalcloudsCERNusesfederationsofKubernetesclusterstosupportmulti-cloudoperations.Multipleclusterscanbeseamlesslyintegratedacrosscloudsofvaryingtechnologies,includingAWS,GCEandOpenStackcloudssuchasCERNandtheT-SystemsOpenTelekomCloudasdemonstratedatKubecon2018.
Integratingvirtualmachines,containerenginesandbare-metalunderasingleframeworkprovidesforeasyviewsonusageaccounting,ownershipandquota.ManilastoragedriversforKubernetesallowtransparentprovisioningoffileshares.ThissupportsboththeITdepartmentincapacityplanningandtheexperimentresourcecoordinatorsindefiningtheprioritiesfortheirworkinggroups.Resourcemanagementpoliciessuchasreassignmentorexpiryofresourcesondepartureofstaffarehandledinconsistentworkflows.
SKTelecom
SKTelecom(SKT),SouthKorea’slargesttelecommunicationsoperator,hasbeenexploringoptimizedapproachesfordeployingOpenStackonKuberneteswiththeaimofputtingcorebusinessfunctionsoncontainerizedOpenStackbytheendof2018.SKTleveragesKollaandOpenstack-Helm.withdeploymentsautomatedbyKubespray.SKTdevotesnearly100%ofit’sdevelopmenteffortstoOpenStack-Helm,andworkscloselywithAT&TtomakeOpenStack-Helmsuccessful.
SKThasalsoincorporatedothertoolsintotheirOpenStackonKubernetesefforts.Forlogging,monitoring,andalarms,theyareusingPrometheusandElasticsearch,Fluent-bit,andKibana,allofwhicharedefaultreferencetoolsintheOpenStack-Helmcommunity.SKTcombinesalloftheseintoasingleclosed-integratedsolutioncalledTACO:SKTAllContainerOpenStack.
SKTspecificallyemphasizesanautomatedcontinuousintegration/continuousdelivery(CI/CD)pipelinearoundcontainerizedOpenstackonKubernetes.SKT’sCIsystemconsistsofJenkins,Rally,Tempest,DockerRegistry,aswellasJiraandBitbucket.SKTalsodevelopedanopensourcetoolcalledCookiemonster,achaos-monkeylikeresiliencytesttoolforKubernetesdeploymentthatperformsresiliencytestsfortheirCIpipeline.
Witheverychange,SKTautomaticallybuildsandtestsboththeOpenStackcontainersandHelmcharts.Daily,theyautomaticallyinstallahighlyavailableOpenStackdeploymentwiththreecontrolnodesandtwocompute-nodes,run400testcasesfromTempestagainstittovalidatetheservices,andfinallyrunresiliencytestingwithCookiemonsterandRally.ThecompleteCIsystemisillustratedinthefollowingdiagram:
SKTautomatesitsdeploymentswithArmada,asub-projectofAirship,whichwasintroducedinthecommunityasanewopeninfrastructureprojectbyAT&T.SKTiscollaboratingincommunitytoprovideenhancementstotheprojectbasedontheirproductionuses.
Inpracticaluse,SKThasalreadyseenalargenumberofbenefitsfromdeployingOpenStackonKubernetesincluding:
SimpleandEasyInstallations.ClusterAuto-Healing.AnabilitytoupgradeandupdateOpenStackwithminimalimpacttorunningservices.Rapidadoptionofadvancedreleasemethodologies,includingblue-greendeployment,canaryreleases.CompleteautomatedmanagementofPythondependenciesthroughcontainerisolation.Securesecretandconfigurationmanagement.Fastandflexibleroll-outsofclusterupdates.
SKTisstilltestingtheapproach,butisactivelymovingtowardsrunningtheirOpenStack-Helmdeploymentsinproduction.Byendofthisyear,SKTwillhaveatleastthreeproductionclusters,withthefourthandlargestcomingonlinein2019.Theseusecasesinclude:
BigDataplatform(plannedtogoliveQ42018)Avirtualdesktopinfrastructureplatform(productionreadybyQ42018)AGeneralpurposeInternalPrivateCloud(plannedtogoliveQ32018)Atelconetworkinfrastructurebuiltonvirtualnetworkfunctions(plannedtoopensometimein2019)
SKTisalsotryingtoimproveautomationontelecominfrastructureoperationbyutilizingcontainerizedVNFsandleveragingcontainers’autohealingandfastscale-outfeatures.InordertoallowinteractionbetweenvirtualmachinebasedVNFsandcontainerizedVNFs,SimplifiedOverlayNetworkArchitecture(SONA),whichisavirtualnetworksolutionforOpenStack,willsupportcommunicationbetweenVMsandcontainers.SONAusestheKuryrprojectforintegrationofOpenStackandKubernetes,anditoptimizesnetworkperformanceusingsoftwaredefinednetworkingtechnologies.
Overall,SKTisfindingthatKuberneteshelpssolvemanyofthecomplexitiesofdeployingandoperatingOpenStack.SimplifyingOpenStackgivesthemapowerfulapproachtodeliveradvancedinfrastructureinnovationforthe5Gera.
FocusingeffortsonOpenstackonKubernetesdramaticallyincreasedtheirinternalcapabilitytodealwiththeevolvingshifttowardmicroservicesincontainersandbecomeacriticalinfrastructurefordeliveringArtificialIntelligence,InternetofThings,andMachineLearning.
Superfluidity
TheSuperfluidityprojectismadeupof18partnersfrom12Europeancountries.Itaimstoenhancetheabilitytoinstantiateserviceson-the-fly,runthemanywhereinthenetwork(core,aggregation,edge)andshiftthemtransparentlytodifferentlocations.SUPERFLUIDITYisaEuropeanResearchproject(Horizon2020)tryingtobuildthebasicinfrastructureblocksfor5Gnetworksbyleveragingandextendingwellknownopensourceprojects.SUPERFLUIDITYwillprovideaconvergedcloud-based5Gconceptthatwillenableinnovativeusecasesinthemobileedge,empowernewbusinessmodels,andreduceinvestmentandoperationalcosts.
Topursuethesegoals,theprojectconsortiumisshiftingawayfromlegacy,VM-basedapplicationstoCloudNativecontainerizedapplications.KuryrservesasabridgebetweenOpenStackvirtualmachines,andKubernetesandOpenShiftcontainerizedservices.
TheprojectmakesuseofManageIQasacentralnetworksfunctionvirtualizationorchestrator(NFVO),AnsibleforApplicationdeploymentandlifecyclemanagement,OpenStackservicesincludingHeat,Neutron,andOctavia,andKubernetesthroughOpenShiftforVMsandcontainersintegration.
ByleveragingAnsibleplaybooksexecutedfromtheManageIQappliance,SUPERFLUIDITYoffersacommonwaytodeployapplications.TheseapplicationsinturnusethecloudorchestrationfunctionalityprovidedbyOpenStackHeattemplatesandOpenShifttemplates.
Theconsortiumdeploys5Gcloudradioaccessnetworks(CRAN)andmobileedgecomputing(MEC)componentswithincontainers.Italsodeployshighthroughputapplicationslikevideostreamingontopofthedistributedinfrastructure.
ShiftingtowardacloudnativeapproachtoapplicationdeliveryallowsforrapidandresilientSUPERFLUIDITYinstallations.ItenablesasmoothtransitionfromVM-basedapplicationsandcomponentstocontainers,whileretainingtheversatilitytoenableVMsforsomespecificapplications.Examplesoftheseapplicationsarespecialsecurityprotectionsornetworkaccelerationrequiredbysingle-routeinput/outputvirtualization(SRIOV).
Inscaleperformancetesting,SUPERFLUIDITYwasabletolaunchapproximately1000podsatarateof22pods/second(withtimemeasuredfromcreationtorunning).ThisremarkableperformancewasachievedbyrunningOpenShiftonVMsmanagedbyOpenStack,withKuryractingasapodnetworkdrivertoavoiddouble-encapsulationperformancehits.
IV.Conclusion
Overthepastfewyears,ascontainershavebecomeanimportanttoolfordevelopersandorganizationsalike,OpenStackhasleverageditsmodulardesignandexpansivecommunitytointegratecontainertechnologiesatmanylevels.ThiscanbeseenbothbythevariousorganizationsbringingcontainersandOpenStackintoproduction,andthenumberofprojectsthatworkalongsidecontainerstodelivernewcapabilities.TheOpenStackFoundationiscommittedtoensuringthatemergingtechnologiescanbeincorporatedandutilizedwithinOpenStack,andcontainersareanimportantexampleofthatcommitment.
Tolearnmore,visittheContainersLandingPage,whereyoucanfindacopyofthisdocumentaswellaslinkstodozensofvideosfocusedontheintegrationsofOpenStackandcontainers.KubernetesSIG-OpenStackhasaSlackchannel,mailinglist,andweeklymeetingifyouengagedirectlywiththecommunitythat’sbuildingKubernetesandOpenStack
integrations.
V.OpenSourceProjectIndex
Airship
Airshipisacollectionofinteroperableandlooselycoupledopensourcetoolsthatprovideautomatedcloudprovisioningandmanagementinadeclarativeway,basedaroundKubernetesasanapplicationplatform.
Ansible
AnsibleisacommonlyusedorchestrationtoolusedtodeployandmanageOpenStackinstallations.
Cinder
OpenStackCinderoffersblockstorageasaservice,providingasingleAPIbackedbyoverseventydifferentpossiblestoragedrivers.
CloudProviderOpenStack
CloudProviderOpenStackistheimplementationoftheKubernetesCloudProviderinterface.ItallowsanOpenStack-hostedKubernetesclustertodirectlyaccessstorageandloadbalancerresourcesintheOpenStackcloud.
Calico
CalicoisanetworkoverlaywithdriversforbothKubernetesandOpenStackthatfeaturesL3-onlyrouting.
Cyborg
CyborgisanOpenStackprojectthatprovidesageneralmanagementframeworkforhardwareacceleratorsincludingFPGA,GPU,ASIC,andothers.Workisinprogresstosurfaceageneralhardwareinterfacetopods.
Docker
Dockerisanopensourcecontainervirtualizationframework,usedtohostcontainerizedapplications.
Helm
HelmistheofficialpackagemanagerforKubernetes.ApplicationdeploymentsaredescribedbyHelm-Charts,whichcanbeautomaticallydeployedandmanagedonaKubernetescluster.
Ironic
IronicistheOpenStackbare-metalservice.RunningeitherasastandaloneserviceorasadrivertoOpenStackNova,itcanmanagethecompletelife-cycleofbare-metalsystems,includingenrollment,provisioning,maintenance,anddecommissioning.
Loci
LOCIisanOpenStackprojecttobuildlightweight,OCIcompliantcontainersforOpenStackprojects.
LXC
LXCisalow-levelcontainervirtualizationinterfacethattakesadvantageofLinuxkernelnamespaceisolationandothertechnologiestocreateisolatedlinuxruntimes.
KataContainers
KataContainersisastandardimplementationoflightweightVirtualMachines(VMs)thatfeelandperformlikecontainers,butprovidetheworkloadisolationandsecurityadvantagesofVMs.
Keystone
KeystoneistheOpenStackIdentityservicethatprovidesmeansforauthenticatingandmanaginguseraccountsandroleinformationprimarilyfortheOpenStackcloudenvironment,butalsoasaplugintootherenvironments,includingKubernetes.
Kolla(Containers)
Kolla(Containers)isanOpenStackprojecttobuildcontainersforeachOpenStackservice.Itincludesasophisticatedbuildandtemplatingsystems,andiscapableofbuildingcontainersfrombothsourceandpackagesonavarietyofhost
operatingsystems.
KollaAnsible
KollaAnsibleisanOpenStackprojectthatusesAnsibletodeployandmaintainafullOpenStackinstallationusingKollacontainers.
Kubernetes
Kubernetesisacontainerorchestrationsystemthatdeliversrobustandhighly-availableapplicationsontopofcloud-infrastructure.
Kuryr
KuryrisanOpenStackprojectthatprovidesaNeutronnetworkoverlaytocontainerruntimes,includingDockerandKubernetes.Itaimstobethe“integrationbridge”forcontainerandVMnetworks.
Magnum
MagnumisanOpenStackprojectthatoffersmanagedcontainerplatformsasaservice,includingKubernetes,DockerSwarm,Mesos,andDC/OSplatforms.Itiscapableofcreatingtenantisolatedapplicationplatformsthroughasimpleuser-facingAPI.
Neutron
NeutronistheOpenStacksoftware-definednetworkingservice,offeringasingleAPItodeliverdynamicnetworkinfrastructurebackedbydozensofnetworkdrivers.
OpenStackAnsible
OpenStackAnsibleisaprojectforbuildingOpenStackservicesintoLXCcontainers,andfordeployingandmanagingOpenStackinstallationswithinthosecontainerizedservices.
OpenStackHelm
OpenStackHelmisanOpenStackprojectthatdeploysandmanagesthelifecycleofOpenStackandsupportinginfrastructureontopofKubernetes(egCephandMariaDB),deliveringproductionreadydeployments,forarangeofusecasesfromsmalledgedeploymentstolargecentraloffices.LeveragingtheHelmpackagemanagementsystem.OpenStackHelmhassupportforbothbaremetal(Ironic)andvirtual(Nova/KVM)workloadmanagement,andisimageagnosticsupportingbothLOCIandKollacontainers.
Qinling
QinlingisanOpenStackprojecttodeliverFunctionsasaService.Qinlingsupportsdifferentcontainerorchestrationplatforms,suchasKubernetesandDockerSwarm,aswellasdifferentfunctionpackagestoragebackendssuchaslocalfile-store,OpenStackSwift,andS3.
Triple-O
TripleOisaprojectaimedatinstalling,upgradingandoperatingOpenStackcloudsusingOpenStack’scloudservicesasthefoundation-buildingonNova,Ironic,Neutron,HeatandAnsibletoautomatecloudmanagement.
Zun
ZunistheOpenStackContainersservice.ItaimstoprovideanAPIserviceforrunningapplicationcontainerswithouttheneedtomanageserversorclusters.
VI.Authors
MembersoftheOpenStackSIG-KubernetesCommunity
JaesukAhn,SKTelecomChristianBerendt,BetacloudSolutionsGmbHAnneBertucio,OpenStackFoundationPeteBirley,AT&TChrisHoge,OpenStackFoundationLingxianKong,CatalystCloudHongbinLu,HuaweiDanielMellado,RedHat,Inc.AllisonPrice,OpenStackFoundationDavidRabel,B1SystemsGmbHSanghoShin,SKTelecom