Post on 13-Feb-2017
ContainerOrchestration:WhichConductor?
ContainerConEurope,Berlin,Oct2016
MikeBright, @mjbright
HaikelGuemar, @hguemar
MarioLoriedo, @mariolet
First...Alittlebitofhistory
Solet'sfirstlookatrecentcontainerhistory...
Unix
con
tain
ers,
Unik
erne
lsLi
nux
cont
aine
rs(L
XC),
Mes
osLX
C(C
loud
Foun
dry,
Dot
Clou
dPa
aS)
Dock
er
Cont
aine
rOrc
hest
ratio
nOp
tions
PaaS
ado
ptio
nso
fDoc
ker
Dock
er"S
war
mm
ode"
,OCI
D...
Mic
ro-O
Ses,
Swar
m,R
kt,L
XD
Dock
erb
uysU
nike
rnel
Sys
tem
s
...2009 2013 2014 2016...2015
@hguemar@mjbright@mariolet
History μ-OSesManyvendorsaredevelopingμ-OSes,smallOS(mainlyLinux-based)tobethebasisforcontainerenginehostswhethertheybebare-metalorvirtualhostmachines.
They'resmall,withfaststartup,usefewresourcesandhaveasmallattacksurfaceandoften"atomic"softwareupdates.
OS VendorCoreOS - (CoreOS)ProjectAtomic - (RedHat)RancherOS - (RancherLabs)Photon - (VMWare)NanoServerOS - (Microsoft)UbuntuSnappyCore - (Canonical)
...Unikernels
μ-Services
μ-services Frommonolithstoμ-services
Rememberwhenhighavailabilitymeantthis...?
Active StandbyAppApp
ServersrunningmonolithicapplicationsinActive-Standbymodes,as1+1,N+1,orN+Morsplitacross3tiers.
Scalingmeantto"scaleup"byaddingCPU,RAM,disk.Butthere'salimittothis...thenyouhaveto"scaleout"@hguemar@mjbright@mariolet
μ-services Frommonolithstoμ-services
Thencameμ-services..
Astheindustrymovedtovirtualizedmicro-servicesthisallowedtoobtaingreaterefficiencies(higherutilisationofresources)andtheredesignofapplicationsallowstoscaleoutandachievehighavailability.
Containersfacilitatethismove,allowingfasterscalingandevengreaterefficiencieswithlessredundancy(noOStoreproduce).
@hguemar@mjbright@mariolet
μ-services Howcontainershelp?
ContainersolutionssuchasDockergobeyondtheisolationcapabilitiesofLXCbyprovidingsimpletousetoolstoenablepackagingofappswiththeirdependenciesallowingportableapplicationsbetweensystems.
Containersarelightweight
Versionedimagescontainingalldependanciescanbeshared
Containersallowtousethesameapplicationbinariesondevelopment,testandproductionsystemswhetherthatbeonalaptop,serverorinthecloud.
It'sanobrainerfordevelopers,whocanbuildandsharetheirownimages
@hguemar@mjbright@mariolet
μ-services Frommonolithstoμ-services
But1000'sofnodesareunmanageable...aren'tthey?
Wecan'ttakecareofour
,
sowehavetotreatthemlike
that'scloudnative!
@hguemar@mjbright@mariolet
Soweneedcontainerorchestration
OrchestrationWhatwasContainerOrchestrationagain?
Architecture-Composition&StitchingWorkflows&Policiesto
Scalein/out(maybeautomatically)Placeworkloadsfor
loadbalancing,faulttolerance,resourceoptimization
Adapttofaults
....
....
Master
Worker Worker Worker
....App1 AppNDistrib
Cluster
Config
(quorum)
Master Master
WorkerTasks:
OrchestrationGettingto"DesiredState"Tomanage100's,1000's,10,000'sofnodesweneedtoexpress"desiredstate"ratherthan"dothis".
Imperative DeclarativeTellsystem Dothis desiredstate
"startanewnode" "3mysqlnodes".
Intelligence Operator OrchestrationEngine.
Flexibility Best Least.
Itisnotlongerfeasibleforanoperatorto
knowtheresourcesavailable(e.g.SSD/HDD,GPU,...)reacttofailure,knowwhentoscale...
@hguemar@mjbright@mariolet
Choiceisgreat-whenyouknowwhatyouwant...
Orchestration
TheBig3-MainOrchestrationChoices
DockerSwarm("SwarmMode")
ApacheMesos
Kubernetes
Orchestration
TheBig3-MainOrchestrationChoices
DockerSwarm("SwarmMode")
ApacheMesos
Kubernetes
...moreChoices...
Rancher(RancherLabs)
Fleet(CoreOS)
Nomad(HashiCorp)
Kontena
OpenStackMagnum@hguemar@mjbright@mariolet
OrchestrationTheBig3-WhatdoesGoogleTrendssay?
ClearlyKuberneteshasaleadinGoogle"searchtrends"
Butwecanexpect"DockerSwarm"tomakequickprogressthankstothenew"swarmmode"
DockerSwarm
DockerSwarm
Dec2014 ...DockerSwarmisannounced
OrchestrationusingDockerCompose
Jun2016 ... SwarmToolkitreleasedOpenSourceOrchestrationToolkit
Jun2016 ... SwarmModeannouncedOrchestrationintegratedintoDockerEngine
Docker1.12isthefirstreleasetointegrate"SwarmMode"TheoriginalDockerSwarmismaintainedforlegacyuse.
"SwarmMode"isarevolutionbringing:
OrchestrationdirectlyintheDockerEngineAdvancednetworkingfeatures
meshnetwork,vxlanLoadbalancingServiceDiscoveryDockertraditionaleaseofuse
docker.com @hguemar@mjbright@mariolet
DockerSwarm
Architecture
....
....
Master
Worker Worker Worker
....App1 AppN DistribClusterConfig(quorum)
Master Master
WorkerTasks:
GossipNetwork
DockerEngine DockerEngine DockerEngine DockerEngine
gRPC
docker.com @hguemar@mjbright@mariolet
DockerSwarm
UsingDocker"SwarmMode"Createanewswarmbycreatingthemasternode:
$dockerswarminit--advertise-addr192.168.2.100Swarminitialized:currentnode(dxn1zf6l61qsb1josjja83ngz)isnowamanager.
JoinanewWorkernodetotheswarm:
$dockerswarmjoin--tokenTOKEN192.168.2.100:2377
JoinanewMasternodetotheswarm:
$dockerswarmjoin-tokenmanager
docker.com @hguemar@mjbright@mariolet
DockerSwarmDemo
DockerSwarm
DockerSwarmDemoCreationofa3nodecluster
Runaserviceontheclusterandscaleitto3replicas
Makearollingupdateoftheservice
Drainanodeofthecluster
docker.com @hguemar@mjbright@mariolet
DockerSwarm
Gettingstarted
BrunoCornecs'Docker101LabNewtoDocker?Cometohttp://sched.co/7oHf
Docker101Lab,9am-Friday7thOctober
https://github.com/bcornec/Labs/tree/master/Docker
Followedby..
JeromePetazzoni'sOrchestrationWorkshopormoreadvanced?Cometohttp://sched.co/7oHx
OrchestratingContainersinProductionatScalewithDockerSwarm,Friday7thOctober
https://github.com/jpetazzo/orchestration-workshop
docker.com @hguemar@mjbright@mariolet
Kubernetes
FromtheGreek:"Steersman,helmsman,sailingmaster"
Kubernetes
Googlecreatedbasedonextensiveexperiencerunningcontainersinternally~billionsofcontainersayear
StartedOct2014,reachedv1.0inJuly2015,nowatv1.4
ManagedbytheCloudNativeComputingFoundationhttps://cncf.io/
CommercialofferingsfromCoreOS(Tectonic)andCanonical
Integratedin:
GKE(GoogleContainerEngine)OpenStackaboveKubernetes
Stackanetes(CoreOS,usesTectonic)SelfhealingOpenStackdemo
Mirantis(OpenStackCI/CDbasedonKolla)VariousPaaS:
RedHatOpenShiftCPHPEStackatov.40Deis
kubernetes.io @hguemar@mjbright@mariolet
Kubernetes
Architecture
....
....
Master
Worker Worker Worker
....App1 AppN etcd
Master Master
WorkerPods: 192.168.100.10
ServicesLabels,Selectors
kubernetes.io @hguemar@mjbright@mariolet
ApacheMesos
ApacheMesos
Themostprovenorchestratortoday,existssince2009.
Canscaleto~10,000nodes.
Usedinproductionby:
UberTwitterPaypalHubspotAirbnbeBayGrouponNetflix
SupportsContainerizerstoisolatetasks
mesos.apache.org
ApacheMesos
Architecture
....
....
Master
Worker(agent)
....Spark AppNDistrib
Cluster
Config
(quorum)
Master Master
WorkerTasks:
ResourcesAvailable
Scheduler
SparkExecutor SparkExecutor Executor
Offer
AcceptRegister
Worker(agent) Worker(agent)
mesos.apache.org
ApacheMesos
MesosisusedinconjunctionwithFrameworkssuchas
Forlongrunningtasks:
Marathon(Mesosphere),AuroraorSingularity
Forjoborchestration:
Chronos"cron",Jenkins
ForBigDataProcessing:
Hadoop,Spark,StormCassandra,ElasticSearch,...
mesos.apache.org@hguemar@mjbright@mariolet
Soisn'tittimewetoldyouwhattochoose?
...let'sjustcomparethem...
What'scommon
DockerSwarmandKubernetesarecreatingrichOrchestrationstackswithintegratedruntimes.
They'removingincrediblyquicky...
Theyareaddingfeaturessuchasnetworkingcapabilities,loadbalancing,services,labels.
Theyhaveamore'declarative'approach
Theysupportorarelookingtosupportdifferentruntimeengines(*)
@hguemar@mjbright@mariolet
Whatadvantages?Docker"Swarm
Mode"Simpletouse(despiteunderlyingcomplexity)
All-in-onecontainerengineplusorchestration
UsesDockerAPIandfamiliardockercommands
Advancednetworking
-meshnetworking-LoadBalancingandServiceDiscovery
Replication
@hguemar@mjbright@mariolet
Whatadvantages?Kubernetes
Richconceptualmodel
Podsasgroupingsofcontainers
LabelsandSelectors(forallcomponents)
Largeecosystem
Networking
-LoadBalancingandServiceDiscovery
Replication
@hguemar@mjbright@mariolet
Whatadvantages?ApacheMesos
Mostmature
Battletestedbymanyserviceproviders
Scalesto10,000nodes
DataCenterOS-appearsas1resource
Notjustcontainers
Manyframeworksavailable
Difficultramp-up
@hguemar@mjbright@mariolet
Handson...
@hguemar@mjbright@mariolet
Hands-on Comealong
Thisafternoon'stutorialsessionledbyMario:Tuesday,October4-15:30-16:20
5Containersfor5Languages:PatternsforSoftwareDevelopmentUsingContainers-MarioLoriedo,RedHat
Tomorrow'slabsessionledbyHaikel:Wednesday,October5-11:00-12:50
ContainerOrchestrationLab:Swarm,Mesos,Kubernetes-HaïkelGuémar,FedoraProject
Labsetupinstructionshere
DockerSwarmKubernetesApacheMesos
@hguemar@mjbright@mariolet
Questions?Thankyou
@hguemar@mjbright@mariolet
Resources
@hguemar@mjbright@mariolet
ResourcesBooksPublisher Title AuthorOReilly DockerCookbook SébastienGoasguen
OReilly DockerUp&Running
KarlMatthias,SeanP.Kane
OReilly UsingDocker AdrianMouat
OReilly[EarlyAccess]KubernetesUp&Running
KelseyHightower
Manning [MEAP]CoreOSinAction MattBailey
Manning [MEAP]KubernetesinAction MarkoLukša
@hguemar@mjbright@mariolet
ResourcesArticles/OrganismsCloudNativeComputingFoundation-Kubernetes,Prometheushttps://cncf.io/
"KubernetestheHardWay,KelseyHightower"-https://github.com/kelseyhightower/kubernetes-the-hard-way
"KubernetesUserGuide,Walkthrought"-http://kubernetes.io/docs/user-guide/walkthrough/
@hguemar@mjbright@mariolet
ResourcesVideosJune2016-ContainerOrchestrationWars,KarlIsenberg,Mesosphere
Mar2016-ContainerOrchestrationwithKubernetes,DockerSwarm&Mesos-Marathon-AdrianMouat,ContainerSolutions
Jan2016-Docker,Kubernetes,andMesos:Compared.,,AdrianOtto,SouthernCaliforniaLinuxExpo
Repos
@hguemar@mjbright@mariolet
Kubernetes
DocumentationGettingstartedguides
CreatingaKubernetesClusterportKubernetestoanewenvironment
inGettingStartedfromScratchUserdocumentation
torunprogramsonanexistingKubernetescluster
KubernetesUserGuide:ManagingApplications
theKubectlCommandLineInterfaceisadetailedreferenceonthekubectlCLIUserFAQ
kubernetes.io @hguemar@mjbright@mariolet
Kubernetes
Documentation-2
ClusteradministratordocumentationforpeoplewhowanttocreateaKubernetesclusterandadministeritintheKubernetesClusterAdminGuide
DeveloperandAPIdocumentationtowriteprogramsusingtheKubernetesAPI,writepluginsorextensions,ormodifycorecodeKubernetesDeveloperGuidenotesontheAPIAPIobjectdocumentation,adetaileddescriptionofallfieldsfoundinthecoreAPIobjects
Walkthroughsandexampleshands-onintroductionandexampleconfigfilesintheuserguideinthedocs/examplesdirectory
ContributionsfromtheKubernetescommunity
inthedocs/contribdirectory
kubernetes.io @hguemar@mjbright@mariolet
Kubernetes
Documentation3Designdocumentationanddesignproposals
tounderstandthedesignofKubernetes,andfeatureproposalsKubernetesDesignOverviewandthedocs/designdirectorydocs/proposalsdirectory
Wiki/FAQthewikitroubleshootingguide
Community,discussion,contribution,andsupport
ConsiderjoiningtheCloudNativeComputingFoundation.Fordetailsaboutwho'sinvolvedandhowKubernetesplaysarole,readtheirannouncement.
kubernetes.io @hguemar@mjbright@mariolet