Reconstructing the SRE

ReconstructingtheSRE

BobWiseCTO

CloudNativeComputingTeam

Copyright © 2017 Samsung SDS Co., Ltd. All rights reserved

This presentation is intended to provide information concerning Samsung’s efforts around containers and container orchestration. We do our best to make sure that information presented is accurate and fully up-to-date. However, the presentation may be subject to technical inaccuracies, information that is not up-to-date or typographical errors. As a consequence, Samsung does not in any way guarantee the accuracy or completeness of information provided on this presentation. Samsung reserves the right to make improvements, corrections and/or changes to this presentation at any time.

The information in this presentation or accompanying oral statements may include forward-looking statements. These forward-looking statements include all matters that are not historical facts, statements regarding the Samsung Data System' intentions, beliefs or current expectations concerning, among other things, market prospects, growth, strategies, and the industry in which Samsung operates. By their nature, forward-looking statements involve risks and uncertainties, because they relate to events and depend on circumstances that may or may not occur in the future. Samsung cautions you that forward looking statements are not guarantees of future performance and that the actual developments of Samsung, the market, or industry in which Samsung operates may differ materially from those made or suggested by the forward-looking statements contained in this presentation or in the accompanying oral statements. In addition, even if the information contained herein or the oral statements are shown to be accurate, those developments may not be indicative developments in future periods.

Logos remain the property of their respective owners.

2

Copyright © 2017 Samsung SDS Co., Ltd. All rights reserved

This presentation is intended to provide information concerning Samsung’s efforts around containers and container orchestration. We do our best to make sure that information presented is accurate and fully up-to-date. However, the presentation may be subject to technical inaccuracies, information that is not up-to-date or typographical errors. As a consequence, Samsung does not in any way guarantee the accuracy or completeness of information provided on this presentation. Samsung reserves the right to make improvements, corrections and/or changes to this presentation at any time.

The information in this presentation or accompanying oral statements may include forward-looking statements. These forward-looking statements include all matters that are not historical facts, statements regarding the Samsung Data System' intentions, beliefs or current expectations concerning, among other things, market prospects, growth, strategies, and the industry in which Samsung operates. By their nature, forward-looking statements involve risks and uncertainties, because they relate to events and depend on circumstances that may or may not occur in the future. Samsung cautions you that forward looking statements are not guarantees of future performance and that the actual developments of Samsung, the market, or industry in which Samsung operates may differ materially from those made or suggested by the forward-looking statements contained in this presentation or in the accompanying oral statements. In addition, even if the information contained herein or the oral statements are shown to be accurate, those developments may not be indicative developments in future periods.

Logos remain the property of their respective owners. So there.

3

ReleasetheKraken.

SDS- CloudNativeComputingTeam

• Our#1jobisimprovingorganizationalvelocity– DeliveringthebusinessvalueofKubernetestoyou,fastest

• RocksolidKubernetesclusterdesignanddeployment,specifictoyou• Optimizeddeploymentpipelinesandcontainerstrategy• 24x7x365Kubernetesoperationssoyoucanfocusonyourbusiness• Organizationalconsultingtorapidlyadapt

• Weare:– IndustryleadersinOperationsAutomation,ClusterOperations,andKubernetesAdoption

– ContributorsandleadersontheKubernetesprojectfor2+years– MaintainersofKraken:production-gradeclustermanagement– DeliveringthisforEnterprisecustomersglobally

Copyright©2017SamsungSDSCo.,Ltd.Allrightsreserved4

MacroTrends

1. MacroTrend:Massiveshiftsinallindustriestowardssophisticatedandcomprehensiveautomationtoenablecompetitiveadvantage

2. Inalltechnologydelivery“CloudNative”architecturesandautomationprevail

3. Outsourcingistheonlypathascompaniescannotadaptandacquireexpertiseinternally

Copyright © 2017 Samsung SDS Co., Ltd. All rights reserved 5

RootCauses:DeepOrganizationalPain• Toolsandprocessesbuiltfortheeraofbaremetalandvirtualmachines

• Devteamsunderpressuretodelivernewfeaturesquickly,notgoingfastenough

• Operationsteamstryingtosupporthastilydeployedfeatures– Qualityissues– Outages– Constantfirefighting– Unhappycustomers– Employeeretentionissues

• Operationsviewedas“justtryingtoblockthings”• “Devops”hascometomeangeneralistswhoareoverwhelmedtryingtohandleeverything

Copyright © 2017 Samsung SDS Co., Ltd. All rights reserved6

TheCloudNativeDisruption

7


CloudNativeHasDramaticallyRaisedtheBar

ClassicEnterpriseITisEspeciallyBehind


Fromthehighlyrecommended:https://www.slideshare.net/adriancockcroft/dockercon-state-of-the-art-in-microservices

CloudNativeMarketRealities

Enormousdifferencesbetweencompaniesintheabilitytoexecuteonsoftwareproductdeliveryhaveemergedonallthreeaxes:

• Velocity• Quality• Efficiency

Copyright©2017 SamsungSDSCo.,Ltd.Allrightsreserved2

CloudNativeMarketRealities

Thecompanieswinninginallthreeofthesecategoriesshareonthingincommon:

• Velocity• Quality• Efficiency


Theme:UsingCloudNativeApproaches

(a) Containerized. Applicationsdeployedinunitsthatcanbeeasilymanagedanddealtwithbyeveryone:developers,productmanagers,andoperations.

(b) Dynamicallymanaged. Automaticallyandresponsivelydeployedbyanorchestrationenginethatconsiderscustomerexperienceandcost.

(c) Micro-servicesoriented. Looselycoupledandrapidlyadaptableservicesthatcanbeinnovatedanddeployedseparatelywhilethesystemasawholecontinuestooperate.


WhatisCloudNative?

CloudNativeOrganizationsAreHighPerformanceBasedOn:

HowTheyBehaveHowTheyMeasure


HighPerformanceOrgsEmbraceChange

14

HighPerformanceOrgsEmbraceRapidChangeWithAutomation

15


PuppetLabs2017 StateofDevops ReportHighlyRecommendedReading

https://puppet.com/resources/whitepaper/2017-state-of-devops-report

MonolithicComplexDependenciesLarge,CentrallyControlledTeamsMonthstoProductionWaterfallProcessManualQAManualSecurityAudits

LotsofSimplePartsIndependentPartsSmall,IndependentTeamsSpecialistsnotGeneralistsSREPatternContinuousIntegrationContinuousDeploymentAutomatedQAUbiquitousAutomationAutomatedSecurity

ChangeResistant ChangeEmbracing

EmbraceTheChange


LotsofSimplePartsIndependentPartsSmall,IndependentTeamsContinuousIntegrationContinuousDeploymentAutomatedQAUbiquitousAutomationAutomatedSecurity


KeyInsight

OptimizedforMTBF OptimizedforMTTR


LotsofSimplePartsIndependentPartsSmall,IndependentTeamsContinuousIntegrationContinuousDeploymentAutomatedQAUbiquitousAutomationAutomatedSecurity


KeyInsight

MeasuredbyMTBF MeasuredbyMTTR

MeanTimeBetweenFailuredrivesfailurepreventionandriskaversionMeanTimeToRepairdrivesresponsivenessandallowsrisks


LotsofSimplePartsIndependentPartsSmall,IndependentTeamsSpecialistsnotGeneralistsSREPatternContinuousIntegrationContinuousDeploymentAutomatedQAUbiquitousAutomationAutomatedSecurity


EmbraceTheChange

LotsofSimplePartsIndependentParts

Small,IndependentTeamsSpecialists,notGeneralistsAdoptionoftheSREPattern

ContinuousIntegrationContinuousDeploymentAutomatedQAUbiquitousAutomationAutomatedSecurity

Architecture(Microservices)

Org

ExecutionFundamentals

LotsofSimplePartsIndependentParts

Small,IndependentTeamsSpecialists,notGeneralistsAdoptionoftheSREPattern

ContinuousIntegrationContinuousDeploymentAutomatedQAUbiquitousAutomationAutomatedSecurity

Architecture(Microservices)

Org

ExecutionFundamentals

CloudNative

Devops Positives• Breakingdownsilos• Nomore“throwingitoverthewall”• Orientationtoinfrastructureascode• Orientationtoautomation• ValuingcodingskillsinoperationsDevops fails:

– Companyrenamesopstodevops.– Improvementfail


Devops Anti-PatternsinLargerOrgs• Renameopstodevops.• Disbandops,“wejustmakethedevs doit”• Addadevops groupwithyetanothersilobetweendevandops• Expecteveryonetoknoweverything

– “Weonlyhavefullstackdevops”– Thisisunrealistic

Instead,wewantapatternforspecializationwithcollaboration….


TheSRE(SiteReliabilityEngineer)• Helpsproductteamsengineerforoperability

– Architecture– Tooling

• Ownscapacityplanning• EnsuresCItoolingisworkingandadequate• Helpsdevelopmentgofasterwithpositiveassistance• Spendsalotoftimeondevelopingtooling,typicallymonitoring,CI,andoperationalanalytics

• EnsureproductmanagementsetsSLOs(servicelevelobjectives)anderrorbudgets– Tracksandenforces


BenTraynor – SREVPatGoogle“OutsideGoogle,weoftenobservethatthereisn'tparityofesteembetweentheSWEandoperationsteams,whichcombinespoorlywiththefactthattheyoftenhavedifferentincentives.That'showweendupwiththemodelthatexistsintheindustrytoday,whereSWEteamswritesomethingandthrowitoverawalltotheoperationsteams,whothentrytomakeitwork,andcan't,andthrowitback,andsoon.”



Dev

Ops

QACI

CD Monitoring

SRE

EvolvingfromDevops:ClusterOps


InfraOps

ClusterOps

Dev

Cluster

Infra

AppOpsApp App

ProductMgt

InfraOpsInfraInfra

ClusterOpsandtheSRERole


InfraOps

ClusterOps

Dev

Cluster

Infra

AppOpsApp App

ProductMgt

InfraOpsInfraInfra

SRE

TechnologyProgressiontoPlatforms…DeconstructionNeeded


Impacttothedevelopment• OperationsmovingupthestackwhileDevelopmentismovingdownthestack• StillalotcomplexityandseparationbetweenOperations/Developmentteams• Outsourcewheneconomiesofscalemakeitmorebeneficialtopushtoa3rd party

BareMetal

OS

OS/VirtualMachines

Application

Hypervisor

OS/VirtualMachines

Application

Dependencies Dependencies

BareMetal

OS

Dependencies

Application

InfrastructureInfrastructure

IAAS

Application Application


PAAS

VM’s IAAS/PAASCo-Lo

BareMetal

OS

Dependencies

Application

Infrastructure

DIY

OS/VirtualMachines

OS/VirtualMachines

3rd Party

Operations

Development

AreaofResponsibility

OrganizationalTrendtoOutsourcing…DeconstructionNeeded


ImprovementstoDevelopmentProcess.• Fasterreleasecyclesmeansquickertomarket.• Smaller,morereliabledeployments.• Quickerrecovery,reduceriskwhenissuesoccur.Shiftinfocusto

MTTRvsMTTF.• Economiesofscalearegettingtothebuildvsbuydecision

InfrastructureOperations

ClusterOperations

AppOperations


ClusterOperations

AppOperations


ClusterOperations

AppOperations

IAAS

Application Application


PAAS

IAAS/PAAS

OS/VirtualMachines

OS/VirtualMachines

FutureState

KubernetesContainersPipelineAlerting,MonitoringPerformance/EfficiencyMaintenanceReporting

SpeedtoMarket

ImprovementstoOrganizationalVelocity.• SubjectmatterexpertsonDay1• Noneedtospinupanewoperationsteam• AllowdevelopmentteamstofocusonProductvsTooling• Morecostefficientthandoingitin-house

MultipleFactors• TechnologyprogressiontoPlatforms• OrganizationalTrendtoOutsourcing

– SomeSREfunctionswillbeoutsideyourorg• Bestpractices:

– SpecialistsoverGeneralists– CollaborationoverSilos


ReconstructingtheSRE


SRE

AREApplication

CRECluster

IREInfra

AREConcerns• ApplicationSLI/SLO• Availabilitytargets• Podautoscaling• Capacityplanning• Deployments/Canaries• ApplicationCI/CD• Apppackagingpractices• AppSLIcapture• Applicationarchitecture• Applicationmonitoring• Applicationlogging

SLO– ServiceLevelObjectiveSLI– ServiceLevelIndicator

CRE/IREConcerns• ClusterSLI/SLO• ClusterCapacity• Nodeautoscaling• Controlplanedesign• Clusterupgrades• Clusterutilization• Under/overcapacity

nodes• Nodespecs• Dockerversion• Network

configuration• Namespacedesign• Failuredomains• Clustermonitoring• Clusterlogging

KubernetesFactors


KubernetesConceptual


Node(Server)

DockerDaemon

KubernetesNodeController

Node(Server)

DockerDaemon


Node(Server)

DockerDaemon


KubernetesControlPlane

Status:Cluster isreadyforwork




ToAPI: Run (1)ofcontainerX

Node(Server)

DockerDaemon


Node(Server)

DockerDaemon


Node(Server)

DockerDaemon


1ofX

ReplicaSet (replicationset)Managespodcount

Node(Server)

DockerDaemon


Node(Server)

DockerDaemon


Node(Server)

DockerDaemon




Status:ClusterisrunningX

Pod(X)

ContainerContainerContainer


1ofX


OrchestratingMultipleApplications



Node(Server)

DockerDaemon


Node(Server)

DockerDaemon


Node(Server)

DockerDaemon


Pod


Pod


Pod


Pod


Pod


Pod


Pod


Pod


Pod


App1

App2

App3

App4

App5

KubernetesNamespace(Environments)


Service

Pod


Pod




main

Pod


Pod




main

Pod


Pod


main

Pod


Pod




main

Pod


Pod


main

Pod


Pod


CLASH



main

Pod


Pod


prod



main

Pod


Pod


prod

main

Pod


Pod


dev

KubernetesNamespace• Virtualcluster• Fundamentaladminunit,mapstodifferentorganizationalpatterns:– Resourcenamescoping,i.e.

• dev,qa,orprod– Teamsorprojects

• Group1,group2– Services

• WhatQuotasareattachedto,andfuturepermission-relatedconcepts– ReflectionofGoogleoperationalphilosophy– Micro/ACLsgettoounwieldy


KubernetesNamespaces:KeytotheCRE/AREContract• Namespacesareformalproductionentities,managedbyCRE• Devpipelinesthatcreate/modifynamespacesgetCREoversight• RBACconfigurationofnamespacesisacriticalrole• CREownsnamespacequotas• AREhasbroadpermissionsinsidethenamespace• CREownsthepodschedulingrules/contraints(taints/tolerations)

• Reportingbrokenoutbynamespace


ManagedClusterOps Services• OurgoalistoenableCustomerstofocusentirelyontheircorebusiness.• Fastestpathtoproductionclusters• Continuousmonitoringandalerting• Performanceandefficiencyanalysis


• ClusterReadinessEvaluation• StrategicBusinessAnalysis• POCDelivery• DevelopmentPipelinePlanning• DedicatedOn-boardingTeam• ManagedClusterOperations• Otherservicestoacceleratebenefitsto

yourbusiness.

ProfessionalAccelerationServices:Ourgoalistoacceleratebenefitstoyourbusinessfromcontainers,cloud-native,andfasterdeployments:

HardwareOperations

ClusterOperations

AppOperations

Quickestandandmostcosteffectivepath

SDS– CNCTCommercialOffering

ContactandProjectInfo• BobWise

– @countspongebob ontwitterandgithub– [email protected]

• CNCThomepage:– http://samsung-cnct.github.io/

• Krakenrepo– ProductiongradeclustermanagementforKubernetes– opensource,Apachelicense– https://github.com/samsung-cnct/kraken– #krakenchannelonkubernetes slack


Q&A

50

Reconstructing the SRE

Software

Transcript of Reconstructing the SRE