SACON

SACONInternational2017

NilanjanDeFireCompass,Inc.

nde@firecompass.com

India|Bangalore|November10– 11|HotelLalit Ashok

ImmutableInfrastructure

SACON

Agenda

• WhatisImmutableInfrastructure?• Advantages/Drawbacks• Howto?• Demos

2

SACON

WhatisImmutableInfrastructure?

• AkaImmutableservers,phoenixservers• Traditionally• “Snow-flake”Serversarecontinuallyupdatedandmodifiedinplace• SSHintotheservers,update,patch,tweakconfig,deploynewcode

• An immutableinfrastructure isanotherinfrastructureparadigminwhichserversarenevermodifiedafterthey'redeployed.• ClassicImmutability• LinuxLiveCDs

3

SACON

RulesforTrulyImmutableServers

• Rule1:• Don’tchangetheOSfilesystem• Don’tinstall/upgrade/downgrade/removepackages• Notevenforsecurityvulnerabilities!• Don’teditconfiguration• Nohot-fixestoyourappcode

• Notevensmallorurgentfixes

• Rule2:• Iftemptedtodochangeanything

• FollowRule1

4

SACON

Semi-immutableservers

• Onlyautomatedsystemupdatesallowed• Nologins/SSH• Nomanualchanges

5

SACON

PhoenixServers

• ImmutableserverswithaTTL(Timetolive)• ServersSelf-destructaftere.g.1week• Recreateserversregularly

6

SACON

Howdoweupgrade?

• Createupdatedserverfromscratch• UpdatedOS• UpdatedApp• UpdatedConfiguration

• Deploy• Validate/testthedeployment• Snapshottheimageandcreatemultipleserversifrequired• Redirectproductiontraffictonewdeployment• Shutdownoldserversoroptionallykeepthemaround,justincase

7

SACON

Advantages– SecurityBenefits

• Reducedattacksurface• HardenedOS• RemovepackagesnotrequiredincludingSSH,bash,wget

• Attackerthrownoutwitholdserver• Backdoorsinstalledbyattackersdonotpersist• Theymaystillgetbackifvulnerabilitiesarenotpatched

• BetterAuditingandMonitoring• Centralizedmonitoring• Logunexpectedfilechanges• Logunexpectednetworkconnections

8

SACON

Advantages

• Definedbycode,henceversion-able• Relativelyeasytoreplicatetoapointintime• Easiertoscalable‘on-demand’• MoreconsistentStagingenvironments

SACON

Drawbacks

• Smallchangesarecumbersome• Solution:Automation

• Debugging/Forensicsaredifficult• Solution:Bundledebuggingtools,shiplogging/debuggingoutputtocentralizedlogservers(e.g.,ELK)

• Someapplications/components,e.g.,databasesoranythingstateful maynotfitwellintothisparadigm• Re-architectyourapplication/components• Implementdatabaserefactoring– separatedatabaseupdatesfromapplicationupdates.

• PassthebucktosomethinglikeAmazonRDSwhichmaintainsthedatabaseoruseAWSEBStopersistthedata

SACON

Drawbacks

• Mayincreasecostifyouplantokeepoldphoenixservers.• Solution:Deleteoldserversautomatically

• DevOpsmaturityisrequiredtoensurethatyoucanrolloutanupdatequickly(e.g.,incaseofa0dayflaw)

11

SACON

Howtoimplement?Recommendations

• Serversinanyvirtualizedenvironment(likecontainers,preferablyinacloudcomputingenvironment)• Isolatedinstances• FastProvisioningfromcustomimages

• Fullautomationofyourdeploymentpipeline• Aservice-orientedarchitecture• Stateless,VolatileApplicationlayer• PersistentDataLayer

• ExternalCentralizedlogging– ELK• Externaldatasources– AWSRDS,Datavolumes

• DedicatedDevOpsteam

12

SACON

Demo

• Immutablecontainersusingdocker• https://docs.docker.com/get-started/

SACON

RelevantTools&Technologies

• ContinuousDeliveryPlatform• Spinnaker,Jenkins,GoCD

• Cloudplatforms• AWS,GCP,Azure,Kubernetes

• Containers• Docker,docker-compose,AWSECS,GoogleContainerengine

• Cloudfunctions– Serverless Architectures• AWSLambda,Googlecloudfunctions

• Netflix’sChaosMonkey

SACON

FurtherReading

• https://www.digitalocean.com/community/tutorials/what-is-immutable-infrastructure• https://www.thoughtworks.com/insights/blog/moving-to-phoenix-server-pattern-introduction• https://martinfowler.com/bliki/ImmutableServer.html• https://devops.stackexchange.com/questions/49/what-are-the-pro-and-cons-of-snowflakes-servers-phoenix-servers-and-immutable-s• https://www.slideshare.net/jpetazzo/immutable-infrastructure-with-docker-and-containers-gluecon-2015• https://www.slideshare.net/SonatypeCorp/there-is-no-server-immutable-infrastructure-and-serverless-architecture

15