BATOUmulti-(component|host|environment|.*)

deployment

Wednesday, 3.July 13

@theuni

Wednesday, 3.July 13

Wednesday, 3.July 13

Wednesday, 3.July 13

Wednesday, 3.July 13

AUTOMATING DEPLOYMENTS IS HARD

Wednesday, 3.July 13

HOW DOES

CONVERGENCE HELP?

Wednesday, 3.July 13

HOW DOES THIS WORK WITH BATOU?

Wednesday, 3.July 13

SOME PERSPECTIVE

Wednesday, 3.July 13

Wednesday, 3.July 13

IT'S NOT THAT BAD.

Wednesday, 3.July 13

Wednesday, 3.July 13

service deployment

Fabric, Capistrano, ...

system configuration Puppet, Chef, ...

provisioning kickstart, Razor, imaging ...

Wednesday, 3.July 13

FTP

bashmkzopeinstance

zc.buildoutfabric

Wednesday, 3.July 13

CONVERGENCE

Wednesday, 3.July 13

"Everything that follows is a result of what you see here."(Dr. Alfred Lanning; I, Robot)

Wednesday, 3.July 13

SIMPLEos.mkdir('foo')with open('foo/bar', 'w') as myfile: myfile.write('asdf')os.chmod('foo/bar', 0755)

Wednesday, 3.July 13

•unexpected system state

•can't resume

•unnecessary updates

os.mkdir('foo')with open('foo/bar', 'w') as myfile: myfile.write('asdf')os.chmod('foo/bar', 0755)

SIMPLISTIC

Wednesday, 3.July 13

CORRECT(?)if not os.path.isdir('foo'): os.unlink('foo')if not os.path.exists('foo'): os.mkdir('foo')try: os.lstat('foo/bar')except OSError: passelse: if os.path.isdir('foo/bar'): shutil.rmtree('foo/bar') else: os.unlink('foo/bar')if (os.path.exists('foo/bar') and open('foo/bar', 'r').read() != 'asdf'): open('foo/bar', 'w').write('asdf'):current = os.stat('foo/bar').st_modeif stat.S_IMODE(current) != 0755: os.chmod('foo', 0755)

Wednesday, 3.July 13

SIMPLE

File('foo/bar', content='asdf', mode=0755, leading=True)

Wednesday, 3.July 13

class File(Component):

namevar = 'path'

def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content)

File('foo/bar', content='asdf', mode=0755, leading=True)

Wednesday, 3.July 13

class File(Component):

namevar = 'path'

def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content)

File('foo/bar', content='asdf', mode=0755, leading=True)

compute target state (no touching!)

Wednesday, 3.July 13

class File(Component):

namevar = 'path'

def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content)

File('foo/bar', content='asdf', mode=0755, leading=True)

compute target state (no touching!)

composition operator

Wednesday, 3.July 13

class File(Component):

namevar = 'path'

def configure(self): self += Presence( self.path, leading=self.leading) self += Mode(self.path, self.mode) self += Content(self.path, self.content)

File('foo/bar', content='asdf', mode=0755, leading=True)

compute target state (no touching!)

composition operator

order matters

Wednesday, 3.July 13

class Presence(Component):

namevar = 'path' leading = False

def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading)

def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded()

def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass

Wednesday, 3.July 13

class Presence(Component):

namevar = 'path' leading = False

def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading)

def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded()

def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass

run "anywhere"

Wednesday, 3.July 13

class Presence(Component):

namevar = 'path' leading = False

def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading)

def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded()

def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass

run on target

run "anywhere"

Wednesday, 3.July 13

class Presence(Component):

namevar = 'path' leading = False

def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading)

def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded()

def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass

run on target

run "anywhere"

after all sub-components

Wednesday, 3.July 13

class Presence(Component):

namevar = 'path' leading = False

def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading)

def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded()

def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass

run on target

only if needed

run "anywhere"

after all sub-components

Wednesday, 3.July 13

class Presence(Component):

namevar = 'path' leading = False

def configure(self): if self.leading: self += Directory( os.path.dirname(self.path), leading=self.leading)

def verify(self): if not os.path.isfile(self.path): raise batou.UpdateNeeded()

def update(self): ensure_path_nonexistent(self.path) with open(self.path, 'w'): pass

run on target

only if needed

run "anywhere"

after all sub-components

keep delegating!

Wednesday, 3.July 13

class Directory(Component):

namevar = 'path' leading = False

def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded()

def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path)

Wednesday, 3.July 13

class Directory(Component):

namevar = 'path' leading = False

def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded()

def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path)

could be done with recursive composition

Wednesday, 3.July 13

class Directory(Component):

namevar = 'path' leading = False

def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded()

def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path)

could be done with recursive composition

refactor with sub-components if too

complex

Wednesday, 3.July 13

class Directory(Component):

namevar = 'path' leading = False

def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded()

def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path)

all methods optional: no configure()

could be done with recursive composition

refactor with sub-components if too

complex

Wednesday, 3.July 13

class Directory(Component):

namevar = 'path' leading = False

def verify(self): if not os.path.isdir(self.path): raise batou.UpdateNeeded()

def update(self): ensure_path_nonexistent(self.path) if self.leading: os.makedirs(self.path) else: os.mkdir(self.path)

all methods optional: no configure()

could be done with recursive composition

pattern: just wipe out what's wrong

refactor with sub-components if too

complex

Wednesday, 3.July 13

CONVERGENCE

resume where needed

handle many system states transparently avoid

unnecessary updates

Wednesday, 3.July 13

COMPONENTS

composition of simple components

no magic bullet, just a lot easier to factor

your code

configure - verify - update

Wednesday, 3.July 13

Wednesday, 3.July 13

SINGLE-COMMAND

Wednesday, 3.July 13

REPEATABLERELIABLE

Wednesday, 3.July 13

SIMPLE

Wednesday, 3.July 13

ENTROPY

Wednesday, 3.July 13

EXPRESSIVENESSREADABILITY

Wednesday, 3.July 13

REUSABLE

Wednesday, 3.July 13

PLATFORM INDEPENDENCE

Wednesday, 3.July 13

DOMAIN AGNOSTIC

Wednesday, 3.July 13

NO ADDITIONAL RUNTIME

DEPENDENCIES

Wednesday, 3.July 13

CONTINUITY

Wednesday, 3.July 13

MINIMAL DOWNTIMES

Wednesday, 3.July 13

Wednesday, 3.July 13

PRACTICAL USAGE

Wednesday, 3.July 13

REQUIREMENTS

Python 2.7

SSH

virtualenvMercurial

Wednesday, 3.July 13

ENVIRONMENTS

[environment]service_user = myservicehost_domain = flyingcircus.iobranch = production

[hosts]multikarl00 = nginx, haproxymultikarl01 = postgres, redis, memcached, crontabmultikarl12 = supervisor, logrotate, doctotext, myappmultikarl13 = supervisor, logrotate, doctotext, myapp

Wednesday, 3.July 13

LOCAL

$ bin/batou-local dev localhostUpdating Hello > File(hello) > Presence(hello)Updating Hello > File(hello) > Content(hello)$ bin/batou-local dev localhost$

Wednesday, 3.July 13

REMOTE$ bin/batou-remote prodtest02.gocept.net: connectingtest01.gocept.net: connectingtest01.gocept.net: bootstrappingtest02.gocept.net: bootstrappingOKOKDeploying test01.gocept.net/helloUpdating Hello > File(hello) > Presence(hello)Updating Hello > File(hello) > Content(hello)OKDeploying test02.gocept.net/helloUpdating Hello > File(hello) > Presence(hello)Updating Hello > File(hello) > Content(hello)OK

Wednesday, 3.July 13

OVERRIDES

class Hello(Component):

hostname = "foo"

[environment]...

[component:hello]hostname = bar

Wednesday, 3.July 13

SECRETS

class Hello(Component):

db_password = none

secrets/production.cfg

[hello]db_password = reallysecretstuff

Wednesday, 3.July 13

SECRETS

class Hello(Component):

db_password = none

secrets/production.cfg

[hello]db_password = reallysecretstuff

SciFibut close

Wednesday, 3.July 13

PROVIDE/REQUIREclass MyApp(Component):

def configure(self): self.provide('appserver', self.host.fqdn)

class HAProxy(Component):

def configure(self): self.backends = \ self.require('appserver')

Wednesday, 3.July 13

PLATFORMSclass HAProxy(Component): ...

@platform('flyingcircus.io', HAProxy)class SystemWideHAProxy(Component):

def configure(self): self += File('/etc/haproxy', ensure='symlink', link_to=self.parent.haproxy_cfg.path)

Wednesday, 3.July 13

VFS MAPPING

./

...

./work

./work/_/etc/haproxy.cfg

class HAProxy(Component):

def configure(self): self += File('/etc/haproxy')

[environment]...[vfs]sandbox = Developer

Wednesday, 3.July 13

FEATURESclass MyApp(Component):

features = ['instance', 'jobrunner']

def configure(self): if 'instance' in self.features: ...

[hosts]hosta = myapp:instancehostb = myapp:jobrunnerhostc = myapp:instance, myapp:jobrunnerhostd = myapp

Wednesday, 3.July 13

Wednesday, 3.July 13

CONVERGENCE

COMPOSITION

DETAILSWednesday, 3.July 13

QUESTIONS?

Wednesday, 3.July 13

batou.readthedocs.org

pypi.python.org/pypi/batou

bitbucket.org/gocept/batou

Wednesday, 3.July 13