Modern, scalable deployment for plone

Scalable, modern deployment for Plone

Christian [email protected]

Tuesday, October 16, 12

Infrastructure


EntropyTuesday, October 16, 12

Platform

Man

agem

ent

Virtualization

Components

OS

Aut

omat

ion

Mon

itori

ng

Hardware

Data center

Application

Supp

ort


Platform

Man

agem

ent

Virtualization

Components

OS

Aut

omat

ion

Supp

ort

Mon

itori

ng

Hardware

Data center

Application


Data center


Platform

Man

agem

ent

Virtualization

Components

OS

Aut

omat

ion

Supp

ort

Mon

itori

ng

Hardware

Application

Data center


Hardware


Platform

Man

agem

ent

Virtualization

Components

OS

Aut

omat

ion

Supp

ort

Mon

itori

ng

Data center

Application

Hardware


• Virtualization is very affordable

• Platform creates interface between app and infrastructure

Virtualization

Components

OS


Platform

Man

agem

ent

Aut

omat

ion

Supp

ort

Mon

itori

ng

Hardware

Data center

Application

Virtualization

Components

OS


• convergence

• idempotency

• versioning

• everything as code

• continuous repairs

Automation


Platform

Man

agem

ent

Virtualization

Components

OS

Aut

omat

ion

Supp

ort

Mon

itori

ng

Hardware

Data center

Application


• Take care of updates and “production-ready” configuration

• Backup and other day-to-day operations

• OS and regularly used components

• Gradual process moving components from user-space installations to platform-wide features

Management


Platform

Man

agem

ent

Virtualization

Components

OS

Aut

omat

ion

Supp

ort

Mon

itori

ng

Hardware

Data center

Application


• Nagios :(

• (Un-)fortunately: #monitoringsucks

Monitoring


Platform

Man

agem

ent

Virtualization

Components

OS

Aut

omat

ion

Supp

ort

Mon

itori

ng

Hardware

Data center

Application


• Request tracker

• custom glue code

• Daily review

• Interrupt shield

Support


Platform

Components

Man

agem

ent

Virtualization

OS

Aut

omat

ion

Supp

ort

Mon

itori

ng

Hardware

Data center

Application


Application deployment and operations


Deployment

• Quick

• Repeatable

• Platform-independent

• single-click


Batou

• Model-based

• Use everywhere

• No effect on runtime

• Orchestration

• Convergency

• Idempotency

• Reusability


class Solr(Component):

def configure(self): self.address = Address(self.host.fqdn, 9000) self.provide('solr:server', self.address)

self += Buildout('solr', python='2.7')

self += Program('solr', command='java', command_absolute=False, args='-jar start.jar', directory=self.expand( '{{component.workdir}}/parts/instance'), restart=True)


[hosts]oshaweb00 = nginx, www, varnish, supervisor, haproxy, docsoshaweb01 = zope, hwthemes, supervisor...oshaweb20 = zope, hwthemes, supervisoroshaweb21 = zeo, openldap, supervisoroshaweb22 = nginx, media2oshaweb23 = nginx, birt, lms, ugm, supervisoroshaweb24 = solr, memcached, supervisoroshaweb26 = mailout

[component:haproxy]strategy = production_strategybots = 3whitelist = 8default = 3...


Components

goodnginx

varnish

haproxy

PostgreSQL

memcached

supervisor

postfix

badapache

mysql, BDB

OpenLDAP


Minimal "production-ready" checklist

• Sane front-end configuration

• System startup/shutdown

• Log rotation

• Database maintenance

• Monitoring: processes and ports

• Service user


Secrets

• One AES-encrypted config file per environment

• Transparent decryption

• Safety-belt to avoid accidental checking in unencrypted files

• Future: PGP-encryption


Environments

• Production vs. Staging

• Structurally identical but smaller

• Only one of many (dev, integration, ...)

• "Fly what you test, test what you fly."


Web applications

• Shared nothing in your app!

• Many small processes

• Do not listen early

• Never store runtime config in the database


Web applications II

• Log long running requests

• Avoid synchronous external requests

• Avoid accidental write requests.

• Use feature switches.


Process control

• only one component to integrate

• turn anything into a daemon

• use superlance (httpok, memmon) for active restarts

• use 3.0 release candidate


Python: virtualenv and buildout

• use both, combined

• buildout might be a hammer, but not everything is a nail


more buildout

• use extends to keep your configs clean and structured

• allow-picked-versions=False

• -t 3

• buildout >= 1.6

• mr.developer

• allowed-hosts = ...


and even more buildout

• multiple small better than one big

• different Python versions

• different subsets of egg versions

• use mirrors - see pypi-mirrors.org

• latency to the PyPI server is the biggest factor in buildout performance


Databases

• As much RAM as you can afford - seriously

• Low latency network (<0.1ms)

• Multi-core if necessary

• Disk usage < 40%


story: more hardware = slower

• Application is slow.

• Customer buys more hardware. Specifically: RAM.

• System is slower now.

• System runs many components. Hard to see through.



• One number stands out: process exit rate is 20/sec. PosgreSQL. What?

• Customer creates new connection for each query.

• PostgreSQL forks master for new connection.

• Fork went from few 10ms to few 100ms



• Diagnose. Find root cause.


HTTP/S termination

• HTTP and SSL

• Virtual hosting

• Redirects

• host normalization

• Proxy to load balancer

• Static files


Load balancing

• LB + HA

• Classify requests

• redispatch + retries

• balance: leastconn

• minconn 1, maxconn 2, threads +2

• Live statistic UI


Monitoring

• again :(

• processes, ports, logs

• model dependencies

• check_webpage.rb

• Business processes (BPI)


Reporting

• report + phone call

• incidents

• changes

• availability

• performance

• trends


SLAs

• Availability

• more keeps your business safe

• less reduces costs

• Friends don't let friends think in "nines".

• Alignment of technical and organisational measures


References

• "The practice of system and network administration" (Limoncelli, et. al.)

• Mark Burgess on config management

• "The twelve-factor app" (Heroku)

• gocept.net/doc


Conclusion I

• Use infrastructure to build platforms

• Platforms should be transparent.

•Deploy applications on platforms.


Conclusion II

• Do not cobble together random stuff from the internet.

• Careful work pays. Buzzwords don't.

• If you hear "enterprise". Run.

• Sapere aude.


Conclusion III

• I don't think IaaS and PaaS in the current extremes are a good answer to many projects.

• Many people experiment with ideas. Everything is becoming more flexible.

• Let's play!


Questions?Tuesday, October 16, 12

Modern, scalable deployment for plone

Technology

Transcript of Modern, scalable deployment for plone