10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

download 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

of 78

Transcript of 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    1/78

    10 deploys per dayDev & ops cooperation at Flickr

    John Allspaw & Paul Hammond

    Velocity 2009

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    2/78

    3 billion photos

    http://flickr.com/photos/jimmyroq/415506736/

    40,000 photos per second

    http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    3/78

    Dev versus Ops

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    4/78

    Its not my machines,its your code!

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    5/78

    Its not my code,its your machines!

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    6/78

    Spock ScottyLittle bit weird

    Sits closer to the boss

    Thinks too hard

    Pulls levers & turns knobs

    Easily excited

    Yells a lot in emergencies

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    7/78

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    8/78

    Says No all the time

    Afraid that new fangled things will break the site

    Fingerpointy

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    9/78

    They say NO all the time

    Because no one tells

    them anything

    Because

    They say NO all the time

    Because the site breaks

    unexpectedly

    Ops stereotype

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    10/78

    Devs job is to add new features

    Ops job is to keep the site stable and fast

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    11/78

    Ops job is NOT to keep the site stable and fast

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    12/78

    Ops job is to enable the business(this is devs job too)

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    13/78

    The business requires change

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    14/78

    But change is the root cause of most outages!

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    15/78

    Discourage change in the interests of stability

    or

    Allow change to happen as often as it needs to

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    16/78

    Lowering risk of changethrough tools and culture

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    17/78

    Dev and Ops

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    18/78

    Ops who think like devsDevs who think like ops

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    19/78

    But thats me!

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    20/78

    You can always think more like them

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    21/78

    Tools

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    22/78

    1. Automated infrastructureIf there is only one thing you do

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    23/78

    Chef

    Puppet

    CFengine

    FAI

    System Imager

    Cobbler

    BCfg2

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    24/78Role &configuration

    management

    OS imaging

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    25/78

    2. Shared version control

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    26/78

    http://www.flickr.com/photos/stewart/461099066/http://www.flickr.com/photos/stewart/461099066/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    27/78

    3. One step build

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    28/78

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    29/78

    3. One step buildand deploy

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    30/78

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    31/78

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    32/78

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    33/78

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    34/78

    http://www.flickr.com/photos/mauren/2429240906/

    http://www.flickr.com/photos/stewart/461099066/http://www.flickr.com/photos/stewart/461099066/http://www.flickr.com/photos/stewart/461099066/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    35/78

    4. Feature flags(aka branching in code)

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    36/78

    1.0 1.1 1.2

    1.0.1

    1.1.1

    1.0.2

    Desktop software

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    37/78

    r2301 r2302

    Web software

    r2306

    http://www.flickr.com/photos/stewart/461099066/http://www.flickr.com/photos/stewart/461099066/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    38/78

    http://www.flickr.com/photos/stewart/461099066/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    39/78

    http://www.flickr.com/photos/stewart/461099066/http://www.flickr.com/photos/stewart/461099066/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    40/78

    #php

    if ($cfg['enable_feature_video']){ }

    {* smarty *}

    {if $cfg.enable_feature_beehive} {/if}

    Feature flags

    http://www.flickr.com/photos/healthserviceglasses/3522809727/http://www.flickr.com/photos/healthserviceglasses/3522809727/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    41/78

    http://www.flickr.com/photos/healthserviceglasses/3522809727/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    42/78

    http://www.flickr.com/photos/healthserviceglasses/3522809727/http://www.flickr.com/photos/healthserviceglasses/3522809727/http://www.flickr.com/photos/healthserviceglasses/3522809727/http://www.flickr.com/photos/healthserviceglasses/3522809727/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    43/78

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    44/78

    http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    45/78

    5. Shared metrics

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    46/78

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    47/78

    Application level metrics

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    48/78

    Application level metrics

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    49/78

    Adaptive feedback loops

    App System Metrics

    RU ok?

    maybe?

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    50/78

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    51/78

    6. IRC and IM robots

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    52/78

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    53/78

    Culture

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    54/78

    1.RespectIf there is only one thing you do

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    55/78

    http://www.flickr.com/photos/aaronjacobs/64368770/

    http://www.flickr.com/photos/chrisdag/2286198568/

    http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    56/78

    http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    57/78

    http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    58/78

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    59/78

    Developers: Talk to ops about the impact of your code:

    what metrics will change, and how?

    what are the risks? what are the signs that something is going wrong? what are the contingencies?

    This means you need to work this out before talking to ops

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    60/78

    2.Trust

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    61/78

    http://www.flickr.com/photos/healthserviceglasses/3522809727/http://www.flickr.com/photos/healthserviceglasses/3522809727/http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    62/78

    http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    63/78

    http://www.flickr.com/photos/williamhook/3468484351/

    http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    64/78

    Ops: Be transparent,

    give devs access to systems

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    65/78

    3.Healthy attitudeabout failure

    http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    66/78

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    67/78

    http://www.flickr.com/photos/toms/2323779363/

    http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    68/78

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    69/78

    http://www.flickr.com/photos/dnorman/2678090600

    http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    70/78

    4.Avoiding Blame

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    71/78

    Fingerpointyness

    http://www.flickr.com/photos/flattop341/260207875/http://www.flickr.com/photos/flattop341/260207875/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    72/78

    Fingerpointyness

    problem!!!argggh!

    time

    freaking out,not talking,

    finding fault

    blaming,covering

    ass

    fixing

    thing

    s

    fixed.

    whining,hiding.

    hurt egos

    figuring itout

    Being productive

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    73/78

    Being productive

    problem!!!argggh!

    time

    fixing

    thing

    s

    fixed.

    feelingguiltyfiguring itout moveon with

    life

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    74/78

    Developers: Remember that someone else will

    probably get woken up when your code breaks

    http://www.flickr.com/photos/alex-s/353218851/

    http://www.flickr.com/photos/allspaw/2819774755/

    http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/http://flickr.com/photos/jimmyroq/415506736/
  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    75/78

    Ops: provideconstructive

    feedback on

    current achesand pains

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    76/78

    1. Automated infrastructure

    2. Shared version control3. One step build and deploy

    4. Feature flags

    5. Shared metrics

    6. IRC and IM robots

    1.Respect

    2.Trust

    3.Healthy attitude about failure4.Avoiding Blame

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    77/78

    This is not easyYou could just carry on shouting at each other

  • 8/14/2019 10+ Deploys Per Day_ Dev and Ops Cooperation at Flickr Presentation

    78/78

    (Thank you)