Self service build and deployment at Netflix (Agile 2013)

Tweet @garethbowles with feedback!

Self-Service Build & Deployment @Netflix

Monday, August 5, 13

• How would your organization be different if all of your engineers could build, test and deploy their own code ...

• ... and were responsible for fixing what they broke at 3am ?

Gareth Bowles

Netflix is the world’s leading Internet television network with more than 36 million members in 40 countries enjoying more than one billion hours of TV

shows and movies per month, including original series.

Source: http://ir.netflix.com

The Challenge

• We need to innovate rapidly, driven by:

• Global competition

• New connected devices

• Continuous customer feedback

• And we need fast rollback

The Challenge• We need to scale to cope with:

• Growing customer base

• Peaks in demand:

• Special events: holidays, Oscars

• Daily fluctuations (weekdays vs. weekends, daytime vs. evening)

Things That Help

• We can push out updates whenever we like

• Company culture

Things That Got in Our Way

A Few Short Years Ago ...

• Monolithic web app

• Single points of failure

• Releases were done by following runbooks

• DC-based infrastructure

• Different teams used different tools

Meeting the Challenge

http://www.slideshare.net/reed2001/culture-1798664

Freedom and Responsibility

• Hire mature people who work well with others

• Give them the context for company success

• Then get out of their way

• But hold them responsible for results

Context, not Control

• Be transparent about what the company needs to succeed

• Minimize the processes people need to go through to achieve success

• Value results, not planning and process

Highly Aligned, Loosely Coupled

• Clear strategy and goals

• Team interactions focus on strategy, not tactics

• Minimal cross-functional meetings

• Occasional post-mortems to increase alignment

What This Helped Us Achieve

• DVD to Streaming

• DC to cloud

• US-only to 40-plus countries

Architecture

Credit: Steve Somers

Key Changes

• Service oriented architecture

• Many small teams, each providing their own interconnected service

• Deploy on Amazon Web Services

• Increased reliance on open source

Highly aligned, loosely coupled

• Services are built by different teams who work together to figure out what each service will provide.

• The service owner publishes an API that anyone can use.

What AWS Provides

• Machine Images (AMI)

• Instances (EC2)

• Elastic IPs

• Load Balancers

• Security groups / Autoscaling groups

Freedom and Responsibility

• Developers deploy when they want

• They also manage their own capacity and autoscaling

• And fix anything that breaks at 3am!

Personaliza-‐Eon Engine User Info Movie

MetadataMovie RaEngs

Similar Movies

Reviews A/B Test Engine

2B requests per day

into the Ne3lix API

12B outbound requests per day to API

dependencies

Personaliza-‐Eon Engine User Info Movie

MetadataMovie RaEngs

Similar Movies

Reviews A/B Test Engine

2B requests per day

into the Ne3lix API

12B outbound requests per day to API

dependencies

Build and Deployment

The Audience

• ~700 engineers

• Large majority are developers

• Test engineers

• Delivery teams

• Operations & reliability engineering

Our Goal

• Lower the barriers to build, test and deployment until the entire process is accessible to every developer.

The Team

• 11 engineers and 1 director (but we’re hiring !)

• Developers, build / release engineers, DevOps

• Specialize, but understand the full stack

• Service oriented

Self-Service Build & Deployment

• Channel best practices

• Promote, don’t dictate

• Make adoption easy

• Make tools flexible

Building and Deploying

Perforce / Git

libraries

source

Ant targets

Groovy all over

snapshot / release libraries / apps

Jenkins

resolve

buildcompile report

publishtest

Artifactory yumAminator

Asgard

Building and Deploying

Perforce / Git

libraries

source

Ant targets

Groovy all over

snapshot / release libraries / apps

Jenkins

resolve

buildcompile report

publishtest

Artifactory yumAminator

Asgard

Is That Really Self-Service ?

Common Build Framework

• Define a build with just a few lines of Ant code

• Templates for libraries and webapps

• Override standard targets if you need to

Jenkins Job DSL• Define Jenkins build jobs using a domain

specific language (based on Groovy)

• Loop to create multiple jobs (e.g. for building different branches)

• Make one change and rerun to update all jobs

• The code is the configuration

• https://wiki.jenkins-ci.org/display/JENKINS/Job

Jenkins Dynaslaves

• Create build slaves in AWS

• Dedicated slave pools for teams

• Scale slave pools up and down on demand

• https://github.com/Netflix-Skunkworks/dynaslave-plugin

From Build to Deployment

Aminator

• Create (“bake”) AMIs

• Image contains a service and everything needed to run it

• Can be automatically triggered as a build step

• https://github.com/Netflix/aminator

How Baking is Different

https://github.com/Netflix/aminator

Traditional:•launch OS•install packages•install app

Generic AMI

Instance

Generic AMI

Instance

Generic AMI

Instance

Generic AMI

Instance

Netflix:•launch OS+app

Generic AMI

Instance

Netflix:•launch OS+app

App AMI Instance

Linux Base AMI (CentOS or Ubuntu)

Java (JDK 6 or 7)

Tomcat

Optional Apache

Monitoring

Log Rotation to S3

Appdynamics Machine Agent

Appdynamics App Agent

monitoring

Application war file, base servlet, platform, interface

jars for dependent services

GC and thread dump logging

Healthcheck, status servlets, JMX interface,

Servo autoscale

At Netflix, the AMI is the unit of deployment.

Asgard• Web UI and REST API for service deployment

and management

• Manage ASGs, ELBs, security groups, ...

• Application -> cluster -> ASG

• Rapid deployment and rollback

• Available to all engineers

• https://github.com/Netflix/asgard

Red / Black Deployment

Netflix has moved the granularity from the

instance to the cluster.

Simple Service Setup Effort• Write the code (variable :-))

• 15 minutes to write a build file and define dependencies

• 15 mins to create a Jenkins build, 2 to 10 mins to run it

• 5 mins to bake an AMI

• 10 mins to deploy in test, another 10 for prod

Just a quick reminder...

(Some of) Netflix is open source:

https://github.com/netflix

Why We Open Source

• Give back to Apache license OSS community

• Motivate, retain, hire top engineers

• Benefit from a shared ecosystem

• Make Netflix solutions into common standards

The Netflix PlatformDiscovery (Eureka)Entrypoints (Edda)

Configuration (Archaius)Zookeeper (Exhibitor)logging (Blitz4j & Honu)

NIWS (Ribbon)GeoBase

Hystrix

Circuit Breakers (Hystrix)Cassandra (Priam &

Astyanax & CassJMeter) Cryptex AKMS

EvCacheZuuli18nL10n

Open Source

https://github.com/Netflix/Cloud-Prize/wiki

Thank You !

Email: gbowles@{gmail,netflix}.com

Twitter: @garethbowles

Linkedin: www.linkedin.com/in/garethbowles

Self service build and deployment at Netflix (Agile 2013)

Technology

Transcript of Self service build and deployment at Netflix (Agile 2013)

IoT Poland 2016 - Over the air deployment bringing business closer to agile

Netflix Phone Support | Netflix Customer Support - 1800-986-3790

Agile Workforce Deployment: Mobilizing Talent at …...Agile Workforce Deployment: Mobilizing Talent at the Speed of Business September 9, 2014 1 Jennifer Connell, CRP, SGMS -T, Director,

Optimize your enterprise with agile ECM · An Agile Enterprise Infrastructure Laserfiche Rio’s agile ECM ecosystem offers: ` Quick and easy deployment. ` Straightforward and adaptable

Rapid Deployment of ERP solutions using Lean & Agile Methods

Continuous Deployment – Nextdoor.fi released every day at Scan-Agile 2011

An Agile Service Deployment Framework and its Application

Agile, Continuous Delivery, devops…. · • Agile – Iterations: Smaller but more frequent changes ... Release and Deployment Management Change Management (if we have time) Configuration

Webinar ITM Platform: Agile Deployment of Project Portfolio Management (PPM)

Microservices Tenets: Agile Approach to Service ... · Microservices Tenets: Agile Approach to Service Development and Deployment ... and Web technologies such as domain-driven design,

Agile Software Quality Function Deployment€¦ · Software QFD, Agile software development, Agile Requirements Engineering, Agile Software QFD . 1. INTRODUCTION The way to use software

CASE STUDY: Integrating Agile Operations to support NFV ... · CASE STUDY: Integrating Agile Operations to support NFV/SDN Deployment & B2B services redefinition, ... Growing ecosystem

Agile Deployment with the World's Largest and Most Trusted ... · Automating the movement and deployment of infrastructure, middleware, ... HOW AKAMAI SIMPLIFIES YOUR DEVOPS JOURNEY

Netflix 2009

Agile Delivery Framework Automation & Deployment With · PDF fileAgile Delivery Framework Automation & Deployment With Puppet ... experience in IT with expertise in DevOps, Agile Methodologies,

Agile Automation - · PDF fileAgile Automation What does “Agile ... What effect does Agile have? ... owns a shared staging environment?) QA pairing with DevOps –confidence in deployment

Hardening the Cloud: Assuring Agile Security in High ... · Hardening the Cloud: Assuring Agile Security in High-Growth Environments ... It is a shared journey for all ... Used Netflix

Salesforce Deployment: From Production Hacker to Agile Guru

Making the Agile Leap to Continuous Deployment

Enterprise Agile Deployment