Download - Continous Delivery in Action

CONTINUOUS DELIVERY

in Action

by Hu Zhenbo (Andy) ThoughtWorks

Friday, April 27, 2012

History


years

4


Millions

0

25

50

75

100

2008 2009 2010 2011

Page Views/Day


0

17.5

35

52.5

70

2008 2009 2010 2011

Team Size70+


2008 2009 2010 2011

Features Delivered Days between major releases


major releases per year

3


days to deploy

20


people involved in deployment

30


Angry CustomerFriday, April 27, 2012

Sad TeamFriday, April 27, 2012

Why?


RequirementFriday, April 27, 2012

Fragile SystemFriday, April 27, 2012

Deploy to DeathFriday, April 27, 2012

2008 2009 2010 2011 2012

Features Delivered Days between major releases


delivery frequency

every 1 or 2 week


days to deploy

0.5


person dedicated for deployment

1


How’d we do it?


One TeamFriday, April 27, 2012

CollaborationFriday, April 27, 2012

Incremental Developing

Decode 4 types of data

21 pts




21 pts

Design

5 pts

Framework

8 pts

Decode Data

8 pts




21 pts

Decode Data A

8 pts

Decode Data C

5 pts

Decode Data B

5 pts

Decode Data D

3 pts


Feature ToggleFriday, April 27, 2012

Feature Toggle

# Feature TogglenonRealTimeRecharge = truerealTimeRecharge = false

Config File #if( $nonRealTimeRecharge )

<div>非即时到帐</div><div> ...

#end

#if( $realTimeRecharge ) <div>即时到帐</div> <div> ...#end

Page File


Deployment Pipeline

Commit Stagecompileanalysisunit test

...

Acceptance Test Stage

User Acceptance Testing

exploratory test...

Non-Functional Testing

performance test...

Production

Increasing confidence in build’s production readiness

Environment become more production-like

Slower Feedback


Deployment PipelineFriday, April 27, 2012

Auto-Deploy

Code Change

JarCommit

Jar

Update Portal

DependenZIP Upload

Stop Server

Unzip ConfigCreate

Log FileStart

ServerBackup


Auto-Deploy

Code Change

JarCommit

Jar

Update Portal

DependenZIP Upload

Stop Server

Unzip ConfigCreate

Log FileStart

ServerBackup

One-click Deploy


Components & Dependencies

Common Core

Payment

Account

Order

Portal

Admin

Mobile


Test QuadrantsFriday, April 27, 2012

Test Pyramid

AutomatedGUI Tests

API Tests

Integration Tests

Component Tests

Automated Unit Tests

ManualSessionBasedTesting


Deployment Test system configuration component communication service is up and running ...

Acceptance Test


Performance TestFriday, April 27, 2012

Blue-Green Deploy

WebServer

AppServer

DBServer

Router


Blue-Green Deploy

WebServer

AppServer

DBServer

v5.1 v5.1 v5.1

Router


Blue-Green Deploy

WebServer

AppServer

DBServer

v5.1 v5.1 v5.1

v5.2 v5.2 v5.2

Router


Data Migration

migrate - 001_create_products_table.sql - 002_add_column_to_products.sql - 003_update_data_for_products.sql

rollback - 001_create_products_table.sql - 002_add_column_to_products.sql - 003_update_data_for_products.sql


Green Database

v5.1


Green Database

v5.1readonly


Green Database

v5.1 v5.1

Backup

readonly


Green Database

v5.1 v5.1

Backup

v5.2

Migrate

readonly


Decouple App & Database

v5.1 v5.2 v5.3

v5.2 v5.3

compatible with db 5.1 & 5.2 compatible with db 5.2 & 5.3


Production Monitoring

HardwareOS

MiddlewareApplication

Voltage

Fan Speed

Temperature

CPU

Memory

HTTP

Disk

DB

DNS

SMTP


Production MonitoringFriday, April 27, 2012

Cross Functional TeamContinuous IntegrationIncremental Developing

Feature ToggleAutomate DeploymentDeployment Pipeline

Blue-Green DeploymentDatabase Migration

Production Monitoring...

Weekly Release


Good Enough?


A Long Way to Go

http://code.flickr.com/




Big ReleaseFriday, April 27, 2012

Continuous DeliveryFriday, April 27, 2012

blue-green deployment

canary releases

data management

dark launch

build pipeline

automatic deployment

branch by abstraction

DevOps

acceptancetest

virtualization

configurationmanagement

Version ControlSystem

could computinginfrastructure as code


Q & A

weibo: @HuZhenbo
