Continuous Deployment at Etsy: A Tale of Two Approaches
-
Upload
ross-snyder -
Category
Technology
-
view
17.163 -
download
4
description
Transcript of Continuous Deployment at Etsy: A Tale of Two Approaches
![Page 1: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/1.jpg)
Continuous Deployment at
March 9, 2013
Ross [email protected]@beamrider9
:A Tale of Two Approaches
1
![Page 2: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/2.jpg)
A quick primer on
2
![Page 3: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/3.jpg)
is:The global marketplace we make together.
3
![Page 4: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/4.jpg)
is:The premier destination for handmade goods,
vintage items, and craft supplies.
simplertimestoys lacklusterco norwesterseaglass
4
![Page 5: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/5.jpg)
quick facts:
• 22+ million members
• 800,000+ active shops
• 18+ million items currently for sale
• 20 cents to list item, 3.5% transaction fee
• 400+ employees (majority in Brooklyn)
(as of March 2013)
5
![Page 6: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/6.jpg)
$0
$200
$400
$600
$800
$1,000
2005 2006 2007 2008 2009 2010 2011 2012
Gross Merchandise Sales ($MM)
Since opening its doors in June 2005, Etsy has grown virtually non-stop.
6
![Page 7: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/7.jpg)
A nice problem to have:
“Our site is so successful, how can we
move fast enough to keep up with demand?”
7
![Page 8: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/8.jpg)
CONTINUOUSDEPLOYMENT
8
![Page 9: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/9.jpg)
:The Early Years
(2005 - 2008)
9
![Page 10: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/10.jpg)
: The Early Years
1. Spend significant time writing code
10
![Page 11: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/11.jpg)
: The Early Years
2. Painful source control merge
1. Weeks writing code
11
![Page 12: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/12.jpg)
: The Early Years
3. Hand off to someone else to deploy
1. Weeks writing code2. Painful merge
12
![Page 13: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/13.jpg)
: The Early Years
4. Deploy, site goes down
1. Weeks writing code2. Painful merge3. Hand off to deployers
13
![Page 14: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/14.jpg)
: The Early Years
5. Roll back deploy
1. Weeks writing code2. Painful merge3. Hand off to deployers
4. Deploy, site down
14
![Page 15: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/15.jpg)
: The Early Years
6. Spend hours (days?) fixing bugs
1. Weeks writing code2. Painful merge3. Hand off to deployers
4. Deploy, site down5. Roll back deploy
15
![Page 16: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/16.jpg)
: The Early Years
7. Go back to step 2
1. Weeks writing code2. Painful merge3. Hand off to deployers
4. Deploy, site down5. Roll back deploy6. Fix bugs
16
![Page 17: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/17.jpg)
: The Early Years
WATERFALL!
17
![Page 18: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/18.jpg)
: The Early YearsPros:
Early Etsy engineers used this release cycle to bootstrap the marketplace from nothing.
Forever grateful.
18
![Page 19: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/19.jpg)
: The Early Years
• Large changesets• Infrequent deploys• Weak confidence in deploy success• Significant time spent deploying• Low ability to experiment/iterate/react• Developer stress/unhappiness
Cons:
19
![Page 20: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/20.jpg)
: The Early YearsBy late 2008, Etsy is still a startup, but has
the deploy process of a much bulkier company.
Popularity is on the verge of outpacing capacity.
20
![Page 21: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/21.jpg)
:Today
21
![Page 22: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/22.jpg)
: Today
1. Small changesets, deployed frequently
22
![Page 23: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/23.jpg)
: Today
2. Engineers deploy the site
1. Small changesets
23
![Page 24: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/24.jpg)
: TodayAnd not just engineers, but also:
• Designers• Product Folks• Upper Management• Board Members• Dogs
24
![Page 25: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/25.jpg)
: Today
3. Deploys are fast and near-effortless
1. Small changesets2. Engineers deploy
25
![Page 26: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/26.jpg)
: Today
4. Most changes behind config flags(safer deploys)
1. Small changesets2. Engineers deploy3. Deploys are fast
26
![Page 27: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/27.jpg)
: Today
5. Graphs/metrics to assess deploy
1. Small changesets2. Engineers deploy3. Deploys are fast
4. Changes behind flags
27
![Page 28: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/28.jpg)
: Today
6. If issues, fix immediately & roll forward
1. Small changesets2. Engineers deploy3. Deploys are fast
4. Changes behind flags5. Copious graphs/metrics
28
![Page 29: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/29.jpg)
This isn’t license to break stuff, quickly.
Engineer-driven QA and solid unit testing are integral parts of the process.
: Today
29
![Page 30: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/30.jpg)
: Today
7. Repeat 25+ times per day, every day
1. Small changesets2. Engineers deploy3. Deploys are fast
4. Changes behind flags5. Copious graphs/metrics6. Fix fast & roll forward
30
![Page 31: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/31.jpg)
Then:
1. Small changesets2. Engineers deploy3. Deploys are fast4. Changes behind flags5. Copious graphs/metrics6. Fix fast & roll forward
1. Weeks writing code2. Painful merge3. Hand off to deployers4. Deploy, site down5. Roll back deploy6. Fix bugs, go to step 2
Now:
31
![Page 32: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/32.jpg)
Etsy Deploy Stats: 2012
• Deployed to production 6,419 times
• On average, 535/month, 25/day
• Additional 3,851 config-only deploys
• 196 different people deployed to prod
• Nov/Dec 2012: deployed 752 times
32
![Page 33: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/33.jpg)
Why does it work?
33
![Page 34: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/34.jpg)
• N = # of deploys• P = probability of site degradation• S = average severity of degradation• T = time to detect/resolve
= N * P * S * TExpectedDowntime
Continuous Deployment Math
34
![Page 35: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/35.jpg)
• N = 1• P = 0.5• S = 0.7• T = 100
Before:
E.D. = 35
• N = 250 ↑↑↑↑• P = 0.1 ↓• S = 0.05 ↓↓• T = 5 ↓↓↓
Now:
E.D. = 6.25
N = # of deploys P = prob. of degradation
S = avg. severity of degradation T = time to detect/resolve
(all numbers completely arbitrary)
Continuous Deployment Math
35
![Page 36: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/36.jpg)
Etsy circa 2013 (400+ employees)acts, in some ways, more like a startup than
Etsy circa 2008 (40+ employees).
Big Takeaway
36
![Page 37: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/37.jpg)
Continuous Deployment makes possible: “Continuous Experimentation”
http://etsy.me/continuous-experimentation
37
![Page 38: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/38.jpg)
Continuous Experimentation
1. Small changes2. Run experiment (A/B test)3. Analyze data4. Re-examine assumptions
Repeat continuously in pursuit of larger goals.
38
![Page 39: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/39.jpg)
Heard since 2010: “Neat experiment, but this will never scale.”
As of 2013, Etsy has 100+ engineers -still going strong.
39
![Page 40: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/40.jpg)
Some Etsy Customizations
Deploying is a first-class feature. Inability to deploy is a P1 incident (same as site down).
40
![Page 41: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/41.jpg)
Some Etsy Customizations
• Internal admin site• API • Big data• Search• Blog• Deployinator itself
We continuously deploy not just the main Etsy website, but as much as possible:
41
![Page 42: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/42.jpg)
Some Etsy Customizations
• Database schema changes• PCI-DSS environment (credit cards)
In the rare case we can’t continuously deploy, we create alternative tools:
We do continuously deploy as much of our payment processing as is safe & legal (98%).
42
![Page 43: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/43.jpg)
Some Etsy Customizations
Keeping deploys fast is paramount and worth the investment in manpower & hardware.
43
![Page 44: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/44.jpg)
Some Etsy CustomizationsContinuous deployment is all about moving
forward, sometimes at the expense of the past.
Our solution: engineering-wide bug rotation, one day a month, every engineer participates.
44
![Page 45: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/45.jpg)
Continuous Deployment is a fantastic recruitment tool for attracting engineers who like to move fast and get stuff done.
Fun Fact:
45
![Page 46: Continuous Deployment at Etsy: A Tale of Two Approaches](https://reader030.fdocuments.in/reader030/viewer/2022020217/546e9aaab4af9faa268b4734/html5/thumbnails/46.jpg)
Learn more:http://codeascraft.etsy.com/
Join the fun:http://www.etsy.com/careers
Etsy open source (Deployinator, StatsD)http://etsy.github.com/
46