1

Synthesizing Continuous Deployment Practices Used in Software Development

Akond Ashfaque Ur Rahman, Eric Helms, Laurie Williams, and Chris Parnin

Department of Computer Science, North Carolina State University

2

Why Continuous Deployment ?

• “one of the top ten strategic technology trends for 2015 ” – Gartner

• emerging software development process

3

Motivation Software companies learn from the experience of other similar software companies

Crossing The Chasm

4

Research Objective

The goal of this paper is to aid software practitioners in implementing continuous deployment through systematic analysis of software practices that are used by software companies

5

How Continuous Deployment is Different ?

XP Scrum Kanban

Continuous Deployment

6

Definition of Continuous Deployment

“as a software process that releases software changes automatically to end-users after they pass the required automated tests. ”

Martin FowlerImage Reference: http://martinfowler.com/

7

Definition of Continuous Delivery

“the software process that actually releases software to production as soon as they are ready, resulting in many deployments to production every day. ”

Image Reference: http://continuousdelivery.com/ Humble & Farley

8

Our Definition

“a software engineering process where incremental software changes are automatically tested, and frequently deployed to production environments. ”

9

Realization of Continuous Deployment Varies

One Example

vs.Facebook Etsy

10

Adoptees

19 adoptees

11

Our Contributions

• A summary and concise definition of 11 continuous deployment practices

• A mapping of the 11 continuous deployment practices and the software companies who are using these practices

• A list of techniques adoptees have used to realize each of the identified continuous deployment practices

12

Research Question

How frequently are different software practices used by software companies that perform continuous deployment?

?

13

Methodology

Identify 11 Practices

Identify Adoptees and Their Practices

Missing ?

Follow-up Inquiries

Data Analysis

Yes

Other 18 adoptees : internet artifacts

Facebook : Feitelson et al.

No

14

1. Automated Deployment (19)- making software available to end-users automatically - conducted in between software acquisition and software

execution without manual effort

Technique Used Total Count Examples of Adoptees

Other Tools 13 CoolBlue, Etsy

Scripting 4 IMVU, Rally Software

BitTorrent 2 Facebook, Pinterest

Deployinator, Ansible, Codeship, Octodeploy etc.

15

2. Automated Testing (19)- practice of automated techniques to perform test case

management, test data generation etc.

Type of Testing Total Count Examples of Adoptees

Unit 15 Etsy, Facebook

Integration 13 Etsy, IMVU

A/B 9 Etsy, Facebook

Functional 8 CoolBlue, Etsy

Acceptance 5 Atlassian, Netflix

Regression 5 CoolBlue, Etsy

Perceptual 1 Google Consumer Surveys

16

2. Automated Testing [Contd.]

Technique Used Total Count

Examples of Adoptees

Automated Test Suites 19 Etsy, Facebook

Separate Testing Team 2 Atlassian, Netflix

17

Technique Used Total Count Example of Adoptees

Other Tools 9 CoolBlue, Google Consumer Surveys

Github Tools 7 Atlassian, EtsyManual Inspection 6 Atlassian, Facebook

Phabricator 2 Facebook, Quora

3. Code Review (16)- practice of presenting software changes to different

stakeholders for comments and approval

Lint checkers, Static analysis tools, coding

convention tools etc.

18

4. Dark Launching (9)- practice of deploying software changes by keeping the functional aspects hidden to end-users

Technique Used Total Count Example of Adoptees

Gatekeeper 1 Facebook

Note: Other adoptees who use this practice did not discuss their techniques

19

5. End-User Communication (10)- practice of communicating with end-users to receive feedback and gather requirements

Technique Used Total Count Example of Adoptees

Official Forums 6 Etsy, Facebook

Social Networking Websites 2 Github, Spreaker

Phone Calls 1 Wealthfront

Web Seminars 1 Rally Software

20

6. Feature Flags (13)- the practice of triggering a specific software branch amongst many software branches

Technique Used Total Count Example of Adoptees

Conditional Framework 1 Rally Software

Configuration Flags 1 Etsy

Note: Other adoptees who use this practice did not discuss their techniques

21

7. Intercommunication (14)- practice of sharing all necessary development and deployment information amongst team members

Technique Used Total Count Example of Adoptees

Conversation Bots 2 Facebook, Flickr

ChatOps 1 Github

Gerrit 1 URLinkedUp

Note: Other adoptees who use this practice did not discuss their techniques

22

8. Monitoring (16)- practice of collecting deployment related information, producing appropriate performance metrics

Technique Used Total Count Example of Adoptees

Graphite 2 Etsy, Github

Nagios 2 IMVU, Rally Software

Splunk 1 Rally Software

Note: Other adoptees who use this practice did not discuss their techniques

23

- practice of deploying the master or a different branch of the software

Technique UsedTotal Count

Example of Adoptees

Trunk Shipment 14 Coolblue, Etsy

Branch Shipment 3 Atlassian, Netflix

9. Repository Use (19)

24

10. Shepherding Changes (14)

- practice of making developers responsible for their software changes

Technique Used

Total Count Example of Adoptees

On-call policy 2 Etsy, Github

Boot camps 1 Facebook

Note: Other adoptees who use this practice did not discuss their techniques

25

11. Staging (17)- practice that controls the deployment of software changes to internal and

production servers

Technique UsedTotal Count

Example of Adoptees

Gradual roll out 17 Etsy, Facebook

Dogfooding 12 Etsy, Facebook

26

Empirical FindingsType of Product Deployed

Adoptee Total Count

Other Type of Websites Atlassian (AT), Google Consumer Surveys (GCS), Kitchensurfing (K), Netflix (N), Outbrain (O), Rally Software (RS), Spreaker (S)

7

Social Networking Websites

Facebook (FB), Flickr (FR), Github (G), IMVU (I), Quora (Q), Pinterest (P) 6

E-Commerce Websites Coolblue (CB), Etsy (E), Flipkart (FK), URLinkedUp (U), Wealthfront (WF) 5

Desktop Software Atlassian (AT) 1Review Websites Yelp (Y) 1

Survey, Cooking, Audio etc.

27

Empirical Findings

Autom

ated

Dep

loym

ent

Autom

ated

Tes

ting

Code R

eview

Dark L

aunch

ing

End-Use

r Com

munica

tion

Featu

re F

lag

Inter

com

munica

tion

Mon

itorin

g

Repos

itory

Use

Shepher

ding C

hange

s

Stagin

g0

3

6

9

12

15

18

Commonality of Practices

Count of Adoptees (Yes) Count of Adoptees (No) Count of Adoptees (Unknown)

Cou

nt

of P

ract

ices

28

Empirical Findings

- Academic articles - Speakerdeck presentations

Blog P

osts

Slides

hare P

rese

ntatio

ns

Info

Q Pre

senta

tions

Video

s

Other

0

5

10

15

20

25

Count of Different Internet Artifacts

Type of Internet Artifacts

Cou

nt

29

Empirical Findings

Autom

ated

Dep

loym

ent

Autom

ated

Tes

ting

Code R

eview

Dark L

aunc

hing

End-U

ser C

omm

unica

tion

Featu

re F

lag

Inter

com

mun

icatio

n

Mon

itorin

g

Repos

itory

Use

Shep

herd

ing C

hang

es

Stag

ing

0

3

6

9

12

15

18

Use of Internet Artifacts and Follow-up Inquiries

Internet Artifacts Follow-up Inquiries

Cou

nt

of P

ract

ices

30

Observations

• Commonality of Continuous Deployment Practices

• Different Types of Testing • Shepherding Changes

31

Limitations

• Keyword-based search mechanism • Evaluation of the 11 software engineering

practices are missing

32

Future Directions

• Understanding how these 11 practices help adoptees to achieve continuous deployment successfully

• Providing guidelines on how software companies can use the identified 11 software practices to implement continuous deployment effectively

33

Conclusion

• Summarizing 11 specific software practices used in industry to implement continuous deployment, and the adoptees that are using these practices

• Mapping of the 11 continuous deployment practices and the software companies who are using these practices

• Observing the commonality of practices used to implement continuous deployment

34

Related Work• G. G. Claps, R. B. Svensson, and A. Aurum, “On the journey to

continuous deployment: Technical and social challenges along the way,” in Information and Software Technology, January, 2015

• N. F. Velasquez, G. Kim, N. Kersten, and J. Humble, “2014 State of DevOps Report,” https://puppetlabs.com/sites/default/files/2014-state- of-devops-report.pdf, 2014.

• M. Leppanen, S. Makinen, M. Pagels, V. Eloranta, J. Itkonen, M. Mantyla, V. Mika, and T. Mannisto, “The Highways and Country Roads to Continuous Deployment,” in IEEE Software, March-April, 2015