Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

Behind the scenesHow we produce OpenStack

Thierry CarrezRelease Manager, OpenStack

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

OpenStack is large & growing

130+ git repositories1.8+ MLOC

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

OpenStack is painful

Difficult to deployDifficult to testDifficult to keep up with the changes

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

OpenStack is open innovation

Anyone can propose a changeLots of different companiesLots of different countries/culturesNo traditional management structure

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL?

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

5 Principles

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

1Not tested Not working→Not automatically tested Not tested→

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

2Code reviewsNobody gets a free pass

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

3Gate-centric developmentI, for one, welcome our new robotic overlords

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

4Time-based releasesHow I stopped worrying and embraced the coordination benefits

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

5Automate all the things...especially the boring tasks

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

Processes

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

6 m

onth

s

Design Summit, 4 days to discuss next release goals

PTLs define release goals

Milestones every 6 weeks

Last milestone, no more features (Feature Freeze)

RC1, creation of the release branch Final release

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

Design Write Code

Code review

Milestone Final

release Tests Gate Design

review Tests

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

Your git branch

master git branch

github

git.o.o

tarballs.o.o

review.o.o

developer

reviewers

infrastructurePEP8

HackingUnit tests

Integration tests

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

Challenges

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

NumbersOn a typical week day

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

1000Patchsets proposed

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

7500Gerrit comments and votes

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

16000Test runs, most spinning complete clouds

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

250Changes merged into OpenStack code

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

Thank you !

Coordination and Leadership challenges in producing OpenStack

Thierry Carrez (@tcarrez)Release management PTL

thierry@openstack.orghttp://fnords.wordpress.com@tcarrez on

ttx on

picture credits Chaos by Daniela Hartman (https://www.flickr.com/photos/29487767@N02/4255026872) Coordination by Doug88888 (https://www.flickr.com/photos/doug88888/4687906267) Boring tasks by Alex Eylar (https://www.flickr.com/photos/hoyvinmayvin/5166095952) Congestion by Joiseyshowaa (https://www.flickr.com/photos/joiseyshowaa/7454479488) Rare issues by University of Liverpool HLS (https://www.flickr.com/photos/liverpoolhls/10740309163) Community focus by Thierry Carrez Watercooler by Eric Domont (https://www.flickr.com/photos/edomond/5348007434) Tension by J.D. Hancock (https://www.flickr.com/photos/jdhancock/7186117659)