Chrome Release Cycle

Author: Anthony Laforge (laforge@chromium.org)

Our Philosophy

Think of any major website, even the fancy Web 2.HTML5 ones... do they have version numbers?We took the same approach to our client software as an online web service.That is... we treat releases as a means of getting features out to users and not goals in and of themselves.It's about flow.

How Our Users Work

Users sign up for Chrome release channels based on the level of stability they want.Updates happen automatically in the background.New features and functionality just flow in without any effort on the user's part.

How the Chrome Team Works

Developers work almost exclusively on a single central trunk (possible due to our try bots and continuous build infrastructure)Our branches stem from points on the trunkWe stabilize branches by pulling in changes from trunk (i.e. everything lands on trunk first)

Where we started (v1-4) - On Paper

In practice, for that final beta...we'd

Cut a branch (when we thought we had everything)Merge about ~500 patches (since we didn't have everything)Spend weeks stabilizing and re-stabilizing issues.Ended up working 1-3 months to get a release out the door, always certainly missing our 13 week plan.And at the end finally shipping something we were happy with... but that left us pretty drained (i.e. the bad flow)

The chain of events

We'd hold releases, sometimes our branch points, until features were done, which meant...Lots of changes got queued up on the trunk, which caused... Engineers merging lots of changes to the branch, which led to...Instability on the branch that we had to deal with and that meant...

That chain of events...

Long lived branches, which...Made supporting the branch (i.e. merging changes) more difficult as we drifted further from the trunk...All of which led to...

Unpredictable release cycles, with date targets we could never hit, and....Engineers who were always in a rush to get their features into "this" release since we couldn't make any promises about the schedule

Which led me to think...

I need a long vacation.There has to be a better way.What's causing things not to flow smoothly?

...Yes...In that order.

Primary GoalsShorten the release cycle and reduce the life span of a branch (make merges easier)Make releases more predictable and easier to scopeReduce the strain on the entire team

BadGood

Goal to Plan - Getting down the cycle

Originally set out a plan to simplify the 12 week cycle, 6 weeks on dev, 6 weeks on beta, and then to stable.Once diagrammed, using blocks to represent the weeks, realized that with 3 channels you could have two overlapping releases running at once, which could get us a stable release ~ every 6 weeks.

Block diagrams

It was a start, but it wasn't the full answer

Sure, turning the wheel faster would make some things, like merges, easier.But without addressing the scope problems, the flow wouldn't be any better and would continue to interrupt releases.Things wouldn't be any more predictable and the whole cycle would just fall apart (more puddles, less of a stream).

Goal to Plan - Controlling Scope

The pace of the schedule sets the boundaries for the amount of work that can be completed.It's important to have specific points in the schedule to review features and cut scope.Establish clear expectations (and engineering practice) to developers that any features not ready to ship at branch will be disabled (i.e. we only cut post branch, never add).

Plan to Pitch

Reached out to the various cross functional groups on the Chrome team, who would all be impacted, before approaching our executives.

Engineering, QA, Product, Marketing, Support, Localization, Security, etc...

There were a lot of concerns to address, but the exercise forced a lot of thought on the implications of the schedule change.It took time, but it made the pitch easier to present to the leadership and the rest of the team.

ConcernsWould large feature development still be possible?

"Yes, engineers would have to work behind flags, however they can work for as many releases as they need to and can remove the flag when they are done."Can the engineers keep up?

"Their pace won't need to change, since features can be disabled there should be no milestone pressure, things ship when they are ready."What would a world look like where we didn't base our marketing on releases?

"We market features, not releases."

And so we implemented it

11 week overlapping schedules.

Our General Rules

The branch point is the end of our development cycleFeatures only get disabled post branch pointFeatures should be engineered so that they can be disabled easily (1 patch)Only stability, security, and critical regressions can ever block a release

Things we struggled withOvercoming team inertia (particularly when we cut scope)Saying no consistently to the teamFighting the urge to return to date driven management decisionsWe initially didn't solve the trunk->dev->branch time to patch problem (daily channel helped).Setting the right burn down point (i.e. branch point).Disabling features.

Key Lessons

Having clear and concise rules is important.Predictable schedules cut down on communication costs and team confusion.It takes work and fore-thought to disable features.Diligent feature tracking is important.

ConclusionSpeed alone isn't always the right answer, it's about keeping things running smoothly.For us, scope was getting in the way of that goal.

We basically wanted to operate more like trains leaving Grand Central Station (regularly scheduled and always on time), and less like taxis leaving the Bronx

(ad hoc and unpredictable).