Copyright © 2011 Constant Contact Inc.
Beyond a Scrum of Scrums: Scaling Up Agile with Kanban
Gil IrizarryProgram ManagerConstant Contact
June 2011
2Copyright © 2011 Constant Contact, Inc.
Agenda
• A bit about me and Constant Contact
• Scrum and Kanban at the team-level
• Scaling up to an engineering org
3Copyright © 2011 Constant Contact, Inc.
My background
• Over 20 years software development and management experience
• Over 5 years in an agile software development environment
• Most recently working with Kanban, prior to that Scrum and RUP
• CSM and PMP certifications, Kanban coaching training with David Anderson
• BS from Cornell, ALM from Harvard, certificate in Management from MIT Sloan
4Copyright © 2011 Constant Contact, Inc.
My contact info
• Contact me if you would like updated slides
5Copyright © 2011 Constant Contact, Inc.
Background on Constant Contact
• SaaS company offering on-line e-mail marketing, event marketing and surveys. Recent enhancements extend the services to the social media space
• >$200MM gross revenue per year
• >800 employees
• >450K paying customers
• Engineering and Operations total about 150 people
• First Scrum team formed in 2006
6Copyright © 2011 Constant Contact, Inc.
The Goal
• We want to release more features more frequently
• Why?
• React quicker to changing market conditions
• Get new features to users more quickly
• Frequent releases are smaller releases: our customers often cannot absorb a large set of changes
7Copyright © 2011 Constant Contact, Inc.
The Challenge
• 13 software development teams
• 5 development and testing server clusters
• 2 production server clusters in 2 data centers (soon to be 3)
• A code base that was first created around 2004 and built up over time
• And all utilizing one operations team
8Copyright © 2011 Constant Contact, Inc.
What do we have?
• A mess of inter-dependencies
• Image from
http://viewyonder.com/2009/07/12/data-center-consolidation-road-to-unified-computing-service-analysis/ - Data Center Spaghetti. This problem exists in many other parts of the enterprise!
9Copyright © 2011 Constant Contact, Inc.
So, how do we address this?
• Scrum of Scrums!
• Um, wait…
10Copyright © 2011 Constant Contact, Inc.
A little review: how things used to be
11Copyright © 2011 Constant Contact, Inc.
This is appropriate if you are building an…
“Originally planned to enter service in May 2008, the [Dreamliner] project has suffered from repeated delays and is now more than three years behind schedule.” -- Wikipedia
12Copyright © 2011 Constant Contact, Inc.
Quick Review of Scrum
• Fixed iterations
• Daily stand-ups
• What did you do yesterday, what did you do today, any impediments
• Retrospectives
• Burn-down chart
• Board with To Do, In Progress and Done states
13Copyright © 2011 Constant Contact, Inc.
Scrum in a slide
Sprint: 4 weeks
3 Meetings
3 Documents
3 Roles
PO
1
SM
3
Product Backlog
1
Sprint BL
2
Burn-down
3
Product
Stakeholders
On-going Collaboration
Sprint Planning
Sprint Planning
1
Daily Scrum
2
SprintReview
SprintReview
3
Daily
TEAM
2
14Copyright © 2011 Constant Contact, Inc.
Scrum of Scrums (of Scrums (of Scrums…))
• How to integrate teams in Scrum?
• Hold a Scrum of Scrums
• What happens if you have too many teams?
• Hold a Scrum of Scrums of Scrums, and so on…
• Add a MetaScrum or two
15Copyright © 2011 Constant Contact, Inc.
Foundational Principles of Kanban
• Start with what you do now
• Agree to pursue incremental, evolutionary change
• Respect the current process, roles, responsibilities & titles
From: http://agilemanagement.net/index.php/Blog/the_principles_of_the_kanban_method (David Anderson)
16Copyright © 2011 Constant Contact, Inc.
5 Core Properties of Kanban
• Visualize the workflow
• Team board states are a reflection of the value stream
• Limit WIP
• Manage Flow
• Implied that flow should be continuous
• Make Process Policies Explicit
• Improve Collaboratively (using models & the scientific method)
17Copyright © 2011 Constant Contact, Inc.
Sample Kanban Board
18Copyright © 2011 Constant Contact, Inc.
Sample CFD
11/9
/201
0
11/1
5/20
10
11/2
1/20
10
11/2
7/20
10
12/3
/201
0
12/9
/201
0
12/1
5/20
10
12/2
1/20
10
12/2
7/20
10
1/2/
2011
1/8/
2011
1/14
/201
1
1/20
/201
1
1/26
/201
1
2/1/
2011
2/7/
2011
2/13
/201
1
2/19
/201
1
2/25
/201
1
3/3/
2011
3/9/
2011
3/15
/201
1
3/21
/201
1
3/27
/201
1
4/2/
2011
4/8/
2011
4/14
/201
1
4/20
/201
1
4/26
/201
10
10
20
30
40
50
60
Makeover Project - Cumulative Flow Diagram
User StoryMockupsReady-DoneIn DevelopmentDev DoneIn TestingComplete
19Copyright © 2011 Constant Contact, Inc.
Compare with Burndown chart
19-J
ul
21-J
ul
23-J
ul
27-J
ul
29-J
ul
2-Aug
4-Aug
6-Aug
10-A
ug
12-A
ug
16-A
ug
18-A
ug
20-A
ug
24-A
ug
26-A
ug
30-A
ug
1-Sep
3-Sep
7-Sep
9-Sep
13-S
ep
15-S
ep
17-S
ep
21-S
ep
23-S
ep
27-S
ep
29-S
ep1-
Oct0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00Social Project - Burndown to date
Social Media Tasks
Linear (Social Media Tasks)
Tas
ks R
emai
ning
20Copyright © 2011 Constant Contact, Inc.
Scrum and Kanban: Let’s combine them!(But how? And why?)
http://en.wikipedia.org/wiki/Reese's_Peanut_Butter_Cups#Marketing_and_advertising
21Copyright © 2011 Constant Contact, Inc.
Transistioning from Scrum to Kanban
• We did not change how teams were currently working
• We modeled existing hand-offs within the team, i.e. each team’s kanban board reflected that teams style of work
22Copyright © 2011 Constant Contact, Inc.
The Dilemma
Remember that teams want (and often need) to work continuously, but releases are discrete.
Multiple teams working together create interdependencies
How to resolve this?
23Copyright © 2011 Constant Contact, Inc.
Transforming Scrum (and Scrum of Scrums)
• At the enterprise-level, team(s) management becomes a problem of dependency management
• Planning, coordination and stand-ups at the departmental and organizational level need to be executed with this mind
24Copyright © 2011 Constant Contact, Inc.
What we have done at CTCT
• At the organization-level:
• Prioritized project list
• All At Once Planning
• Classes of Service
• (Near) Continuous integration and (mostly) automated regression testing
• Dependency and deliverable review
• Release train
25Copyright © 2011 Constant Contact, Inc.
What we have done at CTCT
• At the team-level:
• Continuous delivery
• Better metrics and metrics-driven estimation
26Copyright © 2011 Constant Contact, Inc.
Prioritized Project List
• Product Management, Engineering and Operations leads meet every two weeks to prioritize all development projects
• This becomes the organizational backlog, which drives each team’s backlog
27Copyright © 2011 Constant Contact, Inc.
All At Once Planning
28Copyright © 2011 Constant Contact, Inc.
All At Once Planning
• Prior to the start of an iteration, teams use the prioritized project list to plan their upcoming work.
• Planning involves the identification of deliverables and dependencies.
• Dependencies are discussed with dependent teams.
• A meeting is held (all at once planning mtg) in which all development teams present their dependencies to each other and to the operations team.
29Copyright © 2011 Constant Contact, Inc.
All At Once Planning, cont’d
• At the end of the meeting, each team has their planned deliverables and incoming dependencies.
• If they haven’t already, they determine their capacity and, based on the priorities, commit to a set of work.
• This means that a team may have capacity to do work, but may not get to it in a release if that work pushes the operation team beyond its capacity.
• Once the iteration starts, we will have each team’s set of commitments.
30Copyright © 2011 Constant Contact, Inc.
Classes of Service
• Inevitably, the operations team can often become a bottleneck for development.
• We attempt to manage this through classes of service. We borrow a concept from Kanban that says similar projects are grouped into classes and each class is assigned an allocation.
• For example, we may decide that 20% of ops time should be spent on infrastructure improvements, and 80% spent on servicing development
31Copyright © 2011 Constant Contact, Inc.
(Near) Continuous integration and (mostly)automated regression testing
• It is important that builds, deployments and testing are as automated and continuous as possible. Tools we use:
• Hudson for continuous integration
• Puppet for automated deployments
• Maven for managed builds
• Selenium for automated testing
• svn for source control and versioning
• We still have a lot of work to do in this area
32Copyright © 2011 Constant Contact, Inc.
Dependency and deliverable review
• We have a daily meeting, about 20 minutes, long in which we review each project, not each team.
• Project leads review the deliverables and dependencies to which they have committed and say if they are on track or not.
33Copyright © 2011 Constant Contact, Inc.
Dependency and deliverable review, cont’d
• We found the teams could still miss a deliverable even if they had no impediments. Deliverable tracking provides a better view of the state of the iteration
• However, we still call the review meeting “Scrum of Scrums” because the name stuck!
34Copyright © 2011 Constant Contact, Inc.
Release Train
• Releases are iterative but development is continuous
35Copyright © 2011 Constant Contact, Inc.
Release Train, cont’d
• Scrum aims to make development iterative but this causes problems:
• How do you handle all the testing at the end of the sprint? What if defects are found a day before the sprint ends?
• If a deliverable misses a release (the train), it simply waits to capture the next one.
• To be reiterate (and be explicit): we don’t penalize a team if a deliverable is not done at the end of a release and misses the release train
36Copyright © 2011 Constant Contact, Inc.
Release Train, cont’d
• To make this work:
• Releases must be short
• Must have those tools for automated deployments
• Marketing and support must be flexible enough to react to last-minute product changes
37Copyright © 2011 Constant Contact, Inc.
Kanban at the team-level
• Teams plan continuously
• Teams test continuously
• It’s OK if a team finds a defect on the last day of the release.
• It’s OK if a team starts work for the next release in the current release
• Development and testing can flow more smoothly, because we do not want the end of an iteration to feel like this:
38Copyright © 2011 Constant Contact, Inc.
Kanban at the team-level
39Copyright © 2011 Constant Contact, Inc.
Metrics
• We gather for each team:
• Cycle time on items after grouping them by size:
• Completion time for small, medium and large
• Spread of cycle times
• Work items completed
• Open defects in production, to give a gross measure of technical debt
40Copyright © 2011 Constant Contact, Inc.
Metrics guide planning and estimation
• Over time, we expect that the spread of cycle times for a given item size goes down.
• So, over time, an estimate of completion time for items of a given size should become more accurate.
• We have eliminated planning poker. Work items are just sized as smalls or mediums and the average cycle times for those sizes from the last release become the estimate for the upcoming release.
• Large items are broken down in smaller items
41Copyright © 2011 Constant Contact, Inc.
2010 R7 2010 R8 2011 R1 2011 R2 2011 R3 2011 R40
5
10
15
20
25
30
35Average Cycle Times for work items
Average of Cycle Time (small - 1 Story Point)
Average of Cycle Time (medium - 3 Story Points)
Average of Cycle Time (large - 5 Story Points)
42Copyright © 2011 Constant Contact, Inc.
How did we do?
• Releases used to take 12 weeks. Now they take 5 weeks.
• More importantly, testing used to take 6 weeks. Now it takes 1 week. Testing used to be 50% of the release cycle, but now is just 20%.
• We have a better picture of our release at any given moment
• A year ago, CTCT was not in the social media space. Now our service has ties to Facebook, Twitter and LinkedIn. We are becoming a leader in the social media marketing space.
43Copyright © 2011 Constant Contact, Inc.
How did we do?
44Copyright © 2011 Constant Contact, Inc.
Time for Questions
Top Related