Real Time Process Improvement with ScrumTechnique for Software Development. IEEE Transactions on...
Transcript of Real Time Process Improvement with ScrumTechnique for Software Development. IEEE Transactions on...
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Real Time Process Improvement WithSCRUM: Theory and Background
Real Time Process Improvement WithSCRUM: Theory and Background
Jeff Sutherland, Ph.D.Chief Technology Officer
http://jeffsutherland.com
Jeff Sutherland, Ph.D.Chief Technology Officer
http://jeffsutherland.com
We Make Healthcare MobileTM
© jeffsutherland.com 2004
The Zen of SCRUMSo simple, anyone can implement it!So easy, all can benefit!So subtle, few achieve transcendent performance …How is it that a project manager does nothing, andachieves everything?Interlocking principles emerge product like jigsaw puzzle.Novice removes one piece -- engine never fires on allcylinders …Who can know why?SCRUMmaster must understand deeply and practicerigorously.Only then will team members say, “This experiencechanged my life!”
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Complex Adaptive Systems (cas)Interacting agents respond to stimuli.Stimulus-response behavior is defined in terms of rules.Agents adapt by changing rules as experience accumulates.Agents aggregate into meta-agents whose behavior is emergent.How can a collection of dumb things emerge smart system behavior?
Maamar, Zakaria and Sutherland, Jeff (2000) Toward Intelligent Business Objects: Focusing onTechniques to Enhance Business Objects that Exhibit Goal-Oriented Behaviors. Communicationsof the ACM 40:10:99-101.
Frozen
ChaosFragmentation
casSelf Organization
Web services?
1998 Agents1995 Components1993 Business Objects
1980 Classes1970 Procedures
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Enterprise Systems are cas
Business entities are examples of complex adaptivesystems.Modification time is on the order of months or years,roughly time required to change software.Automating business processes renders parts of thebusiness in software.Business systems have severely constrained rulesets, making ideal test bed for cas concepts.
Sutherland, Jeff and van den Heuvel, Willem-Jan (2002) Developing and integratingenterprise components and services: Enterprise application integration and complexadaptive systems. Communications of the ACM 45:10:59-64.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Objects Meet Requirementsfor Evolution
Variation: there is a continuing abundance ofdifferent elements (class libraries).Heredity or replication: the elements have thecapacity to create copies or replicas of themselves(inheritance).Differential "fitness": the number of copies of anelement that are created in a given time varies,depending on interactions between the features ofthat element and the features of the environment inwhich it persists (reuse).
Daniel Dennett. Darwin's Dangerous Idea: Evolution and the Meanings of Life. Simon andSchuster, 1995, p. 343.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Do Programmers Meet EvolutionaryRequirements?
Algorithms create movement through design space– Simple minded repetitive procedure– Incremental change to adjacent designs
"Cranes" accelerate movement through design space– Sex and genetic engineers– Evolutionary prototyping and smart people– Emergent architecture
Brooks, Fred. No Silver Bullet: Essence and Accidents of Software Engineering. IEEEComputer, Vol. 20, No. 4, April 1987, pp. 10-19.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Change is Imperative:Wasserman's 7 Factors Driving Change
Criticality of time to marketShift in computing economicsPowerful desktop computersExtensive networks and the WebGrowing availability of object technologyWIMP (windows, icons, menus, pointers)Unpredictability of the waterfall model of softwaredevelopment
Tony Wasserman. Toward a Discipline of Software Engineering. IEEE Software 13:6:23-31, Nov 1996.
Too many projects fail
We Make Healthcare MobileTM
© jeffsutherland.com 2004
"Why Are Systems Late, Over Budget, Wrong?""The Waterfall Methodology!" (Paul Bassett)
Analysis Paralysis– static modeling overused– specs are stale bakedDesign-from-Scratch– no generic models– no standard architecturesLarge Project TeamsUser IntermediariesNo Early Warning Signals
Bassett, Paul G. Framing Software Reuse: Lessons from the Real World. YourdonPress Computing Series, 1997.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Wicked Problems: Righteous SolutionsOut of a total cost of $37B for the sample set, 75% of [DOD] projects failed or were neverused, and only 2% were used without extensive modification. Jarzombek. The 5th Annual JAWSS3 Proceedings, 1999.
Wicked problems have no definitive formulation. Each attempt at creating a solutionchanges your understanding of the problem.No stopping rule. The problem-solving process ends when resources are depleted,stakeholders lose interest or political realities change.Solutions are not true-or-false, but good-or-bad … getting all stakeholders to agreethat a resolution is "good enough" can be a challenge.No immediate or ultimate test of a solution.Every implemented solution has consequences.No well-described set of potential solutions. Various stakeholders have differingviews of acceptable solutions.Each problem is essentially unique. Part of the art of dealing with wicked problemsis the art of not knowing too early what type of solution to apply.Each problem can be considered a symptom of another problem. A wicked problemis a set of interlocking issues and constraints that change over time, embedded in adynamic social context.The causes of a wicked problem can be explained in numerous ways.The planner (designer) has no right to be wrong.
Rittel, H and Webber M. Dilemmas in a General Theory of Planning. Policy Sciences, Vol. 4. Elsevier, 1973.Degrace and Hulet's book, Wicked Problems, Righteous Solutions, Prentice Hall, 1990
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Software Development is an EmpiricalProcess
Ziv's Uncertainty Principle in Software Engineering -uncertainty is inherent and inevitable in softwaredevelopment processes and products [Ziv, 1996].
Humphrey's Requirements Uncertainty Principle - fora new software system, the requirements will not becompletely known until after the users have used it.Wegner's Lemma - it is not possible to completelyspecify an interactive system [Wegner, 1995].
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Productivity: All at Once ModelsSingle Super-ProgrammerHandcuffing two programmerstogetherBrook’s Surgical TeamBorland Quattro projectGoldratt’s “The Goal”Senge’s systems thinkingHolland’s complex adaptive systems
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Team Size: Development Effort inMonths
The smaller the better.491 medium sized projects with 35,000-95,000 SLOC (sourcelines of code)Putnam, Lawrence H. and Myers, Ware. Familiar Metrics Management: Small is Beautiful--Once Again. IT Metrics Strategis IV:8:12-16, Cutter Information Corp., August 1998.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Team Size: Development Time inMonths
Sweet spot is 5-7 people491 medium sized projects with 35,000-95,000 SLOC(source lines of code)Putnam, Lawrence H. and Myers, Ware. Familiar Metrics Management: Small is Beautiful--Once Again. IT Metrics Strategis IV:8:12-16, Cutter Information Corp., August 1998.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Bell Labs Report on most productiveproject ever: Borland Quattro for Windows
277Function pointsper staff month
>1008Staff
>5031Time in months
Industry standardBWP1,000,000 lines ofC++ code
BQW
Jones, Capers. AppliedSoftware Measurement, SecondEdition. McGraw Hill, 1997.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
James Coplien. Borland Software Craftsmanship: ANew Look at Process, Quality, and Productivity.Proceedings of the 5th Annual Borland InternationalConference, Orlando, 1994.
One of most remarkable organizations, processes, anddevelopment cultures seen in AT&T Bell Laboratories Pasteurprocess research projectProject management, product management, QA integral to team,all making technical contributionsHigher communication saturation than 89% of projectsMore even distribution of workload“Anti-schismogenetic” – no cliquesHighly iterative developmentStrong architectural interaction with implementationMore time spent in project team meetings than anything else –several hours a dayGerry Weinberg notes that CMM Level 1 and 2 teams need strongmanagerial direction. Level 3 paradigm shift is self-directing team.Borland team was clearly in this category, although not bycommonly accepted criteria.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Team comments on Quattro project
“We are satisfied by doing real work.”“Software is like a plant that grows. You can’tpredict its exact shape, or how big it willgrow.”“There are no rules for this kind of thing—it’snever been done before.”
“Evolutionary development is best technically, and it saves timeand money.”Report of the Defense Science Board Task Force on Military Software. Oct 1987.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Iterative and IncrementalDevelopment: A Brief History
1956 – Benington’s stagewise model – USAF SAGE System1957 – IBM Service Bureau Corp, Project Mercury, IBM FederalSystems Devision – Gerry Weinberg1960 – Weinberg teaching IID at IBM Systems ResearchInstitute1969 - Earliest published reference to IID:
– Robert Glass. Elementary Level Discussion ofCompiler/Interpreter Writing. ACM Computing Surveys, Mar 1969
Larman, Craig and Basili, Vic. Iterative and Incremental Development: A BriefHistory. IEEE Computer, June 2003.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Iterative and Incremental Development:A Brief History
1971 – IBM Federal Systems Division– Mills, Harlan. Top-down programming in Large Systems. In
Debugging Techniques in Large Systems. Prentice Hall, 19711972 – TRW uses IID on $100M Army Site Defense software1975 – First original paper devoted to IID
– Gasili, Vic and Turner, Albert. Iterative Enhancement: A PracticalTechnique for Software Development. IEEE Transactions onSoftware Engineering. Dec 1975.
1977-1980 – IBM FSD builds NASA Space Shuttle software in17 iterations over 31 months, averaging 8 weeks per iteration
– Madden and Rone. Design, Development, Integration: Space ShuttleFlight Software. Communications of the ACM, Sept 1984.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Iterative and Incremental Development:A Brief History
1985 – Barry Boehm’s Spiral Model– Boehm, Barry. A Spiral Model of Software Development and Enhancement.
Proceedings of an International Workshop on Software Process and SoftwareEnvironments. March, 1985
1986 – Brooks, Fred. No Silver Bullet. IEEE Computer, April 1987– Nothing … has so radically changed my own practice, or its effectiveness [as
incremental development].1994 – First SCRUM at Easel Corporation
1994 – DOD must manage programs using iterative development– Report of the Defense Science Board Task Force on Acquiring Defense
Software Commercially. June 1994.1995 – Microsoft IID published
– McCarthy, Jim. Dynamics of Software Development. Microsoft Press, 1995.1996 – Kruchten. A Rational Development Process. Crosstalk. July.
– Origins of RUP
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Iterative and IncrementalDevelopment: A Brief History
1996 – Kent Beck Chrysler Project– Origin of XP
1996 – Larman meets with principal author of DD-STD-2167– David Maibor expressed regret for the creation of the waterfall-based
standard. He had not learned of incremental development at the time andbased his advice on textbooks and consultants advocating the waterfallmethod. With the hindsight of iterative experience, he would recommend IID.
1997 – Coad and DeLuca rescue Singapore project– Origin of Feature-Driven Development
1998 – Standish Group CHAOS Project– Top reason for massive project failures was waterfall methods. “Research
also indicates that smaller time frames, with delivery of software componentsearly and often, will increase success rate.
1999 – Publication of extensive DOD failures– Out of a total cost of $37B for the sample set, 75% of projects failed or were
never used, and only 2% were used without extensive modification.Jarzombek. The 5th Annual JAWS S3 Proceedings, 1999.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Iterative and Incremental Development:A Brief History
2001 – 17 process expert “anarchists” meet atSnow Bird
– Agile Manifesto initiated 100s of books and papers on agiledevelopment
2001 – MacCormack’s study of key success factors– MacCormack, Alan. Product-Develoment Practices that Work.
MIT Sloan Management Review 42:2, 2001.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Manifesto for Agile SoftwareDevelopment
We are uncovering better ways of developing software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools Working software over comprehensive documentation
Customer collaboration over contract negotiation Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
MacCormack Process EvolutionWaterfall model – sequential process maintains adocument trailRapid-Prototyping Model – disposable prototype helpsestablish customer preferenceSpiral Model – series of prototypes identifies majorrisksIncremental or Staged Delivery Model – system isdelivered to customer in chunksEvolutionary Delivery Model – iterative approach inwhich customers test an actual version of the softwareMacCormack, Alan. Product-Development Practices That Work: How Internet Companies BuildSoftware. MIT Sloan Management Review 42:2:75-84, Winter 2001.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
MacCormack Success Factors
Early release of evolving product design to customers.Daily incorporation of new software code and rapidfeedback on design changesA team with broad-based experience in shippingmultiple projectsMajor investment in design of product architecture
MacCormack, Alan. Product-Development Practices That Work: How Internet Companies BuildSoftware. MIT Sloan Management Review 42:2:75-84, Winter 2001.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
SCRUM Origins: Takeuchi and NonakaLessons from Fuji-Xerox, Canon, Honda, NEC, Epson, Brother,3M, Xerox, HP
Old model – Relay Race– Speed and flexibility not adequate in today’s market
New model - Rugby
Takeuchi, Hirotaka and Nonaka, Ikujiro. 1986. The new new product developmentgame. Harvard Business Review 64:1:137-146 (Jan/Feb), reprint no. 86116.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Moving the SCRUM downfield
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Takeuchi and Nonaka Success Factors
Built-in instabilitySelf-organizing project teamsOverlapping development phases“Multilearning”Subtle controlOrganizational transfer of learning“These characteristics are like pieces of a jigsaw puzzle. Each element, by itself,does not bring about speed and flexibility. But taken as a whole, the characteristicscan product a powerful new set of dynamics that will make a difference.”
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Factor 1: Built-in instabilityTop management kicks off development process by signalingbroad goal.Project team is offered extremely challenging goals with widemeasure of freedom.
– Example: Fuji-Xerox gave FX-3500 project team two years to comeup with a copier that cut costs in half
Top management creates an element of tension in the projectteam through challenging requirements with wide freedom toachieve strategic objective.
– Honda Executive: “It’s like putting the team members on the secondfloor, removing the ladder, and telling them to jump or else. Ibelieve creativity is born by pushing people against the wall andpressuring them almost to the extreme.”
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Factor 2: Self-organizing project teamsA project team takes on a self-organizing character as it is drivento a state of “zero information” – where prior knowledge does notapply.Left to stew, the process begins to create its own dynamic order.The project team begins to operate like a start-up company.A group possesses a self-organizing capability when it exhibitsthree conditions.
– Autonomy– Self-transcendence– Cross-fertilization
At some point, the team beginsto create its own concept.
ScrumDown at the Radcliffe Rugby Club
We Make Healthcare MobileTM
© jeffsutherland.com 2004
AutonomyHeadquarters involvement is limited to providingguidance, money, and moral support at the outset.On a day to day basis, management seldom intervenesand the team is free to set its own direction.In a way, top management acts as a venture capitalist
– “We open our purse and keep our mouth closed.”
Example: IBM development of personal computerExample: Honda City project team, average age 27,“Develop the kind of car that the youth segment would like todrive.”
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Self-transcendenceThe project teams appear to be absorbed in the never-ending quest for “the limit.”They elevate their goals through the developmentprocess.By pursuing what appear to be contradictory goals,they devise ways tooverride the status quo andmake the big discovery.Example: Canon AE-1 team
Flyhalf Sarah Schooler of Radcliffe fending off Boston College
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Cross-fertilizationTeam with wide variety of specializations, thoughtprocesses, and behavior patterns carries out newproduct development.Working in one large room is best (Fuji-Xerox).“When all team membersare in one room, othersinformation becomes yourswithout even trying.”
Radcliffe Rugby Football Club
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Factor 3: Overlapping DevelopmentPhases
Self-organizing character of the team produces uniquedynamic or rhythmSashimi system – Fuji Xerox Rugby system – HondaHard merits (demerits)
– Speed and flexibility (watch out for muck and mall)
Soft merits– Share responsibility and cooperation– Stimulates involvement and commitment– Sharpens a problem-solving focus– Develops initiative and diversified skills– Heightens sensitivity to market conditions
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Factor 4: MultilearningLearning by doing in two dimensions
– Across organization– Across specialty
Enhanced learning opportunities– 15% of time devoted to “dreams” – 3M– Peer pressure to study– Send team to Europe to look around – Honda– Bring in top academics and consultants – HP
Everyone learns multiple skills
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Factor 5: Subtle ControlManagement establishes checkpoints
– Prevents instability, ambiguity, and tension from turning intochaos
Emphasis on self-control, control by peer pressure,control by love = “subtle control”Management responsible for:
– Selecting team members for balanced team– Creating an open working environment– Encouraging engineers to go out in the field– Establishing rewards based on group performance– Tolerating and anticipating mistakes– Encouraging suppliers to become self-organizing
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Factor 6: Organizational Transfer of LearningTransfer knowledge outside group
– Scatter successful team to new projects– Institutionalize practice (monthly demos at IDX)
Consciously pursue unlearning– Next generation must be 40% better– Cut product cycle by 80%– Scrap old parts, processes, tools
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Challenges and OpportunitiesWinding the Rubber Band Principle: Broad mandate anddemanding goals create tension.Anti-Waterfall Principle: Operational decisions are madeincrementally. Strategic decisions delayed to last moment.Push/Pull Principle: Differentiation in concept phase,integration dominates in implementation phaseSpread the Wealth Principle: Non-experts take on newtasks.Cuckoo Principle: Successful SCRUMs become companymodels (or they can get crushed because they are different).Control Anti-Pattern: Seniority based companies havedifficult time.
– But in times of desperation, SCRUMs are easily created.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Spiral Methodology
Barry Boehm introduced the Spiral Methodology to "fix" problemswith the Waterfall Methodology. This is the most commonly usedvariant of the Waterfall today.The Spiral methodology "peels the onion", progressing through"layers" of the development process. A prototype lets usersdetermine if the project is on track, should be sent back to priorphases, or should be ended. However, the phases and phaseprocesses are still linear.
Boehm, B.W. A Spiral Model of Software Development and Enhancement.Proceedings of an International Workshop on Software Process and SoftwareEnvironments, Coto de Caza, Trabuco Canyon, California, March 27-29, 1985.Boehm, Barry. A Spiral Model of Software Development and Enhancement. ACMSIGSOFT Software Engineering Notes, August 1986.Boehm, Barry. A Spiral Model of Software Development and Enhancement. IEEEComputer, vol.21, #5, May 1988, pp 61-72.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Iterative Methology
The Iterative methodology improves on the Spiralmethodology.Each iteration consists of all of the standard Waterfall phases,but each iteration only addresses one subsystem. Furtheriterations can add resources to the project while ramping upthe speed of delivery.Improves cost control, reduces risk, ensures delivery of(sub)systems, and improves overall flexibility.Still assumes that the underlying development processes aredefined and linear.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
SCRUMMethodology
The first and last phases (Planning and Closure) consist ofdefined processes.The Sprint phase is an empirical process. It is treated as ablack box that requires external controls.Sprints are nonlinear and flexible. Sprints are used to evolvethe final product.The project is open to the environment until the Closurephase. The deliverable can be changed at any time.The deliverable is determined during the project based on theenvironment.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
SCRUM Flow Diagram
Ken Schwaber. www.controlchaos.com
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Methodology Comparison
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Risk with CurrentMethodologies
Any methodology is better thannothing.Current approaches rests on thefallacy that the developmentprocesses are defined, predictableprocesses.They lack flexibility needed tocope with the unpredictableresults and respond to a complexenvironment.
Schwaber, Ken. SCRUM Development Process.Business Object Design and Implementation(Eds. Jeff Sutherland et al.). London: Springer-Verlag, 1997.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
SCRUM LowersRisk
Development teams need to operate adaptivelywithin a complex environment using impreciseprocesses.SCRUM can accelerate closure by inducing thephenomenon known as "punctuated equilibrium"seen in the evolution of biological species.Levy, Steven. Artificial Life: A Report from the Frontier Where Computers MeetBiology. Vintage Books, 1993.Lewin, Roger. Complexity: Life at the Edge of Chaos. Collier Books, 1994.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Mike Cohn’s Java SCRUM:more functionality, better quality
5400050,803Lines of Java
912Months
604.5People
WaterfallSCRUM
[email protected]://www.mountaingoatsoftware.com/
BQW
SCRUM Waterfall
We Make Healthcare MobileTM
© jeffsutherland.com 2004
SCRUM SummaryScrum is an iterative, incremental process for developing any product or
managing any work. It produces a potentially shippable set of functionality atthe end of every iteration. It's attributes are:an agile process to manage and control development work.a wrapper for existing engineering practices.a team-based approach to iteratively, incrementally develop systems and productswhen requirements are rapidly changinga process that controls the chaos of conflicting interests and needs.a way to improve communications and maximize co-operation.a way to detect and cause the removal of anything that gets in the way ofdeveloping and delivering products.a way to maximize productivity.is scalable from single projects to entire organizations. Has controlled andorganized development and implementation for multiple interrelated products andprojects with over a thousand developers and implementers.is a way for everyone to feel good about their job, their contributions, and that theyhave done the very best they possibly could.is a pattern.
Beedle, Mike; Devos, Martine; Sharon, Yonat; Schwaber, Ken; Sutherland, Jeff (1999) SCRUM: An extensionpattern language for hyperproductive software development. In Harrison, Neil; Foote, Brian; Ronhert, Hans(Eds.) Pattern Languages of Program Design 4. Addison-Wesley Software Patterns Series.
We Make Healthcare MobileTM
© jeffsutherland.com 2004
Thank YouPatientKeeper, Inc.
Brighton Landing East20 Guest Street, Fifth Floor
617-987-0300www.patientkeeper.com