Barry Boehm University of Southern California Richard Turner George Washington University Based on...

Barry BoehmUniversity of Southern California

Richard Turner George Washington University

Based on Balancing Agility and Discipline: A Guide for the Perplexed,

B. Boehm and R. Turner, Addison Wesley, 2004

ICSE 2004 TutorialMay 25, 2004

Balancing Agility and Balancing Agility and Discipline:Discipline:

Evaluating and Integrating Agile and Evaluating and Integrating Agile and Plan-Driven MethodsPlan-Driven Methods

5/25/04 2

Outline

• Tutorial learning objectives– Survey of assumptions about participants

• General software trends and implications– Sources of perplexity about agile, plan-driven

methods• Overview of agile and plan-driven methods

– XP, TSP “days in the life”– Comparisons of differences, strengths,

weaknesses– Two case studies

• Risk-based balance of agility and discipline– Small, medium, large examples

• Observations, current work and a way forward– Hands-on exercise

• Conclusions; review of learning objectives

5/25/04 3

Tutorial Learning Objectives

• Practitioners and managers– Understand agile, plan-driven method

strengths, difficulties– Learn risk-based approach to tailor hybrid

methods– Practice in formulating organizational strategy

• Researchers – Understand research issues, opportunities

• Techniques for achieving rapid quality• Empirical analyses

• Educators– Understand differences in agile, plan-driven

approaches– Understand teaching challenges, opportunities

• Survey courses, project courses

5/25/04 4

Assumptions About Participants - Surveyed at Tutorial

• Generally familiar with plan-driven methods– Waterfall, incremental, spiral, RUP, PSP/TSP– Software CMM, CMMI, SPICE, ISO I2207

• Less familiar with agile methods– XP, Scrum, Crystal, DSDM, FDD

• Representative of mixed domains– Large/small projects, research, education– Variety of software applications domains

• Increasing need for high software dependability

• Increasing software complexity, speed of change

5/25/04 5

Discipline and Agility

• Discipline is the foundation for any continuously successful endeavor– Ingrains and strengthens– Training, practice, honing skills– Extends and supports natural talent– Supports dealing with adverse environments– Creates history and experience

• Agility is Discipline’s counterpart– Releases and invents– Improvisation, evolution, embrace the

unexpected– Takes advantage of insight, natural talent– Applies history and experience in new ways

• Success requires both – in balance

5/25/04 6

Collins’ Good to Great- HarperBusiness, 2001

Bureaucratic Organization

Start-up Organization

Hierarchical Organization

Great Organization

Low

Low

High

High

Ethic of Entrepreneurship

Culture of Discipline

5/25/04 7

Information Technology Trends

Traditional Development

• Standalone systems

• Stable requirements

• Rqts. determine capabilities

• Control over evolution

• Enough time to keep stable

• Stable jobs

• Failures noncritical

• Reductionist systems

• Repeatability-oriented process, maturity models

Current/Future Trends

• Everything connected (maybe)

• Rapid requirements change

• COTS capabilities determine

rqts.• No control over COTS evolution

• Ever-decreasing cycle times

• Outsourced jobs

• Failures critical

• Complex, adaptive, emergent

systems of systems• Adaptive process models

5/25/04 8

Background

• Two approaches to software development – Disciplined (SW-CMM, document-based,

heavy process)– Agile (XP, tacit knowledge, light process)

• Both have strengths and weaknesses• Agile and disciplined proponents are

believers• Many of us are perplexed

5/25/04 9

Key Definitions

• Agile method – – one which fully adopts the four value propositions

and twelve principles in the Agile Manifesto.• Discipline – (per Webster)

– control gained by enforcing obedience or order; – orderly or prescribed conduct or pattern of

behavior; – self-control.

• Plan-driven – – a description for disciplined methods (order is

often defined in plans) • Plan – (per Webster)

– a method for achieving an end (a process plan); – an orderly arrangement of parts of an overall

design (a product plan). – We assume that such plans are documented.

5/25/04 10

The Agile Manifesto

• Individuals and interactionsIndividuals and interactions over processes and tools• Working softwareWorking software over comprehensive documentation• Customer collaborationCustomer collaboration over contract negotiation• Responding to changeResponding to change over following a plan

We are uncovering better ways of developing

software by doing it and helping others do it.

Through this work we have come to value:

That is, while there is value in the items on

the right, we value the items on the left more.

5/25/04 11

Plan-Driven Methods Are Evolving Too

• Evolutionary acquisition and spiral development– U.S. DoD Directive 5000.1, Instruction 5000.2

• Risk-driven level of detail– Of processes: peer reviews vs. Critical Design

Reviews– Of products: GUI prototypes vs. specifications– Of heavyweight methods: Team Software

Process, Rational Unified Process

5/25/04 12

Sources of Perplexity

• Distinguishing method use from method misuse• Citing method misuse vs. intended use

– Claiming XP use when simply “not documenting”– “CMM Level 4 Memorial Library” of 99 2-inch binders

• Overgeneralization based on the most visible instances– XP is Agile; CMM is Plan-Driven

• Multiple definitions– Quality: customer satisfaction or compliance?

• Claims of universality– The pace of IT change is accelerating and Agile

methods adapt to change better than disciplined methods therefore Agile methods will take over the IT world.

– Software development is uncertain and the SW-CMM improves predictability therefore all software developers should use the SW-CMM

5/25/04 13

Sources of Perplexity (2)

• Early success stories– Chrysler project that successfully birthed XP was

later cancelled– Cleanroom has never made it into the

mainstream

• Purist interpretations (and internal disagreements)– “Don't start by incrementally adopting parts of

XP. Its pieces fit together like a fine Swiss watch“

– "An advantage of agile methods is that you can apply them selectively and generatively."

– "If you aren't 100% compliant with SW CMM Level 3, don't bother to bid"

– "With the CMMI continuous interpretation, you can improve your processes in any order you feel is best."

5/25/04 14

Outline









5/25/04 15

Various Agile Methods Available

• Adaptive Software Development - Highsmith

• Crystal - Cockburn• Dynamic System Development

Methodology - DSDM Consortium• eXtreme Programming - Beck,

Cunningham, Jeffries• Feature Driven Development - Coad,

DeLuca• Lean Development – Charette,

Poppendieck• Scrum - Schwaber, Sutherland

5/25/04 16

Extreme Programming (XP)

• Beck, Cunningham and Jeffries at Chrysler– Chrysler Comprehensive Compensation

program– Traditional methods failed to meet corporate

needs

• Based on numerous best practices taken to the “extreme”

• Organization and methodology help maximize practice value

• Most often cited as exemplary of “agile methods”

5/25/04 17

XP Principles – I

• Fundamental values:– Communication– Simplicity– Feedback– Courage

• Good practices pushed to extremes– “If code reviews are good, review code all the

time”– “If testing is good, test all the time”– “If design is good, make it part of everybody’s

daily business”– “If simplicity is good, always leave the system

with the simplest design that supports its current functionality”

– “If architecture is important, everybody will work defining and refining the architecture all the time”

5/25/04 18

XP Principles – II

• Good practices pushed to extremes (2)– “If integration testing is important, then

integrate and test several times a day”– “If short iterations are good, make the

iterations really, really short – seconds and minutes and hours, not weeks and months and years”

– “If customer involvement is good, make them full-time participants”

5/25/04 19

XP: The 12 Practices-Kent Beck, Extreme Programming Explained, Addison Wesley, 1999

• The Planning Game

• Small Releases• Metaphor• Simple Design• Testing• Refactoring

• Pair Programming• Collective Ownership• Continuous

Integration• 40-hour Week• On-site Customer• Coding Standards

Sometimes used generatively; Sometimes used imperatively

5/25/04 20

XP Characteristics

5/25/04 21

The Team Software Process (TSP)-Watts Humphrey, Introduction to the ISP, Addison Wesley, 2000

• Scale-up of Personal Software Process (PSP)– To about 20-person teams

• Strong emphasis on planning and control– 21 scripts, 5 roles/activities, 21 forms

• Risk-driven tailoring can make process more agile

5/25/04 22

Example TSP Role Script

5/25/04 23

TSP Characteristics

5/25/04 24

Days In The Life: XP and TSP

• Comparison of two development teams• Product:

– Tool to process complex sales report generated by legacy system

– Prototype delivered– Task is to enhance and port prototype – 8 month projected schedule– Estimated size: 20,000 SLOC

5/25/04 25

Background Information

TSP/PSP• Training

– Each member has 125-hour PSP for Engineers

– 2 Members 5-day launch coach training

• Tools and environment– Web-based TSP data

collection tool– OO tools– Individual offices and

conference room• Project planning

– 4-day launch session– Develop/tailored all

processes– 180 tasks identified and

plan accepted by stakeholders

XP• Training

– 2 members attended 40-hour XP workshop

– In-house presentations and mentoring

• Tools and environment– Automated test suite

development tool– OO tools– Bullpen and Conference

room/lounge• Project planning

– 1-day customer exploration– 2-day follow-up to prioritize– Agreed on 2-week iteration

length, 3-iteration release length, and release schedule

5/25/04 26

Normal and Crisis Days

• Used Friday as common day• Crisis days are plan-breakers

– New requirements– Short schedule

• Please read the handouts and we’ll discuss

5/25/04 27

Comparison

• Differences– Depth and scope of metrics– Specificity of process– Level and scope of reporting– Customer interface

• Similarities– Collaborative teams– Well-defined roles– Spiral, risk- and increment-driven process– Measurement– Test first (test-driven) approach

• Bottom Line– Both were able to confidently push-back

when necessary

5/25/04 28

Outline









5/25/04 29

Finding Middle Ground

• A pragmatic view• Both approaches have “home grounds”• A broad range of environments and

needs• Trends require better handling of rapid

change• Processes should be the right weight for

the job• We believe risk-based analysis is the key

to finding middle ground

5/25/04 30

Contrasts and Home Grounds

• Comparison is difficult, but we found 4 areas where there are clear differences

• Application– Project goals, size, environment; velocity

• Management– Customer relations, planning and control,

communications

• Technical– How the software is developed

• Personnel– The type and competency of developers and

stakeholders

5/25/04 31

Application Characteristics

• Primary goals– A: Rapid value, responsiveness to change– P: Predictability, stability, high assurance

• Size– A: Small to medium– P: Large

• Environment– A: Turbulent, high change– P: Stable, requirements defined

• Project velocity– A: Code, learn, and fix– P: Architect for parallel, incremental

development

5/25/04 32

Management Characteristics

• Customer Relations– A: Dedicated, collocated; where feasible– P: Contractual; increasingly evolutionary– A: Trust through working software and

participation– P: Trust through process maturity evaluations

• Planning and Control– A: Means to an end– P: Anchor processes, communication

• Project Communications– A: Tacit, interpersonal knowledge– P: Explicit, documented knowledge

5/25/04 33

Technical Characteristics

• Requirements– A: Adjustable, informal stories– P: Formally baselined, complete, consistent

specifications

• Development– A: Simple Design– P: Architecture-based design

• Testing– A: Automated, test-driven– P: Planned, requirements-driven

5/25/04 34

Technical Characteristics (2)Cost of Change: Beck, Li

Cos

t of

Cha

nge

Time

Beck

LiLi

5/25/04 35

Technical Characteristics (3) Cost of change: Poppendieck

• At least two cost-escalation curves• Lean development: Architect to defer

high-stakes decisions – e.g. COTS• Easier to change an unmade decision

Most Changes

Changes in High-stakes Constraints

Time

Co

st o

f C

han

ge

5/25/04 36

Technical Characteristics (4)Cost of Change: TRW

Beck

Li

Steeper Cost-to-fix for High-Risk Elements

0102030405060708090

100

0 10 20 30 40 50 60 70 80 90 100

% of Software Problem Reports (SPR’s)

TRW Project A373 SPR’s

TRW Project B1005 SPR’s

% ofCosttoFixSPR’s

Major Rework Sources:Off-Nominal Architecture-BreakersA - Network FailoverB - Extra-Long Messages

Steeper Cost-to-fix for High-Risk Elements

0102030405060708090

100

0 10 20 30 40 50 60 70 80 90 100

% of Software Problem Reports (SPR’s)

TRW Project A373 SPR’s

TRW Project B1005 SPR’s

% ofCosttoFixSPR’s

Major Rework Sources:Off-Nominal Architecture-BreakersA - Network FailoverB - Extra-Long Messages

5/25/04 37

Technical Characteristics (5)How Much Architecting?

Beck

Liu

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60

Percent of Time Added for Architecture and Risk Resolution

Per

cen

t of T

ime

Ad

ded

to O

vera

ll S

ched

ule

Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution

Added Schedule Devoted to Rework(COCOMO II RESL factor)

Total % Added Schedule

Sweet Spot Drivers:

Rapid Change: leftward

High Assurance: rightward

Sweet spot

10 KSLOC

100 KSLOC

10,000 KSLOC

5/25/04 38

Personnel Characteristics

• Customers– A: CRACK customers throughout development– P: CRACK customers early

• CRACK: Collaborative, Representative, Authorized, Committed, and Knowledgeable

• Developers– A: Heavy mix of high caliber throughout– P: Heavy mix early with lower capability later

• Culture– A: Many degrees of freedom (Thrives on

chaos)– P: Clear policies and procedures (Thrives on

order)• Collocated customers are difficult to

find/keep/keep current on either side

5/25/04 39

Personnel Characteristics (2)Modified Cockburn Levels

(Ski

ll, U

nders

tandin

g)

(Formality, Documentation)

(Ski

ll, U

nders

tandin

g)

(Formality, Documentation)

Level Characteristics

3 Able to revise a method (break its rules) to fit an unprecedented new situation

2 Able to tailor a method to fit a precedented new situation

1A With training, able to perform discretionary method steps (e.g., sizing stories to fit increments, composing patterns, compound refactoring, complex COTS integration). With experience can become Level 2.

1B With training, able to perform procedural method steps (e.g. coding a simple method, simple refactoring, following coding standards and CM procedures, running tests). With experience can master some Level 1A skills.

-1 May have technical skills, but unable or unwilling to collaborate or follow shared methods.

5/25/04 40

Summary of Home Grounds

Characteristics Agile Plan-driven

Application

Primary Goals Rapid value; responding to change Predictability, stability, high assurance

Size Smaller teams and projects Larger teams and projects

Environment Turbulent; high change; project-focused Stable; low-change; project/organization focused

Management

Customer Relations

Dedicated on-site customers, where feasible; focused on prioritized increments

As-needed customer interactions; focused on contract provisions; increasingly evolutionary

Planning/Control

Internalized plans; qualitative control Documented plans, quantitative control

Communications

Tacit interpersonal knowledge Explicit documented knowledge

Technical

Requirements Prioritized informal stories and test cases; undergoing unforseeable change

Formalized project, capability, interface, quality, forseeable evolution requirements

Development Simple design; short increments; refactoring assumed inexpensive

Architect for parallel development; longer increments; refactoring assumed expensive

Test Executable test cases define requirements

Documented test plans and procedures

Personnel

Customers Dedicated, collocated CRACK* performers CRACK* performers, not always collocated

Developers At least 30% full-time Cockburn level 2 and 3 experts; no Level 1B or -1 personnel**

50% Cockburn Level 3s early; 10% throughout; 30% Level 1B’s workable; no Level -1s**

Culture Comfort and empowerment via many degrees of freedom (thriving on chaos)

Comfort and empowerment via framework of policies and procedures (thriving on order)

* Collaborative, Representative, Authorized, Committed, Knowledgeable** These numbers will particularly vary with the complexity of the application

5/25/04 41

Outline









5/25/04 42

Two Case Studies

• Show how that agile and plan-driven methods can be extended

• Provide examples of success and lessons learned

• ThoughtWorks Lease Management– Agile extended with plan-driven

• CCPDS-R– Plan-driven overlaid with agile concepts

5/25/04 43

Thoughtworks Lease Management

• XP replaced ineffective traditional development• Problems when project moved beyond XP

assumptions– The effort to develop or modify a story really does not

increase with time and story number– Trusting people to get everything done on time is

compatible with fixed schedules and diseconomies of scale

– Simple design and YAGNI scale up easily to large projects

– • Disciplined practices enabled XP to scale up

– High-level architectural plans to provide essential big-picture information

– More careful definition of milestone completion criteria to avoid “finishing” but not finishing

– Use of design patterns and architectural solutions rather than simple design to handle foreseeable change

5/25/04 44

CCPDS-R

• USAF Command Center Processing and Display System Replacement for early missile warning

• Government contract environment required plan-driven approach, project needed agility

• Practices implemented to provide agility mapped well to the Agile Manifesto – Individuals and Interactions over Processes and

Tools• Milestone content was redefined • Architecture was organized around the performers’ skill

levels – Working Software over Comprehensive

Documentation• Later PDR demonstrated working software for high-risk

areas – Customer Collaboration Over Contract Negotiation

• Used COCOMO for cost and schedule negotiations– Responding to Change Over Following a Plan

• Automated plans• Architecture for re-use saved effort

5/25/04 45

CCPDS-R (2) Cost of Change

RATIONALS o f t w a r e C o r p o r a t I o n

Project Development Schedule15 20 25 30 35 40

40

30

20

10

Design Changes

Implementation Changes

MaintenanceChanges and ECP’s

HoursChange

RATIONALS o f t w a r e C o r p o r a t I o n

Project Development Schedule15 20 25 30 35 40

40

30

20

10

Design Changes

Implementation Changes

MaintenanceChanges and ECP’s

HoursChangeHours

Change

Cos

t of

Cha

nge

Time

Beck

5/25/04 46

Five Critical Decision Factors

• Size, Criticality, Dynamism, Personnel, Culture

Personnel

Dynamism (% Requirements-change/month)

Culture (% thriving on chaos vs. order)

Size (# of personnel)

Criticality (Loss due to impact of defects)

5030

105

1

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15

Essential Funds Discretionary

Funds Comfort

Single Life

Many Lives

(% Level 1B) (% Level 2&3)

0

10

20

30

40

Agile

Plan-driven

Personnel





5030

105

1

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15


Funds Comfort

Single Life

Many Lives


0

10

20

30

40

Agile

Plan-driven

Figure 6-1

5/25/04 47

Comparing the Case Study Projects

LeaseManagement

CCPDS-R

5/25/04 48

Misconceptions

5/25/04 49

Outline









5/25/04 50

Using Risk to Balance Discipline and Agility - Overview

Step 5. Execute and Monitor

Step 4. Tailor Life Cycle

Step 3. Architecture Analysis

Step 1. Risk Analysis

Step 2. Risk Comparison

Rate the project’s environmental, agility-

oriented and plan-driven risks.

Uncertain about

ratings?

Buy information via prototyping, data

collection and analysis

Compare the agile and Plan-

driven risks

Go Risk-based Agile

Agility risks dominate

Plan-driven risks dominate

Architect application to encapsulate agile parts

Go Risk-based Agile in agile

parts; Go Risk-based Plan-

driven elsewhere

Yes

No

Go Risk-based Plan-driven

Tailor life cycle process around risk patterns

and anchor point commitment milestones

Monitor progress and risks/opportunities,

readjust balance and process as appropriate

Neither dominate

Deliver incremental capabilities according to

strategyNote: Feedback loops present, but omitted for

simplicity

Step 5. Execute and Monitor

Step 4. Tailor Life Cycle

Step 3. Architecture Analysis

Step 1. Risk Analysis

Step 2. Risk Comparison

Rate the project’s environmental, agility-

oriented and plan-driven risks.

Uncertain about

ratings?

Buy information via prototyping, data

collection and analysis

Compare the agile and Plan-

driven risks

Go Risk-based Agile

Agility risks dominate

Plan-driven risks dominate

Architect application to encapsulate agile parts

Go Risk-based Agile in agile

parts; Go Risk-based Plan-

driven elsewhere

Yes

No

Go Risk-based Plan-driven

Tailor life cycle process around risk patterns

and anchor point commitment milestones

Monitor progress and risks/opportunities,

readjust balance and process as appropriate

Neither dominate

Deliver incremental capabilities according to

strategyNote: Feedback loops present, but omitted for

simplicity

5/25/04 51

Risks Used in Risk Comparison

• Environmental risks– Technology uncertainties– Many stakeholders– Complex system-of-systems– Legacy compatibility

• Agility risks– Scalability– Use of simple design– Personnel turnover– Too-frequent releases– Not enough agile-capable people

• Plan-driven risks– Rapid change– Need for rapid results– Emergent requirements– Not enough discipline-capable people

5/25/04 52

Three Examples

• Agent-based systems

• Medium – SupplyChain.com application

• Small – Event Managers application

• Large – National Information System for Crisis Management (NISCM) application

• Each example results in a development strategy based on risk analyses

5/25/04 53

SupplyChain.com Profile

• Turnkey agent-based supply chain management systems

• Distributed, multi-organization teams of about 50 people

• Parts of applications are relatively stable, while others are highly volatile

• Architectures are driven by a few key COTS packages that are also continually evolving

5/25/04 54

SupplyChain.com Home Ground Chart

5/25/04 55

SupplyChain.com Risks

5/25/04 56

SupplyChain.com Mitigations

5/25/04 57

SupplyChain.com Strategy

Startup 1. Teambuilding and Shared

Vision

3. Design, Development and Deployment2. Systems Definition and Architecting

Stakeholders

Furnish CRACK representatives and alternates

Staff and organize to cover all success-critical risk areas

• Develop benefits-realization results chains

• Identify missing stakeholders

• Enhance mutual understanding

• Explore goals, options, technology, prototypes, risks

Review; Commit

to proceed

• Elaborate supply chain operational concept, prototypes, transition strategy

• Evaluate and determine COTS, reuse, and architecture choices

• Prioritize and sequence desired capabilities, outstanding risks

• Establish project organization, overall process, and feature teams

Review; Commit

to proceed

• Ensure representative exercise of incremental capabilities

• Monitor progress and new operational development

• Monitor project progress, risk resolution, and new technology developments

• Identify and work critical project-level actions

• Develop and integrate new feature increments

• Communicate proposed redirections

• Prepare for and execute acceptance tests, installation, operational evolution

• Analyze feedback on current version

• Reprioritize features

• Prepare strategy for next increment

• Consolidate process lessons learned

• Use risk to determine content of artifacts

Project Manager andRisk Management Team

Agile Feature Team LCA LCO

5/25/04 58

Event Managers Project

• Small startup company

• Diverse set of smaller agent-based planning applications

• This project: support for managing the infrastructure and operations of conferences and conventions

• Widening variety of options and interdependencies, makes an agent-based approach highly attractive

5/25/04 59

Event Managers Home Ground Chart

5/25/04 60

Event Managers Risks

5/25/04 61

Event Managers Mitigations

5/25/04 62

Event Managers Strategy

1. StartupTeambuilding and Shared

Vision

2. Design, Development and Deployment

• Establish CRACK joint management team

• Develop shared vision, results chain, life cycle strategy, infrastructure choices

• Establish commitment to proceed, incremental capabilities, backlog

• Develop next-increment features

• Test, exercise, deploy new increment• Re-prioritize next-increment features• Analyze lessons learned; prepare strategy for next increment

Customer Management; Representative Users

Joint Developer-Customer Agile Team

LCA

5/25/04 63

NISCM Profile

• Broad coalition of government agencies and critical private-sector organizations

• Support cross-agency and public-private sector coordination of crisis management activities

• Adaptive mobile network, virtual

collaboration, information assurance, and information integration technology

• Private-sector system-of-systems integration contractor

5/25/04 64

NISCM Home Ground Chart

5/25/04 65

NISCM Risks

5/25/04 66

NISCM Mitigations

5/25/04 67

NISCM Strategy

Startup Teambuilding and Shared

Vision

Design, Development and Deployment

Systems Definition and Architecting

Stable, Higher-criticality Module Teams

Stakeholders

Furnish CRACK representatives and alternates

• Staff and organize to cover all success-critical risk areas

• Prepare for and select competitors for concept definition

• Develop results chains• Identify missing

stakeholders• Enhance understanding• Explore goals, options,

technology, prototypes, risks

• Negotiate mutually satisfactory objectives and priorities

• Formulate top-level operational concept, requirements, architecture, life cycle plan, feasibility rational

• Select winning system integration contract

Hold Life Cycle

Objective Review. If feasibility

rationale is valid,

commit to proceed

• Prepare for/select competitors for component subcontract definition

• Elaborate operational concept, prototypes, transition strategy

• Evaluate/determine COTS, reuse, and architecture choices

• Develop/ validate critical-infrastructure software

• Prioritize/sequence desired capabilities, outstanding risks

• Formulate definitive operational concept, requirements, architecture, life cycle plan, feasibility rationale

• Use to solicit/select component subcontractors

Hold Life Cycle

Architecture Review. If feasibility

rationale is valid,

commit to proceed

• Ensure representative exercise of incremental capabilities

• Monitor progress and new operational development

• Monitor project progress, risk resolution, and new technology developments

• Identify/work critical project-level actions

• Continuously integrate/ test growing software infrastructure and components

Dynamic, Lower-criticality Module Teams

• Develop compatible component-level prototypes, requirements, architectures, plans, critical software, feasibility rationale

• Classify modules by likely stability and criticality

Organize into disciplined and agile module teams

Plan, specify, develop stable, higher-criticality module increments

Plan, specify, develop dynamic, lower-criticality module increments

• Communicate proposed redirections

• Prepare for and execute acceptance tests, installation, operational evolution

• Analyze feedback on current version

• Reprioritize features

• Prepare strategy for next increment

• Use risk to determine content of artifacts

NSO and I3-led ProjectRisk Management Team

LCALCO

5/25/04 68

Risk Profiles Across Examples

5/25/04 69

Outline





weaknesses• Risk-based balance of agility and discipline

– Small, medium, large examples • Observations, current work and a way

forward– Hands-on exercise


5/25/04 70

Summary of Observations

• No silver bullet• Clear agile and

plan-driven home grounds

• Increased demands for agility, velocity, quality

• Balanced methods are emerging

• Build methods up vs. tailor down

• Methods are important; people more so – Values– Communications– Expectations

management

5/25/04 71

No Silver Bullet

• Brooks’ werewolf concerns– Complexity,

conformity, changeability, invisibility

• Agile methods – Handle changeability

and invisibility– Miss on complexity

and conformity• Plan-driven methods

– Handle conformity and invisibility

– Miss on changeability and complexity

5/25/04 72

Future Applications Need Both

• Historically– Many small, non-critical,

well-skilled, agile culture, rapidly evolving projects

– Many large, critical, mixed-skill, ordered culture, stable projects

• In the future– Large projects are no

longer stable– Maintenance of extensive

process and product plans will become too expensive

– Complexity and conformity werewolves are waiting for agile projects

– Attributes of both approaches will be needed

5/25/04 73

Balanced Methods are Emerging

• Agile methods– Crystal Orange– DSDM– FDD– Lean Development

• Plan-Driven methods– Rational Unified

Process– CMMI

• Hybrid– Boehm-Turner Risk-

based– Code Science/Agile

Plus– Results from USC

2004 ARR

5/25/04 74

Build up – Don’t Tailor Down

• Plan-driven methods traditionally– Define heavyweight process guidelines– Advocate tailoring– Treat mass of processes as security blanket

• Agilists traditionally– Begin with the minimum – Add as needed (and justified by cost-benefit)– Start small; extend only where necessary

5/25/04 75

Maybe its not the methods!

• Agile movement has echoed a long line of warning calls

• Success of agile may be due as much to people factors as to technology

• Valuing people over processes is the most important factor in the manifesto

I know I saw something about that in the process somewhere…

5/25/04 76

People

• Of the people, by the people, for the Of the people, by the people, for the peoplepeople

• Separation of concerns is increasingly Separation of concerns is increasingly harmfulharmful

• A few quotes…A few quotes…– The notion of “user” cannot be The notion of “user” cannot be

precisely defined, and therefore it has precisely defined, and therefore it has no place in computer science or no place in computer science or software engineering.software engineering. Dijkstra Dijkstra

– Analysis and allocation of the system Analysis and allocation of the system requirements is not the responsibility requirements is not the responsibility of the software engineering group, but of the software engineering group, but it is a prerequisite for their work.it is a prerequisite for their work. The The Capability Maturity Model for Software.Capability Maturity Model for Software.

– Software engineering is not project Software engineering is not project management.management. TuckerTucker

5/25/04 77

Values

• Reconciling values is a critical people-oriented task

• Stakeholders value different things• Software engineering is usually value-

neutral– Every requirement, object, test case, defect

equally important

• Process improvement and plan-driven methods are inwardly-focused – Aimed at productivity improvement– Not higher value to customer

• Agilists attention to prioritization and negotiation are promising

5/25/04 78

Communications

• Face it, most engineers can’t talk, much less write

• Need to bridge the customer-developer communications gap

• IKIWISI* reigns• Rapid change increases

need for solid communication

• Few sources of guidance– Cockburn’s Agile Software

Development is a good starting place

*I’ll know It When I See It

5/25/04 79

Expectations Management

• Differences between successful and troubled projects is often expectations management

• SW developers have problems with EM – Strong desire to please– Little confidence in prediction– Over confidence in abilities

• Most significant common factor in highly effective plan-driven or agile teams

• EM means not setting up unrealistic expectations– Process mastery– Preparation– Courage

Developers seem to like Sisyphean tasks…

5/25/04 80

Recent Experiences in Hybrid Methods

• The following information is drawn from the USC-CSE Annual Research Review held in March 0f 2004

• Presentations were provided by large companies, consulting firms and research institutions

• All dealt with results and challenges from integrating agile and plan-driven methods

5/25/04 81

Agile Experiences

• Generally successful up to mid-range on 5v dimensions– Most common agile approaches included

selected application of XP and Scrum– Some successes with teams of teams, up to 4

teams– Common approach was applying XP practices in

a Scrum management environment

• Need to reinterpret some XP practices– Metaphor :: shared vision, common terminology– On-site customer :: customer proxy– Collective ownership :: Responsible ownership,

refactoring; designated functional experts– 40-hour work week :: Sustainable development

5/25/04 82

Agile Experiences - 2

• Need to add further techniques– Business process analysis– Risk management– Wall Gantts; story-based earned value– Independent reviews– Big-picture architecture

• Some situation-dependent differences of opinion– Simple design vs. architecture– Optional pair-programming vs. reviews– Stories vs. formalized acceptance criteria– Separate QA, CM

5/25/04 83

Agile Experiences - 3

• Main Challenge: Coexisting with large organization culture– Organizational processes

• Contracts• Requirements• Change control• QA

– Closely-coupled non-agile or legacy systems– Upper manmagement buy-in

• Business case• Controls• Risk

– Sarbanes-Oxley requirements

5/25/04 84

Sarbanes-Oxley

• A new US Law– Congress’ response to Enron, WorldCom, et al– Internal Controls: evaluate and disclose

effectiveness– Disclose fraud– Affects public companies and “significant” vendors

• Development process must include internal controls for– Fraud– Asset Management and Safeguarding– Financial Reporting

• Why is this important to executive management?– Executives can go to jail.– IT management can be held grossly negligent and

sued by a company or shareholders.• Goes into effect this year

5/25/04 85

What an Auditor Looks for…

Processes and tools over individuals and interactionsComprehensive documentation over working software

Contract negotiation over customer collaborationFollowing a plan over responding to change

An Auditor Manifesto?

5/25/04 86

Example Experiences from Presentations

• Northrop Grumman• Lockheed Martin• Alistair Cockburn• Agile Tek• NCSU Study (SABRE and IBM)• Software Experience Center Consortium

(Fraunhofer Center – Maryland)

5/25/04 87

Northrop Grumman Pilot

• Build a large system of analyst workstations working across different databases and networks in a rapid iterative environment using agile processes – Abandon traditional “shall” statement based

requirements development– Quickly develop a common understanding of how

the analysts will utilize the system through a use case development

– Developers to be responsible for implementing use cases not individual subsystem components. Everyone responsible for daily integration and integrity of the entire system.

– Schedule use case development through a series of priority setting meetings with the customer and users

5/25/04 88

Northrop Grumman Results

• There are gains to be made in productivity and reduced delivered defects with use case based system specification and implementation– Need to determine the detail required and

configuration management required to successfully manage this process

– Need customer buy in at the beginning along with agreement on how the system will be sold off.

– The cost to refine the system specification from high level to implementation level is still present, but the result of doing this refinement with use cases is a better common understanding of how the system will function at the end state.

• Embedded Government representatives with the developers are a necessity for rapid development and acceptance– They must have the authority to make

cost/schedule/function trade-offs within the current iteration

5/25/04 89

Lockheed Martin

• 5 programs that used Agile to some degree:– Maritime Systems and Sensors– Aeronautics– Space Systems

• Organized as follows:– Agile processes/practices used– What went well– What didn’t go quite so well

5/25/04 90

Lockheed Martin: Agile Practices Used

• Project Planning– Sprint Planning– Backlog Lists– Risk Management– Manage scope, not cost/schedule/quality

• Design– Use agile modeling techniques– Keep it simple– Document just enough to keep you going

• Implementation and Test– Pair Programming – Refactoring – Test driven design – Continuous integration– Development on the target system

5/25/04 91

Lockheed Martin: What went well

• Team empowerment/group ownership• Plan for and embrace change• Short cycle times allowed for prompt

and frequent feedback• Continuous integration• Customer involvement• Pair Programming

5/25/04 92

Lockheed Martin: Needed improvement

• Increased levels of stakeholder involvement

• Manage expectations• Agile development processes require

agile organizational processes

5/25/04 93

A. Cockburn on Hybrids

• Agile is a priority set & bucket of techniques that usually can be mixed into other priorities

• Agile prioritizes: – Project steering based on the integrated code base– Rapid feedback on product & process– People as a value center – Creativity in overcoming obstacles– Low bureaucratic overhead [not in the manifesto,

but present anyway]• Agile delivers process

– Efficiency – Maneuverability– Predictability (oddly enough)– Not every team values or prioritizes those

• Alternative priorities are to be :– predictable, repeatable, low cost, defect-free, low

liability,– ‘laid-back and enjoyable’

• Can mix to the extent they are not in conflict.– This is where the discussion should move to !

5/25/04 94

Agile Tek and Agile+

• Agile+ is XP + …• + Business Process Analyses (BPAs)• + Story “Actors”• + Delphi-STE Estimation• + Risk-Based Situation Audits (RBSAs)• + Componentized Architecture• + Wall Gantts and Instrument Panels• + Automated Contract and Regression

Testing• + Automatic Document Generation • Strict Pair Programming• 40-Hour Work Week Restriction• + Flexibility to meet special needs

5/25/04 95

Agile Tek: Solutions to ADM issues

• Scaling– Componentized Architecture/Interface Definitions– Automated Build and Test Processes – (Virtual) Team Rooms

• Unpredictability at Macro Scale– Delphi Estimation– STE usage for larger projects

• Vulnerability to changes at system level– Componentized Architecture

• Vague about system testing– Automated Contract and Regression Testing

• Inflexible to special needs– Treat the Special Need as a User Story and prioritize it

accordingly• Some ADM Practices Are Impractical

– Use practices that make sense and work in real-world situations

– Abandon or modify those that don’t• Do not Manage Risks Explicitly

– Use Risk Based Situation Audits– Establish a risk management philosophy

5/25/04 96

NCSU Study (IBM and Sabre)

• Research by Laurie Williams at North Carolina State University

• Preliminary results of ongoing research• Two teams at two companies• Used an XP Evaluation Framework

– Context factors (Boehm-Turner 5-D Chart)– Adherence metrics (Shodan survey)– Results metrics (NCSU)

5/25/04 97

IBM: XP-Context Factors

• Small team (7-10)• Co-located• Web development

(toolkit)• Supplier and

customer distributed (US and overseas)

• Examined one release “old” (low XP) to the next “new” (more XP)

5/25/04 98

Sabre: XP-Context Factors

• Small team (6-10)• Co-located• Scriptable GUI

environment• Customer remote,

multinational, several time zones

• Examined third release “old” (low XP) to the ninth release “new” (sustained XP)

5/25/04 99

NCSU Summary

When used by teams operating within the specified context, the use of a specified subset of XP practices leads to an improvement in . . .

Alternative Hypothesis IBM Case study evidence?

Sabre case study evidence?

internal code structure No No

pre-release quality Yes Yes

post-release quality Yes Yes

programmer productivity

Yes Yes

customer satisfaction Yes N/A

team morale Yes N/A

Two characteristically-agile teams:

5/25/04 100

Software Experience Center

• Founded in 1999• Members include ABB, Boeing,

DaimlerChrysler, Motorola, Nokia• Goal is to improve members’

software competencies, practices • Actively share experiences• Topics include Subcontracting,

Requirements Engineering, Product Lines

• Facilitators, Experience Collectors– Fraunhofer Virtual Institute for

Empirical Software Engineering

5/25/04 101

SEC Study Background

• Based on 4 SEC meetings and one eWorkshop on agile– Experience was openly shared– Decision to compile, report on this

experience • Complemented with internal

reports, published papers• Material from 15 XP-influenced

pilots (various level of detail)• Identified common experiences

across organizations• The final report is awaiting formal

approval

5/25/04 102

Summary of SEC Findings

• Positive improvements on productivity without compromising quality

• Few defects could be traced to XP practices• Best for independent collocated projects

involving few people• Can be used for large, complex, safety-

critical systems with long life cycles• Always requires tailoring to fit the task at

hand• A broader implementation requires changes

to culture and quality system• Use of selected agile practices will become

more and more common• Agile methods will not be used much, but

will influence other processes• Hybrid processes will be the primary way to

apply agile principles

5/25/04 103

Research Challenges and Opportunities

• Techniques for achieving rapid quality– Risk/value-driven process lightening– Agile with lightweight V&V overlay– Capitalizing on domain knowledge– Reuse, component-based approaches– Automating quality enhancement

• Empirical analyses– Distributed agile methods– Hybrid agile/plan-driven methods– Effects of individual practices– Effects of scale, personnel turnover

5/25/04 104

•60% of projects in agile home ground

•60% of effort in plan-driven home ground

Size Distribution

5/25/04 105

Development Methods

5/25/04 106

Do Both Assure Success?

• TSP and agile methods are indeed silver bullets and always succeed spectacularly well.

• People tend to report success more than they do failures.

• The pioneer projects are being done by high-capability early adopters of new methods.

• The pioneer projects are experiencing some Hawthorne effects of performing extra well while being in the spotlight.

• The projects are being compared to particularly poorly performing past projects.

• High improvement numbers make options worth exploring

5/25/04 107

Reifer Scorecard for XP

5/25/04 108

Pair Programming Results

• Short studies are suspect; pairs take about 10 hours to jell

5/25/04 109

Educational Approaches

• Survey courses– Materials here plus excursions into practice– Try, compare inspections and pair

programming– Try, compare refactoring vs. use of design

patterns

• Project courses– Some practices need tailoring to student

constraints– Steep learning curve; precede with survey

course

5/25/04 110

Experimental XP-Like Course

• Goals:– Create an improved version of Agile

COCOMOII– Use XP practices; supplement where needed– Gather software process data

• Characteristics:– Part-time, non-collocated student team– Part-time customer– Personnel with moderate

platform/applications experience

5/25/04 111

5/25/04 112

XP + situational common sense => Lightweight method

• Planning variants– Brainstorming +modified win-win ->

e-Story/Task cards– Bi-weekly meetings– Stories prioritized -> iteration requirements

• Design variants– Brainstorming:: group input:: team buy-in– Architecture determined from shortfalls of

previous arch.– Top level design :: iteration designs– Iteration design :: fulfills iteration

requirements– Prototype GUI, draft class design: pair-work

5/25/04 113

Electronic Story/Task Cards

5/25/04 114

XP + situational common sense => Lightweight method

• Coding variants:– Coding standards:: drafted, team consensus,

will evolve as needed– Customer available bi-weekly for

feedback/report– Tests are being written before any code, all

code is unit & integration tested

• Quality guidelines:– Digital archives of artifacts – Simple, short, risk-based documentation

5/25/04 115

Organizational Way Forward

• Convene key stakeholders to determine organizational goals and strategies

• Rebalance agility and discipline in this organizational context

5/25/04 116

Determining Organizational Goals and Strategies

• Current software organization’s status and needs• Key stakeholders’ needs and trends• How to cope with future trends• The increased pace of change and need for agility• The increased concern with software

dependability and need for discipline• Your ability to satisfy your stakeholders’ evolving

value propositions and to keep up with your toughest competitors

• The increasing gap between supply and demand for Cockburn Levels 2 and 3 people

• Your ability to cope with existing and emerging challenges– COTS integration, evolving Internet/Web

capabilities, distributed and mobile operations, agent coordination, multimode virtual collaboration, and outsourcing jobs

5/25/04 117

Collins’ Good to Great- HarperBusiness, 2001

Bureaucratic Organization

Start-up Organization

Hierarchical Organization

Great Organization

Low

Low

High

High

Ethic of Entrepreneurship

Culture of Discipline

5/25/04 118

Rebalancing Agility and Discipline

In agile or plan-driven home ground

Highly mixed agile/plan driven profile

Graph “typical project” on 5D chart

-Now; 5 years ahead

Develop best home ground continuous process improvement strategy

Treat anomalies as risks to be managed

Develop incremental mixed strategy. Try on pilot project

Integrate, execute process, people, technology plans. Monitor and control using Balanced Scorecard.

In home ground with some anomalies

5/25/04 119

Hands-on Exercise

• Graph your organization’s “typical project” on 5D charts– Now; 5 years from now

• Identify major needs for organizational change– With respect to balancing agility and discipline– With respect to future trends, environmental risks

• Identify most critical first steps for improvement

• Identify pilot project for applying first steps– Application, staffing– Needs for education, teambuilding, culture change

• Summarize in bulleted strategic plan

5/25/04 120

Personnel





5030

105

1

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15


Funds Comfort

Single Life

Many Lives


0

10

20

30

40

Personnel





5030

105

1

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15


Funds Comfort

Single Life

Many Lives


0

10

20

30

40

Figure 6-1

Now

5/25/04 121

In 5 Years

Personnel





5030

105

1

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15


Funds Comfort

Single Life

Many Lives


0

10

20

30

40

Personnel





5030

105

1

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15


Funds Comfort

Single Life

Many Lives


0

10

20

30

40

Figure 6-1

5/25/04 122

Conclusions

• Plan-driven and agile methods both aim to– Satisfy customers– Meet cost and schedule parameters

• Home grounds exist, but the need/opportunity for integration is expanding

• We recommend a self-assessment of your organization and business – Locate your place in the home ground space– Identify areas of change and how they might

move your location– Look for ways to integrate methods to better

respond to change• People concerns dominate method

concerns

Barry Boehm University of Southern California Richard Turner George Washington University Based on...

Documents

Transcript of Barry Boehm University of Southern California Richard Turner George Washington University Based on...