071503 [email protected] © 2003 The Aerospace Corporation PSM User’s Group Conference 15 July...

071503 [email protected] © 2003 The Aerospace Corporation

PSM User’s Group Conference15 July 2003

Rita Creel Senior Engineering Specialist

Computer and Systems Engineering SubdivisionReconnaissance Systems Division

Systems Engineering Measurement

For Acquisition Process Improvement

2071503, PSM User’s Group Conference [email protected]

Acknowledgements

The following individuals have also participated in this work:

Susan Jones Tom Pomroy Amanda Staples Dan Vanderwarker

This briefing was originally presented at an International Council on Systems Engineering (INCOSE) Brown Bag Meeting, held on May 27, 2003, in Centreville, VA.


Agenda

Introduction

Approach & Progress to Date

Next Steps & An Example

Recap


Motivation

The products and processes of an acquirer’s pre-contract award systems engineering activities can either

Reduce risks and increase a program’s chances for success

OR Create problems that are difficult to remedy after contract

award

Completeness, consistency, and accuracy are important characteristics for these products and processes

But they are extremely difficult to achieve!


Rationale for Measurement Focus

Goal: Improve the systems engineering processes and products of the acquirer’s pre-contract award activities

Premise: We need to Baseline where we are today

Characterize current processes and products Collect lessons learned

Define where we want to be tomorrow and move in that direction Analyze lessons learned Identify areas for improvement Select, plan for, and implement small improvements

Check our progress along the way: Are we really improving? Assess changes in processes, products, problems, and results—How?

“You can’t improve what you can’t measure.” –Tom Demarco,Controlling Software Projects, Yourdon Press, NY, 1982


Systems Engineering – INCOSE Definition(paraphrased)

An interdisciplinary approach and means to enable the realization of successful systems, focused on

Defining customer needs and required functionality EARLY in the development cycle

Documenting and managing requirements Designing a top-level architecture that can accommodate change, as

necessary Proceeding with design synthesis and system verification and

validation

A structured process that Proceeds from concept, to production, to operation Considers the complete problem (operations, performance, test,

manufacturing, cost and schedule, training and support, disposal)

What are the acquirer’s systems engineering activities in the pre-contract award time frame?


Acquirer’s Systems Engineering Activities(pre-contract award)

Exactly what IS the acquirer’s systems engineering role in preparing for a major acquisition?

Concept exploration

Technology research and maturation

Architecture studies

Cost and schedule estimation

Requirements definition

Acquisition Strategy Planning

Request For Proposal (RFP) preparation

Source selection

We can’t tackle all these at once—so where do we start?


We Chose RFP Preparation – Why?

RFP sets the foundation for the source selection and for the work to be done on contract

Historically problematic Few organizational process assets to support the process For large systems, requires personnel with a variety of

expertise—availability nearly always a consideration Usually (always?) understaffed

Measurable tasks and products can be readily identified There are lots of things we can measure We only want to collect data that will be truly useful

How are we going to select and define an effective, efficient set of measures?


Agenda

Introduction



Recap


Approach

PHASE 1. Use the Practical Software and Systems Measurement (PSM)1 Process to select and define measures

Consistent with the Capability Maturity Model Integration® (CMMI®)2 Measurement and Analysis Process Area

Flexible approach to implementing measurement based on information needs and objectives

PHASE 2. In the PSM Insight tool, implement a measurement framework for the measures selected in PHASE 1

PHASE 3. Pilot the measures on selected programs

PHASE 4. Document lessons learned, revise the approach and measures as needed, and go back to STEP 1

1Practical Software and Systems Measurement is sponsored by the DoD and US Army (www.psmsc.com) 2Capability Maturity Model Integration, Version 1.1, (SEI-2002-TR-012), ©2002 by Carnegie Mellon University (www.sei.cmu.edu/cmmi)


PSM Process – Measurement Activities

Objectivesand Issues

EvaluateMeasurement

ImprovementActions

AnalysisResults and Performance Measures

Scope of PSM

User Feedback

Analysis Results

Establish &Sustain

Commitment

Technical and ManagementProcesses

PerformMeasurementNew

Issues

Measurement Plan

Core Measurement Process

PlanMeasurement

From PSM Version 5.0b, 21


Breakout of Measurement Activities

Task StepActivity

P la n M e a su re m e n t

P e rfo rm M e a su re m e n t

Id e ntify a n d P rio rit iz e In fo rm a tio n N ee d s

S e le ct a n d S pe c ify M e as u re s

Id e ntify In fo rm a tio n N e e d s



Plan Measurement(activity)

Measurement Plan

Integrate Intothe ProjectProcesses

Select and SpecifyMeasures

Identify and Prioritize

Information Needs New Information Needs

Risk ManagementInformation

Objectives, IssuesProject EnvironmentImprovement Actions

Proposed Changes

Proposed Changes



Identify and Prioritize Information Needs(task)

ProposedChanges

Prioritized InformationNeeds

Information NeedsIdentify

Information Needs

Map Information

Needs

Prioritize Information

Needs

Information NeedsProject Characteristics

Improvement Actions

Risk Management and Financial Management Plans



Identify Information Needs(step 1)

Review and document significant lessons learned from various programs.

Identify desired outcome and objectives. What interfered? Identify risks, issues, and problems.

Determine what kind of information could have helped mitigate risks or prevent problems.

Can measurement provide some or all of that information?

How big and complex is the job?How long do we have to do it, and how is the schedule likely to change?How are we going to track progress toward completion?How realistic is the budget, and how is it likely to change?How are we going to track expenditures?What is the required quality of our products and processes?How are we going to determine actual quality?Do we have the right people, and can we get enough of their time?Do we have the infrastructure we need to support the work?

Example Information Needs


Map Information Needs(step 2)

Develop a list of the acquirer’s “generic” activities and products for pre-contract award

Identify “measurable concepts” that link key issues and information needs to activities and products

Some ExamplesInformation Needs Measurable Concepts Activities/ProductsSize and complexity of job Required functionality/quality, Requirements, capabilities,

stakeholders, interfaces concept study results

Schedule and status Milestones, dependencies, Plans, schedules, RFP completeness, volatility

Funding constraints Budget, cost profile, volatility Financial data, plans

Quality V&V effectiveness, defects ASP, RFP (SOW/SOO, requirements, instructions,

evaluation criteria)

Availability of needed expertise Expertise, availability, turnover Staffing profile, plans

Supporting infrastructure Resource availability, process Policies/practices, tailoring (tools, processes) capability guidelines, hardware,

software, trainingAcronyms: ASP: Acquisition Strategy Plan; RFP: Request for Proposal; SOO: Statement of Objectives;

SOO: Statement of Work; V&V: Verification and Validation


Prioritize Information Needs(step 3)

Prioritize information needs according to The information that’s most critical to have The expected effort required to obtain that information

In our initial discussions on lessons learned, two issues rose to the surface

We never have enough of the right peopleto get the job done right

The quality of the RFP suffers as a result

Top-Priority Information Needs

• Availability of needed expertise & stakeholders with interests that span the life cycle

• Quality of the developing RFP



Measurement Plan







Proposed Changes

Proposed Changes



Agenda

Introduction



Recap & Issues


Select and Specify Measures(task)

CharacterizeProject Context

DefineM easurable

Concepts

Select the ApplicableM easures

SpecifyM easurem ent

Constructs



Select and Specify Measures(task)

Step 1: Characterize Project Context Identify one or more potential pilot projects and associated points

of contact (POCs) Ideally, select projects that are just beginning RFP preparation

Discuss with POC characteristics of the project as well as the intended RFP process and product

Steps 2 & 3: Define Measurable Concepts & Select Applicable Measures

With the above characteristics in mind, select a few “promising” measurable concepts related to the target information need

Let’s say we’ve chosen to begin with “How much effort is being applied to key tasks vs. the effort estimated/planned?”

Step 4: Specify Measurement Constructs


Levels of a Measurement Construct

Information Product

Attribute

Indicator(s)

Derived Measure(s)

Base Measure(s)

Measurement Method

Adapted from PSM Book Updates, Betsy Clark

Measurement Construct

AddressesInformationNeed

ImplementsMeasurableConcept

CharacterizesProduct orProcess

Function

Analysis Model


Information Need

Decision Criteria &IndicatorIndicator

Analysis Model

Derived MeasureDerived Measure

MeasurementFunctions

Base MeasuresBase Measures

MeasurementMethods

The Process- Entities & Attributes

Actual Hours SpentOn Task

Subtract Hours Spent Processing (includingrecovering from) Interruptions From

Total Hours Charged to Task

Factor in:•Work accomplished (quantity and quality)

•Time criticality of task•Importance of task•# interruptions•Staff experience•Staff project familiarity•Relevant historical data!

Example Measurement Construct

Record TimeCharged to Task

Record TimeSpent Processing

Unrelated Interruptions

Record HoursPlanned for Task

Total HoursCharged to Task

Hours Spentin Interruptions

Planned Hours

Plot Interrupt Impact onPlan (Plan vs. Actual,with 10% threshold);

Plot Hours Charged & HoursSpent on Interruptions

If |actual–planned| >10% planned:•Is the plan bad? (if so, replan)•Do we need better “interrupt management”? (if so, strategize)

Are Interruptions Preventing Staff From Working Key Tasks as Planned?

Effort Records (time sheets); Interruption Logs (create form)

Compare Actual vs. Planned Hours Spent on Task


RFP Preparation Effort

0

5

10

15

20

25

30

35

40

45

50

Hou

rs

Hours Scheduled Hours Charged Action Threshold-High

Action Threshold-Low Interrupt Impact Actual Hours-RFP Prep

Example Indicator

RFP Preparation Effort

0

5

10

15

20

25

30

35

40

45

50

Hou

rs

Hours Scheduled Hours Charged Action Threshold-High

Action Threshold-Low Interrupt Impact Actual Hours-RFP Prep



Measurement Plan







Proposed Changes

Proposed Changes



Integrate into the Project Processes(task)


IdentifyM easurem entO pportun ities

DevelopM easurem ent

Procedures

Docum entProject

P lan(for

measurement)


Step 1: Identify Measurement Opportunities Identify pilot efforts Review products and processes Identify measurables

Step 2: Develop Measurement Procedures Collection procedures Analysis procedures

Step 3: Document Measurement Plan for the Project Identify measurement roles, responsibilities, and communications Describe project information needs List and specify measures used to address information needs Define aggregation (roll-up) structures Define reporting mechanisms and frequency

Integrate into the Project Processes(task)




EvaluateMeasurement

ImprovementActions


Scope of PSM

User Feedback

Analysis Results

Establish &Sustain

Commitment



Issues

Measurement Plan


PlanMeasurement



Perform and Evaluate Measurement(activities)

Perform Measurement Activity Analyze and report results Identify and implement improvements to the acquisition process

Evaluate Measurement Activity Ask: Is the measurement process efficient and effective?

What is the effort to collect? What is the effort to analyze? Are the measures used to make decisions?

Ask: Do the measures work? Are they sufficiently informative (accurate, repeatable, meaningful)? If not, what are the shortcomings?

Document lessons learned Identify and implement improvements to measures and the

measurement process




EvaluateMeasurement

ImprovementActions


Scope of PSM

User Feedback

Analysis Results

Establish &Sustain

Commitment



Issues

Measurement Plan


PlanMeasurement



Establish and Sustain Commitment(activity)

Obtain Organizational Commitment Address “What’s in it for me?” Commit to measurement as “the way we do business” Provide training

Define Responsibilities Assign measurement tasks to people

Provide Resources Start with simple tools (e.g., spreadsheets) Provide the time to perform measurement tasks

Review Progress of Measurement Implementation Ensure alignment of measures with information needs Perform regular status checks on Plan Measurement tasks If implementation stalls, address problems quickly


Agenda

Introduction



Recap


Recap

Goal Improve the acquisition

process Approach

Identify and measure effects of improvements

Start with RFP preparation process and products

Progress to Date Identified & prioritized some

information needs Began thinking about

measurable concepts

Next Steps Specify measurement

constructs Begin pilot projects

Collect and analyze data Evaluate & suggest

improvements to acquisition process

Evaluate measures and measurement process

Implement improvements Repeat, targeting another

information need

“Without the right information, you’re just another personwith an opinion.” –Tracy O’Rourke, CEO of Allen-Bradley


Contact Information

For information on this effort:

Rita Creel

Phone: (703) 633-5634

Email: [email protected]

For information on PSM:

www.psmsc.com

For information on the CMMI:www.sei.cmu.edu/cmmi


Backup Slides

PSM Measurement Hierarchy

PSM Mapping of Information Categories, Measurable Concepts, Measures (1 of 2)

PSM Mapping of Information Categories, Measurable Concepts, Measures (2 of 2)


PSM Measurement Hierarchy

Project Inform ationNeed

Inform ation Categories

MeasurableConcepts

Measurem entConstructs(Measures)



PSM Mapping of Information Categories,Measurable Concepts, and Measures(1 of 2)

InformationCategories

MeasurableConcepts

Prospective Measures

Information - Category - Measure Mapping

Schedule and Progress

Resources and Cost

Product S ize and S tability

M ilestone CompletionCritical Pa th Perform anceW ork Unit P rogress

Incremental Capab ility

Personne l E ffort

Physica l S ize and S tability

Functional Size andStability

Environment and Support Resources

Financia l Performance

M ilestone DatesS lack Tim eRequirements TracedRequirements TestedProblem Reports OpenedProblem Reports C losedReviews CompletedChange Requests OpenedChange Requests ResolvedUnits Designed Units CodedUnits IntegratedTest Cases AttemptedTest Cases PassedAction Items OpenedAction Items Comple tedComponents IntegratedFunctionality Integrated S taff LevelDevelopment EffortExperience LevelS taff TurnoverBCW S, BCW P, ACW PBudget CostQuantity Needed Quantity AvailableTime AvailableTime UsedDatabase SizeComponents InterfacesLines of CodeRequirementsFunctional ChangesFunction Po in tsFrom PSM Version 5.0b, 38


PSM Mapping of Information Categories,Measurable Concepts, and Measures(2 of 2)

Inform ationC ateg ories

M easu rab leC on cepts

P rospective M easu res

Information - Category - M easure Mapping

Product Q ua lity

P rocess Pe rform ance

Techno logy E ffectivenessC ustom er Sa tis faction

Functional C orrectness

Supportab ilty -M ain ta inability

E ffic iency

Po rtab ilityU sab ilityD ependab ility -R eliab ilityP rocess C om pliance

Process E ffic iency

Process E ffectiveness

Techno logy S uitab ilityTechno logy Vo la tilityC ustom er Feedback

C ustom er Suppo rt

D efectsAge o f D e fectsTechnica l P erform ance Leve lT im e to R estoreC yclom atic C om plex ityU tiliza tionThroughpu tR esponse T im eS tandards Com p lianceO perato r E rro rsM ean T im e to Fa ilu reR efe rence M atu rity Ra tingProcess Audit F indingsProductiv ityC ycle T im eD efects C on ta inedD efects E scapingR ew ork E ffortR ew ork C om ponentsR equirem ents C ove rageBaseline C hangesSa tis faction R a tingsAward FeeR equests fo r S uppo rtSupport T im e


RFP Review Effectiveness

Product Quality

Process Effectiveness

071503 [email protected] © 2003 The Aerospace Corporation PSM User’s Group Conference 15 July...

Documents

Transcript of 071503 [email protected] © 2003 The Aerospace Corporation PSM User’s Group Conference 15 July...