Post on 31-Dec-2015
071503 Rita.Creel@aero.org © 2003 The Aerospace Corporation
PSM User’s Group Conference15 July 2003
Rita Creel Senior Engineering Specialist
Computer and Systems Engineering SubdivisionReconnaissance Systems Division
Systems Engineering Measurement
For Acquisition Process Improvement
2071503, PSM User’s Group Conference Rita.Creel@aero.org
Acknowledgements
The following individuals have also participated in this work:
Susan Jones Tom Pomroy Amanda Staples Dan Vanderwarker
This briefing was originally presented at an International Council on Systems Engineering (INCOSE) Brown Bag Meeting, held on May 27, 2003, in Centreville, VA.
3071503, PSM User’s Group Conference Rita.Creel@aero.org
Agenda
Introduction
Approach & Progress to Date
Next Steps & An Example
Recap
4071503, PSM User’s Group Conference Rita.Creel@aero.org
Motivation
The products and processes of an acquirer’s pre-contract award systems engineering activities can either
Reduce risks and increase a program’s chances for success
OR Create problems that are difficult to remedy after contract
award
Completeness, consistency, and accuracy are important characteristics for these products and processes
But they are extremely difficult to achieve!
5071503, PSM User’s Group Conference Rita.Creel@aero.org
Rationale for Measurement Focus
Goal: Improve the systems engineering processes and products of the acquirer’s pre-contract award activities
Premise: We need to Baseline where we are today
Characterize current processes and products Collect lessons learned
Define where we want to be tomorrow and move in that direction Analyze lessons learned Identify areas for improvement Select, plan for, and implement small improvements
Check our progress along the way: Are we really improving? Assess changes in processes, products, problems, and results—How?
“You can’t improve what you can’t measure.” –Tom Demarco,Controlling Software Projects, Yourdon Press, NY, 1982
6071503, PSM User’s Group Conference Rita.Creel@aero.org
Systems Engineering – INCOSE Definition(paraphrased)
An interdisciplinary approach and means to enable the realization of successful systems, focused on
Defining customer needs and required functionality EARLY in the development cycle
Documenting and managing requirements Designing a top-level architecture that can accommodate change, as
necessary Proceeding with design synthesis and system verification and
validation
A structured process that Proceeds from concept, to production, to operation Considers the complete problem (operations, performance, test,
manufacturing, cost and schedule, training and support, disposal)
What are the acquirer’s systems engineering activities in the pre-contract award time frame?
7071503, PSM User’s Group Conference Rita.Creel@aero.org
Acquirer’s Systems Engineering Activities(pre-contract award)
Exactly what IS the acquirer’s systems engineering role in preparing for a major acquisition?
Concept exploration
Technology research and maturation
Architecture studies
Cost and schedule estimation
Requirements definition
Acquisition Strategy Planning
Request For Proposal (RFP) preparation
Source selection
We can’t tackle all these at once—so where do we start?
8071503, PSM User’s Group Conference Rita.Creel@aero.org
We Chose RFP Preparation – Why?
RFP sets the foundation for the source selection and for the work to be done on contract
Historically problematic Few organizational process assets to support the process For large systems, requires personnel with a variety of
expertise—availability nearly always a consideration Usually (always?) understaffed
Measurable tasks and products can be readily identified There are lots of things we can measure We only want to collect data that will be truly useful
How are we going to select and define an effective, efficient set of measures?
9071503, PSM User’s Group Conference Rita.Creel@aero.org
Agenda
Introduction
Approach & Progress to Date
Next Steps & An Example
Recap
10071503, PSM User’s Group Conference Rita.Creel@aero.org
Approach
PHASE 1. Use the Practical Software and Systems Measurement (PSM)1 Process to select and define measures
Consistent with the Capability Maturity Model Integration® (CMMI®)2 Measurement and Analysis Process Area
Flexible approach to implementing measurement based on information needs and objectives
PHASE 2. In the PSM Insight tool, implement a measurement framework for the measures selected in PHASE 1
PHASE 3. Pilot the measures on selected programs
PHASE 4. Document lessons learned, revise the approach and measures as needed, and go back to STEP 1
1Practical Software and Systems Measurement is sponsored by the DoD and US Army (www.psmsc.com) 2Capability Maturity Model Integration, Version 1.1, (SEI-2002-TR-012), ©2002 by Carnegie Mellon University (www.sei.cmu.edu/cmmi)
11071503, PSM User’s Group Conference Rita.Creel@aero.org
PSM Process – Measurement Activities
Objectivesand Issues
EvaluateMeasurement
ImprovementActions
AnalysisResults and Performance Measures
Scope of PSM
User Feedback
Analysis Results
Establish &Sustain
Commitment
Technical and ManagementProcesses
PerformMeasurementNew
Issues
Measurement Plan
Core Measurement Process
PlanMeasurement
From PSM Version 5.0b, 21
12071503, PSM User’s Group Conference Rita.Creel@aero.org
Breakout of Measurement Activities
Task StepActivity
P la n M e a su re m e n t
P e rfo rm M e a su re m e n t
Id e ntify a n d P rio rit iz e In fo rm a tio n N ee d s
S e le ct a n d S pe c ify M e as u re s
Id e ntify In fo rm a tio n N e e d s
From PSM Version 5.0b, 22
13071503, PSM User’s Group Conference Rita.Creel@aero.org
Plan Measurement(activity)
Measurement Plan
Integrate Intothe ProjectProcesses
Select and SpecifyMeasures
Identify and Prioritize
Information Needs New Information Needs
Risk ManagementInformation
Objectives, IssuesProject EnvironmentImprovement Actions
Proposed Changes
Proposed Changes
From PSM Version 5.0b, 32
14071503, PSM User’s Group Conference Rita.Creel@aero.org
Identify and Prioritize Information Needs(task)
ProposedChanges
Prioritized InformationNeeds
Information NeedsIdentify
Information Needs
Map Information
Needs
Prioritize Information
Needs
Information NeedsProject Characteristics
Improvement Actions
Risk Management and Financial Management Plans
From PSM Version 5.0b, 33
15071503, PSM User’s Group Conference Rita.Creel@aero.org
Identify Information Needs(step 1)
Review and document significant lessons learned from various programs.
Identify desired outcome and objectives. What interfered? Identify risks, issues, and problems.
Determine what kind of information could have helped mitigate risks or prevent problems.
Can measurement provide some or all of that information?
How big and complex is the job?How long do we have to do it, and how is the schedule likely to change?How are we going to track progress toward completion?How realistic is the budget, and how is it likely to change?How are we going to track expenditures?What is the required quality of our products and processes?How are we going to determine actual quality?Do we have the right people, and can we get enough of their time?Do we have the infrastructure we need to support the work?
Example Information Needs
16071503, PSM User’s Group Conference Rita.Creel@aero.org
Map Information Needs(step 2)
Develop a list of the acquirer’s “generic” activities and products for pre-contract award
Identify “measurable concepts” that link key issues and information needs to activities and products
Some ExamplesInformation Needs Measurable Concepts Activities/ProductsSize and complexity of job Required functionality/quality, Requirements, capabilities,
stakeholders, interfaces concept study results
Schedule and status Milestones, dependencies, Plans, schedules, RFP completeness, volatility
Funding constraints Budget, cost profile, volatility Financial data, plans
Quality V&V effectiveness, defects ASP, RFP (SOW/SOO, requirements, instructions,
evaluation criteria)
Availability of needed expertise Expertise, availability, turnover Staffing profile, plans
Supporting infrastructure Resource availability, process Policies/practices, tailoring (tools, processes) capability guidelines, hardware,
software, trainingAcronyms: ASP: Acquisition Strategy Plan; RFP: Request for Proposal; SOO: Statement of Objectives;
SOO: Statement of Work; V&V: Verification and Validation
17071503, PSM User’s Group Conference Rita.Creel@aero.org
Prioritize Information Needs(step 3)
Prioritize information needs according to The information that’s most critical to have The expected effort required to obtain that information
In our initial discussions on lessons learned, two issues rose to the surface
We never have enough of the right peopleto get the job done right
The quality of the RFP suffers as a result
Top-Priority Information Needs
• Availability of needed expertise & stakeholders with interests that span the life cycle
• Quality of the developing RFP
18071503, PSM User’s Group Conference Rita.Creel@aero.org
Plan Measurement(activity)
Measurement Plan
Integrate Intothe ProjectProcesses
Select and SpecifyMeasures
Identify and Prioritize
Information Needs New Information Needs
Risk ManagementInformation
Objectives, IssuesProject EnvironmentImprovement Actions
Proposed Changes
Proposed Changes
From PSM Version 5.0b, 42
19071503, PSM User’s Group Conference Rita.Creel@aero.org
Agenda
Introduction
Approach & Progress to Date
Next Steps & An Example
Recap & Issues
20071503, PSM User’s Group Conference Rita.Creel@aero.org
Select and Specify Measures(task)
CharacterizeProject Context
DefineM easurable
Concepts
Select the ApplicableM easures
SpecifyM easurem ent
Constructs
From PSM Version 5.0b, 43
21071503, PSM User’s Group Conference Rita.Creel@aero.org
Select and Specify Measures(task)
Step 1: Characterize Project Context Identify one or more potential pilot projects and associated points
of contact (POCs) Ideally, select projects that are just beginning RFP preparation
Discuss with POC characteristics of the project as well as the intended RFP process and product
Steps 2 & 3: Define Measurable Concepts & Select Applicable Measures
With the above characteristics in mind, select a few “promising” measurable concepts related to the target information need
Let’s say we’ve chosen to begin with “How much effort is being applied to key tasks vs. the effort estimated/planned?”
Step 4: Specify Measurement Constructs
22071503, PSM User’s Group Conference Rita.Creel@aero.org
Levels of a Measurement Construct
Information Product
Attribute
Indicator(s)
Derived Measure(s)
Base Measure(s)
Measurement Method
Adapted from PSM Book Updates, Betsy Clark
Measurement Construct
AddressesInformationNeed
ImplementsMeasurableConcept
CharacterizesProduct orProcess
Function
Analysis Model
23071503, PSM User’s Group Conference Rita.Creel@aero.org
Information Need
Decision Criteria &IndicatorIndicator
Analysis Model
Derived MeasureDerived Measure
MeasurementFunctions
Base MeasuresBase Measures
MeasurementMethods
The Process- Entities & Attributes
Actual Hours SpentOn Task
Subtract Hours Spent Processing (includingrecovering from) Interruptions From
Total Hours Charged to Task
Factor in:•Work accomplished (quantity and quality)
•Time criticality of task•Importance of task•# interruptions•Staff experience•Staff project familiarity•Relevant historical data!
Example Measurement Construct
Record TimeCharged to Task
Record TimeSpent Processing
Unrelated Interruptions
Record HoursPlanned for Task
Total HoursCharged to Task
Hours Spentin Interruptions
Planned Hours
Plot Interrupt Impact onPlan (Plan vs. Actual,with 10% threshold);
Plot Hours Charged & HoursSpent on Interruptions
If |actual–planned| >10% planned:•Is the plan bad? (if so, replan)•Do we need better “interrupt management”? (if so, strategize)
Are Interruptions Preventing Staff From Working Key Tasks as Planned?
Effort Records (time sheets); Interruption Logs (create form)
Compare Actual vs. Planned Hours Spent on Task
24071503, PSM User’s Group Conference Rita.Creel@aero.org
RFP Preparation Effort
0
5
10
15
20
25
30
35
40
45
50
Hou
rs
Hours Scheduled Hours Charged Action Threshold-High
Action Threshold-Low Interrupt Impact Actual Hours-RFP Prep
Example Indicator
RFP Preparation Effort
0
5
10
15
20
25
30
35
40
45
50
Hou
rs
Hours Scheduled Hours Charged Action Threshold-High
Action Threshold-Low Interrupt Impact Actual Hours-RFP Prep
25071503, PSM User’s Group Conference Rita.Creel@aero.org
Plan Measurement(activity)
Measurement Plan
Integrate Intothe ProjectProcesses
Select and SpecifyMeasures
Identify and Prioritize
Information Needs New Information Needs
Risk ManagementInformation
Objectives, IssuesProject EnvironmentImprovement Actions
Proposed Changes
Proposed Changes
From PSM Version 5.0b, 55
26071503, PSM User’s Group Conference Rita.Creel@aero.org
Integrate into the Project Processes(task)
From PSM Version 5.0b, 56
IdentifyM easurem entO pportun ities
DevelopM easurem ent
Procedures
Docum entProject
P lan(for
measurement)
27071503, PSM User’s Group Conference Rita.Creel@aero.org
Step 1: Identify Measurement Opportunities Identify pilot efforts Review products and processes Identify measurables
Step 2: Develop Measurement Procedures Collection procedures Analysis procedures
Step 3: Document Measurement Plan for the Project Identify measurement roles, responsibilities, and communications Describe project information needs List and specify measures used to address information needs Define aggregation (roll-up) structures Define reporting mechanisms and frequency
Integrate into the Project Processes(task)
28071503, PSM User’s Group Conference Rita.Creel@aero.org
PSM Process – Measurement Activities
Objectivesand Issues
EvaluateMeasurement
ImprovementActions
AnalysisResults and Performance Measures
Scope of PSM
User Feedback
Analysis Results
Establish &Sustain
Commitment
Technical and ManagementProcesses
PerformMeasurementNew
Issues
Measurement Plan
Core Measurement Process
PlanMeasurement
From PSM Version 5.0b, 21
29071503, PSM User’s Group Conference Rita.Creel@aero.org
Perform and Evaluate Measurement(activities)
Perform Measurement Activity Analyze and report results Identify and implement improvements to the acquisition process
Evaluate Measurement Activity Ask: Is the measurement process efficient and effective?
What is the effort to collect? What is the effort to analyze? Are the measures used to make decisions?
Ask: Do the measures work? Are they sufficiently informative (accurate, repeatable, meaningful)? If not, what are the shortcomings?
Document lessons learned Identify and implement improvements to measures and the
measurement process
30071503, PSM User’s Group Conference Rita.Creel@aero.org
PSM Process – Measurement Activities
Objectivesand Issues
EvaluateMeasurement
ImprovementActions
AnalysisResults and Performance Measures
Scope of PSM
User Feedback
Analysis Results
Establish &Sustain
Commitment
Technical and ManagementProcesses
PerformMeasurementNew
Issues
Measurement Plan
Core Measurement Process
PlanMeasurement
From PSM Version 5.0b, 21
31071503, PSM User’s Group Conference Rita.Creel@aero.org
Establish and Sustain Commitment(activity)
Obtain Organizational Commitment Address “What’s in it for me?” Commit to measurement as “the way we do business” Provide training
Define Responsibilities Assign measurement tasks to people
Provide Resources Start with simple tools (e.g., spreadsheets) Provide the time to perform measurement tasks
Review Progress of Measurement Implementation Ensure alignment of measures with information needs Perform regular status checks on Plan Measurement tasks If implementation stalls, address problems quickly
32071503, PSM User’s Group Conference Rita.Creel@aero.org
Agenda
Introduction
Approach & Progress to Date
Next Steps & An Example
Recap
33071503, PSM User’s Group Conference Rita.Creel@aero.org
Recap
Goal Improve the acquisition
process Approach
Identify and measure effects of improvements
Start with RFP preparation process and products
Progress to Date Identified & prioritized some
information needs Began thinking about
measurable concepts
Next Steps Specify measurement
constructs Begin pilot projects
Collect and analyze data Evaluate & suggest
improvements to acquisition process
Evaluate measures and measurement process
Implement improvements Repeat, targeting another
information need
“Without the right information, you’re just another personwith an opinion.” –Tracy O’Rourke, CEO of Allen-Bradley
34071503, PSM User’s Group Conference Rita.Creel@aero.org
Contact Information
For information on this effort:
Rita Creel
Phone: (703) 633-5634
Email: Rita.Creel@aero.org
For information on PSM:
www.psmsc.com
For information on the CMMI:www.sei.cmu.edu/cmmi
35071503, PSM User’s Group Conference Rita.Creel@aero.org
Backup Slides
PSM Measurement Hierarchy
PSM Mapping of Information Categories, Measurable Concepts, Measures (1 of 2)
PSM Mapping of Information Categories, Measurable Concepts, Measures (2 of 2)
36071503, PSM User’s Group Conference Rita.Creel@aero.org
PSM Measurement Hierarchy
Project Inform ationNeed
Inform ation Categories
MeasurableConcepts
Measurem entConstructs(Measures)
From PSM Version 5.0b, 36
37071503, PSM User’s Group Conference Rita.Creel@aero.org
PSM Mapping of Information Categories,Measurable Concepts, and Measures(1 of 2)
InformationCategories
MeasurableConcepts
Prospective Measures
Information - Category - Measure Mapping
Schedule and Progress
Resources and Cost
Product S ize and S tability
M ilestone CompletionCritical Pa th Perform anceW ork Unit P rogress
Incremental Capab ility
Personne l E ffort
Physica l S ize and S tability
Functional Size andStability
Environment and Support Resources
Financia l Performance
M ilestone DatesS lack Tim eRequirements TracedRequirements TestedProblem Reports OpenedProblem Reports C losedReviews CompletedChange Requests OpenedChange Requests ResolvedUnits Designed Units CodedUnits IntegratedTest Cases AttemptedTest Cases PassedAction Items OpenedAction Items Comple tedComponents IntegratedFunctionality Integrated S taff LevelDevelopment EffortExperience LevelS taff TurnoverBCW S, BCW P, ACW PBudget CostQuantity Needed Quantity AvailableTime AvailableTime UsedDatabase SizeComponents InterfacesLines of CodeRequirementsFunctional ChangesFunction Po in tsFrom PSM Version 5.0b, 38
38071503, PSM User’s Group Conference Rita.Creel@aero.org
PSM Mapping of Information Categories,Measurable Concepts, and Measures(2 of 2)
Inform ationC ateg ories
M easu rab leC on cepts
P rospective M easu res
Information - Category - M easure Mapping
Product Q ua lity
P rocess Pe rform ance
Techno logy E ffectivenessC ustom er Sa tis faction
Functional C orrectness
Supportab ilty -M ain ta inability
E ffic iency
Po rtab ilityU sab ilityD ependab ility -R eliab ilityP rocess C om pliance
Process E ffic iency
Process E ffectiveness
Techno logy S uitab ilityTechno logy Vo la tilityC ustom er Feedback
C ustom er Suppo rt
D efectsAge o f D e fectsTechnica l P erform ance Leve lT im e to R estoreC yclom atic C om plex ityU tiliza tionThroughpu tR esponse T im eS tandards Com p lianceO perato r E rro rsM ean T im e to Fa ilu reR efe rence M atu rity Ra tingProcess Audit F indingsProductiv ityC ycle T im eD efects C on ta inedD efects E scapingR ew ork E ffortR ew ork C om ponentsR equirem ents C ove rageBaseline C hangesSa tis faction R a tingsAward FeeR equests fo r S uppo rtSupport T im e
From PSM Version 5.0b, 39
RFP Review Effectiveness
Product Quality
Process Effectiveness