The Nuts, Bolts and Duct Tape of Establishing a System Engineering Measurement Program Paul J. Frenz...
-
Upload
aidan-horton -
Category
Documents
-
view
214 -
download
2
Transcript of The Nuts, Bolts and Duct Tape of Establishing a System Engineering Measurement Program Paul J. Frenz...
The Nuts, Bolts and Duct The Nuts, Bolts and Duct Tape of Establishing a Tape of Establishing a System Engineering System Engineering
Measurement ProgramMeasurement ProgramPaul J. FrenzPaul J. Frenz
General Dynamics Advanced General Dynamics Advanced Information SystemsInformation Systems
Team AssignmentTeam Assignment
• Establish a Systems Engineering Measurement Program
• Support CMMI Level 5
• Support Organizational Initiatives
• Provide Quantifiable evidence of improvements achieved
Measures ResearchMeasures Research
Investigation of industry practices
• Software Productivity Consortium (SPC)
• Practical Software and Systems Measurement (PSM)
• INCOSE
• Generic web and literature searches
• Summer Intern did industry survey
Research FindingsResearch Findings
Measures such as:• requirements per hour • defects per requirement
can be useful to an organization:• current capabilities and setting goals • don’t provide insight into the daily
management of a program
Measures GuidelinesMeasures Guidelines
Measurements selected must be tied to business goals!
This lead to measurements that supported the goals of faultless execution, customer satisfaction, and quality
Measurement ChallengeMeasurement Challenge
The objective was to give the program leads the measures they needed to support these goals
The measures needed to guide the leads with knowing what to pay attention to and when to pay attention!
Initial Measures EstablishedInitial Measures Established
Building off of our lessons learned within Software Engineering, we established the initial measures using these principles:
• Measure collection should be uncomplicated - preferably automated
• Measures are justified
Initial Measures EstablishedInitial Measures Established
• We use a standard time period - calendar month
• Although we permit tailoring it is discouraged
• Avoid frequent replans and focus on using measures to return to the planned baseline
• Consistent criteria and an approval when a measurement baseline is replanned
Initial Measures SelectedInitial Measures Selected
• Productivity Measure – Requirements per Hour• Inch-stone Schedule Performance• Requirements Volatility• Requirement Volatility Impact • Problem Report Status Aging• Effort Expended to Planned Effort• Peer Review Scheduled to Plan• Peer Review Execution and Results• Requirements Defect Density
Productivity Measure – Productivity Measure – Requirements Per HourRequirements Per Hour
SE Productivity Predictability
0
2
4
6
8
10
12
Jan-0
4
Feb-0
4
Mar-
04
Apr-
04
May-0
4
Jun-0
4
Jul-04
Aug-0
4
Sep-0
4
Oct-
04
Nov-0
4
Dec-0
4
Jan-0
5
Feb-0
5
Re
qu
ire
me
nts
Pe
r 1
00
Ho
urs
EAC SE Requirements*100/EAC HoursOriginal EAC SE Requirements*100/EAC Hours
Inch-stone Schedule Inch-stone Schedule Performance to PlanPerformance to Plan
All Monthly Schedule Performance
0
20
40
60
80
100
120
140
Jan-
04
Feb
-04
Mar
-04
Apr
-04
May
-04
Jun-
04
Jul-0
4
Aug
-04
Sep
-04
Oct
-04
Nov
-04
Dec
-04
Jan-
05
Feb
-05
Mo
nth
ly T
asks
to
Pla
n P
erce
nta
ge
Monthly Tasks Percentage to Plan
Inch-stone Schedule Inch-stone Schedule Performance to PlanPerformance to Plan
All Schedule Performance
0
100
200
300
400
500
600
Jan
-04
Fe
b-0
4
Ma
r-0
4
Ap
r-0
4
Ma
y-0
4
Jun
-04
Jul-
04
Au
g-0
4
Se
p-0
4
Oct
-04
Nov-
04
Dec-
04
Jan
-05
Fe
b-0
5
Ta
sk
s
Planned Tasks Complete Actual Tasks Complete
Requirements VolatilityRequirements Volatility
Requirement Requirement Volatility Impact AnalysisVolatility Impact Analysis
Problem Report (PR) Status Problem Report (PR) Status AgingAging
All Problem Report Status Aging
0
10
20
30
40
50
60
1 2 3 4 5 6+
Months
Pro
ble
m R
ep
ort
s
Open Analysis CCB ReviewAssigned Verification Closed
Effort Expended to Effort Effort Expended to Effort PlannedPlanned
Peer Review Schedule to PlanPeer Review Schedule to Plan
Peer Review Execution and Peer Review Execution and ResultsResults
SE Peer Review Saves Per Page
0
0.5
1
1.5
2
2.5
3
3.5
4R
eq
SR
R
DD
PD
R
CD
R
CIR
L
Saves
Escapes Per PageMajor Saves Per PageMinor Saves Per PageNew Reviews
Peer Review Execution and Peer Review Execution and ResultsResults
SE Peer Review Hours Per Page
0
0.5
1
1.5
2
2.5
3
3.5
4
Req
SR
R
DD
PD
R
CD
R
CIR
L
Ho
urs
Hours Per PageNew Reviews
Requirements Defect DensityRequirements Defect DensitySE Requirement Defect Density
0
0.2
0.4
0.6
0.8
1
1.2
Jan-0
4
Feb-0
4
Mar-
04
Apr-
04
May-0
4
Jun-0
4
Jul-04
Aug-0
4
Sep-0
4
Oct-
04
Nov-0
4
Dec-0
4
Jan-0
5
Feb-0
5
De
fec
t P
erc
en
tag
e
Req Defects PercentageGoal Requirement Defect Density Percentage
Guidance for Implementing Your Guidance for Implementing Your Own Measurement ProgramOwn Measurement Program
Critical to Success:
• Strong measurement champion
• Strong management support
• Start small – limit your base measures to less than 7
Guidance for Implementing Your Guidance for Implementing Your Own Measurement ProgramOwn Measurement Program
Critical to Success:• Provide clear definitions for base measures
• Measure at a high enough level to identify issues without burdening
• Minimize collection effort through the use of tools and automation
Guidance for Implementing Your Guidance for Implementing Your Own Measurement ProgramOwn Measurement Program
Pitfalls:• Measures that have little value to the program• Not being able to explain how to use the data• Trying to collect too many base measures• Poorly defined, ambiguous base measures• Not creating a change in culture where
managing with measures is the norm
SummarySummary
• The development and sustainment of a measurement program requires top down support
• There must be a champion to continually sell the benefits of a measurement program
SummarySummary
• Champion needs depth and breadth of understanding of measures and the ability to enthusiastically communicate this understanding
• The measures need to be overwhelmingly geared to assisting the program in its day-to-day execution
SummarySummary
• Our measurement program has assisted our organization in dramatically improving the execution of our development programs by quantitatively managing key processes and focusing attention on critical tasks to guide our leads through the development process.