System Development Performance Measurement Working Group ... · System Development Performance...

SDPMWG - NDIA Systems Engineering Conference

October 26, 2011 1

PSM

System Development Performance

Measurement Working Group Results

Summary

sponsored by the

NDIA Systems Engineering DivisionSystems Engineering Effectiveness Committee(In conjunction with Practical Software and Systems Measurement)

26 October 2011


October 26, 2011 2

PSM A Top Systems Engineering Issue

Technical decision makers do not have the right

information & insight at the right time to support informed

& proactive decision making or may not act on all the

technical information available to ensure effective &

efficient program planning, management & execution.

In September 2010, the NDIA Systems Effectiveness

Committee chartered a working group to identify a small

set of key leading indicators that would help address

this issue. This presentation provides an overview of the

working group process and its results.


October 26, 2011 3

PSMPSM Working Group Goal

Identify potential high value

– measures,

– indicators, and

– methods

for managing programs, particularly in support of

– making better technical decisions and

– providing better insight into technical risk

at key program milestones during

– Technology Development and

– Engineering and Manufacturing Development

for both the acquirer and supplier


October 26, 2011 4

PSM

Identify and

Prioritize

Information

Needs

Top SE

Issues

(2010)

OSD/Industry

Directives,

Policy,

Guidance, and

Initiatives

Select and

Specify

Leading

Indicators

and

Measure

sources

• Detailed concepts

and questions

• Insight into effects

and root causes

• Consensus on

candidate indicators

Refine

Workshop

Results

Document

Results

Prioritized

Information

Needs

Task

report

• Measures

• Supporting

Information per

template

• Recommendations

Candidate

Indicators

• Background

• Brainstorming

• Analysis of Key Industry

Measurement

Resources (Appendix B)

• Group discussion

Guidance and

Findings

Core Team Workshop Post Workshop

Core Team with

Working Group Review

Working Group Process


October 26, 2011 5

PSM Important Information Needs

Highest Priority Information Needs

(Addressed by current results)

Other Information Needs

(To be considered in the future)

Requirements

Interfaces

Architecture

Staffing and Skills

Technical Performance

Technology Maturity

Affordability

Risk Management

Manufacturability

Testability

Requirements Verification and

Validation

Defects and Errors

System Assurance

Process Compliance

Work Product Progress

Facilities and Equipment

Change Backlog

Review Action Item Closure

As Determined by the Workshop


October 26, 2011 6

PSMPSM Indicator Selection Criteria

• Strongly addresses the information need

• Feasible to produce

• Raw data exists and easily processed

• Already frequently utilized (in common use)

• Provides leading or predictive insight

• Applicable to Technology Development (TD)

and Engineering Manufacturing &

Development (EMD) phases


October 26, 2011 7

PSMPSM Recommended Leading Indicators

Information Need Specific Leading Indicator

Requirements Requirements Stability

Requirements Stakeholder Needs Met

Interfaces Interface Trends

Staffing and Skills Staffing and Skills Trends

Risk Management Risk Burndown

Technical Performance TPM Trend (specific TPM)

Technical Performance TPM Summary (all TPMs)

Technical Maturity Technology Readiness Level

Manufacturability Manufacturing Readiness Level

No recommendations at this time for

Affordability and Architecture


October 26, 2011 8

PSM Example: Interface Stability

Indicator Name Interface Trends

Information

Need(s)

Interfaces

Evaluate the growth, change, and correctness of external interfaces.

Question(s)

Addressed

Is the definition of external interfaces correct and complete?

Measurable

Concept

SE activities associated with correctness and completeness (i.e., approved) and

validation of the definition and design of system external non-hardware interfaces.

Leading Insight

Provided

Evaluates the stability and adequacy of the interfaces between the system under

development to other systems to which it provide or receives information to

understand the risks to other activities towards providing required capability, on-

time and within budget.

Base Measures Total Number of External Interfaces at the end of the reporting period (e.g.,

monthly)

Total Number of External Interfaces Completed(determined by the application of

explicit criteria) at the end of the reporting period

Total Number of External Interfaces Not Yet Defined at the end of the reporting

period

Total Number of External interfaces To Be Resolved (interface defined but not

completed; that is, have outstanding issues) at the end of the reporting period

Total Number of External Interfaces planned to be completed by the end of the

reporting period

Total Number of External Interfaces planned to be resolved by the end of the

reporting period

Derived Measures Total Number of External Interfaces = Total Number of External Interfaces

Completed + Total Number of External Interfaces Not Yet Defined + Total Number

of External Interfaces To Be Resolved


October 26, 2011 9


Decision

Criteria

Interpretation

and Usage

The plans should be based on results expected to be achieved at major milestones.

For example, all external interfaces are defined at PDR and all issues are resolved by

CDR.

For unresolved interfaces use a tolerance band around the plan as

Plan Value +/- <some percent> of Plan Value, e.g., Plan Value +/- 10%. The percent

used should be based on historical experience on successful programs.

For interface definitions use a tolerance band around the plan as

Plan Value +/- <some percent> of (Total Interfaces Not Yet Defined)

For both unresolved interfaces and interface definitions, investigate if a tolerance is

exceeded for the latest reporting period or if the trend over the last several reporting

periods is consistently trending toward a tolerance limit.

If values are below the lower tolerance limit or trending towards the lower tolerance

limit, then identify the dependent program activities impacted, define and evaluate the

risks, and take actions to control the exposure.

If values are above the upper tolerance limit or trending towards the upper tolerance

limit, then the correctness and completeness of the definitions should be reviewed to

ensure the quality requirements have been met.

Additional

Considerations

A similar approach can be used for internal non-hardware interfaces., for example

combat systems

“Complete”, as used here, means the interface has been defined and determined to be

complete through analysis or modeling and simulation, but has not been tested

Showing major milestones on the chart is helpful for interpretation.


October 26, 2011 10


Example

"Time Now" is May-11, which is when all interfaces were planned to be completed. However, fifteen remain, with the chart indicating these will be completed within three months. Corrective action was attempted mid-stream but was not sufficient to recover schedule. Since completions were trending toward and crossing the lower threshold much earlier, the basic lesson learned is to take action aggressively when anomalous behavior is first detected.


October 26, 2011 11

PSMPSM Some Things to Keep in Mind

• The intent of the working group was not to

invent new indicators but to determine an

essential set addressing important system

engineering issues

• Organizations need to determine how to

meet the intent of each indicator (obtain the

insights intended). Don’t dwell on the style of

presentation suggested by the examples


October 26, 2011 12

PSMPSM Future Directions

• Determine leading indicators that address the other

important information needs identified by the working group

• Build a relationship model for government and contractor

information focusing on information needs, data alignment

and measures

• Harmonize contractor reporting and government

requirements

• Determine how to establish collaboration among

government required data and repository mechanisms and

contractor and academia repositories


October 26, 2011 13

PSM The Road Ahead

Major SE Tasks

Contractor held Information

Used Internal to Contractor for

Planning and Bidding

Bidding Relationships

• Cost

• Schedule

• Size

• Technical

• Risk

• Quality

Management Information

• Requirements

• Concept Development

• Architecture

• Modeling and Simulation

• Scenarios

• Interfaces

• Algorithms

• Perf. Monitoring and Analysis

• Trade Studies

PHASE I

ITR ASR

MS-A

PHASE II

Requirements

Development

PHASE III

SRR I SRR II SFR

MS-B

PDR-I CDR

PHASE IV

TRR FRR SVR /

PRR

Engineering & Manufacturing

Development & Demonstration

Technology

Demonstration

Prototypes

MDD MS-C

PDR-II

Production &

Deployment

Systems

Development

Performance

Measures

Government and Contractor Data, Measurement, and Information Needs Not Aligned

Regular Reporting Guidance Tied to Strategic

and Tactical Needs

Current Sources for Indicators and Measures

Independent

Parametric

Modeling Tools

Benchmarks Government Repositories

Industry / Academia Models Not Cooperative

Tactical Guidance High Level

CostEstimating

Relationships

Not Focused for Systems Engineering

1

2

3 Data Collection andReporting Requirements

6

5

4

1. Build a relationship model for government and contractor

information

5. Determine Systems Engineering relationships and

activities to information needs

a.Focus on information needs, data alignment, and

measures a.Requirements to needs traceability

2. Harmonize contractor reporting and government

requirements b.Measurement

3. Review and extract existing common data and

information need requirements c.Parametric analysis

4. Identify information gaps and remedies

6. Connect the government required data and repository to

contractor and academia repositories

Roadmap

Tasks


October 26, 2011 14

PSMPSM Final Report – Table of Contents

• Introduction

• Approach

– Working Group

– Information Needs

– Recommended Indicators

• Benchmarks

• Future Directions

• Appendix A: Working Group Participants

• Appendix B: Key Measurement Resources

• Appendix C: Operational Descriptions of

Recommended Indicators


October 26, 2011 15

PSMPSM Current Status

• Working Group Report has been submitted

to the NDIA Systems Engineering Division

for Approval

• Expected availability: November


October 26, 2011 16

PSMPSM Points of Contact

• For further information on the Working Group, please contact any of the following:

– Mr. Peter McLoone, NDIA SED, Working Group Industry Co-chair ([email protected])

– Mr. Martin Meth, representative for OUSD/DDR&E/MPS ([email protected]) and Working Group OSD Co-chair

– Mr. Garry Roedler, NDIA SED, Working Group Industry Adviser ([email protected])

– Ms. Cheryl Jones, PSM, Working Group Collaboration Co-chair ([email protected])

– Mr. Bob Rassa, NDIA Systems Engineering Division (SED) Chair ([email protected])

– Mr. Alan Brown, NDIA SED Systems Engineering Effectiveness Committee (SEEC) Chair ([email protected])

– Mr. Geoff Draper, NDIA Systems Engineering Division (SED) Vice-chair ([email protected])

mailto:[email protected]








October 26, 2011 17

PSM

Backup


October 26, 2011 18

PSM Requirements & Interfaces

10001100120013001400150016001700

Jan-03

Mar-03

May-03

Jul-03

Sep-03

Nov-03

Jan-04

Mar-04

May-04

Jul-04

Sep-04

Nov-04 N

um

ber

of

Req

uir

emen

ts

Per

cen

t C

han

ge

Requirements StabilitySystem Requirements

Percent Adds Percent Deletes Percent Modifications

Target Total Requirments

SRR PDR CDR

12 12 12 12

15 15 15 15 15 15

18 18 18 18 18 18 18 18 18 18

0

5

10

15

20

25

30

35

40

0

2

4

6

8

10

12

14

16

18

20

Jan-

10

Feb-

10

Mar

-10

Apr-1

0

May

-10

Jun-

10

Jul-1

0

Aug-

10

Sep-

10

Oct-1

0

Nov-

10

Dec-

10

Jan-

11

Feb-

11

Mar

-11

Apr-1

1

May

-11

Jun-

11

Jul-1

1

Aug-

11

Valid

atio

n Ac

tiviti

es

MOE

s / KP

Ps

Stakeholder Needs MetTechnical Measures (MOEs/KPPs) Satisfied and Validation Activities Completed

MOEs Not Satisfied

MOEs Partially Satisfied

MOEs Fully Satisfied

Validation Activities (Plan)

Validation Activities (Actual)

0

20

40

60

80

100

120

140

160

Jan-

10

Feb-

10

Mar

-10

Apr-

10

May

-10

Jun-

10

Jul-1

0

Aug-

10

Sep-

10

Oct

-10

Nov-

10

Dec-

10

Jan-

11

Feb-

11

Mar

-11

Apr-

11

May

-11

Jun-

11

Jul-1

1

Aug-

11

External Interface Completion

Total Number of External Interfaces Completed

Total Number of External Interfaces Not Yet Defined

Total Number of External interfaces To Be Resolved

Total Number of External Interfaces Planned To Be Completed

Lower Threshold for Interface Completion

Upper Threshold for Interface Completion

Total Number of External Interfaces Planned To Be Resolved


October 26, 2011 19

PSM Staffing & Risk

0

10

20

30

40

50

60

Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 Apr-11 May-11 Jun-11 Jul-11 Aug-11 Sep-11 Oct-11

SE Staff(FTEs)

SE Staffing

SE Staf f ing - Actual Upper Threshold Lower Threshold

0

5

10

15

20

25

30

35

Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 Apr-11 May-11 Jun-11 Jul-11 Aug-11 Sep-11 Oct-11

SE Staff(Average

Experience)

SE Staff Experience

SE Staf f ing - Actual Upper Threshold Lower Threshold


October 26, 2011 20

PSMTechnical Performance and Maturity

Manufacturability

Name Respon

sible

Position

/IPT

KPP

or

KSA

Perfor

mance

Spec.

PDR

Status

Actual

MS B

Status

Actual

CDR

Status

Actual

MS C

Status

Planned

FRP

Status

Planned

Aerodynamic Drag (count)

SE IPT <222 225 223 220 187 187

Thermal Utilization (kW) SE IPT <60 56 59 55 51 50

Electrical Power Usage (kW)

SE IPT <201 150 185 123 123 123

Operating Weight (lb) SE IPT <99,000 97,001 101,001 97,001 85,540 85,650

Range (nm) SE IPT >1,000 1,111 1,101 1,111 1,122 1,130

Average Flyaway Unit Cost (number)

SE IPT <1.5 1.3 1.58 1.37 1.35 1.32

*Note: Margin is 10%

System Milestone / Technical Review

TRL (Plan)

TRL (Actual)

MRL (Plan)

MRL (Actual)

Comments / Risk Action Plan

ITR TRL 2 TRL 3 MRL 2 MRL 2 Analysis model based on ABC study

ASR TRL 3 TRL 3 MRL 3 MRL 3 Lab validation of ASIC mfg concept

MS A TRL 4 TRL 3 MRL 4 MRL 3 Study funding delayed 30 d. TRA completed.

SRR TRL 5 TRL 4 MRL 5 MRL 3 Mechanical packaging ICD validation issues. Supplier facility contention elevated.

SFR TRL 6 TRL 5 MRL 6 MRL 5 Prototyped XYZ subsystem w/ test bed I/F. Investigating low yield on lot 6 wafer fab.

PDR / MS B TRL 6 TRL 6 MRL 6 MRL 6 Dwgs on plan. Tin whisker fab issue ok. Producibility plan approved.

CDR TRL 7 MRL 7 Evaluating alternative µW feeds (risk #23).

TRR TRL 7 MRL 8

SVR (FCA PRR) TRL 7 MRL 8

MS C TRL 8 MRL 9

FRP Decision Review

TRL 9 MRL 10

System Development Performance Measurement Working Group ... · System Development Performance...

Documents

Transcript of System Development Performance Measurement Working Group ... · System Development Performance...