The Developmental Evaluation Framework and its Effect on Test … · 2020. 9. 16. · The...

Distribution Statement A: Approved for public release. Distribution is unlimited..

https://www.CTO.mil @DoDCTO


The Developmental Evaluation Framework and its Effect on Test and Evaluation of a Major Defense Space Acquisition

Mr. Rick ThomasDeputy Director for Space Systems, Developmental Test, Evaluation and Assessments Office of the Deputy Director, Engineering, OUSD (R&E)

37th International Test and Evaluation SymposiumSeptember 16, 2020

dkluzik

Cleared

dkluzik

Slides Only

Distribution Statement A: Approved for public release. Distribution is unlimited..Distribution Statement A: Approved for public release. Distribution is unlimited..

Why We Are Here?

Programs benefit by linking decisions, evaluation, test, resources, and schedule to yield performance information of system capability

The Developmental Test, Evaluation and Assessments (DTE&A) office in OSD created the Developmental Evaluation Framework (DEF) as a planning and execution tool to assist a program with its test and evaluation (T&E) strategy

2


Purpose

Present the value and impact the DEF has had on a space program to deliver Warfighter capability

Describe how the program manager implemented the DEF as a planning and execution tool to assist the program in implementing an effective T&E strategy

Provide lessons learned from the Space Fence experience

3


Space Fence Ground-Based Radar System

Space Fence is a sophisticated system to dramatically improve the way U.S. Space Force identifies and tracks objects in space.

4


Space Fence DT&E Background

Starting in 2012, worked closely with Space Fence program office in producing the first DEF to support the program’s Engineering, Manufacturing, Development (EMD) acquisition phase

– Created DEF and included it in Request for Proposal (RFP)

– Vendor leaders stated the DEF in the RFP let them know what the Government needed, resulting in lower bids from vendors

– Bids were approximately 18-20 percent below Government expectations

– The DEF was incorporated into section 3.2 of the Test and Evaluation Master Plan (TEMP) in support of the Milestone B decision

5


What the DEF Is and Does

The DEF guides development of the DT&E strategy by:

– Focusing the thought process on logically identifying critical program decisions

– Defining the information needed to inform the decisions– Defining the test and modeling and simulation (M&S) events

needed to generate data, which when evaluated provides the knowledge

The DEF design calls for the PM to identify key data for assessing progress toward achieving developmental evaluation objectives.

6


DEF ImplementationTechnology

Maturation &Risk Reduction

Engineering &ManufacturingDevelopment

Production& Deployment

Operations& Support

MaterielSolutionAnalysis

A B CCDD CPD

ValidatedUser NeedsStatement

MDDICD FRP

PM developed DEF to support EMD RFP

USD(R&E)Tech Modernization

Activity Focus

D(DTE&A) Span ofEngagement

Vendors provided DEFs in their EMD

proposal

DEF guided DT&E through integrated

testing

Developmental Evaluation Framework is a living document** Acronyms in Notes

7

Expanding DEF Applicationrapid prototyping & fielding,

mission engineering; enterprise architecture

DEF OriginsDeveloped to assist DoDI

5000.02 ACAT 1 programs

DEF DevelopmentThought process to scope and align an evaluation strategy to

inform decision-making

1.

2.

3.


DEF Methodology

8

Decisions

Evaluation

Test / M&S

Resources

Schedule

Def

ine Inform

Def

ine Data

ExecuteDef

ine

Design of Experiments-Based

Test Design

Define

Plan

- Performance- Interoperability- Cybersecurity- Reliability


Developmental DEF Evolution

9

The Program Office modified the DEF over time to address important changes:– More mature cooperation with the vendor and Government

management and updates to decision points

– The addition of cybersecurity

During this period, the 45th and 47th Test Squadrons stood up as Lead Developmental Test Organizations (LDTO) and learned to execute several Developmental Test Risk Reduction events

The test results of these events were returned to the Program Office and folded into Lockheed Martin’s (LM) monthly metric reporting


Integrated T&E Touch Points

10


Space Fence Reporting

Leadership Team utilized DEF as a tool to identify technical performance metrics required to achieve and retire developmental objectives

– LM: Reported system technical performance measures (TPMs) and provided copies to Government

– Government Program Manager: Reported system TPMs quarterly in Defense Acquisition Executive Summery

– Air Force 46th Test Squadron: Reported status of DEF evaluation measures after contractor and final Government DT&E events

11


Example of Score Card

12


DT&E DEF Retrospective

Space Fence DEF

– The winning contractor stated the DEF lets the contractor know what the government wants, resulting in his vendor’s lower bid (ICOTE,); both vendors bids were approximately 18-20 percent below Government expectations

Government PM lauded the DEF Core Team efforts

– Stated the DEF was a solid and usable product for evaluating design and development as well as system capability (performance)

13


Scorecard Lessons Learned

The scorecard, at every level of leadership (contractor, PM, engineers, and DTE&A), provides a quick glance of how the critical performance metrics were satisfied to show progress-to-plan

The Space Fence scorecard was a solid tool for tracking critical technical performance metrics

By highlighting issues in red, the scorecard easily presented to the vendor and Government developers what needed to be focused on and fixed

Many other programs would benefit from program implementation

14


Upcoming DEF Adaptations

DEF can be adapted to use through different acquisition strategies and pathways:– Adaptive Acquisition Framework – Rapid Acquisition/Prototyping– Software (DevSecOps) – Mission Engineering– Enterprise Architecting

Evolving concept of DEF and Operational Evaluation Framework (OEF) as foundations for a Unified Evaluation Framework (UEF) to increase focus on full lifecycle decision-support, as well as Digital Engineering and Mission-Based Systems Engineering (MBSE) linkage

15


Summary

DEFs represent the intersection of program management, systems engineering, and contractor T&E and DT&E, providing a strong foundation for integrating test efforts

DEF has been applied to over 100 programs supporting major program decisions

Programs like SF and its DEF, from inception to RFP, to vendor selection through to OT&E and IOC, enable timely delivery of capabilities with better buying power

SF program delivered on time, meeting its threshold date for its final Acquisition Program Baseline milestone

DEF served as an effective DT&E tool for the PM and contractor to deliver a fully capable system to Warfighters

16


Acronyms• Capability Development Document (CDD)• Capability Production Document (CPD)• Developmental Evaluation Framework (DEF) • Developmental Test and Evaluation (DT&E)• Director, Developmental Test Evaluation and Assessment (D,DTE&A)• Engineering, Manufacturing, Development (EMD)• Full-Rate Production (FRP)• Ground Based Strategic Deterrence (GBSD)• Initial Capabilities Document (ICD)• The Industrial Committee on Operational Test and Evaluation (ICOTE)• Initial Operational Capability (IOC)• Lockheed Martin (LM)• Materiel Development Decision (MDD)• Next Generation Overhead Persistent Infra-Red (NG OPIR)• Office of the Secretary of Defense (OSD)• Operational Test and Evaluation (OT&E)• Request for Proposal (RFP)• Test and Evaluation (T&E)• Test and Evaluation Master Plan (TEMP)• Technology Maturation & Risk Reduction (TMRR)• Technical performance measures (TPM)• Under Secretary of Defense for Research and Engineering (USD(R&E))

17

The Developmental Evaluation Framework and its Effect on Test … · 2020. 9. 16. · The...

Documents

Transcript of The Developmental Evaluation Framework and its Effect on Test … · 2020. 9. 16. · The...