I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Headquarters U.S. Air Force

Systems Engineering and Test & Evaluation Engagement

1

3 March 2010Col Shawn Shanley

SAF/AQR

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Common Themes

Change the way we think about requirements – upfront SE and T&E participation required

Perception test done after the fact, temptation to skip T&E – T&E voice in Development Planning activity Pre-MS A insufficient

Get involved much earlier; address concepts/solutions to ensure requirements unambiguous, testable, feasible, realistic, & measurable

Address not just system under test, but system of systems for effectiveness

Collaboration with acquisition community and processes must be early to influence strategies/plans; don’t forget about reliability growth, info assurance, sustainment test needs

Show DP & PMs value T&E can bring “be part of solution” – end of cycle too late; ensure right people/right skills represent T&E

2

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Main Theme:“Upfront, Early, & Collaborative” Engagement

WSARA of 2009 mandates D,SE and D,DT&E collaboration AF response addressed SE & DT&E strengths & weaknesses

AF has 2 key initiatives to provide for the influences of these functional disciplines earlier in lifecycle

1. Development Planning (DP) Process Concept Characterization & Technical Description (CCTD)

2. Air Force Program Support Review (AF PSR) Independent Tech Assessment for MDAP milestone reviews

Institutionalizing in policy and directive guidance Major 62 series (SE) policy and guidance development effort

3

T&E Influences/Integration with SE are in AF Thrust Areas

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Early SE and T&E

4

CBAJoint Concepts

MS CMS B

Strategic Guidance

MS A

ICD Engineering and Manufacturing CPD

Production and Deployment O&S

MDD

MaterielSolutionAnalysis

JCIDS 5000.02

TechnologyDevelopment

CDD

T&E Engagement• T&E consideration during program formulation• Assessment of Operational Test Readiness• TEMP focus on risk assessment criteria• Technology maturity and integration risk• Independent DT assessment to MDA • TEMP and SEP criteria • Integrated Test

TEMPTES

T&E WIPT

Earlier Involvement

T&E EngagementConcept/Solution selection influenceSEP influenceTES and TEMP influenceT&E risk assessmentDT&E execution planningTRL data evaluation

Development Planning Process AF Program Support Review Process (AF PSR)

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Early SE and T&E in Development Planning

5

CBAJoint Concepts

MS CMS B

Strategic Guidance

MS A

ICD Engineering and Manufacturing CPD

Production and Deployment O&S

MDD

MaterielSolutionAnalysis

JCIDS 5000.02

TechnologyDevelopment

CDD

T&E Engagement• T&E consideration during program formulation• Assessment of Operational Test Readiness• TEMP focus on risk assessment criteria• Technology maturity and integration risk• Independent DT assessment to MDA • TEMP and SEP criteria • Integrated Test

TEMPTES

T&E WIPT

Earlier Involvement

T&E EngagementConcept/Solution selection influenceSEP influenceTES and TEMP influenceT&E risk assessmentDT&E execution planningTRL data evaluation

Development Planning Process AF Program Support Review Process (AF PSR)

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Development Planning(prospective materiel solutions) MSA

Capability Planning(DOTMLPF)

6

ICDDCR

AoA Study Guidance

AFRB

AoAMDD

JROC

Trade Space Characterization

User Needs &Validated Requirements• JCIDS outputs

(if available)• Capability

shortfall• Others

AFROCJROC

Characterization

Candidate Solution Sets Characterization

Concept Characterization Review

Initial Concepts Review

Candidate Solution Sets Selection

CCTDs

Programmatic Analyses

Final Concepts Review

Release Approval

AFROC

Development Planning SE Process: T&E Engagement

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Concept Characterization & Technical Description (CCTD) Elements

Mission / Capability Need Statement / CONOPS

Concept Overview

Trade Space Definition / Characterization

Studies, Analyses, Experiments

Concept Characterization / Design

Program Characterization

Risk Assessment

DOTMLPF Implications

Conclusions (Capability Description; Traceability to Need Statement)

7

Major opportunities for

early T&E program influence

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

SE and T&E in AF Program Support Review

8

CBAJoint Concepts

MS CMS B

Strategic Guidance

MS A

ICD Engineering and Manufacturing CPD

Production and Deployment O&S

MDD

MaterielSolutionAnalysis

JCIDS 5000.02

TechnologyDevelopment

CDD

T&E Engagement• T&E consideration during program formulation• Assessment of Operational Test Readiness• TEMP focus on risk assessment criteria• Technology maturity and integration risk• Independent DT assessment to MDA • TEMP and SEP criteria • Integrated Test

TEMPTES

T&E WIPT

Earlier Involvement

T&E EngagementConcept/Solution selection influenceSEP influenceTES and TEMP influenceT&E risk assessmentDT&E execution planningTRL data evaluation

Development Planning Process AF Program Support Review Process (AF PSR)

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 9

Air ForceProgram Support Review

DP #1—Organize, train, equip-Mentor/advise program office-TRA: ID independent rvw team-Early risk identification-Gain program understanding

DP #2—Provide early status-Provide preliminary report to PM-TRA: approve CTEs-Address risks-Cross flow information

DP #3—Finalize products-Out brief findings to PM-TRA: approve TRLs-Capture open risks-Complete MS deliverables

MS (-11) MS (-3)MS (-10) MS (-9) MS (-8) MS (-7) MS (-6) MS (-4)MS (-5) MS (-2) MS (-1m) MSMS (-12)

DP 2 DP 3 AFRB DAB

Early technical engagement to seamlessly mentor & advise, and independently assess & capture risks…so PMs can act

Guidance &Planning mtg DP 1

DAPSTechnology

Manufacturing

Logistics

ESOH

Eng Integration

Prepare/review/complete PESHE documentation

Conduct risk assessment

Identify CTEs

Identify critical processes

Review logistics readiness

Assess TRLs

SPO address findings

Assess MRLs

Notionaltimeline

OIPT

T&E assessmentPSR assessment integration and findings validation

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Institutionalizing—Policy & Guidance

AFPDs 61-1 Management of Science and Technology (SAF/AQR) * 62-1 Engineering (SAF/AQR) * 63-1/20-1 Acquisition & Sustainment Life Cycle Management (SAF/AQX)

AFIs 10-601 Operational Capability Requirements (AF/A5R) ** 61-xx Research and Development (SAF/AQR) * 62-1xx Life Cycle Engineering (SAF/AQR) * 63-101 Acquisition & Sustainment Life Cycle Management (SAF/AQX) **

Non-directive Guidance AFPAM 63-128 Acquisition & Sustainment (SAF/AQX) Early Systems Engineering Guide (SAF/AQRE) * Development Planning Guide (AFMC/A5C) * CCTD Guide (SAF/AQRE, AFIT/SY (Center for Systems Engineering)) * AF PSR Guide (SAF/AQRE, AFMC/EN, Center ENs, SMC/EA)

* - update or first issuance in CY10** - update currently in coordination

Coordinated with 99 Series, T&E policy

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Challenges

Resources, resources, resources – People, TDY, Time, Funding?

Discipline and stability in processes developed – will we violate them?

Trend towards quicker/urgent fielding, quick capability delivery –short cuts likely inconsistent with discipline seeking

Will T&E organizations set early engagement as a priority –dedicate seasoned testers to early lifecycle planning phases?

Trade offs between what is enough testing and appropriate cost for programs?

Do services begin to POM for this early T&E engagement, specifically for Pre-MS A activity?

Develop the people, need strategic plan to attract right skills/train workforce we have?

11

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Summary

Development Planning (DP) Process & Air Force Program Support Review (AF PSR) Process support “upfront , early, & collaborative” involvement of SE & T&E in acquisition

Properly considered T&E requirements planning & influences on concept/solution selections Pre-MDD & follow on acquisition program development strategies Post-MDD will result in improved program execution & validation

Standardized processes and products are being developed and institutionalized in policy and directive guidance Major 62 series (SE) policy and guidance development effort Coordinated w/ 99 series (T&E) policy

12

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Air Force Acquisitions

13