H e l p i n g Wa r f i g h t e r s a d a p t

UNCLASSIFIED

UNCLASSIFIED

(U) Rapid Test and Evaluation

18 January 2018Thomas C. Clutz, PhD

2

UNCLASSIFIED

UNCLASSIFIED

(U) Background

(U) JIDO enables DoD actions to counter improvised-threats

with tactical responsiveness and, through rapid solution

development and delivery, to support the Combatant

Commands‘ efforts to prepare for and adapt to battlefield

surprise in support of counter-terrorism, counter-insurgencies,

and other related mission areas, including countering

improvised explosive devices.

(U) “Rapid solution development and delivery” requires a rapid test capability

3

UNCLASSIFIED

UNCLASSIFIED

(U) Enabling Rapid Fielding

Vet Deploy Sustain

CONUS Based Testing

Theater Support Testing

In-Theater Testing

Forward Operations

Support

JIDO Risk Mitigation

4

UNCLASSIFIED

UNCLASSIFIED

(U) Vocabulary

• (U) Test – a formal activity designed to provide an official report on/certification for a system’s capabilities- Government sponsored

• (U) Experiment – an activity intended to provide understanding about some (perhaps all) of a system’s capabilities– Government/vendor sponsored

• (U) Demonstration – an activity designed to showcase a system’s capabilities– Vendor sponsored

(U) A specific purpose provides a specific outcome

5

UNCLASSIFIED

UNCLASSIFIED

• Needs for rapid test execution:• (U) Pre-defined assessment processes

– (U) Establish an understanding (if possible, a specific plan) of how a particular technology needs to be tested

– (U) Includes an understanding of classification• (U) Pre-positioned equipment (systems and devices)

– (U) Have all the relevant equipment pre-positioned at the test range

– (U) Have a common representation of a threat device (ideally across all test centers involved in testing a particular capability)

• (U) Pre-positioned funding– (U) Front-load test activity coffers; retain expenditure authority

(U) Rapid Test & Evaluation Principles

(U) The more time spent in test preparation,the less time required for test execution

6

UNCLASSIFIED

UNCLASSIFIED

• Common standards and methodologies

• Statistically relevant testing

• Operational realism

• Consistent T&E across DoD

• Comparability of test results

Consistent, Comparable Results

(U) Test Protocols

7

UNCLASSIFIED

UNCLASSIFIED

(U) Test in Integrated Environments

(U) Realistic testing in integrated, realistic environments increases the chances of utility in a theater environment

8

UNCLASSIFIED

UNCLASSIFIED

M&S can save significant time in testing-but the model must be useful

•Lab testing reveals issues

•Limited realism

Laboratory

I2WD

Pt Mugu

•Most data in a given time

•Difficult to account

for all effects

Modeling &Simulation

•Most realistic

•Least cost effective

APG YPG

China Lake

WSMR

Theater

OpenAir

RF Propagation

CraneVehicle

IntegrationNear Earth

Propagation

(U) Integration of M&S with T&E

9

UNCLASSIFIED

UNCLASSIFIED

• (U) JIDO has centralized the conduct of all JIDO test activities in a single office

• (U) The JIDO Integrated Evaluation Team (JIET) is the nucleus of JIDO test planning– (U) Membership includes: JIDO T&E (facilitator)

Service representatives

• OTA, operators

Program manager

Scientists

(U) Rapid Test & Evaluation Team

(U) Interagency acceptance of the final test plan

10

UNCLASSIFIED

UNCLASSIFIED

(U) Rapid Test & Evaluation Methodology

JIDO Initiative Evaluation Team

Initiative Evaluation Plan

Traceability Matrix

Individual Evaluations

Intro

Assessment Overview

Assessment Issues

MOEs

M&S Strategy

Traceability Matrix

Threat & Intel

Competitive Strategies Group

Data Collection Events

Suitability, Survivability & Design Issues

Estimate of Supportability

Eval Schedule

Executive Summary

Background

PurposeRequirementSystem OverviewSystem Evaluation TimelineData Sources & References

Findings, Analysis & Discussion

EffectivenessSuitabilitySupportability

Recommendations

OperationsSupport ArchitectureFuture Planning

Success Stories

Provide JIDO leadership with

recommendations and the “so what” analysis

based on all data collection

Data Collection Events

Data Elem

ents/Metrics

All initiatives will get an IEP which will

outline the depth of the test efforts

Tracker for all data metrics of interest

correlated with data collection schedule. What will we know,

when?

11

UNCLASSIFIED

UNCLASSIFIED

• (U) Rapid solution development and deliverynecessitates the establishment of rapid test and evaluation capabilities to properly assess each rapid initiative

• (U) Preparation time saves execution time– Established test protocols/plans – Fully equipped test ranges– Prepositioned funding– Useful modeling and simulation– Obtain stakeholder buy-in for the test plan

• (U) Integrated test facilities provide significant value

(U) Summary

(U) A simple, direct approach can be very effective

12

UNCLASSIFIED

UNCLASSIFIED

13

UNCLASSIFIED

UNCLASSIFIED

• (U) JIDO has centralized the conduct of all JIDO test activities in a single office– (U) Theater Support Test (TST) program – provides on-call T&E

capabilities to Combatant Commands (CCMDs) and Services to address current RFIs in support of TTP development, and the fielding of material solutions

– (U) Op SABRE – annual force protection electronic countermeasures compatibility test series conducted in cooperation with FVEY partners

– (U) T&E support to JIDO Rapid Acquisition Initiatives – rapid evaluation support of JIDO initiative development to address warfighter requirements against improvised threats devices

(U) Rapid Test & Evaluation Examples

14

UNCLASSIFIED

UNCLASSIFIED

• (U) Theater Support Test– (U) Open-air, hardware-in-the-loop laboratory, modeling & simulation

test capabilities used to rapidly respond to CCMD RFIs– (U) 500+ RFIs answered to provide test data to support TTP changes Counter UAS (C-UAS) system performance and compatibility

• (U) Rapid fielding of new C-UAS capabilities necessitates the need to rapidlyrespond to performance and compatibility RFIs from CCMDs and Services

• (U) Ex. Warfighter SIG endorsement of OIR Memo directing JIDO to conduct C-UAS compatibility testing for DoD

CREW performance & compatibility trials AEA performance & compatibility Rifleman Radio Green Hash – rapidly executed testing in order to confirm

material and TTP solutions• (U) Op SABRE

– (U) Conducted since 2005 among the FVEY partners to respond to national,CCMD and Service RFIs and inform TTP development to mitigate compatibilityconcerns

(U) Rapid Test & Evaluation

15

UNCLASSIFIED

UNCLASSIFIED

• (U) JIDO Rapid Acquisition Initiatives– (U) MARVIN AEB responded immediately to RFIs resulting from operational

assessments in CENTCOM AOR• Testing replicated false alarm conditions and provided data and

recommended TTP changes to mitigate the impact to performance• Testing explored the use of MARVIN as a solution beyond the designed

problem set– (U) Chameleon Rapidly responded to CCMD RFIs to test and assess mitigation measures

for specific improvised threat; test data confirmed viability of simple riskmitigation measure that was adopted

(U) Rapid Test & Evaluation

16

UNCLASSIFIED

UNCLASSIFIED 12

(U) Warfighter Support Portal (WaSP)

• (U) Origins of WaSP– Theater Support Web Tool (TSWT)

was created as a reachback capabilityto operators in theater

– Offers a quick turnaround on requests for information(RFIs)

– Allows storage, retrieval, and access to RFI data• (U) Tools

– Drupal and MySQL• (U) Key Products

– Maintenance of a direct portal link withtheater to enable rapid test requests

– Complete test report archive– JDEP account administration– C-UAS Compatibility Chart hosting– C-SUAS Working Group

(U) https://wasp.navair-rdte.navy.mil

17

UNCLASSIFIED

UNCLASSIFIED

RAM

P:

CE

Ente

rpris

e pr

oduc

es a

ctio

nabl

e sy

stem

per

form

ance

info

rmat

ion

JEM

S &

othe

r In

fo S

harin

g

Test Data

Test Agencies:

e.g., ATEC, other DT, OA

services

Test

Pla

ns

da

ta

OA DataOperational

Units

• IEP• Traceability Matrix• CE Report:

• Cumulative Relevant T&E results• Proven capes/limitations• Risks/Issues• Recommendations

• Baseline capability

Evaluation Div Chief

Eval Div Tech Advisor

Test & Evaluation Enterprise

IET

IET

Cor

e

CE Programs

Team (IET Lead)

J9

J2

J3

TEC

Dev Rep

M&S

AST Chair

PM Rep

Appr

oval

Cha

in fo

r R

elea

se b

eyon

d C

E

IET Coordinates within context of IPTs to address metrics testing with respect to operational risk, Red/blue teaming, leveraging M&S, Intel, and

CONOPS

Develops and Continuously

Updates

JIDO System

Integrator

Initiative PMs

T&E Coord & MGT

Metrics

CE Test Operations

Team

CE Infrastructure

Team

PrioritiesJIDO

Capability Manager

Eval Data

Current ProductsR

elea

se

Senior Leader/FDO

System D

ata

Initiative&

ThreatD

ataC

E An

alys

is T

eam