1Deputy Under Secretary of the Army Test and Evaluation Office

Big Data in

Test and Evaluation

Prepared for ITEA Annual Symposium

4 October 2016

David Jimenez Director, Army Test and Evaluation

2Deputy Under Secretary of the Army Test and Evaluation Office

Big Picture The tests we conducted over the past 20 years are not

representative of the next 20 years

Hypersonics

Artificial Intelligence

Cognitive Workload

Contested Information

Environments

Technology Parity

Virtual Environments

Autonomy

Cryptology

Requires Continued Investment in Infrastructure & People

Future T&E

Big Data

3Deputy Under Secretary of the Army Test and Evaluation Office

A Big Data Perspective

Small Data Sets

Manual Observation

Little Insights

The Good Old Days

Small

Data

Small Analysis

The Challenge

Big

Data

How do I Determine ?

?

4Deputy Under Secretary of the Army Test and Evaluation Office

Force 2025 and T&E

Service OTAs must determine

whether or not systems are

effective, suitable, and survivable

in support of unified land

operations in an operational

environment dominated by:

• Increased momentum of

human interaction

• Potential overmatch

• Importance of cyber and

space

• Dense urban areas

(megacities)

• Ubiquitous media

• WMD proliferation

• CEMA !

• Big Data !

Increasing complexity on the battlefield increases complexity in T&E. Demand

for data – and the means to use it effectively – is also increasing.

See TRADOC PAM 525-3-1 “The U.S. Army Operating Concept (AOC): Win in a Complex World”

found at: http://www.arcic.army.mil/Concepts/operating.aspx CEMA – Cyber Electromagnetic Activities

http://www.arcic.army.mil/Concepts/operating.aspx

5Deputy Under Secretary of the Army Test and Evaluation Office

Big Data Causing An Evolution in T&E

Yesterday Today

Discrete data sets (usually associated with a single test); small overall file size

Large data sets collected over a test program (may include data from contractor tests, simulators, hardware/software-in-the-loop laboratories, M&S, fielded system, and similar systems)

Meaning derived from expert observations Meaning derived from continuous observation

Workforce has expertise in the system under test

Workforce has expertise in analytics

Evaluation products consumed by small, specialized audience

Evaluation products consumed by broad audience with diverse interests

Central evaluation question: “Did it meet requirements?”

Central evaluation question: “What are the system’s strengths and limitations over the range of conditions found on a complex, interoperating battlefield?”

To focus on the “Why and How” of a system’s operational effectiveness, operational suitability, and survivability, increases the demand for deep analytics.

6Deputy Under Secretary of the Army Test and Evaluation Office

Leverage Advances in

Instrumentation Capabilities

T&E Big Data Challenges

T&E Big Data

Free and shared among responsible practitioners

Amounts of data straining analytical resources

Support model validations

More Reliance on Supercomputing

Need tools to make short order of analysis - visualization, sage, and frame capture

T&E Cadre of the Future Requires Data Scientists and Data Analysts

7Deputy Under Secretary of the Army Test and Evaluation Office

Big Data Changed Everything

Implications for Army Operating Concept and Force 2025: The Force 2025 Soldier will not have known a world without analytics.Our Surroundings

We expect to be able to access analytics – instantly and on demand -- to measure and understand our complex world.

Our neighbors

Our interests

Our health

Our wealth

8Deputy Under Secretary of the Army Test and Evaluation Office

2025 T&E and Big Data Goals Goals: • Utilize knowledge, information, and

data to achieve core mission and business objectives.  Faster, more Accurate Decision-Making

 Cost Optimization

 Quicker Responses to Requests for Information

 More Holistic Test and Evaluation

 Automated tracking items or status

• Make useful big data capabilities available to everyone, but tailored to specific needs.

Sustainment of data for long term

use (Archival)

Discoverability and Access to data

Analytics of historical and

current information

Derive context to inform decision

making

Common Core Requirements:

2025 T&E

Leveraging Historical Data

Faster, More Sophisticated

Analytical Tools

Modeling & Simulation

Design of Experiments

Cloud Computing

9Deputy Under Secretary of the Army Test and Evaluation Office

Data Driven Deep Dive Analysis

 Incident Overview – Middle East Course near mile marker 9.4 – 1L and 1R (Front, Left & Right) Half-shafts broken

 During that test week, vehicle completed 4 passes of this section of Middle East – 1 pass on June 5 (date incident occurred) – 3 passes on June 6

 Large spike in left front spindle, frame, and driver acceleration occurred approximately 10s prior to vehicle stopping on course due to incident.

Sheared Right Side Half Shaft Sheared spline inside Left Hub Cartridge

10Deputy Under Secretary of the Army Test and Evaluation Office

Considerations for Big Data Analytics

Pros: Cons: Available data may be underutilized due to awareness gaps.

• What capabilities already exist? • What lessons have already been learned? • What opportunities exist?

Utilizing big data requires careful planning: • Information system and data management design • Data Collection, Reduction, Analysis (DCRA) • Archiving and sustainment

Utilizing big data requires appropriate tools. • Even small data sets are unmanageable without right tools • Tool development requires planning, time, and resources

“I paid for all this data. What can I do with it?”

Awareness

Planning

Tools

11Deputy Under Secretary of the Army Test and Evaluation Office

The Big Data Community

Field

T&E

User Needs

Materiel Development

S&T

Big Data

(6.1/6.2/6.3)

“Big Data” is a common resource of the Services’ analytical community.

Diverse analytical organizations contribute to and draw from it:

• data acquisition methods • computational resources • models, simulations, laboratories, tools • historical data • expertise

Important questions going forward: • Who manages it for stakeholders? • Who sustains it? • How do we establish business rules for

increased collaboration? • Can we obtain synergies through

collaboration?

12Deputy Under Secretary of the Army Test and Evaluation Office

Performance Test Data Integrated Concept Study *

PURPOSE. Address Army’s need for timely access to T&E data while aligning Army’s storage infrastructure and protocols with DODI 5000.02.

SCOPE.

 Conduct a cost-benefit analysis to determine breath & depth of data to be stored & resources

 Evaluate sensitivity of results to assumption changes and identify risks associated with changes

RESPONSIBILITES.

 AMSAA will appoint a Study Director

 Study Advisory Group (SAG) will oversee the planning & conduct of the study

 SAG Composition: Senior Executive / General Officer from: ASA(ALT) DUSA-TE, CIO/G-6,

DCS, G-3/5/7, AMC *RDECOM, ARL, & AMSAA), TRADOC CAA, & DTIC

* HQDA (DCS, G-3/5/7) Memo, subject: Performance Test Data Integrated Concept Guidance and Directive Study, 26 Feb 16

13Deputy Under Secretary of the Army Test and Evaluation Office

Value of ‘Deep’ Knowledge

EXAMPLE

Bad Event

Increased Survivability

Analysis by Service OTAs

& Others

14Deputy Under Secretary of the Army Test and Evaluation Office

Big Data Analysis Approach

Week’s worth of test data (~100 GB) processed within 2-3 days

1) Download vehicle data files

2) Process data for each week of test

3) Review report for reliability highlights

Run Course Identification Scripts GPS coordinates used to ID course

Generates summary file containing metadata for each file (Vendor,

Vehicle ID, Course, Date, Miles, &

Hours

Run Data Collector Scripts Combine files from similar vehicle, course and date

Generates files with concatenated channel data and flags the files

containing incomplete data

Run Report Generator 1) Displays summary of mileage and hours 2) Compares accelerations, temperatures, and speeds, across multiple