1Deputy Under Secretary of the Army Test and Evaluation Office
Big Datain
Test and Evaluation
Prepared for ITEA Annual Symposium
4 October 2016
David Jimenez Director, Army Test and Evaluation
2Deputy Under Secretary of the Army Test and Evaluation Office
Big PictureThe tests we conducted over the past 20 years are not
representative of the next 20 years
Hypersonics
Artificial Intelligence
Cognitive Workload
Contested Information
Environments
Technology Parity
Virtual Environments
Autonomy
Cryptology
Requires Continued Investment in Infrastructure & People
Future T&E
Big Data
3Deputy Under Secretary of the Army Test and Evaluation Office
A Big Data Perspective
Small Data Sets
Manual Observation
Little Insights
The Good Old Days
Small
Data
Small Analysis
The Challenge
Big
Data
How do IDetermine ?
?
4Deputy Under Secretary of the Army Test and Evaluation Office
Force 2025 and T&E
Service OTAs must determine
whether or not systems are
effective, suitable, and survivable
in support of unified land
operations in an operational
environment dominated by:
• Increased momentum of
human interaction
• Potential overmatch
• Importance of cyber and
space
• Dense urban areas
(megacities)
• Ubiquitous media
• WMD proliferation
• CEMA !
• Big Data !
Increasing complexity on the battlefield increases complexity in T&E. Demand
for data – and the means to use it effectively – is also increasing.
See TRADOC PAM 525-3-1 “The U.S. Army Operating Concept (AOC): Win in a Complex World”
found at: http://www.arcic.army.mil/Concepts/operating.aspx CEMA – Cyber Electromagnetic Activities
5Deputy Under Secretary of the Army Test and Evaluation Office
Big Data CausingAn Evolution in T&E
Yesterday Today
Discrete data sets (usually associated with a single test); small overall file size
Large data sets collected over a test program (may include data from contractor tests, simulators, hardware/software-in-the-looplaboratories, M&S, fielded system, and similar systems)
Meaning derived from expert observations Meaning derived from continuous observation
Workforce has expertise in the system under test
Workforce has expertise in analytics
Evaluation products consumed by small, specialized audience
Evaluation products consumed by broad audience with diverse interests
Central evaluation question: “Did it meet requirements?”
Central evaluation question:“What are the system’s strengths and limitations over the range of conditions found on a complex, interoperating battlefield?”
To focus on the “Why and How” of a system’s operational effectiveness, operational suitability, and survivability, increases the demand for deep analytics.
6Deputy Under Secretary of the Army Test and Evaluation Office
LeverageAdvances in
InstrumentationCapabilities
T&E Big Data Challenges
T&EBig Data
Free and shared among responsible practitioners
Amounts of data straining analytical resources
Support model validations
More Reliance on Supercomputing
Need tools to make short order of analysis -visualization, sage, and frame capture
T&E Cadre of the Future Requires Data Scientists and Data Analysts
7Deputy Under Secretary of the Army Test and Evaluation Office
Big Data Changed Everything
Implications for Army Operating Concept and Force 2025: The Force 2025 Soldier will not have known a world without analytics.Our Surroundings
We expect to be able to access analytics – instantly and on demand -- to measure and understand our complex world.
Our neighbors
Our interests
Our health
Our wealth
8Deputy Under Secretary of the Army Test and Evaluation Office
2025 T&E and Big Data GoalsGoals:• Utilize knowledge, information, and
data to achieve core mission and business objectives. Faster, more Accurate Decision-Making
Cost Optimization
Quicker Responses to Requests for Information
More Holistic Test and Evaluation
Automated tracking items or status
• Make useful big data capabilities available to everyone, but tailored to specific needs.
Sustainment of data for long term
use (Archival)
Discoverability and Access to data
Analytics of historical and
current information
Derive context to inform decision
making
Common Core Requirements:
2025 T&E
Leveraging Historical Data
Faster, More Sophisticated
Analytical Tools
Modeling & Simulation
Design of Experiments
Cloud Computing
9Deputy Under Secretary of the Army Test and Evaluation Office
Data Driven Deep Dive Analysis
Incident Overview– Middle East Course near mile marker 9.4– 1L and 1R (Front, Left & Right) Half-shafts broken
During that test week, vehicle completed 4 passes of this section of Middle East– 1 pass on June 5 (date incident occurred)– 3 passes on June 6
Large spike in left front spindle, frame, and driver acceleration occurred approximately 10s prior to vehicle stopping on course due to incident.
Sheared Right Side Half Shaft Sheared spline inside Left Hub Cartridge
10Deputy Under Secretary of the Army Test and Evaluation Office
Considerations for Big Data Analytics
Pros: Cons:Available data may be underutilized due to awareness gaps.
• What capabilities already exist?• What lessons have already been learned? • What opportunities exist?
Utilizing big data requires careful planning:• Information system and data management design • Data Collection, Reduction, Analysis (DCRA)• Archiving and sustainment
Utilizing big data requires appropriate tools.• Even small data sets are unmanageable without right tools• Tool development requires planning, time, and resources
“I paid for all this data. What can I do with it?”
Awareness
Planning
Tools
11Deputy Under Secretary of the Army Test and Evaluation Office
The Big Data Community
Field
T&E
User Needs
Materiel Development
S&T
Big Data
(6.1/6.2/6.3)
“Big Data” is a common resource of the Services’ analytical community.
Diverse analytical organizations contribute to and draw from it:
• data acquisition methods• computational resources• models, simulations, laboratories, tools• historical data• expertise
Important questions going forward:• Who manages it for stakeholders?• Who sustains it?• How do we establish business rules for
increased collaboration?• Can we obtain synergies through
collaboration?
12Deputy Under Secretary of the Army Test and Evaluation Office
Performance Test DataIntegrated Concept Study *
PURPOSE. Address Army’s need for timely access to T&E data while aligning Army’s storage
infrastructure and protocols with DODI 5000.02.
SCOPE.
Conduct a cost-benefit analysis to determine breath & depth of data to be stored & resources
Evaluate sensitivity of results to assumption changes and identify risks associated with changes
RESPONSIBILITES.
AMSAA will appoint a Study Director
Study Advisory Group (SAG) will oversee the planning & conduct of the study
SAG Composition: Senior Executive / General Officer from: ASA(ALT) DUSA-TE, CIO/G-6,
DCS, G-3/5/7, AMC *RDECOM, ARL, & AMSAA), TRADOC CAA, & DTIC
* HQDA (DCS, G-3/5/7) Memo, subject: Performance Test Data Integrated Concept Guidance and Directive Study, 26 Feb 16
13Deputy Under Secretary of the Army Test and Evaluation Office
Value of ‘Deep’ Knowledge
EXAMPLE
Bad Event
IncreasedSurvivability
Analysis byService OTAs
& Others
14Deputy Under Secretary of the Army Test and Evaluation Office
Big Data Analysis Approach
Week’s worth of test data (~100 GB) processed within 2-3 days
1) Download vehicle data files
2) Process data for each week of test
3) Review report for reliability highlights
Run Course Identification ScriptsGPS coordinates used to ID course
Generates summary file containing metadata for each file (Vendor,
Vehicle ID, Course, Date, Miles, &
Hours
Run Data Collector ScriptsCombine files from similar vehicle, course and date
Generates files with concatenated channel data and flags the files
containing incomplete data
Run Report Generator1) Displays summary of mileage and hours2) Compares accelerations, temperatures, and speeds, across multiple vehicles3) Displays plots of major channels for each unique vehicle, course, and date combination
Generates .pdfreport
15Deputy Under Secretary of the Army Test and Evaluation Office
Analysis in Depth – Data Dependent
Creates context for analysis.
Multiple views of instrumentation
data channel values.
Links discrete, continuous, hierarchical,
and geospatial data types to scenario
timeline.
SASC Bill for FY17 NDAA Section 853. Enhanced use of data to improve acquisition program outcomes.
Army has been investing on the SASC’s position from the analysis and test community.
By FY18 Army T&E and Analytical communities should be up and running with a coherent data analytics and POA&M that gets at the proposed FY17 NDAA.
Unlocking & providing ATEC / RDECOM data to the community is necessary, as well ascontinuing RDT&E into tools to analyze, HPC investments, and visualization aids.
16Deputy Under Secretary of the Army Test and Evaluation Office
Leveraging the Big Data Space:
Use Historical Data to Right-size Future Test
Big Data
Field
T&E
User Needs
Materiel Development
S&T(6.1/6.2/6.3)
Risk Areas = Priority Test Areas
ATEC and AMSAA analyzed 18 million milesof Stryker field and T&E data to develop reliability “risk areas.”
Insights will be used to shape test scope on future versions of the systems.
+
Field Test
Subsystem X
Assembly Y
Component AB
Sub-assembly Z
Block interface W
Widget subsystem
Assembly Case
Sub-component ABC
Nuts and bolts Assembly
Main element
Subsystem Box K
Superstructure Link
Block assembly
Crankstick Beta
Shaft Structure
Drive Component Widget
XYZ Interface
17Deputy Under Secretary of the Army Test and Evaluation Office
Leveraging the Big Data Space:
Developing Cybersecurity Metrics
Big Data
Field
T&E
User Needs
Materiel Development
S&T(6.1/6.2/6.3)
ATEC leveraging Network Integration Evaluation (NIE) events to develop models, methodologies, and metrics for cybersecurity T&E.
Insights will be used to enable earlier-in-life cycle assessments and requirements development.
0.1% Person is untrustworthy
Resource worth $1000
Threat Model for Untrustworthy Insiders
18Deputy Under Secretary of the Army Test and Evaluation Office
Leveraging the Big Data Space:
Improving System Survivability
Big Data
Field
T&E
User Needs
Materiel Development
S&T(6.1/6.2/6.3)
ATEC combined insights about ballistic events on vehicles in theater from:- Intelligence community’s trend analyses
- On-board vehicle instrumentation- Ballistic response data from live fire testing.- Modeling and simulation
Insights used to improve:- Current and future system survivability designs- Test Scope- Test and evaluation methodology- Instrumentation and simulation designs
19Deputy Under Secretary of the Army Test and Evaluation Office
New Data Scientists & Data Analysts
Visual Information
-1084-
IT Management
-2210-
Computer Engineering
-0854-
Mathematical Statistics
-1529-
Expertise in engineering; expertise in data systems, data structures, data mining and programming languages.
Expertise in scientific inquiry into complex relationships and processes using multi-disciplinary analysis tools and techniques – particularly modeling and simulation.
Expertise in statistical tools
and techniques; expertise
in applied mathematics.
Expertise in applying visual design principles to communicate complex information to diverse audiences.
Expertise in data architectures, information systems, and data management.
Computer Science-1550-
Operations Research
-1515-Expertise in high-speed computing systems, data acquisition systems, algorithm analysis and development, and information processing display, control and transfer.
WANTED: Cadre of Data Scientists and Data Analysts
20Deputy Under Secretary of the Army Test and Evaluation Office
Conclusions
• Big Data analysis : Terabytes of Data Greater Insights ?
High potential to leverage learn /understand behaviors of complex systems
High potential of over-analysis for sake of over-analysis
• New generation of Data Scientists needed
• Real data-driven evidence to investigate anomalies - attribution
• Investments required:
New methods and tools to quickly process and analyze Big Data
Support the enterprise decision processes
Develop a sharing culture – DOD data policy evolutions
Big Data will change our T&E enterprise – in ways we don’t completely grasp yet.
Top Related