Big Data in Test and Symposium... –Middle East Course near mile marker 9.4 ... Unlocking &...

download Big Data in Test and Symposium... –Middle East Course near mile marker 9.4 ... Unlocking & providing

of 20

  • date post

    30-Mar-2020
  • Category

    Documents

  • view

    0
  • download

    0

Embed Size (px)

Transcript of Big Data in Test and Symposium... –Middle East Course near mile marker 9.4 ... Unlocking &...

  • 1Deputy Under Secretary of the Army Test and Evaluation Office

    Big Data in

    Test and Evaluation

    Prepared for ITEA Annual Symposium

    4 October 2016

    David Jimenez Director, Army Test and Evaluation

  • 2Deputy Under Secretary of the Army Test and Evaluation Office

    Big Picture The tests we conducted over the past 20 years are not

    representative of the next 20 years

    Hypersonics

    Artificial Intelligence

    Cognitive Workload

    Contested Information

    Environments

    Technology Parity

    Virtual Environments

    Autonomy

    Cryptology

    Requires Continued Investment in Infrastructure & People

    Future T&E

    Big Data

  • 3Deputy Under Secretary of the Army Test and Evaluation Office

    A Big Data Perspective

    Small Data Sets

    Manual Observation

    Little Insights

    The Good Old Days

    Small

    Data

    Small Analysis

    The Challenge

    Big

    Data

    How do I Determine ?

    ?

  • 4Deputy Under Secretary of the Army Test and Evaluation Office

    Force 2025 and T&E

    Service OTAs must determine

    whether or not systems are

    effective, suitable, and survivable

    in support of unified land

    operations in an operational

    environment dominated by:

    • Increased momentum of

    human interaction

    • Potential overmatch

    • Importance of cyber and

    space

    • Dense urban areas

    (megacities)

    • Ubiquitous media

    • WMD proliferation

    • CEMA !

    • Big Data !

    Increasing complexity on the battlefield increases complexity in T&E. Demand

    for data – and the means to use it effectively – is also increasing.

    See TRADOC PAM 525-3-1 “The U.S. Army Operating Concept (AOC): Win in a Complex World”

    found at: http://www.arcic.army.mil/Concepts/operating.aspx CEMA – Cyber Electromagnetic Activities

    http://www.arcic.army.mil/Concepts/operating.aspx

  • 5Deputy Under Secretary of the Army Test and Evaluation Office

    Big Data Causing An Evolution in T&E

    Yesterday Today

    Discrete data sets (usually associated with a single test); small overall file size

    Large data sets collected over a test program (may include data from contractor tests, simulators, hardware/software-in-the-loop laboratories, M&S, fielded system, and similar systems)

    Meaning derived from expert observations Meaning derived from continuous observation

    Workforce has expertise in the system under test

    Workforce has expertise in analytics

    Evaluation products consumed by small, specialized audience

    Evaluation products consumed by broad audience with diverse interests

    Central evaluation question: “Did it meet requirements?”

    Central evaluation question: “What are the system’s strengths and limitations over the range of conditions found on a complex, interoperating battlefield?”

    To focus on the “Why and How” of a system’s operational effectiveness, operational suitability, and survivability, increases the demand for deep analytics.

  • 6Deputy Under Secretary of the Army Test and Evaluation Office

    Leverage Advances in

    Instrumentation Capabilities

    T&E Big Data Challenges

    T&E Big Data

    Free and shared among responsible practitioners

    Amounts of data straining analytical resources

    Support model validations

    More Reliance on Supercomputing

    Need tools to make short order of analysis - visualization, sage, and frame capture

    T&E Cadre of the Future Requires Data Scientists and Data Analysts

  • 7Deputy Under Secretary of the Army Test and Evaluation Office

    Big Data Changed Everything

    Implications for Army Operating Concept and Force 2025: The Force 2025 Soldier will not have known a world without analytics.Our Surroundings

    We expect to be able to access analytics – instantly and on demand -- to measure and understand our complex world.

    Our neighbors

    Our interests

    Our health

    Our wealth

  • 8Deputy Under Secretary of the Army Test and Evaluation Office

    2025 T&E and Big Data Goals Goals: • Utilize knowledge, information, and

    data to achieve core mission and business objectives.  Faster, more Accurate Decision-Making

     Cost Optimization

     Quicker Responses to Requests for Information

     More Holistic Test and Evaluation

     Automated tracking items or status

    • Make useful big data capabilities available to everyone, but tailored to specific needs.

    Sustainment of data for long term

    use (Archival)

    Discoverability and Access to data

    Analytics of historical and

    current information

    Derive context to inform decision

    making

    Common Core Requirements:

    2025 T&E

    Leveraging Historical Data

    Faster, More Sophisticated

    Analytical Tools

    Modeling & Simulation

    Design of Experiments

    Cloud Computing

  • 9Deputy Under Secretary of the Army Test and Evaluation Office

    Data Driven Deep Dive Analysis

     Incident Overview – Middle East Course near mile marker 9.4 – 1L and 1R (Front, Left & Right) Half-shafts broken

     During that test week, vehicle completed 4 passes of this section of Middle East – 1 pass on June 5 (date incident occurred) – 3 passes on June 6

     Large spike in left front spindle, frame, and driver acceleration occurred approximately 10s prior to vehicle stopping on course due to incident.

    Sheared Right Side Half Shaft Sheared spline inside Left Hub Cartridge

  • 10Deputy Under Secretary of the Army Test and Evaluation Office

    Considerations for Big Data Analytics

    Pros: Cons: Available data may be underutilized due to awareness gaps.

    • What capabilities already exist? • What lessons have already been learned? • What opportunities exist?

    Utilizing big data requires careful planning: • Information system and data management design • Data Collection, Reduction, Analysis (DCRA) • Archiving and sustainment

    Utilizing big data requires appropriate tools. • Even small data sets are unmanageable without right tools • Tool development requires planning, time, and resources

    “I paid for all this data. What can I do with it?”

    Awareness

    Planning

    Tools

  • 11Deputy Under Secretary of the Army Test and Evaluation Office

    The Big Data Community

    Field

    T&E

    User Needs

    Materiel Development

    S&T

    Big Data

    (6.1/6.2/6.3)

    “Big Data” is a common resource of the Services’ analytical community.

    Diverse analytical organizations contribute to and draw from it:

    • data acquisition methods • computational resources • models, simulations, laboratories, tools • historical data • expertise

    Important questions going forward: • Who manages it for stakeholders? • Who sustains it? • How do we establish business rules for

    increased collaboration? • Can we obtain synergies through

    collaboration?

  • 12Deputy Under Secretary of the Army Test and Evaluation Office

    Performance Test Data Integrated Concept Study *

    PURPOSE. Address Army’s need for timely access to T&E data while aligning Army’s storage infrastructure and protocols with DODI 5000.02.

    SCOPE.

     Conduct a cost-benefit analysis to determine breath & depth of data to be stored & resources

     Evaluate sensitivity of results to assumption changes and identify risks associated with changes

    RESPONSIBILITES.

     AMSAA will appoint a Study Director

     Study Advisory Group (SAG) will oversee the planning & conduct of the study

     SAG Composition: Senior Executive / General Officer from: ASA(ALT) DUSA-TE, CIO/G-6,

    DCS, G-3/5/7, AMC *RDECOM, ARL, & AMSAA), TRADOC CAA, & DTIC

    * HQDA (DCS, G-3/5/7) Memo, subject: Performance Test Data Integrated Concept Guidance and Directive Study, 26 Feb 16

  • 13Deputy Under Secretary of the Army Test and Evaluation Office

    Value of ‘Deep’ Knowledge

    EXAMPLE

    Bad Event

    Increased Survivability

    Analysis by Service OTAs

    & Others

  • 14Deputy Under Secretary of the Army Test and Evaluation Office

    Big Data Analysis Approach

    Week’s worth of test data (~100 GB) processed within 2-3 days

    1) Download vehicle data files

    2) Process data for each week of test

    3) Review report for reliability highlights

    Run Course Identification Scripts GPS coordinates used to ID course

    Generates summary file containing metadata for each file (Vendor,

    Vehicle ID, Course, Date, Miles, &

    Hours

    Run Data Collector Scripts Combine files from similar vehicle, course and date

    Generates files with concatenated channel data and flags the files

    containing incomplete data

    Run Report Generator 1) Displays summary of mileage and hours 2) Compares accelerations, temperatures, and speeds, across multiple