Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

21
Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

Transcript of Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

Page 1: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

Some Thoughts on ABS V&V

V. E. Middleton Enterprises, LLC.

Page 2: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

“When you can measure what you are speaking about, when you can express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind . . . scarcely advanced to the stage of science.”

William Thomsen, Lord Kelvin, 1804-1907“If I had time … to study, I think I should concentrate almost entirely on the “actualities of war”, the effect of tiredness, hunger, fear, lack of sleep, weather … It is the actualities that make war so complicated and so difficult, and are usually neglected by historians.”

Field Marshall Archibald Wavell, 1883-1950

Author of ‘Soldiers and Soldiering’

Fundamentals

Page 3: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

What’s Driving the Train?

A Changing View of Conflict Effects Based Operations Maneuverist Approach Network Centric Warfare

Physical Information Cognitive

Changing Spectrum of Operations No longer only symmetric warfare Close contact as well as close combat Greater Role of Humans Relative to Materiel

Need for More than First Principles, Attrition-based MS&A

Page 4: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Agent Based Simulation

Modeling Command & Control with Network-Centric SA Make Decisions Order Action Monitor Task Accomplishment Regulate/Adjust Task Accomplishment Possess and Employ Situation Knowledge

Modeling Human Interactions Attitudes and latitudes - the agent narrative The fuzzy end state

Page 5: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Need for Human-Centric Modeling

Ultimately we would like a continuum of Human Factors At one end

Sensory and psycho-physiological factors At the other end

Will, morale, culture etc In the middle, Situation Awareness and Decision-

Making: Concepts with real “face validity” Some accessible theory Metrics/measures possible Can be expressed concretely in simulation

Page 6: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

• Human Systems Representation (psycho-physiological states) and behavior/decision making processes

• Resolution & Fidelity Issues• Individual Human interaction with Terrain & the environment

• Representation of complex terrain (MOUT)• Methodologies unique to soldier operations, e.g.:

- Target acquisition in urban areas and inside structures (complex backgrounds, varying light levels, etc.)- Target engagement process at short ranges

• Representation of “bad” information: incomplete, inaccurate, inconsistent

• Representation of the integration of hardware/equipment with human systems

Some of the Challenges to V&V

Page 7: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Dynamic State Modeling

Page 8: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Intelligent Agents

Agents have Perception: can sense their environment (key point,

perceptions can be subjective, incomplete or just wrong!) Action: can effect change on their environment

Intelligent Agents also have Knowledge: can relate perceptions to world

object "states" and make inferences to supplement perceptual data

Autonomy: can act based on current perceived world state instead of following only pre-programmed actions

For emergent analysis autonomy is the most important feature

Page 9: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Emergent Analysis

Factor Name Low Level High Level DescriptionBlue_Speed 1.2 4.15 Ground speed of blue forces (km/hr)

Mask_Obedience 0.2 0.9 How well soldiers follow orders after they mask

Number_of_UAV 0 2 Number of UAVs available in the scenario

Number_of_ARV 0 4 Number of armed robotic vehicles

JCAD_Sensitivity 2 14 Time until JCAD detects (sec)

Mask_Marksmanship 0.4 0.8 Marksmanship of blue forces after they mask

SWFR_Effect 0.5 1 Internal communications effectiveness

ExternalComm_Effect 0.5 1 External communications effectiveness

Simulation Experimental Design

Page 10: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Problem Statement

I don’t see that our situation is especially improved

The right solution to the wrong problem is not terribly useful

Page 11: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Operational Narative

“You mean no one remembered to bring a rock?”

The operational narrative frames the necessary an salient elements of the analysisA set of standard scenarios is a key element to the ABS Val process

Page 12: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Conceptual Model

The conceptual model must be intelligible to both the analyst AND to the decision maker

Page 13: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Tool selection and application

Appropriate choice and appropriate application of tools are both required for sound results

Page 14: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Unexpected Results?

Page 15: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Validation Process Steps

Definition of a valid problem statement

Construction of a set of valid operational narratives/ scenarios/use cases.

Selection/adaptation/ development of a conceptual model

Selection and validation of a reliable simulation

Validity of the experimental design and its implementation

a well-formulated problem statement expressed in the “right” MOPs &MOEs

1) face validity and credibility derived from the developer(s)/

authoritative agencies, 2) internal consistency,3) intensity/depth of individual cases,

breadth of the entire set; Degree of completeness, fidelity,

resolution Representation of scenarios/use cases

based on the fit between the simulation and conceptual model - fidelity, resolution, completeness

Determined by feedback, assessment & interpretation of simulation results.

Validation CriteriaStep

Page 16: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Backups

Page 17: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Intelligent Agent Behavior Engine

EnvironmentDefinition

DataAcquisition

DataAssimilation

KnowledgeBase

Behavior

MeteorologicalPhenomena

Terrain

ThreatWeaponsEffects

OtherEntity

Behaviors

.

.

.

SensoryPerception

• Visual• Aural• Tactile

MessageTraffic

Filtration

Fusion

Integration

DataBase

DecisionEngine

TaskPerformance• Rate• Accuracy

Decisions

FormattedData

Handoff

Models

Physics

Based

Page 18: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Dynamic, protection/ operability tradeoffs

Integrated Insults

Terrain-dependentNBC contamination

SmokeLimited Fatigue

Terrain dependentIC movement rates

MOUT movementformations/rules

Pk/Ph

Thermal stressChemical agents

Pk/Ph, Pdet, PacqDirect/Indirect fireIndependent error

budgetsSimple suppressionPre-set or HITL target

selection

Suppression as a functionof situation awareness

Realistic close combatin MOUT and the openfield

Perfectcommunications

Perfect SituationAwareness

HITL decisionsbased on perfectknowledge

Terrain/loaddependentmovement rates

Perfect navigationPre-set or HITL

route/speedadjustments

Survivability Lethality Mobility Command and Control

Improved detection of ICtargets

Integrated error budgetsStressor effects on error

budgets

IFF Combat IDImperfect knowledgeHITL decisionsSimple rule-based

situation awareness

Dynamic humanresponse to terrain:- Route selection

- Optimal use of "Position"

concealment

State of the Simulation Art

Limited re-supply

Expenditure of consumable resources

Sustainability

Estimation of metabolic workload

Statistical use of open field terrain protection Dynamic

redistribution of unit resources

Situation Awareness as dynamic contingency response (pattern recognition/integrated factors)

Incremental addition of ballistic protectionNon-lethal weapon

effects

Simple barriers

Vulnerability to projectiles/ fragments

Blunt trauma

Dynamic terrain interaction

Task dependent incapacitation

Current Capability

Significant Challenges

Near-TermAchievableKey:

- Cover and

Intra-building movement

Macro-nutrient physiology and energy balance

Soldier load item utility-based optimization

Integrated effects of fatigue on performance

Page 19: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

New Metrics: Essential Elements New MOPs to quantify the performance of information

systems and subsystems. Current human factors engineering measures address

aspects of human interaction with such systems, e.g.; rate of signal processing and other perceptual interface questions, very few address the quality and usability of information or trade-offs between acquisition and comprehension issues.

New MOEs to quantify the effectiveness of close contact instead of close combat. This includes the need for measures of situation

awareness and its contribution to greater ability to react to dynamic conditions.

The mapping that describes the contributions of various MOPs of interest to the MOEs associated with those MOPs.

Page 20: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Extending conventional outcome measures - Control?

Attrit Neutralize Influence Monitor

destroy suppress restrict detect

kill interdict inhibit observe

damage detain restrain check

incapacitate surround curb regulate

reduce contain lead inspect

Page 21: Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.

V. E. Middleton

Representing Agent Perception & Knowledge

MOPs are simulation inputs MOEs are simulation outputs Knowledge structures must represent data

filtration, fusion, and integration - cues& alerts, inference, pattern matching, other decision elements

MOPs & MOEs must support these structures - Bayesian Belief Networks, Fuzzy Cognitive Maps, Recognition Primed Decision Making, Dempster Schaefer Uncertainty theory…