Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.
-
Upload
clement-watkins -
Category
Documents
-
view
214 -
download
0
Transcript of Some Thoughts on ABS V&V V. E. Middleton Enterprises, LLC.
Some Thoughts on ABS V&V
V. E. Middleton Enterprises, LLC.
V. E. Middleton
“When you can measure what you are speaking about, when you can express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind . . . scarcely advanced to the stage of science.”
William Thomsen, Lord Kelvin, 1804-1907“If I had time … to study, I think I should concentrate almost entirely on the “actualities of war”, the effect of tiredness, hunger, fear, lack of sleep, weather … It is the actualities that make war so complicated and so difficult, and are usually neglected by historians.”
Field Marshall Archibald Wavell, 1883-1950
Author of ‘Soldiers and Soldiering’
Fundamentals
V. E. Middleton
What’s Driving the Train?
A Changing View of Conflict Effects Based Operations Maneuverist Approach Network Centric Warfare
Physical Information Cognitive
Changing Spectrum of Operations No longer only symmetric warfare Close contact as well as close combat Greater Role of Humans Relative to Materiel
Need for More than First Principles, Attrition-based MS&A
V. E. Middleton
Agent Based Simulation
Modeling Command & Control with Network-Centric SA Make Decisions Order Action Monitor Task Accomplishment Regulate/Adjust Task Accomplishment Possess and Employ Situation Knowledge
Modeling Human Interactions Attitudes and latitudes - the agent narrative The fuzzy end state
V. E. Middleton
Need for Human-Centric Modeling
Ultimately we would like a continuum of Human Factors At one end
Sensory and psycho-physiological factors At the other end
Will, morale, culture etc In the middle, Situation Awareness and Decision-
Making: Concepts with real “face validity” Some accessible theory Metrics/measures possible Can be expressed concretely in simulation
V. E. Middleton
• Human Systems Representation (psycho-physiological states) and behavior/decision making processes
• Resolution & Fidelity Issues• Individual Human interaction with Terrain & the environment
• Representation of complex terrain (MOUT)• Methodologies unique to soldier operations, e.g.:
- Target acquisition in urban areas and inside structures (complex backgrounds, varying light levels, etc.)- Target engagement process at short ranges
• Representation of “bad” information: incomplete, inaccurate, inconsistent
• Representation of the integration of hardware/equipment with human systems
Some of the Challenges to V&V
V. E. Middleton
Dynamic State Modeling
V. E. Middleton
Intelligent Agents
Agents have Perception: can sense their environment (key point,
perceptions can be subjective, incomplete or just wrong!) Action: can effect change on their environment
Intelligent Agents also have Knowledge: can relate perceptions to world
object "states" and make inferences to supplement perceptual data
Autonomy: can act based on current perceived world state instead of following only pre-programmed actions
For emergent analysis autonomy is the most important feature
V. E. Middleton
Emergent Analysis
Factor Name Low Level High Level DescriptionBlue_Speed 1.2 4.15 Ground speed of blue forces (km/hr)
Mask_Obedience 0.2 0.9 How well soldiers follow orders after they mask
Number_of_UAV 0 2 Number of UAVs available in the scenario
Number_of_ARV 0 4 Number of armed robotic vehicles
JCAD_Sensitivity 2 14 Time until JCAD detects (sec)
Mask_Marksmanship 0.4 0.8 Marksmanship of blue forces after they mask
SWFR_Effect 0.5 1 Internal communications effectiveness
ExternalComm_Effect 0.5 1 External communications effectiveness
Simulation Experimental Design
V. E. Middleton
Problem Statement
I don’t see that our situation is especially improved
The right solution to the wrong problem is not terribly useful
V. E. Middleton
Operational Narative
“You mean no one remembered to bring a rock?”
The operational narrative frames the necessary an salient elements of the analysisA set of standard scenarios is a key element to the ABS Val process
V. E. Middleton
Conceptual Model
The conceptual model must be intelligible to both the analyst AND to the decision maker
V. E. Middleton
Tool selection and application
Appropriate choice and appropriate application of tools are both required for sound results
V. E. Middleton
Unexpected Results?
V. E. Middleton
Validation Process Steps
Definition of a valid problem statement
Construction of a set of valid operational narratives/ scenarios/use cases.
Selection/adaptation/ development of a conceptual model
Selection and validation of a reliable simulation
Validity of the experimental design and its implementation
a well-formulated problem statement expressed in the “right” MOPs &MOEs
1) face validity and credibility derived from the developer(s)/
authoritative agencies, 2) internal consistency,3) intensity/depth of individual cases,
breadth of the entire set; Degree of completeness, fidelity,
resolution Representation of scenarios/use cases
based on the fit between the simulation and conceptual model - fidelity, resolution, completeness
Determined by feedback, assessment & interpretation of simulation results.
Validation CriteriaStep
V. E. Middleton
Backups
V. E. Middleton
Intelligent Agent Behavior Engine
EnvironmentDefinition
DataAcquisition
DataAssimilation
KnowledgeBase
Behavior
MeteorologicalPhenomena
Terrain
ThreatWeaponsEffects
OtherEntity
Behaviors
.
.
.
SensoryPerception
• Visual• Aural• Tactile
MessageTraffic
Filtration
Fusion
Integration
DataBase
DecisionEngine
TaskPerformance• Rate• Accuracy
Decisions
FormattedData
Handoff
Models
Physics
Based
V. E. Middleton
Dynamic, protection/ operability tradeoffs
Integrated Insults
Terrain-dependentNBC contamination
SmokeLimited Fatigue
Terrain dependentIC movement rates
MOUT movementformations/rules
Pk/Ph
Thermal stressChemical agents
Pk/Ph, Pdet, PacqDirect/Indirect fireIndependent error
budgetsSimple suppressionPre-set or HITL target
selection
Suppression as a functionof situation awareness
Realistic close combatin MOUT and the openfield
Perfectcommunications
Perfect SituationAwareness
HITL decisionsbased on perfectknowledge
Terrain/loaddependentmovement rates
Perfect navigationPre-set or HITL
route/speedadjustments
Survivability Lethality Mobility Command and Control
Improved detection of ICtargets
Integrated error budgetsStressor effects on error
budgets
IFF Combat IDImperfect knowledgeHITL decisionsSimple rule-based
situation awareness
Dynamic humanresponse to terrain:- Route selection
- Optimal use of "Position"
concealment
State of the Simulation Art
Limited re-supply
Expenditure of consumable resources
Sustainability
Estimation of metabolic workload
Statistical use of open field terrain protection Dynamic
redistribution of unit resources
Situation Awareness as dynamic contingency response (pattern recognition/integrated factors)
Incremental addition of ballistic protectionNon-lethal weapon
effects
Simple barriers
Vulnerability to projectiles/ fragments
Blunt trauma
Dynamic terrain interaction
Task dependent incapacitation
Current Capability
Significant Challenges
Near-TermAchievableKey:
- Cover and
Intra-building movement
Macro-nutrient physiology and energy balance
Soldier load item utility-based optimization
Integrated effects of fatigue on performance
V. E. Middleton
New Metrics: Essential Elements New MOPs to quantify the performance of information
systems and subsystems. Current human factors engineering measures address
aspects of human interaction with such systems, e.g.; rate of signal processing and other perceptual interface questions, very few address the quality and usability of information or trade-offs between acquisition and comprehension issues.
New MOEs to quantify the effectiveness of close contact instead of close combat. This includes the need for measures of situation
awareness and its contribution to greater ability to react to dynamic conditions.
The mapping that describes the contributions of various MOPs of interest to the MOEs associated with those MOPs.
V. E. Middleton
Extending conventional outcome measures - Control?
Attrit Neutralize Influence Monitor
destroy suppress restrict detect
kill interdict inhibit observe
damage detain restrain check
incapacitate surround curb regulate
reduce contain lead inspect
V. E. Middleton
Representing Agent Perception & Knowledge
MOPs are simulation inputs MOEs are simulation outputs Knowledge structures must represent data
filtration, fusion, and integration - cues& alerts, inference, pattern matching, other decision elements
MOPs & MOEs must support these structures - Bayesian Belief Networks, Fuzzy Cognitive Maps, Recognition Primed Decision Making, Dempster Schaefer Uncertainty theory…