04 September 2015 UK OFFICIAL

© Crown copyright 2015 Dstl

04 September 2015

UK OFFICIAL

© Crown copyright 2015. Published with the permission of the Defence Science and Technology Laboratory on behalf of the Controller of HMSO


04 September 2015

UK OFFICIAL

Introduction

Purpose: to make you think about what underlies the rules

and tables in a wargame. In particular:

• What you need to know about the data.

• Where that data might come from, with examples of three

different approaches used by Dstl.

• Making peace with the fact you will never have 100% of

the data you want.


04 September 2015

UK OFFICIAL

What this brief will cover

• A user’s perspective on input data

• Three methods for generating that data:

– Trials & Experimentation

– Performance & System Modelling

– Historical Analysis

• Checking your data

• Q&A

Caveat: Most of this presentation focuses on objective

data. This is only a part of the data conundrum that

wargamers face. A topic for later discussion.


04 September 2015

UK OFFICIAL


04 September 2015

UK OFFICIAL

Data in wargames

• Most wargames use numbers and rules.

• Different wargames use different levels of data aggregation to

produce these numbers (e.g. entity vs Coy).

• Unless you understand where all the numbers and rules come

from all wargames include black boxes.

• Lots of detail means lots of potential small black boxes.

• Often you want black boxes. E.g. players, emergent phenomenon.

• Make sure you’re comfortable with your black boxes.

Light Gun Battery

Manoeuvre Move

Artillery Power ISR Range

Medium

3 / 6 1 (2) 0

Artillery Range 1

2d6 Roll 1:5 1:4 1:3 1:2 1:1 2:1 3:1 4:1 5:1 6:1 7:1 8:1

2Attacker

WithdrawsEngaged

Defender

Withdraws

Defender

Defeated

Defender

Defeated

Defender

Defeated

Defender

Defeated

Defender

DefeatedExchange

Defender

Defeated

Defender

Defeated

Defender

Defeated

3Attacker

WithdrawsContact Engaged Exchange Exchange

Defender

Withdraws

Defender

Defeated

Defender

Defeated

Defender

Defeated

Defender

Defeated

Defender

Defeated

Defender

Defeated

4Attacker

Withdraws

Attacker

WithdrawsContact Engaged

Defender

Withdraws

Defender

WithdrawsExchange

Defender

Defeated

Defender

Defeated

Defender

Defeated

Defender

Defeated

Defender

Defeated

5Attacker

Withdraws

Attacker

Withdraws

Attacker

WithdrawsEngaged Contact Exchange

Defender

Withdraws

Defender

WithdrawsContact

Defender

Defeated

Defender

Defeated

Defender

Defeated

6Attacker

Defeated

Attacker

Withdraws

Attacker

Withdraws

Attacker

WithdrawsEngaged Contact

Defender

Withdraws

Defender

Withdraws

Defender

Withdraws

Defender

Withdraws

Defender

Defeated

Defender

Defeated

7Attacker

Defeated

Attacker

Withdraws

Attacker

Withdraws

Attacker

WithdrawsEngaged Engaged

Defender

Withdraws

Defender

Withdraws

Defender

Withdraws

Defender

Withdraws

Defender

Withdaws

Defender

Defeated

8Attacker

Defeated

Attacker

Defeated

Attacker

Withdraws

Attacker

Withdraws

Attacker

WithdrawsEngaged Contact

Defender

Withdraws

Defender

Withdraws

Defender

Withdraws

Defender

Withdaws

Defender

Defeated

9Attacker

Defeated

Attacker

Defeated

Attacker

Defeated

Attacker

Defeated

Attacker

Withdraws

Attacker

WithdrawsContact Contact

Defender

Withdraws

Defender

Withdraws

Defender

Withdaws

Defender

Withdraws

10Attacker

Defeated

Attacker

Defeated

Attacker

Defeated

Attacker

Defeated

Attacker

Withdraws

Attacker

WithdrawsEngaged Contact Contact Contact

Defender

Withdaws

Defender

Withdraws

11Attacker

Defeated

Attacker

Defeated

Attacker

DefeatedContact

Attacker

Defeated

Attacker

Withdraws

Attacker

WithdrawsEngaged

Defender

Defeated

Defender

DefeatedContact

Defender

Withdraws

12Attacker

Defeated

Attacker

Defeated

Attacker

Defeated

Defender

Withdraws

Attacker

Defeated

Attacker

Defeated

Attacker

DefeatedExchange Engaged

Defender

Defeated

Defender

DefeatedContact

– Trust

– Judgement

– Good logging

– Ability to review & discuss

– Validation & Verification

– Acceptable risk


04 September 2015

UK OFFICIAL

Users perspective

What I want to know about the data in my wargame:

• Where did it come from?

• Is it a good representation of what I want to investigate?

• Is it credible or is it counterintuitive? If so why?

• What are the uncertainties, boundaries & caveats?

– How will these impact the wargame?

– What caveats do I need to put on my outputs?


04 September 2015

UK OFFICIAL

Data sources Simple

Morale

1 2 3 4 5

1d6 1:1 2:1

Close Combat

Prob. Of Kill

Historical Analysis

Trials & Experimentation

Judgement

Systems & Performance

Modelling


04 September 2015

UK OFFICIAL

Morale

1 2 3 4 5

1d6 1:1 2:1

Close Combat

Prob. Of Kill

Systems & Performance

Modelling

Historical Analysis


Judgement (incl. soft effects & emergent

phenomenon)

Data sources In Practice


04 September 2015

UK OFFICIAL


From the Fields to the Tables

Mark Pickering



04 September 2015

UK OFFICIAL

Contents

• What real world data do we need?

• How do we collect data?

• What causes variation?

• From data to game.


04 September 2015

UK OFFICIAL

What real world data do we need?

• Probability of hit, or “P(hit)”

• Achievable rates of fire

• Lethality


04 September 2015

UK OFFICIAL

Collecting data

• Modelling the ballistics – does not represent ‘a muddy field’.

• Trials – better real world data, but expensive.


04 September 2015

UK OFFICIAL

What can be modelled?

• Lots of highly detailed aspects, such as:

– Weather

• Pressure

• Humidity

• Wind

– Manufacturing variation

• Propellant quality

• Density variation in round

• Barrel defects

– Etc.

• But some things can’t be easily modelled, such as…..


04 September 2015

UK OFFICIAL


04 September 2015

UK OFFICIAL

How do we measure P(hit)?

• With difficulty


04 September 2015

UK OFFICIAL

P(hit)

• Miss distance

• What range?

• Limited data


04 September 2015

UK OFFICIAL

P(hit) - Hit grid


04 September 2015

UK OFFICIAL

From data to game

• Example Game Mechanism

– To hit roll = Probability of hit or P(hit)

– Armour save roll = Probability of not penetrating the armour

– To wound roll = Probability of ‘kill’ or P(kill)

Hit Armour Save Wound


04 September 2015

UK OFFICIAL

Aim Point

To hit roll


04 September 2015

UK OFFICIAL

To hit roll


04 September 2015

UK OFFICIAL

Probable hit

To hit roll


04 September 2015

UK OFFICIAL

Armour save roll

Hit area


04 September 2015

UK OFFICIAL

Hit area

Armour View image from: War Thunder – www.warthunder.com

Armour save roll


04 September 2015

UK OFFICIAL

Lethality View image from: War Thunder – www.warthunder.com

Hit area

To wound roll


04 September 2015

UK OFFICIAL

Systems & Performance Modelling

Generating the rules

Dan Ledwick



04 September 2015

UK OFFICIAL

Systems and performance

modelling • I.e. what happens when X shoots Y?

• Information from trials and experimentation.

– P(hit).

– Level of damage per hit.

• Simulates outcomes of specific events.

• Reports at required level.

– E.g. individual bullet effects aggregated to Bde level.

• Faster and cheaper than full trials.

• Highly repeatable.

• Can represent potential future equipment.


04 September 2015

UK OFFICIAL

Systems and performance modelling

• Focus on Vulnerability/Lethality

• Overview

• Example

• Data uses


04 September 2015

UK OFFICIAL

Vulnerability/Lethality overview

• Determine the result of a weapon attacking a

target

– Weapon performance

– Armour performance

– Damage to target components

– Resulting effect on target functionality

• Usually involves running a computer

simulation

– Large number of engagements simulated


04 September 2015

UK OFFICIAL

Vulnerability/Lethality example

Effectors

Initially: Brick

From interaction: Glass shards

Example: brick thrown at glasshouse.

Assessment Process

1. Trigger: Effector generator – brick.

2. Propagate brick.

3. Interaction with target component.

- Trigger generation of glass shards.

4. Component response – damage algorithms.

5. Propagate brick and glass shards.

6 and 7. Interaction/Component response.

- Repeat as necessary

1 2

3

4

5

6

7


04 September 2015

UK OFFICIAL

Vulnerability/Lethality data uses • Identifying vulnerable areas of vehicles

to prioritise protection improvements

• Characterising weapon performance against a target set

• Higher level wargames and models

– Computer simulations, Manual

• Training simulations

– DFWES, AWES


04 September 2015

UK OFFICIAL

Wargaming Generation

Trials and

Experimentation

System and

Performance

modelling

Tabletop Computer Algorithms

Judgement Wargame

Generation

Historical

Analysis


04 September 2015

UK OFFICIAL

Historical Analysis for Wargaming

Stevie Ho



04 September 2015

UK OFFICIAL

Data and wargaming

• Historical Analysis (HA) provides real-world data grounded

in reality.

– Increases buy-in when you can say: “this actually happened

before.”

• Testing the theoretical vs the actual.

• Can provide data on things trials and experiments cannot

or will not.

– Particularly in the operational and strategic spaces.

• However, you cannot create new historical data, you have

to work with what you get.

– Exact real world cases are rare, so historical analogy is often

a requirement.


04 September 2015

UK OFFICIAL

Definition

• The use of mathematical, statistical and other forms of

analysis to understand historical engagements,

operations, campaigns and conflicts for the purpose of

providing impartial analysis and sensitive decision support

to policy makers.

• Critical to sensible and fully informed policy making.


04 September 2015

UK OFFICIAL

Origins

• 1980s – Falklands War.

Field trials vs. Falklands War

• Combat Psychology Example: soldiers are much braver if they

know they’re in no real danger.


04 September 2015

UK OFFICIAL

What data does HA use?

• Anything that will tell us what we want to know or allow us

to infer an acceptable estimate thereof.

• Primary data sources

– War diaries

– Post Op Reports

– Operational Data Sources

• Secondary data sources

– Official Histories

– Reference Books

– Academic Studies


04 September 2015

UK OFFICIAL

What methods does HA use?

• Quantitative analysis

– Regression Analysis, correlation etc.

– Stats packages – R, Minitab, SPSS etc.

– Excel formulae, charts, graphs etc.

• Qualitative analysis

– Historical Research, Framework Analysis etc.

– Literature reviews

– Case studies


04 September 2015

UK OFFICIAL

Large-N Studies

Quantitative

Statistical analysis Generalised results

Potential over-abstraction

Single Case Studies

Qualitative

In-depth analysis Contextual detail

Not representative

Comparative Analysis

Qualitative or Quantitative

5 – 30 cases Pattern Matching

Selection Bias

Increasing Depth

Increasing Abstraction

The Real World

Spectrum of historical analysis


04 September 2015

UK OFFICIAL

Large-N Studies

Quantitative

Statistical analysis Generalised results

Potential over-abstraction

Single Case Studies

Qualitative

In-depth analysis Contextual detail

Not representative

Comparative Analysis

Qualitative or Quantitative

5 – 30 cases Pattern Matching

Selection Bias

Increasing Depth

Increasing Abstraction

The Real World

This is similar to wargaming.

Different types of wargame will be

appropriate depending on your question and

depending on your data availability.

More data and material allows you to

produce a greater number of different kinds

of wargame, but one size does not fit all.

Spectrum of historical analysis


04 September 2015

UK OFFICIAL

What does HA offer? • Multidisciplinary approach an advantage.

– Data from tactical up to grand strategic level.

– Understanding of interaction of qualitative and quantitative

factors.

• HA can highlight the important factors and back or

disprove perceived wisdom.

– Particularly important when the HA goes against commonly

held beliefs or perceptions.

• These factors can be fed into wargames or wargames can

be designed to highlight the importance of these factors.

• Can be blended with trials and experimentation data.

– A mixture of from HA, trials, experimentation and judgement

can help robustness, especially in the future space.


04 September 2015

UK OFFICIAL

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0 5 10 15 20 25

Period

Pro

po

rtio

n o

f fat

alat

ies

attr

ibu

ted

to IE

Ds

Army

Marines

Iraq – US Army / USMC IED Fatalities Introduction of USMC MOJAVE VIPER Training

Example study: HA of value of training


04 September 2015

UK OFFICIAL


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0 5 10 15 20 25

Period

Pro

po

rtio

n o

f fat

alat

ies

attr

ibu

ted

to IE

Ds

Army

Marines

Iraq – US Army / USMC IED Fatalities Introduction of USMC MOJAVE VIPER Training

However, at this time the USMC also

introduced a new protective vehicle.

Which one was more important?


04 September 2015

UK OFFICIAL

0

0.001

0.002

0.003

0.004

0.005

0.006

0.007

0.008

Aug-41 Feb-42 Sep-42 Apr-43 Oct-43 May-44 Nov-44

Date

% t

ota

l ja

pa

ne

se

to

nn

ag

e s

un

k/p

atr

ol

da

y

<4 patrols

>4 patrols

USN Submarines - Pacific Average proportion of Japanese fleet sunk each patrol day, each month by experienced and inexperienced captains.

Paired t-test p value < 0.001.



04 September 2015

UK OFFICIAL

Checking Your Data

A User’s Peace of Mind

James King


04 September 2015

UK OFFICIAL

Checking your tables

• Comparison to historical operations.

• Is it a reasonable (and credible) representation

of the situation you’re trying to wargame?

– Judgement of people who have conducted similar

operations.

– Military/SME judgement.

• Look over as many cases as possible, not just

one example. Beware single data points.


04 September 2015

UK OFFICIAL

Data provenance

Best Case Worst Case

Very Common Very Rare Common

Availability

Lo

ts o

f R

ea

l W

orld

Cases

Exte

nsiv

e T

ria

ls

Tria

ls

Exp

erim

en

tatio

n

Mo

de

llin

g

His

torica

l Analo

gy

Exp

ert

Op

inio

n

(with

ou

t e

vid

en

ce

)

‘Exp

ert

’ Op

inio

n

(little

rele

va

nt

exp

erie

nce)

Judgement

Exp

ert

Op

inio

n

(with

evid

en

ce)

Provenance


04 September 2015

UK OFFICIAL

Availability

Lo

ts o

f R

ea

l W

orld

Cases

Exte

nsiv

e T

ria

ls

Tria

ls

Exp

erim

en

tatio

n

Mo

de

llin

g

His

torica

l Analo

gy

Exp

ert

Op

inio

n

(with

ou

t e

vid

en

ce

)

‘Exp

ert

’ Op

inio

n

(little

rele

va

nt

exp

erie

nce)

Judgement

Exp

ert

Op

inio

n

(with

evid

en

ce)

Provenance

Data provenance

Best Case Worst Case

Very Common Very Rare Common


04 September 2015

UK OFFICIAL

And finally…

• What you need to know about the data.

– Think about what underlies the rules and tables in your wargame.

– Log your assumptions wherever possible.

– When designing a wargame make sure you can get the data you will need.

• Where that data might come from, with examples of three different

approaches used by Dstl.

– Trials & Experimentation, Systems & Performance Modelling, Historical Analysis.

– There are multiple sources of data, each with their own strengths and

weaknesses. Judgement ties them all together to make a wargame.

– Wargames and data sources are not isolated, but can test and inform each other.

• Making peace with the fact you will never have 100% of the data you want.

– Be comfortable with uncertainties and black boxes. People are black boxes!

– Be sure you are happy with the data you have. Understand the relationship

between the impact the data will have, and the certainty you need.

– Beware single data points!


04 September 2015

UK OFFICIAL

Questions?

04 September 2015 UK OFFICIAL

Documents

Transcript of 04 September 2015 UK OFFICIAL