Aide Memoire on - dln-rad.forces.gc.ca

of 113

Aide Memoire on

Intelligence

Analysis Tradecraft

UNCLASSIFIED

For training

purposes only.

Version française disponible.

Canadian Forces

Intelligence Command

Version 6.00

August 2015

of 113

Aide Memoire on

Intelligence Analysis Tradecraft

Version 6.0 August 2015

Table of Contents

INTRODUCTION ....................................................................................................................................................... 6

BEST PRACTICES FOR INTELLIGENCE ANALYSTS .................................................................................................. 6 ELEMENTS OF ANALYSIS ......................................................................................................................................... 7

Evidence (Uncertainty). ....................................................................................................................................... 7 Assumptions. ........................................................................................................................................................ 7 Factors. ................................................................................................................................................................ 8 Assessing the Future. ........................................................................................................................................... 8 Drivers and Criteria. ............................................................................................................................................ 9 Assessments, Scenarios and Hypotheses. ............................................................................................................. 9

THE INTELLIGENCE CYCLE ................................................................................................................................... 10 LEVELS OF INTELLIGENCE SUPPORT .................................................................................................................... 10 CATEGORIES OF INTELLIGENCE ............................................................................................................................ 12 TYPES OF STRATEGIC ANALYSIS PRODUCTS ........................................................................................................ 12 ANALYTIC RIGOUR ................................................................................................................................................ 13 TEMPORARY POSTMODERNISM ............................................................................................................................. 13

REFLECT ON THE PROBLEM, DETERMINING POSSIBLE APPROACHES. ............................................ 14

WHAT IS THE QUESTION? ...................................................................................................................................... 14 W5H & SW. ........................................................................................................................................................ 14 (W5H & SW)C4 .................................................................................................................................................. 15 Policy Prescription. ........................................................................................................................................... 15

WHAT IS THE NATURE OF THE QUESTION? ............................................................................................................ 15 Puzzles, Mysteries and Messes. .......................................................................................................................... 15 Three Kinds of Questions. .................................................................................................................................. 16 How much detail? .............................................................................................................................................. 16 Time. .................................................................................................................................................................. 16 Client Knowledge. .............................................................................................................................................. 16

ISSUE REDEFINITION .............................................................................................................................................. 17 IDENTIFYING FACTORS .......................................................................................................................................... 17 SUMMARY ............................................................................................................................................................... 17

BE RESOURCEFUL AND SYSTEMATIC WHEN COLLECTING INFORMATION, DOCUMENTING

SOURCES AND NOTING CAVEATS ON USAGE. ............................................................................................. 18

GUIDING FACTORS ................................................................................................................................................. 18 INT COMPARISON CHART: PROS AND CONS ........................................................................................................ 18

OSINT................................................................................................................................................................ 19 HUMINT ........................................................................................................................................................... 19 SIGINT .............................................................................................................................................................. 20 IMINT................................................................................................................................................................ 20

COLLECTION PLANNING ........................................................................................................................................ 21 Requirements. ..................................................................................................................................................... 21 Indicators. .......................................................................................................................................................... 22 Collection Plans. ................................................................................................................................................ 23

DATA SORTING ....................................................................................................................................................... 26

of 113

DOCUMENTATION OF SOURCES ............................................................................................................................. 26 Annotated Bibliography. .................................................................................................................................... 26

ROTBERG, ROBERT I. WHEN STATES FAIL – CAUSES AND CONSEQUENCES. PRINCETON: PRINCETON UNIVERSITY

PRESS, 2004. ISBN 0691116717 (DG’S LIBRARY) (UNCLAS) ................................................................................. 27 Keep track on-line. ............................................................................................................................................. 27 Analysts Notebook. ............................................................................................................................................. 27 Cornell Note Taking System. .............................................................................................................................. 28

CLASSIFICATION MARKING ................................................................................................................................... 28 Classifications. ................................................................................................................................................... 30 Control System Marking. ................................................................................................................................... 31 Sub-Control Systems. ......................................................................................................................................... 31 Dissemination Control Marking......................................................................................................................... 31

AUDIT TRAIL .......................................................................................................................................................... 31 SUMMARY ............................................................................................................................................................... 31

CRITICALLY EVALUATE THE QUALITY OF ALL INFORMATION ......................................................... 32

DATA TRIAGE ......................................................................................................................................................... 32 DATA DIAGNOSTICS ............................................................................................................................................... 32

Provider Diagnostic. .......................................................................................................................................... 32 Information Diagnostic. ..................................................................................................................................... 34 Relevance Check. ............................................................................................................................................... 35 Denial and Deception. ....................................................................................................................................... 37

DATA QUALITY DIAGNOSTIC ................................................................................................................................. 38 PROPAGANDA ......................................................................................................................................................... 39

How to identify propaganda - Red Flags ........................................................................................................... 39 CHECKLISTS FOR EVALUATING INFORMATION .................................................................................................... 40

HUMINT Reporting ........................................................................................................................................... 40 Alpha-Numeric Source-Data Rating .................................................................................................................. 40 Four Step - Web Page Evaluation Checklist. ..................................................................................................... 40

DEVELOP MULTIPLE HYPOTHESES / EXPLANATIONS ............................................................................. 42

FORMULATING HYPOTHESIS ................................................................................................................................. 42 Black Swan Rule................................................................................................................................................. 42 Value Added. ...................................................................................................................................................... 42 Three Basic Approaches. ................................................................................................................................... 42 Types of Hypotheses. .......................................................................................................................................... 43 A Good Hypothesis ............................................................................................................................................. 43

HYPOTHESES (AND SCENARIO) GENERATION ....................................................................................................... 43

CHALLENGE ASSUMPTIONS, MINDSETS AND BIASES............................................................................... 44

ASSUMPTIONS ......................................................................................................................................................... 44 Implication of Assumptions. ............................................................................................................................... 44 Key Assumptions Check. .................................................................................................................................... 44

MINDSETS AND BIASES ........................................................................................................................................... 45 Implication of Mindsets. ..................................................................................................................................... 45 Implication of Biases. ......................................................................................................................................... 46

COUNTERING MINDSETS AND BIASES ................................................................................................................... 46 PREVENTION OF FIXED MINDSETS ....................................................................................................................... 47

BUILD COLLABORATIVE NETWORKS ............................................................................................................ 48

CANADIAN SECURITY AND INTELLIGENCE COMMUNITY ..................................................................................... 48 WHY COLLABORATE? ........................................................................................................................................... 48 CREATING A COLLABORATIVE CULTURE ............................................................................................................. 49 TEAM/GROUP COLLABORATION ........................................................................................................................... 50

Models of Collaboration. ................................................................................................................................... 50 Developing Effective Teams. .............................................................................................................................. 51

of 113

Team Formation. ................................................................................................................................................ 51 CONSENSUS VS DISSENT ......................................................................................................................................... 52

Common Pitfalls with Small Groups. ................................................................................................................. 52 Benefitting from Diversity. ................................................................................................................................. 53 Advocacy vs Objective Inquiry. .......................................................................................................................... 53 Forcing a Consensus. ......................................................................................................................................... 54 Encouraging Consensus. .................................................................................................................................... 55

USE STRUCTURED ANALYTIC TECHNIQUES ............................................................................................... 56

WHY USE STRUCTURED TECHNIQUES? ................................................................................................................. 56 CATEGORIES OF TECHNIQUES ............................................................................................................................... 56 STRUCTURED ANALYTIC TECHNIQUES MAP ........................................................................................................ 57 USE WORK GROUPS ................................................................................................................................................ 58

Team A / Team B. ............................................................................................................................................... 58 Red Team. .......................................................................................................................................................... 59 Red Cell. ............................................................................................................................................................. 60 Devil’s Advocacy. ............................................................................................................................................... 61

DEVELOP ................................................................................................................................................................. 62 Structured Brainstorming................................................................................................................................... 62 Delphi Method.................................................................................................................................................... 63 Environmental Scanning. ................................................................................................................................... 64 Outside-In Thinking. .......................................................................................................................................... 65 Hypothesis Generator. ....................................................................................................................................... 66 Hypothesis Review Technique. ........................................................................................................................... 67 Reframing the Question...................................................................................................................................... 68

EVALUATE ............................................................................................................................................................... 69 Data Diagnostic. ................................................................................................................................................ 69 Key Assumptions Check. .................................................................................................................................... 70 Indicators of Change. ......................................................................................................................................... 72 Indicators Validator. .......................................................................................................................................... 73 Analysis of Competing Hypotheses. (ACH). ...................................................................................................... 74

EXPLORE NETWORKS .............................................................................................................................................. 75 Social Network Analysis. .................................................................................................................................... 75 Link Charts. ........................................................................................................................................................ 76

COMPARE ................................................................................................................................................................ 77 Cross-Impact Matrix. ......................................................................................................................................... 77 Structured Comparison. ..................................................................................................................................... 78 Weighted Rankings. ............................................................................................................................................ 79 Change Analysis. ................................................................................................................................................ 80

GENERATE SCENARIOS ............................................................................................................................................ 81 Cone of Plausibility. ........................................................................................................................................... 81 Quadrant Hypothesis Generation Technique. .................................................................................................... 82 Alternative Futures Analysis. ............................................................................................................................. 83

UNDERSTAND POSSIBILITIES ................................................................................................................................... 84 Force Field Analysis. ......................................................................................................................................... 84 Counterfactual Reasoning. ................................................................................................................................. 85 Bow-Tie Technique. .......................................................................................................................................... 86

DEMONSTRATE ........................................................................................................................................................ 88 What If? Analysis. .............................................................................................................................................. 88 Chronologies and Timelines. ............................................................................................................................. 89 Mind Mapping. ................................................................................................................................................... 90 Matrices. ............................................................................................................................................................ 91 Decision Trees.................................................................................................................................................... 92 High Impact - Low Probability. ......................................................................................................................... 93

REVIEW PRE-PUBLICATION ..................................................................................................................................... 94 Dialectic Inquiry. ............................................................................................................................................... 94

of 113

Pre-Mortem Assessment. .................................................................................................................................... 95 MATCHING THE TECHNIQUES TO THE TASKS ....................................................................................................... 96

WRITE CLEAR, CONCISE, AND CLIENT FOCUSED REPORTS .................................................................. 99

CFINTCOM ANALYTIC PRODUCT STANDARDS .................................................................................................. 99 PRESENTATION OF ANALYSIS .............................................................................................................................. 100 LANGUAGE OF UNCERTAINTY ............................................................................................................................. 101 EXPRESSING ANALYTIC CERTAINTY ...................................................................................................................... 102

Provide strong judgements ............................................................................................................................... 102 Likelihood ........................................................................................................................................................ 102 Analytic confidence .......................................................................................................................................... 103

FOOTNOTES .......................................................................................................................................................... 104 THE REASONING PROCESS .................................................................................................................................. 105

Deductive Reasoning. ....................................................................................................................................... 105 Inductive Reasoning. ........................................................................................................................................ 105 Abductive Reasoning. ....................................................................................................................................... 105

FALLACIES............................................................................................................................................................ 106 PRE-ATIP PROCESSING ....................................................................................................................................... 106

THE TWO-HOUR CHALLENGE ........................................................................................................................ 108

BEST PRACTICES FOR INTELLIGENCE ANALYSIS REVIEW ................................................................. 110

ANALYTIC RIGOUR MAP .................................................................................................................................. 113

The Aide Memoire will be a work in progress for some time.

Anyone with suggestions as to how to improve this version should send their

comments to Gudmund Thompson ([email protected]) or Ramine

Shaw ([email protected]), noting the version number to which the

comments refer.

mailto:[email protected]

of 113

Introduction

The purpose of this Aide Memoire is to help defence intelligence analysts and managers to

produce clear, concise, insightful, and client focused analytic reports. While the material herein

may be useful for investigators (criminal, counter-intelligence, etc.), technical and/or scientific

analysts, the Aide Memoire is specifically intended to help guide Assistant Chief of Defence

Intelligence analysts working in the Regional and Transnational Directorate. As a result, the

material will not focus on the forensic, technical, scientific, or investigative aspects of analysis,

but rather on the tradecraft that will help unravel the difficult and complicated issues of

understanding and interpreting human motivations, intentions and future actions.

Analysts being guided by this Aide Memoire will need to be able to move beyond deduction and

induction, as the dominant descriptor of their reasoning process, to abduction or inference to the

best explanation.

Best Practices for Intelligence Analysts1

The Aide Memoire is structured to follow the Best Practices for Intelligence Analysts, developed

by the Intelligence Analyst Learning Program.

The Best Practices are reviewed in some detail starting on page 110.

1 Developed by John Pyrik for use in the Canadian interdepartmental Intelligence Analyst Learning Program.

Best Practices for Intelligence Analysts

1. Reflect on the problem, determining possible

approaches. 2. Be resourceful and systematic when collecting

information, documenting sources and noting caveats on usage.

3. Critically evaluate the quality of all information. 4. Develop multiple hypotheses / explanations. 5. Challenge assumptions, mindsets and biases. 6. Build collaborative networks. 7. Use structured analytic techniques. 8. Write clear, concise, well-documented, and client-

focused reports.

of 113

Definitions

Elements of Analysis

Analysts wishing to study how to produce and communicate insightful and forward looking

analysis will be confronted by a seemingly unending, duplicative and contradictory list of terms.

In order to deal with and differentiate between these various elements of analysis, this Aide

Memoire will use the terms evidence (uncertainty), assumptions, factors, criteria, drivers,

assessments, scenarios and hypotheses. These terms are defined below.

Evidence (Uncertainty).

Evidence is any available circumstance, piece of data, information or research, or analytic

reflection, etc. that supports, influences or contributes to a belief, proposition, or analytic

assessment.

Sometimes evidence can be considered as factual, or

irreproachable, especially in the scientific, technical and

forensic realms. Generally, however, and especially in

analysis that deals with the human condition, capabilities

or intentions, evidence should be dealt with as

embodying an element of uncertainty. From an analytic

perspective, evidence should almost always be

understood as occupying a position along an uncertainty

spectrum or range, where one end describes the lowest

possible value (zero percent in the graphic on the right) and

the other end the highest possible value (100 percent in this

example). By dealing with evidence as an uncertainty that

occupies an identified position along a range, analysts will

(or should) be open to the possibility that the selected

position on the spectrum may not be exactly correct.

Evidence is generally more compelling when dealing with

things that have already happened (a telephone intercept may

indicate that an individual was implicated in a plot), than it is when trying to discern the future

(an equivalent telephone intercept is likely less compelling when contemplating an individual’s

potential role in an potential, possible or upcoming plot).

Assumptions.

Sometimes, in analysis, the uncertain characteristic of evidence, or indeed the

lack of conclusive evidence, produces a debilitating stalemate that causes the

analytic process to stall. In these cases, it is often helpful to decide to “take

something for granted” in order to allow the analysis to proceed.

An assumption, then, is defined as something that is taken for granted, or

or

Assumption

or

Assumption

Evidence (Uncertainty)

100%0%


100%0%


When considering the fuel

efficiency of a new car, an

analyst would understand that

the indicated 4.2 litres/100 km

may actually be 4.6 or indeed

possibly 3.8 litres/100 km.

of 113

accepted as being true, without proof, for the purpose of argument or action. In an intelligence

context, an assumption allows some notion or idea to be

considered as fact, for the purpose of the analysis. A

fundamental characteristic of an assumption is that it either

is, or it isn’t!

In good intelligence analysis, assumptions, especially

assumptions that are considered key to the analytic line, need

to be specifically identified, in order to warn the reader that

the conclusions are contingent on the veracity of the

assumption.

Factors.

Factors play a very important role in analytic assessments, especially predictive ones, as they

allow the analyst to describe or contemplate influential elements without being constrained by

the specificity of evidence, or assumptions. Factors,

like evidence, can be measured, that is they can be

understood as incorporating a range (low to high), but

unlike evidence, where the analyst is expected to

make a determination as to where on the range the

evidence rests, or an assumption, which either is or

isn’t, a factor’s real value lies in its ability to facilitate

comparison when taken as a whole. This comparison

is made possible when the analyst identifies the level

of influence (weak to strong) that a factor has in the

particular analysis being contemplated. Exactly the

same factors may have markedly different levels of

influence in different situations.

Assessing the Future.

In producing predictive analysis dealing with future

issues, various approaches are available. The analyst

may identify specific pieces of evidence that are

expected to be important, and use an assessment of their likely position on the uncertainty scale

to predict the future. Or, the analyst can simply identify a number of assumptions and predict the

future based on them. Or, the analyst might identify the factors that will be influential in

determining how the future will unfold; and then, by examining the likely influence of each of

the factors on the issue at hand, produce an assessment of what the future will likely hold. This

latter approach is considered to be the best in many circumstances.

This is not to say, in situations where something has happened regularly for a considerable

period of time, that an assumption that this will continue cannot be useful in assessing the future.

Further, where evidence is strong and well understood, it is reasonable to use that understanding

to predict the future. However, in both cases, the good analyst will look for other factors that

Assumption

An analyst contemplating the

purchase of a new family car

may want to identify as a key

assumption the expectation

that the family unit (husband,

wife, two kids) will remain

intact throughout the process.

Factor

Gas consumption and reliability may

both be factors in the purchase of a

new car. Gas consumption is

measurable (1 to 25 litres per100 km)

as is reliability (1- to 5-star rating),

but their influence in the decision

making process, when they are

considered as a whole, may be very

different. A pensioner may feel that

gas consumption is far more

important than would someone that

has just secured a high paying job.

Similarly, a new mother may think

that reliability is critical to her car

purchasing decision.

of 113

Evidence (Uncertainties) Factors

Assumption

Hyp

oth

esis

Hyp

oth

esis

Hyp

oth

esis

Assumption

Hyp

oth

esis

Scenario (s)

Assessment

Evidence (Uncertainties) Factors

Assumption

Hyp

oth

esis

Hyp

oth

esis

Hyp

oth

esis

Assumption

Hyp

oth

esis

Scenario (s)

Assessment

may become influential in determining whether or not the future will look like the past and base

the analysis on this broader understanding.

Drivers and Criteria.2

A driver is a factor whose influence is significantly

more dominant than others. There can be several

drivers in a complex situation, and the term driving

factor may be used. Clearly identifying drivers or

driving factors in analysis is often very useful.

A criterion is also a factor, but it generally embodies

a specific standard against which something can be

judged or decided. A criterion is a condition that

either must or must not be fulfilled for the assessed

scenario to exist, making it (like an assumption)

binary.

Assessments, Scenarios and Hypotheses.3

An assessment is an analytic position. It may arise directly from evidence or a factor or two

(indeed, it may arise directly from an assumption!). Generally, however, assessments are

informed by scenarios or hypotheses (even if they are unstated) – which in turn rely on evidence,

factors and assumptions.

A scenario is an imagined or projected sequence

of events that can be used to demonstrate how a

potential future or specific end-state could come

about. In order to be effective, scenarios must

be based on a “realistic” interpretation of events

– not a fanciful or imaginative one that the

intelligence client may interpret as

unreasonable.

A hypothesis is a proposition set forth as an

explanation for the occurrence of some

specified event or phenomena. A hypothesis

may explain a scenario, or several hypotheses

may describe separate aspects of a single

scenario. Hypotheses can be disproven (the

Black Swan phenomenon – see page 42), but they can rarely be proven. If a hypothesis is

disproven, it does not necessarily mean that the scenario it supports in invalid, though the

scenario would likely need to be adjusted to compensate.

2 Based on work done by Maj Richard Little, 2014.

3 Based on work done by Maj Richard Little, 2014.

Factors, Drivers and Criteria

In an analysis regarding the purchase

of a new car, factors may include:

- gas consumption,

- cargo capacity,

- safety, and

- reliability.

A driver, or driving factor, would

almost certainly be price, while, for a

family with small children or an

aging parent, 4-door might be a

criterion.

of 113

The Intelligence Cycle

While various intelligence agencies

and authors have made numerous

modifications and/or additions to

the intelligence cycle, it remains

the foundation upon which all

intelligence work is done. The

intelligence cycle is generally

thought to consist of four phases:

1. Identify your intelligence

needs.

2. Collect the information.

3. Analyse the information

4. Report the information.

There are, however, numerous other interpretations. The one used to describe intelligence-led

policing is shown below.

PRIORITY

SETTING

DIRECTION &

PLANNINGCOLLECTION

EVALUATION

ANALYSIS DISSEMINATIONDIRECTION &

PLANNINGINVESTIGATION

ARREST

DISCLOSURE &

PROSECUTION

IMPACT

ASSESSMENT

INTELLIGENCE

CYCLE

ENFORCEMENT

CYCLE

COLLATION

PRIORITY

SETTING

DIRECTION &

PLANNINGCOLLECTION

EVALUATION



ARREST

DISCLOSURE &

PROSECUTION

IMPACT

ASSESSMENT

INTELLIGENCE

CYCLE

ENFORCEMENT

CYCLE

COLLATION

PRIORITY

SETTING

DIRECTION &

PLANNINGCOLLECTION

EVALUATION



ARREST

DISCLOSURE &

PROSECUTION

IMPACT

ASSESSMENT

INTELLIGENCE

CYCLE

ENFORCEMENT

CYCLE

COLLATION

PRIORITY

SETTING

DIRECTION &

PLANNINGCOLLECTION

EVALUATION



ARREST

DISCLOSURE &

PROSECUTION

IMPACT

ASSESSMENT

INTELLIGENCE

CYCLE

ENFORCEMENT

CYCLE

COLLATION

1. Identify

your intelligence

needs.

2. Collect

the information.

3. Analyse

the information.

4. Report

the information.

1. Identify

your intelligence

needs.

2. Collect

the information.

3. Analyse

the information.

4. Report

the information.

1. Identify

your intelligence

needs.

2. Collect

the information.

3. Analyse

the information.

4. Report

the information.

1. Identify

your intelligence

needs.

2. Collect

the information.

3. Analyse

the information.

4. Report

the information.

of 113

Levels of Intelligence Support

It is generally accepted that intelligence analysis

can be separated into three levels, though the

specific definitions of these levels is often the

source of heated debate. For the purposes of

this aide memoire, the three levels of

intelligence are defined as follows:

Strategic Intelligence – is required for the

formulation of policy and plans at the

departmental/agency, national, or

international level. This is the highest level

intelligence, and as such should be all-source, estimative, and focused on future or

comprehensive threats.

Operational Intelligence – is required for the planning and preparation for future operations

and/or investigations.

Tactical Intelligence – is required for the planning and execution of ongoing operations.

Nothing says, however, that these levels need to be symmetrical. Indeed, by separating each

level of intelligence into an operating environment and intelligence support, one can envisage a

very asymmetric juxtaposition, where each level of intelligence support provides input to all

levels of the operational environment.4

4 Developed from “a conceptual model of military intelligence in the contemporary operating environment”

proposed by Jim Cox in his doctoral thesis, Apr 2009.

Strategic

Operational

Tactical

Strategic

Operational

Tactical

Strategic

Operational

Tactical

Strategic

Operational

Tactical

Strategic

Level Operational

Level

Tactical

Level

Operating

Environment

Strategic

Level Operational

Level

Tactical

Level

Tactical

Intelligence

Operational

Intelligence

Strategic

Intelligence

Intelligence

Support

Tactical

Intelligence

Operational

Intelligence

Strategic

Intelligence

OperatingEnvironment

IntelligenceSupport

StrategicLevel Operational

Level

TacticalLevel

TacticalIntelligence

OperationalIntelligence

StrategicIntelligence


IntelligenceSupport


Level

TacticalLevel




Strategic

Level Operational

Level

Tactical

Level

Operating

Environment

Operating

Environment

Strategic

Level Operational

Level

Tactical

Level

Tactical

Intelligence

Operational

Intelligence

Strategic

Intelligence

Intelligence

Support

Tactical

Intelligence

Operational

Intelligence

Strategic

Intelligence


IntelligenceSupport


Level

TacticalLevel





IntelligenceSupport


Level

TacticalLevel




of 113

Categories of Intelligence

Similarly, intelligence analysis can generally be divided into three categories, and while

definitions differ, this aide memoire will use the following:

Basic Intelligence – is intelligence, on any subject, which may be used as reference material

in planning. It provides background material on potential adversaries and/or operational

environments, and can deal with equipment, capabilities, personalities, infrastructure, socio-

political and cultural matters, or geography and other environmental information. It can be

permanent, or subject to change, it is normally independent of a specific crisis, and is often

maintained in databases.

Current Intelligence – is intelligence that reflects the current situation at either the strategic,

operational, or tactical level. It almost certainly describes current events and their

consequences, but also may predict changes and estimate future intent. Current intelligence

is perishable and may change rapidly.

Estimative Intelligence – is intelligence that looks to the future in an attempt to prevent

strategic or tactical surprise. Warning intelligence is a form of estimative intelligence.

Types of Strategic Analysis Products

Context analysis or environmental scanning is a way to analyze the environment in which an

intelligence client or decision maker operates. It deals with both internal and external issues and

while it often concentrates on the macro aspects of the situation (the drivers or factors), it may

also be very focused on important details.

Warning analysis provides senior decision makers with effective warning against a range of

complex regional and global actors and events, the adverse outcome of which could threaten

national and Allied interests.

Opportunity analysis is the strategy of assessing the potential for a change to affect an outcome,

relationship, advantage, etc. for either an adversary or a client, and the consequence of that

change.

A strategy of providing clients with assessments that evaluate both the opportunities and the risks

of various options they might be considering will always be valued. The analyst’s role is to

identify the forces and factors that may be influential in the situation and to focus on how to

mitigate bad scenarios from unfolding and how to enhance the prospects for positive

developments to occur.

With any opportunity analysis, three additional key questions should be addressed. First, what

are the benefits of implementing this policy, change or enhancement? Next, what adverse effects

are likely to occur when the implementation takes place? Finally, how will the implementation

affect the overall operation, relationship, etc?

of 113

Analytic Rigour

“Analytic rigour” is a term that is widely used, but few analysts or managers can actually

describe what it entails – not very helpful!

For the purposes of this Aide Memoire, analytic rigour is based on the CFINTCOM Analytic

Product Standards (page 99), the CFINTCOM-approved Probability Terms (page 103), the

concept of Analytic Confidence (page 103), the Data Diagnostic process (page 69), which

includes Paul and Elder’s template for “Analysing the Logic of an Article” (page 35), and the

Denial and Deception checklists (page 37); and the Key Assumptions Check (page 70).

Summary:

To exhibit analytic rigour, intelligence analysts should:

- Make accurate judgements,

- Be clear,

- Be insightful, timely and relevant, and

- Highlight trends over time

This should be demonstrated through:

- Identifying confidence in analytic judgements,

- Identifying assumptions,

- Considering alternate hypotheses,

- Identifying indicators, and

- Applying structured analytic techniques.

Finally, analytic rigour depends on the quality and reliability of the evidence as examined

through the lens of the provider, the information itself, its relevance, and the potential for

denial and deception.

See page 113 for a graphic of Analytic Rigour, developed using CmapTools5 software.

Temporary Postmodernism6

Good intelligence analysts, in the process of preparing an analytic product, should make a point

of (within reason) challenging everything. One way of doing this is by temporarily becoming a

postmodernist. From this perspective, the analyst will adopt the position of someone who

believes:

- that there is no privileged access to the truth (as some religious leaders or

environmental advocates would have us think);

- that knowledge is socially constructed (and as societies change, so does knowledge);

- that there are no clear standards for ‘progress’ (a more technologically-assisted or

economically-advanced or culturally-diverse society is not necessarily a better one);

- that no culture is inherently superior; and finally

- that our perspective influences our reality (think about five blind guys holding onto

various parts of an elephant that they are experiencing for the first time).

5 Developed by the Institute for Human and Machine Cognition.

6 Developed from a talk given by Dr Jeffrey Tang of JMU to the 5-Eyes Analytic Training Workshop - 19 Nov 2010

of 113

Reflect on the Problem, Determining Possible

Approaches.

What is the Question?

In virtually all aspects of intelligence analysis, the first, and often most perplexing problem is

determining the question that needs to be answered. Whether the analyst is defining a scientific

research question, dealing with a current issue, or supporting an operation and/or investigation,

the importance of this formative aspect cannot be over-estimated.

W5H & SW.

It is perhaps intuitive that the oft-cited W5H can be helpful here, and analysts

are encouraged to answer as many of these questions as possible in defining

the intelligence problem. W5H is, however, insufficient in that it does not

deal with the issue of SO WHAT? Indeed, without an identifiable SW, any

piece of intelligence analysis, no matter how insightful, or clear, may simply

be busywork.

Often the details of what needs to be produced can be found in departmental

policy or standard operating procedures, but there will be times when the

analysis is in response to a client request, and others where the analyst acts as

initiator.

When responding to a client’s request, it is important that the actual client’s needs are identified,

in detail, to the analyst. Given that there are often managers and/or client relations personnel

between the client and the analyst, it is important that everyone involved in the process is attuned

to this requirement. Often, the best way to ensure the analyst understands the client’s

requirements is for the two of them to engage in a discussion of the issue at hand.

When considering an analyst-initiated analysis project, the analyst must be prepared to “sell” the

benefit of the initiative. To this end, the analyst should define the issue, the likely consequences,

and the potential impact on operations, policy and planning.

The questions, who, what, where, when, why, how, and so what, have a broad application in the

processes of intelligence analysis. The table below shows how these seven questions can be

applied when thinking about:

the client of the analysis (and to the client of the client of the analysis),

the content of the analytic product,

the contacts (or sources) that the analyst is relying upon, and

the individuals (or organisations) with whom the analyst should collaborate when

undertaking any analysis.

W5H & SW

Who?

What?

When?

Where?

Why?

How?

So what?

of 113

(W5H & SW)C47

Client Content Contact Collaboration

Who is the client? Who is it about? Who? Who is this? Who has extra

information?

What is wanted? What is it about? What? What is being

offered?

What might be

added?

Where is he

located?

Where did/will it

happen? Where?

Where is the info

coming from?

Where are they

located?

When does she

need it?

When did/will it

happen? When?

When was the info

acquired?

When can I reach

them?

Why is it wanted? Why did/will it

take place? Why?

Why is it being

proffered? Why not try?

How can it be

delivered?

How did/will it

come about? How?

How was it made

available?

How can I reach

them?

What is the

implication?

What is the

implication? So What?

What is the

implication?

What is the

implication?

Policy Prescription.

The old adage that intelligence analysis should be policy/operationally relevant but

policy/operationally neutral is extremely important and must be kept in mind throughout the

process. The role of intelligence analysis is to inform operations and policy development, to

make tasks of decision making easier. Intelligence assessments should not tell the clients what to

do; though judgements concerning the consequences of courses of action may be offered.

What is the nature of the question?

Puzzles, Mysteries and Messes.

Analytic problems run the gamut from simple to extremely complicated. A puzzle is the

simplest. A puzzle has clearly defined boundaries, and only one correct answer. An example of

a puzzle is: “Who put the bomb on Air India flight 182?". There is one right answer, and that

answer is at least theoretically knowable even if it is difficult or impossible to prove in court.

A problem is more complex than a puzzle. A mystery has clear boundaries, but has no single

correct answer. An example of a problem in this sense is: “How do we bring the perpetrators of

the Air India bombing to justice?” This is a problem for several reasons, one of which is that

different individuals and legal systems have different opinions and rules on what constitutes

justice in a case of ideologically-motivated mass murder.

7 Developed from a presentation given by Dennis St. John, NSA, at the 4-Eyes Analytic Training Workshop, hosted

by James Madison University’s Institute for National Security Analysis on 1 Sep 2009.

of 113

1

One

System

requires

evidence

& reasoning

within a

system

a correct

answer

knowledge

2

No

System

calls for

stating a

subjective

preference

a subjective

opinion

cannot be

assessed

3

Multi

System

requires

evidence

& reasoning

within multiple

systems

better & worse

answers

judgement

1

One

System

requires

evidence

& reasoning

within a

system

a correct

answer

knowledge

1

One

System

requires

evidence

& reasoning

within a

system

a correct

answer

knowledge

2

No

System

calls for

stating a

subjective

preference

a subjective

opinion

cannot be

assessed

2

No

System

calls for

stating a

subjective

preference

a subjective

opinion

cannot be

assessed

3

Multi

System

requires

evidence

& reasoning

within multiple

systems

better & worse

answers

judgement

3

Multi

System

requires

evidence

& reasoning

within multiple

systems

better & worse

answers

judgement

A mess is the most complex of the three, and is also known as a “wicked problem.” A mess is a

complex issue without defined boundaries, and may even lack any clear formulation. Messes

tend to be interactive open systems, unquantifiable, and to contain multiple interrelated

uncertainties. Messes do not have single, correct solutions, and they do not even provide any

way of knowing when a solution has been reached. Instead they have a range of approaches

which may improve the situation, but which carry with them both intended and unintended

consequences which change the nature of the mess. An example of a mess is: “How do we

prevent terrorism?”

Three Kinds of Questions.8

Another way of approaching a problem is to

figure out what type of question it involves. Is

it a question with one definitive answer? Is it a

question that calls for a subjective choice? Or

does the question require the consideration of

competing points of view?

Paul and Elder provide the diagram on the right

as a way of demonstrating these three options.

How much detail?

When determining the nature of the question, it

is important to consider the amount of detail

that will be required to adequately deal with the

issue. There is little sense in expending the

time and energy to produce an all-

encompassing paper on an issue when the only

thing that is required is a simple yes or no answer. Likewise, a short answer may be completely

inadequate in other circumstances.

Time.

An insightful, detailed assessment provided after its usefulness has expired is a waste of time.

Determine when the client requires the assessment and then meet that timing!

Client Knowledge.

Another consideration is the level of knowledge of the client. Again, this is a spectrum where on

one end an expert in the issue only needs to be provided with the specific answer to the question

at hand. This type of decision maker often has a preconceived idea of what the answer “should”

be. The challenge for the analyst, if the answer is in fact something else, will be to convince this

expert that an alternative explanation is appropriate. At the other end of the spectrum is a client

that does not even know that the issue exists. For this latter group of intelligence users, the

8 From The Miniature Guide to Critical Thinking Concepts and Tools, Paul and Elder, p.18

of 113

analyst must define the issue in terms the client can understand (unfamiliar words and concepts

will need to be explained), take special care that the logic of the assessment is transparent, and,

throughout, keep in mind that the presentation will need to be compelling enough to hold a

potentially wavering attention.

Issue Redefinition9

Many analytic projects start with an issue statement outlining: what is the issue, why is it an

issue, and how will it be addressed. Unfortunately, it is often the case that the “issue definition”

stage has been given short shrift by either the client (who may or may not know what is actually

wanted) or the analytic team’s client relations group. All this to say, the “issue” that is presented

to the analyst is often not the question that needs to be answered.

To deal with this shortcoming, analysts or analytic teams may want to experiment with different

ways to redefine an issue. This is important, because seemingly small differences in how an

issue is defined can have significant effects on the direction of the research and consequent

analytic products.

Reframing the Question is a structured analytic technique (see page 68) which can be used to get

the analytic process started. While it will not guarantee the analyst pursues the right avenue of

inquiry, it may forestall unnecessary effort on a poorly stated issue.

Identifying Factors

Once the question has been defined, analysts should then undertake the identification of the

factors, criteria and drivers that can be expected to be helpful in understanding the issue - from

the perspective that will be useful to the client.

This identification process can be undertaken through the use of the structured analytic

techniques in the Develop group. The Environmental Scanning techniques – SWOT, Activity

Systems Model, STEMPLES and Understanding Groups (see page 64), and Outside-In Thinking

technique (see page 65) can be especially effective. A quick session with the client can also be

invaluable at this stage.

Summary

The process of reflecting on the problem and determining possible approaches is fundamental to

the tradecraft of analysis. Without a specific and thorough knowledge of the question that needs

to be answered, the type of problem being addressed, the amount of detail that is required, and

the client familiarity with the issue, much analytic time and effort will undoubtedly be wasted.

Concomitantly, and perhaps more alarming, without this prior knowledge, the probability that

the analyst will provide an answer that is inappropriate, incomplete, or unhelpful will increase;

perhaps irrevocably damaging the client’s confidence in intelligence.

9 Based on Richards J. Heuer, Jr. And Randolph H. Pherson’s Structured Analytic Techniques for Intelligence

Analysts, copyright 2011 by CQ Press.

of 113

Be Resourceful and Systematic When Collecting

Information, Documenting Sources and Noting

Caveats on Usage.

Once the analyst has determined the question that needs to be answered, the type of problem that

it is, and the detail and timeframe in which it needs to be answered, the next phase of the process,

collection, can begin.

Here, the human condition predisposes us to develop an explanation based on previous

experience and then set out to find evidence that supports this hypothesis. This is to be resisted

as vigorously as possible.

Indeed, during this portion of the analytic process, the objective is simply to gather

data/evidence/information which may be relevant to the question at hand – even if it appears to

be contrary to an initial belief. This should be done as thoroughly and as broadly as possible –

keeping in mind the timeframe/complexity of the task – and the actual question.

Guiding Factors

Analysts undertaking work on a new subject should never be content to rely on the information

that is pushed to them by others, and they should probably not attempt to assimilate everything

that is available on the issue. While analysts will often be constrained by the unavailability of

information, due to the secrecy surrounding much of intelligence work, more often than not, they

will be faced by an overwhelming overabundance of data – much of which is irrelevant to the

task at hand.

One of the best ways of triaging the available information, and determining what type of

information should examined, is by focusing on the data that is related to the factors that have

been identified as likely to provide insight into the problem being studied. That will allow the

analyst to set aside the other, unhelpful but often very interesting, data. This in turn will

facilitate quicker understanding of the issue, in terms that will be of interest to the client.

Often, analysts will be well served by looking outside of their normal environment to collect the

data that is relevant to the issue at hand.

INT Comparison Chart: Pros and Cons10

It is generally accepted that not all sources of information are created equal. Given scarce

resources (especially time) it is often important that the main collection effort be directed against

the sources of highest potential value. To this end, the following chart contrasts and compares

intelligence garnered from open sources with that which is acquired from human, and technical

(signals and imagery) sources.

10

Developed by John Pyrik for use in the Canadian interdepartmental Intelligence Analysis Learning Program.

of 113

OSINT

Collection risk: low

Cost: low

Pro

broadest coverage

good for identifying emerging issues (can be very up-to-date)

easy to access

generally, minimal risks in collection

cheap

no restrictions on use (easy to share) quality varies, but good for a “first pass”

Con

high volume

unstructured

may alert a sophisticated target

may contain diatribes, polemics, hyperbole, propaganda, and some nuts!

difficult to assess quality (scholarly journals to personal blogs)

language issues

HUMINT

Collection risk: high

Cost: medium

Pro

first-hand information

richest and most detailed information

goes to motives and intent

target will not usually know information reported

flexible - can cover anything (in theory)

only possible source for some hard targets

relatively inexpensive

Con

difficult to recruit good sources on hard targets

may take a long time to set-up a source

policy may limit who can be recruited

sources have biases and personal agendas

they make mistakes, exaggerate their role or access (which may be very limited or

indirect) and they embellish information

money does not guarantee loyalty

a source may be a fabricator, double agent, or dupe

information is hard to verify (i.e. “single and sensitive source”)

good tradecraft is necessary to cope with these problems

large administrative burden (for organization and handler), medium to high personal

risk

of 113

SIGINT

Collection risk: low*

Cost: high

*Note: While the personal risk is low, SIGINT collection is politically sensitive and therefore a

high risk for governments.

Pro

large volume of high quality information

remote access and usually not dangerous to collect

broad coverage of potential targets

potential to provide insight into plans and intentions

Con

hard to use or share due to high classification

passive (even random)

high volumes of low yield intelligence makes gold hard to find

costly - requires intensive resources equipment, code-breaking, decoding, translating

and disseminating

many insignificant and significant actors who are poorly informed

many key targets know how to avoid detection

deception easy for knowledgeable targets

low tech targets with no SIGINT profile may be most important challenge to store

and retrieve

IMINT

Collection risk: low

Cost: high

Pro

persuasive

very broad coverage

provides essential information on physical items

intensive magnification from satellites provides very specific information, especially

on military assets, infrastructure, damage assessment, activities which leave marks

good for change detection

photo interpretation is a well-developed art

adversary probably underestimates our capability

Con

not always available

more useful for capability than for intent

more useful for tactical/military than strategic/civilian

material can be misinterpreted

many targets can disguise or hide assets

of 113

Collection Planning

The collection process, especially one preparing for a complicated or long-term project, can

often benefit from the assembly of a collection plan. This can be a formalised process like

DND’s CCIRM (Collection Coordination and Intelligence Requirements Management) or a

much simpler, tailored-to-the-task,

spreadsheet or matrix. In either case, the

collection plan will record, in a single place,

what the analysts need, and where they think

they can get it.

Requirements.

Requirement is the term that is used – by

some agencies – to describe any number of

aspects of “what needs to be done” at various

stages in the intelligence process. This lack

of specificity can be confusing (a client’s

requirement is a piece of intelligence advice, a

collector’s requirement is a piece of information, an analyst’s requirement may be an analytic

product, etc.), but the term is helpful nonetheless.

From the analyst’s perspective, an

Intelligence Requirement (IR) is probably

best understood as a question that describes

a gap in intelligence knowledge, while a

Priority Intelligence Requirement (PIR),

as its name suggests, is a gap in intelligence

knowledge that should be filled on a priority

basis. Often Priority Intelligence

Requirements encompass numerous

Intelligence Requirements such as is shown

in the box at the right.

PIRs and IRs such as those shown in the box, however, are often far too broad to be of much use

to a collector or researcher. (Can you imagine asking an imagery analyst to answer the IR –

How much uranium is Lilliput enriching?) For the purposes of identifying exactly what it is the

analyst needs to know, Essential Elements of Information (EEI) and Indicators may be

identified. Using the Lilliput strategic threat example, a reasonable EEI for an imagery analyst

may be:

What is the metric volume of tank six at power-plant Alpha, located at 102.22N / 44.33W?

Similarly, a measurable indicator may be:

A heat-bloom of more than 4,000C at nuclear plant Bravo located at 102 º 22’ N / 40º 33’ W.

When considering the Client Requirement –

What is the strategic threat from Lilliput?

The Analysts PIR may be:

What is Lilliput’s nuclear weapons

capability?

While IRs may be:

How much uranium is Lilliput enriching?

Does Lilliput have a strategic weapons

delivery platform?

Collection Coordination and Intelligence

Requirements Management

DND’s CCIRM is not only a way for members of

the Intelligence Community to coordinate

requirements. CCIRM is also a tracking

mechanism that ensures producers are

accountable for the collection and/or processing

they are tasked with. Even more importantly,

CCIRM tracking serves as the correspondence-

of-record for this activity, upon which senior

leadership can base decisions about

prioritization, resource allocation, and policy.

of 113

Many other terminologies can be used to describe exactly the same things as PIRs IRs, EEIs and

indicators. Indeed the Criminal Intelligence Collection Plan, on page 24, uses “Objective” and

“Investigative Questions.”

Indicators.11

The main function of an indicator is to help identify persons, activities, developments, or trends

of interest. Indicators can be used to:

Identify which scenario or alternative future is emerging.

Alert one to unanticipated developments that might otherwise go undetected.

Validate existing hypotheses or viewpoints.

Make the warning process more rigorous.

A pre-established set of indicators are often used to:

Suggest a target’s activities or behaviour is consistent with an established pattern

(backward-looking).

Suggest a given hypothesis is correct or a predicted scenario is emerging (forward-

looking).

The best indicators satisfy all five of the following characteristics. (The first three are critical for

any good indicator, the fourth and fifth are not always possible to satisfy.)

Observable/Collectible/Practical. An indicator must be able to be observed and collected

at suitable time periods.

Valid. An indicator must accurately measure the concept being considered.

Reliable. Data collection must be reliable and different people must be able to see the

same thing.

Stable. An indicator should be useful over time to allow comparisons.

Unique. An indicator should only measure one thing.

Indicators of Change.

In this process, an analyst or team creates a list of indicators or signposts of observable events

that could be expected to become apparent if a postulated situation is developing. The technique

(see page 72) can be used whenever an analyst needs to track an event over time or monitor and

evaluate changes.

11

Adapted from Handbook of Analytic Tools & Techniques, Pherson, 2008, p 16

of 113

Indicators Validator.

Indicators, unfortunately, often do not demonstrate all of the desired characteristics as well as the

analyst would like. Indeed, a critical question that is often left un-asked is whether a given

indicator would appear only in the scenario to which it is assigned, or whether it might also

appear in one or more alternative scenarios. The structured analytic technique Indicator

Validator (see page 73) is specifically designed to expose the diagnostic value of indicators in an

attempt to ensure that they are not relied upon inappropriately.

Collection Plans.

Collection plans vary widely in form, format and content, though most will contain some or all

of the following:

Reference to a specific intelligence problem.

Increasingly granular questions that need to be answered.

A list of agencies/sources from which the information can be sought.

A way of indicating which agencies/sources have been tasked/asked/queried on each

specific question.

A way of indicating the status of the requests.

Follow-up action required.

Below is a pair of examples of Collection Plans,12

each dealing with a different strategic security

issue, and each using different terminology to describe the information being sought.

12


Scenario 1 Indicator

Scenario 2

Scenario 3

Scenario 4

Indicator

Indicator

Indicator

Indicators

ValidatorMost

Discriminating Indicators

(Listed in rank order

and by

scenario.)

(Evaluates the

likelihood of each indicator to emerge in each scenario.)

- Highly Likely- Likely- Possible- Unlikely- Highly Unlikely


Scenario 2

Scenario 3

Scenario 4

Indicator

Indicator

Indicator

Indicators

ValidatorMost



and by

scenario.)

(Evaluates the




Scenario 2

Scenario 3

Scenario 4

Indicator

Indicator

Indicator

Indicators

ValidatorMost



and by

scenario.)

(Evaluates the




Scenario 2

Scenario 3

Scenario 4

Indicator

Indicator

Indicator

Indicators

ValidatorMost



and by

scenario.)

(Evaluates the



of 113

Criminal Intelligence Collection Plan Objective: COULD CANADIAN RADIOISOTOPES BE USED TO MAKE A

"DIRTY" BOMB

Investigative

Questions

Information Sources Collection

Method

Collection

Method Details 1. What

radioisotopes could

be used in a dirty

bomb?

2 What

radioisotopes are

present in Canada?

3. What level of

interest exists in

terrorist groups to

use a dirty bomb?

4. How accessible

are radioisotopes

along their supply

chain/life cycle?

A) Internal Databases

a) NCDB (3, 4)

b) PROS (3, 4)

c) SCIS (1, 2, 3, 4)

d) etc

B) Open Sources

a) Internet (1, 2, 3, 4)

b) Newspapers (1, 2, 3,

4)

c) Conf. mat. (1, 2, 3, 4)

d) Sci / Tech lit. (1, 2, 4)

C) Dom. Depts / Agencies

a) Health Canada (1, 2)

b) Transport Canada (2,

4)

c) ITAC (1, 2, 3, 4)

d) etc

D) Provincial / Local

Agencies

a) Law enfor. (3, 4)

b) Emerg. Mgmt. (1,2, 4)

c) Min. Environment (2,

4)

E) Foreign Depts / Agencies

a) DHS (1, 2, 3, 4)

b) CIA (1, 2, 3, 4)

c) FBI (1, 2, 3, 4)

d) FEMA (1, 2, 3, 4)

F) Etc.

A) a-g: Intelex &

direct query

B) Intelex

C) Liaison Officers &

direct contact. in-

person and by

telephone. interviews

and questionnaire).

D) Liaison Officers &

direct contact

E) Liaison Officers &

direct contact

A)f) May require

travel to BC. $3,000

B) May need to

purchase reports, buy

subscription, or get

special access. $500

C) Information

sharing issues (non-

derivative use); may

require legal advice.

Possible travel (site

visits). $1,000

D) Information

sharing issues.


visits). $5,000

E) Information

sharing issues.


visits). $10,000

of 113

MSOC East Information Collection Plan Date Created: 20 Dec 06

Intelligence Problem: What is the threat to Canadian Vessels transiting the Suez Canal, Red

Sea and Gulf of Aden Region

Classification: Unclassified

Information Required NLT: 04/01/07

PIR IR

Sources and

Agencies Results Status

OS

INT

IMB

ON

I

TR

AN

SC

OM

ITO

CG

CS

E

AS

TJ

IC

Completed

Pending

Incomplete

PIR 001:

What is the

terrorist

threat within

the specified

region?

IR 001.01 -

What are the

known terrorist

factions within

the region?

X X X X

1.ONI - Reports rec'd.

2. TRANSCOM - Nil

report

3. ITOCG - 3 x associated

reports

rec'd

4. ASTJIC - Regional TRA

rec'd

IR 001.02 -

What have been

the known or

suspected

terrorist

associated

events in this

region?

X X X X X

1. OSINT – Searched,

multiple

results

2. ONI - Maritime reports

rec'd -

(strong assessment)

3. TRANSCOM - Nil

report

4. ITOCG - Included IR

001.01

5. ASTJIC - Included IR

001.01

GAP - Who are

potential emerging

factions in light of the

expanding insurgent

activity in Iraq?

GAP - Will they exploit

this region, and / or

target it?

IR 001.03 -

What have been

the regional

terrorist targets

for attack?

X X X X

1. OSINT - Searched with

multiple

results

2. ONI - Reps rec'd

3. ITOCG - Included IR

001.01

4. ASTJIC - Included IR

001.01

GAP - What have been,

if any, the specific

maritime targets?

ONI - forwarding

additional synopsis

Conclusion pending

of 113

Data Sorting13

Data Sorting is particularly effective during the initial data gathering stage when information

elements can be broken out into categories or subcategories for compilation and comparison.

These groupings should be aligned with the factors that were identified during the earlier Identify

Factors (see page 17) phase. The technique is most helpful when massive stores of data can be

assembled in a software program, such as a spreadsheet, where, by grouping material under the

appropriate factor, trends, similarities, differences and gaps can be identified.

The method consists of five steps:

Step 1: Review the factors that have been identified as being important to understanding

the issue at hand and, if appropriate, devise categories and sub-categories into which the

evidence can be broken down. Then place the evidence (data, circumstance, information

or research, or analytic reflection, etc.) into the appropriate portion of the spreadsheet.

Step 2: Review the material in the database or spreadsheet to identify key fields that may

allow you to uncover possible patterns or groupings or gaps.

Step 3: Group those items according to the schema you defined in step 1.

Step 4: Choose a category and sort the data in that category looking for trends or gaps.

Step 5: Review your sorted evidence to see if there are alternative ways to sort it.

Documentation of Sources

In the process of collecting information, it is often important that a record be kept of what has

been found, and where it stored, so that these details may be recalled later. Some analysts can

benefit from very sophisticated computerized document retrieval programs that can handle all of

their information storage and retrieval requirements. Others, however, will need to access a

diverse and mutually exclusive set of information sources (books, TV programs, lectures and

interviews, Internet and classified sources, and personal experience to name but a few). In either

case, tools that will allow the analyst to recollect the data can be of great assistance. To this end,

annotated bibliographies, the Analyst’s Notebook software, and the Cornell Note Taking system

are highly recommended.

Annotated Bibliography.

Annotated bibliographies should, if possible, contain the following items of information:

Source (agency, publication, individual, etc.)

Report number, serial number or message reference number,

Item or article title (abbreviated if necessary),

Date (of publication/storage),

Classification and caveats,

Storage location, and

A short description of the salient points.

13

This section is based on the DIA Analytic Methodologies, A Tradecraft Primer: Basic Structured Analytic

Techniques, First Edition, March 2008, p 33-35.

of 113

Keep track on-line.

Keeping track of on-line sessions can be done using free software such as:

Cogitum Co-citer captures the selected text, its Internet address, its title and the date. You

can also assign your own comments to the texts.

Net Snippets is a sophisticated way to make your Internet research experience more

productive than ever.

Analysts Notebook.

Especially for analysts conducting

investigations in which relationships

between persons, data, events, and

activities are important, the link

diagrams produced by the Analysts

Notebook (or similar software)

provides a convenient method of

assembling and displaying data.

The advantage of using link diagrams

to store data is that they can:

integrate data from different

sources,

highlight key relationships,

serve as a visual briefing aid

for team members, prosecutors

and juries,

show when cases overlap

(when multiple charts are

combined), and

provide a cumulative snapshot of a case.

Examples of annotated bibliography entries.

Heuer Jr, Richard J. Psychology of Intelligence Analysis. Langley: CIA, 1999

http://www.odci.gov/csi/books/19104/index.html (Unclas)

Exceptional & very readable introduction to the issues of analytical bias as it

can apply to cross-cultural situations (as well as analysis in general).

Rotberg, Robert I. When States Fail – Causes and Consequences. Princeton:

Princeton University Press, 2004. ISBN 0691116717 (DG’s Library) (Unclas)

Very good work – intriguing insight into state collapse and failure.

http://www.cogitum.com/

http://www.netsnippets.com/basic/index.htm

http://www.odci.gov/csi/books/19104/index.html

of 113

While every analyst will have a personal preference for how to store data, some basic rules can

be useful:

show organizations as boxes and people as circles (icons may also be used),

place people within the box for their organizations,

link individuals and organizations with lines (solid for confirmed links, dotted or dashed

for suspected links),

use coloured links to represent phone calls, financial transactions, commodity flows, etc.,

avoid crossing lines,

place key entities in the centre, and

always include a legend or key.

Additional information can be added to a chart:

grade and source type

o every icon and link can be graded

o the origin of the data can be coded as

source type

use descriptive link types

o links can be generic or indicate the

nature of the association, e.g. owner, subscriber, member, etc.

attributes - visual cues can be added to icons and links

cards

o behind each icon and link, you can create cards to hold additional information

o you can cut and paste to these cards.

o the content is searchable.

The Analysts Notebook is also described on page 76.

Cornell Note Taking System.

Sometimes, information gathering entails attending lectures or

participating in debriefings. If this is the case, the Cornell Note

Taking system provides an easy-to-use, organised, way to

record information.

Note Taking Area: Record the presentation/discussion

as fully and as meaningfully as possible. Use

abbreviations, diagrams and examples.

Cue Column: Soon afterward, sum up the detailed notes

by writing keywords and phrases in the Cue Column.

These concise jottings are clues or “cues” for Reciting,

Reviewing, and Reflecting.

Summaries: Sum up each page of your notes in a

sentence or two. Summarizing clarifies meanings and

relationships, reinforces continuity, and strengthens

memory.

Classification Marking

Alias

Address Postal

BIO Info

DOB : 1972/08/23

Place : Montréal , Quebec

Alias

Address Postal

BIO Info

DOB : 1972/08/23

Place : Montréal , Quebec

C

u

e

s

Note

taking

area

Summaries

C

u

e

s

Note

taking

area

Summaries

of 113

In accordance with the Policy on Canadian Government Security (PGS), security classifications

should appear, in capital letters, at the top and bottom of classified documents. Subsequent

dissemination control markings may or may not be capitalised.

In accordance with DND Security Instructions – Ch.27 and the CFINTCOM Intelligence

Production Publication Standard, every title, paragraph, bullet, graph, insert, map, etc. must be

marked as appropriate for that specific portion of the product. And, a document (or a portion

thereof) is to be graded according to its own content (that is the highest element of content that it

contains), and not because of its relationship or reference to another document (or portion

thereof). The consequences of compromise (see the table on the next page) should be used to

help guide the determination classifications, especially when source material is not being directly

referred to (in which case the classification of the source material is to be cited).

While intelligence analysis generally proceeds from an examination and evaluation of

information from classified sources, this, in itself, does not mean that every assessment is

necessarily classified. Similarly, the title of an analytic piece, even though the analysis is highly

classified, needs to be classified based on its own content, and not on its relationship to the

analysis.

There is, however, the distinct possibility that compromise of a compilation of a number of lower

classified elements (paragraphs, titles, pictures, etc.) may increase the degree of injury to the

national interest, because of the compilation; requiring the use of a higher classification. The

complications that this anomaly may produce can be avoided by ensuring that at least one of the

paragraphs (or graphs, or bullets, or inserts, etc.) of the document contains sufficient material to

be of the higher classification in its own right (that is – based on its content). Then, the

document can reflect the classification of the highest element that it contains without any

difficulties.

The listing of product titles emanating from a specific group of analysts during a specific time

period is an example of this compilation effect. Even though all the titles may all be

unclassified, their collocation and identification as emanating from a specific analytic team

during a specific time period would certainly merit protection at the confidential or secret level

as appropriate. In this case, the sentence introducing the list would be classified at the higher

level, while the titles themselves would remain unclassified.

of 113

Classifications.

The actual number of

security classifications in

Canada is quite limited.

As shown in the table,

Confidential, Secret, and

Top Secret are only to be

used when dealing with

matters of National

interest. Protected A, B,

and C, on the other hand are reserved for other interests such as protecting personal, business or

other information.

The following table details some of the likely consequences if Top Secret, Secret, or Confidential

material were to be compromised. The examples are not exhaustive.

Consequences of Compromise

Top Secret

Threat to the stability of Canada or friendly nations.

Loss of life.

Exceptionally grave damage to the effectiveness or security of

Canadian and Allied forces.

Exceptionally grave damage to relations with friendly

governments.

Exceptionally grave damage to the effectiveness of extremely

valuable intelligence operations.

Severe long-term damage to the Canadian economy.

Secret

Increased international tension.

Serious damage to international relations.

Serious damage to the operational effectiveness of the Canadian

Forces

Serious damage to valuable intelligence operations.

Significant threats to the national critical infrastructure.

Serious damage to civil order.

Confidential

Damage to Canada’s diplomatic relations.

Damage to the operational effectiveness of the Canadian Forces.

Damage in the short term to economic interests.

Damage to the effectiveness of intelligence operations.

Level Degree of injury to the National Interest

Degree of injury to Other Interests

Top Secret Exceptionally Grave

Secret Grave

Confidential Limited

Protected C Exceptionally Grave

Protected B Grave

Protected A Limited

of 113

Control System Marking.14

Control systems are in place within Canada and the 5-eyes nations (Australia, Canada, Great

Britain, New Zealand and USA) to give additional protection to classified information derived

from or concerning sensitive sources, methods, or techniques. The control systems in use within

Canada are:

COMINT – relating to signals intelligence, and

TALENT KEYHOLE – relating to satellite reconnaissance systems and products.

Sub-Control Systems.

Material from or referring to especially sensitive sources and methods may be further

compartmentalised and disseminated to a limited number of recipients on a strict need-to-know

basis. Sub-control system markings include:

GAMMA – provides additional protection for very sensitive COMINT reports, and

ECI (Exceptionally Controlled information) – provides additional protection for very

sensitive SIGINT operations.

Dissemination Control Marking.

Dissemination control markings are used to limit the distribution of material to specific

individuals, groups or nationalities. A dissemination control marking can take any form,

provided it is understood by the reader (for example Task Force Kilo Eyes Only). Dissemination

control markings include:

FVEY – a designation for 5-eyes,

ORCON – a US marking to indicate that dissemination beyond listed addressees is

subject to approval of the originator, and

CEO – Canadian Eyes Only.

Audit Trail

Maintaining an accurate record of the location, author, classification, and caveats restricting the

use of information gathered for the purpose of intelligence analysis can be a daunting task. It is,

however, a vital element in the process and mandatory in many instances, especially when

judicial or inquiry processes are plausible.

Summary

By being resourceful and systematic when collecting information the analyst is in a better

position to make accurate, insightful assessments that will meet the client’s needs.

By documenting sources and noting caveats on usage, the analyst will be better able to recall,

and utilise the material that is found.

14

Developed from the CSE Publication The SIGINT Classification System, OPS-5-15, 22 Aug 2007.

of 113

Critically Evaluate the Quality of all Information

Just because a large amount of evidence, properly documented and caveated, has been collected

does not mean that it is all usable. Indeed, much of the data that can be accumulated on any

topic can be wrong, out of context, or deceitful.

Data Triage

When faced with a large

amount of data, and a short

time frame, analysts may

want to do a form of triage

where each report or element

of data is initially judged

relevant or not and then

credible, not credible, or

possibly credible. Having done this classification, the analyst can then deal with the material

that is most likely to contain the needed material.

Data Diagnostics

Diagnosis of the data available can often reveal shortcomings

that, once recognised, can be militated against. To this end,

the analyst should consider the provider of the information,

the quality of the information itself, the relevance of the

information to the problem at hand, and the possibility of

denial or deception.

Provider Diagnostic.15

An evaluation of each provider of information to

determine reliability and credibility will give insight into

the strength and weakness of the material being

provided. Irrespective of the “INT” providing the data,

the analyst should consider, at a minimum, the access

and motivation of both the agency itself, and that of the

agency employee, processor or provider. Analysts can

systematically consider the following questions when

evaluating providers:

15

This section is developed from the DIA Analytic Methodologies, A Tradecraft Primer: Basic Structured Analytic

Techniques, First Edition, Mar 2008, p 11-13, augmented by material from a Denial and Deception Analysis

Workshop presented 8-9 Apr 2009 by members of the US Foreign Denial and Deception Committee.

Relevant

Not Relevant

Credible Not Credible Possibly Credible

Credible Not Credible Possibly Credible

Data Diagnostics

Provider

Information

Relevance

Denial and Deception

Provider Diagnostic Overview

Agency (HUMINT, SIGINT,

COMINT, OSINT, etc.)

Handler (Agent handler,

editor, translator, processor,

interpreter, etc.)

Access

Motivation

of 113

Human Intelligence (HUMINT)

What agency published the report and what is the mandate/political leaning of that

organisation?

Has this agency produced useful material in the past?

Who wrote the report, and what role do they fill in the organisation?

Who controls the source of the information?

How was the source acquired (defector, émigré, agent, walk-in, recruited, etc.)?

How much was paid for the information?

If the report comes from another country – how good is that country at handling

contacts or running agents?

What is the agency’s evaluation of the source within the report?

What reliability does the agency producing the report attach to the information?

Has the reporting officer interjected his or her opinions or assessments?

Is it likely that the handler interpret correctly what the source actually meant (i.e.

were there language, social or expertise barriers)?

Imagery Intelligence (IMINT)

What agency provided the image and what is the mandate/political leaning of that

organisation?


Who wrote the accompanying assessment, and what role do they fill in the

organisation?

Who controls the source of the imagery?

How was the image acquired (satellite, hand held, etc.)?

What does the target know about our imagery sources and capabilities?

Is the collection strategy really able to answer the question being posed:

o Is the frequency of coverage sufficient?

o Is the coverage taking place at the right time of day/year?

o Are the right sensors being used?

What imaging systems (including foreign and commercial) is the target likely

sensitive to and/or guard against?

Communications Intelligence (COMINT)


organisation?


Who wrote the report, and what role do they fill in the organisation?

Who controls the source of the information?

What type of communications circuit is the information coming from (military,

civilian, satellite, air-breathing, etc.)?

Is the circuit enciphered or plain text; was the collection routine or special?

Who translated the conversation, and what organisation do they belong to?

What is the language or transcription proficiency of the translator?

Do the translators understand slang or technical terms associated with the topic?

What does the target know about collection capabilities?

of 113

Open-Source Intelligence (OSINT)


organisation?


Who produced the material, and what role do they fill in the organisation?

If based on a foreign language: Who translated the report and what organisation do

they belong to? What is their ability? Do they understand slang? Are they

experienced with this type of problem?

Information Diagnostic.

Having considered the impact of the provider, the analyst should then begin to look at the

information itself. This can be done from two aspects, the quality of the information and the

logic of the information.

Quality of Information.16

Examining the quality of the information independent of the source of the information is

important as critical information can occasionally be found in reports from sources judged to

have low access and a poor record (and vice versa). At a minimum, analysts should ask

themselves the following questions when evaluating the quality of information:

Is the information first-, second-, or third-hand?

Is there information from a separate INT (or source) that corroborates this report?

Is the information consistent or inconsistent with previous information?

Is this a complete transcript (verbatim) or a processed (analysed) summary of the

material?

o Is the report a snippet of a much larger report, conversation, etc?

What was the frequency of collection?

Have there been any recent changes in the frequency of collection?

What was the duration of the collection?

Logic of the Information.

Examining the logic of the information is also very useful and the Paul and Elder template – on

the following page – can be extremely helpful in this regard. While the template is geared

toward analysing the logic of an “article,” the same process can be applied to reporting of almost

any genre.

16


Techniques, First Edition, Mar 2008, p 15-16.

of 113

Template for Analysing the Logic of an Article17

1. The main purpose of this article is ________________.

(State as accurately as possible the author’s purpose for writing the article.)

2. The key question that the author is addressing is ______________.

(Figure out the key question in the mind of the author when s/he wrote the article.)

3. The most important information in this article is ________________.

(Figure out the facts, experiences, data the author is using to support her/his conclusions.)

4. The main inferences/conclusions in this article are ___________________.

(Identify the key conclusions the author comes to and presents in the article.)

5. The key concept(s) we need to understand in this article are ______________. By these

concepts the author means ___________________.

(Figure out the most important ideas you would have to understand in order to understand the

author’s line of reasoning.)

6. The main assumption(s) underlying the author’s thinking is (are) _______________.

(Figure out what the author is taking for granted [that might be questioned].)

7a. If we take this line of reasoning seriously, the implications are ________________.

(What consequences are likely to follow if people take the author’s line of reasoning seriously?)

7b. If we fail to take this line of reasoning seriously, the implications are ___________.

(What consequences are likely to follow if people ignore the author’s reasoning?)

8. The main point(s) of view presented in this article is (are) ___________________.

(What is the author looking at, and how is s/he seeing it?)

Relevance Check.

This check, while potentially time consuming, may help ensure that the analyst does not rely on

information that is not relevant to the central issues being assessed. A simplistic approach to

determining relevance is to examine the salience18

of the material. In this regard, the following

questions should be asked:

Does this relate (economically, socially, politically, or militarily, etc.) to the main

intelligence problem?

17

Copied from The Miniature Guide to Critical Thinking Concepts and Tools, Paul and Elder, p.13 18


Techniques, First Edition, Mar 2008, p 17-18.

of 113

Does this relate to subordinate issues associated with the main intelligence problem?

Does this make sense with what we know?

Does this make sense with what we think?

Does this beg further questions or highlight changes that need to be addressed

analytically?

Is this consistent with previous information? If not, what caused the change?

Beyond salience, however, relevance can be examined through the lenses of the biases that often

prompt analysts to give credence to material inappropriately. Some of the cognitive pitfalls19

that need to be considered in this regard are:

The vividness criterion – information that is vivid, concrete, and personal has a

greater impact on our thinking than pallid, abstract information that may actually

have substantially greater value as evidence.

The oversensitivity to consistency – in one sense, consistency is clearly an

appropriate guideline for evaluating evidence, but under some circumstances,

consistency can be deceptive. Information may be consistent only because it is highly

correlated or redundant, or because it is drawn from a very small or biased sample.

The bias favouring centralized direction – analysts often overestimate the extent to

which other countries [or groups, or individuals] are pursuing coherent, rational, goal-

maximizing policies, because this makes for more coherent, logical, rational

explanations.

The similarity of cause and effect – heavy things make heavy noises; dainty things

move daintily, etc. is generally true when dealing with physical properties, but there

is little reason for analysts to assume that economic events have primarily economic

causes, that big events have important consequences, or that little events cannot affect

the course of history.

Internal vs. external causes of behaviour – analysts are often inclined to infer that the

behaviour of others is caused by broad personal qualities or dispositions (attitudes,

beliefs, and personality), and expect that these qualities will determine the actor’s

behaviour under other circumstances. Often, not enough weight is assigned to

external causes (incentives and constraints, role requirements, social pressures, etc.).

Illusory correlation – judgements about correlation are fundamental to all intelligence

analysis. For example, assumptions that worsening economic conditions lead to

increased political support for an opposition party, that domestic problems may lead

to foreign adventurism, that …. But, an illusory correlation occurs when people

perceive a relationship that does not in fact exist.

19

The definitions in this section are drawn/quoted from Psychology of Intelligence Analysis, Heuer, 2003, ch 10-11.

of 113

Denial and Deception.

An important aspect of evaluating data – whether it comes from HUMINT, SIGINT, Imagery,

MASINT, etc., or you found it on the Internet – is to determine if denial (restricting access to

information) or deception (steering observers away from information) are being used.

An important question for analysts is:

What does the target think our collection/data gathering capabilities are?

To identify and counter denial and deception, analysts should:20

Identify and consider the opponent’s best options, even if there is no current evidence

they are pursuing them.

Beware of obvious, neat patterns that point to one option.

Collect intelligence through as many different and reliable means as possible.

Avoid giving undue importance to a limited number of data points that appear to be

consistent.

Keep an open mind and envision a range of possibilities.

If the possibility of deception is a concern, key reporting can be assessed based on five sets of

criteria:21

Does the

potential deceiver

have Motive,

Opportunity, and

Means (MOM) to

deceive?

Would this

deception be

consistent with

Past Opposition

Practices (POP)?

Do we have

cause for concern

regarding the

Manipulability of

Sources

(MOSES)?

What can we learn from our Evaluation of Evidence (EVE)?

20

These points come from the GFF An Intelligence Analysis Primer: Six Steps to Better Intelligence Analysis, Mar

2008, p 20-21. 21

From the CIA Tradecraft Review, A Tradecraft Primer: Structured Analytic Techniques for Improving

Intelligence Analysis, Volume 2 Number 2, Jun 2005, p 14.

Motive, Opportunity and Means

(MOM)

Manipulation of Sources (MOSES)

Motive – What are the deceiver’s goals?

Channels – What means are available?

Risks – What are the risks of discovery?

Costs – Can deception be accomplished?

Feedback – Can deceiver monitor its use?

Is the source reliable?

Does the source have access?

How good are the source’s bona fides?

Is the source vulnerable to control or manipulation by the target?

Past Opposition Practice (POP) Evaluation of Evidence (EVE)

Does the deceiver have a history of

deception?

Does the deception fit past patterns?

If not, are there other historical precedents?

If not, are there changed circumstances that would explain this form of deception?

How accurate is the source’s

reporting?

Is the whole chain of evidence available?

Does evidence from one source conflict with others?

Do other sources of information

provide corroborating evidence?

Is the absence of evidence unusual?

of 113

However, it is also possible that the

source is honestly reporting that which

he/she believes to be true. In that case,

analysts need to ask:

Is the Source of the

information simply Naïve

(SON)?

Analysts should be most concerned

about the possibility of deception in the

following situations:22

The analysis hinges on a single key piece of information;

The potential deceiver has a history of being deceptive;

Data is received at a critical time when those involved have a great deal to gain/lose.

Accepting the new information would require the decision maker to expend or divert

significant resources;

Accepting the new information would cause the analyst to alter a key assumption or

key judgement;

The adversary or the competitor could track its adversary’s behaviour and decision-

making process through an established feedback channel.

Data Quality Diagnostic23

A Data Quality Diagnostic is one way to make analytical arguments or intelligence gaps more

apparent and is useful because it prompts a focus on four areas, specifically, the provider of the

evidence, the information provided, its relevance, and the possibility the evidence is part of a

denial and deception effort. Although a quality diagnostic can not ensure the accuracy of

analytic judgements, its use usually enhances the credibility and usability of intelligence

assessments.

When you conduct a diagnostic review of evidence, use the matrix below to compile your

findings. Rate your confidence in the evidence as High, Medium, Low, or Questionable. For

Denial and Deception, rate the likelihood of deception occurring.

22

Copied from Critical Thinking for Strategic Intelligence, Pherson and Pherson, 2013, p 105. 23

Developed from a USEUCOM CTSA Course Diagnostic Exercise, Mar 2008

Source Naivety (SON)

Is it just that the well meaning source is naïve?

Might the source be attempting to influence operational

plans or policy to advance a personal agenda?

Is the source evangelically fervent about the subject under

discussion?

Does the source subscribe to “non-mainstream” beliefs?

Might the source simply be trying to please you or the

interlocutor?

3.

2.

1.

Deceptio

n

Rele

va

nt

Da

ta

Pro

vid

er

Overa

ll Ratin

g

Diagnostics

Data Quality Diagnostic

of 113

Propaganda

Propaganda is simply another form of denial and deception.

How to identify propaganda - Red Flags24

The Publication

1. Known for extremist views.

The Source

2. An obscure “institute” or academic and likely a mouthpiece for a special interest group.

Where do they get their funding? Who are the directors? What are their backgrounds?

The Content

3. Strong emotional aspect

a) Ominous, stirring or patriotic music and images

b) Associates a person, event or idea with something hated or feared, e.g. Nazis

c) Use of slogans, e.g. "blood for oil" , "cut and run" , “united we stand”

d) Use of virtue words, e.g. Peace, happiness, security, wise leadership, freedom, liberty …

4. Poor reasoning

a) Illogical (or non-intuitive) relationships between concepts

b) Sweeping conclusions from mere anecdotal evidence

c) Issue framed to favour one point of view

d) Irrelevant or questionable data

e) Vague, undefined terms

5. Misrepresentation - false or missing information (half of the story)

6. Oversimplification

a) Simple answers to complex social and political questions

b) Blame assigned to an individual or group (i.e. scapegoating).

c) Misleading stereotypes or labels.

d) Blanket statements

7. Persuasive Aim

a) cites or associates prominent figures to a position idea, argument or action.

b) repeats ideas until they are accepted as truth

c) presents ideas as the view of the majority (so get on the bandwagon)

d) opposition (to author’s premise) would be unpatriotic, undemocratic, inhumane

24


of 113

Checklists for Evaluating Information

HUMINT Reporting

Evaluation Status

o New,

o Developing,

o Established

Reporting Record

o On trial,

o No reason to

doubt,

o Partially

corroborated,

o Reliable

Access

o Indirect,

o Direct,

o Quoting,

o Opportunistic,

o Occasional,

o Regular

Four Step - Web Page Evaluation Checklist.25

Next page.

25


Alpha-Numeric Source-Data Rating

Reliability of Source Credibility of the

Information

A Completely reliable 1 Confirmed by other sources

B Usually reliable 2 Probably true

C Fairly reliable 3 Possibly true

D Not usually reliable 4 Doubtful

E Unreliable 5 Improbable

F Reliability cannot be

judged 6 Truth cannot be judged

of 113

Name of Website: Assessed Reliability

Low 1- 2- 3 -4- 5 High URL / Address: http://

1. Type Advocacy Business Info/Ref News Personal Entertainment Meta-tags – Who are they trying to attract to their

website (view / source)?

Older Versions – How did the site evolve?

archive.org)?

2. Content Accuracy - errors of fact or logic

- misspellings, poor grammar

- incorrect dates

Authority - author unqualified, uncited

- poor reputation

- sources undocumented

Objectivity - any blatant bias (terms, etc)?

- persuasive aim?

- single or multiple POV?

- any sponsors or advertising?

Currency

- When was it last updated?

- Any dead links?

Coverage - omissions?

3. Owner / Author: Full legal company name:

-Check copyright and privacy statements.

Who registered the domain?

Who incorporated the company? Officers / directors?

4. Affiliations and Associations? Who do they link to?

Shared premises? Tip: Google phone numbers and

addresses

Who links to them?

- Nature of association

- Affect on credibility

What do others say about them? Google names.

Consistent with similar sites?

Date Completed

Analyst / Investigator

Four Step

Web Page Evaluation Checklist

Security Classification

http://www.archive.org/

http://www.sitemapdoc.com/

http://www.networksolutions.com/whois/index.jhtml

http://www.google.com/

http://www.linkpopularity.com/

http://www.google.com/

of 113

Develop Multiple Hypotheses / Explanations

Formulating Hypothesis

Generating hypotheses26

is at the heart of intelligence analysis, and is one of the main ways in

which analysts try to provide meaning for their data. Analysts should always try to generate

several explanatory hypotheses, to make sure that they have covered a wide range of

possibilities. In intelligence analysis, as in science, hypotheses are preliminary explanations, put

forward for examination and testing, and not a final judgement. This is one of the aspects of

effective analysis in which creativity and imagination are most important.

Black Swan Rule.

No matter how many white swans one finds to prove that all swans are white, it only takes one

black swan to disprove this hypothesis. Disproving a hypothesis is far more emphatic and valid

than trying to prove it.

Value Added.27

Generating multiple hypotheses at the start of a project helps analysts avoid common analytic

pitfalls such as:

Coming to premature closure;

Being overly influenced by first impressions;

Selecting the first answer that appears “good enough” (satisficing);

Focusing on a narrow range of alternatives representing marginal, not radical, change;

Opting for what elicits the most agreement or is desired by the boss;

Selecting the alternative that avoids a previous error or replicates a past success.

Three Basic Approaches.

There are, in the broadest sense, three basic approaches28

to developing hypotheses or

explanations that can be used by intelligence analysts.

The first is Situation Specific. In this case, the issue is seen as unique and the

information/intelligence gathered is dealt with concretely, not as generalisation. As the situation

develops, a single scenario is created that hangs together as a plausible narrative. Newer

information is then compared with the original narrative for consistency, and over time the

picture is increasingly fleshed out until a convincing storyline has been built. This approach, on

its own, risks being dismissive of information that does not fit the evolving picture, possibly

over-emphasising the value of confirmatory information.

26

Quoted from the An Intelligence Analysis Primer: Six Steps to Better Intelligence Analysis, Mar 2008, p.25 27

Quoted from the Handbook of Analytic Tools & Techniques, Pherson, 2008, p.11 28

The three approaches to hypothesis generation is developed from the DIS Principles of Defence Intelligence

Analysis 2nd

Edition - Summer 2006 p 38-39

of 113

The second approach is Theory Based, though the theory is often simply a set of commonly

accepted assumptions. While the Situation Specific approach attempts to extract a plausible

narrative from existing/emerging data, the Theory Based approach starts with a narrative and

attempts to fit the data to it. Problematically, any strongly held theory (either science-based or

assumption-based) will tend to resist the proper consideration of divergent information.

The third approach is that of Comparison. Here, the evidence at hand is compared to other

well-understood situations or circumstances. While this is generally useful in the technical

intelligence realm where new data is compared to known systems and capabilities, it becomes

problematic when truly comparable analogies are difficult to identify.

Note that these three approaches are not mutually exclusive.

Types of Hypotheses.

Several types of hypotheses29

can be generated, such as:

Narrow but Deep – These hypotheses will look at the particulars of a specific case, such

as a country or organisation, and attempt to discover causes of a particular situation,

identify key drivers or factors, and speculate on ways the situation may develop. This

approach is most useful when the analyst has substantial data.

Wide but Shallow – In contrast, these hypotheses will consider similar situations in a

number of similar cases. This approach is likely to be especially helpful if the analyst has

only limited data on a particular situation.

Interdisciplinary – Another way to generate hypotheses is to look at the problems from

a different perspective. If the analyst has been thinking about the issue from a political or

military point of view, considering it from the perspective of economics, sociology or

some other discipline may help to provide new insights and possible explanations.

A Good Hypothesis

Is written as a statement, not as a question.

Is testable and falsifiable.

Predicts the anticipated results clearly.

Hypotheses (and Scenario) Generation

The generation of hypotheses or scenarios can be facilitated through the use of the structured

analytic techniques contained in the Develop and Generate Scenarios groupings.

Structured Brainstorming. (See page 62)

Outside-In Thinking. (See page 65)

Hypothesis Generator. (See page 66)

Hypothesis Review. (See page 67)

Quadrant Hypothesis Generation. (See page 82) and Alternatives Futures. (See page 83)

29

Developed from the GFF An Intelligence Analysis Primer: Six Steps to Better Intelligence Analysis, Mar 2008,

p.25

of 113

Challenge Assumptions, Mindsets and Biases

The process of perception links people to their environment and is critical to accurate

understanding of the world about us. Accurate intelligence analysis obviously requires accurate

perception. Yet research into human perception demonstrates that the process is beset by many

pitfalls. Moreover, the circumstances under which intelligence analysis is conducted are

precisely the circumstances in which accurate perception tends to be most difficult.30

The limitation of the human mind will impair or cloud the ability of an analyst to assimilate,

interpret, and understand information. Knowing these limitations and guarding against these

tendencies may, however, help analysts produce intelligence that is more complete and objective.

Assumptions

Implication of Assumptions.

As is shown in the box, assumptions are the

fog through which information in interpreted

to arrive at inferences or conclusions. An

assumption is an idea that is taken for granted,

it may be explicit (clearly stated) or implicit

(not stated at all) – IT MAY BE

INCORRECT! It will often take the guise of

common knowledge.

Identifying assumptions can be one of the most difficult challenges that an analyst will face, but

if an influential assumption is incorrect, the likelihood that the resulting inference or conclusion

is correct drops precipitously.

(Information, in this context, is made up of evidence, indicators, factors and/or proof, [Stuff

That Matters] which are all different ways of describing observable phenomena that have

relevance to an analytical problem.)

Key Assumptions Check.31

One of the best ways of mitigating the influence of assumptions is the Key Assumption Check

analytic technique (see page 70).

The goal of the key assumptions check is not to undermine or abandon key assumptions; rather,

it is to make them explicit and identify what information or developments would demand

rethinking them.

30

Quoted from Psychology of Intelligence Analysis, Heuer, 2003, p5 31

From the CIA Tradecraft Review, A Tradecraft Primer: Structured Analytic Techniques for Improving

Intelligence Analysis, Volume 2 Number 2, Jun 2005, pp 7-9

Proof

Evidenc

e

on

Information

The Reasoning Process

Assumptions

Indication

s

Factor

s

Assessment

Factors

Evidence

Inference

Conclusion

Information

The ReasoningProcess

Assumptions

Indications

Proof

Appreciation

Proof

Evidenc

e

on

Information


Assumptions

Indication

s

Factor

s

Assessment

Factors

Evidence

Inference

Conclusion

Information

The ReasoningProcess

Assumptions

Indications

Proof

Appreciation

of 113

Mirror-Imaging

(Ethnocentricity)

This is the inability to see the

world through the eyes of a

different national or ethnic group

or the inability to put aside one’s

own cultural attitudes and

imagine the world from the

perspective of those belonging to

a different group. An ethnocentric

perspective is especially

dangerous in the intelligence

context because it can distort

important aspects of strategic

thinking, especially where

problems of perception and

prediction are involved.

Checking for key assumptions requires analysts to consider how their analysis depends on the

validity of certain premises, which they do not routinely question or believe to be in doubt. A

four step process will help analysts:

Step 1: Review what the current analytic line on this issue appears to be; write it down

for all to see.

Step 2: Articulate all the premises, both stated and unstated in finished intelligence,

which are accepted as true for this analytic line to be valid.

Step 3: Challenge each assumption, asking the questions of the Key Assumptions Check

analytic technique. Explore why it “must” be true and whether it remains valid under all

conditions.

Step 4: Refine the list of key assumptions to contain only those that “must be true” to

sustain your analytic line; consider under what conditions or in the face of what

information these assumptions might not hold.

Mindsets and Biases

A substantial body of research in cognitive psychology and decision making is based on the

premise that … cognitive limitations cause people to employ various simplifying strategies and

rules of thumb to ease the burden of mentally processing information to make judgements and

decisions. These simple rules of thumb are often useful in helping us deal with complexity and

ambiguity. Under many circumstances, however, they lead to predictably faulty judgements

known as cognitive biases. Cognitive biases are mental errors caused by our simplified

information processing strategies. It is important to distinguish cognitive biases from other forms

of bias, such as cultural bias, organizational bias, or bias that results from one’s own self-interest.

In other words, a cognitive bias does not result from any emotional or intellectual predisposition

toward a certain judgement, but rather from subconscious mental procedures for processing

information. A cognitive bias is a mental error that is

consistent and predictable.32

Implication of Mindsets.

Mindsets33

are mental models that people develop over time,

based on factors such as education, family environment,

professional experience and travel. Mindsets are easy to form

but difficult to change. Data that does not fit into a given

mindset is often dismissed or devalued. Examples of mindsets

are:

Self-delusion – ambiguous information is interpreted

in favour of what is already believed.

Mirror Imaging – people assume that others, even in a

different culture, would deal with a situation more or

less the same way in which they would.

Groupthink – a small group, with strong leadership,

32

Quoted from Psychology of Intelligence Analysis, Heuer, 2003, p110 33

From the GFF An Intelligence Analysis Primer: Six Steps to Better Intelligence Analysis,, Mar 2008, p 23

of 113

tries to reach a decision under pressure, but it discounts alternatives (and therefore is

more likely to make the wrong decision) in the interest of maintaining the solidarity of

the group.

Paradox of Expertise – it is exactly those who have worked intently on an issue for a

long time who are least able to detect potential change; moreover, their experience helps

to make these experts overconfident about the accuracy of their judgements.

Implication of Biases.

Biases34

arise from the structure of the mind, and human beings are all born with mental

frameworks and dispositions that form the basis for eventual biases. Examples include:

Hindsight Bias – looking back on an event, once the outcome is known, and wrongly

believing that events were clearer in nature and easier to predict than they actually were.

Availability Bias – the mind gives greater weight to data that is vivid, concrete, and

personal than to more abstract information such as statistics.

Pattern Bias – the mind attributes a pattern or a controlling force in situations when data

are actually random or coincidental.

Confirmation Bias – we see what we expect to see, and more data is needed to recognise

and understand what has not been expected.

Biases were examined earlier when considering the relevance of evidence. See page 36 for an

explanation of the biases of:

Vividness

Oversensitivity to consistency

Tendency to favour centralised

direction

Similarity of Cause and Effect

Internal vs external causes

Illusory correlation

Countering Mindsets and Biases35

Mindsets and biases are often mitigated by simply implicating an individual or individuals into

the analytic process that do not share the mindset or biases of the original analyst or group. The

Review Pre-Publication and Use Work Groups families of structured analytic techniques are

quite good at this; particularly:

Devil’s Advocacy. (See page 61)

Red Cell. (See page 60)

Team A/Team B (See page 58)

Dialectic Inquiry. (See page 94)

Red Team. (See page 59)

34

From the GFF An Intelligence Analysis Primer: Six Steps to Better Intelligence Analysis,, Mar 2008, p 23 35

Adapted from the GFF An Intelligence Analysis Primer: Six Steps to Better Intelligence Analysis,, Mar 2008, p

24

Self Test:

How long have I held my current views?

Am I dismissive of alternate views?

Do I only read sources that reinforce my

views?

Do I “cherry pick” evidence that fits with my

views while discounting information that

doesn’t?

of 113

Prevention of Fixed Mindsets 36

Much work is being done on the cognitive and neuroscience aspects of fixed mindsets and many

researchers are concluding that factors such as Western child-rearing practices, chronic daily

work routines, and the standard American diet, are significant contributing factors. While there

is little we can do about our child-rearing practices, there are steps we can take to avoid the

development of fixed mindsets.

Research is showing that analyst managers can help their analysts avoid the debilitating

effects of fixed neurological structures caused by specialised long-term work routines by

implementing work programs that are analogous to a cross-training physical fitness

regime. That is, analysts should be regularly challenged by very diverse work tasks,

forcing them to constantly use various functions and areas of their brains.

Recent groundbreaking explorations of the connection between exercise and the brain’s

performance show that even moderate exercise will supercharge mental circuits to beat

stress, sharpen thinking, enhance memory, and much more.

The brain in general and its capacity to change and grow in particular is very sensitive to

environmental influences like stress, dehydration, nutrition and diet. To this end, along

with dietary restriction, fasting, and calorie restriction, potentially helpful nutrients so far

examined include:

o Omega-3 fatty acids

o Curcumin (the major component of turmeric), green tea

o Antioxidants: Flavonoids, Vitamin E, Resveratrol

o Ketogenic diet (Coconut Oil, Medium Chain Triglycerides)

o Dietary branched chain amino acids

o Caffeine

36

Developed from a briefing given by Dr Michael Axel of the Berlin Free University to the 5-Eyes Analytic

Training Workshop on 11 Feb 2013.

of 113

Build Collaborative Networks

Canadian Security and Intelligence Community

Why Collaborate?37

Collaboration is a fundamental intelligence community concept and an essential element of

analytic tradecraft. By bringing broad-based experience to bear on complex intelligence issues,

clients can be provided with the context, perspective, and implications that these issues demand.

The best intelligence analysis is produced through collaboration.

It is almost impossible to conceive of an intelligence product that can be created solely by a

single analyst or production element. A report detailing development of a new weapons system

must go beyond the technical characteristics of the system and include: a discussion of the

military strategy, force structure, and infrastructure that is available or necessary to support it

use; the direct and indirect effect on deployed or deploying forces; and the political, diplomatic

37

Developed from the draft DIA Tradecraft Note on Collaboration, dated 27 Aug 2010.

of 113

and foreign relations impacts of the new system. A product discussing cyber threats falls short if

it does not discuss the implications to the nations, organisations, or personnel potentially affected

by the development. The argument – that the integration of all knowledge pertinent to a given

issue is not an intelligence responsibility – can only result in leaving clients to perform the

integration on their own.

Collaboration is an active and iterative process of partnering, in all phases of the intelligence

process, with others who share an interest in the issue (and indeed, some who do not).

Collaboration distinguishes intelligence analysis from other information professions such as

academic research and journalism and it includes not only other members of the intelligence

community, but outside experts as well. Well done, collaboration:

enriches finished intelligence with multidisciplinary analysis,

reduces the impact of individual bias on the analysis,

helps identify and deal with assumptions,

helps clarify uncertainties,

aids in the identification of influential factors, and

encourages the incorporation of alternative hypotheses.

Absent appropriate collaboration, products will have a higher chance of being incomplete,

confusing, or even contradictory.

Creating a Collaborative Culture38

The creation of a culture of collaboration is neither simple, nor spontaneous. Rather it is a

deliberate process requiring the commitment of both analysts and analyst managers. To this end,

analysts should:

Initiate collaborative relationships with colleagues outside their work unit, directorate,

department/agency, and country, to include other analysts, operators, policy makers,

academics, NGO, PVO, and others.

Utilise collaborative tools to overcome organisational and geographic boundaries.

Credit those who have collaborated on projects and provide appropriate feedback to

individuals outside of the community. People are more likely to collaborate again if they

are aware of their impact.

Concomitantly, analyst managers should:

Organise and lead topically focused forums creating relaxed environments for analysts

and others to meet and form the foundations for communities of interest.

Insist that analysts drafting products solicit input from all stakeholders, especially those

outside the analysts’ work unit, before putting pen to paper.

Hold analysts accountable for initiating and sustaining effective relationships with others

outside their immediate work unit during periodic performance review.

Recognise and reward analysts who lead communities of interest on their issues or

otherwise collaborate effectively with their counterparts.

Develop a process/system of handing-over contacts from one analyst to another during

personnel changes.

38

Developed from the draft DIA Tradecraft Note on Collaboration, dated 27 Aug 2010.

of 113

Team/Group Collaboration39

Teams are often used in intelligence analysis with the objective of producing better analysis than

could be done by an individual working alone. Groups or teams often have:

More resources than individuals

Greater diversity of resources

More flexibility in deploying their resources

More opportunities for collective learning

The potential for “synergy”

However, the performance of groups/teams is often subpar when compared to “nominal” groups.

Why is this the case?

Poor leadership – this is the most often cited reason.

A team is used for work that is better performed by an individual.

The task is too hard.

The team is set up in a way that stymies its potential – this may be the most common

reason.

When should a team be used for analytic work? Sometimes there is no real choice, but when

there is, consider the following:

When the desired result is a creative composition – never use a team. The analogy here

is: while it takes a full orchestra to render Tchaikovsky's famous 1812 Overture,

Tchaikovsky himself imagined and then produced the composition.

When confronted by an urgent, focused, multi-disciplinary problem – always use a team.

But when there is room to choose, what type of team is called for?

Models of Collaboration.40

Four models of collaboration worth considering are: Task

Force, Experts Group, Professional Community, and

Interested Parties. To decide which model is appropriate,

two questions need to be asked:

1. Where might the answer(s) lie? In a small or

large group?

2. What degree of coordination and control is

needed? High or Low?

These two questions will result in four combinations of

small/large and high/low.

Small Group + High Control = Task Force.

39

Adapted from a briefing to the GFF COI POI on 24 Feb 2009 by J. Richard Hackman from Harvard University. 40 Developed from an article by John Pyrik published in the Nov 2009 edition of the Intelligence Analyst Training

Newsletter.

Interested

Parties

Experts

Group

Professional

Community

Task

Force

De

gre

e o

f O

rga

nis

atio

n

Lo

w -

----

----

-H

igh

Group Size

Small/Closed --- Large/Open

Interested

Parties

Experts

Group

Professional

Community

Task

Force

Interested

Parties

Experts

Group

Professional

Community

Task

Force

De

gre

e o

f O

rga

nis

atio

n

Lo

w -

----

----

-H

igh

Group Size

Small/Closed --- Large/Open

of 113

Here, cooperation is mandatory, activities are centrally directed, and the results

are generally for the benefit of one party.

Small Group + Low Control = Experts Group

In an Experts Group, cooperation is voluntary, activities are determined

collectively, and benefits are shared equally (or at least are accessible to all

parties). Experts groups are often used to collaborate on a product that will be

published by one of the participants.

Large Group + High Control = Professional Community

The Professional Community is generally a large and diverse group with a high

degree of coordination. The process of interaction is often structured and

membership has enduring multi-party benefits.

Large Group + Low Control = Interested Parties

Here, the process of interaction is informal and ad hoc and the benefits accrue

primarily to one party. The boundaries of the group are often blurred and could

involve a potentially unlimited number of self-selected "interested parties"

associated in a loose fashion.

Developing Effective Teams.

The role of the team leader is crucial in the

development of good teams. A good team leader

will:

Decide about the kind of team that is

appropriate for the work to be done.

Get the essential and enabling conditions

in place – and keep them there.

Coach at the margins to help the team take

full advantage of its favourable

performance circumstances.

For more about designing and leading effective teams, see the book Leading Teams or visit

http://www.leadingteams.org. Alternatively, the Team Diagnostic Survey assesses the standing

of a team on the conditions that foster team effectiveness and provides a diagnostic profile of the

team’s strengths and weaknesses. For online access, go to https://research.wjh.harvard.edu/TDS.

Team Formation.41

Creating a team from a divergent group of individuals, be they from disparate sections of the

same organisation, inter-departmental or even international, can be a challenge. One way to help

ensure that all members are in agreement about the team’s main purposes and, more importantly,

are able to bring the full range of their knowledge, skill and experience to bear in accomplishing

these purposes, can be facilitated through the use of a team formation exercise. One such

process is as follows:

41

Developed from the Blue Team Launch Exercise, produced by J. Richard Hackman from Harvard University.

Developing effective teams:

Make the purpose of the team

clear – identify a specific leader.

Identify (and inform the team of)

individual member knowledge,

contacts, and expertise.

Identify additional needs – assign

a team member to attain them.

Establish clear team norms.

Make “course corrections” as

necessary.

http://www.leadingteams.org/

https://research.wjh.harvard.edu/TDS

of 113

Step One: Selection of Timekeeper and Recorder (these roles should be filled by the team

leader if there is one).

Step Two: Individual Work – have each individual identify:

o the main purpose of the team as they understand it,

o the main consequences of the team achieving its purpose, and the team not

achieving its purpose, and

o on a scale of 1 to 5, how challenging it will be for the team to achieve its purpose.

Step Three: Discussion by Pairs – have each pair identify:

o how their separate descriptions of the main purpose agree and disagree,

o the knowledge, skill, experience, perspectives, and relevant outside contacts that

each person brings to the team that could be helpful in achieving the team’s main

purpose (this should be accomplished by interviewing each other), and

o a short list (less than four items) of things they want to report to, or ask of the

remainder of the team.

Step Four: Reconvene, Report and Discuss – the recorder or team leader should facilitate

the discussion of, and summarise and record the following:

o the knowledge, skill, experience, perspectives and relevant outside contacts that

are available to the team,

o the additional expertise that is needed, and who specifically is in the best position

to obtain it from resources outside the team, and

o team norms.

Step 5: Wrap Up – the team should:

o identify anything else that needs to be addressed before work can start,

o if a team leader is not already in place, appoint one, and

o based on what has been discussed and concluded, decide on what the team should

do to get off to a good start with actual work on the task.

Consensus vs Dissent42

Common Pitfalls with Small Groups.

Research suggests that the desire for consensus is an important cause of poor group decisions.

Development of a group consensus is usually perceived as success, but, in reality, it is often

indicative of failure. Premature consensus is one of the more common causes of sub-optimal

group performance. It leads to failure to identify or seriously consider alternatives, failure to

examine the negative aspects of the preferred position, and failure to consider the consequences

that might follow if the preferred position is wrong.

42

Developed and quoted from Richards J. Heuer, Jr. And Randolph H. Pherson’s Structured Analytic Techniques

for Intelligence Analysts, pp 300-303 & pp 257-260, copyright 2011 by CQ Press.

of 113

Other problems that are less obvious, but no less significant have been documented extensively

by academic researchers. It frequently happens that some reasonable satisfactory solution is

proposed that all members can agree with, and the discussion is ended without further search to

see if there may be a better answer. Such a decision often falls short of the optimum that might

be achieved with further inquiry.

A phenomenon known as group polarization leads, in certain predictable circumstances, to a

group decision that is more extreme than the average group member’s view prior to the

discussion. Social loafing is the phenomenon that people working in a group will often expend

less effort than if they were working to accomplish the same task on their own. In any of these

situations, the result is often an inferior product that suffers from a lack of analytic rigour.

Benefitting from Diversity.

Improvement of group performance requires an understanding of these problems and a

conscientious effort to avoid or mitigate them. The literature on small-group performance is

virtually unanimous in emphasizing that groups make better decisions when their members bring

to the table a diverse set of ideas, opinions, and perspectives.

Laboratory experiments have also shown that even a single dissenting opinion, all by itself,

makes a group’s decisions more nuanced and its decision-making process more rigorous. The

research also shows that the benefits from dissenting opinions occur regardless of whether or not

the dissenter is correct. The dissent stimulates a reappraisal of the situation and identification of

options that otherwise would have gone undetected.

To be effective, however, dissent must be genuine, not generated artificially as in some

application of the Devil’s Advocacy technique. It should also be reasonable. If the person

voicing dissenting views is known to the group as a habitual contrarian or maverick, his or her

comments run the risk of being dismissed by the group regardless of merit.

Advocacy vs Objective Inquiry.

Diversity of opinion is, of course, a double-edged sword as it can become a source of conflict

which degrades group effectiveness. Analysts must engage in inquiry, not advocacy, and they

must be critical of ideas but not critical of people.

In a task-oriented team environment or where Devil’s Advocacy, Red Cell, or other such

techniques are in use, analysts adopt the role of advocate which often leads to emotional conflict

and reduced team effectiveness. Advocates tend to examine evidence in a biased manner,

accepting at face value information that seems to confirm their own point of view and subjecting

any contrary evidence to highly critical evaluation. Advocacy is appropriate in the courtroom,

but it is not an appropriate method of discourse within a team, especially when power is

unequally distributed among the participants, when information is unequally distributed, and

when there are no clear rules of engagement – especially about how the final decision will be

made.

of 113

On the other hand, an effective mitigation of all of these challenges can be found in a process of

objective inquiry – for example, see the Dialectic Inquiry process outlined on page 94 – which

can lead to new and creative solutions to problems, especially when it occurs in an atmosphere of

civility, collaboration, and common purpose. The table below displays the differences between

advocacy and the objective inquiry expected from a team member or colleague.

Advocacy Inquiry

Concept of decision

making A contest

Collaborative problem

solving

Purpose of discussion Persuasion and lobbying Testing and evaluation

Participants’ role Spokespeople Critical thinkers

Pattern of behaviour Strive to persuade others

Defend your position

Downplay weaknesses

Present balanced arguments

Remain open to alternatives

Accept constructive criticism

Minority views Discouraged or dismissed Cultivated and valued

Outcome Winners and losers Collective ownership

Forcing a Consensus.

Notwithstanding the insistence above that dissenting views are valuable in intelligence analysis,

there are times when a consensus simply must be achieved, or where the strongly held views of

an individual or group is becoming inimical to achieving the task at hand. In these cases, it is

important to be able to manage the confrontation that is almost inevitable. To this end,

adversarial collaboration techniques, based on the forced requirement to understand and address

the other side’s position, rather than simply dismiss it, may be of use. In the techniques that

follow, mutual understanding of the other side’s position is a bridge to productive collaboration:

Key Assumptions Check – Given the difficulty inherent in identifying assumptions, and

their taken-for-granted nature, the Key Assumptions Check (see page 70) can be a very

effective method to help understand what underlies a conflicting judgement. If a Key

Assumptions Check has not already been done, each side can undertake the process and

then share the results with the other. Discussion should then focus on the rationale for

each assumption and suggestions for how the assumptions might be either confirmed or

refuted. The discussion should focus on refuting the other side’s assumptions, rather than

on supporting one’s own.

Analysis of Competing Hypotheses – When opposing sides are dealing with a collegial

difference of opinion, with neither side firmly locked into its position, ACH (see page 74

may provide a good structured format for helping to identify and discuss the differences.

Both parties agree on a set of hypotheses and then rate each item of evidence or

assumption as consistent or inconsistent with each hypothesis. When a disagreement

arises, the difference often points to previously unrecognised assumptions or to some

interesting rationale for a different interpretation of the evidence. The use of ACH may

not result in the elimination of all the differences of opinion, but it can be a big step

of 113

toward understanding these differences and determining which might be reconcilable

through further intelligence collection or research.

Mind Mapping – This process can graphically display the logical relationship between

each element of an argument (see page 90) and, if two sides agree to work together to

create a single Mind Map, with the rationale both for and against a given conclusion.

Such a map would show where the two sides agree, and where they diverge, and why.

The visual representation of the argument makes it easier to recognise weaknesses in

opposing arguments. An alternative approach might be to focus the discussion on

alternative contrasting Mind Maps.

Mutual Understanding – When analysts disagree, the disagreement is often exacerbated

by the fact that they have a limited understanding of the other side’s position and logical

reasoning. The Mutual Understanding approach addresses this problem directly. After

an exchange of information on their positions, the two sides meet together with a

facilitator, moderator, or decision maker. Side 1 is required to explain to Side 2 its

understanding of Side 2’s position. Side 1 must do this in a manner that satisfies Side 2

that its position is appropriately represented. The roles are then reversed, and Side 2

explains its understanding of Side 1’s position. This mutual exchange is often difficult to

do without really listening to and understanding the opposing view and what it is based

upon. Once all the analysts accurately understand each side’s position, they can discuss

their differences more rationally and with less emotion.

The Nosenko Approach – This process is named for a Soviet intelligence officer who

defected to the US in 1964, and the ground rules used by the CIA’s analytic team to try to

determine the “truth” about him. After reviewing the evidence, each officer on the team

identified those items of evidence thought to be of critical importance in making a

judgement on Nosenko’s bona fides. Any item that one officer stipulated as critically

important had to be addressed by each of the other members. These ground rules can be

applied in any effort to abate a long-standing analytic controversy. The key point that

makes the rules work is the requirement that each side must directly address the issues

that are important to the other side and thereby come to understand the other’s

perspective. This process guards against the common propensity of analysts to make

their own arguments and then simply dismiss the other side’s arguments as unworthy of

consideration.

Encouraging Consensus.43

One way to achieve a consensus is through the realization of a series of little consensuses. Using

this iterative approach, the following questions should be asked and answered - in turn - by the

group:

What is the problem that needs to be addressed?

What caused the problem?

What are the alternative solutions?

What are the criteria for a good solution?

What is the best solution?

43

Daniel Lapensée, CHRA, YouTube – Comment render les réunions efficaces, 3 Jul 2011

of 113

Use Structured Analytic Techniques

Why use Structured Techniques?

The analytic tools and techniques that have been mentioned so far in this Aide Memoire and

which follow all, potentially, have three positive attributes:

First, they may instil rigor in the process of analysis by providing a checklist that can be

followed to guide the process.

Second, they may actually bring into existence a better analysis than would have been

produced without them.

Finally, they may enhance the persuasiveness of arguments and provide a useful

visualisation when presenting the assessment.

Be careful though, using the wrong technique, using faulty or incomplete data, or allowing an

inappropriate assumption or mindset to prevail will almost certainly result in poorer analysis and

wasted time.

Categories of Techniques

There are a number of ways of dividing structured analytic techniques into groups. One of these

is shown on the back of the IALP’s Best Practices for Intelligence Analysts card. In that schema,

the techniques are classified as either:

Diagnostic – these techniques are primarily aimed at making analytic arguments,

assumptions, or intelligence gaps more transparent.

Challenge, or contrarian – these techniques are intended to explicitly challenge current

thinking.

Imaginative – these are thinking techniques that aim to develop new insights, different

perspectives and/or alternative outcomes.

The structured analytic techniques in this Aide Memoire, however, have been classified

according to how they contribute to the analytic tradecraft process. To this end, nine separate

groupings have been identified as follows (see also the CMap on page 57):

Use Work Groups – these techniques are various ways of organising analytic teams in

order to tackle intelligence problems.

Develop – these techniques are intended to help analysts explore issues and come up with

factors and alternatives.

Evaluate – these techniques help analysts evaluate the usefulness (veracity, or

appropriateness) of evidence, assumptions and indicators.

Explore Networks – these two techniques use link processes to explore networks and

relationships.

Compare – these techniques rank things according to the factors or criteria.

Generate Scenarios – these techniques use factors to develop scenarios.

Understand Possibilities – these techniques use factors to explore possibilities.

Demonstrate – these techniques will help an analyst illustrate or sequence analysis in a

compelling or understandable way.

of 113

Review Pre-Publication – these techniques provide ways to examine analysis before it is

published.

The techniques are, below, shown alphabetically and by family group:

Technique Page

Alternative Futures Analysis. 83

Analysis of Competing Hypotheses 74

Bow-Tie Technique 86

Chronologies and Timelines 89

Cone of Plausibility 81

Counterfactual Reasoning 85

Cross-Impact Matrix 77

Change Analysis 80

Data Diagnostic 69

Decision Trees 92

Delphi Method 63

Devil’s Advocacy 61

Dialectic Inquiry 94

Environmental Scanning 64

Force Field Analysis 84

High Impact – Low

Probability 93

Hypothesis Generator 66

Hypothesis Review Technique 67

Indicators of Change 72

Indications Validator 73

Key Assumptions Check 70

Link Charts 76

Matrices 91

Mind Mapping 90

Outside-In Thinking 65

Pre-Mortem Assessment 95

Quadrant Hypothesis Generation

Technique 82

Red Cell 60

Red Team 59

Reframing the Question 68

Social Network Analysis 75

Structured Brainstorming 62

Structured Comparison 78

Team A – Team B 58

Weighted Rankings 79

What If? Analysis 88

Structured Analytic Techniques Map

of 113

Use Work Groups

Team A / Team B.

Purpose: To provide a self-check that confirms, refutes, or offers an alternative (competing) assessment.

Description: Team A/Team B is a competition, akin to a debate. However, unlike Devil’s Advocacy,

which simply tries to refute an argument, it permits the expression of different views. Separate teams are

assigned the same problem. They may be told to take opposite sides or they may be left to arrive at

independent conclusions.

Process: When debate is polarized between two views, the Team A/Team B approach helps clarify the

differences between the two viewpoints. When it is suspected that analysts have developed an inflexible

mindset, it can force a re-evaluation.

The process has two phases, analysis and debate.

In the Analysis Phase:

Begin by identifying the two (or

more) competing hypotheses or points

of view.

Form teams (or designate individuals) to

develop the best case that can be made for

each hypothesis.

In the Debate Phase:

Both teams present their arguments

before an independent jury.

The teams critique each other’s

assessments then respond to criticisms.

The jury or decision maker listens, questions, and decides which team has the stronger case and may also

recommend actions to be taken, e.g. further research.

Strengths: Assigning analysts to the team that fits their existing view expedites the process and makes

for strong arguments. The reverse takes more time, but forcing analysts to argue “the other side” can often

make them more aware of their own assumptions, mindsets and biases.

The process can help opposing experts see the merit in each other’s arguments.

It also allows those with opposing views the opportunity to express their views and feel that they have

been heard.

For the decision maker, this technique exposes and explains important analytic differences within the

expert community. Senior officials often learn more by weighing well-argued conflicting views than

from reading an assessment that masks substantive differences or drives analysts to the lowest common

denominator.

Weaknesses: The process requires time and resources. Competition may lead to workplace acrimony and

anchoring. Results may be skewed if one team has a political agenda.

of 113

Use Work Groups

Red Team.

Purpose: To produce assessments which represent probable viewpoints and intentions of the subjects of

the analysis.

Description: Red Teams have been used historically in planning and implementing military exercises, in

which the Red Team plays the opposing force. A Red Team is a group brought together to simulate the

intentions, planning, and sometimes the capabilities of the subject group. A Red Team can produce

predictions of possible or probable future actions by the subjects. A Red Team can also identify

vulnerabilities in our security, planning, and other aspects of our capabilities which the subject could

exploit.

Analytic Red Teams in intelligence are useful in predictive assessments and alternative analyses,

potentially at both the tactical and strategic levels. This is not the only possible use of this technique, for

example operational Red Teams which employ hacking techniques are used to actively test IT security.

Members of a Red Team need to have a rich understanding of the subjective and objective worlds of the

subjects, including: cultural norms and values, motives, operating doctrine, operational environment,

sensitivity to risk, and other relevant factors such as their subjective understanding of our intentions and

capabilities. Operational Red Teams require intelligence support to successfully imitate their subjects’

intentions and capabilities.

Strengths: A well-informed Red Team can give valuable insights into the courses of action available and

acceptable to an adversary. This can provide a balance against biases, especially mirror imaging.

Weaknesses: Without a rich understanding of the subjects, members of a Red Team may tend to fall into

mirror imaging and so produce confirmation of potentially false understandings. There can also be

significant organizational challenges to using a Red Team approach if the purpose is not widely

understood or if individuals are too invested in the positions being tested.

of 113

Use Work Groups

Red Cell.

Purpose: A Red Cell is an internal think tank created to help an organization develop broader peripheral

vision and avoid surprise.

Description: In response to 911, the CIA created a Red Cell that would think unconventionally about the

full range of relevant analytic issues. Their short “think pieces” about plausible events for which there

was currently little or no evidence were intended to provoke thought rather than to provide authoritative

assessment.

A Red Cell does not automatically take the opposite or contrarian view – it is not a challenge technique. It

merely tries to raise awareness.

The process starts with a question such as: “Could the UN in New York be attacked using garbage barges

laden with explosives?”

Any feasible question is OK – although, ideally it should be one that no one else has asked. The next step

is generating a series of events or scenario that would lead to the event occurring, i.e. “What if?” analysis.

Finally, a short but (hopefully) convincing argument is presented that summarizes the evidence and

reasoning.

A Red Cell can be set-up permanently or on an ad hoc basis to deal with specific issues (in which case it

might be referred to as a “Team A / Team B” exercise or a “Red Team” if it simulates the actions or

decision-making process of an adversary).

If successful, Red Cell reports persuade experts, analysts, and decision makers to re-examine their views

and re-task resources to prove or disprove the scenario advanced by the Red Cell.

A Red Cell needs a diverse group of people who are “out of the box” thinkers. Not afraid to speak out or

appear the fool, they should be ingenious and resourceful. Members could include psychologists,

philosophers, professors, and analysts from different units and agencies who think differently from the

ones who typically work on the account (i.e. file / issue / problem). Outside technical experts may be

included or consulted (depending on the problem).

To ensure a continual flow of fresh insights, new members can be rotated in every three months.

Management should also give permanent Red Cell units lots of latitude (two thirds of CIA Red Cell

reports are self-initiated).

Strengths: Helps to combat assumptions, mindsets, and biases.

Weaknesses: A dedicated Red Cell that constantly critiques or “shows-up” the analysis of others, can

create internal tensions. One that “colours too far outside the lines” risks being ignored or mocked. Red

Cell reports may provoke hostility, especially within conservative, hierarchical organizations. “As if we

didn’t have enough problems already,” could be one reaction. If Red Cell reports undermining

mainstream analysis are distributed externally, the agency could appear indecisive. Small organizations

may not be able to afford removing staff from their regular duties to work in a Red Cell. Some argue

creativity cannot be structured.

of 113

Use Work Groups

Devil’s Advocacy.

Purpose: An analytical technique used to test an argument by attempting to prove it flawed or

fundamentally incorrect.

Description: The original advocatus diaboli [Devil's Advocate] was, from 1587 to 1983, part of the

Roman Catholic Church's formal process of recognizing individuals as saints. As the name suggests, his

role was to take the opposing view against his opponent the advocatus Dei [God's advocate] promoting an

individual as a saint.

This competitive approach to testing a hypothesis or argument is also seen in the structured confrontations

between prosecution and defence in courts of law.

It can also be placed in the context of Hegelian dialectic, in which thesis generates antithesis, leading to a

higher synthesis.

A Devil's Advocate exercise can be effective in testing hypotheses in terms of finding any weaknesses in

logic.

This technique can be used at several stages during the development of an intelligence assessment. For

example, a Devil's Advocate could review the project plan, the working hypotheses, and/or the final draft

of the assessment.

While the typical approach to multiple competing hypotheses is based on analysts developing and testing

their own hypotheses, the Devil's Advocate technique brings in an outsider to test the hypotheses. The

individual or team playing the role of Devil's Advocate develops and presents arguments against the

hypotheses and conclusions presented in the assessment under review.

An individual or team playing the Devil's Advocate will need to study not only the hypotheses or

assessment it is testing, but must also study the supporting information.

Strengths: An effective Devil's Advocate can find flaws in a hypothesis or assessment which might

otherwise not be noticed by the authors, and a hypothesis which has survived a thorough and rigorous

testing through this process can be given more credibility.

A Devil’s Advocate also reduces “group think.”

Weaknesses: A Devil's Advocate can suffer from the same biases as the author of the hypothesis, and so

miss the same flaws.

Use of the Devil's Advocate approach can generate conflict and morale problems in an intelligence group

if the individuals involved identify too closely with the positions being argued and tested.

A weak Devil's Advocate may also create a misleading impression that all sides of an issue have been

considered, when in reality they have not. This could lead to false confidence and over-commitment.

of 113

Develop

Structured Brainstorming.44 Purpose: A form of brainstorming that generates new analytic ideas, hypotheses, or concepts through an

unconstrained individual or group process.

Description: This technique works best when an individual is willing to work as part of a group to

develop multiple ideas, hypotheses, or concepts. It can be used either at the beginning of an analytic

project to help generate the initial hypotheses or at a later stage if the initial result proves inadequate. New

information may be found that could cause the analyst to return to this technique to integrate it into the

existing hypotheses.

Process: Creative thinking works best when a trained facilitator is available to ensure the session is

fruitful. The creative thinking process actually consists of two phases: a divergent phase, where group

members create new ideas via brainstorming, and a convergent phase, where group members cluster ideas

for review, consolidation, and follow-up action.

Divergent Phase

Organize the group. Group members should come from a variety of backgrounds (cross

fertilization is important). Cognitive diversity, different points of view, and a wide range of

experience are important. Small groups tend to function better than large ones; five to seven

participants is a good target.

Focus on a specific topic or question. It should not be so broad that no solution is possible or so

narrow that creativity won’t help. Make clear to all members in advance that discussion will not

be constrained by current positions or available evidence.

Have everyone write down at least one idea before discussion starts. Use paper, white boards,

or Post-it notes to record ideas. That will allow easy clustering of ideas during the convergent

phase.

Have the group verbally generate as many ideas as possible. When a group has one or more

strong personalities, the facilitator can have the members stop all verbalization and write their

ideas down and post them where others can read them and build on any idea. Listen closely as

others talk; this will help generate ideas. Suspend judgement; do not eliminate ideas; what looks

crazy at first may become valuable later, after more thought or when new data is received.

Let the first session last for 45-60 minutes or until a noticeable decline in activity takes

place. Then take a break. Keep going for two more sessions, ending each when the activity falls

off. After the third such period, it is time to stop the divergent phase.

Convergent Phase

Group the ideas by theme, then set aside any that do not easily fit with any group. Then

through voting or other means, select the themes or outliers that deserve further attention.

After the session is over, have the individuals spend time alone to silently review the

submission and consider: Which of the alternatives are reasonable and would meet the goals?

What are the alternative’s shortcomings? What are the alternative’s benefits?

44

Developed from the DIA’s A Tradecraft Primer: Basic Structured Analytic Techniques, First Edition, Mar 2006,

pp 21-24 and Handbook of Analytic Tools & Techniques, Pherson, 2008, pp19-20

of 113

Develop

Delphi Method.

Purpose: To produce reliable consensus judgements from a large and diverse set of experts.

Description: The Delphi method is an iterative survey approach to generating consensus judgements.

The progressive series of questionnaires is a structured, anonymous debate among the participants.

The first set of questionnaires is sent out to the panel of experts who provide their individual judgements.

The next set of questionnaires includes anonymous summaries of the responses to the previous

questionnaire, the panel members are asked to consider these other responses and to revise or maintain

their own judgements accordingly. A reliable consensus of opinion usually develops in the second

iteration, but the process can be repeated further if necessary. Sometimes the survey will reveal stable

disagreement in the form of two or more factions holding incompatible opinions.

The Delphi method can be used in predictive assessments, in priority setting assessments, and in

generating expertise-based analytical tools.

A successful Delphi depends on good survey design, sound analysis of the survey results, and on

involving respondents with valid and relevant expertise.

Strengths: Studies of Delphi method have shown it to produce more reliable consensus judgements than

face-to-face meetings. A Delphi survey can also be more efficient and economical that arranging and

facilitating a meeting / workshop of the appropriate experts.

Weaknesses: Errors in survey design, survey results analysis, or in selecting the participants can lead to

invalid results.

of 113

Develop

Environmental Scanning. Purpose: Environmental Scanning provide checklists that may help an analyst explore a subject area in

broad terms.

Process: Having selected an appropriate environmental scanning process, use each of its component parts

to explore the matter under exploration.

SWOT

Strengths

Weaknesses

Opportunities

Threats

Activity Systems Model

45 See the diagram on the right.

STEMPLES46

47

Social (culture, attitudes/perceptions, education, population, health, welfare, etc.)

Technological (developments, funding, access to technology, patents/licensing, IT, etc.)

Environmental (climate, living conditions, impact of weather, etc.)

Military (capabilities, developments, doctrine, control & command, etc.)

Political (leadership, political system, policy, pressure groups, elections, etc.)

Legal (current & future legislation, regulatory processes, judicial system, etc.)

Economic (internal economy, inflation, global markets, industry, interest rates, etc.)

Security (police, reforms, private companies, terrorism, criminal networks, etc.)

Understanding Groups48

What defines this group (geography, history, purpose, etc.)?

Are there internal factions (religious, ethnic, linguistic, generational, etc.)?

How is power distributed (matriarchic, patriarchal, wealth, fear, moral authority, etc.)?

What motivates and inhibits individual conduct?

What pressures and stresses are present in the group?

How is the culture changing?

What unusual values, attitudes or beliefs are present? What are the taboos?

How are outsiders viewed?

What are the capabilities and intentions of this group?

Do members of this group think as you do?

45

Adapted from the US DIA’s “Activity Systems Model (expanded)” demonstrated by JMITC instructors at the Feb

2013 5-Eyes Analytic Training Workshop (they had derived it from Yjro Engestrom). 46

Copied from the DIS Handbook Quick Wins for Busy Analysts, 2012, p 9 47

One could also consider psychological, ethical, demographic and other factors. 48

Based on the check list developed by John Pyrik for use in the Canadian Interdepartmental Intelligence Analyst

Learning Program.

Tools

Actor (s) Task (s)

Roles RulesOrganisation

Community Governance

Context

Motivation Outcomes

Tools

Actor (s) Task (s)Actor (s) Task (s)

Roles RulesOrganisationRoles RulesOrganisation

Community GovernanceCommunity Governance

Context

MotivationMotivation OutcomesOutcomes

of 113

Develop

Outside-In Thinking.49 Purpose: Outside-In Thinking is used to identify the range of systemic forces, factors, and trends that

would have an impact on shaping an issue, allowing analysts to incorporate this broader conceptual

framework into their analysis.

Process: Once a generic description of the problem or phenomenon under study has been generated, the

analyst can use an environmental scanning process (see page 64) to trigger new ideas. The process is:

First, to list all of the factors over which the

subject may be able to exert limited influence

(such as resources, partners, methodologies,

etc.).

Then, to list all of the forces and factors that

could have an impact on the topic, but over

which the subject can exert little or no influence

(such as historical precedent, the growth of the

Internet, etc.).

Then the analyst can assess, specifically, how each of these forces and factors might have an impact on

the problem and generate new collection taskings to fill the information gaps.

The technique is most useful in the early stages of the analytic process when analysts need to identify all

the critical factors that could influence how a particular situation will develop as well as potential gaps in

the reporting. The technique is also useful if a large database is being assembled by helping to ensure that

an important field is not forgotten in the database architecture.

Strengths: Outside-In Thinking encourages analysts to move “outside their inbox” to rethink a problem

by employing a broader conceptual framework. By casting one’s net broadly, analysts are more likely to

see an important dynamic or to include a relevant alternative hypothesis. The process can provide new

insights and uncover relationships that were not evident from the reporting. In doing so, the technique

helps analysts think in terms that extend beyond day-to-day reporting to identify previously un-

considered, but fundamental forces and factors.

49

Developed from the Handbook of Analytic Tools & Techniques, Pherson, 2008, p.33

What can be influenced?

What cannot be influenced?

of 113

Develop

Hypothesis Generator.50

Purpose: The Hypothesis Generator provides a structured mechanism for generating a wide array of

hypotheses.

Description: This process uses a critical examination of a lead hypothesis to generate plausible

alternatives.

Process: First, crisply define the issue, activity, or behaviour that is subject to examination and establish a

lead hypothesis. (The lead hypothesis could be the one you were given, the most obvious explanation, or

the conventional wisdom.) Then:

Critically examine the lead hypothesis by identifying and listing its key elements. Use Who,

What, When, Where, Why, How and So What (in this or any other order) to evaluate all

dimensions of the hypothesis (note that some of these questions may not be appropriate for the

task at hand).

Generate plausible alternatives for each key element (see the example).

Generate a list of all possible permutations (see the box below).

Discard any permutation that simply makes no sense.

Evaluate the credibility of the

remaining permutations by

challenging the key assumptions of

each component.

Assign a “credibility score” to each

permutation using a 1 to 5 point scale

(see box on right).

Sort the permutations, listing them

from most to least credible (not

shown).

The permutations at the top of the list become the alternative hypotheses most deserving of

attention.

50

Developed from the Handbook of Analytic Tools & Techniques, Pherson, 2008, p.12

Etc.EtcEtcEtc

5Permutation 1-3-33.35Permutation 1-3-23.22.35Permutation 1-3-13.1

4Permutation 1-2-33.32Permutation 1-2-23.22.21.11Permutation 1-2-13.1


Etc.EtcEtcEtc


4Permutation 1-2-33.32Permutation 1-2-23.22.21.11Permutation 1-2-13.1


Example:

Terrorists are trying to harm us by providing illegal drugs through the Internet.

Element 1

1. terrorists,

2. drug companies,

3. entrepreneurs,

4. etc.

Element 2

1. harm us,

2. make money,

3. sabotage health care

programs,

4. etc.

Element 3

1. illegal drugs,

2. illegal prescription drugs,

3. legal prescription drugs,

4. etc.

Example:


Element 1

1.1 terrorists,

1.2 drug companies,

1.3 entrepreneurs,

1.4 etc.

Element 2

2.1 harm us,

2.2 make money,

2.3 sabotage health care

programs,

2.4 etc.

Element 3

3.1 illegal drugs,

3.2 illegal prescription drugs,

3.3 legal prescription drugs,

3.4. etc.

Example:


Element 1

1. terrorists,

2. drug companies,

3. entrepreneurs,

4. etc.

Element 2

1. harm us,

2. make money,

3. sabotage health care

programs,

4. etc.

Element 3

1. illegal drugs,

2. illegal prescription drugs,

3. legal prescription drugs,

4. etc.

Example:


Element 1

1.1 terrorists,

1.2 drug companies,

1.3 entrepreneurs,

1.4 etc.

Element 2

2.1 harm us,

2.2 make money,

2.3 sabotage health care

programs,

2.4 etc.

Element 3

3.1 illegal drugs,

3.2 illegal prescription drugs,

3.3 legal prescription drugs,

3.4. etc.

of 113

Develop

Hypothesis Review Technique.51 Purpose: An effective technique to mitigate mirror-imaging bias and to understand an adversary’s

potential mitigation strategies.

Description: After generating hypotheses to answer a given question, use this technique to gain a better

understanding of alternatives from the adversary’s point of view.

Process: For each alternative or

hypothesis to be reviewed, perform the

following steps from the point of view of

the adversary. Ensure all the steps for one

alternative are complete before going on

to the next.

List all the benefits or pluses for

the alternative being reviewed

from the adversary’s point of

view. (Why would this alternative

be a good choice?)

List all the risks or minuses for

the alternative being reviewed

from the adversary’s point of

view. (What detracts from this

alternative being a good choice?)

Review and consolidate the risks,

merging and eliminating as

appropriate.

Identify possible risk mitigation strategies that the adversary may adopt for each risk or minus.

Compare the benefits and unalterable risks for all options.

Finally, consider the results of this review on the overall analysis – amend if necessary.

Strengths: This technique can mitigate mirror-imaging bias and the natural negative bias. It can be

applied to any problem at any point in the analytic process as a simple, fast, and effective technique to

gain insight into the cultural-based adversary’s point of view, and of the potential implications of the

various options.

Weaknesses: This method can generate false impressions of a given alternative or the benefit-versus-risk

calculus if the analyst does not have an adequate understanding of the adversary’s culture-based point of

view.

Note: A similar, but slightly more complicated process – called the Adversarial Options Matrix – is

shown at the bottom of page 91.

51


pp 27-29

Plusses Minuses Mitigation Strategies

First Hypothesis or Alternative

1.

2.

3.

Etc.

1.

2.

3.

Etc.

1.a

1.b

2.a

2.b

3.a

3.b

Etc.

Second Hypothesis or Alternative

Etc. Etc. Etc.

Third Hypothesis or Alternative

Etc. Etc. Etc.

of 113

Develop

Reframing the Question.52 Purpose. This reframing technique provides a vehicle through which analysts can experiment with

different ways of defining an issue.

Description. Reframing the Question should be used whenever an analyst begins to assess a new issue or

problem, or begins a new research endeavour. Proper issue identification can save a great deal of time

and effort that is easily misspent on research and analysis of a poorly stated issue.

Process. Start with the best possible statement of the issue and then re-consider that statement as follows:

Paraphrase. Redefine the issue without losing the original meaning. Review the results to see if

they provide a better foundation upon which to conduct the research and assessment to gain the

best answer. Example: the original question, How much of a role does Aung Sung Sui Kyi play in

the ongoing unrest in Burma? is rephrased as, How active is the NLD headed by Aung Sung Sui

Kyi in the current antigovernment riots in Burma?

180 Degrees. Turn the issue on its head. Is the issue the one asked or the opposite of it? Example:

the original question, How much of the PLA ground capability would be involved in an initial

assault on Taiwan? is rephrased as, How much of the PLA ground capability would NOT be

involved in the initial Taiwan assault?

Broaden the Focus. Instead of focusing on only one piece of the puzzle, step back and look at

several pieces together. What is the issue before you connected to? Example: the original

question, How corrupt is President Musharraf? leads to the question, How corrupt is the

Pakistani government?

Narrow the Focus. Can the issue be broken down further? Take the question and ask about the

components that make up the problem. Example: the original question, Will the EU ratify a new

constitution? can be broken down to, How do individual member states view the new EU

constitution?

Redirect the Focus. What outside forces impinge on this issue? Is deception involved? Example:

the original question, What are the terrorist threats against the US homeland? is revised to, What

opportunities are there to interdict terrorist plans?

Ask Why. Ask why of the initial issue or question. Develop a new question based on the answer.

Then ask why of the second question and develop a new question based on that answer. Repeat

this process until you believe the real problem emerges. This process is especially effective in

generating possible alternative answers.

52


Techniques, First Edition, March 2008, pp.5-6

of 113

Evaluate

Data Diagnostic.

Purpose. The purpose of the Data Diagnostic process is to give analysts a process to use when

considering the veracity and usefulness of data.

Description. A thorough diagnosis of the available data (evidence, information, opinions, etc.) can often

reveal shortcomings that, once recognised, can be mitigated. To this end, the analyst should consider the

provider of the information, the information itself, the relevance of the information to the problem at

hand, and the probability of the existence of denial and/or deception.

Process. The Data Diagnostic process is divided into four parts:

Provider Diagnostic. Evaluation of the provider should consider:

o The agency, or witness, or source of the information

o And the handler (editor, agent handler, translator, processor, interpreter, etc.)

o In terms of:

Access and

Motivation.

Information Diagnostic. The information (data, evidence, opinions, etc.) itself should be

examined by considering its:

o Quality

Source access

Corroborability

Consistency

Completeness

o Logic

Main purpose

Key question

Important info

Inferences

Concepts

Assumptions

Implications (positive and negative)

Point of View

Relevance. A simplistic approach to relevance is to examine the salience of the information

(data, evidence, opinions, etc.) to the matter at hand (i.e. is it related, consistent, complete?).

Beyond salience, however, relevance can be examined through the lenses distorted by the

influence of biases that often prompt analysts to give credence to material inappropriately. Some

of these cognitive pitfalls are: the vividness criterion, the oversensitivity to consistency, the bias

favouring centralised direction, the similarity of cause and effect, internal vs. external causes of

behaviour, and the illusory correlation.

Denial and Deception. Finally, the possibility of denial and deception should be examined

through five sets of criteria:

o Does the potential deceiver have Motive, Opportunity, and Means (MOM) to deceive?

o Would this deception be consistent with Past Opposition Practices (POP)?

o Do we have cause for concern regarding the Manipulability of Sources (MOSES)?

o What can we learn from our Evaluation of Evidence (EVE)?

o Is the Source of the information simply Naïve (SON)?

of 113

Evaluate

Key Assumptions Check.53

Purpose: List and review the key working assumptions on which fundamental judgements rest.

Description: A Key Assumptions Check is most

useful at the beginning of an analytic project when

an hour or two can be instrumental in ensuring that

the impending assessment does not rest on flawed

premises. Rechecking assumptions can also be

valuable at any time prior to finalising judgements

to insure that the assessment does not rest on flawed

premises.

Process: Checking for key assumptions requires

analysts to consider how their analysis depends on

the validity of certain premises, which they do not

routinely question or believe to be in doubt. A four

step process can help with this challenge:

Review what the current analytic line on the

issue appears to be; write it down.

Articulate all the premises, both stated and

unstated in finished intelligence, which are

accepted as true for this analytic line to be

valid.

Challenge each assumption, probing why it “must” be true and whether it remains valid under all

conditions. The questions on the right will help in this regard.

Refine the list of key assumptions to contain only those that “must be true” to sustain the analytic

line; consider under what conditions or in the face of what information these assumptions might

not hold.

Alternatively - How to do it54

1. Identify your analytic line to be tested. It might look as follows:

RED is GREEN’s most important military supplier. RED has also been pivotal in assisting

GREEN with its WMD programmes which are now reaching maturity.

2. List all of the key assumptions that you believe underpin the analytic line, i.e. those that are accepted as

being true for the conclusions to be valid. For the example above these would look as follows:

Number Assumptions

1 RED is supplying GREEN militarily

2 GREEN has no other significant military supplier

3 RED has provided GREEN with non-military goods and training

53

Developed from the CIA A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence

Analysis, Jun 2005, pp 7-9 54

Copied from the UK Defence Intelligence Quick Wins for Busy Analysts, 2012, pp 18-19

Questions to ask during this process:

o How much confidence exists that

this assumption is correct?

o What explains the degree of

confidence in this assumption?

o Is this key assumption more likely a

key uncertainty or key factor?

o Could this assumption have been

true in the past, but not now?

o If this assumption proves to be

wrong, would the analytic line be

significantly impacted?

o Has this process identified new

variables that need further analysis?

of 113

4 GREEN has WMD programmes

5 GREEN depends on RED assistance for its WMD programmes

6 GREEN’s WMD programmes are reaching maturity

7 ...etc.

After you have developed as complete a list as you can, go back and critically examine each assumption

using the following questions to aid your thinking:

• If it were false, how seriously would this undermine the analytic line?

• How much confidence do you have that this assumption is valid?

- Why do you have this degree of confidence?

- Under what circumstances might this assumption be false?

- Could it have been true in the past but no longer true today?

- What would we expect to see if this assumption were true?

- Why aren’t we seeing these indicators?

Based on these, score each assumption according to two criteria:

• RELEVANCE:

- Largely irrelevant to analytic line (0)

- Important - analytic line would be significantly less likely if assumption were false (1)

- Essential - analytic line cannot be true without assumption (2)

• SUPPORT:

- Unsupported or very questionable (0)

- Correct with some caveats (1)

- Solid (2)

3. We are looking for the shaky, load-bearing assumptions. Find the assumptions which score highest for

“relevance”. Of these, the assumptions with the lowest “support” scores are the key uncertainties. A

matrix template like the one below should be used to filter your key assumptions and provide a clear

structure for the exercise. Use a comments column to record the rationale behind the results of your

confidence check. This might relate to the quality and quantity of evidence, reliability of sources etc.

In this example, the scores suggest that items 4 and 5 require revisiting as they are essential to the analytic

line but unsupported. Item 3 can be ignored.

Consider whether the key uncertainties identified have revealed collection requirements. The number of

key uncertainties will also dictate whether/how much the analytic line requires further research and

analysis, including contact with collectors, to ensure it is as robust as possible and accurately reflects the

available information.

Number Assumption Relevance Support

1 RED is supplying GREEN militarily 2 2

2 GREEN has no other significant military supplier 1 1

3 RED has provided GREEN with non-military goods and

training 0 2

4 GREEN has WMD programmes 2 1

5 GREEN depends on RED assistance for its WMD

programmes 2 0

of 113

Evaluate

Indicators of Change.55 Purpose: To periodically review a list of observable events or trends to track events, monitor targets, spot

emerging trends, and warn of anticipated change.

Description: An analyst or team creates a list of indicators or

signposts of observable events that could be expected to

become apparent if a postulated situation is developing. The

technique can be used whenever an analyst needs to track an

event over time or monitor and evaluate changes.

Process: Whether used alone, or in combination with other

structured techniques, the process is the same:

Identify a set of competing hypotheses or scenarios.

Create separate lists of potential activities, statements,

or events that are expected to become manifest for

each hypothesis or scenario – and preferably only for

that one scenario or hypothesis.

Regularly review and update the indicators lists to see

which are changing.

Identify the most likely or most correct hypotheses or

scenarios, based on the number of changed indicators

that are observed.

Developing sets of indicators for each scenario or hypothesis

may be useful in distinguishing whether or not a development is

emerging or not emerging.

Strengths: By providing an objective baseline for tracking

events or targets, indicators instil rigor into the analytic process

and enhance the credibility of analytic judgements. Including an

indicators list in a finished product builds a more concrete case

for the analytical judgements and may help a client track

developments. By laying out a list of critical variables, analysts

will also be generating hypotheses regarding why they expect to

see the presence of such factors. In so doing, analysts make the analytic line much more transparent and

available for scrutiny by others.

Weaknesses: Analysts can become fixated on watching for the identified indicators, missing signals or

developments that are not anticipated.

55


Analysis, Jun 2005, pp 12-13

Start Point

Indicator

Possible Path

Scenario

Start Point

Indicator

Possible Path

Scenario

of 113

Evaluate

Indicators Validator.56

Purpose. This is a simple tool for assessing the diagnostic power of

indicators.

Description. Once the analyst (or the analytic team) has created a set of

alternative scenarios (or future worlds), and has also generated a set of

indicators for each scenario (or world), an indications validation process

may be undertaken to help determine the diagnostic value of the

indicators.

Process. In this process, the analysts populate a matrix similar to that

used for the Analysis of Competing Hypotheses (see page 74), involving the

following steps:

List the scenarios or worlds on the top of the matrix (as is done for

hypotheses in ACH).

List the indicators on the left side of the matrix (as is done with evidence in

ACH).

In each cell of the matrix, assess whether, for each particular

scenario, the indicator is:

o Highly likely to appear,

o Likely to appear,

o Could appear,

o Unlikely to appear, or

o Highly unlikely to appear.

Determine the diagnostic value of each indicator by assigning

a numeric value to each cell. If the “home” indicators are rated

HL for each attendant scenario, then scenarios with an L have

a value of 1, C = 2, U = 3 and HU = 4. (If the “home”

indicator is only rated as L [D and F in the example], then each

of the other values are one point lower.) The indicators with

the highest total score are the most discriminating, while

indicators with low scores are of little use.

Re-sort the indicators placing the most discriminating

indicators on the top of the matrix and the least discriminating

indicators at the bottom where they can be eliminated.

With the non-discriminating indicators eliminated, the analyst

should review the re-ordered list of indicators. If most indicators

for a particular scenario have been eliminated, the analyst should

develop new, and more diagnostic, indicators for that scenario (and

recheck their diagnostic value by applying the Indicators Validator

process to them as well).

56

Based on Richards J. Heuer, Jr. And Randolph H. Pherson’s Structured Analytic Techniques for Intelligence

Analysts, copyright 2011 by CQ Press, and an Indicators Validator paper, Jan 2009, by Randy Pherson.

Scenario Indicator Value

1. A. 1

B. 1

2. C. 1

D. 2

3. E. 1

F. 2

4. G. 1

H. 1

Highly Likely

Likely

Could

Unlikely

Highly Unlikely

Highly Likely

Likely

Could

Unlikely

Highly Unlikely

Scenario

Indicator 1. 2. 3. 4.

A. (6) HL C C C

B. (9) HL C U HU

C. (11) U HL HU HU

D. (9) HU L HU HU

E. (12) HU HU HL HU

F. (9) HU HU L HU

G. (11) HU U HU HL

H. (10) C HU HU HL

Scenario

Indicator 1. 2. 3. 4.

A. HL

B. HL

C. HL

D. L

E. HL

F. L

G. HL

H. HL

of 113

Evaluate

Analysis of Competing Hypotheses. (ACH). 57 58

Purpose: ACH is a tool which can aid judgement on important issues through the careful weighting of

alternative explanations or conclusions. It helps to overcome, or at least minimize, some of the cognitive

limitations that make prescient intelligence analysis so difficult to achieve. ACH should not be expected

to reveal the “right answer,” but it may help identify likely hypotheses and allow the elimination of less

likely ones.

Process: ACH is a process as follows:

Identify the issue to be examined and specify the possible hypotheses to be considered. These

should, in a perfect world, be both mutually exclusive and all encompassing.

Make a list of significant evidence and assumptions. That is, everything that is relevant to

evaluating the hypotheses – including evidence, arguments, assumptions, and the absence of

things one would expect to see if a hypotheses were true or false.

Determine the credibility score (high, medium or low) for each piece of evidence or assumption

by considering the source, the quality and logic of the information, its relevance, and the

potential for denial and deception.

Prepare a matrix, with the hypotheses across the top and the evidence/assumptions/etc. and its

corresponding credibility score down the side. Consider each piece of evidence or assumption

against each of the hypotheses and determine its diagnosticity. Is it consistent (C), inconsistent

(I), very inconsistent (II) or neutral (N)? Score the data. Inconsistent pieces receive

a score of two marks and very inconsistent

receive four (consistent and neutral pieces

receive no marks). For each inconsistent or

very inconsistent piece, the credibility

weighting is then considered, and one extra

mark is added if the weighting is medium

and two extra marks if it is high.

Total the scores at the bottom of each

column.

Draw tentative conclusions about the

relative likelihood of each hypothesis.

Analyse how sensitive your conclusion is to

a few critical items of evidence or

assumptions. Consider the consequences

for your analysis if that

evidence/assumption were wrong, or

misleading.

Report conclusions – remember, the process only identifies hypotheses which are least likely.

Software: This form of analysis can be done in MSWord in a table, using MSExcel or with the simple

and free ACH Tool, (http://www2.parc.com/istl/projects/ach/ach.html).

57

Developed from Psychology of Intelligence Analysis, Heuer, 2003, Chapter 8. 58

Scoring process developed from The Art of Intelligence, Lahneman and Arcos, 2014, p. 32.

Issue or Problem

Being Examined

Hypotheses

Ranking ( II / I / N / C )Diagnostic

Weighting

( H / M / L )

Evidence

Assumption

Argument

Absence 654321

Total

Hypothesis 6

Hypothesis 5

Hypothesis 4

Hypothesis 3

Hypothesis 2

Hypothesis 1

Issue or Problem

Being Examined

Hypotheses

Ranking ( II / I / N / C )Diagnostic

Weighting

( H / M / L )

Evidence

Assumption

Argument

Absence 654321

Total

Hypothesis 6

Hypothesis 5

Hypothesis 4

Hypothesis 3

Hypothesis 2

Hypothesis 1

http://www2.parc.com/istl/projects/ach/ach.html

http://www2.parc.com/istl/projects/ach/ach.html

of 113

Explore Networks

Social Network Analysis.

Purpose: To reveal patterns of interaction and the allocation of social and personal capital within a group

of people.

Description: Social Network Analysis (SNA) uses sociograms and mathematical computations to reveal

significant aspects of behaviour in groups. The sociograms (network analysis charts) can show the

relative importance of individuals in a network, and the nature and relative strengths of the links between

individuals.

Concepts used in SNA include:

Centrality - the relative position of an

individual within a network. Centrality

reveals influence and control over

communications and other exchanges.

Density - the number of links in a network

as a percentage of the number of possible

links. In general, high density groups adapt

more easily to the loss of an individual,

while low density groups are harmed less

by an informant in their midst.

Redundancy - the extent to which two or more individuals are equivalent to each other. The

equivalence may be in terms of their network role, or their functional capabilities.

Betweenness - is a measure of connectiveness. Individuals or groups that have many

paths between them have higher betweenness than those that do not.

Cutpoints - individuals who are the sole connections between network components.

SNA is a useful technique in targeting intelligence. It can break groups into smaller, separate entities, and

can identify individuals whose removal would cause the greatest disruption in the group’s functioning.

Detailed information about the connections between individuals in the group under study is required,

including the nature and content of those connections.

Strengths: Identifies the structural strengths and vulnerabilities of a group. In groups with a hierarchical

structure, SNA reveals patterns of communication, exchange and influence. In groups without formal

hierarchy or structure, the social network is the primary organizing principle.

Weaknesses: Requires rich data about the content of links between individuals, which can be difficult to

acquire when studying covert groups.

Recommended Readings:

“Notions of Position in Social Network Analysis” pp 1-35 in Sociological Methodology, Vol 22.

Stephen P. Borgatti and Martin G. Everett, 1992.

“Destabilizing Networks” pp 79-92 in Connections 24(3). Kathleen M. Carley, Ju-Sung Lee, and

David Krackhardt. 2002.

4

10 5

2

13

81215

13

9

14

26256

716

21

2417181920

22 23

4

10 5

2

13

81215

13

9

14

26256

716

21

2417181920

22 23

of 113

Explore Networks

Link Charts. Purpose: The objective of a link diagram is to graphically depict relationship data.

These diagrams aid investigators and analysts by uncovering, interpreting, and displaying complex

information in easily-understood chart form. (See also page 27.)

Process:

Assemble all of the information.

Identify entities of interest and their

associations.

Construct an association matrix and

populate it with the information of

interest.

Identify the nature of the associations.

Draw a preliminary network diagram.

Refine the diagram.

Finally, develop hypotheses.

While individual analysts, and projects, will

have different ways of showing data, the

following are recommended:

show organizations as boxes and people as circles (icons

may also be used),

place people within the box for their organizations,

link individuals and organizations with lines (solid for

confirmed links, dotted or dashed for suspected links),

use coloured links to represent phone calls, financial

transactions, commodity flows, etc.,

avoid crossing lines,

place key entities in the centre, and

include a legend or key.

Strengths: Link Charts:

integrate data from different sources,

highlight key relationships,

serve as a visual briefing aid for team members,

prosecutors and juries,

show when cases overlap (when multiple charts

are combined),

provide a cumulative snapshot of a case.

Software: The Analyst's Notebook (i2inc.com) – the

Cadillac of charting software (about US$4,400) – is used

by over 2,000 law enforcement, military and intelligence

organizations. http://www.visualanalytics.com/index.cfm.

Toolbar

PaletteChart

Area

Toolbar

PaletteChart

Area

Perso

n 1

Perso

n 2

Perso

n 3

Perso

n 4

Act

ivity

1

Locat

ion

1

Locat

ion

2

Locat

ion

3

Locat

ion

4

Perso

n 5

Locat

ion

5

Veh

icle

1

Certain

Association

Suspected

Known

Owner

Suspected

Owner

?

?

?

?

?

? Collection

Gap

Perso

n 1

Perso

n 2

Perso

n 3

Perso

n 4

Act

ivity

1

Locat

ion

1

Locat

ion

2

Locat

ion

3

Locat

ion

4

Perso

n 5

Locat

ion

5

Veh

icle

1

Association

Known

Owner

Suspected

Owner

Inferred

Relationship

Inferred

Relationship?

?

?

?

?

? Collection

Gap

?

?

?

?

?

? Collection

Gap

I

I

I

I

I

Perso

n 1

Perso

n 2

Perso

n 3

Perso

n 4

Act

ivity

1

Locat

ion

1

Locat

ion

2

Locat

ion

3

Locat

ion

4

Perso

n 5

Locat

ion

5

Veh

icle

1

Certain

Association

Suspected

Known

Owner

Suspected

Owner

?

?

?

?

?

? Collection

Gap

Perso

n 1

Perso

n 2

Perso

n 3

Perso

n 4

Act

ivity

1

Locat

ion

1

Locat

ion

2

Locat

ion

3

Locat

ion

4

Perso

n 5

Locat

ion

5

Veh

icle

1

Association

Known

Owner

Suspected

Owner

Inferred

Relationship

Inferred

Relationship?

?

?

?

?

? Collection

Gap

?

?

?

?

?

? Collection

Gap

Perso

n 1

Perso

n 2

Perso

n 3

Perso

n 4

Act

ivity

1

Locat

ion

1

Locat

ion

2

Locat

ion

3

Locat

ion

4

Perso

n 5

Locat

ion

5

Veh

icle

1

Certain

Association

Suspected

Known

Owner

Suspected

Owner

?

?

?

?

?

? Collection

Gap

?

?

?

?

?

? Collection

Gap

Perso

n 1

Perso

n 2

Perso

n 3

Perso

n 4

Act

ivity

1

Locat

ion

1

Locat

ion

2

Locat

ion

3

Locat

ion

4

Perso

n 5

Locat

ion

5

Veh

icle

1

Association

Known

Owner

Suspected

Owner

Inferred

Relationship

Inferred

Relationship?

?

?

?

?

? Collection

Gap

?

?

?

?

?

? Collection

Gap

I

I

I

I

I

Cabal

Bill’s

Diner

Mini Mall

Andy

Jim

George

Chuck

Bird

Business

Oma

Cabal

Bill’s

Diner

Mini Mall

AndyAndy

JimJim

George

ChuckChuck

Bird

Business

OmaOma

Cabal

Bill’s

Diner

Mini Mall

AndyAndy

JimJim

George

ChuckChuck

Bird

Business

OmaOma

Cabal

Bill’s

Diner

Mini Mall

AndyAndy

JimJim

George

ChuckChuck

Bird

Business

OmaOma

http://www.i2inc.com/

http://www.visualanalytics.com/index.cfm

of 113

Compare

Cross-Impact Matrix.59

Purpose: By comparing and contrasting the variables of a complex problem, analysts can systematically

examine how each factor in a particular context influences all other factors to which it appears to be

related.

Description: When analysts are estimating or forecasting future events,

they consider the dominant forces and potential future events that might

influence the outcome. The Cross-Impact Matrix provides a context for

discussing the relative influence of these forces or events and a structure

for managing that complexity.

Process: Having settled on a firm understanding of the issue at hand, the

Cross-Impact Matrix process is as follows:

Develop a list of variables and/or events that would likely have some effect on the issue being

studied.

Create a matrix and put the

variables or events down the

left side of the matrix and the

same variables or events

across the top.

Use the matrix to consider

and record the relationship

between each variable or

event and every other

variable or event. For

example:

o Does the presence of Variable 1 increase or diminish the influence of Variables 2, 3, 4,

etc.? Or, does the occurrence of Event 1 increase or decrease the likelihood of Events 2,

3, 4, etc.?

o If one variable does affect the other, record its magnitude in the variable’s row. (Note

that each relationship is considered twice. Variable 1’s influence on Variable 2 is

recorded in the Variable 1 row, while Variable 2’s influence on Variable 1 is recorded in

the Variable 2 row – and they might not be the same.)

Study the matrix to determine:

o The direction and the magnitude of the various relationships.

o Which variables have the greatest impact on others (in the example, variables 2 and 5

have the strongest influence).

o If there are any variables that might reinforce each other either directly (variables 2 and

4), or through a third variable (variables 3, 1 and 5). Combinations of variables that

reinforce each other can lead to surprisingly rapid changes in a situation.

Strengths: The Cross-Impact Matrix often reveals that variables or issues once believed to be simple and

independent are, in reality, interrelated.

59

Developed from Richards J. Heuer, Jr. And Randolph H. Pherson’s Structured Analytic Techniques for

Intelligence Analysts, pp 104-107, copyright 2011 by CQ Press.

Magnitude

++ Strong positive

+ Positive

Neutral

- Negative

-- Strong Negative

Variable 1 Variable 2 Variable 3 Variable 4 Variable 5

Variable 1 + ++

Variable 2 -- - ++ +

Variable 3 ++ - +

Variable 4 ++ +

Variable 5 -- + ++ +

of 113

Compare

Structured Comparison. 60

Purpose: To rank-order priorities for complex issues with large sets of options.

Description: This technique uses sets of defined criteria to allow a formal comparison of entities or

phenomena. The set of attributes (criteria, qualities) and their values will be specific to each problem and

must be very carefully established.

Structured Comparison / Multiple Criteria

Analysis are approaches to decision making

used in the private sector, military and

government. In the context of intelligence

analysis, structured comparison tools have to be

specific to the decision-making problem. For

example, set of attributes designed to rank-order

organized crime groups in terms of their

strengths and weaknesses will not be valid for

terrorist groups.

The attribute set must be developed by the

appropriate range of subject matter experts in order to ensure relevance to the

problem. The criteria must have clear working definitions to calibrate the

technique, and so ensure the reliability of the results. Analysts will need to have a

good understanding of the criteria as defined in the technique in order to apply it.

Strengths: More objective, comprehensive, and reliable than unstructured

comparison exercises. The procedure can be repeated over time, and can be

applied by multiple groups of analysts; it increases credibility of rank ordering,

and permits auditing of a process.

Weaknesses: Time and resources are needed to create specific attribute sets. If the process and results

are not reviewed there is the potential for respondents to slant their input, whether consciously or

subconsciously, in an attempt to promote their preferred results.

Example: In order to develop the matrix above, high, medium, low and nil definitions of each of the

criteria (A, B, C, etc.) would need to be developed and agreed upon. Then, for each entity or

phenomenon an assessment would need to be made for each of the criteria to determine the appropriate

level. Colour coding and sorting could be done either manually, or using an IT solution.

60

Developed from Project Sleipnir – An Analytical Technique for Operational Priority Setting, a paper presented by

Steven J. Strang, RCMP, at the 2005 International Conference on Intelligence Analysis, McLean, VA,.May 2 to 4,

2005.

E

D

C

B

A

7651234Criteria

Qualities

Entities or Phenomenon

Complex Issue to be Examined

E

D

C

B

A

7651234Criteria

Qualities

Entities or Phenomenon

Complex Issue to be Examined

Unknown

Nil

Low

Medium

High

Unknown

Nil

Low

Medium

High

https://analysis.mitre.org/proceedings/Final_Papers_Files/135_Camera_Ready_Paper.pdf

of 113

Compare

Weighted Rankings.61

Purpose: A technique used by an individual or group to gain confidence in the assessment of available

alternatives by weighting criteria in importance.

Process: Having determined (using a separate

technique) the question to be answered, there are

eight steps to accomplish a weighted ranking review:

Identify all the major criteria for ranking,

Pair-rank the criteria by rating each

individual criteria against each other criteria

in turn, and

Select top several criteria and weight them

in percentiles.

Then, having identified the items to be ranked:

Construct a weighted-ranking matrix; enter

the items to be ranked, the

selected criteria and their

weights,

Pair-rank each of the items to

be ranked by each criterion

and record the votes,

Multiply the number of votes

by criterion’s weight,

Add the weighted values for

each item and total, and

Determine the final rankings.

Conclude by conducting a sanity

check of the results and reviewing the

impact of the weighted criteria on the

final result.

Example: To answer the question – What is the best method for conducting a terrorist attack? The

top box identifies, pair ranks, and weights the major criteria (what does best mean in this context?). Then,

having also identified the items to be ranked, the second box shows the pair ranking of each item against

each of the selected criteria, the multiplication of the number of votes by the criteria’s weight, the total for

each item and the final ranking. A sanity check needs to be done before accepting that the man-portable

bomb is really the best method for conducting a terrorist attack.

61


pp 63-67, and the USEUCOM CTSA Training Package.

Ranking the Major Criteria

Votes Ranking Weight

Physical Destruction IIII (4) 3 .27

Political Disruption II (2) 4

Social Disruption I (1) 6

Economic Disruption II (2) 5

Ease of Delivery IIIII I (6) 1 .4

Ease of Acquisition IIIII (5) 2 .33

Cultural Impact I (1) 7

Selected Criteria

Items to be Ranked

Ease of

Delivery:

.4

Ease of

Acquisition:

.33

Physical

Destruction:

.27

Total Final

Ranking

Dirty Bomb 3x.4=1.2 2x.33=0.7 6x.27=1.6 3.5 7

Poison Gas 2x.4=0.8 5x.33=1.65 5x.27=1.55 4.0 6

Car Bomb 5x.4=2.0 6x.33=2.0 2x.27=0.5 4.5 3

Crash Airplane 4x.4=1.6 4x.33=1.3 4x.27=1.1 4.0 5

Blow up Airplane 6x.4=2.4 3x.33= 1.0 3x.27=.8 4.2 4

Man-Portable Bomb 8x.4=3.2 8x.33=2.6 1x.27=0.3 6.1 1

Suicide Bomb 7x.4=2.8 7x.33=2.3 0x.27=0 5.1 2

Biological Attack 1x.4=0.4 1x.33=0.3 7x.27=1.9 2.6 8

Nuclear Bomb 0x.4=0.0 0x.33=0.0 8x.27=2.2 2.0 9

of 113

Compare

Change Analysis.62

Purpose: Change Analysis is a technique for gaining insight into the impact of incremental change over

extended periods of time that may otherwise go unnoticed.

Description: Incremental change can often be difficult to notice, and while the aggregate of the change

over time may be large, each actual change may be too small to attract notice. Change Analysis is a

process whereby modifications in the criteria or factors that encompass or define an issue or relationship

are studied to determine whether or not alterations have occurred over time.

Process: As with many aspects of intelligence analysis, the first and most important step in undertaking a

Change Analysis is the selection of the issue and the timeframe to be analysed. The issue can be virtually

anything for which defining criteria or factors can be identified (the state of an international relationship;

the influence of an insurgent; etc.), and the timeframe is what ever is appropriate. Once the issue and the

time have been determined, then:

Select the comparison criteria or factors - Determine the criteria or factors that will give the best

insight into the most telling implications of the change. The best criteria or factors will:

o Be observable, measurable or determinable at each interval.

o Give valid and reliable insight into the amount of change occurring.

o Remain stable over the timeframe of the event or relationship.

o Taken together, provide unique insight into the change in the event or relationship being

assessed. They should imply one conclusion and only one. If other interpretations and/or

hypotheses are plausible, additional or more diagnostic indicators should be identified.

Establish appropriate time intervals – The timeframe, determined initially along with the issue,

needs to be broken into discrete segments that will illuminate the amount and speed of change

taking place. These segments should:

o Begin at a logical start point for the phenomena to be assessed.

o Be consistent (monthly, yearly, quarterly, etc.).

o Be tied to watershed events, such as elections, or recurring crises.

o Facilitate the measurement of the comparison criteria or factors.

Record the status of each criterion or factor at each interval – Determine the state of each

criterion or factor at the end of each interval, using a measurement process that is consistent in

detail and focus throughout.

Assess the change – Examine both the amount of change between the time intervals and the

overall change across the entire timeframe. Identify and assess the implications of:

- The rate of change for each criterion between time intervals and for the entire timeframe.

- The correlation between external events and the degree of change observed (did an

external event cause less or greater change compared to other periods?).

- Turning points that have been identified – and their implication for the future.

- Continued change at the current rate.

Weakness: Using selection of poor or inappropriate comparison criteria, factors, timeframe, or intervals

can cause misleading results and even a failure to see important changes.

62

Developed from the “Change Analysis” structured analytic technique authored by Jay Hillmer, DIA, Oct 2010.

of 113

Generate Scenarios

Cone of Plausibility.63

Purpose: To identify drivers that shape current events in a subject area, and then generate a series of

plausible scenarios that are expected to describe the endpoint situation.

Description: The Cone of Plausibility is a disciplined scenario generation technique that helps analysts

imagine plausible futures and their effects. It is valuable in understanding the drivers that shape current

events and as a form of Strategic Warning.

Process: The Cone of Plausibility process begins with a very careful definition of the issue to be

examined. This will generally be in the form of - What will ____ look like in __?, where the second

____ is a specific time (be it two weeks, two

months, five years, etc.) .

Once the question and time frame have been

decided upon, the analyst needs to first identify and

then describe the drivers that are most useful in

defining the issue. It is important both that the

drivers and their descriptions accurately portray the

current situation, and that they be expected to

generally remain valid throughout the period.

Then, with a firm and thorough understanding of the

issue at hand, encapsulated in the drivers and

descriptions, the analyst can go on to assemble a Baseline Scenario. This scenario assumes that the

drivers and their descriptions will remain valid throughout the period. Next, a Plausible Scenario can be

produced by changing one (or more? – but be careful) driver and description – usually the one that is least

likely to hold true. Finally, a Wildcard (or another Plausible) Scenario can be generated by changing a

different baseline driver and description. Typically, the wildcard describes a future that is low probability

but high impact, or it may be the most dangerous scenario.

Requirements: Analysts involved in scenario generation through the Cone of Plausibility technique need

to have a thorough understanding of the subject under study, need to be able to select and define the

drivers that are most likely to remain valid throughout the timeframe selected, and need to be able to

imagine future impacts.

Strengths: The Cone of Plausibility Technique is particularly useful in providing decision makers with

discussion material, both through the drivers that define the subject area and the generated scenarios.

63

Developed from UK DIS materials.

Now ?

DriversDescriptions

Scenario 2:Plausible

Scenario 1:Baseline

Scenario 3:Wild Card

Now ?Now ?

DriversDescriptions


Scenario 1:Baseline


Now ?Now ?

DriversDescriptions


Scenario 1:Baseline


Now ?Now ?

DriversDescriptions


Scenario 1:Baseline


of 113

Generate Scenarios

Quadrant Hypothesis Generation Technique.64

Purpose: A rapid method of generating four hypotheses based on the two main factors, criteria or drivers

affecting the topic or event.

Description: For use when a topic or event has only two key driving forces that are easily identified and

with a wide consensus.

Process: The technique should be undertaken as

follows:

First, identify the two key factors (criteria or

drivers) and the descriptions of their

extremities through techniques such as

brainstorming or surveying experts. (If there

are more than two factors, use the Alternative

Futures Analysis technique – page 83.)

Then, draw a quadrant with the two factors

on the horizontal and vertical axes. (Public

Attitudes and Government Policy in the

example.)

Determine and place the extremes of the

factors at the end of each axis. (Outreach and

Crackdown for Government Policy, Anti-

Muslim Sentiment and Sympathy & Support

for Public Attitudes in the example.)

Finally, in each quadrant, formulate the

hypothesis that best leads to the end state that

is suggested by the combination of the

applicable extremes of the two drivers.

Strengths: The technique is especially good for developing future outcomes, in the form of four

hypotheses, from the interaction which the main forces either have or could have on the outcome.

Discussion of what comprises the two main factors (criteria or drivers), and identifying their extremes,

can be a useful exercise in itself.

Weaknesses: The technique depends upon the correct identification of two major driving forces. Where

there are numerous major driving factors, or there are significant disagreements on which forces are major

drivers, the technique should not be counted on.

64


pp 25-26

Pub

lic

Att

itud

es

Topic: Home Grown Terrorism

Key Drivers:

Government Policy

Public Attitudes

Anti-Muslim

Sentiment

Sympathy

& Support

Government PolicyOutreach Crackdown

Hypothesis #1

Hypothesis #4

Hypothesis #2

Hypothesis #3

Pub

lic

Att

itud

es


Key Drivers:

Government Policy

Public Attitudes

Anti-Muslim

Sentiment

Sympathy

& Support


Hypothesis #1

Hypothesis #4

Hypothesis #2

Hypothesis #3

of 113

Generate Scenarios

Alternative Futures Analysis.

Purpose: To systematically identify a range of alternative futures, monitor for signposts, and thereby

avoid surprise.

Description: Alternative futures Analysis takes over when the

two factor limit of the Quadrant Hypothesis Generation (see

page 82) technique is too restrictive. In this technique, the

group of experts collectively identifies three (or more) factors,

criteria or drivers that are likely to shape an issue.

Then, during the “quadrant crunching” phase, each of the

factors is compared to each of the other factors in a 2x2 matrix,

producing a different hypotheses, “future world” or scenario in

each quadrant, as shown.

Strengths: Most other efforts to anticipate future events usually end up

with minor variations of current trends. The Alternative Futures

technique forces analysts to look at not only the most likely scenarios

but also those that might seem unlikely, even counter-intuitive. Some of

these may have high impact.

Weaknesses: This is a highly efficient and effective technique for

generating an extremely broad set of alternatives when faced with very

little data and high degrees of uncertainty, but,

it takes time and effort to do well, and

a diverse group of participants is critical.

Also, it is difficult to manage more than three factors. Four can be

combined six different ways producing 24 scenarios, six can be

combined 15 ways producing sixty scenarios, etc.

On Completion: Analysts can then ask: What signposts or indicators would exist on the path to each

“future world”?

Home Grown Terrorism

Key Drivers

A. Government Policy

B. Public Attitudes

C. Immigration Trends

A

B

A

C

B

C

Home Grown Terrorism

Key Drivers

A. Government Policy

B. Public Attitudes

C. Immigration Trends

A

B

A

C

B

C

A

B

A

C

B

C

Pub

lic

Att

itud

es


Anti-Muslim

Sentiment

Sympathy

& Support


Valuing

Diversity

Touchy

Feely

Hard

Times

Necessary

Measures

Defining potential outcomes

Pub

lic

Att

itud

es


Anti-Muslim

Sentiment

Sympathy

& Support


Valuing

Diversity

Touchy

Feely

Hard

Times

Necessary

Measures

Defining potential outcomes


Select the scenarios deserving the most attention

Hypotheses

#1Hard

Times

Hypotheses

#4

Hypotheses

#3

Hypotheses

#5

Conflict

Brewing

Hypotheses

#7

Hypotheses

#9

Civil

Strife

Hypotheses

#11

A

B C C

A B

Hypotheses

#6

Hypotheses

#10

Legend – Deserving the most attention


Select the scenarios deserving the most attention

Hypotheses

#1Hard

Times

Hypotheses

#4

Hypotheses

#3

Hypotheses

#5

Conflict

Brewing

Hypotheses

#7

Hypotheses

#9

Civil

Strife

Hypotheses

#11

A

B C C

A B

Hypotheses

#6

Hypotheses

#10

Legend – Deserving the most attention

of 113

Understand Possibilities

Force Field Analysis.65

Purpose: To better understand forces or key factors, criteria or drivers influencing an issue or change. It

is also a useful tool for generating signposts and indicators.

Description: Force Field Analysis investigates the balance of forces acting on an issue or influencing a

change. It attempts to identify the most important players and target-groups involved in the issue and their

relative influence. It is excellent in providing a vehicle to promote discussion of a topic.

If a change is contemplated, Force Field

Analysis can help identify potential opponents

and allies. At the onset of a study, it clarifies

key factors or forces. It also helps challenge

assumptions about these factors and highlights

those that are stabilising or destabilising.

Process: To carry out a Force Field Analysis:

Describe, in detail, the plan or proposal

for change.

List all of the forces supporting the

change in one column.

List all of the forces resisting the

change in another.

Scores can be assigned to each force showing

relative strengths. These values can then be

used to compare relative weights of the forces

for verses the forces against the proposal. They

can also be used to show the effect of a change

in the influence of a factor, if it can be modified.

The “exam question” must be clearly identified and understood by the study team before embarking on

the identification of positive and negative forces.

Strengths: Force Field Analysis breaks down the core forces / drivers that influence or impact a

situation. It allows users to create, and then consider, a wider array of alternative scenarios and

possibilities. It is particularly useful when timelines are short, or when a relatively simple yes/no

assessment is required.

Weaknesses: The technique does not generate a specific “baseline” scenario and the forces identified as

impacting on the subject may be too numerous, overly complicating the process. When generating

scenarios, because of the polar nature of the forces, the gray areas may not be adequately examined.

Software: A Force Field Analysis Worksheet and instructions are available at

http://www.mindtools.com/rs/ForceField.

65

Developed from UK DIS material.

Forces for

change

Forces against

changeForces for

change

Forces against

change

http://www.mindtools.com/rs/ForceField

of 113


Counterfactual Reasoning.66

Purpose: Counterfactual Reasoning is a structured process for generating alternate future scenarios that

are set in context.

Description: This technique is

particularly suited to answering

what-if questions where the

context includes: the different

situations which could bring those

scenarios about, and the other

events which will also be taking

place during the same time

period.

Process: Once the analytic

challenge has been defined (What

will B look like if A happens?),

the Counterfactual Reasoning

technique uses the following seven step process:

Generate antecedent scenarios – Identify the possible ways that A could come about.

Rank the antecedent scenarios – In this step we rank order the scenarios in terms of the

probability of each scenario occurring, and the length/complexity of each scenario (in other

words, the number of different sequences of events that need to converge to bring about the

scenario).

Generate intermediate events – Identify the relevant events that will independently occur

irrespective of A, and potentially influencing B.

Rank the intermediate events – In this step we need to examine the likelihood that the

intermediate events will be more or less important given the occurrence of event A. That is, does

the likelihood of the intermediate events being influential increase, decrease or remain the same if

A occurs?

Generate consequent scenarios – Identify potential outcomes (B) that are consistent with both the

antecedent scenarios and the intermediate states.

Rank the consequent scenarios – Determine whether or not the scenarios can have their

probability quantified (most likely/most dangerous/most costly/etc.). If they can, rank order them

– especially if there are lots of them.

Select the most significant consequent scenarios – Finally, identify the scenarios that are most

likely/most dangerous/most costly for inclusion in an analytic product.

Strengths: Using this technique to answer a “What will B look like if A happens?” question, prompts the

analyst to: make reference to the events that will likely cause A; incorporate the influence of events that

will take place irrespective of A but which may influence B; and put the consequences (B) in order of

probability.

66

Developed from Dr Noel Hendrickson’s presentation on counterfactual reasoning given to the 4-Eyes Analytic

Training Workshop, 12-16 Jan 2009 and Counterfactual Reasoning – A Basic Guide for Analysts, Strategists, and

Decision Makers, Dr Hendrickson, The Proteus Monograph Series Vol 2, Issue 5, Oct 2008.

Timeline

Today Future

Antecedents

Intermediates

Timeline

Today Future

Antecedents

Intermediates

of 113


Bow-Tie Technique. 67

Purpose: The Bow Tie Analytic Technique is used to model the causes and consequences of a specific

“undesirable event” and can help identify actions that have the best chance of reducing the most likely

causes and mitigating the worst consequences.

Description: This

technique is useful in

that it identifies both the

causes that can lead to

the undesirable event

and the consequences

emanating there from.

The technique also

allows the development

and consideration of

both proactive and

reactive control

measures, and can be

used to show the effect

of a policy, or physical

change on the issue.

Usage: The process begins by identifying, as specifically as possible, the “event” that is to be examined.

This can be a single undesirable event – such as the loss of control of a vehicle (see the example in the

boxes), or a failure of policy – the diplomatic or military eviction from a locale, or a political

development – the rise to power of a dictatorial leader. The event does not, however, need to be negative.

Equally plausible are desirable goals, such as valid elections or the achievement of an objective.

Once the event has been defined, analysts should identify causes that could bring this event about.

Causes can be thought of as factors or drivers – i.e. circumstances, facts or influences that, either

separately or in combination with each other could cause the event to take place. Depending on

the number of causes identified, a process of ranking them – by likelihood or impact – may be

necessary.

Then, analysts should identify the consequences of the undesirable event. These may take

numerous forms (physical, psychological, internal or external, etc.) and it may be necessary to

separate them into categories or rank them in order of priority or impact.

Having established the elements of the “bow tie,” analysts should

begin to consider the impact of both proactive and reactive

control measures on the issue. In this example, three categories

have been used, physical or engineered, maintenance, and

procedural. Maintenance, in this regard engenders a combination

of physical or engineered activity or presence, with a policy

statement. This is, however, only one approach to categorising

the controls and the groupings in the environmental scanning techniques may actually be more

suited to this problem being studied.

67

Developed from a briefing provided by Mike Standbrook, of BC Hydro, to CAPIA, on 23 Sep 2009. (According

to Mike Standbrook, Royal Dutch Shell used a similar process 20+ years earlier).

The

Event

Consequence ACause 1

Consequence B

Consequence C

Consequence D

Cause 2

Cause 3

Cause 4

Consequence E

Consequence F

Proactive Controls Reactive Controls

Elimination

of causes

Reduction of

probability of

undesirable event

Reduction of

consequences

Recovery from

consequences

Undesirable Event

The

Event

Consequence ACause 1

Consequence B

Consequence C

Consequence D

Cause 2

Cause 3

Cause 4

Consequence E

Consequence F

Proactive Controls Reactive Controls

Elimination

of causes

Reduction of

probability of

undesirable event

Reduction of

consequences

Recovery from

consequences

Undesirable Event

Physical or Engineered

Maintenance

Procedural

Controls:

Physical or Engineered

Maintenance

Procedural

Controls:

of 113

Probability

Co

ns

eq

ue

nces

Pre-Policy

Post-Policy

Probability

Co

ns

eq

ue

nces

Pre-Policy

Post-Policy

Proactive controls are considered

in two ways. First, the

elimination of causes category

sets out to find ways, physical,

maintenance or policy related,

that will eliminate the existence

of the cause. (If an elimination,

or even amelioration, can be

effected, perhaps the undesirable

event will not take place.)

Second, the proactive control will

consider ways that the probability

of the undesirable event can be

reduced. The controls identified

in the first category are almost

certain to be specific to one

cause, while the controls

identified in the second category

may be associated with one or

more causes.

Reactive controls are also

considered in two ways. First

are physical or policy initiatives

that will reduce the potential

impact of the consequences or

cause the shift from a more to a

less severe consequence.

Second, are initiatives that will assist in the recovery from the consequences. It should be noted

that reactive initiatives are unlikely to be consequence specific.

Finally, if useful, analysts can consider the impact that a change in

either the physical or policy realm may provoke. This can be shown

in many ways, including using the construct in the box on the right,

where the probability of each of the consequences being manifest,

pre-change, is shown dashed, and the probability of each

consequence being manifest post-change is shown solid.

Loss of

Control of

Vehicle

Human error

Slippery road

conditions

Vehicle malfunction

Sudden road

obstruction

Cause Undesirable Event

Don’t drive

when roads

are slippery

Proactive Controls

Elimination

of causes

Reduction of

probability of

undesirable event

Gravel roads

when slippery

Construct roads

underground

Lower speed

limit

Driver

training

4x4

Capability

Periodic vehicle

check-ups

Appropriate

Tires

Walk don’t

drive policy

New vehicles

Regular vehicle

maintenance

Loss of

Control of

Vehicle

Human error

Slippery road

conditions

Vehicle malfunction

Sudden road

obstruction

Cause Undesirable Event

Don’t drive

when roads

are slippery

Proactive Controls

Elimination

of causes

Reduction of

probability of

undesirable event

Gravel roads

when slippery

Construct roads

underground

Lower speed

limit

Driver

training

4x4

Capability

Periodic vehicle

check-ups

Appropriate

Tires

Walk don’t

drive policy

New vehicles

Regular vehicle

maintenance

Reactive Controls

Reduction of

consequences

Recovery from

consequences

Undesirable Event

Loss of

Control of

Vehicle

Consequences

Minor Injuries

and/or Damage

Life

Threatening

Injury

Fatality

No Injuries or

Damage

ABS Brake

System

Air Bags

Mandatory

seat belt

use

911

Emergency

Response

OnStar

First-Aid

Supplies

on Board

Reactive Controls

Reduction of

consequences

Recovery from

consequences

Undesirable Event

Loss of

Control of

Vehicle

Consequences

Minor Injuries

and/or Damage

Life

Threatening

Injury

Fatality

No Injuries or

Damage

ABS Brake

System

Air Bags

Mandatory

seat belt

use

911

Emergency

Response

OnStar

First-Aid

Supplies

on Board

of 113

Demonstrate

What If? Analysis.68

Purpose: What If? Analysis can be a useful technique for developing the “analytic proof” that may help

clients understand how a particular event could happen, even if that event seems unlikely at the present

time. Note: The UK-developed Backcasting-Light (Quick I&W) process is very similar.69

Description: By shifting the focus from whether the event could occur to how it may happen, What If?

Analysis encourages analysts to suspend judgement about the likelihood of the event and focus more on

what developments – even unlikely ones – might enable such an outcome. The technique shifts

discussion from “Could it happen?” to:

How much does it matter?

How is it most likely to come about?

Has the probability of the event happening changed?

The technique also gives decision makers:

A better sense of what they might be able to do today to either prevent an untoward development

from occurring or to leverage an opportunity for advancing their interests.

A specific list of indicators to monitor to see if developments point to the event actually

occurring.

Process: Individuals or groups can begin a What If? Analysis by either clearly stating the conventional

analytic line and then stepping back to consider what alternative outcomes are too important to dismiss,

even if unlikely. Or, by simply pretending that what could happen has already occurred. It is important

to be precise in defining both the event and its impact; sometimes it may even be helpful to posit a

triggering event.

Once the endpoint (that is the event and its impact) has been established, analysts or teams should:

Develop a chain of argumentation – based as much on logic as evidence – to explain how this

event could have come about.

Work backwards from the event in concrete ways – specifying what must actually occur at each

stage of the scenario.

Identify one or more plausible pathways or scenarios to the unlikely event; very often more than

one will appear possible.

Generate a list of indicators or “observables” for each scenario that would point to the events

starting to play out.

Assess the level of damage or disruption for a negative scenario and how difficult it would be to

overcome.

Assess the overall impact of a positive scenario and how best it could be enabled.

Rank the scenarios in terms of which deserve the most attention by taking into consideration the

difficulty of implementation and the potential impact.

Monitor the indicators on a periodic basis.

68


Analysis, Jun 2005, pp 26-27, and the Handbook of Analytic Tools & Techniques, Pherson, 2008, p.35.

69

Quick Wins for Busy Analysts, Jul 2012, pp 32-33.

of 113

Demonstrate

Chronologies and Timelines.70

Purpose: By providing a method of displaying data over time, chronologies and timelines provide a

visual way to look for relationships and patterns in the data connecting persons, places, organisations,

phone numbers, and other activities.

Description: Chronologies list events in the order

they occurred, usually in narrative or bulleted form.

This allows the analyst to identify gaps or

unexplained time periods, consider the implications

of the absence of evidence, and draw conclusions or

make recommendations.

Timelines arrange information graphically along a

chronological spectrum. Information can be

presented visually by categorising the data and

displaying it above and below the line. This allows

the analyst to look for links between timelines of activities of persons or organisations, identify gaps or

unexplained time periods, and consider the implications of

the absence of evidence.

Chronologies and timelines are useful throughout the

analytic process for structuring and visualising data. At the

beginning of the analysis, they can be used to sort the data

by time period and to visually display time series data.

During the analytic process, newly acquired data can be

added to chronologies and timelines as events happen or

when additional information comes to light. In the final

product, chronologies and timelines are particularly useful for displaying the data for clients.

Strengths: Chronologies and

timelines help in organising and

sorting information over time or into

designated time periods. Once the

data is structured, the analyst can more

efficiently examine relationships

among the various items noted. The

analyst can quickly see the flow of the

data, identify relationships, discover

gaps and generate requirements for

additional research or investigation,

and present results to clients.

70

Developed from the Handbook of Analytic Tools & Techniques, Pherson, 2008, p.9, Graphic from the DIA’s A

Tradecraft Primer: Basic Structured Analytic Techniques, First Edition, Mar 2006, p 38

A chronology is usually a simple list of events, typically from left to right.

Who is giving the orders?

of 113

Intelligence

Analysis

Alternate

Explanations

Not

Data dumpKey

Assumptions

Evidence

BLUF

Support

inferences

Exec

Summaries

Amplify in

subsequent

Audience

read

initials

Important

points

Position in

para lead

sentences

Summary in

lead para

Para lead

sentences

Selected

Relevant

Sourced

Footnoted

Customer

need

Identify

indicators

Deal with

carefullyHighlight

Integral

Taken for

granted

Against

For

Highlight

gaps

Provide

credibility

Single

cannot

Seldom

black & white

Imperfect

knowledge

Intel

customers

Want

insight

Well

read

Intelligence

Analysis

Alternate

Explanations

Not

Data dumpKey

Assumptions

Evidence

BLUF

Support

inferences

Exec

Summaries

Amplify in

subsequent

Audience

read

initials

Important

points

Position in

para lead

sentences

Summary in

lead para

Para lead

sentences

Selected

Relevant

Sourced

Footnoted

Customer

need

Identify

indicators

Deal with

carefullyHighlight

Integral

Taken for

granted

Against

For

Highlight

gaps

Provide

credibility

Single

cannot

Seldom

black & white

Imperfect

knowledge

Intel

customers

Want

insight

Well

read

Demonstrate

Mind Mapping.

Purpose: To improve critical and creative thinking by depicting relationships visually.

Description: Analysts often start collecting information without sufficient consideration of the problem

or question they are facing. Whether what is needed is a conceptual framework or a collection plan, a

mind map can better illuminate the path ahead.

The four basic steps to

creating a mind map:

Write your topic

in the centre of a

large piece of

paper;

From the central

topic, radiate out

key words and

the most

important ideas

you have about

the topic, each on

a separate, thick

line;

Branch thinner

lines off the ends

of the appropriate

main lines, to

show supporting data (the more important the data, the closer it should be to the central topic or

idea); and

Use images and colors freely in your own special code to show people, topics, themes,

associations or dates, and to make the Mind Map more memorable.

Mind maps are well-suited to brainstorming because they don’t impose linear thought (like a hierarchical

“table of contents” approach does). One thought often spawns another, sometimes connecting things in

unexpected ways.

Mind maps can be created by hand using pencil and paper, on the web (Comapping, Mindomo,

MindMeister, and bubbl.us), or with software (ConceptDraw, FreeMind, TopicScape, iMindMap,

PersonalBrain, or MindManager).

Strengths: Mind maps are simple to create. They are intuitive, reflecting the way you think. They

visually capture, organize and communicate ideas and information effectively. Users have greater

retention and recall over traditional text-based approaches. They provide a single view of all the aspects

of a project. Lots of information can be taken in at a glance.

Note: The mind map in the box above uses the material in the Presentation of Analysis section on page

100.

http://comapping.com/

http://www.mindomo.com/

http://www.mindmeister.com/

http://bubbl.us/

http://www.conceptdraw.com/en/products/mindmap/ap-creative-critical-thinking.php

http://freemind.sourceforge.net/wiki/index.php/Main_Page

http://www.3d-scape.com/

http://www.imindmap.com/

http://www.thebrain.com/

http://www.mindjet.com/

of 113

Demonstrate

Matrices.71

Purpose: This technique produces a grid with as many cells as required to sort data and gain insight.

Description: Matrices are useful whenever there are more options or more intricate data than can be

conceptualized at one time without a visual representation. Whenever information can be reduced to a

matrix, it provides analytic insights.

Process: Matrices can be rectangular, square, or triangular depending on the purpose and number of rows

and columns required to enter the data.

Step One: Draw a matrix with sufficient columns and rows to enter the two sets of data to be

compared.

Step Two: Enter the range of data or decision criteria along the horizontal axis (first column) and

along the vertical axis (first row).

Step Three: In the grid squares in between, note the relationships or lack thereof in the cell at the

intersection between the two associated data points.

Step Four: Review the hypotheses developed for the issue in light of the relationships shown in

the matrix and, if appropriate, develop a new hypothesis(es) based on the insight gained from the

matrix.

Strengths: Matrices are

exceptionally useful in isolating

critical data when there is an

abundant amount of overall

information relevant to an issue.

When used to review data related to

options, such as the analysis of

competing hypotheses, it enables

analytic focus on each option,

improving comparison. Matrices allow elements of a problem to be separated and categorized by type, for

comparison of different types of information or of pieces of the same type of information. Matrices also

allow analysts to identify patterns or

correlations within the information.

Weaknesses: The two-dimensional

design of matrices limits their use

for collating data on complex issues.

Leaving out pertinent data easily

oversimplifies an issue.

71


pp 41-42. Second graphic – Adversarial Options Matrix from DIA CTSA Training Package, 2008.

Options Reasons for

Adopting

Reasons to

Reject Adoption

Intended

Results of Adopting

Implications of

Adopting

Indicators

that Option Adopted

of 113

Demonstrate

Decision Trees.

Purpose: To establish chains of decisions and/or events which illustrate a comprehensive range of

possible future actions.

Description: Decision trees chart the range of options for a given decision point, give estimates of value

or probability for each option, and show possible outcomes.

Decision trees are useful to aid our

own decision making by explicitly

comparing options, or to create a

model of our subjects’ decision

making and possible actions.

A decision tree illustrates a

comprehensive set of possible

decisions and the possible outcomes

of those decisions. It can be used to

assess the probability of any given

sequence of decisions.

A decision tree is structurally

similar to critical path analysis and to program evaluation and review technique (PERT) charts. However,

both these techniques show only the activities and connections which need to be undertaken to complete a

complex task. A timeline analysis as done in support of a criminal investigation is essentially a decision

tree drawn after the fact, showing only the

paths taken.

Analysts producing a decision tree to model

a subject’s decision making need to have a

rich understanding of the subjective and

objective worlds of the subject(s), including:

cultural norms and values, motives,

operating doctrine, operational environment,

sensitivity to risk, and other relevant factors.

Conditional Probabilities: Decision trees

can be particularly useful for analysts in

determining the conditional probabilities of

an event – as shown in the box on the right.

Strengths: Decision trees are simple to understand and interpret. A decision tree can be generated by a

group through brain-storming, and can also be posted for addition & comment over a period of time.

Both approaches can increase the completeness of the set of possible options and consequences.

Weaknesses: Relies on the accuracy of data used, completeness of the range of options assessed, and on

the validity of the qualitative probabilities/values estimated for each option. Also, a detailed decision tree

can present the misleading impression that all possible options and/or outcomes have been considered.

Decision points are shown as squares,

and circles represent possible outcomes.

of 113

Demonstrate

High Impact - Low Probability.72 Purpose: High Impact-Low Probability is a communication technique designed to convince a decision

maker that a seemingly unlikely event, that would have major consequences, might actually occur.

Description: High Impact-Low Probability analysis should be used when clients or intelligence staffs

need to be alerted to the potential that a seemingly long-shot development poses a credible threat.

Process: If the considered opinion is that an event is unlikely:

Clearly describe the unlikely event.

Define the high impact outcome of the event occurring as precisely as possible. Consider both

the actual event and the secondary impacts.

Identify information or evidence that suggests that the unlikely event is already actually

occurring.

Postulate additional triggers that would propel events in this unlikely direction, or factors that

would greatly accelerate timetables.

Develop one or more plausible pathways that would explain how this unlikely event could unfold.

Focus on the specifics of what must happen at each stage of the process.

Generate a list of indicators or signposts that would help identify that events were beginning to

unfold in this way.

Identify factors that could deflect this bad outcome or encourage a positive one (an opportunity

analysis).

Once the lists of indicators are developed, analysts must periodically review and update them as

necessary.

Strengths: High Impact-Low Probability analysis counters the natural tendency to focus attention on only

the most probable scenarios. This format also allows analysts to explore the consequences of an event –

particularly one not deemed likely by conventional wisdom – without having to challenge the main-line

judgement or to argue with others about how likely an event is to happen.

Mapping out the course of an unlikely, yet plausible, event can uncover hidden relationships between key

factors and assumptions; it also can alert analysts to oversights in the mainstream analytic line.73

Convincing others, particularly decision makers concerned about putting out today’s fires, that they

should devote time to considering a remote possibility for which there is currently weak evidence

requires: an authoritative or credible author; plausible arguments with compelling scenarios and open-

minded readers who are receptive to unconventional ideas. Absent some or all of those criteria, the High

Impact-Low Probability technique provides a process that, once the reader’s attention has been captured

by the clear description of the event and its consequent impacts, will lead the reader through a logical

progression designed to cater for and overcome the cognitive hurdles that the human mind is prone to

erecting.

72

Developed from the Handbook of Analytic Tools & Techniques, Pherson, 2008, p.36 73

CIA A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis, Jun 2005, p 24

of 113

Review Pre-Publication

Dialectic Inquiry.74 Purpose: Dialectical Inquiry challenges and disputes the assumptions and evidence used to frame and

support an existing analytic position to arrive at plausible alternative explanations through discussion.

Description: Dialectic Inquiry, like Devil’s Advocacy, uses intra-group conflict as the basis for solving

problems. However, it goes beyond merely finding flaws. Through constructive discussion, it attempts to

produce plausible alternative analysis. This technique is a valuable way to counter mindsets and bias

through the critical, but collegial, examination of existing analytic positions.

Process: Typically the dialectic process begins with a

written product, perhaps even a draft, where the

analytic position is identified and supporting evidence

and assumptions are clearly stated.

A team of analysts then examines the product for

evidence and assumptions, trying to imagine plausible

alternative interpretations of the assumptions and

evidence used. The team then seeks additional,

unused, evidence or assumptions that may contradict the stated analytic position and produce an

alternative interpretation.

Throughout this process, the key information, assumptions and their alternative interpretations are

examined, exposing any flaws. After discussing and reconciling any disagreements in interpretation, the

group agrees on which information and intelligence has

diagnostic value, and which assumptions are still valid. Finally,

the team may, if appropriate amend the initial analytic position

based on surviving information, intelligence and assumptions.

Strengths: By attacking and challenging assumptions and

evidence, the dialectic inquiry technique counters bias and helps

with prediction. If done thoroughly, only the best solutions will

survive critical examination and all flaws in the argument will

be detected. The technique maximises the evaluation of sources and information and enables confidence

levels to be expressed more accurately.

Weaknesses: Dialectic examination does not guarantee a better assessment. There is always the

possibility that the team will bring bias and mindsets to the table. Analysts need to be able to step outside

of their analytic comfort zone and to criticize their existing assumptions, biases and mindsets, and to

consider unused assumptions and evidence.

The technique stresses critical examination over group harmony.

Lastly, Dialectic Inquiry must be done by a team and it can take a lot time - particularly if the new

analytic position is subjected to further dialectic examination as purists would suggest is required.

74

Developed from UK DIS material.

During Dialectic Inquiry, lists are

produced of elements that could be used

against the initial analytic position:

Evidence Used

Assumptions Used

Information or Intelligence Not Used

Assumptions Not Considered

Throughout the process, ask:

If this evidence or assumption

turns out to be false, what is the

impact on the analysis?

of 113

Review Pre-Publication

Pre-Mortem Assessment.75 Purpose: The Pre-Mortem Assessment is a systematic assessment of how a key analytic judgement,

decision, or plan of action could go spectacularly wrong. It is, however, conducted prior to finalising an

analytic judgement or decision. The Pre-Mortem Assessment asks an analyst, or team of analysts, to

imagine that a report has gone forward, and now, several months or years later, it is learned that the

analytic judgement was flat wrong or the project failed spectacularly. The objective, now, is to figure out

how it could have failed.

Process: Analysts should begin by employing a structured brainstorming, or similar technique, to elicit

plausible explanations for the unexpected outcome. They should then develop a list of indicators that

would provide early warning that events are not proceeding as anticipated. Finally, a comprehensive

assessment of all possible sources of analytic error can be assembled by considering:

Contradictory or Anomalous Evidence – try to imagine how rejected or ignored data might be a

key clue to an alternative hypothesis. Is there any evidence that you rejected as unimportant

because it didn’t appear relevant, or you didn’t believe it?

Absence of Information – are there major information gaps? Did the absence of information

influence how the team arrived at its conclusion? Were there unusual information gaps?

Key Assumptions – could any be unsubstantiated, or need to be better caveated. What is the

impact of an invalid assumption? What would the impact be on the bottom line?

Credibility of Critical Evidence – if any evidence, or stream of reporting, turns out to be

incorrect, what impact will this have on the analysis? Is key evidence dated? Could sources turn

out to be unreliable? How credible was the critical evidence? Was a single source, INT, or

stream of information relied upon?

Deception – does anyone have motive,

opportunity, or means to deceive you or

provide misleading information? Is

there a past history of deception?

Finally, the possibility of common analytic

pitfalls should be considered. Were alternative

hypotheses considered? Might a change in the

broad environment (technological change,

globalisation, environmental change, etc.) have

an impact on the analysis? Etc.

Change of Frame: To symbolise the different role analysts are playing in this process, meetings should

be scheduled exclusively for this purpose, be led by a different person, and preferably should be held at a

different location. An experienced facilitator may be helpful.

This change in the frame of reference changes the group dynamics. The critical perspective will always

generate more critical ideas. Team members who may have previously suppressed questions or doubts

because they lacked confidence or wanted to be good team players are now empowered to express those

divergent thoughts. If this change in perspective is handled well, each team member will know that they

win points with their colleagues for being critical of the previous judgement, not for supporting it.

75

Developed from the Handbook of Analytic Tools & Techniques, Pherson, 2008, pp.31-32, and a draft version of

Richards J. Heuer, Jr. and Randolph H Person’s Structured Analytic Techniques for Intelligence Analysts, Mar

2009.

What if my main conclusion or key

judgement turns out to be flat wrong?

Did I ignore

contradictory

evidence?

What should

have the absence

of evidence

told me?

Were my

key assumptions

valid?

Did

deception go

undetected?

How reliable

was my key

evidence?

What if my main conclusion or key

judgement turns out to be flat wrong?

Did I ignore

contradictory

evidence?

What should

have the absence

of evidence

told me?

Were my

key assumptions

valid?

Did

deception go

undetected?

How reliable

was my key

evidence?

Did I ignore

contradictory

evidence?

What should

have the absence

of evidence

told me?

Were my

key assumptions

valid?

Did

deception go

undetected?

How reliable

was my key

evidence?

of 113

Matching the Techniques to the Tasks76

For the inexperienced analyst, the problem is often one of not knowing which structured analytic

technique to choose for the problem at hand. The following series of questions are intended to

encapsulate as much of the breadth of analytic issues as possible, and for each, to provide

guidance as to which technique would be appropriate. This is not the definitive selection

process, indeed expertise and familiarity with the subject issue will often provide better

guidance, but it is a place to start.

What is it you want to accomplish?

Do you want to define a project or issue?

o Delphi Method

o Environmental Scanning

o Nominal Group Technique

o Reframing the Question

o Structured Brainstorming

Are you just getting started, wanting to generate a list of important factors, criteria or

drivers?


o Outside-In Thinking

o Mind Mapping


Do you have a list, and want to prioritize it?

o Weighted Rankings

o Structured Comparison

Do you want to examine or visualise relationships between factors, indicators, players,

etc.?

o Chronologies and Timelines

o Cross-Impact Matrix

o Decision Trees

o Force Field Analysis

o Link Charts

o Mind Mapping

o Social Network Analysis


Do you want to make sense of a recent event, or assess the likely outcome of an evolving

situation?

o Alternative Futures Analysis

76

Developed from a draft version of Richards J. Heuer, Jr. and Randolph H. Phearson’s Structured Analytic

Techniques for intelligence Analysts, Mar 2009. (Published by CQ Press, 2011)

of 113

o Hypothesis Generator


o Quadrant Hypothesis Generation Technique

Do you want to monitor a situation or issue and avoid being surprised?


o Bow-Tie Technique

o High Impact - Low Probability

o Indicators of Change

o Indicators Validator

Do you want to generate and test hypotheses?

o Analysis of Competing Hypotheses

o Hypotheses Generator

o Hypotheses Review

o Quadrant Hypothesis Generation Technique

Are you trying to foresee the future?


o Cone of Plausibility

o Counterfactual Reasoning

o Delphi Method


Are you concerned about the effect of assumptions, mindsets and biases in your

analysis?

o Analysis of Competing Hypotheses

o Data Diagnostic

o Dialectic Inquiry

o Delphi Method

o Devil’s Advocacy

o Key Assumptions Check

o Pre-Mortem Assessment

Do you want to see things from other perspectives?

o Cross-Impact Matrix

o Delphi Method

o Devil’s Advocacy

o Hypothesis Review Technique


o Pre-Mortem Assessment

o Red Cell

o Red Team

o Reframing the Question

o Team A / Team B

of 113

Do you need to create a visualisation of an issue?

o Chronologies and Timelines


o Counterfactual Reasoning

o Decision Trees

o Force Field Analysis

o Link Charts

o Matrices

o Mind Mapping


Do you need to contribute to the warning process?



o High Impact - Low Probability

o Indicators of Change

o Indicators Validator


o What If? Analysis

of 113

Write Clear, Concise, and Client Focused Reports

CFINTCOM Analytic Product Standards77

While recognizing that there is a wide variety of analytic products and target audiences,

intelligence analysis is best served by having all analytic elements adhere to a common set of

standards. To this end, all analytic products should:

1. Make accurate judgements and assessments. Analysts should apply expertise, logic and

reasoning to make the most accurate judgements and assessments possible given the time and

information available and known information gaps. Where products are estimative, the

analysis should identify and correctly characterize the impact of key factors affecting

outcomes or situations.

2. Be clear. Analytical products should incorporate the Bottom Line Up Front structure, be

internally consistent, and use language and syntax that convey meaning unambiguously.

Graphics and images should be readily understandable and should illustrate, support, or

summarize key evidence or analytic judgements.

3. Be insightful, timely and relevant. Analytic products should provide timely information and

insight on issues relevant to the products' intended consumers and/or provide useful context,

warning, or opportunity analysis.

4. Properly expresses confidence or uncertainties in analytic judgements. Where appropriate,

analytic products should indicate both the level of confidence in analytic judgements and

explain the basis for ascribing it. Sources of uncertainty – including information gaps and

significant contrary reporting – should be noted and linked logically and consistently to

confidence levels in judgements.

5. Properly identify assumptions. Analytic products should explicitly identify the critical

assumptions on which the analysis is based and explain the implications for the analysis if

those assumptions prove to be incorrect.

6. Consider alternative hypotheses. Where appropriate, analytic products should identify and

explain the strengths and weaknesses of alternative hypotheses, viewpoints, or outcomes in

light of both available information and information gaps. Analytic products should explain

how alternatives are linked to key assumptions and/or assess the probability of each

alternative.

7. Identify indicators. As appropriate, analytic products should identify indicators that would:

enhance or reduce confidence or prompt revision of existing judgements; signal whether

assumptions are more or less likely to be correct; or help clarify which alternative hypothesis,

viewpoint, or outcome is more likely or is becoming more likely.

77

Developed from the US ODNI’s Rating Scale for Evaluating Analytic Tradecraft Standards, (2007 ?).

of 113

8. Use structured analytic techniques. To the extent possible, analysis should incorporate

insights from the application of structured analytic technique(s) appropriate to the topic being

analyzed.

9. Properly describe the quality and reliability of evidence. Analytic products should explain

which evidence is key to analytic judgements and why. When appropriate, factors

significantly affecting the weighting that the analysis gives to available information, such as

denial and deception, access, motivation, relevance or other factors affecting the quality and

potential reliability of the information, should be included in the product.

10. Exhibit consistency of analysis over time, or highlight changes and explain rationale.

Analytic products should deliver a key message that is either consistent with previous

production on the topic from the same analytic element or, if the key analytic message has

changed, highlight the change and explain its rationale and implications.

Presentation of Analysis

Well-presented intelligence analysis saves intelligence consumers time by using the structure of

the Bottom Line Up Front (BLUF).78

Assessments should also provide supporting evidence for

any inferences being drawn, explain the influence of key assumptions, and, when appropriate,

offer alternative explanations or hypotheses.

BLUF requires that the important elements of an assessment be made obvious, first, by

summarizing them in the lead paragraph, and then by sequentially positioning them in the lead

sentences of the subsequent paragraphs. By doing this, the audience needs only to read the initial

sentences to gain an understanding of the material. Any amplifying items should, consequently,

be placed in the subsequent sentences of the paragraph. By using this structure, accurate

executive summaries can be produced by incorporating the lead sentences of each paragraph

either in whole or in part.

Assessments should always present evidence to support any inferences being drawn. Simply

providing a data dump of seemingly-relevant material is not the objective here. Rather, only

carefully selected, truly relevant, properly sourced and footnoted material should be presented.

The objective is not for the analyst to show how much he or she knows, but to provide the

material that the client needs to know. Inherent in the selection of material is a good knowledge

of the client’s needs.

Key assumptions also need to be carefully dealt with in intelligence analysis. To this end, the

assessment should highlight key elements that are being ‘taken for granted’ and upon which the

integrity of the analysis rests. Good analytic products will also identify indicators that can be

expected to show whether – or not – the stated assumptions continue to hold true. Intelligence

gaps should also be highlighted when appropriate.

Finally, alternative hypotheses or explanations can provide a level of credibility to an assessment

that a single-minded explanation cannot. Even the best analyst is unlikely to have perfect

78

Based on Communicating with Intelligence, by James S. Major, Scarecrow Press, 2008

of 113

knowledge of any intelligence issue being examined, and, while it is tempting to be categorical

in assessment, issues are seldom black and white. Further, our intelligence clients are, in the

most part, intelligent, well-read individuals who are more interested in an insightful treatment of

a subject than they are of the opinion of an unknown analyst.

Language of Uncertainty

An important aspect of any intelligence analysis is an expression of the likeliness that the

assessed event or development will or will not occur.

While the box on the left, below, suggests a definite hierarchy to the probability or likeliness

terms that an analyst may choose, the box on the right shows just how broad the interpretation of

the same terms can be. As a consequence, the development of and strict adherence to a set of

terms and definitions is strongly advised. If this is not possible, then ensure that the terms to be

used are specifically defined in a place where the reader cannot help but see them.

Improbable

Beyond aReasonable

doubt

Certain

unlikely

Likely

Groundsto suspect

Possible

Probable

Groundsto believe

0 10 20 30 40 50 60 70 80 90 100

Improbable

Beyond aReasonable

doubt

Certain

unlikelyunlikely

LikelyLikely

Groundsto suspect

Possible

Groundsto suspect

Possible

ProbableProbable

Groundsto believeGroundsto believe

0 10 20 30 40 50 60 70 80 90 100

Likely

Probable

Improbable

unlikely

Groundsto suspect

Groundsto believe

Beyond aReasonable

doubt

Possible

Certain

Likely

Probable

Improbable

unlikely

Groundsto suspect

Groundsto believe

Beyond aReasonable

doubt

Possible

Certain

of 113

Expressing Analytic Certainty79

Provide strong judgements

Intelligence analysis is all about making judgements on the basis of information at hand,

assessing the situation based on sound analytical techniques and methodologies, and

understanding the target based on observations of past patterns of activity and research.

Intelligence reporting does not usually take the form of conclusive direct evidence (if it did we

would only be collating and evaluating facts). Our senior clients recognise that assessments are

made based on the best information available at the time of the assessment. The harm of a

wrong call is likely to be less than the harm of being irrelevant through overly conservative,

qualified, or out of date/late assessments. Analysts could qualify judgements so much that they

can never be wrong—but calling everything possible adds little value for decision makers.

When we make a judgement, we should do so:

with as few qualifications as possible,

with enough supporting evidence and logic to give authority to the judgement, and

with clarification of any vulnerabilities in the assessment.

Analytic certainty is based on two distinct components: likelihood and confidence, which we

define as follows:

Likelihood—a statement that expresses the probability, that is, the chance than an event

or development will or will not happen.

Confidence—an expression of the strength of our assessment based primarily on our

depth of understanding of the issue, the persuasiveness of the reasoning (taking into

account gaps and assumptions) and the potential for deception.

Likelihood

The following figure shows the terms that we will use to express the various levels of likelihood.

These words are roughly associated with the scale of 0/10 through 5/10 to 10/10. These numbers

are not intended to express a percentage or other numerical interpretation, but are simply to

indicate likelihood. Something that is conceivable is likely to be manifest in three out of ten

instances, for example.

In instances where an exploratory analysis is being undertaken, or when we simply do not know,

we will use terms such as may, might, unable to assess and undetermined.

79

Based on DIA Tradecraft Note 01-15: Expressing Analytic Certainty, 5 Jan 2015.

of 113

Analytic confidence

Certainty is unlikely to be achieved, because of the nature of the intelligence challenge. If the

required information could be relied on, came from unimpeachable sources and was readily

available, there would not be a requirement for intelligence in the first place. The analytic

confidence level must therefore be made as clear as possible to the clients.

Analytic confidence is not an expression of an analyst’s personal belief that a judgement is

correct, nor is it a measure of the likeliness that the event will occur in the future. Rather, it is an

expression of the maximum degree of confidence that can be justifiably assigned to a judgement,

based on three factors:

Evidence – the strength of the knowledge base, to include the quality of the evidence and

our depth of understanding about the issue;

Assumptions - the number and importance of assumptions used to fill information gaps;

and

Reasoning - the strength of the logic underpinning the argument, which encompasses the

number and strength of analytic inferences as well as the rigour of the analytic

methodology applied to the product.

Each successively higher level of analytic confidence in the graphic below indicates a greater

degree of commitment.

When it needs to be stated, and that will not be very often, analytic confidence will be

characterised as high, moderate or low. If we have complete confidence in something, we would

probably be stating the obvious, and if we have no confidence, we probably will not be writing

on the subject.

of 113

When confidence is medium or low, this confidence needs to be made obvious in the narrative

(either integrated into the text or in a separate box) and the factors contributing to this lower-

than-normal confidence would need to be explored in some detail.

Footnotes

Almost every analytic product will benefit

from a thorough and consistent

identification of the sourcing information

for all significant and substantive

reporting or other information upon which

the product’s analytic judgements,

assessments, estimates, or confidence

levels depend. The documentation of

sources in footnotes or endnotes enhances

the credibility and transparency of

intelligence analysis enabling readers and

intelligence managers to better understand

the quantity and quality of information

underlying the analysis. Footnoting is

particularly useful in the pre-publication,

editorial and approval phases, and may be

crucial in a post-publication inquiry.

Endnotes or footnotes, intended to achieve this end, should contain the following five items of

information, when available:

the abbreviated classification of the material from this source that was used in the paper

(not necessarily the overall classification of the source document),

the source (agency, publication, individual, etc.),

the report number, serial number or message reference number,

the item or article title (abbreviated if necessary), and

the date.

Examples of Footnotes

Email DFAIT Williams, K, Nairobi (Cabnet), Burundi, 1

Mar 2007 (U)

Media Reports East Africa Standard (Nairobi), Kagame quits army

ahead of poll, 13 Aug 2003 (U)

BBC Monitoring, quoting Rwanda Radio, President

leaves for Visit to Belgium, 10 Mar 2004 (U)

Websites

EIU, Country Profile Nigeria, 31 Dec 2003 (U)

of 113


Reasoning, the act of forming conclusions, may be inductive, deductive, or abductive.

Deductive Reasoning.

Deductive reasoning80

is reasoning which uses deductive arguments to move from given

statements (premises) to conclusions, which must be true if the premises are true. An example of

deductive reasoning, given by Aristotle, is

All men are mortal. (major premise)

Socrates is a man. (minor premise)

Socrates is mortal. (conclusion)

Inductive Reasoning.

Induction,81

or inductive reasoning, is the process of reasoning in which the premises of an

argument are believed to support the conclusion but do not entail it; i.e. they do not ensure its

truth. Induction is a form of reasoning that makes generalizations based on individual instances.

It is used to ascribe properties or relations to types based on a number of observations or

experiences; or to formulate laws based on observations of patterns. The following are examples

of induction:

All observed crows have been black.

Therefore:

All crows are black.

This exemplifies the nature of induction: inducing the universal from the particular. However,

the conclusion is not certain. Unless we can systematically falsify the possibility of crows of

another colour, the statement (conclusion) may actually be false.

I always hang pictures on nails.

Therefore:

All pictures hang from nails.

Assuming the first statement to be true, this example is built on the certainty that "I always hang

pictures on nails" leading to the generalisation that "All pictures hang from nails". However, the

link between the premise and the inductive conclusion is weak. No reason exists to believe that

just because one person hangs pictures on nails that there are no other ways for pictures to be

hung, or that other people cannot do other things with pictures. Indeed, not all pictures are hung

from nails; moreover, not all pictures are hung.

Abductive Reasoning.

80

Wikipedia 81

Wikipedia

of 113

Abduction,82

or inference to the best explanation, is a method of reasoning in which one chooses

the hypothesis that would, if true, best explain the relevant evidence. Abductive reasoning starts

from a set of accepted facts and infers their most likely, or best, explanations. Abduction often

involves both inductive and deductive arguments. However, as the conclusion in an abductive

argument does not follow with certainty from its premises it is best thought of as a form of

inductive reasoning. What separates abduction from the other forms of reasoning is an attempt to

favour one conclusion above others, by attempting to falsify alternative explanations or by

demonstrating the likelihood of the favoured conclusion, given a set of more or less disputable

assumptions.

Fallacies

Intelligence reports need to be firmly based on irrefutable logic. Some errors in reasoning or

flawed arguments occur so often that they are called fallacies, a few of the most common of

which are:

Inadequate sampling occurs when a sample, too small to represent an adequate measure,

is used to draw a conclusion.

Post Hoc ergo Propter Hoc occurs when it is assumed that if event B occurs after event

A, then A caused B.

Personification occurs when human characteristics are ascribed to objects or concepts.

Prevalent Proof occurs when mass opinion is used as a method of verification.

Appeal to Authority occurs when the opinion of a recognised expert is automatically

seen as valid.

Ad Hominem occurs when an argument is dismissed by attacking the person making the

argument.

False Dichotomy occurs when a set of possibilities is arbitrarily reduced to only two

opposing ones.

Non Sequitur refers to an argument in which the conclusion does not follow the premise;

a logical connection is implied where none exists.

Tautology is an argument that uses circular reasoning; the conclusion is its own premise.

Pre-ATIP Processing83

Pre-ATIP processing is meant to facilitate an efficient response to ATIP requests and keep track

of the responses that have already been provided. This process is especially important when

someone other than the author will have to respond to a request. It also allows the author to

indicate those passages that are particularly sensitive while the memory is still fresh.

For each paper, a table is prepared, indicating: ATIP number and request, or anticipatory;

subject/author; and severances.

Once the table has been filled out, it can be saved as an electronic file. If necessary, a hard-copy

can be printed and attached to the full document.

82

Wikipedia 83

This process is the initiative of George Betts, Global Division IAS.

of 113

ATIP: ATI Request

A-2005-01111

Sea Slug notes

or (Anticipatory)

Subject/Author: IM -26/05 :Security Implications of Sea Slugs, 06 Dec 05;

Joe Smith

Overall Classification: Secret

Severances

Key Judgement 2 (S)

“Expansion of sea slugs pods...

Sever In Accordance With

Para 16(1) - Law Enforcement and

Investigations

Key Judgement 3 (S)

“Sea slugs are well established...”


Para 15(1) - Law Enforcement and

Investigations

Para 6, sentence beginning (S)

“Deep under the ocean .”


Para 13(1) - Information obtained in

confidence

of 113

The Two-Hour Challenge

Users of this aide memoire can be forgiven if they observe that many of the techniques outlined

are best left to those with lots of time; i.e. the guys and gals that need to deal with the pressures

of producing a product every two months and who can be found staring out the window, for

hours on end, in quiet contemplation. If the conclusion is that only the prima dona strategic

analysts can possibly benefit from the Best Practices for Intelligence Analysis, it would be

wrong. Indeed, all analysts can reap significant benefit from the processes outlined in this aide

memoire.

Let’s consider that you have been asked to:

recommend a deployment route for impending operation Z,

explain what happened in location Y last night and the impact on Canada,

recommend a course of action to deal with an outbreak of X,

assess the utility of source W, or

assess the likely reaction of V to upcoming U.

Oh, by the way, you have two hours to do this in!!

So, what do you do? Running around with your hair on fire is one option, but perhaps not the

most efficient use of your time. What say you:

Start by figuring out what the question really is. In this regard, you may want to classify

the issue as a puzzle, mystery or mess. You may also want to paraphrase, broaden,

narrow or redirect the focus, ask “Why?” or turn the question 180 degrees in order to help

identify the real issue that needs to be addressed. You almost certainly also want to

determine which factors, drivers or criteria will have an important role to play in

explaining the issue.

Once you know what the question is and which factors will be important in answering it

for your client, gather the data you need. Given the two-hour time constraint, this likely

rules out asking for re-directed satellite coverage. Indeed, you probably want to focus on

your most trusted, readily available, sources of information. Keep track of the sources,

classifications and caveats on usage – you don’t need to create work for security

infraction investigators!

Then, think critically about the sources, quality and relevance of the information you

have available. Denial and deception, propaganda and exaggerations can all have a

negative impact on your analysis.

At this point – and not before!!! – develop multiple hypotheses or explanations. If only

one comes to mind, try harder. Do a quick quadrant hypothesis generation or try to do

some inside-out thinking. Remember, the paradox of expertise suggests that the more

you know about a subject area, the more likely you are to miss something.

Now is the time to consider the use of structured analytic techniques. Not all of them

take forever. Indeed, the diagnostic techniques like the Force Field Analysis,

Chronologies and Timelines, and Weighted Rankings can be very helpful in a short

period of time. They also produce graphical representations that may be useful in

educating/informing your client.

Finally, produce your report. Keep the bottom line up front and ensure that the

confidence you have in your judgements is made crystal clear. Include footnotes.

of 113

Throughout this process, you should constantly challenge your assumptions and look for

the negative influence of bias. If you have time, ask someone in your network, or a

trusted outsider, to help.

Analysts almost certainly follow this process intuitively. The problem with the instinctive

method, however, is someday something may be omitted, and that momentary failure may result

in an assessment that gets someone inconvenienced or killed. A concerted effort to apply the

Best Practices may, on the other hand, help the analyst sleep better at night. A consistent

application of this process may also be handy if an analyst is forced to face an investigative

tribunal that is looking into an operational failure.

Judge: And how did you

come to make this

assessment?

You: As I am an internationally recognised expert in this field, I

simply drew on my experience to produce the assessment. The

short time constraint I was working under precluded any other

approach.

Or

You: Well sir, notwithstanding the short time constraints we were

working under:

- I took the time to ensure I understood both the type and

specifics of the question and the factors that would be

instrumental in answering it.

- I kept track of the data I gathered from the most

trustworthy and easily accessed sources, keeping track of

the distribution restrictions.

- I critically evaluated the source, content and relevance of

each piece of data.

- I developed multiple hypotheses.

- I utilised diagnostic techniques to help put the data in

context and produce representational graphics.

- Only then did I produce a report in which I placed the

bottom line up front and was very clear with my

confidence in the judgements I had included.

Throughout this process I was cognizant of the assumptions I was

working with and the potential for bias in my thinking. I

consulted both co-workers and my supervisor in an attempt to

mitigate the negative effects of both.

of 113

Best Practices for Intelligence Analysis Review84

The Best Practices for Intelligence Analysts (BPfIA) Review is a systematic procedure that can

be undertaken by a small team or group can use to identify weaknesses in its own analysis (or by

a manager as a guide to help check on the analytic process undertaken by a team). The goal is to

confirm, or if necessary modify, both the analysis and the group’s overall confidence in its

judgements.

In order to extract the maximum benefit from this technique, all team or group members need to

don a hypothetical black hat85

and become critics rather than supporters of their own analysis.

From this re-framed position,86

they should critically review the analysis, answering the

following questions, in an attempt to generate new ideas.

Reflect on the Problem

o Should one expect to find a single correct answer, a most likely answer with one

or more alternatives, or a number of possible explanations for future

development?

o Is the appropriate question being answered, focusing on the factors that are

relevant to the client?

o Does the analysis deal with the W5H-SW?

o Does the analysis cater to the client’s needs (policy relevant, timely and

appropriately detailed)?

Collect Information

o Was a collection plan assembled?

o Were requirements identified?

o Were indicators identified and validated?

o How many different sources (INTs) were used?

Critically Evaluate Information

o For each important/critical piece of information:

What was the reliability and credibility was assigned to the provider?

What was the quality of the information itself?

What was the relevance of the information?

Is any evidence so dated that it may no longer be valid?

And what is the likelihood that deception was involved?

o Are there major information gaps?

84

Inspired by the Structured Self-Critique Technique that was developed by Richards Heuer and Randy Pherson and

appeared in the draft Chapter 9 Structured Analytic Techniques for Intelligence Analysts, Mar 2009. 85 The Black Hat is one element of Edward de Bono's "Six Thinking Hats" approach to decision-making. Using this

approach, the White hat provides facts and information, the Red hat injects feelings and emotions, the Yellow hat

provides positive judgements, the Green hat focuses on alternatives and learning, the Blue hat injects the big picture,

and the Black hat focuses on critical judgements. 86

A frame is any cognitive structure that guides the perception and interpretation of what one sees. An individual or

group can change their frame of reference by simply changing the questions it asks or changing the perspective from

which it asks them. This technique asks team members to reverse their role from advocate to critic in order to

explore potential weaknesses in the previous analysis. Effectiveness depends largely on how fully and

enthusiastically the group embraces the imaginative or alternative role they are playing. Just going through the

motions is of little value. (Extracted from the draft Chapter 9 of Heuer and Pherson’s Structured Analytic

Techniques for Intelligence Analysts, Mar 2009.

of 113

Is this absence of information readily explainable?

o Is there any anomalous evidence that would have been important if it had been

believed or if it could have been related to the issue at hand?

o Is the issue relatively stable, or is the situation undergoing, or about to undergo,

significant change?

o If the information deals with decision-making, is the behaviour based on:

a rational actor model,

the result of bargaining between political or bureaucratic forces,

the result of standard organisational processes, or

the whim of an authoritarian leader?

o Does the team have the cultural expertise to judge the thought processes of the

culture being reported on?

Hypotheses

o What alternative hypotheses were considered?

What evidence was there for and against these (not-selected) hypotheses?

Are there reasonable alternative interpretations of the evidence that would

give greater credibility to a rejected hypothesis?

o Might a change in the broad environment (technological change, globalisation,

environmental change, etc.) have an impact on the analysis?

Assumptions, Mindsets and Biases

o What are the key assumptions inherent in this assessment?

How recent and reliable is the evidence supporting each of these

assumptions?

o Which one or two assumptions would have the greatest impact on the judgement

if they turned out to be wrong?

Collaboration

o Did the analytic team seek a broad range of diverse opinions by including analysts

from other offices, agencies, academia, or the private sector?

Structured Techniques

o What analytic techniques were used in the preparation of this assessment?

Are there other techniques that may have been more appropriate or which

may have achieved a different analytic result?

Communicate

o Is the logic in the assessment transparent? Can the logic trail be mindmapped or

otherwise demonstrated?

o Is the bottom line up front and obvious?

o Are all probability terms in keeping with the appropriate standard? If not, are the

non-standard usages made clear?

o Is the level of analytic confidence clearly stated?

Does the confidence level accurately reflect the group’s confidence in the

evidence that the analysis rests upon?

Does the confidence level accurately reflect the group’s confidence in the

analytic reasoning that informs the analysis?

Does the confidence level accurately reflect the group’s confidence that

the analysis was not compromised by deception?

of 113

After answering each of these questions, the group should take off the black hats and consider

the appropriateness of both the assessment itself and the level of confidence expressed in the

assessment. Modifications should then be made if necessary.

The change in frame of reference gives the analytic group an opportunity to change its dynamics

and perhaps generate critical new ideas. Team members that have previously suppressed

questions or doubts because they lacked confidence or wanted to be good team players are now

empowered to express those divergent thoughts. If this change in perspective is handled well (a

facilitator may be required), each team member will know that they win points with their

colleagues for being critical of the previous judgement, not for supporting it.

The success of this process, however, depends in large measure on the team members’

willingness and ability to make the transition from supporters to critics of their own ideas. Some

analysts lack the intellectual flexibility to do this well.

of 113

Analytic Rigour Map

Aide Memoire on - dln-rad.forces.gc.ca

Documents

Transcript of Aide Memoire on - dln-rad.forces.gc.ca