CIA Tradecraft Primer Structured Analytic Techniques

8/7/2019 CIA Tradecraft Primer Structured Analytic Techniques

1/45

A Tradecraft Primer:

Structured Analytic Techniquesfor Improving Intelligence Analysis

Prepared by the US Government

March 2009


2/45

This primer highlights structured analytic techniquessome widely used in the private

sector and academia, some unique to the intelligence profession. It is not a

comprehensive overview of how intelligence officers conduct analysis. Rather, the

primer highlights how structured analytic techniques can help one challenge judgments,

identify mental mindsets, stimulate creativity, and manage uncertainty. In short,

incorporating regular use of techniques such as these can enable one to structure

thinking for wrestling with difficult questions.


3/45

Introduction ................................................................................................................. 1

How To Use These Techniques .................................................................................. 5

Diagnostic Techniques................................................................................................ 7

Key Assumptions Check..................................................................................... 7

Quality of Information Check............................................................................. 10

Indicators or Signposts of Change.................................................................... 12

Analysis of Competing Hypotheses (ACH)....................................................... 14

Contrarian Techniques............................................................................................... 17

Devils Advocacy................................................................................................ 17

Team A/Team B ................................................................................................. 19

High-Impact/Low-Probability Analysis............................................................... 22

What If? Analysis............................................................................................. 24

Imaginative Thinking Techniques .............................................................................. 27

Brainstorming..................................................................................................... 27

Outside-In Thinking............................................................................................ 30

Red Team Analysis............................................................................................. 31

Alternative Futures Analysis............................................................................... 34

Strategies for Using Structured Analytic Techniques................................................ 38

Selective Bibliography ............................................................................................... 40

T A B L E O F C O N T E N T S


4/45


5/45

1

Intelligence

analysts should

be self-conscious

about their reasoningprocesses. They

should think about

how they make

judgments and reach

conclusions, not just

about the judgments

and conclusions

themselves.

Richards Heuer,The Psychology of

Intelligence Analysis

THE MIND-SET CHALLENGE

Using the analytic techniques contained

in this primer will assist analysts indealing with the perennial problemsof intelligence: the complexity ofinternational developments, incompleteand ambiguous information, and theinherent limitations of the human mind.Understanding the intentions andcapabilities of adversaries and otherforeign actors is challenging, especiallywhen either or both are concealed.Moreover, transnational threats todaypose even greater complexity, in thatthey involve multiple actorsincluding

nonstate entitiesthat can adapt andtransform themselves faster than thosewho seek to monitor and contain them.Finally, globalization has increased thediversity of outcomes when complex,interactive systems such as financial flows,regional economies or the internationalsystem as a whole are in flux.2

The first hurdle for analysts is identifyingthe relevant and diagnostic informationfrom the increasing volume of ambiguousand contradictory data that is acquiredthrough open source and clandestinemeans. Analysts must also pierce theshroud of secrecyand sometimesdeceptionthat state and nonstate actorsuse to mislead. A systematic approachthat considers a range of alternativeexplanations and outcomes offers oneway to ensure that analysts do not dismisspotentially relevant hypotheses andsupporting information resulting in missedopportunities to warn.

Cognitive and perceptual biases inhuman perception and judgment areanother important reason for analysts to

consider alternatives. As Richards Heuerand others have argued, all individualsassimilate and evaluate informationthrough the medium of mental models(sometimes also called frames ormind-sets). These are experience-based constructs of assumptions andexpectations both about the world ingeneral and more specific domains.These constructs strongly influence whatinformation analysts will acceptthat is,data that are in accordance with analystsunconscious mental models are more

likely to be perceived and rememberedthan information that is at oddswith them.

Mental models are critical to allowingindividuals to process what otherwisewould be an incomprehensible volume ofinformation. Yet, they can cause analyststo overlook, reject, or forget importantincoming or missing information that isnot in accord with their assumptions andexpectations. Seasoned analysts maybe more susceptible to these mind-setproblems as a result of their expertiseand past success in using time-testedmental models. The key risks of mind-sets are that: analysts perceive what theyexpect to perceive; once formed, they areresistant to change; new information isassimilated, sometimes erroneously, intoexisting mental models; and conflictinginformation is often dismissed or ignored.

I N T R O D U C T I O N

1 Richards J. Heuer, Jr.,The Psychology ofIntelligence Analysis (Washington: Center for theStudy of Intelligence, 1999).

2 These observations were drawn from a lengthiertreatment of cognitive bias found in the ShermanKent Centers Occasional Paper, Making Sense ofTransnational Threats, Vol. 3, No. 1, October 2004.

1


6/45

2

Common Perceptual and Cognitive Biases

Perceptual BiasesExpectations. We tend to perceive what we

expect to perceive. More (unambiguous)

information is needed to recognize an

unexpected phenomenon.

Resistance. Perceptions resist change even

in the face of new evidence.

Ambiguities. Initial exposure to ambiguous

or blurred stimuli interferes with accurate

perception, even after more and better

information becomes available.

Biases in Evaluating EvidenceConsistency. Conclusions drawn from a

small body of consistent data engender

more confidence than ones drawn from a

larger body of less consistent data.

Missing Information. It is difficult to

judge well the potential impact of missing

evidence, even if the information gap

is known.

Discredited Evidence. Even though

evidence supporting a perception may

be proved wrong, the perception may not

quickly change.

Biases in Estimating Probabilities

Availability. Probability estimates are

influenced by how easily one can imagine an

event or recall similar instances.

Anchoring. Probability estimates are

adjusted only incrementally in response to

new information or further analysis.

Overconfidence. In translating feelings of

certainty into a probability estimate, people

are often overconfident, especially if they

have considerable expertise.

Biases in Perceiving Causality

Rationality. Events are seen as part of

an orderly, causal pattern. Randomness,

accident and error tend to be rejected

as explanations for observed events. For

example, the extent to which other people

or countries pursue a coherent, rational,

goal-maximizing policy is overestimated.

Attribution. Behavior of others is attributed

to some fixed nature of the person or

country, while our own behavior is attributed

to the situation in which we find ourselves.

Intelligence analysts must actively reviewthe accuracy of their mind-sets by applyingstructured analytic techniques that willmake those mental models more explicitand expose their key assumptions.The techniques found in this primer aredesigned to assist in this regard by:

Instilling more structure into the

intelligence analysis.

Making analytic arguments moretransparent by articulating them andchallenging key assumptions.

Stimulating more creative, out-of-the-

box thinking and examining alternativeoutcomes, even those with lowprobability, to see if available data mightsupport these outcomes.

Identifying indicators of change (or

signposts) that can reduce the chancesof surprise.

Incorporating findings derived from thesetechniques into our intelligence productsalso serves the policymaker by:

Highlighting potential changes that

would alter key assessments orpredictions.

Identifying key assumptions,

uncertainties, intelligence gaps anddisagreements that might illuminaterisks and costs associated with policychoices.

Exploring alternative outcomes for which

policy actions might be necessary.


7/45

3

Strategic Assumptions That Were Not Challenged

1941 World War II

Japan would avoid all-out war because it recognized US military superiority.

Given that US superiority would only increase, Japan might view a first strike as the only

way to knock America out of the war.

1950s Korean War

China would not cross the Yalu River in support of the North Korean government.

Red China could make good on its threats to counter US aggression against the North.

1962 Cuban Missile Crisis

The Soviet Union would not introduce offensive nuclear weapons into Cuba.

The Kremlin could miscalculate and believe it could create a fait accompli that a young

US President would not be prepared to reverse.

1973 Yom Kippur War

Arabs knew they could not win because they had failed to cooperate in the past andstill lacked sufficient air defenses to counter Israeli airpower.

A surprise Arab attack, even if repelled, could wound Israel psychologically and promptinternational calls for cease-fires and diplomatic negotiations.

1989 German Unification

East Germany could not unify with the West Germany against the wishes of theSoviet Union.

The Soviet Unionunder Gorbachevmight not be prepared to intervene militarily in

Eastern Europe as it had in the past.

1998 Indian Nuclear Test

Conducting a nuclear test risked international condemnation and US sanctions andwould threaten a newly elected coalition government.

A successful and surprise nuclear test could boost Indian nationalist pride and solidifypublic support for a shaky coalition government.

2003 Iraqs WMD Programs

Saddam failed to cooperate with UN inspectors because he was continuing to developweapons of mass destruction.

If Iraqi authorities had destroyed their WMD stocks and abandoned their programs,they might refuse to fully acknowledge this to the UN to maintain Iraqs regional status,

deterrence, and internal regime stability.


8/45

4


9/45

5

The analytic techniques in this primerare designed to help individual analysts,as well as teams, explore and challenge

their analytical arguments and mind-sets. Some techniques are fairly simpleto understand and employsuch asBrainstorming and Devils Advocacy.Others are more complex and demand agreater degree of analytical sophistication,resource commitment, and time. All thetechniques are included because theyhave helped other analysts avoid rigidways of thinking or assisted them inexploring new outcomes or implicationsof an intelligence problem.

The techniques are grouped by theirpurpose: diagnostic techniques areprimarily aimed at making analyticarguments, assumptions, or intelligencegaps more transparent; contrariantechniques explicitly challenge currentthinking; andimaginative thinking

H O W T O U S E T H E S E T E C H N I Q U E S

techniques aim at developing newinsights, different perspectives and/ordevelop alternative outcomes. In fact,

many of the techniques will do somecombination of these functions. However,analysts will want to select the tool thatbest accomplishes the specific taskthey set out for themselves. Althoughapplication of these techniques aloneis no guarantee of analytic precision oraccuracy of judgments, it does improvethe sophistication and credibility ofintelligence assessments as well as theirusefulness to policymakers. As RichardsHeuer notes in his own work on cognitivebias, analysis can be improved.3

3 Heuer, Psychology of Intelligence Analysis, p. 184.


10/45


11/45

7

KEY ASSUMPTIONS CHECK

List and review the key working

assumptions on which fundamentaljudgments rest.

WHEN TO USE

A Key Assumptions Check is most usefulat the beginning of an analytic project. Anindividual analyst or a team can spendan hour or two articulating and reviewingthe key assumptions. Recheckingassumptions also can be valuable atany time prior to finalizing judgments,to insure that the assessment does notrest on flawed premises. Identifying

hidden assumptions can be one of themost difficult challenges an analystfaces, as they are ideas heldoftenunconsciouslyto be true and, therefore,are seldom examined and almost neverchallenged.

A key assumption is any hypothesis thatanalysts have accepted to be true andwhich forms the basis of the assessment.For example, military analysis may focusexclusively on analyzing key technicaland military variables (sometimes called

factors) of a military force and assumethat these forces will be operated in aparticular environment (desert, openplains, arctic conditions, etc.). Postulatingother conditions or assumptions,however, could dramatically impact theassessment. Historically, US analysis ofSoviet-Warsaw Pact operations againstNATO had to assume a level of non-Soviet Warsaw Pact reliability (e.g., wouldthese forces actually fight?). In this case,there was high uncertainty and dependingon what level of reliability one assumed,the analyst could arrive at very different

conclusions about a potential Sovietoffensive operation. Or when economistsassess the prospects for foreign economic

reforms, they may consciously, or not,assume a degree of political stability inthose countries or the region that mayor may not exist in the future. Likewise,political analysts reviewing a developingcountrys domestic stability mightunconsciously assume stable oil prices,when this key determinant of economicperformance and underlying social peacemight fluctuate. All of these exampleshighlight the fact that analysts often relyon stated and unstated assumptions toconduct their analysis. The goal is not toundermine or abandon key assumptions;rather, it is to make them explicit andidentify what information or developmentswould demand rethinking them.

VALUE ADDED

Explicitly identifying working assumptionsduring an analytic project helps:

Explain the logic of the analytic

argument and expose faulty logic.

Understand the key factors that shape

an issue.

Stimulate thinking about an issue.

Uncover hidden relationships and links

between key factors.

Identify developments that would cause

you to abandon an assumption.

Prepare analysts for changed

circumstances that could surprise them.

D I A G N O S T I C T E C H N I Q U E S


12/45

8

Key Assumptions Check: The 2002 DC Sniper Case

The outbreak of sniper shootings in the Washington, DC area in the fall of 2002 providesa good example of how this technique could have been applied. After the initial flurry ofshootings, the operating assumption that quickly emerged was that the shootings were thework of a single, white male who had some military training and was driving a white van.If law enforcement officials had conducted a Key Assumptions Check, they could havebroken this statement into its key components and assessed the validity of each statementas follows:

Key Assumption Assessment

The sniper is a male.

The sniper is acting alone.

The sniper is white.

The sniper has militarytraining/experience.

The sniper is driving awhite van.

Highly likely (but not certain) given past precedent withserial killers. We are taking little risk by not looking for afemale.

Highly likely (but not certain) given past precedents.

Likely, but not as certain, given past precedents. We wouldbe taking some risk if we rule out nonwhites as suspects.

Possible, but not sufficient reason to exclude fromconsideration potential suspects who have not had anymilitary training.

Possible because you have a credible eyewitness accountbut worthy of continuing scrutiny given the number ofwhite vans in the area (more than 70,000 registered in theMaryland suburbs of Metropolitan Washington, DC) andthat different kinds of vehicles are being described.

A Key Assumptions Checkcould have allowed law enforcement ofcials to:

Avoid jumping to conclusions (the sniper is white, has military training, and is driving

a white van) that did not hold up under closer scrutiny. By explicitly examining eachassumption, officials could have avoided prematurely narrowing down the potential pool

of suspects to a group that did not include the actual perpetrator. Similarly, they might

have been more cautious about accepting that the sniper was driving a white van.

Be receptive to new leads and citizen tips, such as eyewitness reports that the sniper ed

the scene driving a specific model Chevrolet.

More seriously consider evidence that subsequently became available, which

contradicted a key assumption. If officials had stated explicitly that they were assuming

the sniper was acting alone, they might have been sensitive to new information that

contradicted that key assumption. Often this type of information gets lost in the noise if

the analyst has not already thought about what key assumptions he or she is making.


13/45

9

THE METHOD

Checking for key assumptions requiresanalysts to consider how their analysis

depends on the validity of certainpremises, which they do not routinelyquestion or believe to be in doubt. A four-step process will help analysts:

1. Review what the current analytic line onthis issue appears to be; write it downfor all to see.

2. Articulate all the premises, both statedand unstated in finished intelligence,which are accepted as true for thisanalytic line to be valid.

3. Challenge each assumption, askingwhy it must be true and whether itremains valid under all conditions.

4. Refine the list of key assumptions tocontain only those that must be trueto sustain your analytic line; considerunder what conditions or in the face ofwhat information these assumptionsmight not hold.

QUESTIONS TO ASK DURING THISPROCESS INCLUDE:

How much condence exists that this

assumption is correct?

What explains the degree of condence

in the assumption?

What circumstances or information

might undermine this assumption?

Is a key assumption more likely a key

uncertainty or key factor?

Could the assumption have been true in

the past but less so now?

If the assumption proves to be wrong,

would it significantly alter the analyticline? How?

Has this process identied new factors

that need further analysis?


14/45

10

QUALITY OF INFORMATION CHECK

Evaluates completeness and soundness

of available information sources.

WHEN TO USE

Weighing the validity of sources is a keyfeature of any critical thinking. Moreover,establishing how much confidenceone puts in analytic judgments shouldultimately rest on how accurate andreliable the information base is. Hence,checking the quality of information usedin intelligence analysis is an ongoing,continuous process. Having multiplesources on an issue is not a substitute

for having good information that hasbeen thoroughly examined. Analystsshould perform periodic checks ofthe information base for their analyticjudgments. Otherwise, important analyticjudgments can become anchored to weakinformation, and any caveats attachedto those judgments in the past can beforgotten or ignored over time.

If a major analytic assessment is planned,analysts should individually or collectivelyreview the quality of their information

and refresh their understanding of thestrengths and weaknesses of pastreporting on which an analytic line rests.Without understanding the context andconditions under which critical informationhas been provided, it will be difficultfor analysts to assess the informationsvalidity and establish a confidence level inan intelligence assessment.

VALUE ADDED

A thorough review of informationsources provides analystsas well

as policymakerswith an accurateassessment of what we know and whatwe do not know. It is also an opportunityto confirm that sources have been citedaccurately. In the case of HUMINT, this willrequire extensive review of the sourcesbackground information and access as wellas his or her motivation for providing theinformation. Similarly, reviewing technicalsourcing can sometimes reveal inadvertenterrors in processing, translation, orinterpretation that otherwise might havegone unnoticed.

In addition, a quality of information checkcan be valuable to both collectors andpolicymakers:

It can help to detect possible deception

and denial strategies by an adversary.

It can identify key intelligence gaps and

new requirements for collectors.

It can assist policymakers in

understanding how much confidenceanalysts are placing on analyticjudgments.


15/45

11

THE METHOD

An analyst or a team might begina quality of information check by

developing a database in whichinformation is stored according to sourcetype and date, with additional notationsindicating strengths or weaknesses inthose sources.4 Ideally, analysts wouldhave a retrieval and search capability onthe database, so that periodic reviewsare less labor intensive and result in amore complete review of all sourcesused in past analysis. For the informationreview to be fully effective, analysts willneed as much background informationon sources as is feasible. Knowing the

circumstances in which reporting wasobtained is often critical to understandingits validity. With the data in hand, analystscan then:

Review systematically all sources for

accuracy.

Identify information sources that appear

most critical or compelling.

Check for sufcient and strongcorroboration of critical reporting.

Reexamine previously dismissed

information in light of new facts orcircumstances that cast it in adifferent light.

Ensure that any recalled reporting is

identified and properly flagged for otheranalysts; analysis based on recalledreporting should also be reviewed todetermine if the reporting was essential

to the judgments made.

Consider whether ambiguous

information has been interpreted andcaveated properly.

Indicate a level of condence that

analysts can place in sources, whichare likely to figure in future analyticassessments.

Quality of Information Problem on Iraq

. . . Analysts community wide are unable to make fully informed judgments on the

information they received, relying instead on nonspecific source lines to reach their

assessments. Moreover, relevant operational data is nearly always withheld from analysts,

putting them at a further analytical disadvantage . . .

Senate Select Committee on Intelligence, Report on the US Intelligence Communitys

Prewar Intelligence Assessments on Iraq

Analytic errors included over-reliance on a single, ambiguous source, [and, in addition

to collection shortfalls] failure of analysts to understand fully the limitations of technical

collection . . .

The Intelligence Community relied too heavily on ambiguous imagery indicators . . .

Commission on the Intelligence Capabilities of the United States Regarding

Weapons of Mass Destruction.

4Analysis of Competing Hypotheses (ACH),discussed later in the primer, is a useful techniquefor exploring the possibility that deception couldexplain the absence of evidence.


16/45

12

INDICATORS OR SIGNPOSTS OF

CHANGE

Periodically review a list of observableevents or trends to track events, monitor

targets, spot emerging trends, and warnof unanticipated change.

WHEN TO USE

An analyst or team can create anindicators or signposts list of observableevents that one would expect to see if apostulated situation is developing; e.g.,economic reform, military modernization,political instability, or democratization.Constructing the list might require only a

few hours or as much as several days toidentify the critical variables associatedwith the targeted issue. The techniquecan be used whenever an analyst needsto track an event over time to monitorand evaluate changes. However, it canalso be a very powerful aid in supportingother structured methods explained laterin this primer. In those instances, analystswould be watching for mounting evidenceto support a particular hypothesis, low-probability event, or scenario.

When there are sharply divided viewson an issue, an indicators or signpostslist can also depersonalize theargument by shifting analytic attentionto a more objective set of criteria. Usingan indicators list can clarify substantivedisagreements, once all sides agreeon the set of objective criteria used tomeasure the topic under study.

VALUE ADDED

By providing an objective baseline fortracking events or targets, indicators

instill rigor into the analytic processand enhance the credibility of analyticjudgments. An indicators list included

in a finished product also allows thepolicymaker to track developments andbuilds a more concrete case for the analytic

judgments. By laying out a list of criticalvariables, analysts also will be generatinghypotheses regarding why they expectto see the presence of such factors. In sodoing, analysts make the analytic line muchmore transparent and available for scrutinyby others.

THE METHOD

Whether used alone, or in combination withother structured analysis, the process is thesame:

Identify a set of competing hypotheses orscenarios.

Create separate lists of potential

activities, statements, or events expectedfor each hypothesis or scenario.

Regularly review and update the

indicators lists to see which are changing.

Identify the most likely or most correct

hypotheses or scenarios, based on thenumber of changed indicators that are

observed.

Developing two lists of indicators for eachhypothesis or scenario may prove usefulto distinguish between indicators that adevelopment is or is not emerging. This isparticularly useful in a What If? Analysis,when it is important to make a case that acertain event is unlikely to happen.


17/45

13

Tracking the Potential for Political Instability in a Foreign Country

Topics Indicators 1999 2000

I II III IV I II III

Government

Capacity

Quality of leadership/organizational capabilities

Responsiveness to popular demands

Ability to deliver basic goods and services

Internal Security Capabilities

Effectiveness of civil/criminal justice systems

Legitimacy of

Regime

Breadth and depth of political participation

Perceived level of corruption

Human rights violations

Weakness of civil society

Pervasiveness of transnational criminal organizations

External support for government

Opposition

Activity

Ethnic/religious discontent

Military discontent with civilian government

Popular demonstrations/strikes/riots

Insurgent/separatist/terrorist group activity

External support for opposition

Threat of conflict with or in neighboring state

Economic

Factors

Weakness of domestic economy/unemployment/inflation

Degree of income disparity

Capital flight

Decreased access to foreign funds

Reduced trade openness

Environmental

Issues

Extent of environmental degradation

Food/energy shortages

Ability to respond to natural disasters

Presence of Trigger Mechanisms ( if present)

Contested elections

Unpopular changes in food/energy prices

Sudden imposition of unpopular policies

Coup plotting

Government mismanagement of natural disaster or nationalemergency

Death of key figure

Serious concernSubstantial concern

Moderate concern

Low concern

Negligible concern

Tracking the Potential for Political Instability in an Indicators Matrix .Analysts tracked the potential forregime change in 2000 and identified a list of indicators, to which they posed the question, is this occurring or not?

Analysts also went further and developed a list of trigger mechanisms that might bring about a political shift.


18/45

14

ANALYSIS OF COMPETING

HYPOTHESES (ACH)

Identification of alternative explanations(hypotheses) and evaluation of all

evidence that will disconfirm rather thanconfirm hypotheses.

WHEN TO USE

Analysis of Competing Hypotheses(ACH) has proved to be a highly effectivetechnique when there is a large amountof data to absorb and evaluate. Whilea single analyst can use ACH, it ismost effective with a small team thatcan challenge each others evaluation

of the evidence. Developing a matrixof hypotheses and loading alreadycollected information into the matrix canbe accomplished in a day or less. If thedata must be reassembled, the initialphases of the ACH process may requireadditional time. Sometimes a facilitator orsomeone familiar with the technique canlead new analysts through this processfor the first time.

ACH is particularly appropriate forcontroversial issues when analysts want

to develop a clear record that showswhat theories they have considered andhow they arrived at their judgments.Developing the ACH matrix allowsother analysts (or even policymakers)to review their analysis and identifyareas of agreement and disagreement.Evidence can also be examined moresystematically, and analysts have foundthat this makes the technique ideal forconsidering the possibility of deceptionand denial.

VALUE ADDEDACH helps analysts overcome threecommon mistakes that can lead toinaccurate forecasts:

Analysts often are susceptible to being

unduly influenced by a first impression,based on incomplete data, an existinganalytic line, or a single explanation thatseems to fit well enough.

Analysts seldom generate a full set

of explanations or hypotheses at theoutset of a project.

Analysts often rely on evidence to

support their preferred hypothesis,but which also is consistent with otherexplanations.

In essence, ACH helps analysts to avoidpicking the first solution that seemssatisfactory instead of going through allthe possibilities to arrive at the very bestsolution.

THE METHOD

ACH demands that analysts explicitlyidentify all the reasonable alternativehypotheses, then array the evidenceagainst each hypothesisrather thanevaluating the plausibility of eachhypothesis one at a time. To create a levelplaying field, the process must:

Ensure that all the information and

argumentation is evaluated and givenequal treatment or weight whenconsidering each hypothesis.

Prevent the analyst from prematureclosure on a particular explanation orhypothesis.

Protect the analyst against innate

tendencies to ignore or discountinformation that does not fitcomfortably with the preferredexplanation at the time.

To accomplish this, the process shouldfollow these steps:

Brainstorm among analysts with

different perspectives to identify allpossible hypotheses.

List all signicant evidence and

arguments relevant to all thehypotheses.


19/45

15

Analyze how sensitive the ACH results

are to a few critical items of evidence;should those pieces prove to be wrong,

misleading, or subject to deception, howwould it impact an explanations validity?

Ask what evidence is not being seen but

would be expected for a given hypothesisto be true. Is denial and deception apossibility?

Report all the conclusions, including the

weaker hypotheses that should still bemonitored as new information becomesavailable.

Establish the relative likelihood for thehypotheses and report all the conclusions,including the weaker hypothesesthat should still be monitored as newinformation becomes available.

Identify and monitor indicators that would

be both consistent and inconsistent withthe full set of hypotheses. In the lattercase, explore what could account forinconsistent data.

Prepare a matrix with hypotheses

across the top and each piece ofevidence on the side. Determine

whether each piece of evidenceis consistent, inconsistent, or notapplicable to each hypothesis.5

Rene the matrix and reconsider the

hypothesesin some cases, analystswill need to add new hypotheses andre-examine the information available.

Focus on disproving hypotheses rather

than proving one. Talley the piecesof evidence that are inconsistent andconsistent with each hypothesis to see

which explanations are the weakest andstrongest.

5 The diagnostic value of the evidence will emergeas analysts determine whether a piece of evidenceis found to be consistent with only one hypothesis,or could support more than one or indeed allhypotheses. In the latter case, the evidence canbe judged as unimportant to determining whichhypothesis is more likely correct.

Terrorism in Tokyo From Aum Shinrikyo

Weight H: 1 H: 4 H: 2 H: 3

Kooky

Cult

Terrorist

Group

Political

Movement

Criminal

Group

Inconsistency Score -1.0 -1.0 -2.0 -3.0

E3 Attacks on Journalists MEDIUM I N I I

E2 Religious Affiliation MEDIUM C I I I

E4 Established Party MEDIUM N N C I

E1 Blind Leader Mastsumoto MEDIUM C C C C

ACH Matrix of Terrorism in Tokyo.In March 1995, a largely unknown group attacked the Tokyo subways byusing a highly lethal nerve agent known as sarin. ACH provides a mechanism to carefully examine all the evidence and

possible explanations for understanding what type of group could have been responsible. In simplified form, the above

matrix arrays each piece of evidence on the vertical axis and then evaluates each in terms of the items consistencywith four possible explanations for the terrorist attack in Tokyo (horizontal axis). Analysts rate a piece of evidence as

consistent (C); inconsistent (I); or neutral (N) . This process allows analysts to see that some evidence will be consistent

with more than one hypothesis and be less valuable in disproving hypotheses.


20/45


21/45

17

DEVILS ADVOCACY

Challenging a single, strongly held

view or consensus by building thebest possible case for an alternativeexplanation.

WHEN TO USE

Devils Advocacy is most effective whenused to challenge an analytic consensusor a key assumption regarding a criticallyimportant intelligence question. Onthose issues that one cannot afford toget wrong, Devils Advocacy can providefurther confidence that the current analyticline will hold up to close scrutiny. An

individual analyst can often assume therole of the Devils Advocate if he or shehas some doubts about a widely heldview, or a manager might designate acourageous analyst to challenge theprevailing wisdom in order to reaffirm thegroups confidence in those judgments.In some cases, the analyst or a teamcan review a key assumption of acritical judgment in the course of theirwork, or more likely, a separate analyticproduct can be generated that arraysall the arguments and data that supporta contrary hypothesis. While this caninvolve some analytic time and effort,when a group of analysts have workedon an issue for a long period of time, itis probably wise to assume that a strongmind-set exists that deserves the closerscrutiny provided by Devils Advocacy.

VALUE ADDED

Analysts have an obligation topolicymakers to understand wheretheir own analytic judgments might beweak and open to future challenge.Hence, the Devils Advocacy processcan highlight weaknesses in a currentanalytic judgment or alternatively help toreaffirm ones confidence in the prevailingjudgments by:

Explicitly challenging key assumptions

to see if they will not hold up undersome circumstances.

Identifying any faulty logic or information

that would undermine the key analyticjudgments.

Presenting alternative hypotheses

that would explain the current body ofinformation available to analysts.

Its primary value is to serve as a checkon a dominant mind-set that can developover time among even the best analystswho have followed an issue and formedstrong consensus that there is only oneway of looking at their issue. This mind-set phenomenon makes it more likely thatcontradictory evidence is dismissed or

not given proper weight or consideration.An exercise aimed at highlighting suchevidence and proposing another way ofthinking about an issue can expose hiddenassumptions and compel analysts to reviewtheir information with greater skepticismabout their findings. The analyst couldcome away from the exercise more certainthat: 1) the current analytic line was sound;2) the argument is still the strongest, butthat there are areas where further analysisis needed; or 3) some serious flaws in logicor supporting evidence suggest that theanalytic line needs to be changed or at

least caveated more heavily.

THE METHOD

To challenge the prevailing analytic line, theDevils Advocate must:

Outline the mainline judgment and

key assumptions and characterize theevidence supporting that current analyticview.

Select one or more assumptionsstated

or notthat appear the most susceptible

to challenge.

Review the information used to determine

whether any is of questionable validity,whether deception is possibly indicated,or whether major gaps exist.

C O N T R A R I A N T E C H N I Q U E S


22/45

18

Highlight the evidence that could

support an alternative hypothesis orcontradicts the current thinking.

Present to the group the ndings

that demonstrate there are flawedassumptions, poor quality evidence, orpossible deception at work.

Consider drafting a separate contrarian

paper that lays out the arguments fora different analytic conclusion if thereview uncovers major analytic flaws.

Be sure that any products generated

clearly lay out the conventional wisdomand are identified as an explicitly

Devils Advocate project; otherwise,the reader can become confused as tothe current official view on the issue.


23/45

19

TEAM A/TEAM B

Use of separate analytic teams that

contrast two (or more) strongly heldviews or competing hypotheses.

WHEN TO USE

A Team A/Team B approach is differentfrom Devils Advocacy, where the purposeis to challenge a single dominant mind-set. Instead, Team A/Team B recognizesthat there may be competing and possiblyequally strong mind-sets held on an issuethat need to be clarified. Sometimesanalysts confuse the two techniques bydrafting a Team B exercise that is really a

Devils Advocacy exercise.

If there are at least two competingviews within an analytic office orperhaps competing opinions withinthe policymaking community on a keyissue, then Team A/Team B analysis canbe the appropriate technique to use.Developing a full-blown Team A/Team Bexercise requires a significant commitmentof analytic time and resources, so it isworthwhile considering if the analytic issuemerits this kind of attention.

A longstanding policy issue, a criticaldecision that has far-reaching implications,or a dispute within the analytic communitythat has obstructed effective cross-agencycooperation would be grounds for usingTeam A/Team B. If those circumstancesexist, then analysts will need to review allof the data to develop alternative papersthat can capture the essential differencesbetween the two viewpoints.

VALUE ADDED

Managers have found that when there areoffice tensions among competing factions

of analysts, a Team A/Team B approach canhelp opposing experts see the merit in theother groups perspective. The process ofconducting such an exercise can reducethe friction and even narrow the differences.At a minimum, it allows those holdingopposing views to feel that their views havebeen given equal attention.

For the policymaker, this technique helpsto surface and explain important analyticdifferences within the expert community.Often senior officials can learn more by

weighing well-argued conflicting views thanfrom reading an assessment that maskssubstantive differences or drives analysisto the lowest common denominator.By making the key assumptions andinformation used for each argument moretransparent, a policymaker can judge themerits of each case, pose questions backto the analysts, and reach an independentjudgment on which argument is thestrongest. Moreover, highlighting alternativeviews puts collectors on notice that theyneed to be searching for new information

that can confirm or disconfirm a range ofhypotheses.

If opposing positions are well established,it can be useful to place analysts on teamsthat will advocate positions they normallydo not support; forcing analysts to arguethe other side can often make them moreaware of their own mind-set.


24/45

20

A Team A/Team B View: Chinas Military

A: Chinas Hollow Militaryby Bates Gill and Michael OHanlon

B:Chinas Military: A SecondOpinion by James Lilley and Carl Ford

. . . We believe that the recent clamor

over Chinas strategic ambitions is greatly

overblown. Most of the Chinese aims that

run counter to US interests are in fact

not global or ideological but territorial

in nature and confined primarily to the

islands and waterways to Chinas south

and southeast . . .

An enormous gap separates Chinas

military capabilities from its aspirations.The PRCs armed forces are not very good

and not getting better very fast. Whatever

Chinas concerns and intentions, its

capacity to act upon them in ways inimical

to US interests is severely limited, and will

remain so for many years . . .

The PRCs power projection capabilities,

too, are constrained by huge weaknesses

especially in areas such as aerial refueling,

electronic warfare, command and control

and amphibious and air assault assets.

China owns considerably less top-level

equipment than medium powers like Japan

and Britain; it owns even less than smaller

powers such as Italy, South Korea or The

Netherlands . . . The resources it devotes

to acquiring modern weaponry are akin to

those of countries spending $10-20 billion

a year on defense . . .

. . . We think they got it half right. China

is no military superpower and will not

acquire that status for some years to

come. But measured in terms of its

capacity to challenge key US allies in East

Asia, Chinas capabilities have grown

exponentially . . .

By emphasizing direct comparisons

between the defense capabilities of the

United States and the PRC, the authorscreate an artificial and misleading

construct . . . What the regime gives

every indication of striving for is sufficient

military clout to achieve its aims in Asia.

In the short term, it wishes to intimidate

Taiwan sufficiently to bring about

unification on Beijings terms . . .

. . . Investments of the sort Beijing is

making can mean only one thing: China

is determined to improve the PLAs

fighting capability. While most nations

are reducing defense expenditures in the

postCold War era, China is one of the few

doing the opposite . . . Across the board,

the PLA is engaged in a major spending

effort to upgrade weapons and equipment

and improve its operational capability.

According to the Pentagon, these efforts

have already enhanced Chinas ability to

project military power . . .

Excerpted from journal articles found in The National Interest, Fall and Winter 2000/2001.


25/45

21

THE METHOD

Analysis Phase.A Team A/Team Bexercise can be conducted on an

important issue to:

Identify the two (or more) competing

hypotheses or points of view.

Form teams or designate individuals to

develop the best case that can be madefor each hypothesis.

Review all pertinent information that

supports their respective positions.

Identify missing information that would

buttress their hypotheses.

Structure each argument with an explicit

presentation of key assumptions,key pieces of evidence, and carefularticulation of the logic behind theargument.

Debate Phase.An oral presentation ofthe alternative arguments and rebuttals inparallel fashion can then be organized for

the benefit of other analysts:

Set aside time for an oral presentation

of the alternative team findings; this canan informal brainstorming session or amore formal debate.

Have an independent jury of peers

listen to the oral presentation andbe prepared to question the teamsregarding their assumptions, evidence,or logic.

Allow each team to present their case,challenge the other teams arguments,and rebut the opponents critique of itscase.

Let the jury consider the strength of

each presentation and recommendpossible next steps for further researchand collection efforts.


26/45

22

HIGH-IMPACT/LOW-PROBABILITY

ANALYSIS

Highlights a seemingly unlikelyevent that would have major policy

consequences if it happened.

WHEN TO USE

High-Impact/Low-Probability Analysisis a contrarian technique that sensitizesanalysts to the potential impact ofseemingly low probability events thatwould have major repercussions on USinterests. Using this technique is advisablewhen analysts and policymakers areconvinced that an event is unlikely but

have not given much thought to theconsequences of its occurrence. Inessence, this can be a warning that theintelligence and policy communitiesmust be alert to an unexpected but notimpossible event. For example, the fallof the Shah, the collapse of the SovietUnion, and the reunification of Germanywere all considered low probability eventsat one time; however, analysts mighthave benefited from considering theconsequences of such events and howthey might plausibly have come about.

VALUE ADDED

Mapping out the course of an unlikely,yet plausible, event can uncover hiddenrelationships between key factors andassumptions; it also can alert analyststo oversights in the mainstream analyticline. In addition, an examination of theunthinkable allows analysts to developsignposts that may provide early warningof a shift in the situation. By periodicallyreviewing these indicators an analystis more likely to counter any prevailing

mind-set that such a development ishighly unlikely.

THE METHOD

If there is a strongly held view that an eventis unlikely, then postulating precisely the

opposite should not be difficult.

Dene the high-impact outcome clearly.

This process is what will justify examiningwhat most analysts believe to be a veryunlikely development.

Devise one or more plausible

explanations for or pathways to the lowprobability outcome. This should be asprecise as possible, as it can help identifypossible indicators for later monitoring.

Insert possible triggers or changes inmomentum if appropriate. These canbe natural disasters, sudden healthproblems of key leaders, or neweconomic or political shocks that mighthave occurred historically or in otherparts of the world.

Brainstorm with analysts having a broad

set of experiences to aid the developmentof plausible but unpredictable triggers ofsudden change.

Identify for each pathway a set of

indicators or observables that wouldhelp you anticipate that events werebeginning to play out this way.

Identify factors that would deect a

bad outcome or encourage a positiveoutcome.


27/45

23

High Impact of a Low-Probability Event: Pearl Harbor

. . . So far as relations directly between the United States and Japan are concerned, there

is less reason today than there was a week ago for the United States to be apprehensive

lest Japan make war on this country.

Were it a matter of placing bets, the undersigned would give odds of five to one that the

United States and Japan will not be at war on or before December 15; would wager three

to one that the United States and Japan will not be at war on or before the 15th of January

(i.e, seven weeks from now); would wager even money that the United States and Japan

will not be at war on or before March 1 (a date more than 90 days from now) . . .

State Department Special Assessment, 27 November 1941

High-Impact/Low-Probability Analysis on 9/11

The 9/11 Commission report includes examination of speculative analysis that was

provided to senior policymakers highlighting what were thought to be highly unlikely

scenarios that would have a very high impact. One of those items was the 6 August 2001

Presidents Daily Brief, which stated:

Al-Qaida membersincluding some who are US citizenshave resided in or traveled to

the US for years, and the group apparently maintains a support structure that could aid

attacks. Two al-Qaida members found guilty in the conspiracy to bomb our embassies in

East Africa were US citizens, and a senior EIJ [Egyptian Islamic Jihad] member lived in

California in the mid-1990s.

A clandestine source said in 1998 that a Bin Ladin cell in New York was recruiting

Muslim-American youth for attacks.

We have not been able to corroborate some of the more sensational threat reporting, such

as that from a service in 1998 saying that Bin Ladin wanted to hijack a US aircraft to gain

the release of Blind Shaykh Umar Abd al-Rahman and other US-held extremists.

9/11 Commission Report, declassified and approved for release 10 April 2004.


28/45

24

WHAT IF? ANALYSIS

Assumes that an event has occurred with

potential (negative or positive) impactand explains how it might come about.

WHEN TO USE

What If? analysis is another contrariantechnique for challenging a strong mind-set that an event will not happen or thata confidently made forecast may not beentirely justified. It is similar to a High-Impact/Low-Probability analysis, but itdoes not dwell on the consequencesof the event as much as it acceptsthe significance and moves directly to

explaining how it might come about.

VALUE ADDED

By shifting the focus from whether anevent could occur to how it may happen,analysts allow themselves to suspendjudgment about the likelihood of the eventand focus more on what developmentseven unlikely onesmight enable such anoutcome. An individual analyst or a teammight employ this technique and repeatthe exercise whenever a critical analyticjudgment is made.

Using this technique is particularlyimportant when a judgment restson limited information or unprovenassumptions. Moreover, it can freeanalysts from arguing about theprobability of an event to consideringits consequences and developing someindicators or signposts for its possibleemergence. It will help analysts addressthe impact of an event, the factors thatcould causeor alterit, and likelysignposts that an event is imminent.

A What If? analysis can complement adifficult judgment reached and providethe policymaker a thoughtful cautionto accepting the conventional wisdomwithout considering the costs and risksof being wrong. This can help decision-makers consider ways to hedge their bets,even if they accept the analytic judgmentthat an event remains unlikely.

THE METHOD

Like other contrarian methods, What If?analysis must begin by stating clearly

the conventional analytic line and thenstepping back to consider what alternativeoutcomes are too important to dismiss,even if unlikely. Brainstorming over a fewdays or weeks can develop one or moreplausible scenarios by which the unlikelyevent occurs:

Assume the event has happened.

Select some triggering events that

permitted the scenario to unfold to helpmake the what if more plausible; for

example, analysts might postulate thedeath of a leader, a natural disaster, orsome economic event that would start achain of other events.

Develop a chain of argumentation based

as much on logic as evidence to explainhow this outcome could have comeabout.

Think backwards from the event in

concrete waysthat is, specifying whatmust actually occur at each stage of the

scenario is often very useful.

Identify one or more plausible pathways

or scenarios to the unlikely event; veryoften more than one will appear possible.

Generate a list of indicators or

observables for each scenario thatwould help to detect the beginnings ofthe event.

Consider the scope of the positive and

negative consequences of each scenarioand their relative impacts.

Monitor the indicators developed on a

periodic basis.


29/45

25

What If? Analysis: An Unlikely Outcome in Yugoslavia, 1990.

The possibility of muddling through is very low. In the unlikely event that it happens, this iswhat it would look like.

Memories of the internecine civil war during World War II and fear of another destructive

conflict would lead the two most numerous South Slav peopleSerbs and Croatsto

reach some political accommodation. A compromise that preserves Yugoslavia would

include:

Basic principles:

-No change in existing Republic borders.

-No change in Yugoslavias existing international status.

-Mutually recognized sovereignty of each republic

Confederal institutions:

-A single foreign ministry, to which diplomatic representatives would be accredited.

-A central military organization with a joint General Staff responsible for planning.

-A central bank, determining macroeconomic policy, a common currency,

Powers reserved to republics.

-Veto over actions of the Confederal Authority.

-Control of internal security, including guarantee of minority rights.

-Operational control over some or all military units stationed on the republics territory.

-Raising taxes and allocating funds to discharge mutually agreed confederal

responsibilities.

Only the Serbs can open the door to a confederal Yugoslavia, and Serbias leader,

Slobodan Milosevic, holds the key. Some observers felt there are pressures on him to try.

If he does not, he would give his opponents the leverage to remove him. The potential

penalties of failure to compromise would be too great, in this view, for the peoples and

leaders of Yugoslavia to forgo every effort to find a compromise.

Excerpts from the declassified NIE: Prospects for Yugoslavia, October 1990.


30/45


31/45

27

I M A G I N A T I V E T H I N K I N G T E C H N I Q U E S

BRAINSTORMING

An unconstrained group process

designed to generate new ideas andconcepts.

WHEN TO USE

Brainstorming is a widely used techniquefor stimulating new thinking and it canbe applied to virtually all of the otherstructured analysis techniques as anaid to thinking. Typically, analytic willbrainstorm when they begin a projectto help generate a range of hypothesesabout their issue.

Brainstorming, almost by definition,involves a group of analysts meeting todiscuss a common challenge; a modestinvestment of time at the beginningor critical points of a project can takeadvantage of their different perspectivesto help structure a problem. This groupprocess allows others to build on aninitial idea suggested by a member of thebrainstorming session.

An individual analyst also can brainstormto produce a wider range of ideas than

a group might generate, without regardfor other analysts egos, opinions, orobjections. However, an individual will nothave the benefit of others perspectives tohelp develop the ideas as fully. Moreover,an individual may have difficulty breakingfree of his or her cognitive biases withoutthe benefit of a diverse group.

VALUE ADDED

This technique can maximize creativityin the thinking process, force analyststo step outside their normal analytic

mind-sets, and suspend their typicalgood judgment about the practicalityof ideas or approaches. More generally,brainstorming allows analysts to see awider range of factors that might bearon the topic than they would otherwiseconsider. Analysts typically censorout ideas that seem farfetched, poorly

sourced, or seemingly irrelevant to thequestion at hand. Brainstorming givespermission to think more radically or

outside the box. In particular, it can sparknew ideas, ensure a comprehensive look ata problem or issues, raise unknowns, andprevent premature consensus around asingle hypothesis.

THE METHOD

Paradoxically, brainstorming should bea very structured process to be mostproductive. An unconstrained, informaldiscussion might produce some interestingideas, but usually a more systematicprocess is the most effective way to

break down mind-sets and produce newinsights. In particular, the process involvesa divergent thinking phase to generate andcollect new ideas and insights, followedby a convergent phase in which ideasare grouped and organized around keyconcepts. Some of the simple rules to befollowed include:

Never censor an analysts ideas no

matter how unconventional they mightsound.

Rather nd out what prompted thethought, as it might contain the seedsof an important connection between thetopic and an unstated assumption.

Give yourself enough time to do

brainstorming correctly. It usually takesone hour to set the rules of the game,get the group comfortable, and exhaustthe conventional wisdom on the topic.Only then will the truly creative ideasbegin to emerge.

Involve at least one outsider in the

processthat is, someone who does notshare the same educational background,culture, technical knowledge or mind-set as the core group but has somefamiliarity with the topic.

A two-phase, twelve-step, structuredprocess is often used to get the most out ofthe brainstorming sessions:


32/45

28

Brainstorming and Divergent Thinking:

Perspectives of an Experienced Practitioner

. . . First, leave rank at the door and focus on a democracy of ideas. Thoughts from

experts, senior officers, and supervisors are of course, valuable and welcome, but such

experts are not permitted to cut off debate by citing their seniority. In fact, it pays to invite

junior officers as well as senior ones who are not involved directly in working the issue

under discussion. Some of the most creative ideas at brainstorming sessions frequently

come from relatively junior people who can look at a problem with fresh perspective, or

from senior ones who are not experts on the issue.

Second, make sure there is no official analytic line. One of the most significant blocks to

new thinking is the presence of a long held analytic line that analystsand even more so

managersare reluctant to change. Rather than trying to fit ideas into the framework of

what weve said before, analysts need to feel free to go wherever bits of the evidence

and informed supposition take them. They must feel free to throw out seemingly strangebut plausible ideas that might be based on historical precedent and instinct rather than

on concrete information. Facilitators can stimulate this process by deliberately posing

an alternative outcome to a problem that differs starkly from the accepted analysis or by

proposing a contrary way to think about an issue.

Third, dont permit killer phrases like that would not work or that could not happen

to be voiced out loud. Effective brainstorming starts with ideas and possibilities, not with

practicalities and self-imposed obstacles to fresh perspectives. Force the group to get as

wide a range of ideas out for discussion as possible. At some point, a set of ideas might

be winnowed down and subject to tests of workability, but that comes laternot during

brainstorming.

Fourth, keep the brainstorming session to no more than 90 minutes. There is no hard and

fast rule, but somewhere between 60 and 90 minutes, the idea stream starts to dry up,

people repeat themselves, and jokes replace creative ideas.

Fifth, record ideas in a visible way. Lots of people take notes at brainstorming sessions

for their own use, and that is good. We have found it valuable to have someone jot down

the ideas presented on large paste-a-note sheets that we put on the walls. This allows

participants in brainstorming sessions to react to ideas generated earlier. Moreover,

analysts are encouraged to participate in such exercises if they see their own ideas put

down in writing. Having a notional record of brainstorming also helps the analyst who ends

up writing a report based on the discussion. Ideas that may not be used in one report are

invariably put to use later, so it is good to have a record of them . . .


33/45

29

Divergent Thinking Phase:

Distribute Post-It notes and pens or

markers to all participants. Typically,10-12 people works best.

Pose the problem in terms of a focal

question. Display it in one sentence ona large easel or whiteboard.

Ask the group to write down responses

to the question, using key words that willfit on the small Post-It note.

Stick all the notes on a wall for all to

seetreat all ideas the same.

When a pause follows the initial ow of

ideas, the group is reaching the endof their conventional thinking and thenew divergent ideas are then likely toemerge.

End the collection stage of the

brainstorming after two or three pauses.

Convergent Thinking Phase:

Ask the participants as a group

to rearrange the notes on the wallaccording to their commonalities orsimilar concepts. No talking is permitted.Some notes may be moved several

times as notes begin to cluster. Copyingsome notes is permitted to allow ideasto be included in more than one group.

Select a word or phrase that

characterizes each grouping or clusteronce all the notes have been arranged.

Identify any notes that do not easily t

with others and consider them eitheruseless noise or the beginning of anidea that deserves further attention.

Assess what the group has

accomplished in terms of new ideasor concepts identified or new areas

that need more work or furtherbrainstorming.

Instruct each participant to select one

or two areas that deserve the mostattention. Tabulate the votes.

Set the brainstorming groups priorities

based on the voting and decide on thenext steps for analysis.


34/45

30

OUTSIDE-IN THINKING

Used to identify the full range of basic

forces, factors, and trends that wouldindirectly shape an issue.

WHEN TO USE

Analysts find this technique most usefulat the conceptualization of an analyticproject, when the goal is to identify all thecritical, external factors that could influencehow a particular situation will develop. Itwould work well for a group of analystsresponsible for a range of functional and/orregional issues. When assembling a largedatabase that must identify a number of

information categories or database fields,this technique can aid in visualizing theentire set of categories that might beneeded in a research effort. Often analystsrealize only too late that some additionalinformation categories will be needed andthen must go back and review all previousfiles and recode the data. With a modestamount of effort, Outside-in Thinkingcan reduce the risk of missing importantvariables early in the analytic process.

VALUE ADDED

Most analysts spend their time concentratingon familiar factors within their field oranalytic issue. That is, they think from theinsidenamely, what they controloutto the broader world. Conversely, thinkingfrom the outside-in begins by consideringthe external changes that might, over time,profoundly affect the analysts own field orissue. This technique encourages analyststo get away from their immediate analytictasks (the so-called inbox) and thinkabout their issues in a wider conceptual

and contextual framework. By recasting theproblem in much broader and fundamentalterms, analysts are more likely to uncover

additional factors, an important dynamic, ora relevant alternative hypothesis.

THE METHOD

The process begins by developing ageneric description of the problem or thephenomenon under study. Then, analystsshould:

List all the key forces (social,

technological, economic, environmental,and political) that could have an impacton the topic, but over which one can

exert little influence (e.g., globalization,social stress, the Internet, or the globaleconomy).

Focus next on key factors over which

an actor or policymaker can exert someinfluence. In the business world thismight be the market size, customers, thecompetition, suppliers or partners; in thegovernment domain it might include thepolicy actions or the behavior of allies oradversaries.

Assess how each of these forces could

affect the analytic problem.

Determine whether these forces actually

do have an impact on the particular issuebased on the available evidence.


35/45

31

RED TEAM ANALYSIS

Models the behavior of an individual

or group by trying to replicate how anadversary would think about an issue.

WHEN TO USE

Frequently, analysts face the challengeof forecasting how a foreign leader ordecisionmaking group may behave whenit is clear that there is a risk of fallinginto a mirror-image problem. That is,analysts can sometimes impute to aforeign actor the same motives, values, orunderstanding of an issue that they hold.Traditional analysis sometimes assumes

that foreign leaders or groups will behaverationally and act as the analystswould if faced with the same threats oropportunities. History has shown thatforeign leaders often respond differentlyto events because of different cultural,organizational, or personal experiences.

Red Team analysis tries to consciouslyplace analysts in the same cultural,organizational, and personal setting(putting them in their shoes) in whichthe target individual or group operates.

Whereas analysts normally work from theposition of the blue (friendly forces), ared team of analysts attempts to work inthe environment of the hostile forces.

VALUE ADDED

Like Devils Advocacy and Team A/Team Btechniques, Red Team analysis is aimedat freeing the analyst from the prison of awell-developed mind-set; in this case, theanalysts own sense of rationality, culturalnorms, and personal values. Whereasanalysts usually operate as observers

of a foreign adversary, the Red Teamtechnique transforms the analyst into anactor operating within the adversarysculture and political milieu. This form ofrole playing is useful when trying toreplicate the mind-set of authoritarianleaders, terrorist cells, or other non-Western groups that operate under verydifferent codes of behavior or motivations.

Often this technique can introduce new ordifferent stimuli that might not have beenfactored into traditional analysissuch as

the targets familial ties or the internationalpolitical, economic, and military pressuresfelt by the individual. For example, RedTeam participants might ask themselves:What would my peers, family, or tribeexpect me to do? Alternatively, a RedTeam analyst might pose the questionto his colleagues: How do we perceivethe external threats and opportunities?Finally, the Red Team technique can factorinto its analysis the way in which personalpower and status might influence a targetsbehavior.

THE METHOD

On issues that lend themselves to RedTeam analysis, a manager needs to builda team of experts with in-depth knowledgeof the operating environment, the targetspersonality, and the style of thinking used.The team should be populated not justwith those who understand the language,but also with people who might haveexperienced the culture, share the ethnicbackground, or have worked in a similaroperational environment. Once established

and separated from traditional analysis, theteam members should:

Put themselves in the adversarys

circumstances and react to foreign stimulias the target would.

Develop a set of rst-person questions

that the adversary would ask, suchas: How would I perceive incominginformation; what would be my personalconcerns; or to whom would I look for anopinion?

Draft a set of policy papers in which the

leader or group makes specific decisions,proposes recommendations, or laysout courses of actions. The more thesepapers reflect the cultural and personalnorms of the target, the more they canoffer a different perspective on theanalytic problem.


36/45

32

A Red Team PerspectiveIrans Military Strategy Vis-a-vis the United States

. . . The United States and Israel may be contemplating military operations against Iran, as per

recent media reports . . . A week-long combined air and ground maneuver has just concluded

in five of the southern and western provinces of Iran, mesmerizing foreign observers, who

have described as spectacular the massive display of high-tech, mobile operations, including

rapid-deployment forces relying on squadrons of helicopters, air lifts, missiles, as well as

hundreds of tanks and tens of thousands of well-coordinated personnel using live munitions.

Learning from both the 2003 Iraq war and Irans own previous experiences of the 1980-88

war with Iraq and the 1987-88 confrontation with US forces in the Persian Gulf, Iranians have

focused on the merits of a fluid and complex defensive strategy that seeks to take advantage

of certain weaknesses in the US military superpower while maximizing the precious few areas

where they may have the upper hand, e.g.; numerical superiority in ground forces, guerrilla

tactics, terrain, etc.

Any US attack on Iran will likely be met first and foremost by missile counter-attacks engulfing

the southern Persian Gulf states playing host to US forces, as well as any other countrye.g.,

Azerbaijan, Iraq, or Turkey allowing their territory or airspace to be used against Iran. The

rationale for this strategy is precisely to pre-warn Irans neighbors of the dire consequences,

with potential debilitating impacts on their economies for a long time, should they become

accomplices of foreign invaders of Iran.

Another key element of Irans strategy is to increase the arch of crisis in places such as

Afghanistan and Iraq, where it has considerable influence, to undermine the United States

foothold in the region, hoping to create a counter-domino effect wherein instead of gaining

inside Iran, the US would actually lose territory partly as a result of thinning its forces andmilitary overreach.

Irans counter-psychological warfare, on the other hand, seeks to take advantage of the death-

fearing American soldiers who typically lack a strong motivation to fight wars not necessarily

in defense of the homeland. A war with Iran would definitely require establishing the draft in the

US, without which it could not possibly protect its flanks in Afghanistan and Iraq.

There is a sense of national-security siege in Iran these days, in light of a tightening security

belt by the US benefiting from military bases in Iraq, Turkey, Azerbaijan, Uzbekistan,

Tajikistan, Kyrgyzstan, as well as Kuwait, Saudi Arabia, Qatar, Bahrain, Oman, and the island-

turned garrison of Diego Garcia. From Irans vantage point, the US, having won the Cold War,

has turned into a leviathan unhinged capable of manipulating and subverting the rule of

international law and the United Nations with impunity, thus requiring a sophisticated Iranianstrategy of deterrence, that, in the words of certain Iranian media pundits, would even include

the use of nuclear weapons . . .

Excerpts from Kaveh L. Afrasiabi (Tehran University), How Iran Will Fight Back,

cited in Asia Times Online Ltd., 2004.


37/45

33

Red Team analysis is not easy to conduct.It requires significant time to develop ateam of qualified experts who can think like

the adversary. The team has to distanceitself from the normal analysis and workas though living in the targets world.Without a sophisticated understanding ofthe culture, operational environment, andpersonal histories of the foreign group,analysts will not be able to behave or thinklike the enemy. Analysts can never trulyescape their own experiences and mind-sets, but this technique can at least preventthem from falling into mirror-imagingunconsciously.

The most novel feature of Red Teamanalysis is its presentation.

The analysis is often in a rst person

formatthat is, drafted as memos to orfrom a leader or group.

Red Team analysis avoids the use of

caveats or qualifications and assumesthat the recipient understands that

the paper is aimed more at provokingthought or challenging the conventionalunderstanding of how an adversarythinks.

Such papers are rarely coordinated

among other experts and do not purportto represent the consensus view on anissue.

Red Team papers do not plot out allpossible courses of action but seek togive a prediction based on the targets

special personal, organizational, orcultural experiences.


38/45

34

ALTERNATIVE FUTURES ANALYSIS

Systematically explores multiple ways a

situation can develop when there is highcomplexity and uncertainty.

WHEN TO USE

Alternative futures analysis (often referredto as scenarios) is most useful whena situation is viewed as too complex orthe outcomes as too uncertain to trusta single outcome assessment. First,analysts must recognize that there ishigh uncertainty surrounding the topic inquestion. Second, they, and often theircustomers, recognize that they needto consider a wide range of factors thatmight bear on the question. And third,they are prepared to explore a range ofoutcomes and are not wedded to anypreconceived result. Depending on howelaborate the futures project, the effortcan amount to considerable investment intime, analytic resources, and money.A team of analysts can spend severalhours or days organizing, brainstorming,and developing multiple futures;alternatively, a larger-scale effort canrequire preparing a multi-day workshop

that brings together participants(including outside experts). Such anundertaking often demands the specialskills of trained scenario-developmentfacilitators and conferencing facilities.

This technique is a sharp contrast tocontrarian techniques, which try tochallenge the analysts high confidenceand relative certitude about an eventor trend. Instead, multiple futuresdevelopment is a divergent thinkingtechnique that tries to use the complexity

and uncertainty of a situation to describemultiple outcomes or futures that theanalyst and policymaker should consider,rather than to predict one outcome.

VALUE ADDED

Alternative futures analysis is extremelyuseful in highly ambiguous situations, whenanalysts confront not only a lot of knownunknowns but also unknown unknowns.What this means is that analysts recognizethat there are factors, forces, and dynamicsamong key actors that are difficult toidentify without the use of some structuredtechnique that can model how they wouldinteract or behave. As the outcomes are notknown prior to the futures exercise, analystsmust be prepared for the unexpected andwilling to engage in a more free-wheelingexchange of views than typically occursin order to imagine the future. Given

the time and resources involved, scenarioanalysis is best reserved for situations thatcould potentially pose grave threats orotherwise have significant consequences.

From past experience, analysts havefound that involving policymakers in thealternative futures exercise is the mosteffective way to communicate the results ofthis exploration of alternative outcomes andsensitize them to key uncertainties. Mostparticipants find the process of developingsuch scenarios as useful as any finished

product that attempts to capture the resultsof the exercise. Analysts and policymakerscan benefit from this technique in severalways:

It provides an effective means of weighing

multiple unknown or unknowablefactors and presenting a set of plausibleoutcomes.

It can help to bound a problem by

identifying plausible combinations ofuncertain factors.

It provides a broader analytic framework

for calculating the costs, risks, andopportunities presented to policymakersby different outcomes.

The future is plural.

Peter Schwartz,

author ofThe Art ofthe Long View anda widely acclaimedscenario developer.


39/45

35

Hypothetical Threats to the Homeland:Using Spectrums to Define Potential Targets

HIGH

(Weapons ofMass Destruction)

S

ophistication

o

fWeapon

Intended Impact

LOW(Rifle/Handgun)

BROAD(General USpopulation)

SELECTIVE(Military/Police

post)

Kidnapping senior policyofficials child

Coordinated car bombsat shopping centers

Mortar attack on USmilitarys home base

Sniper attacks on home

base Guard Post

Information warfare

Introduce contaminateddrugs into domesticsupply chain

BW attack on militarybase water supply

RPG or SAM attack onmilitary aircraft

A Futures Exercise. The graphic captures four potential futures to understand how foreign insurgentsmight carry out an attack on the United States. A brainstorming exercise helped analysts identify twokey uncertainties (the sophistication of weapons used by the insurgents and the intended impact of theattack) and arrayed these factors on a graph as the x and y axes. The four resulting quadrants in the2 x 2 matrix allowed analysts to visualize potential targets from the various combinations (low to highsophistication of weapons and selective to broad intended impact of an attack). For example, if a grouppossessed highly sophisticated weapons and intended a broad attack on the United States, potentialtargets could include computer networks and domestic drug supplies. Having filled in a quadrant,analysts can then turn to devising likely indicators or signposts of such a future.


40/45

36

It aids analysts and policymakers in

anticipating what otherwise would besurprising developments by forcing

them to challenge assumptions andconsider possible wild cards ordiscontinuous events.

It generates indicators to monitor

for signs that a particular future isbecoming more or less likely, so thatpolicies can be reassessed.

THE METHOD

Although there are a variety of ways todevelop alternative futures, the mostcommon approach used in both the public

and private sectors involves the followingsteps:

Develop the focal issue by

systematically interviewing experts andofficials who are examining the generaltopic.

Convene a group of experts (both

internal and external) to brainstormabout the forces and factors that couldaffect the focal issue.

Select by consensus the two mostcritical and uncertain forces and convertthese into axes or continua with themost relevant endpoints assigned.

Establish the most relevant endpoints

for each factor; e.g., if economicgrowth were the most critical, uncertain

force, the endpoints could be fastand slow or transformative andstabilizing depending on the type ofissue addressed.

Form a futures matrix by crossing the

two chosen axes. The four resultingquadrants provide the basis forcharacterizing alternative future worlds.

Generate colorful stories that describe

these futures and how they couldplausibly come about. Signposts or

indicators can then be developed.

Participants, especially policymakers, canthen consider how current decisions orstrategies would fare in each of the fourworlds and identify alternative policiesthat might work better either across all thefutures or in specific ones. By anticipatingalternative outcomes, policymakers have abetter chance of either devising strategiesflexible enough to accommodate multipleoutcomes or of being prepared and agilein the face of change.


41/45

37


42/45

38

These structured analytic techniquescan be used in a variety of ways whenanalysts begin a new assessment. Some

can be used equally effectively at multiplepoints in the process and can promotean analysts ability to keep an open mind,to consider multipleincluding highlyunlikelyhypotheses, to challengeconventional wisdom, and to assess theimpact of important information gaps ordeception on analytic judgments andconfidence levels. The Timeline for UsingAnalytic Techniques provides somethoughts on when to use one or more ofthem during the course of an analystsresearch and writing.

S T R A T E G I E S F O R U S I N G S T R U C T U R E D

A N A L Y T I C T E C H N I Q U E S

Starting Out

At the beginning of an analytic project,analysts are always wise to consider

brainstorming and assumptions checksto insure that important factors are notbeing missed or taken for granted.Similarly, outside-in-thinking cansometimes put an analytic project into abroader international context, in whichfactors outside the lead analysts areaof responsibility might impact on his orher analytic judgments. For instance,economic assumptions about the priceof oil might be key to a regional politicalanalysts understanding the prospectsfor political stability in an oil-exporting

A Timeline for Using Analytic Techniques

Throughout an Analytic Project

Begin Paper/

Commence Project

Deliver Final Paper/

Brief Key Findings

Brainstorming

Key Assumptions Check

What If? Analysis

High Impact/Low Probability Analysis

Outside-In Thinking

Brainstorming

Key Assumptions Check

Team A/Team B

Analysis/Debate

Red Team Red Team

Devils Advocacy

Deception Detection Deception Detection

Gaming, Modeling, and Simulation*

Scenarios and Alternative Futures Analysis

ACH

Indicators

*Gaming, Modeling, and Simulation are among the more sophisticated techniques taught ina more advanced analytic methods course and usually require substantial commitments ofanalyst time and corporate resources.


43/45

39

country or a underdeveloped countryentirely dependent on expensive energyimports. A High Impact/Low Probability

assessment can also sensitize analystsearly on to the significance of dramaticevents that might affect their analytic lines.

Some techniques like Indicators andSignposts or Analysis of CompetingHypotheses (ACH) can be usefulthroughout a project and revisitedperiodically as new informationis absorbed and analyzed. ACH,in particular, is a good tool to usethr

CIA Tradecraft Primer Structured Analytic Techniques

Documents

Transcript of CIA Tradecraft Primer Structured Analytic Techniques