How can we best support evidence-

informed policy making?

Martin McKee

European Observatory on Health

Systems and Policies

Where am I?

You’re 30 metres above

the ground in a balloon

You must be a researcher

Yes. How did you know?

Because what you told me is

absolutely correct but completely

useless

You must be a policy maker

Yes, how did you know?

Because you don’t know where you are, you don’t know where you’re going, and now

you’re blaming me

The problem

“… in public policy making, many

suppliers and users of social research are

dissatisfied, the former because they are

not listened to, the latter because they

do not hear much what they want to

listen to…”Lindblom, 2002

Reaching policy makers

The world is complex

Source: UK Foresight Report on Obesity

Even for policy makers

Source: US Department of Defense (NY Times 28 April 2010)

A few sociologists write the best English

they are capable of writing.... Others,

however – and a vast majority – write in

a language that has to be learned

almost like Esperanto. It has a private

vocabulary which, in addition to strictly

sociological terms, includes new words

for the commonest actions, feelings and

circumstances. It has the beginnings of

a new grammar and syntax, much

inferior to English grammar in force and

precisions. So far as it has any effect on

standard English, the effect is largely

pernicious.

Cowley M. Sociological habit patterns in

linguistic transmogrification” Reporter,

1965

The problem with researchers

American Journal of Sociology, Vol. 73, No. 5

(Mar., 1968), pp. 617-627

But at least some try...

We have all had the experience. You

start a new job, be it academic or

corporate. You walk into a

presentation by a colleague. Before

the third slide comes up on the

screen, you find yourself totally lost

in a morass of method, terminology,

and tradition beyond your

comprehension. You know the

speaker is brilliant and you have full

confidence in the conclusions;

however, you find it impossible to

follow the logic that takes the work

from beginning to end.

SUPPORT for evidence-informed policy making

Oxman et al. Health Research Policy and Systems 2009 7(Suppl 1):S1 doi:10.1186/1478-4505-7-S1-S1

Source: Health Research Policy and Systems 2009, 7(Suppl 1):S4

Evidence in support of decisions

Oxman et al. Health Research Policy and Systems 2009 7(Suppl 1):S1 doi:10.1186/1478-4505-7-S1-S1

Realistic evaluation

• Not “what works?”

• “What works for whom and in

what circumstances?”

• Focus on heterogeneity in

evidence synthesis rather than

identification of central tendency

Making the decison

Source: Oxman et al. Health Research Policy and Systems 2009 7(Suppl

1):S16 doi:10.1186/1478-4505-7-S1-S16

What do policy-makers identify as

facilitators and barriers in use of research?

• Facilitators:– personal contact – timely relevance– inclusion of summaries with policy recommendations

• Barriers: – absence of personal contact– lack of timeliness or relevance of research– mutual mistrust– power and budget struggles

• Two-way personal communication (most common suggestion) may improve the appropriate use of researchevidence, but it might also promote its selective and inappropriate use

Innvær et al, 2002

Applying the lessons in practice

Applying the lessons in practice (2)

BlackWhite

With thanks to Douglas Adams

The problem with policy makers

Post modernism in the USA“The aide said that guys like me were

‘in what we call the reality based community,’ which he defined as people who ‘believe that your solutions emerge from your judicious study of discernable reality.’ I nodded and murmured something about enlightenment principles and empiricism. He cut me off. ‘That’s not the way the world really works any more,’ he continued. ‘ We’re an empire now, and when we act we create our own reality.’”

Source: Peter Oborne: The rise of political lying

... and not just in the USA• “If I am being honest, which I will

be … I don’t think I have anything to lose by being honest at this stage in my political career”

Peter Mandelson on being appointed as a European

Commissioner

• “It was a fascinating insight. He talked about being honest as if it was something you might take up at a certain age, like angling or DIY, an optional extra tacked onto your life”

Simon Hoggart, British political commentator

Insights from contemporary politics

A unifying theme – deep dysfunctionalism and dialogues of the deaf

Insights from Wonderland (plus ca change....)

Alice laughed. "one can't believe impossible things.“"I daresay you haven't had much practice," said the Queen. "When I was your age, I always did it for half-an-hour a day. Why, sometimes I've believed as many as six impossible things before breakfast."

"When I use a word,' Humpty

Dumpty said in rather a scornful

tone, 'it means just what I choose

it to mean — neither more nor

less.”

So many out-of-the-way things

had happened lately, that Alice

had begun to think that very few

things indeed were really

impossible.

'Would you tell me, please, which

way I ought to go from here?'

'That depends a good deal on where

you want to get to,' said the Cat.

'I don't much care where —' said

Alice.

'Then it doesn't matter which way

you go,' said the Cat

'Have you guessed the riddle yet?' the Hatter said,

turning to Alice again.

'No, I give it up,' Alice replied: 'what's the answer?'

'I haven't the slightest idea,' said the Hatter.

'Nor I,' said the March Hare.

Alice sighed wearily. 'I think you might do

something better with the time,' she said, 'than

waste it in asking riddles that have no answers.'

A simple question:

What is the role of government in a crisis?

Shame on anyone that makes this tragedy political, socio-economic or racial. …in the land of opportunity and personal responsibility the individual is ultimately accountable.Robert Buckley, Decatur, USA

BBC web site

…the federal government's lethal ineptitude wasn't just a consequence of Mr. Bush's personal inadequacy; it was a consequence of ideological hostility to the very idea of using government to serve the public good. For 25 years the right has been denigrating the public sector, telling us that government is always the problem, not the solution. Why should we be surprised that when we needed a government solution, it wasn't forthcoming?

Paul Krugman (NY Times 5th Sept 2005)

A crucial distinction

• Are policy makers:

– Uninformed?

• can be compensated for by provision of information

• In practice, people use heuristics (information shortcuts, “filling in the blanks”)

– Misinformed?

• conventionally thought to require corrections

• may require action against fundamental biases

Kuklinski et al, 2000

A theory of motivated political

reasoning

• Policy makers are motivated by two goals:

– Accuracy goals

• Seek out and carefully consider relevant evidence to reach the best conclusion

– Partisan goals

• Apply reasoning powers in defence of a prior specific conclusion

• Both virtually always present but the balance varies

– When do partisan biases overwhelm objectivity?

– Why do seemingly rational people become sceptical when faced by overwhelming evidence?

Taber & Lodge, 2006

Seeking and interpreting evidence

• Subjects undertook computer-based exercise where stated purpose was to assimilate evidence to explain to others (gun control, affirmative action)

– Initial assessment of each subject’s position on the issue

– On-line searching and synthesis of evidence to be placed on an information board

– Evidence clearly attributable to sources known to have different views on issue (e.g. Brady Campaign and NRA)

– Computer tracked search patterns and time spent reading different arguments

Taber & Lodge, 2006

Key findings

• Prior attitude effect– Subjects see evidence that they agree with as stronger and

more relevant that evidence they disagree with

• Disconfirmation bias– Subjects actively denigrate evidence they disagree with while

accepting evidence they agree with at face value

• Confirmation bias– When given control over sources of information, subjects seek

out confirmatory evidence and avoid what might challenge their prior beliefs

• Attitude polarization– Partisans presented with same evidence with diverge in their

attitudes

Taber & Lodge, 2006

Confirmation bias in action

• Subjects were given, at random, one of four news stories on diabetes to read, each formatted as article from authoritative source

• Four different causal explanations were embedded in text:

– No specific cause (controls)

– Genetic

– Lifestyle

– Social determinants

Gollust et al, AJPH 2009

Probability of agreeing with the statement that ‘‘People with diabetes got their illness

because of the social or economic circumstances in which they live,’’

Gollust et al,

AJPH 2009

Support for collective societal

responses to diabetes

Gollust et al,

AJPH 2009

Correcting misconceptions

• Subjects initially categorized on a conservative-liberal scale

• Exposed to factually incorrect stories on the effect of US tax cuts and Weapons of Mass Destruction in Iraq, followed by an authoritative correction

• Among those who sympathised with the initial message the correction either failed to change their misperception or actually reinforced it.

Nyhan & Reifler, 2010

Addressing misinformation

• Subjects divided into 3 groups– A. Given facts about scale and duration of US welfare

payments (much less than most people think)

– B. Completed multiple choice quiz on welfare

– C. Given no information

• Extremely high level of wrong answers

• Those most wrong also most confident they are right

• Giving facts has almost no effect on those misinformed

• “It is the ‘I know I’m right’ syndrome that poses the most formidable problem”

Kuklinski et al, 2000

Addressing misinformation• Subjects asked

– how much is spent on welfare?

– How much should be?

– Should welfare be cut?

• Those who say:

– amount spent is high

– What should be spent is much lower

– And discover what is spent is really even lower than they think should, can be persuaded to become more generous

• Authors’ conclusion: to correct misinformation you need to “hit them between the eyes”

Kuklinski et al, 2000

Mechanisms• PET scanning of brains of 15 Democrats

and 15 Republicans just before 2004 US Presidential Election

• Presented with sets of clearly

contradictory statements by candidates• Individuals reported no contradictions in

statements from their own party but contradictions in statements from opposing party

• PET scanning showed that, when confronted with contradictions from own party, brain both switched off centres dealing with negative emotions and switched on centres dealing with positive emotions

• This happened without involving centres involved in reasoning

Westen et al. J Cognitive Neuroscience, 2006

External influences: the media

Mother's fury as nanny state brands her

healthy daughter, 5, 'fat and at risk of

heart disease‘

Elf n' safety warning: You could die laughing

at these amazing pictures of the Nanny State

stating the obvious

'Nanny state' row as problem kids are targeted before birth

Government 'going too far with nanny

state'

What earthly business is it of the Nanny

State if I let my children eat strawberry-

flavoured liquorice shoelaces?

But do the media really matter?

• Between 1996 and 2000, Fox News entered cable TV markets of 20% of US towns.

• Entry of Fox News to a town had a significant impact on voting patterns, with 3-8% shift in favour of Republicans

• More detailed analysis of coverage of specific senate races showed that this was a general ideological shift and not one favouring particular candidates

DellaVigna & Kaplan. The Fox News Effect: Media Bias and Voting (NBER Working Paper No. 12169

External influences: the neighbours

• Support for welfare higher among people who live near to many welfare recipients of the same race– Geographic isolation may be a cause of

“separate-group” thinking.

• Support for welfare lower among people who live near to welfare recipients of another race– People have a hostile reaction to recipients

of another race, but sympathetic reaction to recipients of the same race

• Support for welfare greater among whites who – “have …had a black person for dinner in

your home in the last few years?”

Source: Luttmer, 2001/ Alesina 2003

External influences: vested interests

• In 1994, the chief executives of the leading tobacco companies went before a US Senate committee and testified under oath that “nicotine is not addictive”

• "Nobody believed them" -Supreme Court Justice Antonin Scalia(duck shooting partner of Dick Cheney)

“Nicotine is addictive. We are, then, in the

business of selling nicotine—an addictive drug

effective in the release of stress mechanisms.”

1963 industry document

The frontal attacks...

• “Gratuitous intervention” – Peter Skrabanek

• “Health fascism” – Bruce Charlton

W***kers

The Guardian letters page today is full of the

anguished squeals of the niconazis up in

arms because they haven’t been able to get

a law to jail anyone who even thinks about

having a fag.

Dr Anna Gilmore, Prof Martin McKee,

London School of Hygiene and Tropical

Medicine. Tropical Medicine? What the hell

is that all about? And no-one consulted

smokers about the ban during the sham

consultation. 55% of pub-goers are smokers

by the way.

External influences: vested interests

0

5

10

15

20

25

30

35

40

Exposure to movie tobacco use

grade 5

grade 6

grade 7

grade 8

0-50 51-100 101-150 > 150

Pe

rce

nt

Tri

ed

Sm

okin

g

Song et al., 2007

So can we maintain trust?(stories of children and TLAs)

• BSE – is your daughter’s

hamburger safe?

� MMR – is your son’s

vaccine safe?

Summary

• Researchers can do much more to make their messages clear, relevant, and timely– Obscure and incomprehensible language is not the

aim (reviewers of specialist journals please note)

• But policy-makers need to confront their biases– The quick and easy answer (which fits with your pre-

conceived notion) may not necessarily be the right one

• We all need to understand how where we live, what we see, and how the messages around us are shaped influences what we believe is true