Explanatory Evaluation:how to better understand effects

of interventions

Henk SligteKohnstamm Institute for Educational Research

University of AmsterdamHenk.Sligte@UvA.NL

http://kohnstamminstituut.uva.nl/htm/english.htm

Research for Dutch Ministry of Education

• Report (in Dutch)• Pater, C., Sligte, H., van Eck,

E (2012). Verklarende evaluatie, een methodiek. Amsterdam: Kohnstamm Instituut

• Explanatory Evaluation, a methodology for policy evaluation

http://www.kohnstamminstituut.uva.nl/rapporten/pdf/ki882.pdf

Evaluation

• Evidence based policy (and practice): effect (impact) evaluation what works?

• Dutch Ministry: we need explanations for effects we need a methodology to complement impact evaluation

• The other side of the medal: explanatory evaluation what works for whom in what circumstances? How and why does the policy work (or not)? Finer granularity is needed

• In future: both types to be applied

Tentative question here and now:If the research

method works in explaining effects of policy interventions, can it then also work

in explaining interventions and

innovations in education, in

schools?

Effect (Impact) evaluation

• Through (quasi)experiments demonstration of causal relation between intervention and found effects– Experiment: Controlled Randomized Trials– Quasi: natural experiment– Difference-in-difference: compare with business as

usual (the whole population minus the experimental)

Pretest-posttest modelCompare the experimental groups with similar

control or reference groupsEffects can exclusively be ascribed to intervention

BUT:

• The rationale (the why) for these effects remains unknown: black box

• Explanatory evaluation focuses specifically on this and can further validate the results of an impact assessment (effect evaluation)

• Questions addressed:– How and why does an intervention work?– If there is no or a smaller effect than expected,

why is that the case?– Which unwanted / unintended effects occur

(also unexpected positive)?

The development of the methodology

• Literature study, especially Pawson, R. & Tilley, N. (1997). Realistic Evaluation. London: SAGE Publications

• Study of two Dutch policy interventions in the area of early school leaving – VM2: prevent drop-out– De Wijkschool: stimulate drop-outs because of

multiple problems (drugs, criminality, debts, broken families, etc) to go back to school

Effects of VM2

• VM2– School drop-out in first year vocational

education enormous– Intervention: combine systems of preparatory

and regular vocational education– Effects: compare experimental group with

population as a whole on numbers of dropouts– Conclusion: less dropout… “in xx percentpoint

of the cases a significant lower rate of dropouts was the result”…

– How to find more differential effects and how to understand them better?

Explanatory evaluation (EE)

• Realistic Evaluation: get under the surface of the direct observable...

• What works for whom in what circumstances and how and why?

• Ex ante (before), ex durante (during), ex post (after)

• Central to answering this question is the reconstruction of the policy (or program) theory

• The policy theory is the sum of assumptions or hypotheses that explain how the policy should work These can be represented in statements like:

• If (intervention) ... Then ... (Outcome), and in some cases also conditions But (take care that…)

EE: contexts differ

• In the reconstruction of the theory, it is important to have an eye for the conditions and factors that play a role in a variety of contexts.

• The context (or conditions) can be physical, but also social, for example, certain groups or characteristics of target groups (an intervention works only for motivated students).

• Attention to circumstances helps explain why an intervention works better in one case than in the other case.

• The intervention works through certain mechanisms, mechanisms that make "things work" and can be seen as the driving force behind an intervention.

EE• Analysis of policy documents• Creation of concrete causal generative schemes

(=hypotheses) on the basis of this analysis • Context-Mechanism-Outcome (CMO) schemes

– Problem mechanisms, leading to unwanted outcomes or results in (parts of the) society

– Intervention that generates Change Mechanism(s)– Change mechanisms overcome problem mechanisms– Better outcomes

• Check with staff members (at Ministry) that were responsible for policy intervention

• Adjust CMO-schemes, make new CMO-schemes

EE: field studies

• Test the theory, test the hypotheses (CMO)• Enter the field studies. The researcher

must have the different groups of actors in the picture.

• Consider what information to whom can be achieved, what should be asked from whom: who knows what?

• Formulate relevant questions for each (type of) informant to answer and think about the most appropriate method for each type involved persons.

Field studies

• Reconstruction and comparison with the policy theory takes place in a learning dialogue at different levels.

• Types (levels) of respondents:– Policy implementers (e.g. project leaders)– Intermediaries (e.g. teachers)– Target group (e.g. students, elderly, etc)

• Researchers: open attitude but focus on understanding and consensual knowledge (do we agree that this is what really happened?)

EE: Field studies

• Each actor from his own role in the operation of the policy can reflect on assumed mechanisms and give an explanation of (un) planned and (un) desired effects.

• Question to the dialogue partners is– whether the assumptions indeed work for them– are assumptions about their behaviour correct,– what side effects they face.

• Cyclical process: stop rules...

EE: Critical evaluation

• Here the explanations are found why the policy intervention has or has not the desired effects, the goal of an explanatory evaluation.

• If the effects of an intervention are not as positive as expected, the researcher may ask the following questions: – Is the theory or policy measure(s) plausible?– Was the theory sufficiently differentiated? – Was implementation successful?– Were necessary conditions met? – What should be modified?

• Learning effects for policy makers…

Can EE work for educational practice?

R&D in education

• Development interventions/innovations• Do you measure effects (outcomes) and

how? Pre-test/post-test?• Compare with reference/control groups?• What do you measure?

– Knowledge, skills, attitudes, higher order skills (reflection, learn to learn, learn to see activities as learning), motivation? Other, new things?

• Find out what works: outcomes, effects• Find out what works for whom in what

contexts and why

Explanatory evaluation

• What is the theory behind the Development?• Reconstruct the assumptions (ifthen/CMO)• Distinguish assumptions at levels:

– The ideas, the theories– The structures and systems developed and used– The anticipated effects on various groups of

actors• Do field research• Evaluate whether your assumptions are

shared with various actors at different levels: test the hypotheses

Generic interview scheme

• THEN – NOW – LATER• Contexts-Mechanisms-Outcomes• Start with NOW• What Outcomes realised? Differentiate (level

of ideas-theories, structures-systems, behaviour)

• THEN: What Hypotheses? What Processes (mechanisms) caused the Outcomes?

• What crucial success & failure factors?• LATER: what new/adjusted outcomes?• How to achieve the outcomes?

PAST PRESENT FUTURE

1) What Outcomes are now realized? a) ideas/theories

b) structures/new roles c) the workfloor

Differences in Contexts?

2) Go back to start of project

3) What Hypotheses? What Mechanisms

Caused the Outcomes?Success and failure

factors?

4) Go to end of project

5) What Outcomesare expected?

What to be realized? What new ideas/theories? What success and failure?

6) What Interventions and Mechanisms

are needed to achieve the Outcomes?

????

??