Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact...
Transcript of Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact...
![Page 1: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/1.jpg)
Introduction to randomized impact evaluations
Seema Jayachandran Northwestern University
USI Matchmaking conference – July 2012 – Colombo
![Page 2: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/2.jpg)
What is Evaluation?
Evaluation
Program Evaluation
Impact Evaluation
![Page 3: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/3.jpg)
Evaluation
Program Evaluation
Impact Evaluation
Program Evaluation
![Page 4: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/4.jpg)
Evaluation
Program Evaluation
Impact Evaluation
Monitoring and Evaluation
Monitoring
![Page 5: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/5.jpg)
Monitoring
Evaluation
Program Evaluation
Impact Evaluation
Program Evaluation
![Page 6: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/6.jpg)
Components of Program Evaluation
• Needs Assessment
• Program Theory Assessment
• Process Evaluation
• Impact Evaluation
• Cost Effectiveness
• What is the problem?
• How, in theory, does the program fix the problem?
• Does the program work as planned?
• Were its goals achieved? The magnitude?
• Given magnitude and cost, how does it compare to alternatives?
![Page 7: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/7.jpg)
Who is this evaluation for?
• Academics • Donors
– Their Constituents
• Politicians / policymakers • Technocrats • Implementers • Beneficiaries • Proponents, Skeptics
![Page 8: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/8.jpg)
How can impact evaluation help us?
• Surprisingly little hard evidence on what works • Can do more with given budget with better evidence • If people knew money was going to programs that
worked, could help increase pot for anti-poverty programs
• Instead of asking “do aid/development programs work?” should be asking: – Which work best, why and when? – How can we scale up what works?
![Page 9: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/9.jpg)
Programs and their Evaluations: where do we start?
Intervention
• Start with a problem • Verify that the problem
actually exists • Generate a theory of why the
problem exists • Design the program • Think about whether the
solution is cost effective
Program Evaluation
• Start with a question • Verify the question hasn’t
been answered • State a hypothesis
• Design the evaluation • Determine whether the value
of the answer is worth the cost of the evaluation
![Page 10: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/10.jpg)
IMPACT EVALUATION Measuring how well it worked
![Page 11: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/11.jpg)
Did we achieve our goals?
• Primary outcome (impact) • Example: Does intervention to protect
springs reduce child diarrhea?
• Also distributional questions: • Example: What was the impact for
households with good v. bad sanitation practices?
![Page 12: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/12.jpg)
How to measure impact?
Impact is defined as a comparison between:
1. the outcome some time after the program has been introduced
2. the outcome at that same point in time had the program not been introduced (the “counterfactual”)
![Page 13: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/13.jpg)
What is the impact of this program?
Time
Prim
ary
Out
com
e
Impact
Counterfactual
Program starts
![Page 14: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/14.jpg)
Impact: What is it?
Time
Prim
ary
Out
com
e
Impact Counterfactual Program starts
![Page 15: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/15.jpg)
Impact: What is it?
Time
Prim
ary
Out
com
e
Impact Counterfactual
Program starts
![Page 16: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/16.jpg)
Counterfactual
• The counterfactual represents the state of the world that program participants would have experienced in the absence of the program (i.e. had they not participated in the program)
• Problem: Counterfactual cannot be observed • Solution: We need to “mimic” or construct
the counterfactual
![Page 17: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/17.jpg)
Constructing the counterfactual
• Usually done by selecting a group of individuals that did not participate in the program
• This group is usually referred to as the control group or comparison group
• How this group is selected is a key decision in the design of any impact evaluation
![Page 18: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/18.jpg)
Selecting the comparison group
• Idea: Select a group that is exactly like the group of participants in all ways except one: their exposure to the program being evaluated
• Goal: To be able to attribute differences in outcomes between the group of participants and the comparison group to the program (and not to other factors)
![Page 19: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/19.jpg)
Non random assignment
HQ
2006
Income per person, per month
1000
500
0 Treat Compare
1457
947
![Page 20: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/20.jpg)
Impact evaluation methods
1. Randomized Experiments • Also known as:
– Random Assignment Studies
– Randomized Field Trials
– Social Experiments
– Randomized Controlled Trials (RCTs)
![Page 21: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/21.jpg)
Impact evaluation methods
2. Non- or Quasi-Experimental Methods a. Pre-Post b. Simple Difference c. Differences-in-Differences d. Multivariate Regression e. Statistical Matching f. Interrupted Time Series g. Instrumental Variables h. Regression Discontinuity
![Page 22: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/22.jpg)
Constructing the Counterfactual
• Randomized: – Use random assignment of the program to
create a control group which mimics the counterfactual.
• Non-randomized: – Argue that a certain excluded group mimics
the counterfactual.
![Page 23: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/23.jpg)
Random assignment
2006
Income per person, per month
1000
500
0 Treat Compare
1457 1442
![Page 24: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/24.jpg)
How impact differs from process?
• When we answer a process question, we need to describe what happened.
• When we answer an impact question, we need to compare what happened to what would have happened without the program
![Page 25: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/25.jpg)
RANDOMIZED EVALUATION The “gold standard” for Impact Evaluation
![Page 26: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/26.jpg)
The basics
Start with simple case: • Take a sample of program applicants • Randomly assign them to either:
§ Treatment Group – is offered treatment § Control Group - not allowed to receive
treatment (during the evaluation period)
![Page 27: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/27.jpg)
Some variations on the basics
• Assigning to multiple treatment groups – Example: Information campaign on treating
water + Provision of public water taps
• Assigning of units other than individuals or households
§ Schools § Local Governments § Villages or neighborhoods
![Page 28: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/28.jpg)
Random Sampling and Random Assignment
Randomly sample from area of interest
![Page 29: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/29.jpg)
Randomly sample from area of interest
Randomly assign to treatment and control
Random Sampling and Random Assignment
Randomly sample from both treatment and control (Example: households within neighborhood)
![Page 30: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/30.jpg)
Key advantage of experiments
Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment,
any difference that subsequently arises between them can be attributed to the program rather than to other factors.
30
![Page 31: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/31.jpg)
“Piped water to households in Morocco”: Treatment vs. Control villages at baseline
Variables Control Group
Treatment Group
Difference Difference in
Std Deviations
Household size 4.55 4.73 -0.18 7%
Income index 4.21 4.42 -0.21 8%
Index of knowledge of diarrhea causes
1.25 1.23 0.02 2%
Main water source = public tap
0.44 0.43 0.01 2%
Distance to public tap (meters)
139 134 5 3%
Source: Devoto et al. (2011)
![Page 32: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/32.jpg)
Key steps in conducting an experiment
1. Design the study carefully
2. Randomly assign people to treatment or control
3. Collect baseline data
4. Verify that assignment looks random
5. Monitor process so that integrity of experiment is not compromised
![Page 33: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/33.jpg)
Key steps in conducting an experiment (cont.)
6. Collect follow-up data for both the treatment and control groups
7. Estimate program impacts by comparing mean outcomes of treatment group vs. mean outcomes of control group.
8. Assess whether program impacts are statistically significant and practically significant.
![Page 34: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/34.jpg)
Example
![Page 35: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/35.jpg)
The Problem
• 13% of world population lacks access to “improved water sources”
• Nearly 2 million children die each year from diarrhea
• 20% all child deaths (under 5 years old) are from diarrhea
![Page 36: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/36.jpg)
7/2009 Spring Cleaning - SITE 36
![Page 37: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/37.jpg)
The Goal
• MDG: “reduce by half the proportion of people without access to sustainable drinking water”
![Page 38: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/38.jpg)
The Solution(s)
![Page 39: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/39.jpg)
Spring cleaning in Kenya
• Key question: What is the impact of the clean springs program on the rate of water borne illness?
• Methodological Question: How should we estimate the impact of the program?
![Page 40: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/40.jpg)
Spring Cleaning Sample
Target Population
(200)
Not in evaluation
(0)
Evaluation Sample
(200)
Total Population
(562 springs)
Random Assignment
Year 2 (50)
Years 3,4 (100)
Year 1 (50)
![Page 41: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/41.jpg)
Data collection
• Data from both treatment and control areas
• Measured water quality (E. coli) at both springs and houses
• Representative sample of households using a spring (7-8 households per spring)
• Household survey: child anthropometrics, mother-reported diarrhea, hygiene practices, etc.
![Page 42: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/42.jpg)
Impact of clean springs intervention
• 66% reduction in source water E. coli concentration
• 24% reduction in household E. coli concentration
• 25% reduction in incidence of diarrhea for children under 3 years old
• No significant change for 5-12 year olds
![Page 43: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/43.jpg)
Is contaminated source water really the main problem?
• Quantity of water: Could be a stronger determinant of health than quality of water (Curtis et al, 2000)
• Hygiene: Water quality helps little without hygiene (Esrey, 1996) – 42% in the Kenya live without a toilet at home
• Low demand: People are more willing to pay for convenient water than clean water
![Page 44: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/44.jpg)
Alternative Solution(s)?
![Page 45: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/45.jpg)
Making Policy from Evidence
Intervention Impact on Diarrhea
Spring protection (Kenya) 25% reduction in diarrhea incidence for ages 0-3
![Page 46: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/46.jpg)
Making Policy from Evidence
Intervention Impact on Diarrhea
Spring protection (Kenya) 25% reduction in diarrhea incidence for ages 0-3
Source chlorine dispensers (Kenya) 20-40% reduction in diarrhea
Home chlorine distribution (Kenya) 20-40% reduction in diarrhea
Hand-washing (Pakistan) 53% drop in diarrhea incidence for children under 15 years old
Piped water in (Urban Morocco) 0.27 fewer days of diarrhea per child per week
![Page 47: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/47.jpg)
III – Why randomize?
![Page 48: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/48.jpg)
Why randomize? – Conceptual Argument
If properly designed and conducted, randomized experiments provide the most credible method to estimate the impact of a program
48
![Page 49: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/49.jpg)
Why “most credible”?
Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment,
any difference that subsequently arises between them can be attributed to the program rather than to other factors.
49
![Page 50: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/50.jpg)
IV – Conclusions
![Page 51: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/51.jpg)
• There are many ways to estimate a program’s impact
• This workshop argues in favor of one: randomized experiments – Conceptual argument: If properly designed and
conducted, randomized experiments provide the most credible method to estimate the impact of a program
– Empirical argument: Different methods can generate different impact estimates
Conclusions - Why Randomize?
![Page 52: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/52.jpg)
When to do a randomized evaluation?
• When there is an important question you want/need to know the answer to
• Timing--not too early and not too late
• Program is representative not gold plated – Or tests a “proof of concept”
• Time, expertise, and money to do it right
• Develop an evaluation plan to prioritize
![Page 53: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/53.jpg)
When NOT to do an RE
• When the program is premature and still requires considerable “tinkering” to work well
• When the project is on too small a scale to randomize into two “representative groups”
• If a positive impact has been proven using rigorous methodology and resources are sufficient to cover everyone
• After the program has already begun and you are not expanding elsewhere
![Page 54: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/54.jpg)
If you ask the right question, you’re more likely to care
• Start with a question • Verify the question hasn’t been answered • State a hypothesis • Design the evaluation • Determine whether the value of the answer is worth the
cost of the evaluation
• With key questions answered from impact evaluations, process evaluation can give your overall impact
• A few high quality impact studies are worth more than many poor quality ones
Developing an evaluation strategy
![Page 55: Introduction to randomized impact evaluations · PDF fileIntroduction to randomized impact evaluations Seema Jayachandran Northwestern University USI Matchmaking conference – July](https://reader034.fdocuments.in/reader034/viewer/2022051720/5a77add57f8b9a0d558e1886/html5/thumbnails/55.jpg)
Questions?