Better Decision-Making with RCTs: A Guide for Financial Service Providers.
-
Upload
pauline-harrell -
Category
Documents
-
view
213 -
download
0
Transcript of Better Decision-Making with RCTs: A Guide for Financial Service Providers.
![Page 1: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/1.jpg)
Better Decision-Making
with RCTs:A Guide for Financial Service Providers
![Page 2: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/2.jpg)
2
Overview
• Who we are• Experimentation with Randomized
Controlled Trials–What is an RCT– Choosing metrics– Logistics–When are RCTs useful/when are they not
• RCT Toolkit
November 5, 2015
![Page 3: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/3.jpg)
Introductions
November 5, 2015
Julia BrownProgram Manager
Lucia GoinResearch Associate
Nora GregoryResearch Associate
Kim Smith Sr Research Associate
Jonathan Zinman, DartmouthScientific Director
Dean Karlan, Yale
![Page 4: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/4.jpg)
4
Where we work
51 CountriesSupported by 18 Country ProgramsNovember 5, 2015
![Page 5: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/5.jpg)
5
US Finance Initiative
26 Projects (8 RCTs, 18 Pilots)22 Partner OrganizationsNovember 5, 2015
![Page 6: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/6.jpg)
6
THE PROBLEM
November 5, 2015
Limited Evidence
on what works best
Limited Useof available
evidence
Ineffective Policies
and Programs
HiPPO Thinking
But we’ve always done it that way!
![Page 7: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/7.jpg)
7
THE SOLUTION
November 5, 2015
Better Policies
& Program
s
MORE EVIDENCE, LESS POVERTY
Design & Evaluate
potential solutions to poverty problems
Mobilize & Support
decision-makers to use evidence
![Page 8: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/8.jpg)
8
What is Impact?
November 5, 2015
IMPACTof current practices
IMPACT of new • Products/services• Features• Marketing
On Your Bottom Line • On Your Clients • On Your Mission
![Page 9: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/9.jpg)
9
What is Impact?
November 5, 2015
Savi
ngs
Bala
nces
Time
Program Start
![Page 10: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/10.jpg)
10
What is Impact?
November 5, 2015Correlation: 0.947091
Per capita consumption of cheese (US)Number of people who died by becoming entangled in their bedsheets
![Page 11: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/11.jpg)
11November 5, 2015
Aver
age
Inco
me
Time
Program Start
Received Intervention
Did Not Receive Intervention
![Page 12: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/12.jpg)
12
The Counterfactual
• The counterfactual represents the state of the world that program participants would have experienced in the absence of the program (i.e. had they not participated in the program)
• Problem: Counterfactual cannot be observed
November 5, 2015
![Page 13: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/13.jpg)
13
• Solution: We need to “mimic” or construct the counterfactual
November 5, 2015
The Counterfactual
![Page 14: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/14.jpg)
14
Randomized Controlled Trials
November 5, 2015
COMPARISON
![Page 15: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/15.jpg)
15November 5, 2015
• Impact is defined as a comparison between:
1. the average outcome for people who were in the program
--and--
2. the average outcome for the comparison group Average savings balances for those who received text
messages: $550― Average savings balances for those in the comparison group: $325 Impact of text messages on savings:
$225
![Page 16: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/16.jpg)
16November 5, 2015
![Page 17: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/17.jpg)
17
Selecting Outcomes & Metrics
November 5, 2015
Outcome - What do you care about?
• Improved savings behavior
Metric - How is it measured?
• Number of savings deposits per month• Average size of savings deposit• Average daily balance of savings account
![Page 18: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/18.jpg)
18
Selecting Metrics
November 5, 2015
OUTPUTS OUTCOMES
• Account openings
• Message receipt
• Web visits
• Savings• Repayment• Income
![Page 19: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/19.jpg)
19
Selecting Data
November 5, 2015
• Administrative data Demographic data Transaction data
• Credit report data
• Survey data Overall financial health
![Page 20: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/20.jpg)
20
Selecting Metrics
November 5, 2015
Collect data that your organization will use
Set thresholds for success and failure before you start
![Page 21: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/21.jpg)
21
Logistics – How many people do you need?
November 5, 2015
Intervention Comparison0
20
40
60
80
100
Savings Balances
![Page 22: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/22.jpg)
22
Logistics – How many people do you need?
November 5, 2015
Intervention Comparison0
20
40
60
80
100
Savings Balances
![Page 23: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/23.jpg)
23
Logistics – How many people do you need?
November 5, 2015
Intervention Comparison0
20
40
60
80
100
Savings Balances
![Page 24: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/24.jpg)
24
Logistics – How many people do you need?
November 5, 2015
![Page 25: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/25.jpg)
25
Logistics - Randomization
November 5, 2015
![Page 26: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/26.jpg)
26
Example: Super Savers CD
November 5, 2015
1) Identify target population
2) Randomizea. Control
b. Counseling
c. Savings
3) Collect outcome data
![Page 27: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/27.jpg)
27
When NOT to Randomize
• Sample size
• Staff capacity
• Intervention not ready to be tested
• No feasible way of randomizing
• No way of measuring outcomes
• Organization not prepared to respond to results
November 5, 2015
![Page 28: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/28.jpg)
28
When is the Right Time to Randomize?
• When you have a good sense of needs, program theory, and process
• When there is an important, specific and testable question you want/need to know the answer to
• Time, expertise, and money to do it right• Appropriate sample size• Results can be used to inform programming and
policy November 5, 2015
![Page 29: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/29.jpg)
29
The Role for Researchers
• Sort through trickier RCT designs• Outsource some of the work • Objectivity and credibility• Creativity and experience
November 5, 2015
![Page 30: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/30.jpg)
30
A Toolkit for RCTs
November 5, 2015
![Page 31: Better Decision-Making with RCTs: A Guide for Financial Service Providers.](https://reader035.fdocuments.in/reader035/viewer/2022062805/5697bfe91a28abf838cb6fc2/html5/thumbnails/31.jpg)
Thank You!