Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation...
-
Upload
lesley-lawson -
Category
Documents
-
view
215 -
download
2
Transcript of Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation...
![Page 1: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/1.jpg)
Will your study have enough power?
CYRUS SAMIINEW YORK UNIVERSITY
ieGovern Impact Evaluation WorkshopIstanbul, TurkeyJanuary 27-30, 2015
: #ieGovern
![Page 2: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/2.jpg)
Definition
THE POWER OF YOUR STUDY MEASURES THE CONFIDENCE YOU CAN HAVE THAT IT WILL LEAD YOU TO DRAW THE RIGHT CONCLUSION.
![Page 3: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/3.jpg)
What we will not do today
![Page 4: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/4.jpg)
What we will not do today
![Page 5: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/5.jpg)
Definition
THE POWER OF YOUR STUDY MEASURES THE CONFIDENCE YOU CAN HAVE THAT IT WILL ALLOW YOU TO DRAW THE RIGHT CONCLUSION.
BUT WHY MIGHT YOU DRAW THE WRONG CONCLUSION?
![Page 6: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/6.jpg)
Definition
THE POWER OF YOUR STUDY MEASURES THE CONFIDENCE YOU CAN HAVE THAT IT WILL ALLOW YOU TO DRAW THE RIGHT CONCLUSION.
BUT WHY MIGHT YOU DRAW THE WRONG CONCLUSION?
THE WORLD IS MESSY.
![Page 7: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/7.jpg)
![Page 8: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/8.jpg)
Definition
POWER = EFFECT SIZE + NUMBERS - MESSY
![Page 9: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/9.jpg)
Messy
“WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?”
![Page 10: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/10.jpg)
Messy
“WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?”
EFFECTS ARE MESSY.
![Page 11: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/11.jpg)
Messy
WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?
DEPENDS ON THE KIND OF PERSON: “EAGER BEAVER”
“NORMAL BUREAUCRAT”
“LAZY SOB”
![Page 12: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/12.jpg)
Messy
WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?
Type of person With incentive Without incentive Effect of incentive
Eager beaver 10 cases/day 10 cases/day 0
Normal bureaucrat 10 cases/day 2 cases/day 8
Lazy SOB 2 cases/day 2 cases/day 0
![Page 13: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/13.jpg)
![Page 14: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/14.jpg)
Messy
WHAT IS THE EFFECT OF AN INCENTIVE SCHEME?
IF INCENTIVE SCHEME APPLIED TO WHOLE DEPARTMENT:
EFFECT = (8 X 0) + (8 X 8) + (8 X 0) = 64 MORE CASES/DAY
Type of person With incentive Without incentive
Effect of incentive
Number of these types?
Eager beaver 10 cases/day 10 cases/day 0 8
Normal bureaucrat
10 cases/day 2 cases/day 8 8
Lazy SOB 2 cases/day 2 cases/day 0 8
![Page 15: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/15.jpg)
Numbers and messy
SUPPOSE OUR STUDY WORKS WITH 2 OFFICES.
1 GETS THE INCENTIVE SCHEME AND 1 DOESN’T.
THEN WE MEASURE THE PRODUCTIVITY EFFECT.
WHAT COULD HAPPEN?
![Page 16: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/16.jpg)
32 16
16 X 6 = 96
![Page 17: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/17.jpg)
40 16
24 X 6 = 144
![Page 18: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/18.jpg)
16 8
8 X 6 = 48
![Page 19: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/19.jpg)
16
40
-24 X 6 = -144
![Page 20: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/20.jpg)
Numbers and messy
IF WE ONLY WORK WITH 2 OFFICES, WE HAVE A STRONG POSSIBLY OF DRAWING WILDLY MISTAKEN CONCLUSIONS.
![Page 21: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/21.jpg)
Numbers and messy
WHAT IF WE WORK WITH ALL SIX OFFICES?
![Page 22: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/22.jpg)
32 16
40
16
16
8
2 X 48 = 96
![Page 23: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/23.jpg)
24 32
40
8
40
16
2 X 16 = 32
![Page 24: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/24.jpg)
24 32
40
8
16
16
2 X 40 = 80
![Page 25: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/25.jpg)
32 32
40
8
16
16
2 X 16 = 32
![Page 26: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/26.jpg)
Numbers and messy
COMPARE:
![Page 27: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/27.jpg)
![Page 28: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/28.jpg)
Numbers and messy
WITH EVEN MORE:
![Page 29: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/29.jpg)
POWER = EFFECT SIZE + NUMBERS - MESSY
![Page 30: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/30.jpg)
POWER = EFFECT SIZE + NUMBERS - MESSY
Numbers help to compensate for messy.
![Page 31: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/31.jpg)
POWER = EFFECT SIZE + NUMBERS - MESSY
Numbers help to compensate for messy.
Sometimes increasing numbers is straighforward.
Sometimes you need to be creative:• For example, information campaigns experiments for
universal policies.
![Page 32: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/32.jpg)
POWER = EFFECT SIZE + NUMBERS - MESSY
Other things you can do to reduce the problem of messy:• Stratification techniques• Analytical techniques (regression adjustment)
![Page 33: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/33.jpg)
POWER = EFFECT SIZE +
NUMBERS - MESSY
![Page 34: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/34.jpg)
POWER = EFFECT SIZE +
NUMBERS - MESSY
If the effect is REALLY large or small, then even with small numbers and lots of messy you can draw reasonable conclusions.
But if the effect is small, you need lots of numbers and ways to reduce messy to draw good conclusions.
![Page 35: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/35.jpg)
POWER = EFFECT SIZE +
NUMBERS - MESSYOf course, you do not know the effect size.
So, what assumption should you make about it?
Define a minimal desirable effect that would lead you to adopt the intervention you are studying.
You want to have power to detect this minimal desirable effect.
![Page 36: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/36.jpg)
![Page 37: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/37.jpg)
Formally…
Minimal detectable effect
Test critical value Power critical value
Effect estimator standard deviation (“messy”) as a function of sample size (numbers)
![Page 38: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/38.jpg)
Conclusion
THE WORLD IS MESSY. EFFECTS ARE MESSY.
RELIABLE EVIDENCE REQUIRES THAT WE HAVE ADEQUATE NUMBERS. SOMETIMES WE NEED TO BE CREATIVE WITH THIS.
WE WANT ADEQUATE POWER SO WE DON’T WASTE $$$ ON UNRELIABLE OR INCONCLUSIVE STUDIES.
![Page 39: Will your study have enough power? C YRUS S AMII N EW Y ORK U NIVERSITY ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 : #ieGovern.](https://reader038.fdocuments.in/reader038/viewer/2022110322/56649d175503460f949ed540/html5/thumbnails/39.jpg)
RESEARCH DESIGN COURSE MATERIALS:
HTTP://CYRUSSAMII.COM/?PAGE_ID=1603