Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D....
Transcript of Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D....
![Page 1: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/1.jpg)
Do Students Know Best? Choice, Classroom Time and Academic Performance
Ted Joyce Baruch College and Graduate Center, CUNY and NBER
Sean Crockett
Baruch College and Graduate Center, CUNY
David A. Jaeger Graduate Center, CUNY, University of Cologne, CESifo, and NBER
Onur Altindag
Graduate Center, CUNY
Stephen D. O’Connell Graduate Center, CUNY
Dahlia Remler
Baruch College and Graduate Center, CUNY and NBER
![Page 2: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/2.jpg)
Effect of Classroom Time
• Does time in the classroom matter for student outcomes? – Given rich online materials available today
• Randomized experiments solve biased treatment selection problem – Irreplaceable – Notation: Randomized Clinical Trial (RCT)
![Page 3: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/3.jpg)
Randomized Experiments: Limitations in a World of Choice
• Unwillingness to participate in RCT less generalizable
• Students get to choose format – Being able to choose motivation – Students choose based on what works for them
(heterogeneous treatment effects)
• Time, expense limit RCTs… – How close can observational studies come?
![Page 4: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/4.jpg)
Research Questions • What is the effect of time in the classroom on
student outcomes? • Do students sort into different learning formats?
– What factors drive sorting?
• Does that sorting affect the estimates of classroom time on performance? – Parallel experimental and observational study – How close can observational estimates come? – Do students know best? [can’t fully answer…]
![Page 5: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/5.jpg)
Outline of Talk – Part I
• Review key studies on online effects
• Introduce the DRPT—outline our “quasi-DRPT”
• Review Potential Outcomes Framework
• Evidence to support quasi-DRPT assumptions
![Page 6: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/6.jpg)
Outline of Talk – Part 2
• Econometric model: experimental v. observational effects
• Selection – Survey about likely selection drivers
• Results by class size • Implications
![Page 7: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/7.jpg)
Several Relevant Literatures
• Effect of Attendance on performance (Romer 1993; Dobkin et al. 2010) – 1 GPA (Romer); -.17 SD (Dobkin)
• Large database analyses (Xu & Jaggers 2013;
Bettinger et al. 2014; Hart et al. 2014) -.14 SD (X-J); -10 pts (H); -0.09 , IV=-0.28 SD
![Page 8: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/8.jpg)
Several Relevant Literature (cont)
• RCTs of online learning: (Figlio et al. 2013;
Bowen et al. 2014; Alpert et al. 2015; Joyce et al. 2015)
2-3 pts (F); no diff (B); no diff blended (A)
Participation issues
![Page 9: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/9.jpg)
Several Relevant Literature (cont)
• Doubly Randomized Preference Trials (DRPT) • Medical literature (Bradley 1993)
• Janevic et al. (2003); • Shadish et al. (JASA 2008); Marcus et al
(2012) • Long, Little, Lin (JASA 2008a,2008b)
![Page 10: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/10.jpg)
Doubly Randomized Preference Trial
Consent
Randomization
Randomization
Experiment Treated
Controls
Choice Treated
Controls
![Page 11: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/11.jpg)
DRPT uses • Observational and experimental estimates
generalize to same population
• Can selection bias can be eliminated in observational studies...which variables needed
• How effects (experimental estimates) differ by preference – & from choosing
Cites: Shadish, Clark & Steiner (JASA 2008); Janevic et al (2003); Marcus et al (2012); Long, Little & Lin (2008a, b); Cook, Shadish & Wong (2008)
![Page 12: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/12.jpg)
Our Quasi DRPT
• Run a Randomized Experiment in fall of 2013
• Repeat study in fall of 2014 – Identical except students choose format
• In 2014 we do a detailed baseline survey: Can we predict format choice with a rich set of covariates and “eliminate” Omitted Variables Bias?
![Page 13: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/13.jpg)
Recap of Results from RCT Joyce, et al. Econ of Ed. Review, June 2015
• Randomized intro econ students into – Treatment: format with a 50% reduction in class time – Control: traditional – Providing online content to both groups
• Primary Results: – Class time matters: treatment group scored average
2.5 percentage points lower on exams (about .2 SDs) – Relatively little evidence of treatment heterogeneity – No differences in non-cognitive effort (attendance, time
spent with online materials), nor withdrawal. – Experimental estimates of attendance are about ¼ of
observational estimates from past studies
![Page 14: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/14.jpg)
Micro Principles (ECO 1001) • ECO 1001 required course for all students
applying to the business program – Enrollment ≈ 1000 students each fall – Over 100 are evening students, 25 Honors students
& another 40 students taught by adjunct faculty
• Professor A and Professor B – Both Taught ECO1001 for the past 6 years – Both have similar and strong teaching evaluations
• Registrants offered 5 points on final grade for participating
![Page 15: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/15.jpg)
Professors and Classrooms Monday Tuesday Wednesday Thursday Friday
Morning A/Small: Traditional
A/Small: Traditional
B/Large:
Compressed
Afternoon A/Small: Compressed
B/Large:
Traditional
B/Large: Traditional
![Page 16: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/16.jpg)
Research Design • Course: ECO 1001, Microeconomic Principles • Two class formats
– Traditional meets 2 x per week for 75 minutes each – “Compressed” meets 1 x per week for 75 minutes – Online materials identical for both formats (quizzes, videos) – Professors used same slides in both formats, but went faster
with less time for questions, etc. in the compressed format • Two professors, each teaching both formats
– Professor A: always in small classroom with 114 seats – Professor B: always in large classroom with 274 seats
• Two semesters of ECO 1001, Microeconomic Principles – Fall 2013: 725 Students were randomized into formats – Fall 2014: 766 Students chose format
![Page 17: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/17.jpg)
Doubly Randomized Preference Trial
Consent
Randomization
Randomization
Experiment Traditional
Compressed
Choice Traditional
Compressed
![Page 18: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/18.jpg)
Potential Outcomes Framework Define the causal effect for person “i” δi = (P1i – P0i) f=1 compressed format; f=0 traditional format Since δi is unobsevable we estimate the expected value
E(δi) = E(P1i – P0i)
![Page 19: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/19.jpg)
Potential Outcomes Framework: Randomized Arm
Pi = (1-di)*P0i + di*P1i di = 1 if compressed; di = 0 if traditional
E(Pi|di=1) = (P1i|di=1) E(Pi|di=0) = (Pi0|di=0) E{(Pi|di=1) - (Pi|di=0)} = E(P1i|di=1) - E(P0i|di=0)
δr = E(P1i|di=1) - (P0i|di=1) + E(P0i|di=1) - (P0i|di=0) Treatment of
Treated (TOT) Selection bias
![Page 20: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/20.jpg)
Potential Outcomes Framework: Choice Arm
Pi = (1-di)*P0i + di*P1i di = 1 if compressed and 0 traditional
E(Pi|X, di=1) = (P1i|X, di=1) E(Pi|X, di=0) = (Pi0|X, di=0) E(Pi|X,di=1) - (Pi|X,di=0) = E(P1i|X,di=1) - (P0i|X,di=0) δc = E(P1i|X,di=1) - (P0i|X, di=1) + E(P0i|X,di=1) - (P0i|X,di=0)
Treatment of Treated (TOT)
Selection bias
![Page 21: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/21.jpg)
Diff-in-Diff of Treatment Effects
𝛿𝛿𝑐𝑐 = 𝐸𝐸𝑥𝑥{𝐸𝐸 𝑃𝑃1𝑖𝑖 − 𝑃𝑃0𝑖𝑖 𝑋𝑋 }
𝛿𝛿𝑟𝑟 = 𝐸𝐸[𝑃𝑃1𝑖𝑖 − 𝑃𝑃0𝑖𝑖}
𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶 = 𝛿𝛿𝑟𝑟 − 𝛿𝛿𝑐𝑐
Treatment effect from randomized arm
Treatment effect from choice arm
![Page 22: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/22.jpg)
Why Might Choice Affect Outcomes? • Learning style may be better suited to one
format or another – If students choose the format that they expect will
improve their performance, then the effect of format likely will be less in the choice than in the randomized arm
• If students choose format for reasons such as convenience, it could increase difference in the treatment effect of format between the two designs
![Page 23: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/23.jpg)
Quasi Doubly Randomized Preference Trial (as Implemented)
Haphazard Randomization
Experiment (Fall 2013)
Choice (Fall 2014)
Randomization
Traditional
Compressed
Traditional
Compressed
Consent (96%)
Consent (100%)
![Page 24: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/24.jpg)
Haphazard Randomization into Format
• “Haphazard randomization” into format – Can’t postpone…
• Covariate balance betw semesters excellent
• No reason at all to think there are any systematic differences between students in the two semesters – Almost as good as doing DRPT in one semester, but
with more power
![Page 25: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/25.jpg)
Baseline Characteristics 2013 2014 Diff NormD
Baruch GPA 3.03 3.04 -0.01 -0.01
SAT Verbal 540.7 556.5 -15.8*** -0.13
SAT Math 604.0 608.7 -4.7 -0.04
Credits 44.59 44.88 -0.29 -0.01
Part time 0.07 0.08 -0.01 -0.01
White 0.27 0.29 -0.02 -0.03
Asian 0.45 0.46 -0.01 -0.02
Native English 0.53 0.55 -0.03 -0.04
*** p<.01
![Page 26: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/26.jpg)
Baseline Characteristics
![Page 27: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/27.jpg)
Baseline Characteristics Within 2013
![Page 28: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/28.jpg)
Balance Across Formats in Choice Year
• Formats balanced in randomized experiment year (2013), of course
• Balance across formats in choice year (2014) – For variables measured in both years – No treatment selection bias seen in variables
strongly predictive of performance • Math SAT, GPA…
– From this evidence, could still be selection bias in survey variables (2014 only) or unmeasured variables
![Page 29: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/29.jpg)
Baseline by Format: Choice Comp Trad Diff NormD
Baruch GPA 3.05 3.03 0.02 0.03
SAT Verbal 555.2 557.8 -2.6 -0.02
SAT Math 604.9 612.4 -7.5 -0.06
Credits 44.76 44.99 -0.24 -0.01
Part time 0.07 0.09 -0.01 -0.04
White 0.28 0.30 -0.02 -0.03
Asian 0.46 0.47 -0.01 -0.01
Native English 0.59 0.53 0.06 0.08
*** p<.01
![Page 30: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/30.jpg)
Baseline Characteristics within 2014
![Page 31: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/31.jpg)
Selection into Compressed
![Page 32: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/32.jpg)
Outline of Talk – Part 2
• Choice effect: – How does effect of compressed vary choice v.
experiment?
• Preference effect: – How does effect of compressed vary by
preference for compressed v. hybrid?
• Selection: What factors drive format choice? • Results by class size • Implications
![Page 33: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/33.jpg)
Econometric Specification
Difference between randomized and choice estimates, controlling for variables available in both years: Choice Effect
δ = δr − δc
![Page 34: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/34.jpg)
Student Performance Measures • Midterm:
– % of 30 multiple choice questions answered correctly • Final
– % of 40 multiple choice questions answered correctly • Midterm+Final
– % of 70 Midterm and Final questions answered correctly
• Overall Grade: – Curved Midterm (35%) + curved Final (45%) + Aplia
(20%) + 5% points experiment participation bonus. • Also: Withdrawal
![Page 35: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/35.jpg)
Effect of Class Time on Student Performance
![Page 36: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/36.jpg)
Cognitive Outcomes: 2013 vs. 2014¥ (standard deviations)
Joyce et al. (2014) 36 1/23/2014
¥All estimates adjusted for student characteristics
Midterm Final MT+F Overall
2013 (δr) -0.20*** -0.14 -0.19** -0.20***
2014 (δc) -0.12 -0.13* -0.14* -0.14**
(δr-δc) -0.08 -0.01 -0.05 -0.07 N 1,332 1,333 1,333 1333
![Page 37: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/37.jpg)
Cognitive Outcomes: 2013 vs. 2014¥ (standard deviations)
Joyce et al. (2014) 37 1/23/2014
¥All estimates adjusted for student characteristics; includes only those with preferred choices in 2014
Midterm Final MT+F Overall
2013 (δr) -0.19*** -0.13* -0.18*** -0.19***
2014 (δc) -0.13* -0.09 -0.12* -0.12*
(δr-δc) -0.06 -0.04 -0.06 -0.07
N 1,243 1,244 1,243 1243
![Page 38: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/38.jpg)
Expressed Preferences • At each semester start, students asked what format
they preferred
• In Choice year, 14% (of those who expressed preference) did not get preference, roughly equal between formats
• In Experimental year, being randomized to a
format made you decide you liked it: – In compressed arm: 77% preferred compressed – In traditional arm: 43% preferred compressed – Better to ask before assignment, if possible
![Page 39: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/39.jpg)
Preference Differences
Pref dummies are: Actual_Preferred Omitted category: Got Traditional, Preferred Traditional (TT)
Does the compressed effect vary by (expressed) preferred format?
![Page 40: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/40.jpg)
Midterm Final MT+F Overall Got Comp, Want Comp
-0.21** (0.08)
-0.17* (0.09)
-0.22** (0.09)
-0.17*** (0.06)
Got Comp, Want Trad
-0.26* (0.13)
-0.04 (0.14)
-0.16 (0.14)
-0.10 (0.11)
Got Trad, Want Comp
-0.04 (0.08)
-0.09 (0.09)
-0.08 (0.08)
-0.06 (0.06)
Comp effect, among want Comp
-0.17 (0.08)
-0.08 (0.09)
-0.14 (0.08)
-0.12 (0.07)
Comp effect Diff pref comp – pref trad)
-0.43 (0.17)
-0.11 (0.18)
-0.30 (0.17)
-0.22 (0.13)
N 605 605 605 605
Standard Errors in Parentheses; Only Known Preferences Omitted category: Got Trad, Want Trad
2013 (Experiment): Preference Differences
![Page 41: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/41.jpg)
Midterm Final MT+F Overall Got Comp, Want Comp
-0.13* (0.07)
-0.11 (0.08)
-0.13* (0.07)
-0.12** (0.05)
Got Comp, Want Trad
-0.15 (0.12)
-0.35** (0.15)
-0.30** (0.14)
-0.22** (0.10)
Got Trad, Want Comp
-0.18 (0.15)
0.01 (0.15)
-0.08 (0.15)
-0.09 (0.11)
Comp effect, among want Comp
0.05 (0.15)
-0.12 (0.16)
-0.06 (0.16)
-0.03 (0.11)
Comp effect Diff pref comp – pref trad)
-0.10 (0.20)
-0.47 (0.23)
-0.35 (0.22)
-0.26 (0.16)
N 648 648 648 648
2014 (Choice): Preference Differences
Standard Errors in Parentheses; Only Known Preferences Omitted category: Got Trad, Want Trad
![Page 42: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/42.jpg)
Results • Compressed effect about .2 SD • Effect sizes smaller when students choose
– but not statistically significantly
• Differences by preference in compressed effect suggest – Those who prefer traditional do better in
traditional • But little power, stat sig
• In theory, small selection bias on unmeasured variables possible
![Page 43: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/43.jpg)
Original Research Goal on Selection • Cook, Rubin on prospective observational
studies • Every possible approach to learn every
possible variable that might drive format choice (selection) – Conducted informal interviews
• Create and field survey to measure all variables prospectively
• See if can reproduce randomized experiment – But we came close without them!
![Page 44: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/44.jpg)
Survey: Hypothesized Format Drivers • Online vs. in-person orientation • Perceived strength in quant, non-quant courses • Need for structure • Attitudes towards risk • Hours worked, if any • Commute time
• Survey Instrument
– Some constructs had multiple indicators – Most on 5-point Likert response format (“Strongly
Agree,” “Agree,” “Neither Agree or Disagree,” “Disagree”, “Strongly Disagree”)
![Page 45: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/45.jpg)
Traditional Compressed P-value (χ2 )
My learning style well-suited suited to hybrid w/ less face-to-face class time
3.30 2.71 0.00***
Ever took hybrid/fully online 0.32 0.29 0.26
If yes, worked well 2.93 2.58 0.06*
Traditional lectures work well for me 2.30 2.51 0.02**
Prefer quant courses to writing courses
2.57 2.54 0.00***
Prefer electronic devices to read 3.38 3.23 0.16
Interaction w/ prof & other students helpful
1.81 2.12 0.00***
Commute time to school 2.42 2.52 0.30
Paid work 2.51 2.35 0.15
![Page 46: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/46.jpg)
Prior Student Learning Experiences: 2014
![Page 47: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/47.jpg)
Selection • Few variables drive format selection • Variables that drive format selection
– Mostly learning style – Uncorrelated w/ performance predictors (e.g.,
GPA, Math SAT) – Do not affect performance
• No detectible treatment selection bias! • In theory, small choice effect could be
contaminated by residual treatment selection bias due to unmeasured variables…. Unlikely
![Page 48: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/48.jpg)
Effects by Class Size/Professor
• Superb balance across format in class-size & professor
• But design makes it impossible to disentangle effect of professor from effect of class size
• We believe effect is class size, not professor • Professor A and Professor B
– Both Taught ECO1001 for the past 6 years – Both have similar and strong teaching evaluations
![Page 49: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/49.jpg)
Professors and Classrooms Monday Tuesday Wednesday Thursday Friday
Morning A/Small: Traditional
A/Small: Traditional
B/Large:
Compressed
Afternoon A/Small: Compressed
B/Large:
Traditional
B/Large: Traditional
![Page 50: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/50.jpg)
Prior Teaching Evaluations scale is 1 (worst) through 5 (best)
![Page 51: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/51.jpg)
Cognitive Outcomes: 2013 vs. 2014 Professor A/Small Class¥
(standard deviations)
Joyce et al. (2014) 51 1/23/2014
¥All estimates adjusted for student characteristics
Midterm Final MT+F Overall
2013 (δr) -0.12 -0.10 -0.12 -0.12
2014 (δc) -0.11 -0.14 -0.14 -0.15
(δr-δc) -0.01 0.04 0.02 0.03 N 383 383 383 383
![Page 52: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/52.jpg)
Performance within Professor A
![Page 53: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/53.jpg)
Cognitive Outcomes: 2013 vs. 2014 Professor B/Large Class¥
(standard deviations)
Joyce et al. (2014) 53 1/23/2014
¥All estimates adjusted for student characteristics
Midterm Final MT+F Overall
2013 (δr) -0.27** -0.18** -0.26*** -0.28***
2014 (δc) -0.16** -0.12 -0.14 -0.16
(δr-δc) -0.11 -0.06 -0.08 -0.12 N 949 950 950 950
![Page 54: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/54.jpg)
Performance within Professor B
![Page 55: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/55.jpg)
Implications • Lessons for RCTs
– Choice effects may be important (although not so much in this case)
– Possible to get estimates of choice effects using DRPT framework
• Lessons for observational studies – May not suffer from treatment selection bias! – Prospective observational studies doable
• Lessons for both – Prospectively measure likely heterogeneity
dimensions, like format preference!
![Page 56: Do Students Know Best? Choice, Classroom Time …...Onur Altindag Graduate Center, CUNY Stephen D. O’Connell Graduate Center, CUNY Dahlia Remler Baruch College and Graduate Center,](https://reader030.fdocuments.in/reader030/viewer/2022040721/5e2e5d824750b30ce931a378/html5/thumbnails/56.jpg)
Implications • Administrators looking to reduce costs with
fewer class meetings: be aware of performance penalty – Trade off with class size
• Treatment effects may be heterogeneous – Students may know what works best and choose
accordingly • Preference effects may matter
– Beware forcing those who don’t want hybrid/online into hybrid/online