SLIDES: Productivity Versus Motivation: Using Field Experiments...

. . . . . .

SLIDES: Productivity Versus Motivation: Using FieldExperiments and Structural Econometrics to BetterUnderstand Adolescent Human Capital Production.

(manuscript coming soon)

Brent Hickman (University of Chicago)

John List (University of Chicago)

Joseph Price (Brigham Young University)

Christopher Cotton (Queen’s University)

Hickman, List, Cotton Price () Productivity vs Motivation 1 / 84

. . . . . .

The researcher would often like to answer questions that depend onindividual characteristics that are either fundamentally unobservable or hardto quantify. EXAMPLES:

What are the factors underlying academic success?◮ Learning Productivity◮ Motivation◮ Environmental Variables

Related questions:◮ How best to intervene (at the individual level)?◮ Where do demographic academic disparities come from (at the group

level)?


. . . . . .

FIELD EXPERIMENTAL RESEARCH DESIGN: We do not aim totest a policy effect, but rather, to generate observables which will identify astructural model of decision-making based on agent heterogeneity:

θE “Efficiency/Productivity Type”◮ Indexes some aspect of ability

θL “Leisure Preference/Motivation Type”◮ Measures incentives required to induce a certain level of output,

holding θE fixed


. . . . . .

General Model OverviewPRINCIPAL: values some output O by agents

◮ Accomplishment of tasks O = Q◮ Accumulated proficiency O = S

PRODUCTION TECHNOLOGY: τ(O; θE)◮ t = τ(o; θE) time units required for output o, given θE

PRODUCTION COSTS: C(O; θE) = θLc(t)

LABOR CONTRACT: (O, P(O))

AGENT DECISION PROBLEM:

maxo∈R+

{P(o)− θLc [τ(o; θE)]}

Research Goal: To develop an experimental design to track total outputAND time inputs which, combined with exogenous contract variation,are enough to identify τ, c, and the joint distribution of (θE, θL)Hickman, List, Cotton Price () Productivity vs Motivation 4 / 84

. . . . . .

HOW TO TELL PRODUCTIVITY/MOTIVATION APART?IDEAL EXPERIMENT: Consider an avg agent named Suzie.

...1 Clone Suzie

...2 Lock the two clones in separate, identical observation rooms for amonth. Each room contains:

◮ A couch and entertainment center with an Xbox and TV,◮ A desk with a set of work tasks to accomplish.

...3 Offer different incentives to spend time on work instead of leisure◮ Offer Clone #1 $5 per unit of output (tasks accomplished)◮ Clone #2 gets $10 per unit.

...4 Monitor Suzies’◮ Time choices,◮ Rate of progress,◮ Total output.


. . . . . .

HOW TO TELL PRODUCTIVITY AND MOTIVATIONAPART?

So we can’t exactly do that, but we can come close!

...1 “Clone” a large group of agents through randomization◮ Offer each group different incentives to work

...2 Place working materials online◮ Suzie can take the observation room with her...◮ More natural, less invasive

...3 Progress/time tracking using built-in online capability


. . . . . .

STRUCTURAL IDENTIFICATION SKETCHNEEDED INGREDIENTS:(1) Joint Distribution of inputs (time) and outputs (completed tasks) AND(2) exogenous contract variation

...1 θE◮ Standard panel data idea: by observing individual cumulative

production histories, we get a WITHIN-CHILD TIME SERIES

◮ e.g., Specify Cobb-Douglas production technology

τ(q; θE) = δ1

(q

θE

)δ2

× u

and estimate θE as a child-specific fixed effect

...2 θL and c(t)◮ Conditional on θE, EXPERIMENTAL INCENTIVES VARIATION serves

as an instrument

◮ Can non-parametrically separate idiosyncratic costs (θL) from commonbaseline costs (c)


. . . . . .

OUTLINE: STRUCTURAL IDENTIFICATION OF A2-DIMENSIONAL PRINCIPAL-AGENT MODEL

...1 One-dimensional identification based on◮ D’Haultfeouille and Fevrier (DF2015)◮ Torgovitsky (T2015)

...2 Extension: 2-D identification with distributions of outputs AND inputs


. . . . . .

ONE-DIMENSIONAL MODEL

1-D PRINCIPAL-AGENT MODEL

(motivation heterogeneity only)

TYPES: θ ∈ R++, θ ∼ Fj(θ), j = A,D,K

COSTS:C(q; θ) = θc(q), c′(q) > 0, c′′(q) > 0 Q ∼ Gj(q), j = 1, 2, . . . , J

LABOR CONTRACTS: (Q, P(Q))

◮ Pj(Q) = bj + wjQ, j = 1, 2, . . . , J◮ Exogenous contract assignment◮ Linearity is for simplicity; not crucial here...


. . . . . .

1-D MODEL

A agent’s objective function is

maxq∈R+

{πj(θ) = Pj(q)− θc(q)

},

FOCs defining optimal production choice:

θj(q) =

(Pj)′(q)

c′(q)(monotone decreasing) (1)

Also, let Qj(θ) ≡ θ−1j

STRUCTURAL PRIMITIVES: Fj(θ), and c(q)OBSERVABLES: Pr

j (q), Gj(q)


. . . . . .

1-D IDENTIFICATION a la DF2015/T2015

EXCLUSION RESTRICTIONS:...1 cj(q) = c(q), ∀j = 1, . . . , J....2 Fj(θ) = F(θ), j = 1, . . . , J.

NORMALIZATION:for some (θ∗, q∗) ∈ [θ, θ]× R+ let θ1(q∗) = θ∗.

This is innocuous because

C(q; θ) = θc(q) ⇔ C(q; θ) = (ιθ) (c(q)/ι) , ∀ι ∈ R+

I.e., normalization simply fixes total cost unitsθ∗ measured relative to baseline θ∗ type.


. . . . . .


“HORIZONTAL TRANSFORM” OPERATOR:

F(θ) = 1 − G1(Q1(θ)) = 1 − G2(Q2(θ))

⇒ Q1(θ) = (G1)−1 [(G2) (Q2(θ))] ≡ H1,2 [Q2(θ)] .

(2)

⇒ Tell me (θ, q) under contract 1 and I can figure out q′ for that same θunder contract 2.

“VERTICAL TRANSFORM” OPERATOR:

θj(q) =(

Pj)′(q)/c′(q)

θ1(q) =(P1)

′ (q)(P2)

′ (q)θ2(q) ≡ V1,2 [θ2(q)]

(3)

⇒ Tell me (θ, q) under contract 1 and I can figure out θ′ for that same qunder contract 2.


. . . . . .


Q

θ(Q

)CASE 1: 2 CONTRACTS NO CROSSING

θ

1(q)

θ2(q)

(q*,θ*) NORMALIZED(q,θ) IDENTIFIED

Figure: Case 1: 2 Contracts, No Crossing


. . . . . .


Q

θ(Q

)CASE 2: 2 CONTRACTS WITH A CROSSING

θ

1(q)

θ2(q)


Figure: Case 2: 2 Contracts, W/Crossing


. . . . . .


Q

θ(Q

)CASE 3: 3 NON−PARALLEL CONTRACTS

θ

1(q)

θ2(q)

θ3(q)


Figure: Case 3: 3 Non-Parallel Contracts


. . . . . .

TWO-DIMENSIONAL MODEL EXTENSION

TYPES: θ = (θE, θL) ∈ R2++

PRODUCTION TECHNOLOGY:◮ Let τ(q; θE) = t denote qty of time input t required to achieve output

level q, given type θE

PRODUCTION COSTS: C(t; θL) = θLc(t)

◮ Costs are convex in q: d2c[τ(q;θe)]dq2 > 0

◮ Identifying assumptions: separability and convexity

STRUCTURAL PRIMITIVES: τ(q; θE), c(t), and FθE,θL(θE, θL)

CONTRACTS USED:◮ Pk(Q) = bk + wkQ, j = 1, . . . , J


. . . . . .

IDENTIFICATION OF θE : Cobb-Douglas Production

τ(q; θE) =

(q

θE

)δ2

× u (∗)

OBSERVABLES: for each agent i w/positive payout (i.e., total outputqi ≥ 2 units), and for each individual unit produced pik, k = 1, . . . , qi,

τpik ≡ total observed time to achieve pik units of output

(∗) ⇒ log(τpik) = δ2 log(pik)− δ2 log(θEi) + εpik , (∗∗)

Assume E[[1 pik θEi]

⊤εpik

]= 0 across i and k


. . . . . .

IDENTIFICATION OF θE, τ: Cobb-Douglas ProductionIDENTIFICATION OF

(δ2, {θEi}I

i=1

):

STEP 1: For each i and pik = 2, . . . , qi we difference to get

(∗∗) ⇒ log(

τpik

τpi,k−1

)= δ2 log

(pik

pi,k−1

)+ (εpik − εpi,k−1)

which identifies δ2.

STEP 2: For each i and pik = 1, . . . , qi ≥ 2, we have

log(τpik)− δ2 log(pik) = γiDi + vpik ,

where Di is a dummy for student i.

STEP 3: (∗∗) implies that

θEi = exp(−γi

δ2

)


. . . . . .

IDENTIFICATION OF θE, τ: Cobb-Douglas ProductionIDENTIFICATION OF

(δ2, {θEi}I

i=1

):

STEP 1: For each i and pik = 2, . . . , qi we difference to get

(∗∗) ⇒ log(

τpik

τpi,k−1

)= δ2 log

(pik

pi,k−1

)+ (εpik − εpi,k−1)

which identifies δ2.

STEP 2: For each i and pik = 1, . . . , qi ≥ 2, we have

log(τpik)− δ2 log(pik) = γiDi + vpik ,

where Di is a dummy for student i.

STEP 3: (∗∗) implies that

θEi = exp(−γi

δ2

)Hickman, List, Cotton Price () Productivity vs Motivation 18 / 84

. . . . . .

IDENTIFICATION OF θL, c:Agent’s Obj. Fn.: maxq∈R+

{Pj(q)− θLc [τ(q; θE)]

}, j = 1, 2, 3

FOCs: θLj(q; θE) =

(Pj)′(q)

c′ [τ(q; θE)] τ′(q; θE). (4)

EXCLUSION RESTRICTIONS, NORMALIZATION:...1 cj(t) = c(t), j = 1, 2, 3....2 τj(q; θE) = τ(q; θE), j = 1, 2, 3....3 Fj(θL|θE) = F(θL|θE), j = 1, 2, 3.

(i.e., randomization implies no selective entry, and work tasks are thesame across incentive groups)

...4 For some (θ∗L, θ∗E, q∗) ∈ [θL, θL]× [θE, θE]× R+ we fix

θL1(q∗; θ∗E) = θ∗L.


. . . . . .

IDENTIFICATION OF θL, c:

HORIZONTAL TRANSFORM: Recall that

F(θL|θE) = 1 − Gj

[Qj(θL; θE)

∣∣∣θE

]= 1 − Gj′

[Qj′(θL; θE)

∣∣∣θE

], j = j′

Therefore,

⇒ Qj(θL; θE) =(Gj

)−1

[(Gj′

)−1(

Qj′ [θL; θE]∣∣∣θE

) ∣∣∣∣∣θE

]≡ Hj,j′

[Qj′(θL; θE)

∣∣∣θE

], j = 1, 2, 3, j = j′

(5)


. . . . . .

IDENTIFICATION OF θL, c:

VERTICAL TRANSFORM: Recall that

θLj(q; θE) =

(Pj)′(q)

c′ [τ(q; θE)] τ′(q; θE), j = 1, 2, 3

Therefore,

⇒ θLj(q; θE) =

(Pj)′(q)(

Pj′)′(q)

θLj′(q; θE)

≡ Vj,j′[θLj′(q; θE)

∣∣∣θE

], j = 1, 2, 3, j = j′.

(6)


. . . . . .

IDENTIFICATION OF θL, c:IDENTIFICATION ARGUMENT (assuming no mass points in Gj):

...1 STEP 1: θE = qt directly identifies the joint distribution

FθE,Q,j(θE, q) ⇒ Gj(Q|θE)

...2 STEP 2: given (θ∗L, θ∗E, q∗), we know that Gj(q|θ∗E), HT, VT identifyθLj(q; θ∗E), j = 1, 2, 3 for q ≥ 2

...3 STEP 3: Plugging θL1(·; θ∗E) into the FOC identifies c(t)

...4 STEP 4: Knowing c(·) identifies θL2(q; θE), θL3(q; θE) for arbitrary(θE) using the FOCs

...5 STEP 5: Inverting q for a given θE, then {θE, θL} the jointdistribution FθE,θL(θE, θL) is nonparametrically identified.


. . . . . .

APPLICATION: “Motivation Vs Productivity: A Study ofHC Investment Using Field Experiments and StructuralModeling” Cotton, Hickman, List, Price (2015)

MOTIVATION: Measures of childhood human capital (e.g., SAT scores,educational attainment) are known to be linked to important outcomes:

College attendance/GPA/Graduation Probability

Employment, Productivity, Income

Even Marital/Family Stability

Early development also drives socioeconomic inequality


. . . . . .

Raw Ability vs. Work Ethic

Consider Suzie, a 7th grader who scores highly on a math exam. We askWHY?

TWO COMPETING EXPLANATIONS (both hinge on Suzie’sunobservable characteristics):

...1 LEARNING PRODUCTIVITY: relatively small amounts of Suzie’sstudy time translate into big gains in her math understanding...

...2 MOTIVATION: Suzie loves learning math or doesn’t care as muchabout leisure activities, so she chooses to spend more time studyingthan others...


. . . . . .

Raw Ability vs. Work Ethic (cont’d)POLICY IMPLICATIONS:Knowing more about Suzie’s characteristics and time choices can inform uson how best to intervene if she struggles later on.

...1 If Suzie is highly motivated but slow to learn,◮ Resource problem? (e.g., tutoring)◮ Study Skills deficiency? (e.g., help her to use time more efficiently)

...2 If Suzie is a fast learner but lacking motivation,◮ Inspiration problem? (e.g., feeling unengaged by her coursework)◮ Incentives problem?

...3 At the group level, we could decompose gender/race gaps intodifferences in unobservables...

◮ Mainly a productivity difference?◮ Mainly a motivational difference?◮ Some mixture?


. . . . . .

Research Design:Experimental variation in short-term time investment incentives.Observations of time investment into learning in a controlled, butnatural field setting

◮ Online Mathematics portal w/ 5th-8th grade students & Common Corecurriculum

◮ 3 randomly assigned wage contracts for completing math learning taskson website

◮ Automated Monitoring of time use decisions

◮ Progress in math proficiency measured in classroom, working withstudents’ regular teachers


. . . . . .

CHICAGO HUMAN CAPITAL STUDY

Mathematics study held at public schools in Chicago suburbs◮ VERY heterogeneous population, racially, culturally, socioeconomically

Adjacent grades combined for the study (5/6, 7/8)◮ 5/6=GROUP 2◮ 7/8=GROUP 3


. . . . . .

WWW.CHICAGOMATHGAME.ORG

MC questions taken from state standardized testing materials over thepast decade

◮ CA, FL, IL, MN, NY, New England, TX, WI, OH, NAEP (National)◮ All questions similar to curriculum in use at subjects’ schools◮ All questions separated into 5 subject categories, based on Common

Core◮ Questions arranged into “quizzes”

⋆ 6 questions each⋆ 5 subject-specific quizzes per category⋆ 55 general quizzes⋆ Passing criterion: at least 5/6 correct⋆ Practice run for missed problems⋆ Random ordering each attempt

Supplementary Instructional Materials (glossary, web videos, practiceproblems w/ solutions)


. . . . . .

Experimental Design (grades 5-8):Pre-test from same pool of questions

Direct wage contracts for completing quizzes: (Q, P(Q)), where

Pj(Q) =[bj + mjQ

]· 1{Q ≥ 2}, j = 1, 2, 3

◮ CONTRACT 1: (b1, m1) = ($15, $0.75)◮ CONTRACT 2: (b2, m2) = ($10, $1)◮ CONTRACT 3: (b3, m3) = ($5, $1.25)

Individual randomassignment w/in each school/gradeStudents informed of rules on an information sheet

◮ Their own score from practice test◮ Quiz passing criterion◮ Wage contract/total earnings potential◮ 10-day work period + login for website

Post-test (with 1/4 overlap from pre)


. . . . . .



Pj(Q) =[bj + mjQ

]· 1{Q ≥ 2}, j = 1, 2, 3


Individual randomassignment w/in each school/grade

Students informed of rules on an information sheet◮ Their own score from practice test◮ Quiz passing criterion◮ Wage contract/total earnings potential◮ 10-day work period + login for website



. . . . . .



Pj(Q) =[bj + mjQ

]· 1{Q ≥ 2}, j = 1, 2, 3


Individual randomassignment w/in each school/gradeStudents informed of rules on an information sheet

◮ Their own score from practice test◮ Quiz passing criterion◮ Wage contract/total earnings potential◮ 10-day work period + login for website



. . . . . .

Experimental Design:

OTHER OBSERVABLES:Student Pre/Post Survey

◮ Extracurricular/leisure activites, gaming consoles◮ Math preferences◮ Intrinsic vs Extrinsic Motivation

Student note taking on Test booklet

Parent Post Survey◮ Family Structure, housing, income◮ Education background for both parents, STEM ability◮ Parenting Styles, time use◮ Prize Preferences

Home values (County tax records)


. . . . . .

Preliminary Descriptive Results

WEBSITE USAGE:◮ 1565 students (48%) used website at least once◮ For site users, avg time spent was 2 hours◮ Over 16 hours in come cases

QUIZ OUTPUT:◮ Avg # of quizzes passed was 21.5 out of 80 possible◮ Avg # of quiz attempts was 39


. . . . . .


0 10 20 30 40 50 60 70 80

0.4

0.5

0.6

0.7

0.8

0.9

1

TOTAL PASSED QUIZZES

EM

PIR

ICA

L C

DF

CONTRACT 1, 5th−6th GRADESCONTRACT 2, 5th−6th GRADESCONTRACT 3, 5th−6th GRADES


. . . . . .


“Struggling Workhorse” (SW):◮ Bottom half on pre-test, but top half (w/in incentive group) for

minutes spent on website◮ 434 students out of 2,380 (18.2% overall)

“Struggling SuperWorkhorse” (SSW):◮ Bottom half/top quarter OR bottom quarter/top half◮ 287 students out of 2,380 (12.1% overall)

“Low Motivation” (LM):◮ Top half/bottom half◮ 460 students out of 2,380 (19.3% overall)


. . . . . .

ESTIMATION

STAGE 1 (PRODUCTION TECH): Estimate δ2, {θEi}Ii=1 using

standard methods for (unbalanced) panel data.

STAGE 2: Now, how to estimate c(t), {θLi}Ii=1, and FθE,θL(θE, θL)?

METHOD 1: Nonparametric Kernel-based estimator◮ Partial Identification Problem: 50% of students choose zero output◮ Can only bound their 2-D type within a region of (θE, θL) space◮ Also cannot include covariates in model since high-dimensional

nonparametrics are infeasible

METHOD 2: Spline-based semi-parametric estimator◮ Flexible B-spline specification for c(t)◮ Joint normality assumption to extrapolate non-identified region of

(θE, θL) space◮ Also allows for inclusion of covariates...


. . . . . .

ESTIMATION METHOD 1: Kernel Smoothing

Given STAGE 1, we have (qij, θEij) for each student i in contract j...1 Within each contract sample, estimate the conditional CDF

GjQ|θE

(q|θE) =I

∑i=1

1(qij ≤ q)κ(

θE − θEij

hj

),

where κ is a density function and hj is a bandwidth.

...2 Use the HT and VT operators to estimate θL(q; θ∗E)

...3 Plug θL(q; θ∗E) into FOC to get c(t), which applies to all θE

...4 Use FOCs and c(t) to recover θLij(qij; θEij) for each student in thedata with 2 ≤ qij < 80


. . . . . .

ESTIMATION METHOD 1: Kernel SmoothingPROS: Simple, method follows directly from identification argumentCONS: Statistically inefficient for two reasons:

...1 Kernel smoothing in multiple dimensions very data hungry

...2 Method does not directly impose theory on CDF estimates

0 10 20 30 40 50 60 70 800

0.2

0.4

0.6

0.8

1

Q

CO

ND

ITIO

NA

L C

DF

GQ

|θE

(q|θ

E*)

CONTRACT 1 UQCONTRACT 2 UQCONTRACT 3 UQ

0 10 20 30 40 50 60 70 800

2

4

6

8

θ L2(q

;θE*)

Q


. . . . . .

ESTIMATION METHOD 2: Semiparametric GMMFirst, parameterize

c(t; αc) =M

∑m=1

αcmCm(t)

where the Cm(·)’s are a set of B-spline basis functions.Knots in t space chosen as uniform quantiles

Next, assume joint log-normality of types

(log θE, log θL) = BVN ([µE, µL], Σ),

where Σ = [µe, µl , σ2e , σ2

l , σel ]

Parametric form characterizes selection equations:Pr[zero output] = Pr

[bj + 2mj < θL c (τ(2/θE); αc) ; αel

]Pr[ f ull output] = Pr

[mj ≥ θL c′

(τ(80/θEij); αc

)τ′(80/θEij); αel

]


. . . . . .

ESTIMATION METHOD 2: Semiparametric GMMFirst, parameterize

c(t; αc) =M

∑m=1

αcmCm(t)

where the Cm(·)’s are a set of B-spline basis functions.Knots in t space chosen as uniform quantiles

Next, assume joint log-normality of types

(log θE, log θL) = BVN ([µE, µL], Σ),

where Σ = [µe, µl , σ2e , σ2

l , σel ]

Parametric form characterizes selection equations:Pr[zero output] = Pr

[bj + 2mj < θL c (τ(2/θE); αc) ; αel

]Pr[ f ull output] = Pr

[mj ≥ θL c′

(τ(80/θEij); αc

)τ′(80/θEij); αel

]Hickman, List, Cotton Price () Productivity vs Motivation 37 / 84

. . . . . .

ESTIMATION METHOD 2: Semiparametric GMM

QUESTION: Why not just estimate via Tobit Maximum Likelihood?

ANSWER: The censoring point (extensive margin for work vs not) is afunction of the fixed effects θEi and θLi, which are also parameters to beestimated. This violates regularity conditions needed for MLE to benumerically well-behaved and consistent...


. . . . . .


Additional Conditions to impose:Orthogonality conditions from production technology

Normalization: θ∗L =P′

1(q∗)

c′[τ(q∗;θ∗E);αc]τ′(q∗;θ∗E)= 1, G2(q∗) = 0.75

FOCs: θLij =(Pj)

′(qij)

c′[τ(qij;θEij);αc]τ′(qij;θEij), ∀i, j,

Shape Restrictions from linear constraints (B-splines=magic!):Boundary Condition: c(0; αc) = 0 ⇒ αc1 = 0Monotonicity: c′(t; αc) > 0 ⇒ αck < αc,k+1, ∀kConvexity:

c′′[τ(q; θE); αc

] (τ′(q; θE)

)2+ c′

[τ(q; θE); αc

]τ′′(q; θE) > 0


. . . . . .


Additional Model Components:Decompose (θE, θL) as functions of student covariates(socioeconomics, leisure opportunities, home environment, subjectpreferences, etc...)

◮ Specify types as a single index of covariates X:

θE = xβE1e1 · xβE2

e2 · · · xβEkek · ηE

θL = xβL1l1 · xβL2

l2 · · · xβLklk · ηL

where ηE and ηL are the components of a student’s type that areorthogonal to observables.

◮ Incorporate information from:⋆ Organizational skills/habits using test booklet data⋆ Survey data (math attitude, motivation, home environment)⋆ Census block group data on poverty/stability measures


. . . . . .

ESTIMATION METHOD 2: Semiparametric GMMAdditional Model Components:

Relate estimates to math pre-test score S and score change ∆S usingpost-test scoresCobb-Douglas production:

Si = AsθαsEE θαsL

L Usi

∆Si = A∆θα∆EE θα∆L

L U∆i

where we assume E[[1, θEi, θLi]

⊤ log Uji]= 0 for j = s, ∆.

This provides a link between our learning by doing production modeland traditional outcome measures of human capital.

IMPORTANT NOTE: involving covariates or test scores in the empiricalmodel requires the joint normality assumption for feasibility and to correct

for sample selection (i.e., mass points at zero output).


. . . . . .

Stage 1 Results:Production Technology estimate: τ(q; θe) =

(q

θE

)δ2

δ2 = 0.9969 (not statistically different from 1)

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

5

10

15

20

25

30

35

40

45

θE for Group 2 Investors

His

tog

ram

Val

ues


. . . . . .

Stage 2 Kernel-Based Results:

0 100 200 300 400 500 6000

20

40

60

80

100

120

140

X: 180Y: 14.57

TIME SPENT t (IN MINUTES)

CO

MP

EN

SA

TIN

G D

IFF

ER

EN

TIA

L

c(t)

(in

$ u

nit

s)

BASELINE UTILITY COST FUNCTION ESTIMATERELATIVE TO θ

L=1 (median identified type)

X: 360Y: 47.03

$7.84/hr = 6 hrs labor supplied

$4.86/hr = 3 hrs labor supplied


. . . . . .

Stage 2 Kernel-Based Results:

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

2

4

6

8

10

12

θE

θ L

Scatterplot of Type Estimates(Investors Only)

LOCUS OF NON−IDENTIFICATION

(due to mass point at q=0)


. . . . . .

Race Comparison (kernel-based results)

−5 0 5 10 15 20 25 30 35 400

0.2

0.4

0.6

0.8

1Race Comparisons (for only identified types)

PRE−TEST SCORE (out of 36)

EM

PIR

ICA

L C

DF

BLACK/HISPANICWHITE/ASIAN

−5 −4 −3 −2 −1 00

0.2

0.4

0.6

0.8

1

log( θE)

EM

PIR

ICA

L C

DF

BLK/HSPWHT/ASN

−4 −2 0 2 40

0.2

0.4

0.6

0.8

1

log( θL)

EM

PIR

ICA

L C

DF

BLK/HSPWHT/ASN

I.e., BLK/HSP students take longer to accomplish learning tasks but aremore willing to substitute from leisure time toward work.


. . . . . .

Gender Comparison (kernel-based results)

−5 0 5 10 15 20 25 30 35 400

0.2

0.4

0.6

0.8

1Gender Comparisons (for only identified types)

PRE−TEST SCORE (out of 36)

EM

PIR

ICA

L C

DF

FEMALEMALE

−5 −4 −3 −2 −1 00

0.2

0.4

0.6

0.8

1

log( θE)

EM

PIR

ICA

L C

DF

FEMALEMALE

−4 −2 0 2 40

0.2

0.4

0.6

0.8

1

log( θL)

EM

PIR

ICA

L C

DF

FEMALEMALE

I.e., FEMALE students take longer to accomplish learning tasks but aremore willing to substitute from leisure time toward work.


. . . . . .

Empirical Output Distributions by Race

0 10 20 30 40 50 60 70 800

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

QUIZZES PASSED

EM

PIR

ICA

L C

DF

s

FULL OUTPUT DISTRIBUTION

OBSERVED MINORITY OUTPUTOBSERVED NON−MINORITY OUTPUT


. . . . . .

Output Distributions by Gender

0 10 20 30 40 50 60 70 800

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

QUIZZES PASSED

EM

PIR

ICA

L C

DF

s

FULL OUTPUT DISTRIBUTION

OBSERVED FEMALE OUTPUTOBSERVED MALE OUTPUT


. . . . . .

SEMIPARAMETRIC STRUCTURAL ESTIMATES

NEARING COMPLETION AND COMING SOON...

...1 Production Counterfactuals◮ Equalize marginal distributions of different characteristics/covariates

across demographic groups, one at a time (decomposition)

◮ What level of incentives needed to close race/gender gaps in output?


. . . . . .

AS A PREVIEW, WE INCLUDE DESCRIPTIVE ANALYSES OFCOVARIATES BY RACE AND GENDER HERE:


. . . . . .

COVARIATES by Gender

0 0.5 1 1.5 2 2.5 3

x 105

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1MEDIAN NEIGHBORHOOD INCOME BY GENDER

US DOLLARS

t−TEST P−Value=0.94717FEMALEMALE


. . . . . .


0 10 20 30 40 50 60 70 800

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1PROBLEM SOLVING STRATEGIES BY GENDER

PRE−TEST: TOTAL MARKINGS (0−144)

t−TEST P−Value=2.21e−06FEMALEMALE


. . . . . .


0 10 20 30 40 50 60 700

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

PRE−TEST:ELIMINATION/CIRCLING/STARRING (0−108)


0 5 10 15 20 250

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

PRE−TEST:PROBLEM NOTES (0−36)

EM

PIR

ICA

L C

DF

s



. . . . . .


0 1 2 3 4 5 6 7 80

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1HOMEWORK TIME BY GENDER

SELF−REPORTED HOURS:AVG WEEKDAY + AVG WEEKEND DAY



. . . . . .


0 1 2 3 4 5 60

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1STRUCTURED ACTIVITY TIME BY GENDER




. . . . . .


0 0.5 1 1.5 2 2.5 3 3.5 40

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1EXTRA−CURRICULAR ACTIVITIES BY GENDER

SELF−REPORTED COUNT (0−4)



. . . . . .


0 1 2 3 4 5 60

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1UNSTRUCTURED SOCIAL TIME BY GENDER




. . . . . .


0 1 2 3 4 5 60

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1SCREEN TIME BY GENDER




. . . . . .


−1 −0.5 0 0.5 1 1.5 20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1STEM PREFERENCE BY GENDER

COMPOSITE SCORE (0−2)



. . . . . .


−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1MATH PREFERENCE SHIFT BY GENDER

SELF−REPORTED SHIFT



. . . . . .


−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1MATH UNDERSTANDING SHIFT BY GENDER




. . . . . .


0 1 2 3 4 5 6 7 80

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1EX−INT MOTIVATIONAL SCORE BY GENDER

COMPOSITE SCORE



. . . . . .


0 0.5 1 1.5 2 2.5 30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

EXTRINSICMOTIVATION SCORE


0 1 2 3 4 5 60

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

INTRINSICMOTIVATION SCORE

EM

PIR

ICA

L C

DF

s



. . . . . .


0 1 2 3 4 5 6 70

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1TOTAL HELPER COUNT BY GENDER

TOTAL COUNT (self−reported)

EM

PIR

ICA

L C

DF

s



. . . . . .


0 0.5 1 1.5 2 2.5 3 3.5 40

0.5

1

TOTAL ADULT HELPER COUNT (self−reported)

CD

Fs


0 0.5 1 1.5 2 2.5 3 3.5 40

0.5

1

TOTAL PEER HELPER COUNT (self−reported)

CD

Fs


0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.5

1

TOTAL ADULT HELPER FRACTION

CD

Fs



. . . . . .

COVARIATES by Race

0 0.5 1 1.5 2 2.5 3

x 105

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1MEDIAN NEIGHBORHOOD INCOME BY RACE

US DOLLARS

t−TEST P−Value=3.1566e−247BLACK/HISPANICWHITE/ASIAN


. . . . . .

COVARIATES by Race

0 10 20 30 40 50 60 70 800

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1PROBLEM SOLVING STRATEGIES BY RACE

PRE−TEST: TOTAL MARKINGS (0−144)

t−TEST P−Value=0.0087581BLACK/HISPANICWHITE/ASIAN


. . . . . .

COVARIATES by Race

0 10 20 30 40 50 60 700

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

PRE−TEST:ELIMINATION/CIRCLING/STARRING (0−108)


0 5 10 15 20 250

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

PRE−TEST:PROBLEM NOTES (0−36)

EM

PIR

ICA

L C

DF

s



. . . . . .

COVARIATES by Race

0 1 2 3 4 5 6 7 80

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1HOMEWORK TIME BY RACE




. . . . . .

COVARIATES by Race

0 1 2 3 4 5 60

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1STRUCTURED ACTIVITY TIME BY RACE




. . . . . .

COVARIATES by Race

0 0.5 1 1.5 2 2.5 3 3.5 40

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1EXTRA−CURRICULAR ACTIVITIES BY RACE

SELF−REPORTED COUNT (0−4)



. . . . . .

COVARIATES by Race

0 1 2 3 4 5 60

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1UNSTRUCTURED SOCIAL TIME BY RACE




. . . . . .

COVARIATES by Race

0 1 2 3 4 5 60

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1SCREEN TIME BY RACE




. . . . . .

COVARIATES by Race

−1 −0.5 0 0.5 1 1.5 20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1STEM PREFERENCE BY RACE

COMPOSITE SCORE (0−2)



. . . . . .

COVARIATES by Race

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1MATH PREFERENCE SHIFT BY RACE




. . . . . .

COVARIATES by Race

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1MATH UNDERSTANDING SHIFT BY RACE




. . . . . .

COVARIATES by Race

0 1 2 3 4 5 6 7 80

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1EX−INT MOTIVATIONAL SCORE BY RACE

COMPOSITE SCORE



. . . . . .

COVARIATES by Race

0 0.5 1 1.5 2 2.5 30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

EXTRINSICMOTIVATION SCORE


0 1 2 3 4 5 60

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

INTRINSICMOTIVATION SCORE

EM

PIR

ICA

L C

DF

s



. . . . . .

COVARIATES by Race

0 1 2 3 4 5 6 70

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1TOTAL HELPER COUNT BY RACE

TOTAL COUNT (self−reported)

EM

PIR

ICA

L C

DF

s



. . . . . .

COVARIATES by Race

0 0.5 1 1.5 2 2.5 3 3.5 40

0.5

1

TOTAL ADULT HELPER COUNT (self−reported)

CD

Fs


0 0.5 1 1.5 2 2.5 3 3.5 40

0.5

1

TOTAL PEER HELPER COUNT (self−reported)

CD

Fs


0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.5

1

TOTAL ADULT HELPER FRACTION

CD

Fs



. . . . . .

THE END


. . . . . .

Time Accounting Issues: Data Truncation

PROBLEM: sometimes students begin a web session and then walk awayin the middle, so how to tell spurious time observations apart and what to

do about it?

DIAGNOSING: Look for “holes" in the support of the distribution of webpage times, t.

...1 Bin time data by subject category, activity type

...2 Find first place where a kernel density estimator would equal zero◮ That is, for bandwidth h = cσN−1/5,◮ first order time observations from least to greatest: t1 < t2 < · · · < tN

and◮ find t∗ ≡ min {tn | tn+1 − tn > 2h}.

...3 If t∗ exists, truncate all observations t > t∗

...4 Repeat from step 2 until t∗ = ∅


. . . . . .

Time Accounting Issues: Data Truncation

PROBLEM: sometimes students begin a web session and then walk awayin the middle, so how to tell spurious time observations apart and what to

do about it?

DIAGNOSING: Look for “holes" in the support of the distribution of webpage times, t.

...1 Bin time data by subject category, activity type

...2 Find first place where a kernel density estimator would equal zero◮ That is, for bandwidth h = cσN−1/5,◮ first order time observations from least to greatest: t1 < t2 < · · · < tN

and◮ find t∗ ≡ min {tn | tn+1 − tn > 2h}.

...3 If t∗ exists, truncate all observations t > t∗

...4 Repeat from step 2 until t∗ = ∅Hickman, List, Cotton Price () Productivity vs Motivation 82 / 84

. . . . . .

Time Accounting Issues: Data TruncationEXAMPLE: 8th Grade Probability, Non-Practice Page Load (question)Times

Truncation point: 13.1 minutesPercent of truncated observations: 0.63%

0 20 40 60 80 100 120 140 160 180 2000

200

400

600

800

1000

1200

1400

1600

8th Grade Probability Question Page Load TimesWith Truncation Point

His

togr

am v

alue

s

RESOLUTION: replace truncated observations by within-bin,within-student censored meanHickman, List, Cotton Price () Productivity vs Motivation 83 / 84

. . . . . .

Time Accounting Issues: Output Time (τ) AggregationDEFINITION: “Paid time” is time spent attempting a quiz that waseventually passed, prior to first success on that quiz.DEFINITION: Students spend some time on quiz attempts which neverlead to a paid success, and occasionally re-attempt a quiz after passing itthe first time. Call this “unpaid time” .

METHOD 1: Drop all unpaid time

METHOD 2: Evenly spread unpaid time over all observed successes.

METHOD 3: Apply unpaid time serially to success times in progress.

◮ Tommy passes Q1 in 15 min. Then he spends 2 min re-attempting Q1.Then he spends 1 min attempting Q5 without success. Then he spends3 min attempting Q2 leading to a pass. Finally, he spends 1 moreminute on Q5 but without success.

◮ Total output: 2 units◮ τ1 = 15min◮ τ2 = τ1 + 2 + 1 + 3 = τ1 + 6min


SLIDES: Productivity Versus Motivation: Using Field Experiments...

Documents

Transcript of SLIDES: Productivity Versus Motivation: Using Field Experiments...