II. DESIGN AND METHODS IN THEEARLY HEAD START STUDY

Richard A.Faldowski,Rachel Chazan-Cohen, JohnM.Love, andCheri Vogel

The EHSREP followed 3,001 low-income families who applied to receiveEarly Head Start (EHS) services between July 1996 and September 1998 at 17program sites around the country. Upon application, families were randomlyassigned to receive either EHS plus local community services or usualcommunity services alone (i.e., all services, except EHS, available in localcommunities) through the child’s third birthday (Administration forChildren and Families [ACF], 2002a; Love et al., 2005).

In March 1996, ACF, the federal agency responsible for the evaluation,selected 15 programs that were able to recruit twice as many families as theycould serve and had a strong local research partner. ACF also ensured that, inaggregate, the sites would be nationally distributed and would include themajor programmatic approaches, settings, and family demographics thatwere characteristic of the 68 programs then funded. When these criteriaresulted in too-few center-based programs, ACF added two more sites.Although the 17 programs cannot be considered a nationally representativesample of all EHS programs, analyses of key program features demonstratedthat the program and family characteristics of the 17 reflected EHS programsoperating at that time (ACF, 2002a; Love et al., 2005).

The EHS study was designed to document the consequences of receivingeither EHS services or other community services up until age 3 combined withsubsequent Head Start or other formal early care and education programs onchildren’s school readiness and parent functioning in the preschool years(ACF, 2007a). This chapter focuses on key design issues andmethods relevantfor understanding the substantive analyses reported in this monograph.

Corresponding author: John M. Love, 1016 Canyon Park Drive, Ashland, OR 97520,email: jlove@mind.net

20

EHS DESIGN (UP TO AGE 3)

During the sample intake period, which extended over 26 months, 3,001families were randomly assigned to the program and control groups, roughly inequal numbers. About a quarter of the research sample enrolled duringpregnancy, but all families had to have a child under the age of 1 year. Formoreinformation on the randomization process, please refer to the project’stechnical impact reports (ACF, 2002a; Administration on Children, Youth, andFamilies [ACYF], 2001). At the time of enrollment, parents completed HeadStart Family Information System (HSFIS) ProgramApplication and Enrollment Forms forthe programs.Demographic informationwas extracted from these forms for usein the research. We used this information to create subgroups defined by familycharacteristics at baseline, and to adjust for differences in the characteristics ofprogramandcontrol groupmembers whenestimating program impacts (i.e., togain precision in our estimates). We also used data from the forms to comparethe characteristics of interview respondents with nonrespondents, and toconstruct weights to adjust for potential nonresponse bias.

Data collection included both time in program assessments as well aschild age-based assessments. The time-in-program assessments were all parentinterviews and included the following:

Parent Services Follow-Up Interviews (PSI) Targeted for Collection 6, 15, and 26Months After Random Assignment. These interviews collected information on(1) the use of services provided both by EHS and other providers, such ashome visits, case management, parenting education, health care, employ-ment assistance, and child care; (2) progress toward economic self-sufficiency,such as employment status, welfare receipt, and participation in educationand training programs); (3) family health; and (4) children’s health. MostPSIs were conducted by telephone with the focus child’s primary caregiver,although some interviews were conducted in person for those not reachableby phone.

Exit Interviews When Children Reached 3 Years of Age. These interviews wereconducted only with EHS parents (not the control group) when children were3 years old and had to transition out of Early Head Start. The exit interviewsobtained retrospective information on the use of EHS services. Wheneverpossible, the interviews were conducted in conjunction with the age 3 parentinterviews (see below), but in some cases were conducted in conjunction withthe 26-month PSIs.

The child age-based data collections included:Parent Interviews (PI) Targeted for Collection When Children Were 14, 24, and

36 Months Old. These interviews obtained a large amount of informationfrom the primary caregivers about their child’s development and familyfunctioning. These data usually were collected in person, but some PIs orportions of them were conducted by telephone when necessary.

21

DESIGN AND METHODS

Child and Family Assessments Targeted for CollectionWhen ChildrenWere 14, 24,and 36 Months Old. Field interviewers provided data on their observations ofchildren’s behavior and home environments. Interviewers conducted directchild assessments (such as Bayley assessments) and videotaped semistruc-tured parent–child interactions. Quality of center-based care was alsoobserved at this time.

This monograph reports results for assessments at ages 2 and 3 (some ofthese data have been reported elsewhere—CF, 2002; ACYF, 2001; Love et al.,2005). We did not analyze impacts at 14 months because some families hadbeen in the program only a short time.

DESIGN OF THE PREKINDERGARTEN FOLLOW‐UP (AGE 5)

The prekindergarten follow-up study was designed to follow former EHSand control group children and families from the time children turned 3years old until the summer preceding their scheduled kindergarten entry.Two types of data collection were conducted. First, families participated ina telephone tracking interview. These tracking interviews helped researchersmaintain contact with sample members and collected information onchildren’s enrollment in formal early care and education programs, alongwith parents’ education and employment status. Second, during the spring orsummer preceding each child’s scheduled kindergarten entry, familiesparticipated in a parent interview, direct child assessments, videotapedparent–child interactions, and early care and education provider observationsand interviews. Twelve percent of the parent interviews were conducted inSpanish.

DESCRIPTIONS OF MEASURES

Measures were chosen to capture multiple aspects of child and familyfunctioning. The selection of measures for the prekindergarten phase of thestudy was guided by two major goals. First, there was the need to maintaincontinuity with the birth to 3 phase of the project. Second, when new orsubstitute measures were selected, measures that had been employed inHeadStart FACES (ACF, 2003, 2006, 2007a, b) or other national surveys (e.g., EarlyChildhood Longitudinal Study, Birth Cohort [ECLS-B]; National Center forEducation Statistics, n.d.; National Institute of Child Health and HumanDevelopment Study of Early Child Care [NICHD SECC], n.d) were givenpriority (Brooks-Gunn, Fuligni, & Berlin, 2003). Table 1 documents themeasures within each of the major outcome domains that are included inthe analyses reported in this monograph. We do not report on all measures

22

TABLE1

COMPREHENSIVELIST

OFO

UTCOMESANALYZ

ED

INSU

BSE

QUENTCHAPTERS

Domain

Outcome

Source

Possible

Ran

geMean(SD)

Interpretation

ChildNeg

ativeSo

cial-Emotional

&Positive

Approaches

toLearningOutcomes

CBCLAgg

ressiveBeh

avior(age

s2,

3,5)

Paren

tRep

ort

0–38

Age

2:12

.6(6.76)

Higher

scores:More

aggressive

beh

avior

Age

3:11

.1(6.47)

Age

5:10

.9(6.73)

FACESSo

cial

Beh

aviorProblems(age

5)Paren

tRep

ort

0–24

Age

5:5.6(3.58)

Higher

scores:More

beh

aviorproblems

ChildNeg

ativityTowardParen

tDuringPlay

(age

s2,

3,5)

Observer

Rating/

Paren

t–Child

PlayVideo

1–7

Age

2:1.7(0.98)

Age

3:1.3(0.57)

Age

5:1.2(0.56)

Higher

scores:Higher

ratesat

whichthech

ildshows

ange

r,hostility,

ordislike

towardtheparen

tduring

play

ChildEngage

men

tDuringPlay(age

s2,

3,5)

Observer

Rating/

Paren

t–Child

PlayVideo

1–7

Age

2:4.3(1.14)

Age

3:4.7(1.01)

Age

5:4.7(0.95)

Higher

scores:Higher

ratesat

whichthech

ildshows,

initiates,an

d/ormaintainsinteractionwiththe

paren

t;an

dco

mmunicates

positive

regard

and/or

positive

affect

totheparen

tduringplay

Sustained

Atten

tionWithObjectsDuringPlay

(age

s2,

3)

Observer

Rating/

Paren

t–Child

PlayVideo

1–7

Age

2:5.0(0.94)

Age

3:4.9(0.95)

Higher

scores:More

sustained

attentionwithobjects

duringplay

FACESPositive

Approaches

toLearning(age

5)Paren

tRep

ort

0–14

Age

5:12

.0(1.88)

Higher

scores:More

positive

approachto

learning

Observed

BayleyEmotionReg

ulation(age

s2,

3)Observer

Rating

1–5

Age

2:3.6(0.80)

Age

3:3.9(0.76)

Higher

ratings:betterem

otionregu

lationan

dab

ilityto

shifttasks,less

frustrationan

dneg

ativity

Observed

LeiterEmotionReg

ulation

(Leiter-RExaminer

RatingScales)(age

5)

Interviewer

Rating/

ChildAssessm

ent

Raw

Scores:0–

66

Scaled

Scores:46

–113

Age

5:91

.1(9.80)

Higher

scores:Greater

levelsofen

ergy,positive

emotion,

andlack

ofan

xiety;

aswellas

appropriatelevelsof

self-reg

ulationan

dindistractibility

Observed

Atten

tion(age

5)(L

eiter-R

Examiner

RatingScales)

Interviewer

Rating/

ChildAssessm

ent

Raw

Scores:0–

30

Scaled

Scores:1–

10

Age

5:8.6(1.97)

Higher

scores:Greater

levelsofattention,co

ncentration,

focu

s,an

dindistractibilitywhileperform

ing

challengingtasks

ChildLan

guage/

Cogn

itive/

Academ

icSk

ills

MacArthurCDIVocabulary

(age

2)Paren

treport

0–10

0Age

2:54

.8(22.95

)Higher

scores:Large

rvocabulary

EnglishRecep

tive

Vocabulary

(Peabody

Picture

Vocabulary

TestIII)

(age

s3,

5)

ChildAssessm

ent

Raw

Scores:0–

204

Scaled

Scores:40

–160

Age

3

SS:83

.0(15.56

)

Higher

scores:Greater

Englishreceptive

vocabulary

Age

5

SS:91

.5(15.16

)

(Continued)

23

DESIGN AND METHODS

TABLE1.

(Continued)

Domain

Outcome

Source

Possible

Ran

geMean(SD)

Interpretation

Span

ishRecep

tive

Vocabulary

(TestdeVocabulario

enIm

agen

esPeabody)

(age

s3,

5)

ChildAssessm

ent

Raw

Scores:0–

125

Scaled

Scores:40

–160

Age

3

SS:95

.1(8.16)

Higher

scores:Greater

Span

ishreceptive

vocabulary

SS:84

.4(23.1)

BayleyMDI(age

s2,

3)ChildAssessm

ent

49–1

34Age

2:89

.1(13.68

)Higher

scores:higher

developmen

talfunctioning

Age

3:90

.6(12.63

)

BayleyMDI<

85(age

s2,

3)ChildAssessm

ent

0–1

Age

2:0.37

(0.48)

1¼

Childhas

anMDI<

85

Age

3:0.30

(0.46)

Woodco

ck-Johnson-Revised

Letter-Word

Iden

tification

(age

5)

ChildAssessm

ent

Raw

Scores:0–

57

Scaled

Scores:40

–160

Age

5

SS:89

.3(13.66

)

Higher

scores:Greater

Englishletter/word

knowledge

preliteracy

skills

Span

ishWoodco

ck-M

unoz-Revisad

aIden

tifcacion

deLetrasyPalab

ras(age

5)

ChildAssessm

ent

Raw

Scores:0–

58

Scaled

Scores:40

–160

Age

5:86

.9(16.9)

Higher

scores:Greater

Span

ishletter/word

knowledge

preliteracy

skills

Woodco

ck-Johnson-Revised

Applied

Problems

(age

5)

ChildAssessm

ent

Raw

Scores:1–

60

Scaled

Scores:40

–160

Age

5

SS:88

.3(20.08

)

Higher

scores:Greater

representation,co

unting,

and

simple

addition/subtractionem

ergingnumeracyskills

Woodco

ck-M

unoz-Revisad

aProblemas

Aplicados

(age

5)

ChildAssessm

ent

Raw

Scores:0–

59

Scaled

Scores:40

–160

Age

5:88

.7(17.15

)Higher

scores:Greater

representation,co

unting,

and

simple

addition/subtractionem

ergingnumeracyskills

Leiter-RAtten

tionSu

stained

ChildAssessm

ent

Raw

Scores:74

–74

Scaled

Scores:1–

19

Age

5

SS:11

.0(3.18)

Higher

scoresindicategreaternumber

ofco

rrectan

swers

withfewerrors,sugg

estinggreatervigilance

and

focu

sedattentionduringarepetitivetask:n

egativeraw

scoresoccurwhen

more

errors

than

correctan

swers

aregiven

Leiter-RAtten

tionSu

stained

TotalCorrect

ChildAssessm

ent

Raw

Scores:0–

74Age

5Higher

scores:Greater

number

ofco

rrectan

swers

Scaled

Scores:1–

19SS

:10

.9(3.11)

Leiter-RAtten

tionSu

stained

TotalErrors

ChildAssessm

ent

Raw

Scores:0–

74

Scaled

Scores:1–

19

Age

5

SS:10

.7(2.75)

Higher

errorscaled

scoresco

rrespondto

lower

rawerror

scores;therefore,higher

errorscaled

scoresindicate

fewer

number

ofinco

rrectan

swers

ChildHealth

ERVisitDueto

Acciden

torInjury

(age

s2,

3)Paren

tRep

ort

0–1

Age

2:0.08

(0.27)

1¼

ERvisitdueto

acciden

torinjury

Age

3:0.11

(0.31)

AnyIm

munizations(age

s2,

3)Paren

tRep

ort

0–1

Age

2:0.97

(0.16)

1¼

Immunized

Age

3:0.98

(0.13)

(Continued)

24

TABLE1.

(Continued)

Domain

Outcome

Source

Possible

Ran

geMean(SD)

Interpretation

ChildHas

Individualized

EducationPlan

(age

s2,

3,5)

Paren

tRep

ort

0–1

Age

2:0.04

(0.19)

Age

3:0.07

(0.25)

1¼

Childhas

individualized

educationplan

Age

5:0.8(0.27)

Spee

chProblems(age

5)Paren

tRep

ort

1–3

Age

5:0.22

(0.41)

Lower

scores:Childhas

fewer/less

seriousspee

ch

problems

Paren

tingan

dtheHomeEnvironmen

t

HOMELan

guage&

Literacy(age

s2,

3,5)

Paren

tRep

ort

þInterviewer

Observation

0–12

Age

2:10

.2(1.71)

Age

3:10

.5(2.02)

Age

5:10

.7(3.12)

Higher

scores:Homeen

vironmen

tismore

supportiveof

child’slangu

agean

dliteracy

Readsto

ChildDaily

(age

s2,

3,5)

Paren

tRep

ort

0–1

Age

2:0.55

(0.50)

1¼

Someo

neat

homeread

sto

childdaily

Age

3:0.54

(0.50)

Age

5:0.32

(0.47)

ChildSp

anke

dWithin

PastWee

k(age

s2,

3,5)

Paren

tRep

ort

0–1

Age

2:0.50

(0.50)

1¼

Childspan

kedwithin

pastweek

Age

3:0.50

(0.50)

Age

5:0.36

(0.48)

Paren

tSu

pportiven

essDuringPlay(age

s2,

3,5)

Observer

Rating/

Paren

t–Child

PlayVideo

1–7

Age

2:4.0(1.10)

Age

3:3.9(0.93)

Age

5:4.0(1.01)

Higher

scores:Greater

deg

reeto

whichtheparen

tis

emotionally

available

andphysically

andaffectively

presentto

thech

ildduringplay

Paren

tDetachmen

tDuringPlay(age

s2,

3,5)

Observer

Rating/

Paren

t–Child

PlayVideo

1–7

Age

2:1.4(0.86)

Age

3:1.23

(0.60)

Age

5:1.35

(0.77)

Higher

scores:Greater

deg

reeto

whichtheparen

twas

unaw

are,

inattentive,an

d/orindifferentto

thech

ild

duringplay

ChildHas

Reg

ularBed

time(age

s2,

3)Paren

tRep

ort

0–1

Age

2:0.59

(0.49)

1¼

Childhas

regu

larbed

time

Age

3:0.59

(0.49)

8Teach

ingActivities(age

s2,

3,5)

Paren

tRep

ort

0–16

Age

2:4.51

(0.84)

Age

3:4.4(0.85)

Higher

scores:Greater

engage

men

tin

more

types

ofor

more

freq

uen

tteachingactivities

withthech

ild

Age

5:11

.1(3.93)

Children’sBooks

(26ormore)(age

5)Paren

tRep

ort

0–1

Age

5:0.62

(0.49)

1¼

Has

26ormore

children’sbooks

Paren

tAtten

dsMee

tings/Open

Houses(age

5)Teach

erInterview

0–1

Age

5:0.8(0.38)

1¼

Paren

tattended

mee

tings/open

housesthisyear

Fam

ilyWell-B

eingan

dMen

talHealth

Dep

ression:Composite

International

Diagn

ostic

Interview,Dep

ressionModule

(age

2)

Paren

tInterview

0–10

0Age

2:15

.9(32.59

)Probab

ilityofdep

ression

(Continued)

25

DESIGN AND METHODS

TABLE1.

(Continued)

Domain

Outcome

Source

Possible

Ran

geMean(SD)

Interpretation

Dep

ression:Cen

terforEpidem

iologicStudies—

Dep

ressionScale—

Short

Form

(age

s3,

5)

Paren

tInterview

0–36

Age

3:7.7(7.00)

Age

5:8.0(7.15)

Higher

scores:Greater

number/freq

uen

cyofdep

ressive

symptoms

Paren

tingDistress(age

s2,

3)Paren

tInterview

12–6

0Age

2:25

.4(9.30)

Higher

scores:more

distressassociated

withparen

ting

Age

3:25

.2(9.59)

Fam

ilyConflict(age

s2,

3)Paren

tInterview

1–4

Age

2:1.7(0.54)

Higher

scores:more

conflict

Age

3:1.7(0.53)

Someo

nein

Household

Had

Alcohol/

DrugProblem

DuringPastYe

ar(age

5)

Paren

tInterview

0–1

Age

5:0.1(0.29)

1¼

Exp

osure

tohousehold

drugoralco

holproblems

within

thepastyear

ChildWitnessedViolence

(age

5)Paren

tInterview

0–1

Age

5:0.11

(0.32)

1¼

Childex

posedto

violence

Paren

tWas

Abusedin

PastYe

ar(age

5)Paren

tInterview

0–1

Age

5:0.09

(0.28)

1¼

Paren

tab

usedin

pastyear

Paren

tSe

lf-Sufficiency

Paren

tEmployed(age

s2,

3)Paren

tInterview

0–1

Age

2:0.72

(0.45)

1¼

Employedsince

enrollmen

torsince

last

interview

Age

3:0.85

(0.36)

Tim

eParen

tEmployedin

Past6Months(age

5)Paren

tInterview

1–5

Age

5:3.5(1.73)

Lower

values:Greater

fractionoftimeem

ployedduring

past6months(1

¼all,2¼

most

ofthetime,

3¼

abouthalfthetime,

4¼

less

than

halfthetime,

5¼

never)

Paren

tin

SchoolorJobTraining(age

s2,

3)Paren

tInterview

0–1

Age

2:0.43

(0.50)

Age

3:0.54

(0.50)

1¼

Paren

tin

schoolorjobtrainingsince

enrollmen

tor

timesince

last

interview

Inco

me(annual

atages

2,3;

monthly

atage,

5)Paren

tInterview

n.a.

Age

2:$1

4,41

5($12

,168

)Higher

numbers:Higher

inco

me

Age

3:$1

6,80

9($12

,865

)

Age

5:$2

,287

($1,80

8)

Form

alProgram

Exp

erience

Age

s3–

5

Form

alProgram

Exp

erience

Age

s3an

d4

Paren

tInterview

0–1

Age

5:0.42

(0.49)

1¼

Childin

form

aled

ucational

program

atages

3an

d4

Everin

HeadStart

Paren

tInterview

0–1

Age

5:0.51

(0.50)

1¼

Childattended

HeadStartat

somepointbetwee

n

ages

3an

d4

26

collected, yet we selected those that reflect all key domains of outcomesstudied.Wheremultiplemeasures were used to assess a domain, the followingrules were used to choose what to report: priority was given to (1) measuresthat we used at more than one time point; (2) psychometric strength; and(3) the measures’ total scores rather than individual scales, unless there was acompelling reason why an individual scale would be of particular interest.Information on psychometrics of the measures is included elsewhere for the0–3 phase of the study (ACF, 2002a, ACYF, 2001) so we summarize here onlymeasures reported for the prekindergarten phase (unless a measure was alsoused at ages 2 or 3).

Child Social, Emotional, and Approaches to Learning Outcomes

The Child Behavior Checklist (CBCL)—Aggressive Behavior Problems subscalewas used at all three ages (2, 3, and 5), to capture children’s aggressivebehavior. The CBCL for ages 1.5–5 (Achenbach & Rescorla, 2000) is a widelyused standardized assessment of child behavioral, emotional, and socialfunctioning. Items yield scores of 0, 1, and 2 (whether the statement is “nottrue,” “somewhat or sometimes true,” or “very true or often true” of the child).The aggressive behavior subscale includes 19 items, including itemsindicating defiance, poor impulse control, antisocial behavior, and maladap-tive attention seeking. Total scores on the subscale could range from 0 to 38.The observed internal consistency at age 5 was a ¼ .89 (n ¼ 2,014).

FACES Social Behavior Problems is a summary of 12 negative behaviorsreported by mothers. For each item, parents rated their children on a 3-pointscale—“not true,” “somewhat or sometimes true,” and “very true or oftentrue.” The scale combines 12 problem behavior items into a total problembehaviors score that represents the overall degree to which a child hasproblems with aggressive or disruptive behavior, hyperactivity, and withdrawntypes of behavior. The FACES total problem behaviors score ranges from 0 to24, with higher scores representing more frequent or severe behaviorproblems. In the FACES study, the internal consistency of the total problembehavior scores for Head Start children was .76 in 2002 (ACF, 2006, p. A-32).At age 5 in the EHSREP sample, the internal consistency was the same:a ¼ .76 (n ¼ 2,029).

Child Behavior During Parent–Child Semistructured Play. We collected similardata across the three waves of data collection, although the situation changedsomewhat. In the age 2- and 3-year assessments, the parent and child weregiven three bags of interesting toys and asked to play with the toys in sequence.At the prekindergarten assessment, the dyad was given two cans of Play-Doh, arolling pin, and a cookie cutter. At all ages, the task was videotaped, and childbehaviors were coded by child development researchers on a scale adapted forthis evaluation from the Three Box coding scales used in the NICHD Study of

27

DESIGN AND METHODS

Early Child Care (Owen, Barefoot, Vaughn, Dominguez, & Ware, 1996).Three aspects of children’s behavior with the parent were rated on a 7-pointscale: Negativity Toward Parent, the degree to which the child shows anger,hostility, or dislike toward the parent; Engagement, the extent to which thechild shows, initiates, or maintains interaction with the parent; and SustainedAttention With Objects, the degree to which the child is involved with the toyspresented. These were coded only at 2 and 3 years of age.

Coding methods and reliability on 2- and 3-year data are reportedelsewhere (Brady-Smith et al., 2013; Fuligni & Brooks-Gunn, 2013; Fuligniet al., 2013), so we focus on the prekindergarten data collection here.Coders were required to attain at least 85% agreement to within 1 pointof a “gold standard” rater before independently coding interactions.Thereafter, a random 9% of all tapes assigned were used to check coders’ongoing reliabilities. Intercoder reliability was assessed on ratings of 157tapes. Intercoder agreement was high for both the child variables (96%for child engagement of parent and 99% for negativity toward parent).Intraclass correlations (ICCs), which can be considered a generalizationof kappa coefficients for multiple raters (Fleiss & Cohen, 1973; Rae,1988; Shrout & Fleiss, 1979), ranged between .64 and .70 as they are basedon the substantially more stringent criterion of exact agreement (seeFuligni & Brooks-Gunn, 2013, for information on all observational coding inEHS).

The FACES Social Skills and Positive Approaches to Learning scale assesseschildren’s positive social interaction skills as well as their behavioraldispositions toward learning. Comprising seven items, it assesses the degreeto which children exhibit skills in making friends and accepting their ideas,enjoying learning and trying new things, showing imagination, comforting/helping others, and wanting to hear positive feedback. Like the FACESproblem behavior scales, for each item, parents rated their children on a3-point scale—“not true,” “somewhat or sometimes true,” and “very true oroften true.” The social skills and positive approaches to learning scale scoreranges from 0 to 14, with higher scores indicating higher levels of socialskills and more positive approaches to learning (ACF, 2006, p. A-31–A-32).In the FACES study, the internal consistency of this scale was .66 (ACF, 2006,p. A-32). At age 5 in EHSREP, the internal consistency was similar: a ¼ .64(n ¼ 2,048).

Leiter-R Examiner Rating Scales (Roid & Miller, 1997) were used to rateemotion regulation at the prekindergarten assessment (the Bayley BehavioralRating Scale [BRS; Bayley, 1993] was used at the earlier ages). At theconclusion of the child assessment, assessors completed a set of ratings aboutthe child’s behavior observed throughout the testing session. The Leiter-RExaminer Rating Scales evaluate children’s behavior in eight domains:(1) attention, (2) organization and impulse control, (3) activity level,

28

(4) sociability, (5) energy and feelings, (6) mood and regulation, (7) anxiety,and (8) sensory reactivity. Each item in the domains is rated on a 4-point scaleanchored to the rate at which a child exhibited a behavior during the testingsession: (0) “Rarely/never occurred” (i.e., less than roughly 10% of the time),(1) “Sometimes occurred” (i.e., around 10–50% of the time), (2) “Oftenoccurred” (i.e., approximately 50–90% of the time), or (3) “Usually/alwaysoccurred” (i.e., more than 90% of the time). We report on the Attentiondomain given the salience to school functioning, as well as one compositescale, Emotion Regulation (comprising energy and feelings, mood andregulation, anxiety, and sensory reactivity subscales). Raw scores on thesubscales and composites were converted to scaled scores by application ofage-appropriate norms. In the EHS prekindergarten child assessment,internal consistencies of the subscales and composites of the Leiter-Rexaminer ratings ranged from .81 (sociability subscale) to .96 (cognitive/social composite). The internal consistencies of the attention rating and theemotion regulation composite, were a ¼ .93 (n ¼ 1,821) and a ¼ .93(n ¼ 1,796).

Child Language, Cognitive, and Academic Skills Outcomes

At ages 3 and 5, the PPVT-III and TVIP were used to assess language,because the MacArthur Communicative Development Inventories (Fensonet al., 2000) used at age 2 were no longer appropriate. The PPVT-III measureslistening comprehension of spoken words in standard English for childrenand adults age 2½ and over (Dunn & Dunn, 1997; Williams & Wang, 1997).The child is presented with four pictures and is asked to point to the picturethat matches the word spoken by the interviewer. The PPVT-III was normedon a nationally representative sample of children and adults of various ages sothat raw scores can be converted to age-adjusted, standardized scores with amean of 100 and a standard deviation of 15. At age 5, its internal consistencywas .96 (n ¼ 1,691). Similarly, the Test de Vocabulario en Imagenes Peabody(TVIP)measures the listening comprehension of spoken words in Spanish forSpanish-speaking and bilingual children from age 2½ to 18 (Dunn, Padilla,Lugo, & Dunn, 1986). The TVIP was normed on a sample of Mexican andPuerto Rican children of various ages so that raw scores can be converted toage-adjusted, standardized scores with amean of 100 and a standard deviationof 15. In the EHS sample of Spanish-speaking children, the alpha was .96(n ¼ 174).

Two subscales of the Woodcock-Johnson Tests of Achievement—Revised(Woodcock & Johnson, 1990) were used to assess children’s preacademicskills: Letter Word Identification, ability to identify decontextualizedindividual letters and words; and Applied Problems, skills in analyzing

29

DESIGN AND METHODS

and solving practical problems in mathematics. The only analogousmeasure at ages 2 and 3 was the Bayley Mental Development Index(MDI; Bayley, 1993). The internal consistency of the Letter-WordIdentification test with preschool children averages .92 (Woodcock &Johnson, 1990, p. 100), whereas in this sample a was .84 (n ¼ 1,757). TheSpanish version of the test, the Baterıa Woodcock-Munoz-R Identificacion deLetras y Palabras (Woodcock &Munoz-Sandoval, 1996), was used for childrenwhose primary language was Spanish. The internal consistencies of theIdentifcacion de Letras y Palabras test with Spanish-speaking FACES studychildren was .83 in 2002 (ACF, 2006, pp. A-16–A-17). In the EHSprekindergarten evaluation a was .96 (n ¼ 115). The Spanish version ofApplied Problems, the Woodcock-Munoz-R Problemas Aplicados (Woodcock &Munoz-Sandoval, 1996) was used for children whose primary languagewas Spanish. Since the Woodcock-Munoz-R Problemas Aplicados wasdeveloped as a parallel measure to the Woodcock-Johnson-R AppliedProblems test; scores on the two versions are directly comparable andthey were combined in analyses. The internal consistency of the AppliedProblems test with preschool children averages .91 (Woodcock & Johnson,1990, p. 100), and the internal consistency in FACES was also .91 in2002 (ACF, 2006, pp. A-16–A-17). In EHS, the internal consistency wasa ¼ .85 (n ¼ 1,870).

Leiter International Performance Scale-Revised—Attention Sustained (Leiter-R;Roid & Miller, 1997) was developed as a set of cross-culturally appropriateintellectual functioning assessments for individuals with limited verbalabilities. The Attention Sustained subtest for 4- to 5-year-olds is a timedcancellation task. Children are presented with a target figure (“flower,”“butterfly,” “funny guy,” or “goat”) and are asked to find and cross out asmany of the target figures on the page as possible and to work as fast as theycan within the allotted time (which varies by target figure from 30 to 60seconds). Poor performance in the absence of a diagnosable motorimpairment is typically attributed to difficulty in sustaining attention to adetailed task (Roid & Miller, 1997, p. 118). The Attention Sustained testresults in three raw scores: a total correct score, a total errors score, and a totalcorrect adjusted-for-errors score. Each set of scores is converted to scaledscores by application of age-appropriate norms. Because of the wide range ofchildren’s ages at the prekindergarten assessment, only scaled scores areused in analyses in this monograph. The Leiter-R manual reports anAttention Sustained internal consistency for 4- to 5-year-olds of .83 (p. 157),and test–retest reliability for 6- to 18-year-olds of .85 (Roid & Miller, 1997, p.162). In FACES, the internal consistency of the attention sustained test scoresamong 4- to 5-year-old Head Start children was a ¼ .80 (n ¼ 920) in 2002(ACF, 2006, pp. A-17–A-18). The internal consistency in the EHS study wasa ¼ .75 (n ¼ 1,782).

30

Child Health

At ages 2 and 3 years, parents were asked if the child had visited theemergency room because of accident or injury. At the same time, parents were askedabout the child’s immunization status, i.e., whether they were up to date onimmunizations. At 2, 3, and 5 age points, parents were asked if their child hadan identified disability and an Individualized Education Plan (IEP). At age 5,parents were also asked about the presence and severity of a speech problem.

Parenting and the Home Environment

At all three ages we used the HOME Support for Language and Literacysubscale of the Home Observation for Measurement of the Environment (HOME),one of the most widely used measures designed to assess characteristics of achild’s home environment important for stimulating optimal child develop-ment (Bradley, 1994; Bradley & Caldwell, 1988; Caldwell & Bradley, 1984). Ashortened form was used in the EHS Study (Mott, 2004). The Support ofLanguage and Literacy subscale measures the breadth and quality of themother’s speech and verbal responses to the child during the home visit, asrated by the interviewer; whether the parent encourages the child to learnshapes, colors, numbers, and the alphabet; the presence of books, toys, andgames accessible to the child; and whether the parent reads to the childseveral times per week. Items are obtained by a combination of parent reportand interviewer observation (Fuligni, Han, & Brooks-Gunn, 2004).

Parent Behavior During Parent–Child Semistructured Play measures parentbehavior during a semistructured play task. (See details above in childoutcomes.) Several aspects of parents’ behavior with their child were rated on a7-point scale (Fuligni, Brooks-Gunn,&Brady-Smith, 2008).Here we report twoaspects, one negative and one positive: Supportiveness, an average of parentalsensitivity, cognitive stimulation, and positive regard during play with the child;and Detachment, the extent to which the parent is inattentive to the child,inconsistently attentive, or interacts with the child in an indifferent manner(Fuligni&Brooks-Gunn, 2013; Fuligni et al., 2013; Brady-Smith et al., in press).Intercoder reliability of the child andparent variables was assessed on ratings of157 tapes. Intercoder agreement was high for both the parent variables (94%for supportiveness and 92% for detachment). ICCs, which can be considered ageneralization of kappa coefficients for multiple raters (Fleiss & Cohen, 1973;Rae, 1988; Shrout & Fleiss, 1979), ranged between .64 and .70 as they are basedon the substantially more stringent criterion of exact agreement.

Specific parent interview questions analyzed as parenting outcomesincluded frequencies of engagement in teaching and play activities,5 whethersomeone reads to the child daily, the number of children’s books in the household,and whether the child had been spanked within the past week. At age 2, parentswere asked about whether they had a regular bedtime for their child. The parent

31

DESIGN AND METHODS

had to name the time and report that the child went to bed at that time at least4 of the past 5 weekdays.

Family Well‐Being and Mental Health

The Parental Distress subscale of the short form of the Parenting StressIndex was used at 2 and 3 years.

The Center for Epidemiological Studies Depression Scale. Short Form (CESD-SF) was used to measure symptoms of depression at all three ages (Ross,Mirowsky, &Huber, 1983). The scale includes 12 items taken from the full, 20-itemCESD scale (Radloff, 1977). Respondents were asked the number of daysin the past week they had a particular symptom. Symptoms include poorappetite, restless sleep, loneliness, sadness, and lack of energy. In the EHSprekindergarten sample, the estimated internal consistency of caregiver CES-D scores was a ¼ .88 (n ¼ 2,033).

The Family Conflict subscale of the Family Environment Scale was used at 2and 3. Parents respond to items on a 4-point scale, where 4 indicates higherlevels of agreement with statements such as, “We fight a lot,” and “We hardlyever lose our tempers.” Items were recoded and averaged so that 4 indicatedhigh level of conflict (Moos & Moos, 1976).

The prekindergarten parent interview included a number of itemsrelated to family risk factors. Specific parent interview health and well-beingquestions included whether anyone in the household had drug or alcoholproblems, whether the child had witnessed violence, and whether the parent hadbeen abused.

Parent Self‐Sufficiency

In the Parent Services Interviews (corresponding with the 2- and 3-year-old data) and in the 5-year-old interview, parents were asked about self-sufficiency, including Education (parents were asked about education and jobtraining programs that they had participated in during the follow-up period);Employment (parents were asked about jobs that they had held during thefollow-up period); and Family Income (for the first two waves, parents wereasked about their family income during the past year). At the age 5 interview,parents were asked about monthly household income.

ANALYSIS OF RESPONSE RATES AND RESPONSE BIAS

Response Rates

High response rates, maximizing preservation of the original randomizedEHS sample, are a key requirement for ensuring the validity of follow-up study

32

results. In calculating response rates, nonrespondents are defined as anysample member for whom a designated data element was not collected. Theyinclude sample members whose children died (and who were not, by design,followed up), withdrew from the study, could not be located, or refused aninterview/assessment/observation. At least one element of tracking orprekindergarten data was collected for 78% of the original 3,001 samplemembers (see Figure 1). Prekindergarten data were collected on 2,142 (71%)of the sample. Figure 1 shows the sample disposition from the original 3,001sample members randomized to the number with complete parent interviewdata.

Response Bias Analysis6

Effects that sample attrition exerts on the composition of the follow-upsample—expressed as response bias—are as important as response rates.Internal validity of findings is maximized when there are few differences inbaseline characteristics between treatment and control respondents (Cook &

FIGURE 1.—Sample retention and attrition at each data collection point in the EHSREPNote: Overall sample disposition for 14 months through prekindergarten parent interviews.

33

DESIGN AND METHODS

Campbell, 1979; Shadish, Cook, & Campbell, 2002). Attrition bias, anespecially important threat to internal validity in longitudinal studies, isminimized when the follow-up sample still resembles, on baseline character-istics, the original sample that was randomized. Attrition bias is tested bycomparing treatment group respondents to nonrespondents and comparisongroup respondents to nonrespondents.

Overall, analysis indicates that the treatment and control groups arecomparable at the prekindergarten follow-up point, thereby maintaininginternal validity of the results. Among prekindergarten respondents, only twoof the differences between program and control groups are statisticallysignificant, even at the liberal significance level of .10 (analyses available uponrequest). This is fewer than half of the five differences that would be expectedby chance. Therefore, we conclude that the impact analyses reported inChapters III and IV represent valid assessments of the effects of the EHStreatment.

Furthermore, despite some statistical differences between respondentsand nonrespondents onmany important characteristics, the prekindergartensample remains comparable to the baseline sample and the magnitude ofattrition bias may be modest (analyses available upon request). Specifically,the baseline and prekindergarten samples are similar on such key character-istics as age of mother at birth of focus child, mother’s educationalattainment, whether employed or in school/training, number of childrenin the household, income, age of focus child at enrollment, focus child withpreterm birth or low birthweight, or having established biological orenvironmental risks. The EHS sample at prekindergarten overrepresentscenter-based programs (22.7% of respondents were in center-based programsvs. 13.4% of nonrespondents) and programs that were later implementers ofthe Head Start Program Performance Standards; while under-representinghome-based and incomplete implementers (see ACF, 2002a, for definitions oflevels of program implementation). The sample at prekindergarten issomewhat less disadvantaged than the baseline sample: Respondents weremore likely to own their own home, less likely to be receiving TANF or FoodStamps, more likely to be low risk, andmore likely to be living with a spouse orother adults.

These differences suggest that some attrition bias may be manifest in anumber of characteristics and some caution is warranted when drawinginferences from the prekindergarten analyses and applying them to theoriginal EHS sample. The magnitudes of the statistically significant differ-ences, however, are not large and values of the characteristics remain well withthe ranges of variability observed among low-income families eligible for EHSservices nationally, suggesting that practical implications of the potential biasmay be small. Furthermore, the program and control groups remaincomparable on baseline characteristics. We conclude, therefore, that findings

34

from the prekindergarten sample in the EHS continue to have internalvalidity for assessing the impacts of EHS.

CONCLUSIONS

The next several chapters focus on questions about the enduring effectsof EHS program participation as well as the contributions of formal programexperiences ages 3–5. Although the specific questions considered in eachchapter differ and the methods employed to address them vary, all are set inthe context of the research design, methods, and instrumentation describedhere. The age 5 response rates were consistent with patterns and variations bydata collection component observed in the age 3 data collection wave (seeACF, 2002a, chapter II). Importantly, the respondents in the program andcontrol groups were virtually indistinguishable on a host of baselinecharacteristics at the age 5 assessment point. Finally, although a number ofdifferences between baseline and follow-up samples (respondents vs.nonrespondents) on baseline characteristics were present, the magnitudesof the differences were generally modest and the values well within rangestypically observed among EHS families.

NOTES

5. The teaching activities, which are also listed inTable 1, include (1) Told child a story; (2)Taught child letters, words, or numbers; (3) Taught child songs or music; (4) Worked on artsand crafts together; (5) Played with indoors toys or games; (6) Played a game, sport, orexercised together; (7) Took child along while doing errands; and (8) Involved child inhousehold chores.

6. Results available from authors upon request.

35

DESIGN AND METHODS