Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development...
-
Upload
valerie-jenkinson -
Category
Documents
-
view
217 -
download
0
Transcript of Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development...
![Page 1: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/1.jpg)
Statistical Analysis Overview ISession 2
Peg Burchinal
Frank Porter Graham
Child Development Institute,
University of North Carolina-Chapel Hill
![Page 2: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/2.jpg)
Overview: Statistical analysis overview I-b
• Nesting and intraclass correlation
• Hierarchical Linear Models
– 2 level models
– 3 level models
![Page 3: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/3.jpg)
Nesting
• Nesting implies violation of the linear model assumptions of independence of observations
• Ignoring this dependency in the data results in inflated test statistics when observations are positively correlated– CAN DRAW INCORRECT CONCLUSIONS
![Page 4: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/4.jpg)
Nesting and Design• Educational data often collected in schools,
classrooms, or special treatment groups– Lack of independence among individuals -> reduction in
variability• Pre-existing similarities (i.e., students within the cluster are more
similar than a students who would be randomly selected)• Shared instructional environment (i.e., variability in instruction
greater across classroom than within classroom)
• Educational treatments often assigned to schools or classrooms – Advantage: To avoid contamination, make study more
acceptable (often simple random assignment not possible)– Disadvantage: Analysis must take dependencies or
relatedness of responses within clusters into account
![Page 5: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/5.jpg)
Intraclass Correlation (ICC)
• For models with clustering of individuals – “cluster effect”: proportion of variance in the
outcomes that is between clusters (compares within-cluster variance to between-cluster variance)
– Example – clustering of children in classroom. ICC describes proportion of variance associated with differences between classrooms
![Page 6: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/6.jpg)
Intraclass Correlation
• Intraclass correlation (ICC) – measure of relatedness or dependence of clustered data– Proportion of variance that is between clusters
– ICC or = b / (b + w)
– ICC = 0 } no correlation among individuals within a cluster
= 1 } all responses within the clusters are identical
![Page 7: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/7.jpg)
Nesting, Design, and ICC
• Taking ICC into account results in less power for given sample size – less independent information
• Design effect = mk / (1 + (m-1))– m= number of individuals per cluster– K=number of clusters– =ICC
• Effective sample size is number of clusters (k) when ICC=1 and is number of individuals (mk) when ICC=0
![Page 8: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/8.jpg)
ICC and Hierchical Linear Models
• Hierarchical linear models (HLM) implicitly take nesting into account– Clustering of data is explicitly specified by
model– ICC is considered when estimating standard
errors, test statistics, and p-values
![Page 9: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/9.jpg)
2 level HLM
• One level of nesting– Longitudinal: Repeated measures of individual
over time• Typically - Random intercepts and slopes to
describe individual patterns of change over time
– Clusters: Nesting of individuals within classes, families, therapy groups, etc.
• Typically - Random intercept to describe cluster effect
![Page 10: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/10.jpg)
2 level HLM Random-intercepts models
• Corresponds to One-way ANOVA with random effects (mixed model ANOVA)
• Example: Classrooms randomly assigned to treatment or control conditions– All study children within classroom in same condition
– Post treatment outcome per child (can use pre-treatment as covariate to increase power)
– Level 1 = children in classroom
Level 2 = classroom
ICC reflects extent the degree of similarity among students within the classroom.
![Page 11: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/11.jpg)
2 Level HLMRandom Intercept Model
• Level 1 – individual students within the classroom– Unconditional Model: Yij = B0j + rij
– Conditional Model: Yij = B0j + B1 Xij + rij
• Yij= outcome for ith student in jth class
• B0j= intercept (e.g., mean) for jth class
• B1= coefficient for individual-level covariate, Xij
• rij= random error term for ith student in jth class,
E ( rij) = 0, var (rij) =
![Page 12: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/12.jpg)
2 Level HLMRandom Intercept Model
• Level 2 – Classrooms – Unconditional model: B0j
= 00 + u 0j
– Conditional model: B0j = 00 + 01 Wj1 + 02 Wj2 + u 0j• B0j j= intercept (e.g., mean) for jth class• 00 = grand mean in population• 01 = treatment effect for Wj, dummy variable indicating
treatment status-.5 if control; .5 if treatment
• 02 coefficient for Wj2, class level covariate• u 0j = random effect associated with j-th classroom
E (uij) = 0, var (uij) =
![Page 13: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/13.jpg)
2 Level HLMRandom Intercept Model
• Combined (unconditional)– Yij = 00 + u 0j + rij
• Yij = B0j + rij
• B0j = 00 + u 0j
• Combined (conditional)– Yij = 00 + 01 Wj + 02 Wj2 + B1 Xij + u 0j + rij
• Yij = B0j + B1 Xij + rij
• B0j = 00 + 01 Wj + 02 Wj2 + u 0j
• Var (Yij ) = Var ( u 0j + rij ) = (
• ICC = = (
![Page 14: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/14.jpg)
Example2 level HLM Random Intercepts
• Purdue Curriculum Study (Powell & Diamond)– Onsite or Remote coaching– 27 Head Start classes randomly assigned to onsite
coaching and 25 to remote coaching– Post-test scores on writing– Onsite: n=196, M=6.70, SD=1.54
Remote: n=171, M=7.05, SD=1.64
![Page 15: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/15.jpg)
Example2 level HLM Random Intercepts
• Level 1: Writingij = B0j + B1 Writing-preij + rij
B1 =.56, se=.05, p<.001
E ( rij) = 0, var (rij) = 1.67
• Level 2: B0j = 00 + 01 Onsitej + u 0j
00 (intercept- remote group adjusted mean) = 3.74, se =.31
01(Onsite-Remote difference) = -.37, se=.17, p=.03
E (uij) = 0, var (uij) =
• ICC = (
![Page 16: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/16.jpg)
2 Level HLM - Longitudinal (random-slopes and –intercepts models)
• Corresponds NOT to One-way ANOVA with random effects
• Example: Longitudinal assessment of children’s literacy skills during Pre-K years– Level 1 = individual growth curve
Level 2 = group growth curve
![Page 17: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/17.jpg)
Level 1- Longitudinal HLM
• Level 1 – individual growth curve – Unconditional Model: Yij = B0j + B1j Ageij + rij
– Conditional Model: Yij = B0j + B1j Ageij + B2 Xij + rij• Yij= outcome for ith student on the jth occasion• Ageij = age at assessment for ith student on the jth occasion
• B0j= intercept for ith student• B1j= slope for Age for ith student• B2= coefficient for tiem-varying covariate, Xij\
• rij= random error term for ith student on the jth occasion E ( rij) = 0, var (rij) =
![Page 18: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/18.jpg)
Level 2 – Longitudinal HLM• Level 2 – predicting individual trajectories
– Unconditional model: B0j = 00 + u 0j
B1j = 10 + u 1j
– Conditional model: B0j = 00 + 01 Wj1 + 02 Wj2 + u 0j
B1j = 10 + 11 Wj1 + 12 Wj2 + u 1j
• B0j= intercept for ith student B1j= slope for Age for ith student
• 00 = intercept in population10 = slope in population
• 01 = treatment effect on intercept for Wj, student -level covariate
11 = treatment effect on slope for Wj, student -level covariate
![Page 19: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/19.jpg)
Level 2 – Longitudinal HLM• Level 2 – predicting individual trajectories
– Unconditional model: B0j = 00 + u 0j
B1j = 10 + u 1j
– Conditional model: B0j = 00 + 01 Wj1 + u 0j
B1j = 10 + 11 Wj1 + u 1j
• u 0j = random effect for individual intercept u 0j = random effect for individual slope• E (u0j) = 0, var (u0j) =
E (u1j) = 0, var (u1j) = cov u 0j, u 1j) =
var u 0j, u 1j)=
• level 1 and 2 error terms independent cov (rij, T) = 0
![Page 20: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/20.jpg)
Example – Longitudinal HLM• Purdue Curriculum Study (Powell &
Diamond)Level 1 – estimating individual growth curves for
children in one treatment condition (Remote)– Level 2 – estimating population growth curves
for Remote condition
Blending Pre Post Follow-up
N
M (sd)
187
9.48 (5.34)
171
13.75 (4.57)
63
15.14 (4.60)
![Page 21: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/21.jpg)
Example
• Level 1: blendingij = B0j + B1j Ageij + rij
estimated• Level 2: B0j = 00 + 01 Wj1 + u 0j
B1j = 10 + u 1j
Estimated results
Intercept 00 = 11.86 (se=.48), 00 = 10.03**
season 01 = 2.43* (se=.70)
Slope 10 = 1.51* (se=.60), 11 = 4.24** 10 = -1.45**
![Page 22: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/22.jpg)
3 level HLM • 2 levels of nesting• Examples
– Longitudinal assessments of children in randomly assigned classrooms
• Level 1 – child level data• Level 2 – child’s growth curve• Level 3 – classroom level data
– Two levels of nesting such as children nested in classrooms that are nested in schools
• Level 1 – child level data• Level 2 – classroom level data• Level 3 – school level data
![Page 23: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/23.jpg)
3 level Model-Random Intercepts• Children nested in classrooms, classrooms nested
in schools– Level 1 child-level model Yijk = ojk + eijk
• Yijk is achievement of child I in class J in school K
• ojk is mean score of class j in school k
• eojk is random “child effect”
– Classroom level model ojk = 00k + r0jk
• 00k is mean score for school k
• r0jk is random “class effect”
– School level model 00k = 000 + u00k
• 000 is grand mean score
• u00k is random “school effect”
![Page 24: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/24.jpg)
3 level Model-Random Intercepts• Children nested in classrooms, classrooms nested
in schools– Level 1 child-level model Yijk = ojk + eijk
• eojk is random “child effect”,
E (eijk) = 0 , var(eijk) =
– Within classroom level model ojk = 00k + r0jk
• r0jk is random “class effect”,
E (r0jk ) = 0 , var(r0jk ) =
Assume variance among classes within school is the same
– Between classroom (school) 00k = 000 + 01 trt + u00k
E (u00k ) = 0 , var(u00k ) =
![Page 25: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/25.jpg)
Partitioning variance
• Proportion of variance within classroom
• Proportion of variance among classrooms within schools
• Proportion of variance among schools
![Page 26: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/26.jpg)
3 Level HLM – level 2 longitudinal and level 3 random intercepts
• Typically – treatment randomly assigned at classroom level, children followed longitudinally (e.g., Purdue Curriculum Study)– (within child) Level 1: Yijk = 0j k + 1j k Ageijk + rijk
E (eijk) = 0 , var(eijk) =
– (between child ) Level 2: 0jk
= 00k + r 0jk; 1j k = 10k + r 1jk
E (r0jk ) = 0 , var(r0jk ) = E (r1jk ) = 0 , var(r1jk ) =
– (between classes) Level 3: 00k = 00 + u00k; 10k = 10 + u10k
E (u00k ) = 0 , var(u00k ) = E (u10k ) = 0 , var(u10k ) =
![Page 27: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/27.jpg)
Example Purdue Curriculum Study
• Level 1 – individual growth curve• Level 2 – classroom growth curve• Level 3 – treatment differences in classroom growth
curves
Writing Pre Post Follow-up
Onsite
M (se)
N=199
5.98 (1.49)
N=196
6.70 (1.54)
N=79
6.92 (1.74)
Remote
M (se)
N=187
6.01 (1.55)
N=171
7.04 (1.64)
N=63
7.48 (1.62)
![Page 28: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/28.jpg)
Purdue Curriculum Study
![Page 29: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/29.jpg)
Threats
• Homogeneity of variance – at each level– Nonnormal data with heavy tails– Bad data– Differences in variability among groups
• Normality assumption– Examine residuals– Robust standard error (large n)
• Inferences with small samples
![Page 30: Statistical Analysis Overview I Session 2 Peg Burchinal Frank Porter Graham Child Development Institute, University of North Carolina-Chapel Hill.](https://reader034.fdocuments.in/reader034/viewer/2022051401/56649c7b5503460f9492f2ae/html5/thumbnails/30.jpg)
3 Level HLMLongitudinal assessments of
individual in clustered settings