Presentation at National Early Childhood Inclusion Institute Chapel Hill, NC May 21, 2014 Patricia...

Measures that Support Implementation of High Quality

Inclusive Practices

Presentation at National Early Childhood Inclusion InstituteChapel Hill, NC

May 21, 2014

Patricia Snyder, PhDProfessor and David Lawrence Jr. Endowed Chair in Early Childhood Studies University of Florida

Mary Louise Hemmeter, PhDProfessor Department of Special EducationVanderbilt University

Elena Soukakou, PhDSenior LecturerUniversity of Roehampton, London, UK

Moderator: Pamela J. Winton, PhDSenior Scientist & Director of Outreach FPG Child Development Institute

Agenda

Welcome and introductionOverview of three measuresQuestion & answer after each measure

Facilitated discussion

Teaching Pyramid Observation Tool (TPOT) (Fox, Hemmeter, & Snyder, 2013) Available from Paul Brookes Publishing

Embedded Instruction Observation System --Teacher Version (EIOS-T) (Crowe, Snyder, Crow, Mullin, & Embedded Instruction Project, 2011) Available from the authors

Inclusive Classroom Profile (ICP) (Soukakou, 2012).Available from the author

Overview of the Inclusive Classroom

Profile (ICP) Dr. Elena Soukakou,

Author

Copyright © 2013 by Elena P. Soukakou

Measures inclusive, classroom-level practices that support the individualized needs of children with disabilities

ICP Content

Structured observation measure1–7 point rating scale12 items

ICP Structure

Young children with

disabilities can experience low

quality in classes that

are otherwise rated as being

of high quality

Wolery, et al., 2000

National Professional Development Center on Inclusion

1. Adaptation of space and materials2. Adult involvement in peer interactions3. Adult guidance of children’s activities and play4. Conflict resolution5. Membership6. Relationships between adults and children7. Support for social communication8. Adaptation of group activities9. Transitions between activities10. Feedback11. Family-professional partnerships12. Monitoring children’s learning

ICP Items

As a research toolAs a classroom evaluation tool

As a professional development tool


How Can the ICP Be Used?

Children with identified disabilities in the context of classroom activities and social interactions with adults and peers

Teachers, teacher assistants, specialists


Who Is Being Observed?

Observation Teacher

interview Document

review


Administration

1.Exploratory research2.Conceptualization and domain delineation3. Item generation4.Expert review5.Pilot study in the UK

See Soukakou, E. P. (2012). Measuring quality in inclusive preschool classrooms: Development and validation of the Inclusive Classroom Profile (ICP). Early Childhood Research Quarterly, 27(3), 478-488)

ICP Development: Five Phases

1st pilot study in the UK showed promising results on reliability & validity (Soukakou, 2012)

2nd pilot study in the US in collaboration with:


Pilot Studies on the ICP

NC Department of Instruction,Exceptional Children

Did assessors learn to use the ICP with accuracy?

What is the evidence for reliability and validity?

Did assessors find the ICP useful and acceptable for program evaluation?


ICP Pilot Study (US): Research Questions

51 inclusive classrooms in one state Public Pre-K (5),

Head Start (13), Developmental Day programs (13), Other child care centers (20)

150 children with disabilities Mean age of children= 4.43 years


Sample: Classrooms

51 ICP assessments 50 ECERS-R assessments Assessor survey for

gathering data on ICP acceptability


Procedures

ITEM ICC ICP 1 Adaptation of Space, Materials and Equipment

.62 ICP 2 Adult Involvement in Peer Interactions

.78 ICP 3 Adult Guidance of Children’s Play

.11 ICP 4 Conflict Resolution

.70 ICP 5 Membership

.84 ICP 6 Relationships between Adults and Children

.75 ICP 7 Support for Communication

.51 ICP 8 Adaptations of Group Activities

.72 ICP 9 Transitions between Activities

.95ICP 10 Feedback

.60ICP 11 Family-Professional Partnerships

.99ICP 12 Monitoring Children’s Learning

.99

Results: Inter-Rater Reliability

ECERS-R Scale ICP Total Score

Space and Furnishings 0.48***Personal Care 0.21**Language and Reasoning 0.47***Program Structure 0.29*Activities 0.30*Interactions 0.38**Parent and Staff 0.38**ECERS Total Score 0.48***

Results: Rank-Order Correlations Between ICP and ECERS

Note: *p<.05, **p<.01, ***p<.001

Results: Discriminant Validity

Mean(SE)/B(SE)

Child Care 3.67 (0.15)a

Developmental Day

5.12 (0.19)b

Head Start 4.64 (0.19)b

Public Pre-K 4.76 (0.30)b

Note: Means not sharing superscripts are significantly different.

On a 1–5 point scale, 4 assessors:

Rated the importance of the ICP constructs measured very highly (m= 5)

Would highly recommend the ICP measure to others (m=5)

Found the measure easy to administer (m= 4)

Felt well prepared after the reliability training observations (m=4)


Results: Social Validity

Assessors established adequate administration and reliability proficiency upon training.

Evidence for construct validity.

Differences in quality across types of programs

Assessors found the ICP easy to use and useful for program evaluation


Summary of Findings

Training program for users. Online overview materials at http://npdci.fpg.unc.edu/measuring-quality-inclusion-inclusive-classroom-profile

Professional development curriculum for PD providers/consultants


Next Steps

http://npdci.fpg.unc.edu/measuring-quality-inclusion-inclusive-classroom-profile




http://pdc.fpg.unc.edu

PDC@FPG

Putting Knowledge to Work

ICP training available at PDC@FPG

Questions?

Teaching Pyramid Observation Tool

(TPOT)Mary Louise Hemmeter

Lise FoxPatricia Snyder

Tertiary Intervention: Few Children

Secondary Prevention:

Some Children

Universal Promotion: All Children

The Pyramid Model: Promoting Social and Emotional Competence and Addressing

Challenging Behavior

27

TPOT Materials

28

TPOT Score Sheet & TPOT Manual

TPOT Sample Items

http://products.brookespublishing.com/Teaching-Pyramid-Observation-Tool-TPOT-for-Preschool-Classrooms-Set-Research-Edition-P727.aspx

Teaching Pyramid Observation Tool (TPOT)

The TPOT was developed to measure the fidelity with which teachers implement Pyramid Model practices

Provides information that can be used to:Describe “quality” of implementation of TPOT practicesCompare implementation within and across

teachers/classroomsSupport program-wide implementation and improvement

activities Identify needs of teachers for training and implementation

support

Using the TPOT Observations

Conducted for a minimum of 2 hours Must observe centers or free play, at least one

teacher-directed activity, and the transitions between activities

Focus of observation is primarily lead teacher’s implementation of practices, but consider all adults

InterviewsFor those practices that might not or cannot

be observed during the 2-hour observation

Organization of the TPOT: Three Subscales Key Practice Items: Multiple indicators

associated with each item Each indicator rated yes, no, or N/O (only when noted)

Red Flags Each item rated yes or no

Using Effective Strategies for Responding to Challenging Behavior Item only scored when challenging behavior observed Includes three indicators for responses to each incident of

challenging behavior Each of these three Indicators rated as yes or no for each

incident

14 Key Practice Items

Observation items1. Schedules, routines, and

activities (SR)2. Transitions between activities

(TR)3. Supportive conversations (SC)4. Promoting engagement (ENG)5. Providing directions (PD)6. Collaborative teaming (CT)7. Teaching behavior

expectations (TBE)8. Teaching social skills and

emotional competencies (TSC)

Observation and interview items9. Teaching friendship skills (FR)10. Teaching children to express

emotions (TEE)11. Teaching problem-solving

(TPS)

Interview items12. Interventions for children

with persistent challenging behavior (PCB)

13. Connecting with families (COM)

14. Supporting families in using Pyramid Model practices (INF)

Example Key Practice Item

33

Item

Indicator

Score

Columns

No Opportunity

Key Practice Item: Schedules, Routines, & ActivitiesObservation Only

34

Key Practice Item: Teaching Friendship Skills Observation AND Interview

35

Interview Questions for Teaching Friendship Skills

Red Flags Subscale

37

Challenging Behavior Subscale

38

Defining the Behavior – What to Note

40

Studies to Examine Psychometric Integrity of TPOT™ Scores

Primarily Head Start classrooms TPOT observations

50 classrooms 2 raters 3 occasions

300 total TPOTs Generalizability study to look at

dependability of scores across items, raters, occasions of measurement

CLASS (Pianta, LaParo, & Hamre, 2008)

50 classrooms Administered between 2nd and 3rd TPOT

observation Inter-observer agreement for 34% (n = 17) Convergent score validity

Work reported was supported, in part, by Institute of Education Sciences grant (R324A07212) to Vanderbilt University. The information and opinions expressed are those of the authors, not the funding agency.

Generalizability Study Findings Less than .01% of

variance on key practice indicators due to raters

<1% of variance on key practice indicators due to occasion

6.12% of variance on key practice indicators due to classroom/teacher

Phi coefficient = absolute decisions

G coefficient = relative decision Phi key practice indicators = .89 G key practice indicators = .95 Phi red flags = .76 G red flags = . 84

Good news! Dependability in rank ordering of

classrooms and dependability in scores across raters, occasions

41

Snyder, P., Hemmeter, M.L., Fox, L., Bishop, C., & Miller, M.D. (2013). Developing and gathering psychometric evidence for a fidelity instrument. Journal of Early Intervention, 35, 150-172.

42

TPOT™ and CLASS™ Correlations

N = 50TPOT Key Practices

Emotional Support (ES)

.70

Classroom Organization (CO)

.73

Instructional Support (IS)

.76

Snyder, P., Hemmeter, M.L., Fox, L., Bishop, C., & Miller, M.D. (2013). Developing and gathering psychometric evidence for a fidelity instrument. Journal of Early Intervention, 35, 150-172.

Noteworthy Correlations:TPOT™ and CLASS™

TPOT Key Practices subscale and every CLASS dimension and domain

TPOT Red Flags subscale and every CLASS dimension and domain (negative relationships)

General teaching items on TPOT (SR, TR, SC, ENG, PD) and each CLASS dimension and domain

Most targeted teaching items on TPOT (TBE, TSC, TEE, TPS, FR) and Instructional Support CLASS domain

TPOT Connecting with Families with each dimension and domain on CLASS

Additional detail in Chapter 7 in TPOT manual43

Another Study: TPOT™ and ECERS-R

Noteworthy correlationsTPOT Key Practices subscale and overall ECERS-R10 of the 14 TPOT key practice items and overall

ECERS-RTPOT Red Flags subscale and overall ECERS-R

(negative relationships)

Additional detail in Chapter 7 in TPOT manual

44

45

Another Study: Pre-SET and Select TPOT-Pilot Version Key Practice Items(N = 31 Classrooms)

Note. Adapted from Steed and Pomerleau (2012). N = 31 classrooms. a = Seven environmental items included on pilot version of TPOT. * = p < .05 ** = p < .01

Additional detail in Chapter 7 in TPOT manual

46

1 2 3 40

10

20

30

40

50

60

70

80

90

100

Use of TPOT in Potential Efficacy Study (N = 40)Control

InterventionT

ota

l T

PO

T s

co

re

Training

Sept. Nov. Feb. Apr.

Comparison

Interven-tion

Coaching

Figure 1. Mean TPOT scores across 4 waves. Total TPOT indicators = 108. Wave 4 [t(40.03)=6.80, p<.001, Cohen’s d=2.6) Hemmeter, M.L., Fox, L., Snyder, P., & Algina, J. (2011, April). Efficacy of a classroom-wide model for promoting social-emotional development and preventing challenging behavior. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.

Additional Results from Potential Efficacy Study: SSIS Child Outcomes

Adjusted Means Cohen’s d Effect Size

Intervention Control

Target Children

Social 88.6 84 .41

Problem Behavior 108.7 115.5 -.52*

Non Target Children

Social 103.8 96.4 .46*

Problem Behavior 95.2 99 -.29

Mean frequency positive social interactions for 60 min observation session across 4 waves. Average frequency of positive social interactions for target children in each classroom used to derive means for each group.Mean experimental Wave 4 = 19.96 (SD = 10.7); control Wave 4 = 15.65 (SD = 8.5)

Additional Results from Potential Efficacy Study: Target Child Social Interaction Behaviors

Using the TPOT to Inform Decision Making Using the TPOT in coaching

Running TPOTFormal TPOTGoal setting/action planning

Using the TPOT program wideMonitor implementation of PW implementationPlan professional development

Using the TPOT in monitoring/evaluation

Sched

ules, ro

utines, an

d activiti

es

Transiti

ons betw

een ac

tivities

Supporti

ve co

nversa

tions

Promoting c

hildren

’s enga

gemen

t

Provid

ing dire

ctions

Collaborati

ve te

aming

Teach

ing beh

avior e

xpect

ations

Teach

ing socia

l skills

and em

otional co

mpetencie

s

Teach

ing frie

ndship sk

ills

Teach

ing child

ren to

expres

s emotions

Teach

ing pro

blem so

lving

Interve

ntions for c

hildren

with

persist

ent c

hallen

ging b

ehav

ior

Connecting w

ith fa

milies

Supporti

ng Fam

ily use

of the P

yramid M

odel prac

tices

Key Prac

tices Su

bscale

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Percentage of Indicators Observed for Key Practice ItemsInitial - 8 Teachers November - 8 Teachers April - 8 Teachers

Questions?

Embedded Instruction Observation System - Teacher Version

(Crowe, Snyder, Crow, Mullin, & Embedded Instruction Project, 2011)

Work reported was supported, in part, by Institute of Education Sciences grant (R324A070008) to the University of Florida. The information and opinions expressed are those of the authors, not the funding agency.

Manual

Scoring Sheet

Description of EIOS-T

Continuous event observational coding system

Used by teachers to record the frequency and accuracy of complete learning trials embedded within ongoing classroom activities, routines, and transitionsAdult or environmentally arranged antecedents to

elicit a targeted child behaviorWhether the target child behavior occurredConsequences and additional help (if appropriate)

provided following child behavior

Development of EIOS-T

Adapted from Embedded Instruction Observation System (EIOS) - Research Version 2.0 (Snyder et al., 2010)

EIOS was primary dependent measure in a Goal 2 study funded by IES*

Adapted from the the Teacher Performance Rate and Accuracy Scale (Ross, Singer-Dudek, & Greer, 2005) Designed to quantify interlocking three-term

contingencies or “learn units”*Institute of Education Sciences Project No. R324A070008: Impact of Professional Development on Preschool Teachers’ Use of Embedded Instruction Practices

Interlocking Three-Term Contingencies

Three-Term Contingencies for

Teacher

Instructional Components

Three-Term Contingencies for

Child

Teacher Antecedent 1 Child [reaches for ball]

Teacher Behavior 1 Teacher: “Say ball.” Child Antecedent

Teacher Consequence 1/Teacher Antecedent 2

Child: “Ball.” Child Behavior

Teacher Behavior 2 Teacher: [gives child a balI]

Child Consequence

Teacher Consequence 2 Child [plays with ball]

Note. Table adapted from Ross et al., 2005

From: Barton, E., Bishop, C., & Snyder, P. (in press). High quality instruction through complete learning trials: Blending intentional teaching with embedded instruction. Young Exceptional Children Monograph.

Targeted Child Behavior

Priority Learning Targets

Universal

Individualized

IEP Content

General Curriculum Commercial

GeneralCurriculum

LocallyDeveloped

Targeted Curricula(e.g., early literacy, social emotional)

Early Learning Foundations

EIOS-R

Focused on documenting Occurrence of learning

trials Categories of

antecedents, consequences and error corrections Adult-delivered Peer-delivered Environmentally

arranged Procedural accuracy of

learning trial components

EIOS-T

Focused on documenting Occurrence of learning

trials Category of

antecedent delivered Teacher-delivered Environmentally

arranged

Presence or absence of complete learning trial components

EIOS-T Development Simplified and reduced EIOS-R codes Developed coding manual and scoring sheet Developed and piloted training

Piloted with preschool teacher who participated in Goal 2 study

Revised coding manual based on teacher feedback

Piloted with three preschool teachers in a multiple baseline across teachers single-subject experimental study (Bishop, Snyder, & Crow, 2014)

EIOS-T Codes

Following EIOS-T Coding

Pilot Data Using EIOS-T Multiple baseline across three teachers single-

subject experimental study (Bishop et al., 2014) When provided feedback about their coding accuracy,

2 of 3 teachers were able to record accurately the occurrence of learning trials

When provided with feedback about their coding accuracy, all 3 teachers were able to record accurately the fidelity with which they implemented complete learning trial components

When teachers began to accurately record the occurrence and accuracy with which they delivered CLT components, the fidelity with which they implemented CLTs increased

Questions?

How are these measures similar?

Could these measures be used in an integrated fashion? How would that integration occur?

What is the role of measures such as these in terms of supporting inclusion within the broader early childhood quality movement?

Discussion

THANK YOU

Data Slides to Show if Needed


ICP Sample: Lead Teacher Characteristics*

Head

StartChild Care

Public

Pre-KDev Day Total

Mean Mean Mean Mean Mean

Course hours in special education

9.25 1.18 1.5016.0

87.52

Number of years of teaching child(ren) with a disability

10.77

5.95 4.40 5.48 6.91

Number of years of teaching in EC

13.62

12.85

5.60 8.1011.1

2* Based on teacher report


ICP Sample: Classroom and Child Characteristics

Head Start

Child Care

PublicPre-K

Dev. Day

Total

Mean Mean Mean Mean Mean

Number of children in classroom

17.15 16.55 15.00 16.38 16.51

Number of children with IEP in classroom

2.62 2.15 2.40 4.69 2.94

Age of youngest child (Yrs) 3.54 3.20 4.00 3.69 3.49

Age of oldest child (Yrs) 4.54 4.30 4.60 4.62 4.47

Number of adults in classroom 2.38 1.70 2.20 3.46 2.37

Children with IEP/adults ratio 1.13 1.22 1.07 1.43 1.23

All children/adults ratio 7.66 10.91 7.10 5.02 8.21

ECERS-R score 4.95 4.58 5.14 5.31 4.92

ICP score 4.64 3.67 4.76 5.12 4.39

Speech and language (38%) Developmental delay (37%) Autism (12%) Other health impairment (5%) Sensory impairment (4%) Multiple disabilities (1%) Orthopedic impairment (1%) Don’t know (2%) * Based on teacher report

ICP Sample: Children’s Primary Diagnoses*

88% of classrooms had at least one child with a moderate or severe level of disability in at least one area. *

* Based on teacher responses using an adaptation of the ABILITIES Index (Simeonsson & Bailey, 1991).


ICP Sample: Children with Disabilities

Head Start

Child Care

Public

Pre-K

Dev.

Day TotalChildren receiving majority of services in classroom

17.65%

37.21%

8.33%67.21

%57%


ICP Sample: Percent of Children Receiving the Majority of Specialized Services in the Classroom

In North Carolina, the percentage of children receiving the majority of specialized services in the classroom is 50.97%*

Nationally, the percentage of children receiving the majority of specialized services in the classroom is 41.67%*

*OSEP Report to Congress, 2011

EIOS-T Teacher Coding Accuracy

Average Percent Occurrence Agreement

No Training EIOS-T Training

EIOS-T Feedback

Natalie 12 (0 – 20) 62 (38 – 75) 90 (75 – 100)

Rhonda 31 (9 – 50) 36 (9 – 50) 47 (18 – 76)

Brenda 33 (0 – 66) 41 (20 – 63) 90 (78 – 100)

Average Percent Component Agreement

No Training EIOS-T Training

EIOS-T Feedback

Natalie N/A 66 (40 – 83) 88 (80 – 100)

Rhonda N/A 66 (33-83) 83 (69-100)

Brenda N/A 67 (39-89) 82 (61-83)

Teacher Implementation of Complete Learning Trials

Perc

enta

ge o

f Pro

ced

ura

lly C

orr

ect

Learn

ing T

rials

Presentation at National Early Childhood Inclusion Institute Chapel Hill, NC May 21, 2014 Patricia...

Documents

Transcript of Presentation at National Early Childhood Inclusion Institute Chapel Hill, NC May 21, 2014 Patricia...