April 20, 2023 1

Regional Educational Laboratory - Southwest

The Effects of Teacher Professional Development on Student Achievement:

Finding from a Systematic Review of Evidence

Presented by:Kwang Suk Yoon (AIR)Teresa Duncan (AIR)

Sylvia Wen-Yu Lee (National Taiwan U)Kathy Shapley (Edvance Research, Inc.)

American Education Research AssociationNew York

March 27, 2008

April 20, 2023 2

Teacher Professional Development

One of key policy strategies for standards-based reform efforts

No Child Left Behind Act• Teacher Quality provisions• “High-quality” professional development

• Expectations: PD activities are to be regularly evaluated for their impact on teacher effectiveness and improved

student achievement Until recently, little systematic effort on vetting the

PD effectiveness

Background

April 20, 2023 3

Objective

AIR completed a fast-turnaround study, sponsored by the Regional Education Laboratory Southwest (RELSW), which was funded by IES

To conduct a systematic review of research-based evidence on the effects of teacher professional development (PD) on student achievement

April 20, 2023 4

Research-based Evidence on the effects of PD on student achievement

Challenges in demonstrating research-based evidence

Quality of professional development Workable theory of actions

• Logic model Sufficient implementation

Quality of empirical study Valid causal inferences Rigorous study design

April 20, 2023 5

Overview of methodology

Systematic review Using explicit & transparent methods Following a set of standards Being accountable, replicable, and updatable

Review protocol Aligned with What Works Clearinghouse (WWC)

evidence standards Study selection criteria Multi-stage, multi-coder review process

• Screening, coding & reconciliation Evidence Review Tool (ERT)

April 20, 2023 6

Study selection criteria

Topic: Inservice teacher professional development Population: K-12 students and their teachers Subject: reading/English/language arts, mathematics, or science Study design

Randomized controlled trial (RCT) Quasi-experimental design (QED) with matched comparison group

Student achievement outcome Measures and their psychometric properties Time: 1986-2006 Country: Australia, Canada, the United Kingdom, or the United

States

April 20, 2023 7

Overview of the review process

April 20, 2023 8

Literature search & prescreening

Literature searches Keyword searches on 7 major electronic databases Contacted key researchers Identified 1,343 citations potentially addressing the

effects of PD on student achievement

Prescreening Quickly scanning the abstracts for a few selection

criteria (e.g., empirical study?) Narrowed down to 132 relevant studies

April 20, 2023 9

Reasons for failing Stage-1 screening criteria Of 132 relevant studies, about two-thirds failed in the “study

design” criterion

Stage-1 coding

April 20, 2023 10

Stage-2 Coding

Determining the study quality ratings N = 27 relevant studies eligible for quality

ratings 9 met WWC evidence standards

5 met evidence standards with reservations 4 met evidence standards without reservations

18 failed to meet the evidence standards RCT – randomization, attrition, disruption, etc. QED – baseline equivalence, attrition,

disruption, etc.

April 20, 2023 11

Nine studies

Carpenter et al., 1989 (RCT) Cole, 1992 (RCT) Duffy et al., 1986 (RCT) Marek & Methven, 1991 (QED) McCutchen et al., 2002 (QED) McGill-Franzen et al., 1999 (RCT) Saxe et al., 2001 (QED) Sloan, 1993 (RCT) Tienken, 2003 (RCT with group

equivalence problems)

April 20, 2023 12

Documentation of studies

Effect size Characteristics of PD

• Form• Duration, contact hours• Content – Kennedy’s (1998) classification• Provider• Participants (volunteers?)• Information about the implementation of PD

Stage-3 coding

April 20, 2023 13

Results (1)

Paucity of rigorous studies Only 9 studies met evidence standards Mostly small-scale, underpowered efficacy trials

Distribution of 9 studies By study design

• 5 RCT• 4 QED

By content area• Concentrated in reading/English/language arts

By grade level• All focused on elementary school level

April 20, 2023 14

Results (2)

Overall effect size Average of 20 effect sizes (drawn from 9 studies) = .54

• Of 20 effect sizes, 12 were not statistically significant.

Effects by subject area Fairly consistent across three subject areas

Effects by form, duration, and intensity of PD Lack of variability in form: all workshop or summer institute Some evidence of the effect of intensive PD

Effects by content of PD No consistent pattern Failed to replicate Kennedy’s (1998) finding

April 20, 2023 15

Conclusion

There is some evidence of positive effects of PD on student achievement. Caveats The limited number of studies and the variability in

their professional development approaches preclude detailed conclusions about the effectiveness of particular professional development programs or about the effectiveness of professional development by such features as form and content.

The PD impact research is still at its developmental stage.

April 20, 2023 16

Suggestions (1)

Doing more well-conducted efficacy trials, replications, and effectiveness trials

Improving the design of PD impact studies Redressing common reasons for failing WWC

evidence standards• For example, lack of baseline equivalence in QED

Increasing statistical power to detect effects Aligning outcome measures with PD Examining mediation effects Addressing potential confounding effects of PD

with other important instructional factors (e.g., curriculum)

April 20, 2023 17

Suggestions (2)

Adequate documentation of PD & study PD: theory of action, implementation Sample and cluster, if any Group assignment Baseline equivalence Effect size (ES)

Use of structured abstract to facilitate research synthesis Structured abstract (Mosteller et al, 2004) Claim-based structured abstract (Kelly & Yin,

2007)

April 20, 2023 18

Contact info & links

Kwang Suk Yoon kyoon@air.org (202-403-5358)

Link to our RELSW report on the IES website http://ies.ed.gov/ncee/edlabs/projects/project.asp?

id=70 Link to our presentation material

http://www.pdal.net/reports.asp

Thank you!

April 20, 2023 19

Message

A “stainless steel” law of systematic reviews may be operating—namely, “the more rigorous the review, the less evidence there will be that the intervention is effective.” (Peter Rossi)

April 20, 2023 20

Logic Model