What Works

Clearinghouse:

Early Childhood EducationElizabeth W. Cavadel

Mathematica Policy Research

Today’s Presentation Early childhood development and child care

Understanding the What Works Clearinghouse (WWC)

Introduction to the WWC

Discussion of the backbone of the WWC

Examining products released by the WWC, including relevant examples

Identifying whether a study was reviewed, and its WWC rating

Searching for particular evidence from Intervention Reports

Group discussion

Early care and education

Approximately 60% of children ages 3 to 5 are in a “care

arrangement” during the day (Child Trends, 2012)

Estimated that 25 - 41% are in full-time, center-based,

care

38% of children under 3 spend at least 35 hours/week in

non-parental care (NCLS-B, 2002)

Research shows quality matters and yes, the quality of care

is highly variable, across both income and types of care

What quality means in early childhood:

Strong adult-child interactions: supporting social-emotional

development

Reciprocal caregiving

Increasing complexity

Rich environmental exploration: supporting cognitive development

Varied experiences and exposure

The ability to interact with, and act upon, the environment

Language: supporting language and literacy development

Amount and diversity of talk

Types of talk

Conversations

Domain specific knowledge: science, math, early literacy skills

What is the WWC?

Problem and Context

• Substantial amount of research on education interventions• Large variation in quality• No common standards• Synthesizing findings is not

simple• Studies summarized many ways

• How does that affect choices?• Often based on personal

experience and ideology • Infrequently on strong research

WWC Mission

To be a central and trusted source of scientific evidence for what works in education

Develops and implements standards for reviewing and synthesizing education research

Assesses the credibility of research evidence on the effectiveness of interventions (does not directly assess interventions)

Transparent, objective, and high quality

http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_info_what_06

1015.pdf

What is the WWC?

The WWC reviews evidence of

effectiveness of programs, policies, or

practices by using a consistent and

transparent set of standards. The WWC

doesn’t rank, evaluate, or endorse

interventions.

WWC: Influencing Research and Practice

9

Producers of Information

• Universities

• Research organizations

• Developers

Consumers of

Information

• State and federal govt.

• Education departments

• Superintendents

• School boards

• Principals and teachers

• Parents and the public

• Funders and

consultants

Research

Standards Effectiveness Information

Evidence-Based

Decisions

Improved Student

Outcomes

Research

Topics

What Works

Clearinghouse• Develop standards

• Train reviewers

• Identify research

• Review research

• Summarize evidence

• Disseminate findings

The Review: Backbone of a WWC

Product

Review of publication to determine whether it supports causal

inferences

Three sets of design standards

Group design standards

Pilot single-case design standards

Pilot regression-discontinuity design standards

Four ratings

Meets WWC design standards without reservations

Meets WWC design standards with reservations

Does not meet WWC design standards

Not eligible for review

Review Protocols: Sets Parameters for Review

http://ies.ed.gov/

ncee/wwc/Publica

tions_Reviews.aspx

?f=All%20Publicatio

n%20and%20Produc

t%20Types,5;#pubs

earch

Are the groups formed randomly?

Is sample attrition high or low?

Were groups similar before the intervention

began?

Are there confounding factors or concerns with

outcomes?

http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_info_rates_061015.pdf

Key questions for a group design review

http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_info_reporting_061

015.pdf

Improvement Index

Effectiveness Rating

Extent of Evidence

What does the WWC report?

WWC Products

WWC Website: All Resources and

Products

http://whatworks.ed.gov

http://ies.ed.gov/ncee/w

wc/

Intervention Reports

An intervention report is a summary of findings of

the most reliable research on a given curriculum,

program, practice, or policy in education.

http://ies.ed.gov/ncee/wwc/pdf/intervention_reports/wwc_fasttrack_100714.p

df

http://ies.ed.gov/ncee/wwc/pdf/intervention_reports/wwc_socialskills_020513.

pdf

http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_info_making_061

015.pdf

1. Define scope of review

2. Conduct a comprehensive

search

3. Screen and review studies

4. Combine findings

5. Summarize the review

Steps in Writing an Intervention Report

Searching for an intervention report

http://ies.ed.gov/ncee/wwc/

Publications_Reviews.aspx?f=A

ll+Publication+and+Product+Ty

pes%2c5%3b

Example Intervention Report

What can you learn from an

intervention report?

http://ies.ed.gov/ncee/wwc/interventionreport.aspx?sid

=578

Parts of an Intervention Report

Program description

Research

Effectiveness

http://ies.ed.gov/ncee/wwc/pdf/intervention_reports/wwc_careeracademies_09

2215.pdf

Inside an Intervention Report

Program Information

Research Summary

Effectiveness Summary

References

Appendices

http://ies.ed.gov/ncee/wwc/pdf/intervention_reports/wwc_careeracademies_09

2215.pdf

Single Study Reviews

Assess one study against WWC standards

Not a systematic review on that particular intervention

First stage for some single study reviews is a “Quick Review”

Email blast giving sense of whether study will meet standards, ideally one week after media attention

What can you learn from a quick

review?

http://ies.ed.gov/ncee/wwc/quickreview.aspx?sid=239

What can you learn from a single

study review?

http://ies.ed.gov/ncee/wwc/SingleStudyReview.aspx?sid=236

http://ies.ed.gov/ncee/wwc/pdf/single_study_reviews/wwc_fitkids_082515.pd

f

WWC Practice Guides

A practice guide is a set of recommendations for educators to address challenges in their classrooms and schools. The level of research evidence supporting each recommendation is provided.

Expert panel to develop recommendations

Conduct a comprehensive literature search

Review each study against WWC standards

Summarize the review

http://ies.ed.gov/ncee/wwc/Publications_Reviews.aspx?f=All%20Publication%20and%20Prod

uct%20Types,3;#pubsearch

What is in a practice guide?

http://ies.ed.gov/ncee/wwc/pdf/practice_guides/wwc_algebra_040715.p

df

http://ies.ed.gov/ncee/wwc/pdf/practice_guides/early_math_pg_111313

.pdf

Recommendations

How-to steps

Lots of Examples

What can an example do for you?

WWC Videos

http://ies.ed.gov/ncee/wwc/webinar.aspx

If you are a researcher,

http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_gsa_v1.pdf

Has a study been reviewed by the WWC?

http://ies.ed.gov/ncee/wwc/ReviewedStudies.as

px

Find What Works

http://ies.ed.gov/ncee/wwc/findwhatworks.aspx

Contact InformationThe What Works Clearinghouse

http://whatworks.ed.gov

On Facebook at

www.facebook.com/whatworksclearinghouse

On Twitter at

www.twitter.com/WhatWorksED

And a widget you can add to your own pages

http://ies.ed.gov/ncee/wwc/Widget.aspx

Elizabeth Cavadel

ecavadel@mathematica-mpr.com

M. C. Bradley

cbradley@mathematica-mpr.com