Post on 06-Apr-2018
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
1/28
The Implementation and Efectiveneo Supplemental Educational Service
A Review and Recommendations or Program Improvement
Carolyn J. Heinrich University of Texas, Austin
Patricia Burch University of Southern California
March 2012
www.americanprogress.org www.aei.o
American Enterprise Institutefor Public Policy Research
Tightening Up Title I
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
2/28
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
3/28
1 Introduction and summary
4 Participation in SES and its impacts
6 Evidence on SES eects
10 Whats happening during an hour o SES?
14 Recommendations or policy and program change in SE
17 Conclusion
18 Reerences
21 Endnotes
22 About the authors and acknowledgements
Contents
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
4/28
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
5/28
Introduction and summary | American Enterprise Instit
Introduction and summary
Te 1965 Elemenary and Secondary Educaion Ac, or ESEA, creaed ile I ed-
eral unding o help provide all sudens wih an equal opporuniy o receive he
highes-qualiy educaion possible. Providing uoring or sruggling sudens is
one o many possible uses o ile I unds. Oher uses include eacher proessional
developmen, compuer labs, insrucional maerials, eacher assisans, and more.
ESEA was renamed he No Child Le Behind Ac, or NCLB, when i was reau-
horized in 2002. NCLB aimed o close he achievemen gap in public educaion.I requires public schools ha have no made adequae yearly progress on es
scores or a leas wo consecuive years o oer parens o children in low-income
amilies he opporuniy o receive exra academic assisance, oherwise known
as supplemenal educaional services, or SES. SES consiss primarily o uoring
oered ouside he regular school-day hours. Consisen wih he inen o he law
o promoe accounabiliy, exibiliy, and choice, SES is implemened a he local
level and draws largely on he privae secor o oer eligible sudens a range o
choices or ree uoring ouside o regular school hours.
No new ederal monies were allocaed o suppor he delivery or managemen
o SES. Te law lays ou crieria and guidelines or sae and local educaional
agencies in approving SES providers, arranging or heir services, and managing
conracs and nancial sysems. School disrics wih eligible schools are obligaed
o se aside 20 percen o heir ile I unding or SES and o measure provider
eeciveness in increasing suden achievemen.1
In arranging or SES, sae and local educaional agencies are able o draw on a
airly well-esablished marke o aer-school uoring programs. As SES expanded
uoring opporuniies or low-income sudens a subsanial number o diverseorganizaions enered he marke o compee or available SES unds. Tey
adverise widely varying hourly raes, uor qualicaions, uoring session lengh,
insrucional sraegies, and curriculums.2
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
6/28
2 Center or American Progress | The Implementation and Eectiveness o Supplemental Educational Services
Te ux in he SES vendor marke is considerable. Many smaller organizaions
ener and leave aer atracing ew sudens, while ohers have rapidly expanded
heir share o sudens served. Some school disrics have also operaed heir own
SES programs, hough as his is condiional on he disric making adequae yearly
progress, disric roles as providers also come and go. Te subsanial year-o-year
ucuaions complicae sae and local educaional agency eors o comply wihNCLB requiremens in ideniying organizaions ha provide services consisen
wih sae and local insrucional programs and wihdrawing approval rom pro-
viders ha ail o increase suden academic achievemen or wo years.3
In heory, parens and sudens should be holding SES vendors accounable
hrough heir choices o providers. Tey osensibly use inormaion disribued
by school disrics and SES providers o ideniy he bes provider o mee he
childrens needs. Sudens who become aware o heir eligibiliy may choose
o regiser or SES wih a specic SES provider, and SES providers invoice he
school disric or he number o hours SES sudens atend, up o a maximumper-suden dollar allocaion.
Te more eecive providers would increase heir share o sudens atending
SES over ime i he program worked as he law inended. Te service agreemen
beween a disric and is SES providers is, eecively, a cos-reimbursemen con-
rac, however, wih no perormance coningencies. In addiion, only sae educa-
ional agencies, no disrics, have auhoriy o approve SES providers and esablish
program crieria, such as an accepable suden/uor raio or providers o mee.
Te ac ha SES akes place ouside o he regular school classroom and ha
insrucional pracices are known o vary signicanlyno only beween pro-
viders, bu also wihin he same provider depending on he seting and specic
uorurher challenges sae and local agency eors o acquire knowledge o
SES conen and eeciveness.
Paricia Burch and Annalee Good poin ou ha he eaures o SES ha are key
o is eecivenessaciviies and resources used in insrucion, he naure o
ineracions beween sudens and providers, and insiuional and srucural
elemens ha inuence uoring pracicesare among he leas visible o saesand school disrics. 4 Ta said, aer-school sudy and uoring programs have
long been in operaion, including ederally unded programs, and here is a large
body o research on heir implemenaion and eecs, include sudies specically
ocused on SES.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
7/28
Introduction and summary | American Enterprise Instit
In his paper, we review sudies on he eecs o SES on suden achievemen and
updae he evidence on wha makes SES eecive or why i ails. We do his by
drawing on expanding disric evaluaion eors and he published lieraure in
his area as well as our own recen and ongoing mulisie, mulimehod sudies o
he implemenaion and eeciveness o SES.
In paricular, we address hese key quesions:
Who atends SES and or how many hours?Wha are he esimaed eecs o SES (rom our sudy and ohers), and how do
hey compare o hose o alernaive inervenions?Wha is happening in an invoiced hour o SES?Wha policy changes or levers migh improve SES?
We conclude wih recommendaions or program and policy change o make SES
more eecive in ligh o he expeced reauhorizaion o ESEA and ile I his year.
Our basic recommendaions include:
Sudens paricipaing in SES need o ge more hours o higher-qualiy and
appropriaely diereniaed insrucion or SES o be eecive. Saes and school disrics need o beter monior and conrol service qualiy
and delivery and ake a closer look a wha online providers are doing in an hour
o SES insrucion. SES resources should be direced primarily o sudens in lower grades, sudens
who are English language learners or who have disabiliies, and hose who are
mos severely underperorming in school. School disrics, SES providers and heir uors, and regular school-day eachers
and parens need o beter coordinae heir eors o increase he success o SES
in raising suden achievemen and closing achievemen gaps.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
8/28
4 Center or American Progress | The Implementation and Eectiveness o Supplemental Educational Services
Participation in SES and its impacts
Since SES began school disrics have been under pressure o comply wih ederal
requiremens o make SES available o as many eligible sudens as unding allows
and o assess vendor eeciveness in increasing he achievemen o paricipaing
sudens. Some school disric accounabiliy and evaluaion unis atemp o mea-
sure program eeciveness and in some cases SES provider ecacy. Boh disric
sa and researchers , however, ace numerous challenges o properly evaluaing
suden- and vendor-level SES eecs.
One o he bigges challenges is ha who ges uored in SES programs and or
how long is inuenced by a variey o acors, including suden and amily char-
acerisics and program adminisraion.
Paricipaion in SES is volunary among sudens eligible or SES. NCLB requires
school disrics o use he same daa o deermine eligibiliy or SES ha hey use
or making wihin-disric ile I allocaionshisorically hey use inormaion
on ree school lunch eligibiliyand school disrics are required o noiy ami-
lies o heir childrens eligibiliy and he availabiliy o approved SES providers.
Disrics have o esablish addiional crieria o deermine which eligible sudens
ge access o services i more sudens are expeced o sign up or SES han here
are unds available o serve hem (preerably beore regisraion opens). Bu no
all sudens decide o ollow hrough in atending wih a chosen provider even
i hey are eligible and given he opporuniy o regiser or SES. And some sop
atending beore heir oal SES dollar allocaion is expended.
Dierences exis beween SES-eligible sudens who regiser and atend SES and
hose who do no or a number o reasons. Sudens more requenly absen dur-
ing he regular school day, or example, are more likely o orego an aer-schoolopion. Analyses rom our mulisie sudy show ha he percen o days absen
in he prior school year is one o he mos consisen, negaive predicors o boh
regisraion or SES and SES atendance across he sudy disrics. We also ound
ha one o he mos consisen, posiive predicors o SES regisraion and aten-
dance is wheher he SES-eligible suden atended SES in he prior school year.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
9/28
Participation in SES and its impacts | American Enterprise Instit
As long as one can measure dierences such as hese in sudens who regiser/
atend and hose who do no, hey can be adjused in esimaing SES eecs. I
hose regisering and atending dier in ways ha are no observed or measured
and ha relae o suden achievemenin he level o encouragemen hey
receive rom regular school-day eachers, or examplegeneraing accurae esi-
maes o SES eecs may insead prove very dicul.
Our research also shows ha i is imporan o separaely model and accoun or
he muliple sages o SES selecionregisraion, atendance, and he number
o hours atendedbecause he inuence o suden characerisics diers across
hem. For insance, we nd across muliple sies and years ha whies, Hispanics,
and Asians are signicanlyless likely o regiser or or atend SES. Bu i hey
atend hey are signicanly more likely han Arican Americans o atend 40 or 60
hours. English language learners, or ELL sudens, are more likely o regiser and
atend more hours han non-ELL sudens.
Ideally we would like o know he added value o each addiional hour o SES
atended. Pu anoher way, wha number o hours o SES do sudens need o
atend beore we sar o see eecs on heir reading and mah achievemen?
o esimae his saisically, we need o have sucien numbers o sudens
atending a range o SES hours. In pracice, he number o hours sudens atend
SES is inuenced by acors such as he rae per hour charged by SES providers
and he dollars allocaed per suden by disrics or SES, as well as suden and
program characerisics.
One school disric in our mulisie sudy, or example, allocaed approximaely
$1,300 per suden or SES while over 70 percen o he paricipaing sudens
received SES rom a provider charging $75 or more per hour. Te maximum
hours o uoring a SES provider could oer a suden a his rae per hour and per
suden allocaion was abou 18 hours over he school year.
Across he disrics in our sudy, we observed he same peaks (or clumping) o
sudens in he number o hours hey atended SES, wih he larges peaks close
o 0 or 20 hours (depending on grade level), and smaller spikes around 40 and60 hours o SES atended. Te implicaion is ha we are able o more reliably
esimae SES eecs a hese common hresholds or levels o atendance where
here are sucien numbers o sudens observed (using rigorous mehods ha
eecively adjus or selecion).
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
10/28
6 Center or American Progress | The Implementation and Eectiveness o Supplemental Educational Services
Evidence on SES effects
Evaluaions o SES and/or provider-specic eecs are noably decien in many
saes and school disrics. Some saes and disrics rely only on inormaion
sel-repored by providers or rom relaively eeble daa-gahering eors such as
paren saisacion raes rom volunarily compleed surveys (wih very low and
selecive paricipaion).Oher disrics atemp o make he bes use o he daa
hey collec on suden SES atendance and provider invoices or operaing SES
o evaluae is eeciveness. Sill, only a handul o he larger disrics have he
in-house capabiliies o apply more rigorous approaches ha accoun or sudenselecion and oher esimaion problems.
In ac, some consisency in sudy ndings can be ound among he more
advanced disric evaluaion eors. Chicago Public Schools or CPS evaluaions
in 2003-04, 2004-05, 2006-07, and 2007-08 repored larger gains in reading and
mahemaics or sudens receiving a leas 40 hours o uoring and or sudens
in grades 4 hrough 8 who were notELL and who received at least30 hours o
SES uoring.5
Consisen wih oher CPS ndings, a Los Angeles Unied School Disric
sudy ound low SES paricipaion, and airly small program eecs even
among sudens wih he highes levels o SES atendance. Te eecs were
atribued primarily o improved perormance by elemenary sudens.6 In addi-
ion, sudies in Minneapolis and Milwaukee Public Schools, where average SES
hours atended are paricularly low, did no nd saisically signican, posiive
eecs o SES paricipaion.7
Mathew Springer and co-auhors poin ou ha very ew sudies rigorously adjus
or dierences in he characerisics o sudens who choose o paricipae in SES.8
Tey ideniy only our sudies besides heir own ha did so.9 I is imporan o
mach up sudens who look similar in heir characerisics in order o draw cred-
ible conclusions abou SES eeciveness based on comparisons o sudens who
paricipae in SES wih hose who orego he opporuniy o atend SES.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
11/28
Evidence on SES eects | American Enterprise Instit
Ta said, an imporan insigh rom he broader lieraure on aer-school uor-
ing programswhich is consisen wih ha o SES evaluaions o daeis
ha reaching some minimal hreshold o uoring hours appears o be criical o
producing measurable eecs on sudens achievemen (as measured primarily
by es scores).
Paricia Lauer and co-auhors conduced a synhesis o research on ou-o-school-
ime programsspecically in response o NCLB requiremens o oer SES
and ound ha eec sizes were larger or programs ha were more han 45 hours
in duraion, hough hey became smaller or hose longes in duraion. 10
In our own ongoing research esimaing he eecs o SES we nd 40 hours o
uoring o be a criical hreshold. Below 40 hours we do no ideni y any saisi-
cally signican eecs o SES on sudens mah and reading gains (as measured
by changes in es scores). In addiion, we nd eecs on boh mah and reading
achievemen or elemenary sudens who receive a leas 40 hours o SES buonly gains in mah or middle school sudens. Sudies by Mathew Springer and
co-auhors and on Zimmer and co-auhors likewise ound more consisen, posi-
ive eecs o SES on sudens mahversus readinggains in heir sudies o
SES in large, urban school disrics.11
Anoher common nding in sudies o SES is ha younger childrenspecically,
elemenary school sudensare more likely o atend SES (aer regisering or a
program) and o atend more hours han middle school or high school sudens.12
Unorunaely, some sudies, including one ha explored he eecs o SES across
muliple school disrics (Balimore, Chicago, Long Beach, Los Angeles, Palm
Beach, Philadelphia, San Diego, and Washingon, D.C.), did no esimae SES
eecs by grade level (or elemenary, middle, or high school).13
Sill, hese sudies did explore he poenial cumulative eecs o SES or sudens
who atended more han one year. Tey ound subsanially larger eecs on su-
den mah and reading es score gains associaed wih atending more han one
year o SES. Tis suggess ha more hours o SES are needed o produce larger
program eecs even i hey occur over more han one school year.
Tis evidence, along wih oher corroboraing ndings in he sudies discussed
here, poin o he essenial role o he number o SES hours received in generaing
eecs on sudens mah and reading gains.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
12/28
8 Center or American Progress | The Implementation and Eectiveness o Supplemental Educational Services
As indicaed above, an imporan challenge in geting more hours o SES o su-
dens is he hourly rae SES providers charge. Tis, in combinaion wih disric
(per-suden) maximum allocaions o SES dollars, limis he oal number o
hours o uoring sudens can receive. Sae and local educaional agencies do no
have auhoriy o proscribe or conrol he hourly raes charged by SES providers
(oher han disric-operaed ones), hough hey may speciy a range.
One would expec ha providers charging higher hourly raes would be delivering
higher-qualiy uoring services. In our curren and prior research, however, we
see litle correlaion beween provider characerisicssuch as suden-eacher
raios, oal hours oered, suden atendance, curriculum design, and ohers
and hourly raes charged oher han wheher a provider is online. 14
Furher, in our ongoing sudyincluding approximaely 300 providerswe nd
ha even when conrolling or suden selecion ino online vendors and he num-
ber o hours o SES sudens atend, sudens atending wih online SES providersare less likely o gain in mah and reading (relaive o ofine providers). Tis
nding is disconcering given ha in our sample, online vendors charged signi-
canly more or heir services ($24 per hour more) han oher providers.
We now look a eec sizes because hey are sandardized and can hereore be
compared across sudies. An eec size is he changemeasured in sandard
deviaionsin an average sudens oucome ha can be expeced i he suden
paricipaes in SES. A sandard deviaion ells you how dieren (or ar away) a
value is rom a given average (or a given sample). So a larger sandard deviaion
implies a bigger eec.
While here are some dierences in esimaed SES-eec sizesor mah and read-
ing and across sudies wih dieren samples, reamen measures, and approaches
o esimaing eecshere is also clearly some congruence in ndings.
Te average increase in mah es score gains o 0.09 sandard deviaions repored
by on Zimmer and co-auhors or sudens atending any SES is approximaely
he same ha we nd or middle school sudens who atend a leas 40 hours o
SES in our mulisie sudy.15
Te sudy by Mathew Springer and co-auhors alsonds increases in es score gains o abou 0.09 sandard deviaions in mahema-
ics (and 0.076 sandard deviaions in reading).16 In an alernaive specicaion
ha accouned or hose who regisered bu did no atend SES, however, hey
did no nd saisically signican eecs on reading or sudens atending SES
(comparable o our mulisie sudy resuls).
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
13/28
Evidence on SES eects | American Enterprise Instit
Our mulisie sudy separaely esimaes SES eecs or elemenary school
sudens. We nd comparably sized eecs or mah and reading (eec sizes o
approximaely 0.06 sandard deviaions, range: 0.054-0.076) ha are jus slighly
smaller han hose or middle school sudens.
Summaries o he eeciveness o similar educaional inervenions ha are ar-geed o elemenary and middle school sudens and inend o supplemen regular
school-day insrucion sugges ha hese eec sizes or SES are small. Based
on eec sizes o similarsupplemenal educaional inervenions compiled by
Carolyn Hill and co-auhors rom randomized sudies and a mea-analysis o oher
mea-analyses, an esimaed (one-year) eec size o 0.06 sandard deviaions is
one-ourh o one-h he size o mean eec size esimaes rom educaional iner-
venions argeed oward elemenary school sudens (0.23 o 0.33).17 Mean-eec
sizes rom randomized sudies o inervenions or middle school sudens, ranging
rom 0.27 o 0.51, imply he average SES-eec sizes range rom one-enh o one-
hird he size o eecs o oher similar educaional inervenions or hese sudens.
A he same ime, he ndings on SES eecs are very consisen wih hose
repored or inervenions more generally under NCLB ha rely on sandardized
achievemen ess o hold disrics and saes accounable or resuls. A Naional
Academy o Sciences sudy concluded ha school-level, es-based programs and
incenives under NCLB have eec sizes, on average, o approximaely 0.04 o 0.08
sandard deviaions, wih measurable eecs o dae ha are concenraed in ele-
menary grade mahemaics and are small in relaion o expeced improvemens. 18
Finally, o pu hese eec sizes ino perspecive wih regular school achievemens,
we esimae ha on average, not including administrative costs, school disrics
spend an amoun equal o abou 15 percen o heir average annual per-pupil coss
on SES (or hours o SES invoiced by providers). Using 0.06 sandard deviaions
as he average gain rom SES paricipaion, his is equal o abou 11 percen o 16
percen o he average annual gains in mah (0.54) and reading (0.38) by hird
and h graders on naionally-normed ess.
Wha his means is ha or elemenary sudensor whom sudies are mos likely
o nd saisically signican impacs o SESSES is, a bes, nearly as cos eec-ive in producing suden achievemen gains as spending on regular school-day
aciviies. In oher words, he gains rom paricipaion in SES relaive o average or
regular school-day gains in mah and reading are, a beswihou ull accouning
o cossapproximaely equal o or slighly less han hose achieved wih he same
racion o spending by schools disrics on regular school-day aciviies.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
14/28
10 Center or American Progress | The Implementation and Eectiveness o Supplemental Educational Services
Whats happening during an
hour of SES?
NCLBs inenion was o aciliae as ree a choice as possible or sudens and
parens selecing SES providers and program ypes. Under he law, school disrics
canno impose requiremens on uors, and he only auhoriy hey have in ermi-
naing a providers conrac occurs when he provider violaes disric policies or
oher such erms o a conrac.
Disric sa who are responsible or he adminisraion o SES conend ha heir
hands are ied in monioring providers. Tey also poin ou ha SES uors dono have o mee highly qualied sandards or have specic raining. Furher,
hey allege ha sae educaional agencies are lax in evaluaing providers, se-
ing minimum sandards or uoring qualiy or requesing essenial inormaion
on applicaions or assessing and monioring qualiy, or ollowing hrough on
disric complains abou provider incompeence or misconduc. And wih very
ew resources or program adminisraion, le alone monioring and evaluaion,
disric sa are sreched o nd ime o observe SES providers and beter under-
sand wha is aking place in an hour o SES or which disrics are invoiced.
A disinguishing eaure o our mulisie, mulimehod sudy o he implemena-
ion and eeciveness o SES is an in-deph qualiaive componen designed o
dene key elemens o SES program models and o ideniy how policy and imple-
menaion poenially mediae or inuence SES impacs. Wha do we see happen-
ing in pracice (a he classroom level) in an invoiced hour o SES? How does his
vary across dieren SES provider setings, disrics, ormas, and approaches o
uoring? And how does i relae o program eeciveness?
Because he law inenionally oers SES providers wide-ranging exibiliy in he
design o heir programs, assessing program qualiy or deliy o evidence-basedpracices is a somewha elusive ask. Sill, considering he law and inormaion
specied in provider conracs wih disrics, some direcives or conen ocus,
locaion, and he use o research-based pracices are clear.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
15/28
Whats happening during an hour o SES? | American Enterprise Institu
For one, SES providers should make reading and mahemaics he conen ocus
o insrucion, and insrucion has o be provided ouside o he regular school day.
Providers are no required o oer services o sudens wih disabiliies, or SWD,
or ELL sudens. Bu i providers oer hese services he law requires hem o be
adverised, and disrics are responsible or providing hese services i no provider
is able o or agrees o do so.
In addiion, he law saes ha SES uoring should be high-qualiy, research-
based, and specically designed o increase suden academic achievemen
[Secion 1116(e)(12)(C)]. In our sudy, we inerpre research-based pracices
as bes pracices idenied as making a measurable impac on suden achieve-
men by ou-o-school-ime or OS lieraure or conen area specialiss.
Best practices for out-of-school time tutoring and their use in SES
Litle research exiss on bes pracices specic o SES. Even so, prior research on
OS programs ells us ha high-qualiy programs are characerized by:
Consisen and susained insrucional ime Small grouping paterns (no larger han a suden-o-eacher raio o 10:1, bu
smaller is beter) Curriculum ha is conen rich, diereniaed o suden needs, and conneced
o sudens regular school-day learning Insrucion (or conen delivery) ha is varied (srucured and unsrucured,
independen and collecive), acive (no desk ime or workshees), ocused on
skills developmen, sequenced o achieve skill developmen objecives, and
explici in is argeing o specic skills Posiive relaionships beween uors, sudens, and peers eachers/uors wih boh conen and pedagogical knowledge and coninuous
suppor, as well as consrucive evaluaion, rom heir adminisraors 19
o ideniy hese bes pracices in SES sessions Burch and colleagues designed
a sandardized observaion insrumen o sysemaically collec inormaion on
eaching mehods and insrucional maerials in use and o ideniy he impaco dieren ormas, resources (curriculum maerials, sang), and insrucional
mehods on sudens observed levels o engagemen.20
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
16/28
12 Center or American Progress | The Implementation and Eectiveness o Supplemental Educational Services
o dae, Burch and colleagues have observed 99 uoring sessions across a range o
providers in ve, large urban school disrics in our saes. Te providers included
online, in-home, in-school, and communiy-based uors; or-pro, no-or-pro,
disric-provided, and aih-based organizaions; providers wih large marke
share (in erms o sudens served), wo or more years o SES provision, and
wih higher-han-average levels o suden atendance; and providers adverisingservices o SWD and ELL populaions.
In general, he model o uoring observed ended o ake he orm o radiional
academic learning environmens, wih sudens being uored in esed subjecs
mahemaics and readingand ypically insruced in a whole group orma wih
more han one suden and one ocal aciviy.
In oher words, raher han providing somehing innovaive, acive, and very di-
eren rom he regular school day, SES was based on radiional orms o eacher-
direced insrucion.21 For insance, uoring sessions o wo or more sudens weresrucured ino a single aciviy wih no op-ou aciviy in abou wo-hirds o
sessions observed. esearch on OS argues, wihou qualicaion, or diereni-
aed programming ha responds o sudens dieren learning syles or needs.
In addiion, sudens atending SES who migh learn bes via projec-based learning,
ars inegraion, or links o communiy-based aciviies encounered ew oppor-
uniies o his sor in our mulisie sudy observaions. Perhaps mos roubling,
however, very ew uors wih raining or experience in ELL or SWD insrucion
were presen during uoring, and wih very ew excepions, neiher curriculum nor
insrucion was ailored in any way o he unique needs o hese sudens.
On a more encouraging noe, uors were observed engaging wih sudens in
a predominanly posiive way across disrics and ormas. SES consisenly
occurred in small groups wih uoring sessions raing highly on indicaors o bes
pracices such as provide consrucive criicism; encourage paricipaion rom
disengaged sudens; and lisen acively and atenively o sudens.
Te observaion insrumen developed by Burch and colleagues also allowed
or assessmen o how consisen uors were in heir uoring pracices acrossan enire session and how much instructional ime sudens were receiving.
Irrespecive o he orma, sudens received less insrucional ime han wha was
adverised or invoiced by providers, hough he magniude o hese dierences
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
17/28
Whats happening during an hour o SES? | American Enterprise Institu
varied by orma. In more han hal o all observaions wih wo or more su-
densprimarily o-line, school-based setingssudens ha sared a session
were observed missing par o he session or leaving he uoring session alo-
geher, or sudens came in lae. We call his atendance ux.
We conrmed hrough inerviews wih uors and provider adminisraors haschool-based SES programs oen compee wih oher aer-school programs or
sudens ime, and classrooms wih muliple sudens required coordinaion and
se-up ha cu ino insrucional ime.
egardless, he quaniaive and qualiaive ndings o our mulisie sudy sugges
ha sudens are no geting enough hours o high-qualiy, diereniaed SES
insrucion o produce signican gains in heir learning. Tis is no a problem
ha will be resolved only by seting minimum hours sandards or SES providers
given ha invoiced hours do no equal insrucional ime.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
18/28
14 Center or American Progress | The Implementation and Eectiveness o Supplemental Educational Services
Recommendations for policy
and program change in SES
Public suppor exiss, including among many parens, or coninuing an iner-
venion in an ESEA reauhorizaion ha provides ree, exra academic assisance
o sruggling sudens. Bu he bes available evidence o dae suggess ha SES
has been minimally eecive. I produces only small eecs or a relaively small
racion o sudens, primarily elemenary aged, who ge a sucien number o
hours o uoring.
Sudens paricipaing in SES need o ge more hours oinstructional ime, andhey need o receive higher-qualiy and appropriaely diereniaed insrucion.
We oer he ollowing recommendaions or improving he eeciveness o SES
ha address he program design, he responsibiliies o sae and local educa-
ional agencies in implemenaion, and he ron-line role o SES providers in
delivering services.
States and districts need more levers of control over SES providers
Te curren division o responsibiliies beween sae and local educaional
agencies or monioring and accounabiliy o SES providers is no working. Te
disrics conrac wih and pay SES vendors o provide services, bu hey have no
auhoriy o se sandards or uor or service qualiy or minimum hours o SES
per enrolled suden or hourly raes charged.
esearch shows a srong relaionship beween he inensiy o SEShe num-
ber o hours atendedand is eeciveness, and SES provider hourly raes
direcly inuence he number o hours o SES sudens can atend. Te ac ha
we observed widely varying hourly raes across and wihin providerswih oneprovider charging less han hal he rae per hour in one disric han i charged in
anohersuggess ha saes and disrics need o exer more conrol over hourly
raes wih he accepable ranges hey speciy.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
19/28
Recommendations or policy and program change in SES | American Enterprise Institu
More generally, we recommend ha disrics be allowed and encouraged o
negoiae perormance-based conracs wih SES providers ha aciliae greaer
conrol over hourly raes and minimum SES hours provided. Tey also need more
say in uor qualicaions and curriculum (paricularly or serving ELL and SWD
populaions) and oher programmaic and nancial managemen acors.
Saes could similarly negoiae perormance-based conracs wih disric-
operaed providers. Some racion o ile I resources should be allocaed o
managing SES providers i perormance-based conracs are o be eecively
designed and managed.
Redirect SES resources toward students in lower grades and ELL and
students with disabilities
Anoher consisen nding in disric-based sudies and oher research is ha parici-paion and atendance in SES are signicanly lower or high school sudens han
or elemenary and middle school sudens. o dae, no sudies have shown SES o
be eecive or high school sudens, and hey show greaer atendance ux as well.
Alernaively, we ound ha ELL sudens were more likely o regiser or and atend
SES, and ye here was litle knowledge o or accommodaion or he special curricu-
lum and insrucional needs o ELL or sudens wih disabiliies in SES sessions.
Tese ndings sugges a policy change ha would redirec SES resources rom he
high school level o lower grades and oward new eors and programming o be-
er serve ELL and SWD enrollees.
Take a closer look at online providers
Online SES providers generally charge more or heir services. Tis is possibly
because hey ypically provide compuers or insrucion ha heir sudens are
subsequenly allowed o keep. No surprisingly, he share o sudens signing
up wih online providers has grown as sae and local educaional agencies havegenerally cracked down on providers use o incenives o atrac sudens o
heir programs and encourage atendance. Ofine providers are no longer able
o oer comparably atracive enrollmen incenives as he online providers who
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
20/28
16 Center or American Progress | The Implementation and Eectiveness o Supplemental Educational Services
can promise a compuer. Disric sa expressed concern ha some sudens
atend online SES only long enough o earn he compuer, and sudens ge
signicanly ewer hours o insrucion wih online providers due o heir higher
hourly raes.
We recommend ha sae and local educaional agencies underake a horoughassessmen o wha online providers oer in an SES sessionincluding qual-
iy and diereniaion in he curriculumand consider which crieria or key
elemens should bear on he hourly rae-seting o SES providers. Tis should be
done in combinaion wih heir assessmens o oher providers.
Inormaion gahered on provider perormance on hese crieria should be com-
municaed widely o sudens and parens. Again, disrics will need addiional
nancial and echnical suppor o underake hese evaluaions and eecively dis-
seminae he ndings o sudens and parens.
Focus on the students who need the most academic help
Large urban school disrics in paricular are increasingly sreched o provide
SES o all eligible sudens who sign up. Is likely ha in he absence o addiional
unding even ewer sudens will be served i policies are implemened o increase
he number o hours a given SES paricipan atends, as we recommend.
Some disrics have already had o esablish addiional eligibiliy crieria beyond
sricer low-income requiremens, and we urher recommend ha hese crieria
emphasize reaching hose sudens who are mos severely underperormingwih
adequae levels o uoring.
Better coordination among SES providers and other parties
Finally, i will likely ake greaer coordinaion and communicaion beween SES
providers and heir uors, school adminisraors, regular school-day eachers,
and parens o eecively serve hese sudens mos in need o exra academicassisance. Increased cooperaion, ransparency, and openness in program
managemen a boh sae and local levelswhich we saw aking roo in some
disricswill also be essenial o improving SES eeciveness.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
21/28
Conclusion | American Enterprise Institu
Conclusion
Our own research and oher sudies conrm a srong relaionship beween he
inensiy o SES, or he number o hours o SES atended, and is eeciveness.
Te inensiy o he inervenion is parly ied o he hourly rae charged or
SES, which deermines he number o possible hours o SES available o a given
suden. SES providers use o ime or insrucion and suden atendance ux
likewise aec he level and qualiy o SES received.
Sill, even among hose receiving a level o uoring necessary o generae eecs,he magniude o hese eecs is small as gauged by eec sizes or similar kinds o
ou-o-school-ime inervenions. Findings rom observaions o uoring sessions
sugges ha he minimal eecs may also sem rom criical omissions in he qual-
iy and characer o insrucional pracices. Among hese criical omissions is pro-
gramming ha addresses he special needs o ELL and SWD sudens, a problem
made more roubling by he ac ha hese sudens are signing up and atending
SES a higher raes han oher sudens.
We recommend program changes ha should ulimaely help sudens parici-
paing in SES ge more hours o higher-qualiy and appropriaely diereniaed
insrucion. Saes and school disrics need o ake seps o beter monior and
conrol service qualiy and delivery, and hey also need o invesigae wha is ak-
ing place in insrucion provided by online vendors, who operae he mos cosly
and dicul o monior programs.
In disrics aced wih increasingly sreched resources, SES should be direced
primarily o sudens in lower grades, ELL and SWD sudens, and hose who are
mos severely underperorming in school. o be successul in heir service o su-
dens wih he greaes need or academic assisance, school disrics, SES provid-ers and uors, regular school-day eachers, and parens need o beter coordinae
heir eors o increase he insrucional ime received by sudens and he qualiy
o SES insrucion delivered.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
22/28
18 Center or American Progress | The Implementation and Eectiveness o Supplemental Educational Services
References
Burch, P. and ohers. 2011. Preliminary Findings o aMulisie Sudy o he Implemenaion and Eecs oSupplemenal Educaional Services (SES). Workingpaper. Universiy o Souhern Caliornia and Universiyo Wisconsin.
Burch, P., A. Good, and M. Sewar. 2010. Mulisie Eval-uaion o Supplemenal Educaional Services Observa-ion Insrumen. Universiy o Souhern Caliornia andUniversiy o Wisconsin.
Burch, P. 2009. Hidden Markets: Te New Education Priva-tization. New York: ouledge.
Burch, P. and A. Good. 2009. Geting o he Core: Teole o Insrucional Seting in Federally Mandaeduoring. Working paper presened a he 2009
American Educaional esearch Associaion Coner-ence, San Diego, CA.
Burch, P., M. Seinberg, and J. Donovan. 2007. Supple-menal Educaional Serv ices and NCLB: Policy
Assumpions, Marke Pracices, Emerging Issues. Edu-
cational Evaluation and Policy Analysis 29 (2):115-133.
Chicago Public Schools. 2009. Te 2008 SupplemenalEducaional Services Program: Year 5 SummaiveEvaluaion. Chicago: Chicago Public Schools Oce oExended Learning Opporuniies, esearch, Evalua-ion and Accounabiliy.
Chicago Public Schools. 2008. Te 2007 SupplemenalEducaional Services Program: Year 4 SummaiveEvaluaion. Chicago: Chicago Public Schools Oce oExended Learning Opporuniies, esearch, Evalua-ion and Accounabiliy.
Chicago Public Schools, Oce o esearch, Evaluaion,and Accounabiliy. 2007. SES uoring Programs:
An Evaluaion o Year 3 in he Chicago Public Schools.Chicago: Chicago Public Schools.
Chicago Public Schools, Oce o esearch, Evaluaion,and Accounabiliy. 2005. SES uoring Programs:
An Evaluaion o he Second Year. Chicago: ChicagoPublic Schools.
Durlak, . and . Weissberg. 2007. Te impac o aer-school programs ha promoe personal and socialskills. Chicago: CASEL.
Governmen Accounabiliy Oce. 2006. No Child Le
Behind Ac: Educaion Acions Needed o ImproveLocal Implemenaion and Sae Evaluaion o Supple-menal Educaional Services. Washingon: GAOepor 06-758.
Heinrich, C.J., .H. Meyer, and G. Whiten. 2010. Supple-menal Educaion Services under No Child Le Behind:
Who Signs Up, and Wha Do Tey Gain?EducationalEvaluation and Policy Analysis 32 (June): 273-298.
Heisad, D. 2005. Analysis o 2005 supplemenal educa-ion services in Minneapolis public schools: An applica-ion o mached sample saisical design.Minneapolis:
Minneapolis Public Schools.
Hill, C.J., H.S. Bloom, A. . Black, and M.W. Lipsey. 2008.Empirical Benchmarks or Inerpreing Eec Sizes inesearch. Child Development Perspectives 2 (3): 172-177.
Lauer, P.A. and ohers. 2006. Ou-o-School-ime Pro-grams: A Mea-Analysis o Eecs or A-isk Sudens.Review of Educational Research 76 (2): 275-313.
Litle, P., Wimer, C., and Weiss, H. 2008. Programs inhe 21s Cenury: Teir Poenial and Wha i akeso Achieve I. Cambridge: Harvard Family esearch
Projec.
Naional Academy o Sciences. 2011. Incenives andes-based Accounabiliy in Educaion.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
23/28
Reerences | American Enterprise Institu
ickles, J.H. and M.K. Barnhar. 2007. Te Impac oSupplemenal Educaional Services Paricipaion onSuden Achievemen: 2005-06. Los Angeles: Los
Angeles Unied School Disric Program Evaluaionand esearch Branch, Planning, Assessmen andesearch Division.
Springer, M.G., M.J. Pepper, and B. Ghosh-Dasidar. 2009.
Supplemenal Educaional Services and Suden esScore Gains: Evidence rom a Large, Urban SchoolDisric. Working paper, Vanderbil Universiy.
Vandell, D., E. eisner, and K. Pierce. 2007. Oucomeslinked o high-qualiy aerschool programs: Longiu-dinal ndings rom he sudy o promising pracices.Irvine: Universiy o Caliornia; Washingon: PolicySudies Associaes.
Zimmer, . and ohers. 2006. Evaluaion o wo Ou-o-School Program in Pitsburgh Public Schools: NoChild Le Behinds Supplemenal Educaional Servicesand Sae o Pennsylvanias Educaional AssisanceProgram. Working Paper, RND Corporaion.
Zimmer, . and ohers. 2007. Sae and Local Imple-menaion o he No Child Le Behind Ac: Volume
Iile I School Choice, Supplemenal EducaionalServices, and Suden Achievemen. Washingon: U.S.Deparmen o Educaion, Oce o Planning, Evalua-ion and Policy Developmen.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
24/28
20 Center or American Progress | The Implementation and Eectiveness o Supplemental Educational Services
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
25/28
Endnotes | American Enterprise Institu
Endnotes
1 In addition, 5 percent o this unding can be set aside or transporta-tion or students eligible or choice programs.
2 Patricia Burch, Matthew Steinberg, and Joseph Donovan, Supple-mental Educational Services and NCLB: Policy Assumptions, MarketPractices, Emerging Issues, Educational Evaluation and Policy
Analysis 29 (2) (2007):115-133; Patricia Burch, Hidden Markets: TheNew Education Privatization (New York: Routledge, 2009).
3 The Government Accountability Oce reported in 2006 thatthree-ourths o states experienced challenges in developing datasystems or tracking and analyzing SES inormation and assessingstudents academic progress, with no state producing a conclusiveassessment o SES providers eects on student academic outcomes.
4 Patricia Burch and Annalee Good, Getting to the Core: The Role oInstructional S etting in Federally Mandated Tutoring. Workingpaper (University o Southern Caliornia, 2009).
5 Chicago Public Schools, Oce o Extended Learning Opportunities,Research, Evaluation and Accountability, The 2008 S upplementalEducational Services Program: Year 5 Summative Evaluation (2009);Chicago Public Schools Oce o Extended Learning Opportunities,Research, Evaluation and Accountability, The 2007 S upplementalEducational Services Program: Year 4 Summative Evaluation (2008);Chicago Public Schools, Oce o Research, Evaluation and Ac-countability, SES Tutoring Programs: An Evaluation o Year 3 in theChicago Public Schools (2007); Chicago Public Schools, O ce oResearch, Evaluation, and Accountability, SES Tutoring Programs:An Evaluation o the Second Year (2005).
6 Jordan H. Rickles and Melissa K. Barnhart, The Impact o Supple-mental Educational Services Participation on Student Achievement:2005-06 (Los Angeles: Los Angeles Unifed School District Program
Evaluation and Research Branch, Planning, Assessment and Re-search Division, 2007).
7 David Heistad, Analysis o 2005 supplemental education servicesin Minneapolis public schools: An application o matched samplestatistical design (Minneapolis: Minneapolis Public Schools, 2005);Carolyn J. Heinrich, Robert H. Meyer, and Greg Whitten, Supple-mental Education Services under No Child Let Behind: Who SignsUp, and What Do They Gain? Educational Evaluation and Policy
Analysis 32 (2) (2010): 273-298.
8 Matthew G. Springer, Matthew J. Pepper, and Bonnie Ghosh-Dastidar,Supplemental Educational Services and Student Test Score Gains:
Evidence rom a Large, Urban School District. Working paper(Vanderbilt University, 2009).
9 The our other studies identifed by Springer, Pepper, and Ghosh-Dastidar are: Ron Zim mer and others, Evaluation o Two Out-o-School Program in Pittsburgh Public Schools: No Child Let Behind's
Supplemental Educational Services and State o Pennsylvania'sEducational Assistance Program. RAND Working Paper (RANDCorporation, 2006); Ron Zimmer and others, State and LocalImplementation o the No Child Let Behind Act: Volume ITitleI School Choice, Supplemental Educational Services, and StudentAchievement (Washington: U.S. Department o Education, Oceo Planning, Evaluation and Policy Development, 2007); Heinrich,Meyer and Whitten, Supplemental Education Services under NoChild Let Behind; Heistad, Analysis o 2005 supplemental educa-tion services in Minneapolis public schools.
10 Patricia A. Lauer and others, Out-o-School-Time Programs: AMeta-Analysis o Eects or At-Risk Students,Review of EducationalResearch 76 (2) (2006): 275-313.
11 Springer, Pepper and Ghosh-Dastidar, Supplemental EducationalServices and Student Test Score Gains; Zimmer and others, Evalua-tion o Two Out-o-School Program in Pittsburgh Public Schools.
12 Patricia Burch and others, Preliminary Findings o a Multisite Studyo the Implementation and Eects o Supplemental EducationalServices (SES). Working paper (University o Southern Caliornia andUniversity o Wisconsin, 2011); Springer, Pepper and Ghosh-Dastidar,
Supplemental Educational Services and Student Test Score Gains;Zimmer and others, State and Local Implementation o the NoChild Let Behind Act.
13 Zimmer and others, State and Local Implementation o the No ChildLet Behind Act; Springer, Pepper and Ghosh-Dastidar, Supple-mental Educational Services and Student Test Score Gains.
14 Burch and others, Preliminary Findings o a Multisite Study o theImplementation and Eects o Supplemental Educational Services(SES): Heinrich, Meyer and Whitten, Supplemental EducationServices under No Child Let Behind.
15 Zimmer and others, State and Local Implementation o the No ChildLet Behind Act.
16 Springer, Pepper and Ghosh-Dastidar, Supplemental EducationalServices and Student Test Score Gains.
17 Carolyn J. Hill and others, Empirical Benchmarks or InterpretingEect Sizes in Research, Child Development Perspectives2 (3) (2008):172-177.
18 National Academy o Sciences Report o the Committee on Incen-tives and Test-based Accountability, Incentives and Test-basedAccountability in Education, (2011).
19 Joseph A. Durlak and Roger P. Weissberg, The impact o ater-school programs that promote personal and social skills (Chicago:CASEL, 2007); Priscilla Little, Christopher Wimer, and Heather Weiss,
Programs in the 21st Century: Their Potential and What it takes toAchieve It (Cambridge: Harvard Family Research Project, 2008) ;Debra Lowe Vandell, Elizabeth Reisner, and Kim M. Pierce, Out-comes linked to high-quality aterschool programs: Longitudinalfndings rom the study o promising practices. (Irvine: University oCaliornia and Washington: Policy Studies Associates, 2007).
20 Patricia Burch, Annalee Good, and Mary Stewart, Multisite Evaluationo Supplemental Educational Services Observation Instrument(University o Southern Caliornia and University o Wisconsin-Madison, 2010).
21 Homework help and test preparation activities were present in somesessions but were not the dominant orm o academic support.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
26/28
22 Center or American Progress | The Implementation and Eectiveness o Supplemental Educational Services
About the authors
Carolyn Heinrich is he Sid ichardson Proessor o Public Aairs and he direc-
or o he Cener or Healh and Social Policy a he Lyndon B. Johnson School
o Public Aairs a he Universiy o exas a Ausin. Her research ocuses on
social welare and educaion policy, public managemen and perormancemanagemen, and social-program evaluaion. She requenly works direcly in
her research wih governmens a all levels, including wih he ederal govern-
men on evaluaions o workorce developmen programs, wih saes on heir
social welare and child suppor programs, school disrics in he evaluaion o
supplemenal educaional services and oher educaional inervenions, and
governmens such as Brazil and Souh Arica on heir povery reducion and
human capial developmen programs. In 2004, Proessor Heinrich received he
David N. Kershaw Award or disinguished conribuions o he eld o public
policy analysis and managemen by a person under age 40. Prior o her appoin-
men a he Universiy o Wisconsin-Madison in 2003, Proessor Heinrich wasan assisan proessor a he Universiy o Norh Carolina-Chapel Hill and held
an academic research appoinmen a he Universiy o Chicago. She received
her docorae in public policy sudies rom he Universiy o Chicago.
Patricia Burch is an associae proessor a he ossier School o Educaion,
Universiy o Souhern Caliornia. Prior o his, she was an assisan proessor in
he Deparmen o Educaional Policy Sudies, Universiy o Wisconsin, Madison.
Burch received her B.A. in English rom Oberlin College in 1985, her M.A. in
educaion rom Harvard Universiy in 1991, her M.A. in sociology rom Sanord
Universiy in 1998, and her Ph.D. in educaion rom Sanord Universiy in 2000.
Over he pas decade, Dr. Burch has conduced major sudies and evaluaions o
K-12 educaion reorm such as SAGE (he Wisconsin class-size iniiaive), sys-
emic insrucional reorm in large urban school disrics (Seatle, Chicago, and
Philadelphia), proessional developmen reorms (Chicago, Philadelphia, and New
York Ciy), and school-linked services (Wes Virginia, New Mexico, Boson, and
San Diego). Her newes work examines he role o privae rms as inuences in he
design and implemenaion o K-12 educaion policy and how public educaion
is being ransormed below he radar by new orms o privaizaion. Her book
Hidden Markets: Te New Education Privatization was published by ouledgein 2009. She currenly serves as a co-principal invesigaor on a our-year sudy
iled, A Muli-Sie Evaluaion o he Implemenaion and Impac o Supplemenal
Educaional Services, unded by he Insiue o Educaion Sciences.
http://www.lafollette.wisc.edu/calendar-news/2004/2004-fall.html#heinrichhttp://www.lafollette.wisc.edu/calendar-news/2004/2004-fall.html#heinrich8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
27/28
About the authors and acknowledgements | American Enterprise Institu
Acknowledgments
Te auhors hank he major under o heir ongoing research on supplemenal
educaional services, he Insiue or Educaional Sciences, P/Award number:
305A090301, Educaion Policy, Finance, and Sysems esearch Program, Goal 3.
8/2/2019 The Implementation and Effectiveness of Supplemental Educational Services
28/28
About the Center or American Progress
The Center or American Progress is a nonpartisan re-
search and educational institute dedicated to promoting
a strong, just and ree America that ensures opportunityor all. We believe that Americans are bound together by
a common commitment to these values and we aspire
to ensure that our national policies reect these values.
We work to fnd progressive and pragmatic solutions
to signifcant domestic and international problems and
develop policy proposals that oster a government that
is o the people, by the people, and or the people.
Center or American Progress
1333 H Street, NW, 10th Floor
Washington, DC 20005
Tel: 202.682.1611 Fax: 202.682.1867
www.americanprogress.org
About the American Enterprise Institute
The American Enterprise Institute is a community o
scholars and supporters committed to expanding libe
increasing individual opportunity, and strengtheninree enterprise. AEI pursues these unchanging ideal
through independent thinking, open debate, reasone
argument, acts, and the highest standards o researc
and exposition. Without regard or politics or prevailin
ashion, we dedicate our work to a more prosperous
saer, and more democratic nation and world.
The American Enterprise Institute
1150 Seventeenth Street, N.W.
Washington, DC 20036
Tel: 202.862.5800 Fax: 202.862.7177
www.aei.org
American Enterprise Institutefor Public Policy Research