TAH TATE EGISLATURE ENERAL ESSION · OFFICE OF THE LEGISLATIVE FISCAL ANALYST ‐ 3 ‐ FEBRUARY...
14
U TAH S TATE L EGISLATURE 2017 G ENERAL S ESSION O FFICE OF THE L EGISLATIVE F ISCAL A NALYST ‐1‐ F EBRUARY 13, 2017, 9:54 AM LFA L EGISLATIVE FISCAL ANALYST W HAT D RIVES S TUDENT A CHIEVEMENT A CROSS S TATES ? HIGHER EDUCATION APPROPRIATIONS SUBCOMMITTEE STAFF:JESS ESPLIN AND JILL L. CURRY,PH.D. I SSUE B RIEF E XECUTIVE S UMMARY A strong education system can confer many benefits on a society. One way to measure the strength of an education system is through K‐12 student achievement. This state‐level analysis across time finds that educational attainment of adults has the largest and most consistent impact on K‐12 student achievement. The percent of children living in single parent households also produces relatively consistent effects. Other measures such as spending per student and student‐teacher ratio fail to reach statistical or substantive significance in this study. This study and other academic works consistently find that spending alone does not drive improved academic achievement. Further research should examine student achievement within Utah to determine which inputs, including areas of spending, produce the greatest impact on student achievement in an intra‐state analysis. T HE I MPORTANCE OF E DUCATION “Inclusive, good‐quality education is a foundation for dynamic and equitable societies.” ‐Desmond Tutu Education is the bedrock of society. Societies that are more educated tend to experience increased prosperity from a more productive workforce, 1 less crime, 2 better overall health, 3 and increased civic engagement. 4 The most direct path toward an educated society is a strong primary education system. The simplest way to assess the strength of a primary education system is by examining student achievement. Thus, student achievement can give us insight into the core of a society. B ACKGROUND AND L ITERATURE R EVIEW Despite the importance of student achievement, there is not unanimous agreement on what factors drive student success. Many studies produce conflicting results and anecdotal data muddy the water even further. Class size is one indicator that produces varying results. Using data from Tennessee’s Project STAR—a randomized experiment in the 1980s that assigned students to a small class (13 to 17 students), a regular‐sized class (22 to 25 students), or a regular‐sized class with a teacher’s aide—Krueger (1999) 5 finds that the students in small classes score higher on standardized tests than the students in both regular‐sized classes. In contrast, Hoxby’s (2000) 6 study of class size in Connecticut does not find a statistically significant effect of class size on student achievement. Teaching experience is another indicator with a varying effect on student achievement. Teacher experience significantly affected reading test scores in Rockoff (2004), 7 but Staiger and Rockoff (2010) 8 find that student achievement rises rapidly with teacher experience, but only for the first few years of a teacher’s career before the effect flattens out. Spending on education is one of the most debated predictors of student achievement. Currently, the most prominent support for increased spending is Jackson, Johnson, and Persico (2016) 9 who find that significant and sustained increases in public education funding (in the form of court‐mandated school finance reforms) lead to increased educational attainment and income in adulthood. In contrast, Hanushek is one of the most‐cited critics of the general impact of school spending on student outcomes (e.g. Hanushek 2003). 10 Case studies produce similarly conflicting results. Advocates for increased school spending often point to Massachusetts. A 1993 law transformed the Massachusetts school funding system to give more state money to districts with large populations of low‐income students. The Revere School District alone, comprised of mostly low‐ income students, saw a 20 percent increase in high school graduation rates after additional funding was spent on hiring more teachers and paying for training, new textbooks, reading coaches, and a technology team. Paul Reville,
Transcript of TAH TATE EGISLATURE ENERAL ESSION · OFFICE OF THE LEGISLATIVE FISCAL ANALYST ‐ 3 ‐ FEBRUARY...
UTAH STATE LEGISLATURE 2017 GENERAL SESSION
OFFICE OF THE LEGISLATIVE F ISCAL ANALYST ‐ 1 ‐ FEBRUARY 13, 2017, 9:54 AM
LFA LEGISLATIVE FISCAL ANALYST
WHAT DRIVES STUDENT ACHIEVEMENT ACROSS STATES?
HIGHER EDUCATION APPROPRIATIONS SUBCOMMITTEE
STAFF: JESS ESPLIN AND JILL L. CURRY, PH.D.I S SU E BR I E F
EXECUTIVE SUMMARY
Astrongeducationsystemcanconfermanybenefitsonasociety.OnewaytomeasurethestrengthofaneducationsystemisthroughK‐12studentachievement.Thisstate‐levelanalysisacrosstimefindsthateducationalattainmentofadultshasthelargestandmostconsistentimpactonK‐12studentachievement.Thepercentofchildrenlivinginsingleparenthouseholdsalsoproducesrelativelyconsistenteffects.Othermeasuressuchasspendingperstudentandstudent‐teacherratiofailtoreachstatisticalorsubstantivesignificanceinthisstudy.Thisstudyandotheracademicworksconsistentlyfindthatspendingalonedoesnotdriveimprovedacademicachievement.FurtherresearchshouldexaminestudentachievementwithinUtahtodeterminewhichinputs,includingareasofspending,producethegreatestimpactonstudentachievementinanintra‐stateanalysis.
THE IMPORTANCE OF EDUCATION
“Inclusive,good‐qualityeducationisafoundationfordynamicandequitablesocieties.”
‐DesmondTutu
Educationisthebedrockofsociety.Societiesthataremoreeducatedtendtoexperienceincreasedprosperityfromamoreproductiveworkforce,1lesscrime,2betteroverallhealth,3andincreasedcivicengagement.4Themostdirectpathtowardaneducatedsocietyisastrongprimaryeducationsystem.Thesimplestwaytoassessthestrengthofaprimaryeducationsystemisbyexaminingstudentachievement.Thus,studentachievementcangiveusinsightintothecoreofasociety.
BACKGROUND AND LITERATURE REVIEW
Despitetheimportanceofstudentachievement,thereisnotunanimousagreementonwhatfactorsdrivestudentsuccess.Manystudiesproduceconflictingresultsandanecdotaldatamuddythewaterevenfurther.Classsizeisoneindicatorthatproducesvaryingresults.UsingdatafromTennessee’sProjectSTAR—arandomizedexperimentinthe1980sthatassignedstudentstoasmallclass(13to17students),aregular‐sizedclass(22to25students),oraregular‐sizedclasswithateacher’saide—Krueger(1999)5findsthatthestudentsinsmallclassesscorehigheronstandardizedteststhanthestudentsinbothregular‐sizedclasses.Incontrast,Hoxby’s(2000)6studyofclasssizeinConnecticutdoesnotfindastatisticallysignificanteffectofclasssizeonstudentachievement.Teachingexperienceisanotherindicatorwithavaryingeffectonstudentachievement.TeacherexperiencesignificantlyaffectedreadingtestscoresinRockoff(2004),7butStaigerandRockoff(2010)8findthatstudentachievementrisesrapidlywithteacherexperience,butonlyforthefirstfewyearsofateacher’scareerbeforetheeffectflattensout.
Spendingoneducationisoneofthemostdebatedpredictorsofstudentachievement.Currently,themostprominentsupportforincreasedspendingisJackson,Johnson,andPersico(2016)9whofindthatsignificantandsustainedincreasesinpubliceducationfunding(intheformofcourt‐mandatedschoolfinancereforms)leadtoincreasededucationalattainmentandincomeinadulthood.Incontrast,Hanushekisoneofthemost‐citedcriticsofthegeneralimpactofschoolspendingonstudentoutcomes(e.g.Hanushek2003).10
Casestudiesproducesimilarlyconflictingresults.AdvocatesforincreasedschoolspendingoftenpointtoMassachusetts.A1993lawtransformedtheMassachusettsschoolfundingsystemtogivemorestatemoneytodistrictswithlargepopulationsoflow‐incomestudents.TheRevereSchoolDistrictalone,comprisedofmostlylow‐incomestudents,sawa20percentincreaseinhighschoolgraduationratesafteradditionalfundingwasspentonhiringmoreteachersandpayingfortraining,newtextbooks,readingcoaches,andatechnologyteam.PaulReville,
FEBRUARY 13, 2017, 9:54 AM ‐ 2 ‐ OFF ICE OF THE LEGISLATIVE F ISCAL ANALYST
WHA T D R I V E S S T U D E N T A C H I E V EM EN T A C R O S S S T A T E S ?
HarvardprofessorandformerMassachusettsstateeducationsecretary,said,“WhenyoulookatMassachusetts’overallperformancenationally,wehavegonefromthemiddleofthepacktothetopofthepack.”11
CriticsofincreasedschoolspendingoftenpointtothecaseofNewJersey.In1985,theNewJerseySupremeCourt’srulinginAbbottv.Burkeledthestatetoincreasespendinginitspoorestdistricts.TheCamdenSchoolDistrictspentroughly$23,000perstudentinthe2015‐2016schoolyear—about2.5timesthenationalaverage—but“there’snorealevidencethatthey’reclosingtheachievementgaporthatthey’redoingsignificantlybetter,”Hanushekargues.12
Utahspendsat,ornearthebottom,perstudentofallstatesinthecountry.TheU.S.CensusBureaurecentlyreleased2014datathatshowUtahspends$6,500perpupil,incomparisontoNewYork’s$20,610perpupil,thehighestamountnationally.Thenationalaverageforperpupilspendingis$11,009.13Intermsofstudentoutcomes,amongthestatesUtahfaresbetterthanitsperpupilspending.MostrecentlyUtah’sfourthgradersscoredsevenpointsabovethenationalaverageontheNAEP4Sciencetestandthestate’seighthgradersscored13pointsabovethenationalaverageontheNAEP8Sciencetest.Similarly,UtahfourthandeighthgradestudentsperformabovethenationalaverageonNAEPReadingandMathematicstests.
Withsomanyconflictingresultsregardingwhatimpactsstudentachievement,itisdifficulttosaywithconfidencewhatfactorswillproducegainsinstudentoutcomes.Interestingly,despitedifferencesintheirfindings,Jacksonetal.andHanushekdocometosomesimilarconclusions.Jacksonetal.note“increasedschoolfundingalonemaynotguaranteeimprovedoutcomes,butourfindingsindicatethatprovisionofadequatefundingmaybeanecessarycondition.Importantly,wefindthathowthemoneyisspentmaybeimportant”(214).14Likewise,Hanushek(2003)statesthathisstudy“doesnotmeanthatmoneyandresourcesnevermatter”(89).Hefurthernotesthat“nogooddescriptionofwhenandwherethesesituationsoccurisavailable,sothatbroadresourcepoliciessuchasthoselegislatedfromcentralgovernmentsmayhitsomegoodusesbutalsohitbadusesthatgenerallyleadtooffsettingoutcomes”(89).15Thus,despitedifferentresultsregardingtheimpactofschoolfunding,bothstudiesarecarefultonotethatforfundingtohaveanimpactitmustbetargeted.However,duetothemultitudeofstudiesproducingavarietyofresults,thequestionremainsabouthowfundingforpubliceducationshouldbetargeted.Togainfurtherinsightintothisquestion,weconductedananalysisofstudentachievementacrossthestates.
METHODOLOGY
INDICATORS
Thestate‐levelanalysisofstudentachievementisbasedonapaneldatasetwithobservationsforall50statesfrom2003to2013(theDistrictofColumbiaisexcluded).Thedatautilizedintheanalysisincludemeasuresofstudentoutcomesandachievement,spendingonpubliceducation,relevantdemographics,andvariouseducationalinputsforeachstateacrosstime.Thisstudyincludesthreemeasuresofstudentoutcomesand11indicatorspredictedtoimpactthem.ThedescriptivestatisticsandsourcesforeachmeasureareinTableA.1.intheappendix.Eachindicatorisdescribedbelow.
NAEP4Indexisastudentoutcomemeasure.ItistheaverageoftheaggregatedscoresforfourthgradestudentsontheNAEPmathematicsandreadingtests.16TheindexedscorewascreatedforeachstatebyaveragingtheNAEPmathematicsandNAEPreadingscores(bothrangefromzeroto500)forfourthgradestudents.Ifeitherthemathematicsorreadingscorewasmissingthentheavailablescorewasusedfortheindexratherthananaverage.
NAEP8Indexisalsoastudentoutcomemeasure.ItisanindexscorecreatedforeachstatebyaveragingtheaggregatedscoresforeighthgradestudentsontheNAEPmathematicsandreadingtests.17LiketheNAEP4index,boththeNAEPmathematicsandreadingscoresforeighthgradestudentsrangefromzeroto500.Ifeitherthemathematicsorreadingscorewasmissingthentheavailablescorewasusedfortheindexratherthananaverage.
Graduationrateisthethirdstudentoutcomemeasure.ThegraduationrateusedinthisstudyistheAveragedFreshmanGraduationRate(AFGR)ortheAdjustedCohortGraduationRate(ACGR)dependingontheyear.TheAFGRisusedfor2003through2010.TheAdjustedCohortGraduationRate(ACGR),whichstateswererequiredto
OFFICE OF THE LEGISLATIVE F ISCAL ANALYST ‐ 3 ‐ FEBRUARY 13, 2017, 9:54 AM
WHA T D R I V E S S T U D E N T A C H I E V EM EN T A C R O S S S T A T E S ?
usebeginningin2011,isusedfromthento2013.18TheACGRisthepercentageofstudentswhograduateinfouryearswitharegularhighschooldiploma.Thismeasureisanadjustedratebecausestudentsareaddedwhosubsequentlytransferintothecohortforthegraduatingclassandstudentsaresubtractedwhotransferoutofthecohort.19
Currentspendingperstudentisthetotalannualspendingonpubliceducation(local,state,andfederal)dividedbythetotalnumberofstudentsenrolledinthepubliceducationsystembystate.
Singleparenthouseholdisameasureofthepercentofchildrenlivinginsingleparenthouseholds.
Medianhouseholdincomeisthemedianhouseholdincomebystatein2015dollarsandmeasuredinthethousands.20
Educationalattainmentisthepercentofthepopulationage25andolderwithabachelor’sdegreeorhigher.
Student‐teacherratioisthenumberofstudentsdividedbythenumberofteachersinpubliceducation.
EnglishlanguagelearnersisameasureofthepercentofpublicschoolstudentsinprogramsforEnglishlanguagelearners.21
Highqualityteachersmeasuresthepercentofcoreacademicclassesinallschoolstaughtbyhighlyqualifiedteachers.HighlyqualifiedisdefinedandusedhereasitisinNoChildLeftBehind.Ingeneral,tobedeemedhighlyqualifiedteachersmusthaveabachelor’sdegree,fullstatecertificationorlicensure,anddemonstratecompetencyineachsubjecttheyteach.22
Teacherexperience23(<4years)isanindicatorofthepercentofteacherswithlessthanfouryears24ofteachingexperience.
Teacherexperience(≥15years)isthepercentofteacherswith15ormoreyears25ofteachingexperience.
CrimerateisthenumberofcrimesasreportedintheUniformCrimeReportsper100,000inthepopulation.
Volunteerhourspercapitameasurestheaveragenumberofvolunteerhourspercapitaperstate.
ANALYSIS
Asdiscussedabove,thisanalysisincludesthreemeasuresofstudentoutcomes:anindexofNAEP4Scores,anindexofNAEP8Scores,andtheGraduationRate.Asaninitialassessment,acomparisonwasdonebetweenallstatesandUtahforallthreeofthesemeasures.First,infigure1isaplotoftheNAEP4Indexscoresovertime.Anychartfeatureinbluerepresentsvaluesforall50stateswhereasthechartfeaturesinorangerepresentvaluesforthestateofUtah.Thebluelines(insidetheblueboxes)representthemedianNAEP4IndexScoresandtheblueX’srepresentthemeanNAEP4IndexScoresforthoseyears.TheorangelinesandorangeX’srepresenttheNAEP4IndexScoresforUtahforthoseyears.Thebluetopandbottomlines(beyondtheblueboxes)arethelocalmaximumandminimum,respectively,andthebluedotsareoutlyingdatapoints.26
InexaminingtheNAEP4IndexScores,thefiguredemonstratesthat,foreachyear,Utah’sscoreisequaltoorhigherthanthemeanscoreforallstatesexceptin2009whereUtahhasaNAEP4IndexScoreof229.76andthemeanscoreforallstatesis230.22(adifferenceof0.46points).Utah’sscoreisequaltoorhigherthanthemedianscoreforallstatesexceptin2007and2009.In2007,Utah’sscoreis230.33andthemedianscoreforallstatesis231.47(adifferenceof1.14points).In2009,themedianscoreforallstatesis231.83whichishigherthanUtah’sscorethatyearby2.07points.
FEBRUARY 13, 2017, 9:54 AM ‐ 4 ‐ OFF ICE OF THE LEGISLATIVE F ISCAL ANALYST
WHA T D R I V E S S T U D E N T A C H I E V EM EN T A C R O S S S T A T E S ?
Figure2providesasimilarfigurefortheNAEP8IndexScores.Inanalyzingthesescores,Utah’sscoreisequaltoorhigherthanthemeanscoreforallstatesexceptin2007.In2007,Utah’sscoreis271.66andthemedianscoreforallstatesis273.56(adifferenceof1.90points).
ThegraduationrateforallstatescomparedtoUtahwasalsoexaminedovertime.Figure3indicatesthatthegraduationrateinUtahisequaltoorhigherthanthemeanandmediangraduationrateforallstatesineveryyearanalyzedexcept2008and2011.In2008thegraduationrateforUtahwas74.3percentwhilethemeanandmediangraduationrateforallstateswere76.1and76.4percent,respectively.In2011,Utah’sgraduationratewas76percentwhilethemeangraduationrateforallstateswas78.9percentandthemediangraduationratewas80percent.
OFFICE OF THE LEGISLATIVE F ISCAL ANALYST ‐ 5 ‐ FEBRUARY 13, 2017, 9:54 AM
WHA T D R I V E S S T U D E N T A C H I E V EM EN T A C R O S S S T A T E S ?
Asmeasuredbythethreestudentoutcomeindicators,theperformanceofstudentsinUtahiseitherequaltoorexceedstheaverageperformanceacrossallstatesinmostyears.However,thefiguresabovedonotshedlightonwhatfactorssignificantlyimpactstudentsuccess.Toanalyzewhatisdrivingstudentachievement,astate‐levelanalysiswasconductedusingarandomeffectspanelregressionwiththepanelcomposedofstatesovertime.Therandomeffectsmodelisusedbecauseweexpectthevariationacrossstatestoberandom.Inotherwords,differencesacrossstatesmayimpactstudentoutcomes.
Alongwiththemeasuresofstudentachievement,wehave11indicatorstheorizedtoaffectstudentachievement.Thesefactors,whicharedefinedabove,includespendingonpubliceducation,relevantdemographics(percentofchildreninsingleparenthouseholds;medianhouseholdincome;educationalattainment;Englishlanguagelearners;crimerate;andvolunteerhourspercapita),andvariouseducationalinputs(student‐teacherratio;highqualityteachers;teacherexperience(<4years);andteacherexperience(≥15years))foreachstateacrosstime.Weexpectthatincreasesinthepercentofchildreninsingleparenthouseholds,student‐teacherratio,percentofEnglishlanguagelearners,percentofteacherswithfewerthanfouryearsofexperience,andcrimeratewillleadtodecreasedstudentachievement.Concurrently,weexpectincreasesincurrentspendingperstudent,medianhouseholdincome,educationalattainment,thepercentofhighqualityteachers,teacherswith15ormoreyearsofexperience,andvolunteerhourspercapitawillleadtoincreasesinstudentachievement.
RESULTS
Table1presentstheresultsofeachofthethreestudentachievementanalyses.Eachcolumnevaluatestheimpactoftheindicatorsononeofthethreestudentachievementmeasures.
4THGRADETESTSCORES
ThefirstcolumnassessestheimpactofourindicatorsonNAEP4IndexScores.Sixoftheindicatorsarenotstatisticallysignificantincludingcurrentspendingperstudent,medianhouseholdincome,student‐teacherratio,percenthighqualityteachers,teacherswithlessthanfouryearsofexperience,andcrimerate.Thefiveremainingindicatorsproducestatisticallysignificantresults.Thesemeasuresincludepercentofchildreninsingleparenthouseholds,educationalattainment,percentEnglishlanguagelearners,teacherswith15ormoreyearsofexperience,andvolunteerhourspercapita.
AsexpectedpercentofchildreninsingleparenthouseholdsandpercentEnglishlanguagelearnersleadtodecreasedNAEP4IndexScores.Also,asexpected,increasesineducationalattainmentleadtoincreasesinNAEP4IndexScores.
FEBRUARY 13, 2017, 9:54 AM ‐ 6 ‐ OFF ICE OF THE LEGISLATIVE F ISCAL ANALYST
WHA T D R I V E S S T U D E N T A C H I E V EM EN T A C R O S S S T A T E S ?
Teacherswith15ormoreyearsofexperienceandvolunteerhourspercapitawereexpectedtohaveapositiveimpactonstudentachievement,but,inthisanalysis,theyleadtolowerNAEP4IndexScoresalthoughthemagnitudeoftheirimpactissmall.
IntheanalysisofNAEP4IndexScores,thethreesignificantindicatorswiththegreatestimpactareeducationalattainment,percentofchildreninsingleparenthouseholds,andEnglishlanguagelearners.Educationalattainmentispositivewithavalueof0.659,meaningthatasthepopulationaged25andolderwithabachelor’sdegreeorhigher(hereafterreferredtoasthecollege‐educatedpopulation)increasesbyonepercent,theNAEP4IndexScoresincreaseby0.7pointsonaverageacrosstimeandbetweenstates.Inotherwords,asthecollege‐educatedpopulationincreasesby10percent,theNAEP4IndexScoreincreasesbyaboutsevenpointsonaverageacrosstimeandbetweenstateswhichisgreaterthanthestandarddeviationofaboutsixfortheNAEP4IndexScore.
TheeffectofeducationalattainmentonNAEP4IndexScoresisillustratedinfigure4whichcontainsthepredictedvaluesofNAEP4IndexScoresaseducationalattainmentincreasesfromabout16percentto40percent.Asthepercentofthecollege‐educatedpopulationincreasesfrom16to40,theNAEP4IndexScoresincreasefrom222to238.
(1)NAEP4IndexScores
(2)NAEP8IndexScores
(3)GraduationRate
<0.001 <0.001 <0.001(<0.001) (<0.001) (<0.001)‐0.244** ‐0.190** ‐0.125(0.063) (0.062) (0.085)<‐0.001 ‐0.0001** ‐0.0002**(<0.001) (<0.001) (<0.001)0.659** 0.747** 0.612**(0.097) (0.104) (0.135)‐0.227 ‐0.189 ‐0.211(0.121) (0.121) (0.125)‐0.265** ‐0.147 0.120(0.075) 0.078 (0.105)0.023 0.036 0.050(0.020) (0.019) (0.029)‐0.063 ‐0.110** ‐0.145**(0.037) (0.035) (0.050)‐0.058** ‐0.035** ‐0.0020.008 (0.008) (0.011)‐0.001 ‐0.001** ‐0.002**(<0.001) (<0.001) (<0.001)‐0.043* ‐0.011 0.009(0.021) (0.020) (0.023)231.740** 273.364** 81.232**5.087 5.079 6.695
N(observations) 300 300 550N(groups) 50 50 50
TABLE1AssessingStudentAchievementAcrosstheStates,2003‐2013
Volunteerhourspercapita
constant
Notes:**p<0.01;*p<0.05;two‐tailedtests.Coefficientsderivedusingrandomeffectspanelregression.Standarderrorsareinparentheses.
Student‐teacherratio
Currentspendingperstudent
%childreninsingleparenthouseholds
Medianhouseholdincome
Educationalattainment
Englishlanguagelearners
Highqualityteachers
Teacherexperience(<4years)
Teacherexperience(≥15years)
Crimerate
OFFICE OF THE LEGISLATIVE F ISCAL ANALYST ‐ 7 ‐ FEBRUARY 13, 2017, 9:54 AM
WHA T D R I V E S S T U D E N T A C H I E V EM EN T A C R O S S S T A T E S ?
Thepercentofchildreninsingleparenthouseholdsisalsosignificantwithavalueof‐0.244.Thus,asthepercentofchildrenlivinginsingleparenthouseholdsincreasesbyonepercent,theNAEP4IndexScoresdecreaseby0.24pointsonaverageacrosstimeandbetweenstates.Inotherwords,a10percentincreaseinthepercentofchildrenlivinginsingleparenthouseholdsleadstoadecreaseinNAEP4IndexScoresofabout2.5pointsonaverage.TheeffectofthisindicatoronNAEP4IndexScoresisillustratedinfigure5.Asthepercentofchildreninsingleparenthouseholdsincreasesfrom17to49,theNAEP4IndexScoresdecreasefrom233to225.
TheeffectofEnglishlanguagelearnersissimilartosingleparenthouseholds.ThevalueforEnglishlanguagelearnersis‐0.26soasthepercentofEnglishlanguagelearnersincreasesbyonepercent,theNAEP4IndexScoresdecreaseby0.26pointsonaverageacrosstimeandbetweenstates.Inotherwords,a10percentincreaseinthepercentofEnglishlanguagelearnersleadstoadecreaseinNAEP4IndexScoresofabout2.6pointsonaverage.
FEBRUARY 13, 2017, 9:54 AM ‐ 8 ‐ OFF ICE OF THE LEGISLATIVE F ISCAL ANALYST
WHA T D R I V E S S T U D E N T A C H I E V EM EN T A C R O S S S T A T E S ?
8THGRADETESTSCORES
ThesecondanalysisusesNAEP8IndexScoresandfindsthatfiveoftheindicatorsarenotstatisticallysignificant(currentspendingperstudent,student‐teacherratio,Englishlanguagelearners,percenthighqualityteachers,andvolunteerhourspercapita).Sixofthemeasuresproducestatisticallysignificantresults.Thesemeasuresincludepercentofchildreninsingleparenthouseholds,medianhouseholdincome,educationalattainment,teacherswithlessthanfouryearsofexperience,teacherswith15ormoreyearsofexperience,andcrimerate(seetable1).
Asexpectedpercentofchildreninsingleparenthouseholds,teacherswithlessthanfouryearsofexperience,andcrimerateleadtodecreasedNAEP8IndexScores.Also,asexpected,increasesineducationalattainmentleadtoanincreaseinNAEP8IndexScores.Teacherswith15ormoreyearsofexperienceandmedianhouseholdincomewereexpectedtohaveapositiveimpactonstudentachievement,but,inthisanalysis,theyleadtolowerNAEP8IndexScoresalthoughthemagnitudeoftheirimpactissmall.
Theeffectofeducationalattainmentispositivewithavalueof0.747,meaningthatasthecollege‐educatedpopulationincreasesbyonepercent,theNAEP8IndexScoreincreasesby0.75pointsonaverageacrosstimeandbetweenstates.Inotherwords,asthecollege‐educatedpopulationincreasesby10percent,theNAEP8IndexScoresincreasebyabout7.5pointsonaverageacrosstimeandbetweenstateswhichisgreaterthanthestandarddeviationofabout6.7fortheNAEP8IndexScore.TheimpactofeducationalattainmentonNAEP8IndexScoresisillustratedinfigure6whichshowsthatasthecollege‐educatedpopulationincreasesfrom16to40percent,theNAEP8IndexScoresincreasefrom264to282.
Thepercentofchildreninsingleparenthouseholdsisalsosignificantwithavalueof‐0.190.Thus,asthepercentofchildrenlivinginsingleparenthouseholdsincreasesbyonepercent,theNAEP8IndexScoredecreasesby0.19pointsonaverageacrosstimeandbetweenstates.Inotherwords,a10percentincreaseinthepercentofchildrenlivinginsingleparenthouseholdsleadstoadecreaseinNAEP8IndexScoresofabouttwopointsonaverage.TheeffectofthisindicatoronNAEP8IndexScoresisillustratedinfigure7.Asthepercentofchildreninsingleparenthouseholdsincreasesfrom17to49,theNAEP8IndexScoredecreasesfrom275to269.
Theeffectofteacherswithlessthanfouryearsofexperienceissimilartosingleparenthouseholds.Thevalueforinexperiencedteachersis‐0.11soasthepercentofteacherswithlessthanfouryearsofexperienceincreasesbyonepercent,theNAEP8IndexScoredecreasesby0.11pointsonaverageacrosstimeandbetweenstates.Inotherwords,a10percentincreaseinthepercentofinexperiencedteachersleadstoadecreaseinNAEP8IndexScoresof
OFFICE OF THE LEGISLATIVE F ISCAL ANALYST ‐ 9 ‐ FEBRUARY 13, 2017, 9:54 AM
WHA T D R I V E S S T U D E N T A C H I E V EM EN T A C R O S S S T A T E S ?
aboutonepointonaverage.
HIGHSCHOOLGRADUATION
Thethirdanalysisusesgraduationrateandfindsthatsevenoftheindicatorsarenotstatisticallysignificant(currentspendingperstudent,percentofchildreninsingleparenthouseholds,student‐teacherratio,Englishlanguagelearners,percenthighqualityteachers,teacherswith15ormoreyearsofexperience,andvolunteerhourspercapita).Fourofthemeasuresproducestatisticallysignificantresults.Thesemeasuresincludemedianhouseholdincome,educationalattainment,teacherswithlessthanfouryearsofexperience,andcrimerate(seetable1).
Asexpected,teacherswithlessthanfouryearsofexperienceandcrimerateleadtodecreasedgraduationrates.Also,asexpected,increasesineducationalattainmentleadtoanincreaseingraduationrates.Medianhouseholdincomewasexpectedtohaveapositiveimpactonstudentachievement,but,inthisanalysis,itleadstolowergraduationratesalthoughthemagnitudeofitsimpactissmallalongwiththeeffect‐sizeforcrimerate.
Theeffectofeducationalattainmentispositivewithavalueof0.612,meaningthatasthecollege‐educatedpopulationincreasesbyonepercent,thegraduationrateincreasesby0.61percentonaverageacrosstimeandbetweenstates.Inotherwords,asthecollege‐educatedpopulationincreasesby10percent,thegraduationrateincreasesbyabout6percentonaverage.Theimpactofeducationalattainmentonthegraduationrateshowninfigure8demonstratesthatasthecollege‐educatedpopulationincreasesfrom16to40percent,thegraduationrateincreasesfrom70to85percent.
Theeffectofteacherswithlessthanfouryearsofexperienceis‐0.15soasthepercentofteacherswithlessthanfouryearsofexperienceincreasesbyonepercent,thegraduationratedecreasesby0.15percentonaverageacrosstimeandbetweenstates.Inotherwords,a10percentincreaseinthepercentofinexperiencedteachersleadstoadecreaseinthegraduationrateofabout1.5percentonaverage.
FEBRUARY 13, 2017, 9:54 AM ‐ 10 ‐ OFF ICE OF THE LEGISLATIVE F ISCAL ANALYST
WHA T D R I V E S S T U D E N T A C H I E V EM EN T A C R O S S S T A T E S ?
DISCUSSION AND CONCLUSION
Afterconductingouranalysis,wefindthatsomemeasuresaffectstudentachievement.Acrossallthreeanalyses,educationalattainment—thepercentofcollege‐educatedadults—maintainsasignificantandsubstantiveimpactonstudentachievement.ThisfindingsuggeststhattheeducationalattainmentofadultsimpactsoutcomesforK‐12students.ThepercentofchildreninsingleparenthouseholdshasasignificantimpactonstudentachievementasmeasuredbytheNAEP4andNAEP8IndexScores.Itisnotasignificantpredictorofgraduationrate.
ThepercentofteacherswithlessthanfouryearsofexperienceproducessignificantresultsforNAEP8IndexScoresandthegraduationrate.ThisfindingsupportstheworkofStaigerandRockoff(2010)27whofindthatteacherexperiencemattersmostinthefirstfewyearsofteaching.Twomeasuresthatdonotproduceanystatisticallysignificantresultsarecurrentspendingperstudentandstudent‐teacherratio.Inthisstudy,theseindicatorsdonothaveasignificantimpactonstudentachievement.Theremainingmeasuressometimeshaveasignificantimpactonstudentachievementand,forsomeofthese,theirimpactissmallinmagnitude.
Oneobjectiveinconductingthisanalysiswastoilluminatehoweducationfundingshouldbetargeted.Thisstudysuggeststhatfundingtoincreasethecollege‐educatedpopulationcouldproducesignificantgainsinK‐12studentachievement.Theresultsalsoindicatethatincreasingsupportforchildrenlivinginsingleparenthouseholdsmightsignificantlyimpactstudentachievement.
Thisstudyislimitedinscopeandmagnitude.Wecouldnotincludemeasuresoflong‐termeducationaloutcomes,suchasstudents’totaleducationalattainmentandtheirincomeinadulthood.Thiswouldprovidemoreinformationonhowspendingaffectslong‐termoutcomesofstudentsuccess,beyondtestscoresandhighschoolgraduation.Weacknowledgetheinadequacyoftheteacherqualityandexperiencevariablesincludedinthismodel.Weusedthebestdataavailabletousforthisstudyandnotethattherearenolarge‐scaledataavailableonteacherpracticeswithintheclassroom,assuggestedinsomestudies28tobethemostinfluentialfactorsonstudentachievement.Futureresearchshouldtakethisneedintoaccountanddevelopmorereliableandaccuratemeasuresofteachereffectiveness.
Ourstate‐levelstudyhasotherlimitationsaswell.Ourresearchassessesoverallacademicproficiency,asmeasuredbyNAEPscoresandhighschoolgraduationrates,generalizingresultsfortheentirepublicschoolstudentpopulationintheU.S.Therefore,thisstudycannotmeasureachievementgapsamongdifferentpopulationsofpublicschool
OFFICE OF THE LEGISLATIVE F ISCAL ANALYST ‐ 11 ‐ FEBRUARY 13, 2017, 9:54 AM
WHA T D R I V E S S T U D E N T A C H I E V EM EN T A C R O S S S T A T E S ?
students.Theexaminationofdisparitiesinachievementamonggroupsofstudentsispartofabroaderexaminationineducationresearchofeducationalequality,butthisstudyisnotdesignedtoaddressit.
AdditionalresearchisneededtocomprehensivelyanswerthecompellingquestionthathascapturednationalattentionandattentioninUtah—howdoweimproveeducationaloutcomes?Asabroadstate‐levelanalysis,thisstudyfindsthatastatewideincreaseinpubliceducationfundingwouldnotguaranteebetteroutcomesregardlessofsocioeconomicconditions.Ourfindingsdosuggestthatthepolicyinitiativesthatwilllikelyimprovestudentachievementwillrequireadditionalspending.Thesepolicyinitiativesincludeincreasingthenumberofcollege‐educatedadultsandprovidingadditionalsupporttostudentswholiveinsingleparenthouseholds.However,asaconsequenceofitsbroadanalysis,theresultsofthisstudylikelyconcealnuanceineducationalinputsanddemographicdifferencesbetweenstates.AcloseranalysisattheschooldistrictlevelwithinUtahmayprovideamoredetailedprescriptionforfundingUtah’spubliceducationsystem,tailoredtoitsdemographiccharacteristicsandsocioeconomicneeds.Anintra‐stateanalysisofdifferentialfundingneedsbetweenschooldistrictsisnecessarytodeterminewhethertargetedfundingincreaseswouldhaveasignificanteffectoneducationaloutcomes.
FEBRUARY 13, 2017, 9:54 AM ‐ 12 ‐ OFF ICE OF THE LEGISLATIVE F ISCAL ANALYST
WHA T D R I V E S S T U D E N T A C H I E V EM EN T A C R O S S S T A T E S ?
Indicators MeanStandardDeviation Minimum Maximum N Years
NAEP4IndexScores1 229.55 5.97 212.85 245.09 300 2003,2005,2007,2009,2011,&2013NAEP8IndexScores1 272.60 6.67 256.49 288.79 300 2003,2005,2007,2009,2011,&2013GraduationRate(percent)2 77.26 7.36 51.30 91.40 550 2003–2013Currentspendingperstudent3 9,822.21 2,722.66 4,876.88 19,463.97 550 2003–2013%childreninsingleparenthouseholds4 32.21 5.30 17.00 49.00 550 2003–2013MedianhouseholdIncome(inthousands)5 55,243.28 8,480.67 37,825.00 80,007.00 550 2003–2013Educationalattainment(percent)6 27.13 4.79 16.30 40.29 550 2003–2013Student‐teacherratio7 15.62 2.77 10.69 30.02 550 2003–2013Englishlanguagelearners(percent)8 6.24 4.79 0.53 25.49 550 2003–2013Highqualityteachers(percent)9 93.65 8.02 34.30 100.00 550 2003–2013Teacherexperience(<4years)(percent)10 16.31 4.27 6.00 28.30 550 2003–2013Teacherexperience(≥15years)(percent)10 55.42 21.39 29.50 89.80 550 2003–2013Crimerate(per100,000)11 3,473.22 851.72 1,788.43 6,224.14 550 2003–2013Volunteerhourspercapita12 36.99 10.41 14.12 100.13 550 2003–2013Sources:
10NationalCenterforEducationStatistics;https://nces.ed.gov/surveys/sass/tables_list.asp11U.S.FederalBureauofInvestigation(FBI):UniformCrimeReports;U.S.CensusBureau(BOC);Moody'sAnalyticsCalculated12CorporationforNationalandCommunityService;VolunteeringandCivicLifeinAmerica
5U.S.CensusBureau,CurrentPopulationSurvey,AnnualSocialandEconomicSupplements.Accessedathttp://www.census.gov/data/tables/time‐series/demo/income‐poverty/historical‐income‐households.html.6U.S.CensusBureau(BOC):AmericanCommunitySurvey,CurrentPopulationSurvey,andPopulationEstimates,Projections;Moody'sAnalyticsEstimated7U.S.DepartmentofEducation:NationalCenterforEducationalStatistics(NCES)‐CommonCoreofData(CCD)‐PublicElementary/SecondarySchoolUniverseSurvey;Moody'sAnalyticsCalculated8U.S.DepartmentofEducation,NationalCenterforEducationStatistics,CommonCoreofData(CCD),"LocalEducationAgencyUniverseSurvey,"2003‐04through2013‐14.9EDFacts/ConsolidatedStatePerformanceReport,2004‐05:http://www2.ed.gov/admins/lead/account/consolidated/index.html
TABLEA.1DescriptiveStatisticsandSourcesofMeasures
1NationalCenterforEducationStatistics;http://nces.ed.gov/nationsreportcard/naepdata/2NationalCenterforEducationStatistics;CommonCoreofData3Moody'sAnalytics4U.S.CensusBureau,AmericanCommunitySurvey,KidsCountDataCenter(AnnieE.CaseyFoundation)
OFFICE OF THE LEGISLATIVE F ISCAL ANALYST ‐ 13 ‐ FEBRUARY 13, 2017, 9:54 AM
WHA T D R I V E S S T U D E N T A C H I E V EM EN T A C R O S S S T A T E S ?
1EricA.Hanushek,EricA.,JensRuhose,andLudgerWoessmann,“Itpaystoimproveschoolquality,”EducationNext,16,no.3(Summer2016):52‐60.
2DanChirasandDominicCrea,“TheEffectofEducationonCrime:EvidencefromPrisonInmates,Arrests,andSelf‐Reports,”TheAmericanEconomicReview,94,no.1(March2004):155‐189.
3DavidCutlerandAdrianaLleras‐Muney,“EducationandHealth:EvaluatingTheoriesandEvidence,”inMakingAmericansHealthier:SocialandEconomicPolicyasHealthPolicy,ed.RobertF.Schoeni,JamesS.House,GeorgeA.Kaplan,andHaroldPollack(NewYork:RussellSageFoundation,2008),29‐60.
4StevenJ.RosenstoneandJohnMarkHansen,Mobilization,Participation,andDemocracyinAmerica(NewYork:Longman,2002).
5AlanB.Krueger,“ExperimentalEstimatesofEducationProductionFunctions,”QuarterlyJournalofEconomics,114,no.2(May1999):497‐532.
6CarolineM.Hoxby,“TheEffectsofClassSizeonStudentAchievement:NewEvidencefromPopulationVariation,”QuarterlyJournalofEconomics,115,no.4(November2000):1239‐1285.
7JonahE.Rockoff,“TheImpactofIndividualTeachersonStudentAchievement:EvidencefromPanelData,”AmericanEconomicReview,94,no.2(May2004):247‐252.
8DouglasO.StaigerandJonahERockoff,“SearchingforEffectiveTeacherswithImperfectInformation,”JournalofEconomicPerspectives,24,no.3(Summer2010):97‐118.
9C.KiraboJackson,RuckerC.Johnson,andClaudiaPersico,“TheEffectsofSchoolSpendingonEducationalandEconomicOutcomes:EvidencefromSchoolFinanceReforms,”QuarterlyJournalofEconomics,131,no.1(February2016):157‐218.
10EricA.Hanushek,“TheFailureofInput‐BasedSchoolingPolicies,”TheEconomicJournal,113,no.485(February2003):64‐98.
11CoryTurner,KevinMcCorry,LisaWorf,SarahGonzalez,KirkCarapezza,andClaireMcInerny."CanMoreMoneyFixAmerica'sSchools?"AllThingsConsidered,NationalPublicRadio,April25,2016.
12SarahGonzalez,"WhatHappenedWhenOneofNewJersey'sPoorestSchoolDistrictsIncreasedSpending."WNYCNews,April24,2016.
13EducationalFinanceBranch.EconomicReimbursableSurveysDivision.PublicEducationFinances:2014.Washington,DC:U.S.CensusBureau,June2016.
14Jacksonetal.,2016.
15Hanushek,2003.
16TheNAEP4Sciencescoresareexcludedbecausethetestswereonlyadministeredin2009,2011(2011scoresarenotavailable),and2015andtheNAEPSciencescalerangesfromzeroto300ratherthanzeroto500liketheNAEPmathematicsandreadingscales.
17TheNAEP8Sciencescoresareexcludedbecausethetestswereonlyadministeredin2009,2011,and2015andtheNAEPSciencescalerangesfromzeroto300ratherthanzeroto500liketheNAEPmathematicsandreadingscales.
18Inthefollowinginstances,theACGRisnotavailablesotheAFGRisused:Idaho(2011,2012,2013);Kentucky(2011,2012);andOklahoma(2011,2012).
19TheAFGRisnotadjustedtoaccountforstudentsenteringorexitingthecohortduringhighschoolsoifthereisout‐migrationtheAFGRwillunderstatethegraduationratecomparedtotheACGR(in‐migrationwilloverstatethegraduationrate).Also,thediplomacountintheAFGRmayincludestudentswhotookmorethanorlessthanfouryearstograduatewhichwilloverstatethegraduationratecomparedtotheACGR.Ingeneral,theACGRmeasureprovidesmorecomparabilityacrossstates.Forany
FEBRUARY 13, 2017, 9:54 AM ‐ 14 ‐ OFF ICE OF THE LEGISLATIVE F ISCAL ANALYST
WHA T D R I V E S S T U D E N T A C H I E V EM EN T A C R O S S S T A T E S ?
statesmissingadatapoint,theaverageofthestate’sgraduationratefromtheprioryearandthefollowingyearwasused.Thisimputationwasonlyusedtoderivefivedatapointsoutof600totaldatapoints.
20Medianhouseholdincomeismeasuredin2015ConsumerPriceIndexforAllUrbanConsumersResearchSeries(CPI‐U‐RS)adjusteddollarstomeasurehouseholdincomeconsistentlyovertime.
21For2003through2004,2003‐2004valueswereused.For2005through2007,theaverageofthe2003‐2004and2008‐2009figureswasused.For2008and2009,the2008‐2009figureswereused.Theremainingyearshavedatacorrespondingtotheyear.
22Thevaluesfor2005areusedfor2003and2004aswellbecausedataforthoseyearsarenotavailable.
23AllteacherexperiencedataarefromtheSchoolandStaffingSurvey(SASS).Thesurveywasadministeredforthe2003‐2004,2007‐2008,and2011‐2012schoolyears.The2003‐2004percentagesareusedfor2003through2006;the2007‐2008percentagesareusedfor2007through2010,andthe2011‐2012percentagesareusedfor2011through2013.TheNationalTeacherandPrincipalSurvey(NTPS)hasnowreplacedtheSASSandthe2015‐2016dataarenotyetavailable.
24For2003‐2004,thedataavailablearepercentofteacherswiththreeorfeweryearsofexperience.
25For2003‐2004,thedataavailableareteacherswithfourormoreyearsofexperience.
26Datapointsareoutlyingiftheyfallbeyond1.5timestheinterquartilerange(IQR)whichisthedifferencebetweenthe75thand25thpercentilesofthedata.Utahdoesnothaveabox,maximumorminimum,oroutlyingpointsbecauseitsscoreisonlyonedatapointsothesevaluescannotbecalculated.
27StaigerandRockoff,2010.
28EricA.Hanushek,"EducationandtheNation'sFuture."InBlueprintforAmerica,ed.GeorgeP.Shultz,(Stanford,CA:HooverInstitutionPress,2016),89‐108.AnyaKamenetz,“5BigIdeasThatDon’tWorkInEducation.”NationalPublicRadio,2015.
