Oakland, CA 94607-5200 CHANCELLORS Dear … · 2020. 8. 14. · Adaptive Learning Technology Pilot...

UNIVERSITY OF CALIFORNIA

BERKELEY • DAVIS • IRVINE • LOS ANGELES • MERCED • RIVERSIDE • SAN DIEGO • SAN FRANCISCO SANTA BARBARA • SANTA CRUZ

1111 Franklin StreetOakland, CA 94607-5200Phone: (510) 987-9074http://www.ucop.edu

March 23, 2017

Improving student success is at the core of our University principles. Campuses acrossthe UC system continuously explore using evolving technologies alongside effectivemethodologies to enhance teaching and student learning. Recently, three campusesconducted year-long pilot studies on using adaptive learning technology to improvestudent success in entry level mathematics and chemistry. Each of the pilots showedpositive results.

UC Davis, UC Santa Barbara, and UC Santa Cruz each sought to address an issue thatmany entering freshmen face — the need to fill foundational knowledge gaps in order toplace well and succeed in first-year math or chemistry.

Through their implementation of summer bridge programs using the ALEKS adaptivelearning system, the campuses saw higher placement and improved student performancein these courses. They also showed that the students who participated generally did aswell as other students who initially placed into the course without the foundationalsummer help. One campus, UC Santa Barbara, provided additional support using ALEKSduring the academic year, yielding positive results as well.

A report highlighting the 2015-2016 pilot findings and recommendations by the three UCcampuses is attached as a PDF.

I encourage you to consider the findings and recommendations outlined in this report anddecide what conversations you might have at your campuses to advance meeting the needsof your students in their pursuit of academic success.

Attachment

cc: Executive Vice Chancellors and ProvostsProvost DorrUndergraduate Deans

Yours very truly,

Napolitano

CHANCELLORS

Dear Colleagues:

President

Adaptive Learning Technology Pilot Report

December 2016

AdaptiveLearningTechnologyPilot

December30,2016

TABLEOFCONTENTS

Introduction...........................................................................................................................1

AdaptiveLearning..................................................................................................................2

WhatisALEKS?.......................................................................................................................3

HowwasALEKSImplemented?..............................................................................................4

UCDavisPilot.........................................................................................................................5Summary...................................................................................................................................................................................5Keyresearchquestionsandanalysis...........................................................................................................................6

UCSantaBarbaraPilot..........................................................................................................10Summary................................................................................................................................................................................11Keyresearchquestionsandanalysis........................................................................................................................11

UCSantaCruzPilot................................................................................................................15Summary................................................................................................................................................................................16Keyresearchquestionsandanalysis........................................................................................................................16

SummaryofFindings.............................................................................................................20

Challenges.............................................................................................................................21

Recommendations................................................................................................................23Researchandacademics.................................................................................................................................................24Recommendation1......................................................................................................................................................24Recommendation2......................................................................................................................................................24Recommendation3......................................................................................................................................................24

Organizationalenhancements......................................................................................................................................25Recommendation4......................................................................................................................................................25Recommendation5......................................................................................................................................................25Recommendation6......................................................................................................................................................25

Technologyandinfrastructure....................................................................................................................................25Recommendation7......................................................................................................................................................25Recommendation8......................................................................................................................................................26

Conclusion.............................................................................................................................26

References............................................................................................................................28


December30,2016|Page1

INTRODUCTIONThisdocumentpresentsa2015–2016pilotstudyattheUniversityofCalifornia(UC)exploringtheuseofadaptivelearningtechnology,inthiscasetheAssessmentandLEarningKnowledgeSpaces(ALEKS)system,toenhancestudentsuccessandimproveinstruction.Followinganoverviewofthetechnologyandthestudy’spremise,thereportpresentsdetailedaccountsofthecampusresearchgroups’approachesandfindings.ThreeUCcampusesparticipatedinthispilot,whichoriginatedasoneoftheprogrammaticcommitmentsmadebyUCinassociationwiththegovernor’s2015–2016budget.AUCDavisresearchteamagreedtoleadtheproject,withresearchteamsfromUCSantaBarbaraandUCSantaCruzalsoparticipating.TheUCDavisteam,basedinthatcampus’sCenterforEducationalEffectiveness,hadbeguninvestigationsinthisarearelatedtomathematicsandsciencescourses.ResearchgroupsfrombothUCSantaBarbaraandUCSantaCruzhadalsoidentifiedthepotentialforadaptivelearningtechnologyinthesedisciplines.Eachofthecampusgroupswasmotivatedtoparticipateinthispilottoaddressacommonchallengefacingasegmentofincomingfreshmen—thattheyareoftenunpreparedfortherigorsofentry-levelcalculusandchemistrycourses.Onsomecampuses,thesestudentsplaceinto“workload”coursesthataddressbasicconceptsbutdonotcountascredittowardastudent’sdegree;onothers,studentsplaceintodevelopmentalcoursesthatgivecredit.Ineithercase,thesestudentsfallbehindtheirfreshmencohortsintheirfirstyear.Persistencewasalsoaconcernforcampuses,asmanystudentseitherdropthesecoursesorfailtoearnafor-creditgrade.Allthreeresearchgroupsalsosuspectedthat1)thesestudentsdonotstayinScience,Technology,EngineeringandMathematics(STEM)disciplines,and2)theymayexperienceadelayintimetodegreeasaresultoffallingbehind.Eachcampusresearchgrouphadexperiencewithadaptivelearningwithinthecontextofitsowncampus.AndeachcampushadexperiencewithALEKS,acontentandassessmentplatformthatfocusesonmathematicsandsciencecontentmastery.Thus,ALEKSbecametheadaptivelearningtechnologycomponentusedinallthreepilots.EachcampusteamimplementedtheALEKSsystemtoachieveitsindividualresearchgoals.



ADAPTIVELEARNINGTheconceptofadaptivelearningisnotnew.Infact,goodteachingpracticeshavelongembracedtheconcept—bychangingtheinstructionalapproachorthelearningcontenttoaddressstudents’individualstrugglesorgapsinlearning.Inthelast25years,numerouseducationaltechnologiesandsoftwaresystemshavebeendevelopedwiththeaimofmonitoringstudentperformanceandaddressingstudentlearninggapsacrossacademicdisciplines.Morerecently,givenadvancesintechnologiesthatapplyelementsofcognitiveresearchtostudentlearning,thenumberofsoftwaresystemspurportingtobeadaptiveandtoshowpositivelearningoutcomeshasincreased.Educatorsnowfaceagrowingnumberofoptionsfortechnologyproductsthatpresentdifferent,sometimesconfusing,claimsaboutlearningandthatposesignificantorganizationalimpactsandcosts-of-ownership.Educatorswanttounderstandhowthesystemsworkandhowtheyhelpstudentslearn.Buteducatorsandtechnologysuppliershavenotbeenabletofindoragreeuponacommondefinitionofadaptivelearning. In2012,theBillandMelindaGatesFoundation(Gates)gatheredagroupofuniversityleadersfromacrosstheU.S.todiscussadaptivelearningandthetechnologiesthatsupportit.Believingthatadaptivelearningshowedpotentialtoassistcollegestudentsinearningtheircredentialsmoreefficiently,GatesconvenedrepresentativesfromtheUniversityofTexasatAustin,AmericanPublicUniversity,theKentuckyCommunityandTechnicalCollegeSystem,andtheAmericanAssociationofStateCollegesandUniversities,amongothers.ThegatheringgeneratedaGates-fundedstudyofadaptivelearningandtechnologyinhighereducation.EducationGrowthAdvisors,nowTytonPartners,completedtheinitialcomprehensivestudyand,withcontinuedsupport,returnedtothesubjectin2015.Theysoughttounderstandhowadaptivelearninginhighereducationhadchangedsincetheinitialstudyandhowthechangesmightaffectthefutureadoptionofsuchtechnologies.Havinggatheredqualitativeinterviewswithleadersfrommorethan20institutionsandsurveyresponsesfrom35suppliersofadaptivelearningsolutions,thegroupdefinedadaptivelearning: “Solutionsthattakeasophisticated,data-driven,and,insomecases,nonlinearapproachtoinstructionandremediation,adjustingtoeachlearner’sinteractionsanddemonstratedperformancelevelandsubsequentlyanticipatingwhattypesofcontentandresourcesmeetthelearner’sneedsataspecificpointintime.” Theresult,LearningtoAdapt2.0:TheEvolutionofAdaptiveLearninginHigherEducation,(2016)revealedfiveemergentthemesacrossinstitutionsofhighereducation.Thefirstthemeis



thattherehasbeenlittlechangeintheabilityofinstitutionstomovefrompilottobroadimplementationacrosstheinstitution.Thisisimportanttounderstandingthecomplexitiesofimplementation-to-scale.Whileadaptivelearninghasmadesignificantleapsinfeaturegrowth,andhasbeenshowntoberelevanttocompetency-basededucation(twoofthefivethemes),institutionsfacemanydifficultchallengesthatrelatetointegrationofsuchsystemsintoexistinginstitutionalinfrastructureandtotheorganizationalandacademicworkflowsoftheinstitution.Thesechallengesaresignificant,andoftencostly.In2016,PearsonLearningreleasedDecodingAdaptive,areportbyEdSurge,anorganizationfoundedin2011withthepurposeofconnectingeducatorsandeducationaltechnologiststoinformationandresearchaboutwhattechnologycoulddotosupportteachingandlearning.Thereportdefinedadaptivelearninganddescribedvariousadaptivetoolsandsystems.Inthisreporttheresearchersassertedadefinitionofadaptivelearning:“…Digitaltools…thatcanrespondtoastudent’sinteractionsinrealtimebyautomaticallyprovidingthestudentwithindividualsupport.”Thesetworecentandsignificantendeavorsagreeonsomecommoncharacteristicsofadaptivelearning:

• Individualization• Adjustmenttothelearner’sinteractionsandperformance,usingdata• Adjustmentmadeinrealtime• Multiplepathways,potentiallynonlinear• Newcontentandresourcesrelevanttotheindividual’sdemonstratedneed

Atthesametime,bothrecognizethecomplexitiesinstitutionsfaceinevaluatingandimplementingadaptivelearningsystemswithintheacademicandtechnologycomponentsoftheinstitution’sorganization.

WHATISALEKS? TheAssessmentandLEarninginKnowledgeSpaces(ALEKS)systemwasdevelopedthroughresearchinmathematicalcognitivescienceknownasKnowledgeSpaceTheory,initiatedinthe1980sbyprofessorJean-ClaudeFalmagneatNewYorkUniversity(NYU)andtheUniversityofCalifornia,Irvine,andprofessorJean-PaulDoignonattheUniversityofBrussels.Accordingto



ALEKS’publishedinformation,“thecomplexeducationalsoftwarebasedonKnowledgeSpaceTheoryiscapableofefficientlyandaccuratelyassessingknowledgeinvariousdisciplines.”Thesystemassessesthestudent’s“knowledgestate”andcreatesalistofwhatthestudentisreadytolearn.Bymonitoringinrealtimethestateofthestudent’sknowledge,ALEKScontinuouslyoffersstudentsthecontenttheyarereadytolearn.Eachstudentbeginswithanassessmentofcurrentcourseknowledgewhereinthesystemchooseseachquestionbasedontheresponsetothepreviousone,presentingahighlyvariable,individualizedassessmentforeachstudent.ALEKSdevelopsaviewofthestudent’sknowledgestate,whichisrepresentedvisuallyinamulticoloredpiechart.Aftertheassessmentandthepresentationofthepiechart,thestudententersthelearningmode.Thestudentchoosesatopic(fromaselectionbasedonprerequisiteknowledgedemonstratedintheassessment)andworksthroughpracticeproblemsthatteachthetopic.Thestudentconsistentlymustgetthepracticeproblemscorrecttoachievemasterybeforemovingontoanothertopic.Thepiechartrepresentsthestudent’sgrowingmasteryoftopicsandasthestudentprogresses,heorsheachievesamorecompletepiechart.Periodically,ALEKSwillreassesstopicstoensuretheyareretained,andthesystemcontinuestoadjusttothestudent’slearning.WhileALEKSoffersanumberofproducts,theUCresearchteamschosetwodistinctproducttypesfortheAdaptiveLearningTechnologyPilot.ALEKSPPLfocusesonMathematicsPlacementandPreparation(UCSantaCruz),andALEKSHEScienceprovidescoursestoprepareforgeneralchemistryandsupportseachchemistrycoursethroughoutanacademicsequence(UCDavisandUCSantaBarbara).

HOWWASALEKSIMPLEMENTED?ALEKSwasimplementedateachofthecampusesindifferentwaysduringsummer2015,fallquarter2015andwinterquarter2016toaddresseachcampusteam’sspecificresearchquestionsrelatedtoinstructionalsupportforstudents’learningandachievementinbeginningchemistryandmathematics.AllthreecampusteamsemployedtheALEKSsystemduringthesummer,eithertopreparefreshmanstudentstotakeanentry-levelmathorchemistrycourse(torefreshandbuildmissingknowledgetopics)orasaplacementorre-placementmechanismthatwouldaccomplishthe



sameobjectivewhileeffectivelyreducingenrollmentinprerequisiteworkloadcourses.Campusteamsthentrackedstudentperformanceinthecoursestheytookduringthefallquarterandtheirsubsequentperformanceinthenextcourseinthesequenceduringwinterquarter.Tofurthersupportstudentsuccess,duringthefallquarter2015andwinterquarter2016,UCSantaBarbaraemployedALEKSinthecourseashomeworksupport.Inthepilot,ALEKSdidnotcomprisethewholeofanacademiccourse,althoughothercampusesmayusethesysteminthatway.Instead,thestudenteither1)preparedfortheacademiccourse,2)placedorre-placedintothecourse,or3)usedALEKSassupportduringtheacademiccoursework.Membersofthethreeparticipatingcampusresearchteamsmetonseveraloccasionsduringtheacademicyeartosharetheirstudygoals,approaches,experiences,toolsandmethods.Allidentifiedpositiveresults(someofmoresignificancethanothers)aswellaschallengestousingALEKSinthelearningenvironment.Thesedetailsfollow.

UCDAVISPILOTPILOTTYPE PreparationandPlacement

RESEARCHTEAM MarcoMolinaro,assistantviceprovostforEducationalEffectiveness

CatherineUvarov,formerpostdoctoralfellow

AlbertoGuzman-Alvarez,formeranalyst

WithsupportfromtheprovostandtheDepartmentofChemistry

CAMPUSCOURSES GeneralChemistry2A(CHE2A)andWorkLoadChemistryprepcourse(WLD41C)

ALEKSCOURSES SummerPrepforGeneralChemistry(SP-Chem)

NUMBEROFSTUDENTSIDENTIFIED 1099(butnotallinvitedtoparticipate)

NUMBEROFSTUDENTSCOMPLETED 274of551

SummaryIncollaborationwiththeUCDavisChemistryDepartment,theresearchersconductedapilotstudyofanonlinesummerpreparatoryprogramforchemistryusingALEKS.Aimedatincomingfreshmen,theonlinechemistrypreparationwasusedasanalternativetotheplacementexamsand/orworkloadchemistrycoursethatwouldnormallysatisfytheprerequisiterequirementforenrollinginCHE2Aforthe2015–16academicyear.Participationwasoptional,byinvitation.Ofnote:Duringthestudy,studentsreceivedtimedemailsfromresearchstaffregarding



participationanddeadlines,buttheyessentiallyworkedontheirownwithoutanyfacultyorTAintervention.Students’successfulcompletionofthesummerpreparatorycourse(achievementofpieat95percentorbetter)fulfilledtheprerequisiterequirementtoenrollinCHE2Awithouttheirtakingaplacementexamorthein-personpreparatorychemistrycourseWLD41C,forwhichstudentsdonotearnacademiccredit.

Keyresearchquestionsandanalysis

Question1 ConclusionsDoesusingALEKSasapreparatorycourseduringsummerhelpstudentstosucceedinGeneralChemistry2A?

• Studentsappearwell-preparedtosucceedintheGeneralChemistry2Acourse,andtheydoaswellasstudentswhoplacedintothecoursewithoutALEKS.

• StudentsappeartodobetterthanthosewhotooktheworkloadcourseWLD41Corwhorepeatedthecourse.

ANALYSIS

Usingscoresonpre-andpostassessmentsandthecommonfinalexamasprimarymeasuresofstudentperformance,theresearcherslookedatdataforfallquarter2015andwinterquarter2016asacombineddataset.Comparisongroupswereidentifiedandwerenarrowedtoincludeonly1)thosewhoplaceddirectlyinGeneralChemistry2A,2)thosewhocompletedALEKSSummerPrepforGeneralChemistryandplacedintoandtookthecourse,3)thosewhocompletedWLD41Candpassed,thusenteringGeneralChemistry2Aasprepared,and4)thosewhowererepeatingGeneralChemistry2A(withnootherinterventionsorre-placements).

FINDINGS

TheresearchersfoundnostatisticallysignificantdifferencesinoverallcourseperformancebetweenthestudentswhodirectlyplacedintoGeneralChemistry2AandthosewhocompletedALEKSSummerPrepandplacedinthecourse.However,theyfoundsignificantgainsinpre-andpost-statesbetweenthestudentswhofinishedALEKSSP-ChemandstudentswhopassedtheWLD41Ccourse.Overall,studentswhofinishedALEKSSP-Cheminthesummergained6.9percentagepointsmorethanthosewhotooktheWLD41Ccoursefirst.ThegainsaremoresignificantbetweentheALEKSSP-Chemstudentandthosewhorepeatedthecourse—scoring11.5percentagepointsmorebetweenpre-andpost-states.Usingfinalexamscores,amoredirectmeasureofstudentcoursegradeoutcome,thegainwasabithigher:8.81and13.51percentagepoints,respectively.



Whenusingunderrepresentedminority(URM)statustoevaluatewhetheranachievementgappersistsbetweenstudentpopulations,theresearchersfoundnosignificantdifferencebetweenthestudentswhodirectlyplacedinGeneralChemistry2AandthosewhocompletedALEKSSP-Cheminthesummer.SimilargainsusingfinalexamdataexistwithURMstudentswhocompletedALEKSSP-ChemandthosewhocompletedtheWLD41Ccoursefirst—witha10.3percentage-pointgainfortheALEKSSP-Chemstudents.

CONCLUSIONS

Ingeneral,studentswillingtoutilizeALEKSoverthesummerappearwell-preparedforCHE2Asuccess,performaswellasthosestudentswhoplacedintothecoursewithoututilizingALEKS,andgradebetterthanthosestudentswhotakeWLD41firstorwhorepeatCHE2Awithoutanyotherintervention.Theeffectappearstobethatmorestudentsareabletotakethecredit-bearingcourseinthefirsttermoffered,ratherthanbeingrequiredtoenterworkload(WLD)courses.TheseconclusionsholdtrueforURMstudentsaswell.

Question2 Conclusions

ArecertainstudentpopulationsutilizingALEKSmorethanothers?

• WhiletherearedifferencesinthepercentagesofstudentsbycollegewhostartedandcompletedALEKSinthesummer,noclearreasonisevidentfromthedata.

• Bothlow-incomeandURMstudentshavesignificantlylowercompletionratesthanstudentsnotidentifiedaslow-incomeorURM.

ANALYSIS

ThepilotteamcheckedhowstudentswereusingALEKSoverthesummer,andclusteredstudentsbasedontheirusagepatterns.TheteamperformedhierarchicalclusteranalysisinordertodetermineiftherewerecorrelationsbetweenusagepatternsandsubsequentperformanceinCHE2A,andifanydemographicdifferencesappearedintheclusters.Theresearchersformedclustersaround:1)overallchangeinthestudent’spiemastery,2)howearlyinthesummerthestudentreachedmaximumpiemastery,3)therateofthestudent’spiemasterychange,and4)thedegreetowhichthechangeinpiemasterywasgradualorintermittent.Theyalsoappliedninecharacteristics,includingtimespent,dailypiemasterychange,daysworkedtoachievemastery,daystomaximumpiemastery,daysspentinALEKS,initialassessmentfromALEKSandSATscore.ForsummerusageofALEKS,theteamanalyzed424studentresults,placingstudentsinoneof10clusters.Verysmallsamplesizesinsomeoftheclusterspreventedrobuststatisticalanalysis



ofperformance.Theyusedvisualizationsoftwaretoshowclusterperformanceinmasteringpieovertimeduringthesummer.StudentpopulationsincludedstudentsbyColleges,includingLettersandSciences,Engineering,BiologicalSciences,AgricultureandEnvironmentalSciences;anddemographics,includingURM,low-income,first-generationandfemale.

FINDINGS

TheresearchersfoundthatstudentswhocompletedALEKSinthesummertookdiverseapproachestogainpiemastery,evengivensimilarSAT(total)andinitialassessmentscores.ThiscanbeattributedtothenatureoftheALEKSsystem,whichishighlyindividualizedandbasedonadaptiveresponse.ManyofthestudentswhocompletedALEKSsucceededbygraduallyworkingonmasterythroughoutthesummer.Whenlookingacrossthestudentpopulations,withapproximatelyequalpercentagesfromeachcollege(withtheCollegeofLettersandScienceshavingaslightlylowerpercentageofstudents),thepercentageofengineeringstudentswhocompletedALEKSwasmuchhigherthanthatofstudentsinothercolleges.Completionbyengineeringstudentswas64percentofthatschool’stotalpopulation,withanaveragecompletionbystudentatothercollegesat53percent.Whileoneofthepilotgoalswastonarrowtheperformancegapbetweenstudentpopulations,whenlookingatdemographicstheresearchersfoundadisparitybetweenthedemographicpopulationandthetotalpopulationwhostartedandcompletedALEKS.Forinstance,forURMstudentscomparedtothosenotfromtheURMgroup,41percentofURMstudentsstartedandfinished,while65percentofnon-URMstudentsdid.Amonglow-incomestudents,48percentstartedandfinished,while62percentofnon-low-incomestudentsdid.Andforfirst-generationstudents,thepercentagedifferencewasessentiallythesame,48percentcompletionvs.63percentcompletionforstudentsnotofthatdemographicstatus.Femalestudents,ontheotherhand,hadlessdisparity,at55percentoffemalestudentsand59percentofmalestudentscompletingALEKS.

CONCLUSIONS

WhiletherearedifferencesinthepercentagesofstudentsbycollegewhostartedandcompletedALEKSinthesummer,noclearreasonisevidentfromthedata.And,whenlookingatthedemographicpopulationsandtheirdifferencesinpercentagethatstartandcomplete,thedatasetsweredeterminedtobetoosmalltodrawmeaningfulconclusions.Buttheresearchers



donotethatURMstudentsshowsignificantlydifferentcompletionratescomparedtonon-URMstudents.Factorsthatmayexplainthosedifferenceswerenotincludedinthisanalysis.

Question3 ConclusionsWhatarethebarriersofuse(ofALEKS)? • Noneofthepotentialbarriersidentifiedinthe

studywerefoundtobesignificant.

ANALYSIS

StudentswhostartedALEKS(anddidnotcomplete)andthosewhodidnotstartatallwereaskedtoselectexternalfactorsthatmayhavebeenimpedimentstostartingorcompletingALEKSSP-Chem.Asmallergroupofstudents,comprisingthosewhodidcompleteALEKSSP-ChemandthosewhochosetoenrollinWLD41,weregiventheMUSICModelforAcademicMotivationandSTEMIdentitysurveytolookintonon-cognitivemeasuresofpersistence.TheMUSICModelsurveymeasuresacrossfiveconstructsofmotivation(empowerment,usefulness,success,interestandcaring)andthreeconstructsofSTEMidentity(identification,abilityandcareer).Inthesurvey,studentsratedona6-pointLikertscalethedegreetowhichtheyagreedordisagreedwithaseriesofstatementsthatmayhavecontributedtonotstartingornotcompletingALEKSduringthesummer.Theresearcherscreateddensitygraphsofstudentratingstocomparethetwogroupsacrosstheconstructs.

FINDINGS

ThetopreasonthatstudentsreportedfornotcompletingALEKSinpreparationforthefallCHE2AcoursewashavingachievedapassingscoreonthechemistryandmathplacementexamsgivenatUCDavis.47percentofrespondentsreportedthisreason.Thenext-highestreasonreportedwaslackoftime,atapproximately30percent.Adropto16.7percentsclusteredaroundsuchreasonsasfeltconfidentinchemistry,didn’tfeelthesummercoursewasnecessary,ordecidedtotaketheWLDcourseinsteadofdoingsummerprep.Otherreasons,citedby10.8–13.7percentofrespondentsincludedconflictwithsummertravel,familyobligations,don’tneedtotakechemistry,andjobconflict.Farfewercitedlackoftechnology,summerprograminvolvement,junkmail,orother.Responseswerenotanalyzedaccordingtodemographicstatus,suchasURMorfirst-generationcollegestudent.IntheMUSICModelforAcademicMotivationandSTEMIdentity,overall,studentswhocompletedALEKSSP-ChemandstudentswhoenrolledinWLD41Cgavebothcourseshighratingsonallfiveconstructsofmotivation.IntheCaringconstruct,ALEKSSP-ChemisonlyslightlyhigherthanWLD41C,likelyduetotwofactors:1)thenatureoftheWLD41Ccourseas



aface-to-faceexperience,and2)theresearchteamhavingsentperiodicandpersonalizedemailsoverthesummertotheALEKSSP-CHEMstudents.TheresearchersbelievethesemayhavecontributedtoanoverallhighercaringratingfortheALEKSSP-Chemcourse.

CONCLUSIONS

Theresearchersdidnotstrivetoconnectstudents’reasonsfornotcompletingALEKSSP-Chemtotheresultsofthemotivationandidentitysurveyfromsomeofthecohorts.Further,noneofthebarrierstocompletionhavebeenfoundtobesignificanttoovercomeformorecompleteparticipationandmoresuccessfulperformanceinCHE2A.Still,theresearchershavedetermined(basedonthefindingthatnearlyhalfofthestudentsdidnottakeALEKSSP-ChembecausetheysuccessfullyplacedintotheCHE2Acourse)thatthecampuswillnotincludethisgroupinanyfuturecomparisons.TheUCDavisteamdididentify,however,thatrunningasummerbridgeprogramforincomingfreshmenhasseveralchallenges.First,thestudentsarenotyet“fullymatriculated,”i.e.,theyhavenotenteredtheuniversityasmatriculatedstudentsforthefall,sotheyhavenotyetengagedinmatterslikeself-identification(asneedingaccommodationorothersupport).Furthermore,thesefreshmenhavenotbeguntoenjoytheorganizationalstructuresthecampusoffers,suchasacademicadvisementandfinancialaid,norhavetheybeguntodeveloprelationshipswiththeirfreshman-yearfacultyforsupport.Theresearchersdidnotethattheytriedtopreventpossiblebarrierstoparticipation,suchasthelackofcommunicationandsupport(addressedthroughtheemailssenttostudentswhobeganALEKSSP-Chem),andthecostofALEKS(typicallythestudent’sresponsibility,butcoveredbythecampusordepartmentforthepilot).

UCSANTABARBARAPILOTPILOTTYPE PreparationandHomeworkHelp

RESEARCHTEAM CarlGutiérrez-Jones,DepartmentofEnglishandassociatevicechancelloranddeanofUndergraduateEducation(2015–2016)

LindaAdler-Kassner,interimco-deanofUndergraduateEducationandprofessorofWritingStudies

StevenVelasco,directorofInstitutionalResearch

DarbyFeldwinn,DepartmentofChemistry

MargaritaSafronova,DepartmentofPoliticalScience

CAMPUSCOURSES Chemistry1A(CHE1A)



ALEKSCOURSES GeneralChemistry(Firstquarter—withpartialuseinsummerprep)

NUMBEROFSTUDENTSIDENTIFIED 598

NUMBEROFSTUDENTSCOMPLETED 598

SummaryEarlierresearchbyUCSantaBarbarachemistryfacultymembersrevealedthatmanystudentsenteringthestudyofchemistrystruggledwithfundamentalmathandscienceconceptsandhaddifficultyapplyingtheseconceptstoproblemsassociatedwithentry-levelchemistry.UCSantaBarbarafacultyelectedtouseALEKSasasummercoursepriortothebeginningofthefallquarter2015,andasahomeworkassignmentduringtheterm,alongsideworkwithintheCHE1Acourse.TheaimofUCSantaBarbara’spilotwastostudytheeffectofusingALEKSonoverallsuccessinCHE1A,agatewaycourse(onerequiredinanumberofmajors),andtoascertainthedegreetowhichALEKSimprovedperformanceinCHE1A.


Question1 ConclusionsDoesperformanceontheALEKSinitialassessmentknowledgecheckadministeredinthefirstALEKSsummercourseserveasapredictivetoolforsuccessinChemistry1A?

• CompletionoftheALEKSsummercoursehasasignificantandpositiveeffectonstudents’finalgradesinCHE1A.

ANALYSIS

ThestudypopulationconsistedofallstudentsinChemistry1Ainfallquarter2015withcomparisongroupsforthosewhodidcompletethesummercourseandthosewhodidnot.Resultsofsummerplacement,coursegradesofB-orabove,andthefinalgradeinCHE1Awereusedasprimarymeasureswhilecontrollingforavarietyofacademicpreparationcharacteristics(suchashighschoolGPAandSATscore)anddemographiccharacteristics(suchasgender,first-generationcollegestatusandsocioeconomicfactors).SuccesswasdefinedasagradeofB-orabove.

FINDINGS

TheresearchersfoundthatthestudentswhocompletedALEKSassignmentsinthesummerandwhosucceeded—earningascoreofB-orbetterinCHE1A—increasedtheirprobabilityofreceivingsuchascoreby30percent,whencontrolledforotherfactors.ThedatafurthersuggestthattheALEKSsummercourseinitialassessmentisasignificantpredictorofsuccessin



CHE1A.TheestimatesindicatethatforeveryonepercentincreaseinALEKSsummercoursetopicmasterydemonstratedintheinitialplacement,theprobabilityofsuccessinCHE1Aincreasesbynearlyhalfapercent.

CONCLUSIONS

CompletionoftheALEKSsummercoursehasasignificantandpositiveeffectonstudents’finalgradeinCHE1A.

Question2 ConclusionsDofinalgradesvaryamongstudentswhodoanddonotcompletetheprequarterassignment?

• CompletionoftheALEKSsummercoursehasasignificantandpositiveeffectonstudents’academicperformanceinCHE1A.

ANALYSIS

Todeterminedifferencesbetweenfinalgradesamongstudentswhocompletedordidnotcompletethesummercourse,theresearcherscalculatedaverageandmedianfinalCHE1Acoursegrades.Usinga4.0gradescale,theresearchersdidnotcorrectforacademicpreparationorotherdemographicinformation.

FINDINGS

Studentswhocompletedthesummercoursereceivedhigheraveragegradesthanstudentswhodidnotcompletethesummercourse.Whilethegroupofnoncompletersisonly10percentofthesizeofthegroupofcompleters(56studentsvs.526students),theresearchersfoundnearlyagrade-and-a-halfdifferencebetweenthemeangradesofsummercompletersandnon-completersofthesummercourse:2.61to1.21,respectively.Themediangradesshowedanevenlargerdifference:3.00to1.00.However,theresearchersnotethatmuchofthedifferenceinaveragegradesislikelytobeexplainedbyprioracademicpreparationandothercharacteristicsofstudents.Thus,theresearcherslookedfornetimpactwhenconsideringthoseothervariables.Still,resultssuggestthatstudentswhocompletethesummercourseinALEKSreceiveanestimated.79highergradethanstudentswhodonotcompletethesummercourse.

CONCLUSIONS

Again,theresearchersnotethatcompletionoftheALEKSsummercoursehasasignificantandpositiveeffectonstudents’academicperformanceinCHE1A.



Question3 Conclusions

DoestheuseofALEKSduringtheterm,whenincorporatedintoCHE1Aashomeworkhelp,contributetostudents’abilitiestodeveloptheknowledgeandstudyhabitsnecessaryforsuccessinChemistry1A?

• ThereisverylittleevidencethatuseofALEKScontributestostudents’abilitiestoreflectonanddeveloptheirapplicationofknowledgeinCHE1A.

ANALYSIS

TheresearchersexaminedtherelationshipbetweenaStudyBestPracticesIndex(SBPI)andmetricsconcerningastudent’suseofALEKSinthefallquarter.TheSBPIisbasedonclosed-endresponsestoaself-assessmentdistributedtostudentsduringthemidtermexaminCHE1A.Theindexconsistsofself-reportedpracticesandbehaviorsaboutthestudent’sengagementinvariousactivitiesthatshouldleadtosuccessinthecourse.Theteamusedthisindexasaproxyforthepracticeofgoodstudyhabits.TheresearcherslookedtodeterminewhetherornottheuseofALEKS(asmeasuredbythenumberoftopicslearnedperhourandtotaltimespentinALEKS)isassociatedwithhigherlevelsofSBPI.

FINDINGS

UCSantaBarbararesearchersfoundnoevidencethattheALEKSmeasures(topicslearnedperhourandtotaltimespentinALEKS)wererelatedtotheSBPI.Infact,whenlookingatotherfactors,specificallyhighschoolGPA,adefinitepositiveassociationbetweenGPAandhigherlevelsofSBPIwasfound.Noothercontrolsforacademicpreparationshowedanysignificanteffect.

CONCLUSIONS

TheresearchersindicatedthereisverylittleevidencethatuseofALEKScontributestostudents’abilitiestoreflectonanddeveloptheirapplicationofknowledgeinCHE1A.

Question4 ConclusionsDoesreflectiononlearningviaALEKS(andothercourseactivities)asitisincorporatedintoCHE1Acontributetostudents’abilitiestodevelopandapplyknowledgewithinChemistry1A?

• Onlytheidentifiedmissedquestions(onthemidterm)appearsignificantlyrelatedtoastudent’sabilitytodevelopandapplyknowledgeinthecourse.

• Thiseffectisnegatedwhencontrolssuchasprioracademicachievement,highschoolGPAandSATscoreareapplied.



ANALYSIS

Forthisquestion,UCSantaBarbararesearchersusedopen-endedquestionsincorporatedintothemidtermself-assessmentgiventostudentstakingCHE1A.Studentswereaskedto:1)identifytheproblemstheyhadseenbefore,butwhichtheymissedonthemidtermexam,2)saywhetherornottheyhadstudiedthemissedmaterialpriortotheexam,and3)discusshowtheywouldchangetheirstudyhabitstoensureabettergradeonfutureexams.Theteamhypothesizedthatuseofandreflectiononlearningcontributedtostudents’abilitiestodevelopandapplyknowledgeandcorrectlyanswerquestionsfromALEKSembeddedinthefinalexam.

FINDINGS

AccordingtoUCSantaBarbararesearchers,theabilitytoidentifyandreflectonpracticesandprocesseslikethoselistedaboveareassociatedwithmetacognitivedevelopment,ortheabilitytoconsciouslyrecognizeone’sownprocessesandactionsinalearningcontextandtomakedecisionsaboutlearninginthatcontextbasedonrecognition.Metacognitiveawarenesshasbeenidentifiedasimportantforlearningsuccess.Theresearcherscodedstudents’responsesintothethreedimensionsof:1)identifiedmissedquestions,2)studiedmissedprior,and3)changedhabitasaresult.Although285studentsparticipatedinthecourse,notallofthestudentsrespondedtothequestions,whichtheresearcherspointoutlimitstheutilityofthefindings.

CONCLUSIONS

Theresearchersconcludedthatonlytheidentifiedmissedquestions(onthemidterm)dimensionappearssignificantlyrelatedtoastudent’sabilitytocorrectlyanswertheALEKS-relatedproblemonthefinalexam.However,thiseffectisnotpresentwhencontrolsforprioracademicachievement,suchashighschoolGPAandSATscore,areputinplace.Theotherdimensionsshowedweakandnegativecorrelationstoperformanceonthefinalexam.

Question5 ConclusionsForstudentsinthelowestquartileoftheALEKSassessment,whatcorrelation,ifany,existsbetweenstudents’ALEKSuse,performanceonparticularexamquestionsandstudents’finalgrades?

• TheresearchersfoundcompletionoftheALEKSsummercourseanduseofALEKSinthefallquartertobesignificantlyrelatedtotheachievementofhighergradesinCHE1Aforstudentsinthelowestquartile.



ANALYSIS

ToexploretheimpactofALEKSuseamongthelowestquartilestudents,UCSantaBarbararesearchersusedamodelthatallowedthemtoassesstheestimatedchangeinstudents’CHE1Agradebasedonavarietyofexplanatoryfactors,includingALEKSsummercoursecompletion,previousacademicperformanceanddemographiccharacteristics.Inthesummer2015ALEKSpilot,137of594studentswereinthelowest-quartileinitialassessment.Thelowest-quartilestudentsachieved35percentoftopicsknownintheinitialassessmentcomparedto59percentforallotherstudents.80percentofthesestudentsalsocompletedtheALEKSsummercoursecomparedto90percentofallstudents.Inall,lowest-quartilestudentslearnedfewertopicsperhourandspentmoretimeinALEKS.

FINDINGS

StudentswhocompletedtheALEKSsummercoursehadahigheraverageCHE1Agradethanstudentswhodidnotcompletethecourse,representingastatisticallysignificantcorrelationbetweenfinalgradeandcompletionofALEKSduringthesummer.Further,studentsinthelowestquartileareestimatedtohaveanaveragegradeof.45higheriftheycompletedtheALEKSsummercoursecomparedtothosewhodidnot.Thestrongestpredictorinthemodelsforlowest-quartilestudents,similartothemodelsforallstudents,isthenumberoftopicslearnedperhourduringthefall.And,forstudentsinthelowestquartile,whilecontrollingforpreviousacademicperformanceanddemographiccharacteristics,therecontinuestobeapositiveandstatisticallysignificantrelationshipbetweentopicslearnedperhourandfinalgradeinCHE1A.

CONCLUSIONS

AmongtheleastpreparedstudentsinthelowestquartileoftheALEKSsummercourseinitialassessment,theresearchersfoundcompletionoftheALEKSsummercourseanduseofALEKSinthefallquartertobesignificantlyrelatedtotheachievementofhighergradesinCHE1A.Theresearchersindicatethistobeaverypositiveresult,providingevidencethatthechancesofsuccesscanberaisedthroughtheuseofinnovativetoolsandinterventionssuchasALEKS.

UCSANTACRUZPILOTPILOTTYPE PreparationandPlacement

RESEARCHTEAM JayePadgett,DepartmentofLinguisticsandinterimviceprovostforStudentSuccess

DebraLewis,DepartmentofMathematics



JulianFernald,directorofInstitutionalResearch,AssessmentandPolicyStudies

CAMPUSCOURSES Math2(CollegeAlgebra),Math3(Precalculus),Math11A(CalculuswithApplications),Math19A(CalculusforScience,EngineeringandMathematics)

ALEKSCOURSES ALEKSPlacement,PreparationandLearninginMathematics(ALEKSPPL)

NUMBEROFSTUDENTSIDENTIFIED 3974(3,625enteringfreshmen+349transfers/other)

NUMBEROFSTUDENTSCOMPLETED 2,686(tookALEKSplacement;manydidnotneedanyofthesecourses)

SummaryUCSantaCruzpilotedtheuseofALEKSPPLinthesummertoallowincomingstudentstoplaceintoandtakemoreadvancedcoursesinmaththantheyotherwisewouldhave.Thepilot’smaingoalwastoshiftincomingfreshmenenrollmentsupwardfrommorepreparatoryordevelopmentalcoursestothemorecollege-readycourses,withoutjeopardizingstudents’performanceinmath.


Question1 ConclusionsHowmanystudentstooktheassessmentmorethanonce?Howdidtheirplacementscorechange?Didthepilotsucceedinshiftingenrollmentstoahighermathcourse?

• Morestudentstookmoreadvancedclasseswithoutendangeringtheiracademicsuccess.

• EnrollmentinthelowerlevelMath2courseinfall2015was49percentofwhatitwasinfall2014.

ANALYSIS

BasedontheirinitialplacementscorewithALEKSPPL,studentswereencouragedtocontinueinALEKSPPLandretaketheplacementassessment,sometakingtheassessmentasecondorthirdtime.OfthetotalnumberofstudentswhotooktheALEKSPPLplacement,771retooktheexamatleastonce(ofwhom722wereenteringfreshmen).Theresearchersremovedstudentswhoplacedintothehighest-placementtier(sincethesestudentscouldhavenoincentivetoreassess),finding32.6percentofallstudentsand35.1percentofenteringfreshmenreassessed.Theresearcherslookedfurtheratthestudents’re-placementtiers,sincethesetierswouldindicatethemathematicscoursesatUCSantaCruzforwhichthestudentwouldbeeligible.Ofthe722enteringfreshmenwhoretooktheplacement,83.8percentimprovedbyatleastonetier.Theresearchersthencomparedthehistoricenrollmentsandcoursecompletionsforthefirsttwomathcourses,Math2(CollegeAlgebra)andMath3(Precalculus)fromthe



2011–12academicyearthroughthe2015–16academicyearusingthesummerpilotofALEKSPPLasapivotpointtocheckitsimpactonenrollment.

FINDINGS

WhilethenumbersofstudentswhocompletedMath2andMath3coursesincreasedeachyearbetweenthe2011–12and2014–15academicyears,afterintroducingALEKSPPLasare-placementmechanisminthesummerof2015,researchersnotedasignificantdecreaseinthenumbersofstudentswhotookthosecoursesthroughthe2015–16academicyear.Theresearchersfurtherexaminedfall2014and2015performancedatainallthemathematicscoursesaffectedbytheintroductionoftheALEKSPPLplacementmechanismtodetermineiftherewasaneffectonthepassratesinthesecourses.Theresearchersnotedadramaticoverallshiftinstudentsfrommoreintroductorytomoreadvancedmathcourses.Theynotedthatperformancehasnotshifteddramaticallyoverall,citingthatpassratesinthreeofthefivecoursesrosewhiletheyfellsomewhatintwoofthecourses(Math3andMath19A).

CONCLUSIONS

Theresearchersconcludedthatthepilotwassuccessfulingettingmorestudentsintoadvancedclasseswithoutendangeringtheiracademicsuccess.Theresearchersdonotethatotherfactorsmayhaveimpactedenrollmentshifts.Forthecohortofstudentswhotypicallytakecollege-levelalgebra(Math2)—enteringfreshmen—therewasa10percentdropinthesizeoftheincomingclassby419students(from4,037infall2014to3,618infall2015).YetenrollmentinCollegeAlgebradecreasedby51percentfromfall2014tofall2015.And,therewereincreasesinenrollmentinthehigher-levelmathcourses.

Question2 ConclusionsSomestudentsinitiallyplacedintocourseXbutre-placedintoahighercourseYafterworkingwithinALEKSPPLtoimprove.AretheoutcomesforthesestudentsasgoodastheyareforthosewhoplacedinitiallyintocourseY?Howdidthesetwogroupscompareinasubsequentmathcourse?

• Studentswhore-placedintoacourseafterinitiallyplacinglowergenerallydidaswellasstudentswhoinitiallyplacedintothesamecourse.

• Studentswhore-placedintoafallcourseafterinitiallyplacinglowerdidaswellinasubsequentwintercourseasstudentswhoinitiallyplacedintothesamefallcourse.

ANALYSIS

Theresearcherslookedatperformanceinthreecourses:Math3(Precalculus),Math11A(CalculuswithApplications)andMath19A(CalculusforScience,EngineeringandMathematics),firstcomparingthegroupofstudentswhore-placedintothecourseviaALEKSPPLassessment



withthosewhoinitiallyplacedintothecourse.Theysoughttodetermineiftherewasasignificantdifferenceinstudentperformanceoverallinthecourse.Toaddressthequestionofhowwellthestudentswhore-placedfaredovertime,theresearchersalsoexaminedfactorsthatpredictedsuccessoftheseplacementgroupsinsubsequentmathcourses—usingthesecoursesequences:Math3(Precalculus)àMath11A(CalculuswithApplications)Math3àMath19A(CalculusforScienceEngineeringandMath)Math11AàMath11B(CalculuswithApplications,course2)Math19AàMath19B(CalculusforScience,EngineeringandMath,course2)Theteamexaminedstudentperformanceoutcomesineachsequenceandtheirgroupashavingplacedinitiallyorre-placedafterALEKSPPL.

FINDINGS

Whentakingcertaindemographicfactorsintoaccount,suchasURM,gender,first-generationstatusandPellGrant-recipientstatus,theresearchersfoundnosignificantdifferenceincoursegradeorpassratebetweenthegroupwhore-placedandthegroupwhoplacedintothecourseinitially.Infact,theresearchersnotethatthestrongestpredictorofsuccessisprioracademicpreparationasmeasuredbySATmathscoreandhighschoolGPA.ThisheldtrueforMath3andMath11A.ForMath19A,whilethefindingsrevealedthatre-placementdidcontributetopredictingthefinalgrade,itwasnotasstrongapredictorasthemathSATscoreorhighschoolGPA.Whenlookingathowstudentsfaredovertime,insubsequentcourses,theresearchersfoundthatstudents’groupplacementwasofnoconsequencetosuccessfulperformanceinlatercourses.StudentswhoworkedtheirwayuptoPrecalculusplacementviaALEKSPPLdidaswellinthenextmathcourseasthosewhoplacedintoPrecalculustobeginwith,allelsebeingequal.ThesameistrueforstudentswhotookCalculuswithApplicationsorCalculusforScience,EngineeringandMathematics.Fromtheresearchers’perspective,thisisapositivefindingthatsupportstheoverallgoalofhelpingstudentsgainintime-to-degreewithoutsacrificingacademicperformance.

CONCLUSIONS

TheUCSantaCruzresearchersconcludedthatALEKSPPLwashelpfultothosestudentswhodidretaketheplacementandwhoplacedhigher,thustakingandgenerallysucceedinginthe



higher-levelmathcourse.And,whencontinuinginthemathematicssequence,studentswhore-placedcontinuedtodoaswellasthosewhoinitiallyplacedintothesamecourse.

Question3 ConclusionsHowdooutcomesdependonplacementscoresinALEKSPPL,bothinitialandfinal,orontheamountoftimespentinALEKSPPL?

• ResearchersdidnotfindthatimprovedperformanceinthemathcourseswasassociatedwithmoretimewithinALEKSPPL.

• Thestudent’sscorewasonlyweakly(initialscore)ornotassociated(re-placementscore)withimprovedperformanceinthemathcourses.

ANALYSIS

TheUCSantaCruzresearchteamwantedtoexplorewhetherornottheamountoftimespentinALEKSPPLhelpedpredictthefinalcoursegrade,andwhetherthestudent’sscoreafterreplacing(initialscorevs.re-placementscore)helpedpredictthefinalcoursegrade.TheresearchersexaminedthecontributionofeachofthesefactorsaswellasthatoftheSATmathscoretofinalgradesinPrecalculus(Math3),CalculuswithApplications(Math11A)andCalculusforScience,EngineeringandMath(Math19A).

FINDINGS

WiththeMath3course,theresearchersfoundthatthebestpredictoroffinalgradewasthestudent’sSATmathscore,accountingfor37percentofthevarianceinfinalcoursegradepoint.Thenextbestpredictor,atsevenpercent,wasthetimespentworkinginALEKSPPLtore-place.However,timespentworkinginALEKSPPLbeforeplacementwasnegativelyassociatedwiththefinalcoursegrade.Further,thestudent’sinitialplacementscoreaccountedforonlytwopercentofthevariance,andthefinalplacementscoredidnotcontributetoanyprediction.AnanalogousanalysisforCalculuswithApplications(Math11A)andCalculusforScience,EngineeringandMath(Math19A)foundthatonlyastudent’sinitialplacementscorewithinALEKSPPLhelpedtopredictthefinalcoursegrade,thoughaccountingforonly13percent(Math11A)oreightpercent(Math19A).

CONCLUSIONS

Theresearchteamconcludedthatastudent’sprioracademicpreparation,asrepresentedbytheSATmathscoreand/oraninitialplacementscore,explainsmostofthefinalcoursegrade.UCSantaCruzresearchersdidnotfindthatimprovedperformanceinthemathcourseswasassociatedwithmoretimewithinALEKSPPLorwithastudent’sscoreinALEKSPPLafterre-placing.



Goingfurther,theresearchersstatedtheremaybemanyreasonsthatstudentsmaytakelongerworkingwithinALEKSPPL,notallofwhichimplyanadvantage,andsomeofwhichareclearlybeyondthecontroloftheALEKSsystemtoaddress(suchasanxietyabouttakingthenextassessmentorlackofconfidence).TheresearchersnotedonesignificantchallengeinthatstudentsworkwithinALEKSPPLontheirown,andtheenvironmentinwhichtheyworkisnotcontrolled,norwastherefacultymediationfromalearningperspective.

SUMMARYOFFINDINGSOverall,eachofthecampusesinvolvedintheAdaptiveLearningTechnologyPilotfindthatwhenALEKSisusedbystudentsasitisintended—withanestablishedachievementgoalorpiemasterylevel,andwithstudentpersistencetowardthatgoal—resultsarepositiveinrelationtoastudent’soverallperformanceinthecoursetowhichitisapplied.Insomecases,thesamepositiveresultsareevidentinat-riskpopulations,suchasURM,low-incomeandfirst-generationstudents.Whilethosepopulationsmaydemonstrateslightlylowerperformanceresults,theresultsarestillpositivewithintheoveralldemographicgroup.However,ALEKSisnota“plug-and-play”panaceaforpoorstudentperformanceintheseselectedmathandchemistrycourses.Whenimplementedalongsideacourseduringtheacademicterm,itdoesrequirefacultyorTAmonitoringandappropriatecommunicationwithstudentsabouttheirlearningandtheirperformance,regardlessofhowtheyareachievingresultsinALEKS.Thiskindofsupportmaydifferdependingontheimplementationmodelputintoplace.UsingALEKSaspreparationforentry-levelcalculusorchemistryTheimplementationofALEKSaspreparationfortakinganentrylevelcoursetypicallyoccursinthesummerwithincomingfreshmen.BothUCDavisandUCSantaBarbarareportpositiveresultswithstudentswhocompletedtheirsummerALEKSassignment.ForUCDavis,morestudentsbeganCHE2AthantookWLD41C(workloadchemistry),oneofthegoalsthecampuswantedtoachievethroughsummerpreparation.AndstudentswhocompletedALEKSSP-ChemperformedaswellasstudentswhoplacedintothecourseinitiallyanddidnotneedtotakeALEKSinthesummer,andalsoperformedsignificantlybetterthanstudentswhotookWLD41C,orwhorepeatedthecoursewithoutintervention.UCDavisensuredthatstudentswouldnot



havetopayfortheALEKSsystemandsentregular,personalizedemailstostudentswhostartedALEKSthroughoutthesummer.ForUCSantaBarbara,whilestudentswhocompletedthesummerALEKScoursewere30percentmorelikelytoreceiveagradeofB-orabovethanstudentswhodidnotcompletethesummerassignment,theaveragefinalgradefortheCHEM1AcourseinthefallwasC+comparedtoaDforthosewhodidnotcompletetheALEKSassignment.UsingALEKStoplaceorre-placeatahigherlevelinentry-levelchemistryormultilevelmathsequencesIntheUCDavisALEKSsummerprogram,successfullycompletingALEKSandachievinga95percentorhighermasterymettheprerequisitestoplaceinCHE2A.OfthosewhoplacedintothehighercoursegiventheirsummerALEKSSP-Chemwork,notallelectedtotakeCHE2A.SomeelectedWLD41C.Still,asnotedearlier,UCDavisfoundnostatisticallysignificantdifferenceinperformancebetweenthosestudentswhoinitiallyplacedintothecourseanddidnottakeALEKSandthosewhocompletedALEKSsuccessfullyandthentookthecourse.AtUCSantaCruz,therewasasignificantdropinthenumberofstudentsneedingtotakeMath2(CollegeAlgebra),oneofthekeygoalsofthepilot.And,re-placingintoahigherlevelcourse,Math3,didnotnegativelyaffectstudents’academicperformanceinthecourse.Still,theresearchersdidreportadropinthenumberofincomingfreshmen,whichcouldhavecontributedtosomeofthatresult.UsingALEKSasongoingacademicsupport—homework—duringanacademictermTheUCSantaBarbarateamimplementedALEKSduringthetermalongsidetheacademiccourse.TheirfindingsindicatethatmasteryofALEKStopicsduringthequarterdoeshaveastatisticallysignificantrelationshiptostudents’finalcoursegrades,andthatthisrelationshipisattributedtothenumberoftopicslearnedperhour.PersistencewithALEKSandmasteringtopicsappearstohavecontinuedpositiveimpacts,evenwithlowest-quartilestudents.

CHALLENGESAsnotedintheGatesreport(2016),highereducationinstitutionscontinuetofacechallengestoimplementadaptivelearningtechnologiesatscale.Thereportnotedthatwhilemanymoreinstitutionshavebeguntopilotadaptivelearningsystemsduringthelastthreeyears,fewhavetransformedpilotsintofull-scaleimplementation.Thateffortisasignificantone,requiring



institutionalstrategicobjectivesthattakeintoconsiderationallfacetsofitsorganizationalstructure,fromacademicstoresearchtotechnologytofinanceandaccountability.SomeofthespecificchallengesnotedbytheUCresearchersinconductingtheirpilotsarerelatedtosummerimplementation.Insummer,theparticipatingstudentshavebeenadmitted,buttheyhavenotyetfullymatriculatedatthecampus.Theyhavenotyetfullyengagedwiththeuniversity,maynothavereceivedtheirfinancialaidorattendedorientation,ormaynothavemetwithacademicadvisersoridentifiedasneedingaccommodationofanykindwhenthesummerprogramstarts.Thus,akeychallengeforsummerprogramsiscoordinatedcommunicationattheuniversityandwiththestudent.Thecampusesnotedtheneedfortimelyandinformativecommunicationwithstudentsthroughchannelswithwhichtheyengageduringthesummer,suchasStudentAffairsandAcademicAdvisement.Campusesnotedthatcollaborationwiththesegroupsandtransparencyregardingthegoalsofthesummerprepandplacementprogramsneedtobeformalizedsothatappropriateandappropriatelytimedcommunicationstostudentsencouragetheirparticipationandultimatechancesofsuccessinthesummerprogramand,thus,theirfallcourses.AnotherchallengethecampusesfaceisthecostoftheALEKSsystem.Atmanycampuses,ALEKSsystemcostsarepassedontothestudentas“coursematerialsfees.”Andthesefeesvarybasedonthecampusandthelengthoftimethestudentwillusethesystem(from$35toaccessthesystemforatermtonearly$100toaccessfortwoyears).Costscanbeachallengeforstudentsandaddtotheoverallcostofmaterialsacrosstheacademicyear.UCSantaCruzfoundamethodtopayforALEKScostswithoutpassingfeesontothestudent.Withasummerprogram,manystudentswhohavebeenidentifiedasparticipantsinALEKShavenotreceivedtheirfinancialaidpackagesintimetosupportsummerusage.InthecaseofUCDavis,theresearchersmadearrangementswiththevendortodelaythebillingforusageuntilthebeginningoftheacademicterm,whenstudentsreceivedtheirfinancialaid.Regardlessofusageinthesummerorduringtheacademicyear,costingeneralisachallengebecauseitraisesinstitutionalconcernsaboutstudentsandthefeestheyarerequiredtopay.Attimes,discussionofpassingcoststostudentscanbecontroversialatacampusorwithindepartments.SomethinkthatthecampuspassesentirelytoomanycostsontostudentsandthatstudentsareburdenedbytextbookandeTextbookfees,aswellastechnologyfees.InthiscontextALEKSrepresentsanothercosttostudents,inadditiontothecoursematerialsfeestheypayforthecourseandthetechnologyfeestheypayfortheterm.



Whilenoneoftheresearchersindicatedanyknownissueswithstudentsintheirpilotsrelatedtoaccessibility,thefactisthattheALEKSplatformisnotcompletelyaccessible.ItdoesnotmeetADArequirements,Sections504and508,norWorldWideWebConsortium(W3C)andWebContentAccessibilityGuidelines(WCAG)version2.0,whichthefederalgovernmentrequires.Currentscreen-readertechnology(usedbyblindstudentsto“read”thetextonthescreen)cannottranslatecertainmathematicalandscientificformulasnorcontentthatishighlyvisualinnature.WhilestudentswhouseALEKSmaymakerequestsforaccommodationsthroughtheirStudentswithDisabilitiescenters,studentsstillmayhavedifficultycompletingthecontentwithinALEKSineitherreasonableortimelyfashion.Certainlythiscouldposedifficultyforsummerbridgestudentswhohavenotyetmatriculatednoridentifiedtotheircampusasrequiringaccommodations.WhiletheALEKSCorporationhasaddressedtheirapproachtoupgradingtheirplatformandcode,stillmoreworkneedstobedonetoinvestigatesolutionsinthisregard.Asreportedbythecampusesinthepilots,theyarenotawareofstudentswhocouldnotcompletetheworkinALEKSduringtheirpilotsduetotheneedforaccommodation.Otherchallengesappeartohamperadoptiononalargerscale—acrossadepartmentorinallclassesofaspecificsubject.ResearchersnotedthatmuchcommunicationabouttheirresearchandfindingsandmuchmoreconversationwillbeneededatindividualcampusestoorganizedepartmentsorprogramsaroundadoptingALEKS.Itisonethingtoconductpilotsandseepositiveresults;itisanothertousethoseresultstoplanatransformationintoaprogramorpracticewithamuchlargerpopulation.Therearemattersofacademicpolicy,departmentalprograms,instructionalintegrationanddesign,facultyfreedomandorganizationalprocessestoconsideranddefineinordertosucceedatscale.And,technicalintegrationintoauniversity’sinfrastructureisacostlyendeavor.That,too,wouldneedcarefulplanningamongnumerousdepartmentssothatuseraccessanddataaccessforreportingandintegratedviewsofstudentsuccessarepossible.Whileeachoftheparticipatingcampusesreportedthereareongoingdiscussionsattheircampusesregardingprogrammatic,policyandorganizationalconcerns,discussionsregardinginstitutionalstrategicplanningwerenotreported.

RECOMMENDATIONSWhilethecampusgroupsformedrecommendationsfortheirowncampusdepartmentprogramsandongoingwork,someoftheserecommendationsmaybeappliedmoregenerallyacrossUCcampuseswhosefacultyconsiderusingALEKS—oranyadaptivelearningtechnology—alongsidetheiracademiccoursesinmathematicsandscience.Theserecommendationsare



notcomprehensivenorshouldtheybeinterpretedtosuggestefficacysystemwide.Rather,theyarearesultofthespecificpilotsandcampusgoalsforthosepilotsthattookplaceinaspecificperiodoftime.Theyareintendedtoinformongoinginvestigationsintotheefficacyofadaptivelearninginthecontextofcampusacademicprogramsintendedtosupportstudentsuccessandimproveinstruction.

Researchandacademics

RECOMMENDATION1

CampusesshouldwidelysharetheirresearchandlearningsabouttheuseofALEKSacrosstheUCsystem.SomeoftheUCcampuseshaveexperienceusingALEKSoveranumberofyearsinvariousmodelsofimplementation.ThecollectedexperiencesofUCresearchfacultywouldhelppromoteunderstandingofpositiveresults,programdesignsandrecommendedpractices.Campusesshouldexplorevariousforumsforsharing,throughsystemwidemeetings,reportstoappropriateadministrativeleadershiporAcademicSenatecommittees,useoftheCaliforniaDigitalLibrary,andcontinuedformalandinformalcollaborations.

RECOMMENDATION2

Participatingcampusesshouldcontinuetheirworkrelatedtothepositivefindingsintheirresearch.Bydoingso,theywillcollectmorestudentdata;alargerdatasetwillrendermorevalidfindings,andcould,overtime,revealadditionalinsights.Further,thecontinuedworkwillhelpclarifyprogramdesignsthatmaybeconsideredforadoptionwithinmoreacademicdisciplinesandmorebroadlyacrossUC,addressingsomeofthechallengesuniversitiesfacewhenconsideringimplementingprogramstoscale.

RECOMMENDATION3

CampusesshoulddefineclearacademicgoalsregardingexpectedstudentacademicachievementinspecificdisciplinesaddressedbysystemssuchasALEKS.TheSTEMdisciplinesappeartobebest-poisedtoundertakethiskindofacademicplanningandprogramdesign,whichisnotinsignificantfromeitheracontentperspectiveorstudent-andfaculty-supportperspective.WhilesystemssuchasALEKSbenefitfromyearsofdevelopmentandcontainagreatdepthandbreadthofcontent,academicdepartmentsandfacultyarechallengedwithmatchingthelearningoutcomesandknowledgesequencesacrosstheircourses(suchasthemathematicssequenceoutlinedbyUCSantaCruz)withthespecificcoursesandknowledgesequencesrepresentedwithintheadaptivelearningsystem.



Organizationalenhancements

RECOMMENDATION4

Campusesshouldidentifyallacademicandadministrativerolesandpointsofcontactrequiredtosupportstudentsandfaculty/TAswhoareutilizingadaptivelearningsolutionssuchasALEKS.Duringthesummer,whenincomingstudentshavenotyetfullyenteredthesystem,communicationacrossdepartmentssuchasstudentaffairs,academicadvisementandtheacademicunitrunningtheprogramneedstooccur.Thegoalsoftheprogramshouldbeclear,andtheintendedcommunicationstostudentsneedtobeknowntotheappropriateindividualswhoseroleitistoworkwithincomingstudents.Duringtheacademicyear,aswell,similarcommunicationandroleclarityishighlyrecommendedandmaybebeneficialtootherstudentsupportgroups.

RECOMMENDATION5

FacultyandTAsshouldalwaysbetrainedandpreparedtouseALEKSanditsreportingsystemsothatappropriateinstructionalinterventionsorindividualcommunicationswithstudentsabouttheirlearningoccursinwaysthatsupportstudentlearningandsuccess.Withthisinplaceaspartofanyadaptivelearningtechnologyimplementation,instructionalmethodscanbeimprovedalongsidestudentresults.

RECOMMENDATION6

Campusesshouldinvestigateacoherentandcost-effectivemodelthatalleviatestheburdenofextracosttostudentsalreadyburdenedwithhighcoursematerialsfees(textbooksandthelike).WhenALEKSisusedalongsideanacademiccourse,itpresentsanadditionalcostwhenatextbookorothercorematerialsfeesareexpected.Insummer,whenALEKSisusedforplacement,itisofconcerntothecampusdepartmentsthatstudentswouldbechargedforthesystem’suse.Someviewitasadepartment’sresponsibilityiftheimplementationinvolvesplacementorresearch.

Technologyandinfrastructure

RECOMMENDATION7

CampusesshouldensurethatallstudentsparticipatinginALEKShaveaccesstoanyaccommodationstheymightneedinordertosuccessfullycompletetheworkinALEKS.Thisposeschallengesinsummerprogramswithincomingfreshmenwhohavenotyetidentifiedas



needingaccommodations.Atpresent,thecampusesarenotawareofstudentswhomaychoosenottoparticipateorwhodon’tcompleteALEKSinthesummerbecauseoftheneedforaccommodations.ClearandfrequentcommunicationwithstudentsaswellasmonitoringtheirusageandprogresswithinALEKSmayhelpidentifystudentswithneeds,butitdoesnotalleviatetheimpactoftheproblem.TheprovidersoftheALEKSsystemwillneedtolooktoevolvingtechnologiestoaddresstheaccessibilityissue.

RECOMMENDATION8

Campusesshouldbegintohavebroaderandfocuseddiscussionsaboutstudentlearningdataandacademicusesofthatdataacrosstheinstitution,andincludeinfrastructureplanninginmeaningfulways.Someuniversitiesreportthesediscussionstobestrategicallyimportantandcriticaltoachievingacademicgoalsfortheinstitution,includingthesuccessfuleducationofitsstudentbody.StudentAffairs,AcademicAdvisement,AcademicResearch,TeachingandLearningandDiversityandEngagementhavevaryinginterestinandneedforsuchdata.Students,too,wouldbenefitfromaccesstothisdata,andincoherentways.Thechallengesofintegratingsystemsandaccesstodataforeffectiveusebyadministration,facultyandstudentsremainalargeconcernformanyuniversitysystems.

CONCLUSIONTheAdaptiveLearningTechnologyPilotstudyatUCusingtheALEKSsysteminmathematicsandchemistrysuggeststhatthesystemcanbehelpfultocertainstudentpopulationswhenspecificprogramobjectivesaredefinedandorganizationalconsiderationsareaddressedbeforethesystemisimplemented.StudentsbenefitwhenthesuccesstargetsareclearlystatedandwhentheyaremotivatedtosucceedandpersistinALEKSuntiltheirtargetsaremet.AtthesametimethepilotstudyrevealedchallengesrelatedtomorewidespreadimplementationofALEKS,includingcosttostudents,accessibilityofthecontenttostudentswithdisabilities,andcommunicationwithincomingstudents,particularlyduringthesummer,viewedasanimportanttimetopreparestudentsandimprovetheirchancesforsuccessduringfreshmanyear.Fromaninstitutionalperspective,mattersofacademicpolicy,programintegration,facultyfreedom,costandorganizationalprocessesrepresentsignificantchallengestointegratingsuchtoolsacrossadepartment.EachcampusresearchteambelievesthatALEKShadapositiveimpactinthecontextitwasappliedattheircampus,butALEKSisnotapanaceaforstudentsuccess.Theresearcherswill



sharetheirresultswidelyandcontinueinvestigationsintousingadaptivelearningtoolssuchasALEKStosupportstudentlearningandsuccess.



REFERENCESEdSurge.(2016).DecodingAdaptive.London:Pearson.TytonPartners.(2016).LearningtoAdapt2.0:TheEvolutionofAdaptiveLearninginHigherEducation.Boston:TytonPartners.IncollaborationwiththeBillandMelindaGatesFoundation.

Oakland, CA 94607-5200 CHANCELLORS Dear … · 2020. 8. 14. · Adaptive Learning Technology Pilot...

Documents

Transcript of Oakland, CA 94607-5200 CHANCELLORS Dear … · 2020. 8. 14. · Adaptive Learning Technology Pilot...