1
The impact of preparation on TSA and
BMATtestresults–aninstitutionalcase
studyatOxfordUniversity
FinalReportMay2019
Jenny Lenkeit, Daniel Caro and Hubert Ertl, Sam Cheadle, Gosia
Turner,AlisonMatthews,SaminaKhanandJo-AnneBaird
withShailenPopatandSamanthaCurle
DepartmentofEducation,UniversityofOxford
UndergraduateAdmissions,UniversityofOxford
StudentDataManagementandAnalysis,UniversityofOxford
Project sponsored by Cambridge English Language Assessment (Admissions TestingService)
OUCEA/19/3
2
Contents
Executivesummary..........................................................................................................................................5
Projectcontext................................................................................................................................................5
Theresearch....................................................................................................................................................5
Keyfindings.....................................................................................................................................................5
Testpreparationmatters......................................................................................................................5
Testpreparationdiffersbyschooltype..........................................................................................5
Resultson theTSAandBMATwere related to student characteristics, potentiallyraisingequityissues................................................................................................................................6
Implications.....................................................................................................................................................6
1. Introduction................................................................................................................................................7
Testpreparation............................................................................................................................................8
2. AnalysisofOxfordadmissiondata.................................................................................................13
2.1 Sample..............................................................................................................................................14
2.2 Variables.........................................................................................................................................14
2.3 Descriptiveanalysis...................................................................................................................16
2.3.1 TSAandBMATperformancebycharacteristicsofapplicants...........................17
2.4 Regressionresults.......................................................................................................................21
2.5 Consistencychecks.....................................................................................................................24
3. AnalysisofBrasenoseCollegesurveydata.................................................................................27
3.1 Amountofpreparation.............................................................................................................27
3.2 Supportfromschools................................................................................................................28
3.3 Useofexampreparationresources....................................................................................28
3.4 Perceptionsofthetests............................................................................................................28
4. InterviewswithOxfordstudents....................................................................................................29
5. Discussionofresultsfromtheexplorationofrelationshipbetweenstudentcharacteristics,testperformanceandtestpreparation(Phase1)...........................................31
Researchquestions...............................................................................................................................31
Limitations................................................................................................................................................33
6. Oxfordsurveydataandadmissiondata2017/2018cohort..............................................33
6.1 Development of survey instrument on Oxford student test preparation andadministration.............................................................................................................................................33
6.2 Data...................................................................................................................................................34
3
6.2.1 Responseratesandanalyticalsample..........................................................................34
6.2.2 Variables.....................................................................................................................................34
6.3 Descriptionoftheanalyticsampleandnon-respondents.........................................36
7. AnalysisofOxfordsurveydataandadmissiondata2017/2018cohort.......................37
7.1 Descriptiveanalysisoftheextentandnatureoftestpreparation........................37
7.1.1 Extentandnatureoftestpreparationbytesttakers..............................................37
7.1.2 Extentandnatureoftestpreparationbystudentbackgroundcharacteristics 44
7.2 Associations between student background characteristics, test preparationandtestperformance...............................................................................................................................50
7.3 Mediatingandmoderatingeffectsofstudentbackgroundcharacteristics.......55
7.3.1 Mediatingeffectsof studentbackgroundcharacteristics for theassociationbetweentestpreparationandtestperformance.....................................................................55
7.3.2 Moderatingeffectsofstudentbackgroundcharacteristicsfortheassociationbetweentestpreparationandtestperformance.....................................................................56
7.4 Associations between time of application, test preparation and testperformance.................................................................................................................................................57
7.5 Learningstrategiesandtestperformance.......................................................................57
7.5.1 Thelearningstrategyscales..............................................................................................57
7.5.2 Associations between learning strategies, student backgroundcharacteristicsandtestperformance...........................................................................................59
8. Discussionandlimitations.................................................................................................................60
Limitations....................................................................................................................................................61
Discussionofkeyfindings......................................................................................................................62
Towhatextentdostudentsprepareforuniversityadmissionstestsandwhatisthenature of this preparation amongst students with different backgroundcharacteristics?.......................................................................................................................................62
What are the connections between student characteristics, including priorqualifications,andtestpreparationandperformance?........................................................62
Isthetimeofapplicationconnectedwithpreparationfor,andbetterperformancein,tests?.....................................................................................................................................................64
Whatlearningstrategiesareconnectedwithbettertestperformance?.......................64
Implications..................................................................................................................................................64
References.........................................................................................................................................................65
Appendices........................................................................................................................................................68
4
AppendixA.TSAandBMATtables.....................................................................................................68
AppendixB....................................................................................................................................................72
AppendixC.BrasenoseCollegesurvey:selectedquotes...........................................................73
B1.Amountofpreparation................................................................................................................73
B.2Supportfromschools...................................................................................................................73
B.3Useofexampreparationresources.......................................................................................74
B.4Perceptionsofthetests...............................................................................................................74
AppendixD.Interviewguide.................................................................................................................76
AppendixE:SurveyInstrument...............................................................................................................79
5
Executivesummary
Projectcontext
TheprojectevaluatedtheimpactoftestpreparationforUniversityadmissionstestsontestperformancewithdatafromOxfordUniversity.Thetopicisofparticularrelevancebecauseanincreasingnumberofundergraduateprogrammes–nationallyintheUKbutalso internationally - require applicants to take an admissions test. The outcomes ofthese tests are considered for admission to University in addition to prior schoolattainment measures such as GCSE scores. However, how students prepare foradmissionstestsandifthispreparationhasanimpactontheirperformanceinthetestisanunder-researchedarea.
Theresearch
This project first explored the relationships between student characteristics and testperformance with Oxford University Admissions data from 2010 to 2015. Second,drawingoninformationfromtheBrasenoseCollegesurveyonhowstudentsprepareforadmissions tests, findings from a literature review and qualitative interviews with asample of students, a survey was developed. It asked for information on how far inadvance students started preparing for the test, the sources they used for testpreparation,thesupporttheyreceivedfromvariousagentsandthestrategiesstudentsemployedtoprepareforthetest.Third,thesurveywasadministeredtostudentswhocommencedtheirstudiesatOxfordUniversityinOctober2017anddatawasanalysedwithmultivariateanalysistechniques.
Keyfindings
Testpreparationmatters.
Nearlyall studentsprepare to take the test to someextentand frequentlyuseawiderange of online resources. The most popular sources were the Oxford UniversityWebsite,theofficialtestproviderwebsiteandTheStudentRoom,withthemajorityofthosewhousedthemfindingthemtobequiteorextremelyhelpful.
The most relevant aspect of test preparation that is positively related to testperformanceistheamountoftimestudentsstartpreparinginadvanceofthetest.
Testpreparationdiffersbyschooltype.
Schools play an important role in supporting students to take and prepare for theadmissionstestbyofferingthemopportunitiestoasksubject-specificquestionsorworkwith a subject teacher, receive general support/mentoring from a teacher andpreparinginworkshopsorclassesfortheadmissionstest.
6
Theamountofsupport thatstudentsreceivetoprepare for thetestsdiffersbyschooltypeandindicatorsofsocio-economicdeprivation.Studentsfromstateschoolsreceivefewer opportunities from schools to prepare for the admissions test. Students fromstate schools and those with an Overall WP Flag also have the least access to otherforms of support: they received feedback on the past test papers they practiced lessoften and were less likely to have talked to former pupils at their school/college orformerorcurrentOxfordstudentswhotookthetest.
ResultsontheTSAandBMATwererelatedtostudentcharacteristics,potentially
raisingequityissues.
A-Level results were highly associated with TSA and BMAT test performance andthereforesuccessofbeingadmittedtoanOxfordUniversityundergraduateprogramme.A-Level results remain positively related to test performance even for students whowereadmittedtothecourseforBMATbutnotforTSA.
Other characteristics being equal, BME applicants, female applicants and those fromdeprivedsocio-economicareasperformed lesswell inboth theTSAandBMAT.Thesecharacteristics were associated with lower performances in TSA and BMATperformanceevenforstudentswhowereadmittedtoacourse.Students’demographicand socio-economic background characteristics explain a larger variance of BMATscoresthanTSAscores.
Startingtoprepareearlyforthetestscounterstherelativedisadvantageassociatedwithsocio-economic deprivation, state school attendance and gender in TSA performancebutnotinBMATperformance.
Implications
The results of this project give evidence on how prior attainment and studentbackground characteristics are related to performance in admissions tests. Thisunderstanding is essential for Universities’ policies on equality. For the University ofOxford, specifically, the resultshavedirect implications for itsoutreachwork, as theyprovide clear evidence that admissions tests and support in preparing for them is ahigherhurdleforapplicantsfromunder-representedgroups–especiallythoseinstateschoolsandfromsocio-economicallydeprivedareas–forwhomadviceandsupportfortaking these tests is essential. In addition, test providers may re-evaluate availablepreparation materials as not all of them are used with similar frequency or areperceivedashelpfulbyallstudents.
7
1. Introduction
This study investigated the preparation of applicants for admissions tests forundergraduate programmes at the University of Oxford. At Oxford, admissionsprocessesofagrowingnumberofundergraduateprogrammesincludeadmissionstestsand the results of the tests influence decisions onwhether applicants are invited forinterviewsand,potentially,whethertheyareofferedaplace.
Admissions tests are becoming more important in admissions processes in selectiveEnglish higher education institutions (HEIs) but preparation is an under-researchedarea (Kirkup et al., 2010).While a 2010BIS report concluded that SAT-style tests do‘not add any additional information, over and above that of GCSEs and A levels (orGCSEsalone),atasignificantlyusefullevel’(Kirkupetal.,2010,4),selectiveuniversitieshave foundadmissions testshelpful indistinguishingbetween largenumbersofhigh-achievingapplicants.
HEIs in theUKoften considerprior attainment atGCSEalongwithother informationfromcandidates for admission touniversity.Withevidenceonprior attainment,HEIsaim to judge the suitability of candidates for the course and their ability to completecourses successfully. Admissions selection criteria are consistent with meta-analysesshowingthatpriorattainmentiscorrelatedwithperformanceinuniversitycourses.Therelationship,however,isnotlarge.Previousacademicperformancehasaneffectsizeof.30 in medicine, explaining for example 23% of variance in overall performance inmedicalschool(Ferguson,James,&Madeley,2002).Forsocialscienceandhumanities,effectsizesofd=.31 tod=.67were found(Peers& Johnston,1994).Particularly,priorattainmentdoesnotdiscriminatewellamonghighlyqualifiedcandidates.ThisisoneofthereasonswhysomeHEIswithhighlyselectivecourseshave introducedadmissionstestsaspartofadmissioncriteria.AnotherreasonisthatmanystudentsapplytoHEIswith different qualifications and, compared to the constantly changing nationalqualificationstakenbyschoolstudents,admissionstestsprovideacommonbenchmarkofacademicability.Andstillanotherreasonisthatadmissionstestsmakeitpossibletocomparestudentsfromverydifferenteducationalbackgrounds.
AtOxford, an increasingnumberof undergraduateprogrammes require applicants totake an admissions test. Of the 248 undergraduate courses offered in 2016, 232requiredapre-interviewtestortestsandonly14courseshavenotestatall.Thereare11separate tests,althoughsomeof these,suchasModernLanguages,aremadeupofmultiple individual papers. In total 85%of all applicants registered for pre-interviewtestsin2015.TheThinkingSkillsAssessment(TSA)iscurrentlyusedforadmissionstonine Oxford undergraduate programmes (Economics and Management, ExperimentalPsychology, Geography, History and Economics, Human Sciences, Philosophy andLinguistics, PPE, Psychology and Linguistics, Psychology and Philosophy). TheBiomedical AdmissionsTest (BMAT) is used forMedicine andBiomedical Sciences at
8
Oxford.TheTSAandBMATaredevelopedandadministeredbyCambridgeAssessmentAdmissionTesting(CAAT)andarisefromthethinkingskillstradition(Fisher,2005).
The thinking skills tradition emerges from work in psychology and philosophy.Although there aredifferent strands, they share the common idea that thinking skillscanbetaughtdirectlyaspartofthecurriculum,andthattheseskills involvecognitiveprocessessuchasarguingacase,decision-making,problemsolving,explainingcauses,evaluating, comparing and contrasting, judging credibility, clarifying and interpretingideasandotherhigher-orderthinkingskills(Fisher,2005).
Extensive research, including meta-analyses and experimental studies, shows thatthinking skills teaching programmes can promote higher educational attainment andtransferable skills across different subject areas and contexts (Adey & Shayer, 1994;Fisher, 2005; Higgins & Hall, 2004). Thinking skills are particularly relevant at thehighereducation level,wherestudentsaregenerallyexpected to think for themselvesandworkindependently.Consistently,researchshowsthatthinkingskillstestsliketheTSAandBMATpredictcoursedegreemarksinuniversityoverandaboveA-levels(e.g.,Black, 2012; Emery, 2009;O’Hare&McGuinness, 2015). That is another reasonHEIsconsiderthesetestsaspartoftheadmissionselectioncriteria.
TheTSAhas two sections. The first is composed of 50multiple-choice questions andstudentshave90minutestocompletethem.Thesecondsectionisawritingtask,whichstudentshave30minutestocomplete.Asthewritingsectionscoresarenotusedaspartof the selection process toOxfordUniversity courses, data from section 2 of the TSAwere not included in this research. TSA section 1 addresses “problem-solving skills,including numerical and spatial reasoning. Critical thinking skills, includingunderstandingargumentandreasoningusingeverydaylanguage.”1
The BMAT is composed of three sections; candidates are allowed 60minutes for thefirst sectionand30minutes for sections2 and3,making the examination2hours induration.The firstsection iscomposedof35multiple-choicequestionsandaddresses“generic skills in problem solving, understanding arguments and data analysis andinference.” Section2 contains27multiple-choicequestions,which test “The ability toapplyscientificknowledgetypicallycoveredinschoosScienceandMathematicsbytheageof16(forexample,GCSEintheUKandIGCSEinternationally).”Thefinalsectionisawriting task inwhich candidates choose fromoneof threequestionsdesigned to test“Theabilitytoselect,developandorganiseideas,andtocommunicatetheminwriting,conciselyandeffectively.”2
Testpreparation
Testpreparationhasbeenthesubjectofconsiderableresearchmorebroadlyandwasintensively researched in the 1980s in the US, in relation to the SATs (university
1 https://www.admissionstesting.org/for-test-takers/thinking-skills-assessment/tsa-oxford/about-tsa-oxford/
2https://www.admissionstesting.org/for-institutions/about-our-tests/biomedical-admissions-test/
9
entrance tests).A general conclusion from thisworkwas that no test candistinguishbetween aptitude and performance: they assess performance (Powers, 1985; Slack&Porter, 1980). It hadpreviously been argued that some tests could assess underlyingpotential,evenifastudentwasnotcurrentlyperformingatahighlevel.Arecentcaseinwhich this notion was disabused was Durham University’s Centre for EducationalMeasurement’s11+test,asresearchindicatedthattestpreparationhadaneffectuponoutcomes.3 Thisrecentexampledemonstratesthatdespitetheearlierresearchinthisarea,perceptionsthatapureformofpotentialcanbeassessedpersistinsomequarters.
Studies on test preparation, cramming and drilling have found increases in students’scores(e.g.Bangert-Drowns,Kulik,andKulik1983;BuntingandMooney2001;MessickandJungeblut1981;Powers1986;Sturman2003).Accesstotestpreparationmaterials,similarityofthematerialstotheactualassessments,thenumberofpracticetests,timeontaskandhigherabilitystudentshaveallbeenassociatedwithhigherperformances(Kulik, Bangert-Drowns & Kulik, 1984; Powers and Swinton, 1984; Griffin, Carless &Wilson,2013).Analyticalandquantitativetasksshowhighergainsfromrepeatedtest-takingthandoverbaltasks(Hausknecht,Halpert,DiPaolo&MoriartyGerrard,2007).These studies have typically been conducted for tests that do not have an associatedcurriculum,justliketheBMATandtheTSA(unlike,forexample,A-levels).Section2ofthe BMAT does assess scientific knowledge, though, so may operate more likecurriculum-relatedassessmentsforthatproportionofthetest.
Fourexplanations forstudents’ scoregainsexist in the literature (Arendasy,Sommer,Guiterrez-Lobos&Punter,2016)(Table1.1).Thefirststatesthattestfamiliarityinitselfis a cause of increased scores, even if this gain is not relevant to the construct beingassessed.Suchgainsdonotimpactuponstudents’abilities,onlyontheirperformances.Asecondmodelpositsthatdrillingontest-specifictechniques(suchashowtoeliminateoptionalresponseswithoutadeepunderstandingof thequestion)wouldhavesimilareffects to model 1 in terms of impact upon performance and ability. A third modelinvolvesteachingconstruct-relevantmaterialtostudentssotheyperformbetterontestquestionsandthisimpactstheirabilityinadomain-specificmanner.Afourthmodelisthatofgeneralteachingandlearningstrategieswhichwouldbeexpectedtobothimpactuponperformanceandproducegeneralisedimprovementsinstudents’abilities.
Table1.1:Testpreparationmodels
Model Construct
relevance
Cause Impacton
performance
Impacton
ability
1 No Testfamiliarity Yes No
2 No Testcoachingtipsandtricks
Yes No
3https://www.theguardian.com/education/2016/sep/12/tutor-11plus-test-grammar-schools-disadvantaged-pupils
10
3 Yes Itemsolutionstrategies
Yes Yes:test-ordomain-specific
4 Yes Teaching Yes Yes
InresearchonamedicalentranceexaminationattheUniversityofVienna,Arendasyetal.(2016)demonstratedthatModel3bestexplainedtheirfindings.Testpreparation,intheir sample of students, had test-specific and domain-specific effects but in keepingwith other studies, the gains from test preparation did not transfer to generalintelligence(g)oreventogeneralnaturalscienceknowledge.Itwaspreviouslyknownthat practising test items did not increase general intelligence (Arendasy & Sommer,2013),butthefindingwithrespecttodomain-specificgeneralisationwasnew.
Preparation for thinking skills admissions tests like the TSA and BMAT is an under-researchedarea.There is evidence thatGCSEandA-levels results correlatepositivelywithSAT-styletests(Kirkupetal.,2010),butlittleisknownaboutpreparationfortheTSAandBMATtests.InarecentstudyGallacher,McElweeandCheung(2017)studiedpreparationstrategiesfortheBMATwithsurveydatafromaconveniencesample.TheauthorsfoundthatthegreatmajorityofcandidatesusetheBMATwebsiteforaccessingpreparationmaterials. Regression analysis results suggested that attempting practicepapersundertimedconditionswasaneffectivestrategyforhighertestscoresacrossallBMAT sections. There was no indication that extra help from schools, commercialcoursesorcomingfromanIndependentschoolwaspositivelyrelatedtohigherBMATscoresonceotherfactorswerecontrolledfor.
Conceptually, enhancing thinking skills through educational programmes and studies(GCSE,A-Levels,thinkingskillscourses)shouldbedistinguishedfrompreparationsthataimto improve test-takingskills.While the formerwouldsupportskillsnecessary forstudents’ study, the latter would merely develop strategies to enhance test-takingskills.4
ThisstudywillinvestigatepreparationfortheTSAandBMATwithdatafromon-courseUniversityofOxfordstudents.Thefollowingresearchquestionswillbeaddressed.
Researchquestions
Theguidingresearchquestionthisstudyseekstoansweris:
- What is the impact of preparation for university admissions tests on testperformance?
Thisquestioncomprisesanumberofsub-questions:
4 This project did not aim to disentangle the two concepts of preparation for thinking skills tests(theoreticallyorempirically)andinstrumentshavenotbeendesignedaccordingly.Whereappropriateadistinctionisattemptedinthediscussionofresults.
11
- Towhatextentdostudentsprepareforuniversityadmissionstestsandwhatisthe nature of this preparation amongst students with different backgroundcharacteristics?
- Whataretheconnectionsbetweenstudentcharacteristicsandtestpreparationandperformance?
- Whichpriorqualificationsandsubjectsareconnectedwithtestpreparationandbettertestperformance?
- Is the time of application connected with preparation for, and betterperformancein,tests?
- Whatamount, timingandnatureofpreparationare connectedwithbetter testperformance?
- Istheinfluenceoftestpreparationontestperformancemediatedormoderatedbystudentcharacteristicsandbackground?
- What learningstrategiesandsourcesofsupportareconnectedwithbetter testperformance?
Studyphases
Theresearchquestionsareaddressedintwoconnectedphases.
Phase1:Explorationofstudentcharacteristics, testperformanceandadmissionstestspreparation
1. Exploration of the relationship between student characteristics and testperformanceMultiple regression analysis examined the association between applicantcharacteristics and performance in TSA andBMAT tests for students from theUK. A number of characteristicswere considered, such as BME status, gender,priorattainment(GCSEandA-Levelresults),andschooltypeorigin.One of the underlying questions was whether certain GCSEs and A-Levelsincreasethelikelihoodofgoodtestresultsornot,i.e.whichqualificationsseemto provide a good preparation for these two admissions tests. This analysisallowedustodeterminewhethertestresultdifferencesthatseemtobelinkedtocandidatebackgroundpersistwhenGCSEandA-Levelattainmentwascontrolledfor. The work of student-level data also provided insights into connectionsbetween test results and other background variables (e.g., type of schooling,POLARquintile).
2. AdditionalanalysisbasedonstudentsurveyresponsesInaseparatesurvey,collectedbyBrasenoseCollegeatOxfordoverthelastthreeyears, studentswere asked in an open-ended questionwhether and how they
12
preparedforadmissionstests,includingwhethertestpreparationwasprovidedby students’ schools orbasedon students’ own initiative.Thematic analysis ofthis data allowed us to identify potential connections between amount andnatureofpreparation,typeofschoolingandotherbackgroundvariables.
3. Application date as a proxy for the amount of test preparationPreliminary analysis of with Brasenose College admissions data has revealedthat thedateof application is significantly related to admissions success.Aftercontrolling for GCSE performance, applicants who apply early have a higherchanceofobtaininganofferandgainingaplace.Onepossibleexplanationisthatearlyapplyingapplicantshavemoretimeandsupporttoprepareforadmissionstests and are thereforemore likely to achieve higher test scores and a higherproportion of offers. The study will explore this and alternative explanationsfurther, using the university wide dataset, and directly examining therelationship between application date and TSA and BMAT performance.
4. Qualitative investigation of the impact of preparation on test experience andresultsBasedoninsightsgainedin1–3andpreviousresearchontestpreparationthisstudy generates qualitative data to explore the usefulness of differentpreparationmaterials andmodes. This was achieved by conducting follow-upinterviewswithasampleofsurveyrespondentswhohaveexperienceddifferentlevels and types of preparation. The interviews allowed us to explore in aqualitativemannersomeofthewidely-heldassumptionsregardingpreparationforuniversityadmissionstests,forinstancethatmaterialrecommendedbytestdevelopers is the focus of students’ preparations. Most importantly theinterviewsinformedthedevelopmentofthesurveyinstrumentinPhase2.
Phase2:Analysisofadmissionstestspreparationandtestperformance
Basedonthe findingsofPhase1asurveywasdevelopedandpiloted(AppendixE) tonewlyadmittedundergraduatestudentswhocommencedtheirstudiesinOctober2017.Thesurveyincludesitemsinthefollowingareas:
• Howearlystudentsstartedpreparingfortheadmissionstests• Howmuchstudentspreparedfortheadmissionstests• Useofpasttestpapersandwaysofpracticingwiththem• Sourcesofsupportinpreparationprocess• Usefulnessofdifferenttypesofpreparationmaterials
13
Additionally, the survey included the scales on student learning strategies (e.g.,memorisation, elaboration, and control strategies), widely used in the context of thePISAstudy(Marsh,Haug,Artelt&Baumert,2006;Säälik,Nissinen&Malin,2015)andadaptedforhigh-stakestests inpreviousresearchbytheOxfordUniversityCentreforEducational Assessment (Baird, Hopfenbeck, Elwood, Caro & Ahmed, 2015). Thequestionnaire includes various types of questions, ranging from closed questions, toLikert scale items and open-ended questions. The survey is administered by OxfordUniversity’sStudentDataManagementandAnalysisteamviaOnlineSurveys(formerlyBOS) and piloted with a substantial sample of incoming students commencing theirstudiesinOctober2017.Thesurveyislinkedtostudentrecords,whichallowsanalysissimilar to thatundertaken for smaller samplesof students inPhase1, but employingmuchmoredetaileddataontestpreparationandsophisticatedmethods.
Next to descriptive statistics, multivariate analysis is employed. Factor analysisevaluatedthestructureofstudentresponses(e.g.,learningstrategiesscale).Regressionanalysis examined the association between student preparation variables and testresultswhilst taking intoaccountdifferentialeffects forbackgroundcharacteristicsofstudents (e.g., BME status and gender). The latter is in linewith findings from large-scale assessment suggesting that preparation strategies vary by the student's genderandsocio-economicstatus(e.g.,OECD,2010).Thedatafromthesurveywillalsobeusedto explorewhether the influenceof testpreparationon testperformance ismediated(explained) or moderated (differentially affected) by student characteristics andbackground.
Aimandstructureofthereport
ThisFinalReportpresentstheresultsofPhase1andPhase2.Itisstructuredasfollows:
Sections 2 to 5 discuss the findings from Phase 1. Section 2 reports analysis of theassociation between TSA/BMAT scores and characteristics of students (e.g., GCSE/A-levelsandsocio-economicbackground)usinglarge-scaleOxfordadmissiondataacrossfiveyearsanddifferentcourses.Section3presentsopen-endedresponsesofstudentsatBrasenoseCollegeonhowtheyprepared foradmissionexams.Section4presents theprocedure of developing and administering interviews with 10 candidates. Section 5discussesmainresultsofPhase1.
Sections6to8reportonthefindingsfromPhase2.Section6outlinesthedevelopmentof the survey, the sample and used variables for the 2017/2018 cohort. Section 7presents the analysis and results including descriptive statistics and associationsbetween test preparation, student background characteristics and test performance.Section8discussedtheoverallfindingsoftheproject.
2. AnalysisofOxfordadmissiondataThis sectionpresents resultsof analysisofOxfordadmissiondataon the relationshipbetween test scores on admissions tests (i.e., TSA and BMAT) and applicants’
14
characteristics. Results of descriptive analysis and regression analysis are presented.Twosamplesareconsideredforanalysis,onefortheTSAtestandanotherfortheBMATtest.
2.1 Sample
The total TSA sample consisted of 9,886 home (UK domiciled) applicants across the2010 – 2015 Universities and Colleges Admissions Service (UCAS) cycles. The totalBMATsampleconsistedof7,175homeapplicantsacrossthe2009–2015UCAScycles.Formultivariateanalysis,thesamplewasfurtherrestrictedtoapplicantswithGCSE/A-level results and relevant socio-economic characteristics (available by UK postcode),resultinginafinalTSAsamplesizeof7,032andafinalBMATsamplesizeof5,301.TheTSA sample included applicants for the following courses: Economics&Management(n=2,169),ExperimentalPsychology(n=741),Geography(n=674),‘Politics,Philosophyand Economics’ (n=3,168) and ‘Psychology, Philosophy and Linguistics’ (n=280). TheBMAT sample included applicants to the following courses: Biomedical Sciences(n=190)andMedicine(n=5,111).5
2.2 Variables
ForbothTSA6andBMATscorestheoverallscoreswereusedintheanalysis.TSAandBMATscoreshavebeenstandardisedbyZ-scoring.NotethatscoreswerestandardisedwithineachUCAScycleandcoursegroupseparately.Thiswascarriedoutfortwomainreasons. (1) Standardisationwithin eachUCAS cycle aims to account for year to yearfluctuationsinthedifficultyofthetest.Notethatwithincyclestandardisationwasalsoapplied to GCSE and A-level attainment measures. (2) Due to the limitations in thecentral admissions database (ADSS), scores supplied by Cambridge Assessment havebeen recorded for all courses taking TSA as well as medicine applicants taking theBMATacrosstheUCAScycles.InBiomedicalSciences,BMATscoreshavebeenrecordedin a standardised format across the UCAS cycles. This left no other option than tostandardisealltestscoreswithineachcourseseparately.7 Further,BMATscoreswereweightedaccordingtoOxford’sselectionprocedures.8
The following variables were considered as statistical predictors of TSA and BMATscores.
5Note:NumbersdeviatefromtheInterimReport,astestscoresfortheBiomedicalSciencescoursehavebeen incorrectly assigned to TSA due to ADSS test storage issues. Subsequent analysis has beencorrectedforthiserrorinthepresentreport.
6FortheTSAonlythemultiplechoicesectionsassessingcriticalthinkingandproblemsolvingskillsareincludedintheoverallscore.
7Note:StandardisationofBMATtestscoresacrossMedicineandBiomedicalSciencescoursesdoes,dueto thedifferent formats inwhich thescoreswererecorded,noteliminatecomparability issuesof thescores. However, Appendix B compares the analysis conducted in Table 2.3 for BMAT with thecombinedMedicineandBiomedicalSciencescoreswithresultsforBMATscoresofMedicinestudentsonly.DifferencesinresultsarenegligibleandsubsequentanalysisforBMATisconductedandreportedwiththecombinedstandardisedscoresfromMedicineandBiomedicalSciences.
8https://www.medsci.ox.ac.uk/study/medicine/pre-clinical/statistics
15
• School type: Dummy variables indicate applicant’s school type (state,independent,orother).Stateandotherschoolsarecomparedwithindependentschoolsinregressionmodels.The‘other’categoryismainlyoverseasschools.
• BME:Dummyvariablesindicateblackandminorityethnic(BME)status(yes,no,ordon’tknow/refuse).
• Application time: The number of days between 1st September and theapplication date (UCAS form received by Oxford). The reference date of 1stSeptember was chosen because this is close to the application deadline. Theprecise reference date does not affect the statistical results. Smaller valuesreflectearlyappliers.Resultingregressioncoefficientsweremultipliedby10foreaseofinterpretation.
• ACORNFLAG:Dummyvariable indicates socio-economic adversity according toUKpostcode.Whereacandidate’spostcodefallsintoACORNgroupsfourorfive('FinanciallyStretched'and'UrbanAdversity')theapplicationwasflagged.
• POLAR3FLAG:Dummyvariable indicates lowparticipation inhigher educationaccordingtoUKpostcode.ThePOLARclassificationplacesregionsintoquintilesbased on the rate of young participation inHigher Education. Applicants fromthelowesttwoquintileswereflagged.
• PRE16SCHOOLFLAG: Dummy variable indicates performance of applicant’sschoolatGCSEisbelownationalaverage.
• POST16SCHOOLFLAG: Dummy variable indicates performance of applicant’sschoolatA-levelisbelownationalaverage.
• WPOVERALLFLAG(wideningparticipation):Dummyvariableindicateswhethertheapplicanthasobtainedoneofmoreschoolflags(preorpost16)andoneormoregeodemographicflags(ACORNorPOLAR3)orhaspreviouslybeenincare.
• GCSEA*s:ThenumberofA* grades (only including applicantswith5ormoreGCSEs).GCSEscoresweretransformedintoz-scores.
• A-levelA/A*s:ThenumberofAorA*grades. A-levelsscoresweretransformedinto z-scores. (Note that A-levels are taken after the TSA or BMAT for moststudents,butwere included in themodelasaproxy forapplicants’ concurrentattainmentlevels.)
• Schoolhistoryofapplication:Itisequaltotherawnumberofapplicantsbetween2011and2016,perschool.ItreflectstheschoolhistoryofapplyingtoOxford.
• UcasCycle:This is theapplication cycle inwhichanapplicationwasmade, andrefers to the year of entry for the majority of those applying. For example,applications made in the 2014 Universities and Colleges Admissions Service(UCAS) Cycle were made by 15 October 2013, for entry in October 2014(majority),ordeferredentryinOctober2015(minority).
• A-level subjects: Dummy variables indicate applicant’s A-levels choices (Art,Biology, Business Studies, Chemistry, Computing, Economics, English, EnglishLanguage, English Literature, Further Mathematics, Further Additional,Geography, History, Languages, Law, Mathematics, Philosophy, Physics,Psychology,ReligiousEducation,ScienceOther,SocialScience,andOther).
16
2.3 Descriptiveanalysis
Table 2.1 and 2.2 report descriptive statistics for selected variables in the TSA andBMATsamplesanalysed.Percentageofapplicantsandmeantestscoresarereportedbyschooltype.
Table 2.1 and 2.2 show that the percentage of applicants from disadvantaged areasaccording to ACORNFLAG and POLAR3FLAG is relatively low in independent schools(around4%forTSAapplicantandaround7%forBMATapplicants),buthigherinstateandotherschools(around14%inTSAapplicantandbetween16%and20%inBMATapplicants). In termsof educational attainment, candidates from independent schoolsperformbetteratGCSEandA-levelsthancandidatesfromstateandotherschools.GCSEandA-levelperformanceisparticularlylowforcandidatesfromotherschools.
Table2.1:TSAanalyticsample:descriptivestatistics(%ofapplicantsandmean
scores).Home(UKdomiciled)applicantsonly(N=9,886).
School,type Independent Other StatePercentagesFemale 38% 34% 42%BME 22% 30% 24%ACORNFlag 3% 14% 13%POLAR3Flag 5% 15% 14%Pre16SchoolFlag 2% 7% 12%Post16SchoolFlag 1% 7% 29%OverallWPFlag 1% 6% 10%MeansGCSE(z-score) 0.44 -0.68 -0.18A-levels(z-score) 0.14 -0.33 -0.13TSA* 62.31 59.31 60.45TSA(z-score) 0.36 -0.10 0.11
*Excludes recorded TSA scores of less than 10 to exclude outliers which were considered to beunexplainedmissingdata.Note:HomeapplicantswhohavetakentheTSAonly(2010–2015UCAScycles).Forcomparability,GCSEandA-levelsscoreswerescaled(i.e.,z-cores)tohaveameanofzeroandastandarddeviationofone.
Table2.2:BMATanalyticsample:descriptivestatistics(%ofapplicantsandmean
scores).Home(UKdomiciled)applicantsonly(N=7,175).
School,type Independent Other StatePercentagesFemale 52% 60% 54%BME 35% 44% 38%ACORNFlag 6% 20% 16%POLAR3Flag 8% 16% 17%
17
Pre16SchoolFlag 3% 6% 15%Post16SchoolFlag 1% 7% 29%OverallWPFlag 1% 8% 13%MeansGCSE(z-score) 0.38 -0.85 0.00A-levels(z-score) 0.23 -0.54 0.00BMAT* 56.85 49.09 54.13BMAT(z-score) 0.35 -0.39 0.09
*ExcludesrecordedBMATscoresoflessthan10.Note:Homeapplicantswhohave taken theBMATonly (2009–2015UCAS cycles). For comparability,GCSEandA-levelsscoreswerescaled(i.e.,z-cores) tohaveameanofzeroandastandarddeviationofone.
2.3.1 TSAandBMATperformancebycharacteristicsofapplicantsAs with GCSE and A-levels results, Table 2.1 and 2.2 suggest that applicants fromindependent schools perform slightly better in the TSA and BMAT compared toapplicants from state schools and other schools.Note thatmature students aremorelikelytobeincludedinthe‘other’category,withapproximately17%beingaged19oroverinthisgroup,comparedwith5%intheoveralldataset.
Wewantedtoevaluatewhetherperformancegapsinadmissionstestsacrossdifferentschools were explained by prior attainment of candidates. Figure 2.1 displays TSAperformance by school type for candidates with equivalent GCSE results, that is,belongingtothesameGCSEattainmentquartile(seealsoTableA1inAppendixA).Thex-axis locates candidates according to GCSE quartiles and the y-axis depicts TSAstandardisedscores.
When we look at candidates with equivalent GCSE results, evidence of a TSAperformance gap favouring independent schools can only partly be confirmed.CandidatesofindependentschoolsandstateschoolsperformfairlysimilarlyintheTSAtest, particularly at higher GCSE levels. There is only weak evidence that candidatesfrom independent schools perform better than candidates from state schools at thelowestGCSEquartile.ANOVAanalysisconfirmedthatthedifferenceissignificant(F(1,9068) = 6.06,p = 0.014). Still, it is the attainment in GCSE that explainsmost of thevariationinTSAscoresbetweenstateandindependentschools.
These results are explored further in regression analysis controlling for morecharacteristicsinthenextsection.
Figure 2.1: AverageTSA standardised scores byGCSE quartiles and school type
(homeapplicantsonly)
18
Figure 2.2 presents the TSA performance for female and male candidates by GCSEquartile (see also Table A2 in Appendix A). Results consistently show that malecandidates performed better than female candidates in TSA across GCSE attainmentlevels.ANOVAanalysisconfirmedthatthedifferenceissignificant(F(1,9293)=340.02,p < 0.001). The gender gap is relatively stable across students with different GCSElevels.
Figure2.3depictsthegapinTSAoutcomesforBMEandWhitecandidatesacrossGCSEquartiles(seealsoTableA3inAppendixA).CandidateswithBMEbackgroundperformworse inTSA thanWhitecandidates.ANOVAanalysisconfirmedthat thedifference issignificant (F (1, 9289) = 130.30, p < 0.001). The TSA score gap related to BMEbackground appears to be larger for students with lower GCSE scores, in the firstquartile(F(3,9289)=6.39,p<0.001).
Figure2.2:AverageTSAstandardisedscoresbyGCSEquartilesandgender(home
applicants only)
-1.0
-0.5
0.0
0.5
1.0
Q1 Q2 Q3 Q4
Sta
nd
ard
ise
d T
SA
te
st
sco
re
GCSE attainment quartile
Independent State
-1.0
-0.5
0.0
0.5
1.0
Q1 Q2 Q3 Q4
Sta
nd
ard
ise
d T
SA
te
st
sco
re
GCSE attainment quartile
Female Male
19
Figure 2.3: Average TSA standardised scores by GCSE quartiles and BME
background
Similarly,Figures2.4to2.6displayperformancegapsinBMATscoresacrossdifferenttypes of school, student gender and BME status by prior attainment of candidates.Candidates from independent schools score significantlyhigher than those fromstateschools(F(1,6595)=48.50,p<0.001).ThedifferenceisstableacrossGCSEquartiles(F(3,6595)=0.96,p=0.41.N.S.)(Figure2.4).
Figure 2.5 presents the BMAT performance for female andmale candidates by GCSEquartile.ResultsconsistentlyshowthatmalecandidatesperformedsignificantlybetterthanfemalecandidatesinBMATacrossGCSEattainmentlevels(F(1,6745)=324.48,p<0.001).ThegendergapisstrongeratlowerandhigherGCSElevels(F(3,6745)=2.77,p<0.05).
Figure 2.6 presents the BMAT performance for BME status by GCSE quartile. ResultsconsistentlyshowthatcandidateswithaBMEstatusperformedsignificantlylowerthancandidateswithoutBMEstatus inBMATacrossGCSEattainment levels (F(1,6423)=182.90,p < 0.001). The gap is stronger at lowerGCSE levels (F (3, 6423) = 9.83,p <0.001).
-1.0
-0.5
0.0
0.5
1.0
Q1 Q2 Q3 Q4
Sta
nd
ard
ise
d T
SA
te
st
sco
re
GCSE attainment quartile
White BME
20
Figure2.4:AverageBMATstandardisedscoresbyGCSEquartilesandschooltype
Figure2.5:AverageBMATstandardisedscoresbyGCSEquartilesandgender
Figure2.6:AverageBMATstandardisedscoresbyGCSEquartilesandBMEstatus
-1.0
-0.5
0.0
0.5
1.0
Q1 Q2 Q3 Q4
Sta
nd
ard
ise
d B
MA
T t
est
sco
re
GCSE attainment quartile
Independent State
-1.0
-0.5
0.0
0.5
1.0
Q1 Q2 Q3 Q4
Sta
nd
ard
ise
d B
MA
T t
est
sco
re
GCSE attainment quartile
Female Male
21
2.4 Regressionresults
Linear regression models were used to investigate statistical prediction of TSA andBMAT scores. Unstandardised regression coefficients and associated standard errorsarereportedinTable2.3.
Table 2.3: Regression of TSA and BMAT scores on student and school
characteristics
VariablesTSA BMAT
Coef. SE Sig. Coef. SE Sig.Intercept 16.70 11.79 -63.25 11.03 ***Gender(Female=1) -0.21 0.02 *** -0.25 0.02 ***Schooltype(Other=1) -0.13 0.23 0.43 0.30 Schooltype(State=1) 0.07 0.02 ** -0.04 0.02
BME(don'tknow/refuse=1) -0.18 0.04 *** -0.29 0.05 ***BME(Yes=1) -0.39 0.02 *** -0.27 0.02 ***ACORNFlag -0.10 0.04 * -0.11 0.04 **POLAR3Flag 0.04 0.03 0.01 0.03 Pre16SchoolFlag -0.05 0.04 -0.05 0.03 Post16SchoolFlag -0.09 0.03 ** -0.05 0.03 OverallWPFlag9 -0.06 0.05 -0.13 0.05 **Applicationtime -0.43 0.19 * -0.45 0.19 *GCSEA*s(zscored) 0.23 0.01 *** 0.19 0.01 ***A-levelA/A*s(zscored) 0.20 0.02 *** 0.21 0.01 ***Schoolhistoryofapplication 0.04 0.01 *** 0.01 0.01 *UcasCycle -0.01 0.01 0.03 0.01 ***
9TSA:467applicantswereWPflaggedoutofasampleof7,032.BMAT:479applicantswereWPflaggedoutofasampleof5,301.
-1.0
-0.5
0.0
0.5
1.0
Q1 Q2 Q3 Q4
Sta
nd
ard
ise
d B
MA
T t
est
sco
re
GCSE attainment quartile
White BME
22
A-LevelArt 0.02 0.05 -0.01 0.06 A-LevelBiology 0.01 0.03 0.06 0.06 A-LevelBusinessStudies 0.03 0.06 0.17 0.28 A-LevelChemistry -0.04 0.03 0.31 0.14 *A-LevelComputing 0.18 0.17 0.26 0.33 A-LevelEconomics 0.10 0.03 *** 0.24 0.07 **A-LevelEnglish -0.01 0.07 0.22 0.14 A-LevelEnglishLanguage -0.08 0.06 0.04 0.17 A-LevelEnglishLiterature -0.07 0.03 * -0.07 0.05 A-LevelFurtherMathematics 0.17 0.03 *** 0.27 0.04 ***A-LevelFurtherAdditional 0.00 0.00 * 0.00 0.00 A-LevelGeography -0.11 0.03 ** -0.11 0.06 A-LevelHistory -0.06 0.03 * -0.01 0.04 A-LevelLanguages -0.09 0.03 ** -0.05 0.03 A-LevelLaw -0.08 0.10 -0.63 0.37 A-LevelMathematics 0.22 0.03 *** 0.12 0.03 ***A-LevelOther 0.13 0.03 *** 0.09 0.06 *A-LevelPhilosophy 0.29 0.05 *** 0.25 0.15 *A-LevelPhysics 0.16 0.03 *** 0.11 0.03 ***A-LevelPsychology 0.11 0.04 ** 0.07 0.06 A-LevelReligiousEducation 0.10 0.04 ** 0.17 0.07 *A-LevelScienceOther 0.07 0.15 -0.31 0.20 A-LevelSocialScience 0.07 0.08 -0.14 0.21
p<0.05(*),p<0.01(**)andp<0.001(***)
Thecombinedsetofpredictorsexplains27%ofthevarianceoftheTSAscoreandtheBMAT score. Overall, results of the TSA model with the full set of control variablesindicated:
Significantlyhigherperformancefor,
• Stateschoolsapplicants(relativetoindependentschoolapplicants)• ApplicantsfromschoolswithahistoryofapplyingtoOxford• ApplicantswithhigherA-LevelandGCSEscores• Applicantswhoappliedearlier• Applicants who chose the following A-level courses: Economics, Further
Mathematics,Mathematics,Philosophy,Physics,Psychology,ReligiousEducationandOther.
Andsignificantlylowerperformancefor,
• BMEapplicants(relativetoWhiteapplicants)• Femaleapplicants• ACORNflaggedapplicants
23
• Post16schoolflaggedapplicants• A-levelcourses:EnglishLiterature,Geography,History,Languages
Overall,resultsoftheBMATmodelindicated:
Significantlyhigherperformancefor,
• ApplicantsfromschoolswithahistoryofapplyingtoOxford• ApplicantswithhigherA-levelandGCSEscores• Applicantswhoappliedearlier• ApplicantswhoappliedinalaterUCAScycle• A-level subjects: Chemistry, Economics, Mathematics, Further Mathematics,
Philosophy,Physics,andReligiousEducationandOther.
Significantlylowerperformancefor,
• Femaleapplicants• BMEapplicants(relativetoWhiteapplicants)• ACORNflaggedapplicants• OverallWPapplicants(relativetonon-flaggedapplicants)
In descriptive analysis we had shown that applicants from independent schoolsperformedbetterthancandidatesfromstateschoolsintheTSAandBMATtests.FortheTSA,wehad seen that this differencewas significant, however, that the gapbetweencandidates from independent and state schools was largely explained by GCSE priorattainment. Additionally, regression results in Table 2.3 indicate that applicants fromstateschoolsperformslightlybetterinTSAthancandidatesfromindependentschoolsonce the full set of socio-demographic and prior attainment characteristics areconsidered. The table also shows that there is no significant difference betweencandidates from independent or state schools in BMAT, once the same set of socio-demographicandpriorattainmentcharacteristicsareconsidered.
That is, applicants from state schools perform better (TSA) or equal to (BMAT)applicants from independent schools with equivalent GCSE/A-levels results, socio-economicbackground,andothercharacteristicsconsidered inmodels (seeTable2.3).Whilsta reducednumberof candidates in stateand independent schoolsmight shareprior attainment and socio-economic background characteristics included in models,whatregressionresultsdocommunicateclearlyisthatlowerTSA/BMATperformanceof candidates from state schools is not an attribute of state schools, but rather isexplained by prior attainment and socio-economic characteristics of candidatesattendingindependentandstateschools.
Consistently with results in Figures 2.2 and 2.5, regression models show that malecandidates performbetter than female candidates.Also, consistentlywith Figures 2.3and 2.6, regression models indicate that BME candidates perform worse thanWhite
24
applicants.TheTSA/BMATgapinperformancebetweenBMEandWhitecandidates isnotexplainedbytheirpriorattainmentorsocio-economicbackground.ThusthereareothercharacteristicsnotincludedinregressionmodelsexplainingtheperformancegapbetweenBMEandWhitecandidates.
Regression results also show that early applicants perform better in TSA/BMATadmissionsteststhanthosewhoapplylater.Theapplicationtimevariablereflectsdaysbetween1stSeptemberandtheapplicationdate,withsmallervaluesindicatingearlierapplications.Bi-variatecorrelationswith timeofapplicationarenegative forboth theTSAandBMATresults (Pearson’s r= -0.12), indicating thatearlierapplicantsperformbetter in the test, consistentlywith regressionresults.Furtheranalysis in thisprojectwill aim to explain the mechanisms for the positive effect of applying early onTSA/BMATresults.
2.5 Consistencychecks
As the survey in Phase 2 was administered to on-course Oxford students only,consistency checkswereperformed to evaluate inconsistencies in regression findingsfor1)allapplicantsand2)on-coursestudentpools.Regressionanalysiswerererunforon-courseTSA(N=1,478)andBMAT(N=835)studentsonly.ResultsareshowninTable2.4.
Table 2.4: Regression of TSA and BMAT scores on student and school
characteristics,includingon-coursestudentsonly
VariablesTSA BMAT
Coef. SE Sig. Coef. SE Sig.Intercept 2.64 24.54 59.71 24.10 *Gender(Female=1) -0.26 0.04 *** -0.24 0.05 ***Schooltype(Other=1) 0.36 0.45 0.62 0.74 Schooltype(State=1) 0.03 0.05 -0.05 0.05
BME(don'tknow/refuse=1) 0.10 0.14 -0.16 0.17 BME(Yes=1) -0.22 0.05 *** 0.05 0.05 ACORNFlag -0.21 0.09 * -0.23 0.11 *POLAR3Flag 0.07 0.07 0.03 0.08 Pre16SchoolFlag -0.04 0.09 -0.10 0.08 Post16SchoolFlag -0.15 0.06 * -0.19 0.07 **OverallWPFlag -0.01 0.11 -0.11 0.13 Applicationtime 0.22 0.36 -0.01 0.36 GCSEA*s(zscored) 0.03 0.03 -0.11 0.04 **A-levelA/A*s(zscored) 0.02 0.05 0.15 0.08 Schoolhistoryofapplication 0.02 0.02 0.03 0.02 UcasCycle 0.00 0.01 -0.03 0.01 *A-LevelArt -0.05 0.11 0.01 0.11
25
A-LevelBiology -0.04 0.08 -0.11 0.15 A-LevelBusinessStudies 0.22 0.20 0.13 0.63 A-LevelChemistry 0.04 0.08 0.22 0.37 A-LevelComputing 0.59 0.53 0.81 0.63 A-LevelEconomics 0.09 0.06 0.11 0.15 A-LevelEnglish 0.02 0.18 0.37 0.26 A-LevelEnglishLanguage -0.29 0.18 0.00 0.00 A-LevelEnglishLiterature 0.00 0.07 0.02 0.13 A-LevelFurtherMathematics 0.20 0.07 ** 0.20 0.10 *A-LevelFurtherAdditional 0.00 0.00 0.00 0.00 A-LevelGeography -0.16 0.08 -0.12 0.15 A-LevelHistory 0.02 0.07 0.05 0.11 A-LevelLanguages -0.04 0.07 -0.05 0.09 A-LevelLaw -0.15 0.28 0.00 0.00 A-LevelMathematics 0.31 0.08 *** 0.16 0.10 A-LevelOther 0.06 0.06 0.17 0.15 A-LevelPhilosophy 0.05 0.10 -0.29 0.32 A-LevelPhysics 0.12 0.08 0.12 0.09 A-LevelPsychology 0.00 0.08 0.23 0.16 A-LevelReligiousEducation 0.16 0.07 * 0.10 0.17 A-LevelScienceOther 0.18 0.43 0.00 0.44 A-LevelSocialScience -0.33 0.19 0.00 0.00 p<0.05(*),p<0.01(**)andp<0.001(***)
Asexpectedanumberof,butnotall,thesamecharacteristicsweresignificantstatisticalpredictorsofTSAperformanceforon-coursestudentsonly:
Significantlylowerperformancefor,
• BME(relativetoWhite)applicants• Femaleapplicants• Acornflaggedapplicants• Post16schoolflaggedapplicants
Note,however,thatthefollowingcharacteristicsarenolongersignificantbasedonthelimitedon-coursepool:
• Schooltype• SchoolhistoryofOxfordapplications• Candidateswhoappliedearlier• ApplicantswithhigherA-LevelandGCSEscores
Also, a smallernumberofA-Level courses are significant statisticalpredictorsofTSAperformance.
A-levelcoursesassociatedwithhigherperformance:
26
• FurtherMathematics,Mathematics,ReligiousEducation
For BMAT, statistical predictors of performance for on-course students only are thefollowing:
Significantlylowerperformancefor,
• Femaleapplicants• Acornflaggedapplicants• Post16schoolflaggedapplicants• ApplicantswithhigherGCSEscores• ApplicantswhoappliedinanearlierUCAScycle
Note,however,thatthefollowingcharacteristicsarenolongersignificantbasedonthelimitedon-coursepool:
• BMEapplicants(relativetoWhiteapplicants)• OverallWPapplicants(relativetonon-flaggedapplicants)• SchoolhistoryofOxfordapplications• Candidateswhoappliedearlier• ApplicantswithhigherA-Levelscores
Only Further Mathematics A-Levels is a significant statistical predictor of BMATperformance.
StudentsacceptedtooneoftheUniversity’scoursesareamoreselectiveandthereforeheterogeneous group than the group of applicants in focus in Table 2.3. We canthereforeassumethatthisispartlythereasonwhysomecharacteristicsarenolongerpredictiveoftestperformancewhenlookingaton-coursestudentsonly,becausetheirvariance in the on-course student group is smaller. Hence, we see that, while schooltype, schoolhistoryofOxfordapplications,earlierapplicationandhigherA-LevelandGCSE scores were statistically predictive of TSA test performance for the applicantgroup,thesecharacteristicsarenolongerassociatedwithtestperformancewithintheon-course student group.Within this group there is however still sufficient variancewithregardtoBMEstatus,genderandsocio-economicdeprivation(ACORNFlag,post16schoolflag),forthosecharacteristicstopredictTSAtestperformance.
Similarly, for BMAT test scores, BME status, overall WP flag, the school’s history ofOxford applications, earlier application and higher A-Level scores applicants do notstatisticallypredicttestperformancewithintheon-coursestudentgroup,whiletheydidwithintheapplicantgroup.Wecanthusassumethattheon-coursestudentsinMedicineandBiomedical science courses aremore similarwith regard to these characteristicsand their related test scores. They aremore heterogeneous, however,with regard togender, indicators of socio-economic deprivation and GCSE scores, as thesecharacteristicsremainsignificantlyrelatedtoBMATtestperformance.
27
3. AnalysisofBrasenoseCollegesurveydata
First year students at Brasenose College were asked about their preparation foradmissionstestsinthecontextofasurvey.Surveyresponsesinformedthepreparationof interviews and, ultimately, the development of a survey administered to OxfordstudentsinPhase2ofthisproject.
Brasenose College provided a file with data for 141 respondents (47 in 2013, 44 in2014,50 in2015),ofwhich116 indicatedthat theyhadsatanadmissionstest(38 in2013,37in2014,41in2015).
Theywereaskedtorespondtothefollowingopenendeditem:
‘Ifyouhadtotakeanaptitudetestduringtheadmissionsprocess–couldyousayhowyou prepared, how much input your school had, and any other thoughts orrecollections?’
Theresponsesreceivedforthisitemrangedfromoneorafewwordstoagoodnumberof longer sentences. Most responses included information on the type and extent ofpreparationandtheroleoftheschoolinpreparation.Onlyaminorityofresponsesalsoprovidedotherthoughtsandrecollections.Table3.1presentsanoverviewofresults.
Studentswho responded to the surveywerenot identified as takersof a specific test(TSA,BMAT,other),henceresponsesmayrelatetoanyadmissionstesttaken.However,survey responses could be grouped in the following thematic areas: amount ofpreparation,supportfromschools,useofexampreparationresources,andperceptionsof the test.Selectedquotes foreachof theseareasarepresented inAppendixC.Mainresultsarepresentedbelow.
Table3.1:BrasenoseCollege:preparationandsupportbysubjectarea
2013 2014 2015 Total % Prepared
‘yes’ %Littleornosupport
fromschool%
Usableresponses 38 37 41* 116* 98° 84.4 51 44.0Arts&Humanities 12 17 12 41 35.3 29 70.7 20 48.8JointDegree 7 4 1 12 10.3 12 100 2 16.7Sciences 9 10 15 34 29.3 32 94.1 14 41.2SocialSciences 10 6 7 23 19.8 20 87.0 13 56.5*includes6respondentswhosesubjectareaisnotclear;°includes5respondentswhosesubjectareaisnotclear
3.1 Amountofpreparation
Amongst the116respondents thathadsatadmissions tests therewerevery fewwhoindicatedtheyhadnotpreparedforthetest.Thevastmajorityofstudentshadpreparedfor the admissions test (84.4%), but the self-reported degree of preparation variedvastly,with14respondentsindicatingthattheyonlyhadverylimitedpreparation.The
28
reasonsgivenforlimitedpreparationvariedfromalackofguidanceonhowtopreparetootherprioritiesandunforeseencircumstances.
MostofthestudentswhoreportedverylimitedtestpreparationwerestudyingasubjectintheArtsandHumanities(9),whereasonlyoneSciencesstudentfellintothisgroup.Overall, just over 70% of Arts andHumanities students explicitly indicated that theyhad undertaken substantial levels of test preparation, compared with nearly 85% ofstudentsoverall(seeTable3.1).
3.2 Supportfromschools
Overall,51of116studentsindicatedsubstantialsupportfromtheirschoolinresponseto the item stated above, 25 indicated little or very little support and a further 26explicitlymade itclear that theyhadnohelp fromtheirschool,apart fromtheschoolproviding some administrative support such as helping in registering for the test. Inmanycases,studentswhoreportedlimitedtestpreparationalsoindicatedthattheyhadnoorverylittlesupportfromtheirschool.Supportfromschoolswasmostwide-spreadforstudentsintheSocialSciences(seeTable3.1).
3.3 Useofexampreparationresources
Irrespective of whether students received support from their school, the mostfrequentlymentionedtypeofpreparationwasworkingthroughpastorpracticepapers.A clear majority of students made, often extensive, use of these papers (i.e., 70students). In a good number of responses it becomes clear that students tried tosubstitutethelackofsupportfromschoolbyworkingextensivelywithpasttestpapers.
Othermaterial/sourcesmentionedincludedprintedhandbooksonadmissionstests,thewebsitesoftestproviders,onlinepracticetests,subject-relevanttextbooks,chatrooms,andsourcesmadeavailableonOxfordCollegeorDepartmentwebpages.
3.4 Perceptionsofthetests
In terms of themore general perceptions of the admissions test by students, a goodnumberofrespondentspointed toasignificant levelofuncertaintyregardingwhat toexpect in the test. This perception seems to be unrelated to the level of preparationstudents did, and in someways developing a strategy for dealingwith the feeling ofuncertaintybecamepartofthepreparationprocess.
Studentsalsodescribedhowthisuncertaintywas‘resolved’,inasfarastheactualteststurned out to be much harder than they had expected. However, there were someinstances in which the experience of taking the test was described as enjoyable orrewarding,notleastbecausetheyweredifferenttowhatstudentshadtoprepareforA-Levels.
29
4. InterviewswithOxfordstudents
To develop a survey for Phase 2, Brinkmann and Kvale’s (2015) seven stages of aninterviewprocesswereusedasaguide.BrasenoseCollegesurveyresponses(Section3)informedthedevelopmentofinterviewswithstudentsatOxfordandinadditionseveralthemesemergedfromtheliteraturereviewandthroughdiscussionswithintheresearchteam which guided the development of interview questions. Further, a pilot wasconductedwiththreecurrentOxfordUniversitystudentsatBrasenoseCollegewhohadpreviouslytakenanOxfordUniversityadmissionstest.Thepilottogetherwithresultsofthe literature reviewand theBrasenoseCollege surveypointed to six thematic areas.The themes are presented in Table 4.1. The themes together with more detailedprompts for follow-up questions were useful in maintaining the researchers’ focusduringtheinterviewprocessastheybridgetheinterviewquestionswiththeresearchaims. The wording of the questions was open aiming at evaluative and descriptiveresponses.ThefullinterviewguideispresentedinAppendixD.
Table4.1:ThematicInterviewGuide
ThemeA TheStudentThemeB PreparationfortheTestThemeC UsefulnessofthePreparationThemeD LogisticsSurroundingtheTestThemeE PerceptionsoftheRelevanceandImportanceoftheTestThemeF AdviceforFutureStudents
These multiple perspectives were considered important for the development of anuancedviewofthefindings(Flyvbjerg2006).ParticipantswereviewedaswhatFoley(2012)calls ‘a respondent’as theywerechosenbecause theyhavespecialknowledgearound preparation for admissions tests and were thus positioned as experts. Thisreassured the participants that they were not being tested and ensured that theinterviewwasmore factual and descriptive rather than emotional (Foley 2012). Thequestionsrequiredparticipantstobereflectivewhichwaslikelytoleadtoavarietyofresponsesdependingupontheparticipants’opennessandabilitytoarticulate.Priortomeetingthem,theresearcherswerenotawareofthepersonalityof theparticipantorhowmotivatedtheyweretoparticipate.
Theaimoftheinterviewswastoprovidethickdescriptionandrichdetailpertainingtopreparation undertaken for Oxford University admissions tests. Ten students wereselected for the main interview. The selection process was as follows. The samplingframe was the Definitive Dataset that contains information on all applicants whoappliedtoOxfordin2016(tocommencetheirstudiesin2017).Itwaslimitedtothosewhometandacceptedtheirofferandtocoursesthatrequiretakinganadmissionstest.It was important to achieve a good representation of all school types. A stratifiedrandomsamplingwasusedtoselect10studentsfromindependentschools,10students
30
from state schools and 8 students from ‘other’ schools. In total 48 students wereselected.Thenthelistwascross-checkedwiththelistofcurrentstudentstorevealthattwoofthechosenstudentsdidnotcommencetheirstudies leaving46students inthesample.
The46studentswereinvitedbyemailtoparticipateintheinterviewintwoblocks,firsttwentythentwentysix,aimingatobtaining10positiveresponses.Asthetakeupfromthestudentswastoolow,itwasnecessarytoapproachanother44studentsinordertosummon10interviewees.Thesewereselectedinexactlythesamewayastheoriginalsample of 46. The 10 students in the final interview sample received each a £10incentiveperinterview.TableA4(AppendixA)givesanoverviewofthecharacteristicsoftheinvitedsampleandthe10studentswhodidtheinterview.
Interviews were conducted in a private room so that the conversation could berecorded confidentially andwithout interruption.Noteswere takenof any reflectionsby the researchers to aid the formulation of follow-up questions. The interviewschedules were semi-structured and were used as a guide with flexibility to exploreotheraspects.Theresearchersactivelylistenedtowhatwassaidandhowitwassaidsothatitcouldbedecidedwhichaspectsofananswertopursuewithsecondaryquestions,beinginformedbyknowledgeofthetopicandguidedbyempathy(Brinkmann&Kvale2015). AsRubinandRubin (2005)advise ‘youshowempathybyaskingquestions toobtainthedetailsthatallowyoutoimaginewhatyourintervieweeshaveexperienced’(2005:81).The researcherswereaware thatparticipantswouldvary in thequalityofdata that they can provide and recognised that somemay be reticent to share theirhonest views, so they attempted to ‘motivate and facilitate interviewee accounts’throughtheirtoneandbodylanguage(Brinkmann&Kvale2015:193).Datacollectionand some initial analysis occurred simultaneously so that judgements could bemadeabout which secondary questions to ask. Interviews followed criteria for a qualityinterviewprovidedbyBrinkmannandKvale’s(2015)andlistedinTable4.2.
Table4.2:BrinkmannandKvale’s(2015:181)criteriaforaqualityinterview
Rich,specific,andrelevantanswersfromtheinterviewee
Theshortestinterviewer'squestionsandlongestsubjects'answerspossible
The degree to which the interviewer follows up and clarifies the meanings of therelevantaspectsoftheanswers
The degree to which the interviewer verifies their interpretations of the subject'sanswersoverthecourseoftheinterview
QualitativeanalysisoftheinterviewdatacontributedtodevisingasurveyforPhase2oftheproject.ThefinalsurveyquestionnaireispresentedinAppendixE.
31
5. Discussionofresultsfromtheexplorationofrelationshipbetweenstudent characteristics, test performance and test preparation
(Phase1)
The aim of this project is to study the impact of preparation for TSA and BMATadmissions testsusing the caseofOxford.Anumberof researchquestionshavebeenpostulated.ThissectionprovidespreliminaryresponsestotheresearchquestionswiththedatasetsutilisedinPhase1.
Researchquestions
Towhatextentdostudentsprepareforuniversityadmissionstestsandwhatisthenature of this preparation amongst students with different backgroundcharacteristics?Results of the Brasenose College survey indicated that the greatmajority of studentsprepared for admissions tests (84%). The degree of preparation, however, variedremarkablyacrosscourseswithstudentsintheArtsandHumanitiesreportinglimitedtest preparation compared to students in the Sciences. About half of the studentsindicated substantial support from schools, one quarter indicated very little supportfrom schools, and another quarter indicated explicitly that they had no help fromschools. Similarly, interviewswith ten students indicated great variability in termsofschool support, with students who had received regular and intense support fromschools to studentswho had not received support at all. Very few students reportedsupportfromfamilyandfriends.
Beyond school support, students indicated making extensive use of past papers inpreparingforadmissionstests.
The Brasenose College survey is restricted to one College. The survey in Phase 2examines further the preparation for admissions tests for students with differentbackgroundcharacteristics.
Whataretheconnectionsbetweenstudentcharacteristicsandtestpreparationandperformance?ResultsofOxfordadmissiondatasetsin2010–2015acrossdifferentcoursesrevealedconnections between student characteristics and performance in the TSA and BMATtest.
Consistently, applicants with higher A-level and GCSE scores and applicants fromschoolswithahistoryofapplyingtoOxfordperformedbetterintheTSAandBMATtestthantherest,evenaftercontrollingforanumberofstudentandschoolcharacteristics.
BMEand female applicants aswell as applicants fromdeprived socio-economic areasperformedworseinbothtests,independentlyofothercharacteristics.
32
Candidates from independent schools performed better in the TSA and BMAT thancandidatesfromstateschools.However,candidatesfromStateandindependentschoolswith equivalent GCSE results performed fairly similar in admissions tests. That is,underperformance of state schools in admissions tests is mainly explained by priorattainmentofcandidates.
ThereisaninteractionbetweentheschooltypeandGCSEperformance,withcandidatesfromindependentschoolsperformingslightlybetterthancandidatesfromstateschoolsatlowerGCSElevelsfortheTSAtest.
Which prior qualifications and subjects are connectedwith test preparation andbettertestperformance?OxfordadmissiondatasetsmadeitpossibletolookattherelationshipbetweenA-Levelchoices and performance in the TSA and BMAT tests. For both tests, specific A-Levelchoices such as Economics, Mathematics, Physics, and Religious Education relatedpositivelytoperformanceinTSAandBMATtests,whereastheGeographyA-levelwasnegatively related to performance in both tests even after student and schoolcharacteristicsaretakenintoaccount.
Isthetimeofapplicationconnectedwithpreparationfor,andbetterperformancein,tests?Previousanalysis(Section2)hadshownthatearlyappliersweremorelikelytoobtainan offer. Thiswas true independently of the course,UCAS cycle, the applicant’s priorattainment,schooltype,andschoolhistoryofapplyingtoOxford.Theassociationwithtime of application was around one fifth as predictive as GCSEs for explaining offerdecisions. Given this evidence, it was postulated that the time of application wasassociatedwithadmissionstestsresults.
EvidencefromOxfordadmissionsdataindicatethatearlyappliersperformbetterintheTSAandBMATtests.Theresultsholdevenafteranumberofcharacteristicsrelatedtothe applicant and his/her schools are taken into account. Differentmechanismsmayexplainthisresult(e.g.,preparationtime,culturalcapital,motivation).Forexample,itispossible that early appliers use more time for preparing for exams, or that earlyappliers and their families are better informed about the system, or even that earlyappliers aremoremotivated than late appliers. Analysis in Phase 2with survey dataexploredsomeofthepossiblemechanismsunderlyingtheassociationbetweentimeofapplicationandadmissionstestscores.
What amount, timing, and nature of preparation are connected with better testperformance?ThisquestionisaddressedinPhase2andresultspresentedaccordingly.
Istheinfluenceoftestpreparationontestperformancemediatedormoderatedbystudentcharacteristicsandbackground?ThisquestionisaddressedinPhase2andresultspresentedaccordingly.
33
What learning strategies and sources of support are connectedwith positive testperformance?ThisquestionisaddressedinPhase2andresultspresentedaccordingly.
Limitations
Likely themost important limitationof this study is thatourdata is restricted toon-coursestudentsatOxford.AsitissuggestedbytheBrasenosedata,themajorityofon-coursestudentspreparedforadmissionstests(84%).However,wehavenoinformationonpreparation for those candidateswhowerenotofferedaplace.One couldassumethat the amount of preparation and preparation strategies varied for candidatesadmittedandnotadmittedtoOxford.Infact,variationbetweenthesetwogroupsmightbeveryvaluableforidentifyingeffectivepreparationstrategies.Totheextentthatouranalysisislimitedtoon-coursestudents,ourfindingsdonotrepresentthewholerangeof performance in admissions tests and some effects might be underestimated. Forexample, the association with socio-economic deprivation will be underestimated ifnon-admitted candidates disproportionallymore often come from socio-economicallydeprivedareas.
6. Oxfordsurveydataandadmissiondata2017/2018cohort
6.1 Development of survey instrument on Oxford student test
preparationandadministration
The survey instrumenthasbeendevelopedon thebasis of the findings fromPhase1and in particular those of the qualitative interviews. It was administered to fresherswhocommencedtheirstudiesinOctober2017.Thequestionnaire(AppendixE)asksanumberofquestionsabouthowstudentshavepreparedfortheadmissionstest.Theseinclude: how far in advance students started preparing for the test, the sources theyusedfortestpreparationandthesupporttheyreceivedfromvariousagents.Inadditionthesurveyincludedascaleonstudentlearningstrategies,widelyusedinthecontextofthe PISA study (Marsh et al., 2006) and adapted for high-stakes tests in previousresearch by the Oxford University Centre for Educational Assessment (Baird et al.,2015).
TheonlinesurveywasadministeredbyOxfordUniversity’sStudentDataManagementandAnalysisteamviaOnlineSurveys(formerlyBOS).Thesurveywasopenbetween9thofMarchand22ndofApril2018, inordertoallowstudentssufficient timetorespondbetweentheendofHilaryandthebeginningofTrinityTerm.
Information collected through the survey was linked to respondents’ studentadmissions recordswhich allowed for similar analysis to that undertaken for smallersamples of students in Phase 1, but employing much more detailed data on testpreparationandmoresophisticatedmethods.
34
6.2 Data
Table6.1givesanoverviewof thecohortandrespondent information. InMichaelmasTerm2017atotalof3,160studentscommencedtheirundergraduatestudiesatOxfordUniversity. Of these, 2,737 (86.6%) students needed to take an admissions test. 703studentstooktheTSAtestand181tooktheBMATtest,thatis,22.2%and5.7%ofthetotalcohort,respectively.
Table6.1:Overviewofcohortandrespondents
StudentscommencinginMT2017
Studentstaking
admissionstest
StudentstakingTSA
StudentstakingBMAT
Surveyrespondents
BMATtesttakers
TSAtesttakers
N 3160 2737 703 181 783 70 202%ofcohort 100.0 86.6 22.2 5.7 24.8 2.2 6.4%ofrespondents 8.9 25.8
6.2.1 Responseratesandanalyticalsample783studentsrespondedtothesurveywhichconstitutes28.6%responserate,ofwhich202(6.4%ofthetotalcohort;25.8%ofrespondents)areTSAtesttakersand70(2.2%ofthetotalcohort;8.9%ofrespondents)areBMATtesttakers(Table6.1).TheresponserateamongTSAtesttakerswas28.7%andamongBMATtesttakerswas38.6%.
Ofthe202TSAtesttakerseightstudentsdidnotindicateinthesurveythattheytooktheTSA test, although theywere identifiedby theUniversity’s admission informationdataasTSAtesttakers.ThosestudentsareexcludedfromtheTSAsampleastheymighthave answeredquestions in the survey in relation to another test. Table 6.2 gives anoverviewoftheanalyticalsample.
Table6.2:Overviewofanalyticalsample
Survey
respondentsBMAT
respondentsTSA
respondentsOthertestrespondents
N 783 70 194 519
%ofrespondents 100 8.9 24.8 66.3
6.2.2 VariablesArangeof informationhasbeencollected through thequestionnaire,notallofwhichcanbereasonablyaddressedinthecontextofthisreport.Variablesusedfordescriptiveanalysisareself-explanatoryregardingtheircategoriesandarenotlistedbelow.Forthemultivariateanalyses(sections7.2to7.5)thefollowingvariableshavebeenused:
35
• TSAtestscore: istheaverageoftwoseparatemarksfromtheCriticalThinkingandProblemSolvingtests,eachweightedequally.ThevariableiscontinuousandhasbeenstandardisedtohaveameanofM=0andastandarddeviationofSD=1.
• BMATtest score: isa combinedscore fromthreeBMAT test sections.Foreachsection students receive amark that is converted into a score. The combinedtotal score weights sections 1 and 2 with 40% and section 3 with 20%. Thevariable is continuouswith a range of 48 to 91 and has been standardised tohaveameanofM=0andastandarddeviationofSD=1.
• Gender:isabinaryvariableindicatingwhetherthestudentisfemale(0)ormale(1);
• BME:isabinaryvariableindicatingwhetherthestudenthasaBMEstatus(1)ornot(0);
• Fee status: is represented by three dummy variables indicating whether thestudenthasaHome,EUornon-EUfeestatus;
• SchoolGroupUK:isabinaryvariableindicatingwhetherthestudentattendedanindependent(0)orstate(1)schoolintheUK;
• SchoolGroupInternational: isabinaryvariable indicatingwhether thestudentattendedaschoolintheUK(0)oraninternationalschooloutsidetheUK(1);
• A-LevelBand:indicateswhetherthestudentachieved‘Less than A*A*B, A*AB or
AAB’ (0), AAB (1), AAA(2),A*AB(3),A*AA(4),A*A*B(5),A*A*A*(6),3A*s(7),4A*s (8) or 5A*s or better (9). For themultivariate analysis the variable wasstandardisedtohaveameanofM=0andastandarddeviationofSD=1;
• Application time: is a continuous variable indicating the number of days thathavelapsedbetweenthestudents’applicationandtheapplicationdeadline(15-OCT-2016).Largervaluesreflectearlierappliers;
• WPOVERALLFLAG(wideningparticipation): isadummyvariable that indicateswhethertheapplicanthasobtainedoneormoreoftheschoolperformanceflags(preorpost16)andoneormoresocio-economic flags(ACORNorPOLAR3)orhaspreviouslybeenincare(1)ornot(0);
• Timepreparation:isacategoricalvariableindicatinghowmuchtimeinadvancestudents started preparing for the test: 'not at all' (0), 'just before' (1), '1-2monthsbefore'(2),'3-6monthsbefore'(3),'6-12monthsbefore'(4),'Morethanayearbefore'(5).Notethatthisvariablegivesnoindicationofeffortordegreeofpreparationinthistimeduration;
• Amount practiced: is a variable combining information from two questions:whetherstudentspracticedpastpapersandhowmuchtheypracticedpasttestpapers.Thevariableindicatesthatstudents‘didnotpracticeatall’(0),‘practicedverylittleorsomewhat’(1),‘practicedquiteabitoragreatdeal’(2);
• Onlineresources: isabinaryvariable indicatingwhetherstudentsoverallusedonlineresources ‘very littleorsomewhat’(0) ‘quiteabitoragreatdeal’ (1)to
36
prepare for the test. The variable originally had four categories, but to ensuresufficient numbers on each category they were collapsed into two formultivariateanalysis;
• Family support: is a binary variable indicating whether students receivedsupportfromtheirfamily(1)ornot(0);
• Schoolencouragement:isabinaryvariableindicatingwhetherastudent’sschoolencouragedthemtoapplytoOxford(1)ornot(0);
• Controlstrategies:isascaleindicatingtheextenttowhichstudentsusecontrolstrategies to prepare for the test (M=0; SD=1; α=0.81110). It is based on fivevariables. Example item: ‘I tried to figure out which ideas I had not reallyunderstood’. Students answered all items by indicating how often they used aparticularstrategyona5-pointLikertscale(always–never).
• Memorisation: is a scale indicating the extent to which students usememorisationstrategiestoprepareforthetest(M=0;SD=1;α=0.916).Itisbasedon four variables. Example item: ‘I tried tomemorise asmuch of the revisionmaterialaspossible’.Studentsansweredall itemsby indicatinghowoftentheyusedaparticularstrategyona5-pointLikertscale(always–never).
• Elaboration:isascaleindicatingtheextenttowhichstudentsusememorisationstrategies to prepare for the test (M=0; SD=1; α=0.792). It is based on fourvariables. Example item: ‘I tried to understand the revisionmaterial better byrelating it to what I already knew’. Students answered all items by indicatinghow often they used a particular strategy on a 5-point Likert scale (always –never).
6.3 Descriptionoftheanalyticsampleandnon-respondents
Thesamplecomprised58%females.Themajorityofrespondents(84%)wereWhiteasopposed toBME.Threequarters of respondentswereUK students (75%), 11%werefrom the EU and 13% were non-EU. There were 14% of respondents who had anOverall WP Flag11. There were 37% of respondents who came from independentschools (as opposed to state schools) and 23% of respondents came from overseasschools.TableA5(Appendix)givesanoverviewofthesamplesstatistics.
Inordertocheckwhetheroursampleofrespondentsissubstantiallydifferentfromthenon-respondentsacomparisonofbackgroundcharacteristicswasconducted.Figure6.1summarisesthedifferencesbetweenthetwogroups.
10 The Cronbach’s Alphameasures the internal consistency of the scale, α values higher than 0.8 areconsideredrobustscales.
11StudentsareflaggedwiththeOverallWPFlagiftheyhaveatleastoneofschoolperformanceflag(pre-16orpost-16flag)ANDatleastonesocio-economicflag(ACORNorPOLAR)ORiftheywerepreviouslyincare.
37
Figure 6.1: Background characteristics by survey respondents and non-
respondents
Chi-Squaretestsrevealthat,apartfromthepercentageofstudentswithBMEstatus,alldifferences between the two groups are significant. Thus, the respondents areoversampled in the sense that significantly more female students, those from EUcountries,thosefromstateschools(UKonly)andinternationalschoolsandthosewithanOverallWPFlaghaveansweredthequestionnaire.Inaddition,respondentsachievesignificantlymoreoftenanA*ABA-Levelscorethannon-respondents(notpictured).
ThereisnosignificantdifferencebetweentheTSAtestscoresofNon-respondents(M=68.6, SD = 6.13) andRespondents (M= 67.9, SD = 6.17; t (688) = 1.236, p = .0.746).Further,nosignificantdifferencebetweentheBMATtestscoresofNon-respondents(M=64.09,SD=6.72)andRespondents(M=66.33,SD=7.93;t(176)=-2.022,p=.263)wasidentified.
7. Analysis of Oxford survey data and admission data 2017/2018cohort
7.1 Descriptiveanalysisoftheextentandnatureoftestpreparation
7.1.1 ExtentandnatureoftestpreparationbytesttakersThis section looks only at the identified analytical sample (n=783) and distinguishesbetweenTSA,BMATandothertesttakerswhereappropriate.
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Female
BME
UK
EU
Non-EU
Overall WP flag
Independent School (vs. State School [UK only])
International School (vs. UK school)
% Non-Respondents % Respondents
38
Themajorityofstudentsstartedpreparingfortheadmissionstest1-2monthsormoreinadvanceofthetestdate(Figure7.1).Thisis independentofwhethertheypreparedfor the TSA, BMAT or another admissions test. Only a small number of studentsindicatedthattheydidnotprepareatallforthetest.TSAtesttakersstartpreparingforthe test later (1-2 months before) than other test takers (including BMAT) who aremorelikelytostarttheirpreparations3-6monthsbeforethetestdateorevenearlier.Thesedifferencesaresignificant.
Figure7.2indicatesthatover90%oftesttakerspracticedpastpaperswhenpreparingfor theadmissions test.Chi-Square tests reveal thatother test takersare significantlyless likely and TSA takers are significantly more likely to prepare for the test bypracticingpastpapers.Moreover,BMATtakerspreparesignificantlymorethanTSAtesttakers(Figure7.3).
Therewerefurtherquestionsabouthowstudentspracticedpastpapers.77%indicatedthattheypracticedundertimedconditions,35.2%indicatedreceivingfeedbackonthepaperstheypracticedand88.8%statedtheyusedamarkscheme.Figure7.4providesfurther information about howuseful students perceived thesemodes to be for theirpreparation.
Figure7.1:Timestudentsstartpreparingfortheadmissionstest
16%
20%
11%
44%
55%
53%
26%
17%
26%
9%
5%
9%
Other
TSA
BMAT
When did you start preparing for the test?
I dont remember I did not prepare at all Just before the test
1-2 months before the test 3-6 months before the test 6-12 months before the test
More than a year before the test
39
Figure7.2:Studentsconfirming that theypracticedpastpaperswhenpreparing
fortheadmissionstest
Figure7.3:Amountstudentspractisedpastpapers
Both practicing past papers under timed conditions and receiving feedback on themwere perceived as extremely helpful in the preparation process by the majority ofstudentswhousedthosemodes.Themarkschemewasalsomainlyperceivedasquitehelpful.Onlyasmallpercentageofstudents(<15%)indicatedthattheydidnotperceivethedifferentwaysofpractisingpastpapersashelpful.
Inordertopreparefortheadmissionstest,studentsusedonlineresources.Figure7.5displays the differences between overall use of online resources by test takers. Chi-Square significance tests reveal that BMAT test takers indicated they used onlineresourcessignificantlymoreoftenthanothertesttakersincludingTSAtesttakers.This
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Other
TSA
BMAT
Did you practise past papers when preparing for the test?
7%
4%
3%
23%
24%
16%
36%
44%
32%
34%
28%
49%
Other
TSA
BMAT
Very little Somewhat Quite a bit A great deal
40
finding ishoweverhardlysurprisingasmoreonlineresourcesareavailable forBMATthanforTSA.
Figure7.4:Helpfulnessofdifferentmodestopracticepastpapers
Figure7.5:Overalluseofonlineresourcestopreparefortheadmissionstest
Thesurveyinstrumentalsoinquiredinmoredetailaboutthetypesofonlineresourcesstudents used to prepare for the admissions tests. Overall most students used theOxfordUniversity’swebsite,theofficialtestproviderwebsitesandtheStudentRoominorder to prepare for the admissions tests. They less often accessed other onlineresourcessuchasYouTubeorschool/collegewebsites.Figure7.6indicatesthatthere
13%
11%
4%
51%
39%
47%
36%
50%
49%
Mark scheme
Feedback received
Timed conditions
Not at all helpful Not very helpful Quite helpful Extremely helpful
20%
16%
9%
25%
26%
11%
28%
28%
36%
27%
30%
44%
Other
TSA
BMAT
Overall, how much did you use online resources?
Very little Somewhat Quite a bit A great deal
41
are substantial differences between the different test takers and their use of specificonlineresources.Chi-Squaretestsrevealthefollowingsignificantdifferences:OthertesttakersaresignificantlymorelikelytousetheOxfordUniversity’swebsiteandlesslikelytouseanofficial testproviderwebsite. Incontrast,BMATtest takersaresignificantlylesslikelytousetheOxfordUniversity’swebsiteandmorelikelytousetheofficialtestproviderwebsiteorotherwebsitesdedicatedtothetest.TSAtesttakersarealsomorelikely to use the official test provider website but less likely to use other websitesdedicatedtothetest.
The majority of students who used the Oxford University website, the official testproviderwebsiteandotherwebsitesdedicated to the testperceived themasquiteorextremelyhelpful(Figure7.7). But, therearealsosubstantialproportionsofstudentswho perceived the Oxford University website (29%), The Student Room (44%) andotherwebsitesdedicatedtothetest(19%)asnotveryhelpful.
Figure7.6:Useofonlineresourcestopreparefortheadmissionstest
Figure7.7:Perceivedhelpfulnessofdifferentonlineresourcesforadmissionstest
preparation
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Another school/college website(s)
Other student forum websites (besides the
Student Room)
Your school/college website
Other
YouTube
Other websites dedicated to the test
The Student Room
Oxford University website
Official test provider website
TSA BMAT Other
42
Studentswerefurtheraskedtoprovideinformationaboutsupporttheyreceivedfromtheir families in the process of preparing for the test. Only a few students receivedsupport from their families and the differences between the test takers are notsignificant (Figure 7.8). Themajority of studentswho did receive support from theirfamilies indicated that this was financial support (13%), working out how to bestapproachthetest(10%)goingthroughpasttestpapers(9%),revisingsubject-specificmaterial(4%)andotherformsofsupport(6%).
Figure7.8:Familysupportreceived
Withregardtosupportreceivedfromtheirschools,themajorityofstudentsindicatedthat theywere already considering applying to Oxford and their schools encouraged
9%
25%
7%
28%
24%
17%
26%
62%
61%
49%
53%
62%
35%
38%
80%
83%
29%
14%
44%
19%
14%
48%
35%
20%
17%
Oxford University website
Official test provider website
The Student Room
Other websites dedicated to the test
YouTube
Other
Your school/college website
Another school/college website(s)
Other student forum websites (besides The Student
Room)
Extremely helpful Quite helpful Not at all/not very helpful
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Other
BMAT
TSA
43
them(Figure7.9).Onlyafewstudents indicatedthattheirschools infactdiscouragedthem to apply. Similar proportions of students indicate that their schools supportedthem in preparing for the admissions test by providing opportunities to ask subject-specificquestionsorworkwithasubjectteacher,providingworkshopsandmentoring.Schoolslessoftenprovidedtheopportunitytouseclassroomperiodstoprepareforthetest(Figure7.10).Notethatresultsforschoolsupportarenotdistinguishedbytesttypebecausethereisnoreasonableassumptionthatschoolsupportshoulddifferbythetestthatstudentstake.Differencesbystudentbackgroundcharacteristicsarepresentedinsection7.1.2.
Figure7.9:Schoolencouragementreceived
Figure7.10:Schoolsupportreceived
Students fell back on a range of other strategies and resources to prepare for theadmissionstest(Figure7.11)suchasstudyingthesubjectsyllabusortextbooks,talking
0% 20% 40% 60% 80% 100%
No, they actually discouraged me from
applying
No, they did not
Yes, I was already considering it and
they encouraged me
Yes, they suggested I should consider
applying
Did your school/college encourage you to apply to Oxford?
0% 20% 40% 60% 80% 100%
Opportunity to use some class periods
to prepare for the test
General support/mentoring from a
teacher at your school/college
Provide workshops or classes to help
prepare for the admissions test
Opportunity to ask subject-specific
questions or work with a subject
teacher at your school/college
Forms of school support
44
tocurrentorformerOxfordstudentswhotookthetestorprivatetutorials.Chi-Squaretestsrevealanumberofsignificantdifferencesbetweenthetesttakers.Comparedwithstudents takingotherentrancexaminations in thedataset,BMATandTSA test takerssignificantlymoreoftenusedcommerciallypublishedtestpreparationbookstopreparefor the test (which may not be available for all of the other tests); BMAT takerssignificantlymoreoftenattendedcommercialpreparationcoursesandusedtheservicesofanexternalagencyinordertoprepareforthetest;BMATandothertesttakersusedsubjectsyllabus/textbookssignificantlymoreoften;other test takers talked to formerpupilsof their schoolor current/formerOxford students significantlymoreoftenandattendedOxfordOpenDaysmoreoften.
Figure7.11:Otherstrategiesandresourcesusedforpreparation
7.1.2 Extent and nature of test preparation by student backgroundcharacteristics
Thissectionlooksattheidentifiedanalyticalsample(n=783)anddifferentiatesresultsby different student background characteristics. All descriptive analyses wereconductedforthefollowingbackgroundcharacteristics:Gender,BMEstatus,Feestatus,OverallWPFlag,SchoolGroup(UKonly),SchoolGroup(International).However,onlydifferences that are significant between different student groups are reported anddiscussed.
Figures 7.12 to 7.14 present the differences ofwhen students start preparing for theadmissions test by their characteristics. UK students were significantly less likely toreportnopreparationforthetestthanothers,andnon-EUstudentsweresignificantly
0% 20% 40% 60% 80% 100%
Other
Used services of an external agency
Had private tutorials
Attended commercial preparation course(s)
Talked to current or former Oxford students
Talked to former pupils at my school/college
Attended Oxford Open Days
Used subject syllabus/textbooks
Read commercially published test preparation book(s)
TSA BMAT Other
45
morelikelytoreportnopreparation.Non-EUstudentswerealsosignificantlylesslikelytoreportthattheystartedpreparing1-2monthsbeforethetestthanothers.
Significantlymore students fromUK state schools reported preparing just before thetestthanthosefromindependentschools,andtheywerealsosignificantlylesslikelytoreport preparing 3-6 or 6-12 months before the test than those from independentschools(Figure7.13).Further,studentsfromschoolsoutsidetheUKsignificantlymoreoftendidnotprepareforthetestatall.
Figure7.12:Timestudentsstartpreparingfortheadmissionstestbyfeestatus
Figure7.13:TimestudentsstartpreparingfortheadmissionstestbySchoolType
(UKonly)
5%
6%
16%
18%
20%
49%
52%
35%
24%
17%
27%
9%
6%
8%
UK
EU
Non-EU
I don't remember I did not prepare at all Just before the test
1-2 months before the test 3-6 months before the test 6-12 months before the test
More than a year before the test
9%
20%
43%
51%
32%
20%
13%
7%
Independent
State
I dont remember I did not prepare at all Just before the test
1-2 months before the test 3-6 months before the test 6-12 months before the test
More than a year before the test
46
Figure7.14:TimestudentsstartpreparingfortheadmissionstestbySchooltype
(International)
Almostallstudentspracticedpasttestpapersbutfemalestudents(Figure7.15),thosewith an Overall WP Flag (Figure 7.16) and those from International schools (Figure7.17)practicedpasttestpaperssignificantlylessoftenthantheircounterparts.
With regard to the ways in which they practiced (mark scheme, feedback, timedconditions), significant differences are only found for students from independentschoolswhoreceivedfeedbacksignificantlymoreoften(56.9%)comparedtothose instateschools(31.4%)and forstudents fromUKschoolsoverall (40.3%),comparedtointernationalschools(15.4%).
Figure7.15:Amountstudentspracticedpastpapersbygender
Figure7.16:AmountstudentspracticedpastpapersbyOverallWPFlag
6%
16%
19%
49%
45%
24%
21%
9%
6%
UK school
International School
I dont remember I did not prepare at all Just before the test
1-2 months before the test 3-6 months before the test 6-12 months before the test
More than a year before the test
0% 20% 40% 60% 80% 100%
% practiced test papers
Very little
Somewhat
Quite a bit
A great deal
Male Female
47
Figure 7.17: Amount students practiced past papers by School Group
(International)
Figure7.18:OveralluseofonlineresourcesbySchoolType(International)
0% 20% 40% 60% 80% 100%
% practiced test papers
Very little
Somewhat
Quite a bit
A great deal
Overall WP Flag No Overall WP Flag
0% 20% 40% 60% 80% 100%
% practiced test papers
Very little
Somewhat
Quite a bit
A great deal
International school UK school
20%
13%
23%
28%
30%
25%
27%
34%
UK school
International school
Very little Somewhat Quite a bit A great deal
48
SignificantdifferencesintheoveralluseofonlineresourcesarefoundonlyforstudentsfromUKschoolscomparedtostudentsfrominternationalschools,wherestudentsfromUKschoolindicatedthattheyuseonlineresources‘verylittle’significantlymoreoften(Figure7.18).
Differences in family support received are not significant between students fromdifferent background groupswith one exception; students from international schools(39.4%) received significantly more support from their families than those from UKschools(29.6%).Thedifferenceissignificantandisduetohigheramountsoffinancialsupport(29%)thatstudentsfromInternationalschoolsreceivecomparedtotheirpeersfromUKschools(8%).
Figure 7.19 presents the differences in school support by students fromdifferentUKschools.Onlya relativelysmallpercentageof students reported that their schoolsdidnotencouragethemtoapply(10%).Butstudentsfromstateschoolsindicatedthattheywere less often encouraged to apply (12%) than students from independent schools(6%).Theywerealsolessoftenencouragedwhentheywereconsideringitthemselvesthanstudentsfromindependentschools(63%vs.74%)..Thedifferencesaresignificant.This result prompts a closer look at the kind of support that independent schoolstudentsreceivecomparedtostateschoolstudents.
Figure7.19SchoolencouragementbySchoolGroup(UKonly)
Figure 7.20 and significance tests reveal that students in independent schools areofferedsignificantlymoreopportunitiestoasksubject-specificquestionsorworkwithasubjectteacher,receivegeneralsupport/mentoringfromateacherandmayprepareinworkshopsorclassesfortheadmissionstestthanstateschoolstudents.
0% 20% 40% 60% 80% 100%
No, they actually discouraged me from
applying
No, they did not
Yes, I was already considering it and
they encouraged me
Yes, they suggested I should consider
applying
State Independent
49
Figure7.20FormsofschoolsupportreceivedbySchoolGroup(UKonly)
Students from International schools (Figure 7.21) were also significantly less oftenencouraged to apply and less often encouraged when they were considering itthemselvesthanstudentsfromUKschools.
Figure7.21:SchoolsupportbySchoolType(International)
Female students (27.3%) read commercially published test books significantly moreoftenthanmalestudents(18.1%).BMEstudents(39.6%)readcommerciallypublishedtest books significantly more often than non-BME students (20.5%). This may beattributabletoahigherpercentageofBMEstudentstakingtheBMAT(36.1%)thantheTSA (12.4%). BME students were also more likely to have attended commercialpreparationcourses(13.5%vs.6.0%).
0% 20% 40% 60% 80% 100%
Provide workshops or classes to help
prepare for the admissions test
Opportunity to use some class periods
to prepare for the test
General support/mentoring from a
teacher at your school/college
Opportunity to ask subject-specific
questions or work with a subject
teacher at your school/college
State Independent
0% 20% 40% 60% 80% 100%
No, they actually discouraged me from
applying
No, they did not
Yes, I was already considering it and
they encouraged me
Yes, they suggested I should consider
applying
International school UK school
50
StudentswithanOverallWPFlag talked to formerpupilsat their school/collegewhotook the test significantly lessoften than thosewithoutanOverallWP flag (8.3%vs.19.7%) andWP students reported using other strategiesmore often (4.8%vs. 1.4%)(bothdifferencesaresignificant).
Studentsfromindependentschoolstalkedtoformerpupilsattheirschool/collegewhotookthetestsignificantlymoreoftenthanthosefromstateschools(32.3%vs.11.0%),andalsoweremorelikelythanstateschoolstudentstohavetalkedtocurrentorformerOxford students who they knew (21.4% vs. 10.8%) (both significant) and to haveattended Oxford Open Days (32.2% vs. 24.4%). Students from International schoolsusedtheservicesofanexternalagencysignificantlymoreoftenthanthoseatUKschools(6.0%vs.1.6%),andalso talked to formerpupilsat their school/collegewho tookanadmissionstestsignificantlylessoften(11.3%vs.18.4%).Studentsfrominternationalschools also attended Oxford Open Days less often (6.0% vs. 27.9%) than those atschoolintheUK.
7.2 Associations between student background characteristics, test
preparationandtestperformance
This sections looks at associations between student background characteristics, testpreparationandtestperformance.Analysesareconductedforstudentsoftheanalyticalsample who took the TSA (n=194) and BMAT (n=70) test. Analyses are conductedseparately for the two groups but coefficients are comparable because dependentvariables(TSAtestscoreandBMATtestscore)havebeenstandardisedtohaveameanofM=0andastandarddeviationofSD=1.Further,becausesamplesarerelativelysmall,particularlyfortheBMATsample,aparsimoniousapproachtomodellingisadopted,i.e.where independent variables are identified as non-significant predictors of testperformance,theyareexcludedinthesubsequentanalysis.
Inthefirststepregressionanalysiswereconductedtopredicttestscoresbytheamountandnatureoftestpreparation.Table7.1summarisestheresults.Models1to5forTSAandModels7to11forBMATevaluatetherelationshipbetweeneachpredictorwiththetestscore individually. InModels6and12allvariableswereusedtogethertopredictthe testscore.Forboth tests,whenstudentsstartpreparing for the test issignificant.Thus,theearlierthestudentstartspreparingforthetestthehigherthetestscores.Theassociationsremainstatisticallysignificantwhenallotherpredictorsarecontrolledfor.Initially,theuseofonlineresourcesissignificantlyrelatedtoBMATtestscore,butwhenother predictors are controlled for, the association becomes insignificant. All otherindicatorsoftestpreparationarenotsignificantlyrelatedtotestscores.Consequently,theyareexcludedfromsubsequentanalysis.
51
Table7.1:Regressioncoefficientsfortestperformanceontestpreparationbytest
Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig.
fixedeffectsTimepreparation 0.173 (0.08) * 0.217 (0.09) *
Amountpracticed 0.187 (0.14) 0.045 (0.17)
Onlineresources 0.256 (0.15) 0.166 (0.15)
Familysupport -0.009 (0.16) -0.150 (0.16)
Schoolencouragement -0.013 (0.17) -0.112 (0.17)
randomeffectsexplainedvariance(in%) 12.1
Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig.
fixedeffectsTimepreparation 0.423 (0.13) ** 0.287 (0.15) *
Amountpracticed 0.432 (0.24) 0.233 (0.23)
Onlineresources 0.821 (0.28) ** 0.564 (0.32)
Familysupport 0.191 (0.27) 0.007 (0.26)
Schoolencouragement 0.115 (0.28) -0.003 (0.26)
randomeffectsexplainedvariance(in%) 17.30
p<0.05 (*), p<0.01 (**) and p<0.001 (***)
Model7
Model1 Model2 Model3 Model4
Model12Model11Model10Model9Model8
TSAtestscore
BMATtestscore
Model5 Model6
52
In a second step relationships between student background characteristics and test
preparation with test scores were evaluated (Table 7.2). As in the previous table,
Models1to8forTSAandModels11to18forBMATevaluatetherelationshipbetween
eachpredictorwith the testscore individually.Models9and10(TSA)and19and20
(BMAT)combineanumberofvariablestopredictthetestscore.
For the TSA test score, gender, OverallWP Flag and UK school group are significant
predictorswhenlookedatindividually.Malestudentsscorehigherwhilethosewithan
OverallWPflagandthosefromstateschoolsperformlower.Whenthosepredictorsare
evaluated in amultiple regression togetherwithwhen students startedpreparing for
thetestandtheiroveralluseofonlineresources(Model9),onlythetimewhenstudents
startedpreparingforthetestissignificantlypositivelyrelatedtotestscore.Thegender
coefficient closely misses the significance level. Model 10 additionally evaluates the
associationbetweenA-Levelbandandtestscores,butthisisnotasignificantpredictor
ofTSAtestscoreswheneitherlookedatindividually(Model8)orwhenothervariables
arecontrolledfor(Model10).
Similarly, for theBMAT test scores,havinganOverallWPFlagandhavingattendeda
state school are negatively associated with test performance when evaluated
individuallywhereasoverseasstudentsperformbetter.Moreover,studentswithbetter
A-Levels score significantly higher in the BMAT test. In the combined Model 19,
students from state schools perform significantly lower than those from independent
schoolsand femalestudentsperformsignificantly lesswell thanmalestudents.12Test
preparationvariablesarenotsignificantoncebackgroundcharacteristicsarecontrolled
for, i.e. characteristics that predict high performance in BMAT scores are out of
students’ control. Thismeans, in contrast to results for TSA,where preparing for the
testearliermitigateddisadvantagesfromdemographiccharacteristics,femaleandstate
schoolstudentsstillperformlowerinBMATdespitehavingequaltimepreparingforthe
tests as male and independent school students. Model 20 additionally evaluates the
associationbetweenpriorattainmentandtestscoresandthisissignificant,evenwhen
othervariablesarecontrolledfor.
ThevariablesinTable7.2explain19.7%of test scorevariance forTSA,but45.1% for
BMAT. Thus, students’ background characteristics seem to be substantially more
predictive of test scores for BMAT than for TSA. The amount of explained variance
remains fairly constant when additionally controlling for A-Level band (23.2% TSA;
52.6%BMAT).
Note, although Fee status (OS) is initially significant, it could not be included in the
combinedmodelforcomputationalreasons.Infact,studentsintheBMATsamplewith
anOverseasFeeStatusareonlyn=3.
12ItshouldbenotedthatmalestudentsareunderrepresentedintheBMATsample(20%)andthatthis
mayoverestimatethegendereffect.
53
54
Table7.2:Regressioncoefficientsfortestperformanceontestpreparationandstudentbackgroundcharacteristicsandpriorattainmentbytest
Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig.fixedeffectsGender(male) 0.428 (0.15) ** 0.311 (0.16) 0.277 (0.17)BME -0.25 (0.26)Feestatus(EU) 0.328 (0.21)Feestatus(OS) 0.254 (0.21)OverallWPFlag -0.51 (0.25) * -0.3 (0.25) -0.19 (0.27)SchoolgroupUK(State) -0.52 (0.18) ** -0.34 (0.19) -0.46 (0.21) *SchoolgroupINT(Intern.) 0.325 (0.17)A-Levelband -0.01 (0.08) -0.07 (0.08)Timepreparation 0.312 (0.10) ** 0.303 (0.10) **Onlineresources 0.028 (0.16) 0.012 (0.17)randomeffectsexplainedvariance 19.70 23.20
Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig.fixedeffectsGender(male) 0.571 (0.30) 0.72 (0.27) ** 0.515 (0.26) *BME 0.082 (0.26)Feestatus(EU) 0.17 (0.42)Feestatus(OS) 1.015 (0.57) *OverallWPFlag -0.99 (0.37) ** -0.58 (0.34) -0.58 (0.32)SchoolgroupUK(State) -0.98 (0.22) *** -0.68 (0.23) ** -0.81 (0.22) ***SchoolgroupINT(Intern.) 0.599 (0.40)A-Levelband 0.404 (0.17) * 0.272 (0.12) *Timepreparation 0.222 (0.13) 0.188 (0.13)Onlineresources 0.274 (0.28) 0.273 (0.28)randomeffectsexplainedvariance 45.10 52.60p<0.05(*),p<0.01(**)andp<0.001(***)
Model16 Model17 Model18 Model19 Model20
TSAtestscore
BMATtestscore
Model7 Model8 Model9 Model10
Model11 Model12 Model13 Model14 Model15
Model1 Model2 Model3 Model4 Model5 Model6
55
7.3 Mediating and moderating effects of student backgroundcharacteristics
In order to better understand the associations between student backgroundcharacteristics, test preparation and test performance, mediation and moderationanalysisisconducted.Mediationanalysisisusedtoinvestigatewhethertheassociationsbetween test preparation and test performance are to some extent associationsbetween student background characteristics and test performance because they areconfoundedwith each other, i.e. that for examplemale students use online resourcesmoreoftenandbecausemalestudentshaveonaveragehighertestperformanceitlookslikeonlineresourcesareassociatedwithtestperformance.Moderationanalysisisusedto investigatewhethertheassociationbetweentestpreparationandtestperformancevaries by different student background characteristics, i.e. that for example theassociationbetweenthetimestudentsstartpreparinginadvanceandtestperformanceisdifferentiallystrongforstudentswithorwithoutanOverallWPFlag.
7.3.1 Mediating effects of student background characteristics for theassociationbetweentestpreparationandtestperformance
In the regression models student gender, having an Overall WP Flag, being from anindependent or state school and prior attainment were at some point identified assignificant predictors of test performance on TSA or BMAT. Thus this selection ofbackgroundvariablesisusedinthemediationanalysis.Moreover,whenstudentsstartpreparing for the test and the overall use of online resources were identified assignificantpredictorsoftestperformance.Inaddition,theamountofpracticingpasttestpapers is included, as the previous descriptive analysis revealed several significantdifferencesonthisvariablebetweenstudentsofdifferentbackgroundcharacteristics.
Figure 7.21 shows the general mediation model conducted separately for the fourdifferent background characteristics. For each background characteristic a separatemediationmodelwasrunbothforTSAandBMATtestscoresasdependentvariables.
Neither of the selected student background variables significantlymediate any of theassociationsbetweentheTSAtestscoreandwhenstudentsstartpreparingforthetest,the amount they practice and their use of online resources. For BMAT, the onlysignificantmediationeffectwasfoundbetweentheuseofonlineresourcesandgender.That is, the positive association between the use of online resources is substantiallymediatedbygenderasthedirecteffectisinfactsmallerthantheindirecteffect.Thus,male students overall achieve higher on the BMAT test and because they use onlineresourcesmoreoftenthanfemales,itinitiallyappearsasiftheuseofonlineresourcesispositively related to test performance. However, a substantial amount of thisassociationisconfoundedwiththefactthatmalestudentsgenerallyattainhigherscoresinthetest.
56
Figure 7.21: Generalmediationmodel to evaluate the association between testpreparationandtestperformance
7.3.2 Moderating effects of student background characteristics for theassociationbetweentestpreparationandtestperformance
Again, the four student background characteristics (gender, Overall WP Flag, SchoolGroupUK,andpriorattainment)andthreetestpreparationvariables(thetimestudentsstart preparing in advance for the test, the overall use of online resources and theamountofpracticingpasttestpapers)wereselectedforthemoderationanalysisforthesamereasonsasoutlinedinsection7.2.1.
Figure7.22 shows thegeneralmediationmodel. Separatemodelswere conducted foreach of the four different background characteristics by each of the test preparationvariables.
Figure 7.22: Generalmediationmodel to evaluate the association between testpreparationandtestperformance
Time preparation
Amount practiced
Online resources
Background characteristics
Test score
Background characteristics
Test score Test preparation
57
Neither TSA nor BMAT test scoremodels show any significantmoderating effects ofstudents background characteristics on the relationship between test preparationvariables and test score. That is, the relationships between test preparation and testscores are not significantly different for students of different categories on thebackgroundcharacteristics.
7.4 Associationsbetweentimeofapplication,testpreparationandtestperformance
Thissectionevaluateswhetherthetimeelapsedbetweenthestudents’applicationandtheapplicationdeadline(15-OCT-2016)isrelatedtotestpreparationandperformance.Thevariablereflectsthedifferencebetweenthosetwodateswherelargervaluesreflectanearlierapplication.
Table7.3summarisestheresultsoftheregressionanalysesoftestperformanceontestpreparation and application time. The time that elapsed between the application andtheapplicationdeadlineisnotasignificantpredictoroftestperformance,eitherwhenevaluatedseparatelyorinrelationwithwhenstudentsstartpreparingforthetestandtheiruseofonlineresources.Whilethisisinlinewiththeresultsforon-coursestudentsinPhase1, it isworthremembering thatapplication timewassignificantlyassociatedwithtestscores in thesampleofapplicants,whereearlierapplicantsscoredhigher inbothtests.
7.5 Learningstrategiesandtestperformance
7.5.1 ThelearningstrategyscalesStudentswereaskedanumberofquestionsrelatingtothestrategiestheyappliedwhenpreparingfortheadmissionstest.Informationonstudents’learningstrategiesiswidelyusedinthecontextofthePISAstudy(Marshetal.,2006)inordertopredicteducationaloutcomes. The items used in this survey have been adapted for high-stakes tests inpreviousresearchbytheOxfordUniversityCentreforEducationalAssessment(Bairdetal.,2015).Overall,studentsprovidedinformationon14questions.Factoranalysiswasusedtomapthe14itemsontothreescalesidentifiedbyMarshandcolleagues(2006).These are: Control strategies, memorisation and elaboration. Table 7.4 gives anoverviewofitems,factorandscalecharacteristics.Withtheexceptionofone,allitemsweresuccessfullymappedontothethreescalesaccordingly.
58
Table7.3:Regressioncoefficientsfortestperformanceontestpreparationandapplicationtimebytest
TSAtestscore BMATtestscore Model1 SE Sig. Model2 SE Sig. Model1 SE Sig. Model2 SE Sig.fixedeffects Applicationtime -0.128 (0.65) -0.371 (0.65) 0.646 (0.97) 0.422 (0.89) Timepreparation 0.162 (0.08) * 0.318 (0.14) *Onlineresources 0.225 (0.15) 0.52 (0.31) randomeffects explainedvariancein% 3.6 16.5 p<0.05(*),p<0.01(**)andp<0.001(***)
59
Table 7.4: Overview of items used for learning strategy scales and statisticalinformationVariables Scale Eigenvalue Explained
VarianceReliability(α)
IstartedbyfiguringoutexactlywhatIneededtolearn.
Controlstrategies
2.895 48.1 0.811
I checked if I understood what I hadread.
Controlstrategies
I made sure that I remembered themost important points in the revisionmaterial
Controlstrategies
If I did not understand something, Ilooked for additional information toclarifyit.
Controlstrategies
I tried to figureoutwhich ideas Ihadnotreallyunderstood.
Controlstrategies
I tried to memorise all the materialrequired
Memorisation 3.22 74.5 0.916
Itriedtolearnmynotesbyheart. Memorisation
I tried to memorise as much of therevisionmaterialaspossible
Memorisation
ItriedtomemorisewhatIthoughtwasimportant.
Memorisation
I tried to relate new information toknowledgefromothersubjects.
Elaboration 2.469 49.7 0.792
I figured out how the informationmightbeusefulintherealworld.
Elaboration
I tried to understand the revisionmaterialbetterbyrelatingittowhatIalreadyknew.
Elaboration
I studied material that went beyondwhatwasexpectedforthetest
Elaboration
7.5.2 Associations between learning strategies, student backgroundcharacteristicsandtestperformance
RegressionanalysiswasusedtoevaluatewhetherthethreedifferentlearningstrategyscaleswereassociatedwithTSAandBMATtestscores.However,nosignificantresultscouldbereportedforeithertest.Nevertheless,acloserlookathowdifferenttesttakersandstudentswithdifferentbackgroundcharacteristicsmakeuseoflearningstrategieswhenpreparingforthetestrevealedsomesignificantdifferences.Specifically,studentspreparing for theBMAT test applied all three learning strategies – control strategies,memorisationandelaboration–moreoftenthanthosepreparingfortheTSA.ThesameistrueforBMEstudents,althoughitisworthnoting,thatsubstantiallymoreBMATtesttakersareBMEstudents(36.1%)thantheTSAtakers(12.4%).Further,femalestudents
60
applymemorisation strategiesmore often thanmale students. Considering that bothBMAT and TSA have major thinking skills components, this may not be the mostsuitablestrategyandmaypotentiallyberelatedtotheconsistentlylowertestresultsforfemalestudents in theBMAT. Inaddition,students fromschoolsoutsidetheUKapplymemorisationtechniqueslessoften.
Table 7.5: Descriptive statistics for learning strategies scale by test type andbackgroundvariables
Controlstrategies Memorisation Elaboration M SD Sig.# M SD Sig.# M SD Sig.#Testtypes
BMAT 0.52 0.56 *** 0.97 0.75 *** 0.51 0.71 ***TSA -0.19 0.97 -0.35 0.78 -0.18 0.90
Gender
Female 0.07 0.91
0.10 0.99 * 0.06 0.90
Male -0.13 0.96 -0.18 0.89 -0.10 0.91BMEstatus
Non-BME -0.11 0.95 *** -0.03 0.95 ** -0.07 0.87 **BME 0.48 0.56 0.60 1.04 0.31 0.84
OverallWPFlag
No -0.02 0.92
0.07 0.99
-0.01 0.87
Yes 0.18 0.87 0.22 1.05 0.10 0.95SchoolGroup(UK)
Independent 0.01 0.95
0.09 1.08
-0.06 0.95 State -0.01 0.90 0.07 0.95 0.03 0.82
SchoolGroup(International)
UK 0.00 0.92
0.08 1.00 ** -0.01 0.87
International -0.03 1.00 -0.32 0.80 -0.03 1.04#SignificancedifferenceinbetweenmeansistestedbyT-testsforindependentsamples.
8. Discussionandlimitations
TheprojectaimedtoevaluatetheimpactoftestpreparationforUniversityadmissionstestson testperformancewithdata fromOxfordUniversity.The topic isofparticularrelevancebecauseanincreasingnumberofundergraduateprogrammes–nationallyinthe UK but also internationally - require applicants to take an admissions test. TheoutcomesofthesetestsareconsideredforadmissiontoUniversityinadditiontopriorschoolattainmentmeasuressuchasGCSEscores.Howstudentsprepareforadmissionstestsandifthispreparationhasanimpactontheirperformanceinthetestishoweveranunder-researchedarea.Particularly,theprojectaddressedthefollowingoverarchingresearchquestions:
61
- Towhatextentdostudentsprepareforuniversityadmissionstestsandwhatisthe nature of this preparation amongst students with different backgroundcharacteristics?
- What are the connections between student characteristics, including priorqualifications,andtestpreparationandperformance?
- Is the time of application connected with preparation for, and betterperformancein,tests?
- Whatlearningstrategiesareconnectedwithbettertestperformance?
In order to address these questions the first phase of the project explored therelationships between student characteristics and test performance with OxfordUniversity Admissions data from 2010 to 2015. Drawing on information from theBrasenoseCollegesurveyonhowstudentsprepareforadmissionstests,findingsfromaliterature review and qualitative interviewswith a sample of students, a surveywasdeveloped. Itasked for informationonhow far inadvancestudentsstartedpreparingforthetest,thesourcestheyusedfortestpreparation,thesupporttheyreceivedfromvariousagentsandthestrategiesstudentsemployedtoprepareforthetest.Thesurveywas administered to students who commenced their studies at Oxford University inOctober2017.
Limitations
Inconsideringtheresults,itisimportanttobearinmindsomeofthelimitationsoftheresearchthatmayaffectourinterpretationofthem.
Althoughitwouldhavebeenvaluabletobeabletocomparethepreparationstrategiesbetweenstudentswhowereacceptedtotheircourseandthosewhowerenot,thiswasnotpracticallypossible,andthusthesurveysamplewaslimitedonlytothosewhoarecurrentstudents.
Thesestudentsthusrepresentalimitedbreadthoftheentireapplicantpoolandhenceamorehomogeneousgroup,particularlyintermsoftheirrangeoftestresults.It isalsothecase thatsomeof theresultsmightbeunderestimated(e.g.OverallWPFlag,BMEstatus),andthismayaccountforthelackofassociationseeninsomeareas,forexamplebetweentestresultsandschoolhistoryofapplying.
It is also important to note that the survey is based on self-reportedmeasures, andthere may be some differences between groups in the way in which they representthemselves-forexamplefemalesmaybelikelytobemorereservedintheirresponsesthanmales.
Neverthelesstherearekeyfindingsfromthisprojectwhicharediscussedbelow.Theseseemtobeallthemoreimportant,becausetheyareobserveddespitethesmallerandmorehomogeneousgroupofon-coursestudentsassessedinthisstudy.
62
Discussionofkeyfindings
Towhatextentdostudentsprepare foruniversityadmissions testsandwhat isthe nature of this preparation amongst students with different backgroundcharacteristics?Thereisevidencethatnearlyallstudentspreparetotakethetesttosomeextent.Onlineresourceswerecitedasfrequentlyusedbyallstudents.Whilethemostpopularonlinesource used was the Oxford University Website, students draw substantially on anumber of other sources to prepare for the test, such as the official test providerwebsiteandTheStudentRoom,withthemajorityofthosewhousedthemfindingthemtobequiteorextremelyhelpful.Moreover,studentsusedotherstrategiesandresourcestoprepareforthetest,suchasreadingcommerciallypublishedtestpreparationbooks,subject/syllabus books, attending Oxford Open Days and talking to current Oxfordstudentsorformerstudentswhotookthetest.
Allstudentspreparefortheadmissionstestindependentoftheirschooltypeandsocio-economic background.What is interesting though, is that the amount of support thatstudentsreceiveinordertoprepareforthetestsdiffersbythosecharacteristics.Mostnotably,studentsfromstateschoolsandthosewithanOverallWPFlagseemtohavetheleastaccesstosupport.Theyreceivedfeedbacklessoftenonthepasttestpaperstheypracticedandwerelesslikelytohavetalkedtoformerpupilsattheirschool/collegeorformerorcurrentOxfordstudentswhotookthetest.Itcanbeassumedthattheseformsofsupportarelessreadilyavailabletothem,ratherthantheyarelesslikelytodrawonthem.Inadditionstudentsfromstateschoolsaresomewhatlesslikelytobeencouragedto apply to Oxford and have fewer opportunities to prepare for the test in schoolcontexts,suchasteachermentoringorschoolinitiatedworkshops.Feweropportunitiesto prepare and weaker support networks may also be the reason why state schoolstudentsstartpreparing later forthetest.This istheonlyvariablerelatingtosupportthat is significantly positively related to test performance when other preparationstrategies are considered (both for TSA and BMAT) and when student backgroundvariables are controlled for (TSA only). This finding is in linewith previous researchthatevidencesthepositiverelationshipbetweentestpreparationandtestperformance(e.g.Bangert-Drowns,Kulik, andKulik1983;Bunting andMooney2001;Messick andJungeblut1981;Powers1986;Sturman2003).However,astheseresultsemergeinthegroup of on-course students only, it is reasonable to assume that the relative lack ofpreparation opportunities and support networks – through schools and former testtakers–playsanevenbiggerroleatanearlierstageintheselectionprocess.
What are the connections between student characteristics, including priorqualifications,andtestpreparationandperformance?Unsurprisingly, A-Level results are related to test scores for the applicant pool.However, when looking at on-course students only who took the TSA no significant
63
relationship was found anymore. This means, A-Level results seem to be highlyassociated with TSA scores and hence of being admitted to a course. The on-coursestudentsthenarearelativelyhomogeneousgroupintermsofA-Levelresultsandtestscores for which no significant relationship between A-Levels and test scores wasfound.
Withinthegroupofon-coursestudentswhotooktheBMAT,A-Levelswerestillhighlyassociatedwithtestperformance.ItseemsthatamoreheterogeneousgroupintermsofA-Levels(andpossibletestscores)hasbeenacceptedtocoursesthatrequiretheBMAT.Althoughnotpossiblewiththedataathand,itwouldhavebeeninterestingtoevaluatewhethertheyareinfactpredictiveofsuccessinthecoursesthemselves.
BME applicants, female applicants and those from deprived socio-economic areasperformedlesswellinboththeTSAandBMAT,othercharacteristicsbeingequal,whichconsequentlyaffectsthechancesofbeingacceptedtoacourse.Withinthegroupofon-coursestudentsthesizeoftheeffectsofthesecharacteristicshasdecreased,butlargelyremain to present as disadvantaging factors in TSA and BMAT performance. Theseresultscouldonlypartlybeconfirmedwiththemorerecent2017/2018studentcohort,possiblyduetotherestrictedsample.Here,femalestudents,thosefromdeprivedsocio-economicareasandfromstateschoolsperformedlesswellontheTSA.However,whenmultiple background characteristics and test preparation strategies are taken intoaccount,associationsbetweengender,socio-economicareasandtestperformancearelargelyexplained.ItisalsoworthnotingthatforTSA,testpreparationseemstocounterthe relative disadvantage associated with socio-economic deprivation, state schoolattendanceandgender.
ForBMAT,thepicture looksslightlydifferent.Students fromdeprivedsocio-economicareas, state schools and with lower A-Level results performed less well on the test.However,afteraccountingformultiplebackgroundcharacteristicsandtestpreparation,being female, from a state school and having lower A-Level results remainedsignificantly negatively associated with test performance. There is also evidencestudents’backgroundcharacteristicsexplainalargervarianceofBMATscoresthanTSAscores.
We did not find relevant evidence that student background variables mediate ormoderatetheassociationsbetweentestpreparationandtestscores.
Theseresultshaveimplicationsfordiscussionsaroundequityinadmissionstests.WhydoBME, femaleandapplicants fromsocio-economicallydeprivedareasunderperforminTSAandBMATdespiteequalGCSEandA-Level results?Similarly,whydostudentsfromstateschoolsunderperformincomparisontotheirpeersatindependentschools,despite GCSEs, A-Levels and other background characteristics being equal?With thedataathandwecannotanswerthesequestionsandtheremightbeothercharacteristicsnotincludedinouranalysisthatareabletoexplainthesegapsintestperformance.
64
Isthetimeofapplicationconnectedwithpreparationfor,andbetterperformancein,tests?ResultsprovideevidencethatapplicantswhoapplyearlierperformhigheronTSAandBMAT with other characteristics such as prior attainment, demographic and socio-economic characteristics being equal. However, for on-course students, it is the timestudents started preparing in advance for the test that positively predicts testperformanceinbothTSAandBMAT,ratherthanthetimeoftheirapplication.
Whatlearningstrategiesareconnectedwithbettertestperformance?Results show that students preparing for the BMAT test applied the three learningstrategies – control strategies,memorisation and elaboration –more frequently thanstudentspreparingfortheTSA.Also,femalestudentsusememorisationstrategiesmoreoften than male students. As both BMAT and TSA have major thinking skillscomponents,memorisationmaynotbethebeststrategyandmaypotentiallyberelatedtotheconsistentlylowertestresultsforfemalestudentsinBMAT.
Implications
There are practical implications of understanding these results. If preparation foradmissions tests has a positive impact on test performance, clearly those with lessopportunities andnetworks supportingpreparation are at a disadvantage.Advantagecanbegainedfrompractice,andthisneedstobecommunicatedexplicitlytothosewhowilltakethetests.Itseemsclearthatmoreadvantagedstudentsareprofitingfromthisknowledgealready,anditisfairthatthisknowledgebeshared.
Therearepracticalformsofoutreachthatmaybehelpful,includingdevelopingclearerguidance around the efficacy of preparation, and the importance of starting suchpreparationwell in advanceof the admissions round.This should includeadvice thattheattempttomemoriselotsofinformationmaynotbethebeststrategytoprepareforadmissions tests. Theremay also be consideration of how further ‘demystification’ oftheadmissionsprocesstoOxfordcanbetakenforward–clarifyingboththetimingandnatureoftheprocessesinvolved.
Moreover,previousresearchhasshownthattestpreparationaffectstestspecificskillsrather than general intelligence or even general subject knowledge (Arendasy et al.,2016).Consequently,theremayalsobemeritinreviewingtheemphasisplacedontestsintheadmissionsprocess,andwhethertheyareputtingalreadyadvantagedsubgroupsofstudentswithmoreopportunitiesandsupportforpreparationfurtherahead.
65
References
Adey, P.S. & Shayer, M. (1994). Really raising standards: Cognitive intervention andacademicachievement.London:Routledge
Arendasy,M.E.& Sommer,M. (2013).Quantitative differences in retest effects acrossdifferentmethodsusedtoconstructalternatetestforms.Intelligence,41(3),181–192.
Arendasy, M.E., Sommer, M., Guiterrez-Lobos, K., & Punter, F.J. (2016). Do individualdifferences in test preparation compromise the measurement fairness ofadmissiontests?Intelligence,55,44–56.
Baird, J.-A.,Hopfenbeck,T.N.,Elwood, J.,Caro,D.,&Ahmed,A.(2015).Predictability inthe Irish Leaving Certificate. Oxford University Centre for EducationalAssessmentandQueen’sUniversity,Belfast.
Bangert-Drowns,R.L.,Kulik, J.A.,&Kulik,C.-L.(1983).Effectsofcoachingprogramsonachievementtestperformance.ReviewofEducationalResearch,53(4),571–585
Black,B.(2012).AnoverviewofaprogrammeofresearchtosupporttheassessmentofCriticalThinking.ThinkingSkillsandCreativity,7,122–133.
Brinkmann,S.&Kvale,S.(2015).Sevenstagesofaninterviewinvestigation(PartII),in:S. Brinkmann (Ed.), InterViews: Learning the craft of qualitative researchinterviewing.Sage,LosAngeles,London,NewDelhi,Singapore,WashingtonDC.
Bunting,B.P.&Mooney,E. (2001).Theeffectsofpracticeandcoachingontestresultsfor educational selection at eleven years of age.Educational Psychology, 21(3),243–253.
Emery, J.L. (2009). The predictive validity of the BioMedical Admissions Test forpreclinicalexaminationperformance.MedicalEducation,43(6),557-564.
Flyvbjerg, F. (2006). Five misunderstandings about case-study research. QualitativeInquiry,12(2),219-245.
Ferguson,E.,James,D.,&Madeley,L.(2002).Factorsassociatedwithsuccessinmedicalschool:Systematicreviewoftheliterature.BritishMedicalJournal,324,952–957.
Fisher,A. (2005). ‘ThinkingSkills’ andadmission tohigher education.Reportpreparedfor the University of Cambridge Local Examinations Syndicate. Cambridge:UCLES.
Foley, L.J. (2012). Constructing the respondent. In: J. F Gubrium, J. A. Holstein, A. B.Marvasti, & K. D. McKinney (Eds.), Handbook of interview research: Thecomplexityofthecraft.ThousandOaks,CA:Sage,pp.305–315.
Gallacher, T., McElwee, S., & Cheung, K. (2017). BMAT 2015 test preparation surveyreport.CambridgeAssessment–AdmissionsTesting.WorkingPaper.
66
Griffin,B.,Carless,S.,&Wilson,I.(2013).Theeffectofcommercialcoachingonselectiontestperformance.MedicalTeacher,35(4),295-300.
Hausknecht,J.P.,Halpert,J.A.,DiPaolo,N.T.,&MoriartyGerrard,M.O.(2007).Retestinginselection:Ameta-analysisofcoachingandpracticeeffectsfortestsofcognitiveability.JournalofAppliedPsychology,92,373–385.
Higgins,S.&Hall,E.(2004).Pickingthestrawberriesoutof the jam:Thinkingcriticallyaboutsystematicreviews,narrativereviewsandmeta-analysis.BERAConferencePaper,UMISTSeptember2004.
Kulik, J.A., Bangert-Drowns,R.L.,&Kulik, C.-L.,C. (1984). Effectiveness of coaching foraptitudetests.PsychologicalBulletin,95(2),179-188.
Kirkup,C.,Wheater,R.,Morrison,J.,Durbin,B.,&Pomati,M.(2010).Useofanaptitudetestinuniversityentrance:avaliditystudy.BISResearchPaperno26(London).
Marsh,H.,Haug,K.-T.,Artelt,C.,&Baumert,J.(2006).OECD’sbriefself-reportmeasureof educational psychology’s most useful affective constructs: Cross-cultural,psychometriccomparisonsacross25countries. International JournalofTesting,6(4),311–360.
Messick, S. & Jungeblut, A. (1981). Time and method in coaching for the SAT.PsychologicalBulletin,89(2),191–216.
OECD(2010).PISA2009results:Learningtolearn–studentengagement,strategiesandpractices(VolumeIII).http://dx.doi.org/10.1787/9789264083943-en
O’Hare,L.&McGuinness,C.(2015).Thevalidityofcriticalthinkingtestsforpredictingdegree performance: A longitudinal study. International Journal of EducationalResearch,72,162–172.
Peers, I.S., & Johnston, M. (1994). Influence of learning context on the relationshipbetween A-level attainment and final degree performance: A meta-analyticreview.TheBritishJournalofEducationalPsychology,64(11),1–18.
Powers, D.E. (1985). Effects of test preparation on the validity of a of a graduateadmissionstest.AppliedPsychologicalMeasurement,9(2),179–190.
Powers, D.E. (1986). Relations of test item characteristics to test preparation/testpracticeeffects:aquantitativesummary.PsychologicalBulletin,100(1),67–77.
Powers,D.E.&Swinton,S.S.(1984).Effectsofself-studyforcoachabletest itemtypes.JournalofEducationalPsychology,76(2),206–278.
Rubin, H.J. & Rubin, I.S. (2005). Qualitative interviewing: The art of hearing data.ThousandOakes:SagePublications,Inc.
Säälik,Ü.,Nissinen,K.,&Malin,A.(2015).Learningstrategiesexplainingdifferencesinreading proficiency. Findings of Nordic and Baltic countries in PISA 2009.LearningandIndividualDifferences,42,36-43.
67
Slack,W.&Porter,D.(1980).TheScholasticAptitudeTest:Acriticalappraisal.HarvardEducationalReview,50(2),154–175.
Sturman, L. (2003). Teaching to the test: Science or intuition? Educational Research,45(3),261–273.
68
Appendices
AppendixA.TSAandBMATtables
Table A1. Average TSA standardised scores by CGSE quartiles and school typegroup(homeapplicantsonly)
GCSEquartiles
Schoolgroup Q1 Q2 Q3 Q4Independent -0.15(410) 0.18(838) 0.42(1169) 0.61(1396)
Other -0.2(123) 0.27(52) 0.65(30) 0.49(26)
State -0.31(1847) 0.13(1520) 0.38(1165) 0.6(1031)
Note:Samplesizeinparenthesis.
TableA2.AverageTSAstandardisedscoresbyCGSEquartilesandgender(homeapplicantsonly) GCSEquartiles
Gender Q1 Q2 Q3 Q4Female -0.52(847) -0.03(981) 0.19(983) 0.42(1110)
Male -0.14(1533) 0.27(1429) 0.55(1381) 0.75(1343)
Note:Samplesizeinparenthesis.
Table A3. Average TSA standardised scores by CGSE quartiles and BMEbackground(homeapplicantsonly)
GCSEquartiles
BMEbackground Q1 Q2 Q3 Q4White -0.14(1616) 0.2(1730) 0.48(1668) 0.69(1682)
Notknown/Refused -0.28(164) 0.25(179) 0.22(183) 0.48(194)
BME -0.65(600) -0.06(501) 0.24(513) 0.38(577)
Note:Samplesizeinparenthesis
69
TableA4. Frequenciesofbackgroundcharacteristics from the interviewsampleandanalyticalsample
Characteristics Sample(46) Analyticalsample(10)
N % N %Female 19 41.3 5 50Male 27 58.7 5 50Subjectstudy(collapsed)
Humanities 16 34.8 1 10MedicalSciences 7 15.2 3 30
MPLS 14 30.4 3 30SocialSciences 9 19.6 3 30
Subjectstudy(detailed)
Economics&Management 2 4.3
EngineeringScience 4 8.7 1 10English 8 17.4
English&ModLanguages 3 6.5
ExperimentalPsychology 1 2.2 1 10History 4 8.7 1 10
Law 2 4.3 1 10Mathematics 6 13
Medicine 6 13 1 10ModernLanguages 1 2.2
Physics 4 8.7 1 10PPE 5 10.9 2 20
Feestatus
UK 40 87 9 90EU 3 6.5 0
non-EU 3 6.5 1 10SchoolType
Independent 20 43.5 6 60State 19 41.3 3 30Other 7 15.2 1 10
70
TableA5:Descriptivestatisticsofsamplesusedintheanalysis
Respondents Non-Respondents TSA BMAT
N 783 2,737 194 70Variables % % % %
Female 57.6 47.1 57.7 80BME 16.5 18.1 12.4 36.1UK 75.5 87.9 71.6 87.1EU 11.5 7.4 13.4 8.6Non-EU 13 13.7 14.9 4.3OverallWPflag 14.2 10.9 12.2 11.5IndependentSchool 36.6 46.6 31.4 38.7InternationalSchool 22.6 18.8 27.3 10.4ALevelBand
LessthanA*A*B,A*ABorAAB 0.2 0.7 AAB 2.0 1.0 1.6 AAA 7.1 6.8 7.8 A*AB 3.4 1.7 2.3 A*AA 17.9 20.6 18.8 14.8A*A*B 0.7 1.1 A*A*A 22.9 26.0 21.9 183A*s 13.9 12.1 38.3 47.54A*s 30.9 28.7 9.4 19.7
5A*sorbetter 1.1 1.2 Practisingpastpapers*
Practicedpastpapers 98.4 97.1Usedmarkscheme 88.4 88.2Receivedfeedback 16.2 32.4
Practicedundertimedconditions 84.3 89.7Onlineresources*
OxfordUniversitywebsite 60.3 32.9Officialtestproviderwebsite 63.9 87.1
Otherwebsitesdedicatedtothetest 11.9 38.6Yourschool/collegewebsite 5.2 4.3
Anotherschool/collegewebsite(s) 0.5 2.9TheStudentRoom 32 35.7
Otherstudentforumwebsites 2.1 4.3YouTube 4.1 10Other 6.7 7.1
Familysupport* 30.1 28.6Schoolsupport*
Theydiscouraged 3.6 2.9Theydidnotencourage 19.7 21.4
Theyencouraged 58.0 57.1Theysuggested 16.1 18.6
Provideworkshops/classes 30.6 42.9Useclassperiods 1.6 2.9
Mentoring 21.2 24.3Subject-specificquestions 20.7 44.3
Otherstrategiesandresourcesused* Commerciallypublishedtest
preparationbooks 34 65.7
Commercialpreparationcourses 3.6 20Servicesofanexternalagency 0.5 7.1
Privatetutorials 3.6 4.3Subjectsyllabus/textbooks 5.2 65.7
71
Formerpupilsatmyschool/college 7.7 22.9CurrentorformerOxfordstudents 9.8 17.1
OxfordOpenDays 13.9 21.4Other 1 1.4
Respondents Non-Respondents TSA BMAT
Variables M SD M SD M SD M SDTSAscore(standardised) -0.07 1.00 0.03 1.00 0.00 1.00 BMATscore(standardised) 0.19 1.09 -0.12 0.92 0.00 1.00Timelapsed 13.05 52.61 6.39 6.14Time preparation (0 'Not at all' - 5'Morethan1yearbefore') 3.08 0.87 3.36 0.85
Amountpracticingpastpapers(0'Verylittle'-3'Agreatdeal') 1.95 0.83 2.26 0.84Usefulnessinwaysofpractisingpastpapers(0'Notatallhelpful'-3'Extremelyhelpful')
Markscheme 2.28 0.71 2.32 0.68Feedback 2.35 0.71 2.27 0.70
Practicingundertimedconditions 2.48 0.56 2.69 0.47Overalluseonlineresources(1'Verylittle'-4'Agreatdeal') 2.72 1.06 3.16 0.94Helpfulnessonlineresources(0'Notatall/notveryhelpful-3'Extremelyhelpful')
OxfordUniversitywebsite 0.73 0.57 0.91 0.60Officialtestproviderwebsite 1.11 0.60 1.46 0.62
Otherwebsitesdedicatedtothetest 0.91 0.79 1.19 0.68Yourschool/collegewebsite 1.50 0.71 0.33 0.58
Anotherschool/collegewebsite(s) 0.00 1.00 0.00TheStudentRoom 0.68 0.57 0.52 0.59
Otherstudentforumwebsites(besidestheStudentRoom) 0.75 0.50 0.67 0.58
YouTube 1.38 0.52 0.86 0.38Other 0.38 0.51 0.60 0.55
Learningstrategies Controlstrategies -0.19 0.97 0.52 0.56
Memorisationstrategies -0.35 0.78 0.97 0.75Elaborationstrategies -0.18 0.90 0.51 0.71
* Percentages relate to students who indicated that they used this particular strategy, resource orreceivedaparticularsupport.
72
AppendixB
Table A6: Regression results for combined BMAT test scores (Medicine andBiomedicalSciences)andBMATtestscoresforMedicineonly
VariablesBMAT BMAT-Medicineonly
Coef. SE Sig. Coef. SE Sig.
Intercept -63.25 11.03*** -64.09 11.06***Gender(Female=1) -0.25 0.02*** -0.23 0.02***Schooltype(Other=1) 0.43 0.30 0.41 0.30Schooltype(State=1) -0.04 0.02 -0.04 0.02BME(don'tknow/refuse=1) -0.29 0.05*** -0.31 0.05***BME(Yes=1) -0.27 0.02*** -0.28 0.02***ACORNFlag -0.11 0.04** -0.11 0.04**POLAR3Flag 0.01 0.03 0.00 0.03Pre16SchoolFlag -0.05 0.03 -0.03 0.04Post16SchoolFlag -0.05 0.03 -0.03 0.03OverallWPFlag -0.13 0.05** -0.15 0.05**Applicationtime -0.45 0.19* 0.00 0.00*GCSEA*s(zscored) 0.19 0.01*** 0.19 0.01***A-levelA/A*s(zscored) 0.21 0.01*** 0.21 0.02***Schoolhistoryofapplication 0.01 0.01* 0.00 0.00*UcasCycle 0.03 0.01*** 0.03 0.01***A-LevelArt -0.01 0.06 0.02 0.06A-LevelBiology 0.06 0.06 0.02 0.06A-LevelBusinessStudies 0.17 0.28 0.18 0.28A-LevelChemistry 0.31 0.14* 0.40 0.28A-LevelComputing 0.26 0.33 0.26 0.33A-LevelEconomics 0.24 0.07** 0.23 0.07**A-LevelEnglish 0.22 0.14 0.22 0.14A-LevelEnglishLanguage 0.04 0.17 0.11 0.18A-LevelEnglsihLiterature -0.07 0.05 -0.09 0.05A-LevelFurtherMathematics 0.27 0.04*** 0.26 0.04***A-LevelFurtherAddidional 0.00 0.00 0.00 0.00A-LevelGeography -0.11 0.06 -0.12 0.06*A-LevelHistory -0.01 0.04 -0.01 0.04A-LevelLanguages -0.05 0.03 -0.05 0.03A-LevelLaw -0.63 0.37 -0.64 0.37A-LevelMathematics 0.12 0.03*** 0.11 0.03**A-LevelOther 0.09 0.06* 0.14 0.07*A-LevelPhilosophy 0.25 0.15* 0.31 0.15*A-LevelPhysics 0.11 0.03*** 0.11 0.03***A-LevelPsychology 0.07 0.06 0.08 0.06A-LevelReligiousEducation 0.17 0.07* 0.17 0.07**A-LevelScienceOther -0.31 0.20 -0.25 0.21A-LevelSocialScience -0.14 0.21 -0.16 0.21p<0.05(*),p<0.01(**)andp<0.001(***)
73
AppendixC.BrasenoseCollegesurvey:selectedquotes
B1.Amountofpreparation‘I sat an aptitude test but didn't really do anything in the way of preparationbecause Iknew itwouldbeon topics Ihadn'tencounteredbeforeandwouldbelookingtoassessmymoregeneralskillsasahistorian,thatIcouldn'treallyrevise’(Arts&Humanities,2014).
B.2Supportfromschools‘TooktheTSA.Schoolhadzeroinput’(SocialScience,2013).
‘Ihad to take theBMAT,myschooldidn'thavea lotof input inmypreparation’(Sciences,2014).
‘…my schoolheld classesonce aweekgoingover them (past testpapers) for acoupleofmonthsbefore(thetestwastaken)’(Sciences,2014).
‘Myschoolgaveseveralhalfhourmeetingstodiscussgeneralwaysofapproachingthe…test’(Arts&Humanities,2014).
‘HadtotaketheTSA–hadacoupleofsessionsatschoolandpreparedusingthepastpapers/questionbooks’(SocialSciences,2013).
‘… our teacher gave us a powerpoint slide detailing the types of questions thatmight arise and how best to logically go about answering them’ (Joint Degree,2015).
‘preparedbydoingthepastpapers,schoolhassome(notveryuseful)classesonit’(ArtsandHumanities,2015).
‘Myschoolgavemesomeclasses inadvancebutthesewereof littleuse’(Arts&Humanities,2015).
‘Schoolheldworkshopsbutwerenotuseful’(Sciences,2015).
‘My schools was very supportive but couldn’t do too much to help’ (SocialSciences,2013).
‘Schoolhadnoinputbecausetheydidn’treallyunderstandwhatitwas…’(Arts&Humanities,2014).
‘…they(schoolteachers)weren’tquitesureexactlyhowbesttohelpallthetimeastheywereunfamiliarwiththesystem(Sciences,2014).
‘Iwasvery luckybecausemyschoolpreparedmewell forthetest, Idida lotofpracticetestsandmyteacherwaswillingtomarkthemandtalkmethroughsomefeedback’(Arts&Humanities,2015).
‘… a couple of meetings with a maths teacher at my school helped me to gothroughthem(practicemathsquestions)’(Science,2013).
74
‘Ididpastpapersandhadfeedbackfromahistoryteacher(…who)toldmewhereIwasgoingwrongandhowIcouldimprove(Arts&Humanities,2014).
‘Prepared quite a lot for the MAT. School put on twice weekly lunchtimeextensionsinpreparation.Bythetimeoftakingthetesthadcompletedallofthepastpaperquestionsavailableonthewebsite’(Science,2014).
‘IwroteouteverypastHAT testandmyschoolmarkedmyessaysandgavemefeedback’(Arts&Humanities,2014).
‘Didallthepastpapers,myschoolheldclassesonceaweekgoingoverthemforacoupleofmonthsbefore’(Science,2014).
B.3Useofexampreparationresources‘MyschoolhadverylittleinputbeyondadministeringtheTSA.IpreparedjustbydoingasmanypastpapersasIcouldfind’(SocialSciences,2013).
‘Ididalloftheavailablepracticetests,butreceivednoassistancefrommyschool’(SocialSciences,2013).
‘Iboughtthegrammarbookthatstudentshereuse(IwasadvisedinBlackwells)andIwentthroughit,makingnotes’(Arts&Humanities,2013).
‘I was given very little guidance bymy school so resorted to asking students Iknewhad studiedatOxfordabout thebestway toprepare’ (Arts&Humanities,2013).
‘Myschooldidnotorganisemuchtohelpme,however,aformerOxfordundergradwhonowworksatmyschoolgavemeafewpointersonwhattolookfor’(Arts&Humanities,2013).
‘ImainlypreparedbyrevisingmyGCSEsciencecourse’(Sciences,2014).
‘IdidtheHATsoIjustprintedoffeverypastpaperandmadesurethatIknewmyA level history syllabus so I could draw upon pre-existing knowledge’ (Arts &Humanities,2013).
‘…overthehalftermholidayIensuredIhadasoundknowledgeofmyAStopics’(JointDegree,2013).
‘Myschoolprovidedmewithallgrammartextbooks Ineed formyGermantest’(JointDegree,2013).
‘AsIdoModernLanguages,Ialsogenerallyrevisedgrammar’(Arts&Humanities,2013).
‘…thecontent(ofthePAT)wasfarremovedfromtheASsyllabus’(Science,2013).
B.4Perceptionsofthetests‘…wasquiteintimidating,andhadlittleideaaboutwhatIwasabouttosit’(SocialSciences,2013).
75
‘Didn’t really know what to expect from it (the test). Would have liked moreguidance…’(Arts&Humanities,2013).
‘Would have liked more guidance, especially on to how it featured in theadmissionsprocess’(Arts&Humanities,2013).
‘Itwasstressfulwithdifferentrumoursabouthowmuchweightingtheresulthadonadmissionprospects…’(Sciences,2014).
‘ThetestweremuchharderthananythingIhadpreparedfor’(Arts&Humanities,2013).
‘I remember the test being very challenging – harder than themock tests – butveryinterestingatthesametime’(Arts&Humanities,2014).
‘Thetestwasmoreenjoyablethanexpected’(JointDegree,2014).
‘Ireallyenjoyedsittingitstrangely’(Arts&Humanities,2014).
‘ThetestsweretoughbutIthinkrewardingandnecessary’(Sciences,2013).
‘The paper itselfwas unique and challenging, and a test in hindsight I probablyenjoyed’(Sciences,2015).
‘Test was challenging but enjoyable having the opportunity to write somethingentirelydifferenttoA-levels’(Arts&Humanities,2015).
76
AppendixD.Interviewguide
InformationandConsentOne of the aims of this research project is to learn how students prepare for Oxfordadmissionstests.Weareinterestedtoknowabouttheexperiencesthatyouhadsothatit can inform us about the connection between test preparation and admissionsoutcomes which we hope will contribute to developing effective support for futureapplicants.Youareinvitedtotakepartinaninterviewlastingaround45minutes. Theinterviewwillbeaudiorecordedandthetranscriptionwillbesenttoyouforverificationbeforeweanalyse it.The informationyouprovidewillbekept strictly confidential andyouranswers will be aggregated with those of other participants so that you remainanonymous.Theinformationyouprovidewillbeusedforthepurposesofresearchandstudentadviceonly.TheresearchisledbyDrDanielCarofromtheDepartmentofEducation,UniversityofOxford. Should you have any questions please contact him([email protected]).For questions regarding the ethical approval process please contact the head ofDepartment of Education Research Ethics Committee, Dr Liam Gearon([email protected]).Pleaseconfirmthat,
• youhavereadandunderstoodtheinformationaboutthisstudy.• youunderstandthatyoucanwithdrawfromthestudywithoutconsequenceat
anytimesimplybyinformingtheresearcher.• you are aware of who to contact should you have questions following my
participationinthisstudy.• you understand that this project has been reviewed by and received ethical
clearance through the University of Oxford Central University Research EthicsCommittee.
• youagreetoparticipateinthisstudy.Name(Interviewee): ________________________________Date: ________________________________Signature: ________________________________ Researcher(Interviewer): ________________________________Date: ________________________________ Signature: ________________________________
77
YouQ.Whichcoursedo/didyoustudyandwhendidyoustart?Q.Whichtestdidyoutakeaspartoftheadmissionsprocessandwhenwasthat?PreparationforthetestQ.Whendidyoufindoutthatthattestwaspartoftheadmissionsprocess?
o Whereandfromwhom?o Wereyouclearaboutwhatwasinvolvedinthetest?
Q. Whowould you say helped you to understandwhat the testwas about andwhat
preparationsyouwouldneedtomake?Weretheyexperts/experienced?o Howdideachpersonhelp?o Howmuchdidyoufigure-outforyourself?
UsefulnessofPreparationQ.Didyouprepareforthetest?Q.Ifso,whatdidyoudo?
o Organizeyourstudyspaceo Useflowchartsanddiagramso Practicepastexampaperso Explainyouranswerstootherso Organizestudygroupswithfriends
Q.Whichapproachesweremostusefultoyouandwhichwerelessso?Q.Whichmaterialsdidyoufindtobethemostusefulandwhichwerelessso?o PastPaperso Preparationbookso Preparationcourseso Oxfordwebsiteo Testproviderwebsiteo SubjectsyllabusQ.Wereyourpreparationsusefulforthetestorwasitdifferenttowhatyouexpected?Q.Wastheformateasytounderstand,doandcompleteontime?LogisticssurroundingthetestQ.Didyouregisterforthetestyourself?
o Didanyoneelsehelpwiththat?Q.Didyouhavetopayafeetotakethetest?
78
o Wasitmanageable?
Q.Wherewasthetestandhoweasywasittogetthere?Q.Whatwasitliketakingthetestinthatvenue?PerceptionsofImportanceofandRelevanceofthetestQ. When youwere preparing to apply, how important did you think the testwas ingettingyouaplaceatOxford?Q.Wasthetestreferredtolaterintheadmissionsprocess?Q.Whatwouldyouradvicebeforstudentswhowilltakethetestnextyear?
79
AppendixE:SurveyInstrument
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Top Related