e-Exams The story so far…transformingexams.com/files/e-exams_Dec_2015_96dpi.pdf · e-Exams The...
Transcript of e-Exams The story so far…transformingexams.com/files/e-exams_Dec_2015_96dpi.pdf · e-Exams The...
e-Exams
Thestorysofar…(2013-2015)
TransformingExams.com
Compiled by Mathew Hillier, Dec 2015
TransformingExamsAcrossAustraliaAustralianGovernmentOfficeforLearningandTeachingNa:onalGrantID15-4747
Acknowledgements-DiamondandGoldSponsors!SupportforthisprojecthasbeenprovidedbytheAustralianGovernmentOfficeforLearningandTeaching.TheviewsexpresseddonotnecessarilyreflecttheviewsoftheAustralianGovernmentOfficeforLearningandTeachingorpar:cipa:ngins:tu:ons.Addi:onalsupportisbeingprovidedbyMonashUniversity. D
emo&Guides
Dr Mathew Hillier; Leader OLT national project. Seed leader
Monash University (was University of Queensland)
Dr Andrew Fluck; Originator of USB e-exam concept. Seed partner.
University of Tasmania
Dr Michael Cowling, Mr Kenneth Howah Seed trial site.
Central Queensland University
Dr Kim Blackmore Australian National University Assoc. Prof. Paul Newhouse Edith Cowan University Dr Matthew Bower, Prof. Dominic Verity Macquarie University Prof. Marilyn Baird (H), Mr Scott Grant (A) Monash University Assoc. Prof. Shona Leitch RMIT University Dr Ruth Geer, Mr Bruce White University of South Australia TransformingExams.com
Civil Service Exam Under Emperor Jen Tsung (fl.1022) from a history of Chinese emperors(colour on silk): Bibliothèque Nationale, Paris, France.
Is this your exam space?
Thedissonanceofitall!RealworldofworkExams
4
World Economic Forum – How will digital change your working world. https://agenda.weforum.org/wp-content/uploads/rtr2m8vm1-628x330.jpg
Exams at Monash Caufield in 2015 (mathew.hilier[at]monash.edu)
Wearefacedwithagrowingdisconnectbetweenthewayhighstakestes*ngisconductedusingpenonpaperexamsandstudents’everydayexperiencesofstudy,workandlife.
21stcenturyemployabilityskillsWaysofthinking • crea:vityandinnova:on
• cri:calthinking,problemsolving• learningtolearn,metacogni:on
Waysofworking • communica:on• collabora:on(teamwork)
Toolsforworking • informa:onliteracy• ICTliteracy• disciplineresourcesandtools
Livingintheworld • ci:zenship–localandglobal• lifeandcareer• personalandsocialresponsibility(including
culturalawarenessandcompetence)
5
http://atc21s.org/ Binkley, M., Erstad, O., Hermna, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining Twenty-First Century Skills. In Griffin, P., Care, E., & McGaw, B. Assessment and Teaching of 21st Century Skills, Dordrecht, Springer.
21Clearners
• skilleduseoftools• ac:velearningratherthanpassivereceivingofknowledge• authen:clearningexperiencesratherthancontrivedtasks• construc:onratherthaninstruc:on• task(notprocess)oriented• justin:melearning• searchnotmemorise• u:lisesocialnetworks• doesn’tknowanswerbutknowswheretofindit• Googlenotlibraries• collaboratenotcompete.
6
Elliott, B (2007)
BobbyEllio]andassessment1.5to2.0
7
Tradi:onalassessment(Assessment1.0)
Computer-basedassessment(Assessment1.5)
Tool-assistedassessment(Assessment2.0andbeyond)
Elliott, B (2007) “Assessment in the age of Web 2.0” Scottish Qualifications Authority. http://wiki.cetis.ac.uk/images/d/de/Assessment_2_v2.pdf
Ye Olde style… mostly paper-based mostly classroom-based very formalised (in terms of administration) highly synchronised (in terms of time and place) highly controlled (in terms of contents and marking).
Add computer …
21C • user-generated content, blogs, • the power of the crowd, wikipedia • data on an epic scale, • architecture of participation, easy use • network effects, • openness. OER, Mashups
Future à Web 3.0., IoT, Ubiquitous
HypeCycleforEduca:on2015• Cloud email • Mass notification • OS learning repository • E-book readers • Tablets • Mashups • Self-publishing
8
• MOOCS • Low/mid range handsets
2014
2014 Numerous technology tools are available.
Targe:ng...
• Supervised• Highstakes• Oncampus• Largescale
(imagecredit:DrFluckUTAS)
Whatwearenotspecificallyaddressinghereisoffcampus,
onlineonly,distanceeduca*on,crossins*tu*onalstudents
–thereareextraissues(later!)and
somepossiblee-solu*onstoaddresstheseneeds.
9
Whye-Exams?Ra:onale:Concerns,drivers,possiblesolu:onsfore-Exams.Atruly'wicked','messy'problemandalongroadtogetitright!...
Presentedra:onaleat2013andSeedgrantfindingsat2014,2015conferences.Moretocome…andexploredindepthinHillier&Fluck(2013)h]p://www.ascilite.org.au/conferences/sydney13/program/papers/Hillier.pdf
ta.vu/eexam-map
e-Exams:Online,Offline,OnCampusorDistance
11
Online
(net)
• Spaceissuesforins:tu:ons.• Improvedexammanagement
efficiency.• Equipment:needcomputerlabsto
caterfor2000atonceorBYO.• Moresecure:liveITmonitoring/
controlandspacesaresupervised.• Needsreliablenetwork.
• Nospaceissueforins:tu:ons.• Moreefficientexam
management.• Studentssupplyequipment.
• Lesssecure:ITmonitoringbutwiderspacesareunsupervised.
• Needsreliablenetwork.
Offline
• Spaceissuesforins:tu:ons.• Lessefficientexammanagement.• Equipment:needcomputerlabsto
caterfor2000atonceorBYO.• Moresecure:ITcontrolpossible,
spacesaresupervised.• Networkreliabilitynotanissue.
• Nospaceissueforins:tu:ons.• Lessefficientexammanagement.• Studentssupplyequipment.
• Lesssecure:nousefulmonitoring/supervision
• Networkreliabilitynotanissue.OnCampus(controlledspaces) Distance(athome)
Therearetrade-offsforanye-examsolu:on.
APossibleFutureAnevolu:onratherthanarevolu:on.Someaspectsmayoccurquickerthanothersdependingonpar:cularimplementa:on,technicalmodelschosen,socio-cultural-policyenvironmentcondi:ons.OutlookforAustralasia.(DenmarkandNorwayalreadyhave‘internetinexams’).
12Source: Hillier & Fluck (2015) http://transformingexams.com/files/hillier_fluck_2015_exam_futures.pdf
Wherewearegoing:Post-paperexamsWeneedgreaterpedagogicalflexibilityandmoreauthen:cassessmentsintheexamroom.…alignment!
13
Simula:ons,toolsofthetrade,virtualexperiments…
'Windows'sopwareviaWINE.E.g.CAD/3Dmodeling,Celes:a.
Moodlequizwithmedia(automarked).
14
Physical hardware can be connected to the internet or we can use software simulations of labs and experiments.
SimulatedLabs
VirtualImmersiveEnvironments
TAwebinarh]p://transformingassessment.com/events_1_april_2015.php15
Sco]Grant(MonashUniversity,Australia)h]p://www.virtuallyenhancedlanguages.com
VirtualImmersiveEnvironmentsTaskBasedLearningorTaskBasedLanguageLearning(TBLL).Includescommunica:onac:vi:es,usinglanguagetocarryouttasks,languageusethatismeaningfultothelearnerandhasapurpose,andcommunica:onac:vi:esthatreflectreal-lifeac:vi:eswithauthen:cmaterials.Learnbydoing:Purchasesuppliesthencooknoodles–allinChinese.
16
MoodleandOpenSimWorkingTogether
Dataflowsasifthestudentwas
doingtheac*vityintheLMS
Asetofscripts
forMoodleand
VWthatactsas
abridge.
Student
undertakes
assessmentin
thevirtual
world
SetupQuizin
theLMS.
Resultsare
storedinthein
gradebook.
Videos:TransformingAssessmentYoutubeChannel
Undertakinganassessmentac:vityintheVWini:atesdatatransferstotheLMS.
BasicExamples-Interac:veapps
18
1. Studentclicksalinkembeddedinthequiztolaunchaseparateapp.
2. Undertakesataskasinstructed.
3. Thenrespondsusingselectedresponseornumericalinputtosuit.
BasicExamples-spreadsheetinquizSpreadsheetlinkedtoquiz
19
Examples–Conversa:onSim(Monash)StudentsrespondtoaseriesstatementsviaMCQs(maybe,yes,no)withfeedbackperchoicetosimulateaconversa:one.g.Moodlelessonac:vity.
h]p://conversa:onsim.org/Nelson,R&Dawson,P(2013)Assessment-as-learning:introducingtheConversa:onSim,TAwebinar/e-AssessmentScotland,21Augh]p://transformingassessment.com/eAS_2013/events_21_aug_2013.php
Examples–ShorttextresponseStudentstypeinashortsentenceresponsewhichcanbemarkedbycomputerbasedonpa]ernmatching.AvailableinMoodlenow.Exampleques:onExamplesetngs
SallyJordan,OpenUniUK,TAwebinar,5June2013transformingassessment.com/events_5_june_2013.php
Kine:cenergywillbeatmaximumwhenatthebo]omoftheslide.
evalua:on
Examples–Confidenceques:onsConfidencebasedapproachespenaliseguessing.Studentsneedtochoosearesponseanddeclaretheirlevelofcertainty.AvailableinMoodlenow.
22
UniversityCollegeLondon
CertaintyvMarkExpected
Certaintylevelsandconsequences
TonyGardner-Medwin,UCL,TAWebinar6April2011h]p://transformingassessment.com/events_6_april_2011.php
e-ExamSystemAffordances
23
Per:nentFeatures AffordancesA'Wholecomputer'environment(OS,LMS,applica:ons…)onas:ck.
Vastlyexpandedpedagogicalscopeoverthatofabrowserwindow.
TypedstudentresponsesviaWordprocessor,constructedviaapps(humanmarked)oron-boardlearningmanagementsystemquiz(computermarked).
Catersforintroduc:ontoadvanceduses.Componentsadded/removedtosuit.Electroniccollec:onfacilitatesanaly:cs,itemresponseanalysis…
Nolivenetworkrequiredduringexam,evenforLMSques:ons.
Robust.Greatercontrol.(networkcouldbeusedforadmin)
Studentownedequipmentusedashostandlepuntouched.
Anethicalapproachtoscalability(noinvasivesopwaretoinstall)
Modular,opensourcecodebaseandcommodity'offtheshelf'components.
Leveragingpopularandsustainableprojectsforbe]erefficiency.Fully'known'(no'blackbox').Available!
OneversionworksonmostIntelbasedlaptops-Apple,'windows',Linux,thathaveaUSBport.
Onesopwareversioncanserveall.Streamlinesdevelopmentandmaintenance.
Bootable ‘live’ USB drive
Wherewearenow:PaperEquivalent(UQ)Worddoc!Ques:ontypesused:shortanswer/essay,matching,constructatable,labeladiagram/image(byfillingatable).Manualmarking.
24
Wherewearenow:PaperPlus(UTAS)Worddoc,plusapps.Ques:ondocumentwithlinkstolaunchlocalappsandresources:graphics,Scratchprogrammingtools,presenta:onslides,PDFs.Allon-boardtheUSBs:ck.‘ITinEduca:on’exams:AndrewFluck,UTAS.
25
Scratch SDK
Exam doc
e-ExamWorkflowusedinTrials
26
Postsession:retrieveresponsesandassessment
Examroomuse
ResponsesretrievedfromUSBs. Collatede-responsessenttoacademic.
CreatemasterUSB(tested)
USBsduplicatedperstudentAcademiccreatesexamlearningmaterial
Set-up:prepareexamlearningmaterials Pre-session:Studentlaptopsetup&prac:ce.
CollectUSBs(responses)
1. Studentsenterroom.
2. GivenUSB.3. Bootlaptop.4. Doexam.5. ReturnUSB.6. Leaveroom.
e-Examsystem
takesover
laptop.
Ubuntu
LiveUSB.
LibreOffice.
FirstandMostRecente-ExamsUQ
27
VETS2100 S2 2014
DENT4092 S1 2015
Used standard teaching rooms, sought rooms with tables and power sockets.
ß VETS: hand-writers sat in rows. Attempted to separate typists and hand-writers where possible. DENT: typists at the back, à hand-writers at the front.
Spaces–Triedinavariety
28
FloorPlansPurple=typistsOrange=hand-writers
VETS
DENT
BIOL OCTY
CRIM PHTY
WalkThroughofCurrente-ExamPlaxorm
2.HolddownALTthenpoweron.KeepholdingALTun:lyousee…
29
2.Poweronwhiletapping‘bootkey’(e.g.F12or…)Keeptappingun:lyousee…(orsimilar)
*Win8:firstneedtodisablesecureboot.
1. Start with the computer turned OFF. Then insert USB stick
Apple Other/Windows*
Taptaptap...Holddown
Pressandreleasepower
Pressandreleasepower
WalkThrough
3.Selectayellowicon.EFIbootor‘Windows’
Couldbeeitherone!Sojusttry.Ifyouget
Trytheotherone!30
3.SelectUSBdevice.Itmightbelabeledsomethingelseandprobablywontbefirst.
Apple Other/Windows
e-Exam
e-Exam
WalkThrough
4A.Somesystemmessagesmayappear,ifsojustwaitandsee.4B.Examsystemshouldstart.
5.Arriveate-Examsystemdesktop.
31
WalkThrough6.StudentnowtypesintheirstudentIDnumberandname.ClickStartExam.
32
WalkThrough7.Examfileopensreadytoenterexamdetailsandresponses.
33
Note:originalfilecopiedandstudentnumberprefixedtofilename.
WalkThrough8.Studenttypesresponsesintoareasindicated.
34
WalkThrough9.Remembertosave(CTRLS).WhendoneuseFile>Exit.
35
WalkThrough10.Shutdownthesystem.Whenthesystemhaspoweredoff,handintheUSBS:ck.
36
Shutdownbu]on
Ready?
37
TrialOutcomes
Resultsfrom2014-2015trialsfollow.
38
e-ExamsSeedWork2013to2015
OLTProjectleader/Presenter: DrMathewHillier,UniversityofQueenslandOLTProjectcollaborator: DrAndrewFluck,UniversityofTasmaniaOLTProjectsystemdeveloper: MarisaEmerson,UniversityofQueenslandUQcourseacademics:
Acknowledgement:SupportforthisprojecthasbeenprovidedbytheAustralianGovernmentOfficeforLearningandTeaching.TheviewsexpresseddonotnecessarilyreflecttheviewsoftheAustralianGovernmentOfficeforLearningandTeachingorpar:cipa:ngins:tu:ons.
Dr Arosha Weerakoon (Dentistry) Dr David Booth (Zoology), Elizabeth Springfield (Occupational Therapy), Katrina Williams (Physiotherapy), Prof. Malcolm Jones (Veterinary Biology), Rebekah Scotney (Veterinary Technology) and Dr Robin Fitzgerald (Criminology) Getthedemoanduserguides
h]p://transformingexams.com
UQe-ExamTrialOutcomes
40
DatacollectedfromstudentsResultsavailableuponrequest,someath]p://transformingexams.com/research.html
• 2013pre-projectonlinesurvey(UQwide:briefresultsshown)– UQstudentssurveyedabouttheirpreconcep:onsaboute-exams.
ASCILITEpaperHillier2014h]p://transformingexams.com/files/hillier_2014_ascilite_full_paper_prepress.pdf
• 2014-5Trialspre-examshortsurvey(8courses–typistsonly).– Conductedatthepre-examprac:cesetupsessions.– Covered:studentpreliminaryimpressions,technicalhardwarecompa:bility.
• 2014-5Trialspost-examextendedsurvey(8courses–resultsshown)– Conductedattheconclusionoftheexam(intheroom-allstudents).– Covered:ra:onale,studentexamexperience,reac:ontoexamsessioncondi:ons,e-examsystemimpressions,examwri:ngstrategiesandproduc:on,generalnon-examwri:ngstrategies.Resultfor2014(sixcourses)h]p://transformingexams.com/files/Hillier_2015_ascilite_fp.pdf
• 2015Analysisoftextproduc:on(DENTonly)– Marksvwordcount,typingvhandwri:ng(moretocome;languagedensity…)
ResearchStudyPhases
41
Phase 1 Institution wide online survey (see Hillier 2014, 2015). Phase 2, Step 1 e-Exam Trial Expression of interest
Typists Handwriters
Phase 2, Step 2 Pre-exam preparation survey
Phase 2, Step 3 Type the exam Handwrite the exam
Phase 2, Step 4 Post-exam survey Par:cipa:oninPhase1:approx.928respondents(Nov2013-Nov2014)Par:cipa:oninPhase2:Eightcourses(sixin2014,twoin2015)
Phase1SurveyDesign• Surveyconstructedtocoverarangeofpossibleconcerns.
• PriorresearchDermo(2009),Limetal.,(2006),MogeyandHartley(2012),Sorensen(2013),Fluck,Pullen&Harper(2009),Fluck(2011),Fluck(2013),Hillier&Fluck(2013)providedameanstoscopetheissues.
• AsurveybyDermo(2009)providedthecore.– Acknowledgethatwewouldbeusingitinadifferentmanner(preratherthenpost).
– But!weneverintendedtoreplicateit,insteadweusedthisasameansforelici:ngstudentconcernsacrossarangeofissues.
– Studentswouldlargelyberespondingspecula:velybasedontheirpreconcep*ons.(instruc:onsgiventostudentsaccordingly)
42
Phase1Ins:tu:onSurvey
Mobiledeviceownership(excludingdesktopcomputers)wasanaverageof2.3devicesperstudent(standarddevia:onof0.8).
43
94%
84%
41%
34%
0% 20% 40% 60% 80% 100%
OwnLaptop
OwnSmartPhone
OwnTablet
OwnDesktopComputer
PercentageofRespondentsThatOwnEachTypeofDevice
Phase1Theques:ons
44
Plustwoopenendedcommentques:ons
Phase1Par:cipa:on• 488#students(37%males,63%females)=1%*• 9%post-grad,remainderwereundergrads(withanevenspreadacrossyearlevels).
• 45programs,thosewithatleast10arelisted:
45
*Krejcie & Morgan (1970) state that for a population of 50,000 a random sample 381 would be sufficient to be representative in relation to opinions expressed by respondents to 95% confidence. (it wasn’t random). #analysis performed on responses received at Feb 2014. Subsequent responses up to Nov 2014 were 928.
Phase1Analysisa. Themesdrawnfromopenendedques:onson
currentlyheld‘concerns’aboute-examsandgeneralcomments.Ourfocushere!
b. Sta:s:cs*usedtoexplorethebodyofopinionrepresentedbyLikertscalesratherthanasasearchforasingletruth.Tendedtos:cktonon-parametrictests.
ReportedfurtherinHillier,M.(2014).TheVeryIdeaofe-Exams:Student(Pre)concep:ons.PresentedattheAustralasianSocietyforComputersinLearninginTer:aryEduca:onconference,Dunedin,NewZealand.Retrievedfromh]p://ascilite.org/conferences/dunedin2014/files/fullpapers/91-Hillier.pdf
46
Phase1ThemesPreconcep:ons.
47
0 20 40 60 80 100
Technicalreliability
Chea:ng
Matchwithdiscipline
Keyboardingprowess
ComputerliteracyComment Count
Emergent themes from Phase 1 survey
Thetechnologybeingunreliablestressesmeoutmorethanthethoughtofdoingtheexam–Lawstudent
ArealprogrammerwouldbelookinguptheAPIsfortheirlanguageevery:metheywantedtodosomething,buttheycan'tbecausethey'reforcedtoonlyusepaper-basednotestheyhaveonhand.It'sinfuria:ng-Computersciencestudent.
Asamatureagedstudent,IwouldfeelatadisadvantagedoingacomputerisedexamasIamnotascomputerliterateasmanyoftheyoungerstudents-Chemistrystudent.
It'struethat'computerisedexamsfavoursomestudentsmorethanothers'-i.e.,theonesthatareproficienttypistsovertheonesthataren't-butthesameistrueofpaper-basedexamina:ons,whichfavourthosewiththeabilitytoworkthroughstrongpainintheirwri:nghand–Artsstudent.
Phase1Findings:MatchtoDisciplineByprogram(major)
48See Hillier (2014) for more. SD D N A SA
Phase1Findings:IwouldliketoType
49
“Iwouldliketobeabletotypeanswersinanexam”Byprogram(major)
See Hillier (2014) for more. SD D N A SA
Phase1InterimConclusionStudentswere• Cau:ouslyop:mis:c• Justoverhalfwouldliketoseeane-examop:on.Meanagreement(3.3)“Iwouldliketobeabletotypeanswersinanexam.”
• Werea]unetothenatureoftheirdisciplineandhowtheideaofane-exammightfit.
• Thefearoftheunknown(?)espregardingtechnicalfailuresandreliability.
50
Phase2:UQe-ExamTrials2014-2015
51
Datacollectedfromstudents(opt-in)• Viapre-projectonlinesurvey(UQwide–2013-2014):
Hillier,M.(2014).TheVeryIdeaofe-Exams:Student(Pre)concep:ons.PresentedattheAustralasianSocietyforComputersinLearninginTer:aryEduca:onconference,Dunedin,NewZealand.Retrievedfromh]p://transformingexams.com/files/hillier_2014_ascilite_full_paper_prepress.pdf
• Viapre-examshortsurvey(8courses–typistsonlynext).– Conductedatthepre-examprac:cesetupsessions.– Covered:studentpreliminaryimpressions,technicalhardwarecompa:bility.
• Viapost-examextendedsurvey(8courses)– Conductedattheconclusionoftheexam(intheroom).– Covered:ra:onale,studentexamexperience,reac:ontoexamsession
condi:ons,e-examsystemimpressions,examwri:ngstrategiesandproduc:on,generalnon-examwri:ngstrategies.
– (six2014courses)h]p://transformingexams.com/files/Hillier_2015_ascilite_fp.pdf
• Analysisoftextproduc:on(DENTonly)– Marksvwordcount,typingvhandwri:ng(moretocome;languagedensity…)
Theeightcoursesinthetrials2014-2015
Condi:ons• First‘toeinthewater’trials.• Par:cipa:onwasop:onal.• Midtermexamsworth15%to20%ofthecoursegrade.Note:Splitgroup=typistsandhand-writersindifferentrooms. 52
Course Minutes DescripVon
ANIM2014AnimalBiology 45 MixedshortanswerandMCQ(type'x')[splitgroup]
BIOL2014Zoology 50 Shortanswer(Mul:plechoicesec:ondonepenonOMRsheet)[splitgroup]
CRIM2014Criminology 70 Singlelongessayresponsesec:on(andaMul:plechoicesec:ondonepenonOMRsheet)
OCTY2014Occupa:onalTherapy 90 MixedshortanswerandMCQ(type'x')
PHTY2014Physiotherapy 15 Diagnosis:watchvideoandwriteintotable.Doneinsmallgroupsof16priortoOSCE.
VETS2014Veterinarytechnology 90 Theory,mostlyshortanswer.(withinternalandexternalgroups)
CRIM2015Criminology 90 Singlelongessayresponsesec:on(andaMul:plechoicesec:ondonepenonOMRsheet)[splitgroup]
DENT2015ResearchMethodsinDen:stry 60 Theory,shortanswer,onecalcula:onques:on
TrialPhaseA]ri:on
53
Numberoftypistsateachstageofthetrial(Surveyresponses)• Notallrespondentscompletedeveryques*on.
• Anumberofstudentselec*ngtohand-writedidnotfillintheEOIandthe
post-examsurveysoareslightlyunderrepresented.
• Similarlynotalla\endeesatthepre-examset-upsessionreturnedasurvey
(~90%+did).
Steps of trial Yes Maybe Total typists Attrition No -
hand-write 1 Expression of Interest 241 241 420 2.1 Pre - before try 124 17 141 100 38 2.2 Pre - after try 112 19 131 10 52 4 Exam (after) 98 98 33 549 Tableupdatedtoinclude2015participants.Finaltypistsbasedonreturnedsurveys.
Pre-examFirstImpressionsSelectedpre-examsessionsurveyques:ons(typistsonly)Studentscametotesttheirlaptopandtrythesystemacoupleofweekspriortotheexam.
54
QuesVon N Mean SD
Thewri]eninstruc:onswereeasytofollow 140 4.0 1.0
Itwaseasytolearnthenecessarytechnicalsteps 137 4.0 1.0
Itwaseasytostartmycomputerusingthee-ExamUSBs:ck 140 4.1 1.2
IfeelconfidentIwillbeabletodothesestepsinarealexam 138 4.0 1.1
Thesopwarewithinthee-ExamSystemwaseasytouse 137 4.1 1.1
Inowfeelrelaxedabouttheideaofusinge-Examformyupcomingexam 138 3.8 1.1
Iwouldliketouseacomputerforexamsinthefuture*(newin2015) 32 4.1 0.9
Updatedtoincludes12015results.
Bars represent medians. Means shown for clarity. Strongly Disagree Strongly Agree
TrialPhase:Pre-examSurvey
55
Datacollectedfromstudentsatpre-examset-up/prac:cesessions(2014-2015).Beforetryinge-exam Apertryinge-exam
TrialPhase:Pre-examSurveyOpentextcomments–concernsandpraise[2014]
56
0 5 10 15 20 25
fear of technical failure/crash remember procedure/use difficulties
scrolling/touchpad use power/battery
fear of data loss drawing/diagram difficulty fear of computer damage
newness/unfamiliarity security
panic eye strain
fail the exam rules/contingency
general positive/praise good ease to use
convenient fence sitting
0 5 10 15 20 25
Goodconcept.Providesanalterna:vetowri]enexams.Easytonavigate.
Simplyadifferentformatofexam.Rememberingthisfortheexam.
Thatmycomputermayfreezeornotworkproperly.Randomtechnologicalmalfunc:on.
Fairlysurei'llusethelaptopbutjustneedtothinkaboutitali]lemore.
AmIallowedtouseamouse?Willitaffectthesystem?Scrollingopposite.
Availabilityofpower.Ba]eryrunout.
Doingsomethingwrongandpanicking
Woulditsavemyanswersproperly.
Numberoflaptopspassedandtechnicalissues[2014].
So…wehadapoolofloanlaptops.UpgradetonextversionofOSwillhelptoo.
Pre-examLaptopTes:ng
57
80%
20%
compatablemachines
incompatablemachines
0 2 4 6 8 10 12
notcompa:ble/unknown
graphicsdriverissues
manualstart
screenresolu:onissues
needclearerinstruc:ons
ba]erydrain
usererror
hardwarefault/keyboardbroken
save-asproblem(defaultsave
difficulty/complexityinset-up
slowtoboot
‘Wintel’LaptopConfigura:onTherequiredsetngadjustmentsappliedtoallowthelaptoptobootfromthee-ExamUSBs:ck(version12.04).Collatedsemester22014onward.
58
Note: Apple laptops do not require adjustment. To do: Need to collate from earlier sessions.
BrandsandOpera:ngSystemsBYOlaptopstock
59
Able to boot from e-Exam USB stick
Estimated Battery Life in Hours
Hours
Count Includess12015results–8cohorts.
Phase2:UQe-ExamTrials2014-2015
60
Datacollectedfromstudents(opt-in)• Viapre-examprojectonlinesurvey(UQwide):
Hillier,M.(2014).TheVeryIdeaofe-Exams:Student(Pre)concep:ons.PresentedattheAustralasianSocietyforComputersinLearninginTer:aryEduca:onconference,Dunedin,NewZealand.Retrievedfromh]p://transformingexams.com/files/hillier_2014_ascilite_full_paper_prepress.pdf
• Viapre-examshortsurvey(8courses–typistsonly).– Conductedatthepre-examprac:cesetupsessions.– Covered:studentpreliminaryimpressions,technicalhardwarecompa:bility.
• Viapost-examextendedsurvey(8courses–allstudents–next)– Conductedattheconclusionoftheexam(intheroom).– Covered:ra:onale,studentexamexperience,reac:ontoexamsession
condi:ons,e-examsystemimpressions,examwri:ngstrategiesandproduc:on,generalnon-examwri:ngstrategies.
– 2014(sixcourses)h]p://transformingexams.com/files/Hillier_2015_ascilite_fp.pdf
• Analysisoftextproduc:on(DENTonly)– Marksvwordcount,typingvhandwri:ng(moretocome;languagedensity…)
Typistsandhand-writersbycourse
61
Typed15%
Handwrote85%
Combinedallcohorts
Cohort Typed HandwroteCRIM2014 25.4% 74.6%PHTY2014 18.8% 81.2%VETS2014 12.4% 87.6%ANIM2014 4.4% 95.6%OCTY2014 11.1% 88.9%BIOL2014 9.9% 90.1%CRIM2015 12.1% 87.9%DENT2015 28.8% 71.2%
Proportionoftypistsandhandwritersineachoftheeightcohorts2014-2015
0% 20% 40% 60% 80% 100%
ProporRonoftypistsandhandwritersbycohortTyped Handwrote
Genderra:os:Typistsandhand-writers
62
Typists 15%
Hand-writers 85%
All Eight Cohorts
ModeMWU 27737.5
Z -1.516Sig.(2-tailed) 0.13
Finding: Gender did not play a role in the choice to type. Includess12015results–8cohorts.
Reasonsfortypingtheexam(2014)
63
(added30October2014)
[2014]
Phase2Post-examTypist’sComments
64
Itiscleaner,ImakelotsofmistakeswhenI'mwri:nganditusuallyendsinlotsofscribbleseverywhere.
Ihaveterriblehandwri:ng.Feltbadaboutit. Icouldgetinfodownfaster
andexaminercouldreadit.
Quickertypingandtheabilitytoeditorcompletelydeletemyanswerwithoutcompromisingonspace.
Youcanwriteasmuchasyouotherwisewouldbutdon'tgetasorehandwhentyping.
Typingismorenaturalforme.IthinkbestwhenIamtyping.
[2014]
Reasonsforhandwri:ngtheexam(2014)
65
(added30October2014)
[2014]
Phase2Post-examHand-writers
66
It'seasiertohandwrite.Thoughprobablynoteasierforyoutoreadmywri:ng.
Threeyearsofpriorexamswri:ngsos:ckwithwhatyouknow.
Lazytobringlaptop.
Ifeltmorecomfortablehandwri:ngasnothingcangowrong&Iwasn'trelyingonthecomputersystemtocompletemyexam.
IthinkmoreaboutwhatI'mwri:ngwhenIhandwritebutmyhandgetssoreanditisn'tfast.
Iwasini:allyplanningtotypethisexambutdecidedagainstitduetotheunpredictabilityofmachines
I'maslowtyperandfeeldisadvantaged.
Iamworriedaboutcomputermalfunc:ons.
[2014]
Post-examImpressions
67
Boxplots: responses from typists. Bars represent medians. Means shown for clarity. N = 91
Strongly Disagree Strongly Agree
4.4
4.0
4.2
4.5
4.3
Updatedtoincludes12015results–8cohorts.
Student(typists)impressionsofusingtheexamsystem
Post-examImpressionsDidtypiststhinktheexamsuitedtheuseofcomputers?
68
Boxplots: responses from typists by cohort. Bars represent medians. Means shown for clarity. Overall mean agreement 4.2 Largely that was a ‘yes’. However two factors at play: a) Self-selecting sample.
Typists would be positive. b) Exam was ‘paper
equivalent’ thus not taking advantage of what was possible with IT e.g. multimedia, simulations etc
Strongly Disagree Strongly Agree
Updatedtoincludes12015results–8cohorts.
3.9
3.6
4.4
4.8
4.2
4.6
4.0
4.2
Reac:ontocondi:onsintheexamTypistshadamoreposi:veexperienceoverallbutnosignificantdifferenceswerereportedfor:merunningoutandstresslevels.
69
Mann-WhitneyU 21172.5 23645 23065 24252Z -2.539 -0.647 -1.081 -0.447
Asymp.Sig.(2-tailed) >.05 n/s n/s n/s
Typists = purple, Hand-writers = orange
Strongly Agree
Strongly Disagree
Updatedtoincludes12015results–8cohorts.
4 3.7 2.7 2.6 2.6 2.7 3.5 3.5Means
Reac:ontocondi:onsintheexamOverallexamexperiencebycohort
70
stronglydisagree(notgood)stronglyagree(be]er)
Updatedtoincludes12015results–8cohorts.
Reac:ontocondi:onsintheexamTimeavailabilitybycohort
71
stronglydisagree(notgood)stronglyagree(be]er)
Updatedtoincludes12015results–8cohorts.
Generalkeyboardandwri:ngprowessStudentreportedtypingandwri:ngskills
72
Key: Purple (top) = typists, Orange (bottom) = hand-writers Bars represent medians. Means shown for clarity. Mann-Whitney U test results shown.
Means Sig 3.28, 3.48 n/s 4.46, 4.31 n/s 3.37, 3.61 n/s 4.49, 3.88 sig >.001 4.23, 3.49 sig >.001 4.52, 3.67 sig >.001
Strongly Disagree Strongly Agree Updatedtoincludes12015results–8cohorts.
73
Was the sound of typing distracting? In each boxplot Typers (left) and Hand writers (right)
Means 3.4 2.3Both exhibited significant differences to >.01 on Mann-Whitney U test Likert Scale: 5 = Strongly Agree, 1 = Strongly Disagree Updated to include S1 2015 results.
The cohorts ANIM2014, BIOL2014 and CRIM2015 were removed from the analysis because typists and hand writers sat in different rooms. Those that could hear typing (who selected 5, 4 or 3) were included in the determination of distraction by typing sound. Cohort exams were held in different venues.
2.2 3.3
74
Was the sound of typing distracting (VetSci)? In each boxplot Typers (left) and Hand writers (right)
Means 3.4 2.47
Likert Scale: 5 = Strongly Agree, 1 = Strongly Disagree
VetSci Course: VETS both the internal and external cohorts used same room, but at different times (4 weeks apart). VETS internal: warm day, ceiling fans and construction noise. VETS external: cooler day, no fans, quiet. Environmental conditions and acoustics play a large role in the degree to which ‘typing noise’ becomes a distracting factor. Hand-writers were not all quiet either!
2.85 3.57
FutureInten:onsTypistsweremoreposi:vetowardsfuturee-Exams,asexpected,buthand-writerswerenotnega:veasawhole.
75
Mean N SDTypists 4.2 39 0.8Hand-writers 2.2 167 1.2
Mann-WhitneyU672.5Z-7.961
Sig(2tail)>.001
SD SA
Typists = purple, Hand-writers = orange Updatedtoincludes12015results(4cohorts-new question added in Semester 2, 2014). .
Post-examImpressionsHand-wri:ngintheexam
76
Boxplots: responses from hand-writers. Bars represent medians. Means and counts shown for clarity. Note: 1= Strongly Disagree, 5 = Strongly Agree
m N3.9*162.6 243.2 252.8 482.8 462.3 802.41071.8107
m N3.4 763.8 243.7 253.6 482.9 443.5 803.51093.2107
KruskalWallisTestChi-Square 61.060 19.631
df 7 7Asymp.Sig. 0.000 0.006
Updatedtoincludes12015results–8cohorts.
Are some students over estimating the neatness of their hand writing?!
* Note 20% response rate by VETS for this item. All others near 90%
TrialTechnicalIssuesIssuelog(2014):15ofthe69whotypedreported‘technicalissues’viathepost-examsurvey.1morewasiden:fiedbyobserva:on.Themajoritywereminor.
77
Issue N Notes,AddiVonalObservaVons,SuggestedSoluVonsBoot/startup 2 Inrealitymostpar:cipantsneededassistance/forgotbootkey.
Familiarity:needtoprac*ce!
EnteringID 0 Allgood.(somestudentsentered‘s’ratherthan8digitnumberbutsystemcopesfine).Usingthesopware
1 Somedidnotknowhowto'exit'gracefully(i.e.Filesave,fileexit,shutdown).Needtoprac*ce!Inves*gatean'I’mfinished'script/bu\on.
Ba]ery 0 Mostpluggedin.Powerneedstobeavailable.Savingfiles 0 Allgood.(no:cedonestudentused‘saveas’whensavewas‘greyedout’)nowfixedSopwarecrashed/computerfroze
4 1xOld2009whiteMacbook.Officesuitequittodesktop.3xSystemdriveranoutofspacecausingthesystemtocrash(nowfixed).
Touchpad/mouse
7 Sensi:vityreportedbypar:cipants.Someadjustmentsweremade.
USBwiredmicehighlyrecommended!Inves*gatedrivers.
Scrolling 15 TwofingerscrollingoppositetoOSX,keyboardshortcuts.Smallscrollbars.Sensi:vity.Familiarity:needtoprac*ce.Largerscrollbars.Inves*gateauserselectableop*onfor
touchpad/scrollbehavior(andre-mappingofkeyboardshortcuts).
Furtherdevelopmentisneededtoaddresstheseissues.Warningsremaininreadmefilesavailableonpublicdownloadsites.
78
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
I write more words when
I write faster when
I think more carefully before I start writing when
I pause to think most when
I write in a style that feels more normal when
I try not to make changes unless they are really important when
I change, move or correct words or phrases most when
I think the overall structure/argument of my responses is better when
I make more effective use of the time available when
I go back and read over my responses before submitting most when
I feel more stressed when
I am more likely to run out of time when
Overall I feel I perform better in an exam when
Using my computer Same equally Using a pen and paper
Student consideration of general exam conditions when using computer versus pen [2014]: All six cohorts. Response pairs: those who typed (line 1) & those that hand-wrote* (line 2)
* Note - Many of those that hand-wrote their exam had no prior experience of using a computer for an exam so the results presented here are largely speculative on their part. However, it is reasonable to assume that they drew on their general use of computers. Note! Updated March 2015 edition places ‘same equally’ in the middle rather than on the right.
79
0%# 20%# 40%# 60%# 80%# 100%#
I think carefully before I start writing when using my computer
I think carefully before I start writing when using pen and paper
I take notes in lectures using my computer
I take notes in lectures using pen & paper
I make quick, rough notes before writing essays/reports properly using my computer
I make quick, rough notes before writing essays/reports properly using pen and paper
I make a detailed plan before writing essays/reports properly using my computer
I make a detailed plan before writing essays/reports properly using pen and paper
I just start writing (there is no plan!) when using my computer
I just start writing (there is no plan!) when using pen and paper
I make lots of notes using pen & paper
I tend to go back and re-read and revise my writing quite a lot
I prepare most of my assignments using a computer
Yes# No#
Writing strategies under non-exam conditions – general writing habits [2014]: All six cohorts. Response pairs: Typers (line 1) and Hand writers (line 2)
ZSig.(2-tail)
-4.25 .000***
-1.75 .080*
-1.56 .120
-2.98 .003***
-3.19 .001***
-1.84 .067
-2.96 .003***
-1.20 .230
-0.08 .934
-0.12 .904
-1.40 .161
-0.52 .606
-1.48 .138
Nonparametric U & Z used to compare those who typed in the exam to those that hand wrote. Note! The September 2014 edition of this chart was incorrectly reversed against the stats.
Typingandwri:ngabili:esStudenttypingandwri:ngingeneral
80
4.5 3.8 4.2 3.5 4.5 3.9 3.3 3.5 4.4 4.3 3.3 3.4 Means
Updatedtoincludes12015results–8cohorts.
Likertsc
ale:1=stronglydisa
gree,5=stronglyagree.
Mann-WhitneyU 14703 13079.5 14514 18196.5 18969 19746.5Z -4.708 -5.677 -4.762 -1.694 -1.366 -0.676Sig.(2-tailed) >.001 >.001 >.001 n/s n/s n/s
Typers (left) and Hand writers (right)
Looks like a ‘yes’!
Did the nature of prior experience of e-exams impact on the decision to type this exam?
All participants, 6 cohorts [2014]. OfthosewithPriorexp. AllMann-WhitneyU 502Z -2.734Sig.(2-tailed) >.01
Yes No
81
82Looks like a ‘yes’!
Does the nature of prior experience of e-exams impact future intended use? [2014]
All participants, all cohorts. Hand writers, all cohorts.
All HandwritersMann-WhitneyU 22.5 21Z -3.262 -2.248Sig.(2-tailed) >.01 >.05
Phase2:UQe-ExamTrials2014-2015
83
Datacollectedfromstudents(opt-in)• Viapre-examprojectonlinesurvey(UQwide):
Hillier,M.(2014).TheVeryIdeaofe-Exams:Student(Pre)concep:ons.PresentedattheAustralasianSocietyforComputersinLearninginTer:aryEduca:onconference,Dunedin,NewZealand.Retrievedfromh]p://transformingexams.com/files/hillier_2014_ascilite_full_paper_prepress.pdf
• Viapre-examshortsurvey(8courses–typistsonly).– Conductedatthepre-examprac:cesetupsessions.– Covered:studentpreliminaryimpressions,technicalhardwarecompa:bility.
• Viapost-examextendedsurvey(8courses–allstudents–next)– Conductedattheconclusionoftheexam(intheroom).– Covered:ra:onale,studentexamexperience,reac:ontoexamsession
condi:ons,e-examsystemimpressions,examwri:ngstrategiesandproduc:on,generalnon-examwri:ngstrategies.
– 2014(sixcourses)h]p://transformingexams.com/files/Hillier_2015_ascilite_fp.pdf
• Analysisoftextproduc:on(DENTonly)– Marksvwordcount,typingvhandwri:ng(moretocome;languagedensity…)
AnalysisofExamResponses-DENT
84
DENT Mid Semester Exam S1 2015 – • Six short answer questions • 20% of course • Scripts N = 68 • 19 typed • 49 handwritten
Analysis • Production (word count) • Marks per question and overall
Produc:onDENT
85
DENT Mid Semester Exam S1 2015 – Six short answer questions; 20% of course.
Scripts N = 68 (19 typed, 49 handwritten)
Score Mann-Whitney U 295, Z -2.333, Sig .02
Words Mann-Whitney
U 245.5 Z -3.007 Sig .003
WordCountforeachQues:onDENT
86
Comparing number of words typed and handwritten by question number. Significance per question by mode. Mann-Whitney: Q1 >.05 Q2 >.05 Q3 NS Q4 >.05 Q5 >.05 close Q6 NS
Domorewordsmeanbe]ermarks?
87
DENT 2015 exam. It depends! A higher hand written word count generally led to passing and better marks, but lesser words did not always result in poor marks. Typists did better overall. More typed words only slightly increased marks. Not claiming causation!
Moreinforma:on….Demoset-upGuide,StudentPrac:ceandUserGuideh]p://transformingexams.com
Demovideosstart-up,useandrecoveryexamples.Appleh]p://ta.vu/eexam-demo-a'Wintel'(Dell)h]p://ta.vu/eexam-demo-dContact:m.hillier[at]uq.edu.au
e-ExamProjectResources
88
ReferencesDermo,J.(2009).e-Assessmentandthestudentlearningexperience:Asurveyofstudentpercep:onsofe-assessment.Bri:sh
JournalofEduca:onalTechnology,40(2),203–214.h]p://doi.org/10.1111/j.1467-8535.2008.00915.xFluck,A.(2013).Implementa:onofon-campusdigitalexamina:onprac:ces.PresentedattheTransformingAssessmentina
DigitalEra,Melbourne,Australia.Retrievedfromh]p://www.eexams.org/stories/Documents/TA%20in%20a%20Digital%20Era%20-%20Fluck.pdf
Fluck,A.E.(2011).eExamsforTransforma:onofHigherEduca:on.PresentedattheAustralianAssocia:onforResearchinEduca:onConference,HobartTasmania.Retrievedfromh]p://www.aare.edu.au/data/publica:ons/2011/aarefinal00107.pdf
Fluck,A.,&Hillier,M.(2014).eExamsTransformingCurriculum.InNowIT’sPersonal(pp.151–158).Adelaide,Australia:ACEC.Retrievedfromh]p://acec2014.acce.edu.au/sites/2014/files/a]achments/eExams%20paperd%20-%20REV2b.docx
Fluck,A.,Pullen,D.,&Harper,C.(2009).CaseStudyofaComputerBasedExamina:onSystem.AustralasianJournalofEduca:onalTechnology,25(4),509–523.
Hillier,M(2015)"Totypeorhandwrite:student'sexperienceacrosssixe-Examtrials",ASCILITEConference,Perth,29Nov-2Dech]p://transformingexams.com/files/Hillier_2015_ascilite_fp.pdf
Hillier,M&Fluck,A(2015)"Apedagogicalendgameforexams:alook10yearsintothefutureofhighstakesassessment",ASCILITEConference,Perth,29Nov-2Dec.h]p://transformingexams.com/files/hillier_fluck_2015_exam_futures.pdf
Hillier,M.(2014).TheVeryIdeaofe-Exams:Student(Pre)concep:ons.PresentedattheAustralasianSocietyforComputersinLearninginTer:aryEduca:onconference,Dunedin,NewZealand.Retrievedfromh]p://ascilite.org/conferences/dunedin2014/files/fullpapers/91-Hillier.pdf
Hillier,M.,&Fluck,A.(2013).Arguingagainfore-examsinhighstakesexamina:ons.InH.Carter,M.Gosper,&J.Hedberg(Eds.),ElectricDreams(pp.385–396).MacquarieUniversity.Retrievedfromh]p://www.ascilite.org.au/conferences/sydney13/program/papers/Hillier.pdf
Lim,E.C.H.,Ong,B.K.C.,Wilder-Smith,E.P.V.,&Seet,R.C.S.(2006).Computer-basedversuspen-and-papertes:ng:students’percep:on.AnnalsoftheAcademyofMedicine,Singapore,35(9),599–603.
Mogey,N.,&Hartley,J.(2012).Towriteortotype?Theeffectsofhandwri:ngandword-processingonthewri]enstyleofexamina:onessays.Innova:onsinEduca:onandTeachingInterna:onal,50(1),85–93.h]p://doi.org/10.1080/14703297.2012.748334
Sorensen,E.(2013).Implementa:onandstudentpercep:onsofe-assessmentinaChemicalEngineeringmodule.EuropeanJournalofEngineeringEduca:on,38(2),172–185.h]p://doi.org/10.1080/03043797.2012.760533
220moreat:h]ps://www.zotero.org/groups/e-assessment/items/tag/e-examSeee-ExamProjectpageath]p://transformingexams.com/research.html 89
e-ExamsOnlineConferenceRecordings
Viewath]p://ta.vu/eAS20141. Monday,8September-GMT07:00:BYODon-campuse-examsatUniversityofTasmania(UTAS).AndrewFluck,
UniversityofTasmania,Australia.2. Wednesday,10September-GMT07:00:Bring-your-own-laptope-examforalargeclassatNUS.SeowTeckKeongand
JefferyTay,Na:onalUniversityofSingapore.3. Thursday,11September-GMT07:00:LargescalefullyonlineBYODfinalexams:Notyourparentsmul:plechoice.Rob
Peregoodoff,UniversityofBri:shColumbia,Canada.4. Friday,12September-GMT07:00:Finland'sna:onalmatricula:onexamsgoeselectronic.MatLa]u,Matricula:on
Examina:onBoard,Finland.5. Tuesday,16September-GMT07:00:eOSCE-robustreal:meelectronicmarkingforclinicalexamina:ons.Sebas:an
HunkelerandDrPhilippeZimmermann,Ins:tuteofMedicalEduca:on,UniversityofBerne,Switzerland.6. Wednesday,17September-GMT07:00:Gamifica:onofClickerswithBYOD.PaulLam,ChineseUniversityofHong
Kong.7. Thursday,18September-GMT07:00:SafeExamBrowser:Amodularapproachtosecureandflexibleonline-exams.
DanielR.SchneiderandTobiasHalbherr,SwissfederalIns:tuteofTechnologyZurich.8. Friday,19September-GMT07:00:TenYearsofe-ExamsatFreieUniversitatBerlin:anOverview.AlexanderSchulz&
NicolasApostolopoulos,FreeUniversityBerlin,Germany. 90
End
CitethisresourceHillier,M(2015)“e-Exams:Thestorysofar”.December.