10 September 2014, 07:00 GMT

Bring-your-own-laptop e-Exam for a Large Class at NUS Seow Teck Keong and Jeffery Tay, National University of Singapore

Your hosts

Professor Geoff Crisp, Dean Learning and Teaching, RMIT University geoffrey.crisp[@]rmit.edu.au

Dr Mathew Hillier, Institute for Teaching and Learning Innovation, University of Queensland mathew.hillier[@]uq.edu.au

Transforming Assessment is supported by

eAS Online Conference

Just%to%let%you%know:%%By#par'cipa'ng#in#the#session#you#acknowledge#and#agree#that: The#session#may#be#recorded,#including#voice#and#text#chat#communica'ons#(a#recording#indicator#is#shown#inside#the#webinar#room#when#this#is#the#case).#We#may#release#recordings#freely#to#the#public#which#become#part#of#the#public#record.#We#may#use#session#recordings#for#quality#improvement,#or#as#part#of#further#research#and#publica'ons.

BRING-YOUR-OWN-LAPTOP E-EXAM FOR A LARGE CLASS AT

NUS

SEOW#TECK#KEONG#Senior#Lecturer#

Department#Of#Biological#Sciences#

JEFFERY#TAY#Associate#Director#

Centre#For#Instruc'onal#Technology#

Na'onal#University#of#Singapore#

WHY DO WE DO E-EXAMS? •  Authen'city#

•  Showing#high#resolu'on#medical#images#rather#than#photocopied#versions#•  Videos/Audio#clips#to#mimic#what#students#will#encounter#in#realUlife#situa'ons#in#their#

chosen#area#of#study#•  Enable#ques'on#analy'cs#•  Improved#efficiency#and#fairness#in#grading#and#repor'ng#•  Audi'ng#•  Ease#of#storage#(electronic#vs#hard#copy#scripts)#•  Reduce#human#error#

•  Administra've#processes#e.g#handling#of#scripts,#marks#transfer#•  Illegible#student#handwri'ng#

•  BYOD#(2012)#•  Using#the#personal#learning#environment#(aka#laptop)#of#the#student#•  Reduce#students’#examina'on#stress#•  Reduce#system#delay#(transi'on#from#typing#to#pen#and#paper#during#exam)#

Lockdown Software

BYOD / PC Cluster

Inhouse solution PC Cluster

VPN PC Cluster

E-EXAMS SINCE 2006

0#

5#

10#

15#

20#

25#

2006/2007#

2007/2008#

2008/2009#

2009/2010#

2010/2011#

2011/2012#

2012/2013#

2013/2014#

Num

ber%o

f%assessm

ents%

Academic%Year%

CA#

Final#

EUAssessments#in#NUS#•  Since#2006#(8#years)#•  Average#200#students#per#run#•  Both# con'nuous# assessment# as#well#as#high#stakes#

Facul'es#•  Centre#for#English#Language#•  Compu'ng#•  Law#•  Medicine#•  School#of#Public#Policy#•  Science#

Lockdown Software

BYOD / PC Cluster

Inhouse solution PC Cluster

VPN PC Cluster

E-EXAMS SINCE 2006

0#

5#

10#

15#

20#

25#

2006/2007#

2007/2008#

2008/2009#

2009/2010#

2010/2011#

2011/2012#

2012/2013#

2013/2014#

Num

ber%o

f%assessm

ents%

Academic%Year%

CA#

Final#

NUS E-EXAMS @ 2006

VPN#+#PC#Cluster#PC#clusters#•  Laptop#are#costly#and#bulky#•  Easier#to#manage#•  Ownership#of#devices#reduces#risk#of#chea'ng#

VPN#•  Provided#a#guest#environment#to#access#eUexam#resources#

•  Restricted#access#based#on#creden'als#

NUS E-EXAMS @ 2006

Solu'on#•  Limits#network#connec'on#only#to#exam#server#

•  Loads#a#guest#environment#with#only#internet#explorer#available,#all#other#programs#are#hidden#

NUS E-EXAMS @ 2006

Pros#•  Easy#for#student,#just#sit#down#and#take#exam#

•  Easy#for#staff#to#create#ques'ons#•  Auto#mark#solu'on#ensures#zero#errors#

Cons#•  Lengthy#setup#and#teardown#•  User#constantly#experience#system#lagging#

•  Connec'vity#and#stability#issues#•  Noisy#keyboards#caused#distrac'on#to#students#

•  Not#scalable#•  License#costs#•  Does#not#completely#lock#down#the#system#

Lockdown Software

BYOD / PC Cluster

Inhouse solution PC Cluster

VPN PC Cluster

E-EXAMS SINCE 2006

0#

5#

10#

15#

20#

25#

2006/2007#

2007/2008#

2008/2009#

2009/2010#

2010/2011#

2011/2012#

2012/2013#

2013/2014#

Num

ber%o

f%assessm

ents%

Academic%Year%

CA#

Final#

NUS E-EXAMS @ 2009

Problems#•  Prior#system#not#feasible#in#the#long#term#

•  A#lot#of#possible#security#loopholes#make#it#unsuitable#for#high#stakes#exam#

Conduct#feasibility#study#on#commercially#available#solu'ons#•  None#of#the#reviewed#solu'ons#are#able#to#meet#our#requirements#completely#

Conclusion#•  Ahempt#to#do#an#inUhouse#solu'on#

NUS E-EXAMS @ 2009 Inhouse#Solu'on#+#PC#Cluster#PC#clusters#•  Laptop#are#costly#and#bulky#•  Easier#to#manage#•  Ownership#of#devices#reduces#risk#of#chea'ng#

InUhouse#Solu'on#•  Backend#dashboard#to#setup#and#manage#exam#

session#•  Implement#the#following#lockdown#protocols#

•  Limit#accessible#URLs,#Ingress#and#Egress#Firewall#

•  Limit#keystrokes#and#key#combina'ons#•  Restrict#access#to#local#storage#•  Prevent#the#use#of#USB#devices#•  Limited#'me#access#

NUS E-EXAMS @ 2009

Pros#•  Easy#for#student,#just#come#and#take#exam#

•  Easy#for#staff#to#create#ques'ons#•  Auto#mark#solu'on#ensures#zero#errors#

•  Ease#of#system#administra'on#

•  Stricter#security#controls#compared#to#the#previous#implementa'on#

•  Stricter#authen'ca'on#protocols#compared#to#previous#implementa'on#

Cons#•  Lengthy#setup#and#teardown#•  Connec'vity#issues#•  User#constantly#experience#system#slowness#

•  Noisy#keyboards#caused#distrac'on#to#students#

•  Not#scalable#•  Only#able#to#work#in#a#PC#Cluster#•  License#costs#•  Does#not#completely#lock#down#the#system#

Lockdown Software

BYOD / PC Cluster

Inhouse solution PC Cluster

VPN PC Cluster

E-EXAMS SINCE 2006

0#

5#

10#

15#

20#

25#

2006/2007#

2007/2008#

2008/2009#

2009/2010#

2010/2011#

2011/2012#

2012/2013#

2013/2014#

Num

ber%o

f%assessm

ents%

Academic%Year%

CA#

Final#

NUS E-EXAMS @ 2012 Problems#•  Macbooks#and#BYOD#becoming#popular,#but#current#

system#is#incompa'ble#with#both#trends#

•  Constant#upgrades#required#to#fix#new#security#vulnerabili'es#and#opera'ng#system#taking#a#toll#on#limited#resources#

•  New#requirements#

•  open#book#

•  Increased#class#size#(>#200)#

Conduct#feasibility#study#on#available#solu'ons#•  Respondus#Lockdown#Browser#found#to#be#suitable#

candidate#with#caveats#(LDB)#

NUS E-EXAMS @ 2012

LDB#+#PC#Cluster#or#BYOD#•  LDB#supports#lockdown#of#both#Windows#and#Mac#plakorms#

•  Implement#the#following#lockdown#protocols#

•  Limit#accessible#URLs#

•  Limit#keystrokes#and#key#combina'ons#

•  Restrict#access#to#local#storage#

NUS E-EXAMS @ 2012

Pros#•  Easy#for#staff#to#create#ques'ons#•  Auto#mark#solu'on#ensures#zero#errors#

•  Support#BYOD,#Windows#and#Mac#plakorms#

•  Support#viewing#of#PDF#documents#

Cons#•  Students#need#to#install#and#verify#that#LDB#runs#on#their#machine#before#the#exam#

•  Limited#lockdown#capabili'es#compared#to#previous#implementa'on#

•  BYOD#introduces#new#challenges#compared#to#using#a#PC#Cluster#e.g#dedicated#electrical#socket,#stable#network#connec'vity#

•  S'll#no#support#for#Math#and#freehand#inputs#

•  Possible#increase#in#manpower#requirements#due#to#BYOD#

CASE STUDY – LSM1301 E-EXAM @ NOV 2013

Background#•  LSM#1301#–#General#Biology#•  Enrollment#–#672#students#•  Final#Examina'on#–#Open#book,#mul'ple#choice,#BYOD#

#First#LARGE#scale,#high#stakes#eUexam#in#NUS#

CASE STUDY – LSM1301 E-EXAM @ NOV 2013

Various#stakeholders#involved#to#ensure#objec'ves#are#met#•  Computer#Centre:#Core#Infrastructure#

•  Centre#for#Instruc'onal#Technology:#BYOD#eUexam,#technical#consultant,#single#point#of#contact,#training#

•  Department,#faculty,#Registrar’s#office:#Logis'cal#support#

Objec've#•  Ensure#students#are#acceptable#and#prepared#with#the#change#in#exam#format#

•  Ensure#actual#eUexam#runs#as#smoothly#as#possible#for#the#672#students#

CASE STUDY – LSM1301 E-EXAM @ NOV 2013 Prepara'on#•  Trial#run/mock#exam#

•  Prepare#the#students#for#change#•  First#point#of#contact#with#actual#devices#to#

be#used#in#exam#•  Prior#iden'fica'on#of#poten'al#problem#

areas#to#prevent#these#to#occur#during#the#actual#exam#

•  Standby#devices:#10%#of#cohort#•  Equipment#test#at#actual#exam#loca'ons#•  Standby#equipment#e.g#wireless#AP,#switches#

•  Backup#exam#loca'on#

CASE STUDY – LSM1301 E-EXAM @ NOV 2013

Technical#issues#that#occurred#Mock#Exam#•  Sonware#incompa'bility:#CIT#ahempted#to#fix#

the#incompa'bility,#most#of#them#were#caused#by#the#students#themselves#rather#than#the#sonware#

•  Lockdown#circumven'on:#Under#certain#condi'ons#the#lock#down#is#circumvented,#these#are#then#reported#back#to#vendor#for#them#to#fix#

##

#

#Actual#Exam#•  5%#U#Power#surge:#Students#might#be#using#an#

incorrect#power#adapter,#causing#a#power#surge#and#blanking#out#power#to#other#students#

•  10%#U#Laptop#crashing/hanging/unable#to#connect#to#WIFI#

•  2%#U#Inconsistency/Limita'on#of#lockdown:#Due#to#the#different#opera'ng#systems#and#devices,#some#of#them#have#a#different#level#of#lockdown#compared#to#the#rest#(e.g#the#ability#to#use#the#built#in#dic'onary)#

STUDENTS’ PERCEPTION STUDY

Exam#–#22#Nov#2013#672#students#

PreUexam#survey#–#midUOct#2013#658#students#(98%#response)#

PostUexam#survey#–#midUDec#2013#399#students#(59%#response)#

Focus#group#discussions#–##mid#to#end#Mar#2014#15#students#(2%#response)#

PRE-EXAM AND POST-EXAM SURVEYS

0%#

10%#

20%#

30%#

40%#

50%#

60%#

70%#

80%#

PreUExam# PostUExam#Nega've# Posi've#

QUALITATIVE FEEDBACK – PRELIMINARY FINDINGS

Posi've#•  More#efficient#

•  Clicking#vs#shading#•  Referencing#of#notes##(‘ControlUF’#or#‘CommandUF’#keys)#

•  Personal#learning#space#•  Familiarity#with#own#laptop#

•  Well#organised#•  Due#to#useful#trial#run#and#mock#exam#

Nega've#•  Technical#issues#

•  Dropped#connec'ons,#slow#downloading#of#notes#

•  Unable#to#review#webcast#recordings#of#lectures#

•  Limita'ons#of#lockdown#browser#

•  Exam#taking#techniques#•  Markings,#notes#on#ques'on#paper#

•  Quaran'ne#

CLOSING NOTES

Conclusions#•  Pedagogy#influences#examina'on#modes#which#in#

turn#affects#the#implementa'on#

•  Closed#book#=#Full#or#limited#lockdown#

•  Open#book#=#No#or#limited#lockdown#

•  Closed#book/full#lockdown#is#close#to#impossible#with#a#BYOD#implementa'on#even#with#commercial#solu'ons#

•  Open#book#with#no#or#limited#lockdown#is#possible#with#certain#commercial#solu'ons#

•  Constant#maintenance#and#upgrade#required#to#ensure#solu'on#is#ahead#of#hackers#and#students#trying#to#beat#the#system.#

•  Anything#short#of#no#lockdown#is#a#temp'ng#treat#for#Compu'ng#students,#showing#success#in#hacking#the#system#is#beher#than#scoring#100%#for#the#exam#

•  High#stakes#nature#introduces#addi'onal#complexity#

•  How#long#do#we#need#to#keep#the#data#and#ensure#backward#compa'bility?#

•  Do#we#need#to#consider#a#more#stringent#electronic#supervision#to#ensure#security#and#fairness?#

•  How#far#do#we#go#to#audit#student’s#ac'ons?#

•  Is#there#a#need#to#cater#for#N+2#failure,#to#ensure#that#even#if#everything#fails,#there#is#s'll#a#backup?#

WHAT’S NEXT?

•  Looking#for#an#easy#to#deploy#solu'on#that#is#as#effortless#as#the#flick#of#a#switch#•  Currently#no#holy#grail#solu'on#that#is#able#to#address#all#modes#of#exams#

•  Adapta'on#of#new#technologies#in#eLearning#is#picking#up#pace,#thus#a#comprehensive#solu'on#maybe#forthcoming.##

What#do#we#want?#

U  A#secure#and#usable#solu'on#that#is#expensive##

U  A#usable#and#cheap#solu'on#that#is#insecure,#or#

U  A#secure#and#cheap#solu'on#that#is#not#usable?#

e-Assessment Scotland 2014 Online Presenter contact:

teckeong@nus.edu.sg t@nus.edu.sg

With thanks from your hosts Professor Geoff Crisp, Dean Learning and Teaching, RMIT University geoffrey.crisp[@]rmit.edu.au Dr Mathew Hillier, Institute for Teaching and Learning Innovation, University of Queensland mathew.hillier[@]uq.edu.au

eAS Online Conference