© 2005 The University of Vermont University of Vermont College of Medicine Online Exams Embrace the...
-
Upload
kerry-hodge -
Category
Documents
-
view
212 -
download
0
Transcript of © 2005 The University of Vermont University of Vermont College of Medicine Online Exams Embrace the...
© 2005 The University of Vermont
University of VermontCollege of Medicine
Online Exams
Embrace the opportunity, avoid the pitfalls
Copyright: The University of Vermont This work is the intellectual property of the University of Vermont. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors
© 2005 The University of Vermont
Testing
© 2005 The University of Vermont
Who we are
University of Vermont College of Medicine• Located in Burlington, Vermont• Founded in 1822• Nation’s seventh oldest medical school• One of 125 medical schools in the US • Enrollment
o 399 medical studentso 93 graduate studentso 12 MD/PhD students
• Facultyo 678 full time facultyo 383 part time facultyo 711 volunteer faculty
• Affiliated with Fletcher Allen Health Care (FAHC) o a level 1 trauma center
• Allied with Maine Medical Center (MMC)
© 2005 The University of Vermont
Why On-Line Exam Delivery
Vermont Integrated Curriculum (VIC)o Transforms medical education
• Learning defined by body system, disease, and skill set• Patient-based learning integrates with basic science
o Merges tradition and innovation• Lectures, labs, and seminars join case-based studies,
collaborative small groups, and virtual learning• The art and science of medicine are fused with an advanced hybrid
online learning system
o Assesses competencies• Questions linked to teaching• Statistical analysis• Competency tracking
o 12 courses over first 18 months, some with as many as 120 faculty; 8 clinicalexperiences in two states
© 2005 The University of Vermont
History
• Courses no longer resided in departments; now overseen centrally
• Needed to assess the efficacy of the new curriculum – was it working?
• Wanted single platform for assessment, and it had to produce detailed statistics for analysis
• National licensing exams were computerized• Wanted to embrace potential of technology
© 2005 The University of Vermont
Requirements
• Size/Volumeo Any one exam, up to 100 students at a time; up to 150 questions.o One large exam delivered simultaneously with other smaller exams
• Need for hybrid functions for pedagogical purposes – especially image-based disciplines
o Embedded interactive multi-media as part of question delivery.
• Valid student assessmento Alignment of exam questions to competencies
• Security >= paper based assessmentso Inability to do anything but take the examo Minimize the ability to look at another's worko Maximize exam/question reusability
• Grading/Statisticso Grading/Statistical analysis that at minimum matched existing bubble
sheet process.
© 2005 The University of Vermont
Technologies
• Blackboard is the core platformo Test Manager/Gradebook
o Force completion, single attempt, randomized, password protected
• Respondus for uploading questions.o Images, videos, sounds uploaded separately into Blackboard
• Baseline laptops o Ensure consistent delivery and behavior.
• Secure Exam browser and privacy screens o Help discourage cheating
• Fault tolerant platform/platform monitoring toolso Maximize availability.
• Application tuning, generous hardware and exam delivery process maximize performance.
© 2005 The University of Vermont
Technology considerations
• Benchmark and simulate exam delivery under load.o Exam size, randomization, number of students,
simultaneous initiation, and simultaneous exams are all key factors.
o Simulation tools valuable in predicting system performance. Microsoft Application Center Testing (ACT)
o Real world simulation validates entire process.
• Network connectivityo Wired vs Wireless
• Baseline clientso Reduce variables and unforeseen problems.o Have a spares/support strategy in placeo Beware of SMS packages/Windows updateso Teardown disruption
© 2005 The University of Vermont
Technology considerations
• Disaster planningo Develop backup strategieso Understand system behavior and recovery strategies
• New versionso Develop a test script, and be methodical evaluating
every releaseo Understand the types of questions and multimedia
being utilized in exams, o Run your load test.
• Future considerationso Bb 7 advertises improved performance. Validation in
process.o Tablet PCs possible for next yearo Testing throughout the country for 4th year courses
© 2005 The University of Vermont
Process – development/delivery
• Exam developmento New questions added via Blackboard or Respondus.o On-Line exam review by faculty members.
• Exam deploymento Exam scheduled for release 15 min prior to scheduled
exam start.o Password protected. Algorithm based on encrypted
exam name plus something we assign.
• Exam deliveryo Students get setup 5 – 15 min prior to exam starto Support team present during exam kickoff.o Launch ~ 16 students at a timeo Loaner laptops/spare parts (Network cables, power
bricks, mice etc) on hando Save often
© 2005 The University of Vermont
Process -- statistics
• Exam performance analysiso Blackboard provides the cumulative statisticso A custom developed tool extracts detailed
exam informationo Excel pivot-table generated reports include:
• Individual student performance
• Answer key
• Enables grade adjustment for questions determined to be statistically invalid
o SAS calculates performance based statistics• Discrimination index
• Difficulty index
• Intra-examination reliability index
© 2005 The University of Vermont
Process – analysis/adjustment
• Exam Results Analysiso Questions flagged if <60% correcto Difficulty and Discrimination indices – good
discriminators kept; poor ones reviewedo Poorly written questions or material not covered =
question droppedo Faculty development opportunity as they review
their question performance
• Exam Review (for students)o After exam adjustedo SQL Reporting Services for individual exam results.o Students see master question number they got
wrong, their answer and correct answero Question text projected on screen
© 2005 The University of Vermont
People
• COMET team o Exam deployment and delivery issues
• COMIS team o laptop help and network connectivity concerns
• Curricular proctors o Timing and accommodations
• Course director o Content oversight and delivery parameters
• Assessment director o Statistical analysis
© 2005 The University of Vermont
Demonstration
© 2005 The University of Vermont
Opportunities
• What do faculty want to achieve with exams?o Ease of grading? o New question types?o Rich media?
• What can faculty achieve with exams?o Incorporating video or soundo New question types
• Find the correct location• Multiple answer• Random from a question pool
o Case based or question by question examso Just in time exams o Testing as content has been taught
© 2005 The University of Vermont
Considerations
• Studentso Access to/familiarity with technology
• Facultyo Do they know what they want?o Will they make the commitment to work with the technology?o Are they prepared to change their testing paradigm?
• Processo What are the student & faculty expectations for exam
review?o Are summary statistics enough?o How will they drop questions?o Off site/Distance learning limitations
© 2005 The University of Vermont
Questions?
© 2005 The University of Vermont
Contact Us
Contact us:
Jill JemisonE-Learning [email protected](802) 656-0076
Jim HeintzSenior Technical Program [email protected](802) 656-9770