Caveon Webinar Series: The Good and Bad of Online Proctoring
-
Upload
caveon-test-security -
Category
Education
-
view
798 -
download
1
description
Transcript of Caveon Webinar Series: The Good and Bad of Online Proctoring
Upcoming Caveon Events
• NCME Conference, April 26 – 30th, San Francisco Test Security I: Policy Issues: Technical/Statistical/Methodological Issues
– Presenters include: Dr. John Fremer, Caveon
• USDLA Conference, April 28- May 1, St. Louis
Will the real learner please stand up? – Presenters include Caveon’s Jamie Mulkey, Ed.D. and Patrick Martin
• Caveon Webinar Series: Next session, May 2
What You Need to Know about High Stakes Cheating in Your Schools– Presenters include former State Assessment Director, Dr. Mike Stetter,
and Dr. John Fremer, President, Caveon Consulting Services
Latest Publications
• Handbook of Test Security – Now available for purchase! We’ll share a discount code before end of session.
• TILSA Guidebook for State Assessment Directors on Data Forensics – soon to be released
Caveon Online
• Caveon Security Insights Blog– http://www.caveon.com/blog/
• twitter– Follow @Caveon
• LinkedIn– Caveon Company Page– “Caveon Test Security” Group
• Please contribute!
• Facebook– Will you be our “friend?”– “Like” us!
www.caveon.com
“The Good and Bad of Online Proctoring”
Harry LaymanExecutive DirectorDigital Assessment PlanningThe College Board
April 17, 2013
Caveon Webinar Series:
Dr. David FosterCEOCaveon Test Security
Agenda for today
• Recap of Structure and Content of recent paper “Online Proctoring Systems Compared”
• Introduction and Definitions for Online Proctoring Discussion• Test Security Threats Addressed by Online Proctoring• Online Proctoring Features and Threat Impacts• Seven Critical Online Proctoring Best Practices• Odds and Ends
– Research Support– Future Features– Trends
• Lessons for Test Program Users
Online Proctoring Systems Compared:Purpose of Paper
• Published by Dave Foster and Harry Layman– http://bit.ly/proctoring
• Background of paper• Recent paper provides a framework for describing different
offerings and several tables of feature comparisons• Lots of detail, not entirely complete – will remain a work in
progress• A good starting point as you begin to think about features /
benefits and your specific risk profile and threats of concern
Online Proctoring Systems Compared: Structure and Content
• Overview of Online Proctoring – what it is, what it is not, and primary value proposition
• An enumeration of 8 vendor products / services included in the scope of the review– Mix of capabilities and approaches; offerings not homogenous/equivalent and so
comparisons need to be more holistic– We have not yet investigated all options equally, so the “comparisons” are
informative but not complete; look for updates over time
• Seven groups of features or characteristics of Online Proctoring offerings organized into a series of comparison tables
• This presentation will “hit the highlights,” setting online proctoring in the general context of test security threats, and provide a high-level discussion of the attributes and features of Online Proctoring systems that can mitigate those threats.
Online Proctoring Is / Is not:
• Online Proctoring refers specifically to using an internet-based approach to remotely monitor individual test administrations, replacing traditional approaches that have relied on “eyes in the room”
– Not CCTV, roving personnel, or using volunteers in user-selected locations (libraries, etc.)
• Online Proctoring is NOT a “find-your-own-proctor” model as used for decades in distance education exams
Online Proctoring Provides:
• Key value proposition in a nutshell:
– Independent, bias-free proctors
– Professional proctors (training, supervision, logging, accountability, etc)
– Technology-assisted audio, visual and activity monitoring support (e.g., log files, video, lock-down, keystroke monitoring, test session control, real-time intervention, etc.)
General Proctoring Requirements
• Online Proctor During Entire Exam • Continuous Internet • Encryption for Data Transfer • Schedule Availability • Proctor Management
– Supervised – Training – Career Path – Certification
• Program Customization
Security Threats / Scope
• Online Proctoring is primarily concerned with
– Threats during test administration; and hence
– Attempts to either capture test content, or
– Get inappropriate assistance in any way.
Security Threats / Scope
• Online Proctoring is only concerned with a subset of the overall set of program vulnerabilities, threats and risks. What are they?
• The subset are those threats that are detectable by the online proctor (or onsite proctor) and over which he or she has some control
Test Security Threats: A Finite Universe
• Only Six Categories of Cheating Threats– Using pre-knowledge of test content– Colluding with others– Using cheating aids– Using a proxy test taker– Hacking into scoring system– Copying answers from other test takers
• Only Six Categories of Stealing Threats– Hacking into a system and stealing test files– Capturing content by digital photography devices– Capturing content by electronic recording of the screen– Memorizing content– Transcribing content verbally (on paper or recording device)– Getting content from test program insider
Test Security Threats: A Finite Universe
• Only Six Categories of Cheating Threats– Using pre-knowledge of test content– Colluding with others– Using cheating aids– Using a proxy test taker– Hacking into scoring system– Copying answers from other test takers
• Only Six Categories of Stealing Threats– Hacking into a system and stealing test files– Capturing content by digital photography devices– Capturing content by electronic recording of the screen– Memorizing content– Transcribing content verbally (on paper or recording device)– Getting content from test program insider
Down to Six! Test Security ThreatsRelevant for Online Proctors
• Cheating Threats– C1. Colluding with others– C2. Using cheating aids– C3. Using a proxy test taker– C4. Copying answers from other test takers
• Stealing Threats– S1. Capturing content by digital photography devices– S2. Transcribing questions verbally (on paper or
recording device)
C1. Colluding with Others
• How can a proctor effectively detect and deal with collusion?
– Camera that can see a large portion of the room– Camera that can pick up sounds and talking– Ability to pause test session, and perhaps suspend it– Ability to contact test taker, record/store
session and time-stamp incident– Proctors should be prohibited
from viewing test taker screen
C2. Using Cheating Aids
• How can a proctor effectively detect and deal with a test taker using a cheating aid?
– Camera that can see a large portion of the room, particularly desk, computer, keyboard, test taker head/arms/hands
– Strong lockdown program to prevent access to online aids– Ability to contact test taker, pause
test session, and perhaps suspend it– Ability to record/store session and
time-stamp incident
C3. Using a Proxy Test Taker
• How can a proctor effectively detect and deal with a proxy test taker?
– Strong initial authentication– Continuous authentication methods– Ability to contact test taker, pause test session, and
perhaps suspend it– Ability to record/store session and time-stamp incident
C4. Copying Answers from Another Test Taker
• How can a proctor effectively detect and deal with a test taker copying from another?
– Camera that can see a large portion of the room, including other test takers allowed in the room
– Ability to contact test taker, pause test session, and perhaps suspend it
– Ability to record/store session and time-stamp incident
S1. Capture Test Content with Digital Photography
• How can a proctor effectively detect and deal with the digital photography of the test questions?
– Camera that can see a large portion of the room, along with head/hands/torso of test taker
– Prevent proctor from viewing test screens– Ability to contact test taker, pause test session, and
perhaps suspend it– Ability to record/store session and time-stamp incident
S2. Transcribing Questions Verbally
• How can a proctor effectively detect and deal with the transcription of test questions?– Camera that can see a large portion of the room, along
with head/hands/torso of test taker– Prevent proctors from viewing the test taker screen– Ability to contact test taker, pause test session, and
perhaps suspend it– Ability to record/store session and
time-stamp incident
Seven Critical Online Proctoring Best Practices
1. Use wide-view camera with microphone
2. Require lockdown program
3. Prevent proctor view of test screens
4. Allow and require proctor to communicate with test taker
5. Allow proctor to control the test session
6. Use strong authentication methods
7. Record, store and time-stamp test session, along with proctor logs
First: Use a Capable Webcam
• Is external to the computer; connected by 2-4 feet of USB cable and set back
• Has large field of view (approx. 80° up to 360°)• Allows proctor to see testing workstation and head,
torso, arms, hands of test taker• Camera angle and placement
does not allow viewing the
workstation screen• Has integrated microphone
Second: Use a Capable Lockdown Program
Operating System/Computer/Browser• Prevent right-click • Prevent min/max windows
• Prevent printing • Prevent Copy/Paste
• Prevent function keys • Prevent running of applications
• Prevent important key combos • Prevent launch of applications
• Hide Taskbar and Desktop • Prevent communication tools
• Hide menus and icons • Prevent browser control/navigation
Third: Prevent Proctor View of Test Workstation Screen
• What are the reasons a proctor should not view the content of a test taker’s workstation screen?
– Security Reason #1: Viewing the screen may encourage collusion between proctor and test taker
– Security Reason #2: Seeing the items of a high-stakes exam on a proctor’s screen may encourage the theft of the test content
– Privacy Reason: It’s not appropriate for a proctor to view how a test taker answers test questions
Fourth: Allow Proctor to Communicate with Test Taker
• Chat• Canned Messages• Technical Support call if necessary
Fifth: Allow Proctor to Control Test Session
• Control allows proctor to deal effectively with detected threats– Pause the test
• cover or remove the current question• require acknowledgement and compliance from the test
taker• provide a warning
– Suspend the test (can restart at later time)– Cancel the test
• Proctor rules for these actions are subject to customization by the testing program
Sixth: Use Strong Authentication Methods
• Strong– Fingerprint readers– Facial recognition– Keystroke pattern recognition– Voice recognition
• Avoid weak authentication methods– Viewing of government-issued IDs through webcam
Seventh: Record and Store Sessions
• Record entire test session and proctor logs/communications
• Time stamp incidents to make them easy to locate
• Provide easy-to-use review application for both video of test session and proctor logs
• Store long-term or until no longer needed
Best Practices Effects on Security Incidents
• If these best practices are followed what benefits can be expected?
– Better detection of threats or attempted breaches– Professional, un-biased, and competent response by
proctors– Deterrence: Fewer incidents going forward– Stronger evidence basis for decision making and taking any
action, including legal action
Research Support
• Reliable data – on site or online – is hard to come by
• Limited data available for online proctoring; what is there is encouraging
• More reporting by participating organizations will increase the data pool
• Need studies also from academic researchers
It’s Only Going to Get Better: Future Proctoring Features
• Improvements (e.g., better cameras; better proctor training, etc.)
• New features: Automated proctoring technologies– Inappropriate keystroke detection – Audio level detection and display – Detection of suspicious response patterns with Real-Time
Data Forensics– Better facial, voice recognition software on the horizon
More Trends
• More providers, may see increasing stratification between cost and quality / professionalism (as with other security services)
• New technologies may allow better detection, or lower costs – or both– Augmenting human monitoring of online video with real-
time “artificial intelligence” could allow for greater proctor-to-examinee ratios with even better security
Lessons for Program Users
• Do it right or be an easy target• Strong security is always needed for tests where
cheating and theft occur, but not everyone has the same set threats or concerns
• Different audiences may have different levels of technical sophistication, risk, time sensitivity, etc.
• Online proctoring when done properly can be as capable as any other well-run proctoring method at reducing test security risks
HANDBOOK OF TEST SECURITY
• Editors - James Wollack & John Fremer• Published March 2013• Preventing, Detecting, and Investigating Cheating• Testing in Many Domains
– Certification/Licensure– Clinical– Educational– Industrial/Organizational
• Don’t forget to order your copy at www.routledge.com– http://bit.ly/HandbookTS (Case Sensitive)– Save 20% - Enter discount code: HYJ82
Questions?
Please join our “Caveon Test Security” LinkedIn group and submit questions into the discussion box or contact [email protected]
THANK YOU!
- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Security”
Harry LaymanExecutive DirectorDigital Assessment PlanningThe College Board
Dr. David FosterCEOCaveon Test Security
Again, find our Online Proctoring Systems Compared paper at http://bit.ly/proctoring