Caveon Webinar Series: Weathering the Perfect Test Security Storm - February 2015
Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014
-
Upload
caveon-test-security -
Category
Education
-
view
179 -
download
5
description
Transcript of Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014
![Page 1: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/1.jpg)
Steve Addicott, Vice PresidentDennis Maynes, Chief Scientist
Caveon Test Security
Caveon Webinar Series:
Six Security Challenges to Your High Stakes Test Programand How Data Forensics May Help Thwart Them
January 22, 2014
![Page 2: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/2.jpg)
Upcoming Caveon Events
www.caveon.com
Caveon Webinar Series: Next session, February 19 Protecting Your Tests Using U.S. Copyright Law
ATP Innovations In Testing Annual Conference • March 2-5 in Scottsdale, AZ• Check out our sessions here: • http://www.caveon.com/atp-2014-innovations-in-testing-caveon-s
essions/• Visit us in Booth 33 or make an appointment to talk to us about
your specific test security or test development concerns.
![Page 3: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/3.jpg)
Agenda for Today
• Magnitude of the Challenges
• Six Challenges
• Potential Solutions/Approaches
• Role of Data Forensics
• Summary
![Page 4: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/4.jpg)
Magnitude of the Problem
2012 ATP Security Committee Survey Results
• Exact matches of exams on the internet? • 41% of test sponsors (who completed the survey)
• $88,000,000 - $223,000,000!!!!• Overall cost estimate for replacing compromised
exams
• Intangible Losses• Validity of certificates• Credibility of program• Confidence in certificate holders
![Page 5: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/5.jpg)
Six Challenges
1. Proxy test taking 2. Braindump usage 3. Test theft 4. Technology 5. Stakeholder support 6. Test administration models
“Caveon Speaks Out on IT Exam Security” http://www.caveon.com/articles/it_exam_security.htm
![Page 6: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/6.jpg)
Proxy Test Taking
•2007: Contracted with a proxy test taker for $1,000• In a few weeks, the certificate was “awarded.”• Data analysis discovered
• The test site:• registered with a false mailing address• affiliated with a mobile site• operated by the proxy test taking organization
• Tests at five more test sites were “very similar” / “in collusion”• Estimated number of proxy-taken exams was 500 in 6 months
• We infer that:• This organization was paid $1 million for
proxy test taking services for a single exam title in one year.
![Page 8: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/8.jpg)
We Believe
• Proxy test takers• Legitimate test sites, but…
• Front room and back room• Operate multi-nationally• Super-human performance• Branching out to other certifications • Sophisticated
• “Whack-a-mole” – they move on
![Page 9: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/9.jpg)
Braindump Usage
![Page 10: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/10.jpg)
Braindump/Theft Usage Case 2012
• Test taker 313 took the exam on 1/25 at 10 am• 97% of the live items were disclosed on 1/25 at 4 pm.• The items were “near-exact” (recorded and transcribed)
• Four test takers from the same company (296, 297, 310, and 311) took the exam on 1/23 and 1/24. • Theft probably occurred on 1/23.
• Eleven more took the exam between 1/25 and 2/29.
• Assuming independence, the similarity had a vanishingly small probability (<10-38).• The imputed answer key had 10 wrong answers for 60
questions.• It’s more likely for the Powerball winner to win the next 4
jackpots!
![Page 11: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/11.jpg)
We Believe
• Braindump usage is rampant (may exceed 1 in 6 test takers)
• Not just for “profiteers” anymore—small groups
• Some braindumpers have gotten smarter.• Are reacting to new test design tactics
• Some braindumpers are naïve.• Education is key.• Invalidating scores will deter braindump usage.
![Page 12: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/12.jpg)
Test Theft
• Testking.com and pass4sure.com • dominant web-based providers of stolen
content.
• More popular on Google than the word, “braindump” – Google Trends 1/2014.
![Page 13: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/13.jpg)
A Real-Life Example
• Medical certification program
• Administration to 3,500 candidates on Saturday
• Anonymous email on Wednesday– “I thought you should know…”– ENTIRE ITEM BANK ATTACHED!!
![Page 14: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/14.jpg)
![Page 15: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/15.jpg)
About Stolen Tests
• Exact copies with answers• Copies of digital files (hacking)?
• Near-exact copies without answers• Digital recording with answer key imputation?
• Reconstructed copies• Recalled or memorized questions?
• Theft triggers• Announced exam republications• When pass rates drop
• Publication of stolen content appears to take about two weeks
![Page 16: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/16.jpg)
Technology
• Bluetooth-enabled ear pieces
• Spy cameras
• Other communication tools
![Page 17: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/17.jpg)
Technology
![Page 18: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/18.jpg)
Technology
![Page 19: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/19.jpg)
Technology
![Page 20: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/20.jpg)
Technology
![Page 21: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/21.jpg)
Technology
![Page 22: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/22.jpg)
Stakeholder Support
![Page 23: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/23.jpg)
In Our Experience
• Legal departments are reluctant to invalidate scores and to revoke certifications
• Many partnering organizations are opposed to sanctions
• Executive “buy in”-- Leadership may not understand the extent of fiscal and ancillary losses
• Poor communication plans – Internal & externalEnsuring that tests measure what they are intended to measure will yield positive effects for the candidates and the sponsoring organizations.
![Page 24: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/24.jpg)
Stakeholder Support Can Be Won
Although the number of individuals who pass their exams as a result of fraudulent exam prep or test taking behavior is very small, it can have a big impact on the value of your certification. EMC is committed to providing the highest level of exam security and does take action when fraudulent exam practices are uncovered. Every month we perform a statistical analysis of all exam result(s). Any exam results found to be questionable - with a high probability of being the result of exam fraud - we revoke. We have been doing this for over two years with great success.
-Liz Burns, EMC Proven Professional Program Manager, posted on the EMC Community Network, August 27, 2009
![Page 25: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/25.jpg)
Test Administration Models
• Security breaches are more likely when…• Tests are administered 24/7
• CBT vs. Paper/Pencil doesn’t matter• Franchised test sites are used• Test prep schools run test sites• Rules are suspended at conferences
• Item compromise is more likely to occur by theft than exposure
![Page 26: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/26.jpg)
We Believe
• The publish-and-forget approach is inherently insecure when tests are administered 24/7.
• Different test administration models may require different security measures and approaches than those taught in schools or used by traditional scheduled testing administrations.
• Test security costs vary with different test administration models.
![Page 27: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/27.jpg)
Test Security is a Process, Not a State
Protect
Detect
Respond
Improve
Measure and Manage
![Page 28: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/28.jpg)
Protect Against Security Breaches
• Test taker and test developer agreements• Education for test takers• Require participation in security
investigations• Messaging
• Cisco Exam Compliance Video Tutorials• https://learningnetwork.cisco.com/community/certifications/
policies_reference_tools/earned-it-videos
• Security Audits of Policies and Procedures• Background checks of test site personnel• Security training of test site personnel• Registered copyrights• Deter through enforcement actions
![Page 29: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/29.jpg)
Detect and Respond
• Detect using data forensics• Similarity to detect sites operated by
proxies, braindump users, and coaching schools
• Latency to detect proxies and braindump users
• EVT™ items to detect braindump users • Respond to potential breaches when
detected• Policies need to clearly support using
statistics• Just-in-time analysis or delayed scores
remove messiness of score invalidations
![Page 30: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/30.jpg)
Exam Inoculation
• Active area of research• “Inoculate the exam” against test
fraud• Does not require score invalidation
or test site shutdowns• Requires frequent republication of tests
• Use innovative measurement techniques (EVT) to detect when to republish
• Use continuous test development model so that new items are always available when the exam must be republished
• Will require adjustments to processes used by the psychometric and test development staff
![Page 31: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/31.jpg)
Data Forensics Detection
Statistical Anomalies
Testing Irregularities
Security Violations
Security Breaches
Test Fraud
![Page 32: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/32.jpg)
Type I Versus Type II Errors
• Focus on test score validity, not candidate behavior.
• Type I error: Improperly deciding a the test score is invalid.
• Type II error: Failing to detect when the test score is invalid.
• Using low probabilities decreases Type I errors and increases Type II errors.• This is a conservative approach.• Errors of allowing invalid scores to stand
are preferred over invalidating valid scores.
![Page 33: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/33.jpg)
Communicating with Stakeholders
• Set appropriate expectations• Clearly convey what data forensics can
and cannot do• Policies of “zero tolerance” and “see no
evil” are not reasonable.
• Present and report key metrics• Number of invalid tests which were
detected• Number of test sites which appear to be
errant• Number of test questions which needed
to be replaced
![Page 34: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/34.jpg)
Questions?
![Page 35: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/35.jpg)
Caveon Online
• Caveon Security Insights Blog• http://www.caveon.com/blog/
• Twitter - Follow @Caveon• LinkedIn
• Caveon Company Page/Caveon Test Security Group/Caveon Security Minute Group
• Facebook• “Like” us!
www.caveon.com
![Page 36: Caveon Webinar Series: Six Security Challenges to Your High Stakes Test Program Jan 2014](https://reader037.fdocuments.in/reader037/viewer/2022102901/55619e40d8b42ad9538b4837/html5/thumbnails/36.jpg)
Thank you!
Steve AddicottVice [email protected]@SdAddicott
Dennis MaynesChief [email protected]@DennisMaynes
- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Security”