Upcoming Caveon Events

• Caveon Webinar Series: Next session, May 17

“Using Decision Theory to Score Accurate Pass/Fail Decisions” Presented by Jamie Mulkey and Larry Rudner

• Caveon at NCSA – June 19-21, National Harbor, MD

Visit our booth and spend some time talking with us one-on-one!

Latest Publications

• Handbook of Test Security – Now available for purchase! We’ll share a discount code before end of session.

• TILSA Guidebook for State Assessment Directors on Data Forensics – coming soon!

Caveon Online

• Caveon Security Insights Blog– http://www.caveon.com/blog/

• twitter– Follow @Caveon

• LinkedIn– Caveon Company Page

– “Caveon Test Security” Group• Please contribute!

• Facebook– Will you be our “friend?”

– “Like” us!

www.caveon.com

“What You Should Know about High Stakes Cheating in Your Schools”

Mike Stetter EdDSenior Security ConsultantFormer State Assessment Director

May 2, 2013

Caveon Webinar Series:

John Fremer PhDPresidentCaveon Consulting Services@TestSecurityGuy

AGENDA

A. THE TEST SECURITY SITUATION

B. CHALLENGES FOR SCHOOL DISTRICT LEADERS

C. TOP TEN THINGS THAT YOU CAN DO TO

STRENGTHEN YOUR TEST SECURITY

D. GAINING A GREATER SECURITY ADVANTAGE

A. THE TEST SECURITY SITUATION

Who cheats?

How many of them are there?

How often do they cheat?

JOSEPHSON INSTITUTE SURVEY RESEARCH

• 95% admitted to cheating in high school

• 51% believed they must cheat to get ahead

• 90% were pleased with their own morality

40,000 College Students Surveyed in 2010

Under 17 Over 50

Believe cheating is necessary 50% 10%

OK to deceive your boss 31% 8%

OK to keep incorrect change 49% 15%

OK to lie to your spouse 45% 22%

7,000 Survey Respondents

SURVEY OF ADULTS OVER 50 AND YOUTH UNDER 17

29% felt pressure to cheat on standardized tests

34% felt pressure to change students’ scores for the better

21% knew educators that changed scores on students’ tests

8% admitted to changing students’ scores due to pressure

Detroit Free Press, July 26, 2011

MICHIGAN EDUCATOR SURVEY

School District Cheating Scandals

In The News

DC Public Schools

More than 100 schools

displayed suspicious

answer sheet erasures

So far, 2 teachers have been forced to resign

USA TODAY June 22, 2012

Houston Independent

School District

Teachers at 2 elementary

schools caught cheating

Changed student answer sheets on State exams

The Houston Chronicle February 25, 2013

Atlanta Public Schools

179 educators in

58 schools implicated

in dishonest test practices

41 have resigned or retired. 35 indicted by Grand Jury

Former Superintendent indicted on conspiracy

TIMENewsFeedApril 13, 2013

Analyzed test results from

69,000 public schools

Suspicious test scores, similar to those in Atlanta,

were found in 196 school districts

Dishonest test practices on State level assessments

appears to be a national problem

April 13, 2013

August 25, 2011

Duncan: No link between cheating and NCLB

Places the blame on school leaders

Regardless of who the perpetrators are . . .

School District Administrators will feel the heat!

B. CHALLENGES FOR SCHOOL DISTRICT LEADERS

• Proctors’ potential conflict of interest

• Non-Standardized environment

• Examinee familiarity

• Transporting and storing test materials

SCHOOL DISTRICT TEST SECURITY MATRIX

• District Superintendent

• District Assessment Coordinator

• School Test Coordinator

DISTRICT TEST ROLES SECURITY MATRIX

The full Matrix is posted on the Caveon website:

http://bit.ly/testroles

SCHOOL DISTRICT TEST SECURITY MATRIX

Essential Proactive

Steps

During Test Window

Before & After

Test Window

Superintendent

District Assessment

Director

Building Test Coordinator

District Superintendent

• Assign clear responsibility & roles for test security

• Maintain communications with all local stakeholders on assessment activities, results

• Advocate for test program best practices

SCHOOL DISTRICT TEST SECURITY MATRIX

District Assessment Director

• Supervise test training for all proctors

• Protect secure materials and test session integrity through detailed, accountable procedures

• Develop/Update district Test Security Plan

SCHOOL DISTRICT TEST SECURITY MATRIX

Building Test Coordinator

• Confirm every test proctor completes training, and understands Do’s & Don’ts during testing

• Observe test sessions to confirm procedural compliance

• Directly supervise all handling of secure test materials

• Report suspected test irregularities

SCHOOL DISTRICT TEST SECURITY MATRIX

C. TOP 10 THINGS YOU CAN DO

TO STRENGTHEN YOUR TEST SECURITY

#1 – COMMUNICATION

• Raise test security awareness

• Include security references in communications

• Issue periodic security focused communications

#2 – EXPECTATIONS

• Develop a system-wide Student Honor Code

• Emphasize the importance of honest test taking

• Issue clear expectations to test staff for professional behavior in test administration

#3 – PROCEDURES

• Review all procedures for security related content

• Emphasize procedures for handling test materials

• Review / update security procedures annually

#4 – TRAINING

• Review your test security training materials

• Track who completes the training

• Provide brush-up training annually

#5 – OBSERVATIONS

• Perform announced pre-test session site visits

• Perform unannounced test session observations

• Follow-up on all observation outcomes

#6 – FEEDBACK

• Seek feedback from test administrators / proctors

• Review procedures for reporting test irregularities

• Provide an anonymous tip line

#7 – STUDY YOUR TEST DATA CAREFULLY

• Very Unusual Gains

• Other Indices

• Similarity

• Erasures

• Individual Patterns

• Timing (In Computer Testing)

#8 – PREPARING FOR POSSIBLE SECURITY INCIDENTS

• Develop a “Security Incident Response Plan”

• Specify procedures, roles and responsibilities

• List potential outcomes for security infractions

#9 – LEARNING FROM OTHERS

• CCSSO TILSA SCASS “Guidebook on Test Security”

• Wollack & Fremer “Handbook of Test Security”

• Caveon Webinars and Security Insights Newsletter

Learn about developments in prevention and detection

#10 – IF YOU DEVELOP YOUR OWN TESTS

• Reduce exposure rate of test questions

• Randomize question locations

• Rotate answer options

DATA FORENSICS

D. GAINING A GREATER SECURITY ADVANTAGE

Data Forensics is the analysis of student test data

using sophisticated proprietary computer programs

to statistically identify instances likely representing

test fraud, collusion, and other security violations.

DATA FORENSICS

A statistical measure of similarity within test results

or test response patterns that highlight instances

with a very high likelihood of a test security violation.

Similarity Index

DATA FORENSICS

HOW CREDIBLE IS A FLAGGED SECURITY INDEX?

• Chance of being hit by lightning = 1 in 1 Million

• Chance of winning the lottery = 1 in 10 Million

• Chance of DNA false-positive = 1 in 30 Million

• Chance of students flagged for Similarity doing their own work = 1 in 1 Trillion

Caveon’s data forensic diagnostic screening service for

school districts – a cost-effective means to monitor for

• Answer copying

• Test coaching

• Proxy test taking

• Collusion and other security violations

THE CAVEON SECURITY SCREEN

• Custom forensic analyses of student test records

• Data output by subject, by school and by student

• Data interpretation with a Caveon test security specialist

• Advice on follow-up with highlighted security anomalies

• Consultation with test security planning and evaluation

• Security focused analysis of policies and procedures

THE CAVEON SECURITY SCREEN

SAMPLE FORENSICS OUTPUT DATA REPORT

Consultation

• Consultation with a Caveon Test Security Specialist

• Advice on Test Security planning and evaluation

• Interpretation of forensic data output reports

• Guidance on test irregularity follow-up

THE CAVEON SECURITY SCREEN SUMMARY

• Similarity Index analyses

• Perfect Test and Identical Test Analyses

• School level data reported by grade and by subject

• Flagged student output with cluster details for pairs and groups

Data Forensics

Michael W. Stetter, D.Ed.

Senior Security Consultant

Caveon Test Security

(302) 415-0375

mike.stetter@caveon.com

For information on the Caveon Test Security Screen

HANDBOOK OF TEST SECURITY

Editors - James Wollack & John Fremer

Published March 2013

Preventing, Detecting, and Investigating Cheating

Testing in Many Domains

Certification/Licensure

Clinical

Educational

Industrial/Organizational

Don’t forget to order your copy at www.routledge.com

http://bit.ly/HandbookTS (Case Sensitive)

Save 20% - Enter discount code: HYJ82

- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Security”

Mike StetterSenior Security ConsultantCaveon Test Security

John FremerPresidentCaveon Consulting Services@TestSecurityGuy

Thank You!