Caveon Webinar Series: Online Proctoring Best Practices Oct 2013
Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016
-
Upload
caveon-test-security -
Category
Education
-
view
275 -
download
2
Transcript of Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016
![Page 1: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/1.jpg)
www.caveon.com 1
Caveon Webinar Series
Mastering the U.S. DOE Peer Review Requirements on Test Security
January 27, 2016
![Page 2: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/2.jpg)
www.caveon.com 2
Objective
Today, we will overview the Test Security requirements of Peer Review, and share some readily available resources to help you meet them.
![Page 3: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/3.jpg)
www.caveon.com 3
Today’s Presenters
Dr. John Fremer Dr. John Olson
Marc J. Weinstein, J.D.
![Page 4: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/4.jpg)
www.caveon.com 4
Agenda for Today• Purpose of Peer Review• Background of ESSA• Our Background• The Directive• Critical Element 2.5• Four Requirements• Example Evidence
• Resources Available• Summary• Q&A
![Page 5: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/5.jpg)
www.caveon.com 5
Purpose of the Peer Review Process• To support States by identifying expectations
that they can use as they develop, administer, and improve their assessment systems
• To help States prepare for the peer review of their assessment systems
• To guide the peer reviewers in the review of the State assessment systems
![Page 6: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/6.jpg)
www.caveon.com 6
Key Dates for Peer Review
Due to ESSA being signed into law:• January date cancelled• Revised review dates • April 2016 • June 2016
![Page 7: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/7.jpg)
www.caveon.com 7
From the U.S. DOE
“The field has learned a lot about the importance of having strong test security measures, before, during, and after the assessments .”
Ann Whalen, Acting Assistant Secretary for Elementary and
Secondary Education, letter to CCSSO, September 25, 2015
![Page 8: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/8.jpg)
www.caveon.com 8
ESSA Act – December 2015Some background – from NCLB to ESSA.
![Page 9: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/9.jpg)
www.caveon.com 9
From the U.S. Senate on ESSA
Senator HATCH. [regarding ESSA] I ask the chairman, is it your understanding that the references in section 1201 to activities and services that ensure and improve the ‘‘validity and reliability of state assessments’’ were intended to allow funds to be used for test security activities and services designed and utilized to prevent, detect, and respond to testing irregularities and incidents that threaten the validity of assessment results?
See, Congressional Record at S8469 (December 8, 2015) .
![Page 10: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/10.jpg)
www.caveon.com 10
From the U.S. Senate on ESSASenator ALEXANDER. Mr. President, the Senator is correct. Student assessments must be designed and administered with a high degree of quality assurance. State assessment results can be used as the basis for critical decisions affecting the lives of students and the funding and operation of schools, and given the significant taxpayer investment for statewide assessments, we must provide States with the flexibility to use funds to preserve and maintain the integrity and validity of these important assessments.
See, Congressional Record at S8469 (December 8, 2015).
![Page 11: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/11.jpg)
www.caveon.com 11
ESSA – December 2015
• The U.S. Department of Education will seek public comments on ESSA before issuing implementation regulations later in 2016.
• Accountability systems under the new law go into effect for school year 2017-18.
• Test security continues to be a U.S. DOE priority under the new law.
![Page 12: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/12.jpg)
www.caveon.com 12
• Co-authors of CCSSO PublicationsCo-Editor of HoTS
Our Background
Olson & Fremer , 2013 Fremer & Olson, 2015 Wollack & Fremer, 2013
![Page 13: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/13.jpg)
www.caveon.com 13
The Directive, DefinitionsCritical Element• Statement of a relative requirement• State must submit evidence meeting the
requirementExamples of Evidence• Generally illustrative• Variety of evidence
acceptable• Intended to facilitate
preparation of peer review submission
• Must address the critical element in whole or in part
![Page 14: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/14.jpg)
www.caveon.com 14
Critical Element 2.5
Prevent test security irregularities and ensure the integrity of test results through:
• Prevention• Detection• Remediation• Investigation
![Page 15: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/15.jpg)
www.caveon.com 15
Example Evidence, 2.5Per the U.S DOE:
• State Test Security Handbook
• Summary results or reports of internal or independent monitoring, audit, or evaluation of the State’s test security policies, procedures and practices
![Page 16: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/16.jpg)
www.caveon.com 16
Resources to Meet Critical Element 2.5State Test Security Handbook
![Page 17: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/17.jpg)
www.caveon.com 17
Part 1 - PreventionPrevention of any assessment irregularities including : • Implementing proper test preparation
guidelines and administration procedures
• Maintaining the security of test materials
• Providing annual training at the district and school levels
![Page 18: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/18.jpg)
www.caveon.com 18
Example Evidence, Prevention
Per the U.S DOE, implemented & documented:
• Test administration manual (TAM)• Chain of custody procedures• Efforts to reduce the likelihood of
unauthorized communication• Security procedures for
accommodations providers• Protocol and consequences for violations• Internal communication procedures
![Page 19: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/19.jpg)
www.caveon.com 19
Example Evidence, Prevention
Additional suggestions:
• State Test Security Audit or Evaluation Report
• State Test Security Handbook• Technical Report with appropriate sections• DFA and monitoring procedures• Training materials for DTCs, STCs, TAs and
Proctors
![Page 20: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/20.jpg)
www.caveon.com 20
A List of Resources to Help*• TILSA Guidebook, Section 3• TILSA Lessons Learned Report, Section 2• Operational Best Practices for State
Assessments• Handbook of Test Security, Chapter 6 &7• Standards for Educational and
Psychological Testing by AERA/APA/NCME • Test and Data Integrity in the
Administration of Statewide Student Assessment Programs, NCME (2013)
*This is a partial list. You will be provided with a comprehensive list of linked resources after the session, which will address Parts 1-4.
![Page 21: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/21.jpg)
www.caveon.com 21
Part 2 - Detection
Detection of Testing Irregularities
![Page 22: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/22.jpg)
www.caveon.com 22
Example Evidence, DetectionPer the U.S DOE, implemented and documented:
• Incident reporting procedures• Data analysis procedures• Historical data of previous year’s
incidents
![Page 23: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/23.jpg)
www.caveon.com 23
Example Evidence, DetectionAdditional Suggestions:• Those in Part 1, and:• Procedures for securing testing room
from cheating aids• Data Forensics analysis• Monitoring the proctoring process• Web & social media monitoring• Incident intake and tracking
mechanism• Training materials• Tip line
![Page 24: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/24.jpg)
www.caveon.com 24
Part 3 - Remediation
Remediation following any test security incident involving any of the State’s assessments
![Page 25: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/25.jpg)
www.caveon.com 25
Example Evidence, Remediation
Per the U.S DOE, policies & procedures in place, including:
• Clear definition of an incident• Levels of severity identified
• Security Incidence Response Plan• Communication strategies• Stakeholders identified • Media relations tactics
![Page 26: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/26.jpg)
www.caveon.com 26
Example Evidence, Remediation
Additional Suggestions:• Those in Part 1-2, and:
• Escalation Path Flowchart• Documentation of procedures for
responding to testing breaches• Documentation of state's consequences
for confirmed violations
![Page 27: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/27.jpg)
www.caveon.com 27
Part 4 - Investigation
Investigation of any alleged or factual test irregularities
![Page 28: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/28.jpg)
www.caveon.com 28
Example Evidence, Investigation
Per the U.S DOE, investigation procedures including policies for:
• Securing evidence• Rules for investigating• Standards for districts• Privacy policies• Escalation path flowchart
![Page 29: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/29.jpg)
www.caveon.com 29
Example Evidence, Investigation
Additional Suggestions:Those in Part 1-3, and:• Data forensics analysis indicating irregularities• Web patrol findings• Testing Irregularity or Incident Report from a
school or district• Log of tips received by state DOE • Licensure action/other sanctions imposed
following investigation• Risk analysis• Investigative Response Plan• Investigative Training
![Page 30: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/30.jpg)
www.caveon.com 30
Key Points Summarized
• Detailed, documented policies and procedures are key.
• A State Test Security Handbook is an invaluable tool for this process.
• Keep careful records of past efforts and incidents.
• Training must extend beyond test administrators.
![Page 31: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/31.jpg)
www.caveon.com 31
We can help!• Assistance with preparing Peer
Review documentation• Security Audit• Security Handbook• Planning and implementing data forensics analyses• Investigation Training and Process
development
![Page 32: Caveon Webinar Series - Mastering the US DOE Test Security Requirements January 2016](https://reader035.fdocuments.in/reader035/viewer/2022070515/587bf78c1a28ab7c668b4d73/html5/thumbnails/32.jpg)
www.caveon.com 32
Thank You! Follow Caveon on twitter @caveonCheck out our blog www.caveon.com/blogLinkedIn Group “Caveon Test Security”
Dr. John [email protected]
Dr. John [email protected]
Marc Weinstein, [email protected]