Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob...
-
Upload
ethan-chambers -
Category
Documents
-
view
216 -
download
0
Transcript of Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob...
![Page 1: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/1.jpg)
Exam Security IssuesLegal, Psychometric, and Administrative Perspectives
Amin Saiar, PhDBob Albert, MBA
Amanda Wolkowitz, PhDJennifer Ancona Semko, JD
![Page 2: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/2.jpg)
Overview of Presentation
1. Introduction to Security Risks and Analytics
2. Case Study: Invalidation of Examinee Scores
3. Legal Considerations in Cheating Investigation and Intervention
![Page 3: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/3.jpg)
Introduction to Security Risks and Analytics
Amin Saiar, PhDJohn Weiner, MSGreg Hurtz, PhD
![Page 4: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/4.jpg)
Security Analytics
• Can be used to investigate issues or events, or to identify issues to investigate
• It is most effective when combined with other methods of intervention
• Play a critical role in testing-program management
![Page 5: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/5.jpg)
Exam Security RisksRisk Behavior Analytic
Cheating Collusion / Copying Response similarity
Piracy HarvestingTesting time vs. items answered
Proxy Stand-inBiometrics, photo,
proctor
Volatile Retake Quasi-key accessTest 2 - Test 1 comparison
Compromised content
Change in performance Pass rate analyses
![Page 6: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/6.jpg)
Response Similarity
• Used to identify examinees with an unexpectedly similar number of matching responses with another examinee
• An examinee is flagged if number of matched responses are significantly higher from distribution of matched scores
• In such case, collusion or copying may have occurred
ABACABDCADBCCBBAAB
DBACCBCDAABCABDCAB
![Page 7: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/7.jpg)
Collusion vs. Copying
An important distinction because the type of damage done is different and analytics differ in ability to detect
them
Copying
Copying of answers from a source
Collusion
Advance knowledge of some or all of the
items
![Page 8: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/8.jpg)
Monte Carlo Simulations
• Simulated data created to test different indices’ ability to correctly identify collusion and copying
• Different percentages of “examinees” throughout the ability range were randomly selected to “cheat” on different percentages of items
• Response operator characteristic (ROC) curve used to measure detection ability
![Page 9: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/9.jpg)
Indices Compared
• HT (Sijtmsa & Meijer)– Nonparametric person-fit statistic
• Kappa (Cohen)– Cohen’s kappa inter-rater agreement
coefficient for categorical items• J Indices (Weiner, Hurtz, & Saiar)
– Regression-based approaches to identify large residuals
![Page 10: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/10.jpg)
J2 Index
1. Compute Max_nMatch by form Each candidate’s maximum # of matching responses with any other candidate
2. Conduct regression analysisTest score onto Max_nMatch
3. Compute residualsObserved minus predicted
4. Standardize residuals within sampleConversion to z-score
0 20 40 60 80 100 120
0
20
40
60
80
100
120
140
![Page 11: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/11.jpg)
ROC Curve
• Plots true positive rate against false positive rate
• Useful in determining if actual cheaters are being detected
![Page 12: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/12.jpg)
Area under ROC Curve
![Page 13: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/13.jpg)
Takeaways
• Certain analytics and investigative methods are most appropriate for specific types of cheating
• The specific risks that a program is susceptible to should be identified or determined
• The use of multiple analytic tools and investigative methods is advised
![Page 14: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/14.jpg)
Case Study:Invalidation of Examinee
Scores
Bob Albert, MBAAmanda Wolkowitz, PhD
![Page 15: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/15.jpg)
National Dental Examining Board of Canada (NDEB)
• 1953 Act of Parliament – establish qualifying conditions for a single national standard of qualification for general dentists in Canada
• Issues certificates to those who meet the standard
![Page 16: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/16.jpg)
10 Provincial Licensing Authorities
![Page 17: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/17.jpg)
NDEB Certification Process
Written Examination
and OSCE
Certified and portable
Graduates of Accredited Programs
Graduates of Non-
accredited Programs
NDEB Equivalency Process
Accredited Qualifying Program
![Page 18: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/18.jpg)
Bylaws
• 20.01 explains the consequences of compromising an examination or assessment to examinees:– Results may be voided– Privilege of repeating the examination or
assessment may be denied– A person may be dismissed immediately
prior to a full report
![Page 19: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/19.jpg)
Assessment of Clinical Judgment (ACJ)
• 1 day, 3 booklet paper-pencil based (2 case-based and 1 radiology) with extended match questions
• Administered annually in June and December
• Must earn an equated test score of 75 to pass the exam
![Page 20: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/20.jpg)
Sample questionQuestion: (select one or more correct answers.)There is evidence of calculus on the
Distal of tooth 1.8
Mesial of tooth 1.8
Distal of tooth 1.7
Mesial of tooth 1.7
Distal of tooth 1.6
Mesial of tooth 1.6
+.25
-1
(0)
-1
+.25
-1
-1
+.25
+.25
-1
-1
Distal of tooth 1.5
Mesial of tooth 1.5
Distal of tooth 1.4
Mesial of tooth 1.4
Distal of tooth 1.3
![Page 21: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/21.jpg)
The 2014 ACJ Incident
• Approximately 634 participants nationally
• 203 participants at the Toronto site• Participants seated according to a
pre-set seating chart• Exam is proctored by invigilators and
supervised by a test administrator.
![Page 22: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/22.jpg)
Suspicious Behavior Reported
• Invigilator alerted by a participant of two individuals having conversations during the exam.
• After noting the incident, participants asked to stop talking and not have communications.
• Test Administrator notified of the incident and noted the seat numbers.
![Page 23: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/23.jpg)
Suspicious Behavior Suspected
• Exam wide answer sheets collected and scanned
• Commercial based detection analysis software not possible due to multiple choice/response item structure
• In-house group comparison statistical analysis revealed significance similarities in responses
![Page 24: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/24.jpg)
• Alpine Testing was engaged to analyze and report on the suspicious behavior.
Suspicious Behavior Reported
![Page 25: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/25.jpg)
What did we find?
![Page 26: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/26.jpg)
In your testing program, have you successfully invalidated candidates’ scores based on questions about the validity of that score?
A. No. We have not identified specific candidates whose scores were in question.
B. No. We identified candidates whose scores were in question, but did not invalidate their scores.
C. Yes. We have invalidated candidates’ scores based on questionable validity.
![Page 27: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/27.jpg)
Case Study
• June 2014 administration– 634 total participants– 203 completed paper-pencil exam at
site of interest
![Page 28: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/28.jpg)
Case Study
• 120 items divided into 3 “books”– Extended match multiple-choice– Each option has a point value between -
3 and 1 (including fractional amounts) associated with it
– Sum of points equals item score– Item score range: 0 to 1
![Page 29: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/29.jpg)
1. Test StatisticsAll ASCT
SiteNon-ASCT Site
Participant Count
654 203 451
Mean 83.0 83.4 82.9SD 8.6 8.1 8.8Alpha Reliability
0.779 0.758 0.787
No suspicious findings when comparing overall test statistics.
![Page 30: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/30.jpg)
2. Correlation Analysis
• Correlation of all 203 participants at site
Pair r p ID #1 Seat
ID #2 Seat
1 0.87 <0.001 46 452 0.61 <0.001 202 53 0.60 <0.001 93 114 0.63 <0.001 161 1495 0.62 <0.001 129 120
Participants with Both Statistically Significant Correlations and Correlations Above r = 0.60
![Page 31: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/31.jpg)
2. Correlation Analysis
• What does this mean?– Pair 1 had < 1 in
10,000,000,000,000,000,000,000, 000,000,000,000,000 (< 1 in ten undecillion) chance of having this much similarity in their responses
– The chances of winning the Powerball are much greater (1 in 175,000,000)
![Page 32: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/32.jpg)
3. Identical Response Option
203 participants = 20,503 paired Comparisons
Pair 1 shared identical responses to 84.2% of the items. This is over 20% moreidentical item responses than any other pair at this site.
Pair 1
![Page 33: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/33.jpg)
4. Seating Chart
• Seats of those with significantly high correlations
![Page 34: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/34.jpg)
4. Seating Chart
• Seats of those with high number of identical responses (at least 69)
![Page 35: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/35.jpg)
5. Probability Analysis on Pair
• Responses of Pair 2407– 21 items in which both scored 0– Exact same response to 17 of these 21
• What is the probability that two randomly selected examinees would have 17 or more identical wrong answers on these 21 items?
![Page 36: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/36.jpg)
5. Probability Analysis on Pair
Less than 1 in 33 million
![Page 37: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/37.jpg)
Summary of Pair 24071. Proctor noticed test misconduct2. Highest correlation at test site with a correlation unlikely
to occur by chance3. Highest number of identical item responses that were
highly unlikely to occur by chance (100% identical in book 1 of 3 books)
4. Highest number of identical wrong responses receiving 0 points that were highly unlikely to occur by chance
5. Seated right next to each other
![Page 38: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/38.jpg)
What happened to these candidates?
![Page 39: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/39.jpg)
Suspicious Behavior Reprimanded
• After legal counsel and sufficient proof of compromise…
• Examinations committee voted to void the participants’ results and be “ineligible to participate in any future NDEB assessments or examinations”.
• Communicated the results to the public on the NDEB web site.
![Page 40: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/40.jpg)
Additional Procedures
• Alpine Testing contracted to develop a design to conduct high level screening analyses of test scores.
• NDEB is integrating the design into its own custom question bank system.
• NDEB will incorporate the reports into its routine post exam item performance analyses.
![Page 41: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/41.jpg)
Legal Considerations in Cheating Investigation and
Intervention
Jennifer Ancona Semko, JD
![Page 42: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/42.jpg)
Key Question: Can you defend your actions?
Three fundamentals:
(1) Matter of contract(2) Deference (3) Good faith
![Page 43: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/43.jpg)
What’s a candidate agreement?
Do I need one?• Contract: An agreement between two or more persons which creates an obligation to do or not to do a particular thing. A legal relationship consisting of the rights and duties of the contracting parties. Black’s Law Dictionary, Sixth Edition
• Your agreement with test takers defines the relationship
• Memorializes your (and their) rights and obligations
• If done properly, makes expectations (and remedies) clear
![Page 44: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/44.jpg)
What does your candidate agreement say?• Are candidates on notice that sharing items is a breach?• Are candidates on notice that studying from recalled items
is improper?• “It must be OK . . . they told us about everything else,
but not about this.”• Did you reserve the right to invalidate scores? Suspend or
ban access to examination? To take other action?• What are the grounds for action? Is there a “catch all”?• Are candidates required to cooperate in investigations?• Do you regularly review your agreement language?• Do you have uniform security procedures and policies in
place?
![Page 45: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/45.jpg)
When was the last time you reviewed and updated your Candidate Agreement?
A. Our program regularly reviews and updates our Candidate Agreement.
B. No one has touched our Candidate Agreement in at least a few years.
C. We have a Candidate Agreement?
![Page 46: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/46.jpg)
Rights of Exam Administrators
• Murray v. Educational Testing Service, 170 F.3d 514 (5th Cir. 1999)– “Several courts, including this one, have recognized
the importance of allowing ETS to assure itself of the validity of students’ scores through internal review procedures.”
– “. . . ETS has an obligation to provide, or use its best efforts to provide, only valid scores to the colleges and universities that rely on ETS’s services.”
– “Moreover, ETS has the right to protect its own reputation by assuring the reliability of the information it provides.”
![Page 47: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/47.jpg)
Rights of Exam Administrators
• Scott v. ETS, 600 A.2d 500 (N.J. Super. 1991)– “We are satisfied that the relevant public and private
interests are fairly accommodated by a procedure which permits ETS to cancel scores upon an adequate showing of substantial question as to their validity, without any necessity for a showing of actual cheating or other misconduct.”
• K.D. v. ETS, 386 N.Y.S.2d 747 (1976)– “. . . If [ETS] reasonably believed that the test scores . . .
did not accurately reflect [plaintiff’s] aptitude for law school, it acted within its right to protect its own image as well as its obligation to the schools who are its clients in canceling plaintiff’s scores . . .”
![Page 48: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/48.jpg)
Deference to Exam Administrators
Murray v. ETS, 170 F.3d 514 (5th Cir. 1999) (SAT Exam)
• Louisiana basketball player; needed 820 on SAT
• Scored 700, then 1300– Similarity to nearby student (3 in 100 million odds)– Scored 800 on retake
• “ETS’s contract with Murray clearly and explicitly reserved to ETS the right to withhold any scores ETS had reason to believe were not valid. The only contractual duty ETS owed to Murray was to investigate the validity of Murray’s scores in good faith.”
![Page 49: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/49.jpg)
Good faith applies to football, too
Langston v. ACT, 890 F.2d 380 (11th Cir. 1989) (ACT Exam)
• Alabama football player; scored 10 on ACT; then 20
• Inconsistent with GPA; unusual similarity to nearby student
• “Under the governing law, the outcome of plaintiff’s case does not turn on whether or not plaintiff cheated on his exam, but only on whether or not ACT carried out its contractual obligations in good faith.”
![Page 50: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/50.jpg)
Did you act in good faith?
• Court will look for evidence of good faith actions
• Consistency, uniformity, fairness
• Consideration of all available information (including test taker’s response)
• State actors also owe Constitutional Due Process
![Page 51: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/51.jpg)
Which creates stronger foundation for action?
• Your Candidate Agreement Says:
(A)“Candidates proven to have cheated during the examination may have their scores invalidated.”
OR
(B)“We reserve the right to cancel any test score if, in our sole opinion, there is adequate reason to question its validity.”
![Page 52: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/52.jpg)
What is the most significant obstacle preventing you from invalidating candidate scores where appropriate?
A. Don’t have the data needed to support invalidation.
B. Lack of management/legal department support for invalidation decisions.
C. Unsure of our program’s rights to invalidate.
D. Something else.
![Page 53: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/53.jpg)
Legal Considerations: Take-aways
• DON’T have to prove cheating• DO have to act in good faith• It’s about validity, not cheating• Matter of contract, deference, good
faith• Candidate Agreement language is
key
![Page 54: Exam Security Issues Legal, Psychometric, and Administrative Perspectives Amin Saiar, PhD Bob Albert, MBA Amanda Wolkowitz, PhD Jennifer Ancona Semko,](https://reader037.fdocuments.in/reader037/viewer/2022103100/56649f1e5503460f94c36249/html5/thumbnails/54.jpg)
THANK YOU• Bob Albert, MBA
– Chief Technology Officer, National Dental Examining Board Of Canada– [email protected]
• Amin Saiar, PhD– Manager of Psychometric Services, PSI Services LLC– [email protected]
• Jennifer Ancona Semko, JD– Partner, Baker & McKenzie LLP– [email protected]
• Amanda Wolkowitz, PhD– Psychometrician, Alpine Testing Solutions– [email protected]