Detecting and addressing contract cheating in online ...transformingassessment.com › sites ›...
Transcript of Detecting and addressing contract cheating in online ...transformingassessment.com › sites ›...
![Page 1: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/1.jpg)
29 Apr 2020: 07:00AM UTC Detecting and addressing contract cheating in online assessment Associate Professor Phillip (Phill) Dawson Deakin University, Melbourne, Australia Justtoletyouknow:
Bypar'cipa'nginthewebinaryouacknowledgeandagreethat:Thesessionmayberecorded,includingvoiceandtextchatcommunica'ons(arecordingindicatorisshowninsidethewebinarroomwhenthisisthecase).Wemayreleaserecordingsfreelytothepublicwhichbecomepartofthepublicrecord.Wemayusesessionrecordingsforqualityimprovement,oraspartoffurtherresearchandpublica'ons.
Webinar Series
e-Assessment SIG
WebinarHostsProfessorGeoffCrisp,DeputyVice-Chancellor&Vice-PresidentAcademicUniversityofCanberrag.crisp[at]canberra.edu.auDrMathewHillier,MacquarieUniversitymathew.hillier[at]gmail.com
![Page 2: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/2.jpg)
deakin.edu.au/cradle @phillipdawson
Detecting and addressing contract cheating in online assessment Associate Professor Phillip (Phill) Dawson Centre for Research in Assessment and Digital Learning (CRADLE) Deakin University, Melbourne, Australia
DeakinUniversityCRICOSProviderCode:00113B
Slidesandresourcesat:hUp://philldawson.com/ta-chea'ng
![Page 3: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/3.jpg)
Many thanks to
A/Prof Wendy Sutherland-Smith
Rebecca Awdry …and the anonymous participants in our studies and workshops!
…academic integrity researchers and experts around the world…
![Page 4: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/4.jpg)
![Page 5: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/5.jpg)
Some declarations before we go any further…
I support AfL & AI. I think cheating is a symptom of broader problems. I think universities have a responsibility try to prevent and detect cheating. I receive research funding from ed tech companies (including Turnitin), but these are my opinions, not theirs. My mum helped me contract cheat in year four.
![Page 6: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/6.jpg)
Tohellwithsuspense
![Page 7: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/7.jpg)
1. Contract cheating is a big problem.
Three things to take from this presentation
2. We need to balance academic integrity and assessment security
3. You can learn to detect contract cheating
![Page 8: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/8.jpg)
1. Contract cheating is a big problem. -Students don’t find it, it finds them. -A significant proportion of students contract cheat. -Only a tiny fraction of contract cheating gets detected.
![Page 9: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/9.jpg)
![Page 10: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/10.jpg)
![Page 11: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/11.jpg)
…research paper…
…hate stats…
…due tomorrow…
Amigud, A. (2020). Cheaters on Twitter: an analysis of engagement approaches of contract cheating services. Studies in Higher Education, 45(3), 692-705. doi:10.1080/03075079.2018.1564258
![Page 12: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/12.jpg)
![Page 13: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/13.jpg)
![Page 14: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/14.jpg)
![Page 15: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/15.jpg)
Australia, extrapolated from Bretag et al 2018
~80k total avg.
~2k per uni
UK
![Page 16: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/16.jpg)
Banning essays won’t save us. Authentic assessment won’t save us. Contract cheating can occur with any task type. (Ellis et al, 2019)
Ellis, C., van Haeringen, K., Harper, R., Bretag, T., Zucker, I., McBride, S., . . . Saddiqui, S. (2019). Does authentic assessment assure academic integrity? Evidence from contract cheating data. Higher Education Research & Development, 1-16. doi:10.1080/07294360.2019.1680956
![Page 17: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/17.jpg)
Rapid turnaround times won’t save us. Contract cheaters can probably do it quicker than students. (Wallace & Newton, 2014)
Wallace, M. J., & Newton, P. M. (2014). Turnaround time and market capacity in contract cheating. Educational Studies, 40(2), 233-236. doi:10.1080/03055698.2014.889597
![Page 18: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/18.jpg)
Exams won’t save us. ‘Third party cheating’ is likely more common in exams than assignments. ‘Third party cheating’ is likely detected less often in exams than assignments. (Harper et al, 2020)
Harper, R., Bretag, T., & Rundle, K. (2020). Detecting contract cheating: examining the role of assessment type. Higher Education Research & Development, 1-16. doi:10.1080/07294360.2020.1724899
![Page 19: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/19.jpg)
Remote proctoring might help, but vendors won’t let me test it out. Treat their claims with caution. (Dawson, forthcoming)
Dawson, P. (forthcoming). Defending Assessment Security in a Digital World: Preventing E-Cheating and Supporting Academic Integrity in Higher Education. Routledge.
![Page 20: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/20.jpg)
2. We need to balance -promoting academic integrity -assuring assessment security
![Page 21: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/21.jpg)
Fundamental values of academic integrity • Honesty • Trust • Fairness • Respect • Responsibility • Courage
https://academicintegrity.org/fundamental-values/
![Page 22: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/22.jpg)
![Page 23: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/23.jpg)
![Page 24: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/24.jpg)
![Page 25: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/25.jpg)
![Page 26: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/26.jpg)
Addressing contract cheating requires…
Academic Integrity • Trusting • Educative • Proactive
Think ‘crime prevention’
Assessment security • Detecting • Punitive • Proactive or reactive
‘policing’ or ‘surveillance’
It’s a balance, not a dichotomy
![Page 27: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/27.jpg)
Talk with students. Contract cheating sites lie and even blackmail. (Rowland et al, 2018; Yorke, in press)
Rowland, S., Slade, C., Wong, K.-S., & Whiting, B. (2018). ‘Just turn to us’: the persuasive features of contract cheating websites. Assessment & Evaluation in Higher Education, 43(4), 652-665. doi:10.1080/02602938.2017.1391948 Yorke, J., Sefcik, L., & Veeran-Colton, T. (in press). Contract cheating and blackmail: a risky business? Studies in Higher Education. doi:10.1080/03075079.2020.1730313
![Page 28: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/28.jpg)
Talk with students. “You don’t always get what you pay for.”
(Sutherland-Smith & Dullaghan)
Sutherland-Smith, W., & Dullaghan, K. (2019). You don’t always get what you pay for: User experiences of engaging with contract cheating sites. Assessment & Evaluation in Higher Education, 44(8), 1148-1162. doi:10.1080/02602938.2019.1576028
![Page 29: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/29.jpg)
0 5 10 15 20 25 30 35 40
Short turnaround time
Heavily weighted task
Series of small graded tasks
Research, analysis and thinking skills
No ‘right’ answer
Integrate knowledge/skills vital to programme
Relevant professional skills
Major part of nested task
Small part of nested task
In-class task
Personalised and unique
Viva
Reflection on practicum Bretag, T., Harper, R., Burton, M., Ellis, C., Newton, P., van Haeringen, K., et al. (2019). Contract cheating and assessment design: exploring the relationship. Assessment & Evaluation in Higher Education, 44(5), 676-691.
Listen to students. Students’ perceptions of the likelihood of contract cheating (%)
![Page 30: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/30.jpg)
Assessment security: “measures taken to harden assessment against attempts to cheat; this includes approaches to detect and evidence attempts to cheat, as well as approaches to make cheating more difficult.” (Dawson, forthcoming)
Dawson, P. (forthcoming). Defending Assessment Security in a Digital World: Preventing E-Cheating and Supporting Academic Integrity in Higher Education. Routledge.
![Page 31: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/31.jpg)
3. You can learn to detect contract cheating -Indicators -Training -Software
![Page 32: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/32.jpg)
![Page 33: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/33.jpg)
Can markers detect contract cheating? (Dawson & Sutherland-Smith, 2018)
Can training improve detection rates? (Dawson & Sutherland-Smith, 2019)
Can software improve detection rates? (Dawson, Sutherland-Smith & Ricksen, in press)
Dawson, P., & Sutherland-Smith, W. (2018). Can markers detect contract cheating? Results from a pilot study. Assessment & Evaluation in Higher Education, 43(2), 286-293. doi:10.1080/02602938.2017.1336746 Dawson, P., & Sutherland-Smith, W. (2019). Can training improve marker accuracy at detecting contract cheating? A multi-disciplinary pre-post study. Assessment & Evaluation in Higher Education, 44(5), 715-725. doi:10.1080/02602938.2018.1531109 Dawson, P., Sutherland-Smith, W., & Ricksen, M. (in press, published online 2019). Can software improve marker accuracy at detecting contract cheating? A pilot study of the Turnitin authorship investigate alpha. Assessment & Evaluation in Higher Education, 1-10. doi:10.1080/02602938.2019.1662884
![Page 34: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/34.jpg)
0 20 40 60 80 100
Trained to look for it
Looking for it
Not looking for it
Accuracy at detecting contract cheating (%)
e.g. Lines 2016; Medway et al 2018
Lines, L. (2016). Ghostwriters guaranteeing grades? The quality of online ghostwriting services available to tertiary students in Australia. Teaching in Higher Education, 1-26. doi:10.1080/13562517.2016.1198759 Medway, D., Roper, S., & Gillooly, L. (2018). Contract cheating in UK higher education: A covert investigation of essay mills. British Educational Research Journal, 44(3), 393-418. doi:10.1002/berj.3335
![Page 35: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/35.jpg)
0 20 40 60 80 100
Trained to look for it
Looking for it
Not looking for it
False positive rate (%)
e.g. Lines 2016; Medway et al 2018
Lines, L. (2016). Ghostwriters guaranteeing grades? The quality of online ghostwriting services available to tertiary students in Australia. Teaching in Higher Education, 1-26. doi:10.1080/13562517.2016.1198759 Medway, D., Roper, S., & Gillooly, L. (2018). Contract cheating in UK higher education: A covert investigation of essay mills. British Educational Research Journal, 44(3), 393-418. doi:10.1002/berj.3335
![Page 36: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/36.jpg)
![Page 37: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/37.jpg)
Trying to look for contract cheating is a good first step. Most of the indicators of contract cheating were task-specific.
Dawson, P., & Sutherland-Smith, W. (2018). Can markers detect contract cheating? Results from a pilot study. Assessment & Evaluation in Higher Education, 43(2), 286-293. doi:10.1080/02602938.2017.1336746 Dawson, P., & Sutherland-Smith, W. (2019). Can training improve marker accuracy at detecting contract cheating? A multi-disciplinary pre-post study. Assessment & Evaluation in Higher Education, 44(5), 715-725. doi:10.1080/02602938.2018.1531109
![Page 38: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/38.jpg)
Potential generic indicators of contract cheating
reflection done poorly
metadata
unusual mistakes
not using materials from class
reads as something by a generalist
strange formatting
does not address question
wrong task type
![Page 39: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/39.jpg)
![Page 40: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/40.jpg)
Training works if you have access to contract cheating assignments
Dawson, P., & Sutherland-Smith, W. (2019). Can training improve marker accuracy at detecting contract cheating? A multi-disciplinary pre-post study. Assessment & Evaluation in Higher Education, 44(5), 715-725. doi:10.1080/02602938.2018.1531109
![Page 41: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/41.jpg)
Viva had 100% detection rate (Needs replication before we publish but it’s a good sign)
Tomorrow! Reg http://taw.fi/30apr2020
![Page 42: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/42.jpg)
![Page 43: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/43.jpg)
‘Cheat-proofing’ every act of assessment is probably impossible and definitely a bad idea.
Focus on securing those acts of assessment that matter to the degree program outcomes. Focus on developing academic integrity in the others.
![Page 44: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/44.jpg)
Thank you!
Slides and resources at: http://philldawson.com/ta-cheating
![Page 45: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/45.jpg)
1. Contract cheating is a big problem. -Students don’t find it, it finds them. -Only a tiny fraction of contract cheating gets detected
Three things to take from this presentation 2. We need to balance -promoting academic integrity -assuring assessment security
3. You can learn to detect contract cheating -Indicators -Training -Software
Slides and resources at: http://philldawson.com/ta-cheating
![Page 46: Detecting and addressing contract cheating in online ...transformingassessment.com › sites › default › files › ... · Detecting and addressing contract cheating in online](https://reader030.fdocuments.in/reader030/viewer/2022041112/5f1780360f0a5109ff5624f0/html5/thumbnails/46.jpg)
Webinar Series
Webinar Session feedback With thanks from your hosts Professor Geoff Crisp, Deputy Vice-Chancellor & Vice-President Academic University of Canberra g.crisp[at]canberra.edu.au Dr Mathew Hillier, Macquarie University mathew.hillier[at]gmail.com Recording available http://transformingassessment.com
e-Assessment SIG Register http://taw.fi /30apr2020
Next session 30 April 2020