Data collection and learning analytics in programming education
-
Upload
petri-ihantola -
Category
Data & Analytics
-
view
31 -
download
0
Transcript of Data collection and learning analytics in programming education
![Page 1: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/1.jpg)
Programming Assessment and
Data Collection
Petri Ihantola
![Page 2: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/2.jpg)
Programming Assessment and
Data Collection
Petri IhantolaAssistant Professor at Tampere University of Technology (2014 - ), D.Sc. (Tech) from Aalto University in 2011, Software Engineer in Test at Google (2007-2009), Teaching various large-class programming courses at Aalto University, former Helsinki University of Technology (2004 - 2014)
![Page 3: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/3.jpg)
Arto Vihavainen, Ville Karavirta, Juha Helminen,
Juha Sorva, Otto Seppälä, ...
![Page 4: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/4.jpg)
![Page 5: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/5.jpg)
![Page 6: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/6.jpg)
![Page 7: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/7.jpg)
image: http://www.fatandsassymama.com/wp-content/uploads/2013/08/baking.jpg
Programming is a process
![Page 8: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/8.jpg)
Programming is a process
Feedback should be provided from how students do what they do, not only whether
the end product tastes good or not
![Page 9: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/9.jpg)
Traditionally, feedback has focused on the end products
image: https://www.flickr.com/photos/clement127/15004844674 cc (by-nc-nd)
![Page 10: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/10.jpg)
Traditionally, feedback has focused on the end products
correctness, efficiency, style, design, ...
Ala-Mutka. A survey of automated assessment approaches for programming assignments. Computer Science Education, 15(2):83-102, 2005.
![Page 11: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/11.jpg)
May encourage ineffective trial and error processes
image: https://www.flickr.com/photos/oliveira_comp/14261335089 cc (by-nc-sa)
![Page 12: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/12.jpg)
May encourage ineffective trial and error processes tackled by limiting the number of
submissions/feedback, using time penalties, making each exercise unique,
organizing contests, ... Ihantola et al. 2010. Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research. 86-93.
![Page 13: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/13.jpg)
Hey, wait a moment... isn't this already already an example of providing
feedback from the proces
![Page 14: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/14.jpg)
So what makes it hard to provide even better feedback
(from processes)?
![Page 15: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/15.jpg)
So what makes it hard to provide even better feedback
(from processes)?
![Page 16: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/16.jpg)
Systems collect data
But when trying to get the big picture, we still have to do many assumptions
image: unknown
![Page 17: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/17.jpg)
image: unknown
![Page 18: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/18.jpg)
Houston, we have a problem
image: NASA, PD
![Page 19: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/19.jpg)
Let's look at easier problems first
Ihantola & Karavirta (2011). Two-Dimensional Parson’s Puzzles: The Concept, Tools, and First Observations. In: Journal of Information Technology Education: Innovations in Practice 10, pp. 1–14.
![Page 20: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/20.jpg)
Helminen, Ihantola, Karavirta, Malmi (2012). How Do Students Solve Parsons Programming Problems? – An Analysis of Interaction Traces. In Proceedings of the 8th International Computing Education Research Conference, pp. 119–126, Auckland, New Zealand.Karavirta, Helminen, Ihantola (2012). A mobile learning application for parsons problems with automatic feedback. In: Koli Calling ’12: Proceedings of the 12th Koli Calling International Conference on Computing Education Research. Koli, Finland: ACM, pp. 11–18. ISBN: 978-1-4503-1795-5. (best system paper award)
Looks like the student got stuck here, lets
help.
![Page 21: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/21.jpg)
Back to real life and real programming environments
![Page 22: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/22.jpg)
Back to real life and real programming environments
![Page 23: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/23.jpg)
How much information is lost when storing snapshots
at different granularities?
submissions, save points, key-strokes
Vihavainen, Luukkainen & Ihantola. 2014. Analysis of source code snapshot granularity levels. In Proceedings of the 15th Annual Conference on Information technology education (SIGITE '14). ACM
![Page 24: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/24.jpg)
Novice programmers
image: https://www.flickr.com/photos/donnieray/8658314801/ cc (by)
![Page 25: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/25.jpg)
● Introduction to Programming (MOOC)● Spring 2014, University of Helsinki● 1166 students● 93231 submissions● 1.3 million saves, runs and tests● 37 million events (insert, remove, paste)
Novice programmers
![Page 26: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/26.jpg)
● 50% of students work on assignments that they never submit - no information on the progress in such (harder?) assignments
● Programmers with previous experience move more straightforward (make less sidesteps)
● 6.3 snapshots / submission and 30 key events / snapshot
Some findings
![Page 27: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/27.jpg)
So... collect the data while you can. It cannot be regenerated, e.g., interpolated.
![Page 28: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/28.jpg)
Any examples of what to do with more accurate data?
![Page 29: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/29.jpg)
Can we automatically detect student’s perceived difficulty
as they are working on programming tasks?
Petri Ihantola, Juha Sorva, and Arto Vihavainen. 2014. Automatically detectable indicators of programming assignment difficulty. In Proceedings of the 15th Annual Conference on Information technology education (SIGITE '14). ACM, New York, NY, USA, 33-38. (best paper award)
![Page 30: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/30.jpg)
Can we understand how the way of how students type
their code evolves over time?
Arto Vihavainen, Juha Helminen, and Petri Ihantola. 2014. How novices tackle their first lines of code in an IDE: analysis of programming session traces. In Proceedings of the 14th Koli Calling International Conference on Computing Education Research (Koli Calling '14). ACM, New York, NY, USA, 109-116.
![Page 31: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/31.jpg)
What next?
![Page 32: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/32.jpg)
The three main goals of feedback are to help a learner understand and learn about
1. the learning goals2. own progress towards these goals3. activities needed to make better process
Hattie & Timperley (2007). The Power of Feedback. Review of Educational Research, 77(1), 81-112.
![Page 33: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/33.jpg)
Time perspective in educational data mining will change to more fine grained
![Page 34: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/34.jpg)
Plenty of research opportunities from course-level analysis to modeling
individual students
![Page 35: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/35.jpg)
However, we should not ignore the vast amount of
previous research
e.g., Juha Helminen, Petri Ihantola, and Ville Karavirta. 2013. Recording and analyzing in-browser programming sessions. In Proceedings of the 13th Koli Calling International Conference on Computing Education Research (Koli Calling '13). 13-22.
![Page 36: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/36.jpg)
ITiCSE working group in July
![Page 37: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/37.jpg)
https://us.pycon.org/2015/events/edusummit/
Python Education Summitvoting of the topics is open
![Page 38: Data collection and learning analytics in programming education](https://reader030.fdocuments.in/reader030/viewer/2022032714/55aaec091a28ab7d388b45c4/html5/thumbnails/38.jpg)
Thank [email protected] @ihantolawww.linkedin.com/in/ihantolatinyurl.com/PetriPublications