PAZAPA: identifying and supporting at-risk students
29
PAZAPA: identifying and supporting at-risk students Serge Verlinde Jordi Heeren Nathalie Nouwen KU Leuven, Leuven Language Institute
Transcript of PAZAPA: identifying and supporting at-risk students
PAZAPA: identifying and supporting at-risk students
Serge VerlindeJordi Heeren
Nathalie NouwenKU Leuven, Leuven Language Institute
2
French proficiency in Flanders is on the decline
French is definitely needed on the labourmarket
uncomfortable position of French language teachers at higher education institutions
Reseach question
3
How toidentify& support at-risk students (for French)
byproviding relevant feedback at the right moment?
Rather than providing students with the message ‘we missed you last week; please catch up the next days’ […], intervention can now focus on concrete learning behaviors: working on the tendency to procrastinate, searching for alternative learning materials that may be of more challenging nature. In otherwords: the learning feedback will change from product feedback to feedback provided at the process level, the most effective type of feedback.
Tempelaar, Rienties and Nguyen (2017). Towards Actionable Learning AnalyticsUsing Dispositions
Subject of the study
4
• 367 students1st year of fac. of Law (2015-2016)university of Leuven (KU Leuven)
• selected from dataset of 3909 students (2015-2017)
Teaching approach: blended learning
5
language for specific purposes course
basic grammar (and vocabulary) exercises
student data
initial testresult
learningmanagementsystem
end mark -continuous assessment
Student data
6
• previous studies• end mark for previous studies• number of hours for French and Mathematics• mother tongue
• self-evaluation (0-10)
Initial test
7
• C-test
o 100 itemso often used as placement test
Initial test
8
• Did the students take the test seriously? YES
Initial test
9
• Predictive value of test scores?o Correlation test score – exam score:
• R = .55 (p<0,001)o Regression analysis
• R² = 0.50%
variables β pEnd resultsecundary school
.292 <0.001
Test score .265 <0.001Secundary school type
.252 <0.001
…
YES
Initial test
10
• Division into groups (n = 367)
exam < 10 exam >= 10 % successtest <= 40 32 4 11.1%test <= 50 44 22 33.3%test <= 60 45 67 59.8%test <= 70 16 80 83.3%test > 70 5 52 91.2%
I: 102
II: 112
III: 153
LMS: exercises
11
• 20 grammatical topics, free order• common “fill in the blanks” exercises
• 1 exercise = 2 attemptso first attempt => correction + fact sheeto second attempt => focus on mistakes
correction + fact sheet
• goal: 7/10 for 17/20 topics > reward > final mark
Exercises
12
• feedback_1: correction + fact sheet (after both attempts)
Exercises
13
• feedback_2: score compared to score of their peers
Exercises
14
• feedback_3: dashboard
Exercises
15
• Do weaker students start their exercises later?
start ac. year
wake-up call
I: 42.4% III: 65%
YES
(II: 61.9%)
Exercises
16
• Do exercises help students get better marks?
>= 17/20 ex. >= 7
< 17/20 ex.>= 7
pass 198 27fail 102 40
YES, probably…
Chi2 = 15.251; df = 1; p = 0.0001
Exercises
17
• Are the exercises too difficult?
% first attempt(s)
% last attempt
averageattempts/topics
I 35.2% 75.3% 2.6
II 38.8% 76.4% 2.7
III 43.9% 76.2% 2.1
NO
Exercises
18
• Are some exercises more difficult than others? YES
Exam mark
19
• exam mark * self-evaluation
Research question
20
How toidentify& support at-risk students (French course)
byproviding relevant feedback at the right moment?
Feedback BEFORE analyses
21
• Mixed product and process feedback
for every ex.: correction + fact sheet => second attempt
for every ex.: score comparison (peers)
wake-up call 1 (mid-Nov.): mail with variable message
wake-up call 2 (mid-March): mail with variable message
dashboard
Feedback AFTER analyses
22
• Process feedback
for every ex.: correction + fact sheet => second attempt
for every ex.: score comparison (peers)
wake-up call 1 (mid-Nov.): email with variable message
wake-up call 2 (mid-March): email with variable message
dashboard
asap: personalized email
Feedback AFTER analyses
23
• email to students witho test score <= 50 (if test taken seriously)
o self-evaluation < 5
• content:o start asap with the exerciseso making exercises increases your chances of successo start with exercises on [items]o be aware that you will need at least 3 attempts => 7/10o later on: do not forget to focus on [items]
Feedback AFTER analyses
24
• email to students witho test score <= 50 (if test taken seriously)
o self-evaluation > 6• content:
o you may have overestimated yourselfo start asap with the exerciseso making exercises increases your chances of successo start with exercises on [items]o be aware that you will need at least 3 attempts => 7/10o later on: do not forget to focus on [items]
Conclusion
25
descriptive learning analytics
predictive learning analytics
better identification of at-risk studentsdispositional learning analytics
process feedback
Future work
26
• More motivated group division?• Findings supported by evidence from other groups?• Impact of early (process) feedback?
• Analysis of answers?
Impact of early (feedback) process?
27
2015-2016
2016-2017
I: 42.4% III: 65%(II: 61.9%)
I: 60,3% III: 62,6%(II: 61.8%)
Impact of early (feedback) process?
28
YESBUT… (data from the day before the wake-up call)
2015-16 numberof
2016-17
20 5.6% 0 1.4% 5216 60.2% < 5 74.2% 27059 16.4% 5-9 11.8% 4342 11.7% 10-16 8.2% 3022 6.1% >= 17 4.4% 16
need for constant monitoring…
