https://mfeldstein.com/purdue-course-signals-data-issue-explainer/

Learning dashboards for first‐year students

the (non)sense of chances of success and predictive models

Tinne De LaetTinne.DeLaet@kuleuven.be

@TinneDeLaet

“Learning analytics is about collecting traces that learners leave behind and using those traces to improve learning.”

- Erik Duval

Learning Analytics and Educational Data Mining, Erik Duval’s Weblog, 30 January 2012, https://erikduval.wordpress.com/2012/01/30/learning-analytics-and-educational-data-mining/ 3

Learning Analytics?

Learning Dashboards?

4Dashboard Confusion, Stephen Few, Intelligent Enterprise, March 20, 2004

“A dashboard is a visual display of the most important information needed to

achieve one or more objectives; consolidated and arranged on a single

screen so the information can be monitored at a glance.”- Stephen Few

Successful Transition from secondary to higher Education using Learning Analytics

enhance a successful transition from secondary to higher education by means of

learning analytics design and build analytics dashboards,

dashboards that go beyond identifying at-risk students, allowing actionable feedback for all

students on a large scale.

Achieving Benefits from Learning Analytics

research strategies and practices for using learning analytics to support students during

their first year at university developing the technological aspects of

learning analytics, focuses on how learning analytics can be used

to support students.

5

www.stela-project.eu

@STELA_project2015‐1‐UK01‐KA203‐013767

www.ableproject.eu

@ABLE_project_eu

562167‐EPP‐1‐2015‐1‐BE‐EPPKA3‐PI‐FORWARD

STELA ♥ ABLE

6

actionable feedbackstudent-centered

program level

inclusive

first-year experience

institution-wide

Learning Analytics

actual implementation

[!] Feedback must be “actionable”.

7

Warning!

Male students have

10% less probability to

be successful.

You are male.

Warning!

Your online activity is

lagging behind.

action?

?

action?

?

8

awareness

(self-)reflection

sensemaking

impact

data

questions

answers

behavior changenew meaning

Verbert K, Duval E, Klerkx J; Govaerts S, Santos JL (2013) Learning analytics dashboard applications. American Behavioural Scientist, 10 pages. Published online February 2013.

[!] Feedback must be “actionable”.

learning dashboards @KU Leuven

interaction

self-reflection

LISSA

STUDENTADVISOR STUDENT LASSI –

learning skills

REX - scoresPOS – futurestudents

[!] Start with the available data.

Lots of data may eventually become available in the future …

…. already start with what is available

10

(*)

(*) Zarraonandia, T., Aedo, I., Díaz, P., & Montero, A. (2013). An augmented lecture feedback system to support learner and teacher communication.British Journal of Educational Technology, 44(4), 616-628.

Case studydashboard interaction student – study advisor

Study advisor – student conversations

12

Should I consideranother program?

Can I still finish the bachelor in 3 years?

How should I composemy program for next year?

What is the personalsituation?

How can I help?

What is the best next step?

[!] Use all available expertise. 

13

visualization experts

practitioners / end-users

researchers LA

researchers first-year study success

Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue.In IEEE Transactions on Learning Technology (http://ieeexplore.ieee.org/document/7959628/).

LISSA dashboardhttps://able.cs.kuleuven.be/demo‐september/2016/2

[!] Wording matters.

15

73% chance of success

73% of students of earlier cohorts with the same

study efficiency obtained the bachelor degree

http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/

LISSA dashboard

16

Three examination periods

observations, interviews, questionnaires

pilot with two engineering programs

Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology

LISSA: evaluation – observations

17

15 observations

insights(-) factual(+) interpretative (!) reflective

Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology

Evaluation – interviews

“When students see the numbers, they are surprised, but now they believe me.

Before, I used my gut feeling, now I feelmore certain of what I say as well”.

“It’s like a main thread guiding the

conversation.”

“I can talk about what to do with the results, instead of each time looking for the data and

puzzling it together.”

“Students don’t know where to look during the conversation, and avoid eye contact.

The dashboard provides them a point of focus”.

“A student changed her study method in June and could now see it paid off.”

LISSA supports a personal dialogue.

the level of usage depends on the experience and style of the study advisors

fact-based evidence at the side narrative thread

key moments and student path help to reconstruct personal track

“I can focus on the student’s personal

path, rather than on the facts.”

“Now, I can blamethe dashboard and

focus on collaboratively looking

for the next step to take.”

18

LISSA: status

19

26 programs >4500 students114 student advisors

training of study advisors

http://blog.associatie.kuleuven.be/tinnedelaet/lissa-learning-dashboard-supporting-student-advisers-in-traditional-higher-education/

Millecamp M., Gutiérrez F., Charleer S., Verbert K., De Laet T.# (2018). A qualitative evaluation of a learning dashboard to support advisor-student dialogues. Proceedings of the 8th International Learning Analytics & Knowledge Conference. LAK. Sydney, 5-9 March 2018 (pp. 1-5) ACM.

dashboards for three examination periods

LISSA: evaluation – student questionnaires

20

26 programs @KU Leuven291 student questionnaires

first examination period

“Confronting, but useful”

“I want to use thisdashboard at home.”

“Also show the sub‐gradesfor labs, … ”

“How can I know the data is trustworth?”

“Can’t these visualizations besend to students?” “Crisp and clear.”

21

0

0

1

1

1

1

4

2

1

4

4

3

29

21

36

37

49

42

176

112

156

132

141

169

80

155

93

116

92

72

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

1.     The dashboard is clarifying and surveyable.

2.     The shown information regarding my studysituation is correct.

3.     The shown position with respect to my fellowstudents (histograms per exam and global…

4.     A conversation with my student advisors helpedme to gain insight in my study trajectory.

5.     The visualisation is of added value to theconversation with the student advisor.

6.     The shown information provide me insight inmy current situation.

Student questionnaire January 2018 (N=291)

Strongly Disagree Disagree Neither Agree or Disagree Agree Strongly Agree

[!] Do not oversimplify. Show uncertainty.

22

• reality is complex

• measurement is limited

• individual circumstances

• need for nuance

• trigger reflection

http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/

[!] Be careful with predictive algorithms.

23http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/

• reality is complex

• measurement is limited

• individual circumstances

• need for nuance

• trigger reflection

Case studystudent-facing dashboards

[!] Start with the available data.

25

data already available?

administrative (examples)

student records course grades

systems (examples)

LMS access logs advisor meetings

)

Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Small data as a conversation starter for learning analytics: exam results dashboard for first-year students in higher education. Journal of Research in Innovative Teaching & Learning, , 1-14.

[!] Think beyond the obvious data.

26

• Don’t think too traditional.

• Many institutions are collecting survey data for educational research.

[!] Not all data is usable. 

27

example data from a traditional course with “VLE as a file system”

test scores

activity/week (#days)

weeks of the year

[!] Not all data is usable. 

28

example data from a course with flipped classroom & blended learning

exam scores

activity (# of modules used)

Not a single student using less than 10

modules passed the course.

Most of the successful students used 15 modules or more.

[!] Keep Learning Analytics in mind when designing learning activities. 

29

Learning Analytics Learning Design

INFORM

ENABLE

If LA indeed contributes to improved learning design…

… don’t make it an afterthought

30

Does my concentrationmatter?

How is my time management?

I feel uncertain.Is this normal?

How can I improve my concentration?

data already available?

administrative (examples)

student records course grades

[!] Think beyond the obvious data.

31

systems (examples)

LMS access logs advisor meetings

surveys (examples)

quality insurance LASSI

~ 30 LASSI questions(shortened version)

“Learning Skills”

Example: When preparing for an exam, I create questions that I think might be included.

Example: I find it difficult to maintain my concentration while doing my coursework.

Example: I find it hard to stick to a study schedule.

raw scores(selected 5 out of 10)

CONCENTRATION

MOTIVATION

FAILURE ANXIETY

TEST STRATEGY

TIME MANAGEMENT

norm scores(in Flemish HE context)

Example: STRONG

Example: AVERAGE

Example: LOW

Example: VERY STRONG

Example: VERY WEAK

32

Meta cognitive abilities

Pinxten, M., Van Soom, C., Peeters, C., De Laet, T., Langie, G., At-risk at the gate: prediction of study success of first-year science and engineering students in an open-admission university in Flanders—any incremental validity of study strategies? Eur J Psychol Educ (2017). readySTEMgo Erasmus+ project https://iiw.kuleuven.be/english/readystemgo

Dashboard learning skills

33

students complete LASSI questionnaire

students received personalized emailwith invitation for dashboard

4367 students in 26 programsin 9 faculties @KU Leuven

demo: https://learninganalytics.set.kuleuven.be/lassi-1718/ (KU Leuven login)

2 programs @TU Delft

Feedback model

1. What is this about?

2. How am I doing?

3. How does this relates to others?

4. Why is this relevant?

5. What can I do about it?

34

35

3. How does this relates to others?

2. How am I doing?

1. What is this about?

@studyProgram@

@yourScore@

4. Why is this relevant?

5. What can I do about it?

36

37

5. What can I do about it?

Response

38

3868 students (89%) used dashboard

Student feedback?

39http://blog.associatie.kuleuven.be/tinnedelaet/learning-dashboard-for-actionable-feedback-on-learning-and-studying-skills/

How CLEAR is this info?

stars stars

Students that click through

Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness.In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham.

40

better learning skills

More intense users

Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness.In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham.

41

worse learning skills

[!] Give students “the key”.

42

• Student has the key to own data.

• Student takes initiative to share/discuss own data.

• GDPR as opportunity!

Dashboard positioning test

43https://feedback.ijkingstoets.be/ijkingstoets-10-ir/index.html (10ir0demo)

[!] Acceptance precedes impact.

44

• Involve stakeholders from the start and value their input!

COmmunicationCOoperation

• Demonstrate usefulness.

• Take care of ethics and privacy.

• Best scenario: students & study advisors as ambassadors

CO

CO

Impact?

survey before intervention 2nd year students 2016-2017 experiences first-year feedback 41 vragen, 5-point Likert scale pen & paper

dashboards LISSA LASSI (learning skills) 3 x REX (grades)

Survey after intervention 2nd year students 2017-2018

Impact?

During the first year I received sufficient information regarding my academic achievements.

46

Engineering Science (p<0.001)

Impact?

The information I received helped to position myself with respect to my peers.

47

Engineering Science (p<0.001)

Impact?

48

The information I received made me reflect.

The information I received made me adapt my behaviour.

[!] Context matters!

• available data

• national and institutional regulations and culture

• educational vision

• educational system, size of population ..

• …

Don’t just copy existing LA solutions!

49

Summary

case studies 11 findings/recommendations

[!] Use all available expertise.

[!] Start with the available data.

[!] Look beyond the obvious data.

[!] Not all data is usable.

[!] Wording matters.

[!] Don’t oversimplify. Show uncertainty.

[!] Beware of predictive algorithms.

[!] Keep Learning Analytics in mind when designing

learning activities.

[!] Give students “the key” to their data.

[!] Acceptance precedes impact.

[!] Context matters!

humble approach

small data

involvement of stakeholders, especially practitioners

actionable feedback

scalability

traditional university settings

Is this Learning Analytics?

Future?

51

Continue and extend dashboards @KU Leuven?

Transfer to other universities?

extension?

Project team @

52

Sven CharleerAugmentHCI, Computer Science department

PhD researcher ABLE

Katrien VerbertAugmentHCI, Computer Science department

Copromotor of STELA & ABLE

Carolien Van SoomLeuven Engineering and Science Education CenterHead of Tutorial Services of ScienceCopromotor of STELA & ABLE

Greet Langie Leuven Engineering and Science Education Center

Vicedean (education) faculty of Engineering TechnologyCopromotor of STELA & ABLE

Tinne De LaetLeuven Engineering and Science Education CenterHead of Tutorial Services of Engineering ScienceCoordinator of STELAKU Leuven coordinator of ABLE

Francisco GutiérrezAugmentHCI, Computer Science departmentPhD researcher ABLE

Tom BroosLeuven Engineering and Science Education CenterAugmentHCI, Computer Science departmentPhD researcher STELA

Martijn MillecampAugmentHCI, Computer Science department

PhD researcher ABLE

Special thanks to study advisors for their cooperation, advice, feedback, and support!Jasper, Bart, Riet, Hilde, An, Katrien, …

♥

53

2

3

1

7

2

1

10

44

23

36

3

11

64

97

81

74

36

29

150

115

126

110

119

91

61

30

56

59

128

157

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

7.     The dashboard makes me more aware on my currentstudy situation.

8.     The dashboard makes me forecast the differentpossibilities in my future study trajectory.

9.     The dashboard helps me to reflect on my past andcurrent study behaviour or study trajectory.

10.   The dashboards stimulates me to adapt my approachin my studies for the future (study behaviour or study…

11.   If I will have a new conversation after one of the nextexamination periods, I hope that the visualisation will be…

12.  I would like to consult the information on my own.

Student questionnaire January 2018 (N=291)

Strongly Disagree Disagree Neither Agree or Disagree Agree Strongly Agree

How to determine thresholds for different

groups?

LISSA dashboard

54

upper and lower group: clear message

middle group as small as possible

Do not overfit! (nuance)