Learning dashboards for actionable feedback · learning analytics design and build analytics...

51
Learning dashboards for actionable feedback the (non)sense of chances of success and predictive models Tinne De Laet [email protected] @TinneDeLaet

Transcript of Learning dashboards for actionable feedback · learning analytics design and build analytics...

Page 1: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Learning dashboards for actionable feedback

the (non)sense of chances of success and predictive models

Tinne De Laet

[email protected]@TinneDeLaet

Page 2: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

“Learning analytics is about

collecting traces that learners

leave behind and using those

traces to improve learning.”

- Erik Duval

Learning Analytics and Educational Data Mining, Erik Duval’s Weblog, 30 January 2012, https://erikduval.wordpress.com/2012/01/30/learning-analytics-and-educational-data-mining/ 2

Learning Analytics?

Page 3: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Learning Dashboards?

3Dashboard Confusion, Stephen Few, Intelligent Enterprise, March 20, 2004

“A dashboard is a visual display of the

most important information needed to

achieve one or more objectives;

consolidated and arranged on a single

screen so the information can be monitored

at a glance.”

- Stephen Few

Page 4: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Successful Transition from secondary to higher

Education using Learning Analytics

enhance a successful transition from

secondary to higher education by means of

learning analytics

design and build analytics dashboards,

dashboards that go beyond identifying at-risk

students, allowing actionable feedback for all

students on a large scale.

Achieving Benefits from Learning Analytics

research strategies and practices for using

learning analytics to support students during

their first year at university

developing the technological aspects of

learning analytics,

focuses on how learning analytics can be used

to support students.

4

www.stela-project.eu

@STELA_project

2015-1-UK01-KA203-013767

www.ableproject.eu

@ABLE_project_eu

562167-EPP-1-2015-1-BE-EPPKA3-PI-FORWARD

Page 5: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

STELA ♥ ABLE

5

actionable feedback

student-centeredprogram level

inclusive

first-year experience

institution-wide

Learning Analytics

actual implementation

Page 6: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

[!] Feedback must be “actionable”.

6

Warning!

Male students have

10% less probability to

be successful.

You are male.

Warning!

Your online activity is

lagging behind.

action?

?

action?

?

Page 7: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

7

awareness

(self-)reflection

sensemaking

impact

data

questions

answers

behavior change

new meaning

Verbert K, Duval E, Klerkx J; Govaerts S, Santos JL (2013) Learning analytics dashboard applications. American Behavioural Scientist, 10 pages. Published online February 2013.

[!] Feedback must be “actionable”.

Page 8: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

8

interaction

self-reflection

LISSA REX - grades

STUDYADVISER

STUDENT

Erasmus+ project ABLE

LASSI – learning skills

The dashboards

Page 9: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

[!] Start with the available data.

Lots of data may eventually become

available in the future …

…. already start with what is available

9

(*)

(*) Zarraonandia, T., Aedo, I., Díaz, P., & Montero, A. (2013). An augmented lecture feedback system to support learner and teacher communication.

British Journal of Educational Technology, 44(4), 616-628.

Page 10: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Case studydashboard interaction student – study advisor

Page 11: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Study advisor – student conversations

11

Should I consider

another program?

Can I still finish the

bachelor in 3 years?

How should I compose

my program for next

year?

What is the personal

situation?

How can I help?

What is the best

next step?

Page 12: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

[!] Use all available expertise.

12

visualization experts

practitioners / end-users

researchers LA

researchers first-year study success

Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue.

In IEEE Transactions on Learning Technology (http://ieeexplore.ieee.org/document/7959628/).

Page 13: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

13

LISSA dashboard

Page 14: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

[!] Wording matters.

14

73% chance of success

73% of students of earlier cohorts with the same

study efficiency obtained the bachelor degree

http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/

Page 15: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

LISSA dashboard

15

Three examination periods

observations, interviews, questionnaires

pilot with two engineering programs

Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology

Page 16: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

LISSA: evaluation – observations

16

15 observations

insights(-) factual(+) interpretative (!) reflective

Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology

Page 17: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Evaluation – interviews

“When students see the numbers, they are

surprised, but now they believe me.

Before, I used my gut feeling, now I feel

more certain of what I say as well”.

“It’s like a main thread

guiding the

conversation.”

“I can talk about what to do with the results,

instead of each time looking for the data and

puzzling it together.”

“Students don’t know where to look during the

conversation, and avoid eye contact.

The dashboard provides them a point of focus”.

“A student changed her

study method in June and

could now see it paid off.”

LISSA supports a personal dialogue.

the level of usage depends on the experience and style of the study advisors

fact-based evidence at the side narrative thread

key moments and student path help to reconstruct personal track

“I can focus on the

student’s personal

path, rather than on

the facts.”

“Now, I can blame

the dashboard and

focus on

collaboratively looking

for the next step to

take.”

17

Page 18: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

LISSA: status

18

26 programs >4500 students114 student advisors

training of study advisors

http://blog.associatie.kuleuven.be/tinnedelaet/lissa-learning-dashboard-supporting-student-advisers-in-traditional-higher-education/

Millecamp M., Gutiérrez F., Charleer S., Verbert K., De Laet T.# (2018). A qualitative evaluation of a learning dashboard to support advisor-student

dialogues. Proceedings of the 8th International Learning Analytics & Knowledge Conference. LAK. Sydney, 5-9 March 2018 (pp. 1-5) ACM.

dashboards for three examination periods

Page 19: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

LISSA: evaluation – student questionnaires

19

26 programs @KU Leuven291 student questionnaires

first examination period

“Confronting, but useful”

“I want to use thisdashboard at home.”

“Also show the sub-gradesfor labs, … ”

“How can I know the data is trustworth?”

“Can’t these visualizations besend to students?” “Crisp and clear.”

Page 20: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

20

0

0

1

1

1

1

4

2

1

4

4

3

29

21

36

37

49

42

176

112

156

132

141

169

80

155

93

116

92

72

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

1.     The dashboard is clarifying and surveyable.

2.     The shown information regarding my studysituation is correct.

3.     The shown position with respect to my fellowstudents (histograms per exam and global…

4.     A conversation with my student advisors helpedme to gain insight in my study trajectory.

5.     The visualisation is of added value to theconversation with the student advisor.

6.     The shown information provide me insight inmy current situation.

Student questionnaire January 2018 (N=291)

Strongly Disagree Disagree Neither Agree or Disagree Agree Strongly Agree

Page 21: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

[!] Do not oversimplify. Show uncertainty.

21

• reality is complex

• measurement is limited

• individual circumstances

• need for nuance

• trigger reflection

http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/

Page 22: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

[!] Be careful with predictive algorithms.

22http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/

• reality is complex

• measurement is limited

• individual circumstances

• need for nuance

• trigger reflection

Page 23: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Case studystudent-facing dashboards

Page 24: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

[!] Start with the available data.

24

data already available?

administrative (examples)

student records course grades

systems (examples)

LMS access logs advisor meetings

)

Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Small data as a conversation starter for learning analytics: exam results dashboard for first-year students in higher

education. Journal of Research in Innovative Teaching & Learning, , 1-14.

Page 25: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

[!] Think beyond the obvious data.

25

• Don’t think too traditional.

• Many institutions are collecting survey

data for educational research.

Page 26: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

[!] Not all data is usable.

26

example data from a traditional course with “VLE as a file system”

test scores

activity/week (#days)

weeks of the year

Page 27: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

[!] Not all data is usable.

27

example data from a course with flipped classroom & blended learning

exam scores

activity (# of modules used)

Not a single student using less than 10

modules passed the course.

Most of the successful students used 15 modules or more.

Page 28: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

[!] Keep Learning Analytics in mind when designing learning activities.

28

Learning Analytics

Learning DesignINFORM

ENABLE

If LA indeed contributes to improved

learning design…

… don’t make it an afterthought

Page 29: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

29

Does my concentration

matter?

How is my time

management?

I feel uncertain.

Is this normal?

How can I improve

my concentration?

Page 30: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

data already available?

administrative (examples)

student records course grades

[!] Think beyond the obvious data.

30

systems (examples)

LMS access logs advisor meetings

surveys (examples)

quality insurance LASSI

Page 31: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

~ 30 LASSI questions(shortened version)

“Learning Skills”

Example: When preparing for an exam, I create questions that I think might be included.

Example: I find it difficult to maintain my concentration while doing my coursework.

Example: I find it hard to stick to a study schedule.

raw scores(selected 5 out of 10)

CONCENTRATION

MOTIVATION

FAILURE ANXIETY

TEST STRATEGY

TIME MANAGEMENT

norm scores(in Flemish HE context)

Example: STRONG

Example: AVERAGE

Example: LOW

Example: VERY STRONG

Example: VERY WEAK

31

Meta cognitive abilities

Pinxten, M., Van Soom, C., Peeters, C., De Laet, T., Langie, G., At-risk at the gate: prediction of study success of first-year science and engineering students in an

open-admission university in Flanders—any incremental validity of study strategies? Eur J Psychol Educ (2017).

readySTEMgo Erasmus+ project https://iiw.kuleuven.be/english/readystemgo

Page 32: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Dashboard learning skills

32

students complete LASSI questionnaire

students received personalized emailwith invitation for dashboard

4367 students in 26 programsin 9 faculties @KU Leuven

demo: https://learninganalytics.set.kuleuven.be/lassi-1718/ (KU Leuven login)

2 programs @TU Delft

Page 33: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Feedback model

1. What is this about?

2. How am I doing?

3. How does this relates to others?

4. Why is this relevant?

5. What can I do about it?

33

Page 34: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

34

3. How does this relates to others?

2. How am I doing?

1. What is this about?

@studyProgram@

@yourScore@

Page 35: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

4. Why is this relevant?

5. What can I do about it?

35

Page 36: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

36

5. What can I do about it?

Page 37: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Response

37

3868 (89%) used dashboard

Page 38: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Student feedback?

38http://blog.associatie.kuleuven.be/tinnedelaet/learning-dashboard-for-actionable-feedback-on-learning-and-studying-skills/

How CLEAR is this info?

stars stars

Page 39: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Students that click through

Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness.In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham.

39

better learning skills

Page 40: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

More intense users

Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness.In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham.

40

worse learning skills

Page 41: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

[!] Give students “the key”.

41

• Student has the key to own

data.

• Student takes initiative to

share/discuss own data.

• GDPR as opportunity!

Page 42: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Dashboard positioning test

42https://feedback.ijkingstoets.be/ijkingstoets-10-ir/index.html (10ir0demo)

Page 43: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

[!] Acceptance precedes impact.

43

• Involve stakeholders from the start and

value their input!

COmmunication

COoperation

• Demonstrate usefulness.

• Take care of ethics and privacy.

• Best scenario:

students & study advisors as ambassadors

CO

CO

Page 44: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Impact?

survey before intervention

2nd year students 2016-2017

experiences first-year feedback

41 vragen, 5-point Likert scale

pen & paper

dashboards

LISSA

LASSI (learning skills)

3 x REX (grades)

Survey after intervention

2nd year students 2017-2018

Page 45: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Impact?

During the first year I received sufficient information regarding my academic achievements.

45

Engineering Science (p<0.001)

Page 46: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Impact?

The information I received helped to position myself with respect to my peers.

46

Engineering Science (p<0.001)

Page 47: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Impact?

47

The information I received made me reflect.

The information I received made me adapt my behaviour.

Page 48: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

[!] Context matters!

• available data

• national and institutional regulations

and culture

• educational vision

• educational system, size of population ..

• …

Don’t just copy existing LA solutions!

48

Page 49: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Summary

case studies 11 findings/recommendations

[!] Use all available expertise.

[!] Start with the available data.

[!] Look beyond the obvious data.

[!] Not all data is usable.

[!] Wording matters.

[!] Don’t oversimplify. Show uncertainty.

[!] Beware of predictive algorithms.

[!] Keep Learning Analytics in mind when designing

learning activities.

[!] Give students “the key” to their data.

[!] Acceptance precedes impact.

[!] Context matters!

humble approach

small data

involvement of stakeholders, especially practitioners

actionable feedback

scalability

traditional university settings

Is this Learning Analytics?

Page 50: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Future?

50

Continue and extend dashboards @KU Leuven?

Transfer to other universities?

extension?

Page 51: Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Project team @

51

Sven CharleerAugmentHCI, Computer Science department

PhD researcher ABLE

Katrien VerbertAugmentHCI, Computer Science department

Copromotor of STELA & ABLE

Carolien Van SoomLeuven Engineering and Science Education CenterHead of Tutorial Services of ScienceCopromotor of STELA & ABLE

Greet Langie Leuven Engineering and Science Education Center

Vicedean (education) faculty of Engineering TechnologyCopromotor of STELA & ABLE

Tinne De LaetLeuven Engineering and Science Education CenterHead of Tutorial Services of Engineering ScienceCoordinator of STELAKU Leuven coordinator of ABLE

Francisco GutiérrezAugmentHCI, Computer Science departmentPhD researcher ABLE

Tom BroosLeuven Engineering and Science Education CenterAugmentHCI, Computer Science departmentPhD researcher STELA

Martijn MillecampAugmentHCI, Computer Science department

PhD researcher ABLE

Special thanks to study advisors for their cooperation, advice, feedback, and support!Jasper, Bart, Riet, Hilde, An, Katrien, …