THE STUDENT ACTIVITY METER FOR AWARENESS AND

SELF-REFLECTIONSten Govaerts, Katrien Verbert, Erik Duval, Abelardo Pardo.

1

SigCHI 2012, Long Case Study.

Tuesday 20 March 12

2

http://bit.ly/stenpub

Tuesday 20 March 12

OVERVIEW

• Problem statement

• Objectives

• Design & Implementation

• Evaluations

• Conclusion

3

Tuesday 20 March 12

PROBLEM

4

http://www.ypfp.org/classroom

Tuesday 20 March 12

PROBLEM

4

http://www.ypfp.org/classroom

Tuesday 20 March 12

OBJECTIVES

• self-monitoring for learners

• awareness for teachers

• time tracking

• learning resource use and recommendations

• part of Learning Analytics research [ACM LAK conf., Siemens 2011, Duval 2011]

5

http://free-extras.com/images/obama_hope_progress-2848.htm

Tuesday 20 March 12

STUDENT ACTIVITY METER (SAM): DEMO.

6

http://dilbert.com/strips/comic/2000-02-24/

Tuesday 20 March 12

STUDENT ACTIVITY METER

7

Tuesday 20 March 12

STUDENT ACTIVITY METER

8

Tuesday 20 March 12

STUDENT ACTIVITY METER

9

Tuesday 20 March 12

TASK

• How would you evaluate SAM?

• how/where do you get data from?

• which criteria could you evaluate?

• which methods use to evaluate those criteria?

• iterative? which iterations?

10

http://callofduty.wikia.com/wiki/Task_Force_141

Tuesday 20 March 12

METHODOLOGY.

• design-based research [Obrenović 2011]: give students & teachers a tool and evaluate

• iterative process of design & evaluation

• paper mockups

• usability studies

• surveys on usefulness

11http://funnytogo.com/pictures/evolution/timeline.htm

Tuesday 20 March 12

12

iter. evaluation goal methodology demo-graphics data

I.usability, satisfaction,

preliminary usefulness

task-based interview with think-aloud, SUS

& MSDT

12 CS students at KULeuven

timetracking via Twitter

II.assessing teacher

needs, use & usefulness

online survey with Likert items, yes/no and open questions

19 teachers & TA’s with

CGIAR coursesMoodle logs

III.

assessing teacher needs, expert

feedback, use & usefulness

online survey with Likert items, yes/no and open questions

12 participants from LAK MOOC

Moodle logs

IV. use, usefulness & satisfaction

structured face-to-face interview with tasks &

open questions

11 teachers & TA’s at Carlos III

Uni. Madrid

VM with extensive

tool tracking

Tuesday 20 March 12

ITERATION ONE

• usability and user satisfaction evaluation

• 12 CS students, using a -based time tracker

• 2 evaluation sessions:

• task based interview with think aloud (after 1 week of tracking)

• user satisfaction (SUS & MSDT) (after 1 month)

Tuesday 20 March 12

• in general, people understand the visualizations well!

• some issues were uncovered...

LEARNABILITY, ERRORS & EFFICIENCY

Tuesday 20 March 12

• in general, people understand the visualizations well!

• some issues were uncovered...

LEARNABILITY, ERRORS & EFFICIENCY

Tuesday 20 March 12

• in general, people understand the visualizations well!

• some issues were uncovered...

LEARNABILITY, ERRORS & EFFICIENCY

Tuesday 20 March 12

• in general, people understand the visualizations well!

• some issues were uncovered...

LEARNABILITY, ERRORS & EFFICIENCY

Tuesday 20 March 12

• in general, people understand the visualizations well!

• some issues were uncovered...

LEARNABILITY, ERRORS & EFFICIENCY

Tuesday 20 March 12

[Bangor et al. 2008]

USER SATISFACTION

• average SUS score: 73%

Tuesday 20 March 12

[Bangor et al. 2008]

USER SATISFACTION

• average SUS score: 73%

Tuesday 20 March 12

[Bangor et al. 2008]

USER SATISFACTION

• average SUS score: 73%

Tuesday 20 March 12

[Bangor et al. 2008]

USER SATISFACTION

• average SUS score: 73%

Tuesday 20 March 12

demographics

evaluation goal

design changes negative positive

I. 12 CS students

usability, satisfaction, preliminary usefulness

1st iteration small usability issues

•↑learnability•↓errors•good satisfaction•usefulness positive

Tuesday 20 March 12

ITERATION TWO

• 20 persons: 3 CGIAR, 2 Law, 8 CS teachers & 7 CS TA’s.

• An online survey about usefulness, teacher issues and how the tool can resolve these.

• on average: 40 mins are spent using SAM.

17

Tuesday 20 March 12

CGIAR CASE STUDY

Provide feedback to the students

Being aware of what students are doing

Knowing about collaboration and communication

Knowing which documents are used and how much

Knowing how and when online tools have been used

Finding the students who are not doing well

Finding the best students

Knowing how much time students spent

Knowing if external learning resources are used

issue for teacher addressed

✔

✗

?✔

✔

✔?

✔

??

✔

✔✔??✔?

✔

18

1 2 3 4 5

Issues

Provide feedback to the studentsBeing aware of what students are doingKnowing about collaboration and communication *Knowing which documents are used and how muchKnowing how and when online tools have been usedFinding the students who are not doing well

Finding the best students

Knowing how much time students spentKnowing if external learning resources are used

Addressed by SAM

1 2 3 4 5

Tuesday 20 March 12

CGIAR CASE STUDY

Provide feedback to the students

Being aware of what students are doing

Knowing about collaboration and communication

Knowing which documents are used and how much

Knowing how and when online tools have been used

Finding the students who are not doing well

Finding the best students

Knowing how much time students spent

Knowing if external learning resources are used

issue for teacher addressed

✔

✗

?✔

✔

✔?

✔

??

✔

✔✔??✔?

✔

18

Tuesday 20 March 12

CGIAR CASE STUDY

Provide feedback to the students

Being aware of what students are doing

Knowing about collaboration and communication

Knowing which documents are used and how much

Knowing how and when online tools have been used

Finding the students who are not doing well

Finding the best students

Knowing how much time students spent

Knowing if external learning resources are used

issue for teacher addressed

✔

✗

?✔

✔

✔?

✔

??

✔

✔✔??✔?

✔

18

Tuesday 20 March 12

CGIAR CASE STUDY

19

Tuesday 20 March 12

CGIAR CASE STUDY

good indicator for effortunderstand the workload

use for course design optimization

obtain course overview

increase awareness

detect outliers

progress evolutioncompare students

want better 1 to 1 comparison tool

more details on student

want to search for students

more metricswanted

19

Tuesday 20 March 12

CGIAR CASE STUDY

good indicator for effortunderstand the workload

use for course design optimization

obtain course overview

increase awareness

detect outliers

progress evolutioncompare students

want better 1 to 1 comparison tool

more details on student

want to search for students

more metricswanted

19

Tuesday 20 March 12

demographics

evaluation goal

design changes negative positive

I.12 CS

students

usability, satisfaction, preliminary usefulness

1st iteration small usability issues

•↑learnability•↓errors•good satisfaction•usefulness positive

II.19

teachers & TA’s

assessing teacher

needs, use & usefulness

help functionresource

recomm. not useful

•provides awareness•all vis. useful•many uses•90% wants it

Tuesday 20 March 12

ITERATION THREE

• open course on learning and knowledge analytics, http://bit.ly/dWYVbX

• 12 visual analytics enthousiasts + experts (who also teach)

• almost identical survey to CGIAR case.

21

Tuesday 20 March 12

ITERATION THREE

• open course on learning and knowledge analytics, http://bit.ly/dWYVbX

• 12 visual analytics enthousiasts + experts (who also teach)

• almost identical survey to CGIAR case.

21

Tuesday 20 March 12

LAK CASE STUDY

Provide feedback to the students

Being aware of what students are doing

Knowing about collaboration and communication

Knowing which documents are used and how much

Knowing how and when online tools have been used

Finding the students who are not doing well

Finding the best students

Knowing how much time students spent

Knowing if external learning resources are used

issue for teacher addressed

✔

✗✔

✔

✔?

?

?

✔?

✔?

✔?

✗

✗?

?

22

Tuesday 20 March 12

LAK CASE STUDY

Provide feedback to the students

Being aware of what students are doing

Knowing about collaboration and communication

Knowing which documents are used and how much

Knowing how and when online tools have been used

Finding the students who are not doing well

Finding the best students

Knowing how much time students spent

Knowing if external learning resources are used

issue for teacher addressed

✔

✗✔

✔

✔?

?

?

✔?

✔?

✔?

✗

✗?

?

22

Tuesday 20 March 12

LAK CASE STUDY

Provide feedback to the students

Being aware of what students are doing

Knowing about collaboration and communication

Knowing which documents are used and how much

Knowing how and when online tools have been used

Finding the students who are not doing well

Finding the best students

Knowing how much time students spent

Knowing if external learning resources are used

issue for teacher addressed

✔

✗✔

✔

✔?

?

?

✔?

✔?

✔?

✗

✗?

?

22

Tuesday 20 March 12

LAK CASE STUDY

23

Tuesday 20 March 12

LAK CASE STUDY

24http://maranash.blogspot.com/2011/01/conference-shout-out.html

Tuesday 20 March 12

LAK CASE STUDY

chronological course dwell time

verify the status of the classroom activity

self-reflection to measure progress and increase

motivation

find students experiencing problems and low

engagement

more metrics

more data

24http://maranash.blogspot.com/2011/01/conference-shout-out.html

Tuesday 20 March 12

LAK CASE STUDY

chronological course dwell time

verify the status of the classroom activity

self-reflection to measure progress and increase

motivation

find students experiencing problems and low

engagement

more metrics

more data

24http://maranash.blogspot.com/2011/01/conference-shout-out.html

Tuesday 20 March 12

IDEA’S FROM EXPERTS

detailed information per student

the used resource types

detailed information of 2 students

detailed usage stats of resources

stats or vis. on content creation

25

Tuesday 20 March 12

IDEA’S FROM EXPERTS

1

2

3

4

5 detailed information per student

the used resource types

detailed information of 2 students

detailed usage stats of resources

stats or vis. on content creation

25

Tuesday 20 March 12

IDEA’S FROM EXPERTS

1

2

3

4

5 detailed information per student

the used resource types

detailed information of 2 students

detailed usage stats of resources

stats or vis. on content creation

25

Tuesday 20 March 12

demographics

evaluation goal

design changes

negative positive

I.12 CS

students

usability, satisfaction, preliminary usefulness

1st iterationsmall usability

issues

•↑learnability•↓errors•good satisfaction•usefulness positive

II.19 teachers

& TA’s

assessing teacher needs, use & usefulness

help functionresource

recomm. not useful

•provides awareness•all vis. useful•many uses•90% want it

III.12

participants

assessing teacher

needs, expert feedback, use & usefulness

re-orderable parallel

coordinates with

histograms

most addressed needs are indecisive

•provides awareness and feedback•many uses•66% want it•recomm. can be useful

Tuesday 20 March 12

ITERATION FOUR

27

• a CS course on C++ programming

• 11 people: 7 teachers, 2 TA’s & 1 course planner

• richer data set: tracking from programming environment

• qualitative study using a structured face-2-face interview with 25 open questions.

Tuesday 20 March 12

ITERATION FOUR

27

• a CS course on C++ programming

• 11 people: 7 teachers, 2 TA’s & 1 course planner

• richer data set: tracking from programming environment

• qualitative study using a structured face-2-face interview with 25 open questions.

Tuesday 20 March 12

ACHIEVABLE TASKS

28

identify students doing well and at risk

understand time spent better

better course overview

detect students for assistance

course planning

monitor tool and resource use

get a time summary

conflicting visions on students doing well

and at risk

http://www.tdpri.com/forum/telecaster-discussion-forum/48431-ic503z-cap-52-ri-blackguard-book.html

Tuesday 20 March 12

ACHIEVABLE TASKS

28

identify students doing well and at risk

understand time spent better

better course overview

detect students for assistance

course planning

monitor tool and resource use

get a time summary

conflicting visions on students doing well

and at risk

http://www.tdpri.com/forum/telecaster-discussion-forum/48431-ic503z-cap-52-ri-blackguard-book.html

Tuesday 20 March 12

USER SATISFACTION

29

• average SUS score: 69,69%

[Bangor et al. 2008]

Tuesday 20 March 12

USER SATISFACTION

29

• average SUS score: 69,69%

all: want to continue using it9/11: give it to students

[Bangor et al. 2008]

Tuesday 20 March 12

demo-graphics

evaluation goal

design changes negative positive

I.12 CS

students

usability, satisfaction, preliminary usefulness

1st iterationsmall usability

issues

•↑learnability•↓errors•good satisfaction•usefulness positive

II.19

teachers & TA’s

assessing teacher needs, use & usefulness

help functionresource

recomm. not useful

•provides awareness•all vis. useful•many uses•90% want it

III.12

participants

assessing teacher needs, expert

feedback, use & usefulness

re-orderable PC with

histograms

most addressed needs are indecisive

•provides awareness and feedback•many uses•66% want it•recomm. can be useful

IV.11

teachers & TA’s

use, usefulness & satisfaction

filter & search, icons,

zooming in line chart, editing PC

axes

conflicting visions of students

doing well or at risk

•provides time overview•provides course overview•PC assist with detecting problems•many uses & insights•100% want it

Tuesday 20 March 12

CONCLUSION

• SAM enables:

• to find students doing well and at risk• a better course overview• understanding student time spending

• almost all participants want to continue using SAM

• SAM enables to find a wide variety ofnew insights

31

Tuesday 20 March 12

2

slides will appear on http://www.slideshare.net/stengovaerts

slides will appear on http://www.slideshare.net/stengovaerts 32

Tuesday 20 March 12