EVALUATING THE STUDENT ACTIVITY METER:

TWO CASE STUDIES.Sten Govaerts, Katrien Verbert & Erik Duval

http://www.role-project.eu

1

Thursday 8 December 11

OVERVIEW

• Problem statement

•Objectives

•Design & Implementation

• Evaluations

• Conclusion & Future work

2

Thursday 8 December 11

PROBLEM

3

Thursday 8 December 11

PROBLEM

3

Thursday 8 December 11

OBJECTIVES

• self-monitoring for learners

• awareness for teachers

• time tracking

• learning resource use and recommendations

4

Thursday 8 December 11

STUDENT ACTIVITY METER (SAM): DEMO.

5

Thursday 8 December 11

THREE EVALUATIONS

• with CS students• ICWL 2010 paper: “Visualizing Activities for

Self-reflection and Awareness”

• with CGIAR courses and teachers

• with Learning and Knowledge Analytics course participants

6

Thursday 8 December 11

2010 PAPER

• usability and user satisfaction evaluation with 12 CS students

• in general, people understand the visualizations well!

• solved some issues and requests since

Thursday 8 December 11

2010 PAPER

• usability and user satisfaction evaluation with 12 CS students

• in general, people understand the visualizations well!

• solved some issues and requests since

Thursday 8 December 11

2010 PAPER

• usability and user satisfaction evaluation with 12 CS students

• in general, people understand the visualizations well!

• solved some issues and requests since

Thursday 8 December 11

2010 PAPER

• usability and user satisfaction evaluation with 12 CS students

• in general, people understand the visualizations well!

• solved some issues and requests since

Thursday 8 December 11

METHODOLOGY.

• design-based research: give students & teachers a tool and evaluate

• usability studies

• paper mockups

• surveys

8

Thursday 8 December 11

CASE STUDY

• 20 persons: 3 CGIAR, 2 Law, 8 CS teachers & 7 CS TA’s.

• An online survey about usefulness, teacher issues and how the tool can resolve these.

• on average: 40 mins are spent using SAM.

9

Thursday 8 December 11

CGIAR CASE STUDY

10

Provide feedback to the students

Being aware of what students are doing

Knowing about collaboration and communication

Knowing which documents are used and how much

Knowing how and when online tools have been used

Finding the students who are not doing well

Finding the best students

Knowing how much time students spent

Knowing if external learning resources are used

issue for teacher addressed

✔

✗

?✔

✔

✔?

✔

??

✔

✔✔??✔?

✔

Thursday 8 December 11

CGIAR CASE STUDY

10

Provide feedback to the students

Being aware of what students are doing

Knowing about collaboration and communication

Knowing which documents are used and how much

Knowing how and when online tools have been used

Finding the students who are not doing well

Finding the best students

Knowing how much time students spent

Knowing if external learning resources are used

issue for teacher addressed

✔

✗

?✔

✔

✔?

✔

??

✔

✔✔??✔?

✔

Thursday 8 December 11

CGIAR CASE STUDY

11

Bar chartParallel coordinatesLine chart

✔✔✔

Thursday 8 December 11

CGIAR CASE STUDY

12

Thursday 8 December 11

CGIAR CASE STUDY

good indicator for effortunderstand the workload

use for course design optimization

obtain course overview

increase awareness

detect outliers

progress evolutioncompare students

want better 1 to 1 comparison tool

more details on student

want to search for students

more metricswanted

12

Thursday 8 December 11

CGIAR CASE STUDY

good indicator for effortunderstand the workload

use for course design optimization

obtain course overview

increase awareness

detect outliers

progress evolutioncompare students

want better 1 to 1 comparison tool

more details on student

want to search for students

more metricswanted

12

Thursday 8 December 11

CASE STUDY

• open course on learning and knowledge analytics, http://bit.ly/dWYVbX

• 12 visual analytics enthousiasts + experts (who also teach)

• almost identical survey to CGIAR case.

13

Thursday 8 December 11

CASE STUDY

• open course on learning and knowledge analytics, http://bit.ly/dWYVbX

• 12 visual analytics enthousiasts + experts (who also teach)

• almost identical survey to CGIAR case.

13

Thursday 8 December 11

LAK CASE STUDY

14

Provide feedback to the students

Being aware of what students are doing

Knowing about collaboration and communication

Knowing which documents are used and how much

Knowing how and when online tools have been used

Finding the students who are not doing well

Finding the best students

Knowing how much time students spent

Knowing if external learning resources are used

issue for teacher addressed

✔

✗✔

✔

✔?

?

?

✔?

✔?

✔?

✗

✗?

?

Thursday 8 December 11

LAK CASE STUDY

14

Provide feedback to the students

Being aware of what students are doing

Knowing about collaboration and communication

Knowing which documents are used and how much

Knowing how and when online tools have been used

Finding the students who are not doing well

Finding the best students

Knowing how much time students spent

Knowing if external learning resources are used

issue for teacher addressed

✔

✗✔

✔

✔?

?

?

✔?

✔?

✔?

✗

✗?

?

Thursday 8 December 11

LAK CASE STUDY

15

Bar chartParallel coordinatesLine chart

???

Thursday 8 December 11

LAK CASE STUDY

16

Thursday 8 December 11

LAK CASE STUDY

chronological course dwell time

verify the status of the classroom activity

self-reflection to measure progress and increase

motivation find students experiencing problems and low

engagement

more metrics

more data16

Thursday 8 December 11

LAK CASE STUDY

chronological course dwell time

verify the status of the classroom activity

self-reflection to measure progress and increase

motivation find students experiencing problems and low

engagement

more metrics

more data16

Thursday 8 December 11

IDEA’S FROM EXPERTS

17

detailed information per student

the used resource types

detailed information of 2 students

detailed usage stats of resources

stats or vis. on content creation

Thursday 8 December 11

IDEA’S FROM EXPERTS

1

2

3

4

5

17

detailed information per student

the used resource types

detailed information of 2 students

detailed usage stats of resources

stats or vis. on content creation

Thursday 8 December 11

CONTINUED WORK...

18

Thursday 8 December 11

CONTINUED WORK...

18

Thursday 8 December 11

FUTURE WORK

• extend SAM with feedback of experts

• evaluate the usefulness in a larger group of students

• evaluate with teachers during and after a course.

19

Thursday 8 December 11

YOUR DATA IN SAM ?!

20

ask me!sten.govaerts@cs.kuleuven.be

Thursday 8 December 11

2

slides will appear on http://www.slideshare.net/stengovaerts

slides will appear on http://www.slideshare.net/stengovaerts 21

Thursday 8 December 11