From Findability to Awareness: Metadata in Music and Technology Enhanced Learning.

Post on 06-May-2015

740 views 1 download

description

Slides of my PhD defense on September 11, 2012.

Transcript of From Findability to Awareness: Metadata in Music and Technology Enhanced Learning.

FROM FINDABILITY TO AWARENESS: METADATA IN MUSIC AND TECHNOLOGY

ENHANCED LEARNING

Sten Govaerts

Promotor: Prof. dr. ir. Erik DuvalCo-promotor: Dr. Katrien Verbert

Vinden en beseffen: Metadata in muziek en E-learning

1

Wednesday 12 September 12

http://www.britannica.com/bps/media-view/96031/1/0/0 2

Wednesday 12 September 12

3

Wednesday 12 September 12

http://www.flickr.com/photos/markkelley/1022720488/ 4

Wednesday 12 September 12

http://www.flickr.com/photos/markkelley/1022720488/ 4

Wednesday 12 September 12

METADATA• title: Maiden Voyage• main artist: Herbie Hancock• artists: Freddie Hubbard, George

Coleman, Ron Carter, Anthony Williams• record label: Blue Note, Capitol

Records• instruments: trumpet, tenor

saxophone, piano, bass, drums.• recording date: March 17, 1965• studio: Van Gelder Studio, New Jersey• version: remastered by Rudy Van Gelder

5

Wednesday 12 September 12

CONTEXT.

6

Wednesday 12 September 12

OVERVIEW.

7

Wednesday 12 September 12

OVERVIEW.

7

Wednesday 12 September 12

OVERVIEW.

7

Wednesday 12 September 12

OVERVIEW.

7

Wednesday 12 September 12

OVERVIEW.

7

Wednesday 12 September 12

OVERVIEW.

7

Wednesday 12 September 12

OVERVIEW.

7

Wednesday 12 September 12

OVERVIEW.

7

Wednesday 12 September 12

OVERVIEW.

7

Wednesday 12 September 12

OVERVIEW.

7

Wednesday 12 September 12

OVERVIEW.

7

Wednesday 12 September 12

OVERVIEW.

RQ1: How can metadata be leveraged to provide efficient and

effective access to content?7

Wednesday 12 September 12

OVERVIEW.RQ2: How can metadata creation

be made more scalable?

7

Wednesday 12 September 12

OVERVIEW.

RQ3: How can metadata standards be selected for specific

functionalities of applications?7

Wednesday 12 September 12

OVERVIEW.RQ4: How can attention metadata be leveraged to improve

awareness and self-reflection?

7

Wednesday 12 September 12

DOMAINS.

Music Technology-enhanced Learning (TEL)

8

Wednesday 12 September 12

STEP 1 – DEFINE.

9

Wednesday 12 September 12

STEP 1 – DEFINE.

RQ3: How can metadata standards be selected for specific

functionalities of applications?9

Wednesday 12 September 12

METADATA SCHEMA<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:dc="http://purl.org/dc/elements/1.1/"> <rdf:Description rdf:about="http://media.example.com/audio/guide.ra"> <dc:creator>Herbie Hancock</dc:creator> <dc:title>Maiden Voyage</dc:title> <dc:contributor>Freddie Hubbard</dc:contributor> <dc:contributor>George Coleman</dc:contributor> <dc:contributor>Ron Carter</dc:contributor> <dc:contributor>Anthony Williams</dc:contributor> <dc:description>Maiden Voyage is the fifth album led by jazz musician Herbie Hancock, and was recorded by Rudy Van Gelder on May 17, 1965 for Blue Note Records.</dc:description> <dc:date>1965-03-17</dc:date> <dc:publisher>Blue Note</dc:publisher> <dc:type>Music album</dc:type> <dc:format>audio/mp3</dc:format> <dc:rights>copyrighted by Capitol Records, Inc.</dc:rights> </rdf:Description> </rdf:RDF>

Dub

lin C

ore

10step 1 - define.Wednesday 12 September 12

METADATA SCHEMA@prefix mo: <http://purl.org/ontology/mo/> .@prefix : <#> .@prefix dc: <http://purl.org/dc/elements/1.1/> .@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .@prefix event: <http://purl.org/NET/c4dm/event.owl#> .

:work a mo:MusicalWork ; dc:title "Herbie Hancock - Maiden Voyage" .

:performance a mo:Performance ; dc:title "Maiden Voyage" ; mo:performance_of :work ; mo:performer <http://dbpedia.org/page/Herbie_Hancock> ; dc:date "1965-03-17"^^xsd:date . event:place <http://dbpedia.org/page/Van_Gelder_Studio> ; mo:performance_of <http://dbpedia.org/page/Maiden_Voyage_(Herbie_Hancock_album)> . mo:recorded_as [ a mo:Signal ; mo:published_as :track1 ; dc:title "Original recording at the Van Gelder Studio" ; ] .

Mus

ic O

ntol

ogy

11step 1 - define.Wednesday 12 September 12

METADATA SCHEMA:track1 a mo:Track ; mo:track_number 1 ; dc:title "Maiden Voyage". :signal mo:time [ tl:duration "PT2M14S"^^xsd:duration; tl:timeline :tl; ]. :chorus1 a ps:Chorus; rdfs:label "First chorus"; event:time [ tl:timeline :tl; tl:start "PT0S"^^xsd:duration; tl:duration "PT9S"^^xsd:duration; ].

:verse1 a ps:Verse; rdfs:label "First verse"; event:time [ tl:timeline :tl; tl:start "PT9S"^^xsd:duration; tl:duration "PT33S"^^xsd:duration; ].

Mus

ic O

ntol

ogy

11step 1 - define.Wednesday 12 September 12

FINDING AN APPROPRIATE METADATA SCHEMA...

Corthaut, Nik; Govaerts, Sten; Verbert, Katrien; Duval, Erik. Connecting the dots: music metadata generation,

schemas and applications. ISMIR 2008.12step 1 - define.Wednesday 12 September 12

...FOR THE RIGHT APPLICATION.

Corthaut, Nik; Govaerts, Sten; Verbert, Katrien; Duval, Erik. Connecting the dots: music metadata

generation, schemas and applications. ISMIR 2008.13step 1 - define.Wednesday 12 September 12

...FOR THE RIGHT APPLICATION.

Corthaut, Nik; Govaerts, Sten; Verbert, Katrien; Duval, Erik. Connecting the dots: music metadata

generation, schemas and applications. ISMIR 2008.13step 1 - define.Wednesday 12 September 12

STEP 2 – CREATE.

14

Wednesday 12 September 12

STEP 2 – CREATE.

RQ2: How can metadata creation be made more scalable?

14

Wednesday 12 September 12

country of origin of an artist.

http://www.flickr.com/photos/twicepix/4437851857/15

Wednesday 12 September 12

COUNTRY & CONTINENT

• useful for :

• subgenres• popularity• recommendations

16step 2 - create.Wednesday 12 September 12

COUNTRY & CONTINENT

• useful for :

• subgenres• popularity• recommendations

• expensive to annotate

16step 2 - create.Wednesday 12 September 12

COUNTRY & CONTINENT

• useful for :

• subgenres• popularity• recommendations

• expensive to annotate

• very little existing research

16step 2 - create.Wednesday 12 September 12

MASHUP.

twitter

last.fm

website

dapper

yahoo! pipes

last.fm

last on am/fm

google app engine

google maps

freebase

last.fm

youtube

17step 2 - create.Wednesday 12 September 12

Evaluation:ground truth data set of 11275 artists

last.fm

freebase

biography

combination

0% 25% 50% 75% 100%

77%

71%

90%

91%

Accuracy

last.fm

freebase

biography

combination

0% 15% 30% 45% 60%

59%

55%

26%

7%

Coverage

Three data sources

• last.fm• freebase• biographies

Govaerts, Sten; Duval, Erik. A Web-based approach to determine the origin of an artist. ISMIR 2009.

18step 2 - create.Wednesday 12 September 12

Evaluation:ground truth data set of 11275 artists

last.fm

freebase

biography

combination

0% 25% 50% 75% 100%

77%

71%

90%

91%

Accuracy

last.fm

freebase

biography

combination

0% 15% 30% 45% 60%

59%

55%

26%

7%

Coverage

Three data sources

• last.fm• freebase• biographies

Govaerts, Sten; Duval, Erik. A Web-based approach to determine the origin of an artist. ISMIR 2009.

Ennio Morricone (born 10 November 1928 in Rome) is an Italian composer especially noted for his film scores. He has composed the scores of more than 500 films and TV series. Although only 30 of these are for Western films, it is for this work which he is best known. Morricone’s sparse style of composition for the genre is particularly exemplified by the soundtracks of the classic spaghetti westerns The Good, the Bad and the Ugly (Sergio Leone, 1966) and Once Upon A Time In The West (Sergio Leone, 1968). [...]In 2003 he scored the Japanese Taiga drama about Miyamoto Musashi, Japan’s legendary warrior. In 2005 he was on a world tour with the Roma Sinfonietta Orchestra, touring from cities like New York, Bilbao, Bratislava and Tokyo. In 2006, Morricone collaborated with Morrissey by composing the string part for his song “Dear God Please Help Me”, recorded in Rome’s Forum Music Village Studios. In recognition of his vast body of work and dedication, Morricone received an honorary Academy Award in 2007; Morricone had received nominations for Best Original Score in the past, but had failed to win.

18step 2 - create.Wednesday 12 September 12

Evaluation:ground truth data set of 11275 artists

last.fm

freebase

biography

combination

0% 25% 50% 75% 100%

77%

71%

90%

91%

Accuracy

last.fm

freebase

biography

combination

0% 15% 30% 45% 60%

59%

55%

26%

7%

Coverage

Three data sources

• last.fm• freebase• biographies

Govaerts, Sten; Duval, Erik. A Web-based approach to determine the origin of an artist. ISMIR 2009.

18step 2 - create.Wednesday 12 September 12

Evaluation:ground truth data set of 11275 artists

0% 25% 50% 75% 100%

last.fm

freebase

biography

combination

91%

90%

71%

77%

Accuracy

last.fm

freebase

biography

combination

0% 15% 30% 45% 60%

59%

55%

26%

7%

Coverage

Three data sources

• last.fm• freebase• biographies

Govaerts, Sten; Duval, Erik. A Web-based approach to determine the origin of an artist. ISMIR 2009.

18step 2 - create.Wednesday 12 September 12

Evaluation:ground truth data set of 11275 artists

0% 25% 50% 75% 100%

last.fm

freebase

biography

combination

91%

90%

71%

77%

Accuracy

0% 15% 30% 45% 60%

last.fm

freebase

biography

combination

7%

26%

55%

59%

Coverage

Three data sources

• last.fm• freebase• biographies

Govaerts, Sten; Duval, Erik. A Web-based approach to determine the origin of an artist. ISMIR 2009.

18step 2 - create.Wednesday 12 September 12

Evaluation:ground truth data set of 11275 artists

0% 25% 50% 75% 100%

last.fm

freebase

biography

combination

91%

90%

71%

77%

Accuracy

0% 15% 30% 45% 60%

last.fm

freebase

biography

combination

7%

26%

55%

59%

Coverage

Three data sources

• last.fm• freebase• biographies

}

Govaerts, Sten; Duval, Erik. A Web-based approach to determine the origin of an artist. ISMIR 2009.

18step 2 - create.Wednesday 12 September 12

Evaluation:ground truth data set of 11275 artists

0% 25% 50% 75% 100%

last.fm

freebase

biography

combination

91%

90%

71%

77%

Accuracy

0% 15% 30% 45% 60%

last.fm

freebase

biography

combination

7%

26%

55%

59%

Coverage

Three data sources

• last.fm• freebase• biographies

}

}Govaerts, Sten; Duval, Erik. A Web-based approach to

determine the origin of an artist. ISMIR 2009.

18step 2 - create.Wednesday 12 September 12

LAST.FM MASHUP

http://www.cs.kuleuven.be/~sten/lastonamfm19step 2 - create.

Wednesday 12 September 12

musical genre.

http://www.flickr.com/photos/7593077@N03/5076100406/in/photostream/ 20

Wednesday 12 September 12

USING SEARCH ENGINES

21step 2 - create.Wednesday 12 September 12

USING SEARCH ENGINES

21step 2 - create.Wednesday 12 September 12

USING SEARCH ENGINES

21step 2 - create.Wednesday 12 September 12

USING SEARCH ENGINES

21step 2 - create.Wednesday 12 September 12

USING SEARCH ENGINES

21step 2 - create.Wednesday 12 September 12

CLASSIFICATION WITH SEARCH ENGINES

using co-occurrence

M. Schedl, T. Pohle, P. Knees, G. Widmer, “Assigning and Visualizing Music Genres by Web-based Co-occurrence Analysis”, Proceedings of the 7th International Conference on Music Information Retrieval, 2006, pp. 260-265.

22step 2 - create.Wednesday 12 September 12

CLASSIFICATION WITH SEARCH ENGINES

using co-occurrence

M. Schedl, T. Pohle, P. Knees, G. Widmer, “Assigning and Visualizing Music Genres by Web-based Co-occurrence Analysis”, Proceedings of the 7th International Conference on Music Information Retrieval, 2006, pp. 260-265.

22step 2 - create.Wednesday 12 September 12

23step 2 - create.Wednesday 12 September 12

Rock:

Blues:

Country:

Jazz:

Pop:

Metal:

23step 2 - create.Wednesday 12 September 12

Rock:

Blues:

Country:

Jazz:

Pop:

Metal:

0,013

0,009

0,013

0,005

0,0150,009

23step 2 - create.Wednesday 12 September 12

RESULTS

• 1st results were much worse

• what happened?

• re-run the original experiment

• evaluate on the same data set: 1995 artists and 9 genres.

• different search engines: Google, Yahoo! and Microsoft.

• over time: 8 times over a period of 36 days.

24step 2 - create.Wednesday 12 September 12

25step 2 - create.Wednesday 12 September 12

25step 2 - create.Wednesday 12 September 12

26step 2 - create.Wednesday 12 September 12

WHAT TO USE?

• use Google when it’s stable else rely on Yahoo!

• when is it stable? test with a small set

• some artists get classified incorrectly on bad days

• compare the accuracy achieved with the test set to the average.

27step 2 - create.Wednesday 12 September 12

WHAT TO USE?

• use Google when it’s stable else rely on Yahoo!

• when is it stable? test with a small set

• some artists get classified incorrectly on bad days

• compare the accuracy achieved with the test set to the average.

Govaerts, Sten; Corthaut, Nik; Duval, Erik. Using search engine for classification: does it still work? adMIRe WS, ISM2009.

27step 2 - create.Wednesday 12 September 12

STEP 4 – UTILISE.

28

Wednesday 12 September 12

STEP 4 – UTILISE.

28

Wednesday 12 September 12

STEP 4 – UTILISE.Ch 5 & 6

RQ1: How can metadata be leveraged to provide efficient and

effective access to content?28

Wednesday 12 September 12

FEDERATED SEARCH.

Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated search and social recommendation widget. SRS WS, CSCW 2011.

Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning Environments: the ROLE Interoperability framework. EC-TEL 2011.

step 4 - utilise. 29

Wednesday 12 September 12

FEDERATED SEARCH.

Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated search and social recommendation widget. SRS WS, CSCW 2011.

Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning Environments: the ROLE Interoperability framework. EC-TEL 2011.

step 4 - utilise. 29

Wednesday 12 September 12

FEDERATED SEARCH.

Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated search and social recommendation widget. SRS WS, CSCW 2011.

Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning Environments: the ROLE Interoperability framework. EC-TEL 2011.

step 4 - utilise. 29

Wednesday 12 September 12

FEDERATED SEARCH.

Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated search and social recommendation widget. SRS WS, CSCW 2011.

Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning Environments: the ROLE Interoperability framework. EC-TEL 2011.

step 4 - utilise. 29

Wednesday 12 September 12

FEDERATED SEARCH.

Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated search and social recommendation widget. SRS WS, CSCW 2011.

Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning Environments: the ROLE Interoperability framework. EC-TEL 2011.

step 4 - utilise. 29

Wednesday 12 September 12

FEDERATED SEARCH.

Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated search and social recommendation widget. SRS WS, CSCW 2011.

Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning Environments: the ROLE Interoperability framework. EC-TEL 2011.

step 4 - utilise. 29

Wednesday 12 September 12

FEDERATED SEARCH.

Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated search and social recommendation widget. SRS WS, CSCW 2011.

Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning Environments: the ROLE Interoperability framework. EC-TEL 2011.

step 4 - utilise. 29

Wednesday 12 September 12

step 4 - utilise.

CONTEXT

• Personal Learning Environment:

• customizable• re-use, creation & mashup of tools, resources

• enable users to access content

• in different contexts

30

Wednesday 12 September 12

step 4 - utilise.

CONTEXT

• Personal Learning Environment:

• customizable• re-use, creation & mashup of tools, resources

• enable users to access content

• in different contexts

30

Wednesday 12 September 12

step 4 - utilise.

CONTEXT

• Personal Learning Environment:

• customizable• re-use, creation & mashup of tools, resources

• enable users to access content

• in different contexts

30

Wednesday 12 September 12

WHAT’S A WIDGET ?!?

31step 4 - utilise.Wednesday 12 September 12

WHAT’S A WIDGET ?!?

31step 4 - utilise.Wednesday 12 September 12

STEP 5 & 6 – CAPTURE & PERSONALISE.

32

Wednesday 12 September 12

STEP 5 & 6 – CAPTURE & PERSONALISE.

Ch 5 & 6

RQ1: How can metadata be leveraged to provide efficient and

effective access to content?

32

Wednesday 12 September 12

step 5 & 6 - capture/personalise.

ATTENTION METADATA

Attention metadata does not describe the content itself, but what people do

with the content.

33

Wednesday 12 September 12

step 5 & 6 - capture/personalise.

ATTENTION METADATA

Attention metadata does not describe the content itself, but what people do

with the content.

copyright by Warner Bros. Pictures

33

Wednesday 12 September 12

step 5 & 6 - capture/personalise.

ATTENTION METADATA

Attention metadata does not describe the content itself, but what people do

with the content.

copyright by Warner Bros. Pictures

33

Wednesday 12 September 12

UI.

34step 4 - utilise.Wednesday 12 September 12

UI.

34step 4 - utilise.Wednesday 12 September 12

step 5 & 6 - capture/personalise.

Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated search and social recommendation widget. SRS WS, CSCW 2011.

Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning Environments: the ROLE Interoperability framework. EC-TEL 2011.

SOCIAL RECOMMENDATIONS.

35

Wednesday 12 September 12

step 5 & 6 - capture/personalise.

Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated search and social recommendation widget. SRS WS, CSCW 2011.

Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning Environments: the ROLE Interoperability framework. EC-TEL 2011.

SOCIAL RECOMMENDATIONS.

35

Wednesday 12 September 12

step 5 & 6 - capture/personalise.

Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated search and social recommendation widget. SRS WS, CSCW 2011.

Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning Environments: the ROLE Interoperability framework. EC-TEL 2011.

SOCIAL RECOMMENDATIONS.

35

Wednesday 12 September 12

step 5 & 6 - capture/personalise.

Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated search and social recommendation widget. SRS WS, CSCW 2011.

Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning Environments: the ROLE Interoperability framework. EC-TEL 2011.

SOCIAL RECOMMENDATIONS.

35

Wednesday 12 September 12

PAGE RANKHYPERLINK

WEBSITE

A node is important if and only if many other important nodes point to it

Rank of node i:

36step 5 & 6 - capture/personalise.

Wednesday 12 September 12

friend

connection

saved/shared

liked

liked

disliked

friend

connection

saved/shared

R1

R2

R3

R4

R5

Erik

Sandy

Sten

OUR CASE

37step 5 & 6 - capture/personalise.

Wednesday 12 September 12

THREE CASE STUDIES2 31

38step 5 & 6 - capture/personalise.

Wednesday 12 September 12

KU LEUVEN & EPFL

• 15 PhD students.

39step 5 & 6 - capture/personalise.

Wednesday 12 September 12

KU LEUVEN & EPFL

• 15 PhD students.

• usability, satisfaction, usefulness

39step 5 & 6 - capture/personalise.

Wednesday 12 September 12

KU LEUVEN & EPFL

• 15 PhD students.

• usability, satisfaction, usefulness

• usefulness of recommendations: 11/14 pro.

39step 5 & 6 - capture/personalise.

Wednesday 12 September 12

KU LEUVEN & EPFL

• 15 PhD students.

• usability, satisfaction, usefulness

• usefulness of recommendations: 11/14 pro.

• user satisfaction: System Usability Scale (SUS) & MS Desirability Toolkit

39step 5 & 6 - capture/personalise.

Wednesday 12 September 12

KU LEUVEN & EPFL

• 15 PhD students.

• usability, satisfaction, usefulness

• usefulness of recommendations: 11/14 pro.

• user satisfaction: System Usability Scale (SUS) & MS Desirability Toolkit

• SUS score: 66,25%

39step 5 & 6 - capture/personalise.

Wednesday 12 September 12

KU LEUVEN & EPFL

• 15 PhD students.

• usability, satisfaction, usefulness

• usefulness of recommendations: 11/14 pro.

• user satisfaction: System Usability Scale (SUS) & MS Desirability Toolkit

• SUS score: 66,25% KU Leuven: high (75%)EPFL + one KU Leuven: low (50%)

39step 5 & 6 - capture/personalise.

Wednesday 12 September 12

WHY THE DIFFERENT SUS?

• 1st phase by 2 interviewers

40step 5 & 6 - capture/personalise.

Wednesday 12 September 12

WHY THE DIFFERENT SUS?

• 1st phase by 2 interviewers

40step 5 & 6 - capture/personalise.

Wednesday 12 September 12

WHY THE DIFFERENT SUS?

• 1st phase by 2 interviewers

40step 5 & 6 - capture/personalise.

Wednesday 12 September 12

WHY THE DIFFERENT SUS?

• issues:

• distraction from unrelated widgets

• layout too dense

• height of widgets too small

• not evaluating the widget but the whole experience...

41step 5 & 6 - capture/personalise.

Wednesday 12 September 12

DESIRABILITY...

42step 5 & 6 - capture/personalise.

Wednesday 12 September 12

DESIRABILITY...

42step 5 & 6 - capture/personalise.

Wednesday 12 September 12

DESIRABILITY...

42step 5 & 6 - capture/personalise.

Wednesday 12 September 12

FESTO

• LMS (CLIX) extended with widgets

• 25 heterogeneous participants

• evaluation setup:

1. video introduction to the new system

2. period of experimentation

3. questionnaire

43step 5 & 6 - capture/personalise.

Wednesday 12 September 12

FESTO

• LMS (CLIX) extended with widgets

• 25 heterogeneous participants

• evaluation setup:

1. video introduction to the new system

2. period of experimentation

3. questionnaire

43step 5 & 6 - capture/personalise.

Wednesday 12 September 12

FESTO RESULTS

44step 5 & 6 - capture/personalise.

Wednesday 12 September 12

FESTO RESULTS

88% would recommend it to their colleagues

81% agreed that it helps them to be more effective

44step 5 & 6 - capture/personalise.

Wednesday 12 September 12

JIAO TONG UNIVERSITY

• offers higher education for adults with a full-time job

• early prototype with

• evaluation setup:

1. task list (ETA 15 mins)

2. questionnaire (in Chinese) (partially from ResQUE)

45step 5 & 6 - capture/personalise.

Wednesday 12 September 12

JIAO TONG UNIVERSITY

• offers higher education for adults with a full-time job

• early prototype with

• evaluation setup:

1. task list (ETA 15 mins)

2. questionnaire (in Chinese) (partially from ResQUE)

45step 5 & 6 - capture/personalise.

Wednesday 12 September 12

RESULTS

• ran over 3 weeks, 13 respondents

• results for federated search widget:• 69% used the federated search widget

• 56% found the recommendations relevant to the course

• 38% wanted to continue using the federated search widget

• Personal learning environment results• 23% found it useful for the course

• 39% found it easy to use, 15% found it navigationally consistent

• 46% were satisfied with the personal learning environment

46step 5 & 6 - capture/personalise.

Wednesday 12 September 12

WHY?

• an established LMS augmented with PLE features brand new system

• cultural differences?

• slow internet connection (long loading and querying times)

• lesson learned: the choice of evaluation environment is very important

47step 5 & 6 - capture/personalise.

Wednesday 12 September 12

WHY?

• an established LMS augmented with PLE features brand new system

• cultural differences?

• slow internet connection (long loading and querying times)

• lesson learned: the choice of evaluation environment is very important

47

users perceive federated search as useful and effective IF the content in the repositories relates

to their tasks.[Belliston et al., 2007], [Korah et al., 2010], [Gore, 2008]

step 5 & 6 - capture/personalise.

Wednesday 12 September 12

WHY?

• an established LMS augmented with PLE features brand new system

• cultural differences?

• slow internet connection (long loading and querying times)

• lesson learned: the choice of evaluation environment is very important

47

users perceive federated search as useful and effective IF the content in the repositories relates

to their tasks.[Belliston et al., 2007], [Korah et al., 2010], [Gore, 2008]

our evaluation indicates that users benefit from the social recommendations.

recommendations from friends are preferred over system recommendations. [Shinha et al., 2001]

step 5 & 6 - capture/personalise.

Wednesday 12 September 12

STEP 7 – CREATE AWARENESS.

48

Wednesday 12 September 12

STEP 7 – CREATE AWARENESS.Ch 7

48

Wednesday 12 September 12

STEP 7 – CREATE AWARENESS.Ch 7

RQ4: How can attention metadata be leveraged to improve

awareness and self-reflection?

48

Wednesday 12 September 12

PROBLEM

http://www.ypfp.org/classroom

49step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

PROBLEM

http://www.ypfp.org/classroom

49step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

STUDENT ACTIVITY METER

• visualisations of attention metadata

• self-monitoring for learners

design

evaluation

step 5 & 7 - capture/create awareness.

implementation

50

Wednesday 12 September 12

STUDENT ACTIVITY METER

• visualisations of attention metadata

• self-monitoring for learners

• awareness for teachersdesign

evaluation

step 5 & 7 - capture/create awareness.

implementation

50

Wednesday 12 September 12

STUDENT ACTIVITY METER

• visualisations of attention metadata

• self-monitoring for learners

• awareness for teachers

• time trackingdesign

evaluation

step 5 & 7 - capture/create awareness.

implementation

50

Wednesday 12 September 12

STUDENT ACTIVITY METER

• visualisations of attention metadata

• self-monitoring for learners

• awareness for teachers

• time tracking

• learning resource use and recommendations

design

evaluation

step 5 & 7 - capture/create awareness.

implementation

50

Wednesday 12 September 12

STUDENT ACTIVITY METER

• visualisations of attention metadata

• self-monitoring for learners

• awareness for teachers

• time tracking

• learning resource use and recommendations

• part of Learning Analytics research [ACM LAK conf., Siemens 2011, Duval 2011]

design

evaluation

step 5 & 7 - capture/create awareness.

implementation

50

Wednesday 12 September 12

demo...

http://www.leveraas.com/

Govaerts, Sten; Verbert, Katrien, Duval, Erik; Pardo, Abelardo. The Student Activity Meter for Awareness and Self-reflection. CHI 2012.

51

Wednesday 12 September 12

demographics

evaluation goal

design changes negative positive

I. 12 CS students

usability, satisfaction, preliminary usefulness

1st iteration small usability issues

•↑learnability•↓errors•good satisfaction•usefulness positive

52step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

step 5 & 7 - capture/create awareness.

ITERATION TWO & THREE

• CGIAR: 19 teachers - LAK: 12

• An online survey about usefulness, teacher issues and how the tool can resolve these.

53

Wednesday 12 September 12

step 5 & 7 - capture/create awareness.

Provide feedback to the students

Being aware of what students are doing

Knowing about collaboration and communication

Knowing which documents are used and how much

Knowing how and when online tools have been used

Finding the students who are not doing well

Finding the best students

Knowing how much time students spent

Knowing if external learning resources are used

issue for teacher

?✔

✔?

??

✔✔??✔?

add-ressed

✔✔

✔?

?

?

issue for teacher

add-ressed

?

✔?

✔?

?

?

CGIAR LAK1 2 3 4 5

Issues

Provide feedback to the studentsBeing aware of what students are doingKnowing about collaboration and communication *Knowing which documents are used and how muchKnowing how and when online tools have been usedFinding the students who are not doing well

Finding the best students

Knowing how much time students spentKnowing if external learning resources are used

Addressed by SAM

1 2 3 4 5

54

Wednesday 12 September 12

step 5 & 7 - capture/create awareness.

Provide feedback to the students

Being aware of what students are doing

Knowing about collaboration and communication

Knowing which documents are used and how much

Knowing how and when online tools have been used

Finding the students who are not doing well

Finding the best students

Knowing how much time students spent

Knowing if external learning resources are used

issue for teacher

?✔

✔?

??

✔✔??✔?

add-ressed

✔✔

✔?

?

?

issue for teacher

add-ressed

?

✔?

✔?

?

?

CGIAR LAK

54

Wednesday 12 September 12

step 5 & 7 - capture/create awareness.

Provide feedback to the students

Being aware of what students are doing

Knowing about collaboration and communication

Knowing which documents are used and how much

Knowing how and when online tools have been used

Finding the students who are not doing well

Finding the best students

Knowing how much time students spent

Knowing if external learning resources are used

issue for teacher

?✔

✔?

??

✔✔??✔?

add-ressed

✔✔

✔?

?

?

issue for teacher

add-ressed

?

✔?

✔?

?

?

CGIAR LAK

54

Wednesday 12 September 12

CGIAR CASE STUDY

55step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

CGIAR CASE STUDY

good indicator for effortunderstand the workload

use for course design optimization

obtain course overview

increase awareness

detect outliers

progress evolutioncompare students

want better 1 to 1 comparison tool

more details on student

want to search for students

more metricswanted

55step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

CGIAR CASE STUDY

good indicator for effortunderstand the workload

use for course design optimization

obtain course overview

increase awareness

detect outliers

progress evolutioncompare students

want better 1 to 1 comparison tool

more details on student

want to search for students

more metricswanted

55step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

LAK CASE STUDY

http://maranash.blogspot.com/2011/01/conference-shout-out.html56step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

LAK CASE STUDY

chronological course dwell time

verify the status of the classroom activity

self-reflection to measure progress and increase

motivation

find students experiencing problems and low

engagement

more metrics

more data

http://maranash.blogspot.com/2011/01/conference-shout-out.html56step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

LAK CASE STUDY

chronological course dwell time

verify the status of the classroom activity

self-reflection to measure progress and increase

motivation

find students experiencing problems and low

engagement

more metrics

more data

http://maranash.blogspot.com/2011/01/conference-shout-out.html56step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

demographics

evaluation goal

design changes

negative positive

I.12 CS

students

usability, satisfaction, preliminary usefulness

1st iterationsmall usability

issues

•↑learnability•↓errors•good satisfaction•usefulness positive

II.19

teachers & TA’s

assessing teacher

needs, use & usefulness

help functionresource

recomm. not useful

•provides awareness•all vis. useful•many uses•90% want it

III.12

participants

assessing teacher

needs, expert feedback, use & usefulness

re-orderable parallel

coordinates with

histograms

most addressed needs are indecisive

•provides awareness and feedback•many uses•66% want it•recomm. can be useful

57

Wednesday 12 September 12

step 5 & 7 - capture/create awareness.

ITERATION FOUR

• a CS course on C++ programming

• 11 people: 7 teachers, 2 TA’s & 1 course planner

• richer data set: tracking from programming environment

• qualitative study using a structured face-2-face interview with 25 open questions.

58

Wednesday 12 September 12

step 5 & 7 - capture/create awareness.

ITERATION FOUR

• a CS course on C++ programming

• 11 people: 7 teachers, 2 TA’s & 1 course planner

• richer data set: tracking from programming environment

• qualitative study using a structured face-2-face interview with 25 open questions.

58

Wednesday 12 September 12

ACHIEVABLE TASKS

identify students doing well and at risk

understand time spent better

better course overview

detect students for assistance

course planning

monitor tool and resource use

get a time summary

conflicting visions on students doing well

and at risk

http://www.tdpri.com/forum/telecaster-discussion-forum/48431-ic503z-cap-52-ri-blackguard-book.html 59step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

ACHIEVABLE TASKS

identify students doing well and at risk

understand time spent better

better course overview

detect students for assistance

course planning

monitor tool and resource use

get a time summary

conflicting visions on students doing well

and at risk

http://www.tdpri.com/forum/telecaster-discussion-forum/48431-ic503z-cap-52-ri-blackguard-book.html 59step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

ACHIEVABLE TASKS

identify students doing well and at risk

understand time spent better

better course overview

detect students for assistance

course planning

monitor tool and resource use

get a time summary

conflicting visions on students doing well

and at risk

http://www.tdpri.com/forum/telecaster-discussion-forum/48431-ic503z-cap-52-ri-blackguard-book.html 59step 5 & 7 - capture/create awareness.

all: want to continue using it9/11: give it to students

Wednesday 12 September 12

demo-graphics

evaluation goal

design changes negative positive

I.12 CS

students

usability, satisfaction, preliminary usefulness

1st iterationsmall usability

issues

•↑learnability•↓errors•good satisfaction•usefulness positive

II.19

teachers & TA’s

assessing teacher needs, use & usefulness

help functionresource

recomm. not useful

•provides awareness•all vis. useful•many uses•90% want it

III.12

participants

assessing teacher needs, expert

feedback, use & usefulness

re-orderable PC with

histograms

most addressed needs are indecisive

•provides awareness and feedback•many uses•66% want it•recomm. can be useful

IV.11

teachers & TA’s

use, usefulness & satisfaction

filter & search, icons,

zooming in line chart, editing PC

axes

conflicting visions of students

doing well or at risk

•provides time overview•provides course overview•PC assist with detecting problems•many uses & insights•100% want it

60

Wednesday 12 September 12

CONCLUSION

• SAM is perceived useful and contributes to better awareness.

61step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

CONCLUSION

• SAM is perceived useful and contributes to better awareness.

• SAM enables:

61step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

CONCLUSION

• SAM is perceived useful and contributes to better awareness.

• SAM enables:

• to detect potential students doing well and at risk

61step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

CONCLUSION

• SAM is perceived useful and contributes to better awareness.

• SAM enables:

• to detect potential students doing well and at risk• a better course overview

61step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

CONCLUSION

• SAM is perceived useful and contributes to better awareness.

• SAM enables:

• to detect potential students doing well and at risk• a better course overview• understanding student time spending

61step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

CONCLUSION

• SAM is perceived useful and contributes to better awareness.

• SAM enables:

• to detect potential students doing well and at risk• a better course overview• understanding student time spending• to find a wide variety of new insights

61step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

CONCLUSION

• SAM is perceived useful and contributes to better awareness.

• SAM enables:

• to detect potential students doing well and at risk• a better course overview• understanding student time spending• to find a wide variety of new insights

• almost all want to continue using SAM

61step 5 & 7 - capture/create awareness.

Wednesday 12 September 12

CONTRIBUTIONS.

1. A structured method for selecting an appropriate metadata schema

2. The design & evaluation of a metadata generator for the origin of an artist

3. The re-evaluation of a general metadata generation approach

4. The design & evaluation of federated search tool with social recommendations

5. The design & evaluation of visualisations of attention metadata to support awareness and self-reflection.

62

Wednesday 12 September 12

CONTRIBUTIONS.

1. A structured method for selecting an appropriate metadata schema

2. The design & evaluation of a metadata generator for the origin of an artist

3. The re-evaluation of a general metadata generation approach

4. The design & evaluation of federated search tool with social recommendations

5. The design & evaluation of visualisations of attention metadata to support awareness and self-reflection.

Ch 8

62

Wednesday 12 September 12

CONTRIBUTIONS.

1. A structured method for selecting an appropriate metadata schema

2. The design & evaluation of a metadata generator for the origin of an artist

3. The re-evaluation of a general metadata generation approach

4. The design & evaluation of federated search tool with social recommendations

5. The design & evaluation of visualisations of attention metadata to support awareness and self-reflection.

RQ 3

Ch 8

62

Wednesday 12 September 12

CONTRIBUTIONS.

1. A structured method for selecting an appropriate metadata schema

2. The design & evaluation of a metadata generator for the origin of an artist

RQ 3

Ch 8

62

Wednesday 12 September 12

CONTRIBUTIONS.

1. A structured method for selecting an appropriate metadata schema

2. The design & evaluation of a metadata generator for the origin of an artist

3. The re-evaluation of a general metadata generation approach

RQ 3

Ch 8

62

Wednesday 12 September 12

CONTRIBUTIONS.

1. A structured method for selecting an appropriate metadata schema

2. The design & evaluation of a metadata generator for the origin of an artist

3. The re-evaluation of a general metadata generation approach

RQ 3

RQ 2RQ 2

Ch 8

62

Wednesday 12 September 12

CONTRIBUTIONS.

1. A structured method for selecting an appropriate metadata schema

2. The design & evaluation of a metadata generator for the origin of an artist

3. The re-evaluation of a general metadata generation approach

4. The design & evaluation of federated search tool with social recommendations

RQ 3

RQ 2RQ 2

Ch 8

62

Wednesday 12 September 12

CONTRIBUTIONS.

1. A structured method for selecting an appropriate metadata schema

2. The design & evaluation of a metadata generator for the origin of an artist

3. The re-evaluation of a general metadata generation approach

4. The design & evaluation of federated search tool with social recommendations

RQ 3

RQ 2RQ 2

RQ 1

Ch 8

62

Wednesday 12 September 12

CONTRIBUTIONS.

1. A structured method for selecting an appropriate metadata schema

2. The design & evaluation of a metadata generator for the origin of an artist

3. The re-evaluation of a general metadata generation approach

4. The design & evaluation of federated search tool with social recommendations

5. The design & evaluation of visualisations of attention metadata to support awareness and self-reflection.

RQ 3

RQ 2RQ 2

RQ 1

Ch 8

62

Wednesday 12 September 12

CONTRIBUTIONS.

1. A structured method for selecting an appropriate metadata schema

2. The design & evaluation of a metadata generator for the origin of an artist

3. The re-evaluation of a general metadata generation approach

4. The design & evaluation of federated search tool with social recommendations

5. The design & evaluation of visualisations of attention metadata to support awareness and self-reflection.

RQ 3

RQ 2RQ 2

RQ 1

RQ 4

Ch 8

62

Wednesday 12 September 12

publications.

http://dailypracticeblog.com/how-paper-claims-will-be-impacted-by-5010/ 63

Wednesday 12 September 12

domain steps RQ conf. publication

music

define, create, utilise

(RQ1, RQ3) ISMIR’06 Govaerts, Sten; Corthaut, Nik; Duval, Erik. Moody tunes: the rockanango project

music

create (RQ2) ISMIR’07 Govaerts, Sten; Corthaut, Nik; Duval, Erik. Mood-ex-machina: towards automation of moody tunes

music defineRQ3 ISMIR’08

Corthaut, Nik; Govaerts, Sten; Verbert, Katrien; Duval, Erik. Connecting the dots: music metadata generation, schemas and applications

music define

RQ2 ISMIR’09 Govaerts, Sten; Duval, Erik. A Web-based approach to determine the origin of an artist

music

create

RQ2 ISMIR’09 Govaerts, Sten; Duval, Erik. A Web-based approach to determine the origin of an artist

music

create RQ2 ISM’09-WS

Govaerts, Sten; Corthaut, Nik; Duval, Erik. Using search engine for classification: does it still work?

TEL

capture, create

awareness

RQ4 ICWL’10Govaerts, Sten; Verbert, Katrien; Klerkx, Joris; Duval, Erik. Visualizing activities for self-reflection and awareness

TEL

capture, create

awarenessRQ4 ICWL’11 Govaerts, Sten; Verbert, Katrien; Duval, Erik. Evaluating the

student activity meter: two case studies

TEL

capture, create

awareness

RQ4 CHI’12Govaerts, Sten; Verbert, Katrien, Duval, Erik; Pardo, Abelardo. The Student Activity Meter for Awareness and Self-reflection

TEL

utilise, personalise,

capture

RQ1CSCW’11

-WSGovaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated search and social recommendation widget

TEL

utilise, personalise,

capture RQ1 EC-TEL’11Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning Environments: the ROLE Interoperability framework

64

Wednesday 12 September 12

period project domain RQ steps pubs.

2005 - 2007

Aristo IWT project 1

music RQ2, RQ3

define, create, (utilise)

ISMIR’06, ISMIR’07, ISMIR’08, ISMIR’09, ISM’09-

WS

2007 - 2009

Aristo IWT project 2

music RQ2, RQ3

define, create

ISMIR’06, ISMIR’07, ISMIR’08, ISMIR’09, ISM’09-

WS

2009 - now ROLE TEL

RQ1, RQ4

utilise, personalise, capture, create

awareness

ICWL’10, ICWL’11, ECTEL’11,

CHI’12

10/2010-12/2010

Research stay at EPFL TEL

RQ1, RQ4

utilise, personalise, capture

CSCW’11-WS

65

Wednesday 12 September 12

somemetrics...

66

Wednesday 12 September 12

CITATIONS PER YEAR.

2007 2008 2009 2010 2011 20120

8

15

23

30nu

mbe

r of

cita

tions

year

external citationsself citations

total citations: 72external citations: 49h-index: 6

67

Wednesday 12 September 12

CITATIONS PER YEAR.

2007 2008 2009 2010 2011 20120

8

15

23

30

12

23

8

711

6

23

6

3

num

ber

of c

itatio

ns

year

external citationsself citations

total citations: 72external citations: 49h-index: 6

67

Wednesday 12 September 12

CITATIONS PER PAPER.

external citationsself citations

0

4

8

11

15

num

ber

of c

itatio

ns

ISM

IR 2

006

ISM

IR 2

007

ISM

IR 2

008

ISM

IR 2

009

(1)

ISM

IR 2

009

(2)

adM

IRe

2009

IJET

EL 2

010

ICW

L 20

10

SRS

2011

ECT

EL 2

011

EFEP

LE 2

011

ICW

L 20

11

CH

I 201

2

papers68

Wednesday 12 September 12

CITATIONS PER PAPER.

external citationsself citations

0

4

8

11

15

5

9 4

2

9

2

6

1 1

1

6

4

7 2

32

212 2 1

num

ber

of c

itatio

ns

ISM

IR 2

006

ISM

IR 2

007

ISM

IR 2

008

ISM

IR 2

009

(1)

ISM

IR 2

009

(2)

adM

IRe

2009

IJET

EL 2

010

ICW

L 20

10

SRS

2011

ECT

EL 2

011

EFEP

LE 2

011

ICW

L 20

11

CH

I 201

2

papers68

Wednesday 12 September 12

TIME SPENT.

time

1: schemas 2: origin 2: search 4,5&6: search & recomm. 5&7: awareness

0% 25% 50% 75% 100%

time per RQ

RQ1 RQ2 RQ3 RQ4

69

Wednesday 12 September 12

TIME SPENT.

time 15% 20% 10% 18% 37%

1: schemas 2: origin 2: search 4,5&6: search & recomm. 5&7: awareness

0% 25% 50% 75% 100%

time per RQ 18% 30% 15% 37%

RQ1 RQ2 RQ3 RQ4

69

Wednesday 12 September 12

70

Wednesday 12 September 12

70

Thank you for your attention!Questions?

Wednesday 12 September 12