A conceptual model for the annotation of

audiovisual heritage in a media studies context

Liliana Melgar, Marijn Koolen, Jaap Blom, Eva Baaren, Roeland Ordelman

Workshop “Audiovisual Data And Digital Scholarship: Towards Multimodal Literacy”

Digital Humanities

Krakow, 2016

What do “wish lists” and research have in common?

“Digital humanists are

motivated annotators” (Walkowsky & Barker, 2014)

Annotating is one of

the so-called

“scholarly primitives” (Unsworth, 2000)

A (oral) historian’s case

How cultural

activism is used

in the claim for

identity of “x”

ethnic group?

© AAC Library

© southernfoodways.org

...“My focus is on the

representations of the past as

acts of storytelling (or personal

“narratives”)”

The “Bricolage” of qualitative data analysis

Buster Keaton. 'Sherlock Jr.' (1924)

© Tumblr.com

“Moving images are a blind medium”(Sandom & Enser, 2001)

Added challenges of AV media

Analysis of our research use cases

Media

aesthetics

Media

representations

Cross-media

analysis

Social history of

the media

Social and

cultural history

Fan culture

studies

Genre studies

Discursive

representations

(media linguistics)

Visual

representations

Public debates

War and

conflict

Journalism

(data-driven,

narrative)

Postcolonial

studies

Migration

Desmet case

Reality tv

Memories from

the Moroccan-

Berber activists in

NL/FL

Muslims/islam

representations

Amateur

film/video

Youtube

broadcasts

World War IIIndonesian

decolonisation

Public contention

Refugee crisis

RQ:

How to provide

system support

for manual

annotation tasks

of AV media in

the context of

Digital

Humanities

infrastructures?

Requirements

analysis

Research use

cases

Method

Use case / requirements analysis: one to one interviews with media scholars

Literature review

Analysis of the data models, functionalities and interface features of current

tools that support video annotation

CAQDAS VIDEO ANALYSISPROFESSIONAL

VIDEO EDITORS

DOMAIN SPECIFIC

APPLICATIONS

GENERIC WEB AV

ANNOTATION

➔ Transana

➔ Nvivo

➔ Atlas.ti

➔ Anvil

➔ Advene

➔ Elan

➔ Lignes du temp

➔ Final Cut Pro

➔ Adobe’s

Premiere ProCC

➔ Cinemetrics

➔ Linked TV

➔ ArtTube

➔ Youtube

➔ Synote

Preliminary Outcomes

Study of the information

annotating behavior of

media scholars

Analysis of existing tools for

video annotation

A process model

(use case-driven)

AV media-centered data

model

A concept model

Study of the information

annotating behavior of

media scholars

Analysis of existing tools for

video annotation

A process model of

annotation in a research

workflow (use case-

driven)

A media-centered model

of AV annotation

(data-oriented)

A concept -integrative

model of annotation

Preliminary Outcomes

Conceptual

model

Object:

Media

Document

Actor

Annotation

(output)

Process

Task

Motivation

Context

Conceptual

model,

dimensions

Scholar/

(oral)

historian

Thematization

ManualInterpreting

Themes / Narratives

Oral history interviews

Academic research

Current Open

Annotation Data Model

Current Open Annotation Data

Model for scholarly research

The Open Annotation

Collaboration (OAC)

Project (Hunter et al., 2010)

Adds context and agent = Actor

Method, task, and purpose (in stages)

Previous work has identified stages in the

scholarly research process

Bron et al., 2012. “Overview of the phases in the media studies

research cycle with associated search processes and changes in

the research question (RQ). Arrows indicate possible sequences.”

Traditional scholarly

research process in

the previous use

cases include an

“analysis” phase,

which is mostly done

through manual

annotations, named

as:

-Coding (Grounded

theory analysis)

-Thematization

(narratives)

-Content analysis

Previous work has also identified stages in search process

(Huurdeman & Kamps, 2014; Koolen…)

?RQ

-Storytelling

-Reporting

Exploratory data

analysis

Data

preparationData enrichment

Explore (other)

datasets

Research phases & annotation process

Pre-focused annotation➔ Bookmarking

➔ Open coding (initial coding)-

-tagging

➔ Commenting (memos,

tasks)

Focused annotation➔ Focused coding

➔ Classification/

Thematization /

Narrative

➔ Codebook

➔ Commenting

(analytical memos)

Annotation preparation

➔ Defining coding

schemes (layers)

➔ Select data

enrichment services

per layer

Based on studies about research stages (Bron et al., 2015), search stages (Huurdeman & Kamps, 2014, Koolen et al., 2015); Qualitative data analysis theory (Charmaz, 2006); Concept of

annotation (Agosti et al., 2012; Melgar, 2016); CLARIAH use cases (interviews with media scholars) and discussions during CLARIAH WP5 requirement analysis.

Corpus selection Corpus analysis & enrichment

ExplorationContextualization

(Assembling)PresentationAnalysis

A (oral) historian’s case

How cultural

activism is used

in the claim for

identity of “x”

ethnic group?

Data

preparationExplore

(other)

datasets

Data enrichment

Exploratory data

analysis

Focused

annotation

Corpus selection Pre-Focused

annotation

Conclusion and future work

• We need to approach “annotation” in a broader scope

• We have come up with three models for annotation:

• A conceptual model of annotation

• A process model(s)

• A media-centered, data-oriented model

• Future work includes

• Implementation of our model in the CLARIAH services

• Evaluation with media scholars

• New requirements and further development for each

research use case

References

Bron, Marc, Jasmijn van Gorp, and Maarten de Rijke. “Media Studies Research in the Data-Driven Age: How Research Questions

Evolve.” Journal of the Association for Information Science and Technology, 2015, n/a-n/a. doi:10.1002/asi.23458.

Huurdeman, Hugo C., and Jaap Kamps. “From Multistage Information-Seeking Models to Multistage Search Systems.” In Proceedings of

the 5th Information Interaction in Context Symposium, 145–154. IIiX ’14. New York, NY, USA: ACM, 2014.

doi:10.1145/2637002.2637020.

Koolen, Marijn, Toine Bogers, Antal van den Bosch, and Jaap Kamps. “Looking for Books in Social Media: An Analysis of Complex

Search Requests.” In Advances in Information Retrieval - 37th European Conference on IR Research, ECIR 2015, Vienna, Austria,

March 29 - April 2, 2015. Proceedings, edited by Allan Hanbury, Gabriella Kazai, Andreas Rauber, and Norbert Fuhr, 9022:184–

196. Lecture Notes in Computer Science, 2015. doi:10.1007/978-3-319-16354-3_19.

Sandom, Christine, and P.G.B. Enser. “VIRAMI: Visual Information Retrieval for Archival Moving Imagery.” Milano, Italy: Archives &

Museum Informatics, 2001. http://www.archimuse.com/publishing/ichim01_vol1/sandom.pdf.

Unsworth, John. “Scholarly Primitives: What Methods Do Humanities Researchers Have in Common, and How Might Our Tools Reflect

This?” London: King’s Collegue, 2000. http://people.brandeis.edu/~unsworth/Kings.5-00/primitives.html.

Walkowski, Niels-Oliver, and Elton T.E. Barker. “Digital Humanists Are Motivated Annotators.” Laussane, Switzerland, 2014.

http://dharchive.org/paper/DH2014/Paper-296.xml