DRAFT

DIVE into the Event-Based Browsing of Linked Historical Media

Victor de Boera,b,∗, Johan Oomena, Oana Inelb, Lora Aroyob, Elco van Staverenc, Werner Helmichd,Dennis de Beursd

aNetherlands Institute for Sound and Vision, Hilversum, the NetherlandsbDept. of Computer Science, VU University Amsterdam, the Netherlands

cDutch National Library, the Hague, the NetherlandsdFrontwise, Utrecht, the Netherlands

Abstract

DIVE is a linked-data digital cultural heritage collection browser. It was developed to provide innovativeaccess to heritage objects from heterogeneous collections, using historical events and narratives as thecontext for searching, browsing and presenting of individual and group of objects. This paper describesthe DIVE Web Demonstrator1. We also discuss how the collection metadata the demonstrator uses areenriched through a hybrid workflow. The demonstrator uses semantics from these enriched collections, theirvocabularies and linked data vocabularies to establish connections between the collection media objectsand the events, people, locations and concepts that are depicted or associated with those objects. Theinnovative interface combines Web technology and theory of interpretation to allow for browsing this networkof data in an intuitive ”infinite” fashion. DIVE focuses to support digital humanities scholars in their onlineexplorations and research questions.

Keywords: Digital History, Heterogeneous Data Cloud, Digital Hermeneutics, Historical Events,Crowdsourcing

1. Background

The Web has offered cultural heritage institu-tions and their public a medium, changing their tra-ditional task from information interpreters to thatof information providers [1] and collections are be-ing made digitally available in increasing numbers.Public repositories such as Europeana and the Digi-tal Public Library of America, for instance, offer ac-cess to tens of millions of digital artifacts from mu-seums, archives and libraries. This urges culturalheritage institutions to rethink the access provisionstrategies to their collections to allow the public tointerpret and contribute to their collections.

Search and browsing interfaces provide access toboth professionals as well as the general public,

∗Corresponding authorEmail addresses: v.de.boer@vu.nl (Victor de Boer),

joomen@beeldengeluid.nl (Johan Oomen),oana.inel@vu.nl (Oana Inel), lora.aroyo@vu.nl (LoraAroyo), elco.vanStaveren@kb.nl (Elco van Staveren),werner@frontwise.com, dennis@frontwise.com (Dennis deBeurs)

searching for cultural heritage objects in either asingle collection or in multiple collections at thesame time. The traditional information access tocultural heritage assumes that experts interpret andcurate their collections in such a way that the usersof their information systems perform simple or com-plex keyword search to come to a selection of itemsmatching the query. However, research has shownthat many users seek more exploratory forms ofbrowsing [2].

Recent technical innovations, which includeLinked Data, make it possible to create interac-tive access to cultural heritage collections not onlythrough direct textual keyword search, but alsothrough structured links between cultural heritageobjects and related events, persons, places and con-cepts. In the Agora project2, browsing of culturalheritage collections through events and their linksto collection object has proved successful for sup-porting the interpretation of end users, and thusrealizing the so-called “digital hermeneutics” [3].

2http://agora.cs.vu.nl

Preprint submitted to Elsevier June 18, 2015

DRAFT

2. The DIVE Demonstrator

Building on the event modelling of the Agoraproject, the DIVE demonstrator presented in thispaper implements the event-based browsing of cul-tural heritage objects from two heterogeneous his-torical collections. Within DIVE, new interactionconcepts for events and event-based narratives havebeen explored and developed. We explicitly supporta diversity of user groups, including Digital Human-ities researchers, professional (commercial) usersand the general public. The DIVE Web demonstra-tor is a linked-data digital cultural heritage collec-tion browser. It was developed to provide innova-tive access to heritage objects from heterogeneouscollections, using historical events and narrativesas the context for searching, browsing and presen-tation of individual objects and groups of objects.DIVE provides a novel way to support digital hu-manities scholars in their online explorations andresearch questions.

The heterogeneous collections made availablethrough the demonstrator are interlinked in a com-mon linked data network. This interconnected net-work of events, persons, places and concepts pro-vides context to the cultural heritage objects, whichare represented in the same networks. Thus, the ob-jects are contextualized with events and narratives,which is crucial for the findability and hermeneu-tics. Core contribution is the innovative user inter-face supporting information interpretation in multi-media collections through dynamic browsing expe-rience with linked data and explicit event represen-tation. DIVE expands on linked media browserssuch as NoTube N-screen3 or HyperTed4 by sup-porting exploration of multiple types of linked me-dia and an event-centric approach. This event-based browsing can also be found in tools such asseen.co5 or Eventify6.

3. The Collection Data

The DIVE demonstrator allows for browsing ofheterogeneous linked datasets as long as they con-tain media objects (such as images or videos) whichare enriched through links with entities such asevents, persons and places. currently, content

3http://nscreen.notu.be/4http://linkedtv.eurecom.fr/Hyperted/5http://seen.co6https://eventify.it/

from two cultural heritage institutions are enriched,linked and made available. We here describe theoriginal data and the enrichment process.

3.1. Data Sources

Figure 1 shows examples of the media objects inthe two collections included in the current versionof the DIVE demonstrator.

• The Netherlands Institute for Sound and Vi-sion (NISV)7 archives Dutch broadcasting con-tent, including television and radio content.A part of the NISV collection of broadcastvideo was published as Open Data on theOpenimages.eu platform8. Within the DIVEproject, a subset of this collection of 100 ran-domly selected videos was ingested through theOAI-PMH protocol9. This collection consistsmainly of Dutch news broadcasts items fromthe period. These videos have a typical dura-tion of between a 1 and 10 minutes. For thesevideos descriptive metadata is available includ-ing free-text content description.

• The Dutch National Library (KB)10 providesaccess to a number of historical datasets. Inthe DIVE demonstrator, we use the KB ANPRadio News Bulletin dataset11. This datasetis made up of digitized typoscripts (radio newsscripts, to be read during news broadcasts)from the period 1937-1984. These have beenmade public through a Web interface and API.Here, the scanned images, OCRed contentand descriptive metadata is available. Theoriginal data and metadata are available inDutch. We ingested 2210 transcripts into theDIVE demonstrator. These were selected fromroughly the same period and topics as theNISV dataset to ensure that links between thecollections could be established.

3.2. Data Conversion and Enrichment

The textual descriptions and descriptive meta-data for both collections are retrieved and are con-verted to RDF. From the textual descriptions we

7http://www.beeldengeluid.nl8http://openimages.eu9We are currently expanding this small subset

10http://www.kb.nl11http://radiobulletins.delpher.nl/

2

DRAFT

Figure 1: Examples of the media objects from the two col-lections: the top shows one typoscript from the KB dataset.Not only the textual content but also the annotations areof interest. The second image shows a video item from theOpenImages collection of NISV with the descriptive meta-data. The items share a mention of a geographical location(the city of Dordrecht).

extract events as well as places, persons and con-cepts linked to those events which in turn are de-picted by the cultural heritage objects (videos ornews bulletins). For this extraction, we employan ensemble of methods. The DIVE system in-corporates a hybrid workflow for event enrichmentin video collections. In this hybrid workflow themachines and the crowd collaborate in the pro-cess of extracting relevant events and event-relatedconcepts in video collections. In the first stageNamed Entity Recognition (NER) and Event ex-traction tools for Dutch text are used in order toretrieve a set of relevant concepts from the videodescriptions. In a second stage, crowdsourcing

through the CrowdTruth platform12 is employed tohave human-recognized entities and to refine the re-sults from Natural Language Processing. Section 4presents in detail this workflow. For the extractionof the News Bulletins, we also use the results of theNER employed by the KB, which is based on theStanford parser, optimized for Dutch texts.

The results from the different tools and thecrowdsourcing are consolidated to RDF. The data ismodeled using the Simple Event Model (SEM) [4].This model allows for the representation of events,actors, locations and temporal descriptions. We ex-tend SEM with other Linked Data schemas, e.g.DC, SKOS, OpenAnnotation and FOAF to repre-sent other types of resources linked to the mediaobjects13. Links are established between the dif-ferent datasets using the Amalgame alignment tool[5]. Amalgame allows for interactive alignment ofterms, using a variety of filtering and (exact andfuzzy) text matching algorithms. In this case, weemployed exact matching with minimal preprocess-ing of text labels. Amalgame allows for the outputof different types of relations (including for exam-ple skos:exactMatch or skos:closeMatch), in thiscase, we opted to include the found correspondencesas owl:sameAs relations14. Links are also estab-lished to external sources, including Wikipedia andDBpedia. The resulting dataset is stored in an RDFTriple store, which provides a SPARQL endpoint15.

Figure 2 shows the general setup of the project,the data ingestion and enrichment pipeline as wellas the interface layers.

4. Hybrid Event Enrichment Workflow

This section presents the hybrid workflow (Fig-ure 3) used for enriching the video collection withevents and event-related concepts. The workflowtakes advantage of both machines output and crowdjudgments in order to support collection interpre-tation and navigation through event-based researchand exploration. The central idea of this approachis harnessing the disagreement among machine andcrowd annotators in order to optimize the extrac-tion of events and their role fillers, mainly from

12http://crowdtruth.org/13The DIVE datamodel is visualized at https://github.

com/biktorrr/dive/blob/master/imgs/dive_model_v3.pdf14For more information, we refer the reader to the Amal-

game website at http://semanticweb.cs.vu.nl/amalgame/15The ClioPatria triple store can be accessed at http://

semanticweb.cs.vu.nl/dive/

3

DRAFT

EXPLORATIVE “DIGITAL SUBMARINE” UI SUPPORT FOR EVENT-BASED COLLEC- TION INTERPRETATION AND NAVIGATION TOUCH-BASED

SEARCHING AND RETRIEVAL OF EVENTS AND OBJECTS THROUGH SPARQL QUERIES SMART IMAGE CACHE PROVIDES VISUAL REPRESENTATIONS OF ENTITIES RETRIEVED FROM COMMONS

DIVE RDFS DATAMODEL BASED ON SIMPLE EVENT MODEL (SEM), OPENANNOTATION (OA) AND SKOS

MEDIA OBJECTS FROM OPEN APIS FROM CULTURAL HERITAGE PARTNERS

USER EXPERIENCE

API

TRIPLE STORE

ENTITY EXTRACTION

DATASETS

OPENIMAGES.EU DELPHER.KB.NL

CROWD-MACHINE PIPELINE FOR EVENT EXTRACTION AND ENTITY LINKING

Figure 2: An overview of the DIVE project and demon-strator: cultural heritage content is collected and enrichedthrough the event generators. The resulting graph represen-tation is stored in a triple store, which is queried by the Webinterface layers.

video descriptions. The entire workflow was devel-oped on the CrowdTruth platform.

4.1. Machine Annotation

Machine annotation is applied at the level of bothvideo content and video description. The videopre-processing consists of keyframes extraction andvideo segmentation through the open source FFm-peg16 multimedia framework. The aim of thesetransformations is to adjust the media item in sucha way that it captures enough event-related fea-tures, while keeping a suitable amount of details tovisualize these features. The keyframes are mainvideo components that intend to depict single vari-ous scenes, people, groups, locations, while the seg-ments cover a broader range of event-related as-pects. Thus, the aim of the keyframes and videosegments is to guide and support the user experi-ence in the event-based browsing.

In order to detect depicted or associated videoconcepts, we process the video description. We

16https://www.ffmpeg.org/

apply a set of six NLP tools to identify thenamed entities (NE) present in the video de-scription: NERD17, TextRazor18, THD19, DBpe-diaSpotlight20, Lupedia21 and SemiTags22. Eventhough some of these extractors are already incor-porated in the NERD framework, we observed thatwhen NERD is used with the so-called combinedstrategy (i.e., it enables to launch all extractorswith a conflict resolution) it might fail in recogniz-ing the correct entity type. Most of the misclassifi-cations appear between Location and Organizationtypes. Each method receives a description as inputand returns for each recognized named entity thefollowing: (i) entity name; (ii) start offset; (iii) endoffset; (iv) entity types; (v) entity resource.

The state-of-the-art NE extractors (for a reviewsee [6]) provide good results, but it is extremelydifficult to find one extractor that performs well onheterogeneous topics such as the TV-news broad-casts from NISV. On the one hand, each extractoruses different algorithms and training data and istargeted on recognizing only some specific namedentities. On the other hand, other extractors arebeing more reliable on different document typessuch as newspapers or medical articles [7]. The ideaof aggregating multiple machine extractors outputhas been developed in [8, 9] as a majority vote strat-egy. However, we build our approach on the dis-agreement notion. By evaluating the disagreementamong multiple extractors we can potentially im-prove our results as an extractor can return: (i)missed entities by other extractors; (ii) alternativeentity spans, where a span consists of a contigu-ous segment of the video description that is beinganalyzed; (iii) missed or misclassified entity typesand resources by other extractors. Furthermore, wedo not remove entities with low confidence or rel-evance scores because these values are most of thetime computed by black-box metrics.

In order to harness the disagreement among themachine annotators, we take each entity that wasextracted by one or multiple machine extractorsand group it based on its DBpedia class in the fol-lowing clusters: Locations, People, Time and OtherConcepts. In the case where we have overlapping

17http://nerd.eurecom.fr/18https://www.textrazor.com/19https://ner.vse.cz/thd/20http://dbpedia-spotlight.github.io/demo/21http://www.old.ontotext.com/lupedia22http://ner.vse.cz/SemiTags/app/index

4

DRAFT

ENTITY EXTRACTION

EVENTS CROWDSOURCING AND LINKING TO CONCEPTS THROUGH

CROWDTRUTH.ORG

SEGMENTATION & KEYFRAMES

LINKING EVENTS AND CONCEPTS TO KEYFRAMES

Figure 3: Hybrid machine-crowd event enrichment workflow

entities spans, i.e., multiple spans for the same en-tity such as ”D. van Staveren” and ”Staveren” wedo not discard any alternative, but keep all of them.For entities where the extractors did not return atype, but they did return a resource URI (eitherfrom DBpedia or Wikipedia) we perform variousqueries to DBpedia in order to retrieve the entityclass. Thus, by harnessing the disagreement be-tween extractors, we reject the notion of majorityvote and consider that any information from any ex-tractor can be right, even if it was a singular choice.Further, the resulted clusters are used in a seriesof crowdsourcing experiments aiming to link theevents with their role fillers (Participants, Location,Time and other Concepts). Moreover, we extractthe timestamps mentioned in the video descriptionsand use the keyframes close to these timestampsin order to link the aforementioned concepts withvisual parts of the video. Keyframes that are ex-tracted around these timestamps have a high prob-ability of depicting the concepts described in thatpart of the description. This is mainly desired forthe explorative browsing of the collection.

4.2. Crowdsourcing

A crowdsourcing task consists of a unit of work,small and clear enough for crowd workers to pro-vide annotations. The crowdsourcing experimentshave as input the integrated output of the machineprocessing, the clusters, as well as the video descrip-tion. Their aim is to use human computing in orderto resolve the issues that machines are still hav-ing problems to clarify, e.g. (i) extracting events,(ii) linking events with their role fillers. Therefore,the first crowdsourcing task aims to extract all the

existing events from the video descriptions. Sec-ond, we focus on linking the extracted events withtheir role fillers - participants, location and timeconcepts.

As mentioned above, the first task (Fig. 4) fo-cuses on determining the events depicted in thevideo descriptions. Thus, the input received bythe crowd contains the text of a video descrip-tion. The crowd is asked to highlight all the wordphrases that could potentially refer to an event.The crowd can verify the selections made and re-move the ones that were mistakenly highlighted.Given this open template, we aim at gathering mul-tiple event-granularities, which would further im-prove the event interpretation. We consider event-granularities the different span alternatives for anevent such as ”toespraak” or ”toespraak tot gen-odigden”. In the current approach we considerevents the longest spans annotated by the crowdworkers (i.e. series of consecutive words that havea high score of being events from the crowd work-ers).

The second task (Fig. 5) focuses on linking theevents from the previous step with the entities clas-sified as participants, location and time from themachines output. The input received by the crowdconsists of two text boxes with the video descrip-tion. The first box contains highlighted events(resulted from the previous crowdsourcing task),while the second box contains highlighted Loca-tion/Participants/Time concepts. The crowd isasked to create links between the highlighted eventsand concepts by clicking the concepts of both textboxes. A link can be removed by clicking on it.

Since at this level we deal with human annota-

5

DRAFT

Figure 4: Crowdsourcing Template for Event Extraction

Figure 5: Crowdsourcing Template for Linking Events with Event-related Concepts

tors, it is extremely important to deal with the an-swers quality. As human annotation is a processof semantic interpretation, often described by thetriangle of reference [10], we have since observedthat disagreement signals low quality and can bemeasured at any corner of this triangle. This is themain concept incorporated in CrowdTruth frame-work and presented through the CrowdTruth met-rics [11]. Since the metrics have been presentedmultiple times [12], we describe here only the adap-tation of the annotation vector that is the mainconcept behind the disagreement metrics.

The result of each worker for the event highlight-ing task is stored in a vector having the length equalto the total number of words in the video descrip-

tion. Thus, in each worker annotation vector wemark all the highlighted words. The annotationvector for each video description represents the ag-gregation of all the worker vectors. For the secondtask, we have a set of events Ei and a set of pos-sible related concepts Cj . The annotation vectorcontains all the possible combinations Ei-Cj .

For both crowdsourcing tasks we have used thefollowing settings: (i) each video description wasannotated by 15 workers; (ii) each worker was al-lowed to perform only a maximum of 10 annota-tions; (iii) only the workers from Dutch-speakingcountries were allowed to provide annotations. Forevaluating and validating the correctness of theCrowdTruth metrics, i.e. the spam workers anno-

6

DRAFT

tations were correctly removed, three people per-formed manual evaluation for 100 workers, equallydistributed between spammers and non-spammers.The performance score given by the evaluators (1-spam, 0-non-spam, 0.5-unclear) was used to com-pute the precision, accuracy, recall and F1 score.The outcome of the evaluation proved that thespam workers were correctly identified given thehigh F1 score and accuracy of 0.9, precision of 0.87and recall of 0.94.

5. User Experience

The DIVE demonstrator is an end-user applica-tion, openly available on the Web. The demonstra-tor’s UI is designed for two user groups. The pri-mary user group is that of cultural heritage domainresearchers, to browse the integrated collections inan explorative fashion. However, the demonstratoris set up in an intuitive way, allowing the generalpublic to navigate the integrated graph in withoutthe need of academic prior knowledge. It uses rea-soning over the graph to display related items to theone that the user is focusing on. Specifically, prop-erties indicating specific relations between entitiesare modeled as sub-properties of a more generic “isrelated to” property. Similarly, specific entity typesare modeled as sub-classes of a more generic en-tity class. These generic properties and classes areused in the queries that the UI employs. RDFS rea-soning ensures that the specific related entities areretrieved. Furthermore, transitivity of relations isused to display entities that are related indirectlyand in the interface module, a number of relationpatterns are defined that determine which (directlyor indirectly related) entities are shown. These pat-terns are implemented in the SPARQL queries usedby the interface module. This goes well beyondstandard information retrieval. The event-relatedfilters allow to identify groups of objects depict-ing or associated to specific events, related people,locations and times. Additionally, we project allbrowsing results on an interactive timeline, provid-ing alternative filtering and access to the relatedobjects.

5.1. User Experience Design

5.1.1. Design principles

The core of digital hermeneutics is formed by twocomponents: object-event relationships and event-event relationships. By making explicit relation-ships between the objects and events as well as

between the events themselves we can facilitateusers in their processes of accessing and interpret-ing objects in online cultural heritage collections.In DIVE we aim at implementing those relationsin an intuitive event-centric browsing interface forbrowsing cultural heritage collections by means ofunderlying linked data graph.

Considerable effort was put in creating an inter-face with a clear identity and an engaging user ex-perience that invite users to continue exploring atdifferent levels of detail. Users become explorersdiving deeper into the data, like a diver “deeperand deeper into an ocean trench discovering newspecies”. This metaphor makes the interface a “dig-ital submarine”, which provides navigation controlsas well as supportive and manipulative tools. Thedesign of the interface also forms an innovative “in-finite exploration path”, which unlocks the poten-tial of touch-based explorative user interfaces.

5.1.2. Design process

The demonstrator’s User Interface is the resultof a careful design process in which web and in-teraction design experts, data experts, humanitiesresearchers and end users collaborated. It is bothvisually attractive as well as functional and makesuse of the visual nature of much of the media ob-jects as well as the extracted knowledge. By makingexplicit relationships between the collection objectsand events, and their related properties, e.g. peo-ple involved, locations and times we can facilitateusers in their access and interpretation processes ofobjects in online cultural heritage collections.

The extensive design phase in which multipleconcepts have been tested resulted in the DIVE“infinity browsing” interface, a combination of twocore interaction concepts that involve a horizontallevel supporting the breadth and a vertical levelsupporting the depth of information explorationand interpretation.

5.1.3. Horizontal browsing

The horizontal level displays the result set of ob-jects related to the seed keyword search in a dy-namic presentation. At this level, user’s explorationis supported by event-centric filters making explicitthe relation of each object to either the depictedand associate events and their properties, e.g. peo-ple, locations and times involved in the events.Consistent color coding is used for each propertytype to allow for a quick discovery of desired typeof objects. Large result sets are represented as a

7

DRAFT

colored barcode as an overview of the amount andcomposition of event properties in the search re-sult. Objects are represented by type-color, type-icon, title and an image and associated with a setof buttons providing detailed information for eachobject, e.g. description, source and external links.To allow for active user engagement and sharingof personal perspectives and interpretations, userscan add comments to each object, share entitiesand save them in private or shared collections. Ad-ditionally, we provide a set of related objects fromthe Europeana collections23.

Typical interactions at this level are:

• Pinch or scroll the elements of the color bar-code zooms in on the objects to reveal moreinformation, e.g. image, title, icon of the event-related property (for example, an icon for lo-cation indicates that this object depicts a lo-cation of the related event).

• Drag right or left the row of related objects re-veals previous or next object on the horizontallevel.

• Arrows are used for navigating to previous ornext objects in the row.

• Search option & event filters allow to show sub-sets of related objects.

5.1.4. Vertical browsing

The vertical level is formed by the user explo-ration history, as a path of selections on the hor-izontal level - leading to the last selected object.Each selection of an object results in a new rowwith related entities loaded under the selected ob-ject. Users can scroll back to a previous step, zoomout, choose another object and build a new pathfrom there. This allows for fluid dynamics in col-lection exploration, discovery of alternative paths,and ultimately supports deep interpretation of cul-tural heritage collections. Current work includesadding the option to save exploration histories asa collection so that users can revisit or share theirbrowsing experiences.

5.2. Implementation Details

The interface is developed in an iterative inte-grated process of design and development. It’s mainbuilding blocks are made in PHP and Symfony2 onthe server side, and HTML5, Javascript and CSS3

23http://europeana.eu/

on the client-side. A number of libraries are used toprovide specific functionality: jQuery handles themajor part of the functionalities like DOM interac-tion and manipulation, event handling and AJAX.Velocity.js is used to improve the performance ofanimations. Hammer.js supports the handling oftouch events. Moment.js makes dates manageable.

5.2.1. Optimization and Speed

As the handling and displaying of large amountsof information can be near the feasible limits ofthe web browsers, optimizations have been madeto keep the interface fast and responsive. On theserver, query results are cached and served with thecorrect cache headers by the API. Data is servedwith the correct cache headers and gzip compres-sion. On the interface side the queuing and serialprocessing of data calls contributes to fast first re-sponses and gradual loading. Queued calls can beremoved before being triggered, in case user inter-action (e.g. selecting a different row) makes themobsolete. Other examples of interface optimizationsinclude the gradual buildup of DOM elements24 andlazyloading of images, stepwise loading and publish-ing of deeper relation data and eventually limitingvisual effects and animations for large collections.When loading the related data, many synchronouscalls are made to the API25. the On programminglevel the use of Prototypes improves object creationspeed for large amounts of entity and row objects.These efforts results in a smooth and fast browsingexperience on the desktop, and makes the interfaceaccessible by tablets.

The interface acquires data from the data layerusing the triple store’s SPARQL API. Severalqueries are used to search entities by keyword, getrelated entities (events, persons, etc.) and get en-tity details. Key focus of the queries is to returnthe right data, within an acceptable time frame.owl:sameAs relations are included in the query toinclude entities that are not yet combined is a su-perclass. With the current dataset/server combina-tion, typical query speed measures about 2000ms

24Experiments showed that adding DOM elements onlywhen required reduces the average DOM/class build timefor an entity from over 10ms to around 4ms.

25By saving/exiting a session, the session lock is releasedand multiple calls can be handled much faster by the server.This lead to a significant reduction of response times. For ex-ample, when retrieving indirectly related entities for “Rotter-dam”, 1926 entities are found, with the session exit strategy,the response time reduces from around 14000ms to 4500ms.

8

DRAFT

for the keywords based text search query, 300msfor entity details, and between 200ms and 2000msfor related entities26. The last range is heavily re-lated to the amount of related entities. The return-ing data is handled by an adapter that maps thedatafields to an internal format which is used tobuild the interface representations. This approachrelieves the clients of unnecessary data parsing andcontributes to compatibility with other data sets.

5.2.2. Smart Image Cache

To provide a visual representation for other enti-ties, the DIVE frontend includes a “smart imagecache”. This module provides an image lookupbased on keywords associated with a DIVE entityand also caches the response because of the po-tentially long lookup times involved. We considerthis cache to be “smart” in the sense that inter-nally it is implemented as calling a specific imagefile, but the module does a distributed lookup in anumber of Open image APIs . Specifically, imagesare retrieved from the five most relevant Wikipediasearches using the Wikipedia API27. If no imagesare found, another query is made to the OpenCul-tuurData API28 which covers an extensive set ofDutch open heritage- and cultural data. This lat-ter API was very useful for our domain specificallyas it provided a lot of news related images thatmatch the old videos and radio news bulletins inthe DIVE data set. The quality and availability ofimages through this system is acceptable and pro-vides a powerful way of filling in the (visual) datagaps. These images increase the user experienceby supporting the visual navigation through the in-terface and rememberability and recognizability ofindividual entities.

5.2.3. Tablet-first

Figure 6 shows the current version of the in-terface, optimized for tablets and modern webbrowsers. The demonstrator UI is designed specifi-cally for multiple platforms and features a respon-sive design. In fact, the UI was designed witha “tablet-first” principle, where touch-actions andgestures are primarily considered for the browsing

26Live examples of these three types of queriesare maintained at http://tinyurl.com/dive-query1,http://tinyurl.com/dive-query2 and http://tinyurl.

com/dive-query3 respectively27http://www.mediawiki.org/wiki/API:Main_page28http://www.opencultuurdata.nl/api/

of the graph. Among the design features specif-ically made for touch-screen devices are 1) largetouch areas ; 2) pinch and swipe navigation for re-sizing the result set or zooming in on the specificmedia object and browsing horizontally and ver-tically through the graph; 3) speed optimizationssuch as asynchronous loading, code optimizations,limited effects to keep performance up on (generallyslower) touch devices.

6. Current Status and Future Work

6.1. Current Status

The DIVE demonstrator has been published athttp://dive.beeldengeluid.nl. We are contin-ually updating both the data as well as the user in-terface. Currently, a subset of data from the two in-stitutions is available in the datastore and throughthe interface. We are scaling this up to more datafrom the two institutions as it becomes available, aswell as (Dutch) historical data from other sources.

6.2. Current Work

In the context of the DIVE+ project29, we are ex-tending the DIVE demonstrator in four ways. First,we extend the DIVE data with two new collections(i.e. Amsterdam Museum30 and Tropenmuseum31)as well as links to external linked open datasets,such as DBpedia, AAT and ULAN.

Second, we aim at extending the user interfacewith an intuitive way to deal with event narratives.

Third, we extend the crowdsourcing pipeline dis-cussed in Section 4 for historical events extrac-tion to also image annotation. Through extensionson the CrowdTruth platform, we will be able todeal with diversity of interpretation and perspec-tives through a joint pipeline for machine and hu-man processing of media and provides a deep anal-ysis of the quality of the crowdsourced data. Itwill also contribute to mapping various “narratives”and identification of “perspectives” associated withthem, to advance research on data representation,structuring and categorizations.

Finally, we propose a set of user studies to testthe usability of the approach with a variety of end

29DIVE+ is part of the Netherlands eScience Center pro-gram for enabling scientific discovery for digital humanities(http://www.esciencecenter.nl/).

30http://www.amsterdammuseum.nl31http://tropenmuseum.nl

9

DRAFT

Figure 6: Screenshot of the interface. This page shows a person (Berlage), with two related events and KB media objectsassociated with them, as well as a related place and person. In the bottom, a timeline is also shown.

users. This will include Information Retrieval-typeuser evaluations, with a specific aim at measuringthe success of explorative search. In this way we willbe able to gain insights on the scalability, robust-ness and reusability of the DIVE digital hermeneu-tics approach, as well as its usability for history re-searchers and general audience. We will also log us-age by members of the general public as the demon-strator will be made available as part of the websitesof NISV and KB.

The aim is therefore to advance the way in whichhistory researchers and general audience interactwith digital heritage collections.

Acknowledgements

This work is partially supported by NWOCATCH program (http://www.nwo.nl/catch)and the Dutch Ministry of Culture. The authorswould like to thank Theo van Veen and Olaf Janssenfor their help with the KB data.

References

[1] K. Mueller, Museums and virtuality, in: In R. Parry,editor, Museums in a Digital Age, chapter 30, pages295-305 Routledge, 2007.

[2] M. v. Erp, J. Oomen, R. Segers, C. v. d. Akker,L. Aroyo, G. Jacobs, S. Legene, L. v. d. Meij, J. v.Ossenbruggen, G. Schreiber, Automatic heritage meta-data enrichment with historic events, in: Museums andthe Web 2011, 2011.

[3] C. van den Akker, S. Legene, M. van Erp, L. Aroyo,R. Segers, L. van der Meij, J. van Ossenbruggen,

G. Schreiber, B. Wielinga, J. Oomen, , G. Jacobs, Dig-ital hermeneutics: Agora and the online understandingof cultural heritage, in: Proc. of the 3rd InternationalWeb Science Conference (WebSci ’11). ACM, New York,NY, USA, 7 pages., 2011.

[4] W. R. van Hage, V. Malaise, R. H. Segers, L. Hollink,G. Schreiber, Design and use of the simple event model(sem), Web Semantics: Science, Services and Agentson the World Wide Web 9 (2).URL http://www.websemanticsjournal.org/index.

php/ps/article/view/190

[5] J. van Ossenbruggen, M. Hildebrand, V. de Boer,Interactive vocabulary alignment, in: Research andAdvanced Technology for Digital Libraries - Interna-tional Conference on Theory and Practice of DigitalLibraries, TPDL 2011, Berlin, Germany, September26-28, 2011. Proceedings, 2011, pp. 296–307. doi:

10.1007/978-3-642-24469-8_31.[6] A. Gangemi, A comparison of knowledge extraction

tools for the semantic web, in: The Semantic Web: Se-mantics and Big Data, Springer, 2013, pp. 351–366.

[7] G. Rizzo, M. van Erp, R. Troncy, Benchmarking theextraction and disambiguation of named entities on thesemantic web, in: 9th International Conference on Lan-guage Resources and Evaluation (LREC’14), 2014.

[8] L. Chen, S. Ortona, G. Orsi, M. Benedikt, Aggregatingsemantic annotators, Proceedings of the VLDB Endow-ment 6 (13) (2013) 1486–1497.

[9] S. Hellmann, J. Lehmann, S. Auer, M. Brummer, Inte-grating nlp using linked data, in: The Semantic Web–ISWC 2013, Springer, 2013, pp. 98–113.

[10] L. Aroyo, C. Welty, Truth is a lie: 7 myths about humanannotation, AI Magazine, 2015.

[11] L. Aroyo, C. Welty, The Three Sides of CrowdTruth,Journal of Human Computation 1 (2014) 31–34.

[12] O. Inel, et al., Crowdtruth: Machine-human computa-tion framework for harnessing disagreement in gather-ing annotated data, in: The Semantic Web–ISWC 2014,Springer, 2014, pp. 486–504.

10