KIT – University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association

INSTITUTE OF APPLIED INFORMATICS AND FORMAL DESCRIPTION METHODS (AIFB)

www.kit.edu

On-the-fly Integration of Static and Dynamic Linked Data

Andreas Harth (KIT), Craig Knoblock (USC), Steffen Stadtmüller (KIT), Rudi Studer (KIT), Pedro Szekely (USC)

2 On-the-fly Integration of Static and Dynamic Linked Data

Outline

Motivation

Scenario and Overview

Modelling Sources: Karma

Accessing and Integrating Sources: Data-Fu

Demo

Conclusion

3 On-the-fly Integration of Static and Dynamic Linked Data

Motivation

The relevance of many types of data perishes or degrades over time (e.g., weather information, moving objects)

Timely decision making requires access to live data and

inclusion of new sources in a flexible manner.

Our goals(Near) real-time access to a variety of data sources in a range of data formats and access modalities

Rapidly integrate sources via modeling and to generate a Linked Data interface to live sources

4 On-the-fly Integration of Static and Dynamic Linked Data

Static vs. Dynamic Sources

Various sources have different update intervals (from minutes to weeks)

We treat the access to all sources in the same way via polling (HTTP GETs)

Thus, the only distinction between „static“ and „dynamic“ sources is how fast we refresh the query results for each source

5

Scenario

On-the-fly Integration of Static and Dynamic Linked Data

Marine Vessels (AIS)

Vehicles(Campus Cruisers)

POIs(Crunchbase,

OSM, Wikimapia)

Venues/Events(Eventful, LastFM)

Buses/Stops(LA Metro)

6

Architecture

On-the-fly Integration of Static and Dynamic Linked Data

7 On-the-fly Integration of Static and Dynamic Linked Data

Karma

Interactive tool for rapidly extracting, cleaning, transforming, integrating, and publishing data

Hierarchical Sources

ServicesModel

KarmaTabularSources

Database

…

See http://isi.edu/integration/karma/ for more info and download

8

Modelling Sources with Karma

Karma is a data integration toolMap data onto an ontology to generate Linked Data

Karma extension to enable the on-the-fly lifting of API I/O data according to a pre-defined mapping model

On-the-fly Integration of Static and Dynamic Linked Data

Linked API

Web API

Vehicles(Campus Cruisers)

9 On-the-fly Integration of Static and Dynamic Linked Data

Linked Data Access to Event APIs

LastFM API

Given a lat/lon of a location, return a list of event identifiers

http://km.aifb.kit.edu/services/lastfmwrap/geo.getevents?lat={?lat}&long={?lon}Given an event identifier, return details about the event

http://lastfm.rdfize.com/events/{event-id}

Eventful API

List events given a keyword search term and a date range

http://km.aifb.kit.edu/services/eventfulwrap/search?location={?loc}&date={?date}

Venues/Events(Eventful, LastFM)

10 On-the-fly Integration of Static and Dynamic Linked Data

LastFM Data-Fu Program (I)

Program at http://km.aifb.kit.edu/services/data-fu/lastfm

with input lat/lon (in RDF via HTTP POST)

Rule to search for events at given location:

{ ?p geo:long ?lon . ?p geo:lat ?lat . } => { [] http:mthd http:GET ; http:requestURI <http://km.aifb.kit.edu/services/lastfmwrap/geo.getevents?lat={?lat}&long={?lon}> . } .

“For the input point with lat/long

perform an HTTP GET

at the KIT LastFM Wrapper URIconstructed with the lat/long”

11 On-the-fly Integration of Static and Dynamic Linked Data

LastFM Data-Fu Program (II)

Rule for retrieving information about the found events, including geolocation of event:

Query to return a table with lat/lon and label to transform to KML/Google Earth:

{ ?e rdf:type lode:Event. } => { [] http:mthd http:GET ; http:requestURI ?e . } .

“For every resource of type event

perform an HTTP GETat the resource URI”

:q1 qrl:select ( ?event ?place ?label ?lat ?lon ) ;qrl:where { ?event <http://purl.org/NET/c4dm/event.owl#place> ?place . ?event rdfs:label ?label . ?place geo:lat ?lat . ?place geo:long ?lon .} .

“Output is every entity with latitude, longitude and associated label”

12 On-the-fly Integration of Static and Dynamic Linked Data

Data Source Characteristics

13 On-the-fly Integration of Static and Dynamic Linked Data

Demo

Load http://people.aifb.kit.edu/aha/2013/d3/index.kml into Google Earth

Location of buses and ships are updated

14 On-the-fly Integration of Static and Dynamic Linked Data

Conclusion

System interoperation in distributed environments with Linked Data as interface

Rapid integration of new sources (via Karma models and Data-Fu scripts)

Realtime access to networked data via Data-Fu scripts/programshttp://code.google.com/p/data-fu/

Ability to rapidy integrate new sources via Karma modelshttp://www.isi.edu/integration/karma/

Future workModular organisation of programs

Manipulating resource state (Read-Write Linked Data)

Optimisations for limited bandwidth environments

15 On-the-fly Integration of Static and Dynamic Linked Data

Challenges

Data is provided at different places, by different owners, often over the web (decentralised data publishing)

Data and links are provided in a many different formats/protocolsDevelopers have to gain a deep understanding of every API by reading textual descriptions

Applications (user agents) are supposed to follow links as found during runtime of the application

Developers have to define their desired interaction at design time

Developers have to write individually tailored code to consume services in applications