WP3 System Architecture& System Integration

By (Stein) Runar Bergheim

Asplan Viak Internet

PATHS Project Review12th March 2014, Luxembourg

Overview of PresentationFormal tasks & deliverables

Revisiting the PATHS system architecture

What is new in version 2

Challenges encountered and their solutions

Summing up

PATHS Project Review12th March 2014, Luxembourg

WP3 DeliverablesD3.1: Specification of System Architecture Due Month 8, August 2011 [submitted: 11.10.2011, +1 month]

D3.2: First Prototype and Documentation Month 16, April 2012 [submitted: 23.05.2012, +1 month]

D3.3: Second Prototype and Documentation Month 28, April 2013 [submitted: 03.05.2013, ~roughly on time]

PATHS Project Review12th March 2014, Luxembourg

Data

The PATHS System Revisited

PATHS Web API (JSON/XML)

UserInterfrace

DatabaseSolr

IndexAuth

Query /View

Edit /Delete

DataLoader

Cloud

WP1,4

Wp2

PATHS Project Review12th March 2014, Luxembourg

WHAT IS NEW IN V2.0WP3 – System Architecture

PATHS Project Review12th March 2014, Luxembourg

New Features 2nd Prototype

Vocabul-aries,

thesauri

2nduser

interface

Mobileapplicatio

n

Recomm-endation

s

Visual-ization

Geo-location

PATHS Web API (JSON/XML)

Cloud

Existing system

1stuser

interface

DataLoader

Dates Types

PATHS Project Review12th March 2014, Luxembourg

Extended Data Model

...uses URIs as keys

PATHS Project Review12th March 2014, Luxembourg

Database + Solr Index

~1.88 mill items

~19.9 mill b.ground links

~51.1 mill similarity links

~44k multi-hierarchical topics

PostgreSQL

Solr

PATHS Project Review12th March 2014, Luxembourg

System OverviewMain Web APIPostgreSQL,

PostGIS, Apache, IIS, Solr, Tomcat,

C#.NET

Prototype UIHTML5, Javascript,

Python

Recommender methods

Java, Perl, PostgreSQL

Sentiment methodsJavascript

JavaJSON

PATHS Project Review12th March 2014, Luxembourg

PATHS API Web ServicesUsr: Users, auth and behavior

Path: Path and node authoring & retrieval

Item: ESEPaths interaction, background links, similarity links

Topic: topic hierarchies, query

Search: query items, paths

SolrProxy: access to Solr select end-point

PATHS Project Review12th March 2014, Luxembourg

PATHS API Clients2nd Prototype http://explorer.paths-project.eu

iPad Application Link/screenshot

Branching paths, map based visualization, thesaurus browser

Exploration on mobile platform, social media interaction

PATHS Project Review12th March 2014, Luxembourg

CHALLENGES & THEIR SOLUTIONS

System Architecture and Integration

PATHS Project Review12th March 2014, Luxembourg

Parallel vs SequentialParallel execution of dependant WPs Work Package 2, 3 and 4 have parallel activities but sequential

dependencies

Impact, solution, lessons learned Some sub-optimal but necessary technology decisions WP3 is developed using a sub-set of data WP4 is developed assuming, not knowning, how WP3 will work Extra integration work at the end of the process A lot of valuable knowledge about the optimization of different

systems, technologies and service interactions Summarized in D3.3

PATHS Project Review12th March 2014, Luxembourg

Platform IssuesPlatform complexity The platform includes a «scientific» quantity of technologies, various

stages of releases: beta, 0.x versions,

Platform compatibility A component designed for Apache with Python on Linux does NOT

necessarily work the same way on IIS with Python on Windows – notable at migration

Impact, solution, lessons learned Inconvenience for user testing, field trials Extra efforts required for intensive debugging Lessons learned: dependency awareness Learnings summarized in D3.3, D5.3

PATHS Project Review12th March 2014, Luxembourg

Data API vs local dataVirtualized data vs local data layers The overhead of doing multiple parallel/sequential queries over HTTP

are far greater than doing the same towards i.e. a local database or service

Challenging the REST architecture Necessary to make authoring-API session-aware, i.e. authoring is not

done through a RESTful service

Impact, solution, lessons learned An application that is going to use an API needs to be aware at design

time Validating alternative technologies, e.g. MongoDB for multi-hierarchies Valuable knowledge on Web APIs, useful in follow-up of exploitation

plan, documented in several of the presentations on slide-share, publications

PATHS Project Review12th March 2014, Luxembourg

WP3: Summing UpStrong points Enables others to build apps on

the same virtual data layer Demonstrates fully virtualized

system architecture with sophisticated app

Modular architecture allows for re-use of all OR parts of the PATHS system

Open standards with interfaces to mainstream of European GLAM institutions

The use of URIs as keys permits ANY data – not only ESE

Opportunities ahead Many - in various stages of

follow-up, please refer to presentation on Exploitation Plan

Questions & Thanks for the Attention

(Stein) Runar Bergheim

rb@avinet.no