Kick-off meetingDelft, April 9-11 2014

7FP – SPACEno. 607131

Data management

@ Fast First step in data management – repository

Gerrit Hendriksen

Gerben J. de Boer

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

Preview

tailored data> WCS> WFS> SOS> SOAP

> netCDF-CF- OPeNDAP> ISO SQL– PostGIS> SVN

> GIT> http> ftp

> KML> WMS> WFS

> CSWgraphics of data

standard dataraw

data

catalogueof data

work done on server

work done on client

Exchange, develop

standards

spatio-temporal

standards

database standards

(lab and field data)

ISO standards

whatwherewhenwho whyhow…data URLs

smart phone & tablet usersscientists professionals

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

In relation with WP’s/ Tasks

D4.3D4.3

D4.1D4.1

D4.5D4.5

T5.2T5.2

D4.8D4.8

D3.3D3.3

datamanagementdatamanagement

DisseminationDissemination

D4.8D4.8D4.1D4.1 T5.2T5.2D3.3D3.3

T5.1T5.1

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

Techniques in data management

Version control (SubVersion, GIT,…)

OGC netCDF+ OPeNDAP

OGC CF Conventions

OGC PostGIS

GeoServer

CSW

ncWMS, ADAGUC

Data & tools as-isvia version control

General purposedata storage:syntax standard

Geospatialsemantic standard

Server side OGCtailoring & graphics

Discoverymeta-data catalog

ISO SQL(PostgreSQL

)

geospatial semantics

servlets

ETL

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

Data management systeem

(DAS)

modelling

(DDS)

(DAS)(DAS)(DMS)

Mobile applications

Monitoring Network

EO/RS

EO/RS tools

Version control

DAS – Data Acquisition Sub-system DMS - Data Management Sub-system DDS - The Data Dissemination Sub-system

Catalogueservice

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

Open source and - standards

• Software• PostgreSQL• Geoserver• Geonetwork• Mapnik• (QGIS)• Python• OpenStreetMap

• OGC (exchange of data)

• Semantic vocubularies• CF• WORMS (PESI)

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

Reason

?

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

Quality assurance

Vessel visualisation of location and speed of fish tracks carried out during monitoring.

The red lines show deviation from the protocol (maximum speed between 5-10 km/hour). It seemed that this was due

to input errors.

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

Clear standards

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

Clear standards

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

For FAST

• Data will be available on an opensource platform:– Scientists– Basis of dissemination- and calculation platform

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

First step in generalization

• Repository• Web-based system for version control• Allows a worldwide community of people

to collaborate online• Everyone (with proper credentials) can

add/alter/remove and retrieve data• Nothing can be added/altered/removed

without being noticed• Makes sure that changes are merged

without conflicts

• Bear in mind, it is password controlled, so not fully open

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

Frequently used commands

Commit

Checkout (sparse)

Update

Add/Delete/Modify

Log

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

Checkout (sparse)

Create a local copy (only once) Checkout Depth (Windows)

•Fully recursive•Immediate children, including folders•Only file children•Only this item

svn checkout --depth (Command line)•Infinity•Immediates•Files•Empty

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

Update

Download: Server -> Local copy

Note: update before working on a file!

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

Commit

Upload: Local copy –> server Always add a log message!

Command line: svn commit -m”[insert log message]”

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

Add/delete

• Svn addUsed when a new file or folder is added to the repository, before committing. Otherwise, the file will only stay local

• Svn deleteUsed when a file or folder should be deleted. Otherwise, the file or folder is only deleted locally and will reappear at the following update

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

• Language: English (folder names, file names, code)

• Don’t use spaces in file/folder names, preferably no capitals

• Always add a log message (also in English)

• Update regularly!

conventions

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

In practice

• Install subversion client– Tortoise (http://tortoisesvn.net/)

• Sparse checkout of the repository of you country• Identify some data• Commit data• Checkout the data

• http://repos.deltares.nl/repos/FAST   (mind the capitals)

• temporary username: oegast• temporary password: 0terra

Let’s do it!!!

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

Meta-data for cataloguing: save *.xml

• Think about copyright and ownership already now!• Think about life cycle after end of FAST: shift to library / Datacentre• DOI

KIC

K-O

FF

ME

ET

ING

. Del

ft, 9

-11

Apr

il 20

14

7FP – SPACE

OpenEarth wiki (openearth.eu)