Constructing a Federated Data Distribution Service for Universities
-
Upload
safe-software -
Category
Technology
-
view
343 -
download
0
description
Transcript of Constructing a Federated Data Distribution Service for Universities
![Page 1: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/1.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Jakob Lagerstedt, Mats HögströmConstructing a Federated Data Distribution Service for Universities
![Page 2: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/2.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Overview
Background
Previous Attempts
A modern solution
Some tricks learned
Results
![Page 3: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/3.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Background
We are from the
Swedish University of Agricultural Sciences
![Page 4: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/4.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Background
Data and researchers in Sweden
![Page 5: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/5.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Background
Not free
Image courtesy of KJGarbutt (CC BY 2.0)
![Page 6: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/6.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Background
Strange formats
Image courtesy of Toronto Public Library Special Collections (CC BY 2.0)
![Page 7: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/7.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Previous attempts
Distribution was difficultRequired authorization and data preparation
Researchers would share pieces of data on USB drives (sneakernets)
Image from http://japanese.engadget.com/2005/11/12/hdd-circa-1975/
![Page 8: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/8.jpg)
CONNECT. TRANSFORM. AUTOMATE.
A modern solution
Geographic Extraction Tool
![Page 9: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/9.jpg)
CONNECT. TRANSFORM. AUTOMATE.
A modern solution
Geographic Extraction Tool
![Page 10: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/10.jpg)
CONNECT. TRANSFORM. AUTOMATE.
A modern solution
Geographic Extraction Tool
![Page 11: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/11.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade ()
Do not panic
All the code is in the handouts if you are interested!
Image courtesy of Nico Kaiser (CC BY 2.0)
![Page 12: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/12.jpg)
CONNECT. TRANSFORM. AUTOMATE.
FME server Rest API
Token service /fmetoken/service/generate
HTTP get /fmerest/repositories/" + your_fme_repo_name + ".json
Tricks of the trade (Servlet)
Image courtesy of abraham.williams (CC BY 2.0)
![Page 13: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/13.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (Shutdown Python script)
To enable unsupported functions
Zip & Email & Remove files that have been downloaded, Anything you want!
![Page 14: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/14.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (External script example 1 handout)
Use an external script to enable faster debugging
Monkey patch sys or import something from path (yes we actually use the linux version)
![Page 15: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/15.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (SpatiaLite example 2 handout)
Wonderful format for mixed environments
We have Qgis, Arcmap, FME, they all read it!
Image courtesy of juicyverve (CC BY 2.0)
![Page 16: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/16.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (SpatiaLite)
However, FME does not read as efficiently as other formats
![Page 17: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/17.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (SpatiaLite)
does not set spatial index
![Page 18: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/18.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (SpatiaLite)
Can fix
![Page 19: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/19.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (Google Fusion Tables)
Fusion tables are nice for reporting
But often writes fail (we do not know why, reported the issue)
![Page 20: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/20.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (KML network link)
KMLs are nice for reporting
![Page 21: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/21.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Results
23 Universities
100K distributed data sets ( 400 * total area of country)
6000 Unique users
Image courtesy of saamiblog (CC BY 2.0)
![Page 22: Constructing a Federated Data Distribution Service for Universities](https://reader034.fdocuments.in/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/22.jpg)
Thank You!
Questions? Handout link
http://pluto.slu.se/drop/public.php?service=files&t=2e0c13550e0c71e963e6f6bb2c52569c
For more information: jakob.lagerstedt | mats.hogstrom @slu.se Swedish University of Agricultural Sciences
CONNECT. TRANSFORM. AUTOMATE.