RDIF@AWI Raw Data Ingest FrameworkRDIF@AWI – Raw Data Ingest Framework AWI Research Platforms...
1
Polar5/6 Devices Neumayer Devices Mya II Devices Internet AWI 26.10.2013, Peter Gerchow, [email protected] RDIF@AWI – Raw Data Ingest Framework AWI Research Platforms BREMERHAVEN Am Handelshafen 12 27570 Bremerhaven Telefon 0471 4831-0 www.awi.de Temp. Data Exchange (/hs/datex) PANGAEA Archive (/hs/usero) Centralized near Realtime Data Metadata Generation /hs/usero/polarstern/fishsounder/ANT24_2/ ANT24_2-D20090711-T103035.raw ANT24_2-D20090711-T104035.raw ANT24_2-D20090711-T105035.raw … ANT24_2-D20090711-T170035.raw ANT24_2-D20090711-T171035.raw ANT24_2-D20090711-T172035.raw Mastertrack (DShip) Documentation / Manuals • Summary Description • Data Storage and Archiving • Device and System Responsibles • DShip – Data Acquisition System AWI internal File based Data Access /hs/usero/polarstern/fishsounder/ANT24_2/ ANT24_2-D20090711-T103035.raw ANT24_2-D20090711-T104035.raw ANT24_2-D20090711-T105035.raw … ANT24_2-D20090711-T170035.raw ANT24_2-D20090711-T171035.raw ANT24_2-D20090711-T172035.raw … MAC using smb://hssrv1/usero … PC Using \\hssrv1\usero access„readonly“ Web based Internet Access Simple Pre Processing of Data Access Protection by Web Application Centralized offline Data Transport Heincke Devices Polarstern Devices Shipborne Devices DShip-System with Stationbook Multibeam Echosounder Ferrybox … Sediment Echosounder Weather Station TSG (2x) External Devices CTD ADCP Navigation System OFOS … Raw Data Storage with Tape Library Data Storage with Tape Library Dship (/hs/dms) Projects (/hs/bsys) (/hs/csys) (/hs/gsys) (…) Universal Archive (/hs/usera) PANGAEA Archive (/hs/usero) Access Protection by Filesystem Metadata Generation File Structure Validation Peter Gerchow Data Access for Validation and Post Processing v2
Transcript of RDIF@AWI Raw Data Ingest FrameworkRDIF@AWI – Raw Data Ingest Framework AWI Research Platforms...
Polar5/6 Devices
Neumayer Devices
Mya II Devices
Internet
AWI
26.10.2013, Peter Gerchow, [email protected]
RDIF@AWI – Raw Data Ingest Framework AWI Research Platforms
BREMERHAVEN
Am Handelshafen 12
27570 Bremerhaven
Telefon 0471 4831-0
www.awi.de
Temp. Data Exchange (/hs/datex)
PANGAEA Archive
(/hs/usero)
Centralized near Realtime Data
Metadata Generation
/hs/usero/polarstern/fishsounder/ANT24_2/
ANT24_2-D20090711-T103035.raw
ANT24_2-D20090711-T104035.raw
ANT24_2-D20090711-T105035.raw
…
ANT24_2-D20090711-T170035.raw
ANT24_2-D20090711-T171035.raw
ANT24_2-D20090711-T172035.raw
Mastertrack (DShip)
Documentation / Manuals
• Summary Description
• Data Storage and Archiving
• Device and System Responsibles
• DShip – Data Acquisition System
AWI internal File based Data Access
/hs/usero/polarstern/fishsounder/ANT24_2/
ANT24_2-D20090711-T103035.raw
ANT24_2-D20090711-T104035.raw
ANT24_2-D20090711-T105035.raw
…
ANT24_2-D20090711-T170035.raw
ANT24_2-D20090711-T171035.raw
ANT24_2-D20090711-T172035.raw
… MAC using smb://hssrv1/usero
… PC Using \\hssrv1\usero
access„readonly“
Web based Internet Access
Simple Pre Processing of
Data
Access Protection by
Web Application
Centralized offline Data Transport
Heincke Devices
Polarstern Devices
Shipborne Devices
DShip-System with Stationbook
Multibeam Echosounder
Ferr
ybo
x …
Sediment Echosounder
Wea
ther
Sta
tio
n
TSG
(2
x)
External Devices
CTD ADCP
Nav
igat
ion
Sys
tem
OFOS
… Raw Data Storage with Tape Library
Data Storage with Tape Library
Dship (/hs/dms)
Projects (/hs/bsys) (/hs/csys) (/hs/gsys)
(…)
Universal Archive (/hs/usera)
PANGAEA Archive (/hs/usero)
Access Protection by
Filesystem
Metadata Generation
File Structure Validation
Peter Gerchow
Data Access for Validation and Post Processing
v2
