SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant...
Transcript of SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant...
![Page 1: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/1.jpg)
SmartMet Server Providing INSPIRE Compliant MetOcean Data
Roope Tervo, Mikko Rauhala, Mikko Visa, Mika Heiskanen Finnish Meteorological Institute
![Page 2: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/2.jpg)
• Finnish Meteorological Institute opened its data in 2013.
• Basically everything that FMI has property rights was opened.
• Both (near) real-time and historical and climatological data.
• Data is provided in freely in machine readable format.
05/09/17 2
FMI Open Data
https://en.ilmatieteenlaitos.fi/open-data
SmartMet Server Providing INSPIRE Compliant MetOcean Data
![Page 3: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/3.jpg)
FMI Open Data Portal follows INSPIRE requirements.
FMI Open Data Portal
Meta data Services
The very same data portal works as Open Data and INSPIRE portal.
05/09/17 3
ISO19115 WFS WMS CSW
Grid Series Observations
Time Series Observations
Data Models O&M
Simple Feature
GRIB NetCDF GeoTiff
SmartMet Server Providing INSPIRE Compliant MetOcean Data
![Page 4: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/4.jpg)
View Service (WMS) • Based on GeoServer • Only the most common layers
published • Provides a quick view to the
data
05/09/17 4 SmartMet Server Providing INSPIRE Compliant MetOcean Data
![Page 5: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/5.jpg)
Catalog Service (CSW)
• Based on GeoNetwork • Provides high level metadata
05/09/17 5 SmartMet Server Providing INSPIRE Compliant MetOcean Data
![Page 6: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/6.jpg)
Download Service (WFS 2.0)
• Web Feature Service (WFS) 2.0 Simple Profile
• Based on stored queries • Predefined data sets with
possibility for additional parameters (i.e. time and area)
• Based on SmartMet Server
05/09/17 6 SmartMet Server Providing INSPIRE Compliant MetOcean Data
![Page 7: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/7.jpg)
In a Nutshell • Data and product server for MetOcean data • High capacity & availability
• FMI installation handles over 30 000 000 requests each day • Data is extracted and products generated on-demand • INSPIRE Compliant • Operative since 2008
• FMI client services (since 2008) • Finnish Meteorological Institute (FMI) Open Data Portal (since 2013) • Going to be used at Copernicus C3S Climate Data Store (ECMWF)
• Open source
05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 7
![Page 8: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/8.jpg)
In a Nutshell • Several input sources
• GRIB-, NetCDF-, etc. files (multi-dimensional grid data)
• PostGIS database (vectors)
• Point database (point observations)
• Several output interfaces and formats • WMS, WFS 2.0
• JSON, XML, ASCII, HTML, SERIAL
• GRIB1, GRIB2, NetCDF
05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 8
![Page 9: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/9.jpg)
Usage
05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 9
FMI Open Data Portal & FMI INSPIRE Data Services
Backend for clients’ web services
Integration to clients’ systems Backend for
mobile applications
Backend for FMI Client Services
Backend for FMI public pages
• Basis of most FMI product generation
![Page 10: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/10.jpg)
• Published in 2016 in GitHub • https://github.com/fmidev/smartmet-server
• https://hub.docker.com/u/fmidev/ • MIT Licence • Documentation in GitHub
• FMI host the development • Small contributions with pull requests
• In larger contributions, implementation plan is recommended (in GitHub wiki)
• CLA (Contributor Licence Agreement) is required
05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 10
Open Source
![Page 11: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/11.jpg)
Architecture
05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 11
• Frontend • Load balancer • Knowledge about
backend services • Backend
• Different backends may contain different services
• Plugin-based architecture • Engines provide shared
access to the data • Plugins provide services
(APIs) built upon engines
![Page 12: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/12.jpg)
Most Important Components
05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 12
• Frontend • Provides HTTP 1.1 server • Monitors status of backend services and provides
load balancing • Provides LRU product cache
• Data Engines (providing C++ API) • Querydata engine provides access to the grid data • Observation engine provide access to the point data
in database • Geonames and gis engines provide geolocation
information
![Page 13: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/13.jpg)
Most Important Components
05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 13
• Plugins (providing HTTP API) • WMS: Generates SVG images from grid
data on-demand, which are rendered to requested raster format
• WFS: Point data output for grid data and observations
• Timeseries: Custom point data interface with support for aggregate values over time and area
• Download / WCS: Grid data output
![Page 14: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/14.jpg)
Post-Processing Capabilities
05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 14
• Corrects the data based on accurate DEM (up to 30 meter resolution) and land/water information
• Calculates derivative parameters
• Support for aggregate values over time and area
![Page 15: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/15.jpg)
Producing INSPIRE Data Products Observations
05/09/17 15 SmartMet Server Providing INSPIRE Compliant MetOcean Data
Database SmartMet
Server obsengine
SmartMet Server WFS
![Page 16: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/16.jpg)
Producing INSPIRE Data Products Point Forecasts
05/09/17 16 SmartMet Server Providing INSPIRE Compliant MetOcean Data
File System
SmartMet Server qengine
Memory mapped data.
Server provides logic for
interpolation of data for
requested area and time.
SmartMet Server WFS
![Page 17: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/17.jpg)
Producing INSPIRE Data Products Grid Forecasts 1/2
File System
SmartMet Server qengine
Server provides relevant meta
data to fetch the data content
SmartMet Server WFS
05/09/17 17 SmartMet Server Providing INSPIRE Compliant MetOcean Data
![Page 18: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/18.jpg)
Producing INSPIRE Data Products Grid Forecasts 2/2
File System
SmartMet Server qengine
SmartMet Server Download
Memory mapped data.
Server provides logic for
interpolation of data for
requested area and time.
05/09/17 18 SmartMet Server Providing INSPIRE Compliant MetOcean Data
![Page 19: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/19.jpg)
FMI Setup In 2017
05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 19
• 2 frontends • RAM: 256G
• CPU: 24x 2.10GHz
• OS: RHEL7 • 7 backends
• RAM: 12G
• CPU: 24x2.50GHz
• OS: RHEL7 • Load Balancer
• F5 BIG IP 11
![Page 20: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/20.jpg)
WFS 140 ms/req WMS 130 ms/req Timeseries 30 ms/req Autocomplete 4 ms/req
Performance Production (FMI Setup)
05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 20
• 30 – 80 M req/day • Baseline 200 req/s • Peaks over 650 req/s
Average response times Typical load
![Page 21: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/21.jpg)
Performance Load Tests (with 5 servers)
05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 21
• Scenario based on operative use at FMI • Peaks over 4300 req/s • Avg 173 ms, 95% of responses in 244 ms, median 54 ms • Possibly heavy data requests require QoS management
• Independent queues for slow and fast queries
![Page 22: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/22.jpg)
Roadmap
05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 22
Native GRIB and NetCDF support for input data
Support for GRIB and NetCDF data as input data without converting data to internal data format
Clustering support over Internet
Possibility to provide data from it’s original source via single API (bring users to data)
![Page 23: SmartMet Server - INSPIRE · PDF file05/09/17 SmartMet Server Providing INSPIRE Compliant MetOcean Data 6 . In a ... (providing HTTP API) ... • Peaks over 650 req/s](https://reader031.fdocuments.in/reader031/viewer/2022022500/5aa409847f8b9a185d8b6d87/html5/thumbnails/23.jpg)
www.fmi.fi
https://github.com/fmidev https://hub.docker.com/u/fmidev/
https://en.ilmatieteenlaitos.fi/open-data
http://roopetervo.com http://www.slideshare.net/tervo