(1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization...
Transcript of (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization...
![Page 1: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/1.jpg)
PAVICSPower Analytics for Visualization of Climate
Science
Blaise Gauvin St-Denis1, Tom Landry2,David Huard1, David Byrns2,
Diane Chaumont1, Samuel Foucher2
(1) Ouranos, (2) CRIM
CANARIE Research Software Developers Workshop, May 2-3 2017
![Page 2: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/2.jpg)
And all the team members...
Mario Beaulieu
Valérie Bourduas-Crouhen
David Byrns
Claude Chapdelaine
Isabelle Charron
Diane Chaumont
Marianne Corvellec
François-Xavier Derue
Samuel Foucher
Félix Gagnon-Grenier
Blaise Gauvin St-Denis
Renaud Hébert-Legault
David Huard
Mourad Labassi
Tom Landry
Travis Logan
Frédéric Osterrath
Dominique Paquin
Jean-François Rajotte
![Page 3: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/3.jpg)
Presentation plan
- Context, objectives and collaborations
- Platform development
- Current state of the platform
- Challenges of multidisciplinary software development
![Page 4: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/4.jpg)
- Consortium on Regional Climatology and Adaptation to Climate Change
http://www.ouranos.ca
Advanced software modeling and development
Speech and text
Emerging technologies and data science
Vision and Imaging
http://www.crim.ca
- Computer Research Institute of Montreal
![Page 5: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/5.jpg)
Context
- Climate data volume and number of requests towards climate service centers constantly growing
- e.g. one 2d variable on a regional North American grid @ 22 km resolution for a daily simulation from 1950-2100 is roughly 20 gb
- Significant efficiency improvement required- Ouranos and CRIM established a partnership
![Page 6: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/6.jpg)
Planned functionalities (phase I)- Processes : regions extraction, spatial and temporal means, bias
correction, spatial analogs, climate indicators
- Workspaces, access rights and resource sharing
- Connectivity to our local data and ESGF data nodes
- Search engine with support for key-value search
- Traceability of operations (workflows)
- Dynamic visualization of NetCDF files and regions of interest
![Page 7: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/7.jpg)
Adopted standards
- NetCDF
- Climate and Forecast Metadata (CF Conventions)
- GeoJSON, ESRI shapefile
- Open Geospatial Consortium (OGC)
- Earth System Grid Federation (ESGF) API
![Page 8: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/8.jpg)
Collaborations
![Page 9: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/9.jpg)
Birdhouse
- Collection of processes to support climate data analysis- Part of the big COPERNICUS projects- Uses OGC standards (WPS)
- Language: Python
- Operations on climate data through OpenClimateGIS from NOAA (OCGIS)
- Regular video conferences
http://bird-house.github.io/
![Page 10: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/10.jpg)
- Federation in charge of distributing climate data from the large climate model intercomparison projects
- Financing: DOE, NASA, NOAA, NSF, MPI, ANU, IPSL, BADC
- Ouranos/CRIM participated at the Face2Face 2016 conference
- Commitment to contribute to ESGF Compute Working Team (CWT)
on tests for climate data analysis processes
- Tom Landry (CRIM) now member of ESGF Executive Committee (XC)
- Projected CMIP6 (and other MIPs) data volume: 25-50 petabytes from
mid-2017 to the end of 2018
- Video conferences every two weeks for CWT, monthly for XC
![Page 11: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/11.jpg)
Additional initiative: OGC Testbed 13
CRIM is financed in the “Earth Observation Cloud” thread by NRCan (confirmed) and US Department of Energy (pending) to:
- Develop DataCube functionalities (Synthetic Aperture Radar -> Sentinel 2 & Radarsat 2)- Develop hybrid cloud horizontal elasticity (Docker, PyWPS, Kubernetes, OpenStack)- Coordinate with ESA-funded participants for specs and advanced use cases- Integrate tightly in PAVICS platform- Develop a use case for forestry (e.g. land cover mapping)- Align developments with the scientific modelling thread from NASA
Several concerns and use cases shared by CANARIE community! A different way to approach innovation and international funding of projects.
![Page 12: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/12.jpg)
Technologies
![Page 13: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/13.jpg)
“Data” technologies
- THREDDS server for NetCDF data
- Access via OPeNDAP- GeoServer for polygonal
and image data- Solr as catalog for search
criterias- Globus for massive data
transfers
![Page 14: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/14.jpg)
“Processes” technologies
- Web Processing Service (WPS, PyWPS)
- Birdhouse - NOAA OCGIS- Apache Spark, NASA SciSpark- ESGF Compute Working Team API
![Page 15: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/15.jpg)
“Interfaces” technologies
- Node.js / React- OpenLayers- Web Map Service (WMS)- Plotly- yt-project.org- Jupyter notebook
![Page 16: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/16.jpg)
Deployment
![Page 17: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/17.jpg)
https://hub.docker.com/u/pavics/
![Page 18: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/18.jpg)
https://github.com/Ouranosinc
![Page 19: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/19.jpg)
![Page 20: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/20.jpg)
![Page 21: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/21.jpg)
![Page 22: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/22.jpg)
![Page 23: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/23.jpg)
Challenges of multidisciplinary science
![Page 24: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/24.jpg)
A solution to multidisciplinary research obstacles
![Page 25: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/25.jpg)
A natural consequence of specialization
- The fact that individual researchers are increasingly specialized raises the “cost” of interacting with other disciplines;
- Due to these costs, multidisciplinary projects are often run in parallel, with no real dependencies and synergies between teams;
- Open source code has helped tremendously, but there is a large resource gap between installing software and having a working application, especially in earth system modeling.
![Page 26: (1) Ouranos, (2) CRIM Diane Chaumont1, Samuel Foucher2 …PAVICS Power Analytics for Visualization of Climate Science Blaise Gauvin St-Denis1, Tom Landry2, David Huard1, David Byrns2,](https://reader034.fdocuments.in/reader034/viewer/2022051823/5fed946caa8edc7f7b35e5ab/html5/thumbnails/26.jpg)
An interface between disciplines
- Offering specialized services through a standard web interface
(e.g. WPS) could make specialized expertise available to others.- GreatLakesHydrology_WPS(precipitation_ts, temperature_ts) -> net_basin_supply
- GreatLakesRouting_WPS(net_basin_supply, initial_conditions) -> gl_water_levels
- GreatLakesRegulation_WPS(gl_water_levels, ottawa_flow) -> ontario_discharge
PAVICS workspaceWorkflow Editor / Manager
WPS Catalogue
Process provider
PAVICS workspace
Process providerProcess provider