Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing...
Transcript of Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing...
![Page 1: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/1.jpg)
Latest climate4impact developmentsConnection to CLIPC
Provenance in Processing
CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC, STFC, IPSL
Christian Pagé
Maarten Plieger, Wim Som de Cerff, Ernst de Vreede, Andrej Mihajlovski & Alessandro Spinuso,
Antonio Cofiño & Manuel Vega Saldarriaga,
Lars Barring
Ronald Hutjes & Fokke de Jong,
Sandro Fiore & Alessandro d’ Anca
ESGF Face to Face 2016
Washington, DC
![Page 2: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/2.jpg)
• Platform for researchers to explore climate
data and perform analysis
• Connects to ESGF web services
– Searches ESGF using search API
– Security arranged via certificates and OpenID
– Uses THREDDS Catalogs and OpenDAP
• Uses ADAGUC WMS/WCS
– Web Map Services for visualization
– Web Coverage Services for data transformation
• PyWPS to perform calculations / process data
– ICCLIM climate indices calculation, data
reduction
– Personal store for processing outcomes
• Backend for other portals (CLIPC)
What is climate4impact?
ESGF Face to Face 2016
Washington, DC
![Page 3: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/3.jpg)
• Many iterations with users
• Tooltips with explanations
• Quick select menus
• ES-DOC integration
• Preview of data
• Export to CSV
• Reusable & modular
• Never finished
Improved search UI for climate impact researchers
![Page 4: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/4.jpg)
• WPS DescribeCoverage UI
• Lightweight
• Links to preview
• Links to basket / cart
• Reusable & modular
Improved processing UI for WPS services
previewbasket
title
delete
identifier
abstract value
![Page 5: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/5.jpg)
• Data reduction
• Subsetting
• GIS formats
• Reprojection
• Regridding
Improved wizard for subsetting and regridding WPS
![Page 6: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/6.jpg)
• ADAGUC WMS and WCS supports GeoJSON
• GeoJSON can be rasterized to grids using the Web Coverage Service
• The Web Coverage Service is used in Web Processing Services
GeoJSON support for polygons via WMS/WCS
![Page 7: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/7.jpg)
• Calculates statistics per region
• Mean, min, max, etc ...
WPS for statistics per region
• Outputs CSV table and NetCDF files
• Based on NUTS regions in GeoJSON
Minumum per region
Mean per region
Maximum per region
![Page 8: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/8.jpg)
• CLIPC is aimed at boundary workers, C4I is aimed at climate impact researchers
• CLIPC is directly connected to climate4impact (WMS, CSW, WPS, basket api)
• Uses access tokens for authentication (UUID’s, 10 hours valid)
• Instruction movies: http://www.clipc.eu/getting-started/documentation-and-videos
CLIPC Portal uses Climate4impact web services
User basket at
CLIPC portalUser basket at
Climate4impact portal
![Page 9: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/9.jpg)
PyWPS
Opendap
Inputs
netcdf4
netcdf4
Combine
WPS
nc
norm
1
nc
norm
2
nc
weigh
nc
weigh
nc
out
WCS
comb
Provenance DB
Monitoring and Exploration of WPS workflows via Provenance
Combine WPS, five steps involved Visual analytics techniques on provenance
• Highlighting data-reuse, even for cached data
• User interactions
• Exploitation of resources
NetCDF dependencies
Searchable metadata
• Provenance module: WPS_PROV
• Provenance metadata is stored in NetCDF
• W3C PROV-DM standard
![Page 10: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/10.jpg)
Visualisation of W3C PROV Output from NetCDF
![Page 11: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/11.jpg)
• Displays SVG graphic created from provenance variable in NetCDF
Accessible in Climate4impact portal and CLIPC portal
![Page 12: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/12.jpg)
Climate4impact has currently ~750 registered users
Usage statistics in short:
Access to C4I
Access tokens
Browser session
Certificates
![Page 13: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/13.jpg)
Thanks for listening!!!
ESGF Face to Face 2016
Washington, DC
![Page 14: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/14.jpg)
End
![Page 15: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/15.jpg)
• Combines two maps into a new one: New climate indicators can be made
• Provenance module for WPS: WPS_PROV:– Uses W3C PROV-DM standard for reporting in XML
– Provenance is stored in NetCDF: Standardises attributes and variable as
storage containers of provenance
Provenance in CLIPC WPS combine service
![Page 16: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/16.jpg)
Monitoring and Exploration of WPS workflows via Provenance
Visual analytics techniques on provenance data, highliting data-reuse, users interactions,
exploitation of resources..
NetCDF dependencies
Searchable metadata
WPS Runtime Monitoring
![Page 17: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/17.jpg)
Combine WPS: Five steps involved
Visualisation Tools
• metadata change
• knmi_provenance
variable
– bundle json
– lineage json
– prov-dm w3c (xml)
1. WCS Processes
2. Normalisation Processes
3. Weighing Processes
4. Combination
5. Piping modules
pywps
Opendap Service
Inputs
netcdf4
netcdf4
Combine WPS
nc
norm
1
nc
norm
2
nc
weigh
nc
weigh
nc
out
WCS
comb
Provenance DB
![Page 18: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/18.jpg)
CLIPC Portal connected to climate4impact
Climate4impact backend services CLIPC frontend
CLIPC Infrastructure
MyProxyX509
PyWPS (WPS)
ImpactportalJava, Servlets, JSPBasket, Tokenapi, OpenID, OAuth2,
OpenDap, Front-end
Impact data nodeTHREDDS OpenDAP
ADAGUC Viewer (WMS)
ADAGUCServer (WMS)
PostgreSQL
MySQL
Tomcat
Drupal CMS
Ap
ache H
TTPD
Pro
xy
Climate4impact infrastructure
WMS
WCS
WPS
CSW
OpenDAP
Storage API
(basket)
W3C PROV
Access
token
DiscoveryCatalogue
Openlayers3Visualization
CSWGeonetworks
IS-ENES + CLIPC Processing scripts
ICCLIM ClipCombine … … ……
FTPUpload
![Page 19: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/19.jpg)
Combine architecture: File A + File B = Result
![Page 20: Connection to CLIPC Provenance in Processing · Connection to CLIPC Provenance in Processing CERFACS, KNMI, University of Cantabria, SMHI, Wageningen University & Research, CMCC,](https://reader030.fdocuments.in/reader030/viewer/2022040410/5ecdf08b4a408c2f443d1ca1/html5/thumbnails/20.jpg)
WMS /impactportal/adagucserver/<accesstoken>/?source=<opendapurl>&service=WMS&request=GetCapabilities
WCS /impactportal/adagucserver/<accesstoken>/?source=<opendapurl>&service=WCS&request=GetCapabilities
WPS /impactportal/WPS/<accesstoken>/?service=WMS&request=GetCapabilities
OpenDAP /impactportal/DAP/<accesstoken>/<userid>/<file>
HTTP download /impactportal/DAP/<accesstoken>/<userid>/<file>