Notebooks, reproducibility and other topics in ocean ... · Notebooks, reproducibility and other...

C. Troupin

GHER-University of Liège

Notebooks, reproducibility andother topics in ocean sciences

The material (slides, exercises) are made available through GitHubathttps://github.com/gher-ulg/COST-EUMETSAT-Training

https://github.com/gher-ulg/COST-EUMETSAT-Training

Who knows/uses …?

GitHub

orcID

Who knows/uses …?

GitHub orcID

Problem:how to guarantee reproducibility of

results?

How to go from data to results?

▶ Read the publication?▶ Read the manual?▶ Get and re-use code referenced in publication?


▶ Read the publication?

▶ Read the manual?▶ Get and re-use code referenced in publication?


▶ Read the publication?▶ Read the manual?

▶ Get and re-use code referenced in publication?


▶ Read the publication?▶ Read the manual?▶ Get and re-use code referenced in publication?

Notebooks: interactive computational environments

Notebooks combine:1 code fragments that can be executed,2 text for the description of the application and3 figures illustrating the data or the results.


Notebooks combine:1 code fragments that can be executed,2 text for the description of the application and3 figures illustrating the data or the results.

”Digital Playground”

”Data Story Telling”

”Computational Narratives”


http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261



Interactiveenvironments:

what exists today?

http://rmarkdown.rstudio.com/

Possible to export in journal or presentation formats https://github.com/rstudio/rticles

LATEX templates for different journals

http://rmarkdown.rstudio.com/

https://github.com/rstudio/rticles

https://zeppelin.apache.org/

Languages can be mixed in the same notebook Users can write their own interpreter (language backend)

https://zeppelin.apache.org/

http://jupyter.org/ More than 40 language kernels available Can be used as a multi-user server ( jupyterhub )

http://jupyter.org/

http://beakernotebook.com/

Usage of different languages in different cells,within the same notebook

Installation and multi-language

http://beakernotebook.com/

https://cocalc.com/ ”Collaborative Calculation in the Cloud” Support of many languages Users to upload their files on the platform

https://cocalc.com/

Exercise 1subsetting using nco

Goal: extract a regional subset from fieldTool: nco

NetCDF subsetting

Notebook file: NetCDF-regridding/netCDF-subsetting.ipynbLanguage: bash (not usual)

1 Run the notebook cell-by-cell

2 Create a new notebook File / New Notebook3 Modify the bounding box and perform subsetting4 Repeat the operations on one of your files5 Export the notebook as a pdf file

Exercise 2regridding using nco

Goal: re-interpolateTool: nco + ESMF

NetCDF regridding

Notebook file: NetCDF-regridding/netCDF-regrid.ipynbLanguage: bash

1 Run the notebook cell-by-cell

2 Create a new grid file with a different resolution

3 Perform again the regridding4 Try the regridding on one of your file

use create-netCDF-grid.ipynbif you don’t know how to create such a file

Another step

towards reproducibility

How to identify scientists/researchers?

Let’s work with ORCID

Source: Academicons

http://jpswalsh.github.io/academicons/

How to identify datasets and publications?

Digital object identifier

Ocean Observation Science

Generate Dataset Software tool PublicationAnalysis Results

Source code, notebook

Upload

Login

Upload


Generate Dataset

Software tool PublicationAnalysis Results


Upload

Login

Upload




Upload

Login

Upload

Zeno-what???

Zeno-what???

Zenodo – http://zenodo.org/

A platform to upload papers, datasets, software codes…and to get permanent identifiers

http://zenodo.org/

Zeno-what???After Zenodotus1st superintendent of the Library of Alexandria and 1st critical editor of Homer

By O. Von Corven - Tolzmann, Don Heinrich, Alfred Hessel and Reuben Peiss. The Memory of Mankind. NewCastle, DE: Oak Knoll Press, 2001, Public Domain,https://commons.wikimedia.org/w/index.php?curid=2307486

https://commons.wikimedia.org/w/index.php?curid=2307486




Upload

Login

Upload

A recent (and real) example: MedSea Atlas

The publication? ”Mediterranean Sea Hydrographic Atlas…” 10.5194/essd-2018-9

The author? Sissy Iona (Hellenic Center for Marine Research) 0000-0001-6878-4671

The data? MedSea – T and S observation collection V2 10.12770/8c3bd19b-9687-429c-a232-48b10478581c

The products? MedSea – T and S Annual Climatology 10.5281/zenodo.1146976 via

The method/tool? DIVA Version 4.6.11 10.5281/zenodo.400970 via

Conclusions Suggestions

1 Create an account in ORCID(https://orcid.org/) 2 minutes

2 Create an account in Zenodo(http://zenodo.org/) 2 minutes

3 Publish your code along with your paper 15 minutes4 Make your data public ∞ minutes

https://orcid.org/

http://zenodo.org/




3 Publish your code along with your paper 15 minutes

4 Make your data public ∞ minutes

https://orcid.org/

http://zenodo.org/




3 Publish your code along with your paper 15 minutes4 Make your data public ∞ minutes

https://orcid.org/

http://zenodo.org/

Anything missing?

Working with

in situ observations

”Without sufficient observations,useful prediction will likely never bepossible”

”Models will evolve and improve, but,without data, will be untestable, andobservations not taken today are lostforever.”

C. Wunsch et al. (2010) PNAS

”Without data assimilation, anyattempt to produce reliable forecastsis almost certain to end in failure.”

https://www.metoffice.gov.uk/learning/making-a-forecast/first-steps



How to access the data?Copernicus Marine Environment Monitoring Service (CMEMS)catalog: http://marine.copernicus.eu

http://marine.copernicus.eu

How to access the data?

1 Create user2 Login into the system3 Select the data set of interest from the catalog4 Download from FTP

Exercise 3: find sea watertemperature near Hamburg

using in situ data

In situ temperature

Data: folder INSITU_BAL_NRT_OBSERVATIONS_013_032index_latest.txt: list of files with bounding box, timecoverage etc latest directory: netCDF files

Tool: notebook CMEMS_INSTAC/read_CMEMS_indexfile.ipynbTasks:

1 Read the index file2 Represent the data points on a figure (map)3 Find the closest data point4 Read the temperature at the point

Creating a presentation from your notebook

http://nbviewer.jupyter.org/github/gher-ulg/COST-EUMETSAT-Training/blob/master/CMEMS_INSTAC/slides_insitu.ipynb




How to access the data?In Situ Thematic Assembly Center:http://www.marineinsitu.eu/dashboard/

http://www.marineinsitu.eu/dashboard/

Notebooks, reproducibility and other topics in ocean ... · Notebooks, reproducibility and other...

Documents

Transcript of Notebooks, reproducibility and other topics in ocean ... · Notebooks, reproducibility and other...