GUODA: A Unified Platform for Large-Scale Computational Research on Open-Access Biodiversity Data
-
Upload
matthew-j-collins -
Category
Data & Analytics
-
view
28 -
download
0
Transcript of GUODA: A Unified Platform for Large-Scale Computational Research on Open-Access Biodiversity Data
iDigBio is funded by a grant from the National Science Foundation’s Advancing Digitization of Biodiversity Collections Program. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
GUODA: A Unified Platform for Large-Scale Computational Research on Open-Access Biodiversity DataMatthew Collins, Alexander Thompson, Jorrit Poelen, Jennifer Hammock
2
What is GUODA?Global Unified Open Data Access
An informal collaboration between technologists from organizations like EOL , ePANDDA, and iDigBio as well as independent biodiversity informaticists. We share data use cases, best practices, infrastructure, code, and ideas around the science that can be done by analyzing large open-access biodiversity datasets.
http://guoda.bio
3
What our members are interested inComputation with biodiversity data• Research at scale• Lowering barriers to accessing computation• Reproducibility
Matthew CollinsTechnical Operations
Manager - iDigBio
Jorrit PoelenIndependant
Alexander Thompson Software Products
Lead - iDigBio
Jennifer HammockMarine Theme
Coordinator - EOL
Nathan BirdSoftware
Developer - iDigBio
4
An example use of GUODA
Does anyone use catalog numbers in remarks fields to document relationships between specimen records in iDigBio?
(We’re at TDWG so we’ve got to do something with identifiers, right?)
5
A term-document index of iDigBio(idb_df .select(idb_df["uuid"], idb_df["uuid"]) .where(sql.column("note") != "") .withColumn("tokens", udf_tokenize(sql.column("note"))) .select(sql.explode(sql.column("tokens"))) .groupBy(sql.column("uuid"), sql.column("token")) .count())
6
What terms match catalognumber?joined = (idb_df_ids .join(idb_tf_df, on=idb_df_ids["idb_catalognumber"] == idb_tf_df["token"]) .join(idb_df_notes, on=sql.column("uuid") ==
idb_df_notes["note_uuid"]) .withColumn("catalognumber_len", sql.length(sql.column("idb_catalognumber"))) )
7
What do we find?
A few things like record bd347847…Has a remark Part of Collection at FH: barcode-00374180.
Which matches record 826da57a...
Histogram of matching catalognumber length
8
How long did that take to write?
< 200 lines of code (including whitespace and comments)
1 intermittent day of coding
https://github.com/iDigBio/idb-spark
9
How long did that take to run?73.5 million records in iDigBio to 151 million document:term:counts
40 minutes
Joined back to iDigBio resulting in 2.9 billion terms found in the catalognumber field
3 hours 40 minutes
11
Servers!MesosHDFSSparkMarathonDockerCassandra
Infrastructure
Advanced Computing and Information Systems Labhttp://acis.ufl.edu
12
Data is half the toolCopies of whole datasets
• Stored locally• Refreshed automatically
Re-represent datasets in a useful structure for high performance computing - parquet on HDFS:https://github.com/bio-guoda/guoda-datasets
13
Interfaces to GUODA
• Jupyter Notebooks for end-users
• Containers for API and web services
• Persistent storage for application state
• Hangouts calls every 2-4 weeks
14
The front door to GUODA
Notebooks
“Literate Programming”
Comments, code, and outputs all together in a readable document that describes what is being done
17
What would you do with it?Have a Github account and want to write code? This is an alpha quality system.
http://jupyter.idigbio.org
Or talk to us if you want to host an application on our systems
iDigBio is funded by a grant from the National Science Foundation’s Advancing Digitization of Biodiversity Collections Program. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
idigbio.org/wikifacebook.com/iDigBio
twitter.com/iDigBio
vimeo.com/iDigBio
idigbio.org/rss-feed.xml
idigbio.org/events-calendar/export.ics
Get involved!