D00001275 Flexible & Interoperable Data Transfer (FIT) Protocol Rev 1.8
Andy Petrella_Med@Scale by Data Fellas: Scalable and Interoperable Genomics data services, what...
23
by Data Fellas, Data Enthusiasts v 4.0 (July, 13th ‘15) Scalable and Interoperable data services Applied to Genomics
-
Upload
dataconomy -
Category
Data & Analytics
-
view
94 -
download
1
Transcript of Andy Petrella_Med@Scale by Data Fellas: Scalable and Interoperable Genomics data services, what...
by Data Fellas, Data Enthusiasts v 4.0 (July, 13th ‘15)
Scalable and Interoperable data servicesApplied to Genomics
Young Belgian Startup
The Data Fellas Startup
Data ScienceXavier Tordoir@xtordoir
Andy Petrella@noootsab
Data Processing
Scalable Machine Learning
Micro Services oriented
Data Fellas EcosystemWe’ve worked with
Data Fellas: EvangelizingTrainingScala
Apache Spark (BE, in September)http://spark4devs.data-fellas.guru/
Distributed Machine Learning
Pipeline (Oakland, August)http://bigdatascala.bythebay.io/training.html
Apache Spark (SFO with BoldRadius, August)
TalksScala IO, Devoxx Belgium, Devoxx France, Scala Days, KTH, KUL, Spark Meetup London, …
more to come (Italy, …)
PMC Member at Strata NYPMC member at DevoxxPMC Member at Foss4G
First: Data ScienceAnalysis
Spark Notebook
First: Data ScienceAnalysis
Production
Project Generator
Mesos / C* / DCOS
First: Data ScienceAnalysis
Production
Distribution
Micro Service / Binary format
Marathon
First: Data ScienceAnalysis
Production
DistributionRendering
SChema for output
GG / D3 …
First: Data ScienceAnalysis
Production
DistributionRendering
Discovery
Service Metadata
SOLR , …
First: Data ScienceAnalysis
Production
DistributionRendering
Discovery
CatalogSpark Notebookusing Services too
First: Data ScienceAnalysis
Production
DistributionRendering
Discovery
Share Analyses
Share Results
Share Datasets
First: Data Science
Project Code Name:
Shar3
Next: Applied TO Genomics
Genomics data is pretty big
● 100,000’s genomes in 2015● 1,000,000’s … ● 100,000,000’s … ● …
Next: Applied TO Genomics
Genomics data is pretty big and of High dimensionality
One genome:○ 3 billions bases (basic DNA component) sequence○ 30 - 60 x coverage for quality○ 10’s to 100’s millions variants (variable bases
from one individual to the next)
Next: Applied TO Genomics
e.g. 1000genomes project:
● 200TB compressed data● organised in files/directories● data formatted following specs in a … PDF
Data and services schemas are required
What we do with genomics data?
Lots of Querying and Learning:
E.G.
● Population structure is a fundamental basis● Querying relationships between genomes and other
biological features
Hey… no one has all data!
Metadata
What we do with genomics data?
Lots of Querying and Learning:
E.G.
● We do some specific Modelling on some data…
Hey… no two serve the same computations!
Service Discovery
Interoperability
So, no one has all data … BUT all should be able to talk…
Interoperability (GA4GH)
Interoperable… Analysis
Production
DistributionRendering
Discovery
Share Analyses
Share Results
Share Datasets
Interoperable & scalable…
GA4GH + Shar3 = Med@Scale
+ ADAM & spark+ In Memory optimization (Tachyon)+ Deployment (e.g. DCOS)
Wrap-UP
Follow us @DataFellas and get notified about our
+ sharing platform at scale: Shar3
+ Google Genomics At Home (^.^): Med@Scale
+ future plans: modules for Trading, Geospatial, other medical data, …
ReferencesAdam: https://github.com/bigdatagenomics/adamBdg-Formats: https://github.com/bigdatagenomics/bdg-formats
GA4GH website: http://genomicsandhealth.org/GA4GH data working group: http://ga4gh.org/
@Spark-Notebook: https://github.com/andypetrella/spark-notebook/
Med-At-Scale: https://github.com/med-at-scale/high-health
Data Fellas: http://data-fellas.guru/ Training: http://spark4devs.data-fellas.guru/
Q/ATHANKS!
