Collaborative data-driven...

Collaborative data-driven science


Mike Rippin


Background and History of SciServer

Major Objectives

Current System

SciServer Compute – Now

SciServer Compute – Future

Q&A

2


4

“The Project aims to create a sustainable collaborative ecosystem built around several large scientific data sets for the broader science community, based upon the expertise developed for the Sloan Digital Sky Survey (SDSS) SkyServer and associated projects.”


5

NSF Cooperative Agreement

5 years duration, just completed first 3

Development of Cyberinfrastructure

Science Driven


6

Alex Szalay - PI Mike Rippin – PM Ani Thakar, Gerard Lemson, Jordan Raddick,

Bonnie Souter – Associate Directors (Team Leads)

Technical Team: Dmitry Medvedev, Manuchehr Taghizadeh-Popp, Jai Won Kim, Sue Werner, Victor Paul, Jan Vandenberg, Lance Joseph, Alainna White, Laszlo Dobos


Started with the SDSS SkyServer

Goal: instant access to rich content

Idea: bring the analysis to the data

Interactive access at the core

7


Interactive science on petascale data

Create scalable open numerical laboratories

Large footprint across many disciplines

Use commonly shared building blocks

Major national and international impact

Ani Thakar, JHU 8


10

Cyber Infrastructure

Science Collaboration

SDSS Integration

Outreach & Education


11



12

Database storage & Query:

Data analysis:

Data exploration:

User sign-on:

File storage:


13


14


15

Hosted Data

Personal Data

Single Sign-On

Qu

ery

Co

mp

ute



16

Hosted Data


Astronomy Cosmology

Turbulence Genomics

Materials Science Oceanography


17

Co

mp

ute


Server Cluster


18

Co

mp

ute


Server Cluster

VM VM


19

Co

mp

ute


VM

Docker Docker Docker


20

Co

mp

ute


Docker

Jupyter


21

Co

mp

ute


Docker

Jupyter

INTERACTIVE & SYNCHRONOUS


22

Engine for executing analysis on data sets Environment for executing Python Notebooks

Interactively Utility API Libraries in Python and R Interacts with ALL other SciServer

components that have a WS API:◦ Login Portal for authentication◦ CASJobs for Queries◦ SkyServer and SkyQuery for Astronomy data◦ SciDrive for Storage


23


24


25


26


27


28


30

HPC

CloudStore

DBSkyServer

Compute

SciDrive

indexing

cycles/byte

SciServer


31

Build on VM/Docker Architecture

Scalable non-interactive, asynchronous Job management (JOBM)

Rich Access Controls (RACM)

Distributed compute execution (COMPM)

Support Python, R, Matlab


32

Dashboard UI

SciDrivePlugins

CASJobs

Resource and Access Control Manager

Compute Manager

D D

D D

D D D

Server Cluster

Compute Manager

D D

D D

D D D

Server Cluster

Job Manager

Job List MetadataACLs

Resources

Script Job

Script Job

Query Job

PUSH

PUSH

PUSHPULL

PULLAsync

Async

Jupyter


33

Dashboard UI

SciDrivePlugins

CASJobs


Compute Manager

D D

D D

D D D

Server Cluster

Compute Manager

D D

D D

D D D

Server Cluster

Job Manager


Resources

Script Job

Script Job

Query Job

PUSH

PUSH

PUSHPULL

PULLAsync

Async

Jupyter


34

Dashboard UI

SciDrivePlugins

CASJobs


Compute Manager

D D

D D

D D D

Server Cluster

Compute Manager

D D

D D

D D D

Server Cluster

Job Manager


Resources

Script Job

Script Job

Query Job

PUSH

PUSH

PUSHPULL

PULLAsync

Async

Jupyter


35

Dashboard UI

SciDrivePlugins

CASJobs


Compute Manager

D D

D D

D D D

Server Cluster

Compute Manager

D D

D D

D D D

Server Cluster

Job Manager


Resources

Script Job

Script Job

Query Job

PUSH

PUSH

PUSHPULL

PULLAsync

Async

Jupyter

Jupyter Notebook (INVISIBLE)

<code>{

CJ.Query()SD.Write()Open.File()

}

CASJobs Persistent File SciDrive

MyDB ScratchDBScratch File SDSS Turbulence MatSci

EXTERNAL

D


36

SciServer Compute Interactive is live now

Supports Python, R, Jupyter Runs on a 4 node cluster Access to several domain databases

Asynchronous Job Execution early 2017

Please register with SciServer and try it out

Collaborative data-driven...

Documents

Transcript of Collaborative data-driven...