Elastic-RA Virtual Collaborative

Environment for Scientific Computing and Data Analysis in the

Cloud

Karim Chinekarim.chine@cloudera.co.uk

o Open-source (GPL) software environment for statistical computing and graphics

o Lingua franca of data analysis.

o Repositories of contributed R packages relatedto a variety of problem domains in life sciences,social sciences, finance, econometrics, chemometrics, etc. are growing at an exponential rate.

o R Website: http://www.r-project.org/o CRAN Task View: http://cran.r-project.org/web/views/o CRAN packages : http://cran.cnr.berkeley.edu/o Bioconductor: http://www.bioconductor.org/o R Metrics: https://www.rmetrics.org/

Scientific Computing Environments

www.scilab.org

http://root.cern.ch

www.sagemath.org

www.sas.com

office.microsoft.com

www.mathworks.com

www.scipy.org

www.spss.com

www.wolfram.com

From: John Fox, Aspects of the Social Organization and Trajectory of the R Project, R Journal-Feb 2009

The ‘s Success Story

"Give me a place to stand, and I shall move the earth

with a lever"

Scientific Computing Software, HPC and Usability

Extract from the NetSolve/GridSolve Description Document

The emergence of Grid computing as the prototype of a next generation cyberinfrastructure for sciencehas excited high expectations for its potential as an accelerator of discovery, but it has also raisedquestions about whether and how the broad population of research professionals, who must be thefoundation of such productivity, can be motivated to adopt this new and more complex way of working.The rise of the new era of scientific modeling and simulation has, after all, been precipitous, and manyscience and engineering professionals have only recently become comfortable with the relatively simpleworld of the uniprocessor workstations and desktop scientific computing tools. In that world, softwarepackages such as Matlab and Mathematica represent general-purpose scientific computingenvironments (SCEs) that enable users — totaling more than a million worldwide — to solve a widevariety of problems through flexible user interfaces that can model in a natural way the mathematicalaspects of many different problem domains.Moreover, the ongoing, exponential increase in the computing resources supplied by the typicalworkstation makes these SCEs more and more powerful, and thereby tends to reduce the need for thekind of resource sharing that represents a major strength of Grid computing [1]. Certainly there arevarious forces now urging collaboration across disciplines and distances, and the burgeoning Gridcommunity, which aims to facilitate such collaboration, has made significant progress in mitigating thewell-known complexities of building, operating, and using distributed computing environments. But it isunrealistic to expect the transition of research professionals to the Grid to be anything but halting andslow if it means abandoning the SCEs that they rightfully view as a major source of their productivity.We therefore believe that Grid computing’s prospects for success will tend to rise and fall according toits ability to interface smoothly with the general purpose SCEs that are likely to continue to dominatethe toolbox of its targeted user base.

Arnold, D. and Agrawal, S. and Blackford, S. and Dongarra, J. and Miller, M. and Seymour, K. and Sagi, K. and Shi, Z. and Vadhiyar, S.

Elastic-R targets major e-Science use cases

o Lower the barriers for accessing cyber infrastructures.

o Bridge the gap between existing SCEs and grids/clouds

o Help dealing with the data deluge (take the computation to the data)

o Enable collaboration within computing environments

o Simplify the science gateways creation and delivery process

o Provide the building blocks of a user-friendly platform for reproducible computational research

o Provide a cloud-based e-Learning environment for computational sciences and statistical education

o Lower the barriers for using distributed computing, make HTC accessible to all research professionals

o Bridge the gap between different mainstream SCEs and between SCEs and workflow workbenches

o Provide a universal computing toolkit for scientific applications and scalability frameworks.

o Provide a portal for scientific computing on demand, collaboration and computational resources sharing

o Merge on demand HPC capabilities, real time collaboration and social networking

Computational ComponentsR packages : CRAN, Bioconductor, Wrapped C,C++,Fortran code

Scilab modules, Matlab Toolkits, etc.

Open source or commercial

Computational ResourcesHardware & OS agnostic computing engine : R, Scilab,..

Clusters, grids, private or public clouds

free: academic grids or pay-per-use: EC2, Azure

Computational User InterfacesWorkbench within the browser

Built-in views / Plugins / Spreadsheets

Collaborative views

Open source or commercial

Computational ScriptsR / Python / Groovy

On client side: interactivity..

On server side: data transfer ..

Stateful or stateless, automatic mapping of R data objects and functions

Computational Application Programming Interfaces

Java / SOAP / REST, Stateless and stateful

Computational Data StorageLocal, NFS, FTP, Amazon S3, Amazon EBSfree or commercial

Generated Computational Web Services

Elastic-R

Elastic-R is a ubiquitous plug-and-play platform for scientific and statistical computing Elastic-R lowers the barriers of accessing cyberinfrastructures

Elastic-R on Infrastructure-as-a-Service style Cloud

Anatomy of an Elastic-R machine instance on Amazon EC2

Heartbeat Restful WS over SSL

Public Clouds

Private Cloud

Elastic-R portal: single facade to public and private clouds

Exercise 1: Get started with Amazon EC2

Task 1 : Sign Up For Amazon Elastic Compute Cloud Open the following link : http://aws.amazon.com/ec2/Click on and follow the instructions

Task 2: Log on to AWS Management ConsoleOpen the following link http://aws.amazon.com/console/Choose Amazon EC2 in the Combo box and click on

Task 3: Run an Ubuntu Machine InstanceSelect the AMIs Panel, Select « All Images », enter the following AMI ID : ami-1437dd7dRight-click on the unique line displayed and choose « Launch Instance »

AMIs Panel

-In the « Request Instances Wizard », click on « Continue » twice-At the « CREATE KEY PAIR » step, choose « Proceed without a Key Pair » and click on « Continue »-At the « CONFIGURE FIREWALL » step, choose « Create a new Security Group », fill in the name and the Description (with a name of your choice) and click on « Continue »-At the final step « REVIEW », click on « Launch » then click « Close »

-Open the « Instances » Panel, wait for the Status icon to change from orange to green

Task 4: Shut down the Machine Instance- Right-click on the newly created instance, choose « Terminate » and confirm (« Yes, terminate »)

Exercise 2: Get started with Elastic-R

- Open The following link : www.elastic-r.org- Log on using the login/password you received by email before the tutorialAlternatively use one of the following accounts login:sc10-1/passwd:sc10-1 login:sc10-2/passwd:sc10-2 login:sc10-3/passwd:sc10-3 login:sc10-4/passwd:sc10-4

-In the « Cloud Console » panel, provide your Amazon.com login and password

- Keep the default values for machine : « R 2.11.0 Ubuntu 10.04 Lucid – 32 bits », keep the default value for capacity : «Small–32 bits–1 core–Memory 1.7 G», keep Disk to «None» and provide a label for the machine- Click on « Launch New Machine »

-The Console displays « Retrieving AWS Keys.. » then « Launching.. » then « Machine started successfully …»

-Click « ok » and wait for a new line to appear in the list of Available Machines (with the label you provided)

-When the machine is ready, you get connected automatically, you can now interact with the R session using the R console and the R graphics

-Type the following expressions in the R console : x=rnorm(100) var(x)plot(x)hist(x)

- Open the menus and views and try the different available features

Elastic-R is a collaborative Virtual Research Environment.Users can share their machine instances, stateful remote engines, data,..

Amazon Virtual Private Cloud

Subnet 2

Subnet 3

Subnet 1

The Elastic-R portal itself is an EC2 machine instance. Any number of portals can be run on EC2 for decentralized and private collaboration

Exercise 3: Share a machine instance and collaborate with other attendees.

-Open the collaboration Menu- Enter the list of Elastic-R users logins you will share your machine with in « Sharing Rights Editor » and click on « Refresh »-Providing only logins means granting all possible rights on all available engines. Specific sharing rights can be appended to the login -Examples: john:rw , allow user john to view and update peter:r:EngineTest, allow user peter to view only the session data on the engine EngineTest

- The new sharing rights appear in the « Machine Sharing Rights » panel

-Your collaborators can see your machine in their lists of available machines and can connect to it-Try the different collaboration features(consoles broadcast, graphic annotation, whiteboard, spreadsheets,..

-Open the Collaboration Menu and click on “Lead”, change the layout of your environment (resize menus, open and close views). Your collaborators’ browsers follow your Ajax workbench layout

Visual Graphic User Interface Builder

Elastic-R Java Workbench

Plugins Repository

• myPlugin• myDashboard

Upload plugin

Elastic-R AJAX Workbench

Standalone Application AccessibleFrom a URL

Users can create easily Java User Interfaces (plugins) that use the full capabilities of a stateful and remote R engine and share them as URLs

Software+Services=Applications convergence + ubiquitous collaboration.The server-side toolkit: R + spreadsheet models + virtual gui widgets.

Reproducible research: A scientist can snapshot her computational environment and her data. She can archive the snapshot or share it with others.

Elastic-R AMI 1

R 2.10 + BioC 2.5

Elastic-R AMI 2

R 2.9 + BioC2..3

Elastic-R AMI 3

R 2.8+BioC2.0

Elastic-R Amazon Machine Images

Elastic-R EBS 1

Data Set XXX

Elastic-R EBS 2

Data Set YYY

Elastic-R EBS 3

Data Set ZZZ

Elastic-R EBS 4

Data Set VVV

Elastic-R AMI 2

R 2.9 +

BioC 2.3 Elastic-R EBS 4

Data Set VVV

Amazon Elastic Block Stores

Elastic-R AMI 2

R 2.9 +

BioC 2.3 Elastic-R EBS 4

Data Set VVV

Elastic-R.org

Exercise 4: Create collaboratively and distribute a scientific dashboard using the Elastic-R server-side widgets designer

- Open The « Views » Menu and check « R Panels », a java applet is loaded - Right-click on the empty panel, choose “New Slider”- Keep the default R expression : slider.make("n", x=93,y=15,panel="ssprimary") and click “Apply”- A slider appears, change the value of the slider and check the value of the R variable “n” in the Console- Change the value of nusing the R console, and notice the change in the slider : the slider is a Bidirectional mirror of n-Right-click on the panel And choose “New Chart”,keep the default expressionand click “Apply”-A Bar Chart Mirroring the R variable “y” appears-Right-click on the Panel, Choose “New Macro”, keep the default expression: the Macro is triggered when “n”changes and updates “y”as being a function of n

-Change the slider and notice the chart update

-Click on « Show Direct URL » and copy the displayed URL to the clipboard -Open a new Browser Window an paste the url in the browser’s address bar-Notice the java applet being loaded and the interactive standalone dashboard-Send the URL by email to you neighbour

Exercise 5: call Elastic-R cloud engines from Office documents, mirror an R-enabled server-side spreadsheet to Excel (Windows users only).

Exercise 5 bis: Install R packages, generate variables, create new files, spreadsheets and virtual widgets’ panels. Snapshot your environment and share it with others.

- Open the « Tools » menu an click on the Word icon, choose Open Document with Word, read the displayed message.

-Inside the word document, type an R expression, select it and press Alt-F1.-Type an R expression producing a graphic, select it and press ALT-F1.

-Open the « Tools » menu an click on the Excel icon, choose Open Document with Excel- Open the « Spreadsheet » view in the browser. Update the cells in the browser, notice what happens in Excel, update the cells in Excel and notice what happens in the browser

The scientist can control any number of stateful R engines from within an R session on the cloud or on his machine. He can use them for parallel computing

Exercise 6: Run two 64-bit Elastic-R machine instances with 4 engines each. Use the 8 engines from within a main R session to apply a function to a large dataset.

-Open the « Portal Info » Menu and choose profile « expert »-In the cloud console view, enter : your amazon login, your amazon password , “R.2.11.1 – Ubuntu 10.04 – 64 bits”, “Large-64 bits-2 cores-..”, Shutdown after “1” Hour, Disk “None”, empty label, Number “2”, Engine Names “A,B,C,D”, Memory Min “1024”, Memory max “1024”, check “Use SSL”, click “Launch New Machine”

- Select the two new machines and click on « Get Workers Links »

- In the R console, notice the R expressions for the Rlinks creation and the Rlinks successful creation notification-Type the following R commands in the R console:>RTo display the vector of Rlinks mapping all the engines in the two Large instances>rlink.console(R[1], 'ls()')To send the expression ‘ls() ’ to the R engine referenced by R*1+>rlink.put(R[1],'n')To send the variable n to the first engine> cl<-cluster.make(R)To create a logical cluster holding all the R engines in the two large instances>cluster.console(cl, ‘ls()’)To send the expression ‘ls()’ to the consoles of all the engines>library(vsn)>data(kidney)>l=list()>for (i in c(1:100)) l[[as.character(i)]]=kidneyTo build a list of 100 item, each holding an ExpressionSet.>cluster.console(cl, 'library(vsn)')To load the library vsn on all workers>cluster.apply(cl, ‘l’, ‘justvsn’)To normalize the 100 ExpressionSet in parallelUsing the 8 engines.

Elastic-R Portal :

www.elastic-r.org

Articles about the project:

Chine K. (2010). Open Science in the Cloud: Towards a Universal Platform for Scientific andStatistical Computing. In Handbook of Cloud Computing. (Chapter 19). Springer US.

Karim Chine, "Learning Math and Statistics on the Cloud, Towards an EC2-Based Google Docs-likePortal for Teaching / Learning Collaboratively with R and Scilab," icalt, pp.752-753, 2010 10th IEEEInternational Conference on Advanced Learning Technologies, 2010

Karim Chine, "Scientific Computing Environments in the age of virtualization, toward a universalplatform for the Cloud" pp. 44-48, 2009 IEEE International Workshop on Open Source Software forScientific Computation (OSSC), 2009

Karim Chine, "Biocep, Towards a Federative, Collaborative, User-Centric, Grid-Enabled and Cloud-Ready Computational Open Platform" escience,pp.321-322, 2008 Fourth IEEE InternationalConference on eScience, 2008

Linkedin Group:

http://www.linkedin.com/groups?home=&gid=2345405

Links

ACS: Madi Nassiri Amazon: Simone Brunozzi, Deepak Singh AT&T Research Labs: Simon Urbanek ATUGE: Imen Essafi, Béchir

Tourki, Ilyes Gouja, HatemHachicha, Amine Elleuch Auckland Centre for eResearch: Nick Jones Banca d'Italia: Giuseppe Bruno

Bio-IT World: Kevin Davies BNP Paribas: Ousseynou Nakoulima Cambridge Healthtech Institute: Cindy Crowninshield City

University of New York: Mario Morales, Makram Talih Columbia University: Omar Besbes Dassault Systèmes: Omri Ben Ayoun,

Patrick Johnson Dataspora: Michael E. Driscoll EDF: Alejandro Ribes EBI: Alvis Brazma, Wolfgang Huber, Kimmo Kallio, Misha

Kapushesky, Michael Kleen, Alberto Labarga, Philippe Rocca-Serra, Ugis Sarkans, Kirsten Williams, Eamonn Maguire EPFL:

Darlene Goldstein ESPRIT: Farouk Kammoun, Tahar. Benlakhdar e-Taalim: Nadhir Douma ETH Zürich: Yohan Chalabi, Diethelm

Würtz, Martin Mächler European Commission: Konstantinos Glinos, Enric Mitjana, Monika Kacik, Ioannis Sagias FHCRC: Martin

Morgan, Nianhua Li, Seth Falcon Google: Olivier Bosquet FVG LLC: Lisa Wood Harvard University: Tim Clark, Sudeshna Das,

Douglas Burke,Paolo Ciccarese IBM: Jean-Louis Bernaudin, Pascal Sempe, Loic Simon, Lea A Deleris, Alex Fleischer, Alain Chabrier

Imperial College London: Asif Akram, Vasa Curcin, John Darlington, Brian Fuchs Indiana University:Michael Grobe INRIA: David

Monteau, Christian Saguez, Claude Gomez, Sylvestre Ledru JISC: John Wood, David Flanders Johnson & Johnson - Janssen

Pharmaceutica: Patrick Marichal KXEN: Eric Marcade Lancaster University: Robert Crouchley, Daniel Grose Leibniz Universität

Hannover: Kornelius Rohmeier LIAMA: Baogang Hue, Kang Cai Limagrain: Zivan Karaman Mekentosj: Alexander Griekspoor,

Matt Wood Microsoft: Eric Le Marois, Tony Hey Mubadala: Ghazi Ben Amor Nature Publishing Group: Ian Mulvany, Steve Scott

NCeSS: Peter Halfpenny, Rob Procter, Marzieh Asgari-Targhi, Alex Voss, YuWei Lin, Mercedes Argüello Casteleiro, Wei Jie, Meik

Poschen, Katy Middlebrough, Pascal Ekin, June Finch, Farzana Latif, Elisa Pieri, Frank O'Donnell New York Java User Group:

Frank D Greco OeRC: Dimitrina Spencer, Matteo Turilli, David Wallom, Steven Young OMII-UK: Neil Chue Hong, Steve Brewer

OpenAnalytics: Tobias Verbeke Oracle: Dominique van Deth, Andrew Bond OSS Watch: Ross Gardler Platform Computing:

Christopher Smith Royal Society: James Wilsdon San Diego Supercomputer Center: Nancy R. Wilkins-Diehr Sanger Institute: Lars

Jorgensen, Phil Butcher Shell: Wayne.W.Jones, Nigel Smith Société Générale: Anis Maktouf Stanford University: John Chambers,

Balasubramanian Narasimhan, Gunter Walther SYSTEM@TIC: Karim Azoum Technische Universität Dortmund: Uwe Ligges,

Bernd Bischl Technoforge: Pierre-Antoine Durgeat Tekiano: Samy Ben Naceur Télécom-ParisTech: Isabelle Demeure, Georges

Hebrail, Nesrine Gabsi The Generations Network: Jim Porzak Total: Yannick Perigois Tunisian Ministry of Communication

Technologies: Naceur Ammar, Lamia Chaffai-Sghaier, Mohamed Saïd Ouerghi, Syrine Tlili Tunisian Ecole Polytechnique: Riadh

Robbana UC Berkeley: Noureddine El Karoui, Terry Speed UC Davis: Rudy Beran, Debashis Paul, Duncan Temple Lang UCL:

Daniel Jeffares UCLA: Ivo Dinov, Jeroen Ooms UC San Diego: Anthony Gamst UCSF: Tena Sakai Université Catholique de

Louvain: Christian Ritter University of Cambridge: Ian Roberts, Robert MacInnis Peter Murray-Rust, Jim Downing University of

Manchester: Carole Goble, Len Gill, Simon Peters, Richard D Pearson, Iain Buchan, John Ainsworth University of Plymouth: Paul

Hewson University of Split: Ivica Puljak UTK: Ajay Ohri World Bank Group-IFC: Oualid Ammar Yahoo: Laurent Mirguet, Rob

Weltman Independant:Charles Dallas, Romain François

Acknowledgments