Nectar cloud workshop ndj 20110331.2

40
NeCTAR Research Cloud Workshop 30 – 31 March 2011, Melbourne, Australia Nick Jones [email protected] www.eresearch.auckland. ac.nz Thoughts from the east, on research in the cloud

description

 

Transcript of Nectar cloud workshop ndj 20110331.2

Page 1: Nectar cloud workshop ndj 20110331.2

NeCTAR Research Cloud Workshop30 – 31 March 2011, Melbourne, Australia

Nick [email protected]

Thoughts from the east, on research in the cloud

Page 2: Nectar cloud workshop ndj 20110331.2

Genomics, Genetics, Bioinformatics, Molecular Modelling:NZ Genomics LtdMaurice Wilkins CentreAlan Wilson CentreVirtual Institute of Statistical Genetics

Genomics, Genetics, Bioinformatics, Molecular Modelling:NZ Genomics LtdMaurice Wilkins CentreAlan Wilson CentreVirtual Institute of Statistical Genetics

Wind Energy, Geothermal & Minerals Exploration:• GNS Exploration• Institute for Earth Science and Engineering• Centre for Atmospheric Research

Wind Energy, Geothermal & Minerals Exploration:• GNS Exploration• Institute for Earth Science and Engineering• Centre for Atmospheric Research

Earthquakes, Tsunami, Volcanoes:• Natural Hazards Research Platform• DEVORA Auckland Volcanic Field• GeoNet

Earthquakes, Tsunami, Volcanoes:• Natural Hazards Research Platform• DEVORA Auckland Volcanic Field• GeoNet

Invasive Species, Water / Land Use, Emissions:• Agricultural Greenhouse Gas Centre• Bio-Protection Research Centre• National Climate Change Centre

Invasive Species, Water / Land Use, Emissions:• Agricultural Greenhouse Gas Centre• Bio-Protection Research Centre• National Climate Change Centre

Human Development, Bioengineering, Social Statistics:• National Research Centre for Growth and Development• Auckland Bioengineering Institute• Liggins Institute• Malaghan Institute• Social Sciences Data Service

Human Development, Bioengineering, Social Statistics:• National Research Centre for Growth and Development• Auckland Bioengineering Institute• Liggins Institute• Malaghan Institute• Social Sciences Data Service

Nanotechnology and High Technology Materials:MacDiarmid InstituteMaterials TRST

Nanotechnology and High Technology Materials:MacDiarmid InstituteMaterials TRST

Page 3: Nectar cloud workshop ndj 20110331.2

Roadmap of NZ e-Infrastructure

National eScience Infrastructure

BeSTGRID – Compute / Data Services

NZ Genomics Ltd

BeSTGRID Federation Tuakiri – Research Access Federation

Research Data Infrastructure

KAREN Advanced Research Network

Weather / Atmosphere NIWA HPC – P575

BlueFern BlueGene/L

Genomics

Data

Identity

HPCeScience

Network

Page 4: Nectar cloud workshop ndj 20110331.2

NZs eResearch Infrastructures

HPC & Distributed ComputingBeSTGRID, since 2006• $2.5M + $800k• Largest national grid users in

Australasia• Built capability, established a

model for sharing resources

NZ eScience Infrastructure (NeSI)• ~$50M ($27M Crown)• 4 HPC centres across NZ• Signing agreements now, running

for 4 years > review > ongoing…

NZ Genomics Ltd (NZGL)• National genomics service• 4 major partners• Funded in 2008• Almost operational..

Page 5: Nectar cloud workshop ndj 20110331.2

NZ Genomics Ltd

Centralised Cloud (?)

~$3.5M

Services federated with NeSI

Page 6: Nectar cloud workshop ndj 20110331.2

Science Users

Principal Investors

UJV

Procurement Planning

Operational management

Data & Grid middleware to manage access

Outreach

NIWA• People

• HPC capacity

• Storage

• Portion of facilities, power, depreciation costs

Auckland• People

• New cluster

• New storage

• Portion of facilities, power, depreciation costs

Principal and Associate Investors• Access through a “single front door”• Specialised concentrations of capability at each institution• Receive government coinvestment• Capacity available to institutions reflects their level of investment• Managed and metered access to resources, across all resource types• Institutions’ access costs covered by investment

Services

Canterbury• People

• HPC

• New storage

• Portion of facilities, power, depreciation costs

Government

• Invest over 4 years, plus out-years

AgResearch / Otago• People

• New cluster

• New storage

• Portion of facilities, power, depreciation costs

Managed Access

Legend

Financial flows

Access

Research institutions (non investors)• Access through a “single front door”• Capacity scaled out from partners capabilities• Managed and metered access to resources, across all resource types• Access fee initially calculated as partial costs of metered use of resources and reviewed annually

Industry• Access through a “single front door”• Capacity scaled out from partners capabilities• Managed and metered access to resources, across all resource types• Access fee calculated as full costs of metered use of resources

Governance

Principal Investors• Transparent• Collaborative

Cluster &Services

HPC & Services

HPC & Services

Cluster &Services

Page 7: Nectar cloud workshop ndj 20110331.2

Board(7 members)

Director(Fulltime Position)

Team @AgResearch

Team @Auckland

Team @Canterbury

Team @NIWA

Strategic Advisory Group

( 7 members)

Governance

Management & Services

4 Board nominated researchers

Chair (independent via UJV)1 independent (via UJV & MoRST)

1 from each Principle Investor1 nominated by Associate Investors

Chair3 NZ researchers3 offshore researchers

Management TeamProgramme & CommunicationsFinance & Administration

User Advisory Group

Following roles are covered across all teams (in varying portions):Manager / Technical Lead / Systems / Middleware / Applications

Page 8: Nectar cloud workshop ndj 20110331.2

Director

Programme Manager

Communications Manager

Finance & Administration

User Advisory Group

Management & Services

4 Board nominated researchers

Services

AgResearch/Otago

ManagerTechnical LeadApplicationsMiddlewareSystems

Auckland

ManagerTechnical LeadApplicationsMiddlewareSystem

Canterbury

ManagerTechnical LeadApplicationsMiddlewareSystem

NIWA

ManagerTechnical LeadApplicationsMiddlewareSystem

SLASLA SLA SLA

Management

Page 9: Nectar cloud workshop ndj 20110331.2

Core Functions

Page 10: Nectar cloud workshop ndj 20110331.2

Land of the long white cloud..

• Within NZGL and NeSI, we have reasonable scale investments into virtualised infrastructure

• Our aim is to build at least a federated cloud platform, if not a single research cloud (perhaps with mulitple availability zones)

Page 11: Nectar cloud workshop ndj 20110331.2

Migrating users’ tools into the Cloud

Tools

Desktop/Server

*aaS

Cloud

Page 12: Nectar cloud workshop ndj 20110331.2

Who are our users?• Researchers• HPC users• System Administrators• Application Developers

What are the funders expectations?– Facebook for scientists? – VM and development platforms– Scaled out server infrastructure & development platform efficiencies and coordination – National collaborative infrastructure

Needs and solutions depend on which layer of the cloud you’re creating:

SaaS, PaaS, IaaS

Page 13: Nectar cloud workshop ndj 20110331.2

*aaS Examples

Galaxy (Genetic Marker Design)

– analysis modules– workflows

Biocommons– multisite– sheep.biocommons.org.nz

(Genomics)– analysis modules– pipelines

• Scale out analyses?

• Self service

• Customised / templated services

• Reuse of methods?

Page 14: Nectar cloud workshop ndj 20110331.2

www.bestgrid.orgwww.nesi.org.nztechnical.bestgrid.orgwww.eresearch.org.nz

www.nzssds.org.nz

df.bestgrid.org

Page 15: Nectar cloud workshop ndj 20110331.2

• Schedulers• LRMS

IaaS: Compute Elements

• Computational Libraries• Compilers• Grisu + Globus, VDT

PaaS: Scriptable

Environments

• Finite Element, Fluid Dynamics, Statistics

• Grisu clients• GSI-SSH• Web portals

SaaS: Packaged

Applications

• Eucalyptus• Nimbus• Amazon EC2• Storage Service• File Service

IaaS

• Biocommons• GenePattern• Galaxy• Drupal multisite website platform• Shibboleth Federation Services• iRODS

PaaS

• Sakai• Bionimbus• sheep.biocommons.org.nz, etc• DataFabric

SaaS

Applications & Services HPC

+

Page 16: Nectar cloud workshop ndj 20110331.2

Joining the Grid to the Cloud

Issues, early 2010:• Maturity of solution• Overhead of VM

provisioning

Cloud Workshop 18/02/2011• https://

wiki.heprc.uvic.ca/twiki/bin/view/Main/CloudWorkshop

Examines queue, launches VMs on Nimbus, EC2, Eucalyptus

Let users define the computational image

Provide a library for them to share,

discover, launch

Page 17: Nectar cloud workshop ndj 20110331.2

GPU

GigE

Memory Cores

Network

Storage

Clusters

Virtual Labs

Services Adhoc servers

Databases

Websites

Portals

Research Cloud

GPU

Network

GigE Memory

Cores

Storage

GPU

GigE

Memory Cores

GPU

GigE

Memory Cores

IaaS

PaaS

SaaS

Page 18: Nectar cloud workshop ndj 20110331.2

Users Issues & NeedsWhat are their issues?– Accessibility– Usability– Domain specificity– Funding sources (opex, capex, none)– Sustainability

What are their needs?– Self service / on demand– Applications Registry

• Ease of discovery

– Consultancy• Migration, adaptation

– Mature Service Delivery

Page 19: Nectar cloud workshop ndj 20110331.2

Manapouri underground power station

Come… meet the cloud…..

Page 20: Nectar cloud workshop ndj 20110331.2

• Eucalyptus• Nimbus• Amazon EC2IaaS

• Drupal multisite website platform• Compute Platform• Shibboleth Federation Services• Biocommons• GenePattern• Galaxy

PaaS

• Genetic Marker Design• Sakai• Bionimbus• sheep.biocommons, etc• Grisu Compute Jobs• DataFabric• HPC Applications

SaaS

Operating Environments

Package Maintenance

Systems Management

Data Networks

Node Interconnects

Application/Service DevelopmentCommunity

BuildingSoftware

Engineering Integration

Application/Service AdministrationContent Customisation Community

Building

Derinkuyu Underground City, Cappadocia, Turkey

Page 21: Nectar cloud workshop ndj 20110331.2

CONSOLE OPERATOR AT SAIGON SATELLITE TERMINAL. Soldier monitors satellite traffic and selects satellite to be used.

• Eucalyptus• Nimbus• Amazon EC2IaaS

• Drupal multisite website platform• BeSTGRID Compute Platform• Shibboleth Federation Services• Biocommons• GenePattern• Galaxy

PaaS

• Genetic Marker Design• Sakai• Bionimbus• sheep.biocommons, etc• Grisu Compute Jobs• DataFabric

SaaS

Operating Environments

Package Maintenance

Systems Management

Data Networks

Node Interconnects

Application/Service DevelopmentFunctional

RequirementsSoftware

Engineering Integration

Application/Service AdministrationContent Customisation Community

Development

Page 22: Nectar cloud workshop ndj 20110331.2

eResearch Tools development

In the past, staff have been systems operators and middleware developers– few business analysts and user centered designers

Who is driving product and service development?

Who discovers needs, translates into eResearch Tools?

Page 23: Nectar cloud workshop ndj 20110331.2

Apps

Federated services Consistent service interfaces

University

CommunityInstitute

University

Community

Institute

National Research

Cloud

Institutional services

Where are the users now?

Under their desks!

In institutional facilities

In externally supported communities

Migrating users to the Cloud

And, we want them to join us!

Page 24: Nectar cloud workshop ndj 20110331.2

Migrating eResearch Tools

Monolithic systems aren’t compatible with HPC, Grid, nor the Cloud

• Applications require scaling out, whether databases, web based analysis engines, desktop applications

MigrationeResearch

ToolsDevelopment

Research Cloud –

SaaS

Research Cloud – PaaS

Research Cloud - IaaS

Page 25: Nectar cloud workshop ndj 20110331.2

Going the last mile?Who is supporting the researcher, to understand their needs, define their requirements, and implement their solutions?

• Are there services and applications either already in place, or under development, to meet needs, and fill capacity created?

Who will cross the boundaries between these groups?

Researchers

Applications & Services

developers

Platform service

providers

Infrastructure service

providers

Page 26: Nectar cloud workshop ndj 20110331.2

Who are our users?

Who are the users? – Researchers – technically savvy ones..– HPC users.. maybe

–System Administrators–Application Developers

“All researchers that need to use eresearch tools already know how to and are doing so..”

“If you can’t write the code yourself, how can you trust the results? Surely you must be able to write the code!?”

Page 27: Nectar cloud workshop ndj 20110331.2

What do I talk about, when I talk about Cloud?

Instead of the machinery:

VMs, hypervisors, schedulers, block storage, virtual networks, hosts, networks, firewalls

How about capabilities:

Scalability: Methods, Communities

Self Service

Sustainability (through preservation)

Where do we do this well?Data?

• Preservation• Curation

HPC?• Algorithms• Methods• Hardware optimisation techniques

When communicating to end users, common vocabularies in other communities aren’t about the machinery, they’re higher level…

Page 28: Nectar cloud workshop ndj 20110331.2

What do I talk about, .. ScalabilityWhat is the equivalent of algorithms and optimal approaches in the Cloud era?

The knowledge, skills, and methods to optimally use the cloud?

How to be elastic (scale up/down, on demand)

How to manage failure

Multi-tenancy

Building communities, leveraging network effects

Page 29: Nectar cloud workshop ndj 20110331.2

What do I talk about, .. Preservation

What are the research processes and artefacts we are interacting with?

Methodologies..enshrined as VMs?

e.g. research codes, which take cuts and bruises to re-instantiate = knowledge artefacts that have value and are scarce

What about workflows?Shouldn’t we wait until we have enshrined these methods into sophisticated workflows, and ontologies?

.. So we should throw away any method until it reaches a predefined threshold of maturity? Because before then, it has no value?

Oh, so all methods are important?that’s up to the researcher to say.. Method construction and archive,

self service.

Ahh, so we’re close to being able to archive and preserve rich research methods?

for those that enshrine their methods in software based systems… YES!

Reuseable workflows and services for large collaborative communities?

… like Virtual Organistions, they only suit certain levels of scale and maturity

Individual research codes, workflows, pipelines

= research notebooks for those who efficiently enshrine their thoughts and ideas in code

These are people we should support!

And, their experiments things we should preserve..

Page 30: Nectar cloud workshop ndj 20110331.2

What did we learn from the grid?

So..

…. what do we need to get right,

to be successful?

Page 31: Nectar cloud workshop ndj 20110331.2

Get the team right

Composition & Coordination of distributed team essential• Fully committed FTEs• Cross site responsibilities, including shared

systems administration• Strong leadership and programme

management

Page 32: Nectar cloud workshop ndj 20110331.2

Strong guidance

Which resources are getting used, why?

• Prescriptive framework necessary to ensure participation

… need a well defined service

model

Page 33: Nectar cloud workshop ndj 20110331.2

Should we build new institutions?Why do we want to build institutions?

Scale efficiencies

Sustainability

.. and reliable targeted accountable services

We often take project funding, and aspire to build new institutions

Creating institutions is and will continue to be expensive and difficult..

Our institutions are long lived, and can absorb most technologies, given care and enough time to mature

Page 34: Nectar cloud workshop ndj 20110331.2

Really think about CoordinationWhy do we need Government money?

Scale?

We’re partly seeing a coordination failure (coordination of capital and resources)

It is the incentive to bring out institutions into alignment..

Funding:work with funders to align opex and capex. Crown and coinvestors fund both in NZ, clarifying intent of investments

Otherwise, whoever funds the operating, eats the cake

Ensure KPIs incentivize sustainability

Incentives and obligations … clarify expectations to coinvestors, sector

“Single Front Door”

Coordinate with broadest scale IT community that has responsibility and capability

– Research labs can be good, though often will be introspective / focus on local solutions / research outputs

– Drive innovations into core IT Services groups; strong coordination from top level sponsors required

– Collaborate nationally and internationally

Page 35: Nectar cloud workshop ndj 20110331.2

Usability really counts !!

Make it easy!• Authentication … Authentication …

Authentication … • Consistent User Interfaces?

1. Don’t get in my way2. Don’t break the abstraction3. Don’t give me middleware!!

Page 36: Nectar cloud workshop ndj 20110331.2

Build communitiesIs the aim to build aggregations of communities?• Support national scale communities?

… but these may not exist

will need to build communities (developers, end users, administrators, leaders)

Who will mediate within and between

communities, seek out commonalities, and

take ownership of developments?

Page 37: Nectar cloud workshop ndj 20110331.2

Support the long tail• There’s an endless list of applications and services, that individual

researchers need and want• We often focus on the large communities, common platforms, and

scalability

• To make a Research Cloud useful for the majority of the community, we need to design for great diversity– We can do this incrementally

• We’re looking to learn, fast, what works for researchers, and what doesn’t

>>> we need agility, and very strong engagement with users communities

Page 38: Nectar cloud workshop ndj 20110331.2

Thanks

NeCTAR Research Cloud Workshop30 – 31 March 2011, Melbourne, Australia

Nick [email protected]

Thoughts from the east, on research in the cloud

Page 39: Nectar cloud workshop ndj 20110331.2

Image Credits

• www.nzhistory.net.nz, the website of NZHistory.net.nz, New Zealand history online, at Manatū Taonga. Licensed by Manatū Taonga for re-use under the Creative Commons Attribution-Non-Commercial 3.0 New Zealand Licence.

• www.history.army.mil, Communications Electronics 1962-1970, Department of the Army

• www.travelpod.com, Travel Blog, 7.08.2010, http://www.travelpod.com/members/wbeardsl

Page 40: Nectar cloud workshop ndj 20110331.2

GoalsNeeds of users, as identified at this workshop

Policy implications• Applications & Services that are in common national usage• Enable provision of excellent cloud services to researchers• Nodes operating within a prescribed framework

Framework• Consistent user interface

– to the range of applications and services running at the distributed nodes• Applications and Services

– Selected by Panel– Data analysis, visualisation, collaboration, security, application and service platforms, portals / interfaces to HPC and commercial

cloud providers• Infrastructure as a Service

– Hosting Platform as a Service offerings for research communities• Distributed Nodes

– Selected by panel• Prescriptive operating model

– Lead Node will create & operate framework; monitor service delivery• Access, interop, application migration, security, licensing, accounting, implementation practicalities, monitoring and maintenance

– Common layer that each node can interoperate with