AstroCompute in the CloudAmazon Web Services & the SKA

Tim Cornwell & Miles Deegan18 May 2015

• SKA Office:– Miles Deegan, EPM, SDP & TM– Tim Cornwell, SKA Architect– Jeff Wagg, SKA Project Scientist

• Amazon Web Services– Brendan Bouffler – aka ‘Boof’

• Global lead, SKA & HPC, AWS, SciCo (scientific computing)

– Jamie Kinney – Head of SciCo

Today’s hosts

Footer text

• Introduction – MD• AstroCompute and the SKA – TC• Grant applications and open data – MD• Using AWS for Scientific Computing – Boof• Q&A

Agenda

• AWS funded to the tune of $1M• To be used by the end of calendar year 2016• Half the funds cover the cost of hosting up to 1 PB of data

in AWS S3• The rest covers the cost of compute cycles• Will cover how this $500k can be applied for and used

later• Now over to Tim…

Introduction to the SKA/AWS AstroCompute Programme

Footer text

SKA Phase 1 (SKA1)Budget: €650M, construction start 2017

5

AustraliaSouthern Africa

SKA1_MID197 Dishes including:64 x MeerKAT dishes

133 x SKA dishes

SKA1_LOWLow Frequency Aperture Array

Stations

SKA Phase 2 (SKA2)Cost: TBD; construction start 2022

6

AustraliaSouthern Africa

SKA2_MID2500 Dishes

SKA2_AAMid Frequency Aperture Array

Stations

SKA2_LOWLow Frequency Aperture Array

Stations

SKA1 logical view

7

• Science Data Processing system is part of the telescope– One system per telescope

• Data flow so large that dedicated facility is needed• Telescope becomes adaptive to e.g. cancel calibration

effects• Steps are: acquire, edit, calibrate, make image,

analyse, with iterative cycles• Too much data to allow guiding by humans• But analysis step requires some human guidance and

performance• Analysis rich in visualization, feature identification,

catalog queries

How SKA processes data

8

• Lead by U. Cambridge (Paul Alexander)– Core team: ASTRON (NL), CSIRO (AU), MeerKAT (SA),

Cambridge (UK), ICRAR (AU), CHPC (SA), Hartree Centre (UK)• Responsible for processing of visibility data into images• Strong industry engagement

– IBM, INTEL, NVIDIA, etc.• Heavy compute load

– SKA1: 30 to 300PF• Massive data rate

– SKA1: 0.3 to 3 TB/s

Science Data Processing

9

Science Data Processor Local M&C

Science Data Processor

Telescope Manager

Cor

rela

tor /

B

eam

form

er

Data Routing Ingest

Visibility processing

Multiple Reads

Time Series Search

Multiple Reads

Data BufferData Routing

Time Series Processing

Image Plane Processing

Data Prodcuts

Sky Models, Calibration

Parameters ...

Meta Data

Master ControllerMaster Controller Local M&C Database

Tiered Data Delivery

• National/international science and engineering centres

• Similar to CERN Tier1• May specialize in science areas e.g. Epoch of

Reionization• Local archive of science data products• Analysis of science data products

– e.g. multi-EB HI emission cube• Funded separately from SKA• SKA Board has yet to fully define RSECs

Regional Science and Engineering Centres (RSECs)

10

Data Delivery to RSECs

Tiered Data Delivery

Astronomer

Regional Centre

Cloud

Sub-set of Archive

Data routing

Regional Centre

Sub-set of Archive

Regional Centre

Sub-set of Archive

Cloud access

SDP Core Facility South Africa

SDP Core Facility Australia

• SKA1 Science Data Processing Centres• Dedicated data centres• or Cloud-based (e.g. Amazon or public)• Both models are still options

• Region Science and Engineering Centres• Dedicated data centres• or Cloud-based (e.g. Amazon or public)• Expect both models based on local preference

Heterogeneous delivery

12

• Facilitate growth of knowledge about Clouds• Produce science results not otherwise possible• Move large data sets onto the Amazon Cloud• Produce tools• Jumpstart use of Cloud for all radio astronomy

Purpose of grants program

13

• Preferable if the data is open– out of any proprietary period imposed by the

observatory• And open means open in principle to anyone

(not just grant holders)• Use of proprietary data sets does not rule out

a proposal– But the storage cost comes out of the grant, not

just the compute cost• Had some initial conversations, need to follow

up

We need data – up to 1 PB

Footer text

• Science and/or software:– Data reduction, source identification, catalogue

development, tool development, archive management, porting, benchmarking, visualization…

• A $ estimate is required:– The AWS Simple Monthly Calculator– http://calculator.s3.amazonaws.com/index.html

• We expect most grants will be in the region of $10k• To receive a grant you will need an AWS account to get

the credits– Got one already? Let us know the 10-digit account number

Grant proposals – we need…

Footer text

• Huge amount of detail – 3, 4 pages max• CVs• Publication lists• Lists of invited talks• Etc etc• Meant to be a lightweight process; not like

applying for a formal grant with a national agency

We don’t need

Footer text

• Assurance of a concrete outcome, either as a science result, a resource, or a tool for use by others. Examples include a published peer-reviewed paper, a student project, an archive of some data, software with open-source license committed to github or similar and openly available.

• Training of students or postdocs• Relevance to SKA science or technology• Well resourced plan• An initial plan for how to make use of AWS

resources.

Criteria

Footer text

• Cannot use the funds for Reserved Instances• Other instance purchased types are OK – on-

demand or spot• AWS recommendation is to use the spot

market– Name your own price – jobs run when bid exceeds

current price– Should mean more cycles per $– Price fluctuates going on supply and demand– But potentially job interruption

• But there are ways of readily coping with this

Some guidelines and constraints

Footer text

• Call for proposals:– https://www.skatelescope.org/ska-aws-

astrocompute-call-for-proposals/• Deadline for responses: 1 June 2015• Successful bidders will be contacted before

1 July

Applications

Footer text

• Quick review on a quarterly basis– TC & MD

• Progress may lead to further grants being awarded, depends on demand

Progress tracking by SKAO

Footer text

• t.cornwell@skatelescope.org• m.deegan@skatelescope.org• bouffler@amazon.com• ska-astrocompute@amazon.com

– AWS capabilities and tech questions to the AWS Scientific Computing Group

Contact Details

Footer text

Now for Q&A