AstroCompute in the Cloud - Public Website€¦ · • Amazon Web Services – Brendan Bouffler –...
Transcript of AstroCompute in the Cloud - Public Website€¦ · • Amazon Web Services – Brendan Bouffler –...
AstroCompute in the CloudAmazon Web Services & the SKA
Tim Cornwell & Miles Deegan18 May 2015
• SKA Office:– Miles Deegan, EPM, SDP & TM– Tim Cornwell, SKA Architect– Jeff Wagg, SKA Project Scientist
• Amazon Web Services– Brendan Bouffler – aka ‘Boof’
• Global lead, SKA & HPC, AWS, SciCo (scientific computing)
– Jamie Kinney – Head of SciCo
Today’s hosts
Footer text
• Introduction – MD• AstroCompute and the SKA – TC• Grant applications and open data – MD• Using AWS for Scientific Computing – Boof• Q&A
Agenda
• AWS funded to the tune of $1M• To be used by the end of calendar year 2016• Half the funds cover the cost of hosting up to 1 PB of data
in AWS S3• The rest covers the cost of compute cycles• Will cover how this $500k can be applied for and used
later• Now over to Tim…
Introduction to the SKA/AWS AstroCompute Programme
Footer text
SKA Phase 1 (SKA1)Budget: €650M, construction start 2017
5
AustraliaSouthern Africa
SKA1_MID197 Dishes including:64 x MeerKAT dishes
133 x SKA dishes
SKA1_LOWLow Frequency Aperture Array
Stations
SKA Phase 2 (SKA2)Cost: TBD; construction start 2022
6
AustraliaSouthern Africa
SKA2_MID2500 Dishes
SKA2_AAMid Frequency Aperture Array
Stations
SKA2_LOWLow Frequency Aperture Array
Stations
SKA1 logical view
7
• Science Data Processing system is part of the telescope– One system per telescope
• Data flow so large that dedicated facility is needed• Telescope becomes adaptive to e.g. cancel calibration
effects• Steps are: acquire, edit, calibrate, make image,
analyse, with iterative cycles• Too much data to allow guiding by humans• But analysis step requires some human guidance and
performance• Analysis rich in visualization, feature identification,
catalog queries
How SKA processes data
8
• Lead by U. Cambridge (Paul Alexander)– Core team: ASTRON (NL), CSIRO (AU), MeerKAT (SA),
Cambridge (UK), ICRAR (AU), CHPC (SA), Hartree Centre (UK)• Responsible for processing of visibility data into images• Strong industry engagement
– IBM, INTEL, NVIDIA, etc.• Heavy compute load
– SKA1: 30 to 300PF• Massive data rate
– SKA1: 0.3 to 3 TB/s
Science Data Processing
9
Science Data Processor Local M&C
Science Data Processor
Telescope Manager
Cor
rela
tor /
B
eam
form
er
Data Routing Ingest
Visibility processing
Multiple Reads
Time Series Search
Multiple Reads
Data BufferData Routing
Time Series Processing
Image Plane Processing
Data Prodcuts
Sky Models, Calibration
Parameters ...
Meta Data
Master ControllerMaster Controller Local M&C Database
Tiered Data Delivery
• National/international science and engineering centres
• Similar to CERN Tier1• May specialize in science areas e.g. Epoch of
Reionization• Local archive of science data products• Analysis of science data products
– e.g. multi-EB HI emission cube• Funded separately from SKA• SKA Board has yet to fully define RSECs
Regional Science and Engineering Centres (RSECs)
10
Data Delivery to RSECs
Tiered Data Delivery
Astronomer
Regional Centre
Cloud
Sub-set of Archive
Data routing
Regional Centre
Sub-set of Archive
Regional Centre
Sub-set of Archive
Cloud access
SDP Core Facility South Africa
SDP Core Facility Australia
• SKA1 Science Data Processing Centres• Dedicated data centres• or Cloud-based (e.g. Amazon or public)• Both models are still options
• Region Science and Engineering Centres• Dedicated data centres• or Cloud-based (e.g. Amazon or public)• Expect both models based on local preference
Heterogeneous delivery
12
• Facilitate growth of knowledge about Clouds• Produce science results not otherwise possible• Move large data sets onto the Amazon Cloud• Produce tools• Jumpstart use of Cloud for all radio astronomy
Purpose of grants program
13
• Preferable if the data is open– out of any proprietary period imposed by the
observatory• And open means open in principle to anyone
(not just grant holders)• Use of proprietary data sets does not rule out
a proposal– But the storage cost comes out of the grant, not
just the compute cost• Had some initial conversations, need to follow
up
We need data – up to 1 PB
Footer text
• Science and/or software:– Data reduction, source identification, catalogue
development, tool development, archive management, porting, benchmarking, visualization…
• A $ estimate is required:– The AWS Simple Monthly Calculator– http://calculator.s3.amazonaws.com/index.html
• We expect most grants will be in the region of $10k• To receive a grant you will need an AWS account to get
the credits– Got one already? Let us know the 10-digit account number
Grant proposals – we need…
Footer text
• Huge amount of detail – 3, 4 pages max• CVs• Publication lists• Lists of invited talks• Etc etc• Meant to be a lightweight process; not like
applying for a formal grant with a national agency
We don’t need
Footer text
• Assurance of a concrete outcome, either as a science result, a resource, or a tool for use by others. Examples include a published peer-reviewed paper, a student project, an archive of some data, software with open-source license committed to github or similar and openly available.
• Training of students or postdocs• Relevance to SKA science or technology• Well resourced plan• An initial plan for how to make use of AWS
resources.
Criteria
Footer text
• Cannot use the funds for Reserved Instances• Other instance purchased types are OK – on-
demand or spot• AWS recommendation is to use the spot
market– Name your own price – jobs run when bid exceeds
current price– Should mean more cycles per $– Price fluctuates going on supply and demand– But potentially job interruption
• But there are ways of readily coping with this
Some guidelines and constraints
Footer text
• Call for proposals:– https://www.skatelescope.org/ska-aws-
astrocompute-call-for-proposals/• Deadline for responses: 1 June 2015• Successful bidders will be contacted before
1 July
Applications
Footer text
• Quick review on a quarterly basis– TC & MD
• Progress may lead to further grants being awarded, depends on demand
Progress tracking by SKAO
Footer text
• [email protected]• [email protected]• [email protected]• [email protected]
– AWS capabilities and tech questions to the AWS Scientific Computing Group
Contact Details
Footer text
Now for Q&A