SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC...
Transcript of SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC...
![Page 1: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/1.jpg)
Piers Harding // February, 2017
SKA SDP-COMP Middleware: The intersect with commodity
computing
![Page 2: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/2.jpg)
Overview
● SDP Middleware – why is this important
● What are the options
● Middleware – where is industry heading
● What are NZA doing
![Page 3: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/3.jpg)
Quick recap of SKA Context for SDPContext Diagram
These are off-site! (In Perth & Cape
Town)
These are off-site! (In Perth & Cape
Town)Ref. J Taylor - 2016
Murchison Region, AU; Karoo Desert, SA
Two independent SDPs
![Page 4: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/4.jpg)
SDP Scope SKA Phase 11
Ref. SKA-TEL-SDP-0000001 SDP Preliminary Architecture Design P Alexander et al
![Page 5: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/5.jpg)
Middleware: Where is it?
![Page 6: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/6.jpg)
SDP COMP Middleware: why is this important?
![Page 7: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/7.jpg)
SDP COMP Middleware: opportunities
● Deal with the uncertainty and pain of growth
● the opportunities to do things differently
● Adopt modern software architecture and management
● Less about jobs (batch) and more about services
● Decouple bespoke software from hardware and platform (as much as possible)
● Guard against becoming a single purpose platform
● Position to take advantage of future innovation
![Page 8: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/8.jpg)
Middleware: Project aspirations
● Commodity computing – COTS
● Reduce investment in bespoke development - “let others do as much
as possible”
● Control costs – initial and ongoing
● Openness – preference for open source and open standards –
enable participation
![Page 9: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/9.jpg)
Middleware: SDP developer infrastructure
● External entities will need to write code to insert in the pipelines
● Must define APIs and interfaces, publish and give reference
implementations
● Provide development tools
● Testing environments
● “encapsulate in tools and environments so I can run at home”
![Page 10: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/10.jpg)
Middleware: what are the options?
![Page 11: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/11.jpg)
Middleware: Focus on containerisationa modern software paradigm
![Page 12: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/12.jpg)
Containerisation: how it (should!) works
● In kernel virtualisation using cgroups, and namespaces
● Containers launched from immutable images – share layers
● Packaging and dependency encapsulation
● Philosophy:
● 1 container == 1 service (preferably 1 process)
● Immutability – IO to services, external config
● Cattle not Pets
● Efficient – operational density increased - no OS boot, small images
● Enables cohabitation – heterogeneous hosts, and container versions
![Page 13: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/13.jpg)
Containerisation: Why should you care?
● Changes the way systems are architected and managed - SDLC
● Focuses on delivering services that are:
● Robust (self-healing)
● Scalable – resource aware, and scheduling capabilities
● High availability – continuous operation
● Developers closer to the platform – environmental consistency
● Delegate all but specific operational functions to the platform
![Page 14: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/14.jpg)
Container Orchestration: Why should you care?
● Centralises core functions such as:
● Telemetry
● Monitoring
● Logging
● Scheduling
● Scaling
● Focuses on resources as services
![Page 15: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/15.jpg)
Middleware and developers:
● Pipeline software developers interact with the middleware
● It becomes their API
● It defines the application process flow and their design limitations
● And their workflow (SDLC) – dev, test, prod, packaging, sharing,
debugging
![Page 16: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/16.jpg)
SKA: could be Service oriented by design
● Many characteristics of a service:
● Soft real-time
● Tight performance requirements
● Scalability and scheduling key – service flavours
● Continuous operation is an aspiration
● Unknown future processing requirements
![Page 17: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/17.jpg)
SKA-SDP: But, there is a problem!
● data rates are vast – 11Tbps
– per node – 254MB/s, 6TB temp, 3TB shared*
● The buffer storage
● Critical process overlap
● We cannot terminate nodes without:
● load balancing the ingest
● Using shared storage for the buffer
● Service recovery/resume
* 6 hour observation -51GB Grids * 52 max
![Page 18: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/18.jpg)
Observation flow & overlapFocus on Imaging Pipeline (biggest)
0 3 6 9 12 15 18 21 24 3 12
Continuous calibration - soft real-time
9.4 wksObservation 1 – 6 hours
Data Ingestion
Image Pipeline
Observation 2 – 6 hoursData Ingestion
Image Pipeline
Observation 3 – 6 hours Data Ingestion
IngestObservation 4 – 3 hours
Image
Image Pipeline
Data Ingestion
Image Pipeline
Observation 3 – 6 hours
Ingest without processing!
Processing timeline - hours
![Page 19: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/19.jpg)
Middleware: where is the industry heading?
![Page 20: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/20.jpg)
Traditional HPC● Is batch
● generally doesn’t have real-time considerations – Mesos
● Infrastructure Down time is OK (generally not considered)
![Page 21: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/21.jpg)
Emerging HPC Technologies
● Approaching real-time
● Aligned with Advanced Analytics
● Focus on operational efficiency
● Container based technologies – isolation, density, replication
● Service oriented – Spark, ImpalaDB, Kubernetes, Docker Swarm
● Evolving fast – Google GCE with GPUs, AWS ECS with GPUs
● Coming: serverless architecture, FaaS - AWS Lambda (exists but no GPU), OpenWhisk
● Resurgence in compiled languages – Go + GPU
● Not there yet
![Page 22: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/22.jpg)
Middleware: what are NZA doing?
![Page 23: SKA SDP-COMP Middleware: The intersect with commodity computing · 2017-03-09 · Emerging HPC Technologies Approaching real-time Aligned with Advanced Analytics Focus on operational](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed79cff27e9e96258456b38/html5/thumbnails/23.jpg)
Investigating design options
● Centred on Containerisation, Storage, Telemetry & Logging –
allocated tasks from the SKAO
● Also looking into:
● Platform Management – Software defined Infrastructure from
the SysAdmin and DevOps point of view
● Orchestration & Scheduling
● Solution architecture