PKI-Anwendungen 6/2/2015 1 Base framework. PKI-Anwendungen 6/2/2015 2 Bottom frame.
Cloud Computing mit mathematischen Anwendungen€¦ · Automated program start after machine launch...
Transcript of Cloud Computing mit mathematischen Anwendungen€¦ · Automated program start after machine launch...
KIT – University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association
4.Cloud Applications
www.kit.edu
Cloud Computing mit mathematischen Anwendungen Dr. habil. Marcel Kunze Engineering Mathematics and Computing Lab (EMCL) Institut für Angewandte und Numerische Mathematik IV Karlsruhe Institute of Technology (KIT)
2
Examples of Cloud Applications
1. Rendering of movies 2. Management of digital data 3. Office work 4. Collaborative work
Cloud Computing | SS2011 | M.Kunze
3
Cloud Architectures using SQS, S3 and EC2
Cloud Computing | SS2011 | M.Kunze
4 Cloud Computing | SS2011 | M.Kunze
Cloud Applications (1): Rendering of Movies
! Architecture: Weakly coupled services ! Use storage service for files ! Use compute service for work ! Organize workflow by use of a queuing service
1. Store files 2. Store messages with
instructions 3. Start computation
5 Cloud Computing | SS2011 | M.Kunze
6 Cloud Computing | SS2011 | M.Kunze
7 Cloud Computing | SS2011 | M.Kunze
8 Cloud Computing | SS2011 | M.Kunze
9 Cloud Computing | SS2011 | M.Kunze
10 Cloud Computing | SS2011 | M.Kunze
11 Cloud Computing | SS2011 | M.Kunze
12
Cloud Applications (2): SmugMug http://www.smugmug.com/
Cloud Computing | SS2011 | M.Kunze
13
Cloud Applications (2): SmugMug http://www.smugmug.com/
Management of Pictures and Movies – Millions of users – Billions of pictures – Revenue: Several million $ per year – 19 employees - IT Services: Amazon Web Services - User management: OpenID
Mashup M
ashu
p S3, EC2, FPS, …
Application = Mashup of services of various providers
Cloud Computing | SS2011 | M.Kunze
14
Cloud Applications (3): Windows Live
! Windows 7 ships with ready to go cloud applications ! Windows Azure Software Development Kit
Cloud Computing | SS2011 | M.Kunze
15
Cloud Applications (4): Google Apps
! Google Apps offers office applications ! Collaboration within the company (e.g. KIT)
Cloud Computing | SS2011 | M.Kunze
16 Cloud Computing | SS2011 | M.Kunze
17
docs.google.com
Cloud Computing | SS2011 | M.Kunze
18
Optimize Linear Equations
! An example of when and how to use Solve ! Let's say you're a farmer and you want to decide what to grow. You know
that each crop will bring in a certain amount of money and take a certain amount of land, capital, and fertilizer to grow
! You can grow three things: wheat, corn, and broccoli ! Each of them has its own costs and brings in a given amount of money.
However, there are only limited resources available: ! Money available: $170 ! Land available: 70 acres ! Fertilizer available: 100 tons
Cloud Computing | SS2011 | M.Kunze
19
Optimize Linear Equations
! Here's how you can express this information in the spreadsheet:
Cloud Computing | SS2011 | M.Kunze
20
Optimize Linear Equations
! These are the formulas used in the spreadsheet: ! Money (Cell B7) =9*A3+8*B3+7*C3 ! Land (Cell B8) =7*A3+8*B3+9*C3 ! Fertilizer (Cell B9) =2*A3+10*C3 ! Profit (Cell B16) =6*A3+8*B3+C3
! When you run Solve, it can tell you what to grow:
Cloud Computing | SS2011 | M.Kunze
21 Cloud Computing | SS2011 | M.Kunze
Hausaufgabe 4
Einem IaaS-Anbieter steht ein Rechenzentrum mit 100 Schränken und einer gesamten Anschlussleistung von 1.500 kW zur Verfügung. In jedem Schrank sind Rechner mit insgesamt 1.000 CPU-Cores und 1,5 TeraByte Memory installiert. Es sollen 3 verschiedene Klassen von virtuellen Maschinen angeboten werden: Micro, Small und Large. Jede Klasse hat spezifische Anforderungen an Memory, Zahl von CPU-Cores, Energieverbrauch und bringt einen gewissen Gewinn. Die Managementwerkzeuge können maximal 80.000 Instanzen verwalten. Large Small Micro Nebenbedingung Memory (GB) 10 5 1 Max. 150 TeraByte CPU-Cores 8 2 1 Max. 100.000 CPU-Cores Energiebedarf (W) 50 25 15 Max. 1.500 kW Gewinn (Cent/h) 10 4 2 Zu maximieren Verwenden Sie zur Optimierung des Gewinns die Solve-Funktion der Tabellenkalkulation aus den Google Apps.
• Wie viele Instanzen sollten von jedem Typ gefahren werden, damit der Gewinn maximal ist? • Wie groß ist der Gewinn pro Stunde? • Wie groß ist in diesem Fall der Energiebedarf?
22
Cloud Research and Development
Cloud Computing | SS2011 | M.Kunze
23 Cloud Computing | SS2011 | M.Kunze
24
Open Cirrus Cloud Computing Research Testbed http://opencirrus.org
! An open, internet-scale global testbed for cloud computing research ! A tool for collaborative research ! Data center management & cloud services
! Resources ! Multi-continent, multi-datacenter, cloud computing system ! Federated “Centers of Excellence” around the globe
! each with 100–400+ nodes and up to ~PB storage ! and running a suite of cloud services
! Structure ! Sponsors: HP Labs, Intel Research, Yahoo! ! Founding partners: IDA Singapore, KIT, UIUC ! New partners: CESGA, CMU, ETRI, MIMOS, RAS,
China Mobile, China Telecom
! Cover story in IEEE Computer, vol 43, no 4, April 2010
Cloud Computing | SS2011 | M.Kunze
25
Cloud Systems Research and Architecture
! Perform research in the following areas: ! Cloud services ! Datacenter federation ! Datacenter management and automation ! Data-intensive applications and systems ! Cloud management tools, e.g. KOALA
http://koalacloud.appspot.com/
! Methodology ! Perform experiments also on a low system level ! Utilize flexible cloud computing frameworks ! Compare different configurations and implementations
! Simple, transparent, controllable cloud computing infrastructure
Cloud Computing | SS2011 | M.Kunze
26 Cloud Computing | SS2011 | M.Kunze
Proprietary Cloud Computing Stacks
27 Cloud Computing | SS2011 | M.Kunze
Open-source Cloud Stack
! OpenCirrus researchers have complete access to the hardware and software platform
28
Open Cirrus Blueprint
IT-Infrastructure Layer (Physical Resource Sets)
Cloud Infrastructure Services
Cloud Application Services
Virtual Resource Sets
Eucalyptus OpenNebula
Cloud Computing | SS2011 | M.Kunze
29 Cloud Computing | SS2011 | M.Kunze
Physical Resource Sets (PRS)
! PRS service goals ! Provide mini-datacenters to researchers ! Isolate experiments from each other ! Stable base for other research
! PRS service approach ! Allocate sets of physical co-located nodes, isolated inside VLANs. ! Leverage existing software (e.g. Utah Emulab, HP OpsWare) ! Start simple, add features as we go ! Base to implement virtual resource sets
! Hardware as a Service (HaaS)
30 Cloud Computing | SS2011 | M.Kunze
Virtual Resource Sets (VRS)
! Basic idea: Abstract from physical resource by introduction of a virtualization layer
! Concept applies to all IT aspects: CPU, storage, networks and applications, …
! Main advantages ! Implement IT services exactly fitting customer‘s varying need ! Deploy IT services on demand ! Automated resource management ! Easily guarantee service levels ! Live migration of services
! Infrastructure as a Service (IaaS) ! Implement compute and storage services ! De-facto standard: Amazon Web Services interface
31 Cloud Computing | SS2011 | M.Kunze
32 Cloud Computing | SS2011 | M.Kunze
What is it?
! Open-Source toolkit for building cloud infrastructures ! Originates from the EC funded RESERVOIR project
! Orchestrates storage, network and virtualization technologies to enable the dynamic placement of multi-tier services on distributed infrastructures, combining both data center resources and remote cloud resources, according to allocation policies
! Provides internal and cloud administration and user interfaces for the full management of the IaaS cloud platform
33 Cloud Computing | SS2011 | M.Kunze
Architecture
Cloud Plugins: Amazon EC2 and ElasticHosts connectors
34 Cloud Computing | SS2011 | M.Kunze
Feature Function
Internal Interface • Unix-like CLI for fully management of VM life-cycle and physical boxes • XML-RPC API and libvirt virtualization API
Scheduler • Requirement/rank matchmaker allowing the definition of workload and resource-aware allocation policies
• Support for advance reservation of capacity through Haizea
Virtualization Management
• Xen, KVM, and VMware • Generic libvirt connector (VirtualBox planned for 1.4.2)
Image Management • General mechanisms to transfer and clone VM images
Network Management • Definition of isolated virtual networks to interconnect VMs
Service Management and Contextualization
• Support for multi-tier services consisting of groups of inter-connected VMs, and their auto-configuration at boot time
Security • Management of users by the infrastructure administrator
Fault Tolerance • Persistent database backend to store host and VM information
Scalability • Tested in the management of medium scale infrastructures with hundreds of servers and VMs (no scalability issues has been reported)
Installation • Installation on a UNIX cluster front-end without requiring new services • Distributed in Ubuntu 9.04 (Jaunty Jackalope)
Flexibility and Extensibility
• Open, flexible and extensible architecture, interfaces and components, allowing its integration with any product or tool
Features
35 Cloud Computing | SS2011 | M.Kunze
36 Cloud Computing | SS2011 | M.Kunze
What is it?
! Open-Source software infrastructure for implementing Cloud computing on clusters from UC Santa Barbara
! EUCALYPTUS - Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems
! Implements Infrastructure as a Service (IaaS) – gives the user the ability to run and control entire virtual machine instances (Xen, KVM) deployed across a variety of physical resources
! Interface compatible with Amazon EC2 ! Includes Walrus, a basic implementation of Amazon S3 interface ! Potential to interact with the same tools, known to work with
Amazon EC2 and S3 ! Eucalyptus is an important step to establish an open Cloud
computing infrastructure standard
37
Architecture
! Cloud Controller (CLC) ! Collects resources information of the CCs ! Manages the S3 / EBS Services ! Cloud Meta-Scheduler
Cloud Computing | SS2011 | M.Kunze
Public Network
CLC
S3 / EBS DataBase
Private Network
CC
NC NC NC
Private Network
CC
NC NC NC
Public Network
! Cluster Controller (CC) ! Select informa5on of free
resources ! Control the alloca5ons of the
VMs on the NCs
! Node Controller (NC) ! Controls the Xen /
KVM Hypervisor ! Sends resource
informa5on to the Cluster Controller
38 Cloud Computing | SS2011 | M.Kunze
! Open-Source implementation of Google App Engine ! Project driven by UC Santa Barbara ! App Engine is a platform service that allows to run Web apps in
Python (and JAVA) on the Google infrastructure ! AppScale works transparently on Cloud infrastructures like
Eucalyptus ! AppScale is compatible with Google App Engine ! Applications for Google App Engine can be developed and tested
in a private cloud
AppScale http://appscale.cs.ucsb.edu
Quelle: Navraj Chohan
39 Cloud Computing | SS2011 | M.Kunze
Tashi http://wiki.apache.org/incubator/TashiProposal
Cluster Manager
Node
Node
Node
Node
Node
Storage Service
Virtualization Service
Node
Scheduler
! Co-Scheduling of CPU and data
40
R&D Examples
! Job flow to run Monte-Carlo simulation ! HPC as a Service ! Cloud management services
Cloud Computing | SS2011 | M.Kunze
41
! Grid example: Run 1000 Jobs to generate 1 million Monte-Carlo events ! The Grid is a complex system
A Grid Job Flow Scenario
Cloud Computing | SS2011 | M.Kunze
42
A Cloud Job Flow looks much smarter
! Works the same like on a local computer ! Develop and debug the program locally
! Build a virtual machine with the following characteristics ! Automated program start after machine launch (e.g. Amazon EC2) ! Do the data processing (e.g. generate 1000 Monte-Carlo events) ! Write output to persistent cloud storage (e.g. Amazon S3) ! Automated shut down once finished to stop accounting
! Just instantiate the suiting number of machines in the cloud ! Simply launch 1000 machines to produce 1 million events ! Cost is the same:
! Run 1 machine 1000 hours ! Run 1000 machines 1 hour
! No need for batch queues and scheduling (“unlimited resources”)
Cloud Computing | SS2011 | M.Kunze
43
HPC as a Service
Tradi9onal HPC Architecture … ! is characterized by very specific computer clusters designed for special applica5ons ! offers pre-‐defined opera5on systems and user environments only ! serves one single applica5on at a given 5me ! provides restricted user access ! provides management privileges exclusively to administrators
Concept of HPCaaS ! Capability of using clustered servers and
storage as resource pools, fully automated management
! Individual cluster configura5on on-‐demand ! Flexibility to serve mul5ple user groups and
applica5ons with varying requirements ! Customers gain resource management
privileges
Cloud Computing | SS2011 | M.Kunze
44
Cloud Management: KOALA http://koalacloud.appspot.com
Cloud Computing | SS2011 | M.Kunze
! Mobile management of hybrid cloud resources as SaaS solution ! Plan: Develop iPad/iPhone/Android application
IaaS
PaaS
KOALA
45
! Advantages and Disadvantages of a Cloud-‐based Management of IaaS wrt. a local solu9on (e.g.: Elas9cfox, Hybridfox) ! Advantages:
! Flexibility concerning the used browser ! Support of EC2/S3/EBS and Eucalyptus ! No local installa5on necessary (except of the private key) ! Cloud installa5on corresponds to the cloud compu5ng idea
! Disadvantages: ! Users have to trust the provider of the management applica5on concerning
the data privacy and opera5on availability ! Advantages and Disadvantages of KOALA wrt. the Amazon Management Tools
(especially AWS Management Console) ! Advantages:
! Support of EC2/S3/EBS and Eucalyptus ! KOALA can run in a private cloud (via AppScale)
! Disadvantages: ! Not all AWS-‐Features implemented (5ll now) ! No support by Amazon
KOALA hSp://koalacloud.appspot.com -‐ hSp://code.google.com/p/koalacloud
Cloud Computing | SS2011 | M.Kunze
46
! AppScale runs in EC2 ! KOALA runs in AppScale ! The cloud can be managed
out of itself ! It is not necessary to store
credentials at Google
KOALA hSp://koalacloud.appspot.com -‐ hSp://code.google.com/p/koalacloud
Cloud Computing | SS2011 | M.Kunze
47
Good to know
! All course info is on http://studium.kit.edu ! http://www.math.kit.edu/mitglieder/lehre/cloudcomp2011s/
Cloud Computing | SS2011 | M.Kunze
48 Cloud Computing | SS2011 | M.Kunze