Open Cloud Consortium: An Update (04-23-10, v9)
-
Upload
robert-grossman -
Category
Technology
-
view
2.002 -
download
1
description
Transcript of Open Cloud Consortium: An Update (04-23-10, v9)
![Page 1: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/1.jpg)
Open Cloud Consortium: An Update
Robert GrossmanOpen Cloud Consortium
April 23, 2010
www.opencloudconsortium.org
![Page 2: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/2.jpg)
Part 1.
Overview of theOpen Cloud Consortium (OCC)
www.opencloudconsortium.org
2
![Page 3: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/3.jpg)
501(3)(c) Not-for-profit corporation Supports the development of standards,
interoperability frameworks, and reference implementations.
Manages testbeds: Open Cloud Testbed and Intercloud Testbed.
Manages cloud computing infrastructure to support scientific research: Open Science Data Cloud.
Develops benchmarks.
3
www.opencloudconsortium.org
![Page 4: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/4.jpg)
OCC Members
Companies: Aerospace, Booz Allen Hamilton, Cisco, InfoBlox, Open Data Group, Raytheon, Yahoo
Universities: CalIT2, Johns Hopkins, MIT Lincoln Lab, Northwestern Univ., University of Illinois at Chicago, University of Chicago
Government agencies: NASA Open Source Projects: Sector Project
4
![Page 5: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/5.jpg)
OCC Working Groups
1. Large Data Cloud Working Group2. Open Cloud Testbed Working Group.3. Intercloud Testbed Working Group4. Open Science Data Cloud Working Group
![Page 6: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/6.jpg)
Storage Services
Compute Services
Applications
Virtual Network Manager
Data Services
Network Transport
Virtual Machine Manager
Metadata Services
Identity Manager
IaaS
PaaS
Apps
![Page 7: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/7.jpg)
Part 2. Intercloud Testbed
7
Cloud 1
Cloud 2
![Page 8: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/8.jpg)
We have several cloud standards…
Infrastructure as a Service– Virtual Data Centers (VDC)– Virtual Networks (VN)– Virtual Machines (VM)
Platform as a Service– Cloud Compute Services– Data/Table Cloud Services– Cloud Storage Services
Open Virtualization Format (OVF)
Open Cloud Computing Interface (OCCI)
SNIA Cloud Data Management Interface (CDMI)
Large Data Cloud Interoperability Framework
![Page 9: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/9.jpg)
Where are the Gaps?
Infrastructure as a Service– Virtual Data Centers (VDC)– Virtual Networks (VN)– Virtual Machines (VM)– Physical Resources
Platform as a Service– Cloud Compute Services– Data as a Service
Open Virtualization Format (OVF)
Open Cloud Computing Interface (OCCI)
SNIA Cloud Data Management Interface (CDMI)
Large Data Cloud Interoperability Framework
Naming entities in IaaS & PaaS Bridging IaaS & DaaS Services that span multiple VMs, ….
![Page 10: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/10.jpg)
Bridging the Gaps…A Small Step
Infrastructure as a Service– Virtual Data Centers (VDC)– Virtual Networks (VN)– Virtual Machines (VM)– Physical Resources
Platform as a Service– Cloud Compute Services– Data as a Service
Open Virtualization Format (OVF)
Open Cloud Computing Interface (OCCI)
SNIA Cloud Data Management Interface (CDMI)
Large Data Cloud Interoperability Framework
Metadata service linking IaaS and DaaS
Metadata service naming and linking entities in the IaaS layers
![Page 11: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/11.jpg)
Part 3. Large Data Cloud Working Group
11
Standards for integrating and interoperating large data cloud services such as those provided by Hadoop and similar systems.
![Page 12: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/12.jpg)
Focus of Working Group
12
Cloud Storage Services
Cloud Compute Services (MapReduce, UDF, & other programming frameworks)
Table-based Data Services
Relational-like Data Services
App App App App App
App App
App App
Developing APIs for this framework.
![Page 13: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/13.jpg)
Benchmarks for Large Data Clouds
Until recently, the only benchmark used was Terasort (sorting 10 billion 100 byte records)
Replaced by Gray Sort and Minute Sort Gray Sort tries to maximize TB / min sorted on
100 TB or more of data. Hadoop holds the current Gray Sort and
Minute Sort records. Problem: sort is just one of the types of work
load for analytic applications
![Page 14: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/14.jpg)
MalStone
MalGen – generates synthetic data with realistic distributions.
MalStone A & B – “stylized” computations that can be used as benchmarks for architectures, software and systems for large data clouds.
Open source and available at malgen.googlecode.com
14
![Page 15: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/15.jpg)
Part 4. Open Cloud Testbed
![Page 16: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/16.jpg)
Condominium Clouds In a condominium cloud, you buy your own rack
or bunch of racks. The racks are managed and operated by the
condominium association, in this case the OCC. If your rack is 120 TB, you get the rights to c. 40
TB of storage in the cloud. The rest is a shared resource.
The Open Cloud Testbed is a condo cloud managed by the OCC.
16
![Page 17: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/17.jpg)
Open Cloud Testbed
Phase 2 9 racks 250+ Nodes 1000+ Cores 10+ Gb/s
Phase 3 (2011) – we will stand up some 100 Gb/s links.
MREN
CENIC Dragon
Hadoop Sector/Sphere Thrift KVM VMs Eucalyptus VMs
C-Wave
![Page 18: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/18.jpg)
Part 5. Open Science Data Cloud Working Group
18
![Page 19: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/19.jpg)
Open Science Data Cloud
19
Astronomical dataBiological data (Bionimbus)
Networking data
Provide a long term home for selected scientific data sets and support elastic cloud-based analysis & integration of the data.
![Page 20: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/20.jpg)
Part 6. Image Processing for Disaster Relief Using Elastic Clouds
![Page 21: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/21.jpg)
The Challenge
When a disaster strikes, there is usually an immediate and critical need for computing power to process images.
Example, there was a delay getting current images of Haiti to non-government organizations (NGO) after earthquake on January 12, 2009.
![Page 22: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/22.jpg)
The Idea …The OCC Elastic Cloud for Disaster Relief
Set up a permanent elastic cloud that is available to assist with disaster relief.
Establish connections to sources of images that can be enabled at times of need.
Set up a network of volunteers with accounts on the cloud and knowledge of the tools that can swarm when needed.
Use as a test of large data cloud standards and interoperability.
![Page 23: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/23.jpg)
Image Processing on Large Data Clouds
Data parallel applications– Parallelism is often required at file or directory level– Data locality is important– Parallel disk IO is also critical
Requirements– The input data size can be at 10+ TB per day– Want to integrate with open source libraries such as
OSSIM
![Page 24: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/24.jpg)
Distributed File Systems & Image Processing
Sector is broadly similar to the Hadoop Distributed File System
Main differences– Hadoop directly implements a distributed block based file
system– Sector is a layer over a native file system
Sector does not split files– A single image will not be split, therefore when it is being
processed, the application does not need to read the data from other nodes via network
– A directory can be kept together on a single node as well, as an option
![Page 25: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/25.jpg)
Get Involved…
… Join our volunteer effort.
![Page 26: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/26.jpg)
Part 7. Virtual Networks
![Page 27: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/27.jpg)
How Long Does It Take…
![Page 28: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/28.jpg)
… To Move A Cloud Application Spanning Multiple VMs Between Clouds?
![Page 29: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/29.jpg)
… To Add A New Rack to a Cloud Service?
![Page 30: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/30.jpg)
… To Add Another Public Cloud to A Private/Public Cloud?
![Page 31: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/31.jpg)
We Have Several Ways of Defining Virtual Networks….
VN-Link
VLAN
VPNs
BGP
MPLSOpenFlow
Open vSwitch
vSwitchCloudSwitch
![Page 32: Open Cloud Consortium: An Update (04-23-10, v9)](https://reader035.fdocuments.in/reader035/viewer/2022081602/55510c10b4c9057b478b5619/html5/thumbnails/32.jpg)
But No Vendor Neutral VN Standard That
That scales to 100,000+ VMs Supported by multiple vendors Spans multiple physical switches Supports VN Mobility Provides strong isolation of VN Is easy for VMs to join and leave VNs Includes management interfaces ….