From Grid Computing to Cloud Computing – The IBM Approach · After Cloud Computing...
Transcript of From Grid Computing to Cloud Computing – The IBM Approach · After Cloud Computing...
© 2005 IBM Corporation
IBM India
From Grid Computing to Cloud Computing – The IBM ApproachGaruda Partner Meet ,4th March 2008,Bangalore,India
P. Sambath Narayanan Ph.DIndia Systems & Technology LabIBM
2
Agenda
I. Important Challenges
III. Important Technologies, Trends and Standards
V. Case Studies
VII.From GRID Computing to Cloud Computing
IBM India
© 2005 IBM Corporation
Terminology
Virtualization Service Orientation Data management information Data Service Policy Management Interoperable Automation Lifecycle
IBM India
© 2005 IBM Corporation
I. Some Important Challenges
Data Management – Right Time & Right Data
Network Bandwidth and Latency
Security
Software and Standards
Need for many Grid based Scientific and Commercial Applications
Enable smooth scaling in many dimensions
Integration with the physical world
IBM India
© 2005 IBM Corporation
IBM India
© 2005 IBM Corporation
IBM India
© 2005 IBM Corporation
Few Important Challenges(contnd.)
Data Management is a Challenge Diverse usage scenarios
Volume of data - TBs
Right data at right time
Format of data
Heterogenity of systems at all level
Bandwidth, transfer, manipulation and analysis of large volume of data
IBM India
© 2005 IBM Corporation
Few Important Challenges (contnd.)
Network Bandwidth
Large volume of data needs to be transferred across the network
Ensuring right data to be available at the right time
Latency, Bandwidth, transfer, manipulation and analysis of large volume of data
Cost of Bandwidth
IBM India
© 2005 IBM Corporation
Grid Security
Intrusion Detection
Anti VirusManagement
PolicyManagement(Auth,Privacy,federation
User Managmnt.
KeyManagmnt
Tru
st Mod
el
Secu
re logg
ing
Policy Expression and exchange
Bindings security(transport, protocol,messagesecurity
Secure conversations
Credential & Id Translation
Access controlenforcement Audit
Service/end-pointpolicy
Mappingrules
AuthorizePolicy
PrivacyPolicy
IBM India
© 2005 IBM Corporation
II. Key Technologies
Virtualization
Storage / Data Management
Grid Security
Grid Software
IBM India
© 2005 IBM Corporation
Keywords
Virtualization Service SOA Data management information Data Service Policy Management Interoperable Automated Lifecycle
IBM India
© 2005 IBM Corporation
Grid Technology Evolution
OGSA
Managed,SharedVirtual System
Globus Toolkit
Standards,GT2Many Deployments
Many DeploymentsScientific Applications
1990 1995 2000 2005 2007
Grid
Ad
optio
n &
A
ccep
tanc
e
IBM India
© 2005 IBM Corporation
Grid Open Standards
OASIS (organization for the advancement of structured information standard)
WS – Resource Framework
WS – Notification
Open Grid Services Architecture-GGF OGSA Basic Profile
OGSA Security Profile
Basic Execution Services (OGSA-BES)
Job Submission Description Language (JSDL)
Data Access and Integration Services (DAIS)
Configuration Description, Deployment, and Lifecycle Management (CDDLM)
OGSA Byte I/O (Byte IO)
IBM India
© 2005 IBM Corporation
OGSA Design Principles
Service Orientation to virtualize resources Everything is a service
From Web service Standard interface service mechanisms, multiple protocols
bindings, local/remote transparency
From Grids Service semantics, reliability and security models
Life cycle management, discovery and other services
Multiple hosting environments C,J2EE,.NET
IBM India
© 2005 IBM Corporation
Technology Classification & Trends
Serial Applications
Parallel Applications
•Multi-threaded
•MPI
•open
Client Server
•CORBA
•COM/DCOM
•.NET, J2EE
•Home grown work
• distribution
P2P
•App Integration
•Reliable Messaging
•Reliable execution
Service virtualization
•Web services
•Service registration,
Discovery, invocation
•Location independent
•Lift App off the servers
Mainframes
Storage
•DAS
Open Systems
•Unix, Linux,Windows
Storage
•DAS
Clusters
•DRM
Storage•DAS
Infra. Virtualization
•Grid
•OGSA
•Data Grid
•Service provisioning
Infrastructure
Application technologies
Monolithic Open Distributed Virtualized
16
Virtual I/O paths
Virtualization-Single system & partitioning
Features Micro-partitioning
Share processors across multiple partitions
Minimum Partition: 1/10 processor
AIX 5L V5.3 or Linux*
Virtual I/O ServerShared EthernetShared SCSI & Fiber ChannelInt Virtualization ManagerAIX 5L V5.3 & Linux partitions
Partition LoadManagerAIX 5L V5.2 & V5.3 supportedBalances Processor & memory request
Partition Mobility
* = SLES 9 or RedHat v3 with update 3
POWER Hypervisor
LinuxAIX 5L
V5.2
Dynamically Resizable
3 Cores
2Cores
ManagerServer
LPAR 2AIX 5L V5.3
LPAR 1AIX 5L V5.2
LPAR 3Linux
PLM PartitionsUnmanaged Partitions
POWER Hypervisor
PLM Agent PLM Agent
AIX 5L
V5.3
6Cores
6 Cores
Lin
ux
AIX
5L
V5.
3
AIX
5L
V5.
3
AIX
5L
V5.
3
AIX
5L
V5.
3
AIX
5L
V5.
3
Lin
ux
Micro-partitioning
1Cores
Linux Linux
3 Cores
AIX 5L
V5.3
3Cores
EthernetSharing
Virtual I/O Server
Partition
StorageSharing
Int VirtManager
IBM India
© 2005 IBM Corporation
Virtualization – Information/StorageTechnology
Helps in addressing Data Management Challenges Integrated view of storage, fs and DB driven by
standard
Data transformation, security and replication
IBM India
© 2005 IBM Corporation
Virtualization - WorkloadTechnology
Workload Management Challenge
Single logical view of workload scheduling
Different type of scheduling environments and domains
Workload virtualization strategy is to create a single, logical view of workload scheduling.
This will enable users to accelerate performance of multiple large application workloads across their organization, leveraging and orchestrating IT resources in a flexible and dynamic fashion.
IBM India
© 2005 IBM Corporation
Virtualize Unlike Resources
Virtualize the
Enterprise
Virtualize Outside the Enterprise
Virtualize Like Resources
Single Systems& Partitioning
Cluster
Simple(2-4)
Sophisticated(4+)
Workload Management
SAN Volume Controller
Information Integrator
GPFS
NFS V4
SAN FS
Integrated Cluster Environment
XDExtended Deployment
LoadLeveler
LoadLeveler MultiCluster
CSMCluster Systems Manager
IntelligentOrchestrator
Systems Edition
Enterprise Edition
Provisioning Manager
EnterpriseWorkloadManager
Information
IBM Grid Toolbox
LSF
GridServer
Symphony
MultiCluster
MP Enterprise
Who does virtualization
IBM India
© 2005 IBM Corporation
Overcoming Network Challenges
Through efficient utilization of the Network. IBM Download Grid example. Explained in later slides.
Integrating with Global Research Networks
National Research & Education Networks. Supporting Research and Education communities
Specialized ISP
Lambda Grid
IBM India
© 2005 IBM Corporation
IBM India
© 2005 IBM Corporation
Overcoming Grid Security Challenges
Three key attributes of Grid Security Model Enables integration and interoperability
Creation and management of dynamic trust domains
Supports dynamic creation of services
OGSA Security Web services security standard
Grid Security Infrastructure (GSI) Portion of the Globus tool kit that implements security function
IBM India
© 2005 IBM Corporation
Virtualize Unlike Resources
Virtualize the Enterprise
Virtualize Outside the Enterprise
Virtualize Like Resources
Single Systems & Partitioning
Cluster
Simple(2-4)
Sophisticated(4+)
China GridMinistry of Education
People’s Republic of China
National Digital Mammography Archive
IBM
Across the Spectrum: Real Life References
IBM India
© 2005 IBM Corporation
Earth System Grid(ESG) - Case Study Overcoming Data Management Challenges
Service = RepositoryStorage Repository for Model generated atmospheric data3200 Users91,000 filesMore than 150 TB of data downloadedMore than 300 research papers
0
100
200
300
400
500
600
11/1
/04
12/1
/04
1/1/
05
2/1/
05
3/1/
05
4/1/
05
5/1/
05
6/1/
05
7/1/
05
8/1/
05
9/1/
05
10/1
/05
11/1
/05
12/1
/05
1/1/
06
2/1/
06
3/1/
06
4/1/
06
5/1/
06
6/1/
06
7/1/
06
8/1/
06
9/1/
06
10/1
/06
GB
/da
y
Daily 7-Day Average
GB
/day
600
IBM India
© 2005 IBM Corporation
ESG Architecture & Technologies
Climate data Metadata catalog
NcML (metadata schema)
OPenDAP-G (aggregation and subsetting)
Data management Data Mover Lite
Storage Resource Manager
Globus toolkit Globus Security Infrastructure
GridFTP
Monitoring and Discovery Services
Replica Location Service
Security Access control
MyProxy
User registration
DataSubsetting
AccessControl
UserRegistration
OPeNDAP-GMyProxy SRM DISKCache
ESG Web Portal
NCARCache
NCARMSS
RLS SRM
ORNLHPSS
RLS SRM
RLS
SRM
RLS LANLCache
searchbrowse
download
WebBrowser
DML DataUser
publishWebBrowser
DataProvider
MonitoringServices
DataPublishing
ClimateMetadata
CatalogsBrowsing
UsageMetrics
DataDownload
DataSearch
NERSC
MSS, HPSS: Tertiarydata storage systems
IBM India
© 2005 IBM Corporation
Energy Exploration – Case Study
Service = Seismic Computing 3-D Seismic imaging is the most resource intensive Grid Enabled system for Seismic Imaging Gulf of Mexico 3-D Marine Surveys Estimated run times on a cluster(128 cpu,2.4 GHz,Pentium) Compute intensive wave equation provides better accuracy
Slides are based on the work done by 3DGeo Team. See Reference Material
IBM India
© 2005 IBM Corporation
Parallelization of PSDM on Multiple Clusters
Clusters from 3DGeo processing centres and Clusters from SDSC
MPICH-G2 / MPICH-GP(Kum Rye Park)
Globus Tool kit DCs –
SantaClara,Houston,SanDiego Supercomputing Centre
This slide is based on the work done by 3DGeo Team. See Reference Material
IBM India
© 2005 IBM Corporation
Medical Education Over Access GridWork Done by J.Silverstein, U. Chicago
IBM India
© 2005 IBM Corporation
National Digital Mammography ArchiveElectronic Medical Record data grid and repository
Motivation To help doctors and medical students learn more about breast
cancer and related diseases Challenges
Managing and storing of huge files for fast retrival
Annual NDMA volume could exceed 5.6 peta bytes per year– Image size 160 MB per study
Minimum daily traffic estimated 28 TB
NETwork bandwidth and response
Encryption of patient data and transmission across public networks
IBM India
© 2005 IBM Corporation
caBIG: sharing of infrastructure, applications, and data.
DataIntegration!
Service Oriented Science – Cancer & Biology
IBM India
© 2005 IBM Corporation
IBM India
© 2005 IBM Corporation
Technology Evolution
OGSA
Managed,SharedVirtual System
Globus Toolkit
Standards,GT2Many Deployments
Many DeploymentsScientific Applications
1990 1995 2000 2005 2007
Grid
Ad
optio
n &
A
ccep
tanc
e
IBM India
© 2005 IBM Corporation
Business Challenges
With demand for IT resources hard to predict, service providers usually over-provision resources in order to support peak demands and ensure continuous service availability and quality, while other systems run at lower capacity,
IBM India
© 2005 IBM Corporation
Cloud Computing Defined
Large pools of systems are linked together to provide IT services
Service-based online economy resources and services are transparently provisioned
and managed.
IBM India
© 2005 IBM Corporation
The Need - Cloud Computing
Dramatic growth in connected devices
Real-time data streams
the adoption of service oriented architectures
Web 2.0 applications
Open collaboration, social networking and mobile commerce.
Massive increase in the scale of IT environments
driving the need to manage them as a unified cloud.
IBM India
© 2005 IBM Corporation
Business SolutionAfter Cloud Computing
Cloud-computing-based technologies that will enable the borderless delivery of IT services based on actual demands to keep costs competitive.
Seamless delivery of services to consumers regardless of demand or available computing resources
Virtualization and Grid Technologies
IBM India
© 2005 IBM Corporation
Cloud Computing Example
Delivery of online entertainment. Distribution of television shows, movies and other videos
are moving to the Web the cloud computing technologies would enable a
network of service providers to host the different media. Using cloud computing technology, the broadcasters can join forces to reach a service cooperation contract that enables them to tap into advanced services including content distribution, load balancing, and overlay networking across different platforms in different countries.
If there is large demand for a show hosted by a particular site, it can dynamically 'hire' additional servers and services from other sites that are not being used.
IBM India
© 2005 IBM Corporation
Blue Cloud
Series of cloud computing offerings
Allow corporate data centers to operate more like the Internet
Enable computing across a distributed, globally accessible fabric of resources, rather than on local machines or remote server farms.
IBM India
© 2005 IBM Corporation
Reference Material
A Virtualization Experience: IBM Worldwide Grid Implementation, Moon Kim et al., IBM Red Books, IBM
Grid 2, Edited by Ian Foster & Carl Kesselman,Elsevier,2004 Grid computing for energy exploration,
D.Beve,S.E.Zarantonello,N.Kaushik,I.Musat
IBM India
© 2005 IBM Corporation
Summary
1) Storage, Security, Network and Application availabilty are major Grid challenges
2) Virtualization is an important technology for the Grid3) Many large Grid projects have been working successfully4) Think of Grid for variety of services, not just for computing
alone5) Grid, virtualization and service orientation have many things in
common
© 2005 IBM Corporation
IBM India
From Grid Computing to Cloud Computing – The IBM ApproachGaruda Partner Meet ,4th March 2008,Bangalore,India
P. Sambath Narayanan [email protected]