Convergence Characteristics for Clusters, Grids, and P2P

networks

Rajkumar BuyyaGrid Computing and Distributed Systems

(GRIDS) Lab. The University of MelbourneMelbourne, Australiawww.gridbus.org

WW Grid

Panel at the P2P conference, Linkoping, Sweden

2

Where Did I come From ?

*

*

*

*

Melbourne

Stockholm+Linkoping

Singapore

London

3

GRIDS Lab @ the U. of Melbourne, The Gridbus Project:

www.gridbus.org Conduct R&D in Distributed Computational Economy for

Service-Oriented Computing: Architecture, Specification, and Open Source Reference

Implementation in in Collaboration with Any Global Peers. Platform: Cluster, Grid, P2P for diverse applications.

Distributed Application Composition Tools Grid Economy & Scheduling (via Nimrod-G Broker) GridSim Toolkit: Grid Modeling and Simulation (Java based): Libra: Economic Cluster Scheduler Grid Bank: Accounting and Transaction Management World Wide Grid (WWG) – A Global Testbed P2P/Jxta-based Compute Power Market Application Enabler Projects:

Virtual Laboratory Toolset for Drug Design High-Energy Physics and the Grid Network (HEPGrid) Brain Activity Analysis on the Grid Future: GridEmail for Spam Management

—read only if sender pays!

4

Books

5

Technology Evolution: Cluster, Grid, P2P

6

2100

2100 2100 2100 2100

2100 2100 2100 2100

Desktop SMPs or SuperComputers

LocalCluster

GlobalCluster/Grid

PERFORMANCE

Inter PlanetaryGrid!

•Individual•Group•Department•Campus•State•National•Globe•Inter Planet•Galaxy

Administrative Barriers

EnterpriseCluster/Grid

?

Scalable Seamless Computing: Breaking Administrative Barriers

7

Basic Elements

Security

Resource Allocation & Scheduling

Data locality

Network Management

System Management

Resource Discovery

Uniform Access

Computational Economy

Application Development Tools

8

Cluster Architecture

Sequential Applications

Parallel Applications

Parallel Programming Environment

Cluster Middleware

(Single System Image and Availability Infrastructure)

Cluster Interconnection Network/Switch

PC/Workstation

Network Interface Hardware

Communications

Software

PC/Workstation

Network Interface Hardware

Communications

Software

PC/Workstation

Network Interface Hardware

Communications

Software

PC/Workstation

Network Interface Hardware

Communications

Software

Sequential Applications

Sequential Applications

Parallel ApplicationsParallel

Applications

9

Cluster Architecture

Sequential Applications

Parallel Applications

Parallel Programming Environment

Cluster Middleware

(Single System Image and Availability Infrastructure)

Cluster Interconnection Network/Switch

PC/Workstation

Network Interface Hardware

Communications

Software

PC/Workstation

Network Interface Hardware

Communications

Software

PC/Workstation

Network Interface Hardware

Communications

Software

PC/Workstation

Network Interface Hardware

Communications

Software

Sequential Applications

Sequential Applications

Parallel ApplicationsParallel

Applications

10

A Typical Grid Computing Environment

Grid Resource Broker

Resource Broker

Application

Grid Information Service

Grid Resource Broker

databaseR2R3

RN

R1

R4

R5

R6

Grid Information Service

2

11

A Typical Grid Computing Environment

Grid Resource Broker

Resource Broker

Application

Grid Information Service

Grid Resource Broker

databaseR2R3

RN

R1

R4

R5

R6

Grid Information Service

2

12

A Typical P2P Computing Environment

Peer Agent

Peer Agent

Application

Peer Discovery Service

Peer Agent

P1

P3

pN

P2

R7

Who

can

hel

p ?

Peer

P 2, P

7 can

hel

p!

pM

Request

Response

p4 p5

RequestSorry, I am busy.

13

A Typical P2P Computing Environment

Peer Agent

Peer Agent

Application

Peer Discovery Service

Peer Agent

P1

P3

pN

P2

R7

Who

can

hel

p ?

Peer

P 2, P

7 can

hel

p!

pM

Request

Response

p4 p5

RequestSorry, I am busy.

14

Cluster, Grid, P2P: Characteristics

Characteristic Cluster Grid P2PPopulation Commodity

ComputersHigh-end computers Edge of network

(desktop PC)

Ownership Single Multiple Multiple

Discovery Membership Services

Centralised Index & Decentralised Info

Decentralized

User Management Centralised Decentralised Decentralised

Resource management

Centralized Distributed Distributed

Allocation/Scheduling

Centralised Decentralised Decentralised

Inter-Operability VIA based? No standards yet No standards

Single System Image Yes No No

Scalability 100s 1000? Millions? [@Home]

Capacity Guaranteed Varies, but high Varies

Throughput Medium High Very High

Speed(Lat. Bandwidth)

Low, high High, Low High, Low

15

Nimrod-G: Modeling Distributed Computing in Economic Terms

Grid Middleware

Nimrod/G Client Nimrod/G ClientNimrod/G Client

Grid Information Server(s)

Schedule Advisor

Trading Manager

Nimrod-G Engine

GridStore

Grid Explorer

GE GISTM TS

RM & TS

Grid Dispatcher

RM: Local Resource Manager, TS: Trade Server

Globus, Legion, Condor, etc.

G

G

CL

Globus enabled node.Legion enabled node.

GL

Condor enabled node.

RM & TSRM & TS

C L

16

On-Demand Computing

On-Demand Computing

CostCostDeadlineDeadline

Legion hosts

Globus Hosts

Bezek is in both Globus and Legion Domains

Arlington

Alexandria

Richmond

HamptonNorfolk

Virginia BeachChesapeakePortsmouth

Newport News

Roanoke

Ap p om a toxRive r

Ja m esRive r

Shena nd oa hRive r

Ra p p a ha nnoc kRive r

Potom a cRive r

VIRGINIA77

81

64

64

66

85

17

Active Sheet Nimrod-G:Transparent Spreadsheet Processing from PCs to

Clusters to World-Wide Grids

NimrodNimrodProxyProxy

Nimrod-GNimrod-G

World-Wide Grid

18

MEG(MagnetoEncephaloGraphy) Data Analysis on the Grid: Brain Activity

Analysis

Life-electronics laboratory,AIST

Data Analysis

•Provision of expertise in the analysis of brain function•Provision of MEG analysis

Data Generation

Nimrod-G

64 sensors MEG

Results

Analysis All pairs (64x64) of MEG data by shifting the temporal region of MEG data over time: 0 to 29750: 64x64x29750 jobs

World-Wide Grid•[deadline, budget, optimization preference]

1

5

4

3

2

[Collaboration with Osaka University, Japan]

Backup Slides

20

Evolution Time Line

* Sputnik

1960 1970 1975 1980 1985 1990 1995 2000

* ARPANET

* Email* Ethernet

* TCP/IP* IETF

* Internet Era * WWW Era

* Mosaic

* XML

* PC Clusters* Crays * MPPs

* Mainframes

* HTML

* W3C

* P2P

* Grids

* XEROX PARC wormCO

MP

UTIN

GN

ETW

OR

KIN

G

* Web Services

* Minicomputers

* PCs

* WS Clusters

* PDAs* Workstations

* HTC

21

What is Cluster/Grid ?

A type of parallel and distributed system that enables the sharing, selection, & aggregationof resources distributed in _____/_______ administrative domains depending on their availability, capability, performance, cost, and users quality of service requirements.

A Single

Multiple

Cluster Grid

22

CPM: DC Economy-based P2P Computing

(Jxta based Implementation)

Accounting

Bill

Resources

(Provider)

User (Consumer)

CPM Agent

Trader

Market Server

Market Repository

Job Management

- Discovery

- Membership

23

Drug Design: Data Intensive Computing on Grid

It involves screening millions of chemical compounds (molecules) in the Chemical DataBase (CDB) to identify those having potential to serve as drug candidates.

Protein

Molecules

Chemical Databases(legacy, in .MOL2 format)

[Collaboration with WEHI for Medical Science, Melbourne]

24

Virtual Drug DesignA Virtual Lab for “Molecular Modeling for Drug Design” on P2P Grid

“Screen 2K molecules in 30min. for $10”

Grid Market Directory

ResourceBroker

Grid Info. Service

GTS

GTS

GTS

GTS

“Give me list PDBs sourcesOf type aldrich_300?”

“serv

ice co

st?”

(GTS - Grid Trade Server)

PDB2

“get mol.10 from pdb1 & screen it.”

Data Replica Catalogue

“service providers?”

GTS

PDB1

“mol.10 please?”

“mol.5 please?”

(RB maps suitable Grid nodes and Protein DataBank)