The Grid

”Enter the GRID”

af Kristian Mandrup

Indeks Intro Overview Architecture Solutions Future Conclusions & discussion

What is it ? The Next-Gen Internet?  A 21st century time machine?

Intro Future of collaborative problem-

solving Internet's next evolutionary step The Grid is a new class of

infrastructure Link computers in new ways Open up storage and transaction

power as Web opened up content

Intro (2) Era of distributed, networked

computing is just beginning The WWW a taste, the Grid a vision Answer to the enterprise

computing crisis (ECC)

Vision Applies interconnected model used by power utilities to access

services, software and hardware resources as part of virtual supercomp.

Executes jobs on best suited, least loaded systems in a seamless, transparent and secure way

On-demand access to computational power, data bases and services

Manage resource sharing and co-ordinated problem solving across dynamic, multi-institutional virtual organisations both in eScience and eBusiness

Provides scalable, secure, high-performance mechanisms for discovering and negotiating access to remote resources

Geographically distributed groups can work together in new ways

Background (evolution) Breakthrough technologies

Begun in the research environment Moved to open standards Applied to business applications

What we are seeing with Grid standards

Background (history) Desire to connect supercomputers into

"metacomputers" that could be remotely controlled

Vision of the Grid started in 1960s Envisioned a computer facility operating "like

a power company or water company" Word "grid" borrowed from the electricity grid Any compatible device could be plugged in

anywhere on the Grid and be guaranteed a certain level of resources, regardless of where those resources might come from

Evolution 1G Grids

Involved local "meta-computers" with basic services such as distributed file systems and site-wide single sign on.

1G Grids were totally custom made 2G Grids

Underlying software services and communications protocols Grids offered basic building blocks, but deployment involved significant customization

Interoperability among 2G Grid systems very difficult 3G Grids

Solves deployment and interoperability issues by providing standard interfaces

Today it feasible to realize the Grid vision Global Grid Forum (GGF) created in November, 2000

Demand Science & Industry

High-energy physics, needs extra resources to manage and analyze huge amounts of data

Science and industry participants require level of reliability not offered by current peer-to-peer initiatives

Strong need to efficiently manage availability of distributed infrastructures, applications and services

Computational resources are failing to keep up with what scientists demand of them

Demand (technical) Doubling periods (months)

Network bandwidth 9 Storage capacity 12 Computing power 18

Computer power is falling behind storage !

Demand (example) Scientists create high-resolution

simulations need petabyte archives CERN's Large Hadron Collider (LHC) will

produce multiple petabytes (1015 byte) of data per year

Scientists demand 10+ Gb/s to work remotely on petabyte data sets

Law of diminishing returns ???

Demand (solutions) If communication is unlimited and free

Not restricted to using local resources to solve problems

Use collective computing power of research collaboration or buy from provider

Look at large datasets using special collaboration and visualization tools

Use remote resources to do things not possible using local resources

Benefits Aggregates compute power and delivers

it as a network service Grid Engine presents users to a seamless,

integrated computing capability Facilitate the deployment of compute

farms, the basic building blocks of grid computing

Making large amounts of compute power available for applications and users

Benefits ”sales talk” Raise productivity Maintain availability Minimize downtime Shorter time to market Reduces costs by better utilisation of

resources Quicker and better results Increased quality and innovation Do things not possible before Increased ROI (Return On Investment)

Potential problems Social and political dimensions

(like WWW) Sharing between strangers where

no history of trust

Uses Development of semiconductors Bioinformatics Mechanical design Software development Oil/gas exploration Financial analysis Academic and research pursuits

Architecture (Infrastructure) Open Grid Services Architecture (OGSA)

Integration of Grid and Web services technologies

Open Grid Services Infrastructure (OGSI) Grid Resource Access and Management

(GRAM) protocol and service Remote resource allocation and process

creation Monitoring Management services

Architecture (OGSA) Open Grid Services Architecture

Establish standard interfaces and behaviours for distributed system management

Management of service instances (persistent or transient) Defines fundamental WDSL interfaces: to establish a Grid

service in the open source Global Toolkit 3.0 (GT3) Grid service instance: maintains a set of service data

elements by encapsulating XML fragments in standard containers

FindServiceData operation: queries this information and allows notification of service existence and modifications in service

Includes GT3 (Global Toolkit 3) Core and Base Services

Architecture 1) physical devices or resources 2) Core communication and

authentication protocols cryptographically secure mechanisms - verifying identity of users and resources

3) Protocols, services, and APIs Implement interactions across

collections of resources Directory and brokering services for

resource discovery and allocation Monitoring and diagnostic services Data replication services Membership and policy services

4) User applications

Security Unlike the Web, the Grid is being designed from the

ground up as a secure system Accept only messages coming from special hosts

and reserved ports Integration with Kerberos5 and DCE exists

Authentication, authorization, and policy Client and a server need to mutually authenticate

each other. No distinction between client and server. Server

one moment, client another moment. Special requirements for managing transaction

Security (method) Single sign-on: Via creation of a proxy credential Mapping to local security mechanisms: Grid

security infrastructure maps to local solutions at each site

Delegation: Sub-computations created at sites A and B. Both communicate with each other and access files at site C

Community authorization and policy: infeasible for each resource to keep track of community membership and privileges. Group membership identified with cryptographic credential issued by trusted third party

Security (how it works) User calls on computational

resources of sites A and B Communicate with each other ,

read files located at site C. Each step requires authorization

and authentication Mediating requests requires the

Grid Security Infrastructure (GSI)

Provides: Single sign-on Run-anywhere authentication

service Support for delegation of

credentials to sub-computations Local control over authorization Mapping from global to local

user identities

Implementation requirements Implementing architecture requires

uniform mechanisms Creating and managing services on remote

computers Supporting single sign-on to distributed

resources Transferring large datasets at high speed Forming large distributed virtual communities Maintaining information about existence, state,

and usage policies of community resources

Solutions Sun Microsystems acquired Gridware, a private

developer of Distributed Resource Management (DRM) software, in July 2000

Becomes Grid Engine project Grid Engine project goals:

New open standards for DRM Standard API for application integration

Grid Engine Portal (GEP) Provides a Java based capability for enabling highly secure internet access

to applications that run on an existing Grid Engine grid Loosely coupled to Grid Engine, SunONE Portal Server

Globus Toolkit (1996) Standards-based protocols for distributed system management for open

source implementation

Using the Grid Steps to take

Discover resources exist. Negotiate access to resources Configure hardware/software to use

resources Avoid compromising security of self or

remote resources

How it works Obtaining: authentication credentials Querying: Information system and replica catalog to

determine availability of computers, storage systems, and networks, and location of required input files (collective services)

Submitting: requests to appropriate computers, storage systems, and networks to initiate computations, move data, and so forth (resource protocols)

Monitoring: the progress of the various computations and data transfers, notifying the user when all are completed, and detecting and responding to failure conditions (resource protocols)

Status Grid Engine software has been

ported to many operating systems, including Solaris, Linux

Current v.5.4 can be downloaded

The Future The Grid may give birth to a global

file-swapping network or a members-only citadel for moneyed institutions

The future of the Grid is unknown !

Conclusions & discussions Is this the distributed systems

utopia ? Is anything missing ? What is the next step after ”the

Grid” – ”the Matrix” ??? What is ”the Matrix” ? CM-systemers rolle ?