Distributed Virtual Computer (DVC):Distributed Virtual Computer (DVC):Simplifying the Development of High-Performance Simplifying the Development of High-Performance

Grid ApplicationsGrid Applications

Nut Taesombut and Andrew A. Chien

Department of Computer Science and EngineeringUniversity of California, San Diego

Workshop on Grids and Advanced Networks (GAN’04)Chicago, Illinois

April 22, 2004

04/19/23 2

OutlineOutline

• Background and Motivation• Distributed Virtual Computer (DVC)• Example Application• Related Work• Summary and Future Work

04/19/23 3

Emerging Opportunity of Lambda GridsEmerging Opportunity of Lambda Grids

• Network Advances and Trends– DWDM optical paths (or lambdas) enable

– Dedicated High Bandwidth

– Dynamic Configuration

• Lambda Grid – Distributed, shared resources interconnected by plentiful lambdas

– Configurable Connections and Capacity

– Deterministic Communication Performance

– Novel Communication Capabilities (e.g. optical multicast)

04/19/23 4

OptIPuter ProjectOptIPuter Project

• OptIPuter – Large-scale Research Project on Impact of Lambdas on System Software and next-generation E-science– International Testbed for Experimentation (UCSD, UIC, UCI, Amsterdam, etc.)

– Leading E-science Drivers (Neuroscience, Geophysical/Earth Sciences)

– 3-D Data Analysis, Visualization and Collaboration Applications

– Data-intensive and Real-time, Distributed data sources/sinks

– Wealth of Innovative System Software Research (protocols, DVC, storage, etc.)

Smarr, Papadopoulos, Ellisman, Orcutt, Chien – UCSDDeFanti, Leigh - UIC

http://www.optiputer.net

04/19/23 5

MotivationMotivation

• Building Grid Applications is Difficult!– Applications must deal with complexity of resource environment

– Resource Heterogeneity, Performance, Communication

– Multi-Organization Security, Resource Management

– Shared and Untrusted Resource Environments

– Low-Quality Networks

• Adding Low-level Management of Network Complicates the Task– No Uniform Interfaces (routers, switches, end nodes)

– Wildly Different Semantic Level (BELOW IP!)

– Novel Capabilities (e.g. multicast, RDMA, etc.)

• Key Requirements– A new abstraction which simplifies Grid environments

– A view which integrates communication as first class

04/19/23 6

Example of Grid ComplexityExample of Grid Complexity

• Access to Resources Across Multiple Namespaces

• User-Controlled Configuration of Dynamic Lambda Network

• Heterogeneous Communication

Internet

10.1.2.61

10.1.2.68

10.1.3.34

http://zebra.ucsd.edu:5220

http://intania.uic.edu:4566

http://dream.uci.edu:4010

04/19/23 7

Distributed Virtual Computer (DVC)Distributed Virtual Computer (DVC)

• A Simple Execution Environment for Grid Applications– Set of LambdaGrid resources (connections, resources)

– Naming, access, and management services

– Transparently shared amongst Applications

• Simplify Use of Network and Grid Capabilities– Automate compute/data resource binding

– Automate dynamic λ-configuration; expose novel λ-capabilities

– Leverage existing Grid Technologies (Globus, NWS etc.)

DVCDVC

04/19/23 8

DVC Design Principles DVC Design Principles

• Separate Resource Config/Mgmt and Application Programming– Resource Environments Configured to Spec

– Applications simply use them

• Aggregate and Bind Grid Resources; Present as Workgroup– Central resource control

– Single namespace

– Unified resource access mechanisms

– Trusted and secure environment

– Controllable performance

• Enable Collective Resource Views – Unified naming structures (e.g. collective names)

– Collective properties (e.g. group communication, trust, access control)

04/19/23 9

Example: Locally Simplified Grid ProgrammingExample: Locally Simplified Grid Programming

• Single Control Domain• Unified Naming Mechanism• Uniform Use of Different Communication Mechanisms (e.g. protocols)

DVC Domain

comp3

str1

comp2

comp1

Simple View of DVC

Internet

10.1.2.61

10.1.2.68

10.1.3.34

http://zebra.ucsd.edu:5220

http://intania.uic.edu:4566

http://dream.uic.edu:4010

04/19/23 10

How DVC’s Simplify Application Grid ProgrammingHow DVC’s Simplify Application Grid Programming

• Automate Resource Binding and Configuration– Reduce user interaction through resource broker and manager

• Unify Resource Naming and Access Mechanisms– Hide heterogeneity through simple abstractions

• Transparently Enable Security Protection– Implement cryptographic operations at the middleware layer

• Monitor and Control Resource and Communication Performance– Detect asynchronous events and notify application based on subscription

04/19/23 11

Realizing Simplified Application Grid ProgrammingRealizing Simplified Application Grid Programming

• DVC Manager– Single master controller

– Resource selector/negotiator/scheduler

– Trust mediator and security authority

– Synchronizer of global state information

• Ghost Managers– Slave managers, each running at each bound resource

– Job process controller at remote resource

– Communication mediator

– Resource status monitor and reporter

DVC

DVC Manager

Ghost Manager

Control Flow

Data Flow

04/19/23 12

Example: Dynamic Configuration of Example: Dynamic Configuration of Lambda GridLambda Grid

• Example Application: – BIRN (Biomedical Information Research Network)

UCI

NCMIR/UCSD

SDSC

Harvard

UCLA

UNC

DukeBIRN DVC

• DVC Advantages– On-demand creation of a private Grid resource workgroup– Transparent use of novel communication capabilities

– high-speed multi-point communication– SAN-like storage access across geographically distributed resources

04/19/23 13

Example: Dynamic Configuration of Example: Dynamic Configuration of Lambda GridLambda Grid

• Sequence to Create a BIRN DVC1. Create a resource configuration specification and send a request to bind

resources

2. Create resource groups (i.e. for collective data source and sink)

3. Create multipoint-to-point and point-to-multipoint communication sessions

4. Define the properties of communication sessions (e.g. security and communication mechanisms)

UCI

NCMIR/UCSD

SDSC

Harvard

UCLA

UNC

Duke

ucsd

harvardduke

sdsc

uci uci

unc

GTP + enc + auth

TCP + Optical Multicast

Physical-Level View of BIRN DVC Application-Level View of BIRN DVC

Grp1

Grp2

04/19/23 14

Related WorkRelated Work

• Abstractions of Distributed Resources– PVM [Geist94]

• Grid Middleware– Globus System

• Grid Programming Tools– GridRPC [Nakada02], MPICH-G2 [Karonis03], Condor-G [Frey01]

– GrADS [Berman01], GridLab [Allen03],

• Federation of Resources– Virtual Organization (VO) [Foster01]

• Distributed Virtual Computer– Provide an Application-Focused Dynamic Resource Container

– Dynamic resource configuration and sharing policies

04/19/23 15

Summary and Future WorkSummary and Future Work

• Summary– DVC’s provide simple computing environments for Grid applications

– Locally simplified resource workgroup

– DVC’s allow on-demand instantiation and dynamic configuration of Lambda-Grid

– DVC’s enable simple use of novel communication capabilities

• Future Work– Develop the full implementation of the DVC Prototype

– Implement as Web Services (i.e. WS-RF specification)

– Deploy the prototype on the OptIPuter Testbed

– Demonstrate with OptIPuter applications (Bioinformatics and Geophysical)

– Explore other system technologies that can be integrated into the DVC framework

– Real-time System

– High-Performance Distributed File System

04/19/23 16

Thank YouThank You

• Questions and Remarks?

• Contact Information: – Nut Taesombut (nut@cs.ucsd.edu)

• OptIPuter Website:– www.optiputer.net