The MicroGrid: A Scientific Tool for Modeling

GridsAndrew A. Chien

SAIC Chair Professor

Department of Computer Science and Engineering

University of California, San Diego

April 30, 2001

Andrew A. Chien – GrADS Site Visit (4/01)

2

Outline

• Motivation• What is a MicroGrid?• Validating Models• Status • Future Work

Andrew A. Chien – GrADS Site Visit (4/01)

3

Motivation

• Need tools to study complex dynamic Grid behavior» complex non-linear dynamic behavior» Tightly couple communication, computing, and storage resources» Performance, Availability, Failure

• Complementary approaches useful, but insufficient» MacroGrids

– Limitations of scale and actual configuration– Major logistical efforts

» Other Simulations– Network-only (internet/networking)– Application level (simple resource models)

» Enable design of robust, reliable, good performing Grids and Grid applications

Andrew A. Chien – GrADS Site Visit (4/01)

4

Grid Application Developer

• How will my software behave on the projected hardware configuration? (performance)

• How will it behave dynamically? (robustness)• How will it interact with other Grid applications an uses of the system?• How can I make this a robust, stable, reusable application?

“Cactus”

“Zeus-MP”

“Netsolve”

“SF-Express”

“Distributed Viz”

“GTomo”

“Tardis”

Andrew A. Chien – GrADS Site Visit (4/01)

5

Grid System Software Developer

• Libraries – network, performance instrumentation, runtime environment (e.g. Globus)

• Program Preparation System – dynamic compilers, runtime, etc.• Do these things work and how well? • With what applications and what range of applications?

“GrADS”

“PPS”

“Nimrod”

“Globus”

“NWS”

Grid Researchers

Andrew A. Chien – GrADS Site Visit (4/01)

6

Grid System Administrator

• What if I change my resource access policies? • What if I add/take away these resources?• What if I change the “price” charged for resources?• What happened to my Grid when it melted down last week?

Andrew A. Chien – GrADS Site Visit (4/01)

7

MicroGrid Goals

• Runtime environment for GrADS experiments (a la MacroGrid)• Develop technology and tools to support specialized Grid

communities (a la MacroGrid)

• Realistic modelling of a broad range of Grid systems, applications, environments, and dynamic behavior» Execution of real applications (tools and middleware)

» Scale to large experiments

» High fidelity simulation, support variety of speed + fidelity– Network, compute, memory, disk

» Observable, repeatable behavior

Andrew A. Chien – GrADS Site Visit (4/01)

8

Outline

• Motivation• What is a MicroGrid?• Validating Models• Status • Future Work

Andrew A. Chien – GrADS Site Visit (4/01)

9

Grid Application

Virtual Grid

MicroGrid Software

LAN Workgroup Scalable Cluster Heterogeneous Environment

MicroGrid Modeling

• A scientific tool for modeling Computational Grids» Run arbitrary Grid applications on any virtual Grid resources» Allow the study of complex dynamic behavior of large

systems

Andrew A. Chien – GrADS Site Visit (4/01)

10

MicroGrid Today

• Processor speed modeling• Memory size modeling• Virtualized Resource description (GIS/MDS)• Network Virtualization• Online Network Simulation

• => runs the Globus 1.1.3 software• => runs Globus applications on a Linux/Alpha testbed

Andrew A. Chien – GrADS Site Visit (4/01)

11

Using a MicroGrid

• Find some physical resources• Configure a Virtual Grid• Submit a Globus Job to it• Observe Execution (which occurs in virtual time)• DeConfigure the Virtual Grid

Grid Application

Virtual Grid, “MicroGrid”

MicroGrid Software

Andrew A. Chien – GrADS Site Visit (4/01)

12

Outline

• Motivation• What is a MicroGrid?• Validating Models• Status • Future Work

Andrew A. Chien – GrADS Site Visit (4/01)

13

MicroGrid Validation

• Simulate an benchmarks and applications• various Grid systems

• Run simulations on the physical hardware• Compare to published results

Andrew A. Chien – GrADS Site Visit (4/01)

14

Validation on Micro-benchmarks

• Memory Capacity Modeling• Processor Speed Modeling• NSE Network Modeling

• Each resource model is validated

Andrew A. Chien – GrADS Site Visit (4/01)

15

Validation on NPB Benchmarks

• Comparison to published cluster NPB results » Set parameters based on known published relative resource

performance -- processor and network performance» Alpha cluster (Alpha’s + 100Mbit Ethernet) and HPVM cluster

• Overall execution time matches within 4%

Andrew A. Chien – GrADS Site Visit (4/01)

16

NPB over WAN

•vBNS•A fictional Cluster

•Varying WAN bandwidth

Andrew A. Chien – GrADS Site Visit (4/01)

17

NPB over WAN (Cont.)

•No background network traffic

•Performance is insensitive to network bandwidth

•Shows a simulation of hypothetical cluster on WAN

Andrew A. Chien – GrADS Site Visit (4/01)

18

Internal Behavior of NPB

• Autopilot tools for Program Tracing (in MicroGrid environment)

• Traces from MicroGrid and real Grid • Match within 5%

Andrew A. Chien – GrADS Site Visit (4/01)

19

Validation on Large Applications

• Cactus PDE Solver Framework on Alpha cluster• WaveToy program, various Matrix sizes• Execution time matches within 7%

Andrew A. Chien – GrADS Site Visit (4/01)

20

Outline

• Motivation• What is a MicroGrid?• Validating Models• Status • Future Work

Andrew A. Chien – GrADS Site Visit (4/01)

21

MicroGrid Today

• Uses Globus 1.1.3• Supports Globus 1.1.3 applications and tools

• Incorporates models for» Processor speed » Memory capacity » Virtualized Resource Description (GIS/MDS)» Network Virtualization» Online Network Simulation

• Used via standard submission interfaces• Not yet available for external users, improving robustness and

adding modules

Andrew A. Chien – GrADS Site Visit (4/01)

22

What have we learned?

• Demonstrated accurate simulation of Grid environments and applications

• Demonstrated ability to support existing applications and tools (critical for significant experiments)

• Existing network simulation tools are inadequate• Existing network traffic models are inadequate• Deriving network configuration information is

challenging• Extrapolation of results is a major challenge due to

nonlinearity of behavior

Andrew A. Chien – GrADS Site Visit (4/01)

23

What have we learned? (cont)

• There’s a LOT more work to be done to support» large-scale, high speed simulations,» with flexible choice of resource models,» simulating a wide range of environments (config,

background activity, etc.), and» executing on a wide range of physical hardware resources.

Andrew A. Chien – GrADS Site Visit (4/01)

24

Milestones

Year 1:• Develop Initial Version of MicroGrid toolkit • Empirical study of application behavior based on

MicroGrid toolkit

Year 2:• GrADS runtime environment and applications on the

MicroGrid (in progress)

Andrew A. Chien – GrADS Site Visit (4/01)

25

Outline

• Motivation• What is a MicroGrid?• Validating Models• Status • Future Work

Andrew A. Chien – GrADS Site Visit (4/01)

26

Ongoing and Future Activities

• System Development (Better MicroGrid)» Scalable On-line Network simulation – Xin “Paff” Liu» Variable speed simulation (efficiency) – Ranjita Bhagwan» Network Traffic Modeling (background & coupled load) –

Xianan Zhang» Disk Speed Modeling (I/O intensive workloads) – Huaxia Xia

• Other current activities (Validation, Software)» Scalapack modeling – Match GrADS results» Cactus modeling – Match GrADS results» Porting to x86 Linux » Robustify and package for external release

Andrew A. Chien – GrADS Site Visit (4/01)

27

Summary

• Demonstrated that MicroGrid approach can produce accurate results in modeling» Grid applications» Grid infrastructures» Dynamic behavior

• Working software• Significant validation

» Micro-benchmarks; Full benchmarks; Applications

• … Need to get MicroGrid software to the next level of capability …

Andrew A. Chien – GrADS Site Visit (4/01)

28

MicroGrid Team

• Dr. Andrew Chien (PI)• Graduate Students:

» Xin “Paff” Liu, Ranjita Bhagwan, Xianan Zhang, Huaxia Xia

• Former:» Dr. Hyo Jung Song (Postdoc)» Dr. Kenjiro Taura (U Tokyo Professor)» Dennis Jakobsen (MS)

• For more information see » http://hipersoft.rice.edu/grads/project/micro.html» http://www-csag.ucsd.edu/

Andrew A. Chien – GrADS Site Visit (4/01)

29