Networking and Management Frameworks for Cluster-Based

GraphicsBenjamin Schaeffer

Integrated Systems Lab

University of Illinois

Syzygy Snapshots

Mathematics: Hyperbolic Geometry

Syzygy

• Library for Virtual Reality on PC clusters

• Supports multiple programming styles– Distributed Scene Graph– Synchronized application instances

• Portable: Linux, Win32, Irix

• High performance and high quality

The Problem

• PC graphics and networking performance is becoming comparable to the graphics supercomputers popular in the 1990’s

• How can we exploit this?

• Unfortunately, PC’s can only host 1 high performance graphics card (AGP bus)

• PC cluster is the answer

The Problem

• PC cluster has much lower communications bandwidth between components (shared memory vs. network)

• This bottleneck means that PC cluster VR software must be highly optimized to get high performance

• Different applications will require different architectures. Difficult for the library writer!

The Problem

• Cluster applications are more difficult to manage than applications running on a single computer.

• Starting the application?

• Stopping the application?

• Configuring the application?

• What if the cluster is heterogeneous?

Related Work

• CAVElib: Cluster support in the latest version. Limited automatic data sharing. Sockets-based network facility for other data sharing.

• NetJuggler and Cluster Juggler: Two different extensions to VR Juggler with cluster support. Multiple copies of the application share an input event stream.

Related Work

• WireGL: Replacement for OpenGL shared library that broadcasts OpenGL primitives over the network to be displayed by cluster nodes.

• Avango: Shared scene graph for shared virtual environments. Can be adapted for cluster-based graphics.

Application Architecture

• Two basic application architectures (there can be more) for cluster-based graphics

• Client/server

• Master/Slave

Client/Server

• The application runs on a central node and sends rendering information to the render nodes. Uses a fixed protocol.

• Generic rendering clients run on the render nodes. These are the same for all applications.

• Examples: WireGL, Avango (?), Syzygy distributed database

Master/Slave

• Multiple copies of the application run, one on each render node.

• A master instance of the application controls what the slave instances do, via broadcasting state information each frame.

• No fixed protocol. Unique for each application.• Examples: CAVElib, Cluster Juggler, Syzygy

Master/Slave application framework

Pros and Cons

• Client/Server can require high bandwidth (WireGL).

• Client/Server can require software rewriting (scene graphs). These higher-level protocols can be network efficient.

• Master/slave in many cases requires low bandwidth.

• Master/slave might not be suitable for complex applications.

Syzygy

• Provides a common networking and management framework for building cluster VR applications.

• Why? Many different ways to do cluster VR, each appropriate for different situations.

• As practical examples, build a distributed scene graph and a master/slave application framework.

Communications Layer

Phleet I/O FrameworkMedia Protocols

Media Objects I/O Drivers

Application Frameworks

Messaging

• Syzygy communication is message-based.• Programmer can construct message types

and languages from those types.• Binary data format with built-in translation

between different machine architectures (big-endian to little-endian).

• Built-in message serialization and de-serialization

Communication Objects

• Syzygy includes server objects that can manage multiple full-duplex network connections. Communication occurs via Syzygy messages.

• Also client objects that communicate with the servers.

• Fault tolerance is built in. Objects can activate in any order, stop, restart, reconnect, disconnect, etc.

Synchronization

• BarrierServer object controls synchronization across the cluster.

• BarrierClient objects connect to it.• Built on the standard Syzygy client/server objects.• Synchronization groups are dynamic. Clients can

enter and leave the group at any time.• Fast. Buffer swap latency is comparable to the

ping time on the cluster network for ethernet.• Important for cluster graphics!

Synchronized Buffer Transfers

• For efficiency in graphics, information should be transferred once per frame. Small messages need to be buffered.

• Syzygy includes support for objects that automatically buffer messages in the background, transferring the buffers on the graphics buffer swap.

• Double-buffering of messages is used to get the most out of slow networks. While it is buffering the new messages, Syzygy is transferring the last buffer.

ser v

ercl

ient

clie

ntSend frame (n)

Fill frame (n+1)

Draw frame (n-1)

Receive frame (n)

Send Connection

Request

Send State

RecvState

Draw frame (n)

Idle Recv frame (n+1)

Idle

Send frame (n+1)

Fill frame (n+2)

Send frame (n+2)

Fill frame (n+3)

Draw frame (n+1)

Recv frame (n+2)

Draw frame (n+1)

Recv frame (n+2)

Phleet

• Distributed operating system (minimal)

• szgserver controls.

• szgd provides remote execution services.

• szgserver is merely a connection broker. Most communication occurs directly between programs.

Phleet

• Parameter database stored in szgserver. Meta-config file that can be managed from the command line of any computer.

• Global locks

• Simple message API routes through szgserver to all managed programs (like Unix signals). Can tell a program to reload its parameters or exit.

Distributed Scene Graph

• Client/server. One node serves geometry to the render nodes.

• Bandwidth is automatically conserved. Only scene changes are sent from one frame to the next.

• Semantic nature of scene graph helps ease programming.• Robust. Render clients can disconnect and reconnect

while the application is running. Dynamically change your display. Similarly, stop your application and start a new one without bringing down the render clients.

Distributed Scene Graph

• Built on a general hierarchical distributed database.

• Nodes in the database are created and later altered via sending Syzygy messages to the database.

• Incremental changes can be made to a node (for instance, altering just some points in a point set). Saves bandwidth.

• An API is provided to aid the programmer in generating these messages.

Distributed Scene Graph

• Node types include:– Transform – Points– Lines– Triangles– Texture coordinates– Textures– Visibility– Text Billboard– Etc.

SyzygyServer

Barrier Server

Graphics Server

Input Client

szgd

szgd

szgd

InputServer

Barrier Client

Render

Master/Slave Framework

• Master/slave = same application runs synchronized on render nodes.

• One node is the master and distributes I/O or other control info.

• Framework makes it easy to write a synchronized application, handle input devices, and manage the whole thing.

• A good way to write a custom distributed graphics protocol!

Master/Slave Framework

• Programmer registers memory chunks.• Event loop:

– Pre-exchange callback (computations), input device polling on master.

– Data exchange (handled by library)– Post-exchange callback. Input device state is

synchronized on all application instances. As is registered memory.

– Draw callback.– Synchronization.

Master Slave Slave

Barrier Server Barrier Client Barrier Client

Input Client

InputServer

Master/SlaveApplication

Detail

Practical Experiences

• Volume visualization using master/slave framework.

• Visualization of MPI-based codes using distributed scene graph.

• Quake 3 level viewer. Master/Slave.• Mathematical visualizations: Sphere eversion and

hyperbolic geometry. Master/slave.• Fully articulated avatars animated by motion

capture data. Distributed scene graph.

CAVE Quake 3

////

CARMEN: shared virtual world

////

Mathematics: Hyperbolic Geometry

.edu/.edu/.edu/.edu/

Supercomputer Performance Analysis

Getting the Software

• Syzygy is licensed under the GNU LGPL

• It comes with everything you need to start showing demos right away!

• Download the latest release at www.isl.uiuc.edu (follow the software link)

• Or ask for a free Syzygy-0.4 CD after this talk!