RMesa (Remote Mesa)

Kutty S Banerjee

Emmanuel Agu

Remote Mesa

• OpenGL Implementation on Mobile Clients

• Split the OpenGL pipeline b/w Mobile Host and Server “adaptively”

• Granularity is each stage of GL pipeline

• Provide support for remote fetching of meshes

Mesa3D

• RMesa built on top of Mesa3D

• Software implementation of OpenGL.

• Started by Brian Paul,lots of contributors

• Technically, Mesa is OpenGL like API

• Latest stable version is 6.0 Major Number of Mesa =(Minor Number of OpenGL)+1

• E.g., Mesa 6.0 = OpenGL 1.5

Pipeline Splitting

OpenGL Pipeline

OpenGL Pipeline Overview DiagramOpenGL Reference Manual (Addison-Wesley Publishing Company)

Detailed OpenGL Pipeline

Split Pipeline?

• Client weak

• Borrows resources from powerful graphics server

• Saves Battery Life

• Renders more pixels on client

• Adaptively distributes stages on client-server

Pipeline Splitting

Stage 1 Stage 2 Stage 3 Stage 4 Stage 5

Stage 1 Stage 2 Stage 3 Stage 4 Stage 5

Client Pipeline

Server Pipeline

Client Blocks

Shipping vertices to server

Fetching processed vertices from server

Pipeline Splitting in RMesa

• OpenGL calls classified:• State Changers (glNormal,glColor)• Vertex feeders (glVertex,glRect)• Buffers (glFinish, glFlush)• Special (Display Lists, Vertex Arrays, Textures)

• Backend of these calls in Mesa is “Pipeline Factory”

• RMesa modifies Pipeline Factory!!

Running Mesa Pipeline

glBegin(GL_TRIANGLES);

glColor3f(1.0,0.0,0.0);

glVertex3f(0.2,0.5,0.8);

glColor3f(0.0,1.0,0.0);

glVertex3f(0.9,0.3,0.5);

glColor3f(0.0,0.0,1.0);

glVertex(1.4,1.2,1,3);

glEnd();

Within the draw() method

Function draw()

• Between glBegin and glEnd pair vertices collected

• When pipeline run, vertices flushed downpipeline

• NOTE!! New vertices created in pipelineE.g., Clipping!!

• In this case glColor merely changes the GL State.

Vertex Information

Vertex Data

1. Fog

2. Coordinates

3. Color

4. Color (Secondary)

5. Edge Flag

6. Normal

7. Texture Coordinates

•Vertex & Vertex Data required by client & server!!

• Optimization? Don’t need to ship Fog, Color, Secondary Color, Texture Coordinates, Normal for Projection Stage

• But Projection Matrix required to be shipped!!

• All state information, matrices, stack stored in GLContext!! Maintained by all GL Implementations.

FCENT

Networking Inside RMesa

• Winsock 2.0 & BSD Sockets• Win32 and Linux (n/w abstracted)

• RPC• Win32 RPC (OSF – Linux)• Marshalling ? Slow?• Creating interface for entire GL library -

troublesome

• Sockets vs RPC?• Do self marshalling• Tailor made for RMesa

Performance Metrics

• Speed? RMesa vs Mesa– On Desktops– On Mobile Clients

• Battery Life

• Image resolution

`

Summary!!RMesa:

1. Client performs “Primitive Assembly” stage.

2. Ships data over to server for projection stage and blocks

3. Server receives

• Projection Matrix

• Vertex Coordinates

4. Server performs the projection operation .

5. Ships modified coordinate values back to client!!

Primitive Assembly

User Defined Clipping

Projection Stage

This is where in the OpenGL pipeline weare talking of!!

RMesa Adaptive

RMesa -Adaptive

• Maps stages on clients capabilities

• A wrist watch maps all stages to server

• Receives 2D image

• RMesa on a laptop with GeForce FX Go5700 can itself be a server!!(Me not in sales at nVidia)

Constraints – Mobile ClientMobile Constraints:

1. Battery Life

2. CPU power

3. Memory

4. GPU limitation

5. Network congestion

1. Will rendering drain my battery?

2. CPU fast? Can network to server be faster?

3. Memory enough? Need to store on server?

4. Graphics card – enough geometry power?

5. Is network fast?

Intelligence AgencyFeatures:

1. Static Reads:

1. CPU power

2. Memory

3. Graphics Hardware

4. Architecture

2. Dynamic Reads:

1. Battery Life

2. Network congestion

3. CPU utilization

•Intelligraph reads client’s capabilities

•Static Reads: One time read by the agency of the client, done at startup!!

•Dynamic Reads: Periodic surveillance of the client’s configurations and changes

•Maximum Polygons rendered without server support

•Empirical decision - Intelligraph

Intelligraph and RMesa

Intelligraph and RMesa

Intelligraph

RMesaMesh

Simplification

StageMapper

RMesa – Interaction with Intelligraph

Max vertices client can render alone

How to increase this number by stage mapping?

Feeds Vertices to RMesa Pipeline

Fetching Remote Files

Rendering Remote FilesRemote File Render Process:

1. Client queries server for a list of files containing 3D data.

2. Selects a file and informs the server to fetch it.

3. It can also order server to render the polygons in the file instead of fetching the file.

• Application stage provides vertices to geometry stage.

• Allows the application stage to be remote (Retained mode graphics)!!

• Client can fetch file from server and then with the apply stage mapping to the polygons so obtained.

• Extreme case – Client fetches 2D image from server file!!

• RMesa provides extensions to OpenGL calls to support Remote Rendering of Files!!

Rendering Remote Files

Application

OpenGL

RMesa

Extension

Client

OpenGL

RMesa

Extension

Server

File Server

RMesa for Programmers

Configuration FilesclientRMesa Config file:

• StageMapping

• ModelView=client

• Projection=server

• Lighting=client

• Texture=server

• Fragment =client

• Rasterization=client

. .

. .

• Server

• Server Address

• Server Port

• Intelligraph Address

• Intelligraph Port

• The clientRMesa.config file helps in locating the RMesa server and Intelligraph.

• Stage Mapping – details client server stages Note: Temporary solution!!

• The configuration file needs to be in the same directory as that of the executable.

• serverRMesa.config provides port number of server

Compile Link with RMesa

• Compile as usual

• Link with OpenGL (RMesa Implementation)

• clientRMesa.config in current directory

• InitRMesa() – to use extended features

• Without InitRMesa – regular single machine OpenGL appln

RMesa and java3D

• java3D for OpenGL

• Uses RMesa OpenGL transparently!!

• Java applications abstracted completely!!

Project StatusStatus Bar:

• Milestones accomplished

• Geometric pipeline split

• Mapping of stages b/w

client-server

• Socket & RPC implementation of n/w

• Work Under Progress

• Texture & Lighting

• Rasterization Stage splitting

UbiCom ProjectAugmented Reality:(AR)

• Model synthetic objects in real world scene

• Viewer sees through special glasses

• Optical AR and Video AR

• Requirement for low latency graphics scene modeling

• Constraints on processing power, memory, graphics hardware on client mobile device (glasses)

• Dynamic LOD

• Adaptive Resource Contract

•Optical AR- Virtual objects are merged into scene via half transparent mirror

• Video AR- Real world scene is tracked and modeled in 3D and virtual objects immersed into it. More real but slow!!

• Head moves very fast therefore scene modeling speed requirements are extremely high.

• Done using client – server model

• Reads in the client power and chooses a particular Simplification model. Done dynamically!!

WireGL Project

• WireGL product of Stanford Graphics• Client – server model• Client sends GL commands• Server does geometry & rasterization on a single

node!!• O/p of Rasterization send to Image Composition

and lastly display• Nodes form cluster• Soft First

WireGL & RMesa : State Tracking

• Each server connects to many clients

• How much data should client send server each time?

• How does server keep pace with client’s GL Context?

• State Tracking!!

• Involves Tree Hierarchy and context switching. Slows server

WireGL & RMesa State Tracking

• WireGL uses “Tree Hierarchy” on client

• Server has “Context Switching”

• Slows down and increases latency

• RMesa state at the client.

• “LookAhead State” shipped to server

• No Context Switching on server

AnyGL Project

• Derivative of WireGL

• Separates Geometry and Rasterization

• State Tracking present but involves more sophisticated “Timestamp Protocol”

• Uses hardware for Image reassembly called Lightning-2

• Also provision for software reassembly

RMesa (Remote Mesa)Version 1.0

Kutty S Banerjee

Emmanuel Agu