Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer...
-
Upload
phebe-horton -
Category
Documents
-
view
219 -
download
0
description
Transcript of Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer...
Virtual Environments: System Architectures
Anthony SteedSimon JulierDepartment of Computer ScienceUniversity College London
http://www.cs.ucl.ac.uk/teaching/VE
Overview
• Problem Statement & Requirements
• Data Representations (Contents)
• Execution Models (Dynamics)
• Latency
Reminder - VE is an Immersive,Mediated Communication Medium
User
Interface Devices
Environment
User
Synthetic Environment
Real Environment
Mediated Medium
Ellis’s Content, Geometry, Dynamics Model
• Contents is data• Dynamics is code or
rules to change content• Today we will look at
common data and code representations
Contents:Actors and
Objects
Geometry:Dimensions, Metrics and
Extent
Dynamics:Interaction Rules
“Virtual Environments and Environmental Instruments”, S. Ellis, 1996
Environment
Sources and Sinks of Data
• Input “Data”– Data from trackers (e.g. 3D position plus 3D rotation for head
and hand = 12D data)– Data from input devices (e.g. 2D joystick, buttons)– Possibly audio, haptic, physiological input
• Output “Data”– Displays systems, audio, video
Displays Have Different Requirements
• Video (N copies – for stereo and multiple screens)– Maintain copy of visual state– Render as fast as possible (~60Hz)– Synchronise with other renders
• Audio– Maintain copy of audio state– Render without glitches (requires fast interrupt)
• Haptics– Maintain copy of haptic data
Render as fast as possible (~1000Hz)
“Under the Hood” of the Code
• We will find code modules for many different tasks:– Managing data and assets– Reading devices – Audio rendering– Video rendering– User input– Networking– Complex calculations such as physics (see seminar
tomorrow!)
Overview
• Problem Statement & Requirements
• Data Representations (Contents)
• Execution Models (Dynamics)
• Latency
Graphs
• A graph consists of vertices and edges
• Vertices define the “state” information
• Edges define “relationships”
• Scene-graphs are directed and acyclic
Arbitrary graphDirected graphDirected acyclic graph
Scene-graphs
• In a scene-graph, vertices are often called nodes– Store state information– Can include arbitrary property
information• All graphs have a root node
which defines the base of the tree
• All other nodes divided into two types:– Group nodes– Leaf Nodes
Root node
Group nodes
Leaf nodes
Group Nodes
• Group nodes have multiple nodes as children– Child nodes can be other group nodes or leaf nodes
• Applies common state information to multiple objects– State information propagates down the graph
• Examples include:– Transformations– Switch nodes– Effects
• Bump mapping, scribing, specular highlights
– (In recent times) Shader programs
Examples (OpenSceneGraph)
Anisotropic Lighting
ScribingCartoon
Bumpmapping
Leaf Nodes
• Leaf nodes cannot have children• State information relates to the appearance of specific
objects• Examples include:
– Geometry– Image based rendering
• Billboards• Impostors
Examples (OpenSceneGraph)
Impostors Billboards
Overview
• Problem Statement & Requirements
• Data Representations (Contents)
• Execution Models (Dynamics)
• Latency
Dynamics
• These are the rules of interaction between the contents
• These can be:– Differential equations of Newtonian dynamics to describe
kinematic and dynamic relationships– Grammatical rules for pattern-matched triggered actions
• Many different ways of doing this from imposing numerical approximations to Newtonian physics, through to plain old C++ / Java / XVR coding
Dynamics Representation Model
• Separate from the contents, how do we want to represent our dynamics?
• Its leads to quite critical computer science questions about the separation between code and data
• From a scene-authors point of view two basic models– Standalone process model– Within-scene-graph model
Implementing Dynamics as Standalone Processes• Dynamics implemented
as separate processes / threads
• Can change state of the graph in arbitrary ways– Change values of nodes– Add / remove nodes
Dynamics
Problems of Standalone
• Treats scene-graph as a black-box, makes initial scene-creation very easy (load nodes in to scene-graph, changes a few fields on nodes)
• Composing scenes together, or instantiating existing objects is very hard
• End up with quite complicated code• The code probably has variables in local scope which
“should” be in the scene-graph• Couples together rendering and animation to the same
frame rate
Implementing Dynamics Within the Scene-Graph
• Fairly “autonomous” dynamics can be achieved by embedding dynamics within the scenegraph
• Animations are group nodes which apply state changes to their children
• Examples include:– Animation paths– Particle systems
Animation Node
Animated nodes
Example Animation and Particle System
Problems with Within-Scene-Graph Model
• Difficult to get all the code in there– e.g Difficult to write code that interfaces to other applications– Some valiant attempts such as VRML97/X3D
• Does provide for easy composition of scenes• Different parts of the scene do need an explicit way of
talking to each other– In VRML/X3D there are routes– In many other systems there is an event system
• Many systems provide a scripting language support– Python and Lua are very common
Execution Model Ties Everything Together
• So far we we’ve talked about a disparate set of systems:– A master environment– Separate representations for different output modes– User interfaces for controlling the environment
• The execution model “glues” all these parts together– Closely related to distributed systems as well
Execution Models Tying Things Together
• Example:– The position on an object is changed– The update needs to be reflected in:
• The master database• The different scenegraphs• Over the network (if connected)
– How can all of this be coordinated?• Two main models:
– Kernel model– Actor model
Simple Kernel Model
• Treats a VR application like a traditional graphical application:
while(true){
read_trackers();set_body_position();//Changes scene graphdo_animation(); //Changes scene graphrender_left_eye();render_right_eye();render_sound();poll_trackers();
}
do_animation
• After the user’s position is set by the read_trackers and set_body_position (see later in course) …
• … do_animation manipulates the scene• As we have noted, is responsible for:
– Animations, physics, interaction• This is where the standalone or all the within-scene-
graph dynamics are updated
Problems with Simple Kernel Model
• Everything happens in serial order• Rendering only happens at a fixed rate, so if part of
the animation slows down, the rendering slows down (very noticeable in many video games!)
• What if there are different output requirements such as haptics (1000Hz)?
• What if the input rates are much higher (e.g. 200Hz)
Modified Kernel Model (1)
• Has a fast loop which calls different functions at different frame rates:
while(true){
fast_function();if (elapsed>30ms) slow_function();
}fast_function(){read_trackers(), do_haptics();}
slow_function(){render_left_eye();render_right_eye(); render_sound();}
Modified Kernel Model (2)
• Uses a simple main loop, for the video rendering, but “background” threads for reading devices
• Main thread is very much like the Simple Kernel Model, but the function read_trackers and poll_trackers are now in a separate thread, and the main thread just copies information
• This is a good match to how operating systems actually schedule non-blocking input and output
Pros / Cons of the Kernel Model
• Advantages:– Simple to understand– Application programmer keeps their own data structures within the loop
• Disadvantages:– Implementation needs care because of different update rates– Usually requires some awareness of parallel programming issues– Lots of complexity ends up in the do_animation() method
Actor Model• Virtual environment is
realised by a set of collaborating asynchronous processes (actors)
• Actors send messages to one another
• Processes share a common database
• Database is a central repository of the scene graph
DatabaseAudio
Video1
Video2
TrackingSpeech
Collision
Application
Setting Object State Using in the Actor Model
• Setting the object state is often achieved using the subject-observer design pattern
• The object in the database is the subject• Different renderers / networking systems are the
observers• When the subject’s state is updated, the observers are
automatically notified
Pros / Cons of the Actor Model
• Advantages:– Application program does not care about distribution / what
rendering systems used– Update rates are locally very good– Scales well to multiple cores
• Disadvantages:– Difficult to code efficiently– Difficult to query between code modules in different actors,
needs very clear separation of responsibilities– End to end latency difficult to control (see next section)
Overview
• Problem Statement & Requirements
• Data Representations
• Execution Models
• Latency
End-to-End Latency
• Total time from head movement to scene movement
Mine, Mark (1993). Characterization of End-to-End Delays in Head-Mounted Display Systems, UNC Chapel Hill Computer Science Technical Report TR93-001.
End-to-end latency
Problems
• Single process• For the main CPU, tracker processing is actually
mostly just “waiting”, could schedule something else IF and only IF (IFF) we know how long it will take– Previously we did poll and read separately
• If there are multiple video streams, we could do them in parallel IFF rendering changes no state in the scene graph (depending on your scene graph this isn’t a good bet!)
Example Timing Problem
Animate Render Animate Render
T0 = tracker poll
T1 = video output
Animate Render Animate Render
T0 = tracker poll
T1 = video output
Ideal case
Worst case
Summary
• Representing the environment is difficult• The representation has to be rich enough to capture the contents,
geometry and dynamics• Each display mode requires its own form of the environment to
optimise the display• Want to make content as rich as possible to support dynamic
models• Otherwise behaviour is expressed only in code.• At run-time there are logically concurrent processes (rendering,
collision, audio etc…)• Execution models need to reflect this concurrency