Post on 11-Jan-2016
Tools for Edutainment
Sensory Interfaces, Augmented Reality and Edutainment
BackgroundWu Pin Fong
Virtual Reality Consulting company, Xtensory Inc, California, America
Total 25 years industrial experienceDone virtual reality a long time – 14 years
Emphasis mixed reality support for industry
Virtual Reality Background
Started
Worked at Digital Equipment (now part of Hewlett Packard) 1990
Start the virtual reality program at Digital
First major US company to support virtual reality
Undocumented VR History Working for a major american
computer company
Able to work with all the key VR developers
VR small community then
Relationships
Jaron Lanier, VPL ResearchScott Fisher, NASADr Tom Furness, HITL Labs, University
of WashingtonCarolina Cruz , CAVE inventor, Iowa
State UniversityRudy Darken, Naval Post Graduate
School
Others William Bricken
Autodesk HITL Labs
Foundation for WorldToolKit
Device manufacturers Ascension, Polhemius, Intersense, Immersion, etc
Jaron Lanier VPL Research Taught me Virtual Reality
Worked on VR project For SIGGRAPH 1990
VPL Research in 1990 Very exciting
New technology
Mark Bolas (later to start FakeSpace)
Tom Furness Director, HITL Labs Came from Aerospace
Medical Research Laboratory (USAF), Wright-Patterson AFB, OH.
Helped get funding from Digital Equipemt for HITL Labs
Electronic Visualization Lab - CAVE The CAVE is a
projection based virtual reality system developed at the Electronic Visualization Lab created by Carolina Cruz-Neira
Naval Post Graduate School Rudy Darken, Director MOVES Institute,
Monterey Post Graduate School
Edutainment game - America's Army
Some ProjectsNASAAmerican army, navyWright patterson air force base, OHFAABank of AmericaMayo ClinicToshibaHITL labs, University of WashingtonHasbroLockheed SchlumbergerMonterey naval schoolHeitmanCompaq
OverView Need make it easy to develop
mixed reality games
Provide Standard components
Why components Many mixed reality games involve people
who are not virtual reality experts
Most edutainment projects assume completely new software Control of actual real data Expert programmers
Real world need interface with legacy software
Edutainment games
Out of research into actual industrial use
Today Discuss one standard component -
device support Example of its use Special device support – Haptics A haptics example Component for building virtual cities
for use in mixed reality games
Device Support Component
XvsLink Standard device component for device support
Paper - A Component-based Sensor Architecture for CAVEs (CAVE Automated Virtual Environments), Proceedings of Immersive Projection Technology Workshop, Iowa State University, 2000.
Industry tested since 1996
Not research Used in actual products
Toshiba University of Pennsylvania –virtual
human Jack Mayo Clinic
Standard Device Interface
Importance of components
Example of components – standardised device interfaces
XvsLink Purpose Instrument existing legacy software
Linux, Windows, Sun Solaris, HP, SGI
Easy to use VR devices
Interfaces to non VR devices
Work Closely with Device Manufacturers So closely actually referenced in
their manuals Example Intersense IS600 manual
Focused - Does not provide user interaction techniques
scene graph (or transform tree)
graphics techniques
Provides handle the device control layer
provide communication between hosts
hook to higher levels of an edutainment system
XvsLink Functional Diagram
Device supportSession capture &
playback
Network supportUsers
Developersadministrators
Virtual vehicle
Legacy software
User interaction analysis
One industrial Difference Need to monitor user interaction
with the game
Capture user data for later playback and analysis
Layered Architecture
Application
device interface transport
XvsLink
User interaction tools
Legacy software
Component Device Interface Provide Easy Access
Common Interface to different devices
Access to VR peripherals from any application
Storage and replay of device data
XvsLink
XvsLink is a C++ class library with both a C and C++ interface that provides a standard device interface for integrating devices into simulator applications.
Make devices invisible to user Writing a driver for a new device
usually a few days
Special commands for specific device drivers
Windows and linux kernel level drivers
Operation Open networked, serial, USB, shared
memory connections
Invisible to user Generic devices by functions
connection automatically restored if the device server is shut down and restarted
Design
generic devices polled or queued local or remote actual or virtual devices identified by logical name
Design
polled or queued- application can specify
polled reads data asynchronously typical example: tracker, valuator
queued circular typical example: keyboard, button
Local and Remote devices remote devices treated as local
devices
Local client and server
queries for device capabilities
Design
local or remote device application unaware of source
can use actual or virtual data Device data from hardware or
calculations
Device Mapping
interfaces specify a set of capabilities
device is mapped onto one or more interfaces
Client connects to the different interfaces
run-time bindings
Design
identified by logical name
Device can map to any name
new names can be created as needed
Extensible
Generic Type devices
Layered devices
Multiple-behavior devices
Design
generic devices locator valuator button
Device added dynamically
dynamic device support
examples Intersense tracker Fakespace wand, pinchGlove Phantom haptic
Support
Linux, windows, SUN, HP, SGI
hooks to other tools, packages
Hooks for legacy systems
Example of XvsLink’s Use
Virtual Collaborative Clinic
NASA – Virtual Collaborative Clinic
1998
Dr. Muriel Ross, head of NASA's biometrics research at Ames
Out of research into deployment usually no news
Virtual Collaborative clinic
Collaboration Partners
ObjectivesMedical support in remote locations
surgical try-outs in mixed reality after which the most adequate procedure can be stored to be used in the actual operation.
project computer images onto the patient in order to guide physicians during surgical interventions.
Special device support – Haptics
Haptic Rendering of Virtual Stimuli
Haptic
Haptic Information Haptic information is a form of sensory
stimulation that is acquired through collisions with objects
Using the Haptic Sensory Channel
Force, vibration, texture object shape, compliance, impact,
contact, sliding, slipping, torque and kinematic constraint.
encoding the feel and movement of real or virtual objects during manipulation or exploration
Haptic Exploratory Procedures
Human Sensing
Force Feedback
Force Feedback
Rutgers
Haptics
Example
Xtensory’s Tactile Controller
Shape Memory Alloy (SMA)
a system of long stroke vertical pin-rods, using a coil-form Shape Memory Alloy (SMA) as an actuator.
This material can be stretched or deformed from its original shape but would spontaneously return to its original shape when heated.
Tactor Sensor
Tactor Array
a 16 by 16 array of pin-rods
high-speed update rates
Tactor Array in Action
Operation
orchestrating the vertical motion of a dense array of pin-rods. The effect is similar to a children's "Pop up book", where a flat surface transforms into a 3D form.
the driving method is scalable so that tactor arrays of any size can be constructed
A Haptics Example
Call Options Game
Done for major american bank to train its traders
Conceptual architecture and framework for mixing real and virtual data
Added haptically rendering virtual stimuli
The Game - Call Options The options portfolio consist of options with
3 different times to expiration. The gamma value of an option is
increasing dramatically just before expiration if the option is about at-the-money.
Local peaks or valleys therefore occur for each strike, near to maturity . (Long positions give positive gamma and short positions give negative gamma).
Using Haptics Peaks and valleys shown visually
rendered haptically Mix real and virtual data
In Operation
Another Haptics Example
Enclosed Hazardous Spaces
Narrow field of View
Constrained Space Need sense of touch to move haptic orienting cues play a critical
role
Inspection Game Mix real object – containment vessel
Virtual hazards
Tactile sensing
Training
Virtual City Support Component
Why Many mixed reality games will use
actual city buildings, streets, locations
Need make it easy to mix real and virtual for actual cities
Military – urban operations
Urban Operations In the last 10 years, however, the
U.S. has committed military forces to far-flung urban operations in Panama, Somalia, Liberia, Bosnia and now Kosovo.
The Marine Corps alone has been involved in urban operations in 237 of its last 250 deployments.
Urban Operations Chracterised by extremely short firing
ranges imposed by a city's long canyons of high-rise buildings and dense thickets of trackless housing and teeming alleys
Need for local infantry autonomy
No command and control capability
Intelligence & Reconnaissance
TelegraphOptics
Motion Imagery, Hyperspectral
Radio, ComputerImageAircraft, Camera
Horse,Drums,Smoke, Mirrors, etc…
WrittenEyes, Ears
FeetMouthEyes, Ears
TransmissionReportCollection
Requirements Quick and easy way to build virtual cities
for training games
Acuisition of motion imagery from camera phones, UAVs
Build virtual cities from analysis of motion imagery
Time Frame - Late 1990’s
Xtensory Contracts Enhanced Multiplatform, Integrated Virtual Urban
Warfare Simulation. Navy SBIR N99-063.
Soldier-Computer Interface for Enhanced Interaction and Communication. Army SBIR A95-035
Immersive Visualization of Complex Situations for Mission Rehearsal. Army SBIR A95-089.
Component Motion Imagery analysis (acquired
from another company)
Polygon processing into low polygon count models, autocad, 3DS file formats
Good way to build virtual cities
Conclusion
Shown how use of component tools help in making industrial strength edutainment games