ML2VR Providing MATLAB Users an Easy Transition to Virtual...
Transcript of ML2VR Providing MATLAB Users an Easy Transition to Virtual...
![Page 1: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/1.jpg)
ML2VRProviding MATLAB Users an Easy Transition to Virtual Reality and Immersive Interactivity
David J. Zielinski Ryan P. McMahan Wenjie Lu Silvia Ferrari
![Page 2: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/2.jpg)
Motivation:
DiVE: Duke Immersive Virtual Environment. 6-Sided CAVE-type Virtual Reality System
Benefits of using Virtual Reality● Represent models to scale● Increased spatial understanding● Advanced interaction techniques● Collaborative exploration (CAVE-
type systems)(Open House Every Thursday 4:30pm)
![Page 3: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/3.jpg)
Motivation:
Engineering Students
DiVE
?
![Page 4: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/4.jpg)
Question: What software do engineers use for simulation
and modeling?
![Page 5: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/5.jpg)
Answer: MATLAB"MATLAB is a numerical computing environment and programming language. Developed by MathWorks, MATLAB allows matrix manipulations, plotting of functions and data, and implementation of algorithms." -Wikipedia
![Page 6: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/6.jpg)
Our Goal:
MATLAB DiVE
![Page 7: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/7.jpg)
Potential Existing Solutions:Name Features Issues
MATLAB Simulink 3D Toolbox (Commercial Software Module)
Create and modify VRML scenegraphs.
Targeted at Desktop display systems.
Avizo CalculusMatlab (Commercial Software Module)
Run MATLAB scripts inside Avizo to process data.
Can't utilize user's existing rendering code.
Chromium(open-source project)
Capture and Distributes OpenGL to Render Nodes.
No input device abstraction layer.
Syzygy (open-source framework)
Provides input device and display abstraction layer.
No way to obtain MATLAB's graphics.
![Page 8: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/8.jpg)
Potential Existing Solutions:Name Features Issues
MATLAB Simulink 3D Toolbox (Commercial Software Module)
Create and modify VRML scenegraphs.
Targeted at Desktop display systems.
Avizo CalculusMatlab (Commercial Software Module)
Run MATLAB scripts inside Avizo to process data.
Can't utilize user's existing rendering code.
Chromium(open-source project)
Capture and Distributes OpenGL to Render Nodes.
No input device abstraction layer.
Syzygy (open-source framework)
Provides input device and display abstraction layer.
No way to obtain MATLAB's graphics.
ML2VRMATLAB Syzygy
Our Solution:
![Page 9: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/9.jpg)
Let's see ML2VR in action!
![Page 10: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/10.jpg)
1 CPU 1 HMDUniversity of Texas at Dallas
1 CPU 4 ProjectorsUniversity of Indiana
1 CPU 1 ProjectorDuke University
Common VR System Topologies
![Page 11: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/11.jpg)
What is OpenGL?
MATLAB OpenGL Screen
fill3([0 0 0],[1 0 0],[1 0 1], glBegin(GL_TRIANGLES) glVertex3f(0,0,0); glVertex3f(1,0,0); glVertex3f(1,0,1);glEnd();
"OpenGL is an API (Application Progamming Interface) for rendering 2D and 3D computer graphics. The API is typically used to interact with a GPU, to achieve hardware-accelerated rendering. OpenGL was developed by Silicon Graphics Inc. in 1992." -Wikipedia
vertex = corner
![Page 12: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/12.jpg)
What does OpenGL intercept mean? 1. We add our replacement opengl32.dll to the MATLAB directory. 2. At runtime, MATLAB now loads our replacement opengl32.dll3. When MATLAB calls an opengl function, our function gets called4. After sending out the data, we pass the data onto the real
opengl32.dll function
![Page 13: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/13.jpg)
What does OpenGL intercept mean? 1. We add our replacement opengl32.dll to the MATLAB directory. 2. At runtime, MATLAB now loads our replacement opengl32.dll3. When MATLAB calls an opengl function, our function gets called4. After sending out the data, we pass the data onto the real
opengl32.dll function
glVertex3f(...)
glVertex3f (...){ // send to GPU}
cplxmap(...)
MATLAB command
normal case
![Page 14: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/14.jpg)
What does OpenGL intercept mean? 1. We add our replacement opengl32.dll to the MATLAB directory. 2. At runtime, MATLAB now loads our replacement opengl32.dll3. When MATLAB calls an opengl function, our function gets called4. After sending out the data, we pass the data onto the real
opengl32.dll function
glVertex3f(...)
glVertex3f (...){ // send to GPU}
glVertex3f (...){
send_szg_glVertex3(...);real_opengl.glVertex3f (...);
}cplxmap(...)
MATLAB command intercept
case
normal case
![Page 15: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/15.jpg)
System Architecture Challenges ● Complicated MATLAB simulations may run at low frame rate.
● Our Syzygy content viewer runs at a higher frame rate. ● Can we utilize this "frame rate mismatch?"
SimulatorSickness
Interaction Latency
![Page 16: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/16.jpg)
Detailed System Architecture
Graphics content is generated by MATLAB simulation
That content is displayed by our Syzygy viewer
![Page 17: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/17.jpg)
Detailed System Timeline
● MATLAB simulation generates content● Syzygy viewer receives a content frame● While waiting for new content, Syzygy viewer allows user to view content from different
viewpoints (head based rendering).● Fast head based rendering = reduced simulation sickness
![Page 18: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/18.jpg)
● Minimal additions to access data from VR devices
MATLAB Scriptvr = vr_interface(ServerIP,ServerPort); while 1 [event,btn,pos]=vr.get_button_event(); if event==1 && btn==0 % modify data here end clf; % clear screen surf(dataX,dataY,dataZ); % render surface drawnow; % flush end
![Page 19: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/19.jpg)
Programming APIvr = vr_interface(ServerIP,ServerPort);
Parameters:vr.set_return_type(value) vr.set_transform_matrix(value)
Events: helps with low frame rate simulations[event, button, pos3]=vr.get_button_event() [event, axis, pos3, val]=vr.get_joystick_event()
Polling:[matrix4]=vr.get_sensor_state(sensorID) [val]=vr.get_button_state(buttonID) [val]=vr.get_joystick_state(axisID) Navigation:vr.do_navigation_translate(vec3) vr.do_navigation_rotate(obj, vec3, amount)
![Page 20: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/20.jpg)
Case Study: Robot Path Planning
Silvia FerrariDuke University
Wenjie LuDuke University
● Artificial potential function used to plan the motion of the robot● Potential function recalculates the best path to the target while
adapting to the dynamic addition and removal of obstacles.● We added in ML2VR functions to make the simulation interactive.
Key:
RobotObstacleGoal
Original Desktop Simulation Now in VR !
G. Zhang, and S. Ferrari, “An adaptive artificial potential function approach for geometric sensing,” in IEEE Decision and Control, 2009, pp. 7903-7910.
![Page 21: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/21.jpg)
New Features Robot Demo
Raycasting
Move the goal
![Page 22: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/22.jpg)
New Sci Vis Examplesconeplot and streamline (complicated 3D visualization)
surf demo with parameter modification with joystick x-axis
![Page 23: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/23.jpg)
Interaction Examples01_click02_joystick03_touch04_drag_pos305_drag_matrix406_virtual_hand_pos307_virtual_hand_matrix408_raycasting09_sphere_spawner10_polling11_navigation
![Page 24: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/24.jpg)
Contributions:
● Open source cross-platform software system.
● MATLAB scripts can easily obtain input device data.
● Decoupled (fast) head based rendering.
● Successful case study of a desktop to VR conversion.
● Suite of Interaction programming examples.
![Page 25: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/25.jpg)
March is ML2VR Month!What Where When
Friday Forum Talk Duke University March 8th
Poster Session @ FIP Symposium
Duke University March 10th-11th
Poster @ IEEE VR 2013 Orlando, Florida March 16th-23rd
Invited Speaker @ "Immersive Visualization Bootcamp"
Indianapolis, Indiana
March 25th-27th
![Page 26: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/26.jpg)
Software Robustness● More OpenGL primitives● Dynamic buffers● Line rendering issues● Desktop simulator refinement
Evaluation● What is the benefit of immersive visualization
and interaction for MATLAB?
Future Work
![Page 27: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/27.jpg)
Future WorkRobotics Applications
Quadcopter Simulation Human-Robot Collaboration Simulation
![Page 28: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/28.jpg)
Future Work Front + Back Ends
ML2VRMATLAB Syzygy
Mathematica
Camille GoudeseuneUniversity of Illinois
FreeVR
VR Juggler
R
Bill ShermanIndiana University
Judy VanceIowa State University
![Page 29: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/29.jpg)
Future Work More Collaborators
Michael "Gus" R. Gustafson
Teaches EGR 53L, a Computational Methods in Engineering w/ MATLAB course, that is required for all Pratt (Duke Engineering) first-year students.
Regis Kopper
New director of the DiVE.
Ross Tredinnick
Living Environments Laboratory(UW Madison CAVE)
![Page 30: ML2VR Providing MATLAB Users an Easy Transition to Virtual ...people.duke.edu/~djzielin/slides/ml2vr_presentation.pdf · ML2VR Providing MATLAB Users an Easy Transition to Virtual](https://reader030.fdocuments.in/reader030/viewer/2022041302/5e12ee86c4aeab6ff1048268/html5/thumbnails/30.jpg)
David J. ZielinskiDuke University
AuthorsRyan P. McMahanUniversity of Texas at Dallas
Acknowledgements:
Rachael B. Brady Eric E. Monson Sarah V. Goetz
Support:National Science Foundation.IGERT Grant No. DGE-1068871.
Silvia FerrariDuke University
Wenjie LuDuke University