3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture:...
-
Upload
claud-hudson -
Category
Documents
-
view
226 -
download
0
Transcript of 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture:...
![Page 1: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/1.jpg)
3DDI Visualization MURI
UC Berkeley and MIT
![Page 2: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/2.jpg)
UC-MIT
3DDI: Overview
Project pipeline:
3D capture:Modeling,simulation
Rendering 3D Display
Applications:Tele-surgery
TrainingCollaboration
![Page 3: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/3.jpg)
UC-MIT
3DDI: Goals• Direct Interaction: no
gloves or glasses.
• Animated content: interaction in real time.
• Content is real-world: 3D models from live capture and modeling.
Laser scanner
3D display
Virtual object
![Page 4: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/4.jpg)
UC-MIT
Task: 3D capture using range scanner• To build a solid-state, high-accuracy electronic
range-finding scanner.
• The system should serve as a replacement for mechanical scanners and motion-capture devices and be usable indoors and outdoors.
• Desired performance: Outdoors, sub-meter accuracy at 100s of meters,
scans in less than a second. Indoors, millimeter accuracy at several meters, scans
at 20-60 frames/sec.
![Page 5: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/5.jpg)
UC-MIT
Subtasks
• Fabrication, testing and improvement of high-power, flip-bonded VCSEL arrays.
• Integration of scanner components. Design of custom elements (modulator, amplifier and power supplies).
• Purchase and integration of coupling optics.• Illumination demo with VCSEL source,
photomultiplier and CCD.• Wrote code for image sequence processing
and calibration.• Static scan example, integration with dynamic
authoring tool (Steve Chenney).
![Page 6: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/6.jpg)
UC-MIT
3D Imaging System, U.C.B.
Fuji Lens
MCPImaging Optics
CCDVCSEL Array
Power Supply HF Signal
Portable Platform!
IR Light
![Page 7: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/7.jpg)
UC-MIT
• Bottle image: depth range ~ 1.2m. Accuracy ~ 0.3cm
First Scanned image
![Page 8: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/8.jpg)
UC-MIT
Task: Model Capture Using Pose Cameras
• Urban geometry
• Textures/BRDFsfor reillumination
… How can we import 3D scene data quickly and automatically?… Starting point for visualization, design, simulation, teaching.
… Develop effectivesensors, automatedand semi-automatedsoftware tools for rapid environment capture
• Synergistic efforts at UCB: 3D scanner Illumination capture
![Page 9: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/9.jpg)
UC-MIT
Goals of integrated effort• Acquire geo-referenced digital imagery
of MIT campus from ground, air
• Extract building exteriors from imagery,using fully automatic techniques
• Model building interiors semi-automaticallyfrom existing 2D building floorplans
• Attach dense interior phototexturesto geometry, semi-automatically
• Integrate photometrics, interaction,and dynamic simulation from UCB
![Page 10: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/10.jpg)
UC-MIT
Acquisition of geo-referenced imagery• Argus platform performs sensor fusion
of imagery, navigation information
![Page 11: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/11.jpg)
UC-MIT
Geo-referencing of multiple nodes• Currently semi-automated process requir-
ing less than one person-second per image
![Page 12: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/12.jpg)
UC-MIT
Texture extraction• Estimation based on weighted medians
![Page 13: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/13.jpg)
UC-MIT
Progress year 3• Acquire hemispherical interior imagery
• Merge ground, aerial geo-ref’d imagery
• Extension to temporal modeling Continuous site modeling of changing site Test: Building 20 demolition, construction
![Page 14: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/14.jpg)
UC-MIT
Task: Capturing Geometry and Reflectance from Photographs
• Input from Cameras, Pose Cameras, Laser Scanners
• Output to Conventional and 3D Displays
![Page 15: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/15.jpg)
UC-MIT
Progress year 1
• Extend Facade to Parametrized Curved Objects
• Visibility Processing and Real-time Rendering
• Campanile Movie
• High Dynamic Range Photography
![Page 16: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/16.jpg)
UC-MIT
Research Highlights Year 1
• Façade: extended to circularly symmetric objects.
• Façade: Accelerated using α-blending.
Campanile Movie shown at SIGGRAPH’97
![Page 17: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/17.jpg)
UC-MIT
Progress year 2
• Photometric Properties of Architectural Scenes
• Capturing and Using Complex Natural Illumination
• Video Motion Capture
![Page 18: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/18.jpg)
UC-MIT
Research Highlights Year 2• Calculation of radiance with known
(outdoor) illumination:
• Re-rendering under novel lighting:
![Page 19: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/19.jpg)
UC-MIT
Research Highlights Year 2• Rendering synthetic objects into real scenes using
HDR photography. Real+Synthetic objects:
![Page 20: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/20.jpg)
UC-MIT
Research Highlights Year 2
• Acquisition of motion data from video using kinematic models:
![Page 21: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/21.jpg)
UC-MIT
Progress year 3
• Reflectance Recovery from MIT Pose Camera Data
• Inverse Global Illumination
![Page 22: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/22.jpg)
UC-MIT
A Synthetic Sunrise Sequence
5:00am 5:30am 6:00am 6:30am
7:00am 8:00am 9:00am 10:00am
One Day at the End of March
![Page 23: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/23.jpg)
UC-MIT
Inverse Global Illumination Algorithm Developed
Reflectance Properties
Radiance Maps
Geometry Light Sources
![Page 24: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/24.jpg)
UC-MIT
Real vs. Synthetic for Original Lighting
![Page 25: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/25.jpg)
UC-MIT
Real vs. Synthetic for Novel Lighting
![Page 26: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/26.jpg)
UC-MIT
Progress Year 4
• Input Multiple range scans
of a scene Multiple photographs
of the same scene
• Output Geometric meshes of
each object in the scene
Registered texture maps for objects
![Page 27: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/27.jpg)
UC-MIT
Overview
RangeImages
RadianceImages
PointCloud
PointGroups
Meshes SimplifiedMeshes
CalibratedImages
TextureMaps Objects
Registration Segmentation Reconstruction
PoseEstimation
Texture MapSynthesis
![Page 28: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/28.jpg)
UC-MIT
Segmentation Results
![Page 29: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/29.jpg)
UC-MIT
Camera Pose Results
• Accuracy: consistently within 2 pixels• Correctness: correct pose for 58 out of 62 images
![Page 30: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/30.jpg)
UC-MIT
Texture-Mapping and Object Manipulation
![Page 31: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/31.jpg)
UC-MIT
Image-based Modeling and Rendering• 3rd Generation--Vary spatial
configurations in addition to viewpoint and lighting
Novel Viewpoint Novel Viewpoint & Configuration
![Page 32: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/32.jpg)
UC-MIT
Texture-Mapping and Object Manipulation
![Page 33: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/33.jpg)
UC-MIT
Task: Authoring Huge, Dynamic Visual Simulations
• Efficiency Too much time is spent computing needless dynamic
state, and dynamic authoring is not integrated with geometric design.
• Control Physics doesn’t do what an author wants
• Success is measured through speedups and the control of example scenarios.
![Page 34: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/34.jpg)
UC-MIT
3D Capture
Modeling,Simulation
Rendering
3D Display
• Take models from measured data. Eg: architecture
• Author scenarios and simulate the dynamics. Eg: a traffic accident
• Provide dynamic models for efficient rendering.
• Integration example: Simulating with a scanned bottle.
How it relates to MURI
![Page 35: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/35.jpg)
UC-MIT
Year 1: Culling with consistency
• Exploit viewer uncertainty to achieve efficient dynamics culling
• Significant speedups demonstrated: Around 5x for test environments. Arbitrary depending on the world.
• Tools released for VRML authoring.
• Papers in I3D, VRML98 and CGA.
![Page 36: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/36.jpg)
UC-MIT
Year 2 and 3: Directing Scenarios
• Use physical sources of randomness (eg. rough surfaces, variable initial conditions) to direct physical simulations
• Year 2: Directing a single body
• Year 3: Directing multiple interacting bodies
• Along the way: Fast multi-body simulation techniques
![Page 37: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/37.jpg)
UC-MIT
Integration Example: Details
• Captured data and 3d rendering must be linked by an authoring phase.
• Extract radius information from 3D bottle scan, plus estimate of variance.
• Simulate using MCMC to achieve a goal - balls are deflected by bottles to land in the right place.
• Render on autostereoscopic display.
![Page 38: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/38.jpg)
UC-MIT
Task: Integration of Modeling and Simulation• Incorporate data from multiple sources:
Geodetic capture (MIT); floorplan extrusion, instancing (UCB)
• Geometry compilation for responsiveness: Scaleable, persistent proximity/visibility database (UCB,
MIT)
• Natural, extensible constraint-based interaction Object associations framework (UCB)
• Physically-based kinematics: Fire simulation (UCB; shown in ‘98) Impulse-response simulation (UCB)
![Page 39: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/39.jpg)
UC-MIT
Several generations of system components:• 1990-93: WalkThrough system (UCB)
Rapid visualization of complex models
• 1993-94: Radiosity integration (Princeton) Diffuse illumination throughout model
• 1994-95: Object associations (UCB) Natural object instancing & placement
• 1994-97: FireWalk, Impulse (UCB) Physically-based fire, kinematic simulations
• 1996-99: Façade, Skymaps (UCB) High-fidelity photo-assisted modeling
• 1996-99: City Scanning (MIT) Acquisition of extended urban models
![Page 40: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/40.jpg)
UC-MIT
Dataset Integration: Geo-referencing• Argus data is geodetically registered
![Page 41: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/41.jpg)
UC-MIT
Dataset Integration: Exterior structure• Exteriors in UCB FireWalk framework
![Page 42: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/42.jpg)
UC-MIT
Integration of UCB object associations• Infrastructure supports editing at any scale
![Page 43: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/43.jpg)
UC-MIT
Exterior to interior transition
• Seamless transition to Tech Square interior
![Page 44: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/44.jpg)
UC-MIT
Transition: building approach
• Gravity association keeps us to local ground
![Page 45: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/45.jpg)
UC-MIT
Visibility modifications: exterior, interior
• Cell-portal visibility applies throughout
![Page 46: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/46.jpg)
UC-MIT
Door passages using object assocations
• Opening doors to allow passage
![Page 47: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/47.jpg)
UC-MIT
Integration of UCB floorsketch, firewalk
• Tech Square interiors modeled by procedural floorplan extrusion, furniture instancing
![Page 48: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/48.jpg)
UC-MIT
Integration of UCB Impulse-Response• Automated generation of RBL objects
Requires specification as union of convex parts
• Initial integration: population, visualization
![Page 49: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/49.jpg)
UC-MIT
Extension to Impulse: sleeping objects• Added “sleep state” for objects coming to
rest
![Page 50: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/50.jpg)
UC-MIT
Extension to Impulse: interaction• Added interactive application of forces
![Page 51: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/51.jpg)
UC-MIT
Task: Novel 3D Displays
• Re-design the MIT holographic-video display for heightened utility.
• Design a new autostereoscopic video display for multiple viewers.
![Page 52: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/52.jpg)
UC-MIT
Relationship to the rest of the field:
• The holographic video display is the first of its kind, and is unique in its size (75mm x 125 mm) and its capability for rapid interaction.
• The autostereoscopic display is unique in its ability to provide binocular stereo video to multiple viewers in arbitrary locations, without the use of viewing aids such as spectacles.
![Page 53: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/53.jpg)
UC-MIT
Interactive Holographic Video
![Page 54: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/54.jpg)
UC-MIT
Autostereoscopic Display
Multiple viewers(three, so far)Micropolarizer-based spatial multiplexing
![Page 55: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/55.jpg)
UC-MIT
Viewer Tracking in Progress
recognizer finds left eye(s).
video signal to viewer-tracking LCD
![Page 56: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/56.jpg)
UC-MIT
Task: Telesurgery
To integrate elements of the MURI pipeline for visualization in the performance and training of surgery:
• Capture of anatomical data
• Modeling of deformable objects
• Haptic interaction with models
• 3D display of models
![Page 57: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/57.jpg)
UC-MIT
Progress year 1
Developed virtual environment for surgical training:
• Organ models from Visible Human data
• Simple deformable modeling, using 2D meshes of masses-springs-dampers
• Basic instrument interactions, without force feedback: grasping, cutting, stapling, electrocautery
• Commercial laparoscopic interface without force feedback (Immersion Corp.)
![Page 58: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/58.jpg)
UC-MIT
Progress years 2 & 3
• Added haptic capability to surgical simulation:
Custom 4 degree of freedom laparoscopic interface, based on commerical 3 DOF device (Sensable Tech Phantom)
• Non-linear, graded finite-element modeling for real-time performance and good accuracy & scalability.
• Tested environment in surgical training course at UCSF
![Page 59: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/59.jpg)
UC-MIT
Current Simulation:Gallbladder removal
Removal of soft tissue using electrocautery tool
![Page 60: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/60.jpg)
UC-MIT
3DDI: Overview
Project pipeline:
3D capture:Modeling,simulation
Rendering 3D Display
Applications:Tele-surgery
TrainingCollaboration
![Page 61: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/61.jpg)
UC-MIT
Programmatic Evaluation I
• Research on components and system integration successful One example of complete pipeline--from
scanning to display shown (bottle). Multiple examples of integration of two or
more modules -- walkthru, outdoor reflectance modeling, simulation
![Page 62: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/62.jpg)
UC-MIT
Programmatic Evaluation II
• Research on components has not connected well with original applications. The virtual surgery work does not make much use of the technologies developed in the project.
• Propose shift of primary motivating application to urban model capture, visualization and simulation
![Page 63: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/63.jpg)
UC-MIT
Budget Adjustments
• MIT Phase out research on holographic display Continue autostereoscopic development Increase funding of urban modeling
• UCSF Virtual surgery: phase out completely
• UC Berkeley Increase funding for modeling from laser
scanner data
![Page 64: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/64.jpg)
UC-MIT
3DDI: Overview
Project pipeline:
3D capture:Modeling,simulation
Rendering 3D Display
Application:Exterior and InteriorUrban Environments
![Page 65: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/65.jpg)
UC-MIT
Scenario: Rapid Capture of, andTraining in, Urban Environments
• Acquire high-fidelity geometric and photometric models of real environments
• Provide ability to simulate, visualize and physically interact with this environment
• Enhance photorealism with 3D displays
![Page 66: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/66.jpg)
UC-MIT
An Example Sequence of Interactions
• The images in the following sequence obviously appear synthetic; we want to achieve this functionality while maintaining photorealism.
![Page 67: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/67.jpg)
UC-MIT
Flyby of Model of Real Urban Environment• Can be modified by adding virtual buildings
![Page 68: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/68.jpg)
UC-MIT
Seamless Exterior to Interior Transition
• Incorporate geometric, photometric detailto increase photorealism, immersion
![Page 69: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/69.jpg)
UC-MIT
Compiled Proximity, Visibility Information• Increases interactivity, decreases network
traffic among multiple users of model
![Page 70: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/70.jpg)
UC-MIT
Interact with Physical Objects• Increased ability for natural interaction with
running physical simulation
![Page 71: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/71.jpg)
UC-MIT
Directing Behaviors• Construct problem solving contexts for
training
![Page 72: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/72.jpg)
UC-MIT
Research and Engineering Aspects
• Instrumentation
• Exterior capture
• Interior capture
• Real-time Interaction
• Directable Dynamics
• 3D Display
• System Integration
• Representation!
![Page 73: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/73.jpg)
UC-MIT
Progress over the last 12 months• VCSEL array scanner
• Modeling from range and image data: Berkeley
• Modeling from pose cameras: MIT
• Authoring with dynamics
• Real-time Simulation of Physically Realistic Global Deformations
• System integration in Walkthru framework
![Page 74: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/74.jpg)
UC-MIT
Progress on VCSEL array scanner• Design of row-addressable VCSEL array to
provide a scanning source.
• Fabrication of chip prototypes with bonding to silicon.
• Testing and characterization.
![Page 75: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/75.jpg)
UC-MIT
Progress on Modeling: MIT
• Acquisition sensor improvements
• Faster, more accurate spherical imagery
• Improved sub-pixel edge detection
• Automated rotational alignment
• Improved texture, occlusion estimation
• Off-planar relief estimation (Fua, Leclerc)
• Symbolic window extraction (Wang)
• Framework for indoor/outdoor visibility
![Page 76: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/76.jpg)
UC-MIT
Progress on Authoring with Dynamics
• Combine the visibility structure of a model with a model of object dynamics Objects guarantee where they will not be Cull dynamics safely
• Objective: Frame rate depends on number of objects
in view
• Demonstration: A complex world where frame rate (largely)
depends on number of objects in view
![Page 77: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/77.jpg)
UC-MIT
Progress on Real-time Simulation of Physically Realistic Global Deformations
•Combines the best features of several models FEM accuracy (theory of elasticity) No distortion (due to the nonlinear strain) Diagonalized mass matrix (similar to particle
system) Graded mesh size of O(n^2) (comparable to
BEM)
![Page 78: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/78.jpg)
UC-MIT
System Integration in Walkthru Framework
• Create an object-oriented, extensible databasein which various types of models can be stored.
• Develop rendering paradigms by which this DB can be explored by many users simultaneously.
• Create the hooks for the attachment of simulators which may allow “work-on-demand” control.
• Allow all interactions to happen over the Internetbetween different types of computers and OS.
![Page 79: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/79.jpg)
UC-MIT
Walkthru Framework
• Environment to model complex dynamic worlds with user interaction: Cell-based visibility culling: Pre-loading of scene parts -- based on
expected demand, derived from user motion.
• Generic simulation interface; integrated: CFAST: NIST’s Fire Simulator IMPULSE: Rigid Body Dynamics
![Page 80: 3DDI Visualization MURI UC Berkeley and MIT. UC- MIT 3DDI: Overview Project pipeline: 3D capture: Modeling, simulation Rendering3D Display Applications:](https://reader035.fdocuments.in/reader035/viewer/2022062322/56649e7f5503460f94b82e4e/html5/thumbnails/80.jpg)
UC-MIT
Progress Summary
• Built a shared, object-oriented data base.
• Extensions beyond just geometry (SYLIF).
• Tools for model generation from floorplans.
• Used in joint model developments with MIT.
• Integration of scene data from Malik’s group.