From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical...
Transcript of From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical...
![Page 1: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/1.jpg)
From Static to Dynamic
Visualization ofReal-Time Imaging Data
Stefan Bruckner
Department of InformaticsUniversity of Bergen
![Page 2: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/2.jpg)
Real-Time Medical Imaging (1)
• Acquisition of live in vivo image data of the human body– Imaging of dynamic processes
Cardiology (heart), gastroenterology (stomach & bowel), nephrology (kidney), hepatology (liver), (bladder), obstetrics (fetus)
– Interventional imagingSurgery (e.g., tumor resections, neurosurgery, bypass surgery), biopsies
2
![Page 3: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/3.jpg)
Real-Time Medical Imaging (2)
• Radiography
– FluoroscopyOnly 2D projections, ionizing radiation
• Computed Tomography (CT)
– CT Fluoroscopylow spatial/temporal resolution,high radiation doses
• Magnetic Resonance Imaging (MRI)
– FLASH (Fast Low Angle Shot) MRIOnly single/few slices, limited availability
3
![Page 4: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/4.jpg)
Basic Ultrasound Imaging
4
![Page 5: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/5.jpg)
Ultrasound Characteristics
• Non-invasive
• Cheap
• High resolution
– Spatially
– Temporally
• Noise
– Random
– Speckle
5
![Page 6: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/6.jpg)
Common Ultrasound Modes
• 2D Ultrasound– B-Mode
• 3D Ultrasound– Static 3D imaging
• 4D Ultrasound– Dynamic 3D imaging
• Doppler Ultrasound– Color Doppler: directional– Power Doppler: non-directional
• Contrast Ultrasound– Microbubbles-based contrast agents
• Elastography– Mechanical tissue properties
6
![Page 7: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/7.jpg)
Challenges
• Real-time imaging means that no part of the visualization pipeline can be considered pre-processing
– Limited computational budget
– Degree of interaction limited
– Constant changes
7
![Page 8: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/8.jpg)
Outline
• Visualization of 3D/4D ultrasound data
• Recent advances in
– Filtering
– Classification
– Illumination
– Fusion and Guidance
8
![Page 9: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/9.jpg)
FILTERING
From Static to Dynamic
Visualization of Real-Time Imaging Data
![Page 10: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/10.jpg)
Filtering
• Noisy character of ultrasound imaging makes filtering particularly important for 3D visualization
10
![Page 11: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/11.jpg)
Lowest Variance Filtering
• Remove speckle and random noise
• Structure-preserving filtering
– Determine local structure orientation
– Filter along direction of lowest variance
Solteszova el al. 2012: Lowest-Variance Streamlines for Filtering of 3D Ultrasound 11
![Page 12: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/12.jpg)
Local Structure Orientation
Solteszova el al. 2012: Lowest-Variance Streamlines for Filtering of 3D Ultrasound
• Sample local voxel neighborhood on on a sphere
12
![Page 13: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/13.jpg)
Directional Filtering
• Streamline integrationalong direction oflowest variance
FORWARD
BACKWARD
Solteszova el al. 2012: Lowest-Variance Streamlines for Filtering of 3D Ultrasound 13
![Page 14: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/14.jpg)
Results
Solteszova el al. 2012: Lowest-Variance Streamlines for Filtering of 3D Ultrasound 14
![Page 15: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/15.jpg)
4D Filtering (1)
• Acceptable complexity of filtering method is limited by the target frame rate
– Idea: only filter voxels that contribute to the final rendered image
– Problem: filtering changes data values and hence can affect visibility globally
– Solution: conservatively estimate a voxel’s visibility after filtering
15Solteszova el al. 2014: Visibility-Driven Processing of Streaming Volume Data
![Page 16: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/16.jpg)
4D Filtering (2)
• Only a fraction of voxels actually influence the final image due to transparency and occlusion
16Solteszova el al. 2014: Visibility-Driven Processing of Streaming Volume Data
![Page 17: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/17.jpg)
Visibility-Driven Filtering
17Solteszova el al. 2014: Visibility-Driven Processing of Streaming Volume Data
![Page 18: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/18.jpg)
Prediction of Filter Behavior
• Opacity of a filtered value of minimum and maximum of a neighborhood
• Possible for all convolution-based filters with normalized non-negative weights
• Lookup tables for conservative visibility mask calculation
Solteszova el al. 2014: Visibility-Driven Processing of Streaming Volume Data 18
![Page 19: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/19.jpg)
Results (1)
Solteszova el al. 2014: Visibility-Driven Processing of Streaming Volume Data
unfilteredregularfiltering
5 fps
visibilityoptimized
10 fps
=
19
![Page 20: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/20.jpg)
Results (2)
20Solteszova el al. 2014: Visibility-Driven Processing of Streaming Volume Data
![Page 21: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/21.jpg)
CLASSIFICATION
From Static to Dynamic
Visualization of Real-Time Imaging Data
![Page 22: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/22.jpg)
Classification
• Mapping of data values to optical properties (usually color and opacity)
• Several challenges
– Low dynamic range
– Significant amount of noise and speckle
– Varying intensities for the same tissue
– Fuzzy boundaries
22
![Page 23: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/23.jpg)
Variational Classification
• Simultaneous denoising and opacity assignment
• Variational approach based on isovalue and gradient
Fattal and Lischinski 2001: Variational Classification for Visualization of 3D Ultrasound Data 23
![Page 24: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/24.jpg)
Scale Space Filtering
• Automatic adjustment of the global opacity transfer function based on scale-space filtering
24
Hönigmann et al. 2003: Adaptive Design of a Global Opacity Transfer Function for Direct Volume Rendering of Ultrasound Data
![Page 25: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/25.jpg)
Predicate-based Classification
• Problem: classification of 3D ultrasound data for volume visualization– Standard 1D transfer functions
don’t work well for ultrasound
– Additional attribute dimensions can help, but classification space becomes difficult to navigate
• Approach: define a set of point predicates which can be combined via logical operations
25Schulte zu Berge et al. 2014: Predicate-based Focus-and-Context Visualization for 3D Ultrasound
![Page 26: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/26.jpg)
Predicate Library
• Set of different local and non-local predicates 𝑃 = (𝑓𝑃: 𝑋 → 𝑡𝑟𝑢𝑒, 𝑓𝑎𝑙𝑠𝑒 , 𝜅𝑃, 𝛿𝑃)– 𝜅𝑃 is an importance factor
– 𝛿𝑃 is the color modulation
• Examples of possible predicates– Range-based predicates
– Direction-based predicates
– Signal-to-Noise ratio predicate
– Vesselness predicate
– Confidence predicate
– Label predicate
26Schulte zu Berge et al. 2014: Predicate-based Focus-and-Context Visualization for 3D Ultrasound
![Page 27: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/27.jpg)
Predicate Setup
• Simple widget to assign importances and colors
• Combination of predicates with Boolean operations (and, or, not)
27Schulte zu Berge et al. 2014: Predicate-based Focus-and-Context Visualization for 3D Ultrasound
![Page 28: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/28.jpg)
Visual Mapping
• Importance-driven layered compositing, cf. [Viola et al. 2004, Rautek et al. 2007]
• High-importance layers receive higher visibility (depth relationships can be overridden)
• Predicates only affect hue and opacity, luminance comes from data values
28Schulte zu Berge et al. 2014: Predicate-based Focus-and-Context Visualization for 3D Ultrasound
![Page 29: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/29.jpg)
Predicate Histogram
• Sketch-based interface for predicate setup
• User draws positiveand negative sketch
• Importance of each predicate is modulated accordingly
29Schulte zu Berge et al. 2014: Predicate-based Focus-and-Context Visualization for 3D Ultrasound
![Page 30: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/30.jpg)
Results (1)
• Shoulder dataset: combines visualization of bone and muscle tissue
30Schulte zu Berge et al. 2014: Predicate-based Focus-and-Context Visualization for 3D Ultrasound
![Page 31: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/31.jpg)
Results (2)
• Path of the carotid artery is shown in red
31Schulte zu Berge et al. 2014: Predicate-based Focus-and-Context Visualization for 3D Ultrasound
![Page 32: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/32.jpg)
Results (3)
• Achilles tendon is shown in red
32Schulte zu Berge et al. 2014: Predicate-based Focus-and-Context Visualization for 3D Ultrasound
![Page 33: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/33.jpg)
RENDERING
Recent Developments in Ultrasound Visualization
![Page 34: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/34.jpg)
Volume Rendering (1)
34
image plane
volume
eye
light source
![Page 35: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/35.jpg)
Volume Rendering (2)
35
in-scattering
absorption out-scattering
emission usuallyignored
![Page 36: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/36.jpg)
Local Volume Illumination
• Only a function of gradient direction and light source parameters
– Volumetric absorption between light source and sample point is ignored no shadows
– Multiple scattering is ignored no color bleeding effects
36
conventionalrendering
fetoscopicimage
![Page 37: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/37.jpg)
Light Propagation in Tissue
• Human skin (and tissue in general) is translucent
– Red penetrates deeper than blue and green light
– Light scatters predominantly in forward direction
– Light propagation tends to become isotropic after multiple scattering events
37
![Page 38: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/38.jpg)
Fetoscopic Illumination Model
38
volume data
indirect light
direct light
scattering
shadows
ambient
specular
tone mapping
final image
Varchola 2012: Live Fetoscopic Visualization of 4D Ultrasound Data
![Page 39: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/39.jpg)
Fetoscopic Illumination Model
39
volume data
indirect light
direct light
scattering
shadows
ambient
specular
tone mapping
final image
Varchola 2012: Live Fetoscopic Visualization of 4D Ultrasound Data
![Page 40: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/40.jpg)
Direct Lighting (1)
Light is attenuated along its way through the volume
40
![Page 41: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/41.jpg)
Direct Lighting (2)
41Kniss et al. 2003: A Model for Volume Lighting and Modeling
![Page 42: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/42.jpg)
Light Source Extent (1)
42hard shadows soft shadows
![Page 43: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/43.jpg)
Light Source Extent (2)
43
![Page 44: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/44.jpg)
Soft Shadows
44Patel et al. 2013: Instant Convolution Shadows for Volumetric Detail Mapping
![Page 45: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/45.jpg)
Kernel Size (1)
45
shadow softness - low shadow softness - medium shadow softness - high
![Page 46: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/46.jpg)
Kernel Size (2)
46
shadow softness - low shadow softness - medium shadow softness - high
![Page 47: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/47.jpg)
Fetoscopic Illumination Model
47
volume data
indirect light
direct light
scattering
shadows
ambient
specular
tone mapping
final image
Varchola 2012: Live Fetoscopic Visualization of 4D Ultrasound Data
![Page 48: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/48.jpg)
Indirect Lighting (1)
Light is scattered multiple times before it reaches the eye
48
![Page 49: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/49.jpg)
Indirect Lighting (2)
49Kniss et al. 2003: A Model for Volume Lighting and Modeling
![Page 50: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/50.jpg)
Chromatic Light Attenuation
50
color intensity (RGB)
position along diffusion profile
light orientation
R
G
B
![Page 51: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/51.jpg)
Forward Scattering (1)
51
rendering without scattering rendering with scattering
![Page 52: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/52.jpg)
Forward Scattering (2)
52
![Page 53: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/53.jpg)
Fetoscopic Illumination Model
53
volume data
indirect light
direct light
scattering
shadows
ambient
specular
tone mapping
final image
Varchola 2012: Live Fetoscopic Visualization of 4D Ultrasound Data
![Page 54: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/54.jpg)
Front and Back Lighting
54
Light positioned in front Light positioned behind the scene
![Page 55: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/55.jpg)
Local Ambient Occlusion (1)
• Evaluate the average visibility of each point
– Perform sampling in a small spherical neighborhood
– Modulate ambient illuminationintensity by the result
55
![Page 56: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/56.jpg)
Local Ambient Occlusion (2)
56
with ambient termwithout ambient term
![Page 57: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/57.jpg)
Fetoscopic Illumination Model
57
volume data
indirect light
direct light
scattering
shadows
ambient
specular
tone mapping
final image
Varchola 2012: Live Fetoscopic Visualization of 4D Ultrasound Data
![Page 58: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/58.jpg)
Specular Highlights
58
![Page 59: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/59.jpg)
Fetoscopic Illumination Model
59
volume data
indirect light
direct light
scattering
shadows
ambient
specular
tone mapping
final image
Varchola 2012: Live Fetoscopic Visualization of 4D Ultrasound Data
![Page 60: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/60.jpg)
Implementation
• GPU-based implementation using DirectX
– Available as HDlive in GE’s latest generation of ultrasound machines (Voluson E8 / Expert)
– Interactive performance of 15-20 fps limited by data acquisition
60
![Page 61: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/61.jpg)
Results (1)
61
conventional rendering fetoscopic rendering
![Page 62: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/62.jpg)
Results (2)
62
conventional rendering fetoscopic rendering
![Page 63: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/63.jpg)
Results (3)
63
conventional rendering fetoscopic rendering
![Page 64: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/64.jpg)
Results (4)
64
fetoscopic renderingconventional rendering
![Page 65: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/65.jpg)
Results (5)
65
photograph acquired with fetoscope[A Child is Born, Nilson and Hamberger]
fetoscopic rendering[Picture of the Month, Ultrasound in
Obstetrics & Gynecology 38(5)]
![Page 66: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/66.jpg)
Benefits
• Approximates realistic illumination in real-time
• Robust against noise and artifacts
• Better 3D perception may have diagnostic benefits
• Currently investigating other application scenarios (e.g., cardiac)
66
cleft lip: better visibility of border and separation
down syndrome: inclanation of palpepralfissures
![Page 67: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/67.jpg)
Cardiac Ultrasound
67
![Page 68: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/68.jpg)
FUSION AND GUIDANCE
From Static to Dynamic
Visualization of Real-Time Imaging Data
![Page 69: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/69.jpg)
Fusion and Guidance
• Fusion: combine multiple modalities to improve diagnostic value
– Registered CT/MRI scans, blood flow, etc.
• Guidance: augment images with additional information
– Orientation and navigation aids, etc.
69
![Page 70: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/70.jpg)
B-Mode/Doppler Fusion
• Integrated visualization of B-Mode and Doppler data
• Non-photorealistic silhouette rendering for reduced visual clutter
70Petersch et al. 2007: Blood Flow in Its Context: Combining 3D B-Mode and Color Doppler Ultrasonic Data
![Page 71: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/71.jpg)
Vector Flow Imaging Visualization
• Vector Flow Imaging provides 3D velocity information
– Pathlets-based visualization
– Pathline integration on the GPU
Angelelli et al. 2014: Live ultrasound-based particle visualization of blood flow in the heart 71
![Page 72: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/72.jpg)
Anatomical Context
72Burns et al. 2007: Feature Emphasis and Contextual Cutaways for Multimodal Medical Visualization
• Tracked 2D probe registered with pre-interventional CT scan
• Cutways for unoccluded depiction of the ultrasound slice
![Page 73: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/73.jpg)
Guidance in Liver Examinations
Jennifer N. Gentry
Viola et al. 2008: Illustrated Ultrasound for Multimodal Data Interpretation of Liver Examinations
• Couinaud segmentation: divides the liver into different sections dependent on the blood vessels
• Registration to a liver modelfor real-time Couinaudoverlays during the scan
73
![Page 74: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/74.jpg)
Cardiac Ultrasound Guidance
• Real-time augmentation of the ultrasound slice using an animated heart model
74Ford et al. 2012: HeartPad: Real-Time Visual Guidance for Cardiac Ultrasound
![Page 75: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/75.jpg)
CONCLUSIONS
From Static to Dynamic
Visualization of Real-Time Imaging Data
![Page 76: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/76.jpg)
Conclusions (1)
• Selection of recent approaches for improved visualization of ultrasound data
• Importance of 4D ultrasound as a cheap and effective imaging modality is ever-increasing
• Technological advances (e.g. beamforming) offer continuous improvements in frame rate and image resolution
• Live 4D data is still very challenging and many problems remain unsolved
76
![Page 77: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/77.jpg)
Conclusions (2)
• Technical challenges– Real-time filtering, segmentation, registration,
rendering, …
• Visualization challenges– Integration of anatomy and physiology
(more after the break)
– Visualization of high-speed processes
– Interaction with real-time visualizations
– Quantitative visualization
– Collaborative visualization
77
![Page 78: From Static to Dynamic Visualization of Real-Time Imaging Data 2015. 11. 9. · Real-Time Medical Imaging (1) •Acquisition of live in vivo image data of the human body –Imaging](https://reader033.fdocuments.in/reader033/viewer/2022060520/604ee379b7745f41e20b14f4/html5/thumbnails/78.jpg)
Thank you for your attention!
Acknowledgements
Veronika Solteszova, Åsmund Birkeland, Paolo Angelelli, Ivan Viola, Alexey Karimov, Andrej Varchola, M. Eduard Gröller,
Erik Steen, Gerald Schröcker, Daniel Buckton