Global Static Indexing for Real-time Exploration of Very...
Transcript of Global Static Indexing for Real-time Exploration of Very...
![Page 1: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/1.jpg)
CASC VP 1
Global Static Indexing for Real-time Exploration of Very Large Regular Grids
Valerio Pascucci and Randall J. Frankpascucci,[email protected]
Lawrence Livermore National Laboratory
![Page 2: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/2.jpg)
CASC VP 2
Outline• Motivation
• Previous work
• Data layout
• 2n tree indexing
• Performance for slicing large grids
• Conclusions and future work
k
I
i j
(bla bla[vis`01], bla bla bla [sc`00] ……)
![Page 3: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/3.jpg)
CASC VP 3
We must achieve real-time interaction with large datasets on a wide variety of platforms.
The problem
• Extremely large datasets 0.5TB/timestep (8k,8k,8k,(time)).
• Interactive rendering for real-time data exploration.
• Target platforms: desktop,parallel server, cluster.
![Page 4: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/4.jpg)
CASC VP 4
Previous Work• Out-of-core geometric algorithms
[Goodrich, Tsay, Vengroff, Vitter ‘93]
[Vitter ‘00][Matias, Segal, Vitter ‘00]
[Asano, Ranjan, Roos, Welzl ’95][Arge Miltersen ’99]
• Out-of-core visualization[Chiang, Silva ‘97][Sutton, Hansen ‘99]
[Livnat,Shen,Johnson ’96][El-Sana, Chiang’00]
[Bajaj, Pascucci, Thompson, Zhang ‘99]
• Space filling curves(image processing, multidimensional databse, geomtric datastructure …)
[Bandou, Kamata.’99][Balmelli, Kovacevic, Vetterli ’99]
[Parashar, Browne, Edwards, Klimkowski ’97]
[Niedermeier, Reinhardt, sanders ‘97][wise’00]
[Hans Sagan ’94] [Lawder king ’00][Griebel Zumbusch ‘99]
![Page 5: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/5.jpg)
CASC VP 5
We apply three fundamental techniques to the visualization of large simulation data.
Our approach
• Multi-resolution geometric representation:– adaptive view-dependent refinement;– minimal geometric output for selected error tolerance.
• Cache oblivious external memory data layouts:– exploit spatial and resolution coherency;– no need for complicated paging techniques.
• Progressive processing: – continuously improved rendering;– scalability with the resources without budgeting.
![Page 6: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/6.jpg)
CASC VP 6
We focus on the progressive computation of slices (any orientation) of large 3D rectilinear grids.
• Rectilinear grid• Sub-sampling octtree
• 1D order = hierarchical3D Z-order curve
• Coarse to fine slice refinement
![Page 7: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/7.jpg)
CASC VP 7
General Data Layout
Grouping the data by level of resolution
Data coherent Progressive refinement of a hierarchical
geometric data-structure
Grouping the data by geometric proximity
![Page 8: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/8.jpg)
CASC VP 8
General Data Layout
![Page 9: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/9.jpg)
CASC VP 9
General Data Layout
![Page 10: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/10.jpg)
CASC VP 10
General Data Layout
![Page 11: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/11.jpg)
CASC VP 11
General Data Layout
![Page 12: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/12.jpg)
CASC VP 12
General Data Layout
![Page 13: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/13.jpg)
CASC VP 13
General Data Layout
![Page 14: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/14.jpg)
CASC VP 14
General Data Layout
nD to 1D mapping:
I → I*
I → l find the level of resolution lCl (pre)compute the number of elements in the levels coarser than l
I → I’ index of the element within its level of resolution
I* = Cl + I’
C0 C1 C2 C3
![Page 15: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/15.jpg)
CASC VP 15
We exploit the correlation of bin/quad/oct-trees with the Lebesgue space-filling curves.
The Lebesgue curve is also known as Z-order, Morton, …. Curve.Special case of the general definition introduced by Guiseppe Peano in 1890.
![Page 16: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/16.jpg)
CASC VP 16
We turn the recursive definition of the Z-order curve into a hierarchical subsampling scheme.
(a) (b)
(c) (d)
coarse data new level data
![Page 17: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/17.jpg)
CASC VP 17
We obtain a multi-resolution hierarchical representation which is not exactly a 2n-tree.
• Not exactly a quad-tree ……
Level 0
Level 1
![Page 18: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/18.jpg)
CASC VP 18
We obtain a multi-resolution hierarchical representation which is not exactly a 2n-tree.
• Not exactly a quad-tree ……
Level 0
Level 1
Level 2
![Page 19: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/19.jpg)
CASC VP 19
Level 3
We obtain a multi-resolution hierarchical representation which is not exactly a 2n-tree.
• Not exactly a quad-tree ……
Level 0
Level 1
Level 2
![Page 20: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/20.jpg)
CASC VP 20
The 1D index I* can be computed in a simple and efficient way in any dimension.
I*
I
1514131211109876543210
1514131110976532112840Z-order 4
3210Z-order 2
0Z-order 0
210
Cl = 22(l-1) _I_22lI’ =I* = Cl + I’
_I_22(l+1)- - 1
I
i j
![Page 21: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/21.jpg)
CASC VP 21
The 1D index I* can be computed in a simple and efficient way in any dimension.
I*
I
ki j
![Page 22: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/22.jpg)
CASC VP 22
Overall the hierarchical Z-order yields a cache oblivious hierarchical data layout.
Distribution in the grid of each constant size block of data
0 1 2 3 4 5 6 7 8 9
![Page 23: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/23.jpg)
CASC VP 23
Theoretical analysis shows a gain of orders of magnitude independently of the block size.
1
10
100
1000
10000
100000
1000000
10000000
100000000
000000000
1 2 4 8 16 32 64 128
256
512
1024
2048
4096
8192
32*32*32
3-bits shift
1-bit shift
1
10
100
1000
10000
100000
1000000
10000000
100000000
1000000000
1 4 16 64 256
1024
4096
32 3̂-BLOC
3-BITS
1-bit shift
32K blocks 64K blocks
Cache oblivious !!!
![Page 24: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/24.jpg)
CASC VP 24
Real speedup matches theoretical expectations:more than 10x improvement, platform scalable.
• 2048x2048x1920 dataset (we have run up to 8192x8192x7680)
• 20MB memory cache
• Translation and rotation tests (average over 3 primary axis)250Mhz SGI Onyx (1024x1024)
0.001
0.010
0.100
1.000
10.000
100.000
1000.000
1 10 100
Resolution
Seco
nds
ARR-Rot
BIT-Rot
BLK-Rot
ARR-Trans
BIT-Trans
BLK-Trans
500Mhz PIII Laptop (512x512)
0.001
0.010
0.100
1.000
10.000
100.000
1 10 100
Resolution
Seco
nds
ARR-Rot
BIT-Rot
BLK-Rot
ARR-Trans
BIT-Trans
BLK-Trans
![Page 25: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/25.jpg)
CASC VP 25
Conclusions and Future Work
• Results: an implicit scheme for coherent spatial, multi-resolution regular grid data access
• Simple address remapping
• Read/write access
• No additional data overhead
• Supports progressive access
• Near term applications
• Volume rendering
• Time critical iso-contouring
• Future work
• “View dependent” parameterization
• Unstructured/temporal hierarchies
• Improved interpolation
• Distributed implementation
![Page 26: Global Static Indexing for Real-time Exploration of Very ...pascucci.org/pdf-presentations/SC2001-talk.pdf · CASC VP 3 We must achieve real-time interaction with large datasets on](https://reader033.fdocuments.in/reader033/viewer/2022051904/5ff61193d1a4580e27092b48/html5/thumbnails/26.jpg)
CASC VP 26
UCRL-VG-143542
This work was performed under the auspices of the U.S. Department of Energy by University of California Lawrence Livermore National Laboratory under contract No. W-7405-ENG-48.