Point Based Graphics – State of the Art and Recent...

Point Based Graphics – State of the Art and Recent Advances

Markus Gross

Computer Graphics Laboratory

ETH Zürich [email protected]

http://graphics.ethz.ch

A SIGGRAPH 2009 Course

2

Acknowledgements • Mario Botsch, Gael Guennebaud, Simon Heinzle, Richard

Keiser, Oliver Knoll, Edouard Lamboray, Matthias Müller, Miguel Otaduy, Cengitz Oeztireli, Mark Pauly, Denis Steinemann, Matthias Teschner, Tim Weyrich, Martin Wicke, Stephan Wuermlin, Matthias Zwicker

• Leo Guibas, Stanford University • Leif Kobbelt, RWTH Aachen • Andy Nealen, Marc Alexa, TU Darmstadt • Bart Adams, Phil Dutre, Uni Leuven • Hanspeter Pfister, Jeroen van Baar, MERL

3

Purpose of this Course

Points are useful primitives for graphics and modeling!

4

Overview

1. Motivation 2. Representation 3. Processing, Editing, and Modeling 4. Rendering and Display 5. Physics Based Animation 6. Point Based Video 7. Lessons learned

5

Course Material

1. The slides 2. Our book on Point Based Graphics

Gross, M.; Pfister, H.: Point Based Graphics, Morgan

Kaufmann, 2007

6

1. Motivation

7

Polynomials....

Rigorous mathematical concept Robust evaluation of geometric entities Shape control for smooth shapes

Require proper parameterization Discontinuity modeling Topological flexibility

Reduce p, refine h!

8

Triangles...

Simple geometric primitives Hardware to support them Digital processing Explicit topology The widely accepted queen of graphics primitives

Separation of geometry and attributes Complex LOD management Compression and streaming is highly non-trivial

9

Getting to the point…

Natural representation for many 3D acquisition systems No separation of geometry and appearance/attributes No separation of surfaces and volumes

No connectivity or topology …possibly more…?

10

History of Points in Graphics – Particle systems [Reeves 1983] – Points as a display primitive [Whitted, Levoy 1985] – Oriented particles [Szeliski, Tonnesen 1992] – Particles and implicit surfaces [Witkin, Heckbert 1994] – Rendering Architectures [Grossmann, Dally 1998] – Digital Michelangelo [Levoy et al. 2000] – Surfels [Pfister et al. 2000] – QSplat [Rusinkiewicz, Levoy 2000] – Point Clouds [Linsen, Prautzsch 2001] – Point set surfaces [Alexa et al. 2001] – Radial basis functions [Carr et al. 2001] – Surface splatting [Zwicker et al. 2001] – Randomized z-buffer [Wand et al. 2001] – Sampling [Stamminger, Drettakis 2001] – Pointshop3D [Zwicker, Pauly, Knoll, Gross 2002] – Raytracing [Alexa et al. 2003] – Boolean Operations [Adams et al. 2003] – Modeling [Pauly et al. 2003] – blue-c [Gross et al. 2003] – Meshless Physics [Muller et al. 2004] – Spherical MLS [Guennebaud, Gross 2007].....many more

11

Points – A Motivation • 3D content creation pipeline

Points generalize Pixels !

1997-2009

12

2. Representation

13

Surface Model

• Compute continuous surface from a set of discrete point samples

discrete set of

point samples P = { pi, ci, mi, ... }

continuous surface

interpolating or approximating P

14

Surface Model

• Moving least squares (MLS) approximation – Surface defined as stationary set of projection

operator P implicit surface model

– Weighted least squares optimization • Gaussian kernel function

– local, smooth – mesh-less, adaptive

(Alexa, Levin, Amenta, et al.)

15

Point Set Surfaces (PSS) [Levin 2003], [Alexa et al. 2001,2003]

2D-example: smooth curve from a set of points

using moving least squares (MLS) approximation

16

Simple PSS Definition

x

pi

ni

17

Simple PSS definition

x

18

Simple PSS definition

Issues

• Loss of detail, tight fits, stability

19

Issues

• Loss of detail, tight fits, stability

20

PointGraphics 2007 21

Spherical MLS

• Projection onto algebraic sphere

• Improved Stability • Curvature for free • Very fast on GPU

(Guennebaud, Gross, Siggraph 2007)

22

Algebraic Point Set Surfaces

• Key idea: – plane fit sphere fit

• Benefits: – low sampling density – tight approximation – efficiency

23

Issue

• How to fit a sphere onto a set of points ?

minimize some distance metric!

?

24

Algebraic fit

• Algebraic distance:

– the sphere is described by

• Algebraic fit:

!

plane equation !

add constraint(s)

25

Low Sampling Rates

planar fit spherical fit

26

Stability

APSS SPSS vs

27

Progressive Downsampling

from 150k to 5 pts ...

28

Differential Operators

=> e.g., accessibility shading

Local Kernel Regression • LKR for implicit function to surface:

(Oztireli, Gunnebaud, Gross, Eurographics 2009)

29

Surface Definition • Surface definition becomes

30

Our Surface Definition • LKR Formulation of [Kolluri]

• Robust error function

• Iteratively Reweighted Least Squares (IRLS)

31

Our Surface Definition • Surface definition becomes

32

Results

• Edges and Corners

33

Results • A tough one !

34

Results

• s

35

Results • Stability

36

Results • Noise & Outliers

37

38

3. Processing, Editing, Modeling

39

Local Surface Analysis

• local neighborhood (e.g. k-nearest) • eigenvectors span covariance

ellipsoid

• surface variation

• smallest eigenvector is least-squares normal

• measures deviation from tangent plane curvature (Linsen, Prautzsch, Garland)

40

Local Surface Analysis (Pauly, Gross, Kobbelt,

IEEE Vis 2002)

40 40

41

Resampling - Particles

original model

296,850 points

uniform repulsion

2,000 points

adaptive repulsion

3,000 points

• Resample surface by distributing particles – Relaxation - Adjust repulsion radius – Upsampling possible (Heckbert)

42

Particle Simulation

43

Resampling- Simplification

296,850 points 2,000 points remaining

contraction pairs

• Iteratively contracts point pairs (Hoppe) – Uses quadric error metric (Heckbert) – Similar to QSlim

44

3D Image Editing

• Interactive 3D painting • Cleaning • Carving • Textureing and

antialiasing • Modeling • Spectral processing • ...of point sampled geometry

From 2D Pixels to 3D Points

45

Pointshop 3D

• Interactive system for point-based surface editing • Generalize 2D photo editing concepts and

functionality to 3D point-sampled surfaces • Use 3D surface pixels (surfels) as versatile display and

modeling primitive

Does not require intermediate triangulation

irregular point-sampled model

irregular point-sampled model Pointshop 3D

Input Output Surface editing

(Zwicker, Pauly, Knoll, Gross,

Siggraph 2002)

46

Concept

Resampling Editing Operator

u

Parameterization

v

47

Parametrization • Constrained minimum distortion parameterization (Levy, Siggraph 2001)

C(X) = X(u j ) x j{ }j

2+

2X

u2du =min.

feature points distortion

!

Extension to irregular point clouds Multigrid solver for resulting sparse linear least squares problem

48

Parametrization

• Landmarks set interactively by the user (Levy)

landmarks parametrization

49

Examples • Painting textures

50

Examples

• Engraving surface detail

51

Examples

• Filtering appearance and geometry

52

Examples • Multiscale feature extraction

(Pauly, Keiser, Gross,

Eurographics 2003)

53

3D Haptic Painting

• Mass-spring skeleton – Physically-based

deformation – Force feedback

• Point-sampled surface – Geometric deformation – Paint transfer

(Adams, Wicke, et al. PBG 2004)

54

Paint Transfer

1. Sample Collection

2. Paint Buffer Construction

3. Object Sample Projection

4. Brush Sample Projection

5. Paint Model Evaluation

6. Reprojection

No separation of geometry and texture, no connectivity!

55

Brush Splitting


Paint Transfer


Results

58

Painted Bunnies

Nemo Day ‘n Night Flower

• No separation of geometry and texture • No charting and parametrization

59

Surface Processing Toolbox Overview

60

Manual Hole Filling

1) Scan with holes 2) Application of MLS spray can

3) Cont. use of MLS spray can 4) Point relaxation

(Weyrich et al. 2004)

61

Manual Artifact Removal

1) Initial geometry 2) Eraser 3) MLS Spray Can

4) Point Relaxation 5) MLS Smoother

62

Multi-Scale Modeling

• Scale spaces: levels of smoothness

same degrees of freedom on smoothest level

63

Multi-Scale Modeling

• Discretization: base domain + orthogonal diff.

discretization

in scale

discretization

in space

(Pauly, Kobbelt, Gross, TOG 2006)

64

Filtering

65

Interactive Editing

• editing metaphor – continuous free-form

deformation function – smooth transition

between deformed and un-deformed region

– deformation function composed of simple translation and rotation components

66

Example • Large Deformations, Dynamic Resampling

(Pauly, Keiser, Kobbelt, Gross Siggraph 2003)

10,000 points 271,743 points

67

• Interpolate scalar attributes

Dynamic Sampling

68

Free-form Deformation

69

Boolean Operations

+ - -

• Signed distance function by MLS operator

70

Download: graphics.ethz.ch/Pointshop3D

71

4. Rendering

72

Surfels

– Framebuffer resolutions stay roughly the same

– Size of typical triangles in complex 3D models project to < 1 pixel

– Point or pixel based rendering methods become more and more attractive

– Points store several surface attributes (surfels)

– To render, forward project each point separately

Surfels

• 1. View independent sampling (preprocessing)

• 2. Rendering (runtime)

Geometry Surfel

LDC Tree Reduced LDC Tree

Sampling and Texture Prefiltering

3-to-1 Reduction Optional

Block Culling

Forward Warping

Visibility Splatting

Texture Filtering

Deferred Shading

Image Reconstruction and Antialiasing

(Pfister, Zwicker, Baar,

Gross, Siggraph 2000)

73

LDC Tree

Level 0

Level 1

Level 2

•Store only pointers to surfels at level 0. •Do not store empty blocks.

74

75

…Problems… • After projection the image may contain holes

• Texture and edge aliasing

Image Buffer

Visibility Splatting

Object space Z-Buffer Image Buffer

76

Image Reconstruction

Image Buffer Output Image

•2D filtering operations in screen space

77

78

Forward Warping

204 K points 347 K points

Sampling Artifacts screen space

pixel sampling

80

A Closer Look onto Sampling

128 x 192

minification

magnification

aliasing

holes

• Unified approach to reconstruction and aliasing • EWA, Heckbert 86

Splatting

splatting

screen space

81

Warping Reconstruction Kernels

object space screen space

reconstruction kernel

warped reconstruction kernel

forward projection

82

83

Signal Processing View Object Space

Sample

Screen Space

Warp

Screen Space

Filter Screen Space

Gaussian Kernels

Gaussian

reconstruction kernel

Gaussian

low-pass filter

screen space screen space

84

Gaussian Kernels – Closed under affine mappings and convolution

Gaussian resampling filter

“screen space EWA”

• Analytic expression of the resampling filter can be

computed efficiently

85

86

Irregular Textures

pixel sampling

optimized screen space EWA

sampling pattern

(Zwicker, Pfister, Baar,

Gross Siggraph 2001)


Examples • Combine reconstruction and bandlimitation

88

Improvements: Phong Shading + Shadow Maps

(Botsch at al. PBG 2005)

89

Ray-Tracing • Compute ray intersection with MLS surface

(Adams, Keiser, Dutre, Pauly,

Gross, Guibas, EG 05)


Examples

(Alexa et al. 2004, Adams et al. 2005)

91

Point Rendering Hardware • ASIC and FPGA design • 30 Mio. EWA splats/s • Lean architecture • FP units for geometry • 0.25 mu, 500KG

92

Integration Into Poly Pipelines

• Re-use of existing GPU units

• Novel Units – Splat rasterizer – Ternary depth test – Accumulation – Surface reconstruction

buffer – Normalize

92

(Weyrich et al., Siggraph 2007)

93

Video

93

Point Processing Units

• Spatial search: kNN and eNN – Common in most point-operations – Kd-tree [Bentley 75] and priority queue

94

Heinzle, Gunnebaud, Botsch, Gross Graphics Hardware 2008

Coherent Neighbor Cache (eNN)

• Find neighbors in slightly bigger radius.

95

Reuse if .

Coherent Neighbor Cache (kNN)

• Find (n+1) neighbors. • Reuse result for spatially close query.

96

Re-use if .

The Architecture

97

Coherent Neighbor Cache

98

• Eight cached neighborhoods

• Problem: parallel queries in kNN module

Interleave spatially similar queries

Pro

toty

pe

Kd-Tree Traversal

99

Processing Module

100

• Multithreaded quad-port bank of 16 registers

• 128 threads

• Programmability using FPGA-technology.

Results kNN and MLS

101

CUDA: x4

CPU: x1.5

FPGA: x1

CUDA: x2.4

CPU: x1.4

FPGA: x1

CUDA w/o sort: x4.0

CUDA: x1.6

CPU: x1.1 FPGA: x1

CUDA w/o sort: x3.1

MLS CPU: x0.4

MLS CUDA x3.8

Results Approximation Error (MLS projection)

102

Error

Cache hits

103

5. Physically-based Animation

104

Concept • Point-based approach for

physically-based animation

– Point based volume (physics) physical elements

– Point based surface (appearance) surface elements

(Müller, Keiser, Nealen, Pauly, Gross, Alexa,

SCA 2004)

surfels {si} simulation

nodes {pj}

105

Body forces + new external forces = next integration step

We start with pure elasticity, and add plasticity/flow thereafter

Simulation Loop • Start with undeformed object and apply

external forces (per physical element)

Add external forces Time integration Gradient of displacement field Strain (Greens strain) Stress (Hookean material) Body forces (from elastic energy)

106

Topological Separation

Offline simulation at ca. 5 sec./frame

107

Topological Separation

Offline simulation at ca. 5 sec./frame

• Collision detections of penetrating point sets

108

Viscous Fluids

(Keiser, Gasser, Gross, PBG 2005)

• Combine Navier-Stokes and solid mechanics

109

Melting

110

Shell Physics- Lena

(Wicke, Steinmann,

Gross, EG 2005)

• Point shell structures (fibres) connect surface

111

Fracture and Cracks

• Crack initiation – where stress above threshold – crack created by inserting 3 crack nodes

• each carrying 2 opposing surfels • connection is crack front

external

force

external

force

one fracture

surface

crack front

Pauly et al. Siggraph 2005

112

Example

113

Resampling

114

Fracture

(Pauly, Keiser, Adams, Dutre, Gross, Guibas

Siggraph 2005)

• Dynamic sampling of facture surface

115

Mesh-Points

(Steinemann, Otaduy, Gross,

SSCA 2006)

116

6. Point Based Video

(Gross et al. Siggraph 2003)

117

System Overview

Gross et al. Siggraph 2003


119

3D Mirror

Displays 2007 120

121

3D Video Pipeline

122

3D Video Representation

• 3D Video fragement • Generalizes pixels to 3D points

123

Differential 3D Video Stream

green: new Insert red: expired Delete blue: color changed UpdateCol white: color unchanged UpdatePos possibly black: background

124

3D Video Stream

Waschbüsch et al. PG 2005

125

3D Video Recorder

126

MoreExamples

Lamboray et al. ICIP 2004 Würmlin et al. VMV 2005

127

Active 3D Video Acquisition

Stereo cameras

Texture camera

Structured light projector

128

Final 3D Video

129

Results

130

So…Points…

• Points are a useful modeling/graphics primitive

• Sample based approach to graphics and modeling • „Signal Processing“ view • Resample without restructuring

• Local operations on large datasets • Do not store topology • Complement on other graphics representations • Graphics pipelines for points?

• Consumer electronics?

131

Research Themes

• Illustration and stylistic visualization • Implicits and Isosurfaces - MLS • Procedural point geometry • Add semantics to point samples

(„sameness“) • Compression and out of core

representations • Dynamic point clouds • More advanced filters for point

sampled data • Explore combinations of primitives • Hardware for point graphics

132

Resources

• http://graphics.ethz.ch/points/

• http://graphics.ethz.ch/pointshop3d/ • http://www.graphics.ethz.ch/publications/

tutorials.php

• http://graphics.ethz.ch/events/pbg/08/

133

Researchers/Groups (Random&Incomplete)

• Marc Alexa – TU Berlin • Hanspeter Pfister – Harvard • Marc Pauly – ETH Zurich • Matthias Zwicker – University of Berne • Tim Weyrich – UC London • Mario Botsch – University of Bielefeld • Leo Guibas – Stanford University • Leif Kobbelt – RWTH Aachen • Marc Stamminger – University of Erlangen • Gael Guennebaud – INRIA Bordeaux

134

Researchers/Groups (Random&Incomplete)

• Nina Amenta – UC Davis

• Amitabh Varshney – University of Maryland • Michael Wand – MPI – Saarbrücken • Bart Adams – Stanford website

• Loic Barthe, Mathias Paulin – Toulouse • Wojciech Matusik – Adobe Research • Renato Pajarola – University of Zurich • Marc Levoy – Stanford University

135

Thank You!

Point Based Graphics – State of the Art and Recent...

Documents

Transcript of Point Based Graphics – State of the Art and Recent...