CS 395: Adv. Computer Graphics Introduction to Image-Space Methods Image-Space Methods Watt & Watt:...
-
date post
19-Dec-2015 -
Category
Documents
-
view
234 -
download
0
Transcript of CS 395: Adv. Computer Graphics Introduction to Image-Space Methods Image-Space Methods Watt & Watt:...
CS 395: Adv. Computer GraphicsCS 395: Adv. Computer Graphics
Introduction toIntroduction to
Image-Space MethodsImage-Space Methods
Watt & Watt: Chapter 8,Watt & Watt: Chapter 8,
"Pixel is Not a Little Square!" –AR Smith"Pixel is Not a Little Square!" –AR Smith
Jack TumblinJack Tumblin
[email protected]@cs.northwestern.edu
Image Space MethodsImage Space Methods
• A ‘Digital Image’ = a data structureA ‘Digital Image’ = a data structurefor displayed imagesfor displayed images
• ?What can it ?What can it dodo, besides display?, besides display?
what can we change?what can we change?– Position; Position; offset, stretch, squeeze, smear, cut,…offset, stretch, squeeze, smear, cut,…
Try It! (Java Applets)Try It! (Java Applets) http://www.mrl.nyu.edu/~hertzman/nudge/ http://www.mrl.nyu.edu/~hertzman/nudge/
– Combine:Combine: +/-*, Compositing, (depth images?) +/-*, Compositing, (depth images?)– Separate:Separate: matte or ‘segmenting’ matte or ‘segmenting’– Elaborate:Elaborate: make ‘more of the same’ picture make ‘more of the same’ picture
‘‘Real’ Images: Lens-Focused LightReal’ Images: Lens-Focused Light
• 3D3D2D Light Intensity Map 2D Light Intensity Map I(I(x,yx,y))• Angle(Angle(,,) ) Position( Position(x,yx,y) ) • ‘‘BlurringBlurring’—sharpness set by focus, lens quality’—sharpness set by focus, lens quality
I(I(x,yx,y))Image Plane Image Plane
IntensityIntensity
Ang
le(
Ang
le(
,, ))
Pos
ition
(P
ositi
on( x
,yx,y ))
Viewed SceneViewed Scene
‘‘Digital’ Images: 2D Grid of NumbersDigital’ Images: 2D Grid of Numbers
• NO intrinsic meaning, but ...NO intrinsic meaning, but ...• Widely assumed to representWidely assumed to represent
– Point Samples of a “smoothed” 2D intensity surfacePoint Samples of a “smoothed” 2D intensity surface– Uniform sampling pattern (but not always)Uniform sampling pattern (but not always)
xx
yy
(!weasel-word!)(!weasel-word!)
Real vs. DigitalReal vs. Digital
• Real Image == 2D Light Intensity map: I(x,y)Real Image == 2D Light Intensity map: I(x,y)
• Digital Image == 2D grid of numbers: I(m,n)Digital Image == 2D grid of numbers: I(m,n)
I(x,y)I(x,y)
x,yx,y
I(m,n)I(m,n)
m,nm,n
(pixels)(pixels)
Digital Images As VectorsDigital Images As Vectors
• ‘‘Stack up’ pixel values: VERY LONG vector Stack up’ pixel values: VERY LONG vector – 1 digital image == 1 point in N-dim. Space1 digital image == 1 point in N-dim. Space– Nearby points == Similar imagesNearby points == Similar images– All possible digital images: All possible digital images:
a grid of N-D pointsa grid of N-D points
– Space of all practical Space of all practical digital images: digital images:
• ~8Meg dimensions ~8Meg dimensions (2Mpix * RGBA) (2Mpix * RGBA)
• discrete, quantizeddiscrete, quantized
I(m,n)I(m,n)
m,nm,n
II0000
II0101
II0202
……II1010
II1111
II1212
……
I = I =
II00 00 I I01 01 I I02 02 II03 03 II04 04 II05 05
Digital Images As VectorsDigital Images As Vectors
• Sensible element-by-element operations:Sensible element-by-element operations:– Add, subtract, scale two images: Add, subtract, scale two images:
0.5 +0.5 + (I (I11 - I - I2 2 ) = out) = out
-- ==II11 II22 outout
Digital Images As VectorsDigital Images As Vectors
• Sensible element-by-element operations:Sensible element-by-element operations: ‘ ‘Alpha blending’ or ‘dissolve’:Alpha blending’ or ‘dissolve’:
--Weighted sum of two images: Weighted sum of two images: 0 < 0 < < 1 < 1
++ == II11 (1-(1-) I) I22 outout
Digital Images As VectorsDigital Images As Vectors
Compositing:Compositing:
:: Use an Use an imageimage
as ‘opacity’:as ‘opacity’:
++ == .* I.* I11 (1-(1-) .* I) .* I22outout
Extended Compositing MethodsExtended Compositing Methods
• Environment MattesEnvironment Mattes
Alpha Matte Environment Matte Photograph
Digital Images As VectorsDigital Images As Vectors
Exploration of the space-of-all-images:Exploration of the space-of-all-images:• Image shift, scale, rotation, warp, etc.Image shift, scale, rotation, warp, etc.
as vector operation? awkward...as vector operation? awkward...• ‘‘Video Textures’ (Schodl et al., SIGG2000)Video Textures’ (Schodl et al., SIGG2000)
http://www.gvu.gatech.edu/perception/projects/videotexture/http://www.gvu.gatech.edu/perception/projects/videotexture/
• ‘‘Face Space’, ‘eigenfaces’, etc:Face Space’, ‘eigenfaces’, etc:– Gather many example images of facesGather many example images of faces
Use PCA to find desirable axes: Use PCA to find desirable axes: gender, age, facial expression...gender, age, facial expression...
• Depth images: is this still 2D? Depth images: is this still 2D?
Texture Synthesis: Learned VectorsTexture Synthesis: Learned Vectors
• Texture Synthesis/elaborationTexture Synthesis/elaborationEfros98, Wei/Levoy99, Ashikhmin2000,...Efros98, Wei/Levoy99, Ashikhmin2000,...
Input Sample
Synthesized Result
Texture Synthesis: Learned VectorsTexture Synthesis: Learned Vectors
• Core Idea: Core Idea:
– for each pixel, for each pixel, make a vector of neighborhood pixel values:make a vector of neighborhood pixel values:
– Make a ‘dictionary’ of these vectors:Make a ‘dictionary’ of these vectors:given a vector , given a vector ,
– what are the most likely values of ?what are the most likely values of ?
Dictionary: crude, local pattern learningDictionary: crude, local pattern learning– Use dictionary to build a new image from Use dictionary to build a new image from
‘seed’ pixels/regions chosen at random.‘seed’ pixels/regions chosen at random.
II0000
II0101
II0202
……II1010
II1111
II1212
……
I = I =
Texture Synthesis: Learned VectorsTexture Synthesis: Learned Vectors
• Refinements & Extensions:Refinements & Extensions:– Faster, simpler methods (Ashkhmin: real-time!)Faster, simpler methods (Ashkhmin: real-time!)
• http://www.cs.utah.edu/~michael/ts/http://www.cs.utah.edu/~michael/ts/ – Hertzmann(2001): Image analogiesHertzmann(2001): Image analogies
http://mrl.nyu.edu/projects/image-analogies/http://mrl.nyu.edu/projects/image-analogies/
• Given images A,B,C, ‘learn’ AGiven images A,B,C, ‘learn’ AB mapping. then find DB mapping. then find DSolves ‘A is to B’ as ‘C is to D’ problemsSolves ‘A is to B’ as ‘C is to D’ problems
• Recreates some local effects—brush strokes, blur, etc.Recreates some local effects—brush strokes, blur, etc.• Wider variety of user-guided reconstruction..Wider variety of user-guided reconstruction..
– Recent extensions: Recent extensions: video textures, depth, curves, shapes, video textures, depth, curves, shapes, textures on surfaces, colorizing B/W textures on surfaces, colorizing B/W
images...images...
Texture Synthesis: Modified VectorsTexture Synthesis: Modified Vectors
• ‘‘Image Inpainting’ –repair damaged images Image Inpainting’ –repair damaged images http://www.ece.umn.edu/users/marcelo/restoration.htmlhttp://www.ece.umn.edu/users/marcelo/restoration.html
• ‘‘Reaction/Diffusion’ textures: Reaction/Diffusion’ textures: http://www.gvu.gatech.edu/people/faculty/greg.turk/reaction_diffusion/reaction_diffusion.htmlhttp://www.gvu.gatech.edu/people/faculty/greg.turk/reaction_diffusion/reaction_diffusion.html
iteratively compute new pixel intensity from iteratively compute new pixel intensity from neighbors + rules. Image evolves over timeneighbors + rules. Image evolves over time
• Anisotropic Diffusion, Curvature Flow: Anisotropic Diffusion, Curvature Flow: iterative rules (PDEs) for progressive iterative rules (PDEs) for progressive edge-preserving image smoothingedge-preserving image smoothing
! NOT EQUIVALENT !! NOT EQUIVALENT !
? Digital image: pixel-by-pixel operations ? ? Digital image: pixel-by-pixel operations ?
? Real Image: point-by-point operations ?? Real Image: point-by-point operations ?
Source of much frustration & error!Source of much frustration & error!
Digital Image Digital Image ==== Real Image Real Image
Also see Alvy Ray Smith’s article entitled:Also see Alvy Ray Smith’s article entitled:
“A pixel is NOT a little square,“A pixel is NOT a little square,
a pixel is NOT a little square,a pixel is NOT a little square,
a pixel is NOT a little square!”a pixel is NOT a little square!” found here.found here.
Real Real DigitalDigital
– Image Image IntensityIntensity– Image Image Spectral DistributionSpectral Distribution (‘color’) (‘color’)– Image Image GeometryGeometry (Position,Angle, Distortion, …) (Position,Angle, Distortion, …)– Image Image ResolutionResolution (‘sharpness’ or ‘focus’) (‘sharpness’ or ‘focus’)
Real Images:Real Images: Smooth,Smooth,
Continuous, Continuous,
Variable, Variable,
CompleteComplete
Unlimited…Unlimited…
Digital Images:Digital Images: Sampled,Sampled, Discrete,Discrete, Quantized, Quantized, Fixed Fixed Limited …Limited …
Conversion Method?Conversion Method?
• Real Image == 2D Light Intensity map: I(x,y)Real Image == 2D Light Intensity map: I(x,y)
• Digital Image == 2D grid of numbers: I(m,n)Digital Image == 2D grid of numbers: I(m,n)
I(x,y)I(x,y)
x,yx,y
??
I(m,n)I(m,n)
m,nm,n
Digital Digital Real : ‘Reconstruction’ Real : ‘Reconstruction’
A pixel sets strength of a real A pixel sets strength of a real Basis functionBasis function(aka ‘reconstruction filter’s impulse (aka ‘reconstruction filter’s impulse
response’)response’)
I(m,n)I(m,n)
m,nm,n
00 0.50.5-0.5-0.5
1.01.0 I(x,y)I(x,y)
x,yx,y
‘‘Box’Box’
Digital Digital Real : ‘Reconstruction’ Real : ‘Reconstruction’
A pixel sets strength of a real A pixel sets strength of a real Basis functionBasis function(aka ‘reconstruction filter’s impulse (aka ‘reconstruction filter’s impulse
response’)response’)
I(m,n)I(m,n)
m,nm,n
00 1.01.0-1.0-1.0
1.01.0 I(x,y)I(x,y)
x,yx,y
‘‘Linear’Linear’
Digital Digital Real : ‘Reconstruction’ Real : ‘Reconstruction’
A pixel sets strength of a real A pixel sets strength of a real Basis functionBasis function(aka ‘reconstruction filter’s impulse (aka ‘reconstruction filter’s impulse
response’)response’)
I(m,n)I(m,n)
m,nm,n
00 1.01.0-1.0-1.0
1.01.0I(x,y)I(x,y)
x,yx,y
2.02.0
‘‘Cubic’Cubic’
RealRealDigital: ‘Sampling’Digital: ‘Sampling’
• If I(x,y) is ‘smooth enough’ already,If I(x,y) is ‘smooth enough’ already,
• ?Why not just grab I(x,y) values at (m,n) ??Why not just grab I(x,y) values at (m,n) ?
I(x,y)I(x,y)
x,yx,y
I(m,n)I(m,n)
m,nm,n
‘‘Smooth enough’ to Sample?Smooth enough’ to Sample?
• Because ‘smooth enough’ is Because ‘smooth enough’ is undefinedundefined;;
• Samples may hit spurious peaks, valleysSamples may hit spurious peaks, valleys
I(x,y)I(x,y)
x,yx,y
I(m,n)I(m,n)
m,nm,n
BAD!BAD!
Real Real Digital : ‘Pre-filter’ Digital : ‘Pre-filter’
‘‘Smoothing’ Defined : Linear Pre-filterSmoothing’ Defined : Linear Pre-filter
Pixel = Pixel = local weighted averagelocal weighted average of real image of real imagearound pixel sample pointaround pixel sample point
I(x,y)I(x,y)
x,yx,y
m,nm,n
00 1.01.0-1.0-1.0
1.01.0
2.02.0
‘‘Cubic’Cubic’weightsweights
WeightWeight1.01.0
Pre-Filter functionsPre-Filter functions
• Pre-filters == anti-aliasingPre-filters == anti-aliasing in Comp Graphics in Comp Graphics
• SHOULDSHOULD be done by be done by
continuous integration,continuous integration,
• USUALLYUSUALLY done by done by
super-samplingsuper-sampling
? Quality Measures ?? Quality Measures ?00 1.01.0-1.0-1.0
1.01.0
2.02.0
‘‘Bi-cubic’Bi-cubic’filterfilter
00 1.01.0-1.0-1.0
1.01.0‘‘Bi-linear’Bi-linear’
filterfilter
00 0.50.5-0.5-0.5
1.01.0
‘‘Box’Box’filterfilter
‘‘Smoothness Measures’Smoothness Measures’
• Sine wave magic:Sine wave magic:1)1) ALLALL (continuous)(continuous) functionsfunctions are a are a (possibly infinite)(possibly infinite)
weighted sum of sinusoids: weighted sum of sinusoids: f(x) = Af(x) = A00sin(wsin(w00x) + Bx) + B00cos(wcos(w00x)+x)+
A A11sin(wsin(w11x) + Bx) + B11cos(wcos(w00x)+…x)+…
2)2) ForFor ALL ALL FILTERS FILTERS (box, linear, cubic,…, anything…)(box, linear, cubic,…, anything…)
sinusoid in sinusoid in scaled, shifted sinusoid out scaled, shifted sinusoid out
THUS– compute response to ALL signals from sinusoids THUS– compute response to ALL signals from sinusoids
3)3) High-Frequency Sinusoids in a signal hold High-Frequency Sinusoids in a signal hold ALLALL of its sampling/aliasing troubles! of its sampling/aliasing troubles!
1-D Aliasing Example1-D Aliasing Example
Real image:Real image: I(x,y) =sin(2 I(x,y) =sin(2fx) f=3.0fx) f=3.0
Real
Aliasing: Aliasing: when bad conversionwhen bad conversion
scrambles the frequenciesscrambles the frequenciesof sinusoidsof sinusoids
1-D Aliasing Example1-D Aliasing Example
Real image:Real image: I(x,y) =sin(2 I(x,y) =sin(2fx) f=3.0fx) f=3.0
Sample at: Sample at: x(n) = n / 36.0 (~12 samples per cycle)x(n) = n / 36.0 (~12 samples per cycle)
Sample
Real
Aliasing ExampleAliasing Example
Real image:Real image: I(x,y) =sin(2 I(x,y) =sin(2fx) f=3.0fx) f=3.0
Sample at: Sample at: x(n) = n / 36.0 (~12 samples per cycle)x(n) = n / 36.0 (~12 samples per cycle)
Reconstruct:Reconstruct: (use ‘box’ filter) (use ‘box’ filter) approximates f=3.0 approximates f=3.0
Sample
Real
Display
Aliasing ExampleAliasing Example
Real image: I(x,y) =sin(2Real image: I(x,y) =sin(2fx) f=3.0fx) f=3.0
Sample at: x(n) = n / 36.0 (~12 samples per cycle)Sample at: x(n) = n / 36.0 (~12 samples per cycle)
Reconstruct: (use ‘box’ filter) Reconstruct: (use ‘box’ filter) approximates f=3.0 approximates f=3.0
Another Real Image:Another Real Image: I(x,y) = sin(2 I(x,y) = sin(2fx) where f= 39.0fx) where f= 39.0
Sample
Real
Display
Real
Aliasing ExampleAliasing Example
Real image: I(x,y) =sin(2Real image: I(x,y) =sin(2fx) f=3.0fx) f=3.0
Sample at: x(n) = n / 36.0 (~12 samples per cycle)Sample at: x(n) = n / 36.0 (~12 samples per cycle)
Reconstruct: (use ‘box’ filter) Reconstruct: (use ‘box’ filter) approximates f=3.0 approximates f=3.0
Another Real Image:Another Real Image: I(x,y) = sin(2 I(x,y) = sin(2fx) where f= 39.0fx) where f= 39.0
Sample at: Sample at: x(n) = n / 36.0 (~0.92 samples per cycle) x(n) = n / 36.0 (~0.92 samples per cycle)
Sample
Real
Display
Sample
Real
Aliasing ExampleAliasing Example
Real image: I(x,y) =sin(2Real image: I(x,y) =sin(2fx) f=3.0fx) f=3.0
Sample at: x(n) = n / 36.0 (~12 samples per cycle)Sample at: x(n) = n / 36.0 (~12 samples per cycle)
Reconstruct: (use ‘box’ filter) Reconstruct: (use ‘box’ filter) approximates f=3.0 approximates f=3.0
Another Real Image:Another Real Image: I(x,y) = sin(2 I(x,y) = sin(2fx) where f= 39.0fx) where f= 39.0
Sample at: Sample at: x(n) = n / 36.0 x(n) = n / 36.0 (~0.92 samples per cycle)(~0.92 samples per cycle)
Reconstruct: Reconstruct: (use ‘box’ filter)(use ‘box’ filter) approximates f=3.0 !! approximates f=3.0 !!
Sample
Real
Display
Sample
Real
Display
Aliasing ExampleAliasing Example
Real image: I(x,y) =sin(2Real image: I(x,y) =sin(2fx) f=3.0fx) f=3.0
Sample at: x(n) = n / 36.0 (~12 samples per cycle)Sample at: x(n) = n / 36.0 (~12 samples per cycle)
Reconstruct: (use ‘box’ filter) Reconstruct: (use ‘box’ filter) approximates f=3.0 approximates f=3.0
Another Real Image:Another Real Image: I(x,y) = sin(2 I(x,y) = sin(2fx) where f= 39.0fx) where f= 39.0
Sample at: Sample at: x(n) = n / 144.0 x(n) = n / 144.0 ( 4.0 samples per cycle)( 4.0 samples per cycle)
Reconstruct: Reconstruct: (use ‘box’ filter)(use ‘box’ filter) approximates f=39.0 approximates f=39.0
Sample
Real
Display
SampleSample
Real
DisplayDisplay
Aliasing ExampleAliasing Example
? How much sampling is enough? ? How much sampling is enough?
• Depends on quality of Depends on quality of reconstructionreconstruction filter filter
• Rule-of-Thumb: at least 4 samples per cycle of the Rule-of-Thumb: at least 4 samples per cycle of the highest frequency sinusoidhighest frequency sinusoid
• Nyquist CriterionNyquist Criterion: 2 samples/cycle is theoretical min. : 2 samples/cycle is theoretical min.
Sample
Real
Display
SampleSample
Real
DisplayDisplay
Conclusion:Conclusion:
• Digital images are compact, indirect, Digital images are compact, indirect, ambiguous descriptions of light distributionsambiguous descriptions of light distributions
• Digital images are vectorsDigital images are vectors
• Digital image patterns can be learnedDigital image patterns can be learned
• Image-space manipulations must explicitly Image-space manipulations must explicitly address real-digital (discrete-continuous) address real-digital (discrete-continuous) conversions to avoid aliasing artifacts.conversions to avoid aliasing artifacts.
Aliasing ExampleAliasing Example
Real image:Real image: I(x,y) =sin(2 I(x,y) =sin(2fx) f=3.0fx) f=3.0
Sample at: Sample at: x(n) = n / 36.0 (~12 samples per cycle)x(n) = n / 36.0 (~12 samples per cycle)
Reconstruct:Reconstruct: (use ‘box’ filter) (use ‘box’ filter) approximates f=3.0 approximates f=3.0
Another Real Image:Another Real Image: I(x,y) = sin(2 I(x,y) = sin(2fx) where f= 39.0fx) where f= 39.0
Sample at: Sample at: x(n) = n / 36.0 x(n) = n / 36.0
Reconstruct: Reconstruct: (use ‘box’ filter)(use ‘box’ filter) approximates f=3.0 approximates f=3.0
Sample
Real
Display
Sample
Real
Display
Image Space MethodsImage Space Methods
• Intro: What can we do with images?Intro: What can we do with images?• Re-sampling:Continuous & Discrete FiltersRe-sampling:Continuous & Discrete Filters
– Interpolation: What’s ‘in-between’ a pixel?Interpolation: What’s ‘in-between’ a pixel?– Antialiasing: What makes a good sampling?Antialiasing: What makes a good sampling?– Math Rigor: the Fourier TransformMath Rigor: the Fourier Transform– Examples: box, triangle, Mitchell-NetravaliExamples: box, triangle, Mitchell-Netravali
• Compositing: TransparencyCompositing: Transparency• Un-Compositing: Matte SeparationsUn-Compositing: Matte Separations• Further extensions: Further extensions:
– Environment MatteEnvironment Matte– Polynomial Texture MapPolynomial Texture Map– Texture Elaboration, Image Inpainting(Sigg2000)Texture Elaboration, Image Inpainting(Sigg2000)
OLDOLD
• ‘‘Image’Image’==A 2D map of light intensities from a lens==A 2D map of light intensities from a lens• ‘‘Digital Image’Digital Image’==a 2D grid of numbers (pixels)==a 2D grid of numbers (pixels)
• PROBLEMS: PROBLEMS: – What is What is betweenbetween all those points? all those points?– WhyWhy go digital? Why is it go digital? Why is it betterbetter than film? than film? – How can I combine the best of many pictures? Edit How can I combine the best of many pictures? Edit
easily?easily?
GOAL:GOAL:– More Flexibility; let me do more than just display!More Flexibility; let me do more than just display!
Image-Space TechniquesImage-Space Techniques
GOAL: More flexibility!GOAL: More flexibility!
• Compositing /Matte: cut-and-paste, Compositing /Matte: cut-and-paste, transparency , the ‘digital optical bench’ transparency , the ‘digital optical bench’ (Pixar`84)…(Pixar`84)…
• Warp: image as a ‘rubber sheet’, you can Warp: image as a ‘rubber sheet’, you can cut, stretch, and change at willcut, stretch, and change at will
• Environment Matte (Salesin99…)Environment Matte (Salesin99…)
• Video Textures(Schodl 2000)Video Textures(Schodl 2000)
• Polynomial Texture Map (Malzburg2001)Polynomial Texture Map (Malzburg2001)