A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify...
Transcript of A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify...
![Page 1: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/1.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
A (Very Short) Primer to Image Processing
Davide Bacciu
Dipartimento di InformaticaUniversità di [email protected]
Intelligent Systems for Pattern Recognition (ISPR)
![Page 2: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/2.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Image Format
Images are matrices of pixel intensities or color values (RGB)
Other representations exist, but not of interest for thecourseCIE-LUV is often used in image processing due toperceptual linearity
Image difference is more coherent
![Page 3: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/3.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Machine Vision Applications
Region of interest identification
Object classification
![Page 4: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/4.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Machine Vision Applications
Image Segmentation Semantic segmentation
![Page 5: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/5.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Machine Vision Applications
Automated imagecaptioning
...and much more
![Page 6: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/6.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Key Questions?
How do we represent visual information?InformativeInvariant to photometric and geometric transformationsEfficient for indexing and querying
How do we identify informative parts?Whole image? Generally not a good idea...Must lead to good representationsEdges, blobs, segments
![Page 7: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/7.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Image Histograms
Represent the distribution of some visual information onthe whole image
ColorEdgesCorners
Color histograms are one of the earliest image descriptorsCount the number of pixels of a given color (normalize!)Need to discretize and group the RGB colorsAny information concerning shapes and position is lost
![Page 8: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/8.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Color Histograms
Images can be compared, indexed and classified based ontheir color histogram representation
%Compute histogram on s i n g l echannel
[ yRed , x ] = i m h i s t ( image ( : , : , 1 ) ) ;%Disp lay histogrami m h i s t ( image ( : , : , 1 ) ) ;
impor t cv2 #OpenCVimage = cv2 . imread ( " image . png " )# loop over the image channelschans = cv2 . s p l i t ( image )co lo rs = ( " b " , " g " , " r " )f o r ( chan , co l o r ) i n z ip ( chans , co lo rs ) :
h i s t = cv2 . c a l c H i s t ( [ chan ] , [ 0 ] , None , \[ 256 ] , [ 0 , 256] )
![Page 9: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/9.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Describing Local Image Properties
Capturing information on image regionsExtract multiple local descriptors
Different locationDifferent scale
Several approaches, typically performing convolutionbetween a filter and the image region
Need to identitygood regions ofinterest (later)
![Page 10: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/10.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Intensity Vector
The simplest form of localized descriptor
Normalize w to make the descriptorinvariant w.r.t. affine intensity changes
No invariance to pose, location, scale (poorly discriminative)
d =w− w‖w− w‖
![Page 11: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/11.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Distribution-based Descriptors
Represent local patches by histograms describing properties(i.e. distributions) of the pixels in the patch
What is the simplest approach you can think of?Histogram of pixel intensities on a subwindowNot invariant enough
A descriptor that is invariant toIllumination (normalization)Scale (captured at multiple scale)Geometric transformations (rotation invariant)
![Page 12: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/12.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Scale Invariant Feature Transform (SIFT)
1 Center the image patch on a pixel x , y of image I2 Represent image at scale σ
Controls how close I look at an image
Convolve the image with a Gaussian filter with std σ
Lσ(x , y) = G(x , y , σ) ∗ I(x , y)
G(x , y , σ) = exp(−x2 + y2
2σ2
)
![Page 13: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/13.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Gaussian Filtering of an Image
Create the Gaussian filter%A gaussian f i l t e r between −6 and +6h=13 , w=13 , sigma =5;%Create a mesh of p i x e l po in t s i n [−6 ,+6][ h1 w1]= meshgrid(−(h−1) / 2 : ( h−1) /2 , −(w−1)
/ 2 : ( w−1) / 2 ) ;%Compute the f i l t e rhg= exp(−(h1 .^2+w1. ^ 2 ) / (2∗ sigma ^2) ) ;%Normalizehg = hg . / sum( hg ( : ) ) ;
Then, convolve it with theimageOr you use library functions todo all this for you
![Page 14: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/14.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Gaussian Filtering of an Image
Create the Gaussian filter%A gaussian f i l t e r between −6 and +6h=13 , w=13 , sigma =5;%Create a mesh of p i x e l po in t s i n [−6 ,+6][ h1 w1]= meshgrid(−(h−1) / 2 : ( h−1) /2 , −(w−1)
/ 2 : ( w−1) / 2 ) ;%Compute the f i l t e rhg= exp(−(h1 .^2+w1. ^ 2 ) / (2∗ sigma ^2) ) ;%Normalizehg = hg . / sum( hg ( : ) ) ;
Then, convolve it with theimageOr you use library functions todo all this for you
I s c a l e = i m g a u s s f i l t ( I , sigma ) ;
σ = 5
σ = 0.05
![Page 15: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/15.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Scale Invariant Feature Transform (SIFT)
1 Center the image patch on a pixel x , y of image I2 Represent image at scale σ3 Compute the gradient of intensity in the patch
Magnitude mOrientation θ
Use finite differences:
mσ(x , y) =√(Lσ(x + 1, y)− Lσ(x − 1, y))2 + (Lσ(x , y + 1)− Lσ(x , y − 1))2
θσ(x , y) = tan−1((Lσ(x , y + 1)− Lσ(x , y − 1))(Lσ(x + 1, y)− Lσ(x − 1, y))
)
![Page 16: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/16.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Gradient and Filters
A closer look at finite difference reveals
Gx =[
1 0 −1]∗ Lσ(x , y)
Gy =
10−1
∗ Lσ(x , y)
So
mσ(x , y) =√
G2x + G2
y and θσ(x , y) = tan−1(
Gy
Gx
)=
![Page 17: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/17.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Gradient Example
%Compute g rad ien t w i th c e n t r a l d i f f e r e n c e on x , y d i r e c t i o n s[ Gx, Gy ] = imgrad ientxy ( Ig , ’ c e n t r a l ’ ) ;
%Compute magnitude and o r i e n t a t i o n[m, the ta ] = imgrad ien t (Gx, Gy) ;
Ig m θ
![Page 18: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/18.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Scale Invariant Feature Transform (SIFT)
1 Center the image patch on a pixel x , y of image I2 Represent image at scale σ3 Compute the gradient of intensity in the patch4 Create a gradient histogram
4x4 gradient windowHistogram of 4x4 samples per window on 8 orientation binsGaussian weighting on center keypoint (width = 1.5σ)4× 4× 8 = 128 descriptor size
![Page 19: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/19.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
SIFT Descriptor
Normalize to unity for illumination invarianceThreshold gradient magnitude to 0.2 to avoid saturation(before normalization)Rotate all angles by main orientation to obtain rotationalinvariance
![Page 20: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/20.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
SIFT Facts
For long time the most used visual descriptorHOG: Histogram of oriented gradientsSURF: Speeded Up Robust FeaturesORB: an efficient alternative to SIFT or SURFGLOH: Gradient location-orientation histogram
SIFT is also a detector, although less used
SIFT in OpenCVimpor t cv2. . . #Image Readgray= cv2 . cv tCo lo r ( img , cv2 .COLOR_BGR2GRAY)s i f t = cv2 . x features2d . SIFT_create ( )#1 − Detect and then d i sp laykp = s i f t . de tec t ( gray , None )kp , des = s i f t . compute ( gray , kp )#2 − Detect and d i sp laykp , des = s i f t . detectAndCompute ( gray , None )
![Page 21: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/21.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Fourier Analysis
Images are functions returning intensity values I(x , y) onthe 2D plane spanned by variables x , yNot surprisingly, we can define the Fourier coefficients of a2D-DFT as
H(kx , ky ) =N−1∑x=1
M−1∑y=1
I(x , y)e−2πi(
xkxN +
ykyM
)
In other words, I can write my image as sum of sine and cosinewaves of varying frequency in x and y directions
![Page 22: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/22.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
The Convolution Theorem
The Fourier transform F of the convolution of two functions isthe product of their Fourier transforms
F(f ∗ g) = F(f )F(g)
Transforms convolutions in element-wise multiplications inFourier domainSuppose we are given an image I (a function) and a filter g(a function as well)......their convolution I ∗ g can be conveniently computed as
I ∗ g = (F )−1(F(I)F(g))
where (F )−1 is the inverse Fourier transform
Convolutional neural networks can be implemented efficiently inFourier domain!
![Page 23: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/23.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Image PR with DFT
1 Make a filter out of a pattern using Fourier transform F2 Convolve in Fourier domain and reconstruct with F−1
3 Threshold high pixel activation to generate response mask
![Page 24: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/24.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Global DescriptorsLocal DescriptorsSpectral Analysis
Practical Issues with DFT on Images
Previous example, in Matlab:[N,M] = s ize ( I ) ;mask = i f f t 2 ( f f t 2 ( I ) .∗ f f t 2 ( charPat ,N,M) ) > th resho ld ;
The DFT is simmetric (in both directions):Power spectrum is re-arranged to havethe (0,0) frequency at the center of theplot
The (0,0) frequency is the DC componentIts magnitude is typically out of scale w.r.t.other frequencies
H(0,0) =N−1∑x=1
M−1∑y=1
I(x , y)e0
Use log(abs(H·,·)) to plot the spectrum (orlog-transform the image)
![Page 25: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/25.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Edge DetectionBlob DetectorsInterest Point Detectors
Visual Feature Detector
RepeatabilityDetect the same feature in different image portions and indifferent images
Photometric - Changes in brightness and luminanceTranslation - Changes in pixel locationRotation - Changes to absolute or relative angle of keypointScaling - Image resizing or changes in camera zoomAffine Transformations - Non-isotrophic changes
![Page 26: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/26.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Edge DetectionBlob DetectorsInterest Point Detectors
Edge Detection
![Page 27: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/27.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Edge DetectionBlob DetectorsInterest Point Detectors
Edges and Gradients
Image gradient (graylevel)
∇I =[∂I∂x,∂I∂y
]direction of change of intensityEdges are pixel regions where...
Intensity gradient changes abruptly
The return of finite difference methods
Gx =∂I∂x≈ I(x + 1, y)− I(x − 1, y)
Gy =∂I∂y≈ I(x , y + 1)− I(x , y − 1)
Gx +1 0 −1+1 0 −1+1 0 −1
Gy +1 +1 +1
0 0 0−1 −1 −1
Prewitt
operators
![Page 28: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/28.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Edge DetectionBlob DetectorsInterest Point Detectors
Convolving Gradient OperatorsImage
Magnitude
Gx
Gy
![Page 29: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/29.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Edge DetectionBlob DetectorsInterest Point Detectors
Sobel Operator
An additional level of smoothing of the central difference
Gx =
+1 0 −1+2 0 −2+1 0 −1
Gy =
+1 +2 +10 0 0−1 −2 −1
![Page 30: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/30.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Edge DetectionBlob DetectorsInterest Point Detectors
In Code
Matlab%Create an h o r i z o n t a l ( x ) P r e w i t t f i l t e rh = f s p e c i a l ( ’ p r e w i t t ’ ) ; %Try a lso ’ sobel ’%Convolve i t to the image IgimH = i m f i l t e r ( Ig , h , ’ r e p l i c a t e ’ ) ;%Transpose f i l t e r f o r the y−d e r i v a t i v eimV = i m f i l t e r ( Ig , h ’ , ’ r e p l i c a t e ’ ) ;%MagnitudeM = u in t8 ( s q r t ( double ( ( imHor . ^ 2 ) + ( imVer . ^ 2 ) ) ) ) ;%Then p l o t . . .im too l ( imH) ; %etc . . .
Python# p r e w i t t maskskerne lx = np . ar ray ( [ [1 ,1 ,1 ] , [0 ,0 ,0 ] , [ −1 , −1 , −1] ] )kerne ly = np . ar ray ( [ [ −1 ,0 ,1 ] , [ −1 ,0 ,1 ] , [ −1 ,0 ,1 ] ] )
#convo lv ing f i l t e r simg_prewi t t x = cv2 . f i l t e r 2 D ( img_gray , −1, kerne lx )img_prewi t t y = cv2 . f i l t e r 2 D ( img_gray , −1, kerne ly )
#sobel (CV_8U i s the output data type , ks ize i s the kerne l s i ze )img_sobelx = cv2 . Sobel ( img_gray , cv2 .CV_8U,1 ,0 , ks ize =3)img_sobely = cv2 . Sobel ( img_gray , cv2 .CV_8U,0 ,1 , ks ize =3)
![Page 31: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/31.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Edge DetectionBlob DetectorsInterest Point Detectors
Blob Detection
Blobs are connected pixelsregions with little gradientvariabilityLaplacian of Gaussian (LoG)gσ(x , y) has maximum responsewhen centered on a circle ofradius
√2σ
∇2gσ(x , y) =∂2gσ
∂x2 +∂2gσ
∂y2
Typically using a scale normalized response
∇2normgσ(x , y) = σ2
(∂2gσ
∂x2 +∂2gσ
∂y2
)
![Page 32: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/32.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Edge DetectionBlob DetectorsInterest Point Detectors
LoG Blob Detection
1 Convolve image with a LoG filter at different scalesσ = kσ0 by varying k
2 Find maxima of squared LoG response1 Find maxima on space-scale2 Find maxima between scale3 Threshold
The LoG filter can be approximatedas a Difference of Gaussians (DoG)for efficiency
gkσ0(x , y)−gσ0(x , y) ≈(k − 1)σ2
0∇2g(k−1)σ0
![Page 33: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/33.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Edge DetectionBlob DetectorsInterest Point Detectors
Affine Detectors
Laplacian-based detectors are invariant to scale thanks tothe maximization in scale-space
Still not invariant to affine transformations
![Page 34: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/34.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Edge DetectionBlob DetectorsInterest Point Detectors
Maximally Stable Extremal Regions (MSER)
Extract covariant regions (blobs) that are stable connectedcomponents of intensity sets of the imageKey idea is to take blobs (Extremal Regions) which arenearly the same through a wide range of intensitythresholdsThe blobs are generated (locally) by binarizing the imageover a large number of thresholds
Invariance to affine transformation of image intensitiesStability (they are stable on multiple thresholds)Multi-scale (connected components are identified byintensity stability not by scale)Sensitive to local lighting effects, shadows, etc..
You can then fit an ellipse enclosing the stable region
![Page 35: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/35.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Edge DetectionBlob DetectorsInterest Point Detectors
Intuition on the MSER Algorithm
Generate frames from the image by thesholding it on allgraylevels
Capture those regions that from a small seed of pixel growto a stably connected regionStability is assessed by looking at derivatives of regionmasks in time (most stable⇒ minima of connected regionvariation)
![Page 36: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/36.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Edge DetectionBlob DetectorsInterest Point Detectors
MSER in Code
Again, in OpenCVimpor t cv2. . .#Load the mser de tec to r from OpenCVmser = cv2 . MSER_create ( )reg ions = mser . detectRegions ( img , None )#Create a convexhu l l enc los ing s tab le reg ionsh u l l s = [ cv2 . convexHul l ( p . reshape(−1, 1 , 2 ) ) f o r p i n reg ions ]#Draw detected reg ions on image copyv i s = img . copy ( )cv2 . p o l y l i n e s ( v is , hu l l s , 1 , (0 , 255 , 0 ) )cv2 . imshow ( ’ img ’ , v i s )
Matlab%Run MSER and re tu rns reg ionsreg ions = detectMSERFeatures ( Ig ) ;f i g u r e ; imshow ( Ig ) ; %p l o t imagehold on ;p l o t ( reg ions ) ; %over lap reg ions
%A l t e r n a t i v e l y can p l o t ac tua l reg ionsp l o t ( regions , ’ showPixe lL is t ’ , t rue , ’
showEl l ipses ’ , f a l s e ) ;
![Page 37: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/37.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Image Segmentation
The process of partitioning an image into set of homogeneouspixels, hoping to match object or their subparts
A naive approach? Applyk-means to pixels color (typicallyL*a*b) hoping to cluster togetherregionsA slightly less naive approach?Apply k-means to pixels colorand (x , y) position hoping toenforce some level of spatialinformation in clusters
![Page 38: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/38.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Normalized Cuts (Ncut)
Node = pixelaij = affinity between pixels(at a certain scale σ)
A cut of G is the set ofedges such whoseremoval makes G adisconnected graphBreaking graph into piecesby cutting edges of lowaffinityNormalized cut problem
NP-hardApproximate solution asan eigenvalue problem
Code: https://www.cis.upenn.edu/~jshi/software/
![Page 39: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/39.jpg)
Representing Visual ContentIdentify Informative Parts
Segmentation
Pixel Issue
Pixels in image are a lot!Ncut can take ages to completeLikewise many other advanced segmentation algorithms
Efficiency trick⇒ SuperpixelsGroup together similar pixelsCheap, local oversegmentationImportant that superpixels donot cross boundaries
Now apply segmentation/fusionalgorithms to superpixels: Ncut,Markov Random Fields, etc.
Code: https://ivrl.epfl.ch/research/superpixels
![Page 40: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/40.jpg)
Representing Visual ContentIdentify Informative Parts
SegmentationConclusions
Take Home Messages
Image processing is very much about convolutionsLinear masks to perform gradient operationsGaussian functions to apply scale changes (zooming in andout)
Visual content can be better represented by localdescriptors
Histograms of photo-geometric propertiesSIFT is intensity gradient histogram
Computational efficiency is often a driving factorConvolutions in Fourier domainSuperpixelsLightweight feature detector? Random sampling
![Page 41: A (Very Short) Primer to Image Processing · 2020-02-28 · Representing Visual Content Identify Informative Parts Segmentation A (Very Short) Primer to Image Processing Davide Bacciu](https://reader030.fdocuments.in/reader030/viewer/2022040917/5e91356392668a39a4642174/html5/thumbnails/41.jpg)
Representing Visual ContentIdentify Informative Parts
SegmentationConclusions
Next Lecture
Generative and Graphical ModelsIntroduction to a module of 6 lecturesA (very quick) refresher on probabilities (from ML)
Probability theoryConditional independenceInference and learning in generative models
Graphical models representationDirected graphical modelsUndirected graphical models