Super Resolution
-
Upload
roberto-medico -
Category
Education
-
view
555 -
download
0
Transcript of Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Super Resolution
Federico D’Amato Roberto Medico
University of Florence
June 9, 2014
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Outline
1 Introduction
2 Image Registration
3 Imaging Process
4 IBP
5 Irani and Peleg Algorithm
6 Gradient-like Method
7 Color SR
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Super Resolution Techniques
Super Resolution is a class of techniques that enhance theresolution of an imaging system. There are 3 main approachesto SR reconstruction of an high-resolution image from lowerresolution image(s):
• Interpolation-based• Example-learning-based• Multi-image-based
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Interpolation-based
Figure: Interpolaton methods try to achieve a best approximation of apixel’s color and intensity based on the values at surrounding pixels
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Example-learning-based
Correspondences between low-resolution and high-resolutionimages are learned from a set of training images. The trainingset consists of high-resolution / low-resolution pairs.
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Multi-Image Super Resolution
Super-Resolution from image sequences attempts toreconstruct the original scene image with high resolution givena set of observed images at lower resolution.
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Why Super Resolution?
Limit of camera resolution:• Spatial limit→ determined by spatial density of optical
sensor• Optical blur→ determined by the lens
How to improve camera resolution?• Direct method: improving imaging system by
manufacturing technique (pixel density, lens size)• Use of Super-resolution reconstruction:
• Use of spatial sub-pixel movement information betweenframe
• Reconstruction from low-resolution image sequences tohigh-resolution image
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Digital Imaging System
Key components:1 the sensor⇒ limit on highest spatial frequency2 the lens⇒ optical blur
Figure: Image acquisition process
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Spatial Aliasing
Spatial aliasing is an effect that causes different signals tobecome indistinguishable (or aliases of one another) whenspatially-sampled. When a digital image is recorded, areconstruction is performed by the imaging device→ if theimage data is not properly processed during sampling orreconstruction, the reconstructed image will differ from theoriginal image (it’s called an ’alias’ of the original scene)
Figure: One signal and its alias
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Aliasing components
• Sensor is constructed from a finite number of discretepixels→ reconstruction of real world scene is affected byaliasing effects
• It’s impossible to completely remove aliasing componentsusing anti-aliasing filters⇒ information in the aliasedcomponents is used to recover spatial frequenciesbeyond sensor resolution
• It’s the possible to use information to improve the imageresolution
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Aliasing effect on patterns of increasing frequency→ poor (orcompletely wrong) image reconstruction
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Figure: Sub-Pixel shifted signals
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Naive approach
• How can we compute the valueof pixel X?
• By applying some interpolationtechnique (e.g. bilinear) toneighbours A,B,C,D of X
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Multi-image Approach
• LR image resolution: MxN• Images displacement: half a
pixel• Combining the pixel of the LR
images in a more dense grid2Mx2N returns an image athigher resolution.
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Outline
1 Introduction
2 Image Registration
3 Imaging Process
4 IBP
5 Irani and Peleg Algorithm
6 Gradient-like Method
7 Color SR
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Registration
• Computation of the changes (displacements) betweenimages is known as registration
• 2D Rotation matrix:
• Displacement are computed between one image g0 (takenas reference image) and all the others image.Displacement between gk and g0 can be written as:
g0(x , y) = gk (x cos(Θ)−y sin(Θ)+a, y cos(Θ)+x sin(Θ)+b)
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Figure: Example of rigid registration
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
• Expand sin Θ and cos Θ to the first two terms of their Taylorseries:
g0(x , y) ≈ gk (x + a− yΘ− xΘ2
2, y + b + xΘ− y
Θ2
2)
• Expand gk to the first term of its Taylor series:
g0(x , y) ≈ gk (x , y)+(a−yΘ−xΘ2
2)∂gk
∂x+(b+xΘ−y
Θ2
2)∂gk
∂y
• The error function between gk and g0 is:
E(a,b,Θ) =∑
[gk (x , y) + (a− yΘ− xΘ2
2)∂gk
∂x+
+(b + xΘ− yΘ2
2)∂gk
∂y− g0(x , y)]2
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
• ∂E∂a = 0, ∂E
∂b = 0, ∂E∂Θ = 0
• Ignoring non-linear terms and small coefficients we get thefollowing system of linear equations, whose solution(a,b,Θ) minimizes the difference between g0 and gkwarped by (a,b,Θ):∑
g2x a +
∑gxgyb +
∑Agx Θ =
∑gxgt∑
g2y b +
∑gxgya +
∑Agy Θ =
∑gygt∑
A2Θ +∑
Agyb +∑
Agxa =∑
Agt
where gx = ∂gk∂x , gy = ∂gk
∂y , gt = g0 − gk and A = xgy − ygx
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Iterative refinement
When it’s not possibile to assume that the displacementsbetween g0 and gk are sufficiently small, an iterativerefinement algorithm is used:
1 Assume no motion between frames2 for n=0,1,..
• Compute (a(n),b(n),Θ(n)) and add the computed motionto the current estimate (a,b,Θ)
• Warp frame gk towards g0 using (a,b,Θ) and return to 2.
The process ends when (a(n),b(n),Θ(n)) ≈ 0.
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Outline
1 Introduction
2 Image Registration
3 Imaging Process
4 IBP
5 Irani and Peleg Algorithm
6 Gradient-like Method
7 Color SR
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Receptive Field
The receptive field of a LR pixel (m,n) of the kth LR image isdefined by its center (x , y) and its shape, determined by theregion of support of hPSF (·) in the high resolution grid. Thecenter (x , y) can be computed by:
x = ak + sxm cos Θk − syn sin Θk
y = bk + sxm sin Θk + syn cos Θk
where• (ak ,bk ) is the translation of the kth image from g0
• Θk is the rotation between the kth image and g0
• sx and sy are the upscaling factors in x and y directions
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Figure: Receptive Field
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
• Imaging process can be modeled as:
gk (m,n) = σk (hPSF (f (x , y)) + ηk (x , y))
where• gk is the kth observed LR image• f is the original image• hPSF is a blurring operator• ηk is an additive noise term• σk is a non-linear function that digitizes and quantizes
image into pixels (including displacement)• (x , y) is the center of the receptive field of the detector
whose output is gk (m,n)
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Figure: Simulated Imaging Process
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Blurring Operator Estimate
Given a generic imaging device, we can empyrically estimateits blurring function h(·) analyzing the output of the imagingprocess of well-known sample scenes.
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Outline
1 Introduction
2 Image Registration
3 Imaging Process
4 IBP
5 Irani and Peleg Algorithm
6 Gradient-like Method
7 Color SR
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Iterative Back Projection
• Iterative algorithm based on a set of K low resolutionimages of the same scene with known displacements
• Goal: to improve an initial guess of the HR imageiteratively minimizing an error function usingback-projection
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Hypothesis
• Assumptions:• displacement between images can be described by three
parameters:• a, horizontal shift• b, vertical shift• Θ, rotation angle
• ignores acceleration of the camera while imaging a singleframe
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Data:• f 0: initial guess of the HR image• gk : set of LR observed images• hPSF , (ak ,bk ,Θk ) ∀k = 1, ..,K
for n = 0,1, .. do
1 Compute the set of K simulated LR images {g(n)k } from f (n)
2 Compute en between gk and gk(n)
if en > ε thenUpdate guess f (n+1) by back-projecting the error on f (n)
elsereturn f (n)
endend
Algorithm 1: Iterated Back Projection
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Outline
1 Introduction
2 Image Registration
3 Imaging Process
4 IBP
5 Irani and Peleg Algorithm
6 Gradient-like Method
7 Color SR
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Simulated Imaging Process
How can we programmatically simulate the device imagingprocess?
Def.A low resolution pixel y is influenced by a high resolution pixelx if x ∈ y’s receptive field
Def.A low resolution image g is influenced by a high resolutionpixel x if ∃y ∈ g influenced by x
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
g(n)(~y) =∑~x
f (n)(~x)hPSF (~x − ~z~y )
where• hPSF is the point-spread kernel of the imaging blur• ~x is an HR pixel• ~y is a LR pixel influenced by ~x• ~z is the center of y ’s the receptive field
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Idea
• Given the g(n)k simulated LR images, the goal is to
minimize the error between {g(n)k } and {gk }.
• The minimization is obtained with the iterativeback-projection scheme, where ek = gk − g(n)
k isweighted computing the influence of every LR pixel ~y onHR pixel ~x , using:
hBP(~x − ~z~y )∑~y∈
⋃k Yk,~x
hBP(~x − ~z~y )
where Yk ,~x is the set: {~y ∈ gk | ~y is influenced by ~x}• ~y has more influence when ~x is close to ~z~y , center of ~y ’s
receptive field
• The error is then multiplied by a factor:hBP(~x−~z~y )
c
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Guess Improvement
• The value of ~x in the next guess f (n+1) is calculatedsumming up the weighted errors on all the LR pixel ~y itinfluences
f (n+1)(~x) = f (n)(~x)+∑
~y∈⋃
k Yk,~x
(gk (~y)−g(n)k (~y))
(hBP(~x − ~z~y ))2
c∑
~y∈⋃
k Yk,~xhBP~x~y
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Back-Projection Kernel
• hBP~x~y affects how much the error on LR pixel ~y (influenced
by ~x) contributes to the value of HR pixel ~x in the nextguess f (n+1)
• hBP affects the characteristics of the solution image, e.g.its smoothness
• A possible choice is hBP = hPSF
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Initial Guess Choice
• Initial guess f (0) can influence the output of the algorithm,i.e. which HR image is reached first
• One possibile choice of f (0) is taking the average of theupscaled LR images gk
• This choice doesn’t affect performance
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Algorithm Complexity
The complexity is O(KN min{M, log N}) where:• K is the number of LR images• N is the size of the HR image f• M is the size of HPSF kernel
Parallelism can be used to compute the contributions of LRpixels indipendently.
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Improvements
• Stable Pixels: HR pixels that don’t change value for 2consecutive iterations won’t be considered in the followingiterations
• Noise reduction: minimal and maximal values ofgk (~y)− g(n)
k (~y) are ignored in computing the weightedaverage of the contributions of the LR pixels in the iterativeback-projection scheme
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Outline
1 Introduction
2 Image Registration
3 Imaging Process
4 IBP
5 Irani and Peleg Algorithm
6 Gradient-like Method
7 Color SR
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Error Function
The error function to minimize is given by the MSE between thesimulated images g(n)
k and the observed images gk :
e(n) =
∑k (gk − g(n)
k )2
K
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Algorithm
f (n+1) = f (n) − λG
where• G =
∑k HBP(g(n)
k − gk )
• g(n)k is the kth simulated LR image at the nth iteration
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Outline
1 Introduction
2 Image Registration
3 Imaging Process
4 IBP
5 Irani and Peleg Algorithm
6 Gradient-like Method
7 Color SR
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
YIQ representation
• Y component represents the luminance information• I and Q represent the chrominance information
Most of the energy is concentrated in the Y component.
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Color Super Resolution
It’s possible to use the SR algorithm even on color images,going through 4 steps:
1 transform the color images in YIQ representation2 apply the SR algorithm to the Y component images3 register the images at the two (I,Q) chrominance image
sequences using parameters found in 2. Create anaverage for each of the I and Q components
4 fuse the HR Y component and LR I and Q components togenerate a HR RGB image
F. D’Amato, R. Medico Super Resolution
Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
References
• ’Image sequence enhancement using sub-pixeldisplacements’, Keren, D., Peleg, S. ; Brada, R.; Dept. ofComput. Sci., Hebrew Univ., Jerusalem, Israel
• ’Video Super-resolution Reconstruction Based onSub-pixel Registration and Iterative Back Projection’,Journal of Electronic Imaging,Vol. 18, No. 1, 2009,Feng-Qing Qin, Xiao-HaiHe, Wei-Long Chen, Xiao-MinYang, and Wei Wu
• ’Improving resolution by image registration’, Michal Irani,Shmuel Peleg
• ’Super-Resolution’, Pradeep Gaidhani
F. D’Amato, R. Medico Super Resolution