Image ProcessingImage ProcessingIB Paper 8 – Part AIB Paper 8 – Part A

Ognjen ArandjeloviOgnjen Arandjelovićć

http://mi.eng.cam.ac.uk/~oa214/

Lecture RoadmapLecture Roadmap

Face geometry Lecture 1:

Geometric image transformations

Lecture 2:

Colour and brightness enhancement

Lecture 3:

Denoising and image filtering

– – Image Denoising and Filtering –Image Denoising and Filtering –

Image Noise SourcesImage Noise Sources

Image noise may be produced by several sources:

Quantization

Photonic

Thermal

Electric

DenoisingDenoising

To effectively perform denoising, we need to consider the following issues:

Signal (uncorrupted image) modelTypically piece-wise constant or linear

Noise model (from the physics of image formation)Additive or multiplicative, Gaussian, white, salt and pepper…

Salt and Pepper NoiseSalt and Pepper Noise

Gaussian NoiseGaussian Noise

Modelling NoiseModelling Noise

Most often noise is additive:

Observed pixel luminance

True luminance Noise process

Additive Gaussian Noise – ExampleAdditive Gaussian Noise – Example

A clear original image was corrupted by additive white Gaussian noise:

Original, uncorrupted image Additive Gaussian noise

Additive Gaussian Noise – ExampleAdditive Gaussian Noise – Example

A clear original image was corrupted by additive white Gaussian noise:

Additive Gaussian noise Additive Gaussian noise

Additive Gaussian Noise – ExampleAdditive Gaussian Noise – Example

Taking a slice through the image can help us visualize the behaviour of noise better:

Temporal Average for Video DenoisingTemporal Average for Video Denoising

A video feed of a static scene can be easily denoised by temporal averaging, under the assumption of zero-mean additive noise:

Pixel luminance estimate

Pixel luminance in frame i

Average noise energy is reduced by a factor of N:

Temporal Averaging – ExampleTemporal Averaging – Example

Consider our noisy CCTV image from the previous lecture and the result of brightness enhancement:

Original image Brightness enhanced image

Temporal Averaging – ExampleTemporal Averaging – Example

The effect of temporal averaging over 100 frames is dramatic:

But note that moving objects cause blur.

The clarity of image detail is much improved.

Spatial AveragingSpatial Averaging

Although attractive, a static video feed is usually not available. However, a similar technique can be used by noting:

Images are mostly smoothly varying

Original smoothly varying signal and the signal corrupted with zero mean Gaussian noise

Simple Spatial AveragingSimple Spatial Averaging

Thus, we can attempt to denoise the signal by simple spatial averaging:

The result of averaging each neighbouring 7 (± 3) pixels

Simple Spatial Averaging – ExampleSimple Spatial Averaging – Example

Using out synthetically corrupted image:

Additive Gaussian noise Spatially averaged using 5 х 5 neighbourhood

Simple Spatial Averaging – ExampleSimple Spatial Averaging – Example

Consider the difference between the uncorrupted image and the corrupted and denoised images:

Before averaging After averaging

RMS difference = 29 RMS difference = 12

Simple Spatial Averaging – AnalysisSimple Spatial Averaging – Analysis

The result of averaging looks good, but a closer inspection reveals some loss of detail:

Difference image Magnified patch

Simple Spatial Averaging – AnalysisSimple Spatial Averaging – Analysis

To formally analyze the filtering effects, rewrite the original averaging expression:

Rectangular pulseConvolution integral

1D Convolution1D Convolution

A quick convolution re-cap:

f(x) h(x)

Flip and slide over

Discrete 1D ConvolutionDiscrete 1D Convolution

In dealing with discrete signals:

Flip and slide over

… 234 233 228 240 241 241 … 0 0 1 2 2 1

f(x)

h(x)

… 234 233 228 240 241 241 …

1 2 2 1 0 01 2 2 1 0 0

228+ 480+ 482+ 241 + …

2D Convolution2D Convolution

The concept of linear filtering as convolution with a filter (or kernel) extends to 2D and the integral becomes:

We shall be dealing with separable filters only

in which this is equivalent to two 1D convolutions:

Simple Spatial Averaging – AnalysisSimple Spatial Averaging – Analysis

By considering the effects of convolution in the frequency domain, we can now see why there was loss of detail:

Rectangular pulse function

Fourier transform

The sinc function

High frequencies are damped

White Noise ModelWhite Noise Model

This insight allows to devise the denoising filter in a principled way by considering the SNR over different frequencies:

Signal frequency spectrum

Noise frequency spectrum

Frequency

Ene

rgy

Frequency

Ene

rgy

White Noise ModelWhite Noise Model

This insight allows to devise the denoising filter in a principled way by considering the SNR over different frequencies:

Pass Do not pass

The Ideal LPF AgainThe Ideal LPF Again

As when we dealt with reconstructing a signal from a set of samples, we can low-pass filter by convolving with the sinc function in the spatial domain:

The key limitation is that the sinc function has a wide spatial support

Thus, in practice we often use filters that offer a better trade-off in terms of spatial support and bandwidth

Gaussian Low Pass FilterGaussian Low Pass Filter

The Gaussian LPF is one of the most commonly used LPFs. It possesses the attractive property of minimal space-bandwidth product.

1D Gaussian 2D Gaussian as a surface

2D Gaussian as an image

Gaussian LPF – Toy ExampleGaussian LPF – Toy Example

Using the Gaussian filter on our toy 1D example produces a nearly perfect filtering result:

RMS error reduction from 0.1 to 0.02

Gaussian LPF – ExampleGaussian LPF – Example

Using out synthetically corrupted image:

Additive Gaussian noise LP filtered using a Gaussian with

Low, Band and High-Pass FiltersLow, Band and High-Pass Filters

A quick recap of relevant terminology:

Frequency

Ga

in

Low-pass Band-pass High-pass

Low, Band and High-Pass FiltersLow, Band and High-Pass Filters

A summary of main uses:

Low-pass: denoising

High-pass: removal of non-informative low frequency components

Band-pass: combination of low-pass and high-pass filtering effects

Gaussian High-Pass FilterGaussian High-Pass Filter

A high pass filter can be simply constructed from the Gaussian LPF:

Convolution with the delta function leaves the function unchanged

High-pass filter Low-pass filter

Gaussian HPF – Toy ExampleGaussian HPF – Toy Example

Consider the effects of high pass filtering our 1D toy example:

Original signal High-pass filter output

The result is not dependent on the signal mean

Maximal responses around discontinuities

Gaussian HPF – ExampleGaussian HPF – Example

Consider the effects of high pass filtering an image:

Original image High-pass filtered image

Information rich intensity discontinuities are extracted.

High Frequency Image ContentHigh Frequency Image Content

An example of the importance of high-frequency content:

?

+Low-pass filter High-pass filter

High Frequency Image ContentHigh Frequency Image Content

And the result of the experiment is…

HPFs in Face Recognition HPFs in Face Recognition

High-pass filters are used in face recognition to achieve quasi-illumination invariance:

Original image of a localized face

High-pass filtered

Filter Design – Matched FiltersFilter Design – Matched Filters

Consider the convolution sum of a discrete signal with a particular filter:

When is the filter response maximal?

… 234 233 228 240 241 241 …

1 2 2 1 0 01 2 2 1 0 0

228+ 480+482+ 241 + …

Filter Design – Matched FiltersFilter Design – Matched Filters

The summation is the same as for vector dot product:

The response is thus maximal when the two vectors are parallel i.e. when the filter matches the local patch it overlaps.

… 234 233 228 240 241 241 …

1 2 2 1 0 0

Filter Design – Intensity DiscontinuitiesFilter Design – Intensity Discontinuities

Using the observation that maximal filter response is exhibited when the filter matches the overlapping signal, we can start designing more complex filters:

Kernel with maximal response to intensity edges

0.5 0.0 -0.5

Filter Design – Intensity DiscontinuitiesFilter Design – Intensity Discontinuities

Better yet, perform Gaussian smoothing to suppress noise first:

Noise suppressing kernel with high response to intensity edges

Gaussian kernel

Unsharp Masking EnhancementUnsharp Masking Enhancement

The main principle of unsharp masking is to extract high frequency information and add it onto the original image to enhance edges:

image

HPF

+ output

Original edge Enhanced

Unsharp Masking EnhancementUnsharp Masking Enhancement

Unsharp mask filtering performs noise reduction and edge enhancement in one go, by combining a Gaussian LPF with a Laplacian of Gaussian kernel:

Gaussian smoothing Convolution with –ve Laplacian of Gaussian

+ =

Result

Unsharp Masking – ExampleUnsharp Masking – Example

Consider the following synthetic example:

Gaussian smoothed then corrupted with Gaussian noise

Unsharp Masking – ExampleUnsharp Masking – Example

After unsharp masking:

Gaussian smoothed then corrupted with Gaussian noise

– – That is All for Today –That is All for Today –