ELEG404/604: Digital Imaging &Photography

Gonzalo R. ArceDepartment of Electrical and Computer Engineering

University of Delaware

Chapter II

1/90

ELEG404/604

Image Enhancement in the Spatial DomainI Algorithms for improving the visual appearance of imagesI Gamma correctionI Contrast improvementsI Histogram equalizationI Noise reductionI Image sharpening

I Optimality is in the eye of the observer: Ad hoc.

2/90

ELEG404/604

What is a histogramEach individual histogram entry is defined as

h(i) = Number of pixels in l with the intensity value ifor all 0≤ i < K, that is

h(i) = card{(u,v) |I (u,v) = i}h(0) is the number of pixels with the value 0, h(1) the number of pixels withthe value 1, and so forth.

3/90

ELEG404/604

HistogramsI Used to conclude if an image is properly exposed.I Digital cameras often provide a real-time histogram overlay on the

viewfinder.I Used to improve the visual appearance of an image.I Used as a ’‘forensic‘’ tool for determining what type of processing has

been applied to an image.

4/90

ELEG404/604

Histograms

Three very different images with identical histograms.

5/90

ELEG404/604

Image Acquisition- ExposureI Histograms make classic exposure problems apparent.I Large span of the intensity range at one end is largely unused other end is

crowded with high-value peaks represents poorly exposed image.

(a) Underexposed, (b) properly exposed, (c) overexposed

6/90

ELEG404/604

Image Acquisition- ContrastI Range of intensity values effectively used.I Difference between the image’s maximum and minimum pixel values.I A full−contrast image makes effective use of the entire range of available

intensity values from a= amin · · ·amax = 0 · · ·K−1 (black to white).

(a) Low contrast, (b) Normal contrast, (c) High contrast

7/90

ELEG404/604

Image Acquisition- Dynamic RangeI Number of distinct pixel values in an image.I Ideal case: Image uses all K usable pixel values.

(a) High dynamic range, (b) low dynamic range with 64 intensity values, (c)extremely low dynamic range, 6 intensity values

8/90

ELEG404/604

CompressionEffect of JPEG compression. The original image (a) with only two differentgray values, (b) histogram. JPEG compression, results in additional grayvalues histogram (d).

9/90

ELEG404/604

Color Image HistogramsHistograms of an RGB color image: (a) original image, (b) luminancehistogram hLum,(c−e) RGB color components, and associated componenthistograms hR,hG,hB (f−h). All three color channels have saturationproblems. The spike in the distribution resulting from this is found in themiddle of the luminance histogram (b).

10/90

ELEG404/604

Spatial Domain Processing

I Utilize neighborhood operations

g(x,y) = T [f(x,y)]

I Simple case: point operations

s= T (r)

I Contrast stretchingI Thresholding

11/90

ELEG404/604

Gray Level Transformation Curves

I More general transformationsI LogI Inverse LogI nth powerI nth root

I Used to map narrow dark (log/root) orbright (inverse log/power) range to agreater dynamic range

12/90

ELEG404/604

Fourier Spectrum ExampleThe spectrum has a large dynamic rangeI Example: 0 to 106.I Compress range to view visually (dB’s)

Left: Fourier spectrum. Right: Result of applying the log transformation.

13/90

ELEG404/604

Power-Law (Gamma) Transformations

I Power-law transformation:

s= crγ

I Many devices require gamma correctionI CRTs have power function intensity-to-voltage

responsesI Monitor specificI Applied in color planes

14/90

ELEG404/604

Gamma Correction Example Monitor

Top Left: linear-wedge gray-scale image. Top right: response of monitor tolinear wedge. Bottom left: Gamma-corrected wedge. Bottom right: Output ofmonitor

15/90

ELEG404/604

Gamma Correction Example - Aerial

γ=3

γ=8γ=4

Top Left: aerial image. Top right, Bottom left and bottom right: results of applying Gamma correction with c = 1 and γ = 3,4,5 respectively.

16/90

ELEG404/604

Histogram Processing

I Light, dark, and low contrast images haveconcentrated histograms

I Images with uniform histogramsI Contain the full range of gray valuesI Have high contrastI Better general visual appearance

17/90

ELEG404/604

Histogram EqualizationI We focus on the (normalized) scalar mapping

s= T (r) 0≤ r ≤ 1

where the following are satisfied:I T (r) is single-valued and monotonically increasing in [0,1]I 0≤ T (r)≤ 1 for 0≤ r ≤ 1

I The single-valued condition allows the inverse transformation to be defined

r = T−1(s) 0≤ s≤ 1

18/90

ELEG404/604

Probability Density Function

I Let the PDF of r be the pr(r)I The CDF is Pr(r) = ∫ r

0 pr(w)dwI Note CDFs are monotonically increasing and have range [0,1]

I Defined the RV s= T (r)I The PDF of a RV function is

ps(s) = pr(r)∣∣∣∣drds

∣∣∣∣

19/90

ELEG404/604

CDF Distribution

I Set T (r) = Pr(r)

ds

dr= dT (r)

drd[∫ r0 pr(w)dw]

dr= pr(r)

I Thus the PDF of s is

ps(s) = pr(r)∣∣∣∣drds

∣∣∣∣pr(r)

∣∣∣∣ 1pr(r)

∣∣∣∣ = 1

I The CDF is uniformly distributed.

20/90

ELEG404/604

Histogram Equalization

21/90

ELEG404/604

Histogram EqualizationA 3-bit image (L= 8) of size 64×64(MN = 4096) has the intensitydistribution in table.

Intensity levels are integers in [0,L−1] = [0,7].

22/90

ELEG404/604

Histogram Equalization

Values of the histogram equalization are obtained using:

s0 = T (r0) = 70∑j=0

Pr(rj) = 7pr(r0) = 1.33

Similarly,

s1 = T (r1) = 71∑j=0

Pr(rj) = 7pr(r0) + 7pr(r1) = 3.08

and s2 = 4.55, s3 = 5.67, s4 = 6.23, s5 = 6.65, s6 = 6.86, s7 = 7.00

23/90

ELEG404/604

The s values still have fractions because they were generated by summingprobability values, so we round them to the nearest integer:

s0 = 1.33→ 1s1 = 3.08→ 3s2 = 4.55→ 5s3 = 5.67→ 6

s4 = 6.23→ 6s5 = 6.65→ 7s6 = 6.86→ 7s7 = 7.00→ 7

24/90

ELEG404/604

Histogram Equalization

I CDF mapping of gray valuesI Yields uniformed histogramI Simple, parameter-free

I Discrete caseI Results not strictly uniformI Implementation issues

25/90

ELEG404/604

Histogram EQ Mappings

26/90

ELEG404/604

Histogram Equalization Result

Image

Histograms

Cumulative

Histograms

Original image and corresponding histograms. Notice that new peaks arecreated in the resulting histogram h (d) by merging original histogram cells.

27/90

ELEG404/604

Filters

What is a filter?Main difference between filters and point operations is that filters generallyuse more than one pixel to compute each new pixel input.

No point operation can blur or sharpen an image.

28/90

ELEG404/604

Filters

Each new pixel value I ′(x,y) is computed as a function of the pixels in acorresponding region in the original image I.

29/90

ELEG404/604

Spatial Filtering

I Spatial filtering is based on amoving window or mask

I In the linear filtering case:

g(x,y) = Σas=−aΣb

t=−bw(s, t)f(x+s,y+t)

I More compactly

R = Σmi=1nwizi

Must take into account bordereffects

30/90

ELEG404/604

FilteringTo sharpen or filter an image that was taken out-of-focus and is blurred, eachpixel is replaced with a linear combination of its neighbors.

31/90

ELEG404/604

Filtering

32/90

ELEG404/604

Simple Smoothing Masks

I Simplest linear filter: spatial average: reduces noise, but introducesblurring

I Distance weight samplesI Centrally located samples are more importantI Reduces blurring (somewhat)I Example above: simple integer arithmetic

I Alternative approach: utilize spectral characteristics

33/90

ELEG404/604

34/90

ELEG404/604

Border geometry

Filter can be applied only at locations (x,y) where the filter matrix H of size(2K+ 1)× (2L+ 1) is fully contained in the image.

35/90

ELEG404/604

Linear FiltersExamples of linear filters. The “box” filter (a) and the Gauss filter (b) areboth smoothing filters with all-positive coefficients. The “Laplace” or“Mexican hat” filter (c) is a difference filter.

36/90

ELEG404/604

Spatial Filtering

37/90

ELEG404/604

Smoothing Example

I Square averaging filter resultsI Window sizes: 3, 5, 9, 15, and 35

38/90

ELEG404/604

NonLinear Filters

Any image structure is blurred by a linear filter. Image structures such as edges(top) or thin lines (bottom) are widened, and the local contrast is reduced.

39/90

ELEG404/604

Order-Statistic Filters

I Linear (weighted sum) filtersI Blur edges and detailsI Are susceptible to outliers

I Order-statistic filters (nonlinear)I Preserve edges and detailsI Are less susceptible to outliers

I Spatially ordered samples: z1, z2, · · · , zNI Rank ordered samples: z(1), z(2), · · · , z(N)

z(1) ≤ z(2) ≤ ·· · ≤ z(N)MED[z1, z2, · · · , zN ] = z((N+1)/2)

Selection-type filter

40/90

ELEG404/604

Filtering

Sometimes it is very useful to apply a non-linear filter such as the median.

41/90

ELEG404/604

Effects of a minimum filter on various local signal structuresOriginal signal (top) and result after filtering (bottom), where color barsindicate the extent of the filter. Step edge (a) and linear ramp (c) are shiftedto the right. Narrow pulse (b) is removed.

42/90

ELEG404/604

Minimum Filter

43/90

ELEG404/604

Maximum Filter

44/90

ELEG404/604

Minimum and Maximum filters applied to a grayscale image

The original image is corrupted with ’‘salt and pepper‘’ noise (a). The 3×3pixel minimum filter eliminates the bright dots and widens all dark imagestructures (b). The maximum filter shows the exact opposite effects (c).

45/90

ELEG404/604

Median Filter

Computation of a 3×3 median filter. The nine pixel values extracted from thewindow are sorted and the center value is the median

46/90

ELEG404/604

Effects of Median FilterEffects of a 3×3 median filter on two-dimensional structures. Isolated dotsare eliminated (a), as are thin lines (b). Step edge remains unchanged (c),corner is rounded off (d).

47/90

ELEG404/604

Median Filter

48/90

ELEG404/604

Linear smoothing filter vs median filter

(a) Original image corrupted with ’‘salt-and-pepper‘’ noise. (b) Linear 3×3pixel box filter (b) reduces noise but the entire image is blurred. (c) The

median filter eliminates the noise dots and keeps the structures largely intact.

49/90

ELEG404/604

Salt and Pepper Noise Example

Left: X-ray Image of circuit board corrupted by salt and pepper noise. Middle: Noisereduction with a 3x3 averaging mask. Right: Noise reduction with a 3x3 median filter.I Bad sensors and bit errors yield salt-and-pepper noiseI Heavy tailed noise distribution. Other examples: Laplacian, Cauchy, a-StableI Outliers (generally) located in the extremes of the ordered set. Do not affect median

50/90

ELEG404/604

Weighted Median Filters (WMF)

The median filter can be generalized to weighted order statistic filters.

Given x(n) = [x1, . . . ,xN ], and a set of weights w1, . . . ,wN , the WMF outputis given by

y(n) = MEDIAN [x1♦w1,x2♦w2, . . . ,xN♦wN ] (1)where xi♦wi denotes replication:

xi♦wi =wi times︷ ︸︸ ︷

xi,xi, . . . ,xi (2)and wi ∈ Z+

51/90

ELEG404/604

Weighted median example

Each pixel value is replicated multiple times, as specified by the weight matrixW. For example, the value 0 from the center pixel is inserted three times. Thepixel vector is sorted and the center value (2) is the weighted median.

52/90

ELEG404/604

Center Weighted Median Filter (CWMF)A special case is the center weighted median filter (CWMF)

y(n) = MEDIAN[x1,x2, . . . ,xc−1,xc♦wc,xc+1, . . . ,xN ] (3)where c= (N + 1)/2 =N1 + 1 is the index of the center sample.

For wc = 1, the CWMF reduces to a median filter.

For wc ≥N the CWMF is an identity filter.

By varying the parameter

more smoothing less1←−−−−−−−−→

wcN

53/90

ELEG404/604

Property:Let y(n) be the output of a CWMF, then

y(n) = MEDIAN [x1, . . . ,xc−1,xc♦wc,xc+1, . . . ,xN ]= MEDIAN [x(k),xc,x(N−k+1)]

where k = (N + 2−wc)/2 for 1≤ wc ≤N , and k = 1 for wc >N .Hence,

y(n) =

xc if x(k) ≤ xc ≤ x(N+1−k)x(k) if xc ≤ x(k)x(N+1+k) if xc ≥ x(N+1−k)

54/90

ELEG404/604

Original

55/90

ELEG404/604

Salt and pepper noise

56/90

ELEG404/604

5× 5 window - wc = 1

57/90

ELEG404/604

5× 5 window - wc = 3

58/90

ELEG404/604

5× 5 window - wc = 5

59/90

ELEG404/604

Methods for extending the image to facilitate filtering alongthe borders

Nonexisting pixels outside the original image are either set to some constant value (a), take on the value of the closest

border pixel (b), are mirrored at boundaries (c), or repeat periodically along the coordinate axes (d).

60/90

ELEG404/604

Sharpening Filters

I Objective: Highlight and enhance fine detailI Details may have been blurred in acquisition processI Method: utilize first- and second-order derivativeI Derivatives identify signal changes (details/features)I First-derivative requirements:I Zero in flat regionsI Nonzero along rampsI Second-derivative requirements:I Zero in flat regionsI Zero along ramps of constant slope

61/90

ELEG404/604

First derivative in one dimensionOriginal image (a), horizontal intensity profile f(x) along the center imageline (b), and first derivative f ′(x) (c).

df

dx(x)≈ f(x+ 1)−f(x−1)

2 = 0.5 · (f(x+ 1)−f(x−1)).

62/90

ELEG404/604

Second DerivativeI Utilize difference equations: First derivative:

∂f

∂x= f(x+ 1)−f(x)

Second derivative∂2f

∂x2 = f(x+ 1) +f(x−1)−2f(x)

63/90

ELEG404/604

Derivative Observations

I First-order derivatives generate thick edgesI Second-order derivativesI Have stronger response to detailsI Produce a double response at step changesI Order of response strength. Point, line, step.

I Second-order derivative is therefore preferred for enhancementI Use isotropic (rotation invariant) formulation

64/90

ELEG404/604

Partial Derivatives and the GradientDerivative of a multidimensional function taken along one of its coordinatespartial derivative;

∂I∂x

(x,y) and ∂I∂y

(x,y)

The gradient function is

∇I(x,y) =[∂I∂x(x,y)∂I∂y (x,y)

]

The magnitude of the gradient,

|∇I|(x,y) =√√√√(∂I

∂x(x,y))2 + (∂I

∂y(x,y))2

65/90

ELEG404/604

Derivative Filters

The gradient approximation of the horizontal derivatives is the coefficientmatrix:

HDx = [−0.5 0 0.5] = 0.5 · [−1 0 1]

where the coefficients −0.5 and +0.5 correspond to the image elementsI(x−1,y) and I(x+ 1,y). Vertical component of the gradient

HDy =

−0.50

0.5

= .5 ·

−101

66/90

ELEG404/604

Partial Derivatives of a Two Dimensional Function

(a) Synthetic image,(b) first derivatives in the horizontal direction ∂I/∂x. (c) vertical direction ∂I/∂y; (d)magnitude of the gradient |∇I| (d).

67/90

ELEG404/604

Edge Operators-Various ApproachesDiffer in the type of filter used for estimating the gradient components andhow they are combined.I Roberts Operator: Simplest and oldest edge finder. Employs 2 small filters

2×2 for estimating the directional gradient along the image diagonals:

HRx =

(0 1−1 0

)HRy =

(−1 0

0 1

)

68/90

ELEG404/604

Edge Operators-Various Approaches

I Prewitt and Sobel Operators: Classic methods that differ in the filtersthey use:I Prewitt: Computes an average gradient across 3 neighboring lines or columns

respectively

HPx =

−1 0 1−1 0 1−1 0 1

HPy =

−1 −1 −10 0 01 1 1

I Sobel: The smoothing part assigns higher weight to the current center line and

column, respectively:

HSx =

−1 0 1−2 0 2−1 0 1

HSy =

−1 −2 −10 0 01 2 1

69/90

ELEG404/604

Estimates for Local Gradient Components

I Prewitt:∇I(x,y)≈ 1

6 ·(

(I ∗HPx )(x,y)

(I ∗HPy )(x,y)

)I Sobel:

∇I(x,y)≈ 18 ·(

(I ∗HSx )(x,y)

(I ∗HSy )(x,y)

)

70/90

ELEG404/604

Edge Strength and OrientationDenote the scaled filter results as:

Dx(x,y) =Hx ∗ I and Dy(x,y) =Hy ∗ I

In both cases, the local edge strength E(x,y) is defined as the gradientmagnitude √

(Dx(x,y))2 + (Dy(x,y))2

an the local edge orientation angle φ(x,y) is

φ(x,y) = arctan(Dy(x,y)Dx(x,y))

An improved version of the Sobel filter:

HS′x = 1

32

−3 0 3−10 0 10−3 0 3

HS′y = 1

32

−3 −10 −30 0 03 10 3

71/90

ELEG404/604

Edge Operators

Process of gradient-based edge extraction. Linear filters Hx and Hy producetwo gradient images, Dx and Dy. They are used to compute the edgestrength E and orientation φ for each image position (x,y).

72/90

ELEG404/604

Laplacian Edge OperatorProblem with edge operators based on first derivatives; edges are as wide asthe underlying intensity transition. Alternative makes use of the secondderivatives.Laplacian is the simplest isotropic derivative:

∇2f = ∂2f

∂x2 + ∂2f

∂y2

I

∇2f ≈ [f(x+ 1,y) +f(x−1,y) +f(x,y+ 1) +f(x,y−1)]−4f(x,y)

I Isotropic to 90◦rotationsI Add diagonal derivatives to make it 45◦ isotropic

73/90

ELEG404/604

Mask Implementations and Enhancement

I Similar definition produces a signchange

I Enhancement adds (subtracts)derivative and observed image

g(x,y) ={f(x,y)−∇2f(x,y) if the center is negativef(x,y) +∇2f(x,y) if the center is positive

74/90

ELEG404/604

Example

Top Left: image of the north pole of the moon. Top Right: Laplacian-filtered image.Bottom Left: Laplacian image scaled for display purposes. Bottom Right: Image enhancend

75/90

ELEG404/604

Example

Top Left: Composite Laplacian mask. Top Right:scanning electron microscope imageBottom Left: filtered image with first Laplacian mask. Bottom Right: filtered image withsecond Laplacian mask

76/90

ELEG404/604

77/90

ELEG404/604

Unsharp MaskingUnsharp masking process:

fi(x,y) = f(x,y)−f(x,y)

I Originally a darkroom process combining a blurred negative and positivefilm

Generalization: high-boost filteringfhb(x,y) = Af(x,y)−f(x,y)fhb(x,y) = (A−1)f(x,y)−fs(x,y)

I Using the Laplacian to obtain the sharp image fs(x,y)

fhb ={Af(x,y)−∇2f(x,y) if the center is negativeAf(x,y) +∇2f(x,y) if the center is positive

78/90

ELEG404/604

79/90

ELEG404/604

Edge Sharpening with second derivativeOriginal intensity function f(x), first derivative f ′(x), second derivativef ′′(x), and sharpened intensity function f̆(x) = f(x)−w ·f ′′(x)

80/90

ELEG404/604

Results of Laplace filter HL

(a) Image I(a), (b) second partial derivative ∂2I/∂2u in the horizontaldirection, (c) second partial derivative ∂2I/∂2v in the vertical direction, and(d) Laplace filter ∇2I(u,v).

81/90

ELEG404/604

Edge Sharpening with the Laplace filterOriginal image and marked line (a, b), result of Laplace filter HL (c, d), andsharpened image (e, f).

82/90

ELEG404/604

Unsharp Mask

(a,b) Original image and detail,(c) intensity profile of line; USM filtering withGaussian smoothing σ = 2.5 (d - f) and 10.0 (g - i). Sharpening factor is 1.0(100%).

83/90

ELEG404/604

Mask Implementation

Standard Laplacian sharpening: A= 1.Increasing A, reduces the level of sharpeningI Scales (brightens) original image

84/90

ELEG404/604

High-Boost Example

Top Left: original image. Top Right: Laplacian of the original image withmask(left) in previous image (A= 0). Bottom Left: Laplacian of the originalimage with mask(right) in previous image (A= 1). Bottom Right: Laplacianof the original image with mask(right) in previous image (A= 1.7).

85/90

ELEG404/604

Edges and Contours

I Edges and contours, are of high importance for the visual perception andinterpretation of images.

I Edges: local intensity changes distinctly along a particular orientation.The amount of change with respect to spatial distance is the firstderivative of a function.

86/90

ELEG404/604

Edge Detection

87/90

ELEG404/604

Edge Detection

The derivative of a digital image canbe found by applying the linear filter.

88/90

ELEG404/604

Edge Detection

The edges are then thinned by applying a threshold.

89/90

ELEG404/604

Sobel Example

Left: Optical image of contact lens. Right: Sobel gradient. Commonapplication: edge detectionI Threshold sobel outputI Binary edge mask

90/90

ELEG404/604

Comparison of Edge Operators