INF 5860 Machine learning for image classification

23.1.18Background in image convolution and filteringAnne Solberg

Today

• Image filtering• 2D convolution• Edge detection filters

• Implementing convolution in python

• PURPOSE: give a short background to those without a background in image analysis

• Convolutional nets use convolution as the basic operation.

INF 5860 3

INF 5860 4

Properties of the human visual system

• We can see light intensities over a broad range– The largest is about 1010 times higher than the lowest we can sense.

• We can only see a certain number of levels simultaneously,– About 50 different gray levels, but many more colors.

• When we focus on a different area, the eye adapts and we see localintensity differences.

INF 5860 5

Neural processes in the retina

• Amplifies edges.

• Stimulating one part suspendsother parts.– See one orientation at a time

• Increased contrast in edgesbetween uniform regions– Called Mach band

INFINF 5860 6

Optical illusions• Illusional contours Straight or curved lines

• Multistable images Simultaneous contrast

Image filtering• One of the most used operations on images• A filter kernel is applied either in the image domain or 2D

Fourier domain.• Applications:

– Image enhancement– Image restoration– Image analysis – preprocessing

• Used to:– Reduce noise– Enhance sharpness– Detect edges and other structures– Detect objects– Extract texture information

INF 5860 7

Spatial filtering

• A filter is given as a matrix:

• The size of the matrix and the coefficients decides the result.

INF 5860 8

INF 2310 9

2-D convolution

• Output image: g. Input image f.

• w is a m×n filter of size m=2m2+1, n=2n2+1, m2=m//2, n2=n//2

• m and n usually odd. • We will use square filters m=n• Output image: weighted sum of input image pixels

surrounding pixel (x,y). Weights:w(j,k).• This operation is done for every pixel in the image

, , ,

Step 1: convolution: rotate the image 180 degrees

10

In this case, nothing changes

Rotation in Python

17.3.2017 INF 5860 11

Step 2: Iterate over all locations where thefilter overlaps the image

INF 5860 12

Multiply the image and the maskCompute the result for location (x,y)

Step 3

INF 5860 13

Multiply the filter and the image coefficients, and sum to get the resultfor ONE pixel in the output image

Repeat for next output pixel

INF 5860 14

Repeat for next output pixel

INF 5860 15

The solution is

INF 5860 16

• To get the result for one pixel in the output image, we compute

• Straightforward implementation: use two for-loops over j and k to compute g(x,y)

• To get the result for ALL pixels in the image with size (X,Y), we wouldneed two additional for-loops over all pixels:for x in range(n2: X-n2):

for y in range(m2:Y-n2):

The inner sum (formula above) can be more efficiently implemented usingnp.sum.For color images: add an outer for loop over the bands (c=1:3 for RGB images) INF 5860 17

, , ,

, , ,

IN this case we compute the result only where thefilter completely overlaps the image

18

Mean filters

• 3×3:

• 5×5:

• 7×7:

• Scale the result by thesum of the coefficientsto keep the range of theinput image

111111111

91

1111111111111111111111111

251

1111111111111111111111111111111111111111111111111

491

19

OriginalFiltered 3x3

20

9x925x25

21

Gradients

• Gradient of F along r in direction

• Largest gradient when

• This is the angle g where

• gx=F/x and gy =F/x are the horisontal and verticalcomponents of the gradient.

• Gradient points in the direction where the function has thelargest increase.

ry

yF

rx

xF

rF

gggg xF

yF

yF

xF sincos0cossin

0

rF

sincosyF

xF

rF

22

Gradient magnitude and direction

• Gradient direction:

• Gradient magnitude

gg

g

x

y

gg

tancossin

x

yg g

g1tan

2/122

maxyx gg

rF

23

Gradient-filters• Prewitt-operator

• Sobel-operator

• On this slide, the filters are denoted h, not w

111000111

),(101101101

),( jiHjiH yx

121000121

),(101202101

),( jiHjiH yx

These filter do gradient computation in one direction, and smoothing in the otherdirection

Computing the gradients

• Find the horisontal edges by convolving with hx:– Compute

• Find the verdical edges by convolving with hy:– Compute:

• Compute gradient magniture and direction:

INF 2310 24

Gradient example

INF 5860 25

INF2310 26

Size of the resulting imageAlt.1. Compute the result only where the filter fits inside the input

image. – Assume the image has X x Y pixels and that the filter has size m x n

(m,n odd numbers)– Output size:

(M-m+1)x(N-n+1)• 3x3-filter: (M-2)x(N-2)• 5x5-filter: (M-4)x(N-4)

Alt.2. Keep the size of the input image– Will normally be used for deep learning– Common for image filtering also– Special considerations at the border

• Zero-pad the input image• Or, change the filter size at

the boundary

Filtrering II: Correlation

• Difference from convolution: Pluss replaces minus.– No rotation!

• Normally applied in convolutional nets as the net itselfwill estimate the filter coefficients (and they could be estimated with the rotation included)

INF2310 27

a

as

b

bt

tysxftshyxg ),(),(),(

Like to learn more:

• See lecture notes from INF2310 (notes in English)– http://www.uio.no/studier/emner/matnat/ifi/INF2310/v17/undervisni

ngsmateriale/slides_inf2310_s17_week06.pdf– http://www.uio.no/studier/emner/matnat/ifi/INF2310/v17/undervisni

ngsmateriale/slides_inf2310_s17_week07.pdf

INF 5860 28

This weeks exercise

• Task: complete the notebooks: – math_operations.ipynb– indexing.ipynb– convolution.ipynb

• See lecture notes on how to start jupyter notebooks• https://jupyter-notebook-beginner-

guide.readthedocs.io/en/latest/

INF 5860 29

Useful links:

• http://cs231n.github.io/python-numpy-tutorial/

• http://cs231n.github.io/ipython-tutorial/

INF 5860 30