Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓...

52
1

Transcript of Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓...

Page 1: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

1

Page 2: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

2© 2015 The MathWorks, Inc.

Image Processing and

Computer Vision with MATLAB

Justin Pinkney

Page 3: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

3

Image Processing, Computer Vision

Image Processing Computer Vision Deep Learning

, and Deep Learning

Page 4: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

4

Deep Learning vs Image Processing

The perception:

“Deep learning has made ‘traditional’ image processing obsolete.”

or

“Deep learning needs millions of examples and is only good for classifying

pictures of cats anyway.”

Page 5: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

5

Deep Learning and Image Processing

The reality:

Deep learning and image processing are effective tools to solve different

problems.

and

Computer Vision tasks are complex, use the right tool for the job.

Page 6: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

6

What are real world problems like?

Page 7: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

7

An Example

Page 8: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

8

Page 9: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

9

Take a picture ⭢ Solve the puzzle

Page 10: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

10

Breaking down the problem

Find the puzzle Find the boxes

4 1 ⋯ ? ?7 ? ⋯ ? ?⋮ ⋮ ⋯ ⋮ ⋮? ? ⋯ 7 ?? 7 ⋯ ? ?

Read the numbers

4 1 ⋯ 6 87 8 ⋯ 3 2⋮ ⋮ ⋯ ⋮ ⋮4 2 ⋯ 7 56 7 ⋯ 4 1

Solve

Deep Learning Image Processing Deep Learning Optimisation

Page 11: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

11

Step 1 – Find the puzzle

Find the puzzle Find the boxes

4 1 ⋯ ? ?7 ? ⋯ ? ?⋮ ⋮ ⋯ ⋮ ⋮? ? ⋯ 7 ?? 7 ⋯ ? ?

Read the numbers

4 1 ⋯ 6 87 8 ⋯ 3 2⋮ ⋮ ⋯ ⋮ ⋮4 2 ⋯ 7 56 7 ⋯ 4 1

Solve

Page 12: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

12

Step 1 – Find the puzzle

▪ Varied background and object of

interest

▪ Uncontrolled image acquisition

▪ Good problem for deep learning

Page 13: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

13

Finding things with deep learning

Object detection Semantic segmentation

Page 14: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

14

Page 15: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

15

Page 16: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

16

How do we train a semantic segmentation network?

▪ Manage input data and apply

pre-processing

▪ Use a well established

network architecture

configured for our problem

▪ Set up training options

▪ Train the network

%% Input data

inputSize = [512, 512, 3];

train = pixelLabelImageDatastore(imagesTrain, labelsTrain, ...

'OutputSize', inputSize(1:2));

test = pixelLabelImageDatastore(imagesTest, labelsTest, ...

'OutputSize', inputSize(1:2));

%% Setup the network

numClasses = 2;

baseNetwork = 'vgg16';

layers = segnetLayers(inputSize, numClasses, baseNetwork);

layers = sudoku.weightLossByFrequency(layers, train);

%% Set up the training options

opts = trainingOptions('sgdm', ...

'InitialLearnRate', 0.01, ...

'ValidationData', test, ...

'MaxEpochs', 10, ...

'Plots', 'training-progress');

%% Train

net = trainNetwork(train, layers, opts);

Page 17: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

17

Page 18: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

18

Page 19: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

19

Page 20: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

20

An Example – Sudoku Solver

Find the puzzle Find the boxes

4 1 ⋯ ? ?7 ? ⋯ ? ?⋮ ⋮ ⋯ ⋮ ⋮? ? ⋯ 7 ?? 7 ⋯ ? ?

Read the numbers

4 1 ⋯ 6 87 8 ⋯ 3 2⋮ ⋮ ⋯ ⋮ ⋮4 2 ⋯ 7 56 7 ⋯ 4 1

Solve

Deep Learning

Page 21: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

21

Step 2 – Find the boxes

Find the puzzle Find the boxes

4 1 ⋯ ? ?7 ? ⋯ ? ?⋮ ⋮ ⋯ ⋮ ⋮? ? ⋯ 7 ?? 7 ⋯ ? ?

Read the numbers

4 1 ⋯ 6 87 8 ⋯ 3 2⋮ ⋮ ⋯ ⋮ ⋮4 2 ⋯ 7 56 7 ⋯ 4 1

Solve

Page 22: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

22

Step 2 – Find the boxes

▪ Well defined problem:

– 4 intersecting straight lines

– Dark ink on light paper

– Grid of 9 x 9 equally sized boxes

▪ Need good precision to localise

individual boxes.

▪ Good image processing problem.

Page 23: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

23

Page 24: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

24

Step 2 – Find the boxes

Adaptive thresholding imbinarize

Morphological operations

imopen/imclose

Region property analysis

regionprops

Robust line detection

hough/houghpeaks

Geometric warping

fitgeotrans/imwarp

Image processing pipeline

Page 25: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

25

Page 26: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

26

An Example – Sudoku Solver

Find the puzzle Find the boxes

4 1 ⋯ ? ?7 ? ⋯ ? ?⋮ ⋮ ⋯ ⋮ ⋮? ? ⋯ 7 ?? 7 ⋯ ? ?

Read the numbers

4 1 ⋯ 6 87 8 ⋯ 3 2⋮ ⋮ ⋯ ⋮ ⋮4 2 ⋯ 7 56 7 ⋯ 4 1

Solve

Deep Learning Image Processing

Page 27: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

27

Step 3 – Read the numbers

Find the puzzle Find the boxes

4 1 ⋯ ? ?7 ? ⋯ ? ?⋮ ⋮ ⋯ ⋮ ⋮? ? ⋯ 7 ?? 7 ⋯ ? ?

Read the numbers

4 1 ⋯ 6 87 8 ⋯ 3 2⋮ ⋮ ⋯ ⋮ ⋮4 2 ⋯ 7 56 7 ⋯ 4 1

Solve

Page 28: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

28

Step 3 – Read the numbers

Page 29: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

29

How can we read numbers?

Optical Character

Recognition

(OCR)

3

?

?

?

?

Page 30: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

30

How can we read numbers?

3

5

8

7

-

Deep Learning

Page 31: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

31

We can make synthetic training data

Page 32: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

32

We can make (lots of) synthetic training data

Page 33: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

33

We can make (lots of) synthetic training data

Page 34: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

34

Page 35: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

35

Page 36: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

36

Debugging a network

97.8% accuracy

?3

5

8

7

-

Page 37: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

37

Debugging a network

activations()

3

5

8

7

-

Page 38: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

38

Debugging a network with t-SNE

Page 39: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

39

Page 40: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

40

An Example – Sudoku Solver

Find the puzzle Find the boxes

4 1 ⋯ ? ?7 ? ⋯ ? ?⋮ ⋮ ⋯ ⋮ ⋮? ? ⋯ 7 ?? 7 ⋯ ? ?

Read the numbers

4 1 ⋯ 6 87 8 ⋯ 3 2⋮ ⋮ ⋯ ⋮ ⋮4 2 ⋯ 7 56 7 ⋯ 4 1

Solve

Deep Learning Image Processing Deep Learning

Page 41: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

41

Step 4 – Solve

Find the puzzle Find the boxes

4 1 ⋯ ? ?7 ? ⋯ ? ?⋮ ⋮ ⋯ ⋮ ⋮? ? ⋯ 7 ?? 7 ⋯ ? ?

Read the numbers

4 1 ⋯ 6 87 8 ⋯ 3 2⋮ ⋮ ⋯ ⋮ ⋮4 2 ⋯ 7 56 7 ⋯ 4 1

Solve

Page 42: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

42

How can we solve a sudoku?

Page 43: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

43

Step 4 – Solve

Page 44: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

44

Page 45: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

45

Deep Learning

Page 46: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

46

Image Processing

Page 47: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

47

Deep Learning

Page 48: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

48

Optimisation

Page 49: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

49

Page 50: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

50

An Example – Sudoku Solver

Find the puzzle Find the boxes

4 1 ⋯ ? ?7 ? ⋯ ? ?⋮ ⋮ ⋯ ⋮ ⋮? ? ⋯ 7 ?? 7 ⋯ ? ?

Read the numbers

4 1 ⋯ 6 87 8 ⋯ 3 2⋮ ⋮ ⋯ ⋮ ⋮4 2 ⋯ 7 56 7 ⋯ 4 1

Solve

Deep Learning Image Processing Deep Learning Optimisation

Page 51: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

51

Image Processing and Computer Vision with MATLAB

▪ Both deep learning and image processing are great tools for computer

vision.

▪ Use the right combination of tools to get the job done.

▪ MATLAB makes it easy and efficient to do both image processing and deep

learning together.

▪ MATLAB helps you integrate a computer vision algorithm into the rest of

your workflow.

Page 52: Image Processing and Computer Vision with MATLAB ↓ Robust line detection hough/houghpeaks ↓ Geometric warping fitgeotrans/imwarp Image processing pipeline ...

52

Try out the example code

(Search for: Deep Sudoku Solver)

Try what’s new in Image Processing

and Computer Vision toolboxes

Talk to us about your (computer

vision) problems