1

Leafsnap: classification

Mariia Dmitrieva

Mohamed Elawady

Paper

“Leafsnap: A Computer Vision System for Automatic

Plant Species Identification”

Neeraj Kumar, Peter N. Belhumeur, Arijit Biswas, David

W. Jacobs,W. John Kress, Ida C. Lopez, and Joao

V.B. Soares

European Conference on Computer Vision 2012

2

Outline

1.Introduction

2.Recognition Process

2.1. Classification

2.2. Segmentation

2.3. Feature Extraction

2.4. Comparison

4.Results

5.Future Directions

6.Demo

3

Outline

1.Introduction

2.Recognition Process

2.1. Classification

2.2. Segmentation

2.3. Feature Extraction

2.4. Comparison

4.Results

5.Future Directions

6.Demo

4

1. Why & Who…

5

Who

Columbia University

University of Maryland

Smithsonian Institution

Why

Book

Scout

6

1. Plant Species Identification

7

1. Framework

Outline

1.Introduction

2.Recognition Process

2.1. Classification

2.2. Segmentation

2.3. Feature Extraction

2.4. Comparison

4.Results

5.Future Directions

6.Demo

8

2. Recognition Process

4. Comparison

Compare the features to those from a labeled database of leaf image and returning the species with the closest matches

3. Feature Extraction

Select curvature features from the binarized image representing the shape of the leaf

2. Segmentation

Obtain a binary image separating the leaf from the background

1. Classification

Whether the input image is a valid leaf or not

9

2.1. Classification

Input ImagePre-

Processing Step

Compute GIST

features

Perform SVM

classifier

Leaf or Non-leaf

10

2.1. SVM Classifier

11

Which one is the best?

Support

Vectors

SVM

Line that maximizes

the minimum margin

among only support vectors

2.2. Segmentation

Color

• High variable across different leaves of the same spices

Venation Pattern

• Undetectable due to the poor image quality of most phone cameras

Flowers

• Only present at limited times of year

Leaf Shape

• Good at one condition: photograph them against light and non textured background

Initial Segmentation using EM

Removing False Positive Regions

Removing The Stem

12

2.2. SegmentationInitial Segmentation using EM I

RGB

Convert

to HSV

HSV

HSV

Hue

HueSaturation

SaturationValue

Value

SV

SV with H=0

Leaf

Background

13

2.2. SegmentationInitial Segmentation using EM II

Leaf

Background

EM

SV

14

2.2. SegmentationRemoving False Positive Regions

INPUTInitial

Segmentation

Result of

Current Step

Dilation + Elimination

Small Regions

15

2.2. SegmentationRemoving the stem

INPUTInitial

Segmentation

Result of

Current Step

Opening and

Difference

Operations

Remove

False Positive

Regions

16

2.3. Extraction

17

Extraction

Comparison

2.3. Extraction: Curvature

Complications Rotations

Scale Changes

Axis Alignment

Complex Boundaries

Segmentation Errors

18

coarse scale

fine scale

19

2.3. Extraction

Integral Measures

20

2.3. Extraction

Multiscale Curvature Measures

2.3. Extraction

Histogram of the Curvature

21

2.3. Extraction

Advantages of the HoCS

Fast

Invariant to rotation

Not requiring alignment

Insensitive to small segmentation and

discretization errors

Independent of the topological complexity

22

2.4. Comparison

Nearest neighbors search

Database:

23,915 lab images

5,129 mobile phone images

Broussonettia papyrifera

23

2.4. Comparison

Nearest neighbor search

Comparison by histogram intersection distance:

0.31 seconds

Top 25 results are presented

24

B

i

ii baNbad ),min(),(

Outline

1.Introduction

2.Recognition Process

2.1. Classification

2.2. Segmentation

2.3. Feature Extraction

2.4. Comparison

4.Results

5.Future Directions

6.Demo

25

26

1st match is right

69% of time

Within top 5 matches

93% of time

4. Results

5. Future directions

New objects

27

Education

Tracking

around the Earth

6. Demo Video

28

Video By “Into Mobile”

http://www.youtube.com/watch?

v=k02C7p7mQ_c

Bibliography

“Introduction to Support Vector Machines”

http://docs.opencv.org/doc/tutorials/ml/introduction_to_svm/introd

uction_to_svm.html

“A Computer Vision System for Automatic Plant Species

Identification”

http://homes.cs.washington.edu/~neeraj/projects/leafsnap/base/pr

esentations/2012_leafsnap/leafsnap-eccv2012.pptx

“Integral invariants for robust geometry processing” Pottmann, H.,

Wallner, J., Huang, Q.X., Yang, Y.L., Computer Aided Geometric

Design 26, 37–60 (2009)

29

30

Thanks For Listening !

Leafsnap: classification

31

QUESTIONS