Image archive and leaf classifier SPECIFIC ENABLERS

17
IMAGE ARCHIVE AND LEAF CLASSIFIER SPECIFIC ENABLERS Stuart E. Middleton, Banafshe Arbab-Zavar, Stefano Modafferi, Ken Meacham and Zoheir Sabeur University of Southampton IT Innovation Centre ENVIROFI specific enabler 17 th January 2013 “ENVIROfying” the Future Internet

description

“ENVIROfying” the Future Internet. Image archive and leaf classifier SPECIFIC ENABLERS. Stuart E. Middleton, Banafshe Arbab-Zavar , Stefano Modafferi , Ken Meacham and Zoheir Sabeur University of Southampton IT Innovation Centre ENVIROFI specific enabler 17 th January 2013. - PowerPoint PPT Presentation

Transcript of Image archive and leaf classifier SPECIFIC ENABLERS

Page 1: Image  archive and leaf classifier  SPECIFIC ENABLERS

IMAGE ARCHIVE AND LEAF CLASSIFIER SPECIFIC ENABLERSStuart E. Middleton, Banafshe Arbab-Zavar, Stefano Modafferi, Ken Meacham and Zoheir SabeurUniversity of Southampton IT Innovation CentreENVIROFI specific enabler17th January 2013

“ENVIROfying” the Future Internet

Page 2: Image  archive and leaf classifier  SPECIFIC ENABLERS

• WP1 pilot use case• Image archive

• Architecture• User interface

• Leaf classifier• Architecture• Algorithms• User interface

OverviewImage archive and leaf classifier specific enablers

2

Page 3: Image  archive and leaf classifier  SPECIFIC ENABLERS

• WP1 pilot: Citizens in Tuscany• Data sources

• Proof of concept• CROWD SOURCING FROM SIR HAROLD HILLIER GARDENS, UK• HTTP://WWW3.HANTS.GOV.UK/HILLIERGARDENS

• User trial• CROWD SOURCING VIA WP1 PILOT IN THE TUSCANY REGION

• Image archive to record crowd-sourced leaf images• Web portal & backend service (Italian & English)• Integrated mobile phone platform• Support for general public and botanical experts

• Leaf image + auxiliary images + geo-tag + metadata

WP1 pilot use caseImage archive and leaf classifier specific enablers

3

Page 4: Image  archive and leaf classifier  SPECIFIC ENABLERS

• Leaf classifier to label unknown images• Web portal & backend service (Italian & English)• Integrated mobile phone platform

• Biodiversity ontology support• Scientific names (Latin)• Common names (Italian, English)• Domain ontology URI’s (e.g. TaxMeOn)• Natura 2000 habitat codes

• Value proposition• Supporting crowd sourced leaf observations allows image data

collection by volunteers at a scale beyond traditional methods

WP1 pilot use caseImage archive and leaf classifier specific enablers

4

Page 5: Image  archive and leaf classifier  SPECIFIC ENABLERS

Image archive architectureImage archive and leaf classifier specific enablers

5

Image, Geotag, User Metadata

Web browser

Users (crowd sourcing)

User

Mobile Data Acquisition Framework (MDAF)

Mobile observation server

HTTP RESTfulImage archive service

OWLIM (metadata)mySQL (data)

Image archive service

Domain experts

Expert

Mobile device

Image archive UI

Image(s), Geotag(s),User metadata

Database syncronization

Expert metadata

Image recordsImage

records

Crowd sourcing(web upload and mobile support)

Expert review of labels

Page 6: Image  archive and leaf classifier  SPECIFIC ENABLERS

Image archive user interfaceImage archive and leaf classifier specific enablers

6

Page 7: Image  archive and leaf classifier  SPECIFIC ENABLERS

Leaf classifier architectureImage archive and leaf classifier specific enablers

7

Unlabelled image

Web browser

Users (general public)

User

Mobile Data Acquisition Framework (MDAF)

Mobile observation server

HTTP RESTful SPSLeaf classifier process

OWLIM (metadata)mySQL (data)

Image classifier service

Mobile device

Leaf classifier UI

Unlabelledimage(s)

SPS request- image URI's

Classification label set(s)

Training setsignatures

SPS request- image URI's

Classificationlabel set(s)

Expert

Expert reviewed training set

Training set

Classification label set(s)

Label set

Labelset(s)

Users request classifications(unlabelled images)

Top N matches returned(leaf classifier algorithm)

Page 8: Image  archive and leaf classifier  SPECIFIC ENABLERS

• Classic benchmark datasets• e.g. Swedish leaf: 1,125 images, 15 species

• NO SHADOWS• LIMITED ROTATION

• Crowd-sourced datasets challenging!• e.g. Hillier Gardens (IT Innovation): 1400 images, 54 species

• SHADOWS• NATURAL OUTDOOR LIGHTING• ARBITRARY ROTATION

Leaf classifier algorithmsImage archive and leaf classifier specific enablers

8

Page 9: Image  archive and leaf classifier  SPECIFIC ENABLERS

• Segmentation - Colour-based Expectation-Maximization• HSV colour space; discard hue due to the high level of noise• Colour-based EM algorithm for pixel classification using k-means

clustering to initialize the EM algorithm (Belhumeur 2008)• Three clusters are considered representing: leaf; shadow and

background.

Leaf classifier algorithmsImage archive and leaf classifier specific enablers

9

P. Belhumeur, et al."Searching the World’s Herbaria: A System for Visual

Identification of Plant Species." ECCV. 2008. 116-129.

Page 10: Image  archive and leaf classifier  SPECIFIC ENABLERS

Leaf classifier algorithmsImage archive and leaf classifier specific enablers

10

Belhumeur 2008 tried segmentation with two clusters- problems handling shadows

LeafShadow

Background

We use three clusters forleaf, shadow, background

- shadows eliminated

• Segmentation - Colour-based Expectation-Maximization

Page 11: Image  archive and leaf classifier  SPECIFIC ENABLERS

Leaf classifier algorithmsImage archive and leaf classifier specific enablers

11

← The 3 clusters are re-classified based on cluster’s properties. Here, both leaf and shadow clusters were subsequently classified as leaf.

• Segmentation - Examples

Page 12: Image  archive and leaf classifier  SPECIFIC ENABLERS

• Feature extraction - Inner Distance Shape Context (Ling, 2007)

• Matching - fusion of two matching methods based on confidence levels:• Point-based IDSC matching• Contour matching

Leaf classifier algorithmsImage archive and leaf classifier specific enablers

12

Inner-distance connections between sampled points Inner-distance shape context Point correspondence between two images of

the same class

H. Ling, D. W. Jacobs. Shape Classification Using the Inner-Distance. 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, pp. 286 - 299.

Page 13: Image  archive and leaf classifier  SPECIFIC ENABLERS

• Distinctive classes

Leaf classifier algorithmsImage archive and leaf classifier specific enablers

13

Vitex Agnus-Castus

P(Best match) = 100%Confidence = 100%

Quercus Polycarpa

P(Best match) = 100%Confidence = 99.82%

Alnus Glutinosa 'Pyramidalis‘P(Best match) = 100%Confidence = 99.66%

Platanus ’Pyramidalis’

P(Best match) = 100%Confidence = 97.60%

Acer MonspessulanumP(Best match) = 100%Confidence = 97.5%

Tilia Tomentosa'Petiolaris'P(Best match) = 100%Confidence = 81.85%

Populus Nigra

P(Best match)=93.33%Confidence = 76.67%

Rhamnus Alpina

P(Best match)=92.86%Confidence = 82.28%

Cornus Sanguinea

P(Best match)=90.32%Confidence = 74.91%

Fagus Sylvatica 'Grandidentata'P(Best match)=90.00%Confidence = 77.78%

Ulmus

P(Best match)=90.00%Confidence = 66.48%

Page 14: Image  archive and leaf classifier  SPECIFIC ENABLERS

• Erroneous results can be caused by:• Similarity between the leaf shape of different species• Error in segmentation• Insufficient number of training samples

Leaf classifier algorithmsImage archive and leaf classifier specific enablers

14

Species name P(Best match)%

Confidence%

Error in classification caused by:

Shape similarity Error in segmentation

Insufficient training samples

Carpinus Betulus 84.09 67.68 x - - Acer Saccharum subsp. Leucoderme

83.33 81.48 x - -

Sorbus Degenii 80.65 61.65 x - - Ostrya Carpinifolia 78.57 44.05 x - - Crataegus_Crus-Galli 75.86 63.22 x x - Magnolia x Soulangeana

74.19 49.64 x x -

Acer Platanoides 'Globosum'

66.67 52.96 x - -

Quercus Robur 64.29 51.19 - x x Pyrus x Michauxii 63.64 51.01 x x x Magnolia x Loebneri 63.33 57.04 x x - Fraxinus 14.29 8.73 - x x

Examples ofsimilar shapes

Acer Platanoides 'Globosum'

Acer Saccharum subsp Leucoderme

Platanus ’Pyramidalis’

Magnolia x Loebneri

Magnolia x Soulangeana

Carpinus Betulus

Ostrya Carpinifolia

Rhamnus Alpina

Ulmus

Page 15: Image  archive and leaf classifier  SPECIFIC ENABLERS

• Hillier Gardens dataset results• Current dataset: 1400 images, 54 species• Mean probability of correct first match: 85.18%• Mean confidence in correct classification: 73.88%

Leaf classifier algorithmsImage archive and leaf classifier specific enablers

15

Page 16: Image  archive and leaf classifier  SPECIFIC ENABLERS

Leaf classifier user interfaceImage archive and leaf classifier specific enablers

16

Page 17: Image  archive and leaf classifier  SPECIFIC ENABLERS

Thank you for your attentionStuart E. Middleton

{sem}@it-innovation.soton.ac.ukwww.ENVIROFI.eu

twitter.com/ENVIROFI

The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under Grant Agreement Number 284898

17