1UMC – HCI seminar series

Human Computer Interaction

Query by Sketch

Chi-Ren Shyu

Department of Computer Engineering and Computer ScienceUniversity of Missouri-Columbia

Columbia, MO 65211, U.S.A.

2UMC – HCI seminar series

Query by Sketch -- an HCI topic?

Human side: query behavior, retrieval result evaluation, training of semantics and database indexing.

Computer side: query processing from human’s sketches, feature extraction from sketch objects, database search mechanisms, etc.

3UMC – HCI seminar series

Some Applications and Systems

Photo (natural scene images) IBM QBIC http://wwwqbic.almaden.ibm.com/

GIS – Egenhofer at U. of Maine Spatial Query by sketch http://www.spatial.maine.edu/~max/RL.html

Medical Image – University of Missouri – ColmbiaWebHIQS http://diglib1.cecs.missouri.edu:3243/FinalProj/index1.html(experimental website)

4UMC – HCI seminar series

QBS in Diagnostic Image Databases

Diagnoses often involve findings related to spatial relationship among lesions and landmarks.

Pathologies could be detected based on many visual patterns that are “sketchable”, such as shapes of tumors, distribution of nodules, ect.

Query: “Retrieve database images that have similar lesion-landmark location with the sketched query (image)”

5UMC – HCI seminar series

Query methods in Diagnostic Image DBS

Sketch on a blank sheet – users are provided with a GUI for drawing organs, landmarks, and lesions.

Sketch on an organ template – users are provided with a template of an organ and a GUI for drawing landmarks and lesions.

Sketch on an existing image – users are provided with a real diagnostic image and a GUI for drawing landmarks and lesions.

6UMC – HCI seminar series

Research opportunity – Feature Extraction from Sketches

Shapes from sketch objects, texture from sketched regions, gray scale from sketched vicinity, etc. Domain expertise will help to design computer vision algorithms for this purpose.

Forming a feature vector for image retrieval. Searching a high-dimensional database for

retrieving the most similar sketches?

7UMC – HCI seminar series

Research opportunity – Spatial Modeling from Sketches

Anatomical landmarks in HRCT lung: Lung regions (Automatic Extraction) Fissures (Human in the loop) Lobe partitions (HIL/AE)

Lesions vs. landmarks: Interior Adjacent Across

8UMC – HCI seminar series

Research opportunity – Spatial Modeling from Sketches - 2

Why these three lesions should be retrieved?Query lesion Sketched by

the user

9UMC – HCI seminar series

Spatial Modeling from Sketches - 3

Degree of Truth for (inner-Adjacent, To-the-right)

(0.34, 0.00) (0.15, 0.00) (0.55, 0.31) (1.00, 0.82)

(1.00, 0.96) (0.83, 0.84) (0.21, 0.99) (0.00, 1.00)

What about multiple landmarks with multiple lesions?

10UMC – HCI seminar series

Research opportunity – Semantic Query from Sketches

Semantic or so-called ontology is very subjective and user-oriented. Not good for “general purposed” database retrieval. GREAT for Sketches!

The hope is: Sketches might be able to bring more desirable retrieval results from user’s aspects.

11UMC – HCI seminar series

Semantic Query from Sketches - 2

12UMC – HCI seminar series

Semantic Query from Sketches - 3

Bronchial Array

Number of bronchial structures

Average Size of Bronchial Structures

Coverage of Bronchial Structures in PBR

Grey difference between Wall and Lumen

Wall and Lumen Ratio

Many Few

Big Small

Sparse Dense

Blur Sharp

Thin Thick

Sketch bronchial structures on an organ template and fillgray scales to different part of the object, such as bronchial

Walls and lumens.

13UMC – HCI seminar series

Semantic Query from Sketches - 4

code=Keller

code=LungGMean code=Cystic

code=dark mlo=15 mhi=35 dlo=1 dhi=3

code=bright mlo=40 mhi=120

code=Structure

code=Big mlo=2 mhi=20

code=Small mlo=40 mhi=120

code=light mlo=31 mhi=52

Users sketch on many training imagesand form a semantic tree structure

for future retrieval.

14UMC – HCI seminar series

Semantic Query from Sketches - 5

)(

1

2

)(),(1

)(

1

2

)(

)(

)(

)(

)(

fj

hi

e

ad

amx

fj

hifj

lo

fj

lo

e

ad

xam

fj

amxfor

e

amamxfor

amxfor

e

xa

hi

fj

hi

fj

hi

lo

fj

lo

fj

lo

For fuzzy gangs, this looks familiar. We can form membership functions for the semantic

terms by using user’s Sketches.

0

0.5

1

1 6 11 16 21 26 31 36 41mlo=3, dlo=1, elo=2, mhi=9, dhi=6, ehi=2

mlo=25, dlo=0.5, elo=2, mhi=25, dhi=1, ehi=2

mlo=11, dlo=3, elo=2, mhi=15, dhi=12, ehi=2

mlo=21, dlo=5, elo=2, mhi=31, dhi=5, ehi=2

mlo=21, dlo=5, elo=1, mhi=31, dhi=5, ehi=3

15UMC – HCI seminar series

Semantic Query from Sketches - 6

Ranking database images by semantics

LNNG/CYSTIC/CYSTIC1 LNNG/HONEY/HONEY1 LNNG/TIB/TIB1

Perfect Solution

Good Solution

Partial Solution

16UMC – HCI seminar series

Semantic Query from Sketches - 6

17UMC – HCI seminar series

Research opportunity – Query Languages for Sketches

SELECT I.iidFROM LungImageDatabase IWHERE SIM( SKETCH(Qid,GUI), I.FE(iid)) <

thresholdORDER BY SIM;

A possible SQL extension for QBS:

18UMC – HCI seminar series

Summary

Query by sketch is not new, however, it has high potential for many applications, especially in the areas of diagnostic image database, GIS, criminal investigation, etc.

In the CECS department, we have many existing research projects that are highly correlated to the concept of QBS. We should have some sort of synergistic efforts to have some interesting projects in the near future.