A Contact Sheet Approach toSearching Untagged Images on Smartphones

Wolfgang Richter*, Kiryong Ha*, Alok Shankar*, Ardalan Amiri Saniꝉ, Jan Harkes*, Lin Zhongꝉ, Mahadev Satyanarayanan*

(*Carnegie Mellon University, ꝉRice University)

Problem

Low Fidelity Evaluation

How do we efficiently search mobile devices in near real-time?

Lessons and Contributions

Solution: Contact Sheets

Two key factors:

CloudOlder,Uploaded

Recent,not uploaded

0

50

100

150

200

250

300

350

400

Full 640x480 320x240 160x120 80x60 40x30 20x20

Bandw

idth

(K

B)

Fidelity

To save energy and bandwidth, we degrade objects being searched and pull full fidelityversions in a just-in-time search.

Diamond ModificationsMISTAndroid Application

C2DM

PollingCamera

UI

Local Storage

Mobile API

Cloud API

MISTDataretriever Diamond GUI

SessionServerdiamondd

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

Full Fidelity 640x480 320x240 160x120 80x60 40x30 20x20

Energ

y C

onsu

mptio

n (

mJ)

Fidelity

No BatchingBatching

• Released Android Market App• Python Flask Web Application • Mobile API via HTTP • Cloud API via HTTP• Full Fidelity Retrieval • Push-based Google C2DM • Polling-based HTTP• Just-in-Time Search

How far can we degrade images in a search before it loses meaning?

Dataset: • 12, 963 Flickr Scraped ImagesFidelity Reduction: • Resolution • JPEG Quality FactorBenchmarked Three Computer Vision Algorithms:

• No One-Size-Fits-All• Significant Energy Savings• Significant Bandwidth Savings

Lessons

Help Us:

Energy Bandwidth Potential Savings?

Contributions

0

0. 2

0. 4

0. 6

0. 8

1

640x480 320x240 160x120 80x60 40x30 20x20

Filt

er

Acc

ura

cy

Fidelity

0.991

0.899

10095755025

0

0. 2

0. 4

0. 6

0. 8

1

640x480 320x240 160x120 80x60 40x30 20x20

Filt

er

Fa

lse

Ne

ga

tive

Ra

te

Fidelity

0.040

10095755025

0

0. 2

0. 4

0. 6

0. 8

1

640x480 320x240 160x120 80x60 40x30

Filt

er

Acc

ura

cy

Fidelity

0.907

0.803

10095755025

0

0. 2

0. 4

0. 6

0. 8

1

640x480 320x240 160x120 80x60 40x30

Filt

er

Fa

lse

Ne

ga

tive

Ra

te

Fidelity

0.124

0.042

10095755025

0

0. 2

0. 4

0. 6

0. 8

1

640x480 320x240 160x120 80x60

Filt

er

Fa

lse

Ne

ga

tive

Ra

te

Fidelity

0.575

0.281

10095755025

0

0. 2

0. 4

0. 6

0. 8

1

640x480 320x240 160x120 80x60

Filt

er

Acc

ura

cy

Fidelity

0.650

0.424

10095755025

RGB Histogram Face DetectionBrightness