Post on 21-Dec-2015
UIST 2004 Kim, Seitz, Agrawala
Video-Based Document Tracking:
Unifying Your Physical and Electronic Desktops
Jiwon Kim Steven M. Seitz Maneesh Agrawala
University of Washington Microsoft Research
UIST 2004 Kim, Seitz, Agrawala
Unifying physical andelectronic desktops
Record video of paper on physical desktop
Video camera
Desktop
UIST 2004 Kim, Seitz, Agrawala
Unifying physical andelectronic desktops
Record video of paper on physical desktop– Tracking
Video camera
Desktop
UIST 2004 Kim, Seitz, Agrawala
Unifying physical andelectronic desktops
Record video of paper on physical desktop– Tracking– Recognition
Video camera
Desktop
UIST 2004 Kim, Seitz, Agrawala
Unifying physical andelectronic desktops
Record video of paper on physical desktop– Tracking– Recognition– Linking
Video camera
Desktop
UIST 2004 Kim, Seitz, Agrawala
Applications
• Find lost document• Browse remote desk
Video camera
Desktop
UIST 2004 Kim, Seitz, Agrawala
Applications
• Find lost document• Browse remote desk• Find electronic
version
Video camera
Desktop
UIST 2004 Kim, Seitz, Agrawala
Applications
• Find lost document• Browse remote desk• Find electronic
version• History-based
queries
Video camera
Desktop
UIST 2004 Kim, Seitz, Agrawala
Related Work
• Interactive desktops
Self-Organizing Desk[Rus et al. 97]DigitalDesk [Wellner 93]
UIST 2004 Kim, Seitz, Agrawala
Related Work
• Interactive desktops• Augmented paper
PADD [Guimbretière 03]
UIST 2004 Kim, Seitz, Agrawala
Related Work
• Interactive desktops• Augmented paper
PADD [Guimbretière 03] CyberCode[Rekimoto et al. 00]
UIST 2004 Kim, Seitz, Agrawala
Related Work
• Interactive desktops• Alternative media• Object tracking & recognition
SIFT [Lowe 04]
UIST 2004 Kim, Seitz, Agrawala
System Overview
Video of desk Images from PDF
Track & recognize
T T+1
Desk Desk
Internal representation
UIST 2004 Kim, Seitz, Agrawala
System Overview
Video of desk Images from PDF
Track & recognize
T T+1
Desk Desk
Internal representation
Scene Graph
UIST 2004 Kim, Seitz, Agrawala
System Overview
Video of desk Images from PDF
Track & recognize
T T+1
Desk Desk
Internal representation
Where is my W-2?
UIST 2004 Kim, Seitz, Agrawala
System Overview
Video of desk Images from PDF
Track & recognize
T T+1
Desk Desk
Internal representation
Where is my W-2?
Answer
UIST 2004 Kim, Seitz, Agrawala
System Overview
Video of desk Images from PDF
Track & recognize
T T+1
Desk Desk
Internal representation
Where is my W-2?
UIST 2004 Kim, Seitz, Agrawala
Tracking & Recognition
… …
… …
Desk Desk
Event
tut-article.pdf
sanders01.pdf
objectspaces.pdf kidd94.pdf
lowe04sift.pdf
UIST 2004 Kim, Seitz, Agrawala
Assumptions
• Document– Corresponding electronic copy exists– No duplicates of same document
UIST 2004 Kim, Seitz, Agrawala
Assumptions
• Document– Corresponding electronic copy exists– No duplicates of same document
• Motion– 3 event types: move/entry/exit– One document at a time– Only topmost document can move
UIST 2004 Kim, Seitz, Agrawala
Non-Assumptions
• Desk need not be initially empty• Stacks may overlap
UIST 2004 Kim, Seitz, Agrawala
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
“A document moved from (x1,y1) to (x2,y2)”
before after
UIST 2004 Kim, Seitz, Agrawala
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
“A document moved from (x1,y1) to (x2,y2)”
Document Recognition
before after
File1.pdf
File2.pdf
File3.pdf
UIST 2004 Kim, Seitz, Agrawala
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
“A document moved from (x1,y1) to (x2,y2)”
Document Recognition
before after
File1.pdf
File2.pdf
File3.pdf
Scene Graph Update
Desk Desk
UIST 2004 Kim, Seitz, Agrawala
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
“A document moved from (x1,y1) to (x2,y2)”
Document Recognition
before after
File1.pdf
File2.pdf
File3.pdf
Scene Graph Update
Desk Desk
UIST 2004 Kim, Seitz, Agrawala
Event Detection
time
Threshold
Event Frames
time
… …
Motion Frames
Image motion
UIST 2004 Kim, Seitz, Agrawala
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
“A document moved from (x1,y1) to (x2,y2)”
Document Recognition
before after
File1.pdf
File2.pdf
File3.pdf
Scene Graph Update
Desk Desk
UIST 2004 Kim, Seitz, Agrawala
Event Interpretation
Move
Entry
Exit
1. Move vs. Entry/Exit
before after
UIST 2004 Kim, Seitz, Agrawala
Event Interpretation
Move
Entry
Exit
1. Move vs. Entry/Exit
before after
UIST 2004 Kim, Seitz, Agrawala
Event Interpretation
• Use SIFT [Lowe 04]
– Rotation- and scale-invariant– Highly distinctive (128-bit vector)
UIST 2004 Kim, Seitz, Agrawala
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
“A document moved from (x1,y1) to (x2,y2)”
Document Recognition
before after
File1.pdf
File2.pdf
File3.pdf
Scene Graph Update
Desk Desk
UIST 2004 Kim, Seitz, Agrawala
Document Recognition
…
File1.pdf File2.pdf File3.pdf File4.pdf File5.pdf File6.pdf
• Match against PDF image database
…
UIST 2004 Kim, Seitz, Agrawala
Document Recognition• Performance analysis
– Tested 20 pages against database of 162 pages
UIST 2004 Kim, Seitz, Agrawala
Document Recognition• Performance analysis
– Tested 20 pages against database of 162 pages
– ~200x300 pixels per document for reliable match
Document Resolution
Recognition Rate
UIST 2004 Kim, Seitz, Agrawala
Document Recognition• Performance analysis
– Tested 20 pages against database of 162 pages
– ~200x300 pixels per document for reliable match
Document Resolution
Recognition Rate
300
0.9
UIST 2004 Kim, Seitz, Agrawala
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
“A document moved from (x1,y1) to (x2,y2)”
Document Recognition
before after
File1.pdf
File2.pdf
File3.pdf
Scene Graph Update
Desk Desk
UIST 2004 Kim, Seitz, Agrawala
Results
• Input video– ~40 minutes– 1024x768 @ 15 fps– 22 documents, 49 events
• Running time– Video processed offline– No optimization– A few hours for entire video
UIST 2004 Kim, Seitz, Agrawala
Future Work
• Enhance realism– Handle more realistic desktops– Real-time performance
UIST 2004 Kim, Seitz, Agrawala
Future Work
• More applications– Support other document tasks
• E.g., attach reminder, cluster documents
UIST 2004 Kim, Seitz, Agrawala
Future Work
• More applications– Support other document tasks
• E.g., attach reminder, cluster documents
– Beyond documents
UIST 2004 Kim, Seitz, Agrawala
Future Work
• More applications– Support other document tasks
• E.g., attach reminder, cluster documents
– Beyond documents
UIST 2004 Kim, Seitz, Agrawala
Future Work
• More applications– Support other document tasks
• E.g., attach reminder, cluster documents
– Beyond documents
UIST 2004 Kim, Seitz, Agrawala
Future Work
• More applications– Support other document tasks
• E.g., attach reminder, cluster documents
– Beyond documents