Motivation - University of...
Transcript of Motivation - University of...
![Page 1: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/1.jpg)
Motivation
Where is my W-2 Form?
![Page 2: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/2.jpg)
Video-based Tracking
Camera view of the desk
Camera
Overhead video camera
![Page 3: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/3.jpg)
Example
40 minutes, 1024x768 @ 15 fps
![Page 4: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/4.jpg)
System Overview
UserInput Video
![Page 5: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/5.jpg)
System Overview
User
Internal Representation
Video Analysis
Vision Engine
Desk Desk
T T+1
Input Video
![Page 6: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/6.jpg)
Query Interface
System Overview
Query(where is my W-2 form?)
Internal Representation
Input Video User
Video Analysis
Desk Desk
T T+1
![Page 7: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/7.jpg)
System Overview
Query(where is my W-2 form?)
Internal Representation
Input Video User
Video Analysis
Answer
Query Interface
Desk Desk
T T+1
![Page 8: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/8.jpg)
System Overview
User
Internal Representation
Video Analysis
Input Video
Vision Engine
Desk Desk
T T+1
![Page 9: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/9.jpg)
Vision Problem
… …
![Page 10: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/10.jpg)
Vision ProblemEvent
… …
![Page 11: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/11.jpg)
Vision Problem
…
Event
… …
Desk
![Page 12: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/12.jpg)
Vision Problem
Desk
… …
Event
… …
Desk
![Page 13: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/13.jpg)
Vision Problem
… …tut-article.pdf
sanders01.pdf
objectspaces.pdf kidd94.pdf
lowe04sift.pdf
Event
… …
Desk Desk
![Page 14: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/14.jpg)
Vision Problem
… …
Event
Scene Graph(DAG)
… …
tut-article.pdf
sanders01.pdf
objectspaces.pdf kidd94.pdf
lowe04sift.pdf
Desk Desk
![Page 15: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/15.jpg)
Assumptions
• Simplifying– Corresponding electronic copy exists
![Page 16: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/16.jpg)
Assumptions
• Simplifying– Corresponding electronic copy exists– 3 event types: move/entry/exit– One document at a time– Only topmost document can move– No duplicate copies of same document
![Page 17: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/17.jpg)
Assumptions
• Simplifying– Corresponding electronic copy exists– 3 event types: move/entry/exit– One document at a time– Only topmost document can move– No duplicate copies of same document
• Other– Desk need not be initially empty
![Page 18: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/18.jpg)
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
Document Recognition
Scene Graph Update
![Page 19: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/19.jpg)
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
Document Recognition
before after
Scene Graph Update
![Page 20: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/20.jpg)
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
“A document moved from (x1,y1) to (x2,y2)”
Document Recognition
before after
Scene Graph Update
![Page 21: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/21.jpg)
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
“A document moved from (x1,y1) to (x2,y2)”
Document Recognition
before after
File1.pdf
File2.pdf
File3.pdf
Scene Graph Update
![Page 22: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/22.jpg)
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
“A document moved from (x1,y1) to (x2,y2)”
Document Recognition
before after
File1.pdf
File2.pdf
File3.pdf
Scene Graph Update
![Page 23: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/23.jpg)
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
“A document moved from (x1,y1) to (x2,y2)”
Document Recognition
before after
File1.pdf
File2.pdf
File3.pdf
Scene Graph Update
![Page 24: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/24.jpg)
Event Detection
… …
![Page 25: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/25.jpg)
Event Detection
time
Frame Difference
… …
![Page 26: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/26.jpg)
Event Detection
time
Threshold
Event Frames
time
… …
Frame Difference
![Page 27: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/27.jpg)
Event Detection
before after
Event Frames
… …
![Page 28: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/28.jpg)
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
“A document moved from (x1,y1) to (x2,y2)”
Document Recognition
before after
File1.pdf
File2.pdf
File3.pdf
Scene Graph Update
![Page 29: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/29.jpg)
Event Interpretation
Move
before after
![Page 30: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/30.jpg)
Event Interpretation
Move
Entry
before after
![Page 31: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/31.jpg)
Event Interpretation
Move
Entry
Exit
before after
![Page 32: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/32.jpg)
Event Interpretation
Move
Entry
Exit
Motion: (x,y,θ)
before after
![Page 33: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/33.jpg)
Event Interpretation
Move
Entry
Exit
1. Move vs. Entry/Exit
before after
![Page 34: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/34.jpg)
Event Interpretation
Move
Entry
Exit
2. Entry vs. Exit
before after
![Page 35: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/35.jpg)
Event Interpretation
• Use SIFT [Lowe 99]
– Scale Invariant Feature Transform– Distinctive feature descriptor– Reliable object recognition
![Page 36: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/36.jpg)
Move vs. Entry/Exit
before after
![Page 37: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/37.jpg)
Move vs. Entry/Exit
before after
![Page 38: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/38.jpg)
Move vs. Entry/Exit
before after
![Page 39: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/39.jpg)
Move vs. Entry/Exit
before after
![Page 40: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/40.jpg)
Move vs. Entry/Exit
before after
![Page 41: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/41.jpg)
Move vs. Entry/Exit
before after
![Page 42: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/42.jpg)
Move vs. Entry/Exit
before after
![Page 43: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/43.jpg)
Move vs. Entry/Exit
before after
![Page 44: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/44.jpg)
Move vs. Entry/Exit
before after
![Page 45: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/45.jpg)
Move vs. Entry/Exit
before after
![Page 46: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/46.jpg)
Move vs. Entry/Exit
before after
![Page 47: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/47.jpg)
Move vs. Entry/Exit
before after
Motion: (x,y,θ)
![Page 48: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/48.jpg)
Entry vs. Exit
before after
Example 1 (entry)
![Page 49: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/49.jpg)
Entry vs. Exit
before after
Example 1 (entry)
![Page 50: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/50.jpg)
Entry vs. Exit
before after
Example 1 (entry)
![Page 51: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/51.jpg)
Entry vs. Exit
before after
…
File1.pdf File2.pdf File3.pdf File4.pdf File5.pdf File6.pdf
…
![Page 52: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/52.jpg)
Entry vs. Exit
before after
Example 2 (entry)
![Page 53: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/53.jpg)
Entry vs. Exit
before after
…
File1.pdf File2.pdf File3.pdf File4.pdf File5.pdf File6.pdf
…
![Page 54: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/54.jpg)
Entry vs. Exit
before after
![Page 55: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/55.jpg)
Entry vs. Exit
before after
?
![Page 56: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/56.jpg)
Entry vs. Exit
before after
?
![Page 57: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/57.jpg)
Entry vs. Exit
before after
……
![Page 58: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/58.jpg)
Entry vs. Exit
before after
……
Amount of change
time
![Page 59: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/59.jpg)
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
“A document moved from (x1,y1) to (x2,y2)”
Document Recognition
before after
File1.pdf
File2.pdf
File3.pdf
Scene Graph Update
![Page 60: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/60.jpg)
Document Recognition
…
File1.pdf File2.pdf File3.pdf File4.pdf File5.pdf File6.pdf
• Match against PDF image database
…
![Page 61: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/61.jpg)
Document Recognition
…
File1.pdf File2.pdf File3.pdf File4.pdf File5.pdf File6.pdf
• Match against PDF image database
• Performance– Can differentiate between ~100 (or more) documents– ~200x300 pixels per document for reliable match
…
![Page 62: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/62.jpg)
Algorithm OverviewInput
Frames… …
Event Detection
Event Interpretation
“A document moved from (x1,y1) to (x2,y2)”
Document Recognition
before after
File1.pdf
File2.pdf
File3.pdf
Scene Graph Update
![Page 63: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/63.jpg)
Scene Graph Update
before after
Motion: (x,y,θ)
Desk
![Page 64: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/64.jpg)
Scene Graph Update
before after
Motion: (x,y,θ)
Desk
![Page 65: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/65.jpg)
Scene Graph Update
before after
Motion: (x,y,θ)
Desk Desk
![Page 66: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/66.jpg)
Scene Graph Update
before after
Motion: (x,y,θ)
Desk
?
![Page 67: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/67.jpg)
Scene Graph Update
before after
Motion: (x,y,θ)
Desk Desk
![Page 68: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/68.jpg)
Scene Graph Update
Desk DeskDeskDesk
…
![Page 69: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/69.jpg)
Scene Graph Update
Desk DeskDeskDesk
…
![Page 70: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/70.jpg)
Photo Sorting Example
![Page 71: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/71.jpg)
Current Directions
• Handle more realistic desktops• Speed up processing time• Other useful functionalities
– Written annotation– Version management– Bookmark– Multi-user queries
![Page 72: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/72.jpg)
For More Information
• Our publications– Jiwon Kim, Steven M. Seitz, Maneesh Agrawala. The Office of
the Past. IEEE Workshop on Real-time Vision for HCI, 2004.– Jiwon Kim, Steven M. Seitz, Maneesh Agrawala. Video-based
Document Tracking: Unifying Your Physical and Electronic Desktops. To appear in Proceedings of UIST, 2004.
• Other related work– David G. Lowe. Distinctive image features from scale invariant
keypoints. International Journal of Computer Vision, 2004.– Pierre Wellner. Interacting with paper on the DigitalDesk.
Communications of the ACM, 36(7):86.97,1993.– D. Rus and P. deSantis. The self-organizing desk. In
Proceedings of International Joint Conference on Artificial Intelligence, 1997.
![Page 73: Motivation - University of Washingtongrail.cs.washington.edu/projects/office/doc/rtv4hci_dtracking_talk.pdfFile1.2.3.4.5.6.pdf • Match against PDF image database • Performance](https://reader034.fdocuments.in/reader034/viewer/2022050118/5f4e92a5a9d17024145c1064/html5/thumbnails/73.jpg)
For More Information
• Our publications– Jiwon Kim, Steven M. Seitz, Maneesh Agrawala. The Office of
the Past. IEEE Workshop on Real-time Vision for HCI, 2004.– Jiwon Kim, Steven M. Seitz, Maneesh Agrawala. Video-based
Document Tracking: Unifying Your Physical and Electronic Desktops. To appear in Proceedings of UIST, 2004.
• Other related work– David G. Lowe. Object recognition from local scale-invariant
features. International Conference on Computer Vision, 1999.– Pierre Wellner. Interacting with paper on the DigitalDesk.
Communications of the ACM, 36(7):86.97,1993.– D. Rus and P. deSantis. The self-organizing desk. In
Proceedings of International Joint Conference on Artificial Intelligence, 1997.