Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context
description
Transcript of Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context
![Page 1: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/1.jpg)
Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context
Felix Loew
Master Thesis (TU München)
21.01.2005
![Page 2: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/2.jpg)
• Context-Awareness and Augmented Reality
• Vision Based Tracking: Natural Feature Tracking
• A „Magic Book“ User Study
• Software Architecture (DWARF )
• Expected results and conclusions
Outline
![Page 3: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/3.jpg)
Context-Awareness and Augmented Reality
• A system has to adopt to the user´s behaviour!
• A system has to learn how it is used!
• New intuitive methods of interaction are required!
• The system has to disappear (Marc Weiser)!
![Page 4: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/4.jpg)
The „Magic Book“ in this context
• User has to „learn“ how it is used
• Very error-prone to movement– Movement of the handheld device is not
considered in the tracking routine– Vision based tracking loses orientation– User has to reinitialize the tracking (looking on
the marker again)– Immersion decreases!
![Page 5: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/5.jpg)
My Idea: Enhancing the Magic Book
Inertial Tracker: Intersense, Inertial Cube3 (~120FPS)
Vision Based Tracking: NFT (~20 FPS)
Movement
![Page 6: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/6.jpg)
• Context-Awareness and Augmented Reality
• Vision Based Tracking: Natural Feature Tracking
• A „Magic Book“ User Study
• Software Architecture (DWARF )
• Summary, Conclusions & Limitations
Outline
![Page 7: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/7.jpg)
Basics: Natural Feature Tracking
Frame 1 Frame 2
Frame 3 Frame 4
![Page 8: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/8.jpg)
• NFTARToolkit (Kato): 2D planar scenes• Estimation of feature position in next frame
– It is not realistic that no movement occurs– Linear estimation– Kalman Filter
• Within a window the “best match” with the template is searched (Template Matching)
• For every pixel within window the similarity with the template is calculated (Normalized Cross Correlation)
Basics: Natural Feature Tracking
![Page 9: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/9.jpg)
Template Matching
![Page 10: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/10.jpg)
• Calculates similarity between template and area around search point– values [-1..1]
• Best match is considered as feature point in current frame (threshold ~0.7)!
• Computational expensive!– O (templateSize2 * searchSize2)
Normalized Cross Correlation
![Page 11: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/11.jpg)
My approach
• Consider movement as well!
• Use additional tracking device (Inertial Tracker / 3DOF)
• Naive approach:much movement large search window
low movement small search window
• Dynamic configuration during runtime
![Page 12: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/12.jpg)
What is much movement? Which movements actually occur?How does a user use such an application?
USER STUDY ON MAGIC BOOK
My approach
![Page 13: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/13.jpg)
• Context-Awareness and Augmented Reality
• Vision Based Tracking: Natural Feature Tracking
• A „Magic Book“ User Study
• Software Architecture (DWARF )
• Expected results and conclusions
Outline
![Page 14: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/14.jpg)
• Goals– Detecting specific movement patterns– Derive Movement model– Estimation of a correlation between changes in
orientation and feature points coordinates
User Study Setup
Orientation (Handheld, 3D)
Position (FP-Coords, 2D)
![Page 15: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/15.jpg)
User Study Setup
• Tracking Orientation and Position (6DOF)– Magnetic Tracker:
Flock of Birds
– Bird1: Magic Book
– Bird2: Handheld device
– Calibration to Magic Book coordinate system (Bird1)
![Page 16: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/16.jpg)
• Magnetic Tracker– Timestamp
– Position P = (px,py,pz)
– Orientation (quaternion) Q = (v,s)
• NFT– Timestamp
– 4 feature points per frame• Id
• Screen-Position S = (sx,sy)
Logging
![Page 17: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/17.jpg)
• Magnetic Tracker
pics\2_20_1.Case2.Flock.xls
• NFT
pics\2_20_1.Case2.Coords.xls
Logging
![Page 18: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/18.jpg)
User Tasks
• Comparable Tasks (Navigation Tasks)– User has to answer certain questions (Overview,
Focus and Search Tasks)• „What is the girls hair color?“
• „How many people do you see?“
• Free Tasks– User can use Magic Book freely
![Page 19: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/19.jpg)
Evaluation
• Ideas for evaluation (Correlation)– Which angular offset (FOB) causes which
coordinate offset (NFT)?– Which angular velocities occur?– Which rotations occur?– Which changes of position occur?– Is there a common movement profile or are
there serious differences between different users?
![Page 20: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/20.jpg)
Evaluation
• Find mapping between:
• Configure Software Architecture with this information
orientation search window
![Page 21: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/21.jpg)
• Context-Awareness and Augmented Reality
• Vision Based Tracking: Natural Feature Tracking
• A „Magic Book“ User Study
• Software Architecture (DWARF)
• Expected results and conclusions
Outline
![Page 22: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/22.jpg)
Software Architecture (DWARF)
NFT ARToolkit
![Page 23: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/23.jpg)
Dynamic Configuration
![Page 24: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/24.jpg)
• Context-Awareness and Augmented Reality
• Vision Based Tracking: Natural Feature Tracking
• A „Magic Book“ User Study
• Software Architecture (DWARF)
• Expected results and conclusions
Outline
![Page 25: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/25.jpg)
Conclusions & Summary
• Hybrid Tracking approach to consider context information as well– Movement context– 2DO: What about user context?
• User Study on Magic Book (work in progress)– Expected conclusions on usage of Magic Book– Find correlation between movement and NFT properties– Process a configuration out of the study– Derive Movement model of the Magic Book– Limitation: Very task specific (features are not moving, only 2D)!
• Software Architecture
![Page 26: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/26.jpg)
• Heaps of work– I won’t cover everything I want to do
• Other “sensor-fusion” approaches might be more suited (Kalman-Filtering)
• Evaluation of User Study will be hard
• Is there an abstraction for NFT based applications?
Limitations & 2DO´s
![Page 27: Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context](https://reader030.fdocuments.in/reader030/viewer/2022012919/5681592e550346895dc65cc6/html5/thumbnails/27.jpg)
Thanks
• I would be glad if some of you want to join the study!