Mobile Motion Tracking using Onboard Camera
description
Transcript of Mobile Motion Tracking using Onboard Camera
![Page 1: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/1.jpg)
Mobile Motion Tracking using Onboard Camera
Supervisor: Prof. LYU, Rung Tsong Michael Prepared by: Lam Man Kit
Wong Yuk Man
![Page 2: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/2.jpg)
Agenda
Motivation & ObjectivePrevious Work Improvement & New components Sample Applications
Conclusion
Q&A
![Page 3: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/3.jpg)
Motivation
Rapid increase in the use of camera-phone
Camera-phone can perform image processing tasks on the device itself
Enhance human-computer interaction by mobile phone
![Page 4: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/4.jpg)
Objective
Implement real-time motion tracking on Symbian phone, without requiring additional hardware
Translational motion trackingRotational motion tracking
To develop a tracking engine for other mobile devices developers to use
![Page 5: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/5.jpg)
Previous Work
A translational motion tracking engineBlocking matching• SEA (Successive Elimination Algorithm)• PPNM• PDE (Partial Distortion Elimination)• Adaptive Window Search• Spiral Scan
Feature selection
A translational motion tracking engine
![Page 6: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/6.jpg)
Block Matching Algorithm
![Page 7: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/7.jpg)
Block Matching Algorithm
Evaluate the "goodness" of a match by Sum of Absolute Difference
Select the candidate block with the lowest error
N
i
N
j
jiYjiXyxSAD1 1
|),(),(|),(
![Page 8: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/8.jpg)
Feature SelectionA good feature block:
High variance -> complex blockNot in a repeated pattern
If a good feature block is found, the performance of the motion tracking is improved
We have made improvements to our existing algorithm
![Page 9: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/9.jpg)
squares
Old Feature Selection
Divide the current frame into squares
Apply Feature Selection Algorithm to each squaresHowever, the best feature block may appear between two squares
![Page 10: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/10.jpg)
New Feature Selection
More blocks are sampledSample 26x26 variances of blocks with size 15x15 in a 54x54 window
Search in spiral way and stop searching when selection criteria are matched
Prefer to find a feature block in the centerPrevent out of bound problem (block appear out of screen)
![Page 11: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/11.jpg)
New Feature SelectionIf a feature block is found on the edge, the following will happen:The tracking algorithm will fail to find the exact matchSolution:
Apply additional constraint
![Page 12: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/12.jpg)
New Feature selectionIn order to find a feature block that is not on the edge, we apply a checking algorithmImportant difference in all directions => interest point
![Page 13: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/13.jpg)
New Feature Selection
Now our newest feature selection algorithm becomes:
Find a block with a large variance -> which indicate the complexity of the blockCheck if the block is on the edge or not by calculating the SAD between the candidate block and its 4 neighbors
• If either one of the SAD is small -> the block is on edge -> reject
• Else the block is not on edge
![Page 14: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/14.jpg)
SSD is used instead of SAD
SSD = Sum of Squared DifferenceLast term, SAD is used as matching criteriaSAD is commonly used because of its lower complexity (faster)But we chose to use SSD finally
![Page 15: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/15.jpg)
SSD is used instead of SAD
Experiment shows that SSD has better performance in accuracy andAlgorithm with SSD run as fast as that with SADReason (of why as fast as SAD):
Elimination effects from SEA, PPNM & PDE become large when SSD is used Compensate the higher complexity of SSD
![Page 16: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/16.jpg)
SSD is used instead of SADSEA and PPNM lower bound is adjusted using the inequality (12) proposed in the following paper
J.J. Francis and G. de Jager. A Sum Square Error based Successive Elimination Algorithm for Block Motion Estimation (2002)
![Page 17: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/17.jpg)
Experimental Result on Symbian phone
Frame rateMaximum frame rate supported by Nokia 6600 is about 14 frames/secRunning our algorithm (1/0.007 frames/sec) once only for each frame do not slow down the displaying frame rate, in other word, the frame rate can still be 14 frames/sec, our algorithm doesn’t lag the display
![Page 18: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/18.jpg)
Rotation Tracking Engine
As we rotate the phone, the object can still be tracked correctly (center of object roughly equals center of green box)
Observation and Motivation
![Page 19: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/19.jpg)
Rotation Tracking Engine
Our approachIf two blocks are tracked at the same time, angle of line connecting the two blocks reflect the tiling angle of the phone
Two-block approach
![Page 20: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/20.jpg)
Rotation Tracking Engine
Another approachTrack one block by simple motion tracking engine, then find which rotation gives the best matchFail if tracking object is the same for different angle
One-block approach
![Page 21: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/21.jpg)
Rotation Tracking Engine
Modification of motion tracking algorithm to suit rotation tracking
Linear Adaptive method used in translation motion tracking is not used here because phone’s motion can be both linear motion and circular motion
![Page 22: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/22.jpg)
Rotation Tracking EngineWhat to predict?
Given the coordinates of the tracking blocks in the last 2 previous framesPredict the coordinates of the blocks in the next frame
θθ
A simplified mathematic problem (the following assumptions are approximately correct)
All three lines pass through the circle’s center
End points of the three lines lie on the circle
Line L2
Line L1
Line L3
![Page 23: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/23.jpg)
Rotation Tracking Engine
SolutionAngle between L2 and L3 = θ
• θ = Tan-1(slope(L2)) – Tan-1(slope(L3))
Coordinates of the next tracking block (xL1, yL1) are calculated by multiplying the column matrix of coordinates of the previous block with a rotation matrix
2
2
1
1
cossin
sincos
L
L
L
L
y
x
y
x
![Page 24: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/24.jpg)
Rotation Tracking EngineSolution
The prediction of the position of the next tracking block should also take the translational movement into accountHorizontal displacement = Tx
• Tx = ( xL21 + xL22
– xL31 - xL32
)/2
Vertical displacement = Ty • Ty = ( yL21
+ yL22– yL31
- yL32
)/2
Coordinates of the next tracking block (xL1, yL1) is calculated by
1111
cossin
sincos
12
2
1
1
L
L
L
L
y
x
Ty
Tx
y
x
![Page 25: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/25.jpg)
Rotation Tracking Engine
A simple drawing to show the detected rotation angle of the phone
![Page 26: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/26.jpg)
Rotation Tracking Engine
Reducing error by using levelApply threshold on output, increase/decrease one level only when change is largeTo give less sensitive but more desirable output for game e.g. skiing game: skier face only to 7 directionsReduce difficulty, increase reliabilityA small rotation or a small detection error will not increase/decrease one level
![Page 27: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/27.jpg)
Conditions to be a good background for both engines
ObjectiveMax. performance measurementIncrease usability
Condition:Feature selection can always find a good feature pointWithin certain distance
• No repeat pattern• Very distinct pattern
Pattern is nearly the same when rotated. E.g. Circle which is good for rotation detection
![Page 28: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/28.jpg)
Virtual Mouse
An application that make full use of the translational motion detection engineRemote control the mouse of PC by Symbian phone using motion trackingAdvantages:
It allows input in all directions It provides high levels of control Joystick can still be used as rough moving while camera input can be used as fine translation
![Page 29: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/29.jpg)
Virtual Mouse
ServerIn server side (Windows), the Bluetooth is configured to provide RFComm Service and server regards the Bluetooth transmission port as Comm. Port Server receives message from that Comm. PortCall function in MouseAction.h to make the mouse move and trigger mouse click event
![Page 30: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/30.jpg)
Virtual Mouse
ClientSearch for Bluetooth device nearbyConnect to the selected Bluetooth deviceEvery time motion tracking algorithm finishes, results are sent to server (12 times/sec)Joystick and keypad can also be used to control mouse in PC (Multi-button mouse)
![Page 31: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/31.jpg)
Car Racing GameA “Car Racing Game” is developed using the motion tracking engine.Game lags. It is because:
The engine takes time to find the motion vectorCPU speed of the Symbian phones are lowThe game engine thread uses most of the CPU power while the thread for updating the screen cannot proceed
Solutions:Double BufferingDirect Screen AccessSingle Thread
![Page 32: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/32.jpg)
Double Buffered AreaDouble-buffering – two complete color buffers
One is displayed while the other is calculating next scene to be drawn,When the drawing of the next frame is completed, the content of the back buffer is copied to the front screen
Computing Frame 2
Show Frame 2
Buffer 1 Buffer 2copy
Frame 1
Computing……
Frame 2
Front Screen
Frame 2 ready
![Page 33: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/33.jpg)
Direct Screen AccessIn Symbian, traditional drawing requires the help of Window Server
Overhead in context switchingLower speed
Access the screen directly in 3 ways:Creating and using CfbsScreenDeviceAccessing screen memory directly.Using CdirectScreenAccess.
Using DSA can bypass the Window ServerGet rid of context switchingFaster
![Page 34: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/34.jpg)
Traditional Graphic Programming
Update Display
Device Screen and Keypad
Window Server
RWsSession RWsSessionRWsSession
Application 1 Application 1 Application 1
Key Presses
![Page 35: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/35.jpg)
Direct Screen Access
2. If successful, a region for drawing to is returned to the application.
Window Server
1. Request to Window Server to work in Direct Screen Access mode
Application
![Page 36: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/36.jpg)
Single Thread
Prevent context switchingNo need to do synchronization
![Page 37: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/37.jpg)
Car Racing Game
With the use of efficient graphic programming algorithms, the tracking engine will not affect the performance of the game
It shows that we can use the engine to develop other applications without performance degradation
![Page 38: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/38.jpg)
Skiing Game
Sample program from the book “Developing Series 60 Applications: A Guide for Symbian Os C++ Developers”Based on this game, we plug in our rotation tracking engine into it
The game becomes more interactiveRotate the phone to control the angle of the skier
![Page 39: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/39.jpg)
Skiing Game
![Page 40: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/40.jpg)
Skiing Game
The process of using the engine is very simple
Create an instance of the tracking engineCall the function of the engine to find the motion vectorUse the motion vector for the game logic
![Page 41: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/41.jpg)
Skiing Game
![Page 42: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/42.jpg)
Demo
![Page 43: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/43.jpg)
Experimental Result on Symbian phone
Testing EnvironmentPlatform
• Symbian Phone Nokia 6600 Algorithm
• Our final (hybrid-type) Algorithm (Block matching algorithm featured with Adaptive Window, Spiral Scan, SEA, PPNM and PDE method)
Algorithm parameter• Block size = 17 x 17 (pixels)• Search window size = 16 x 16 (pixels)
Number of block to track in each run of algorithm = 1 block ONLY
![Page 44: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/44.jpg)
Experimental Result on Symbian phone
Testing Result (avg time to run)Final algorithm • 7ms
Final algorithm without adaptive window method and SEA, PPNM method • 22ms
Full Exhaustive Search algorithm (the most simplest one) • 55ms
![Page 45: Mobile Motion Tracking using Onboard Camera](https://reader035.fdocuments.in/reader035/viewer/2022070409/568144de550346895db1ab09/html5/thumbnails/45.jpg)
Q & A