LYU0503 Document Image Restoration on Mobile Phone Using Onboard Camera
Mobile Motion Tracking using Onboard Camera
-
Upload
giacomo-chapman -
Category
Documents
-
view
26 -
download
0
description
Transcript of Mobile Motion Tracking using Onboard Camera
![Page 1: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/1.jpg)
Mobile Motion Tracking using Onboard Camera
Supervisor: Prof. LYU, Rung Tsong Michael Prepared by: Lam Man Kit
Wong Yuk Man
![Page 2: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/2.jpg)
Outline
Motivations
Objective
Methods
Results
Future Work
Q&A
![Page 3: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/3.jpg)
Motivations
Rapid increase in the use of camera-phone.
Commonly used for taking photos or capturing video only.
Is it possible to add more values to the camera and make full use of it ?
![Page 4: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/4.jpg)
Motivations
Unlike traditional camera, camera-phone has more functions than just taking photos.
Camera-phone can perform image processing tasks on the device itself.
Symbian OS makes programming on mobile phones possible.
![Page 5: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/5.jpg)
ObjectiveTo add more values to camera-phone, so that it enhances the human-computer interaction.Implement real-time motion tracking on Symbian phone, without requiring additional hardware.Movement detected acts as an innovative input method for different applications:
Camera mouse to control the cursorNew input method for interactive gamesGesture input
![Page 6: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/6.jpg)
Objective
Novel ubiquitous computing applications can be developed !
![Page 7: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/7.jpg)
Real-World Interaction
![Page 8: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/8.jpg)
Motion Estimation
Motion estimation is a process to find the motion vector of the current frame from reference frame(s).
Optical flowBlock matching
Block matching algorithm is an integral part for most of the motion-compensated video coding standards. Eg MPEG 1, MPEG 2, H.263.
![Page 9: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/9.jpg)
Block MatchingDivide the previous frame to small rectangular blocks.Find the best match for the reference block in current frame.Calculate motion vector between the previous block and its counterpart in the current frame.Typical size for a block: 16x16 pixels.Search Range W: typically 16 or 32 pixels.Similarity Measures:
Mean Absolute Error (MAE)Mean Square Error (MSE)Sum of the Absolute Difference (SAD)
SAD is used in our project.
Current frame
Previous frame
MV
![Page 10: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/10.jpg)
Block Matching2W + 1
2BW + 1
2BH + 12H + 1
Search Window
Block in previous frame
Search Window (in current frame)
A region which has the same center as the selected block in the previous frame, extended by w pixels in both directions
BW = 1BH = 1W = 4H = 4
W
center pixel
![Page 11: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/11.jpg)
Block-match Motion Estimation
Two kinds of methods commonly used:
Fast Search Algorithms• 2-D Logarithmic Search• 3-Step Search (3SS) • Diamond Search
Exhaustive Search Algorithm (ESA)
![Page 12: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/12.jpg)
Fast Algorithms
Fast Search Algorithms:2-D Logarithmic Search3-Step Search (TSS) Diamond Search
Assumption:The matching error monotonically increases as the search position moves away from the optimal motion vector
![Page 13: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/13.jpg)
Fast Algorithms - TSSThree-Step Search (TSS)
1st Step:• Search 8 surroundings
and the central point• Distance = w/2 pixels• Find the best match
2nd Step:• Use previous best match
as center• Repeat 1st step with
distance = w/4 pixels
3th Step:• Repeat 1st step with
distance = w/8 pixels
Searched only 25 points
1
1
11
11
1 1
1
23
2
2
222
2
2333 3 3
33
Center of Block
Search Window
1 2 3
![Page 14: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/14.jpg)
Fast AlgorithmsAdvantages:
Extremely fast
Disadvantages:All fast algorithms greatly rely on a monotonically increasing match criteria around the location of the optimal motion vector
• Easily fall into local minimum
limited number of positions examined (only 25 points) inside the search window, only find suboptimal solution
![Page 15: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/15.jpg)
Exhaustive SearchAll candidates within search window are examined(2w+1)2 positions should be examinedAdvantage: Good accuracy; Finds best matchDisadvantage: High computational load. Impractical for real-time applicationsSolution
Fast Exhaustive Search
2W + 1
2BW + 1
2BH + 12H + 1
Search Window
Block
W
![Page 16: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/16.jpg)
Fast Exhaustive Block Matching Algorithms
Much Faster
No performance Loss
Idea: exclude many search positions while still finding best match:
SEA ( Successive Elimination Algorithm)PNSA ( Progressive Norm Successive Algorithm)
SEA and PNSA can be calculated quickly
![Page 17: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/17.jpg)
SEA algorithmSAD of two blocks X and Y is defined as
By Minkowski inequality
Thus,
By calculating the block-sum difference first, we can eliminate
many candidate blocks (if D > SAD) before doing slow SAD
There exist fast method to calculate the block-sum for SEA
About 2 times faster than exhaustive search !!
N
i
N
j
jiYjiXyxSAD1 1
|),(),(|),(
|)(||)(||)()(| 22112121 yxyxyyxx
),(|),(),(|1 11 1
YXSADjiYjiXN
i
N
j
N
i
N
j
Fast
Slow
Denoted as D
![Page 18: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/18.jpg)
Fast Exhaustive Block-Matching Algorithm
update
….
SAD SAD SAD
….
SAD….
Search range=2W+1
SEA
PNSA
The smallest SAD
Total No of candidate block: (2w+1)2
Probability of eliminating invalid candidate block:SEA < PNSA < SADComputation Load:SEA < PNSA < SAD
![Page 19: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/19.jpg)
Feature SelectionWhich block should be chosen for tracking?
Flat-colored block is not good.
A block in a region of repeated pattern is not good.
Why is the “eye” a good candidate?It is a good tracking location because of the brightness difference between the black and skin colors.
How do we find a good feature block?Is that block good?
Is that block good?
No
No
It is a good block !!
![Page 20: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/20.jpg)
Feature SelectionGoal:
Find a good reference block for tracking
Criteria:The candidate block should have great SAD with it’s neighborsIt contains “complex” information
Great SAD with neighbors blockPrevent ambiguous detectionSpeed up the searching algorithmMany candidate blocks are eliminated by the tree in upper level
Complex block Prevent choosing flat region as reference blockEnhance the performance of PDE (Partial Distortion Elimination)
![Page 21: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/21.jpg)
PDE (Partial Distortion Elimination)
Simple, small overheadComparison can be done Halfway
Stop if the sub-blocks SAD between block X and Y is already larger than the previous minimum SAD
Removes unnecessary computations efficientlyif the feature block Y has high complexity
It will have great SAD with block XIncrease chance of halfway stop
We implement a simple feature selection algorithm based on the above criteria
X: candidate blockY: feature block
![Page 22: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/22.jpg)
Feature SelectionDivide the current frame to small rectangular blocksFor each block, sum all the pixels value, denoted as Ixy (Intensity of the block)Calculate the variance of each block which represent the complexity of the blockUse Laplacian Mask for each block
The Laplacian operator indicates how the reference block differs from the neighbors
• Flat background > small output• Dissimilar with neighbors > large output
Select the block which has the largest Ixy and large variance as the feature block
Laplacian Mask
![Page 23: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/23.jpg)
Adaptive Search Window
Conventional methodSearch window is defined as a rectangle with the same center as block in previous frame, extended by W pixels in both directions.
Search Window
Block
Center of Search Window
![Page 24: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/24.jpg)
Adaptive Search WindowProposed method
Center of the search window is predicted based on the previous displacement and previous predicted displacement Example
• Previous motion vector is (1,0), i.e. one pixel to the right
• The predicted center of search window can be the position next to the center of the previous block
Search Window
Block
Center of Search Window
![Page 25: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/25.jpg)
Adaptive Search Window
MotivationTo Increase the speed of fast full search algorithm by searching the most probably optimal position first
• Need to corporate with Spiral Scan
To increase the chance of finding the true optimum point
• Explained in the following slides
![Page 26: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/26.jpg)
Conventional Search WindowWe used web camera to track the motion of an object and graph showing its x-axis velocity against time is plotted
Due to the limited size of search window, if an object is moving too fast, the optimal position would fall out of the search window, detection error results
|Velocity| < W pixels/sAssume the algorithm is run every second
![Page 27: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/27.jpg)
Adaptive Search WindowBased on the previous optimal position and motion vector, we estimate the next optimal position, and this will be the center of the search window
P: Predicted Displacement P’: Previous Predicted DisplacementL: Learning Factor, range is [0.5, 1.0]D: Previous Displacement
DLPLP
')1(
![Page 28: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/28.jpg)
Adaptive Search WindowApplying adaptive search window method, the relative velocity fall within the range [-20,20], all true optimum points fall into the search window and thus no serious detection error results
Relative velocity = actual disp. – expected disp.
|Acceleration (relative velocity)| < W pixels/s
W: Search RangeAssume the algorithm is run every second
![Page 29: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/29.jpg)
Raster Scan MethodConventional Block Scanning Method
when we use the previous block to find a best match in the current frame, we calculate the Sum of Absolute Difference (SAD) of the previous block with the current block at the left top position of the search window first. Then scan from top to bottom, from left to right.
Simply to implement
Small code overhead1 2 3 4
5 6 7 8
9 10 11 12
13
14 15 16
Search Window
# represents thepriority of each current block in block matching
![Page 30: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/30.jpg)
Spiral Scan MethodProposed Block Scanning Method
Observation• The order of scanning can affect the time to reach the
optimum candidate block • When SEA, PPNM or PDE method are used, this can affect
the amount of computation
• When adaptive search window method is used, the motion vectors are center biased
Objective• Search the motion vector around the center of a search
window first• Higher chance to meet the optimal position earlier
algorithm run faster
),(|),(),(|1 11 1
YXSADjiYjiXN
i
N
j
N
i
N
j
![Page 31: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/31.jpg)
Spiral Scan MethodFirst find the SAD at the center of the search windowThen find the SAD at position that are n pixels away from the center where n = [1,BW]
Search Window
![Page 32: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/32.jpg)
Spiral Scan MethodProposed Block Scanning Method
Result• Require larger memory space
– If fast calculation of block sum is used together, the whole block sum 2D array is needed to be stored.
• A bit larger code overhead– Degradation in speed not significant
– Spiral Scan Complexity: O(DX2)
– SAD Complexity: O(DX2 BW2)
• Speed of Algorithm significantly improved, when use with adaptive search method, about 2-3 times speed-up in real-time motion tracking
![Page 33: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/33.jpg)
Table showing the time required to find the motion vector at different regions using different algorithms (Each algorithm is run 5 times using 2.0GHz CPU)Result
Time
Typical Point
ESA
SAD
Algorithm
ESA
PDE
SAD
Algorithm
SEA
PPNM
SAD
Algorithm
SEA
PPNM
PDE
SAD
Algorithm
Spiral
SEA
PPNM
PDE
SAD Algorithm
Adaptive
Spiral
SEA
PPNM
PDE
SAD
Algorithm
High Gradient
High Variance 2078ms 484ms 156ms 109ms 16ms 16ms
Low Gradient
High Variance2203ms 515ms 578ms 375ms 94ms 47ms
Low Gradient
Low Variance2078ms 360ms 359ms 203ms 79ms 47ms
Optimal Motion Vector = (-5, 12), Previous Motion Vector = (-2, 4) AffectingSpiral Scan and Adaptive Search Window algorithm
Static Frame Motion Tracking
This is just to illustrate speed is possible to be improved with our final
algorithm, a more general measurement will be
presented at the next slide
![Page 34: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/34.jpg)
Algorithm using adaptive spiral method can reduce the average distance between search window’s center and optimum block’s position, thus improve the speed of algorithm ( as illustrated in the previous slide )
Result
Real-Time Motion Tracking
Average Velocity/Distance ~ 15 pixels Average Velocity/Distance ~ 6 pixels
![Page 35: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/35.jpg)
Summary Video Source captured by camera Extractor Source Frames
Already selected feature block?
Feature Selection
delayBlock-matching Algorithm using two image frames
Transmitter
Server Application
Frame tYes
No
Frame t-1
A feature block is selected as reference block
MV of reference block
e.g. Bluetooth
![Page 36: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/36.jpg)
Contribution
Proposed a method to improve the block matching algorithm for our application
Adaptive Spiral Method• Improve performance• Require larger memory space
New combination• Adaptive Spiral SEA PPNM PDE SAD
algorithm
![Page 37: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/37.jpg)
Testing Platform on Window In order to test the performance of our algorithms, we have written a GUI program using Window MFC and OpenCV library.
![Page 38: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/38.jpg)
Testing Platform on Symbian We finally built an application on Symbian as an testing platform to further test and fine tune our algorithm. Ready for other applications to build on top and use the motion tracking result directly.
![Page 39: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/39.jpg)
Simple Applications finishedA “pong” game written in C#
Play in Window using Web camera as input device
A “pong” game written in Symbian Language
Play in Symbian phone using onboard camera
![Page 40: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/40.jpg)
Future Work
Further improve the block matching algorithm by hierarchical method
Study and implement algorithms to detect rotation angle
Develop virtual mouse application
Develop multiplayer game
Build motion tracking API on Symbian
![Page 41: Mobile Motion Tracking using Onboard Camera](https://reader036.fdocuments.in/reader036/viewer/2022062721/56813745550346895d9ed89b/html5/thumbnails/41.jpg)
Q & A