A Keystone-free Hand-held Mobile Projection System Li Zhaorong And KH Wong Reference: Zhaorong Li,...

A Keystone-free Hand-held Mobile Projection System

Li ZhaorongAnd

KH WongReference: Zhaorong Li, Kin-Hong Wong, Yibo Gong, and Ming-Yuen Chang, “An Effective Method for Movable Projector Keystone Correction”, IEEE Transactions on Multimedia, VOL. 13, NO. 1, Feb 2011.

keystone correction 1

Outline

• Introduction• Methodology

– Pro-cam pair calibration– Projection region detection and tracking– Automatic keystone correction

• Experimental results• Conclusion


Introduction


Aim of this work

• A mobile projector keystone correction method– project keystone free content on a general flat

surface– without adding markings or boundaries on the

surface


Aim of this work

• Configuration


Aim of this work

• Desired Results

Keystoned projection Corrected projection


http://www.youtube.com/watch?v=y5XYdeh8Bno


Motivation

• Mobile projector becomes popular– Flexible projector-screen position– More freedom of display control such as viewing angle,

distance etc– General flat surface can be used as screen– Physical size and cost shrinks quickly– Emergence of mobile devices with embedded projector– Others……


Motivation

• Limitation: keystone distortion !– When projecting onto a screen at oblique

position, the projection region becomes a trapezoid instead of a rectangle

– Traditional static projection system also face this problem

– Stringent for mobile projection system


Motivation

• Existing keystone correction method– All for static projector system, not suitable for

mobile projection, eg.• Su [1]: requires bounded screen• Li et al[2]: requires bounded screen • Raskar [3]: high computation, not real-time• One-time correction, not continuous


Motivation

• Special need for mobile keystone correction– Screen independence

• no specially designed or position-fixed screen should be required

– Continuous processing in real time• Continuous correction instead of one-time correction is

expected


Our approach

• Propose an effective method that can continuously correct the keystone distortion on a general markless screen

• Only additional device used is a webcam attached with the projector


Our approach

• We add a green frame to the projector screen and project it to the display screen

• Track the green frame use the camera and then automatically correct the keystone correction


Contributions

• An artfully designed particle filter tracker for tracking projection region

• An efficient and accurate recovery algorithm for recovering 3d projection region

• A continuous and markless mobile projector keystone correction system


Methodology


Overview

• Three major modules– Projector-camera pair calibration– Projection region detection and tracking– Automatic keystone correction

3D projectionregion recovery

Projector-camera pair calibration

G, K

Corrected projection

Keystoned projection

Pre-warped image3D rectangle

Keystone correctionProjection region tracking

Camera frame

Flow chart


Part 1: Projector-camera pair calibration

• Projector-camera relationship– Projection matrix from 3D camera coordinate system

to projector image plane

• Benefits– Independent from projector movement– No need to estimate explicit parameters– Easy to estimate



• Projective model

p c x GXpxcX

G

: 2D projector image pixel

: 3D point in camera frame

: 3x4 projection matrix

: scale factor



• Estimation method– Project cross to an ordinary cardboard with

known size– Collect a number of projector-camera

corresponding crosses– Calculate the 3D coordinate of the cross– Estimate G matrix using SVD


Part 2: Projection region detection and tracking

• Add a green frame to the projection region• Detect the frame in the initial frame• Track the frame in the subsequent frames


Detection• Based on the Canny edge map and Hough line segments• Find a quadrangle satisfying following criteria

– each side of the quadrangle longer than a threshold– opposite sides should have similar lengths– each angle within the range from 30 to 150 degree– the overlapping ratio of the line segments to the four sides of the

formed quadrangle bigger than a threshold– the quadrangle approximately located around the center of the

camera image


Tracking

• The relationship between projector screen and the projection region in camera

p c x Hx

Hpxcx

: the homography matrix

: the projector image pixel

: the camera image pixel


Tracking

• The homography matrixT

1[ ]d

tn

H J R K

JRtn

Kd

: intrinsic parameter matrix of projector

: rotation matrix of the projector w.r.t camera

: translation vector of the projector w.r.t camera

: normal of the screen

: distance of the screen from the camera: intrinsic parameter matrix of camera

H is dominated by n and d since

J,R,t,K are all fixed


Tracking

• Tracking state vector

• Particle filter tracking– State dynamic model– Observation model– Initialization

T[ , ]ds n


Tracking

• State dynamic model

1 1 1( | ) ( , )k k k kp Uniform s s s e s e

1,k ks s

e

: the state of k-1 and k frame

: the uncertainty of the movement of the projector


Tracking

• Observation model– Re-project each particle to the camera

T1[ ]

d

JtnH JR K

3 3 3 1[ , ] [ , ] JR Jt G G

c 1 p x H x


Tracking

• Observation model– Compute likelihood by comparing the re-projected

quadrangle with the edge map• Specifically, check how many edge points on each of four sides• The likelihood of each side is the percentage of the edge points

among all side points• The likelihood of the particle is the sum of likelihoods of four sides


Tracking

• Initialization– The detected quadrangle in the detection stage is

used to initialize the particle filter– First, recover the 3D position of the quadrangle

[using the method in Part 3]– Second, compute its normal and distance from the

camera


Part 3: Automatic keystone correction

• Three steps– Recover 3D projection region– Look for inscribed rectangle– Pre-warp projection image


Recover 3D projection region

• Solve following equation set for each corner of the projection region using SVD

c c x KXp c x GX

: the corner of the projection region in camera

: the 3D corner of the projection region

: the corner of the projector screen

cxpxcX


Recover 3D projection region

• Incorporate co-planarity constraint

2 2 2 2T T c T T c4 41 i 2 i 1 i 2 iT c T c T c T c

1 13 i 3 i 3 i 3 i

2c c c c c c1 4 1 2 1 3

i i i iu v

c c

i i

k X k X g X g X

k X k X g X g X

X X X X X X

000000000000000000000000000000000000000000

T T T1 2 3, ,k k kT T T1 2 3, ,g g g

K

G: three rows of

: three rows of

: weight

Use the SVD solution as the initial value


Look for inscribed rectangle

• Simple geometric intersection


Pre-warp projection image

• Re-project inscribed rectangle into the camera image

• Find homography between the projection and projector screen

• Pre-warp the display image using the homography


Pre-warp projection image

Pre-warped projection image Display result


Experimental results


Devices

• PC with 2.16GHz CPU, 1 GB memory• Optoma mobile projector 1280x1024• Logitech Quickcam Pro 4000 webcam 320x240


Projector camera pair calibration

• Error distribution

Back-projection error corresponding to 80% inliers is 4.2 pixels


Projection region tracking

• Mean and std tracking error w.r.t different noise levels on simulation data


Projection region tracking

• Mean and std tracking error on real data are 3.4 and 3.6 pixels

Trajectory of the real projector movement


Keystone correction

• Some real correction results



Keystone correction

• The error of keystone correction against different poses of the projector


Keystone correction

• Comparison of our keystone correction module with the static projector keystone correction


Speed

• 16 fps on our platform• The per-frame processing time is about 60 ms• The pre-warping step occupies most of the

time (about 36 ms)


Conclusion

• Proposed an effective mobile keystone correction method

• Mobility and Markinglessness are the most distinguishing features

• Limitation: need to project a green frame• Future work: Use invisible IR LED to eliminate the

green frame


Reference• [1] R. Sukthankar, R. Stockton, and M. Mullin, “Smarter presentations:

Exploiting homography in camera-projector systems,” in Intl. Conf. on Computer Vision, 2001.

• [2] B. Li and I. Sezan, “Automatic keystone correction for smart projectors with embedded camera,” in Intl. Conf. on Image Processing, 2004

• [3] R. Raskar and P. Beardsley, “A self-correcting projector,” in Intl. Conf. on Computer Vision and Pattern Recognition, 2001

• [4] Zhaorong Li, Kin-Hong Wong, Yibo Gong, and Ming-Yuen Chang, “An Effective Method for Movable Projector Keystone Correction”, IEEE Transactions on Multimedia, VOL. 13, NO. 1, Feb 2011.


Thank you


A Keystone-free Hand-held Mobile Projection System Li Zhaorong And KH Wong Reference: Zhaorong Li,...

Documents

Transcript of A Keystone-free Hand-held Mobile Projection System Li Zhaorong And KH Wong Reference: Zhaorong Li,...