LYU0503 Document Image Restoration on Mobile Phone Using Onboard Camera

11

LYU0503LYU0503Document Image RestorationDocument Image Restoration

on Mobile Phoneon Mobile PhoneUsing Onboard CameraUsing Onboard Camera

Supervisor:Supervisor:Prof. Michael R.LyuProf. Michael R.Lyu

Group Member:Group Member:Leung Man Kin, StephenLeung Man Kin, StephenNg Ying Kit, TommyNg Ying Kit, Tommy

22

OutlineOutline

IntroductionIntroduction ApproachApproach Stitching AlgorithmStitching Algorithm Geometric TransformationGeometric Transformation ProgressProgress ConclusionConclusion Future PlanFuture Plan

33

OutlineOutline


44

IntroductionIntroduction

Use mobile phone to capture photo Use mobile phone to capture photo of document.of document.

The document can be stored in the The document can be stored in the phone or sent to other people by phone or sent to other people by several means.several means.

55

ObjectiveObjective

Use camera to take photo of different Use camera to take photo of different parts of a document.parts of a document.

Use suitable algorithm to merge the Use suitable algorithm to merge the photos back to a whole document.photos back to a whole document.

Transmit the document by several Transmit the document by several meansmeans• e.g. Fax, MMS, etc.e.g. Fax, MMS, etc.

66

MotivationMotivation Most of phones nowadays have built-in camerMost of phones nowadays have built-in camer

a.a.

Resolution of camera is not enough for documResolution of camera is not enough for documentent• The text on the document would be blurredThe text on the document would be blurred

Symbian phone can provide several transmissiSymbian phone can provide several transmission function. E.g. Fax, MMSon function. E.g. Fax, MMS

Fax at any time!Fax at any time!

77

OutlineOutline


88

ApproachApproach

Take photos of different part of Take photos of different part of documentdocument

Use stitching algorithm to merge the Use stitching algorithm to merge the photos into a photophotos into a photo

Stitching algorithm is our major Stitching algorithm is our major research area of our projectresearch area of our project

99

AssumptionAssumption

The document consists mainly of The document consists mainly of text.text.

The camera has macro mode.The camera has macro mode.• Sharp Close Up.Sharp Close Up.

1010

ApproachApproach

Procedure of our designProcedure of our design• Take a photo of full document first.Take a photo of full document first.

• Then ask the user to take photos of Then ask the user to take photos of different partsdifferent parts

Overlay a rough picture on the screen of the Overlay a rough picture on the screen of the phone.phone.

1111

ApproachApproach

A4 document

1212

OutlineOutline


1313

Stitching AlgorithmStitching Algorithm

Stitching needs to find corresponding Stitching needs to find corresponding points in two imagespoints in two images

two different approachestwo different approaches• Direct alignment Direct alignment

Compare images pixel by pixelCompare images pixel by pixel e.g.e.g.

• Feature-based registrationFeature-based registration Use some methods to find feature pointsUse some methods to find feature points Match features in two imagesMatch features in two images

images. two theare and , 10

2

01 IIxIuxIuEi

iiSSD

1414

Stitching AlgorithmStitching Algorithm Feature-based registration is chosen.Feature-based registration is chosen.

• Usually, direct alignment is less efficient.Usually, direct alignment is less efficient.• Need to consider different scale, rotation, etc.Need to consider different scale, rotation, etc.

There are several methods to extract There are several methods to extract feature.feature.• E.g. corner detector.E.g. corner detector.

Scalable Invariant Feature Transform Scalable Invariant Feature Transform (SIFT) is used to extract the feature.(SIFT) is used to extract the feature.• The features are invariant to image scale and The features are invariant to image scale and

rotation.rotation.

1515


How to get the feature point?How to get the feature point?• Identifies key locations in scale space by Identifies key locations in scale space by

looking for locations that are maxima or looking for locations that are maxima or minima of a difference-of-Gaussian minima of a difference-of-Gaussian function.function.

2

22

222

1,,

yx

eyxG

yxIyxGkyxGyxD ,*,,,,,,

1616


Varying Varying σσ

Repeat the above process using subsampled image Repeat the above process using subsampled image for several timesfor several times• Decrease the resolution by 2.Decrease the resolution by 2.

2

22

222

1,,

yx

eyxG

1717


Find the maxima or minima by comparinFind the maxima or minima by comparing 26 neighbors.g 26 neighbors.

They are the key feature points.They are the key feature points.

1818


Property of the feature keypointsProperty of the feature keypoints• Achieve partial invariance to local variations Achieve partial invariance to local variations

like affine or 3D projections.like affine or 3D projections.

• Need to filter feature keypoints furtherNeed to filter feature keypoints further Reject keypoints that have low contrast.Reject keypoints that have low contrast.

1919


How to store the feature?How to store the feature?• For each keypoint, image gradient magnituFor each keypoint, image gradient magnitu

des and orientations are sampled to store tdes and orientations are sampled to store the feature.he feature.

2020

Stitching AlgorithmStitching Algorithm Compare the features in two imagesCompare the features in two images

• Get the position to stitchGet the position to stitch

However, the photo taken may suffer from However, the photo taken may suffer from geometric distortiongeometric distortion• e.g. Lens distortion, taking photo with different anglese.g. Lens distortion, taking photo with different angles

2121


Use the matched feature to calculate thUse the matched feature to calculate the transformation parameters. e transformation parameters.

Transform images to the correct orientaTransform images to the correct orientation before mergingtion before merging

2222

SIFT


Extract feature keypoint

Matching keypoints in two images

Findtransformation parameter

by matched keypoints

Transform imagesusing the parameter

Merge all the images

2323

OutlineOutline


2424

Lens CorrectionLens Correction

Transformation

2525

Transformation parameters ai

Transformation parameters bi

Image2

Image3

Image1

2626

Take several

overlapping images

of the document……

2727

High Resolution

Document Image

Restored!

2828

OutlineOutline


2929

Porting to Symbian PhonePorting to Symbian Phone

On the PC, the stitching algorithm is impOn the PC, the stitching algorithm is implemented using C# and Matlab.lemented using C# and Matlab.

We should port the algorithm to the SymWe should port the algorithm to the Symbian phone.bian phone.

Symbian OS use C++ as the language.Symbian OS use C++ as the language.

3030


Difficulty:Difficulty:• Their syntaxes are not the same.Their syntaxes are not the same.

e.g. set / get keyword in C#e.g. set / get keyword in C#• Symbian and C# provide different API.Symbian and C# provide different API.

Math library, Array library.Math library, Array library.• Some of data types are different in Symbian Some of data types are different in Symbian

C++.C++.

3131


Difficulty:Difficulty:• The processor on Symbian phone is not poThe processor on Symbian phone is not po

werful.werful.

• The memory space on Symbian phone is noThe memory space on Symbian phone is not large.t large.

We may need modify the algorithm so thWe may need modify the algorithm so that it would suit the phone.at it would suit the phone.

3232

OutlineOutline


3333

ConclusionConclusion

We have studied different stitching algorWe have studied different stitching algorithmsithms

We have implemented the stitching algoWe have implemented the stitching algorithm on PCrithm on PC

We are porting the stitching algorithm oWe are porting the stitching algorithm on Symbian phonen Symbian phone

3434

OutlineOutline

IntroductionIntroduction ObjectiveObjective MotivationMotivation ApproachApproach Stitching algorithmStitching algorithm Geometric TransformationGeometric Transformation ProgressProgress ConclusionConclusion Future PlanFuture Plan

3535

Future PlanFuture Plan

Implement the stitching on the Symbian Implement the stitching on the Symbian phone.phone.

Modify SIFT or design other algorithms fModify SIFT or design other algorithms for more efficient performance on mobile or more efficient performance on mobile phonephone

3636

Future PlanFuture Plan

Improve the algorithm of merging.Improve the algorithm of merging.

Study super-resolution to make text Study super-resolution to make text clearer.clearer.

LYU0503 Document Image Restoration on Mobile Phone Using Onboard Camera

Documents

Transcript of LYU0503 Document Image Restoration on Mobile Phone Using Onboard Camera