Dr. István Marosi Scansoft-Recognita, Inc., Hungary SSIP 2005, Szeged Character Recognition...

Dr. István MarosiScansoft-Recognita, Inc., Hungary

SSIP 2005, Szeged

Character Recognition Internals

04 Jul 2005 Istvan Marosi

OCR InternalsMain tasks of an OCR system:

Image acquisitionLayout recognitionText recognitionUser assisted correctionResult exportation



Image acquisitionGet image

B/W ScanningGray ScanningColor ScanningLoad from image file

Preprocess imageLayout recognitionText recognitionUser assisted correctionResult exportation



Image acquisitionGet imagePreprocess image

Color separationThresholdingDespecklingRotationDeskewing

Layout recognitionText recognitionUser assisted correctionResult exportation

Color SeparationDe-speckle, de-skew


The Preprocessed ImageJoined chars


Joined charsThe Preprocessed Image


The Preprocessed ImageJoined chars


The Preprocessed ImageBroken chars



Image acquisition

Layout recognitionText zones

Columns of flowed textTablesInverse text

Graphic zonesText recognitionUser assisted correctionResult exportation



Image acquisition

Layout recognitionText zonesGraphic zones

Line ArtPhoto

Text recognitionUser assisted correctionResult exportation



Image acquisitionLayout recognition

Text recognitionSegmentationCalculation of Feature Vector ElementsClassificationLanguage AnalysisVoting

User assisted correctionResult exportation


SegmentationWhat are those pixel groups belonging to a single letter?


Calculation of FV Elements: Contour Tracing

Find a (new) white-black transitionFollow the “edge” of the pixels using the MIN or MAX ruleAdministrate the already traced white-black transitionsCollect information while going aroundAnd repeat the process on new shapes ...


Contour TracingFind a (new) white-black transition

Follow the “edge” of the pixels using the MIN or MAX ruleAdministrate the already traced white-black transitionsCollect information while going aroundAnd repeat the process on new shapes ...


Contour TracingFind a (new) white-black transitionFollow the “edge” of the pixels using the MIN or MAX rule

if black(a) then turn(ccw)else if black(b) then forwardelse turn(cw)

a b


Contour TracingFind a (new) white-black transitionFollow the “edge” of the pixels using the MIN or MAX rule

if black(a) then turn(ccw)else if black(b) then forwardelse turn(cw)

a b

if white(b) then turn(cw)else if white(a) then forwardelse turn(ccw)

ab


Contour TracingFind a (new) white-black transitionFollow the “edge” of the pixels using the MIN or MAX ruleAdministrate the already traced white-black transitionsCollect information while going aroundAnd repeat the process on new shapes ...


Some Easily Calculatable Data

Problem #1

Turning CW: In=In-1+1Turning CCW: In=In-1-1 Going Forward: In=In-1



Problem #2

Turning CW: In=In-1+1Turning CCW: In=In-1-1 Going Forward: In=In-1



Problem #3

Going Up: In=In-1-Xn

Going Down: In=In-1+Xn

Going Right: In=In-1

Going Left: In=In-1



Problem #4

Going Up: In=In-1-Xn

Going Down: In=In-1+Xn

Going Right: In=In-1

Going Left: In=In-1






AB A B


A

B AB

Classification; Training modelsRestricted Coulomb Energy (RCE) Network(Dr. Leon Cooper, Dr. Charles Elbaum)


Classification; Training modelsRestricted Coulomb Energy (RCE) Network(Dr. Leon Cooper, Dr. Charles Elbaum)

A

B AB


Classification; Training modelsNestor Learning System (NLS)



Default radius Rmax



Decreased radius



Decreased radius Rmin



Pass 2Decreased radius


VotingText recognition in OmniPage Pro

OCR Engines available:Caere’s engine (codename: Salt & Pepper)

Recognita’s engine (codename: Paprika)

ScanSoft’s engine (codename: Fireworx)


Text recognition in OmniPage ProOCR Engines available:

Caere’s engine (Salt & Pepper)

Uses a Matrix Matching based algorithmfeature set: 40 cells of an 8x5 gridgood overall description of a shapeweaker at detailed structure

Recognita’s engine (Paprika)

Uses a Contour Tracing based algorithmfeture set: convex and concave arcs on the contourgood detailed description of a shapeweaker at overall structure

Voting





ScanSoft’s engine (Fireworx)

Segmentation algorithms:

Voting





ScanSoft’s engine (Fireworx)

Segmentation algorithms:Developed by independent groupsHave different strengths and weaknesses

Voting


Text recognition in OmniPage ProOCR Engines availableSegmentation algorithms

Conclusion:They are complementaryLet’s create a voting system

Voting


Voting strategiesExternal „Black box”voting

~20% gain

Image

Paprika Salt &Pepper

Vote

Txt 3 Txt 1

Dict

Final Txt

Voting

Fire-worx

Txt 2


Voting strategiesExternal „Black box”voting

Internal „Shape”voting

Voting Image

Paprika

Fire-worx

BronzeTxt 3

Txt 2

Dict

Final Txt

Salt &Pepper

Txt 1


Paprika

Original segmentation:Every independent connected component is a

character

Good segmentation: recognizeBad segmentation: reject

Image

Recognize originalsegmentationK.B.

Voting


Paprika

Image

Recognize originalsegmentation

Txt 2Train adaptive classifier

from original shapes

K.B.

AdaptiveK.B.

VotingTxt 1


Paprika

Try several segmentationsLoop if unrecognizable

Image




Recognize broken andjoined shapes

K.B.

AdaptiveK.B.

VotingTxt 1

Dict


Paprika

Image





K.B.

AdaptiveK.B.

Train adaptive classifierfrom ‘ugly’ shapes

VotingTxt 1

Dict


Paprika

Image


Txt 3




K.B.

AdaptiveK.B.

Train adaptive classifierfrom ‘ugly’ shapes

Recognize more brokenand joined shapes Try several segmentations

Loop if unrecognizable

VotingTxt 1

Dict


Image

Paprika

Fire-worx

BronzeTxt 3

Txt 1

Dict

Final Txt

Salt &Pepper

Txt 1

Voting strategies

~60% gain

Voting



Image acquisitionLayout recognitionText recognition

User assisted correctionBy the user’s random editing...

Pop-up verifierManual Training

By proofreading of doubtful wordsResult exportation



Image acquisitionLayout recognitionText recognition

User assisted correctionBy the user’s random editing...By proofreading of doubtful words

Correct: User dictionaryChanged: IntelliTrain

Remember trained charactersApply them on following pages

Result exportation


IntelliTrainRecognized word: sorneUüng


IntelliTrainRecognized word: sorneUüngFixed word: something


IntelliTrainRecognized word: sorneUüngFixed word: somethingSubstitutions found: m rn

thi Uü


IntelliTrainRecognized word: sorneUüngFixed word: somethingSubstitutions found: m rn

thi UüPerform automatically:

Learn image pattern and substitution infoFind similar substituted (‘blue’) text on actual pageMatch against pattern of substitution and correctFind such errors on following pages, too



Image acquisitionLayout recognitionText recognitionUser assisted correction

Result exportationCombine pages into a Document

Header / Footer recognitionPage numbersHyperlinks (e.g. „See Table 20”)

Save results



Image acquisitionLayout recognitionText recognitionUser assisted correction

Result exportationCombine pages into a DocumentSave results

doc filee-mailSpeech synthesizer

Dr. István Marosi Scansoft-Recognita, Inc., Hungary SSIP 2005, Szeged Character Recognition...

Documents

Transcript of Dr. István Marosi Scansoft-Recognita, Inc., Hungary SSIP 2005, Szeged Character Recognition...