HANDWRITING RECOGNITIONA PERSPECTIVE ON TWO DECADES OF INNOVATIONS

Venu Govindaraju

University at Buffalo

New York, USA

Organizing Multiple Experts for Efficient Pattern Recognition

Active Pattern Recognition Using Genetic Programming

A Complexity Framework for Combination of Classifiers in Verification and Identification Systems

Image Processing using Ontology Concepts for Image Segmentation

Language Motivated Approaches for Human Action Recognition and Spotting

Towards a Globally Optimal Approach for Learning Deep Unsupervised Models

PATT

ERN

R

ECO

GN

ITIO

N

Sequential Pattern Classification without Explicit Feature Extraction

Exploiting the Gap between Human and Machine Abilities in Handwriting Recognition for Web Security Applications

Stochastic Modeling of High-level Structures in Handwritten Word Recognition

A Stochastic Framework for Font Independent Devanagari OCR

Language Models and Automatic Topic Categorization for Information Retrieval in Handwritten Documents

Automatic Recognition of Handwritten Medical Forms for Search

Statistical Techniques for Efficient Indexing and Retrieval of Document Images

Enhancing Cyber Security through the use of Synthetic Handwritten CAPTCHAs

Methods for Biomedical Image Content Extraction Toward Improved Multimodal Retrieval of Biomedical Articles

A Semi Supervised Framework for Handwritten Document Analysis

Bayesian Background Models for Retrieval of Handwritten Documents

Accents in Handwriting: A Hierarchical Bayesian Approach to Handwriting Analysis

Multilingual Word Spotting in Offline Handwritten Documents

Enhancement and Retrieval of Low Quality Handwritten Documents

Hierarchical and Dynamic-Relational Models for Handwriting Recognition

Probabilistic Random Field based Text Identification

DO

CU

ME

NT

AN

ALY

SIS

2

Minutia-Based Partial Fingerprint Recognition

Integrating Facial Expressions and Skin Texture in Face Recognition

Integrating Minutiae Based Fingerprint Matching with Local Correlation Methods

A Novel Multi-sample Fusion Methodology for Improving Biometric Recognition

A Framework for Fingerprint Enhancement and Feature Detection

A Framework for Efficient Fingerprint Identification using a Minutiae Tree

Face Modeling and Biometric Anti-spoofing using Probability Distribution Transfer Learning

Intrusion Detection using Spatial Information and Behavioral Biometrics

BIO

ME

TR

ICS

3

5

O

Stone

Tablet

to

iPad

tablet

Full Circle

7

8

9

Air Writing

11

E-signatures and Finger Doodles

This AI Algorithm Learns Simple Tasks as Fast as We Do

Software that learns to recognize written characters from just one example may point the way

towards more powerful, more humanlike artificial intelligence. MIT Technology Review, Dec 2015.

AI

All Things Handwritten

Dynamic lexiconsand integrated segmentation

Interactive lexicons and

multiresolution

Serial fusion and lexicon density

Principled fusion methods

Indexing and Spotting

Text retrieval

CAPTCHAs and security

Accents, signatures, and

security

Personal archives

Flipped classrooms

Key Innovations

1

9

9

5

-

2

0

0

0

2

0

0

0

-

2

0

0

5

2

0

1

0

-

2

0

1

5

2

0

0

0

-

2

0

0

5

2

0

1

5

-

2

0

2

0

LEXICONS

1Lexicon

Postal

ML Success Story - Highlighted in the CCC Symposium on “Computing Research That Changed the

World” (2009); Seminal work of Kim, Govindaraju (PAMI 1997) at the core of the technology

10-10-20 Rule

Lexicon

H. Xue, V. Govindaraju, “On the dependence of handwritten word recognizers on lexicons”, IEEE Transactions

Pattern Analysis and Machine Intelligence, IEEE Computer Society Press, 24(12): 1553-1564 (2002)

18

30% ZIPs contain less than 100 street names; Max streets returned is 3,071

Lexicon

Dynamic Lexicon

Rank Recog result

1 glyburide

2 bumetanide

3 indapamide

Rank Recog result

1 fosinopril

2 perindopril

3 benazepril

Rank Recog

result

1 metoprolol

2 metolazone

3 torsemide

MedicalFaxed prescriptions

20

Lexicons

Lexicon

R. Jayadevan, U. Pal, and F. Kimura: Recognition of Words from Legal Amounts of Indian Bank Cheques.

ICFHR 2010

• Census forms recognition

Lexicon of professions

• Prescription forms

Lexicon of medicines

• Registry of land ownerships

Lexicon of owner

1 2 3 4 5 6 7 8 9

w[7.6]

w[7.2]

r[3.8]

w[5.0]

w[8.6]

o[7.6]r[6.3]

d[4.9]

w[5.0]

o[6.6]

o[6.0]

o[7.2]o[10.6] d[6.5]

d[4.4]

r[7.5]r[6.4]

o[7.8]r[8.6]

r[7.6]

o[8.3]

o[7.7]r[5.8]

1 2 3 4 5 6 7 8 9

o[6.1]

G. Kim, Venu Govindaraju: A Lexicon Driven Approach to Handwritten Word Recognition for Real-Time

Applications. IEEE Trans. Pattern Anal. Mach. Intell. 19(4): 366-379 (1997)

21

Lexicon

Integrated Segmentation

WORD

WELL

WORLD

HOUSE

SCHOOL

Hyderabad A D A - - - A - A

A - A A - A A -A - A - -A - A - - A - -

A - - - D - -

Interactive Features

a. Delhi b. Kolkatta c. Patna d. Dehradun

Lexicon

a) Amherst

b) Buffalo

c) Boston

d) None of the above

S. Madhvanath, V. Govindaraju, “The role of holistic paradigms in handwritten word recognition”, IEEE

Transactions Pattern Analysis and Machine Intelligence, IEEE Computer Society Press, 23(2): 149-164 (2001)

Lexicon

Interactive Features

PJ. Park, V. Govindaraju, and S. Srihari,: OCR in a Hierarchical Feature Space. IEEE Trans. Pattern Anal.

Mach. Intell. 22(4): 400-407 (2000)

24

Lexicon

Multiresolution features

A*

Lexicon

Lookup table: 29 x 10 x 85 (quad tree, 4 levels)

Interactive Lexicons

a. New Delhi b. Kolkattac. Patna d. Dehradun

Lexicon

Length

Short (<5 chars)

Long (>8 chars)

Ascenders / Descenders

Beginning (< 3 chars)

End (> 6 chars)

Parts

1

>1

Holes / Loops

0

>0

i-dots / t-crossings

IMPACT

1995-2000 Today

27

Lexicon

Lex size %

10 96

100 91

1000 (Top 50) 80 (98)

20000 (Top 100) 62 (94)

Postal Encoding 40

Lex size %

10 ~99

100 ~99

1000 ~95

Postal Encoding 95

FUSION

2Fusion

Sergey Tulyakov, Venu Govindaraju: Use of Identification Trial Statistics for the Combination of

Biometric Matchers. IEEE Transactions on Information Forensics and Security 3(4):719-733 (2008)

29

Fusion

Serial Fusion

Lexicon : NRecognizer

C1

Top x%

Recognizer

Cn

Top y%Recognizer

C2

• How does one determine x, y ?

• How does one determine the ordering of C1, C2, … Cn

• Is there a difference based on whether it is an Identification task or

Verification task?

• Can it be a generalized principled approach?

30

Fusion

Lexicon Density

V. Govindaraju, P. Slavik*, and H. Xue, “Lexicon density as a measure for performance evaluation of handwritten

recognizers”, IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Computer Society Press,

24(6): 789-800 (2002)

• Lexicon size as feature

• Different lexicons with the

same size have inconsistent

results

• Lexicon density as metric

• Recognizer specific

• Performance is consistent

with the same level of density

R. Milewski, A. Bharadwaj, and and V. Govindaraju, “Automatic Recognition of Handwritten Medical Forms for

Search Engines”, International Journal of Document Analysis and Recognition, Springer, 2009.

Bootstrapping for Large Lexicons

Fusion

Pruning Lexicons

Fusion

CLT to RLT CL to RL CLT to ALT

HR 7.48% 7.42% 17.58%

Error Rate 10.78% 10.88% 24.53%

Map: M matchers x N classes scores to N combined scores

Principled Approach

Lex N

Lex 1C1

Ci

CN

Lex 2

:

:

:

:

:

:

:

:

1

Ns

1

1s

1

is

:

:

j

Ns

js1

:

: j

is

M

Ns

Ms1

:

:Mis

NS

1S

:

:iSf

Verification

Verify if combined score of

hypothesis >θ

),,,( 21 M

iiii sssfS

Identification

Class of maximum combined

score iNi

S,...,1

maxarg

Fusion

Lex 1 Lex i Lex N

Recognizer C1

Sergey Tulyakov, Venu Govindaraju: Use of Identification Trial Statistics for the Combination of Biometric

Matchers. IEEE Transactions on Information Forensics and Security 3(4):719-733 (2008)

34Fusion

Score Matrix

Recognizer C1

Recognizer CM

Optimal Methods

Fusion

trialsj

kk

j

kk

j

Csp

Csp

)|(

)|(

Likelihood Ratio

Vi fS

Ni fS

Reject:Accept?i

S

NiiS

,...,1maxarg

? Vi fS

?Ni fS

C1 C2 C1 ^ C2 C1 v C2 LR Weighted

sum

NN

54.8 77.2 48.9 83.0 68.8 81.6 81.7

A B C …

.95 .89 .76 …

A B C …

.80 .54 .43 …

trialsj

kk

j

k

j

kk

j

k

j

k Ctsp

CtspC

)|,(

)|,(maxarg

Sergey Tulyakov, Chaohong Wu, Venu Govindaraju: On the Difference between Optimal Combination

Functions for Verification and Identification Systems. IJPRAI 24(2): 173-191 (2010)

Fusion

IMPACTPractical

• Dependent Scores

• Independent Matchers

37

IMPACTTheoretical

Fusion

)}({ , ji

j

iii sfS

Low

Medium I

Medium II

High

)(MC f

)(MNC f

)(NMC f

)(NMNC f

INDEXING AND RETRIEVAL

3Retrieval

Retrieval

Historical Documents

Catalin I. Tomai, Bin Zhang, Venu Govindaraju: Transcript mapping for historic handwritten document

images. IWFHR 2002: 413-418

Transcript mapping

Retrieval

1755: p.12, p.22, …

…

Express: p.12, p.14, ..

…

May: p12, p.45, ..

Word Spotting

Retrieval

42

Retrieval

• Imperfect Segmentation

• Multiple Writers

• Use recognition scores

)(#)|()|(ˆ,

it

w

ji wPcwPqefr

Impact

Lexicon Controlled Historical Medical Postal Checks

10K 1K 10K 1K 4K 100 40

Top 1 57 67 12 28 20 95 99

Top 10 74 75 32 72 42 99 99

CAPTCHAS , ACCENTS AND

SECURITY

4 Security

Security

New Captchas

Security

𝜑𝑖 𝑡 = 𝜃𝑠𝑖 +𝜃𝑒𝑖 − 𝜃𝑠𝑖

𝐷𝑖න0

𝑡

𝑣𝑡𝑖 𝜏 𝑑𝜏ith component trajectory :

0 5 10 15 20 25

0

2

4

6

8

10

12

14

x (mm)

y (

mm

)

Neuromotor Model

Security

Chetan Ramaiah, Réjean Plamondonm Venu Govindaraju: A Sigma-Lognormal Model for Handwritten Text

CAPTCHA Generation. ICPR 2014: 250-255

47

Accents

Security

IMPACT

48

Security

Dynamic lexiconsand integrated segmentation

Interactive lexiconsand multiresolution

Serial fusion and lexicon density

Principled fusion methods

Indexing and Spotting

Text retrieval

CAPTCHAs and security

Accents, signatures, and

security

Personal archives

Flipped classrooms

Marianne Craig Moore was an American Modernist poet, critic, translator, and

editor. Her poetry is noted for formal innovation, precise diction, irony, and wit

Transform based method works

for both gray scale and binary

images

MultilingualMADCAT

Neighboring patches likely to have the same class label

),(),()j(i,

iioi

jih yxxx

dependency Observ : ; similarity Nbd : oh

Hand vs Machine vs Graphics

),(),(1),( jien

xxDjiD

jih eexx

: dominant gap between words

. dominant gap between linesx

y

)),(/(1),( iim yxD

iio eyx

Distance in Spatial Space (Dn ) and Feature Space (De:)

2

2

,

2

2

,

2

)(

2

)(),(

y

ydy

x

xdxjiD

jiji

n

Dm: Distance in Feature Space

MRF

Flipped Class

Summary

Thank You venu@cubs.buffalo.edu