Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir...

75
Validating the use of Validating the use of Handwriting as a Handwriting as a Biometric and it’s Biometric and it’s Forensic Analysis Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang Technological University

Transcript of Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir...

Page 1: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

Validating the use of Validating the use of Handwriting as a Biometric Handwriting as a Biometric and it’s Forensic Analysisand it’s Forensic Analysis

Graham Leedham & Vladimir Pervouchine

C2i, School of Computer Engineering

Nanyang Technological University

Singapore

Page 2: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

2

Structure of this TalkStructure of this Talk

1. Brief History of Handwriting

2. A look at the Variability of Handwriting

3. A look at Forensic Document Analysis

4. Computer Tools to assist FDE’s

5. Is handwriting an accurate biometric?

6. Study of the effectiveness of features used by FDE’s

Page 3: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

3

History of HandwritingHistory of Handwriting 25000 years ago Cave paintings are the oldest pictures ever found.

Many were made more than 25,000 years ago by 'stone age' cave dwellers using sticks, sharp stones or their fingers. For 'paint' they used charcoal, coloured earth and vegetable dyes.

Early man could not write, so to remember things or leave messages he drew pictures on cave walls, rocks, bones or wet clay. Gradually, over the years, pictures became symbols, and then letters to form alphabets of signs to represent sounds.

Page 4: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

4

History of HandwritingHistory of Handwriting SUMERIAN CUNEIFORM Some of the earliest examples of a writing

system come from the Sumerian people who lived in the Middle East between 4000 and 6000 years ago.

'Cuneiform' means 'wedge-shaped' because the inscriptions were made by pressing the triangular tip of a reed or a stick (stylus) into wet clay tablets. The wedge marks were combined into signs representing objects and ideas. At first there were over 2000 different signs, but the Sumerians gradually reduced their 'alphabet' to about 600 symbols.

It was also at about this time that Chinese characters began to emerge independently in China with symbols being written on bone and shells. They too were originally pictures and symbols to represent ideas and objects. These were the earliest forms of writing. And subsequently had its own history of development.

Page 5: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

5

History of HandwritingHistory of Handwriting

4000 years ago EGYPTIAN HIEROGLYPHICS While cuneiform was spreading

throughout Mesopotamia, a different writing system was being developed in nearby Egypt. From about 5000 years ago the Egyptians used a form of stylised picture writing called hieroglyphics. ('Writing of the Gods'.)

Page 6: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

6

History of HandwritingHistory of Handwriting 2500 years ago GREEK About 2500 years ago, the ancient

Greeks were using an alphabet very much like our own. In fact, the word 'alphabet' comes from the first and second Greek letters, 'alpha' and 'beta'.

The Greek alphabet was developed from the Phoenician writing system. The Phoenicians were great sailors and merchants who traded with many countries in the Mediterranean, taking their writing with them. But the Greeks added signs for vowels because the Phoenician alphabet contained only consonants.

Page 7: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

7

History of HandwritingHistory of Handwriting

2000 years ago ROMAN When the Romans conquered Greece

just over 2000 years ago they took over the Greek alphabet and altered the shape of many letters. The letters of the English alphabet come directly from the Roman alphabet, although we have added three extra letters: J, U and W.

Page 8: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

8

History of HandwritingHistory of Handwriting

800 years ago GOTHIC After the Roman Empire collapsed in the

5th century AD, it was mainly monks who kept up the art of writing. Soon every monastery had its own scriptorium where manuscripts were copied, decorated and bound into books.

Page 9: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

9

The Magna Carta written in 1215The Magna Carta written in 1215(written in Latin)(written in Latin)

Page 10: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

10

History of History of HandwritingHandwriting

500 years ago ITALIC In the 15th century a group of Italian

scholars working in Florence decided that Gothic was difficult to read so they developed a new script. This style soon became popular all over Europe. Even today we still call styles like this 'Italic' because they came from Italy.

Page 11: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

11

History of HandwritingHistory of Handwriting

300 years ago COPPERPLATE The 'Copperplate' style of handwriting was taught

by writing masters from the 17th century onwards. Sometimes known as the English running hand, this neat easy-to-read script was also easy to write. Word after word could be produced without lifting the pen between letters. Until the invention of the typewriter it was widely used for business records and legal documents.

Page 12: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

12

History of HandwritingHistory of Handwriting

Several handwriting styles are now taught in schools around the world. There is less variation in style.

Page 13: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

13

Variability of HandwritingVariability of Handwriting

Individual

styles

develop

Page 14: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

14

Variation Variation of the of the word word “the” “the” written by written by 8 different 8 different writers.writers. Source: Harrison, 1981 Source: Harrison, 1981

Page 15: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

15

Variation of the letters “G” and “R” written by 15 different writers.

Source: Harrison, 1981

Page 16: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

16

Example of variation in letter formation styles in 10 letters from 9 different writers.

Source: Harrison, 1981

Page 17: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

17

Handwriting can be produced by Handwriting can be produced by many different writing instrumentsmany different writing instruments

Page 18: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

18

Handwriting Handwriting has been used has been used as a legal or as a legal or official seal for official seal for centuriescenturies

“Set your hand to the document.”

“Make your mark.”

Page 19: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

19

Forgery / Disguise / AlterationForgery / Disguise / Alteration

(i) Is the writing FORGED? (the author is not who he claims to be and is attempting to assert the writing is the same as someone else’s) or

(ii) Is the writing DISGUISED? (the author wishes to deny doing the writing at a later date) or

(iii) Is the writing ALTERED? (Has someone modified or altered the original document?)

Page 20: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

20

Hierarchy of Handwriting Hierarchy of Handwriting Recognition ProblemsRecognition Problems

Automatic processing of handwritten documents

Recognition formachine transcription

Mathematical formulaePrinted charactersNumeralsAlphabetic charactersSymbols

Cursive scriptWhole wordsSeparate characters

Writing analysis forAuthentication

Signature verificationWriter identificationForgery identificationDisguised writing

(OFF-LINE & ON-LINE)

Page 21: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

21

Current Methods Current Methods used by Forensic used by Forensic Document ExaminersDocument Examiners

Primarily involves manual extraction and comparison of various global and local visible features.

They are usually doing a comparison test between a “Questioned Document” and a set of “Known Documents”.

The objective is to determine whether the “Questioned Document” was, or was not, written by a particular individual.

The “Questioned Document” may be in disguised handwriting.

Page 22: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

22

Global features: Handwriting size, word spacing, line spacing, arrangement of words, margin patterns, baseline patterns, line quality, spelling, grammar …

Local features: character size, height-width ratios of characters, Size and shape of loops, letter slant, letter design, letter spacing,

writing pressure, speed variation, t-crossings and i-dots, hooking, punctuation

marks ...

See texts: Harrison W.R. (1981), Suspect Documents, their Scientific Examinations, Nelson-Hall Inc., Illinois. Hilton O. (1993), Scientific Examination of Questioned Documents, CRC Press, Florida.

FOR MORE INFO...

Current Methods used by Current Methods used by Forensic Document ExaminersForensic Document Examiners

Page 23: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

23

Current Methods used by Forensic Current Methods used by Forensic Document ExaminersDocument Examiners

Hidden writing left as pressure indentations on sheets below the one written on are recovered using ElectroStatic Detection Apparatus (ESDA) equipment.

Page 24: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

24

FISH - Forensic Information System for Handwriting - used by the bundeskriminalamant, Germany to maintain a database of known and unknown writers. A handwriting sample is characterised by interactive extraction of several global and local features to create a database of handwriting which can be indexed to locate similar handwriting to that in a Questioned Document.

Current Methods used by Current Methods used by Forensic Document ExaminersForensic Document Examiners

Page 25: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

25

Current Methods used by Forensic Current Methods used by Forensic Document ExaminersDocument Examiners Tick sheets, or comparison charts, (used in the UK

and other countries) are created showing side-by-side comparison of known and questioned words or characters. The comparison is subjective and based on local and global features. The degree of similarity is graded on a five point scale –

1. Was written by…., 2. High probability it was written by....,3. Probable/could well have been written by…., 4. No evidence, 5. Inconclusive.

Page 26: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

26

Example of a manually produced comparison chart to show thatthese writings were produced by different writers:

Source: Harrison, 1981

Page 27: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

27

Forgery / Disguise / AlterationForgery / Disguise / Alteration

(i) Is the writing FORGED? (the author is not who he claims to be and is attempting to assert the writing is the same as someone else’s) or

(ii) Is the writing DISGUISED? (the author wishes to deny doing the writing at a later date) or

(iii) Is the writing ALTERED? (Has someone modified or altered the original document?)

Page 28: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

28

Example of a manually producedcomparison chart toshow disguised handwriting.

Source: Harrison, 1981

Page 29: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

29

Forgery / Disguise / AlterationForgery / Disguise / Alteration

(i) Is the writing FORGED? (the author is not who he claims to be and is attempting to assert the writing is the same as someone else’s) or

(ii) Is the writing DISGUISED? (the author wishes to deny doing the writing at a later date) or

(iii) Is the writing ALTERED? (Has someone modified or altered the original document?)

Page 30: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

30

EARLY COMPUTER TOOLS TO ASSIST DOCUMENT EXAMINERS:

The early computer tools were predominantly in the use of the COMPUTER IMAGE PROCESSING and image handling techniques.

e.g. Behenen and Nelson, 1992.

And in the ENHANCEMENT of poor images as encountered in ESDA lifts or provide alternative methods to view the hidden writing.

e.g. Baier et al., 1987

As well as a number of attempts at WRITER IDENTIFICATION

e.g. Kuckuck et al., 1979

Page 31: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

31

Our Earlier Research attempted to Our Earlier Research attempted to ease the workload of the FDEease the workload of the FDE

1. Oct 1993 - Dec 1994: (FODES)Holcombe G., An experimental image processing environment for the forensic examination of questioned documents, MSc Dissertation, University of Essex, 1995.

2. Oct 1994 - Dec 1997: (WIS)Greening C., Automatic writer identification for forensic document analysis, PhD Dissertation, University of Essex, 1998.

3. May 1996 - April 1999: (FOX)Applied Research Fund Project (RG25/95) “Image Analysis Tools for Authentication and Enhanced Classification of Handwritten Script Using Forensic Techniques” carried out at Nanyang Technological University.

Objective: to investigate the possible use of computer basedimage handling tools to assist a document examiner in the analysis of a handwritten document and preparation of the evidence.

Page 32: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

32

Achievements:Prototype system comprising tools to:1. Scan and view images, segment text from background.2. Various automatic and interactive tools to process the handwritten documents: line, word and character segmentation, chart generation, slant variation, loop, ascender and descender feature extraction, animated word, character or signature visualisation, stroke sequence extraction ….

Previous Research at NTU Previous Research at NTU (& Essex University)(& Essex University)

Page 33: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

33

1. Handwriting Extraction :• Background removal• Shadow noise removal• Salt and pepper noise removal

3. Feature Extraction:• Baseline patterns and angle• Word slant• Average stroke width• Height of main body, ascenders• Depth of descenders• Loop features: area, slant, circle dissimilarity index

4. Simulation :• Ruler guided writing• Slant manipulation

2. Line, Word, Character Segmentation:• Text is available

scanned image

noise freeimage

variousfeatures

simulatedimages

TOOLS TO ASSIST DOCUMENT EXAMINERS:

Purpose: To enable document examiners to produce display charts and physical measurements.

Work at Essex University and Nanyang Technological University: Leedham, Sagar, Solihin, Holcombe, Chong, Greening (1993-2000)

FOX & WIS systems

Page 34: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

Previous Research Results Achieved

• Supplementary software for visual comparison of letter formation has been implemented.• A set of rule-based algorithms have been developed for feature extraction.

Screenshot of visualizationScreenshot of visualization softwaresoftware

Page 35: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

35

TOOLS TO ASSIST DOCUMENT EXAMINERS:

One of the first systems to be installed to provide database matching of handwriting features for individuals was the FISH system introduced by the German law enforcement agency (Bundeskriminalamt) which semi-automatically extracts features to characterise handwriting and store them in a database. (Shown at the 5th IGS, 1991, Tempe, by Manfred Hecker)

Work in Germany & Netherlands:BKA, Schomaker, Franke, de Jong, et al. 1994 -

A recent research consortium has sought to extend this work in the

WANDA system (see http://www.ai.rug.nl/alice/wanda/)

Page 36: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

36

TOOLS TO ASSIST DOCUMENT EXAMINERS:

Work in Australia:Found, Rogers, Sita et al. 1995 -

Investigating handwritten signature complexity and tools to assist document examiners. Providing objective means for the subjective techniques employed by document examiners.

Other work:

There are numerous other researchers currently working on writer identification and tools to identify forged handwriting:

Eg. Bensefia et al. from Rouen University, FranceFairhurst et al. from Kent University, UKCha & Tappert from Pace University, USAUeda et al., Nara College of Technology, Japan

Page 37: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

37

So what is all this about???

People have been doing HANDWRITING for 1000’s of years,

Crime involving FORGED, DISGUISED or ALTERED handwriting is common in cheques, anonymous letters, wills and other examples where deception can lead to illegal gain or advantage.

FORENSIC DOCUMENT EXAMINERS have been authenticating handwriting in one form or another for more than 100 years.

Many TEXT BOOKS have been written describing the analysis techniques and providing CASE STUDIES.

Today all LAW ENFORCEMENTS AGENCIES practice Questioned Document Examination and employ Forensic or Questioned Document Examiners along with other Forensic Scientists.

There exist a number of PROFESSIONAL ORGANISATIONS around the world which foster professional training and accreditation to forensic Document Examination.

What is the problem?

Page 38: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

38

THE PROBLEM IS…..

Other branches of Forensic Science such as DNA analysis, blood, fibre and soil analysis are supported by a wealth of CHEMICAL, BIOLOGICAL and PHYSICAL KNOWLEDGE obtained from years of scientific research and published in learned journals.

Forensic Document Examiners do not have any similar SCIENTIFIC BASIS to support their EXPERT OPINION.

The current acceptability of Forensic Document Examiners expert opinion is based on the CREDIBILITY, EXPERIENCE and STANDING of the FDE. They have little scientific evidence to support their opinion. Their judgement is subjective and not an exact science.

Recent legal challenges (eg Daubert, 1993 and Starzecypzel, 1995) have brought this lack of scientific support into the limelight.

Page 39: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

39

OBJECTIVES……

During the past 20 years there has been sporadic interest in the application of computer systems to ASSIST AND SUPPORT the work of Forensic Document Examiners.

In this rest of this presentation some of our current research work is presented which:

1. Provides computer support to the analysis methods used by document examiners

2. Seeks scientific evidence to support the analysis methods used by forensic document examiners.

Whilst much of the research carried out in handwriting processing and recognition provides considerable scientific knowledge about the education of writing styles, cognitive processes and motor-control activities in the production and recognition of handwriting, only a limited amount of the work is directly applied to the work of the FDE.

Page 40: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

40

Research Motivation (restating)Research Motivation (restating)Most of the techniques employed by document examiners are based on

standard practices and previous experience. The Forensic Document Examiner as an Expert Witness is highly reliant on their personal standing and credibility.

There is very little scientific justification for many of their practices and procedures. The only truly scientific examination is the chemical analysis and dating of ink and paper.

Several court rulings have brought into question the scientific basis of the expert document examiners testimony.

Eg United States vs Starzecpyzel, 1995

Page 41: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

41

Is handwriting a biometric?Is handwriting a biometric?

The argument is that - because handwriting is a practiced ballistic skill, it represents some uniqueness to the individual which can be used to identify the individual.

However skilled forgery is widespread, and individual variability due to health, stress writing implement, writing stance etc… is significant. Where does the genuine end and the forgery begin?

Page 42: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

42

Scientific basis for forensic document examination

CURRENT APPROACHESCURRENT APPROACHES

APPROACH 1. Is handwriting unique?

Feature extraction:Proves it is a biometric. Mainly computational features + some document examiner features

APPROACH 3. Is it possible to verify the effectiveness of features employed by forensic document examiners?

Feature extraction:Justifies the use of features/methods used by Document Examiners.

APPROACH 2. Are professional forensic document examiners better than other people at handwriting examination?

Justifies the need for tools to assist forensic document examiners in what they do.

Page 43: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

43

APPROACH 1: IS HANDWRITING UNIQUE?

Two tests for establishing error ratesTwo tests for establishing error rates

IdentificationAlgorithm

Handwriting Sample

Handwriting Sample 1

Handwriting Sample 2

Writer 1

Writer n

Same Writer

Different Writer

(a)

VerificationAlgorithm

(b)

Work on individuality at CEDAR (Srihari et al. Journal of Forensic Sciences, 2002)

Establish discriminatory power of handwritingUse objective methods

Algorithms suitable for software implementationRelate methods to FDE (forensic document examination) procedures

Rigorous testing, establish error-ratesPeer-review

Page 44: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

44

APPROACH 1: IS HANDWRITING UNIQUE?

POVIDING TOOLS TO ASSIST DOCUMENT EXAMINERS:

CEDAR-FOX toolset developed at CEDAR

Page 45: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

45

APPROACH 2: ARE PROFESSIONAL FORENSIC DOCUMENT EXAMINERS BETTER THAN OTHER PEOPLE IN HANDWRITING EXAMINATION?

Work at Drexel University, USA: Kam et al (1994- )

andWork at LaTrobe University, Australia:Found, Rogers et al (1994-)

Both groups performed a number of experiments comparing professional FDE’s and lay people.

They concluded that “Professional document examiners DO possess writer-identification skills absent in the general population.”

Page 46: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

46

APPROACH 3: IS IT POSSIBLE TO VERIFY THE EFFECTIVENESS OF FEATURES USED BY FORENSIC DOCUMENT EXAMINERS?

Handwriting has been shown to be discriminative when identifying whether an unknown writer is one a group of N known writers and verifying whether two documents are from the same or different writers. (Srihari et al) Document examiners have been proven to be better than lay people at writer identification/verification, including forgery identification. (Kam et al & Found, Rogers et al)

The methods employed by document examiners are well documented in various texts.

Page 47: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

47

The comparison methods used by FDE’s are frequently QUALITATIVE / SUBJECTIVE and the resulting evidence also qualitative / subjective.

QUANTITATIVE / OBJECTIVE ANALYSIS of the methods used by FDE’s is necessary to determine techniques and methods which can be supported by scientific proof.

APPROACH 3: IS IT POSSIBLE TO VERIFY THE EFFECTIVENESS OF FEATURES USED BY FORENSIC DOCUMENT EXAMINERS?

This is what we are trying to do.

Page 48: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

48

APPROACH 3: OBJECTIVES OF THE STUDYAPPROACH 3: OBJECTIVES OF THE STUDY

1. To develop a system to automatically extract structural features from individual handwritten characters.

2. To assess the uniqueness and individuality of the structural or visually observable features used by Forensic Document Examiners

3. To determine whether the features are unique to a writer and can be used for author identification

Page 49: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

49

SELECTION OF CHARACTERS TO STUDYSELECTION OF CHARACTERS TO STUDY

Cannot examine all characters as feature sets must be individually tailored and algorithms written:

Choose frequently occurring characters

Characters must potentially possess writer-specific features. (Capital letters as well as letters that consist of several strokes, like those with ascenders or descenders, bear more individual information than simple characters like “i” or “c”.)

Robust automatic feature extraction of the writer-specific features must be achievable

Based on an analysis of 220,000 words from three novels, the characters

chosen for analysis were ‘d’, ‘y’, ‘f’ and grapheme ‘th’.

The three letters and one grapheme represent >12% of typical script.

Page 50: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

50

Frequencies of letters and graphemes Frequencies of letters and graphemes in English textsin English texts

Letters / graphemes should:

• be frequent

• have ascenders / descenders

• not have too simple shape

Page 51: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

51

Features of “d”Features of “d”

1. Height

2. Width

3. Height to width ratio

4. Relative height of ascender

5. Slant of ascender

6. Final stroke angle

7. Fissure angleIn this study we concentrate on structural micro-features extracted from characters and graphemes. These are a subset of the features used by forensic document examiners.

Page 52: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

52

Example of a manually produced comparison chart to show thatthese writings were produced by different writers:

Source: Harrison, 1981

Page 53: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

53

Features of “y”Features of “y”

8. Height

9. Width

10.Height to width ratio

11.Relative height of descender

12.Descender loop completeness

13.Descender slant

14.Final stroke angle

15.Slant at point

TY

Page 54: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

54

Features of “f”Features of “f”

16.Height

17.Width

18.Height to width ratio

19.Presence of loop at FT

20.Presence of loop at FB

21.Slant

Page 55: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

55

Features of “th’Features of “th’ 22.Height

23.Width

24.Height to widht ratio

25.Distance HC

26.Distance TC

27.Distance TH

28.Angle between TH and TC

29.Slant of t-stem

30.Slant of h-stem

31.Position of t-bar

Page 56: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

56

Data used for evaluation :Data used for evaluation : Individual characters were extracted manually from 600 samples of

the CEDAR letter representing data from 200 writers.

To decrease variation of a character form caused by the preceding and the following characters, samples of characters “d” and “y” were extracted only from the end of words, and samples of grapheme “th” were extracted only from the beginning of words.

All samples of character “f” were extracted because there were only 8 occurrences of this character in the letter.

There were at most 10 samples of character “d”, 8 samples of “y”, 8 samples of “f”, and 9 samples of “th” extracted from each of the 600 documents.

Letter “d” “y” “f” “th”

Samples per writer 30 24 24 27

Page 57: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

57

Automatic Feature Extraction :Automatic Feature Extraction : The algorithms used two version of an image: a

binarised image of a character and its skeleton, which was obtained by thinning the binarised image.

After all images had been processed by feature extraction programs, the feature values were verified manually for each image. An image was excluded from the set of patterns used in the later study if any of the features was extracted incorrectly.

Correct Feature Extraction

f d y th

92% 85% 85% 87%

Page 58: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

58

We assume that an FDE canWe assume that an FDE can : :(i) effectively utilise more subtle features than our system

does, for it is very hard to express in strict mathematical terms many of the document examiner features, and

(ii) the person can determine which features should, and which should not, be used in a particular case (that is, having looked at handwritten samples, an expert is able to select only the important features for handwriting comparison).

As a consequence, an expert FDE should be able to distinguish writers or establish authorship of questioned documents better, on average, than our system.

Our research is thus aimed at determining a lower bound on the accuracy of writer discrimination

Page 59: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

59

Feature RelevanceFeature RelevanceWe want to place each feature into one of three classes according to the results of feature subset selection experiments:

1. Indispensable - it was selected in each optimal feature set

2. Partially relevant - selected in some of the optimal feature sets

3. Irrelevant features - not selected in any optimal feature set

This provides information about the features which should always be chosen, which features can be substituted by others (only some of this subset need to be chosen), and features which should be excluded.

Page 60: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

60

Evaluation of the Feature SetEvaluation of the Feature Set

An induction algorithm was used to evaluate a feature set. We used n-fold cross validation, called Distal, to evaluate a

feature subset (Yang et al., 1997) The training data was divided into n approximately equal

partitions and the induction algorithm was then run n times each time leaving one subset for test and using the other n − 1 parts for training.

The classification accuracy obtained from n tests was then averaged and associated with the corresponding feature subset.

The wrapper approach was used to find all the best feature subsets (John et al., 1994)

Page 61: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

61

Evaluation of the feature setEvaluation of the feature set

A binary 31-bit string[0 1 1 1 0 0 … 0 1]

The original feature set[ f1 f2 f3 f4 f5 f6 … f30 f31 ]

Feature set to evaluate[ f2 f3 f4 f31 ]

DistalN-fold cross validation

ANN

Accuracy value for the feature set

Page 62: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

62

Classifier used to Evaluate the Classifier used to Evaluate the Feature SetFeature Set All feature values were treated as real numbers. The

normalised Manhattan distance was used as the distance measure for DistAl

where k is the number of features (31), mini and maxi are the minimum and maximum values of the ith feature in the data set respectively.

k

i ii

ii FF

kFFd

1

,2,1

21 minmax

1,

Page 63: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

63

Search for the best feature setsSearch for the best feature sets

Genetic Algorithm

Evolution of population of

strings

Evaluation of accuracy

(string fitness) for each string by

n-fold cross validationArray of fitness values

Next generation of strings

Best strings found

Initial randomly generated strings

An exhaustive search is not feasible)

Page 64: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

64

Search for the best feature setsSearch for the best feature sets

For some writers the amount of patterns obtained for one of the four characters was too small because of errors in the feature extraction stage.

To make the results of experiments comparable for all single characters and the four-character set, patterns from 165 different writers were used (writers who had less than 15 patterns for any of the character due to errors in feature extraction stage were excluded from the study).

Page 65: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

65

ResultsResults

Character “d” “y” “f” “th” “dyf th”

Accuracy, % 16 20 26 36 58

Classification Accuracy for 165 writers

Results for single character performance are in agreement withthose of Srihari et al,, 2002 even using a different feature set.

Page 66: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

66

ResultsResults

Feature for the grapheme “th” consititue the greatest number of indispensible features. (8 out of 13).

Four features were irrelevant (angles in “d” and loops in “f”).

Indispensable features, fi

Partially relevant features, fi

Irrelevant features, fi

1, 2, 11, 16, 18, 22, 25, 26, 27, 28, 29, 30, 31

3, 4, 5, 8, 9, 10, 12, 13, 14, 15, 17, 21, 23,

24

6, 7, 19, 20

Page 67: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

67

ConclusionsConclusions

Many of the features FDE’s use contain discriminatory power.

Genetic Algorithms are again demonstrated to be effective at identifying indispensable and dispensable features.

Analysis of graphemes is more accurate than individual characters (also noted by FDE methods).

These results are only valid for identifying or verifying genuine handwriting samples. The detection of forgery or disguise is unlikely to be successful with this method.

Page 68: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

68

ConclusionsConclusions

Only a small set of characters and one grapheme were analysed. It is not possible to draw absolute conclusions based on such limited analysis.

We CAN say that more formalisation of document examiner features and analysis on a wider range of letters and graphemes is likely to provide a solid scientific basis for the techniques and features currently used by FDE (and also enable useful tools to be developed to enhance their exmination performance).

Page 69: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

69

FUTURE WORKFUTURE WORK To improve the accuracy of feature extraction:

- A more effective algorithm should be found for letter image skeletonization

- Some of the angular feature measurements need to be reconsidered and the accuracy improved

- Important features, and particularly loop features, need to be further defined for measurement of different feature values corresponding to different kinds of loops

The analysis must be extended to include more writers and more examples of each letter before a conclusive result can be obtained.

Page 70: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

70

ADDENDUM: Intelligent ADDENDUM: Intelligent Skeletonisation and Penstroke Skeletonisation and Penstroke Direction and Sequence RecoveryDirection and Sequence Recovery

Feint line of g’s

descender

People are particularlygood at determining pen-stroke directionand even thefluency of the writing

Page 71: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

71

Intelligent Skeletonisation and Penstroke Intelligent Skeletonisation and Penstroke Direction and Sequence RecoveryDirection and Sequence Recovery

The imperfect skeleton produced by a popular skeletonisation algorithm

An idealised skeleton of the letter also showing pen-down and pen-up points

We need to integrate the subtleties of human visual perception with knowledge of forensic document examination.

Page 72: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

A New Skeletonisation schemeA New Skeletonisation scheme

Purpose: approximate pen trajectory so that individual features of handwriting are retained.

Approach: three stages.– Vectorisation: initial skeletal branches are formed.– Stroke formation: the branches are merged into strokes

and hidden loops are recovered.– Adjustment of skeleton: spline knot positions are

adjusted.

Page 73: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

Some experimental resultsSome experimental results

Page 74: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

ObservationsObservations A new method of skeletonisation was developed which

preserves structural features of handwriting The method allows extraction of structural features more

accurately than was possible with original thinning and postprocessing

The method allows the extraction of additional structural features that could not be reliably extracted previously

Writer identification has been demonstrated to be improved.

More details to appear in

Vladimir Pervouchine, Graham Leedham, and Konstantin Melikhov, Handwritten character skeletonisation for forensic document analysis, Accepted for presentation at the 20th Annual ACM Symposium on Applied Computing, Santa Fe, New Mexico, 13-17 March 2005.

Page 75: Validating the use of Handwriting as a Biometric and its Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang.

SUMMING UPSUMMING UP

Handwriting remains a mechanism for authorisation. Legal challenges to the authenticity of the handwriting will continue.

While handwriting has changed rapidly over the past few 100 years there is less stability now. (Some would argue that handwriting skills are degrading because of the widespread use of IT.)

Scientific support to processes and procedures practiced by FDE’s to provide that authenticity is beginning.

There remain many challenges. E.g. Detecting skilled forgery and performing verification in the presence of handwriting changes due to illness.