3 2 1 0 1 2 3

Current Research in Forensic Toolmark Analysis

Helping to satisfy the “new” needs of forensic scientists with state of the art microscopy,

computation and statistics

Outline• Introduction• Instruments for 3D toolmark analysis• 3D toolmark data• The statistics:

• Identification Error Rates• “Match” confidence• “Match” probability

• Statistics from available practitioner data

• All forms of physical evidence can be represented as numerical patternso Toolmark surfaceso Dust and soil categories and spectrao Hair/Fiber categories and spectrao Craniofacial landmarkso Triangulated fingerprint minutiae

• Machine learning trains a computer to recognize patterns o Can give “…the quantitative difference between an identification and

non-identification”Moran o Can yield identification error rate estimateso May be even confidence measures for I.D.s

Quantitative Criminalistics

Data Acquisition For Toolmarks

Comparison MicroscopeConfocal Microscope Focus Variation MicroscopeScanning Electron Microscope

2D profiles3D surfaces(interactive)

Screwdriver Striation Patterns in Lead

Bullet base, 9mm Ruger Barrel

Bullets

Bullet base, 9mm Glock Barrel

Close up: Land Engraved Areas

• Statistical pattern comparison!• Modern algorithms are called

machine learning• Idea is to measure

features that characterize physical evidence

• Train algorithm to recognize “major” differences between groups of featureswhile taking into account

natural variation and measurement error.

What can we do with all this microscope data?

• Visually explore: 3D PCA of 760 real and simulated mean profiles of primer shears from 24 Glocks:

• ~45% variance retained

Support Vector Machines• Support Vector Machines (SVM) determine

efficient association rules• In the absence of specific knowledge of probability

densities

SVM decision boundary

Refined bootstrapped I.D. error rate for primer shear striation patterns= 0.35% 95% C.I. = [0%, 0.83%]

(sample size = 720 real and simulated profiles)

18D PCA-SVM Primer Shear I.D. Model, 2000 Bootstrap Resamples

How good of a “match” is it?Conformal PredictionVovk

• Data should be IID but that’s it C

umul

ativ

e #

of E

rror

s

Sequence of Unk Obs Vects

80% confidence20% errorSlope = 0.2

95% confidence5% errorSlope = 0.05

99% confidence1% errorSlope = 0.01

• Can give a judge or jury an easy to understand measure of reliability of classification result

• This is an orthodox “frequentist”approach

• Roots in Algorithmic Information Theory

• Confidence on a scale of 0%-100%• Testable claim: Long run I.D. error-

rate should be the chosen significance level

Conformal Prediction

Theoretical (Long Run) Error Rate: 5%

Empirical Error Rate: 5.3%

14D PCA-SVM Decision Modelfor screwdriver striation patterns

• For 95%-CPT (PCA-SVM) confidence intervals will not contain the correct I.D. 5% of the time in the long run

• Straight-forward validation/explanation picture for court

• An I.D. is output for each questioned toolmark• This is a computer “match”

• What’s the probability the tool is truly the source of the toolmark?

• Similar problem in genomics for detecting disease from microarray data• They use data and Bayes’ theorem to get an

estimate

How good of a “match” is it?Efron Empirical Bayes’

JAGS MCMC Bayesian over-dispersed Poisson with intercept, on test set

A Bayesian Hierarchical Model: Believability Curve

Bayes Factors/Likelihood Ratios • In the “Forensic Bayesian Framework”, the Likelihood

Ratio is the measure of the weight of evidence.• LRs are called Bayes Factors by most statistician• LRs give the measure of support the “evidence” lends to

the “prosecution hypothesis” vs. the “defense hypothesis”• From Bayes Theorem:

Bayes Factors/Likelihood Ratios • Using the fit posteriors and priors we can obtain the likelihood ratiosTippett, Ramos

Known match LR values

Known non-match LR values

• Two large scale published studieso 10-Barrel TestHamby:

o 626 practitioners (24 countries)o 15 “unknowns” per test seto At least one bullet from each of the 10 consecutively manufactured barrels o # examiner errors committed = 0

o GLOCK Cartridge Case TestHamby: o 1632 9-mm Glock fired cartridge caseso 1 case per Glocko All cartridge cases pair-wise compared o # of pairs of cartridge cases judged to have enough surface detail

agreement to be (falsely) “matching” = 0o AFTE Theory of Identification standard used: www.swggun.org

Available Large Scale Practitioner Studies

• 0% error rate is the “frequentist” estimateoWe looked to sports statistics for low scoring

gameso“Bayesian” statistics provide complementary

methods for analysisoCan work much better in estimating small

probabilities

So does that mean the error rate is 0%?

Available Large Scale Practitioner Studies

• For 10-Barrel we need to estimate a small error rate

• For GLOCK we need to estimate a small random match probability (RMP)

• Use Bayesian “Beta-binomial” method when no “failures” are observed (Schuckers)

Available Large Scale Practitioner Studies

• Basic idea of the reverend Bayes:

Prior Knowledge × Data = Updated Knowledge

a + baError Rate/RMP =

Posterior(a,b | data) Uninf(a,b) × Beta-Binomial(data | a,b)

Get updated estimates of Error rate/RMP

Available Large Scale Practitioner Studies

• So given the observed data and assuming “prior ignorance” o Posterior error rate/RMP distributions:

Average Examiner Error Rate0.011%

[0.00023%, 0.040%]

RMP0.000086%

[0.0000020%, 0.00031%]

Posterior Dist. 10-Barrel Posterior Dist. GLOCK

Available Large Scale Practitioner Studies

Acknowledgements• Professor Chris Saunders (SDSU)• Professor Christophe Champod (Lausanne)• Alan Zheng (NIST)• Research Team:

• Dr. Martin Baiker• Ms. Helen Chan• Ms. Julie Cohen• Mr. Peter Diaczuk• Dr. Peter De Forest• Mr. Antonio Del Valle• Ms. Carol Gambino• Dr. James Hamby

• Ms. Alison Hartwell, Esq.• Dr. Thomas Kubic, Esq.• Ms. Loretta Kuo• Ms. Frani Kammerman• Dr. Brooke Kammrath• Mr. Chris Lucky• Off. Patrick McLaughlin• Dr. Linton Mohammed• Mr. Nicholas Petraco• Dr. Dale Purcel• Ms. Stephanie Pollut

• Dr. Peter Pizzola• Dr. Graham Rankin• Dr. Jacqueline Speir• Dr. Peter Shenkin• Ms. Rebecca Smith• Mr. Chris Singh• Mr. Peter Tytell• Ms. Elizabeth Willie• Ms. Melodie Yu• Dr. Peter Zoon