Error Rates and Random Match Probabilities (RMP) Based on

the RUGER 10-Barrel Test and the GLOCK Cartridge Case Tests

James E. Hamby, Ph.D., David J. Brundage, M.S., Steven A. Norris, B.S., Nicholas D.K.

Petraco, Ph.D., and James W. Thorpe, Ph.D.

Conclusions on Research Conducted over nearly 25 years involving the

RUGER 10-Barrel Test, GLOCK Cartridge Case Tests

Base error rates were established for both studies using advanced Bayesian

techniques

The Identification of Bullets Fired From 10 Consecutively Rifled RUGER 9mm Pistol Barrels

A Research Project Involving 626 Scientists From Labs In 24 Countries

The 10-Barrel Test

James E. Hamby, Ph.D.,International Forensic Science Laboratory & Training Centre, Indianapolis, Indiana

David J. Brundage, M.S.,Forensic Science Consultant, Nashville,Tennessee

James W. Thorpe, Ph.D.,University of Strathclyde, Glasgow, Scotland(Retired)

A total of 240 test sets were produced by Dave Brundage, Mickey French and Jim Hamby – all then employed at the Indianapolis-Marion County Forensic Laboratory, Indianapolis, IN

Recently 20 additional test sets were produced by Hamby and 10 sets sent to Europe and other countries for examination. The various test sets have used 4 different types of 9mm ammunition.

Of the 626 participants in this research project, nine have used some type of ‘ballistics’ imaging

equipment.

Of the 626 participants in this worldwide project,9,382 of the possible 9,390

unknown bullets were identified. (15 unknowns x 626 participants = 9,390)

Three bullets were mutilated during test firing (tank rash) and five ‘unknown’ bullets

could not be associated to the known bullets by two young examiners-in-training

- listed as inconclusive.

This worldwide research project and numerous other research projects reported

in the scientific literature – the AFTE Journal, the Journal of Forensic Sciences, and others - since the early 1930’s should provide data that will obviate some of the

concerns within the legal community -especially in the United States.

Request for Assistance

• We request that those members who haven’t participated ‘take’ the test in their laboratory & submit their answers!

• We also request that you return the set if you are no longer using it as there are labs that have requested test sets. Some labs may have two sets in their possession.

The Examination, Evaluation and Identification of 9mm Caliber Cartridge

Cases Fired from 1,632 GLOCK Semiautomatic Pistols Manufactured Over a 23 Year Period - Using Optical Microscopy, Confocal Microscopy and Computational Pattern Recognition

GLOCK Cartridge Case Test

This paper was presented at the 2013 AAFS Meeting by Petraco & Hamby with Norris as a co-author. The 1,632 fired cases

were examined and the original 617 fired cases evaluated using

the heritage IBIS.

1,632 GLOCK Cartridge Cases

13

This GLOCK research involved both IBIS and examiner based

results. (over 1,300,000 exams)

Nick Petraco will discuss the Error Rates/RMP for both research projects in a few minutes.

Modified SWGGUN Power Point Using the excellent SWGGUN power

point, I have made a few changes that work well when testifying especially in

Frye or Daubert Hearings. The following are a few of the changes. Our beloved AV man–Mike–has the power point if you are interested.

The results of a subjective examination are based on an individual’s opinion based on his/her training and experience. This does not mean that this type of examination is unreliable or unscientific.

There is subjectivity in every science and in every test, whether it is a doctor diagnosing a head cold, a physician examining an x-ray or a cytotechnologist examining slides for cancer cells (pap smears).

Definition: Subjective Examination

Grant or Accepted Government Teaching Programs for Firearms Identification

• National Institute of Justice, Washington, D.C.• NIJ - in conjunction with the National Forensic Science Technology Center

(NFSTC) – funded a distance learning program titled ‘Firearms Examiner Training Program’ to augment training for new examiners,

• Federal Bureau of Investigation (FBI) • Since 1986, the FBI has offered a one week course titled ‘Specialized

Techniques in Firearms Identification’. The course has been attended by several hundred firearms examiners.

• Bureau of Alcohol, Tobacco and Firearms (ATF)• Since 1999, the ATF has offered a one-year long course titled the ‘National

Firearms Examiner Academy. To date, 140+ firearms examiners have attended the NFEA. (Funded by ATF and NIJ)

• California Criminalistics Institute (CCI), CA DOJ• For several years, CCI has offered a variety of specialized courses in

firearms and toolmark related fields.

•

Grant or Accepted Government Teaching Programs for Firearms Identification

• United Nations Office on Drugs and Crime (UNODC)• In 2013, the UNODC – with funding from the Canadian Government – is

developing a limited service forensic laboratory for the Palestinian Civil Police (PCP) Temporary Forensic Science Laboratory (TFSL) in Ramallah, Palestine.. The TFSL will have the following forensic specialties:

• Forensic Chemistry – Drug and Arson Analysis• Questioned Documents Analysis• Firearms and Tool Mark Identification

• United States Department of State • In 2014, the US Department of State – through the Bureau of International

Narcotics and Law Enforcement Affairs, US Embassy, Belize, funded a program to strengthen the Belize National Forensic Science Laboratory by providing technical assistance related to the handling, processing and analysis of ballistics (firearms / tool marks) related evidence.

• Other programs such as ICITAP teach firearms and tool mark Identification around the world.

SummaryFirearm & Toolmark Identification meets the reliability standard put forth by both Daubert or Frye requirements. The science has been accepted in every state and Federal court that uses Daubert or Frye.

Examples: A recent (9-11) Indiana Supreme Court Decision (Frye) upheld the reliability of Firearms and Tool Mark Identification; and a recent (12-11) Kansas District Court Decision (Frye) upheld ‘general acceptance’ in the field. A recent (7-13) Florida Court

Issued a Daubert decision upholding the science. Cases are available for review on the SWGGUN website (under ARK) at www.swggun.org.

Data AnalysisAverage Examiner Error Rate and Random Match Probability

• The Good News:o 10-Barrel Test: # examiner errors committed = 0oGLOCK Cartridge Case Test: # of falsely

matched cartridge cases = 0

THIS SHOULD BE GREAT!We WANT error rates of 0% for good

classification systems!!

Data AnalysisAverage Examiner Error Rate and Random Match Probability

• The Bad News:o You have to deal with the U.S. court system

o0% error rates present an opportunity to muddy waters with the “unrealistic study” criticisms

oLittle court interest in understanding the technicalities of estimating small error rates.

Data AnalysisAverage Examiner Error Rate and Random Match Probability

• 0% error rate is the “frequentist” estimateo “Bayesian” statistics provide complementary

methodsoCan work much better in estimating small probabilitiesoWe looked to sports statistics for low scoring games

So what can we do?

Data AnalysisAverage Examiner Error Rate and Random Match Probability

• For 10-Barrel we need to estimate a small error rate

• For GLOCK we need to estimate a small random match probability (RMP)

• Use Bayesian “Beta-binomial” method when no “failures” are observed (Schuckers)

Data AnalysisBayesian Methodology

• Basic idea of the reverend Bayes:

Prior Knowledge × Data = Updated Knowledge

a + ba

Error Rate/RMP =

Posterior(a,b | data) U(a,b) × Beta-Binomial(data | a,b)

Get updated estimates of Error rate/RMP

Data AnalysisBayesian Methodology

• So given the observed data and assuming “prior ignorance” o Posterior error rate/RMP distributions:

Average Examiner Error Rate0.011%

[0.00023%, 0.040%]

RMP0.000086%

[0.0000020%, 0.00031%]

Posterior Dist. 10-Barrel Posterior Dist. GLOCK

Data AnalysisExtrapolations

# of Participants # Cartridge Cases Examined With No Errors in I.D.

Exa

min

er E

rror

Rat

e (%

)

RM

P (

%)

GLOCK10-Barrel

Actual: 626 participants

Actual: 1632 cartridge cases

Done