The Practice of Firearm/Toolmarks and a Reasonable Path ...€¦ · Firearm toolmarks examinations...

Copyright Cliff Spiegelman 1

The Practice of Firearm/Toolmarks and a Reasonable Path Forward

Cliff Spiegelman, Texas A&M UniversityCoauthor William A. Tobin


Three Parts Of The Talk

What are Firearm Toolmarks

What statements etc. does the scientific method allow?

Explanation of types of error with examples

Three families of experiments to start laying the

statistical foundation for firearm/toolmarks

Discussion


Firearm/toolmarks

Firearm toolmarks examinations and comparisons are often used in investigations of homicides involving a firearm, spent bullets and/or cartridge cases that are recovered from crime scenes. Most frequently, one or more firearms are recovered during investigation of a shooting incident and typically submitted for forensic comparisons with bullets and/or cartridge cases recovered from the scene. The forensic practice used to associate or eliminate a particular firearm as the murder weapon is based on comparisons of characteristics imparted to bullets and cartridge cases during cycling of a cartridge through the gun, and is known as firearm/toolmarks examination.


Typical testimony

If it is concluded that the submitted weapon “matches”the crime scene bullets, the firearm/toolmarks examiner typically testifies at trial that the crime scene bullets were fired from the gun to the exclusion of all other possible weapons, although sometimes “to a practical certainty.”

What Claims does the Scientific Method Support?

It supports claims that have been reasonably verified by hypotheses tests and corroborated by others. This means repeatable and reproducible and robust reproducible (in the case of broad claims) results

Repeatability: The same experimental set up gets the same resultsReproducibility: Similar setups that use different examiners at different locations get the same experiments resultsRobust Reproducibility: Widely varied experimental setups get the supporting results


What the scientific method does not allow

Without repeatability claims should be restricted to the (crime) lab and the original experiment. Nothing can be said about any other experiment.

Without reproducibility claims should be restricted to the (crime) labs participating in the experiment and experiments using the same factors and in the same way.

Without robust reproducibility claims cannot be extended beyond the specific factors used in the experiment.


Error According To The Apple Dictionary

Technical: a measure of the estimated difference

between the observed or calculated value of a

quantity and its true value.

Law: a mistake of fact or of law in a court's opinion,

judgment or order.

Phrases: see the error of one's ways realize or

acknowledge one's wrongdoing.


Two Categories Of Error Used In This Presentation

Systematic or sometimes referred to as biasBetter know examples:

Publication bias: publishing only positive resultsDifferent instrumentation as in different light sources (much more about this later)

Random errorBetter know examples:

Variation among examiners trained within the same school ('CMS' or classical)Variation in marks from the same firearm


More Bias Examples

Underreported, summary dismissal, or unreported false IDs

Missouri and other cases reported in “Gun evidence often wide of scientific mark” in the Austin American Statesman

Anthony Ray Hinton exonerated after 30 years on death row

Former Detroit firearm unit (~10% error rate)

Blame the lab or examiner rather than the process

Question? Any such cases in the FBI firearm/toolmark unit?

Present publications in trade journals as publications in science journals


AAS Article Highlighted By AFTE Website


More Bias Examples

Myopic view of resultsNot involve the broader scientific community in research investigations but keep them within 'house’

Myopic view of results spreads by publication bias etc..

Restrictions to visible light (more later)Not combining human with computer determinations Using tortured language to deny a practical error rate

‘match to a practical certainty’ or ‘practical impossibility of being wrong’


More on Bias

An example of publication bias on steroids

Have a trade organization claim it’s publishes a science journal and publish non statistically designed, not double blinded studies and other pseudo-experiments as broad based proof of efficacy.


More Random Error

Within bullet brand variation in charge and hence variation in striations

Within bullet brand variation in bullet hardness (antimony levels) and hence variation in striations

Random variations in light output

Locations of bullet damage from striking bone or buildings

Within bullet brand variation in cartridge hardness and hence variation in ejector marks


More Random Error

Within bullet brand variation in primer cup hardness and

hence variation in firing pin impressions

Cleanliness of weapon components

Condition of lubrication system, nested within lubrication

nature and regime.

……………………………etc..


Some Factors To Consider During Firearm/Toolmark Experiments

Factors not considered or discussed, and sample size issues:Ammunition typeAmmunition chargeCartridge case hardnessPrimer cup hardnessBreechface hardnessFiring pin hardnessDifferent batches of GlocksNon-Glock firearms“feeds and speeds” of production (or, alternatively, economic conditions of the manufacturing environment)


Factors To Consider (Continued)Ids from firing pinsIds from ejector marksIds from breechface marksIds from combination of ejector, firing pin, and breechface marksWeapon cleanlinessParticipants’ experience as toolmarks examinersParticipants asked to handle test as casework, but no measures of effectiveness for this instruction reportedBreak-in period for pistols Lubrication regimeCondition of lubrication system, nested within lubrication regime


Factors (Continued)

Temper of workpieceMicrostructure of workpieceFinishing processes (assertion only by author that “breechface is unaffected during all remaining operations” but no description as to what constituted “remaining operations” for interpretation of possible metallurgical effects)Effect of checking random sample of test cartridges for clear marks by experimenter before proceeding with the experiment was not investigated. This cannot be done in practice, as criminals do not check to insure their cartridges are clearly identifiable before leaving them at crime scenes.

Still More Factors

Batch and sample not chosen randomly; Two 9 mm Lugers were not chosen randomlyAcceptable undersized ammunitionBrand of ammunitionResponse measures Fabrication tooling materials, if anyFabrication tooling hardnessType of workpieceAlloy used for workpiece


Possible Experimental Approaches to begin Laying a Statistical Foundation for Firearm

ToolmarksThree classes of experiments will be presented

1. To build trust and confidence and motivate SOP writing. To build a team that can function together.2. To assess examiners’ abilities to differentiate class, subclass, and individual characteristics3. To assess error rates and assess their dependence upon many factors. To assess the importance of many factors and variables


Experiments Continued

The experimental approaches should proceed sequentially and under no circumstances should the third set of experiments be performed before the first two.The reason is that without an SOP and without understanding examiners’ ability to distinguish various types of marks, error assessment will not serve a scientific purposes well. They can serve ‘got you’ goals. For example…


1. Building Confidence And Trust And Starting A Detailed SOP

Pick a moderately priced pistol and film test fires from itSend the same test fired bullets (or cartridge casings) and video to participants

Ask participants if the bullet was fired from the test weapon and to give the complete list of marks used or that could be used to match the bullet to the test gun. A template (2 dimensional blueprint, if need be colors can be used to demonstrate a 3rd dimension) of the bullet should be provided for examiners to show what marks that they used. Lands and groves should be numbered so that results from different examiners can be compared


Experiment Type 1 Continued

After the results are returned summaries should be sent to all participants for discussion

Discrepancies should be discussedFollow on experiments could remove some lands and groves from the original test bullets and the experiment repeated

Eventually an SOP starts developing


2. Assess Examiners’ Abilities to Differentiate Class, Subclass, and Individual Characteristics

(We use bullets but a similar approach can be used for cartridge

casings)

Select a few production runs and choose 2 consecutively manufactures runs from each. The runs should be chosen to cover different types of subclass characteristics and unique identifying marks (dirty lubrication vs. clean lubrication)

Before each firing of the pistols make a map of each pistol barrel (using cameras and microscopy)Film test fires from each pistol and label them to the proper pistol and make video available to participating examiners


Experiment Type 2 Continued

Provide a diagram to the FTE of all marks on the bullets. Ask them to label which ones are class, subclass, and unique characteristics. (Do not ask them to match bullets to guns. )After the results are returned summaries should be sent to all participants for discussion

Discrepancies should be discussed. The goal is to characterize how well different types of marks can be characterized.

Follow on experiments could try different ammunition brands and types (of bullets-jacketed, hollow point etc.)


3. Error Rate Assessment

Perform screening experiments from among the 50+ factors and variables to find out the most relevant and important ones affecting examiner performance.

Physical factors such as gun type, and ammunition type, microscopes and lighting, etc.Social factors such as training and certification type

These experiments will need complete buy in. Social psychologists should be consulted to help make tests representative of casework


Example of a screening experiment

Subjective Comparison Procedures Still Subject To Statistical Laws And Issues

Multiple comparison issuesModeling errors (misspecified likelihoods particularly in tail regions)

Need confidence statementsPopulations need to be clearly defined

Data with annotation are neededAnalyses need to grounded within the field of statistics

A

D

Com

posi

tion

A

B

C

D

Com

posi

tion

By chaining, ‘D’ becomes “analytically indistinguishable” from ‘A’

A Hypothetical: Example: Madrid Bombing

Inference

Deductive inference:The fingerprints correspond…… therefore the suspect is the offender

Not appropriate in most cases in forensic scienceWould involve explicit exclusion of all other sources than the suspected one

Inference

Inductive inferenceIf the suspect is the offender, then the fingerprints should correspond…… the fingerprints correspond …… therefore, it is possible that the suspect and the offender are the same person

Appropriate to most cases

Inductive inference

Intuitively depends on:

Inductive inference

Intuitively depends on:Similarity between trace and control materialRarity of control materialBackground circumstances

Current Situation

It is very difficult for an audience to appreciate what the scientist truly means, and how to use the conveyed information to reach a decision

What and How are not considered separately in Forensic Science

No real consensus on the What(not even the beginning of one in the US)No study on the How

PresenterPresentation NotesReinforce that it is not only about knowing what we mean, but also ensuring that the audience unerstand it. NOTE

There is no good way to report these conclusions

Weight of evidence convey information regarding:Level of agreement between trace and control objectsLevel of “rarity” of the characteristics of the tracePotentially error rate(s)Potentially relevance, transfer and persistence

It does not:Make assumptions on size of population of potential offendersInvolve considering factors unrelated to the evidence

But weight of evidence conveys the information in an obscure way, and the audience may not be able to readily use it in its decision-making process

There is no good way to report these conclusions

Weight of evidence convey information regarding:Level of agreement between trace and control objectsLevel of “rarity” of the characteristics of the tracePotentially error rate(s)Potentially relevance, transfer and persistence

It does not:Make assumptions on size of population of potential offendersInvolve considering factors unrelated to the evidence

But weight of evidence conveys the information in an obscure way, and the audience may not be able to readily use it in its decision-making process

Thompson el al. (2013) Do Jurors Give Appropriate Weight to Forensic

Identification Evidence? Journal of Empirical Legal Studies 10(2) 359-397

Psychology of Effective Communication

Three main theories:Frequency theory – Theorizes that human beings are more competent with counts than with probabilities because they have been exposed to them more across evolution.Cognitive experiential approach – Originates from psychodynamics: different personality types, some relying more on numbers, some relying more on intuition. Intuition represents a lower level of development than numeracy.Fuzzy trace theory – Originates from cognitive research: individuals rely on their gist (substance of information –intuition ) and verbatim (exact representation of information - numeracy) to make decisions. Intuition represents a higher level of development than numeracy.


Three main theories:Frequency theory – Theorizes that human beings are more competent with counts than with probabilities because they have been exposed to them more across evolution.Cognitive experiential approach – Originates from psychodynamics: different personality types, some relying more on numbers, some relying more on intuition. Intuition represents a lower level of development than numeracy.Fuzzy trace theory – Originates from cognitive research: individuals rely on their gist (substance of information –intuition ) and verbatim (exact representation of information - numeracy) to make decisions. Intuition represents a higher level of development than numeracy.

Human relies on the least precise gist representations

necessary to make a decision


According to these theories, comprehension of information (verbal or numerical) and resulting actions/decisions are generally influenced by:

Ability to mentally conceptualize the problemFormat of the informationExpectationSeverity of the possible outcome

Ability to retrieve knowledge/values from memoryPast experienceSpecific contextCueing of relevant knowledge/values to consider

Ability to apply reasoning processesProcessing interferences

So right now what would the scientific method allow for testimony

1. In the opinion of the examiner the bullets and/or cartridge casings have marks consistent with being fired from the evidence weapon.

The prosecutor and defense attorney should ask how many weapons might be consistent with this evidence.

The examiner can answer a small percentage but should say that they do not know how small and that a few thousand weapons would be a small percentage.

So right now what would the scientific method allow for testimony

2. In the opinion of the examiner the evidence weapon cannot be eliminated as making the marks on the bullets and/or cartridge casings found at the crime scene.

The prosecutor and defense attorney should ask how many weapons could there be that would not be eliminated as providing this evidence?The examiner can answer a small percentage but should say that they do not know how small and that a few thousand weapons would be a small percentage.

How can these statements be made stronger?

By using statistically designed experimentsEstimate the probability of misidentifying ‘matches’ due to subclass characteristics.

Use uncertainty intervals in testimonyEstimate the probability of misidentifying a ‘match’ from a weapon that does not share subclass characteristics.

Use uncertainty intervals in testimony

Lack of statistical and mathematical input hurts us all

If the wrong person is convicted in part or wholly due to overstated testimony or faulty forensic practice (say dry labbing) then the guilty person is free to commit other crimes, such as murder and rape. Of course long years on death row or worse for the wrongfully convicted are unintended consequences of the current forensic practice. This is a fixable problem.

Using IR images instead of visual light


More IR detail

The Practice of Firearm/Toolmarks and a Reasonable Path Forward Three Parts Of The TalkFirearm/toolmarksTypical testimonyWhat Claims does the Scientific Method Support?What the scientific method does not allowError According To The Apple DictionaryTwo Categories Of Error Used In This PresentationMore Bias ExamplesAAS Article Highlighted By AFTE WebsiteMore Bias ExamplesMore on BiasMore Random ErrorMore Random ErrorSlide Number 15Some Factors To Consider During Firearm/Toolmark ExperimentsFactors To Consider (Continued)Factors (Continued)Still More FactorsPossible Experimental Approaches to begin Laying a Statistical Foundation for Firearm ToolmarksExperiments Continued1. Building Confidence And Trust And Starting A Detailed SOPExperiment Type 1 Continued2. Assess Examiners’ Abilities to Differentiate Class, Subclass, and Individual Characteristics�(We use bullets but a similar approach can be used for cartridge casings) Experiment Type 2 Continued3. Error Rate AssessmentExample of a screening experimentSubjective Comparison Procedures Still Subject To Statistical Laws And IssuesSlide Number 29Slide Number 30A Hypothetical: Example: Madrid BombingInferenceInferenceInductive inferenceInductive inferenceCurrent SituationThere is no good way to report �these conclusionsThere is no good way to report �these conclusionsPsychology of Effective CommunicationPsychology of Effective CommunicationPsychology of Effective CommunicationSo right now what would the scientific method allow for testimonySo right now what would the scientific method allow for testimonyHow can these statements be made stronger?Lack of statistical and mathematical input hurts us allUsing IR images instead of visual lightMore IR detailMore IR detailSlide Number 49

The Practice of Firearm/Toolmarks and a Reasonable Path ...€¦ · Firearm toolmarks examinations...

Documents

Transcript of The Practice of Firearm/Toolmarks and a Reasonable Path ...€¦ · Firearm toolmarks examinations...