Standards, Validation, and Evaluation Studies to Aid the ...Anal. Chem. Division Jan Redman Funding:...

M.C. Kline – NIJ Applied Technologies and Partnerships Conference

February 1, 2006

http://www.cstl.nist.gov/biotech/strbase/NISTpub.htm 1

Standards, Validation, and Evaluation Studies to Aid the

Forensic DNA Community

Margaret C. Kline and John M. ButlerNational Institute of Standards and Technology, Chemical Science and Technology Laboratory, Gaithersburg, MD

NIJ Applied Technologies and Partnerships ConferenceFebruary 1, 2006 (Hilton Head, SC)

DisclaimersFunding: Interagency Agreement 2003-IJ-R-029 between the National Institute of Justice and NIST Office of Law Enforcement Standards

Points of view are those of the authors and do not necessarily represent the official position or policies of the US Departmentof Justice. Certain commercial equipment, instruments and materials are identified in order to specify experimental procedures as completely as possible. In no case does such identification imply a recommendation or endorsement by the National Institute of Standards and Technology nor does it imply that any of the materials, instruments or equipment identified are necessarily the best available for the purpose.

Our publications and presentations are made available at: http://www.cstl.nist.gov/biotech/strbase/NISTpub.htm

Presentation Outline

• Importance of Quality Results in DNA Testing

• Introduction to NIST and its Role

• Standard Reference Materials

• Validation Resources

• Evaluation and Interlaboratory Studies

Quality Is Essential in Forensic DNA Testing

• DNA results impact lives – the guilty can be implicated in a crime and the innocent can be exonerated

• Scientific attacks against the science behind DNA testing are rare in court now. Rather the focus is on demonstrating that quality results were obtained.

• DNA databases involve comparisons of DNA profiles analyzed at different times or in different locations

DNA Testing Requires a Reference Sample

Crime Scene Evidence compared to Suspect(s) (Forensic Case)Child compared to Alleged Father (Paternity Case)Victim’s Remains compared to Biological Relative (Mass Disaster ID)Soldier’s Remains compared to Direct Reference Sample (Armed Forces ID)

A DNA profile by itself is fairly useless because it has no context…

DNA analysis for identity only works by comparison – you need a reference sample

Elements for Guaranteeing Quality Results in Forensic DNA Testing

• Accepted Standards and Guidelines for Operation• Laboratory Accreditation• Proficiency Testing of Analysts• Standard Operating Procedures• Validated Methods• Calibrated Instrumentation• Documented Results• Laboratory Audits• Trustworthy Individuals


February 1, 2006


Many Labs Are Moving to ISO 17025 as Part of Their Laboratory Accreditation

http://www.ascld-lab.org/international/indexinternational.html

a specific program of accreditation based upon the requirements of ISO/IEC 17025 –General requirements for the competence of testing and calibration laboratories

Requires measurement traceability to national measurement standards from a National Metrology Institute

ISO/IEC 17025

5.6 Measurement traceability

5.6.1 General

All equipment used for tests and/or calibrations, including equipment for subsidiary measurements (e.g. for environmental conditions) having a significant effect on the accuracy or validity of the result of the test, calibration or sampling shall be calibrated before being put into service. The laboratory shall have an established programme and procedure for the calibration of its equipment.

ISO/IEC 17025

5.6.2.1.2 There are certain calibrations that currently cannot be strictlymade in SI units. In these cases calibration shall provide confidence in measurements by establishing traceability to appropriate measurement standards such as:

• the use of certified reference materials provided by a competent supplier to give a reliable physical or chemical characterization of a material;

•the use of specified methods and/or consensus standards that areclearly described and agreed by all parties concerned.

Participation in a suitable programme of interlaboratory comparisons is required where possible.

NIST History and Mission• National Institute of Standards and Technology

(NIST) was created in 1901 as the National Bureau of Standards (NBS). The name was changed to NIST in 1988.

• NIST is part of the U.S. Department of Commerce with a mission to develop and promote measurement, standards, and technology to enhance productivity, facilitate trade, and improve the quality of life.

• NIST is the National Metrology Institute for the U.S.

• NIST supplies over 1,300 Standard Reference Materials (SRMs) for industry, academia, and government use in calibration of measurements.

DNA typing standard

NIST Gaithersburg Campus

Advanced Chemical Sciences Laboratory (Building 227)

Located in Gaithersburg, Maryland, on approximately 234 hectares (578 acres) just off Interstate 270 about 25 miles northwest of Washington, D.C.

Administration (Building 101)

~2,500 staff

http://www.nist.gov

Overview of NIST Organization

Chemical Sciences and Technology Laboratory (CSTL) – 5 divisions

Biotechnology Division

DNA Technologies Group

Human Identity Project Team

ITL (7)

MSEL (6)

Physics (7)

BFRL (5)

http://www.cstl.nist.gov/biotech/DNATechologies/Human_Identity.htm

Structural Biology

Cell and Tissue Measurement

Bio-processing

Analytical Chemistry

Process Measurements

Surface & Microanalysis

Science

Physical & Chemical Properties

EEEL (8)

MEL (5)

Technology Services (4)

Clinical Diagnostics/

EDRN

DNA Damage & Repair

MtDNA/ Mutation Detection

Biochemical Sciences Division

DNA Measurements Group


February 1, 2006


NIST Human Identity Project Team

Mike CoblePete ValloneJohn Butler(Project Leader)

Margaret Kline

Amy Decker Becky HillDave DuewerAnal. Chem. Division

Jan Redman

Funding: Interagency Agreement 2003-IJ-R-029 between National Institute of Justice (NIJ) and NIST Office of Law Enforcement Standards (OLES)

Chris DeAngeliscomputer programmer

Current Areas of NIST Effort with Forensic DNA

• Standards– Standard Reference Materials– Standard Information Resources (STRBase website)– Interlaboratory Studies

• Technology– Research programs in SNPs, miniSTRs, Y-STRs, mtDNA, qPCR– Assay and software development

• Training Materials– Review articles and workshops on STRs, CE, validation– PowerPoint and pdf files available for download

Quality Assurance Standards for Forensic DNA Testing Laboratories

(October 1, 1998)

STANDARD 9.5The laboratory shall check its DNA procedures annually or whenever substantial changes are made to the protocol(s) against an appropriate and available NIST standard reference material or standard traceable to a NIST standard.

FBI’s DNA Advisory Board

Congress Passed the DNA Identification Act of 1994 (Public Law 103 322)

Formalized the FBI's authority to establish a national DNA index for law enforcement purposes.

Standard Reference Materials (SRMs)

• Relevant Forensic DNA SRMs– SRM 2391b (DNA profiling – STRs, D1S80, DQA1/PM)– SRM 2392-I (mtDNA, Cell line HL-60)– SRM 2395 (Y-chromosome)– SRM 2372 (Human DNA quantitation); in development

• Provides national/international traceability and compatibility (aids in ISO 17025 compliance)

http://www.nist.gov/srm

2002200019981995199019851980 2004

SRM 2372SRM 2395SRM 2392SRM 2391SRM 2390

DNA Technology

FBI s

tops

runn

ing

RFL

P

Hum

an G

enom

e dr

aft

U.S

. Y-S

TRs s

elec

ted

FBI Q

A m

anda

te

Euro

pean

Y-S

TRs c

hose

nC

OD

IS S

TR lo

ci c

hose

n

Firs

t U.S

. mtD

NA

cas

e

Firs

t sin

glep

lex

STR

kits

FSS

mtD

NA

FBI s

ingl

e-lo

cus R

FLP

PCR

with

Taq

enz

yme

Aut

omat

ed se

quen

cers

Jeff

reys

mul

ti-lo

cus R

FLP

And

erso

n m

tDN

A

MSS3

43 types2221164 STRs

Recertification

2006

QS04

Forensic DNA Timeline with NIST SRM Development 1995: SRM 2391 PCR-based DNA Profiling Standard

D1S80 LocusVariable NumberTandem Repeat

(VNTR)6 – Pre-amplified D1S80 samples1 – D1S80 allelic ladder1 – size standard

10 – Genome DNA Extracts2 – Cell line cells spotted on

903 paper for extraction

800 bp

400 bp

Silver stained gels for STR monoplexes

DQ-alpha & PolyMarker reverse dot blot hybridization

Samples originally selected to possess

all DQA1 types


February 1, 2006


2003: SRM 2391b released•22 STR Loci•D1S80 and DQA1/PM (still certified)•Many labs using 16plex STR kits

Identifiler STR kit

PowerPlex 16 STR kit

PowerPlex 16 BIO

22 autosomal STRscharacterized across

12 DNA samples

22 autosomal STRscharacterized across

12 DNA samples

NIST SRM 2391bAll current commercial kit

STR loci are certified

Standard Reference Material® (SRM)

A NIST SRM is prepared and used for three main purposes:

1) to help develop accurate methods of analysis;

2) to calibrate measurement systems used to facilitate exchange of goods, institute quality control, determine performance characteristics, or measure a property at the state-of-the-art limit;

3) to ensure the long-term adequacy and integrity of measurement quality assurance programs.

The Current Task:

SRM 2372: Human DNA Quantitation Standard

Challenge:What is a nanogram of genomic DNA ?

From interlaboratory studies we know there is a factor of 1.6 in the measurement systems currently in use. But the range is 20 fold.

SRM 2372Human DNA Quantitation Standard

A B

C Component A: MaleComponent B: FemaleComponent C: Mixture

Anticipated 2006 issue

NOT AVAILABLE AT THIS TIME Planned Amounts: Each component 50 µL of Human

Genomic DNA with a concentration targeted @ 50 ng/µL. The [DNA] for each component will be listed in the materials Certificate of Analysis.

Some of the Information Resources on the NIST STRBase Website

• …/str_fact.htm STR Fact Sheets on Core Loci• …/multiplx.htm Multiplex STR Kit Information• …/y_strs.htm Y-Chromosome Information• …/var_tab.htm Variant Alleles Reported• …/mutation.htm Mutation Rates for Common STRs• …/str_ref.htm Reference List with ~2,500 Papers• …/training.htm Downloadable PowerPoints for Training• …/validation.htm Validation Information• …/miniSTR.htm miniSTR Information• …/address.htm Addresses for Scientists• …/NISTpub.htm Publications & Presentations from NIST

http://www.cstl.nist.gov/biotech/strbase


February 1, 2006


Validation Resources• Survey initiated at June 2004 NIJ meeting and

conducted last summer resulted in 53 responses• Talk at Promega meeting Oct 2004• Validation summary sheets• Validation website on STRBase• Workshop conducted August 2005 at NFSTC (DVD to

be released as part of President’s DNA Initiative training)– http://www.cstl.nist.gov/biotech/strbase/validation/validationworkshop.htm

• We invite submission of your internal validation studies for inclusion in the NIST validation website

http://www.cstl.nist.gov/biotech/strbase/validation.htm

New Validation Homepage on STRBasehttp://www.cstl.nist.gov/biotech/strbase/validation.htm

Forensic Science International 148 (2005) 1-14

Other information and conclusions

How?

What validated?Where published?

VALIDATION WORKSHOP

Laboratory Internal Validation Summaries

Soliciting Information on Studies Performed by the Community

Evaluations by Our Project TeamOur project team has assisted in beta-testing numerous products:

– Applied Biosystems (Foster City, CA)• GeneMapperID, Yfiler kit

– Marligen Biosciences (Ijamsville, MD)• Signet Y-SNP kit

– Promega Corporation (Madison, WI)• PowerPlex Y kit

– ReliaGene Technologies, Inc. (New Orleans, LA)• Y-PLEX 5, Y-PLEX 12 kits

– Roche Molecular Systems (Alameda, CA)• mtDNA HVI/HVII LINEAR ARRAY

• We are regularly participating in EDNAP/ENFSI European studies regarding Y-STRs, SNPs (FSS 21plex and SNPforID 52plex), and degraded DNA samples

• We have supplied information to assist U.S. companies in assay development: Orchid Cellmark (validation of autosomal SNPs); Bode Technology Group (miniSTR assays)

Contact us if you are interested in

collaborating

NIST Initiated Interlaboratory Studies

Data analysis currently on-going ...Preliminary data shown at NIJ Grantees (June 2005), ISFG (Sept 2005), Promega (Sept 2005)

69Mixture Interpretation Study (Jan-Mar 2005)

Kline, M.C., Duewer, D.L., Redman, J.W., Butler, J.M. (2005) Results from the NIST 2004 DNA Quantitation Study, J. Forensic Sci. 50(3):571-578

80DNA Quantitation Study (Jan-Mar 2004)

Kline, M.C., Duewer, D.L., Redman, J.W., Butler, J.M. (2003) NIST mixed stain study 3: DNA quantitation accuracy and its influence on short tandem repeat multiplex signal intensity. Anal. Chem. 75: 2463-2469.

Duewer, D.L., Kline, M.C., Redman, J.W., Butler, J.M. (2004) NIST Mixed Stain Study #3: signal intensity balance in commercial short tandem repeat multiplexes, Anal. Chem. 76: 6928-6934.

74Mixed Stain Study #3 (Oct 2000-May 2001)

Duewer DL, Kline MC, Redman JW, Newall PJ, Reeder DJ. (2001) NIST Mixed Stain Studies #1 and #2: interlaboratory comparison of DNA quantification practice and short tandem repeat multiplex performance with multiple-source samples. J. Forensic Sci. 46: 1199-1210

45Mixed Stain Studies #1 and #2 (Apr–Nov 1997 and Jan–May 1999)

Kline MC, Duewer DL, Newall P, Redman JW, Reeder DJ, Richard M. (1997) Interlaboratory evaluation of STR triplex CTT. J. Forensic Sci. 42: 897-906

34Evaluation of CSF1PO, TPOX, and TH01 (CTT)

# Labs PublicationsStudies involving STRs

MSS3

QS04

MIX05

http://www.cstl.nist.gov/biotech/strbase/interlab.htm CTT (1995)

Size of Smaller Allele (bp)

Size

of L

arge

r Alle

le

(bp)

185 190 195

185

190

195

200 TH01

111111111111111111333333

333333333333333 44444

4444444444444445555555

55555555555555 666

666666666666666

77777

77777777777777

888822222222222222222222

145 150 155

155

160

165

vWA

111111

111

11333

333

33333 4

44444

444

44

5

5555

5555

666

666

666

6

777777

77777

2222

22

22

22

230 235 240

235

240

245

TPOX

111111

1111

111111

11111

33333

33

33333 3333

33333

44444

44

4444

444444

4444

55555

55

5555

5 5555

555

6666

66

666

666666

6666

77777

7777

7777777777

8888

22222

22

2222

222222

2222

310 315 320

310

315

320

325CSF1PO

1111

11111

1

11111111

11

33

333

333333

3

3333333333

44444

444444

4

4444444444

5

555555

55555

55555

555

6666

666666

6

66666

66666

7777

7777777777777

777

8888

2222

2

22

22222

2222222

222

aaaaa

aa

aaaaa

aaaaaaa

aaa bbb

bb

bb

bbbbb

bbbbbbb

bb

Demonstrated the need for allelic ladders, regardless of the precision of the analytical instrumentation.

J. Forensic Sci. 1997; 42: 897-906

Figures were created by graphing the bp sizes of allele 1 against allele 2 for the 7 samples (same amplification kit).

5 bp


February 1, 2006


h

h

h

z

z

z

l

l

l

l

0.5

0.1

0.05

510.50.10.05

1

5

Reported [DNA]N, ng/µL

Rep

orte

d [D

NA

] P, n

g/LTarget

plot of replicate samples N and P.

Quantitation Issue from Mixed Stain 2 (1999)

J. Forensic Sci. 46: 1199-1210

NMSS#3 Reported DNA concentrations of the samples, ng/µL

X

W

V

U

T

SR

5

1

0.5

Rep

orte

d [D

NA]

, ng

/ µL

Interlaboratory SummaryQuantiBlot

Your Values

The National Institute of Standards and TechnologyGratefully Acknowledges the Participation of the

Laboratory XYZ

In the 2001 Interlaboratory Challenge Exercise “Mixed Stain Study #3”

Sample Quantitation Sample Typing

Margaret C. Kline, Study Coordinator

0.3

0.5

1

3

5

1010

R S T UV WX

2.5%

25%Median75%

97.5%

DN

A C

once

ntra

tion,

ng

/ µL

Others

99.5 99 98 96 90 80 70

You

3

2

1

0

Donor Alleles (Major + Minor) Identified, %

Non

-Don

or A

llele

s R

epor

ted,

%

This feedback can be helpful to a laboratory to know where they stand relative to other labs to illustrate opportunities for improvement.

See Kline, M.C., et al. (2003) Anal. Chem. 75: 2463-2469

DNA Quantitation Accuracy in STR Typing

aaa a

a

a aappp

p

p

ppa p

-3 -2 -1 0 1 2 3

Direct,Non-blot

TE

E

TT TTT TT T

T TT

T TTTT

T T T TTTT

TTF

F

FDDDE E

E

E

E

E

EEE

EEE

EEE EE E

EA

A AA ATFDEA

-3 -2 -1 0 1 2 3

Direct,Blot

29

8 766

55431

11

0

00

00 0

000 000

00

00

000

6 510-3 -2 -1 0 1 2 3

PCRRT

Concordance

App

aren

t Pre

cisi

onAmong Participate Results for QS04

Concordance – is a direct multi-material analogue of bias

Apparent precision – is analogous to precision but also incorporates sample-specific measurement differences or “matrix effects”.

“Bold” characters represent the median performance for all results submitted for a particular method

The semi-circles delimit 1, 2 and 3 standard deviations of total comparability.

T 29

8 766

55431

11

0

00

00 0

000 000

00

00

000

6 510-3 -2 -1 0 1 2 3

PCRRT

Concordance

App

aren

t Pre

cisi

on

0 = Quantifiler1 = Alu RT-PCR5 = BRCA16 = CFS-HUMRTKline, et al. (2005) J. Forensic Sci. 50(3):571-578

Interlaboratory Comparisons

60 data setsLaboratory Performances with Real-Time PCR Methods

Comparing results from 8 different samples using

10 different methods

DNA Standard

Ave

rage

CT

valu

e

27.00

27.60

28.20

28.80

0 1 2 3 4 5 6 7

Quantifiler Results 1.6 ng of genomic DNA; n=4

Variations in CT due to the DNA standard used area apparent

∆CT = 1.13 between STDs 2 & 6 (factor of 2.2)


February 1, 2006


Acknowledgments

Mike Coble

Pete Vallone

John Butler

Margaret Kline

Amy Decker

Becky Hill

Dave Duewer

Jan Redman

Chris DeAngelis

Funding from interagency agreement 2003-IJ-R-029 between NIJ and the NIST Office of Law Enforcement Standards

NIST Human Identity Project Team

This presentation available as pdf file from http://www.cstl.nist.gov/biotech/strbase/NISTpub.htm

[email protected]

Standards, Validation, and Evaluation Studies to Aid the ...Anal. Chem. Division Jan Redman Funding:...

Documents

Transcript of Standards, Validation, and Evaluation Studies to Aid the ...Anal. Chem. Division Jan Redman Funding:...