Guide to Selecting Qualified WET Laboratories
Robert N. Brent, Ph.D. DynCorp Science and Engineering Group
Why is Lab Selection Important?
You, the permittee, are responsible
for:
• Meeting monitoring frequency requirements
• Certifying the quality of data
– “I certify under penalty of law …the information submitted is, to the best of my knowledge and belief, true, accurate, and complete…”
• Living with the test results
– Test failures can initiate additional testing, TIEs, TREs, enforcement actions, fines, public scrutiny
Why is Lab Selection Important?
You are the one paying
• In a 1998 WERF Study
– C.dubia chronic test: $100 - $2,300
– Fathead minnow chronic test: $238 - $5,500
• With multiple outfalls, monthly testing requirements, and multi-species monitoring, test costs can be significant
– 3 outfalls X 12 samples/yr X 3 species X $1000/test = $108,000
• Know what you are paying for!
Why is Lab Selection Important?
Quality does matter
• There is a wide range of laboratory performance
– Ex: EPA’s Variability Guidance Document
Percentiles of Intralaboratory CVs (coefficient of variation) for C. dubia Chronic Test
25th percentile CV17%
0
5
10
15
20
25
0 10 20 30 40
Test
Res
ult
50th percentile CV27%
0
5
10
15
20
25
0 10 20 30 40
Test
Res
ult
75th percentile CV45%
0
5
10
15
20
25
0 10 20 30 40
Test
Res
ult
A Tale of Two Labs
Ex: WET Interlaboratory Variability Study
0
10
20
30
40
50
60
70
80
1 2 3 4 1 2 3 4SampleLab
Va
ria
bili
ty (
Co
ntr
ol %
CV
)
0
5
10
15
20
25
30
Co
ntr
ol P
erf
orm
an
ce
(M
ea
n)
Variability
ControlPerformance
inva
lid
inva
lid
inva
lid
false +
-------------A------------- -------------B-------------
A Tale of Two Labs
Ex: WET Interlaboratory Variability Study
0
10
20
30
40
50
60
70
80
1 2 3 4 1 2 3 4SampleLab
Va
ria
bili
ty (
Co
ntr
ol %
CV
)
0
5
10
15
20
25
30
Co
ntr
ol P
erf
orm
an
ce
(M
ea
n)
Variability
ControlPerformance
inva
lid
inva
lid
inva
lid
false +
-------------A------------- -------------B-------------
$936 / test
$936 / test
$500 / test
$500 / test
Obstacles to Selecting Qualified Labs
Procurement Regulations
• Many permittees are required to select lowest responsive and responsible bidder
• This doesn’t always mean selecting lowest bidder
– Must be responsive to everything that you request in the solicitation
– Must be responsible for delivering exactly what you specify in the contract
• Solution: You determine the exact specifications for what you want
– Government Ex: $600 hammer
– WET Ex: lab must demonstrate intralab variability of <27%
• Talk to your procurement specialist about incorporating detailed specifications into the lab solicitation process
Obstacles to Selecting Qualified Labs
Insufficient knowledge and resources for identifying qualified laboratories
• Solution: This workshop will provide
– Practical tools for evaluating laboratory quality
– Things to look for in a qualified lab
– Questions to ask
– Sample test data sets to quiz your lab
General Evaluation Criteria
Capacity
• Will the lab be able to test your samples when you need them tested?
• What is the labs maximum and typical capacity (tests per week)?
– Look for maximum capacity to greatly exceed typical capacity
– Quality usually decreases when operating near capacity limits
• What limits capacity? (space, staff, organisms, equipment)
– Look for limits that can be easily remedied
General Evaluation Criteria
Staff
• What is the level of experience and education of staff from top to bottom?
– Remember: it is the staff at the bottom (technicians) that will have the most contact and interaction with your samples
– Experienced and well-trained staff is critical to consistently generating high quality data
• What is the turnover rate?
– Look for low turnover at the top
– Turnover at the bottom may be high, but look for rigorous training program
General Evaluation Criteria
Organizational Structure
• What is the chain of command?
• Who will you be interacting with?
– Look for a dedicated account manager
– Look for a laboratory contact as well
Certification
• Does the lab carry any State or national certifications?
– Not all States have lab certification programs, but beware of labs without certification in a State where there is a certification program
General Evaluation Criteria
Historical Performance
• How many tests does the lab run annually?
– Look for 50 – several hundred (for common test methods)
• What percentage of tests were successfully completed (met TAC) without retesting?
– Look for 90 -100%
• How many other clients do they have with similar waste streams (municipal, industrial, etc.) to yours?
– Familiarity with similar waste streams may lead to better, quicker problem resolution
General Evaluation Criteria
Reporting
• Does the lab’s reporting standards and format meet the requirements of your regulatory authority?
• Does the lab’s reporting format meet your needs
– Look for simple but detailed reports
Ex: You should be able to find the test result (NOEC, IC25, etc.) within 60 seconds
Ex: Report should be detailed enough for someone to completely recalculate the test results
Ex: You should be able to find the temperature of treatment A on Day 4 of the test
General Evaluation Criteria
Awareness
• Is the lab aware of recent developments in WET testing? – Look for lab to be aware of:
Proposed WET method changes and upcoming new versions of WET manuals
WET Method Guidance Document (EPA 821/B-00/004)
WET Variability Guidance Document (EPA 833/R-00/003)
Quality Control Evaluation Criteria
Organism Source and Quality
• Where does the lab get their organisms?
– In-house cultures vs. commercial suppliers
– Look for consist source
• How many organisms are available on a daily basis?
– Look for capacity to exceed usage
• What QC measures are used to assess organism health?
– Look for reference toxicant testing and other measures (reproduction and survival in cultures)
• How often are culture crashes experienced?
Quality Control Evaluation Criteria
Dilution Water Source and Quality
• What dilution water types does the lab use?
• Where does the dilution water come from or how is it prepared?
– Look for consistent high quality source
– Look for experience with different dilution water types
• How does the lab assess dilution water quality?
– Look for chemical testing and toxicity testing of new dilution water batches
Quality Control Evaluation Criteria
Reference Toxicant Control Charts
• Reference toxicant control charts are one of the best ways to assess laboratory performance (if you know what to look for)
• Labs are required to conduct monthly reference toxicant tests
• Test results are plotted on a chart to show variability over time
0
0.5
1
1.5
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Ref
. To
x. C
on
c. IC25
Mean
-2STD
+2STD
Evaluating Reference Toxicant Control Charts
Look for control charts of LC50 (for acute methods) or IC25 (for chronic methods), not NOECs
Look for 20 points
Look for curved limits
0
0.5
1
1.5
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Ref
.To
x. C
on
c. IC25
Mean
-2STD
+2STD
Evaluating Reference Toxicant Control Charts
Look at scale
0
0.5
1
1.5
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Re
f. T
ox
. Co
nc
. IC25
Mean
-2STD
+2STD
-0.50
0.51
1.52
2.53
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Re
f. T
ox
. Co
nc
. IC25
Mean
-2STD
+2STD
0
0.5
1
1.5
2
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Re
f. T
ox
. Co
nc
. IC25
Mean
-2STD
+2STD
0
0.5
1
1.5
2
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
DateR
ef.
To
x. C
on
c. IC25
Mean
-2STD
+2STD
DifferentScale
SameScale
Evaluating Reference Toxicant Control Charts
Look at CV
• CV = Standard deviation / Mean * 100%
• Compare CV to national percentiles (Variability Guidance Doc Table B-1)
0
0.5
1
1.5
2
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Re
f. T
ox
. Co
nc
. IC25
Mean
-2STD
+2STD
0
0.5
1
1.5
2
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Re
f. T
ox
. Co
nc
. IC25
Mean
-2STD
+2STD
CV = 17% CV = 40%
0
0.5
1
1.5
2
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Re
f. T
ox
. Co
nc
. IC25
Mean
-2STD
+2STD
Evaluating Reference Toxicant Control Charts
Look for results beyond control limits
• 1 in 20 will exceed limits by chance alone
• Beware of >2 exceedances in 20
Quality Control Evaluation Criteria
Control Charts for Additional QC Measures
• Look for control charts of:
– Control performance over time (i.e., control reproduction)
– Control CV over time
– PMSDs (percent minimum significant difference) over time
• Compare with national percentiles from Variability Guidance Document (Tables B-7 to B-8)
Evaluation of PMSD Control Charts
Look for PMSD mean versus national percentiles
Look for PMSDs that exceed 90th percentile
0
10
20
30
40
50
7/24/98 12/6/99 4/19/01 9/1/02
Date
PM
SD
(%
)
PMSD
Mean
50th %tile
90th %tile
Quality Control Evaluation Criteria
Statistical Analysis and Endpoint Calculation
• Can the laboratory correctly perform the recommended statistics and properly calculate test endpoints?
– EPA found that a large percentage of laboratories in the WET Interlaboratory Variability Study made one or more errors in the calculation of test results
• Test laboratories with sample data sets
– 3 sample data sets provided
– Ask laboratories to calculate LC50, survival NOEC, sublethal NOEC, and IC25 using the recommended statistical flowcharts and concentration-response evaluation guidance (where appropriate)
Sample Test Data Set #1
Sample Test Data Set #1 - Ceriodaphnia dubia Survival and Reproduction Test
Survival NOEC 25LC50 40.613
Reproduction NOEC 12.5Reproduction IC25 22.969
1-tail, 0.05 levelof significance
0
5
10
15
20
25
30
D-C
ontr
ol
6.25
12.5
*25 50 100
Da
y 6
Re
pro
du
cti
on
Sample Test Data Set #2
Requires use of concentration-response guidance
Sample Test Data Set #2 - Ceriodaphnia dubia Survival and Reproduction Test
Survival NOEC 100LC50 >100
Reproduction NOEC 100Reproduction IC25 >100*
* IC25 calculated as 86.82, but based on concentration-response evaluation, this result was determined to be anomalous and IC25 reported as >100
0
5
10
15
20
25
30
35
40
D-C
ontr
ol
6.2
5
12.5 25
50
100
Da
y 6
Re
pro
du
cti
on
Sample Test Data Set #3
Requires use of concentration-response guidance
Sample Test Data Set #3 - Fathead Minnow Larval Survival and Growth Test
Survival NOEC 50LC50 51.404
Growth NOEC 50*Growth IC25 51.446
* 6.25% treatment was significantly different from control, but based on concentration-response evaluation, this treatment was determined to be anomalous and NOEC reported as highest concentration not significantly different from the control
0
0.1
0.2
0.3
0.4
0.5
0.6
D-C
ontr
ol
*6.2
5
12.5 25 50 100
7 D
ay B
iom
ass
Conclusions
It’s your money and your reputation at stake:
• Choose a quality laboratory
When you find a quality laboratory, stick with them
When you don’t know, ask
Top Related