Post on 02-Jan-2016
2
Introduction Choice of proper instrument or method of analysis is a
major preventive quality assurance activity that will affect the quality of the service provided
The process of choosing a new instrument or kit can be divided into two parts:o Method selectiono Method evaluation
4
1- METHOD SELECTION The goal of the selection process is to choose a test
method that best suits the laboratory's service requirements
The primary consideration for making this selection should be based on the test method's usefulness in providing medically relevant data
The demands on the laboratory are imposed by the users of its services and this information can be determined by communicating with these users
In making the selection among various methods, the desired characteristics should be carefully considered
5
1- METHOD SELECTION
Three classes of characteristics should be considered:
A- The Method's Application Characteristics: These are the characteristics of a test method that
determine whether it can or cannot be implemented in a particular laboratory, include factors such as:
• Personnel • Turnaround time • Cost per test
• Space and utility requirements • Rate of analysis • Types of specimens
• Safety considerations • Run size, materials • Sample volume
6
1- METHOD SELECTION
B- Methodology Characteristics: These contribute to the quality of the method such as:
o Chemical sensitivity and specificity, o manner of calibration, o and optimization of the reaction conditions
C- Performance Characteristics: Those that determine how well the method performs in
its practical application. They include:o the method's linear range (also known as the analytical or working
range), o its precision, o and its accuracy
7
2- SOURCES OF INFORMATION You need to have a full information before deciding to
buy a new instrument or kit The best sources of information are:
o The laboratory technical literature (Journals) It will have evaluations and recommendations for different methods and
instruments Use the literature to find the method's application, methodology, and
performance characteristicso Conferences & Exhibitions
Vendors come to show their wares with demonstrations and discussions of their product
o Word-of-mouth recommendations From reliable and trustworthy sources Contact others to find out what they are using and if they are happy or
unhappy with their choice
11
3- METHOD EVALUATION Once an instrument or test kit has been selected as a
possible candidate for use, it should be carefully evaluated before making a final commitment
The evaluation process should be logically structured so that a minimum of time and effort is invested to obtain maximum results
Goals of the evaluation includes:o The first determinations to be made in a method evaluation
are the imprecision and inaccuracy, which should be compared with the maximum allowable error based on medical criteria
o Discover if the new method fits into the framework of the laboratory's organization and workload
12
3- METHOD EVALUATION Imprecision is estimated from studies in which multiple
aliquots of the same specimen (with a constant concentration) are analyzed repetitively (Replication Experiment)
Inaccuracy, or the difference between a measured value and its actual value, is due to the presence of a systemic error
Systemic error can be due to constant or proportional error and is estimated from Recovery study and interference study
13
Allowable Analytic Error (Ea)
Probably the most important aspect of method evaluation is to determine if the random and systematic error (total error) is less than the Ea
The error that can be tolerated without invalidating the medical usefulness of the test
The Clinical Laboratory Improvement Amendments of 1988 (CLIA 88) have published Eas for an array of clinical tests
15
3- METHOD EVALUATION The evaluation should last for no less than one week and
no more than 60 days. Barnett and Westgard recommend a minimum of 20
days for a complete evaluation that includes day-to-day precision studies.
Enough time should be allowed to perform all of the necessary evaluation experiments and to observe the instrument's day-to-day variation. o Too short a time period may result in important data being
missed or misrepresented. o Too long an evaluation period may last longer than the
expiration date of reagents, calibrators, and controls, and also incorporate long-range instrument variation.
16
A- Instrument and Method Evaluation Steps:
Construct a good evaluation plan and establish a
working procedure detailing calibration, operation, and
maintenance during the evaluation period.
Acquire enough reagents, calibrators, and controls to
last the evaluation period.
The plan should be designed as a series of logical steps
so that failure in the preliminary steps will cancel the
continuation of the process.
17
A- Instrument and Method Evaluation Steps:
The course that the evaluation process takes will depend on the availability of an acceptable comparative method
If the comparative method has satisfactory and well-documented precision and accuracy, then the evaluation should determine if the new method is as accurate and precise as the old.
If a satisfactory comparative method is not available for comparison, then other experiments should be used to determine the new method's precision and accuracy.
If possible, the entire evaluation should be conducted by a single individual.o Variations between individual techniques and bias can be
excluded if one person, conducts all of the experiment.
18
B- Linearity Check The first step in the Evaluation process is to perform a
Linearity Check It is important to assess the useful analytical range of a
laboratory method, o i.e., the lowest and highest test results that are reliable and can be
reported
This confirms that the instrument or method is linear over its published working range stated by Manufacturers
19
B- Linearity Check
How to perform linearity check:o An aqueous or protein-based standard of high concentration
is diluted to obtain a series of samples of lesser but known concentration.
o The range of samples should cover both ends of the method’s range of linearity.
o Analyze the samples and compare the results to the expected values by constructing a scatter plot
o Place the expected value on the x-axis and the obtained value on the y-axis and draw a line of agreement
20
B- Linearity Check
Example of linearity experiment (glucose method with a
published range from 0-500 mg/dl)
A known glucose standard of 500 mg/dl is diluted (as
shown in the figure) with deionized water
All dilutions should be made from a stock sample
avoiding serial dilutions
Dilution errors are magnified in serial dilutions
Use volumetric glassware and good pipetting technique
when preparing the dilutions
21
B- Linearity Check Analyze each sample in triplicate and average the results. Accept the method's linearity if the line of agreement has a
slope of 1.000 ± 0.03 and an intercept of zero plus or minus the within-run precision of the method.o (If the within-run precision of the method is not known, it will be
determined in the next series of experiments; consequently this part of the evaluation may not be completed until the replication experiments are completed).
Investigate pipetting errors as the probable cause of a nonlinear line by repeating the experiment with a freshly diluted series of samples.
If the method is not linear over its published range, stop and investigate the problem before any further experiments arecarried out.
When a method loses linearity on either end of its range the line will start to curve.
26
C- Replication Experiment This evaluation experiment is used to demonstrate a test
method's precision and random error Three different replication studies are performed:
o within-run, o within-day, o and day-to-day
Choose three samples that have the same matrix or physical qualities as the patient samples that the method will be analyzing
The three should represent the low, normal, and high physiological concentrations of the analyte in question
27
C- Replication Experiment The within-run replication experiment measures
precision or the lack of it caused by random error within an analytical runo Each sample is analyzed a minimum of 20 times within a single
analytical run
The within-day replication experiment measures the amount of random error between runs that occurs within a single dayo Each of the three samples is analyzed a minimum of 20 times
throughout the day in several analytical runs
The day-to-day replication experiment measures the amount of random error inherent in the method from day-to-dayo Analyze each sample daily for a minimum of 20 consecutive days
28
C- Replication Experiment
For most chemical constituents, lyophilized control
serum or some similar stable material should be used for
the day-to-day replication experiments
Though there are some differences between prepared
controls and patient specimens, the control material will
remain stable over the 20-day period and chances of
obtaining it in the desired concentrations and quantities
are greater than using only patient samples
29
For each of the replication experiments, calculate a mean, mode, standard deviation (s), and coefficient of variation (CV) for each sample
The greater the imprecision of the method, the larger the standard deviation will be
If the distribution of the values is due to random chance, then it should have a normal or Gaussian distribution with the mean equal to the mode
If the distribution is skewed as the result of some random error, then the mode and the mean will not be equal
C- Replication Experiment
30
C- Replication Experiment A method lacks precision if it has a large coefficient of
variation and there is a substantial difference between the mean and the mode
Such random error is unacceptable and potentially dangerous to the patient.
Ninety-five percent of the values should fall within two standard deviations of the mean
One in 20 samples will be beyond the 2s limits If more than 1 sample in 20 or 2 in 40 are beyond 2s for
unexplained reasons (other than acknowledged errors in pipetting, short sampling, or some other obvious problem) the method should be discarded as having too great a degree of random error and thus is potentially dangerous to the patient
31
C- Replication Experiment The acceptable CV will vary, depending on the medical
significance of the analyte that the method is measuring The within-run and within-day replication experiments
are optimistic evaluations of the method’s precision but are useful in determining if the method is performing properly
The day-to-day replication experiment is a better indication of the method’s overall precision
32
D- Recovery Experiment The recovery experiment is performed to estimate the
proportional systematic error in the absence of a reliable comparative or reference method
This is the type of error whose magnitude increases as the concentration of analyte increases
The error is often caused by a substance in the sample matrix that reacts with the sought for analyte and therefore competes with the analytical reagent
33
D- Recovery Experiment A patient sample is spiked with varying amounts of pure
standard and then analyzed A baseline sample and samples with concentration at the
medical decision levels should be prepared After analyzing the samples in triplicate and averaging
the values for each, the expected results are compared to the actual results and the degree of proportional systematic errors defined as the percent of the added analyte recovered
Westagrd states that the limits of acceptability occur when the percentage of error is less than the determined allowable analytical error (95% of the limit of allowable error) at the medical decision levels
34
D- Recovery Experiment If it is determined that the allowable error of a value is
limited to 95% of that value without any harm or influence on the physician’s course of treatment, then any proportional systematic error greater than 5% at that decision level will judge the method to be unacceptable
36
D- Recovery Experiment
Interpretationo This method has a 90% recovery rate or a 10%
proportional systematic rate based on this experimento The acceptable level of recovery or accuracy will vary,
depending on the analyte and the concentration
37
E- Interference Experiment The interference experiment is performed to estimate the
systematic error caused by other materials that may be present in the specimen being analyzed
We describe these errors as constant systematic errors because a given concentration of interfering material will generally cause a constant amount of error regardless of the concentration of the analyte being tested in the specimen
As the concentration of interfering material changes, however, the size of the error is expected to change
38
E- Interference Experiment The volume of the interferer should be small relative to
the original test sample to minimize the dilution of the patient specimen
However the amount of dilution is not as important as maintaining the exact same dilutions for the pair of test samples
Using the test method in question, a sample is analyzed in triplicate to determine a value
A specific amount and concentration of a substance thought or known to be an interfering substance is added to this sample and the sample is then reanalyzed
39
E- Interference Experiment The difference between the result of the sample with and
without the substance is calculated The sample is repeated with several different
concentrations of the interfering substance The acceptability of this experiment, according to
Westagrd, is that the difference between the two values, one with and the other without interfering substance, should be less than the allowable analytical error at that concentration
40
E- Interference Experiment Interference to be tested
o Bilirubin can be tested by addition of a standard bilirubin solutiono Hemolysis is often tested by removing one aliquot of a sample,
then mechanically hemolyzing or freezing and thawing the specimen before removing a second aliquot.
o Lipemia can be tested by addition of a commercial fat emulsion, such as Liposyn (Abbott Laboratories) or lntralipid (Cutter Laboratories)
o Additives to specimen collection tubes can be conveniently studied by drawing a whole blood specimen, then dispensing aliquots into a series of tubes containing the different additives
41
E- Interference Experiment
Experiment:o As example, determination of glucose will be usedo Creatinine is an interfering substance in this method.
By adding creatinine to a sample in measured amounts the amount of interference can be determined
o Prepare the sample as follows:Sample #1, 0.9 ml serum + 0.1 ml deionized water (baseline)
Sample #2, 0.9 ml serum + 0.1 ml of 50 mg/dl creatinine STD
Sample #3, 0.9 ml serum + 0.1 ml of 100 mg/dl creatinine STDo Each sample is prepared and run in triplicate and the
results are averaged.
42
E- Interference Experiment
Results:
Amount of glucose measured
Amount of Creatinine added Interference
Sample # 1 100 mg/dl 0 mg/dl 0 mg/dl
Sample # 2 104 mg/dl 5 mg/dl + 4 mg/dl
Sample # 3 111 mg/dl 10 mg/dl +11 mg/dl
43
E- Interference Experiment
Calculations:1. Amount of creatinine added
2. Amount of InterferenceInterference = Conc. Measured (test) — Conc. Measured (baseline)
Interference = 104 mg/dl 100 — 100 mg/dl = 4 mg/dl
Conc. Added = 50 mg/dl X 0.1 ml
= 5 mg/dl0.1 ml + 0.9 ml
Conc. Added = STD. Conc. X ml STD. Added
ml of serum+ ml of STD
44
E- Interference Experiment
Interpretation:o The experiment shows that increasing amounts of
creatinine will result in an increasing positive interference
o The acceptable level of recovery and accuracy will vary depending on the analyte and the concentration
o In this case, further investigating is recommended
45
F- Comparison Of Methods The comparison of methods is a procedure that
determines accuracy and precision by comparing the test method to a method of known precision and accuracy
The comparative method can be either a recognized reference method (is one that has a well documented low level of imprecision and inaccuracy) or a method of analysis that the laboratory wishes to replace.
Choose a method that is well documented and widely accepted
Check the comparative method carefully during the experiment to ensure that it is performing properly
Run controls and use proper procedures in performing the comparison on both methods
46
F- Comparison Of Methods When using a known reference method, all of the
observed differences between the methods can be attributed to the new method.
When the comparative method is the one that is being used and the bias and analytical error is known, then part of the observed analytical error noted between the two methods can be attributed to the comparative method, with the remaining error belonging to the test method
The difference between the two methods at medical decision levels should be less than the allowable analytical error for that concentration of the analyte
47
F- Comparison Of Methods This experiment involves simultaneously analyzing split
samples on both the test and comparative method Only patient samples should be used for evaluation
between methods Lyophilized, aqueous, or ethylene glycol-based control
samples have slightly different physical properties than fresh human serum or plasma and may react differently in different systems
Westgard recommends that each sample be run in duplicate, at different analytical runs o This is done in order to check on the validity of the experimental
observations and to detect random errors, such as mixing up of samples and short sampling
48
F- Comparison Of Methods The replicates should be analyzed the same day on the
same instrument and within as short a time as possible The difference between the duplicate measurements
should be less than or equal to the determined between-run precision
If the differences are grater than this, suspect an error Analyze a minimum of 40 samples, 5 samples a day for 8
days Select a variety of concentrations so that the entire linear
or working range of the new method will be represented This requires the pre-selection of samples but comparing
samples only in the " normal " range can cover analytical differences between the methods at medical decision levels
49
F- Comparison Of Methods
Reduce opportunities for bias and mistakes to occur:
1. Limit the number of technologists participating in the experiment.
2. Try to run samples on both instruments at the same time to limit the effects of sample deterioration.
3. Use acceptable techniques in preparing reagents and samples for analysis.
50
F- Comparison Of Methods There are several ways of approach in evaluating
the comparison between two methods:1. Plot the results on a scattergram
2. Linear regression analysis
3. The paired t-Test
4. The F-Test
5. The correlation of coefficient
51
F- Comparison Of Methods
1. One of the easiest and most visually studied is to plot the results on a scattergram, draw a best straight line through the points, and determine the slope and intercept of the line
o The slope and the intercept offer a fair evaluation of the agreement between the methods
o Data should be plotted on the scattergram on a daily basis and inspected for outliers so that original samples can be reanalyzed as needed
o If the technologist waits until all of the samples are analyzed there will be no opportunity to investigate outliers or to reanalyze samples
52
F- Comparison Of Methods Plot the test method on the y-axis and the reference or
comparative method on the x-axis. A best fit straight line can be drawn through the plotted
points A visual inspection of the line and the points around it can
give a preliminary assessment of the agreement between the methods