AACB ASM 2003

THE POWER OF ERROR DETECTION OF

WESTGARD MULTI-RULES: A RE-EVALUATION

Graham JonesDepartment of Chemical Pathology

St Vincent’s Hospital, Sydney

AACB ASM 2003

Background• Westgard multi-rules are claimed to increase the

power of error detection of laboratory QC procedures.

• Power Function Charts can quantify the ability of these rules to detect changes in assay performance.

• Examples of Power Function Charts are available on the Westgard QC website (www.westgard.com)

• In this poster I re-evaluate the power of error detection of QC rules which require data from more than one QC run (Multi-run rules).

AACB ASM 2003

Hypothesis

• That the correct model for assessing the Power of Error Detection for multi-run QC rules should only show benefit for these rules if the error has not already been detected in QC runs required to gather data for those rules.

• This hypothesis was modelled and compared to data on the Westgard website.

AACB ASM 2003

Nomenclature

• Single-run rules– All data is contained in a single QC run

• For n=2 includes 13s and 22s

• For n=4 includes 13s and 22s and 41s

• Multi-Run Rules– Requires data from more than one QC run

• For n=2 includes 41s and 10x

• For n=4 includes 8x

AACB ASM 2003

Methods

• Power Function Charts were produced using a Microsoft Excel spreadsheet.

• QC results were simulated using a random number generator with a normal distribution.

• Changes in bias were modelled by adding various constants to the output.

• QC rules were evaluated by the frequency with which they were triggered at changes in bias.

• Westgard multi-rules with n=2 were evaluated for bias detection: 13s/22s/41s/10x.

• Changes in random error were not modelled.

AACB ASM 2003

Hypothesis - Graphical Display

Mean

-3SD

+3SD

+2SD

-2SD 1 2 3 4 5

QC run - within-run rules evaluate performance (13s/22s)

QC run - within-run rules evaluate performance (13s/22s) - multi-run rule evaluates performance (10x across both materials) - Only adds benefit if shift NOT detected by QC events 1-4

Change in assay bias

This display uses 10x as an example of a multi-run rule

AACB ASM 2003

00.10.20.30.40.50.60.70.80.9

1

0.0 1.0 2.0 3.0 4.0

1.3s/2.2s/4.1s/10x1.3s/2.2s/4.1s1.3s/2.2s1.3s

Pro

babi

lity

for

Rej

ectio

n

Shift in Bias (multiples of SD)

ResultsA

B

Graph A- Original data from Westgard

website

Graph B- Model of data from Westgard website.- Multi-run rules fire even if shift would

have been detected previously.

AACB ASM 2003

00.10.20.30.40.50.60.70.80.9

1

0.0 1.0 2.0 3.0 4.0

10x

4.1s

2.2s

1.3s

00.10.20.30.40.50.60.70.80.9

1

0.0 1.0 2.0 3.0 4.0

1.3s/2.2s/4.1s/10x

1.3s/2.2s/4.1s

1.3s/2.2s

1.3s

Pro

babi

lity

for

Rej

ectio

n

D

C

Graph C- Westgard data adjusted for hypothesis.- Multi-run Rules fire only if shift would

NOT have been detected previously.

Graph D- Model of individual rules from Graph C- Multi-run Rules fire only if shift would

NOT have been detected previously.Shift in Bias (multiples of SD)

Shifts detected with 90% certainty from full multi-rulesShifts detected with 90% certainty from within-run rules.

AACB ASM 2003

Results• A power Function Chart from the Westgard website

showing multi-rules for bias detection with n=2 is shown in graph A.

• The change in bias which Westgard claims full Multi-rules can detect with 90% certainty is about 2.0 times the SD of the assay (Graph A).

• My model of the Westgard data, with multi-run rules triggered even if the change in bias would been previously detected, agrees well with the website data (Graph B).

• In the Westgard model the multi-run rules (10x and 41s) enhance the error detection over the within-run rules (graphs A and B)

AACB ASM 2003

• The model excluding multi-run rules when a shift would have been detected previously is shown in Graph C.

• When these previously-detected shifts are removed from the data, the assay bias which can be detected with 90% certainty is reduced to about 3.3 times the assay SD (Graph C).

• With this model the multi-run rules do not add to the within-run rules for confident error detection.

• When the individual rules are plotted it can been seen that the multi-run rules never add to the error detection with 90% certainty.

• The multi-run rules can be considered warning rules.

AACB ASM 2003

Conclusion

• The multi-run rules, as described on the Westgard website, give a falsely low estimate of the change in bias which can be detected with 90% certainty.

• The 10x and 41s rules add little to the overall error detection at the 90% confidence level with 2 QC samples per run.

• Multi-run rules are similarly non-contributory with 4 QC samples per run (data not shown).