Comparing Two Measurement Devices

32
4 2 5 1 0011 0010 1010 1101 0001 0100 1011 Comparing Two Measurement Devices Brian Novatny 2003

description

Comparing Two Measurement Devices. Brian Novatny 2003. Each measurement made by an instrument or measuring device consists of the true, unknown level of the characteristic or item measured plus an error of measurement. - PowerPoint PPT Presentation

Transcript of Comparing Two Measurement Devices

Page 1: Comparing Two Measurement Devices

42510011 0010 1010 1101 0001 0100 1011

Comparing Two Measurement Devices

Brian Novatny2003

Page 2: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Background• Each measurement made by an instrument

or measuring device consists of the true, unknown level of the characteristic or item measured plus an error of measurement.

• In practice it is important to know whether or not the variance in errors of measurement of an instrument, or the imprecision of measurement, is suitably small as compared to the variance of the characteristic or product measured, or the product variability.

Page 3: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Background

• For efficiency of the measuring process, the variance in errors of measurement should be several or many times smaller than the variability of the characteristic measured or product variance.

• High measurement error causes the power of most statistical tests to decrease unless compensated for by larger sample sizes– Power = 1 - Beta– Beta is the Type II error

Page 4: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Measurements are indicated by

• Device 1 = B1 + Xi + Ei1

• where B1 = bias for device 1• Xi = true value• Ei1 = random errors for device 1

• Device 2 = B2 + Xi + Ei2

Page 5: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Regression Approach

• Y = b0 +b1*X• Device 1 = Intercept + Slope * Device 2

• Intercept value should be zero, if not, it indicates bias of the two devices

• Slope term should be around 1 indicating “Similarity”

• Mean Square Error estimate “Precision”• R-Square term estimates some measure of

strength

Page 6: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Problems with Regression

• The X variable (independent variable) should be measured without error– this is never the case, but errors in the X variable should be

small, and it won’t be when comparing devices– they will be on equal footing

• Asymmetry - specifically designate one device to predict the other– prefer symmetric approach where is doesn’t matter which

variable is the input and which is the output

• Inverse regression can pose several problems when trying to resolve the asymmetry problem

Page 7: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Problems with Regression

• What should the R-square value be?– No objective justification

• How to handle the case when there are multiple measuring devices?– Pairwise comparisons– multiple testing error problem

Page 8: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Some Solutions

• Grubbs model– uses sums and differences

• Pittman and Morgan along with Maloney and Rastogi– provide proofs and refinements to Grubbs model

• Blackwood and Bradley– multivariate test on bias and precision

• Tan and Iglewicz modify the standard regression approach based on Mandel’s work of Errors in Variables

Page 9: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Grubbs Model

• Involves calculating the Sums and Differences of the two devices

• The differences will be used to estimate bias– standard paired t-test with n-1 degrees of freedom

• Performing a correlation analysis analysis on the sums and differences is used to estimate precision of the two devices– follow student’s t distribution with n-2 degrees of freedom

• Provides two independent tests– one for bias of the two measurement devices– one for precision equivalency

Page 10: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Simultaneous Test for Precision and Bias

• Multivariate approach instead of independent tests

• Differences are regressed on the Sums – Difference = Intercept + Slope * Sums

• Model F-test uses the UNCORRECTED SUMS OF SQUARES to get the correct number for the df– instead of the familiar corrected sums of squares– the overall Type I error (Alpha) rate is exact

Page 11: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Simultaneous Test for Precision and Bias

• If the Model F indicates significance, then tests for the Bias and Precision are just the individual F-tests– the overall test is generally more powerful – it can reject the equivalence assumption of the

two devices even though each individual test does not

Page 12: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Simultaneous Test for Precision and Bias

• The Precision test is exact

• The Bias test is exact only when the precision between the two devices is equal– use the paired difference t-test otherwise– more powerful– Uses UMVU (uniform minimum variance

unbiased) estimate of the variance

Page 13: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Advantage of Simultaneous Test

• Type I Error Rate is exact• Overall test could reject even though

individual test do not– power of test

• Statistical modeling– usual array of diagnostics

• residuals

Page 14: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Regression approach

• Regression can still be used, but an adjustment to the model must be made

• In Simple Linear Regression, the results (Beta Hat) are achieved by minimizing the sum of squared residuals in the direction of the dependent variable

• The correction is to achieve Beta hat by minimizing the sum of squared residuals in the direction of -Lambda/Beta hat

Page 15: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Regression approach

• Lambda is a calculated value to adjust the slope calculations– called the Precision Ratio

• ratio of the machines repeatability• machine 1 / machine 2 or inverse

• Lambda is determined by performing the standard Gauge RxR studies or taking repeated values

Page 16: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Regression approach

• The approach still uses regression and Lambda value helps solve the asymmetry problem– as lambda approaches infinity, implies X

values approach 0, so Beta hat is Sxy/Sxx, which is where X is the independent variable

– as lambda approaches 0, implies Y values approach 0, so Beta hat is Syy/Sxy, which is where Y is the independent variable

Page 17: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Regression approach

• Approach handles precision but not bias

• Uses Polar coordinates for confidence intervals– Slope = TAN (Theta)– Intercept = Tau/COS (Theta)

Page 18: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Example

Device1 Device2 Sum Diff 5.00 4.73 9.73 0.27

5.17 4.83 10.00 0.33 5.17 4.63 9.80 0.53 5.00 4.37 9.37 0.63 8.50 7.03 15.53 1.47 5.67 4.50 10.17 1.17 8.00 7.03 15.03 0.97 9.00 7.93 16.93 1.07 8.50 7.50 16.00 1.00 11.17 9.57 20.73 1.60 9.00 7.70 16.70 1.30

Page 19: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Standard Regression Analysis

• R-Square = 98.24%• SQRT MSE = 0.31• Device 2 = -0.28 + 1.2 * Device 1• 95% Confidence Intervals– Intercept (-1.07,0.52)– Slope (1.07,1.31)

• Conclusion– No Bias, but not similar

Page 20: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Simultaneous Test for Bias and Precision

• Regress Diff on Sums (Uncorrected SS)• Model F = 73.22 ==> p-value = 0.0000027• Therefore, devices are different relative to their

bias and precision• Individual Precision Test– F = 17.5 ==> p-value 0.0024

• Individual Bias Test (not exact)– F = 128.93 ==> p-value = 0.0000012

• Paired t-test has a T value = 6.98 and p-value less then 0.00001

Page 21: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Correct Conclusion

• The two devices measure differently

• Strong Bias

• Strong lack of precision (repeatability)

Page 22: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Potential Problems

• Methods do not account for a difference in Gain, or slope of devices

• Devices might measure equally well or poor at the low and high ends of the scale, but the relationship is not constant– collect data at one end of the data range– power of the test could be compromised

Page 23: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Multiple Measuring Devices

• Grubbs and others propose technique for three measuring devices– comparisons when one device is a “Standard”– with three devices, get a more powerful test

• Multivariate methods lead to fuller choice of sub-hypothesis and can be used regardless of the number of measurement devices

Page 24: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Multiple Measuring Devices

• One method involves performing a multivariate regression on q-1 measurement devices– independent variable = mean of each part– dependent variable = deviations from that mean

• Independent variable is averaged each part across all the measurement devices

• Dependent variable is calculated by the differences of each value from that mean

Page 25: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Multiple Measuring Devices

• Generally have to fit a Full model and a Reduced model (intercepts only)

• Then compare the two models – usually through some matrix manipulation

• Technique can be performed by most software packages that can perform MANOVA techniques

Page 26: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Authors Opinion

• As the title says, this is just my opinion and not based on any concrete proof– such as simulation studies

• My preferred method of analysis would be the Multivariate approach using Blackwood and Bradley’s Regression with the Uncorrected Sums of Squares– this procedure seems to have a more powerful

test in finding differences– eliminates the possibility of getting a negative

variance, which Grubbs method could get

Page 27: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Authors Opinion

• With the Multivariate approach, there is a natural extension to testing more than two measuring devices

• Of course, there is no reason to try both the multivariate approach and Grubbs approach since they are easily computed using standard data analysis techniques

Page 28: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Final Comments

• These methods are not to replace Gage RxR studies, but to evaluate two devices against each other

• Each device should be tested for bias and repeatability and linearity as desired– corrective action should be taken as needed

Page 29: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

Final Comments• The test for Bias is only a test for

agreement between the two devices, not a bias against a standard– both devices could be grossly off from the

standard (but in the same direction and amount)

• If there is a claim that one device is “superior” to another (better precision), these methods could prove the validity of the claim and provide the precision estimates

Page 30: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

References for Two Device Comparison

• Grubbs, F.E. (1973). “Errors of Measurement, Precision, Accuracy and the Statistical Comparison of Measuring Instruments” , Technometrics Vol. 15 pp. 53-66

• Bradley, E.L. and Blackwood, L.G (1991). “An Omnibus Test for Comparing Two Measuring Devices”, Journal of Quality Technology, Vol. 23 pp. 12-16

• Tan, C.Y. and Iglewicz, B. (1999). “Measurement-Methods Comparisons and Linear Statistical Relationship”, Technometrics, Vol. 41 pp. 192-201

Page 31: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

References for Multiple Device Comparison

• Christensen, R. and Blackwood, L.G (1993). “Tests for Precision and Accuracy of Multiple Measuring Devices”, Technometrics, Vol. 35 pp. 411-420

• Bedrick, E.J. (2001). “An Efficient Scores Test for Comparing Several Measuring Devices”, Journal of Quality Technology, Vol. 33 pp. 96-102

Page 32: Comparing Two Measurement Devices

4251

0011 0010 1010 1101 0001 0100 1011

[email protected]