Statistical Issues in Assessing Hospital Performance [PDF, 694KB]
Statistical Process Control for Assessing LC … · Statistical Process Control for Assessing LC...
Transcript of Statistical Process Control for Assessing LC … · Statistical Process Control for Assessing LC...
Statistical Process Control for Assessing LC MS/MS
Performance
Michael Bereman
Department of Biological Sciences
North Carolina State University, Raleigh, NC
Skyline Users Group Meeting
2014 American Society for Mass Spectrometry Conference
Baltimore, Maryland
June 15, 2014
What is Statistical Process Control (SPC)?
No definition of quality - SPC describes “quality” as inversely proportional to
variance
• Special cause vs. chance cause variation
• 1st implemented in Japan post WWII
• Revived in USA by the automotive industry in the 1980’s
• Sales, marketing, finance, clinical diagnostics
Walter Shewhart
(1891-1967)
A powerful collection of tools used in achieving process stability and improving
capability through the identification and reduction of assignable causes of
variation
LC MS/MS is a Process
Performance Metrics
ID Metrics (from a standard)
- Total number of PSMs
- Total number of peptide IDs
- Total number of protein IDs
Peptide ID-free metrics
- Retention time
- Fwhm
- Peak asymmetry
- Peptide abundance
- Mass measurement
accuracy
uncontrollable
Temperature
Humidity
Peptide stability
Electronics
Autosampler
co
ntr
ollab
le
A process is everything required to turn an input(s) into the desired output
Shewhart Control Charts
x
+2σ
-2σ
Time
Center LineUpper control
line
Lower control line
-3 -2 -1 0 1 2 3
Process Metric
x+2σ-2σ
Fre
qu
en
cy
-3-2
-10
12
3
Pro
cess M
etric
x+
2σ
-2σ
Frequency
Primary tool in SPC – used for monitoring process output
Shewhart, W.A. J. Frankl. Inst. 226, 163, 1938
Pareto Analysis80/20 Principle - Identifies the most significant problems – vital few
x
+2σ
-2σ
Time
x
+2σ
-2σ
Time
x
+2σ
-2σ
Time
Peak Area
RT
FWHM
N=5
N=3
N=0
FWHMN
um
ber
of
ou
tlie
rs
5
3
0
RT Peak Area
Cu
mu
lati
ve %
Pareto Chart
Juran, J.M., Quality Control Handbook 1974
How Does One Monitor Performance in LC MS/MS
1. Run a standard before each analysis or at the beginning of each day
……………………………
2. Run a standard systematically throughout a study
……………………………
……………………..………
Reference set n=3….10
3. Run a reference set
QC std
Samples
QC stdQC std
QC stds QC std
Scan Cycle for Quality Control Standards
MS1 MS2 MS2 MS2 MS2 MS2 MS2
Targeted MS/MS
Time (min)
Full MS1 Scan
10 40
Targeted MS/MSMonitor several peptides across the gradient
1. RT reproducibility
2. Peak asymmetry
3. FWHM
4. Peptide abundance
5. Mass measurement accuracyMS related
LC related
10 20 30 40Time (min)
Rela
tive A
bu
nd
an
ce
Motivation for Tool Development
1. Quantitative method to assess instrument performance
2. Visual tools
3. Methods for process monitoring
4. Versatile (SRM, MS1, Targeted MS/MS, high RP, low RP, and vendor neutral)
5. Easy and fast!
Peak A
rea
QC number QC number
“good” “bad”
K.AEFVEVTK.L K.AEFVEVTK.L
Statistical Process Control in Proteomics (SProCoP)
Implemented as an external tool (App store) in Skyline
Bereman, MS et. al., JASMS 25(4) 581. 2014
Step 2 – Defining Output
User can specify number of
runs (i.e., Reference Set)
Default = 3
MMA in ppm
(only for high RP instruments)
Saves individual charts
Simple – Yet Powerful Charts to Assess
Performance1. Control Chart Matrix
2. Pareto Chart
3. Box Plot of MMA
Shewhart Control Charts in SProCoP
Mean (Ref set)
-3s
+3s
3 - Sigma Quality Performance (~1 out of 370 to be a false positive)
CV across all runs
Peptide sequence
Metr
ic
Interpretation of Control Charts
Detect systematic trends – instead of single semi random occurrences
1. Do not rely on single peptide/metric measurements
2. How do the other peptides for that particular metric perform?
3. Is there an obvious systematic trend?
4. How does the next QC run perform?
QC Num QC Num
Peak a
sym
Peak a
sym
1. Utilizing SProCoP – Monitoring LC MS/MS –
System Suitability
……………………………………………….
Reference set n=10
Bereman, MS et. al., JASMS 25(4) 581 2014
Bath Gas replenished
LCL
In Control
Out of Control
MS
1 P
eak A
rea
QC Num
Peak A
rea
QC Num
Bereman, MS et. al., JASMS 25(4) 581 2014
Importance of Systematic Evaluation – Problem may have not been
diagnosed as quickly
1. Utilizing SProCoP – Monitoring LC MS/MS –
System Suitability
Trap 2 (3.9 cm)
Trap 1 (4.0 cm)
Trap 3 (4.1 cm)
1 lysate + 5 QC stds
Trap + Column
Replace Trap
Trap + Column
2. Utilizing SProCoP – Identification of Sources of
VariationDo self packed traps affect the LC MS/MS process?
Bereman et al., MCP 2013
Pareto chart provides nice summary of data – and points to which metric is most variable
5 QC runs were ran on the first trap – thresholds established
Bereman, MS et. al., JASMS 25(4) 581 2014
2. Utilizing SProCoP – Identification of Sources of
Variation
RT PA fwhm P Sym
RT
(m
ins
)P
eak
are
as x
10
-9
MS1 Peak Area
Targeted MS/MS Peak Area
Σ (fragment ions)
Peak A
rea
Peak A
rea
0
2
4Ratio of Peak Areas
(fragment ions/MS1)*100
% o
f M
S 1
3. Longitudinal Instrument Tracking - SProCoP
Feb 2014 May 2014
N=140 QC standards
*
VLDALDSIK
VLDALDSIK
QC Number
1 6 12 19 26 33 40 47 54 61 68 75 82 89 96 104 113 122 131
1
2
3
4
5
UCL
LCL
CL
% M
S 1
Peak A
rea
Longitudinal Instrument Tracking - SproCoP
Future Metric – Spray Stability
10 20 30 40Time (min)
10 20 30 40Time (min)
34.6 35.4 34.6 35.4
Stable Spray Unstable Spray
stable unstable
Pea
k A
rea
Difficult to quantify
FFVAPFPEVFGK FFVAPFPEVFGK
Increased variance in abundances – decreased power to identify differences
and can lead to false positives
Qualitative detection is based on duty cycle
p=0.02
n=5 n=5
Runs Test to Determine When ESI Stability Changes
Runs Test
23min
2523 min 25
68 runs 16 runs
𝑺𝒊𝒈𝒏 = 𝑰𝑵+𝟏 − 𝑰𝑵
unstable spray stable spray
10 20 30
Bin data ~ 2 min
min
Unstable spray was smoothed (Boxcar n=3) to mimic stability
+ ++ +++++ -
--- - - - -
ab
un
dan
ce
+ +-+ -
+++ -
-
--+
- +
-
Retention Time
2 runs
Avg run
length = 8
10 runs
Avg. run
length = 1.6
Average Run Length Differentiates ESI Stability
32
34
36
38
40
42
44
0 20 40 60
Ru
n L
en
gth
28 38
Rela
tive
ab
un
dan
ce
RT
RT
stable
stable
Less stable
Less stable
Define spray stability based on reference set – track metric in control
chart
34.0 35.5
m/z= 692.87
m/z= 692.87
Summary – What Can SProCoP do for YOU – ?
*ESI Stability *Multivariate Methods
*Thursday Poster #288
*Features will be available in next version
RT reproducibility
FWHM
Peak asymmetry
*Robustness of trap
Chromatography
Peptide Intensity (peak areas)
MMA
Fragmentation efficiency
Mass Spectrometry
All from a couple of mouse clicks!
SProCoP