Joseph M. Juran Center for Research in Supply Chain, Operations, and Quality · 2014-03-07 ·...

Juran Center for Leadership in Quality

Professor Kevin LindermanAcademic Co-Director of Joseph M. Juran Center for Research in Supply

Chain, Operations, and Quality Carlson School of Management

University of Minnesota

Joseph M. Juran Center for Research in Supply Chain, Operations, and Quality


Origins of the Juran Fellows Program


“This will be a place where leaders engage with scholars to shape critical

questions, where new knowledge is developed, translated, and

disseminated, and, above all, where Quality scholars are trained. It is my

hope that, generations from now, historians will note that we helped

create the ‘Century of Quality.’”J.M. Juran

Quote from Dr. Juran


Juran Fellowship AwardApplicationContent1. Career information 2. Executive summary3. Project description (up to 15 pages)

4. Short curriculum vitae (1-2 pages)

5. Letter of recommendation from a faculty advisor


Evaluation Criteria 1. Importance of the Problem

(20% of score)2. Connection to Existing Quality Research

(20% of score)3. Appropriate Methodology

(20% of score)4. Relevance to Scholarship and Practice

(20% of score)5. Timeline and Probability of completion

(10% of score)


Community of Juran Fellows

6


Community of Juran Fellows


The Future…

From seedlings to orchards


Recent Juran Fellow Winners

Anna ErroreUniversity of Palermo (Italy)

Ujjal Kumar MukherjeeCarlson School of Management


Anna ErroreUniversity of Palermo (Italy)Visiting PhD student at the Carlson School of Management

Definitive Screening Designs for Quality Improvement


Anna ErroreUniversity of Palermo, Italy

(visiting PhD student at Carlson School of Management, University of Minnesota)

William LiChristopher J. Nachtsheim

Carlson School of Management, University of Minnesota

Definitive Screening Designs and Discrete Choice Experiments for Quality Improvement


DOE in Business and Industry

All models are wrong; some models are useful.

Experiments, statistically designed or not, are a component of the learning process. How well one succeeds will be a function of adherence to the scientific method, the most rapid means for speeding the learning process.

George Box, 1978

Juran, J. M., and Godfrey, A. B. (1999). “Juran’s quality handbook (Vol. 2)”. New York, McGraw Hill.


In quality applications DOE plays a fundamental role in a huge variety of situations, from design and development of new products, to product or process improvement practices.

In conducting experiments one of the most challenging tasks is the design of the experimental settings.

Assumptions Design Experiment Analysis

DOE in Business and Industry


The term `Screening Design' refers to an experimental plan that is intended to find the few significant factors from a list of many potential ones.

Primary purpose is to identify significant main effects Three empirical principles

Effects sparsity Effects heredity Effects hierarcy

Screening Experiments

Source: Li, Xiang, Nandan Sudarsanam, and Daniel D. Frey. "Regularities in data from factorial experiments." Complexity 11.5 (2006): 32-45.


Example

The reactor experiment of Box, Hunter, and Hunter (2005)

5 factors: Feed Rate (A), Catalyst (B), Stir Rate (C), Temperature (D), Concentration (E).

The response is the percent reacted (y).

Example of a Screening Experiment


The fitted equation is:y = 65.5+9.75B+ 5.375D−3.125E +6.625BD−5.5DE

Obtained with a full factorial experimental design: 25 = 32 runs

In practice full factorial designs are very rare, especially for screening, because the run size increases exponentially with the number of factors.

Example of a Screening Experiment


Design efficient screening designs Linear models (quantitative response) Non-Linear models (qualitative response)

Identify and refine methods for the analysis

Purpose of this project

CostInformation

Get the right information at the minimum cost


Over reliance on a single model

Optimal design depends on a pre-stated model

Full Factorial designs optimal for full-factorial model

Screening designs assume only first-order effects active(resolution III fractional factorial, Plackett Burman designs)

If higher order effects are active results may be misleading


Statistical efficiency Minimum alias (confounding) of the effects Small run sizes

Desirable design features

CostInformation


Definitive screening designs

Jones, B., & Nachtsheim, C. J. (2011) A Class of Three-Level Designs for Definitive Screening in the Presence of Second-Order Effects, Journal of Quality Technology 43(1), 1-15.

2012 ASQ Brumbaugh Award

2012 Lloyd S. Nelson


Definitive screening designs

Errore, A., Jones, B., Li, W., Nachtsheim, C. J. (under review) Two-Level Definitive Screening Designs.

2 level factors

3+2 level factors

3 level factorsFirst paper of this

project(under review)Presented at

Spring Research Conference 2013

“Statistics in Industry and Technology”

University of California Los Angeles

June 20th -22nd


Three-level DSDs

Jones, B., & Nachtsheim, C. J. (2011) A Class of Three-Level Designs for Definitive Screening in the Presence of Second-Order Effects, Journal of Quality Technology 43(1), 1-15.


Jones, B., & Nachtsheim, C. J. (2013). Definitive Screening Designs with Added Two-Level Categorical Factors. Journal of quality technology, 45(2), 121-129.

Three-level plus two-level DSDs

Added 2-level categorical factors

Two construction methods: DSD-augmented,

unbiased designs Orth-augmented,

orthogonal designs


Goals:Clear the estimation of the ME from the 2fiHigh D-efficiency in the ME modelSmall run size

Alias = 0

Orthogonal or nearly orthogonalMin run size

Only 2-level factors Categorical factors or too costly to pick three levels Need to correctly identify active factors

Two-level DSDs

Juran Center for Leadership in QualityIntercept ME

2m

1 m

Construction method

m

m

Start random

Search exchange

Judge improvements


Example

-1 1 -1 1 -1 -1 -11 1 1 -1 1 -1 -1

-1 1 -1 -1 -1 -1 1-1 1 1 1 -1 1 11 1 -1 -1 -1 1 -1

-1 -1 1 -1 -1 1 -1-1 1 -1 -1 1 1 -11 -1 1 -1 1 1 1

-1 -1 -1 1 -1 1 11 -1 1 1 1 1 -11 -1 -1 -1 1 -1 -1

-1 -1 1 1 1 -1 11 1 -1 1 1 -1 11 -1 1 1 -1 -1 1

1. Optimize a 7x7 design

2. Foldover the 7x7 design to a 14x7

Screening design with 7 factors


Example –comparisons with other designs

7 factors

Typical design choices

- Fractional FactorialResolution III

8 runs


7 factors

Typical design choices:

- Plackett Burman

12 runs



Example

7 factors

- Margolin (1969)Weighing design

D-efficiency = 79.12 % Ave abs corr= 0.4286



Example

7 factors

- Two-level DSD

D-efficiency = 91.65% Ave abs corr= 0.1837



Current and future work

Design Two-level DSDs (paper under review) Extension to non-linear models and discrete choice

experiments (WIP) Analysis

Identification and selection of the correct active terms in the analysis stage (WIP)


LEARNING FROM HIGH TECH INNOVATION FAILURES:APPLICATION OF ‘BIG DATA’ PREDICTIVE ANALYTICS ON MEDICAL DEVICE FAILURES

Ujjal Kumar MukherjeePhD Student, Carlson School of Management, UMN

Kingshuk K. SinhaProfessor, Mosaic Company Professor of Corporate Responsibility

Supply Chain & Operations, Carlson School of Management, UMN

March 11th, 2014

Juran Center for Leadership in QualityJuran Center for Leadership in Quality

OVERALL RESEARCH AGENDA

Understanding sources of technology failures in medical devices Technology life-cycle (age of underlying technology) Differential impact on sources: Stages of development and manufacturing

Predicting device failures Failure risk estimation based on unstructured market data (user experience) Predictive analytics of ‘Big’ unstructured datasets Improving precision of prediction Judgment bias in failure detection

Improving usage and adoption of new technology in healthcare Context of surgical robot Field study in a large multi-specialty hospital in US Looking at doctors’ and surgical team learning and adoption Organizational capability building of complex and critical technology in the

context of healthcare


INNOVATION FAILURES IN THE MEDICAL DEVICE INDUSTRY

Source: FDA Database

• “Innovation failure” adds substantial cost and inconvenience to the system• In the second quarter of 2012, 123.5 million device units were recalled • Increasing complexity of devices as well as increased speed of innovation. • Failures are not desirable but are not always avoidable. • Important to mitigate the impact of failures if one were to happen. • Therefore, failure risk prediction is critical to ensure quality of innovation.


OBJECTIVES OF THE RESEARCH

In this study we are using a large unstructured dataset and BIG DATA analytics to build a PREDICTIVE MODEL of failure risk of high tech innovations.

Research Questions:

Do field failure data provide a credible signal for early detection and prediction of innovation failure risk? (Prediction)

What technology, firm or industry factors individually and in combination can improve the precision of prediction of innovation failure risk? (Precision)

Do firms systematically exhibit a judgment bias in interpreting signals related to innovation failures? What factors influence the judgment bias, if present? (Consistency)


FAILURE PREDICTION: USING SIGNAL DETECTION THEORY

Rate of adverse effects reported by users as signal of failure

Under normal working conditions some adverse effect reports are likely - Noise

However, systematic failures lead to abnormal adverse effect rate - Signal

The idea is to monitor market reports on an ongoing basis

Build predictive risk models for device failures

Classify user reports into likelihood of device failures

Statistically, separating the Signal from Noise is important.


VISUALIZING PREDICTION OUTCOME IN SIGNAL DETECTION THEORY (SDT)

Predicted

False True

Actual False Correct Rejection Rate False Alarm Rate (FAR)

True Miss Rate (MR) Hit Rate (HR)

Accuracy of the Prediction: ‘Area under the curve (AUC)’ of the ROC curve

Receiver operating characteristics (ROC) curve

Both are expensive to the firm


HYPOTHESES

Hypothesis 1 (H1): Field failures predict innovation failure.

Hypothesis 2 (H2): Field failures supplemented by factors related to design, production, supply chain, regulation and technology life cycle of the innovation reduces the prediction error of the innovation failure.

Hypothesis 3A (H3A): Signal to noise ratio (in H2) is associated with systematic bias in the prediction of innovation failure. • High signal to noise ratio is associated with under-reaction bias and low signal to

noise ratio is associated with over-reaction bias.

Hypothesis 3B (H3B): Severity of innovation failure (in H2) is associated with systematic bias in the prediction of innovation failure. • High severity is associated with over-reaction bias and low severity is associated

with under-reaction bias.


DATA

Primary Data Source1 Adverse events FDA manufacturer and user reported adverse event database

(MAUDE) ~ 3 Million usable datapoints. Fairly unstructured with missing data and mis-classification (BIG DATA). (1992-2010)

2 Recalls FDA Recall database (2002-2010)

3 Firm level product development data

FDA approval database, Compustat database (1998-2010)

4 Industry competition FDA approval database, Compustat database (1998-2010)

5 Manufacturing changes and Global Manufacturing at the firm level.

FDA user and manufacturer facility registration database; FDA supplemental approval text files (1998-2010).

6 Supplier information FDA user and manufacturer facility registration database; FDA supplemental approval text files (1998-2010).

7 Device classification FDA approval database (1992-2010)

8 Usage classification FDA approval database (1992-2010)

9 Adverse event severity FDA / CDRH Patient reports database ~ 6 million data entries (1992-2010).

10 Implant / non-implant FDA / CDRH device database (1998-2010)

11 Device approval classification

FDA 510(k) and PMA / PMA supplement database (1998-2010)


CONCLUSION FROM SEVERAL PREDICTIVE ANALYTIC MODELS ON THE DATA

1. Density of adverse effects (Measure of rate) are significant predictor of recalls.

2. Product development intensity parameters are significant predictors of recalls.

3. Supply chain changes increases recall likelihood in the short run. However, data indicates that generally firms recover from short run glitches in the long run.

4. Frequency of manufacturing process changes are significant predictors of recalls.

5. Global manufacturing significantly increases the likelihood of recalls.

6. Competitive intensity increases the likelihood of recalls.

7. Regulation type PMA has higher likelihood of recalls compared to 510K.


PREDICTIVE MODELING AND MODEL SELECTION

Generalized Linear Mixed Models (GLMM)

Generalized Additive Models (GAM)

False Positive Rate

True

Pos

itive

Rat

e

True

Pos

itive

Rat

e

False Positive Rate

REGRESSION MODELS



ENSEMBLE CLASS MODELS

False Positive Rate

True

Pos

itive

Rat

e

True

Pos

itive

Rat

e

False Positive Rate

Boosting Classifier Random Forest Classifier



Method Initial Model Sensitivity (AUC)

Final Model Sensitivity (AUC)

LASSO - 0.79Mixed Logistic Regression 0.61 (0.593-0.624) 0.81 (0.765-0.823)Generalized Additive Model 0.64 0.86Cox Hazard Model 0.55 0.74Naïve Bayes 0.66 0.75Neural Network 0.65 0.85Support Vector Machine (Radial Basis Function Kernel)

0.59 0.74

Boosting 0.59 0.78Random Forest 0.82 (0.815-0.824) 0.95 (0.925-0.961)Number of devices: 5,873, Train set size: 98,650, Test set size: 28,641 and Total sample size (n) : 127,291

Considerably high prediction accuracy.


Variance

Seve

rity

Low High

Low

Hig

h

Under-reaction

Over-reactionOver-reaction

Failure JudgmentsVariance Mean Ratio vs. Judgment Bias

FIRMS EXHIBIT SOME JUDGMENT BIAS IN DETECTING FAILURE SIGNALS

Predictive analytic framework proposed will aid decision and judgment

Aimed to mitigating some of the cognitive biases

Make decision support system more data and information based

Experience and human judgment can never be fully replaced but can be aided with data

analytics


CONCLUSIONS

1. We have shown that it is possible to predict innovation failures with fairly high degree of accuracy.

2. We have also been able to analyze the influence of some critical covariates in improving the accuracy of predicting innovation failure.

3. We have shown that judgment bias exists in managing innovation failure.

4. Use of data analytic decision support can help improve precision of failure detection and mitigate judgment bias to a considerable extent.


ACKNOWLEDGEMENTS

Juran Center Dissertation Grant, 2013

Social and Business Analytic Collaboration (SOBACO), UMN Grant, 2013

American Hospital Association Grant, 2012

Dissertation Fellowship, Carlson School, UMN, 2014


THANK YOU


Juran Fellowship Award

WewelcomeyourParticipation!

Kevin [email protected]

Joseph M. Juran Center for Research in Supply Chain, Operations, and Quality · 2014-03-07 ·...

Documents

Transcript of Joseph M. Juran Center for Research in Supply Chain, Operations, and Quality · 2014-03-07 ·...