APPLICATION OF CUSUM ANALYSIS IN THE · PDF fileBMJ 1998;316;1662-23 Breast and Endocrine...

APPLICATION OF CUSUM ANALYSIS

IN THE ASSESSMENT OF (TRAINEE)

PERFORMANCE IN THYROID SURGERY:

AN AID TOWARDS QUANTITATIVE

BENCHMARKING?

Hisham AN and *Lim TO

Department of Breast and Endocrine Surgery

Putrajaya Hospital and *Clinical Research Centre,

Kuala Lumpur Hospital, Kuala Lumpur,

Malaysia

Background

The social contract between doctors and the public

is being renegotiated. The contract says:

“In return for guaranteeing that we will be treated by

competent doctors who will respect our dignity and

offer services better than those of local garage we the

public will give you doctors status, above average

income and the privilege of regulating yourselves”.

Richard Smith, Editor

BMJ 1998;316;1662-23

Breast and Endocrine Surgery

Putrajaya Hospital

Background

This contract is being renegotiated by the public

through; i.e. parliament, the media, patients’

organization and by the profession.

If this contract falls apart: self regulation will be lost.

Public empowerment:

Media

Increase litigation

Medical legal issues


Putrajaya Hospital

Background


Putrajaya Hospital

Tun Abdullah Badawi,

Former PM Malaysia

The development of human capital and research and development…is absolutely crucial …..

For economic prosperity to be widely shared, it must be founded upon progress in the areas of research & innovation as well as human and institutional capacity building. There is a need, therefore, to develop clear and comprehensive strategies that will foster the development of human capital …”

Background


Putrajaya Hospital

3 MANTRA UTAMA:

SATU MALAYSIA

RAKYAT DIDAHULUKAN.

PENCAPAIAN DIUTAMAKAN.

ONE MALAYSIA.

PEOPLE FIRST. PERFORMANCE NOW

Y.A.B. DATUK SERI NAJIB

PRIME MINISTER OF MALAYSIA

03.04.2009

Background

People first and Performance Now

Key Performance Index: Predefined standard of performance

and to be achieved in respective areas

National Key Result Areas: Benchmarking/ Standard set

Achievement of the target set


Putrajaya Hospital

Background


Putrajaya Hospital

Module1: Redefining Performance

• Explain the importance of measuring the

“what” and the “how” that makes up performance.

• Establish metrics to measure what is achieved.

Module 2: Importance Of Managing Performance

• Understand the importance of establishing Performance

Measurements in an organization.

• Learn what it takes to be a Performing Knowledge

Worker

Background


Putrajaya Hospital

Module 3: Managing KEY RESULT AREAs (KRAs)

• Define Key Result Areas and Its Importance

• Define what is an “Area of Focus” and identify

what are the attributes?

• Identify and Derive KRAs

• Establish the Relationship between the KRAs

& day-to day activities

• How to transform and adopt KRAs for practical usage

Background


Putrajaya Hospital

Module 4: Establishing KEY PERFORMANCE INDICATORS (KPIs)

• Identify the Key Components of KPIs and the different

category of Performance Measurements.

• Identify the different levels and types of target.

• Learn how to determine the “right” targets.

Background


Putrajaya Hospital

Module 5: Performance Tracking

• Develop action plan to measure, monitor & manage

performance

• Manage information, Contacts & Activities

• Learn what it takes to overcome pitfalls

Background

How do we meet the standard of performance?

Good clinical practice: set the standard for doctors

expected to meet. (in a positive way)

How are we to show that the profession and its

member are keeping up with the latest evidence

and maintaining their skills?


Putrajaya Hospital

Background

Current practice in monitoring performance:

Close monitoring by the supervisor

Log book: observed, assisted and performed

Numbers/quota for related procedures

Certification in basic and advanced surgical skills (lab)

Subjective endorsement of technical skills:

Despite good paper qualifications, fear and doubts:

young surgeons’ (trainees’) performance


Putrajaya Hospital

Quality Assurance study

(Outcomes)

Safety of procedure

(techniques)

Patients and diseases

(Patients)

Performance variation

(Surgeons)

Huge Goitre

Retrosertnal goitre

Fungating goitre

Background

Measuring and comparing surgical outcomes:

-Morbidity and mortality: departmental level

i.e. wound infection, anastomotic leakages, reoperation

-Event reporting

-POMR (centers/not individual)

Vague or inappropriate to be useful,

Poor individual performer could be dismissed as casemix

problem since stratifications were not available

Retrospective review


Putrajaya Hospital

Background

Assessing the trainee surgeon early in a training

program to ensure that satisfactory outcome might

be anticipated benefits the trainee and the program

Identifying the poor performer for remedial purposes

or exclusion is often difficult without the implication

of subjective observer bias.

The use of knowledge-based comparative test score

does not correlate well with technical ability.


Putrajaya Hospital

Background

The process of assessment in the surgical practice

remains subjective without any quantitative reference

to a predefined standard.

To date there is no universal acceptable method

of monitoring performance and competency apart

from close supervision and observation.


Putrajaya Hospital

It is one of a series of statistical tests developed in

World War II as quality control tests in production

lines.

It is a cumulative sum (or trending) of predefined

scores (direct and indirect measures) versus the

index number of a series of consecutive procedures1-9.


Putrajaya Hospital

The CUSUM analysis:

The CUSUM analysis:

This is a sequential analysis to allow the

observer to decide if the production was “in control”

(within a defined quality boundaries) or had become

“out of control”

Identify the need to stop the process as well as to choose

the definitions of stopping the process


Putrajaya Hospital

Mathematically:

CUSUM score is determined after the performance

of each consecutive procedure when the outcome

measures is known.

It is designed to detect small but persistent shift

in the clinical process1-9. STATA multipurpose

statistical package is used to construct the chart

and analysis.


Putrajaya Hospital

The CUSUM analysis:

The formula for calculation relates to the nth procedure:

CUSUM Cn= max (0, Cn-1+Xn– k), where C0= 0,

Xn is the outcome measure for nth procedure, standardised

to have a zero mean and unit standard deviation (SD)

k is the reference value as determined by an agreed

standard of performance


Putrajaya Hospital

The CUSUM analysis:

Charting Trend

Acceptable level : the CUSUM curve plateau

Unacceptable level : the CUSUM produce a curve

that continue to slopes upward

and crosses the horizontal line1-7

(alarming the trend)


Putrajaya Hospital

The CUSUM analysis:

Background

In the continuous effort to improve the quality of

surgical care we employed the technique of CUSUM

(cumulative sum) analysis to evaluate its applicability

in the assessment of surgical trainee performance and

competency in thyroid surgery


Putrajaya Hospital

Background


Putrajaya Hospital

1. Pre-defined standard:

-Direct measurement: Death, near miss injury, etc

-Indirect measurement: duration, blood loss, etc

2. Stratification of cases.

Primary or reoperative surgery, etc

(focal reference point/international standard)

3. Individual performer. Trainees etc

Patients and Methods

A total of 189 consecutive patients who had primary

thyroid surgery were accrued in this prospective study

from June 1998 to December 1999.

The assessment was stratified into 3 groups:

a. 52 (27.5%) patients had hemithyroidectomy

b. 72 (38.1%) patients had total thyroidectomy

c. 65 (34.4%) patients had thyroid surgery under

local anaesthesia (LA).


Putrajaya Hospital


In the absence of published standard, we

accept the direct measures of identification

/preservation of the both the laryngeal nerves

as the predefined standard. When the nerve is

not seen it is considered as near miss injury.

1. The recurrent laryngeal nerve (k:95%)

2. The external laryngeal nerve (k:60%).

(In thyroid surgery if the RLN is injured,

the outcome is immediately known) Breast and Endocrine Surgery

Putrajaya Hospital

Recurrent laryngeal nerve


Putrajaya Hospital

2-3% of all medico-legal claims in General Surgery

involved the consequences of thyroid surgery and

nearly exclusively these were related to RLN

(BJS 1994;81;1555-6)

Large statistical series (12,000) of thyroid surgery

reported the rate of permanent RLN palsies:

1.2% when nerve is seen and exposed

5.2% when nerve is not seen and exposed

(Surgery 1994;115;139-44)

Whatever technique is used in thyroid surgery,

virtually all endocrine surgeons today would

consider it unacceptable not to identify the RLN

during thyroid surgery.

Thompson NW

Surgery 1973

The technique of thyroidectomy was adopted

and standardized as described previously

described .

Delbridge et al

ANZ J Surg 1992


Putrajaya Hospital


Techniques of thyroid surgery

82.5% (52) dissection the RLN was posterior to

the inferior thyroid artery

17.5% (11) dissection the RLN was anterior to

the inferior thyroid artery

Tubercle of ZuckerkandlRec. laryngeal nerve


Putrajaya Hospital

Technique of identifying the RLN

Two extra-laryngeal branches of the

RLN passing behind the grade 3

Zuckerkandl's tubercle

Hisham et al:

ANZ J Surg 2002:70;251-253

ZT

Grade 3 Zuckerkandl's tubercle

causing significant pressure symptoms

The recurrent laryngeal nerve (k:95%) Breast and Endocrine Surgery

Putrajaya Hospital

Not seen Single branch 2 branches 3 branches NRLN

General: 11(2.2%) 323(64.4%) 164(32.6%) 3(0.6%) 1(0.2%)

Hisham et al, ANZ J Surgery 2002:72:887-9

Secondary:

Benign re-op 7(10.6%) 41(62.1%) 17(25.8%) 1(1.5%)

Completion Ca 1(4.1%) 19(79.2%) 4(16.7%) 0(0.0%)

Recurrent Ca 11(36.7%) 17(56.7%) 2(6.6%) 0(0.0%)

Hisham et al, (in press)


Position Anterior/On the gland Tracheo-esophageal groove Lateral/post

General 23(6.0)% 231(60.8%) 18(4.9%)/109(28.3%)

Hisham et al, ANZ J Surgery 2002:72:887-9

Secondary

Benign 21(35.6%) 28(52.5%) 7(11.9%)

Completion 0 16(69.6%) 7(30.4%)

Recurrent 1(5.3%) 12(63.1%) 6(31.6%)

Reference 6.6% 70% 23.4%

Hunt et al BJS 1968;55:63-66


External laryngeal nerve

Editorial review

The Neglected nerve in thyroid surgery: The case of routine

identification of the external laryngeal nerves.

In this article the authors (Hisham et al, ANZ J Surg 2001:71;212-4)

state that routine identification of ELN should be the benchmark to

avoid nerve injury.

In achieving an identification rate of more than 90% they have

certainly set a very high benchmark for other endocrine surgeons

to reach.

Professor L Delbridge

ANZ Surg 2001:71;199

The external laryngeal nerves (k:60%)Breast and Endocrine Surgery

Putrajaya Hospital

Cernea (1992)

Head & Neck


Type 1

23 (35.9%)

Type 2a

24(36.2%)

Type 2b

4(6.3%)

1 cm

Not seen

13 (20.3%)

Type 1

35 (16.06%)

Type 2a

113(54.84%)

Type 2b

54(24.77%)

Not seen

16 (7.34%)

Hisham (2001)

ANZ J Surg


Putrajaya Hospital

Hisham (2001) 14(8%) 24(12%) 81(41%) 77(39%)

Eur J Surg

The external laryngeal nerves (k:60%)

Type 1 Type 2a Type 2b

Cernea et al

(Head and Neck 1992:14(5):380-383)


Secondary Not seen Type 1 Type 2a Type 2b

Benign re-op 52(72.2%) 2(2.8%) 10(13.9%) 8(11.1%)

Completion Ca 5(20.8%) 4(16.7%) 14(58.3%) 1(4.2%)

Recurrent Ca 22(71.0%) 2(6.4%) 7(22.6%) 0 (0.0%)

(In press)

In general 14(8%) 24(12%) 81(41%) 77(39%)

Eur J Surg 2001;167:662-5

1 cm



We assessed the direct measures of:

1. Identification and preservation of RLN

2. Identification and preservation of ELN

3. Conversion rate of thyroidectomy under LA

4. Mortality and Morbidity


Putrajaya Hospital


We also assessed the indirect measures of:

1. Duration of operation,

2. Amount of blood loss and blood transfusion,

3. The weight of goiter,

4. The visual analogue scoring scale for

pain score and rating of the procedure

(applied to those patients who had

thyroid surgery under LA)


Putrajaya Hospital

CUSUM controlled chart

Performance of Consultant and Trainee

Identification of Recurrent Laryngeal NerveOutcome:

Identification

of RLN

CUSUM for Series of Thyroid Surgery

Trainee

Surgeon

n

0 10 20 30

0

.92

0

Total thyroidectomy

Hemi thyroidectomy

1.84

Results


Putrajaya Hospital

ResultsCUSUM controlled chart

Performance of Consultant and Trainee

Identification of External Laryngeal NerveOutcome:

Identification

of ELN

CUSUM for series of Thyroid Surgery

n

Trainee

Surgeon

0 10 20 30

0

0

Total thyroidectomy

Hemi thyroidectomy

1.9

3.8 1.9


Putrajaya Hospital

CUSUM controlled chart

Performance variation: Duration of operation

ResultsC

US

UM

CUSUM for series of Thyroid Surgery

n0 5 10 15 20 25 30

0

4.4

8.8

13.2

0

4.4

Trainee

Surgeon

Total thyroidectomy

Hemi thyroidectomy


Putrajaya Hospital

Results Performance variation

CUSUM controlled chart: Duration of operation

Duration of op

CU

SU

M

CUSUM for series of

Thyroid surgery under LA

n

0 5 10 15 20 25 30 35 40

0

4.4

8.8

13.2

17.6

Trainee

Surgeon


Putrajaya Hospital

(Hisham et al, ANZ J Surg 2001:71;212-4)

Results CUSUM controlled chart

Performance variation : Pain score of operation

VAS pain score

CU

SU

M

CUSUM for series of

thyroid surgery under LA

n

0 5 10 15 20 25 30 35 40

0

4.4

8.8

13.2

17.6

Trainee

Surgeon


Putrajaya Hospital

(Hisham et al, ANZ J Surg 2002:72;287-9)

Conclusions

1. The application of CUSUM analysis in the

assessment of trainee performance in thyroid

surgery should be given due consideration as

an aid towards quantitative benchmarking 1-9.

2. It offers the best objective method of monitoring

the performance of trainee to the maintenance of

competency of the surgeon over a period of time3-7.


Putrajaya Hospital

Conclusions

3. Nonetheless, there remain unresolved issues over

the confidentiality and its medico-legal implications

to the quality of care in the surgical practice1-2.


Putrajaya Hospital

References

1. Johnson JN. Making self-regulation credible. Br Med J 1998;316:1847-1848.

2. Irvine D. The performance of doctors. I: Professionalism and self-regulation in

a changing world. Br Med J 1997;314:1540-2.

3. Bolsin S, Colson M. The use of the Cusum technique in the assessment of trainee

competence in new procedures. Int J Qual Health Care 2000;12:433-8.

4. Van Rij AM, McDonald JR, Pettigrew RA et al. Cusum as an aid to early assessment

of the surgical trainee. Br J Surg 1995;82: 1500-1503.

5. Williams SM, Parry BR, Schlup MM. Quality control: An application of the cusum.

Br Med J 1992;304: 1359-1361.

References

6. Kestin IG. A statistical approach to measuring the competence of anaesthetic trainees

at practical procedures. Br J Anaesth 1995;75:805-809.

7. Parry BR, Williams SM. Competency and the colonoscopist: A learning curve.

Aust NZ J Surg 1991; 61:419-422.

8. De Leval, MR. Analysis of a cluster of surgical failures: Application to a series of

neonatal arterial switch operations. J Thorac Cardiovasc Surg 1994;107: 914-924.

9. Bartlett A, Parry B. Cusum analysis of trends in operative selection and conversion

rates for laparoscopic cholecystectomy. Aust N Z J Surg 2001;71:453-456.

1: Brown SM, Benneyan JC, Theobald DA, et al

Binary cumulative sums and moving averages in nosocomial infection cluster detection.

Emerg Infect Dis. 2002 Dec;8(12):1426-32.

2: Robinson IA, Blackham RB, Cozens NJ, Sharp J.Related Articles,

Good practice in head and neck fine needle aspiration cytology as assessed by CUSUM.

Cytopathology. 2002 Dec;13(6):335-42.

3. Forbes TL, De Rose G, Harris KA.Related Articles,

A CUSUM Analysis of Ruptured Abdominal Aortic Aneurysm Repair.

Ann Vasc Surg. 2002 Sep;16(5):527-33.

4. de Oliveira Filho GR.Related Articles,

The construction of learning curves for basic skills in anesthetic procedures:

an application for the cumulative sum method.

Anesth Analg. 2002 Aug;95(2):411-6, table of contents

5: Buchmann P, Steurer J.Related Articles,

[The learning curve in surgery: possibilities and limits of this method]

Swiss Surg. 2002;8(3):106-9. German

6: Lim TO, Soraya A, Ding LM, Morad Z.Related Articles,

Assessing doctors' competence: application of CUSUM technique in monitoring doctors' performance.

Int J Qual Health Care. 2002 Jun;14(3):251-8.

References

http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=12498659&dopt=Abstract



http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Display&dopt=pubmed_pubmed&from_uid=12485168









7:Novick RJ, Fox SA, Stitt LW, Kiaii BB, et al Links

Assessing the learning curve in off-pump coronary artery surgery via CUSUM failure analysis.

Ann Thorac Surg. 2002 Jan;73(1):S358-62. No abstract available.

8. Novick RJ, Fox SA, Stitt LW, Swinamer SA, Lehnhardt KR, Rayman R, Boyd WD. Links

Cumulative sum failure analysis of a policy change from on-pump to off-pump coronary artery bypass grafting.

Ann Thorac Surg. 2001 Sep;72(3):S1016-21.

PMID: 11565718 [PubMed - indexed for MEDLINE]

9: Bartlett A, Parry B.Related Articles, Links

Cusum analysis of trends in operative selection and conversion rates for laparoscopic cholecystectomy.

ANZ J Surg. 2001 Aug;71(8):453-6.


10: Ravn LI, Sprehn M, Pedersen CB.Related Articles, Links

[The Cusum score. A tool for evaluation of clinical competence]

Ugeskr Laeger. 2001 Jun 25;163(26):3644-8. Danish.


References








Thank You


Putrajaya Hospital


Putrajaya Hospital

APPLICATION OF CUSUM ANALYSIS IN THE · PDF fileBMJ 1998;316;1662-23 Breast and Endocrine...

Documents

Transcript of APPLICATION OF CUSUM ANALYSIS IN THE · PDF fileBMJ 1998;316;1662-23 Breast and Endocrine...