DETERMINING SURGICAL EFFICACY

Non-randomized andrandomized trialsJeremy Wright

AbstractAlthough good quality randomized controlled trials provide high level

evidence for informing practice, they can be difficult to undertake within

surgery. Non-randomized trials, whilst easier to conduct, often contain

many biases that make it difficult to establish causation. It is no surprise,

therefore, that much surgical practice is established through little

evidence of efficacy, often being driven by external factors such as

industry and personal preference. The ways in which surgical practice

has evolved despite the evidence will be discussed. The difficulties of

undertaking randomized and non-randomized trials will also be outlined.

Keywords clinical trials; laparoscopy; non-randomized; randomized;

randomized controlled trials

Introduction

The randomized controlled trial (RCT), or systematic review of

RCTs, has become widely accepted to be the ‘gold standard’

evidence for evaluating the efficacy of clinical treatments. Through

the rigorous randomization process, group differences between

those receiving the treatment and those not receiving the treatment

are minimized. Moreover, the control group provides a ‘non-

treatment’ group whose health outcomes are compared with those

receiving treatment. In surgical trials it is usual to compare a ‘new’

treatment with a ‘standard’ treatment. It is rare for patients to

undergo sham surgery for ethical reasons but when this does occur

the results can be surprising. Any measured treatment effect,

therefore, can more confidently be said to be as a result of receiving

treatment. There are, however, many situations, particularly

when the treatment effect is very large, when a RCT is not required;

for example, performing a tracheostomy for tracheal obstruction.

There are also situations when it is difficult or unethical to carry

out a RCT; for example, when a second look operation is required

to assess outcome but the procedure, which carries risks, does not

provide any benefit to the patient.

Nevertheless, some important surgical RCTs have been

conducted, offering good quality evidence to support or refute

treatment efficacy. The recommendations from these trials,

however, often take many years to be implemented, with some

appearing to have little impact on surgical practice.

In this article two well-established treatments for common

surgical problems will be looked at specifically: laparoscopic

cholecystectomy and laparoscopic adhesiolysis. These will be

Jeremy Wright MBBS MBA FRCOG is a consultant gynaecologist at Ashford

& St Peter’s NHS Trust and Clinical Director of the MSc in Advanced

Gynaecological Endoscopy at the Postgraduate Medical School,

University of Surrey, UK. Conflicts of interest: none declared.

SURGERY 27:9 371

examined to see how they have become routine surgical practice,

despite the evidence from clinical trials available at that time.

The difficulties of undertaking RCTs within surgery and the role

that high quality non-randomized trials play in informing

surgical practice will then be discussed.

Laparoscopic cholecystectomy

In 1992, laparoscopic cholecystectomy was developing rapidly in

the United Kingdom. Petelin, of ‘dissector’ fame, a technically

skilled laparoscopic surgeon from Texas, was popularizing the

technique of laparoscopic cholecystectomy1 and it was enthusi-

astically taken up by surgeons in continental Europe and the

United Kingdom. Whilst laparoscopic cholecystectomy was

becoming popular amongst some surgeons, Shanahan and

Knight2 from the Pancreato-biliary unit at St George’s Hospital,

London, UK were concerned by an increasing number of

common bile duct injuries occurring during the procedure. They

undertook a telephone survey of 10 hospitals in the South of

England, discovering that the incidence of bile duct injuries was

1:53 following laparoscopic cholecystectomy compared with

1:300 at open surgery.2 At that time, it was suggested that mini-

laparotomy cholecystectomy, performing the procedure through

an incision that was <5cms, was a safer procedure, with

a similar rate of common bile duct injury to that experienced

during traditional open cholecystectomy. Moreover, patients

undergoing a mini-laparotomy cholecystectomy appeared to

have a similar hospital stay to those undergoing laparoscopic

cholecystectomy.

As Shanahan and Knight2 point out, the higher risk of

common bile duct injury associated with laparoscopic cholecys-

tectomy might be due to a learning curve effect and the low risk

associated with mini laparotomy might be due to it being per-

formed mainly by committed biliary surgeons who were skilled

in the technique.

Although at the time there was a lack of evidence to support

the introduction of laparoscopic cholecystectomy, and what

evidence there was suggested that mini laparotomy was a safer

and quicker approach, laparoscopic cholecystectomy grew in

popularity amongst surgeons. Media exposure was fuelling

patient demand for this new procedure and the instrument

makers responded very quickly to the demand. It was also a time

when hospital expenditure was available, the hospitals being

caught up at that time in ‘fee for service’ and a fear of losing

patients. It could be argued, therefore, that laparoscopic chole-

cystectomy was introduced into the United Kingdom on the basis

of a case series of 1236 patients reported by Cuschieri et al.3 in

1991 in which there were no reported deaths and an audit report

by Dunn4 in which there were two deaths in 1653 cases. At the

same time Voyles et al.5 reported a series of 500 patients treated

laparoscopically without any common bile duct injury, suggest-

ing that the initial high complication rate was resolving, at least

in the hands of those publishing in the literature of the time.

In April 1996 Majeed et al. reported a prospective randomized

trial comparing laparoscopic and small incision cholecystectomy

in 200 patients.6 This well-conducted trial found no difference in

any of the outcome parameters except that the laparoscopic

procedure took significantly longer to perform. By this time

however, laparoscopic cholecystectomy was such a well-

� 2009 Elsevier Ltd. All rights reserved.

DETERMINING SURGICAL EFFICACY

established technique that the trial was largely overlooked and

most surgeons in training will now learn the laparoscopic tech-

nique and will rarely have the opportunity or need to perform an

open cholecystectomy.

A later meta-analysis of laparoscopic versus mini-laparotomy

cholecystectomy RCTs7 reported no differences in morbidity and

no difference in operative time in a subgroup of trials published

after the year 2000, when the procedure was more established

and surgeons were further along the learning curve.

Laparoscopic adhesiolysis

Postoperative adhesions are commonly regarded as a cause of pain

and repeated hospital admissions. Although a laparotomy

approach to adhesiolysis has never gained acceptance, the advent

of laparoscopy has meant that patients with chronic pain are often

offered laparoscopic adhesiolysis in an attempt to try and reduce

their pain. Although non-randomized studies of laparoscopic

adhesiolysis report a reduction in pain of between 38% and 87%,

there is also an associated morbidity of around 10%.8

In a well-conducted blinded RCT,8 patients were assigned to

a diagnostic laparoscopy alone or a laparoscopic adhesiolysis,

the procedure being concealed from both the patients and

assessors. The patients’ pain was subsequently assessed at 1 year

by visual analogue scores, and their use of analgesics and quality

of life scores were also measured. Participants in both groups

reported good pain relief and significantly improved quality of

life, with no differences between the two groups. The authors

concluded, therefore, that adhesiolysis cannot be recommended

as a treatment for adhesions in patients with chronic abdominal

pain.8

Despite the high quality evidence produced by this study,

laparoscopic adhesiolysis continues to be undertaken for patients

with chronic abdominal pain. It is associated with considerable

morbidity, inadvertent enterostomy, damage to the mesentery

leading to areas of ischaemic bowel and bleeding from the

mesenteric vessels. Recurrent adhesions from surgery also

remain a problem.

Difficulties in conducting randomized trials in surgery

There are many reasons why surgical RCTs pose difficulties,

although not all are entirely specific to surgery. Commonly cited

reasons are:

� Difficulties in the recruitment of patients. Studies have shown

that key reasons for patient reluctance to participate in RCTs are:

the dislike of possibly receiving the placebo or potentially inferior

treatment; the potential side effects or complications of the new

treatment; the trial treatment may not be the best option; and

a general unease with the research process. Surgical RCTs can

have the added difficulty in that the patient may have to agree to

have a surgical procedure, with its associated risks, when there is

a 50% chance that they are assigned to a ‘non-treatment’ group

where they may not receive the best therapeutic benefit. To some

extent, this problem can be overcome by the use of a crossover

design, where half of the patients receive the treatment in the

first phase of the trial and the other half receive it in the latter

phase of the trial.

� Surgical procedures tend to develop over time, with each

development stage improving the technique by a small amount.

SURGERY 27:9

Such small improvements require large sample sizes to demon-

strate differences between intervention and control.

� A new surgical procedure generally requires the acquisition

of new skills, which can take a considerable time to develop

before a trial can be conducted. Surgical trials are commonly

criticized for being conducted too early in the learning curve

when complication rates are higher and the chances of a posi-

tive outcome are lower. However, during the time that it takes

for the surgeon to develop these new skills, the new procedure

becomes accepted as ‘normal practice’ and therefore it becomes

difficult to justify or secure funding for a trial. A further problem

arises in being able to determine when the surgeon has the

expertise to participate in the trial, with studies varying in the

number of procedures necessary to achieve technical

competence.

� Changes in surgical procedures are often influenced by the

manufacture of surgical instruments and therefore a widespread

shift towards adopting the new procedure can occur. Once this

happens, it is difficult to undertake a trial on what has become

accepted practice.

� It can be difficult to blind patients to the group assignment

and therefore the knowledge of being in the ‘new treatment’

group may influence the perceived benefits from surgery.

With all these difficulties surrounding the design of surgical

trials, the majority are conducted in big centres where the

patient population may be different, often with tertiary referral

of complex cases, to that of a district general hospital. Moreover,

the expertise of the surgeons in the trials may be greater than the

average expertise in general hospitals. It is important, therefore,

that the results are interpreted with these potential biases in

mind.

Although randomized surgical trials are difficult to conceive

and there are enormous problems to overcome with patient

selection, blinding and ensuring that exclusions do not make

them irrelevant to the real world, they are very important in

establishing whether procedures offer clinical efficacy. Sadly,

funding for surgical trials is lacking and the confluence of

clinical trial specialists, statistical planning and surgical

expertise is rare. Ethical and research governance issues make

the implementation of a trials culture a difficult challenge when

ideally all patients undergoing a surgical procedure should be

invited to take part in a trial to continually and rigorously

test treatment efficacy and safety. Patients entered into trials do

better, irrespective of which arm they are in, than patients

who are not in trials and this adds to the compelling reasons to

properly and rigorously assess what we do to our patients and

why we do it.

Surgical procedures are of course very different from drug trials.

They are often conceived and subsequently expanded by individ-

uals or small research groups and compared to drug trials have

modest cost implications. They are said to have a high degree of

‘determinism’ in that the outcomes are clear and they are rarely, if

ever, compared to a placebo treatment. They are also not free from

selection biases in that only participants thought to be fit enough to

undergo the new procedure will be invited to have the procedure,

or alternatively, if a less invasive procedure is being developed,

only those who have a poor chance of survival with the standard

procedure will be invited to take part.

372 � 2009 Elsevier Ltd. All rights reserved.

DETERMINING SURGICAL EFFICACY

The value of non-randomized trials in surgery

Although deemed to provide the highest quality evidence, RCTs

can suffer from being underpowered, being applicable to a very

selective type of patient group, and having biases arising from

inadequate blinding and concealment. This is particularly impor-

tant when the treatment has side effects that allow patients to

detect that they are receiving the treatment and not the placebo. It

has been suggested, therefore, that the majority of RCTs are of

insufficient quality to establish causation with confidence.9 It is

also important to recognize that the value of randomization (i.e.

reduction in biases) may not be overly important for all studies,

particularly those that demonstrate a very large treatment effect.

Indeed, if the treatment effect is larger than the possible effect from

all of the potential biases, there is probably no need for

a randomized double-blind study. What is important, however, is

that the study is properly designed to minimize any biases and that

the treatment effect is large enough to account for the potential

affect of biases. Indeed, a non-randomized study needs to

demonstrate a larger treatment effect size than what you would be

prepared to accept from a RCT.

Since surgical RCTs are notoriously difficult to conduct, it is

unsurprising that the majority of research on the efficacy of

surgical procedures is undertaken through non-randomized

studies. Rigorous research design, however, can help to mini-

mize biases and reduce the confounding factors that often lead to

misinterpretation of the results. Confounding is a situation in

which extraneous variables correlate both with the dependent

variable and the independent variable and therefore may cause

the study to be subject to bias. Confounding may well have

occurred in the case series of laparoscopic cholecystectomy

previously reported in this article5 if the patients in the laparo-

scopic cholecystectomy group were fitter or less likely to have

had previous surgical procedures than those in the mini-lapa-

rotomy group. This selection bias is often present when a new

surgical technique is being tested as the surgical skills required

for the more complex cases have yet to be developed. Yet if the

laparoscopic group are at less risk of complications before they

receive the surgery, this is likely to lead to an incorrect conclu-

sion that laparoscopy is safer or equivalent in risks to mini-

laparotomy cholecystectomy.

One of the advantages of a non-randomized trial is that it is

possible to allow the surgeons to continue to operate using the

technique that is most familiar to them. Key to the design of a non-

randomized trial, therefore, is the careful selection of respondents

in the treatment and the control groups to ensure that both groups

are similar in terms of any potential confounding factors (for

example, body mass index, previous treatments, age, gender,

morbidity). If group differences do occur, it is important that

appropriate statistical tests are used to adjust for any potential

biases. A further element of the research design that can effectively

reduce any biases is the use of an objective outcome measure.

Measures such as pain and quality of life are very subjective, being

influenced by many factors. In a non-randomized and non-blinded

study, where patients are aware of the treatment that they have

received, it is better to use an objective outcome measure, which is

less likely to be influenced by external factors.

The surgical discipline is, however, adopting a ‘trials culture’

and there have been good if contradictory trials on laparoscopic

SURGERY 27:9 373

colectomy. An early American trial,10 showed no real difference

between laparoscopic and ‘open’ colectomy, but later trials showed

much reduced morbidity for successfully completed laparoscopic

colectomy.11 However, these trials show that conversion from

laparoscopic surgery to open surgery is associated with significant

increased morbidity and that laparoscopic rectal surgery remains

more morbid than laparoscopic colonic surgery.12

Conclusion

Whilst RCTs can provide good quality evidence to inform

surgical practice, the difficulties involved in undertaking these

types of trials often mean that they are underpowered and are

only applicable to a very specific group of patients. Good quality

non-randomized trials can also provide high quality evidence but

need to be designed so that biases are minimized, as well as

needing to demonstrate a large treatment effect. One of the

reasons why trials have not been extensively used in surgical

practice is that new interventions are often intuitively justified as

they have large treatment effects and offer favourable risk/

benefit ratio when applied to the right patients. Changes in

surgical technique are in fact little regulated other than by local

clinical governance rules. Changing clinical surgical practice,

particularly in the field of laparoscopic surgery, is increasingly

driven by the equipment manufacturers who have sponsored

surgical meetings, run their own courses and offer extremely

good, if selective, training to surgeons wishing to learn new

techniques. A

REFERENCES

1 Braasch JW. Laparoscopic cholecystectomy and other procedures.

Arch Surg 1992 Aug; 127(8): 887.

2 Shanahan D, Knight M. Laparoscopic cholecystectomy. BMJ 1992 Mar

21; 304(6829): 776e7.

3 Cuschieri A, Dubois F, Mouiel J, et al. The European experience

with laparoscopic cholecystectomy. Am J Surg 1991 Mar; 161(3):

385e7.

4 Dunn DC. Voluntary confidential audit of outcome of surgery. BMJ

1991 Nov 16; 303(6812): 1272.

5 Voyles CR, Petro AB, Meena AL, et al. A practical approach to

laparoscopic cholecystectomy. Am J Surg 1991 Mar; 161(3):

365e70.

6 Majeed AW, Troy G, Nicholl JP, et al. Randomized, prospective,

single-blind comparison of laparoscopic versus small-incision

cholecystectomy. Lancet 1996 Apr 13; 347(9007): 989e94.

7 Purkayastha S, Tilney H, Georgiou P, et al. Laparoscopic

cholecystectomy versus mini-laparotomy cholecystectomy: a meta-

analysis of randomized control trials. Surg Endosc 2007; 21:

1294e300.

8 Swank DJ, Swank-Bordewijk SC, Hop WC, et al. Laparoscopic

adhesiolysis in patients with chronic abdominal pain: a blinded

randomized controlled multi-centre trial. Lancet 2003 Apr 12;

361(9365): 1247e51.

9 Howick J, Glasziou P, Aronson J. The evolution of evidence hierar-

chies: what can Bradford Hill’s ‘guidelines for causation’ contribute?

J R Soc Med; 2009: 186e94.

10 Phillips EH, Franklin M, Carroll BJ, et al. Laparoscopic colectomy.

Ann Surg 1992 Dec; 216(6): 703e7.

� 2009 Elsevier Ltd. All rights reserved.

DETERMINING SURGICAL EFFICACY

11 Weeks JC, Nelson H, Gelber S, et al. Short-term quality-of-life

outcomes following laparoscopic-assisted colectomy vs open

colectomy for colon cancer: a randomized trial. JAMA 2002 Jan 16;

287(3): 321e8.

12 Guillou P, Quirke P, Thorpe J, et al. Short-term endpoints of

conventional versus laparoscopic-assisted surgery in patients with

SURGERY 27:9 374

colorectal cancer (MRC CLASSIC trial): multicentre, randomized

controlled trial. Lancet 2005; 365: 1718e26.

FURTHER READING

Piantadosi S. Clinical trials: a methodological perspective. 2nd edn.

Hoboken, NJ: John Wiley & Sons, 2005.

� 2009 Elsevier Ltd. All rights reserved.