Non-randomised studies

Methodologies for a new era summer school

School of Applied Social Studies, University College Cork

22 June 2011

Dr Paul MontgomeryJennifer Burton

Why bother?

The data from a good study can be analysed in The data from a good study can be analysed in many ways, but no amount of clever analysis many ways, but no amount of clever analysis

can compensate for the problems with the can compensate for the problems with the design of a study.design of a study.

(Altman, 1991)(Altman, 1991)

Introduction

It is sometimes impossible or undesirable to influence events in a human sample You may not be able to control group allocation It may be unethical to expose or withhold an

intervention

An appreciation of the varieties of study designs available can reduce the need to reinvent the wheel and to rediscover the mistakes of others. It can also help to end the “scandal of research”’

(Altman, 1993)

Aims

Identify types of questions that can be answered using non-randomised methods

Describe several non-randomised designsHighlight the strengths and weaknesses of

common non-randomised designs

Question to designPrevalence/ incidenceRisk and protective factorsPrognosisHarmEffectiveness

Internally Valid

Evidence of Effectiveness

Meta-Analysis of Randomised Trials Randomised Trials

Non-randomised studies Cohort studies

Case-Control studies

Non-comparative studies Case Series (open trial)

Case reports

Expert opinion

The Hierarchy of EvidenceFor Intervention Studies

Illustrative examples, hypothesis generating studies

Question to DesignIn practice, methods tend to be

complementary in answering questions

Each method may be used to answer several types of questions

Several studies may help tease apart a question

PrevalenceHow many people have mental health

disorders?

SurveysCensus (one-time full population)

Ideal but expensive, difficult, likely to miss people from marginal groups

Cross section (one-time sample) Must consider many potential biases due to

geography, time, etc.

Cross-Sectional Survey

Identify a sample of adults representative of the population

Measure symptoms of mental health problems

Calculate the number of people above a given threshold

Surveys

Usefully estimate prevalence or incidence and associations

Comparisons may be made between different subgroups to identify associations

Risk and ProtectionDoes smoking cause cancer?

Risk and ProtectionWhat

factors can predict falls in the elderly?

Retrospective CohortTake all elderly people in OxfordLook at

Characteristics of their homes Individual factors (e.g. medication use) Other predictors?

Look for association between these factors and falls

Retrospective CohortBackward looking surveyRelatively inexpensive and practicalGood for detecting latent outcomesProne to several sources of bias (selection,

participant recall, etc.)

Risk and ProtectionWhat are the factors that contribute to chronic

fatigue syndrome?

Prospective CohortTake all babies born in a given

period in 1970Survey them regularlyLook for correlations between

variables (e.g. maternal depression) and outcomes (e.g. chronic fatigue)

Prospective Cohort

Identifies temporal relationshipsCan examine multiple effects of exposure Loss-to-follow-up can be a problem Inefficient for the evaluation of rare problems

unless the attributable risk is high e.g. 1970 British Cohort Study (ongoing)

http://www.cls.ioe.ac.uk/

Risk and ProtectionIs fish oil good for my mental

health?

Ecological Studies

Compare countries’ consumption of fish oil to their rates of depression

Increased consumption of fish oil lowers a nation’s rate of depression (Hibbeln 2001), but eating fish is not the only difference among countries

Ecological StudiesLarge unit of analysis (e.g. countries)May identify population-level risk and protective

factorsBecause the unit of observation is not the

individual subject, they are subject to the ecological fallacy when they overlook important sources of variance

Risk and ProtectionIs running bad for my knees?

Case ControlIdentify a group of runnersThen find a group of people who

don’t run matched for age, sex, weight and other variables

Test for associations between knee problems and being a runner

Case Control Inexpensive and practicalGood for generating hypothesesLacks a temporal dimensionUnless data come from a population-based

survey, cannot give incidence and prevalence data

Prognosis

My husband has just taken 4 times the recommended dose of purple pills.

What’s going to happen?

Case ReportAn otherwise healthy 60 year old man was prescribed 10 mg of

vardenafil (Levitra, Bayer) for sexual dysfunction. Because this was ineffective, he increased the dose to 40 mg. Three hours later, he had a tonic-clonic seizure, seen by his relatives.

On admission to hospital, neurological examination, brain magnetic resonance imaging, and electroencephalography after sleep deprivation were normal. Stress electrocardiography, echocardiography, and cardiac scan with dipyridamole test as well as carotid doppler ultrasonography did not show concomitant cardiac diseases. The man was told to stop using vardenafil.

Two months later he had a new tonic-clonic seizure, four hours after taking 30 mg of vardenafil. At eight months' follow-up he is seizure-free without treatment.

Pasquale Striano, Federico Zara, Carlo Minetti (professor of paediatrics), Salvatore Striano (2006). Epileptic seizures can follow high doses of oral vardenafil. BMJ;333:785.

Case ReportInexpensive and quickMay draw attention to important

clinical and research issuesIn rich detail, describes conditions

and outcomes May not be representative, does not

usually provide evidence of causation

Case Series

Several case studiesDraws attention to patterns in client populationsCommon in aetiological research

Break

EffectivenessDoes abstinence education reduce the

likelihood of premarital sex?

Pre-post (single group)Take a class of kidsAsk them if they will have sex before marriageThey attend an abstinence-based education

programmeAsk them if they will have sex before marriage

Pre-post (single group) Inexpensive, generally easier than controlled

studiesProvides some evidence of temporal

relationshipsUsually lacks a plausible counterfactual (i.e.

what would have happened in the absence of intervention)

EffectivenessDoes Head Start

improve IQ?

Between GroupLook at all the kids in New York

born in 1980 who were eligible for Head Start

Compare those who attended to those who did not attend

If possible, collect measures before and after attendance

Between GroupProvides a counterfactual

scenario, can give evidence of temporal relationships.

Groups may differ on both measured and unmeasured variables, observed differences may be attributable to factors other than the intervention.

EffectivenessDoes having regular contact with a social

worker improve outcomes for fostered children?

Historical ControlCompare children in foster care

since the 1944 education act to children in foster care before then.

If possible, include measures before and after enrolment for children in each group.

Historical ControlProvides a counterfactual scenarioGroups may differ on both

measured and unmeasured variables, observed differences may be attributable to factors other than the intervention

EffectivenessDo intensive

police crackdowns reduce gun

violence?

Time-Series Identify areas with high levels of gun crime and

identify peak timesRepeatedly use crackdowns during periods of

high crimeCompare times with the intervention to periods

without the intervention

Time-SeriesProvides a counterfactual scenarioTimes may be different Often requires complicated

statistical analyses to control for differences in baseline variables, time trends, etc.

Conclusion

What is your question?What types of study design might

contribute to an answer?Think ‘Horses for Courses’