The following lecture has been approved for

Adults

This lecture may contain information, ideas, concepts and discursive anecdotes that may be thought provoking and

challenging

It is not intended for the content or style of delivery to cause offence

Any issues raised in the lecture may require the viewer to engage in further thought, insight, reflection or critical evaluation

Critical EvaluationCritical Evaluationofof

Clinical Research Clinical Research

Dr. Craig A. JacksonDr. Craig A. JacksonSenior Lecturer in Health PsychologySenior Lecturer in Health Psychology

Division of Trauma & Critical CareFaculty of HealthUCE Birmingham

hcc.uce.ac.uk/craigjacksonhcc.uce.ac.uk/craigjackson

Session OutlineSession Outline

• Main Research DesignsMain Research Designs•ExperimentsExperiments RCTsRCTs•ObservationObservation Case-ControlCase-Control Cohort studiesCohort studies

• Critical Evaluation CriteriaCritical Evaluation Criteria• Ethical clearance / considerationsEthical clearance / considerations

• Sample and Population issuesSample and Population issues

• Methods & Data collectionMethods & Data collection

• AnalysesAnalyses

• Write up & Publication issuesWrite up & Publication issues

Brief Research HistoryBrief Research History

Role of Bran Fibre dietary increases in IBS patients -- 1997Role of Bran Fibre dietary increases in IBS patients -- 1997((Randomised Controlled Trial)

Mental Health of UK Farmers using OP Pesticides (X2) -- 1997-2000Mental Health of UK Farmers using OP Pesticides (X2) -- 1997-2000(Epidemiological Surveys)(Epidemiological Surveys)

Neurobehavioural Performance of desert-based Oil Drillers -- 1998-2000Neurobehavioural Performance of desert-based Oil Drillers -- 1998-2000(Clinical assessment)

Temporary Hearing Loss in Student Bar Staff – 2000-2002 Temporary Hearing Loss in Student Bar Staff – 2000-2002 (Epidemiological Survey)

Benefits of Occupational Health Advice in Primary Care Settings -- 2001-2004Benefits of Occupational Health Advice in Primary Care Settings -- 2001-2004(Randomised Controlled Trial)

Smaller-Scale projects – (Tri-Services, NHS Personnel, NHS Patients)Smaller-Scale projects – (Tri-Services, NHS Personnel, NHS Patients)(Cross-sectional Surveys, Clinical Trials)

Budget Airline Pilot Fatigue – 2002Budget Airline Pilot Fatigue – 2002(Cross-sectional Survey)

Multiple roles of psychologist, statistician, and methodology designerMultiple roles of psychologist, statistician, and methodology designer

Formats of Clinical ResearchFormats of Clinical Research

Experimental vs. ObservationalExperimental vs. Observational

Longitudinal vs. Cross-sectionalLongitudinal vs. Cross-sectional

Prospective vs. RetrospectiveProspective vs. Retrospective

LongitudinalLongitudinal

ProspectiveProspective

ExperimentalExperimental

Randomised Controlled TrialRandomised Controlled Trial

ObservationalObservational

LongitudinalLongitudinal Cross-sectionalCross-sectional

SurveySurveyRetrospectiveRetrospectiveProspectiveProspective

Case control studiesCase control studiesCohort studiesCohort studies

Qualititative VS Quantitative ResearchQualititative VS Quantitative Research

False oppositionFalse opposition

Observational methods equally validObservational methods equally valid

Complementary rolesComplementary roles

Qualitative equally as hard to doQualitative equally as hard to do

(if not harder)(if not harder)

Quantitative

Qualitative

Quantitative Research DesignsQuantitative Research Designs

approach

Observational

Experimental

Survey

Epidemiology

Case - control

Cohort study

Postal questionnaire

Laboratory

RCT

PatientsStaff

Healthy

Experimental StudiesExperimental Studies

Investigator makes interventionInvestigator makes intervention

A “manipulation”A “manipulation”

Then studies the effects of that intervention Then studies the effects of that intervention

Features:Features:

Comparison e.g. before vs. afterComparison e.g. before vs. after control vs. treatmentcontrol vs. treatment

Always longitudinal Always longitudinal

Always prospective Always prospective

Experimental Experimental

Clinical Clinical

TrialsTrials

RCTsRCTs

Experimental StudiesExperimental Studies

Evaluate effectiveness of intervention / therapyEvaluate effectiveness of intervention / therapy

Use similar samples who reflect populationUse similar samples who reflect population

Comparable groupsComparable groups

Differences in outcomes due to interventions (not differences between groups)Differences in outcomes due to interventions (not differences between groups)

Independent Variable (IV) alters Dependent Variable (DV)Independent Variable (IV) alters Dependent Variable (DV)

Best “evidence” of cause and effectBest “evidence” of cause and effect

Sometimes inconclusiveSometimes inconclusive

Types of Experimental StudiesTypes of Experimental Studies

Between Subjects StudiesBetween Subjects Studies

Each group receives different treatmentEach group receives different treatment

Groups comparedGroups compared

Within Subjects StudiesWithin Subjects Studies

Each individual is measured before & after interventionEach individual is measured before & after intervention

Advantage that each participant is own controlAdvantage that each participant is own control Between subject variability removedBetween subject variability removed

patientspatients

Treatment groupTreatment group

Control groupControl group

Outcome measuredOutcome measured

Outcome measuredOutcome measured

patientspatients Outcome measured #1Outcome measured #1 TreatmentTreatment Outcome measured #2Outcome measured #2

Traditional Experimental DesignsTraditional Experimental Designs

Between subjects studiesBetween subjects studies

Within Subjects studies Within Subjects studies

Within Subjects StudiesWithin Subjects Studies

Cross-over-studies Cross-over-studies Each patient receives treatment in sequenceEach patient receives treatment in sequence““Washout” period between treatmentsWashout” period between treatmentsOrder of treatments randomisedOrder of treatments randomised

Matched-pairs studyMatched-pairs study

Parallel studyParallel study

Patient in arm 1 matched with patient in arm 2Patient in arm 1 matched with patient in arm 2

Match based on prognostic / socio-economic factorsMatch based on prognostic / socio-economic factors

Data is linkedData is linked

Paired individualsPaired individuals

Treatment 1Treatment 1

Treatment 2Treatment 2

Group AGroup A

Group BGroup B

Treatment 2Treatment 2

Treatment 1Treatment 1

Gp AGp A

Gp BGp B

Control GroupsControl Groups

Allow comparison in Between Group studies Allow comparison in Between Group studies

Evaluations without comparison?Evaluations without comparison?

Types of Control GroupsTypes of Control Groups•““no treatment” group no treatment” group likely to be confounded by having conditionlikely to be confounded by having condition

•““placebo” groupplacebo” group ethically dodgy?ethically dodgy?

•““low dose” grouplow dose” group avoids ethical issuesavoids ethical issues

•““standard treatment” groupstandard treatment” group avoids ethical issuesavoids ethical issues

•““gold standard” group gold standard” group avoids ethical issuesavoids ethical issues

•““historical controls”historical controls” unreliable due to many confoundersunreliable due to many confounders

Comparison Groups: Random SamplingComparison Groups: Random Sampling

Ensures generalizability of findings to larger pop.Ensures generalizability of findings to larger pop.e.g. in-patient sample limitationse.g. in-patient sample limitations

Treatment effects better detected if there is little between-group variabilityTreatment effects better detected if there is little between-group variability

Exclusion Criteria & Inclusion Criteria keep groups comparableExclusion Criteria & Inclusion Criteria keep groups comparable

Paradox:Paradox: greater uniformity of sample = less generalisable to general population greater uniformity of sample = less generalisable to general population

Control Groups: Random AllocationControl Groups: Random Allocation

Doesn’t guarantee groups will be homogonousDoesn’t guarantee groups will be homogonous

Ensures allocation independent of patient featuresEnsures allocation independent of patient features

Avoids (sub)conscious allocation biasAvoids (sub)conscious allocation biase.g. severely sick people into treatment groupse.g. severely sick people into treatment groups

Guarantees Guarantees allocationallocation to be bias-freeto be bias-free

Non-homogenous groups may still occurNon-homogenous groups may still occurdue to chance – random errorsdue to chance – random errors

Stratified randomisation Stratified randomisation for each prognostic factor e.g. weight, age, sexfor each prognostic factor e.g. weight, age, sex

Drug XDrug X

53 yrs53 yrs

81% male81% male19% fem19% fem

Gp AGp A(500)(500)

Gp BGp B(500)(500)

Drug YDrug Y

27 years27 years

53% male53% male47% fem47% fem

PopulationPopulation(60 million)(60 million)

SampleSample(1000)(1000)

Randomised Controlled Trials in GP & Primary CareRandomised Controlled Trials in GP & Primary Care

90% consultations take place in GP surgery90% consultations take place in GP surgery

RCT is actually 50 years oldRCT is actually 50 years old

Potential problemsPotential problems

2 Key areas:2 Key areas: Recruitment BiasRecruitment Bias

Randomisation BiasRandomisation Bias

Over-focus on failings of RCTsOver-focus on failings of RCTs

RCTs in General Practice & Primary CareRCTs in General Practice & Primary Care

• RCTs justified inRCTs justified in situations of genuine clinical uncertaintysituations of genuine clinical uncertainty• Provides rigorous, sound basis for evaluating treatmentsProvides rigorous, sound basis for evaluating treatments• Samples large enough to establish any worthwhile benefit Samples large enough to establish any worthwhile benefit

(effectiveness or cost, or both) (effectiveness or cost, or both)

Need for larger numbers of patients Need for larger numbers of patients

More than are availableMore than are available to single practicesto single practices

Requires “club together” approachRequires “club together” approach

GPs: no contractual obligationGPs: no contractual obligation

(i)(i) unwilling to take part if no immediate benefit for patientsunwilling to take part if no immediate benefit for patients

(ii)(ii) while possibly disruptingwhile possibly disrupting the delivery of health carethe delivery of health care

RCTs in General Practice & Primary CareRCTs in General Practice & Primary Care

GPs conflict of interest between:GPs conflict of interest between:Role and Wish to benefit future patientsRole and Wish to benefit future patientsAcademic meritAcademic merit

Long term nature of practitionerLong term nature of practitioner and patient relationship and patient relationship May engender loyaltiesMay engender loyaltiesUnfairly coerce patientsUnfairly coerce patients to give consentto give consent

Patients' fearsPatients' fears about:about:ConfidentialityConfidentialityRisks of the interventionRisks of the interventionApparent disadvantage of being allocated to a control groupApparent disadvantage of being allocated to a control group

may further inhibit recruitmentmay further inhibit recruitment

Fail to recruit Fail to recruit consecutiveconsecutive patientspatients may introduce potential for selection bias may introduce potential for selection bias

RCTs in General Practice & Primary CareRCTs in General Practice & Primary Care

May disrupt primary care May disrupt primary care Too much disruption = no reflection of real practiceToo much disruption = no reflection of real practice

MethodologicalMethodological problems reduce scientific reliability of the results problems reduce scientific reliability of the results (Recruitment & Randomisation) (Recruitment & Randomisation)

General practice General practice notnot a laboratory a laboratoryPatientsPatients are not experimental animalsare not experimental animals

Case-control studies, retrospectiveCase-control studies, retrospective and prospective cohort studies, and and prospective cohort studies, and descriptive studies aredescriptive studies are all acceptable methods. all acceptable methods. Observation is OKObservation is OK

Should accept alternative methodsShould accept alternative methods when RCT too difficult or flawedwhen RCT too difficult or flawed

RCT DeficienciesRCT Deficiencies

Trials too smallTrials too small

Trials too shortTrials too short

Poor qualityPoor quality

Poorly presentedPoorly presented

Address wrong questionAddress wrong question

Methodological inadequaciesMethodological inadequacies

Inadequate measures of quality of life (changing)Inadequate measures of quality of life (changing)

Cost-data poorly presentedCost-data poorly presented

Ethical neglectEthical neglect

Patients given limited understandingPatients given limited understanding

Poor trial managementPoor trial management

PoliticsPolitics

MarketeeringMarketeering

Why still the dominant model?Why still the dominant model?

Observational StudiesObservational Studies

Investigator observes existing situationInvestigator observes existing situation

DescribesDescribes

Analyses Analyses

Interprets Interprets

No influence on events No influence on events

Longitudinal observation studiesLongitudinal observation studies

case-control studies: retrospectivecase-control studies: retrospective

cohort-studies: prospective cohort-studies: prospective

Cross-sectional observation studiesCross-sectional observation studies

surveys examining subjects at one point in timesurveys examining subjects at one point in time

based on random sample of interest populationbased on random sample of interest population

Observational StudiesObservational Studies

Look for associationsLook for associations• Cause -> EffectCause -> Effect• Exposure – IllnessExposure – Illness• EpidemiologicalEpidemiological• Incidence, cause, preventionIncidence, cause, prevention

No control group necessaryNo control group necessary

Cannot use classical experimentationCannot use classical experimentation

No randomisationNo randomisation

Bias is a realistic problemBias is a realistic problem

Case-Control StudyCase-Control Study

Identify group with condition / illness Identify group with condition / illness (cases)(cases)

Identify group without condition / illness Identify group without condition / illness (controls)(controls)

Both groups compared for exposure to Both groups compared for exposure to (hypothesized)(hypothesized) risk factors risk factors

Greater exposure to risk factor in cases than controls = Greater exposure to risk factor in cases than controls = “causal relation”“causal relation”

Beware:Beware:Lead time biasLead time biasRecruitment of cases at similar points in timeRecruitment of cases at similar points in timeNewly diagnosed cases (biases?)Newly diagnosed cases (biases?)

Selection of ControlsSelection of Controls

Cases have Lung Cancer + Smoking ExposureCases have Lung Cancer + Smoking Exposure

Controls could be other hospital patients (other disease) or “normals”Controls could be other hospital patients (other disease) or “normals”

Matched Cases & Controls for age & genderMatched Cases & Controls for age & gender

Option of 2 Controls per CaseOption of 2 Controls per Case

Smoking years of Lung Cancer cases and controls Smoking years of Lung Cancer cases and controls (matched for age and sex)(matched for age and sex)

CasesCases ControlsControlsn=456n=456 n=456n=456

FF PPSmoking yearsSmoking years 13.7513.75 6.126.12 7.57.5 0.040.04

(± 1.5)(± 1.5) (± 2.1)(± 2.1)

Case-Control Study: Other BiasesCase-Control Study: Other Biases

Recall BiasRecall BiasCases > associations with exposuresCases > associations with exposures

Unreliable MemoriesUnreliable MemoriesRetrospective natureRetrospective natureOver-reliance on recallOver-reliance on recall

Unreliable RecordsUnreliable RecordsPoor hospital recordsPoor hospital recordsRepetitive, incomplete, inaccurate, irretrievable, interpretationRepetitive, incomplete, inaccurate, irretrievable, interpretation

Interview BiasInterview BiasDifferent interviewersDifferent interviewers

Cohort StudyCohort Study

ID and examination of a group (cohort)ID and examination of a group (cohort)

Followed over time (20 years common!)Followed over time (20 years common!)

Looking for disease development / other end-pointLooking for disease development / other end-point

Aetiology of disease (based on data collected)Aetiology of disease (based on data collected)Data more reliable than case-control studiesData more reliable than case-control studies

• Requires large NRequires large N• Requires long follow upRequires long follow up• InefficientInefficient• Expensive (espec. rare outcomes)Expensive (espec. rare outcomes)

Cohort Study: MethodsCohort Study: Methods

Subjects classified into 2 (or more groups)Subjects classified into 2 (or more groups)e.g. exposed vs non exposede.g. exposed vs non exposed

End point: groups compared for cancer symptom statusEnd point: groups compared for cancer symptom status

Comparison of Brain cancers between users and non-users of mobile Comparison of Brain cancers between users and non-users of mobile phonesphones

Brain CaBrain Ca No CaNo Ca

mobile phone usermobile phone user 292292 108108 400400

non-phone usernon-phone user 8989 313313 402402

381381 421421 802802

Cohort Study: Other BiasesCohort Study: Other Biases

Lost to follow upLost to follow upBias if reason related to exposureBias if reason related to exposureValidity affectedValidity affected

Group sizes changeGroup sizes changeMembership changes e.g ex-smokersMembership changes e.g ex-smokersDifferential mortalityDifferential mortality

Change in circumstanceChange in circumstancee.g. job changee.g. job changeExposures need calculation or re-calculationExposures need calculation or re-calculation

Surveillance biasSurveillance biasInvestigator aware of group membershipInvestigator aware of group membershipInvestigating exposed members moreInvestigating exposed members more

prospectively measure risk factors prospectively measure risk factors cohortcohort end point measuredend point measured

aetiologyaetiologyprevalenceprevalence

developmentdevelopmentodds ratiosodds ratios

retrospectively measure risk factors retrospectively measure risk factors start point measuredstart point measured casescases

aetiologyaetiologyodds ratiosodds ratiosprevalenceprevalencedevelopmentdevelopment

Observational studiesObservational studies

CohortCohort (prospective) (prospective)

Case-ControlCase-Control (retrospective) (retrospective)

Cross Sectional StudyCross Sectional Study

Subjects contacted & surveyed just onceSubjects contacted & surveyed just once

Questionnaire (post, email, phone)Questionnaire (post, email, phone)

Random sample of defined pop.Random sample of defined pop.

Limited causalityLimited causalityNot temporal relationshipsNot temporal relationshipsLittle insight into aetiologyLittle insight into aetiologySource of descriptive dataSource of descriptive dataPrevalence ratesPrevalence ratesVolunteer biasVolunteer biasNon responsesNon responsesSelf-selectionSelf-selectionUnrepresentative sampleUnrepresentative sample

Critical Appraisal CriteriaCritical Appraisal Criteria

• Researchers’ planResearchers’ plan• Ethical clearance / considerationsEthical clearance / considerations• SampleSample

•SizeSize•BiasBias•AllocationAllocation

• Methods & Data collectionMethods & Data collection•ValidValid•ReliableReliable•MeasurableMeasurable•AccurateAccurate

• AnalysisAnalysis• Write upWrite up

•AccurateAccurate•ClearClear•ReplicableReplicable

Planning and DesignPlanning and Design

Ethical clearance & considerationsEthical clearance & considerations

Good research should be...Good research should be...

JustifiedJustified

Well plannedWell planned

Appropriately designedAppropriately designed

Ethically approvedEthically approved

• Research should be driven by protocolResearch should be driven by protocol• Pilot studies should have a written rationalePilot studies should have a written rationale• Protocols should answer specific questionsProtocols should answer specific questions• Not just “collecting data”Not just “collecting data”• Protocols must be agreed by all contributors & participantsProtocols must be agreed by all contributors & participants• Keep the protocol as part of the Research record / logKeep the protocol as part of the Research record / log

Ethical misconduct not to meet this standard? – Not yetEthical misconduct not to meet this standard? – Not yet

Design & Ethical ApprovalDesign & Ethical Approval

Statistical issues should be considered before data collectionStatistical issues should be considered before data collection

Power calculations are (becoming) essentialPower calculations are (becoming) essential

Formal documented ethical approval is required for all research involving Formal documented ethical approval is required for all research involving

(i) (i) peoplepeople(ii)(ii) medical recordsmedical records(iii)(iii) anonymous human tissue (Nuffield Council on Bioethics)anonymous human tissue (Nuffield Council on Bioethics)

Fully informed consent should always be soughtFully informed consent should always be sought

If not possible (deceptive studies) a research ethics committee should decideIf not possible (deceptive studies) a research ethics committee should decide

WMA Research Ethics ChecklistWMA Research Ethics Checklist

• people’s rights and claimspeople’s rights and claims• different sorts of interestsdifferent sorts of interests and their relative strengthand their relative strength• human well-beinghuman well-being• loss of lifeloss of life

• what would be good or bad for peoplewhat would be good or bad for people

• democratic acceptancedemocratic acceptance

• consultationconsultation

• sensitive momentssensitive moments• benefits and harmsbenefits and harms• griefgrief and distressand distress• an obligation to make sacrifices for the community;an obligation to make sacrifices for the community;

• entitlement of the community to deny autonomy and violate bodily integrity in public interestentitlement of the community to deny autonomy and violate bodily integrity in public interest• the systemthe system of justiceof justice• public safetypublic safety• public policy considerationspublic policy considerations• dangerdanger• civil libertiescivil liberties• individual autonomyindividual autonomy• lives and liberties of citizenslives and liberties of citizens

Sample sizeSample size

Population characteristicsPopulation characteristics

The Importance of Sample SizeThe Importance of Sample Size

• Forgotten in many studiesForgotten in many studies

• Little consideration givenLittle consideration given

• Appropriate size needed to confirm / refute hypothesesAppropriate size needed to confirm / refute hypotheses

• Small samples far too small to detect anything but the grossest differenceSmall samples far too small to detect anything but the grossest difference

• Non-significant results become reported as significant – Type 2 errors occurNon-significant results become reported as significant – Type 2 errors occur

• Too large a sample – Too large a sample – unnecessary waste of (clinical) resourcesunnecessary waste of (clinical) resourceswaste of patient time, inconvenience, discomfortwaste of patient time, inconvenience, discomfort

Essential to assess optimal sample size before investigationEssential to assess optimal sample size before investigation

How Many Make a Sample?How Many Make a Sample?

““8 out of 10 owners who expressed a preference, said their cats 8 out of 10 owners who expressed a preference, said their cats preferred it.”preferred it.”

How confident can we be about such statistics?How confident can we be about such statistics?

8 out of 10?8 out of 10?80 out of 100?80 out of 100?

800 out of 1000?800 out of 1000?80,000 out of 100,000?80,000 out of 100,000?

25 cell clusters25 cell clusters

22 cell clusters22 cell clusters

24 cell clusters24 cell clusters

21 cell clusters21 cell clusters

Total Total = 92 cell clusters= 92 cell clustersMean Mean = 23 cell clusters= 23 cell clustersSDSD = 1.8 cell clusters= 1.8 cell clusters

26

25

24

23

22

21

20

Multiple Measurement of small sampleMultiple Measurement of small sample

It all depends on the size of your needleIt all depends on the size of your needle

NN AgeAge IQIQ

11 2020 10010022 2020 10010033 2020 10010044 2020 10010055 2020 10010066 2020 10010077 2020 10010088 2020 10010099 2020 1001001010 2020 100100

TotalTotal 200200 10001000MeanMean 2020 100100SDSD 00 00

NN AgeAge IQIQ

11 1818 10010022 2020 11011033 2222 11911944 2424 10110155 2626 10510566 2121 11311377 1919 12012088 2525 11911999 2020 1141141010 2121 101101

TotalTotal 216216 11021102MeanMean 21.621.6 110.2110.2SDSD ± 4.2± 4.2 ± 19.2 ± 19.2

NN AgeAge IQIQ

11 1818 10010022 2020 11011033 2222 11911944 2424 10110155 2626 10510566 2121 11311377 1919 12012088 2525 11911999 2020 1141141010 4545 156156

TotalTotal 240240 11571157MeanMean 2424 115.7115.7SDSD ± 8.5± 8.5 ± 30.2 ± 30.2

Small samples spoil researchSmall samples spoil research

Qualitative studies need to sample wisely too…Qualitative studies need to sample wisely too…

Asian GPs’ attitudes to ANP Asian GPs’ attitudes to ANP

Objective: Objective: To determine attitudes to ANP among Asian doctors in East Birmingham PCTTo determine attitudes to ANP among Asian doctors in East Birmingham PCT

Method:Method:Send invitation to 55 Asian GPs (Approx 47% of East Birmingham PCT)Send invitation to 55 Asian GPs (Approx 47% of East Birmingham PCT)

Intends to interview (30mins) with first 20 GPs who respondIntends to interview (30mins) with first 20 GPs who respond

Sample would be 36% of Asian GPs – and only 17% of GPs in PCT Sample would be 36% of Asian GPs – and only 17% of GPs in PCT

Severely Biased Research (and ethically dodgy too)Severely Biased Research (and ethically dodgy too)

Achieving a high response rate to a questionnaire is vitalAchieving a high response rate to a questionnaire is vitalas helps ensures a normal distribution of responses?as helps ensures a normal distribution of responses?

Postal questionnaires rarely get a response rate > 40%Postal questionnaires rarely get a response rate > 40%

Unless respondents have a vested interest in the outcomeUnless respondents have a vested interest in the outcome

Bias?Bias?

Most efficient (best) response rates usually happen when respondents have to Most efficient (best) response rates usually happen when respondents have to do very little to take part in the studydo very little to take part in the study

Multiple phase projects see a depletion in numbers at every stageMultiple phase projects see a depletion in numbers at every stage

Quick “in and out” one-stop approach is bestQuick “in and out” one-stop approach is best

Population SamplesPopulation Samples

RANDOM samplingRANDOM sampling

OPPORTUNISTIC samplingOPPORTUNISTIC sampling

CONSCRIPTIVE samplingCONSCRIPTIVE sampling

QUOTA samplingQUOTA sampling

5’6” 5’7” 5’8” 5’9” 5’10” 5’11” 6’ 6’1” 6’2” 6’3” 6’4”5’6” 5’7” 5’8” 5’9” 5’10” 5’11” 6’ 6’1” 6’2” 6’3” 6’4” Height Height

% o

f pop

ulat

ion

% o

f pop

ulat

ion

A Normally Distributed Sample of a PopulationA Normally Distributed Sample of a Population

A POPULATIONA POPULATION

REPRESENTATIVE SAMPLE REPRESENTATIVE SAMPLE (theoretical)(theoretical)

ACCESSIBLE ACCESSIBLE SAMPLESAMPLE(actual)(actual)

Are this lot are REPRESENTATIVE of the POPULATION ?Are this lot are REPRESENTATIVE of the POPULATION ?

Sampling a PopulationSampling a Population

POPULATIONSPOPULATIONSCan be mundane or extraordinaryCan be mundane or extraordinary

SAMPLESAMPLEMust be representativeMust be representative

INTERNALY VALIDITY OF SAMPLEINTERNALY VALIDITY OF SAMPLESometimes validity is more important than generalisabilitySometimes validity is more important than generalisability

SELECTION PROCEDURESSELECTION PROCEDURESRandomRandom

OpportunisticOpportunisticConscriptiveConscriptive

QuotaQuota

ECOLOGICAL VADLIDITYECOLOGICAL VADLIDITYParticipants in their natural environmentParticipants in their natural environment

Sampling KeywordsSampling Keywords

DeploymentDeployment

RANDOM SAMPLING RANDOM SAMPLING

RANDOM ASSIGNMENT RANDOM ASSIGNMENT How to assign the sample into different treatments or groupsHow to assign the sample into different treatments or groupsRelated to the INTERNAL VALIDITY of the researchRelated to the INTERNAL VALIDITY of the researchEnsures groups are similar (EQUIVALENT) to each other prior to TREATMENTEnsures groups are similar (EQUIVALENT) to each other prior to TREATMENT

Waste of time randomly sampling but not randomly allocatingWaste of time randomly sampling but not randomly allocating

Having a choice in this matter is a luxuryHaving a choice in this matter is a luxury

How many makes a sample?How many makes a sample?

POWER OF STUDY CALCULATIONPOWER OF STUDY CALCULATION

Statistical method of calculating the number of subjects needed in a project Statistical method of calculating the number of subjects needed in a project Based upon…..Based upon…..

Expected variance of subjects’ scoresExpected variance of subjects’ scores

Useful size of any differences between groupsUseful size of any differences between groups

Significance level (e.g. 5 % or 1 %)Significance level (e.g. 5 % or 1 %)

Power levelPower level

The larger the differences you are looking for between groups, then the fewer The larger the differences you are looking for between groups, then the fewer subjects are needed. Looking for small differences between groups requires subjects are needed. Looking for small differences between groups requires larger numbers of subjectslarger numbers of subjects

BiasBias

BiasBias

Validity of study depends on Validity of study depends on avoiding biasavoiding bias

Bias = “Systematic distortion of results due to unforeseen factors”Bias = “Systematic distortion of results due to unforeseen factors”

Group 1 = pillGroup 1 = pill Group 2 = no pillGroup 2 = no pill

How will the “no pill”group progress?How will the “no pill”group progress?

Any effects of them “Any effects of them “knowingknowing” they have no treatment?” they have no treatment?

Handling differences may influence + complicate trial resultsHandling differences may influence + complicate trial results

Known as Known as confounding factorsconfounding factors

To minimize bias… To minimize bias… control groupcontrol group randomisationrandomisation blindingblinding

Gulf War Gulf War SyndromeSyndrome

Depleted Uranium Depleted Uranium WeaponryWeaponry

PesticidesPesticides

THIS IS AN INHERENT PROBLEM WITH THIS IS AN INHERENT PROBLEM WITH HEALTH RESEARCH HEALTH RESEARCH

COMBAT IT WITH LARGE SAMPLES COMBAT IT WITH LARGE SAMPLES AND CLEVER METHODOLOGYAND CLEVER METHODOLOGY

StressStress

Selection Bias Selection Bias

Sampling properly is CrucialSampling properly is Crucial

Samples may be askewSamples may be askew

Specialist publications attract a specialist response groupSpecialist publications attract a specialist response group

Exists a self-selection bias of those with special interestsExists a self-selection bias of those with special interests

Controversial topics, or litigious areasControversial topics, or litigious areas

TelecommsTelecomms

Call Call CentresCentres

Bird FluBird Flu

Hospital Hospital infectioninfection

Bias – The placebo effect really does work!Bias – The placebo effect really does work!

Most effectiveMost effective medication known medication known In approx. 30% of pop.In approx. 30% of pop.Subjected to more clinical trialsSubjected to more clinical trials than any other medicament than any other medicament Nearly always does better thanNearly always does better than anticipatedanticipatedThe range of susceptible conditions seems limitlessThe range of susceptible conditions seems limitlessDoes not always occurDoes not always occurPresent in subjective and objective outcomesPresent in subjective and objective outcomesNegative outcomes can occur (Nocebo effect)Negative outcomes can occur (Nocebo effect)

•Big pills better than smaller pillsBig pills better than smaller pills•Red pills better than blueRed pills better than blue•4 pills better than 24 pills better than 2•30% of pop.30% of pop.•Sham surgery vs arthroscopy for osteoarthritisSham surgery vs arthroscopy for osteoarthritis

Patient’s “knowledge” of their treatment causes biasPatient’s “knowledge” of their treatment causes bias

e.g. Benedetti & the Turin studye.g. Benedetti & the Turin study

Subject Variables that potentially bias / confound researchSubject Variables that potentially bias / confound research

STABLE FACTORSSTABLE FACTORS SITUATIONAL FACTORSSITUATIONAL FACTORS

AgeAge Alcohol (recent use)Alcohol (recent use)EducationEducation Caffeine (recent use)Caffeine (recent use)SexSex Nicotine (recent use)Nicotine (recent use)SocioeconomicsSocioeconomics Medication (recent use)Medication (recent use)LanguageLanguage Paints, glues, pesticides (recent) Paints, glues, pesticides (recent) HandednessHandedness Near visual acuityNear visual acuityComputer experienceComputer experience Restricted movement (injury)Restricted movement (injury)Caffeine (habitual use)Caffeine (habitual use) Cold / fluCold / fluAlcohol (habitual use)Alcohol (habitual use) StressStressNicotine (habitual use)Nicotine (habitual use) Arousal / FatigueArousal / FatigueMedication (habitual use)Medication (habitual use) SleepSleepPaints, glues, pesticides (habitual use)Paints, glues, pesticides (habitual use) Screen luminanceScreen luminanceDiabetesDiabetes Time of dayTime of dayEpilepsyEpilepsy Time of yearTime of yearOther CNS / PNS diseaseOther CNS / PNS diseaseHead injury (out >1 hr)Head injury (out >1 hr)Alcohol / drug addictionAlcohol / drug addictionPhysical activityPhysical activity

Blinding: Importance of doing itBlinding: Importance of doing it

Investigator or Patient know treatment = BiasInvestigator or Patient know treatment = Bias

Observations and Judgements become less reliableObservations and Judgements become less reliable

Patient responses change:Patient responses change:Positive outcomes in active armPositive outcomes in active armNegative outcomes in passive armNegative outcomes in passive arme.g. known cancer diagnoses and deterioratione.g. known cancer diagnoses and deterioration

Use max. degree of blindness possibleUse max. degree of blindness possiblee.g. make patient and investigator both blind if possiblee.g. make patient and investigator both blind if possible

e.g. A.A.Mason & Congenital Ichthyosis and Hypnosis 1951e.g. A.A.Mason & Congenital Ichthyosis and Hypnosis 1951

Blinding: Methods of doing itBlinding: Methods of doing it

Double-blindDouble-blind patient & investigator blindpatient & investigator blind

Data managerData manager

Patient interactionPatient interactionTreatment typeTreatment type

Blinding: MethodsBlinding: Methods

Double-blindDouble-blind patient & investigator blindpatient & investigator blind

Single-blindSingle-blind patient blindpatient blind

Triple-blindTriple-blind patient & investigator & data monitor blindpatient & investigator & data monitor blind

Double-dummyDouble-dummy 2 treatments2 treatmentspatients get 2 pills (1 active, 1 dummy)patients get 2 pills (1 active, 1 dummy)

Open trialsOpen trials patient & investigator aware of treatment patient & investigator aware of treatment

Randomisation in a double-blind trialRandomisation in a double-blind trialEnvelope technique commonEnvelope technique common

Un-blinding – ethical necessityUn-blinding – ethical necessity

Un-blinding a problematic studyUn-blinding a problematic study

Breaking code – anticipated in the planning stagesBreaking code – anticipated in the planning stages

Criteria for breaking code – established and agreedCriteria for breaking code – established and agreed

Emergency access to randomisation codeEmergency access to randomisation code

Treatment stopped and patient withdrawnTreatment stopped and patient withdrawn

Formal monitoring process – review and make recommendationsFormal monitoring process – review and make recommendations

MethodsMethods

Data CollectionData Collection

Background on ResearchBackground on Research

• Large-scale Large-scale

• QuantitativeQuantitative

• Can be descriptive Can be descriptive “ “2% of women think they are beautiful”2% of women think they are beautiful”

• Can be inferential Can be inferential “ “Significantly more singletons think they’re beautiful (46%) than married (23%)Significantly more singletons think they’re beautiful (46%) than married (23%)

• Done with a sample of patients, respondents, consumers, or professionalsDone with a sample of patients, respondents, consumers, or professionals

• Differences between any groups assessed with hypothesis testingDifferences between any groups assessed with hypothesis testing

Important that sample size must be large enough to detect any Important that sample size must be large enough to detect any such difference if it truly existssuch difference if it truly exists

Survey ResearchSurvey Research

Questionnaire is a fundamental component of most researchQuestionnaire is a fundamental component of most research

Most MSc / MPhil /PhD projects use survey methodsMost MSc / MPhil /PhD projects use survey methods

Can be very efficientCan be very efficient

Weaknesses Weaknesses

weak / dubious questionnairesweak / dubious questionnaires

non-valid questionnairesnon-valid questionnaires

biased samplesbiased samples

biased responsesbiased responses

poor response ratepoor response rate

ValidityValidity

Does the survey measure what it says it is measuringDoes the survey measure what it says it is measuring

ReliabilityReliability

Does the survey yield stable Does the survey yield stable

data over timedata over time

How accurate are large scale surveys / questionnaires?How accurate are large scale surveys / questionnaires?

XXXX XXXX XXXX XXXXXX XXX

XXXXX XX XX

XXXXXX XXXXhappy happy sad sad

XX

Likert scalesLikert scales

““How do you feel right now?”How do you feel right now?”

XXXX XXXXXXXX XXXX

XXXXXX

XX

happy happy sad sad

Bending the DataBending the Data

““How do you feel right now?”How do you feel right now?”

Non-Responders just as ImportantNon-Responders just as Important

Postal surveys may accrue poor response rates (e.g. 20%) from pop.Postal surveys may accrue poor response rates (e.g. 20%) from pop.

May need to re-write to pop. to re-recruit bigger sampleMay need to re-write to pop. to re-recruit bigger sample

Inefficient to write to all pop. againInefficient to write to all pop. again

Need to re-write to non-responders and NOT respondersNeed to re-write to non-responders and NOT responders

Impossible in anonymous studies with no linkageImpossible in anonymous studies with no linkage

Can be done with confidential studiesCan be done with confidential studies

ResearcherResearcher Potential SamplePotential Sample

1000 people

540 consents

540 blank questionnaires

Under-powered studyn = 210

Response rate of 21%

210 completed questionnaires

Diminishing returns of multi-stage recruitmentDiminishing returns of multi-stage recruitment

Invitation letter & consent form

Acceptance letter & consent form

““Unethical” practices proved to increase response ratesUnethical” practices proved to increase response rates

TechniqueTechnique Likelihood of participationLikelihood of participation

Cash incentive Cash incentive X 2X 2 (Brown, (Brown, et al.et al. 1997, Roberts 1997, Roberts et al.et al. 2000) 2000)

Warn respondents of follow up (need linkage)Warn respondents of follow up (need linkage) X 1.4X 1.4

Drop out must be explained by the respondentDrop out must be explained by the respondent X 1.3X 1.3

Choice to opt out given to respondentsChoice to opt out given to respondents X 0.7X 0.7

(Edwards (Edwards et al.et al. 2002) 2002)

Finally. . . . The best research is simple in designFinally. . . . The best research is simple in design

““Some people hate the very name of statistics but.....their power of Some people hate the very name of statistics but.....their power of

dealing with complicated phenomena is extraordinary. They are the dealing with complicated phenomena is extraordinary. They are the

only tools by which an opening can be cut through the formidable only tools by which an opening can be cut through the formidable

thicket of difficulties that bars the path of those who pursue the science thicket of difficulties that bars the path of those who pursue the science

of man.”of man.”

Sir Francis Galton, 1889Sir Francis Galton, 1889

Further ReadingFurther Reading

Altman, D.G. “Designing Research”. In: Altman, D.G., (ed.) Altman, D.G. “Designing Research”. In: Altman, D.G., (ed.) Practical Statistics Practical Statistics For Medical ResearchFor Medical Research. London, Chapman and Hall, 1991; 74-106.. London, Chapman and Hall, 1991; 74-106.

Bland, M. “The design of experiments”. In: Bland, M., (ed.) Bland, M. “The design of experiments”. In: Bland, M., (ed.) An introduction to An introduction to medical statisticsmedical statistics. Oxford, Oxford Medical Publications, 1995; 5-25.. Oxford, Oxford Medical Publications, 1995; 5-25.

Daly, L.E., Bourke, G.J. “Epidemiological and clinical research methods”. Daly, L.E., Bourke, G.J. “Epidemiological and clinical research methods”. In: Daly L.E., Bourke, G.J., (eds.) In: Daly L.E., Bourke, G.J., (eds.) Interpretation and uses of medical statistics.Interpretation and uses of medical statistics. Oxford, Blackwell Science Ltd, 2000; 143-201.Oxford, Blackwell Science Ltd, 2000; 143-201.

Jackson, C.A. “Study Design” & “Sample Size and Power”. In: Gao Smith, F. Jackson, C.A. “Study Design” & “Sample Size and Power”. In: Gao Smith, F. and Smith, J. (eds.)and Smith, J. (eds.) Key Topics in Clinical Research Key Topics in Clinical Research. Oxford, BIOS scientific . Oxford, BIOS scientific Publications,Publications, 2002 2002..

Jackson, C.A. “Jackson, C.A. “Planning Health & Safety Research Projects in the Planning Health & Safety Research Projects in the Workplace”. Workplace”. Croner Health and Safety at Work Special ReportCroner Health and Safety at Work Special Report 2002; 62: 1-16. 2002; 62: 1-16.

Kumar, R. Kumar, R. Research Methodology: a step by step guide for beginnersResearch Methodology: a step by step guide for beginners. . Sage, London 1999.Sage, London 1999.

Further ReadingFurther Reading

Abbott, P. and Sapsford. Abbott, P. and Sapsford. Research methods for nurses and the caring Research methods for nurses and the caring professions.professions. Open University Press, Buckingham 1988. Open University Press, Buckingham 1988.

Bowling, A. Measuring Health. Open University Press, Milton Keynes 1994Bowling, A. Measuring Health. Open University Press, Milton Keynes 1994

Polit, D. & Hungler, B. Nursing research: Principles and methods (7th ed.). Polit, D. & Hungler, B. Nursing research: Principles and methods (7th ed.). Philadelphia: Lippincott, Williams & Wilkins 2003.Philadelphia: Lippincott, Williams & Wilkins 2003.

Council for International Organizations of Medical Sciences (CIOMS). Council for International Organizations of Medical Sciences (CIOMS). IInternational Guidelines for Ethical Review of Epidemiological Studies nternational Guidelines for Ethical Review of Epidemiological Studies World Health Organisation, Geneva 1991.World Health Organisation, Geneva 1991.

Nuffield Council on Bioethics. Nuffield Council on Bioethics. Human tissue: Ethical and legal issues. Human tissue: Ethical and legal issues. Nuffield Nuffield Council on Bioethics, London 1995.Council on Bioethics, London 1995.

World Medical Association. World Medical Association. Ethical Principles for Medical Research Involving Ethical Principles for Medical Research Involving Human Subjects. Human Subjects. Declaration of Helsinki, 2002. (Washington Amendment).Declaration of Helsinki, 2002. (Washington Amendment).