Research CurriculumResearch CurriculumSession II –Study Subjects, Session II –Study Subjects,

Variables and Outcome MeasuresVariables and Outcome Measures

Jim Quinn MD MSJim Quinn MD MS

Research Director , Division of Emergency MedicineResearch Director , Division of Emergency Medicine

Stanford UniversityStanford University

OverviewOverview

Study SubjectsStudy Subjects- SamplingSampling

- RecruitmentRecruitment

VariablesVariables- Types of outcome measuresTypes of outcome measures

- Precision, accuracy, validity, reliabilityPrecision, accuracy, validity, reliability

Study SubjectsStudy SubjectsGeneralizing the ResultsGeneralizing the Results

““Research is only interesting to others if Research is only interesting to others if they can apply it to their practice”they can apply it to their practice”

Study SubjectsStudy Subjects

Subjects in the study sample should be Subjects in the study sample should be representative of the population of interestrepresentative of the population of interest

Depending on study different populations Depending on study different populations may yield different results.may yield different results.

- Examples: General population, ED Examples: General population, ED patients, Clinic Patients, Attitudes of patients, Clinic Patients, Attitudes of patientspatients

- Laceration studies, syncope studyLaceration studies, syncope study

Study SubjectsStudy Subjects

Specify the best clinical and demographic Specify the best clinical and demographic characteristics of the study population to characteristics of the study population to best answer questionbest answer question

Appropriate sampling from that target Appropriate sampling from that target populationpopulation

Results = truth in the study Results = truth in the study

Best possible chance to have the results Best possible chance to have the results generalizable.generalizable.

Selection CriteriaSelection CriteriaDefining the Target PopulationDefining the Target Population

Inclusion CriteriaInclusion Criteria

- defines the main characteristics of the defines the main characteristics of the target population – be specifictarget population – be specific

Selection CriteriaSelection Criteria Defining the Target PopulationDefining the Target Population

Exclusion CriteriaExclusion Criteria

- Individuals whose characteristics may Individuals whose characteristics may interfere with the quality of the resultsinterfere with the quality of the results

E.g. – rare events, poor follow-upE.g. – rare events, poor follow-up

- May compromise generalizability of the - May compromise generalizability of the studystudy

SamplingSampling

Convenience SampleConvenience Sample

Consecutive SampleConsecutive Sample

Probability SamplesProbability Samples

- Simple Random Sample- Simple Random Sample

- Stratified Random SampleStratified Random Sample

- Cluster SamplesCluster Samples

RecruitmentRecruitmentGoalsGoals

A sample that represents the target A sample that represents the target populationpopulation

- Non responders, lost follow-up- Non responders, lost follow-up

Enough subjects to meet sample size Enough subjects to meet sample size requirementsrequirements

- Play it safe, overestimate- Play it safe, overestimate

- There is always fewer patients than you - There is always fewer patients than you think!think!

““The best way to eliminate disease The best way to eliminate disease is to study it!”is to study it!”

Outcome MeasuresOutcome Measures

Selection of Variables and ScalesSelection of Variables and Scales

Selection of VariablesSelection of VariablesPractical Points/Precision/AccuracyPractical Points/Precision/Accuracy

Continuous VariablesContinuous Variables- ““discrete” variablesdiscrete” variables- rich in informationrich in information- Potential sample size “relief”Potential sample size “relief”

CategoricalCategorical- DichotomousDichotomous- NominalNominal- Ordinal Ordinal

Measurement ScalesMeasurement Scales

Categorical VariablesCategorical Variables

- Phenomena often not suited for Phenomena often not suited for measurement (e.g. Death)measurement (e.g. Death)

- DichotomousDichotomous

- NominalNominal

- Ordinal – categories have order but no Ordinal – categories have order but no specific numerical or uniform difference specific numerical or uniform difference

Measurement ScalesMeasurement Scales

Continuous (infinite values)Continuous (infinite values)

Ordered discrete (ordinal with numerical Ordered discrete (ordinal with numerical meaning)meaning)

- Statistically handled very similarly- Statistically handled very similarly

Measurement ScalesMeasurement ScalesSummarySummary

CategoricalCategorical

- Scales may have more meaning and make Scales may have more meaning and make more sense. more sense.

- Less information, need large numbersLess information, need large numbers

ContinuousContinuous

- some times hard to determine meaningful some times hard to determine meaningful differencesdifferences

- sample size friendlysample size friendly

Attributes of Outcome MeasuresAttributes of Outcome MeasuresPrecisionPrecision

Is the measure “reproducible, reliable and Is the measure “reproducible, reliable and consistent”consistent”

Subject to random error and variabilitySubject to random error and variability

- Observer variabilityObserver variability

- Instrument variabilityInstrument variability

- Subject variabilitySubject variability

Assessing PrecisionAssessing Precision

Inter and Intraobserver reproducibilityInter and Intraobserver reproducibility

Within and between instrument Within and between instrument reproducibilityreproducibility

- Continuous variables – Coefficient of Continuous variables – Coefficient of variationvariation

- Categorical – kappa statisticCategorical – kappa statistic

Enhancing PrecisionEnhancing Precision

Standardize measurement methodsStandardize measurement methods

Train and certify observersTrain and certify observers

Refining the instrumentsRefining the instruments

Automating the instrumentAutomating the instrument

Repetition (reduces random error)Repetition (reduces random error)

AccuracyAccuracy

““Does the variable actually measure or Does the variable actually measure or represent what it intends to”represent what it intends to”

Assessed by comparison to a “Gold Standard”Assessed by comparison to a “Gold Standard”

Different than precision, but many things that Different than precision, but many things that improve precision improve accuracyimprove precision improve accuracy

A function of systematic errorA function of systematic error- Observer biasObserver bias- Subject BiasSubject Bias- Instrument BiasInstrument Bias

Enhancing AccuracyEnhancing Accuracy

Standardized measurement methodsStandardized measurement methods

Training observersTraining observers

Refining instrumentsRefining instruments

Automating instrumentsAutomating instruments

Making Unobtrusive measuresMaking Unobtrusive measures

BlindingBlinding

Calibration of InstrumentsCalibration of Instruments

ValidityValidityAccuracy when there is no “Gold Standard”Accuracy when there is no “Gold Standard”

- Measuring an abstract or subjective Measuring an abstract or subjective phenomena (e.g. – pain, quality of life)phenomena (e.g. – pain, quality of life)

- Content Validity (Face, Inherent or Content Validity (Face, Inherent or sampling validity)sampling validity)

- Construct ValidityConstruct Validity

- Criterion Related Validity (Predictive Criterion Related Validity (Predictive Validity)Validity)

Final ThoughtsFinal Thoughts

An outcome measure should be sensitive An outcome measure should be sensitive enough to determine important clinical enough to determine important clinical differencesdifferencesIt should be associated with only the It should be associated with only the characteristic of interestcharacteristic of interestMeasurements should involve data Measurements should involve data collection that is efficient in time and costcollection that is efficient in time and costEfficiency is improved by increasing the Efficiency is improved by increasing the quality of each item measuredquality of each item measured