STUDY DESIGN: PILOT STUDIES

STUDY DESIGN:

PILOT STUDIESPILOT STUDIES

Charles Flexner, MDCharles Flexner, MD

Johns Hopkins UniversityJohns Hopkins University

Pilot Studies:

A Case Presentation

What is a Pilot Study???

Pilot StudyPilot Study Developmental StudyDevelopmental Study Feasibility StudyFeasibility Study Phase I StudyPhase I Study Small Exploratory Clinical TrialSmall Exploratory Clinical Trial Hypothesis Generating StudyHypothesis Generating Study

As opposed to hypothesis-testingAs opposed to hypothesis-testing Etc.Etc.

Semantics

The initial study examining a new method or The initial study examining a new method or treatment. treatment.

-- NCI Dictionary of Cancer Terms-- NCI Dictionary of Cancer Terms

A small scale preliminary study conducted before A small scale preliminary study conducted before the main research in order to check the feasibility the main research in order to check the feasibility or to improve the design of the research. or to improve the design of the research.

-- Wikipedia!-- Wikipedia!

What is a Pilot Study?

““In a comprehensive literature search using In a comprehensive literature search using Medline and the Web of Science we could find no Medline and the Web of Science we could find no formal methodological guidance as to what formal methodological guidance as to what constitutes a pilot study.”constitutes a pilot study.” --- GA Lancaster, S Dodd, PR Williamson. --- GA Lancaster, S Dodd, PR Williamson.

Design and Analysis of Pilot Studies: Design and Analysis of Pilot Studies: Recommendations for Good Practice. Recommendations for Good Practice. Journal of Journal of Evaluation in Clinical PracticeEvaluation in Clinical Practice, 2004; 10(2):307-12., 2004; 10(2):307-12.



--- GA Lancaster, et al. Journal of Evaluation in Clinical Practice, 2004; 10(2):307-12.

What question is usually being asked in a What question is usually being asked in a pilot study?pilot study?

The Bottom Line

What question is usually being asked in a What question is usually being asked in a pilot study?pilot study?

““How do I get started?”How do I get started?” ““Is this technique / intervention / data mining Is this technique / intervention / data mining

exercise feasible?”exercise feasible?” “ “Do I need preliminary data to prove that my Do I need preliminary data to prove that my

sample size is correct?”sample size is correct?” ““How much data do I need to justify doing How much data do I need to justify doing

further studies related to this hypothesis?”further studies related to this hypothesis?”

The Bottom Line

Pilot Studies:

Some published examples

Educated uncertaintyEducated uncertainty Feasibility concernsFeasibility concerns Sample size concernsSample size concerns Critical developmental needsCritical developmental needs

Assay development and/or standardizationAssay development and/or standardization New study instrumentNew study instrument New device or technique for endpoint measurementNew device or technique for endpoint measurement

Good reasons to do a “Pilot Study”

Inadequate literature reviewInadequate literature review Need to generate hypothesesNeed to generate hypotheses Running out of: Running out of:

TimeTime Money Money PatientsPatients PatiencePatience

LazinessLaziness

Bad reasons to do a “Pilot Study”

Test integrity & feasibilityTest integrity & feasibility Recruitment & consentRecruitment & consent Intervention (e.g. tolerance, compliance, retention)Intervention (e.g. tolerance, compliance, retention) Data collection (e.g. forms, interface, time)Data collection (e.g. forms, interface, time) EquipmentEquipment Other procedures (e.g. randomization)Other procedures (e.g. randomization)

Refine methods and proceduresRefine methods and procedures Confirm or revise sample size estimatesConfirm or revise sample size estimates

PILOT STUDIES: “Pretesting”

Recruitment & consent:Recruitment & consent: Get the types of participants that we think we will get?Get the types of participants that we think we will get? Any important segments of the target population being left Any important segments of the target population being left

out?out? People turn down the opportunity to participate in our study? People turn down the opportunity to participate in our study?

(what proportion? able to meet the sample size requirement in (what proportion? able to meet the sample size requirement in time? recruitment pool large enough? expand the inclusion time? recruitment pool large enough? expand the inclusion criteria or go multi-center?)criteria or go multi-center?)

Is it obvious who meets and who does not meet the eligibility Is it obvious who meets and who does not meet the eligibility requirements?requirements?

Can this be learned without a formal feasibility study? Can this be learned without a formal feasibility study?


Compliance and Retention:Compliance and Retention: Acceptability / tolerability of interventionAcceptability / tolerability of intervention Participants fail to comply with protocol Participants fail to comply with protocol

requirements? (what proportion? need to modify requirements? (what proportion? need to modify protocol? revise analytic plan?)protocol? revise analytic plan?)

Participants fail to finish our study? (what Participants fail to finish our study? (what proportion? reduce participant burden? add run-in? proportion? reduce participant burden? add run-in? increase sample size?)increase sample size?)


Data Collection:Data Collection: Are all the important/required data items collected? (run Are all the important/required data items collected? (run

through the analytic plan)through the analytic plan) Is there enough room on the data collection form for all of Is there enough room on the data collection form for all of

the data you receive?the data you receive? Who will be recording the data? (standardized training? Who will be recording the data? (standardized training?

standardized equipment & calibration? standardized standardized equipment & calibration? standardized procedures? blinding/masking procedures?)procedures? blinding/masking procedures?)

Any problems entering collected data?Any problems entering collected data? Are the data collection instruments validated and reliable in Are the data collection instruments validated and reliable in

the target population?the target population?


“Murphy’s Law: anything that can go wrong will go wrong.“The reason to run a pilot study is to ensure that the things that do go wrong, go wrong during the pilot study so we can fix them before we start the full study!” – Nae-Yuh Wang

Hard EndpointsHard Endpoints Soft EndpointsSoft Endpoints Clinical EndpointsClinical Endpoints Surrogate EndpointsSurrogate Endpoints BiomarkersBiomarkers Genes (SNP’s, Haplotypes, GWAS)Genes (SNP’s, Haplotypes, GWAS)

PILOT STUDIES: Endpoints

Don’t forget! The value of n-of-1 studies

and self-experimentation

Importance of adhering to the classical Importance of adhering to the classical experimental methodexperimental method Hypothesis testingHypothesis testing Sample size calculationSample size calculation Proper endpoint selectionProper endpoint selection Proper study populationProper study population

Give preference to publishable endeavors!Give preference to publishable endeavors!

The Bottom Line

Do it right the first time!!!”- Scott Zeger, PhD

STUDY DESIGN: PILOT STUDIES

Documents

Transcript of STUDY DESIGN: PILOT STUDIES