1

A Pilot Study: What, Why, and How

Soleh Ayubi09-20-2012

2

Until 2012, no one ever published trials paper related to social media…

3

Until 2012, no trials paper related to social media are available…

2006

2007

2008

2009

2010

2011

2012

Writing papers

Gathering ideas

and writing

proposals

Submitting

proposals and

waiting for the results

Get a funding,Administ

rative process, consolida

te teams,

etc.

Preparing the study:

IRB, logistics, recruitme

nt, etc.

Trials

4

What is a Pilot Study? A pilot projector study as an experimental,

exploratory, test, preliminary, trial or try out investigation (Concise Oxford Thesaurus, 2002).

“ ...test of the methods and procedures to be used on a larger scale if the pilot study demonstrates that the methods and procedures can work” (A Oxford Dictionary of Epidemiology, 2001)

“...investigation designed to test the feasibility of methods and procedures for later use on a large scale or to search for possible effects and associations that may be worth following up in a subsequent larger study” (Everitt B, 2006).

5

What is a Pilot Study? Synonymous with a feasibility study (Thabane.

L, et al., 2010). The main goal of pilot studies is to assess

feasibility so as to avoid potentially disastrous consequences of embarking on a large study - which could potentially “drown” the whole research effort (Thabane. L., 2010).

Pilot studies are required to do prior to full-scale phase III trials (G.A. Lancester, et al., 2004).

6

Reasons for Pilot Study Major funding bodies (UK Medical Research Council and

NIH) require evidences from a pilot study before allocating fund for RCTs (G.A. Lancester, et al., 2004).

It permits preliminary testing of the hypotheses that leads to testing more precise hypotheses in the main study. It may lead to changing some hypotheses, dropping some, or developing new hypotheses (G.A. Lancester, et al., 2004; Thabane. L., 2010).

It often provides the researchers with ideas, approaches, and clues they may not have foreseen before conducting the pilot study. Such ideas and clues increase the chances of getting clearer findings in the main study (G.A. Lancester, et al., 2004; Thabane. L., 2010).

7

Questions for Pilot Study1. Process: This assesses the feasibility of the steps

that need to take place as part of the main study;2. Resources: This deals with assessing time and

budget problems that can occur during the main study;

3. Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centers;

4. Scientific: This deals with the assessment of treatment safety, determination of dose levels and response or other interventions, and estimation of treatment effect and its variance (Van Teijlingen. et al., 2001; Thabane. L, et al., 2010)

8

Questions for Pilot Study (2) G.A. Lancester, et al., 2004; Thabane. L., 2010

Process: This assesses the feasibility of the processes that are key to the success of the main study Sample size and sample characteristics for the main study

Is it obvious who meets and who does not meet the eligibility requirements? Are the eligibility criteria sufficient or too restrictive?

Recruitment rates Clinical or surrogate end-point for the main study

Resources: This deals with assessing time and resource problems that can occur during the main study Retention rates (Non) compliance or adherence rates Understanding of study questionnaires or data collection tools:

Do subjects provide no answer, multiple answers, qualified answers, or unanticipated answers to study questions?

Length of time to fill out all the study forms

9

Questions for Pilot Study (3)G.A. Lancester, et al., 2004; Thabane. L., 2010

Management: This covers potential human and data management problems Determining capacity:

Will the study participants overload your phone lines, mail box, or overflow your waiting room?

Determining process time How much time does it take to mail out a thousand surveys?

Are the equipment, computer, and system readily available when and where it is needed?

What happens when it breaks down or gets stolen? Can the software used for capturing data read and understand the

data? What’s its capacity? Determining center willingness and capacity

Do the centers do what they committed to doing? Do investigators have the time to perform the tasks they committed to doing? Are there any capacity issues at each participating center?

10

Questions for Pilot Study (4)G.A. Lancester, et al., 2004; Thabane. L., 2010

Scientific: This deals with the assessment of intervention, treatment safety, dose, response, effect and variance of the effect Are the interventions accepted? What challenges do study personnel have? Are there any problems entering data into the computer? Can data coming from different sources or activities be

matched? Is it safe to use the study drug/intervention? What is the safe dose level? Do patients respond to the drug? What is the estimate of the treatment effect? What is the estimate of the variance of the treatment effect?

11

Sample SizeG.A. Lancester, et al., 2004; Thabane. L., 2010

In general, sample size calculation may not be required for some pilot studies.

It should be representative of the target study population.

It should be based on the same inclusion/exclusion criteria as the main study.

It should be large enough to provide useful information about the aspect that are being assessed for feasibility.

12

Data AnalysisG.A. Lancester, et al., 2004; Thabane. L., 2010

Conclusion as a target: Stop main study not feasible Continue but modify protocol feasible with

modifications Continue without modification but monitor closely

feasible with close monitoring Continue without modification feasible as is.

The analysis of a pilot study should be mainly descriptive or should focus on confidence interval estimation; avoid statistical significance, hypothesis testing, or powered test.

13

ReportingG.A. Lancester, et al., 2004; Thabane. L., 2010 The title or abstract should indicate that the study is a “pilot” or

“feasibility” Clearly define why do we need a pilot and relate with the specific

scientific primary and secondary objectives and hypotheses for the main study; Clearly define what are the outcome measures of the pilot; It is important to clearly indicate the feasibility objectives as the primary focus for the pilot.

Interpretation of the results should focus on feasibility, taking into account the stated criteria for success of feasibility, study hypotheses, sources of potential bias, confounding factors, and the risk associated the study.

Results from hypothesis testing should be treated as preliminary and interpreted with caution, as no formal power calculations have been carried out.

Discuss the external validity as default, the result may not generalize to the target population.