A pilot study is a smaller version of a planned study. The purpose of the pilot study is to test the design of the larger study to improve its likelihood of success. Unfortunately, there are several misconceptions about what a pilot study is for.

Should I Run A Pilot Study to Determine Safety?

No. In most cases, we want to say that a treatment or device is safe because we saw few or no adverse events in the pilot. For most things that are studied in clinical research, adverse events are rare occurrences. A pilot study is simply not going to be powered to estimate the rate of adverse events with suitable precision.

Suppose, for example, that our pilot study for a particular drug has \(N=20\). During the course of study we saw no serious adverse events. Can we conclude that the drug is safe? No. Even though 0% of the subjects had a serious adverse event, the upper confidence limit is as high as 20.05%, depending on the method used to calculate the confidence interval. In this case, it is not reasonable to make conclusions about the drug's safety.

On the other hand, if we see large numbers of serious adverse effects in our pilot study, it may indicate that the treatment or device is not suitable and therefore we can not proceed with the larger study. Such a decision would necessarily need to consider the wide confidence intervals for the estimate of the rate of serious adverse events and to be taken in conjunction with the IRB.

Should I Use Pilot Data in the Larger Study?

No. A key idea in clinical research is that the data in a study are all sampled in the same way. If it is not, an additional source of variation may be added to the study. One of the main purposes of a pilot study is to tests the methods and protocols used for data collection. Often, these methods and protocols are tweaked in the larger study. Even if there are no changes made between the pilot study and larger study, it is possible that some other factor may change. Therefore, it cannot be guaranteed that the data from the pilot study and the larger study will be sampled similarly.

In the case of a properly designed adaptive trial, you may be able to combine the data with the requisite adjustment to account for the inflated Type I error, but this is a rare case.

Should I Use Pilot Data to Determine Sample Size?

No. Though it is often done, it can be argued that it is not appropriate to use pilot data to determine sample size for the larger study (see Friedman et al., Kraemer et al. and Leon et al.).

The basic arguments against using a pilot study to determine the sample size of the larger study is that due to the small sample size in a pilot study, the estimate of the population effect size is too variable to be reliable enough for use in the power calculation for the larger study.

Using the example from Leon et al., suppose we are estimating \(\Delta\), the population effect size. For the sake of simplicity, assume that the study comprises of two groups, say placebo and control, and that \(\Delta\) represents the between group difference of a normally distributed outcome. We estimate \(\Delta\) by \(d=\frac{\bar{X_1} - \bar{X_2}}{s}\), the sample effect size, where \(\bar{X_i}\) is the sample mean from group \(i\) and \(s\) is the sample standard deviation.

We can approximate the 95% confidence interval for \(d\) by:

\begin{equation*} d \pm \frac{4}{\sqrt{N}} \end{equation*}

where \(N\) is the sample size.

In our pilot study, let's say \(N=18\) and \(d=0.5\). The confidence interval of \(d\) is \((-0.17, 1.17)\). This is a very wide interval. If we estimate the number of sample size requirements for the larger study based on this confidence interval for \(d\), we will get an estimate between 12 and 576. Using the data we gleaned from our pilot study, we risk both under-powering and over-powering our larger study depending on the true value of \(\Delta\), which as a population estimate, we cannot know.

This argument begs the question: how should we determine the effect size to use in our sample size calculations?

I would argue, as Leon et al. do, that the sample size calculation should be based on clinically meaningful effect size. By using a generally agreed upon clinically meaningful effect size we power the larger study to detect a useful effect size instead of the imprecise estimate of the population effect size which may lead to an under-powered or over-powered study.

Should I Use the Pilot Study to Determine the Study's Feasibility?

Yes. This is the one of the main purposes of a pilot study. By running through the processes used in subject/sample acquisition, randomization (if the larger study is a randomized control trial), intervention administration and data acquisition, we can test the feasibility of each step. If, at any step, something was found not to work at all or to be infeasible, that part of the trial should be removed all together or the study canceled.

The importance of a pilot study is that we can make these decisions before we invest the money, time and resources on the design and running of the larger trial.

Should I Use the Pilot Study to Refine Techniques?

Yes. This is the another of the main purposes of a pilot study.

Suppose our pilot study is running and we find that a particular step or procedure in the study does not work quite right. By finding this out in a pilot study, the study's protocol can be adjusted and refined before the larger study is started, a much easier and cheaper proposition.

How do I Size a Pilot Study?

A pilot study is not intended to test a hypothesis. Therefore we cannot talk about p-values and effect sizes; they should not be put forth in the protocol at all. Without p-values and effect sizes, we cannot perform a power analysis.

As stated earlier, the goal of a pilot study is to test the study's feasibility and work out any kinks in the design, including how subjects are enrolled, randomized, treated and so forth. This goal is one factor in the size of the study. Another factor is budget. How many subjects can you afford to have in the study? How long can you afford for the study to run?

Balancing these two factors: the number of subjects you need to test the study's design and the number of subjects you can afford, provides the answer to how many subjects are needed.

Do I Need a Pilot Study?

It depends.

That is a fairly unsatisfactory answer, but the truth of the matter is that, though a pilot study is a fundamental part of clinical research, they are not always feasible. Therefore we have to use pragmatics to decide if we should use a pilot study.

Is the pilot study required by the funding mechanism I am using? If so, then there is really no question. It is required.

If the pilot study is not required, it can still be a good idea to use one. The pilot study provides important information about how the study will work. If you are launching a large study with a large budget, a pilot study is a good investment to make sure that your study is properly designed, that the protocols can and are being followed and that any changes that need to be made can be made before the larger study begins. While it cannot determine with reasonable certainty the effect size that will be seen or the safety of what is being studied, it is a reasonable investment to ensure that your data is being properly and efficiently collected.

References

  • Friedman, LM.; Furberg, CD.; DeMets, DL. Fundamentals of Clinical Trials. 3. New York: Springer; 1998.
  • Kraemer HC, Mintz J, Noda A, Tinklenberg J, Yesavge JA. Caution regarding the use of pilot studies to guide power calculations for study proposals. Archives of general psychiatry. 2006; 63:484–489.
  • Leon, AC; Davis, LL; Kraemer HC. The Role and Interpretation of Pilot Studies in Clinical Research. Journal of Psychiatric Research. 2011; 45(5): 626-629.