Pilot Study

A pilot study is a small scale, preliminary investigation conducted before a full scale study to test the feasibility, procedures, instruments, and methods that are planned for a larger project. Its primary purpose is to identify and address potential problems in the study design, logistics, and execution before substantial resources are committed to the main study.

Launching a full scale study without preliminary testing is one of the most costly mistakes a researcher can make. Problems with recruitment rates, instrument performance, data collection procedures, intervention delivery, participant retention, and analytical approaches are far better discovered and corrected in a small pilot study than after thousands of participants have been enrolled and millions of dollars have been spent. Pilot studies save time, money, and effort by revealing practical and methodological issues that cannot be anticipated from the design alone. They also give estimates of parameters needed to plan the main study, such as recruitment and attrition rates and the standard deviation of the outcome, although estimates from a small sample are imprecise. Effect sizes are not among them: a pilot's effect estimate is too unreliable to set the main study's sample size, which should rest on the smallest effect that would be clinically important. Funding agencies increasingly expect applicants to have pilot data demonstrating that their proposed methods are feasible before committing to major grants.

Pilot studies test one or more aspects of the planned full scale study on a smaller scale. The scope and design of the pilot depend on what needs to be tested. In the conceptual framework of Eldridge and colleagues (2016), pilot studies are a subset of feasibility studies: feasibility study is the overarching term for work asking whether a future study can be done, and a pilot study is one in which the future study, or part of it, is conducted on a smaller scale. The feasibility and methodological pilots described below are pilot studies in this sense.

Feasibility pilots assess whether the study can be conducted as planned. They examine recruitment and enrollment rates (can enough eligible participants be found and enrolled within the planned time frame?), retention rates (do participants complete the study, or do they drop out at unacceptable rates?), intervention fidelity (can the intervention be delivered as designed, and do clinicians or other providers adhere to the protocol?), and data collection procedures (are questionnaires completed fully, are measurements taken accurately, and are data management systems functioning?).

Methodological pilots evaluate the performance of measurement instruments, the appropriateness of outcome measures, the suitability of statistical analysis plans, and the validity of assumptions underlying the sample size calculation. They may reveal that an outcome measure has a ceiling effect (most participants score at the maximum), that an instrument has poor reliability in the target population, or that the variability of the outcome is much larger than assumed, requiring a larger sample than planned.

Pilot studies are typically not powered to detect statistically significant treatment effects, and their results should not be interpreted as definitive evidence for or against an intervention's efficacy. Their sample sizes are justified by the precision needed for feasibility estimates, pragmatic considerations and rules of thumb (often 12 to 30 participants per group for clinical trials, though the appropriate number depends on the specific objectives of the pilot), not by a power calculation for the treatment effect.

Randomized pilot trials are reported using the CONSORT extension for randomised pilot and feasibility trials (2016). It distinguishes pilot trials from definitive trials, asks authors to report any prespecified criteria used to judge whether, or how, to proceed to the definitive trial, and states that formal hypothesis testing for effectiveness is not recommended. It emphasizes reporting feasibility outcomes (such as recruitment rates, protocol adherence, and instrument performance) rather than efficacy outcomes.

Internal pilot studies are embedded within the main study, using the initial participants as both a pilot sample and part of the full study. This approach avoids the delay and additional cost of a separate pilot study, but it requires that no changes to the protocol are made as a result of the pilot data, or that any changes are accounted for in the analysis.

Consider a pilot study before a large scale research project, particularly when the intervention, population, setting, or measurement approach has not been previously tested. Pilots are especially important for complex interventions, multicenter trials, studies in hard to reach populations, and studies using novel measurement instruments. Use the pilot study results to refine the protocol, improve procedures, and justify the feasibility of the main study to funding agencies and ethics committees.

A common mistake is treating a pilot study as a miniature version of the main trial and reporting p values and efficacy conclusions from the underpowered sample. Pilot studies are not designed to test hypotheses; they are designed to test procedures. A related error is using the pilot's effect estimate to set the main study's sample size: estimates from small samples are unreliable, so the calculation should rest on the smallest effect that would be clinically important. Another error is failing to act on the lessons learned from the pilot, conducting the full scale study with the same problems that the pilot identified. Some researchers skip the pilot altogether when time or funding pressures are acute, a decision that often proves more costly in the long run when preventable problems arise in the full study. Finally, not reporting pilot study results deprives the research community of valuable information about the feasibility of specific methods and designs.

A research team planning a randomized controlled trial of a text message based smoking cessation program in pregnant women conducts a six month pilot study at three prenatal clinics. The pilot enrolls 40 participants (20 per arm) and assesses recruitment rates (finding that 35% of eligible women agree to participate, lower than the assumed 50%), intervention engagement (finding that women prefer twice daily messages to five times daily), and outcome measurement (finding that biochemical verification of smoking status using salivary cotinine is feasible at clinic visits but not at home assessments). Based on these findings, the team revises the recruitment strategy, adjusts the message frequency, and modifies the verification protocol before launching the full trial with 500 participants.