Methodological Challenges in Online Trials
Journal of Medical Internet ResearchPublished 3 April 2009Open access
Elizabeth Murray, Zarnie Khadjesari, Ian R. White, Eleftheria Kalaitzaki, Christine Godfrey, Jim McCambridge
Citations205
SJR quartileQ1
SJR score1.99
SNIP1.96
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The major methodological issues that arise in trials are described (recruitment, randomization, fidelity of the intervention, retention, and data quality), how the online context affects these issues are considered, and potential solutions are used to illustrate potential solutions.
Abstract
Health care and health care services are increasingly being delivered over the Internet. There is a strong argument that interventions delivered online should also be evaluated online to maximize the trial's external validity. Conducting a trial online can help reduce research costs and improve some aspects of internal validity. To date, there are relatively few trials of health interventions that have been conducted entirely online. In this paper we describe the major methodological issues that arise in trials (recruitment, randomization, fidelity of the intervention, retention, and data quality), consider how the online context affects these issues, and use our experience of one online trial evaluating an intervention to help hazardous drinkers drink less (DownYourDrink) to illustrate potential solutions. Further work is needed to develop online trial methodology.
Keywords
PsychologyHealth Professions
JAMAEmpirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials
5,580 Citations1995Klaus‐Peter Schulz
Empirical evidence is provided that inadequate methodological approaches in controlled trials, particularly those representing poor allocation concealment, are associated with bias.
Journal of Medical Internet ResearchThe Law of Attrition
2,589 Citations2005Günther Eysenbach
The need for a “science of attrition” is argued, that is, a need to develop models for discontinuation of e health applications and the related phenomenon of participants dropping out of eHealth trials, as well as measures to be reported including the relative risk of dropping out or of stopping the use of an application.
The LancetExternal validity of randomised controlled trials: “To whom do the results of this trial apply?”
2,438 Citations2005Peter M. Rothwell
A review of the determinants of external validity in trial publications and systematic reviews is usually inadequate, a checklist for clinicians is presented, and recommendations for greater consideration of External validity in the design and reporting of RCTs are made.
Enlighten: Publications (The University of Glasgow)Developing and Evaluating Complex Interventions: New Guidance
1,892 Citations2006Peter Craig, Paul Dieppe +4 more
This research highlights the need to understand more fully the rationale behind the continued use of EMMARM, as well as the barriers to doing so, in patients and clinicians.
Psychological MedicineInternet-based cognitive behaviour therapy for symptoms of depression and anxiety: a meta-analysis
1,643 Citations2006Viola Spek, Pim Cuijpers +4 more
In general, effect sizes of internet-based interventions for symptoms of anxiety were larger than effect sizes for depressive symptoms; however, this might be explained by differences in the amount of therapist support.
BMJDesigning and evaluating complex interventions to improve health care
1,334 Citations2007Neil Campbell, Elizabeth Murray +8 more
The importance of ground work in getting usable results is explained and the effectiveness of complex interventions can be difficult and time consuming.
BMJHealth related virtual communities and electronic support groups: systematic review of the effects of online peer to peer interactions
1,296 Citations2004Günther Eysenbach, John Powell +3 more
Journal of Medical Internet ResearchThe Effectiveness of Web-Based vs. Non-Web-Based Interventions: A Meta-Analysis of Behavioral Change Outcomes
999 Citations2004Dean Wantland, Carmen J. Portillo +3 more
An improvement in outcomes for individuals using Web-based interventions to achieve the specified knowledge and/or behavior change for the studied outcome variables showed an improvement in outcome size comparisons.
Journal of Medical Internet ResearchUsing the Internet for Surveys and Health Research
810 Citations2002Günther Eysenbach, Jeremy C Wyatt
This paper concerns the use of the Internet in the research process, from identifying research issues through qualitative research, through using the Web for surveys and clinical trials, to pre-publishing and publishing research results.
Cochrane Database of Systematic ReviewsInteractive Health Communication Applications for people with chronic disease
718 Citations2005Elizabeth Murray, Joanne Burns +3 more
IHCAs were found to have a negative effect on clinical outcomes, and consumers who wish to increase their knowledge or social support amongst people with a similar problem may find an IHCA helpful, however, consumers whose primary aim is to achieve optimal clinical outcomes should not use an IhCA at present.
Clinical TrialsAre missing outcome data adequately handled? A review of published randomized controlled trials in major medical journals
557 Citations2004Angela Wood, Ian R. White +1 more
It is shown that missing outcome data are a common problem in randomized controlled trials, and are often inadequately handled in the statistical analysis in the top tier medical journals.
Journal of Medical Internet ResearcheHealth Trends in Europe 2005-2007: A Population-Based Survey
456 Citations2008Per Egil Kummervold, Catherine Chronaki +6 more
Use of the Internet for health purposes is growing in all age groups and for both men and women, with especially strong growth among young women.
Health Technology AssessmentRecruitment to randomised trials: strategies for trial enrolment and participation study. The STEPS study
440 Citations2007Marion Campbell, Claire Snowdon +9 more
It is suggested that future trials should consider the different needs at different phases in the life of trials, and place greater emphasis on 'conduct' (the process of actually doing trials).
Professional Psychology Research and PracticeInternet interventions: In review, in use, and into the future.
364 Citations2003Lee M. Ritterband, Linda Gonder‐Frederick +4 more
A critical examination of the current state of the available literature focused on the development and testing of Internet interventions is presented, followed by a discussion of possible future directions and implications for research and clinical practice.
AddictionWeb‐based self‐help for problem drinkers: a pragmatic randomized trial
278 Citations2008Heleen Riper, Jeannet Kramer +4 more
The intervention showed itself to be effective in reducing problem drinking in the community and is one of the first randomized controlled trials on a web-based self-help intervention without therapist guidance for self-referred problem drinkers among the adult general population.
Review of the effectiveness of treatment for alcohol problems.
264 Citations2006Duncan Raistrick, Nick Heather +1 more
Journal of Medical Internet ResearchReach, Engagement, and Retention in an Internet-Based Weight Loss Program in a Multi-Site Randomized Controlled Trial
223 Citations2007Russell E. Glasgow, Candace C. Nelson +7 more
A single personalized mailing increases enrollment in Internet-based weight loss and patient characteristics related to each of these important factors may be different, and more comprehensive analyses of determinants of enrollment, engagement, and retention in eHealth programs are needed.
Journal of Medical Internet ResearchRates and Determinants of Repeated Participation in a Web-Based Behavior Change Program for Healthy Body Weight and Healthy Lifestyle
202 Citations2007Marieke W. Verheijden, Marielle P. Jans +2 more
It is found that obese people were more likely to participate in follow-up than people of normal body weight and the non-stigmatizing way of addressing body weight through the Internet may be part of the explanation for this.
Cochrane Database of Systematic ReviewsStrategies to improve recruitment to research studies
197 Citations2007James Mapstone, Diana Elbourne +1 more
It is not possible to predict the effect most interventions will have on recruitment, and funders and researchers should ensure that the evaluation of recruitment strategies are incorporated into research studies.
BMJMissing data
186 Citations2007Douglas G. Altman, Martin Bland
A few missing observations are a minor nuisance, but a large amount of missing data is a major threat to a study's integrity, so it is important to try to maximise data collection.
Journal of Medical Internet ResearchChallenges of Internet Recruitment: A Case Study with Disappointing Results
185 Citations2005Malcolm Koo, Harvey A. Skinner
Overall, the recruitment rate was disappointingly low and recruitment using Internet technologies including email, electronic discussion boards, Usenet forums, and websites did not prove to be an effective approach for soliciting young subjects to participate in the research.
BMJAdequacy and reporting of allocation concealment: review of recent trials published in four general medical journals
175 Citations2005Catherine Hewitt, Seokyung Hahn +3 more
The prevalence of adequate allocation concealment and its association with the statistical significance of trial results are reviewed and an attempt is made to identify randomised controlled trials published from January 2002 to December 2002.
Journal of Computer-Mediated CommunicationThe Story of Subject Naught: A Cautionary but Optimistic Tale of Internet Survey Research
174 Citations2006Joseph A. Konstan, B. R. Simon Rosser +3 more
In a web-based, sexual behavior risk study using a rigorous response validation protocol, 124 invalid responses out of 1,150 total were identified, nearly all of which were due to repeat survey submissions from the same participants.
UCL Discovery (University College London)Interactive Health Communication Applications for people with chronic disease (Withdrawn Paper. 2004, art. no. CD004274.pub3)
172 Citations2005Eleanor J. Murray, Joanne Burns +3 more
Health Technology AssessmentInterferon alfa (pegylated and non-pegylated) and ribavirin for the treatment of mild chronic hepatitis C: a systematic review and economic evaluation
162 Citations2007Jonathan Shepherd, John E. Jones +4 more
It is shown that patients with histologically mild HCV can be successfully treated with both pegylated and non-pegylated interferon alfa, and early treatment and watchful waiting strategies are associated with acceptable cost-per-QALY estimates.
AIDS CareRecruitment and retention of an online sample for an HIV prevention intervention targeting men who have sex with men: the smart sex quest project
150 Citations2004Sheana Bull, Laura V. Lloyd +2 more
While the Internet is a valuable tool for conducting research, conducting this longitudinal research online was severely affected by a loss to follow-up, and analyzing outcome data was hampered by significant differences between those who did and did not complete the study.
Controlled Clinical TrialsPerceptions of equipoise are crucial to trial participation: a qualitative study of men in the ProtecT study
141 Citations2003Nicola Mills, Jenny Donovan +4 more
Patients' perceptions of randomization and reasons for consent or refusal to participate in the ProtecT study (a randomized trial of surgery, radiotherapy, and monitoring for localized prostate cancer) were explored, and belief in clinical equipoise was key to participants' consent to randomization.
Journal of Medical Internet ResearchInternet-Based Interactive Health Intervention for the Promotion of Sensible Drinking: Patterns of Use and Potential Impact on Members of the General Public
132 Citations2007Stuart Linke, Elizabeth Murray +2 more
The Web-based intervention was highly used, and those who stayed with the program showed significant reductions in self-reported indicators of dependency, alcohol-related problems, and mental health symptoms; however, this association cannot be assumed to be causal.
Health Technology AssessmentA systematic review and economic evaluation of computerised cognitive behaviour therapy for depression and anxiety
110 Citations2002Eva Kaltenthaler, Phil Shackley +4 more
Assessment of the clinical effectiveness of cognitive behaviour therapy for treating anxiety, depression and phobias and the cost-effectiveness of CCBT with CBT by conventional methods and with treatment as usual found some evidence of poor-to-moderate quality that CCBT is as effective as TCBT in clinically depressed, anxious or phobic populations.
Statistical Methods in Medical ResearchUses and limitations of randomization-based efficacy estimators
102 Citations2005Ian R. White
Situations where intention-to-treat analysis is insufficient and a randomization-based method is useful include provision of patient information, exploration of treatment-covariate and treatment-time interactions, meta-analysis, and equivalence trials.
Behavior Research MethodsNonequivalence of on-line and paper-and-pencil psychological tests: The case of the prospective memory questionnaire
100 Citations2005Tom Buchanan, Tarick Ali +5 more
The prospective memory questionnaire (PMQ) was used in an on-line study exploring links between drug use and memory, and only two factors could be recovered; the other two subscales were essentially meaningless.
Journal of Medical Internet ResearchIssues in evaluating health websites in an Internet-based randomized controlled trial
100 Citations2002Günther Eysenbach
An Internet-based, cognitive therapy, self-help program to be used as a stand-alone intervention for mild-to-moderate depression, or as an adjunct to traditional services for more severe depression, failed to find an effect across the entire sample.
Family PracticePatients' views and feelings on the community-based teaching of undergraduate medical students: a qualitative study
95 Citations2002Karen J. Coleman
Patients enjoy their involvement in community-based teaching and perceive themselves as making a valuable contribution, and the findings of the research will be reassuring for doctors who presently are involved and those who plan to be involved in the future.
Alcohol and AlcoholismDOWN YOUR DRINK: A WEB-BASED INTERVENTION FOR PEOPLE WITH EXCESSIVE ALCOHOL CONSUMPTION
92 Citations2003Stuart Linke
Web site interventions for excessive drinkers are feasible and merit evaluation of their effectiveness, and little information was obtained from those who dropped out, but some reported that the programme was too time-consuming.
Journal of Medical Internet ResearchThe Internet and Clinical Trials: Background, Online Resources, Examples and Issues
88 Citations2005James Paul, Rachael Seib +1 more
The historical development of the Internet and the randomized controlled trial is reviewed, the Internet resources available that can be used in a clinical trial are described, some examples of online trials are reviewed and the advantages and disadvantages of using the Internet to conduct a clinicaltrial are described.
BMJConducting clinical trials over the internet: feasibility study
88 Citations2003Timothy E. McAlindon, Margaret Formica +3 more
Alcohol and AlcoholismDevelopment of a Psychologically Enhanced Interactive Online Intervention for Hazardous Drinking
70 Citations2008Stuart Linke, Jim McCambridge +3 more
Presentation of detailed information on the theoretical underpinning, content and structure of an intervention makes it easier to interpret the results of any evaluation and is likely to be of use to those developing other online interventions for alcohol or other health behaviours.
SSRN Electronic JournalOxford Internet Survey 2005 Report: The Internet in Britain
66 Citations2005William H. Dutton, Corinna di Gennaro +1 more
This report describes users, ex-users and non-users, based on such attributes as their demographic characteristics, uses of, and attitudes towards, ICTs, and how people with different backgrounds use and think about the Internet.
Journal of Medical Internet ResearchFollowing Up Nonrespondents to an Online Weight Management Intervention: Randomized Trial Comparing Mail versus Telephone
64 Citations2007Mick P. Couper, Andy Peytchev +3 more
Mail is an effective way to reduce attrition to online surveys, while telephone follow-up might lead to overestimating the weight loss for both the treatment and control groups.
Drug and Alcohol DependenceComparative psychometric study of a range of hazardous drinking measures administered online in a youth population
57 Citations2008Bonnita A. Thomas, Jim McCambridge
A range of hazardous drinking measures exhibit sound psychometric properties when administered online, and internal consistency and test-retest correlation statistics were generally satisfactory, providing evidence of reliability.
BMC Public HealthThe DYD-RCT protocol: an on-line randomised controlled trial of an interactive computer-based intervention compared with a standard information website to reduce alcohol consumption among hazardous drinkers
40 Citations2007Elizabeth Murray, Jim McCambridge +6 more
If a comparable reduction of sick leave duration of patients with chronic LBP of who attend an outpatient clinic can be achieved, such reductions will be obviously substantial for the Netherlands and will have a considerable impact.
Addictive BehaviorsTest–retest reliability of an online measure of past week alcohol consumption (the TOT-AL), and comparison with face-to-face interview
37 Citations2008Zarnie Khadjesari, Elizabeth Murray +5 more
The TOT-AL is a reliable, time efficient means of ascertaining alcohol intake, equivalent to that obtained face-to-face, and these findings support the use of this approach to online alcohol assessment in populations with access to the Internet.
Journal of Medical Internet ResearchCan Clinical Trials Requiring Frequent Participant Contact Be Conducted Over the Internet? Results From an Online Randomized Controlled Trial Evaluating a Topical Ointment for Herpes Labialis
23 Citations2004Margaret Formica, Karim Kabbara +2 more
Evaluated the feasibility of a Web-based randomized clinical trial requiring frequent participant interaction, using a 6-month, double-blind, randomized, placebo-controlled pilot trial of a topical ointment containing dioctyl sodium sulfosuccinate intended for treatment of recurrent herpes labialis, and achieved high overall retention rates.
Evaluation and Program PlanningeHealth recruitment challenges
22 Citations2006Debbe Thompson, Ashanti Canada +6 more
Recruitment challenges associated with enrolling African-American girls aged 8-10 years in an eHealth obesity prevention program are presented, their effect on the recruitment plan, and potential implications for eHealth research are presented.