Issues in Comparisons Between Meta-analyses and Large Trials
JAMAPublished 8 April 1998
John P. A. Ioannidis
Citations253
SJR quartileQ1
SJR score5.35
SNIP10.71
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
To evaluate protocols comparing meta-analyses and large trials in order to understand if and why they disagree on the concordance of these 2 clinical research methods, systematic comparison of protocol designs, study selection, definitions of agreement, analysis methods, and reported discrepancies are evaluated.
Abstract
Comparisons of large trials with meta-analyses may reach different conclusions depending on how trials and meta-analyses are selected and how end points and agreement are defined. Scrutiny of these 2 major research methods can enhance our appreciation of both for guiding medical practice.
Keywords
Decision SciencesMathematics
BMJBias in meta-analysis detected by a simple, graphical test
56,268 Citations1997Matthias Egger, George Davey Smith +2 more
Funnel plots, plots of the trials' effect estimates against sample size, are skewed and asymmetrical in the presence of publication bias and other biases Funnel plot asymmetry, measured by regression analysis, predicts discordance of results when meta-analyses are compared with single large trials.
BiometricsOperating Characteristics of a Rank Correlation Test for Publication Bias
17,035 Citations1994Colin B. Begg, Madhuchhanda Mazumdar
An adjusted rank correlation test is proposed as a technique for identifying publication bias in a meta-analysis, and its operating characteristics are evaluated via simulations, and the test statistic is a direct statistical analogue of the popular "funnel-graph".
Psychological BulletinWeighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.
8,537 Citations1968Jacob Cohen
The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi.
JAMAImproving the quality of reporting of randomized controlled trials. The CONSORT statement
3,343 Citations1996Colin B. Begg
Annals of Internal MedicineQuantitative Synthesis in Systematic Reviews
2,477 Citations1997Joseph T. F. Lau, John P.A. Ioannidis +1 more
A stepwise description of the tasks that are performed when statistical methods are used to combine data and the question, Are the results of the different studies similar (homogeneous)?
Epidemiologic ReviewsQUANTITATIVE METHODS IN THE REVIEW OF EPIDEMIOLOGIC LITERATURE1
1,655 Citations1987Sander Greenland
New England Journal of MedicineDiscrepancies between Meta-Analyses and Subsequent Large Randomized, Controlled Trials
1,184 Citations1997Jacques LeLorier, Geneviève Grégoire +3 more
The outcomes of the 12 large randomized, controlled trials that were studied were not predicted accurately 35 percent of the time by the meta-analyses published previously on the same topics.
American Journal of Public HealthUses of ecologic analysis in epidemiologic research.
805 Citations1982Hal Morgenstern
This paper reviews the major types of ecologic study designs, the analytic methods appropriate for each, the limitations ofEcologic data for making causal inferences and what can be done to minimize these problems, and the relative advantages of Ecologic analysis.
Controlled Clinical TrialsPublication bias and clinical trials
721 Citations1987Kay Dickersin, Simon Chan +3 more
It appears that nonpublication was primarily a result of failure to write up and submit the trial results rather than rejection of submitted manuscripts, implying the existence of a publication bias of importance both to meta-analysis and the interpretation of statistically significant positive trials.
The LancetSumming up evidence: one answer is not always enough
659 Citations1998Joseph Lau, John P. A. Ioannidis +1 more
Here a more rounded approach to meta-analyses is argued, arguing that their strengths outweigh their weaknesses, although the latter must not be brushed aside.
The LancetIntravenous magnesium sulphate in suspected acute myocardial infarction: results of the second Leicester Intravenous Magnesium Intervention Trial (LIMIT-2)
534 Citations1992K L Woods, Nick Fletcher +2 more
Intravenous magnesium sulphate treatment is a simple, safe, and widely applicable treatment and its efficacy in reducing early mortality of myocardial infarction is comparable to but independent of, that of thrombolytic or antiplatelet therapy.
Statistics in MedicinePractical methodology of meta‐analyses (overviews) using updated individual patient data
492 Citations1995Lesley Stewart, Cochrane Working Group On Meta‐Analysis Using Individual Patient Data
Meta-analyses using updated individual patient data may provide the most reliable means of combining data from similar randomized controlled trials, and practical advice on initiating and maintaining collaboration and methods of data checking and validation are given.
The LancetOvercoming the limitations of current meta-analysis of randomised controlled trials
427 Citations1998Janice Pogue, Salim Yusuf
It is believed that rigorous meta-analyses undertaken according to these principles will lead to more reliable evidence about the efficacy and safety of interventions than either retrospective meta-analysis or individual trials.
New England Journal of MedicineThe Promise and Problems of Meta-Analysis
405 Citations1997John C. Bailar
This issue of the Journal, LeLorier et al.1 compare the findings of 12 large randomized, controlled trials with the results of meta-analyses of the same problems.
American Journal of Epidemiologyp Values, Hypothesis Tests, and Likelihood: Implications for Epidemiology of a Neglected Historical Debate
305 Citations1993Steven N. Goodman
An analysis using another method promoted by Fisher, mathematical likelihood, shows that the p value substantially overstates the evidence against the null hypothesis.
New England Journal of MedicineA Show of Confidence
303 Citations1978Kenneth J. Rothman
The preoccupation with significance tests is embodied in the focus on whether the P value is less than 0.05; results are considered "significant" or "not significant" according to whether or not the P values are less than or greater than this value.
CHEST JournalClinical Recommendations Using Levels of Evidence for Antithrombotic Agents
281 Citations1995Deborah J. Cook, Gordon H. Guyatt +3 more
The LancetPredictive ability of meta-analyses of randomised controlled trials
228 Citations1995José Villar, G. Carroli +1 more
Further evaluation of the meta-analytical method is needed if the qualitative and quantitative results it yields are to be better understood, and a meta-analysis demonstrating a protective effect from an intervention of more than 40% had a 60% probability of correctly predicting results of the same magnitude of the largest trial.
Statistics in MedicineAn empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials
218 Citations1998Christopher H. Schmid, Joseph Lau +2 more
The significant correlation between the CR and the TE suggests that, rather than merely pooling the TE into a single summary estimate, investigators should search for the causes of heterogeneity related to patient characteristics and treatment protocols to determine when treatment is most beneficial and that they should plan to study this heterogeneity in clinical trials.
BMJMeta-analysis and the meta-epidemiology of clinical research
210 Citations1997C. David Naylor
This week's BMJ contains a pot-pourri of materials that deal with the research methodology of meta-analysis, with many studies have documented publication bias favouring clinical trials that show a significant treatment effect.
BMJThe relation between treatment benefit and underlying risk in meta-analysis
169 Citations1996Stephen J. Sharp, Simon G. Thompson +1 more
Meta-analyses of clinical trials comparing a treated group with a control group should be replaced either by statistically appropriate (but more complex) approaches or, preferably, by analyses which investigate the dependence of the treatment effect on measured baseline characteristics of the patients in each trial.
Annals of Internal MedicineDiscordance between Meta-analyses and Large-Scale Randomized, Controlled Trials: Examples from the Management of Acute Myocardial Infarction
132 Citations1995Steven Borzak, Paul M. Ridker
A major strength of meta-analysis is its ability to discern small but potentially important clinical effects, and this work examines possible reasons for the discrepancy between meta-analytic and megatrial evidence about nitrate therapy and magnesium therapy.
Statistics in MedicineTHE POPULATION RISK AS AN EXPLANATORY VARIABLE IN RESEARCH SYNTHESIS OF CLINICAL TRIALS
128 Citations1996MARTIN W. McINTOSH
This paper represents clinical trials in a bivariate two-level hierarchical model, and demonstrates how to estimate the parameters of the model by both maximum likelihood and Bayes procedures, and shows that using estimates of population risk as fixed covariates will result in bias.
PubMedMeta-analysis of randomized controlled trials. A concern for standards;.
119 Citations1995David Moher, Ingram Olkin
Journal of Clinical EpidemiologyCan treatment that is helpful on average be harmful to some patients? A study of the conflicting information needs of clinical inquiry and drug regulation
115 Citations1996Ralph I. Horwitz, Burton H. Singer +2 more
Differences in results across centers of a multicenter RCT may reflect important distinctions in the clinical conditions of enrolled subjects, which help to identify subgroups of patients in which treatment that has an average overall benefit may be harmful for some patients.
Journal of Clinical EpidemiologyThe impact of high-risk patients on the results of clinical trials
113 Citations1997John P.A. Ioannidis, Joseph Lau
Human immunodeficiency virus-related trials and trials of magnesium in acute myocardial infarction are analyzed and it can be seen that very sick patients of high risk become the major determinants of how many events occur in the whole population.
The American Journal of MedicineComplexity and contradiction in clinical trial research
110 Citations1987Ralph I. Horwitz
Results of this review suggest that pooled analyses of conflicting results of randomized clinical trials (meta-analyses) may be misleading by obscuring important distinctions among trials, and that enhanced flexibility in strategies for data analysis will be needed to ensure the clinical applicability of randomizedclinical trial results.
Controlled Clinical TrialsSample size determination in clinical trials with time-dependent rates of losses and noncompliance
109 Citations1986Edward Lakatos
A method of adjusting event rates in the presence of time-dependent rates of losses, noncompliance, and the like for sample size calculation is presented.
Annals of Internal Medicineβ-Blockers after Myocardial Infarction: Influence of First-Year Clinical Course on Long-Term Effectiveness
86 Citations1993Catherine M. Viscoli, Ralph I. Horwitz +1 more
The aim of the study was to demonstrate an analytic strategy that addressed the question of long-term therapeutic effectiveness while attempting to minimize the risk for bias in the answer.
The LancetMega-trials and management of acute myocardial infarction
72 Citations1995KL Woods
These issues are illustrated by the examples of nitrates, angiotensin-converting-enzyme inhibitors, and magnesium in acute myocardial infarction but have general relevance to the validity and generalisability of simple trials.
Controlled Clinical TrialsAdvantages and limitations of meta-analytic regressions of clinical trials data
69 Citations1992Jesse A. Berlin, Elliott M. Antman
Metaanalysis can explore variations in as well as summarize results of randomized trials, although metaregression has advantages, study characteristics are often strongly associated with each other, leading to collinearity.
American Journal of OphthalmologyLarge trials vs meta-analysis of smaller trials. How do their results compare?
50 Citations1997JC Cappelleri, J.P.A. Ioannidis +5 more
Controlled Clinical TrialsThe relationship between study design, results, and reporting of randomized clinical trials of HIV infection
44 Citations1997John P.A. Ioannidis, Joseph C. Cappelleri +2 more
Meta-analyses of published trials for specific treatments demonstrated a sizable treatment benefit for all the examined medications regardless of whether these medications were officially approved, controversial, or abandoned, raising concerns about either publication bias or unjustifiable rejection of potentially useful medications.
Journal of Acquired Immune Deficiency Syndromes & Human RetrovirologyPredictors and Impact of Patients Lost to Follow-Up in a Long-Term Randomized Trial of Immediate Versus Deferred Antiretroviral Treatment
42 Citations1997John P. A. Ioannidis, Roland Bassett +4 more
Losses to follow-up probably decreased substantially the observed number of primary endpoints, curtailed the power of the trial to demonstrate any difference between immediate and deferred initiation of antiretroviral therapy, and may have introduced large bias in the estimated hazard ratio for the primary endpoint and its statistical significance.
Statistics in MedicineCHANGES IN CLINICAL TRIALS MANDATED BY THE ADVENT OF META-ANALYSIS
24 Citations1996Thomas C. Chalmers, JOSEPH LAU
Heterogeneity of patients entering clinical trials is desirable and should be specifically studied, and it should never be concluded that an average outcome is applicable to all future patients.
JAMALarge Trials vs Meta-analysis of Smaller Trials-Reply
1 Citations1997John P. A. Ioannidis
It is puzzling why these authors attribute to us an a priori belief that large trials usually support the results of meta-analyses of small trials, and why Klebanoff et al seem to believe that de facto all treatments should work the same for all patients regardless of the underlying risk of the disease.