A re-analysis of the Cochrane Library data: the dangers of unobserved heterogeneity in meta-analyses

Ref ID 161
First Author E. Kontopantelis
Journal PLOS ONE [ELECTRONIC RESOURCE]
Year Of Publishing 2013
URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3724681/pdf/pone.0069930.pdf
Keywords Cochrane
Statistical
Heterogeneity
General medical
Problem(s) Inadequate analysis of heterogeneity
Lack of statistical expertise in handling of quantitative data
Number of systematic reviews included 2801
Summary of Findings From 2801 Cochrane reviews, 32,005 meta-analyses and 25,392 subgroup meta-analyses were re-analysed. Re-analysing all meta-analyses with a new method to detect heterogeneity found that in cases where heterogeneity had originally been detected but ignored, 17–20% of the statistical conclusions changed. Rates were much lower where the original analysis did not detect heterogeneity or took it into account, between 1% and 3%. A comparison between real and simulated data using the DerSimonian-Laird method suggested that mean true heterogeneity is higher than assumed or estimated in practice (higher than moderate) but the standard method fails to detect it, especially for meta-analyses of few studies. Most methods that allow the heterogeneity estimator to be zero, including the common DerSimonian-Laird approach, perform well on average, even for extreme distributions of the effects. However, these methods quite often fail to detect heterogeneity and thus produce biased estimates and conclusions, especially for small meta-analyses.
Did the article find that the problem(s) led to qualitative changes in interpretation of the results? Yes
Are the methods of the article described in enough detail to replicate the study? No