A re-analysis of the Cochrane Library data: the dangers of unobserved heterogeneity in meta-analyses

Ref ID	161
First Author	E. Kontopantelis
Journal	PLOS ONE [ELECTRONIC RESOURCE]
Year Of Publishing	2013
URL	https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3724681/pdf/pone.0069930.pdf
Keywords	Cochrane Statistical Heterogeneity General medical
Problem(s)	Inadequate analysis of heterogeneity Lack of statistical expertise in handling of quantitative data
Number of systematic reviews included	2801
Summary of Findings	From 2801 Cochrane reviews, 32,005 meta-analyses and 25,392 subgroup meta-analyses were re-analysed. Re-analysing all meta-analyses with a new method to detect heterogeneity found that in cases where heterogeneity had originally been detected but ignored, 17–20% of the statistical conclusions changed. Rates were much lower where the original analysis did not detect heterogeneity or took it into account, between 1% and 3%. A comparison between real and simulated data using the DerSimonian-Laird method suggested that mean true heterogeneity is higher than assumed or estimated in practice (higher than moderate) but the standard method fails to detect it, especially for meta-analyses of few studies. Most methods that allow the heterogeneity estimator to be zero, including the common DerSimonian-Laird approach, perform well on average, even for extreme distributions of the effects. However, these methods quite often fail to detect heterogeneity and thus produce biased estimates and conclusions, especially for small meta-analyses.
Did the article find that the problem(s) led to qualitative changes in interpretation of the results?	Yes
Are the methods of the article described in enough detail to replicate the study?	No