Dealing with substantial heterogeneity in Cochrane reviews. Cross-sectional study

Ref ID 414
First Author J. B. Schroll
Journal BMC MEDICAL RESEARCH METHODOLOGY
Year Of Publishing 2011
URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3056846/pdf/1471-2288-11-22.pdf
Keywords Cochrane
Error
Heterogeneity
General medical
Problem(s) Inadequate analysis of heterogeneity
Weaknesses identified in some Cochrane reviews
Errors in effect estimate calculations or data synthesis
Number of systematic reviews included 60
Summary of Findings One third of reviews were regarded as problematic with regards to handling of heterogeneity and choice of statistical model for data synthesis. For 10% of included reviews, a significant result using a fixed-effects model changed to a non-significant when a random effects model was used. Appropriate caution was not expressed around the reliability of the pooled treatment effect in the majority of these reviews. One review calculated mean differences instead of standardized mean differences, although the outcomes were measured on very different scales. Whilst two thirds of reviews overall were devoid of major problems in relation to their handling of heterogeneity, only 27 reviews (45%) gave a rationale for choice of statistical model.
Did the article find that the problem(s) led to qualitative changes in interpretation of the results? Yes
Are the methods of the article described in enough detail to replicate the study? No