| Ref ID | 414 |
| First Author | J. B. Schroll |
| Journal | BMC MEDICAL RESEARCH METHODOLOGY |
| Year Of Publishing | 2011 |
| URL | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3056846/pdf/1471-2288-11-22.pdf |
| Keywords |
• Heterogeneity • Cochrane • General medical • Error |
| Problem(s) |
• Inadequate analysis of heterogeneity • Weaknesses identified in some Cochrane reviews • Errors in effect estimate calculations or data synthesis |
| Article Type | Empirical |
| Article Subtype | Cross-sectional survey/Methodological systematic review |
| First Author Country | Denmark |
| Aim | To assess how authors addressed different degrees of heterogeneity from a random sample of Cochrane reviews published in 2008, which presented a result in its first meta-analysis with substantial heterogeneity (I squared greater than 50%). |
| Level of Investigation | Descriptive |
| Summary of Findings | One third of reviews were regarded as problematic with regards to handling of heterogeneity and choice of statistical model for data synthesis. For 10% of included reviews, a significant result using a fixed-effects model changed to a non-significant when a random effects model was used. Appropriate caution was not expressed around the reliability of the pooled treatment effect in the majority of these reviews. One review calculated mean differences instead of standardized mean differences, although the outcomes were measured on very different scales. Whilst two thirds of reviews overall were devoid of major problems in relation to their handling of heterogeneity, only 27 reviews (45%) gave a rationale for choice of statistical model. |
| Number of systematic reviews included | 60 |
| Number of eligible systematic reviews assessed | 3385 |
| Treatment impacted | Yes |
| Treatment impacted description | |
| Interpretation impacted | Yes |
| Interpretation impacted description | Choice of statistical model was in some cases found to change significant treatment effects to non-significant. Fixed-effects models were sometimes found to be inappropriately applied to some meta-analyses without justification. |