Inter-review agreement of risk-of-bias judgments varied in Cochrane reviews

Ref ID 343
First Author N. Konsgen
Journal JOURNAL OF CLINICAL EPIDEMIOLOGY
Year Of Publishing 2020
URL https://www.jclinepi.com/article/S0895-4356(19)30647-X/fulltext
Keywords • General medical
• Risk of bias
• Cochrane
Problem(s) • Flawed risk of bias undertaken
• Weaknesses identified in some Cochrane reviews
Article Type Empirical
Article Subtype Cross-sectional survey/Methodological systematic review
First Author Country Germany
Aim To assess the level of agreement between Cochrane reviews of overlapping randomised controlled trials regarding risk-of-bias judgments for Cochrane reviews of tobacco up to November 2017.
Level of Investigation Descriptive
Summary of Findings The level of agreement between Cochrane reviews which included overlapping randomised controlled trials regarding risk of bias judgments ranged from slight to substantial depending on the risk of bias domain: ‘‘allocation concealment’’: kappa = 0.51 (0.41 to 0.61), kappa = 0.75; ‘‘blinding’’: kappa = 0.19 (0.02 to 0.37), kappa = 0.52; ‘‘blinding of outcome assessment’’: kappa = 0.43 (0.14 to 0.72) kappa = 0.67; and ‘‘incomplete outcome data’’: kappa = 0.15 (-0.03 to 0.32), kappa = 0.64. For ‘‘blinding of participants and personnel’’ and ‘‘selective reporting’’, kappa could not be calculated. The raw agreement was 0.40 and 0.42, respectively.
Number of systematic reviews included 53
Number of eligible systematic reviews assessed 71
Treatment impacted No
Treatment impacted description
Interpretation impacted Not Applicable
Interpretation impacted description