Inter-review agreement of risk-of-bias judgments varied in Cochrane reviews

Ref ID 343
First Author N. Konsgen
Journal JOURNAL OF CLINICAL EPIDEMIOLOGY
Year Of Publishing 2020
URL https://www.jclinepi.com/article/S0895-4356(19)30647-X/fulltext
Keywords Cochrane
Risk of bias
General medical
Problem(s) Flawed risk of bias undertaken
Weaknesses identified in some Cochrane reviews
Number of systematic reviews included 53
Summary of Findings The level of agreement between Cochrane reviews which included overlapping randomised controlled trials regarding risk of bias judgments ranged from slight to substantial depending on the risk of bias domain: ‘‘allocation concealment’’: kappa = 0.51 (0.41 to 0.61), kappa = 0.75; ‘‘blinding’’: kappa = 0.19 (0.02 to 0.37), kappa = 0.52; ‘‘blinding of outcome assessment’’: kappa = 0.43 (0.14 to 0.72) kappa = 0.67; and ‘‘incomplete outcome data’’: kappa = 0.15 (-0.03 to 0.32), kappa = 0.64. For ‘‘blinding of participants and personnel’’ and ‘‘selective reporting’’, kappa could not be calculated. The raw agreement was 0.40 and 0.42, respectively.
Did the article find that the problem(s) led to qualitative changes in interpretation of the results? Not Applicable
Are the methods of the article described in enough detail to replicate the study? Yes