| Ref ID | 467 |
| First Author | B. Tendal |
| Journal | BMJ |
| Year Of Publishing | 2009 |
| URL | https://www.bmj.com/content/bmj/339/bmj.b3128.full.pdf |
| Keywords |
• Cochrane • Author • General medical • Error • Expertise • Statistical |
| Problem(s) |
• Errors in effect estimate calculations or data synthesis • Lack of statistical expertise in handling of quantitative data • Data extraction errors and double counting |
| Article Type | Empirical |
| Article Subtype | Inter-rater agreement study |
| First Author Country | Denmark |
| Aim | To assess the inter-observer variation between 5 experienced methodologists and 5 PhD students for extraction and calculation of the first standardised mean difference related to extraction of continuous and numerical rating scale data from trial reports for use in Cochrane meta-analyses. |
| Level of Investigation | Analytical |
| Summary of Findings | Agreement was 53% at trial level and 31% at meta-analysis level. Reasons for disagreement reasons for disagreement were broadly related to: different choices, exclusion of a trial, and data extraction errors. In 14 out of the 100 standardised mean differences calculated at the meta-analysis level, individual observers reached different conclusions than the originally published review. |
| Number of systematic reviews included | 10 |
| Number of eligible systematic reviews assessed | 32 |
| Treatment impacted | Yes |
| Treatment impacted description | |
| Interpretation impacted | Yes |
| Interpretation impacted description | Ten per cent of the disagreements at the meta-analysis level amounted to an standardised mean difference of at least 1, which is far greater than the effect of most of the treatments we use compared with no treatment |