Raw data for: GRADE and CINeMA showed moderate to substantial concordance in the evaluation of certainty of the evidence
Description
Since 2014, two approaches have been developed to assess the certainty of the evidence of network meta-analysis results: the approach developed by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) and the Confidence in Network Meta-Analysis (CINeMA) approach, inspired by the methodology developed by the GRADE Working Group. GRADE and CINeMA were applied to four networks of different sizes and structures related to different outcomes in different clinical conditions and settings by 13 evaluators. All domains in CINeMA have three levels of ratings (i.e. no concern, some concerns or major concerns). GRADE has also three levels of ratings (i.e. not serious, serious and very serious) except for two domains: publication bias has two (i.e. undetectable versus strongly detectable), and imprecision has four ratings (i.e. not serious, serious, very serious and very very serious).
Files
Steps to reproduce
Our unit of analysis was the number of evaluations. Given the ordinal nature of the ratings, we used Gwet's AC and the corresponding 95% confidence interval (CI) to measure the IRR within each method and for the concordance between methods for each network in the overall assessment (Gwet 2008). For the concordance between GRADE and CINeMA, we used the medians of assessors' responses. In order to account for the severity of disagreements between ordinal ratings, we applied linear weights, meaning that weights decrease linearly as the disagreement between ratings increases. Gwet's AC handles multiple raters for the measurement of IRR and it also handles chance agreement, especially in the presence of imbalanced marginal distributions, the main cause of the "paradox" of Cohen's Kappa, or small sample sizes. We upload all the materials used by assessors for facilitating their ratings both with GRADE and CINeMA.