Inferring Rater Agreement with Ordinal Classification

In several contexts ranging from medical to social sciences, rater reliability is assessed in terms of intra (-inter) rater agreement. The extent of rater agreement is commonly characterized by comparing the value of the adopted agreement coefficient against a benchmark scale. This deterministic approach has been widely criticized since it neglects the influence of experimental conditions on the estimated agreement coefficient. In order to overcome this criticism, in this paper an inferential procedure for benchmarking is presented. The proposed procedure is based on non-parametric bootstrap confidence intervals. The statistical properties of the proposed procedure have been studied for different sample sizes (i.e. n = 10, 30, 50, 100 items), rating scale dimensions (i.e. k = 2, 3, 5, 7 categories) and bootstrap confidence intervals (i.e. percentile and Bias-Corrected and Accelerated) via a Monte Carlo simulation.