RRep: A composite index to assess and test rater precision

In subjective evaluation systems, raters act as measurement instruments providing useful evaluations for taking strategic and/or operational decisions. The assessment of rater evaluative ability in terms of accuracy and precision is of critical importance since rater unreliability may compromise the quality of the decision-making process. The focus of this paper is on rater precision: we propose a novel composite index to assess the rater ability to provide evaluations repeatable over time and reproducible over scales. The extent of rater precision is qualified via a nonparametric benchmarking procedure. The properties of both proposed index and benchmarking procedure have been analysed via a Monte Carlo simulation study.