An Information Theoretic Measure to Evaluate Parsing Difficulty Across Treebanks

With the growing interest in statistical parsing, special attention has been recently devoted to the problem of comparing different treebanks, with the goal of assessing which languages or which domains are more “difficult” to parse relative to a given model. A common methodology in comparing parsing difficulty across treebanks is based on the use of the standard labeled precision and recall measures. As an alternative, in this article we propose an information theoretic measure, called the expected conditional cross-entropy (ECC). One important advantage with respect to standard performance measures is that ECC can be directly expressed as a function of the parameters of the model. We evaluate ECC across several treebanks for English, French, German and Italian, and show that ECC is an effective measure of parsing difficulty, with an increase in ECC being always accompanied by a degradation in parsing accuracy.