A Modified Weighted Kappa for the Comparison of Different Measurement Scales

This paper investigates possible causes of the discordance between two ordinal scales evaluating the same good or service. In psychometric literature, Cohen’s Kappa is one of the most important index to evaluate the strength of agreement, or disagreement, between two nominal variables, in particular in its weighted version. In this paper a Modified Weighted Kappa is proposed. A proper procedure to determine the lower and upper triangle in a non-square table is also implemented, so to generalize the index in order to compare two scales with different number of categories. With the aim to verify the tendency of a scale to have a mainly positive or negative rating compared to a different one, a parametric test is set up. A study with real data is also conducted.