It seems the code for kappa is wrong

Question

It seems the code for kappa is wrong

Closed this issue 8 years ago · 4 comments

I tested mKAPPA with one document and 5 raters and the results are very strange

mKAPPA([1,1,1,1,1],[1 2],'ordinal')
ans =
   NaN

(while I would expect to get 1)

and

mKAPPA([1,1,1,1,2],[1 2],'ordinal')
ans = 
  -2.7756e-16

(while I would expect to get a number close to 1)

Answer 1 · 2017-04-06T18:04:24.000Z

I close this because these degenerate cases are not handled well by Kappa index, not by your code.

Answer 2 · 2017-04-06T18:10:39.000Z

Hi Alessandro,
You're correct. This is an issue with the kappa formulation as opposed to the mKAPPA function. In your case, I might recommend using mSCOTTPI. Please let me know if you have any other issues or questions.

Answer 3 · 2017-04-14T20:58:43.000Z

Thank you very much for your answer. I'll look into it!

Answer 4 · 2017-04-15T16:38:56.000Z

mSCOTTPI([1,1,1,1,1],[1,2],'ordinal')
ans =
NaN

mSCOTTPI([1,1,1,1,2],[1,2],'ordinal')
ans =
-0.2500

mSCOTTPI([1,1,1,2,2],[1,2],'ordinal')
ans =
-0.2500

mSCOTTPI([1,1,2,2,2],[1,2],'ordinal')
ans =
-0.2500

mSCOTTPI([1,2,2,2,2],[1,2],'ordinal')
ans =
-0.2500

It seems that for one document and 5 raters this kind of systems doesn't make sense...