Cohen's kappa calculation

[afergadis]

In your calculation, you always take the same number of instances num_intsances and not the number of instances that ann1 & ann2 have annotated. So, by your example, for annotators b and c num_instances = 15 but the common annotated instances are only 5.

df = self.df.dropna(subset=[ann1, ann2])
ann1_labels = df[ann1].values.tolist()
ann2_labels = df[ann2].values.tolist()
num_instances = self.df.shape[0]

I think that the last line of the above snippet should not be self.df.shape[0] but df.shape[0]

num_instances = df.shape[0]