/proficiency-metric

Primary LanguagePythonApache License 2.0Apache-2.0

An Information Theoretic Metric for Multi-Class Categorization

python

The implementation of the Proficiency Metric in various settings:

  • predeval.py:ConfusionMX: Classification

  • predeval.py:MuLabCat: Multi-Label Categorization

paper

The research paper describing the Proficiency Metric.

data

The test data for the results in the paper:

  • annotated: human-labeled examples

  • results: algorithmically categorized examples

  • labeler[123].txt: KDD Cup 2005

  • Trec_beta: Example test data for ERD 2014 - http://web-ngram.research.microsoft.com/erd2014/Datasets.aspx

  • sk: 100 Magnetic queries annotated with Wikipedia concepts in similar format as ERD 2014 (Trec_beta)

  • magnetic: hand-annotated examples from searches in the Magnetic.com database of search events