/recsys_eval

How Sensitive is Recommendation Systems' Offline Evaluation to Popularity?

Primary LanguageJupyter NotebookApache License 2.0Apache-2.0

How Sensitive is Recommendation Systems' Offline Evaluation to Popularity

This is the implementation of the following paper:

@InProceedings{recsys_eval19,
  author    = {Amir H. Jadidinejad and Craig Macdonald and Iadh Ounis},
  title     = {How Sensitive is Recommendation Systems' Offline Evaluation to Popularity?},
  booktitle = {In Workshop on Offline Evaluation for Recommender Systems (REVEAL2019) at the 13th ACM Conference on Recommender Systems.},
  year      = {2019},
}

Requirements

Results

The following plot summarizes the results of popularity-stratified sampling:

popularity stratified sampling

By setting P threshold to maximum, evaluation of models is corresponding to the offline recommendation system's evaluation:

See the paper or our poster for more details.

How to reproduce?

Use the corresponding Jupyter notebook to reproduce the results of each dataset (MovieLens, Amazon) for a specific popularity threshold P: