Reenvisioning the comparison between Neural Collaborative Filtering and Matrix Factorization

Configuration file, models and basic script to reproduce in Elliot the experiments from the paper: Reenvisioning-the-comparison-between-Neural-Collaborative-Filtering-andMatrix-Factorization

Preliminary steps

Clone this repo and create a pyhton environment to run these experiments:

CONDA

cd <repo_path>
conda create --name elliot_env python=3.8
conda activate elliot_env
pip install --upgrade pip

VIRTUALENV

cd <repo_path>
virtualenv -p /usr/bin/python3.6 venv # your python location and version
source venv/bin/activate
pip install --upgrade pip

Replicate all baselines performance

The results of Table 1-3 and Figure 2-3-4 were generated with the following scripts and by varying the model under test

Movielens:

python experiments_ml1m.py --model <model_name>

Pinterest:

python experiments_p20.py --model <model_name>

The placeholder <model_name> must be in the range [most_pop, MF, NeuMF, ials, slim, easer, rp3beta, pure_svd]. Be careful it is case sensitive.

All hyperparameters and configuration details for each models are placed into configuration files folder.

These experiments generate all needed files to execute the overall evaluation to receive all performance and statistical significance tests.

To execute the complete evaluation, copy all recommendation files from the result folder into a separate one, and then run the following script choosing the appropriate dataset name (movielens or pinterest)

python evaluation.py --dataset <dataset_name>

This script is driven by the file named test_config_{movilens/pinterest}.yml. Edit the placeholder with the specific recommendation folder.

MF vs NeuMF

The first and third columns of table 2 and highlight points in figure 1 (line blue and green) were generated by running these scripts and varying the embedding dimension from 24 to 192 into the appropriate configuration file and the model's name to test (MF or NeuMF).