LibRerank is a toolkit for re-ranking algorithms. There are a number of re-ranking algorithms, such as PRM, DLCM, GSF, miDNN, SetRank, EGRerank, Seq2Slate. It also supports LambdaMART and DNN as initial ranker. In addition, an actively maintaing paper list on neural re-ranking for recommendation can be found here.
pip install --user virtualenv
~/.local/bin/virtualenv -p python3 ./venv
source venv/bin/activate
sudo apt-get install curl
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt-get install git-lfs
git lfs install
git clone https://github.com/LibRerank-Community/LibRerank.git
cd LibRerank
make init
Run initial ranker
bash example/run_ranker.sh
Run re-ranker
bash example/run_reranker.sh
Model parameters can be set by using a config file, and specify its file path at --setting_path, e.g., python run_ranker.py --setting_path config. The config files for the different models can be found in example/config. Moreover, model parameters can also be directly set from the command line. The supported parameters are listed as follows.
| argument | usage |
|---|---|
--data_dir |
The path to the directory where the data is stored |
--save_dir |
The path to the directory where the models and logs are stored |
--model_type |
The algorithm of reranker, including DNN and LambdaMARTPLEASE ATTENTION: Before training lambdaMART, you need to train DNN to get the pre-trained embedding |
--setting_path |
The path to the json config file, like files in example\config |
--data_set_name |
The name of the dataset, such as ad and prm |
--epoch_num |
The number of epoch for DNN model |
--batch_size |
Batch size for DNN model |
--lr |
Learning rate for DNN and lambdaMART |
--l2_reg |
The coefficient of l2 regularization for DNN model |
--eb_dim |
The size of embedding for DNN model |
--tree_num |
The number of trees for lambdaMART model |
--tree_type |
The type of tree for lambdaMART model, including lgb and sklearn |
| argument | usage |
|---|---|
--data_dir |
The path to the directory where the data is stored |
--save_dir |
The path to the directory where the models and logs are stored |
--setting_path |
The path to the json config file, like files in example\config |
--data_set_name |
The name of the dataset, such as ad and prm |
--initial_ranker |
The name of initial ranker, including DNN, lambdaMART. |
--model_type |
The name of the algorithm, including PRM, DLCM, GSF, SetRank, miDNN,Seq2Slate, EGR_evaluator, EGR_generator. |
--epoch_num |
The number of epoch |
--batch_size |
Batch size |
--lr |
Learning rate |
--l2_reg |
The coefficient of l2 regularization |
--eb_dim |
The size of embedding |
--hidden_size |
The size of hidden unit, usually the hideen size of LSTM/GRU |
--keep_prob |
Keep prob in dropout |
--metric_scope |
The scope of metrics, for example when --metric_scope=[1, 3, 5], MAP@1, MAP@3, and MAP@5 will be computed |
--max_norm |
The max norm of gradient clip |
--rep_num |
The number of repetitions during the training of the generator in EGRerank |
--group_size |
The group size for GSF model |
--c_enrropy |
The entropy coefficient in the loss for the generator in EGRerank |
--evaluator_path |
The path to the evaluator model ckpt when training the generator in EGRerank PLEASE ATTENTION: It's necessary to train the evaluator before generator |
DNN: a naive algorithm that directly train a multi-layer perceptron network with input labels (e.g., clicks).
LambdaMART: the implementation of the LambdaMART model in From RankNet to LambdaRank to LambdaMART: An Overview
DLCM: the implementation of the Deep Listwise Context Model in Learning a Deep Listwise Context Model for Ranking Refinement.
PRM: the implementation of the Personalized Re-ranking Model in Personalized Re-ranking for Recommendation
GSF: the implementation of the Groupwise Scoring Function in Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networks.
miDNN: the implementation of the miDNN model in Globally Optimized Mutual Influence Aware Ranking in E-Commerce Search
SetRank: the implementation of the SetRank model in SetRank: Learning a Permutation-Invariant Ranking Model for Information Retrieval.
Seq2Slate: the implementation of sequence-to-sequence model for re-ranking in Seq2Slate: Re-ranking and Slate Optimization with RNNs
EGRerank: the implementation of the Evaluator-Generator Reranking in AliExpress Learning-To-Rank: Maximizing Online Model Performance without Going Online
We process two datasets, Ad and PRM Public, containing user and item features with recommendation lists for the experimentation with personalized re-ranking. The details of processed datasets are summarized in the following table
| Dataset | #item | #list | # user feature | # item feature |
|---|---|---|---|---|
| Ad | 349,404 | 483,049 | 8 | 6 |
| PRM Public | 2,851,766 | 1,295,496 | 3 | 24 |
Depending on the length of the initial ranking, the maximum length of initial lists (re-ranking size n) is set to 10 and 30 for Ad and PRM Public, respectively.
The original Ad dataset records 1 million users and 26 million ad display/click logs, with 8 user profiles (e.g., id, age, and occupation), 6 item features (e.g., id, campaign, and brand). Following previous work, We transform records of each user into ranking lists according to the timestamp of the user browsing the advertisement. Items that have been interacted with within five minutes are sliced into a list and the processed data is avaliable here. The detailed process is here.
The original PRM public dataset contains re-ranking lists from a real-world e-commerce RS. Each record is a recommendation list consisting of 3 user profile features, 5 categorical, and 19 dense item features. Due to the memory limitation, we downsample the dataset and the remained data is avaliable here. The detailed process is here.
Please cite as:
@article{liu2022neural,
title={Neural Re-ranking in Multi-stage Recommender Systems: A Review},
author={Liu, Weiwen and Xi, Yunjia and Qin, Jiarui and Sun, Fei and Chen, Bo and Zhang, Weinan and Zhang, Rui and Tang, Ruiming},
journal={arXiv preprint arXiv:2202.06602},
year={2022}
}