Recommendation System Simulations

Outline of simulation

Teacher model (left panel) tells agents the probability to choose any item recommended to them (Bernoulli trial for each choice)
Student model (right panel) estimates the probability agents will choose any item (including items not yet recommended)
Recommendation algorithm (right panel) recommends items to agents. There are 4 strategies
- Greedy: recommend new items to each agent that are most likely to be chosen
- Epsilon-Greedy: 10% probability: recommend items at random, 90% probability: recommend new items to each agent that are most likely to be chosen
- Random: Recommend items at random
- Oracle: Idealized case where student model is exactly the teacher model. This strategy shows upper bounds in recommendation algorithm performance.

Required libraries:

pandas: Tested on 1.1.5
numpy: Tested on 1.19.5
tensorflow: Tested on 2.6.1
scipy: Tested on 1.5.4

To run

Modify required parameters in AlgorithmicBias_Github.py
Typical simulation can be run with: $ python AlgorithmicBias_Github.py

What this code does

Output are all parameters for model simulations and time-varying features of model, including item popularity, which can be used to calculate Gini coefficient, cumulative item popularity, etc.

Parameters

main()

n: number of agents (default = 4000)
m: number of items (default = 200)
k: Teacher model latent dimensions (default = 4)
ID: Random seed to create unique teacher model. Set to an int number: seed number, else if set to None: new seed created at each realization
eps: Epsilon greedy parameter. 0.0: greedy strategy, 0.1: default epsilon-greedy strategy, 1.0: random strategy
seed: Seed to initial conditions (what user-item pairs are initially 1 or 0 for initial student model training). Used in conjunction with ID. If set to None, new seed made each realization
uniform_beta: Whether elements of the Beta matrix for the Teacher model are all the same (uniform_beta = False) or IID values between 0 and 1 (uniform_beta = True)
GT: If true, we use the oracle strategy (idealized upper bound on performance)
realization: The unique number associated with a given realization of the simulation
(Legacy): rand_rec: If true, we the model runs the random strategy. This is exactly the same as eps = 1.0 and so can be set to False

collect_AB_sims

r: number of items recommended before the student model is retrained
fract_available: initial fraction of data used to train the student model (default = 0.001)
embeddings: latent features in student model (we output both 2 and 5 to test robustness). This parameter is "k'" in the paper.

Output

File name lists basic parameters for code, including whether uniform_beta = True (feature of teacher model) Pickle file with the following keys:

'P': Teacher model P
'Q': Teacher model Q
'n': Number of agents
'm': Number of items
'k': Latent features in teacher model
'beta': Beta scalar parameter for teacher model. If uniform_beta = True, this is always 0.0.
'r': Number of items recommended before model is retrained
'fract_available': Initial fraction of data student model is trained on
'epsilon': Parameter for epsilon-greedy strategy (epsilon = 0.0 for greedy strategy, 0.1 for epsilon greedy strategy, and 1.0 for random strategy)
'embeddings': Latent features in student model ("k'" in paper; default = 5)
'realization': Always 1. Modifications to code can allow multiple realizations to be saved in 1 .pkl file
'sim_data': data recorded at each timestep
'final_R_views': Final user-item matrix (R^{data} in the paper)
'final_U': Final P matrix for student model
'final_V': Final Q^T matrix for student model
'gt_U': Teacher model P (legacy)
'gt_V': Teacher model Q^T (legacy)

sim_data

This key contains several features for each timestep (listed in order):

Simulation timestep (values from 1 to m)
Minimum valence error (Brier score)
Number of user-item pairs not recommended
Popularity of each item (how many times they were chosen). Used to find mean item popularity, Gini coefficient (statistics for Figs. 2, 4, 5b-c)
Two statistics:
- (Legacy): MSE error between student and teacher model probabilities
- Brier score between trained student model and all collected user-item pairs (statistic for Fig. 3, 5a)
(Legacy): [the item the student model predicts each user is most likely to choose, the rank the student model gives to the ground truth most preferred item for each agent]
(Legacy): Correlations between student model-predicted popular items or users who pick many items and the teacher model ground truth

Please cite as

Keith Burghardt, Kristina Lerman. Emergent Instabilities in Algorithmic Feedback Loops. arXiv preprint: arXiv:2201.07203 (2022)

Bibtex:

@article{Burghardt2022,   
author={Keith Burghardt and Kristina Lerman},   
title={Emergent Instabilities in Algorithmic Feedback Loops},   
journal={arXiv preprint: arXiv:2201.07203},    
year={2022}}

KeithBurghardt/RecSim