Tutorial: Multi-Stakeholder Recommender Systems (MSRS)

Introduction

Recommender systems are able to produce a list of recommended items tailored to user preferences, while the end user is the only stakeholder in the system. However, there could be multiple stakeholders in several applications or domains, e.g., e-commerce, advertising, educations, dating, job seeking, and so forth. It is useful to produce the recommendations by balancing the needs of different stakeholders. This tutorial covers the introductions to multi-stakeholder recommender systems (MSRS), introduces multiple case studies, discusses the corresponding methods and challenges to develop MSRS. Particularly, a demo based on the MOEA framework will be given in the talk.

Demo

The demo shows case of utility-based multi-stakeholder recommender systems (UBMSRS) by using an educational data as case study.

Setting: To simplify the problem, the demo performs a hold-out evaluation by using 70% as training and 30% as testing. Recommendation performance was evaluated based on F1 and NDCG over the top-N recommendations, while you can change the value of N in the configuration file "setting.conf". Items with rating > 3 are considered as "relevant" items in the top-N recommendations.
MOEA: In terms of the multi-objective optimizers (MOO), we use the library MOEA. We use the default learning parameters in MOEA. To further tune up the models, change the source codes by referring to the Beginner's guide to the MOEA Framework
Baselines: UBRec refers to the utility-based multi-criteria recommendation model which considers student preferences only. Rankp refers to the simple ranking based on the utility of items from the perspective of instructors or professors only. These baseline approaches will produce different objectives as shown at the end in the setting.conf
How to run: Change the setting in setting.conf, and run "java -jar UBMSRS.jar -c setting.conf" by using JRE 8 or above

Setting.conf

data.path: tell the demo where you data sets are. Make sure all necessary data are in the same folder.
expectation.learn: turn on to learn student expectations by using MOO. Otherwise, load student expectations
runbaseline: demo will run baseline approaches first, if it is turned on
expectation.filename: assign the file of student expectations you want to load
topN: assign the value of N for top-N recommendations
maxeval: assign the maximal number of evaluations in the MOO
maxf1, maxndcg, ...: these are the best performance from the baseline approaches. You do not need to change these values unless you found better baseline results

Data Sets

Data files and explanations

ratings_student.csv and ratings_instructor.csv are the raw rating data given by students and the instructor respectively
ratings_student_train.csv and ratings_student_test.csv are the splitted data by using 70% as training and 30% as testing
ratings_student_candidates.csv contains the pair of users and items, while the items are the candidate items to be recommended according to the train/test split. We have used the independent biased matrix factorization to predict the multi-criteria ratings in <App, Data, Ease> in this data.
expectations_student_learned_by_UBRec_NDCG_0.214.csv is the file of learned student expectations by using the baseline UBRec. The optimal NDCG is 0.214. You can also save a new one if you can learn better expectations.

We collected this educational data for multi-stakeholder recommendations, and the information of student identifications have been removed. If you reuse this data for the purpose of research, please cite the following papers.

Yong Zheng, J.R. Toribio. The role of transparency in multi-stakeholder educational recommendations. User Model User-Adap Inter 31, 513–540 (2021). (Note: there is a user study in this paper)
Yong Zheng, Nastaran Ghane, and Milad Sabouri. "Personalized Educational Learning with Multi-Stakeholder Optimizations." Adjunct Proceedings of the ACM conference on User Modelling, Adaptation and Personalization (UMAP). ACM. 2019.
Yong Zheng. "Multi-Stakeholder Personalized Learning with Preference Corrections." Proceedings of the 18th IEEE International Conference on Advanced Learning Technologies (ICALT). IEEE. 2019.

Tutorials

We have tutorials at ACM RecSys 2019 and ACM CIKM 2019.