/Concept4CRS

Primary LanguagePython

CONCEPT – An Evaluation Protocol on Conversation Recommender Systems with System-centric and User-centric Factors

Contact: Chen Huang (Sichuan University)

  • We propose an comprehesive conversational recommender system (CRS) evaluation protocol, called Concept. It considers both system- and user-centric factors and conceptualizes them into three characteristics, which are further divided into six primary abilities.

  • We have released the dataset, created by utilizing Concept, publicly to aid the research community in making advancements in CRS. A total of 6720 conversation data is recorded to collect the interactions between off-the-shelf CRS and simulated users who demonstrate different personas and preferences.

  • Our paper evaluate and analyze the strengths, weaknesses, and potential risks of off-the-shelf CRS models.

  • To clarify, our contribution lies in the evaluation protocol, not the dataset. The dataset is generated dynamically alongside the execution of the protocol.

Paper

Click Me

File Information

  1. The folder dataset/dialog_data contains conversation data of 4 off-the-shelf CRS models (i.e., KBRD, BARCOR, UNICRS, CHATCRS). Each conversation data is in json format. Additionally, our data metadata is based on This Git Repository
  2. The folder code contains the code of Concept (i.e., the user-CRS interaction and the evaluation). The code is also based on This Git Repository

Reference

If you make advantage of the Concept in your research, please cite the following in your manuscript:

@misc{huang2024concept,
      title={Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factors}, 
      author={Chen Huang and Peixin Qin and Yang Deng and Wenqiang Lei and Jiancheng Lv and Tat-Seng Chua},
      year={2024},
      eprint={2404.03304},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}