This is the codes and datasets for the papers: LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts.
- Ubuntu 16.0.4
- Python 3.5
- Pytorch 0.4.1
- Sklearn >= 0.19.1
-
Processed dataset can be directly used for our codes to reproduce the results reported in the paper. It should be downloaded from Google Drive, and put in the folder /data.
-
Raw dataset consists of the videos and the corresponding live comments that directly downloaded from the Bilibili video websites. It can be found at Google Drive or Baidu Pan. After processed with the scripts in the folder /data, it can be transformed into the processed datasets above.
-
Step 1: Download the processed dataset above
-
Step 2: Train a model
python3 codes/transformer.py -mode train -dir CKPT_DIR -
Step 3: Restore the checkpoint and evaluate the model
python3 codes/transformer.py -mode test -restore CKPT_DIR/checkpoint.pt -dir CKPT_DIR
- Step 1: Extract the frames from the videos and the comments from the .ass files.
python3 data/extract.py - Step 2: Convert the extracted images and text into the format required by our model.
python3 data/preprocess.py - Step 3: Construct the candidate set for the evaluation of the model.
python3 data/add_candidate.py
-
More details regarding the model and the dataset can be found in our paper.
-
The code is currently non-deterministic due to various GPU ops, so you are likely to end up with a slightly better or worse evaluation.
Hopefully the codes and the datasets are useful for the future research. If you use the above codes or datasets, please cite:
@inproceedings{livebot,
author = {Shuming Ma and
Lei Cui and
Damai Dai and
Furu Wei and
Xu Sun},
title = {LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts},
booktitle = {{AAAI} 2018},
year = {2018}
}