This repo provides a baseline model for our proposed task: AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistant.
Click to see visualization demo:
(1) PyTorch. See https://pytorch.org/ for instruction. For example,
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
(2) PyTorch Lightning. See https://www.pytorchlightning.ai/ for instruction. For example,
pip install pytorch-lightning
Comming soon.
See encoder.md. (Comming soon after data released)
Select the config file and simply train, e.g.,
CUDA_VISIBLE_DEVICES=0 python train.py --cfg configs/q2a_fps1_vit_b16+bert_b.yaml
The evaluation will be performed after each epoch. You can use Tensorboard, or just terminal outputs to record evaluation results.
If you find our work helps, please cite our paper.
@article{assistq,
author = {Benita Wong and
Joya Chen and
You Wu and
Stan Weixian Lei and
Dongxing Mao and
Difei Gao and
Mike Zheng Shou},
title = {AssistQ: Affordance-centric Question-driven Task Completion for Egocentric
Assistant},
journal = {arXiv:2203.04203},
year = {2022}
}