This repository contains annotation for the RareAct dataset as well as an evaluation script for computing the wAP and sAP metrics described in the paper.
- Python 3
- Pandas
- Scikit-learn
You can download the videos zipped into one file here. The video names are the YouTube ids of the videos.
The annotation file is hosted in the github repo and named as rareact.csv
Here is a description of each column:
| Column Name | Type | Example | Description |
|---|---|---|---|
id |
int | 14 |
Unique ID for the annotated video segment. |
video_id |
string | 7frRY7aGwMU |
YouTube ID of the video where the segment originated from (unique per video). |
start |
int | 3 |
Start time in seconds of the action segment. |
end |
int | 5 |
End time in seconds of the action segment. |
class_id |
int [0-148] | 8 |
The class identifier of the actions (verb, noun). Maximum id: 148. |
verb |
string | cut |
Action verb describing the interaction. |
noun |
string | laptop |
Object noun subject of the interaction. |
annotation |
int [0-4] | 1 |
Annotation for the given clip and (verb, noun) class. 1: Positive. 2: Hard negative (only verb is right): 3: Hard negative (only noun is right). 4: Hard negative (Both verb and noun are valid but verb is not applied to noun). 0: Negative. |
We provide an evaluation python script.
To run an evaluation you need first to create a prediction output numpy matrix of shape 7607x149.
where each row represent the samples ordered similarly as in rareact.csv and each column is the prediction score for each of the action class_id.
To compute the mWAP just run:
python compute_score.py predictions.npy To compute the mSAP (n=100) just run:
python compute_score.py predictions.npy 100 where predictions.npy is the prediction output numpy array as described above.
If you find this dataset useful, please cite the following paper:
@article{miech20rareact,
title={RareAct: A video dataset of unusual interactions},
author={Miech, Antoine and Alayrac, Jean-Baptiste and Laptev, Ivan and Sivic, Josef and Zisserman, Andrew},
journal={arxiv:2008.01018},
year={2020},
}