If you want to get the dataset annotation, please fill out the form.
we present the GUIDE (Guideline-Guided) dataset, which contains 3.5K videos of 560 instructional tasks in 8 domains related to our daily life. Specifically, we annotate each instructional task with a guideline, representing a common pattern shared by all task-related videos. On this basis, we annotate systematic specific steps, including their associated guideline steps, specific step descriptions and timestamps.
You can use our dataset to evaluate the model's single-video and multi-video comprehension simultaneously!
This task evaluates the models’ capabilities to understand the procedural temporal knowledge of the instructional video. In this task, models have to generate a set of instructional step captions.
INPUT: <vid><Video></vid>
OUTPUT: {Specific Step 1: ... , Specific Step 2: ... , Specific Step 3: ... , ...}
This task evaluates the models’ capabilities to analyze correlations across videos. In this task, models have to mine the common pattern in task-related videos and summarize a guideline from them.
INPUT: <vid><Task-related Video 1></vid> , <vid><Task-related Video 2></vid>, ...
OUTPUT: {Guideline Step 1: ... , Guideline Step 2: ... , Guideline Step 3: ... , ...}
To explore the impact of guidelines on step captioning, we propose the guideline-guided captioning task. In this task, models have to generate specific step captions under the guide of guideline.
INPUT: {Guideline Step 1: ... , Guideline Step 2: ... , Guideline Step 3: ... , ...} + <vid><Video></vid>
OUTPUT: {Specific Step 1: ... , Specific Step 2: ... , Specific Step 3: ... , ...}
{
"query": "How to Fry mud snail",
"category": "Food",
"guideline_steps_num": 2,
"guideline_steps": {
"1": "Processing Mud Snails.",
"2": "Cooking with Ingredients."},
"videos": {
"3xumfqxgm76yq6q": {
"specific_step": {
"0": {
"title": " Clean the Mud Snail ",
"guideline_step": "Processing Mud Snails.",
"bounds": [5.0, 34.0]},
"1": {
"title": " Making Sauce ",
"guideline_step": "Cooking with Ingredients.",
"bounds": [35.0, 51.0]},
...... ,},
"3xkfuf9htyq7mbc": {
"specific_step": {
"0": {
"title": " Water boiling and mud snail scalding ",
"guideline_step": "Processing Mud Snails.",
"bounds": [5.0, 8.0]},
"1": {
"title": " Making Sauce ",
"guideline_step": "Cooking with Ingredients.",
"bounds": [9.0,15.0]},
...... ,},
...... ,}
},
......
- Video Collection(Manually): Collect high-quality instructional videos.
- Automatic Annotation
- Manual Annotation(Manually): Manually optimized annotations.
@inproceedings{GUIDE,
author = {Jiafeng Liang, Shixin Jiang, Zekun Wang, Haojie Pan, Zerui Chen, Zheng Chu, Ming Liu, Ruiji Fu, Zhongyuan Wang and Bing Qin},
title = {GUIDE: A Guideline-Guided Dataset for Instructional Video Comprehension},
booktitle = {IJCAI},
year = {2024},
}