ActioNet: An Interactive End-to-End Platform for Task-Based Data Collection and Augmentation in 3D Environments
Our dataset can be found in the './dataset/{collection_instance}' folders.
It is created using AI2-THOR (https://ai2thor.allenai.org/).
Each data file has the naming convention of './dataset/{collection_instance}/{task}_{floor_plan}'. In each file, there are two lists:
- The first list shows the task and the floor plan
- The second list shows the actions taken to complete the task
The './dataset/resources/task_descriptions' folder contains the descriptions of the tasks for each collection instance.
The './dataset/resources/user_tasks' folder contains the collection instances and tasks that each user is in charge of.
The statistics below are obtained from running the './get_stats.py' file.
Tasks sorted by complexity (number of steps taken):
- Easy: ['Close the shower curtain', 'Fill up the bathtub with water', 'Fill the sink with water', 'Off kitchen light', 'Open Blinds', 'Use the handphone', 'Close the blinds', 'Turn on shower head', 'Turn off the living room light', 'Find the egg in the room', 'Turn on all the floor lamp', 'Turn off all the bedroom light', 'Break shower glass', 'Hand towel on towelholder', 'Turn off the table lamp or desk lamp', 'Make coffee', 'Break the mirror', 'Use laptop', 'Read a book', 'Put off a candle', 'Crack the handphone screen']
- Moderate: ['Check the timing on the watch', 'Clean the bed', 'Throw away cracked egg', 'Fill up cup with water', 'Clear the sofa', 'Pour wine into a cup', 'Crack the window', 'Clean the mirror', 'Pour away coffee in a cup', 'Hide the egg', 'sink towel in water', 'Watch television', 'Toast a bread', 'Prepare sliced apple', 'Throw away used tissuebox', 'Clear the fridge', 'Boil water with a kettle', 'Water the houseplant', 'Wash dirty cloths', 'Pour away water from pot', 'Boil water with pot']
- Complex: ['Keep the laptop', 'Clear the bed', 'Throw away used toilet roll and soap bottle', 'Microwave the sliced potato', 'Keep sporting equipment', 'Throw away unused apple slice', 'Fry an egg', 'Collect dirty cloths', 'Keep box inside safe', 'Wash Dishes', 'Keep valuable items in safe', 'Keep kitchen utensils inside drawer', 'Light up one candle with another', 'Boil Potato', 'Store food into fridge', 'Make a breakfast', 'Make lettuce soup', 'Wash all the utensils', 'Make tomato soup', 'Pack things into the box']
Number of tasks: 63
Number of instances: 3038
Max number of actions: 878
Min number of actions: 3
Mean number of actions: 70.5
Standard deviation for actions: 78.3
Mean length of mid-level tasks: 2.2
Tasks sorted by complexity (number of steps taken):
- Easy: ['Off kitchen light', 'Find the egg in the room', 'Make coffee', 'Throw away cracked egg', 'Fill up cup with water', 'Pour wine into a cup', 'Pour away coffee in a cup', 'Hide the egg']
- Moderate: ['Toast a bread', 'Prepare sliced apple', 'Clear the fridge', 'Boil water with a kettle', 'Pour away water from pot', 'Boil water with pot', 'Microwave the sliced potato', 'Throw away unused apple slice']
- Complex: ['Fry an egg', 'Wash Dishes', 'Keep kitchen utensils inside drawer', 'Boil Potato', 'Store food into fridge', 'Make a breakfast', 'Make lettuce soup', 'Wash all the utensils', 'Make tomato soup']
Number of tasks: 26
Number of instances: 1314
Number of instances by task: {'Boil water with a kettle': 51, 'Hide the egg': 47, 'Keep kitchen utensils inside drawer': 61, 'Make coffee': 60, 'Make tomato soup': 60, 'Boil water with pot': 54, 'Fry an egg': 55, 'Wash Dishes': 53, 'Prepare sliced apple': 52, 'Toast a bread': 50, 'Pour away coffee in a cup': 53, 'Throw away cracked egg': 50, 'Pour wine into a cup': 53, 'Clear the fridge': 51, 'Fill up cup with water': 51, 'Make lettuce soup': 59, 'Throw away unused apple slice': 51, 'Off kitchen light': 49, 'Microwave the sliced potato': 52, 'Pour away water from pot': 51, 'Boil Potato': 54, 'Store food into fridge': 51, 'Find the egg in the room': 47, 'Make a breakfast': 48, 'Wash all the utensils': 49}
Max number of actions: 673
Min number of actions: 5
Mean number of actions: 93.5
Standard deviation for actions: 85
Mean length of mid-level tasks: 6
Tasks sorted by complexity (number of steps taken):
- Easy: ['Use the handphone', 'Turn off the living room light', 'Turn on all the floor lamp', 'Turn off the table lamp or desk lamp', 'Use laptop', 'Read a book']
- Moderate: ['Crack the handphone screen', 'Check the timing on the watch', 'Clear the sofa', 'Crack the window', 'Watch television', 'Throw away used tissuebox']
- Complex: ['Water the houseplant', 'Keep the laptop', 'Keep box inside safe', 'Keep valuable items in safe', 'Light up one candle with another', 'Pack things into the box']
Number of tasks: 18
Number of instances: 794
Number of instances by task: {'Keep valuable items in safe': 21, 'Turn off the living room light': 49, 'Keep box inside safe': 20, 'Crack the window': 54, 'Light up one candle with another': 12, 'Water the houseplant': 60, 'Keep the laptop': 24, 'Turn off the table lamp or desk lamp': 50, 'Pack things into the box': 56, 'Throw away used tissuebox': 50, 'Clear the sofa': 48, 'Check the timing on the watch': 50, 'Crack the handphone screen': 51, 'Use laptop': 52, 'Read a book': 50, 'Use the handphone': 50, 'Watch television': 53, 'Turn on all the floor lamp': 44}
Max number of actions: 878
Min number of actions: 4
Mean number of actions: 58.7
Standard deviation for actions: 72.9
Mean length of mid-level tasks: 4
Tasks sorted by complexity (number of steps taken):
- Easy: ['Open Blinds', 'Close the blinds', 'Turn off all the bedroom light']
- Moderate: ['Break the mirror', 'Clean the bed', 'Clear the bed']
- Complex: ['Keep sporting equipment', 'Collect dirty cloths', 'Keep box inside safe', 'Keep valuable items in safe']
Number of tasks: 10
Number of instances: 404
Number of instances by task: {'Collect dirty cloths': 23, 'Close the blinds': 46, 'Clean the bed': 51, 'Keep sporting equipment': 50, 'Clear the bed': 49, 'Turn off all the bedroom light': 48, 'Open Blinds': 45, 'Break the mirror': 51, 'Keep valuable items in safe': 31, 'Keep box inside safe': 10}
Max number of actions: 795
Min number of actions: 3
Mean number of actions: 60.7
Standard deviation for actions: 83
Mean length of mid-level tasks: 1.9
Tasks sorted by complexity (number of steps taken):
- Easy: ['Close the shower curtain', 'Fill up the bathtub with water', 'Fill the sink with water', 'Turn on shower head']
- Moderate: ['Break shower glass', 'Hand towel on towelholder', 'Put off a candle', 'Clean the mirror']
- Complex: ['sink towel in water', 'Wash dirty cloths', 'Throw away used toilet roll and soap bottle']
Number of tasks: 11
Number of instances: 526
Number of instances by task: {'Throw away used toilet roll and soap bottle': 49, 'Hand towel on towelholder': 45, 'Put off a candle': 49, 'Close the shower curtain': 44, 'Break shower glass': 50, 'sink towel in water': 52, 'Fill the sink with water': 47, 'Turn on shower head': 47, 'Clean the mirror': 53, 'Fill up the bathtub with water': 45, 'Wash dirty cloths': 45}
Max number of actions: 230
Min number of actions: 3
Mean number of actions: 38.4
Standard deviation for actions: 36.9
Mean length of mid-level tasks: 2.5
Our annotation program can be found in the './annotate' folder.
It can be run by running the './annotate/gui.py' file.
We have made changes to the 'allenai/ai2thor/unity/Assets/Scripts/PhysicsRemoteFPSAgentController.cs' file in AI2-THOR (https://github.com/allenai/ai2thor) to allow for the customisation of initial scene settings (eg. object state) for some tasks, and the edited file is './annotate/PhysicsRemoteFPSAgentController.cs'.
In the './annotate/resources' folder, we have:
- './demo.mp4': tutorial video displayed for users in our annotation program
- './initial-scene-settings.txt': custom initial scene settings for some of our tasks, in the '['index','task','object', ... ,'actionable_property']' format (https://ai2thor.allenai.org/ithor/documentation/objects/actionable-properties/)
- './keyboard-control.png': keyboard control guide displayed for users in our annotation program
- './tasks': folder containing the tasks for each collection instance, and is used by our annotation program to track progress and display related information during task annotation
To augment our dataset with autogenerated action sequences for various tasks, run './augment_data.py'.
The './make_videos/replay_and_save_frames.py' file is used to replay the actions in the dataset as a series of frames, and save the frames.
The './make_videos/create_video_from_frames.py' file is used to create videos from the frames saved.