This is the dataset that accompanies the paper Predicting Physical World Destinations for Commands Given to Self-Driving Cars accepted at AAAI 2022.
Talk2Car-Destination is an extension to Talk2Car which is built on nuScenes.
Note: The Talk2Car-Trajectory dataset has also been released here. This dataset also contains the Talk2Car-Destination dataset.
Each json from the dataset is a dictionary where the key is the command token and the value is a dictionary of the following format.
{
"image": "img name",
"top-down": "top down image name"
"command": "given command"
"destinations": [[x,y]], #is a list of (x, y) pairs where each pair is a destination in the top-down image
"egobbox_top": [ 4 x 2 list], # contains the corners of the ego vehicle bounding box in the top-down image.
"all_detections_top": [64 x 4 x 2 list], # contains the corners of all detected objects in the top-down image.
"detected_object_classes": [64 list], # contains the class of each detected object.
"all_detections_front": [64 x 4 x 2 list], # contains the corners of all detected objects in the frontal image.
"predicted_referred_obj_index": [64 list], # contains the index of the predicted referred object.
"detection_scores": [64 list], # contains the confidence score of each detected object.
"gt_referred_obj_top": [4 x 2 list], # contains the corners of the ground truth referred object in the top-down image.
}
- Download top-down images here and put the images in the data folder.
- Download the frontal images here and put the images in the data folder.
- Download the frame data here and put the frame_data folder in the data folder.
- Download the Talk2Car-Destination dataset here and put all files in the data folder. We also include pre-extracted commmand embeddings with a Sentence-BERT model in the .h5 files in this zip.
- Run
visualize.py
to visualize a sample of the dataset
Drag the Talk2Car-Destination dataset into the data/commands
folder of Talk2Car.
Next, when calling the get_talk2car_class
, set load_talk2car_destination
to True
.
Talk2Car-Destination will now be loaded.
You can find the baselines in the baselines
folder.
The used object detectors and their weights you can find in the object_detectors
folder.
If you use this dataset, please consider using the following citation:
@inproceedings{grujicic2021predicting,
title={Predicting Physical World Destinations for Commands Given to Self-Driving Cars},
author={Grujicic, Dusan and Deruyttere, Thierry and Moens, Marie-Francine and Blaschko, Matthew},
booktitle={Thirty-Sixth AAAI Conference on Artificial Intelligence},
year={2021},
organization={AAAI Press}
}