Test error on private_test_e2e dataset
Closed this issue · 3 comments
Hello, I would like to ask what I have no problem testing on the mini dataset, but it is wrong to test on the private_test_e2e dataset?
(navsim) tongyy@server:~/TYY/navsim/scripts$ ./evaluation/run_transfuer.sh
2024-05-23 15:49:00,634 INFO {/mnt/datadisk/tongyy/TYY/navsim/navsim/planning/script/builders/worker_pool_builder.py:20} Building WorkerPool...
2024-05-23 15:49:00,770 INFO {/mnt/datadisk/tongyy/TYY/navsim/navsim/planning/utils/multithreading/worker_ray_no_torch.py:51} Not using GPU in ray
2024-05-23 15:49:00,770 INFO {/mnt/datadisk/tongyy/TYY/navsim/navsim/planning/utils/multithreading/worker_ray_no_torch.py:77} Starting ray local!
2024-05-23 15:49:02,660 INFO worker.py:1749 -- Started a local Ray instance.
2024-05-23 15:49:03,740 INFO {/mnt/datadisk/tongyy/.conda/envs/navsim/lib/python3.9/site-packages/nuplan/planning/utils/multithreading/worker_pool.py:101} Worker: RayDistributedNoTorch
2024-05-23 15:49:03,740 INFO {/mnt/datadisk/tongyy/.conda/envs/navsim/lib/python3.9/site-packages/nuplan/planning/utils/multithreading/worker_pool.py:102} Number of nodes: 1
Number of CPUs per node: 136
Number of GPUs per node: 0
Number of threads across all nodes: 136
2024-05-23 15:49:03,740 INFO {/mnt/datadisk/tongyy/TYY/navsim/navsim/planning/script/builders/worker_pool_builder.py:31} Building WorkerPool...DONE!
Loading logs: 0%| | 0/1 [00:00<?, ?it/s]
Error executing job with overrides: ['agent=transfuser_agent', 'agent.checkpoint_path=/mnt/datadisk/tongyy/TYY/navsim/exp/training_transfuser_agent/2024.05.16.20.42.58/lightning_logs/version_0/checkpoints/epoch19.ckpt', 'experiment_name=transfuser_agent_eval', 'split=private_test_e2e']
Traceback (most recent call last):
File "/mnt/datadisk/tongyy/TYY/navsim/navsim/planning/script/run_pdm_score.py", line 46, in main
scene_loader = SceneLoader(
File "/mnt/datadisk/tongyy/TYY/navsim/navsim/common/dataloader.py", line 83, in init
self.scene_frames_dicts = filter_scenes(data_path, scene_filter)
File "/mnt/datadisk/tongyy/TYY/navsim/navsim/common/dataloader.py", line 50, in filter_scenes
and len(frame_list[scene_filter.num_history_frames - 1]["roadblock_ids"]) == 0
TypeError: object of type 'NoneType' has no len()
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Hi, this failure is expected because you're trying to run the PDM Score on the private test set. This set intentionally has held-out information (i.e., for the route and bounding box annotation) to enable a fair competition where participants cannot overfit or cheat for the leaderboard results.
You can still run your model on private_test_e2e and upload a submission file to huggingface to participate in the challenge. Have a look at docs/submission.md for further information.
Best,
Daniel
Ok, thanks. But I encountered this error when creating the submission pickle file. How can I solve it?
(navsim) tongyy@server:~/TYY/navsim$ ./scripts/submission/run_cv_create_submission_pickle.sh
[2024-05-23 18:49:43,569][timm.models._builder][INFO] - Loading pretrained weights from Hugging Face hub (timm/resnet34.a1_in1k)
[2024-05-23 18:49:53,587][timm.models._hub][INFO] - [timm/resnet34.a1_in1k] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
[2024-05-23 18:49:53,917][timm.models._builder][INFO] - Missing keys (fc.weight, fc.bias) discovered while loading pretrained weights. This is expected if model is being adapted.
[2024-05-23 18:49:54,442][main][INFO] - Building Agent Input Loader
Loading logs: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2.09it/s]
Error executing job with overrides: ['agent=transfuser_agent', 'split=private_test_e2e', 'experiment_name=submission_cv_agent_20240523', 'team_name=YuyuTong', 'authors=YuyuTong', 'email=yuyutong9806@163.com', 'institution=plusgo', 'country=suzhou']
Traceback (most recent call last):
File "/mnt/datadisk/tongyy/.conda/envs/navsim/lib/python3.9/site-packages/torch/serialization.py", line 354, in _check_seekable
f.seek(f.tell())
AttributeError: 'NoneType' object has no attribute 'seek'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/datadisk/tongyy/TYY/navsim/navsim/planning/script/run_create_submission_pickle.py", line 32, in main
output = run_test_evaluation(
File "/mnt/datadisk/tongyy/TYY/navsim/navsim/planning/script/run_create_submission_pickle.py", line 81, in run_test_evaluation
agent.initialize()
File "/mnt/datadisk/tongyy/TYY/navsim/navsim/agents/transfuser/transfuser_agent.py", line 51, in initialize
state_dict: Dict[str, Any] = torch.load(self._checkpoint_path)["state_dict"]
File "/mnt/datadisk/tongyy/.conda/envs/navsim/lib/python3.9/site-packages/torch/serialization.py", line 791, in load
with _open_file_like(f, 'rb') as opened_file:
File "/mnt/datadisk/tongyy/.conda/envs/navsim/lib/python3.9/site-packages/torch/serialization.py", line 276, in _open_file_like
return _open_buffer_reader(name_or_buffer)
File "/mnt/datadisk/tongyy/.conda/envs/navsim/lib/python3.9/site-packages/torch/serialization.py", line 261, in init
_check_seekable(buffer)
File "/mnt/datadisk/tongyy/.conda/envs/navsim/lib/python3.9/site-packages/torch/serialization.py", line 357, in _check_seekable
raise_err_msg(["seek", "tell"], e)
File "/mnt/datadisk/tongyy/.conda/envs/navsim/lib/python3.9/site-packages/torch/serialization.py", line 350, in raise_err_msg
raise type(e)(msg)
AttributeError: 'NoneType' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Seems like your checkpoint path isn't set. Maybe try to add agent.checkpoint_path="path/to/checkpoint" \ to the script.