#IndexError: list index out of range when running "python -m scripts.preprocess_libero --suite libero_spatial"
Opened this issue · 8 comments
Khlann commented
Dear author:
I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!
(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial
Traceback (most recent call last):
File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module>
main()
File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main
task_bert_embs_dict = get_task_bert_embs(root)
File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs
task_embs = get_task_embs(cfg, task_names).cpu().numpy()
File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs
tokens = tz(
File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__
encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one
return self.batch_encode_plus(
File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus
return self._batch_encode_plus(
File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus
for key in tokens_and_encodings[0][0].keys():
IndexError: list index out of range
Robot-K commented
same issue
ahadjawaid commented
You have to run the preprocessing step where they split the dataset into validation and training set. You can do this by doing python3 scripts/split_libero_dataset.py --folder data/atm_libero --train_ratio 0.9.
hanbit-o commented
#7 (comment)
this didn't work to me.
AlvinWen428 commented
Hi, I guess this problem is because the downloaded libero dataset is not correctly placed into the folder ATM/data/. Would you please check the folder structure of ATM/data/? It should be like this:
data
libero_spatial
pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate_demo.hdf5
pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate_demo.hdf5
...
libero_object
libero_goal
libero_10
libero_90
Dear author: I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!
(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial Traceback (most recent call last): File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module> main() File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main task_bert_embs_dict = get_task_bert_embs(root) File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs task_embs = get_task_embs(cfg, task_names).cpu().numpy() File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs tokens = tz( File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__ encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one return self.batch_encode_plus( File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus return self._batch_encode_plus( File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus for key in tokens_and_encodings[0][0].keys(): IndexError: list index out of range
lib36157 commented
Hi, I guess this problem is because the downloaded libero dataset is not correctly placed into the folder
ATM/data/. Would you please check the folder structure ofATM/data/? It should be like this:data
libero_spatial
pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate_demo.hdf5
pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate_demo.hdf5
...libero_object
libero_goal
libero_10
libero_90Dear author: I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!
(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial Traceback (most recent call last): File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module> main() File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main task_bert_embs_dict = get_task_bert_embs(root) File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs task_embs = get_task_embs(cfg, task_names).cpu().numpy() File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs tokens = tz( File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__ encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one return self.batch_encode_plus( File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus return self._batch_encode_plus( File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus for key in tokens_and_encodings[0][0].keys(): IndexError: list index out of range
same error. I don't think it's the problem of data structure. Should be some other reasons.
lib36157 commented
Dear author: I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!
(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial Traceback (most recent call last): File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module> main() File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main task_bert_embs_dict = get_task_bert_embs(root) File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs task_embs = get_task_embs(cfg, task_names).cpu().numpy() File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs tokens = tz( File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__ encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one return self.batch_encode_plus( File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus return self._batch_encode_plus( File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus for key in tokens_and_encodings[0][0].keys(): IndexError: list index out of range
just put all these data files into a folder named libero. The directory should be ATM/data/libero/data files. That works for me.
Robot-K commented
Solved it, there's a .DS_Store file under the location, use
for source_h5 in os.listdir(suite_dir):
source_h5_path = os.path.join(suite_dir, source_h5)
if 'hdf5' not in source_h5: # to pass files like DS_store
continue
RenMing-Huang commented
Dear author: I have encountered this promblem. My os is ubuntu 20.04. How to solve it? Thank you!
(atm) khl@khl:~/khl/ATM/ATM$ python -m scripts.preprocess_libero --suite libero_spatial Traceback (most recent call last): File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/khl/anaconda3/envs/atm/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 285, in <module> main() File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 272, in main task_bert_embs_dict = get_task_bert_embs(root) File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 81, in get_task_bert_embs task_embs = get_task_embs(cfg, task_names).cpu().numpy() File "/home/khl/khl/ATM/ATM/scripts/preprocess_libero.py", line 48, in get_task_embs tokens = tz( File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2577, in __call__ encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs) File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2663, in _call_one return self.batch_encode_plus( File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2854, in batch_encode_plus return self._batch_encode_plus( File "/home/khl/anaconda3/envs/atm/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 458, in _batch_encode_plus for key in tokens_and_encodings[0][0].keys(): IndexError: list index out of rangejust put all these data files into a folder named libero. The directory should be ATM/data/libero/data files. That works for me.
thanks, it works for me!