error in dataloding

Question

error in dataloding

hlyyyyy opened this issue 8 months ago · 1 comments

My code:

import numpy as np
import mediapy
from tqdm import tqdm
import dataclasses

from waymax import config as _config
from waymax import dataloader
from waymax import datatypes
from waymax import visualization

config = _config.DatasetConfig(path ='/ssd2/waymo_open_dataset_motion_v_1_2_0/uncompressed/tf_example/testing/testing_tfexample.tfrecord@150',
    max_num_rg_points=30000,
    data_format=_config.DataFormat.TFRECORD,
    max_num_objects=32)

data_iter = dataloader.simulator_state_generator(config=config)
scenario = next(data_iter)

Error:

CUDA backend failed to initialize: Unable to load CUDA. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
2024-01-18 13:29:21.716915: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.717835: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.718194: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.718219: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.718505: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.719481: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720531: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720556: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720603: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720632: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720665: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720695: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720736: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.720842: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.721255: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.721299: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
Traceback (most recent call last):
  File "/ssd1/LightZero/zoo/waymax/envs/test.py", line 17, in <module>
    scenario = next(data_iter)
  File "/usr/local/lib/python3.10/dist-packages/waymo_waymax-0.1.0-py3.10.egg/waymax/dataloader/dataloader_utils.py", line 223, in get_data_generator
    for data in dataset:
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 809, in __next__
    return self._next_internal()
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 772, in _next_internal
    ret = gen_dataset_ops.iterator_get_next(
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/ops/gen_dataset_ops.py", line 3028, in iterator_get_next
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/ops.py", line 5888, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__IteratorGetNext_output_types_31_device_/job:localhost/replica:0/task:0/device:CPU:0}} Key: state/type.  Can't parse serialized Example.
         [[{{node ParseExample/ParseExampleV2}}]] [Op:IteratorGetNext] name: 
2024-01-18 13:29:21.722554: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.724826: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.724857: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.731598: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.734235: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.735144: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.735471: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.735531: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.736403: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.736785: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.739144: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.739335: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.739355: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740241: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740664: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740718: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740735: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740828: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740851: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740896: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740914: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740931: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.740948: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741291: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741480: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741523: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741566: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741588: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741742: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741769: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.741804: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.742204: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.742333: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.742428: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.742498: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.742512: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.742679: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.743450: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.743486: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.743510: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.743614: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.743883: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.744034: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.744415: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.744489: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.744557: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.744632: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.746093: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.748888: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.751985: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.755314: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.759302: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.763016: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.763520: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.769903: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.770057: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.770379: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.771022: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.774376: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.774463: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.776842: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.
2024-01-18 13:29:21.786968: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at example_parsing_ops.cc:98 : INVALID_ARGUMENT: Key: state/type.  Can't parse serialized Example.

tensorflow version: 2.14.0-cpu
when I change the local file path to google cloud path "gs://xxx", the same error occurs, so it's not the problem of the downloaded file.

Do you have any suggestion about this err?
Thanks a lot.

Answer 1 · 2024-05-10T01:02:47.000Z

maybe your downloaded TFrecord didn't align with the config you use
waymo_open_dataset_motion_v_1_2_0,
I just met this problem yesterday.