jiecaoyu/pytorch_imagenet

lmdb error: expected a readable buffer object

Closed this issue · 7 comments

ujsyehao commented

Error details:
folder.py, line 125, in __getitem__ datum.ParseFromString(value) TypeError: expected a readable buffer object
I have used fix_key.sh(edit $Original path and $target path) and run fix_key.sh successfully, but it still reported this error!

ujsyehao commented

I check convert_key.py, the code below:
key_str = key[0:8] target_txn.put(key_str, value)
ex: img key: 00146808_n02992529/n02992529_25531.JPEG -> 00146808,

the key used in the LMDB dataset is not suitable for accessing

However, when I run fix_key.sh, it shows the error "expected a readable buffer object", Is this related to fix_key.sh?

jiecaoyu commented

Hi @ujsyehao , could you post the whole error message?

ujsyehao commented

Hi, this is error info:
188

ujsyehao commented

I have some thoughts:

  • Imagenet dataset is too large, I check images in the file, some are broken, but images are made into lmdb successfully by caffe, and I can use tran_lmdb and val_lmdb to train alexnet with caffe successfully.
  • I add datum = caffe.proto.caffe_pb2.Datum(), for key, value in source_cursor: datum.ParseFromString(value) in conver_key.py, then I run convert_key.py and it doesn't appear the error "datum.ParseFromString(value)", so I am confused, Can you give me some advice?
jiecaoyu commented

Hi @ujsyehao , could help explain more about "I add datum = caffe.proto.caffe_pb2.Datum(), for key, value in source_cursor: datum.ParseFromString(value)"?

I am also confused whether you got error when running fix_key.sh?

ujsyehao commented

  1. The code datum.ParseFromString(value) doesn't cause error in convert_key.py, However, the same code in folder.py will cause the error information "TypeError: expected a readable buffer object"
    189

  2. I can run ifix_key.sh successfully, It has no error information, the transformed LMDB is the same size as original LMDB

  • 👍1
ujsyehao commented

Hi, I found the reason.

If the extracted imagenet dataset has some broken images, caffe will make un-broken images into LMDB, but the key is skipped(like key-00000272_n02104365/n02104365_4376.JPEG and key-00000274_n02268853/n02268853_2885.JPEG sikp key-00000273).

In your folder.py, the index is continuous, so if an index is 273, you turn it to 00000273, However, transformed LMDB(use convert_key.sh) including originial LMDB doesn't have the key(00000273), so its value will be none, ParseFromString(value) will cause the error "expected a readable buffer object"

  • 👍1