CSAILVision/places365

UnicodeDecodeError

xlxlcyRuSReXxlxl opened this issue · 18 comments

xlxlcyRuSReXxlxl commented

Hi all! Running run_placesCNN_basic.py in this repository I run into this following error:
Traceback (most recent call last):
File "", line 4, in
File "/usr/local/lib/python3.5/dist-packages/torch/serialization.py", line 231, in load
return _load(f, map_location, pickle_module)
File "/usr/local/lib/python3.5/dist-packages/torch/serialization.py", line 379, in _load
result = unpickler.load()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 875: ordinal not in range(128)

Any advice? Thanks in advance!

zhoubolei commented

No idea. maybe you could try useGPU = 1. Model is cached using python2.7.

LCorleone commented

i met the same problem! Did you solve it?

LCorleone commented

i use python 3.6, i think the version may be the key!

zhoubolei commented

Indeed, it is the pickle library in python which cannot correctly load the CNN model trained in python2.7 into python3.6. Please let me know if there is some good solution for that.

  • 👍5
LCorleone commented

yep, i just use virtualenv to create a pure environment for python2.7 and all is ok.
thank you for your reply! this is an excellent work!

paffell commented

I solved it with a small change in torch code in serialization.py lines 376-377 by adding the encoding there:
_sys_info = pickle_module.load(f,encoding='latin1')
unpickler = pickle_module.Unpickler(f,encoding='latin1')

I know it is not the cleanest solution, but it works :) To make it cleaner, it should be at least wrapped it by Try - Catch. But I do use Torch for this purpose only

  • 👍20
  • 👎1
  • ❤️5
xlxlcyRuSReXxlxl commented

I solved in the same way as paffel did, and it works pretty well.

  • 👍3
soravux commented

Thanks @paffell for the trick.
For those of you who don't want to modify PyTorch's source code, you can do the following in your code just before loading the model:

from functools import partial
import pickle
pickle.load = partial(pickle.load, encoding="latin1")
pickle.Unpickler = partial(pickle.Unpickler, encoding="latin1")
model = torch.load(model_file, map_location=lambda storage, loc: storage, pickle_module=pickle)

This is for the run_placesCNN_unified.py example. For run_placesCNN_basic.py, replace the model_file in the last line with model_weight.
Note that your pickle namespace will be polluted after this operation. A better idea would be to copy the module first and modify the methods of this copy.

  • 👍67
  • 👎1
  • 😄1
  • 🎉18
  • ❤️15
  • 🚀5
paffell commented

Thanks @soravux for cleaning the solution. I started to use it as well :)

Fangyh09 commented

Thanks @soravux, it works for me too.

Qinxianshen commented

@soravux Thanks so much!!! it's work for me

sakshijain20 commented

UnpicklingError: invalid load key, '\x5c'.

Getting the same error.Can anyone help?

zhangzhengde0225 commented

_pickle.UnpicklingError: invalid load key, '\x04'.

Getting the same error. Tried Soravux's method, still report error.

sanhok99 commented

I solved in the same way as paffel did, and it works pretty well.

will you please mention the line numbers and the file name. Thanks.

LCorleone commented
carandraug commented

The hack suggested in this issue, i.e., the following line:

pickle.Unpickler = partial(pickle.Unpickler, encoding="latin1")

completely breaks toch.load since version 1.11 because in that version, PyTorch subclasses pickle.Unpickler but after this, pickle.Unpickler is a function and not a type which leads to the following error:

[...]
  File "lib/python3.9/site-packages/torch/serialization.py", line 820, in _legacy_load
    class UnpicklerWrapper(pickle_module.Unpickler):  # type: ignore[name-defined]
TypeError: the first argument must be callable

To load models saved in Python2, since PyTorch 1.1.0 (released May 2019), do

torch.load(...., encoding='latin1')
LCorleone commented