use convert_caffe_model.py

Question

use convert_caffe_model.py

Closed this issue 7 years ago · 9 comments

Hello,
When i run the convert_caffe_model.py, I have a problem.

[libprotobuf ERROR google/protobuf/io/coded_stream.cc:207] A protocol message was rejected because it was too big (more than 67108864 bytes). To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
Traceback (most recent call last):
File "convert_caffe_model.py", line 121, in
main()
File "convert_caffe_model.py", line 67, in main
p.ParseFromString(open(caffe_model_filename, 'rb').read())
google.protobuf.message.DecodeError: Error parsing message

please, give me some hints, thank you very much!

Answer 1 · 2017-05-12T15:23:00.000Z

Hi @buaa-luzhi. This looks related to protobuf not being able to handle large file. I have v2.5 protobuf-compiler (installed via apt-get install protobuf-compiler in ubuntu), and it worked fine.

In your case, you may need to follow steps to make from src described in: https://gist.github.com/albertomontesg/d8b21a179c1e6cca0480ebdf292c34d2#file-sports1m-c3d-network-to-keras-ipynb

Good luck!

Answer 2 · 2017-05-13T08:16:02.000Z

Thank you for your enthusiastic reply! I have been installed v2.5 protobuf-compiler, but still have the problem. In the coded_stream.h file, kDefaultTotalBytesLimit = 64 << 20. I have been tried to modify to 512<<20, and then recompilation, but do not modification succeeded, I'm confused. and Keras = 1.2.0 Reading model file=./models/conv3d_deepnetA_sport1m_iter_1900000... [libprotobuf ERROR google/protobuf/io/coded_stream.cc:207] A protocol message was rejected because it was too big (more than 67108864 bytes). To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h. Traceback (most recent call last): File "convert_caffe_model.py", line 121, in <module> main() File "convert_caffe_model.py", line 67, in main p.ParseFromString(open(caffe_model_filename, 'rb').read()) google.protobuf.message.DecodeError: Error parsing message [youtube] dM06AMFLsrc: Downloading webpage |DNS-request| www.youtube.com |S-chain|-<>-127.0.0.1:1080-<><>-4.2.2.2:53-<><>-OK |DNS-response|: www.youtube.com does not exist ERROR: Unable to download webpage: <urlopen error [Errno 1] Unknown error> (caused by URLError(gaierror(1, 'Unknown error'),)) Using TensorFlow backend. I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally [Info] image_dim_order (from default ~/.keras/keras.json)=tf [Info] Using backend=tf [Info] Reading model architecture... Traceback (most recent call last): File "test_model.py", line 225, in <module> main() File "test_model.py", line 136, in main model = model_from_json(open(model_json_filename, 'r').read()) IOError: [Errno 2] No such file or directory: './models/sports1M_weights_tf.json' --------------------------------------------------- You should have something close to the following: --------------------------------------------------- Position of maximum probability: 367 Maximum probability: 0.71422 Corresponding label: basketball Top 5 probabilities and labels: basketball: 0.71422 Thank you! Please, help me! 在2017-05-12 23:23:01，鹿智by1603117@buaa.edu.cn写道： Hi @buaa-luzhi. This looks related to protobuf not being able to handle large file. I have v2.5 protobuf-compiler (installed via apt-get install protobuf-compiler in ubuntu), and it worked fine. In your case, you may need to follow steps to make from src described in: https://gist.github.com/albertomontesg/d8b21a179c1e6cca0480ebdf292c34d2#file-sports1m-c3d-network-to-keras-ipynb Good luck! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 3 · 2017-05-13T09:19:08.000Z

In this link "https://gist.github.com/albertomontesg/d8b21a179c1e6cca0480ebdf292c34d2#file-sports1m-c3d-network-to-keras-ipynb", describes how to recompile the protoc compiler is very vague, Is there any more detailed tips?

Thank you very much！you are so warm heart!

Answer 4 · 2017-05-14T02:31:20.000Z

I have been modified kDefaultTotalBytesLimit=512<< 20 successfully, but still have this problems!

int BufferSize() const;
static const int kDefaultTotalBytesLimit = 512 << 20; // 64MB
static const int kDefaultTotalBytesWarningThreshold = 32 << 20; // 32MB
and then recompile the protoc compiler successfully.

[libprotobuf ERROR google/protobuf/io/coded_stream.cc:207] A protocol message was rejected because it was too big (more than 67108864 bytes). To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
Traceback (most recent call last):
File "convert_caffe_model.py", line 121, in
main()
File "convert_caffe_model.py", line 67, in main
p.ParseFromString(open(caffe_model_filename, 'rb').read())
google.protobuf.message.DecodeError: Error parsing message

Still the same mistake.

I do not why???

Thank you for you help! thank you very much!

From Lu

Answer 5 · 2017-05-14T02:40:17.000Z

hi, Thank you for your reply, thanks! I have been modified kDefaultTotalBytesLimit=512<< 20 successfully, but still have this problems! int BufferSize() const; static const int kDefaultTotalBytesLimit = 512 << 20; // 64MB static const int kDefaultTotalBytesWarningThreshold = 32 << 20; // 32MB and then recompile the protoc compiler successfully. [libprotobuf ERROR google/protobuf/io/coded_stream.cc:207] A protocol message was rejected because it was too big (more than 67108864 bytes). To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h. Traceback (most recent call last): File "convert_caffe_model.py", line 121, in main() File "convert_caffe_model.py", line 67, in main p.ParseFromString(open(caffe_model_filename, 'rb').read()) google.protobuf.message.DecodeError: Error parsing message Still the same mistake. I do not why??? Thank you for you help! thank you very much! From Lu 在2017-05-12 23:23:01，鹿智by1603117@buaa.edu.cn写道： Hi @buaa-luzhi. This looks related to protobuf not being able to handle large file. I have v2.5 protobuf-compiler (installed via apt-get install protobuf-compiler in ubuntu), and it worked fine. In your case, you may need to follow steps to make from src described in: https://gist.github.com/albertomontesg/d8b21a179c1e6cca0480ebdf292c34d2#file-sports1m-c3d-network-to-keras-ipynb Good luck! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 6 · 2017-05-15T17:44:59.000Z

@buaa-luzhi hmmm. i don't have a good answer now. i suppose you were running the newly compiled binary? try to use the full path for the new binary.

Answer 7 · 2017-08-02T17:01:13.000Z

@buaa-luzhi were you able to iron out this issue?

Answer 8 · 2017-08-05T01:56:38.000Z

No 在2017-08-03 01:01:15，鹿智by1603117@buaa.edu.cn写道： @buaa-luzhi were you able to iron out this issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Answer 9 · 2017-09-19T20:17:48.000Z

Closing this issue as I can't reproduce it (and it's outside the scope of this repo).