lood339/SCCvSD

Got Error When Generating 90K Sample Data with generate_train_data.py

nixczhou opened this issue · 4 comments

nixczhou commented

Hi I got this error when generating more 90K Sample training data, but it just work with 10K, any solutions? Please help me :(

dict_keys(['__header__', '__version__', '__globals__', 'points', 'line_segment_index', 'grid_points'])
Traceback (most recent call last):
  File "generate_train_data.py", line 27, in <module>
    sio.savemat(save_file, {'pivot_images':pivot_images,
  File "C:\Users\nico\anaconda3\lib\site-packages\scipy\io\matlab\mio.py", line 296, in savemat
    MW.put_variables(mdict)
  File "C:\Users\nico\anaconda3\lib\site-packages\scipy\io\matlab\mio5.py", line 893, in put_variables
    self._matrix_writer.write_top(var, asbytes(name), is_global)
  File "C:\Users\nico\anaconda3\lib\site-packages\scipy\io\matlab\mio5.py", line 634, in write_top
    self.write(arr)
  File "C:\Users\nico\anaconda3\lib\site-packages\scipy\io\matlab\mio5.py", line 673, in write
    self.write_numeric(narr)
  File "C:\Users\nico\anaconda3\lib\site-packages\scipy\io\matlab\mio5.py", line 699, in write_numeric
    self.write_element(arr)
  File "C:\Users\nico\anaconda3\lib\site-packages\scipy\io\matlab\mio5.py", line 540, in write_element
    self.write_regular_element(arr, mdtype, byte_count)
  File "C:\Users\nico\anaconda3\lib\site-packages\scipy\io\matlab\mio5.py", line 554, in write_regular_element
    tag['byte_count'] = byte_count
OverflowError: Python int too large to convert to C long
lood339 commented

Can you provide details of your running? For example, the data and how your command line.

nixczhou commented

Used data:

  1. worldcup_sampled_cameras.mat
  2. worldcup2014.mat

I only change the n=10000 to n=90000 and then run it using python generate_train_dataa.py
It works for n=10000, but breaks when n=90000

And the I got error message like the one I posted earlier

OverflowError: Python int too large to convert to C long

Any Suggestions?

Capture

lood339 commented

Thanks for the details.
It turns out that the size of saved .mat file can not be larger than 2G. https://stackoverflow.com/questions/33565414/why-does-saving-mat-files-with-scipy-result-in-larger-file-size-than-with-matlab

When n = 10000, the file size is about 1.15GB. As a result, the file size will be about 10GB when n = 90000.
One way to walk around that: saving training data to separated files, for example, 'train_data_90k_1.mat, train_data_90k_2.mat, ...' Each file has 10000 training examples.

  • 👍1
nixczhou commented

Thanks for the reply, I found out that numpy .npz format can handle the file size larger than 2G :)

Here is how I modify the code to save .npz file
image

  • 👍2