MemoryError [solved]

[neoctbits]

When run data_loader.py, i got some problem:

[TL] Vocabulary from /home/lab/Desktop/text-to-image-master/vocab.txt :
vocabulary with 5430 words (includes start_word, end_word, unk_word)
start_id: 1
end_id: 2
unk_id: 5429
pad_id: 0

• tokenized 81890 captions
  img_capt: this flower has bright purple spiky petals and greenish sepals below them
  nltk.tokenize.word_tokenize(img_capt): ['this', 'flower', 'has', 'bright', 'purple', 'spiky', 'petals', 'and', 'greenish', 'sepals', 'below', 'them']
  img_capt_ids: [6, 3, 7, 31, 18, 165, 4, 5, 319, 83, 374, 120]
  id_to_word: ['this', 'flower', 'has', 'bright', 'purple', 'spiky', 'petals', 'and', 'greenish', 'sepals', 'below', 'them']
• 8189 images found, start loading and resizing ...
• loading and resizing took 77.8853452205658s
  n_captions: 81890 n_images: 8189 n_captions_per_image: 10
  n_images_train:8000 n_captions_train:80000
  n_images_test:189 n_captions_test:1890
  Traceback (most recent call last):
  File "data_loader.py", line 168, in
  save_all((images_train_256, images_train), '_image_train.pickle')
  File "data_loader.py", line 165, in save_all
  pickle.dump(targets, f)
  MemoryError

[zsdonghao]

it is MemoryError... you may need to solve it yourself

[neoctbits]

Thanks@zsdonghao，i had resolve the problem using dill module instead of pickle. but it also memery costly. The alternative way is merge data_ loader.py into train_txt2img.py to get rid of pickle.

[zsdonghao]

cool~ thank you I will mark it done~

[qpwo]

@boluanace can you post a GitHub gist of your merged data_loader.py and train_txt2img.py please? I am getting the memory error and it is killing me