MemoryError
Closed this issue · 2 comments
Hi! I just started experimenting with your package for the analysis of some big datasets and have encountered problems with the required allocation of memory: for example, the MNIST tutorial is interruped because of the following error
MemoryError: Unable to allocate 876. GiB for an array with shape (60000, 2500, 784) and data type float64
Also creating a small data set ad hoc, such as 2000 np.array of length 200 gets me a similar error. Running on a 8GB RAM and Intel(R) Core i5-6300U CPU .
Thanks in advance!
Hi @SimoneSped, thank you for bringing this up.
Could you share more details? Does this error appear during the train call? If that's the case, most likely setting a smaller batch_size (e.g. 32, 64 ..) or switching to train_algo = online will help.
The example in the tutorial shows batch_size = -1 which will try and run as many samples as possible at once for each available processor. Unless you have a very powerful machine, this is not advised with big datasets.
net.train(train_algo='batch', start_learning_rate=0.01, epochs=-1, batch_size=32)
Are you running the most recent version from the repo? The tutorial refers to a version that hasn't been released yet on PyPI and there have been a lot of changes since the previous one.
Hi @SimoneSped, if there's nothing else I will close this PR.
Feel free to reopen it if anything else comes up.