enajx/HebbianMetaLearning

MemoryError

TigerStone93 opened this issue · 3 comments

TigerStone93 commented

Thanks for your great work.

However while I run your code, I encountered a MemoryError.

...
    File ".../evolution_strategy_hebb.py", line 219, in _get_population
        return np.array(population).astype(np.float32)
numpy.core._exceptions.MemoryError: Unable to allocate 697. MiB for an array with shape (200, 91328, 5) and data type float64

At first, es.run doesn't show any problem but after several iterations, train_hebb.py occupy too much memory and the system break down because there is no memory left.

I already tried the solution below but it doesn't solve my error.
https://stackoverflow.com/questions/57507832/unable-to-allocate-array-with-shape-and-data-type

I also tried changing --threads argument from -1 to small integer but the system eventually slow down anyway.

I will keep trying several ways and If I find any solution, I will update this issue.

If you have any solution to solve MemoryError problem, please give me any advise.

Thank you in advance.

enajx commented

I can't replicate the error on my computer. I also tested on a google collab notebook and seems to work fine.
It'd seem that you're just low on memory, I'd recommend trying to run it on a different machine to check if you encounter the same error.

  • 👀2
TigerStone93 commented

Dear @enajx

I used several computers but I failed to complete 300 iterations.
System always encountered MemoryError that I explained above after 80~150 iterations.

Belows are computers that I used to training.

  1. Ubuntu 16.04 with 16GB RAM
  2. Ubuntu 18.04 with 128GB RAM
  3. Windows 10 with 16GB RAM

I failed to find the reason of MemoryError so I edited your code to load pre-trained coefficients and weights before start running and then I run 100 iterations 3 times to complete 300 iterations.

Finally I succeeded in duplicating your result on CarRacing-v0 environments.

Thank you for your reply.

enajx commented

For the records, if running on a system with low memory, it might help to remove typecasting to float32 .astype(np.float32) fromevolution_strategy_hebb.py, eg. https://github.com/enajx/HebbianMetaLearning/blob/master/evolution_strategy_hebb.py#L219