Creating Anaconda Environment taking too much memory.
Closed this issue · 8 comments
I am trying to generate py5 from the py5generator and am following the build process from the documentation, but when I run conda env create -f environment.yml, its taking too much memory and even after running for about 30 minutes nothing seems to happen.
Specifications of my system:
OS: Ubuntu 22.04.2 LTS
Python 3.9.13
Java:
openjdk 11.0.18 2023-01-17
OpenJDK Runtime Environment (build 11.0.18+10-post-Ubuntu-0ubuntu122.04)
OpenJDK 64-Bit Server VM (build 11.0.18+10-post-Ubuntu-0ubuntu122.04, mixed mode, sharing)
Ram: 8 Gb
Processor: Intel® Core™ i5-9300H CPU @ 2.40GHz × 8
Hello, @AsadNizami !
You will need to install Java 17 to run the py5 build, but that happens after creating the Anaconda environment, so it can't be the problem here. Creating the environment shouldn't take 30 minutes and should run fine on your machine. Can you tell me if you've used Anaconda on this machine before?
Can you try creating a test environment with this command?
conda create --name testenv python=3.9 numpyIf this is not a recent Anaconda install, can you make sure it is up to date with this command?
conda update -n base -c defaults condaconda create --name testenv python=3.9 numpy is working fine and Anaconda is the latest version
But this happens while building py5. Is there some memory leakage or something?
But this happens while building py5. Is there some memory leakage or something?
When Anaconda says it is "Solving environment" it is trying to find the best permutation of specific package versions to install on your machine. That process does take up a lot of memory, but it is an Anaconda thing, not py5. To debug this, I would try commenting out a lot of packages in the py5 environment file, create the environment, then uncomment a few packages, create the environment again, uncomment some more, etc, until you find what package is pushing this over the edge.
Ok, thanks. I'll try to find the issue.
I just tried this on a machine that has 4GB of RAM and it ran out of memory solving the environment. Thank you for pointing out this problem. This machine had a working py5 build process a few months ago.
If you like, there are a couple of things you can do to help investigate this. The last time I updated the environment.yml file I added hatch to the build process, so removing that might fix this. I would also expect the previous version of the environment.yml file to work OK. It is also possible I messed up with the version numbers (ie, the 2.0.* in autopep8=2.0.*) and have an unsolvable environment file.
I tried commenting out packages and running builds. It took from a few minutes (for the top 5 dependencies) to 40-50 minutes for the complete build. I wasn't expecting it to take this long, so I created this issue. It's running fine now.
40-50 minutes for the complete build
Hmmm, I wouldn't expect it to take that long either. And now it doesn't work at all on my old laptop with 4GB of RAM. I am going to take a closer look at this to see what I can do to improve this. Thanks again for opening this issue.
This is now fixed. I updated the package versions in the environment file and removed some unneeded packages. I was able to create the environment on my old laptop with 4GB of RAM. It didn't take that much time.
In addition, I found this:
By setting the channel priority to strict, I could speed up the process quite a bit.
conda config --set channel_priority strictAnother potential solution that I didn't try is mamba.