Why Is APSIM Consuming Excessive Resources on Google Cloud's Linux Environment?
Closed this issue · 4 comments
What happened?
Hi,
I am running APSIM version 7.2.8.1 on a Linux system in Google Cloud. My setup includes optimized cloud computing with 16 cores and 64GB of memory. Despite this, running APSIM with the attached file (and in many other scenarios) takes over 15 minutes.
During this time, I observed that APSIM uses all 16 cores at 100% utilization. This seems highly unreasonable. Could there be an issue with the APSIM files, the weather files, or is there another way to optimize its operation?
What Operating System are you on?
Windows
APSIM will use all available CPU so using all 16 cores is ok. I notice in your simulations that you are outputting daily. This makes APSIM very slow as there are a lot of numbers to write to the .db file. Try removing this and see if that runs quicker. Report has the ability the aggregate (sum, mean etc) over a period (https://builds.apsim.info/api/nextgen/docs/Report.pdf) which you can use instead of writting daily values.
I just ran this myself on my local machine and the 5.1K simulations only took about 3 minutes to run, which is what I would expect of an experiment of this size. I then ran apsim using wsl to test it on linux and while the simulations ran in about 3 minutes again, it then had a queue of 270K operations that still needed to be written to the database, which took a while, 22 minutes in total.
That seems to suggest that there is a bottleneck in the linux build where it is not able to write to the datastore as quickly. Dean's suggestion above will help (not writing all variables every day to the datastore), however there does seem to be a bug here that needs to be addressed on the linux build.
I've been noticing slow writes (ie "waiting for datastore ...") even on runs of ~1500 lines. Normally that amount of data shouldnt even have time to display a message
A quick way to significantly cut the number of datastore operations is to set the summary verbosity to 0 (i.e. only log errors) for these large factorials. This should also help with memory consumption. SimpleLeaf (used by Soybean) is "noisy" and will write a message almost every day the plant is alive, even if the Report is set to a different interval.
Somewhat unrelated, but if you try to select the simulation filter in the summary component for this simulation APSIM will crash with a segmentation fault.