Really high resource usage on low powered harvesters (e.g. Raspberry) and long time needed to recognize plots
Closed this issue · 49 comments
This is a clone of
#137
since the old issue was closed without fixing the actual problem.
Since a few versions (I believe it all started more or less around when Chia version 1.2.10 codebase was introduced and staking was introduced into SIT) I have noticed extremely high resource usage (CPU time) when harvesting on low power devices (e.g. RaspberryPi). Please note --- this has not been the case a few versions back.
Also extremely slow adding new plots and starting up (from starting harvester up to fully caching up all plots).
In Chia it was nearly fully fixed with the codebase 1.2.11.
Steps to replicate :
on Raspberry Pi set up SIT, add a lot of plots at once. Compared to old versions it takes forever for them to be cached.
Also running SIT harvester on Raspberry uses extreme amount of resources (twice-triple the amount it used before in "old" versions).
Please note --- this is not a configuration issue of the Raspberry as I have three of them running and also testing other forks which introduced a similar behavior when incorporating chia 1.2.9-10 codebase commits. Flax has more or less fixed it updating from 0.1.4 to 0.1.5 (chia 1.2.11).
Can you please take a look at the commits introduced with chia 1.2.11 and incorporate them into the SIT codebase?
Thank you!
//It looks like a lot of people have a performance problem with the harvester, plot recognitnion, processing of plots etc. Especially if the amount of plots is in higher numbers the performance drop is drastical. Starts around 1000+ plots, maybe earlier.
Plot recognition after a new start takes forever, resource usage (CPU) compared to other forks is tripled or more etc.
I am a micro farm on a amd 3200g (approx 4500 plots) and silicoin uses more resources than anything else on the pc
This is a clone of #137 since the old issue was closed without fixing the actual problem.
Since a few versions (I believe it all started more or less around when Chia version 1.2.10 codebase was introduced and staking was introduced into SIT) I have noticed extremely high resource usage (CPU time) when harvesting on low power devices (e.g. RaspberryPi). Please note --- this has not been the case a few versions back. Also extremely slow adding new plots and starting up (from starting harvester up to fully caching up all plots). In Chia it was nearly fully fixed with the codebase 1.2.11.
Steps to replicate : on Raspberry Pi set up SIT, add a lot of plots at once. Compared to old versions it takes forever for them to be cached. Also running SIT harvester on Raspberry uses extreme amount of resources (twice-triple the amount it used before in "old" versions).
Please note --- this is not a configuration issue of the Raspberry as I have three of them running and also testing other forks which introduced a similar behavior when incorporating chia 1.2.9-10 codebase commits. Flax has more or less fixed it updating from 0.1.4 to 0.1.5 (chia 1.2.11).
Can you please take a look at the commits introduced with chia 1.2.11 and incorporate them into the SIT codebase? Thank you!
//It looks like a lot of people have a performance problem with the harvester, plot recognitnion, processing of plots etc. Especially if the amount of plots is in higher numbers the performance drop is drastical. Starts around 1000+ plots, maybe earlier. Plot recognition after a new start takes forever, resource usage (CPU) compared to other forks is tripled or more etc.
The dev branch code has merged Chia 1.2.11 version. You can try to switch to the dev branch and install that version.
Here some first results --
---installed from the dev branch just fine
---initial plot recognition still takes forever vs other forks plus the initial cpu usage is also extremely high -- 100% 1 core
---after it reads all the plots and settles (ca 10-15 minutes) it might behave a little better vs. 1.1.3 (better caching ? ) --- spiking to 100% 1 core usage seem to occur not so frequent as in 1.1.3.... but -- preliminary result -- further observations needed
---still one of the top processes in regard of cpu consumption, which is a pity...
Seems there was no change in the cpu usage or speed of harvester with the dev branch...
Plot recognition still incredibly slow. 100% cpu 1 core usage when doing that.
Plot harvesting still uses triple or more cpu resources than other forks.
Have still a feeling that it goes a little bit less "berserk" and uses 100% a little bit less but can be a placebo.
Nah... ok it was a placebo. CPU Usage and spikes are still roughly the same...
Since only SIT has this problem (and some forks based on chia 1.2.09-10 code) and as you state - the code was merged with chia 1.2.11 (which fixed most of the issues) I assume the problem might be in the introduced staking algorithm that would require optimization ?
But hey.... at least I can "enjoy" now the newest branch on a python 3.9.x system... so that's a positive :)
The following is observed on a beefy system with a number of forks running. These issues are not observed on any of the other forks. OS is Windows Server 2019, fully updated. Dual CPU E5-26xx series. System is running a full SIT node, farmer, and harvester.
-
Seeing the plot recognition perf problem on my farm too, with version 1.1.5. The more plots you have, the worse it is.
-
When switching to the staking tab in the GUI, the action seems to cause a blocking event or to otherwise affect the harvester or farmer, because I consistently see extremely long lookup times after switching to the tab (as reported by ForkFarmer). We're talking 60+ seconds lookup time. The log generation seems to be affected by the event, as there is a significant delay, followed by a dump of a few lines in the log.
An error message of "harvester did not respond" also shows up, though I have not checked whether this happens every time.
looks like a python global lock caused by the staking code.
The following is observed on a beefy system with a number of forks running. These issues are not observed on any of the other forks. OS is Windows Server 2019, fully updated. Dual CPU E5-26xx series. System is running a full SIT node, farmer, and harvester.
- Seeing the plot recognition perf problem on my farm too, with version 1.1.5. The more plots you have, the worse it is.
- When switching to the staking tab in the GUI, the action seems to cause a blocking event or to otherwise affect the harvester or farmer, because I consistently see extremely long lookup times after switching to the tab (as reported by ForkFarmer). We're talking 60+ seconds lookup time. The log generation seems to be affected by the event, as there is a significant delay, followed by a dump of a few lines in the log.
An error message of "harvester did not respond" also shows up, though I have not checked whether this happens every time.
It will look up your plot farmer key in your plot files, so it will take some time to get the result.
Still --- this behavior is observed ONLY in SIT. Only after staking was introduced. It eats up a whole CPU core. Behavior worsens the more plots you load. Impacts everything --- plot loading, harvesting, cpu resources...
Just to make sure I have made it clear enough..... Even though changelog states otherwise --- The issue is still not solved in SIT 1.2.0.
Bump
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
Bump. Still not solved.
Bump - not solved.
BUMP
It's the same for me. Harvester eats all 1 CPU core (it has 2 CPUs and 32 threads at its disposal). At 50TB it still works, but more often it takes 100% of the core and does not work at all with the entire 800TB farm. I do not recommend anyone to seriously get involved in this project. Waste of time. Anyway, when I look at the price, it's not just me.
The other forks and Chia work very well on the same hardware.
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
BUMP -- NOT solved.
Bump...
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
bump - not solved
bump not solved
bump - not solved
aaaaand it is still not solved..... I think silicoin will just die as a crypto before anything meaningful happens here....
.
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
.
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
.
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
.
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
.
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
.
is the development of this dead?
do yourself a favour and move on.
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
.
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
.
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
.
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
.
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
This issue was automatically closed because it has been flagged as stale and subsequently passed 7 days with no further activity.