[Potential Bug]: Idle mules cause the game to hang intermittently?
probablynotpotato opened this issue · 15 comments
Describe the Bug
The game hangs intermittently (<1fps) for a couple seconds, then goes back to normal.
Error clumps like this appear suspiciously at the time the game hangs:
[General] 161169.10 ====================================== [=ERROR=] 161169.10 AIDirector::Move(): Possibly infinite wake request loop: mule.lib.evaluate_tradeoffers on entity 0x1bd13, wake reason mask=0001 (Most likely cause: Script starts and returns immediately) [General] 161169.10 ====================================== [General] 161170.12 ====================================== [=ERROR=] 161170.12 AIDirector::Move(): Possibly infinite wake request loop: mule.lib.evaluate_tradeoffers on entity 0x5d27d, wake reason mask=0001 (Most likely cause: Script starts and returns immediately) [General] 161170.12 ====================================== [General] 161170.12 ====================================== [=ERROR=] 161170.12 AIDirector::Move(): Possibly infinite wake request loop: mule.lib.evaluate_tradeoffers on entity 0x4ce24, wake reason mask=0001 (Most likely cause: Script starts and returns immediately) [General] 161170.12 ====================================== [General] 161170.12 ====================================== [=ERROR=] 161170.12 AIDirector::Move(): Possibly infinite wake request loop: mule.lib.evaluate_tradeoffers on entity 0x1afdf, wake reason mask=0001 (Most likely cause: Script starts and returns immediately) [General] 161170.12 ====================================== [General] 161171.12 ====================================== [=ERROR=] 161171.12 AIDirector::Move(): Possibly infinite wake request loop: mule.lib.evaluate_tradeoffers on entity 0x4f74e, wake reason mask=0001 (Most likely cause: Script starts and returns immediately) [General] 161171.12 ====================================== [General] 161172.12 ====================================== [=ERROR=] 161172.12 AIDirector::Move(): Possibly infinite wake request loop: mule.lib.evaluate_tradeoffers on entity 0x4fdfa, wake reason mask=0001 (Most likely cause: Script starts and returns immediately) [General] 161172.12 ====================================== [General] 161172.98 ====================================== [=ERROR=] 161172.98 AIDirector::Move(): Possibly infinite wake request loop: mule.lib.evaluate_tradeoffers on entity 0x4f74e, wake reason mask=0001 (Most likely cause: Script starts and returns immediately) [General] 161172.98 ====================================== [General] 161172.98 ====================================== [=ERROR=] 161172.98 AIDirector::Move(): Possibly infinite wake request loop: mule.lib.evaluate_tradeoffers on entity 0x4f66f, wake reason mask=0001 (Most likely cause: Script starts and returns immediately) [General] 161172.98 ======================================
To Reproduce
Steps to reproduce the behaviour:
- Have some mules idling, probably? I have anywhere between 0-30 usually.
- Wait
Expected Behaviour
The game should not hang intermittently at all :)
Version Info (please complete the following information):
- X4 Foundations : 3.2
- Mules Extended Mod : Whatever Steam had on June 18 morning UTC (it doesn't say apparently?)
- I downloaded the mod from: Steam
Hmm, can you post your mule settings?
I would assume the error is caused by mules comparing a billion trade offers because they can't find anything. We are thinking about how to mitigate this, but i think the cruelpit lies with MaxJumps 30 or so.
In general the errormessages are harmless, but i hope to remove them in the next patch.
Do you have debug messages turned on by default (they cause a lot of lag)?
If you have debug files enabled, feel free to post 2-3 supply mule logs.
Do you use any other mods?
A lot of them used to idle when build orders were all fulfilled. However, I recently completed some habitats and there's sudden demand for food, they're all working again and I'm not experiencing the slowdowns anymore.
I don't have debug on. I use a bunch of other mods, the trading-related ones are tatertrader and randomminer and the mules are in a civilian fleet.
I also have two separate supply mule fleets with one specifically forbidden to supply build storage, or they all get stuck in a massive build storage dock queue on a new station.
Hmm, how many stations do you operate?
From what i see, it tries to a massiv amount of compare all trade offers in a single frame.
For an immediate "fix", try lowering the max jumps to < 6 and/or setting serve source only.
That way you reduce the number of trade offer comparisions.
Hopefully, we can mitigate the frame drops in the next patch.
Im not sure when it will be, since im pretty busy with work.
There are 7 stations in that sector which the mules are supposed to be serving. All intermediate ware trading is disabled even for player traders as each station is separately self-sufficient, so in practice only end products ever get moved between those stations and they don't need external input either with the exception of the one shipfab.
If I set them to serve source only, then they wouldn't serve all the stations, and the entire point of having them in a fleet is so they don't have to be balanced constantly between stations.
I had half the mules already limited to 6 jumps, I will limit the rest of them as well.
So, loading the save files proided i have no lag at all.
I do get visible lag once i enable debug messages (as expected). I will try to optimze that.
I will check out the other save by z1p. But im not confident i can fix something i cant see.
Perhaps if I tried making it worse? I'll run some quick tests with removing all mules and then stacking a lot of idle ones to see if it makes a difference.
Could this be another mysterious case of one persons system having an issue with specific loads that our CPU's don't really show any pain with?
Starting to sound like that other issue we had with frame rate and we never could produce it on our rigs as we run fairly powerful cpu's.
I think cause of action for me would be to emulate this issue in a fresh save of my own if I don't get lag then I can probably figure out the script profiler and profile his save vs mine and see how long the script takes doing and what it is doing. Then try messing around with various tick's or max delays to try to mitigate it as such when they have nothing to do just don't check as frequently
I am thinking we could do maybe 2 loops
idle loop and one for an active loop, sort of like a power save feature so when they idle for too long with no jobs just don't bother checking frequently at all and then on the next time it does get something to do switch it over to the active loop which is the is basically a copy-paste of the same loop with a couple of small tweaks to have it bounce between them based on the last time it found anything to go fetch.
This would of course mean that once they fall back to the idle loop it they won't scan as frequently and might get a longer delay on finding a trade but once they do find that trade it would kick them into a more freq scan rate.
Would it be possible to allow idle supply mules to work as travel mules? It'd give them something productive to do, which would reduce idle checks and spread the offer checking load by design.
Possible, probably, but I would put that pretty far down our radar. Not everyone would want that so it's a new param to toggle it, and civ fleets would need a patch. Then we'd have to decide how to convert your params of your supply mule into the params for the travel mule. That's way more work than you think.
I think the amount of return on investment for the amount of time required is not very high.
You I know kind of like this travel + supply mule idea.
that is one less mule and it does both things on its own, I guess the real struggle with it would be how to manage the params in a way that is not super complicated looking.
or maybe this could be a new mule, not to replace distribution mule
and be a combo mule, that can supply, distribute and trade... one mule to do it all in a less overall control kinda way but its easy to use.
Still we have this issue to look into about the idle.
what do you think about this duel loop sort of concept where idle mules go into this power save loop where they check infrequently then only spring to life if they have anything to do then after some time as passed of them with nothing to do they drastically cut down on the tick rate.
Just added a small temporary fix for this. Im not super happy with it, but it does prevent most lags.
I think in priciple the idle problem is solved by this.
Unfortunately it seems that the fix only worked for a small number of idling mules; a major construction project completed recently with a large number of mules going idle, my game started to periodically hang again.
I see..
Did you try the Dev branch?
I never published the temp fix public to steam or nexus. You'd have manually install it from https://github.com/Misunderstood-Wookiee/Mules-and-Warehouses-Extended/tree/Dev