New computations is starving in init stage
Closed this issue · 2 comments
renjiezh commented
Describe the bug
While duchy mills are busy on computing, new computations will stuck in init stage until all old computations are complete. This prevent EDPs from fulfilling requisitions of new computations.
Steps to reproduce
- Create a number of computations so that all mills are occupied .
- Wait until all computations are in computing stage. Create another batch of computations.
Component(s) affected
Duchy, EDP
Version
v0.5.5
Environment
halo-cmm-qa
Additional context
According to experiments, new computations only stuck at aggregator but not at worker 1.
renjiezh commented
A workaround is to force mills to override the Computation order and pick up init Computations by changing the creation time.
Use the spanner sql query against all Duchies:
update Computations
set CreationTime = TIMESTAMP_SUB(CreationTime, INTERVAL 3650 DAY)
where CreationTime > "2024-05-31T17:47:00.000000Z" and ComputationStage = 1
The where clause should be modified accordingly to help select the new computations.
To revert the modification of CreationTime:
update Computations
set CreationTime = TIMESTAMP_ADD(CreationTime, INTERVAL 3650 DAY)
where CreationTime <= "2015-01-01T00:00:00.000000Z" and ComputationStage != 1