environmental-forecasting/model-ensembler

Super quick scheduling results in check loop

Closed this issue · 2 comments

[20-09-22 10:26:54    :DEBUG   ] - SLURM find result name: fctest_south-0
[20-09-22 10:26:56    :DEBUG   ] - Executing command sacct -XnP -j 5235255 -o jobname,state,start,end, cwd unset
[20-09-22 10:26:57    :DEBUG   ] - Command successful
[20-09-22 10:26:57    :DEBUG   ] - SLURM find result name: fctest_south-1
[20-09-22 10:26:57    :DEBUG   ] - Executing command sacct -XnP -j 5235253 -o jobname,state,start,end, cwd unset
[20-09-22 10:26:57    :DEBUG   ] - Command successful
[20-09-22 10:26:57    :DEBUG   ] - SLURM find result name: fctest_south-2
[20-09-22 10:27:14    :DEBUG   ] - Executing command sacct -XnP -j 5235254 -o jobname,state,start,end, cwd unset
[20-09-22 10:27:14    :DEBUG   ] - Command successful
[20-09-22 10:27:14    :DEBUG   ] - SLURM find result name: fctest_south-0
^C
(icenet-blue) [jambyr@bslws03 blue]$ sacct -XnP -j 5235254 -o jobname,state,start,end
fctest_south-0|COMPLETED|2022-09-20T10:12:38|2022-09-20T10:12:56

This is because the batcher logic won't consider FINISH_STATES as indicative that the job was run through. batcher needs changing to move forward if FINISH_STATE has been reached prior to a check returning START_STATES.

I thought this had already been addressed but apparently not!

Fixed by alteration