[2.2.1] `until_front_of_line` reports null timestamps and continues without waiting
Closed this issue · 26 comments
RELEASE CANDIDATE READY
➡️ Potential fix for this issue is ready ⬅️
For anyone willing to validate on non-critical workloads, please use eddiewebb/queue@dev:506 and report any issues on #112
Orb version
2.2.1
What happened
I have this configuration:
- queue/until_front_of_line:
only-on-branch: master
only-on-workflow: orange_build
block-workflow: true
time: '10'
circleci-api-key: CIRCLECI_TOKEN
But when two concurrent orange_build workflows runs the second one reports null timestamp and continues:
This Workflow Timestamp: null
Oldest Workflow Timestamp: null
Front of the line, WooHoo!, Build continuing
Downgrading to 1.8.4 solves the issues. But of course missing the new only-on-workflow parameter.
Expected behavior
Should wait for the previous workflow to finish.
The issue sounds familiar. We have block-workflow: true but not sure if that impacts anything. In the comments to #94 it also says
This Workflow Timestamp: "2023-04-27T12:11:43Z"
Oldest Workflow Timestamp: null
Whereas we get:
This Workflow Timestamp: null
Oldest Workflow Timestamp: null
But might still be same underlying issue. So feel free to close if you think it is the same issue.
BTW. I also tried to remove only-on-workflow to make it wait on all workflows on master. But still didn't wait.
Hi,
Changing to version [2.2.0] seems to work
I believe the issue was introduced here
aba14f1#diff-51176543868ecfedca657c95003c21b9aa413af677aa2a9f06a5eaa88f3f2ed7L180-L186
Circle CI,
I think I don't understand the question sorry
Using Github
I am also running into this issue, so far it seems the null values appear only on pipelines that were trigged via webhook (not a commit or merge).
VCS: GitHub
Also confirming 2.2.0 does work with my webhook triggered builds.
VCS: GitHub
I can confirm that 2.2.0 works as well for us
Ok, need to update to use new /pipelines/ endpoint to get the created_at time of the pipeline, which will be consistent across all workflows in that pipeline, and across different sources of change (Github, Bitbucket, webtriggers, etc)
Confirming that 2.2.0 is working, 2.2.1 is not
I'm seeing similar behavior, curiously enough I only started seeing this behavior sometime after 2023-09-04.
Checking time of workflow: asdf
DEBUG: Making API Call to https://circleci.com/api/v2/workflow/asdf
DEBUG: API Success
Workflow was created at: 2023-09-11T13:34:03Z
Checking time of workflow: fasd
DEBUG: Making API Call to https://circleci.com/api/v2/workflow/fasd
DEBUG: API Success
Workflow was created at: 2023-09-11T13:33:21Z
Checking time of workflow: 123
DEBUG: Making API Call to https://circleci.com/api/v2/workflow/123
DEBUG: API Success
Workflow was created at: 2023-09-11T13:20:48Z
Orb parameter block-workflow is true.
This job will block until no previous workflows have *any* jobs running.
Oldest job: 69447
This Workflow Timestamp: null
Oldest Workflow Timestamp: null
Front of the line, WooHoo!, Build continuing
Confirming that 2.2.0 is working, 2.2.1 is not
same here
Confirming that 2.2.0 is working, 2.2.1 is not
Same here as well. Our builds are triggered from GitHub webhooks.
Can confirm the issue. Running v2.2.1 on GitHub webhooks as well.
@eddiewebb
Hmm... idk if this is it but here:
https://github.com/eddiewebb/circleci-queue/blob/main/src/commands/until_front_of_line.yml#L159
fetch_active_workflows(){
cp /tmp/jobstatus.json /tmp/augmented_jobstatus.json
for workflow in `jq -r ".[] | .workflows.workflow_id //empty" /tmp/augmented_jobstatus.json | uniq`
do
echo "Checking time of workflow: ${workflow}"
workflow_file=/tmp/workflow-${workflow}.json
if [ ! -z $TESTING_MOCK_WORKFLOW_RESPONSES ] && [ -f $TESTING_MOCK_WORKFLOW_RESPONSES/${workflow}.json ]; then
echo "Using test mock workflow response"
cat $TESTING_MOCK_WORKFLOW_RESPONSES/${workflow}.json > ${workflow_file}
else
fetch "${CIRCLECI_BASE_URL}/api/v2/workflow/${workflow}" "${workflow_file}"
fi
created_at=`jq -r '.created_at' ${workflow_file}`
echo "Workflow was created at: ${created_at}"
cat /tmp/augmented_jobstatus.json | jq --arg created_at "${created_at}" --arg workflow "${workflow}" '(.[] | select(.workflows.workflow_id == $workflow) | .workflows) |= . + {created_at:$created_at}' > /tmp/augmented_jobstatus-${workflow}.json
#DEBUG echo "new augmented_jobstatus:"
#DEBUG cat /tmp/augmented_jobstatus-${workflow}.json
mv /tmp/augmented_jobstatus-${workflow}.json /tmp/augmented_jobstatus.json
done
}
When you fetch the status of each workflow, at the end you override the file to /tmp/augmented_jobstatus.json, meaning only last iteration of that for loop will have it's data stored.
So when you later call:
https://github.com/eddiewebb/circleci-queue/blame/main/src/commands/until_front_of_line.yml#L211-L213
load_current_workflow_values(){
my_commit_time=`jq '.[] | select( .build_num == '"${CIRCLE_BUILD_NUM}"').committer_date' /tmp/augmented_jobstatus.json`
}
it will not find the data unless your workflow id happens to be last on the list. So you end up with null value.
It looks to me like the problem is using committer_date instead of workflows.created_at in the latest merge commit to main. See e74ca30.
There are no tags, so it's not clear, but I think this is the latest version 2.2.1.
Version 2.2.0, which other users have reported to be working, I think is the previous merge commit to main: 944d963
@eddiewebb Perhaps it would be good to roll the changes to until_front_of_line.yml back in commit e74ca30 as a fix.
Made a PR for the above mentioned change: #108
Milestone 3.0 uses the best across sources of change which is to delegate to build-processor's assignment of pipeline.id See #111
ATTENTION
If you are having ordering issues on 1.x 2.x releases and are eager to push a fix, please try eddiewebb/queue@dev:499
Comments ---> #112
3.1.0 released and fix