[1.6.2] Fails with exit code 22
ldiqual opened this issue · 7 comments
ldiqual commented
Orb version
1.6.2
What happened
Right after upgrading to 1.6.2, queue step failed with the following output (staging is the branch in question):
staging queueable
Exited with code exit status 22
CircleCI received exit code 22
Config:
- queue/until_front_of_line:
time: '30'
circleci-api-key: CIRCLE_TOKEN
Expected behavior
Queue step should not fail
Full logs
#!/bin/bash -eo pipefail
load_variables(){
# just confirm our required variables are present
: ${CIRCLE_BUILD_NUM:?"Required Env Variable not found!"}
: ${CIRCLE_PROJECT_USERNAME:?"Required Env Variable not found!"}
: ${CIRCLE_PROJECT_REPONAME:?"Required Env Variable not found!"}
: ${CIRCLE_REPOSITORY_URL:?"Required Env Variable not found!"}
: ${CIRCLE_JOB:?"Required Env Variable not found!"}
# Only needed for private projects
if [ -z "${CIRCLE_TOKEN}" ]; then
echo "CIRCLE_TOKEN not set. Private projects will be inaccessible."
else
fetch "https://circleci.com/api/v2/me" "/tmp/me.cci"
me=$(jq -e '.id' /tmp/me.cci)
echo "Using API key for user: ${me}"
fi
VCS_TYPE="github"
}
fetch(){
url=$1
target=$2
http_response=$(curl -f -s -X GET -H "Circle-Token:${CIRCLECI_API_KEY}" -o "${target}" -w "%{http_code}" "${url}")
if [ $http_response != "200" ]; then
echo "ERROR: Server returned error code: $http_response"
cat ${target}
else
echo "DEBUG: API Success"
fi
}
fetch_filtered_active_builds(){
if [ "true" != "true" ];then
echo "Orb parameter 'consider-branch' is false, will block previous builds on any branch."
jobs_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}?filter=running"
elif [ -n "${CIRCLE_TAG:x}" ] && [ "" != "" ]; then
# I'm not sure why this is here, seems identical to above?
echo "CIRCLE_TAG and orb parameter tag-pattern is set, fetch active builds"
jobs_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}?filter=running"
else
: ${CIRCLE_BRANCH:?"Required Env Variable not found!"}
echo "Only blocking execution if running previous jobs on branch: ${CIRCLE_BRANCH}"
jobs_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/tree/${CIRCLE_BRANCH}?filter=running"
fi
if [ ! -z $TESTING_MOCK_RESPONSE ] && [ -f $TESTING_MOCK_RESPONSE ];then
echo "Using test mock response"
cat $TESTING_MOCK_RESPONSE > /tmp/jobstatus.json
else
echo "Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLE_TOKEN is set."
fetch "$jobs_api_url_template" "/tmp/jobstatus.json"
if [ -n "${CIRCLE_TAG:x}" ] && [ "" != "" ]; then
jq '[ .[] | select(.vcs_tag | . != null) | select(.vcs_tag | test("") ) ]' /tmp/jobstatus.json > /tmp/jobstatus_tag.json
mv /tmp/jobstatus_tag.json /tmp/jobstatus.json
fi
echo "API access successful"
fi
}
fetch_active_workflows(){
cp /tmp/jobstatus.json /tmp/augmented_jobstatus.json
for workflow in `jq -r ".[] | .workflows.workflow_id //empty" /tmp/augmented_jobstatus.json | uniq`
do
echo "Checking time of workflow: ${workflow}"
workflow_file=/tmp/workflow-${workflow}.json
if [ ! -z $TESTING_MOCK_WORKFLOW_RESPONSES ] && [ -f $TESTING_MOCK_WORKFLOW_RESPONSES/${workflow}.json ]; then
echo "Using test mock workflow response"
cat $TESTING_MOCK_WORKFLOW_RESPONSES/${workflow}.json > ${workflow_file}
else
fetch "https://circleci.com/api/v2/workflow/${workflow}" "${workflow_file}"
fi
created_at=`jq -r '.created_at' ${workflow_file}`
echo "Workflow was created at: ${created_at}"
cat /tmp/augmented_jobstatus.json | jq --arg created_at "${created_at}" --arg workflow "${workflow}" '(.[] | select(.workflows.workflow_id == $workflow) | .workflows) |= . + {created_at:$created_at}' > /tmp/augmented_jobstatus-${workflow}.json
#DEBUG echo "new augmented_jobstatus:"
#DEBUG cat /tmp/augmented_jobstatus-${workflow}.json
mv /tmp/augmented_jobstatus-${workflow}.json /tmp/augmented_jobstatus.json
done
}
update_comparables(){
fetch_filtered_active_builds
fetch_active_workflows
load_current_workflow_values
# falsey parameters are empty strings, so always compare against 'true'
if [ "false" = "true" ] ;then
echo "Orb parameter block-workflow is true."
echo "This job will block until no previous workflows have *any* jobs running."
oldest_running_build_num=`jq 'sort_by(.workflows.created_at)| .[0].build_num' /tmp/augmented_jobstatus.json`
oldest_commit_time=`jq 'sort_by(.workflows.created_at)| .[0].workflows.created_at' /tmp/augmented_jobstatus.json`
else
echo "Orb parameter block-workflow is false."
echo "Only blocking execution if running previous jobs matching this job: ${CIRCLE_JOB}"
oldest_running_build_num=`jq ". | map(select(.workflows.job_name==\"${CIRCLE_JOB}\")) | sort_by(.workflows.created_at)| .[0].build_num" /tmp/augmented_jobstatus.json`
oldest_commit_time=`jq ". | map(select(.workflows.job_name==\"${CIRCLE_JOB}\")) | sort_by(.workflows.created_at)| .[0].workflows.created_at" /tmp/augmented_jobstatus.json`
fi
if [ -z "$oldest_commit_time" ]; then
echo "API Error - unable to load previous job timings. Report to developer."
exit 1
fi
echo "Oldest job: $oldest_running_build_num"
if [ -z $oldest_commit_time ];then
echo "API Call for existing jobs failed, failing this build. Please check API token"
echo "All running jobs:"
cat /tmp/jobstatus.json || exit 0
echo "All running jobs with created_at:"
cat /tmp/augmented_jobstatus.json || exit 0
echo "All worfklow details."
cat /tmp/workflow-*.json
exit 1
fi
}
load_current_workflow_values(){
my_commit_time=`jq '.[] | select( .build_num == '"${CIRCLE_BUILD_NUM}"').workflows.created_at' /tmp/augmented_jobstatus.json`
}
cancel_current_build(){
echo "Cancelleing build ${CIRCLE_BUILD_NUM}"
cancel_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/${CIRCLE_BUILD_NUM}/cancel?circle-token=${CIRCLE_TOKEN}"
curl -s -X POST $cancel_api_url_template > /dev/null
}
#
# We can skip a few use cases without calling API
#
if [ ! -z "$CIRCLE_PR_REPONAME" ]; then
echo "Queueing on forks is not supported. Skipping queue..."
# It's important that we not fail here because it could cause issues on the main repo's branch
exit 0
fi
if [ "*" = "*" ] || [ "*" = "${CIRCLE_BRANCH}" ]; then
echo "${CIRCLE_BRANCH} queueable"
else
echo "Queueing only happens on * branch, skipping queue"
exit 0
fi
#
# Set values that wont change while we wait
#
load_variables
max_time=30
echo "This build will block until all previous builds complete."
echo "Max Queue Time: ${max_time} minutes."
wait_time=0
loop_time=11
max_time_seconds=$((max_time * 60))
#
# Queue Loop
#
confidence=0
while true; do
update_comparables
echo "This Workflow Timestamp: $my_commit_time"
echo "Oldest Workflow Timestamp: $oldest_commit_time"
if [[ ! -z "$my_commit_time" ]] && [[ "$oldest_commit_time" > "$my_commit_time" || "$oldest_commit_time" = "$my_commit_time" ]] ; then
# API returns Y-M-D HH:MM (with 24 hour clock) so alphabetical string compare is accurate to timestamp compare as well
# recent-jobs API does not include pending, so it is posisble we queried in between a workfow transition, and we;re NOT really front of line.
if [ $confidence -lt 1 ];then
# To grow confidence, we check again with a delay.
confidence=$((confidence+1))
echo "API shows no previous jobs/workflows, but it is possible a previous workflow has pending jobs not yet visible in API."
echo "Rerunning check ${confidence}/1"
else
echo "Front of the line, WooHoo!, Build continuing"
break
fi
else
# If we fail, reset confidence
confidence=0
echo "This build (${CIRCLE_BUILD_NUM}) is queued, waiting for build number (${oldest_running_build_num}) to complete."
echo "Total Queue time: ${wait_time} seconds."
fi
if [ $wait_time -ge $max_time_seconds ]; then
echo "Max wait time exceeded, considering response."
if [ "false" == "true" ];then
echo "Orb parameter dont-quit is set to true, letting this job proceed!"
exit 0
else
cancel_current_build
sleep 10 # wait for API to cancel this job, rather than showing as failure
exit 1 # but just in case, fail job
fi
fi
sleep $loop_time
wait_time=$(( loop_time + wait_time ))
done
staging queueable
Exited with code exit status 22
CircleCI received exit code 22
karlparry commented
Also having this issue, pinned our version back to 1.6.1 for now
eddiewebb commented
Thanks for report @karlparry, @ldiqual
Can you share what image you're using? Based on the lack of any output besides "Branch Queueable" , It is surely the introduction of the API wrapper/token check fetch which happens next, but I'm not sure what since this works locally and my images, but version of curl or shell might matter.
fetch(){
url=$1
target=$2
http_response=$(curl -f -s -X GET -H "Circle-Token:${CIRCLECI_API_KEY}" -o "${target}" -w "%{http_code}" "${url}")
if [ $http_response != "200" ]; then
echo "ERROR: Server returned error code: $http_response"
cat ${target}
else
echo "DEBUG: API Success"
fi
}
eddiewebb commented
Hmm, when I copy and paste your debug into a container I sshed into I get reasonable output, so it does make me think curl/shell compatibility.
main queueable
This build will block until all previous builds complete.
Only blocking execution if running previous jobs on branch: main
Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLE_TOKEN is set.
DEBUG: API Success
API access successful
Checking time of workflow: bd272146-1517-4026-8126-d1e32786b929
DEBUG: API Success
Workflow was created at: 2021-11-11T00:08:08Z
Orb parameter block-workflow is false.
Only blocking execution if running previous jobs matching this job: build-auditor/check_context_usage
Oldest job: 2614
This Workflow Timestamp: "2021-11-11T00:08:08Z"
Oldest Workflow Timestamp: "2021-11-11T00:08:08Z"
API shows no previous jobs/workflows, but it is possible a previous workflow has pending jobs not yet visible in API.
Rerunning check 1/1
ldiqual commented
Image is cimg/node:16.13.0 fyi.
eddiewebb commented
Ahh I think I see, I assume you're both overriding the default token name? My new method was old merge that missed the switch to parameter. eddiewebb/queue:73 has "fix" if you'd liek to validate
ldiqual commented
Confirmed that this fixed the issue we had with 1.6.2. Thank you so much!