Upload failing randomly during getOptions

[Charles-Gagnon]

Most of the time the upload task works fine, but occasionally it'll fail with the following error :

[error] "2020-06-12T05:29:37.644Z"  'error from getOptions'
##[error]Error: Command failed: git cat-file -p 33288473e4a7b9bc7218c6bd08387c30cad92176
fatal: Not a valid object name 33288473e4a7b9bc7218c6bd08387c30cad92176

CI yaml in question : https://github.com/microsoft/azuredatastudio/blob/master/.github/workflows/ci.yml#L57

Failing run : https://github.com/microsoft/azuredatastudio/pull/10879/checks?check_run_id=764299238

[Charles-Gagnon]

From my investigation I think this might be a Github actions issue - the GITHUB_SHA value doesn't seem to always match the commit on the PR branch. Opened a thread on the community forum to see if someone there can investigate : https://github.community/t/github-sha-not-always-correct/117863

[eyal0]

I saw this, too. I once noticed that GITHUB can get confused if CI is running on a branch that has since gotten a newer commit on it or has been rebased.

Can you check if that's related? Is CI success/failure related to git push -f?

[eyal0]

Yes, it's a bug with GitHub. I can easily reproduce it by making commits and uploading them successively. On my computer, I type this:

git commit --amend --no-edit; git push -f

Everytime that I type that, a new version is uploaded.

The github CI that I have does a checkout of the latest code of the PR by default but that doesn't match the GITHUB_SHA variable in the case where a commit happened after the action started but before the git clone start.

I am able to consistently cause a failure with this.

[eyal0]

Looking at the checkout@v1 script from GitHub Actions, I found that it was always getting the latest version of the code, which sometimes caused the GITHUB_SHA variable to not match the actual checkout.

However, after upgrading to checkout@v2, I find that that the checkout does match the environment variable, even with my test as described above.

@Charles-Gagnon I see that you are on checkout@v1. Want to try switching to checkout@v2? Also, it can be useful to run "env" inside your workflow, so that you can see the GITHUB_SHA environment variable while you run.

[Charles-Gagnon]

Awesome, thanks for the investigation. Updating the checkout version seems like it should fix the issue since it'll at least always match the SHA. I'll give it a shot and see if we still hit the issue.

Although I'd like to know how exactly this was happening in our case. It seemed to only happen for specific PRs and would consistently happen for that PR as each commit was added - but if it really was a race condition with multiple commits next to each other that was the issue I wouldn't expect to see it so consistently like that.

For example see https://github.com/microsoft/azuredatastudio/pull/10879/checks?sha=a3af28e325aa832cfb6d94c55a156b6c6c3e628c

Specifically this run : https://github.com/microsoft/azuredatastudio/pull/10879/checks?sha=a3af28e325aa832cfb6d94c55a156b6c6c3e628c

which failed, but the next commit wasn't for another couple of hours afterwards (and also failed).

[eyal0]

In short: Between the uploading of the PR and GitHub CI getting around to running it, there were about 3 hours during which main got 3 more commits. The GITHUB_SHA was set 3 hours before the run started and when the run did start, it checked out with the newer main. So it failed because main rolled forward while waiting for CI to begin.

---

In order to run CI, GitHub attempted to create a merge branch between that one and the branch into which you are merging, which is called main in your repo. GitHub runs CI not on the copy that you upload but rather on the code that you upload after merging with main. So it created a merge commit for you. I can see the GITHUB_SHA of it in the coveralls error message. It is aeddbe81c641cd03a55ac10a5f50987128d56666. I can read more about the commit there:

git fetch origin aeddbe81c641cd03a55ac10a5f50987128d56666 ; git checkout aeddbe81c641cd03a55ac10a5f50987128d56666 ; git show

Merge a3af28e325aa832cfb6d94c55a156b6c6c3e628c into 4322234d0b016b214bd768ce20d5599dc0153e99

The first SHA is the copyright fixes. The second one in the history of main, committed by alanren@. So that was the plan for the CI. If you look at Alan's commit, the date is just a little before "copyright fixes". So it makes sense.

However, maybe the CI was very busy with other tasks and did get around to launch for you until later. The raw logs show that it only started at 2020-06-12T02:08:05.5905251Z. But the GITHUB_SHA's date is: Thu Jun 11 23:12:42 2020 +0000 (check the log of aeddbe81c641cd03a55ac10a5f50987128d56666).

So there were ~3 hours between the creation of the job and it's actual run. During that time, main rolled forward to a7110d89160e25cf914030dd467f3d85c0292692, which was the latest main at the time of github checkout, Thu Jun 11 17:54:30 2020 -0700, by Benjin.

So the timeline is:

• Arvind makes a copyright fixes commit
• Moments later, it is uploaded as a PR and Github makes a merge commit against main at Alan's commit. GITHUB_SHA is set.
• About 3 hours pass until GitHub is able to launch the CI.
• checkout@v1 pulls the latest GitHub merge from CI, which is Arvind's copyright fix merged into Benjin's main.
• The mismatch causes coveralls to shit the bed.

[Charles-Gagnon]

Ah, I missed that there was that much of a delay before the action was ran. Sounds good - appreciate the help here!

[EvHaus]

Just documenting for posterity in case others are running into this. For me this was happening because the right ENV variables weren't being set. There's a bunch of remapping that happens inside Coveralls that made it confusing.

The fix was the following:

- name: Determine Git SHA values
  id: set_git_sha
  uses: ZenHubHQ/best-git-sha-action@v1.3
  with:
      github_event_pull_request_head_sha: ${{ github.event.pull_request.head.sha }}
      github_sha: ${{ github.sha }}
      github_event_number: ${{ github.event.number }}
      github_head_ref: ${{ github.head_ref }}
      github_ref: ${{ github.ref }}

- name: Coveralls
  uses: coverallsapp/github-action@master
  with:
    git-commit: ${{ steps.set_git_sha.outputs.sha }}
    git-branch: ${{ steps.set_git_sha.outputs.currentBranch }}
    github-token: ${{ secrets.GITHUB_TOKEN }}