palantir/policy-bot

status checks mismatched between PRs

devinburnette opened this issue · 3 comments

devinburnette commented

I think we found an edge case where policy bot gets confused about which PR it's reporting about. I think maybe because the commits are cached and since a new commit wasn't made between PRs, it doesn't know to bust the cache on the GitHub context or something like that.

Steps to reproduce:

  • Create a new feature branch off main and push a commit
  • Create a PR against main for that commit
  • Create a new branch from that feature branch and push up the same commit
  • Create a PR against main for that commit

When you click "details" in the status check, it will actually show the evaluation of the previous PR and not the current PR.

bluekeyes commented

I haven't tried the reproduction yet, but I think this is a consequence of how GitHub models status checks, rather than an issue with Policy Bot.

Statuses are attached to commit hashes, so a single commit that appears in multiple PRs will contain the union of all statuses added by those PRs. If the two PRs target different branches, this should show two Policy Bot statuses (see #57), with different contexts. If they target the same branch, the Policy Bot statuses will use the same context and overwrite each other. The "Details" link should point to whichever of the two PRs evaluated most recently.

Visiting the Policy Bot details page will post a new status, so if this explanation is correct (and assuming there are no other actions happening on the PRs), you should be able to switch which PR the details link points at by visiting the Policy Bot details page and editing the URL to view the page for the other PR.

devinburnette commented

Yup okay, we are seeing the second case where the 2 PRs target the same branch. One PR was closed and another PR opened with the same commit creating this scenario. Any action to get the new PR to re-evaluate seems to correct the issue, but just thought I'd bring it up here since I hadn't seen it before.

Thanks for confirming it's just how Github models the status checks. I'm not really too worried about this edge case as long as there's no way to utilize this to merge unapproved code, but from what I understand about the issue this won't be possible because any new commit will trigger re-evaluation and a previously approved commit (even if it was on a different PR) is still, well, approved.. for a lack of better words.

bluekeyes commented

Right, as far as I know, this is confusing, but not a bypass. If you propose the same commit to the same branch in two PRs, the approval conditions should be the same. If you propose the same commit to two different branches in two PRs, each branch will post a unique status check, so you can't use the policy on one branch to get approval for another.