woodpecker-ci/woodpecker

Fix pipeline cancelling

Opened this issue · 6 comments

qwerty287 commented

Component

server, agent

Describe the bug

This is mainly a summary issue of #833, #2062, #2911
and #4349

I've been trying to debug this without real success.

I've been using the local backend, and can do the following observations:

  • cancel pipeline while running: completely broken. The commands are finished, the step is marked as success, the pipeline too (#2911)
  • cancelling a pending pipeline seems to work for me

On ci.woodpecker-ci.org, I can see (uses docker backend):

  • cancel pending pipeline, agent is available: the pipeline starts anyways (this probably is #2062)
  • cancel running pipeline: works in general, but new status is failing but should be killed

System Info

next

Additional context

No response

Validations

  • Read the Contributing Guidelines.
  • Read the docs.
  • Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
  • Checked that the bug isn't fixed in the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]
  • Check that this is a concrete bug. For Q&A join our Discord Chat Server or the Matrix room.
zc-devs commented

Woodpecker 2.1.1, Kubernetes.

  • cancel pending pipeline: removed from queue, released resources, killed pipeline status, skipped step status
    Screenshot 2024-01-08 1

  • cancel running pipeline: removed from queue, released resources, success pipeline and step statuses
    Screenshot 2024-01-08 2

fernandrone commented

I've got a related issue, which is somewhat worrisome.

I was able to reproduce the original buck on a 2.3.0 installation with Kubernetes backend. I've observed it's inconsistent: sometimes cancelling will correctly show the running step as killed/cancelled and mark the pipeline as canceled. The last step to run will show "Oh no, we got some errors! Canceled" (remaining steps in the same workflow will show as grey, with the message "This step has been canceled."). Sometimes, it will show the last step to run as successful instead (and remaining steps in the same workflow will also show as grey, with the message "This step has been canceled.").

However if you have a second workflow that depends on the first (i.e. a multi-workflow pipeline, for example ./.woodpecker/a.yml and ./woodpecker/b.yml and "b" depends_on "a"), if workflow "a" is cancelled and we get the bug where its considered successful, than "b" will start running, and we will not have any way to cancel "b", because the cancel button will have been replaced by a Restart button ❗ This could lead to situations where an erroneous deployment is triggered and a developer is unable to stop it, for example.

⚠️ One thing I noticed is that, consistently, if I cancelled the pipeline between steps, that is, while a pod was in the Pending state (in other words, after a step was finished, but before the logs of a new step started to stream), the bug would occur and the pipeline would be marked as successful. However, if I were to cancel it while a step is in mid-execution (so I'm certain that a Pod was in the Running state) then the step would always cancel properly, marking the step and the whole Workflow as failed. Of course, this only applies and has only been tested on the Kubernetes backend.

I'd share links/screenshots but this all happened within our internal servers.

  • 👍4
s00500 commented

Feels related:

I am running the agent in docker compose. Woodpecker 2.7.1

Had this scenario: No agents available, about 30 pipelines created by cron.
Then I canceled all the 29 previous pipelines. When restarting the agent it started to run through all the canceled pipelines and execute them... Kinda scary...

s00500 commented

Ignore my last comment, there where just way more pipelines. the canceled ones stayed canceled, sorry for the confusuin.

wenchao5211 commented

version : 2.7.1

I often have this problem in my production environment, when I push a pr using gitea, the pipeline is triggered, but it is cancelled immediately, there is no log on the page
image

server log

{"level":"error","repo_id":"36","pipeline_id":"9765","workflow_id":"8018","error":"sql: no rows in result set","time":"2024-10-17T07:01:51Z","message":"queue.Done: cannot ack workflow"}
{"level":"error","repo_id":"36","pipeline_id":"9769","workflow_id":"8021","error":"sql: no rows in result set","time":"2024-10-17T07:05:53Z","message":"queue.Done: cannot ack workflow"}
{"level":"error","repo_id":"36","pipeline_id":"9769","workflow_id":"8021","error":"stream: not found","time":"2024-10-17T07:05:53Z","message":"done: cannot close log stream for step 22064"}
{"level":"error","repo_id":"36","pipeline_id":"9769","workflow_id":"8021","error":"stream: not found","time":"2024-10-17T07:05:53Z","message":"done: cannot close log stream for step 22065"}
{"level":"error","repo_id":"36","pipeline_id":"9769","workflow_id":"8021","error":"stream: not found","time":"2024-10-17T07:05:53Z","message":"done: cannot close log stream for step 22066"}
{"level":"error","repo_id":"36","pipeline_id":"9769","workflow_id":"8021","error":"stream: not found","time":"2024-10-17T07:05:53Z","message":"done: cannot close log stream for step 22067"}
{"level":"error","repo_id":"36","pipeline_id":"9769","workflow_id":"8021","error":"stream: not found","time":"2024-10-17T07:05:53Z","message":"done: cannot close log stream for step 22068"}
{"level":"error","repo_id":"36","pipeline_id":"9769","workflow_id":"8021","error":"stream: not found","time":"2024-10-17T07:05:53Z","message":"done: cannot close log stream for step 22069"}
{"level":"error","repo_id":"36","pipeline_id":"9769","workflow_id":"8021","error":"stream: not found","time":"2024-10-17T07:05:53Z","message":"done: cannot close log stream for step 22070"}
{"level":"error","repo_id":"36","pipeline_id":"9769","workflow_id":"8021","error":"stream: not found","time":"2024-10-17T07:05:53Z","message":"done: cannot close log stream for step 22071"}
{"level":"error","repo_id":"36","pipeline_id":"9769","workflow_id":"8021","error":"stream: not found","time":"2024-10-17T07:05:53Z","message":"done: cannot close log stream for step 22072"}
{"level":"error","error":"sql: no rows in result set","time":"2024-10-17T07:06:15Z","message":"queue: evict_at_once: [8020]"}
{"level":"error","repo_id":"36","pipeline_id":"9768","workflow_id":"8020","error":"sql: no rows in result set","time":"2024-10-17T07:06:15Z","message":"queue.Done: cannot ack workflow"}
{"level":"error","repo_id":"36","pipeline_id":"9770","workflow_id":"8022","error":"sql: no rows in result set","time":"2024-10-17T07:06:23Z","message":"queue.Done: cannot ack workflow"}
{"level":"error","repo_id":"36","pipeline_id":"9770","workflow_id":"8022","error":"stream: not found","time":"2024-10-17T07:06:23Z","message":"done: cannot close log stream for step 22074"}
{"level":"error","repo_id":"36","pipeline_id":"9770","workflow_id":"8022","error":"stream: not found","time":"2024-10-17T07:06:23Z","message":"done: cannot close log stream for step 22075"}
{"level":"error","repo_id":"36","pipeline_id":"9770","workflow_id":"8022","error":"stream: not found","time":"2024-10-17T07:06:23Z","message":"done: cannot close log stream for step 22076"}
{"level":"error","repo_id":"36","pipeline_id":"9770","workflow_id":"8022","error":"stream: not found","time":"2024-10-17T07:06:23Z","message":"done: cannot close log stream for step 22077"}
{"level":"error","repo_id":"36","pipeline_id":"9770","workflow_id":"8022","error":"stream: not found","time":"2024-10-17T07:06:23Z","message":"done: cannot close log stream for step 22078"}
{"level":"error","repo_id":"36","pipeline_id":"9770","workflow_id":"8022","error":"stream: not found","time":"2024-10-17T07:06:23Z","message":"done: cannot close log stream for step 22079"}
{"level":"error","repo_id":"36","pipeline_id":"9770","workflow_id":"8022","error":"stream: not found","time":"2024-10-17T07:06:23Z","message":"done: cannot close log stream for step 22080"}
{"level":"error","repo_id":"36","pipeline_id":"9770","workflow_id":"8022","error":"stream: not found","time":"2024-10-17T07:06:23Z","message":"done: cannot close log stream for step 22081"}
{"level":"error","repo_id":"36","pipeline_id":"9770","workflow_id":"8022","error":"stream: not found","time":"2024-10-17T07:06:23Z","message":"done: cannot close log stream for step 22082"}
{"level":"error","repo_id":"36","pipeline_id":"9771","workflow_id":"8023","error":"sql: no rows in result set","time":"2024-10-17T07:07:27Z","message":"queue.Done: cannot ack workflow"}
{"level":"error","repo_id":"36","pipeline_id":"9771","workflow_id":"8023","error":"stream: not found","time":"2024-10-17T07:07:27Z","message":"done: cannot close log stream for step 22085"}
{"level":"error","repo_id":"36","pipeline_id":"9771","workflow_id":"8023","error":"stream: not found","time":"2024-10-17T07:07:27Z","message":"done: cannot close log stream for step 22086"}
{"level":"error","repo_id":"36","pipeline_id":"9771","workflow_id":"8023","error":"stream: not found","time":"2024-10-17T07:07:27Z","message":"done: cannot close log stream for step 22087"}
{"level":"error","repo_id":"36","pipeline_id":"9771","workflow_id":"8023","error":"stream: not found","time":"2024-10-17T07:07:27Z","message":"done: cannot close log stream for step 22088"}
{"level":"error","repo_id":"36","pipeline_id":"9771","workflow_id":"8023","error":"stream: not found","time":"2024-10-17T07:07:27Z","message":"done: cannot close log stream for step 22089"}
{"level":"error","repo_id":"36","pipeline_id":"9771","workflow_id":"8023","error":"stream: not found","time":"2024-10-17T07:07:27Z","message":"done: cannot close log stream for step 22090"}
{"level":"error","repo_id":"36","pipeline_id":"9771","workflow_id":"8023","error":"stream: not found","time":"2024-10-17T07:07:27Z","message":"done: cannot close log stream for step 22091"}
{"level":"error","repo_id":"36","pipeline_id":"9771","workflow_id":"8023","error":"stream: not found","time":"2024-10-17T07:07:27Z","message":"done: cannot close log stream for step 22092"}