Failed upgrade may lead to an endless loop of rollbacks
kovayur opened this issue · 4 comments
Problem
When the reconciler fails to upgrade the release it rollbacks to the previous revision and returns an error. The controller runtime is expected to retry the reconciliation with an exponential backoff, but in reality it keeps reconciling over and over again. I was able to reproduce this behavior for the following use cases:
- Lack of the PATCH permission for the operator service account to update the K8S object.
- Error in the YAML structure caused by a bug in the chart or incorrect values. Example: an env variable in
Deployment
is set by bothvalue
andvalueFrom
tags (stackrox/stackrox#7105). - CRD used in the release has been removed from the cluster.
Every rollback increases the revision count. In my case, the operator spawns thousands of revisions in a matter of minutes.
Root cause
A rolled back revision is no different from the upgraded revision, it has the deployed
status as after a normal upgrade. There always be a diff between the expected state calculated from the CR and the rolled back revision this will lead to a failed upgrade again and again.
There're events that are added in the reconciliation queue aside of the exponential backoff and cause the reconciliation without any delay. These events are:
- CR status is updated on every reconcile. This is because
Irreconcilable
status is updated twice for every reconcile both withFalse
(right before the upgrade) andTrue
(after the upgrade failed). - A failed upgrade and subsequent rollback causes multiple changes in the secrets storage which are watched and each adds an item the reconciliation queue. Let's say that revision 1 was successful, revision 2 is problematic, revision 3 - is the rollback to revision 1. Upon upgrading to version 2 the following events will be triggered:
- Create revision 2 with status
pending-upgrade
- Mark revision 2 as failed
- Create revision 3 with status
pending-rollback
- Mark revision 1 as
superseded
- Mark revision 3 as
deployed
orfailed
depending on the rollback result.
- Create revision 2 with status
There is deduplication in the queue, but still at least one event will be queued without delay.
This is biting us as well, going into backoff seems like the best solution here no?
Hey @acornett21, thanks for sharing the link to this issue.
I don't think it's the same, as the mentioned issue affects operator-sdk and mine is related to helm-operator-plugins.
As far as I understand, the reconciliation logic in these repositories is different.
If operator-sdk is not affected by this issue, perhaps we could learn from there how to fix it in helm-operator-plugins
@kovayur It does look like the reconcile.go
files are unique for each project, I assumed operator-sdk
was the source of truth for this, and helm-operator-plugins
imported this logic. It seems that the implementation in opeator-sdk
for this fix, should be carried into helm-operator-plugins
project.