Delete notification: Operation cannot be fulfilled on replicasets.extensions, the object has been modified
hjacobs opened this issue · 2 comments
hjacobs commented
An exception occurs in v0.6 when --delete-notification is set to a value > 0 and a deployment with a TTL annotation is processed. Note that everything works (events emitted, resources deleted) and the annotations are set as expected, but the exception is a bit annoying:
kube-janitor-69f4484d9b-djc9m janitor 2019-03-23 22:20:28,730 INFO: Deployment nginx will be deleted at 2019-03-23T22:24:30Z (annotation janitor/ttl is set)
kube-janitor-69f4484d9b-djc9m janitor 2019-03-23 22:20:28,756 INFO: ReplicaSet nginx-dbddb74b8 will be deleted at 2019-03-23T22:24:30Z (annotation janitor/ttl is set)
kube-janitor-69f4484d9b-djc9m janitor 2019-03-23 22:20:28,773 ERROR: Failed to clean up: Operation cannot be fulfilled on replicasets.extensions "nginx-dbddb74b8": the object has been modified; please apply your changes to the latest version and try again
kube-janitor-69f4484d9b-djc9m janitor Traceback (most recent call last):
kube-janitor-69f4484d9b-djc9m janitor File "/usr/local/lib/python3.7/site-packages/pykube/http.py", line 239, in raise_for_status
kube-janitor-69f4484d9b-djc9m janitor resp.raise_for_status()
kube-janitor-69f4484d9b-djc9m janitor File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
kube-janitor-69f4484d9b-djc9m janitor raise HTTPError(http_error_msg, response=self)
kube-janitor-69f4484d9b-djc9m janitor requests.exceptions.HTTPError: 409 Client Error: Conflict for url: https://10.3.0.1:443/apis/extensions/v1beta1/namespaces/default/replicasets/nginx-dbddb74b8
kube-janitor-69f4484d9b-djc9m janitor
kube-janitor-69f4484d9b-djc9m janitor During handling of the above exception, another exception occurred:
kube-janitor-69f4484d9b-djc9m janitor
kube-janitor-69f4484d9b-djc9m janitor Traceback (most recent call last):
kube-janitor-69f4484d9b-djc9m janitor File "/kube_janitor/main.py", line 51, in run_loop
kube-janitor-69f4484d9b-djc9m janitor dry_run=dry_run)
kube-janitor-69f4484d9b-djc9m janitor File "/kube_janitor/janitor.py", line 231, in clean_up
kube-janitor-69f4484d9b-djc9m janitor counter.update(handle_resource_on_ttl(resource, rules, delete_notification, dry_run))
kube-janitor-69f4484d9b-djc9m janitor File "/kube_janitor/janitor.py", line 156, in handle_resource_on_ttl
kube-janitor-69f4484d9b-djc9m janitor send_delete_notification(resource, reason, expiry_time, dry_run=dry_run)
kube-janitor-69f4484d9b-djc9m janitor File "/kube_janitor/janitor.py", line 75, in send_delete_notification
kube-janitor-69f4484d9b-djc9m janitor add_notification_flag(resource, dry_run=dry_run)
kube-janitor-69f4484d9b-djc9m janitor File "/kube_janitor/janitor.py", line 52, in add_notification_flag
kube-janitor-69f4484d9b-djc9m janitor resource.update()
kube-janitor-69f4484d9b-djc9m janitor File "/usr/local/lib/python3.7/site-packages/pykube/objects.py", line 142, in update
kube-janitor-69f4484d9b-djc9m janitor self.api.raise_for_status(r)
kube-janitor-69f4484d9b-djc9m janitor File "/usr/local/lib/python3.7/site-packages/pykube/http.py", line 246, in raise_for_status
kube-janitor-69f4484d9b-djc9m janitor raise HTTPError(resp.status_code, payload["message"])
kube-janitor-69f4484d9b-djc9m janitor pykube.exceptions.HTTPError: Operation cannot be fulfilled on replicasets.extensions "nginx-dbddb74b8": the object has been modified; please apply your changes to the latest version and try again
kube-janitor-69f4484d9b-djc9m janitor 2019-03-23 22:21:30,727 INFO: Clean up run completed: resources-processed=2941, deployments-with-ttl=1, replicasets-with-ttl=1
kube-janitor-69f4484d9b-djc9m janitor 2019-03-23 22:22:32,614 INFO: Clean up run completed: resources-processed=2941, deployments-with-ttl=1, replicasets-with-ttl=1
kube-janitor-69f4484d9b-djc9m janitor 2019-03-23 22:23:34,703 INFO: Clean up run completed: resources-processed=2941, deployments-with-ttl=1, replicasets-with-ttl=1
kube-janitor-69f4484d9b-djc9m janitor 2019-03-23 22:24:36,777 INFO: Deployment nginx with 10m TTL is 10m6s old and will be deleted (annotation janitor/ttl is set)
kube-janitor-69f4484d9b-djc9m janitor 2019-03-23 22:24:36,789 INFO: Deleting Deployment default/nginx..
kube-janitor-69f4484d9b-djc9m janitor 2019-03-23 22:24:36,808 INFO: ReplicaSet nginx-dbddb74b8 with 10m TTL is 10m6s old and will be deleted (annotation janitor/ttl is set)
kube-janitor-69f4484d9b-djc9m janitor 2019-03-23 22:24:36,817 INFO: Deleting ReplicaSet default/nginx-dbddb74b8..
kube-janitor-69f4484d9b-djc9m janitor 2019-03-23 22:24:36,834 INFO: Clean up run completed: resources-processed=2941, deployments-with-ttl=1, deployments-deleted=1, replicasets-with-ttl=1, replicasets-deleted=1
hjacobs commented
This probably happens because the deployment already propagates the new annotation to the replica set and kube-janitor needs to reload the replica set before attempting the update (which actually is a no-op in this case).
blacs30 commented
The create_event function fails when the resource with the ttl is a namespace itself because the event is going to be created in the namespace which is being terminated, that is only for the ExpiryTimeReached event, the TimeToLiveExpired is written to the default namespace successfully.