Kube-janitor doesn't work at all if one of the API's is inaccessible
bdevnani3 opened this issue · 4 comments
bdevnani3 commented
When running kube-janitor on our cluster, we got 503's when attempting to access the api metrics/v1alpha1.
2019-08-12 05:06:00,534 DEBUG: Collecting resources for metrics/v1alpha1..
2019-08-12 05:06:00,734 DEBUG: https://172.20.0.1:443 "GET /apis/metrics/v1alpha1/ HTTP/1.1" 503 20
2019-08-12 05:06:00,734 ERROR: Failed to clean up: 503 Server Error: Service Unavailable for url: https://172.20.0.1:443/apis/metrics/v1alpha1/
Traceback (most recent call last):
File "/kube_janitor/main.py", line 51, in run_loop
dry_run=dry_run)
File "/kube_janitor/janitor.py", line 215, in clean_up
for _type in resource_types:
File "/kube_janitor/resources.py", line 39, in get_namespaced_resource_types
for api_version, resource in discover_namespaced_api_resources(api):
File "/kube_janitor/resources.py", line 32, in discover_namespaced_api_resources
r2.raise_for_status()
File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://172.20.0.1:443/apis/metrics/v1alpha1/
This prevented kube-janitor from running at all on our cluster.
Is it by design supposed to fail when one API is inaccessible?
In addition(and I think this has been raised in another issue), this would also be avoided in our specific use-case if we didn't iterate through all but instead just the required APIs(That, of course, doesn't address the root cause).
hjacobs commented
Good point, the API discovery should be more robust.
bdevnani3 commented
I can work on a push that attempts to fix this if that's okay?
hjacobs commented
Yes, a PR is welcome 😄