DevOps-Nirvana/Kubernetes-Volume-Autoscaler

Exception while trying to describe all PVCs

Closed this issue · 3 comments

pblgomez commented

Describe the bug
Checking the app ( as it was working before ) it now throws an error

TypeError: argument of type 'NoneType' is not iterable

To Reproduce
Steps to reproduce the behavior

  1. Step 1
  2. Step 2
  3. Step 3

Expected behavior
To be able to describe pvc

Extra Information Requested

  • Kubernetes Version: 1.24
  • Prometheus Version: kube-prometheus-stack v45.18.0
  • Enable "Verbose" mode in helm chart, and copy/paste the values printed therein
-------------------------------------------------------------------------------------------------------------
               Volume Autoscaler - Configuration
-------------------------------------------------------------------------------------------------------------
             Prometheus URL: http://kube-prometheus-stack-prometheus:9090
         Prometheus Version: 2.42.0
          Prometheus Labels: {}
    Interval to query usage: every 60 seconds
             Scale up after: 5 intervals (300 seconds total)
     Scale above percentage: disk is over 80% full
 Scale up minimum increment: 1000000000 bytes, or 1G
 Scale up maximum increment: 16000000000000 bytes, or 16T
      Scale up maximum size: 16000000000000 bytes, or 16T
        Scale up percentage: 20% of current disk size
          Scale up cooldown: only resize every 22200 seconds
               Verbose Mode: is ENABLED
                    Dry Run: is Disabled
 HTTP Timeouts for k8s/prom: is 15 seconds
-------------------------------------------------------------------------------------------------------------
Exception while trying to describe all PVCs
Traceback (most recent call last):
  File "/app/./main.py", line 64, in <module>
    pvcs_in_kubernetes = describe_all_pvcs(simple=True)
  File "/app/helpers.py", line 335, in describe_all_pvcs
    output_objects["{}.{}".format(item.metadata.namespace,item.metadata.name)] = convert_pvc_to_simpler_dict(item)
  File "/app/helpers.py", line 305, in convert_pvc_to_simpler_dict
    if 'volume.autoscaler.kubernetes.io/last-resized-at' in pvc.metadata.annotations:
TypeError: argument of type 'NoneType' is not iterable
AndrewFarley commented

@pblgomez Thanks for reporting this. I will do some validation on 1.24 and your specific version of Prometheus shortly, and see if I can replicate your bug and will comment back.

pblgomez commented

It's not happening anymore, so it's hard to give you more info.
Feel free to close as it doesn't happen anymore. 🤷‍♂️

AndrewFarley commented

Catching more exceptions in 1783f4d which should catch and handle this gracefully...

  • 👍1