miracle2k/k8s-snapshots

manual deleting for volume (PV) causes the snapshot pod to fail

Opened this issue · 2 comments

i noticed that when i deleted a namespace volumes with it's snapshots manually from GCP console

this caused the tool to fail. i suggest to pass volumes that don't exists and continue the tool work with some error message, the BUG is FETAL for the tool and it cause it to enter crush loop back.

oogleapis.com/compute/v1/projects/X/zones/us-central1-a/disks/gke-X-d039b866-d-pvc-9749b085-9548-11ea-8048-4201ac100009/createSnapshot?alt=json returned "The resource \'projects/X/zones/us-central1-a/disks/gke-X-d039b866-d-pvc-9749b085-9548-11ea-8048-4201ac100009\' was not found">\n']}, {'type': 'SnapshotCreateError', 'message': 'Error creating snapshot', 'data': {}, 'readable': ['Traceback (most recent call last):\n', '  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/__main__.py", line 58, in main\n    loop.run_until_complete(main_task)\n', '  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete\n    return future.result()\n', '  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/core.py", line 610, in daemon\n    await asyncio.gather(*tasks)\n', '  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/core.py", line 575, in backuper\n    await make_backup(ctx, current_target_rule)\n', '  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/snapshot.py", line 125, in make_backup\n    ) from exc\n', 'k8s_snapshots.errors.SnapshotCreateError: SnapshotCreateError: Error creating snapshot {}\n']}]
2020-05-31T06:05:10.716336Z Shutdown complete              [k8s_snapshots.__main__] message=Shutdown complete severity=INFO
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/snapshot.py", line 101, in make_backup
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/snapshot.py", line 180, in create_snapshot
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/asyncutils.py", line 12, in run_in_executor
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/snapshot.py", line 184, in <lambda>
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/backends/google.py", line 255, in create_snapshot
  File "/usr/local/lib/python3.6/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line 840, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 404 when requesting https://compute.googleapis.com/compute/v1/projects/X/zones/us-central1-a/disks/gke-X-d039b866-d-pvc-9749b085-9548-11ea-8048-4201ac100009/createSnapshot?alt=json returned "The resource 'projects/X/zones/us-central1-a/disks/gke-X-d039b866-d-pvc-9749b085-9548-11ea-8048-4201ac100009' was not found">

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/k8s-snapshots", line 11, in <module>
    load_entry_point('k8s-snapshots==0.0.0', 'console_scripts', 'k8s-snapshots')()
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/__main__.py", line 58, in main
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/core.py", line 610, in daemon
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/core.py", line 575, in backuper
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/snapshot.py", line 125, in make_backup
k8s_snapshots.errors.SnapshotCreateError: SnapshotCreateError: Error creating snapshot {}

@miracle2k is this tool still maintained ?

@umomany I don't have a lot of time to invest to fix this bug (or others), but I do review and merge pull requests.