Quickstart doesn't work

Question

Quickstart doesn't work

Closed this issue 5 years ago · 6 comments

evanstucker-hates-2fa commented 6 years ago

I'm on Google Cloud Platform Kubernetes Engine running master version 1.11.6-gke.3. This is what I get when I try quickstart:

$ kubectl logs -n kube-system -f k8s-snapshots-5bb755c6cb-bpb6n
2019-02-02T23:52:05.066886Z rule.heartbeat                 [k8s_snapshots.core] message=rule.heartbeat rules=None severity=INFO
2019-02-02T23:52:05.071043Z kube-config.from-service-account [k8s_snapshots.context] message=kube-config.from-service-account severity=INFO
2019-02-02T23:52:05.122363Z watch-resources.worker.error   [k8s_snapshots.kube] message=watch-resources.worker.error resource_type_name=PersistentVolume severity=ERROR
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 99, in raise_for_status
    resp.raise_for_status()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.0.0.1:443/api/v1/persistentvolumes?watch=true

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/kube.py", line 181, in worker
    for event in sync_iterator:
  File "/usr/local/lib/python3.6/site-packages/pykube/query.py", line 156, in object_stream
    self.api.raise_for_status(r)
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 106, in raise_for_status
    raise HTTPError(resp.status_code, payload["message"])
pykube.exceptions.HTTPError: persistentvolumes is forbidden: User "system:serviceaccount:kube-system:default" cannot watch persistentvolumes at the cluster scope
2019-02-02T23:52:07.105507Z watch-resources.worker.error   [k8s_snapshots.kube] message=watch-resources.worker.error resource_type_name=PersistentVolumeClaim severity=ERROR
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 99, in raise_for_status
    resp.raise_for_status()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.0.0.1:443/api/v1/persistentvolumeclaims?watch=true

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/kube.py", line 181, in worker
    for event in sync_iterator:
  File "/usr/local/lib/python3.6/site-packages/pykube/query.py", line 156, in object_stream
    self.api.raise_for_status(r)
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 106, in raise_for_status
    raise HTTPError(resp.status_code, payload["message"])
pykube.exceptions.HTTPError: persistentvolumeclaims is forbidden: User "system:serviceaccount:kube-system:default" cannot watch persistentvolumeclaims at the cluster scope
2019-02-02T23:52:08.102549Z watch-resources.worker.error   [k8s_snapshots.kube] message=watch-resources.worker.error resource_type_name=SnapshotRule severity=ERROR
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 99, in raise_for_status
    resp.raise_for_status()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.0.0.1:443/apis/k8s-snapshots.elsdoerfer.com/v1/snapshotrules?watch=true

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/kube.py", line 181, in worker
    for event in sync_iterator:
  File "/usr/local/lib/python3.6/site-packages/pykube/query.py", line 156, in object_stream
    self.api.raise_for_status(r)
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 106, in raise_for_status
    raise HTTPError(resp.status_code, payload["message"])
pykube.exceptions.HTTPError: snapshotrules.k8s-snapshots.elsdoerfer.com is forbidden: User "system:serviceaccount:kube-system:default" cannot watch snapshotrules.k8s-snapshots.elsdoerfer.com at the cluster scope

Answer 1 · 2019-02-03T00:02:21.000Z

Applied rbac.yaml and deleted the k8s-snapshots pod. Looks like the result is the same.

$ kubectl logs -n kube-system -f k8s-snapshots-5bb755c6cb-p6s69
2019-02-03T00:01:14.821950Z rule.heartbeat                 [k8s_snapshots.core] message=rule.heartbeat rules=None severity=INFO
2019-02-03T00:01:14.824421Z kube-config.from-service-account [k8s_snapshots.context] message=kube-config.from-service-account severity=INFO
2019-02-03T00:01:14.872437Z watch-resources.worker.error   [k8s_snapshots.kube] message=watch-resources.worker.error resource_type_name=PersistentVolume severity=ERROR
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 99, in raise_for_status
    resp.raise_for_status()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.0.0.1:443/api/v1/persistentvolumes?watch=true

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/kube.py", line 181, in worker
    for event in sync_iterator:
  File "/usr/local/lib/python3.6/site-packages/pykube/query.py", line 156, in object_stream
    self.api.raise_for_status(r)
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 106, in raise_for_status
    raise HTTPError(resp.status_code, payload["message"])
pykube.exceptions.HTTPError: persistentvolumes is forbidden: User "system:serviceaccount:kube-system:default" cannot watch persistentvolumes at the cluster scope
2019-02-03T00:01:16.858636Z watch-resources.worker.error   [k8s_snapshots.kube] message=watch-resources.worker.error resource_type_name=PersistentVolumeClaim severity=ERROR
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 99, in raise_for_status
    resp.raise_for_status()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.0.0.1:443/api/v1/persistentvolumeclaims?watch=true

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/kube.py", line 181, in worker
    for event in sync_iterator:
  File "/usr/local/lib/python3.6/site-packages/pykube/query.py", line 156, in object_stream
    self.api.raise_for_status(r)
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 106, in raise_for_status
    raise HTTPError(resp.status_code, payload["message"])
pykube.exceptions.HTTPError: persistentvolumeclaims is forbidden: User "system:serviceaccount:kube-system:default" cannot watch persistentvolumeclaims at the cluster scope
2019-02-03T00:01:17.859250Z watch-resources.worker.error   [k8s_snapshots.kube] message=watch-resources.worker.error resource_type_name=SnapshotRule severity=ERROR
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 99, in raise_for_status
    resp.raise_for_status()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.0.0.1:443/apis/k8s-snapshots.elsdoerfer.com/v1/snapshotrules?watch=true

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/k8s_snapshots-0.0.0-py3.6.egg/k8s_snapshots/kube.py", line 181, in worker
    for event in sync_iterator:
  File "/usr/local/lib/python3.6/site-packages/pykube/query.py", line 156, in object_stream
    self.api.raise_for_status(r)
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 106, in raise_for_status
    raise HTTPError(resp.status_code, payload["message"])
pykube.exceptions.HTTPError: snapshotrules.k8s-snapshots.elsdoerfer.com is forbidden: User "system:serviceaccount:kube-system:default" cannot watch snapshotrules.k8s-snapshots.elsdoerfer.com at the cluster scope

Answer 2 · 2019-02-03T00:10:14.000Z

Verified that metadata is available to populate GCLOUD_PROJECT:

durable-bond-123456root@k8s-snapshots-6c849b5b4-twn8m:/app# curl -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/project/project-id; echo
durable-bond-123456

Answer 3 · 2019-02-03T00:34:22.000Z

So I'm just stupid regarding service accounts. The solution appears to be adding "serviceAccountName: k8s-snapshots" to the podSpec. Can we get this added to the README.md?

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: k8s-snapshots
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: k8s-snapshots
    spec:
      serviceAccountName: k8s-snapshots
      containers:
      - name: k8s-snapshots
        image: elsdoerfer/k8s-snapshots:v2.0
        env:
        - name: GCLOUD_PROJECT
          value: durable-bond-225016

Answer 4 · 2019-03-08T14:15:16.000Z

I was also stung by this, and then noticed there's an rbac.yaml and crd.yaml which you also need. It would be nicer to have those manifests (along with the deployment yaml) be in a folder, then the quickstart could just be k apply -f manifests.

Answer 5 · 2019-08-26T23:42:22.000Z

I've moved the manifests to a subfolder, but kubectl apply -f manifests is not recommended. (You'd only need one of the crd manifests, and rbac I suppose only if your cluster uses RBAC, plus there is no manifest file there to actually run the service).

Answer 6 · 2019-08-26T23:43:04.000Z

I do believe that the README can be improved. In particular, I am thinking about splitting it into multiple subfiles.