[TOPIC] - Deployment of Portainer inside Kubernetes

Question

[TOPIC] - Deployment of Portainer inside Kubernetes

deviantony opened this issue 4 years ago · 33 comments

Have any feedback on the deployment of Portainer for Kubernetes BETA inside a Kubernetes cluster? Discuss it here.

Answer 1 · 2020-03-31T10:42:58.000Z

I've deployed it in my K3s-cluster with Traefik 2.x (had to switch the service type to ClusterIP and create a matching IngressRoute obviously) and I'm liking it!

Two questions though:

What is the path where data is stored? So I can create a volumeMount for that
Does this support translations? If so, I'm willing to help create a translation to replace all of Portainer's terms (I know you intentionally kept them to stay user friendly) back to Kubernetes' terms (for instance Resource Pools to Namespaces, etc)

Answer 2 · 2020-04-01T02:16:32.000Z

Looks like a straight deployment into DO Managed K8s doesn't appear to work. I've got the install running with an external load balancer and can reach the UI on TCP/9000.

It appears to detect nodes and capacity but can't list namespaces, volumes, cluster level items etc.

What am I missing here?

Answer 3 · 2020-04-01T02:24:19.000Z

Hi @tardoe, thanks for the feedback ! Simple question which browser are you using? We might have found some issues with Firefox but we're investigating.

Would you be able to inspect the network queries in your browser and post the response of the failing requests?

@djpbessems thanks ! Data is stored under /data. We don't have translation support at the moment but we might consider an "advanced mode" in the future. Although it's actually more complicated than "a resource pool = a namespace".

Answer 4 · 2020-04-01T03:51:06.000Z

@deviantony I'm using Chrome.

Answer 5 · 2020-04-01T04:00:26.000Z

@tardoe interesting, seems there is some issue with the Portainer backend. Container logs will be useful to us in that situation. Cheers

Answer 6 · 2020-04-01T04:10:31.000Z

@deviantony
Just looks like a whole heap of:

2020-04-01 15:09:43	
2020/04/01 04:09:43 http: proxy error: unsupported protocol scheme "tcp"
2020-04-01 15:09:43	
2020/04/01 04:09:43 http: proxy error: unsupported protocol scheme "tcp"
2020-04-01 15:09:43	
2020/04/01 04:09:43 http: proxy error: unsupported protocol scheme "tcp"
2020-04-01 15:09:43	
2020/04/01 04:09:43 http: proxy error: unsupported protocol scheme "tcp"
2020-04-01 15:09:43	
2020/04/01 04:09:43 http: proxy error: unsupported protocol scheme "tcp"
2020-04-01 15:09:29	
2020/04/01 04:09:29 http: proxy error: unsupported protocol scheme "tcp"
2020-04-01 15:09:29	
2020/04/01 04:09:29 http: proxy error: unsupported protocol scheme "tcp"
2020-04-01 15:09:29	
2020/04/01 04:09:29 http: proxy error: unsupported protocol scheme "tcp"

Answer 7 · 2020-04-03T04:15:41.000Z

@tardoe I reproduced your problem, I figured that you actually updated the "local" endpoint to rename it. This is what's causing the problem, we'll keep a note about this. Thank you for the feedback !

Answer 8 · 2020-04-16T05:05:52.000Z

Quick update on this topic, we've added new instructions on how to deploy Portainer with Helm: https://github.com/portainer/portainer-k8s#deploy-portainer-using-helm-chart

Answer 9 · 2020-04-21T07:06:20.000Z

Hi Anthony, using your recommended settings for K3S ingress:

ingress:
  enabled: true
  annotations: {}
  hosts:
    - host: portainer.local
      paths:
      - /
  tls: []

I always get the same error, even if I try to add the requested field:

Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Ingress.spec.rules[0].http.paths[0].backend): missing required field "servicePort" in io.k8s.api.networking.v1beta1.IngressBackend

The online release has always the old configuration .Values.service.port

Thanks

Answer 10 · 2020-06-10T20:56:37.000Z

Hi, I have installed portainer beta on my in house kubernetes cluster created with kubeadm. The installation was successful and I can acces the web UI. Portainer sees my cluster. However the dashboard shows nothing and there is this error showing:
"Failure: Cannot read property 'port' of undefined"
I have no idea what is the undefined resource. I can see resource pools and configurations but everything else returns the above error.
Do you have any suggestions?

Thanks!

Answer 11 · 2020-06-10T21:05:03.000Z

Hi @gmoisio sorry I missed your comment. We'll investigate this.

Hi @djbobyd, thanks for the feedback. I believe this could be related to an existing application running inside your cluster that we are failing to parse. Do you have a lot of applications running inside the cluster? We'd need to isolate any application that might be causing the issue, it'd be great if you can share with us the applications running in your cluster via YAML manifests.

Answer 12 · 2020-06-11T08:51:33.000Z

Hi again,
Actually I have almost nothing in this cluster. It is brand new and empty. So far I have installed Canal (Flannel+Calico), Metallb, nginx ingress, nfs-provisioner from kubernetes external-storage project, cert-manager, rook-ceph (but without the ceph cluster), kubernetes-dashboard and portainer itself. That's all.

Answer 13 · 2020-06-11T21:03:30.000Z

@djbobyd would you be able to share with us the manifest that you used to deploy these? You can share it here or ping me directly on our slack channel with this information (https://portainer.io/ > support (top right) > slack)

Answer 14 · 2020-06-12T07:08:32.000Z

Hi @deviantony here is the list of commands that I used. There is nothing specific, it is pretty much a vanilla install.

CANAL
curl https://docs.projectcalico.org/manifests/canal.yaml -O
kubectl apply -f canal.yaml

MetalLB
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

NGINX
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install my-release ingress-nginx/ingress-nginx

NFS Provisioner
helm install stable/nfs-client-provisioner --set nfs.server=x.x.x.x --set nfs.path=/exported/path

Cert Manager
kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.15.1 --set installCRDs=true

Rook Operator
helm repo add rook-release https://charts.rook.io/release
helm install --namespace rook-ceph rook-release/rook-ceph

Kubernetes Dashboard
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml

I am using helm 3. Hope this helps.

Answer 15 · 2020-06-14T23:46:13.000Z

This is great @djbobyd thanks ! We're going to investigate.

Answer 16 · 2020-06-15T12:00:18.000Z

Hi @deviantony, I have the same issue as @djbobyd. I'm using IBM Cloud Kubernetes Service (Nodeport), which is a single kworker with only a test Liberty profile and Portainer. I get "Failure: Cannot read property 'port' of undefined" in Dashboard, Applications, Configuration, Volumes and Clusters. The last message in kubectl logs is "http error: Invalid JWT token (err=Invalid JWT token) (code=401)" but that was several hours ago.

Answer 17 · 2020-06-15T20:43:47.000Z

@probalsil1 thanks for the feedback, this is a javascript issue so you will not see anything in the logs of Portainer. We'll investigate.

Answer 18 · 2020-06-16T09:34:17.000Z

Hi all, an arkade (arkade is a Kubernetes app/helm chart installer) user was confused as to why he was losing data on restart of his pod. It would seem that there is no PersistentVolume support as of yet? alexellis/arkade#118

If you need a little help, Neil knows how to reach us. We could help with this.

Alex

Answer 19 · 2020-06-16T10:24:49.000Z

Hi @alexellis

Yes, data persistence is currently not supported for the beta version. This is something that we are working on in for the release of the Kubernetes version.

It is mentioned in the readme via https://github.com/portainer/kubernetes-beta#why-do-i-need-to-reconfigure-portainer-each-time-it-is-restarted in the meantime.

Thanks for the report!

Answer 20 · 2020-06-17T08:26:17.000Z

Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Ingress.spec.rules[0].http.paths[0].backend): missing required field "servicePort" in io.k8s.api.networking.v1beta1.IngressBackend

The online release has always the old configuration .Values.service.port

@gmoisio You can put service.port settings into your values.yaml to fix this. The chart configuration section should be updated. If help is needed I can send a PR.

Hi, I have installed portainer beta on my in house kubernetes cluster created with kubeadm. The installation was successful and I can acces the web UI. Portainer sees my cluster. However the dashboard shows nothing and there is this error showing:
"Failure: Cannot read property 'port' of undefined"

I have the same issue running with self-hosted k8s v1.18 installed by kubeadm. Every page will give the "Cannot read property 'port' of undefined" error. Application and dashboard page are empty. Is there a "debug mode" for further investigation?

Answer 21 · 2020-06-17T22:00:35.000Z

@Kenblair1226 we've just fixed this, I'll push an updated image soon and keep you updated here.

PR welcomed for the servicePort setting :)

Answer 22 · 2020-06-18T00:10:06.000Z

@Kenblair1226 @probalsil1 @djbobyd we have fixed the "Failure: Cannot read property 'port' of undefined" error and pushed a new image including this fix. Can you give it a try and tell us if it's working well for you? Thanks !

Answer 23 · 2020-06-18T01:08:27.000Z

Just tested, its working now and I dont see the error.

Answer 24 · 2020-06-18T01:27:51.000Z

Confirmed fixed. Everything works now!

Answer 25 · 2020-06-18T02:21:02.000Z

That's great, thanks everyone for the feedback.

Answer 26 · 2020-06-23T10:38:45.000Z

Sorry for the late reply. I also confirm the problem is gone. However the helm chart installation is still deploying the old version.

Answer 27 · 2020-06-23T16:12:36.000Z

I can also confirm the port error problem is fixed and the K8 Beta works well on IBM Cloud Kubernetes Service.

Answer 28 · 2020-07-23T22:36:06.000Z

Is the agent working for kubernetes yet? The wiki say it works, but when I try to add the remote end point, I keep getting error that "check if the server supports the requested api". I am trying to add the k8s cluster to my regular portainer instance which manages other docker servers.

Answer 29 · 2020-07-23T22:41:01.000Z

@geek-baba you'll need the beta version of Portainer with kubernetes support (as well as the beta version of the agent)in order to manage a Kubernetes endpoint, this version is not released yet.

Answer 30 · 2020-07-23T22:43:16.000Z

@geek-baba you'll need the beta version of Portainer with kubernetes support (as well as the beta version of the agent)in order to manage a Kubernetes endpoint, this version is not released yet.

Thanks for the quick reply! Could I test the beta release or its off the limits for genpop?

Answer 31 · 2020-07-23T23:17:42.000Z

@geek-baba yes it's available for testing (should not be deployed in production), just follow the instructions in the README :)

Answer 32 · 2020-07-23T23:30:12.000Z

Sorry if I was not clear, I am running:

portainer 1.24.1 - this manages my docker servers
portainer 1.0.0-k8s-rc - I have installed this on k8s cluster (both portainer and agent), I am able to access this standalone.

I am trying to add 1.0.0-k8s-rc as endpoint to 1.24.1, and I assume I need BETA version of 1.24.1 branch to able to access k8s cluster right? I dont see a beta tag for 1.24.1 branch on github or dockerhub.

Or they are not compatible with each other?

Answer 33 · 2020-07-24T01:24:04.000Z

@geek-baba ah right, these are not compatible yet. You can only manage Kubernetes endpoints from the 1.0.0-k8s-rc for now until we release the new Portainer 2.0 version.