Websocket error
dexma-dev opened this issue · 11 comments
Hello everybody,
i am trying to install the mattermost team edition chart on my cluster but I am running into some websocket issues.
I am experiencing the same issues as described here.
Is this an issue of the helm chart, the docker container or the nginx ingress?
{"timestamp":"2022-03-31 21:10:41.886 Z","level":"error","msg":"Failed to upgrade websocket connection.","caller":"web/context.go:105","path":"/api/v4/websocket","request_id":"a1us537gp3b7xxyoawiuksyrmr","ip_addr":"xxxxx","user_id":"xxxxxx","method":"GET","err_where":"connect","http_code":500,"err_details":"websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header"}
UPDATE:
Please keep in mind that I am on kubernetes version 1.23.1 and i am therefore using the networking.k8s.io/v1 ingress api. I am not sure this issue is related to that. I made the same changes to the ingress as described in #250. Everything else is working as expected.
Kubernetes version: v1.23.1
Ingress: Nginx
Ingress apiVersion: networking.k8s.io/v1
@gabrieljackson / @Szymongib - Any thoughts on this? It looks like the nginx config isn't upgrading the websocket connection.
and i am therefore using the networking.k8s.io/v1 ingress api
Do you create the ingress manually then? Could you share it here so we can investigate?
Thank you for your time and help!
I cloned the repo and changed the ingress definition inside the helm chart (team-edition).
Here is the resulting ingress definition:
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
name: mattermost-mattermost-team-edition
namespace: gitlab
labels:
app.kubernetes.io/instance: mattermost
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: mattermost-team-edition
helm.sh/chart: mattermost-team-edition-6.4.1
annotations:
acme.cert-manager.io/http01-edit-in-place: 'true'
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: 'true'
meta.helm.sh/release-name: mattermost
meta.helm.sh/release-namespace: gitlab
nginx.ingress.kubernetes.io/client-max-body-size: 1G
nginx.ingress.kubernetes.io/proxy-body-size: 1G
spec:
tls:
- hosts:
- mattermost.domain.com
secretName: secret-name
rules:
- host: mattermost.domain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: mattermost-team-edition
port:
number: 8065
status:
loadBalancer: {}
After several failed attempts i also tried to add a second ingress especially for the upgrade of the connection:
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
name: mattermost-mattermost-team-edition-header
namespace: gitlab
labels:
app.kubernetes.io/instance: mattermost
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: mattermost-team-edition
helm.sh/chart: mattermost-team-edition-6.4.1
annotations:
acme.cert-manager.io/http01-edit-in-place: 'true'
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: 'true'
meta.helm.sh/release-name: mattermost
meta.helm.sh/release-namespace: gitlab
nginx.ingress.kubernetes.io/client-max-body-size: 1G
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "Upgrade: $http_upgrade";
more_set_headers "Connection: 'upgrade'";
proxy_set_header "Upgrade: $http_upgrade";
proxy_set_header "Connection: 'upgrade'";
nginx.ingress.kubernetes.io/proxy-body-size: 1G
spec:
tls:
- hosts:
- mattermost.domain.com
secretName: secret-name
rules:
- host: mattermost.domain.com
http:
paths:
- path: /api/v[0-9]+/(users/)?websocket$
pathType: Prefix
backend:
service:
name: mattermost-team-edition
port:
number: 8065
status:
loadBalancer: {}
Thanks! Could you also share some details about your setup? Are you running on some Cloud? Perhaps the Nginx config/chart values could be helpful too.
I am running Bare Metal with the following configurations:
Kubernetes Version: v1.23.1
NGINX Ingress: v2.1.1
Installation Type: Manifests
No special configuration. I followed the installation guide which can be found here.
MetalLB LoadBalancer: v0.12.1
Installation Type: Manifests
No special configuration here either. Followed the installation guide which can be found here.
All of this was running perfectly on kubernetes version 1.19 a couple of weeks ago. I upgraded to the newest version, therefore i am pretty sure it has something to do with the new ingress api or the newer version of the ingress controller.
@Szymongib any ideas where the problem is coming from?
Thanks for the details @CNeumaier. So is the issue only with websockets? Can you access Mattermost at all?
@Szymongib Everything else is working as expected. I am having only issues with the websockets
That is just a blind guess but I think the Ingress controller you are using is a different thing although based on Nginx too.
I think nginx.ingress.kubernetes.io annotation is for https://github.com/kubernetes/ingress-nginx and the link you sent is https://github.com/nginxinc/kubernetes-ingress/ which according to docs uses nginx.org annotation. I have no experience with the latter so it is hard for me to say if it is configured properly.
You could be onto something there. Let me investigate further and maybe use the „normal“ nginx ingress. I will report back
@Szymongib your guess was on point!
I removed my ingress controller and I installed the kubernetes/ingress-nginx version, which resolved the websocket error.
I did not investigate any further why the other ingress controller caused the issue.
Thank you very much for your help!