[Bug] Traefik annotations don't work for any paths (except root for old Traefik versions)
aleks-fofanov opened this issue · 2 comments
Expected Behaviour
All traffic destined to a domain correctly forwarded to a function regardless of the path requested.
Current Behaviour
Depending on installed version of the Traefik, you may observe different behaviors.
If your Traefik version is below v1.7.21, then your function will receive incorrect (duplicated) path, e.g. nodeinfo.example.com/version will be forwarded to gateway/function/nodeinfo/versionversion and your function will get /versionversion as the path. All requests to root path (/) will be forwarded correctly to a function.
If you have Traefik v.1.7.21 or above then you'll get infinite redirect in attempt to request any path (even root)!
The behaviors are different because of this issue that has been addressed in release v1.7.21 of Traefik
Possible Solution
-
Traefik's rewrite target annotation should be changed from
traefik.ingress.kubernetes.io/rewrite-target: /function/FUNC_NAME/$1
to
traefik.ingress.kubernetes.io/rewrite-target: /function/FUNC_NAME
whereFUNC_NAMEis the name of the function.
I've created PR which fixes this issue. -
(Nice to have) Document (where?) that Traefik v1.7.21 or above is required
Steps to Reproduce (for bugs)
Disclaimer. I tested all of the following on linux, so not really sure that this would work on Mac or Windows.
- Create k8s cluster locally
minikube start
- Install traefik as ingress controller. For demonstration purposes, I first install v1.7.20
helm upgrade -i traefik \
--version 1.87.1 \
--namespace kube-system \
--set imageTag=1.7.20 \
--set rbac.enabled=true \
--set dashboard.enabled=true \
--set dashboard.serviceType=NodePort \
--set dashboard.domain=traefik.local.dev \
--set dashboard.ingress.annotations."kubernetes\.io\/ingress\.class=traefik" \
--set deployment.hostPort.httpEnabled=true \
--set deployment.hostPort.dashboardEnabled=true \
--set serviceType=NodePort \
--set containerSecurityContext.capabilities.drop[0]=ALL \
--set containerSecurityContext.capabilities.add[0]=NET_BIND_SERVICE \
--set deploymentStrategy.type=Recreate \
stable/traefik
- Install openfaas with ingress operator enabled
kubectl create ns dev-openfaas
kubectl create ns dev-openfaas-fn
helm upgrade -i openfaas \
--version 5.8.5 \
--namespace dev-openfaas \
--set functionNamespace=dev-openfaas-fn \
--set basic_auth=false \
--set generateBasicAuth=false \
--set ingressOperator.create=true \
--set ingress.enabled=true \
--set ingress.hosts[0].host=gateway.local.dev \
--set ingress.hosts[0].serviceName=gateway \
--set ingress.hosts[0].servicePort=8080 \
--set ingress.hosts[0].path=/ \
--set ingress.annotations."kubernetes\.io\/ingress\.class=traefik" \
openfaas/openfaas
- (Optional) Add Traefik dashboard and OpenFaaS Gateway UI to your hosts if you want to access them via their domain names
echo "$(minikube ip) traefik.local.dev" | sudo tee -a /etc/hosts
echo "$(minikube ip) gateway.local.dev" | sudo tee -a /etc/hosts
or alternatively you can get their services urls to access their UIs
minikube -n kube-system service traefik-dashboard --url
minikube -n dev-openfaas service gateway-external --url
- Deploy function
faas-cli deploy \
--image stefanprodan/podinfo:3.1.0 \
--namespace dev-openfaas-fn \
--name podinfo \
--annotation com.openfaas.health.http.initialDelay="2s" \
--annotation com.openfaas.health.http.path="/healthz" \
--env PODINFO_PORT="8080" \
--gateway $(minikube -n dev-openfaas service gateway-external --url) \
--tls-no-verify
- Deploy FunctionIngress
cat <<EOF | kubectl apply -f -
apiVersion: openfaas.com/v1alpha2
kind: FunctionIngress
metadata:
name: podinfo
namespace: dev-openfaas
spec:
function: podinfo
domain: podinfo.local.dev
ingressType: "traefik"
bypassGateway: false
tls:
enabled: false
EOF
Now the interesting part begins
- Let's see what routing rule we have in Traefik for the ingress the operator created
curl -s $(minikube -n kube-system service traefik-dashboard --url)/api/providers/kubernetes/frontends | jq -r '.["podinfo.local.dev/"].routes'
Output:
{
"/": {
"rule": "PathPrefix:/;ReplacePathRegex: ^/(.*) /function/podinfo/$1$1"
},
"podinfo.local.dev": {
"rule": "Host:podinfo.local.dev"
}
}
Note the two $1 at the end of the string with the rule. This is why our function gets duplicated path with this Traefik version.
- Add function domain to your hosts and let's see how the routing works
echo "$(minikube ip) podinfo.local.dev" | sudo tee -a /etc/hosts
curl -s http://podinfo.local.dev
Output:
{
"hostname": "podinfo-5569bb569c-9wvdw",
"version": "3.1.0",
"revision": "7b6f11780ab1ce8c7399da32ec6966215b8e43aa",
"color": "cyan",
"logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
"message": "greetings from podinfo v3.1.0",
"goos": "linux",
"goarch": "amd64",
"runtime": "go1.13.1",
"num_goroutine": "6",
"num_cpu": "2"
}
And let's also request the other path
curl -s http://podinfo.local.dev/version
Output:
404 page not found
Function actually gets /versionversion as the path.
Let's also make sure that this path is handled by requesting it through gateway:
curl -s http://gateway.local.dev/function/podinfo/version
or
curl -s $(minikube -n dev-openfaas service gateway-external --url)/function/podinfo/version
Output:
{
"commit": "7b6f11780ab1ce8c7399da32ec6966215b8e43aa",
"version": "3.1.0"
}
- Now let's change the annotation on the ingress to the proposed and see if it works with this version of Traefik
kubectl -n dev-openfaas annotate --overwrite ing podinfo 'traefik.ingress.kubernetes.io/rewrite-target=/function/podinfo'
curl -s http://podinfo.local.dev/version
Output:
error finding function podinfoenv.dev-openfaas-fn: server returned non-200 status code (404) for function, podinfoenv
- Let's rollback changes to the ingress and upgrade traefik to v1.7.23 to see how things work with this version
kubectl -n dev-openfaas annotate --overwrite ing podinfo 'traefik.ingress.kubernetes.io/rewrite-target=/function/podinfo/$1'
helm upgrade traefik \
--namespace kube-system \
--reuse-values \
--set imageTag=1.7.23 \
stable/traefik
kubectl -n kube-system rollout status deploy/traefik
- Let's request the same paths and see what we get (spoiler - infinite redirects). I intentionally limited redirects count to keep the output at acceptable level.
curl -sLv --max-redirs 1 http://podinfo.local.dev
Output:
* Rebuilt URL to: http://podinfo.local.dev/
* Trying 192.168.39.39...
* TCP_NODELAY set
* Connected to podinfo.local.dev (192.168.39.39) port 80 (#0)
> GET / HTTP/1.1
> Host: podinfo.local.dev
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Content-Length: 0
< Date: Thu, 18 Jun 2020 11:18:59 GMT
< Location: /function/podinfo/
< Vary: Accept-Encoding
<
* Connection #0 to host podinfo.local.dev left intact
* Issue another request to this URL: 'http://podinfo.local.dev/function/podinfo/'
* Found bundle for host podinfo.local.dev: 0x5566c09a19d0 [can pipeline]
* Re-using existing connection! (#0) with host podinfo.local.dev
* Connected to podinfo.local.dev (192.168.39.39) port 80 (#0)
> GET /function/podinfo/ HTTP/1.1
> Host: podinfo.local.dev
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Content-Length: 0
< Date: Thu, 18 Jun 2020 11:18:59 GMT
< Location: /function/podinfo/function/podinfo/function/podinfo/
< Vary: Accept-Encoding
<
* Connection #0 to host podinfo.local.dev left intact
* Maximum (1) redirects followed
curl -sLv --max-redirs 1 http://podinfo.local.dev/version
Output:
* Trying 192.168.39.39...
* TCP_NODELAY set
* Connected to podinfo.local.dev (192.168.39.39) port 80 (#0)
> GET /version HTTP/1.1
> Host: podinfo.local.dev
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Date: Thu, 18 Jun 2020 11:22:08 GMT
< Location: /function/podinfo/version/version
< Vary: Accept-Encoding
<
* Connection #0 to host podinfo.local.dev left intact
* Issue another request to this URL: 'http://podinfo.local.dev/function/podinfo/version/version'
* Found bundle for host podinfo.local.dev: 0x5584618379d0 [can pipeline]
* Re-using existing connection! (#0) with host podinfo.local.dev
* Connected to podinfo.local.dev (192.168.39.39) port 80 (#0)
> GET /function/podinfo/version/version HTTP/1.1
> Host: podinfo.local.dev
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Content-Length: 0
< Date: Thu, 18 Jun 2020 11:22:08 GMT
< Location: /function/podinfo/function/podinfo/version/version/function/podinfo/version/version
< Vary: Accept-Encoding
<
* Connection #0 to host podinfo.local.dev left intact
* Maximum (1) redirects followed
- Now let's change the annotation on the ingress to the proposed and see how it works
kubectl -n dev-openfaas annotate --overwrite ing podinfo 'traefik.ingress.kubernetes.io/rewrite-target=/function/podinfo'
curl -s http://podinfo.local.dev
Output:
{
"hostname": "podinfo-5569bb569c-9wvdw",
"version": "3.1.0",
"revision": "7b6f11780ab1ce8c7399da32ec6966215b8e43aa",
"color": "cyan",
"logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
"message": "greetings from podinfo v3.1.0",
"goos": "linux",
"goarch": "amd64",
"runtime": "go1.13.1",
"num_goroutine": "6",
"num_cpu": "2"
}
curl -s http://podinfo.local.dev/version
Output:
{
"commit": "7b6f11780ab1ce8c7399da32ec6966215b8e43aa",
"version": "3.1.0"
}
Those who made this far: Thanks for reading and don't forget to clean your hosts file at /etc/hosts
Context
I'm trying to use ingress operator with traefik to forward all traffic from specific domain to function.
Your Environment
-
Docker version
docker version(e.g. Docker 17.0.05 ):
Docker version 19.03.11, build 42e35e61f3 -
What version and distriubtion of Kubernetes are you using?
kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-21T20:58:41Z", GoVersion:"go1.13.11", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
-
Operating System and version (e.g. Linux, Windows, MacOS):
Ubuntu 18.04.4 LTS -
Link to your project or a code example to reproduce issue:
n/a -
What network driver are you using and what CIDR? i.e. Weave net / Flannel
n/a
Thanks for looking into this and for providing detailed instructions 👍
I've commented on the PR, where we'd usually have the test outputs too.