Nginx not finding/using globalconfiguration file & TransportServers aren't working

Question

Nginx not finding/using globalconfiguration file & TransportServers aren't working

D1StrX opened this issue a year ago · 4 comments

Describe the bug
A couple of things, chaining subjects

Nginx controller

The NginxIngress CRD deployment/pod doesn't have the Command-line Arguments - -global-configuration=$(POD_NAMESPACE)/<nginx-configuration-name>
When using the NginxIngress, it doesn' t create the globalconfiguration

yaml spec.controller.enableCustomResources: true spec.controller.globalConfiguration: <nginx-controller-namespace>/<nginx-configuration-name>

or

yaml globalConfiguration: create: true spec: listeners: - name: example-tcp port: 8888 protocol: TCP

The controller.service.customports contains this config, and the port is visible on the service. (should this also be done on the controller?)
customPorts: - name: example-tcp port: 8888 protocol: TCP targetPort: example-app

TransportServers
`
apiVersion: k8s.nginx.org/v1alpha1
kind: TransportServer
metadata:
name: example-tcp
namespace:
spec:
listener:
name: example-tcp
protocol: TCP
upstreams:
- name: example-app
service: example-service
port: 8888
action:
pass: example-app

`

`
apiVersion: k8s.nginx.org/v1alpha1
kind: GlobalConfiguration
metadata:
name: example-tcp
namespace:
spec:
listeners:
- name: example-tcp
port: 8888
protocol: TCP

`

k describe ts -n <nginx-controller-namespace> <ts>
Type Reason Age From Message
Warning Rejected 37s nginx-ingress-controller Listener example-tcp doesn't exist

The service and deployment are in a namespace, different from where the GC & TS is.

To Reproduce
Steps to reproduce the behavior:

Deploy nginx-operator using the documentation (git clone, version -> make -> deploy etc)
Create nginx controller using NgninxIngress
Create GlobalConfiguration & TransportServer, as described above

Expected behavior

GlobalConfiguration file is created when defining it in the NginxIngress
NginxIngress is using the GlobalConfiguration
TransportServer is finding the listener
Custom TCP/UDP ports can be used when having the Nginx Controller deployed through Nginx-Operator

Your environment

Version of the NGINX Ingress Operator - v1.4.0
Version of the Ingress Controller -
Version of Kubernetes. v1.26.3
Kubernetes platform (e.g. Mini-kube or GCP). On-prem, ubuntu
Using NGINX

Additional context
Goal is that a service, with a custom TCP/UDP port (eg 8888), via Ingress object can be used with the nginx.org controller (not the Kubernetes Nginx controller (which does work with custom tcp/udp)).

If someone could explain how to achieve this, would be great 😄 And the issue related to the globalconfiguration could be fixed/explained.

Answer 1 · 2023-04-19T12:29:29.000Z

Hi @D1StrX Thank you for reporting this issue. This might be due to an issue in the helm chart, as in nginxinc/kubernetes-ingress#3753, where the command line argument should still be generated in the manifest, but the name of the GlobalConfiguration does not match the actual GlobalConfiguration created.

If this is the case, due to the way operator works, you can try the following to duplicate the existing GlobalConfiguration with the correct name

kubectl get gc nginxingress-sample-nginx-ingress -n nginx-ingress -o json \
        | jq '.metadata.name = "nginx-ingress"' \
        | kubectl apply -f -

Note that you will need to update the controller custom ports too.

Please let me know if this workaround fixes the error. We are evaluating ways to address this issue and will provide an official fix soon.

Answer 2 · 2023-04-20T19:20:30.000Z

You are correct. GC named nginx-ingress and the argument in the pod matches and works now. Also the rejecting is gone on the TS, the first time.... After deploying a change to the nginx controller it breaks the TS and says 'AddedOrUpdatedWithError'

Configuration for <namespace>/<TS> was added or updated ; but was not applied: error when updating config from ConfigMap: could not get newest config version: could not get expected version: 15 after 1m0s

So the service & controller need to have the custom ports?

Then I am still wondering what is needed on the ingress part, so it works with the created GC & TS. (I have it working with kubernetes nginx controller). Are there specific annotations to be used?

Answer 3 · 2023-06-06T07:19:20.000Z

You are correct. GC named nginx-ingress and the argument in the pod matches and works now. Also the rejecting is gone on the TS, the first time.... After deploying a change to the nginx controller it breaks the TS and says 'AddedOrUpdatedWithError'

This is what worked for me, no promised it makes any sense, or provides more direction on namespaces then the official doco..

I updated GlobalConfiguration manually before using helm yaml values over --set (json ?)

Note: GlobalConfiguration.name: nginx-ingress-controller (helm chart param vs cmdline arg -global-configuration) as per nginxinc/kubernetes-ingress#3753

helm upgrade --install nginx-ingress oci://ghcr.io/nginxinc/charts/nginx-ingress \
     --version 0.17.1 \
     --namespace nginx-ingress \
     -f nginx-ingress-values.yaml

helm get values nginx-ingress -n nginx-ingress > export-nginx-ingress-values-$$.yaml

Note: the end application namespace called example where the TransportServer(s) live..

kubectl -n example apply -f nginx-transport-server-tcp-exampleca.yaml
kubectl -n example apply -f nginx-transport-server-tcp-examplemaster.yaml

FILE: nginx-ingress-values.yaml

USER-SUPPLIED VALUES:
controller:
  disableIPV6: true
  enableCustomResources: true
  globalConfiguration:
    create: true
    spec:
      listeners:
      - name: example-master
        port: 8999
        protocol: TCP
      - name: example-ca
        port: 8669
        protocol: TCP
  ingressClass: layer4-nginx
  service:
    # annotations="metallb.universe.tf/loadBalancerIPs: 1.2.3.4"
    customPorts:
    - name: example-master
      protocol: TCP
      port: 8999
      targetPort: 8999
    - name: example-ca
      protocol: TCP
      port: 8669
      targetPort: 8669

TransportServer resources should have the ingressClassName field set to the value. (especially multi-ingress installs)

FILE: nginx-transport-server-tcp-exampleca.yaml

apiVersion: k8s.nginx.org/v1alpha1
kind: TransportServer
metadata:
  name: example-ca
spec:
  ingressClassName: layer4-nginx
  listener:
    name: example-ca
    protocol: TCP
  upstreams:
  - name: exampleca-app
    service: exampleca
    port: 8669
  action:
    pass: exampleca-app

FILE: nginx-transport-server-tcp-examplemaster.yaml

apiVersion: k8s.nginx.org/v1alpha1
kind: TransportServer
metadata:
  name: example-master
spec:
  ingressClassName: layer4-nginx
  listener:
    name: example-master
    protocol: TCP
  upstreams:
  - name: examplemaster-app
    service: examplemaster
    port: 8999
  action:
    pass: examplemaster-app

FILE: exampleca-service.yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
#   metallb.universe.tf/loadBalancerIPs: 1.2.3.6
  labels:
    io.kompose.service: exampleca
  name: exampleca
spec:
#  type: LoadBalancer
  ports:
    - name: "8999"
      port: 8999
      targetPort: 8999
  selector:
    io.kompose.service: exampleca
status:
  loadBalancer: {}

$ kubectl get -n example svc
NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
exampleca            ClusterIP   10.43.1.2     <none>        8999/TCP            4d2h
examplemaster        ClusterIP   10.43.1.7     <none>        8669/TCP            4d2h

$ kubectl -n nginx-ingress get svc nginx-ingress-controller
NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                                                    AGE
nginx-ingress-controller   LoadBalancer   10.43.69.96     1.2.3.4        8999:31123/TCP,8669:31124/TCP,80:31125/TCP,443:31126/TCP   26h

kubectl -n example get ts
NAME            STATE   REASON           AGE
example-ca       Valid   AddedOrUpdated   23h
example-master   Valid   AddedOrUpdated   23h

kubectl -n example describe ts example-ca
kubectl -n example describe ts example-master

kubectl get events -n example

LAST SEEN   TYPE      REASON           OBJECT                          MESSAGE
11m         Warning   Rejected         transportserver/example-ca       Listener example-ca doesn't exist
9m52s       Normal    AddedOrUpdated   transportserver/example-ca       Configuration for example/example-ca was added or updated
11m         Normal    AddedOrUpdated   ingress/example-gui              Configuration for example/example-gui was added or updated
11m         Warning   Rejected         transportserver/example-master   Listener example-master doesn't exist
9m52s       Normal    AddedOrUpdated   transportserver/example-master   Configuration for example/example-master was added or updated

kubectl exec nginx-ingress-controller-hex-pod -n nginx-ingress -- ls -ltr /etc/nginx/
kubectl exec nginx-ingress-controller-hex-pod -n nginx-ingress -- cat /etc/nginx/stream-conf.d/ts_example_example-master.conf
kubectl exec nginx-ingress-controller-hex-pod -n nginx-ingress -- cat /etc/nginx/stream-conf.d/ts_example_example-ca.conf
kubectl exec nginx-ingress-controller-hex-pod -n nginx-ingress -- cat /etc/nginx/conf.d/example-example-gui.conf
kubectl logs -n nginx-ingress nginx-ingress-controller-hex-pod

$ kubectl get gc nginx-ingress-controller -n nginx-ingress -o yaml | grep -A10 spec
spec:
  listeners:
  - name: example-master
    port: 8999
    protocol: TCP
  - name: example-ca
    port: 8669
    protocol: TCP

$ kubectl -n nginx-ingress get svc nginx-ingress-controller -o yaml | grep -A30 ports
  ports:
  - name: example-master
    nodePort: 31123
    port: 8999
    protocol: TCP
    targetPort: 8999
  - name: example-ca
    nodePort: 31124
    port: 8669
    protocol: TCP
    targetPort: 8669
  - name: http
    nodePort: 31125
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    nodePort: 31126
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/name: nginx-ingress
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 1.2.3.4

RANDOM DEBUG

curl --noproxy '*' -v 'https://IP:PORT' --resolve fqdn:PORT:1.2.3.4
step certificate inspect https://IP:PORT --bundle --short --roots=ca.pem # --insecure

kubectl logs -n nginx-ingress
kubectl describe ing
kubectl exec -n nginx-ingress -- cat /etc/nginx/nginx.conf
kubectl describe vs
kubectl describe vsr
kubectl describe pol
kubectl describe configmap -n nginx-ingress
kubectl port-forward 8080:8080 --namespace=nginx-ingress
kubectl describe ts -n

Experiments.
helm pull oci://ghcr.io/nginxinc/charts/nginx-ingress --untar --version 0.17.1
cd nginx-ingress
kubectl apply -f crds/
;kubectl delete -f crds/
kubectl get crd -A | grep nginx

Reading

F5 NGINX Ingress Controller
https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-helm/#pulling-the-chart
https://docs.nginx.com/nginx-ingress-controller/configuration/transportserver-resource
https://www.nginx.com/blog/announcing-nginx-ingress-controller-for-kubernetes-release-1-7-0/#tcp-udp
https://github.com/nginxinc/kubernetes-ingress/tree/v3.1.1/examples/custom-resources/basic-tcp-udp
https://github.com/nginxinc/kubernetes-ingress/tree/v3.1.1/examples/ingress-resources/tcp-udp

https://www.blinkops.com/blog/troubleshooting-your-nginx-ingress-controller
https://www.blinkops.com/blog/how-to-rollback-your-kubernetes-deployment
https://helm.sh/docs/intro/using_helm/#the-format-and-limitations-of---set

Todo ... look at NGINX Ingress Operator

Answer 4 · 2023-06-17T08:13:43.000Z

I managed to get it to work as well.. but the I had to use an example from the ngnix docs.. somewhere deep in Github of nginx repo. What still doesn't work is Aacustom name for the GC. Only nginx-ingress-default-controller works (tested by letting the CRD create the config file).

Edited: These examples helped me understand the relation/references of all names. TS needs to be deployed in the namespace where the target service resides. GC needs the correct name. listener name of TS and listeners name of GC needs to match. Upstream TS needs have the target service, name can be anything. Action of TS needs to pass the name(s) entered of the TS upstream name.