This role creates files for using OpenID Connect with NGINX Plus in Kubernetes. The role should be idempotent (it can run over itself detecting changes). The goal of the role is to allow you to automate multiple IDPs for OIDC in Kubernetes. This involves generating the nginx-config.yaml and applying it afterwards (this role just generates the file for now).
Note: NGINX Plus Kubernetes Ingress Controller (KIC) has a technology preview of OIDC since 1.10.
Note: This role is still in active development. There may be unidentified issues and the role variables may change as development continues.
Ansible
This role was developed and tested with maintained versions of Ansible. Backwards compatibility is not guaranteed.
Instructions on how to install Ansible can be found in the Ansible website I suggest using pip or venv so you can keep your existing environment clean.
NGINX Plus Ingress
This role assumes you have built the NGINX Plus Ingress with R22 and added the nginx-plus-module-njs package as part of the install. There is a sample file in files/DockerfileForPlusOIDC if you need a reference. The official instructions are on the documentation page.
Kubernetes
This role assumes you have already stood up Kubernetes. This was tested with 1.15, 1.16, 1.17, and 1.18 so far.
OIDC IDP
This role assumes you have already setup your OIDC and OAuth2 provider. An older walkthrough on some of this is available here. The official implementation for NGINX is here.
Variables are listed in the defaults/main.yml file and must be modified in vars/main.yml or in your playbook. This role will not work with the default idp variables. Most are self explanatory but you must add an idp section and the hostname for the first one should be default.
Note: You must define an idp for hostname: default. I call this idp0 and the hostname: default. This must be configured from your OIDC information.
Platform
This should run on a linux OS, however I have run this on my MacBook Pro with success. Post issues for OS in the github issues section.
Ansible
No other Ansible Galaxy roles are needed at the present time. You could build something more robust by adding kubernetes commands and the ingress build also.
Role Installation
You have a couple options here:
You can install the role from galaxy and create a playbook like below
ansible-galaxy install magicalyak.ansible_role_nginx_ingress_oidcYou can Download the git repo and modify the vars as mentioned above (I usually add my customization to the playbook itself).
ansible-galaxy install git+https://github.com/magicalyak/ansible-role-nginx-ingress-oidc.git
#OR
git clone https://github.com/magicalyak/ansible-role-nginx-ingress-oidc.git
cd ansible-role-nginx-ingress-oidc
ansible-galaxy install -f -r requirements.yml
cd -If you clone this role, create a playbook called nginx-oidc-install-custom.yml and it will ignore it for git upload.
---
- hosts: localhost
gather_facts: false
connection: local
vars:
oidc_files_location: "/home/centos/git/ansible-role-nginx-ingress-oidc/files" # Where we place our generated files
oidc_backup: true # Save copies of previous files
oidc_headless: true # For multiple ingress keyvalue store sync (comment out if not desired)
oidc_resolver: 8.8.8.8 # kube-dns.kube-system.svc.cluster.default is the default so you probably want to make this 8.8.8.8
oidc_idps:
idp0: # these names are placeholders, suggest to use idp$i where $i is an incremental number
# You could use the hostname or the client_id if you want, just make this unique per group
hostname: default # make one of these default (this applies to any unmatched host as a catch-all)
oidc_authz_endpoint: "http://127.0.0.1:8080/auth/realms/master/protocol/openid-connect/auth"
oidc_token_endpoint: "http://127.0.0.1:8080/auth/realms/master/protocol/openid-connect/token"
oidc_jwt_keyfile: "http://127.0.0.1:8080/auth/realms/master/protocol/openid-connect/certs"
oidc_client_id: "a2b20239-2dce-4306-a385-ac9clientid"
oidc_client_secret: "kn_3VLh]1I3ods*[DDmMxNmg8xxx"
oidc_logout_redirect: "http://127.0.0.1:8080/auth/realms/master/protocol/openid-connect/logout"
oidc_hmac_key: kn_3VLh]1I3ods*[DDmMxNmg8xxx
oidc_socat_enable: false # this is only needed if we enable socat for foward proxy (mark non-tunnelled idps as false)
idp1:
hostname: cafe.nginx.net # This only will apply the configuration to the host "cafe.nginx.net"
oidc_authz_endpoint: "https://login.microsoftonline.com/dd3dfd2f-6a3b-40d1-9be0-tenantid/oauth2/v2.0/authorize"
oidc_token_endpoint: "https://login.microsoftonline.com/dd3dfd2f-6a3b-40d1-9be0-tenantid/oauth2/v2.0/token"
oidc_jwt_keyfile: "https://login.microsoftonline.com/dd3dfd2f-6a3b-40d1-9be0-tenantid/discovery/v2.0/keys"
oidc_client_id: "f66df7b0-7378-489a-a98d-clientid"
oidc_client_secret: "PourSomeSecretsOnMeButDontUseThisOne"
oidc_logout_redirect: "https://login.microsoftonline.com/dd3dfd2f-6a3b-40d1-9be0-tenantid/oauth2/v2.0/logout"
oidc_hmac_key: ThisHMACNeedsToBeUnique
oidc_socat_enable: true # true means we will create a socat service to handle tunnelling to a forward proxy for this idp
idp2:
hostname: cafe.example.com
oidc_authz_endpoint: "https://login.microsoftonline.com/dd3dfd2f-6a3b-40d1-9be0-tenantid/oauth2/v2.0/authorize"
oidc_token_endpoint: "https://login.microsoftonline.com/dd3dfd2f-6a3b-40d1-9be0-tenantid/oauth2/v2.0/token"
oidc_jwt_keyfile: "https://login.microsoftonline.com/dd3dfd2f-6a3b-40d1-9be0-tenantid/discovery/v2.0/keys"
oidc_client_id: "f66df7b0-7378-489a-a98d-clientid2"
oidc_client_secret: "PourSomeSecretsOnMeButDontUseThisOne2"
oidc_logout_redirect: "https://login.microsoftonline.com/dd3dfd2f-6a3b-40d1-9be0-tenantid/oauth2/v2.0/logout"
oidc_hmac_key: ThisHMACNeedsToBeUnique2
oidc_socat_enable: true # this creates another socat service (named cage-example-com-socat.nginx-ingress.svc.cluster.defult)
oidc_kube_dns: coredns.kube-system.svc.k8s.nginx.net # Only change if you don't use coredns as default (maybe kube-dns.kube-system.svc.cluster.default)
oidc_headless_dns: nginx-ingress-headless.nginx-ingress.svc.k8s.nginx.net # Only change if customized, otherwise it will be nginx-ingress-headless.nginx-ingress.svc.cluster.default
oidc_keyval_size: 1M # keyval store size (1M)
oidc_keyval_id_timeout: 1h # keyval id timeout (1h)
oidc_keyval_refresh_timeout: 8h # keyval refresh timeout (8h)
oidc_errorlog_level: debug # default is error you may want debug initially
ingress_container_pullsecret: regcred # Used for dockerhub credentials - this is the name of the pull secret - (if undefined this is not used)
ingress_allow_cidr: 0.0.0.0/0 # Range for status page (if undefined this is disabled) - 0.0.0.0/0 is not secure change for production
#ingress_prometheus: # Prometheus exporter - If not defined = disabled
# scrape: true
# port: 9113
ingress_type: deployment # deployment or replicaset
ingress_deployment_count: 1 # number of ingress controllers
ingress_imagename: magicalyak/nginx-plus:OIDC # container image name - change this unless you are me
ingress_pullpolicy: Always # container restart policy (maybe you want IfNotPresent)
# socat_enable: true # enable socat for tunneling forward proxy
# socat_namespace: external # namespace for socat service and deployment (default: default)
# socat_proxy_ip: 10.233.65.4 # The IP or dns of your forward proxy
tasks:
- include_role:
name: magicalyak.ansible_role_nginx_ingress_oidc
Verify Files
Use the files in ./files to configure the ingress controller You should have 4 or 5 files depending on your options.
ansible % ls -la files R22-k8s
total 112
drwxr-xr-x 10 tom.gamull staff 320 Jun 17 15:57 .
drwxr-xr-x 7 tom.gamull staff 224 Jun 17 15:54 ..
-rw-r--r-- 1 tom.gamull staff 239 Jun 17 15:57 headless.yaml
-rw-r--r-- 1 tom.gamull staff 18162 Jun 17 15:57 nginx-config.yaml
-rw-r--r-- 1 tom.gamull staff 2081 Jun 17 15:57 nginx-plus-ingress.yaml
-rw-r--r-- 1 tom.gamull staff 11248 Jun 17 15:57 openid_connect.js
-rw-r--r-- 1 tom.gamull staff 3850 Jun 17 15:57 openid_connect.server_confCreate Kubernetes Ingress Resources
Modify the nginx-plus-ingress.yaml if needed for proxy settings, etc. Modify the nginx-plus-service.yaml in the kubernetes-ingress/deployments/service directory as appropriate Create the ingress resource as normal but specify the nginx-config.yaml located here Create the configmaps (these shouldn't change after you import the first time)
First Time Install
NGINX_K8S_GIT_DIR=/home/centos/git/kubernetes-ingress
NGINX_K8S_OIDC_DIR=/home/centos/nginc-openid-connect
cd $NGINX_K8S_GIT_DIR/deployments
git checkout v1.7.2
kubectl apply -f common/ns-and-sa.yaml
kubectl apply -f rbac/rbac.yaml
kubectl apply -f common/default-server-secret.yaml
kubectl apply -f common/vs-definition.yaml
kubectl apply -f common/vsr-definition.yaml
kubectl apply -f common/ts-definition.yaml
kubectl apply -f common/gc-definition.yaml
kubectl apply -f common/global-configuration.yaml
cd $NGINX_K8S_OIDC_DIR
kubectl apply -f nginx-config.yaml
kubectl create configmap -n nginx-ingress openid-connect.js --from-file=openid_connect.js
kubectl create configmap -n nginx-ingress openid-connect.server-conf --from-file=openid_connect.server_conf
kubectl apply -f nginx-plus-ingress.yaml
cd $NGINX_K8S_GIT_DIR/deployments
kubectl apply -f service/nginx-plus-service.yaml # Make sure this exists and you modified it
cd $NGINX_K8S_OIDC_DIRPrevious Install
Uncomment lines if you're upgrading the ingress controller from an earlier version.
NGINX_K8S_GIT_DIR=/home/centos/git/kubernetes-ingress
NGINX_K8S_OIDC_DIR=/home/centos/nginx-openid-connect
#cd $NGINX_K8S_GIT_DIR/deployments
#kubectl apply -f common/ns-and-sa.yaml
#kubectl apply -f rbac/rbac.yaml
#kubectl apply -f common/vs-definition.yaml
#kubectl apply -f common/vsr-definition.yaml
#kubectl apply -f common/ts-definition.yaml
#kubectl apply -f common/gc-definition.yaml
#kubectl apply -f common/global-configuration.yaml
#kubectl delete configmap -n nginx-ingress openid-connect.js
#kubectl delete configmap -n nginx-ingress openid-connect.server-conf
cd $NGINX_K8S_OIDC_DIR
#kubectl create configmap -n nginx-ingress openid-connect.js --from-file=openid_connect.js
#kubectl create configmap -n nginx-ingress openid-connect.server-conf --from-file=openid_connect.server_conf
kubectl apply -f nginx-config.yaml
#kubectl apply -f nginx-plus-ingress.yamlConfigure your applications
Once you finish you can create an application like the cafe example
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: cafe-ingress
annotations:
custom.nginx.org/oidc: "on"
spec:
tls:
- hosts:
- cafe.nginx.net
secretName: cafe-secret
rules:
- host: cafe.nginx.net
http:
paths:
- path: /tea
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80Exposing JWT Claims in the application
Use something similar to below:
---
- hosts: localhost
gather_facts: false
connection: local
vars:
replace_http_custom: True
apply_nginx_config: False
oidc_files_location: "." # Where we place our generated files
oidc_backup: false # Save copies of previous files
oidc_headless: true # For multiple ingress keyvalue store sync (comment out if not desired)
#oidc_external_status_address: 172.16.186.100
oidc_resolver: 8.8.8.8
oidc_idps:
idp0: # these names are placeholders, suggest to use idp$i where $i is an incremental number
# You could use the hostname or the client_id if you want, just make this unique per group
hostname: default # make one of these default
oidc_authz_endpoint: "https://##REPLACEME##/authorize"
oidc_token_endpoint: "https://##REPLACEME##/token"
oidc_jwt_keyfile: "https://##REPLACEME##/keys"
oidc_client_id: "##CLIENTID##"
oidc_client_secret: "##SECRET##"
oidc_logout_redirect: "https://##REPLACEME##/logout"
oidc_hmac_key: vC5FabzvYvFZFBzxtRCYDYX+
idp1:
hostname: cafe.nginx.net
oidc_authz_endpoint: "https://##REPLACEME##/authorize"
oidc_token_endpoint: "https://##REPLACEME##/token"
oidc_jwt_keyfile: "https://##REPLACEME##/keys"
oidc_client_id: "##CLIENTID##"
oidc_client_secret: "##SECRET##"
oidc_logout_redirect: "https://##REPLACEME##/logout"
oidc_hmac_key: vC5FabzvYvFZFBzxtRCYDYX+
idp2:
hostname: cafe.example.com
oidc_authz_endpoint: "https://##REPLACEME##/authorize"
oidc_token_endpoint: "https://##REPLACEME##/token"
oidc_jwt_keyfile: "https://##REPLACEME##/keys"
oidc_client_id: "##CLIENTID##"
oidc_client_secret: "##SECRET##"
oidc_logout_redirect: "https://##REPLACEME##/logout"
oidc_hmac_key: vC5FabzvYvFZFBzxtRCYDYX+
oidc_keyval_size: 1M # keyval store size (1M)
oidc_keyval_id_timeout: 1h # keyval id timeout (1h)
oidc_keyval_refresh_timeout: 8h # keyval refresh timeout (8h)
oidc_errorlog_level: debug # default is error
ingress_type: deployment # deployment or replicaset
ingress_deployment_count: 1 # number of ingress controllers
#ingress_container_pullsecret: regcred # Used for dockerhub credentials (if undefined this is not used)
ingress_allow_cidr: 0.0.0.0/0 # Range for status page (if undefined this is disabled) - 0.0.0.0/0 is not secure change for production
#ingress_prometheus: # Prometheus exporter - If not defined = disabled
# scrape: true
# port: 9113
ingress_imagename: magicalyak/nginx-plus:OIDC # container image name
ingress_pullpolicy: Always # container restart policy
tasks:
- include_role:
name: magicalyak.ansible_role_nginx_ingress_oidc
# This converts the preffered username (email) into a username by stripping off the @domain.com
- name: Add custom variables to nginx-config.yml HTTP Ingress context
blockinfile:
path: "{{ oidc_files_location }}/nginx-config.yaml"
insertafter: "^.{4}js_import oidc from conf.d/openid_connect.js;$"
block: |2
map $jwt_claim_preferred_username $jwt_sample1 {
default $jwt_claim_preferred_username;
~^(?<username>.*)@(?<domain>.*)$ "${username}";
}
marker: " # {mark} ANSIBLE MANAGED BLOCK"
when: replace_http_custom
- name: Apply nginx-config.yaml
k8s:
state: present
src: "{{ oidc_files_location }}/nginx-config.yaml"
when: apply_nginx_configThen in your application ingress add some header values from the jwt claim. Please not these are exposed to the upstream server and not the end user (you won't see these in developer mode on chrome, but your upstream applicaiton will).
# This is the application ingress file for a sample application
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: cafe-ingress
annotations:
custom.nginx.org/oidc: "on"
nginx.org/location-snippets: |
proxy_set_header username $jwt_claim_sub;
proxy_set_header jwt-email $jwt_claim_email;
proxy_set_header jwt-username $jwt_claim_preferred_username;
proxy_set_header jwt-name $jwt_claim_name;
proxy_set_header jwt-aud $jwt_claim_aud;
proxy_set_header jwt-firstname $jwt_claim_given_name;
proxy_set_header jwt-lastname $jwt_claim_family_name;
proxy_set_header jwt-sample1 $jwt_sample1;
spec:
tls:
- hosts:
- cafe.nginx.net
secretName: cafe-secret
rules:
- host: cafe.nginx.net
http:
paths:
- path: /tea
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80Update IDPs
Simply run the playbook again and it will generate a new nginx-config.yml file (you'll see it in yellow as changed) Then just apply the file and you'll only need to configure the app ingress. This should be able to be automated quite easily.
cd /home/centos/nginx-openid-connect
vim nginx-oidc-install-custom.yml # Add the new app and IDP
ansible-playbook nginx-oidc-install-custom.yml
kubectl apply -f nginx-config.yml