services	author
aks, app-service, container-registry, azure-monitor, storage, virtual-network, virtual-machines, private-link, templates	paolosalvatori

Create an Azure Kubernetes Service cluster with the Application Gateway Ingress Controller

This ARM template can be used to deploy a public or private Azure Kubernetes Cluster (AKS) cluster with an Azure Application Gateway and Application Gateway Ingress Controller add-on. The template allows to deploy a rich set of AKS features such as:

Please report any issue you may find using the template, thanks!

Architecture

This diagram shows the overall architecture:

The ARM template deploys:

A new virtual network with 4 subnets:
- AksSubnet: hosts the AKS cluster
- VmSubnet: hosts the jumpbox virtual machine and private endpoints
- AppGatewaySubnet: hosts Application Gateway WAF2
- AzureBastionSubnet: Azure Bastion
The AKS cluster uses a user-defined managed identity to create additional resources like load balancers and managed disks in Azure.
The AKS cluster is composed of a:
- System node pool hosting only critical system pods and services. The worker nodes have node taint which prevents application pods from beings scheduled on this node pool.
- User node pool hosting user workloads and artifacts.
An Azure Bastion resource that provides secure and seamless SSH connectivity to the jumpbox virtual machine directly in the Azure portal over SSL
An Azure Container Registry (ACR) to build, store, and manage container images and artifacts in a private registry for all types of container deployments.
a Storage Account for the boot diagnostic logs of the jumpbox virtual machine
An Application Gateway used by the Application Gateway Ingress Controller
A Web Access Firewall (WAF) Policy associated to the Application Gateway as the root level and HTTP listener level. The Policy is configured in Prevention mode and uses the OWASP 3.1 rule set and a couple of custom rules that demostrate how to block requests when the query string or a header contain a specific string. You can create more sophisticated custom rules to whitelist or blacklist the incoming traffic to the Application Gateway Ingres Controller.
A Key Vault that can be used by workloads running on AKS to retrieve keys, certificates, and secrets
A private endpoint to the Blob Storage Account
A private endpoint to to Azure Container Registry (ACR)
A private endpoint to Key Vault
When you opt for a private AKS cluster, a private endpoint to the control plane / API server hosted by an AKS-managed Azure subscription. In this case, the cluster can communicate with the API server exposed via a Private Link Service using a private endpoint.
When you choose Premium as SKU for ACR, a Private Endpoint is created to allow the private AKS cluster to access ACR via a private IP address. For more information, see Connect privately to an Azure container registry using Azure Private Link.
A Private DNS Zone for the name resolution of the private endpoint to the Blob Storage Account
A Private DNS Zone for the name resolution of the private endpoint to Azure Container Registry (ACR)
A Private DNS Zone for the name resolution of the private endpoint to Key Vault
When you deploy the cluster as private, a Private DNS Zone for the name resolution of the private endpoint to the Kubernetes Server API
A Virtual Network Link between the virtual network hosting the AKS cluster and the above Private DNS Zones
A jumpbox virtual machine to manage the AKS cluster in case you decide to deploy a private AKS cluster.
A Log Analytics workspace to collect the diagnostics logs and metrics from:
- AKS cluster
- Jumpbox virtual machine
- Application Gateway
- Key Vault
- Network Security Group

Deployment

You can use the deploy.sh Bash script to deploy the topology. The following picture shows the resources deployed by the ARM template in the target resource group.

The following picture shows the resources deployed by the ARM template in the MC resource group associated to the AKS cluster:

Application Gateway

Azure Application Gateway is a web traffic load balancer that enables customers to manage the inbound traffic to multiple downstream web applications and REST APIs. Traditional load balancers operate at the transport layer (OSI layer 4 - TCP and UDP) and route traffic based on source IP address and port, to a destination IP address and port. The Application Gateway instead is an application layer (OSI layer 7) load balancer. Azure Application Gateway provides a rich set of features:

For more information, see How an Application Gateway works.

Web Access Firewall Policy for Application Gateway

Web Application Firewall (WAF) is a service that provides centralized protection of web applications from common exploits and vulnerabilities. WAF is based on rules from the OWASP (Open Web Application Security Project) core rule sets. WAF also provide the ability to create custom rules that are evaluated for each request that passes through the WAF. These rules hold a higher priority than the rest of the rules in the managed rule sets. The custom rules contain a rule name, rule priority, and an array of matching conditions. If these conditions are met, an action is taken (to allow or block). Web applications can be the target of malicious attacks that exploit common, known vulnerabilities that include SQL injection attacks, DDOS attacks, and cross site scripting attacks. Preventing such attacks in application code can be challenging and may require rigorous maintenance, patching and monitoring at many layers of the application topology. A centralized web application firewall helps make security management much simpler and gives better assurance to application administrators against threats or intrusions. A WAF solution can also react to a security threat faster by patching a known vulnerability at a central location versus securing each of individual web applications. Existing application gateways can be converted to a Web Application Firewall enabled application gateway very easily. Azure Application Gateway allows the association of a separate WAF policy to each individual listener. For example, if there are three sites behind the same Application Gateway or WAF, you can configure three separate WAF policies (one for each listener) to customize the exclusions, custom rules, and managed rulesets for one site without effecting the other two. If you want a single policy to apply to all sites, you can just associate the policy with the Application Gateway, rather than the individual listeners, to make it apply globally. Application Gateway also supports per-URI WAF Policies. This feature requires the use of a Path-based routing rule instead of a basic routing rule and requires the definition of a URL Path Map where a specific WAF policy can be associated to a given URL. For more information, see Configure per-site WAF policies using Azure PowerShell. The order of precedence for WAF policies is as follows:

If a per-URI WAF policy exists for the current path, this will take effect / apply and no other WAF policy will apply
If no per-URI WAF policy exists for the current path, but a WAF policy exists for the current listener, this policy will apply, and no other WAF policy will take effect
If no WAF policy exists for the current URI and listener, the global WAF policy will apply, if any.

The Application Gateway WAF can be configured to run in the following two modes:

Detection mode: Monitors and logs all threat alerts. You turn on logging diagnostics for Application Gateway in the Diagnostics section. You must also make sure that the WAF log is selected and turned on. Web application firewall doesn't block incoming requests when it's operating in Detection mode.
Prevention mode: Blocks intrusions and attacks that the rules detect. The attacker receives a "403 unauthorized access" exception, and the connection is closed. Prevention mode records such attacks in the WAF logs.

You can configure Application Gateway to store diagnostic logs and metrics to Log Analytics. In this case, also WAF logs will be stored in Log Analytics and they can be queries using Kusto Query Language.

Visio

In the visio folder you can find the Visio document which contains the above diagrams.

References

Azure Kubernetes Service:

Azure Application Gateway:

Azure Application Gateway Ingress Controller

Azure Application Gateway WAF