Create Azure CNI based AKS Cluster with Application Gateway Ingress Controller (AGIC) Using Terraform

AKS clusters can be created with two networking types Kubenet (basic networking) and Azure CNI (advanced networking). If you are using Windows nodes as well in your AKS cluster you must use Azure CNI (Container Networking Interface).  Application gatway ingress controller is a great way to setup Ingress for AKS. Let's look at terraform code to deploy Azure CNI based AKS with AGIC.

First we need to define providers block.

A resource group and virtual network in Azure should be created.

We need to create two subnets.

• One subnet for AKS cluster. This subnet should have enough address space to support Nodes and pods as well, since Azure CNI based AKS cluster will assign each pod with an IP from the subnet of the cluster /18 CIDR allows more than 16,000 IP addresses avaialble to the cluster.
• Second subnet for AKS ingress App Gatway that would be used by AGIC to implement ingress to AKS.

Public IP adress for Ingress App Gatway need to be created.

We can create the AKS ingress App Gateway as the next step. The app gatway is required to be created with basic setup. Therefore dummy values are used for setting up intial backend pool, backend settings, listeneer and routing rule. Once the control is handed over to AGIC, the App Gateway settings will be fully managed by AGIC based on the ingress specified in AKS. Threfore we have to set ignore changes to these settings using terraform lifecycle ignore_changes.

We can setup Azure contianer registry to use as docker registry, which is to be associated with AKS.

Azure AD group can be reffred as data and can be added to AKS as admins.

The AKS cluster is defined interraform as shown below. Lifecycle ignore_changes is added for default node pool node count as to prevent TF from overwriting the autoscaling in AKS in next deployment of terraform. Windows profile is added as we are creating a cluster with Windows container support. Note that the cluster admins are defined, using the previously read Azure AD group. We have defined a name for the node resource group where the node pools and related resources get created. If we do not define a name Azure will generate a name for this resource group. But as we are using the mamanged identity from this resource group to enable AGIC, we need a known name for the resource group of the node pools. The ingress app gatway is defined with the id of the App Gateway created earlier.

Then we can define a windows node pool for the AKS cluster as shown below. Note that the node count is added to ignore in terraform for next deployments.

We should allow AKS cluster to pull docker images from the Azure contianer registry. Threfore AcrPull role is added to Azurr container registry as shown below.

To allow the AGIC to manage the App Gateway setting based on ingress settings deployed to AKS cluster, the AGIC requires contributor role for the managed identity created for AGIC in the AKS node pool resource group. Therefore we need to read the node pool resource group and the managed identity of ingress app gateway as data from terraform once the AKS cluster is created. So the required dependencies are added in the teraform code below.

 

The complete code explained above can be found here in  GitHub.