Handle Nginx Gateway Certificate Refresh While Inplace Upgrade in AKS

 In the post "High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 2 - Deploy Nginx-Gateway-Fabric" we hav discussed how to get nginx gateway setup in AKS. This approach works fine for the first install and if you are using true blue green with a fresh AKS cluster. However, when we use componenets such as elastic search  on AKS (which we will discuss in future posts how to setup elastc search on AKS) ,we have to use inplace AKS upgrades, with new node pools in same cluster, as we want to persist the data on elastic. In such inplace AKS upgrade requirements we will have to upgrade cert manager and nginx gateway as well inplace. When we try to do such upgrades to cert-manager and nginx gateway we are running into a issue as decribed below.

The Issue

Immediately after the upgrade or after a time interval, the dataplane pods of nginx gteway will run into a high CPU situation and will try to create pods. These pods will not be able to start properly as it they will not be able to validate the certificates generated. Ideally this situation should have been handled by the control plane (operator) of niginx gateway. however, it does not do that properly.

Adding SnippetsFIlter to HTTPRoute in Nginx Gateway to Implement Proxy Annotations of Retired Ingress Nginx

 We have discussed "Setup Nginx-Gateway Routes and Create Gateway Policies to Map with Ingress Nginx Annotations" in a previous post. However, we have not setup all that were previously set via ingress-nginx annotations, and only partial setup was made using ClientSettingsPolicy and UpstreamSettingsPolicy. Some proxy annotations that we used with ingress-nginx can only be setup by using a SnippetsFilter. 

Such SnippetsFilter is shown below for example.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 5 - Cleanup Ingress-Nginx from the AKS Cluster

 We have switched traffic from ingress-nginx to nginx-gateway as discussed in the previous blog. The next step is to gent ingress-nginx setup cleaned up from AKS env. This can be done manually. However, when there are mutiple production regions deployed with same deployment, it is easier if we programatically remove all ingress-nginx related routing and setupp, so it can be deployed to all production environments, same way, without issues.

The expectation is to get ingrss nginx removed, and the namespace ingress-nginx also cleaned up from the AKS cluster. Therefore, we can remove the namespace creation yaml as shown below.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 4 - Switch Traffic from Ingress-Nginx to Nginx-Gateway

 In the part 2 and part 3 of this blog series, we have deployed nginx-gateway in AKS, with required route and policy setup to correctly route to elastic search and kibana deployed in AKS. But we are still have live traffic to elastic search and kibana via the retired ingress-nginx, after the deployment of nginx-gateway also in the same cluster. No as we planned we are going to do the release 2 to get the traffic switch from retired ingress-nginx to new nginx-gateway deployed in AKS.

The expectation is to have kibana and elastic host names to resolve to private IP of the nginx-gateway loadbalancer as shown below, and the ability to use elastic search with the url, using nginx-gateway.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 3 - Setup Nginx-Gateway Routes and Create Gateway Policies to Map with Ingress Nginx Annotations

We have discussed how to deploy nginx-gateway in part 2 of this blog series.  As the nginx-gateway is deployed and ready now, we can start setting up HTTP routes, so that it is ready to replace the ingress-nginx (retired) routes already working in the AKS cluster. The example in this blog uses, routes to AKS hosted elastic search, so there are two routes, the kibana dashboard route and the elastic search route used by the apps. The currrent routes use a private DNS, and a private IP within Azure vNET. We have used a new private IP for nginx gateway in deploy nginx-gateway in part 2 , so that current retired  ingress-nginx internal loadbalancer is not affected, while we setup nginx-gateway (uses another private IP for loadbalancer of gateway), making them both available in AKS cluster. Therefore, setting up same host names for gateway routes will not cause issues in existing ingress-nginx setup. After this step of setting up routes, we will have both ingress-nginx (retired) and nginx-gateway with routes, working in same cluster, but live traffic will be still using ingress-nginx.

The expectation is to have routes using nginx-gateway is setup for ealstic search and kibana as shown below.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 2 - Deploy Nginx-Gateway-Fabric

In part 1 "High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 1 - Plan for Smooth Transition", we have discussed the plan to transtion from retired ingess-nginx to nginx-gateway, for an AKS cluster where we have hosted elatic search.  In this post let's look at steps necessary to deploy nginx-gateway.

The expection is to have successfully deployed high available nginx-gateway with nginx gateway fabric.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 1 - Plan for Smooth Transition

 We have discussed setting up ingress-nginx as ingress controller for AKS previously. Ingress-nginx for kubernetes is retiring in March 2026 as announced here in github and in kubernetes documentation. Gateway API is the recommended replacement for ingress-nginx and Nginx Gateway Fabric is an implementation of Gateway API by Nginx. When you have true blue-green deployments with new AKS cluster getting created at each deployment, transitioning from ingress-nginx to nginx gateway fabric is relatively simpler, and we can discuss that in the last blog of this blob post series. However, when you have a single AKS cluster (situations such as hosting elastic search on AKS demands, we use only a single AKS cluster - we will discuss this topic in a later blog series), where blue-green deployments happen using node pools and cluster upgrades as inplace upgrades, the transtion needs to be done over multiple releases to achive zero down time deployments for production. In this blog, let's plan how we can get the transition done in a step by step manner.

The expectation is to get nginx-gateway setup as shown below, so that HTTP routes can be setup via the gateway.

Use Terraform to Switch Azure Key Vault to Use RBAC Permisions from Access Policies Without a Downtime for Applications/Users

 For Azure Key Vaults access policies based permision setup is now legacy and all key vaults will have to use Azure RBAC permisions eventually for data access permisions according to offcial Microsft documentation here. Using terraform we can setup the changes. However, we have to be carefull about the switching to RBAC from access policies in production scenarios to avoid interptions to applications. Taking two step approach, first set RBAC permisions and in a next release performing switch to RBAC for key vault will help the transtion to be smooth. Let's look at how to setup this requirement with terraform.

The expectation is to have a keyvault setup with RBAC permisions as shown below.

 

Using Remote Terraform State

 Sometimes resources common to multiple diffrent setups might need to be created with a common terraform code. In such cases the commeo terraform resources may need to be reffered with its state in  other terraform code. For this requirement we can use terraform remote state. Let's see how we can use terraform remote state step by step in this post.

The expectation is to refer to the Azure resources in remote terraform state as shown below. Here you can see we have reffered to the resource group name and location, and to log analytics workspace id from remote state.

Enable Windows Data Scraping for AKS Managed Prometheus with Azure Managed Grafana

 We have "Setup Managed Prometheus for AKS via Terraform", however, that setup alone will not provide windows metrics from AKS clsuter to Azure managed Grafana. We have to addtionaly, setup Windows exporter and couple of additional configurations to make it work as decribed in the official Microsoft docs here. Let's look at step by step how to enable windows metrics for AKS with managed prometheus.

Expected outcome is getting metrics such as shown below to managed grafana and visualizing them.

Performing Terraform Import via Azure Pipelines for Existing Azure Resources

Importing state to terraform may be required when a resource is manually created previously, now need to be managed by terraform. Or it can be a situation where you Move code from one repo to another for reorganizing purpose and now you need to refer to exesting resource in Azure and map it to new repo terraform code. The state import can be performed with terraform import command manually. However, performin such task manually targeting production environments is not ideal and kind of impossible, in autmated deployment implementations. In this post let's discuss, a Azure pipeline task that can be used to perform the state imports in a rerunnable way.

Such task toupdate terraform state in Azure pipline should be placed between the terraform init and terraform plan tasks as shown below.

Setup Managed Prometheus for AKS via Terraform

 We have discussed "Deploying Azure Managed Grafana with Terraform" ealier. Now that we have managed grafana available in Azure for the AKS clsuters we can enable managed prometheus. You can setup prometheus on your own in AKS however it is beter to use Azure managed prometheus with AKS as it would leverage the capabilities of Azure monitor and Azure managed grafana to better monitoring and alerting abilities and avoid additional work required to setup full monitoring and observability on your own in an AKS cluster. The offcial documentation is here for leaning more information on the setup.

Setting Up Azure Storage Blob Backups with Azure Backup Vault Using Terraform

 To create automated daily backups of Azure storage account blobs we can use Azure Backup Vault resource. It is possible to configure operational backups and vaulted backups for Azure storage blobs with the backup vault backup policies and instances. We need to consider the limitations specified for operational backups and the limitations in vaulted backups, specially the limitation of only 100 blob containers are allowed to be backed up for a given Azure storage account. Another issue with vaulted backups is the newly created containers in an Azure storage will not automatically included in the backups. Therefore, the Azure vaulted backups are working well only when you have predefoined set of blob containers in your implementation, that are configured for vaulted backups. 

Setup Azure File Share Capacity Alert to Slack with Terraform

 Setting up an Azure File Share capacity alert is useful to know when you reach at least 80% of allocated quota for the file share. This will give the teams ample time to increase the allocation to avoid out of space issues. If we are using standard tier for storage account then we need to use one storage account for each file share, to get the correct alert. Sending the alert to slack channel is a useful way to get properly alerted to take action on time. Let's use an example learn how to setup alerts for multiple Azure file shares uing terraform.

Expectation is to get the alerts to slack channel as shown below.

Use Dynamic Block Conditionally in Terraform with a Nested Block

 We have discussed  usage of dynamic blocks conditionally in terraform in the post "Use Dynamic Block Conditionally in Terraform". Sometimes we need to use dynamic with nested blocks. In this post look at such an example of using dynamic block with network rules for Azure event hubs.

We are going to setup a nested dynamic block as shown below.

Disable Soft Delete for Azure File Shares with Terraform

 Azure file shares can be used as mounted persisted volume in AKS clusters to provide necessary storage to pods in AKS. Azure file shares are enabled with soft delete for files by default. However, when the pod mounted storage is used as temporary storage for processing, there is no need to keep the capability of soft deletion. Let's look at how we can disable soft delete for file share storage when deploying the file share with terraform.

Resolving "ERROR: Account has previously been signed out of this application.. Status: Response_Status.Status_AccountUnusable" When Terraform Plan

 While trying to execute terraform plan locally after an az login we sometimes envounter below error.

│ Error: populating Resource Provider cache: listing Resource Providers: loading results: authorizing request: running Azure CLI: exit status 1: ERROR: Account has previously been signed out of this application.. Status: Response_Status.Status_AccountUnusable, Error code: 0, Tag: 540940121

│ Please explicitly log in with:│ az login --scope https://management.azure.com/.default

│ with provider["registry.terraform.io/hashicorp/azurerm"],│   on main.tf line 17, in provider "azurerm":│   17: provider "azurerm" {

Use Dynamic Block Conditionally in Terraform

 Dynamic block allows to create nested multi level block structures in terraform code. Conditional usage of such blocks are really useful in many scenarios. For example, when we create Azure Cosmos DB, if we want to have read regions only in production environment, but only have one region setup for dev and QA environments we can, leverage the capabilities in Dynamic block of terraform. Let's explore with an example.

Deploying Azure Managed Grafana with Terraform

 Grafana can be used to setup monitoring and alerting with AKS. Azure provide an option to setup managed grafana dashboard, which can be integrated with managed protheus for AKS. In this post let's explore terraform code to setup managed grafana instance similar to below.

Deploying Kubernetes Event Driven Autoscaling (KEDA) AKS add-on with Terraform and Azure Pipelines

 We have discussed deploying Kubernetes Event Drivern Autoscaling (KEDA) with workload identity in AKS in the post "Setting Up Kubernetes Event Drivern Autoscaling (KEDA) in AKS with Workload Identity". Then we discussed how to use an Azure DevOps pipeline to automate deployment of KEDA in the post "Deploying Kubernetes Event Drivern Autoscaling (KEDA) with Azure Pipelines Using Helm". If you are setting up KEDA with helm as discussed instead of using AKS KEDA add-on you will have to monitor the  documentation here and ensure supported version is used. However, with AKS it is better to use Microsoft supported KEDA add-on as it will be having better support in case of an issue. Additionally it will be the correct supported version of KEDA getting setup with add-on, based on AKS kubernetes version used according to the documentation here. Let's see what we need to do in terraform and in Azure piplines to get AKS setup with KEDA add-on.