Create AKS Cluster to Setup Valkey Cluster (Open Source Redis)

 Since bitnami redis images have gone commercial now the best open source redis cluster setup option is to use Valkey, which is guranteed to be open source forever. The source code repos for Valkey is here. We have discussed deploying redis cluster on AKS with bitnami in "Setup Redis Cluster with JSON and Search Modules on AKS with Binami Redis Using Custom Image" However, if we want to deploy Valkey/Redis in cluster in AKS and want to access the Valkey/Redis from outside the AKS cluster, we have to setup AKS networking in a way that each pod is allocated with an IP from the Azure virtual network subnet. In this post let's explore how to setup such an AKS cluster correctly.

Why we need pods in AKS to have IPs from virtual network subnet to deploy Valkey?

When we connect to a Valkey/Redis cluster it always provide diffrent IPs of master nodes to client. Therefore, clients need to acess to IPs of Valkey node pods.

We cannot use Azure CNI overlay for this AKS cluster as it would setup internal network CIDR for pod IPs. Such IPs are not visible outside of AKS. Therefore it is not possible use Azure CNI overlay for AKS clsuter, if we are to deploy Valkey clsuter that should be allowed to access outside of AKS.

We can Azure node subnet mode. But this is not ideal as we have to have a large subnet planned to support boith the node needs as well as pods IP requirement.

The best option is to use Azure CNI Pod Subnet, which would be really useful for this need.

Mount Per Pod Managed Disk to AKS Pod to Provide Isolated Disk Space for Processing Large Files

 In AKS processing files on AKS node disk by application pod is not a recommended approach. It could lead to issues in stability of the AKS node itself. Therefore, we have two options. Either mount an Azure fileshare to the pod as described in "Mount Azure Storage Fileshare Created with Terraform on AKS". The other option is to setup a per pod disk, specially when the need is to process larger files. This option is really useful when we have pods running a single job and when it lives for the lifetime of the job. Example case is use of KEDA scale object with per single message processing from rabbitmq as described in "RabbitMQ KEDA Trigger for AKS Deployed Apps to Scale Out the Apps Accurately".

Expentation is to get a per pod disk attached as shown below.

Fix Keda Upgrade on AKS with Helm Fail Due to Field Ownership Conflicts

 AKS runs an internal mutating component called the admissionsenforcer (part of the AKS-managed "admissions enforcer" / addon-manager). After KEDA's ValidatingWebhookConfiguration (keda-admission) is created, AKS automatically injects a namespaceSelector into every webhook entry to exclude AKS control-plane / managed namespaces (so KEDA's webhooks never intercept system pods). When it does this, it takes Server-Side Apply (SSA) field ownership of those namespaceSelector fields under the field manager named admissionsenforcer. 

So after the first deploy:

• keda-admission webhook exists
• AKS has mutated .webhooks[*].namespaceSelector and now owns those fields

In the second deploy, the KEDA chart  tries to re-apply the same ValidatingWebhookConfiguration with the field manager helm. SSA detects that helm wants to set fields already owned by admissionsenforcer → field ownership conflict:

Update Deployment Resource Allocation in AKS with Kubectl

 Sometimes we might have to update resource allocations (memory and CPU) in AKS deployed apps without full redeployment. In such cases we can easly use kubectl to patch the deployment so it increase or reduce resources as we need it and gradually apply to running pods by restartting them gracefully. Let's see how.

Handle Nginx Gateway Certificate Refresh While Inplace Upgrade in AKS

 In the post "High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 2 - Deploy Nginx-Gateway-Fabric" we hav discussed how to get nginx gateway setup in AKS. This approach works fine for the first install and if you are using true blue green with a fresh AKS cluster. However, when we use componenets such as elastic search  on AKS (which we will discuss in future posts how to setup elastc search on AKS) ,we have to use inplace AKS upgrades, with new node pools in same cluster, as we want to persist the data on elastic. In such inplace AKS upgrade requirements we will have to upgrade cert manager and nginx gateway as well inplace. When we try to do such upgrades to cert-manager and nginx gateway we are running into a issue as decribed below.

The Issue

Immediately after the upgrade or after a time interval, the dataplane pods of nginx gteway will run into a high CPU situation and will try to create pods. These pods will not be able to start properly as it they will not be able to validate the certificates generated. Ideally this situation should have been handled by the control plane (operator) of niginx gateway. however, it does not do that properly.

Adding SnippetsFIlter to HTTPRoute in Nginx Gateway to Implement Proxy Annotations of Retired Ingress Nginx

 We have discussed "Setup Nginx-Gateway Routes and Create Gateway Policies to Map with Ingress Nginx Annotations" in a previous post. However, we have not setup all that were previously set via ingress-nginx annotations, and only partial setup was made using ClientSettingsPolicy and UpstreamSettingsPolicy. Some proxy annotations that we used with ingress-nginx can only be setup by using a SnippetsFilter. 

Such SnippetsFilter is shown below for example.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 5 - Cleanup Ingress-Nginx from the AKS Cluster

 We have switched traffic from ingress-nginx to nginx-gateway as discussed in the previous blog. The next step is to gent ingress-nginx setup cleaned up from AKS env. This can be done manually. However, when there are mutiple production regions deployed with same deployment, it is easier if we programatically remove all ingress-nginx related routing and setupp, so it can be deployed to all production environments, same way, without issues.

The expectation is to get ingrss nginx removed, and the namespace ingress-nginx also cleaned up from the AKS cluster. Therefore, we can remove the namespace creation yaml as shown below.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 4 - Switch Traffic from Ingress-Nginx to Nginx-Gateway

 In the part 2 and part 3 of this blog series, we have deployed nginx-gateway in AKS, with required route and policy setup to correctly route to elastic search and kibana deployed in AKS. But we are still have live traffic to elastic search and kibana via the retired ingress-nginx, after the deployment of nginx-gateway also in the same cluster. No as we planned we are going to do the release 2 to get the traffic switch from retired ingress-nginx to new nginx-gateway deployed in AKS.

The expectation is to have kibana and elastic host names to resolve to private IP of the nginx-gateway loadbalancer as shown below, and the ability to use elastic search with the url, using nginx-gateway.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 3 - Setup Nginx-Gateway Routes and Create Gateway Policies to Map with Ingress Nginx Annotations

We have discussed how to deploy nginx-gateway in part 2 of this blog series.  As the nginx-gateway is deployed and ready now, we can start setting up HTTP routes, so that it is ready to replace the ingress-nginx (retired) routes already working in the AKS cluster. The example in this blog uses, routes to AKS hosted elastic search, so there are two routes, the kibana dashboard route and the elastic search route used by the apps. The currrent routes use a private DNS, and a private IP within Azure vNET. We have used a new private IP for nginx gateway in deploy nginx-gateway in part 2 , so that current retired  ingress-nginx internal loadbalancer is not affected, while we setup nginx-gateway (uses another private IP for loadbalancer of gateway), making them both available in AKS cluster. Therefore, setting up same host names for gateway routes will not cause issues in existing ingress-nginx setup. After this step of setting up routes, we will have both ingress-nginx (retired) and nginx-gateway with routes, working in same cluster, but live traffic will be still using ingress-nginx.

The expectation is to have routes using nginx-gateway is setup for ealstic search and kibana as shown below.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 2 - Deploy Nginx-Gateway-Fabric

In part 1 "High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 1 - Plan for Smooth Transition", we have discussed the plan to transtion from retired ingess-nginx to nginx-gateway, for an AKS cluster where we have hosted elatic search.  In this post let's look at steps necessary to deploy nginx-gateway.

The expection is to have successfully deployed high available nginx-gateway with nginx gateway fabric.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 1 - Plan for Smooth Transition

 We have discussed setting up ingress-nginx as ingress controller for AKS previously. Ingress-nginx for kubernetes is retiring in March 2026 as announced here in github and in kubernetes documentation. Gateway API is the recommended replacement for ingress-nginx and Nginx Gateway Fabric is an implementation of Gateway API by Nginx. When you have true blue-green deployments with new AKS cluster getting created at each deployment, transitioning from ingress-nginx to nginx gateway fabric is relatively simpler, and we can discuss that in the last blog of this blob post series. However, when you have a single AKS cluster (situations such as hosting elastic search on AKS demands, we use only a single AKS cluster - we will discuss this topic in a later blog series), where blue-green deployments happen using node pools and cluster upgrades as inplace upgrades, the transtion needs to be done over multiple releases to achive zero down time deployments for production. In this blog, let's plan how we can get the transition done in a step by step manner.

The expectation is to get nginx-gateway setup as shown below, so that HTTP routes can be setup via the gateway.

Use Terraform to Switch Azure Key Vault to Use RBAC Permisions from Access Policies Without a Downtime for Applications/Users

 For Azure Key Vaults access policies based permision setup is now legacy and all key vaults will have to use Azure RBAC permisions eventually for data access permisions according to offcial Microsft documentation here. Using terraform we can setup the changes. However, we have to be carefull about the switching to RBAC from access policies in production scenarios to avoid interptions to applications. Taking two step approach, first set RBAC permisions and in a next release performing switch to RBAC for key vault will help the transtion to be smooth. Let's look at how to setup this requirement with terraform.

The expectation is to have a keyvault setup with RBAC permisions as shown below.

 

Using Remote Terraform State

 Sometimes resources common to multiple diffrent setups might need to be created with a common terraform code. In such cases the commeo terraform resources may need to be reffered with its state in  other terraform code. For this requirement we can use terraform remote state. Let's see how we can use terraform remote state step by step in this post.

The expectation is to refer to the Azure resources in remote terraform state as shown below. Here you can see we have reffered to the resource group name and location, and to log analytics workspace id from remote state.

Visualize Dead Letter Counts in RabbitMQ Deployed in AKS

 Using the prometheus data obtained by "Enabling Prometheus Data Scraping for RabbitMQ Cluster Deployed with RabbitMQ Cluster Operator on AKS with Managed Prometheus", let's create grafana chart to view any messages land in dead letter queues in the RabbitMQ cluster deployed in AKS.

The expection is to have a chart as shown below.

Enable Prometheus Data Scraping for RabbitMQ Cluster Deployed with RabbitMQ Cluster Operator on AKS with Managed Prometheus

 Once we have "Setup Managed Prometheus for AKS via Terraform" and  "Set Up RabbitMQ Cluster in AKS Using RabbitMQ Cluster Operator", we can enable monitoring for RabbitMQ in AKS. To enable Prometheus data scraping for RabbitMQ cluster on AKS, we need to deploy a service monitor. Additionally, we can deploy a pod monitor as well to scrape metrics from the RabbitMQ clsuter operator. When data scraping enabled, we would be able to get the metrics data for RabbitMQ as shown below, on Azure managed grafana using Azure managed prometheus on AKS as the data source, via an Azure monitoring workspace.

Enable Windows Data Scraping for AKS Managed Prometheus with Azure Managed Grafana

 We have "Setup Managed Prometheus for AKS via Terraform", however, that setup alone will not provide windows metrics from AKS clsuter to Azure managed Grafana. We have to addtionaly, setup Windows exporter and couple of additional configurations to make it work as decribed in the official Microsoft docs here. Let's look at step by step how to enable windows metrics for AKS with managed prometheus.

Expected outcome is getting metrics such as shown below to managed grafana and visualizing them.

Using Managed Prometheus Data to Setup Monitoring for AKS Deployed Bitnami Redis Cluster or Standalone

 Based on "Enable Prometheus Data Scraping for Bitnami Redis Cluster Deployed on AKS with Managed Prometheus" and "Enable Prometheus Data Scraping for Bitnami Redis Standalone Deployed on AKS with Managed Prometheus" discussed in the previous posts, let's look at using the managed prometheus data source in Azure Manged Grafana and how to setup monitoring for AKS deployed Bitnami Redis, with few examples.

The expectation is to have charts as shown below.

Enable Prometheus Data Scraping for Bitnami Redis Standalone Deployed on AKS with Managed Prometheus

 We have discussed "Enable Prometheus Data Scraping for Bitnami Redis Cluster Deployed on AKS with Managed Prometheus" in the previous post. Similar way we can setup Prometheus data scraping for standalone redis deployments on AKS as well. Here are the steps.

The expectaion is to have the metrics sidecar run with the standalone redis master and replica pods as shown below and setup a service monitor so that redis metrics data is made available in Azure managed grafana via managed prometheus on AKS. 

Enable Prometheus Data Scraping for Bitnami Redis Cluster Deployed on AKS with Managed Prometheus

 "Setup Redis Cluster with JSON and Search Modules on AKS with Binami Redis Using Custom Image" and "Setup Managed Prometheus for AKS via Terraform" are explained in the previous posts. To enable setting up monitoring and alerting for the Redis Custer deployed in AKS the first step is to enable Prometheus data scraping in the Redis cluster we deployed on AKS. Let's look at the steps in this post.

The expectation is to have redis metrices available to Azure managed grafana via the managed prometheus in AKS as shown below.

Setup Managed Prometheus for AKS via Terraform

 We have discussed "Deploying Azure Managed Grafana with Terraform" ealier. Now that we have managed grafana available in Azure for the AKS clsuters we can enable managed prometheus. You can setup prometheus on your own in AKS however it is beter to use Azure managed prometheus with AKS as it would leverage the capabilities of Azure monitor and Azure managed grafana to better monitoring and alerting abilities and avoid additional work required to setup full monitoring and observability on your own in an AKS cluster. The offcial documentation is here for leaning more information on the setup.