Get Pod Counts via Kubectl in WSL

 It would be usefull to figure out how many pods running in a given AKS cluster sometimes for various diagnostic needs. In this post let's see how we can use kubectl command to get pod count in diffrent combinations.

Below are some examples.

Create AKS Cluster to Host Valkey Cluster (Open Source Redis)

 Since bitnami redis images have gone commercial now the best open source redis clsuter setup option is to use Valkey, which is guranteed to be open source forever. The source code repos for Valkey is here. We have discussed deploying redis cluster on AKS with bitnami in "Setup Redis Cluster with JSON and Search Modules on AKS with Binami Redis Using Custom Image" However, if we want to deploy Valkey/Redis in cluster in AKS and want to access the Valkey/Redis from outside the AKS cluster, we have to setup AKS networking in a way that each pod is allocated with an IP from the Azure virtual network subnet. In this post let's explore how to setup such an AKS cluster correctly.

Why we need pods in AKS to have IPs from virtual network subnet to deploy Valkey?

When we connect to a Valkey/Redis cluster it always provide diffrent IPs of master nodes to client. Therefore, clients need to acess to IPs of Valkey node pods.

We cannot use Azure CNI overlay for this AKS cluster as it would setup internal network CIDR for pod IPs. Such IPs are not visible outside of AKS. Therefore it is not possible use Azure CNI overlay for AKS clsuter, if we are to deploy Valkey clsuter that should be allowed to access outside of AKS.

We can Azure node subnet mode. But this is not ideal as we have to have a large subnet planned to support boith the node needs as well as pods IP requirement.

The best option is to use Azure CNI Pod Subnet, which would be really useful for this need.

Mount Per Pod Managed Disk to AKS Pod to Provide Isolated Disk Space for Processing Large Files

 In AKS processing files on AKS node disk by application pod is not a recommended approach. It could lead to issues in stability of the AKS node itself. Therefore, we have two options. Either mount an Azure fileshare to the pod as described in "Mount Azure Storage Fileshare Created with Terraform on AKS". The other option is to setup a per pod disk, specially when the need is to process larger files. This option is really useful when we have pods running a single job and when it lives for the lifetime of the job. Example case is use of KEDA scale object with per single message processing from rabbitmq as described in "RabbitMQ KEDA Trigger for AKS Deployed Apps to Scale Out the Apps Accurately".

Expentation is to get a per pod disk attached as shown below.

RabbitMQ KEDA Trigger for AKS Deployed Apps to Scale Out the Apps Accurately

 We have discussed "Setting Up RabbitMQ Cluster in AKS Using RabbitMQ Cluster Operator" in a previous post. Baed on rabbitmq messages we may want to setup a trigger to auto scale our apps to handle the workloads similar to what we have done with eventhubs, azure service bus etc, as described in below posts.

• Setting Up (KEDA) Authentication Trigger for Azure Storage Queue/Service Bus in AKS
• Scale Pods in AKS with Kubernetes Event Driven Autoscaling (KEDA) ScaledJob Based on Azure Service Bus Queue as a Trigger
• Multiple KEDA Triggers for a Scaled Job with Event Hubs in AKS

In this post, let's look at usage of keda scale objects and how we can setup to trigger scaling with rabbit mq messages accurately for AKS deployed apps. The expectation is to scale out apps based on number of messgaes in rabbitmq queues and effective process the the work, while scaling in when there is not much workload on the system.

Depending on above message loads the consumer apps scale out and in accrdingly.

Fix Keda Upgrade on AKS with Helm Fail Due to Field Ownership Conflicts

 AKS runs an internal mutating component called the admissionsenforcer (part of the AKS-managed "admissions enforcer" / addon-manager). After KEDA's ValidatingWebhookConfiguration (keda-admission) is created, AKS automatically injects a namespaceSelector into every webhook entry to exclude AKS control-plane / managed namespaces (so KEDA's webhooks never intercept system pods). When it does this, it takes Server-Side Apply (SSA) field ownership of those namespaceSelector fields under the field manager named admissionsenforcer. 

So after the first deploy:

• keda-admission webhook exists
• AKS has mutated .webhooks[*].namespaceSelector and now owns those fields

In the second deploy, the KEDA chart  tries to re-apply the same ValidatingWebhookConfiguration with the field manager helm. SSA detects that helm wants to set fields already owned by admissionsenforcer → field ownership conflict:

Monitor and Profile Memory Usage in a Running .NET Linux Pod in AKS

 Sometimes we need to understand what causes high memeory usages or if any memory leaks in our application setup. When we see pod memeory is not reclaimed for longer intervals, analyzing the usage and diagnosing where exactly is the problem would be useful to fix the issues. Such profiling of memeory, we should be able to do in a running container inside a pod to understand the full details. Let's look at step by step how to profile and monitor memory usage in a .NET app running in a Linux pod in AKS.

Expectaion is to have the ability to monitor live counters as shown below.

Update Deployment Resource Allocation in AKS with Kubectl

 Sometimes we might have to update resource allocations (memory and CPU) in AKS deployed apps without full redeployment. In such cases we can easly use kubectl to patch the deployment so it increase or reduce resources as we need it and gradually apply to running pods by restartting them gracefully. Let's see how.

Handle Nginx Gateway Certificate Refresh While Inplace Upgrade in AKS

 In the post "High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 2 - Deploy Nginx-Gateway-Fabric" we hav discussed how to get nginx gateway setup in AKS. This approach works fine for the first install and if you are using true blue green with a fresh AKS cluster. However, when we use componenets such as elastic search  on AKS (which we will discuss in future posts how to setup elastc search on AKS) ,we have to use inplace AKS upgrades, with new node pools in same cluster, as we want to persist the data on elastic. In such inplace AKS upgrade requirements we will have to upgrade cert manager and nginx gateway as well inplace. When we try to do such upgrades to cert-manager and nginx gateway we are running into a issue as decribed below.

The Issue

Immediately after the upgrade or after a time interval, the dataplane pods of nginx gteway will run into a high CPU situation and will try to create pods. These pods will not be able to start properly as it they will not be able to validate the certificates generated. Ideally this situation should have been handled by the control plane (operator) of niginx gateway. however, it does not do that properly.

Adding SnippetsFIlter to HTTPRoute in Nginx Gateway to Implement Proxy Annotations of Retired Ingress Nginx

 We have discussed "Setup Nginx-Gateway Routes and Create Gateway Policies to Map with Ingress Nginx Annotations" in a previous post. However, we have not setup all that were previously set via ingress-nginx annotations, and only partial setup was made using ClientSettingsPolicy and UpstreamSettingsPolicy. Some proxy annotations that we used with ingress-nginx can only be setup by using a SnippetsFilter. 

Such SnippetsFilter is shown below for example.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 5 - Cleanup Ingress-Nginx from the AKS Cluster

 We have switched traffic from ingress-nginx to nginx-gateway as discussed in the previous blog. The next step is to gent ingress-nginx setup cleaned up from AKS env. This can be done manually. However, when there are mutiple production regions deployed with same deployment, it is easier if we programatically remove all ingress-nginx related routing and setupp, so it can be deployed to all production environments, same way, without issues.

The expectation is to get ingrss nginx removed, and the namespace ingress-nginx also cleaned up from the AKS cluster. Therefore, we can remove the namespace creation yaml as shown below.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 4 - Switch Traffic from Ingress-Nginx to Nginx-Gateway

 In the part 2 and part 3 of this blog series, we have deployed nginx-gateway in AKS, with required route and policy setup to correctly route to elastic search and kibana deployed in AKS. But we are still have live traffic to elastic search and kibana via the retired ingress-nginx, after the deployment of nginx-gateway also in the same cluster. No as we planned we are going to do the release 2 to get the traffic switch from retired ingress-nginx to new nginx-gateway deployed in AKS.

The expectation is to have kibana and elastic host names to resolve to private IP of the nginx-gateway loadbalancer as shown below, and the ability to use elastic search with the url, using nginx-gateway.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 3 - Setup Nginx-Gateway Routes and Create Gateway Policies to Map with Ingress Nginx Annotations

We have discussed how to deploy nginx-gateway in part 2 of this blog series.  As the nginx-gateway is deployed and ready now, we can start setting up HTTP routes, so that it is ready to replace the ingress-nginx (retired) routes already working in the AKS cluster. The example in this blog uses, routes to AKS hosted elastic search, so there are two routes, the kibana dashboard route and the elastic search route used by the apps. The currrent routes use a private DNS, and a private IP within Azure vNET. We have used a new private IP for nginx gateway in deploy nginx-gateway in part 2 , so that current retired  ingress-nginx internal loadbalancer is not affected, while we setup nginx-gateway (uses another private IP for loadbalancer of gateway), making them both available in AKS cluster. Therefore, setting up same host names for gateway routes will not cause issues in existing ingress-nginx setup. After this step of setting up routes, we will have both ingress-nginx (retired) and nginx-gateway with routes, working in same cluster, but live traffic will be still using ingress-nginx.

The expectation is to have routes using nginx-gateway is setup for ealstic search and kibana as shown below.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 2 - Deploy Nginx-Gateway-Fabric

In part 1 "High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 1 - Plan for Smooth Transition", we have discussed the plan to transtion from retired ingess-nginx to nginx-gateway, for an AKS cluster where we have hosted elatic search.  In this post let's look at steps necessary to deploy nginx-gateway.

The expection is to have successfully deployed high available nginx-gateway with nginx gateway fabric.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 1 - Plan for Smooth Transition

 We have discussed setting up ingress-nginx as ingress controller for AKS previously. Ingress-nginx for kubernetes is retiring in March 2026 as announced here in github and in kubernetes documentation. Gateway API is the recommended replacement for ingress-nginx and Nginx Gateway Fabric is an implementation of Gateway API by Nginx. When you have true blue-green deployments with new AKS cluster getting created at each deployment, transitioning from ingress-nginx to nginx gateway fabric is relatively simpler, and we can discuss that in the last blog of this blob post series. However, when you have a single AKS cluster (situations such as hosting elastic search on AKS demands, we use only a single AKS cluster - we will discuss this topic in a later blog series), where blue-green deployments happen using node pools and cluster upgrades as inplace upgrades, the transtion needs to be done over multiple releases to achive zero down time deployments for production. In this blog, let's plan how we can get the transition done in a step by step manner.

The expectation is to get nginx-gateway setup as shown below, so that HTTP routes can be setup via the gateway.

AKS Pod Memory Usage Percentage Chart Using Managed Prometheus Data

 In previous post we have discussed "Setup Managed Prometheus for AKS via Terraform" and "Enable Windows Data Scraping for AKS Managed Prometheus with Azure Managed Grafana". Understanding memory usage % for each application pod running in AKS, and setup alerts for higher uasge is useful to identify memory allocation requirements or any memory leaks in applications. Let's look at how to setup a chart in Azure Managed Grafana using AKS managed prometheus data.

The expectation is to get a chart as shown below.

Access Private Url within Azure vNet with Azure Pipeline Microsoft Hosted Pipeline Agent via AKS pod in the vNet

If we are using Microsoft hosted agents for Azure pipelines to deploy Azure infrastucture and need to access vNet protected urls of services deployed, we can use a pod in AKS cluster within same vNet, as a jump host. This gives us access endpoints in vNet and ability to resolve DNS defined in private DNS zones of the vNet. Let's look at staep by step how to achive this goal, while using a Microsoft hosted agent in Azure pipelines.

The expectation is to access url such as 
http://es-search.sh.aks.ch-demo-dev-euw-002.net/demoindex001/_count  so AKS hosted elastic seach is accessed via a AKS pod and get the results to the pipeline agent as shown below. Since microsoft hosted agent is outside the vNET it cannot directly reach this elastc search (deployed in AKS) url.

Visualize Dead Letter Counts in RabbitMQ Deployed in AKS

 Using the prometheus data obtained by "Enabling Prometheus Data Scraping for RabbitMQ Cluster Deployed with RabbitMQ Cluster Operator on AKS with Managed Prometheus", let's create grafana chart to view any messages land in dead letter queues in the RabbitMQ cluster deployed in AKS.

The expection is to have a chart as shown below.

Enable Prometheus Data Scraping for RabbitMQ Cluster Deployed with RabbitMQ Cluster Operator on AKS with Managed Prometheus

 Once we have "Setup Managed Prometheus for AKS via Terraform" and  "Set Up RabbitMQ Cluster in AKS Using RabbitMQ Cluster Operator", we can enable monitoring for RabbitMQ in AKS. To enable Prometheus data scraping for RabbitMQ cluster on AKS, we need to deploy a service monitor. Additionally, we can deploy a pod monitor as well to scrape metrics from the RabbitMQ clsuter operator. When data scraping enabled, we would be able to get the metrics data for RabbitMQ as shown below, on Azure managed grafana using Azure managed prometheus on AKS as the data source, via an Azure monitoring workspace.

Using Managed Prometheus Data to Setup Monitoring for AKS Deployed Bitnami Redis Cluster or Standalone

 Based on "Enable Prometheus Data Scraping for Bitnami Redis Cluster Deployed on AKS with Managed Prometheus" and "Enable Prometheus Data Scraping for Bitnami Redis Standalone Deployed on AKS with Managed Prometheus" discussed in the previous posts, let's look at using the managed prometheus data source in Azure Manged Grafana and how to setup monitoring for AKS deployed Bitnami Redis, with few examples.

The expectation is to have charts as shown below.

Enable Prometheus Data Scraping for Bitnami Redis Standalone Deployed on AKS with Managed Prometheus

 We have discussed "Enable Prometheus Data Scraping for Bitnami Redis Cluster Deployed on AKS with Managed Prometheus" in the previous post. Similar way we can setup Prometheus data scraping for standalone redis deployments on AKS as well. Here are the steps.

The expectaion is to have the metrics sidecar run with the standalone redis master and replica pods as shown below and setup a service monitor so that redis metrics data is made available in Azure managed grafana via managed prometheus on AKS.