Handle Nginx Gateway Certificate Refresh While Inplace Upgrade in AKS

 In the post "High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 2 - Deploy Nginx-Gateway-Fabric" we hav discussed how to get nginx gateway setup in AKS. This approach works fine for the first install and if you are using true blue green with a fresh AKS cluster. However, when we use componenets such as elastic search  on AKS (which we will discuss in future posts how to setup elastc search on AKS) ,we have to use inplace AKS upgrades, with new node pools in same cluster, as we want to persist the data on elastic. In such inplace AKS upgrade requirements we will have to upgrade cert manager and nginx gateway as well inplace. When we try to do such upgrades to cert-manager and nginx gateway we are running into a issue as decribed below.

The Issue

Immediately after the upgrade or after a time interval, the dataplane pods of nginx gteway will run into a high CPU situation and will try to create pods. These pods will not be able to start properly as it they will not be able to validate the certificates generated. Ideally this situation should have been handled by the control plane (operator) of niginx gateway. however, it does not do that properly.

Adding SnippetsFIlter to HTTPRoute in Nginx Gateway to Implement Proxy Annotations of Retired Ingress Nginx

 We have discussed "Setup Nginx-Gateway Routes and Create Gateway Policies to Map with Ingress Nginx Annotations" in a previous post. However, we have not setup all that were previously set via ingress-nginx annotations, and only partial setup was made using ClientSettingsPolicy and UpstreamSettingsPolicy. Some proxy annotations that we used with ingress-nginx can only be setup by using a SnippetsFilter. 

Such SnippetsFilter is shown below for example.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 5 - Cleanup Ingress-Nginx from the AKS Cluster

 We have switched traffic from ingress-nginx to nginx-gateway as discussed in the previous blog. The next step is to gent ingress-nginx setup cleaned up from AKS env. This can be done manually. However, when there are mutiple production regions deployed with same deployment, it is easier if we programatically remove all ingress-nginx related routing and setupp, so it can be deployed to all production environments, same way, without issues.

The expectation is to get ingrss nginx removed, and the namespace ingress-nginx also cleaned up from the AKS cluster. Therefore, we can remove the namespace creation yaml as shown below.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 4 - Switch Traffic from Ingress-Nginx to Nginx-Gateway

 In the part 2 and part 3 of this blog series, we have deployed nginx-gateway in AKS, with required route and policy setup to correctly route to elastic search and kibana deployed in AKS. But we are still have live traffic to elastic search and kibana via the retired ingress-nginx, after the deployment of nginx-gateway also in the same cluster. No as we planned we are going to do the release 2 to get the traffic switch from retired ingress-nginx to new nginx-gateway deployed in AKS.

The expectation is to have kibana and elastic host names to resolve to private IP of the nginx-gateway loadbalancer as shown below, and the ability to use elastic search with the url, using nginx-gateway.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 3 - Setup Nginx-Gateway Routes and Create Gateway Policies to Map with Ingress Nginx Annotations

We have discussed how to deploy nginx-gateway in part 2 of this blog series.  As the nginx-gateway is deployed and ready now, we can start setting up HTTP routes, so that it is ready to replace the ingress-nginx (retired) routes already working in the AKS cluster. The example in this blog uses, routes to AKS hosted elastic search, so there are two routes, the kibana dashboard route and the elastic search route used by the apps. The currrent routes use a private DNS, and a private IP within Azure vNET. We have used a new private IP for nginx gateway in deploy nginx-gateway in part 2 , so that current retired  ingress-nginx internal loadbalancer is not affected, while we setup nginx-gateway (uses another private IP for loadbalancer of gateway), making them both available in AKS cluster. Therefore, setting up same host names for gateway routes will not cause issues in existing ingress-nginx setup. After this step of setting up routes, we will have both ingress-nginx (retired) and nginx-gateway with routes, working in same cluster, but live traffic will be still using ingress-nginx.

The expectation is to have routes using nginx-gateway is setup for ealstic search and kibana as shown below.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 2 - Deploy Nginx-Gateway-Fabric

In part 1 "High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 1 - Plan for Smooth Transition", we have discussed the plan to transtion from retired ingess-nginx to nginx-gateway, for an AKS cluster where we have hosted elatic search.  In this post let's look at steps necessary to deploy nginx-gateway.

The expection is to have successfully deployed high available nginx-gateway with nginx gateway fabric.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 1 - Plan for Smooth Transition

 We have discussed setting up ingress-nginx as ingress controller for AKS previously. Ingress-nginx for kubernetes is retiring in March 2026 as announced here in github and in kubernetes documentation. Gateway API is the recommended replacement for ingress-nginx and Nginx Gateway Fabric is an implementation of Gateway API by Nginx. When you have true blue-green deployments with new AKS cluster getting created at each deployment, transitioning from ingress-nginx to nginx gateway fabric is relatively simpler, and we can discuss that in the last blog of this blob post series. However, when you have a single AKS cluster (situations such as hosting elastic search on AKS demands, we use only a single AKS cluster - we will discuss this topic in a later blog series), where blue-green deployments happen using node pools and cluster upgrades as inplace upgrades, the transtion needs to be done over multiple releases to achive zero down time deployments for production. In this blog, let's plan how we can get the transition done in a step by step manner.

The expectation is to get nginx-gateway setup as shown below, so that HTTP routes can be setup via the gateway.

AKS Pod Memory Usage Percentage Chart Using Managed Prometheus Data

 In previous post we have discussed "Setup Managed Prometheus for AKS via Terraform" and "Enable Windows Data Scraping for AKS Managed Prometheus with Azure Managed Grafana". Understanding memory usage % for each application pod running in AKS, and setup alerts for higher uasge is useful to identify memory allocation requirements or any memory leaks in applications. Let's look at how to setup a chart in Azure Managed Grafana using AKS managed prometheus data.

The expectation is to get a chart as shown below.