Handle Nginx Gateway Certificate Refresh While Inplace Upgrade in AKS

 In the post "High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 2 - Deploy Nginx-Gateway-Fabric" we hav discussed how to get nginx gateway setup in AKS. This approach works fine for the first install and if you are using true blue green with a fresh AKS cluster. However, when we use componenets such as elastic search  on AKS (which we will discuss in future posts how to setup elastc search on AKS) ,we have to use inplace AKS upgrades, with new node pools in same cluster, as we want to persist the data on elastic. In such inplace AKS upgrade requirements we will have to upgrade cert manager and nginx gateway as well inplace. When we try to do such upgrades to cert-manager and nginx gateway we are running into a issue as decribed below.

The Issue

Immediately after the upgrade or after a time interval, the dataplane pods of nginx gteway will run into a high CPU situation and will try to create pods. These pods will not be able to start properly as it they will not be able to validate the certificates generated. Ideally this situation should have been handled by the control plane (operator) of niginx gateway. however, it does not do that properly.

Adding SnippetsFIlter to HTTPRoute in Nginx Gateway to Implement Proxy Annotations of Retired Ingress Nginx

 We have discussed "Setup Nginx-Gateway Routes and Create Gateway Policies to Map with Ingress Nginx Annotations" in a previous post. However, we have not setup all that were previously set via ingress-nginx annotations, and only partial setup was made using ClientSettingsPolicy and UpstreamSettingsPolicy. Some proxy annotations that we used with ingress-nginx can only be setup by using a SnippetsFilter. 

Such SnippetsFilter is shown below for example.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 5 - Cleanup Ingress-Nginx from the AKS Cluster

 We have switched traffic from ingress-nginx to nginx-gateway as discussed in the previous blog. The next step is to gent ingress-nginx setup cleaned up from AKS env. This can be done manually. However, when there are mutiple production regions deployed with same deployment, it is easier if we programatically remove all ingress-nginx related routing and setupp, so it can be deployed to all production environments, same way, without issues.

The expectation is to get ingrss nginx removed, and the namespace ingress-nginx also cleaned up from the AKS cluster. Therefore, we can remove the namespace creation yaml as shown below.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 4 - Switch Traffic from Ingress-Nginx to Nginx-Gateway

 In the part 2 and part 3 of this blog series, we have deployed nginx-gateway in AKS, with required route and policy setup to correctly route to elastic search and kibana deployed in AKS. But we are still have live traffic to elastic search and kibana via the retired ingress-nginx, after the deployment of nginx-gateway also in the same cluster. No as we planned we are going to do the release 2 to get the traffic switch from retired ingress-nginx to new nginx-gateway deployed in AKS.

The expectation is to have kibana and elastic host names to resolve to private IP of the nginx-gateway loadbalancer as shown below, and the ability to use elastic search with the url, using nginx-gateway.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 3 - Setup Nginx-Gateway Routes and Create Gateway Policies to Map with Ingress Nginx Annotations

We have discussed how to deploy nginx-gateway in part 2 of this blog series.  As the nginx-gateway is deployed and ready now, we can start setting up HTTP routes, so that it is ready to replace the ingress-nginx (retired) routes already working in the AKS cluster. The example in this blog uses, routes to AKS hosted elastic search, so there are two routes, the kibana dashboard route and the elastic search route used by the apps. The currrent routes use a private DNS, and a private IP within Azure vNET. We have used a new private IP for nginx gateway in deploy nginx-gateway in part 2 , so that current retired  ingress-nginx internal loadbalancer is not affected, while we setup nginx-gateway (uses another private IP for loadbalancer of gateway), making them both available in AKS cluster. Therefore, setting up same host names for gateway routes will not cause issues in existing ingress-nginx setup. After this step of setting up routes, we will have both ingress-nginx (retired) and nginx-gateway with routes, working in same cluster, but live traffic will be still using ingress-nginx.

The expectation is to have routes using nginx-gateway is setup for ealstic search and kibana as shown below.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 2 - Deploy Nginx-Gateway-Fabric

In part 1 "High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 1 - Plan for Smooth Transition", we have discussed the plan to transtion from retired ingess-nginx to nginx-gateway, for an AKS cluster where we have hosted elatic search.  In this post let's look at steps necessary to deploy nginx-gateway.

The expection is to have successfully deployed high available nginx-gateway with nginx gateway fabric.

High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 1 - Plan for Smooth Transition

 We have discussed setting up ingress-nginx as ingress controller for AKS previously. Ingress-nginx for kubernetes is retiring in March 2026 as announced here in github and in kubernetes documentation. Gateway API is the recommended replacement for ingress-nginx and Nginx Gateway Fabric is an implementation of Gateway API by Nginx. When you have true blue-green deployments with new AKS cluster getting created at each deployment, transitioning from ingress-nginx to nginx gateway fabric is relatively simpler, and we can discuss that in the last blog of this blob post series. However, when you have a single AKS cluster (situations such as hosting elastic search on AKS demands, we use only a single AKS cluster - we will discuss this topic in a later blog series), where blue-green deployments happen using node pools and cluster upgrades as inplace upgrades, the transtion needs to be done over multiple releases to achive zero down time deployments for production. In this blog, let's plan how we can get the transition done in a step by step manner.

The expectation is to get nginx-gateway setup as shown below, so that HTTP routes can be setup via the gateway.

Route to Management Dashboard Locally for a RabbitMQ Cluster Deployed in AKS via Azure App Gateway and Nginx Ingress Controller

We have discussed "Setting Up RabbitMQ Cluster in AKS Using RabbitMQ Cluster Operator" and how to expose dashboard with "Access Management Dashboard Locally for a RabbitMQ Cluster Deployed in AKS via Port Forwarding or Nginx Ingress Controller Inside VNet". Using the same Nginx route we can setup app gateway route rule so that the RabbitMQ dashboard can be accessed public url.

The expectation  is to route via App Gateway as shown below..

Access Management Dashboard Locally for a RabbitMQ Cluster Deployed in AKS via Port Forwarding or Nginx Ingress Controller Inside VNet

 We have discussed "Setting Up RabbitMQ Cluster in AKS Using RabbitMQ Cluster Operator" in a previous post. RabbitMQ management dashboard is a useful tool to have simple monitoring and inspecting the setup, connections etc. in the deployed RabbitMQ cluster (It is better to implement proper monitoring and alerting, which we will discuss in a next post). let' look at how to enable access to the dashboard of RabbitMQ deployed in the AKS cluster via port forwarding as well as setting up ingress via nginx.

Automate Health Check Validation for AKS Apps with Nginx Ingress Using Azure DevOps Pipelines

 We have discussed "Setup Application Ingress for AKS Using Nginx Ingress controller with Private IP" in a previous post. Once application and nginx ingress setup is deployed it takes some time for containers and pods to be ready for acepting traffic. We should validate whether the application contianers are deployed with correct docker image and running required replicas as specified in the horizontal pod autoscalers. Then we should verify if ingress for apps are setup corectly. If only all these health checks succeeed we can enable live traffic into a newly deployed set of applications in an AKS cluster (This is useful in blue-green deployments with new cluster or node pool to represent the blue or green instance). Let's use PowerShell and write health validation script and get it executed in Azure piplines to automate health check validation for apps deployed to AKS with nginx ingress.

Expected outecome is to validate ingress setup for application in an Azure DevOps pipelines task as below.

Setup Application Ingress for AKS Using Nginx Ingress controller with Private IP

 We have dicussed "Deploy Nginx Ingress Conroller with Private IP (Limited Access to vNET) to AKS with Terraform, Helm and Azure DevOps Pipelines" and then "Automate Validation of Nginx Ingress Controller Setup in AKS with an Azure Pipeline Task" is discussed. As the next step, let's explore how to setup ingress for application deployed in AKS, using Nginx ingress controller deployed with private IP. The purpose is exposoing the application in AKS within the virtual network, so, that other applications in the same virtual network can access the application securely, without having to expose application in AKS publicly.

The expected outcome is to have ingress for apps setup as shown below.

Automate Validation of Nginx Ingress Controller Setup in AKS with an Azure Pipeline Task

 We have dicussed "Deploy Nginx Ingress Conroller with Private IP (Limited Access to vNET) to AKS with Terraform, Helm and Azure DevOps Pipelines" in the previous post. Once deployed it takes few seconds to fewminutes to get the Nginx ingress controller with private IP ready in AKS. Let's explore how to automate AKS Nginx ingress controller validation inAzure pipelines in this post.

Deploy Nginx Ingress Conroller with Private IP (Limited Access to vNET) to AKS with Terraform, Helm and Azure DevOps Pipelines

We have dicussed "Create Azure CNI based AKS Cluster with Application Gateway Ingress Controller (AGIC) Using Terraform" and "Create Azure CNI based AKS Cluster with Application Gateway Ingress Controller (AGIC) Using Terraform" in previous posts. However, Nginx is a popular ingress controller for kubernetes. When using Nginx ingress controller with AKS  we can avoid the cost of an application gateway used for AGIC. In this post we are going to explore setting up Nginx ingress controller for AKS to expose applications running in AKS within the virtual network privately (without exposing them publicly), so , that only the other applications, or Azure services within the vNet can get access to the apps in AKS.

Expected outcome is to setup Nginx ingress controller in AKS with private IP frm AKS subnet as shown below.

Deploying Nginx Ingress to AKS Using a Makefile in a Custom Namespace

 Nginx is a popular ingress controller used in Kubernetes. We can use the yaml file here to deploy Nginx ingress controller to Azure Kubernetes Services. By default the namespace ingress-nginx will be used to deploy the Nginx controller. Let's see the usage of a Makefile to deploy Nginx ingress controller to AKS in a custom namespace utilizing the yaml file here.