Setting Up RabbitMQ Cluster in AKS Using RabbitMQ Cluster Operator

 We have discused "Setting Up RabbitMQ Cluster Operator and Topology Operator via Azure Pipelines in AKS" in the previous post. The deployed cluster oprator in AKS can be used deploy a RabbitMQ cluster. In this post let's explore deploying a production ready RabbitMQ Cluster in AKS (without TLS/SSL - we will explore that in a future post), to be used with apps deployed in same AKS cluster. 

Once successfully deployed the RabbitMQ clsuter the rabbitmq namcspace should have resources shown in the below image. We are deploying a three node RabbitMQ cluster, each pod scheduled in a different Azure availability zone node. Cluster access setup with service/rabbitmq-cluster ClusterIP, as we only need access within AKS cluster for apps. How to use it for local development, we can discuss in a future post.

Setting Up RabbitMQ Cluster Operator and Topology Operator via Azure Pipelines in AKS

Using operators to setup RabbitMQ cluster in  AKS (kubernetes) is a recomended approach. There are two RabbitMQ operators we need to deploy into AKS for setting up a RabbitMQ cluster and managing message setup in the deployed RabbitMQ cluster.

• Cluster Operator: Automate provisioning, management and oprations of a RabbitMQ cluster within AKS.
• Messaging Topology Operator: Managing message topologies within the deployed RabbitMQ cluster in AKS.

In this post let's look at how to seup above two operators in AKS using Azure DevOps pipeline step.

List All Pods and Their Priority Classes with kubectl

Sometmes it would be necessery to identify which priority classes are used in each pod in a kubernetes environment, specially to plan and reorganize priorities in apps deployed. Let's look at a query to view pods with priority classes using kubectl.

Below command will get all pods in all namespaces with their priority classes name and priority value. The highest priority number value is the highet priority.

Linux AKS Scaled Jobs Handle Graceful Termination for dotnet App using IHostedService

 We have discussed "Windows AKS Scaled Jobs Handle Graceful Termination for dotnet App using IHostedService When Preemption" previously. However,  for Linux scaled jobs we have to handle the graceful termination, using a bash script intead of a PowerShell script.

We can create a script with bash as shown below.

Windows AKS Scaled Jobs Handle Graceful Termination for dotnet App using IHostedService When Preemption

 We have discussed "Gracefully Shut Down dotnet 8 IHostedService App - Deployed as a Windows Container in AKS - While Scale In or Pod Deallocations" previously. The approach  works fine for pods deployed as a deployment in kubernetes. Similary to the deployment pods, scaled job pod can be terminated abruptly due to preemption in kubernetes, if a high priority pod is scheduled. One way to reolve abrupt termination of scaled job pod due to preemption, would be to assign all scaled jobs to highest possible priority. However, setting highest prority to all scaled jobs is not a good solution, as the job may not require highest priority and job should be able to scheduled after other high priority app pods. Let's look at a better solution that can be implented with pre stop hook for scaled jobs running with base docker image Windows server core 2022.

Windows Nanoserver Image Pre Stop Hook to Avoid 502 for Requests

The pods deployed to AKS gets terminated due to reschduling, low priority evictions as well as during scaled in. We can add a termination grase period and pre stop sleep time as shown below in Linux and Windows containers to allow, sufficient time to ingress services to get updated about terminating pods. However, Windows nanoserver does not support PowerShell. Therefore, we need to use a specific mechanism to pre stop hook for nanoserver images. For nanoserver images shutdown signal is correctly get sent to the dotnet app. So we can just setup a sllep time for pre stop hook as necessary.

Gracefully Shut Down dotnet 8 IHostedService App - Deployed as a Windows Container in AKS - While Scale In or Pod Deallocations

 Applications implemented with IHostedService in dotnet, deployed to Azure Kubernetes Services (AKS) as containers in pods get terminated when pod recheduling happens or scaling-in opertaions happen. However, unlike Linux containers, the Windows containers does not receive the signal (similar to SIGTERM or SIGINT) to graceful shutdown. Once the pre stop hook is done the container is immediatly killed disregarding the value set in the terminationgraceperiod. Since, the Windows container did not receive a message to start a graceful shut down, and it is killed abruptly, the in flight operations in the Windows app container are abandoned. Such abadoning of operations cause inconsitency in system data and cause system failures. Therefore, it is mandatory to implement a proper graceful shutdown for Windows containers as well. Let's explore the issue in detail and how to implement a proper solution to enable graceful Windows container shut down, for dotent apps implemented with IHostedService. The issue is happening in mcr.microsoft.com/dotnet/runtime:8.0-windowsservercore-ltsc2022 images and the solution is tested with the same.

Windows app pod scaled-in or pod rescedule 

Setting Up Alert for AKS Pod Restarts Using Log Analytics Workspace and Grafana

 Azure Kubernetes Services (AKS)  pod restarts can be obtained from the KubePodInventory of the connected log analytics workspace. This data can be depicted in a graph in grafana as described in the post "Pod Restart Counts Grafana Chart with Azure Monitor for AKS". Let's explore how to use same information to create an alert using Grafana to notify when pod restarts are happening in apps in a given kubernetes namespace. 

The expectation is to fire alerts from Grafana as shwon below. Note that the alerts can target to send emails, slack notficaition etc. which is not discussed in this post.