The pods deployed to AKS gets terminated due to reschduling, low priority evictions as well as during scaled in. We can add a termination grase period and pre stop sleep time as shown below in Linux and Windows containers to allow, sufficient time to ingress services to get updated about terminating pods. However, Windows nanoserver does not support PowerShell. Therefore, we need to use a specific mechanism to pre stop hook for nanoserver images. For nanoserver images shutdown signal is correctly get sent to the dotnet app. So we can just setup a sllep time for pre stop hook as necessary.
Chaminda's DevOps Journey with MSFT
DevOps with Azure DevOps
Wednesday, 2 April 2025
Windows Nanoserver Image Pre Stop Hook to Avoid 502 for Requests
Saturday, 22 March 2025
Gracefully Shut Down dotnet 8 IHostedService App - Deployed as a Windows Container in AKS - While Scale In or Pod Deallocations
Applications implemented with IHostedService in dotnet, deployed to Azure Kubernetes Services (AKS) as containers in pods get terminated when pod recheduling happens or scaling-in opertaions happen. However, unlike Linux containers, the Windows containers does not receive the signal (similar to SIGTERM or SIGINT) to graceful shutdown. Once the pre stop hook is done the container is immediatly killed disregarding the value set in the terminationgraceperiod. Since, the Windows container did not receive a message to start a graceful shut down, and it is killed abruptly, the in flight operations in the Windows app container are abandoned. Such abadoning of operations cause inconsitency in system data and cause system failures. Therefore, it is mandatory to implement a proper graceful shutdown for Windows containers as well. Let's explore the issue in detail and how to implement a proper solution to enable graceful Windows container shut down, for dotent apps implemented with IHostedService. The issue is happening in mcr.microsoft.com/dotnet/runtime:8.0-windowsservercore-ltsc2022 images and the solution is tested with the same.
Windows app pod scaled-in or pod rescedule
Wednesday, 5 March 2025
Setting Up Alert for AKS Pod Restarts Using Log Analytics Workspace and Grafana
Azure Kubernetes Services (AKS) pod restarts can be obtained from the KubePodInventory of the connected log analytics workspace. This data can be depicted in a graph in grafana as described in the post "Pod Restart Counts Grafana Chart with Azure Monitor for AKS". Let's explore how to use same information to create an alert using Grafana to notify when pod restarts are happening in apps in a given kubernetes namespace.
The expectation is to fire alerts from Grafana as shwon below. Note that the alerts can target to send emails, slack notficaition etc. which is not discussed in this post.
Monday, 3 March 2025
Using "grep" with "kubectl logs" to Filter Container/Pod Logs
kubectl logs command helps us to inspect logs of pods in AKS/kubernetes and useful to diagnose issues. However, when there is too much logs it is harder to read through and find out errors easily. Further, filtering out logs for a given timestamp may be useful at times to identify the issues. In this post let's explore usage of grep with kubectl logs command to filter logs.
Let's take first example to filter for a timestamp in keda operator pod logs. Here -i says to ignore case in logs.
kubectl logs keda-operator-79d756dd66-69gsc -n keda | grep -i '2025-03-04T07:20:24'
Wednesday, 29 January 2025
Setup Azure File Share Capacity Alert to Slack with Terraform
Setting up an Azure File Share capacity alert is useful to know when you reach at least 80% of allocated quota for the file share. This will give the teams ample time to increase the allocation to avoid out of space issues. If we are using standard tier for storage account then we need to use one storage account for each file share, to get the correct alert. Sending the alert to slack channel is a useful way to get properly alerted to take action on time. Let's use an example learn how to setup alerts for multiple Azure file shares uing terraform.
Expectation is to get the alerts to slack channel as shown below.
Saturday, 7 December 2024
Use AKS Workload Identity to Login with "az login" in a Container with Azure CLI
We have discussed "Setting Up Azure Workload Identity for Containers in Azure Kubernetes Services" previously. We can use Azure CLI to interact with Azure resources in a container which is using base docker image "mcr.microsoft.com/azure-cli:latest-amd64". Instead of using specific credential information directly in the Azure CLI script in the container, we can use the workload identity to do an az login in a container where workload identity is enabled. Let's look at how to do that in this post.
Friday, 29 November 2024
Use Dynamic Block Conditionally in Terraform with a Nested Block
We have discussed usage of dynamic blocks conditionally in terraform in the post "Use Dynamic Block Conditionally in Terraform". Sometimes we need to use dynamic with nested blocks. In this post look at such an example of using dynamic block with network rules for Azure event hubs.
We are going to setup a nested dynamic block as shown below.
Saturday, 16 November 2024
Disable Soft Delete for Azure File Shares with Terraform
Azure file shares can be used as mounted persisted volume in AKS clusters to provide necessary storage to pods in AKS. Azure file shares are enabled with soft delete for files by default. However, when the pod mounted storage is used as temporary storage for processing, there is no need to keep the capability of soft deletion. Let's look at how we can disable soft delete for file share storage when deploying the file share with terraform.
Popular Posts
-
Setting up an Azure File Share capacity alert is useful to know when you reach at least 80% of allocated quota for the file share . This w...
-
Dynamic block allows to create nested multi level block structures in terraform code. Conditional usage of such blocks are really useful in...
-
We have discueed, that we have to use an environment variable to handle input parameter default values, if we are using trigger for workflo...
-
Applications implemented with IHostedService in dotnet, deployed to Azure Kubernetes Services (AKS) as containers in pods get terminated w...
-
In Azure DevOps YAML pipelines there are several functions available for you to use. replace is such a useful function, which you can use t...