Restart Count Details of Each Pod and Container for App in Grafana Chart with Azure Monitor for AKS

 Pod and container restart counts monitoring are discussed in posts "Pod Restart Counts Grafana Chart with Azure Monitor for AKS" and "Container Restart Counts Grafana Chart with Azure Monitor for AKS". Those two charts show summarized view per service. To view details of pod and container restarts for an individual service we can create another chart as described in this post.

Workaround Solution for Intermittent CrashLoopBackOff in Windows Containers Running on AKS (.NET 6 Apps with System.Net.Sockets.SocketException 11001 and 10060)

Let's look at a temporary solution to the issue Intermittent CrashLoopBackOff in Windows Containers Running on AKS (.NET 6 Apps with System.Net.Sockets.SocketException 11001 and 10060). Same issue is asked in stackoverflow here. Instead of manually deleting pods that run into the issue maually, the cleaner app implemented in this repo is doing autmatic deletion of pods CrashLoopBackOff state with known exception reported in the container log. If the exeception is unknown the pod in CrashLoopBackOff state will not be deleted, and the container log output is printed, in cleaner app logs to show the exception of the pod having CrashLoopBackOff state.

Docker Alpine (mcr.microsoft.com/dotnet/runtime) - Resolve "Microsoft.WindowsAzure.Storage.StorageException: Only the invariant culture is supported in globalization-invariant mode."

.NET app running with mcr.microsoft.com/dotnet/runtime docker alpine images, might run into issue "Microsoft.WindowsAzure.Storage.StorageException: Only the invariant culture is supported in globalization-invariant mode" if your app is requiring globalization. Let's look at what needs to be done to get the issue fixed.

Resolve Windows Docker Build "C:\Program Files\dotnet\sdk\6.0.407\Microsoft.Common.CurrentVersion.targets(3262,5): error MSB4018: System.Runtime.InteropServices.COMException (0x80040154): Retrieving the COM class factory for component with CLSID {7Bxxxxxxxxxxxx} failed due to the following error: 80040154 Class not registered (0x80040154 (REGDB_E_CLASSNOTREG))"

 Building .NET 6 project with Resource files is running into build issue as shown below, when building with .NET 6 nanoserver build images.

C:\Program Files\dotnet\sdk\6.0.407\Microsoft.Common.CurrentVersion.targets(3262,5): error MSB4018: The "GenerateResource" task failed unexpectedly. [C:\src\Demo\Demo.csproj]

C:\Program Files\dotnet\sdk\6.0.407\Microsoft.Common.CurrentVersion.targets(3262,5): error MSB4018: System.Runtime.InteropServices.COMException (0x80040154): Retrieving the COM class factory for component with CLSID {7Bxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} failed due to the following error: 80040154 Class not registered (0x80040154 (REGDB_E_CLASSNOTREG)). [C:\src\Demo\Demo.csproj]

Container Restart Counts Grafana Chart with Azure Monitor for AKS

 We have disucussed pod restart Grafana panel in AKS in the post "Pod Restart Counts Grafana Chart with Azure Monitor for AKS". We can create similar panel for container restarts using managed Grafana in Azure for AKS using Azure monitor.

Expected Outcome

Panel similar to below showing contianer restarts for a applications over time with a table showing last container(s) of application restarts, maximum restarts for each application.

Pod Restart Counts Grafana Chart with Azure Monitor for AKS

 If your pods are often restarting it might indicate a problem you might have in your application deployed to AKS. For example, there was some significant number of restarts seen in .NET 3.1 applications deployed to AKS (reason found as .NET 3.1 issue which is supposedly fixed in .NET 5, so the approach to fix was to update the applications to .NET 6), which were only appearing in development and in staging envronments, while QA environment haven't shown a single restart. Threfore, it is important to monitor the restart counts in pods to identify issues you might not see in development or QA envronements, but may occur in production environments. Let's see how we can create a pod restart count panel in managed Grafana in Azure for AKS using Azure monitor.

Pod Counts Grafana Chart with Azure Monitor for AKS

 We have discussed how to create a Grafana panel for view Horizontal Pod Autoscaler (HPA) desired pod counts in AKS (Azure Kubernetes Services) in the post "Horizontal Pod Autoscaler Desired Replica Count Grafana Chart with Azure Monitor for AKS". As desired count is changing in HPA for an app monitoring how the actual pod count is changing would be really useful insights. Let's look at steps required to implement Grafana chart for actual pod counts over time using Azure monitor fror AKS.

Horizontal Pod Autoscaler Desired Replica Count Grafana Chart with Azure Monitor for AKS

 Horizontal Pod Autoscaler (HPA) in Kubernetes manage the number of instances for each pod deployment based on the scale settings specified. Itw would be good insights to monitor the demand for scaling of each pod. as a prerequicite we need to setup monitoring AKS cluster with Managed Grafana in Azure with Azure Monitor and Log Analytics Workspace.  Let's see how we can create a Grafana pannel to monitor the HPA behaviour over time.