Pod Counts Grafana Chart with Azure Monitor for AKS

 We have discussed how to create a Grafana panel for view Horizontal Pod Autoscaler (HPA) desired pod counts in AKS (Azure Kubernetes Services) in the post "Horizontal Pod Autoscaler Desired Replica Count Grafana Chart with Azure Monitor for AKS". As desired count is changing in HPA for an app monitoring how the actual pod count is changing would be really useful insights. Let's look at steps required to implement Grafana chart for actual pod counts over time using Azure monitor fror AKS.

Expected Outcome

Panel similar to below showing pods for a given app showing running, terminating and pending pods over time.

In above panel you can see actual pod count changes based on the  HPA desired replica count for the same period for same app shown below (for below panel look at post "Horizontal Pod Autoscaler Desired Replica Count Grafana Chart with Azure Monitor for AKS").

Looking at the two charts above really help us to see the demand for instances and how actual scale out and scale in happens based on the demand.

We can use the query below to create the pod counts panel in Grafana with Azure monitor. The variable aksservice used here to allow the user to sect the required deployment in the Grafana board so the panel can be used to see the required app pod counts.

KubePodInventory

| where ClusterName == "aks-chdemo-dev04"

| where Namespace in('mydemo') and ControllerKind == 'ReplicaSet'

| extend pod_label = todynamic(PodLabel)

| extend app_name = todynamic(pod_label[0].app)

| where app_name == "${aksservice:text}"

| summarize running_pods = sumif(1, PodStatus == 'Running' ), terminating_pods = sumif(1, PodStatus == 'Terminating'), pending_pods = sumif(1, PodStatus == 'Pending') by TimeGenerated, tostring(app_name)

| order by TimeGenerated asc

| project TimeGenerated, running_pods, terminating_pods, pending_pods

The variable is defined in the Grafana board.

 The full json for Grafana panel is available in GitHub here. You can replace the id of the panel and the subscription, log analytics workspace name, resource group name etc.

The query is used in Grafana panel editor as shown below. You should select Azure Monitor as datasource and use Logs as Service. Then provide your log analaytics workspace for AKS as the resource.