Horizontal Pod Autoscaler Desired Replica Count Grafana Chart with Azure Monitor for AKS

 Horizontal Pod Autoscaler (HPA) in Kubernetes manage the number of instances for each pod deployment based on the scale settings specified. Itw would be good insights to monitor the demand for scaling of each pod. as a prerequicite we need to setup monitoring AKS cluster with Managed Grafana in Azure with Azure Monitor and Log Analytics Workspace.  Let's see how we can create a Grafana pannel to monitor the HPA behaviour over time.

The expected outcome is similar to below for a given deployment (pod HPA). Here max replicas setting shown in yellow line while green line shows desired number of replicase desired at each point in time, by the HPA based on the scale settings.

The query is as below to create the above panel. The variable aksservice used here to allow the user to sect the required deployment in the Grafana board so the panel can be used to see the required app deployment pod HPA scale counts.

InsightsMetrics

| where Name == 'kube_hpa_status_current_replicas'

| extend pTags = todynamic(Tags)

// get the cluster name | extend clusterName = todynamic(pTags["container.azm.ms/clusterName"])

// get the namespace

| extend ns = todynamic(pTags.k8sNamespace)

// get the hpa name for app

| extend deployment_hpa = todynamic(pTags.targetName)

// this is max replicas set in hpa

| extend max_reps = todynamic(pTags.spec_max_replicas)

// this is desired number of replicas of the app

| extend desired_reps = todynamic(pTags.status_desired_replicas)

| where clusterName == "aks-chdemo-dev04" // add filter for cluster name

| where ns == "mydemo" // add filter for desired namespace

// filter the hpa for a given hpa (app)

// We are using a variable named "aksservice" defined in grafana here

| where deployment_hpa startswith "${aksservice:text}"

| order by TimeGenerated asc

| project TimeGenerated, deployment_hpa, toint(desired_reps), toint(max_reps)

The variable is defined in the Grafana board

 The full json for Grafana panel is available in GitHub here. You can replace the id of the panel and the subscription, log analytics workspace name, resource group name etc.

The query is used in Grafana panel editor as shown below. You should select Azure Monitor as datasource and use Logs as Service. Then provide your log analaytics workspace for AKS as the resource.