Using Managed Prometheus Data to Setup Monitoring for AKS Deployed Bitnami Redis Cluster or Standalone

 Based on "Enable Prometheus Data Scraping for Bitnami Redis Cluster Deployed on AKS with Managed Prometheus" and "Enable Prometheus Data Scraping for Bitnami Redis Standalone Deployed on AKS with Managed Prometheus" discussed in the previous posts, let's look at using the managed prometheus data source in Azure Manged Grafana and how to setup monitoring for AKS deployed Bitnami Redis, with few examples.

The expectation is to have charts as shown below.

Above are just few examples. We can create more charts with metrices. For this post let's look at how to setup ech of above charts as a starting point.

The first step is to ensure you have your Azure Monitoring workspaces enabled with managed promethesus for AKS are visible in Azure managed grafana (Refer post "Setup Managed Prometheus for AKS via Terraform").

In our grafana dasgboard we can add a variable so that the managed prometheus Azure monitor workspace can be selected for each environment in the board. This let's us create a single dashboard andset of visualizations that can be used to monitor multiple environments. We can create a variable for dashoard and add the type as prometheus.

This allows us selecting the required data source prometheus connected with Azure monitor workspace for a given environment in the dashoard.

With above let's look at how to setup redis pod memory vs redis memory usage chart. Pod memory is the total memory consumption in a single pod of redis cluster, where redis memory is actual redis memory usage. However, alerting should be based on pod memory usage as it is what is totaul consumption, out of allocated pod memory.

To get pod memory usage in MiB we can use below query. Here we filter for redis namespace and fitler out redis_insight pod.

sum by (pod) (

  node_namespace_pod_container:container_memory_working_set_bytes{

    cluster="ch-demo-${instance:text}-aks-green",

    namespace="redis",

    pod!="redis-insight-0"

  }

)/1024/1024

Below is how it can be setup with grafana visualization query.

Then we can setup redis memory usage in same visualization with addin another query. The metric redis_memory_used_bytes is avaiilable to us since we have setup the service monitor as described in the post "Enable Prometheus Data Scraping for Bitnami Redis Cluster Deployed on AKS with Managed Prometheus". ${instance:text} here is another variable setup with having text value of dev-eus-001 as example representing the environment.

redis_memory_used_bytes{

    cluster="ch-demo-${instance:text}-aks-green"

}/1024/1024

Then we can compare pod memory usage and redis memory usage as in below.

Then lets see how we can create a visualization to compare redis pod memory usage with pod memory limit. Here we can setup a calculated 70% memry line as well which we can use to setupp an alert if a pod reaches 70% of its memory limit.

max(

  max_over_time(

    cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{

      cluster="ch-demo-${instance:text}-aks-green",

      namespace="redis",

      pod!="redis-insight-0"

    }[$__range]

  )

)/ 1024 / 1024

The quey above gets us the memory limit in redis cluster pods as a single value. It can be setup in grafana as shown below.

Then for the 70% line we can add a trasformation as shown below.

As final query in this chart we can add pod memory usage.

sum by(pod) (

    node_namespace_pod_container:container_memory_working_set_bytes{

      cluster="ch-demo-${instance:text}-aks-green",

      namespace="redis",

      pod!="redis-insight-0"

    }

  )/ 1024 / 1024

All of this gives us the Max limit 70% and usage chart.

The tird visualization of redis connection we cn get via below query.

redis_cluster_connections{

    cluster="ch-demo-${instance:text}-aks-green", namespace="redis"

}

That will give us the chart below.

We can use meny other metrices and create many other visualiations and alerts for AKS deployed redis using managed prometheus in AKS.