Enabling Prometheus Data Scraping for Nginx-Gateway Deployed in - AKS Setup with Managed Prometheus

 We have discussed how to setup Nginx-Gateway instead of Nginx ingress controller in AKS in previous posts listed in the bottom of this post. We have also looked at  "Setup Managed Prometheus for AKS via Terraform" and "Deploying Azure Managed Grafana with Terraform" . For Nginx-Gateway deployed in AKS, we need to enable datascraping so that we can start monitring nginx metrices from managed grafana. Let's look at the steps required.

The expectation is getting metrices ti managed grafana as shown below.

Before we begin anything lets see if ngix metric data is already scraping from AKS. For this we can check ama-metrics pods.

kubectl get pods -n kube-system -l rsName=ama-metrics -o wide

Once we get ama-metrics pods we can see if it has exposed prometeus UI port. using command 

kubectl exec -n kube-system <ama-metrics-pod-name> -- sh -c "netstat -tlnp || ss -tlnp"

Then we can do a port forward as shown below

kubectl port-forward -n kube-system <ama-metrics-pod-name> 9090:9090

Then we can browse to http://localhost:9090/targets and see what metrics are scraping. Here we see only elastic hosted in AKS (Elastic on AKS is a future topic that we will discuss) metrics are scraping, but no nginx-gateway metric available.

How to enable Nginx-Gateway metrics scraping

As the first step we need to see in our deployment of Nginx-Gateway (see posts listed in the bottom of this post to setup Nginx-Gateway in AKS), if the prometheus metrics ports are exposed by the services or by pods. Depending on that we can define required service monitor or pod monitor, to enable data scraping via managed promethues in AKS to managed grafana.

Let's check nginx-gateway services first.

kubect get svc -n nginx-gateway

Then we can see each service ports and annotations, which shows no metrics ports exposed.

kubectl get svc ngf-nginx-gateway-fabric -n nginx-gateway -o json | jq -r '

  "PORTS:",

  (.spec.ports[] | "  \(.name) \(.port)/\(.protocol) -> \(.targetPort)"),

  "ANNOTATIONS:",

  (.metadata.annotations | to_entries[] | "  \(.key)=\(.value)")

'

For the gateway you should use your gateway service name.

kubectl get svc selfhost-apps-gateway-nginx -n nginx-gateway -o json | jq -r '

  "PORTS:",

  (.spec.ports[] | "  \(.name) \(.port)/\(.protocol) -> \(.targetPort)"),

  "ANNOTATIONS:",

  (.metadata.annotations | to_entries[] | "  \(.key)=\(.value)")

'

Let's check nginx-gateway pods.

kubectl get pod -n nginx-gateway

Then in one of the control plane pods we can see what are exposed ports and annotations.

kubectl get pod <ngf-nginx-gateway-fabric-pod-name> -n nginx-gateway -o json | jq -r '

  "PORTS:",

  (.spec.containers[].ports[] | "  \(.name) \(.containerPort)/\(.protocol)"),

  "ANNOTATIONS:",

  (.metadata.annotations | to_entries[] | "  \(.key)=\(.value)")

'

We should inspect gateway data plane pod as well to see the ports and annotations.

kubectl get pod <gateway-nginx-pod-name> -n nginx-gateway -o json | jq -r '

  "PORTS:",

  (.spec.containers[].ports[] | "  \(.name) \(.containerPort)/\(.protocol)"),

  "ANNOTATIONS:",

  (.metadata.annotations | to_entries[] | "  \(.key)=\(.value)")

'

We can see both control plane and data plane pods expose metrics ports. So the consusion is we should setup pod monitors in both control and data planes.

Let's now identify the label selectors we can use for pod monitors.

To identify the labels of control plane and data plane pods of nginx gateway, we can show labels with below shown command.

kubectl get pod -n nginx-gateway --show-labels

Now we have gathered all required information to setup our pod monitors as shown below.

Note that we need to use apiVersion: monitoring.coreos.com/v1

 instead of apiVersion: monitoring.coreos.com/v1

 due to we are using Azure managed procetheus with AKS.

apiVersion: azmonitoring.coreos.com/v1

kind: PodMonitor

metadata:

  name: nginx-gateway-fabric

  namespace: nginx-gateway

spec:

  podMetricsEndpoints:

  - port: metrics

  selector:

    matchLabels:

      app.kubernetes.io/name: nginx-gateway-fabric

      app.kubernetes.io/instance: ngf

  namespaceSelector:

    matchNames:

    - nginx-gateway

---

apiVersion: azmonitoring.coreos.com/v1

kind: PodMonitor

metadata:

  name: selfhost-apps-gateway-nginx

  namespace: nginx-gateway

spec:

  podMetricsEndpoints:

  - port: metrics

  selector:

    matchLabels:

      app.kubernetes.io/name: selfhost-apps-gateway-nginx

      app.kubernetes.io/instance: ngf

  namespaceSelector:

    matchNames:

    - nginx-gateway

We can add below steps to deployment scripts used in Nginx-Gateway deployment as described in the posts lined in the end of this post.

$nginxGatewayMonitorManifest = -join($manifestPath,'nginx_gateway/','nginx_gateway_monitor.yaml');

Write-Host (-join('Deploying Nginx-Gateway monitor with: ',$nginxGatewayMonitorManifest, ' ...'));
    kubectl apply -f $nginxGatewayMonitorManifest;
    Write-Host ('Successfully deployed Nginx-Gateway monitor.');
    Write-Host ('=========================================================');

Once we apply the above pod monitors we can reinspect the promethus UI. It now shows nginx-gateway control plane and the data plane metrics availability.

Nginx Gateway setup posts

• High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 1 - Plan for Smooth Transition
• High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 2 - Deploy Nginx-Gateway-Fabric
• High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 3 - Setup Nginx-Gateway Routes and Create Gateway Policies to Map with Ingress Nginx Annotations
• High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 4 - Switch Traffic from Ingress-Nginx to Nginx-Gateway
• High Availability Deployment of Nginx Gateway Fabric Replacing Retired Ingress Nginx in AKS - Part 5 - Cleanup Ingress-Nginx from the AKS Cluster
• Adding SnippetsFIlter to HTTPRoute in Nginx Gateway to Implement Proxy Annotations of Retired Ingress Nginx
• Handle Nginx Gateway Certificate Refresh While Inplace Upgrade in AKS