Enable Windows Data Scraping for AKS Managed Prometheus with Azure Managed Grafana

 We have "Setup Managed Prometheus for AKS via Terraform", however, that setup alone will not provide windows metrics from AKS clsuter to Azure managed Grafana. We have to addtionaly, setup Windows exporter and couple of additional configurations to make it work as decribed in the official Microsoft docs here. Let's look at step by step how to enable windows metrics for AKS with managed prometheus.

Expected outcome is getting metrics such as shown below to managed grafana and visualizing them.

To enable Window metrics frst we need to setup namespace monitoring.

---

apiVersion: v1

kind: Namespace

metadata:

  name: monitoring

  labels:

    name: monitoring

Then below deamonset with windows exporter should be created.

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: windows-exporter
  name: windows-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: windows-exporter
  template:
    metadata:
      labels:
        app: windows-exporter
    spec:
      securityContext:
        windowsOptions:
          hostProcess: true
          runAsUserName: "NT AUTHORITY\\system"
      hostNetwork: true
      initContainers:
        - name: configure-firewall
          image: mcr.microsoft.com/powershell:lts-nanoserver-1809
          command: ["powershell"]
          args: ["New-NetFirewallRule", "-DisplayName", "'windows-exporter'", "-Direction", "inbound", "-Profile", "Any", "-Action", "Allow", "-LocalPort", "9182", "-Protocol", "TCP"]
      containers:
      - args:
        - --config.file=%CONTAINER_SANDBOX_MOUNT_POINT%/config.yml
        name: windows-exporter
        image: ghcr.io/prometheus-community/windows-exporter:0.31.3 # Refer https://github.com/prometheus-community/windows_exporter/pkgs/container/windows-exporter or https://github.com/prometheus-community/windows_exporter/releases for tags
        imagePullPolicy: Always
        ports:
        - containerPort: 9182
          hostPort: 9182
          name: http
        resources:
          limits:
            memory: 2Gi # the memory limit equals to the request!
            # no cpu limit! this is excluded on purpose
          requests:
            memory: 2Gi
            cpu: "500m"
        volumeMounts:
        - name:  windows-exporter-config
          mountPath: /config.yml
          subPath: config.yml
      nodeSelector:
        kubernetes.io/os: windows
      volumes:
      - name: windows-exporter-config
        configMap:
          name: windows-exporter-config

Next we need to setup two config maps as shown below. 

---
# https://learn.microsoft.com/en-us/azure/azure-monitor/containers/kubernetes-monitoring-enable?tabs=terraform#enable-windows-metrics
# https://learn.microsoft.com/en-us/azure/azure-monitor/containers/prometheus-metrics-scrape-configuration
# https://github.com/Azure/prometheus-collector/blob/main/otelcollector/configmaps/ama-metrics-settings-configmap-v1.yaml
kind: ConfigMap
apiVersion: v1
data:
  schema-version:
    #string.used by agent to parse config. supported versions are {v1, v2}. Configs with other schema versions will be rejected by the agent.
    v1
  config-version:
    #string.used by customer to keep track of this config file's version in their source control/repository (max allowed 10 chars, other chars will be truncated)
    ver1
  prometheus-collector-settings: |-
    cluster_alias = ""
    https_config = true
  default-scrape-settings-enabled: |-
    kubelet = true
    coredns = false
    cadvisor = true
    kubeproxy = false
    apiserver = false
    kubestate = true
    nodeexporter = true
    windowsexporter = true
    windowskubeproxy = true
    kappiebasic = true
    networkobservabilityRetina = true
    networkobservabilityHubble = true
    networkobservabilityCilium = true
    prometheuscollectorhealth = false
    controlplane-apiserver = true
    controlplane-cluster-autoscaler = false
    controlplane-node-auto-provisioning = false
    controlplane-kube-scheduler = false
    controlplane-kube-controller-manager = false
    controlplane-etcd = true
    acstor-capacity-provisioner = true
    acstor-metrics-exporter = true
    local-csi-driver = true
  # Regex for which namespaces to scrape through pod annotation based scraping.
  # This is none by default.
  # Ex: Use 'namespace1|namespace2' to scrape the pods in the namespaces 'namespace1' and 'namespace2'.
  pod-annotation-based-scraping: |-
    podannotationnamespaceregex = "yournamespace1|default|kube-system|monitoring"
  default-targets-metrics-keep-list: |-
    kubelet = ""
    coredns = ""
    cadvisor = ""
    kubeproxy = ""
    apiserver = ""
    kubestate = ""
    nodeexporter = ""
    windowsexporter = ""
    windowskubeproxy = ""
    podannotations = ""
    kappiebasic = ""
    networkobservabilityRetina = ""
    networkobservabilityHubble = ""
    networkobservabilityCilium = ""
    controlplane-apiserver = ""
    controlplane-cluster-autoscaler = ""
    controlplane-node-auto-provisioning = ""
    controlplane-kube-scheduler = ""
    controlplane-kube-controller-manager = ""
    controlplane-etcd = ""
    acstor-capacity-provisioner = ""
    acstor-metrics-exporter = ""
    local-csi-driver = ""
    minimalingestionprofile = true
  default-targets-scrape-interval-settings: |-
    kubelet = "30s"
    coredns = "30s"
    cadvisor = "30s"
    kubeproxy = "30s"
    apiserver = "30s"
    kubestate = "30s"
    nodeexporter = "30s"
    windowsexporter = "30s"
    windowskubeproxy = "30s"
    kappiebasic = "30s"
    networkobservabilityRetina = "30s"
    networkobservabilityHubble = "30s"
    networkobservabilityCilium = "30s"
    prometheuscollectorhealth = "30s"
    acstor-capacity-provisioner = "30s"
    acstor-metrics-exporter = "30s"
    local-csi-driver = "30s"
    podannotations = "30s"
  debug-mode: |-
    enabled = false
  # Note that below section (ksm-config) is a yaml configuration, the settings override the configuration provided by optional parameters during 
  # onboarding for kube-state-metrics, only uncomment and use them if the collection needs to be customized.
  # Default settings - https://github.com/Azure/prometheus-collector/blob/e40947f3eaee8843021c90bd8645ce7875ad1fcf/otelcollector/deploy/addon-chart/azure-monitor-metrics-addon/values-template.yaml#L2
  # https://learn.microsoft.com/en-us/azure/azure-monitor/containers/kubernetes-monitoring-enable?tabs=cli#optional-parameters
  # OSS documentation for kube-state-metrics resources and the metrics - https://github.com/kubernetes/kube-state-metrics/tree/main/docs#exposed-metrics
  # ksm-config: |-
  #   resources: 
  #     secrets: {}
  #     configmaps: {}
  #   labels_allow_list: # object name and label names
  #     pods: 
  #     - app8
  #   annotations_allow_list: # object name and annotation names
  #     namespaces:
  #     - kube-system
  #     - default
metadata:
  name: ama-metrics-settings-configmap
  namespace: kube-system

Finally we can setup pod monitor with below yaml. Note that instead of monitoring.coreos.com/v1
we use  azmonitoring.coreos.com/v1 since we are using Azure managed prometheus for AKS.

---

apiVersion: azmonitoring.coreos.com/v1

kind: PodMonitor

metadata:

  labels:

    app: windows-exporter

  name: windows-exporter

  namespace: monitoring

spec:

  jobLabel: windows-exporter

  selector:

    matchLabels:

      app: windows-exporter

  podMetricsEndpoints:

  - port: http

    scheme: http

All of above can be applied via kubectl apply.

Then you should see windows-eporter per each windows node, running in your AKS cluster and windows metrics will be available in the Manged grafana.