Chaminda's DevOps Journey with MSFT: Mount Per Pod Managed Disk to AKS Pod to Provide Isolated Disk Space for Processing Large Files

In AKS processing files on AKS node disk by application pod is not a recommended approach. It could lead to issues in stability of the AKS node itself. Therefore, we have two options. Either mount an Azure fileshare to the pod as described in "Mount Azure Storage Fileshare Created with Terraform on AKS". The other option is to setup a per pod disk, specially when the need is to process larger files. This option is really useful when we have pods running a single job and when it lives for the lifetime of the job. Example case is use of KEDA scale object with per single message processing from rabbitmq as described in "RabbitMQ KEDA Trigger for AKS Deployed Apps to Scale Out the Apps Accurately".

Expentation is to get a per pod disk attached as shown below.

Before we dicide to use per pod disk in AKS pods we have to check the node pool VM size and for a given VM size how many data disks can be attached. That is the maximum number of pods we can run in such a node. We can use below command to get how many dsk can be attached to each VM size.

az vm list-skus \
  --location azureregion \
  --resource-type virtualMachines \
  --query "[?name=='vmsize'].capabilities[] | [?name=='MaxDataDiskCount'].value

Example:

az vm list-skus \
  --location westeurope \
  --resource-type virtualMachines \
  --query "[?name=='Standard_D8ds_v4'].capabilities[] | [?name=='MaxDataDiskCount'].value

The command will retun the number of disks possible to attach to the VM size.

Or we can verify this in documentation as well. See Max Remote Storage Disks (Qty.) in links below. You can find for each VM series in docs. Examples below.

We should plan to run only number of pods below the number of allowed managed disks in a given AKS node. In my case CPU and memery requirements in each pod only allows couple of pods on a AKS node so 16 disks is more than enough limit. Or you can control this by using

Example TF code. Note max_pods = 15

resource "azurerm_kubernetes_cluster_node_pool" "app_pool" {

  lifecycle {
    ignore_changes = [node_count]
  }

  name                  = "demopool1"
  orchestrator_version  = local.kubernetes_version
  kubernetes_cluster_id = azurerm_kubernetes_cluster.aks_cluster.id
  auto_scaling_enabled  = true
  node_count            = 5
  min_count             = 5
  max_count             = 36
  vm_size               = var.app_vm_size
  os_type               = "Linux"
  os_sku                = "Ubuntu"
  vnet_subnet_id        = var.subnet_id
  os_disk_size_gb       = "512"
  max_pods              = 15 
  scale_down_mode       = "Delete"
  zones                 = ["1", "2", "3"]

Then next step is getting a storage class defined. Note that here we ensure the disk get removed once the pod gets deleted after the work. Disk only lives for lifetime of the pod so the cost is only while a pod is running.

# StorageClass for per-pod disks (StatefulSet volumeClaimTemplates)
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: demo-disk-storage
parameters:
  skuName: StandardSSD_LRS
provisioner: disk.csi.azure.com
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

We need to use a stateful set here is why.

Deployment + PVC

All pods share the same PVC (and the same disk)
Azure Managed Disks are ReadWriteOnce — only one node can attach at a time
If pods land on different nodes, the second pod fails to attach the disk and can't start
You can't auto-provision one disk per pod — you'd have to manage it manually

StatefulSet + volumeClaimTemplates

Kubernetes automatically creates one dedicated disk per pod — pod-0 gets disk-0, pod-1 gets disk-1
No sharing, no contention, no attach conflicts across nodes
With reclaimPolicy: Delete, the disk is destroyed when the pod is deleted — no lingering cost, no cleanup needed
Scales naturally: spin up 5 pods → 5 disks provisioned; scale down → disks deleted automatically

For Azure Managed Disks (ReadWriteOnce), a Deployment physically can't give each pod its own disk. A StatefulSet was designed exactly for this — stable, per-pod storage that lives and dies with the pod.

Here is how we can deploy such a statefulset.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: demo-app
  namespace: demo
  labels:
    app: demo-app
spec:
  podManagementPolicy: Parallel
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Delete
    whenScaled: Delete
  updateStrategy:
    type: RollingUpdate
  minReadySeconds: 30
  selector:
    matchLabels:
      service: demo-app
  template:
    metadata:
      labels:
        app: demo-app
        service: demo-app
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: demo-wi-sa
      nodeSelector:
        "kubernetes.io/os": linux
      priorityClassName: demo-medium-prority-linux
      #------------------------------------------------------
      # setting pod DNS policies to enable faster DNS resolution
      # https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy
      dnsConfig:
        options:
          # use FQDN everywhere 
          # any cluster local access from pods need full CNAME to resolve 
          # short names will not resolve to internal cluster domains
          - name: ndots
            value: "2"
          # dns resolver timeout and attempts
          - name: timeout
            value: "15"
          - name: attempts
            value: "3"
          # use TCP to resolve DNS instad of using UDP (UDP is lossy and pods need to wait for timeout for lost packets)
          - name: use-vc
          # open new socket for retrying
          - name: single-request-reopen
      #------------------------------------------------------
      volumes:
        # `name` here must match the name
        # specified in the volume mount
        - name: demo-configmap-demo-app-volume
          configMap:
            # `name` here must match the name
            # specified in the ConfigMap's YAML
            name: demo-configmap
      terminationGracePeriodSeconds: 300
      containers:
        - name: demo-app
          lifecycle:
            preStop:
              exec:
                command: ["sleep","10"]
          image: demoacr001.azurecr.io/demo/demo-app:1234
          imagePullPolicy: Always
          volumeMounts:
            - mountPath: /etc/config
              name: demo-configmap-demo-app-volume
            - mountPath: /media/data
              name: media-data-volume
          env:
            - name: DEMO_CONFIG_PATH
              value: /etc/config/config_dev.json
            - name: TERMINATION_GRACE_PERIOD
              value: "300"
            - name: MEDIA_PATH
              value: /media/data
            - name: DEMO_POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: CONTAINER_MEMORY_LIMIT
              valueFrom:
                resourceFieldRef:
                  resource: limits.memory
          resources:
                limits:
                  memory: 4Gi # the memory limit equals to the request!
                  # no cpu limit! this is excluded on purpose
                requests:
                  memory: 4Gi
                  cpu: "1"
  volumeClaimTemplates:
    - metadata:
        name: media-data-volume
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: demo-disk-storage
        resources:
          requests:
            storage: 256Gi

Difference from the file share usage as in "Mount Azure Storage Fileshare Created with Terraform on AKS", in above we use a statefulset instead of deployment.

kind: StatefulSet

Then we use below to ensure disk deleted when statefulset deleted or scaled in deleting a pod.

spec:
  podManagementPolicy: Parallel
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Delete
    whenScaled: Delete
  updateStrategy:
    type: RollingUpdate

We remove the volume referece refering the file share volume

volumes:
  - name: fsdemo-data-volume
    persistentVolumeClaim:
      claimName: fsdemo-storage-pvc

Instead we use vlume claim template, with desired disk size., and we refer the storage class created previously.

  volumeClaimTemplates:
    - metadata:
        name: media-data-volume
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: av-disk-storage
        resources:
          requests:
            storage: 256Gi

Then we can refer to the volume claim for disk and mount it same way as we mounted a file share.

volumeMounts:
  - mountPath: /media/data
    name: media-data-volume

This path can be passed to the pod to find the path within pod using env variable.

env:            
  - name: MEDIA_PATH
    value: /media/data

With this we get per pod disk running when a pods are runing as shown below. Note this is a single message processor as mentioned in "RabbitMQ KEDA Trigger for AKS Deployed Apps to Scale Out the Apps Accurately"

You can see a disk getting created per pod. Pods get allocated to same AKS node as well and gets own per pod disk.

From above we can clearly see each pod even though allocated to same node has seperate Axure managed disk. See below.

The important part:


Node: aksavwin000008

Pod                                      Azure Disk
--------------------------------------------------------------
largemetadata-messageprocessor-0  --->  pvc-f8446b27-1156-4c41-acb0-59ef136a376d

largemetadata-messageprocessor-1  --->  pvc-a49a262b-ddbe-42fc-9b83-e5b9391622a9

Same node:


aksavwin000008
       |
       +-- Disk pvc-f8446b27-1156-4c41-acb0-59ef136a376d
       |
       +-- Disk pvc-a49a262b-ddbe-42fc-9b83-e5b9391622a9

And another example:


Node: aksavwin000009

Pod                                      Azure Disk
--------------------------------------------------------------
largemetadata-messageprocessor-2  --->  pvc-1506338e-e10c-40ac-8caa-d3c85c599991

largemetadata-messageprocessor-3  --->  pvc-aff64124-8d8b-4bdd-8755-f3fcd2a0529e

This confirms our StatefulSet behavior is exactly what we wanted:

✅ Each StatefulSet replica has its own PVC
✅ Each PVC has its own PV
✅ Each PV maps to a unique Azure Managed Disk
✅ Multiple pods can share the same Kubernetes node
✅ Each pod still gets isolated storage

This is the normal AKS pattern for StatefulSets using Azure Disk CSI with volumeClaimTemplates.

Below test futher proves per pod disk as you can see with linux pods here in the same AKS node.

1. Kubernetes scheduling shows two Linux pods on the same node

From our output above:


POD                             NODE
---------------------------------------------------------------
largevideo-messageprocessor-1   aks-avlinux-42788959-vmss000006
largevideo-messageprocessor-3   aks-avlinux-42788959-vmss000006

Both pods are on the same AKS Linux node:


aks-avlinux-42788959-vmss000006
        |
        +-- largevideo-messageprocessor-1
        |
        +-- largevideo-messageprocessor-3

2. But each pod has a different PVC

Our PVC mapping:


POD                             PVC
---------------------------------------------------------------------------
largevideo-messageprocessor-1   media-data-volume-largevideo-messageprocessor-1

largevideo-messageprocessor-3   media-data-volume-largevideo-messageprocessor-3

So Kubernetes has created:


aks-avlinux-42788959-vmss000006

    |
    +-- PVC media-data-volume-largevideo-messageprocessor-1
    |
    +-- PVC media-data-volume-largevideo-messageprocessor-3

3. Inside the container, the same parent folder exists

Both pods navigate to:


../media/data/2b1ebab57530-23df66a1-b347-48cc-9838-895eb9525c5f

This path is identical:


/media/data/2b1ebab57530-23df66a1-b347-48cc-9838-895eb9525c5f

But the contents prove they are different disks.

Pod 1


largevideo-messageprocessor-1

/media/data/2b1ebab57530-23df66a1-b347-48cc-9838-895eb9525c5f

    |
    +-- 653a7334-fa93-448a-872b-4f96c70945ee

Pod 3


largevideo-messageprocessor-3

/media/data/2b1ebab57530-23df66a1-b347-48cc-9838-895eb9525c5f

    |
    +-- fc02ff5b-7a45-4498-a666-714d4388956a

If they were sharing the same Azure Disk, we should see:


Pod 1:
media/data/2b1ebab57530...
    |
    +-- 653a7334...


Pod 3:
media/data/2b1ebab57530...
    |
    +-- 653a7334...

because both containers would be looking at the same filesystem.

Instead we see:


Pod 1 filesystem
----------------
/media/data/2b1ebab57530...
        |
        +-- 653a7334-fa93...


Pod 3 filesystem
----------------
/media/data/2b1ebab57530...
        |
        +-- fc02ff5b-7a45...

Same directory structure, different contents.

Final architecture picture


AKS Linux Node
aks-avlinux-42788959-vmss000006

        |
        |
        +-------------------------------+
        |                               |
        |                               |
largevideo-messageprocessor-1     largevideo-messageprocessor-3
        |                               |
        |                               |
        PVC                             PVC
        |                               |
media-data-volume-                    media-data-volume-
largevideo-messageprocessor-1         largevideo-messageprocessor-3
        |                               |
        |                               |
Azure Disk A                       Azure Disk B
        |                               |
        |                               |
653a7334...                         fc02ff5b...

So our test proves three important things:

✅ Two StatefulSet pods can run on the same AKS node
✅ Each pod gets a different Azure Disk/PVC
✅ The application sees the same root folder path due to application logic creating same root folder, but each pod sees only its own disk contents

This is exactly the expected behavior for AKS StatefulSet + Azure Disk CSI + volumeClaimTemplates.

Chaminda's DevOps Journey with MSFT

Saturday, 27 June 2026

Mount Per Pod Managed Disk to AKS Pod to Provide Isolated Disk Space for Processing Large Files