Monitor and Profile Memory Usage in a Running .NET Linux Pod in AKS

 Sometimes we need to understand what causes high memeory usages or if any memory leaks in our application setup. When we see pod memeory is not reclaimed for longer intervals, analyzing the usage and diagnosing where exactly is the problem would be useful to fix the issues. Such profiling of memeory, we should be able to do in a running container inside a pod to understand the full details. Let's look at step by step how to profile and monitor memory usage in a .NET app running in a Linux pod in AKS.

Expectaion is to have the ability to monitor live counters as shown below.

And even creating a dump for dontnet process so we can analyze the dump file in Visual Studio.

How to do

The first step is to get the pod name. You can use kubectl get pod for this purpose. For example see below. You can use grep for further filtering, but it is not mandatory.

kubectl get pod -n demo | grep video-messageprocessor

Then select one of the pods and exec into it with below command.

kubectl exec -it podname -n namespace -- bash

Then in the pod container we can run below command to see the memery usage in bytes.

# cgroup v2 (modern AKS Linux nodes); falls back to v1 if needed cat /sys/fs/cgroup/memory.current 2>/dev/null \ || cat /sys/fs/cgroup/memory/memory.usage_in_bytes

To get per process memory usage we can use below command in the pod container shell.

for pid in /proc/[0-9]*; do p=${pid##*/} read _ rss _ < <(awk '{print $1, $2, $3}' "$pid/statm" 2>/dev/null) || continue cmd=$(tr '\0' ' ' < "$pid/cmdline" 2>/dev/null) printf "%6s %10d KB %s\n" "$p" "$((rss*4))" "${cmd:-[$(cat $pid/comm 2>/dev/null)]}" done | sort -k2 -n -r

Or we can use much simpler way using below command.

for pid in /proc/[0-9]*/status; do awk '/^Name:|^Pid:|^VmRSS:/ {printf "%s ", $2} END {print ""}' "$pid" done | sort -k3 -n -r | head

To further analayze the .NET process memeory usage we can install dotnet tools in the pod container. Execute each of below command one by one in the pod container shell.

# Check what package manager exists (Linux only) ls /usr/bin/apt /usr/bin/apk /usr/bin/microdnf 2>/dev/null # Debian/Ubuntu base. 'tar' is required so 'kubectl cp' works in step 4.3. apt-get update && apt-get install -y procps curl tar # Install dotnet-counters / dotnet-gcdump / dotnet-dump (linux-x64 builds) # (SDK not required - these are self-contained single-file tools) curl -sSL https://aka.ms/dotnet-counters/linux-x64 -o /tmp/dotnet-counters curl -sSL https://aka.ms/dotnet-gcdump/linux-x64 -o /tmp/dotnet-gcdump curl -sSL https://aka.ms/dotnet-dump/linux-x64 -o /tmp/dotnet-dump chmod +x /tmp/dotnet-*

Now we have the dotnet counters setup in our pod container. First we should find the dotnet program id (PID). If we use entrypoint as  ENTRYPOINT ["dotnet", ...]  the PID should be 1. In that case we can just create varaible with 1 as shown below.

DPID=1 cat /proc/$DPID/comm # should print: dotnet

Or we can find the PID of dotnet process with below command.

# Find the dotnet PID (exact match on /proc/<pid>/comm; stops at first hit) DPID= for d in /proc/[0-9]*; do [ "$(cat "$d/comm" 2>/dev/null)" = "dotnet" ] && { DPID="${d##*/}"; break; } done echo "dotnet pid=${DPID:-<not found>}" [ -n "$DPID" ] || echo "No dotnet process found" >&2

Then to view the live counters and monitor them for the dotnet process we can run below command in the pod container shell.

# Live counters (Ctrl+C when done) /tmp/dotnet-counters monitor -p $DPID --counters System.Runtime

This will start a monitoring view of live counters for the dotnet process.

Press Ctrl+C when done monitoring and it will go back to the pod container shell.

To collect a dump we can use below command. However, be cautios as dotnet-gcdump triggers a full blocking GC and can pause the process for seconds. Run it during a maintenance window or on a non-prod replica when possible.

# GC heap snapshot /tmp/dotnet-gcdump collect -p $DPID -o /tmp/heap1.gcdump

Now type exit to exit the container shell and back to WSL in your workstation. cd into a known folder with your workstation wsl if you are not sure, in which folder you are located in wsl. Use ls -l  if you need to see the current folder contents.

# Preferred (requires 'tar' inside the pod - installed in step 4.1) kubectl cp namespace/podname:/tmp/heap1.gcdump ./heap1.gcdump # Fallback if 'tar' is not available in the container kubectl exec -n namespace podname \ -- cat /tmp/heap1.gcdump > heap1.gcdump

Then you will be able to see the dump file in your local windows folder.

Open this file with Visual Studio to analyze further.