Windows AKS Scaled Jobs Handle Graceful Termination for dotnet App using IHostedService When Preemption

 We have discussed "Gracefully Shut Down dotnet 8 IHostedService App - Deployed as a Windows Container in AKS - While Scale In or Pod Deallocations" previously. The approach  works fine for pods deployed as a deployment in kubernetes. Similary to the deployment pods, scaled job pod can be terminated abruptly due to preemption in kubernetes, if a high priority pod is scheduled. One way to reolve abrupt termination of scaled job pod due to preemption, would be to assign all scaled jobs to highest possible priority. However, setting highest prority to all scaled jobs is not a good solution, as the job may not require highest priority and job should be able to scheduled after other high priority app pods. Let's look at a better solution that can be implented with pre stop hook for scaled jobs running with base docker image Windows server core 2022.

For scaled job, which is implemented to execute one job for a queue message, and run to completion, would start app shut down and will wait for job to be competed execution, then will complete the contianer and  the pod. If such pod dotnet process should not be sent with a second ctrl+c, as we have done in "Gracefully Shut Down dotnet 8 IHostedService App - Deployed as a Windows Container in AKS - While Scale In or Pod Deallocations". Instead, we should use only a wait till dotnet process end, in pre stop hook using a PowerShell script as shown below.

Copyable code is below.

param(
    [Int]$waitBeforeShutDownSignal
)

Start-Sleep -Seconds $waitBeforeShutDownSignal;
$dotnetProcess = Get-Process -Name 'dotnet';
$dotnetProcess.WaitForExit();

The above script should be created as a script file, and copied to base contianer image while building scaled job docker image similar to what we have done in the solution of  "Gracefully Shut Down dotnet 8 IHostedService App - Deployed as a Windows Container in AKS - While Scale In or Pod Deallocations"

Then we can use it in scaled job deployment manifest to setup the pre stop hook as shown below.

terminationGracePeriodSeconds: 3600
containers:
  - name: ${aks_app_name}$
    lifecycle:
      preStop:
        exec:
          command: ["powershell.exe","-File","C:/app/wait_windows_container_gracefully.ps1","-waitBeforeShutDownSignal","10"]

This will ensure the preemtion request will not terminate the scaled job, until the termination grace period is over, or the job execution completed. If the termination grace period is setup sufficiently, the job execution to the completion can be guranteed.