We have discussed "Windows AKS Scaled Jobs Handle Graceful Termination for dotnet App using IHostedService When Preemption" previously. However, for Linux scaled jobs we have to handle the graceful termination, using a bash script intead of a PowerShell script.
We can create a script with bash as shown below.
We have discussed "Gracefully Shut Down dotnet 8 IHostedService App - Deployed as a Windows Container in AKS - While Scale In or Pod Deallocations" previously. The approach works fine for pods deployed as a deployment in kubernetes. Similary to the deployment pods, scaled job pod can be terminated abruptly due to preemption in kubernetes, if a high priority pod is scheduled. One way to reolve abrupt termination of scaled job pod due to preemption, would be to assign all scaled jobs to highest possible priority. However, setting highest prority to all scaled jobs is not a good solution, as the job may not require highest priority and job should be able to scheduled after other high priority app pods. Let's look at a better solution that can be implented with pre stop hook for scaled jobs running with base docker image Windows server core 2022.
The pods deployed to AKS gets terminated due to reschduling, low priority evictions as well as during scaled in. We can add a termination grase period and pre stop sleep time as shown below in Linux and Windows containers to allow, sufficient time to ingress services to get updated about terminating pods. However, Windows nanoserver does not support PowerShell. Therefore, we need to use a specific mechanism to pre stop hook for nanoserver images. For nanoserver images shutdown signal is correctly get sent to the dotnet app. So we can just setup a sllep time for pre stop hook as necessary.
Applications implemented with IHostedService in dotnet, deployed to Azure Kubernetes Services (AKS) as containers in pods get terminated when pod recheduling happens or scaling-in opertaions happen. However, unlike Linux containers, the Windows containers does not receive the signal (similar to SIGTERM or SIGINT) to graceful shutdown. Once the pre stop hook is done the container is immediatly killed disregarding the value set in the terminationgraceperiod. Since, the Windows container did not receive a message to start a graceful shut down, and it is killed abruptly, the in flight operations in the Windows app container are abandoned. Such abadoning of operations cause inconsitency in system data and cause system failures. Therefore, it is mandatory to implement a proper graceful shutdown for Windows containers as well. Let's explore the issue in detail and how to implement a proper solution to enable graceful Windows container shut down, for dotent apps implemented with IHostedService. The issue is happening in mcr.microsoft.com/dotnet/runtime:8.0-windowsservercore-ltsc2022 images and the solution is tested with the same.
Windows app pod scaled-in or pod rescedule
We generally use Azue container registry to store our application docker images when we use AKS as the ochestrator for our applications. However, piling up of previous releases images, as well as images used for developer teting in Azure container registry increase costs. Therefore it is important to have a periodic cleanup mechanism setup to remove all unused images form the registry. Let's look at a strategy we can use to cleanup Azure container registry.
We discuss how to enable workload identity for continers running in AKS in the post "Setting Up Azure Workload Identity for Containers in Azure Kubernetes Services (AKS) Using Terraform - Improved Security for Containers in AKS". However, when we use DefaultAzureCredentials and try to run docker containers locally from a development machine, we do not have the workload identity support. With Visual Studio we can run with the Azure AD user and run applications successfully. But if we are using a docker run command and run docker container locally, we will have to use app registration/service principal. Then we have to grant the service principal with required roles in Azure resources we would need to access from the application. Let's take a look at an example with Azure app config.
The missing fonts in windowsservercore-ltsc2022 docker images can be installed as described in the blog post here. However, when we are not using hosted agents, and when we use kubernetes based self hosted build agents, we do not have access to host machine, to perform the all steps described in the post "Adding optional font packages to Windows containers". Since docker is not supported on self hosted build agent running as container in AKS, we have to use az acr build to build the docker images in such cases. To setup fonts in this kind of a situation in a Azure DevOps pipeline we can take the steps described in this post.
.NET 8 was release on November 14. There are docker container images for .NET 8 available for dotnet runtime and can be found with tag list here https://mcr.microsoft.com/v2/dotnet/runtime/tags/list . However, if you want to setup .NET 8 runtime on another specific Linux docker image for example on ubuntu:jammy , amd64/ubuntu:22.04 or with a special image such as ffmpeg Linux server image linuxserver/ffmpeg:amd64-version-6.0-cli, where you might want to run your .NET app to use ffmpeg, In such cases, where you you might have to setup .NET 8 runtime on a specific docker image, with your other tools readily available, details mentioned may come in handy. Lets, see how we can install .NET 8 runtime on base Ubuntu 22.04 images, using a docker file.
Let's look at a temporary solution to the issue Intermittent CrashLoopBackOff in Windows Containers Running on AKS (.NET 6 Apps with System.Net.Sockets.SocketException 11001 and 10060). Same issue is asked in stackoverflow here. Instead of manually deleting pods that run into the issue maually, the cleaner app implemented in this repo is doing autmatic deletion of pods CrashLoopBackOff state with known exception reported in the container log. If the exeception is unknown the pod in CrashLoopBackOff state will not be deleted, and the container log output is printed, in cleaner app logs to show the exception of the pod having CrashLoopBackOff state.
.NET app running with mcr.microsoft.com/dotnet/runtime docker alpine images, might run into issue "Microsoft.WindowsAzure.Storage.StorageException: Only the invariant culture is supported in globalization-invariant mode" if your app is requiring globalization. Let's look at what needs to be done to get the issue fixed.
Building .NET 6 project with Resource files is running into build issue as shown below, when building with .NET 6 nanoserver build images.
C:\Program Files\dotnet\sdk\6.0.407\Microsoft.Common.CurrentVersion.targets(3262,5): error MSB4018: The "GenerateResource" task failed unexpectedly. [C:\src\Demo\Demo.csproj]
C:\Program Files\dotnet\sdk\6.0.407\Microsoft.Common.CurrentVersion.targets(3262,5): error MSB4018: System.Runtime.InteropServices.COMException (0x80040154): Retrieving the COM class factory for component with CLSID {7Bxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} failed due to the following error: 80040154 Class not registered (0x80040154 (REGDB_E_CLASSNOTREG)). [C:\src\Demo\Demo.csproj]
As discussed in the post "Run .NET Core 3.1 Application with Windows Server Core 2019 Docker Image" there could be situations you need to setup .NET Core 3.1 on Windows servercore docker image. However, you might have experienced below error running Windows container, when you setup tools such as .NET runtime by yourself using dockerfile.
Let's see one such example case and how it can be fixed with .NET Core runtime 3.1 installed on Windows servercore docker image.
You would love to run your .NET applications as Linux containers instead of using Windows containers. However, there are situations that you have to run your .NET applications as Windows containers with servercore docker image due to your project having dependencies, such as a third party dll (could be even built by different department in your company) which demands your application to run on Windows servercore docker image. Worst case is sometimes your app is still on .NET 3.1 Core you do not have a servercore image for .NET Core 3.1. (For .NET 5,6,7,8 the Windows servercore images are available.. check here) Therefore you need to setup .NET Core 3.1 runtime on the servercore docker image. Let's see how we can achive this requirement.
You might want to do quick test deploying your containers to AKS or Azure App Services, while development is ongoing. I fyou do not have a pipelines setup yet you may need to manually push your docker images to Azure Contianer Registry (ACR). Let's have a look at the steps need to push a local image built to ACR using docker commands.
Adding Azure Container Registry (ACR) service connection to Azure DevOps is really simple as described in "Create Service Connection in Azure DevOps for Azure Container Registry", when you have the same account you are using for Azure DevOps, is associated with your Azure Subscription. However, this may not be the case always and you may want to push docker images to ACR in an Azure Subscription which is not related to your Azure DevOps organization, such as your customer's production Azure subscription. We have discussed one option with basic authentication in the post "Create Service Connection in Azure DevOps for Azure Container Registry - Using Basic Authentication". Let’s see how to create a service connection for ACR in such situation to utilize it in a deployment pipeline using a Service Principal in this post.
Adding Azure Container Registry (ACR) service connection to Azure DevOps is really simple as described in "Create Service Connection in Azure DevOps for Azure Container Registry", when you have the same account you are using for Azure DevOps, is associated with your Azure Subscription. However, this may not be the case always and you may want to push docker images to ACR in an Azure Subscription which is not related to your Azure DevOps organization, such as your customer's production Azure subscription. Let’s see how to create a service connection for ACR in such situation to utilize it in a deployment pipeline using basic authentication.
To push docker contianer images from Azure Pipelines to Azure Container Registry (ACR) you need to establish a connection between the Azure DevOps Team Project and the Azure Continer Registry in the Azure Subscription. General service principal based Azure Resource Manager Service Connection cannot be used to push docker images to Azure Contianer Registry. We need to instead create a service connection of type docker registry in Azure DevOps. Let's look at few simple steps to get such service connction created.
It is simple to setup this service connection if your Azure subscription and the Azure DevOps belong to the same account as described in this post. If you have your Azure subscription account as a different account form your Azure DevOps account refer one of below options to create the service connectiont to ACR.
While setting up docker on Ubuntu 20.04 you might run into below issue permission denied error is shown while trying to execute any docker command.
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /var/run/docker.sock: connect: permission denied
In windows 10 you can now setup Docker Desktop with Linux container, without using a virtual machine for Linux, by using Windows Subsystem for Linux (WSL). Remembering all manual steps for doing this is bit cumbersome, and therefore let’s look at how we can use PowerShell to automate the task of installing Docker Desktop with WSL version 2, on Windows 10.
In the part 1 of this post, enabling Docker support for ASP.NET Core app and building and pushing the Docker image to Azure Container Service, using Azure DevOps build pipeline with simple steps was described. The image is tagged with the build Id and it is pushed to the Azure Container Registry, so that it can be later deployed to a container orchestrator to run the container. Helm is used to get he deployment done to AKS via Azure DevOps when creating a an ASP.NET Core App, Container Registry and AKS, then getting it deployed automatically with few clicks using Azure Projects as described in the post “Deploy ASP.NET Core App to AKS with Azure DevOps Project”. Let’s look at getting the container image in Azure Container Registry deployed to AKS with three simple steps without using Helm, with Azure Pipelines.
