Cleanup Strategy for Azure Container Registry Based on Azure Pipeline Retained Builds

 We generally use Azue container registry to store our application docker images when we use AKS as the ochestrator for our applications. However, piling up of previous releases images, as well as images used for developer teting in Azure container registry increase costs. Therefore it is important to have a periodic cleanup mechanism setup to remove all unused images form the registry. Let's look at a strategy we can use to cleanup Azure container registry.

As the first step when we build and push images, we have to ensure we are having a tag that can be later identified to cleanup the images from registry. For that we may use build id as the tag. We can use a sningle pipeline as below to build and push all our apps and follow a blue-green deployment strategy (which we can discuss in a later post) deploy AKS infrastructure and applications into production environments.

When we push any app docker image that will be tagged with the build id as shown in below pipeline  tasks.

If using az acr build the buildid can be used for tag as shown below.

 - task: AzureCLI@2
   displayName: 'Build and push docker image'
   retryCountOnTaskFailure: 2
   inputs:
     azureSubscription: 'MyServiceConnection'
     scriptType: pscore
     scriptLocation: inlineScript
     inlineScript: |
       az account set --subscription $(mysubscriptionid)
       az acr build --platform ${{ variables.container_platform }} `
         --registry $(aks_shared_container_registry) `
         --image $(aks_shared_container_registry).azurecr.io/demo/$(aks_app_name):$(Build.BuildId) `
         --file '$(project_path)/Dockerfile' $(Build.Repository.LocalPath)

The above will ensure any image in Azure container registry is having build id as the tag.

We can follow a branch strategy as follows.

branches

• develop - the development branch protected and accept only incoming pull requetsts
• features/* -  branch a devloper or developers work on
• releases/* - branch a release is made from
• archived/* - older release branches

The stratgy is to use feture branches by a developer or developers to do development, and the create a PR to develop branch. The develop branch is used to deploy to dev envoronment, where dev test is poerformed. Then using releases branch a build and deploy to QA, pre production and production environments. This will ensure same docker image QA tested is deployed to next environments (we can discuss more on this in future posts with blue-green depllyment strategy). Once release become older, we create Archived branch from the releases branch and remove the releases branch.

Based on this branching stratgy we can write a script like below to cleanup the Azure container registry, keeping only the docker images tagged with build ids (of all apps pipeline), from releases/* branches and last three build of develop branch. 

$acrName = 'chdemosharedacr';
$allAppsDefId = 284;
$retainedImageTags = [System.Collections.Generic.List[string]]::new();
$skipCleanup = $false;

Write-Host ('Getting releases/* branches...');
Write-Host ('===========================================================');
$releaseBranches = $null;
$releaseBranches = az repos ref list --repository 'aks_blue_green_nginx' --filter 'heads/releases/' --detect | ConvertFrom-Json;

if (($null -ne $releaseBranches) -and ($releaseBranches.Count -gt 0))
{
    foreach ($releaseBranch in $releaseBranches)
    {
        $releaseName = $releaseBranch.name -replace 'refs/heads/releases/','';
        Write-Host (-join('Inspecting release ',$releaseName,' for all_apps pipeline runs...'));
    
        $pipelineRuns = $null;
        $pipelineRuns = az pipelines build list --definition-ids $allAppsDefId --branch $releaseBranch.name --detect | ConvertFrom-Json;

        if (($null -ne $pipelineRuns) -and ($pipelineRuns.Count -gt 0))
        {
            foreach($pipelineRun in $pipelineRuns)
            {
                Write-Host (-join('Adding release ',$releaseName,' all_apps pipeline id:',$pipelineRun.id,' to retained docker image tag list...'));
                $retainedImageTags.Add($pipelineRun.id);
            }
        }
        else
        {
            $skipCleanup = $true;
            Write-Host (-join('The release ',$releaseName,' does not have any all_apps pipeline runs. Set to skip ACR cleanup.'));
        }

        Write-Host ('-----------------------------------------------------------');
    }
}
else
{
    $skipCleanup = $true;
    Write-Host (-join('releases/* branches not found. Set to skip ACR cleanup.'));
}

Write-Host (-join($retainedImageTags.Count, ' image tags will be retained for releases/*.'));
Write-Host ('===========================================================');

Write-Host (-join('Inspecting develop branch for all_apps pipeline runs...'));    
$pipelineRuns = $null;
$pipelineRuns = az pipelines build list --definition-ids $allAppsDefId --branch 'refs/heads/develop' --detect | ConvertFrom-Json;

if (($null -ne $pipelineRuns) -and ($pipelineRuns.Count -gt 0))
{
    Write-Host (-join('Found ', $pipelineRuns.Count, ' of all_apps pipeline runs. Adding only last 3 runs to retained docker image tag list.'));
    Write-Host (-join('Adding develop branch all_apps pipeline id:',$pipelineRuns[0].id,' to retained docker image tag list...'));
    $retainedImageTags.Add($pipelineRuns[0].id);
    Write-Host (-join('Adding develop branch all_apps pipeline id:',$pipelineRuns[1].id,' to retained docker image tag list...'));
    $retainedImageTags.Add($pipelineRuns[1].id);
    Write-Host (-join('Adding develop branch all_apps pipeline id:',$pipelineRuns[2].id,' to retained docker image tag list...'));
    $retainedImageTags.Add($pipelineRuns[2].id);
    Write-Host (-join($retainedImageTags.Count, ' image tags will be retained for releases/* and develop.'));
}
else
{
    $skipCleanup = $true;
    Write-Host (-join('The develop branch does not have any all_apps pipeline runs. Set to skip ACR cleanup.'));  
}

Write-Host ('===========================================================');

if ($skipCleanup)
{
    Write-Warning (-join('ACR cleanup is skipped due to, not found all_apps runs in releases/* or in develop.'));
}
else
{
    Write-Host (-join('Getting all repos from ',$acrName,'...'));
    Write-Host ('===========================================================');

    $acrRepos = az acr repository list --name $acrName | ConvertFrom-Json;

    foreach($acrRepo in $acrRepos)
    {
        if ($acrRepo.StartsWith('demo/'))
        {
            Write-Host (-join('Inspecting ',$acrRepo,' for docker images tags...'));

            $repoTags = $null;
            $repoTags = az acr repository show-tags --name $acrName --repository $acrRepo | ConvertFrom-Json;

            Write-Host (-join('Found ',$repoTags.Count,' docker images tags in acr repo ',$acrRepo,'. Processing...'));

            foreach($repoTag in $repoTags)
            {
                if ($retainedImageTags.Contains($repoTag))
                {
                    Write-Host (-join('The tag ',$repoTag,' for docker image in acr repo ',$acrRepo,' is retained.')) -ForegroundColor Green;
                }
                else
                {
                    $repoImage = $null;
                    $repoImage = -join($acrRepo,':',$repoTag);

                    Write-Host (-join('The docker image ',$repoImage,' is deleting...')) -ForegroundColor Red;
                    az acr repository delete --name $acrName --image $repoImage --yes;
                    Write-Warning (-join('The tag ',$repoTag,' for docker image in acr repo ',$acrRepo,' is deleted.'));
                }
            }

            Write-Host ('-----------------------------------------------------------');
        }
    }
}

The script can be executed with a pipline job as shown below.

jobs:

  - job: cleanup_acr

    workspace:

      clean: all

    displayName: Cleanup ACR

    pool:

      vmImage: ubuntu-22.04

    timeoutInMinutes: 0

    steps:

      - checkout: self

      - task: UseDotNet@2

        displayName: 'Use .NET Core sdk 8.0.x'

        inputs:

          packageType: 'sdk'

          version: '8.0.x'

          installationPath: '$(Agent.ToolsDirectory)/dotnet'

      - task: AzureCLI@2

        displayName: 'Cleanup ACR'

        inputs:

          azureSubscription: 'Azure Chaminda Sponsor'

          scriptType: pscore

          scriptLocation: inlineScript

          inlineScript: |

            $env:AZURE_DEVOPS_EXT_PAT = '$(System.AccessToken)'

           

            az extension add --name azure-devops --allow-preview false --yes

            az extension update --name azure-devops --allow-preview false

            $(System.DefaultWorkingDirectory)/pipelines/scripts/cleanup_acr.ps1

The job can be used in a pipeline.

trigger: none

name: $(SourceBranchName)_$(Date:yyyyMMdd)$(Rev:r)

variables:

  - template: templates/vars/pipeline_vars.yml

jobs:

  - template: templates/jobs/cleanup_acr.yml

We can setup the pipline to execute on a schedule.

Then the periodic cleanup will be executed .