Resolving “TF401019: The Git repository with name or identifier xxxx does not exist or you do not have permissions for the operation you are attempting.” in Azure Pipelines git Submodule Checkout

Submodule in Git repos help you to keep the common code modules in a separate repo and utilize in multiple other repos. In the classic and YAML build pipelines you can checkout git submodules as explained in the post “Git Repo Submodule Checkout in Azure DevOps Build Pipelines”. However, you may encounter the below issue while running such build pipelines having to perform a git submodule checkout.

Cloning into 'D:/a/1/s/Infra/AzCLI'...

remote: TF401019: The Git repository with name or identifier InfraCLI does not exist or you do not have permissions for the operation you are attempting.

fatal: repository 'https://dev.azure.com/yourorg/DevOps/_git/yourrepo/' not found

fatal: clone of 'https://yourorg@dev.azure.com/yourorg/DevOps/_git/yourrepo' into submodule path 'D:/a/1/s/Infra/AzCLI' failed

Failed to clone 'Infra/AzCLI'. Retry scheduled

Azure Service Connection via Variable Group in YAML CD/Release Pipeline Templates

In application deployment we may use different Azure subscriptions to setup infrastructure for non-prod and production environment infrastructure most of the time. In classic release pipelines we could easily create a resource group to setup deployment task needs, which should be used in multiples stages/environments such as Dev, QA, staging or production. For such groups when the task group is using Azure App Service Deployment, Azure CLI tasks we could supply the service connection name via a variable that can be stored in a variable group relevant to the given stage/environment scope. In YAML we can use a template to define the common steps and it is ideal if we could refer to variable group, to hold the service connection information related to the relevant stage/environment. However, the possibility to use the variable group variable for service connection, regardless of whether using a template in YAML or Azure deploy steps used directly in pipeline YAML seems to be not working as expected. Let’s look at the problem and alternative solutions.

Resolving “Response status code does not indicate success: 400 (Bad Request)” in NuGet Packages Push to GitHub Packages with GitHub Actions

GitHub packages registry can be use to keep your organization NuGet packages privately or share them publicly. If you create a package in a private repo it would be private to the repo or for the organization if you use an organization. If you keep your packages in a public repo you can share it publicly. Ideally when you build reusable NuGet packages you would build and publish such packages with GitHub action workflow to your GitHub packages. While implementing this you may encounter “Response status code does not indicate success: 400 (Bad Request)” at the push of the package, due to special needs of GitHub based NuGet packages. Let’s look at them in this post.

Enhanced YAML Conversion for Azure DevOps Classic Build Pipelines

Azure DevOps YAML pipelines is the way to implement pipeline as code and version control the pipelines alongside your application code. However, among Azure DevOps users classic build pipelines are still popular due to its simple, visual ability to implement and understand faster etc. Additionally, classic pipeline agent job or individual step could be converted to YAML by using “View YAML” option available in each step/task as well as for a given agent phase. With a recent update to Azure DevOps new feature has been added to enable exporting entire classic build pipeline as YAML replacing the “View YAML” feature. This is a great improvement feature added to Azure DevOps as “View YAML” feature was only able to consider information in web UI when generating YAML out of classic pipeline, which at times generated incorrect YAML. With the new export YAML feature, now you can derive a reliable YAML implementation of a given classic pipeline. This would help people to implement pipeline quickly using classic pipeline UI and convert and obtain a YAML, which would help pipeline as code adoption easier. Let’s look at how we can use this new export to YAML feature.

Resolve the “Entity Framework tools version '3.1.9' is older than that of the runtime '5.0.0' in Visual Studio Package Manager”

.NET 5 is released recently and you can use entity framework core 5 with your projects by setting up the version 5.0.0 of the packages, Microsoft.EntityFrameworkCore, Microsoft.EntityFrameworkCore.Design, Microsoft.EntityFrameworkCore.SqlServer, Microsoft.EntityFrameworkCore.Tools if you are using SQL server as the database. However, when you execute dotnet ef commands in the package manager console in Visual Studio you might see the message “Entity Framework tools version '3.1.9' is older than that of the runtime '5.0.0' in Visual Studio Package Manager”. Let’s see how we can get it fixed.

Deploying .NET 5 Web App via GitHub Actions to Azure App Service

.NET 5 is released recently and now you can develop applications using .NET. While releasing .NET five with zero delays Azure App Services started supporting .NET 5 with Early Access. This is helping us to deploy our application code, without having to deply them as self-contained with framework. Meaning we can now deploy our .NET 5 Web Apps, APIs or Function Apps to Azure, just as published project binaries and things will start to run without any issues. In this post let’s look at how we can setup a quick build deployment pipeline deploying a .NET 5 web application to Azure using GitHub Actions.

Obtaining Build Number, Build Id etc. of Another CI Build Pipeline in the CD/Release YAML Pipeline

Let’s look at how we can specify the artifact version to download based on the resource CI build linked to the CD pipeline and check on how to obtain the resource CI build pipeline information such as build number, build id definition id etc., in the CD YAML pipeline.

Implementing a CD YAML Pipeline with Deployment Job

We have discussed couple of things related to implementing CD with YAML in the posts “Trigger Deployment YAML Pipeline Once YAML Build Completed” and in “Prevent Checking Out the Repo in CD YAML Pipeline”. However, the checkout of the repo only happens if use the normal job syntax for deployment job as well. Ideally, we should be using job of type of “deployment” in our CD YAML pipelines so the checkout of repo would not happen by default as well as it will be required to define an environment for deployment. The environment helps us to keep track of our deployments. When we implement the CD YAML with deployment jobs download of artifacts would be automatic. Let’s find out how the deployment job implementation looks like.

Selecting Particular Source CI YAM Pipeline Build While Manually Triggering a YAML CD/Release Pipeline

In classic release pipelines in Azure DevOps you can pick which build to use in the linked build artifacts, easily at the create time. The new norm of implementing pipelines with Azure DevOps is becoming YAML way, allowing to keep your pipelines as code and version controlled. How to Separate CI and CD considerations to individual YAML pipelines was discussed in the post “Trigger Deployment YAML Pipeline Once YAML Build Completed”. Therefore, it is important to know how we can select a particular build, when we trigger a deployment/CD YAML pipeline manually. Let’s compare a classic pipeline triggering manually and YAML CD pipeline triggering manually to understand the difference.

Resolving “System.IO.IOException: The response ended prematurely.” in HttpClient Request

Some times a silly mistake can waste lot of time of a developer. The exception “System.IO.IOException: The response ended prematurely.” while making an http client request to call an API from another web app has wasted considerable amount of my time diagnosing the issue. Therefore, thought worth sharing the experience.

Prevent Checking Out the Repo in CD YAML Pipeline

In the previous post “Trigger Deployment YAML Pipeline Once YAML Build Completed” (https://chamindac.blogspot.com/2020/10/trigger-deployment-yaml-pipeline-from.html) we have discussed how to separate the deployment pipeline from the build pipeline, in Azure DevOps YAML pipeline implementations and to trigger a CD pipeline from the CI build once completed. However, if you check carefully that post implementation of CD it is still checking out the code repository, even though it is not necessary to checkout code being the deployment pipeline. Let’s see how we can avoid checking out repo in the CD YAML.

Trigger Deployment YAML Pipeline Once YAML Build Completed

Now it is possible to implement multi stage pipelines with YAML facilitating the implementation of deployment pipelines as well with YAML instead of classic release pipelines in Azure DevOps. However, having build and deployment steps all together in a single pipeline script is not ideal as it looks bit not a nice implementation from my point of view. With the possibility of triggering another YAML pipeline based on completion of another YAML pipeline, it is possible to separate the Build and Deployment concerns into two different YAML scripts implementing two different pipelines. Let’s have a quick look at how we can trigger a YAML deployment pipeline based on another YAML build pipeline.

Set Work Item State on Pull Request Completion

Associated work items help to identify what is completed in a given pull request. If required while completing a pull request you could, set the associated work item to be moved to completed state. With the introduction of new feature, which we are discussing in this post, you would be able to set the work item state (regardless of it is associated work item or not for the pull request) to desired closed, in progress or resolved states based on description specification.

Azure Terraform Infra as Code Deployment via Custom PowerShell with Azure DevOps Pipelines – Part 2 – Execute Plan with Approval

In the previous post “Azure Terraform Infra as Code Deployment via Custom PowerShell with Azure DevOps Pipelines – Part 1 – Create Plan” we have discussed how to generate a terraform plan targeting Azure Infrastructure deployment and upload it to an Azure Git repo. The solution is implemented instead of using terraform task for Azure DevOps, which is available with Microsoft DevLabs extension due to it is having a prerequisite of Azure resource group, storage etc. as described in the post “Why Azure DevOps Terraform Extension Task by Microsoft DevLabs to Deploy Infra to Azure Does Not Work for Me”. As the second part of previous post, let’s explore the steps require to approve the terraform plan and get the plan executed with Azure DevOps pipeline relying on a state kept in Azure Git repo instead of a storage blob, which is eliminating the need of having manually created Azure resources.

Azure Terraform Infra as Code Deployment via Custom PowerShell with Azure DevOps Pipelines – Part 1 – Create Plan

As described in the post “Why Azure DevOps Terraform Extension Task by Microsoft DevLabs to Deploy Infra to Azure Does Not Work for Me”, the Microsoft DevLabs task to plan and apply terraform infrastructure as code, is demanding to store state in Azure blob storage, which requires to create Azure resources manually as prerequisite of using the task. In this post let us look at a custom implementation of terraform plan and apply task utilizing PowerShell, while storing terraform state and plans in Azure Git repo and have an approval in between the plan and apply steps, to enhance the deployment workflow.

Why Azure DevOps Terraform Extension Task by Microsoft DevLabs to Deploy Infra to Azure Does Not Work for Me

Terraform is used as declarative code for infrastructure deployments on multiple cloud platforms including Azure. Azure DevOps provides capability to execute Infrastructure as code (IaC), with CI/CD pipelines, by using tasks added with the extension by Microsoft DevLabs (https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks) available in the marketplace. The terraform task to plan or apply a terraform plan demands to have reources in Azure such as a resource group, storage account, container and a key (terraform state file path). Similar demands are there for even AWS and GCP for when using the task.

Authorizing Terraform to Apply Changes to Azure Using SPN

We have discussed setting up a Windows 10 environment to develop terraform scripts in previous post. Let’s understand how to authenticate terraform to deploy infrastructure on Azure platform using a service principle with this post.

Getting Started with Terraform for Azure in Windows 10

Terraform is a great way to setup infrastructure as code (IaC) for Azure. Terraform helps us to codify and version control our infrastructure needs in multiple platforms, hence, making learning terraform for IaC for Azure would let an individual to easily get adapted to other platforms such as AWS. IN this post let’s have a quick look at preparing a Windows 10 machine to get started with terraform.

Change Routing of Azure Function Apps

When you implement functions in Azure by default the routing is the https://functionappname/api/functionname . However, this implementation would not let you proper organizing of routing when you have multiple function apps in your software application project. You might want to create custom routing to make your function access from other application organized appropriately. Let’s look at default behaviors and how we can setup custom routing.

How to Run GitHub Actions Step When a Failure Occurred in a Previous Step

GitHub Actions are the CI/CD workflow implementation tool built into GitHub repos. While using the GitHub Actions workflows you may want to execute a cleanup, or rollback or even a ticket(issue) creation task in a situation where a job step is failed. In Azure DevOps pipelines each task had control support to easy implantations of the run on failure need. Let’s look at what it is in Azure DevOps then understand how we can achieve same goals in GitHub actions workflow steps.

Allow Azure Services on SQL Server with Azure CLI

Allow Azure services on Azure SQL Server lets other Azure Services such as function apps, app service apps etc. to be connected to an Azure SQL Server without needing to allow the outbound IPs of such services. You can enable this easily using the portal. Let’s look at how we can Allow Azure services suing CLI.

Git Repo Submodule Checkout in Azure DevOps Build Pipelines

Submodule in Git repos help you to keep the common code modules in a separate repo and utilize in multiple other repos. When you clone the git repo you can include submodules by using git clone --recurse-submodules. In Azure pipelines you can enable checking out the code with submodules for build and package purpose. Let’s have a look at the settings to enable submodule checkout in builds.

Cross Repo Branch Policies in Azure Git Repos

Azure Git repos provide protection to branches with branch policies. The cross-repo branch policy in a team project now lets you define policies applicable to a branch pattern, where it would even be applied to future branches which are adhering to the specified pattern. Let’s explore this feature in bit of detail.

Azure Web App Creation with Azure CLI --runtime Specification Issues in PowerShell Scripts

PowerShell scripts and Azure CLI is a good combination to use for creating infrastructure as code targeting Azure platform. When creating an Azure app service app on Linux, you need to provide the --runtime argument specifying the web app runtime or the platform of the source code getting deployed. In a PowerShell window the command with --runtime argument fails, since a piping symbol is used in runtime arguments.

Deploying Infrastructure to AWS Lightsail Using Azure DevOps – Part 3 – Creating a Release Pipeline to Deploy AWS Lightsail Infrastructure

In the previous two post, we have discussed how to write a bash script with AWS CLI to create AWS Lightsail instance and, how to setup a service connection in Azure DevOps for AWS. IN this post, lets explore the steps required to get the bash script executed in a release pipeline to create AWS Lightsail infrastructure.

Deploying Infrastructure to AWS LIghtsail Using Azure DevOps – Part 2 – Creating a Service Connection

In the previous post, we have discussed how to write a bash script with AWS CLI to create AWS Lightsail instance. In order to run this script to create AWS Lightsail instance via Azure DevOps we need to make a service connection to AWS from Azure DevOps. Let’s look at the steps to create such service connection.

Deploying Infrastructure to AWS LIghtsail Using Azure DevOps – Part 1 - Writing IaC Script with bash

AWS Lightsail is easy to use cloud platform services by Amazon Web Services. You can use aws command line capabilities with Azure DevOps to deploy the required infrastructure on AWS Lightsail. Let’s look at step by step to see how we can get AWS Lightsail deployments via Azure DevOps.

Copy Azure Git Repo Branch Policies from One Branch to Another

Some teams when they practice Agile sprints keep a branch for the life of the sprint. In this case they do branch to develop features, make pull requests to sprint branch and release from the sprint branch at the end of the sprint. It is important to protect the sprint branch from incoming pull requests with policies. Since, there is no out of the box way to copy over the policies defined in one branch to another, each sprint the policies need to be created for the new sprint branch, which makes it a cumbersome process. In order to make it possible to copy set of policies defined in a branch to another, in Azure Git repos, now you can use the script made available here.

Pushing NuGet Packages to Azure DevOps Artifact Feeds Manually

In the build pipelines of Azure DevOps we can easily push a NuGet package, using a NuGet push step and selecting the artifact feed, in the Azure DevOps organization or team project. But you may sometimes need to push packages manually to Azure Artifact feeds. Let’s look at how we can do that.

Running EF Commands in Builds with .NET Core 3.1 in Hosted Agents

When you run builds with Entity Framework (EF) commands such as dotnet ef migrations script with .NET Core 2.2 it would work without any issue. However, if you upgrade your projects to use .NET Core 3.1 your build may fail with issue below., when executing dotnet ef migrations script, to generate a script out of your EF migrations.
error NETSDK1045: The current .NET SDK does not support targeting .NET Core 3.1. Either target .NET Core 2.2 or lower, or use a version of the .NET SDK that supports .NET Core 3.1.

Deploying Machine Learning (ML) Model with Azure Pipeline Using Deployable Artifact from Build

We have discussed how to create a Machine Learning (ML) model as a deployable artifact in the post “Training Machine Learning (ML) Model with Azure Pipeline and Output ML Model as Deployable Artifact” which is based on the open source ML repo, (https://github.com/SaschaDittmann/MLOps-Lab.git) with data by Sascha Dittmann, which also contains the code to train a model.

Training Machine Learning (ML) Model with Azure Pipeline and Output ML Model as Deployable Artifact

Training a machine learning model requires wide range of quality data to get the ML model trained in such a way that it can provide accurate predictions. Azure build pipeline can run the python tests written to validate the data quality and the train a model with uploaded data to Azure ML workspace. In this post on “Setup MLOPS workspace using Azure DevOps pipeline” it is clearly explained how to setup an Azure ML workspace in a new resource group dynamically with Azure CLI ML command extension. The post “Setup MLOPS workspace using Azure DevOps pipeline” as well as this post on training a model with Azure pipelines use the open source ML repo (https://github.com/SaschaDittmann/MLOps-Lab.git) with data by Sascha Dittmann, and code to train a model.