Azure Service Connection via Variable Group in YAML CD/Release Pipeline Templates

In application deployment we may use different Azure subscriptions to setup infrastructure for non-prod and production environment infrastructure most of the time. In classic release pipelines we could easily create a resource group to setup deployment task needs, which should be used in multiples stages/environments such as Dev, QA, staging or production. For such groups when the task group is using Azure App Service Deployment, Azure CLI tasks we could supply the service connection name via a variable that can be stored in a variable group relevant to the given stage/environment scope. In YAML we can use a template to define the common steps and it is ideal if we could refer to variable group, to hold the service connection information related to the relevant stage/environment. However, the possibility to use the variable group variable for service connection, regardless of whether using a template in YAML or Azure deploy steps used directly in pipeline YAML seems to be not working as expected. Let’s look at the problem and alternative solutions.

A service connection to Azure is created as below to enable deploying none production web apps etc. to a given subscription.

Then in a variable group scoped for dev stage a variable is defined to keep the service connection name.

A template YAML which contains steps to do the deployment expects a parameter which is supplied as service connection name to Azure deployment tasks.

Template is used in a deployment pipeline and parameter is passed with the variable defined in the variable group which is referred by the stage.

Trying to execute the pipeline will result in immediate failure with below error.

There was a resource authorization issue: "The pipeline is not valid. Job DeployDev: Step input azureSubscription references service connection $(azureConnection) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job DeployDev: Step input azureSubscription references service connection $(azureConnection) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."

Possible Options

Option 1: Use a release variable

Adding release pipeline variable makes the variable to be applied successfully and it makes the pipeline to execute successfully and deploy to Azure using service connection. However, release variable cannot be scoped to env, so for each environment you may have to use different variable and pass it via pipeline as a parameter to template.

Option 2: Define the variables for service connection in release pipeline level in YAML

You can define the Non-Prod and Pod connection info into two variables as shown below and depending on the stage pas them via parameter to the template in each stage. All other stages might use non-while the production stage may use production Azure connection information variable.

Option3: Define the two variables in for non-prod and prod service connection in Release level Variable Group

You can use a common variable group in the release level, and define required connection information variable for each stage or categorize as non-prod and prod to use only two variables.

Conclusion

It seems YAML pipeline must have the service connection name specified at the scope of release in order to apply it to the Azure deployment tasks. Else the run pipeline will fail immediately with unable to determine the validity of the service connection. This was not the case with classic release pipelines where we were able to define the variable at the stage scope and still it would be applied to the Azure tasks. This seems to be due to YAML pipeline jobs need to verify the existence of service connection before the pipeline is started by validating YAML and for that it might be only refereeing to release scope variables and variable groups.