Setup Test Farm as Azure VMs–For TFS2015 Release Management

A test farm which can run test parallel, using more than one test client can improve the time taken to execute the automation test plan. To use resources only on demand setting this up in Azure Virtual machines would be a great option. Let’s look at how we can setup, a test farm, to use with TFS 2015 (TFS 2015 on-premise, update 2.1 onwards).

A fully setup test farm with release management would look like below.(Deployment environments in the release pipeline not shown here for brevity)

Test Farm – Start This environment runs on release agent locally (in corporate network). Expand Variables allows usage of a variable with another configuration variable in release management. Download Artifacts is used to limit the output getting downloaded, to scripts required for starting the test farm in Azure. CreateReleaseRunBlobContainer is used to setup a temporary blob container in azure to allow monitoring of all parallel test machines completion of assigned test execution. Azure resource group deployment step will start the test farm virtual machines, and the wait poweshell script will wait specified number of seconds, after machines started for them to be fully ready. Each of above will be discussed in detail later.

Run Test These environments will run tests parallel in azure VMs. To achieve this, one admin VM is setup in azure with multiple release agents, which deploys test agent to test client. Each test client VM will run a test agent.

Test Farm – Stop This will monitor the blob on azure to determine, if all test environments done with test executions and then shutdown the Azure VMs.

When create release with definition a message popup to say that agents not available for Run Test, this is because Azure machine containing the agent is not started. But this will be OK since the first environment starts Azure VMs.

Once setup fully The test farm can be started on demand and execute tests and shutdown once the test execution done.

In above Test Farm Started, and Tests executed and the Farm Stopped. The Test run steps show as rejected, but actually they are executed fine, only some tests failed marked it as a failed environment (partially succeeded for environment, feature available in VSTS is not yet available for on premise TFS). But this is expected in test run and continue on error for test execution, will enable to run all the tests and after test steps in the given environment (This will e discussed in detail in a later post).

Let’s look at how this can be setup in detail in following posts.

1. Create Virtual Network in Azure
2. Create Virtual Machines in Azure for the Test Farm. A machine to run release agents (admin machine) and any number of test client machines need to run parallel. All the machines should belong to same workgroup.
3. Setup release agents in the admin machine in Azure, to use with each test client following instructions in here.
4. Enable PowerShell remoting in each test client and verify access from the test farm admin VM in Azure.
5. Setup Test Farm, Start and Stop, and Test Clients for Test Execution.

Enable PowerShell Remoting Between Azure VMs in a Workgroup

Following is a guideline on setting up PowerShell remoting in an azure VM so that another azure VM in same Virtual Network, is able to execute PowerShell commands on the remoting enabled virtual machine. This will allow to setup as a test farm to run with visual studio team service or with TFS (will be discussed in a future post).

1. Assign internal static IP address for the virtual machine(s). You can do this by going to Virtual Machine setting in Azure Portal.

Then select the network interface, IP Configurations

Set the assignment to static. This will make the internal IP unchanged, when the machine is restarted.

Doing it with PowerShell explained here.

2. Execute the command to enable PowerShell remoting in the virtual machine. (run PS as administrator)

Enable-PSRemoting

This will throw an error which is bit of misleading and not providing enough information. If your virtual machine is not in a domain this might occur. More details discussed in below link.

https://4sysops.com/archives/enable-powershell-remoting-on-a-standalone-workgroup-computer/

This can be fixed by running the command with –SkipNetworkProfileCheck switch.

Enable-PSRemoting –SkipNetworkProfileCheck

3. From another virtual machine in the same virtual network in azure, you should be able to enter into a remote PowerShell session, and execute commands in the remote machine. But again this fails with another error.

Enter-PSSession –ComputerName <remotemachinenameorip> –Credential <remotemachineadminuser>

Enter-PSSession -ComputerName do-tf-tc02 -Credential do-tf-tc02\doadmin

Connecting to remote server do-tf-tc02 failed with the following error message : The WinRM client cannot process the request. If the authentication scheme is different from Kerberos, or if the client computer is not joined to a domain, then HTTPS transport must be used or the destination machine must be added to the TrustedHosts configuration setting. Use winrm.cmd to configure TrustedHosts. Note that computers in the TrustedHosts list might not be authenticated. You can get more information about that by running the following command: winrm help config. For more information, see the about_Remote_Troubleshooting Help

4. To fix this issue, add remote machine name to trusted hosts in the client of remote PS.

Set-Item -Path WSMan:\localhost\Client\TrustedHosts –Value <remotemachinenameorip>

Set-Item -Path WSMan:\localhost\Client\TrustedHosts -Value do-tf-tc02

With this PS session can be obtained.

5. Test the PS remoting. Create directory to verify the session is working.

Folder get created in the remote machine.

Restart Widows Services–VSTS/TFS Release Management Task - Chamindac.vsts.release.task.restart-win-service

This release task can be used with Visual Studio Team Services or With TFS 2015.2.1 onwards, to restart a windows service with all its dependent windows services, currently running. Source code available in github.

Task requires one mandatory parameter Service Name. For this parameter Service Name or Service Display name can be provided.

Once run it will stop all dependent services that are running currently, and restart the main service, then start the dependent services stopped earlier. The dependent services that were not running when the restart request made, will not be started, so that the system services state will remain same after restart service script run.

Known Issue

If the agent is running with non administrator account like NetworServices account, task will fail with an error. For example attempting a restart of w3svc might fail with below error.

Service 'World Wide Web Publishing Service (w3svc)' cannot be stopped due to the following error: Cannot open w3svc service on computer '.'

Download Artifacts–VSTS/TFS Extension Updated–Issue Fix and Enhancements

The VSTS/TFS Release extension Download Artifacts, had an issue of failing if there is spaces in the build definition name or build drop UNC paths. This issue has been fixed with the latest version 1.1.15.
Updating from version 1.0.18 to 1.1.15 requires to edit the release definition and save it to, fix parameter not found issues, for removed input parameters.
Issue
https://github.com/chamindac/vsts.release.task.download-artifacts/issues/1

Fixed

Enhancements
Provided facility to download sub folders or files, without downloading the entire artifact folder. Source build information is now automatically picked. No Need to provide additional parameters for it.
Artifact Names/Paths to Download separated by ;. Example *(default) as artifact names allow all artifacts download. Multiple artifacts can be specified with pattern Drop1;Drop2. Artifact name with sub path allows to download sub item of a given artifact. Multiple can be specified with pattern Drop1\MyWebProj;Drop1\ReleaseNote.html;Drop2;Drop3\MyWebProj2 . Wild cards in paths are NOT SUPPORTED. * value as default supported to specify all artifacts.

Chamindac.vsts.release.task.download-artifacts is Now Public in VS Marketplace!

The first extension made by me, “Chamindac.vsts.release.task.download-artifacts” is now public in VSTS Marketplace.

Getting started details can be found in this post “VSTS Release Task – Download Artifacts – Filter for Given Artifact Name(s)”

Why this extension is required is explained in post “Multiple Build Artifacts–TFS 2015/VSTS Builds”.

Source code for this extension is available in Github.

VSTS Release Task – Download Artifacts – Filter for Given Artifact Name(s)

This extension “Chamindac.vsts.release.task.download-artifacts” for VSTS and TFS 2015.2.1 upwards, can be used to filter for build artifacts by name, and download only required artifact(s), to the release agent machine. The requirement for this extension is explained in post “Multiple Build Artifacts–TFS 2015/VSTS Builds”. Source code for this extension is available in Github.

Extension “Chamindac.vsts.release.task.download-artifacts” can be downloaded/installed from VSTS Marketplace. To install to VS Team Services click on install, or download for on-premise TFS 2015 (TFS 2015.2.1 upwards supported).

Once installed, in release definition tasks a new task will be available under Utilities.

Always add this task as first task of the release environment and make sure the environment is set to skip artifact download.

The tasks comes with default filled values to behave as normal Download artifacts.

Build artifact names should be specified in ; separated syntax. Or it can be a single artifact name. Or * for all Artifacts.

Artifact destination is set default to behave as similar to, out of the box, build artifact download.

This task will filter the build artifacts with given names and download only the specified ones.

Multiple Build Artifacts–TFS 2015/VSTS Builds

It is possible to create multiple drops in a new TFS 2015 build or in VS Team Services build. This is useful when you want to do a single build but want to have separate drops(artifacts), for different deployment purposes.

One such example is

1. Artifact Drop for deployment of a website
2. Artifact Drop to contain test automation dlls, and required scripts to run test automations, after successful deployment.

In this situation the deployment happens in a web hosting server and the release agent used there is not going to run the test automations. The agent running test automations will be a separate agent, configured in test client(s), administering machine.

Example 2

1. Drop to deploy web server
2. Artifact Drop to deploy to a CRM server or to a Database Server

Same as above it could be two different release agents configured in two different machines, which does not need others build artifact.

But with the above requirement in mind, it is always advisable to build all components, in a single build to make sure the version is properly built and in sync.

Is it possible to have multiple build artifacts in a single build?

Yes. Let’s look at how we can do that. It can be done by, adding more publish or copy publish tasks to your build definition.

One drop can be in File Share while other can be in Server (Can have both in File Share or both in Server as well).

When a build is done with the above settings, two build artifacts get created for the build.

Next Challenge: Download the only required artifact to the relevant release agent.

With the out of the box functionality in TFS 2015.3 or with VS Team Services currently this is not possible. It does not allow to filter for a given artifact when downloading. Only option is skip download (no artifacts download) or download all artifacts.

When the above skip download is unchecked all artifacts get downloaded to agent. No filtering possible.

How to download only required artifact?

A new extension with capability to filter for named build artifacts and download only them is created. The code is available in here. The extension “Chamindac.vsts.release.task.download-artifacts” is available in VSTS Marketplace. Usage of extension is explained in this post “VSTS Release Task – Download Artifacts – Filter for Given Artifact Name(s)”

Deploy .dacpc to Azure DB via VS Team Services Release–Using Hosted Agents

To deploy a .dacpac to Azure DB with VS Team Services you can use “Azure SQL Database Deployment” task.

Setup either Azure Classic service endpoint or Azure RM Service endpoint in the, Team Services team project.

In a build or release definition add an “Azure SQL Database Deployment”  task.  Both classic and RM based setting up shown below.

Provide

1. Path to the .dacpac file. (build artifact drop path when used in a release definition)
2. Azure DB server name
3. Azure DB name
4. Azure DB Server user name
5. Azure DB Server user password
6. Specify firewall rule to use AutoDetect and delete the rule once done.

or you have the option of setting firewall rule like below to get it working. Since you are deleting rule once done this is ok. But it is not that secure to allow all IPs like below.

This can successfully deploy the .dacpac file to Azure DB using VS Team Services Release.