Setup Test Farm as Azure VMs–For TFS2015 Release Management

A test farm which can run test parallel, using more than one test client can improve the time taken to execute the automation test plan. To use resources only on demand setting this up in Azure Virtual machines would be a great option. Let’s look at how we can setup, a test farm, to use with TFS 2015 (TFS 2015 on-premise, update 2.1 onwards).

A fully setup test farm with release management would look like below.(Deployment environments in the release pipeline not shown here for brevity)

Test Farm – Start This environment runs on release agent locally (in corporate network). Expand Variables allows usage of a variable with another configuration variable in release management. Download Artifacts is used to limit the output getting downloaded, to scripts required for starting the test farm in Azure. CreateReleaseRunBlobContainer is used to setup a temporary blob container in azure to allow monitoring of all parallel test machines completion of assigned test execution. Azure resource group deployment step will start the test farm virtual machines, and the wait poweshell script will wait specified number of seconds, after machines started for them to be fully ready. Each of above will be discussed in detail later.

Run Test These environments will run tests parallel in azure VMs. To achieve this, one admin VM is setup in azure with multiple release agents, which deploys test agent to test client. Each test client VM will run a test agent.

Test Farm – Stop This will monitor the blob on azure to determine, if all test environments done with test executions and then shutdown the Azure VMs.

When create release with definition a message popup to say that agents not available for Run Test, this is because Azure machine containing the agent is not started. But this will be OK since the first environment starts Azure VMs.

Once setup fully The test farm can be started on demand and execute tests and shutdown once the test execution done.

In above Test Farm Started, and Tests executed and the Farm Stopped. The Test run steps show as rejected, but actually they are executed fine, only some tests failed marked it as a failed environment (partially succeeded for environment, feature available in VSTS is not yet available for on premise TFS). But this is expected in test run and continue on error for test execution, will enable to run all the tests and after test steps in the given environment (This will e discussed in detail in a later post).

Let’s look at how this can be setup in detail in following posts.

1. Create Virtual Network in Azure
2. Create Virtual Machines in Azure for the Test Farm. A machine to run release agents (admin machine) and any number of test client machines need to run parallel. All the machines should belong to same workgroup.
3. Setup release agents in the admin machine in Azure, to use with each test client following instructions in here.
4. Enable PowerShell remoting in each test client and verify access from the test farm admin VM in Azure.
5. Setup Test Farm, Start and Stop, and Test Clients for Test Execution.