Resolving Azure DevOps Build Error “data at the root level is invalid” in dotnet test

When you try to execute dotnet test with test assemblies in Azure DevOps builds built with Visual Studio 2019 you may run into an error “data at the root level is invalid”. Let’s look at how to solve this issue and get the tests executed in the Azure DevOps build pipelines.

Updating Test Case Work Item Tags and Description based on Test Automation Code Test Category Attributes and Summary Descriptions

A test automation development team has come up with a requirement to keep track of some test automation code related attributes and documentation descriptions in the relevant Test Case work items in Azure DevOps. The documentation description added to each test method in  the code should be captured and updated to Test Case work item summary description. Any test category attribute values should be applied as Tags to the Test Case work item. Let’s look at how we can automate the above requirement using a little bit of PowerShell in combination with Azure DevOps build pipelines.

Using Unified Agent for Executing Automated Tests Without Installing Visual Studio in Test Client–Azure DevOps/VSTS

As described in the post “Running UI Tests with Unified Agent” you can setup unified agent (same agent used for build/deployment in Azure DevOps/VSTS), in an application pool or in a deployment group to execute functional UI tests. The unified agent is really useful as it does not require to setup separate test agent using winRM based Deploy Test Agent task which is required if Run Functional Tests task is used for automated test execution.With unified agent you can use Visual Studio Test task, which is capable of running Selenium based UI tests as well, in addition to Coded UI tests. There was a dependency that you needed to have Visual Studio installed in the test client machine to get the Visual Studio Test task as it is packaged with Visual Studio. However, thanks to the NuGet package “Microsoft Test Platform”  you no longer need to install full Visual Studio in your test client machine to execute automated tests, with unified agent using Visual Studio Test task. Let’s look at how to use “Microsoft Test Platform”  and Visual Studio Test task in a test client using Azure DevOps Release Management.

Running UI Tests with Unified Agent

You needed to setup dedicated machines for testing when using Run Functional Tests and you had to execute, Deploy Test Agent step and Copy Files step as prerequisites. With the introduction of unified agent enabling build, deployment and testing using same agent, now it is possible to setup pool of machines as Test clients to cater multiple project needs. You can even deploy and test using the same agent if you wish to do so.  Unified agent support is available to run tests as multi-machine distributed execution. You do not require to execute copy files task as artifacts downloaded by the agent can be used for running tests. Let’s look at how to prepare a unified agent for running UI tests.

Setup Test Farm as Azure VMs–For TFS2015 Release Management

A test farm which can run test parallel, using more than one test client can improve the time taken to execute the automation test plan. To use resources only on demand setting this up in Azure Virtual machines would be a great option. Let’s look at how we can setup, a test farm, to use with TFS 2015 (TFS 2015 on-premise, update 2.1 onwards).

A fully setup test farm with release management would look like below.(Deployment environments in the release pipeline not shown here for brevity)

Test Farm – Start This environment runs on release agent locally (in corporate network). Expand Variables allows usage of a variable with another configuration variable in release management. Download Artifacts is used to limit the output getting downloaded, to scripts required for starting the test farm in Azure. CreateReleaseRunBlobContainer is used to setup a temporary blob container in azure to allow monitoring of all parallel test machines completion of assigned test execution. Azure resource group deployment step will start the test farm virtual machines, and the wait poweshell script will wait specified number of seconds, after machines started for them to be fully ready. Each of above will be discussed in detail later.

Run Test These environments will run tests parallel in azure VMs. To achieve this, one admin VM is setup in azure with multiple release agents, which deploys test agent to test client. Each test client VM will run a test agent.

Test Farm – Stop This will monitor the blob on azure to determine, if all test environments done with test executions and then shutdown the Azure VMs.

When create release with definition a message popup to say that agents not available for Run Test, this is because Azure machine containing the agent is not started. But this will be OK since the first environment starts Azure VMs.

Once setup fully The test farm can be started on demand and execute tests and shutdown once the test execution done.

In above Test Farm Started, and Tests executed and the Farm Stopped. The Test run steps show as rejected, but actually they are executed fine, only some tests failed marked it as a failed environment (partially succeeded for environment, feature available in VSTS is not yet available for on premise TFS). But this is expected in test run and continue on error for test execution, will enable to run all the tests and after test steps in the given environment (This will e discussed in detail in a later post).

Let’s look at how this can be setup in detail in following posts.

1. Create Virtual Network in Azure
2. Create Virtual Machines in Azure for the Test Farm. A machine to run release agents (admin machine) and any number of test client machines need to run parallel. All the machines should belong to same workgroup.
3. Setup release agents in the admin machine in Azure, to use with each test client following instructions in here.
4. Enable PowerShell remoting in each test client and verify access from the test farm admin VM in Azure.
5. Setup Test Farm, Start and Stop, and Test Clients for Test Execution.

Send Test Result Email After Running Automated Tests with Release Management

Getting automated tests executed after deployment actions, it self is challenging with release management. Below are some helpful links on “how to overcome them”.
Resolve Common Errors - Run Automated Tests with VS Release Management 2013
Disable Lock Screen on Windows 8.1 to Prevent - Automation engine is unable to playback the test because it is not able to interact with the desktop
Increase Default Timeout of Copying Test Binaries to Test Client – Allow Release Management to Run Tests with Increased Timeout
Once everything in place, with a customized TcmExec.ps1, tests get executed with RM.

Wouldn’t it be nice to have an email delivered to the team with the test results like shown below?
To do this it is possible to modify the TcmExec.ps1 (How to get the TcmExec.ps1 is described here), to read the details of the test result file (.trx) and create a formatted email message to sent to the defined recipients. As the first step script is introduced with few more parameters.
Extract the build number from the build location path given that the build path contains it.This is to make Test result email look nicer if the build number is not supplied by build TM server.
Extract test result details into a temp table to format it for the email.
Send email and print result in output log of RM action.
Avoid failing the Release Management Action, even if failing tests found when the email send option is set to true. This will make sure deployment completes and the required team members are informed of the Test Results. and give them the option of proceeding or stopping the release pipeline to next steps, after evaluating test results.
Setup a custom tool in RM using the PowerShell script TcmExec.ps1.
If components are already created with the tool they need to be updated with new parameters manually.

Set up the release template to use the test runner with new parameters. All other common issues mentioned at the linked post in the beginning of this post.

Tests are getting executed in TFS lab environment with RM.
Release management action output log has the details of the test printed.
Email with the test results sent to the defined recipients.
You can download the enhanced TcmExec.ps1 from TechNet Gallery.
https://gallery.technet.microsoft.com/Send-Test-Result-Email-5b4f0d5e

Increase Default Timeout of Copying Test Binaries to Test Client – Allow Release Management to Run Tests with Increased Timeout

Default timeout for downloading test binaries to execute automated tests in a lab environment is 5 minutes. This might not be enough if the test assemblies are high in size, and test client machine is in a different location from the build drop LAN having low bandwidth.
If running via RM server, MTM Test runner, test hangs on active and never get executed.
trx file opened in Visual Studio Ultimate shows the exact issue.
*******************************************************************************************************
Warning    9/25/2015 7:27:10 PM    Warning: Test Run deployment issue: The assembly or module 'Microsoft.Xrm.Sdk.Deployment' directly or indirectly referenced by the test container '\\builddrop\testautomation\xxx.automation.nonui.dll' was not found.   
Error    9/25/2015 7:32:15 PM    Agent vstfs:///LabManagement/TestMachine/36 exceeded deployment timeout period.    vstfs:///LabManagement/TestMachine/36
*******************************************************************************************************
How to resolve
The testcontroller deploys the test binaries and their dependencies from the build drop location to the test agent machines so that test agent can run them. The timeout for that activity is called deployment timeout.
Test setting should be created and set the deploy time out in the test settings, following instructions here. The command to execute is below.
UpdateTestSettings /collection:http://abc:8080/tfs/DefaultCollection /teamProject:myProject /settingsname:My2_0_App /bucketSize:200 /deploymentTimeout:600000
Next step is providing test settings to the test run via the RM server, MTM Test runner. For this TcmExec.ps1 explained here, has a parameter for test settings.
To provide this in the RM tool (custom tool created using the TcmExec.ps1 explained here), add a parameter as shown below.
This will not be applied to existing components created with the RM tool. Update them to have the SettingsName parameter. 
This allows the SettingsName to be supplied to the test via Release Management Template.
Test now executes after successfully downloading the test binaries to the test client.

Disable Lock Screen on Windows 8.1 to Prevent - Automation engine is unable to playback the test because it is not able to interact with the desktop.

You might run into below error with automated testing with lab environments.
*******************************************************************************************************Error calling Initialization method for test class xxxxxxxx: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: Automation engine is unable to playback the test because it is not able to interact with the desktop.  This could happen if the computer running the test is locked or it’s remote session window is minimized.
*******************************************************************************************************

This could be because you have RDP to the test agent machine and you have logged out, minimized the RDP session. This can be prevented by repairing the lab environment and let it restart the test agent machine. Then as long as you do not RDP to test agent it should run automated tests without any issue.
But if your test agent machine is Windows 8 or Windows 8.1 you might still run into the same error, even though you are 100% certain, that after repairing the lab environment, you have not RDP to the test agent machine.
This happens due to the lock screen functionality of Windows 8 and 8.1. to fix this issue disable the lock screen. This can be done in Group Policy Editor. Run gpedit.msc and set the “Do not display the lock screen” to “Enabled” sate.
This will set the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\Personalization with DWord “NoLockScreen” to value 1.
More information on setting lock screen to disable, can be found here.

Disable the screen saver for Test Agent user.
Set “Turn off the display” to never.

In registry set the following DWord “DisableLockWorkstation”. Instructions here.

More instructions in the answer here. Set and verify all of them.

Repair the lab environment again to make sure the restart of the test agent and, as long as no RDP sessions to the test agent machine, automated test will not run into “Automation engine is unable to playback the test because it is not able to interact with the desktop”.

Resolve Common Errors - Run Automated Tests with VS Release Management 2013

There is a tool available in Release Management 2013 to execute automated tests, once the deployment completes.

How to setup this is well explained in article here. http://nakedalm.com/execute-tests-release-management-visual-studio-2013/
With the custom component set as below in release template, you should be able to excute tests.
Below are few issues you might encounter, and steps to fix them.
*******************************************************************************************************
Error 01:
Cannot process argument transformation on parameter
'PlanId'. Cannot convert value "40426" to type "System.Int16". Error: "Value was either too large or too small for an Int16."  Unfortunately this error comes with the component reporting a “succeed”. But log shows the failure. This error is discussed here.

This is obvious error if you have more TFS projects and larger IDs for work items above 32767. You would not be able to use the Microsoft provided tool 
To Resolve:
You need to get the TCM.ps1 from the tool and edit it. Downloading it from RM tools is not possible simply because downloading or any other modification is not available for Microsoft published tools.
In the deployment agent machine, deployment agent user’s temporary folder (component log contains this path with the error), you can find the file.
C:\RMT\RM\T\RM\XX-MTM-Test\201509241626076897123-31\TcmExec.ps1
In this example users temp path is set to  C:\RMT. Default path is C:\users\username\AppData\Local\Temp\.. as shown below. (More info here)
Copy this script and change the Int16 parameters highlighted below to Int.

Then create a new tool in Release Management using the powershell script.
Set arguments as follows.
-File ./TcmExec.ps1 -Title "__TestRunTitle__" -PlanId __PlanId__ -SuiteId __SuiteId__ -ConfigId __ConfigId__ -Collection "__TfsCollection__" -TeamProject "__TeamProject__" -TestEnvironment "__TestEnvironment__" -BuildDirectory "__BuildDirectory__"
Change the component in the template to use the new tool created, instead of provided “MTM Automated Tests Manager”.
*******************************************************************************************************
Error 02:
If MTM (Microsoft test Manager)  not available in the deployment server below error occurs.“ERROR: Unable to locate C:\RMT\RM\T\RM\XX-MTM-Test\IDE\TCM.exe”
To Resolve:
It is mandatory to have MTM(Microsoft test Manager) installed on the server that is initiating the tests. For this purpose separate machine is setup with deployment agent and it is installed with Visual Studio Test Professional 2013. It is not recommended to install Visual Studio in the target deployment server (DevInt/QA/UAT servers should be similar to Production, without having Visual Studio in them). Better option is to keep a separate machine attached to each environment to trigger tests after deployment.

This is used in the Release template to execute tests after deployment.
*******************************************************************************************************
Error 03:
With new tool test execution failed with below error.
“TF30063: You are not authorized to access http://yourtfs:8080/tfs/collection.”
To Resolve:
Add the deployment agent user configured in test initiating machine, to the Test Service Users in the relevant project collection.
*******************************************************************************************************
Error 04:
After fixing all above issues, tests initiated after the deployment with release template.
This time reading results fails with “A Visual Studio testing sku must be installed to use this command.”
To Resolve:
As discussed in this forum thread only working solution was to install “Visual Studio Ultimate (2013)” in the machine which is acting as deployment agent, triggering the tests using the component in the release template.
*******************************************************************************************************
With all above fixed test were executed successfully and results were shown in the tool log.