Updating Test Case Work Item Tags and Description based on Test Automation Code Test Category Attributes and Summary Descriptions

A test automation development team has come up with a requirement to keep track of some test automation code related attributes and documentation descriptions in the relevant Test Case work items in Azure DevOps. The documentation description added to each test method in  the code should be captured and updated to Test Case work item summary description. Any test category attribute values should be applied as Tags to the Test Case work item. Let’s look at how we can automate the above requirement using a little bit of PowerShell in combination with Azure DevOps build pipelines.

The Requirement

• Capture the summary description comment in the code for each test method in related Test Case work item in Azure DevOps.
• Add the TestCategory attribute values as tags to the Test Case work item in Azure DevOps.

The Solution

• To keep the link between the Test Case work item and the test method in code new TestProperty is introduced named TestcaseID. The developer of test code has to add a TestProperty attribute to each test method providing the TestcaseID as the key and the ID of the Test Case work item in Azure DevOps as the value.
• When an assembly is compiled it does not contain the documentation portion of the code such as summary description of a method. To generate the documentation of code alongside assembly as an XML documentation file, you need to enable Generate Documentation in project property pages. To do this right click on the Project in Visual Studio Solution Explorer and click on properties. Open the Build tab and for each build configuration check the XML documentation file option. With this setting enabled we can generate the XML documentation on the code which contains summary description etc. in the CI build as well.
• PowerShell script made available here which should be used in the Continuous Integration (CI) build of the test code to get the Test Case work item updated, with the summary description and add/manage tags based on the TestCategory attributes. Let’s analyze what the script does.
• First it will load the Test automation assembly with reflection using name pattern provided and look for each class and methods of each class to locate test methods. Test Methods are identified using the availability of the TestMethod attribute. We have to use reflection to do this as the XML documentation we generate does not include any information about the attributes used in methods.
• Then it would check if a TestcaseID property has been defined and if not defined will output a warning to the build log. If an ID is specified and if that Test Case work item is not available in Azure DevOps then there would be an error failing the build. This allows the developers of test code to add the TestcaseID when they are ready with the Test Case work item in Azure DevOps without having to modify the entire code base to include it IDs at once.
• The attributes in each test method evaluated to identify the TestCategory attribute values. The values are prefixed with a predefined tag prefix to prevent any overwriting of manually added tags in the test case work item. Script is only updating the tags if any attribute changes are available.
• Using the assembly name XML documentation file located in the script. Documentation of a given method is retrieved and Summary Description from XML is compared with the Test Case Summary Description if any changes happened in the description, it would be updated in the Test Case work item.
• You have to introduce below set of variables to the build pipeline to support the script.
• RestApiVersion – Azure DevOps REST API version to use
• TestAssemblyFilePattern – Name pattern for test assembly
• TestAssemblyPath – Path of the test assembly and dependencies in the build server
• TestCategoryTagPrefix – Prefix to use in tags for TestCategory attribute values
• Enable script to use OAuth token in build as it would be required to access the REST API of Azure DevOps in the script to retrieve and update Test Case work items.
• In the build step make sure to add  below msbuild arguments to enable generating XML documentation and to copy output of the build to the $(Build.ArtifactStagingDirectory) folder.

/p:OutDir="$(Build.ArtifactStagingDirectory)" /p:GenerateDocumentation=true 

• Add the PowerShell script from here to your source code repository. Then use it after the build step in the build pipeline as a PowerShell script task.
• With this setup when you check in/commit test automation code to the repo, the build will be executed and the relevant Test Case work items get updated automatically. A detailed log will be available in the build pipeline.
• In the Test Case work items you can see the changes getting applied via build.
• You can find sample code and the PowerShell script in GitHub here.
• The sample Azure DevOps build can be found in the public Azure DevOps team project.