CI: Publishing Test results and Coverage from TFS to SonarQube

Wanted to setup a TFS Nightly build, to execute all tests and publish results, with Coverage information included, to SonarQube.

TFS: Team Foundation Service on premises
VSO: VisualStudio Online

My context is an enterprise environment, TFS2015. SonarQube is also in intranet.

I need a Build with following steps:

- Step 1: SonarQube - Begin Analysis
- Step 2: Build Solution
- Step 3: Tests Assemblies
- Step 4: Generate Coverage report
- Step 5: SonarQube - End Analysis

I'm leaving outside any other step like Nuget restore not directly involved in tests and coverage.

This is a very simple and straightforward Build you can setup with the predefined tasks available on TFS.

But sometimes things don't work as expected, due to several reasons, in my case, some missing patch on my TFS agent or bad configuration made me impossible to generates coverage and had not enough rights to check inside server so I needed to find an alternative way.

Lets see then how to create a Build, with tests and coverage information to send to SonarQube, in an alternative way.

Step 1 SonarQube - Begin Analyse:

This is the old task, now deprecated:

This is the new version:

We need to initialize SonarQube and tell him from where we want to collect the tests results and coverage.
The arguments passed to Sonar-Begin depends on how we will generate reports.
So, I'm going to next steps to generate the necessary information and later I'll come back to Sonar-Begin step.

Step 2: Build the Solution

I'll use the VStudio task:

This is a very simple task to build the solution with VStudio internal compiler.

The alternative way is to use a MSBuild task.

Step 3: Test assemblies

Two options here,

Option 1, the Visual Studio Tests task:

On the task configuration section Execution Options we provide path and pattern to discover all tests assemblies on our folders.
This step has a very nice option for Code Coverage, in red in image below.

With just checking this option you should have coverage information generated in VStudio format (.coverage) and should be enough... but here is where my Agent stop working and build get blocked for 1hour.

The above task use the VisualStudio test runner. Launching a build and checking the logs, we have this:

 Preparing task execution handler.  
 Executing the powershell script: D:\AgentTFS\_work\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\1.0.84\VSTest.ps1  
 Working folder: D:\AgentTFS\_work\1\s  
 Executing C:\(...)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "D:\AgentTFS\_work\1\s\relboard.Tests\bin\Release\relboard.Tests.dll" /EnableCodeCoverage /logger:trx /TestAdapterPath:  

The runner is vstest.console.exe, it execute MSTests but in my case I have NUnit tests so I want to use NUnit runner.

In order to debug my tests from VisualStudio I was on the need to install NUnit3 Test Adapter from nuget already, and good news, vstest.console.exe is smart enough to discover any test adapter we have installed on our packages folder and pick it up.
So without any extra config, NUnit test runner should be used to execute our tests.

In case of doubt, there is an option to explicitly specify the runner we want.
In Advanced Execution Options / Path to Custom Test Adapter enter the path of NUnit test adapter that should be: packages/NUnit3TestAdapter/lib

If we stay with vstest.console.exe, the tests result will be generated in MS format .trx
Here are the logs of the build:

 Information: NUnit Adapter 3.7.0.0: Test execution complete  
    
 Passed  Should_Pass_This_First_Test  
 Passed  Should_Pass_This_Second_Test  
 Results File: D:\AgentTFS\_work\1\s\TestResults\buildguest_TASKAGENT5-0010 2017-02-27 22_06_31.trx  

Second option, the old school, manual execution:

Switching to manual

Take a Command Line task to execute NUnit tests with nunit-console.exe directly:

NUnit console should be available somewhere on TFS Agent server, usually needs to be installed in some common folder accessible to any build, but I can't do it, remember I have no access to agent.

I'm adding NUnit console to my solution sources. It is not a good idea to add assemblies to source repository but in this case they are more as externals tools (like SVN way) so I have a folder BuildTools with NUnit console there. There is no need to install NUnit, just download it and put it on a folder.
When sources be downloaded to TFS working directoy, NUnit console will be available at following path:

$(Build.SourcesDirectory)\BuildTools\NUnit.Console-3.6.0\nunit-console.exe (or nunit-console-x86.exe)

The Command Line task is configured as follow:

After task be executed, test results will be available on file NUnitResults.xml

Step 4: Coverage

Now for the coverage, the same, another Command Line task.

You can use several tools to generate the coverage, I'm using OpenCover.

As we did with NUnit, I'm adding OpenCover to my folder BuildTools, remember I can't put it somewhere on agent server, which should be the right way to do.

Configure the Command Line task as follow:

Tool:             BuildTools\OpenCover\OpenCover.Console.exe
Arguments:  -output:opencover.xml" -register:user -target:"BuildTools\NUnit.Console_3.6.0\nunit-     console.exe" -targetargs:"UnitTestProject1\bin\Debug\UnitTestProject1.dll"

we are executing OpenCover which will use nunit-console.exe to execute the tests. coverage report will be opencover.xml

But we can also use vstest.console.exe for the same:

Arguments: -output:"opencover.xml" -register:user -target:"%VSINSTALLDIR%\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" -targetargs:"UnitTestProject1\bin\Debug\UnitTestProject1.dll"

and because I want to use NUnit3 adapter, here is the arguments to use OpenCover with vstest.console.exe and with NUnit3 adapter behind:

Arguments: -output:"opencover.xml" -register:user -target:"%VSINSTALLDIR%\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" -targetargs:"UnitTestProject1\bin\Debug\UnitTestProject1.dll /TestAdapterPath:packages/NUnit3TestAdapter/lib"

Mergin Tests and Coverage in One Step

Step 3 is executing the tests and Step 4 it is also doing the same but only to collect coverage data.

We can do both things in one step.

Execute OpenCover with target vstest.console.exe or nunit-console.exe and pass parameter to the target console to generate the test reports at the same time.

In a Command Line task, configure it as:

Tool: BuildTools\OpenCover\OpenCover.Console.exe

You can prefix the Tool path with $(Build.SourcesDirectory) just to be sure it will look inside your source directory.

OpenCover with vstest.console.exe:

Arguments:

-output:"opencover.xml" -register:user -target:"%VSINSTALLDIR%\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" -targetargs:"/logger:trx UnitTestProject1\bin\Debug\UnitTestProject1.dll"

output:

coverage report = opencover.xml

test results = TestResults\buildguest_TASKAGENT5-0010 2017-02-27 22_06_31.trx

OpenCover with nunit-console.exe:

Arguments:

-output:"opencover.xml" -register:user -target:"BuildTools\NUnit.Console_3.6.0\nunit-console.exe" -targetargs:"/results:NUnitResults.xml UnitTestProject1\bin\Debug\UnitTestProject1.dll"

output:

coverage report = opencover.xml

test results = NUnitResults.xml

OpenCover with vstest.console.exe and nunit3 adapter:

Arguments:

-output:"opencover.xml" -register:user -target:"%VSINSTALLDIR%\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" -targetargs:"/logger:trx UnitTestProject1\bin\Debug\UnitTestProject1.dll /TestAdapterPath:packages/NUnit3TestAdapter/lib"

output:

coverage report = opencover.xml

test results = TestResults\buildguest_TASKAGENT5-0010 2017-02-27 22_06_31.trx

Send data to SonarQube

Now go back to Step 1, Sonar - Begin.

Select task:

Go to Advanced / Additional Settings

If you decided to generate tests results with vstest.console in .trx format, then Sonar-Begin task should be configured as follow:

/d:sonar.cs.opencover.reportsPaths="opencover.xml" /d:sonar.cs.vstest.reportsPaths=TestResults\*.trx

If tests results file was generated directly with nunit-console.exe, (not with vstest.console.exe with nunit adapter !) you have then a .xml file and Sonar-Begin should receive following arguments:

/d:sonar.cs.opencover.reportsPaths="opencover.xml" /d:sonar.cs.nunit.reportsPaths=results.xml

all arguments above are one line !

It may be possible that your reports files can't be found by TFS because it is looking in another folder than where sources are.
You can try prefixing all reports paths with $(Build.SourcesDirectory)\

For the .trx report from vstest.console, try with $(Common.TestResultsDirectory)\



References:
https://docs.sonarqube.org/pages/viewpage.action?pageId=6389770
https://docs.sonarqube.org/pages/viewpage.action?pageId=6389772