Override TestRunParameters in .NET Core

Using a .runsettings file with a <TestRunParameters> element is a convenient way to keep sensitive information required by your integration tests, such as for example username and passwords, out of source control. At least if you run your tests on Windows using Visual Studio Test or the VSTest@2 task in Azure Pipelines.

Let’s assume that you have checked in the following MSTest Test Project (.NET Core) into a source control repository:

The checked in .runsettings file contains a dummy value that should be replaced with a valid password at runtime for the test to pass:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <TestRunParameters>
    <Parameter name="password" value="*" />
  </TestRunParameters>
</RunSettings>

Below is an example of a test that will fail unless the value of the supplied password parameter is equal to “secret”. In a real-world scenario, you would probably use the parameter value(s) to connect to and authenticate against some remote service.

[TestClass]
public class SampleTests
{
    public TestContext TestContext { get; set; }

    [TestMethod]
    public void SampleTest()
    {
        const string ParameterName = "password";
        const string ExpectedValue = "secret";
        Assert.AreEqual(ExpectedValue, TestContext.Properties[ParameterName]);
    }
}

The real password is defined and set using a secret variable in the pipeline editor in the Azure DevOps portal, and supplied as an argument to the VSTest@2 task in the YAML file:

pool:
  vmImage: 'vs2017-win2016'

variables:
  buildConfiguration: 'release'

steps:
- task: DotNetCoreCLI@2
  displayName: 'Restore NuGet Packages and Build Solution'
  inputs:
   command: build
   projects: 'SampleTests/SampleTests.csproj'
   arguments: '-c $(buildConfiguration)'

- task: VSTest@2
  displayName: 'Run Sample Test'
  inputs:
    testAssemblyVer2: |
     Tests\bin\$(buildConfiguration)\**\SampleTests.dll
    runSettingsFile: Tests\Settings.runsettings
    overrideTestrunParameters: '-password $(password)'

So far so good. This approach works – even for .NET Core projects. If you however change the virtual machine image (vmImage) from the vs2017-win2016 one that runs Visual Studio 2017 on Windows Server 2016 to ubuntu-16.04, which is another Microsoft-hosted agent that is based on Linux, you can no longer use the VSTest@2 task as it’s only supported on Windows agents and cannot be used on other platforms.

The recommended way to execute unit tests in .NET Core is to use the dotnet test command. Unfortunately, it currently has no support for setting the TestRunParameters in the .runsettings file from the command-line. It only has some limited support for passing runsettings configurations.

You can work around this by creating another temporary .runsettings file during the build process and pass it to dotnet test with the --settings (or -s) parameter:

- task: DotNetCoreCLI@2
  displayName: 'Run Sample Tests'
  inputs:
   command: test
   projects: 'SampleTests/SampleTests.csproj'
   arguments: '-c $(buildConfiguration) --no-build --no-restore -s $(Build.SourcesDirectory)/SampleTests/UpdatedSettings.runsettings'

UpdatedSettings.runsettings is a temporary file that can be created using Powershell before the above DotNetCoreCLI@2 task is run. The file is “injected” with the value of the secret parameter on each build like this:

- powershell: |
   [xml]$doc = Get-Content SampleTests/Settings.runsettings
   $doc.RunSettings.TestRunParameters.ChildNodes.Item(0).value = '$(password)'
   $doc.Save("$(Build.SourcesDirectory)/SampleTests/UpdatedSettings.runsettings")
  displayName: 'Override TestRunParameters'

Unlike VSTest@2, the powershell task runs on both Linux and macOS as well as on Windows.

If you want to try this out for yourself, I have published the complete YAML file along with a minimal code sample on GitHub. If you set up a new project in the Azure DevOps portal, you should be able to push the sample solution to a Git repository of your own and set up a build pipeline based on the YAML file. Don’t forget to define the secret variable.