TestWide

Searching and Filtering Tests in Test Explorer

If you take a quick glance at the Test Explorer window in Visual Studio 2015, you might not notice all of the power that you have in that little window. To start, there is a grouping icon, that allows you to group your tests by various properties. The grouping is a great way to get common tests together so you can easily select them and run them in test explorer.

TestExplorerGroupBy

You have the option to group by:

Group By Description
Class This will group by the class name to which the test belongs. Note: This is not the fully qualified class name, so if you have multiple classes in different namespaces with the same name, they will be hard to differentiate.
Duration This will group by the duration of the last test run. It uses 3 categories when grouping (Fast < 100ms, Medium > 100ms, and Slow > 1 sec).
Outcome This will group by the outcome of the last run of the tests. It uses 3 categories when grouping (Passed Tests, Failed Tests, and Skipped Tests).
Traits This will group tests based on the TestCategory, Owner, Priority, and TestProperty attributes assigned to tests. Note: Tests can have multiple trait attributes assigned to them and thus a single test could appear multiple times in this view.
Project This will group tests by the project to which they belong.

While the grouping is nice, the real power in this dialog is the search feature. From the documentation, you can search on the following:

Qualifier Description
Trait Searches both trait category and value for matches. The syntax to specify trait categories and values are defined by the unit test framework.
Project Searches the test project names for matches.
Error Message Searches the user-defined error messages returned by failed asserts for matches.
File Path Searches the fully qualified file name of test source files for matches.
Fully Qualified Name Searches the fully qualified file name of test namespaces, classes, and methods for matches.
Output Searches the user-defined error messages that are written to standard output (stdout) or standard error (stderr). The syntax to specify output messages are defined by the unit test framework.
Outcome Searches the Test Explorer category names for matches: Failed Tests, Skipped Tests, Passed Tests.

Let’s take an example. Say I have the following tests in my system (implementations removed for brevity):

[TestMethod]
public async Task TestMethodNone()

[TestMethod, TestCategory("Unit")]
public async Task TestMethodUnit()

[TestMethod, TestCategory("DAL"), TestCategory("Unit")]
public async Task TestMethodDALUnit()

[TestMethod, TestCategory("DAL"), TestCategory("Unit")]
public async Task TestMethodDALUnit2()

[TestMethod, TestCategory("DAL"), TestCategory("Integration")]
public async Task TestMethodADALIntegration()

If I group by trait and don’t filter anything, then I’ll see the following tests:

TestExplorerGroupedByTrait

Next, I could filter the tests by specifying I only want tests with the Unit trait. The search term would be Trait:"Unit":

TestExplorerUnitTestsOnly

I can also filter to only show tests that are both DAL and Unit tests by using the search term Trait:"Unit" Trait:"DAL":

TestExplorerUnitAndDAL

If I want to exclude tests with a given attribute, I could exclude all DAL tests by using the minus symbol, so my search term would be Trait:"Unit" -Trait:"DAL":

TestExplorerUnitNotDAL

You can also pair this with other searchable attributes on the tests. So, after a test run, if I want to find all unit tests that failed, I could use the search term Trait:"Unit" Outcome:"Failed":

TestExplorerFailedUnitTests

As you can see, the grouping and filtering available to you in the Test Explorer window is pretty robust; it just takes a little time to dig into it and learn the syntax. The Run unit tests with Test Explorer article on MSDN gives a lot of good information on this topic and is a worthwhile read if you are using this window in your day to day work. Thanks to William Custode for asking a question on StackOverflow that gave me inspiration for this blog post.

Image Credit: Alberto G. via Creative Commons

AzureBannerImage

Editing Files on your Azure Hosted WordPress Site

I host this blog on Windows Azure using WordPress. There have been times where I needed to edit my web.config file or manually remove some plugins that were causing trouble. After searching around quite a bit, I found that the easiest way to do this is to use Visual Studio Online to edit the files in my hosted site.

To start, you need to add the Visual Studio Online extension to your Web App. To do this, select the tools option from your Web App and select the Extensions menu in the Develop section.

AzureTools

Next, select the Add button to add a new extension.

AddExtension

Select Choose Extension and find the Visual Studio Online extension.

VisualStudioExtension

Once installed, you can then select the Visual Studio Online extension from the Extensions view and select the Browse button. This will launch a new window from which you can explore the contents.

Browse

For, example, you can select and open your web.config file and quickly make changes to it.

VSOEditWebConfig

I have found this tool useful a few times, allowing me to enable woff files and to enable SSL on all pages. Hopefully you find it as useful as I do.

Windows

What does Environment.OSVersion return on Windows 10?

Unfortunately, the answer is, it depends.

Consider the following code:

Console.WriteLine(Environment.OSVersion.ToString());

Depending how you execute the code, you may get differing results:

Visual Studio Version Project Type Output
2015 Update 1 Console App Microsoft Windows NT 6.2.9200.0
2015 Update 1 Unit Test Microsoft Windows NT 10.0.10586.0
2015 (No update 1) Console App Microsoft Windows NT 6.2.9200.0
2015 (No update 1) Unit Test Microsoft Windows NT 6.2.9200.0

So, how can we get our console application to be consistent with the test application in VS 2015 update 1? The key is manifest files. According to the Operating System Version documentation:

For applications that have been manifested for Windows 8.1 or Windows 10. Applications not manifested for Windows 8.1 or Windows 10 will return the Windows 8 OS version value (6.2). To manifest your applications for Windows 8.1 or Windows 10, refer to Targeting your application for Windows.

So we can add a manifest file to our console application to specify Windows 10 compatibility:

<?xml version="1.0" encoding="utf-8"?>
<assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1">
  <assemblyIdentity version="1.0.0.0" name="MyApplication.app"/>
  <compatibility xmlns="urn:schemas-microsoft-com:compatibility.v1">
    <application>
      <!-- Windows 10 -->
      <supportedOS Id="{8e0f7a12-bfb3-4fe8-b9a5-48fd50a15a9a}" />
    </application>
  </compatibility>
</assembly>

Now, when we run the console application, we get the output of Microsoft Windows NT 10.0.10586.0

So, what changed in Visual Studio 2015 Update 1? This is speculation, but I am guessing they added a manifest to the executable the runs the unit tests behind the scenes, which caused the Environment.OSVersion to return the new value for Windows 10. As the documentation states:

Identifying the current operating system is usually not the best way to determine whether a particular operating system feature is present. This is because the operating system may have had new features added in a redistributable DLL. Rather than using the Version API Helper functions to determine the operating system platform or version number, test for the presence of the feature itself.

So, if you plan on using Enivronment.OSVersion in the future, be sure to understand the different values it could return depending on how the code is hosted.

Image Credit Les Haines via Creative Commons

Unable to Resolve DNX Dependencies on OSX

I was building dot net core console applications on OSX and ran into an issue where certain core dependencies were not resolving when running dnu build. The error output I would get would be:

Building ConsoleApplication2 for DNX,Version=v4.5.1
  Using Project dependency ConsoleApplication2 1.0.0
    Source: /Users/john/Dev/ConsoleApplication2/project.json

  Unable to resolve dependency fx/mscorlib

  Unable to resolve dependency fx/System

  Unable to resolve dependency fx/System.Core

  Unable to resolve dependency fx/Microsoft.CSharp
  ...

After doing some testing, I found that if I specified the core framework (dnxcore50) to the build command, then it would work properly.

dnu build --framework dnxcore50

After doing some more digging, I found that I did not have the mono framework installed on the machine, and thus the v4.5.1 build was failing. To correct this, I simply had to install the mono framework using dnvm:

dnvm upgrade -r mono

After that, the dnu build works properly and I can continue on my way.

TestWide

Parallel Test Execution in Visual Studio 2015 Update 1 Might Not Be What You Expect

In the Update 1 of Visual Studio 2015, it was announced that MSTest will support running unit tests in parallel. I decided to give this a shot and see exactly how it worked. I started by writing 8 unit tests that all looked like this:

[TestMethod]
public async Task TestMethod7()
{
    Console.WriteLine("1");
    await Task.Delay(5000);
}

Next, I added the RunSettings file to my project with a MaxCpuCount of 6:

<?xml version='1.0' encoding='utf-8'?>
<RunSettings>
  <RunConfiguration>
    <MaxCpuCount>6</MaxCpuCount>
  </RunConfiguration>
</RunSettings>

Finally, I selected my run settings file from the Test Settings Menu:

SelectTestSettings

I ran it and all of my tests still ran in serial. I thought maybe I had done something wrong or perhaps hit a bug with the new feature, so I returned to the update 1 announcement and found my answer. It states:

Parallel test execution leverages the available cores on the machine, and is realized by launching the test execution engine on each available core as a distinct process, and handing it a container (assembly, DLL, or relevant artifact containing the tests to execute), worth of tests to execute.

The separate container bit is the piece I was missing. In order to get my tests to run in parallel, I needed to split up my tests into separate test assemblies. After doing that, I saw that the tests in different assemblies were running in parallel.

The distinction of tests running in parallel across assemblies is a subtle point that may cause confusion if you think just setting the MaxCpuCount in a run settings file is going to give you benefit on a single test assembly. Overall, I am glad to see that Microsoft is still improving MSTest and hopefully they continue to add features that will allow us to better parallelize our testing.

Image Credit: Alberto G. via Creative Commons

1 2 3 13