Monday, May 6, 2013

Running MSTest without Visual Studio

MSTest is a command-line utility that executes unit tests created in Visual Studio with the Unit Test Framework. This article is about the why and how to run Visual Studio unit tests using the MSTest tool, without the need to have Visual Studio installed.

Note: In Visual Studio 2012, the underlying command line tool used to run the tests is actually VSTest.Console.exe, which is optimized for performance and is used instead of MSTest.

It is really important to distinguish between the Visual Studio Unit Testing Framework and MSTest. The former is the whole suite, integrated with Visual Studio, which allows you to create, manage, execute and measure unit tests sets and projects within Visual Studio. The latter, is one of the command-line utilities available to execute such unit tests.

Having MSTest running without Visual Studio is a bit tricky. This is a detailed post about all the steps and files that might be needed, and it might seem as incredibly cumbersome. However, is just about figuring our the correct files and settings needed on the machine. If you want to avoid having Visual Studio installed, and have a little patience, it will work.

A bit about Unit Testing Frameworks…

There are A LOT of unit test frameworks our there. Every technology platform has at least one that you can use, and many work with a wide variety of unit test frameworks. In .NET, there are a few popular unit test frameworks.

Most developers will choose between NUnit or Visual Studio Unit Test Framework (here is a little comparison). Many people choose NUnit in advance scenarios (like BDD approaches), arguing that Visual Studio native’s test framework is still not mature enough and lacks some of the more advance features. However, in most cases you will find everything you need using the native one. Additionally, using NUnit will require external tools and plugins like ReSharper to manage the test suit. Another Framework worth mentioning is xUnit, although I haven’t worked with it.

MSTest without Visual Studio. Why?

If you work with build servers and continuous integration and continuous deployment processes, you will need to be able to automate the process of building, testing and deployment your software applications (check this MSDN article for more information about testing .NET applications).

I have been publishing a series of articles on how to implement this kind of mechanisms and workflows with TeamCity CI Server for .NET solutions. In the past, I have done test automation using NUnit, but I wanted to try a setup with the .NET testing framework for Visual Studio 2012 and ASP.NET MVC 4.0.

In this case, you will need to be able to execute the unit tests on your CI server. In some cases you can install the development IDE on the server (this is commonly done in other platforms too, for example to build iOS applications). However, this is not desirable and is against best practices. What you really want to do here, is to be able to have your CI server executing the unit tests of your solution using the underlying command-line tool.

As I mentioned before, there are a few command-line tools to run unit tests, but in this post I want to focus on MSTest. Let’s jump into it.

Running MSTest without Visual Studio

Visual Studio and the .NET Framework provide all the binaries and configuration needed to run MSTest commands. Running the command-line tool without Visual Studio is a bit tricky. Microsoft doesn’t provide a stand alone version, so you need to copy the require files and settings to the target machine (disclaimer: licensing agreements might prevent you from doing this).

You will need to make sure of the following:

  • The correct MSTest files are copied to the target machine.
  • The MSTest required GAC binaries are copied and registered in the target machine.
  • MSTest can run and execute commands without issues.

Copying MSTest Files

First you need to copy the MSTest files from your development machine (which should have Visual Studio and the .NET Framework installed), to the server machine. Make sure you are copying the right version files.

The following steps assume you have a 64-bit windows installation, hence the “Program Files (x86)” path. If not, just change for “Program Files”. You could create a new location on the target machine for the files you are about to copy, however I recommend to replicate the same folder structure of the development machine.

  1. Copy MSTest.exe and MSTest.exe.config from “C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE”
  2. Copy the “Microsoft.VisualStudio.QualityTools.*” files in this location
    ”C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\ReferenceAssemblies\v2.0”
    ”C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\ReferenceAssemblies\v4.0”
    ”C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\ReferenceAssemblies\v4.5”

    image
  3. Create a folder for all the GAC assemblies you need to copy. You can just named “assemblies”.
  4. Extract all the v11 “Microsoft.VisualStudio.QualityTools.*.dll” (or corresponding version, although you might need previous version too) from the Windows registered assemblies, located in “C:\Windows\assembly”.
    Note: you will need the “gacutil.exe” to extract assemblies from the GAC. The utility should be located on one of these locations:
    C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin
    C:\Program Files (x86)\Microsoft SDKs\Windows\v8.0A\bin\NETFX 4.0 Tools
    You might need to copy the tool, so you can install the GAC assemblies in the server. To install assemblies you use the following command:
    gacutil.exe /i “assembly_name”
  5. Place the extracted assemblies in the “assemblies” folder.
  6. You might need to register the assemblies on the server machine’s GAC. It might be good to keep a separate copy of these files so you can easily do so. Say, another folder named “GACAssemblies”.
  7. You might also need to extract the Microsoft.NET GAC_MSIL assemblies. Copy all 'v11' “Microsoft.VisualStudio.QualityTools.*.dll” and “Microsoft.VisualStudio.TestTools.*.dll” from “C:\Windows\Microsoft.NET\assembly\GAC_MSIL”
  8. Place these assemblies in the “assemblies” folder too.

    image
  9. Copy all v11 “Microsoft.VisualStudio.QualityTools.*.dlls” and “Microsoft.VisualStudio.TestTools.*.dll” from “C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\PrivateAssemblies” to “assemblies”
  10. Place them in the “assemblies” folder.

    image
  11. Copy all v11 “Microsoft.VisualStudio.QualityTools.*.dlls” and Microsoft.VisualStudio.TestTools.*.dll from C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\PublicAssemblies to “assemblies”
  12. Place them in the “assemblies” folder.
  13. Add “assemblies” to the “privatePath” attribute of in “MSTest.exe.config” (the copy you created at the beginning).
  14. Export “HKEY_LOCAL_MACHINE/SOFTWARE/Wow6432Node/Microsoft/VisualStudio/11.0/EnterpriseTools/QualityTools” and apply it to the server machine (look for how to import and export registry keys, is straight forward).
  15. Copy “QTAgent32.exe” and “QTAgent32.exe.config” (you might need other versions) from “C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE”.
  16. Add “assemblies” to the “privatePath” attribute of in “QTAgent32.exe.config”
  17. Copy “msdia110.dll” from “C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger” to “assemblies”
  18. Register “msdia110.dll” with “C:/windows/syswow64/regsvr32.exe /i '../mstest/assemblies/msdia110.dll”.

Here’s a screenshot of the GAC assemblies showing up in the server machine:

image

GAC assemblies that were actually required:

TC_build_web_111

The copied files. Notice that I opted to keep “PrivateAssemblies” and “PublicAssemblies” folders in addition to the “assemblies” one. Also, I was required to copy “QTAgent32_40.exe” and “QTAgent32_40.exe.config”.

TC_build_web_112

Public Assemblies copied:

TC_build_web_118

Private Assemblies copied:

TC_build_web_116

TC_build_web_117

Assemblies List copied:

TC_build_web_113

TC_build_web_114

TC_build_web_115

Ok, that seemed like a painful process. It seems like a lot, but basically is just copying all the files required to run MSTest on the server without having Visual Studio installed.

Once you copied all these files to the server and registered the ones needed in the GAC, plus the registry keys, you should be good to go. However, how can you be sure? and more than that, how can you troubleshoot if you are missing a file or having any other kind of error?

Add the following section to the MSTest.exe.config on the server. This code will set MSTest to generate logs when it runs. That way, after a run (failed or successful), you can check the log and see what is going on.

  1: <system.diagnostics> 
  2:   <trace autoflush="true" indentsize="4"> 
  3:     <listeners> 
  4:       <add name="EqtListener" type="System.Diagnostics.TextWriterTraceListener" initializeData="C:\mstest.log" /> 
  5:     </listeners> 
  6:   </trace> 
  7:   <switches> 
  8:     <add name="EqtTraceLevel" value="Verbose" /> 
  9:   </switches> 
 10: </system.diagnostics>
 11: 

Some Further Advise

Make sure that all the GAC files required were installed in the build server. I first tried to install only the ones that the process was requiring not to fail, but I realized that the process will silently fail if some others are not installed. So to be sure, install all the ones that you see in your local development machine (the one with Visual Studio). The DLL versions you need to install, depend on the Visual Studio version. If you are not sure, install all the versions. Make sure that the MSTest required files are there, including Public and Private assemblies we copied, the assemblies folder containing all assemblies for reference and the required QTAgent files, which might vary, again depending on the version. Copy all available in the development machine if not sure.

Here is the commands to run your tests manually from the command line (assuming you have two existing tests: IndexTest and AboutTest ):

mstest /testcontainer:"D:\TeamCity\buildAgent\work\8a4449e52545215c\web\MvcApplication2\MvcApplication2.Tests\bin\Release\MvcApplication2.Tests.dll" /test:IndexTest

"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\mstest" /testcontainer:"D:\TeamCity\buildAgent\work\8a4449e52545215c\web\MvcApplication2\MvcApplication2.Tests\bin\Release\MvcApplication2.Tests.dll" /test:AboutTest

TC_build_web_122
You can specify which tests you want to run, and also where do you want to publish the results. By default, the reports will be created in the location from where you are running MSTest (if you run the command from the root of the drive, then the results will be created in C:\TestResults and there you will see the output and results of the MSTest process. Also, take into the account that you can modify the MSTest.exe.config file to create diagnostics reports with detailed information.

Normally, MSTest won't have access to create the result reports where MSTest.exe is located because of permission access issues. Make sure you run the commands from locations where the process will have access to create and modify files.

There are a few articles that were really helpful, although not exactly accurate. If you find yourself with issues after going through this article, maybe these references might give you the answer to your specific problem.

Running MSTest 2010 without Visual Studio
MSTest 2010 on the build server without Visual Studio
Copying Assemblies from the GAC
Getting MSTest to the Server and Enabling MSTest Logging Discussion
MSDN: Running Tests
MSTest.exe Command-Line Options

I hope you can find all the answers you need in this article or the referenced links. Please let me know if you have issues or different experiences or approaches setting up automated test mechanisms in your CI server.

6 comments:

  1. Anthony,

    Great post--this is exactly what I needed. I am looking to run coded UI tests on a computer without VS. However, I'm confronted with this problem...

    I copy all the DLLs and XMLs and use gacutil on the test machine (the one without VS2010), and I run my stand-alone app which calls the test dll to run with MSTest--but it won't run. The error in command prompt says that it couldn't find the following:

    Microsoft.VisualStudio.TestTools.RunnerCommandline.Runner.Execute(String[] args)
    Microsoft.VisualStudio.TestTools.RunnerCommandline.Runner.Main(String[] args)

    I saw nothing similar to these under VisualStudio.TestTools, so do you know why I would be getting this error?

    P.S.
    I apologize if I didn't give enough details. I'm just starting out as an intern, so I don't really understand everything that I'm doing with this MSTest business. I'm just following some step-by-steps. If you need any more specifics let me know and I'll try to get them.

    Thanks!
    -Joel

    ReplyDelete
    Replies
    1. Hi Joel,

      Thanks for stopping by. It is a bit of trial and error to get this working.
      I suggest you look for the missing DLLs in your local machine (the one with VS). You need to make sure that the MSTest.exe dependencies are present in the server machine (the one without VS). Don't forget to check the GAC DLLs. It might be that you copied the files, but they are not registered in the GAC. You can also extend the MSTest.exe.config to make it point to specific DLL's or to give you more details about errors.
      Lastly, make sure you are copying the right files. If you have multiple versions of Visual Studio it might be that you copied references from a different version.

      Hope it helps. Let me know.
      -aB

      Delete
  2. Hey Joel, did this work for you? Am looking to run coded ui tests on a machine that does not have VSTS. I installed Test Agent on this box and it copied the required dlls for MStests.exe.
    When I run my tests however I get the below error

    Could not load file or assembly 'Microsoft.TeamFoundation.Client, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

    ReplyDelete
  3. Thansk Joel, quick question: the same steps would work for vstest.config.exe too?

    ReplyDelete
  4. You can just install the test tools separately. http://www.microsoft.com/en-us/download/details.aspx?id=40750

    ReplyDelete
    Replies
    1. Thanks, You saved me a lot of pain

      Delete