MSTest is a command-line utility that executes unit tests created in Visual Studio with the Unit Test Framework. This article is about the why and how to run Visual Studio unit tests using the MSTest tool, without the need to have Visual Studio installed.
Note: In Visual Studio 2012, the underlying command line tool used to run the tests is actually VSTest.Console.exe, which is optimized for performance and is used instead of MSTest.
It is really important to distinguish between the Visual Studio Unit Testing Framework and MSTest. The former is the whole suite, integrated with Visual Studio, which allows you to create, manage, execute and measure unit tests sets and projects within Visual Studio. The latter, is one of the command-line utilities available to execute such unit tests.
Having MSTest running without Visual Studio is a bit tricky. This is a detailed post about all the steps and files that might be needed, and it might seem as incredibly cumbersome. However, is just about figuring our the correct files and settings needed on the machine. If you want to avoid having Visual Studio installed, and have a little patience, it will work.
A bit about Unit Testing Frameworks…
There are A LOT of unit test frameworks our there. Every technology platform has at least one that you can use, and many work with a wide variety of unit test frameworks. In .NET, there are a few popular unit test frameworks.
Most developers will choose between NUnit or Visual Studio Unit Test Framework (here is a little comparison). Many people choose NUnit in advance scenarios (like BDD approaches), arguing that Visual Studio native’s test framework is still not mature enough and lacks some of the more advance features. However, in most cases you will find everything you need using the native one. Additionally, using NUnit will require external tools and plugins like ReSharper to manage the test suit. Another Framework worth mentioning is xUnit, although I haven’t worked with it.
MSTest without Visual Studio. Why?
If you work with build servers and continuous integration and continuous deployment processes, you will need to be able to automate the process of building, testing and deployment your software applications (check this MSDN article for more information about testing .NET applications).
I have been publishing a series of articles on how to implement this kind of mechanisms and workflows with TeamCity CI Server for .NET solutions. In the past, I have done test automation using NUnit, but I wanted to try a setup with the .NET testing framework for Visual Studio 2012 and ASP.NET MVC 4.0.
In this case, you will need to be able to execute the unit tests on your CI server. In some cases you can install the development IDE on the server (this is commonly done in other platforms too, for example to build iOS applications). However, this is not desirable and is against best practices. What you really want to do here, is to be able to have your CI server executing the unit tests of your solution using the underlying command-line tool.
As I mentioned before, there are a few command-line tools to run unit tests, but in this post I want to focus on MSTest. Let’s jump into it.
Running MSTest without Visual Studio
Visual Studio and the .NET Framework provide all the binaries and configuration needed to run MSTest commands. Running the command-line tool without Visual Studio is a bit tricky. Microsoft doesn’t provide a stand alone version, so you need to copy the require files and settings to the target machine (disclaimer: licensing agreements might prevent you from doing this).
You will need to make sure of the following:
- The correct MSTest files are copied to the target machine.
- The MSTest required GAC binaries are copied and registered in the target machine.
- MSTest can run and execute commands without issues.
Copying MSTest Files
First you need to copy the MSTest files from your development machine (which should have Visual Studio and the .NET Framework installed), to the server machine. Make sure you are copying the right version files.
The following steps assume you have a 64-bit windows installation, hence the “Program Files (x86)” path. If not, just change for “Program Files”. You could create a new location on the target machine for the files you are about to copy, however I recommend to replicate the same folder structure of the development machine.
- Copy MSTest.exe and MSTest.exe.config from “C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE”
- Copy the “Microsoft.VisualStudio.QualityTools.*” files in this location
”C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\ReferenceAssemblies\v2.0”
”C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\ReferenceAssemblies\v4.0”
”C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\ReferenceAssemblies\v4.5”
- Create a folder for all the GAC assemblies you need to copy. You can just named “assemblies”.
- Extract all the v11 “Microsoft.VisualStudio.QualityTools.*.dll” (or corresponding version, although you might need previous version too) from the Windows registered assemblies, located in “C:\Windows\assembly”.
Note: you will need the “gacutil.exe” to extract assemblies from the GAC. The utility should be located on one of these locations:
C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin
C:\Program Files (x86)\Microsoft SDKs\Windows\v8.0A\bin\NETFX 4.0 Tools
You might need to copy the tool, so you can install the GAC assemblies in the server. To install assemblies you use the following command:
gacutil.exe /i “assembly_name”
- Place the extracted assemblies in the “assemblies” folder.
- You might need to register the assemblies on the server machine’s GAC. It might be good to keep a separate copy of these files so you can easily do so. Say, another folder named “GACAssemblies”.
- You might also need to extract the Microsoft.NET GAC_MSIL assemblies. Copy all 'v11' “Microsoft.VisualStudio.QualityTools.*.dll” and “Microsoft.VisualStudio.TestTools.*.dll” from “C:\Windows\Microsoft.NET\assembly\GAC_MSIL”
- Place these assemblies in the “assemblies” folder too.
- Copy all v11 “Microsoft.VisualStudio.QualityTools.*.dlls” and “Microsoft.VisualStudio.TestTools.*.dll” from “C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\PrivateAssemblies” to “assemblies”
- Place them in the “assemblies” folder.
- Copy all v11 “Microsoft.VisualStudio.QualityTools.*.dlls” and Microsoft.VisualStudio.TestTools.*.dll from C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\PublicAssemblies to “assemblies”
- Place them in the “assemblies” folder.
- Add “assemblies” to the “privatePath” attribute of in “MSTest.exe.config” (the copy you created at the beginning).
- Export “HKEY_LOCAL_MACHINE/SOFTWARE/Wow6432Node/Microsoft/VisualStudio/11.0/EnterpriseTools/QualityTools” and apply it to the server machine (look for how to import and export registry keys, is straight forward).
- Copy “QTAgent32.exe” and “QTAgent32.exe.config” (you might need other versions) from “C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE”.
- Add “assemblies” to the “privatePath” attribute of in “QTAgent32.exe.config”
- Copy “msdia110.dll” from “C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger” to “assemblies”
- Register “msdia110.dll” with “C:/windows/syswow64/regsvr32.exe /i '../mstest/assemblies/msdia110.dll”.
Here’s a screenshot of the GAC assemblies showing up in the server machine:
GAC assemblies that were actually required:
The copied files. Notice that I opted to keep “PrivateAssemblies” and “PublicAssemblies” folders in addition to the “assemblies” one. Also, I was required to copy “QTAgent32_40.exe” and “QTAgent32_40.exe.config”.
Public Assemblies copied:
Private Assemblies copied:
Assemblies List copied:
Ok, that seemed like a painful process. It seems like a lot, but basically is just copying all the files required to run MSTest on the server without having Visual Studio installed.
Once you copied all these files to the server and registered the ones needed in the GAC, plus the registry keys, you should be good to go. However, how can you be sure? and more than that, how can you troubleshoot if you are missing a file or having any other kind of error?
Add the following section to the MSTest.exe.config on the server. This code will set MSTest to generate logs when it runs. That way, after a run (failed or successful), you can check the log and see what is going on.
1: <system.diagnostics>2: <trace autoflush="true" indentsize="4">3: <listeners>4: <add name="EqtListener" type="System.Diagnostics.TextWriterTraceListener" initializeData="C:\mstest.log" />5: </listeners>6: </trace>7: <switches>8: <add name="EqtTraceLevel" value="Verbose" />9: </switches>10: </system.diagnostics>11:
Some Further Advise
Make sure that all the GAC files required were installed in the build server. I first tried to install only the ones that the process was requiring not to fail, but I realized that the process will silently fail if some others are not installed. So to be sure, install all the ones that you see in your local development machine (the one with Visual Studio). The DLL versions you need to install, depend on the Visual Studio version. If you are not sure, install all the versions. Make sure that the MSTest required files are there, including Public and Private assemblies we copied, the assemblies folder containing all assemblies for reference and the required QTAgent files, which might vary, again depending on the version. Copy all available in the development machine if not sure.
Here is the commands to run your tests manually from the command line (assuming you have two existing tests: IndexTest and AboutTest ):
mstest /testcontainer:"D:\TeamCity\buildAgent\work\8a4449e52545215c\web\MvcApplication2\MvcApplication2.Tests\bin\Release\MvcApplication2.Tests.dll" /test:IndexTest
"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\mstest" /testcontainer:"D:\TeamCity\buildAgent\work\8a4449e52545215c\web\MvcApplication2\MvcApplication2.Tests\bin\Release\MvcApplication2.Tests.dll" /test:AboutTest
You can specify which tests you want to run, and also where do you want to publish the results. By default, the reports will be created in the location from where you are running MSTest (if you run the command from the root of the drive, then the results will be created in C:\TestResults and there you will see the output and results of the MSTest process. Also, take into the account that you can modify the MSTest.exe.config file to create diagnostics reports with detailed information.
Normally, MSTest won't have access to create the result reports where MSTest.exe is located because of permission access issues. Make sure you run the commands from locations where the process will have access to create and modify files.
There are a few articles that were really helpful, although not exactly accurate. If you find yourself with issues after going through this article, maybe these references might give you the answer to your specific problem.
Running MSTest 2010 without Visual Studio
MSTest 2010 on the build server without Visual Studio
Copying Assemblies from the GAC
Getting MSTest to the Server and Enabling MSTest Logging Discussion
MSDN: Running Tests
MSTest.exe Command-Line Options
I hope you can find all the answers you need in this article or the referenced links. Please let me know if you have issues or different experiences or approaches setting up automated test mechanisms in your CI server.