Pragmatic Works Blog

Choosing a Testing Framework in LegiTest

Written by John Welch | Jun 07, 2017

LegiTest has been designed to be a flexible tool for testing data and data-centric applications. When starting the development of LegiTest, we knew that some companies would have existing frameworks for unit testing in their applications. So, we chose to build LegiTest to work with, and extend a few of, the more popular frameworks. This means it can plug into existing build and integration processes very easily.

Based on feedback from our customers, NUnit (http://nunit.org/) and MSTest/VSTest (https://docs.microsoft.com/en-us/visualstudio/test/unit-test-your-code) are the most common frameworks they use. MSTest (which evolved to VSTest in more recent releases of Visual Studio) is used commonly by data developers who don’t work often with .NET code. It ships as part of Visual Studio, so it’s readily available to most developers. NUnit, on the other hand, is more commonly used by developers with a background in .NET development.

We are often asked: “Which framework should I choose/is best/do you recommend?” The short answer is: LegiTest offers all features regardless of framework, and works well with both, so use the one you are most comfortable with. However, if you haven’t already been using a testing framework, that’s a bit harder to understand. To make the decision easier, we’ll go through a few items that can help you decide between them.

First, consider what your organization is already using. If there are already groups using MSTest or NUnit, stick with the standard. Consistency and ease of integration with existing processes outweigh any technical differences between the frameworks. If there is no standard, keep reading.

If your organization has standardized on a test framework that is not MSTest or NUnit, we’d love to hear about it so we can consider adding it to LegiTest. You can submit an enhancement request at http://pragmaticworks.uservoice.com, or email support@pragmaticworks.com.

There are many web resources that go into extremely long (and sometimes bitter) comparisons between them (just search “MSTest vs NUnit” for a sampling). Most of the arguments presented are only relevant if you are going to be doing direct .NET coding of the unit tests. Since LegiTest generates the .NET code for you, you don’t really have to worry about which framework is easier to code against. So for the most part, we can ignore the technical differences. However, in a few cases, the code generated for NUnit is a bit more efficient than the equivalent code for MSTest. This isn’t orders of magnitude difference, but it does give a bit of an edge to NUnit.

Another small area of difference is in memory management of the tests. Particularly, we have encountered some issues in the field with memory management of tests running under MSTest that interact with SSIS packages. The memory isn’t completely cleared between runs of the tests, and this can result in errors interacting with the SSIS packages. Fortunately, there’s a simple workaround – in Visual Studio, under the menu Test. In the Test Settings menu, make sure that “Keep Test Execution Engine Running” is unchecked. This ensures that the test process memory is reset correctly after each test pass. This is only necessary with MSTest.

These are both minor items, but put together, it tilts the scales in favor of NUnit. If your organization has already standardized on MSTest, then we would recommend staying with the standard. If there is no standard, or it’s not a currently supported framework for LegiTest, then we recommend NUnit.