Is there a performance difference between the Visual Studio test runner and TD.Net?

I've semi-recently moved a suite of unit tests from MbUnit to MSTest (ok, I won't groan about why right now), and started using the built-in test runner in Visual Studio. I've noticed that the built-in runner feels slower than TestDriven.NET. Not knowing if this was real or just a feeling I decided to investigate a little.

I did a quick timing of the two on the solution I'm currently working on. I'm using VS 2010 and TD.Net 3.0. I'm timing them manually by starting and stopping a stopwatch so this isn't completely accurate timings, but they give a good enough indication for me.

Full test suite

I did a fresh Get Latest and built the solution first. Then I ran all tests in the solution using TD.Net first and then the built-in VS runner afterwards. Here are the timings:

First run:

• TD.Net: 101 seconds
• VS: 143 seconds => 41% slower

It was noticable how VS spent a long time initalizing. It would first do nothing for a while, then show the test results with a "Test run pending" message, still doing nothing, before it would start running the tests. I'm not sure what it's doing during this time. TD.Net also has a bit of initialization time, but it's not as noticable.

Without changing anything I ran the same tests again. This time the results were more in line with each other:

Second run:

• TD.Net: 78 seconds
• VS: 85 seconds => 9% slower

Both were faster the second time. VS started running tests without too much delay. VS is still slower, but not by as much (9% instead of 41%).

I made a code change in one of the core-level projects on which most other projects depend, causing almost all projects to rebuild. After building successfully I ran both testers again, giving these results:

After rebuild:

• TD.Net: 86 seconds
• VS: 103 seconds => 20% slower

Again, the Visual Studio runner spent a gread deal of time in a "Test run pending" state (about 50 seconds) before starting the testing.

Single test class

It might not be a big deal that VS is a bit slower running the full test suite if you're doing TDD and most frequently run only the few tests covering the module you're working on. I therefore timed the performance of running a single test class only. As before I first ran TD.Net and then VS and timed them:

First run:

• TD.Net: 2-3 seconds
• VS: 8 seconds => +5 seconds, 166% slower

This time the timings were pretty consistent accross repeated runs, and it didn't matter if I ran all 30 tests in the class or just one of them. So it seems almost all of this time is spent in initialization.

If you're actually doing some work you'll have made some code changes before running the tests again, so build time also contributes to the cycle time between tests. I know it's not possible to measure the build time since this varies wildly accross projects, but I wanted to see how this affected the times for my current project. I made a whitespace code change to the SUT and ran the same test again without building first:

After code change, including build time:

• TD.Net: 6 seconds
• VS: 11 seconds => +5 seconds, 83% slower

VS is still the same 5 seconds slower as above, but now this is down to around 83% slower instead of the 166% above.

Conclusion

It's apparent that the VS runner is indeed slower that TD.Net. VS seems to use a lot more time in initialization. It seems to be about 20-40% slower for a full test run, and 80% slower for TDD.

Build times can affect your cycle time a lot more than the test runner, but with well factored modules the build time can be pretty low as you only need to rebuild the project under test and not the whole solution.

These tests were run on a codebase with relatively few tests compred to the amount of code (72 projects and 718 tests). I might have gotten different results if the number of tests matched the code size since initialization time might be more offset by the testing time.

It should be interesting to measure the timing difference between MSTest and another testing framework like N/Mb/XUnit as well.