Daily Archives: February 16, 2006

Shipping Unit Tests with Application Code

I attended the Canberra .NET User Group meeting this evening (Febuary 16th). This month it was Adam Cogan who was presenting “15 Rules to Successful Projects” (related link).

Adam and I (and probably a few other people) have been having an ongoing discussion about one of the things he does in the various tools and systems that he works on. That is the embedding of unit tests in the application code so that a user can run them – often just by clicking a button!

My first reaction to this was mixed. I kinda like the idea but I saw a whole heap of unresolved issues – especially around licensing. Although upon reading the license agreenment for NUnit, I need not have worried.

Tonight I think Adam finally started convincing me that this was a good idea but I still think his implementation of a good idea sucks. Firstly – NUnit is a developer focused tool, so when you run it it basically traverses the code base looking for unit tests and then lays them out in a namespace/class hierarchy as it executes each and every test.

Thats great – but what have you told the user other than something is “all green” not “not all green”?

Adam was also lamenting the fact that Visual Studio Team System’s unit testing framework isn’t really something that you can bundle up and ship with your application like he does with NUnit. The reality is that most people aren’t going to ship unit tests with their product and the level of integration between the testing frameworks and MSBuild and Visual Studio 2005 would make that dependency pretty large.

It occurs to me that what Adam really needs is a system test framework, not a unit test framework. If you look at Adam’s Rules for Better Unit Tests you can see that a lot of them are for checking dependencies like databases, DLLs and web-services. Personally I like my unit tests to be as free standing as possible and take as few external dependencies as possible.

A system test framework on the other hand would be designed to run tests after an application had been initially installed where the test code can rightly expect a fully functioning environment to be present.

The epiphany about this came to me when I saw another recommendation of Adam’s which was to limit the intelligence that you put into setup executables. I like this idea (although I don’t always follow it) and its similar to what BizTalk does with its relatively simple (heh) routine with the post installation ConfigureFramework.exe step.

System tests and post installation steps are actually related – when a system test fails it is usually because some installation step has been missed. So by doing a bunch of system tests when the application starts and then on demand (and whenever the application is crashing) you can associate the specific test failures with remediation steps (general term).

SystemTestRemediationStep

The diagram above summarises how a system test would be related to a remediation step. The benefit of doing it this way rather than just using NUnit is that you not only get the system to tell you how its feeling but you can actually get the system to suggest how to fix itself if possible – from a user experience point of view this is slick.

I’d love to take a stab at implementing something like this when I get some bandwidth:

TaskQueue.Enqueue(Idea);

Yesterdays Weather for Team Foundation Server – Part 2

In the first post in this series I outlined the rough steps that would need to be taken to implement yesterdays weather in Team Foundation Server. Before I get started on making modifications to the the work item definitions I want to cover off why I didn’t take some seemingly more simple options to implement yesterdays weather (because I know they will come up).

  • Why don’t you just use the existing effort recording fields on work items like “Task”? I really wanted this sample to be free standing and not make any assumptions about what particular process template may or may not be being used – I really want to have a consistent reporting story off any work item type.
  • Why don’t you just use Reporting Services to list Yesterdays Weather? Thats a good question, errr, look over there! In all seriousness though I could have done that. The benefit of this approach is that if you want to integrate it into other tools working directly against the work item store is going to give you a better data structure to work against.

Of course the other big reason that I wanted to do it was to show how to program against the TFS client API in order to get work item information out of the store – but you will have to wait for the next post until I cover that. For now – lets get started looking at the basic modifications required to support Yesterdays Weather.

Creating a Dummy Team Project

The first thing that needs to be done is to create a dummy team project to make our modifications against. You could export the process template but as you will see – doing it this way gives you access not only to the process template, but also a way to try out changes quickly.

YWCreateTeamProject

Modifying the Process Template

Once the process template is created the next step is to go any modify the work item definition for one of the types in the team project. You could do this using good old notepad and the work item type exporter (witexport.exe) but if you want a better editing experience I would recommend downloading the Process Template Editor from the VSTS Customization Workspace.

Because Team Foundation Server is so extensible the Process Template Editor has a fairly expansive UI, so instead of documenting every single step required here I’ve recorded a short screen cast which walks you through it (sorry about the rough audio, the aircon is pretty loud). One thing that I didn’t mention in the screen cast is that you actually have to click the save button to persist the changes back into TFS – so don’t forget to do that!

Observing the Results

Once the work item has been modified and saved back into TFS the changes should become visible the next time you open Team Explorer. It is possible that the definition is being cached and if thats the case just right mouse click on the team project in the Team Explorer and select “Refresh”.

Next, run the “All Tasks” query and select one of the returned results. Notice that there is now a Yesterdays Weather tab on the screen and that it has the two fields that we defined in the Process Template Editor.

YWYesterdaysWeatherNo

We can see the conditional rules we put in simply by changing the value of the “Publish Effort” field to true – as you can see the work item view in Visual Studio gives us some visual queues that we need to fill in the total effort field.

YWYesterdaysWeatherYes

If you wanted to, you could then create queries that use this information. Also – using the Process Templat Editor you can choose how these fields are represented in the TFS data warehouse for reporting purposes – if you want to go down that road.

Extracting the Changes

The whole point of this excercise was to get the modifications required to the work item type definition so that we can use them as part of an installer. If I was going to productise this enhancement to TFS I would almost certainly want a reliable installer that could make the changes to the type definitions and even remove them afterwards.

We can’t just simply load in the entire definition because that might break other customisations that people have already made. I used the Process Template Editor to export the work item definition but I could have used witexport.exe as well (you can download the definition here). The first elements that we are looking for are under the fields definition:

WITFields

The other elements we are interested in are the ones that define the layout:

YWWITLayout

Next Steps

Well – hopefully I’ve managed to impart some free standing knowledge there. In my next post I will look at taking the above individual XML fragments and write custom installer routines to install them into existing work item types. Stay tuned!