Getting Started with TFS Integrator

ATTENTION: TFS Integrator is now obsolete. Please use TFS Continuous Integrator and TFS Dependency Replicator. For more information, read the post on why TFS Integrator is now obsolete.


This is a post that I’ve been meaning to write for a while. Earlier this year, when Chris Burrows joined our team we had the opportunity to spend a little bit of time building a TFS extension that Darren and I dreamed up whilst we were working on a client project together.

As you are probably no doubt aware, Team Foundation Server does not ship out of the box with a Continuous Integration capability. A lot of people, including myself consider this a glaring omission – but as a developer I have to appreciate the demands of tight delivery schedules.

What is Continuous Integration?

Continuous Integration (CI) is the process of continually scanning the source code repository for changes, and, once they have been detected automatically kicking off a build process to verify that the code compiles successfully. In many instances teams will tag on a round of build verification tests to ensure that not only does the code compile, but that it doesn’t catch on fire when you try to run it (smoke tests).

While not unique to agile methods, continuous integration is certainly one of the tools that agile development teams to keep a handle on their code base as it evolves throughout an iteration. It is no surprise then that when agile teams came to use TFS that there was a collective “what the” when it didn’t include a CI engine.

How did we build our TFS CI engine?

Fortunately for us, the team at Microsoft did build TFS with an eye on extensibility and they included in the system an “event service” which we can plug into to listen for events in the source code repository. The way it works is that whenever you do something significant in Team Foundation Server, such as checking in a file, that TFS sub-system notifies the TFS eventing system which then broadcasts that message to any subscribers. This is how you get an e-mail when another developer checks in some code.

This same eventing mechanism can actually be instructed to send a notification via the SOAP format to a web-service endpoint which might be hosted in something like ASP.NET, or event Windows Communication Foundation.

TFS Integrator uses this facility to kick off a build in the Team Build component of Team Foundation Server. When TFS Integrator initialises it reads a configuration file to identify which parts of the source tree it is interested in listening to then subscribes to the check-in event notifications that the eventing system sends out.

When it receives a notification it sleeps for a specified period of time then kicks off a build with the Team Build facility. Once the build is completed the eventing system sends how a notification to subscribers as to the outcome. The completed interaction between the system components looks like this:

 

All of this automation makes use of the tools available from the Visual Studio 2005 SDK which you can download from Microsoft.

How to configure TFS Integrator to support Continuous Integration?

Before we start you need to understand that the current build of TFS Integrator relies on the .NET 3.0 runtime (RC1) being installed. For this reason we don’t currently recommend installing it on the same computer as either your application or data tiers. Once you have downloaded and installed the .NET 3.0 runtime (RC1), the next thing that you are going to need to do is download the latest build of TFS Integrator.

The setup package itself does nothing more than drop a few files onto the file system, the rest of the configuration requires some manual intervention. The installation files are dropped into the “TFS Integrator” directory under program files.

The “TFS Integrator” directory contains three seperate configuration files:

  • TfsIntegrator.exe.config
  • ContinuousIntegration.xml
  • DependencyReplication.xml

The first file, TfsIntegrator.exe.config contains the bootstrap information required to get TFS Integrator talking to your Team Foundation Server installation. In this configuration file there are three main settings that you should be interested in. These are BaseAddress, RegistrationUserName and TeamFoundationServerUrl.

BaseAddress is the address of the server itself. We would have liked to have allowed the program to determine this itself, but it can get tricky in multi-homed systems. The value that you need to provide here is the address that Team Foundation Server will be able to communicate to TFS Integrator on via the event notification service. Provided this address maps to the local host, the TFS Integrator service will self register the port as it starts up.

The next setting, RegistrationUserName is the account that you want TFS Integrator to register subscriptions under. Typically this is the same account that is used to run the service (especially in a domain configuration). Finally, the TeamFoundationServerUrl is the address of the Team Foundation Server that TFS Integrator will need to talk to.

Once the initial configuration is done the service can then be installed. So that it can run headless. You do this by running the “TfsIntegrator.exe” program with the “-i” switch.

 

After the installer registers the service you just need to go into the Services MMC snap-in (services.msc) and specify the logon credentials that the TFS Integrator service is going to use. This account should have access to TFS, be the same account as specified in the RegistrationUserName and have sufficient rights to write files to the “TFS Integrator” directory in program files.

Now that the basic configuration for TFS connectivity is out of the way its time to actually modify the ContinuousIntegration.xml file. The ContinuousIntegration.xml configuration file contains all the information that TFS Integrator needs to trigger the build of a “team build type” in Team Foundation Server.

The root element in the document is ContinuousIntegration, and under that you can have one or more TeamProject elements. The TeamProject element defines the name of the team project to which TFS Integrator will listen to events for. This element in turn contains multiple Build elements. The build element defines what path in the version control store to listen for changes in, what build type to kick off, and on what build machine. The sleep attribute defines the settle period that TFS Integrator will wait for further check-in notifications before spinning off the build.

Once you have made the modifications to the above configuration file you should be able to start up TFS Integrator. Congratulations! But what about DependencyReplication.xml?

What the heck is Dependency Replication?

Dependency Replication is a process that most development teams have to do as the scope of what they are trying to achieve grows. The idea is that within your organisation you might have a set of common frameworks that you use and maintain. Rather than linking in the source code into each new project that you undertake you treat the framework like an internal product which you build seperately.

The problem you then have is integrating the updated binaries into the other projects that depend on them. This can be quite a time consuming process, so much so that teams will often give up and end up compiling hundreds of different versions of the same code into their code base – which creates a maintenance nightmare.

TFS Integrator includes a dependency replication engine which extends the continuous integration feature. It does this by listening for the build completion even from TFS and using that to trigger the execution of some code which picks up the compiled build output and checking it back into Team Foundation Server in a specified location.

The effect is that with dependency replication the feedback loop is complete and dependant components can be linked together.

In the case of the framework example we could build up a scenario where TFS Integrator gradually rebuilds your entire code base.

  1. Developer checks in some code in the framework code base.
  2. Check-in notification sent to TFS Integrator.
  3. TFS Integrator kicks off a build in Team Build.
  4. Build completes.
  5. Notification of successful build completion sent to TFS Integrator.
  6. TFS interator checks in some of the drop files into version control.
  7. Check-in notification sent to TFS Integrator.
  8. TFS Integrator kicks off a build in Team Build.
  9. Build completes.
  10. Notification of successful build completion sent to TFS Integrator.
  11. Rinse, repeat.

In the past I’ve spent significant amounts of time getting similar systems going on top of version control systems like Visual SourceSafe where sharing was used as the replication mechanism. I actually think that this approach is much cleaner and more obvious.

How to configure TFS Integrator to support Dependency Replication?

Like continuous integration, the configuration of dependency replication is controlled via a configuration file – in this case, DependencyReplication.xml.

The root element of the configuration file is DependencyReplication. Within it there are multiple TeamProject elements which define the TeamProject and what the root folder is (I’m not sure what the root folder configuration item is off the top of my head, but I suspect its the relative path to which check-ins will be made). Within each TeamProject element, one or more Dependency elements are defined which specify which build type the team  project depends on. Inside each Dependency element is a File element which specifies the drop location relative source file any dependencies that need to get checked back into source control. The destination is the fully specified path to put those files into version control.

In the configuration file above we are saying that SystemB depends on some files from SystemA (Output1.dll and Output2.dll) and that those files should be checked back into the $/SystemB/SystemB/Dependencies location in version control. The interesting thing to note is that the file name is also specified in the Destination path. This allows configuration managers to rename files as part of the replication process.

Once both continuous integration and dependency replication configuration is complete, you need to restart TFS Integrator to get it to pick up its latest settings.

Next Steps

If you are interested in trying out TFS Integrator in your environment, just go ahead and download it. We are very keen to see people use this tool because it is genuinely useful to us, and we think it will be useful for you too. We probaly won’t be giving away the source code at this stage but unless a future version of Team Foundation Server makes it redundant we will continue to make revisions to the code base and support it (as best we can).

If you want to get support for TFS Integrator then I recommend that you join the OzTFS mailing list and community which was recently setup by Grant Holliday, you can subscribe via e-mail. Once you are subscribed you can send an e-mail with your question. If you can’t post to that mailing list for any reason, just send an e-mail to me.

Thanks

Finally – before I sign off on this post, I want to thank a few people. TFS Integrator was not developed by me – in fact I can’t even claim sole responsibility for its design. Both Darren Neimke and Chris Burrows both helped get this software out the door. Chris in particular put in hours above and beyond the call of duty to get this out the door.

About these ads

69 thoughts on “Getting Started with TFS Integrator

  1. Pingback: Rob Caron : TFS Integrator: A New Continuous Integration Option for Team Foundation Server

  2. Pingback: Team System News : VSTS Links - 09/19/2006

  3. Pingback: It's Way Too Early For This : TFS Integrator

  4. Pingback: Continuous integration for TFS « Systems Engineering

  5. Mei Lee

    We’ve done similiar implementation with the CI being a Web service. We have been using bissubscribe but it need to be run on the box. To add the subscription in the install is neat but if you need to change the subscription do you need to reinstall?
    Can the Check-in subscription filter on Team Project or even source code branch? The former is very important since a TFS can host multiple un-related team projects? I know this is done by adding “Artifacts” filtering in the Integration table but how do you use the API to get the filter in along with the subscription? Thanks.

  6. Mitch Denny Post author

    Hi Mei Lee,

    It actually subscribes each time the service starts, so if you change the configuration it will automatically adjust. The install process is actually quite “dumb”.

    Usually TFS will lisen to all check-ins and use its own configuration files to determine which ones it is interested in. This seemed easier than having to maintain two configuration repositories – one with TFS Integrator and one in the event subscriptions.

  7. Mitch Denny Post author

    Hi Mei Lee,

    If memory serves, we subscribe to all CheckInEvent notifications from TFS and do the filtering *inside* TFS Integrator.

  8. Pingback: Blog-a-Styx : TFS Integrator : l'intégration continue avec TFS

  9. Pingback: adamga's WebLog : Team system resources list, updated September 27, 2006

  10. Chris Chan

    Works great!!! We have been using it for about 3 months now. Dependency replication makes integration with other projects much easier.
    The best thing, once you spend time upfront setting it up, it saves you time ongoing in regards to configuration management.

  11. Mitch Denny Post author

    Wes – the installer is pretty lightweight, it just drops some files in C:\Program Files\TFS Integrator.

  12. Pingback: Buck Hodges : More continuous integration with Team Build

  13. Pingback: SRLTeam : Team System Utilities

  14. Pingback: Rob Caron : Continuous Integration to Continuous Distribution

  15. Pingback: Andrew Coates ::: MSFT : VSTS Resource Guide for Developers & Architects

  16. Pingback: MSDN Austria : Visual Studio Team System - Ressourcen

  17. Pingback: Chris Burrows .NET Blog : Setting Up Your Build Environment with TFS

  18. Pingback: TFS Integrator Updated « notgartner

  19. Piers Lawson

    In case anybody else is wondering, the Sleep attribute takes a value in seconds (e.g. Sleep=”60″ means TFS Integrator will let the system settle for 60 seconds before starting the build).

  20. Steve

    I installed TFS Integrator on my build server and created two projects in order to test the Dependency Replication. I have SystemA and SystemB as in the example where SystemB depends on SystemA. Continuous integration works fine for both systems, but no luck with dependency replication. Here’s what the event log looks like when I check in a file into SystemA:

    ————–
    Starting build for SystemA of type SystemAContinuous on Server TFSSRV in the directory c:\builds from TFS Server http://TFSSRV:8080/

    Build Started for team project SystemA of type SystemAContinuous. BuildUrl:vstfs:///Build/Build/12062006_154745_78176

    BuildSucceeded for team project SystemA of type SystemAContinuous at buildUri vstfs:///Build/Build/12062006_154745_78176
    ————–

    SystemA.dll is dropped in my drop location but no further notification occurs in order to update SystemB’s dependencies.

    Do you have any ideas of what might be wrong? My configuration files are exactly the same as the ones provided with the default installation of TFS Integrator.

  21. Piers Lawson

    I was wondering if you were considering open sourcing / codeplexing TFS Integrator… I have to admit to using Reflector to peek under the covers and I was impressed! It seems like a great framework on which to build other features… for instance I was about to start writing a web service to allow finer grain control over the notifications sent out when a build completes (e.g. email some people as every build completes and email others only when a build changes from “Successfully Completed” to “FAILED” etc.).

    A notification service could also be extendable. For example allowing MSN Messenger messages to be sent to interested parties, or creating a work item against the person who checked in the offending change etc etc.

    Obviously with the ease of install and configuration of TFS Integrator extending it would be far preferable than starting from scratch!

  22. Mitch Denny Post author

    Hi Piers,

    At the moment there is no plan to open source TFS Integrator. That said – take a look at the TfsListener class inside the Readify.Useful.TeamFoundation.dll, it is a general purpose listener for TFS so if you want to build something that listens to TFS you just need to use that.

  23. Steve

    Mitch, I found out why the dependency replication was not working for me. In fact I am using a localized version of TFS (French) and the CompletionStatus messages are also localized.

    In the function TfsIntegrator.DependencyReplication.DependencyReplicationServiceWcf.OnNotificationEvent
    the completionstatus is hard coded:

    if (eventRaised.CompletionStatus == “SUCCESSFULLY COMPLETED”)

    In the french version of TFS the message is “TERMINÉ AVEC SUCCÈS”

    So is there something I could to in order to make TFS Integrator work with a localized version of TFS?

  24. Mitch Denny Post author

    Hi Steve,

    That is a good one. I’ve logged this as a bug. At the moment there is no way around it, but we might be able to find another way to determine success of the build.

  25. Pingback: Getting Started with TFS Deployer « notgartner

  26. Matthew

    Hi Mitch,

    I like this tool – it is really cool but I broke mine!

    I had it all working last night and then today tried to point it to out actual TFS Server and now it does not respond to checkins.

    If I do a manual build from Visual Studio, it builds and does the drop no worries.

    The only thing that I see in the event log is “Service started successfully.” from TFS Integrator.

    Is there a debug flag or something that I can set to see what events TFS Integrator is receiving?

    Does it matter if project names have a space in them?

    Thanks mate,

    Matthew

  27. Mitch Denny Post author

    Hi Matthew,

    You can run TFS Integrator with the -d switch to find out what is going on interactively. Make sure that when you do this you change the configuration file to match the username that the process will be running under.

  28. Sandor Drieënhuizen

    Hi Mitch,

    First, thanks for putting effort in this great extension.
    However, I can’t get it to run. I’ve followed the steps in your guide and when I try to start the service, it shows:

    Service cannot be started. System.ServiceModel.AddressAccessDeniedException: HTTP could not register URL http://+:8000/ContinuousIntegration/. Your process does not have access rights to this namespace (see http://go.microsoft.com/fwlink/?LinkId=70353 for details). —>

    I’ve been playing with the base address to no avail. All TFS layers are running on the same machine by the way.

  29. Mitch Denny Post author

    Hi Sandor,

    Sounds like you need to give the account you are running TFS Integrator more rights. I usually give it local admin rights in my development environment as this is often the rights that the team build account has anyway.

  30. Manish

    I use a scheduled task to build System A and this task happens nightly, once this build is over i want to use TFS integrator to manage dependency replication. Is this possible, if so i have configured the TfsIntegrator.exe.config and DependencyReplication.xml as per the directions and when i build SystemA i was expceting System B to build but that didnt happen. The output of System A ia copied into the appropiate directory but from there on it doesnt get checked into TFS in the “Destination” neither does System B builds. Also is there some log which i can see which will tell me what happens when a build is completed successfully?

    Appreciate all the help

    Manish

  31. Manish

    Wanted to find out if dependency replication process will check in only *.dll doenst it support other file extensions too ?? Is there a way to handle other file types too like .tlb .xml.

    Also during this process if some dll are not generated by the dependent project but they are in the “Source” directory wont they be copied and checked in into the destination.

  32. Pingback: ISerializable - Roy Osherove's Blog : What Source Control tool do you use? And more on TFS Vs. Open Source tools

  33. Hugo

    Hi Mitch,

    I can’t use TFSIntegrator when auto-incrementing build numbers.
    I’m checking out the assemblyinfo file, modifying it and checking in again.
    After made this process, the build fires again in a recursive way. It happens because the assemblyinfo file is under the builds ChangedPath configured in ContinuousIntegration.xml.
    Any solutions?

    I sugest to add a Exclude section in Project tag to add files to ignore when checked-in.

  34. Kai

    I am having the same problem as Steve!

    I use a german version of TFS and have tried to get Dependency Replication running for hours now.

    Is there any fix for ?
    TfsIntegrator.DependencyReplication.DependencyReplicationServiceWcf.OnNotificationEvent
    where the completionstatus is hard coded as

    if (eventRaised.CompletionStatus == “SUCCESSFULLY COMPLETED”)

    In the german version of TFS the message is “ERFOLGREICH ABGESCHLOSSEN”

    Would you please release or mail me a version where I can change this string constant in TfsIntegrator.exe.config?

    This would be great as my project dependencies keep going our team nuts ;)

    Thanks!
    Kai

  35. Mitch Denny Post author

    Hi Kai,

    We are currently internally testing a new version of TFS Integrator 1.1 which moves all configuration into the version control repository (like TFS Deployer). I am hoping that my collegue will post about it soon – but in the meantime I have introduced a fix for this issue into the code base, so the next build that we publically release will address the problem.

    Thanks for raising the issue.

  36. Deepthi Rao

    This is a great tool you have built !! Thanks a lot.

    I had a doubt-can we do pseduo-builds in the same ? What i mean by this is, can we configure it to pick in just our changes (that have not been checked in yet) and build it with the rest of the checked in source code-this would be useful for 2 reasons :

    1.If i have a large number of files and first want to check if they build with the source. An actual checkin might cause a build break and unnecessary roll back of my changes
    2.It will help me obtain the new binaries with my changes, which I can use for further testing.

    Please do let me know.

    Thanks ,
    Deepthi

  37. Mitch Denny Post author

    Hi Deepthi,

    TFS Integrator doesn’t actually perform the build. We still get Team Build to do that, in which case you are ultimately limited by what Team Build does. TFS Integrator doesn’t really get involved in the details for the build.

    I guess you could script the build such that it grabs shelvesets.

  38. Deepthi Rao

    Thanks for the reply Mitch. It would be great if we could use it to pick up shelvesets-how easy is it to do this ? Could you give me more pointers to the same ?

  39. Bill Wolohan

    This is a nice tool Mitch! Thanks for putting it out there.

    I have a suggestion. Can you add an expression to be used to filter the check in events? I’m using AssemblyInfoTask to update the version numbers in all my components when building. As part of the build process, it changes the version numbers stored in AssemblyInfo.cs and checks it in. This causes TFS Integrator to start another build, and so on. If I could filter the events you catch I could tell it to ignore checkins by the build account and break the circuit.

    Thanks,
    Bill

  40. Mitch Denny Post author

    Hi Bill,

    That feature is actually in there for v2.0, but I don’t know if we will ever build it given what TFS 2008 has around features.

  41. Abhi Joshi

    Hi Mitch,
    It looks like greate tool, can this tool provides ability to automate check-ins into VSTS repository.
    My scenario is like this :
    I have two diffrent kind of builds, Lets say Build A and Build B. A build would get solution from lets say folder “Asourse” from vsts and build B from “BSource”. now i want that first bulid A get the entire solution from “Asourse” and perform the build and after successfully completition it put the code in to perticuler physical location in the build server (that we can accomplish through simple VSTS build configuration).
    Now I want to automatically move some components (lets say 2 aspx files) from “ASource” to “BSource” folder and start the build B. If the B build will be completed successfully then label of all files under “BSouce” folder should updated and if the B Build weill be failed then It will roll back the components from Folder “BSource”.
    Please advice me how can i achive this kind of approach.
    Thanks…….

  42. Mitch Denny Post author

    Hi Abhi,

    At the moment TFS Integrator doesn’t have a roll back capability. But you could probably figure something out like this in your MSBuild scripts.

  43. Pingback: Buck Hodges : Using CruiseControl.NET to trigger TFS 2005 builds for continuous integration

  44. Pingback: MSDN Blog Postings » Using CruiseControl.NET to trigger TFS 2005 builds for continuous integration

  45. Pingback: Where am I at with things… « notgartner

  46. BJHop

    Hello,

    I have been using Integrator for a while now and all was well until about 3 weeks ago. We had an issue with our TFS server where the C drive was completely full and it crashed hard. Ever since then our TFS Integrator has not worked. I’ve tried everything, I could think of, change the Website where it receives the events, start and stop the service, run the service from the cmd line with -d turned on.

    Nothing, I can see the subscription get added and removed each time I start and stop the service. The URL is open to TFS, TFS Deployer is on the same site and it works fine.

    Nonetheless, whatever I do it just looks like the Integrator is not getting the events. I check when I have it -d mode and have made a check in that should kick of a build yet I see nothing. I mean I would I least think I would see some kind of error if the Integrator was getting the checkin event and dying.

    If you could shed any light on how to better debug this issue that would be great.

    Thanks,

    BJHop

  47. BJHop

    These are the only two subscriptions I see in the tbl_Subscriptions
    169 CheckinEvent http://as73tfsbuild01:9090/ContinuousIntegration/Checkin
    170 BuildCompletionEvent http://as73tfsbuild01:9090/DependencyReplication/BuildCompletion

    Both URL look fine

    If browse to one of them I get the following:
    a:ActionNotSupported
    The message with Action ” cannot be processed at the receiver, due to a ContractFilter mismatch at the EndpointDispatcher. This may be because of either a contract mismatch (mismatched Actions between sender and receiver) or a binding/security mismatch between the sender and the receiver. Check that sender and receiver have the same contract and the same binding (including security requirements, e.g. Message, Transport, None).

  48. Pingback: Getting Started with TFS Continuous Integrator « notgartner

  49. Pingback: Obsolete: TFS Integrator « notgartner

  50. Pingback: Continuous Integration resources for Team Build - GrabBag<T>

  51. Pingback: TFS Guide now available - GrabBag<T>

  52. Pingback: Paul Stovell says… » TrialBalance Update

  53. Pingback: Setting Up Your Build Environment with TFS « MadTechnology

  54. Don

    It looks like greate tool, can this tool provides ability to automate check-ins into VSTS repository.
    My scenario is like this :
    I have two diffrent kind of builds, Lets say Build A and Build B. A build would get solution from lets say folder “Asourse” from vsts and build B from “BSource” /now i want that first bulid A get the entire solution from “Asourse” and perform the build and after successfully completition it put the code in to perticuler physical location in the build server (that we can accomplish through simple VSTS build configuration).

  55. Pingback: TFS Deployer and TFS 2008 « Code Assassin

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s