How to get teamcity agents to run git

TeamCity works like a charm with git. All explained very well in the official docs. No problem at all.

However, in my setup, with teamcity agents running as a server under local system, the blessed git server running gitolite with ssh-keys as access control things got a little hairier.

The problems were many, and here’s what I finally did to get it running:

* Install Git Extensions for Windows (https://code.google.com/p/gitextensions/)

* Download psexec and unzip it (http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx)

* Generate a key using “ssh-keygen -t rsa”. You’ll save yourself a world of trouble if you do NOT use a password to protect the key.

* Use puttygen to create a putty-style key.

* For sake of simplicity, assume the keys are in ~/.ssh aka c:\users\\.ssh, named id_rsa, id_rsa.pub and id_rsa.ppk.

* Create a textfile in home dir named c:\users\<USER>\git_ssh.sh. It should contain the connection script for ssh using plink

* Add a system level environment variable GIT_SSH with the content “c:\users\<USER>\git_ssh.sh” (no quotes).

* Start a cmd-shell as administrator

* Add your git server to the cache by connecting to it using plink. The command line is typically something like “c:\Program Files (x86)\GitExtensions\PuTTY\plink git@gitserver” – your default user name and server name may vary.

* Use psexec to start a command shell as the system account – “psexec -s cmd.exe”

* Add your git server to the cache of the system account – “plink git@gitserver”

* Restart the teamcity service.

So, what happened? The variable GIT_SSH, as per the git manual is the command git uses to set up the connection to remote. We replace it with plink and a reference to the key. Another option commonly used is to use pageant for this, but that is tricky (though not impossible) when it’s a service that needs the info.

Connecting to the server with plink adds the server’s finger print to the list of known hosts. That list is per user (sits in HKCU) which is why we need to do it both for the logged in user and for the local system account.

Could this be done differently? Sure, no doubt! It’s just that after spending the main part of the afternoon in blind alleys I figured I’d offer a pair of shoulders for others to stand on.

If I’d do something different I’d probably consider replacing plink with ssh and specifying the key in the /etc/ssh/ssh_config-file (http://www.manpagez.com/man/5/ssh_config/). That probably eliminates the need for both the psexec trick and setting the GIT_SSH variable. But now I’ve updated all the agents…

–Jesper

Why I don’t install new components

I am no enemy of other people’s code. On the contrary. If my team can leverage the effort put in by someone else, so we can reach our goals quicker I am all for it.

It’s the installation I am against.

Any decent component pack somes with an installer. It puts some stuff in $(ProgramFiles), maybe a few examples under $(MyDocuments) and the dlls in the GAC. Fire up VS and start using the new stuff. Add new references to the GACed assemblies and code away. Looks good, compiles cleanly – great.

Now check this in and ask a co-worker to pull your work. In a real world scenario, things would break quite immediately, because the build agents do not know about what you have installed on your machine. Co-worker also gets the same compile error about missing references.

So, either you fix the build agent yourself or ask the build master to upadte it. Or them. There might be tens or more agents that all need this set of dlls. And of course your co-workers all need to spend time downloading and installing the bits.

After a while, you realize there’s a bug somewhere in the component, but the latest update fixes it. Download and install. Cater for the API changes and check in. Oh, and mail everyone they need to upgrade too, and pretty please mighty build master, could you please upgrade all the 20 agents again?

If things are done fairly rightly, the compile time bits may very well coexist in the GAC. Not so sure about the design time bits in VS. So, if you work in two projects, and one project can’t update right now, which version do you chose to use on your machine?

Better solution: Do not reference files from the GAC. Check in ALL binaries required by your application during compile time (actually you’d want to make available the entire bit set and tool chain your app wants in the process of turning text files into an ISO image ready to send to production, but that’s another chapter in the same story).

For sake of simplicity, assume we use the CVS as file repository.

Check in your fancy new component suite or function library (named <product> in the following examples) under $\xternal\<product>\<version>\<all the files>.

Add a prebuild step to copy from $\xternal\<product>\<version>\<all the files> to $\stage\xternal\<product>.

Make sure your projects reference files from $\stage\xternal\<product>.

Check in.

Note: Never ever check in things under $\stage. It is a temporary directory used for build artefacts. It’s as wise to check that in as checkin in the stupid bin\debug or obj\release directory under your project.

In case you didn’t follow, here’s what will happen on the build machine: Compile succeeds! The bits it needs are there when the compiler looks for them. On your co-workers machine: Compile succeeds, much for the same reason.

Additional benefits materialize when you decide you need to go for that upgrade. Just check in the new files under $\xternal\<product>\<new_version>. Change your prebuild step to copy from <new_version> instead of <version>. There will be no need to upgrade the project references. The references will change, yes, but to new files in the same location.

No need to waste everyone’s time by forcing them to upgrade agents or own machine. Much less risk of running into version incompatibilities, as everything is sandboxed.

If you really want to go all in on this, you should not use the VCS as file repository, but a separate file server (with proper back-up routine in place). Your prebuild step should download/copy from that external server, which will be available to all projects. Thus, you adding the <new_version> will make it available to the project down the hall, should they be interested.

–Jesper

Foundations of a dev team

Last week I spoke to an old friend about what fundamental advice to give to a development team. There are many areas that needs catering to. I tend to start with setting up a machinery into which you put source code and out comes a nicely packaged installer. And then the rest will follow, sort of

Here is my list of build machine/infrastructure centered focus areas – some technical and some attitude based.

* Use a source control system – be it svn, git, p4 or something else.
* Use a build machine (TeamCity, CC.Net, Cruise, Jenkins, …)
* Make that build machine trigger a compile of everything on check-in.
* Write unit tests.
* Run the unit tests on build machine after the checkin-build.
* Run static checkers and source analysis (fxcop, stylecop, cppcheck, cpplint) on the build machine on each check-in.
* Break the build on failure.
* Make build failures obvious (lava-lamps are great!).
* Foster a culture of checking in often.
* Foster a culture of keeping build-broken times to a minimum.
* Do not assign shame to a broken build.
* Make sure whatever your build machine does can be executed on every developer machine.

The above statements leave a lot to be worked out, I know. However, no matter how you solve it you are better off if you do!

If this seems like a tall order, you’ll probably need some help. Not to worry! The internet is full of advice.

Have fun setting up your team and the machinery to support it. I do look forward to your input!

–Jesper Hogstrom

Jesper Got Published: TWIM #21

I wrote a letter about what cancer is to This Week In Microbiology, and it got published in episode #21. Here’s the letter:

Dear Vincent,

the other day I was in a discussion about what can get cancer, something that ultimately boiled down to what cancer really is. Our reasoning went along the lines of establishing that there are organisms containing any number of cells, ranging from one and up. If I remember correctly, C. Elegans has 957 cells. Presumably there is some organism with 956, 955 and so on.

It seems it doesn’t make any sense to talk about a one celled organism developing cancer – though I am interested to have that confirmed! The nematode just mentioned has cell specialization, so it could presumably develop some form of cancer. What is the lower limit of cells an organism must have to succumb to the decease or should the question really be posed in a completely different way?

Also, some organisms of very few cells occasionally gang up and form a super-organism. This includes some slime moulds and the pre-larvae state of jelly fish. Can such “temporary” organisms develop cancer?

The question is grander than just parasites, and I have a feeling that viruses, living or not, have no propensity to develop cancer. Hence my addressing the question to TWIM.

While I have your attention, allow me to once again thank you and everyone in each of the podcast teams for your effort in sharing your knowledge and doing it in such an enjoyable tone and fashion.

All the best,

Jesper
Software architect
Sweden

The question is answered by Vincent andElio near the end of the show.

Recursive msbuild

Today a co-worker asked me a really interesting thing about msbuild. He wanted to extract some information about the binary he just compiled into a textfile and add that file as a resource to the binary he just compiled. It is worth mentioning he had a really good reason to do this as well.

The good reason and the fact that it was quite an interesting question got me thinking on how to do that with minimal effort. Minimal effort is always a good sign of a lazy programmer, and lazy programmers are better. At least that’s what I read somewhere

First of all, it wouldn’t be possible to simply call the target Build again, as msbuild doesn’t invoke the same target twice in one run.

I figured it’d be necessary to call msbuild again, and in the recursively invoked msbuild to all the tricks.

First off, I wrote a simple program to extract some information from a binary file.

Not overly complex, but good enough for demonstration purposes. I compiled this to genapi.exe and moved it to the root folder of the project I wanted to tamper with.

Msbuild is a quite powerful system, and it has numerous extension points. One of these is to override – or actually redeclare – a target named AfterBuild. This target is invoked after the chain of targets pertaining to build is complete. The default implementation of AfterBuild is to do nothing.

At this point I need to mention the importance of understanding the build system used. There are quite a number of problems that can and should be solved by extending and enhancing the build process. The first step on that journey to enlightenment is to start thinking about the build system as part of the product development infrastructure. It is not just some mumbo-jumbo mixed with two bottles of magic that happened to ship with .Net and Visual Studio. It is an orchestration environment of quite some power.

Anyway, start with right-clicking the project node in the solution explorer and select Unload project. Right-click the same node again and select ‘Edit…’. You’ll now get the csproj file as a textfile in your source editor in VS.

The end of the project file looks like

This is exactly what we want to do. First, to try the waters, I wrote a simple AfterBuild target just to make sure it got executed.

Reload the project file, and turn on more logging by selecting Tools | Options | Projects and Solutions | Build and Run and go for Normal in both the combo-boxes. View output from build by Ctrl-W, O.

Rebuild and you’ll get something like

Great. Build failed.

Now let’s do some serious work.

What we need is to generate the “api”. The target for this looks like

The generated file, ‘api.txt’ needs to be included. We could in theory add it to the project once we have it, but imho it makes much more sense to only add it after it is generated. Thus, we need to modify the Resource item. To avoid doing that too early, this must be done in a target as well. Manipulating items and properties in targets is msbuild’s way to handle deferred evaluation.

Now what with the afterbuild target? There are many ways to skin a cat, and I opted for two invocations of the csproj file, first to generate the resource, and then to rebuild the project with the freshly generated file.

However, the code above suffers from a serious mistake. When msbuild is invoked for the second time, it will surely call build, but it will also end up in the AfterBuild-target and call itself again. Infinite recursion. Not good. To prevent this from happening, we need to pass a token indicating we’re in level two, and also check that before executing anything. Improved version looks like

Much better! However, when are we calling the AddResource-target? To fix that we need to do some indirection, as msbuild doesn’t seem to commit item changes to global scope until a target is completed. Thus, instead of invoking the Build-target, we invoke a new target named SecondBuild. It in turn depends on AddResource and Build.

The complete code that works thus looks like:

Staying true to the agile programming cycle, this is where one should pause and refactor. Could the code be written any cleaner? Do we need two invocations of msbuild? As it turns out, no, not really. The same effect can be achieved with the following, more compact version:

There you go. A simple way to include information about a binary in the binary itself. All from one Build request from inside VS.

A very interesting question and a fairly simple and clean answer.

Happy building!

–Jesper

Setting up an msbuild script system

msbuild is not all that hard, but it does come with a whole slew of parameters. You don’t want to memorize them all. Thus, define a good ol’ batch file to launch msbuild:

This simple little file will allow you to type things like build debug or build release without worrying exactly how to specify the target. If you want to make it extra nice, control log level with an environment variable, by adding the following line:

simply set loglevel=/v:n from command line to change the log level.

You may also want to pass in some variables, like current directory

That will give you the property $(StartDir) available inside your project file.

Speaking of which, here’s a minimal sample:

Admittedly very bare bone.

Let’s add some meat:

If you follow the good practice of adding new targets to the item list _availableTargets, you have a help page showing if you pass in no parameters.

Obviously we’ll have to write the actual targets too, but so far we’ve seen that
* We can add things to a list.
* Display that list with a different separator than semi-colon.
* Define properties with escaped characters (by using the hexadecimal ascii values prefixed by a percent character).

It’s far from complete, but it’s a very good start!

–Jesper Hogstrom

Designing a build system

When developing software it is somewhat obvious that the programmers need to be able to build the source code and run the tests on their own machines. It is slightly – but only slightly! – less obvious that the same code needs to build on a separate machine, known as a build machine. There are several purposes of that, mainly to

* Ensure everything compiles after each check-in (structural integrity)

* Execute unit tests to ensure functional integrity.

* Run various kinds of static analysis on the source or binaries.

* Prepare builds for internal (e.g. QA) and external (e.g. customers) consumption.

A build machine is typically controlled by a build orchestrator, like Cruise (now a part of Go), CruiseControl.Net, TFS or TeamCity. It may be a single machine, a system of agents or something else. Let’s call this the build machinery.

The build orchestrator is in some way connected to the version control system (VCS), be it Perforce, Git, Subversion or TFS.

When a file is updated in the VCS, the build orchestrator gets a signal and starts executing a build script, typically written in msbuild, ant/nant, make or some shell language. It is to prefer to use a language that is designed for build orchestration, thus allowing for specification of dependencies, error handling and maybe even logging.

The fundamental components of a build system are thus:

* A build script

* A build machinery

* A version control system

Luckily, the build machinery and version control system are standard products, so you don’t need to worry about rolling your own.

However, the build script is something where you want to invest some energy. The purpose of the build script is to ensure that you can reproduce a bit-identical build from bit-identical sources. It should also have the ability to execute a suite of other targets.

A target is the end result of a build script invocation, or rather, what you want the script to produce – the transformation of inputs to an output.

Some examples of targets you want your build script to produce:

* Debug - A debug build of all sources.

* Release - A release build of all sources.

* Testcases - Compilation of all unit tests.

* Documentation - Collect all documentation source documents and transform them into the desired output format.

* Installer - Collect all files that goes in to the installation file and produce it.

You can also have your build script do a number of other things, such as

* RunTests - Execute all unit tests in the test framework you use.

* RunSmokeTests - Execute all UI level tests using whatever test robot you use.

* Publish<something> – Upload some artifact to a different server for wider consumption.

* Clean<something>, Spotless – Remove output from the source tree, ensuring next build creates everything from source documents.

An important aspect of a build script is that it should be possible to execute both on the build machiner/build agents and the developer machines!  You should be able to “make debug” or “build documentation” or “want testcases” from a command line prompt on all relevant machines. Otherwise it is very hard to perform an action on a developer machine to ensure the check in will pass. It also helps a great deal when developing the build script.

There are some really critical consequences of the above:  The build system must be self-contained! This means everything required during compile time must be reachable by the build machinery – and if it isn’t checked in you need some other way to pin which version (of a tool, SDK or third party tool) you want. Speaking of which, you’ll also need to version control the build scripts themselves.

So, to summarize what you need to get started:

* A build machinery.

* A version control system.

* A build script that automates everything that happens between coding and preparing a testable artifact.

* Everything required during compile time and test time reachable by the build script (either checked in or reachable on another file system).

The build script and repository of prerequisites require by far the most design – or rather – those two items are by far the least described out there.

I’ve designed build systems using various build machineries (CruiseControl.Net, TeamCity, TFS) using both make and msbuild. Nowadays I write msbuild scripts building .Net and C++ projects for a teamcity environment. However, there are a number of concepts that do not change. I’ve found the most important to be:

* Do not rely on what is installed on the machine you’re building on. Everything must be controlled!

* Designate an output/staging directory, and make sure everything that is produced ends up in under that directory.

* Define a structure of all projects and stick to it.

* The build should behave the same way if building from inside an IDE as when building from command line.

Everything must be controlled

Developers typically install something that goes into a build, add some references to globally available (GACed in the .Net world) DLLs and everything works just fine. Until the build happens on a different machine, when things blow up, as the global state of that machine isn’t the same as on the dev machine.

It is imperative that everything you use during compile time is checked in! This means not only your third party components, but also the unit testing framework and the tools used in the build chain, such as help compilers, installer creators and so on.

Often it is quite simple to meet this requirement. At other times it is harder, like when COM-servers need to be registered, registry entries must be found and other evil global conditions that must be fulfilled. As the build master, your job is to understand how you can script things to make all machines look like that have enough of the global state to fool the tools and components you use.

If you adhere to this rule adding a new build agent, or replacing a developer machine is very much simpler, as fewer things need to be installed.

I draw the line at the .Net level – all involved machined are allowed to have .Net installed. It would be possible to write scripts that didn’t rely on the presence of a .Net installation, but I find it not worth it at this point. If we were targeting strictly different .Net versions I might reconsider.

Staging directory

Visual Studio adds a bin and obj directory under each project. Not nice. You’d rather have all output end up under stage\debug or stage\release or stage\installer. Keeping everything under a stage directory reduce the risk of someone checking in binaries they’ve just built, and makes cleaning the tree significantly easier. Instead of traversing the world, you just rmdir stage\debug. Or rmdir stage if you really want to make things spotless.

Achieving this is not incredibly difficult, as all compilers tend to have a flag indicating where they should deliver the output. The trick is finding the sweet spot that minimize the amount of configuration required for each new project.

Project Structure

Project structure makes it easier to find your way around projects, reduce the learning time when moving to a new team and makes it much easier to write a build script that works across the board, as everything looks quite the same. I don’t know what structure makes sense to you, but something like

-$+
  +-Build
  +-Common
  +-Documentation
  +-Source
  +-Stage
  +-Tests
  +-Xternals

is a good start. It is also fairly obvious what goes where. Try to keep the root directory rather clean!

Behave the same from command line and IDE

This is trickier than it sounds. Assume you set up a wrapper msbuild script that calls msbuild to build your solution file. You play nice and pass in whatever properties you need to the second level of msbuild and everything works as expected. From command line.

From inside Visual Studio however, you don’t get the parameters passed in from your script, as VS is now the root caller. Your fellow developers will get a different result than you and the build agents. There are likely several ways to accomplish this, and they all involve some manipulating the project files (csproj, vcxproj). If you are using some other IDE there are surely other ways.

That just about covers the high level aspects of a build system. Admittedly there are details left to cover, but they lack the generality I wanted for this post. The above should work for most circumstances and most platforms.

–Jesper Hogstrom

Using unit test names as specifications

We spend a lot of time writing test cases and they sure serve us good. Wouldn’t it be nice if we could use the tests as a specification of the capabilities of the tested object?

Let’s say we have a test that looks something like

In essence, the tests spell out the specs for MyCoolGizmo! It has the following capabilities
* Adding Only Permitted In Transaction
* Invoking Add AddsItemToList
* InvokingExecuteRunsExecute On All Items
* …

Since that’s exactly how we named the tests (what a coincidence ) all we need to do is extract that from the code in some way. My first choice is always reflection in situations like this.

Here goes:

First, create a command line application.

Add some supporting functions and definitions

We’ll also need to define the interface for the generators and a class to store the logged items in.

The scanner turns out to be quite straight forward

The only remaining this is to define the result generators, which can of course be changed to suit your needs to produce other formats should that be desired.

To compile you’ll need to add a reference to nUnit. Then run it with appropriate switches, something like

Writing msbuild tasks declaratively

It comes a time in every build admins life when writing custom msbuild tasks can no longer be avoided. I myself is no stranger to writing my own tasks. However, when it’s time to tidy up all those pesky ‘Command Exec=”…”‘ it seems there must be a better way than subclassing ToolTask and construct the command line manually by overriding GenerateCommandLineCommands.

What I set out to do was to have a declarative way to define how parameters are constructed, all driven by an engine in a superclass. Declarative means attributes.

As an example, I want to wrap rsync in a task class. rsync has a great number of parameters, and I want to write something like

I also envision that I will have to write more wrappers of command line tools, and I don’t want to write a command line generator for all of them.

There are only a few types of parameters to pass to command line tools:
* Existence of a parameter, like “–recursive”.
* A switch with a value, like “–password-file=FILE”.

Now, if you’re not familiar with rsync, too bad for you! However, the reasoning holds for any executable you want to wrap in a task instead of using Exec.

Thus, in my subclass, I would like to specify this as something like

The way to go is to first define a couple of simple attributes.

Now we need an intermediate class to operate on these attributes. The most fundamental level is to construct the path to the tool.

This makes sure you can set the toolpath if you want to.

As you notice, I subclass ToolTask, as the msbuild team as done a great job of some parts of the infrastructure.

The key part is to make magic in the virtual GenerateCommandLineCommands.

The above, rather simple code, achieves the main objective. Boolean attributes cause a switch to appear or not appear, omission of an attribute prevents a nonsensical switch to appear.

For debugging purposes, let’s add a DryRun option. This should print out the command line, but not execute anything.

At this point you’ll notice that the output is not shown unless you increase the loglevel to msbuild. Why not make that declarative too?

This allows you to specify if the output of your tools when should show up.

In some cases you probably want a different default value than the .net defaults. Luckily there is already an attribute defined for this: DefaultValueAttribute.

In the rsync case, we might want to set recursive as default, specified as something like

Add to our base class the following code

As much as we all love writing documentation, we know that the others won’t read it. Why not make our tasks self documenting? What if we could add “ShowHelp=’true’” to our msbuild script and get the meaning of each parameter explained?

Again, we make use of an existing attribute, DescriptionAttribute.

Add it as follows in your subclass:

and add the following to the base class:

This will yield an output similar to

The full code for the base class:

A sparse implementation of rsync:

As you can see, there is no code – other than the declarative parts – required to wrap a command line excutable into an msbuild command, regardless of the number of parameters it needs.

Happy building!

–Jesper Hogstrom

Chilispetsat glöggrecept

Efter en höst med god skörd av äppelvin har det bu blivit dags att göra glögg. Efter lite experimenterande har jag kommit fram till följande:

Ta en påse Santa Maria Glöggkryddor och lägg i ett kärl.
Ta två skivor (3-4mm) ingefära och lägg i samma kärl.
Klipp ner ringar (2-3mm) av en halv thai chilifrukt i ovan nämnda kärl.
Häll över ca 1.5-2 dl vit sprit, vodka e dyl.

Vänta i ett dygn (det är den svåraste delen ).

Häll ner alltihop i en gryta, häll ner 1 flaska halvtorrt vitt vin (jag tar naturligtvis mitt eget äppelvin!).
Tillsätt ca 1.5dl muscavadosocker. Öka i förhållande till vinets torrhet!
Koka upp under varsam omrörning.

Stäng av plattan och häll ner 0.5-1 dl enkel whiskey (inte rökig singlemalt!) och en skvätt gin. Om du har lite sherry eller portvin så häll på en skvätt av det med. Det mjukar upp smaken.

Rör om och vänta ett par minuter. Häll upp i muggar – jag menar glöggkoppar och njut!

Som tillbehör använder jag kandisocker, mest för att i efterhand kunna balansera sötman i drycken. Lagom sött = inget kandi! Aningen för torrt – 1-2 kandi-bitar. Och så vidare.

Chilin ger glöggen en eldighet som värmer gott i vintermörkret.

När allt tagit slut kan du ta vara på kryddorna, lägga tillbaka dem i kärlet och hälla på ny sprit. Imorgon kvaäll kan du börja om. Då och då kan du lägga i mer av antingen kanelstång, ingefära eller nejlika.

–Jesper