ScriptCs, MSBuild and TaskFactories

Writing as much build scripts as I do, it’s always interesting to improve the scripts. One way is to write custom tasks to wrap command line tools. I elaborated on this topic in the post about writing tasks declaratively.

However. that doesn’t always cut it. There are some overhead created by writing code, compiling it to a dll, figuring out if the source or the dll should be source controlled and in some cases what you wrap is a script, so the number of parameters may be in a flux.

Scripting languages are quite useful in build systems. What came to my attention a while back while attending Leet Speak 2013 was the ScriptCS language.

The problem with adding scripts to an msbuild-based build system is that you end up with lots of code looking like

<Target Name="convert_data" Inputs="in,txt" Outputs="out.txt">
  <Exec Command="scriptcs -script convert_data.csx -- in=in.txt out=out.txt"/>
</Target>

Especially the “exec command” lends itself to lots of mistakes. It is simply nicer imho to be able to write

<Target Name="convert_data" Inputs="in,txt" Outputs="out.txt">
  <convert_data In="in.txt" Out=out.txt"/>
</Target>

However, that brings us back to the problem of having to write a wrapper around the script in a dll and why in that case bother using scriptcs?

Enter task factories. A task factory allows you to specify in msbuild how to dynamically create a task. In the box you’ll find a task factory for c# defined tasks allowing you to write an entire task in c# inside an msbuild script. Kind of neat, though writing c# in a CDATA section in a text editor is not a very convenient way to write code. I miss code coloring and the rest!

Anyway, my idea was to create a task factory that wraps a scripcs script allowing you to use it as a first class msbuild tsk later on. Turns out that is indeed possible. You’ll get something like

<UsingTask
  TaskFactory="ScriptCsTaskFactory"
  TaskName="convert_data"
  AssemblyFile="MsBuild.ScriptCs.dll">

  <ParameterGroup>
    <In Required="true" ParameterType="System.String" />
    <Out Required="true" ParameterType="System.String" />
  </ParameterGroup>
  <Task>convert_data.csx In={MyParam} Out={Out}</Task>
</UsingTask>

<Target Name="convert_data" Inputs="in,txt" Outputs="out.txt">
  <convert_data In="in.txt" Out=out.txt"/>
</Target>

The magic happens of course in the ScriptCsTaskFactory. It allows writing a single piece of code that can wrap any scriptcs script and use it in a type safe way in msbuild.

The task factory class itself looks like this:

using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.Build.Framework;

namespace MsBuild.ScriptCs
{
    public class ScriptCsTaskFactory : ITaskFactory
    {
        private IDictionary<string, TaskPropertyInfo> _parameterGroup;
        private string _taskBody;

        public bool Initialize(
            string taskName,
            IDictionary<string, TaskPropertyInfo> parameterGroup,
            string taskBody,
            IBuildEngine taskFactoryLoggingHost)
        {
            _taskBody = taskBody;
            _parameterGroup = parameterGroup;
            return true;
        }

        public TaskPropertyInfo[] GetTaskParameters()
        {
            return _parameterGroup.Values.ToArray();
        }

        public ITask CreateTask(IBuildEngine taskFactoryLoggingHost)
        {
            return new ScriptCsTask(_taskBody);
        }

        public void CleanupTask(ITask task) { } // Intentionally not implemented
        public string FactoryName { get { return GetType().Name; } }
        public Type TaskType { get { return GetType(); } }
    }
}

It in turn relies on the ScriptCsTask.

using System.Collections.Generic;
using System.Diagnostics;
using System.Text;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;

namespace MsBuild.ScriptCs
{
    public class ScriptCsTask : Task, IGeneratedTask
    {
        private readonly StringBuilder _commandLinePattern;
        private readonly Dictionary<string, object> _parameters = new Dictionary<string, object>();

        public ScriptCsTask(string commandLine)
        {
            _commandLinePattern = new StringBuilder(commandLine);
        }

        public override bool Execute()
        {
            StartScriptCs(ScriptCsArguments());
            return true;
        }

        public void SetPropertyValue(TaskPropertyInfo property, object value)
        {
            _parameters.Add(property.Name, value);
        }

        public object GetPropertyValue(TaskPropertyInfo property)
        {
            var propertyName = property.Name;
            Log.LogMessage(MessageImportance.Low, "Requesting property '{0}'", propertyName);

            if (_parameters.ContainsKey(propertyName))
            {
                return _parameters[propertyName];
            }

            Log.LogError("Property '{0}' not found", propertyName);
            return string.Empty;
        }

        private void StartScriptCs(string commandLine)
        {
            Log.LogMessage(MessageImportance.Low, "Calling ScriptCs with: [[{0}]]", commandLine);
            var p = new ProcessStartInfo(@"scriptcs.exe", commandLine)
            {
                UseShellExecute = false,
                RedirectStandardOutput = true,
                RedirectStandardError = true,                
                CreateNoWindow = true
            };
            var process = Process.Start(p);
            process.OutputDataReceived += (sender, args) => Display(args.Data);
            process.ErrorDataReceived += (sender, args) => DisplayError(args.Data);

            process.Start();
            process.BeginOutputReadLine();
            process.BeginErrorReadLine();

            process.WaitForExit();
        }

        private void DisplayError(string data)
        {
            if (data != null) Log.LogError(data);
        }

        private string ScriptCsArguments()
        {
            var commandLine = BuildCommandLine();
            string scriptCommandLine;
            if (commandLine.Contains("--"))
            {
                scriptCommandLine = commandLine;
            }
            else
            {
                var cmdLine = commandLine.Split(new[] {' '}, 2);
                scriptCommandLine = cmdLine[0];
                if (cmdLine.Length == 2)
                {
                    scriptCommandLine += string.Format(" -- {0}", cmdLine[1]);
                }
            }
            return scriptCommandLine;
        }

        private void Display(string data)
        {
            if (data != null) Log.LogMessage(MessageImportance.Normal, data);
        }

        private string BuildCommandLine()
        {
            foreach (var parameter in _parameters)
            {
                var name = string.Format("{{{0}}}", parameter.Key);
                var value = parameter.Value.ToString();
                _commandLinePattern.Replace(name, value);
            }

            return _commandLinePattern.ToString();
        }
    }
}

What happens is that the task factory takes the task body and passes it to the task as the command line. This command line is then massages using simple string substitution replacing strings on the format “{parameter}” with the value of the parameter named “parameter”.

The massaged string is then used when launching scriptcs.exe.

I posted the code for this on github available as git@github.com:jhogstrom/taskfactories.git.

Share

Kravlista MP3-spelare

Jag tyckte det var dags att uppgradera min 10 år gamla 1GB MP3-spelare, så jag stack iväg till Clas Ohlson och köpte en som verkade hyffsad. Nu är min kravbild lite klarare.

Strömhantering

  • Man skall kunna sätta på den och starta uppspelning med en enda knapp.
  • Knapparna skall alltid fungera, dvs inte trycka på nån knapp för att “aktivera” den för nästa knapptryckning.
  • Det skall finns en sleep-mode.
  • Gärna inbyggt laddningsbart batteri. Om den laddar ett vanligt AAA-batteri via USB är det oxå ok.

Display

  • Backlit LCD-display. Inga färger behövs.
  • Aktuell låt skall synas.
  • Det skall finnas en progressbar som visar hur långt innevarande stycke spelat.
  • Släck skärmen efter 5-10 sekunder.

Innehållshantering

  • Det skall gå att vandra runt i filträdet på ett lätt sätt.
  • Den skall komma ihåg position och låt mellan sessioner.
  • Det skall gå att spela låtarna i tidsstämpelordning (låt för låt, bibliotek för bibliotek)
  • Den skall spola i minst 5x, gärna 10x.
  • När man backar låt skall den starta innevarande låt från början om man kommit mer än 3 sekunder in i låten. Inte föregående låt.

Datorkoppling

  • USB-koppling. Varför inte microUSB på devicen?
  • Gärna USB2
  • Devicen skall mounta som en disk. Inga program för synkning skall behövas.

Knappar och fysisk layout

  • Tydlig taktil skillnad på knapparna. Jag vill inte behöva titta när jag knappar. Jag ändrar volym eller hoppar till nästa låt. Vill veta vad jag trycker på!
  • Knapparna skall sitta BORT från upphängningen.
  • Lursladden uppåt (och knapparna neråt) när den hänger.
  • Ögla för att hänga den i halsbandslurar.
  • Liten och lätt är bra.

Övrigt

  • Ingen volymvakt. När jag cyklar behöver jag ibland höja volymen, och då vill jag inte ha en knölig knappcombo för att öka förbi nån gräns.
  • Den skall slå av sig om inget spelas och inget händer. Väntetid bör vara minst 3 minuter.
  • Minst 2GB minne.
  • Den skall hantera mp3-format.
Share

Handling dangling mounts after schroot terminates unexpectedly

Typically you’ll get something like this at some point:

E: 10mount: umount: /var/lib/schroot/mount/precise-1f302441-6a1d-4adf-8013-24cea78c02d4/dev: device is busy.

Issuing mount you’ll see there are stuff left dangling…

$ mount
...
/proc on /var/lib/schroot/mount/precise-1f302441-6a1d-4adf-8013-24cea78c02d4/proc type none (rw,bind)
/sys on /var/lib/schroot/mount/precise-1f302441-6a1d-4adf-8013-24cea78c02d4/sys type none (rw,bind)
/dev on /var/lib/schroot/mount/precise-1f302441-6a1d-4adf-8013-24cea78c02d4/dev type none (rw,bind)

umount won’t do it:

$ sudo umount -f `mount|grep '/var/lib/schroot'|awk '{print $3}'`
umount2: Device or resource busy
umount: /var/lib/schroot/mount/precise-1f302441-6a1d-4adf-8013-24cea78c02d4: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount2: Device or resource busy
umount2: Device or resource busy
umount: /var/lib/schroot/mount/precise-1f302441-6a1d-4adf-8013-24cea78c02d4/proc: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount2: Device or resource busy
umount2: Device or resource busy
umount: /var/lib/schroot/mount/precise-1f302441-6a1d-4adf-8013-24cea78c02d4/dev: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount2: Device or resource busy

Probably udevd is holding a lock. Let’s see:

$ sudo lsof `mount|grep '/var/lib/schroot'|awk '{print $3}'`|grep udev
udevd 3573 root 0u CHR 1,3 0t0 1029 /dev/null
udevd 3573 root 1u CHR 1,3 0t0 1029 /dev/null
udevd 3573 root 2u CHR 1,3 0t0 1029 /dev/null
udevd 16423 root 0u CHR 1,3 0t0 1029 /dev/null
udevd 16423 root 1u CHR 1,3 0t0 1029 /dev/null
udevd 16423 root 2u CHR 1,3 0t0 1029 /dev/null
udevd 16425 root 0u CHR 1,3 0t0 1029 /dev/null
udevd 16425 root 1u CHR 1,3 0t0 1029 /dev/null
udevd 16425 root 2u CHR 1,3 0t0 1029 /dev/null

Kill it!

$ sudo kill -9 3573 16423

With some luck you can now unmount all the bind-mounts:

$ sudo umount -f `mount|grep '/var/lib/schroot'|awk '{print $3}'`

Unfortunately udevd will respawn, so you may need to reboot and restart from the lsof command. Good luck.

I guess a fix is in the making for schroot

–Jesper

Share

How to get teamcity agents to run git

TeamCity works like a charm with git. All explained very well in the official docs. No problem at all.

 

However, in my setup, with teamcity agents running as a server under local system, the blessed git server running gitolite with ssh-keys as access control things got a little hairier.

The problems were many, and here’s what I finally did to get it running:

* Install Git Extensions for Windows (https://code.google.com/p/gitextensions/)

* Download psexec and unzip it (http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx)

* Generate a key using “ssh-keygen -t rsa”. You’ll save yourself a world of trouble if you do NOT use a password to protect the key.

* Use puttygen to create a putty-style key.

* For sake of simplicity, assume the keys are in ~/.ssh aka c:\users\\.ssh, named id_rsa, id_rsa.pub and id_rsa.ppk.

* Create a textfile in home dir named c:\users\<USER>\git_ssh.sh. It should contain the connection script for ssh using plink

#!/bin/sh

/C/Program\ Files\ \(x86\)/GitExtensions/PuTTY/plink.exe -i /c/users/<USER>/.ssh/id_rsa.ppk "$@"

* Add a system level environment variable GIT_SSH with the content “c:\users\<USER>\git_ssh.sh” (no quotes).

* Start a cmd-shell as administrator

* Add your git server to the cache by connecting to it using plink. The command line is typically something like “c:\Program Files (x86)\GitExtensions\PuTTY\plink git@gitserver” – your default user name and server name may vary.

* Use psexec to start a command shell as the system account – “psexec -s cmd.exe”

* Add your git server to the cache of the system account – “plink git@gitserver”

* Restart the teamcity service.

So, what happened? The variable GIT_SSH, as per the git manual is the command git uses to set up the connection to remote. We replace it with plink and a reference to the key. Another option commonly used is to use pageant for this, but that is tricky (though not impossible) when it’s a service that needs the info.

Connecting to the server with plink adds the server’s finger print to the list of known hosts. That list is per user (sits in HKCU) which is why we need to do it both for the logged in user and for the local system account.

Could this be done differently? Sure, no doubt! It’s just that after spending the main part of the afternoon in blind alleys I figured I’d offer a pair of shoulders for others to stand on.

If I’d do something different I’d probably consider replacing plink with ssh and specifying the key in the /etc/ssh/ssh_config-file (http://www.manpagez.com/man/5/ssh_config/). That probably eliminates the need for both the psexec trick and setting the GIT_SSH variable. But now I’ve updated all the agents…

–Jesper

Share

Why I don’t install new components

I am no enemy of other people’s code. On the contrary. If my team can leverage the effort put in by someone else, so we can reach our goals quicker I am all for it.

It’s the installation I am against.

Any decent component pack somes with an installer. It puts some stuff in $(ProgramFiles), maybe a few examples under $(MyDocuments) and the dlls in the GAC. Fire up VS and start using the new stuff. Add new references to the GACed assemblies and code away. Looks good, compiles cleanly – great.

Now check this in and ask a co-worker to pull your work. In a real world scenario, things would break quite immediately, because the build agents do not know about what you have installed on your machine. Co-worker also gets the same compile error about missing references.

So, either you fix the build agent yourself or ask the build master to upadte it. Or them. There might be tens or more agents that all need this set of dlls. And of course your co-workers all need to spend time downloading and installing the bits.

After a while, you realize there’s a bug somewhere in the component, but the latest update fixes it. Download and install. Cater for the API changes and check in. Oh, and mail everyone they need to upgrade too, and pretty please mighty build master, could you please upgrade all the 20 agents again?

If things are done fairly rightly, the compile time bits may very well coexist in the GAC. Not so sure about the design time bits in VS. So, if you work in two projects, and one project can’t update right now, which version do you chose to use on your machine?

Better solution: Do not reference files from the GAC. Check in ALL binaries required by your application during compile time (actually you’d want to make available the entire bit set and tool chain your app wants in the process of turning text files into an ISO image ready to send to production, but that’s another chapter in the same story).

For sake of simplicity, assume we use the CVS as file repository.

Check in your fancy new component suite or function library (named <product> in the following examples) under $\xternal\<product>\<version>\<all the files>.

Add a prebuild step to copy from $\xternal\<product>\<version>\<all the files> to $\stage\xternal\<product>.

Make sure your projects reference files from $\stage\xternal\<product>.

Check in.

Note: Never ever check in things under $\stage. It is a temporary directory used for build artefacts. It’s as wise to check that in as checkin in the stupid bin\debug or obj\release directory under your project.

In case you didn’t follow, here’s what will happen on the build machine: Compile succeeds! The bits it needs are there when the compiler looks for them. On your co-workers machine: Compile succeeds, much for the same reason.

Additional benefits materialize when you decide you need to go for that upgrade. Just check in the new files under $\xternal\<product>\<new_version>. Change your prebuild step to copy from <new_version> instead of <version>. There will be no need to upgrade the project references. The references will change, yes, but to new files in the same location.

No need to waste everyone’s time by forcing them to upgrade agents or own machine. Much less risk of running into version incompatibilities, as everything is sandboxed.

If you really want to go all in on this, you should not use the VCS as file repository, but a separate file server (with proper back-up routine in place). Your prebuild step should download/copy from that external server, which will be available to all projects. Thus, you adding the <new_version> will make it available to the project down the hall, should they be interested.

 

–Jesper

Share

Foundations of a dev team

Last week I spoke to an old friend about what fundamental advice to give to a development team. There are many areas that needs catering to. I tend to start with setting up a machinery into which you put source code and out comes a nicely packaged installer. And then the rest will follow, sort of :)

Here is my list of build machine/infrastructure centered focus areas – some technical and some attitude based.

* Use a source control system – be it svn, git, p4 or something else.
* Use a build machine (TeamCity, CC.Net, Cruise, Jenkins, …)
* Make that build machine trigger a compile of everything on check-in.
* Write unit tests.
* Run the unit tests on build machine after the checkin-build.
* Run static checkers and source analysis (fxcop, stylecop, cppcheck, cpplint) on the build machine on each check-in.
* Break the build on failure.
* Make build failures obvious (lava-lamps are great!).
* Foster a culture of checking in often.
* Foster a culture of keeping build-broken times to a minimum.
* Do not assign shame to a broken build.
* Make sure whatever your build machine does can be executed on every developer machine.

The above statements leave a lot to be worked out, I know. However, no matter how you solve it you are better off if you do!

If this seems like a tall order, you’ll probably need some help. Not to worry! The internet is full of advice.

Have fun setting up your team and the machinery to support it. I do look forward to your input!

–Jesper Hogstrom

Share

Jesper Got Published: TWIM #21

I wrote a letter about what cancer is to This Week In Microbiology, and it got published in episode #21. Here’s the letter:

 

Dear Vincent,

the other day I was in a discussion about what can get cancer, something that ultimately boiled down to what cancer really is. Our reasoning went along the lines of establishing that there are organisms containing any number of cells, ranging from one and up. If I remember correctly, C. Elegans has 957 cells. Presumably there is some organism with 956, 955 and so on.

It seems it doesn’t make any sense to talk about a one celled organism developing cancer – though I am interested to have that confirmed! The nematode just mentioned has cell specialization, so it could presumably develop some form of cancer. What is the lower limit of cells an organism must have to succumb to the decease or should the question really be posed in a completely different way?

Also, some organisms of very few cells occasionally gang up and form a super-organism. This includes some slime moulds and the pre-larvae state of jelly fish. Can such “temporary” organisms develop cancer?

The question is grander than just parasites, and I have a feeling that viruses, living or not, have no propensity to develop cancer. Hence my addressing the question to TWIM.

While I have your attention, allow me to once again thank you and everyone in each of the podcast teams for your effort in sharing your knowledge and doing it in such an enjoyable tone and fashion.

All the best,

Jesper
Software architect
Sweden

The question is answered by Vincent andElio near the end of the show.

Share

Recursive msbuild

Today a co-worker asked me a really interesting thing about msbuild. He wanted to extract some information about the binary he just compiled into a textfile and add that file as a resource to the binary he just compiled. It is worth mentioning he had a really good reason to do this as well.

The good reason and the fact that it was quite an interesting question got me thinking on how to do that with minimal effort. Minimal effort is always a good sign of a lazy programmer, and lazy programmers are better. At least that’s what I read somewhere :)

First of all, it wouldn’t be possible to simply call the target Build again, as msbuild doesn’t invoke the same target twice in one run.

I figured it’d be necessary to call msbuild again, and in the recursively invoked msbuild to all the tricks.

First off, I wrote a simple program to extract some information from a binary file.

namespace GenApi
{
    using System.Reflection;

    class Program
    {
        static void Main(string[] args)
        {
            var a = Assembly.LoadFrom(args[0]);
            foreach (var type in a.GetTypes())
            {
                Console.WriteLine(type.FullName);
            }
        }
    }
}

Not overly complex, but good enough for demonstration purposes. I compiled this to genapi.exe and moved it to the root folder of the project I wanted to tamper with.

Msbuild is a quite powerful system, and it has numerous extension points. One of these is to override – or actually redeclare – a target named AfterBuild. This target is invoked after the chain of targets pertaining to build is complete. The default implementation of AfterBuild is to do nothing.

At this point I need to mention the importance of understanding the build system used. There are quite a number of problems that can and should be solved by extending and enhancing the build process. The first step on that journey to enlightenment is to start thinking about the build system as part of the product development infrastructure. It is not just some mumbo-jumbo mixed with two bottles of magic that happened to ship with .Net and Visual Studio. It is an orchestration environment of quite some power.

Anyway, start with right-clicking the project node in the solution explorer and select Unload project. Right-click the same node again and select ‘Edit…’. You’ll now get the csproj file as a textfile in your source editor in VS.

The end of the project file looks like

 <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
  <!-- To modify your build process, add your task inside one of the targets below and uncomment it.
       Other similar extension points exist, see Microsoft.Common.targets.
  <Target Name="BeforeBuild">


  </Target>
  <Target Name="AfterBuild">
  </Target>
  -->
</Project>

This is exactly what we want to do. First, to try the waters, I wrote a simple AfterBuild target just to make sure it got executed.

<Target Name="AfterBuild">
   <Error Text="Yes, AfterBuild works..."/>
</Target>

Reload the project file, and turn on more logging by selecting Tools | Options | Projects and Solutions | Build and Run and go for Normal in both the combo-boxes. View output from build by Ctrl-W, O.

Rebuild and you’ll get something like

GenerateTargetFrameworkMonikerAttribute:
Skipping target "GenerateTargetFrameworkMonikerAttribute" because all output files are up-to-date with respect to the input files.
CopyFilesToOutputDirectory:
  IncludeMe -> c:\users\jhm\documents\visual studio 2010\Projects\SelfInclusion\IncludeMe\bin\Debug\IncludeMe.exe
c:\users\jhm\documents\visual studio 2010\Projects\SelfInclusion\IncludeMe\IncludeMe.csproj(56,5): error : Yes, AfterBuild works...

Build FAILED.

Great. Build failed.

Now let’s do some serious work.

What we need is to generate the “api”. The target for this looks like

 <Target Name="GenerateApi"
          Inputs="$(OutputPath)\$(AssemblyName).exe"
          Outputs="api.txt" >

    <Exec Command="genapi.exe $(OutputPath)\$(AssemblyName).exe &gt; api.txt" />
  </Target>

The generated file, ‘api.txt’ needs to be included. We could in theory add it to the project once we have it, but imho it makes much more sense to only add it after it is generated. Thus, we need to modify the Resource item. To avoid doing that too early, this must be done in a target as well. Manipulating items and properties in targets is msbuild’s way to handle deferred evaluation.

 <Target Name="AddResource">
    <ItemGroup>
      <Resource Include="api.txt" />
    </ItemGroup>
  </Target>

Now what with the afterbuild target? There are many ways to skin a cat, and I opted for two invocations of the csproj file, first to generate the resource, and then to rebuild the project with the freshly generated file.

 <Target Name="AfterBuild">
    <MSBuild Projects="Includeme.csproj" Targets="GenerateApi"/>
    <MSBuild Projects="Includeme.csproj" Targets="Build"/>
  </Target>

However, the code above suffers from a serious mistake. When msbuild is invoked for the second time, it will surely call build, but it will also end up in the AfterBuild-target and call itself again. Infinite recursion. Not good. To prevent this from happening, we need to pass a token indicating we’re in level two, and also check that before executing anything. Improved version looks like

 <Target Name="AfterBuild">
    <MSBuild Projects="Includeme.csproj" Targets="GenerateApi"
       Condition="'$(second)' == ''"/>

    <MSBuild Projects="Includeme.csproj" Targets="Build"
       Properties="second=true" Condition="'$(second)' == ''" />

  </Target>

Much better! However, when are we calling the AddResource-target? To fix that we need to do some indirection, as msbuild doesn’t seem to commit item changes to global scope until a target is completed. Thus, instead of invoking the Build-target, we invoke a new target named SecondBuild. It in turn depends on AddResource and Build.

<Target Name="SecondBuild" DependsOnTargets="AddResource;Build"/>

The complete code that works thus looks like:

 <Target Name="AfterBuild">
    <MSBuild Projects="Includeme.csproj" Targets="GenerateApi" Condition="'$(second)' == ''"/>
    <MSBuild Projects="Includeme.csproj" Targets="SecondBuild" Properties="second=true" Condition="'$(second)' == ''" />
  </Target>
  <Target Name="SecondBuild" DependsOnTargets="AddResource;Build"/>
  <Target Name="GenerateApi"
          Inputs="$(OutputPath)\$(AssemblyName).exe"
          Outputs="api.txt" >

    <Exec Command="genapi.exe $(OutputPath)\$(AssemblyName).exe &gt; api.txt" />
  </Target>
  <Target Name="AddResource">
    <ItemGroup>
      <Resource Include="api.txt" />
    </ItemGroup>
  </Target>

Staying true to the agile programming cycle, this is where one should pause and refactor. Could the code be written any cleaner? Do we need two invocations of msbuild? As it turns out, no, not really. The same effect can be achieved with the following, more compact version:

 <Target Name="AfterBuild">
    <MSBuild Projects="Includeme.csproj" Targets="GenerateApi;AddResource;Build" Properties="second=true" Condition="'$(second)' == ''" />
  </Target>
  <Target Name="GenerateApi"
          Inputs="$(OutputPath)\$(AssemblyName).exe"
          Outputs="api.txt" >

    <Exec Command="genapi.exe $(OutputPath)\$(AssemblyName).exe &gt; api.txt" />
  </Target>
  <Target Name="AddResource">
    <ItemGroup>
      <Resource Include="api.txt" />
    </ItemGroup>
  </Target>

There you go. A simple way to include information about a binary in the binary itself. All from one Build request from inside VS.

A very interesting question and a fairly simple and clean answer.

Happy building!

–Jesper

Share

Setting up an msbuild script system

msbuild is not all that hard, but it does come with a whole slew of parameters. You don’t want to memorize them all. Thus, define a good ol’ batch file to launch msbuild:

@echo off
set msbuild20=C:\Windows\Microsoft.NET\Framework\v2.0.50727\MSBuild.exe
set msbuild35=C:\Windows\Microsoft.NET\Framework\v3.5\MSBuild.exe
set msbuild40=C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe

set msbuild=%msbuild40%

set log_quiet=/v:q
set log_minimal=/v:m
set log_normal=/v:n
set log_detailed=/v:d
set log_diag=/v:diag

set _loglevel=%log_normal%

set projectfile=build.msproj

:start
set target=
if "%1" != "" set target=/t:%1
%msbuild% %_loglevel% %target% %projectfile%
shift
if "%1" == "" goto nomoreparams
goto start

:nomoreparams
if ERRORLEVEL 1 goto error
echo Build completed. Congratulations!
goto end

:error
echo There was an error. Sorry it didn't work out.
goto end

:end

This simple little file will allow you to type things like build debug or build release without worrying exactly how to specify the target. If you want to make it extra nice, control log level with an environment variable, by adding the following line:

set _loglevel=%log_normal%
if "loglevel%" != "" set _loglevel=%loglevel%  <-- new line

simply set loglevel=/v:n from command line to change the log level.

You may also want to pass in some variables, like current directory

%msbuild% %_loglevel% %target% /p:StartDir=%CD% %projectfile%

That will give you the property $(StartDir) available inside your project file.

Speaking of which, here’s a minimal sample:

<Project
  ToolsVersion="4.0"
  xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

</Project>

Admittedly very bare bone.

Let’s add some meat:

<Project
  DefaultTargets="Showhelp"
  ToolsVersion="4.0"
  xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

 
  <PropertyGroup>
    <CR>%0A</CR>
  </PropertyGroup>
 
  <Target Name="ShowHelp">
    <Message Text="Available targets"/>
    <Message Text="* @(_availableTargets, '$(CR)* ')"/>
  </Target>
</Project>

If you follow the good practice of adding new targets to the item list _availableTargets, you have a help page showing if you pass in no parameters.

  <ItemGroup>
    <_availableTargets Include="Debug"/>
    <_availableTargets Include="Release"/>
    <_availableTargets Include="Clean"/>
  </ItemGroup>
</Project>

Obviously we’ll have to write the actual targets too, but so far we’ve seen that
* We can add things to a list.
* Display that list with a different separator than semi-colon.
* Define properties with escaped characters (by using the hexadecimal ascii values prefixed by a percent character).

It’s far from complete, but it’s a very good start!

–Jesper Hogstrom

Share

Designing a build system

When developing software it is somewhat obvious that the programmers need to be able to build the source code and run the tests on their own machines. It is slightly – but only slightly! – less obvious that the same code needs to build on a separate machine, known as a build machine. There are several purposes of that, mainly to

* Ensure everything compiles after each check-in (structural integrity)

* Execute unit tests to ensure functional integrity.

* Run various kinds of static analysis on the source or binaries.

* Prepare builds for internal (e.g. QA) and external (e.g. customers) consumption.

A build machine is typically controlled by a build orchestrator, like Cruise (now a part of Go), CruiseControl.Net, TFS or TeamCity. It may be a single machine, a system of agents or something else. Let’s call this the build machinery.

The build orchestrator is in some way connected to the version control system (VCS), be it Perforce, Git, Subversion or TFS.

When a file is updated in the VCS, the build orchestrator gets a signal and starts executing a build script, typically written in msbuild, ant/nant, make or some shell language. It is to prefer to use a language that is designed for build orchestration, thus allowing for specification of dependencies, error handling and maybe even logging.

The fundamental components of a build system are thus:

* A build script

* A build machinery

* A version control system

Luckily, the build machinery and version control system are standard products, so you don’t need to worry about rolling your own.

However, the build script is something where you want to invest some energy. The purpose of the build script is to ensure that you can reproduce a bit-identical build from bit-identical sources. It should also have the ability to execute a suite of other targets.

A target is the end result of a build script invocation, or rather, what you want the script to produce – the transformation of inputs to an output.

Some examples of targets you want your build script to produce:

* Debug - A debug build of all sources.

* Release - A release build of all sources.

* Testcases - Compilation of all unit tests.

* Documentation - Collect all documentation source documents and transform them into the desired output format.

* Installer - Collect all files that goes in to the installation file and produce it.

You can also have your build script do a number of other things, such as

* RunTests - Execute all unit tests in the test framework you use.

* RunSmokeTests - Execute all UI level tests using whatever test robot you use.

* Publish<something> – Upload some artifact to a different server for wider consumption.

* Clean<something>, Spotless – Remove output from the source tree, ensuring next build creates everything from source documents.

An important aspect of a build script is that it should be possible to execute both on the build machiner/build agents and the developer machines!  You should be able to “make debug” or “build documentation” or “want testcases” from a command line prompt on all relevant machines. Otherwise it is very hard to perform an action on a developer machine to ensure the check in will pass. It also helps a great deal when developing the build script.

There are some really critical consequences of the above:  The build system must be self-contained! This means everything required during compile time must be reachable by the build machinery – and if it isn’t checked in you need some other way to pin which version (of a tool, SDK or third party tool) you want. Speaking of which, you’ll also need to version control the build scripts themselves.

So, to summarize what you need to get started:

* A build machinery.

* A version control system.

* A build script that automates everything that happens between coding and preparing a testable artifact.

* Everything required during compile time and test time reachable by the build script (either checked in or reachable on another file system).

The build script and repository of prerequisites require by far the most design – or rather – those two items are by far the least described out there.

I’ve designed build systems using various build machineries (CruiseControl.Net, TeamCity, TFS) using both make and msbuild. Nowadays I write msbuild scripts building .Net and C++ projects for a teamcity environment. However, there are a number of concepts that do not change. I’ve found the most important to be:

* Do not rely on what is installed on the machine you’re building on. Everything must be controlled!

* Designate an output/staging directory, and make sure everything that is produced ends up in under that directory.

* Define a structure of all projects and stick to it.

* The build should behave the same way if building from inside an IDE as when building from command line.

Everything must be controlled

Developers typically install something that goes into a build, add some references to globally available (GACed in the .Net world) DLLs and everything works just fine. Until the build happens on a different machine, when things blow up, as the global state of that machine isn’t the same as on the dev machine.

It is imperative that everything you use during compile time is checked in! This means not only your third party components, but also the unit testing framework and the tools used in the build chain, such as help compilers, installer creators and so on.

Often it is quite simple to meet this requirement. At other times it is harder, like when COM-servers need to be registered, registry entries must be found and other evil global conditions that must be fulfilled. As the build master, your job is to understand how you can script things to make all machines look like that have enough of the global state to fool the tools and components you use.

If you adhere to this rule adding a new build agent, or replacing a developer machine is very much simpler, as fewer things need to be installed.

I draw the line at the .Net level – all involved machined are allowed to have .Net installed. It would be possible to write scripts that didn’t rely on the presence of a .Net installation, but I find it not worth it at this point. If we were targeting strictly different .Net versions I might reconsider.

Staging directory

Visual Studio adds a bin and obj directory under each project. Not nice. You’d rather have all output end up under stage\debug or stage\release or stage\installer. Keeping everything under a stage directory reduce the risk of someone checking in binaries they’ve just built, and makes cleaning the tree significantly easier. Instead of traversing the world, you just rmdir stage\debug. Or rmdir stage if you really want to make things spotless.

Achieving this is not incredibly difficult, as all compilers tend to have a flag indicating where they should deliver the output. The trick is finding the sweet spot that minimize the amount of configuration required for each new project.

Project Structure

Project structure makes it easier to find your way around projects, reduce the learning time when moving to a new team and makes it much easier to write a build script that works across the board, as everything looks quite the same. I don’t know what structure makes sense to you, but something like

-$+
  +-Build
  +-Common
  +-Documentation
  +-Source
  +-Stage
  +-Tests
  +-Xternals

is a good start. It is also fairly obvious what goes where. Try to keep the root directory rather clean!

Behave the same from command line and IDE

This is trickier than it sounds. Assume you set up a wrapper msbuild script that calls msbuild to build your solution file. You play nice and pass in whatever properties you need to the second level of msbuild and everything works as expected. From command line.

From inside Visual Studio however, you don’t get the parameters passed in from your script, as VS is now the root caller. Your fellow developers will get a different result than you and the build agents. There are likely several ways to accomplish this, and they all involve some manipulating the project files (csproj, vcxproj). If you are using some other IDE there are surely other ways.

That just about covers the high level aspects of a build system. Admittedly there are details left to cover, but they lack the generality I wanted for this post. The above should work for most circumstances and most platforms.

–Jesper Hogstrom

Share
←Older