Message Passing

After a week of hiatus from SICP I opened up the book to another blatantly simple yet profoundly important concept: Message Passing.

Essentially the authors talk about passing around "intelligent data objects" that can dispatch other procedures based on the operation name. The implementation in LISP is simply passing a procedure around that accepts the name of the required operation and through a switch calls the appropriate behavior implementation.

This technique is called "Message Passing" and seems eerily familiar to the way methods are dispatched in OO.

Read for yourself by following the link and scrolling to the bottom:

http://mitpress.mit.edu/sicp/full-text/book/book-Z-H-17.html#%_sec_2.4.3

Abstraction: The Sunshine of your Code

As part of a "back to basics" kick I'm currently reading Structure and Interpretation of Computer Programs. This book is a gem and although I'm only through the first two chapters I've been fascinated several times by fundamental and obvious programming concepts that I seem to have overlooked as my career progressed beyond those carefree University days. I'm particularly absorbed in the author's consideration of abstraction.

Abstraction is something I have personally taken for granted even in University. Recalling lectures where profs used analogies of the abstraction of cars or TVs I approached the topic with the blasé "Ya, ya, ya we get it... lets get to the code!" attitude. Now seven years later this book has caused me to reflect upon abstraction and realize that it is the fundamental ingredient of nearly every advancement in the construction of software.

The core idea is in creating Abstraction Barriers to speak a different language in each layer. In doing so, we can work most efficiently with the software itself and more importantly with other humans (of which most are computer language illiterate). To steal the authors' example at the highest level we can all speak about rational numbers, create them, add, subtract and multiply them. Anyone can "program" a calculator to make rational numbers dance. But there's a special class of people we try to avoid at parties who know that a rational number is comprised of a numerator and a denominator. Further there are the computer scientists who would look at a rational number as some kind of pairing of two integers. Now consider that each computer scientist may choose to implement a rational number a different way. Does this change the way the mathemetician's work with numerators and denominators? Does it change the way the average person does math? Absolutely not. But it may be the reason us computer science nerds don't even get invited to those parties.

In essence Abstraction Barriers are the driving force beneath Design by Contract and Interface Driven Design. Object Oriented languages are based on abstraction as there is no inheritance or polymorphism with out it. Even the often abused principle of encapsulation is about protecting the abstraction barrier. Abstraction is the foundation of every OO design pattern ever invented as a design pattern is just summarizing algorithms into a vocabulary. Even the cornucopia of patterns that emerged from Fowler's PoEAA and Evan's Domain Driven Design are again just abstraction barriers applied.

Consider, for example, software engineering tenants such those found in the Domain Driven Design camp which are some of the most powerful tools introduced to software engineering. Layered architectures and Bounded Contexts compartmentalize subsets of Ubiquitous Language into abstraction layers. Each layer or bounded context has a particular language used throughout. For example, Persistence Ignorance is keeping the implementation of physically storing data out of the realm of those who speak about domain and into the realm of those who speak about databases. Knowing the power of abstraction its possible to take the fundamental concept and apply it to your own design techniques.

Quite a while ago I learned the exercise of anthropomorphising software components to help break down software design. I've just recently started to consider abstraction barriers as part of the excercise with great success. For instance, I'm always running into the nagging question of "Where does this particular piece of code belong?". Now, while playing the part of each object in the orchestration I pay attention to the language being used (that is the verbs and nouns of each component's vocabulary) along with the data in play and try to find the lines between abstraction layers. It becomes quite obvious when you've crossed a barrier as the language will change and finding where a piece of code belongs is a simple as identifying components that have the same vocabulary.

For example, on my current project we have a 3D control that works in terms of Geometries. We also have components that can be drawn in this control such as Pipes, Elbows and Tees. The 3D control works entirely with Geometries knowing nothing about Pipes, Elbows or Tees. Conversely Pipes, Elbows and Tees know nothing about Geometries. Here we have two abstraction layers, lets call them the 3D layer and the Domain layer. If I find myself with a piece of code that does something with Geometries its fairly obvious that it should be grouped with the 3D layer. Or more likely I'm tempted to throw some code that works specifically with Pipes on the 3D control. During an intense anthropomorphise excersise this would raise a red flag as the 3D layer doesn't include Pipes in it's vocabulary. I should relocate that code to a service in the Domain layer so the language is consistent. I could then inject the service into the 3D control so it can call on the service to achieve its goal.

As you can see abstraction is a powerful tool for software engineers, and definitely not a concept to be overlooked. So for you kids out there, don't be like me. Treat abstraction with the respect it deserves. For those out in the industry who have forgotten about it whether you read SCIP or not give abstraction a second thought.

If you're interested you can also watch the videos at http://groups.csail.mit.edu/mac/classes/6.001/abelson-sussman-lectures/. They're worth a viewing if at least for a chuckle at the stylish '80s stashes.

Using Feature Branches

One of the biggest changes we've made to our development process in the past month was to introduce feature branches. Historically our team ran with two branches: the trunk and a production maintenance branch. All new development occurred on the trunk while bug fixes were committed to the maintenance branch and merged down to the trunk. This was a somewhat successful approach in that it's simplicity was perfect for a team that had little experience with version control but it suffered one annoying problem: the trunk was largely unstable.

The first indication that we had a problem was that deployment became a timely event. We would wait until there was low risk that the trunk had bugs in it, close our eyes, pray to our respective gods and slam our fists on the big red deploy button. Sure enough there'd be a partially completed or untested feature in the package that would cause problems down the line.

This led to Code Freezes and the Scheduled Commits anti-patterns. Where around deploy time we'd have to postpone the risky commits in favor of the solid bug fixes. From a lean perspective this is a huge waste as there was code just waiting to be committed to the repository and a developer surfing Facebook waiting for Cruise Control to scream out "Woo hoo!".

So, we decided to try using feature branches for new development. A feature branch is simply a branch that has all of the changes from the trunk plus all of the changes required to implement a particular feature. When the feature has been tested and is ready to be released we merge the branch back to the trunk and delete it. The end result is that the trunk stays stable even while new development is occurring because the feature branches are the ones assuming the instability.

Remember: Branches are like puppies so don't create a branch unless you're prepared to take care of it.

The maintenance costs of a branch are that every day or two a developer has to merge all of the changes from the trunk up to the branch. With Subversion 1.5 this is simple task but you should be aware that it could result in a couple of problems. First, you will likely run into situations where a single file changes on both the trunk and the branch resulting in a conflict. These conflicts are no different than the conflicts you receive while doing an svn update and can be resolved as such. Another more severe problem is that changes can be lost if a file is removed on the trunk but changed on the branch. Unfortunately there is no easy way to detect this other than to scan your merge log for deleted files. I have not yet personally experienced this problem but I am hoping that if it does occur my test fixture will catch it. Finally, performing a large merge is much harder than small incremental merges, so merge often! Depending on your team size and the amount of change on the trunk you'll want to tweak your merge frequency. My team is currently merging every day or every other day depending on the state of the trunk. Plus we always make sure to merge the trunk to each feature branch after another feature branch has been merged down to the trunk.

Remember: Merging works with a range of revisions so when merging from the trunk to the branch you need to include ALL revisions since the last merge not just the HEAD revision.

Subversion mergeinfo tracking in SVN version 1.5 makes feature branches a breeze. I remember using feature branches under early SVN builds (and even CVS ) where we would have to track which versions were brought up to the trunk on an index card so we wouldn't duplicate efforts. Now the mergeinfo is tracked inside of SVN properties and SVN is smart enough to handle these tedious tasks for us. So just make sure your working directory is clean and from it run:

svn merge svn://svnserver/myrepository/myproject/trunk

Sometimes, however, you will want to know which revisions have been applied. In this case you can run the following command from your branch working directory to get a list of revisions merged up to the branch:

svn mergeinfo svn://svnserver/myrepository/myproject/trunk

Similarly you may want to know which trunk revisions are eligible to be merged onto the branch. In this case you can run:

svn mergeinfo svn://svnserver/myrepository/myproject/trunk --show-revs eligible

Remember: A small incremental merge is far easier than a massive merge, so merge often! Don't wait until the end of the feature to sync up your branch. You WILL feel pain.

The final task in feature branch maintenance is after the feature has been developed you will need to merge it down to the trunk. This is a step in our Kanban that occurs after Peer Testing but before UAT. In other words we are committing zero defects (as far as we know) to the trunk. Since our branch already contains all the changes from the trunk that occurred during the development of the feature the only changes that need to be merged down are the ones that belong to new feature. The beauty of keeping your branch in sync is that this happens natually in SVN 1.5 by running the merge command with the reintegrate flag from a clean trunk working directory:

svn merge --reintegrate svn://svnserver/myrepository/myproject/branch

Once the branch has been merged down to the trunk you will want to make sure that no more change occurs on it. This is accomplished by simply deleting the branch. It will remain in your repository so you can always browse back to find those changes but developers will not be able to check it out and commit against it.

Thats about all the mechanics there is to feature branches. But the question remains how do we know when we need to create a new feature branch? This is more of a fuzzy area. Some teams branch for every Story but the granularity of these branches is really dependant on what and how often you will be releasing. In general my team will create a feature branch based off a portion of a Stories in a Theme or Epic that should be released together. We call these chunks Minimally Markettable Features and we choose each one so it takes about an iteration to complete. So our feature branches deliver a useful feature to the business every 2 weeks.

The early results show that feature branches have greatly improved our ability to deliver stable releases and in turn make our client much happier. Our testing efforts are far more efficient since UAT now contains stable changes and there is no waste in developer cycles due to Code Freezes or Scheduled Commits. The downside is that feature branches require much more SVN savvy. To successfully implement feature branches your development team should be fairly experienced with version control systems. If your team is struggling to manage a trunk and a single maintenance branch then I would recommend mastering that before using feature branches.

If, however, you feel your team is ready for the next step and would like to implement feature branches I highly recommend reading the branching topics in the Subversion Book. Also get your hands dirty and play around with branches in a sandbox svn repository. You'll quickly get a feel for how they work and what they're all about. Lastly, drop me a line if you're have any questions.

Becoming Lean

I've spent the past three years building up an internal software development team. This team was built with a foundation of principles instilled by XP-styled Agile (or as they were ingrained in me by ThoughtWorks). The team has changed dramatically over the three years, ever increasing in size, skill and agility. We are currently running with what I would consider to be a dream team. Each team member is a skilled and experienced generalist and everyone on the team either has 2+ years of agile experience or is an apt and enthusiastic participant in their own learning process. I couldn't be more excited about our future prospects and what we will deliver to the business in the coming months.

But even fielding this roster we experience some Agile anti-patterns that turned into pains for the team. An analysis of these anti-patterns is leading us towards a Lean Kanban style and away from the traditional Agile. I know there is not much out there about making this transition so I will try to document as much as I can to help others who are attempting the same thing.

The first of our pains was that we could not get a stable velocity. It was not uncommon for us to have a huge variance between iterations... embarrasingly so.

Iterations always seemed to result in hangover and the perpetual work in progress made testing very difficult. Not to mention cutting releases was difficult when they contained half completed work.

When we switching into maintenenance mode between phases everything went out the window. We moved into chaos mode and during this time there was some team churn (for the better) that changed the entire dynamics of the team so any historical metrics were rendered useless. Not to mention our rhythm, and I believe the inertia behind a strong rhythm is important to a dev team.

Finally, while our development team is Agile the business is hunkered down in a Waterfall mindset. They fear fast and cyclical releases opting for several big-bang releases per year. Admittedly we haven't done much to quell those fears since the work in progress problem plagues our own testing process. But even if those fears were gone the business has a gated approval process better suited for the space shuttle than it is for software.

We tried to band-aide these pains. For example we tried to fix the varying velocity by accounting for half-completed stories (Story A is %65 complete) to give a more realistic velocity. We injected empty buffer iterations to allow for work in progress completion and stabilization. We tried to estimate bugs and maintenenance work but unfortunately metrics taken during this span were used to estimate the schedule dates for future development work. As for the waterfall business problem there was unfortunately nothing we could do to reduce the release cycles. Perhaps the real problem is that changing the business is beyond the team's sphere of influence at this point and we have to earn the business' trust before that can be fixed.

After looking at these problems from a root-cause perspective we have decided to focus on the following principles:

1. Reduce work in progress
   
2. Zero defect mentality
3. Kaizen

We've also decided to switch to an iterationless Kanban system because it more closely represents the way we are working. In a subsequent post I will go through these principles and describe how we expect them to work in concert with Kanban to improve our process.

To be continued...

ClickOnce in General

I've had many emails lately about my ClickOnce deployment using NAnt post so I thought I'd post a little bit about how ClickOnce works in general.

First, there are two files necessary to deploy a ClickOnce app: the deployment manifest and the application manifest. The deployment manifest contains version specific information about your app. This is the file that makes version 1.2 different from 1.3. You should have one deployment manifest for every version you deploy.

The application manifest, however, is really the main entry point into ClickOnce. It contains information about which version is the most recent (eg. 1.3) and redirects the client to the appropriate deployment manifest. You will have just a single application manifest for your project. It is the URL of the application manifest you should give to your users so they can open the application.

The tricky thing about ClickOnce manifests is that in addition to containing a list of files necessary to run an application they contain a hash of each file. This hash is like a digital signature and used to prevent ClickOnce from delivering malicious files. So, any time you release a new version (or change your ClickOnce payload) you must do the following (using Mage):

1. Create a new deployment manifest (for the new version)
2. Sign the deployment manifest (to regenerate the hash of each file)
3. Update your application mainfest (so it points to the new deployment manifest from 1)
4. Sign your application manifest

Once that is done you're pretty much good to go :-)

Also, for further reading my colleague Jonas Avellana has a great post on setting up ClickOnce to deploy to multiple environments.

Binsor Method Interceptor Macro

So, I just picked up a story that is one of those really annoying feature requests-- you know the kind: adding log statements to analyze performance of about 40 key use cases scattered all over the app. Its the kind of story every developer dreads and the worst part is that it is throw away code. In a week when data has been collected there will be another story coming down the pipe to revert those changes. While I could've spent the day hunting down the 40 culprits and performing mind numbing cut and paste code injection I decided instead to attempt an implementation that would stick around and have potential uses in other scenarios.

Since our application uses Windsor entirely configured through Ayende's Binsor scripts I figured I'd take a stab at writing my own binsor macro for interceptors. The functionality I required was that only specific methods of a service need to be logged for time and occasionally other contextual information dumped.

I know I could perform the same action by creating a class that implements IInterceptor and registering it in Binsor like:
IoC.Container.Kernel.GetHandler('textProvider').ComponentModel.Interceptors.Add(InterceptorReference(MyInterceptor))

But I wanted something more flexible. Ultimately I want to be able to craft a custom binsor config file and not have to perform any code changes. This way when the story comes down the pipe to revert changes it is as simple as replacing a config file. Additionally having the ability to throw interceptors on specific methods is an enticing feature should we ever need to peer down the business end of the app running in production.

The syntax I was after looked something like this:

component 'textProvider', ITextProvider, HelloWorldProvider:
intercept 'GetText':
Log.Info('Invoking: ' + invocation.Method.Name + ' at ' + DateTime.Now())
stopwatch.Start()
invocation.Proceed()
Log.Info('Duration: ' + stopwatch.Stop())

Make sense? If not, I'll paraphrase in English: I want to intercept the GetText method on my ITextProvider service and perform the block of code that logs the invocation, lets the invocation proceed and then logs the duration of the invocation. Essentially my goal is to inject that block of code as a delegate into a class implementing IInterceptor. Its actually easier than it sounds!

The first step was to create the MethodInterceptor. This is the class that implements IInterceptor:

public class MethodInterceptor : IInterceptor

{

public delegate void InterceptHandler(IInvocation invocation);

private readonly string methodName;

private readonly InterceptHandler interceptHandler;

public MethodInterceptor(string methodName, InterceptHandler interceptHandler)

{

this.methodName = methodName;

this.interceptHandler = interceptHandler;

}

public void Intercept(IInvocation invocation)

{

if(methodName.Equals(invocation.Method.Name))

interceptHandler(invocation);

else

invocation.Proceed();

}

}

Notice the delegate? Thats the magic... otherwise, its pretty straight forward.

Step 2 is to create something that will do the Windsor configuration for us. This class needs to:

• Add a new component for the interceptor
• Add the dependency values to the interceptor component's configuration
• Register the interceptor component as an interceptor on the parent component
  

If you are unclear about interceptors in Windsor I suggest reading up on them in the Windsor documentation.

public class MethodInterceptExtension : IComponentExtension

{

private const string INTERCEPTOR = "interceptor";

private const string INTERCEPTORS = "interceptors";

private readonly MethodInterceptor.InterceptHandler interceptHandler;

private readonly string methodName;

public MethodInterceptExtension(string methodName, MethodInterceptor.InterceptHandler interceptHandler)

{

this.methodName = methodName;

this.interceptHandler = interceptHandler;

}

public void Apply(Component component)

{

var interceptorKey = component.Name + methodName + "Interceptor";

AbstractConfigurationRunner.IoC.Container.Kernel.AddComponent(interceptorKey, typeof (MethodInterceptor));

AbstractConfigurationRunner.IoC.Container.Kernel.GetHandler(interceptorKey).AddCustomDependencyValue("methodName", methodName);

AbstractConfigurationRunner.IoC.Container.Kernel.GetHandler(interceptorKey).AddCustomDependencyValue("interceptHandler", interceptHandler);

AddInterceptorConfigItem(FindOrCreateInterceptorsConfigItem(component), interceptorKey);

}

private static IConfiguration FindOrCreateInterceptorsConfigItem(Component component)

{

foreach (IConfiguration configChild in component.Configuration.Children)

{

if (INTERCEPTORS.Equals(configChild.Name))

return configChild;

}

var interceptorsConfig = new MutableConfiguration(INTERCEPTORS);

component.Configuration.Children.Add(interceptorsConfig);

return interceptorsConfig;

}

private static void AddInterceptorConfigItem(IConfiguration parent, string interceptorKey)

{

parent.Children.Add(new MutableConfiguration(INTERCEPTOR, "${" + interceptorKey + "}"));

}

}

Finally we need to create the actual Macro. Binsor has a very handy base class called BaseBinsorExtensionMacro that I will inherit from so most of the heavy lifting is done for me

public class InterceptMacro : BaseBinsorExtensionMacro<MethodInterceptExtension>

{

public InterceptMacro() : base("intercept", false, new[] {"component"})

{

}

protected override bool ExpandExtension(ref MethodInvocationExpression extension, MacroStatement macro, MacroStatement parent, ref Statement expansion)

{

extension.Arguments.Add(macro.Arguments[0]);

extension.Arguments.Add(BuildInterceptHandler(macro));

return true;

}

private static Expression BuildInterceptHandler(MacroStatement macro)

{

var e = new BlockExpression();

e.Parameters.Add(new ParameterDeclaration("invocation", new SimpleTypeReference("Castle.Core.Interceptor.IInvocation")));

e.Body = macro.Block;

return e;

}

}

In order to understand this class you have to know that behind the scenes what happens is that by default a new instance of a MethodInterceptExtension is created and injected into the Component being registered (ITextProvider in my example). Unfortunately by default Binsor does not know how many or what parameters to pass into the constructor. So we want to modify the call to construct the MethodInterceptExtension so that it takes in the method name that we want to intercept along with a delegate that actually does the intercepting.

The InterceptorMacro just takes the first argument passed to it as the method name we want to intercept. Now for some magic: the macro wraps it's body block in a delegate. Not to be outdone it then modifies the call to construct a new MethodInterceptExtension to accept two new parameters (method name and delegate). Now the MethodInterceptExtension will be constructed with the method name and delegate injected into it. Cool huh??

These three classes are pretty straight forward and the additional functionality in Binsor is incredibly useful. Plus rather than spending a day writing boring log statements I learned heaps about Binsor and Windsor. Tomorrow I will actually use this and craft up a binsor container configuration file in record time...

Top 10 Jonasims From the LinqToYourBrain Talk

Awwya fool... Jonas Avellana (aka DJ LeDude) brought both his coding speedos and .Net skills to the edmug meeting this evening. Jonas delivered a couple of hours of Linq, SQL and .Net 3.5 nyahhhhh with a flair that's all his own. For those who missed it, here are the top 10 Jonasisms of the evening:

10. Awwww ya!!!
9. Thats Frickin' Cool
8. Thats Frickin' Awesome
7. You can do it like that if you're deepcore
6. var crapDataContext = new DragAndDropCrapDataContext(); [typed not vocalized]
5. First, I'm gonna show you the ghetto way...
4. ... now we'll do some Linq Kung Foo
3. Well, you know we don't have many chicks in the west end so the woman's prison is pretty good
2. Aghhhh my eyes!!!
1. I don't know how it works but its pretty awesome

A little bit of a let down was that he didn't find a way to work bescumber into the presentation but he made up for it by delivering an excellent Linq packed presentation. Thanks Jonas! It takes a lot of time and effort to do a presentation like that and myself and the rest of the .Net community really appreciate it. Now you can get back to ghost riding the whip.

You can read more from Jonas on his blog: http://blog.ezed.ca