Soapbox Rants and Raves

I have been experimenting with Hudson for several weeks now and I like the tool much better than Cruise Control. Not to say it does not have its issues, it generally works very really.  I have mostly had issues getting reports to show up in the project. For example, it appears that on a new project, jUnit test results will not show up until after the first completely successfully loop; kind of strange. One issue that is really frustrating me, the Testability Explorer plug-in;  I just can’t seem to get the report integrated, all of the other plug-ins have worked as expected.

There is one big difference between Cruise Control and Hudson, which I have not exactly come to grips with yet: the handling of build errors. Hudson introduces the the concept of Job Stability. Traditionally, I relied on Ant to determine the success or failure of the build, but with Hudson, it is also possible to determine the stability or health. Most plug-ins allow you to provide custom ranges to indicate the job’s stability, based on the number of errors or warnings generated by the component. On most of my projects, we did not allow any PMD or Checkstyle errors, so job stability was never much of an issue. However, there are many projects that allow Checkstyle, PMD, and even jUnit errors; this is where the stability metric can be really helpful. This would also be useful when introducing a new tool into a project and it takes time to cleanup the issues.

I found a very well written article for setting up a Hudson server.  If you are really ambitious, the article walks you through setting up a Tomcat or JBoss installation.  I originally setup Hudson using Tomcat, but most of my experimental work, I have used the built-in Servlet Container; it is a little easier and makes the frequent updates as simple as a mouse click.

My last rave about Continuous Integration (CI)… I happened to find a really interesting document this week, and I just had to share it… The main point that I wanted to stress is the value of CI. The study was from a very small sample, but the results where in line with my personal views. The document is a very good comparison of twelve different tools and provides a good background on why one would use CI. It is easy to read and contains lots of interesting graphs.

The document is a study of Continuous Integration tools by Fontys University of Applied Sciences in Venlo, Netherlands. This study showed that Cruise Control is still the leader in this space; probably due t o the fact that the Cruise Control has been around for almost 10 years. Hudson, which has only been around for 2 years, appears to be closing in fast on the Cruise Control market share. I found another website that shows Hudson with a significant portion of the CI market. To my surprise, there appears to only be a couple of real contenders, all of the other tools are not well represented.

I also wanted to throw in this last graph to reinforce the dominance of Subversion as a version control system. It has been interesting to watch how fast the open-source community moved away from CVS over the last couple of years. I think it will be very interesting to follow the growth of Git. It has received a lot of press over the past year and seems to create a religious debate when compared with other tools. Just what we need! I was shocked to see the significant projects using Git: Ruby on Rails, Android, Fedora, X.org, just to name a few.

I embraced continuous integration many years ago for purely selfish reasons, but it did not take long to see the real value of the process.  I happened to work on a team where developers continually forgot to check in all of their work. Being an early morning person, I would be the first one to sync out  the previous days changes; at least once a week, my work area would be unbuildable because of missing files. I would end up wasting time mocking up an object or figuring out how to work around the problem.

Enter Cruise Control. I originally setup the tool for notification purposes, cruise would monitor the CVS repository for changes and kick off the build script after each check in. If the build failed, anyone who made changes since the last successful build will be notified of the failure via email. To ensure that the developers would fix the problem, we enabled the spam feature. Cruise would continue to attempt building the project until it was successful. This could result in numerous email messages per hour, and was usually a pretty good motivator to fix the problem. We actually had to institute a higher level of encouragement on some projects, the “Bagel Rule”. If you were responsible for breaking the build for over 24 hours, you had to bring in bagels for the team. This rule worked wonders!

Once the developers got used to the process, we started to crank up the expectations. We could use Cruise to improve the quality of our code, enabling tools like Checkstyle and PMD.  If the code did not conform to our standards or was overly complex, the build was marked as a failure.  These tools essentially automated a basic code review  process.  They did not guarantee a good design, but did a pretty good job at ensuring that the code was well structured and maintainable.  The most valuable Cruise integration was the addition of jUnit and Clover.  By executing the jUnit suite after each check in, helped us discover any unintended sided effects induced by the changes.  By combining the execution of the jUnit suite with code coverage data (we happened to use Clover), gave us a quantitative value of the confidence that we could place on our code.  If we had low code coverage, it was impossible to have high confidence in our code’s correctness, or even in our ability to refactor the code and ensure that that it still worked. Conversely, if we had a decent quality jUnit suite and a high code coverage percentage, then we felt pretty good about the correctness o the code base and had complete confidence in our ability to refactor.

Cruise Control has been around for quite a few years and is now a very mature tool. This is one of those areas that I have not recently researched, but was surprised by how many options are now available to development teams. Cruise Control is fairly easy to setup and does exactly what it needs to do, from an continuous integration perspective. However, it does not provide that same amount of information and value that other tools appear to provide. From a commercial product perspective, I would really like to evaluate the Bamboo tool from Atlassian. I have been throughly impressed with the Atlassian product suite. I think they really understand what developers need and how they work.  My favorite open source product is Hudson. It seems to have some pretty big supporters, such as JBoss QA (Check out the link, pretty cool).  What makes Hudson interesting is the plug-in infrastructure and the trending aspects. It is very easy to integrate and publish a variety of metrics in the Hudson dashboard, something that Cruise Control does not do very well at all.  The dashboard gives you a good feeling of the projects health and stability. Hudson keeps a history of metrics, and displays how the project has evolved over time. Pretty cool, if you are interesting in continuously improving your code base; it gives the developers a dynamic picture of their effort.  Configurability is probably one of Hudson’s greatest strengths, more on that later.

What are your experiences?  Just like unit testing, I would not want to develop without my continuous integration safety net!