Mar 28 2011

Continuous Testing Plug-ins verses Continuous Integration

Category: Continuous Integration,Eclipse,TestingPhil @ 12:01 am

This is an extremely interesting concept, but I really wonder how it works in the real world. Toy projects are one thing, but real production size applications are a different story! The principle is very simple; just modify a piece of code in your project and the IDE automatically executes all of the unit tests. Pretty cool concept. But, what if your unit test suite runs for several hours? I believe this is an all too common problem. I have seen many projects dedicate time to building their unit tests, and over time, end up with test suites that can run for multiple hours. There is a very negative side effect associated with this scenario, it can change a developer’s attitude towards checking in code, it actually discourages it. If the developer knows their commits will not be validated for several hours, they tend to check in very infrequently and only when they feel it will not impact other developers. Many factors can contribute to this scenario. The primary factor is that many of these tests are not really unit tests; they typically do not provide the proper isolation (via mocking) and take too long to execute. I’m not going to address those issues directly, but the concept does support and encourage the idea of quick test execution.

Wouldn’t it be cool to run all of the unit tests, in your IDE before checking in the your changes? Obviously, this action is expected by the developer and is completely possible without any special tools. However, it is not automatic and requires extra effort by the developer. Usually, it is easier to let the Continuous Integration server run the tests and fix the fallout later. It seems like a simple problem to solve; create a plug-in that automatically executes all of the unit tests saving a file, requiring no additional effort or interaction from the developer. If you are an Eclipse user, you have two good options, JUnit Max and Infinitest. I would really like to try Kent Beck’s JUnit Max, but you need to purchase a subscription to use (and even try)  the plug-in. That is a show stopper for me, as there are free alternatives such as Infinitest. I evaluated the Infinitest plug-in last year, after it was open-sourced. I did encounter some problems with the plug-in, my Spring-based projects did not work; I was quickly discouraged and abandoned the tool. Months later, I was still intrigued by the idea, so I gave it a try last week on a different project and achieved success!

Unfortunately, Infinitest is not currently installable from the Eclipse Marketplace. It can be quickly installed the old fashioned way from the Eclipse update site.  There is only one setting you may want to tweak, the Slow Test Warning threshold. If  you have unit tests that execute for longer than half a second, Infinitest will add a warning to your Problems View for each one; this could be problematic if you have a large number of long running unit tests. You should also notice a huge new “Continuous Test” status bar in your Eclipse footer. It will update automatically based on your coding activity. JUnit Max is a little more subtle, but provides the same type of feedback mechanism. That’s it, you are now ready to write code and auto test!

I’m not exactly sure how intelligent these plug-ins are, related to determining which unit tests to execute based on what code was changed. Hopefully that is the next level of continuous testing evolution. However, if you modify a single unit test, Infinitest will only run that single test; beyond that, I don’t believe there is much intelligence. If your code modification causes any unit tests to fail, the status bar will turn red. Unfortunately, it does not tell you how many test failed, it only provides the number of tests which it executed. If you mouse over the status bar, it will display the names of the tests which failed, but you can not click on them, they are just there for feedback. You will need to look in the Problems View to discover which test broke, the view will now contain markers for each failed test.

The only issue I have with the plugin, is around user interaction. It does not provide its own View for test results, it simply adds the failures to the existing items in the Problems View. I assumed that the Unit Test View would be updated with the test results, similar to manually running a unit test; however, this was not the case.  It appears that both Infinitest and JUnit Max use the same strategy, providing minimal feedback to the developer. Maybe this a good approach, but I was expecting more visual feedback and confirmation. I associate static problems with the Problems View, such as compiler and code quality errors, not dynamic problems such as test failures. The developer should be able to click on a marker (problem) and be navigated to the exact line of code indicated by that marker, giving them the opportunity to resolve the issue .  Here is my problem with this approach, if the developer clicks on the failed JUnit marker, where will they be navigated? The only possible location is the unit test which failed. Generally speaking, this is not where the problem would be found, it would lie in the code that was just changed. Given this issue, I would rather use a different View for continuously executed tests;   maybe even better, find a way to reuse the existing Unit Test View.

This does bring up an interesting question, do these plug-ins minimize or eliminate the need for Continuous Integration? I don’t believe this is the case, but would seem to eliminate the need for some of the more advanced features the vendors are implementing. One such example is TeamCity’s remote build, pre-commit test feature. There are many factors to consider, but if unit tests execute quickly, I do not see the need for this functionality to be provided by the CI server. If the tests execute for multiple hours, we could be adding more risk into the process, by not committing changes when they are ready to integrate. Remote build, pre-commit testing sounds valuable on paper, but I am struggling to see the real benefits. I tend to think this functionality over complicates the Continuous Integration process. One of the main benefits of Continuous Integration is that it is seamless and automatic. Nothing is required by the development team to reap the CI benefits, simply check in your code early and often.  As more education and process overhead are required by the CI process, it is no longer simple nor free.  The focus should be on doing what makes sense, at the right time, in the right place. These IDE plug-ins seem to have correct perspective, pre-check-in on the developers desk, simple, automatic, and efficient.

I hope that you will try to integrate one of these plug-ins into your personal development process. I believe they will help create a more robust development process, creating fewer Continuous Integration failures and ultimately, a more efficient, faster executing test suite. If you have real world experience with one of these tools, please provide some comments. I am still curious, does this approach work on real projects?

https://www.beilers.com/wp-content/plugins/sociofluid/images/digg_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/reddit_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/dzone_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/stumbleupon_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/delicious_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blinklist_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blogmarks_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/google_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/facebook_48.png


Mar 03 2011

TDD is too hard…

Category: Software Development,TestingPhil @ 2:11 pm
Earlier this year, I participated in a meeting to recommend testing best practices. Being fairly passionate about TDD, and working in an environment where unit testing is fundamentally an afterthought, I assumed the group would certainly recommended TDD as a best practice. Amazingly, TDD failed to get the official endorsement. Many of the committee members stated that TDD was too hard and required too much up front work. Too hard? Really? I was very surprised by their rational, especially since many of them were seasoned developers. The image to the right has nothing to do with software development, but I love the message! I thought there was a rather subtle correlation to TDD: Thinking first can prevent problems. Think First is probably a little misleading, surely, everyone claims to think before embarking on a task. For me, the real value behind TDD is the thought process; I like to connect thinking with the design process, not the coding process
I had the opportunity to participate in a SD Times webinar on Wednesday called “Leaders of Agile: Practical Agile with Test-Driven Development” by Kent Beck and Mark Seemann.  I took some notes on points I thought were relevant to truly adopting TDD.

 

Concept Observation
Write a little test, write a little code, iterate.
– Don’t spend hours writing code
– TDD tells you where you stand
I have worked with a lot of developers that literally spent hours writing code without ever executing it. Once they feel like they are done coding, they start to think about testing. Many times, this testing is done in the same manner as their development, which I will describe as “without a plan”.  Some even avoid structured, assertion-based testing frameworks, they simply spin up the container and execute the code. This approach will certainly work, but would you really say this is the most effective approach? You end up with a huge pile of un-validated, un-executed code.  Unless you are perfect and never make mistakes, you have no clue how many bugs are buried in the code. Even worse, if you discover a design problem or a more elegant approach, how much code will you have to change or toss away? This is not to say that refactoring or eliminating code is bad. The point is, this effort could have been minimized or even prevented by thinking first. Another benefit of starting from the test perspective, is you always know where you stand in the process; the test works or it does not. If the test does not work, you are probably not done.
TDD allows you to defer design decisions
– Refactoring and Retrofitting
– Working test first, Design as you go
If you think about TDD from an iterative perspective, you are basically working from the outside in. You can almost compare it to the Object Oriented principle of encapsulation, hiding or postponing design decisions.  Unit test should be focused on the behavior of the class, not the actual implementation. Because the unit test is focused on behavior, you have the opportunity to defer some of the design/implementation decisions to a later iteration. This later iteration might only be a couple of hours later, but, by thinking first about the desired behavior and interaction, you are not required to commit to an actual design or implementation.  Obviously refactoring can be a big part of TDD and is hopefully not considered a deterrent. Refactoring does have a cost; I believe this cost is justified and minimized by the confidence provided by a good unit test suite and the long term supportability of a elegant design.
TDD = API Design For me, TDD is more about the design process than the actual unit test. TDD forces me to concentrate on how the class will be utilized, enabling me to focus on the requirements of the class, implementing no more functionality than is actually necessary. Implementing the class first, you are basically guessing at the API, only imagining how the class could be used. Adding and subtracting methods as you go, based on how you feel the class might be invoked. I have to believe that this approach will generally not result in the cleanest API. Focusing on the external interaction and dependencies provides a simpler and more exact view of the actual implementation requirements.
Tests are First Class Citizens. This one is very near and dear to my heart. I can’t tell you how many times developers have pushed back on the enforcement of coding standards and quality as it pertains to the unit test suite. Why would anyone want to allow or encourage bad coding practices in the unit test code, that are not allowed in the real code? Do developers have a quality switch that can be turned on and off? Writing good, clean code over here, and sloppy code over there? It should be all about consistency and supportability. Unit tests need to be as understandable and maintainable as the mainline code; they will be maintained as long as the mainline code and over time, could provide more value (comprehension) than the mainline code.
Testable = Loosely Coupled Testability is another topic I find very interesting. I just love the Testability Explorer tool, but was never able to get others excited about this concept. If not designed effectively, unit tests have the potential to become a roadblock of change. Unfortunately, the same point can be made about the actual code; there are certain designs and implementation that can impede future change. The goal is to use patterns and tools that support change, such as dependency injection. Some people think that TDD can encourage coding without thinking; I have the complete opposite view, thinking first, to establish the design. It is my opinion that TDD is just one small tool that helps flush out designs; doing so without significant overhead and simultaneously building a safety net.

 

I was not sure how these “virtual conferences” actually worked, but this one was very well organized and seemed to go off perfectly. I was very impressed with both of the speakers and recommend subscribing to their blogs. They each made numerous points, but I wanted to share a point from each of them that I found encouraging.  I will probably paraphrase them poorly, but hopefully you get the point!

 

Speaker Thoughts…
Kent Beck – “Everything is hard. You need to be eager and willing to learn.” I thought it was refreshing to hear the words, “eager and willing”.  Unfortunately, we are not always eager and willing! You can’t force anyone to learn or change. There has to be a willingness; we need to be open-minded to absorb and process new thoughts. The benefits are not always immediate and may cause some pain, but in the end, learning is what makes us better.
Mark Seemann – “I tend to do a lot of thinking.” I thought this was great! I never hear people say they do a lot of thinking. Many of us tend to jump into the fire, probably because we are not given sufficient time to evaluate at the situation. My challenge to you, demand the time, take the time. Think First!
https://www.beilers.com/wp-content/plugins/sociofluid/images/digg_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/reddit_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/dzone_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/stumbleupon_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/delicious_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blinklist_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blogmarks_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/google_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/facebook_48.png


Jul 11 2010

Unfriendly Developer Pactices – Adding uncessary environmental complexities

A couple of years ago, I began focusing on construction and deployment complexities. I had just joined a new team and began looking for the commonalities between their architectures and environments. Needless to say, they were all different. The most disturbing issue was the amount of effort required for a new developer to start working with an application. If there was any project documentation, it was probably out of date; resulting in hours or even days of trial and error to actually build and deploy the system. In this post, I’m talking about is building and deploying J2EE applications. The following list represents the top developer unfriendly practices I have observed.

Developer Unfriendly Practices
  1. Without first running a setup task in Ant, the project does not build in Eclipse.
  2. The Eclipse .project and .classpath files are not checked into version control system with the source.
  3. The project’s Ant scripts are generic and non-deterministic; they have to be edited for your individual environment before building the project.
  4. The project requires specific libraries or packages to be installed on your machine before it can be built.
  5. Eclipse project not configured as an WTP project.
  6. Minimal jUnits and no continuous integration.
  7. J2EE container dependencies built into the project.

After experiencing these challenges, I began a quest of environmental simplification. I understand there are numerous reasons why projects end up with these unfriendly characteristics,  but leaving them unaddressed was just not in my character! I made it my personal mission to ensure that each project that I worked on, I would try to leave it in a little better shape than when I arrived. This is a never ending activity; I hope that those that come after me will have a similar philosophy and continue my quest. It is amazing how easy and fast an application can atrophy, eliminating all of the positive changes that had been previously applied. To keep things simple, I came up with following three project requirements. I try to weave some aspect of them into the development process and architecture of each project that I work on.

Project Principles
  1. Self-Containment, No External Configuration.
  2. Environmental Awareness
  3. Change Resilient

I hope that most of these principles seem like common sense and are nothing new. Much like many of the XP principals, they are not new or revolutionary, just good reminders of often overlooked practices. I will try to elaborate on each principal in a future post, hopefully, in the near future!

https://www.beilers.com/wp-content/plugins/sociofluid/images/digg_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/reddit_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/dzone_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/stumbleupon_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/delicious_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blinklist_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blogmarks_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/google_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/facebook_48.png


Apr 25 2010

New Agile…

Category: Software Development,TestingPhil @ 5:50 pm

I might be dating myself, but when I was a kid, I remember my parents talking about New Math. I don’t believe that I was ever exposed to New Math, it might not have made it to the small towns in northwestern Ohio! The New Math Wiki was actually very interesting; taking us back to the cold war with Russia, attempting to drive up the American scientific and mathematical skills. For better or worse, New Math was not able to change the status quot and was abandoned.

Not really the point of this blog, but New Math actually stressed the mathematical concept of set theory. This is actually a concept that one of my mentors actually spoke about quite often. Something that I suggest everyone do a little reading on, especially if you are not familiar with the concept.

So, why New Agile?  A friend happened to snap a picture from a white board at his place of employment, illustrating their new 14 week Agile process. Oh, where to begin…. I going to skip the obvious 14 week concept and tie this back to one of my previous posts on testing. I asked the question: Should testing be the limiting factor in the software development process? I am not exactly sure what numbers were used per cycle, as it seemed like it was a discussion point, but it does highlight an interesting trend.
I threw together this little a table to illustrate my point. Based on either combination of values, the project was going to allocate between 57% and 64% of it’s project schedule to testing. Using the worst case numbers, we are talking about 3 times longer to test the code, than actually develop the code. Is it just me or is that crazy? What if you had two (2) months worth of development? Simple math, that would require six (6) additional months to validate!  I realize there are or can be several factors contributing to this equation, but I’m pretty sure that we are not talking about a team of twelve (12) developers to one (1) tester, creating an unbalanced capacity situation.  Just for fun, I threw a Theory column into my table. I can’t remember where I learned or read this, but I’m sure everyone knows this fact as well… the cost of fixing bugs. We should be spending the vast majority of our software development resources on upfront planning; this would include requirements, analysis and design. It would seem that if developers knew exactly what there building and testers knew exactly what they were testing, both teams could do a much higher quality job, probably in less time!

Unfortunately, we seem to be spending too much of our time in the wrong SDLC phase and on the wrong artifacts. I can’t tell you how many times I have seen a hundred (100) page requirements document for a seemingly trivial system. The SDLC mandates that these documents be produced, irregardless of the content’s quality. This is no fault of the business, they simply try to document their needs, given what they know. Because requirements are typically not allowed to evolve and are seldom interpreted correctly by the IT team, the business never gets what they really need. Additionally, the time spent by the testing teams to provide the traceability back to the excessively verbose requirements, while generating and documenting their test cases could almost be considered SDLC overhead. The testers are rarely given a chance to actually automate their test cases, which in my opinion, is the most valuable artifact of their job.

If we could produce smaller, more concise, valuable requirement documents, spend more time doing actual analysis and design work, we could actually take steps toward my Theory column. This approach could and should enable the later phases of the SDLC to be completed with less effort, fewer issues, and in less time. So two questions:

  1. Is this not common sense?
  2. Why is this approached not embraced by technology organizations?  The benefits seem unquestionable…
https://www.beilers.com/wp-content/plugins/sociofluid/images/digg_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/reddit_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/dzone_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/stumbleupon_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/delicious_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blinklist_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blogmarks_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/google_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/facebook_48.png


Apr 20 2010

Making up your own methodology… WetAgile or W-agile

I was reading the Software Development Times last week and found an interesting article about development methodologies. I almost think Agile has turned into a “techno-fad”; a meaningless buzzword, right up there next to architect! Nobody wants to admit they are using the waterfall approach; they would have be considered an technical dinosaur; just look at the history highlights below!  Unfortunately, many projects seem to choose Agile for the wrong reason; there seems to be some kind of coolness, prestige, or promise of advancement associated with using Agile, rather than a true embracement of the methodology.

Can you believe…

  • The waterfall model was first documented in 1970 by Winston Royce.
  • The spiral model was defined by Barry Boehm in 1986 article “A Spiral Model of Software Development and Enhancement”.
  • The Extreme Programming model was introduced by Kent Beck in 1999. I recently suggested following Kent here
  • Interesting fact, Ken Schwaber, one of the SCRUM creators stated that 75% of organizations using SCRUM do not get the benefits they had hoped for using this approach.

So, everyone claims to be “doing Agile”; I think reality tells a slightly different story. I recently observed a group define the allowable methodology options for new projects; they chose Waterfall, Iterative, and Agile. Waterfall and Agile are pretty to definable, but Iterative seemed a little vague to me. After reading this article, I think I actually figured out what it was, WetAgile. Per the author, WetAgile is a term used to describer Waterfall shops that either aspire to be Agile or are in the transition process.
I actually found two versions of the document by the author, Steve Pieczko. The SD Time version seemed to be at a little higher level, while the Enterprise Management Quarterly version has a little more substance.

The real question is why do teams only implement so few aspects of Agile? The author suggest that we typically implement the pieces of Agile that we most understand or believe the Waterfall teams can easily embrace. It seems like the XP techniques are much easier for the development team to implement, specifically because they control their own space; whereas embracing the complete Agile (SCRUM) management process is a little more involved. I believe the problem is that too many people have to be involved to make it work. Throwing a bunch of people in a room and having a daily stand-up meeting does not exactly imply you are doing “Agile”. I find it disappointing that the most valuable part of SCRUM (my opinion), the creation and management of the backlog, is rarely  done. Probably because project management is still very “Waterfall” centric; you start with some requirements and end with a date, nice and simple. Additionally, it is not very easy to build a backlog when requirements, including defects and enhancements, are managed by processes that have nothing to do with the software development life cycle.

Here is the root of the problem, most projects (applications) are managed as series of individual releases, where each release has a concise beginning and end. Each release is a discrete and isolated project. I don’t think applications are ever viewed as perpetual, ongoing, evolving entities.  The subtle difference is that enhancements and defects are usually managed independently and are rarely viewed and prioritized together. The backlog, if it exists is virtual. The business seems to “divine” their most relevant needs out of thing air, ask for some LOEs, and then define the next release based on what can be implemented by the some magical date. That is a far cry of creating, evolving, and managing a complete list of features that ultimately should be built in a application, using a iterative release stream.

I think another significant problem is the perceived cost of creating a release. How great would it be to deliver new functionality to your customer every month? Unfortunately, with today’s audit and traceability requirements, the actual cost of creating a release can outweigh the value. Too many meetings, too many sign-offs, just too much overhead.; Not to bash the testing community, but it seems that the effort required to test and certify a release is also cost prohibitive. There is no possible way that testing teams could manage monthly release cycles using their traditional approaches. Should testing actually be the limiting factor in the software development process? What is really driving up the cost? What is burning up all of the time? Something I will have to research! I would hope that testing could be accomplished just like development, as developers attempt to do test driven development, testers should be following the exact same process. The continuous integration process should be running their suite, same as the developers suite. Would this not be sweet? The next question is, do we really need “User Acceptance Testing”? If the users are truly involved with the SDLC process and all testing is automated and quantifiable, is that not good enough? We seem to add multiple levels of testing, choosing an “accumulative testing strategy”, rather than a single, cohesive plan.

I found an interesting post on Agile Coaching, called “Agile or W-agile“. I think the blog is kind of unique, a series of drawings done on note cards and white boards… actually a pretty good way to get out a message!

Extra Credit Reading…
This article got me thinking: what is the next advancement in software development methodologies? It seems like eXtreme Programming and Agile have been around for many, many years; even though, maybe not widely adopted. I did a quick Google and found an interesting post titled: After Agile, What’s next? There are some interesting questions and several good links.
https://www.beilers.com/wp-content/plugins/sociofluid/images/digg_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/reddit_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/dzone_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/stumbleupon_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/delicious_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blinklist_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blogmarks_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/google_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/facebook_48.png


Apr 01 2010

Who should write unit tests?

Category: Software Development,TestingPhil @ 12:01 am

I have been a big fan of Kent Beck for many years, mostly from all of his work on jUnit and extreme programming.  I have two recommendations for you:

  1. Follow him on Twitter. I cannot tell you how many thought provoking tweets he has provided me, I’m sure that I get at least one good nugget each day!
  2. Read his personal blog at Three Rivers Institute.  It is a really good source of design and development information. Typically, not a quick read, but well worth the investment.

Earlier this month, Kent tweeted about the “Pragmatic Magazine”. I had been to the “Pragmatic Bookshelf” many times in the past, but did not know about their magazine. This turned out to be complete and valuable educational exercise for me.

Tangent…
You can download the “magazine” in multiple formats: PDF, ePub,  or mobi. Since I do all of my reading on my cell phone (Motorola Droid), I downloaded the Aldiko ebook reader. Within a couple of minutes, I was able to open up the magazine on my phone, too cool! My poor Kindle is so neglected; I should try to download the mobi format and see how the reading experience compares. I’m sure it will be much nicer on the Kindle DX, but I always have my phone with me, so the Droid wins!

I was really surprised by the featured articles. I gravitated the to the JSON article; I’m a huge JSON fan, but see it totally ignored in my professional world… I have not figured why this is true, but the article did provide an interesting perspective. One of the other articles that grabbed my attention was on Testing. You have to read it, it is very interesting…

Quoted from the article…
The developers who understand the importance of unit tests and who write comprehensive and subtle tests are exactly the group that need them the least. The “average” or inexperienced developer probably doesn’t  believe in unit tests and will write bad or pointless tests if forced to do so.

So, the question is: Who should write your unit tests? This answer to this question really emphasizes something that I always try to stress to project teams: Do not test your implementation,  test the behavior, the contract, you need to support. So many times, especially if testing is not second nature to the developer and the team uses a code coverage tool, there is a very bad tendency to write tests which are specific to the implementation. To make matter’s worse, some teams do in-package testing of their Java code to exercise and validate the implementation details. I was so fired up on this point last year, so I wrote a blog on it! Needless to say, I think this is a really bad idea and creates unnecessary and undesirable coupling between your unit test and implementation.

The author goes on to suggest that your best developers should only create the interfaces and associated unit tests, let the other developers create the implementation behind the interface. Pretty cool idea. Unfortunately,  I don’t see this taking off in the corporate world. In my experience, most developers tend to write their own unit tests or unit testing is completely outsourced to some third party (way after the fact). Additionally, the “first tier” developers are expected to write the hard code, and that is what they really want to do anyway; the “second tier” developers usually get the leftovers, like unit test cleanup and shell scripts.

I think the whole value system is skewed in the typical corporate software development process. Where should you put the good people to get the most value?  In the past, if you did not cut it as a developer, you would get promoted to the CM or testing team.  Many times, these teams were not valued by the organization, and hence not staffed with top talent. I would prefer the best and brightest developers in the testing space. No matter how good the code is, if the testers are sub-par, then the application is destined to fail. Conversely, if you have marginal developers and a strong test team, I think you have at least a chance of success (providing you have a good design).

I read another interesting article about bug tracking and system testing.  Think about this… rather than creating a problem ticket for issues found in system test, why not create some type of unit / integration test to demonstrate the problem?  The test get checked in by the test team and breaks the continuous integration build process. Now, the development staff just has to fix the problem. Seems kind of streamlined and efficient; I guess it will never work! But I do like the concept!

https://www.beilers.com/wp-content/plugins/sociofluid/images/digg_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/reddit_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/dzone_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/stumbleupon_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/delicious_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blinklist_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blogmarks_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/google_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/facebook_48.png


Mar 30 2010

Unit Testing Not Required…

I assume that everyone performs some kind of unit testing… The real question is what does unit testing mean to you?  For me, it is some form of repeatable, assertion-based approach that can be re-executed by other developers and a continuous integration process. Unit testing is just one piece in my software confidence puzzle. That sounds like a great topic for my next blog! Unfortunately, some developers think that generating some log messages to inspect or stepping through the code with a debugger accomplishes the very same thing. Depending on the competency of the developer, I believe this can be partially true; however what about the next developer who needs to change the code?

Next, you need to place some kind of value on your unit test strategy. Is testing an investment, an expense, or a liability?  I think that unit testing typically starts as some type of organizational mandate. It quickly turns into an expense and ultimately a liability. I have seen very few projects truly embrace and believe in the value of unit testing. It seems that more often than not, a true believer (or zealot) establishes the process and everything goes along smoothly. After a while, the “pressure of the release date” eliminates the need for creating new unit tests; or even worse, the existing unit test are no longer maintained. They become brittle, broken and ignored. As the number of failed unit test grows, is seems very hard to justify the cost of fixing them. At this point, you have a liability problem; the existing failures get in the way of identifying and managing new failures. Who will really notice the the one new failure when there are already fifty other failures? If ignoring unit test failures is allowed by the continuous integration process, the typical developer will have no incentive to fix them. We have now created an environment where the correctness of the unit test only matters at the point when the code is checked in. That statement might not even be true, as there is no guarantee or reason for the developer to ensure that their new unit test executed properly in the continuous integration environment. Chances are, that after some period of time,  that new test will decay and be added to the every growing, ignored list of broken unit tests.

Wow, I’m painting a pretty bleak picture of unit testing! To make things even more discouraging, I recently lost the debate on the merits of unit testing with one of my mentors. I think he is one of the smartest and most practical architect/developer/person that I have ever worked with, yet he saw little to no value in unit testing. To this day, I still cannot completely follow his logic; surely there are other, external factors that could minimize the value of unit testing, such as team competency, cohesiveness, maturity, and the general environment, but I don’t believe unit testing can (or should be) completely eliminated.

Sad, but true…
On a previous project, one of my teammates said that he would not create any unit tests, unless the project manager added additional line items to the project plan, specifically for unit testing his code. I was amazed… Why was this not considered a standard development procedure? It is pretty easy to see why one could argue that unit testing is doomed.

Fortunately, I have worked on a few projects where we considered unit testing an investment and an asset. These projects mandated zero unit test failures and even required some level of code coverage. The teams never considered or required the creation of a unit test to be called out on the project plan; it was simply part of their job. More than once, these unit test suites gave us the confidence to make the right change (re-factor), rather than the typical Band-Aid Software Methodology typically employed to address problems. Sounds like another good blog topic!

Final thoughts…
Here is another good read, Design to Test; old, but still valid today! There are so many good points in this article, they deserve to be reiterated again and again. If you are familiar with the Spring Framework, they should really resonate with you…
  • Test the interface, not the implementation
  • Composition over inheritance
  • Singleton avoidance

Here was one more blog, “Tips for Testing“, that reinforces the increased ROI of unit testing over time. You should at least skim his tips, he seems to have an interesting perspective…

https://www.beilers.com/wp-content/plugins/sociofluid/images/digg_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/reddit_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/dzone_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/stumbleupon_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/delicious_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blinklist_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blogmarks_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/google_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/facebook_48.png


Nov 09 2009

NetBeans Unit Test Creation better than Eclipse? And where should unit test live?

Category: Eclipse,Java,TestingPhil @ 5:00 am

I seem to work on a variety of Java applications and find that unit testing is one of the most varied (implementation-wise) pieces of the development process. These applications, created by different development staffs, many which have evolved over the pass few years are all very unique… kind of makes me wonder why? That is a topic for another post!

One debate I always seem to encounter is around where to actually save the unit tests? Should we do in-package testing or rely on the public API. My preferences are pretty simple:

  • Unit test should not be stored with the code, but rather created under a secondary source root. On my early projects, I created a sub-package under each package, called test, to manage the relevant files (tests and data). I later realized that I could simplify the Ant build process and enable in-package testing by creating a separate source tree. I have seen projects commingle the actual source code, unit tests, and even the test data files, all within the single source tree. This defaults the testing strategy to in-package testing, and discourages (or even prevents) API testing. I find this approach rather messy and unclear. This approach also complicates the Ant build process; at some point during the process, the unit tests, supporting classes and files should be separated from the actual deployable content.
  • I like to name my secondary source root tdd. I know it is not widely practiced, but my hope is that if the developers see that TDD directory every time the open up their IDE, the concept might actually wear off! Maybe someday, one developer (hopefully more) will actually be challenged to write their unit tests first. With Eclipse and jUnit 4.x annotations, I seem to always start with my unit test, and sometimes even refactor code from the unit test into the actual class; kind of a high-bread TDD process, but the thought is always there!
  • I also prefer the public API testing strategy verses in-package testing. This line of thinking always takes me back to the Testability Explorer. The concepts behind this metric enforce the idea public API testing and is worth a quick read. Add the Spring Framework to the mix, enabling dependency injection and I see little need for in-package testing. This is typically accomplished by creating a “test” package as the root of the secondary source tree, with sub-packages reflecting the package hierarchy of your classes to be tested. I truly believe that defaulting to in-package testing allows the developers to be very sloppy and even unaware that they are testing a specific internal implementation, rather than externalized behavior presented by the API… This is VERY bad practice in my opinion.
  • If there is a need for in-package testing, then the sub-package structure of the classes to be tested can be recreated under the secondary source tree, maintaining a clean separation between the source and test cases. These test should be considered an exception to the norm, rather than common practice. I would hope for a 90-10 or 80-20 ratio, with a majority of the test falling under the test package root (in-package is the minority).

I am kind of tied to Eclipse as my IDE, but do play with NetBeans every so often. I think I could switch to NetBeans, my only real requirement is that the IDE must have Emacs key bindings; some habits are just too hard to break! NetBeans looks like an pretty good tool and seems to be very responsive on my Ubuntu box; I especially like the way it manages plug-ins.  One interesting thing that NetBeans does (maybe a little better than Eclipse), is manage unit testing. NetBeans will automatically create the secondary source tree, but seems to default to in-package testing. It is very easy to add the additional test package into the package structure to enable API testing. I also like the way that NetBeans separates the Test Libraries from the regular Libraries. I’m not sure how well this would work when you use a tool like Ivy, but does make it more obvious, as to which libraries are used for execution verses unit testing… If you do happen to generate a unit test from an existing class, NetBeans will generate more code than Eclipse. I’m not sure how useful this code is, but it does try to create the object under test, invoke the get() methods and perform assertions on the returned values. Might be more noise that it is worth, and anyway, you should be writing the unit test first!

The bottom line is that Unit testing should be easy and valuable. If it becomes too hard or complicated, it might be time to re-address how the unit test strategy is being implemented.

https://www.beilers.com/wp-content/plugins/sociofluid/images/digg_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/reddit_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/dzone_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/stumbleupon_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/delicious_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blinklist_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blogmarks_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/google_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/facebook_48.png


Nov 08 2009

Eclipse – Ubuntu 9.1 Button Issue…

Category: Eclipse,TestingPhil @ 5:18 pm
This was making me crazy the other night… I did not spend too much time on it, thinking is was just me. But, after upgrading my laptop and encountering the exact same problem, I knew there was more too it. It seemed like only the cancel buttons actually worked; all of the next and finish buttons did absolutely nothing, other then beep!  I soon realized that I could use the tab key and press enter to move to the next screen; but that did not make me very happy.

A quick Google search turned up several people with the same exact problem. It was kind of interesting. The Ubuntu developers did not want to take responsibility for the problem and said it was an Eclipse issue. The Eclipse community said they never had to do anything special for Eclipse to run on Ubuntu; it was looking like a stand off! I did find the official bug report filed with Launchpad.net, it says the problem is resolved. However, without overriding the following variable, Eclipse is not very usable.

export GDK_NATIVE_WINDOWS=1

It also appears that this problem is affecting many other tools on the new 9.1 release as well. This simple fix will get you back into action, without having to wait for the real fix.

https://www.beilers.com/wp-content/plugins/sociofluid/images/digg_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/reddit_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/dzone_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/stumbleupon_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/delicious_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blinklist_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blogmarks_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/google_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/facebook_48.png


Oct 19 2009

Spring Framework Anti-Patterns

Category: Java,Software Development,TestingPhil @ 4:40 pm

Not that I’m a big fan of the whole “Anti-Pattern” concept, but there is some subtle irony in the title of this post; some of my coworkers will actually understand. Sorry about the mystery for the rest of you, but I think this post is really relevant for those trying to use Spring for the first time. After building my first Spring-based application, I was on the fence concerning Spring’s actual value. Obviously, I made lots of mistakes in my first implementation; I think many of the following anti-patterns can typically happen on your first Spring project; my goal is to make you aware of these pitfalls, such that you can skip past the mistakes made by myself and others.

Interesting Thought Tangent?
While looking for the title to my post, I have found a little snippet that calling “Spring a direct attack on J2EE“. I thought this was a unique way of describing the Spring Framework. For sometime now, I have been on the bandwagon to simplify the development practice through Spring (Spring Core, Security and Remoting). I never really thought about Spring as another religious battle front; such as the classic emacs/vi or mac/pc wars. In essence, I was picking a side: No more J2EE! I guess it was not obvious to me before today, as I really can’t escape the J2EE container; I’m are required to deploy in a J2EE container, even though I make no use of its capabilities; a Servlet container is all most of us really need! Just an interesting thought! Agree or Not?

So where to begin, I guess the first question is: Why do you want to use Spring? What service(s) will it provide for you? I think many developers simply wanted to slam it on their resume! Let’s start with the the most obvious use of Spring, Dependency Injection (WIKI, Fowler). I think this is the hardest thing for most developers to actually understand! If you think you are just writing XML code to create your instances rather than writing Java code, you have already missed the boat. I really believe that most developers have not actually taken advantage of DI in their applications.  They just use Spring to create an alternative static (hard-wired) bean creation strategies. Just for fun, I thought I would highlight some anti-patterns I have seen used before and hopefully never again!

Create a Factory.
Factories are so 1990. This is one hurdle that some developers have a tough time understanding. They continue to think about creating objects via factories. There is NO need to ever create a factory when using Spring. Spring is essentially the a super factory, plus it is actually responsible for the life-cycle of those vended objects. There are scenarios where you need to specifically create objects that Spring manages, in a loop for example; but these are very uncommon. The proper way to solve this problem, is to create a bean that is ApplicationContextAware.  When this bean is created (by Spring itself), Spring will set the application context on this bean. Now, you have the ability to ask for any bean that Spring is managing.
Ask Spring for objects.
This is probably the toughest habit to break and is easiest pattern to get stuck in. This is especially true if you are trying to integrate Spring into an existing project and is your first attempt at Inversion of Control. You will have a much better chance understanding these constructs, if you can start from a clean slate (or at least create a bunch of samples). Spring done right, is truly is a different way of thinking. For example, when working with a context-aware Servlet, you fundamentally NEVER need to ask Spring for a single instance; this is almost a perfect example of IOC and DI.  The whole point of DI is the INJECTION. It is not about ASKING for an instance, it is about being GIVEN an instance;  throw in the IOC principle and your bean will be invoked at the RIGHT time to provide the appropriate behavior.
Domain Object DI.
This was an unusual one for me. I recently observed a project that used Spring to created multiple static Singleton factories to vend out a variety of instances, each factory with it’s own Spring context file. Needless to say, this application was far from easy to understand. To my surprise, there was even a factory specifically implemented for creating the business domain objects. Why this was done is still a mystery to me and seems to violate the true spirit of Spring. Why would I ever want to call DomainFactory.getNewWidget() or DomainFactory.getNewGadget()? The configuration had the domain beans defined in a separate Spring context file and oddly, used static methods to initialize specific properties of these new bean instances. There was no way which enabled the XML configuration to be overridden, allowing for different implementations. Worse yet, you were tied to the initialization behavior of the factory methods! Depending on your definition of a domain object, I see very little reason domain object dependency injection.
Use Ant to create Spring Configurations.
This is an interesting use of Ant in a Spring-based application. The idea here is to generate the Spring configuration and properties file from an Ant task, to specifically configure the context files to each deployment environment. They values are essentially baked into the application and unchangeable.  It is bad enough to substitute property file values, but some developers actually substitute bean name references. This might be a little hard to follow, but the Spring XML configuration would define two different implementations of the same basic function, each with its own name. When the application is built, the Ant tasks would substitute ref=”bean1” for some environments and ref=”bean2” for others; everywhere that implementation was referenced in the configuration files. Call me crazy, but I don’t think this what the Spring creators ever intended!

As I have developed a better understanding of Spring and how to take advantage of it features, we have evolved our Spring applications to a state of “environmental awareness”. Using this solution, all of the context and property files are packaged into the application, for all known environments at build time. This enables us to build the deployment image (jar/war file) one time, using simple Ant tasks, no more templates, no more substitutions, no complexity. At execution time, Spring is now intelligent and can determine it’s execution environment; we can use specific property files for specific environments, as well as use alternate implementations (via DI) for each environment. This is where DI really pays off, allowing us to use the appropriate objects, not just properties, but objects for each environment! No code changes, no crazy Ant tasks, you build once and deploy anywhere. I think this is actually pretty slick and what Spring (core functionality) is all about.

I do have to put in a plug in for testability. Taking advantage of jUnit 4.x annotations and Spring integration, makes unit testing a snap. Using the @Resouce annotation to inject everything (instances) you need into your unit test is amazingly elegant; unit testing could not be made any easier. This approach allows us to have highly controlled, customizable execution environments; we can seamlessly manage unique configurations for local development, continuous integration, as well as any actual deployment environment almost magically.

https://www.beilers.com/wp-content/plugins/sociofluid/images/digg_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/reddit_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/dzone_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/stumbleupon_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/delicious_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blinklist_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/blogmarks_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/google_48.png https://www.beilers.com/wp-content/plugins/sociofluid/images/facebook_48.png


Next Page »