Test Driven Development

Terry wrote a great post about "pragmatic" programming. It brings up the usefulness of test driven development(TDD). It is good that this is debated. It seems like it is almost accepted as fact that TDD is good. Period. Whenever people start accepting something without questioning, it leads to very bad things.

So I am glad that Terry questions TDD. Many folks I work with question it, too. Their arguments are different than Terry's. For example they point out that much what you code in a test has to be duplicated by QA. Of course that's not an issue for some folks, because they don't have QA, but it is an issue in larger organizations. Terry uses the example of Facebook as a large organization that uses no hard rules, i.e. TDD, for development. Also, folks I work with point out that TDD is a recurring tax. You not only spend time coding the test to start with, but you spend more time re-writing the tests every time you make changes to your code. The cost of TDD must be weighed against its benefits, you should not just accept it blindly.

To me, the value of tests greatly depends on the clients of your code. Let's say you have code whose only client is the end user, i.e. "application" code. Generally what matters for that code is that it is accepted by the end user. If it does what they want, that is all that matters as that is the only purpose for the code. I can definitely see how writing tests may not have as much value for this kind of code, though I think there is still some value to it.

Now compare this to code whose clients are other pieces of code. To make it simple imagine application code that calls infrastructure code. In this case it's harder to argue that if the end users of the application are happy, then the infrastructure code is good. Imagine if the infrastructure code has an API that is supposed to produce a formatted date. The infrastructure code could produce the wrong formatting, but the application code could work-around this to keep the end user happy. Maybe that is ok, but things get uglier when somebody "fixes" the infrastructure code, particularly if a second application uses the same infrastructure code. Having fine-grained tests for the infrastructure code becomes more valuable.

For me personally, I like using TDD, though I am not religious about it. There are times that I write a test first, then I write the code. There are other times I write the code, then a test, run the test, tweak the test, fix the code, etc. In general, I like to write tests just because I like to test my code to be sure that it works. I figure if I am going to go through the effort of doing that anyways, I might as well do it in an organized way that could be useful to others. Also, writing test code is at least as useful as writing inline documentation in your code, maybe more useful. A unit test is sample code, or darn close to it.

Fantasy Baseball: ESPN vs. Yahoo!

I've played fantasy sports for more than ten years now. A lot of that has been on ESPN. I've spent a lot of money of fantasy teams. Two years ago I started playing a little on Yahoo Sports as well. Their service is free, but it was clearly not as good as ESPN's.

What's interesting though is that this year there seemed to be some pretty fierce competition between ESPN and Yahoo. ESPN started making their fantasy baseball free for the firs time. They reduced the features and eliminated prizes. They kept around more complicated leagues with prizes, but of course they had a price.

Yahoo continued to expand their features. They how had live scoring, something you had to pay for in the past (and something ESPN has had for several years.) They've also added more bling to their interface, with drag-and-drop rosters.

But now something kind of funny has happened. Well at least it's funny to a programmer like me it's funny. ESPN clearly was unprepared for the popularity brought on by making fantasy baseball free. They had simplified the free version of things, but kept most of the features from the $20-30 version. They've had so many problems. On the first day of the season, there were frozen rosters, wrong scores, broken waivers. They've sent out some emails apologizing for the many problems:

To all of our ESPN Fantasy Baseball players,

We wanted to be sure you had the latest update on what is happening with ESPN Fantasy Baseball, and what we're in the midst of doing to get things back on track for you.

We have every resource, including our full technical team as well as additional specialized technical and quality assurance personnel, working to resolve these issues.

Based on extensive ongoing work and testing, we expect to have the problems affecting Fantasy Baseball resolved by Wednesday. It is possible that between now and then testing could reveal complications that would push the resolution to later in the week. However, we are confident in targeting Wednesday and are taking every step during that time to ensure that the solutions we provide for you will be complete and long-lasting.

While it may sound simple, the core problem lies in issues with transaction information being processed incorrectly and at incorrect times - which presents complexities as data progressively compounds.

Here's the latest on what is being done:

We have made significant progress finding and working towards a fix for the core problems responsible for roster irregularities and incorrect waiver, free agent and other transactions. Due to the nature of software, thorough testing must be performed to ensure that the solutions we implement will fully resolve the issues and preserve the long-term integrity of the game. To do that, we have created parallels of ESPN Fantasy Baseball leagues in a development and quality assurance environment -- simulating the live game activity. In this environment we have been, and continue to run extensive testing of potential fixes for the core problems. There are approximately 70 different variations of our game currently in use, and we are simulating all of these. In parallel, we have programmers and technicians troubleshooting problems that have created scoring and standings irregularities - isolating and testing potential solutions.

Please know -- it is not just rhetoric when we say we continue to work non-stop to resolve this. We owe you the fastest, most effective and stable solution - and that is our sole focus.

We will continue to communicate with you as there are further developments, and you will hear from us again no later than Wednesday at 3 p.m. ET.

We apologize again for the frustration you've experienced and appreciate your continued patience.

You are our number one priority.

John Kosner, senior vice president and general manager, ESPN.com and the entire ESPN.com Fantasy team

Ouch! I like how they've now created "parallels of ESPN Fantasy Baseball leagues in a development and quality assurance environment" and "There are approximately 70 different variations of our game currently in use." Sounds like there are configurations that they didn't test -- at all. They didn't have unit tests for it or functional tests by QA. They've resorted to cloning live data to do their testing against. That's really not good!

I had thought that simply weren't prepared for the scale, but it sounds like their problems are deeper. Talk about an embarrassing situation for their engineering group.

Oh, and for what's it's worth -- I have two ESPN teams, and one Yahoo team. They're all doing crummy so far. My hitting has stunk, though my pitching has been very good. I'm not worrying though. I figure my guys will start hitting. Hopefully ESPN will have their problems worked out by then.

Code Coverage Insanity

I saw this article on OnJava about so called Statement, Branch, and Path Coverage. The gist was that even if your code coverage was good, there could be different paths of execution that aren't being tested. This is obviously true, and I would think most programmers would roll their eyes and say "duh" to this.

The author offers up the Statement and Branch coverages as ways to analyze such a problem, but ultimately dismisses these methods as inadequate. He then offers up Path Coverage. In his section on Path Coverage he brings up two points:

Fortunately, you can use a metric called cyclomatic complexity to reduce the number of paths you need to test.

Ah, the magic phrase. It's the only thing that came to my mind when I read the title. The author then goes out on a limb and says:

Keep your code simple. The lower the cyclomatic complexity of your methods, the better.

He then describes how by determing the cyclomatic complexity, you can then enumerate the scenarios that need to be tested.

This guy's heart is in the right place, but massively enumerated test paths is the worst-case scenario. Just reduce the complexity. It's so much easier than writing thousands of test cases and the accompanying data. Not to mention that it actually improves your code. I'll go out on a limb and say that if you need to enumerate through many test cases because of high complexity, you're going to have bugs. It doesn't matter if you write all these test cases and all this test data. Unit testing can't always save poor design and/or bad code.

technorati tags:java, cyclomatic complexity, unit testing, metrics