ADT Interview: Failures in Unit Testing
(Tue Feb 08, 2005) [/CodeCraft] #
A while back, Mike Gunderloy of Application Development Trends interviewed me for his Five Questions column, which was distributed in a newsletter. With Mike's permission, I thought it would be fun to post the interview one question at a time over several weeks. Think about how you'd answer each question differently. Here's the third one:
Q: Now that we've got some reasonable experience with unit testing, where do you think it fails?
A: I think unit testing fails when we use it as a substitute for acceptance testing. That is, unit tests that pass don't tell us that we built the right thing. They only tell us that what we built works as we, the programmers, expect. So as much as I'm encouraged by the enthusiasm for test-driven development and unit testing, I think we have to start putting an equal amount of emphasis on acceptance testing.
I also see unit testing failing in cases where it's a checkbox item on a project schedule. Just saying that we're doing unit testing says nothing about the quality of the tests. So I like to see a test fail once in a while as it tells me that the test is actually testing something that could break. I also like to open the tests up to review by users of my code and folks with more testing experience to see where I might improve the quality of the tests.
Finally, a common question I hear revolves around what to do about legacy code on a project that doesn't have unit tests. It frightens me to hear that some managers want to get to a certain percentage of test coverage for all legacy code. Unit testing legacy code can be successful if done tactically around sections of code undergoing change or containing bugs. But when unit testing is taken as an all-or-nothing approach, it's neither practical nor economical.
In what ways has unit testing been successful on your project, and where has it failed to deliver the results you expected? When is the last time one of your unit tests unexpectedly failed? How was that feedback helpful? What approaches are you taking to gain confidence to change legacy code economically? I'd enjoy hearing your responses on your blog or via email.
