Before I head off in to the weeds, let me get a couple of things straight: 1) I like unit testing, 2) I like agile, 3) I like getting enough sleep. With that said it's time for me to get something off my chest: I hate interaction-based unit testing and I think too much emphasis on unit tests is bad for quality!

For a while now I've had a basic coverage rule that governs how much unit testing is required from my team.  If it covers 85% of the lines of code in the class the class is covered. We have a nice nightly automation system that ensures it and we're allowed to arbitrarily tweak the % for particular packages or classes. I'm more or less happy with this threshold and if anything I think I'd probably choose to lower that number and not raise it. When I tell my hard-core agile friends about this and mention that I don't care if people write the tests before or after they write the code I get a bunch of snickering and a kind of "I suppose if you think that's good enough for YOUR project" kind of attitude.

It's this attitude that quality is king and that unless you follow these particular rules you're skimping on quality that bugs me. They've married TDD with extreme unit testing and used it as a kind of quality blackmail, or a political correctness. As if quality were a byproduct of this particular set of religious rituals which, when performed just right, enable comprehension of the quality Zen. In fact, it's my belief that too much attention to unit testing actually hurts product quality in a many companies (including and especially many agile ones). Here's why: the ability to refactor code as requirements change over time is a huge factor in maintaining code quality. Excessive unit testing infrastructure hampers your practical ability to refactor things efficiently.

People scream at me when I say this. Unit tests are supposed to enable refactoring by spotting unexpected bugs resulting from the refactoring effort. I agree.  Unit tests help with this, but only up to a point. When they cross the line into extreme-testing they can be pretty much guaranteed to break every single time you refactor the code. Of course this means that when you refactor you will need to rework a bunch of tests and this is where the system falls apart. People don't want to make small refactors when they know that they will be forced to do a ton of extra work to make the change. They are lazy (or perhaps they have an actual deadline, remember those) and so rather than refactoring and having to fix the tests they hack in the least obtrusive way possible leaving the tests untouched. So the tool that was supposed to enable continuous refactoring ends up driving people to hold off on refactoring and doing it in big chunks. Strange irony indeed.

No doubt, purists will complain that this is just lazy practice. It is not. It is practical economic decision making on the part of the developers and failing to account for it is purism run amok. Projects do not get to go on forever. We must make choices about how to most effectively test what we write and when we choose to spend time in one area we DO sacrifice time in another.

Below are some extreme-testing related testing ideologies that can create problems when treated religiously. Some of these ideologies infect regular unit-testing shops, and some TDD shops avoid them.

Class == unit

Sure, some times a class is a unit, but not always. A unit is just as often a collection of two or three or even five or six reasonably related classes. This is especially true if you are a bit more religious about following the Single Responsibility Principle. There is nothing wrong with (for example) testing a factory class along with the two classes it can produce; especially if the generated classes need to interact in such ways as supporting comparisons properly, accepting and aggregating one another (as interfaces or in base class form), etc. By working with the three classes as a unit you provide a better real-world interaction scenario and you test against likely usage. At the same time you'll probably get reasonable coverage with a lot less code (which is a good thing, more code is evil even when it's testing code).

Doing order dependent testing

If you use an interaction-based framework (EasyMock for example) you are very likely to find yourself writing mock behaviors and thinking, "it'll call this, then this, then this, then the first thing again..." Would you accept this level of order dependency amongst regular classes? Then why is it a good thing when working with test classes? If you can ensure testing coverage entirely with stubs (and you almost always can) then you should. Your tests will survive minor changes to the code and still tell you when you break something. That's the actual goal, not having lots of tests. If you can't get the test to cover the most meaningful cases just using stubs you may need to look at why the code is so complex and start fixing it rather than wasting your time testing it. Are there cases for interaction-based tests. Sure, but they should be extremely rare.

Always mocking/stubbing the database layer

Theory: "We're not testing the database persistence tools, we're testing our code. The database tools are expected to work reliably."

Practice: ARE YOU COMPLETELY INSANE! Testing that the system actually reads, writes, and aggregates things at the persistence layer is probably THE single most valuable set of test in the whole system. The tools may work, but they are very easy to use wrong and unless you are writing your DB stubs to parse and interpret the meaning of SQL (or HQL or whatever) how exactly are you going to know that the unit is even vaguely doing what it is supposed to? Naturally there is some work required to get your system set up to be able to run unit tests that talk to a database, but it's pretty easy to do so I'm amazed when people don't do it.

Of course, many of these same people will argue that this kind of testing is better done at the system (integration) level so doing it at the unit level is a waste of time. My experience is that these system level tests almost never get written and when they do they cover only a very small percentage of the potential cases that are covered by the unit-level database tests. The ideal in testing is to write as little as possible while finding the maximum number of real bugs (ones the customer would actually find). The goal is not to achieve some well segregated testing-level-meaning compliance.

Mocking system objects needlessly

Short version: mocking a Map is a form of mental illness.

Always using interfaces

Here's a shocker: not everything is an interface. Some things really are objects with actual state and actual meaning to that state. Creating interfaces on objects needlessly just so that some object under test can be loosely coupled is a perversion of your class hierarchy for testing purposes. Consider extending the supporting class to create a stub. Consider just letting the object be included in the tests and stubbing further down if it's reasonable. Making everything into interfaces and factories is just plain wrong.

TDD systematizes your brain

My final gripe with religious TDD (now I am talking about TDD and not unit testing in general) is that it turns quality in to nothing more than a system.  This is something you can never do. Finding bugs is a real skill. It involves thinking about how the program or component will be used and abused and trying to make sure it survives those uses. When building the unit tests becomes either purely mechanical OR so intertwined with the actual code it causes people to stop thinking with the creative part of their brain it can be down-right destructive to actually preventing bugs.

Of course, my 85% coverage threshold can systematize things as well (to a degree) and so I have serious reservations about it. If at some point I see that it's getting me less value than it's worth I'll cast it aside and find a better approach. Even unit testing need not be the right answer. Will Shipley makes a pretty strong argument that you don't need it, and sense he writes some actual, real-life good software why the heck should anyone complain.

Wipe that smug look off your face

Having said all this a few times to a few people, I've basically gotten tired of hearing unit-testing theory spit back at me chapter and verse from extreme TDD apologists. Ultimately, if you want to spend a huge chunk of your life writing tests that will never find a single bug in your code then go ahead. I'm more interested in delivering working software with the best mix of features and quality I can achieve within whatever practical constraints I have to work with. If less unit testing and more manual testing gets me there great. If I improve quality by staring at the code for an extra two minutes that's great too. In fact, if you can demonstrate that waving chicken bones over the code at twelve midnight during a harvest moon will reduce bugs I'll go for that too. Unfortunately, since magic solutions don't exist, I'll just have to keep treating all my time consuming tasks (analysis, development, prototyping, unit-testing, system testing, code review, etc) as something that I parcel out according to the economics of their value and leave religious fervor for things that warrant it.