Making Good Software

A blog by Alberto G (Alberto Gutierrez)

Written by Alberto Gutierrez

January 27th, 2012 at 7:54 am

The evil unit test.

with 14 comments

Unit tests can be evil, I know that sounds harsh, but I think there is a battle to be fight to make people understand that unit tests are not always good, and that they actually can become evil, specially when overused.

Think in a foreach, most languages have an implementation of this flow statement. Is the foreach a useful utility for programmers? Definitely! What about function pointers? or Design patterns? or ORM frameworks?… All of them are useful utilities, but can you imagine a programmer, or a book naming the following principles?

  • You shall use a foreach every time you need to perform a loop.
  • Every third method shall implement a design pattern.
  • You will never access your database if you are not using an ORM.

Stupid, right? So why is it that there are programmers happy with statements like:

  • Every public method shall have a unit test.
  • Unit tests are not to be deleted. There is always a reason for a unit test to exist!
  • Unit tests must be created before the code they test.
  • Unit test coverage should be 100%.

Unit tests, as any other utility, tool, principle, philosophy etc. are only to be used when applicable and convenient, and not to be taken as religious dogmas. I have already made this point in my previous article, but since it generated so much controversy I thought it would be good to expand on it. If you can’t unit test a piece of code from a behaviour/black box perspective, then you probably shouldn’t write a unit test at all. Your unit test should check what your code is doing, not how. If you create a unit test that it only focus on how your code is, instead of what it does, then you are writing an evil test because:

1. Evil tests create a lock on how the code is implemented.

Every time the implementation of the method changes, even if its behaviour is maintained, the test becomes red. There is no reason to lock the implementation, because there is no advantage on doing so. If there is some code that is especially tricky, locking its implementation with tests won’t make it easier to understand or won’t force anyone to think twice before changing it.

2. Cause duplication.

The beauty of code with good unit tests is that they compliment each other; the code to be tested represents the implementation (the how), and the test guarantees behaviour (the what). On the other hand, evil unit tests look again into the how, so you will have two places where you state how do you think a problem should be solved, only that you will be using different expressions to do so. Is like saying: “2*3 = 2 + 2 + 2”, and then writing a test that says “Guarantee that 2*3 is the addition of 2 three times”, as opposite to “Guarantee that 2*3 is 6”. Testing the implementation and not the behaviour is a waste of time and serves no purpose since you can always go to the code to find what the implementation is.

3. Builds uncertainty on the tests (red is meaningless).

One of the main advantages of having an automated test suite is that feedback loops become shorter. It takes less time from creating a bug to detecting it. When the code gets polluted with bad unit tests, the automated tests seem to start getting broken randomly and this sometimes leads to a point where developers don’t care much anymore about having their tests passing as usual as they should.

4. Decrease productivity.

Given all the circumstances mentioned already, developers dealing with evil unit tests are faced with many inconveniences that forces them to expend time in unproductive tasks, like refactoring useless unit tests, writing them…

5. Discourage change.

Code bloated with bad unit tests is a nightmare to maintain, you make an implementation change, not a behaviour change, and you have to expend ages fixing tests, hence you minimize your implementation changes which is the core idea behind refactoring.

Consider the following principles then:

1. Delete evil tests. There seems to be a taboo around deleting unit tests, is like they are untouchables, well that’s no true, if the test is not adding value, then the test is an evil test and MUST be deleted.

2. Minimise the areas of your code that can’t be effectively unit tested. It is also true that sometimes the problem is that the code is written so that areas that should be unit testable are tied to components that make difficult the unit testing, in this case, the programmers needs to expend more time understanding the principles behind loose coupling and separation of concerns.

3. Write integrations tests for areas that are not unit testable. There is always going to be code in your application that is going to be legitimately not unit testable, for this code, simply don’t write a unit test, write an integration test.

 

14 Responses to 'The evil unit test.'

Subscribe to comments with RSS or TrackBack to 'The evil unit test.'.

  1. In modern times with frameworks that will calculate your taxes and tie your shoes, integration tests take front and center for me.

    Drew

    30 Jan 12 at 7:46 am

  2. Absolutely – even those of us (like me – disclaimer: I work at Typemock) know that there is a time and a place for unit tests and TDD (not the same thing).

    In fact, we wrote about this before http://bitly.com/zdyJfl

    We’re hosting a webinar on Wednesday about different kinds of testing and when a unit test is appropriate and when another kind of test, like an integration test, may be a better choice – http://bitly.com/xeYSYg.

    Avi

    30 Jan 12 at 9:10 am

  3. The shorthand is “write Behavior-driven tests”, or BDD. Testing implementation is terrible terrible terrible.

    Matt Rogish

    30 Jan 12 at 11:00 am

  4. Your article gives a bad view of what I think you’re trying to get at. It sounds more like unit tests are the problem in some places here, but I think you mean that poorly written unit tests are the problem. Unit tests should always display the behavior of the code, never the implementation. If there are implementation details, you’re doing it wrong. And if something is too hard to test, you may not be separating your concerns enough in the code.

    Also, if you’re using TDD to write your unit tests (this is a must), then the last part of number 4 is just wrong. It can take time to rewrite bad unit tests into good ones, but if you write a good unit test, it should not be decreasing your productivity, but increasing it.

    Kevin

    30 Jan 12 at 11:41 am

  5. If every little change suddenly makes 100′s of tests go red, the problem is rarely in the tests. More often, your tests are brittle _because_ your code is brittle (written at the wrong level of abstraction). Refactoring your internal API can make the tests test more meaningful things, which means less tests should break at once.

    In other words, writing future-proof tests involves thinking about the problem a little deeper than normal and writing better, more future-proof code. Every program has billions of ways it _could_ be implemented. But some ways are more maintainable (in the face of arbitrary changes) than others. Writing (and fixing) tests all the time will eventually give you that intuitive knowledge of how to write maintainable code.

    (Just like writing software all the time gives you an intuitive sense of the language, so you make fewer mistakes. Beginners wrestle with the compiler after every change. Experts rarely get compile errors, and when they do they are small typos. Tests are the new compiler errors!)

    BraveNewCurrency

    30 Jan 12 at 12:35 pm

  6. Tests should test behavior, NOT implementation. If I change the behavior without changing the implementation I shouldn’t need to change the test.

    MUCH easier said than done, but it really helps to keep this goal in mind.

    Jaco Pretorius

    30 Jan 12 at 12:54 pm

  7. I think the main reason why therea re statements like

    - Every public method shall have a unit test.

    is because there are still a lot of people who simply do NOt make use of unit tests. On the other hand there are people who use databases without ORM (btw. I don’t like ORMs). So from my experience: software developed with unit tests is more often corrent than other…

    (Always remember: write the test first :p)

    meamu

    30 Jan 12 at 1:12 pm

  8. Unit tests, acceptance tests, and integration tests are different things.
    Unit tests test that the implementation behaves as intended, and generally grope around private internals. These change when implementation changes.
    Acceptance tests test that the public API is implemented correctly. These change when the API/interface change.
    Integration tests test that the end system implements requirements correctly, and change when requirements change.
    Most TDD hate comes from not separating these correctly, or not having suitable tools for each type, or from (gasp!) not actually having a clear understanding of requirements, interface, or implementation.

    Jon W

    30 Jan 12 at 1:32 pm

  9. [...] The evil unit test. | Making Good Software – [...]

  10. Your website doesn’t work in a mobile platform. When I scroll down, the share/like bullshit appears on the left and covers your content.

    xdf

    31 Jan 12 at 9:10 am

  11. You blog is full of shares/like and the ads are all shit to me. Please, people come here for content, not to see your ads!!!!!

    ads

    7 Feb 12 at 7:45 am

  12. As Kevin points out above, unit tests aren’t “evil” in and of themselves. I worry that your article might give the wrong impression to those new to testing.

    I can’t disagree with anything you’re pointing out – sacrosanct tests *do* lock up implementation, productivity, etc – but that’s not a problem w/ the tests, it’s a problem with the team’s methodology. Where I work, tests are a tool and when they are used inappropriately, someone loses an eye. When used correctly, they *increase* productivity, *reduce* duplication, *free* us from implementation locking, and *encourage* change.

    So what’s the big difference? IMHO, it’s constant refactoring. If time is allocated to actually maintain software (instead of constantly writing new features), and if some of that time is spent on making the tests better (less brittle, more clear, less duplication), then the test suite becomes a thing of beauty and so does the code it supports.

    Love where you’re coming from, BTW – but I think you’re telling only half the story.

    Cheers!

  13. [...] The evil unit test. (makinggoodsoftware.com) [...]

  14. [...] The evil unit test. (makinggoodsoftware.com) [...]

Leave a Reply