TL;DR
Writing good, useful tests is hard, and has a high cost in C++. Can you experienced developers share your rationale on what and when to test?
Long story
I used to do test-driven development, my whole team in fact, but it didn’t work well for us. We have many tests, but they never seem to cover the cases where we have actual bugs and regressions – which usually occur when units are interacting, not from their isolated behaviour.
This is often so hard to test on the unit level that we stopped doing TDD (except for components where it really speeds up development), and instead invested more time increasing the integration test coverage. While the small unit tests never caught any real bugs and were basically just maintenance overhead, the integration tests have really been worth the effort.
Now I’ve inherited a new project, and am wondering how to go about testing it. It’s a native C++/OpenGL application, so integration tests are not really an option. But unit testing in C++ is a bit harder than in Java (you have to explicitely make stuff virtual), and the program isn’t heavily object oriented, so I can’t mock/stub some stuff away.
I don’t want to rip apart and OO-ize the whole thing just to write some tests for the sake of writing tests. So I’m asking you: What is it I should write tests for? e.g.:
- Functions/Classes that I expect to change frequently?
- Functions/Classes that are more difficult to test manually?
- Functions/Classes that are easy to test already?
I began to investigate some respectful C++ code bases to see how they go about testing. Right now I’m looking into the Chromium source code, but I’m finding it hard to extract their testing rationale from the code. If anyone has a good example or post on how popular C++ users (guys from the committee, book authors, Google, Facebook, Microsoft, …) approach this, that’d be extra helpful.
Update
I have searched my way around this site and the web since writing this. Found some good stuff:
- When is it appropriate to not unit test?
- https://stackoverflow.com/questions/109432/what-not-to-test-when-it-comes-to-unit-testing
- http://junit.sourceforge.net/doc/faq/faq.htm#best
Sadly, all of these are rather Java/C# centric. Writing lots of tests in Java/C# is not a big problem, so the benefit usually outweights the costs.
But as I wrote above, it’s more difficult in C++. Especially if your code base is not-so-OO, you have to severely mess things up to get a good unit test coverage. For instance: The application I inherited has a Graphics name space that is a thin layer above OpenGL. In order to test any of the entities – which all use its functions directly – I’d have to turn this into an interface and a class and inject it in all the entities. That’s just one example.
So when answering this question, please keep in mind that I have to make a rather big investment for writing tests.
5
Well, Unit Testing is only one part. Integration tests help you with the problem of your team. Integration Tests can be written for all kinds of applications, also for native and OpenGL applications. You should check out “Growing Object Oriented Software Guided by Tests” by Steve Freemann and Nat Pryce (e.g. http://www.amazon.com/Growing-Object-Oriented-Software-Guided-Signature/dp/0321503627). It leads you step by step through the development of an application with GUI and network communication.
Testing Software that was not test driven is another story. Check Michael Feathers “Working Effectively with Legacy Code” (http://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052).
5
It’s a shame TDD “didn’t work well for you.” I think that’s the key to understanding where to turn. Revisit and understand how TDD didn’t work, what could you have done better, why was there difficulty.
So, of course your unit tests didn’t catch the bugs you found. That’s kind of the point. 🙂 You didn’t find those bugs because you prevented them from happening in the first place by giving thought to how the interfaces should work and how to make sure they were tested properly.
To answer, you question, as you have concluded, unit testing code that is not design to be tested is difficult. For existing code it may be more effective to use a functional or integration test environment rather than a unit test environment. Test the system overall focusing on specific areas.
Of course new development will benefit from TDD. As new features are added, refactoring for TDD might help to test the new development, while also allowing development of new unit test for the legacy functions.
2
I haven’t done TDD in C++ so i can’t comment on that, but you’re supposed to test the expected behaviour of your code. While the implementation can change, the behaviour should (usually?) stay the same. In JavaC# centric world, that would mean you only test the public methods, writing tests for the expected behaviour and doing that before implementation (which is usually better said than done :)).