It is quite easy to cover your code by writing tests first using TDD and you know that when to stop once you implemented a feature.
For me it is trickier to decide how many functional tests to write for say a REST service.
Currently I am in a situation where a service returns few fields in JSON and I can create/updade/retrieve/delete an entry. There are cases where errors are thrown if for example for one or more fields wrong value has been assigned. Also there are dependencies and given a value for one field would make another invalid with certain value. So service is doing some validation.
I have few options(maybe you see and suggest more):
- Write functional tests for each case to verify that system behaves as expected and of course write unit tests together with it. And integration tests if you need it.
- Write some functional tests to verify that the response format/status code/headers are valid but not cover each case as you can cover these with integration/unit tests say testing at the service boundary together with persistence layer(repository) without controller.So I could loose some confidence in actually guaranteeing that my REST service will marshal results back as expected.
Obviously there are trade offs. In first scenario – higher confidence, more time writing tests, takes longer to build an artifact because I have more tests that run slower. In second scenario – faster build, quite high confidence but not as high as with the scenario 1, less time spent writing functional tests.
What is your experience/view on this and what do you think is best option?
2
This is too broad of a question to answer specifically. There are a number of textbook approaches to this and no one will be able to tell you exactly how many number of tests to run without analyzing the details of your project. The main thing is that both you and the customer have to be satisfied.
Here is a common strategy that you didn’t mention in your post:
- Create a list of all the tests you can think of without regard to need, cost or difficulty.
- Ask the product owner to rank them in terms of desirability. The customer might think of other tests too.
- Rank the tests yourself in terms of which one you think is most important to mitigate risks of defects. For instance to address parts of the project that especially tricky.
- Combine your ranking and and the customer’s to prioritize all of the tests.
- Starting with the highest priority tests, tell the customer which of the tests can be done within your time/cost constraints.
- Focus your testing on high priority tests.
As one of the commenters alluded to, figuring all of this out is not easy. Assuming you aren’t doing this as a hobby, you will actually have to talk to the customer or product owner to figure out the right number of tests.
1