Is this a test smell or is it even worse?

I have recently been looking at some testscripts which looks a bit like

...
try {
  receiveSomething();
  // something was received even though it shouldn't
  failTest();
} catch (TimeoutException e) {
  // nothing should be received
  succedTest();
}

The problem I have with these types of tests is

They are not deterministic. You don’t know if nothing was sent on purpose or if everything has crashed.
Its very hard to simultaneously test something else, which might actually, in this case, send something.

My thoughts are one the one side, how can this types of tests be designed better, and secondly can this be an indication of a bigger design smell of the software that is being tested?

EDIT

To clairify, these test scripts are used for black box-testing of event-based, complex software, not unit testing, and my feeling is that the ‘doing nothing’ event is a very ambiguous one. 🙂

From what you wrote, I guess your situation is more like this

startAsynchronousComplexOperation();
try {
  receiveSomething(); // listen for events from that async operation above

  // something was received even though it shouldn't
  failTest();
} catch (TimeoutException e) {
  // nothing should be received
  succedTest();
}

and the object under test is not “receiveSomething”, but the asynchronous operation. Thus the outcome of that test depends on the speed of the test machine, if a lot of other processes run on the test machine etc., and this is non-deterministic.

IMHO, this is indeed a design problem, and not just a “test smell”. If your “complex operation” crashes, or just “takes too long”, not only your test will give a “false negative”. Think about how to deal with that situation in production. Is there no requirement to determine for sure that your “complex operation” is still alive and did not crash? So you may think about adding some kind of “watchdog” or “keep-alive-packet-sending” mechanic to your program to get control over that. And once you have such a mechanic available, you can use it in your tests like this:

startAsynchronousComplexOperation();
while(isComplexOperationStillAlive())
{
    try {
       receiveSomething(); // listen for events from that async operation above
       // something was received even though it shouldn't
       failTest();
   } catch (TimeoutException e) {
   }
}
if(didComplexOperationCrash())
    failTest();
else
    succedTest();

They are not deterministic. You don’t know if nothing was sent on purpose or if everything has crashed.

It doesn’t matter, in the context of this test. All that matters is that, under the preconditions defined, nothing is received. Other tests can simulate other preconditions and check their results are as-expected.

Its very hard to simultaneously test something else, which might actually, in this case, send something.

Good! You shouldn’t try to plug other tests into this test. This test is doing exactly what it should be doing. One set of preconditions => One result.

This is the very definition of determinism.

Very hard to answer this without some context.

Were the test case to be for the function point (“System Will not respond to any request after ‘Quiesce’ command issued”) then yes this would be a perfectly valid test case.

Although I would have specified that a valid response was received immediately before the shutdown request – but these type of “operational” tests are very difficult to automate as its unlikely you would be able to issue such system commands from an API.

They are not deterministic. You don’t know if nothing was sent on purpose or if everything has crashed.

A deterministic algorithm as defined in Wikipedia is as such:

is an algorithm which, given a particular input, will always produce the same output

But the algorithms to build the test hardly matter, pretty much because it is supposed to test the proper state after invoking a specific behavior. In other words, the only thing that needs to be deterministic is the code module that you are testing. The test itself is actually definining what the determinination should be.

Its very hard to simultaneously test something else, which might actually, in this case, send something.

As others pointed out, that is a GOOD THING. If I have a test for module A, that has dependencies on Modules B and C, then if TestModuleA fails, how do I know it is ACTUALLY an issue in module A?

Your test is supposed to just verify that Module A provides the correct state for invoking an operation of A. You should have seperate tests for the dependencies to verify their correct behavior too.

Filed under: softwareengineering - @ 20:08

Thẻ: code-smell, functional-testing

Thiết kế website giá rẻ

Danh mục

Is this a test smell or is it even worse?