In my team we have been working on a firmware for a product. The product was released some years ago but we still implement new features in the firmware and regularly provide a new firmware for our customers. Recently we have developed a new product which use the same firmware as the first one. Of course, we had to change the firmware to allow for some new capabilities (while still keeping functionality for existing product intact).
We have a release test specification which contans mostly manual tests which we carry out before we release any new software. We have adapted this for the new product since it has additional capabilities. But most of it is common.
I know that new products are in the pipeline and in my simplistic reasoning we need to repeat our release test for each product every time we release a firmware. So i am looking for a new approach which is less time consuming. One of the things we are already working on is automation, but due to the nature of the product many tests have to be manual.
7
The broader scope of software testing involves analyzing all of your products, supported platforms, and then identifying what needs to be tested to provide a particular degree of assurance about the results.
The most important aspect is to document what you decide and why you decided that. You’ll want to provide that documentation to your clients so they can be assured of your testing process.
For your case:
-
Since you’re expanding, you should look for the commonality between products and firmware releases. Aspects that are truly common don’t necessarily have to be tested on every single target platform. A representative sample should be sufficient.
-
Provide some discipline around your release process so you can understand what has changed between releases. If you can demonstrate that functional area XYZ won’t be affected by any of this release’s changes, then you can either eliminate or reduce the amount of testing of that area. Your product domain will dictate the balance point between eliminate or reduce testing for that cycle.
-
It can be expensive, but consider adding robotic automation around your test processes. Pulling a cable; pressing buttons; dropping the device; etc… can all be performed with a robotics test harness. This approach allows you to test around the clock, which is the other solution for time consuming tasks. In other words, you found more time instead of decreasing the amount of time required.
I would split your tests in three categories:
- (Unit)tests on a (simulated) reference platform. These tests should cover all the software that does not directly interface with hardware components and should include both unit-tests for the various software components and functional tests for all the features. The (simulated) reference platform should be able to support all features (even if there is no single product that has them all) and should be able to simulate hardware failures.
- Tests of the hardware interfaces. These tests should give you confidence in the interfacing between hardware and software. By necessity, these tests must be executed on the hardware itself and repeated for each distinct hardware design. It might be good if these tests are specified/designed by a combined team of hardware and software people.
- System tests. These tests should give confidence that the final assembled system works as designed. They should focus on the most-used features (and perhaps some easy-to-test error scenarios) and be repeated for each product that has a distinct hardware design or non-overlapping feature-set (if two products are technically identical, but one has a reduced feature set, there is no need to test that second one as extensively).