A comment on a previous answer made me think about our hotfix policy. A current project for example is used by about 300 users all around the globe. One location found a strange bug where they issued a specific number into a combobox. That resulted in a complete server error (the session was reset with a 500 http error). I was like: “Yep, hotfix asap!” However, I’m not sure anymore since this bug only affects a specific location and a specific number. Commonly I didn’t care too much and just hotfixed it. Since this question Hotfixing with an open release branch while this release is currently tested I decided to be a more subtile when it comes to the hotfix-decision.
Is there a rule of thumb when to use a hotfix rather than just a bugfix for the next release?
1
I’m not sure there is a rule of thumb to be honest.
In the ideal situation, You are doing micro single feature releases, have a service based architecture with CI and automated testing. So rolling back is an easy decision to make, as you only lose a single feature and can release a new version almost immediately with low risk.
However in practice this is rarely the case, if you do monthly releases and have to roll back, even straight after release, you are going lose a whole bunch of features everyone has been waiting for. Maybe they have even been marketed to customers! The next release is another month away, so its a tough call to make. Hey, that next release might have another bug in it and you will have to roll it back too, the pressure to hot fix will only increase over time.
You can only move slowly towards the ideal, and it will mean a lot of refactoring and DevOps work. But you can pull a frowny face when hot fixes are requested and keep track of how many you do, and how much time is spent doing them.
There are several factors to consider IMO;
- Does the hot fix need to be deployed to the client, or can you “silently” patch the server to deal with it?
- How involved is a client upgrade? Automatic? Two hours of careful migration?
- How often does this happen and to how many users? (You say 1/300, but is that a fact, or the reports you got?)
- How much prestige do you put in the quality of your product? Is it acceptable that you have a (to the user) random crash live until the next scheduled update? Or is it worth the cost of a hot fix to earn the goodwill of swift support?
There isn’t really a rule of thumb. Different organizations handle this differently. Some will refuse to patch significant issues and wait for a future (possibly several release out) to resolve the issue. Others are proactive at resolving issue on the production release.
I’ve worked on a project where we actively applied fixes to the current release. Each fix was tracked into the release branch with one of three outcomes (the first two occurred with roughly the same frequency):
- Apply patch unchanged. (Applicable when the fix was complete.)
- Apply a complete solution in the release. (Applicable when an incomplete fix/workaround was used.)
- Drop the fix. (Extremely rare case where patched functionality did not exist in release.)
Patches released to the current release should become less frequent as the next release comes close to release. This is trade-off of multiple factors.
It is unlikely you have an edge case that applies to only one value at one site. It may be an edge case applying in a few cases, or possibly be fairly common. I’ve found failure cases that applied to significant portions of the users for which no users had reported a problem. Until you’ve identified the cause, you won’t know how common it is likely to be.