Every defect in software is ultimately due to a discrepancy between assumptions and reality. Insidious bugs are simply discrepancies between particularly deep-seated assumptions and reality. The ability to diagnose bugs depends on your capability to question your own assumptions, and that does indeed require an awareness that you don’t know certain things, you just assumed them and used to be right until now.

Obviously the tools of the trade, log files, debuggers etc. are helpful in uncovering such assumptions and re-aligning your world model with the actual system. But until you are ready to question the crucial assumption, e.g. “It can’t be bad input because we have comprehensive input checking”, you will spend your entire time checking the wrong parts of the system, or simply not knowing where else to look in the first place.

What do I need to do to help my, apparently, limited imagination come up with ways that my project could possibly be broken?

In most cases, I’d say absolutely nothing. You shouldn’t be trying to dream up things that might be causing the program to break. You should be systematically determining what is causing it to break.

You should be going into the debugging process with the following pieces of information:

• the steps that were taken and the values that were entered to produce the bug;
• what the program should be doing when given those steps and inputs;
• what the program is doing when given those steps and inputs.

If there’s an error message, get all the information you can about it. If the error message itself isn’t clear, and you don’t know what it means in practice (some error messages aren’t always particularly useful), then use Google, or StackOverflow, or any other online resource to find information about it.

If there isn’t an error message displayed on the front end, check any logs the application writes to for error messages during the time period you reproduced the bug. The code may have executed to completion, but encountered an exception that’s being handled along the way which is throwing off the end result and producing an entry in the logs. Look for those, do the same above and identify exactly what they mean.

If there are stacktraces provided with exceptions thrown by your code (and there should be), look at the lines of code mentioned. The line itself may not be the one that’s actually producing the issue. If you get a NullPointerException in a piece of Java code, the stacktrace will tell you where you tried to use something that was null when you expected it not to be. That doesn’t exactly point you to the line causing the problem, but it generally does tell you what variable doesn’t have the value you expect it to, so you can look at any references/assignments to that variable to determine that the value isn’t being set or that the value is being set incorrectly.

If none of that has helped, fire up your debugger. If you’ve narrowed it down to a section of the code that you know is causing the issue – but you don’t know exactly which line(s) – then step through that. If not, just step through the entire thing. This is where you need to know exactly what the program should be doing with given inputs, because you need to look at each value after each line and determine exactly where it’s deviating from what you’re expecting it to do.

Still don’t have a clue what the problem is? Ask somebody for help. A co-worker, a friend, an online community. Show them all that work you just did. Show them the error messages, the stacktraces, explain what the program does in general terms (if they don’t already know), what it should be doing in this particular instance (e.g. returning the value 4), what it’s actually doing (e.g. returning the value 5). If you’ve narrowed it down to a few lines of code in the debugger, say “I know the issue is caused by these lines in the code, it’s setting the value to X when it should be Y, but I can’t see why that’s happening”.

Spending a few hours staring blankly at your screen definitely isn’t fun, but there’s no reason you should be doing that. If there’s a problem with your code then you need to be reading or stepping through the code.

To some extent, it is like investigating an criminal case, or a mind-boggling puzzle.

First, you got the victim. After digging a while into the case, you identified a few suspects and you also developed a work hypothesis how exactly the victim might got murdered. You continue investigating, looking out for more helpful information, getting you nearer and nearer to the real source of the problem.

It happens that from time to time you walk into a dead end (pun intended). That’s part of it, and there’s nothing wrong with it, as long as you manage to bring yourself back on track as quickly as possible. The key is, to always think about what piece of information you need next, that either supplies your hypothesis (and provides you with further information), or proves it wrong. Then find a way to get that information in an efficient manner, make your conclusions and move on, until you finally are able to convict the guilty.

And sometimes you realize, that all the facts and indications necessary to spot the culprit were already waiting in front of you half an hour ago. Albeit annoying, that’s among the most interesting parts, because if you do a critical review of your actions and mistakes, you may be able to learn and get better. Ask yourself questions like:

• How could I have avoided this waste of time?
• What did I overlook in the first place, and why?
• What unvalidated and/or wrong assumptions did I rely on?

This will train your abilities. It will also develop your gut instinct, so over time you learn to automatically notice all those minorish things that are too easily overlooked, leading you quicker to the right answer. At the end, it’s all about deliberate practice.

Last not least always remember what Sherlock Holmes taught us: When you have eliminated the impossible, whatever remains, however improbable, must be the truth.

What do I need to do to help my, apparently, limited imagination come up with ways that my project could possibly be broken?

Let history be your guide. If your project is well managed then you should have a database of every bug that has ever been fixed in the product along with an analysis of how the bug was found, how it was reproduced, how it was analyzed, and how it was fixed. That is not a write-only database. Read the database, and very soon bug taxonomies will begin to occur to you.

That will give you a good overview of the sorts of things that go wrong in your product. If you are interested more generally in what goes wrong in all kinds of software, particularly with an emphasis on security-impacting defects, I suggest that you read the CWE list: http://cwe.mitre.org/data/index.html

So rather than trying to reproduce and fix a specific defect I believe you are asking about thinking up new tests that you could use to investigate the product to see if the product works under those circumstances for example: what happens if I enter special characters into our sign up page’s password filed, or what happens if I forcefully close the program while it is writing to the database. These cases are indeed hard to think up.

Software development over the past 10 years (Agile / XP / TDD etc.) has come to value meeting the explicit requirements only and then declaring the feature finished, and not finding every possible way to break something (there are possible exceptions, if you are working for NASA or doing white hat security but even then there is a case to be made for calling such things out in the user story’s acceptance criteria).

So if your features explicitly list as acceptance criteria what the systems needs do such as how to handle input, its performance characteristics, user workflow actions, etc. then you have a complete list of what the tests need to be checking for. Testing should be done to validate that requirements have been met, and the best way to do that is to explicitly list all of your requirements. Take a look at the Agile Test Quadrants.

Some people advocate for these tests to be the explicit declaration of the requirements, that need to be written before the code i.e. Test First ( or Test Driven Development).

However I appreciate that you do not sound like you are contemplating a new project where you can set your own development best practices before you start and are instead coming in after the software has been written and are being asked to ‘test it’. This is indeed more challenging but the same principles apply, you can only test it once you know what it was supposed to do. If there is not a comprehensive list of requirements that were met by the development team for you to work from then asking questions is the best way forward. Depending on your team this may need to be done delicately as people who did not explicitly list their requirements before building a software system do not like to be reminded of what they missed but it is essential to performing this task well.

Once you find a requirement – it must be robust / it must be secure, then try to dig deeper and find out how secure it must be, or how much failure is acceptable, there is always a limit, not many people have an NSA proof level of security as a requirement for their application or would want to pay for that. The more you dig the clearer it should be as to what types of security attacks you need to be defending against or easy of use you are aiming for. Some domain knowledge is then useful, in security, in design, in performance etc. which is where you ask even more questions from the experts you can find in your team, or here on SE, or google / books.