Does software testing methodology rely on flawed data?

Yes, demonstrably. Examining the Agile Cost of Change Curve shows that part of Kent Beck’s work on XP (I’m not sure whether it was part of his motivation or his justification) was to “flatten the curve” of defect costs, based on knowledge of the “exponential” curve that lies behind the Code Complete table. So yes, work on at least one methodology – the one that did most to popularise test-first development – is at least in part based on flawed data.

Is there any independent data to corroborate or refute this?

Yes, there certainly is other data you can look to – the largest study I’m aware of is the analysis of defects done at Hughes Aircraft as part of their CMM evaluation program. The report from there shows how defect costs depended with phase for them: though the data in that report don’t include variances so you need to be wary of drawing too many “this thing costs more than that thing” conclusions. You should also notice that, independent of methodology, there have been changes in tools and techniques between the 1980s and today that call the relevance of these data into question.

So, assuming that we do still have a problem justifying these numbers:

how does this impact software development methodology?

The fact that we’ve been relying on numbers that can’t be verified didn’t stop people making progress based on anecdotes and experience: the same way that many master-apprentice trades are learned. I don’t think there was a Journal of Evidence-Based Masonry during the middle ages, but a whole bunch of big, impressive and long-lasting buildings were nonetheless constructed with some observable amount of success. What it means is that we’re mainly basing our practice on “what worked for me or the people I’ve met”; no bad thing, but perhaps not the most efficient way to improve a field of millions of people who provide the cornerstone of the current technological age.

I find it disappointing that in a so-called engineering discipline doesn’t have a better foundation in empiricism, and I suspect (though clearly cannot prove) that we’d be able to make better, clearer progress at improving our techniques and methodologies were that foundation in place – just as clinical medicine appears to have been transformed by evidence-based practice. That’s based on some big assumptions though:

• that the proprietary nature of most software engineering practice does not prevent enough useful and relevant data being gathered;
• that conclusions drawn from these data are generally applicable (because software engineering is a skilled profession, personal variances in experience, ability and taste could affect such applicability);
• that software engineers “in the field” are able and motivated to make use of the results thus obtained; and
• that we actually know what questions we’re supposed to be asking in the first place. This is obviously the biggest point here: when we talk about improving software engineering, what is it that we want to improve? What’s the measurement? Does improving that measurement actually improve the outcome, or does it game the system? As an example, suppose the management at my company decided we were going to decrease the ratio between actual project cost and predicted project cost. I could just start multiplying all my cost estimates by a fudge factor and I’d achieve that “goal”. Should it then become standard industry practice to fudge all estimates?

For my part, the answer to “how does this impact software development methodology” is “not much”.

Whether caught by the developer or the end user, whether it takes more money to fix it after it’s been caught by the user or not, the fact remains that a bug has been found in the system. If caught by the developer, hopefully it’s a quick fix. The same hope holds for bugs caught by the user.

Regardless of actual developer-hour cost to fix a bug caught by an end user, there is the intangible cost of maintaining the stereotype that coders suck at what they do. When a user finds a bug, it’s the developer’s fault. Therefore, each bug the end user finds reduces the user’s confidence in the system. It’s like touring a home you want to buy, and seeing a water stain showing through the ceiling in one corner of the house. That, in itself, is an easy fix, but you wonder what caused it, and what else that root cause may have affected. What’s your peace of mind worth? You may have to tear the walls down back to the studs and visually inspect everything to ensure the stain was from an isolated incident that has been fixed. Knowing that might be a possibility doesn’t make you very confident in the home. Similarly, if the developer who wrote your software missed this very obvious (to you) cosmetic flaw, what’s wrong deeper down?

These intangible costs are avoided the sooner the bug is caught, which is the stated purpose of TDD-style methodologies. A bug caught during typing by the developer or partner in a pair, one caught at compile time, or one caught by unit/integration testing (the layer added by TDD), is a bug that the user never has to know about, that your project manager never has to apologize for, and that you don’t have to be pulled off of whatever you’re doing right this second to switch gears into defect-fixing mode on a part of the system you thought you’d left behind weeks ago.

I’m going to preface this with the fact that most of what I’m finding comes from the 1970s and early 1980s. During this time, sequential process models were far more common than the iterative and/or incremental approaches (the Spiral model or the agile methods). Much of this work is built on these sequential models. However, I don’t think that destroys the relationship, but one of the benefits of iterative/incremental approaches is to release features (an entire vertical slice of an application) quickly and correct problems in it before dependencies are injected and complexity of each phase is high.

I just pulled out my copy of Software Engineering Economics and found a reference to the data behind this chart in Chapter 4. He cites “Design and Code Inspections to Reduce Errors in Program Development” by M.E. Fagan (IEEE, PDF from UMD), E.B. Daly’s “Management of Software Engineering”, W.E. Stephenson’s “An Analysis of the Resources Used in Safeguard System Software Development” (ACM), and “several TRW projects”.

…the relative cost of correcting software errors (or making other
software changes) as a function of the phase in which the corrections
or changes are made. If a software requirements error is detected and
corrected during the plans and requirements phase, its correction is a
relatively simple matter of updating the requirements specification.
If the same error is not corrected until the maintenance phase, the
correction involves a much larger inventory of specifications, code,
user and maintenance manuals, and training material.

Further, late corrections involve a much more formal change approval
and control process, and a much more extensive activity to revalidate
the correction. These factors combine to make the error typically 100
times more expensive to correct in the maintenance phase on large
projects than in the requirements phase.

Bohem also looked at two smaller, less formal projects and found an increase in cost, but far less significant than the 100 times identified in the larger projects. Given the chart, the differences appears to be 4 times greater to fix a requirements defect after the system is operational than in the requirements phase. He attributed this to the smaller inventory of items that comprise the project and the reduced formality that led to the ability to implement simpler fixed faster.

Based on Boehm in Software Engineering Economics, the table in Code Complete is rather bloated (the low end of the ranges is often too high). The cost to make any change within phase is indeed 1. Extrapolating from Figure 4-2 in Software Engineering Economics, a requirements change should be 1.5-2.5 times in architecture, 2.5-10 in coding, 4-20 in testing, and 4-100 in maintenance. The amount depends on the size and complexity of the project as well as formality of the process used.

In Appendix E of Barry Boehm and Richard Turner’s Balancing Agility and Discipline contains a small section on the empirical findings regarding the cost of change.

The opening paragraphs cite Kent Beck’s Extreme Programming Explained, quoting Beck. It says that if the cost of changes rose slowly over time, decisions would be made as late as possible and only what was needed would be implemented. This is known as the “flat curve” and it is what drives Extreme Programming. However, what previous literature found was the “steep curve”, with small systems (<5 KSLOC) having a change of 5:1 and large systems having a change of 100:1.

The section cites the University of Maryland’s Center for Empirically Based Software Engineering (sponsored by the National Science Foundation). They performed a search of available literature and found that the results tended to confirm a 100:1 ratio, with some results indicating a range of 70:1 to 125:1. Unfortunately, these were typically “big design up front” projects and managed in a sequential manner.

There are samples of “small commercial Java projects” run using Extreme Programming. For each Story, the amount of effort in error fixing, new design, and refactoring was tracked. The data shows that as the system is developed (more user stories are implemented), the average effort tends to increase in a non-trivial rate. Effort in refactoring increases about 5% and efforts toward effort fixing increases about 4%.

What I’m learning is that system complexity plays a great role in the amount of effort needed. By building vertical slices through the system, you slow down the rate of curve by slowly adding complexity instead of adding it in piles. Rather than dealing with the mass of complexity of requirements followed by an extremely complex architecture, followed by an extremely complex implementation, and so on, you start very simply and add on.

What impact does this have on the cost to fix? In the end, perhaps not much. However, it does have the advantages of allowing for more control over complexity (through the management of technical debt). In addition, the frequent deliverables often associated with agile mean that the project might end sooner – rather than delivering “the system”, pieces are delivered until the business needs are satisfied or have changed drastically that a new system (and therefore a new project) is needed.

Stephen Kan’s Metrics and Models in Software Quality Engineering has a section in Chapter 6 about the cost effectiveness of phase defect removal.

He starts off by citing Fagan’s 1976 paper (also cited in Software Engineering Economics) to state that rework done in high level design (system architecture), low-level design (detailed design), and implementation can be between 10 and 100 times less expensive than work done during component and system level testing.

He also cites two publications, from 1982 and 1984, by Freedman and Weinberg that discuss large systems. The first is “Handbook of Walkthroughs, Inspections, and Technical Reviews” and the second is “Reviews, Walkthroughs, and Inspections”. The application of reviews early in the development cycle can reduce the number of errors that reach the testing phases by a factor of 10. This reduction in the number of defects leads to reduced testing costs by 50% to 80%. I would have to read the studies in more detail, but it appears that the cost also includes finding and fixing the defects.

A 1983 study by Remus, “Integrated Software Validation in the View of Inspections/Review”, studied the cost of removing defects in different phases, specifically design/code inspections, testing, and maintenance, using data from IBM’s Santa Teresa Laboratory in California. The cited results indicate a cost ratio of 1:20:82. That is, a defect found in design or code inspections has a cost-to-change of 1. If the same defect escapes into testing, it will cost 20 times more. If it escapes all the way to a user, it will multiple the cost-to-correct by up to 82. Kan, using sample data from IBM’s Rochester, Minnessota facility, found the defect removal cost for the AS/400 project to be similar at 1:13:92. However, he points out that the increase in cost might be due to the increased difficulty to find a defect.

Gilb’s 1993 (“Software Inspection”) and 1999 (“Optimizing Software Engineering Specification and Quality Control Processes”) publications on software inspection are mentioned to corroborate the other studies.

Additional information might be found in Construx’s page on Defect Cost Increase, which provides a number of references on the increase in defect-repair cost. It should be noted that Steve McConnell, author of Code Complete, founded and works for Construx.

I recently listened to a talk, Real Software Engineering, given by Glenn Vanderburg at Lone Star Ruby Conference in 2010. He’s given the same talk at Scottish Ruby Conference and Erubycon in 2011, QCon San Francisco in 2012, and O’Reilly Software Architecture Conference in 2015. I’ve only listened to the Lone Star Ruby Conference, but the talk has evolved over time as his ideas were refined.

Venderburg suggests that the all of this historical data is actually showing the cost to fix defects as time progresses, not necessarily as a project moves through phases. Many of the projects examined in the previously mentioned papers and books were sequential “waterfall” projects, where phase and time moved together. However, a similar pattern would emerge in iterative and incremental projects – if a defect was injected in one iteration, it would be relatively inexpensive to fix in that iteration. However, as the iterations progress, lots of things happen – the software becomes more complex, people forget some of the minor details about working in particular modules or parts of the code, requirements change. All of these will increase the cost of fixing the defect.

I think that this is probably closer to reality. In a waterfall project, the cost increases because of the amount of artifacts that need to be corrected due to an upstream problem. In iterative and incremental projects, the cost increases because of an increase in complexity in the software.

Its just simple logic.

Error detected in spec.

Case : Error found while reviewing UseCase/Function spec.
Actions:
        Rewrite paragraph in error.

Case : Error found during unit test.
Actions:
        Fix code.
        (Possibly) rewrite paragraph in spec.
        rerun unit test.

Case : Error found in integration test.
        Fix code.
        (possibly) rewrite paragraph in spec.
        rerun unit test.
        build new release.
        rerun integration test for whole system.

Case : Error found in UAT
        Fix code.
        (possibly) rewrite paragraph in spec.
        rerun unit test.
        build new release.
        rerun integration test for whole system.
        deploy release to UAT.
        rerun UAT tests.


Case : Error found in production
        Fix code.
        (possibly) rewrite paragraph in spec.
        rerun unit test.
        Build release.
        rerun integration test for whole system.
        deploy release to UAT.
        rerun UAT tests.
        deploy release to production.

As you can see the later the error is detected the more people are involved the more work has to be re-done and in any “normal” environment the paper work and bureaucracy increases exponentially once you hit UAT.

This is all without including the costs a business could incur due to an error in production software (lost sales, over ordering, hacked of customers etc. etc.)

I don’t think anyone has ever managed to write a non-trivial system which never had bugs in production, but, anything you can do to catch bugs early will save you time and effort in the long run.
Specification reviews, code reviews, extensive unit testing, using different coders to write the tests etc. etc. are all proven methods of catching bugs early.

I believe this is, and has always been, about risk management and economics. What is the cost of reducing the number of defects vs. the present value of the impact of future defects. The trajectory of the yellow bird being slightly off in Angry Birds does not equate the trajectory of a Tomahawk cruise missile being off. Software developers in either project can’t make decisions based on that table. In this regard, nothing changes.

The way I think this tends to work is via feedback, expensive bugs in the field cause companies to tighten their quality processes while no complaints from the field cause companies to relax it. So over time software development companies will tend to converge or oscillate around something that works for them (+/-). Code Complete may influence some initial values or may pull companies slightly one way or another. A company that spends too much effort removing defects that no one would notice is probably going to lose business to a competitor that has a more optimized approach. On the other hand, a company releasing buggy products will also go out of business.

Some relevant papers from a quick search (read the complete papers, do more research, and form your own opinion):

A Systematic Literature Review of Software Quality Cost Research (2011)

“While the community has thus developed a sound understanding of the
research domain’s structure, empirical validation is often lacking.
Only about a third of the analyzed articles presents a case study or
more extensive empirical results. This appears to be insufficient for
software quality cost research, which strongly relies on quantitative
data to generate new findings. There is thus a need for novel
approaches to gather quality cost data, as well as stronger
cooperation between industry and research to make such data
available.”

Evaluating the Cost of Software Quality (1998)

“Finally, we have seen that it is important to monitor software
conformance and nonconformance costs so that conformance policies can
be adjusted to reduce the total costs of software quality.”

The cost behavior of software defects (2004)

Abstract … “The current research attempts to update our knowledge of the way in which defects and the expense of correcting them (or alternatively, leaving them uncorrected) influences the final cost of software” … “uncorrected defects become exponentially more costly with each phase in which they are unresolved”

Test Coverage and Post-Verification Defects: A Multiple Case Study (2009)

“We also find that the test effort increases exponentially with test
coverage, but the reduction in field problems increases linearly with
test coverage. This suggests that for most projects the optimal levels
of coverage are likely to be well short of 100%.”

Bridge the Gap between Software Test Process and
Business Value: A Case Study (2009)

I cannot answer your first part of the question, as I simply have not checked. But I can formulate an answer to your second question, and perhaps hint at a possible answer to the first.

It should go without saying that some most important factors in the cost of fixing a bug, barring intrinsically hard to use development tools, are the intrinsic complexity of the product, and how well the user can understand that product.

Focusing for a second on code, under the assumption that code is typically written and maintained by developers capable of dealing with the intrinsic complexities of their code (which may not be entirely true and may deserve its own debate), I would dare suggest that of cricital importance in maintenance, and thus in fixing bugs, is the maintainers’ ability to understand said code.

The ability to understand code is greatly enhanced by the use of proven software engineering tools that are, unfortunately, mostly under- or improperly used. Using the right level of abstraction, modularity, enhancing module cohesion and reducing module coupling are critical tools in coping with complexity that need proper use. Coding to interfaces, or, in OOP, avoiding the over-use of inheritance over composition, packaging by feature, are some of the techniques that are often given insufficient attention in coding.

I believe that the realities of competition in the industry put a negative force on the employment of quality enhancing methods to developing software, keeping low the intrinsic quality of software as a measure of ongoing success.

Consequently, I believe that, in the industry, software tends to suffer more from bug fixing costs the bigger it grows. In such products, bugs become harder to fix over time because the system becomes harder to understand as it grows. The concerns introduced by each feature are overly coupled with other concerns, making understandability hard. Or, the right level of abstraction was not employed, making it hard for the maintainer to formulate a proper model of the system and reason about it. Lack of documentation certainly doesn’t help.

There are exceptions. I’m sure Google isn’t functioning at its pace without some solid practices upheld by stellar developers. And others are likely in the same situation. But for a majority of the software, I wouldn’t be surprised if the data did in fact confirm the claim in Code Complete.

Another Answer! This time to address the title question “Does software morhtodoligy rely on flawed data”.

The real answer is “there is no data”. As in there is no large reliable body of data on software projects there defects, successes time to market etc.

All attempts to gather such data have been underfunded, statistically flawed, or,, so specific to a particular project it is not possible to derive general conclusions from.

Furthermore I do not think there ever will be, the software development process is too subjective and slippery for strict measurement. The organizations best placed to gather such data (the large software houses and systems integrators) know in their hearts that any figures gathered from their performance would be deeply embarrassing.

The only organizations that publish numbers on the cost and success of software projects
are government departments, and, only then because they have to, and, yes these numbers are deeply embarrassing no matter how much they massage the figures.

So in conclusion all software studies are necessarily purely subjective because there is no real data on which to base an objective conclusion.