Track Errors To Find More Errors

Conservative estimates indicate that, in large systems, approximately half of all software errors are found in 15% of the modules, and 80% of all software errors are found in 50% of the modules. More dramatic results from Gary Okimoto and Gerald Weinberg indicate that 80% of all errors were found in just 2% of the modules. Thus, when testing software, you might consider that, where you find errors, you will probably find more.

Maintain logs not only of how many errors are found per time period for the project, but also how many errors are found per module. When history shows a module to be highly error-prone, you are probably better off rewriting it from scratch, with an emphasis on simplicity rather than cleverness.


References:
Okimoto, G. and Weinberg, G., Quality Software Management, Vol. 1: Systems Thinking, New York: Dorset House, 1992.

Endres, A., "An Analysis of Errors and Their Causes in System Programs," IEEE Transactions on Software Engineering, June 1975.

A Successful Test Finds An Error

I have often heard a programmer or tester gleefully declare, "Great news! My test was successful. The program ran correctly." This is the wrong attitude to have when running a test. It also supports the position that programmers should never be the sole testers of their own software. A more constructive attitude is that one is testing to find errors. Thus, a successful test is one that detects an error. Look at the analogous situation with a medical test. Suppose you are feeling ill. The doctor sends a sample of your blood to a laboratory. A few days later, the doctor calls to tell you, "Great news! Your blood was normal." That is not great news. You are sick or you wouldn't have gone to the doctor. A successful blood test reports what's wrong with you. The software has [the potential for] bugs. A successful test reports how these bugs manifest themselves. When generating test plans, you should first select tests based on the likelihood that they will find faults. Only after those tests are written and ran should you write tests for the sake of code coverage.


Reference
Goodenough, J., and Gerhard, S., "Toward a Theory of Test Data Selection," IEEE Transactions on Software Engineering, June 1975.

The Obstacle Is the Way

The Obstacle Is the Way: The Timeless Art of Turning Trials into Triumph by Ryan Holiday
My rating: 5 of 5 stars

A perfect book to carry with you everywhere. The chapters are short and standalone so you can skip them or jump around or re-read certain chapters without it interrupting the flow of the book.

View all my reviews

Absence of Errors Fallacy

Though copious errors guarantee worthlessness, zero errors says nothing about the value of software. This is Gerald Weinberg's "Absence of Errors Fallacy." It really puts testing into perspective. It also puts all software engineering and management into perspective. The first part of the principle is obviously true; software with many errors is useless. The second part provides food for thought. It says that, no matter how hard you work to remove errors, you are wasting your time unless you are building the right system. Akao's Quality Function Deployment provides details on one method of ensuring that you are building the right system throughout the live cycle. A corollary to this principle is that all the formal methods, all the testing, and all the product assurance in the world won't help if you are building the wrong system.


Reference:
Weinberg, G., Quality Software Management, Vol. 1: Systems Thinking, New York: Dorset House, 1992.

Shakedown

Doctor Who: Shakedown by Terrance Dicks
My rating: 4 of 5 stars

Can you really have a Doctor Who story that is not about Doctor Who? Yes! In fact, this story is certainly better for having so little focus on the Doctor.

View all my reviews

Doctor Who: The Mind of Evil

Doctor Who: The Mind of Evil by Terrance Dicks
My rating: 3 of 5 stars

The BBC Audio is from the television soundtrack which leaves a fair amount of detail out. It is okay if you are a Doctor Who fan and know who the characters are but in no way should it be considered a standalone book.

View all my reviews

When to Rob a Bank

When to Rob a Bank by Steven D. Levitt
My rating: 3 of 5 stars

There are a few gems in here that should be revisited and reframed for today. There is also a lot that was only relevant in the past. Overall, it is worth reading but maybe with an eye to skipping anything that seems overly out-of-date.

View all my reviews

Wayne of Gotham

Wayne of Gotham by Tracy Hickman
My rating: 2 of 5 stars

While reading this book I felt that there was too much everything. The crazy people were too crazy. The rich people were too snobbish and blind. Batman was too paranoid.

And yet, after finishing and thinking back on it I cannot come up with a better way of handling the story. The crazy people only seem too crazy because fragmented and false memories really do leave people making no sense. And maybe rich people really are snobs and blind to the world beyond their money. And Batman being paranoid is not a new theme and why wouldn't it just get worse as he gets older?!

View all my reviews

Testing Exposes Presence Of Flaws

No matter how through, testing simply exposes the presence of flaws in a program; it cannot be used to verify the absence of flaws. It can increase your confidence that a program is correct, but it cannot prove correctness. To gain true correctness, one must use completely different processes, that is, correctness proofs.


Reference:
Dijkstra, E., "Notes on Structured Programming," in Structured Programming, Dahl, O., et al., Academic Press, 1972.

Don't Write Your Own Test Plans

Not only should you not be the sole tester of your own software, but you should also not be responsible for generating the test data, test scenarios, or test plans for your software. If you are, you may make the same mistakes in test generation that you made in software creation. For example, if you made a false assumption about the range of legal inputs when engineering the software, you would likely make the same assumption when generating test plans.

If you are a programmer and/or designer and your manager has asked you to write your test plans, I recommend you switch the test plan generation responsibility with a fellow programmer and/or designer. If you are a member of a requirements engineering team, with responsibility for system test generation as well, I recommend that members of your team subdivide the responsibilities so that no individual generates tests for requirements that she or he wrote.