A Test Case Includes Expected Results

Documentation for a test case must include the detailed description of the expected correct results. If these are omitted, there is no way for the tester to determine whether the software succeeded or failed. Furthermore, a tester may assess an incorrect result as correct because there is always a subconscious desire to see a correct result. Even worse, a tester may assess a correct result as incorrect, causing a flurry of designer and programmer activity to "repair" the correct code.

Develop an organization standard for test plans that demands the documentation of expected intermediate and final test case results. Your quality assurance organization should confirm that all test plans conform to the standard.


Reference:
Myers, G., The Art of Software Testing, New York: John Wiley & Sons, 1979.

Use Black-box And White-box Testing

Black-box testing uses the specification of a component's external behavior as its only input. It is mandatory to determine if the software does what it is supposed to do and doesn't do what it is not supposed to do. White-box testing uses the code itself to generate test cases. Thus white-box testing might demand, for example, that a certain level of code coverage is obtained. Be aware; however, that even with both black-box and white-box testing, testing can make use of only a small subset of possible data values from the input domain.

To demonstrate how black-box and white-box testing complement each other, let's look at an example. Let's say a procedure's specification states that it should print the sum of all numbers in an input list. When programmed, it looks for one input of 213 and, if it finds it, sets the sum equal to zero. Since that was not in the specification, there is no way to find the error by black-box testing except by accident. White-box testing would demand that paths are more adequately tested, and thus would probably detect the "213" situation. By combining black-box and white-box, you maximize the effectiveness of testing. Neither one by itself does a through test.


Reference:
Dunn, R., Software Defect Removal, New York: McGraw-Hill, 1984.

Track Errors To Find More Errors

Conservative estimates indicate that, in large systems, approximately half of all software errors are found in 15% of the modules, and 80% of all software errors are found in 50% of the modules. More dramatic results from Gary Okimoto and Gerald Weinberg indicate that 80% of all errors were found in just 2% of the modules. Thus, when testing software, you might consider that, where you find errors, you will probably find more.

Maintain logs not only of how many errors are found per time period for the project, but also how many errors are found per module. When history shows a module to be highly error-prone, you are probably better off rewriting it from scratch, with an emphasis on simplicity rather than cleverness.


References:
Okimoto, G. and Weinberg, G., Quality Software Management, Vol. 1: Systems Thinking, New York: Dorset House, 1992.

Endres, A., "An Analysis of Errors and Their Causes in System Programs," IEEE Transactions on Software Engineering, June 1975.

A Successful Test Finds An Error

I have often heard a programmer or tester gleefully declare, "Great news! My test was successful. The program ran correctly." This is the wrong attitude to have when running a test. It also supports the position that programmers should never be the sole testers of their own software. A more constructive attitude is that one is testing to find errors. Thus, a successful test is one that detects an error. Look at the analogous situation with a medical test. Suppose you are feeling ill. The doctor sends a sample of your blood to a laboratory. A few days later, the doctor calls to tell you, "Great news! Your blood was normal." That is not great news. You are sick or you wouldn't have gone to the doctor. A successful blood test reports what's wrong with you. The software has [the potential for] bugs. A successful test reports how these bugs manifest themselves. When generating test plans, you should first select tests based on the likelihood that they will find faults. Only after those tests are written and ran should you write tests for the sake of code coverage.


Reference
Goodenough, J., and Gerhard, S., "Toward a Theory of Test Data Selection," IEEE Transactions on Software Engineering, June 1975.

The Obstacle Is the Way

The Obstacle Is the Way: The Timeless Art of Turning Trials into Triumph by Ryan Holiday
My rating: 5 of 5 stars

A perfect book to carry with you everywhere. The chapters are short and standalone so you can skip them or jump around or re-read certain chapters without it interrupting the flow of the book.

View all my reviews

Absence of Errors Fallacy

Though copious errors guarantee worthlessness, zero errors says nothing about the value of software. This is Gerald Weinberg's "Absence of Errors Fallacy." It really puts testing into perspective. It also puts all software engineering and management into perspective. The first part of the principle is obviously true; software with many errors is useless. The second part provides food for thought. It says that, no matter how hard you work to remove errors, you are wasting your time unless you are building the right system. Akao's Quality Function Deployment provides details on one method of ensuring that you are building the right system throughout the live cycle. A corollary to this principle is that all the formal methods, all the testing, and all the product assurance in the world won't help if you are building the wrong system.


Reference:
Weinberg, G., Quality Software Management, Vol. 1: Systems Thinking, New York: Dorset House, 1992.

Shakedown

Doctor Who: Shakedown by Terrance Dicks
My rating: 4 of 5 stars

Can you really have a Doctor Who story that is not about Doctor Who? Yes! In fact, this story is certainly better for having so little focus on the Doctor.

View all my reviews

Doctor Who: The Mind of Evil

Doctor Who: The Mind of Evil by Terrance Dicks
My rating: 3 of 5 stars

The BBC Audio is from the television soundtrack which leaves a fair amount of detail out. It is okay if you are a Doctor Who fan and know who the characters are but in no way should it be considered a standalone book.

View all my reviews

When to Rob a Bank

When to Rob a Bank by Steven D. Levitt
My rating: 3 of 5 stars

There are a few gems in here that should be revisited and reframed for today. There is also a lot that was only relevant in the past. Overall, it is worth reading but maybe with an eye to skipping anything that seems overly out-of-date.

View all my reviews

Wayne of Gotham

Wayne of Gotham by Tracy Hickman
My rating: 2 of 5 stars

While reading this book I felt that there was too much everything. The crazy people were too crazy. The rich people were too snobbish and blind. Batman was too paranoid.

And yet, after finishing and thinking back on it I cannot come up with a better way of handling the story. The crazy people only seem too crazy because fragmented and false memories really do leave people making no sense. And maybe rich people really are snobs and blind to the world beyond their money. And Batman being paranoid is not a new theme and why wouldn't it just get worse as he gets older?!

View all my reviews