A Test Case Includes Expected Results

Documentation for a test case must include the detailed description of the expected correct results. If these are omitted, there is no way for the tester to determine whether the software succeeded or failed. Furthermore, a tester may assess an incorrect result as correct because there is always a subconscious desire to see a correct result. Even worse, a tester may assess a correct result as incorrect, causing a flurry of designer and programmer activity to "repair" the correct code.

Develop an organization standard for test plans that demands the documentation of expected intermediate and final test case results. Your quality assurance organization should confirm that all test plans conform to the standard.


Reference:
Myers, G., The Art of Software Testing, New York: John Wiley & Sons, 1979.

Use Black-box And White-box Testing

Black-box testing uses the specification of a component's external behavior as its only input. It is mandatory to determine if the software does what it is supposed to do and doesn't do what it is not supposed to do. White-box testing uses the code itself to generate test cases. Thus white-box testing might demand, for example, that a certain level of code coverage is obtained. Be aware; however, that even with both black-box and white-box testing, testing can make use of only a small subset of possible data values from the input domain.

To demonstrate how black-box and white-box testing complement each other, let's look at an example. Let's say a procedure's specification states that it should print the sum of all numbers in an input list. When programmed, it looks for one input of 213 and, if it finds it, sets the sum equal to zero. Since that was not in the specification, there is no way to find the error by black-box testing except by accident. White-box testing would demand that paths are more adequately tested, and thus would probably detect the "213" situation. By combining black-box and white-box, you maximize the effectiveness of testing. Neither one by itself does a through test.


Reference:
Dunn, R., Software Defect Removal, New York: McGraw-Hill, 1984.

Track Errors To Find More Errors

Conservative estimates indicate that, in large systems, approximately half of all software errors are found in 15% of the modules, and 80% of all software errors are found in 50% of the modules. More dramatic results from Gary Okimoto and Gerald Weinberg indicate that 80% of all errors were found in just 2% of the modules. Thus, when testing software, you might consider that, where you find errors, you will probably find more.

Maintain logs not only of how many errors are found per time period for the project, but also how many errors are found per module. When history shows a module to be highly error-prone, you are probably better off rewriting it from scratch, with an emphasis on simplicity rather than cleverness.


References:
Okimoto, G. and Weinberg, G., Quality Software Management, Vol. 1: Systems Thinking, New York: Dorset House, 1992.

Endres, A., "An Analysis of Errors and Their Causes in System Programs," IEEE Transactions on Software Engineering, June 1975.

A Successful Test Finds An Error

I have often heard a programmer or tester gleefully declare, "Great news! My test was successful. The program ran correctly." This is the wrong attitude to have when running a test. It also supports the position that programmers should never be the sole testers of their own software. A more constructive attitude is that one is testing to find errors. Thus, a successful test is one that detects an error. Look at the analogous situation with a medical test. Suppose you are feeling ill. The doctor sends a sample of your blood to a laboratory. A few days later, the doctor calls to tell you, "Great news! Your blood was normal." That is not great news. You are sick or you wouldn't have gone to the doctor. A successful blood test reports what's wrong with you. The software has [the potential for] bugs. A successful test reports how these bugs manifest themselves. When generating test plans, you should first select tests based on the likelihood that they will find faults. Only after those tests are written and ran should you write tests for the sake of code coverage.


Reference
Goodenough, J., and Gerhard, S., "Toward a Theory of Test Data Selection," IEEE Transactions on Software Engineering, June 1975.

The Obstacle Is the Way

The Obstacle Is the Way: The Timeless Art of Turning Trials into Triumph by Ryan Holiday
My rating: 5 of 5 stars

A perfect book to carry with you everywhere. The chapters are short and standalone so you can skip them or jump around or re-read certain chapters without it interrupting the flow of the book.

View all my reviews