Always Stress Test

Software design often behaves just fine when confronted with "normal" loads of inputs of stimuli. The true test of software is whether it can stay operational when faced with severe loads. These severe loads are often stated in the requirements as "maximum of x simultaneous widgets" or "maximum of x new widget arrivals per hour."

If the requirements state that the software shall handle up to x widgets per hour, you must verify that the software can do this. In fact, not only should you test that it handles x widgets, you should also subject the software to x+1 or x+2 (or more) widgets to see what happens and determine when environment violates "acceptable" behavior. After all, the system may not be able to control its environment, and you do not want the software to crash when the environment "misbehaves" in an unexpected manner.


Reference:
Myers, G., The Art of Software Testing, New York: John Wiley & Sons, 1979.

Test Invalid Input

It is natural and common to produce test cases for as many acceptable input scenarios as possible. What is equally important -- but also uncommon -- is to produce an extensive set of test cases for all invalid or unexpected input.

For a simple example, let us say we are writing a program to sort lists of integers in the range of 0 to 100. Test lists should include some negative numbers, some nonintegral numbers, some alphabetic data, some null entries, and so on.


Reference:
Myers, G., The Art of Software Testing, New York: John Wiley & Sons, 1979.

Range: Why Generalists Triumph in a Specialized World

Range: Why Generalists Triumph in a Specialized World by David Epstein
My rating: 5 of 5 stars

I loved it, but I'm biased since I have always been more of a generalist. :-D
Anyway, Range is packed with information. I found myself stopping a lot just to ponder what I had just read. I loved all the stories that were referenced as examples and food for thought.

The "Learning to Drop Your Familiar Tools" chapter was just brilliant. The description of the smokejumpers who just couldn't fathom dropping their tools is poignant and heartbreaking. Even though there are a ton of books on NASA culture and the Challenger and Columbia disasters, I'm certain that I would read a book on this topic if David Epstein wrote it.

View all my reviews

Chain of Communication

The following is paraphrased for my benefit. It is taken from an interview with Rex Geveden documented in the book Range: Why Generalist Triumph in a Specialized World by David Epstein.

I expect disagreement with my decisions at the time we're trying to make decisions, and that's a sign of organizational health. After the decisions are made, we want compliance and support, but we have permission to fight a little bit about those things in a professional way. There is a difference between the chain of command and the chain of communication, and that the difference represents a healthy cross-pressure.

I'm going to communicate with all levels of the organization from the CEO all the way down to the interns, and you can't feel suspicious or paranoid about that. I will not intercept your decisions that belong in your chain of command, but I will give and receive information anywhere in the organization, at any time. I just can't get enough understanding of the organization from listening to the voices at the top.

Problem Solving

Beware of the person who knows all the answers:

Suppose two politicians are running for president, and one goes through the farm section and is asked, "What are you going to do about the farm question?" And he knows right away - bang, bang, bang. Now he goes to the next campaigner who comes through. "What are you going to do on the farm problem?" "Well, I don't know. I used to be a general, and I don't know anything about farming. But it seems to me it must be a very difficult problem, because for twelve, fifteen, twenty years people have been struggling with it, and people say that they know how to solve the farm problem. And it must be a hard problem. So the way I intend to solve the farm problem is to gather around me a lot of people who know something about it, to look at all the experience that we have had with this problem before, to take a certain amount of time at it, and then to come to some conclusion in a reasonable way about it. Now, I can't tell you ahead of time what solution, but I can give you some of the principles I'll try to use - not to make things difficult for individual farmers, if there are any special problems we will have to have some way to take care of them," etc., etc., etc.

Now such a man would never get anywhere in this country, I think. It's never been tried, anyway. This is in the attitude of mind of the populace, that they have to have an answer and that a man who gives an answer is better than a man who gives no answer, when the real fact of the matter is, in most cases, it is the other way around. And the result of this of course is that the politician must give an answer. And the result of this is that political promises can never be kept. It is a mechanical fact; it is impossible. The result of that is that nobody believes campaign promises. And the result of that is a general disparaging of politics, a general lack of respect for the people who are trying to solve problems, and so forth. It's all generated from the very beginning (maybe - this is a simple analysis). It's all generated, maybe, by the fact that the attitude of the populace is to try to find the answer instead of trying to find a man who has a way of getting at the answer.

 —Richard Feynman

And make sure that person is not you.

A Test Case Includes Expected Results

Documentation for a test case must include the detailed description of the expected correct results. If these are omitted, there is no way for the tester to determine whether the software succeeded or failed. Furthermore, a tester may assess an incorrect result as correct because there is always a subconscious desire to see a correct result. Even worse, a tester may assess a correct result as incorrect, causing a flurry of designer and programmer activity to "repair" the correct code.

Develop an organization standard for test plans that demands the documentation of expected intermediate and final test case results. Your quality assurance organization should confirm that all test plans conform to the standard.


Reference:
Myers, G., The Art of Software Testing, New York: John Wiley & Sons, 1979.

Use Black-box And White-box Testing

Black-box testing uses the specification of a component's external behavior as its only input. It is mandatory to determine if the software does what it is supposed to do and doesn't do what it is not supposed to do. White-box testing uses the code itself to generate test cases. Thus white-box testing might demand, for example, that a certain level of code coverage is obtained. Be aware; however, that even with both black-box and white-box testing, testing can make use of only a small subset of possible data values from the input domain.

To demonstrate how black-box and white-box testing complement each other, let's look at an example. Let's say a procedure's specification states that it should print the sum of all numbers in an input list. When programmed, it looks for one input of 213 and, if it finds it, sets the sum equal to zero. Since that was not in the specification, there is no way to find the error by black-box testing except by accident. White-box testing would demand that paths are more adequately tested, and thus would probably detect the "213" situation. By combining black-box and white-box, you maximize the effectiveness of testing. Neither one by itself does a through test.


Reference:
Dunn, R., Software Defect Removal, New York: McGraw-Hill, 1984.

Track Errors To Find More Errors

Conservative estimates indicate that, in large systems, approximately half of all software errors are found in 15% of the modules, and 80% of all software errors are found in 50% of the modules. More dramatic results from Gary Okimoto and Gerald Weinberg indicate that 80% of all errors were found in just 2% of the modules. Thus, when testing software, you might consider that, where you find errors, you will probably find more.

Maintain logs not only of how many errors are found per time period for the project, but also how many errors are found per module. When history shows a module to be highly error-prone, you are probably better off rewriting it from scratch, with an emphasis on simplicity rather than cleverness.


References:
Okimoto, G. and Weinberg, G., Quality Software Management, Vol. 1: Systems Thinking, New York: Dorset House, 1992.

Endres, A., "An Analysis of Errors and Their Causes in System Programs," IEEE Transactions on Software Engineering, June 1975.

A Successful Test Finds An Error

I have often heard a programmer or tester gleefully declare, "Great news! My test was successful. The program ran correctly." This is the wrong attitude to have when running a test. It also supports the position that programmers should never be the sole testers of their own software. A more constructive attitude is that one is testing to find errors. Thus, a successful test is one that detects an error. Look at the analogous situation with a medical test. Suppose you are feeling ill. The doctor sends a sample of your blood to a laboratory. A few days later, the doctor calls to tell you, "Great news! Your blood was normal." That is not great news. You are sick or you wouldn't have gone to the doctor. A successful blood test reports what's wrong with you. The software has [the potential for] bugs. A successful test reports how these bugs manifest themselves. When generating test plans, you should first select tests based on the likelihood that they will find faults. Only after those tests are written and ran should you write tests for the sake of code coverage.


Reference
Goodenough, J., and Gerhard, S., "Toward a Theory of Test Data Selection," IEEE Transactions on Software Engineering, June 1975.

The Obstacle Is the Way

The Obstacle Is the Way: The Timeless Art of Turning Trials into Triumph by Ryan Holiday
My rating: 5 of 5 stars

A perfect book to carry with you everywhere. The chapters are short and standalone so you can skip them or jump around or re-read certain chapters without it interrupting the flow of the book.

View all my reviews