Use Effective Test Completion Measures

Many projects proclaim the end of testing when they run out of time. This may make political sense, but it is irresponsible. During test planning, define a measure that can be used to determine when testing should be completed. If you have not met your goal when time runs out, you can still make the choice of whether to ship the product or slip the milestone, but at least you know whether you are shipping a quality product.

Two ideas for this effective measurement of test progress are:

1. Rate of new error detections per week.
2. After covertly seeding the software with known bugs (called bebugging), the percentage of these seeded bugs thus far found.

An ineffective measure of test progress is the percentage of test cases correctly passed (unless, of course, you know that the test cases superbly cover the requirements).

Reference:

Dunn, R., Software Defect Removal, New York: McGraw-Hill, 1984.

The Dictator's Handbook

The Dictator's Handbook: Why Bad Behavior is Almost Always Good Politics by Bruce Bueno de Mesquita
My rating: 5 of 5 stars

I was expecting stories about evil dictators, not a practical guide to politics. The real-world feel of this book made me continuously think of the book Physics for Future Presidents.

View all my reviews

Use McCabe Complexity Measure

Although many metrics are available to report the inherent complexity of software, none is as intuitive and as easy-to-use as Tom McCabe's cyclomatic number measure of testing complexity. While it is not absolutely foolproof, it results in fairly consistent predictions of testing difficulty. Simply draw a graph of your program, in which nodes correspond to sequences of instructions and arcs correspond to non-sequential flow of control. McCabe's metric is simply e-n+2p where e is the number of arcs, n is the number of nodes, and p is the number of independent graphs you are examining. This is so simple that there is really no excuse not to use it.

Use McCabe on each module to help assess unit testing complexity. Also, use it at the integration testing level where each procedure is a node and each invocation path is an arc to help assess integration testing complexity.


Reference:
McCabe, T., "A Complexity Measure," IEEE Transactions on Software Engineering, Dec 1976.

The Big Bang Theory Does Not Apply

As a project nears its delivery deadline and the software is not ready, desperation often takes over. Suppose the schedule called for two months of unit testing, two months of integration testing, and two months of software system testing. It is now one month from the scheduled delivery. Suppose 50% of the components have been unit tested. A back-of-the-envelope calculation indicates that you are five months behind schedule. You have two choices:

1. Admit the five-month delay to your customer: Ask for a postponement.
2. Put all the components together (including the 50% not yet unit tested) and hope for the best.

In the first case, you are admitting defeat, perhaps prematurely. In the eyes of your managers, you might be giving up before you've done everything in your power to overcome the problem. In the second case, there might be a chance that, when you put it all together, it will work and you'll be back on schedule. Project managers often succumb to the latter because it looks like they are trying everything before admitting defeat. Unfortunately, this will probably add six more months to your schedule since you'll be trying to retrofit quality. You cannot save time by omitting unit and integration testing.

Reference:

Weinberg, G., Quality Software Management, Volume 1: Systems Thinking, New York: Dorset House, 1992.

Always Stress Test

Software design often behaves just fine when confronted with "normal" loads of inputs of stimuli. The true test of software is whether it can stay operational when faced with severe loads. These severe loads are often stated in the requirements as "maximum of x simultaneous widgets" or "maximum of x new widget arrivals per hour."

If the requirements state that the software shall handle up to x widgets per hour, you must verify that the software can do this. In fact, not only should you test that it handles x widgets, you should also subject the software to x+1 or x+2 (or more) widgets to see what happens and determine when environment violates "acceptable" behavior. After all, the system may not be able to control its environment, and you do not want the software to crash when the environment "misbehaves" in an unexpected manner.


Reference:
Myers, G., The Art of Software Testing, New York: John Wiley & Sons, 1979.

Test Invalid Input

It is natural and common to produce test cases for as many acceptable input scenarios as possible. What is equally important -- but also uncommon -- is to produce an extensive set of test cases for all invalid or unexpected input.

For a simple example, let us say we are writing a program to sort lists of integers in the range of 0 to 100. Test lists should include some negative numbers, some nonintegral numbers, some alphabetic data, some null entries, and so on.


Reference:
Myers, G., The Art of Software Testing, New York: John Wiley & Sons, 1979.

Range: Why Generalists Triumph in a Specialized World

Range: Why Generalists Triumph in a Specialized World by David Epstein
My rating: 5 of 5 stars

I loved it, but I'm biased since I have always been more of a generalist. :-D
Anyway, Range is packed with information. I found myself stopping a lot just to ponder what I had just read. I loved all the stories that were referenced as examples and food for thought.

The "Learning to Drop Your Familiar Tools" chapter was just brilliant. The description of the smokejumpers who just couldn't fathom dropping their tools is poignant and heartbreaking. Even though there are a ton of books on NASA culture and the Challenger and Columbia disasters, I'm certain that I would read a book on this topic if David Epstein wrote it.

View all my reviews

Chain of Communication

The following is paraphrased for my benefit. It is taken from an interview with Rex Geveden documented in the book Range: Why Generalist Triumph in a Specialized World by David Epstein.

I expect disagreement with my decisions at the time we're trying to make decisions, and that's a sign of organizational health. After the decisions are made, we want compliance and support, but we have permission to fight a little bit about those things in a professional way. There is a difference between the chain of command and the chain of communication, and that the difference represents a healthy cross-pressure.

I'm going to communicate with all levels of the organization from the CEO all the way down to the interns, and you can't feel suspicious or paranoid about that. I will not intercept your decisions that belong in your chain of command, but I will give and receive information anywhere in the organization, at any time. I just can't get enough understanding of the organization from listening to the voices at the top.

Problem Solving

Beware of the person who knows all the answers:

Suppose two politicians are running for president, and one goes through the farm section and is asked, "What are you going to do about the farm question?" And he knows right away - bang, bang, bang. Now he goes to the next campaigner who comes through. "What are you going to do on the farm problem?" "Well, I don't know. I used to be a general, and I don't know anything about farming. But it seems to me it must be a very difficult problem, because for twelve, fifteen, twenty years people have been struggling with it, and people say that they know how to solve the farm problem. And it must be a hard problem. So the way I intend to solve the farm problem is to gather around me a lot of people who know something about it, to look at all the experience that we have had with this problem before, to take a certain amount of time at it, and then to come to some conclusion in a reasonable way about it. Now, I can't tell you ahead of time what solution, but I can give you some of the principles I'll try to use - not to make things difficult for individual farmers, if there are any special problems we will have to have some way to take care of them," etc., etc., etc.

Now such a man would never get anywhere in this country, I think. It's never been tried, anyway. This is in the attitude of mind of the populace, that they have to have an answer and that a man who gives an answer is better than a man who gives no answer, when the real fact of the matter is, in most cases, it is the other way around. And the result of this of course is that the politician must give an answer. And the result of this is that political promises can never be kept. It is a mechanical fact; it is impossible. The result of that is that nobody believes campaign promises. And the result of that is a general disparaging of politics, a general lack of respect for the people who are trying to solve problems, and so forth. It's all generated from the very beginning (maybe - this is a simple analysis). It's all generated, maybe, by the fact that the attitude of the populace is to try to find the answer instead of trying to find a man who has a way of getting at the answer.

 —Richard Feynman

And make sure that person is not you.

A Test Case Includes Expected Results

Documentation for a test case must include the detailed description of the expected correct results. If these are omitted, there is no way for the tester to determine whether the software succeeded or failed. Furthermore, a tester may assess an incorrect result as correct because there is always a subconscious desire to see a correct result. Even worse, a tester may assess a correct result as incorrect, causing a flurry of designer and programmer activity to "repair" the correct code.

Develop an organization standard for test plans that demands the documentation of expected intermediate and final test case results. Your quality assurance organization should confirm that all test plans conform to the standard.


Reference:
Myers, G., The Art of Software Testing, New York: John Wiley & Sons, 1979.