Problem Solving

Beware of the person who knows all the answers:

Suppose two politicians are running for president, and one goes through the farm section and is asked, "What are you going to do about the farm question?" And he knows right away - bang, bang, bang. Now he goes to the next campaigner who comes through. "What are you going to do on the farm problem?" "Well, I don't know. I used to be a general, and I don't know anything about farming. But it seems to me it must be a very difficult problem, because for twelve, fifteen, twenty years people have been struggling with it, and people say that they know how to solve the farm problem. And it must be a hard problem. So the way I intend to solve the farm problem is to gather around me a lot of people who know something about it, to look at all the experience that we have had with this problem before, to take a certain amount of time at it, and then to come to some conclusion in a reasonable way about it. Now, I can't tell you ahead of time what solution, but I can give you some of the principles I'll try to use - not to make things difficult for individual farmers, if there are any special problems we will have to have some way to take care of them," etc., etc., etc.

Now such a man would never get anywhere in this country, I think. It's never been tried, anyway. This is in the attitude of mind of the populace, that they have to have an answer and that a man who gives an answer is better than a man who gives no answer, when the real fact of the matter is, in most cases, it is the other way around. And the result of this of course is that the politician must give an answer. And the result of this is that political promises can never be kept. It is a mechanical fact; it is impossible. The result of that is that nobody believes campaign promises. And the result of that is a general disparaging of politics, a general lack of respect for the people who are trying to solve problems, and so forth. It's all generated from the very beginning (maybe - this is a simple analysis). It's all generated, maybe, by the fact that the attitude of the populace is to try to find the answer instead of trying to find a man who has a way of getting at the answer.

 —Richard Feynman

And make sure that person is not you.

A Test Case Includes Expected Results

Documentation for a test case must include the detailed description of the expected correct results. If these are omitted, there is no way for the tester to determine whether the software succeeded or failed. Furthermore, a tester may assess an incorrect result as correct because there is always a subconscious desire to see a correct result. Even worse, a tester may assess a correct result as incorrect, causing a flurry of designer and programmer activity to "repair" the correct code.

Develop an organization standard for test plans that demands the documentation of expected intermediate and final test case results. Your quality assurance organization should confirm that all test plans conform to the standard.


Reference:
Myers, G., The Art of Software Testing, New York: John Wiley & Sons, 1979.

Use Black-box And White-box Testing

Black-box testing uses the specification of a component's external behavior as its only input. It is mandatory to determine if the software does what it is supposed to do and doesn't do what it is not supposed to do. White-box testing uses the code itself to generate test cases. Thus white-box testing might demand, for example, that a certain level of code coverage is obtained. Be aware; however, that even with both black-box and white-box testing, testing can make use of only a small subset of possible data values from the input domain.

To demonstrate how black-box and white-box testing complement each other, let's look at an example. Let's say a procedure's specification states that it should print the sum of all numbers in an input list. When programmed, it looks for one input of 213 and, if it finds it, sets the sum equal to zero. Since that was not in the specification, there is no way to find the error by black-box testing except by accident. White-box testing would demand that paths are more adequately tested, and thus would probably detect the "213" situation. By combining black-box and white-box, you maximize the effectiveness of testing. Neither one by itself does a through test.


Reference:
Dunn, R., Software Defect Removal, New York: McGraw-Hill, 1984.

Track Errors To Find More Errors

Conservative estimates indicate that, in large systems, approximately half of all software errors are found in 15% of the modules, and 80% of all software errors are found in 50% of the modules. More dramatic results from Gary Okimoto and Gerald Weinberg indicate that 80% of all errors were found in just 2% of the modules. Thus, when testing software, you might consider that, where you find errors, you will probably find more.

Maintain logs not only of how many errors are found per time period for the project, but also how many errors are found per module. When history shows a module to be highly error-prone, you are probably better off rewriting it from scratch, with an emphasis on simplicity rather than cleverness.


References:
Okimoto, G. and Weinberg, G., Quality Software Management, Vol. 1: Systems Thinking, New York: Dorset House, 1992.

Endres, A., "An Analysis of Errors and Their Causes in System Programs," IEEE Transactions on Software Engineering, June 1975.

A Successful Test Finds An Error

I have often heard a programmer or tester gleefully declare, "Great news! My test was successful. The program ran correctly." This is the wrong attitude to have when running a test. It also supports the position that programmers should never be the sole testers of their own software. A more constructive attitude is that one is testing to find errors. Thus, a successful test is one that detects an error. Look at the analogous situation with a medical test. Suppose you are feeling ill. The doctor sends a sample of your blood to a laboratory. A few days later, the doctor calls to tell you, "Great news! Your blood was normal." That is not great news. You are sick or you wouldn't have gone to the doctor. A successful blood test reports what's wrong with you. The software has [the potential for] bugs. A successful test reports how these bugs manifest themselves. When generating test plans, you should first select tests based on the likelihood that they will find faults. Only after those tests are written and ran should you write tests for the sake of code coverage.


Reference
Goodenough, J., and Gerhard, S., "Toward a Theory of Test Data Selection," IEEE Transactions on Software Engineering, June 1975.

The Obstacle Is the Way

The Obstacle Is the Way: The Timeless Art of Turning Trials into Triumph by Ryan Holiday
My rating: 5 of 5 stars

A perfect book to carry with you everywhere. The chapters are short and standalone so you can skip them or jump around or re-read certain chapters without it interrupting the flow of the book.

View all my reviews

Absence of Errors Fallacy

Though copious errors guarantee worthlessness, zero errors says nothing about the value of software. This is Gerald Weinberg's "Absence of Errors Fallacy." It really puts testing into perspective. It also puts all software engineering and management into perspective. The first part of the principle is obviously true; software with many errors is useless. The second part provides food for thought. It says that, no matter how hard you work to remove errors, you are wasting your time unless you are building the right system. Akao's Quality Function Deployment provides details on one method of ensuring that you are building the right system throughout the live cycle. A corollary to this principle is that all the formal methods, all the testing, and all the product assurance in the world won't help if you are building the wrong system.


Reference:
Weinberg, G., Quality Software Management, Vol. 1: Systems Thinking, New York: Dorset House, 1992.

Shakedown

Doctor Who: Shakedown by Terrance Dicks
My rating: 4 of 5 stars

Can you really have a Doctor Who story that is not about Doctor Who? Yes! In fact, this story is certainly better for having so little focus on the Doctor.

View all my reviews

Doctor Who: The Mind of Evil

Doctor Who: The Mind of Evil by Terrance Dicks
My rating: 3 of 5 stars

The BBC Audio is from the television soundtrack which leaves a fair amount of detail out. It is okay if you are a Doctor Who fan and know who the characters are but in no way should it be considered a standalone book.

View all my reviews

When to Rob a Bank

When to Rob a Bank by Steven D. Levitt
My rating: 3 of 5 stars

There are a few gems in here that should be revisited and reframed for today. There is also a lot that was only relevant in the past. Overall, it is worth reading but maybe with an eye to skipping anything that seems overly out-of-date.

View all my reviews