Don't Integrate Before Unit Testing

Under normal circumstances components are separately unit-tested. As they pass their unit tests, a separate team integrates them into meaningful sets to exercise their interfaces. Components that have not been separately unit-tested are often integrated into the subsystem in a vein attempt to recapture a lost schedule. Such attempts actually cause more schedule delays. This is because a failure of a subsystem to satisfy an integration test plan may be caused now either by a fault in the interface or by a fault in the previously untested component. And much time is spent trying to determine which is the cause.

If you are managing a project, you can do a variety of things to avoid this situation. First and foremost is to develop an integration test plan early (for example, very soon after high-level design is complete). This plan should specify which components are most important to integrate first and in what order components may be integrated. Once you have written this down, allocate appropriate resources to coding and unit testing of specific high-priority components to ensure that integration testers don't spend an inordinate amount of time idle. Second, as it becomes evident that important components for integration testing are going to be unavailable as needed, have the integration testers start developing temporary scaffolding software to simulate the missing components.


Reference:
Dunn, R., Software Defect Removal, New York: McGraw-Hill, 1984.

Angle of Attack

I've been reading Flying Beyond the Stall: The X-31 and the Advent of Supermaneuverability by Douglas A. Joyce. Very quickly what struck me was that the issue that caused the X-31 to crash in 1995 was incorrect Angle of Attack (AoA) sensor data causing the software to override the pilot and crash the plane. Which was the same thing that happened to the Boeing 737 Max planes that crashed.

Maybe it is because I'm getting older but I'm getting more and more concerned about how the Tech industry forces older engineers out and tries to replace them with entry-level programmers. This isn't an natural issue of passing the baton. This is an issue of trying to replace highly skilled and experienced people with minimally skilled and no experience people.

So many of the software issues that we fight today have already been solved. If we just take the time to learn from the past then we could avoid repeating these mistakes.

Spaceman

Spaceman: An Astronaut's Unlikely Journey to Unlock the Secrets of the Universe by Mike Massimino
My rating: 5 of 5 stars

Wow! Who hasn't dreamed of spaceflight?! I loved the humanity that Massimino brings to the subject. This is a story that is told from his perspective but goes through many subjects that many people either relate to or want to know more about - childhood dreams, college, work, NASA, flight, spaceflight, parenthood, etc. It is wonderful to see those things through Massimino's eyes. Space exploration is awesome, and it needs people like Massimino to share the story; to bring us all along.

View all my reviews

Starman Jones

Starman Jones by Robert A. Heinlein
My rating: 5 of 5 stars

I loved the concern that Max had for his library book. That really hooked me into the book. After that it was a fast, fun ride.

View all my reviews

SuperFreakonomics

SuperFreakonomics: Global Cooling, Patriotic Prostitutes And Why Suicide Bombers Should Buy Life Insurance by Steven D. Levitt
My rating: 5 of 5 stars

Second time reading and still worth the read. It can be a little disheartening to know that some (many / maybe most) of the things discussed didn't make it into mainstream in the past 10 years. That is a lesson in itself and the concept is supported by a few of the stories in SuperFreakonomics.

View all my reviews

Achieve Effective Test Coverage

In spite of the fact that testing cannot prove correctness, it is still important to do a through job testing. Metrics exist to determine how thoroughly the code was exercised during test plan generation or test execution. These metrics are easy to use, and tools exist to monitor the coverage level of tests. Some examples include:

1. Statement coverage, which measures the percentage of statements that have been executed at least once.
2. Branch coverage, which measures the percentages of branches in a program that have been executed.
3. Path coverage, which measures how well the possible paths have been exercised.

Just remember that, although "effective" coverage is better than no coverage at all, do not fool yourself into thinking that the program is "correct" by any definition (see Testing Exposes Presence of Flaws).

Reference:

Weiser, M., Gannon, J., and McMullin, P., "Comparison of Structural Test Coverage Metrics," IEEE Software, March 1985.

Use Effective Test Completion Measures

Many projects proclaim the end of testing when they run out of time. This may make political sense, but it is irresponsible. During test planning, define a measure that can be used to determine when testing should be completed. If you have not met your goal when time runs out, you can still make the choice of whether to ship the product or slip the milestone, but at least you know whether you are shipping a quality product.

Two ideas for this effective measurement of test progress are:

1. Rate of new error detections per week.
2. After covertly seeding the software with known bugs (called bebugging), the percentage of these seeded bugs thus far found.

An ineffective measure of test progress is the percentage of test cases correctly passed (unless, of course, you know that the test cases superbly cover the requirements).

Reference:

Dunn, R., Software Defect Removal, New York: McGraw-Hill, 1984.

The Dictator's Handbook

The Dictator's Handbook: Why Bad Behavior is Almost Always Good Politics by Bruce Bueno de Mesquita
My rating: 5 of 5 stars

I was expecting stories about evil dictators, not a practical guide to politics. The real-world feel of this book made me continuously think of the book Physics for Future Presidents.

View all my reviews

Use McCabe Complexity Measure

Although many metrics are available to report the inherent complexity of software, none is as intuitive and as easy-to-use as Tom McCabe's cyclomatic number measure of testing complexity. While it is not absolutely foolproof, it results in fairly consistent predictions of testing difficulty. Simply draw a graph of your program, in which nodes correspond to sequences of instructions and arcs correspond to non-sequential flow of control. McCabe's metric is simply e-n+2p where e is the number of arcs, n is the number of nodes, and p is the number of independent graphs you are examining. This is so simple that there is really no excuse not to use it.

Use McCabe on each module to help assess unit testing complexity. Also, use it at the integration testing level where each procedure is a node and each invocation path is an arc to help assess integration testing complexity.


Reference:
McCabe, T., "A Complexity Measure," IEEE Transactions on Software Engineering, Dec 1976.

The Big Bang Theory Does Not Apply

As a project nears its delivery deadline and the software is not ready, desperation often takes over. Suppose the schedule called for two months of unit testing, two months of integration testing, and two months of software system testing. It is now one month from the scheduled delivery. Suppose 50% of the components have been unit tested. A back-of-the-envelope calculation indicates that you are five months behind schedule. You have two choices:

1. Admit the five-month delay to your customer: Ask for a postponement.
2. Put all the components together (including the 50% not yet unit tested) and hope for the best.

In the first case, you are admitting defeat, perhaps prematurely. In the eyes of your managers, you might be giving up before you've done everything in your power to overcome the problem. In the second case, there might be a chance that, when you put it all together, it will work and you'll be back on schedule. Project managers often succumb to the latter because it looks like they are trying everything before admitting defeat. Unfortunately, this will probably add six more months to your schedule since you'll be trying to retrofit quality. You cannot save time by omitting unit and integration testing.

Reference:

Weinberg, G., Quality Software Management, Volume 1: Systems Thinking, New York: Dorset House, 1992.