Back to Test Basics

by Gary K. Evans

In last month's newsletter I explored the topic of software safety, and how the scale of software failures has escalated from humorous inconveniences to life-threatening gravity. In this newsletter I will discuss some specific ideas for software testing, so we hopefully can minimize the damage our software might cause.

First, the bad news. For an arbitrary program we can never say:

• This software has no errors;
• This software works only as intended;
• This software is safe.

Now, some good news. But it requires understanding what software testing really means, and I am always disturbed that many Quality Assurance people do not subscribe to these definitions.

In his classic The Art of Software Testing, now in a completely updated 2nd edition, Glenford J. Myers offers the following rock-solid principles:

• Testing is the process of executing a program with the intent of finding errors.
• A good test is one that has a high probability of detecting an as-yet undiscovered error.
• A successful test case is one that detects an as-yet undiscovered error.

I hope these principles give you reason to pause a bit. What Myers is saying is: if you just show that a program's functions work, you have not yet tested the program. A successful test is one that breaks the program, not shows the software works as intended.

Let's explore just a little, simple theory about software. Every program takes inputs, processes those inputs, and produces outputs. So we can think of a program as a transformer: transforming those inputs to outputs.

What can we say about inputs and outputs? The figure below shows that inputs and outputs are either valid, or invalid, for the program. This means we have 4 mappings, or scenarios, we must consider.

Scenario A denotes the program receiving valid inputs, and producing valid outputs. This is the desirable case, but signifies a test that verifies the program operates as intended. Lets assume the program returns the square root of an integer. In Scenario A I pass "16" to the program, and it correctly returns "4".

Scenario B denotes the program receiving invalid inputs, and producing valid outputs (e.g., an error or exception message). This again is a desirable case to verify the program operates as intended. I pass "-3" to the program and it correctly returns "Input integer must be non-negative".

Testing gets interesting as we move to the right-hand column. In Scenario C we provide valid inputs, but the program produces invalid outputs. We call this a "bug" - e.g., we pass "16" to our program and it returns "4.398".

In Scenario D we pass an invalid input, and receive an invalid output. This used to be called "GIGO" - Garbage In, Garbage Out. (Interesting that we do not hear this phrase very often anymore.) This scenario describes passing "-9" to our program and it returns "3", instead of an error message.

Now, where is the good news in this? It is in our ability, if we so choose, to approach our software using these simple categories. For developers, we should appropriate the concepts in Bertrand Meyer's Design By Contract. Design By Contract, which is actually implemented in the syntax of his language Eiffel, incorporates the specification of pre-conditions and post-conditions for each method. Pre-conditions are expressions that must be true for the method to perform its function correctly - i.e., the valid input conditions. Post-conditions are boolean conditions describing the result of the method's execution - i.e., the valid output responses.

For testers, this approach yields a method for identifying test cases for both black box and white box testing. When a tester can completely characterize the valid inputs to a method, the tester has also characterized the inverse set of all the invalid inputs.

There is much more detail we could discuss, such as the Cartesian product set defined by multiple inputs to a program or method, or how to efficiently choose representative subsets of test data from test sets that are normally staggeringly huge. I encourage developers, testers and business system analysts to incorporate thinking in terms of valid and invalid inputs and outputs. And read as much as you can about test basics and techniques. Here are some books I can easily recommend to assist you:

• The Art of Software Testing, 2nd edition. Glenford J. Myers, et. al., 2004.
• Testing Computer Software, 2nd edition. Cem Kaner, et. al., 1999.
• Agile Testing, L. Crispin & J. Gregory, 2009.

Have a productive month,

    Gary K. Evans