Home
Up
Services
Training
Articles
About Us
Contact Us

Driven to Test for Success

Glenford J. Myers wrote a classic software testing book, The Art of Software Testing (1979), in which he offered timeless insights on software testing. Here are three definitions from this book:

  1. "Testing is the process of executing a program with the intent of finding errors."
  2. "A good test case is one that has a high probability of detecting an as-yet undiscovered error."
  3. "A successful test is one that detects an as-yet undiscovered error."

I hope quote #3 gives you pause: the converse of this would say that an unsuccessful test is one that does not detect an error. This is quite at odds with the testing philosophy I see in many of my clients, who believe that a test that does not detect an error must indicate the program is doing what it is supposed to do. This is illogical, of course, since if you never test a program you also will never detect any errors. My consulting experience also indicates that the single weakest area of software testing is in unit testing - the testing done by developers on their own code.

Test Driven Development (TDD), also known as Test First Programming, is a way to unit test software in an efficient and continuous manner. For developers, TDD is an opportunity to become a better programmer and designer. For managers and project management, TDD gives you direct visibility into your team's progress, and the quality status of their code base.

Someone rightly wrote, "A test worth writing is a test worth saving". So, TDD is an automated practice that assures you maintain your investment in your unit tests, eliminates human variation in conducting unit tests, and redirects developers' resistance to unit testing.

It is axiomatic in the waterfall software development process that developers cannot test their own code. The reason for this is simple: after a developer constructs the project code, they cannot easily turn around and try to tear down that same code by testing it - i.e. trying to break it. They have invested too much intellectual capital into building the software. Asking them to break it is impractical. This resistance is a psychological barrier to quality, not a technical obstacle. But the result of insufficient, or non-existent, unit testing is that local logic and coding errors are passed downstream to system test, a qualitatively different form of testing that can find unit defects only if they surface during system and functional testing. Worse, when QA cannot find these errors, or runs out of time to perform testing, those errors may be found by customers.

TDD improves this because it changes the relationship between coding and testing. To understand the immeasurable value of TDD we must re-examine what testing is, and is not. Too many organizations assume that the goal of testing is to assure the software performs as it should. In reality, assurance that the software does what it is supposed to do is the goal of development, not testing.

In the traditional, waterfall approach to unit testing, requirements specify the goals of the project, production code is written to satisfy those requirements, then unit tests are defined and executed manually to verify that the production code performs as expected. The following graphic illustrates these relationships:

In TDD, requirements still specify the goals of the project. But in TDD we first write executable tests that quantify the requirements. Then, and only then, do we write the production code that satisfies those tests. The graphic below illustrates this:

You can see that TDD reverses the relationship between the coding and unit test activities. It also reverses the psychological constraints on the developer. In the traditional approach the developer performs a constructive activity of building the code, then performs a destructive activity of attempting to break the same code. In TDD the developer performs a constructive activity of building the test code based on quantifiable properties of the requirements, then performs another constructive activity of building the production code that enables the test to succeed. Two positives, no negatives.

The practice of TDD is deceptively simple. It only requires the developer to do the following steps:

  1. Identify a requirement that is measurable, and can be satisfied by executable code.
  2. Write executable code that is the test of that requirement.
  3. Run the test code: it must fail until you...
  4. ...Write the production code that will cause the test to succeed.

Why must the test fail in step 3 above? This is the exquisite simplicity of TDD, as well as the only puzzling concept to newcomers.

In step 2 you write the executable test code, but you do not write any production code yet. In step 3 you run the test code and it has to fail for two reasons:

  1. The production code that the test is attempting to invoke has not yet been written, and
  2. By forcing the failure, you are assuring yourself that when production code is written or changed, and that code does not satisfy the test, then the test code will fail, alerting you to a defect in the production code. If you never force the test to fail, you can never be sure that in the futurethe test will signal failure in the production code.

Here is a brief and very simple example to get you interested in TDD. Let us consider that we are developing an order entry system. One requirement in such a project would be

        - An Order shall be able to compute its total cost.

This is reasonable, and we understand that an Order consists of one or more items, that we will call LineItems. What unit test would we write for this requirement?

In C# we might write the following Nnuit test code in a file called Program.cs:

[Test]
public void computeOrderCost()
{ 
   Order o = new Order();
   float cost = o.getCost(); 
   Assert.That(cost >= 0.0);
}

If we were running a command-line environment we would compile Program.cs and on line #1 we would get an error that the compiler cannot find Order. If we are working in an IDE such as Visual Studio, the IDE will flag the first line of code with the same condition: cannot find a definition for Order. Both of these results constitute failures of the test code.

So, we now create an Order.cs class file, define the Order class, and return to Program.cs. We rebuild, and now line 1 passes, but line 2 fails with: no method float getCost() defined in class Order. OK, now we add the getCost() method to the Order class:

public float getCost()
{
}

But what do I put as the return? And shouldn't I be searching through all the LineItems to tally their individual extended costs? And shouldn't I introduce a List collection class here to hold the LineItems? And won't I need to define the LineItem class and at least a zero-arg constructor? And.... And you can get totally wrapped around the axle at this point, can't you? And we are only trying to pass at line #2 in our test code!

Here you adopt the heuristic: Always do the simplest thing possible. You can always, and you will always, add more complexity later. For now, just make sure the code works when it is simple.

In our example, the simplest thing possible means I will write getCost() as:

public float getCost()
{
   return 0;
}

That's enough...for now.

I rebuild Program.cs, and it builds! Finally, after two failures, and having to write just a little production code only to overcome the failures, the test method builds. Now, I run the test method, it creates an empty Order object, o, and invokes the order's getCost() method, and gets a valid return of a floating point value of zero. The Nunit Assert statement verifies that the value is non-negative, and all is good.

I now know that my test will fail without an Order, and without an Order.getCost() method with return type of float. I know my interface to and from getCost() is working. Where to go next? Your first thought might be to add the LineItem collection to Order. But let's think again: what is the simplest thing I can do?

The simplest thing is to create a separate test for one LineItem, asking it for its extended cost:

[Test]
public void getLineItemCost()
{
   LineItem li = new LineItem();
   float extcost = li.getExtendedCost();
   Assert.That(extcost >= 0.0);
}

This will fail until we define the LineItem class, and define the method getExtendedCost() that we also define to return a value of 0.0.

Now we have an Order class that can return its total cost, and we have a LineItem class that can return its extended cost. But they are not linked together.

Now we enhance our original computeOrderCost() method to incorporate only one LineItem (that's the simplest thing, right?):

[Test]
public void computeOrderCost()
{ 
   Order o = new Order();
   LineItem li = new LineItem();  //New
   o.add(li);                     //New
   float cost = o.getCost(); 
   Assert.That(cost >= 0.0);
}

Now we add to Order the method add(LineItem), which will save (in a private field) the reference to one LineItem, and we enhance Order's getCost() method:

public float getCost()
{
   float sum = 0;
   sum += li.getExtendedCost(); //li reference set in Order.add()
   return sum;
}

You can see the evolution here. Next, we can enhance the Order.add(LineItem) method to store each LineItem in a collection class, and enhance Order.getCost() to use a foreach loop to obtain the extended cost from each LineItem. All of the LineItems are currently returning zero, but once we can traverse the entire collection, and validate that the total cost is truly zero, then we can enhance the getLineItemCost() test method to create LineItems with non-zero extended costs, and then enhance the Order.getCost() method to verify that we can correctly tally these non-zero extended costs in the collection of LineItems.

At this point, readers who are developers may be thinking, "So what? I could have written all that code right up front. What do I gain taking these small, tedious steps?" Of course you could have just written the code. But what we gained in this approach is the tests. If you don't write these first, you cannot fall back on them to validate your application code.

Another huge benefit of TDD is only a bit more subtle. When you write the tests, and then the production code, those tests are users of your production code. TDD lets you think about your production code and it's interfaces from the perspective of a user, i.e. from client code that is invoking your code. What better way to make sure your interfaces, method names, and argument lists are usable than to design them as a user!

TDD frameworks are referred to as "xUnit" frameworks, and are available for most languages. Some that may be compatible with your development languages are:

A more extensive list of xUnit frameworks is maintained on the website of extreme programming guru Ron Jeffries.

TDD lets you take large, or very small, steps in how you arrive at the final version of each test method. And with each step you can write a lot, or just a little, production code to satisfy the current goals of each test method. Once you write the unit tests, you have them forever. As your project requirements change, so will the unit tests, and so will the production code. But having the tests is everything. When you have 85 unit tests working against your production code, and you are getting "green" bars, you feel good. You know that if you make a code error that causes a working test to fail, you will get a "red" bar alerting you to the failed test. That gives you confidence, confidence to code quickly knowing the tests will keep you accurate. With TDD you are never more than a few seconds away from knowing if all your application code still passes the tests - or has been broken by your last change!

 

Copyright ©2009 Evanetics, Inc. All Rights Reserved.  www.evanetics.com