Are We Safe Yet?

by Gary K. Evans

I travel by airplane a lot, which means I spend too much of my life in a "locked and upright position". Whenever I fly, I have the usual safety concerns: unruly or suspicious fellow-travelers, weather problems, inebriated pilots, and mechanical failures. But as a software developer I have another, uncommon concern that does not disturb the salesman in the next seat, or the soccer-mom in 24A: is the software flying this airplane safe?

I have been developing, and working with software development groups, for 30 years. Almost all of my professional work has been in commercial, business software development. But I remain astounded that not once in 30 years have I ever seen, or heard, a requirement or IT policy that the software under development must be safe. This may be because the discipline of software safety did not exist before the mid-1980s, when Nancy Leveson started her initial research in this area. Or, it might be because many people assume if the software has no bugs, it must be safe - which is patently not a true statement.

Here are some known examples of unsafe software accidents and the escalating damage they have produced over the decades.

In the 1970s:

• A magazine subscription program sent every copy of a magazine to the same address.
• A programmer ported his company's payroll program. When he (and everyone else) got their next paycheck, they were all for $0.00.
• The first U.S. probe to Venus failed because of a missing comma in a FORTRAN DO loop: DO 3 I = 1.3 instead of DO 3 I = 1,3.

In the 1980s:

• The Space Shuttle is completely dependent on the proper operation of its computers; a mission cannot even be aborted if the computers fail.
• A mechanical malfunction in a fly-by-wire flight control system set up an accelerated environment for which the flight control software was not programmed. The aircraft went out of control and crashed.

In the 1990s:

• AT&T's #4ESS long distance switch was designed to send a specific message to other switches when the #4ESS rebooted from a crash. But a bug in the software on the switch receiving the reboot message caused that switch to crash, then it rebooted and sent the same message out...each switch crashing and then crashing all the other switches.
• The Ariane 5 missile built by the European Space Agency reused working software from the Ariane 4. But other code in the Ariane 5 generated floating-point numbers larger than the Ariane 4 software was designed for, causing an overflow condition that resulted in the destruction of the missile 40 seconds after launch.
• And, of course, there was Y2K.

In the 2000s we are confronted with compromised encryption schemes, aircraft such as the Boeing 717 that take-off and land solely under computer control, Denial Of Service and buffer overrun attacks, cyber terrorism, and ubiquitous reliance on computer software from email to pacemaker operation.

But one of the first examples of horrific software failure is the Therac-25, and it deserves some inspection. The Therac-25 was a medical diagnostic machine that emitted either x-radiation or electron beams. Because of a race condition in the kludged operating software that controlled the position of a metal plate in the beam path, and a bug in the user interface code, the machine operator could cause the machine to emit lethal amounts of electron beam radiation into a patient's body, all while the UI indicated that safe amounts of X-radiation were being emitted. The Therac-25 killed five people before the manufacturer removed it from every installation.

But My Software is Safe

You may be thinking, "I don't write software for avionics, or spacecraft, or even telephone systems. My business software cannot hurt anyone, so it has to be safe." Let me disabuse you of this idea with a real scenario on how to kill someone with a word processor.

Consider that you prepare a word processor document like the one below. The " <\t> " symbol represents a TAB character.

You then save the document, and sometime later read it back into your word processor. The document is different now:

We see that a TAB character has been lost. Well, how dangerous can that be? The answer is in what I neglected to put in the original document. I left off the headers corresponding to the columns identified by the tabs. Here is the "real" document after reading it back in:

Sorry, Chester! Fixed in next release!

(This example came from a U.S. Navy Test Plan for document fidelity. I have modified the scenario a bit for simplicity.)

Software safety become an interest of mine when I was in a Communications group quite a few years ago. I was writing driver software for a Front-End Processor, passing data to a mid-range computer. This was a new product line, and the Sales group informed us that one of our first installations would be at a hospital. My wake-up call came when I realized my software would be shipping patient records, prescriptions and diagnoses across a 50-pin, high-speed Common Trunk connection. In this world dropping a bit, or hitting a race condition wasn't just a bug: it could mean ending a life. Our craft has made tremendous progress, but we still cannot boldly state "my software is safe".

In next month's newsletter I will discuss some innovative testing ideas that can help us come closer to "safer", more reliable software. Forward this email to your QA people and let them know.

Have a productive (and safe) month,

    Gary K. Evans