Concept: Test-Ideas List

Introduction

Information used in designing tests is gathered from many places: design models, classifier interfaces, statecharts, and code itself. At some point, this source document information must be transformed into executable tests:

specific inputs given to the software under test
in a particular hardware and software configuration
initialized to a known state
with specific results expected

It's possible to go directly from source document information to executable tests, but it's often useful to add an intermediate step. In this step, test ideas are written into a Test-Ideas List, which is used to create executable tests.

What are Test Ideas?

A test idea (sometimes referred to as a test requirement) is a brief statement about a test that could be performed. As a simple example, let's consider a function that calculates a square root and come up with some test ideas:

give a number that's barely less than zero as input
give zero as the input
test a number that's a perfect square, like 4 or 16 (is the result exactly 2 or 4?)

Each of these ideas could readily be converted into an executable test with exact descriptions of inputs and expected results.

There are two advantages to this less-specific intermediate form:

test ideas are more reviewable and understandable than complete tests - it's easier to understand the reasoning behind them
test ideas support more powerful tests, as described later under the heading Test Design Using the List

The square root examples all describe inputs, but test ideas can describe any of the elements of an executable test. For example, "print to a LaserJet IIIp" describes an aspect of the test environment to be used for a test, as does "test with database full", however, these latter test ideas are very incomplete in themselves: Print what to the printer? Do what with that full database? They do, however, ensure that important ideas aren't forgotten; ideas that will be described in more detail later in test design.

Test ideas are often based on fault models; notions of which faults are plausible in software and how those faults can best be uncovered. For example, consider boundaries. It's safe to assume the square root function can be implemented something like this:

 double sqrt(double x) {     if (x < 0)        // signal error     ...

It's also plausible that the < will be incorrectly typed as <=. People often make that kind of mistake, so it's worth checking. The fault cannot be detected with X having the value 2, because both the incorrect expression (x<=0) and the correct expression (x<0) will take the same branch of the if statement. Similarly, giving X the value -5 cannot find the fault. The only way to find it is to give X the value 0, which justifies the second test idea.

In this case, the fault model is explicit. In other cases, it's implicit. For example, whenever a program manipulates a linked structure, it's good to test it against a circular one. It's possible that many faults could lead to a mishandled circular structure. For the purposes of testing, they needn't be enumerated - it suffices to know that some fault is likely enough that the test is worth running.

The following links provide information about getting test ideas from different kinds of fault models. The first two are explicit fault models; the last uses implicit ones.

These fault models can be applied to many different artifacts. For example, the first one describes what to do with Boolean expressions. Such expressions can be found in code, in guard conditions, in statecharts and sequence diagrams, and in natural-language descriptions of method behaviors (such as you might find in a published API).

Occasionally it's also helpful to have guidelines for specific artifacts. See Guideline: Test Ideas for Statechart and Flow Diagrams.

A particular Test-Ideas List might contain test ideas from many fault models, and those fault models could be derived from more than one artifact.

Test Design Using the List

Let's suppose you're designing tests for a method that searches for a string in a sequential collection. It can either obey case or ignore case in its search, and it returns the index of the first match found or -1 if no match is found.

 int Collection.find(String string,     Boolean ignoreCase);

Here are some test ideas for this method:

match found in the first position
match found in the last position
no match found
two or more matches found in the collection
case is ignored; match found, but it wouldn't match if case was obeyed
case is obeyed; an exact match is found
case is obeyed; a string that would have matched if case were ignored is skipped

It would be simple to implement these seven tests, one for each test idea. However, different test ideas can be combined into a single test. For example, the following test satisfies test ideas 2, 6, and 7:

Setup: collection initialized to ["dawn", "Dawn"]
Invocation: collection.find("Dawn", false)
Expected result: return value is 1 (it would be 0 if "dawn" were not skipped)

Making test ideas nonspecific makes them easier to combine.

It's possible to satisfy all of the test ideas in three tests. Why would three tests that satisfy seven test ideas be better than seven separate tests?

When you're creating a large number of simple tests, it's common to create test N+1 by copying test N and tweaking it just enough to satisfy the new test idea. The result, especially in more complex software, is that test N+1 probably exercises the program in almost the same way as test N. It takes almost exactly the same path through the code.

A smaller number of tests, each satisfying several test ideas, doesn't allow a "copy and tweak" approach. Each test will be somewhat different from the last, exercising the code in different ways and taking different paths.

Why would that be better? If the Test-Ideas List were complete, with a test idea for every fault in the program, it wouldn't matter how you wrote the tests. But the list is always missing some test ideas that could find bugs. By having each test do very different things from the last one - by adding seemingly unneeded variety - you increase the chance that one of the tests will stumble over a bug by sheer dumb luck. In effect, smaller, more complex tests increase the chance the test will satisfy a test idea that you didn't know you needed.
Sometimes when you're creating more complex tests, new test ideas come to mind. That happens less often with simple tests, because so much of what you're doing is exactly like the last test, which dulls your mind.

However, there are reasons for not creating complex tests.

If each test satisfies a single test idea and the test for idea 2 fails, you immediately know the most likely cause: the program doesn't handle a match in the last position. If a test satisfies ideas 2, 6, and 7, then isolating the failure is harder.
Complex tests are more difficult to understand and maintain. The intent of the test is less obvious.
Complex tests are more difficult to create. Constructing a test that satisfies five test ideas often takes more time than constructing five tests that each satisfy one. Moreover, it's easier to make mistakes - to think you're satisfying all five when you're only satisfying four.

In practice, you must find a reasonable balance between complexity and simplicity. For example, the first tests you subject the software to (typically the smoke tests) should be simple, easy to understand and maintain, and intended to catch the most obvious problems. Later tests should be more complex, but not so complex they are not maintainable.

After you've finished a set of tests, it's good to check them against the characteristic test design mistakes discussed in Concept: Developer Testing.

Using Test Ideas Before Testing

A Test-Ideas List is useful for reviews and inspections of design artifacts. For example, consider this part of a design model showing the association between Department and Employee classes.

Figure 1: Association between Department and Employee Classes

The rules for creating test ideas from such a model would ask you to consider the case where a department has many employees. By walking through a design and asking "what if, at this point, the department has many employees?", you might discover design or analysis errors. For example, you might realize that only one employee at a time can be transferred between departments. That might be a problem if the corporation is prone to sweeping reorganizations where many employees need to be transferred.

Such faults, cases where a possibility was overlooked, are called faults of omission. Just like the faults themselves, you have probably omitted tests that detect these faults from your testing effort. For example, see [GLA81], [OST84], [BAS87], [MAR00], and other studies that show how often faults of omission escape into deployment.

The role of testing in design activities is discussed further in Concept: Test-first Design.

Test Ideas and Traceability

Traceability is a matter of tradeoffs. Is its value worth the cost of maintaining it? This question needs to be considered during Activity: Define Assessment and Traceability Needs.

When traceability is worthwhile, it's conventional to trace tests back to the artifacts that inspired them. For example, you might have traceability between an API and its tests. If the API changes, you know which tests to change. If the code (that implements the API) changes, you know which tests to run. If a test puzzles you, you can find the API it's intended to test.

The Test-Ideas List adds another level of traceability. You can trace from a test to the test ideas it satisfies, and then from the test ideas to the original artifact.