Whitebox testing

Structure-based test design techniques are a good way of generating additional test cases that are different from existing tests. They can help ensure more breadth of testing, in the sense that test cases that achieve 100% coverage in any measure will be exercising all parts of the software from the point of view of the items being covered.

Structure-based techniques serve two purposes: test coverage measurement and structural test case design. They are often used first to assess the amount of testing performed by tests derived from specification-based techniques, i.e. to assess coverage. They are then used to design additional tests with the aim of increasing the test coverage.

What is test coverage?

Test coverage measures in some specific way the amount of testing performed by a set of tests (derived in some other way, e.g. using specification-based techniques). Wherever we can count things and can tell whether or not each of those things has been tested by some test, then we can measure coverage. The basic coverage measure is

                                 Number of coverage items exercised

Coverage = --------------------------------------------------- x 100%

                                       Total number of coverage items

where the 'coverage item' is whatever we have been able to count and see whether a test has exercised or used this item.

There is danger in using a coverage measure. 100% coverage does not mean 100% tested! Coverage techniques measure only one dimension of a multi-dimensional concept. Two different test cases may achieve exactly the same coverage but the input data of one may find an error that the input data of the other doesn't. One drawback of code coverage measurement is that it measures coverage

of what has been written, i.e. the code itself; it cannot say anything about the software that has not been written. If a specified function has not been implemented, specification-based testing techniques will reveal this. If a function was omitted from the specification, then experience-based techniques may find it. But structure-based techniques can only look at a structure which is already there.

How to measure coverage

For most practical purposes, coverage measurement is something that requires tool support. However, knowledge of the steps typically taken to measure coverage is useful in understanding the relative merits of each technique. Our example assumes an intrusive coverage measurement tool that alters the code by inserting instrumentation:

§  Decide on the structural element to be used, i.e. the coverage items to be counted.
§  Count the structural elements or items.
§  Instrument the code.
§  Run the tests for which coverage measurement is required.
§  Using the output from the instrumentation, determine the percentage of elements or items exercised.

Statement coverage

Studies and experience in the industry have indicated that what is considered reasonably thorough black-box testing may actually achieve only 60% to 75% statement coverage. Typical ad hoc testing is likely to be around 30% - this leaves 70% of the statements untested. Different coverage tools may work in slightly different ways, so they may give different coverage figures for the same set of tests on the same code, although at 100% coverage they should be the same. We will illustrate the principles of coverage on code. In order to simplify our examples, we will use a basic pseudo-code - this is not any specific programming language, but should be readable and understandable to you, even if you have not done any programming yourself. For example, consider code below,

READ A

READ B

I F A>B THEN C = 0

END IF

To achieve 100% statement coverage of this code segment just one test case is required, one which ensures that variable A contains a value that is greater than the value of variable B, for example, A = 12 and B = 10. Note that here we are doing structural test design first, since we are choosing our input values in order ensure statement coverage.

Statement coverage is calculated by:

                                           Number of statements exercised

Statement coverage = --------------------------------------------- x 100%

                                               Total number of statements

Decision coverage

A decision is an IF statement, a loop control statement (e.g. DO-WHILE or REPEAT-UNTIL), or a CASE statement, where there are two or more possible exits or outcomes from the statement. With an IF statement, the exit can either be TRUE or FALSE, depending on the value of the logical condition that comes after IF. With a loop control statement, the outcome is either to perform the code within the loop or not - again a True or False exit. Decision coverage is calculated by:

                                         Number of decision outcomes exercised

Decision coverage = -------------------------------------------------------- x 100%

                                             Total number of decision outcomes

Condition Coverage

Here the coverage depends on the testing of the Conditional Statements in the program. It mainly concentrates on the Boolean conditional statements in the program. Eg. IF (A=B&&A=C). it mainly concentrates on the various possible input combinations.

                                            Number of conditional outcomes exercised

Condition coverage = -------------------------------------------------------- x 100%

                                                  Total number of conditional outcomes