Whitebox testing

Structure-based test design techniques are a good way of generating additional test cases that are different from existing tests. They can help ensure more breadth of testing, in the sense that test cases that achieve 100% coverage in any measure will be exercising all parts of the software from the point of view of the items being covered.

Structure-based techniques serve two purposes: test coverage measurement and structural test case design. They are often used first to assess the amount of testing performed by tests derived from specification-based techniques, i.e. to assess coverage. They are then used to design additional tests with the aim of increasing the test coverage.

What is test coverage?

Test coverage measures in some specific way the amount of testing performed by a set of tests (derived in some other way, e.g. using specification-based techniques). Wherever we can count things and can tell whether or not each of those things has been tested by some test, then we can measure coverage. The basic coverage measure is

                                 Number of coverage items exercised

Coverage = --------------------------------------------------- x 100%

                                       Total number of coverage items

where the 'coverage item' is whatever we have been able to count and see whether a test has exercised or used this item.

There is danger in using a coverage measure. 100% coverage does not mean 100% tested! Coverage techniques measure only one dimension of a multi-dimensional concept. Two different test cases may achieve exactly the same coverage but the input data of one may find an error that the input data of the other doesn't. One drawback of code coverage measurement is that it measures coverage

of what has been written, i.e. the code itself; it cannot say anything about the software that has not been written. If a specified function has not been implemented, specification-based testing techniques will reveal this. If a function was omitted from the specification, then experience-based techniques may find it. But structure-based techniques can only look at a structure which is already there.

How to measure coverage

For most practical purposes, coverage measurement is something that requires tool support. However, knowledge of the steps typically taken to measure coverage is useful in understanding the relative merits of each technique. Our example assumes an intrusive coverage measurement tool that alters the code by inserting instrumentation:

§  Decide on the structural element to be used, i.e. the coverage items to be counted.
§  Count the structural elements or items.
§  Instrument the code.
§  Run the tests for which coverage measurement is required.
§  Using the output from the instrumentation, determine the percentage of elements or items exercised.

Statement coverage

Studies and experience in the industry have indicated that what is considered reasonably thorough black-box testing may actually achieve only 60% to 75% statement coverage. Typical ad hoc testing is likely to be around 30% - this leaves 70% of the statements untested. Different coverage tools may work in slightly different ways, so they may give different coverage figures for the same set of tests on the same code, although at 100% coverage they should be the same. We will illustrate the principles of coverage on code. In order to simplify our examples, we will use a basic pseudo-code - this is not any specific programming language, but should be readable and understandable to you, even if you have not done any programming yourself. For example, consider code below,

READ A

READ B

I F A>B THEN C = 0

END IF

To achieve 100% statement coverage of this code segment just one test case is required, one which ensures that variable A contains a value that is greater than the value of variable B, for example, A = 12 and B = 10. Note that here we are doing structural test design first, since we are choosing our input values in order ensure statement coverage.

Statement coverage is calculated by:

                                           Number of statements exercised

Statement coverage = --------------------------------------------- x 100%

                                               Total number of statements

Decision coverage

A decision is an IF statement, a loop control statement (e.g. DO-WHILE or REPEAT-UNTIL), or a CASE statement, where there are two or more possible exits or outcomes from the statement. With an IF statement, the exit can either be TRUE or FALSE, depending on the value of the logical condition that comes after IF. With a loop control statement, the outcome is either to perform the code within the loop or not - again a True or False exit. Decision coverage is calculated by:

                                         Number of decision outcomes exercised

Decision coverage = -------------------------------------------------------- x 100%

                                             Total number of decision outcomes

Condition Coverage

Here the coverage depends on the testing of the Conditional Statements in the program. It mainly concentrates on the Boolean conditional statements in the program. Eg. IF (A=B&&A=C). it mainly concentrates on the various possible input combinations.

                                            Number of conditional outcomes exercised

Condition coverage = -------------------------------------------------------- x 100%

                                                  Total number of conditional outcomes

Blackbox Testing

The four black box testing techniques we will cover in detail are:

·         Equivalence partitioning
·         Boundary value analysis
·         Decision tables
·         State transition testing

 Equivalence Partitioning

Here the basic idea behind using the equivalence partition is that instead of checking the various possible inputs we will divide the set of inputs as Valid and Invalid.

Valid-All Inputs grouped as valid are the ones which will be accepted by the software or the ones which will work.

Invalid-It is completely opposite to the Valid, it means all inputs that are not allowed or invalid are categorized.

Equivalence partitions are also known as equivalence classes - the two terms mean exactly the same thing.

The equivalence-partitioning technique then requires that we need test only one condition from each partition. This is because we are assuming that all the conditions in one partition will be treated in the same way by the software. If one condition in a partition works, we assume all of the conditions in that partition will work, and so there is little point in testing any of these others. Conversely, if one of the conditions in a partition does not work, then we assume that none of the conditions in that partition will work so again there is little point in testing any more in that partition.

For example, Let’s assume we are testing a Numeric box which will accept data(I/P) only from 1 to 100(Including 1 and 100). In this case let’s derive the Equivalence partitioning. First the valid input data set is 1to100, also the invalid or unaccepted set of inputs are any value less than 1 and greater than 100. Lets form the Equivalence partition for the same.

This illustrates a very important task of the tester – not only do we test what is in our specification, but we also think about things that haven't been specified. In this case we have thought of the situation where the Input is less than zero.

So for the Equivalence partitioning we will take one Input from each partition, then the resultant test cases for this method are

Test Case1= -12

Test Case2=15

Test Case3=105

Boundary value analysis

Boundary value analysis (BVA) is based on testing at the boundaries between partitions. If you have ever done 'range checking', you were probably using the boundary value analysis technique, even if you weren't aware of it. Note that we have both valid boundaries (in the valid partitions) and invalid boundaries (in the invalid partitions).

As an example, consider a printer that has an input option of the number of copies to be made, from 1 to 100. To apply boundary value analysis, we will take the minimum and maximum (boundary) values from the valid partition (1 and 100 in this case) together with copies to be made, from 1 to 100. The first or last value respectively in each of the invalid partitions adjacent to the valid partition (0 and 101 in this case). In this example we would have three equivalence partitioning tests (one from each of the three partitions) and four boundary value tests. So the Test cases are,

Test Case1=0

Test Case1=1

Test Case1=100

Test Case1=101

Test Case1= -12

Test Case1=15

Test Case1=105

Decision table testing

In the above discussed methods there is no check point to handle if the inputs are given in combinations. So it is not good to use Equivalence Partition and BVA for Combination of inputs. A decision table is a good way to deal with combinations of things (e.g. inputs). This technique is sometimes also referred to as a 'cause-effect' table. The reason for this is that there is an associated logic diagramming technique called 'cause-effect graphing' which was sometimes used to help derive the decision table.

Cause-A Cause is nothing but the Input given to the Software or to the function.

Effect-A effect is nothing but the output.

If you begin using decision tables to explore what the business rules are that should be tested, you may find that the analysts and developers find the tables very helpful and want to begin using them too. Do encourage this, as it will make your job easier in the future. Decision tables provide a systematic way of stating complex business rules, which is useful for developers as well as for testers. Decision tables can be used in test design whether or not they are used in specifications, as they help testers explore the effects of combinations of different inputs and other software states that must correctly implement business rules. Helping the developers do a better job can also lead to better relationships with them.

Testing combinations can be a challenge, as the number of combinations can often be huge. Testing all combinations may be impractical if not impossible. We have to be satisfied with testing just a small subset of combinations but making the choice of which combinations to test and which to leave out is not trivial. If you do not have a systematic way of selecting combinations, an arbitrary subset will be used and this may well result in an ineffective test effort.

Let's look at an example. If you are a new customer opening a credit card account, you will get a 15% discount on all your purchases today. If you are an existing customer and you hold a loyalty card, you get a 10% discount. If you have a coupon, you can get 20% off today (but it can't be used with the 'new customer' discount). Discount amounts are added, if applicable. This is shown below

State transition testing

State transition testing is used where some aspect of the system can be described in what is called a 'finite state machine'. In simple words it can be called as a flow chart or representing the process flow graphically. This simply means that the system can be in a (finite) number of different states, and the transitions from one state to another are determined by the rules of the 'machine'. This is the model on which the system and the tests are based. Any system where you get a different output for the same input, depending on what has happened before, is a finite state system. A finite state system is often shown as a state diagram.

1.      The states that the software may occupy (open/closed or funded/insufficient funds);
2.      The transitions from one state to another (not all transitions are allowed);
3.      The events that cause a transition (closing a file or withdrawing money);
4.      The actions that result from a transition (an error message or being given your cash).

Note that in any given state, one event can cause only one action, but that the same event - from a different state - may cause a different action and a different end state.

Note that a transition does not need to change to a different state (although all of the transitions shown above do go to a different state). So there could be a transition from 'access account' which just goes back to 'access account' for an action such as 'request balance'. Test conditions can be derived from the state graph in various ways. Each state can be noted as a test condition, as can each transition. In the Syllabus, we need to be able to identify the coverage of a set of tests in terms of transitions.

Use case testing

Use case testing is a technique that helps us identify test cases that exercise the whole system on a transaction by transaction basis from start to finish.

A use case is a description of a particular use of the system by an actor (a user of the system). Each use case describes the interactions the actor has with the system in order to achieve a specific task (or, at least, produce something of value to the user). Actors are generally people but they may also be other systems.

Use cases are a sequence of steps that describe the interactions between the actor and the system. Use cases are defined in terms of the actor, not the system, describing what the actor does and what the actor sees rather than what inputs the system expects and what the system outputs. They often use the language and terms of the business rather than technical terms, especially when the actor is a business user. They serve as the foundation for developing test cases mostly at the system and acceptance testing levels.

Use cases can uncover integration defects, that is, defects caused by the incorrect interaction between different components. Used in this way, the actor may be something that the system interfaces to such as a communication link or sub-system. Use cases describe the process flows through a system based on its most likely use. This makes the test cases derived from use cases particularly good for finding defects in the real-world use of the system (i.e. the defects that the users are most likely to come across when first using the system). Each use case usually has a mainstream (or most likely) scenario and sometimes additional alternative branches (covering, for example, special cases or exceptional conditions).