Static Testing

A common question that arise when Static testing technique comes into picture. “How come a code will be evaluated without executing it?” This can be done by manual examination of the software product and requirements or can be done by some tools. The most commonly followed procedure for the static testing technique is “Review process”. The various advantages of the review process in static testing are,

•         Since static testing can start early in the life cycle, early feedback on quality issues can be established, e.g. an early validation of user requirements and not just late in the life cycle during acceptance testing.

•         By detecting defects at an early stage, rework costs are most often relatively low and thus a relatively cheap improvement of the quality of software products can be achieved.

•         Since rework effort is substantially reduced, development productivity figures are likely to increase.

•         The evaluation by a team has the additional advantage that there is an exchange of information between the participants.

•         Static tests contribute to an increased awareness of quality issues.

The Review Process

Reviews are commonly classified as formal or informal. In case of a formal review a large number of members from the company will involve and happens in a most formal way. All the problems identified here will be documented and preserved for fixing the issues.

An informal review is undocumented and will happen between colleagues during the initial stages of the development of the code or program. Here the review can be conducted from a small team with members counting from one or two. As it is of not much importance in the testing process we will look into the formal review process in detail.

Phases of the formal Review

            There are various steps in the review process which are,

1.      Planning

2.      Kick-off

3.      Preparation

4.      Review meeting

5.      Rework

6.      Follow-up

Planning

The review process begins with a request for review by the moderator or the author of the document being reviewed. A moderator is often assigned to take care of the scheduling of the review. Usually the document that is being reviewed will be “Entry checked” in order to avoid waste of time. This is because the document with too many mistakes is not ready for review and it will not waste the time for reviewers and author to review on an incomplete document.

Few other entry criteria rules that can be found in a good document review are,

·         A short check of a product sample by the moderator (or expert) does not reveal a large number of major defects. For example, after 30 minutes of checking, no more than 3 major defects are found on a single page or fewer than 10 major defects in total in a set of 5 pages.

·         The document to be reviewed is available with line numbers.

·         The document has been cleaned up by running any automated checks that apply.

·         References needed for the inspection are stable and available.

·         The document author is prepared to join the review team and feels confident with the quality of the document.

The review team usually consists of four to six participants including the moderator and the author. To improve the effectiveness of the review the moderator will assign the tasks to each participant. The various focuses in the review will be as follows,

·         Focus on higher-level documents, e.g. does the design comply to the requirements;

·         Focus on standards, e.g. internal consistency, clarity, naming conventions, templates;

·         Focus on related documents at the same level, e.g. interfaces between software functions;

·         Focus on usage, e.g. for testability or maintainability.

Kick-off

Kick-off is an optional step in the review process, in this phase the objective and the plans of the reviews and the other details are discussed. This is usually not carried out in the review process.

Preparation

The participants work individually for the document under review and identify defects and other questions or comments about the project. In this step the documented is viewed end to end for any faults or errors and all defects are recorded and mentioned it to the author at the end for corrections and fixing of defects. The speed of the preparation phase is termed as per the number of pages being checked and this is called checking rate.

Review Meeting

The review meeting has three phases Logging, Discussion and Decision Phase. In each phase the various critical decisions about the reviews are carried out.

During the logging phase the issues, e.g. defects, that have been identified during the preparation are mentioned page by page, reviewer by reviewer and are logged either by the author or by a scribe. A separate person to do the logging (a scribe) is especially useful for formal review types such as an inspection. To ensure progress and efficiency, no real discussion is allowed during the logging phase. If an issue needs discussion, the item is logged and then handled in the discussion phase. A detailed discussion on whether or not an issue is a defect is not very meaningful, as it is much more efficient to simply log it and proceed to the next one. Furthermore, in spite of the opinion of the team, a discussed and discarded defect may well turn out to be a real one during rework.

            During the logging phase the focus is on logging as many defects as possible within a certain timeframe. To ensure this, the moderator tries to keep a good logging rate (number of defects logged per minute). In a well-led and disciplined formal review meeting, the logging rate should be between one and two defects logged per minute.

A decision on the document under review has to be made by the participants, sometimes based on formal exit criteria. The most important exit criterion is the average number of critical and/or major defects found per page (e.g. no more than three critical/major defects per page). If the number of defects found per page exceeds a certain level, the document must be reviewed again, after it has been reworked. If the document complies with the exit criteria, the document will be checked during follow-up by the moderator or one or more participants. Subsequently, the document can leave the review process.

Rework

Based on the defects detected, the author will improve the document under review step by step. Not every defect that is found leads to rework. It is the author's responsibility to judge if a defect has to be fixed. If nothing is done about an issue for a certain reason, it should be reported to at least indicate that the author has considered the issue.

Follow-up

The moderator is responsible for ensuring that satisfactory actions have been taken on all (logged) defects, process improvement suggestions and change requests. Although the moderator checks to make sure that the author has taken action on all known defects, it is not necessary for the moderator to check all the corrections in detail. If it is decided that all participants will check the updated document, the moderator takes care of the distribution and collects the feedback. For more formal review types the moderator checks for compliance to the exit criteria.

Types of Review

            A document under review is subject to various types of reviews depending on the nature of the project. A document can go through an informal review and then it can be subject to technical review. So the main aim is to find out the defects of the product but it may take several ways to achieve the same results. Let’s examine various types of reviews carried in an organization.

Walkthrough

            In the walkthrough process the author will guide the participating people through clear understanding of the document and makes them to achieve a common understanding and to gather feedback. This mode is adopted when the participants are from other domains apart from the software domain. The author will explain the content of the document in a step by step manner to reach the participants.

            Within a walkthrough the author does most of the preparation. The participants, who are selected from different departments and backgrounds, are not required to do a detailed study of the documents in advance. Because of the way the meeting is structured, a large number of people can participate and this larger audience can bring a great number of diverse viewpoints regarding the contents of the document being reviewed as well as serving an educational purpose. If the audience represents a broad cross-section of skills and disciplines, it can give assurance that no major defects are 'missed' in the walkthrough. A walkthrough is especially useful for higher-level documents, such as requirement specifications and architectural documents.

In general the following goals can be applicable:

·         To present the document to stakeholders both within and outside the soft ware discipline, in order to gather information regarding the topic under documentation;

·         To explain (knowledge transfer) and evaluate the contents of the document;

·         To establish a common understanding of the document;

·         To examine and discuss the validity of proposed solutions and the viability of alternatives, establishing consensus.

Key characteristics of walkthroughs are:

·         The meeting is led by the authors; often a separate scribe is present.

·         Scenarios and dry runs may be used to validate the content.

·         Separate pre-meeting preparation for reviewers is optional.

Technical review

A technical review is a discussion meeting that focuses on achieving consensus about the technical content of a document. Compared to inspections, technical reviews are less formal and there is little or no focus on defect identification on the basis of referenced documents, intended readership and rules. During technical reviews defects are found by experts, who focus on the content of the document. The experts that are needed for a technical review are, for example, architects, chief designers and key users. In practice, technical reviews vary from quite informal to very formal.

The goals of a technical review are to:

·         Assess the value of technical concepts and alternatives in the product and project environment;

·         Establish consistency in the use and representation of technical concepts;

·         Ensure, at an early stage, that technical concepts are used correctly;

·         Inform participants of the technical content of the document.

Key characteristics of a technical review are:

·         It is a documented defect-detection process that involves peers and technical experts.

·         It is often performed as a peer review without management participation.

·         Ideally it is led by a trained moderator, but possibly also by a technical expert.

·         A separate preparation is carried out during which the product is examined and the defects are found.

·         More formal characteristics such as the use of checklists and a logging list or issue log are optional.

Inspection

            Inspection is the most formal review type. The document under inspection is prepared and checked thoroughly by the reviewers before the meeting, comparing the work product with its sources and other referenced documents, and using rules and checklists. In the inspection meeting the defects found are logged and any discussion is postponed until the discussion phase. This makes the inspection meeting a very efficient meeting.

            The reason for carrying out inspections can be explained by using Weinberg's concept of egoless engineering Weinberg refers to the human tendency to self-justify actions. Since we tend not to see evidence that conflicts with our strong beliefs, our ability to find errors in our own work is impaired. Because of this tendency, many engineering organizations have established independent test groups that specialize in finding defects. Similar principles have led to the introduction of inspections and reviews in general.

The generally accepted goals of inspection are to

·         Help the author to improve the quality of the document under inspection

·         Remove defects efficiently, as early as possible

·         Improve product quality, by producing documents with a higher level of quality

·         Create a common understanding by exchanging information among the inspection participants

·         Train new employees in the organization's development process

·         Learn from defects found and improve processes in order to prevent recurrence of similar defects

·         Sample a few pages or sections from a larger document in order to measure the typical quality of the document, leading to improved work by individuals in the future, and to process improvements.

Key characteristics of an inspection are

·         It is usually led by a trained moderator (certainly not by the author).

·         It uses defined roles during the process.

·         It involves peers to examine the product.

·         Rules and checklists are used during the preparation phase.

·         A separate preparation is carried out during which the product is examined and the defects are found.

·         The defects found are documented in a logging list or issue log.

·         A formal follow-up is carried out by the moderator applying exit criteria.

·         Optionally, a causal analysis step is introduced to address process improvement issues and learn from the defects found.

·         Metrics are gathered and analyzed to optimize the process.