Testing Fundamentals

Topics covered– What is black-box and white-box testing– How static and dynamic testing differ– What high-level techniques can be used for

reviewing a product specification– What specific problems should look for when

reviewing a product specification in detail

Product Specification

The product specification is a written document using words and pictures to describe the intended product.

Produce spec might read something like this for Windows Calculator:

– The Edit menu will have two selections: Copy and Paste. These can be chosen by one of three methods: pointing and clicking to the menu items with the mouse, using access-keys (Alt+E and then C for Copy and P for Paste), or using the standard Windows shortcut keys of Ctrl+C for Copy and Ctrl+V for Paste.

– The Copy function will copy the current entry displayed in the number text box into the Windows Clipboard. The Paste function will paste the value stored in the Windows Clipboard into the number text box.

Black-Box Testing

Black Box Testing is testing without knowledge of the internal workings of the item being tested. For example, when black box testing is applied to software engineering, the tester would only know the "legal" inputs and what the expected outputs should be, but not how the program actually arrives at those outputs.

Black-box testing is sometimes referred to as functional testing or behavioral testing.

The test planning for black-box can begin as soon as the specifications are written.

Advantages of Black-Box Testing

more effective on larger units of code than glass box testing

tester needs no knowledge of implementation, including specific programming languages

tester and programmer are independent of each other tests are done from a user's point of view will help to expose any ambiguities or inconsistencies

in the specifications test cases can be designed as soon as the

specifications are complete

Disadvantages of Black Box Testing

only a small number of possible inputs can actually be tested, to test every possible input stream would take nearly forever

without clear and concise specifications, test cases are hard to design

there may be unnecessary repetition of test inputs if the tester is not informed of test cases the programmer has already tried

may leave many program paths untested cannot be directed toward specific segments of code which may

be very complex (and therefore more error prone) most testing related research has been directed toward glass box

testing

Tests under Black-Box Testing

Equivalence partitioning: Equivalence partitioning is designed to minimize the number of test cases by dividing tests in such a way that the system is expected to act the same way for all tests of each equivalence partition. Test inputs would be selected from each partition.

Boundary value Analysis is a form of black box testing in which input values at the boundaries of the input domain are tested. It has been widely recognized that input values at the extreme ends of, and just outside of, input domains tend to cause errors in system functionality.

Tests under Black-Box Testing (Cont..)

Smoke Test: A smoke test is a cursory examination of all of the basic components of a software system to ensure that they work. Typically, smoke testing is conducted immediately after a software build is made.

The term comes from electrical engineering, where in order to test electronic equipment, power is applied and the tester ensures that the product does not spark or smoke.

White-Box Testing

White-box testing is testing that takes into account the internal mechanism of a system or component (IEEE, 1990).

White-box testing is also known as structural testing, clear box testing, and glass box testing (Beizer, 1995).

In order to implement white box testing, the tester has to deal with the code and hence is needed to possess knowledge of coding and logic i.e. internal working of the code.

White box test also needs the tester to look into the code and find out which unit/statement/chunk of the code is malfunctioning.

Advantages of White-Box Testing

As the knowledge of internal coding structure is prerequisite, it becomes very easy to find out which type of input/data can help in testing the application effectively.

The other advantage of white box testing is that it helps in optimizing the code

It helps in removing the extra lines of code, which can bring in hidden defects.

Disadvantages of White-Box Testing

As knowledge of code and internal structure is a prerequisite, a skilled tester is needed to carry out this type of testing, which increases the cost.

And it is nearly impossible to look into every bit of code to find out hidden errors, which may create problems, resulting in failure of the application.

Types of testing under White/Glass Box Testing Strategy

Unit Testing: The developer carries out unit testing in order to check if the particular module or unit of code is working fine.

Static and dynamic Analysis: Static analysis involves going through the code in order to find out any possible defect in the code. Dynamic analysis involves executing the code and analyzing the output.

Statement Coverage: In this type of testing the code is executed in such a manner that every statement of the application is executed at least once. It helps in assuring that all the statements execute without any side effect.

Types of testing under White/Glass Box Testing Strategy (cont..)

Branch Coverage: Branch coverage testing helps in validating of all the branches in the code and making sure that no branching leads to abnormal behavior of the application.

Security Testing: Security Testing is carried out in order to find out how well the system can protect itself from unauthorized access, hacking – cracking, any code damage etc.

Mutation Testing: A kind of testing in which, the application is tested for the code that was modified after fixing a particular bug/defect.

Performing a High-Level Review

Defining the Software Product is a difficult process. The process is an inexact science and is prone to

having problems. Examine the specification for large fundamental

problems. It will give better understanding of what the software

should do. Better understanding of the whys and hows behind

the specifications.

Pretend to be the Customer

A tester is to pretend to be the customer when received a specification for review.

Do some research about who the customer will be.

Know about the end user. Find out who will be using the software and talk

to them.

Pretend to be the Customer (cont..)

It is important to understand the customer’s expectations.

Remember that the definition of quality means “meeting the customer’s needs”.

Tester must understand those needs to test. Tester is not necessary to be a domain expert Gaining some familiarity with the field the

software applies to would be a great help.

Pretend to be the Customer (cont..)

During the review if you don’t understand it, don’t assume that it’s correct and go on.

Eventually you will have to use specification to design your test cases

Finding bugs along the way is better. Don’t forget about software security. Security must be specified in details.

Research Existing Standards and Guidelines

Difference between standards and guidelines is a matter of degree.

A standard is much more firm than a guideline. Standards should be strictly adhered to. Guidelines are optional but should be followed. As a tester your job is not to define what

guidelines and standards should be applied. This lies with Project Manager.

Several examples of standards and guidelines

Corporate Terminology and Conventions: – If the software is tailored for a specific company, it

should adopt the common terms and conventions used by the employees of that company.

Industry Requirements:– Medical, pharmaceutical, industrial, and financial

industries have very strict standards that their software must follow.

Several examples of standards and guidelines

Government Standards:– The government, especially the military, has strict

standards. Graphical User Interface (GUI):

– There are published standards and guidelines of Microsoft and Apple Macintosh Operating Systems

– How the software should look and fee to a user. Security: Software and its interfaces and protocols

may need to meet certain security standards or levels.

Review and Test Similar Software

Some things to look for when reviewing competitive products include:

Scale:– Fewer or greater features– Less or more code– Will that size difference matter in your testing?

Complexity:– More or less complex– Will this impact your testing?

Review and Test Similar Software

Testability:– Will you have the resources, time, and expertise to

test software such as this? Quality/Reliability:

– Is this software representative of the overall quality planned for your software?

– Will your software be more or less reliable? Security:

– How does the competitor’s software security?

Review and Test Similar Software

There is no substitute for hands-on experience.– Use the similar software– Bang on it.– Demonstrate it.

Gain a lot of experience and it will help during the review of specification.

Review and Test Similar Software

Don’t forget about reading online and printed software reviews and articles about the competitive software.

This can be especially helpful for security issues

They will be well know in the press.

Low-Level Specification Test Techniques: Specification Attributes Checklist

Complete. Is anything missing or forgotten? Is it thorough? Does it include everything necessary to make it stand alone?

Accurate. Is the proposed solution correct? Does it properly define the goal? Are there any errors?

Precise, Unambiguous, and Clear. Is the description exact and not vague? Is there a single interpretation? Is it easy to read and understand?

Consistent. Is the description of the feature written so that it doesn't conflict with itself or other items in the specification?

Specification Attributes Checklist

Relevant. Is the statement necessary to specify the feature? Is it extra information that should be left out? Is the feature traceable to an original customer need?

Feasible. Can the feature be implemented with the available personnel, tools, and resources within the specified budget and schedule?

Code-free. Does the specification stick with defining the product and not the underlying software design, architecture, and code?

Testable. Can the feature be tested? Is enough information provided that a tester could create tests to verify its operation?

Specification Terminology Checklist

Always, Every, All, None, Never. If you see words such as these that denote something as certain or absolute, make sure that it is, indeed, certain. Put on your tester's hat and think of cases that violate them.

Certainly, Therefore, Clearly, Obviously, Evidently. These words tend to persuade you into accepting something as a given. Don't fall into the trap.

Some, Sometimes, Often, Usually, Ordinarily, Customarily, Most, Mostly. These words are too vague. It's impossible to test a feature that operates "sometimes."

Specification Terminology Checklist

Etc., And So Forth, And So On, Such As. Lists that finish with words such as these aren't testable. Lists need to be absolute or explained so that there's no confusion as to how the series is generated and what appears next in the list.

Good, Fast, Cheap, Efficient, Small, Stable. These are unquantifiable terms. They aren't testable. If they appear in a specification, they must be further defined to explain exactly what they mean.

Handled, Processed, Rejected, Skipped, Eliminated. These terms can hide large amounts of functionality that need to be specified.

If…Then…(but missing Else). Look for statements that have "If…Then" clauses but don't have a matching "Else." Ask yourself what will happen if the "if" doesn't happen.