Software Testing T A A T - Anuradha Bhatia | Simply ... · PDF file1/5/2014 ·...
Transcript of Software Testing T A A T - Anuradha Bhatia | Simply ... · PDF file1/5/2014 ·...
Software Testing
Anuradha Bhatia 1
DEFECT MANAGEMENT CONTENTS
I. Introduction, Defect Classification, Defect Management Process
II. Defect Life Cycle, Defect Template
III. Estimate Expected Impact of a Defect, Techniques for Finding Defects, Reporting a Defect.
Software Testing
Anuradha Bhatia 2
I. Introduction i. Software defects are expensive.
ii. The cost of finding and correcting defects represents one of the most expensive software development activities.
iii. While defects may be inevitable, we can minimize their number and impact on our projects.
iv. To do this development teams need to implement a defect management process that focuses on preventing defects, catching defects as early in the process as possible, and minimizing the impact of defects.
v. A little investment in this process can yield significant returns.
1. Defect Classification (Question: Explain the defect classification. – 8 Marks)
i. A Software Defect / Bug is a condition in a software product which does not
meet a software requirement (as stated in the requirement specifications) or
end-user expectations (which may not be specified but are reasonable).
ii. In other words, a defect is an error in coding or logic that causes a program to
malfunction or to produce incorrect/unexpected results.
iii. A program that contains a large number of bugs is said to be buggy.
iv. Reports detailing bugs in software are known as bug reports.
v. Applications for tracking bugs are known as bug tracking tools.
vi. The process of finding the cause of bugs is known as debugging.
vii. The process of intentionally injecting bugs in a software program, to estimate
test coverage by monitoring the detection of those bugs, is known
as bebugging.
There are various ways in which we can classify.
Severity Wise: i. Major: A defect, which will cause an observable product failure or departure
from requirements.
ii. Minor: A defect that will not cause a failure in execution of the product.
iii. Fatal: A defect that will cause the system to crash or close abruptly or effect other applications.
Type of Errors Wise i. Comments: Inadequate/ incorrect/ misleading or missing comments in the
source code
Software Testing
Anuradha Bhatia 3
ii. Computational Error: Improper computation of the formulae / improper business validations in code.
iii. Data error: Incorrect data population / update in database
iv. Database Error: Error in the database schema/Design
v. Missing Design: Design features/approach missed/not documented in the design document and hence does not correspond to requirements
vi. Inadequate or sub optimal Design: Design features/approach needs additional inputs for it to be complete Design features described does not provide the best approach (optimal approach) towards the solution required
vii. In correct Design: Wrong or inaccurate Design
viii. Ambiguous Design: Design feature/approach is not clear to the reviewer. Also includes ambiguous use of words or unclear design features.
ix. Boundary Conditions Neglected: Boundary conditions not addressed/incorrect
x. Interface Error: Internal or external to application interfacing error, Incorrect handling of passing parameters, Incorrect alignment, incorrect/misplaced fields/objects, un friendly window/screen positions
xi. Logic Error: Missing or Inadequate or irrelevant or ambiguous functionality in source code
xii. Message Error: Inadequate/ incorrect/ misleading or missing error messages in source code
xiii. Navigation Error: Navigation not coded correctly in source code
xiv. Performance Error: An error related to performance/optimality of the code
xv. Missing Requirements: Implicit/Explicit requirements are missed/not documented during requirement phase
xvi. Inadequate Requirements: Requirement needs additional inputs for to be complete
xvii. Incorrect Requirements: Wrong or inaccurate requirements
xviii. Ambiguous Requirements: Requirement is not clear to the reviewer. Also includes ambiguous use of words – e.g. Like, such as, may be, could be, might etc.
xix. Sequencing / Timing Error: Error due to incorrect/missing consideration to timeouts and improper/missing sequencing in source code.
xx. Standards: Standards not followed like improper exception handling, use of E & D Formats and project related design/requirements/coding standards
xxi. System Error: Hardware and Operating System related error, Memory leak
xxii. Test Plan / Cases Error: Inadequate/ incorrect/ ambiguous or duplicate or missing - Test Plan/ Test Cases & Test Scripts, Incorrect/Incomplete test setup
xxiii. Typographical Error: Spelling / Grammar mistake in documents/source code
Software Testing
Anuradha Bhatia 4
xxiv. Variable Declaration Error: Improper declaration / usage of variables, Type mismatch error in source code
Status Wise:
i. Open
ii. Closed
iii. Deferred
iv. Cancelled
2. Defect Management Process (Question: Explain the defect management process in software testing with a neat diagram. – 4 Marks)
Figure 1: Defect management Process
i. Defect Prevention-- Implementation of techniques, methodology and standard processes to reduce the risk of defects.
ii. Deliverable Baseline-- Establishment of milestones where deliverables will be considered complete and ready for further development work. When a deliverable is base lined, any further changes are controlled. Errors in a deliverable are not considered defects until after the deliverable is base lined.
iii. Defect Discovery-- Identification and reporting of defects for development team acknowledgment. A defect is only termed discovered when it has been documented and acknowledged as a valid defect by the development team member(s) responsible for the component(s) in error.
iv. Defect Resolution-- Work by the development team to prioritize, schedule and fix a defect, and document the resolution. This also includes notification back to the tester to ensure that the resolution is verified.
Software Testing
Anuradha Bhatia 5
II. Defect life cycle and Defect Template 1. Defect life cycle
(Question: Explain the defect life cycle or a bug life cycle in software
testing. – 8 Marks) i. Defect Life Cycle (Bug Life cycle) is the journey of a defect from its
identification to its closure.
ii. The Life Cycle varies from organization to organization and is governed by the
software testing process the organization or project follows and/or the Defect
tracking tool being used.
Nevertheless, the life cycle in general resembles the following:
Figure 2: Bugs Life Cycle
Status Alternative Status
NEW
ASSIGNED OPEN
DEFERRED
DROPPED REJECTED
Software Testing
Anuradha Bhatia 6
COMPLETED FIXED, RESOLVED, TEST
REASSIGNED REOPENED
CLOSED VERIFIED
Table 1: Defect Status
Defect Status Explanation
i. NEW: Tester finds a defect and posts it with the status NEW. This defect is yet
to be studied/approved. The fate of a NEW defect is one of ASSIGNED,
DROPPED and DEFERRED.
ii. ASSIGNED / OPEN: Test / Development / Project lead studies the NEW defect
and if it is found to be valid it is assigned to a member of the Development
Team. The assigned Developer’s responsibility is now to fix the defect and have
it COMPLETED. Sometimes, ASSIGNED and OPEN can be different statuses. In
that case, a defect can be open yet unassigned.
iii. DEFERRED: If a valid NEW or ASSIGNED defect is decided to be fixed in
upcoming releases instead of the current release it is DEFERRED. This defect is
ASSIGNED when the time comes.
iv. DROPPED / REJECTED: Test / Development/ Project lead studies the NEW
defect and if it is found to be invalid, it is DROPPED / REJECTED. Note that the
specific reason for this action needs to be given.
v. COMPLETED / FIXED / RESOLVED / TEST: Developer ‘fixes’ the defect that is
ASSIGNED to him or her. Now, the ‘fixed’ defect needs to be verified by the
Test Team and the Development Team ‘assigns’ the defect back to the Test
Team. A COMPLETED defect is either CLOSED, if fine, or REASSIGNED, if still
not fine.
vi. If a Developer cannot fix a defect, some organizations may offer the following
statuses:
Won’t Fix / Can’t Fix: The Developer will not or cannot fix the defect due
to some reason.
Can’t Reproduce: The Developer is unable to reproduce the defect.
Need More Information: The Developer needs more information on the
defect from the Tester.
vii. REASSIGNED / REOPENED: If the Tester finds that the ‘fixed’ defect is in fact
not fixed or only partially fixed, it is reassigned to the Developer who ‘fixed’ it.
A REASSIGNED defect needs to be COMPLETED again.
viii. CLOSED / VERIFIED: If the Tester / Test Lead finds that the defect is indeed
fixed and is no more of any concern, it is CLOSED / VERIFIED. This is the happy
ending.
Software Testing
Anuradha Bhatia 7
2. Defect Template (Question: Create the bug template for a login form. – 4Marks)
i. Reporting a bug/defect properly is as important as finding a defect.
ii. If the defect found is not logged/reported correctly and clearly in bug tracking
tools (like Bugzilla, ClearQuest etc.) then it won’t be addressed properly by the
developers, so it is very important to fill as much information as possible in the
defect template so that it is very easy to understand the actual issue with the
software.
1. Sample defect template
Abstract :
Platform :
Testcase Name :
Release :
Build Level :
Client Machine IP/Hostname :
Client OS :
Server Machine IP/Hostname :
Server OS :
Defect Type :
Priority :
Sevierity :
Developer Contacted :
Test Contact Person :
Attachments :
Any Workaround :
Steps to Reproduce
1.
2.
3.
Expected Result:
Actual Result:
Software Testing
Anuradha Bhatia 8
2. Defect report template i. A defect reporting tool is used and the elements of a report can vary.
ii. A defect report can consist of the following elements.
ID Unique identifier given to the defect. (Usually Automated)
Project Project name.
Product Product name.
Release Version Release version of the product. (e.g. 1.2.3)
Module Specific module of the product where the defect was detected.
Detected Build
Version
Build version of the product where the defect was detected
(e.g. 1.2.3.5)
Summary Summary of the defect. Keep this clear and concise.
Description
Detailed description of the defect. Describe as much as possible
but without repeating anything or using complex words. Keep
it simple but comprehensive.
Steps to Replicate
Step by step description of the way to reproduce the defect.
Number the steps.
Actual Result The actual result you received when you followed the steps.
Expected Results The expected results.
Attachments Attach any additional information like screenshots and logs.
Remarks Any additional comments on the defect.
Software Testing
Anuradha Bhatia 9
Defect Severity Severity of the Defect.
Defect Priority Priority of the Defect.
Reported By The name of the person who reported the defect.
Assigned To
The name of the person that is assigned to analyze/fix the
defect.
Status The status of the defect. (See Defect Life Cycle)
Fixed Build Version
Build version of the product where the defect was fixed (e.g.
1.2.3.9)
Table 2: Defect Report Template
3. Defect tracking tools
Following are some of the commonly used defect tracking tools:
i. Bugzilla - Open Source Bug Tracking.
ii. Testlink - Open Source Bug Tracking.
iii. ClearQuest – Defect tracking tool by IBM Rational tools.
iv. HP Quality Center– Test Management tool by HP.
v.
III. Estimate Expected Impact of a Defect, Techniques for Finding Defects, Reporting a Defect.
1. Estimate Expected Impact of a Defect i. There is a strong relationship between the number of test cases and the
number of function points. ii. There is a strong relationship between the number of defects and the
number of test cases and number of function points. iii. The number of acceptance test cases can be estimated by multiplying the
number of function points by 1.2. iv. Acceptance test cases should be independent of technology and
implementation techniques. v. If a software project was 100 function points the estimated number of test
cases would be 120. vi. To estimate the number of potential defects is more involved.
Software Testing
Anuradha Bhatia 10
a) Estimating Defects i. Intuitively the number of maximum potential defects is equal to the
number of acceptance test cases which is 1.2 x Function Points.
b) Preventing, Discovering and Removing Defects i. To reduce the number of defects delivered with a software project an
organization can engage in a variety of activities. ii. While defect prevention is much more effective and efficient in reducing
the number of defects, most organization conduct defect discovery and removal.
iii. Discovering and removing defects is an expensive and inefficient process. iv. It is much more efficient for an organization to conduct activities that
prevent defects.
c) Defect Removal Efficiency i. If an organization has no defect prevention methods in place then they are
totally reliant on defect removal efficiency.
Figure 3: Defect Removal Efficiency
1. Requirements Reviews up to 15% removal of potential defects. 2. Design Reviews up to 30% removal of potential defects. 3. Code Reviews up to 20% removal of potential defects. 4. Formal Testing up to 25% removal of potential defects.
Software Testing
Anuradha Bhatia 11
d) Defect Discovery and Removal
Size in Function Points
Totals Defects Remaining
Max Defects Perfect Medium Poor
100 120 12 66 102
200 240 24 132 204
500 600 60 330 510
1,000 1,200 120 660 1,020
2,500 3,000 300 1,650 2,550
5,000 6,000 600 3,300 5,100
10,000 12,000 1,200 6,600 10,200
20,000 24,000 2,000 13,200 20,400
Table 3: Defect Removal and Recovery
i. An organization with a project of 2,500 function points and was about medium at defect discovery and removal would have 1,650 defects remaining after all defect removal and discovery activities.
ii. The calculation is 2,500 x 1.2 = 3,000 potential defects. iii. The organization would be able to remove about 45% of the defects or
1,350 defects. iv. The total potential defects (3,000) less the removed defects (1,350) equals
the remaining defects of 1,650.
e) Defect Prevention
If an organization concentrates on defect prevention (instead of defect detection) then the number of defects inserted or created is much less. The amount of time and effort required to discover and remove this defects is much less also.
i. Roles and Responsibilities Clearly Defined up to 15% reduction in number of defects created
ii. Formalized Procedures up to 25% reduction in number of defects created iii. Repeatable Processes up to 35% reduction in number of defects created iv. Controls and Measures in place up to 30% reduction in number of defects
created.
Software Testing
Anuradha Bhatia 12
2. Techniques to find defects (Question: Explain any two techniques to find the defect with strength and weakness. – 8 Marks)
a) Quick Attacks:
i. Strengths
The quick-attacks technique allows you to perform a cursory analysis
of a system in a very compressed timeframe.
Even without a specification, you know a little bit about the software,
so the time spent is also time invested in developing expertise.
The skill is relatively easy to learn, and once you've attained some
mastery your quick-attack session will probably produce a few bugs.
Finally, quick attacks are quick.
They can help you to make a rapid assessment. You may not know the
requirements, but if your attacks yielded a lot of bugs, the
programmers probably aren't thinking about exceptional conditions,
and it's also likely that they made mistakes in the main functionality.
If your attacks don't yield any defects, you may have some confidence
in the general, happy-path functionality.
ii. Weaknesses
Quick attacks are often criticized for finding "bugs that don't matter"—
especially for internal applications.
While easy mastery of this skill is a strength, it creates the risk that
quick attacks are "all there is" to testing; thus, anyone who takes a two-
day course can do the work.
b) Equivalence and Boundary Conditions
i. Strengths
Boundaries and equivalence classes give us a technique to reduce an
infinite test set into something manageable.
They also provide a mechanism for us to show that the requirements
are "covered".
ii. Weaknesses
The "classes" in the table in Figure 1 are correct only in the mind of the
person who chose them.
Software Testing
Anuradha Bhatia 13
We have no idea whether other, "hidden" classes exist—for example, if a numeric number that represents time is compared to another time as a set of characters, or a "string," it will work just fine for most numbers.
c) Common Failure Modes
i. Strengths
The heart of this method is to figure out what failures are common for
the platform, the project, or the team; then try that test again on this
build.
If your team is new, or you haven't previously tracked bugs, you can
still write down defects that "feel" recurring as they occur—and start
checking for them.
ii. Weaknesses
In addition to losing its potency over time, this technique also entirely
fails to find "black swans"—defects that exist outside the team's recent
experience.
The more your team stretches itself (using a new database, new
programming language, new team members, etc.), the riskier the
project will be—and, at the same time, the less valuable this technique
will be.
d) State-Transition Diagrams
Figure 4: State Transition Map
Software Testing
Anuradha Bhatia 14
i. Strengths
Mapping out the application provides a list of immediate, powerful test
ideas.
Model can be improved by collaborating with the whole team to find
"hidden" states—transitions that might be known only by the original
programmer or specification author.
Once you have the map, you can have other people draw their own
diagrams, and then compare theirs to yours.
The differences in those maps can indicate gaps in the requirements,
defects in the software, or at least different expectations among team
members.
ii. Weaknesses
The map you draw doesn't actually reflect how the software will
operate; in other words, "the map is not the territory."
Drawing a diagram won't find these differences, and it might even give
the team the illusion of certainty.
Like just about every other technique on this list, a state-transition
diagram can be helpful, but it's not sufficient by itself to test an entire
application.
e) Use Cases and Soap Opera Tests
Use cases and scenarios focus on software in its role to enable a human being
to do something.
i. Strengths
Use cases and scenarios tend to resonate with business customers, and
if done as part of the requirement process, they sort of magically
generate test cases from the requirements.
They make sense and can provide a straightforward set of confirmatory
tests. Soap opera tests offer more power, and they can combine many
test types into one execution.
Software Testing
Anuradha Bhatia 15
ii. Weaknesses
Soap opera tests have the opposite problem; they're so complex that
if something goes wrong, it may take a fair bit of troubleshooting to
find exactly where the error came from!
f) Code-Based Coverage Models
Imagine that you have a black-box recorder that writes down every single line of
code as it executes.
i. Strengths
Programmers love code coverage. It allows them to attach a number—
an actual, hard, real number, such as 75%—to the performance of their
unit tests, and they can challenge themselves to improve the score.
Meanwhile, looking at the code that isn't covered also can yield
opportunities for improvement and bugs!
ii. Weaknesses
Customer-level coverage tools are expensive, programmer-level tools
that tend to assume the team is doing automated unit testing and has
a continuous-integration server and a fair bit of discipline.
After installing the tool, most people tend to focus on statement
coverage—the least powerful of the measures.
Even decision coverage doesn't deal with situations where the decision
contains defects, or when there are other, hidden equivalence classes;
say, in the third-party library that isn't measured in the same way as
your compiled source code is.
Having code-coverage numbers can be helpful, but using them as a
form of process control can actually encourage wrong behaviours. In
my experience, it's often best to leave these measures to the
programmers, to measure optionally for personal improvement (and
to find dead spots), not as a proxy for actual quality.
g) Regression and High-Volume Test Techniques
People spend a lot of money on regression testing, taking the old test
ideas described above and rerunning them over and over.
Software Testing
Anuradha Bhatia 16
This is generally done with either expensive users or very expensive
programmers spending a lot of time writing and later maintaining
those automated tests.
i. Strengths
For the right kind of problem, say an IT shop processing files through a
database, this kind of technique can be extremely powerful.
Likewise, if the software deliverable is a report written in SQL, you can
hand the problem to other people in plain English, have them write
their own SQL statements, and compare the results.
Unlike state-transition diagrams, this method shines at finding the
hidden state in devices. For a pacemaker or a missile-launch device,
finding those issues can be pretty important.
ii. Weaknesses
Building a record/playback/capture rig for a GUI can be extremely
expensive, and it might be difficult to tell whether the application
hasn't broken, but has changed in a minor way.
For the most part, these techniques seem to have found a niche in
IT/database work, at large companies like Microsoft and AT&T, which
can have programming testers doing this work in addition to traditional
testing, or finding large errors such as crashes without having to
understand the details of the business logic.
While some software projects seem ready-made for this approach,
others...aren't.
You could waste a fair bit of money and time trying to figure out where
your project falls.
3. Reporting defects effectively (Question: Explain how defects can be effectively reported. – 4 Marks)
It is essential that you report defects effectively so that time and effort is not
unnecessarily wasted in trying to understand and reproduce the defect. Here are
some guidelines:
i. Be specific:
Specify the exact action: Do not say something like ‘Select Button B’.
Do you mean ‘Click Button B’ or ‘Press ALT+B’ or ‘Focus on Button B and
click ENTER’.
Software Testing
Anuradha Bhatia 17
In case of multiple paths, mention the exact path you followed: Do not say
something like “If you do ‘A and X’ or ‘B and Y’ or ‘C and Z’, you get D.”
Understanding all the paths at once will be difficult. Instead, say “Do ‘A and
X’ and you get D.” You can, of course, mention elsewhere in the report that
“D can also be got if you do ‘B and Y’ or ‘C and Z’.”
Do not use vague pronouns: Do not say something like “In Application A,
open X, Y, and Z, and then close it.” What does the ‘it’ stand for? ‘Z’ or, ‘Y’,
or ‘X’ or ‘Application A’?”
ii. Be detailed:
Provide more information (not less). In other words, do not be lazy.
Developers may or may not use all the information you provide but they
sure do not want to beg you for any information you have missed.
iii. Be objective:
Do not make subjective statements like “This is a lousy application” or “You
fixed it real bad.”
Stick to the facts and avoid the emotions.
iv. Reproduce the defect:
Do not be impatient and file a defect report as soon as you uncover a
defect. Replicate it at least once more to be sure.
v. Review the report:
Do not hit ‘Submit’ as soon as you write the report.
Review it at least once.
Remove any typing errors.