Best Practices

Compuware Best Practices for Testing

Compuware Best Practices for Testing 1 Compuware 2000 All Rights ReservedDo not copy or reproduce

Compuware

RESTRICTED RIGHTS NOTICE SHORT FORM(January, 2000)

Use, duplication, or disclosure by the Government is subject to the restrictions asset forth in subparagraph (c)(1)(ii) of the rights in Technical Data and Computer Software clause

at 52.227-7013.

Copyright 2000 by Compuware Corporation.

All rights reserved. No part of this document covered by the copyright hereon may be copied orreproduced by any meansgraphic, electronic or mechanical, including photocopying,

recording, taping or in information storage and retrieval systemswithout written permissionfrom the publisher.

Compuware, Abend-AID, File-AID, QACenter, QADirector, QAHiperstation, QALoad, QARun,QualityPoint and XPEDITER are trademarks or registered trademarks of Compuware

Corporation. All other company or product names are trademarks of their respective owners. 2000 Compuware Corporation.


Chapter 1 - Overview................................................................................................... 1-1Purpose of this Guide ........................................................................................................ 1-1Audience ............................................................................................................................ 1-1Scope .................................................................................................................................. 1-1

Chapter 2 - Initiating Testing...................................................................................... 2-1Developing a Successful Testing Project ........................................................................... 2-1Software Correctness ........................................................................................................ 2-1

Chapter 3 - Test Planning........................................................................................... 3-1Establishing a Test Strategy.............................................................................................. 3-1Other Test Strategy Considerations................................................................................ 3-10Test Types........................................................................................................................ 3-13Roles and Responsibilities ............................................................................................... 3-16Open Items....................................................................................................................... 3-17Obtaining Test Strategy Sign-Off ................................................................................... 3-17

Chapter 4 - Creating Test Plans.................................................................................. 4-1Introduction....................................................................................................................... 4-1Establishing a Test Plan .................................................................................................... 4-1Other Test Plan Considerations ........................................................................................ 4-9

Chapter 5 - Test Case Development ............................................................................ 5-1Create Decision Tree ......................................................................................................... 5-1Prioritize the Test Cases.................................................................................................... 5-2Document Test Procedures ............................................................................................... 5-2Document Test Data Requirements .................................................................................. 5-2Identify Candidates for Automation................................................................................. 5-2Automate Designated Test Cases ...................................................................................... 5-3

Chapter 6 - Test Environment Preparation................................................................. 6-1Document Initial Environmental Data.............................................................................. 6-1Document the Test Environment Configuration .............................................................. 6-2Identify Data Recovery Points........................................................................................... 6-2Establish Backup, Restoration and Construction Procedures ......................................... 6-2Create Initial Data State.................................................................................................... 6-3

Chapter 7 - Test Execution ......................................................................................... 7-1Review Test Execution Schedule from Test Plan.............................................................. 7-1Restore Test Execution Environment ............................................................................... 7-1


Execute Test Cases ............................................................................................................ 7-2

Chapter 8 - Test Results Analysis................................................................................ 8-1Perform a Flow Analysis ................................................................................................... 8-1Failure Discovery............................................................................................................... 8-2Success Discovery .............................................................................................................. 8-2Measurement Analysis ...................................................................................................... 8-3

Chapter 9 - Management Reporting............................................................................ 9-1Executive Summary........................................................................................................... 9-1Consolidate Measurement Reports ................................................................................... 9-2

Chapter 10 - Productivity Tools and Professional Services....................................... 10-1Introduction..................................................................................................................... 10-1NuMega Tools for Quality Windows Applications......................................................... 10-3Automated Testing Products........................................................................................... 10-5Interactive Analysis, Testing and Debugging Tools ..................................................... 10-14File and Data Management Tools ................................................................................. 10-22Fault Management Tools............................................................................................... 10-29Professional Services ..................................................................................................... 10-43

Attachment A Test Strategy Template ............................................................................ 1Attachment B Test Plan Template.................................................................................. 7Attachment C Test Environment Requirements........................................................... 15Attachment D Sample Test Case .................................................................................. 25Attachment E Sample Application Readiness Report................................................... 31Attachment F Sample Test Summary Report ............................................................... 37Attachment G Test Planning KPA ............................................................................... 41Attachment H Test Case Development KPA ................................................................ 43Attachment I Test Environment Preparation KPA ...................................................... 45Attachment J Test Execution KPA............................................................................... 47Attachment K Test Results Analysis KPA .................................................................... 49Attachment L Management Reporting KPA ................................................................ 51

Chapter 1 - Overview

Compuware Best Practices for Testing 1-1 Compuware 2000 All Rights ReservedDo not copy or reproduce

Chapter 1 - OverviewPurpose of this Guide

Compuware Best Practices for Testing will help an organization implement awell-structured, repeatable, highly efficient testing process.

It contains information pertaining to the activities performed in a typical testingprocess, how to establish this process and references to many Compuwareproducts that can help you automate a particular activity, including examples. Theguides objective is to help you put a successful testing process in place.

AudienceThis guide is a working reference for quality assurance managers, applicationmanagers, testing architects and project managers.

ScopeWhile there are many tasks in a typical testing project, Compuware Best Practicesfor Testing focuses explicitly on the issue of building a repeatable testing process.Other tasks, such as assessing current testing processes, building awareness andestimating size and budget, while important, are not within the scope of thisdocument. Those tasks may be briefly described and referenced but are notexamined in detail.

Note that while this guide recommends a set of processes, it does not advocatethis as the only way to test. Compuware advocates a standard testing process,centered on our QualityPoint testing methodology, which is the basis for much ofthe material in this guide. You can use this guide to enhance, complete or simplybenchmark your current testing process.

Chapter 2 Initiating Testing


Chapter 2 - Initiating TestingDeveloping a Successful Testing Project

One key to a successful software implementation is using software testing processesthat incorporate appropriate verification and validation activities; these ensure thatthe software works when it is implemented. Another key is having projectmanagement processes and project team infrastructure that effectively identify,manage and control testing activities, as well as provide effective communicationsabout the application's readiness throughout the entire testing effort.

Using the Compuware QualityPoint software testing process has a direct impact onthese two key success criteria. QualityPoint incorporates software testing processesand establishes a framework that directly addresses key characteristics of successfulsoftware projects. It ensures you have the correct software and an effective projectmanagement system.

The items below describe how QualityPoint directly impacts the areas of softwarecorrectness and software testing project management.

Software Correctness

Business Requirements

Business requirements are the actions or results necessary for a viable product andfor implementation. Business and technical requirements are supported by the testingeffort.

Business requirements describe what an application must do and how it will perform,but do not specify how it should be implemented. Define business requirements forthe software product or implementation during the first stage of softwaredevelopment. These requirements must be testable and traceable throughout productdevelopment stages. A testing process that does not explicitly support and link to


2-2 Compuware Best Practices for Testing Compuware 2000 All Rights Reserved

Do not copy or reproduce

the business requirements can lead to results that do not meet the organization'sexpectations. This means that resources were expended on testing activities thatdid not support the requirements.

Technical Requirements

Technical requirements describe both the environment and computer systemarchitecture in which the application must perform and integrate, as well as theperformance requirements of the system itself. Defining the technical requirementsfor the software product or implementation is performed during the first stage ofsoftware development. Technical requirements, like business requirements, must alsobe testable and traceable throughout product development stages, and similarimplications can result if the testing process does not explicitly support and link tothe technical requirements. Implications that could arise include:

1. Results that do not meet the organization's computer system architecture andperformance expectations.

2. Resources that are expended on testing activities that are not in support of therequirements.

3. Testing environments that are not adequate to validate the technicalrequirements.

4. Inadequate production environment and infrastructure implementation planning.

There are different types of technical requirements.

Environment Rules

Environment rules are the system infrastructure required to implement theapplication. These may include server and personal computer hardware configuration(memory, video, processor storage needs), software local and network operatingsystems and drivers, database storage and disk space requirements. For example:

the application must run on both the Microsoft Windows 95 and WindowsNT 4.0 operating systems

the application must run on local workstations that have a minimum of 350MHz processors and a minimum of 64 MB memory

the application's data repository must reside within an Oracle 8.0 database.

Performance Rules

Performance rules measure specific system characteristics such as networkutilization, processing speed, response time, resource consumption, throughput andefficiency. The following are examples of performance rules technicalrequirements:

Chapter 2- Initiating Testing


during peak performance hours (8:00 a.m. through 5:00 p.m.), server CPUactivity must not exceed 80%

screen changes within the application must not exceed five seconds.

Test Requirements

Business and technical requirements must be broken down by applicationfunctionality and tested to validate these requirements. The final product is a testrequirement.

For example, if a business requirement states that the system must allow for theaddition of new employees, then appropriate test requirements for this businessrequirement may include:

add a new U.S. employee via the main menu add a new employee from the Current Employee List screen.

Each test requirement may produce one or more test cases.

A process that does not thoroughly address the scope of the test requirements as wellas ensure that the test requirements are linked to the business and technicalrequirements can lead to such problems as:

business and technical requirements unfulfilled because they were notcompletely tested

resources spent on unnecessary or nonexistent test requirementsineffectiveuse of resources.

Furthermore, since test cases are derived from the test requirements, incomplete testrequirements can also lead to the development of an incomplete list of test cases.

A test case is a test procedure combined with a single data record from the data set.The test case is designed to fulfill the test requirements. A comprehensive testingprocess ensures that both the test case types and quantity are known based upon thelinks between the requirements and the understanding of the test approach.Furthermore, these links help ensure that the tests support business and technicalrequirements. Without this information and process, you may encounter suchproblems as:

inability to scope and identify the resources necessary for test executionwhat has to be tested and how long it will take to complete the testing

an incomplete list of test casesnot all business and technical requirementsmay be fully tested.

Software Metrics

Metrics are used in almost all disciplines to determine how well a process works.Within software correctness, metrics objectively evaluate progress toward meeting




the business and technical requirements and goals. Without objectivity, it is difficultto assess how much progress has been made, how much is left to do and potentialproblems or risks to address.

A metric is a number showing a relationship between two variables. A testing metricis a number showing a relationship between two testing variables. The number mustthen be compared with some standard or norm.

An example of a testing metric is the number of defects found during a specific testphase versus total number of system defects. This metric shows the percentage ofdefects identified as a result of testing. It measures test effectiveness.

Measurements are the objective data used to evaluate the process or performance.The lack of correct measurements makes it difficult to identify areas forimprovement. If data is not, or cannot be measured, it is difficult to assess ifimprovement efforts have been successful.

Measurements are a combination of metrics, and when analyzed together theyanswer a specific question. For example, the percentage of defects identified, as aresult of testing, is a metric measuring test effectiveness.

Software Testing MeasurementsConsider all of the following software testing measurements together when makingdecisions about the readiness of the application. These are only some of themeasurements you can use, so make sure that you choose measurements appropriateto your testing situation.

Test Coverage

Application

How many requirements were tested? The answer does not indicate how many testspassed or failed. This can be shown as a percentage or as a trend.

Metrics

number of test requirements number of business and technical requirements number of test cases per test requirement.

Implementation

1. Establish a requirements matrix with links from the business and technicalrequirements to the test cases.



2. Establish a counting scheme for business and technical requirements, testrequirements and test cases.

3. Execute the tests.

4. Count total number of test requirements.

5. Count total number of test cases per test requirement.

6. Count total number of test cases executed per test requirement.

7. Calculate the test requirement coverage percentage by dividing the number oftest cases executed per test requirement by the total number of test cases per testrequirement. This calculation can also be performed with the business andtechnical requirements.

8. To show trends, create a graph with the x-axis representing time and the y-axisrepresenting the test requirement coverage percentage.

9. Plot the percentage on the graph.

Interpretation

TE ST COV ERAGE

0

5

10

15

20

25

30

35

1 2 3 4 5 6

TIME

PER

CENT

AGE

Test coverage is based on the number of requirements covered during the testprocess. For trends, the line should slope upward over time. This means that the testcoverage approaches 100%.




Error Discovery

Application

When are defects detected and how many defects have been detected? The defectsmay be sorted by severity or represented as a total.

Metrics

number of defects severity of defects.

Implementation

1. Establish a discrepancy reporting procedure.

2. Classify defects by severity.

3. Count and log defects.

4. Create a graph with the x-axis representing time and the y-axis representing thenumber of defects by severity or total.

5. Plot defects over time.

Interpretation

Error discovery is based on the discovery rate of errors throughout the test process.

ERROR DISCOVERY

0

5

10

15

20

25

30

35

1 2 3 4 5 6

TIME

NUM

BER

OF D

EFEC

TS

The slope should decrease over time. This is especially true for defects classifiedwith the highest level of severity. You could decide to release the application if thetrend was toward lower priority defects. If there are many low or medium prioritydefects found near the targeted release date, this may be enough to delay release ofthe application.



Open Defects

Application

When are defects open and how many defects are open? The defects may be sortedby severity or represented as a total.

Metrics

number of defects severity of defects.

Implementation




4. Create a graph with the x-axis representing time and the y-axis representing thenumber of defects by severity or total.

5. Plot defects over time.

Interpretation

Open defects are based on the discovery rate of errors throughout the test process.

OPEN DEFECTS

0

5

10

15

20

25

30

35

1 2 3 4 5 6

TIME

NUM

BER

The slope should decrease over time. This is especially true for defects classifiedwith the highest level of severity. You could decide to release the application if thetrend was toward more low priority. If there are many low or medium prioritydefects found near the targeted release date, this may be enough to delay release ofthe application.




Testing Activity

Application

How much testing has occurred? This does not indicate how many tests have passedor failed and can be represented as a percentage or displayed as a trend.

Metrics number of test cases executed number of available test cases.

Implementation

1. Establish a counting scheme for tracking test cases.

2. Count the number of test cases that have been executed.

3. Count the number of available test cases.

4. Calculate a percentage by taking the number of test cases executed and dividingby the number of available test cases.

5. If a trend is being developed, create a graph with the x-axis representing time andthe y-axis representing the test requirement coverage percentage.

6. Plot the percentage.

Interpretation

Testing activity is based on the number of tests executed over the number ofavailable tests.

TEST ACTIVITY

0

5

10

15

20

25

30

35

1 2 3 4 5 6

TIME

PER

CENT

AGE

The trend should slope upward over time, and the percentage should approach 100.



Success Status

Application

How many tests have passed and how much of the application is working correctly?

Metrics number of passed test cases number of available test cases.

Implementation

1. Establish a counting scheme for tracking test cases.

2. Count the number of test cases that have passed.

3. Count the number of available test cases.

4. Calculate a percentage by taking the number of test cases passed and dividing bythe number of available test cases.

5. If a trend is being developed, create a graph with the x-axis representing time andthe y-axis representing the success status.

6. Plot the percentage.

Interpretation

Success status is based on the number of passed test cases over the number ofavailable tests.

SUCCESS STATUS

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6

TIME

PER

CENT

AGE

The trend should slope upward over time, and the percentage should approach 100.




Specific Risks

Application

This measurement tells which tests have failed and which tests have importance inregard to release of the application.

Metrics

number of open defects severity of open defects.

Implementation

1. Establish a requirement matrix that has links from the business and technicalrequirements to the test cases.

2. Establish a counting scheme for business and technical requirements, testrequirements and test cases.




6. Associate defects with the appropriate business or technical requirement.

7. Calculate a total specific risk percentage by taking the total number of opendefects and dividing it by the total number of business and technicalrequirements.

8. If a trend is being developed, create a graph with an x-axis representing time andthe y-axis representing the specific risk percentage.

9. Plot the total specific risk percentage.

10. Start another graph.

11. Calculate a subtotal specific risk trend by taking the number of open defects byseverity and plotting the totals per requirement over time.

Interpretation

Total specific risks are based on the total number of open defects divided by the totalnumber of business and technical requirements. This total can be further brokendown into the number of open defects divided by a specific business requirement.One more level of detail for specific risk can be obtained by dividing the number ofopen defects of a specific severity by the specific business requirement. The



breakdown from the total specific risks is intended to provide trend information inthe form of gross number totals.

Total Risk by all Business Requirements

The desired result is for the trend to be sloping downward over time, and thepercentage should be approaching zero.

Severity by Business Requirement




Total Specific Risks Priority

The resulting trend of each chart should slope downward over time, and thepercentage should approach zero.

Software Testing Project ManagementCost

It is essential to understand all aspects of cost, even though the primary driver maybe labor hours. A comprehensive testing process consists of activities that identify allpotential resource costs including hardware, software, staff labor, contract labor andany additional capital expenditure. These activities include, but are not limited to,identifying the test approach, understanding the environmental needs, identifying thetest cases and determining which test cases are to be automated. Not incorporatingthese types of activities within the testing process can lead to unexpectedexpenditures or having to react and identify options when the funds are not available.

Schedule

The test schedule plans the order of activities and identifies what needs to happenbefore and after testing. Using a comprehensive test approach identifiesdependencies. Then you can schedule the resources and test cases required to supportthe test approach and fulfill requirements. Poor preparation during initial planningcan result in inability to properly execute and analyze test cases. Unexpected delaysresult because time is needed to address the issues while resources remain idle,waiting for direction.

Software testing projects typically use past project data to allocate requirementresources for the testing effort. Depending on the data and the process you use, as



well as their relevancy to the current project, the schedule might be incomplete. Theprimary purpose of having a schedule is to assist in tracking and communicating theoverall progress of the project, usually based upon milestones. The schedule alsoprovides other benefits by identifying needed resources and potential schedule risks.Ensure that the amount of effort required does not exceed the amount of timescheduled for the testing projects completion.Therefore, to improve the schedule's accuracy, the test process must include allactivities required to complete testing. Test process includes test planning, test casedevelopment, test environment preparation, test execution, test results analysis andmanagement reporting and also such factors as:

the software testing environment tester skill levels schedule commitments to the customer corporate culture.

Without considering all aspects, it is highly probable that by the date the applicationunder test is to be implemented, not all testing will be completed since neither thetime nor the resources were appropriately allocated or the different types of testingneeded were not included. In this scenario, there is a high risk that the applicationbeing tested will not meet requirements and will be defective at the time of itsimplementation.

Resources

Resources may include specific hardware or machines, software or operatingsystems, skilled testers, facilities such as a dedicated testing laboratory or networkconnections and operational support.

Activities within the testing process that help identify both the resource types andappropriateness are those that identify:

what types of testing are needed what test cases must be executed what environment is needed what support tools are necessary to manage the project what tools are necessary to automate testing.

The sooner you identify these resources, the sooner you can react to any gaps(unavailability, skills) and the sooner you can either close the gaps (as much aspossible) or develop other plans. Not understanding the resources can lead toproblems such as:

the resources cannot execute the testing needed (the hardware cannot executethe performance test requirements)




the people are not skilled in understanding test results; therefore, the resultsmay not be reliable and may not accurately reflect the testing progress

resources are not available when needed; alternative plans (training, capitalprocurement and the like) to close any gaps cannot be executed within theallotted time.

Specific types of resource attributes include:

Appropriateness

The "right" resources assigned to the "right" task. For example, are only 386workstations available to test an application designed for a Pentium? Theappropriateness of skill levels is just as important; for example, are the peopleassigned to write test cases trained to write them?

Hardware/Software

Hardware and software resources refer to the specific hardware configurations,software requirements, infrastructure environmental needs and support tools neededthroughout the testing project. Areas of consideration:

will the hardware be available at test execution? has the software been installed? does the software require specific levels of service updates/releases?

Human

It is important to understand which people are needed, who is available and whatgaps need to be addressed. The types of attributes or characteristics to understandinclude skills, experience, expertise, authority and responsibility.

Risk

Risk is the chance of an undesirable event or an item that compromises the project'sgoals or requirements. Factors used to describe risk include the probability that theevent will occur and the impact of the event. According to the Software EngineeringInstitute, risk management can be illustrated as a set of functions that are identifiedthroughout the life cycle of a project:

Identify. Locate risks to the project before they become problems. Analyze. Evaluate the impact, probability and timeframe of the risk. Classify

the risks and then prioritize them. Plan. Translate risk information into decisions and mitigating actions

(present and future) and implement those actions. Track. Monitor the risks and mitigation actions. Control. Check for and correct deviations from the mitigation plans.



Communicate. Provide information and feedback, both internal and externalto the project, on the risk activities.

Risk management should be an ongoing activity. The ability to accurately identify,assess and manage the software testing risks relies heavily upon the completenessand accuracy of the testing process and its supporting activities. The less youunderstand about successfully planning, executing and completing any test processcomponent (test planning, execution, analysis and reporting), the less likely you willidentify or assess the risk properly. Without knowing the risks, it is increasinglydifficult to plan for what might go wrong, and therefore it is highly unlikely that theapplication requirements will be completed within expected project cost andschedule requirements.

Probability

Some testing activities can directly assist in assessing the likelihood that a risk willoccur. By understanding how much needs to be done, by when and by whom, youcan significantly improve probability assessment. For example, if it is known thatsome of the resources needed for testing do not meet requirements, it is likely thatmore time must be allotted for test case development and test execution. If resourceidentification, needs and gap analysis did not exist during test planning, theprobability of having available skilled resources wasnt assessed properly.

If significant process changes are to be implemented, the probability of risksoccurring during the affected process is likely to be high. Not knowing if changes arenecessary and what they would be, however, affects a manager's ability to accuratelyassess the associated risks.

Impact

Impact is another factor of riskthe assessment of what will be impacted and themagnitude of the consequence. Again, some of the testing process can help definewhat could be impacted (cost, effort, schedule, performance and correctness) and theextent to which it is impacted (high, medium, low).

Consider the risk of skilled resources being unavailable. A comprehensive testplanning effort produces a complete test project schedule and a resource matrix,identifying types of people and when they are needed. These tools provide themanager with the opportunity to create different scenarios and estimate the impact ofschedule delays and the associated cost overruns among other factors. Consequently,you can assess what might be impacted and the potential consequences.

Issue Resolution

It is essential to create and implement test processes that help identify problems.Identify problems as early as possible so you can develop and monitor action plansthat address and minimize their impact. If problems occur late in the process, the




costs associated with addressing them may be much higher than if they wereidentified sooner.

Similar to risk management, unidentified problems lead to unexpected results ordifficulties. This also leads to unfulfilled requirements or unmet expectations.

Communication

The primary objective of any communications material is to inform project personnelof expected results and goals as well as progress towards those goals. Effectivecommunication must also be:

understandable by the audience appropriate for the audience relevant to the application under test timely accurate.

Goals

If both the management and the project members do not have a clear understandingof project expectations, then the test goals and requirements may not be met. Projectpersonnel may not be aware of the key activities set by the management team onwhich to measure progress. Attempting to align expectations late in the project's lifecycle can lead to resource waste and can put the schedule, resource effort and cost athigh risk.

Status Reporting

Understanding the test planning process, understanding progress towards goals andinterpreting results and measurements are critical in improving a manager's ability toprovide effective status reports.

Even though the goals may be understood, not providing timely, accurate andappropriate status reporting can lead to:

Ineffective Risk Management and Issue Resolution. Unreliable,incomplete or untimely information affects a manager's ability to identifyrisks and problems, as well as creates necessary action plans that can beexecuted in the time remaining.

Ineffective Resource Utilization. Inaccurate information can result inunnecessary or inappropriate action plans being executed.



Organizational Support Issues. When not communicating, people have atendency to believe that information is "hidden" from them or that themanager does not have the project under control which will potentially leadto problems related to trust and confidence. Consequently, this can affect theproject manager's ability to provide additional resources for mitigation orcontingency planning.

Project MetricsMetrics are as important in project management as they are in achieving correctsoftware. Project management measurements typically reflect progress towardachievement of project management goals related to cost, schedule and effort.The lack of correct measurements makes it difficult to identify areas forimprovement. If the process cannot be measured, how can improvement efforts beconsidered successful and how can the magnitude of the improvements be assessedobjectively?Trend Analysis

Trend analysis is the ability to analyze measurements over a period of time to helpassess issues, risks and progress. Data and trend analysis may only be pertinent for aspecific project, or the manager may be able to use earlier project data to assist intrend analysis. For example, with historical rates of early defect detection, the teammay be able to forecast the defect detection rate on the current project.Not performing any trend analysis can result in:

inability to identify risks and issues early inability to develop appropriate action plans to address risks/issues as early as

possible potential ineffective resource utilization since resources may have been used

for unnecessary activities if projections had been available.

Developing the Staffing, Workspace Environment, Facilities, Trainingand External Support Plans

Staffing Plan

The staffing plan is a profile of the project team. This plan incorporates: generic role definition responsibilities skills needed quantity of resources required by role and skill planned start and end dates of team personnel planned cost per hour and actual cost per hour (optional).




You may develop an organization chart as supporting documentation. Anorganization chart contains the project team structure, including support personnel,such as application experts or technical staff. The chart should also indicate reportingrelationships.

Identifying and securing people to match the skills identified in the staffing plan iscrucial to the success of the project. An exact match is difficult to achieve because ofthe unavailability of qualified resources, business economics and schedules. Youmay need to adjust your project plan to accommodate replacements. If experiencedresources are not available, additional staff training may be required.

Roles and Responsibilities

The following paragraphs describe key positions in a testing team.

Application Owner manages the end-users or is the end-user of the application under test (AUT) serves as the customer for the AUT referred to as Application Subject Matter Expert (SME) in some cases

Business Analyst/Lead Business Analyst understands the business functions within the AUT defines business processes that support the AUT identifies business requirements reviews design and specifications defines the training and end-user documentation requirements assists with test execution assists in creating acceptance criteria reviews and analyzes the test results to determine application correctness represents the functional needs of the end-users to the project team

Business Consultant provides QualityPoint project leadership, working with the project manager

to implement QualityPoint software testing processes assists the Project Manager with the design, creation and implementation of

the QualityPoint processes supplies project control information to the client and provides support for

Senior Management instituting the changes required as a result ofimplementing QualityPoint

serves as a primary point of contact between the consultant's company andthe client



manages the consultant organization's resources assigned to the client'sQualityPoint project

Business Expert understands the business perspective of the AUT outlines the business processes that align with the AUT provides the application details such as workflow, required elements on input

screens and other details referred to as a Business Subject Matter Expert (SME) in some cases

Development Manager controls the process for developing and implementing software provides application-related information to the Business Consultant oversees testing activities from an application perspective assists in creating Business Requirements assists in creating acceptance criteria assists in identifying the informational flow process requirements between.

the development organization and the project team membersProject Manager

plans, integrates and executes activities to deliver products that meet theacceptance criteria

manages resources within the project and ensures that team membersunderstand roles and the logical links among QualityPoint activities

maintains an effective flow of communications develops, monitors and controls the QualityPoint project plan provides organizational and logistical support while the Business Consultant

is at the client site serves as a primary point of contact between the client's company and the

consultant's company

Project Sponsor has the overall responsibility and accountability for the success of the

QualityPoint project provides the communication link between the client's senior management

team and the project team provides clarification of critical issues and establishes priorities monitors project execution defines project expectations




Quality Assurance Analyst(s), Lead Quality Assurance Analyst provides leadership for testing effort oversees testing activities from a technical perspective assists in understanding the details of each software testing process and how

each process impacts the client monitors testing activities tracks and analyzes defects develops and tracks project metrics and measurements creates test reports assists in creating acceptance criteria, test phase exit criteria and suspension

and resumption criteria assists in test plan development

Quality Assurance Manager provides testing process and change control process information to the

business consultant establishes the software quality assurance goals oversees software configuration management and release management provides guidelines to support testing and quality practices responsible for test execution and results sign-off provides recommendations on how to proceed based upon the test results

analysis approves test plan

Senior Management appoints the project sponsor and project manager reviews and approves the test process assessment statement of work,

management report, QualityPoint project plan and application readinessreport

identifies the initial project that will be used to implement the client'sQualityPoint software testing process

provides input on cost, resource and schedule limitations and constraints inwhich the project must work

controls the environment factors that are beyond the control of the ProjectManager

establishes and communicates company direction and how the QualityPointproject supports that direction. Provides support for any changes required fora successful QualityPoint implementation

Team Manager, Team Leader



coordinates a defined subset of related project activities within theQualityPoint project

Technical Analyst(s), Lead Technical Analyst provides technical support for a specific application works for the Technical Support manager establishes physical setup of environment

Technical Expert understands the infrastructure elements of the application as it is

implemented understands the support components of the application such as network

configuration, networking protocols and others referred to as Technical Subject Matter Expert (SME) in some cases

Technical Support Manager controls the technical environment where the application under test will be

developed, tested and implemented assists in identifying the test and production environment requirements and

processes directs the resources required to build and maintain the test and production

environments controls the environment backup and restoration procedures

Test Manager provides testing process information to the Business Consultant establishes the test goals develops or approves the test plans provides guidelines to support testing practices responsible for test execution and results sign-off provides recommendations on how to proceed based upon the test results

analysis

Testing Personnel develops test plans executes tests and evaluates execution results logs and communicates test execution results

Workspace Environment

This section addresses many of the objects you must have in place before testing canbegin. These objects include the following:




Facilities

Where the project team will be working on a daily basis, and the facilities, such ascubicle, telephone and security access card are available to each team member. Alsoconsider common facilities, such as fax machine, conference room, printer andcopier, that will be available to the team. If commonly needed facilities are notavailable at the normal work site, determine how and where the team will gain accessto them when needed. Note any constraint on the size of the team that can beaccommodated at the work site. If the project team will normally be working atmultiple physical locations, identify any potential logistic problems.

Hardware

Identify the hardware, both mainframe and PC, you will need to conduct the projectand who will provide it. Include information such as model and size and the hoursand days of availability. List any known minimum requirements.

Software

Identify the software, both mainframe and PC, you will need to conduct the projectand who will provide it. Include information such as version, release and the numberof licenses and simultaneous users. This section may also include assumptions aboutany known minimum requirements.

Communication

Identify the communication links you will need to conduct the project and who willprovide them. List any known requirements.

Facilities and Fixed Assets Plan

A schedule of events and list of items that the Facilities and Information Technologydepartments require to prepare for the project.The plan for facilities should include:

securing cubicles or offices obtaining whiteboards accessing copy machines and faxes accessing conference rooms obtaining phones and voice-mail connecting to the LAN.

The plan for Information Technology should include: project timing software and hardware, including the configuration to support the workload



shared data areas internal and external communications electronic mail access and special requirements.

Develop your facilities plan at the same time as your staffing and project plans.Make special note of machine capacity. Your overall planning effort must ensureenough capacity is available for planned work.

Defining Training Requirements

Your primary training objective is to effectively transfer skills and knowledge abouttesting methods, concepts, procedures and tools to the people involved in testing.You should design training so that the people using the tools daily can be productiveimmediately. Tool training should occur prior to using the tool. If training occurs allat once, students are overwhelmed and the information is lost.

Identify the Skill Sets of Testing Personnel

Why Is This Important?

Documenting the skill sets of the test personnel:

indicates whether the people have the skills, knowledge and capabilitiesrequired for the activities assigned to them

indicates whether the people have remained current on software testing andsoftware quality assurance processes as they relate to current technologies,platforms, information technology solutions and applications

indicates a company's commitment to a quality program and the continuous.improvement of quality programs and initiatives.

Do those responsible for testing attend software testing and quality assuranceseminars/training? Are they members of a professional organization? Understandingthe types of software quality assurance and software testing training the projectpersonnel have attended, as well as relevant QA professional organizations that theyare involved in, indicates whether they are current on software quality assuranceprocesses.

A training budget indicates the company's commitment to quality and identifieswhether funds may be available for training if skill gaps exist.




How Does This Impact Quality?If people do not have the skills, knowledge or experience necessary, their work couldmake delivering correct software impossible. Examples of problems that can beencountered include:

the requirements may not be complete or correct inaccurate resolutions or long cycle times to complete resolution or both the test cases may not be accurate or appropriate (test requirements are not

complete or correct, test procedures are incorrect, test data is not appropriate,valid or correct)

the test results analysis may not be accurate (defects are incorrectlycategorized, assumptions that a passed test case equates to a valid result areoften made, data flow issues are not identified)

it will be difficult to identify appropriate metrics and measurements thatobjectively and accurately reflect progress toward fulfillment of requirements

the test approach may be defined incorrectly. An incorrect test approach canhave the following implications: high risk of planning inaccuracies (too much or too little testing in

planned) incorrect environmental needs assessment the types of people needed and when they are needed may not be assessed

correctly the list of test cases may not be appropriate (they may not support the test

approach; the number could be too little or too much). testing activities may proceed when issues and risks exist that should be

addressed skill gaps not known can affect the ability to accurately assess the probability

associated with a risk; therefore, the appropriate effort and attention may notbe given to the monitoring, mitigation and contingency planning associatedwith a particular risk. The result may be that you have to react quickly to anearlier identified risk late in the project. The impact to schedule andresources to meet the requirements may also be greater than what waspredicted.

Potential Recommendations

Establish a training program for test team members that includes training on theapplication(s) under test, as well as the QualityPoint software testing process.

External Support Staff

A test team needs to rely on experts in the organization to assist with building thetesting process and with running and verifying test results.



Following is a partial list of key personnel you will need to assist your test team.

The Project Manager and his or her staff can provide assistance with thesequencing of applications to be tested. The Project Manager can help with theprocurement of the necessary testing equipment, office space, work facilities,computer resources, financing, human resources, outside services, tool acquisitionand other high level management tasks.

A Database Administrator (DBA) may be needed to assist with the generation andpopulation of test databases to assess each application. Your reliance on the DBAwill depend on the current availability of test databases. If new test databases arerequired, the time, resources and effort to build them must be planned and scheduled.

The process of creating a subset of database data may also require the expertise ofthe DBA to make the test team aware of application relationships between differenttables, databases and flat files.

Another important function of the DBA is to ensure that test databases are backed upand refreshable. If there are database errors encountered during testing, the expertiseof the DBA can save a lot of time. A DBA will be able to quickly analyze, recognizeand resolve problems.

You may need the assistance of OLTP Technical Support (CICS and other). Forsystems with on-line transaction processing (OLTP) components, the need for one ormore test versions of the on-line system is common. The effort to build a test on-linesystem is often difficult and requires coordination between DBAs and the test teamto ensure that the correct data and program set are referenced during testing. Planand schedule sufficient time to build any new on-line test systems that are needed.

The DASD Administrator needs to be aware of your disk space requirements sothat sufficient space can be provided. For most mainframe shops, disk space is at apremium. The space required for each application must be calculated. Due to thelarge volume of DASD required for testing, someone needs to oversee or manage it.

Identify and plan interactions with your Security Administrator to expedite accessto needed data and resources. Include obtaining access to other platforms such asmidrange, intermediate systems and other servers. Delays in accessing neededresources and files during the testing process should be eliminated.

The Technical Support person is responsible for installing and maintaining thetesting tools needed for your testing organization.

In a mainframe environment, the Technical Support department usually has systemsprogrammers who are responsible for all software installation. It is important tonotify Technical Support of any new tools or new releases of products you need foryour testing organization. Scheduling of technical support resources and specialinstallation requirements, like initial program loads (IPLs), may be difficult. Getstarted early.




You must communicate with your Network Support personnel. Access tocommunications facilities, local area network (LAN) servers and printers,workstation configuration, electronic mail and other infrastructure needed to supportthe test team must be available and reliable. Make sure these facilities are in placeand fully supported via a help desk or LAN expert.

You may need to get information on the contents, scheduling and frequency ofproduction jobs, or you may need to schedule special overnight test data builds. Forthese tasks, involve the Operations department and Job Scheduling staff such as aScheduling Administrator, if a job scheduling package is used. Make sure they areaware of your status as you begin to build the test process and as you start testingeach application.

Chapter 3-Test Planning


Chapter 3 - Test PlanningEffective test strategy and test plans are the foundation for successful testing. Thetest strategy is created at the same time the application requirements are defined.The test plan provides the details that, when implemented, will meet theobjectives outlined in the test strategy. Attachment B shows an example test plan.The first purpose of test planning is to identify the items being tested, the testingtasks to be performed, the personnel responsible for each task and the risksassociated with the plan. The second purpose is to develop the scope, approach,resources and schedule of the testing activities. Thorough test planning is criticalto the success of the testing process.

Ideally, the test plan is developed at the time the application requirements aredefined. This provides the test team adequate time to define the tests, locate andconfigure the hardware and software resources, locate and train the humanresource and schedule the tests. The test plan is a "living document" that willchange as the application functions become more clearly defined and stable.

The test team, the application development team, all user groups and managementreview and approve the test strategy and test plan. Once the test strategy has beencompleted, the test plan can be created.

The process for creating the test plan, along with the test plan itself, should bereviewed and agreed to by the test team, the development team, the user/customerteam and ultimately, the management team. This step cannot be overemphasizedbecause it establishes commitment from each contributing team.

Establishing a Test Strategy

The following pages detail how to establish a test strategy, giving the objectivesof each area, and the reasons that it is important to define and document thoseitems. Attachment A shows a sample test strategy template.




Identifier

ObjectiveThe identifier section allows an identification number to be assigned to the teststrategy document that supports your document naming conventions.

Why Is This Important?The identification number is used to cross-reference the test strategy documentwith other documents in the project. A unique identifier makes the documenteasier to recognize in a listing of documents.

Purpose

ObjectiveTo identify the appropriate readership, communicate what the reader(s) shouldexpect to understand after reading the test strategy and provide a brief descriptionof the overall testing strategy. This is achieved by providing a brief description ofthe document's content, the level of detail contained within and the applicationthat will be tested during the project's test effort.

Why Is This Important?This section provides an introduction for the test strategy and a background of theproject and the AUT. It provides an overview of the test strategy and helps set theexpectation for the amount of effort required for the project.

Goals of the Test Effort

ObjectiveThe purpose of the goals section is to identify and document the goals that thetesting effort is to support and achieve.

Why Is This Important?Goals provide an overview of the force driving the project. These forces can bethe need to correct errors, add new functionality and increase current responsetimes. When the goals of the project are specified, testing efforts can be readilytailored to fulfill and achieve those goals.



Scope of the Test Effort

ObjectiveThe purpose of the test scope section is to document and identify the sections ofthe AUT and the business and technical requirements that will and will not betested.

Why Is This Important?Documenting test scope aids in determining project timelines as well asestimating effort. Once business and technical requirements have been defined,key and related deliverables can be identified. After requirements have beendefined, the test objectives can be outlined and responsibilities can be assigned.Efforts can then be focused on those tests that have a higher priority in reachingthe project's objectives.

Test Environment

ObjectiveThe purpose of the test environment section is to document the environment inwhich the tests are to be executed.

Why Is This Important?Documenting the physical components required for test execution helps identifypotential gaps in what is required and what actually exists; appropriate actionplans to obtain closure can then be developed. Consider such things as: operating system version (remember to identify service packs, fix packs and

patches) networking software and version communications protocols

server and workstation machines.

References

ObjectiveThe purpose of the reference section is to identify supplemental information thatmight be helpful to the reader as well as a list of terminology that may be usedthroughout the document and the test effort.




Why Is This Important?Documenting the references provides a way to define specific terminology andacronyms. These terms can be used throughout all documents to provide acommon frame of reference. Using these terms helps to customize documents tobe specific to you and your needs.

The Test Effort's Organization Needs

ObjectiveTo identify the type of personnel required to conduct the testing effort.

Why is This Important?Identifies the required resources for the testing activities to be performed. Theseactivities can include test planning, test case creation, test script development, testtool support, among others. Once the needed personnel are identified, individualtraining needs can be determined and training programs can be created if theavailable people do not meet the skill and knowledge levels required to meet theexpected schedule.

Test Types

ObjectiveTo document the testing to be completed during this test effort.

Why Is This Important?Documenting the test types needed for this effort helps define the testing activitythat will be undertaken. It also provides you with some indication of the resourcesand test environments required for the testing activity. Each of the test typesshould be considered; however, not every type of test is typically employed. Forexample, it is not necessary to do unit testing if the software is vendor-supplied.

Mapping the test types to the test goals assists in directing test activity andvalidates that the testing is actually needed.

Test Methods

ObjectiveTo document the methods of testing to be utilized during test execution.

Why Is This Important?



Documenting the test methods to be used in the test executions helps define theway testing activity will be performed. It also provides you with some indicationof whether testing tools will be required for the testing activity. Also, projectmanagement tools can be identified while completing this section.

Testing Key Process Areas

ObjectiveTo identify the QualityPoint testing key process areas (KPAs) that are applicablefor this testing effort. A brief description of each KPA and expected outputs arealso documented.

Why Is This Important?By reviewing brief outlines of each of the key process areas, you can determinewhich of the areas best suits your needs. KPAs can be used as needed during theproject and can be used independently of each other to complete the entire project.

Acceptance Criteria

ObjectiveThe purpose of the acceptance criteria section is to document what constitutesacceptance of the application based on quantifiable measures.

Why Is This Important?The process helps identify criteria required to help ensure that both applicationsmeet any applicable contractual or regulatory requirements or both; for example,food, pharmaceuticals and nuclear energy industries must meet Food and DrugAdministration regulations.

The criteria can be used to identify appropriate quality control activities andmeasurements that should be included in the testing process.

Quality levels used to accept the software based on the test goals and test scopeallow the test team to determine and communicate a quantifiable state of theAUT. These states may include coverage statistics and pass/fail statistics, amongothers.




Associated Processes

ObjectiveTo document the applicable processes and procedures that must be in place tosupport a successful testing effort. It is acceptable to either reference processdocumentation or document the process.

Why Is This Important?By outlining each of the associated processes, you can determine procedures thatare crucial to ensure a successful testing effort. Examples of associated processesmay include: requirements management processes

configuration management processes

defect management processes.With appropriate documentation, associated processes can be standardized so thatall personnel involved in the testing effort will have a clear understanding of howto report, track or act upon any defects that may be found in the application undertest.

Test Effort Deliverables

ObjectiveTo provide a list of deliverables to be generated during the testing effort and uponits completion. Also, both your and Compuware's responsibilities in creating andapproving the deliverables are defined. For each deliverable, the acceptancecriteria and the approval process requirements are documented.

Why Is This Important?A list of deliverables provides additional detail regarding the expectations andrequirements early on in the testing effort. Understanding the deliverables canassist in estimating the effort and supporting process activities that must beincorporated into the schedule, especially if similar deliverables have beenrequired on past projects.Knowing the criteria for approving the deliverables and knowing who isresponsible for their approval improve the likelihood that the deliverable willmeet the expectations.

Knowing the acceptance criteria provides the opportunity to plan and implementmeasurements or quality control activities during the process to help improve thelikelihood of their acceptance and minimize rework efforts.



Test Effort Assumptions

ObjectiveTo identify the major assumptions that were used in developing the test strategy.This section should include any assumptions, which proved incorrect, may beused as a basis for invoking changes to the test strategy.

Why Is This Important?Assumptions act as constants when creating a test strategy or other projectdeliverable. They provide a basis to make decisions affecting testing efforts.Documenting assumptions provides a framework in which test measurements canbe validated. If an assumption is changed or has been proved inaccurate, allefforts associated with the assumption must be reviewed and adjusted since theycan no longer be used as accurate measures within the context of the test strategy.

Test Effort Constraints

ObjectiveTo identify specific boundaries and constraints that the test effort must operatewithin. Consider business and technical limitations and constraints as they relateto such items as schedule, resources, cost, computer infrastructure, software,security, regulations and other concerns.

Why Is This Important?If limitations and constraints are not understood, the planning process may notresult in a test strategy that will meet its objectives. Realization of limitations andconstraints may occur late in the testing effort as they are encountered.Consequently, management activities may become reactive instead of proactiveand contingency plans may have to be identified and implemented quickly.

Identifying the limitations and constraints early in the test strategy processprovides the opportunity to challenge them, in an effort to reduce or eliminatetheir potential impact. It also provides the opportunity to identify alternativeapproaches instead of implementing quick fixes that may not be optimal ifconstraints are identified late in the testing effort.

Compuware financial constraints are usually not included within this sectionbecause of the sensitivity of financial information.

It is possible that there are no known significant test effort constraints. If so, astatement to that effect should be included.




Test Effort Dependencies

ObjectiveTo identify test effort dependencies that are outside of the test team's control.Also, the purpose is to identify those dependencies that may be within theircontrol, but may have an impact on the test effort's success if they are notmanaged appropriately or are not successful.

Why Is This Important?Testing activities may not be scheduled at the right time or in the right sequence ifdependencies are not understood.

Tracking and communication mechanisms can be established to help monitor thedependencies; this information can then be used to assess the QualityPointschedule risks as well as to identify when risk and issues should be escalated topeople who could be able to provide assistance.

Mitigation and contingency plans can be developed for those dependencies thatare on the critical path or can have a significant impact on the test effort's successif they encounter difficulties.

Test Schedule

ObjectiveTo identify the testing planning details within the project planning schedule. Thisrequires the preliminary schedule of the test effort and needs to be updated frominformation gathered during test case development.

Why Is This Important?The test schedule controls all activities during the test effort. It also providesmanagement with an explicit representation of all test activities.

Determining if the test effort can be executed in the time allotted tells you thelikelihood of truly meeting the expected schedule based upon the level of effortthat is required.

If the current plan does not allow all testing efforts to be completed within theallotted time, you may still have time to identify alternative approaches to reducethe schedule or to request additional time. Having to identify alternatives toreduce the schedule late in the test effort could result in a solution that is notoptimal or does not meet the test goals.

Deadlines and fixed release dates will be a major factor in determining potentialtest schedule risks.



Identify Risks and Issues

ObjectiveTo be prepared for issues that could hinder or stop completion of the testingeffort.

Why Is This Important?This process determines different elements of risk to the testing efforts, includingrisk identification, risk probability and risk impact. Consider these issues:

will resources be available at the time they are needed?

are new processes being implemented?

will testing start early enough?

are requirements documented?

will training be held as necessary?

are there any verification activities prior to executing a test?

are there management changes?

Identifying risks and appropriate action plans can help avoid:

people spending time addressing problems that could have been avoided

requiring resources to address problems at inopportune times when theschedule does not allow for any slack.

Identifying risks also provides information and knowledge of consequences thatassist with decision making. Mitigation planning can minimize the impact of arisk should it actually occur. Without contingency planning, alternatives toresolve problems that occur may not be optimal or may not achieve the objectives.Contingency planning minimizes the disruption when a risk is realized; the teamdoes not have to spend effort determining what to do, and the cycle time toreceive an authorization to proceed should be minimal.




Other Test Strategy Considerations

Testing ApproachThere are four basic approaches to testing. Each approach has several variationsthat provide certain advantages and cause some disadvantages. This chapterexplains the approaches.

Full ProductionWith this approach, production files and data are used during testing. Completeand total files are taken/copied from production and used for testing. You mustperform the following steps:

identify needed cycles and outputs that are to be compared

capture and save needed data and copy needed files to the test environment

modify JCL and run production to save files for later comparison

substitute renovated code and rerun production using saved data

compare identified files.

Advantages saves time and effort in obtaining data and files

gets into performing tests quicker

usually provides a good test of common or frequently used functionality

usually results in shorter test phase duration.

Disadvantages requires using production data, which is often voluminous and possibly

sensitive

consumes large amounts of processing time

usually does not provide a complete test of the functionality

still have to rerun production in the test environment to save needed files.



Full RegressionWith this approach, production data is augmented to test all, or at least a highpercentage, of the applications functionality. The steps usually follow those infull production, but you must also do the following:

examine programs for code coverage

bring in additional production data or create the needed data

modify JCL and re-run until sufficient coverage is attained.

Advantages saves time and effort in creating data files


extremely good test of applications functionality

usually tests all date-related logic due to high code coverage.

Disadvantages requires using production data, which is often voluminous and possibly

sensitive

consumes large amounts of processing time.

Concentrated RegressionThis approach builds a test bed of files and data for performing all tests.Production data is not used, except as an initial source from which to build orextract data. You must perform the following steps:

determine what is required and analyze the best method for obtaining it

create and/or capture required data

create or extract representative files

create or modify JCL

assure adequate code coverage

execute programs to create a baseline.




Advantages most efficient and effective testing approach

saves time and resources during test performance

the more testing that is required, the greater the return on the effort to set up

provides an excellent test of an applications functionality

usually tests all date-related logic due to high code coverage.

Disadvantages requires the most time and effort to set up

longest lead time before getting to testing

requires more involvement from application-knowledgeable personnel.

Compliance AssuranceThis approach is used for testing applications that you feel are compliant. Sincethe code is not being renovated, you do not need to regression test to check forimpact to the functionality. The steps are similar to full production but you mustalso do the following:

examine programs for date processing

bring in additional data or create data with the needed dates

modify or age files to create pseudo-baselines for comparison.

Advantages saves time and effort in creating data files


usually tests all date-related logic due to date-code coverage.

Disadvantages requires using production data, which is voluminous and possibly sensitive

consumes large amounts of processing time.



Test TypesIn addition to determining the testing approach, your test strategy must identifythe types of tests that are required. These types, or phases as they are sometimescalled, are defined below.

Not all of the tests types are required for a given applicationnor for all programs within an application.

A test strategys intent is to indicate where and on what portions of the applicationthe tests will be performed. The test types are described below.

Unit TestingThis test type is the testing of a single screen, unit or program to verify that itoperates correctly. Unit testing can be carried out by testing all of the logic pathsor by concentrating on the code that has been changed. Test all of the logic pathsfor new development. You only need to concentrate on the changed code whenyou are testing maintenance changes. You can implement unit testing in one ofseveral ways:

by the development or maintenance group normally responsible for theapplication

by a core test team

by the renovation center or group making the changes.

When unit testing is performed at a renovation center, limited or selected testcases are performed. By concentrating your effort, you increase the chance offinding errors while expending minimal effort. This method is often referred to asTouch Point testing. A clean compile is all that is necessary to progress to othertesting phases.

On the other hand, you may choose to employ full logic unit testing for criticalprograms and/or test beds with high code coverage.

The method you choose is dependent upon what other testtypes are being performed. This is one of the key purposesfor outlining a testing approach early in the planning cycle.

String TestingThis test type is the testing of a small or logical grouping of programs that aretightly interrelated. Their relationship makes it difficult to individually test theprograms. Usually, the test consists of executing an on-line transaction or batchjob to verify that it functions correctly. Much compliance testing is performed in




this phase, particularly when the reverse transaction method is used to resetfiles. String testing is often combined with unit testing and sometimes withsystem testing.

Again, you can use the full logic or change concentration approach as in unittesting.

System TestingThis test type is the testing of an application to validate that its functionalityperforms as specified. Its objective is to prove that the whole system performscorrectly. You may perform system testing for the entire application or for logicalmodules or subsystems such as daily, weekly and monthly. System testing isoften combined with acceptance testing and/or integration testing.

A part of system testing is to process the baseline that was created against therenovated code to make sure that no problems were introduced during renovation.Depending upon the size and complexity of the application, these baseline testscan be performed in small groupings or for an entire application.

Integration TestingThis test type involves testing an applications external interfaces. The objectiveis to make sure that data is correctly passed between applications and that any filestructure changes are accommodated. Integration testing is almost certain to berequired due to date and file format changes. In addition, bridges or othertemporary measures are more likely to be used and require testing. The inclusionof these bridges adds processing and therefore, the potential for performancetesting. For this reason, integration and performance testing are often combined.

Acceptance TestingThis type of testing is used to ascertain whether an application completes thosefunctions required from a business perspective. Acceptance testing includesensuring that performance is at acceptable levels, and if functional deviationsoccur, suitable alternatives are substantiated. It is often combined with systemand integration testing. Acceptance testing sometimes includes the generation ofspecial or unique outputs or interfaces.

Performance TestingThis test type is performed to determine if an application can operate as neededwhen transactions, concurrent users or file sizes are larger than normallyexpected. Transaction and response speeds, throughput and computer capacity aremeasured to determine if they are within tolerance. Performance testing is done to



ensure that processing has not been significantly degraded. It is not intended totest for optimizing the application.

Interoperability TestingThis test type is used to determine if the renovated application can operate in itsintended environment. Things like the new operating system, middleware (suchas DBMS) and other renovated applications must be installed and operational forinteroperability testing to take place. Unlike the other tests, interoperabilitytesting requires a separate region or logical partition (LPAR). Performing thistype of test is difficult and requires a lot of work. Consequently, interoperabilitytesting is only performed for critical applications. It is most often combined withperformance and implementation tests.

Retrofit TestingThis test type identifies any changes that may have occurred while an applicationwas being renovated and tested. As mentioned throughout this document,changes should not be permitted or

Best Practices

Documents

Transcript of Best Practices