Software Engineering G22.2440-001 · Static Testing Dynamic Testing Black-Box Testing White-Box...

1

1

Software EngineeringG22.2440-001

Session 12 - Main ThemeSoftware Verification, Validation, and Quality

Dr. Jean-Claude Franchitti

New York UniversityComputer Science Department

Courant Institute of Mathematical Sciences

2

AgendaReviewTesting, Verification, and Validaton

Integration and System TestingStatic ConfirmationDynamic TestingTraceability MatricesAutomated Testing

Configuration ManagementSoftware Quality

Software Quality Assurance (SQA)Project MeasurementsSoftware Quality and Agile MethodsProject Management MetricsQuality and Process Standards and Guidelines

SummaryProject (Part 4) - ongoing

2

3

Summary of Previous Session

ReviewSoftware Engineering TeamsPair ProgrammingPair Programming EffectsRefactoringTest Driven Development (TDD)Distributed Development and Agile Methods Scalability Languages, Platforms, and Component InfrastructuresSummary

ReadingsIndividual Assignment #5 (ongoing)Project Part 3 (ongoing)

4

Part I

Software Testing, Verification and Validation

3

5

Testing

6

Why Test?

• No matter how well software has been designed and coded, it will inevitably still contain defects

• Testing is the process of executing a program with the intent of finding faults

• A “successful” test is one that finds errors, not one that doesn’t find errors

• Testing can “prove” the presence of faults (bugs), but can not “prove” their absence

4

7

What to Test?

• Unit test – test of small code unit written by a single programmer (or programming pair) – file, class, subroutine

• Integration test – test of several units combined to form a (sub)system, preferably adding one unit at a time

• System (alpha) test – test of a finished program (a system release) by system testers

• Acceptance (beta) test – test of a finished program by end-users or their representatives

• The larger the unit of testing when defects are detected, the harder and more expensive it is to find and correct the defects (ignoring XP flattened curve…)

8

When to Test?

Early• XPers write unit test cases before coding• Many software processes involve writing

system/acceptance tests in parallel with development

Often• Regression testing: rerun unit, integration and

system/acceptance tests– After refactoring and each iteration– Throughout integration– Before each release

5

9

Who should Test?

• Software authors should not test their own code because– Testers who don’t believe they will find faults generally

don’t find many faults (cognitive dissonance)– Testers who have to fix any faults they find don’t tend

to find very many (avoidance behavior)– Coders want code to be fault free, but effective testers

must want to find faults (conflict of interest)• However, usually authors do unit tests and often

integration tests• Separate team, preferably “independent”, usually

does system tests and/or acceptance tests

10

Development Testing Overview

Unit Testing Integration Testing System Testing

Class or ModuleTesting Package Testing Component

Testing Subsystem Testing

Positive Testing

Negative Testing

Requirements-based Testing

Alpha Testing Beta Testing

AcceptanceTesting

Types of Test Cases

Types of Testing

White Box Testing Black Box Testing

6

11

Classification of Software Testing

Software Testing

Static Testing Dynamic Testing

Black-Box Testing White-Box Testing

12

Economics of Testing

Costs associated with testing software• Planning testing process• Writing test cases• Setting up equipment• Systematically executing test cases• Follow-up on problems identified• Removing [most] defects

7

13


Costs associated with shipping software defects to customers

• Customers lose faith in developers and get very angry

• Customers can lose a great deal of money if their system goes down because of defects

• Expensive to obtain specific information about customer problems and to find and fix their bugs, possibly on-site

14


Also need to consider the loss potential of each individual project developers work on

• Quality is much more important for safety-or mission-critical software than for consumer software (e.g., video games)

• Balance the cost of testing versus the cost of software failures

8

15


To minimize costs, one goal of testing must be to uncover as many defects as possible with as little testing as possible

• It is simply impossible to test every possible input-output combination of the system– There are simply too many permutations and

combinations • Define test cases that have a high likelihood

of uncovering defects

16

Testing: The Process

Software to be tested

Test Case

Output

Test CaseGeneration

Verification

9

17

When to Stop?


Test Case

Output

Enough?No

Yes

Test Coverage

18

Determining Adequacy

• Statement coverage– Statements

• Branch coverage– Both IF and ELSE

• Path coverage• Example

– Stop when 90% of statements covered

10

19

Defining a Test

• Goal – the aspect of the system being tested• Input and system state – specify the actions

and conditions that lead up to the test as well as the input (state of the world, not just parameters) that actually constitutes the test

• Expected behavior – specify how the system should respond or what it should compute, according to its requirements

20

Running a Test

• Degree of scaffolding (addition software needed just to test) depends on whether black box or white box

• Need to record each test case and its results in a testing database (include date, tester, etc.)

• Preferably automate testing

11

21

Automating Testing

• Scripts that setup and run a test suite• Compare actual and expected outputs• Difficult for GUI testing and large external

files/databases

22

Unit Testing Overview

• Unit testing is testing some program unit in isolation from the rest of the system (which often doesn’t exist yet)

• Usually the programmer is responsible for testing a unit during its implementation (even though this violates the rule about a programmer not testing own software)

12

23

Test Harness (Scaffolding)

• A test driver is supporting code and data used to provide an environment for invoking part of a system in isolation

• A stub is a dummy procedure, module or unit that stands in for another portion of a system, intended to be invoked by that isolated part of the system – May consist of nothing more than a function

header with no body– If a stub needs to return values for testing

purposes, it may read and return test data from a file, return hard-coded values, or obtain data from a user (the tester) and return it

24

Unit Testing Strategies

• Black box (specification-based) testing• White box (program-based) testing• Normally perform both (not alternatives!)

13

25

White Box Testing• Test suite constructed by inspecting the

program (code)• Look at specification only to determine

what is an error• Tests control and/or data flow paths through

unit– Complexity usually too high to extend across

unit boundaries– Sometimes applied only to individual

subroutines• Attempt to exercise all statements, all

branches, or all paths

26

14

27

White-box Testing• Example

INPUT-FROM-USER(x);

If (x <= 300) {

INPUT-FROM-FILE(BALANCE);

If (x <= BALANCE)

GiveMoney x;

else Print “You don’t have $x in your account!!”}

else

Print “You cannot withdraw more than $300”;

Eject Card;

1

2

3

4

5

6

x: 1..1000;

28

public class Math {private int number1; private int number2;public Math(int num1, int num2) {

number1 = num1; number2 = num2;}public int add() {

int number3 = number1 + number2;return number3;

}public int subtract() {

int number3 = number1 - number2;return number3;

}public int multiply() {

int number3 = number1 * number2;return number3;

}public double divide() {

double number3 = number1 / number2;return number3;

}

15

29










}

testVar1 = new Math(20, 10);

public void testadd() {// {20 + 10} == {30}int expected= 30;assertEquals(expected,testVar1.add());

}

30










}


public void testsubtract() {// {20 - 10} == {10}int expected= 10;assertEquals(expected,testVar1.subtract());

}

16

31










}


public void testdivide() {// {10 / 20} == {0.5}double expected= 0.5;assertEquals(new Double(expected),

new Double(testVar2.divide())); }

32

Black Box Testing

• Test suite constructed by inspecting the specification (requirements)

• Ideally, no access to source code being tested

• Tests unit against functional and, sometimes, extra-functional specifications

• Attempts to force behavior that doesn't match specification

17

33

34

Based on Requirements

• Could either be based on the “pure”requirements in the eyes of customer

• Or on a requirements analysis or requirements specification document

• The act of writing test cases can drive the requirements gathering process to be more complete and specific

18

35

Blackbox Approaches to System Testing

• Functional tests corresponding to customer-level requirements

• Extra-functional tests corresponding to customer-level constraints

36

Black-box Testing Example

• Discussion: MAC/ATM machine example– Specs

• Cannot withdraw more than $300• Cannot withdraw more than your account balance

Softwarex

Balance

19

37

Blackbox Approaches to System Testing

• Performance tests – evaluate response time, memory usage, throughput, device utilization, CPU time

• Stress tests – push system to or beyond its specified limits to evaluate its robustness and error handling capabilities

• Reliability tests – monitor response to representative user input, counting failures over time to measure or certify reliability

38

Blackbox Approaches to Unit Testing

• Functional testing – exercise code with valid or nearly valid input for which the expected output is known

• Exhaustive testing usually infeasible, so need way(s) to select test cases and determine when “done” testing

• Choose test cases to attempt to find different faults– Equivalence partitioning– Boundary value analysis

20

39

Equivalence Partitioning

• Assume similar inputs will evoke similar responses

• Equivalence class is related set of valid or invalid values or states

• Only one or a few inputs are selected to represent an entire equivalence class

40

Equivalence Partitioning

• Divide input domain into equivalence classes

• Might also divide output domain into equivalence classes – Need to determine inputs to cover each

output equivalence class – Attempt to cover classes of errors

21

41

Range equivalence classes

1. Arbitrary value below range 2. Arbitrary value within range 3. Arbitrary value above range

42

Set equivalence classes

• Member of set • Non-member of set

22

43

Examples

• negative, positive, and zero inputs or outputs• input or output strings within size limits, beyond

size limits, and the empty string• input or output numeric values within size limits

and beyond size limits • input files that exist or not, are readable or not, are

correct format or not• output files that exist or not, are writeable or not,

are correct format or not (if being modified)• combinations of input values that are consistent,

and combinations that are not consistent

44

Boundary Value Analysis

• Extends equivalence partitioning • In practice, more errors found at or near

equivalence class boundaries than within the classes

• Divide input domain into equivalence classes

• Again may also divide output domain into equivalence classes

23

45

Range equivalence classes

1. Value immediately below range 2. First value of range 3. Second value of range 4. Next to last value of range 5. Last value of range 6. Value immediately above range 7. Minimum and maximum values of range's

basic type

46

Scalar set equivalence classes

• Member of set • Values immediately above and below each

member of set • Minimum and maximum values of set

element's basic type

24

47

Non-scalar set equivalence classes

• Member of set • Null value (if not in set) • Other non-member value(s) of correct type

("above" and "below" each set member, "minimum" and "maximum" of type, if such concepts are applicable)

• Value of incorrect type, structurally equivalent (if not detected by compiler)

• Value of incorrect type, not structurally equivalent (if not detected by compiler)

48

Examples

• maximum negative, maximum positive, and zero inputs or outputs

• input or output strings at size limits, one character beyond size limits, the empty string, and strings of one character

• input or output numeric values at size limits and one beyond size limits

• empty input files, files with one character in them, files at maximum file size (if any)

25

49

Whitebox Approaches to Unit Testing

1. Execute all (reachable) statements 2. Execute all branches of logical decisions, including

boundaries of loops 3. Execute all (feasible) control flow paths in

combination 4. Execute all data flow paths (from each variable

definition to all its uses) • Usually applied only to individual subroutines rather

than larger unit • May be combined with black box testing if tool

available to track coverage

50

Statement Coverage

• Requires each statement in unit (or entire program) to be executed at least once

• 85% easy, 100% hard– Unreachable code– Dead code– Complex sequence

26

51

Path Coverage

• Requires each control flow path in unit to be executed at least once– Rarely applied across program units

• May be impossibleread N;SUM := 0;for I = 1 to N do

read X;SUM := SUM + X;

endfor

52

27

53

Basis Path Testing

• Cover all independent control flow paths• Generate flow chart corresponding to subroutine

logic • Each test case should traverse one edge not

covered by another test case • Guarantees execution of every statement at least

once • Set of independent paths is not unique • May have difficulty in finding parameters (and

global variable values) to force a test case - then test in combination with another path

54

Problems with Basis Path Testing

• Expensive to generate flow charts and find test suite

• Tedious and error-prone (unless automated) • Usually reserved for subroutines with high

cyclomatic complexity

28

55

Cyclomatic Complexity

• Upper bound on number of test cases needed (for basis path testing)

• Flow chart must be expanded to distinguish every simple condition (single conditional operator)

• The complexity metric is number of simple logical tests plus 1

56

29

57

58

Loop Testing

• Isolate independent paths within loop • Apply special tests to each loop type • Detect initialization, indexing,

incrementing, and bounding errors at loop limits

30

59

Simple Loops

1. Skip loop entirely 2. Only one pass through loop 3. Two passes through loop 4. m passes through loop, where m = n-1, n,

n+1 passes through loop 5. Cover branching paths within loop

60

31

61

Nested Loops

1. Too expensive to use simple loop approach 2. Apply simple loop tests on innermost loop

while holding outer loops at minimum iteration parameter

3. Apply simple loop tests to next innermost loop while holding outer loops at minimum values and inner loops at typical values

4. Repeat outward through all levels of nesting

62

Other Loops

• Concatenated loops– Treat as either simple or nested depending on

degree of dependency between the loops • Unstructured loops

– Recode the loops to be structured

32

63

Data Flow Paths

• Reconsider control flow paths • Cover path from each parameter or variable

assignment to each of its uses (before reassignment)

64

33

65

Mutation Testing

• Develop specification-based test suite • Program executes test suite correctly • Make small change in a condition • Retain syntactic correctness! • Run test suite and see if there are any

changes in results • Repeat with next condition

66

Mutations

• Change sense of comparison operator • Change arithmetic operator• Change a constant • Change which variable or array index is

used• Delete statement• Delete/insert return statement

34

67

Intuition

• If a mutation doesn't matter, then something is wrong!– Either mutated code is unnecessary – Or there's a problem with the test suite itself

• Expensive: n^2 mutants for n statements, but automatable

• Often basis for validation suites

68

Integration Testing

Integration Strategies• “Big Bang” vs. Incremental• Top-down vs. Bottom-up vs. Sandwich

35

69

“Big Bang” Approach

1. Unit test all modules 2. Link everything together and run

– Difficult to track down errors because bug could be anywhere

70

A

DCB

FE G

36

71

TestA

TestA,B,C,D,

E,F,G

TestB

TestD

TestE

TestC

TestF

TestG

A

DCB

FE G

72

Incremental Approaches

• Emphasis on testing interfaces• Put a few units together at a time

– Ideally add only one unit at a time• Limits scope of errors

37

73

Top-down integration

1. Start with top-level module 2. Move downwards through control

hierarchy 3. Whenever add new modules, perform

regression tests on old modules to make sure no new errors introduced

74

Top-down integration

• Depth-first integrates all modules in a major vertical branch (wrt caller-callee module breakdown structure)

• Breadth-first integrates all modules directly subordinate at each level (or entire layer)

• Intuition in either case is to test program in relatively natural execution mode

• Difficult when stubs must be relatively complicated to fully test other modules

38

75

TestA

TestA,B,C,D

TestA,B,C,D,

E,F,G

A

DCB

FE G

76

TestA

TestA,B,C,D

TestA,B,C,D,

E,F,G

TestB

TestC

TestD

TestE

TestC

TestD

TestF

TestG

A

DCB

FE G

39

77

Bottom-up integration

1. Start with leaf modules, so no stubs required

2. Combine into larger and larger subsystems– Move up caller-callee hierarchy to full logical

groupings– And, eventually, across logical grouping

boundaries

78

Bottom-up integration

• Drivers needed to control and handle test case input/output

• Intuition is to construct program from separately tested pieces

• Tests control aspects of program relatively late

40

79

TestE

TestF

TestG

TestB,E,F

TestC

TestD,G

TestA,B,C,D,

E,F,G

A

DCB

FE G

80

Sandwich integration

• Hybrid• Top-down for highest levels of control

hierarchy (within logical groupings and across system topology)

• Bottom-up for lower levels • Reduces need for drivers and stubs

41

81

TestA

TestB,E,F

TestA,B,C,D,

E,F,G

TestE

TestD,G

TestF

TestG

A

DCB

FE G

82

TestA

TestB,E,F

TestA,B,C,D,

E,F,G

TestB

TestD

TestE

TestC

TestD,G

TestF

TestG

A

DCB

FE G

42

83

Microsoft “Synch-and-Stabilize”

• Working product as soon as possible• Many small parallel teams work on “feature

sets”• Iterate among designing, building and

testing (and involve customers in testing)• Most important features developed and

integrated first• Integrate frequently – daily builds

(analogous to “code and fix”)

84

Milestone 1: Most critical featuresand shared componentsDesign, code, prototype

Usability testingDaily builds

Feature integrationEliminate severe faults

Milestone 2: Desirable featuresDesign, code, prototype


Feature integration

Milestone 3: Least critical featuresDesign, code, prototype


Feature integration and completionRelease to manufacturing

43

85

Testing Database

• What program unit was tested • Actual inputs • Expected outputs • Actual outputs • Include global variables, files, other state

(when relevant) • Who performed tests • Who was informed of errors

86

Links to Other Materials

• Configuration information (e.g., module versions, system release)

• Test scripts or programs (if any) • Scaffolding used (if any) • Bug report (if any) • Error analysis (if any) • Change request (if any) • Anything else...

44

87

Why Keep Records?

• Some parts are necessary for regression testing

• Same bug may be reported multiple times by different customers

• Track staff responsibility - who did what • Process measurement and improvement -

relate quality/quantity of tests to quality of program

88

Regression Testing After Changes

• Change is successful, and no new errors are introduced

• Change does not work as intended, and no new errors are introduced

• Change is successful, but at least one new error is introduced

• Change does not work, and at least one new error is introduced

45

89

Verification and Validation

90

Verification vs. Validation

• Verification: Are we building the product right?

• Validation: Are we building the rightproduct?

46

91

Verifying and Validating Results

• Static confirmation– Reviews, walkthroughs and inspections – Static confirmation processes improve software quality

• Dynamic testing– Based on various types of testing – Must determine who performs testing

• Traceability matrices– Traceability matrices are used throughout the life cycle – Assumes creation and maintenance of the matrices

92

Event-Driven Software Testing

47

93

Event-driven Software

Events

E1

E2

E3

E4

E5

E6

E7

E8

•Graphical-user Interfaces•Web applications•Device drivers•Network protocol implementations•Object-oriented applications•Component-based applications

94

Event-driven Software

E1

E2

E3

E4

E5

E6

E7

E8

•Ordering not predetermined•Number of permutations•Source code not always available•Environment affects behavior

48

95

CaptureTester MANUALLY performs events on the softwareTool records all user inputs and application response

INPUT/RESPONSE

TESTER INPUT

RESPONSE

APPLICATION UNDER TEST

DATABASE

TESTER

Capture/replay Tools

96

ReplayTool replays users actionsVerify application response against expected response

INPUT/RESPONSE

REPLAY INPUT

RESPONSE

APPLICATION UNDER TEST

DATABASE

COMPARE

EXPECTEDRESPONSE

Capture/replay ToolsModifications made to enhance the software

49

97

Automated Testing

98

Automated Testing


Test Case

Output

Enough?No

Yes

Test Coverage

50

99

Automated Testing


Test Case

Output

CoverageEvaluator

Test CaseGenerator

VerifierOR

Test Oracle

TestSpecs

100

Test CaseGenerator

RegressionTester

TestCoverageEvaluator

Test Oracle

GUI Representation

Sample Automated Testing Framework

ExecutingSoftware

Implementation:Tools (Languages/Toolkits)

GUI Implementer

INFORMALSpecifications

Test Executor

51

101

Automated Testing Sample Capabilities

• Automatic Analysis of Software– 1-2 minutes

• Automatic Test Case Generation– 1000s in minutes– Invalid inputs (outside of specified ranges)

• Automatic Verification• Automatic Replay• Automatic Instrumentation• Automatic Coverage Evaluation

102

Sample System: DART

• Daily Automated Regression Testing– ICSM 2003

• Nightly/Daily Smoke Testing• Create GUI “smoke” tests• Every night, checkout source from CVS repository• Build• And run smoke tests• E-mail Bug Reports and successful/unsuccessful

logs

52

103

GUI Ripping

Representation of GUI

AUTOMATED GENERATION

XML EFG

TEST ORACLE TEST CASES

DART PROCESS

FIX BUGS

DEVELOPERSAUTOMATED REPORTING

SETUPITERATIVE

NIGHTLY PROCESS

APPLICATION UNDER TEST - Version (i)

APPLICATION UNDER TEST – Version (i + 1)

DEVELOPMENT

AUTOMATED EXECUTION

Run Test

COVERAGE REPORT

ERROR REPORT

104

Traceability

53

105

Traceability• Traceability information is information which

helps you assess the impact of requirements change. It links related requirements and the requirements and other system representations

• Types of traceability information– Backward-from traceability Links requirements to their sources

in other documents or people– Forward-from traceability Links requirements to the design

and implementation components– Backward-to traceability Links design and implementation

components backs to requirements– Forward-to traceability Links other documents (which may

have preceded the requirements document) to relevant requirements.

106

Backwards/Forwards Traceability

Business plan Design SpecificationRequirements Document

Forward-to traceability Forward-from traceability

Backward-from traceability Backward-to traceability

54

107

Types of Traceability

• Requirements-sources traceability– Links the requirement and the people or

documents which specified the requirement• Requirements-rationale traceability

– Links the requirement with a description of why that requirement has been specified.

• Requirements-requirements traceability– Links requirements with other requirements

which are, in some way, dependent on them. This should be a two-way link (dependants and is-dependent on).

108

Types of Traceability• Requirements-architecture traceability

– Links requirements with the sub-systems where these requirements are implemented. This is particularly important where sub-systems are being developed by different sub-contractors

• Requirements-design traceability– Links requirements with specific hardware or

software components in the system which are used to implement the requirement

• Requirements-interface traceability– Links requirements with the interfaces of external

systems which are used in the provision of the requirements

55

109

Traceability Tables• Traceability tables show the relationships

between requirements or between requirements and design components

• Requirements are listed along the horizontal and vertical axes and relationships between requirements are marked in the table cells

• Traceability tables for showing requirements dependencies should be defined with requirement numbers used to label the rows and columns of the table

110

A Traceability Table

Depends-onR1 R2 R3 R4 R5 R6

R1 * *R2 * *R3 * *R4 *R5 *R6

56

111

Traceability Lists

• If a relatively small number of requirements have to be managed (up to 250, say), traceability tables can be implemented using a spreadsheet

• Traceability tables become more of a problem when there are hundreds or thousands of requirements as the tables become large and sparsely populated

• A simplified form of traceability table may be used where, along with each requirement description, one or more lists of the identifiers of related requirements are maintained.

• Traceability lists are simple lists of relationships which can be implemented as text or as simple tables

112

A Traceability List

Requirement Depends-onR1 R3, R4R2 R5, R6R3 R4, R5R4 R2R5 R6

57

113

Traceability Policies

• Traceability policies define what and how traceability information should be maintained.

• Traceability policies may include– The traceability information which should be maintained. – Techniques, such as traceability matrices, which should be used

for maintaining traceability. – A description of when the traceability information should be

collected during the requirements engineering and system development processes.

– The roles of the people, such as the traceability manager, who are responsible for maintaining the traceability information should also be defined.

– A description of how to handle and document policy exceptions– The process of managing traceability information

114

Factors Influencing Traceability Policies

• Number of requirements – The greater the number of requirements, the

more the need for formal traceability policies.• Estimated system lifetime

– More comprehensive traceability policies should be defined for systems which have a long lifetime.

• Level of organizational maturity– Detailed traceability policies are most likely

to be cost-effective in organizations which have a higher level of process maturity

58

115

Factors Influencing Traceability Policies

• Project team size and composition – With a small team, it may be possible to assess the impact

of proposed informally without structured traceability information. With larger teams, however, you need more formal traceability policies.

• Type of system– Critical systems such as hard real-time control systems or

safety-critical systems need more comprehensive traceability policies than non-critical systems.

• Specific customer requirements– Some customers may specify that specific traceability

information should be delivered as part of the system documentation.

116

Part II

Configuration Management

59

117

The process of identifying, organizing, and controlling changes to the software during development and maintenance.

• SCM is a support activity that makes technical and managerial activities more effective

•SCM operates throughout the SW life-cycle

Software Configuration Management (SCM)

118

Problems of Change

Which component ? Which version ?

• Double (or multiple) maintenance• Updates to shared data• Simultaneous update

60

119

SCM Functions

• Identification of software items and products• Definition of Baselines• Access controls• Progressing defect reports• Progressing change requests• Recording item status• Controlling releases (versions and variants)• Reporting

120

SCM Tasks

• Configuration identification• Configuration control• Status accounting• Configuration audit

61

121

Some Definitions ...

Development item not yet approved, canitem be informally changed

Configuration an approved and accepteditem (CI) deliverable, changes done

through formal changecontrol procedures

122

Typical SW Configuration Items (CIs)

• Management plans• Specifications (requirements, design)• User documentation• Test design, case and procedure specifications• Test data and test generation procedures• Data dictionaries and databases• Source code, executable code• Libraries• Maintenance documentation • Support software

62

123

Milestones and Baselines

MilestoneA milestone is the end of a stage or phase of a project at which one or more deliverables are actually delivered.

BaselinesA baseline is that collection of items which when complete indicates that a milestone in the development process has been reached.

124

Typical Baselines

Phase BaselineFeasibility studyRequirements defn. Functional baseline

SRS, Interface spec. Allocated baseline

Detailed design Design baseline

Source and Object codeUser manualsTest documents Product baseline

Installation Operational baseline

63

125

Baselines

• Baselines serve as the basis for further development

• Baselines can be changed only through formal change control procedures

• Only items that have been approved and obtained through a formal technical review are accepted into the baseline.

126

Configuration Identification

• Identify what the different baselines will consist of

• Set labelling and identification conventions for the CIs

64

127

Basic CI information

• Item identity• Baseline to which it belongs• Relationships to other items• Version• Variant

128

SCM TerminologyVersion

A SW CI having a defined set of functional capabilities.

Revisionschanges to a version to correct only errors in design logic but does not affect documented functional capabilities since none of the requirements have changed.

Variantsa variation of a version developed to run on different types of HW, or to provide slightly different facilities for different users.

65

129

Examples

1.1 1.2 1.3 1.4

successive versions

branching versions (variants)

1.1 1.2 1.3 1.4

1.3.1.1 1.3.1.2

130

Merging

• Two diverging versions may be merged to create a single new version combining both set of change requests.

• Merge operations are typically done interactively with tool assistance

66

131

SCM Terminology

Promotion of a CIA CI may be promoted from one developmental baseline to another to signify a change in a CI’s internal development state.

ReleaseA Release is used to designate certain promotions of CI’s that are distributed outside the development organization.

132

Configuration Control

• Enforces a rigorous change control mechanism

• Requires formal procedures to– request changes– carry out impact analysis– approve changes– carry out changes

67

133

Change Management Methodology• Submission of Change Request (CR)• Technical and business evaluation and

impact analysis• Approval by Change Control Board• Engineering Change Order (ECO) is

generated stating– changes to be made– criteria for reviewing the changed CI

• CI’s checked out• Changes made and reviewed• CI’s checked in

134

Change Control Board

• A group consisting of representatives of user, customer, producer.

• Responsibilities:– to approve, monitor and control baselines– to approve, monitor, and control changes– to authorise changes

• CCB concerns in change approval– technically ok solution, cost, schedule,

configuration of the whole system, user satisfaction

68

135

Software Libraries

• SW libraries provide the means for implementing SCM

• The number and kind of libraries will vary from project to project . It depends on the levels of control needed.

136

Three Kinds of Software Libraries

Dynamic library (programmer’s library)– programmer’s workspace

Controlled library (master library)– used for managing the current baseline(s)

and for controlling changes made to them

Static library (software repository)– used to archive various baselines released

for general use

69

137

Techniques for Storing Versions

• Full files• Forward Delta files• Reverse Delta files• The set of differences between two

versions is called a delta.

138

Forward Delta Files

foward delta files

+ firstversion

Vnversion

Vnversion

User

Vn+1version

Vn+1version

Vnversion

CM System

changes

70

139

Reverse Delta Files

reverse delta files

+Vn+1

new recentversion

Vn+1version

User

Vnversion

Vnversion

CM System

changes

recent version

140

Status Accounting• provides a mechanism for administrative

tracking and reporting of all SW items identified and controlled.

• Examples of Status reports:– the status of proposed changes– the status of approved changes– the baselines and the approved changes

associated with each baseline– the date when each revision of each CI was

recorded– deficiencies identified by configuration

audit

71

141

Configuration Audit

• A configuration audit establishes that product integrity has been maintained and that changes have taken place in an orderly and controlled manner.

• Audit of the SW product• Audit of SCM activities

142

Physical Configuration Audit

– Consists of determining that all items identified as being part of the configuration are present in the Product baseline

– It must also establish that the correct version and revision of each part are included in the product baseline and that they correspond to information contained in the baseline’s configuration status report.

72

143

Functional Configuration Audit

– It verifies that each CI in the product has been tested to determine that it satisfies the functions defined in the specifications or contract(s) for which it was developed.

144

Organizing for SCM

Roles:• Configuration manager

• Change Control Boardincludes representatives of

- user- customer- developer

73

145

SCM Plan

The SCM Plan is prepared in Project Initiation phase. It documents

- what SCM activities are to be done- how they are to be done- who is responsible for doing specific activities

- when they are to happen- what resources are required

146

SCM Tools

Common features of popular PC-based tools (PVCS, MS Visual SourceSafe, Clearcase, etc.):

• Support for controlling all types of files (source code as well as binary)

• Managing changes as deltas• Supporting branching and merging• Identifying and re-creating releases• Providing a project view

74

147

Intersolv PVCS Version Manager

• One of the oldest PC-based version control products

• Large installed base• A fairly rich feature set• Interfaces with other third party tools• Gateways to mainframe-based library

management systems• Comprehensive security

148

Microsoft Visual SourceSafe

• Project support• File sharing• Intuitive GUI interface• Good repository architecture • Powerful security features• Tight integration with Visual Studio

development tools

75

149

SCM Tools for Unix

• SCCS (Source Code Control System)– manages changes to text files– uses a single file (s-file) to store first version

and successive forward deltas

• RCS (Revision Control System)– manages changes to text files– uses reverse deltas to store versions

150

SW Configuration Management Plan

-- IEEE Standard 828-1990 for SCM Plan

1.Introduction

2.SCM Management2.1 Organization2.2 SCM Responsibilities2.3 Applicable policies, directives and procedures

76

151



3.SCM Activities3.1 Configuration identification

3.1.1 Identifying configuration items3.1.2 Naming configuration items3.1.3 Acquiring configuration items

3.2 Configuration control3.2.1 Requesting changes3.2.2 Evaluating changes3.2.3 Approving or disapproving changes3.2.4 Implementing changes

152



(3. SCM Activities)3.3 Configuration Status Accounting3.4 Configuration Audits and Reviews3.5 Interface control3.6 Subcontractor/Vendor control

4.SCM Schedules5.SCM Resources6.SCM Plan maintenance

77

153

Part III

Software Quality

154

Quality Is Hard to Pin Down

• Concise, clear definition is elusive • Often not quantifiable • Many things to many people • “You'll know it when you see it”• Most often defined as set of “ilities”

(attributes)

78

155

Good quality software has …

• Reliability • Human engineering • Understandability • Modifiability • Efficiency • Testability • Portability Notice: all related to maintenance

156

Understandability

• The ability of a reader of the software to understand its function

• Critical for maintenance

79

157

Modifiability

• The ability of the software to be changed by that reader

• Almost defines "maintainability"

158

Reliability

• The ability of the software to perform as intended without failure

• If it isn't reliable, the maintainer must fix it

80

159

Efficiency

• The ability of the software to operate with minimal use of time and space resources

• If it isn't efficient, the maintainer must improve it

160

Testability

• The ability of the software to be tested easily

• Finding/fixing bugs is part of maintenance • Enhancements/additions must also be tested

81

161

Human Engineering

• The ability of the software to be easily used (usability)

• Not easily used implies more support calls, enhancements, corrections

162

Portability

• The ease with which the software can be made useful in another environment

• Porting is usually done by the maintainer

82

163

Quality Assurance Techniques

• Reviews (walkthroughs, inspections)• Standards• Metrics • Testing

164

Inspection

• Formally review the product and identify defects

• Applied to all project documents including but not limited to source code

• Used to increase software quality and improve productivity and manageability of the development process

83

165

Inspection Roles

• The minimum roles required for an inspection are moderator/recorder, author/reader, and one other inspector (3 participants)

• Customer may or may not be present• Moderator - Ensures that the inspection

procedures are followed, and that the other inspectors perform their responsibilities for each step of the inspection process

• Recorder - Records and classifies all the defects detected at the inspection meeting, and assists the moderator in preparing inspection reports

166

Inspection Roles, cont.

• Reader - Leads the team through the product in a comprehensive and logical manner

• Author - Addresses specific questions that the reader is not able to answer, and detects defects based upon their special understanding of the product

• Inspector - All of the inspection team members are inspectors, regardless of their other roles

84

167

Inspection Process

• Define what will be inspected, • when it will be inspected, • who will do the inspection, • what data will be collected, and • what follow-up actions must be taken.

168

Moderator/AuthorFollow-up

ModeratorInspection

ModeratorPreparation

Author/ModeratorPlanning

ResponsibilityActions

85

169

Planning Phase

• Author assembles all software elements necessary for the inspection to allow easy access for the entire inspection team

• Author notifies the Project Coordinator that the product is ready for inspection

170

Planning Phase, cont.

• Minimum criteria, if applicable, includes:– Spelling and grammar checked – Code should compile clean– Compliant with standards– Each inspection item is line numbered for

easy reference by inspectors• Moderator certifies that the product to

be inspected is complete, the team is selected and notified, and all scheduling actions are accomplished

86

171

ModeratorSchedule inspection

PCAssign roles

PCSelect and notify inspection team

ModeratorReview inspection items for completeness

Project Coordinator (PC)

Select moderator from SQA group

AuthorAssemble items to be inspected

ResponsibilityActions

172

Preparation Phase

• Each member of the inspection team is provided access to the product to be inspected

• Each inspector uses the appropriate checklist and standards to review the inspection items

• Defects identified by each inspector are documented in a Defect Log

87

173

Preparation Phase, cont.

• If more than a few spelling and/or grammatical mistakes are found, then inspection may be postponed

• A redlined copy of the inspection item may also be given to the author at the end of the inspection to aid in defect location

174

ModeratorDisperse the combined Defect Log to all participants for review 24 hours prior to the meeting

ModeratorCombine individual Defect Logs in a logical order

InspectorsSend individual Defect Log to Moderator 48 hours prior to inspection meeting

InspectorsRecord identified defects on individual Defect Log

InspectorsReview all items to be inspectedResponsibilityActions

88

175

Inspection Meeting

• Formally review (walkthrough) the product, indicating previously identified defects

• Moderator, with input from the inspection team, determines the inspection disposition (Accepted, Conditional, or Re-inspect)

• PC approves re-inspections and rework efforts recommended by the inspection team

• New combined Defects Log results from the inspection meeting

176

Rules for Inspections

• Maximum time scheduled for an inspection meeting should be about 2 hours

• Start meeting on time and move it along so that it finishes on schedule

• Defer arguments over style or technique• Solutions or alternatives may be mentioned,

but avoid spending too much time on group rework

• Address the product directly, do not attack the author

89

177

Inspection Disposition

• Accepted - The product is complete, without any further verification of rework. Only a very few trivial defects that can be left to the Author to correct.

• Conditional - Conditionally accept the product, with verification of the rework by the Moderator. In this case there are a few major defects, and rework is not expected to create any substantial changes in the product

178

Inspection Disposition

• Re-inspect - Re-inspect the Author’s rework. This disposition requires that the rework be examined by the Moderator, the Author, and at least one other member of the inspection team in a reiteration of the inspection meeting. There are either a substantial number of major defects, or rework that will change the original product

90

179

ModeratorInform PC of inspection disposition

AuthorDetermine estimated rework effort and completion date

ModeratorDetermine results of the inspection

RecorderNote status of each defect and categorize valid defects

Author, InspectorsPresent and clarify defects identified

ReaderSystematically review (walkthrough) the product being inspected

ModeratorConfirm inspection preparationModeratorReview rules for inspectionsResponsibilityActions

180

Follow-up

• Ensure inspection results and decisions are properly recorded and resolved

• All inspectors must agree that the list of defects is accurate and complete

• Author corrects defects• Moderator certifies correction of all defects

(conditional disposition)• Inspection Report created and stored

91

181

ModeratorRecord metrics (e.g., Size of materials inspected; Defects by severity, class, and type; Preparation times prior to inspection; Estimates of rework effort)

ModeratorPost Inspection Report to project repository

Moderator/RecorderComplete Inspection Report

ModeratorValidate corrections made by author (conditional disposition)

AuthorCorrect or resolve all noted defectsResponsibilityActions

182

Project Measurements(Traditional SDLC)

• Team Productivity Measurement– Measure over several releases– Number of people– Number of features – Number of calendar months– Lines of code produced– Defects pre-release– Defect post-release

92

183

Project Measurements(Agile Projects)

• In an agile project, you can determine each iteration's velocity (number of user stories per iteration) and the number of outstanding defects.

• Because people on agile projects tend to perform test-first development the sheer number of defects tends to be fewer.– If the testers and the customers write user acceptance tests in advance of the

first line of code, agile projects tend to have fewer egregious defects.– Problem is obviously to keep the customer involved.

• So if you're measuring productivity, try measuring size, time, number of people, and defects.– Once you've calculated some sort of team productivity, see if you can figure

out personal productivity. Just don't forget that a person's productivity is based on their entire contribution to the product, not just the number of lines of code (or tests or whatever) they contribute.

184

Software Quality and Agile Methods• Acceptance testing• Mock objects

– Mock objects are usually regarded as a programming technique that merely supports existing methods of unit testing

– Fundamentally, mock objects enable an iterative, top-down development process that drives the creation of well designed object-oriented software.

– http://www.xprogramming.com/xpmag/virtualMockObjects.htm

– http://www.xprogramming.com/xpmag/virtualMockObjects.htm#N78

• User interface testing (HTTPUnit, Canoo)• Performance testing• Agile Project Management

– Incorporate an adaptive and highly responsive development approach with a framework for effective project and risk management

93

185

Quality and Process Standards and Guidelines

• ISO 9000• SWEBOK• ISO 15504• SEI’s Capability Maturity Model (CMM)• CMM Integration (CMMI)

186

Part IV

Conclusion

94

187

Course AssignmentsIndividual Assignments

Reports based on case studies or exercises

Project-Related AssignmentsAll assignments (other than the individual assessments) will correspond to milestones in the team project.As the course progresses, students will be applying various methodologies to a project of their choice. The project and related software system should relate to a real-world scenario chosen by each team. The project will consists inter-related deliverables which are due on a (bi-) weekly basis.There will be only one submission per team per deliverable and all teams must demonstrate their projects to the course instructor.A sample project description and additional details will be available under handouts on the course Web site.

188

Course ProjectProject Logistics

Teams will pick their own projects, within certain constraints: for instance, all projects should involve multiple distributed subsystems (e.g., web-based electronic services projects including client, application server, and database tiers). Students will need to come up to speed on whatever programming languages and/or software technologies they choose for their projects - which will not necessarily be covered in class.Students will be required to form themselves into "pairs" of exactly two (2) members each; if there is an odd number of students in the class, then one (1) team of three (3) members will be permitted. There may not be any "pairs" of only one member! The instructor and TA(s) will then assist the pairs in forming "teams", ideally each consisting of two (2) "pairs", possibly three (3) pairs if necessary due to enrollment, but students are encouraged to form their own 2-pair teams in advance. If some students drop the course, any remaining pair or team members may be arbitrarily reassigned to other pairs/teams at the discretion of the instructor (but are strongly encouraged to reform pairs/teams on their own). Students will develop and test their project code together with the other member of their programming pair.

95

189

Very eXtreme Programming (VXP)

• After teams formed, 1/2 week to Project Concept

• 1/2 week to Revised Project Concept• 2 to 3 iterations• For each iteration:

– 1/2 week to plan– 1 week to iteration report and demo

190

Very eXtreme Programming (VXP)(continued)

• Requirements: Your project focuses on two application services

• Planning: User stories and work breakdown• Doing: Pair programming, write test cases before coding,

automate testing• Demoing: 5 minute presentation plus 15 minute demo• Reporting: What got done, what didn’t, what tests show• 1st iteration: Any• 2nd iteration: Use some component model framework• 3rd iteration: Refactoring, do it right this time

96

191

Revised Project Concept (Tips)

1. Cover page (max 1 page)2. Basic concept (max 3 pages): Briefly

describe the system your team proposes to build. Write this description in the form of either user stories or use cases (your choice). Illustrations do not count towards page limits.

3. Controversies (max 1 page)

192

First Iteration Plan (Tips)• Requirements (max 2 pages):• Select user stories or use cases to implement in

your first iteration, to produce a demo by the last week of class

• Assign priorities and points to each unit - A point should correspond to the amount of work you expect one pair to be able to accomplish within one week

• You may optionally include additional medium priority points to do “if you have time”

• It is acceptable to include fewer, more or different use cases or user stories than actually appeared in your Revised Project Concept

97

193

First Iteration Plan (Tips)

• Work Breakdown (max 3 pages): • Refine as engineering tasks and assign to pairs• Describe specifically what will need to be coded

in order to complete each task• Also describe what unit and integration tests will

be implemented and performed• You may need additional engineering tasks that do

not match one-to-one with your user stories/use cases

• Map out a schedule for the next weeks• Be realistic – demo has to been shown before the

end of the semester

194

2nd Iteration Plan (Tips): Requirements

• Max 3 pages• Redesign/reengineer your system to use a

component framework (e.g., COM+, EJB, CCM, .NET or Web Services)

• Select the user stories to include in the new system– Could be identical to those completed for your 1st

Iteration– Could be brand new (but explain how they fit)

• Aim to maintain project velocity from 1st iteration• Consider what will require new coding vs. major

rework vs. minor rework vs. can be reused “as is”

98

195

2nd Iteration Plan (Tips): Breakdown

• Max 4 pages• Define engineering tasks, again try to maintain

project velocity• Describe new unit and integration testing• Describe regression testing

– Can you reuse tests from 1st iteration?– If not, how will you know you didn’t break something

that previously worked?• 2nd iteration report and demo to be presented

before the end of the semester

196

2nd Iteration Report (Tips): Requirements

• Max 2 pages• For each engineering task from your 2nd Iteration

Plan, indicate whether it succeeded, partially succeeded (and to what extent), failed (and how so?), or was not attempted

• Estimate how many user story points were actually completed (these might be fractional)

• Discuss specifically your success, or lack thereof, in porting to or reengineering for your chosen component model framework(s)

99

197

2nd Iteration Report (Tips): Testing

• Max 3 pages• Describe the general strategy you followed for

unit testing, integration testing and regression testing

• Were you able to reuse unit and/or integration tests, with little or no change, from your 1st

Iteration as regression tests?• What was most difficult to test?• Did using a component model framework help or

hinder your testing?

198

Project Presentation and Demo

• All Iterations Due• Presentation slides (optional)

100

199

Readings

ReadingsSlides and Handouts posted on the course web siteDocumentation provided with software engineering toolsTextbook chapters 13-15, 20, 26-27

Project Frameworks Setup (ongoing)As per references provided on the course Web site

AssignmentsSee Handouts: “Project (Part 3)”

200

Next Session:Quantifying Specifications Using Formal

MethodsReviewVerifying Requirements Mathematically

Using Set Theory and Logic Evaluating Agile Software EngineeringSoftware Engineering EthicsWeb Engineering TechniquesMeasuring User SatisfactionRefactoring and Anti-Patterns

Software Engineering G22.2440-001 · Static Testing Dynamic Testing Black-Box Testing White-Box...

Documents

Transcript of Software Engineering G22.2440-001 · Static Testing Dynamic Testing Black-Box Testing White-Box...