Software Testing 13. Integration

Software Testing13. Integration

Daniel Riegelhaupt - Thomas De Vylder - Gertjan Loots - Tim Saquet - Philip De Smedt

2

13. Integration

• Integration in Object-oriented Development

• Integration Patterns

3

13.1.1 Definitions

A System is composed out of Components whoare themselves systems of smaller components

Component(Focus of integration)

System(Scope of integration)

Typical intercomponent interface(Location of integration faults)

Method Class Instance variablesIntraclass messages

Class Cluster Interclass messages

Cluster Subsystem Interclass messagesInterpackage messages

Subsystem System Interprocces communicationRemote procedure callORB servicesOS services

4

13.1.2 Integration testing is essential

3 basic kinds of testing needed to show that components areminimally interoperable:

• Tests of individual components• Tests of the system resulting from the federation of components• Test of components interoperation

Integration testing = a search for component faults that cause intercomponent failures

5

Integration testing is essential

In contrast:• Component scope testing = search for intracomponent

faults• System scope testing = search for faults that lead to a

failure to meet a system scope responsibility

6


• Antidecomposition axiom:A test suite that achieves coverage at system scope will not necessarily result in the same coverage for its components

In other words: even if the program works as a whole this doesn’t mean we test every component fully (some code might never be reached)

• Anticomposition axiom:Adequate testing of components (in isolation) ≠ adequate testing of a system of components

7


Integrating most or all components in a system at the same time is usually problematic :

- Debugging is difficult (bug can be in any interface)- Last minute patches may be necessary (=> low quality, not

adequately tested )- No systematic testing can lead to undetected bugs

Incremental testing !

8


• Advantages of incremental testing:- Interfaces are systematically exercised and proven to

be stable before exercising unproven interfaces- Buggy interface code will be reached; if not triggered- Observe failures are most likely to come from most

recently added component• Incremental testing requires discipline and commitment:

choose a sequence of components by analyzing dependencies (=> architecture must be known

or planned!). Next manage plan and automate

10

• Integration testing takes place at all scopes and is repeated in each

development increment

11


Integration test plan must answer:• what are the major and minor development increments ?• which components interfaces / subsystems will be the focus

of integration ?• In what sequence will components and their interfaces be

exercised ?• Which stubs and drivers must be developed ?• Which test design patterns should be used ?• When can integration testing be considered complete ?

12


Key considerations:• Scope of an increment and dependencies of components• Testability ( # components testability )• Component stability (postpone integration of volatile

components, start with stable ones)• Test environment stability

13

13.1.3 Dependency AnalysisClass/cluster scope dependencies:• Composition and aggregation• Inheritance • Global variables …• pointers to object used as message parameters • …Intercomponent dependencies:• Explicit:

• Message exchange • Remote procedure calls• Interprocess communication

• Implicit: Not considered in Integration testing

14

Dependency Analysis

Explicit dependencies:• often dictate the sequence of testing• typically correspond to interfaces that should be exercised by an

integration TS

15

Dependency Analysis

Make a dependency tree:• by hand (if small)• Use topological search ( Unix: tsort; commercial tools)• Can have multiple roots, cyclical dependencies

Note that tools might have problems finding dependenciesover: Networks, VM , OS …

18

13.1.4 Faults Model

• Configuration/version control problems• Missing, overlapping or conflicting functions• Problems with file or DB ( incorrect/ inconsistent

representation; conflicting view /usage; data integrity ...)• Wrong method called due to coding or binding error• Client sends illegal message to server (violates pre

condition or sequential constraints)• Wrong object bound to message• Wrong parameter or incorrect parameter value

19

Faults Model• Failures due to incorrect memory management

(de)allocation• Incorrect usage of VM , ORB or OS services• Attempt by the IUT to use target environment services

that are obsolete / not supported in the currant version/ release

• Intercomponent conflicts (e.g. thread X will crash when process Y is running)

• Resource contention: target env. can’t allocate resources needed for nominal load (e.g. Need to open 6 windows but IUT crashes when fifth is opened)

IUT = implementation under test

20

13. Integration

• Integration in Object-oriented Development• Integration Patterns

21

Integration patterns

• Big Bang Integration. Attempt to demonstrate system stability by testing all components at the same time. - Useful in a few limited circumstances.

• Bottom-up Integration. Interleave component and integration testing by following usage dependencies. - Works well for small to medium systems.

22


• Top-down Integration. Interleave component and integration testing by following the application control hierarchy. - Works for almost any scope or architecture.

• Collaboration Integration. Choose an order of integration according to collaborations and their dependencies. Exercise interfaces by testing one collaboration at a time. - Works for almost any scope or architecture.

23


• Backbone Integration. Combine Top-down Integration, Bottom-up integration, and Big Bang Integration to reach a stable system that will support iterative development. - Well suited to small to medium systems and

especially useful for embedded applications.

• Layer Integration. Incrementally exercise interfaces and components in a layered architecture. - Applies to layered architectures.

24


• Client/Server Integration. Exercise a loosely coupled network of components, all of which use a common server component. - Appropriate for client/server architectures.

• Distributed Services Integration. Exercise a loosely coupled network of peer-to-peer components. - Appropriate for decentralized networks containing

peer-to-peer nodes.

25


• High-frequency Integration. Develop and rerun an integration test suite test hourly, daily, or weekly. - Can be applied at almost any scope or

architecture.

For complete scope specific considerations See book 13.2.1

26

Big Bang Integration

• Intent- Demonstrate stability with a few test runs

• Context- SUT is stabilized and only a few components have

been added since the last time- SUT is small and testable- Only feasible for monolithic systems

• Tightly coupled system• Bad design

27


• Fault Model- Does the system run? Demonstrate that system

testing can begin

• Strategy- No incremental integration testing- Entire system is built and a test suite is applied

28


29


• Entry Criteria- All components have passed component scope

testing- The VM to be used is stable- No anomalies found that interfere with integration

testing

30


• Exit Criteria- The test suite passes

• Consequences- Many interface bugs: where to look?

• Debugging is HARD: every component is suspect• Even if SUT passes: many hidden interface faults?

31


• Known Uses- Subsystem Big Bang Integration is used in

Backbone Integration.

32

Bottom-Up Integration

• Intent- Add components to the SUT in uses-dependency

order. Components with fewest dependencies first.

• Context- Stepwise verification between tightly coupled

components- Test each component and then integrate it with

already-tested components

33


• Context (2)- Useful when you can defer testing until all

components have been developed• Components developed by teams

- Early assessment of a component that must implement a critical requirement

• Fault Model- See the Integration Faults

34


• Strategy- Test Model

• Dependency tree• For each component: responsibility test suite

- Do not only test responsibilities» Also exercise subcomponents

- Test Procedure• Go from the leaves of the dep tree to the root

- n levels -> n stages• Major steps (see next slide)

35


1. Leaf-level components are coded. Test drivers are coded for the leaf-level components and the leaf level is tested by these drivers.

2. Components on the next higher level are coded. These components use object tested in a previous stage. Iterate for each level.

3. The entire system is exercised using the root-level component.

36


37


38


39


• Strategy- Automation

• One driver for each component• Bottom-up driver corresponds to

- Independent component- Set of components

• Testing several components with a single driver may be possible

- But you should avoid this- Reuse of drivers!- Reduce test driver maintenance

40


• Entry Criteria- VM is stable- Components in each stage meet exit criteria- No anomalies interfere with integration testing

• Exit Criteria- Each component meets its exit criteria- Interface (per subcomponent) has been tested

once or more- All root-level components pass their test suites

41

Bottom-Up Integration• Disadvantages

- Driver development (huge code size)• Promote reuse of drivers and test components

- Changes to a tested component• Revise test configuration

- No direct testing of intercomponent interfaces• Limited to component that is root of the subtree under

test- Getting values or exceptions from lower-level

components = difficult- Critical sections are tested last

• Highest level of control in root -> tested last• Unstable component indications will come when

component is exercised and not when it is integrated.

42


• Advantages - Start testing as soon as any leaf-level component

is ready- Work in parallel

• Using subtrees- Well suited to responsibility-based design

• Only component interfaces are exercised.- Find mistakes in requirements

43


• Known Uses- Used with clusters- Integration by Contract

• Test design based on the relationship between client and server classes

- Object Relations• “To test a class, what other classes must be tested?”

• Read “Testing Object-Oriented Systems” for more information

44

Top-down Integration

• Intent- Add components to the SUT in control hierarchy

order, beginning with top-level control objects

• Context- Iterative and incremental approach- Control structure of the system

• Modeled as a tree• Top-level components have control responsibilities

- Control objects

45


• Context- Control objects

• Essential control strategies: high risk- Focus on control object components first

- Incremental development• Add layer under control layer on each increment

- Concurrent hw/sw development• Develop hardware interfaces and control components at

the same time- Concurrent sw development

• Develop VM and control components at the same time

46


47


48


49


• Disadvantages - Stub development

• Large number of stubs coded• Unforeseen requirements in a lower-level component may

change many top-level components, breaking (part of) the test suite

• Stubs are implementation-specific

- Interoperability of all components in the SUT is not tested until the last component replaces its stub and the test suite passes

50


• Advantages - Early testing and integration- Reduced cost of driver development

• Only one driver instead of a driver per subtree- Upper-level components are tested multiple times

• Each time a lower-level component is added• Regression testing!

- Parallel development• Different levels of components and stubs

51

Collaboration Integration

• Intent- Add sets of components to the SUT that are

required to support a particular collaboration

• Context- Test interfaces between participants of a

collaboration- Organize integration according to collaborations- Not every collaboration must necessarily be

tested• Completed when all components and interfaces have

been exercised

52


• Context- Indicators (or when to use collaboration

integration)• Use it when SUT has clearly defined collaborations that can

cover all components and interfaces in its scope• To demonstrate a working collaboration asap. High

technical risk or critical collaboration.• Some system require more testing than just a pass through

the top of the control hierarchy.- Example: search function with critical performance objectives

• Fault model- See Integration Faults

53

Collaboration Integration• Test Procedure

1. Develop a dependency tree• Map collaborations onto the tree

2. Choose a sequence in which to apply the collaborations• Simplest collaborations first, complex later• Start with the collaboration that requires fewest stubs• Use a state machine that models sequential constraints on

collaborations• Test in order of risk of disruption for system testing

3. Develop the test suite for the first collaboration4. Run the test suite. Debug until the first

collaboration passes. Revise

54


55


56


57

Collaboration Integration• Automation

- Single driver coupled to the interface for the collaboration

- Stubs required• For untested components that do not participate in a

collaboration. Relevant stubs are replaced with each collaboration

• Entry Criteria- VM is stable- Components in collaboration are operable- No anomalies interfere with integration testing

58


• Exit Criteria- Each tested collaboration meets the exit criteria- All component and intercomponent messages in

the system have been exercised at least once

59


• Disadvantages - Intercollaboration dependencies may be subtle- Participants in a collaboration are not tested

separately• Remember Big Bang Integration?

- Requirement of stubs- Testing lower-level components may be difficult- Collaborations may be incomplete

60


• Advantages - Interface coverage in a few test runs

• Possibly without testing all collaborations- Useful for system scope testing

• Focus on end-to-end functionality- Minimal coupling due to a single test interface- Testing and integration may begin when

components in the first collaboration are coded• Between top-down and bottom-up integration

- Minimize driver development costs

61

Backbone Integration

• Intent- Combine elements of Top-down Integration,

Bottom-up Integration and Big Bang Integration to verify interoperability among tightly coupled subsystems

• Context- Used e.g. by embedded system applications.

• Backbone provides services essential for running tests and applications

• Creating doubles for entire backbone would be impractical

62



• Strategy- Test model

• The top and possibly the second level are tested top-down• Application subsystems are developed bottom-up• Backbone components tested in isolation• The tested components are then big banged.

63


- Test Procedure1. Each backbone component is tested first in isolation,

bottom-up. Drivers and stubs are needed. (Fig 1)2. Application control is tested next, top-down. Stubs are

needed. (Fig 2)3. Backbone components are big-banged together and

tested under a driver. (Fig 3)4. Big Bang control components and backbone. (Fig 4)5. Use top-down on the remaining leaves. (Fig 5)

64


Figuur 1

65


Figuur 2

66


Figuur 3

67


Figuur 4

68


Figuur 5

69

Backbone Integration• Automation

- Stubs and drivers are required.

• Entry criteria- System architecture and component interfaces

have been designed and passed careful review- Backbone and application of roughly same size and

complexity.

70


• Exit criteria- All components meet exit criteria for applied test pattern- Each intercomponent interface exercised at least once.- Complete increment passed system scope test suite.

• Advantages - Mitigates disadvantages of Top-Down and Bottom-Up

(curtails their use at the point they lose effectiveness)

• Disadvantages - Requires careful analysis of system structure and

dependencies- Requires stubs and drivers

71


• Known uses- Development of Windows NT can be characterized

as a Backbone Integration followed by High-Frequency Integration.

• Related patterns- Sandwich integration:

• Combination of Top-Down and Bottom-Up• Final configuration meeting in the middle• But Backbone requires more testing of components in

isolation.

72

Layer Integration

• Intent- Uses an incremental approach to verify stability in

a layered architecture.

• Context- Verifies interoperability in systems that can be

modeled as an hierarchy that only allows interfaces between adjacent layers.

• Top layer doesn’t communicate with bottom layer• Instead, top layer communicates with middle layer and the

middle layer communicates with bottom layer

73

Layer Integration

• Fault model- See integration faults- Propagation through layers must meet hard, real-

time deadline Inadequate performance is often a critical failure

• Strategy- Test Model

• Top layer and possible second layer tested top-down.• Middle layer developed bottom-up.• Primary components bottom layer tested in isolation

74

Layer Integration

- Test procedure• May be Top-Down or Bottom-Up

1. Test each layer in isolation using drivers where needed. (Fig 1)2. Perform Top-Down/Bottom-Up Integration of layers.

• Figure 2 shows the test configurations for a Top-Down Integration, Bottom-Up would reverse the order.

• Automation- Requires stubs and drivers for each layer- Driver for top layer must be designed so it can

exercise entire system

75

Layer Integration

Figuur 1

76

Layer Integration

Figuur 2

77

Layer Integration

• Entry criteria (same as Collaboration Integration)- VM is stable, else consider Backbone Integration- Components in collaboration are operable- No anomalies interfere with integration testing

• Exit criteria- Each driven component meets exit criteria for its

test pattern- Each collaboration that traverses two or more

layers has been exercised once.

78

Layer Integration

• Advantages & Disadvantages- Similar to those of Top-Down (for the top-down

variant) and Bottom-Up (for the bottom-up variant)

• Known uses- Protocol stacks are a widely used layered application

79

Client/Server Integration

• Intent- Demonstrate stability of interactions among clients

and servers. - Initially test clients and servers in isolation, then

controlled increases in scope

• Context- Client/Server Integration achieves stepwise

verification between client components that are loosely coupled to a single server component

- Concurrent development/testing is possible

80



• Strategy- Test model

• Test plan must identify clients and servers• In two-tier star basic integration strategy is client-server • In three-tier star both client-server and server-server

configurations, followed by client-server-server configuration

81


- Test procedure• Each client is tested with stub from server• Server is tested with stubs from all client types• Next, pairs of clients tested with actual server (server may

retain stubs for other clients)• Finally, all stubs are removed and individual use case tests

are replayed.

• Two-tier client/server star integration follows a three-step procedure for each client group:

1. Representative client + server stub2. Server + client stub3. Representative client + server

82


Figuur 1

83


• Automation- Test script for a stage should be designed to support

two kinds of use• Should be repeatable to support regression testing upon

integration of more components• Should be extensible so system scope tests may be added

after final integration testing is completed- Debugging/Testing tools mostly platform-specific

• Difficult to perform controlled testing and debugging across multiple platforms

- Replicate an isomorphic test environment to replicate the structure of the architecture

84


• Entry criteria- VM is stable, else consider Backbone Integration- Components in collaboration are operable- No anomalies interfere with integration testing- Multiplatform test suite installed and working- Multiplatform version control system

• Exit criteria- Each driven component meets exit criteria for its test

pattern- Interface to each subcomponent exercised at least once- Test suite passes in test environment that is isomorphic

with respect to the target environment

85


• Advantages - Avoids problems of Big Bang Integration- Order of client/server integration usually has few

constraints integration sequenced by priority/risk

• Disadvantages - Cost of driver and stub development- Testers cannot exercise end-to-end use cases until

midway or late in development cycle.

• Known uses- Read book for extensive example

86


• Entry criteria- VM is stable, else consider Backbone Integration- Components in collaboration are operable- No anomalies interfere with integration testing- Multiplatform test suite installed and working- Multiplatform version control system

• Exit criteria- Each driven component meets exit criteria for its test

pattern- Interface to each subcomponent exercised at least once- Test suite passes in test environment that is isomorphic

with respect to the target environment

87

Integration Patterns

• Distributed Services Integration. Exercise a loosely coupled network of peer-to-peer components. Appropriate for decentralized networks containing peer-to-peer nodes.- See book for more info

• High-frequency Integration. Develop and rerun an integration test suite test hourly, daily, or weekly. Can be applied at almost any scope or architecture.- See book for more info

Software Testing 13. Integration

Documents

Transcript of Software Testing 13. Integration