Strategies for Avoiding Test Fixture Smells duringSoftware Evolution

Michaela Greiler, Andy Zaidman, Arie van Deursen, Margaret-Anne Storey

Test Fixture Smells

2

What is a test smell?

A symptom of a problem

• Can originate from several causes

• Refactoring needed

3

Structure of a Test

4

Test (method)

Test Structure

1. Statement

2. Statement

3. Statement

4. Statement

5. Statement

6. Statement

Setup of Test Fixture

Verify Outcome

Exercise SUT

Teardown Test Fixture

5

Test Fixture

The code that initializes and configures the system under test

6

How to structure the test fixture and where to place it?

Test – Inline Fixture

Setup within method

Test Fixture

8

Test – Implicit

Setup

Method

9

Test – Delegate

Method

createTestAirportOriginTest ()

createTestAirportOriginTest(); explicit invocation

deleteTestAirportOriginTest()

deleteTestAirportOriginTest(); }

10

Delegate

“Most of the complexity of writing tests involves how to write the Test Methods; what to include

inline and what to factor out into Test Utility Methods, and so on.”

(Meszaros)

11

Test Fixture Smells

• General Fixture

• Test Maverick

• Dead Fields

• Lack of Cohesion of Test Methods

• Vague Header

• Obscure In-line setupICST 2013: Greiler, van Deursen, Storey

Automatic Detection of Test Fixture Strategies and Smells

12

Developers recognize test fixture smells as a problem (ICST’13)

13

Developers recognize test fixture smells as a problem (ICST’13)

Resolving these smells after a long time can be problematic.

Immediately awareness of smells density important.

14

1. Investigate the evolution of test fixture smells and2. Understand which software changes lead to increased test smell densities

to develop strategies to avoid fixture smells and to determine the best time to alert developers.

15

Resolving these smells after a long time can be problematic.

Immediately awareness of smells density important.

Azure

39 KLOCs30 T-classes

358 T-methods300 revisions

1 year

JSoup

20 KLOCs 23 T-classes

372 T-methods 973 revisions

2.5 years

Checkstyle

66 KLOCs 156 T-classes

549 T-methods 2251 revisions

9.5 years

PMD

174 KLOCs 118 T-classes

739 T-methods 1900 revisions

5 years

Voldemort

130 KLOCs 132 T-classes

520 T-methods 2900 revisions

1.5 years

Case studies

16

TestEvoHound…is a static analysis tool that analyzes the test

fixture smell evolution.

TestEvoHound…is a static analysis tool that analyzes the test

fixture smell evolution.

1.Revision Checkout2.Build Process 3.Test Fixture Smell Analysis 4.Trend Analysis

Fixture smells among projects

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Azure Checkstyle JSoup PMD Voldemort

Test Mavericks General Fixture Vague Header Dead Fields LCOTM Inline

19

Does Test Fixture Smell Density Increase?

A general growth of test fixture smells over timedoes not occur.

20

Lehman’s law: when a system evolves, its complexity increases unless work is done to maintain or reduce it.

Increase in smell density

Checkstyle: dead fields

Refactoring and forgotten functionality: “Added a helper method to create a configuration for a check...”.

21

Decrease in smell density

Azure: general fixture

New, non-smelly tests added: “Table Client commit [...]”.

22

Smell Dispersion

Over time, smelly classes get smellier

Smells cluster in a few classes

23

Correlations

?Number of t-methods per test class correlates with test fixture smell density

Number of fields do not necessarily correlate with smell density

# T-methods

# Fields

24

Strategies and Recommendations

Keep test classes small

Keep inheritance structures flat

Limit the scope of super classes

Use composition instead of inheritance

Reconsider the “one test class per class” organization

25

Findings

0%

20%

40%

60%

80%

100%

Azure Checkstyle JSoup PMD VoldemortTest Mavericks General Fixture Vague Header

Dead Fields LCOTM Inline

# T-methods# Fields

26

0%

20%

40%

60%

80%

100%

Azure Checkstyle JSoup PMD VoldemortTest Mavericks General Fixture Vague Header

Dead Fields LCOTM Inline

# T-methods# Fields

Thanks! Questions?

Connect with meMichaela Greiler

michaela.greiler@microsoft.com

mgreiler

27