Squish Coco 3.2.3 - Copyright ©2015 froglogic GmbH

CONTENTS

Contents

1 Introduction 1

1.1 Squish Coco - Code Coverage Tool for Tcl, C# and C/C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 CoverageScanner—Instrumentation during the Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 CoverageBrowser—View, Analyse, and Manage, Code Coverage Results . . . . . . . . . . . . . . . . . . . 2

I Quick Start and Tutorials 4

2 Synopsis 5

3 Using Squish Coco 6

4 Creating an instrumented project 7

4.1 Installing Squish Coco . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4.2 C++ on Microsoft Visual Studio using the Microsoft Visual Studio Add-in . . . . . . . . . . . . . . . . . . . 7

4.3 C# on Microsoft Visual Studio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.4 Tcl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.4.1 Using more than one Tcl version on one system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4.5 Command Line Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

5 Generating Instrumentations Without Modifying Projects 12

5.1 GNU Make . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.2 Microsoft NMake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.3 Microsoft Visual Studio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5.4 Microsoft MSBuild . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5.5 Mono C# XBuild . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

6 Instrumenting a simple project 14

6.1 Prerequisites and installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

6.2 Structure of the parser directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

6.3 Compiling and testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

6.4 Instrumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

- i - froglogic GmbH

CONTENTS

6.5 How the project is instrumented . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

6.6 Additional Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

7 Getting started with Qt 18

7.1 Compiling the example application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

7.1.1 The First Code Coverage Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

7.1.2 Interactive testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

7.1.3 Writing unit tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

7.2 Working with code coverage data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

7.2.1 Post mortem analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

7.2.2 Evaluating the impact of a hot fix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

7.2.3 Black-box testing/distributed testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

7.2.4 Verifying if a bug fix is correctly tested . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

7.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

II Code Coverage Overview 33

8 Code Coverage Overview 34

8.1 Code instrumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

8.1.1 Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

8.1.2 Result of coverage analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

8.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

8.3 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

8.4 Other Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

8.4.1 Function Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

8.4.2 Line Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

9 Testing Methodologies 42

9.1 Hit vs Count . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

9.2 Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

9.2.1 Manual tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

9.2.2 Black box tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

9.2.3 Unit Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

9.2.4 Automatic Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

III CoverageBrowser Reference Manual 45

10 Introduction 46

- ii - froglogic GmbH

CONTENTS

10.1 Command Line Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

11 Black box and white box testing 47

12 The windows of CoverageBrowser 48

12.1 The Executions Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

12.1.1 Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

12.1.2 Loading an Execution Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

12.1.3 The Execution Comparison Analysis Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

12.2 The Source Browser Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

12.3 The Function Browser Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

12.4 The Source Viewer Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

12.4.1 Source Display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

12.4.2 Color Convention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

12.4.3 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

12.4.3.1 Editing Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

12.4.3.2 Removing Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

12.5 The Explanation Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

12.6 The Statistics Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

13 Working with CoverageBrowser 60

13.1 Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

13.1.1 Wildcard Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

13.1.2 Regular Expression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

13.1.2.1 Pattern matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

13.1.2.2 String substitution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

13.2 Code Coverage Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

13.3 Code Coverage Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

13.4 Optimized Execution Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

13.5 Comparing Releases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

13.5.1 Reference Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

13.5.2 Coverage analysis of modified/unmodified source code . . . . . . . . . . . . . . . . . . . . . . . . . 64

13.6 Changing the Instrumentation Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

13.6.1 Merging Instrumentations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

13.6.2 Importing Unit Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

13.6.3 Importing Reviewer Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

- iii - froglogic GmbH

CONTENTS

14 Generation of Reports 66

14.1 EMMA-XML Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

14.2 Cobertura-XML Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

14.3 JUnit Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

14.4 Statistics per Source File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

14.5 Statistics per Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

14.6 HTML Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

14.7 Text Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

15 Preferences 69

15.1 Save/Load Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

15.2 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

15.3 Thresholds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

15.4 Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

IV CoverageScanner Reference Manual 71

16 Introduction 72

16.1 Invoking CoverageScanner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

17 Command Line Arguments 74

17.1 Environment variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

17.2 List of options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

17.2.1 File inclusion/exclusion options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

17.2.2 Function inclusion/exclusion options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

17.2.3 Instrumentation options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

17.2.4 Debugging options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

17.2.5 Execution report options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

17.2.6 Miscellaneous options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

17.3 Instrumenting using preprocessor defines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

17.4 Regular Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

18 Library calls 83

18.1 C and C++ Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

18.1.1 __coveragescanner_install() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

18.1.2 __coveragescanner_testname() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

18.1.3 __coveragescanner_teststate() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

18.1.4 __coveragescanner_add_html_comment() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

- iv - froglogic GmbH

CONTENTS

18.1.5 __coveragescanner_clear_html_comment() . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

18.1.6 __coveragescanner_save() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

18.1.7 __coveragescanner_clear() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

18.1.8 __coveragescanner_filename() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

18.1.9 __coveragescanner_register_library() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

18.1.10 __coveragescanner_unregister_library() . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

18.1.11 __coveragescanner_set_custom_io() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

18.2 C# Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

18.2.1 CoverageScanner.__coveragescanner_init() . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

18.2.2 CoverageScanner.__coveragescanner_testname() . . . . . . . . . . . . . . . . . . . . . . . . 88

18.2.3 CoverageScanner.__coveragescanner_teststate() . . . . . . . . . . . . . . . . . . . . . . . 89

18.2.4 CoverageScanner.__coveragescanner_add_html_comment() . . . . . . . . . . . . . . . . . . . 89

18.2.5 CoverageScanner.__coveragescanner_clear_html_comment() . . . . . . . . . . . . . . . . . 89

18.2.6 CoverageScanner.__coveragescanner_save() . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

18.2.7 CoverageScanner.__coveragescanner_clear() . . . . . . . . . . . . . . . . . . . . . . . . . . 90

18.2.8 CoverageScanner.__coveragescanner_filename() . . . . . . . . . . . . . . . . . . . . . . . . 90

18.2.9 CoverageScanner.__coveragescanner_set_custom_io() . . . . . . . . . . . . . . . . . . . . . 90

19 Controlling the instrumentation during the compilation 92

19.1 Source Code Annotations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

19.1.1 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

19.1.2 Manual Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

19.2 C and C++ Pragma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

19.3 C# regions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

20 IDE Support 96

20.1 GNU Makefile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

20.2 CygWin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

20.3 Scratchbox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

20.4 CMake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

20.4.1 Coverage Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

20.4.2 Microsoft Visual Studio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

20.4.3 GNU GCC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

20.5 Qt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

20.5.1 Qt Creator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

20.5.2 qmake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

20.5.3 moc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

- v - froglogic GmbH

CONTENTS

20.6 SCons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

20.7 ARM Keil . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

20.8 Visual DSP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

20.9 Microsoft Visual Studio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

20.9.1 Microsoft Visual Studio .NET C# Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

20.9.2 Microsoft Visual Studio .NET C++ Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

20.9.3 Squish Coco Microsoft Visual Studio Add-In . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

20.9.4 Microsoft Visual C++ Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

20.9.5 Microsoft Visual Studio 6.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

20.10eMbedded Visual C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

20.11Eclipse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

20.12Xcode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

21 Code Coverage of Libraries 116

21.1 Code Coverage of Static/Shared Libraries and DLL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

21.2 Code Coverage of Plugins/Manually Loaded Shared Libraries . . . . . . . . . . . . . . . . . . . . . . . . . 116

21.2.1 Generating Code Coverage Information directly from the Main Application . . . . . . . . . . . . . . 116

21.2.2 Generating Code Coverage Information directly from the Plugin . . . . . . . . . . . . . . . . . . . . 117

21.2.2.1 Code Coverage of Plugins Generated with Microsoft Visual Studio . . . . . . . . . . . . . 118

21.2.2.2 Code Coverage of Plugins Generated with GNU gcc . . . . . . . . . . . . . . . . . . . . . 118

22 Test Suites and Squish Coco 120

22.1 Execution Comment, Name And Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

22.2 Controlling the generation of the execution report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

22.2.1 Generation of an execution report using UNIX signals . . . . . . . . . . . . . . . . . . . . . . . . . 121

22.2.2 Generation of an execution report using Microsoft Windows event . . . . . . . . . . . . . . . . . . . 121

22.2.3 Unit Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

22.3 Support for specific test frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

22.3.1 CppUnit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

22.3.2 QTestLib . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

22.3.3 GoogleTest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

22.3.4 CxxTest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

22.3.5 NUnit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

22.3.6 Squish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

22.3.6.1 General Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

22.3.6.2 Simplified for Reuse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

- vi - froglogic GmbH

CONTENTS

V CoverageScannerTcl Reference Manual 139

23 Command Line Arguments 140

24 Mixing Tcl with C++ 141

24.1 Analysis of Tcl script coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

24.1.1 Demo application build . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

24.1.2 Ad-hoc Tcl coverage run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

24.1.3 Result analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

24.1.4 Filtered Tcl coverage run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

24.1.5 Report Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

24.2 Analysis of C/C++ code coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

24.2.1 Instrumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

24.2.2 Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

24.2.3 Result analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143


24.2.5 Analysis of mixed Tcl and C/C++ code coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

24.2.6 Combined coverage run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

24.2.7 Merging measurement data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144


VI Command Line Tools 146

25 cmreport - Code coverage report generation 147

25.1 File selection options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

25.2 Execution selection options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

25.3 Options for HTML or CSV reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

25.3.1 Obsolete options for CSV reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

25.4 Options for text reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

25.5 Options for EMMA-XML reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

25.6 Options for JUnit reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

25.7 Options for Cobertura reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

26 cmcsexeimport 154

27 cmmerge - Merging Utility 156

- vii - froglogic GmbH

CONTENTS

28 License management 157

28.1 cocolic - License activation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

28.2 Location of the license . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

29 cocolicserver 159

VII Frequently Asked Questions 160

30 Installation 161

30.1 Is there a way to deploy Squish Coco on Windows without any user interractions? . . . . . . . . . . . . . . . 161

30.2 Where are the settings stored? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

30.3 License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

30.3.1 How do I install a license key on my Jenkins CI? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

30.4 Microsoft Visual Studio Installation Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

30.4.1 Microsoft Visual Studio Compiler Warning D9002 . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

30.4.2 Microsoft Visual Studio Add-In is Not Installed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

30.4.3 Microsoft Visual Studio Add-In is Crashing when "My Documents" is on a Network Drive . . . . . . 164

31 Compiling 165

31.1 Using CoverageScanner with ccache (Unix only) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

31.2 Compiling issue when using the Boost library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

31.3 Microsoft MSBuild Does Not Call CoverageScanner on x86 Platforms . . . . . . . . . . . . . . . . . . . . . 166

32 Instrumentation 167

32.1 Compiling Issues of Lambda Expressions with Microsoft VisualStudio 2010? . . . . . . . . . . . . . . . . . 167

32.2 How to exclude a source file from the code coverage analysis? . . . . . . . . . . . . . . . . . . . . . . . . . 167

32.3 My source code contains inline functions, are they instrumented? . . . . . . . . . . . . . . . . . . . . . . . . 168

32.4 Is it normal that the .csmes file contains a copy of the source code? . . . . . . . . . . . . . . . . . . . . . . . 168

33 CoverageBrowser 169

33.1 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169

33.1.1 Is there a way to minimize the CPU usage? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169

33.1.2 When I’m exporting the statistics per methods I can only see the 65536 first entries . . . . . . . . . . 169

34 Unit Tests 170

34.1 How to get the code coverage of a library from my unit tests? . . . . . . . . . . . . . . . . . . . . . . . . . . 170

35 Reports 172

35.1 Is is possible to compute a code coverage report of selection of files? . . . . . . . . . . . . . . . . . . . . . . 172

- viii - froglogic GmbH

http://ccache.samba.org/

CONTENTS

36 Miscellaneous 173

36.1 The generation of an execution report through Windows events does not work when logged on an other account?173

36.2 Is it possible to debug CoverageScanner API? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

36.3 How to report an issue concerning CoverageScanner? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174

VIII Continuous Integration Server 175

A Atlassian Bamboo integration 176

A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176

A.2 Agent configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

A.3 Importing an execution report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

A.4 Merging instrumentation databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

A.5 Generating coverage report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

A.6 Use Case Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

A.6.1 Coverage Report from single execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

A.6.2 Coverage Report for Unit Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

IX Appendix 181

B Release Notes 182

B.1 Squish Coco v3.2.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

B.2 Squish Coco v3.2.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

B.3 Squish Coco v3.2.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

B.4 Squish Coco v3.2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

B.5 Squish Coco v3.1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

B.6 Squish Coco v3.0.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184

B.7 Squish Coco v3.0.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

B.8 Squish Coco v3.0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

B.9 Squish Coco v3.0.0-pre1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186

B.10 Squish Coco v2.1.8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186

B.11 Squish Coco v2.1.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

B.12 Squish Coco v2.1.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188

B.13 Squish Coco v2.1.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188

B.14 Squish Coco v2.1.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

B.15 Squish Coco v2.1.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

B.16 Squish Coco v2.1.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

B.17 Squish Coco v2.1.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

- ix - froglogic GmbH

CONTENTS

B.18 Squish Coco v2.1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

B.19 Squish Coco v2.0.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

B.20 Squish Coco v2.0.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

B.21 Squish Coco v2.0.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

B.22 Squish Coco v2.0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

C Code insertion 194

D Code Coverage Benchmarks 197

D.1 Test Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

D.2 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

E License Agreement 200

F Customizing I/O of CoverageScanner library 203

F.1 Custom I/O using C file access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

F.2 Custom I/O using SFTP protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

G Supported compilers 212

G.1 C# compilers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212

G.2 Microsoft Visual C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212

G.3 Intel C++ Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213

G.4 GNU gcc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213

H CoverageScanner Adaptation to a Tool Suite 214

H.1 Profile Parameters for Architecture Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215

H.2 Profile Parameters for Instrumentation Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215

H.3 Profile Parameters for Preprocessor Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215

H.4 Profile Parameters for Linker Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216

H.5 Profile Parameters for Compiler Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218

H.6 Profile Parameters for Precompiled Headers Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

H.7 Profile Parameters for Custom IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222

H.8 Miscellaneous Profile Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222

H.9 Instrumentation Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

Index 224

- x - froglogic GmbH

Introduction

Chapter

1 Introduction

1.1 Squish Coco - Code Coverage Tool for Tcl, C# and C/C++

Squish Coco is a complete code coverage tool chain for Tcl, C# and C/C++ programs that runs on Apple® Mac OS X, Linux™

and Microsoft® Windows. It analyzes the performance of an application being exercised (e.g., by a test suite), to reveal importantinformation that can improve the testing of Tcl, C# and C/C++ applications. In particular, Squish Coco can:

• Find untested code sections.

• Find redundant tests which can then be eliminated.

Squish Coco makes it possible to identify which portions of the source code are covered by a test, and to detect whether anew test covers source code lines that the existing tests don’t cover.

• Find dead code by displaying code that is never executed.

• Calculate the optimum test execution order so as to maximize test coverage for each run. This is particularly useful formanual testing.

• Analyse two separate versions of an application and compare the differences.

This feature makes it possible to see which tests are affected by source code modifications and also to get some measureof the test coverage of a patch or hot fix.

Squish Coco can be used at every stage of testing and for all testing methodologies (unit tests, automatic tests, manual whitebox tests, black box tests, etc. . . ), and makes it possible to merge multiple execution reports into a single unified report.

The Squish Coco package provides two separate tools and an add-in:

1. CoverageScanner—this analyzes, instruments and generates the Tcl, C# and C/C++ application.

2. CoverageBrowser—this displays and manages the results of the coverage analysis.

3. Microsoft® Visual Studio® Add-In—this is an optional add-in which makes it possible to generate code coverageconfigurations for every Tcl, C# and C/C++ project created by Microsoft® Visual Studio®.

- 1 - froglogic GmbH

CoverageScanner—Instrumentation during the Generation

CoverageScanner CoverageBrowser

Source Code

(.cpp file)C/C++ Preprocessing

Instrumentation

Compiling

Instrumentation(.csmes file)

Executions

(.csexe file)

Application

(.exe file)

Figure 1.1: Overview of Code Coverage Toolchain for Tcl, C# and C/C++

1.2 CoverageScanner—Instrumentation during the Generation

CoverageScanner instruments the source code seen by the compiler (leaving the original code untouched) and generates aninstrumented executable, shared library, or plugin. The instrumentation keeps track of the code that is executed when theexecutable is run or when the library or plugin are used (e.g., how many times each line is executed, or simply whether a linewas executed) and produces an execution report (.csexe file) upon program termination. In order to get the best quality ofcode coverage metrics, CoverageScanner instruments not only at the function and statement level, but also records statementcoverage and decision coverage information. Optionally, it is also possible to insert the test name and its execution status(passed or failed) into the report using a batch script or directly in the application itself. This makes it possible to integrateCoverageScanner into a test framework (e.g., CppUnit or CxxTest), with the generation of the code coverage information foreach test item.

CoverageScanner is a command line tool which substitutes itself for the the compiler that is normally used to build theexecutable, library, or plugin. Its main function is to insert instrumentation instructions into the preprocessed source code andthen to compile this modified code in place of the original code using the normal compiler. CoverageScanner works withalmost any standard compiler, including GNU gcc, GNU g++, Microsoft® Visual Studio® 6.0, .NET, Express, and EmbeddedC++, Intel® C++, etc. A database (.csmes file), which contains the list of instrumentations and a copy of the instrumentedsource code, is generated at the same time for later analysis. CoverageScanner’s ability to work with most compilers is becauseit can be configured to suit most compilers’ needs.

After the code generation, the instrumented executable can be run as usual (or the instrumented library or plugin used viathe program that accesses them)—this will result in an execution report being produced (in a .csexe file) when the programterminates.

InfoCoverageScanner has built-in support for Qt Library which makes it possible to prevent the instrumentationof code generated by the moc.

1.3 CoverageBrowser—View, Analyse, and Manage, Code Coverage Results

The execution report (in the .csexe file) can be analyzed and managed using the CoverageBrowser tool. This tool has a GUI(graphical user interface) through which the user can browse and manage their execution reports. This makes it straightforwardto locate untested code, dead code, and inefficient tests.


http://qt-project.org/doc/qt-4.8/moc.html

CoverageBrowser—View, Analyse, and Manage, Code Coverage Results

CoverageBrowser’s highlights:

• Commenting instrumented source code lines.

• Displaying the list of executions in a tree view.

• Managing portions of code which cannot be tested by identifying them as “Manually Validated”.

• Browsing through the instrumented code.

• Switching between code coverage condition and code coverage branch analysis.

• Displaying a detailed explanation of the state of the instrumentation. This includes:

– User comments.– The state (executed, not executed, partially executed).– The code coverage count.– The list of tests executing the instrumentation.

• Retrieving the execution status from an automated test suite.

• Code coverage analysis of unit tests.

• Exporting statistics to spreadsheets.

• A test benefit analysis mode which makes it possible to see what additional code coverage a set of tests adds.

• An API which makes it possible to control CoverageBrowser using an external Tcl, C# and C/C++application.

• A source code and method browser which displays code coverage statistics per file, per class, per namespace, and per Tcl,C# and C/C++ function.

• Generation of a report in HTML format, which includes statistics per source file, methods, executions, and a list ofunexecuted code lines.

• Black box testing support.

• The ability to comparare coverage data for different versions of the software.

Navigation

Source Navigation

and Statistics

Executions

Management

Manual

Validation

User

Comments

Detailed

Explanation

Class, Namespace,

and Method

Browser

Figure 1.2: CoverageBrowser: Code Coverage Screenshot


Part I Quick Start andTutorials


Synopsis

Chapter

2Synopsis

Code coverage is commonly used in the development of safety critical applications in order to obtain a measure of how muchof the code is tested. This approach is also used more widely, especially when the application quality is important.

To be able to analyze the code coverage that a test suite achieves it is necessary to compile a version of the the application inwhich statements are inserted that monitor the execution of the source code. The generation of such a modified version of theprogram is called instrumentation.

Several kinds of instrumentation are possible. The most common are:

Line coverage:Instrumenting the execution of every executable source code line.

Branch coverage:Instrumenting the execution of each branch’s block (e.g., the body of an if statement).

Decision coverage:Instrumenting each Boolean decision for loop and selection statements (e.g., record both the Boolean expressionitself—true or false—and the body of the while, for or if statement).

Condition coverage:Instrumenting of each sub-expression of Boolean expressions (i.e., like decision coverage but for multi-part Booleanexpressions recording every part).


Using Squish Coco

Chapter

3Using Squish Coco

Squish Coco is a coverage tool for C and C++ code. In principle it does the same job as GNU gcov, but has some additionaland different functionality. For example, Squish Coco:

• works on release and debug builds.

• uses generic code and is compiler independent.

• instruments branches, decisions, and conditions, whereas gcov is limited to line coverage.

• provides a graphical code browser.

• records an instrumentation report for each test run—these reports can be compared (e.g., to see if the coverage hasimproved), or merged.

• supports adding code coverage reports from unit tests into the main application’s instrumentation.

• makes it possible to compare two versions of the application to see what effect the code changes have on code coverage.

• supports black box testing.

• provides an extension to instrument Qt source code that omits the unnecessary Qt infrastructure.

Squish Coco consists of two main tools:

CoverageScanner :This is a compiler wrapper which feeds instrumented source code to the native compiler. Using it is easy—simply tell thebuild tool to use CoverageScanner as the compiler.

CoverageBrowser:This is a GUI tool for visualizing and analysing code coverage based on the reports generated by running the test suite onan instrumented version of the application.

During the compilation, an instrumentation database is generated for each object, library, and executable. This database containsa copy of the source code, the instrumentation points, and once executed, all the test reports.


Creating an instrumented project

Chapter

4Creating an instrumented project

This chapter contains recipes how a software project can be set up for instrumentation. The methods depend on the languagesand the development environments. Here we show how to create a new software project so that it is prepared for instrumentation.This is also a simple way to get acquainted with Squish Coco.

A description of a larger project can be found in Chapter 7, page 18.

4.1 Installing Squish Coco

Installing Squish Coco is straightforward: just download the installer via the links at http://www.froglogic.com/squish/coco and execute it.

4.2 C++ on Microsoft® Visual Studio® using the Microsoft® Visual Studio® Add-In

There is an add-in for Microsoft® Visual Studio® (see Chapter 20.9.3, page 108) that supports the work with Squish Coco. Thefollowing description shows the setup of a project that uses this add-in.

Start Microsoft® Visual Studio® and create a new C++ application:

1. Click on "FileÕNewÕProject..." to pop up the new project wizard.

2. Choose a project type of "Visual C++ÕWin32" and the "Win32 Console Application" template.

3. Enter a project name of squishcoco_sample, then click the "OK" button.

4. When the wizard’s second page appears, click the "Finish" button.

At this stage the application is not yet instrumented. So now we will create a copy of the build and activate the instrumentation:

1. Open the configuration manager by clicking "BuildÕConfiguration Manager...".


http://www.froglogic.com/squish/coco

http://www.froglogic.com/squish/coco

C++ on Microsoft Visual Studio using the Microsoft Visual Studio Add-in

2. In the "Configuration" column, select "New..." in the combobox

3. In the "New Project Configuration" dialog:

(a) Enter Code Coverage in the "Name" field,

(b) Select Release or Debug in the "Copy settings from" selection dialog.

(c) Click the "OK" button.

To activate the instrumentation, use the Microsoft® Visual Studio® Add-In:

1. Click "ToolsÕCode Coverage Build Mode..." to pop up the Squish Coco wizard.

2. In the "Project:" selection dialog, select squishcoco_sample.

3. In the selection dialog "Configuration:", select Code Coverage.

4. In the Configuration section at the bottom, select the radio button "Modify", and then click on the button, "Enablecode coverage for C++ projects".

The Code Coverage configuration has now been modified to generate code coverage information. The "SquishCoco" outputwindow summarizes all the modifications that have been made:

...Modifying configuration ’Code Coverage’ for the project

’squishcoco_test’ for the platform ’Code Coverage|Win32’Compiler ConfigurationAdditional command line arguments

’--cs-on --cs-condition --cs-count --cs-partial-instrumentation’are appended

Linker ConfigurationAdditional command line arguments

’--cs-on --cs-condition --cs-count --cs-partial-instrumentation--cs-libgen=/MD’ are appended

...

Build the squishcoco_sample project. This will cause the executable squishcoco_sample.exe to be built and the codecoverage instrumentation file squishcoco_sample.exe.csmes to be generated. Double click on squishcoco_sample.exe.csmes to inspect this file in CoverageBrowser.

Right now there is no code coverage statistics visible in CoverageBrowser: This is because the application has not yet beenexecuted. Click on squishcoco_sample.cpp in the source list to display the main function. All the instrumented lines areshown grayed out, to indicate that nothing has been executed.

Now execute squishcoco_sample.exe by double clicking it. This will result in a file called squishcoco_sample.exe.csexe being generated. The file contains a code coverage snapshot which can be imported into CoverageBrowser:

1. Click "FileÕLoad Execution Report...".

2. Select the "File" item and enter the path of the squishcoco_sample.exe.csexe file.

3. Click on the "Import" button.

This will cause the code coverage statistics to be updated. Now, in the source code window, the main function’s return statementwill be colored green to indicate that this line has been executed.


C# on Microsoft Visual Studio

4.3 C# on Microsoft® Visual Studio®

Start Microsoft® Visual Studio® and create a new C# application:

1. Click on "FileÕNewÕProject..." to pop up the new project wizard.

2. Choose a project type of "Visual C#ÕWindows" and the "Console Application" template.

3. Enter a project name of squishcoco_sample, then click the "OK" button.

4. When the wizard’s second page appears, click the "Finish" button.

To activate the instrumentation, use the project properties:

1. In the "Solution Explorer", double click "squishcoco_sampleÕProperties" to open the property dialog.

2. Click on the "Build" tab.

3. Select "Conditional compilation symbols" and enter COVERAGESCANNER_COVERAGE_ON.

The code coverage analysis is enabled when the symbol COVERAGESCANNER_COVERAGE_ON is defined from the command line.

Build the squish_coco project. This will cause the executable squishcoco_sample.exe to be built and the code coverageinstrumentation file squishcoco_sample.exe.csmes to be generated. Double click on squishcoco_sample.exe.csmes toinspect this file in CoverageBrowser.

Right now there are no code coverage statistics to be seen in CoverageBrowser: this is because the application has not yet beenexecuted. Click on Program.cs in the source list to display the main function. All the instrumented lines are shown grayed outto indicate that nothing has been executed.

Now execute squishcoco_sample.exe by double clicking it. This will result in a file called squishcoco_sample.exe.csexe being generated. This file contains a code coverage snapshot which can be imported into CoverageBrowser:

1. Click "FileÕLoad Execution Report...".

2. Select the "File" item and enter the path of the squishcoco_sample.exe.csexe file.

3. Click on the "Import" button.

This will cause the code coverage statistics to be updated. Now, in the source code window, the main function’s return statementwill be colored green to indicate that this line has been executed.

4.4 Tcl

i At the moment, this example only works with Tcl v8.5 or earlier. See Section 4.4.1, page 10 below for details.

Create a file hello.tcl with the following content:

if { $argc == 0 } {puts "Hello World!"

} else {for {set i 0} {$i<$argc} {incr i} {


Tcl

if { $i == 0 } {set hello "Hello [lindex $argv $i]"

} elseif { $i == $argc - 1 } {set hello "$hello and [lindex $argv $i]"

} else {set hello "$hello, [lindex $argv $i]"

}}puts $hello

}

Execute hello.tcl using the tclsh interpreter:

$ coveragescannertcl tclsh hello.tcl Jim John James

Two files are generated:

tclsh.csmes:The instrumentation database of hello.tcl

tclsh.csexe:The execution report.

To import the execution report into its instrumentation database, execute:

$ cmcsexeimport -m tclsh.csmes --title="Jim John James" tclsh.csexe

After the import, it is possible to analyze the code coverage data directly with CoverageBrowser or generate a HTML report:

$ cmreport --title="Hello application" -m tclsh.csmes --html=tclsh.html

4.4.1 Using more than one Tcl version on one system

Squish Coco v3.0 does not yet support the use of tclsh under Tcl v8.6. If you want to use coveragescannertcl with tclsh,Tcl v8.5 or earlier is necessary.

In some UNIX®systems (like Debian Linux™), more than one Tcl version can be present at the same time. The executables arethen distinguished by their names. On such a system, the tclsh executable of Tcl v8.5 may be called tclsh8.5.

We can then run CoverageScannerTcl such that it calls tclsh8.5. On its own, this would however change the names ofthe generated files into tclsh8.5.csmes and tclsh8.5.csexe. We can correct this with another commandline option,--cs-output=tclsh. The first command in the example above now becomes:

$ coveragescannertcl --cs-output=tclsh tclsh8.5 hello.tcl Jim John James

Afterwards, the rest of the example can be done as before.


Command Line Tools

4.5 Command Line Tools

Open a console window (MS-DOS Prompt or Command Prompt window on Windows) and make sure that the Microsoft®

Visual Studio® compiler (cl.exe) or GCC is installed on your system.

Create a simple hello.c source file that contains the following code:

#include <stdio.h>

int main(int argc, char *argv[]){

printf("hello\n");return 0;

}

Compile this program using the CoverageScanner compiler wrapper instead of the native compiler. This is done by prependingcs to the command line compiler’s name:

With Microsoft® Visual Studio® With GCC$ cscl.exe hello.c /Fehello.exe $ csgcc hello.c -o hello.exe

Then the executable hello.exe and the file hello.exe.csmes, which contains the code coverage instrumentation, will begenerated.

Execute hello.exe; this will cause the file hello.exe.csexe to be generated. It contains a code coverage snapshot whichcan be imported into hello.exe.csmes using cmcsexeimport:

$ cmcsexeimport --title="Hello execution" -m hello.exe.csmes -e hello.exe.csexe

Once imported, the hello.exe.csexe file is no longer needed and can be deleted. Now hello.exe.csmes contains a singleexecution record. We can generate an HTML report with cmreport:

$ cmreport --title="Hello application" -m hello.exe.csmes --html=hello.html

Here is a summary of the command line options that were used in this command:

--title="Hello application":Set a title for the report. It appears on all generated HTML pages.

-m hello.exe.csmes:Select the instrumentation database.

–html=hello.html:Specify the name of the HTML file that the report should be written to.


Generating Instrumentations Without Modifying Projects

Chapter

5Generating Instrumentations Without Modifying Projects

Squish Coco can also generate code coverage information for a project without the need to modify it. The principle is toprepend to the PATH variable the path of the CoverageScanner compiler wrappers and to set the instrumentation parame-ters with the COVERAGESCANNER_ARGS environment variable. To activate the instrumentation, --cs-on must be present inCOVERAGESCANNER_ARGS. If this is not the case, CoverageScanner is completely deactivated.

5.1 GNU Make

Proceed as follows to instrument a project which can be generated using GNU Make:

export PATH=/opt/SquishCoco/wrapper/bin:$PATHexport COVERAGESCANNER_ARGS=--cs-onmake cleanmake

5.2 Microsoft® NMake

Proceed as follows to instrument a project which can be generated using NMake:

set PATH=%SQUISHCOCO%\visualstudio;%PATH%set COVERAGESCANNER_ARGS=--cs-onnmake cleannmake


Microsoft Visual Studio

5.3 Microsoft® Visual Studio®

Proceed as follows to instrument a project which can be generated using Microsoft® Visual Studio®:

set PATH=%SQUISHCOCO%\visualstudio;%PATH%set COVERAGESCANNER_ARGS=--cs-ondevenv /useenv myproject.sln /Rebuild

5.4 Microsoft® MSBuild

Proceed as follows to instrument a project which can be generated using Microsoft® MSBuild

set PATH=%SQUISHCOCO%\visualstudio;%PATH%set COVERAGESCANNER_ARGS=--cs-onmsbuild /p:UseEnv=true myproject.sln /t:ReBuild

5.5 Mono C# XBuild

Proceed as follows to instrument a project which can be generated using Mono C# XBuild

export COVERAGESCANNER_ARGS=--cs-onmsbuild \

/p:UseEnv=true \/p:UseHostCompilerIfAvailable=true \/p:CscToolPath="${SQUISHCOCO}/wrapper" \myproject.sln /t:ReBuild

The environment variable SQUISHCOCO points to the location of the CoverageScanner executable.


Instrumenting a simple project

Chapter

6Instrumenting a simple project

In the following paragraphs we will show a simple project with unit tests and show how it can be instrumented. The project is asimple expression parser, and it has not many requirements besides a C++ compiler.

The project replicates in miniature an existing project that has been extend with unit tests and for which we will now useSquish Coco to find out how good the test coverage is. Instrumentation should therefore be non-intrusive and should not changethe project very much.

i This project requires the UNIX® version of Squish Coco.

6.1 Prerequisites and installation

In this description we assume that Squish Coco has been installed at its default location, in the directory /opt/Squishcoco/.If that is not the case, replace in the following examples /opt/SquishCoco with the actual installation path.

The project uses CppUnit as the framework for unit tests. Install CppUnit from the website or as part of your distribution.The application can be found in the directory /opt/Squishcoco/samples/parser. Copy the content of the directory to yourworkspace.

$ cp -a /opt/Squishcoco/doc/parser .$ cd parser

6.2 Structure of the parser directory

In the rest of the example we will use parser as our working directory.

It contains C++ source files and header files, together with an unit test file, unittest.cpp. The Makefile has been preparedfor unit tests, but not for instrumentation. The instrumentation is done with the help of the bash script instrumented.


Compiling and testing

6.3 Compiling and testing

Run make to compile the program. It is a simple expression parser and calculator.

$ ./parserEnter an expression an press Enter to calculate the result.Enter an empty expression to quit.

> 2 + 2Ans = 4

> PiAns = 3.14159

> sin(Pi)Ans = 1.22465e-16

> sinn(90)Error: Unknown function sinn (col 9)

> sin(90)Ans = 0.893997

> cos(pi)Ans = -1

>$

We have added some unit tests for the main class, Parser. Look into the file unittest.cpp to see the tests that have beenincluded. Execute it with make tests. You will see that 8 tests have been executed.

6.4 Instrumentation

We have kept the instrumentation separate from the main project. The core of the instrumentation is a short shell script,instrumented. It is a simple wrapper, and calling “instrumented <command>” executes <command> with a few environ-ment variables set. We will do this now. Enter1

$ make clean$ ./instrumented make tests

The first command removes all object files, since we need everything to be recompiled. The second command then compilesthe program with instrumentation and runs the tests. That’s all!

We now have a look at what the script has done and how it has done it. List the contents of your parser directory:

$ lsconstants.h functions.o.csmes parser.cpp unittests.oerror.cpp instrumented parser.h unittests.o.csmeserror.h LICENSE parser.o variablelist.cpperror.o main.cpp parser.o.csmes variablelist.herror.o.csmes main.o unittests variablelist.ofunctions.cpp main.o.csmes unittests.cpp variablelist.o.csmesfunctions.h Makefile unittests.csexefunctions.o NOTICE unittests.csmes

1If you have installed Squish Coco at a location differing from the default, you first have to edit the script and change the value of the shell variableCOCO_DIR.


How the project is instrumented

You see two kinds of files that do not appear as result of normal compilation. The .csmes file contain the information that isneeded for coverage measurement, and the .csexe files contain the results of code execution. The files that end in .o.csmesare temporary files and are only used during compilation.

This time, the only program that was actually executed was unittests, and therefore the only .csexe file is unittests.csexe.To see the coverage results, you can therefore start the CoverageBrowser with the command

$ coveragebrowser -m unittests.csmes -e unittests.csexe

Then the CoverageBrowser will start with a modal window, "Load Execution Result". Click on the "Import" button toload the data.

!By default, CoverageBrowser automatically deletes the .csmes file after it loads it. You can switch this behavioroff by unselecting the "Delete after loading" checkbox. Or, if you don’t, select the "FileÕSave" menu itemto save the execution report in the unittest.csmes file.

For the use of CoverageBrowser, see Part III, page 45. We will now rather describe how the instrumentation is done.

6.5 How the project is instrumented

The file instrumented is a short bash script:

#! /bin/bash

COCO_DIR=/opt/SquishCoco

export PATH=$COCO_DIR/wrapper/bin:$PATHexport COVERAGESCANNER_ARGS=’--cs-on’

"$@"

The variable COCO_DIR just contains the name of the directory in where Squish Coco is installed. Then there are twoexport statements, and the final cryptic statement executes the command line parameters of instrumented. So if you call./instrumented make tests, the command make tests is executed by the script, but in a different environment thannormally.

The important part of it are therefore the two export statements. In the first one, the search part is manipulated so that theprograms in /opt/sqishcoco/wrapper/bin are searched first. This directory contains a lot of files with the same names asthe compilers2 that are supported by Squish Coco:

$ ls /opt/SquishCoco/wrapper/binar g++-4.9 x86_64-linux-gnu-arc89-gcc gcc x86_64-linux-gnu-g++c99-gcc gcc-4.6 x86_64-linux-gnu-g++-4.6...

These programs are the compiler wrappers. With the new PATH, they are executed instead of the real compilers. The compilerwrappers is actually symbolic links to a single program, coveragescanner (see Part IV, page 71). When executed to compile asource file, it creates an instrumented version of the source and then runs the original compiler to compile it.

2There is also ar, which is not a compiler but takes part in the compilation process.


Additional Changes

In the second export statement, additional flags for the compiler wrappers (see Chapter 17, page 74) are set. Here we set onlyone option, --cs-on. If it is not present, the compiler wrappers are inactive and just call the compilers they represent.

The resulting script should work without changes for many simple projects. If more customization is needed, it can often beachieved by adding more options to COVERAGESCANNER_ARGS.

6.6 Additional Changes

It is also convenient to add make targets to handle the files generated by CoverageScanner. In the parser directory, theMakefile has been changed in the following way:

clean: testclean...-$(DEL_FILE) *.o.csmes # (added)

distclean: clean...-$(DEL_FILE) *.csmes *.csexe # (added)

Since the .o.csmes files are needed only for compilation, they can be deleted whenever the .o files are deleted (which is thatwhat make clean does). The .csmes and .csexe files are more precious and should only be deleted when all generated filesare removed. Therefore we have added their deletion statements to the distclean target.


Getting started with Qt

Chapter

7Getting started with Qt

The following example is more complex. We will take Qt’s TextEdit example and use it to illustrate how Squish Coco can beused at different stages of the development process. To cover the whole coding cycle, we will first show how an instrumentedapplication is created, perform manual tests and analyze their results. Then we will create an instrumented unit test.

In a second step (see Chapter 7.2, page 27), we will cover those aspects which are more interesting to product managers:analysing the impact of code changes (e.g., bug fixes)—in particular, tracking their test progress, externalizing testing, andcollecting the code coverage analysis of a complete testing team.

i The Qt framework can directly be downloaded from http://qt-project.org.

The modified TextEdit sample is available in the doc directory of the Squish Coco installation. Copy the directory doc/textedit to your working directory.

7.1 Compiling the example application

In this section, we will work with the files in the directory textedit/textedit_v1.

We want to be able to build our application both normally and with generated test coverage instrumentation code, withouthaving to change our source code. This can be achieved by making a small change to the application’s project (.pro) file. Wecan then use a command line option for qmake to generate an instrumented build instead of a normal one.

To make the project file suitable both for normal and for instrumented builds, we create a set of definitions that can be activatedby a command line switch; in qmake’s terminology this is called a scope. The following listing (see Figure 7.1, page 19) showsa minimal scope for code instrumentation.

The following must be done:

• We must ensure that precompiled headers are disabled when the code is instrumented. qmake allows to do this by settingthe PRECOMPILED_HEADER variable to an empty value.

• It is also necessary to increase the value of QMAKE_LINK_OBJECT_MAX in order to disable the usage of the linker script.We set it here to 10000.


http://qt-project.org

Compiling the example application

• Finally, qmake must be instructed to use CoverageScanner’s wrappers for compilation. This is done by prefixing thenames of the compilation tools with cs.

CodeCoverage {PRECOMPILED_HEADERS=QMAKE_LINK_OBJECT_MAX=10000

QMAKE_CC=cs$$QMAKE_CCQMAKE_CXX=cs$$QMAKE_CXXQMAKE_LINK=cs$$QMAKE_LINKQMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIBQMAKE_AR=cs$$QMAKE_AR}

Figure 7.1: Minimal qmake configuration

These modification are sufficient for most standard C and C++ applications. For Qt applications we must use additional settingsin order to ensure that CoverageScanner does not instrument the source code that is generated by Qt’s tools (e.g., by uic, qrcand the moc).

To exclude qrc resource files from instrumentation, we must tell CoverageScanner not to instrument any file with a namethat begins with qrc_. This can be done with the command line option --cs-exclude-file-regex=qrc_.*. Since we don’twant to have to enter this option manually, we will put it in the .pro file. Similarly, to let CoverageScanner ignore the filesgenerated by uic we can use the same command line option, only this time with a different file matching regular expression:--cs-exclude-file-regex=ui_.*.

Squish Coco also provides a command line option that is specific to applications built using the Qt 4 toolkit: (--cs-qt4). Thisoption ensures that CoverageScanner:

• does not instrument the Q_OBJECT and Q_DECLARE_PLUGIN macros.

• does not instrument code generated by the moc, except that signal emissions and slot receives are instrumented, sincethey are vital to a Qt program’s logic.

We can also exercise some control over the level of instrumentation and what information is reported. For example, we canswitch on the counting of code executions with the --cs-count command line option, or we can enable full instrumentation atdecision/condition level with the --cs-full-instrumentation option. With the --cs-output option we can specify the filethe execution report is written to when the application terminates. (By default the output is written to the file <appname>.csexe,where <appname> is the name of the program that has been executed.)

So, for a Qt 4-based application, the final Squish Coco scope in the application’s .pro file will look something like this.





CodeCoverage {COVERAGE_OPTIONS = --cs-count --cs-full-instrumentationCOVERAGE_OPTIONS += --cs-qt4COVERAGE_OPTIONS += --cs-output=textedit.exeCOVERAGE_OPTIONS += --cs-exclude-file-regex=qrc_.*

QMAKE_CFLAGS += $$COVERAGE_OPTIONSQMAKE_CXXFLAGS += $$COVERAGE_OPTIONSQMAKE_LFLAGS += $$COVERAGE_OPTIONS

QMAKE_CC=cs$$QMAKE_CCQMAKE_CXX=cs$$QMAKE_CXXQMAKE_LINK=cs$$QMAKE_LINKQMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIBQMAKE_AR=cs$$QMAKE_AR}

Figure 7.2: Final qmake configuration

In this form, the scope has been added to the project file textedit_v1/textedit_v1.pro. When we now run qmake withoutoptions, a Makefile for a normal build is generated; but we can also build an instrumented version of the program in thefollowing way:

Linux™/Apple® Mac OS X Microsoft® Windows

qmake CONFIG+=CodeCoveragemake

qmake CONFIG+=CodeCoveragenmake

In either case we still end up with a textedit.exe executable, but with an instrumented build we will also get an additionalfile, textedit.exe.csmes. It contains the instrumentation database.

7.1.1 The First Code Coverage Results

For our very first exercise we will simply execute TextEdit and then quit the application straight away. This will cause a filecalled textedit.exe.csexe to be generated. The file is in a binary format, so it is not human readable. It is an instrumentationdatabase that contains a snapshot of the most recent execution that we have just done.

To see the results we must run the CoverageBrowser tool and load the textedit.exe.csmes instrumentation database (Menu:"FileÕOpen..."). After the file has been opened, no coverage information is available because no execution reports havebeen imported. The instrumented code lines are shown grayed-out and no coverage statistics has been computed (see Figure 7.3,page 21).



Figure 7.3: CoverageBrowser after loading the TextEdit’s instrumentation database

In order to see an execution report, click "FileÕLoad Execution Report...", which invokes the import dialog. As aminimum you should enter the filename (including the full path) of the textedit.exe.csexe file (field: ‘File Name’), andgive the test a name (Field: ‘Name’), e.g. “Start and Quit”. Switch the ‘Delete after loading’ option on because the report is nolonger needed after our import. It is also helpful to set the ‘When file becomes modified’ option to “Open this dialog”, becausethen the file import dialog is automatically opened after each run and the new .csexe file that it created can be added to thedatabase.

After the import has finished, the code coverage information is visible:

• Coverage statistics for the functions and methods of all source files of the application is shown.

• The source window is now colored and shows executed code on a green background and unexecuted code on a redbackground.

• The execution list now contains one selected item called “Start and Quit”, which is the only test execution report we haveso far created.

7.1.2 Interactive testing

CoverageBrowser correctly reveals, for example, that the TextEdit::fileSave() function is not executed. We will try tovalidate this function interactively, guided by the code coverage analysis.

In the source window, all unexecuted source code lines are shown with a red background. (see Listing 7.4, page 22)



bool TextEdit::fileSave(){

if (fileName.isEmpty()){QMessageBox::warning(this, tr("No file name specified"),tr("Save first your document using ’Save As...’ from the menu"),QMessageBox::Ok);

return false;}

QTextDocumentWriter writer(fileName);bool success = writer.write(textEdit->document());if (success)

textEdit->document()->setModified(false);return success;

}

Figure 7.4: CoverageBrowser source view of the function TextEdit::fileSave()

To test this function we must perform the following steps:

1. Start the TextEdit application.

2. Click on the ‘Save’ button: TextEdit should display the error message "Save first your document using ‘Save As. . . ’from the menu".

3. Quit the application.

After these steps have been done and the coverage report imported, CoverageBrowser shows that the return false; linejust after the call to QMessageBox::warning() has been executed (as indicated by the green background). However, the lineif (fileName.isEmpty()) is shown as partially executed (indicated by an orange background); (see Listing 7.5, page 22).


if (fileName.isEmpty()){QMessageBox::warning(this, tr("No file name specified"),tr("Save first your document using ’Save As...’ from the menu"),QMessageBox::Ok);

return false;}



}

Figure 7.5: CoverageBrowser source view after clicking TextEdit’s ‘Save’ button.

The explanation window (see Listing 7.6, page 23) tells us that the value of the expression fileName.isEmpty() was trueduring one execution but was never false (hence, it is considered only partially executed). In order to fully test this expressionwe must click on the ‘Save As. . . ’ button, then choose a filename, and finally click on the ‘Save’ button.



partially executed: fileName.isEmpty()

TRUE FALSE

yesExecution Count: 1Executed by:- Save Clicked

noExecution Count: 0

Figure 7.6: CoverageBrowser explanation window after clicking on the ’Save’ button of TextEdit.

After rerunning the application and doing a “Save as”, the new execution report now has only one source code line that ispartially untested (see Listing 7.7, page 23). In this case, CoverageBrowser reveals that the Boolean variable success wasnever false, which means that saving the document has never failed.

We could force a write failure, and this would ensure that we had 100% code coverage for this function. But we will use adifferent test strategy to get complete code coverage: we will use a unit test and import the execution result into the TextEditinstrumentation database.


if (fileName.isEmpty()){QMessageBox::warning(this,tr("No file name specified"),tr("Save first your document using ’Save As...’ from the menu"),QMessageBox::Ok );

return false;}



}

Figure 7.7: CoverageBrowser source view after clicking TextEdit’s ‘Save As. . . ’ button and then the ‘Save’ button.

7.1.3 Writing unit tests

The unit test infrastructure can be found in the directory textedit_v1_tests/. It contains just one test, which sets an illegalfilename and then tries to execute the TextEdit’s fileSave() function. To do this we use the QTestLib unit testing librarythat is supplied with Qt. The test is contained in the file textedit_v1_tests/tst_textedit.cpp (see Listing 7.8, page 24).



#include "tst_textedit.h"

void TestTextEdit::tst_saveFile() {TextEdit textEdit;textEdit.fileName="/";QVERIFY( !textEdit.fileSave() );

}

QTEST_MAIN(TestTextEdit);

Figure 7.8: Unit test for the TextEdit application, tst_textedit.cpp

To import this test’s instrumentation result into TextEdit’s instrumentation database, the following infrastructure is necessary:

1. A qmake project file with code coverage configured identically with that of the TextEdit project.

2. A post-build rule which automatically executes the test and collects the coverage information.

3. A unit test listener which saves the code coverage data (and the test status—passed or failed) for every executed unit intothe unit test’s own instrumentation database.

4. A way to import the code coverage report into the TextEdit’s instrumentation database.

The unit test will recompile along with textedit_v1/textedit.cpp. To make its results importable into the TextEditinstrumentation database, it is necessary that both executables (TextEdit and the unit test) are instrumented in exactly thesame way. So, for this example, we must use the instrumentation options --cs-count, --cs-full-instrumentation and--cs-qt4.

Unfortunately, these command line options alone are insufficient, because Squish Coco’s default behavior is only to instrumentheader and source files in the current directory. But here we need to instrument the TextEdit application’s sources inaddition to the unit test, so we must use another command line option to specify an additional path for files to instrument:--cs-include-path.

As before, we don’t want to have to remember these command line arguments every time, so we set them in the qmake projectfile, textedit_v1_tests.pro (see Figure 7.9, page 25). With these lines in the unit test’s project file, the result will be that theqmake-generated Makefile creates the tst_textedit.exe executable which, when run, produces the tst_textedit.exe.csexe execution report. We then can use the CoverageBrowser to import this report into the file tst_textedit.exe.csmes.



HEADERS = ../textedit_v1/textedit.h tst_textedit.hSOURCES = ../textedit_v1/textedit.cpp tst_textedit.cpp

CodeCoverage {COVERAGE_OPTIONS = --cs-count --cs-full-instrumentationCOVERAGE_OPTIONS += --cs-qt4COVERAGE_OPTIONS += --cs-output=tst_textedit.exeCOVERAGE_OPTIONS += --cs-include-path=../textedit_v1COVERAGE_OPTIONS += --cs-exclude-file-regex=qrc_.*

QMAKE_CXXFLAGS += $$COVERAGE_OPTIONSQMAKE_CCFLAGS += $$COVERAGE_OPTIONSQMAKE_LFLAGS += $$COVERAGE_OPTIONS

QMAKE_CC=cs$$QMAKE_CCQMAKE_LINK=cs$$QMAKE_LINKQMAKE_CXX=cs$$QMAKE_CXX}

Figure 7.9: An extract from the unit test qmake project file

We can also execute the unit test automatically and import the execution report with a post-build rule. Squish Coco provides anextra command line tool, cmcsexeimport (see Chapter 26, page 154), which imports an execution report into an instrumentationdatabase. The post-build rule first deletes any previous execution report, then executes the test itself, and finally imports theresults into the application’s execution database, tst_textedit.exe.csexe (see Listing 7.10, page 25).

CodeCoverage {win32: MAINDIR=$$replace(PWD,"/","\\")!win32:MAINDIR=$$PWD

unix {QMAKE_POST_LINK = rm $$MAINDIR/tst_textedit_v1.exe.csexe ;QMAKE_POST_LINK += $$MAINDIR/tst_textedit_v1.exe ;QMAKE_POST_LINK += cmcsexeimport -m $$MAINDIR/tst_textedit_v1.exe.csmes \

-e $$MAINDIR/tst_textedit_v1.exe.csexe -t UnitTest}win32 {QMAKE_POST_LINK = del /F $$MAINDIR\\tst_textedit_v1.csexe &QMAKE_POST_LINK += $$MAINDIR\\tst_textedit_v1.exe &QMAKE_POST_LINK += cmcsexeimport -m $$MAINDIR\\tst_textedit_v1.exe.csmes \

-e $$MAINDIR\\tst_textedit_v1.csexe -t UnitTest}}

Figure 7.10: Post-build rules for the import of the execution report into the unit test’s instrumentation database

By default, the coverage data is imported without any information about the executed tests: instead, an execution report called‘UnitTest’ is created, which does not describe which test was executed or if its execution was successful. To provide the missinginformation, we must use the CoverageScanner API and generate an execution report for each test that is executed. An examplethat shows how this is done is available in the chapter 22.3.2. In the example, the API is used in the following way:

• Two Squish Coco source files are added to the qmake project file (see Listing 7.11, page 26).



• The unit test class, TestTextEdit, inherits from TestCoverageObject instead of directly from QObject (see List-ing 7.12, page 26).

HEADERS += testcoverageobject.hSOURCES += testcoverageobject.cpp

Figure 7.11: Including the CoverageScanner listener in textedit_v1_tests.pro

#include "testcoverageobject.h"#include "../textedit_v1/textedit.h"#include <QtTest/QtTest>

class TestTextEdit : public TestCoverageObject{

Q_OBJECTprivate slots:void tst_saveFile();

};

Figure 7.12: The TextEdit unit test header file, tst_textedit.h

The testcoverageobject.cpp file’s source code (see Listing 7.13, page 26) is simple to understand. The file addsa single cleanup() function to QTestLib, which is executed after each unit test item. The code between #ifdef__COVERAGESCANNER__ and #endif is only compiled when CoverageScanner is invoked1. This extra code creates a test nameby combining the test class’s object name and the test function’s name. In the the example, an execution item called unittest/TestTextEdit/tst_saveFile is generated. The slash is used to support the organization of tests in a tree view. The currenttest status (“PASSED” or “FAILED”) is also recorded and added to the execution report by the __coveragescanner_save()function.

...void TestCoverageObject::cleanup(){cleanupTest();

#ifdef __COVERAGESCANNER__

QString test_name="unittest/";test_name += metaObject()->className();test_name += "/";test_name += QTest::currentTestFunction();__coveragescanner_testname(test_name.toLatin1());if (QTest::currentTestFailed())__coveragescanner_teststate("FAILED");

else__coveragescanner_teststate("PASSED") ;

__coveragescanner_save();#endif}

Figure 7.13: An extract from the TestCoverageObject’s source code

At this stage we could start CoverageBrowser, load the TextEdit instrumentation database, and import the unit test’s1The symbol __COVERAGESCANNER__ is defined automatically by CoverageScanner and so does not need to be defined manually.


Working with code coverage data

instrumentation database by clicking "FileÕImport Unit Tests...". An even more convenient alternative is to use thecmmerge tool to automate this step. The cmmerge program is designed to import one instrumentation database’s executionreport into another instrumentation database. This means that we can extend our post-build rules to use the cmmerge programto import the coverage information automatically from the unit test into the TextEdit program’s instrumentation database (seeListing 7.14, page 27).

CodeCoverage {# Merge coverage database into TextEdit databaseunix {QMAKE_POST_LINK += ;QMAKE_POST_LINK += cmmerge -o $$MAINDIR/../textedit_v1/textedit.tmp \

-i $$MAINDIR/../textedit_v1/textedit.exe.csmes \$$MAINDIR/./tst_textedit_v1.exe.csmes &&

QMAKE_POST_LINK += rm $$MAINDIR/../textedit_v1/textedit.exe.csmes &&QMAKE_POST_LINK += mv $$MAINDIR/../textedit_v1/textedit.tmp \

$$MAINDIR/../textedit_v1/textedit.exe.csmes}win32 {QMAKE_POST_LINK += &QMAKE_POST_LINK += echo Merging unit test result into the main application &QMAKE_POST_LINK += cmmerge -o $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes \

-i $$MAINDIR\\..\\textedit_v1\\textedit.exe.csmes \$$MAINDIR\\tst_textedit_v1.exe.csmes &

QMAKE_POST_LINK += COPY /Y $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes \$$MAINDIR\\..\\textedit_v1\\textedit.exe.csmes &

QMAKE_POST_LINK += DEL /F $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes}}

Figure 7.14: Post build rules: merging instrumentation results into the TextEdit instrumentation database

With all these changes to the .pro file in place, we can once again build and run the unit test. Now the CoverageBrowser showsthe fileSave() function to be 100% covered, with the execution list containing our three original manual tests and the singleunit test (see Figure 7.15, page 27).

Figure 7.15: The execution list after all the tests have been executed

7.2 Working with code coverage data

The most common ways in which code coverage is used are for developers to use it to find untested code, and for managers touse it to produce test status reports (e.g., as diagrams).



In addition to fully supporting the common use cases, Squish Coco also provides additional features which make it possibleto go beyond these fundamentals and extend what can be achieved with code coverage. This will be discussed in the currentsubsection.

7.2.1 Post mortem analysis

Recording each test’s coverage data makes it possible to compare their data to answer the question, “What does this test coverthat the others don’t?” This is particularly useful if just one test fails, since it can help us to identify which part of the code isinvolved.

To see how this works in practice, let us return to the TextEdit example. If we click ‘Save’, we will get an error message thatno filename is defined. This is not very convenient for users—we should have designed TextEdit to handle this particular caseby opening the ‘Save As. . . ’ dialog rather than by producing an error.

To identify where in the code this problem arises, we simply compare the ‘Save Clicked’ execution with all other executionsthat involve the ‘Save’ button. To do this we must first switch to the “Execution Comparison Analysis” mode (Menu:"ToolsÕExecution Comparison Analysis"). Select the checkboxes in the “Reference” column for the “tst_saveFile” and“SaveAs clicked before Save clicked” tests. This will make the execution comparison symbol appears in front of the affectednames (see Figure 7.16, page 28). In the “Executions” column, click on “Save clicked” checkbox.

Figure 7.16: The execution list being used to compare different executions

In this mode, the coverage analysis is based only on source code lines which are not executed by “tst_saveFile” and “SaveAsclicked before Save clicked”. This is why the overall coverage decreases to 1.29%: It means that “Save clicked” executes 1.29%more code than the selected tests.

Using cmreport it is possible to generate a HTML report which displays the same information:

cmreport --csmes=textedit_v1/textedit.exe.csmes \--html=textedit.html \--section=execution \--select-reference=".*tst_saveFile" \--select-reference="SaveAs clicked before Save clicked"

If we now look at the source code itself, we will see that only two lines of the TextEdit::fileSave() function (seeListing 7.17, page 29) are not grayed: the lines which pop up the error message. These are the lines that must be modified tochange the “Save” button’s behavior.




if (fileName.isEmpty()){QMessageBox::warning(this,tr("No file name specified"),tr("Save first your document using ’Save As...’ from the menu"),QMessageBox::Ok );

return false;}



}

Figure 7.17: CoverageBrowser source view of the comparison of the “Save clicked” execution.

In this case, changing the fileSave() function is easy—we simply replace the QMessageBox::warning() call with a call tothe fileSaveAs() method. (see Listing 7.18, page 29)


if (fileName.isEmpty())return fileSaveAs();



}

Figure 7.18: The TextEdit’s improved fileSave() function.

7.2.2 Evaluating the impact of a hot fix

Before committing a change or starting to test a hot fix, it is possible to estimate the impact of the code modification.CoverageBrowser is able to perform an analysis on the difference between two source sets and can list the tests that will beaffected (and those which won’t).

Start CoverageBrowser and load the modified TextEdit example’s instrumentation database. Now click on "ToolsÕComparewith..." and select the original version of the TextEdit instrumentation database. CoverageBrowser will now display thesource code like a text comparison application does (e.g., diff).

Click on "ToolsÕAnalysis of Modified Methods" to exclude all unmodified functions from the coverage analysis. In theTextEdit case, doing this will mean that only one function, TextEdit::fileSave(), will be treated as being instrumentedsince that is the only method we have changed (see Figure 7.19, page 30). This also affects the statistic calculations sinceexecution coverage statistics will now be limited to just this function. The test executions whose coverage statistic is not zeroare the ones that are affected by the code modifications we have made.



In our case we have:

• “Save clicked”,

• “SaveAs clicked before Save clicked” and

• “tst_saveFile” (our unit test).

The “Start and Exit” case has a coverage of 0% and so does not execute our modified code. It is for this reason no longer visiblein the execution list. All entries in the ‘Execution’ column are struck through to inform us that these tests are not executed inthe newest version and only present in the reference database.

In other words, only the two manual tests and the unit test listed above must be re-executed to ensure that no regressions havebeen introduced by our code changes.

Figure 7.19: List of tests affected by a code modification

7.2.3 Black-box testing/distributed testing

Up to now we have done white-box testing, i.e. testing where we have access to the source code and which makes use ofour knowledge of the code. It is also possible to use Squish Coco for black-box testing. In other words, we can still do codecoverage analysis without having access to or even knowledge of the source code. If we use this approach, the generatedinstrumentation database will, of course, contain no source code.

To use black-box testing we must create a suitable instrumentation database by clicking "FileÕGenerate Black-BoxConfiguration...". This database, along with the TextEdit executable, can be given to the test team which can then usethem with a simplified version of CoverageBrowser (see Figure 7.20, page 31). This version of CoverageBrowser only supportsthe importing and managing of execution reports since it does not have access to the application’s source code.

Once all the tests are finished, the black-box database can be merged into the original TextEdit instrumentation database usingCoverageBrowser’s merge facilities (Menu: "FileÕMerge with...").



Figure 7.20: Black-box testing results as shown by CoverageBrowser

7.2.4 Verifying if a bug fix is correctly tested

Often, when a small bug fix is made, the effects are very localized and leave most of the source code unchanged. In view ofthis, it is often unnecessary to retest the entire application with the whole test suite.

Squish Coco makes it possible to avoid unnecessary testing. We can tell it to restrict itself to the source code that has changedbetween the original and fixed version of the application (see Chapter 7.2.2, page 29). This allows us to focus purely on theanalysis of the fix. To achieve this, simply load the fixed application’s freshly generated instrumentation database (e.g., for themodified TextEdit application), and compare it with the earlier database for the unfixed version, using Squish Coco’s facilityfor analyzing modified functions.

Figure 7.21: Coverage of the patched function

We have done just such a comparison and the results are shown in in 7.21: The two tests, “Save clicked” and “Start andExit”, cover 85% of the TextEdit::fileSave() function, the only method that was modified for our fix. From this we know


Conclusion

exactly what additional testing is necessary to achieve 100% code coverage for our tests for the fixed version of the application.CoverageBrowser continues to display the list of missing tests (which are only executed using the first version of TextEdit) instrikeout style. This gives a hint of what remains for testing effort.

Using cmreport it is possible to generate a HTML report which displays the same information:

cmreport --csmes=textedit_v2/textedit.exe.csmes \--csmes-reference=textedit_v1/textedit.exe.csmes \--html=textedit.html \--section=execution

7.3 Conclusion

Squish Coco provides code coverage analysis which can be applied to all the usual testing techniques: unit, manual, andblack-box testing. Squish Coco can easily be told to ignore the generated code produced by the Qt library’s tools (moc, qrc,and uic), so that only the code written by developers is instrumented. Test results can be collected into a database and can beused to evaluate how much code coverage our tests achieve and to show which statements are not currently tested. With thisinformation we can target our testing efforts towards 100% test code coverage. In addition, Squish Coco makes it possible tosee what effect a code modification would have in terms of test code coverage without having to test the entire application.

Overall, Squish Coco can help us target our tests to ensure that our applications have as much test coverage as possible, whileavoiding or minimizing test coverage duplication. Furthermore, Squish Coco can help us see what effects changes to our codehave on test coverage, so that we can adapt our test suites accordingly.



Part II Code CoverageOverview


Code Coverage Overview

Chapter

8Overview of Squish Coco’s Code Coverage Analysis

CoverageScanner is a C++ program that—in effect—replaces the usual compiler. It preprocesses the source code using thenative preprocessor program, in the course of which it inserts instrumentation code, and at the end it compiles and links theproject file, just the same as for a normal build.

CoverageScanner’s instrumentation consists of:

1. declaring a static table that contains the of instrumentation results.

2. generating a library which produces an execution report upon program termination, a UNIX® signal or a Microsoft®

Windows event.

3. adding the instrumentation code itself for each relevant C and C++ statement or Boolean expression.

The compiler, linker and preprocessor used are from the native environments (gcc, Visual Studio, etc.). These tools are usedtransparently and the developer has only to prepend ’cs’ to the name of the compiler executable to activate the code coverage1.For example, using ’csgcc’ instruments the source code and uses the ’gcc’ compiler to generate objects.

Another code coverage technique involves adding measurement points according to debug information. Since CoverageScannerproduces instrumented code during the generation phase, it is possible to analyze the optimized code and achieve a morefine-grained analysis than some other tools achieve. (Many compilers produce unusable debug information on optimized code,but this doesn’t affect CoverageScanner.)

CoverageScanner’s principle of code coverage analysis is not to highlight executed source lines, but rather to place marks onexecution paths. This makes the analysis independent of the source code’s formatting and groups all sequential instructionstogether in a single measurement point. This also minimizes the extra memory and processing overhead that the instrumentedcode incurs.

8.1 Code instrumentation

This section describes how CoverageScanner instruments the code, with each aspect described in its own subsection.

1 For Microsoft® Visual Studio®, an additional wrapper is provided which directly uses the Microsoft® ’cl’ and ’link’ (see Chapter 20.9, page 105)


Code instrumentation

8.1.1 Detection

CoverageScanner parses all C++ language constructs and detects:

1. Executed functions.

2. Executed source code lines.

3. Execution paths.

4. Boolean expressions which produce different execution paths (if, for, while, switch, . . . ).

To put this in concrete terms we present the following very simple example function:

1 void foo()2 {3 bool found=false;4 for (int i=0; (i<100) && (!found); ++i) {5 if (i==50) break;6 if (i==20) found=true;7 if (i==30) found=true;8 }9 printf("foo\n");

10 }

Figure 8.1: Source code sample

CoverageScanner first analyses the sequential statements to record each execution path. It determines which instructions needto be instrumented and places a flag to record their execution. This kind of instrumentation is usually called Statement Coverage.To avoid performance problems, CoverageScanner groups all sequential instructions together and records the execution of thewhole group only once.

In the following listing, the instrumented statements are underlined:

1 void foo()2 {3 bool found=false; // [3] not instrumented4 for (int i=0; (i<100) && (!found); ++i) {5 if (i==50) break;6 if (i==20) found=true;7 if (i==30) found=true;8 }9 printf("foo\n"); // [9] not instrumented

10 } // [10]

Figure 8.2: Code coverage at branch level

Lines 3 and 9 are not monitored since lines 3, 9 and 10 are sequential operations. This has no impact on coverage because ifline 10 is executed, then lines 3 and 9 must have be executed before it.

CoverageScanner’s analysis of conditions determines which expressions are used in conditional statements (if, while, for,. . . ). These statements separate the execution path into two or more branches. Code coverage monitors execution by recordingif the condition was true or false during the application’s execution.

This kind of code coverage instrumentation is usually called Decision Coverage.



In the following listing, the conditions instrumented for Decision Coverage are underlined. The instrumented statements aredisplayed with a gray background:

1 void foo()2 {3 bool found=false;4 for (int i=0; (i<100) && (!found) ; ++i) {5 if (i==50 ) break;6 if (i==20 ) found=true;7 if (i==30 ) found=true;8 }9 printf("foo\n");

10 }

Figure 8.3: Code coverage at decision level

The analysis of Boolean expressions is used to determine which combinations are used in a conditional statements’ (if, while,for, . . . ) Boolean operations (and, or, . . . ). This makes it possible to have a more fine grained analysis why a particular codesegment was executed.

This kind of code coverage instrumentation is usually called Condition Coverage.

For example, the statement ‘return 0;’ of ‘if (a || b) return 0;’ can be executed if a or b is true. Analysis of theBoolean expressions will record whether variables a and b were true or false. This tells us if the statement ‘return 0;’ wasexecuted, which it will be if either or both of the two expressions was true during the test.

In the next example, the instrumented Boolean expressions are underlined:

1 void foo()2 {3 bool found=false;4 for (int i=0; (i<100) && (!found) ; ++i) {5 if (i==50 ) break;6 if (i==20 ) found=true;7 if (i==30 ) found=true;8 }9 printf("foo\n");

10 }

Figure 8.4: Code coverage at condition level

Squish Coco’s analysis is not limited to control statements (such as do. . .while or if. . .then constructs). For example,CoverageScanner also instruments the assignment of Boolean expressions. Constant or static expressions are not instrumentedsince their values are computed at compile time or once during program initialization.

In the following example, the instrumented Boolean expressions are underlined:



void foo(){static bool a = x && y;bool b = x && y;bool c = b;int d = b ? 0 : 1;

}

Figure 8.5: Code coverage of assignments

The statement static bool a = x && y is not instrumented because it is static. And the statement bool c = b is notinstrumented because the expression is a value (albeit a Boolean value), not a Boolean expression.

After the detection phase, CoverageScanner inserts the instrumentation statements into the code. Code insertion is described indetail in Section C.

8.1.2 Result of coverage analysis

The CoverageBrowser program features a graphical user interface which it uses to display the analyzed results of theinstrumentation. CoverageBrowser uses color coding to indicate the status of the statements. For this manual we have used thesame colors as the program, but in addition we have applied text effects to aid those who read a monochrome print out.

• bold green indicates statements which are executed.

• shaded bold red indicates statements which are not executed.

• italic orange indicates statements which are partially executed. These are Boolean expressions or conditionalstatements which were always true or always false.

The output for the example would be:

1 void foo()2 {3 bool found=false;4 for (int i=0; (i<100) && (!found); ++i)5 {6 if (i==50) break;7 if (i==20) found=true;8 if (i==30) found=true;9 }

10 printf("foo\n");11 }

Figure 8.6: Code coverage output

The expression i<100 is partially executed because is was always true. The expressions i==50 and i==30 are always false,yet they are shown as being fully executed. Since these expressions are incorporated in an if ... then statement, it is notnecessary to check whether they are true. (The non-execution of the break; and found=true; statements provide the sameinformation).


Performance

8.2 Performance

The insertion of instrumentation increases the code size and also affects the instrumented application’s performance—it willuse more memory and run slower.

For non-conditional expressions the instrumentation code involves a simple write to a fixed memory location. However, forconditional expressions more detailed analysis is needed and this is more computationally expensive.

Overall, an instrumented application will be from 60% to 90% larger and will run 10% to 30% slower.

i Detailed measurements are available in the Appendix (see Chapter D, page 197).

8.3 Statistics

Some developers write more lines of code than others simply by using a particular coding style—for example by puttingopening braces on a line of their own rather than on the same line as, say, an if statement. In view of this, Squish Cococomputes its statistics using a more subtle measure that isn’t susceptible to the the vagaries of coding style: its calculations arebased on the number of executed instrumented instructions compared with the total number of instrumented instructions.

Every instrumented simple instruction (return, break, the last instruction of a function, etc.) is recorded by one singleinstrumentation. A fully instrumented condition in an IF...THEN...ENDIF statement uses two instrumentations: one for thetrue case and one for the false case. If the code is only partially instrumented, only one condition is recorded (either the falsecase or the true case).

The statistics themselves depend of the type of instrumentation. It is not generally possible to compare code coverage at branchlevel with that at decision level: having 80% coverage at decision level does not tell us anything about the coverage at branchlevel, which could be bigger or smaller. The only thing we can be sure of is that reaching 100% coverage is more difficult withcondition coverage than with decision coverage or branch coverage.

In our example, the coverage is between 60% and 75%. The following table shows the details for each instrumentation.

Code coverage type Instrumentations Executed CoverageBranch (see Figure 8.7, page 39) 5 3 60%Decision partial (see Figure 8.9, page 39) 9 7 77%Decision full (see Figure 8.8, page 39) 13 9 69%Condition partial (see Figure 8.11, page 40) 12 9 75%Condition full (see Figure 8.10, page 40) 15 10 66%

In the examples below, the details of the calculations are displayed using subscripts. The first number shows how manyinstrumentations were executed; the second is the number of instrumentations in total.


Statistics

1 void foo()2 {3 bool found=false;4 for (int i=0; (i<100) && (!found); ++i)5 {6 if (i==50) break; 0/1[not executed]7 if (i==20) found=true;1/1[executed]8 if (i==30) found=true; 0/1[not executed]9 }1/1[executed]

10 printf("foo\n");11 }1/1[executed]

Figure 8.7: Code coverage statistics for full branch level instrumentation

1 void foo()2 {3 bool found=false;4 for (int i=0; (i<100) && (!found)2/2[was f alse and true]; ++i)5 {6 if (i==501/2[was f alse but not true]) break; 0/1[not executed]7 if (i==202/2[was f alse and true]) found=true;1/1[executed]8 if (i==301/2[was f alse but not true]) found=true; 0/1[not executed] }1/1[executed]9 printf("foo\n");

10 }1/1[executed]

Figure 8.8: Code coverage statistics for full decision level instrumentation

1 void foo()2 {3 bool found=false;4 for (int i=0; (i<100) && (!found)1/1[was f alse]; ++i)5 {6 if (i==501/1[was f alse]) break; 0/1[not executed]7 if (i==201/1[was f alse]) found=true;1/1[executed]8 if (i==301/1[was f alse]) found=true; 0/1[not executed]9 }1/1[executed]


Figure 8.9: Code coverage statistics for partial decision level instrumentation


Other Metrics

1 void foo()2 {3 bool found=false;4 for (int i=0; (i<100)1/2[was true but not f alse] && (!found)2/2[was f alse and true]; ++i)5 {6 if (i==501/2[was f alse but not true]) break; 0/1[not executed]7 if (i==202/2[was f alse and true]) found=true;1/1[executed]8 if (i==301/2[was f alse but not true]) found=true; 0/1[not executed]9 }1/1[executed]


Figure 8.10: Code coverage statistics for full condition level instrumentation

1 void foo()2 {3 bool found=false;4 for (int i=0; (i<100)1/2[was true but not f alse] && (!found)2/2[was f alse and true]; ++i)5 {6 if (i==501/1[was f alse]) break; 0/1[not executed]7 if (i==201/1[was f alse]) found=true;1/1[executed]8 if (i==301/1[was f alse]) found=true; 0/1[not executed]9 }1/1[executed]


Figure 8.11: Code coverage statistics for partial condition level instrumentation

8.4 Other Metrics

8.4.1 Function Coverage

Squish Coco computes the function coverage according the information provided by the code coverage at branch level. Thismetric consists of the number of times that a function is called.

8.4.2 Line Coverage

Squish Coco supports also the code coverage metric at line level. For this metric, all source code lines that are executed arerecorded. To compute it, it is necessary to compile the source code at least at the decision level with a full instrumentation.

The line coverage is a natural metric which permits easily to see which lines of code are executed. But it is more inaccuratethan the instrumentation at branch level and its results rely on the developer’s coding style.

The following example illustrates this problem:

int main()


Other Metrics

{if (true) return 1;foo();return 0;

}

Executing it would produce the following result:

int main(){if (true) return 1; // Executedfoo(); // Not executedreturn 0; // Not executed

}

This execution correspond to a coverage of 33%.

Since the first line of main contains two executed statements, splitting it in two would increases the number of executed lines,and so the test coverage. So we reformat the main function as follow:

int main(){if (true)return 1;

foo();return 0;

}

The execution produces a code coverage of 66%:

int main(){if (true) // Executedreturn 1; // Executed

foo(); // Not executedreturn 0; // Not executed

}

Another way to increase the coverage by reformatting is to hide an uncovered statement behind an executed one. To do it it isonly necessary to write the whole code of this main function in one line:

int main(){if (true) return 1; foo(); return 0;

}

The execution produces a line coverage of 100%:

int main(){if (true) return 1; foo(); return 0; // Executed

}

This small example illustrate how the result depends on how the source code is formated. Therefore, Squish Coco provides theline coverage as additional measurement to the decision and the condition coverage and so does not allow that a source code isonly instrumented at line level.


Testing Methodologies

Chapter

9Testing Methodologies

This chapter discusses the various testing strategies that can be carried out so as to make the best use of the Squish Coco tools.

9.1 Coverage Hits vs. Counts

If we want to set very stringent standards for an application to ensure that it is of the highest possible quality, executing justportions of its code, or executing code just once, is insufficient.

To address this issue the Squish Coco tools provide two possible approaches:

Code Coverage Count:We can require that each portion of instrumented code is executed a minimum number of times. This can be achievedby adding to the instrumented code so that it counts each execution.1 Naturally, setting such minimums, and the extrabookkeeping itself, can make the application consume more memory and run a bit slower. An important disadvantage ofthis approach is that loops in the source code end up having a high count with only a single execution, so counting thisway is not always useful.

Test Coverage Count:We can require that each portion of instrumented code is executed by a minimum number of test items. This avoidsthe code coverage count disadvantage regarding loops, since by setting a minimum number of test executions for eachportion of instrumented code, we give equal value to loops, recursions, and to instructions that are executed only once.

9.2 Testing Strategies

Squish Coco can be adapted to fit in with many different testing strategies, from unit tests during early development to completeproduct validation (e.g., acceptance tests). CoverageBrowser supports the merging of code coverage results at each stage oftesting and so provides a precise overview of the software’s overall quality.

1This can be done using CoverageScanner’s --cs-on command line option.


Strategies

9.2.1 Manual White Box Tests

White box testing (a.k.a. clear box testing, glass box testing or structural testing) uses an internal perspectiveof the system to design test cases based on internal structure. It requires programming skills to identify all pathsthrough the software. The tester chooses test case inputs to exercise paths through the code and determines theappropriate outputs.

Definition from Wikipedia

Interactive white box tests are easy to do using Squish Coco: once the application has been compiled with CoverageScanner,the test engineer can execute the it, and after each execution they can import the execution report into CoverageBrowserforanalysis.

9.2.2 Manual Black Box Tests

Black box testing takes an external perspective of the test object to derive test cases. These tests can befunctional or non-functional, though usually functional. The test designer selects valid and invalid input anddetermines the correct output. There is no knowledge of the test object’s internal structure.


Black box tests assume that test engineers have no knowledge of the application’s internals—they work purely in terms ofinputs and the corresponding expected outputs (whether of data or of error reports for invalid input).

For this kind of testing it is possible to follow the same pattern as for white box testing, but without providing the test engineerwith the instrumentation database (.csmes file) generated by CoverageScanner. The application itself will generate an executionreport which can be analyzed after the testing cycle. However, this approach has the disadvantage that test engineers cannotmanage their own execution reports (e.g., to add comments).

Squish Coco has a facility which makes black box testing possible without disadvantaging test engineers. The CoverageBrowsertool can generate an instrumentation database (.csmes file) which does not contain any source code or source browsinginformation. This allows test engineers to manage the list of tests and to view the statistics, but it does not provide any sourcelevel coverage information.

9.2.3 Unit Tests

Unit testing is a method by which individual units of source code are tested to determine if they are fit for use.A unit is the smallest testable part of an application. In procedural programming a unit could be an entire modulebut is more commonly an individual function or procedure. In object-oriented programming a unit is often anentire interface, such as a class, but could be an individual method.


CoverageScanner supports the saving of the code coverage data generated by each unit test using its library. (see Chapter 22.2.3,page 123)

As long as all objects are well compiled2, it is possible to merge the unit tests instrumentation database into those of the realapplication. And so see the code coverage of the unit tests on the main application.

2Well compiled means that a source file is not be compiled twice or at least it is not be compiled with different preprocessor options.


Strategies

9.2.4 Automatic Tests

Squish Coco allows us to annotate the execution report with the execution name and state. This does not require any specificlibrary—the information can be inserted directly by editing the execution report before and after the execution of a test item.This makes it possible to integrate the report into an automatic testing framework or for use with a test executing script. (seeChapter 22.1, page 120)


Part III CoverageBrowserReference Manual


Introduction

Chapter

10 Introduction

CoverageBrowser is a graphical user interface program which enables the user to analyze the test coverage.

CoverageBrowser is typically used in the following way:

1. Load an instrumentation database (a .csmes file) that was generated by CoverageScanner.

2. Load a corresponding execution report (a .csexe file). There may be several reports to choose from: CoverageBrowserdisplays them in a tree view where they can be selected or deselected individually for coverage analysis.

3. Search for untested code segments.

4. Mark dead code or code which cannot be tested as “manually validated”.

5. Add comments to the instrumented code.

CoverageBrowser saves all these data (execution reports, comments, etc.) to the instrumentation database.

10.1 Command Line Arguments

Syntax:

coveragebrowser -m <csmes_file> -e <csexe_file> ...

Where:

-m <argument> | --csmes=<argument>:Load an instrumentation database.

-e <argument> | --csexe=<argument>:Open the import dialog which permits to load a specified execution report.

This command line argument requires the command line option --csmes.


Black box and white box testing

Chapter

11Black box and white box testing

CoverageBrowser can be used both for white box testing and black box testing: If no source code information is available inthe instrumentation database (i.e., in the .csmes file), CoverageBrowser will switch to black box testing mode. In this mode,CoverageBrowser has a simpler user interface that does not provide the functionality that is possible only with access to thesource code. Nevertheless, even with this reduced functionality, it is still possible to import and manage executions.

A black box instrumentation database can be generated by clicking on "FileÕGenerate Black-Box Configuration...".It is possible to merge such a database into a white box configuration at a later stage.

Figure 11.1: CoverageBrowser’s simplified user interface for black box tests


The windows of CoverageBrowser

Chapter

12The windows of CoverageBrowser

12.1 The Executions Window

This window is present in both black box and white box testing mode. It shows all application executions that have been done,including details of their code coverage.

12.1.1 Principles

Executions of the instrumented application are displayed in a tree view in the "Executions" window. CoverageBrowser uses aslash ‘/’ as a separator for grouping measurements together.

For example, the tests shown in Figure 12.1, page 48 have the following names:

• SubTest/test1

• SubTest/test2 (12)

• SubTest/test2 (2)

Code Coverage

Statistics

Selection

Execution State

or Regular Expressions

Wildcard Expressions

Selection using

Figure 12.1: The Executions View

The checkbox next to each item can be used to select executions. The "Sources", "Functions" and "Source Viewer"windows only display the code coverage status of the selected executions.

The code coverage percentage of the tests is visualized by gray horizontal bars in the "Coverage" column.


The Executions Window

The input field allows to filter the output with regular expressions (see Chapter 13.1, page 60). Click the button to selectall the visible executions (i.e., all those that have not been filtered out). Or click the button to unselect all the executions.

For a more finely controlled filter, click the "..." button: This will pop up a dialog by which it is possible to set filteringdepending on the execution state and the comments.

Note that the text which is filtered is a test execution’s full name—for example, SubTest/test1.

Click the -+ button to switch to the execution comparison analysis mode (see Chapter 12.1.3, page 51).

The user can set the state of the executed test by clicking into the "State" field of a test. The new state can be any of thefollowing:

"Unknown":Default state.

" Passed ":This state is used to mark the test as passed. The background colour of the "State" field is then green.

" Failed ":This state is used to mark the test as failed. The background colour of the "State" field is then red.

" Need manual check ":This state is used to indicate that the test must be done manually. The background colour of the "State" field is thenorange.

i The name of the test item and its state can also be defined by an external test suite. (see Chapter 22, page 120)

It is possible to rename, delete or merge executions, or add comments to them through the use of CoverageBrowser’s contextmenus and dock windows1. One can use regular expressions to identify the executions to which these modifications are applied.(The regular expression syntax is described in Chapter 13.1, page 60.) Before regular expression-driven actions are carried out,CoverageBrowser shows a preview of what effects the change would have.

To delete executions, right-click into the "Executions" window and select "Delete multiple executions..." from thecontext menu. A window appears into which the name of the executions can be entered. Here are some examples:

• To delete all execution using the wildcard syntax:Execution Name: *

• To delete all executions in TESTS using the wildcard syntax:Execution Name: TESTS/*

• To delete all executions in TESTS using the regular expression syntax:Execution Name: =TESTS/.*

To rename executions, select "Rename multiple executions..." from the context menu. A window appears (see Fig-ure 12.2, page 50) into which expressions for the old and new names of the executions can be entered. Here are someexamples:

• To move all executions to the directory TESTS, set:Actual Execution Name: =.*New Execution Name: TESTS/&

• To move all executions in the directory TESTS to the directory OLD, set:Actual Execution Name: =TESTS/(.*)New Execution Name: OLD/\1

1After merging, the executions used as the source of the new merged execution are deleted.



• To rename all executions in all directories in testname [directory], set:Actual Execution Name: =([^/]*)/([^/]*)New Execution Name: \2 [\1]

Error Message

Error

Regular Expression

Executions to Rename

New Execution Name

Preview

Figure 12.2: Renaming with regular expressions

12.1.2 Loading an Execution Report

The execution report is produced when an instrumented application finishes execution. Its name is defined by the initializationfunction __coveragescanner_install() of the CoverageScanner library (see Chapter 18.1.1, page 83) and always endswith ".csexe". The report contains the list of all executed code segments for each application that was run. The executionreport is never overwritten; execution data are always appended.

To load an execution report, click on "FileÕLoad Execution Report..." or the icon on the toolbar. A dialog like that inFigure 12.3, page 50 will appear.

Default execution nameScript arguments

Execution result

file name

File watch

Import policy

Script file

Default execution status

Figure 12.3: Loading Execution Report Dialog

The file can be loaded directly or by a script. To load it directly, click on "File" and enter the path of the ".csexe" file to loadinto the free form input box, or use the browse button.If the box at "Delete after loading" is selected, the execution report file will be deleted after it has been loaded. Selectthe option "Open this dialog" for the field "When file becomes modified" to reopen the dialog automatically when theexecution report has been modified.



The "Name" field allows the user to specify the name of the imported instrumentation if this is not already specified in theexecution report (see Chapter 18.1.2, page 84). It is also possible to set the status (“Passed”, “Failed” or “To be checkedmanually”) of all imported executions.If more than one instrumentation is imported, an index numer is appended to its default name to make it unique.

The option "Import Preprocessing" allows to select the behaviour in the case of conflicts or redundant executions:

"Ignore Duplicate Executions":Executions which have executed the same code as an execution which are already imported are ignored.

"Import Duplicate Executions":Executions are imported if at least one instrumented source code line is executed.

"Import Duplicate And Empty Executions":All executions are imported.

"Merge all executions with the same name together":All executions with the same name are merged. The execution counts of all instrumentations are added.

Invalid executions are not imported. If "Display import summary" option is selected, a summary is shown after the operationis completed (see figure 12.4, page 51).

Figure 12.4: Loading Executions - Summary

If the execution report is not accessible through the file system, a script can be used. The script must print the content of theexecution report to the standard output (stdout). The standard error output (stderr) is displayed on the screen and can beused for debugging. On success, the script must return the value 0.

12.1.3 The Execution Comparison Analysis Mode

The Execution Comparison Analysis mode is activated by clicking the button -+ . In this mode, one or more executions must

be selected as references and others as those to analyze. CoverageBrowser then shows only those lines as covered that wereexecuted in the analyzed executions but not in the reference executions. Similarly, the coverage statistics displayed in the"Sources" list contain only the percentage of instrumented statements that were executed only by the executions to analyze.


The Source Browser Window

With Execution Comparison Analysis mode it is therefore possible to find changes in the coverage between different runs of aprogram.

To select the reference executions, use the checkboxes in the "Reference" column at the right of the "Executions" window.They must be selected first. Then select the executions to analyze as before in the "Execution" column.

iIf the execution to analyse is present in the list of reference executions, it is implicitly removed from the list. Soif execution A is compared to executions A and B, CoverageBrowser actually compares execution A only with B,since comparing A with itself will provide no useful information.

Activation

of lines which

are only coveraged

Coverage Statistic of

by the selected execution

are only coveraged

by the selected execution

are displayed

Instrumented lines which

Reference

List

Execution

Analysed

Figure 12.5: Execution Comparison Analysis Mode

12.2 The Source Browser Window

i This feature is not available for black box testing.

The "Sources" window can be displayed by clicking on "ViewÕSources".


The Function Browser Window

Statistics

per File

Source

Header and

Source Files

Compiled

Source excludedfrom the instrumentation

Figure 12.6: Source Browser Window

Each line represents a source file. For C and C++, there are sub-entries for included header files which have been instrumented.When an item is clicked, the "Source Window" is displayed.

The "Coverage" column displays a rudimentary code coverage statistics for each source code file. There is an underlyinghorizontal bar in each field whose length represents the fraction of the code that is covered. The color of the main part of thebar is selected according to code coverage statistics for each file and the value of the thresholds (see Chapter 15.3, page 69).

If parts of the code have been manually validated, then there is a second, gray, horizontal bar at the left of the "Coverage"column; its width represents the number of tests that have been only manually validated. The full length of the column stillrepresents the fraction of all tests that have been validated. When the window however also contains a "Manual Validations"column, the gray horizontal bar is only displayed there.

The checkbox in front of each source file allows to exclude or include the source file to the statistics computation. If excluded,the file is treated as if it and its functions had not been instrumented.

The input field allows to filter the content of the window with regular expressions (see Chapter 13.1, page 60). The filterexpression refers to the full path of the source file. (Example: c:\directory\file.cpp)

Icon Shortcut DescriptionCtrl+Shift+F Previous source file

Ctrl+F Next source file

Table 12.1: Source Browser - Shortcuts

12.3 The Function Browser Window


The "Functions" window can be opened by clicking on "ViewÕFunctions".


The Source Viewer Window

Prototype

Statistics

Namespace

or Class

Method

Figure 12.7: Function Browser Window

It displays the code coverage statistics for all functions, classes and namespaces. A click on an item in the window shows thecode for the corresponding object in the "Source Viewer" window and highlights it.

The "Coverage" column displays a rudimentary code coverage statistics for each function. There is an underlying horizontalbar in each field whose length represents the fraction of the code that is covered. The color of the main part of the bar is selectedaccording to code coverage statistics for each file and the value of the thresholds (see Chapter 15.3, page 69).

If parts of the code have been manually validated, then there is a second, gray, horizontal bar at the left of the "Coverage"column; its width represents the number of tests that have been only manually validated. The full length of the column stillrepresents the fraction of all tests that have been validated. When the window however also contains a "Manual Validations"column, the gray horizontal bar is only displayed there.

The input field allows to filter the output with regular expressions (see Chapter 13.1, page 60). The filter expression refers tothe full names of the items, including the class name and the namespace. (Example: MyNamespace::MyClass::MyProc)

12.4 The Source Viewer Window


12.4.1 Source Display

The "Source Viewer" window can be displayed by clicking on "ViewÕNew Source Window".



Selected

InstrumentationFolding

Filter

Comment

Mark

Unselected

Instrumentation

Test and Code

Coverage Count

Figure 12.8: Source Window

The "Source Viewer" window displays the source file or its C or C++ preprocessed view. Clicking on C++CPP enables the user to

toggle between the 2 different views.

The source code is colored with code coverage instrumentations. The colors used are described in section 12.4.2, page 55.By selecting an area with the mouse, corresponding instrumentations are highlighted and a detailed description of them isdisplayed in the "Explanation" window (see Chapter 12.5, page 57). It is possible to navigate between instrumentations usingthe navigation buttons and . Navigation buttons in yellow, blue, red and green permit to jump to the next or previouscomments, manually validated instrumentations, non-executed code parts or executed code parts. Clicking on the source codeselects the nearest instrumentation.

If a comment is entered for an instrumentation, the icon is displayed in the margin.

On the right side, CoverageBrowser displays the test coverage count2 or the code coverage count3 for each line. If a sourcecode line contains more than one instrumentation, CoverageBrowser display the range of their counts.

Mouse Wheel DescriptionWheel Scroll up/downCtrl+Wheel Zoom in/outShift+Wheel Next/previous instrumentation

Table 12.2: Source Display - Mouse Wheel

12.4.2 Color Convention

Instrumentations are displayed in a source window using different colors:

Green - "Executed":An instrumentation is displayed in green when the code has been executed.

Orange - "Partially Executed":An instrumentation is marked as "Partially Executed" when it is not completely executed. This occurs when aBoolean expression was only true or false for example. In the case of a source code line which contains more than

2The test coverage count is obtained by generating the instrumentation with CoverageScanner using the option --cs-hit3The code coverage count is obtained by generating the instrumentation with CoverageScanner using the option --cs-count



Icon Shortcut DescriptionCtrl+Shift+B Previous comment

Ctrl+B Next commentCtrl+Shift+U Previous unexecuted code

Ctrl+U Next unexecuted codeCtrl+Shift+T Previous executed code

Ctrl+T Next executed codeCtrl+Shift+V Previous manually validated instrumentation

Ctrl+V Next manually validated instrumentationCtrl+Shift+I Previous instrumentation

Ctrl+I Next instrumentationCtrl+Shift+F Previous source file

Ctrl+F Next source fileCtrl+J Open a new source window

C++CPP Ctrl+Shift+J Switch between the preprocessor view and the original source

Ctrl+K Add/Edit CommentsCtrl+Shift+K Remove Comments

ok Ctrl+W Mark as Validatedok Ctrl+Shift+W Clear Validation Flag

Ctrl+Z UndoCtrl+Shift+Z Redo

Table 12.3: Source Display - Shortcuts

one instrumentation, the line is marked as "Partially Executed" when one of its instrumentations has not been"Executed". A detailed information is displayed in the "Explanation" window (see Chapter 12.5, page 57).

Red - "Never Executed" or "Execution count too low":An instrumentation is displayed in red when the code is never executed or when the execution count is lower that than theexecution count requested.

Magenta - "Dead-Code":An instrumentation is displayed in magenta when the code cannot be executed.

Blue - "Manually Set To Be Executed":The user has the possibility to mark an instrumentation as ’Manually Validated’. This is usually to exclude dead codeor code which cannot be tested for code coverage statistics. This state is only relevant if executions are in a "NeverExecuted" or "Partially Executed" state.

Gray - "Unknown" or "Hidden":Gray is used when no information about instrumentation is available. This occurs when no executions are selected orwhen comparing executions of tests (see Chapter 12.1.3, page 51).

12.4.3 Comments


12.4.3.1 Editing Comments

It is possible to add a comment by selecting an instrumentation and clicking on the context menu entry "Add/Set Comment",the main menu entry "InstrumentationÕAdd/Set Comment" or the icon on the toolbar.


The Explanation Window

The "Comment" Window 12.9, page 57 appears and allows a comment to be edited. The most recently entered comments can beretrieved by clicking on the "Last Comments" selection field. Basic text formatting is possible using the integrated toolbarbuttons (see 12.4, page 57).

Text

Formating

Comment

Length

Last

Comments

Edit Field

Figure 12.9: Comment Editing

i If a minimal length for a comment is set, the comment can only be entered if this is reached (see Chapter 15.2,page 69).

The comment is printed in the explanation in a yellow box and the icon ( ) is displayed in the source window near the linenumber.

Icon Shortcut DescriptionCtrl+S StrikeoutCtrl+B BoldCtrl+I ItalicCtrl+U UnderlineCtrl+J JustifyCtrl+R Align RightCtrl+L Align LeftCtrl+M CenterCtrl+Z Undo

Ctrl+Shift+Z Redo

Table 12.4: Comments - Shortcuts

12.4.3.2 Removing Comments

It is possible to remove a comment by selecting an instrumentation and clicking on the context menu entry "Clear Comments",the main menu entry "InstrumentationÕClear Comment" or the icon on the toolbar.

12.5 The Explanation Window



The Statistics Window

The "Explanation" Window 12.10, page 58 is a docking window which is automatically updated with a detailed descriptionof the selected instrumentations of the source window. For each instrumentation, the following information is displayed:

1. A short description of the instrumentation state (see Chapter 12.4.2, page 55).

2. The preprocessed source code which is concerned by the instrumentation.

3. For Boolean expressions, the truth-table which shows executed and unexecuted states.

4. The list of executions which are executing the portion of code.

5. User comments.

Preprocessed

Source

Comment

Truth-Table

State

Figure 12.10: Explanation Window

CoverageBrowser displays the truth-table in the case of a Boolean expression which is partially executed. The truth-tableindicates which value the expression has or has not reached during execution.

Example: the truth-table 12.5, page 58 indicates that the expression was false but not true.

TRUE FALSE

no yes

Table 12.5: Truth-Table Example

12.6 The Statistics Window

The "Statistics" Window 12.11, page 58 is a docking window which is automatically updated with the code coveragestatistic for the whole project.

Figure 12.11: Statistics Window


The Statistics Window

If parts of the code are manually validated, then their percentage is also displayed in the coverage statistics. The bar chart hasthen two regions: the percentage of the manually validated code at the left and then the percentage of the code that is coveredby the automatic tests. The numbers in the bar chart refer to all validated code, that which was manually validated together withthat covered by tests.

By clicking the "..." button, the two kinds of validation are split into two separate bars.

Figure 12.12: Statistics Window with Manual Validation, Split


Working with CoverageBrowser

Chapter

13Working with CoverageBrowser

13.1 Filtering with wildcards or regular expressions

CoverageBrowser provides a generic filtering mechanism of rows using wildcard or regular expressions. Wildcard expressionsare activated per default and regular expressions are selected when the expression starts with an equal sign (’=’). Clicking onthe filter icon converts the expression from wildcard into regular form as far as this is possible and vice versa.

Icon Description.* The filter uses regular expression syntax.?* The filter uses wildcard syntax.! Syntax error. More information are displayed in the status bar.

Table 13.1: Filter States

13.1.1 Wildcard Expressions

Element Meaning* any characters (0 or more)? any character[...] set of character

Example: foo*bar match any tests containing the string foo followed by bar.

13.1.2 Regular Expression

The first character must be ’=’ to activate the regular expressions.


Code Coverage Level

13.1.2.1 Pattern matching

13.1.2.2 String substitution

13.2 Code/Test Coverage Level

The menu entry "InstrumentationÕLevel:x" permits to set the targeted code coverage count or, if the compiled withinstrumentation hit support1, the targeted test coverage count.

The level is corresponding of the number of code/test coverage count necessary to consider that an instrumented code isexecuted.Example: Setting the level to 10, will made necessary to execute 10 times the each line of the source code if compiled withcode coverage count. If compiled with code coverage hit, 10 execution runs need to execute each lines of the source code. Themenu entry "ToolsÕTest Coverage Count Mode" and the button 1.2.3. permits to switch between code coverage count and testcoverage count analysis. This simulates the behaviour of the compilation with code coverage hit support2 when the project iscompiled with code coverage count support3.

13.3 Code Coverage Algorithm

CoverageBrowser displays the code coverage analysis (branch, decision or condition) generated be CoverageScanner. But"InstrumentationÕCoverage MethodÕBranch only" permits to reduce the analysis to the code coverage of branches.This produces the same result as compiling with the --cs-branch of CoverageScanner. "InstrumentationÕCoverageMethodÕDecision, Condition and Branches" permits to show the code coverage analysis at the level defined at thecompilation.

Here a short overview of the command line options necessary for each code coverage analysis method:

13.4 Optimized Execution Order

CoverageBrowser is able to calculate an optimized order of the executions (i.e.: the order of tests which permits to maximizethe code coverage after each execution). This order is specially adapted to manual testing: following this order permits toexecute first the tests which are giving a high code coverage and so detecting rapidly errors in the first test executions.

To calculate the execution order proceed as follows:

1. Select a set of executions in the "Executions" window.

2. Click on "ToolsÕOptimized Execution Order...". The window (see Figure 13.1, page 63) will de display.

1Command line option --cs-hit from CoverageScanner. To compile with code coverage count support, the option --cs-count must be added in thecommand line.

2CoverageScanner command line option --cs-hit3CoverageScanner command line option --cs-count


Optimized Execution Order

Element Meaningc Any character represents itself unless it has a special regexp meaning. Thus c matches the character

c.\c A character that follows a backslash matches the character itself except where mentioned below.

For example if you wished to match a literal caret at the beginning of a string you would write \^.\a This matches the ASCII bell character (BEL, 0x07).\f This matches the ASCII form feed character (FF, 0x0C).\n This matches the ASCII line feed character (LF, 0x0A, Unix newline).\r This matches the ASCII carriage return character (CR, 0x0D).\t This matches the ASCII horizontal tab character (HT, 0x09).\v This matches the ASCII vertical tab character (VT, 0x0B).\xhhhh This matches the Unicode character corresponding to the hexadecimal number hhhh (between

0x0000 and 0xFFFF).\0ooo (i.e., zero ooo) matches the ASCII/Latin1 character corresponding to the octal number ooo (between 0 and 0377).. (dot) This matches any character (including newline).\d This matches a digit.\D This matches a non-digit.\s This matches a whitespace.\S This matches a non-whitespace.\w This matches a word character.\W This matches a non-word character.^ The caret negates the character set if it occurs as the first character, i.e. immediately after the

opening square bracket. For example, [abc] matches ’a’ or ’b’ or ’c’, but [^abc] matchesanything except ’a’ or ’b’ or ’c’.

- The dash is used to indicate a range of characters, for example [W-Z] matches ’W’ or ’X’ or ’Y’or ’Z’.

E? Matches zero or one occurrence of E. This quantifier means "the previous expression is optional"since it will match whether or not the expression occurs in the string. It is the same as E{0,1}. Forexample dents? will match ’dent’ and ’dents’.

E+ Matches one or more occurrences of E. This is the same as E{1,}. For example, 0+ will match’0’, ’00’, ’000’, etc. . .

E* Matches zero or more occurrences of E. This is the same as E{0,}. The * quantifier is often usedby a mistake. Since it matches zero or more occurrences it will match no occurrences at all. Forexample if we want to match strings that end in whitespace and use the regexp \s *$ we wouldget a match on every string. This is because we have said find zero or more whitespace followed bythe end of string, so even strings that don’t end in whitespace will match. The regexp we want inthis case is \s +$ to match strings that have at least one whitespace at the end.

E{n} Matches exactly n occurrences of the expression. This is the same as repeating the expression ntimes. For example, x{5} is the same as xxxxx. It is also the same as E{n,n}, e.g. x{5,5}.

E{n,} Matches at least n occurrences of the expression.E{,m} Matches at most m occurrences of the expression. This is the same as E{0,m}.E{n,m} Matches at least n occurrences of the expression and at most m occurrences of the expression.() Permits to group expressions into sub-expressions.| Alternative. Example: "aaa|bbb" matches the string "aaa" or "bbb".

Element Meaning& Matched expression\n sub-expression number n. Example: the regular expression is "(.*):([0-9]*)" matches the string "joe:18".

The replacement string "\1 is \2" will produce the result: "joe is 18"


Comparing Releases

Coverage analysis CoverageScanner command line optionBranch --cs-branch

Decision with full instrumentation --cs-decision --cs-full-instrumentation

Decision with partial instrumentation --cs-decision

Condition with full instrumentation --cs-full-instrumentation

Condition with partial instrumentation (default)

Execution Order

first executions

Execution name

Coverage of the 'n'

Figure 13.1: Optimized Execution Order

13.5 Comparing Code Coverage of Two Software Releases

CoverageBrowser is able to instrumentation database together in order to:

1. check is the modified/unmodified code is correctly tested.

2. find which tests are impacted by a source code modification.

This feature is particularly adapted to compare two releases together which contains small modifications (bug fixes only) and tolimit the tests of the modified code only.

In this mode CoverageBrowser uses the following typographic rules:

Rule Source Window Method List Source List Execution ListNormal font Identical4 source part Identical4 methods Identical4 files Executions available in

both releasesBold Modified methods Modified filesBold+Underline New text inserted New methods New files New executionsBold+Strike Deleted text Deleted methods Deleted files Missing executions

CoverageBrowser comparison and difference algorithm is particularly designed for C and C++ source code and ignore whitespaces and modifications in comments.


Changing the Instrumentation Database

13.5.1 Reference Database

The reference database is the base instrumentation database which are used for the comparison. To select it click on"ToolsÕCompare with..." and select a .csmes database. Switching the working database with the reference database canbe performed by clicking on "ToolsÕSwitch databases".

Once the reference file loaded, additional filter possibilities are available in the "Executions", the "Sources" and the"Methods" window. This filters permits to show/hide, modified, new, deleted or identical procedures and source files.

The "Executions" window displays a mix between the executions of the reference and the current release:

• stroked executions are only available in the reference release. The statistic computed is corresponding to the coverage forthe reference release. This value can be interpreted as an expected code coverage when this tests get re-executed on thecurrent release.

• underlined executions are new tests.

• executions available in both version are not highlighted.

13.5.2 Coverage analysis of modified/unmodified source code

CoverageBrowser is able to limit the code coverage analysis to the modified (resp. unmodified) functions. When selecting thecoverage analysis on the modified (resp. unmodified) functions only, CoverageBrowser treat all unmodified (resp. modified)functions as if they are not instrumented. Limiting the code coverage analysis to modified functions can be a practical way toverify that the new features are tested and to identify the list of tests which have are impacted by a modification.To limit the code coverage to modified function (resp. unmodified functions) click on "ToolsÕAnalysis of ModifiedMethods" (resp. "ToolsÕAnalysis on Identical Methods").

13.6 Changing the Instrumentation Database

13.6.1 Merging Instrumentations

Clicking on the menu entry "FileÕMerge with..." permits to import the executions, the source code and the instrumentationsfrom other .csmes files. Comments and code mark as validated are merged together.

13.6.2 Importing Unit Tests

Clicking on the menu entry "FileÕImport Unit Tests..." permits to import the execution report of unit tests into thecurrent application. Only execution reports of source file present into the main application are imported, instrumentation ofother source files (for example test code) are ignored.


Changing the Instrumentation Database

13.6.3 Importing Reviewer Comments

Clicking on the menu entry "FileÕImport Reviewer Comments..." permits to import comments and manual validationsof a previous version of the current instrumentation database. Comments and manual validations of unmodified functions willbe imported even if if the source code is modified.


Generation of Reports

Chapter

14Generation of Reports


14.1 EMMA-XML Report

Clicking on the menu entry "ReportsÕExport Statistics in EMMA-XML Format..." permits to export code coveragestatistics of in EMMA-XML format. This output format generate only global statistics in a format that is compatible with EMMA.This format allows to use Squish Coco in tools that provide support for EMMA, notably giving an easy way to use Squish Cocowith continuous integration servers like Hudson.

EMMA defines only four categories for coverage: classes, methods, blocks, and lines. Due to the fact that blocks does not makesense with Squish Coco, this cathegories have the following meaning:

EMMA category Meaningclasses A class is considered executed if one of its method is called. Code which are not in a class are located in

the class "" (empty).methods A method is covered if it was called.blocks Code coverage at branch level.lines Line code coverage. (if compiled with line coverage support)conditions Decision/Condition line coverage. (if compiled with Decision/Condition coverage)

14.2 Cobertura-XML Report

Clicking on the menu entry "ReportsÕExport Statistics in Cobertura-XML Format..." permits to export code cov-erage statistics of in Cobertura-XML format. This output format generate only global statistics in a format that is compatible


JUnit Report

with Cobertura. This format allows to use Squish Coco in tools that provide support for Cobertura, notably giving an easyway to use Squish Coco with continuous integration servers like Hudson and SonarQube.

The statistics in the Cobertura report are computed a little bit differently as usual due to the limitations of the report format.The report is in fact a mis between line and condition coverage.

For this reason, we record as Cobertura-condition every branch, decision and condition instrumentation. So for example,the end of a function is marked as one condition to fulfill. Also, the report imposes that each line with a condition is alsoinstrumented at line coverage level. This is not always the case (for example for empty functions) and so in this case we addartificially some instrumentations. And finally, if an instrumented statement is coded in more than one line, it is necessary togenerate one instrumentation for each of this line.

All this have an impact on the computation of the statistics for classes, method and sources. But the values obtained arecomparable to the all other reports.

14.3 JUnit Report

Clicking on the menu entry "ReportsÕExport JUnit Report..." permits to export the test result as JUnit report. Thisreport does not contain coverage data and only list the test execution result (passed or failed) for each test item.

14.4 Statistics per Source File

Clicking on the menu entry "ReportsÕExport Statistics per Source File" permits to export code coverage statisticsof each source file in CSV format.

The file contains the coverage of all combination of source files and executions selected and the execution status (see Figure 14.1,page 67). Comma and semi-colon can be selected as separator in the CSV file using the "File Type" input field.

Source and Executions

Coverage per

Coverage per

Source File

Global Coverage

Execution Status

Coverage per

Executions

Figure 14.1: Statistics Format - CSV table


Statistics per Method

14.5 Statistics per Method

Clicking on the menu entry "ReportsÕExport Statistics per Method" permits to export code coverage statistics of eachfunction and procedure in CSV format.

14.6 HTML Report

Clicking on the menu entry "ReportsÕGenerate HTML Report..." permits to export code coverage statistics (per methods,source files, executions, . . . ) of the selected executions in HTML format. It permits also to list the manually validated andunexecuted code parts.

14.7 Text Report

Clicking on the menu entry "ReportsÕGenerate Text Report..." permits to generate a small text report in the form ofone line per executed/unexecuted item. A distinct line format can be specified for executed or unexecuted lines.

Following keywords are recognized:

%f: Source code file name

%l: Line number

%c: Column number

%m: Explanation

Example: setting the field "Unexecuted Code Fragments" to "%f:%l: %m" will create a text file which contains all unexecutedcode parts. Each line will look like as follows:

foo.cpp:55: Unexecuted: ’return;’


Preferences

Chapter

15 Preferences

15.1 Save/Load Project

"Save/Restore window position":If this option is selected, the position of all windows and toolbars will be restored upon application restart.

"Reload automatically the last project":If this option is selected, the last project opened will automatically be reloaded upon application restart.

"Saves project automatically on exit":Saves the project file automatically without asking on application exit.

15.2 Comments

"Minimum Comment Size":The minimum comment size, is the minimum length requested for a comment.

"Do not request a comment when setting an item to the ’manually validated’ state":This option is used to allow the user to manually modify the state of an instrumentation without entering a comment.

i Enabling this option should be avoided because modifying the state of an instrumentation should beperformed with a valid reason which should be recorded as a comment.

15.3 Thresholds

Thresholds are trigger values that control the background color of:

• the instrumented source files in in the "Sources" window.


Cache

• the instrumented classes or namespaces in the "Methods" window.

• the instrumented functions, methods or procedures in the "Methods" window.

Description:

"Medium/High Coverage Level":If the statistic is above this value, the background color is set to green. Otherwise, the color is orange.

"Low/Medium Coverage Level":If the statistic is below this value, the background color is set to red. Otherwise, the color is orange.

15.4 Cache

Description:

"Execution":Maximum number of executions loaded into the RAM.

"Source":Maximum number of source files loaded into the RAM.


Part IV CoverageScannerReference Manual


Introduction

Chapter

16 Introduction

CoverageScanner is the application which inserts the instrumentation into the source code. It performs the following operations:

1. Runs the C preprocessor.

2. Instruments the preprocessed source code.

3. Compiles the source code using the native C or C++ compiler.

Mainly, during normal usage, it replaces the native compiler.

i CoverageScanner does only support precompiled headers with Microsoft® Visual Studio® .NET tool chain.CoverageScanner automatically deactivates this feature during compilation.

16.1 Invoking CoverageScanner

CoverageScanner works like a pre-processor which calls after instrumenting a source file the native compiler (Microsoft®

cl.exe, GNU g++, . . . ).

Several invocation method are possible:

1. Using the dedicated CoverageScanner executable for a specific compiler: CoverageScanner executable for a specificcompiler is called to cs+’native compiler name’ or alternatively ’native compiler name’+-cs. (example: cl.exe is theMicrosoft® C++ compiler and the corresponding CoverageScanner executable is cscl.exe or cl-cs.exe).

CoverageScanner aims to work then as the native compiler with the difference that the code get instrumented.

Example:

csg++ source.cpp -o application.exe

2. Using CoverageScanner compiler wrapper and activating the instrumentation using --cs-on in the command linearguments:


Invoking CoverageScanner

Squish Coco provides also a replacement of the native compiler called wrapper which extends its command line argumentto support the code coverage feature. To use it it is necessary to add at the first position in the PATH variable the path ofCoverageScanner wrapper. The instrumentation get activated, when adding --cs-on to the command line argument.

Example:

export PATH=/opt/SquishCoco/wrapper/bin:$PATHg++ --cs-on source.cpp -o application.exe

Or, without modifying globally the PATH variable:

PATH=/opt/SquishCoco/wrapper/bin:$PATH g++ --cs-on source.cpp -o application.exe

3. Using CoverageScanner compiler wrapper and activating the instrumentation through COVERAGESCANNER_ARGS environ-ment variable:

It is also possible to set the command line arguments of the wrapper using the COVERAGESCANNER_ARGS environmentvariable (see Chapter 17.1, page 74). In this case it is no longer necessary to modify the command line arguments.

Example:

export PATH=/opt/SquishCoco/wrapper/bin:$PATHexport COVERAGESCANNER_ARGS=’--cs-on’g++ source.cpp -o application.exe

Or, to instrument a complete project built using GNU make:

COVERAGESCANNER_ARGS=’--cs-on’ PATH=/opt/SquishCoco/wrapper/bin:$PATH make all

4. Using --cs-compiler command line option of CoverageScanner:

--cs-compiler permits to select which native compiler should be used. All other command line arguments, except thosespecific to CoverageScanner, are passed through the native compiler.

Example:

coveragescanner --cs-compiler=g++ source.cpp -o application.exe

i The native compiler path needs to appear in the PATH variable. This is necessary for finding the location of thenative compiler after insertion of the instrumentation code.

Compiler support is done with the help of a profile. It contains declarations which make it possible to adapt CoverageScannerto C, C++ and C# preprocessors, compilers and linkers.

For a description of the compilers that are currently supported see Chapter G, page 212. For a description of the profile syntaxsee Chapter H, page 214


Command Line Arguments

Chapter

17Command Line Arguments

Syntax:

cs<compiler> [<cs-options>] <compiler arguments>coveragescanner --cs-compiler=<STRING> [<cs-options>] <compiler arguments>coveragescanner --cs-profile=<STRING> [<cs-options>] <compiler arguments>

where

<compiler>:One of the compilers supported by Squish Coco. Calling cs<compiler> results in a call of CoverageScanner as awrapper of <compiler>.

<cs-options>:One or more of the options that are described in the following sections.

<compiler arguments>:All the other options. They are copied to the command line of the compiler when it is called by CoverageScanner.

For the use of --cs-compiler and --cs-profile see Chapter 17.2.6, page 79.

iThe command line arguments are prioritized by their order in the command line: the latest entered option has ahigher priority than the first one.Example: cscl --cs-include=foo.h --cs-exclude=foo.h will exclude the header file foo.h from thecoverage analysis.

17.1 Environment variables

Apart from the command line, CoverageBrowser also reads options from the environment variable COVERAGESCANNER_ARGS.If more than one option is present, the options must be separated by spaces or tab characters (‘ ’ or ‘\t’). If both command linearguments are present and COVERAGESCANNER_ARGS is set, the command line options take precedence over the options in theenvironment variable.


List of options

17.2 List of options

17.2.1 File inclusion/exclusion options

--cs-include-file-wildcard=<STRING>:This command line option enables the user to include a file in the coverage analysis. The file name is specified using awildcard expression.

Example: --cs-include-file-wildcard=../include/*.h includes all header file names with the extension .hlocated in ../include/.

--cs-exclude-file-wildcard=<STRING>:This command line option enables the user to exclude a file from the coverage analysis. The file name is specified using awildcard expression.

Example: --cs-exclude-file-wildcard=*.h excludes all header file names with the extension .h.

--cs-exclude-file-abs-wildcard=<STRING>:The same as --cs-exclude-file-wildcard, but with an absolute file path.

--cs-include-file-abs-wildcard=<STRING>:The same as --cs-include-file-wildcard, but with an absolute file path.

--cs-include-file-regex=<STRING>:This command line option enables the user to include a file in the coverage analysis. The file name is specified using aregular expression. (see chapter 17.4, page 81 for the syntax)

Example: --cs-include-file-regex=^\.\./include/[^/]*\.h$ includes all header file names with the extension.h located in ../include/.

--cs-exclude-file-regex=<STRING>:This command line option enables the user to exclude a file from the coverage analysis. The file name is specified using aregular expression. (see chapter 17.4, page 81 for the syntax)

Example: --cs-exclude-file-regex=^.*\.h$ excludes all header file names with the extension .h.

--cs-exclude-file-abs-regex=<STRING>:The same as --cs-exclude-file-regex, but with an absolute file path.

--cs-include-file-abs-regex=<STRING>:The same as --cs-include-file-regex, but with an absolute file path.

--cs-exclude-path=<STRING>:This command line option enables the user to exclude the files located in a specific path (or in any of its subdirectories)from the code coverage analysis.

Example: --cs-exclude-path=c:\include excludes all files located in c:\include.

--cs-include-path=<STRING>:This command line option enables the user to include files located in a specific path (or in any of its subdirectories) in thecoverage analysis.

Example: --cs-include-path=c:\include includes all files located in c:\include.

If more than one file inclusion/exclusion option is present, the last one that matches a given file determines whether it isincluded or excluded.

Examples:

• --cs-exclude-file-wildcard=* --cs-include-file-wildcard=*.cpp instruments only .cpp files.


List of options

• --cs-exclude-file-wildcard=* --cs-include-file-wildcard=*.cpp --cs-include-file-wildcard=*.cxxinstruments only .cpp and .cxx files.

• --cs-exclude-file-wildcard=*/3rdparty/* does not instrument any files from the directory 3rdparty.

• --cs-exclude-file-wildcard=*/3rdparty/* --cs-include-file-wildcard=* instruments all files: The firstfilter rule has no effect.

File paths and wildcard expression are always converted to a canonical form which:

• does not contains a ‘.’ or a ‘..’, and

• does not contain a duplicated path separation character.

On Microsoft® Windows the conversion to the canonical form also includes:

• convertion of all characters to lower case and

• replacement of every slash (‘/’) with a backslash (‘\’).

Regular expressions are untouched. For example, it is necessary to ensure that double path separators are not present in them.

Relative file names are computed relatively to the current working directory. For example, depending on the working directoryone has to use different parameters in order to inhibit the instrumentation of a file src/test.c: If one is outside the directorysrc, one may use

$ csgcc src/test.c --cs-exclude-file-wildcard=src/test.c

but when switching to src, the parameters must change:

$ cd src$ csgcc test.c --cs-exclude-file-wildcard=test.c

!Setting inclusion/exclusion expressions on absolute paths should be preferred on complex build systems, sincemodern build system often change the current working directory. Working with absolute paths make it easier toset global rules for a project.

17.2.2 Function inclusion/exclusion options

--cs-include-function-wildcard=<STRING>:This command line option enables the user to include a function in the coverage analysis. The function name is specifiedusing a wildcard expression and must include for C++ code the class name and namespace.

Example: --cs-include-function-wildcard=C::* includes all members of the class C.

--cs-exclude-function-wildcard=<STRING>:This command line option enables the user to exclude a function from the coverage analysis. The function name isspecified using a wildcard expression and must include for C++ code the class name and namespace.

Example: --cs-exclude-function-wildcard=*::Get* excludes all members of any classes which starts with Get.

--cs-include-function-regex=<STRING>:This command line option enables the user to include a function in the coverage analysis. The function name is specifiedusing a regular expression and must include for C++ code the class name and namespace. (see chapter 17.4, page 81 forthe syntax)


List of options

--cs-exclude-function-regex=<STRING>:This command line option enables the user to exclude a function from the coverage analysis. The function name isspecified using a regular expression and must include for C++ code the class name and namespace. (see chapter 17.4,page 81 for the syntax)

If more than one inclusion/exclusion option is present, the last one that matches a given function determines whether it isincluded or excluded.

17.2.3 Instrumentation options

--cs-off:Disables the code coverage analysis of CoverageScanner. This is equivalent to calling the native compiler or linker.

--cs-on:Activates the code coverage analysis of CoverageScanner. This option is set by default.

--cs-hit:Generation of code coverage hit instrumentation.

--cs-count:Generation of code coverage count instrumentation. (default)

--cs-branch:Generation of code coverage instrumentation of branches.

--cs-decision:Generation of code coverage instrumentation of branches and decisions.

--cs-condition:Generation of code coverage instrumentation of branches, decisions and conditions. (default)

--cs-function:Generation of function code coverage instrumentation. (default)

--cs-no-function:Disable the function code coverage instrumentation.

--cs-line:Generation of line code coverage instrumentation. (default) This option requires a full instrumentation and an instrumen-tation at decision or condition level.

--cs-no-line:Disable the line code coverage instrumentation.

--cs-partial-instrumentation:Suppression of redundant conditions.

--cs-full-instrumentation:No suppression of redundant conditions. (default)

--cs-no-exceptions:Do not instrument the catch block of a try. . .catch statement.

--cs-no-assignments:Do not instrument Boolean expressions in assignments.

--cs-boost:The code behind the macro BOOST_FOREACH is not instrumented. (this option is enabled per default)


List of options

--cs-no-boost:No specific handling for the Boost library.

--cs-qt4:Function filter which suppresses the instrumentation of class members (Q_OBJECT, qt_metacall, qt_metacast, . . . )generated by mocfor Qt4 library. The code behind the macro Q_FOREACH is also not instrumented. (this option is enabledper default)

--cs-no-qt4:No specific handling for the Qt4 library.

--cs-qt3:Function filter which suppresses the instrumentation of class members (Q_OBJECT, className, tr, . . . ) generated bymocfor Qt3 library. (this option is enabled per default)

--cs-no-qt3:No specific handling for the Qt3 library.

--cs-no-annotations:No extraction of annotations (comments and manual validations) from the source files.

17.2.4 Debugging options

--cs-verbose-source-lines=<INT>:Maximum number of source code lines dumped in an error message.

--cs-verbose=<STRING>:Generate verbose debug messages. Possible options are:

build: During build, the instrumentation by CoverageScanner is traced. This includes e.g. the information aboutwhich files are instrumented and which are excluded from instrumentation.

api: When the instrumented program is run, the calls to the CoverageScanner library are traced.

By default, the output is written to stderr.

--cs-verbose-file=<FILE>:Define the file used for generating a CoverageScanner log file. This option is meaningless when --cs-verbose is notused.

--cs-keep-instrumentation-files:Do not delete temporary instrumentation files. They will be left under random names in /tmp on Unix and in %TEMP% onWindows.

--cs-warnings=<STRING>:Instrumentation warning behaviour:

none: warnings won’t be displayed.

display: warnings will be displayed via stderr.

error: warnings will trigger a compilation error.

17.2.5 Execution report options

--cs-output=<STRING>:Set the default output file name of the execution report. Per default, CoverageScanner uses the application name (without




List of options

the path). The extension .csexe is added automatically.--cs-output supports several format specifier which are generated during the compile time:

%T: Current time as an integer. (example: 163355 for 16h33 and 55s)

%D: Current date as an integer. (example: 20140130 for the January 30, 2014)

%F: Application/library file name without its path.

%A: Application/library absolute file name.

%B: Application/library file name without its path and extension.

%P: Application/library file path.

%/: File path separator (on Unix a slash ’/’ and on Windows a backslash ’\’)

The following format specifier which are generated during the runtime:

%c: Current working directory. The current working directory is computed before executing the main() function.Even if the current working directory change, this path remains identical during the whole execution.

%p: Process identifier. If for any reason it cannot be computed, %p will be replaced through an empty string.

%t: Current time. If for any reason it cannot be computed, %t will be replaced through an empty string.

%d: Current date. If for any reason it cannot be computed, %d will be replaced through an empty string.

--cs-lock-csexe:Create a lock file which lock the access of the .csexe file during its generation.

--cs-nolock-csexe:Inverse option to --cs-lock-csexe. The preprocessor output is transmitted to CoverageScannerthrough a temporaryfile.

--cs-dump-on-signal=INT:Generate an execution report when a specific signal was received. The signal can be specified by a number (1 to 63) orusing the conventional signal same (SIGINT, SIGTERM, . . . )

--cs-dump-on-event=STRING:Generate an execution report when a specific Microsoft® Windows event was received.

--cs-minimum-api:The CoverageScanner API generated using this command line option has less dependencies to external libraries asusual. This may be necessary for embedded systems or when compiling C# application with the command line switch/noconfig.

17.2.6 Miscellaneous options

--cs-compiler=<STRING>:This command option enables the user to select a profile. This option does not normally need to be specified, due to thefact that the profile name is usually extracted from the command line name. (cscl.exe implicitly selects the profilecl.cspro)

Example: --cs-compiler=cl selects the profile ’cl.cspro’.

--cs-profile=<STRING>:This option permits to set directly the path of CoverageScanner profile. Using this option if not compatible with--cs-compiler.

Example: --cs-profile=%SQUISHCOCO%\cl.cspro is equivalent to --cs-compiler=cl on Microsoft® Windows.

--cs-architecture=<STRING>:Permits to specify the target architecture.


List of options

--cs-libgen=<STRING>:Adds an additional command line option to the compiler which generates the code coverage library. This command canbe used more than once.

--cs-link-instrumentation-tables:Chain all instrumentation tables together during the link time and not during the run time.

--cs-library-after=<STRING>:Add CoverageScanner library after the linker command line argument <STRING>.

--cs-library-at-end=<STRING>:Add CoverageScanner library at the end of the command line.

--cs-parallel-build=<INT>:Maximum number of concurrent code generation/compilation.

--cs-cpp-ext=<LIST>:List of file extensions corresponding to C++ files (ex: .cpp,.cxx)

--cs-c-ext=<LIST>:List of file extensions corresponding to C files (ex: .c)

--cs-native-toolchain=<STRING>:Adds a directory in the search path of the native compiler. This command can be repeated. In this case the first one is theprefered location.

--cs-option-file=<FILE>:Specification of command line option through a file. It can contains every valid CoverageScanner command line optionsor command line options from the native tool. The file must contain one argument per line. The arguments are notencoded (spaces are not escaped and also backslash is treated as a normal character). This command can be repeated.

--cs-vs2010-lambda:Instrument directly the expression of a return statement of a lambda function and permits a better support of lambdaexpression on Microsoft® Visual Studio® 2010. This is only valid for C++11 compiler or Microsoft® Visual Studio® 2010.For C or older C++ compiler, this option has no effect.

--cs-no-cspch:If set, the files with the extension .cspch are not generated. This files are only used to enhance the compilation speedwhen using precompiled headers.

--cs-architecture:Internal option of CoverageScanner which permits to select alternate settings.

--cs-no-abort-on-error:By default, any errors raised while instrumenting C/C++ source code will stop the compilation. This behaviour can beoverridden using the [--cs-no-abort-on-error] if it’s given, the wrapper executable will display an error message butrecover it by compiling the source file without instrumenting it.

--cs-no-exit-handler-installation:Disable the automatic installation of the handler which saves the code coverage report. If this handler is not installed, noexecutions report will be generated when the application exits or when a custom signal or Windows event is received.Calling __coveragescanner_install() permits to install it manually.

--cs-enable-coveragescanner-library-abs-wildcard=<STRING>:: Enable the generation of the CoverageScanner library when the generated binary whose absolute file name is matchingthe given wildcard expression.

--cs-disable-coveragescanner-library-abs-wildcard=<STRING>:Disable the generation of the CoverageScanner library when the generated binary whose absolute file name is matchingthe given wildcard expression.

--cs-memory-pool=<INT>:Generate a CoverageScanner library which uses an own memory pool of a size of INT bytes instead of using mal-loc()/free(). This option is only usefull for operating system which does not support dynamic memory allocation.


Instrumenting using preprocessor defines

17.3 Instrumenting using preprocessor defines

CoverageScanner activates also the instrumentation if some defines are present in the command line option. This permits toinstrument code with an IDE which does not permits to add custom command line options to the compiler or linker.Example: Defining COVERAGESCANNER_COVERAGE_ON is equivalent of adding --cs-on to the command line.

Define EquivalenceCOVERAGESCANNER_RUNTIME_LOG --cs-verbose=api

COVERAGESCANNER_DUMP_ON_EVENT --cs-dump-on-event=Global\COVERAGE

COVERAGESCANNER_COVERAGE_ON --cs-on

COVERAGESCANNER_COVERAGE_HIT --cs-hit

COVERAGESCANNER_COVERAGE_COUNT --cs-count

COVERAGESCANNER_COVERAGE_BRANCH --cs-branch

COVERAGESCANNER_COVERAGE_DECISION --cs-decision

COVERAGESCANNER_COVERAGE_CONDITION --cs-condition

COVERAGESCANNER_COVERAGE_LOCK_CSEXE --cs-lock-csexe

COVERAGESCANNER_COVERAGE_NOLOCK_CSEXE --cs-nolock-csexe

COVERAGESCANNER_COVERAGE_PARTIAL_INSTRUMENTATION --cs-partial-instrumentation

COVERAGESCANNER_COVERAGE_FULL_INSTRUMENTATION --cs-full-instrumentation

COVERAGESCANNER_COVERAGE_NO_EXCEPTIONS --cs-no-exceptions

COVERAGESCANNER_COVERAGE_NO_ASSIGNMENTS --cs-no-assignments

COVERAGESCANNER_MINIMUM_API --cs-minimum-api

17.4 Regular Expressions

Regular expressions are composed of items (see table 17.1, page 82 for the full list) and combining operators. These can be:

• [. . .]: The brackets are used to define a set of characters. For example, [123] corresponds to the characters 1, 2 or 3. Itis possible to define a range using the minus sign: [a-z] corresponds to all characters between a and z.

• [^. . .]: The caret defines a set of characters which should not match. For example, [^0-9] corresponds to any characterexcept a digit.

• ?: The question mark after a regular expression item makes it optional.Example: ab? matches the string a or ab.

• +: The plus operator makes it possible to repeat the regular expression item.Example: ab+ matches every strings ab. . .b.

• *: The star operator after a regular expression item signifies that the item is optional and can be repeated.Example: ab* matches every string ab. . .b and a.

• |: The pipe operator specifies an alternative.Example: a|b matches the character a or b.

• (. . .): The parenthesis groups expressions into sub-expressions.Example: (true)|(false) matches the string true or false.

• Regular Expression Items: The regular expression items are ASCII or magic characters (See table 17.1, page 82 for thecomplete list). If a magic character needs to be used as an ASCII character, it has to be escaped with a backslash (\).


Regular Expressions

Element Description Equivalence Reverse Element^ The caret marks the beginning of the string. For example, ^c: will only

match file names with an absolute path on the drive C:. If you wish tomatch a literal ^ you must escape it by writing \^.

$ The dollar marks the end of the string. For example \.cpp$ will matchevery C++ file name. If you wish to match a literal $ you must escape itby writing \$.

\< Match the start of a word.\> Match the end of a word.. Any character\ Escape character. Example: \. corresponds to the character ’.’ and not

any character\m Alphanumeric character [0-9a-zA-Z] \M

\a Alphabetic character [a-zA-Z] \A

\b Blank character [ t] \B

\c Control character \C

\d Digit [0-9] \D

\g Printable character except space [] \G

\l Lower-case character [a-z] \L

\p Printable character \P

\n Printable character except space or alphanumeric character \N

\s White-space characters (space, form-feed , newline , carriage return ,horizontal tab and vertical tab)

[ fnrtv] \S

\u Uppercase letter [A-Z] \U

\x Hexadecimal digits [0-9a-fA-F] \X

\w Alphanumeric character or underscore [0-9a-zA-Z_] \W

Table 17.1: Non ASCII Regular Expression Items


Library calls

Chapter

18Library calls

18.1 C and C++ Library

The CoverageScanner library is generated during instrumentation and enables the user

1. to initialize the measurement system and select the name of the .csexe file,

2. to install a handler which saves the execution report on exit,

3. in a unit test scenario, to pass information about the tests to the execution report, especially the name of the test, commentsand the test result,

4. to handle dynamically loaded libraries, and

5. to customize the output system for special needs.

The library functions are accessible from any instrumented file, without the need for an #include directive. When a file isinstrumented, a preprocessor symbol __COVERAGESCANNER__ is defined: It can be used to exclude calls to the library functionsfrom the compilation when instrumentation is switched off.

18.1.1 __coveragescanner_install()

Syntax:

void __coveragescanner_install(const char *appname)

i This function is not needed for gcc, clang or Microsoft® Visual Studio®.

Initialize the CoverageScanner library. It should be the first function call in the main() function. The parameter appnamecontains the name of the executable. It is used to set the name of the measurement file to <appname>.csmes.

__coveragescanner_install() registers an exit handler via atexit() in order to save the execution report whenever theapplication terminates. The handler calls the function __coveragescanner_save() to store the execution traces at the end of


C and C++ Library

the program. It is called when the program terminates regularly and also if the signals SIGABRT, SIGTERM, SIGFPE, SIGILL,SIGINT, SIGSEGV and SIGTERM are received.

Suggested usage:

int main(argc,char *argv[]){#ifdef __COVERAGESCANNER____coveragescanner_install(argv[0]);

#endif...

}

Platform dependencies With gcc, clang or Microsoft® Visual Studio®, CoverageScanner automatically installs a min-imal handler. It saves the execution in the file <program_name>.csmes on a normal exit of the application. Calling__coveragescanner_install() allows to set the file name of the execution report and also saves the report at an abnormalexit, like a crash or an interrupt by a signal.

__coveragescanner_install() is not available for a compiler if its profile parameter CUSTOM_SETUP is set to NONE (seeChapter H, page 214).

18.1.2 __coveragescanner_testname()

Syntax:

void __coveragescanner_testname(const char *name)

Set the name of the current test. It will be saved in the execution report and be displayed in the "Executions" window (seeChapter 12.1, page 48) when the .csmes file is loaded into CoverageBrowser.

All code that is executed until the end of the program or the next call of __coveragescanner_save() is considered part ofthe test name. If tests get the same name, CoverageBrowser disambiguates them by numbers, like “MyTest (2)”.

Test names may be hierarchical, with hierarchy levels separated by slashes: If a program defines two tests, “group/test1” and“group/test2”, CoverageBrowser displays them as two tests, “test1” and “test2” under a common heading, “group”.

i This and the following functions until __coveragescanner_save() are especially intended for the use withunit test frameworks (see Chapter 22, page 120).

18.1.3 __coveragescanner_teststate()

Syntax:

void __coveragescanner_teststate(const char *state)

Set the state of the current test. The parameter state may have the following values:

"PASSED":to indicate that the test was successful.


C and C++ Library

"FAILED":to indicate that the test was not successful.

"CHECK_MANUALLY":to indicate that it was not possible to determine whether the test was successful.

If the function is called twice, the first state is overwritten. The test state will be saved in the execution report and be displayedin the "Executions" window (see Chapter 12.1, page 48) when the .csmes file is loaded into CoverageBrowser.

18.1.4 __coveragescanner_add_html_comment()

Syntax:

void __coveragescanner_add_html_comment(const char *comment)

Set am HTML comment to the current execution. There is only one HTML comment active at any time, but it can be puttogether with several calls of __coveragescanner_add_html_comment(). In this case, the values of the various commentparameters are concatenated.

The full comment must follow the HTML syntax, but only the content of the <body> tag is used. A minimal commend mighttherefore be something like “<html><body>My comment</body></html>”.

18.1.5 __coveragescanner_clear_html_comment()

Syntax:

void __coveragescanner_clear_html_comment()

Remove the comment added by calls of the function __coveragescanner_add_html_comment().

18.1.6 __coveragescanner_save()

Syntax:

void __coveragescanner_save()

Save the execution report and reset the status of all instrumentations. The coverage counters for each line of code and theexecution status of the current test are reset, but the HTML comment stays unchanged.

18.1.7 __coveragescanner_clear()

Syntax:

void __coveragescanner_clear()

Reset the coverage counters for all lines of code.

This is useful in a unittest framework. After the framework has started, __coveragescanner_reset() can be called. Thenthe activity of the framework at startup does become part of the code coverage.


C and C++ Library

18.1.8 __coveragescanner_filename()

Syntax:

void __coveragescanner_filename(const char *name)

Set the file name of the execution report. The extension ’.csexe’ is added automatically. If name is NULL, the generation ofexecution reports is disabled.

In name may contain the following escape sequences:

%p: The process identifier, if it is provided by the platform, otherwise the empty string.

%w: The working directory. The value is computed when the CoverageBrowser library is initialised; it does not changeafterwards.

%t: The current time, or the empty string if it cannot be computed.

%d: The current date, or the empty string if it cannot be computed.

Platform dependencies The escape sequences are only available for a compiler if its profile parameter FILE_FORMAT_SPECIFIERis set to YES (see Chapter H, page 214).

Date and time may not be available on some embedded platforms.

18.1.9 __coveragescanner_register_library()

Syntax:

int __coveragescanner_register_library(const char *library_name)

Register a shared library that was loaded on demand. The function must be called after a library has been loaded with dlopen()(UNIX®) or LoadLibrary() (Microsoft® Windows). After it is registered, the code coverage of the library is saved in the samecoverage report file as the coverage of the main application.

It is possible to call __coveragescanner_register_library() more than once, but every call must be matched by acorresponding call of __coveragescanner_unregister_library(). For each library there is a use counter to keep track ofthe registrations.

Returned values:

0: Success.

-1: Error: library could not be loaded.

-2: Error: invalid library name.

1: Success, but the library is already registered. The use count of the library has been incremented.

2: Success, but the library is not instrumented.

Platform dependencies __coveragescanner_register_library() and __coveragescanner_unregister_library()are not available for a compiler if its profile parameter PLUGIN_REGISTRATION_API is set to NO (see Chapter H, page 214).


C and C++ Library

18.1.10 __coveragescanner_unregister_library()

Syntax:

int __coveragescanner_unregister_library(const char *library_name)

De-register a shared library that was loaded on demand. The function must be called before a library is unloaded withdlclose() (UNIX®) or CloseHandle() (Microsoft® Windows).

Returned values:

0: Success.

-1: Error: library was not registered.

-2: Error: invalid library name.

1: Success, but the library was not unregistered because its use count was not null.

2: Success, but the library is not instrumented.

18.1.11 __coveragescanner_set_custom_io()

Syntax:

void __coveragescanner_set_custom_io(char *(*csfgets)(char *s, int size, void *stream),int (*csfputs)(const char *s, void *stream),void *(*csfopenappend)(const char *path),void *(*csfopenread)(const char *path),void *(*csfopenwrite)(const char *path),int (*csfclose)(void *fp),int (*csremove)(const char *filename) )

Define your own set of I/O functions.

Parameters:

csfgets:Read at most size - 1 characters from stream and store them into the buffer pointed to by s. Reading stops after anEOF or a newline is read. If a newline is read, it is stored in the buffer. A ‘\0’ is stored after the last character in the buffer.

csfputs:Write the string s to stream, without its trailing ’\0’.

csfopenappend:Open the file path for appending text at the end.

csfopenread:Open the file path for reading.

csfopenwrite:Open the file path for writing.

csfclose:Flush and close the stream pointed to by fp.


C# Library

csremove:Remove the file filename from the file system.

By default, CoverageScanner writes the execution report (the .csexe file) to the file systems with help of common Clibrary functions. In some situations, e.g. with embedded systems, it is necessary to use other methods. For this reason,CoverageScanner provides __coveragescanner_set_custom_io(), which allows to replace the I/O functions used by__coveragescanner_save().

Default implementation:

char *csfgets(char *s, int size, void *stream) { return fgets(s,size,(FILE *)stream); }int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); }void *csfopenappend(const char *path) { return (void*)fopen(path,"a+"); }void *csfopenread(const char *path) { return (void*)fopen(path,"r"); }void *csfopenwrite(const char *path) { return (void*)fopen(path,"w"); }int csremove(const char *filename) { return remove(filename); }int csfclose(void *fp) { return fclose((FILE*)fp); }

Some examples are available in Appendix F, page 203.

18.2 C# Library

CoverageScanner library is generated during the link process and enables the user to:

1. Generate the execution report.

2. Give a name to the executed tests.

3. Install a signal handler which saved the execution report on exit.

Additionally CoverageScanner sets the preprocessor define __COVERAGESCANNER__ which excludes this function from anormal compilation.

18.2.1 CoverageScanner.__coveragescanner_init()

Syntax:

void CoverageScanner.__coveragescanner_init()

CoverageScanner.__coveragescanner_init() initialize explicitly the CoverageScanner library. Calling this function isonly necessary on C# DLL which are not calling the static module initializer automatically.

18.2.2 CoverageScanner.__coveragescanner_testname()

Syntax:

void CoverageScanner.__coveragescanner_testname(string name)


C# Library

CoverageScanner.__coveragescanner_testname() sets the name of the test which is currently being executed. It will besaved to the execution report and is displayed in the "Executions" window (see Chapter 12.1, page 48) when the loaded inCoverageBrowser.

18.2.3 CoverageScanner.__coveragescanner_teststate()

Syntax:

void CoverageScanner.__coveragescanner_teststate(string state)

CoverageScanner.__coveragescanner_teststate() sets the state of the test which is currently being executed. The stringparameter state can have the following values:

"PASSED":indicates that the test was successfully executed.

"FAILED":indicates that the test was not successfully passed.

"CHECK_MANUALLY":indicates that it was not possible to determinate if the test was successfully executed.

It will be saved to the execution report and is displayed in the "Executions" window (see Chapter 12.1, page 48) when theloaded in CoverageBrowser.

18.2.4 CoverageScanner.__coveragescanner_add_html_comment()

Syntax:

void __coveragescanner_add_html_comment(string comment)

__coveragescanner_add_html_comment() append a comment to the current execution. The comments needs to follow theHTML syntax but only the body is parsed. (The contains of the “<HEAD>” tag is ignored). This function can be called severaltimes, in this case the contains is appended.

18.2.5 CoverageScanner.__coveragescanner_clear_html_comment()

Syntax:

void __coveragescanner_clear_html_comment()

__coveragescanner_add_html_comment() clear comments added using void __coveragescanner_add_html_comment(string comment).

18.2.6 CoverageScanner.__coveragescanner_save()

Syntax:


C# Library

void CoverageScanner.__coveragescanner_save()

CoverageScanner.__coveragescanner_save() saves the execution report and resets the status of all instrumentations.

18.2.7 CoverageScanner.__coveragescanner_clear()

Syntax:

void CoverageScanner.__coveragescanner_clear()

CoverageScanner.__coveragescanner_clear() resets the status of all instrumentations.

18.2.8 CoverageScanner.__coveragescanner_filename()

Syntax:

void CoverageScanner.__coveragescanner_filename(string name)

this function sets the file name of the execution report. The extension ’.csexe’ is added automatically.

i Setting the execution report file name is necessary if the initialization function __coveragescanner_install()is not used.

18.2.9 CoverageScanner.__coveragescanner_set_custom_io()

Syntax:

void CoverageScanner.__coveragescanner_set_custom_io(__cs_fgets_delegate cs_fgets,__cs_fputs_delegate cs_fputs,__cs_fopenappend_delegate cs_fopenappend,__cs_fopenread_delegate cs_fopenread,__cs_fopenwrite_delegate cs_fopenwrite,__cs_fclose_delegate cs_fclose,__cs_remove_delegate cs_remove)

Parameters:

csfgets:Type:public delegate string __cs_fgets_delegate(System.IO.Stream stream)csfgets reads in at most one less than size characters from stream and stores them into the buffer pointed to by s.Reading stops after an EOF or a newline. If a newline is read, it is stored into the buffer. A ’\0’ is stored after the lastcharacter in the buffer.

csfputs:Type:public delegate void __cs_fputs_delegate(string s, System.IO.Stream stream)csfputs writes the string s to stream, without its trailing ’\0’.


C# Library

csfopenappend:Type:public delegate System.IO.Stream __cs_fopenappend_delegate(string path)csfopenappend open the file path for writing at the end.

csfopenread:Type:public delegate System.IO.Stream __cs_fopenread_delegate(string path)csfopenread open the file path for reading.

csfopenwrite:Type:public delegate System.IO.Stream __cs_fopenwrite_delegate(string path)csfopenwrite open the file path for writing.

csfclose:Type:public delegate void __cs_fclose_delegate(System.IO.Stream fp)The csfclose function will flush and close the stream pointed to by fp.

csremove:Type:public delegate void __cs_remove_delegate(string n)csremove removes filename from the file system.

CoverageScanner writes the execution report (.csexe file) into file using the common C# file IO functions. In some situations,for embedded systems for example, it is necessary to use an other way for recording it. For this reason, CoverageScannerprovides CoverageScanner.__coveragescanner_set_custom_io() which permits to replace the IO functions used byCoverageScanner.__coveragescanner_save().


Controlling the instrumentation during the compilation

Chapter

19Controlling the instrumentation during the compilation

CoverageScanner offers the possibility to modify the coverage settings of some source code parts during the compilation. Thispermits, for example, to exclude some source code from the coverage analysis.

19.1 Source Code Annotations

Source code annotations are specific comments which permits to validate some source code parts. Two kind of annotations areexisting:

Comments:comments are simple code comments which can be used for giving advices to the tester. They don’t have any impact onthe code coverage

Manual Validations:Manual validations permits to indicate that some lines of code should be considered as validated. Manual validationshave the same effect for the code coverage as executing the source lines.

Source code annotations can directly be edited with CoverageBrowser or generated by writing specific comments into thesource code.

19.1.1 Comments

To generate comments for a single source line it is only necessary to start a C or C++ comment with the keyword “coco:<comment>”.This will insert a comment for all instrumentations in the current line.

Example: In the following example, the comment “To test the failure, change the current working directory to a read-onlydirectory” will be added to the instrumentations of the line 4.

1 void write_log( const char *text )2 {3 FILE * f = fopen( "log.txt", "a+" );4 if ( f ) /* coco: To test the failure,


Source Code Annotations

5 change the current working6 directory to a read-only directory */7 {8 fputs( text, f ):9 fclose( f );

10 }11 }

Comments can also be generated for multiples source lines using the keywords “coco begin:<comment>” and “coco end”.

Example: In the following example, the comment “To test the failure, change the current working directory to a read-onlydirectory” will be added to the instrumentations of the lines of the function write_log().

1 void write_log( const char *text )2 {3 /* coco begin: To test the failure,4 change the current working5 directory to a read-only directory */6 FILE * f = fopen( "log.txt", "a+" );7 if ( f )8 {9 fputs( text, f ):

10 fclose( f );11 }12 /* coco end */13 }

19.1.2 Manual Validation

Manual validated instrumentation can be generated using a comment starting with the keyword “coco validated:<comment>”.For validated a code area, “coco begin validated:<comment>” and “coco end” keywords can be used. In order to becompatible with PureCov adjustments, CoverageScanner also recognizes “purecov:” line comments with the keywords“inspected” or “tested”. (alternatively “begin inspected”/“begin tested” and “end inspected”/“end tested” formulti-line annotations)

Example: In the following example, the comment “This function does not need to be tested, only used for debugging.” will beadded to the instrumentations of the function write_log().

1 void write_log( const char *text )2 {3 // coco begin validated: This function does not need to be tested, only used for debugging.4 FILE * f = fopen( "log.txt", "a+" );5 if ( f )6 {7 fputs( text, f ):8 fclose( f );9 }

10 // coco end11 }


C and C++ Pragma

19.2 C and C++ Pragma

CoverageScanner defines C and C++ pragmas which permit to control during the compilation the generation of the instrumenta-tion. All control commands are handled in a stack. The pragmas have the following syntax:#pragma CoverageScanner ( string )or: __pragma ( CoverageScanner ( string ) )or: _Pragma ( CoverageScanner ( string ) )

List of supported pragmas:

#pragma CoverageScanner(cov-on) :Enables code coverage instrumentation.

#pragma CoverageScanner(cov-off) :Disables code coverage instrumentation.

#pragma CoverageScanner(cov-hit) :Instruments using code coverage hit.

#pragma CoverageScanner(cov-count) :Instruments using code coverage count.

#pragma CoverageScanner(cov-branch) :Select the code coverage branch as instrumentation method.

#pragma CoverageScanner(cov-decision) :Select the code coverage decision as instrumentation method.

#pragma CoverageScanner(cov-condition) :Select the code coverage condition as instrumentation method.

#pragma CoverageScanner(cov-partial-instrumentation) :Partial instrumentation of conditions and decisions.

#pragma CoverageScanner(cov-full-instrumentation) :Full instrumentation of conditions and decisions.

#pragma CoverageScanner(cov-assignment-on) :Instrument Boolean expressions in assignments.

#pragma CoverageScanner(cov-assignment-off) :Stop instrumenting Boolean expressions in assignments.

#pragma CoverageScanner(pop) :Restore the instrumentation option before the call of the last CoverageScanner pragma.

19.3 C# regions

CoverageScanner defines C# extensions of the #region keyword which permits to control during the compilation the generationof the instrumentation. The #region extension have the following syntax:#region CoverageScanner ( string )

The keyword #endregion restores the generation options.

List of supported regions:


C# regions

#region CoverageScanner(cov-on) :Enables code coverage instrumentation.

#region CoverageScanner(cov-off) :Disables code coverage instrumentation.

#region CoverageScanner(cov-hit) :Instruments using code coverage hit.

#region CoverageScanner(cov-count) :Instruments using code coverage count.

#region CoverageScanner(cov-branch) :Select the code coverage branch as instrumentation method.

#region CoverageScanner(cov-decision) :Select the code coverage decision as instrumentation method.

#region CoverageScanner(cov-condition) :Select the code coverage condition as instrumentation method.

#region CoverageScanner(cov-partial-instrumentation) :Partial instrumentation of conditions and decisions.

#region CoverageScanner(cov-full-instrumentation) :Full instrumentation of conditions and decisions.


IDE Support

Chapter

20Integrated Development Environment Support

20.1 GNU Makefile

Mostly, in makefiles, the C and C++ compiler and the linker are defined using the environment variable CC, CXX and LINK.This can be substituted by CoverageScanner by setting CC, CXX and LINK in the command arguments of make. Exam-ple: make LINK=csg++ CXX=csg++ CC=csgcc

20.2 CygWin

To install CoverageScanner compiler wrapper for GCC and G++ on CygWin, proceed as follows:

1. Open the “Build Environment Selection” application. (%SQUISHCOCO%\toolselector.exe)

2. Select the item “CygWin - www.cygwin.com”.

3. Click on “Install CygWin Support”. The build environment selection dialog displays the list of generated compilerwrapper.

Open then the CygWin console and compile your application using csgcc (or csg++) instead of gcc (or g++).

20.3 Scratchbox

If Squish Coco is installed on the root file system1, a compiler wrapper is created for each compiler supported by Scratchbox.To invoke CoverageScanner, prepend cs to the name of the cross-compiler.

1Installation option:"1) Installation on the local machine (need to be root)"


CMake

20.4 Kitware CMake

CMake is a platform independent build tool from Kitware which can be downloaded from http://www.cmake.org.

20.4.1 Coverage Configuration

Create a new CMake COVERAGE configuration by adding in CMakeLists.txt following lines:

SET(COVERAGE_FLAGS "--cs-on --cs-count")SET( CMAKE_CXX_FLAGS_COVERAGE

"${CMAKE_C_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING"Flags used by the C++ compiler during coverage builds."FORCE )

SET( CMAKE_C_FLAGS_COVERAGE"${CMAKE_CXX_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING"Flags used by the C compiler during coverage builds."FORCE )

SET( CMAKE_EXE_LINKER_FLAGS_COVERAGE"${CMAKE_EXE_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING"Flags used for linking binaries during coverage builds."FORCE )

SET( CMAKE_SHARED_LINKER_FLAGS_COVERAGE"${CMAKE_SHARED_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING"Flags used by the shared libraries linker during coverage builds."FORCE )

SET( CMAKE_STATIC_LINKER_FLAGS_COVERAGE"${CMAKE_STATIC_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING"Flags used by the static libraries linker during coverage builds."FORCE )

MARK_AS_ADVANCED(CMAKE_CXX_FLAGS_COVERAGECMAKE_C_FLAGS_COVERAGECMAKE_EXE_LINKER_FLAGS_COVERAGECMAKE_SHARED_LINKER_FLAGS_COVERAGECMAKE_STATIC_LINKER_FLAGS_COVERAGECOMPILE_DEFINITIONS_COVERAGE

)

Use the variable COVERAGE_FLAGS to specify CoverageScanner command line options.

20.4.2 Microsoft® Visual Studio®

Proceed as follows:

1. Create toolchain definition file called cl.cmake which set the path of the compiler and linker to CoverageScannerwrapper.Example:

# this one is importantSET(CMAKE_SYSTEM_NAME Windows)


http://www.cmake.org

CMake

# specify the cross compilerFILE(TO_CMAKE_PATH "$ENV{SQUISHCOCO}/visualstudio" SQUISHCOCO)SET(CMAKE_C_COMPILER ${SQUISHCOCO}/cl.exe

CACHE FILEPATH "CoverageScanner wrapper" FORCE)SET(CMAKE_CXX_COMPILER ${SQUISHCOCO}/cl.exe

CACHE FILEPATH "CoverageScanner wrapper" FORCE)SET(CMAKE_LINKER ${SQUISHCOCO}/link.exe

CACHE FILEPATH "CoverageScanner wrapper" FORCE)

2. Create a Makefile project and set the toolchain to CoverageScanner wrapper and the build mode to COVERAGE.Example:

cmake.exe -DCMAKE_TOOLCHAIN_FILE=cl.cmake -DCMAKE_BUILD_TYPE=COVERAGE-G "NMake Makefiles" <path_of_cmake_project>

3. Build the project using nmake.

20.4.3 GNU GCC

1. Create toolchain definition file called gcc.cmake which set the path of the compiler and linker to CoverageScanner.Example:

find_program(CODE_COVERAGE_GCC csgcc)find_program(CODE_COVERAGE_GXX csg++)find_program(CODE_COVERAGE_AR csar)

# specify the cross compilerSET(CMAKE_C_COMPILER "${CODE_COVERAGE_GCC}"

CACHE FILEPATH "CoverageScanner wrapper" FORCE)SET(CMAKE_CXX_COMPILER "${CODE_COVERAGE_GXX}"

CACHE FILEPATH "CoverageScanner wrapper" FORCE)SET(CMAKE_LINKER "${CODE_COVERAGE_GXX}"

CACHE FILEPATH "CoverageScanner wrapper" FORCE)SET(CMAKE_AR "${CODE_COVERAGE_AR}"

CACHE FILEPATH "CoverageScanner wrapper" FORCE)

2. Create a Makefile project and set the toolchain to CoverageScanner and the build mode to COVERAGE.Example:

cmake -DCMAKE_TOOLCHAIN_FILE=gcc.cmake -DCMAKE_BUILD_TYPE=COVERAGE-G "Unix Makefiles" <path_of_cmake_project>

3. Build the project using make.


Qt

20.5 Qt Library

20.5.1 Qt Creator

After creating a project using Qt Creator, to automate instrumenting it follow these steps:

• Switch to the Projects mode using the corresponding button in the mode pan located on the left side of the IDE’s interface.

• In the Build Environment section, expand the System Environment pane.

• Modify the PATH variable prepending the path of the wrapper subfolder of CoverageScanner.

• Add a COVERAGESCANNER_ARGS variable and set its value to --cs-on (plus any other arguments you want to pass toCoverageScanner).

• In the Build Steps section, click on Add Build Step then choose Custom Process Step. This step should execute the unittest which code coverage you’re interested in, so choose its directory as the Working Directory and choose its executableas the Command (and optionally set arguments to be passed to the executable).

• Add another Custom Process Step and for this one set the Working Directory as the unit test’s directory and set SquishCoco’s cmcsexeimport tool’s absolute path as the Command. As for the arguments, make sure the csmes and csexe filesare included, for example --title="Execution" -m unit_test.csmes -P -e unit_test.csexe.

– Remember to replace the word unit_test in unit_test.csexe and unit_test.csmes with the actual name ofyour unit test.

• The final Custom Process Step that needs to be added is the one for the cmreport tool, in case report generation waswanted. Again set the Wokring Directory as the unit test’s, and set cmreport’s absolute path as the Command then setthe arguments, for example:

--title="Execution" -m unit_test.csmes --html=report

– Remember again to change unit_test.csmes according to your unit test’s name.

• Note that the last two steps can be repeated for each unit test in the project, if necessary.

Now, everytime you click on Build (not Run), the project will be built and instrumented and optionally, a report would begenerated. Note that in case the project was built before this configuration was performed, it will be required to execute theClean build step before building so that the coverage measurment data gets generated. Also keep in mind that when one of thebuild steps exits with a non-zero exit status code (i.e when there was an error), Qt Creator stops the build and displays whatevererror message the executable that exited has printed out instead of continuing the build, this means that in case some unit testfails, instead of cmcsexeimport running and importing the execution as a failed execution, cmcsexeimport doesn’t run at alluntil the test case failure was fixed.

20.5.2 qmake

qmake is a tool to generate a Makefile in a platform-independent way, using as specification a so-called “project file”. Bydefault, qmake chooses the programs that are used for compilation. We can however use the Squish Coco wrappers by settingsome of qmake’s variables to new values.

In simple cases, we can do this on the command line:

$ qmake QMAKE_LINK=cslink QMAKE_CXX=cscl QMAKE_CC=cscl


SCons

Here we redefine the variables QMAKE_CC, QMAKE_CXX and QMAKE_LINK: They contain the name of the C compiler, the C++compiler and the linker that are used in the Makefiles generated by qmake.

In more complex cases it is however better to create a new scope inside the project file. (A scope is a region in a qmake projectfile that can be activated on demand.) The following listing shows a template for such a scope, named CoverageScanner,which should be sufficient for most most projects.

CodeCoverage {COVERAGE_OPTIONS =

QMAKE_CFLAGS += $$COVERAGE_OPTIONSQMAKE_CXXFLAGS += $$COVERAGE_OPTIONSQMAKE_LFLAGS += $$COVERAGE_OPTIONS

QMAKE_CC=cs$$QMAKE_CCQMAKE_CXX=cs$$QMAKE_CXXQMAKE_LINK=cs$$QMAKE_LINKQMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIBQMAKE_AR=cs$$QMAKE_AR

}

Here we also set the variables QMAKE_LINK_SHLIB and QMAKE_AR, which contain the names of the command to link sharedlibraries and that to generate archives. Furthermore, can use COVERAGE_OPTIONS to set coveragescanner commandlineoptions (see Chapter 17.2, page 75) to customize the project. An empty value for COVERAGE_OPTIONS will also work andresults in a default instrumentation.

If we have a project file that contains such a scope, we can create a Makefile that generates instrumented code with thecommand:

$ qmake CONFIG+=CodeCoverage

20.5.3 moc

The Meta-Object Compiler moc adds automatically new methods to each classes derived from QObject. For example, thetranslation function tr, the source code for all signals, the cast operator qt_cast, . . . In order to instrument the code using theQt Library and not the library itself, CoverageScanner provides the command line options --cs-qt4 for Qt4 (resp. --cs-qt3for Qt3).

In this case:

• Q_OBJECT macro is no more instrumented.

• All signal are instrumented in order to track their emission.

• The glue code necessary for the signal/slot mechanism is not instrumented.

• The code of Q_FOREACH macro is not instrumented on Qt4.

20.6 SCons

To use Squish Coco with SCons, proceed as followings:




ARM Keil

• Prepend the path of CoverageScanner’s wrapper (csgcc, cscl, . . . ) to the PATH environment variable. The PATHenvironment variable should be set to be able to execute CoverageScanner wrappers .

• Set CC, AR and LINK variable to CoverageScanner corresponding wrapper.Example: when using Microsoft® Visual Studio® compiler, set CC to cscl.exe, set LINK to cslink.exe and set AR tocslib.exe.

! Do no use absolute file paths to the compiler wrapper since that some versions of SCons do not handleproperly spaces in file names.

• Add additional code coverage settings to the variables CCFLAGS, ARFLAGS and LINKFLAGS to exclude for example filesfrom the instrumentation.

Here is a code snippet which can be used for Microsoft® Visual Studio® command line tools:

import osfrom os.path import pathsep

env = Environment()

# Add the path of Squish Coco compiler wrapperenv[ ’ENV’ ][ ’PATH’ ] = os.environ[ ’SQUISHCOCO’ ] + pathsep + env[ ’ENV’ ][ ’PATH’ ]# TEMP variable need to be definedenv[ ’ENV’ ][ ’TEMP’ ] = os.environ[ ’TEMP’ ]

# Set the compiler to Squish Coco wrappersenv[ ’CC’ ] = ’cs’ + env[ ’CC’ ] ;env[ ’AR’ ] = ’cs’ + env[ ’AR’ ] ;env[ ’LINK’ ] = ’cs’ + env[ ’LINK’ ] ;

# Code coverage settingscoverageflags = [ ’--cs-count’ ]env[ ’CCFLAGS’ ] = env[ ’CCFLAGS’ ] + coverageflags ;env[ ’ARFLAGS’ ] = env[ ’ARFLAGS’ ] + coverageflags ;env[ ’LINKFLAGS’ ] = env[ ’LINKFLAGS’ ] + coverageflags ;

20.7 ARM® Keil® µVision

To enable the code coverage analysis it is first necessary to compile the project with Squish Coco compiler wrapper for ARMand enable the code coverage analysis during the compilation.

Processed as followings:

1. Select the Squish Coco tool chain instead of the original ARM toolchain.

(a) On the ARM® Keil® µVision IDE click on "ProjectÕManageÕComponents, Environments, Books...".

(b) Click on "Folders/Extensions", and replace .\ARMCC\bin through .\SquishCoco\bin in the field"ARMCC Folder:".


ARM Keil

Figure 20.1: Installation of CoverageScanner on ARM® Keil® µVision: Setting the path of CoverageScanner.

2. Activate the code coverage analysis during the compilation

(a) Click on "ProjectÕOption for Target..."

(b) On the "C/C++" tab, add --cs-on into the "Misc Controls" field.

Figure 20.2: Activating the code coverage analysis for an ARM® compiler

(c) On the "Linker" tab, add --cs-on into the "Misc Controls" field.


Visual DSP

Figure 20.3: Activating the code coverage analysis for an ARM® linker

20.8 VisualDSP®

To enable the code coverage analysis it is first necessary to compile the project with Squish Coco compiler wrapper and enablethe code coverage analysis during the compilation in the VisualDSP® configuration.

Processed as followings:

1. Click on "ProjectÕProject Options..."

2. On the "Compile" tab, add --cs-on into the "Additional options" field.


Visual DSP

Figure 20.4: Activating the code coverage analysis for the compiler

3. On the "Link" tab, add --cs-on into the "Additional options" field.

Figure 20.5: Activating the code coverage analysis for the linker

4. Rebuild your project.

Like for all embedded targets, it is necessary to provide a dedicated I/O since that in most of the cases no file systems areavailable for storing the code coverage information. Also since that embedded applications are generally not exiting, it is alsonecessary to implement a event handler which saves the execution report upon the reception of a specific trigger.

The simulator emulates the support of a file system. To save the coverage report on the current build directory, it is onlynecessary to register in the first lines of the main() a custom file I/O which uses the standard C file API.

Example:

1 #ifdef __COVERAGESCANNER__



2 static int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); }3 static void *csfopenappend(const char *path) { return (void*)fopen(path,"a+"); }4 static int csfclose(void *fp) { return fclose((FILE*)fp); }5 #endif6

7 int main()8 {9 #ifdef __COVERAGESCANNER__

10 __coveragescanner_set_custom_io( NULL,11 csfputs,12 csfopenappend,13 NULL,14 NULL,15 csfclose,16 NULL);17 #endif18 ....19 }

To record a coverage report when a specific trigger occurs, add the following source code lines in its handler:

1 #ifdef __COVERAGESCANNER__

2 __coveragescanner_save();3 __coveragescanner_clear();4 #endif

20.9 Microsoft® Visual Studio®

Squish Coco provides a wrapper for link.exe and cl.exe located on the %SQUISHCOCO%\visualstudio directory. It behavesexactly like the corresponding Microsoft® wrapper except that the code coverage analysis becomes activated when the option--cs-on is added to the command arguments. These wrappers call the Microsoft® tools for compilation or for linkage.

20.9.1 Microsoft® Visual Studio® .NET C# Compiler

To activate the instrumentation of C# source code, it is only necessary to add the define COVERAGESCANNER_COVERAGE_ON inthe properties of the Microsoft® Visual Studio® .NET project. Other defines can be appended to select additional instrumentationoptions. The full list can be found on the chapter 17.3, page 81.

20.9.2 Microsoft® Visual Studio® .NET C and C++ Compiler

To use Squish Coco with Microsoft® Visual Studio® .NET proceed as follows:

1. Add the location of the CoverageScanner wrappers to the first position in the VC++ Directories.For Microsoft® Visual Studio® 2005 or 2008:



(a) Start Microsoft® Visual Studio® 2005 or 2008.

(b) Open the option dialog: click on "ToolsÕPreferences...".

(c) Select the item "ProjectsÕVC++ Directories".

(d) Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of directories. (see Figure 20.6,page 106)

Add the path of the compiler

wrapper for Visual Studio

to the first position

Figure 20.6: Installation of CoverageScanner on Microsoft® Visual Studio® 2005 and 2008: Setting the path of CoverageScanner.

For Microsoft® Visual Studio® 2010:

(a) Start Microsoft® Visual Studio® 2010.

(b) Open a C++ project.

(c) Open the project properties using the context menu of the laded project.

(d) Select the item "Configuration PropertiesÕVC++ Directories".

(e) Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of executable directories. (seeFigure 20.7, page 106)




Figure 20.7: Installation of CoverageScanner on Microsoft® Visual Studio® 2010: Setting the path of CoverageScanner.

2. To activate code coverage analysis:

(a) Open a Visual C or C++ project.



(b) Edit the project settings (click on "ProjectÕProperties").

(c) Add to the option --cs-on to the additional command line arguments of the C or C++ compiler and linker. (seeFigure 20.8, page 107)

(d) In the additional arguments of the linker, add the --cs-libgen which permits to specify which library should beused for the generation of the CoverageScanner library. The table 20.1, page 107 contains the list of recommendedsettings.

(e) For Microsoft® Windows CE applications, append to the linker arguments the command line option--cs-architecture which permits to specify the target platform. The table 20.2, page 107 contains the listof available architectures.

Library Library File Command line optionSingle Threaded LIBC.LIB --cs-libgen=/ML

Static MultiThread LIBCMT.LIB --cs-libgen=/MT

Dynamic Link (DLL) LIBCRT.LIB --cs-libgen=/MD

Debug Single Threaded LIBCD.LIB --cs-libgen=/MLd

Debug Static MultiThread LIBCMTD.LIB --cs-libgen=/MTd

Debug Dynamic Link (DLL) LIBCRTD.LIB --cs-libgen=/MDd

Table 20.1: CoverageScanner library settings for Microsoft® Visual Studio®

Targeted Architecture Command line optionARM Microprocessor --cs-architecture=ARM

ARM Microprocessor (Thumb code) --cs-architecture=THUMB

x86 Microprocessor --cs-architecture=IX86

MIPS16 Microprocessor --cs-architecture=MIPS16

MIPS Microprocessor --cs-architecture=MIPS

MIPS Microprocessor with FPU --cs-architecture=MIPSFPU

SH3 Microprocessor with FPU --cs-architecture=SH3

SH4 Microprocessor with FPU --cs-architecture=SH4

Table 20.2: List of target architectures

line option'--cs-on' here

Add the command

Figure 20.8: Activation of the instrumentation under Visual Studio® .NET.




Add the command

Figure 20.9: Activation of the instrumentation under Visual Studio® .NET.

20.9.3 Squish Coco Add-In for Microsoft® Visual Studio®

Squish Coco provides for Microsoft® Visual Studio® 2005 up to Microsoft® Visual Studio® 2013 an Add-In which doesautomatically the steps described in the section 20.9.2, page 105. The Add-In itself is normally not mandatory to build aninstrumented application.

Two different workflow are supported:

• Enabling/disabling the code coverage analysis on an existing C++ or C# configuration.This method permits to quickly enable the code coverage analysis on an existing build mode. Unfortunately, a completerebuild is then necessary.

• Creating a new configuration which provides the coverage analysis. The Add-In duplicate a specified configuration andapply the code coverage settings on it.This method is more convenient for a developer since that it permits to quickly switch from the Debug/Release tothe Coverage build mode without rebuilding the whole application. Also once the code coverage settings are appliedit is possible to save it permanently in the new configuration. Unfortunately, this method may require some manualadjustments on some projects which are using fixed encoded paths.

To use it:

1. Open a C++ or a C# project.

2. Click on “ToolsÕCode Coverage Build Mode...”.

3. Select the base configuration and the project which should be compiled with code coverage support.

4. And finally:

• Select “Modify” and click on “Enable code coverage” to alter the current build mode and add the code coveragesupport. “Disable code coverage” permits removes all CoverageScanner command line options.

• Select “New” and click on “Create new configuration” to generate a new build mode with code coveragesupport.



Coverage options

Project and Configuration

used as template

the inline code is instrumented

Include paths in which

Creation of a new configuration

Enabling/disabling the coverageanalysis on the current configuration

Figure 20.10: Microsoft® Visual Studio® Add-In

i For Microsoft® Visual Studio® 2010, it might be necessary to add manually the path Squish Coco executables tothe search path. See section 20.9.2, page 105 for more the detailed procedure.

20.9.4 Microsoft® Visual C++ Express

To use Squish Coco with Microsoft® Visual C++ Express proceed as follows:

1. Add the location of the CoverageScanner wrappers to the first position in the VC++ Directories.

(a) Start Microsoft® Visual C++ Express.(b) Open the option dialog: click on "ToolsÕPreferences...".(c) Select the item "ProjectsÕVC++ Directories".(d) Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of directories. (see Figure 20.11,

page 109)




Figure 20.11: Installation of CoverageScanner on Visual C++ Express: Setting the path of CoverageScanner.



2. The activation of the code coverage analysis is similar to Microsoft® Visual Studio® .NET. (see Chapter 20.9.2, page 105)

20.9.5 Microsoft® Visual Studio® 6.0

To use Squish Coco with Microsoft® Visual Studio® 6.0 proceed as follows:

1. Add the location of the CoverageScanner wrappers to the first position in the executable directories.

(a) Start Microsoft® Visual Studio® 6.0.


(c) Select the item "Directories".

(d) Select "Executable files" in the combobox "Show directories for:".

(e) Add the path of the directory visualstudio of the Squish Coco installation2 to the first position in thelist of directories. (example: if Squish Coco is installed on c:\programme\SquishCoco, add the pathc:\programme\SquishCoco\visualstudio, (see Figure 20.12, page 110))




Figure 20.12: Installation of CoverageScanner on Visual Studio® 6.0: Setting the path of CoverageScanner.

2. To activate the code coverage analysis:



(c) Add the option --cs-on to the additional command line arguments of the C or C++ compiler and linker. (seeFigure 20.13, page 111)

(d) In the additional arguments of the linker, add the --cs-libgen which permits to specify which library should beused for the generation of the CoverageScanner library. The table 20.1, page 107 contains the list of recommandedsettings.

2Microsoft® Visual Studio® 6.0 does not handle system variables in the path list. So the %SQUISHCOCO% variable needs to be expanded.


eMbedded Visual C++


Add the command

Figure 20.13: Activation of the instrumentation under Visual Studio® 6.0.


Add the command

Figure 20.14: Activation of the instrumentation under Visual Studio® 6.0.

20.10 Microsoft® eMbedded Visual C++®

To use Squish Coco with Microsoft® eMbedded Visual C++®proceed as follows:

1. Add the location of the CoverageScanner wrappers to the first position in the executable directories.

(a) Start Microsoft® eMbedded Visual C++®.


(c) Select the item "Directories".

(d) Select "Executable files" in the combobox "Show directories for:".

(e) Select "Platform" and the targeted "CPUs".

(f) Add the path of the directory WinCE of the Squish Coco installation3 to the first position in the listof directories. (example: if Squish Coco is installed on c:\programme\SquishCoco, add the pathc:\programme\SquishCoco\WinCE, (see Figure 20.15, page 112))

3Microsoft® eMbedded Visual C++®does not handle system variables in the path list. So the %SQUISHCOCO% variable needs to be expanded.


eMbedded Visual C++



wrapper for Windows CE

Select the Platform and the CPU

Figure 20.15: Installation of CoverageScanner on eMbedded Visual C++®: Setting the path of CoverageScanner.

2. To activate the code coverage analysis:



(c) Add the option --cs-on to the additional command line arguments of the C and C++ compiler and linker. (seeFigure 20.16, page 112)


Add the command

Figure 20.16: Activation of the instrumentation under eMbedded Visual C++®.


Add the command

Figure 20.17: Activation of the instrumentation under eMbedded Visual C++®.


Eclipse

20.11 Eclipse™ IDE for C/C++

Eclipse™ IDE for C/C++ rely on a GNU-GCC command line compiler. Before trying to compile a project with coveragesupport, it is necessary to create a compiler wrapper ((see Chapter G.4, page 213)). To compile a project with code coveragesupport, proceed as follows:

1. Start Eclipse™.

2. Load a C or C++ project.

3. Open the property window (ProjectÕ Properties)

4. Click on "C/C++ Build/Settings"

5. Create a new configuration by clicking on "Manage Configurations..." and select it.

6. Click on "Tools Settings" tab.

7. Click on "GCC C++ Compiler" and prepend cs to the name of the compiler.

8. Click on "GCC C Compiler" and prepend cs to the name of the compiler.

9. Click on "C++ Linker" and prepend cs to the name of the linker.

compiler/linkerexecutable name

Append 'cs' to the

Figure 20.18: Eclipse™ settings

20.12 Apple® Xcode

To use Squish Coco with Apple® Xcode proceed as follows: To activate the code coverage analysis:


Xcode

1. Open a terminal window and set the CPLUSPLUS, LDPLUSPLUS, LD and CC to CoverageScanner compiler wrapper. Thepath of native compiler (clang, clang++, gcc or g++) need to be present in the PATH environment variable. Start Xcodeusing the open command.

If GCC is used as compiler:

SQUISHCOCO=/Applications/SquishCoco/wrapperexport CC=$SQUISHCOCO/gccexport LD=$SQUISHCOCO/gccexport CPLUSPLUS=$SQUISHCOCO/g++export LDPLUSPLUS=$SQUISHCOCO/g++

open /Developer/Applications/Xcode.app

If clang is used as compiler:

SQUISHCOCO=/Applications/SquishCoco/wrapperexport CC=$SQUISHCOCO/clangexport LD=$SQUISHCOCO/clangexport CPLUSPLUS=$SQUISHCOCO/clang++export LDPLUSPLUS=$SQUISHCOCO/clang++export XCODE_TOOLCHAIN_DIR=/Applications/Xcode.app/Contents/Developer/Toolchainsexport XCODE_TOOLCHAIN=$XCODE_TOOLCHAIN_DIR/XcodeDefault.xctoolchain/usr/bin/

export PATH=$XCODE_TOOLCHAIN:$PATH

open /Applications/Xcode.app

2. Open a Xcode C or C++ project.

3. Edit the project settings (click on "ProjectÕEdit Project Settings").

4. Add the option --cs-on to the additional command line arguments of the C and C++ compiler (fields Other C Flagsand Other C++ Flags) and linker (field Other Linker Flags). (see Figure 20.19, page 115)

5. Disable the usage of the precomiled header: Open the settings of the active target ("ProjectÕEdit Active Target")and remove the contains of Prefix Header.


Xcode

Add the compiler

command line option'--cs-on' here

command line option'--cs-on' here

Add the linker

Figure 20.19: Activation of the instrumentation under Apple® Xcode.


Code Coverage of Libraries

Chapter

21Code Coverage of Libraries

21.1 Code Coverage of Static/Shared Libraries and DLL

During the linking operation, CoverageScanner includes all instrumentations of the shared libraries (if these are compiled withCoverageScanner). CoverageBrowser displays the code coverage of the complete application (executable and its libraries) inone view.

iTo get an analysis of the code coverage of a library only, it is necessary to compile the main application andexclude its sources from the code coverage (by adding in the command line --cs-exclude-file-regex=.*for example.)

21.2 Code Coverage of Plugins/Manually Loaded Shared Libraries

Libraries loaded dynamically can also be instrumented, but it is necessary to handle the generation of the execution report inthe main application or in the plugin code itself.

21.2.1 Generating Code Coverage Information directly from the Main Application

Handling the plugins into the main application can be easily performed using the register/unregister mechanism of theCoverageScanner API: It is just necessary to call __coveragescanner_register_library() after loading a library and call__coveragescanner_unregister_library() just before unloading it.

Example:

#include <stdio.h>#include <stdlib.h>#include <dlfcn.h>


Code Coverage of Plugins/Manually Loaded Shared Libraries

intmain(int argc, char **argv){

void *handle;double (*cosine)(double);char *error;

handle = dlopen("libm.so", RTLD_LAZY);#ifdef __COVERAGESCANNER__

__coveragescanner_register_library("libm.so");#endif

if (!handle) {fprintf(stderr, "%s\n", dlerror());exit(EXIT_FAILURE);

}

dlerror(); /* Clear any existing error */

/* Writing: cosine = (double (*)(double)) dlsym(handle, "cos");would seem more natural, but the C99 standard leavescasting from "void *" to a function pointer undefined.The assignment used below is the POSIX.1-2003 (TechnicalCorrigendum 1) workaround; see the Rationale for thePOSIX specification of dlsym(). */

*(void **) (&cosine) = dlsym(handle, "cos");

if ((error = dlerror()) != NULL) {fprintf(stderr, "%s\n", error);exit(EXIT_FAILURE);

}

printf("%f\n", (*cosine)(2.0));#ifdef __COVERAGESCANNER__

__coveragescanner_unregister_library("libm.so");#endif

dlclose(handle);exit(EXIT_SUCCESS);

}

i Calling __coveragescanner_register_library() or __coveragescanner_unregister_library() onan non-instrumented library is allowed.

21.2.2 Generating Code Coverage Information directly from the Plugin

CoverageScanner cannot handle the instrumentation of plugins (i.e. shared libraries loaded manually) during the linking phase.In this case, the library must initialize and store itself the executions.Therefore, the shared library needs to call __coveragescanner_filename() to set the name of the execution file during itsinitialization and __coveragescanner_save() to save the instrumentations when its becomes unloaded.



21.2.2.1 Code Coverage of Plugins Generated with Microsoft® Visual Studio®

The function DllMain is called on the initialization and the termination of the DLL generated using Microsoft® VisualStudio®. When the ’reason’ field is equal to DLL_PROCESS_ATTACH the function __coveragescanner_filename() shouldbe called. To save the measures on exit the function __coveragescanner_save() shall be called when ’reason’ isDLL_PROCESS_DETACH.

Example:

extern "C"BOOL WINAPI DllMain(HINSTANCE hInstance,

DWORD dwReason,LPVOID /*lpReserved*/)

{switch( dwReason ){case DLL_PROCESS_ATTACH:


/* Initialization of the CoverageScanner library. *//* Replace "mylib" with your filename without extension */__coveragescanner_filename("mylib");

#endif...

break;case DLL_PROCESS_DETACH:...


/* Saves the execution report */__coveragescanner_save();

#endifbreak;

}return TRUE;

}

21.2.2.2 Code Coverage of Plugins Generated with GNU gcc

The GNU compiler offers 2 attributes which permits to execute a function when a library becomes loaded or unloaded:

__attribute__ ((constructor)) my_init(void);:This attribute permits to call a function when the library is loaded. Call the function __coveragescanner_filename()in the custom initialization function of the library.

__attribute__ ((destructor)) my_fini(void);:This attribute permits to call a function when the library is unloaded. Call the function __coveragescanner_save() onthe termination of the library.

Example:

static void plugin_load(void) __attribute__ ((constructor)) ;static void plugin_unload(void) __attribute__ ((destructor)) ;

static void plugin_load(void){




/* Initialization of the CoverageScanner library. *//* Replace "mylib" with your filename without extension */__coveragescanner_filename("mylib");

#endif...

}

static void plugin_unload(void){

...#ifdef __COVERAGESCANNER__

/* Saves the execution report */__coveragescanner_save();

#endif}


Test Suites and Squish Coco

Chapter

22Test Suites and Squish Coco

22.1 Execution Comment, Name And Status

Test suites are able to provide to CoverageBrowser the name of the test, a comment in HTML format and also the status of theirexecution (passed or failed). Two methods are possible:

• Using the functions __coveragescanner_testname(), __coveragescanner_add_html_comment() and__coveragescanner_teststate() can be used to specify the test name, its comment and its status. This isparticularly useful for unit tests in which the test framework is compiled into the application to test.(see Chapter 18.1.2,page 84)

• Writing directly into the .csexe file this information. (see below)

The .csexe file is a simple text file on which is it possible to append additional lines to extend the information provided toCoverageBrowser.

To set the name of a test simply add before executing it the following line: §name of the test\nor*name of the test\nThe character * or § must be the first character of the line. The name of the test is placed directly after and a is terminated witha carriage return.

To set the status of a test simply add after executing it the following line: !status\nThe character ! must be the first character of the line. The status of the test is placed directly after and a is terminated with acarriage return. The status can be one of the following strings:

"PASSED":indicates that the test was successfully executed.

"FAILED":indicates that the test was not successfully passed.

"CHECK_MANUALLY":indicates that it was not possible to determinate if the test was successfully executed.

To append an execution comment, insert the contains of a HTML document just after the execution of the application.

Example: the following batch file execute the test First Test and set the status of the execution to CHECK_MANUALLY.


Controlling the generation of the execution report

echo *First Test >> myapp.csexemyappecho "<HTML><BODY>Execution of myapp</BODY></HTML>" >> myapp.csexeecho !CHECK_MANUALLY >> myapp.csexe

22.2 Controlling the generation of the execution report

The default behaviour of CoverageScanner library is to produce an execution report upon the instrumented application exits.This might not be enough in the cases of unit test (for this tests, it is preferable to generate a report after the execution of eachsingle test) or for application like daemons which never terminates.

For these purposes, CoverageScanner library permits also to:

1. Embedded a C code into the instrumented application to control the report generation (22.2.3, page 123).

2. Generate the coverage report on the reception of a Microsoft® Windows event or a UNIX® signal.

22.2.1 Generation of an execution report using UNIX® signals

To generate the execution using UNIX® signals, add --cs-dump-on-signal=SIG to the linker command line. SIG is thesignal number or its common name (example SIGUSR1).

Using the UNIX® kill command it is possible to trigger the report generation. (ex: kill -SIGUSR1 <pid>)

22.2.2 Generation of an execution report using Microsoft® Windows event

To generate the execution using Microsoft® Windows events, add --cs-dump-on-event=EVENT to the linker command line.<EVENT> is a string which identify the event.

Using the Microsoft® function SetEvent(), it is possible to trigger the generation of a coverage report.

A sample of code is provided here:

#include <windows.h>#include <stdio.h>#include <strsafe.h>#include <iostream>

void DisplayError( LPTSTR lpszMessage ){

// Retrieve the system error message for the last-error code

LPSTR lpMsgBuf;DWORD dw = GetLastError();


Controlling the generation of the execution report

FormatMessageA(FORMAT_MESSAGE_ALLOCATE_BUFFER |FORMAT_MESSAGE_FROM_SYSTEM |FORMAT_MESSAGE_IGNORE_INSERTS,NULL,dw,MAKELANGID( LANG_NEUTRAL, SUBLANG_DEFAULT ),( LPSTR ) &lpMsgBuf,0, NULL );

// Display the error message and exit the processstd::cerr << lpszMessage << " (error " << dw << "): " << lpMsgBuf << ’\n’;

LocalFree( lpMsgBuf );}

int main( int argc, char *argv[] ){

if ( argc != 2 ){

std::cerr << "Usage: " << argv[0] << " <event_name>" << ’\n’;return 1;

}

const char * eventName = argv[1] ;

HANDLE ghEvent = OpenEventA(EVENT_MODIFY_STATE,FALSE,eventName);

if ( ghEvent == NULL ){

DisplayError( "Opening event failed" );return 1;

}

if ( ! SetEvent( ghEvent ) ){

DisplayError( "Raising event failed" );CloseHandle( ghEvent );return 1;

}

CloseHandle( ghEvent );return 0;

}


Support for specific test frameworks

22.2.3 Unit Testing

CoverageBrowser import execution results for each object which are part of an application. This mean that, if an unit test usesthe same objects as these for generating the application, CoverageBrowser is able to import the execution result of the unit testand merge it into the code coverage of the compiled application.

Example:An application is composed to 3 files:

app.cpp:This file contains the main() function of the application.

library.cpp:This file contains a set of functions called in app.cpp.

testlibrary.cpp:This contains the test code of the functions of the file library.cpp. testlibrary.cpp have its own main() function.

To generate the application:

cscl app.cpp /Foapp.objcscl library.cpp /Folibrary.objcscl library.obj app.obj /Feapp.exe

To generate the unit test:

cscl testlibrary.cpp /Fotestlibrary.objcscl library.obj testlibrary.obj /Fetestlibrary.exe

To import the execution report from the unit tests into the instrumentation database of the main application, two methods can beused:

• The execution of testlibrary.exe generates the execution report testlibrary.exe.csexe. This report can beloaded in the measurement database app.exe.csmes.

• The execution of testlibrary.exe generates the execution report testlibrary.exe.csexe. This report can beloaded in the measurement database testlibrary.exe.csmes of the test code. This measurement database can beimported into this one from the main application using "FileÕImport Unit Tests..." function of CoverageBrowser.(see Chapter 13.6.2, page 64)

In both cases, only the code coverage analysis of the file library.cpp is loaded, and the execution report of the test code isignored.

22.3 Support for specific test frameworks

22.3.1 CppUnit

CppUnit1 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test.Simply proceed as follows:

1Project page of CppUnit:http://cppunit.sourceforge.net


http://cppunit.sourceforge.net


1. Call __coveragebrowser_install() in the main() function.

2. Write a CppUnit TestListener class which records the code coverage report upon every unit test completion.The listener should set the name (using __coveragescanner_testname()) and clear the instrumentation (using__coveragescanner_clear()) before executing a test item (class member startTest()) to ensure to get only thecoverage data of the concerned test. When an test item is executed, the instrumentation and the execution status should besaved (using __coveragescanner_teststate() and __coveragescanner_save()) in the class member endTest().The class CoverageScannerListener give an implementation example.

3. Add this listener in the test manager of CppUnit (class CPPUNIT_NS::TestResult).In the example below

CoverageScannerListener coveragescannerlistener;controller.addListener( &coveragescannerlistener );

4. Compile the unit test using CoverageScanner.

#include <cppunit/TestListener.h>#include <cppunit/BriefTestProgressListener.h>#include <cppunit/CompilerOutputter.h>#include <cppunit/extensions/TestFactoryRegistry.h>#include <cppunit/TestResult.h>#include <cppunit/TestResultCollector.h>#include <cppunit/TestRunner.h>

class CoverageScannerListener : public CppUnit::TestListener{public:void startTest( CppUnit::Test *test ){m_testFailed=false;


int pos;// Adjusting the name of the test to display the tests// in a tree view in CoverageBrowserstd::string testname = "CppUnit/" + test->getName();while ( (pos=testname.find("::",0)) != std::string::npos )testname.replace(pos,2,"/");

// Reset the code coverage data to get only the code coverage// of the actual unit test.__coveragescanner_clear();__coveragescanner_testname(testname.c_str()) ;

#endif}

void addFailure( const CppUnit::TestFailure &failure ) { m_testFailed=true; }

void endTest( CppUnit::Test *test ){


// Recording the execution state in the coverage reportif (m_testFailed)__coveragescanner_teststate("FAILED");

else__coveragescanner_teststate("PASSED");



// Saving the code coverage report of the unit test__coveragescanner_save();__coveragescanner_testname("");

#endif}

private:bool m_testFailed;// Prevents the use of the copy constructor and operator.CoverageScannerListener( const CoverageScannerListener &copy );void operator =( const CoverageScannerListener &copy );

};

int main( int argc, char* argv[] ){#ifdef __COVERAGESCANNER__

__coveragescanner_install(argv[0]);#endif// Create the event manager and test controllerCPPUNIT_NS::TestResult controller;

// Add a listener that colllects test resultCPPUNIT_NS::TestResultCollector result;controller.addListener( &result );

// Add a listener that print dots as test run.CPPUNIT_NS::BriefTestProgressListener progress;controller.addListener( &progress );

// Add a listener that saves the code coverage informationCoverageScannerListener coveragescannerlistener;controller.addListener( &coveragescannerlistener );

// Add the top suite to the test runnerCPPUNIT_NS::TestRunner runner;runner.addTest( CPPUNIT_NS::TestFactoryRegistry::getRegistry().makeTest() );runner.run( controller );

return result.wasSuccessful() ? 0 : 1;}

22.3.2 QTestLib

QTestLib is a unit test framework for the Qt Library. This environment can easily be adapted to get the code coverage fromeach unit test.

Simply proceed as follows:


2. Write a QObject derived class called TestCoverageObject which records the code coverage report upon every unittest completion.

3. Instead of inheriting from QObject, make all your test cases inheriting from TestCoverageObject.



4. TestCoverageObject class provide its own init() and cleanup() slot, which use the CoverageScanner API to savethe code coverage report. If these slots are also declared in the test case class, it is necessary to rename it in initTest()and cleanupTest().

5. Compile your project with code code coverage enabled.

TestCoverageObject header:

#ifndef _TEST_COVERAGE_OBJECT_H#define _TEST_COVERAGE_OBJECT_H#include <QObject>class TestCoverageObject : public QObject{Q_OBJECTpublic:virtual void initTest() {}virtual void cleanupTest() {}

protected slots:void init() ;void cleanup();

};#endif

TestCoverageObject source:

#include "testcoverageobject.h"#include <QTest>#include <QMetaObject>#include <QString>

void TestCoverageObject::init(){#ifdef __COVERAGESCANNER____coveragescanner_clear();

#endifinitTest();

}

void TestCoverageObject::cleanup(){cleanupTest();


QString test_name="unittest/";test_name+=metaObject()->className();test_name+="/";test_name+=QTest::currentTestFunction();__coveragescanner_testname(test_name.toLatin1());if (QTest::currentTestFailed())__coveragescanner_teststate("FAILED");

else__coveragescanner_teststate("PASSED") ;

__coveragescanner_save();__coveragescanner_testname("");

#endif}



22.3.3 GoogleTest

GoogleTest2 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unittest.Simply proceed as follows:


2. Write a TestEventListenet class which records the code coverage report upon every unit test completion. Thelistener should set the name (using __coveragescanner_testname()) and clear the instrumentation (using__coveragescanner_clear()) before executing a test item (class member startTest()) to ensure to get only thecoverage data of the concerned test. When an test item is executed, the instrumentation and the execution status should besaved (using __coveragescanner_teststate() and __coveragescanner_save()) in the class member endTest().The class CodeCoverageListener give an implementation example.

3. Add this listener in the Append function of the GoogleTest listener (function::testing::UnitTest::GetInstance()->listeners().Append()).


#include <gtest/gtest.h>#include <direct.h>#include <stdlib.h>

class CodeCoverageListener : public ::testing::TestEventListener{public:// Fired before any test activity starts.virtual void OnTestProgramStart(const ::testing::UnitTest& unit_test){}

// Fired before each iteration of tests starts. There may be more than// one iteration if GTEST_FLAG(repeat) is set. iteration is the iteration// index, starting from 0.virtual void OnTestIterationStart(const ::testing::UnitTest& unit_test, int iteration){}

// Fired before environment set-up for each iteration of tests starts.virtual void OnEnvironmentsSetUpStart(const ::testing::UnitTest& unit_test){}

// Fired after environment set-up for each iteration of tests ends.virtual void OnEnvironmentsSetUpEnd(const ::testing::UnitTest& unit_test){}

// Fired before the test case starts.virtual void OnTestCaseStart(const ::testing::TestCase& test_case){}

// Fired before the test starts.

2Project page of GoogleTest:http://code.google.com/p/googletest/


http://code.google.com/p/googletest/


virtual void OnTestStart(const ::testing::TestInfo& test_info){

#ifdef __COVERAGESCANNER____coveragescanner_clear();std::string test_name=std::string(test_info.test_case_name())+ ’/’+ std::string(test_info.name());

__coveragescanner_testname(test_name.c_str());#endif

}

// Fired after a failed assertion or a SUCCESS().virtual void OnTestPartResult(const ::testing::TestPartResult& test_part_result){}

// Fired after the test ends.virtual void OnTestEnd(const ::testing::TestInfo& test_info){

#ifdef __COVERAGESCANNER____coveragescanner_teststate("UNKNOWN");if (test_info.result()){if (test_info.result()->Passed())__coveragescanner_teststate("PASSED");

if (test_info.result()->Failed())__coveragescanner_teststate("FAILED");

}__coveragescanner_save();

#endif}

// Fired after the test case ends.virtual void OnTestCaseEnd(const ::testing::TestCase& test_case){}

// Fired before environment tear-down for each iteration of tests starts.virtual void OnEnvironmentsTearDownStart(const ::testing::UnitTest& unit_test){}

// Fired after environment tear-down for each iteration of tests ends.virtual void OnEnvironmentsTearDownEnd(const ::testing::UnitTest& unit_test){}

// Fired after each iteration of tests finishes.virtual void OnTestIterationEnd(const ::testing::UnitTest& unit_test, int iteration){}

// Fired after all test activities have ended.virtual void OnTestProgramEnd(const ::testing::UnitTest& unit_test){



}} ;

int main(int argc, char **argv){

//initialize CoverageScanner library#ifdef __COVERAGESCANNER____coveragescanner_install(argv[0]);

#endif

::testing::FLAGS_gtest_output = "xml";::testing::UnitTest::GetInstance()->listeners().Append(new CodeCoverageListener);

::testing::InitGoogleTest(&argc, argv); //init google test framework

return RUN_ALL_TESTS(); //run all tests which are listed in this project}

22.3.4 CxxTest

CxxTest3 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test.Simply proceed as follows:


2. Create a CxxTest TestListener class called CoverageScannerListener by subclassing an existing listener (inthe example bellow: ErrorPrinter). It will record the code coverage report upon every unit test completion.The listener should set the name (using __coveragescanner_testname()) and clear the instrumentation (using__coveragescanner_clear()) before executing a test item (class member enterTest()) to ensure to get only thecoverage data of the concerned test. When an test item is executed, the instrumentation and the execution statusshould be saved (using __coveragescanner_teststate() and __coveragescanner_save()) in the class memberleaveTest(). Finally, all test failure members of this class need to be reimplemented to record the test failure.

3. In the main() function call the run() function of CoverageScannerListener instead ofCxxTest::ErrorPrinter().run().


#include <cxxtest/TestRunner.h>#include <cxxtest/TestListener.h>#include <cxxtest/TestTracker.h>#include <cxxtest/ValueTraits.h>#include <cxxtest/ValueTraits.h>#include <cxxtest/ErrorPrinter.h>

class CoverageScannerListener : public CxxTest::ErrorPrinter{public:CoverageScannerListener(

std::ostream &o=std::cout,const char *preLine = ":",const char *postLine = ""

3Project page of CxxTest:http://cxxtest.com


http://cxxtest.com


) : CxxTest::ErrorPrinter( o, preLine , postLine ) { }

int run(){return CxxTest::ErrorPrinter::run();

}

void enterTest( const CxxTest::TestDescription & desc){test_passed=true;


// Adjusting the name of the test to display the tests// in a tree view in CoverageBrowserstd::string testname="CxxTest/";testname += desc.suiteName();testname += "/";testname += desc.testName();

// Reset the code coverage data to get only the code coverage// of the actual unit test.__coveragescanner_clear();__coveragescanner_testname(testname.c_str());

#endifreturn CxxTest::ErrorPrinter::enterTest( desc );

}

void leaveTest( const CxxTest::TestDescription & desc){


// Recording the execution state in the coverage reportif (test_passed)__coveragescanner_teststate("PASSED");

else__coveragescanner_teststate("FAILED");

// Saving the code coverage report of the unit test__coveragescanner_save();

#endifreturn CxxTest::ErrorPrinter::leaveTest( desc );

}

void failedTest(const char *file,unsigned line,const char *expression)

{ // Just record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedTest(file,line,expression );

}

void failedAssert(const char *file,unsigned line,const char *expression



){ // Just record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssert(file,line,expression );

}

void failedAssertEquals(const char *file,unsigned line,const char *xStr,const char *yStr,const char *x,const char *y)

{ // Just record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertEquals(

file,line,xStr,yStr,x,y );

}

void failedAssertSameData(const char *file,unsigned line,const char *xStr,const char *yStr,const char *sizeStr,const void *x,const void *y,unsigned size)

{ // Just record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertSameData(

file,line,xStr,yStr,sizeStr,x,y,size );

}

void failedAssertDelta(const char *file,unsigned line,const char *xStr,const char *yStr,const char *dStr,const char *x,const char *y,



const char *d)

{ // Just record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertDelta(

file,line,xStr,yStr,dStr,x,y,d );

}

void failedAssertDiffers(const char *file,unsigned line,const char *xStr,const char *yStr,const char *value)

{ // Just record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertDiffers(

file,line,xStr,yStr,value );

}

void failedAssertLessThan(const char *file,unsigned line,const char *xStr,const char *yStr,const char *x,const char *y)

{ // Just record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertLessThan(


}

void failedAssertLessThanEquals(const char *file,unsigned line,const char *xStr,const char *yStr,



const char *x,const char *y)

{ // Just record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertLessThanEquals(


}

void failedAssertRelation(const char *file,unsigned line,const char *relation,const char *xStr,const char *yStr,const char *x,const char *y)

{ // Just record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertRelation(

file,line,relation,xStr,yStr,x,y);

}

void failedAssertPredicate(const char *file,unsigned line,const char *predicate,const char *xStr,const char *x )

{ // Just record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertPredicate(

file,line,predicate,xStr,x);

}

void failedAssertThrows(const char *file,unsigned line,



const char *expression,const char *type,bool otherThrown)

{ // Just record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertThrows(

file,line,expression,type,otherThrown );

}

void failedAssertThrowsNot(const char *file,unsigned line,const char *expression)

{ // Just record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertThrowsNot(

file,line,expression );

}

private:bool test_passed;

};

int main(){#ifdef __COVERAGESCANNER____coveragescanner_install(argv[0]);

#endif

// Use "CoverageScannerListener().run()" instead of "CxxTest::ErrorPrinter().run()"return CoverageScannerListener().run();

}

22.3.5 NUnit

Squish Coco provides an addin for NUnit4 version 2.4.4 and above as sample. To install it proceed as followings:

1. Build NUnitSquishCoco.dll using the Microsoft® Visual Studio® project NUnitSquishCoco.vsproj provided in thesample directory.

2. Copy NUnitSquishCoco.dll to the addins folder located in the bin folder where NUnit.exe can be found.

3. Start NUnit.exe and verify that the addin “NUnit Squish Coco” is loaded.4http://nunit.org/


http://nunit.org/


Once installed, as soon as NUnit’s test driver is executing a C# or C++ managed unit test test.dll, it generates automaticallya code coverage execution report test.dll.csexe automatically if test.dll is instrumented with Squish Coco. The codecoverage information is organized into a tree containing the coverage and the execution status for each single unit test. Theexecution report can be then imported into the application’s instrumentation database with CoverageBrowser or cmcsexeimport.

22.3.6 Squish

It is easily possible to run the GUI testing tool Squish together with Squish Coco in order to get the C/C++coverage of aSquishtest suite. A more in-depth analysis is then possible, correlating each test case (and its results) with the respectivecoverage information. This is especially true since Squish Coco features the comparison of individual executions, including thecalculation of an optimal order.

22.3.6.1 General Approach

The approach depicted below is based on the possibility to apply information about the name, the result and a free-formcomment to each execution (stored in .csexe files). See Chapter 22.1, page 120 for the details.

As an example we’ll use Squish for Qt’s addressbook on a Unix-based system and a JavaScript test script.

1. First we’ll initialize the execution data with the name of the Squish test case that is being run.

function main(){

var currentAUT = currentApplicationContext();var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe"var testCase = squishinfo.testCase;var testExecutionName = testCase.substr(testCase.lastIndexOf(’/’) + 1);var file = File.open(execution, "a");file.write("*" + testExecutionName + "\n");file.close();var ctx = startApplication("addressbook");

...

2. Insert the main test script at this point

3. After the main test script we’ll log the result of the test for the coverage tool:

...// wait until AUT shutdownwhile (ctx.isRunning) {

snooze(1); // increase time if not enough to dump coverage data}

// test result summary and statusvar positive = test.resultCount("passes");var negative = test.resultCount("fails") + test.resultCount("errors") + test.

resultCount("fatals");var msg = "TEST RESULTS - Passed: " + positive + " | " + "Failed/Errored/Fatal: " +

negative;var status = negative == 0 ? "PASSED" : "FAILED";var file = File.open(execution, "a");file.write("<html><body>" + msg + "</body></html>\n");file.write("!" + status + "\n")



file.close();}

When you execute the scripts containing these steps, the Squish CocoExecution Report loads with the test case name, status andexecution summary in the execution details and comments.

22.3.6.2 Simplified for Reuse

1. Create a file called squishCocoLogging.js in Test Suite Resources with the following functions:

function getExecutionPath() {var currentAUT = currentApplicationContext();var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe"return execution;

}

function logTestNameToCocoReport(currentTestCase, execution) {var testExecutionName = currentTestCase.substr(currentTestCase.lastIndexOf(’\\’) + 1)

;var file = File.open(execution, "a");file.write("*" + testExecutionName + "\n");file.close();

}

function logTestResultsToCocoReport(testInfo, execution){

var currentAUT = currentApplicationContext();

// wait until AUT shuts downwhile (currentAUT.isRunning)snooze(5);

// collect test result summary and statusvar positive = testInfo.resultCount("passes");var negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") +

testInfo.resultCount("fatals");var msg = "TEST RESULTS - Passed: " + positive + " | " + "Failed/Errored/Fatal: " +

negative;var status = negative == 0 ? "PASSED" : "FAILED";

// output results and status to Coco execution report filevar file = File.open(execution, "a");file.write("<html><body>" + msg + "</body></html>\n");file.write("!" + status + "\n")file.close();

}

A Python version of this code is:

import re

def getExecutionPath():currentAUT = currentApplicationContext()execution = "%(currAUTPath)s\\%(currAUTName)s.exe.csexe" % {"currAUTPath" :

currentAUT.cwd, "currAUTName" : currentAUT.name}



return execution

def logTestNameToCocoReport(currentTestCase, execution):testExecutionName = re.search(r’[^\\]\w*$’, currentTestCase)testExecutionName = testExecutionName.group(0)file = open(execution, "a")file.write("*" + testExecutionName + "\n")file.close()

def logTestResultsToCocoReport(testInfo, execution):currentAUT = currentApplicationContext()# wait until AUT shuts downwhile (currentAUT.isRunning):

snooze(5)

# collect test result summary and statuspositive = testInfo.resultCount("passes")negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") + testInfo.

resultCount("fatals")msg = "TEST RESULTS - Passed: %(positive)s | Failed/Errored/Fatal: %(negative)s" %

{’positive’: positive, ’negative’: negative}if negative == 0:

status = "PASSED"else:

status = "FAILED"

# output results and status to Coco execution report filefile = open(execution, "a")file.write("<html><body>" + msg + "</body></html>\n")file.write("!" + status + "\n")file.close()

2. Add the following function calls after startApplication() in the main test script:

execution = getExecutionPath();

logTestNameToCocoReport(squishinfo.testCase);

In Python:

execution = getExecutionPath()

logTestNameToCocoReport(squishinfo.testCase)

3. At the end of your script, after closing the AUT (for example with steps clicking File > Exit), call the following function:

logTestResultsToCocoReport(test);

In Python:

logTestResultsToCocoReport(test)

4. In the event your AUT closes unexpectedly, or a script error occurs, incorporating a try, catch, finally ensures yourresults still output to the Coco report file.

Your main test script should be similar to the following:



source(findFile("scripts","squishCocoLogging.JS"))

function main(){

startApplication("addressbook");execution = getExecutionPath();logTestNameToCocoReport(squishinfo.testCase, execution);

try {// body of script

}catch(e) {

test.fail(’An unexpected error occurred’, e.message)}finally {

logTestResultsToCocoReport(test, execution)}

}

Python version:

source(findFile("scripts","squishCocoLogging.py"))

def main():startApplication("addressbook")execution = getExecutionPath()logTestNameToCocoReport(squishinfo.testCase, execution)

try:try:

# body of scriptexcept Exception, e:

test.fail("test failed: ", e)finally:

logTestResultsToCocoReport(test,execution)


Part V CoverageScannerTclReference Manual


Command Line Arguments

Chapter

23Command Line Arguments

Syntax:

coveragescannertcl [<cs-options>] <application arguments>

Where:

<application arguments>:Tcl-based application including optional arguments

<cs-options>:Any of the following options:

--cs-output=<STRING>: Base name of the generated .csmes and .csexe files--cs-exclude-path=<STRING>: Exclude files located in specified directory--cs-include-path=<STRING>: Override exclusion of files based on specified directory--cs-exclude-file-abs-regexp=<STRING>: Exclude file if absolute file name matches regular expression--cs-exclude-file-abs-regexp=<STRING>: Override exclusion of file if absolute name matches regular expression--cs-exclude-file-abs-wildcard=<STRING>: Exclude file if absolute name matches wildcard--cs-exclude-file-abs-wildcard=<STRING>: Override exclusion of file if absolute name matches wildcard--cs-include-code-regexp=<STRING>: Regular expression of code that will be instrumented even if it is excluded--cs-exclude-code-regexp=<STRING>: Regular expression of code that will not be instrumented--cs-include-code-wildcard=<STRING>: Wildcard of code that will be instrumented even if it is excluded--cs-exclude-code-wildcard=<STRING>: Wildcard of code that will be instrumented even if it is excluded--cs-include-init-scripts: Enables code instrumentation for standard Tcl initialization scripts--cs-include-unnamed: Instrument Tcl code even if it has no or just an ambiguous file name set. By default such

code is ignored as it is typically just temporary code generated for e.g. an Tk event handler or results fromnested eval calls.

CoverageScannerTcl is a utility which permits to collect code coverage information of a Tcl-based execution. It will collect allscripts parsed by the embedded Tcl interpreter and determine which parts were actually executed.

Once the application exits a pair of .csmes and .csexe files will be written out that contain the code instrumentationinformation and execution data, respectively.

If the --cs-output switch is omitted the base name of the output files will the same as the <application> name.

See (see Chapter 24, page 141) for a recommended set of --cs-exclude-code-regexp switches for typical Tcl/Tk applica-tions.


Mixing Tcl with C++

Chapter

24Mixed Tcl/C/C++ Code Coverage Analysis

24.1 Analysis of Tcl script coverage

24.1.1 Demo application build

Copy the sample to your working diretcory

$ cp -r /opt/SquishCoco/doc/demoapp_tcl ./$ cd demoapp_tcl

Edit Makefile and adjust the following variables

TCLLIBS=-ltk8.5 -litcl3.4 -ltcl8.5TCLINCLUDES=-I/usr/include/tcl8.5

to match your system’s library names and include paths.Build the C++ executable with:

$ make

Test the simple Tk GUI with an invocation of the launcher script:

$ ./proto

24.1.2 Ad-hoc Tcl coverage run

Launch the application under test via the Tcl coverage scanning tool :

$ coveragescannertcl --cs-output=tcl ./proto


Analysis of Tcl script coverage

The GUI will show up as normal while code coverage is monitored. At this point the application might already be shut downvia the "FileÕExit" menu entry or the [X] button in the window’s title bar. Observe the two files tcl.csmes and tcl.csexebeing generated. The latter contains data about the code covered in the last execution. The former is a measurement databasecontaining information about the instrumented code and possibly imported previous executions. In this case the base name ofthe output files was set via the --cs-output switch. If omitted it will be deduced from the application name.

24.1.3 Result analysis

The results can be view interactively in the GUI provided by Coco. Run

$ coveragebrowser

to launch it. Select "FileÕOpen..." to open the tcl.csmes file. You’ll see all Tcl files and procedures listed in the Sourcesand Methods tab, respectively. When selecting either of them a source code viewer will open that shows instrumented codewith a gray background color. Note that the coloring is still limited to instrumented, i.e. seen code only. To check which codegot actually executed (or not) select "FileÕLoad Execution Report..." and open tcl.csexe. Now red and green colorsdenoting positive and negative coverage will show up for files, procedures and the code itself.

24.1.4 Filtered Tcl coverage run

As you might have noticed that above ad-hoc run produced results for several unnamed and internal files. These stem frominternal Tcl/Tk files or e.g. mouse event handlers that result in on-the-fly eval calls. Compared to the application-specific Tclfiles the coverage information for these internal pieces of code is of little interest.

The easiest way to hide the non-relevant pieces of code is through usage of the various --cs-exclude-code options. Arecommended list of switches will be shown in the repetition of the previous run with filtering options applied:

$ coveragescannertcl --cs-output=tcl --cs-exclude-code-regex="^\s*[^\n]*\s*$"--cs-exclude-code-regex="(tcl|tk)Init" --cs-exclude-code-regex="namespace eval ::itcl"--cs-exclude-code-regex="^# This file is generated"--cs-exclude-code-regex="tk::Menu(Motion|Leave)" ./proto

When viewing the new tcl.csmes in CoverageBrowser you’ll notice that the list of files is reduced to the relevant canvas.tclx,input.tclx, mainwindow.tclx, point.itcl and rect.itcl scripts.

24.1.5 Report Generation

All of text, XML and HTML reports can be generated from the CoverageBrowser GUI. With future automation of nightly testruns or Continuous Integration (CI) in mind we’ll create the report from the command line. And for this very purpose we’ll alsopretend that the execution data has not been imported into the measurement database, yet. This step can be accomplished likethis from the command line:

$ cmcsexeimport --title="First run" -m tcl.csmes -e tcl.csexe

The resulting proto.csmes file now contains everything we need to generated a report:

$ cmreport --title="Coverage Data" -m tcl.csmes --html=report.html


Analysis of C/C++ code coverage

The resulting report.html file (including a report_html/ directory) can now be visited with a Web browser. It includes alisting of executions, source files, functions and metrics on a global level, per file and per functions.

24.2 Analysis of C/C++ code coverage

24.2.1 Instrumentation

While Tcl code can be instrumented for coverage analysis on-the-fly this step requires a rebuild for C or C++ applications. Butthis can be accomplished with almost no change to your regular application build. Here are the two possible approaches:

• Modify your build system to make use of the code scanning tools csgcc, csg++, etc. that will both instrument the codeand later use the original compiler to compile the code as usual.

• Sneak Coco’s compiler wrappers named gcc, g++, etc. into the executable search PATH. They will again instrument thecode and then let the real compiler do its job.

We’ll make use of approach number 2. First, clean the existing object files:

$ make clean

Then, recompile the application with a changed PATH and a switch that enables the compiler wrappers. For brevity, we’ll dothis all in a single line:

$ PATH=~/SquishCoco/wrapper/bin:$PATH COVERAGESCANNER_ARGS="--cs-on" make

As a result we’ll not only get the main executable (proto.exe) but also a corresponding instrumentation file proto.exe.csmes.Information about the instrumented code (gray background color) can optionally be reviewed at this point

$ coveragebrowser -m proto.exe.csmes

24.2.2 Execution

Let’s run the application

$ ./proto

and quit it via its GUI right after. The current working directory will now contain a proto.exe.csexe containing informationabout completed execution.

24.2.3 Result analysis

Same as with Tcl the results can now be reviewed in CoverageBrowser. A short cut invocation that opens both the instrumentationand execution files is shown here:

$ coveragebrowser -m proto.exe.csmes -e proto.exe.csexe




The process of a report generation works analogous to the approach shown for Tcl code above.

24.2.5 Analysis of mixed Tcl and C/C++ code coverage

Coverage data for the Tcl and C/C++ parts of a hybrid application can be gathered independently as shown above. Either in asingle or multiple executions. The only extra step needed is the merging of the data into a single measurement database. Whichthen can be used to generate a single, combined report taking both languages into account.

24.2.6 Combined coverage run

Assuming the instrumentation of the C/C++ code is still in place we can gather coverage data for both types of languages with asingle run:

$ coveragescannertcl --cs-output=tcl ./proto

With the proto.exe.csmes file still in place from the application rebuild we’ll now have two sets of files:

1. tcl.csexe

2. tcl.csmes

3. proto.exe.csexe

4. proto.exe.csmes

At this point the coverage data for Tcl and C/C++ could be analysed independently using cmcsexeimport. But we want togain an overall picture and will therefore merge the results in the next section.

24.2.7 Merging measurement data

Instead of using the Coverage Browser GUI we’ll again use the command line tools. In the first step we’ll merge theinstrumentation files stemming from the two languages into one:

cmmerge -o all.csmes tcl.csmes proto.exe.csmes

To save some disk space the --append option can be used rather than creating the new all.csmes file.

In the second step the two execution files are merged using the merge policy for the 2nd language:

$ cmcsexeimport -m all.csmes -e tcl.csexe --title="Hybrid" --passed$ cmcsexeimport -m all.csmes -e proto.exe.csexe -p merge --title="Hybrid" --passed

For the merge to actually happen the title has to be identical.




An HTML report encompassing coverage for both application parts can be generated with a single command line call:

$ cmreport --title="Hybrid Tcl/C++ Coverage" -m all.csmes --html=report.html

Select "Source Files" to see both Tcl (various files) and C++ code (main.cpp) being listed in a combined report.


Part VI Command Line Tools


cmreport - Code coverage report generation

Chapter

25cmreport - Code coverage report generation

cmreport is a utility which permits to generate Text, HTML, XML or CSV reports from an instrumentation data base (a .csmesfile). It generates exactly the same reports as these generated by CoverageBrowser (see Chapter 14, page 66).

Example: The simplest use case is the generation of an HTML report from a single instrumentation database. If the name of thefile is project.csmes, we may write:

$ cmreport --csmes=project.csmes --html=report

This command creates a file report.html and a directory report_html. The file report.html is the main page of the reportand can be opened with the browser, while report_html contains all the other files. Most of the other command line optionsjust modify the content of such a report.

Syntax:

cmreport -m <csmes_file> ...

Where:

-m <file_name> | --csmes=<file_name>:Set the name of the main .csmes file (mandatory).

-mr <file_name> | --csmes-reference=<file_name>:Set the name of the .csmes reference file. With this option, cmreport compares two software releases.

--release-comparison=<argument>:Comparison mode when comparing two software releases. Possible values:

none: Display only the differences between two software releases.

modified_functions: Display code coverage only for modified functions.

unmodified_functions: Display code coverage only for identical functions.

--source-type=<argument>:Source type used for the analysis. Possible values:

preprocessed: The preprocessed code is analyzed.

original: The original source code is analyzed. C macros are ignored in this mode.

-l <number> | --level=<number>:Set the code coverage level. <number> must be an integer greater than 0.


File selection options

--max-threads=<number>:Maximal number of threads used for the report computation. By default, there as many threads as there are CPUs on thesystem.

--stat:Print global coverage statistics to the standard output

25.1 File selection options

The following options allow to specify the source files that occur in the report.

--include-file-abs-regex:Include source files into the reports using regular expressions

--exclude-file-abs-regex:Exclude source files from the reports using regular expressions

--include-file-abs-wildcard:Include source files into the reports using wildcards

--exclude-file-abs-wildcard:Exclude source files from the reports using wildcards

25.2 Execution selection options

The following options allow to specify the executions that occur in the report.

-s <regex> | --select=<regex>:Select executions using regular expression. If not used, all executions are implicitly selected.

-d <regex> | --deselect=<regex>:Deselect executions using regular expression

--deselect-to-check:Deselect all executions with the status “CHECK_MANUALLY”

--select-passed:Select all executions with the status “PASSED”

--deselect-passed:Deselect all executions with the status “PASSED”

--select-failed:Select all executions with the status “FAILED”

--deselect-failed:Deselect all executions with the status “FAILED”

--select-unknown:Select all executions with unknown status


Options for HTML or CSV reports

--deselect-unknown:Deselect all executions with unknown status

--executions-reference-from-csmes=<argument>:Select all executions present in the specified .csmes file.

-sr <regex> | --select-reference=<regex>:Select reference executions using regular expression. This option activates the comparison of executions together.

-dd <regex> | --deselect-reference=<regex>:Deselect reference executions using regular expression. This option activates the comparison of executions together.

--deselect-to-check-reference:Deselect all reference executions with the status “CHECK_MANUALLY”

--select-passed-reference:Select all reference executions with the status “PASSED”

--deselect-passed-reference:Deselect all reference executions with the status “PASSED”

--select-failed-reference:Select all reference executions with the status “FAILED”

--deselect-failed-reference:Deselect all reference executions with the status “FAILED”

--select-unknown-reference:Select all reference executions with unknown status

--deselect-unknown-reference:Deselect all reference executions with unknown status

25.3 Options for HTML or CSV reports

Command line arguments:

-h <file_name> | --html=<file_name>:Browsable HTML report output file name

--html-single=<file_name>:HTML report output file name (single file)

--csv-excel=<file_name>:CSV report (optimized for Excel and OpenOffice) output file name

--execution-level=<argument>:Code coverage level for executions

--css=<file_name>:CSS style sheet

--icon=<file_name>:Icon

--title=<file_name>:Title


Options for HTML or CSV reports

--section=<argument>:Section of the generated report. This option can be repeated to select more than one section type. If nothing is selected,the cmreport generates all relevant sections automatically.

The possible sections are:

global: Global statistics

function: Function statistics

class: Class/namespace statistics

execution: Execution statistics

source: Source file statistics

directory: Directory file statistics

manually-validated: Manually validated code fragments

unexecuted: Unexecuted code fragments

dead-code: Dead code fragments

executed: Executed code fragments

--global-threshold-low-medium=<num>:Global threshold setting

--global-threshold-medium-high=<num>:Global threshold setting

--source-threshold-low-medium=<num>:Source/directory threshold setting

--source-threshold-medium-high=<num>:Source/directory threshold setting

--function-threshold-low-medium=<num>:Function/class/namespace threshold setting

--function-threshold-medium-high=<num>:Function/class/namespace threshold setting

-b | --coverage-branch :Code coverage on branch level only

--coverage-condition :Code coverage at decision/condition level

--line-coverage :Line coverage analysis

--function-coverage :Function coverage analysis

-t | --test-coverage :Test count mode

-D | --debug :Debug flag

--csv-field-separator=<char>:Field separator for a CSV file

--csv-comma=<char>:Separator (, or .) used for floats in a CSV file


Options for text reports

25.3.1 Obsolete options for CSV reports


--csv-method=<argument>:Generate a CSV report file for each method

--csv-source=<argument>:Generate a CSV report file for each file

--csv-field-separator=<argument>:Field separator for a CSV file

--csv-comma=<argument>:Separator (, or .) used for floats in a CSV file

25.4 Options for text reports


-t <file_name> | --text=<file_name>:Text report output file name

--format-executed=<argument>:Line format for the executed for code fragments

--format-unexecuted=<argument>:Line format for the unexecuted for code fragments

--format-dead-code=<argument>:Line format for dead code fragments

The following escape sequences are recognized:

%f: Source code file name

%l: Line number

%c: Column number

%m: Explanation

It is allowed to to set the output file to an empty string (i.e.: --text=), in this case the standard output of the console is used.

25.5 Options for EMMA-XML reports



Options for JUnit reports

--emma=<argument>:Generate an EMMA-XML report

--emma-source-pattern=<argument>:Source file pattern for EMMA-XML reports

--emma-package-pattern=<argument>:Package name pattern for EMMA-XML reports

The patterns have the following format:

%f: file name (without path)

%b: file name without path and extension

%e: file extension

%p: file path (without file name)

%P: absolute file path (without file name)

i For Jenkins-CI users, it is not recommended to set these parameters. The default values are chosen to work withthe EMMA-XML plugin.

25.6 Options for JUnit reports


-j <file_name> | --junit=<file_name>:Generate an JUnit report

JUnit reports does not contain any code coverage information. The report list only the tests executed, their status (passed orfailed) and their comments.

25.7 Options for Cobertura reports


--cobertura=<argument>:Generate a Cobertura-XML report

--cobertura-source=<argument>:Root path of the source code when generating a Cobertura-XML report

--sonarqube-project=<argument>:Generate a Cobertura-XML report for a SonarQube project

To generate a Cobertura reports two work flows are supported:


Options for Cobertura reports

1. Specifying directly all necessary arguments to generate the report which are:

(a) the output file name (--cobertura=<argument> command line option)

(b) the root path of the source directory tree (--cobertura-source=<argument> command line option)

2. Indicating the path of a SonarQube project properties file sonar-project.properties(--sonarqube-project=<argument> command line option): in this case, the root path of the source tree isextracted from its contains using sonar.cxx.coverage.reportPath, sonar.cxx.coverage.itReportPath orsonar.cxx.coverage.overallReportPath property.


cmcsexeimport

Chapter

26cmcsexeimport - Command Line Import Utility

Syntax:

cmcsexeimport -m <csmes_file> -t <title> [-p <policy>] csexe_file csexe_file ... csexe_file

Where:

-m <argument>|--csmes=<argument>:CSMes file name

-e <argument>|--csexe=<argument>:CSExe file name (argument can be repeated to process more than one file)

-l <argument>|--csexe-list=<argument>:Text file which contains a list of CSExe files. One file per line.

-t <argument>|--title=<argument>:Execution title

-p <argument>|--policy=<argument>:Import policy: ignore_duplicates (default), import_duplicates_and_empty, import_duplicates or merge

-d|--delete:Delete the imported file after processing it.

-P|--passed:Execution marked as passed.

-F|--failed:Execution marked as failed.

-C|--check-manually:Execution marked as to be checked manually.

-D|--debug:Verbose output.

<argument>:CSExe file name (argument can be repeated to process more than one file)

-f <argument>|--csmes-save-frequency=<argument>:Permits to commit the imported execution report to the instrumentation database after reading argument number ofexecution reports.


cmcsexeimport

cmcsexeimport is a utility which permits to import an execution report (.csexe file) into an instrumentation data base (.csmesfile). It behaves exactly like CoverageBrowser load of execution report function (see Chapter 12.1.2, page 50).


cmmerge - Merging Utility

Chapter

27cmmerge - Merging Utility

Syntax:

cmmerge [<options>] -o <outfile> <infile>...cmmerge --blackbox -o <outfile> <infile>

where:

-o <outfile>|--output=<outfile>:.csmes output file. (Required.)

<infile>:.csmes input file. Only one input file is allowed if --blackbox is selected.

-b|--blackbox:Generate a .csmes file for black-box testing.

<options>:Any of:

-a|--append: Merge the input file(s) with the existing content of the output file. <outfile> must already exist.

-i <reference>|--instrumentation-and-execution=<reference>: Include only execution results that belong to<reference> (which is a .csmes file).With this option, cmmerge merges <reference> with the <infile>s, but includes the instrumentations andexecutions of an <infile> only if they also occur in <reference>. If --append is set, <outfile> is viewed as apart of <reference>.This option is useful for unit tests, where the <reference> is the program for which the test is written, whilethe <infile>s contain the execution results of the unit tests.

-r <reference>|--reviews-only=<reference>: Merge <reference> with the comments and manual validations inthe <infile>s. Only validations and comments of unmodified functions are imported. (<reference> is a .csmesfile too).

--delete: Delete the input files from disk after importing them.

-v|--verbose: Verbose output

cmmerge is a small utility which permits to merge several instrumentation databases (.csmes file) together. It behaves exactlylike the CoverageBrowser merge function (see Chapter 13.6.1, page 64).


License management

Chapter

28License management

28.1 cocolic - License activation

Syntax:

cocolic <arguments>

Where:

--fetch-license-key=<argument>:Retrieve a license key using an activation code.

--no-proxy:Disabling proxy server usage.

--http-proxy=<server>:<port>:Retrieve a license key using the HTTP proxy server <server> (port <port>)

--socks5-proxy=<server>:<port>:Retrieve a license key using the SOCKS5 proxy server <server> (port <port>)

--proxy-user=<argument>:Proxy user identification.

--proxy-password=<argument>:Proxy user password.

--license-key=<argument>:Set a license key directly.

--machine-identifier:Display the machine identifier.

--enable-node-locked:Enable the node locked license. In this case the license server configuration will be ignored.

--disable-node-locked:Disable the node locked license. In this case the license server configuration will be used, if configured.

cocolic can be use to activate Squish Coco on a computer without graphical interface available.


Location of the license

28.2 Location of the license

When searching for a license, Squish Coco first tries to find a node-locked license and then to locate a license server.

The license file for a node-locked license is .squishcoco-3-license in the professional edition of Squish Coco. In thenon-commercial edition, its name is .squishcocononcommercial-3-license.

If the environment variable SQUISHCOCO_LICENSEKEY_DIR is set, Squish Coco treats the content as a directory name and triesto find there the license file. Otherwise, it searches for it in several predefined locations:

1. Under Windows, it uses the directories %HOMEPATH%, %HOMEDRIVE%\%HOMEPATH%, %HOME%, %USERPROFILE% or the rootdirectory (‘C:\’).

2. Under Unix, it uses the directories $HOMEPATH, $HOME, or the root directory (‘/’).

If no license file is found, Squish Coco will try to read the specification of a license server from the environment variableSQUISHCOCO_LICENSE_SERVER. If the variable is not set, Squish Coco tries to read the specification from the file .squishcoco-3-licserver (or from .squishcocononcommercial-3-licserver for the non-commercial edition). It searches for thisfile in the same predefined directories as before for the license file.

In both cases, the specification of the license server consists of a host name or an IP address, possibly followed by a colon and aport number.

! Setting SQUISHCOCO_LICENSEKEY_DIR globally for all users is not recommended. It would force all users to usethe same license, but each license key is specific to user and host.


cocolicserver

Chapter

29 cocolicserver - License Server

Syntax:

cocolicserver <arguments>

Where:

-c <path>|--config=<path>:Path to configuration file (provided by froglogic)

-l <path>|--logfile=<path>:Name of file to write log messages to; defaults to standard output

--service-install:Installs cocolicserver as a Windows service such that it will start automatically when booting the system. Can also beused to update the configuration of a previously installed service.

--service-uninstall:Uninstalls the cocolicserver Windows service.

--daemon:Only on Linux™and Apple® Mac OS X: launches cocolicserver as a daemon background process

cocolicserver can be used to serve Squish Coco licenses from a central computer; clients can connect to the server andcocolicserver will make sure that no more than the available number of licenses will be used concurrently.


Part VII Frequently AskedQuestions


Installation

Chapter

30 Installation

30.1Is there a way to deploy Squish Coco on Windows without any userinterractions?

Squish Coco installer supports the following command line options:

/S: runs the installer in silent mode

/SD: runs the installer in silent mode and suppress all popup dialog

/D: cpecification of the installation directory

For example, to install Squish Coco in background:

SquishCocoSetup.exe /SD

For example, to install Squish Coco in background on c:\program files\squishcoco:

SquishCocoSetup.exe /SD "/D=c:\program files\squishcoco"

30.2 Where are the settings stored?

On Windows: %APPDATA%\squishcocoOn Linux $HOME/.config/squishcoco


License

30.3 License

30.3.1 How do I install a license key on my Jenkins CI?

To activate a license proceed as followings:

1. Edit the Jenkins CI job which uses Squish Coco.

2. Add a build step of the type “Execute Windows Batch File” for Windows or “Execute Shell” on Linux with thecommand:

cocolic --fetch-license-key=PRODUCTID

PRODUCTID is the product identifier available on the customer account.

3. Execute the job, after the execution, a new license key will be fetched and installed.

4. If this operation is sucessfull, remove the build step used to create a license key.

i For user behind a proxy server, it is necessary to set the proxy server through the command line option--http-proxy. Simply execute cocolic without any arguments to see the list of available options.

30.4 Microsoft® Visual Studio® Installation Issues

30.4.1 Microsoft® Visual Studio® Compiler Warning D9002

Microsoft® Visual Studio® compiler produces following warning:

cl : warning D9002: Unknown Option "--cs-on" ignored.

The path to CoverageScanner compiler wrapper is missing, add it as described in the documentation. (see Chapter G.2,page 212)

30.4.2 Microsoft® Visual Studio® Add-In is Not Installed

Verify that Microsoft® Visual Studio® Add-In is loaded by Microsoft® Visual Studio®:

1. Open "My Document" folder.

2. Open the folder "Visual Studio 2005" (or "Visual Studio 2008")

3. Open the folder "Addins"

4. You should find the files TestCocoonVs2005AddIn.dll and TestCocoonVs2005AddIn.addin. If the files are missing,please reinstall Squish Coco.


Microsoft Visual Studio Installation Issues

5. Start Microsoft® Visual Studio®.

6. Click on "ToolsÕOptions...". Select the entry Environment - Add-in/Macro Security. Verify that%VSMYDOCUMENTS%\addins is present in the path, and that the macros are enabled.

Figure 30.1: Microsoft® Visual Studio® Option Dialog

7. Click on "ToolsÕAdd-In Manager...". The list of addins loaded are displayed. Verify that Squish Coco is checkedand also that the startup flag is set.

Figure 30.2: Add-In Manager

8. Open a C or C++ project.

9. Click on "ToolsÕCode Coverage Build Mode...". Microsoft® Visual Studio® Add-In window should appear.


Microsoft Visual Studio Installation Issues

30.4.3Microsoft® Visual Studio® Add-In is Crashing when "My Documents" is on aNetwork Drive

Due to some security policy, loading DLL in Microsoft® Visual Studio® from a network share is not allowed. The solution is toinstall the Microsoft® Visual Studio® Add-In on the local drive.

Proceed as follows:

1. Create a folder called "C:\VisualStudioPlugins"

2. Start Visual Studio. Open the option dialog and add "C:\VisualStudioPlugins":

(a) Click on Tools MenuÕOptions.

(b) Select EnvironmentÕAdd-In/Macro Security.

(c) Add the path "C:\VisualStudioPlugins".

3. Move the contains of "My Documents\Visual Studio 2010\Addins\*SquishCoco*" to"C:\VisualStudioPlugins".

4. Restart Microsoft® Visual Studio®.


Compiling

Chapter

31Compiling

31.1 Using CoverageScanner with ccache (Unix only)

ccache is a compiler cache which permits to speedup the recompilation of objects. To use CoverageScanner with ccache it isnecessary to set the PATH variable as followings:

export PATH=/opt/SquishCoco/wrapper/bin:/opt/ccache/bin:$PATH

Create then a ccache wrapper got g++ and gcc as follows (as root):

mkdir -p /opt/ccache/binln -s /usr/bin/ccache /opt/ccache/bin/g++ln -s /usr/bin/ccache /opt/ccache/bin/gcc

Which this configuration, calling gcc will:

1. call CoverageScanner’s gcc wrapper

2. which call ccache’s gcc wrapper

3. which finally call the real gcc binary.

31.2 Compiling issue when using the Boost library

When compiling a project using the boost library, the following errors occurs during the compilation:

5>c:\boost\boost_1_47_0\boost\type_traits\type_with_alignment.hpp(294):error : col 2:syntax error

5>Error:Could not insert insert instrumentation in file test.cpp

The project is instrumenting the boost library, it is necessary to disable the instrumentation of it. Simply add--cs-exclude-file-abs-wildcard="c:\boost\boost_1_47_0\*" to the list of the compiler command line arguments.







Microsoft MSBuild Does Not Call CoverageScanner on x86 Platforms

31.3 Microsoft® MSBuild Does Not call CoverageScanner on x86 Platforms

Proceed as follows:

• Open "Visual Studio x64 Win64 Command Prompt (2010)" console.

• Add the PATH to Squish Coco Microsoft® Visual Studio® compiler wrapper for x64 platforms:

set PATH=%SQUISHCOCO%\visualstudio_x64;%PATH%

• Build your project with the property UseEnv set to true:

msbuild /p:UseEnv=true myproject.sln /t:ReBuild


Instrumentation

Chapter

32 Instrumentation

32.1 Compiling Issues of Lambda Expressions with Microsoft® Visual Studio® 2010?

Microsoft® Visual Studio® 2010 des not provide a complete support of C++11 and so expressions in the form “[](int x){return x; }” may not compile once instrumented. To support such expressions, Squish Coco provides an experiemental

instrumentation than can be activated through the command line option --cs-vs2010-lambda.

32.2 How to exclude a source file from the code coverage analysis?

Excluding some source files from the code coverage analysis needs to be performed during the compilation. Two methods arepossible:

1. Compile using CoverageScanner with the command line option --cs-exclude-file-regex,--cs-exclude-file-wildcard and --cs-exclude-path.

2. Use CoverageScanner pragmas to deactivate the instrumentation, by adding the following lines after all #includecommands.


#pragma CoverageScanner (cov-off)#endif

i Adding this portion of code at the top of the source file (and so before all #include commands), will alsodeactivate the coverage analysis of the headers.


My source code contains inline functions, are they instrumented?

32.3 My source code contains inline functions, are they instrumented?

Inline functions are instrumented like other functions. To avoid instrumenting templates and inline functions of 3rd partylibrary or the standard C++ library, CoverageScanner instrument all functions and procedures of the source file and theheaders which are not present in the system directories. On UNIX®, this correspond to /opt, /usr and $QTDIR. On Microsoft®

Windows, this correspond to the standard application installation directory and the QTDIR path. If its header file is in thesedirectories, it is necessary to tell CoverageScanner to instrument it too. The command line options --cs-include-file-regex,--cs-include-file-wildcard and --cs-include-path permits to force the instrumentation of additional files.

32.4 Is it normal that the .csmes file contains a copy of the source code?

Yes. The .csmes file contains all information necessary for CoverageBrowser. That’s why the source code and the preprocessedsource file is also included.


CoverageBrowser

Chapter

33CoverageBrowser

33.1 Performance

33.1.1 Is there a way to minimize the CPU usage?

CoverageBrowser needs a lot of CPU resources and disk accesses to calculate the statistics for each execution. This calculationis performed in background, but can also be deactivated. Just proceed as follows: hide the column Coverage of the dockingwindow "Executions" using the context menu "Show/Hide ColumnsÕCoverage".

33.1.2When I’m exporting the statistics per methods I can only see the 65536 firstentries

This is a limitation of Microsoft Excel or OpenOffice. Excel 97 can only display 16384 rows. Excel 2000 and Excel 2003 candisplay up to 65536 rows and newer version of Excel can display up to 1048576 rows. OpenOffice Calc is limited to using32000 rows.


Unit Tests

Chapter

34 Unit Tests

34.1 How to get the code coverage of a library from my unit tests?

We suppose that the project contains N libraries in the directories libprojN. Each library generated are static libraries, andthey are generated in libprojN. We have N unit tests, all of them are located in testprojM directories.

So the directory tree looks as followings:

libproj1| libproj1.a+-- libproj1.a.csmes

libproj2| libproj2.a+-- libproj2.a.csmes

......libprojN| libprojN.a+-- libprojN.a.csmes

testproj1| testproj1.exe+-- testproj1.exe.csmes

testproj2| testproj2.exe+-- testproj2.exe.csmes

.....testprojM| testprojM.exe+-- testprojM.exe.csmes

First we need to execute the unit tests:

for TESTDIR in testproj*do

cd $TESTDIR./$TESTDIR.execd ..


How to get the code coverage of a library from my unit tests?

done

After execting this bash script, a .csexe file is created for each unit test:

testproj1| testproj1.exe| testproj1.exe.csexe+-- testproj1.exe.csmes

testproj2| testproj2.exe| testproj2.exe.csexe+-- testproj2.exe.csmes

.....testprojM| testprojM.exe| testprojM.exe.csexe+-- testprojM.exe.csmes

We import then the testprojM.exe.csexe file into its testprojM.exe.csmes:

for TESTDIR in testproj*do

cmcsexeimport -m $TESTDIR/$TESTDIR.exe.csmes -e $TESTDIR/$TESTDIR.exe.csexe \--delete --title="$TESTDIR"

done

Now, the files testprojM.exe.csmes are containing the coverage information of the unit tests and the libraries tested. Thefile testprojM.exe.csexe is deleted since that it is no more used.

It is now possible to import the coverage information of the unit tests in the .csmes file of each library tested:

TESTPROJs=$(ls testproj*/testproj*.exe.csmes)for LIBRARY_DIR in libproj*do

LIBPROJ=$LIBRARY_DIR/$LIBRARY_DIR.a.csmescmmerge -a -o $LIBPROJ -i $LIBPROJ $TESTPROJs

done

This command imports all unit test results into each libprojN.a.csmes. The -i command line option permits to ignore thesource files which are not present into libprojN.a.csmes. This permits to ignore the test code during the merge process.


Reports

Chapter

35Reports

35.1 Is is possible to compute a code coverage report of selection of files?

cmreport has the possibility to exclude files from a code coverage report. If we want for example to compute a report of all filespresent only in mysrc directory, then proceed as followings:

1. First, exclude all sources files: the command --exclude-file-abs-wildcard="*" exclude all source files.

2. Finally, include all sources files from the directory mysrc: the command--include-file-abs-wildcard="*/mysrc/*" include all source files whose absolute path constains /mysrc/.

A complete command would be:

cmreport --csmes=project.csmes \--html=project.html \--exclude-file-abs-wildcard="*" \--include-file-abs-wildcard="*/mysrc/*"


Miscellaneous

Chapter

36 Miscellaneous

36.1The generation of an execution report through Windows events does not workwhen logged on an other account?

Windows events are created per user and not accessible to other user on the same machine. Windows support also system wideevents, these event have a name whose prefix is Global\.

So to generate a code coverage report using Windows events from an application launched by an other user, it is necessary touse system whide events.

Example:

• Compile your application with the argument --cs-dump-on-event=Global\COVERAGE,

• Start your application under the account User1,

• On the account User2, execute the command --cs-dump-on-event=Global\COVERAGE.

36.2 Is it possible to debug CoverageScanner API?

To debug functions provided by the CoverageScanner API it is necessary to:

1. force CoverageScanner to keep the source file generated after compiling and linking. Therefore it is necessary to add--cs-keep-instrumentation-files to the compiler and linker command line arguments.

2. compile CoverageScanner library with debug information. Therefore it is necessary to add --cs-libgen=-g for GCC(resp. --cs-libgen=/Zi for Microsoft® Visual Studio®) to the linker command line option.

Once the application is recompiled, it is possible to set a breakpoint in __coveragescanner_save() and step through thesource code.


How to report an issue concerning CoverageScanner?

36.3 How to report an issue concerning CoverageScanner?

It is possible to generate a log file which permits to investigate on issues concerning CoverageScanner. Proceed as follows onMicrosoft® Windows:

1. Click on StartÕProgramsÕSquish CocoÕSupportÕCoverageScanner Log Recording This will install a debugversion of CoverageScanner which will produce a set of log files.

2. Reproduce your issue.

3. In the console window, press enter to install the normal version of CoverageScanner.

4. The log files are located in %SQUISHCOCO%\logfiles. Zip it together and send it to Squish Coco support(k [email protected]).

Proceed as follows on Linux™:

1. In a console window:

cd /opt/SquishCoco/{version of Squish Coco}mv coveragescanner coveragescanner.backcp coveragescannerdbg coveragescanner

This will install a debug version of CoverageScanner which will produce a set of log files.

2. Reproduce your issue.

3. Restore the release version of CoverageScanner:

cd /opt/SquishCoco/binmv coveragescanner.back coveragescanner

4. The log files are located in /tmp/squishcoco or %TEMP%\SquishCoco_logfiles. Zip it together and send it to SquishCoco support (k [email protected]).

! The log file may contains part of your source code. If this is an issue, simply remove the concerned lines withyour favorite editor.


Part VIII Continuous IntegrationServer


Atlassian Bamboo integration

Appendix

AAtlassian Bamboo integration

A.1 Introduction

Atlassian Bamboo is a continuous integration server used to build, test and release software. The Squish Coco plug-in providesseamless Squish Coco integration with Bamboo. Users can add one or more Squish Coco tasks to a Bamboo Job. Each SquishCoco Task can perform the following actions:

1. Import an execution report into an instrumentation data base

2. Merge several instrumentations databases together

3. Generate code coverage report

User can choose only one action to be performed, two actions or all three actions. If more than one action is chosen then theyare executed in mentioned above order.

Figure A.1: Squish Coco Task configuration overview

After executions finish, Artifacts for Squish Coco (like Coverage Report) can be stored to allow efficient defect troubleshooting.


Agent configuration

A.2 Agent configuration

An Agent is a service that executes Bamboo builds, tests and deployments. The Agent Capabilities setting allows users tospecify which agents are capable of executing Squish Coco jobs. In order to execute it , we must first install Squish Coco atAgent host. Next, we need to setup Agent-specific capability called ?coco? with Squish Coco installation directory as a Value.(go to Bamboo administration|BUILD RESOURCES|Agents|AgentName|Agent-specific capabilities). Later on, during Jobconfiguration, we need to set ?coco? capability existence as a Job requirement. This way, Squish Coco jobs will be executedonly on Agents where Squish Coco in installed.

A.3 Importing an execution report

This action calls cmcsexeimport to import execution report (CSExe file) into an instrumentation data base (CSMes file). Userneed to provide absolute path or relative path (from Bamboo build working directory) for both CSExe file and CSMes file andexecution title. When this action is activated, the following command will be executed on Agent host:

cmcsexeimport --debug --title=<Title> -m <CSMesFile> -e <CSExeFile>

A.4 Merging instrumentation databases

This action calls cmmerge to merge several instrumentation databases (CSMes files) together. We need to provide CSMesoutput file full or relative path name that will be generate as a result of merge. If we provide relative path (or just filename),then CSMes output file will be generated in Bamboo working directory or relative sub directory. CSMes input files need tobe specified, multiple files can be used separated by comma. When this action is activated, the following command will beexecuted on Agent host:

cmmerge ?verbose -o <CSMesOutputFile> <CSMesInputFile1> ? <CSMesInputFileN>

Additionally user can choose option to ?Merge only instrumentations and executions present in the reference file?. This optionis useful for importing CSMes files with Unit Tests. When activated, used need to provide the name of CSMes reference file.This will cause that above cmmerge command will be executed with additional ?-i? option.

cmmerge ?verbose -o <CSMesOutputFile> -i <CSMesRefFile> <CSMesInputFile1> ? <CSMesInputFileN>

A.5 Generating coverage report

The last action calls cmreport to generate HTML Report. We need to provide CSMes file name (either absolute or relativepath). CSMes file shall contain previously imported execution report (i.e. by using first action offered by plugin).

cmreport ?title=<BambooJobName> -m <CSMesFile> ?debug ?html=coco/report.html

Generated report will be stored in coco subdirectory, therefore we can define Artefact with ?coco/**? as a Copy pattern. Thisallows us to view the report directly in Bamboo after job execution. The report title is build using Bamboo Job name.


Use Case Examples

A.6 Use Case Examples

A.6.1 Coverage Report from single execution

We need to activate first action ?Import an execution report into an instrumentation data base? to import execution report(CSEXE file) into an instrumentation data base (CSMES). Finally, we need to activate third action ?Generate code coveragereport?. CSMES file defined for this action is the same as for first action.

Figure A.2: Squish Coco Task configuration for example addressbook application

A.6.2 Coverage Report for Unit Tests

In this scenario we would like to generate coverage report for unit tests execution. To achieve this scenario in Bamboo we needto use Squish Coco task twice in one job.

After the build we have two CSMes files: one for our application and second for Unit Tests. We need to merge thoseinstrumentation data bases where CSMes file for our application will be a reference during the merge. After the merge newCSMes output file is created.


Use Case Examples

Figure A.3: Squish Coco Task configuration for merging Unit Tests executions

Using second Squish Coco task execution report from Unit Test run need to be imported into CSMes file (which was created inprevious task - CSMes output file). Last action ?Generate code coverage report? is used to generate HTML coverage report.


Use Case Examples

Figure A.4: Squish Coco Task configuration for importing execution and generating report


Part IX Appendix


Release Notes

Appendix

B Release Notes

B.1 Squish Coco v3.2.3

Changes between Squish Coco v3.2.3 and Squish Coco v3.2.2:

CoverageScanner :

• Bug Fix: Memory usage reduced. This fixes an "out-of-memory" error message during the compilation.

• New Feature: Explicit initialization of the CoverageScanner API is no longer necessary for C# DLLs. (The programInjectModuleInitializer.exe is no longer needed)



CoverageScanner :

• Bug Fix: More meaningful instrumentation of the C# keywords using, lock, unsafe and fixed.

• New Feature: If the debugger is connected, CoverageScanner runtime information is displayed in the Microsoft®

Visual Studio® console.

• New Feature: In C# files, the “dump on event” feature can be enabled through a macro definition.

Microsoft® Visual Studio® Add-In:

• New Feature: Configuration of the debug output of Squish Coco.

• New Feature: Configuration of the Windows event handler.


Squish Coco v3.2.1



CoverageScanner :

• New Feature: Support of Shift-JIS text encoding.

• New Feature: Execution report of Silverlight C# binaries is stored on an Isolated Storage when storing on the localfile system is not allowed.

• Bug Fix: Support of strong-named C# assemblies.

CoverageBrowser:

• New Feature: The Preferences dialog now allows to switch off syntax highlighting.



CoverageScanner :

• New Feature: Precise error message when a merge failure occurs.

CoverageBrowser:

• New Feature: Thresholds for the overall statistics.

• New Feature: Statistics of manual validated instrumentation (for each function, source code and for the completeproject).

cmreport:

• New Feature: Statistics of manual validated instrumentation (for each function, source code and for the completeproject) for the HTML and CSV report.

cmmerge:

• Speed improvements.

• New Feature: --delete permits to delete all input files after a successful merge.

• New Feature: --blackbox permits to generate an instrumentation database for black-box testing.



Squish Coco:

• New Feature: NUnit addin.

Tool Selector :


Squish Coco v3.0.2

• Bug Fix: Detection of CygWin installations for x64 systems corrected.

CoverageScanner :

• VisualDSP® support.

• Bug Fix: Support of UTF16 source files.

• New Feature: For embedded systems, CoverageScanner is able to generate execution reports in 8.3 format. In thiscase the report extension is .cse and not .csexe.

• New Feature: New command line switch --cs-no-abort-on-error: if used, instrumentation errors raised whileprocessing C/C++ source code no longer abort the build.

cmreport:

• New Feature: Refactoring of the HTML report:

– New Feature: Single HTML file and full HTML report have now the same look and feel.– New Feature: Source directories statistics in a clickable tree view.– New Feature: Class directories statistics in a clickable tree view.– New Feature: Statistic tables are interactively sortable.– New Feature: Tool-tip for a detailed information on each table item.

• New Feature: CSV table export for all informations that can be exported through a HTML report.

• New Feature: The source, execution and function dialog can now be interactively switched between a list and treeview.

• New Feature: CoverageBrowser reacts automatically when the instrumentation database file is regenerated andreload it automatically.

CoverageBrowser:

• New Feature: The source, execution and function dialog can now be interactively switched between a list and treeview.

• New Feature: CoverageBrowser reacts automatically when the instrumentation database file is regenerated andreload it automatically.



Squish Coco:

• New Feature: Solaris support.

CoverageBrowser:

• Bug Fix: EMMA-XML Report generation corrected when excluding source files.

• New Feature: Drag and drop of .csmes and .csexe files supported.

CoverageScanner :

• New Feature: Format specifier %c added to --cs-output.

• Bug Fix: Issues with strings in __declspec extension fixed.

• Bug Fix: Deal with non-instrumentable OpenMP code.

• Bug Fix: C++11 string literal support. (example: R"delim(")delim")

• Bug Fix: UTF-8 source file can now be compiled.


Squish Coco v3.0.1

• Bug Fix: C++ Line comments ending a source file are no more generating a parsing issue.

• Bug Fix: C# linq expressions are now instrumented.

• New Feature: New C# API function CoverageScanner.__coveragescanner_init().

• New Feature: Dead code detection in constant boolean expressions:

– Dead code detection for if ( false ) ... or for if ( true ) ... branches.– Expression do statement while ( false ) is treated if the statement was not included into ado ... while ( ... ) construct.

– Dead code detection of while ( false ) .... loops.– Statements ’while ( true ) ...’ and ’do ... while ( true );’ are now treated as the infinite loop

’for(;;) ...’. The conditions are no more instrumented to avoid false negative coverage instrumentations.

• Bug Fix: New command line switch for C++ mixed mode: --cs-architecture=CLR

• Bug Fix: Several parsing issue for C# and C++ code fixed.

• Bug Fix: Support of multiple header files included with/FI when generating precompiled headers.

• Bug Fix: C# support for Microsoft® Visual Studio® 2013.

Tool Selector :

• New Feature: generation of all wrappers for all installed version of GCC compilers.



CoverageBrowser:

• New Feature: Validating manually an instrumentation validates also all depending instrumentations. For example,when marking a return statement as validated, all sequential lines before are also automatically validated.

CoverageScanner :

• New Feature: Source code annotation though code comments.

• Bug Fix: clang compiler support on Apple® Mac OS X.


• Bug Fix: Excluding source directories from the instrumentation is working again.


Changes between Squish Coco v3.0.0 and Squish Coco v3.0.0-pre1:

CoverageScanner :

• Bug Fix: Line coverage information of nested compound statements are now computed correctly.

• Change: --cs-include-path/--cs-exclude-path works now recursively on all subdirectories.

• New Feature: --cs-output supports now format specifiers (%f, %P, . . . )


Squish Coco v3.0.0-pre1

• Bug Fix: lib.exe wrapper support now Module-Definition (.def) files

• Bug Fix: Compilation of expression in the form CLASS c = a && b are now supported even if the operator || or&& are not logical operators.

CoverageBrowser:

• New Feature: Disabling the difference view of a source file is possible.

• New Feature: Importing reviewer comments from a previous instrumentation database.

cmmerge:

• New Feature: Importing reviewer comments (–reviews-only command line option)

B.9 Squish Coco v3.0.0-pre1

Changes between Squish Coco v3.0.0-pre1 and Squish Coco v2.1.8:

CoverageBrowser:

• New Feature: Usability enhancement of the folding functionality in the source window:

– possibility to define a number of lines before and after the folded line part of the context and which are notfolded.

– folded lines are chosen the instrumentation filter dialog.

• New Feature: For the code coverage at branch or decision/condition level, the executed (resp. not executed) lineswhich are not instrumented are displayed in pastel green (resp. pastel red).

• New Feature: Line coverage information is now also available in the preprocessed view of the source code.

• New Feature: Automatic copying to the clipboard of a selected source code fragment.

• New Feature: Generation of Cobertura reports.

CoverageScanner :

• New Feature: simple support of GNU linker script for MinGW.

• New Feature: clang support on Apple® Mac OS X.

• New Feature: Configuration of instrumentation warnings and errors.

• Bug Fix: throw statement are now instrumented in expressions. (example: int a = b ? 0 : throw 0 ;)

• Bug Fix: /Yc and /Yu can now be used simultaneously for the Microsoft® Visual Studio® compiler.

• New FeatureBETA: Tcl support.

• New FeatureBETA: C# support.

cmreport:

• New Feature: Generation of Cobertura reports.




Squish Coco v2.1.7

CoverageBrowser:

• New Feature: Configuration of the color scheme.

• New Feature: Excluding source files interactively from the instrumentation.

• New Feature: Configuration of the maximum number of threads which can be used for a computation.

• New Feature: Progress information during the generation of a code coverage report.

CoverageScanner :

• New Feature: Better detection of dead code after a switch/case statement.

• New Feature: Optimized handling of C++ header which permits to reduce the size of the instrumentation databaseand the time for generating reports.

cmreport:

• New Feature: Excluding source files from the report generation.

• New Feature: Configuration of the maximum number of threads which can be used for a computation.

• Change: Function parameter are now appended to each function in the EMMA-XML report.

cmcsexeimport:

• New Feature: cmcsexeimport is now able to import more than one execution report file.



CoverageBrowser:

• Change: Navigation buttons are now grouped into one toolbar.

• Speed improvement of the source code viewer.

• Speed improvement of the computing of the list of tests which are executing a source line.

• Speed improvement of the of the text search in the source code function.

• New Feature: Source code viewer:

– New Feature: Sources are displayed in a tabbed view.– New Feature: Preview of the instrumentation of a whole file into the scroll bar.– New Feature: Tooltip which displays the detail of an explanation of a source line.– New Feature: More intuitive text search function (search when typing into the source code window) and

interactive highlighting of the matched patterns.– New Feature: Dock/undock line number column and statistic column.

CoverageScanner :

• Bug Fix: CoverageScanner is now able to handle "Whole Program Optimization" (option /GL) and "Use Link TimeCode Generation" (option /LTCG) of Microsoft® Visual Studio®.

• New Feature: Command line option --cs-vs2010-lambda which permits to instrument lambda functions ofMicrosoft® Visual Studio® 2010

• New Feature: Support for "for each" C++ syntax extension of Microsoft® Visual Studio®.

• Bug Fix: Support of ccache wrapper.

• Bug Fix: CoverageScanner detects now dead code after a try/catch block.

• Bug Fix: Precompiled header issues.


Squish Coco v2.1.6

• Bug Fix: Handling source code lines which are ending with a backslash.

cmreport:

• Speed improvement of the generation of EMMA-XML reports.

• Speed improvement of the generation of HTML reports.

• New Feature: HTML report displays the detailed instrumentation report in a tooltip.

• Change: HTML report is generated in a directory whose name ends with _html. This avoids a name clash underUnix where the executable has in most of the cases no extension.



CoverageBrowser:

• New Feature: Possibility to choose the source code type used for comparing software releases (original source codeor preprocessed source code)

CoverageScanner :

• New Feature: New command line options which permits to specify command line arguments into a text file(--cs-option-file).

• New Feature: New command line options which permits to specify to select directly the CoverageScanner profile(--cs-profile).

cmreport:

• New Feature: possibility to customize the EMMA-XML report



CoverageScanner :

• New Feature: Support for precompiled headers for Microsoft® Visual Studio® compiler.

• New Feature: Xoreax IncrediBuild support (http://www.incredibuild.com/incredibuild-product-overview.html)

• Bug Fix: Wildcard expressions on relative paths are computed on the canonical path. (example:--cs-include-file-wildcard=include/* matches include/foo.h and module/../include/bar.h)

• New Feature: New command line options which permits to specify which extensions are used forC++ (--cs-cpp-ext) and C (--cs-c-ext) files.

• New Feature: New command line option which permits to set the tool chain path (--cs-native-toolchain).

• Change: Verbose command line option (--cs-verbose) permits to select if the build, the instrumentation or theCoverageScanner API is instrumented. The verbose output of the instrumented file is generated into a log file(name of the execution report with the extension.cslog).

• Bug Fix: -m32 and -m64 command line support for GCC.


http://www.incredibuild.com/incredibuild-product-overview.html

http://www.incredibuild.com/incredibuild-product-overview.html

Squish Coco v2.1.4

• Bug Fix: Compilation issue on Microsoft® Visual Studio® for instrumented C++0x lambda function corrected.

• Bug Fix: Support of __stdcall as default calling convention when compiling with Microsoft® Visual Studio®.

• New Feature: New command line option for UNIX® platform which permits to generate a coverage report on thereception of a signal (--cs-dump-on-signal).

• New Feature: New command line option for Microsoft® Windows platform which permits to generate a coveragereport on the reception of an event (--cs-dump-on-event).

• Change: Instrumentation tables of libraries are chained together during the runtime when compiling with Microsoft®

Visual Studio®. The old behaviour can be restored with --cs-link-instrumentation-tables.

cmreport:

• Change: cmreport generate now per default a code coverage report at decision/condition level with line coveragefor uncovered lines.


• Change: Microsoft® Visual Studio® Add-In does not disable the precompiled headers anymore.



CoverageBrowser:

• New Feature: Import summary display the imported execution comment.

• New Feature: New instrumentation mode available which permits to displays the computed line coverage ofuninstrumented lines. This enables the readability of the source code but have no impact on the coverage statistic.

CoverageScanner :

• Bug Fix: Support for /clr:safe and clr:pure command line option.

• Optimization: CoverageScanner calls now directly the native compiler without calling a second instance of it.

• Bug Fix: The line coverage was wrongly calculated before some switch/case statements.

License Manager:

• New Feature: The license manager is now a wizard dialog.

• New Feature: Command line tool to activate a license on Windows and Unix



CoverageScanner :

• New Feature: /MP command line (parallel build) supported for Microsoft® Visual Studio® compiler.

• Bug Fix: Parsing issues of C++11 templates corrected.

• New Feature: The Build Environment Selection tool generate CoverageScanner wrapper for CygWin.


Squish Coco v2.1.2


• New Feature: Since that CoverageScanner instruments now all non system folders, the selection dialog of the list ofdirectories to include is removed. A dialog which permits to select the list of directories to exclude replace it.

Linux™ installer:

• New Feature: Possibility to specify a custom installation directory.

cocolic:

• New Feature: Command line interface on Unix.

• Bug Fix: More verbose error message in the case of network failures.



CoverageScanner :

• Bug Fix: Multi line #pragma supported.

• Bug Fix: ::new keyword supported.


• Bug Fix: Support of solutions with folder.

• Bug Fix: Support of solutions which mix C++ and C# projects.



CoverageScanner :

• Change: CoverageScanner instruments now per defaults all sources files except those which are in the systemdirectory (/usr, /opt or C:\Windows) or the application installation directory (C:\Program Files).

• New Feature: Automatic detection of the runtime library necessary when compiling using Microsoft® Visual Studio®.Adding /MD or /MT to the build command argument of the CoverageScanner library is no more necessary.

• Bug Fix: Crash when using wildcard expression on file name on UNIX®.

• New Feature: The instrumentation table are now generated during the runtime. This permits to handle stubs, whichoverwrite objects from libraries.

• New Feature: Constructor which use member initialization list are instrumented even if their body is does notcontains code.

• New Feature: C++ CLR language extension of Microsoft® Visual Studio® supported.

CoverageBrowser:

• New Feature: Navigation buttons are disables if they are irrelevant. After reaching the end of search, the navigationbuttons start from the beginning again.

• Bug Fix: Clicking on a function in the method list, show it entirely in the source window.


Squish Coco v2.1.0

• Bug Fix: Statistics count was wrongly computed when the instrumentation is computed with line coverage supportand the test count mode was activated.

• New Feature: Importing an execution report does not generate an error message any more. If some executionscannot be imported, the information window is automatically shown. A list of imported executions and errors aredisplayed on it.

• New Feature: Windows x64 package available.

cmreport:

• New Feature: EMMA-XML report generated now condition coverage statistics. This requires Jenkins EMMA-XMLplugin version 1.29 or above.

• New Feature: JUnit report generation.



Squish Coco:

• New Feature: Recognition of dead code inside a function.

• New Feature: Function coverage.

• New Feature: Line coverage.

CoverageScanner :

• Code coverage count is enabled per default.

cmreport:

• Bug Fix: EMMA-XML report corrected: the line field correspond now to the line coverage.



Squish Coco:

• Bug Fix: License activation issues.



cmreport:


Squish Coco v2.0.1

• Improving the speed of the production of HTML reports through parallelization of the generation of the sourcecode view.

CoverageScanner :

• Bug Fix: __try and __finally keyword now handled on Windows for C files.

• Bug Fix: g++ compiler wrapper treats files with .c extension as C++ files.

• Bug Fix: On Windows, command line containing directory name which ends with a backslash are now correctlyescaped.

• Bug Fix: ar and ld wrapper are now working, and do not report that the profile cannot be found.

• New Feature: The code behind the macro BOOST_FOREACH is no more analyzed, unless the command line--cs-no-boost is used.

• New Feature: New command line option --cs-output-abs.

• Change: The support for Qt3, Qt4 and Boost is per default activated. Disabling it can be performed using commandline switches --cs-no-qt3, --cs-no-qt4 and --cs-no-boost.

• Change: The full instrumentation at decision and condition level is now the default setting for the instrumentation.



CoverageScanner :

• Bug Fix: Support setting define which contains escaped quotes per command line (ex:"-DVERSION=\"version 1.1\"")

• Bug Fix: Compilation issue of TextEdit sample when spaces are in the path corrected.

License Dialog:

• Bug Fix: Retrieving a license key is now interruptible and displays a progress bar.


Changes between TestCocoon v1.6 and Squish Coco v2.0.0:


• New Feature: Support for Microsoft® Visual Studio® 2010.

• Bug Fix: x64 platform are now recognized and handled.

CoverageBrowser:

• New Feature: Supporting linking several times the same object into an application.

• New Feature: Possibility to filter out execution which doesn’t cover any code.

• New Feature: Possibility to filter out source files and functions which are not instrumented.

• Bug Fix: Calculation of statistics is able to saturate all available CPU.

• Bug Fix: After opening a new instrumentation database, no executions become selected.


Squish Coco v2.0.0

• New Feature: Possibility to choose between the native file dialog and the dialog provided by Qt.

• New Feature: Displaying the execution of a reference release into the list of execution. Executions of the referencerelease are strikeout, new executions are underlined.

• New Feature: Generation of browsable HTML output.

CoverageScanner :

• Empty functions are no more instrumented.

• New Feature: C++0x support.

• New Feature: Verbose output of the CoverageScanner output when using command line option --cs-verbose.

• New Feature: Saving the contains of a tree view waits until all statistics are computed.

• New Feature: Q_FOREACH macro es now handled to avoid false negative code coverage statistics. Once a look isexecuted once, the Q_FOREACH macro is considered as 100% covered.

cmcsexeimport:

• New Feature: Execution comments in HTML form are parsed from the execution report. This permits to add extralog information to the execution report.

cmreport:

• Bug Fix: Correction of an EMMA-XML issue: global statistics are now ordered in EMMA-XML report as following:Line, Method, Class and then Block

• New Feature: Generation of browsable HTML output.


Code insertion

Appendix

C Code insertion

(This chapter is a continuation of the example in Section 8.1.)

After the detection phase, CoverageScanner inserts the instrumentation code itself as the following examples illustrate. (Notethat the inserted code is displayed in bold blue.)

Sequential Statement:The statements are instrumented before their execution. The instrumentation consists in allocating a Boolean variablewhich detects if the code was executed or not.

Example:

a=foo();a++;break;

will be changed into:

a=foo();a++;{ inst[0]=1; break; }

inst[0] is set to 1 if the ‘break’ statement is executed.

Conditional Statements and Boolean Expressions (full instrumentation):For Boolean expressions the same principle applies except that in addition to recording the execution itself, the state (trueor false) is also recorded.

Example:

if ( a<b )return 1;


if ( (a<b) ? inst[0]=1 : inst[1]=1,0 )return 1;

inst[0] is set to 1 if the Boolean expression a<b was true.

inst[1] is set to 1 if the Boolean expression a<b was false.


Code insertion

Conditional Statements and Boolean Expressions (partial instrumentation):In some cases, recording the value of a Boolean expression is unnecessary for sequential statement instrumentation.For example, the statement if (b)return 0; else return 1; is completely covered by a statement coverage—afterall, recording whether b becomes true or false doesn’t provide any extra information. Similarly, for the statementif (b)return 0; it is only necessary to check if b was false. By default, Squish Coco suppresses the generation ofredundant instrumentation in order to minimize the instrumented code’s size and to maximize execution speed.

Example:

if ( a<b )return 1;


if ( a<b )return 1;

else inst[0]=1 ;

inst[0] is set to 1 if the Boolean expression a<b was false. Since the statement coverage records the instructionreturn 1;, it is not necessary to record if a<b was true;

The generated instrumentation code that needs to be inserted to provide statement coverage for the earlier foo() functionexample means that the function’s code is transformed to the code shown here:

1 char inst[5];2 void foo()3 {4 bool found=false;5 for (int i=0; (i<100) && (!found); ++i)6 {7 if (i==50 ) { inst[0]=1;break;}8 if (i==20 ) { inst[1]=1;found=true;}9 if (i==30 ) { inst[2]=1;found=true;}

10 inst[3]=1; }11 printf("foo\n");12 inst[4]=1; }

Figure C.1: Code coverage instrumentation at branch level

If we insert the instrumentation code necessary to support decision coverage into this example, the resulting code will look likethis:

1 char inst[13];2 void foo()3 {4 bool found=false;5 for (int i=0; ((i<100 && !found)?inst[0]=1:inst[1]=1,0); ++i)6 {7 if ((i==50?inst[2]=1:inst[3]=1,0)) {inst[4]=1; break;}8 if ((i==20?inst[5]=1:inst[6]=1,0)) {inst[7]=1; found=true;}9 if ((i==30?inst[8]=1:inst[9]=1,0)) {inst[10]=1; found=true;}


Figure C.2: Code coverage instrumentation at decision level


Code insertion

If we insert the instrumentation code necessary to support condition coverage into this example, the resulting code will looklike this (except that we have wrapped one line to fit better on the page):

1 char inst[15];2 void foo()3 {4 bool found=false;5 for (int i=0;((i<100)?inst[0]=1:inst[1]=1,0) &&6 ((!found)?inst[2]=1:inst[3]=1,0); ++i) {7 if ((i==50?inst[4]=1:inst[5]=1,0)) {inst[6]=1; break;}8 if ((i==20?inst[7]=1:inst[8]=1,0)) {inst[9]=1; found=true;}9 if ((i==30?inst[10]=1:inst[11]=1,0)) {inst[12]=1; found=true;}


Figure C.3: Full code coverage instrumentation at condition level

Here is what the code would look like if we inserted the partial instrumentation code for condition coverage (again, with oneline wrapped):

1 char inst[12];2 void foo()3 {4 bool found=false;5 for (int i=0; ((i<100)?inst[0]=1:inst[1]=1,0) &&6 ((!found)?inst[2]=1:inst[3]=1,0); ++i) {7 if (i==50 ) {inst[4]=1; break;} else inst[5]=1;8 if (i==20 ) {inst[6]=1; found=true;} else inst[7]=1;9 if (i==30 ) {inst[8]=1; found=true;} else inst[9]=1;


Figure C.4: Partial code coverage instrumentation at condition level


Code Coverage Benchmarks

Appendix

DCode Coverage Benchmarks

D.1 Test Algorithm

The sorting algorithm used for the tests is quicksort.Source code:

#define SIZE 10000000#define NB_TESTS 4#include <stdio.h>#include <stdlib.h>#include <sys/times.h>#include <time.h>

void sort (int array[],int size);

main( ){int i,t ;long duration[NB_TESTS];long duration_val;clock_t starttime,endtime ;int *array;

for (t=0;t<NB_TESTS;t++){starttime=clock();array=(int*)malloc(sizeof(int)*SIZE);for ( i = 0 ; i < SIZE ; i++ )array[i]=(i*7)%SIZE;

sort(array,SIZE);free(array);endtime=clock();duration[t]=1000*((double)(endtime-starttime))/CLOCKS_PER_SEC;fprintf(stderr," %dms",duration[t]);

}duration_val=duration[0];for (t=1;t<NB_TESTS;t++)


Benchmarks

if (duration_val>duration[t])duration_val=duration[t];

printf("%d",duration_val);}

static void quicksort ( int array[], int low, int high ){int pos ;if ( low < high ){int item, i, j, t ;item = array[low] ;i = low ;j = high ;while ( i < j ){while ( array[j] > item )j = j - 1 ;

while ( array[i] <= item && i < j )i = i + 1 ;

if ( i < j ){t = array[i] ;array[i] = array[j] ;array[j] = t ;

}}pos = j ;t = array[low] ;array[low] = array[pos] ;array[pos] = t ;

quicksort ( array, pos + 1, high ) ;quicksort ( array, low, pos - 1 ) ;

}}

void sort (int array[],int size){quicksort ( array, 0, size-1 ) ;

}

D.2 Benchmarks


Benchmarks

Compiler Normal Execution(time)

ExecutionBranch Coverageor Function Cover-age(time)

ExecutionDecision CoverageorLine Coverage(time)

ExecutionCondition Coverage(time)

GCC without optimization 6840ms 8030ms (+17.3%) 9400ms (+37.4%) 9830ms (+43.7%)GCC with optimization -Os 4160ms 4430ms (+6.4%) 6180ms (+48.5%) 6360ms (+52.8%)GCC with optimization -O1 3530ms 4250ms (+20.3%) 5420ms (+53.5%) 5970ms (+69.1%)GCC with optimization -O2 3950ms 4040ms (+2.2%) 5080ms (+28.6%) 5320ms (+34.6%)GCC with optimization -O3 3860ms 4030ms (+4.4%) 5060ms (+31%) 5230ms (+35.4%)

Table D.1: Benchmark (sorting algorithm)

Compiler Native Compilation(bytes)

Branch CoverageorFunction Coverage(bytes)

Decision CoverageorLine Coverage(bytes)

Condition Coverage(bytes)

GCC without optimization 2128 3452 (+1324) 3996 (+1868) 4052 (+1924)GCC with optimization -Os 1940 3196 (+1256) 3716 (+1776) 3760 (+1820)GCC with optimization -O1 1968 3172 (+1204) 3736 (+1768) 3792 (+1824)GCC with optimization -O2 2100 3452 (+1352) 4096 (+1996) 4160 (+2060)GCC with optimization -O3 3544 3548 (+4) 4364 (+820) 4428 (+884)

Table D.2: Sorting algorithm code size of the object


License Agreement

Appendix

ELicense Agreement

Squish Coco Commercial License AgreementAgreement version 1.1

This froglogic Squish Coco Commercial License Agreement ("Agreement")is made by and between froglogic GmbH ("froglogic"), and you (eitheran individual or a legal entity) ("Licensee").

IMPORTANT-READ CAREFULLY:

1. froglogic developed and originated Squish Coco including sourcecode, documentation and example programs ("Licensed Software"). TheLicensed Software is protected by copyright laws and internationalcopyright treaties, as well as other intellectual property laws andtreaties. froglogic retains all rights not expressly granted.

2. By installing, copying, or otherwise using the Licensed Software,Licensee agrees to be bound by the terms of this Agreement. IfLicensee does not agree to the terms of this Agreement, Licenseeshould not install, copy, or otherwise use the Licensed Software. Inaddition, by installing, copying, or otherwise using any updates orother components of the Licensed Software that Licensee receivesseparately as part of the Licensed Software ("Updates"), Licenseeagrees to be bound by any additional license terms that accompany suchUpdates. If Licensee does not agree to the additional license termsthat accompany such Updates, Licensee may not install, copy, orotherwise use such Updates.

3. Upon Licensee’s acceptance of the terms and conditions of thisAgreement, froglogic grants Licensee the right to use the LicensedSoftware solely for Licensee’s internal business purposes in themanner provided below.

4. Options(a) Evaluation License

Licensee may use the Licensed Software for a limited time(evaluation period) to be determined and stated by froglogic,solely for the purpose of determining whether the LicensedSoftware meets Licensee’s requirements.

After the evaluation period has finished, Licensee must either:


License Agreement

(i) discontinue use of the Licensed Software,(ii) acquire a commercial license (see section (b)),

Commercial use of a derived work of the Licensed Software isnot allowed under the scope of the evaluation license.

(b) Node-Locked LicenseLicensee may install and use the Licensed Software locally onone identified computer (Node), with the Node executing theLicensed Software for only one Authorized User. Licensee maytransfer a license certificate to another individual and Nodeafter notification to and approval from froglogic, but not moreoften than every 4 weeks, provided the future Licensee isemployed or contracted by the same individual or legal entityas the initial Licensee.

(c) Floating User LicenseLicensee may install the Licensed Software on an unlimitednumber of computers. All machines using the Licensed Softwaremust have the ability to communicate with a license server. Thenumber of Floating Users that may use a component of theLicensed Software concurrently at any time is limited by thenumber of Floating User Licenses purchased for suchuse. Regardless of whether such use is by persons or computersystems.

5. Licensee may modify the Licensed Software except altering orremoving any details of ownership, copyright, trademark or otherproperty right connected with the Licensed Software.

6. Licensee may not distribute the Licensed Software, modified orunmodified, separately or as part of any software package.

WARRANTY DISCLAIMER

7. The Licensed Software is licensed to Licensee "as is". To themaximum extent permitted by applicable law, froglogic on behalf ofitself and its suppliers, disclaims all warranties and conditions,either expressed or implied, including, but not limited to, impliedwarranties of merchantability, fitness for a particular purpose, titleand non-infringement with regard to the Licensed Software.

LIMITATION OF LIABILITY

8. If, froglogic’s warranty disclaimer notwithstanding, froglogic isheld liable to Licensee, whether in contract, tort or any other legaltheory, based on the Licensed Software, froglogic’s entire liabilityto Licensee and Licensee’s exclusive remedy shall be, at froglogic’soption, either (A) return of the price Licensee paid for the LicensedSoftware, or (B) repair or replacement of the Licensed Software,provided Licensee returns to froglogic all copies of the LicensedSoftware as originally delivered to Licensee. froglogic shall notunder any circumstances be liable to Licensee based on failure of theLicensed Software if the failure resulted from accident, abuse ormisapplication, nor shall froglogic under any circumstances be liablefor special damages, punitive or exemplary damages, damages for lossof profits or interruption of business or for loss or corruption ofdata. Any award of damages from froglogic to Licensee shall not exceedthe total amount Licensee has paid to froglogic in connection with


License Agreement

this Agreement.

SUPPORT AND UPDATES

9. If this license, as specified in the license certificate, includesa support and update agreement, Licensee will be eligible to receiveemail based software support and access to Updates to the LicensedSoftware for a period not to exceed one year from the date of initialdelivery, in accordance with froglogic’s then current policies andprocedures, if any. Such policies and procedures may be changed fromtime to time.

GENERAL PROVISIONS

10. This Agreement may only be modified in writing signed byauthorized representatives of Licensee and froglogic. In case of aconflict between this Agreement and the terms of any purchase order orother ordering document, this Agreement shall prevail. If any provisionof this Agreement is found void or unenforceable, the remainder willremain valid and enforceable according to its terms. If any remedyprovided is determined to have failed for its essential purpose, alllimitations of liability and exclusions of damages set forth in thisAgreement shall remain in effect.

11. This Agreement shall be construed, interpreted and governed by thelaws of Germany, the legal venue to be Hamburg City Court. froglogicreserves all rights not specifically granted in this Agreement.


Customizing I/O of CoverageScanner library

Appendix

FCustomizing I/O of CoverageScanner library

F.1 Custom I/O using C file access

The following example shows how to generate the execution report using the standard C file API.

To compile the example on Microsoft® Windows:

cscl customiofile.c

To compile the example on Linux™ or Apple® Mac OS X:

csgcc customiofile.c -o customiofile

Source code:

#include <stdio.h>#include <string.h>#define VERBOSE 1

static int csfputs(const char *s, void *stream){#if VERBOSEfprintf(stderr,"csfputs:%s\n",s);

#endifreturn fputs(s, (FILE *)stream);

}

static void *csfopenappend(const char *path){#if VERBOSEfprintf(stderr,"csfopenappend:%s\n",path);

#endifreturn (void*)fopen(path,"a+");

}

static void *csfopenread(const char *path){


Custom I/O using C file access

#if VERBOSEfprintf(stderr,"csfopenread:%s\n",path);

#endifreturn (void*)fopen(path,"r");

}

static void *csfopenwrite(const char *path){#if VERBOSEfprintf(stderr,"csfopenwrite:%s\n",path);

#endifreturn (void*)fopen(path,"w");

}

static char *csfgets(char *s, int size, void *stream){char * ret;ret = fgets(s, size, (FILE *)stream);

#if VERBOSEfprintf(stderr,"csfgets:%s\n",s);

#endifreturn ret;

}

static int csremove(const char *filename){#if VERBOSEfprintf(stderr,"csremove:%s\n",filename);

#endifreturn remove(filename);

}

static int csfclose(void *fp){#if VERBOSEfprintf(stderr,"csfclose\n");

#endifreturn fclose((FILE*)fp);

}

int main(){char location[256];int lg_location;

printf(".csexe file name (without extension}:"); fflush(stdout);fgets(location,sizeof(location),stdin);lg_location=strlen(location);if (lg_location)location[lg_location-1]=’\0’; // strip \n

#ifdef __COVERAGESCANNER____coveragescanner_set_custom_io( csfgets,

csfputs,csfopenappend,csfopenread,csfopenwrite,


Custom I/O using SFTP protocol

csfclose,csremove);

__coveragescanner_install(location);#endif}

F.2 Custom I/O using SFTP protocol

The following example shows how to generate the execution report directly on a SFTP server. The SFTP server is part ofSSH v2 and is available on most of the Unix platforms. On Microsoft® Windows, a free SSH server can be downloaded fromhttp://www.freesshd.com.

To compile the example on Microsoft® Windows:

1. Download libSSH2 from http://www.libssh2.org. Generate the library and set the environment variable LIBSSH2 tothe location of the libSSH2 source code.

2. To compile the example:

cscl %LIBSSH2%\win32\debug_dll\libssh2.lib -DWIN32 --cs-libgen=/MTd/MTd -I %LIBSSH2%\include ws2_32.lib customiosftp.c

3. Execute custom_io_sftp.exe.

To compile the example on Linux™:

1. Install the development package of libssh2.

2. To compile the example:

csgcc -lssh2 customiosftp.c -o customiosftp

3. Execute custom_io_sftp.

Source code:

#define VERBOSE 1

#ifdef WIN32# include <winsock2.h>#define LIBSSH2_WIN32#define LIBSSH2_API#else#include <unistd.h># include <sys/socket.h># include <netinet/in.h>#include <arpa/inet.h>#endif#include <libssh2.h>#include <libssh2_sftp.h>


http://www.freesshd.com

http://www.libssh2.org


#include <sys/types.h>#include <fcntl.h>#include <errno.h>#include <stdio.h>#include <ctype.h>

static LIBSSH2_SESSION *session=NULL;static LIBSSH2_SFTP *sftp_session=NULL;static int sock=0;

static void extract_location(const char *location, char *server, char *user, char*passwd, char

*file){int i,j;int lg_location;lg_location=strlen(location);for (i=0;i<lg_location;i++){if (location[i]==’\n’)break;

server[i]=location[i];}server[i]=’\0’;i++;

for (j=0;i<lg_location;i++,j++){if (location[i]==’\n’)break;

user[j]=location[i];}user[j]=’\0’;i++;

for (j=0;i<lg_location;i++,j++){if (location[i]==’\n’)break;

passwd[j]=location[i];}passwd[j]=’\0’;i++;

for (j=0;i<lg_location;i++,j++){if (location[i]==’\0’)break;

file[j]=location[i];}file[j]=’\0’;

}

static void close_sftp_session(){if (sftp_session)libssh2_sftp_shutdown(sftp_session);



sftp_session=NULL;

if (session){libssh2_session_disconnect(session, "Normal Shutdown, Thank you for playing");libssh2_session_free(session);

}session=NULL;

if (sock){

#ifdef WIN32Sleep(1000);closesocket(sock);

#elsesleep(1);close(sock);

#endif}sock=0;

}

static int open_sftp_session(const char *server,const char *user,const char *passwd){struct sockaddr_in sin;int rc;

sock = socket(AF_INET, SOCK_STREAM, 0);

sin.sin_family = AF_INET;sin.sin_port = htons(22);sin.sin_addr.s_addr = inet_addr(server);if (connect(sock, (struct sockaddr*)(&sin),

sizeof(struct sockaddr_in)) != 0){close_sftp_session();return 0;

}

/* Create a session instance

*/session = libssh2_session_init();if(!session){close_sftp_session();return 0;

}

/* ... start it up. This will trade welcome banners, exchange keys,

* and setup crypto, compression, and MAC layers

*/rc = libssh2_session_startup(session, sock);if(rc){close_sftp_session();return 0;



}

libssh2_session_set_blocking(session, 1);

if (libssh2_userauth_password(session, user, passwd)){close_sftp_session();return 0;

}

sftp_session = libssh2_sftp_init(session);

if (!sftp_session){close_sftp_session();return 0;

}return 1;

}

static int csfputs(const char *s, void *stream){#if VERBOSEfprintf(stderr,"csfputs:%s\n",s);

#endifreturn libssh2_sftp_write((LIBSSH2_SFTP_HANDLE*)stream,s,strlen(s));

}

static void *csfopenappend(const char *location){LIBSSH2_SFTP_HANDLE *handle;char server[1024];char user[1024];char passwd[1024];char file[1024];LIBSSH2_SFTP_ATTRIBUTES attrs;

extract_location(location,server,user,passwd,file);#if VERBOSEfprintf(stderr,"csfopenappend %s:%s:%s\n",server,user,file);

#endifif (open_sftp_session(server,user,passwd)){handle = libssh2_sftp_open(sftp_session, file, LIBSSH2_FXF_CREAT|LIBSSH2_FXF_WRITE,

LIBSSH2_SFTP_S_IRUSR|LIBSSH2_SFTP_S_IWUSR|LIBSSH2_SFTP_S_IRGRP|LIBSSH2_SFTP_S_IROTH );

}elsereturn NULL;

if (handle){if (libssh2_sftp_fstat(handle,&attrs)==0){ /* Go to the end of the file */libssh2_sftp_seek(handle,attrs.filesize);

}



}return handle;

}

static void *csfopenread(const char *location){char server[1024];char user[1024];char passwd[1024];char file[1024];

extract_location(location,server,user,passwd,file);#if VERBOSEfprintf(stderr,"csfopenread %s:%s:%s\n",server,user,file);

#endifif (open_sftp_session(server,user,passwd))return (void*) libssh2_sftp_open(sftp_session, file, LIBSSH2_FXF_READ, 0);

elsereturn NULL;

}

static void *csfopenwrite(const char *location){char server[1024];char user[1024];char passwd[1024];char file[1024];

extract_location(location,server,user,passwd,file);#if VERBOSEfprintf(stderr,"csfopenwrite %s:%s:%s\n",server,user,file);

#endifif (open_sftp_session(server,user,passwd))return (void*) libssh2_sftp_open(sftp_session, file, LIBSSH2_FXF_CREAT|LIBSSH2_FXF_WRITE,

LIBSSH2_SFTP_S_IRUSR|LIBSSH2_SFTP_S_IWUSR|LIBSSH2_SFTP_S_IRGRP|LIBSSH2_SFTP_S_IROTH );

elsereturn NULL;

}

static char *csfgets(char *s, int size, void *stream){size_t ss;ss = libssh2_sftp_read((LIBSSH2_SFTP_HANDLE*)stream,s,size-1);if (ss){s[ss]=’\0’;

#if VERBOSEfprintf(stderr,"csfgets:%s\n",s);

#endifreturn s;

}elsereturn NULL;

}



static int csremove(const char *location){int ret;char server[1024];char user[1024];char passwd[1024];char file[1024];

extract_location(location,server,user,passwd,file);#if VERBOSEfprintf(stderr,"csremove %s:%s:%s\n",server,user,file);

#endifif (open_sftp_session(server,user,passwd)){ret = libssh2_sftp_unlink(sftp_session,file);close_sftp_session();return ret;

}elsereturn -1;

}

static int csfclose(void *fp){#if VERBOSEfprintf(stderr,"csfclose\n");

#endifreturn libssh2_sftp_close((LIBSSH2_SFTP_HANDLE*)fp);

}

int main(){char location[1024];char tmp[1024];

#ifdef WIN32WSADATA wsadata;

WSAStartup(WINSOCK_VERSION, &wsadata);#endif

location[0]=’\0’;

printf("server IP:"); fflush(stdout);fgets(tmp,sizeof(tmp),stdin);strcat(location,tmp);

printf("user:"); fflush(stdout);fgets(tmp,sizeof(tmp),stdin);strcat(location,tmp);

printf("passwd:"); fflush(stdout);fgets(tmp,sizeof(tmp),stdin);strcat(location,tmp);

printf(".csexe file name (without extension}:"); fflush(stdout);fgets(tmp,sizeof(tmp),stdin);



strcat(location,tmp);

location[strlen(location)-1]=’\0’;

#ifdef __COVERAGESCANNER____coveragescanner_set_custom_io( csfgets,

csfputs,csfopenappend,csfopenread,csfopenwrite,csfclose,csremove);

__coveragescanner_install(location);#endif}


Supported compilers

Appendix

GSupported compilers

The following compilers are currently supported by CoverageScanner:

G.1 C# compilers

The command line C# compiler of Microsoft® Visual Studio® .NET and Mono C# compiler are supported.

Native Command CoverageScanner Command’mcs’ ’csmcs’

’gmcs’ ’csgmcs’’dmcs’ ’csdmcs’’csc’ ’cscsc’

G.2 Microsoft® Visual C++

The command line compiler and linker of Microsoft® Visual C++ and Microsoft® Visual C++ Toolkit 2003 are supported.

Native Command CoverageScanner Command’cl’ ’cscl’

’lib’ ’cslib’’link’ ’cslink’


Intel C++ Compiler

G.3 Intel® C++ Compiler

The C and C++ command line compiler from Intel® is supported.

Native Command CoverageScanner Command’icl’ ’csicl’’icc’ ’csicc’

’icpc’ ’csicpc’

G.4 GNU gcc

Only g++ and gcc command line compilers are directly supported.To implement support for GNU cross-compilers proceed as follows:

Native Command CoverageScanner Command’gcc’ ’csgcc’’g++’ ’csg++’’ar’ ’csar’

1. Copy ’coveragescanner’ to the file ’cs+<compiler name>’.(ex: copy coveragescanner.exe csarm-linux-gcc.exe)

2. Copy the profile ’gcc.cspro’ or g++.cspro’ to ’<compiler name>+.cspro’.(ex: copy gcc.cspro arm-linux-gcc.cspro)

3. The new GNU cross-compiler ’cs+<compiler name>’ can now be used. It inserts the instrumentations and calls the’<compiler name>’ for compilation.

i The installation script of Squish Coco creates automatically the corresponding compiler wrapper of every GNUcompiler present. So, it is normally not necessary to create such compiler configuration by hand.


CoverageScanner Adaptation to a Tool Suite

Appendix

HCoverageScanner Adaptation to a Tool Suite

CoverageScanner can be adapted to any compiler using a profile. This file is called ’<tool name>.cspro’ and must be placed inthe Squish Coco installation directory. For each profile the CoverageScanner executable should be renamed to ’cs<tool name>’:in this case CoverageScanner directly loads the profile for ’<tool name>’. The command line option --cs-compiler makes itpossible to explicitly to select a profile.

It is possible to rename the CoverageScanner executable to the same name as the native compiler1. In this case CoverageScannerwill work exactly like the native tool, and the code coverage analysis will only be activated when command line option --cs-onis used.

i In order to call the native tool, its location must be in the PATH environment variable after the location ofCoverageScanner.

The profile permits CoverageScanner:

• to interpret the command line of the compiler (see Table H.5, page 218).

• to interpret the command line of the linker (see Table H.4, page 216).

• to customize the output of the coverage analysis 2 (see Table H.7, page 222).

A profile file can contain:

Comments:comments must start with a # character at the beginning of a line and ends at the end of the line.

Global parameters:parameters are defined with the following syntax:<item>=<value>Only one definition per line is allowed.

Parameters for a specific architecture:parameters for a specific architecture are defined with the following syntax:<item>[<architecture>]=<value>Only one definition per line is allowed and the definition must be placed after the global parameter definition.

1Example: rename coveragescanner.exe to cl.exe instead of cscl.exe.2This can be necessary on embedded systems which do not have access to a file system.


Profile Parameters for Architecture Settings

H.1 Profile Parameters for Architecture Settings

ARCHITECTURE=<string>:List of command line arguments which specify the architecture used during the compilation. This parameter should beplaced in the first lines of the profile.

ARCHITECTURE_APPEND=<YES/NO/ALLOW>:If set, the argument of the command line option, which defines the architecture, is directly appended (no space charactersare placed between the option and the arguments).

H.2 Profile Parameters for Instrumentation Settings

COVERAGESCANNER_DUMP_ON_EVENT=<regular expression>:Regular expression which activate the Windows event handler listening the global event Global\COVERAGE.

COVERAGESCANNER_RUNTIME_LOG=<regular expression>:Regular expression which activate the verbose mode of the CoverageScanner API.

COVERAGESCANNER_COVERAGE_ON=<regular expression>:Regular expression which activate the code coverage analysis (equivalent to --cs-on) if this expression match on acommand line argument.

SOURCE_IGNORE_INSTRUMENTATION=<list>:List of source file which are not instrumented. The list is a list of wildcard expressions which matches absolute filepaths. Environment variables can be access by placing the name between two dollars. (ex: $HOME$ to access to HOMEenvironment variable)

H.3 Profile Parameters for Preprocessor Settings

PREPROCESSOR_HIDE_OPTION_NO_ARG=<string>:List of single options (without arguments) which should not be transmitted to the native preprocessor.

PREPROCESSOR_HIDE_OPTION_ONE_ARG=<string>:List of options with one argument which should not be transmitted to the native preprocessor.

PREPROCESSOR_HIDE_OPTION_ONE_ARG_APPEND=<YES/NO/ALLOW>:If set, the argument of the options to hide is directly appended (no space characters are placed between the option and thearguments).

PREPROCESSOR_KEEP_OPTION_ONE_ARG=<string>:List of options with one argument which must be transmitted to the native preprocessor.

PREPROCESSOR_KEEP_OPTION_ONE_ARG_APPEND=<YES/NO/ALLOW>:If set, the argument of the options which must be transmitted is directly appended (no space characters are placed betweenthe option and the arguments).

PREPROCESSOR_DEFINE_OPTION=<string>:Preprocessor command line option which permits to enter C or C++ defines. (ex: -D)


Profile Parameters for Linker Settings

PREPROCESSOR_DEFINE_OPTION_APPEND=<YES/NO/ALLOW>:Allows appending the preprocessor defines just after its command line option string. (without spaces, ex: -DNDEBUG)

PREPROCESSOR_INCLUDE_OPTION=<string>:Preprocessor command line option which permits to include directly some source files. (ex: /FI)

PREPROCESSOR_INCLUDE_OPTION_APPEND=<YES/NO/ALLOW>:Allows appending the include file name defines just after its command line option string. (without spaces, ex: /FImy-header.h)

PREPROCESSOR_DEFINE_SEPARATOR_OPTION=<string>:Separator of the define list. For example, if set to ,, CoverageScanner will detect the defines DEF1 and DEF2 whenparsing the string DEF1,DEF2.

PREPROCESSOR_CMD=<string>:Preprocessor command line. CoverageScanner does only analyse preprocessed files. The command generated the C orC++ file used for the code coverage analysis.The following variables are available:

$TMP1$. . . $TMP9$: Temporary files names. The temporary files are removed on exit.

$OPTIONS$: Preprocessing options. The preprocessing option are extracted from the command line arguments andcontains the list of includes, defines, etc. . .

$TOOL$: Compiler command (TOOL variable).

$ARCHITECTURE$: Architecture detected or specified by --cs-architecture.

$PROFILE_PATH$: Path of the profile file.

$SOURCE$: Source file name.

$OUTPUT$: Preprocessor output file name. If this variable is present in the command line, it contains the name of atemporary filename used as output for the preprocessor. If not present, it is assumed that the preprocessorgenerate its output to stdout.

The preprocessor output is stdout. If this parameter is omitted in the configuration, the preprocessor step is skipped.

i The preprocessor command must set the define __COVERAGESCANNER__ to 1.

(ex: $TOOL$ $OPTIONS$ -D__COVERAGESCANNER__=1 -E $SOURCE$)

H.4 Profile Parameters for Linker Settings

LINKER_HIDE_OPTION_NO_ARG=<string>:List of single options (without arguments) which should not be transmitted to the native linker.

LINKER_HIDE_OPTION_ONE_ARG=<string>:List of options with one argument which should not be transmitted to the native linker.

LINKER_HIDE_OPTION_ONE_ARG_APPEND=<YES/NO/ALLOW>:If set, the argument of the options to hide is directly appended (no space characters are placed between the option and thearguments).

OBJ_EXT=<string>:List of extension for object files separated with a semi-column. (ex: .obj)

DYN_LIB_EXT=<string>:List of extensions for dynamic libraries separated with a semi-column. (ex: .dll)


Profile Parameters for Linker Settings

STATIC_LIB_EXT=<string>:List of extensions for static libraries separated with a semi-column. (ex: .lib)

UNIX_LIBRARY_NAME=<YES/NO>:The library name follows the Unix standard.

LINK_LIBRARY_OPTION=<string>:Link option which permits to select libraries (ex:-l).

LINK_LIBRARY_OPTION_APPEND=<YES/NO/ALLOW>:Append the library directly after the link option string (ex: -lfoo) if LINK_LIBRARY_OPTION_APPEND is set.

LINK_OUTPUT_OPTION=<string>:Linker output option (ex:-o)

LINK_OUTPUT_OPTION_APPEND=<YES/NO/ALLOW>:Append the linker output file directly after the option string (without spaces, ex: -ofoo.o) ifLINK_OUTPUT_OPTION_APPEND is set.

LINK_OUTPUT_DEFAULT=<string>:Linker output default filename.The following variable is available:


$BASENAME_SRC$: Base name of the the first source file. The base name is the filename without extension.

(ex: $BASENAME_SRC$.exe)

LINK_ADDITIONAL_ARGUMENTS=<string>:Additional arguments passed to the linker when instrumenting the source code.

DEFAULT_LIB_PATH=<string>:List of directories which contains the shared libraries (ex: /usr/library;/usr/share/library).

DLL_OPTION=<string>:Linker command line option which permits to generate a shared library or a DLL (ex: /DLL).

DLL_OUTPUT_STATIC_LIB=<string>:Static library (associated with the DLL) output file name (ex: /IMPLIB:foo.lib).

DLL_OUTPUT_STATI_LIB_APPEND=<YES/NO/ALLOW>:Append the static library (associated with the DLL) output file directly after the option string (without spaces, ex: /IM-PLIB:foo.lib) if DLL_OUTPUT_STATIC_LIB_APPEND is set.

STATIC_LIB_AS_DEFAULT=<YES/NO>:Instrumentation database name takes as base name the static library name associated with a DLL.

LIBRARY_PATH_OPTION=<string>:Link option which permits to add additional directories to the path of directories (ex:-L)

LIBRARY_PATH_OPTION_APPEND=<YES/NO/ALLOW>:Append the directory name directly after the option string (ex: -L/foo) if LIBRARY_PATH_OPTION_APPEND is set.

AR_COMMAND_FORMAT_OPTION=<YES/NO>:The GNU librarian (ar) has a specific command line syntax which is supported when this option is set.

STDIN_MRI_SCRIPT_OPTION=<list>:This option indicates that an MRI script is used to generate a library. The script is provided by the standard input.

GENERATE_COVERAGESCANNER_LIBRARY=<YES/NO>:If set, the CoverageScanner library is generated during the linking operation. This should be disabled when generatingstatic libraries.


Profile Parameters for Compiler Settings

SKIP_GENERATE_COVERAGESCANNER_LIBRARY_OPTION=<string>:Permits to skip the generation of CoverageScanner library if a specific option appears in the command line.

AUTODETECT_MS_RUNTIME=<YES/NO>:Detect automatically the runtime library used when generating an application using Microsoft® Visual Studio®. Thisoption compiles the CoverageScanner library with /MT, /MD or /ML if --cs-libgen is not used to set it. If --cs-libgenset /MT, /MD or /ML, this option does not have any effect.

AUTODETECT_MS_RUNTIME_OPTION=<list>:List of command line option which selects the runtime library (for Microsoft® Visual Studio®:/MT;/MD;/ML;/MLd;/MTd;/MDd)

INSTRUMENTATION_TABLES_LINKED_DURING_RUNTIME=<YES/NO>:Instead of linking all instrumentation tables when calling the linker, link it during the runtime of the application.

PLUGIN_REGISTRATION_API=<YES/NO>:If set, __coveragescanner_register_library() and __coveragescanner_unregister_library() are providedbythe CoverageScanner API to register instrumented plugins during the execution.

FILE_FORMAT_SPECIFIER=<YES/NO>:If set, file format specifier (see command line option --cs-output) are supported at the runtime.

INSTRUMENTATION_TABLES_OF_DLL_LINKED_DURING_RUNTIME=<YES/NO>:Instead of linking all instrumentation tables of DLL when calling the linker, link it during the runtime of the application.

COVERAGESCANNER_REGISTRATION_UID=<YES/NO>:Generate an unique symbol in each static and dynamic library to retrieve the code coverage counters.

INJECT_COVERAGESCANNER_LIBRARY_AT_END=<YES/NO>:If YES, the CoverageScanner library is always inserted at the end of the linker command line arguments.

GNU_LINKER_SCRIPT=<YES/NO>:Support of GNU linker scripts.

VS_DEF_FILE_OPTION=<string>:Command line option for specifying the module definition file (.def file).(ex: VS_DEF_FILE_OPTION=/DEF:)

VS_DEF_FILE_OPTION_APPEND=<YES/NO/ALLOW>:If set, the command line option for specifying the module definition file is directly appended to the file name (no spacecharacters are placed between the option and the arguments).

H.5 Profile Parameters for Compiler Settings

COMPILER_ONLY=<YES/NO>:If true, the native tool is only a compiler which has no linker capabilities.

COMPILER_HIDE_OPTION_ONE_ARG=<string>:List of options with one argument which should not be transmitted to the native compiler.

COMPILER_HIDE_OPTION_ONE_ARG_APPEND=<YES/NO/ALLOW>:If set, the argument of the options to hide is directly appended (no space characters are placed between the option and thearguments).

BYTECODE_EMULATOR=<string>:If set, the call of the native toolchain is made by calling a byte code intepreter.

Example: BYTECODE_EMULATOR=/math/to/mono.exe permits for force the call of mcs.exe in a specific Mono emulator.



TOOL=<string>:Native compiler/linker command (example: gcc).The following variables are available:

$PROFILE$: Profile name without extension.

REMOVE_EXT=<string>:List of file extension, separated with a semi-column, of files to remove from the command line arguments.

IGNORE_EXT=<string>:List of file extension, separated with a semi-column, of files to ignore. Ignored files are not instrumented.

C#_EXT=<string>:List of extension for C# files separated with a semi-column. (ex: .cs)

C_EXT=<string>:List of extension for C files separated with a semi-column. (ex: .c)

CPP_EXT=<string>:List of extension for C++ files separated with a semi-column. (ex: .cpp;.cxx)

CPP_LANGUAGE_OPTION=<string>:List of command line options which are switching the compiler in C++ mode.

C_LANGUAGE_OPTION=<string>:List of command line options which are switching the compiler in C mode.

LANGUAGE_SELECTION_OPTION=<string>:List of command line options which permits to select the computing language (C, or C++).

LANGUAGE_SELECTION_OPTION_APPEND=<YES/NO/ALLOW>:Append the selected language directly after the option string.

LANGUAGE_SELECTION_OPTION_C_KEYWORD=<string>:List of arguments of the language selection command line option which force the switching in C mode.

LANGUAGE_SELECTION_OPTION_CPP_KEYWORD=<string>:List of arguments of the language selection command line option which force the switching in C++ mode.

COMPILER_CMD=<string>:Compiler command line.The compiler command line is used to generate the CoverageScanner library during the linking phase.The following variables are available:


$COMPILER$: Compiler command (COMPILER variable).

$SOURCE$: Source file name.

$DESTINATION$: Destination file name.

$LIBGEN$: Contains of the command line option --cs-libgen.

(ex: $COMPILER$ -c $SOURCE$ -o $DESTINATION$)

COMPILER_ADDITIONAL_ARGUMENTS=<string>:Additional arguments passed to the compiler when instrumenting the source code.

COMPILE_OPTION=<string>:Compile command line option. (ex: -c)

COMPILER_OUTPUT_OPTION=<string>:Compiler output option. (ex: -o)



COMPILER_OUTPUT_OPTION_APPEND=<YES/NO/ALLOW>:Append the compiler output directly after the option string (without spaces, ex: -ofoo.o) ifCOMPILER_OUTPUT_OPTION_APPEND is true.

COMPILER_OUTPUT_DEFAULT=<string>:Compiler output default filename.The following variable is available:


$BASENAME_SRC$: Base name of the first source file. The base name is the filename without extension.

(ex: $BASENAME_SRC$.obj)

CALLING_CONVENTION=<string>:CALLING_CONVENTION sets the calling convention (__stdcall, __cdecl or __fastcall) on Windows platform.

FUNCTION_ATTRIBUTE=<string>:FUNCTION_ATTRIBUTE sets additional compiler attribute to each function of the CoverageScanner library.

DESTRUCTOR_ATTRIBUTE=<string>:DESTRUCTOR_ATTRIBUTE sets additional compiler attribute which permits to the functions, which reset the code coverageinformation when the application get shutdown or when a dynamic library get unloaded, to be automatically called. Thisattribute is is only necessary when the instrumentation tables are registered during the runtime.

CONSTRUCTOR_ATTRIBUTE=<string>:CONSTRUCTOR_ATTRIBUTE sets additional compiler attribute which permits to the functions, which initialize the codecoverage information when the application starts, to be automatically called. This attribute is is only necessary when theinstrumentation tables are registered during the runtime.

DESTRUCTOR_PRAGMA=<string>:DESTRUCTOR_PRAGMA sets additional compiler pragma attribute which permits to the functions, which reset the codecoverage information when the application get shutdown or when a dynamic library get unloaded, to be automaticallycalled. The variable $SYMBOL$ is parsed and is replaced through the symbol on which the pragma is applyed. Thisattribute is is only necessary when the instrumentation tables are registered during the runtime.

CONSTRUCTOR_PRAGMA=<string>:CONSTRUCTOR_PRAGMA sets additional compiler attribute which permits to the functions, which initialize the codecoverage information when the application starts, to be automatically called. The variable $SYMBOL$ is parsed and isreplaced through the symbol on which the pragma is applyed. This attribute is is only necessary when the instrumentationtables are registered during the runtime.

DLL_EXPORT=<string>:DLL_EXPORT contains the compiler attribute which permits to export a symbol from a DLL/shared library.

DLL_IMPORT=<string>:DLL_IMPORT contains the compiler attribute which permits to import a symbol from a DLL/shared library.

FORCE_DLL_EXPORT=<YES/NO>:FORCE_DLL_EXPORT permits to force the usage of DLL_EXPORT attribute during the linking phase instead of DLL_IMPORTattribute.

FORCE_DLL_EXPORT_OPTION=<string>:FORCE_DLL_EXPORT_OPTION permits to force the usage of DLL_EXPORT attribute when a appears in the command line.

PARALLEL_COMPILATION_OPTION_APPEND=<YES/NO/ALLOW>:Indicates if the number of processors are directly appended to the command which enables the parallel build.

PARALLEL_COMPILATION_OPTION=<list>:Command line option which enables the parallel build. If this command line argument is followed by an integer, thisvalue indicates the maximum number of parallel build to execute.


Profile Parameters for Precompiled Headers Support

SYSTEM_INCLUDES=<list>:List of system include directories. (ex: /usr/include)

INCLUDE_PATH_OPTION=<string>:Compiler option which permits to add additional include directories (ex:-I)

INCLUDE_PATH_OPTION_APPEND=<YES/NO/ALLOW>:Append the include directory name directly after the option string (ex: -I/foo) if INCLUDE_PATH_OPTION_APPEND is set.

CUSTOM_MALLOC_INCLUDE=<string>:Additional include for a custom malloc()/free() function (ex: <stdlib.h>)

CUSTOM_MALLOC_FUNCTION=<string>:Custom body of a malloc() function. The body should return a void* and the parameter int size indicate the numberof bytes allocated. (ex: return malloc((size_t)size);)

CUSTOM_FREE_FUNCTION=<string>:Custom body of a free() function. The parameter void* ptr is the pointer to free. (ex: free(ptr);)

LOCALE_SYSTEM_FILE_ENCODING=<YES/NO>:If YES, the locale system configuration of text encoding is used for compiling a source file.

OBJECT_DIR_OPTION=<string>:Compiler option which permits to specify the directory on which the objects are stored. (ex: -object-dir=debug/)

OBJECT_DIR_OPTION_APPEND=<YES/NO/ALLOW>:Append the directory directly after the option string (without spaces, ex: -object-dir=foo)

H.6 Profile Parameters for Precompiled Headers Support

PCH_EXT=<string>:List of extension for precompiled header files separated with a semi-column. (ex: .pch)

CREATE_PCH_OPTION_APPEND=<YES/NO/ALLOW>:Indicates if the PCH header generated is directly appended to the command line option.

CREATE_PCH_OPTION=<list>:List of command line options which are generating precompiled headers. (ex: /Yc)

USE_PCH_OPTION_APPEND=<YES/NO/ALLOW>:Indicates if the PCH header used is directly appended to the command line option.

USE_PCH_OPTION=<list>:List of command line options which permits to select a precompiled header. (ex: /Yu)

PCH_OUTPUT_OPTION=<string>:Command line option to specify the precompiled header output. (ex: /Fp)

PCH_OUTPUT_OPTION_APPEND=<YES/NO/ALLOW>:Append the precompiled header output directly after the option string (without spaces, ex: -Fpfoo.pch) ifPCH_OUTPUT_OPTION_APPEND is true.

PCH_OUTPUT_DEFAULT=<string>:Default precompiled header output file name.The following variable is available:

$TMP1$. . . $TMP9$: Temporary files names. The temporary files are removed on exit.$BASENAME_SRC$: Base name of the first source file. The base name is the filename without extension.

(ex: $BASENAME_SRC$.pdb)


Profile Parameters for Custom IO

H.7 Profile Parameters for Custom IO

CSEXE_FOPEN=<string>:’fopen function used by the generation of the execution report file.

CSEXE_FCLOSE=<string>:’fclose function used by the generation of the execution report file.

CSEXE_FPUTS=<string>:fputs function used by the generation of the execution report file.

CUSTOM_SETUP=<NONE/POSIX/MS/GNU/MSCE/C++>: NONE: if set to NONE the function __coveragescanner_install()will not be provided by CoverageScannerThis parameter should be set to NONE on embedded systems which donot provide the functions signal() and atexit().The code coverage information is saved only when __coveragescanner_save() is called.

POSIX: CoverageScanner provides a function __coveragescanner_install(), which installs a handler which savesthe execution report when the application terminates. The handler is installed using POSIX functions (signal()and atexit()).If __coveragescanner_install() is not called, the code coverage information is only saved when__coveragescanner_save() is called.

GNU: This is similar to the POSIX setting except that, if __coveragescanner_install() is not called, a defaulthandler is used which saves the code coverage data in the file coverage.csmes on a normal application exit.If __coveragescanner_install() is not called, the code coverage information is not saved on abnormalexit (crash, software interruption, . . . ).

MS: Same as GNU but for Microsoft® Visual Studio® compiler.

C++: Install a handler written in C++ which saves the code coverage data in the file coverage.csmes upon normalapplication exit. CoverageScanner provides a function __coveragescanner_install(), which only changesthe name of the destination file.

MSCE: Same as C++ but for Microsoft® eMbedded Visual C++® compiler.

H.8 Miscellaneous Profile Parameters

HIDE_OPTION_NO_ARG=<string>:List of single options (without arguments) which should not be transmitted to the native tool (compiler or linker).(ex: -Yd;/Yd for disabling the precompiled headears on Microsoft® Visual Studio®)

HIDE_OPTION_ONE_ARG=<string>:List of options with one argument which should not be transmitted to the native tool (compiler or linker).(ex: -Yl;-Yc;-Yu;-YX;/Yl;/Yc;/Yu;/YX for disabling the precompiled headears on Microsoft® Visual Studio®)

HIDE_OPTION_ONE_ARG_APPEND=<YES/NO/ALLOW>:If set, the argument of the options to hide is directly appended (no space characters are placed between the option and thearguments).

PDB_EXT=<string>:List of extension for debug information files separated with a semi-column. (ex: .pdb)

PDB_OUTPUT_OPTION=<string>:Command line option for specifying the debug information file.(ex: PDB_OUTPUT_OPTION=/Fd)


Miscellaneous Profile Parameters

PDB_OUTPUT_OPTION_APPEND=<YES/NO/ALLOW>:If set, the command line option for specifying the debug information file is directly appended to the file name (no spacecharacters are placed between the option and the arguments).

SBR_EXT=<string>:List of extension for symbol browser files separated with a semi-column. (ex: .sbr)

SBR_OUTPUT_OPTION=<string>:Command line option for specifying the symbol browser information file.(ex: SBR_OUTPUT_OPTION=/Fr;/FR)

SBR_OUTPUT_OPTION_APPEND=<YES/NO/ALLOW>:If set, the command line option for specifying the symbol browser information file is directly appended to the file name(no space characters are placed between the option and the arguments).

USE_RESPONSE_FILE=<YES/NO/ALLOW>:Pack all compiler/linker command line option into a response file. This is generally necessary on Microsoft® Windowsdue to the command line limitation length. The option ALLOW consists of using a response file only if a response file isused in the command line arguments of the native compiler. A response file is only used if the if the command line sizereach a limit defined by MINIMUM_COMMAND_LINE_SIZE_FOR_RESPONSE_FILE.

MINIMUM_COMMAND_LINE_SIZE_FOR_RESPONSE_FILE=<integer>:If the command line arguments are lower that the specified size, no response file is used.

RESPONSE_FILE_EXT=<string>:List of file extensions for the response file.

RESPONSE_FILE_OPTION=<string>:Command line option for specifying the response file.(ex: RESPONSE_FILE_OPTION=@)

RESPONSE_FILE_OPTION_APPEND=<YES/NO/ALLOW>:If set, the command line option for specifying the response file is directly appended to the file name (no space charactersare placed between the option and the arguments).

CODE_STYLE=<C++/C#>:Language supported.

EXIT_FUNCTIONS=<string>:List of functions which causes an application exit.(ex: EXIT_FUNCTIONS=abort;exit)

ENABLE_COVERAGESCANNER_PER_DEFAULT=<YES/NO>:Enable code coverage analysis per default. This is equivalent to implicitly set the command line option --cs-on. Thisoption has only an impact if the native executable has the same name as CoverageScanner wrapper.

DEACTIVATE_COVERAGESCANNER=<YES/NO>:Disable CoverageScanner globally if set to YES.

DEACTIVATE_COVERAGESCANNER_OPTION_NO_ARG=<string>:List of command line option which deactivates CoverageScanner.(ex: DEACTIVATE_COVERAGESCANNER_OPTION_NO_ARG=-M;-MM;-MD;-MMD)

DEACTIVATE_COVERAGESCANNER_OPTION_ONE_ARG=<string>:List of command line option with one argument which deactivates CoverageScanner.

DEACTIVATE_COVERAGESCANNER_OPTION_ONE_ARG_APPEND=<YES/NO/ALLOW>:If set, the command line option which disables CoverageScanner is directly followed by an argument (no space charactersare placed between the option and the arguments).

COVERAGESCANNER_LIBRARY_OBJECT=<string>:Name of the CoverageScanner object containing its library code. On Unix, its default value is __cs_library.o. OnMicrosoft® Windows, its default value is __cs_library.obj.


Instrumentation Parameters

EXIT_VALUE_ON_SUCCESS=<list>:List of exit value that are return by the native linker or compiler and which means that the command was sucessfull.

ERROR_FORMAT=<string>:Format of the errors. The following items are parsed:

%m: Error message.

%f: File name.

%l: Line.

%c: Column.

ERROR_FORMAT_LINE=<string>:Same as ERROR_FORMAT but used when no column information are available.

ADDITIONAL_ARGUMENT_SEPARATOR=<string>:Additional character used as separator for the command line arguments.(ex: ADDITIONAL_ARGUMENT_SEPARATOR=, for VisualDSP++ compiler)

CORRECT_FILENAME_IN_COMPILER_MESSAGE=<YES/NO>:If enabled, CoverageScanner replace in the compiler/linker messages the file name used for the instrumentation throughthe real source filename of the project.

SUPPRESS_APTH_ENTRY=<YES/NO>:If set to YES, CoverageScanner alter the PATH variable and removes its own location when calling the native compiler.

ATEXIT=<string>:Command line used to register the application exit handler which saves the execution report upon the application exit.

FORCE_ESCAPING_RULES_RESPONSE_FILE=<NO/MS/POSIX>:Rules for escaping command line arguments in response files:

POSIX: Escaping rules for UNIX® platforms

MS: Escaping rules for Microsoft® Windows.

NO: Native escaping rules for the current platform.

FILESYSTEM_8.3=<YES/NO>:If set, the extension of the execution report file name is .cse. If the application file name is application.exe thegenerated report will be application.cse in this case and not application.exe.csexe

H.9 Instrumentation Parameters

RETURN_INSTRUMENTED_IN_EXPRESSION=<YES/NO>:If true, return statement are instrumented into the returned expression (ex:return inst[0]++, x;).


Appendix

H Index

--cs-architecture, 79, 80, 107, 216--cs-boost, 77--cs-branch, 61, 63, 77, 81--cs-c-ext, 80, 188--cs-compiler, 73, 74, 79, 214--cs-condition, 77, 81--cs-count, 19, 24, 55, 61, 77, 81--cs-cpp-ext, 80, 188--cs-decision, 63, 77, 81--cs-disable-coveragescanner-library-abs-wildcard, 80--cs-dump-on-event, 79, 81, 121, 173, 189--cs-dump-on-signal, 79, 121, 189--cs-enable-coveragescanner-library-abs-wildcard, 80--cs-exclude, 74--cs-exclude-code, 142--cs-exclude-code-regexp, 140--cs-exclude-code-wildcard, 140--cs-exclude-file-abs-regex, 75--cs-exclude-file-abs-regexp, 140--cs-exclude-file-abs-wildcard, 75, 140--cs-exclude-file-regex, 19, 75, 116, 167--cs-exclude-file-wildcard, 75, 167--cs-exclude-function-regex, 77--cs-exclude-function-wildcard, 76--cs-exclude-path, 75, 140, 167, 185--cs-full-instrumentation, 19, 24, 63, 77, 81--cs-function, 77--cs-hit, 55, 61, 77, 81--cs-include, 74--cs-include-code-regexp, 140--cs-include-code-wildcard, 140--cs-include-file-abs-regex, 75--cs-include-file-abs-wildcard, 75--cs-include-file-regex, 75, 168--cs-include-file-wildcard, 75, 168--cs-include-function-regex, 76--cs-include-function-wildcard, 76--cs-include-init-scripts, 140--cs-include-path, 24, 75, 140, 168, 185--cs-include-unnamed, 140--cs-keep-instrumentation-files, 78, 173--cs-libgen, 80, 107, 110, 173, 218, 219--cs-library-after, 80--cs-library-at-end, 80--cs-line, 77--cs-link-instrumentation-tables, 80, 189--cs-lock-csexe, 79, 81--cs-memory-pool, 80--cs-minimum-api, 79, 81--cs-native-toolchain, 80, 188--cs-no-abort-on-error, 80--cs-no-annotations, 78--cs-no-assignments, 77, 81--cs-no-boost, 78, 192--cs-no-cspch, 80--cs-no-exceptions, 77, 81--cs-no-exit-handler-installation, 80--cs-no-function, 77--cs-no-line, 77--cs-no-qt3, 78, 192--cs-no-qt4, 78, 192

--cs-nolock-csexe, 79, 81--cs-off, 77--cs-on, 12, 17, 42, 72, 73, 77, 81, 105, 107, 110, 112, 114, 214, 215, 223--cs-option-file, 80, 188--cs-output, 19, 78, 79, 140, 142, 184, 185, 218--cs-output-abs, 192--cs-parallel-build, 80--cs-partial-instrumentation, 77, 81--cs-profile, 74, 79, 188--cs-qt3, 78, 100--cs-qt4, 19, 24, 78, 100--cs-verbose, 78, 188, 193--cs-verbose-file, 78--cs-verbose-source-lines, 78--cs-verbose=api, 81--cs-vs2010-lambda, 80, 167, 187--cs-warnings, 78#endregion, 94#pragma, 94#region, 94$ARCHITECTURE$ (Profile variable), 216$BASENAME_SRC$ (Profile variable), 217, 220, 221$COMPILER$ (Profile variable), 219$DESTINATION$ (Profile variable), 219$LIBGEN$ (Profile variable), 219$OPTIONS$ (Profile variable), 216$OUTPUT$ (Profile variable), 216$PROFILE$ (Profile variable), 219$PROFILE_PATH$ (Profile variable), 216$SOURCE$ (Profile variable), 216, 219$TOOL$ (Profile variable), 216_Pragma, 94__COVERAGESCANNER__, 83, 88, 216__coveragescanner_add_html_comment(), 85, 89, 120__coveragescanner_clear(), 85, 124, 127, 129__coveragescanner_clear_html_comment(), 85__coveragescanner_filename(), 86, 117, 118__coveragescanner_install(), 50, 80, 83, 84, 90, 222__coveragescanner_register_library(), 86, 116, 117, 218__coveragescanner_reset(), 85__coveragescanner_save(), 83–85, 88, 117, 118, 124, 127, 129, 222__coveragescanner_set_custom_io(), 87, 88__coveragescanner_testname(), 84, 120, 124, 127, 129__coveragescanner_teststate(), 84, 120, 124, 127, 129__coveragescanner_unregister_library(), 86, 87, 116, 117, 218__pragma, 94

ADDITIONAL_ARGUMENT_SEPARATOR (Profile option), 224Apple Mac OS X, 1, 20, 159, 185, 186, 203Apple Xcode, 113, 115AR_COMMAND_FORMAT_OPTION (Profile option), 217ARCHITECTURE (Profile option), 215ARCHITECTURE_APPEND (Profile option), 215ARM Keil uVision, 101, 102ATEXIT (Profile option), 224AUTODETECT_MS_RUNTIME (Profile option), 218AUTODETECT_MS_RUNTIME_OPTION (Profile option), 218

bash, 14, 16benchmark, 199

225

INDEX

BYTECODE_EMULATOR (Profile option), 218

C, 1–3, 19, 34, 53, 55, 63, 72, 73, 83, 92, 94, 96, 100, 105–107, 110, 112–114, 135, 163,213, 215, 216

C++, 1–3, 14, 19, 34, 53, 55, 63, 72, 73, 83, 92, 94, 96, 100, 105–107, 110, 112–114,135, 163, 185, 213, 215, 216

C#, 1–3, 9, 73, 88, 91, 94, 105, 135, 182, 183, 185, 186, 212, 219C#_EXT (Profile option), 219C_EXT (Profile option), 219C_LANGUAGE_OPTION (Profile option), 219Cache, 70CALLING_CONVENTION (Profile option), 220ccache, 165CMake, 97CMakeLists.txt, 97cmmerge, 156Cobertura, 66, 67, 152, 186Code coverage count, 42, 55, 61Code coverage hit, 42, 61CODE_STYLE (Profile option), 223Color

Blue, 56Gray, 56Magenta, 56Orange, 55Red, 55, 56

Comments, 56, 57, 69COMPILE_OPTION (Profile option), 219COMPILER_ADDITIONAL_ARGUMENTS (Profile option), 219COMPILER_CMD (Profile option), 219COMPILER_HIDE_OPTION_ONE_ARG (Profile option), 218COMPILER_HIDE_OPTION_ONE_ARG_APPEND (Profile option), 218COMPILER_ONLY (Profile option), 218COMPILER_OUTPUT_DEFAULT (Profile option), 220COMPILER_OUTPUT_OPTION (Profile option), 219COMPILER_OUTPUT_OPTION_APPEND (Profile option), 220Condition Coverage, 36CONSTRUCTOR_ATTRIBUTE (Profile option), 220CONSTRUCTOR_PRAGMA (Profile option), 220CORRECT_FILENAME_IN_COMPILER_MESSAGE (Profile option), 224cov-assignment-off

pragma, 94cov-assignment-on

pragma, 94cov-branch

pragma, 94region, 95

cov-conditionpragma, 94region, 95

cov-countpragma, 94region, 95

cov-decisionpragma, 94region, 95

cov-full-instrumentationpragma, 94region, 95

cov-hitpragma, 94region, 95

cov-offpragma, 94region, 95

cov-onpragma, 94region, 95

cov-partial-instrumentationpragma, 94region, 95

CoverageBrowser MenuContext Menu

Add/Set Comment, 56Clear Comments, 57

FileGenerate Black-Box Configuration. . . , 30, 47Import Reviewer Comments. . . , 65Import Unit Tests. . . , 27, 64, 123Load Execution Report. . . , 8, 9, 21, 50Merge with. . . , 30, 64Open. . . , 20Save, 16

Instrumentation

Add/Set Comment, 56Clear Comment, 57Level:x, 61

ProjectEdit Active Target, 114Edit Project Settings, 114

ReportsExport JUnit Report. . . , 67Export Statistics in Cobertura-XML Format. . . , 66Export Statistics in EMMA-XML Format. . . , 66Export Statistics per Method, 68Export Statistics per Source File, 67Generate HTML Report. . . , 68Generate Text Report. . . , 68

ToolsAdd-In Manager. . . , 163Analysis of Modified Methods, 29, 64Analysis on Identical Methods, 64Code Coverage Build Mode. . . , 163Compare with. . . , 29, 64Execution Comparison Analysis, 28Optimized Execution Order. . . , 61Options. . . , 163Switch databases, 64Test Coverage Count Mode, 61

ViewFunctions, 53New Source Window, 54Sources, 52

coveragescanner, 100CoverageScanner() directive, 94, 95CoverageScanner.__coveragescanner_add_html_comment(), 89CoverageScanner.__coveragescanner_clear(), 90CoverageScanner.__coveragescanner_clear_html_comment(), 89CoverageScanner.__coveragescanner_filename(), 90CoverageScanner.__coveragescanner_init(), 88, 185CoverageScanner.__coveragescanner_save(), 89–91CoverageScanner.__coveragescanner_set_custom_io(), 90, 91CoverageScanner.__coveragescanner_testname(), 88, 89CoverageScanner.__coveragescanner_teststate(), 89COVERAGESCANNER_ARGS, 12, 17, 73, 74, 99COVERAGESCANNER_COVERAGE_BRANCH, 81COVERAGESCANNER_COVERAGE_CONDITION, 81COVERAGESCANNER_COVERAGE_COUNT, 81COVERAGESCANNER_COVERAGE_DECISION, 81COVERAGESCANNER_COVERAGE_FULL_INSTRUMENTATION, 81COVERAGESCANNER_COVERAGE_HIT, 81COVERAGESCANNER_COVERAGE_LOCK_CSEXE, 81COVERAGESCANNER_COVERAGE_NO_ASSIGNMENTS, 81COVERAGESCANNER_COVERAGE_NO_EXCEPTIONS, 81COVERAGESCANNER_COVERAGE_NOLOCK_CSEXE, 81COVERAGESCANNER_COVERAGE_ON, 9, 81, 105COVERAGESCANNER_COVERAGE_ON (Profile option), 215COVERAGESCANNER_COVERAGE_PARTIAL_INSTRUMENTATION, 81COVERAGESCANNER_DUMP_ON_EVENT, 81COVERAGESCANNER_DUMP_ON_EVENT (Profile option), 215COVERAGESCANNER_LIBRARY_OBJECT (Profile option), 223COVERAGESCANNER_MINIMUM_API, 81COVERAGESCANNER_REGISTRATION_UID (Profile option), 218COVERAGESCANNER_RUNTIME_LOG, 81COVERAGESCANNER_RUNTIME_LOG (Profile option), 215CPP_EXT (Profile option), 219CPP_LANGUAGE_OPTION (Profile option), 219CREATE_PCH_OPTION (Profile option), 221CREATE_PCH_OPTION_APPEND (Profile option), 221csar, 213cscl, 212cscsc, 212csdmcs, 212CSEXE_FCLOSE (Profile option), 222CSEXE_FOPEN (Profile option), 222CSEXE_FPUTS (Profile option), 222csg++, 213csgcc, 213csgmcs, 212csicc, 213csicl, 213csicpc, 213cslib, 212cslink, 212csmcs, 212CSV, 67, 68, 147, 149–151, 183, 184CUSTOM_FREE_FUNCTION (Profile option), 221CUSTOM_MALLOC_FUNCTION (Profile option), 221


INDEX

CUSTOM_MALLOC_INCLUDE (Profile option), 221CUSTOM_SETUP (Profile option), 84, 222CygWin, 96, 184, 189

DEACTIVATE_COVERAGESCANNER (Profile option), 223DEACTIVATE_COVERAGESCANNER_OPTION_NO_ARG (Profile option), 223DEACTIVATE_COVERAGESCANNER_OPTION_ONE_ARG (Profile option), 223DEACTIVATE_COVERAGESCANNER_OPTION_ONE_ARG_APPEND (Profile option), 223Decision Coverage, 35DEFAULT_LIB_PATH (Profile option), 217DESTRUCTOR_ATTRIBUTE (Profile option), 220DESTRUCTOR_PRAGMA (Profile option), 220DLL_EXPORT (Profile option), 220DLL_IMPORT (Profile option), 220DLL_OPTION (Profile option), 217DLL_OUTPUT_STATI_LIB_APPEND (Profile option), 217DLL_OUTPUT_STATIC_LIB (Profile option), 217DYN_LIB_EXT (Profile option), 216

Eclipse, 113EMMA, 66, 151, 152, 184, 187, 188, 191, 193ENABLE_COVERAGESCANNER_PER_DEFAULT (Profile option), 223ERROR_FORMAT (Profile option), 224ERROR_FORMAT_LINE (Profile option), 224Execution Report, 50EXIT_FUNCTIONS (Profile option), 223EXIT_VALUE_ON_SUCCESS (Profile option), 224

FILE_FORMAT_SPECIFIER (Profile option), 86, 218FILESYSTEM_8.3 (Profile option), 224FORCE_DLL_EXPORT (Profile option), 220FORCE_DLL_EXPORT_OPTION (Profile option), 220FORCE_ESCAPING_RULES_RESPONSE_FILE (Profile option), 224FUNCTION_ATTRIBUTE (Profile option), 220

GENERATE_COVERAGESCANNER_LIBRARY (Profile option), 217GNU_LINKER_SCRIPT (Profile option), 218

HIDE_OPTION_NO_ARG (Profile option), 222HIDE_OPTION_ONE_ARG (Profile option), 222HIDE_OPTION_ONE_ARG_APPEND (Profile option), 222HTML, 3, 10, 11, 28, 32, 68, 85, 89, 120, 142, 145, 147, 149, 177, 179, 183, 184, 188,

192, 193Hudson, 66, 67

Icons..., 49, 59

, 57, 57, 57, 57, 57, 56, 57, 57, 57, 55–57, 57, 56, 57, 50, 49

! , 60.* , 60?* , 60

, 49, 53, 54, 56, 56, 56, 56, 56, 55, 56, 53, 56, 56, 57

ok , 56C++CPP , 55, 56

, 56, 56, 56, 56, 55, 56, 53, 56, 49

1.2.3., 61-

+ , 49, 51

ok , 56IGNORE_EXT (Profile option), 219INCLUDE_PATH_OPTION (Profile option), 221INCLUDE_PATH_OPTION_APPEND (Profile option), 221INJECT_COVERAGESCANNER_LIBRARY_AT_END (Profile option), 218INSTRUMENTATION_TABLES_LINKED_DURING_RUNTIME (Profile option), 218INSTRUMENTATION_TABLES_OF_DLL_LINKED_DURING_RUNTIME (Profile option), 218

Jenkins, 162JUnit, 67, 152

LANGUAGE_SELECTION_OPTION (Profile option), 219LANGUAGE_SELECTION_OPTION_APPEND (Profile option), 219LANGUAGE_SELECTION_OPTION_C_KEYWORD (Profile option), 219LANGUAGE_SELECTION_OPTION_CPP_KEYWORD (Profile option), 219LIBRARY_PATH_OPTION (Profile option), 217LIBRARY_PATH_OPTION_APPEND (Profile option), 217LINK_ADDITIONAL_ARGUMENTS (Profile option), 217LINK_LIBRARY_OPTION (Profile option), 217LINK_LIBRARY_OPTION_APPEND (Profile option), 217LINK_OUTPUT_DEFAULT (Profile option), 217LINK_OUTPUT_OPTION (Profile option), 217LINK_OUTPUT_OPTION_APPEND (Profile option), 217LINKER_HIDE_OPTION_NO_ARG (Profile option), 216LINKER_HIDE_OPTION_ONE_ARG (Profile option), 216LINKER_HIDE_OPTION_ONE_ARG_APPEND (Profile option), 216Linux, 1, 10, 20, 159, 174, 190, 203, 205LOCALE_SYSTEM_FILE_ENCODING (Profile option), 221

make, 15, 17Meta-Object Compiler (moc), 2, 19, 32, 78, 100Microsoft Visual Studio, 1, 7, 9, 11, 13, 72, 80, 83, 84, 97, 101, 105–109, 118, 134,

162–164, 166, 167, 173, 182, 185–190, 192, 212, 218Microsoft Visual Studio .NET, 72, 105, 212Microsoft Visual Studio Add-In, 1, 7, 8, 109, 162–164, 182, 185, 189, 190, 192Microsoft Windows, 1, 20, 34, 76, 79, 86, 87, 121, 168, 174, 189, 203, 205, 223, 224Microsoft Windows CE, 107Minimum Comment Size, 69MINIMUM_COMMAND_LINE_SIZE_FOR_RESPONSE_FILE (Profile option), 223moc, 2, 19, 32, 78, 100Mono C#, 13, 212MSBuild, 13, 166

OBJ_EXT (Profile option), 216OBJECT_DIR_OPTION (Profile option), 221OBJECT_DIR_OPTION_APPEND (Profile option), 221

PARALLEL_COMPILATION_OPTION (Profile option), 220PARALLEL_COMPILATION_OPTION_APPEND (Profile option), 220PCH_EXT (Profile option), 221PCH_OUTPUT_DEFAULT (Profile option), 221PCH_OUTPUT_OPTION (Profile option), 221PCH_OUTPUT_OPTION_APPEND (Profile option), 221PDB_EXT (Profile option), 222PDB_OUTPUT_OPTION (Profile option), 222PDB_OUTPUT_OPTION_APPEND (Profile option), 223PLUGIN_REGISTRATION_API (Profile option), 86, 218pop

pragma, 94PREPROCESSOR_CMD (Profile option), 216PREPROCESSOR_DEFINE_OPTION (Profile option), 215PREPROCESSOR_DEFINE_OPTION_APPEND (Profile option), 216PREPROCESSOR_DEFINE_SEPARATOR_OPTION (Profile option), 216PREPROCESSOR_HIDE_OPTION_NO_ARG (Profile option), 215PREPROCESSOR_HIDE_OPTION_ONE_ARG (Profile option), 215PREPROCESSOR_HIDE_OPTION_ONE_ARG_APPEND (Profile option), 215PREPROCESSOR_INCLUDE_OPTION (Profile option), 216PREPROCESSOR_INCLUDE_OPTION_APPEND (Profile option), 216PREPROCESSOR_KEEP_OPTION_ONE_ARG (Profile option), 215PREPROCESSOR_KEEP_OPTION_ONE_ARG_APPEND (Profile option), 215

qmake, 18–20, 24, 25, 99, 100qrc, 19, 32Qt Library, 2, 99, 100, 125

Reload, 69REMOVE_EXT (Profile option), 219RESPONSE_FILE_EXT (Profile option), 223RESPONSE_FILE_OPTION (Profile option), 223RESPONSE_FILE_OPTION_APPEND (Profile option), 223RETURN_INSTRUMENTED_IN_EXPRESSION (Profile option), 224

SBR_EXT (Profile option), 223


INDEX

SBR_OUTPUT_OPTION (Profile option), 223SBR_OUTPUT_OPTION_APPEND (Profile option), 223SCons, 100, 101Shortcut

Ctrl+B, 56, 57Ctrl+F, 53, 56Ctrl+I, 56, 57Ctrl+J, 56, 57Ctrl+K, 56Ctrl+L, 57Ctrl+M, 57Ctrl+R, 57Ctrl+S, 57Ctrl+Shift+B, 56Ctrl+Shift+F, 53, 56Ctrl+Shift+I, 56Ctrl+Shift+J, 56Ctrl+Shift+K, 56Ctrl+Shift+T, 56Ctrl+Shift+U, 56Ctrl+Shift+V, 56Ctrl+Shift+W, 56Ctrl+Shift+Z, 56, 57Ctrl+T, 56Ctrl+U, 56, 57Ctrl+V, 56Ctrl+W, 56Ctrl+Wheel, 55Ctrl+Z, 56, 57Shift+Wheel, 55Wheel, 55

SKIP_GENERATE_COVERAGESCANNER_LIBRARY_OPTION (Profile option), 218SonarQube, 67, 152, 153SOURCE_IGNORE_INSTRUMENTATION (Profile option), 215Statement Coverage, 35STATIC_LIB_AS_DEFAULT (Profile option), 217STATIC_LIB_EXT (Profile option), 217Status

Dead-Code, 56Executed, 55Execution count too low, 56Hidden, 56Manually Set To Be Executed, 56Never Executed, 56Partially Executed, 55Unknown, 56

STDIN_MRI_SCRIPT_OPTION (Profile option), 217SUPPRESS_APTH_ENTRY (Profile option), 224SYSTEM_INCLUDES (Profile option), 221

Tcl, 1–3, 9, 10, 186Test coverage count, 42, 55, 61Thresholds, 69TOOL (Profile option), 219

uic, 19, 32UNIX, 10, 14, 34, 86, 87, 121, 168, 189, 190, 224UNIX_LIBRARY_NAME (Profile option), 217USE_PCH_OPTION (Profile option), 221USE_PCH_OPTION_APPEND (Profile option), 221USE_RESPONSE_FILE (Profile option), 223

VisualDSP, 103, 184VS_DEF_FILE_OPTION (Profile option), 218VS_DEF_FILE_OPTION_APPEND (Profile option), 218

XBuild, 13XML, 142, 147, 151, 152


Squish Coco 3.2.3 - Copyright ©2015 froglogic GmbH · CONTENTS 18.1.5 __coveragescanner _clear...

Documents

Transcript of Squish Coco 3.2.3 - Copyright ©2015 froglogic GmbH · CONTENTS 18.1.5 __coveragescanner _clear...