Squish Coco 4.0 - froglogic · CONTENTS Contents I Overview2 1 Squish Coco – A code coverage tool...

Squish Coco 4.0.3

Copyright © 2003–2011 Sébastien Fricker, 2011–2017 froglogic GmbH

CONTENTS

Contents

I Overview 2

1 Squish Coco – A code coverage tool for Tcl, QML, C# and C/C++ 3

1.1 CoverageScanner – Instrumentation as part of the build process . . . . . . . . . . . . . . . . . . . 4

1.2 CoverageBrowser – Viewing, analyzing and managing code coverage results . . . . . . . . . . . . . 5

II Setup and Tutorials 7

2 Installation and Setup 8

2.1 Choosing a license . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.2.1 Installation of a license server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.2.2 Running the license server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.3 Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3 Instrumentation of a simple project 11

3.1 UNIX and macOS setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.1.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.1.2 Structure of the parser directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.1.3 Compiling and testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.1.4 Instrumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.1.5 How the project is instrumented . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.1.6 Additional changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.2 Microsoft Windows setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.2.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.2.2 Structure of the parser directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.2.3 Compiling and testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15


Squish Coco 4.0.3 - i - froglogic GmbH

CONTENTS

3.2.5 How the project is instrumented . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.2.6 Additional changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.3 Beyond the minimal instrumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.3.1 Excluding code from instrumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.3.2 Making the test names visible . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.3.3 Patch file analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.3.4 The patch analysis report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.3.5 Reverse patches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.3.6 Bug location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4 Getting started with Qt 23

4.1 Compiling the example application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.1.1 The first code coverage results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.1.2 Interactive testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.1.3 Writing unit tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.2 Working with code coverage data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.2.1 Post mortem analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.2.2 Evaluating the impact of a hot fix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.2.3 Black-box testing/distributed testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.2.4 Verifying if a bug fix is correctly tested . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

III About Code Coverage 40

5 Squish Coco’s code coverage analysis 41

5.1 Coverage metrics supported by Squish Coco . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

5.1.1 Coverage metrics and safety standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

5.2 Description of the coverage metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

5.2.1 Statement Block Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

5.2.2 Decision Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

5.2.3 Condition Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

5.2.4 Multiple Condition Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

5.2.5 MC/DC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.3 Display of the results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.4 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Squish Coco 4.0.3 - ii - froglogic GmbH

CONTENTS

5.5 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

5.6 Problems with line coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

6 Testing Methodologies 55

6.1 Hit vs Count . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

6.2 Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

6.2.1 Manual tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

6.2.2 Black box tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

6.2.3 Unit Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

6.2.4 Automatic Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

IV CoverageScanner Reference 58

7 Introduction 59

7.1 Invoking CoverageScanner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

8 Command Line Arguments 61

8.1 Environment variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

8.2 List of options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

8.2.1 Coverage modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

8.2.2 Fine-tuning of the instrumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

8.2.3 File and path inclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

8.2.4 Function inclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

8.2.5 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

8.2.6 Execution report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

8.2.7 Command line modification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

8.2.8 Generated code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

8.2.9 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

8.3 Instrumenting using preprocessor defines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

8.4 Regular Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

9 Library calls 73

9.1 C and C++ Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

9.1.1 __coveragescanner_install() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

9.1.2 __coveragescanner_testname() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

9.1.3 __coveragescanner_teststate() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

Squish Coco 4.0.3 - iii - froglogic GmbH

CONTENTS

9.1.4 __coveragescanner_add_html_comment() . . . . . . . . . . . . . . . . . . . . . . . . . 75

9.1.5 __coveragescanner_clear_html_comment() . . . . . . . . . . . . . . . . . . . . . . . 75

9.1.6 __coveragescanner_save() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

9.1.7 __coveragescanner_clear() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

9.1.8 __coveragescanner_filename() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

9.1.9 __coveragescanner_register_library() . . . . . . . . . . . . . . . . . . . . . . . . . 76

9.1.10 __coveragescanner_unregister_library() . . . . . . . . . . . . . . . . . . . . . . . 77

9.1.11 __coveragescanner_set_custom_io() . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

9.2 C# Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

9.2.1 CoverageScanner.__coveragescanner_init() . . . . . . . . . . . . . . . . . . . . . . 78

9.2.2 CoverageScanner.__coveragescanner_testname() . . . . . . . . . . . . . . . . . . . 78

9.2.3 CoverageScanner.__coveragescanner_teststate() . . . . . . . . . . . . . . . . . . 79

9.2.4 CoverageScanner.__coveragescanner_add_html_comment() . . . . . . . . . . . . . . 79

9.2.5 CoverageScanner.__coveragescanner_clear_html_comment() . . . . . . . . . . . . 79

9.2.6 CoverageScanner.__coveragescanner_save() . . . . . . . . . . . . . . . . . . . . . . 79

9.2.7 CoverageScanner.__coveragescanner_clear() . . . . . . . . . . . . . . . . . . . . . 80

9.2.8 CoverageScanner.__coveragescanner_filename() . . . . . . . . . . . . . . . . . . . 80

9.2.9 CoverageScanner.__coveragescanner_set_custom_io() . . . . . . . . . . . . . . . . 80

10 Controlling the instrumentation during the compilation 82

10.1 Source Code Annotations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

10.1.1 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

10.1.2 Manual Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

10.2 C and C++ Pragma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

10.3 C# regions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

11 Code Coverage of Libraries 86

11.1 Code Coverage of Static/Shared Libraries and DLL . . . . . . . . . . . . . . . . . . . . . . . . . . 86

11.2 Code Coverage of Plugins/Manually Loaded Shared Libraries . . . . . . . . . . . . . . . . . . . . 86

11.2.1 Generating Code Coverage Information directly from the Main Application . . . . . . . . . 86

11.2.2 Generating Code Coverage Information directly from the Plugin . . . . . . . . . . . . . . . 87

11.2.2.1 Code Coverage of Plugins Generated with Microsoft Visual Studio . . . . . . . . 88

11.2.2.2 Code Coverage of Plugins Generated with GNU gcc . . . . . . . . . . . . . . . . 88

Squish Coco 4.0.3 - iv - froglogic GmbH

CONTENTS

12 CoverageScanner Adaptation to a Tool Suite 90

12.1 Profile Parameters for Architecture Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

12.2 Profile Parameters for Instrumentation Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

12.3 Profile Parameters for Preprocessor Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

12.4 Profile Parameters for Linker Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

12.5 Profile Parameters for Compiler Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

12.6 Profile Parameters for Precompiled Headers Support . . . . . . . . . . . . . . . . . . . . . . . . . 98

12.7 Profile Parameters for Custom IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

12.8 Miscellaneous Profile Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

12.9 Instrumentation Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

V QML Coverage Reference 103

13 Qt QML coverage 104

13.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

13.2 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

13.2.1 Finding the right version of the archive . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

13.2.2 Linux™ installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

13.2.3 macOS installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

13.2.4 Microsoft® Windows installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

13.3 CocoQML Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

13.3.1 Compile a Qt application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

13.3.2 Enable code coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

13.3.3 Excluding source files from coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

13.4 Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

VI CoverageScannerTcl Reference 109

14 Command Line Arguments 110

15 Mixed Tcl/C/C++ Code Coverage Analysis 112

15.1 Analysis of Tcl script coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

15.1.1 Demo application build . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

15.1.2 Ad-hoc Tcl coverage run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

15.1.3 Result analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

Squish Coco 4.0.3 - v - froglogic GmbH

CONTENTS

15.1.4 Filtered Tcl coverage run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

15.1.5 Report Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

15.2 Analysis of C/C++ code coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114


15.2.2 Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

15.2.3 Result analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114


15.2.5 Analysis of mixed Tcl and C/C++ code coverage . . . . . . . . . . . . . . . . . . . . . . . 115

15.2.6 Combined coverage run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

15.2.7 Merging measurement data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115


VII CoverageBrowser Reference 117

16 Introduction 118

16.1 Command Line Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

17 Black box and white box testing 119

18 The windows of CoverageBrowser 121

18.1 The Executions Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

18.1.1 Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

18.1.2 Loading an Execution Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

18.1.3 The Execution Comparison Analysis Mode . . . . . . . . . . . . . . . . . . . . . . . . . . 125

18.2 The Source Browser Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

18.3 The Function Browser Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

18.4 The Source Viewer Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

18.4.1 Source Display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

18.4.2 Color Convention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

18.4.3 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

18.4.3.1 Editing Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

18.4.3.2 Removing Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

18.5 The Explanation Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

18.6 The Statistics Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

Squish Coco 4.0.3 - vi - froglogic GmbH

CONTENTS

19 Working with CoverageBrowser 133

19.1 Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

19.1.1 Wildcard Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

19.1.2 Regular Expression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

19.1.2.1 Pattern matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

19.1.2.2 String substitution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

19.2 Code Coverage Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

19.3 Code Coverage Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

19.4 Optimized execution order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

19.5 Bug Location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

19.5.1 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

19.5.2 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

19.6 Patch Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

19.7 Comparing Releases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

19.7.1 Reference Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

19.7.2 Coverage analysis of modified/unmodified source code . . . . . . . . . . . . . . . . . . . . 142

19.8 Changing the Instrumentation Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

19.8.1 Merging Instrumentations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

19.8.2 Importing Unit Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

19.8.3 Importing Reviewer Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

20 Generation of Reports 144

20.1 HTML/CSV Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

20.2 EMMA-XML Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

20.3 Cobertura XML Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

20.4 JUnit Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

20.5 Text Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

21 Preferences 146

21.1 Save/Load Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

21.2 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

21.3 Thresholds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

21.4 Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

Squish Coco 4.0.3 - vii - froglogic GmbH

http://emma.sourceforge.net/

CONTENTS

VIII Other Coco Tools 148

22 cmreport – Code coverage report generation 149

22.1 File selection options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

22.2 Execution selection options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

22.3 Release comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

22.4 Options for HTML or CSV reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

22.4.1 Obsolete options for CSV reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

22.5 Options for text reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

22.6 Options for EMMA-XML reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

22.7 Options for JUnit reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

22.8 Options for Cobertura reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

23 cmcsexeimport – Command Line Import Utility 156

24 cmmerge – Merging Utility 158

25 cocolic – License activation 159

26 cocolicwizard – GUI for license activation 161

27 cocolicserver – License Server 162

IX Howtos 164

28 Creating an instrumented project 165

28.1 C++ on Microsoft Visual Studio using the Microsoft Visual Studio Add-in . . . . . . . . . . . . . . 165

28.2 C# on Microsoft Visual Studio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

28.3 Tcl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

28.3.1 Using more than one Tcl version on one system . . . . . . . . . . . . . . . . . . . . . . . . 168

28.4 Command Line Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168

29 Generating instrumentations without modifying projects 170

29.1 GNU Make . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

29.2 Microsoft NMake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

29.3 Microsoft Visual Studio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

29.4 Microsoft MSBuild . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

29.5 Mono C# XBuild . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

Squish Coco 4.0.3 - viii - froglogic GmbH


CONTENTS

30 Test suites and Squish Coco 172

30.1 Execution comment, name and status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

30.2 Unit testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

31 Control of execution report generation 174

31.1 Execution report generation with UNIX signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174

31.2 Execution report generation with Microsoft Windows events . . . . . . . . . . . . . . . . . . . . . 174

31.2.1 A program to send events to applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

31.2.2 Global events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

31.2.3 Write your own program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176

32 Customizing I/O of CoverageScanner library 178

32.1 Custom I/O using C file access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

32.2 Custom I/O using SFTP protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180

33 Generation of diff files 187

33.1 Comparing directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

33.2 Version control systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

X Coco Integration Handbook 189

34 Support for IDEs and toolkits 190

34.1 GNU Makefiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

34.2 CygWin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

34.3 Scratchbox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

34.4 Kitware CMake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

34.4.1 Adding a new build type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

34.4.2 Microsoft Visual Studio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

34.4.3 Microsoft NMake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

34.4.4 GNU GCC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

34.5 Qt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

34.5.1 Qt Creator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

34.5.2 qmake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

34.5.3 moc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

34.5.4 qbs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

34.6 SCons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

Squish Coco 4.0.3 - ix - froglogic GmbH

http://www.cygwin.com

CONTENTS

34.7 ARM Keil . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

34.8 Green Hills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

34.8.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

34.8.2 Command Line Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

34.8.3 CoverageScanner Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

34.9 Visual DSP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

34.10Microsoft Visual Studio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

34.10.1 Microsoft Visual Studio .NET C# Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . 203

34.10.2 Microsoft Visual Studio .NET C++ Compiler . . . . . . . . . . . . . . . . . . . . . . . . . 203

34.10.3 Squish Coco Microsoft Visual Studio Add-In . . . . . . . . . . . . . . . . . . . . . . . . . 206

34.10.4 Microsoft Visual C++ Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208

34.10.5 Microsoft Visual Studio 6.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208

34.11eMbedded Visual C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

34.12Eclipse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211

34.13Xcode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212

35 Support for specific test frameworks 214

35.1 CppUnit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

35.2 QTestLib . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216

35.3 GoogleTest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218

35.4 CxxTest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

35.5 NUnit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

35.6 Squish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

35.6.1 General Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225

35.6.2 Simplified for Reuse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226

36 Atlassian Bamboo integration 229

36.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229

36.2 Agent configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230

36.3 Importing an execution report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230

36.4 Merging instrumentation databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230

36.5 Generating coverage report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230

36.6 Use Case Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231

36.6.1 Coverage Report from single execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231

36.6.2 Coverage Report for Unit Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232

Squish Coco 4.0.3 - x - froglogic GmbH

CONTENTS

37 Jenkins CI integration 234

37.1 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234

37.1.1 Getting a license . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234

37.1.2 Getting an EMMA plugin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235

37.2 Adaption of the project to Jenkins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235

XI Frequently Asked Questions 237

38 Installation 238

38.1 Is there a way to deploy Squish Coco on Windows without any user interactions? . . . . . . . . . . 238

38.2 The Microsoft Visual Studio Add-In is not installed . . . . . . . . . . . . . . . . . . . . . . . . . . 238

38.3 Microsoft Visual Studio Add-In is crashing when "My Documents" is on a network drive . . . . . . 240

38.4 Installation problems on macOS 10.12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240

38.5 Code coverage for C# is no longer working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240

39 Compiling 241

39.1 Using CoverageScanner with ccache (Unix only) . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

39.2 Compiling issue when using the Boost library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

39.3 Microsoft MSBuild Does Not Call CoverageScanner on x86 Platforms . . . . . . . . . . . . . . . . 242

40 Instrumentation 243

40.1 Compiling Issues of Lambda Expressions with Microsoft VisualStudio 2010? . . . . . . . . . . . . 243

40.2 How to exclude a source file from the code coverage analysis? . . . . . . . . . . . . . . . . . . . . 243

40.3 My source code contains inline functions, are they instrumented? . . . . . . . . . . . . . . . . . . . 244

40.4 Is it normal that the .csmes file contains a copy of the source code? . . . . . . . . . . . . . . . . . . 244

41 CoverageBrowser 245

41.1 Is there a way to minimize the CPU usage? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245

41.2 When I’m exporting the statistics per methods I can only see the 65536 first entries . . . . . . . . . 245

42 Unit Tests 246

42.1 How to get the code coverage of a library from my unit tests? . . . . . . . . . . . . . . . . . . . . . 246

43 Reports 248

43.1 Is is possible to compute a code coverage report of selection of files? . . . . . . . . . . . . . . . . . 248

Squish Coco 4.0.3 - xi - froglogic GmbH

http://jenkins-ci.org/

http://ccache.samba.org/

CONTENTS

44 Miscellaneous 249

44.1 Is it possible to debug the CoverageScanner API? . . . . . . . . . . . . . . . . . . . . . . . . . . . 249

44.2 How to report an issue concerning CoverageScanner? . . . . . . . . . . . . . . . . . . . . . . . . . 249

XII Appendix 251

A Release Notes 252

A.1 Squish Coco v4.0.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253

A.2 Squish Coco v4.0.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253

A.3 Squish Coco v4.0.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253

A.4 Squish Coco v4.0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253

A.5 Squish Coco v3.4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254

A.6 Squish Coco v3.4.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254

A.7 Squish Coco v3.3.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255

A.8 Squish Coco v3.3.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255

A.9 Squish Coco v3.3.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256

A.10 Squish Coco v3.3.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256

A.11 Squish Coco v3.2.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

A.12 Squish Coco v3.2.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

A.13 Squish Coco v3.2.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

A.14 Squish Coco v3.2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

A.15 Squish Coco v3.1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

A.16 Squish Coco v3.0.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259

A.17 Squish Coco v3.0.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260

A.18 Squish Coco v3.0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260

A.19 Squish Coco v3.0.0-pre1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260

A.20 Squish Coco v2.1.8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261

A.21 Squish Coco v2.1.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262

A.22 Squish Coco v2.1.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262

A.23 Squish Coco v2.1.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

A.24 Squish Coco v2.1.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264

A.25 Squish Coco v2.1.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264

A.26 Squish Coco v2.1.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265

A.27 Squish Coco v2.1.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265

Squish Coco 4.0.3 - xii - froglogic GmbH

CONTENTS

A.28 Squish Coco v2.1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266

A.29 Squish Coco v2.0.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266

A.30 Squish Coco v2.0.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266

A.31 Squish Coco v2.0.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267

A.32 Squish Coco v2.0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267

B File system and registry 268

B.1 Location of the installed files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268

B.1.1 Microsoft Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268

B.1.2 Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

B.1.3 macOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

B.2 Location of the license . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

B.2.1 Node-locked licenses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

B.2.2 Address of the license server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270

B.3 Location of the program settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270

B.4 Location of the installation log file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270

C Supported compilers 271

C.1 C# compilers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271

C.2 Microsoft Visual C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271

C.3 Intel C++ Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272

C.4 GNU gcc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272

C.4.1 Special compiler versions for cross-compiling . . . . . . . . . . . . . . . . . . . . . . . . . 272

D Code insertion 273

E Code Coverage Benchmarks 277

E.1 Test Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277

E.2 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278

F License Agreement 280

Index 283

Squish Coco 4.0.3 - 1 - froglogic GmbH

Part I Overview


Squish Coco – A code coverage tool for Tcl, QML, C# and C/C++

Chapter

1Squish Coco – A code coverage tool for Tcl,QML, C# and C/C++

Squish Coco is a complete code coverage tool chain for Tcl, QML, C# and C/C++ programs that runs on macOS,Linux™ and Microsoft® Windows. It analyzes the performance of an application being exercised (e.g. by a test suite),to reveal important information that can improve the testing of Tcl, QML, C# and C/C++ applications.

In particular, Squish Coco helps to

• find untested code sections.

• find redundant tests which can then be eliminated.

Squish Coco can identify which portions of the source code are covered by a test. It can detect whether a newtest covers lines in the source code that the existing tests do not cover.

• find dead code by displaying code that is never executed.

• calculate the optimum test execution order so as to maximize test coverage for each run.

This is particularly useful for manual testing.

• analyze two separate versions of an application and compare the differences.

This makes it possible to see which tests are affected by source code modifications and also to get somemeasure of the test coverage of a patch or hot fix.

Squish Coco can be used at every stage of testing and for all testing methodologies (unit tests, automatic tests, manualwhite box tests, black box tests, etc.). A test suite can be divided into independent parts: Squish Coco makes itpossible to merge multiple execution reports into a single unified report.

The main components of the Squish Coco package are two separate tools and an add-in:

1. CoverageScanner – a program to analyze and instrument a Tcl, QML, C# and C/C++ application.

2. CoverageBrowser – a program to display and manage the results of the coverage analysis.

3. The Microsoft® Visual Studio® Add-In – an optional add-in to configure code coverage for C# or C++ projectsthat are developed under Microsoft® Visual Studio®.

There are also several helper programs for report generation, coverage data manipulation and license management.


CoverageScanner – Instrumentation as part of the build process

CoverageScanner CoverageBrowser

Source Code

(.cpp file)C/C++ Preprocessing

Instrumentation

Compiling

Instrumentation(.csmes file)

Executions

(.csexe file)

Application

(.exe file)

Figure 1.1: Overview of the code coverage toolchain for Tcl, QML, C# and C/C++

1.1 CoverageScanner – Instrumentation as part of the buildprocess

CoverageScanner instruments the source code that is seen by the compiler without touching the original code. Itgenerates an instrumented executable, shared library or plugin. The instrumentation keeps track of the code that isexecuted when the executable is run or when the library or plugin is used. This allows to measure e.g. how manytimes each line is executed, or whether a line was executed at all. When the instrumented program terminates, itproduces an execution report.

In order to get high-quality coverage measurements, CoverageScanner not only instruments at the function andstatement level but also records statement coverage and decision coverage information. Optionally it is also possibleto insert the test name and execution status (i.e. “passed” or “failed”) into the report. This can be done by a scriptor directly from the application. CoverageScanner can then be integrated into a test framework (like CppUnit orCxxTest), where it generates separate coverage information for each test.

CoverageScanner is a command line tool that replaces the compiler normally used to build the executable, library, orplugin. It inserts special instrumentation instructions into the preprocessed source code and then uses the originalcompiler to compile the modified code. At the same time, a database (the .csmes file) is generated for later analysis.It contains the list of instrumentations and a copy of the instrumented source code.

After compilation, the instrumented executable can be run as usual (and an instrumented library or plugin can be usedvia the program that accesses them), but now, when the program terminates, an execution report is produced (the.csexe file).

CoverageScanner works with almost any standard compiler, including GNU gcc, GNU g++, Microsoft® VisualStudio® 6.0, .NET, Express, and Embedded C++, Intel® C++, etc. For most other compilers, CoverageScanner can beeasily configured.

i CoverageScanner has built-in support for the Qt Framework which makes it possible to disable the instrumen-tation of the moc-generated code.


http://qt-project.org

http://qt-project.org/doc/qt-4.8/moc.html

CoverageBrowser – Viewing, analyzing and managing code coverage results

1.2 CoverageBrowser – Viewing, analyzing and managingcode coverage results

The execution report in the .csexe file can be analyzed and managed with the CoverageBrowser tool. This programhas a GUI (graphical user interface) by which the user can browse and manage their execution reports. This makes itstraightforward to locate untested or unreachable code and inefficient tests.

A summary of CoverageBrowser’s abilities:

• Adding comments to instrumented source code lines.

• Displaying the list of executions in a tree view.

• Marking portions of code which cannot be tested as “Manually Validated”.

• Browsing through the instrumented code.

• Switching between the coverage levels: decision coverage, condition coverage and statement block coverage.

• Displaying a detailed explanation of the state of the instrumentation. This includes:

– User comments.

– The state each instrumented statement: executed, not executed, partially executed.

– The number of times an instrumented statement was executed.

– A list of the tests in which an instrumented statement was executed.

• Retrieving the execution status from an automated test suite.

• Code coverage analysis of unit tests.

• Exporting results to spreadsheets.

• A test benefit analysis mode makes it possible to see which additional code coverage a set of tests adds.

• An API which makes it possible to control CoverageBrowser by an external Tcl, QML, C# and C/C++application.

• A source code and function browser which displays code coverage statistics for each file, each class, eachnamespace and for each Tcl, QML, C# and C/C++ function.

• Generation of a report in HTML format. It includes statistics per source file, methods, executions, and a list ofunexecuted code lines.

• Black box testing support.

• The ability to compare coverage data for different versions of the software.


CoverageBrowser – Viewing, analyzing and managing code coverage results

Navigation

Source Navigation

and Statistics

Executions

Management

Manual

Validation

User

Comments

Detailed

Explanation

Class, Namespace,

and Method

Browser

Figure 1.2: CoverageBrowser: Code coverage screen shot


Part II Setup and Tutorials


Installation and Setup

Chapter

2Installation and Setup

This chapter describes the installation and setup of Squish Coco on your machine and a later update of software andlicenses.

2.1 Choosing a license

There are two kinds of licenses, node-locked and floating licenses.

A node-locked license is bound to a specific machine and a specific user account. On that account, you can run asmany Squish Coco processes as you like. Note that if you run Squish Coco from a Continuous Integration Server, theserver has a separate account; in this case Squish Coco needs an additional node-locked license.

Floating licenses are bound to processes, not to computers. One needs as many floating licenses as there are SquishCoco processes running at the same time. A Squish Coco process can be

1. an instance of CoverageScanner or of the compiler wrappers,

2. an instance of the CoverageBrowser,

3. each instance of the command line tools, cmmerge, cmcsexeimport and cmreport.

Every compiler call that is instrumented by Squish Coco is counted separately because it requires one instance of theCoverageScanner to run. Builds with several compilations in parallel should however cause no real problem since aprocess that cannot get a license will wait until one is freed.

Floating licenses are managed by a license server (see Chapter 27, page 162). The server must be running on amachine that is reachable from the machine on which Squish Coco runs. froglogic will provide a license file whichdetermines the maximal number of Squish Coco processes that can be active at the same time. (The license serveritself is not counted among the processes that need a license.)


Installation

2.2 Installation

An installer for Squish Coco can be downloaded via http://www.froglogic.com/secure.

The installers for the various platforms supported by Squish Coco have a common name scheme, but the way they mustbe executed differs. The naming scheme is SquishCocoSetup_x.y.z_〈platform〉.〈suffix〉, where x.y.z is the programversion, 〈platform〉 describes the operating system and other details of the installation, and 〈suffix〉 is an operatingsystem specific suffix. (In the noncommercial version, SquishCoco is replaced by SquishCocoNoncommercial.)

Microsoft® Windows:The installer is a file SquishCocoSetup_x.y.z_Windows_x86.exe or SquishCocoSetup_x.y.z_Windows_x64.exe and must be executed.

Linux™:The installer is a file SquishCocoSetup_x.y.z_〈platform〉.run and must be executed with bash, as in

$ bash SquishCocoSetup_3.3.2_Linux_x86_64.run

Note that the CentOS version of the installer is also valid for RedHat Linux™.

macOS:The installation package is a file of the form SquishCocoSetup_x.y.z_〈platform〉.pkg. Click on it and theinstaller will be started.

If no valid license is present for your account, the installer will run the License Wizard (see Chapter 26, page 161) atthe end of the installation. The License Wizard will then allow to configure the license.

2.2.1 Installation of a license server

If you have chosen a floating license, you need to specify a machine on your local network on which the licenseserver program runs.

The license server needs a configuration file to run. It specifies the number of licenses served, the machine on whichit runs, and the port used by the server. To specify the configuration file, Squish Coco must be installed first.

Install therefore Squish Coco on the license server machine. After the installation, cocolicwizard will be started.Ignore it. Instead, run the command cocolicserver --server-identifier and redirect its output to a file. Howthis is done depends slightly on the operating system.

Microsoft® Windows:In a command window, type

c:\project> "〈Windows Coco〉\cocolicserver.exe" --server-identifier > machine.txt

Linux™:Open an command shell and type

$ /opt/SquishCoco/bin/cocolicserver --server-identifier > machine.txt

macOS:Open an command shell and type

$ /Applications/SquishCoco/cocolicserver --server-identifier > machine.txt


http://www.froglogic.com/secure

Updates

This will generate a file machine.txt in the directory in which the command was executed. Send it to froglogic,together with the number of requested licenses (and, if necessary, the port number).

2.2.2 Running the license server

froglogic sends you a configuration file, which is usually called licserver.cfg. With it, the license server can run.In the simplest case, write

$ cocolicserver -c licserver.cfg

The server then provides licenses to other machines in the same network. For additional options of cocolicserver, seeChapter 27, page 162.

2.3 Updates

A new license can be installed with with cocolic or cocolicwizard.

To update to a newer version of Squish Coco, download and install it. It will then overwrite the previous version. Thelicense will not be touched, you can continue to use it.

If a new compiler is installed, then the Squish Coco installer should be run again, so that Squish Coco becomes awareof the new compiler. This includes new versions of Microsoft® Visual Studio®: If Visual Studio 2013 was presentduing the Squish Coco installation and Visual Studio 2015 is added, the Squish Cocoinstaller needs to be run.

If instead the compiler is updated, Squish Coco should work mostly without changes.

! One exception is the C# compiler. If it is updated, the Squish Coco setup will be overwritten. SquishCoco needs then to be reinstalled to make code coverage working again.


Instrumentation of a simple project

Chapter

3Instrumentation of a simple project

In this section we will show a small project with unit tests and show how it can be instrumented. The project is asimple expression parser, and it has not many requirements besides a C++ compiler.

The project replicates in miniature an existing project that has been extend with unit tests and for which we will nowuse Squish Coco to find out how good the test coverage is. Instrumentation should therefore be non-intrusive andshould not change the project very much.

i The procedures to set up a project for instrumentation under UNIX® and macOS differ from that for Microsoft®

Windows. The setup section has therefore two versions, which appear in the following sections.

3.1 UNIX® and macOS setup

3.1.1 Setup

The parser example can be found in Squish Coco’s installation directory. Under UNIX®, this is the directory /opt/SquishCoco/ or, if you have installed Squish Coco locally, the subdirectory SquishCoco/ of your home directory.1

Under macOS, the installation directory is /Applications/SquishCoco/.

We will refer to it as the SquishCoco/ directory, wherever it is located. The Squish Coco examples, together withtheir supporting programs, are in SquishCoco/samples/, and the parser is in SquishCoco/samples/parser/.This directory contains three versions of the program, in the directories parser_v1/ to parser_v3/. They representthe parser in different stages of its development.

The example uses CppUnit as its unit test framework. There is a version of CppUnit in the SquishCoco/samples/directory, and the parser example is prepared to use it.

Copy therefore now the content of the whole samples/ directory to your workspace and make parser_v1/ yourworking directory. If Squish Coco is installed in /opt/SquishCoco/, this is done in the following way:

$ cp -a /opt/SquishCoco/samples .$ cd samples/parser/parser_v1

1If you have chosen a custom directory to install Squish Coco, the example will not work without modifications.


http://www.freedesktop.org/wiki/Software/cppunit

UNIX and macOS setup

Make also sure that the Squish Coco tools are in your search path. If they are not so already, you can now do it bywriting

$ . cocopath.sh

(Do Not forget the dot at the beginning!) Now programs like coveragebrowser can be called from the commandline.

3.1.2 Structure of the parser directories

We we will use samples/parser/parser_v1/ as our working directory. It contains C++ source files and headerfiles, together with an unit test file, unittests.cpp. The makefile has been prepared for unit tests, but not forinstrumentation. The instrumentation is done with the help of the bash script instrumented. (There are also somefiles that are needed in the Microsoft® Windows version. We will ignore them here.)

i The makefile is called gmake.mak to distinguish it from the Microsoft® Windows makefile, nmake.mak.Therefore one has to write make -f gmake.mak in places where otherwise a make would be enough.

3.1.3 Compiling and testing

Run “make -f gmake.mak” to compile the program. It is a simple expression parser and calculator.

$ ./parserEnter an expression an press Enter to calculate the result.Enter an empty expression to quit.

> 2 + 2Ans = 4

> PiAns = 3.14159

> sin(Pi)Ans = 1.22465e-16

> sinn(90)Error: Unknown function sinn (col 9)

> sin(90)Ans = 0.893997

> cos(pi)Ans = -1

>$

We have added some unit tests for the main class, Parser. Look into the file unittests.cpp to see the tests thathave been included. Execute it with make -f gmake.mak tests. You will see that 14 tests have been executed, andalso that one of them has failed. This is for greater realism and also allows us to see later how Squish Coco handlestest failures.



3.1.4 Instrumentation

We have kept the instrumentation separate from the main project. The core of the instrumentation is a short shellscript, instrumented. It is a simple wrapper, and calling “instrumented 〈command〉” executes 〈command〉 witha few environment variables set. We will do this now. Enter

$ make -f gmake.mak clean$ ./instrumented make -f gmake.mak tests

The first command removes all object files, since we need everything to be recompiled. The second command thencompiles the program with instrumentation and runs the tests. That’s all!

We now have a look at what the script has done and how it has done it. List the contents of your parser directory:

$ lsconstants.h functions.o.csmes parser.cpp unittests.oerror.cpp instrumented parser.h unittests.o.csmeserror.h LICENSE parser.o variablelist.cpperror.o main.cpp parser.o.csmes variablelist.herror.o.csmes main.o unittests variablelist.ofunctions.cpp main.o.csmes unittests.cpp variablelist.o.csmesfunctions.h Makefile unittests.csexefunctions.o NOTICE unittests.csmes

You see two kinds of files that do not appear as result of normal compilation. The .csmes files contain the informationthat is needed for coverage measurement, and the .csexe files contain the results of code execution. The files thatend in .o.csmes are temporary files and are only used during compilation.

This time, the only program that was actually executed was unittests, and therefore the only .csexe file isunittests.csexe. To see the coverage results, you can therefore start the CoverageBrowser with the command

$ coveragebrowser -m unittests.csmes -e unittests.csexe

Then the CoverageBrowser will start with a modal window, "Load Execution Result". Click on the "Import"button to load the data. (This button will also appear als "Import & Delete", depending on the settings of thebrowser.)

!By default, CoverageBrowser automatically deletes the .csexe file after it loads it. You can switchthis behavior off by unselecting the "Delete after loading" checkbox. Or, if you do not, select the"FileÕSave" menu item to save the execution report in the unittests.csexe file.

For the use of CoverageBrowser, see Part VII, page 118. We will now rather describe how the instrumentation isdone.

3.1.5 How the project is instrumented

The file instrumented is a short bash script:

#!/bin/bash

. getcoco.sh # Get Coco variables

export PATH=$COCO_WRAPPER_DIR:$PATH



export COVERAGESCANNER_ARGS=’--cs-on’

"$@"

At its beginning, the shell script getcoco.sh sets the shell variable COCO_WRAPPER_DIR. It contains the name of thedirectory in which Squish Coco is installed. Then there are two export statements, and the final cryptic statementexecutes the command line parameters of instrumented. So if you call ./instrumented make -f gmake.maktests, the command make -f gmake.mak tests is executed by the script, but in a different environment than

normally.

The important part of it are therefore the two export statements. In the first one, the search part is manipulated sothat the programs in /opt/SquishCoco/wrapper/bin/ are searched first. This directory contains a lot of files withthe same names as the compilers2 that are supported by Squish Coco:

$ ls /opt/SquishCoco/wrapper/binar g++-4.9 x86_64-linux-gnu-arc89-gcc gcc x86_64-linux-gnu-g++c99-gcc gcc-4.6 x86_64-linux-gnu-g++-4.6...

These programs are the compiler wrappers. With the new PATH, they are executed instead of the real compilers. Thecompiler wrappers are actually symbolic links to a single program, coveragescanner (see Part IV, page 59). Whenexecuted to compile a source file, they create an instrumented version of the source and then run the original compilerto compile it.

In the second export statement, additional flags for the compiler wrappers (see Chapter 8, page 61) are set. Here weset only one option, --cs-on. If it is not present, the compiler wrappers are inactive and just call the compilers theyrepresent.

3.1.6 Additional changes

It is also convenient to add make targets to handle the files generated by CoverageScanner. In the parser directory,the Makefile has been changed in the following way:

clean: testclean...-$(DEL_FILE) *.o.csmes # (added)

distclean: clean...-$(DEL_FILE) *.csmes *.csexe # (added)

Since the .o.csmes files are needed only for compilation, they can be deleted whenever the .o files are deleted(which is that what make clean does). The .csmes and .csexe files are more precious and should only be deletedwhen all generated files are removed. Therefore we have added their deletion statements to the distclean target.

2There is also ar, which is not a compiler but takes part in the compilation process. This is the output under UNIX®; under Mac OS X, thedirectory /Applications/SquishCoco/wrapper/ contains a much smaller list of files.


Microsoft Windows setup

3.2 Microsoft® Windows setup

3.2.1 Setup

The parser example can be found in the directory 〈Windows Coco〉\parser. This directory contains three versionsof the program, in the subdirectories parser_v1 to parser_v3. They represent the parser in different stages of itsdevelopment.

The example uses CppUnit as its unit test framework. There is a version of CppUnit in the 〈Windows Coco〉\squishcocodirectory, and the parser example is prepared to use it.

Since these directories are write-protected, you need to create your own working copies. Copy therefore thetwo directories 〈Windows Coco〉\squishcoco\parser and 〈Windows Coco〉\squishcoco\cppunit-1.12.1 to adirectory of your choice. Then remove the write protection of the directories and all the files contained in them.

3.2.2 Structure of the parser directories

We will use parser\parser_v1 as our working directory. It contains C++ source files and header files, togetherwith an unit test file, unittests.cpp. The Makefile has been prepared for unit tests, but not for instrumentation.The instrumentation is done with the help of the batch file instrumented.bat. (There are also some files that areneeded in the UNIX®version. We will ignore them here.)

i The makefile is called nmake.mak to distinguish it from the UNIX® makefile, gmake.mak. Therefore one hasto write nmake /F nmake.mak in places where otherwise a nmake would be enough.

3.2.3 Compiling and testing

We will do the compilation of the example on the command line. To get a command window, execute the batch fileCocoCmd.bat that is located in the parser_v1 directory. In this window, the Microsoft® Visual Studio® commandline tools (like nmake) are accessible, and also the main Squish Coco programs (like CoverageBrowser).

Run “nmake /F nmake.mak” to compile the program. It is a simple expression parser and calculator.

$ C:\code\parser\parser_v1>parser.exeEnter an expression an press Enter to calculate the result.Enter an empty expression to quit.

> 2 + 2Ans = 4

> PiAns = 3.14159

> sin(Pi)Ans = 1.22465e-16

> sinn(90)Error: Unknown function sinn (col 9)

> sin(90)Ans = 0.893997

> cos(pi)Ans = -1


http://www.freedesktop.org/wiki/Software/cppunit


>C:\code\parser\parser_v1>

We have added some unit tests for the main class, Parser. Look into the file unittests.cpp to see the tests thathave been included. Execute it with nmake /F nmake.mak tests. You will see that 14 tests have been executed,and also that one of them has failed. This is for greater realism and also allows us to see later how Squish Cocohandles test failures.


We have kept the instrumentation separate from the main project. The core of the instrumentation is a short shell script,instrumented.bat. It is a simple wrapper, and calling “instrumented.bat 〈command〉” executes 〈command〉with a few environment variables set. We will do this now. Enter

C:\code\parser\parser_v1>nmake /F nmake.mak cleanC:\code\parser\parser_v1>instrumented.bat nmake /F nmake.mak tests

The first command removes all object files, since we need everything to be recompiled. The second command thencompiles the program with instrumentation and runs the tests. That’s all!

We now have a look at what the script has done and how it has done it. List the contents of your parser directory:

C:\code\parser\parser_v1>dir /D

Directory of C:\code\parser\parser_v1

[.] LICENSE unittests.exe[..] main.cpp unittests.exe.csexeconstants.h main.obj unittests.exe.csmeserror.cpp main.obj.csmes unittests.experror.h Makefile unittests.liberror.obj nmake.mak unittests.objerror.obj.csmes NOTICE unittests.obj.csmesfunctions.cpp parser.cpp variablelist.cppfunctions.h parser.h variablelist.hfunctions.obj parser.obj variablelist.objfunctions.obj.csmes parser.obj.csmes variablelist.obj.csmesinstrumented README.squishcocoinstrumented.bat unittests.cpp

35 File(s) 1,079,919 bytes2 Dir(s) 35,159,457,792 bytes free

You see two kinds of files that do not appear as result of normal compilation. The .csmes files contain the informationthat is needed for coverage measurement, and the .csexe files contain the results of code execution. The files thatend in .obj.csmes are temporary files and are only used during compilation.

This time, the only program that was actually executed was unittests, and therefore the only .csexe file isunittests.exe.csexe. To see the coverage results, you can therefore start the CoverageBrowser with the com-mand

C:\code\parser\parser_v1>coveragebrowser -m unittests.exe.csmes -e unittests.exe.csexe



Then the CoverageBrowser will start with a modal window, "Load Execution Result". Click on the "Import"button to load the data. (This button will also appear als "Import & Delete", depending on the settings of thebrowser.)

!By default, CoverageBrowser automatically deletes the .csmes file after it loads it. You can switchthis behavior off by unselecting the "Delete after loading" checkbox. Or, if you do not, select the"FileÕSave" menu item to save the execution report in the unittests.csexe file.

For the use of CoverageBrowser, see Part VII, page 118. We will now rather describe how the instrumentation isdone.

3.2.5 How the project is instrumented

The file instrumented.bat is a short batch file:

@echo offsetlocal

set PATH=%SQUISHCOCO%\visualstudio;%PATH%set COVERAGESCANNER_ARGS=--cs-on

call %*

endlocal

The variable SQUISHCOCO contains the name of the directory in where Squish Coco is installed. It is set by SquishCoco during installation.

At the beginning, the setlocal command ensures that the following commands change the environment variables onlytemporarily. Then there are two set statements, and the final call statement executes the command line parameters ofinstrumented. So if you call “instrumented nmake /F nmake.mak tests”, the command “nmake /F nmake.mak tests” is executed by the batch file, but in a different environment than normally. At the end, endlocal undoesthe changes in the environment variables.

The important part of the script are therefore the two set statements. In the first one, the search part is manipulatedso that the programs in 〈Windows Coco〉\squishcoco\visualstudio are searched first.3 This directory containsfiles with the same names as the compilers and the linker

C:\code\parser\parser_v1>dir /d "\Program Files\squishcoco\visualstudio"

Directory of C:\Program Files\squishcoco\visualstudio

[.] cl.exe link.cspro msvcr100.dll[..] lib.cspro link.execl.cspro lib.exe msvcp100.dll

8 File(s) 5,662,299 bytes2 Dir(s) 35,053,502,464 bytes free

The .exe files in this directory are the compiler wrappers. With the new PATH, they are executed instead of the realcompiler. The compiler wrappers are actually copies of a single program, coveragescanner.exe (see Part IV, page 59).

3This is the version for the 32 bit compiler. There is also a directory 〈Windows Coco〉\\visualstudio_x64 for 64 bit compilation.


Beyond the minimal instrumentation

When executed to compile a source file, they create an instrumented version of the source and then run the originalcompiler to compile it.

In the second set statement, additional flags for the compiler wrappers (see Chapter 8, page 61) are set. Here we setonly one option, --cs-on. If it is not present, the compiler wrappers are inactive and just call the compilers theyrepresent.

The resulting script should work without changes for many simple projects. If more customization is needed, it canoften be achieved by adding more options to COVERAGESCANNER_ARGS.

3.2.6 Additional changes

It is also convenient to add make targets to handle the files generated by CoverageScanner. In the parser_v1directory, the file nmake.mak has been changed in the following way:

clean: testclean...-$(DEL_FILE) *.obj.csmes # (added)

distclean: clean...-$(DEL_FILE) *.csmes *.csexe # (added)

Since the .obj.csmes files are needed only for compilation, they can be deleted whenever the .obj files are deleted(which is that what make clean does). The .csmes and .csexe files are more precious and should only be deletedwhen all generated files are removed. Therefore we have added their deletion statements to the distclean target.

3.3 Beyond the minimal instrumentation

In the following sections we will show additional abilities of Squish Coco. They will require small changes in thecode of the project.

3.3.1 Excluding code from instrumentation

The coverage information generated so far has a problem: It covers too many files. The problematic files are thosethat belong to the testing framework and not to the tested program. Including them would create artificially lowcoverage rates.

With Squish Coco, one can exclude files from coverage by additional command line options. In parser_v2, this hasbeen done. Look into parser_v2/instrumented (or parser_v2\instrumented.bat under Microsoft® Windows).In it, three additional command line options have been set, which we will now explain:

• The option --cs-exclude-path=../../cppunit-1.12.1 excludes the source files of a directory and all itssubdirectories. Here we use it to exclude all the files of the CppUnit framework.

You can use slashes or backslashes with this option—Squish Coco normalizes them internally.

• The options --cs-exclude-file-wildcard=unittests.cpp and --cs-exclude-file-wildcard=CppUnitListener.cppexclude specific files. We use them to exclude the files unittests.cpp and CppUnitListener.cpp. (The



second file is described below.)

3.3.2 Making the test names visible

For the next modification, we want to change the project such that we know not only whether a line of code is coveredby tests, but also by which tests it is covered. For this we will add calls of the CoverageScanner C/C++ library (seeChapter 9.1, page 73) to the code, to tell Squish Coco the names of the tests and where they begin and end.

An updated version of the project can be found in the directory parser_v2. The greatest difference to the version inparser_v1 is that the file CppUnitListener.cpp has been added. It is copied almost verbatim from Chapter 35.1.The file contains a class CppUnitListener and a new main() function. The main() function in unittests.cpphas been removed, but the file is otherwise unchanged.

CppUnitListener.cpp provides a unit test listener which allows to hook into the framework before and after theexecution of each test. One can thus record additional test information, like the name and the result of a test, inthe code coverage data without modifying the test code itself. (For a listing of CppUnitListener.cpp and anexplanation how it works, see Chapter 35.1)

Now you can execute this program with the same way as its previous version. View the results in the CoverageBrowser.In the "Executions" subwindow you can now see the code coverage for each single test. You can especially see thatit was the test testInvalidNumber that had failed.

Hover with your mouse over one of the lines in the source window, and select a line that was marked in dark green.This means that it was executed by the tests4. You will see a tooltip with a list of the tests that executed this line. Tokeep the tooltip short, there is a maximal number of tests to display there. To see a full list, click on the line. Then thefull information about the line is shown in the "Explanation" window.

3.3.3 Patch file analysis

Now consider the following scenario: In a large project, a last-minute patch has to be evaluated. It is not enough timeto run the full test suite, but some risk assessment needs to be done. For situations like this, Squish Coco provides thefeature of patch analysis. With it, one can specifically display the code coverage for the changed lines of code, andfind the tests in a large suite that cover them. One can now see how risky the changes are.

We will simulate this situation in our example. In a new version of the parser, the character classification functionsin the code, isWhiteSpace(), isAlpha(), etc., have been changed and use the standard C classification functions,like isspace(), instead of strchr(). The new version of the parser can be found in the directory parser_v3.

We will now compare it with the version in parser_v2, but neither run the tests nor even compile it. Instead we needthe following two pieces of information:

1. The coverage data from parser_v2, as generated in the previous section.

2. A patch file showing the differences between the two directories. There is already a diff file in the parserdirectory that you can use, parser.diff.

The diff file must be in the “unified” difference format. This is the standard output format of the difffunctionality of many version control systems, e.g. of git diff (see Chapter 33, page 187). Under UNIX®-like systems, the patch file can also be generated by the diff utility. It would be invoked from the parserdirectory in the following way:

$ diff -u parser_v2 parser_v3 > parser.diff

4Lines in bright green were also executed, but not instrumented by Squish Coco.


https://en.wikipedia.org/wiki/Diff_utility#Unified_format


There is also a Microsoft® Windows version of GNU diff in the parser directory, therefore the same commandworks in a Windows command shell too.

Start the CoverageBrowser. Then load the instrumentation database parser_v2/unittests.csmes via the menuentry "FileÕOpen...", and the measurements file parser_v2/unittests.csexe with "FileÕLoad ExecutionReport...".

Now select the menu entry "ToolsÕPatch File Analysis...". When the "Patch File Analysis" dialogappears:

• Enter a title in the "Title" box.

• Enter the path to the patch file in the "Patch File" box.

• Enter a path (including the file name) for the report in the "Type" box.

• Set the "Tooltip Maximal Size" to 5 (or any value greater than zero).

• Make any other option adjustments.

Then click "Open" to view the report in the browser.

3.3.4 The patch analysis report

The report consists of three tables that summarize the influence that the patch has on code coverage, and then anannotated version of the patch file.

The two tables in the section “Overview” contain statistics about the number and kind of the lines that were influencedby the patch.

The first table groups the patched lines in the code by the results of the tests that executed them. One can see herehow much influence the patch has on tests that have passed (and now could fail) or failed (and could now succeed).There are also entries for manually checked tests and for those whose status is unknown. In our example, we did notregister the test results and all our tests are counted as “Unknown”.

The second table shows the kind of changes to expect in the test coverage after the patch has been applied. It consistsof three columns, containing the statistics about removed and inserted lines and their sum. From the first two columnsone can see whether the test coverage for the patched code grows or falls. (In the parser example, it stays the same.)The last line in the table is also important: It shows the number of lines which Squish Coco could not classify asinserted or removed. Patch analysis is a heuristics, after all.

The section “List of tests influenced by the modifications” is a list with the names of the tests that executed thepatched code, together with their result. It is helpful for a qualitative analysis of the patch. In our example, we cansee that all tests execute code that is affected by the patch.

Figure 3.1: Coverage of the patched lines by the tests



More details can be found in the “Patch File” section of the report. It is an annotated version of the original patch file,with the old version of the text in red and the new version in green. Lines that did not change are shown in gray. Themost important column is “Tests”, which shows for each code line the the number of tests that executed it (if it isremoved) or will probably execute it after the patch is applied. A tooltip shows the names of these tests.

3.3.5 Reverse patches

Now suppose that a patch has already been checked in. It has changed the application’s behavior, and you want towrite more test to make sure that all new code has been covered. In this case, a reverse patch analysis is helpful.

To simulate a reverse patch analysis with the parser example, switch to the directory parser_v3 and compile the codethere in the same way as before parser_v2. Then start the CoverageBrowser and load this time the instrumentationdatabase parser_v3/unittests.csmes and the measurements file parser_v3/unittests.csexe. This can alsobe done on the command line, e.g. in the form

$ cd parser_v3$ coveragebrowser -m unittests.csmes -e unittests.csexe

Generate the patch analysis report as before. Squish Coco recognizes automatically that patch.diff is a patch to thecurrent version of the program, and generates a corresponding report. This patch report contains the same data as thereport for the forward patch, but the headers have changed to reflect the new interpretation.

3.3.6 Bug location

CoverageBrowser can compute the probable location of an error in the source code by comparing the coverage dataof all tests that were run on a program.

The algorithm expects that there at least one test had failed. In our example, the parser does not detect correctly thatthe exponent of a floating point number was invalid. It is possible to write the following:

$ cd parser_v2$ ./parserEnter an expression an press Enter to calculate the result.Enter an empty expression to quit.

> 1E+Ans = 1

But 1E+ is not a valid number because it has no digits after the +.

To find out which line in the source code causes this, we need a few additional tests. The following tests are alreadypart of the test suite in the file unittests.cpp:

Test input Expected result Execution status1.1 1.1 Passed

1.1e1 11 Passed1.1e+1 11 Passed1.1e-1 0.11 Passed1.1e+ error message Failed

With these tests already contained in the test suite, we can use CoverageBrowser to find a candidate for the line thatcaused the bug. There is a "Bug Location" window in the CoverageBrowser, but it is by default disabled. You canenable it by checking the field "Bug Location" in the menu "ViewÕBug Location".



After the window is enabled, select all the executions in the "Executions" window and click the "Compute" button.The window will then be filled with a list of possible bug locations, with the most probable (according to SquishCoco) first. In our case there is only one candidate: the line 263 of the file parser.cpp. This line is the locationwhere the sign ’+’ of the exponent is parsed.

At this point a fix could be implemented, but since this is only a demonstration, we will not do it here.

Figure 3.2: Bug location for the parser sample


Getting started with Qt

Chapter

4Getting started with Qt

The following example is more complex. We will take Qt’s TextEdit example and use it to illustrate how SquishCoco can be used at different stages of the development process. To cover the whole coding cycle, we will first showhow an instrumented application is created, perform manual tests and analyze their results. Then we will create aninstrumented unit test.

In a second step (see Chapter 4.2, page 33), we will cover the aspects which are more interesting to product managers:analyzing the impact of code changes (e.g. bug fixes)—in particular, tracking their test progress, externalizing testing,and collecting the code coverage analysis of a complete testing team.

i The Qt framework can be downloaded directly from https://www.qt.io/developers.

The modified TextEdit sample is available in the doc directory of the Squish Coco installation. Copy the directorydoc/textedit to your working directory.

4.1 Compiling the example application

In this section, we will work with the files in the directory textedit/textedit_v1.

We want to be able to build our application both normally and with generated test coverage instrumentation code,without having to change our source code. This can be achieved by making a small change to the application’s project(.pro) file. We can then use a command line option for qmake to generate an instrumented build instead of a normalone.

To make the project file suitable both for normal and for instrumented builds, we create a set of definitions that canbe activated by a command line switch; in qmake’s terminology this is called a scope. The following listing (seeFigure 4.1, page 24) shows a minimal scope for code instrumentation.

The following must be done:

• We must ensure that precompiled headers are disabled when the code is instrumented. qmake allows us to dothis by setting the PRECOMPILED_HEADER variable to an empty value.


https://www.qt.io/developers

Compiling the example application

• It is also necessary to increase the value of QMAKE_LINK_OBJECT_MAX in order to disable the usage of thelinker script. We set it here to 10000.

• Finally, qmake must be instructed to use CoverageScanner’s wrappers for compilation. This is done byprefixing the names of the compilation tools with cs.

CodeCoverage {PRECOMPILED_HEADER=QMAKE_LINK_OBJECT_MAX=10000

QMAKE_CC=cs$$QMAKE_CCQMAKE_CXX=cs$$QMAKE_CXXQMAKE_LINK=cs$$QMAKE_LINKQMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIBQMAKE_AR=cs$$QMAKE_ARQMAKE_LIB=cs$$QMAKE_LIB}

Figure 4.1: Minimal qmake configuration

These modification are sufficient for most standard C and C++ applications. For Qt applications we must useadditional settings in order to ensure that CoverageScanner does not instrument the source code that is generated byQt’s tools (e.g., by uic, qrc and the moc).

To exclude qrc resource files from instrumentation, we must tell CoverageScanner not to instrument any file with aname that begins with qrc_. This can be done with the command line option --cs-exclude-file-abs-wildcard=

*/qrc_*. Since we do not want to have to enter this option manually, we will put it in the .pro file. Similarly, to letCoverageScanner ignore the files generated by uic we can use the same command line option, only this time with adifferent file matching regular expression: --cs-exclude-file-abs-wildcard=*/ui_*.

Squish Coco also supports by default applications that are built with the Qt 4 toolkit.1 This ensures that Coverage-Scanner:

• does not instrument the Q_OBJECT and Q_DECLARE_PLUGIN macros, and

• does not instrument code generated by the moc, except that signal emissions and slot receives are instrumented,since they are vital to a Qt program’s logic.

We can also exercise some control over the level of instrumentation and what information is reported. For example,we can switch on the counting of code executions with the --cs-count command line option, or we can enable fullinstrumentation at decision/condition level with the --cs-full-instrumentation option. With the --cs-outputoption we can specify the file the execution report is written to when the application terminates. (By default theoutput is written to the file 〈appname〉.csexe, where 〈appname〉 is the name of the program that has been executed.)

So, for a Qt 4-based application, the final Squish Coco scope in the application’s .pro file will look something likethis:

1If needed, this behaviour can be disabled with the option --cs-no-qt4.





CodeCoverage {COVERAGE_OPTIONS = --cs-output=textedit.exeCOVERAGE_OPTIONS += --cs-exclude-file-abs-wildcard=*/qrc_*

QMAKE_CFLAGS += $$COVERAGE_OPTIONSQMAKE_CXXFLAGS += $$COVERAGE_OPTIONSQMAKE_LFLAGS += $$COVERAGE_OPTIONS

QMAKE_CC=cs$$QMAKE_CCQMAKE_CXX=cs$$QMAKE_CXXQMAKE_LINK=cs$$QMAKE_LINKQMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIBQMAKE_AR=cs$$QMAKE_ARQMAKE_LIB=cs$$QMAKE_LIB}

Figure 4.2: Final qmake configuration

In this form, the scope has been added to the project file textedit_v1/textedit_v1.pro. When we now runqmake without options, a Makefile for a normal build is generated; but we can also build an instrumented version ofthe program in the following way:

Linux™/macOS Microsoft® Windows

qmake CONFIG+=CodeCoveragemake

qmake CONFIG+=CodeCoveragenmake

In either case we still end up with a textedit.exe executable. But with an instrumented build we will also get anadditional file, textedit.exe.csmes. It contains the instrumentation database.

4.1.1 The first code coverage results

For our very first exercise we will simply execute TextEdit and then quit the application straight away. This willgenerate a file called textedit.exe.csexe. The file is in a binary format, so it is not human readable. It is aninstrumentation database that contains a snapshot of the most recent execution that we have just done.

To see the results we must run the CoverageBrowser tool and load the textedit.exe.csmes instrumentationdatabase. This can be done with the menu entry "FileÕOpen...". After the file has been opened, no coverageinformation is available because no execution reports have been imported. The instrumented code lines are showngrayed out and no coverage statistics has been computed (see Figure 4.3, page 26).



Figure 4.3: CoverageBrowser after loading the TextEdit’s instrumentation database

In order to see an execution report, click "FileÕLoad Execution Report...", which invokes the import dialog.You need to enter at least the file name (including the full path) of the textedit.exe.csexe file into the field "FileName". You also need to give the test a name. Enter in the field "Name" e.g. the text “Start and Quit”. Switch the"Delete after loading" option on because the report is no longer needed after our import. It is also helpfulto set the option "When file becomes modified" to “Open this dialog”, because then the file import dialog isautomatically opened after each run. The new .csexe file that it created in the test can then be added to the database.

After the import has finished, the code coverage information is visible:

• The coverage statistics for the functions and methods of all source files of the application is shown.

• The source window is now colored. It shows executed code on a green background and unexecuted code on ared background.

• The execution list now contains one selected item called “Start and Quit”, the only test execution report wehave cerated so far.

4.1.2 Interactive testing

CoverageBrowser shows reveals, for example, that the TextEdit::fileSave() function is not executed. We willnow validate this function interactively, guided by the code coverage analysis.

In the source window, all unexecuted source code lines are shown on a red background (see Listing 4.4, page 27).



bool TextEdit::fileSave(){

if (fileName.isEmpty()){QMessageBox::warning(this, tr("No file name specified"),tr("Save first your document using ’Save As...’ from the menu"),QMessageBox::Ok);

return false;}

QTextDocumentWriter writer(fileName);bool success = writer.write(textEdit->document());if (success)

textEdit->document()->setModified(false);return success;

}

Figure 4.4: CoverageBrowser source view of the function TextEdit::fileSave()

To test this function, we must perform the following steps:

1. Start the TextEdit application.

2. Click on the "Save" button: TextEdit should display the error message "Save first your document using ‘SaveAs. . . ’ from the menu".

3. Quit the application.

After these steps have been done and the coverage report imported, CoverageBrowser shows that the “return false;” line just after the call to QMessageBox::warning() has been executed, as indicated by the green background).However, the line “if (fileName.isEmpty())” is partially executed: This is shown by an orange background (seeListing 4.5, page 27).


if (fileName.isEmpty()){QMessageBox::warning(this, tr("No file name specified"),tr("Save first your document using ’Save As...’ from the menu"),QMessageBox::Ok);

return false;}



}

Figure 4.5: CoverageBrowser source view after clicking TextEdit’s ‘Save’ button.



The explanation window (see Listing 4.6, page 28) tells us that the value of the expression “fileName.isEmpty()”was true during one execution but was never false. It is therefore considered as only partially executed. In order tofully test this expression, we must click on the ‘Save As. . . ’ button, then choose a file name, and finally click on the‘Save’ button.

partially executed: fileName.isEmpty()

TRUE FALSE

yesExecution Count: 1Executed by:- Save Clicked

noExecution Count: 0

Figure 4.6: CoverageBrowser explanation window after clicking on the ’Save’ button of TextEdit.

After rerunning the application and doing a “Save as”, the new execution report has only one source code line that ispartially untested (see Listing 4.7, page 28). In this case, CoverageBrowser shows that the Boolean variable successwas never false, which means that saving the document has never failed.

We could force a write failure, and this would ensure that we had 100% code coverage for this function. But we willuse a different test strategy to get complete code coverage: We will use a unit test and import the execution result intothe TextEdit instrumentation database.


if (fileName.isEmpty()){QMessageBox::warning(this,tr("No file name specified"),tr("Save first your document using ’Save As...’ from the menu"),QMessageBox::Ok );

return false;}



}

Figure 4.7: CoverageBrowser source view after clicking TextEdit’s ‘Save As. . . ’ button and then the ‘Save’ button.

4.1.3 Writing unit tests

The unit test infrastructure can be found in the directory textedit_v1_tests/. It contains just one test, which setsan illegal filename and then tries to execute TextEdit’s fileSave() function. To do this, we use the QTestLibunit test library that is part of Qt. The test is contained in the file textedit_v1_tests/tst_textedit.cpp (seeListing 4.8, page 29).



#include "tst_textedit.h"

void TestTextEdit::tst_saveFile() {TextEdit textEdit;textEdit.fileName="/";QVERIFY( !textEdit.fileSave() );

}

QTEST_MAIN(TestTextEdit);

Figure 4.8: Unit test for the TextEdit application, tst_textedit.cpp

To import the instrumentation result of this test into TextEdit’s instrumentation database, the following infrastructureis necessary:

1. A qmake project file with code coverage, configured identically to that of the TextEdit project.

2. A post-build rule, which automatically executes the test and collects the coverage information.

3. A unit test listener, which saves the code coverage data (and the test status—passed or failed) for every executedtest into the unit test’s own instrumentation database.

4. A way to import the code coverage report into TextEdit’s instrumentation database.

The unit test will recompile along with textedit_v1/textedit.cpp. To make its results importable into theTextEdit instrumentation database, it is necessary that both executables, TextEdit and the unit test, are instrumentedin exactly the same way. Since we use only the default instrumentation, this requirement is already fulfilled.

Unfortunately, this is not everything we need to do. Squish Coco’s default behavior is to instrument only headerand source files in the directory it is invoked, but here we need to instrument the TextEdit application’s sources inaddition to the unit test. Therefore we must use another command line option to specify an additional path for files toinstrument: --cs-include-path.

As before, we do not want to have to remember these command line arguments every time, so we set them in theqmake project file, textedit_v1_tests.pro (see Figure 4.9, page 30). With these lines in the unit test projectfile, the qmake-generated Makefile will create the tst_textedit.exe executable which, when run, produces theexecution report tst_textedit.exe.csexe. We then can use the CoverageBrowser to import this report into thefile tst_textedit.exe.csmes.



HEADERS = ../textedit_v1/textedit.h tst_textedit.hSOURCES = ../textedit_v1/textedit.cpp tst_textedit.cpp

CodeCoverage {COVERAGE_OPTIONS = --cs-output=tst_textedit.exeCOVERAGE_OPTIONS += --cs-include-path=../textedit_v1COVERAGE_OPTIONS += --cs-exclude-file-abs-wildcard=*/qrc_*

QMAKE_CXXFLAGS += $$COVERAGE_OPTIONSQMAKE_CCFLAGS += $$COVERAGE_OPTIONSQMAKE_LFLAGS += $$COVERAGE_OPTIONS

QMAKE_CC=cs$$QMAKE_CCQMAKE_LINK=cs$$QMAKE_LINKQMAKE_CXX=cs$$QMAKE_CXX}

Figure 4.9: An extract from the unit test qmake project file

We can also execute the unit test automatically and import the execution report with a post-build rule. Squish Cocoprovides an extra command line tool, cmcsexeimport (see Chapter 23, page 156), which imports an execution reportinto an instrumentation database. The post-build rule first deletes any previous execution report, then executes thetest itself, and finally imports the results into the application’s execution database, tst_textedit.exe.csexe (seeListing 4.10, page 30).

CodeCoverage {win32: MAINDIR=$$replace(PWD,"/","\\")!win32:MAINDIR=$$PWD

unix {QMAKE_POST_LINK = rm $$MAINDIR/tst_textedit_v1.exe.csexe ;QMAKE_POST_LINK += $$MAINDIR/tst_textedit_v1.exe ;QMAKE_POST_LINK += cmcsexeimport -m $$MAINDIR/tst_textedit_v1.exe.csmes \

-e $$MAINDIR/tst_textedit_v1.exe.csexe -t UnitTest}win32 {QMAKE_POST_LINK = del /F $$MAINDIR\\tst_textedit_v1.csexe &QMAKE_POST_LINK += $$MAINDIR\\tst_textedit_v1.exe &QMAKE_POST_LINK += cmcsexeimport -m $$MAINDIR\\tst_textedit_v1.exe.csmes \

-e $$MAINDIR\\tst_textedit_v1.csexe -t UnitTest}}

Figure 4.10: Post-build rules for the import of the execution report into the unit test’s instrumentation database

By default, the coverage data is imported without any information about the executed tests. Instead, an executionreport called ‘UnitTest’ is created, which does not describe which test was executed or whether its execution wassuccessful. To provide the missing information, we must use the CoverageScanner API and generate an executionreport for each test that is executed. An example that shows how this is done is available in the chapter 35.2. In theexample, the API is used in the following way:



• Two Squish Coco source files are added to the qmake project file (see Listing 4.11, page 31).

• The unit test class, TestTextEdit, inherits from TestCoverageObject instead of directly from QObject(see Listing 4.12, page 31).

HEADERS += testcoverageobject.hSOURCES += testcoverageobject.cpp

Figure 4.11: Including the CoverageScanner listener in textedit_v1_tests.pro

#include "testcoverageobject.h"#include "../textedit_v1/textedit.h"#include <QtTest/QtTest>

class TestTextEdit : public TestCoverageObject{

Q_OBJECTprivate slots:void tst_saveFile();

};

Figure 4.12: The TextEdit unit test header file, tst_textedit.h

The source code of the file testcoverageobject.cpp (see Listing 4.13, page 32) is simple to understand. Thefile adds a single cleanup() function to QTestLib, which is executed after each unit test item. The code between#ifdef __COVERAGESCANNER__ and #endif is only compiled when CoverageScanner is invoked2. This extra codecreates a test name by combining the test class’s object name and the test function’s name. In the the example,an execution item called unittest/TestTextEdit/tst_saveFile is generated. The slash is used to support theorganization of tests in a tree view. The current test status (“PASSED” or “FAILED”) is also recorded and added tothe execution report by the __coveragescanner_save() function.

2The symbol __COVERAGESCANNER__ is defined automatically by CoverageScanner and so does not need to be defined manually.



...void TestCoverageObject::cleanup(){cleanupTest();

#ifdef __COVERAGESCANNER__

QString test_name="unittest/";test_name += metaObject()->className();test_name += "/";test_name += QTest::currentTestFunction();__coveragescanner_testname(test_name.toLatin1());if (QTest::currentTestFailed())__coveragescanner_teststate("FAILED");

else__coveragescanner_teststate("PASSED") ;

__coveragescanner_save();#endif}

Figure 4.13: An extract from the TestCoverageObject’s source code

At this stage we could start CoverageBrowser, load the TextEdit instrumentation database, and import the unittest’s instrumentation database by clicking "FileÕImport Unit Tests...". An even more convenient alternativeis to use the cmmerge tool to automate this step. The cmmerge program is designed to import one instrumentationdatabase’s execution report into another instrumentation database. This means that we can extend our post-build rulesto use the cmmerge program to import the coverage information automatically from the unit test into the TextEditprogram’s instrumentation database (see Listing 4.14, page 33).


Working with code coverage data

CodeCoverage {# Merge coverage database into TextEdit databaseunix {QMAKE_POST_LINK += ;QMAKE_POST_LINK += cmmerge -o $$MAINDIR/../textedit_v1/textedit.tmp \

-i $$MAINDIR/../textedit_v1/textedit.exe.csmes \$$MAINDIR/./tst_textedit_v1.exe.csmes &&

QMAKE_POST_LINK += rm $$MAINDIR/../textedit_v1/textedit.exe.csmes &&QMAKE_POST_LINK += mv $$MAINDIR/../textedit_v1/textedit.tmp \

$$MAINDIR/../textedit_v1/textedit.exe.csmes}win32 {QMAKE_POST_LINK += &QMAKE_POST_LINK += echo Merging unit test result into the main application &QMAKE_POST_LINK += cmmerge -o $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes \

-i $$MAINDIR\\..\\textedit_v1\\textedit.exe.csmes \$$MAINDIR\\tst_textedit_v1.exe.csmes &

QMAKE_POST_LINK += COPY /Y $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes \$$MAINDIR\\..\\textedit_v1\\textedit.exe.csmes &

QMAKE_POST_LINK += DEL /F $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes}}

Figure 4.14: Post build rules: merging instrumentation results into the TextEdit instrumentation database

With all these changes to the .pro file in place, we can once again build and run the unit test. Now the Coverage-Browser shows the fileSave() function to be 100% covered, with the execution list containing our three originalmanual tests and the single unit test (see Figure 4.15, page 33).

Figure 4.15: The execution list after all the tests have been executed

4.2 Working with code coverage data

The most common ways in which code coverage is used are for developers to use it to find untested code, and formanagers to use it to produce test status reports (e.g., as diagrams).

In addition to fully supporting the common use cases, Squish Coco also provides additional features which makeit possible to go beyond these fundamentals and extend what can be achieved with code coverage. This will bediscussed in the current subsection.



4.2.1 Post mortem analysis

Recording each test’s coverage data makes it possible to compare their data to answer the question, “What does thistest cover that the others do not?” This is particularly useful if just one test fails, since it can help us to identify whichpart of the code is involved.

To see how this works in practice, let us return to the TextEdit example. If we click ‘Save’, we will get an errormessage that no file name is defined. This is not very convenient for users—we should have designed TextEdit tohandle this particular case by opening the ‘Save As. . . ’ dialog rather than by producing an error.

To identify where in the code this problem arises, we simply compare the ‘Save Clicked’ execution with all otherexecutions that involve the ‘Save’ button. To do this we must first switch to the “Execution Comparison Analysis”mode (Menu: "ToolsÕExecution Comparison Analysis"). Select the checkboxes in the “Reference” columnfor the “tst_saveFile” and “SaveAs clicked before Save clicked” tests. This will make the execution comparisonsymbol appears in front of the affected names (see Figure 4.16, page 34). In the “Executions” column, click on “Saveclicked” checkbox.

Figure 4.16: The execution list being used to compare different executions

In this mode, the coverage analysis is based only on source code lines which are not executed by “tst_saveFile” and“SaveAs clicked before Save clicked”. This is why the overall coverage decreases to 1.29%: It means that “Saveclicked” executes 1.29% more code than the selected tests.

Using cmreport it is possible to generate a HTML report which displays the same information:

cmreport --csmes=textedit_v1/textedit.exe.csmes \--html=textedit.html \--section=execution \--select-reference=".*tst_saveFile" \--select-reference="SaveAs clicked before Save clicked"

If we now look at the source code itself, we will see that only two lines of the TextEdit::fileSave() function (seeListing 4.17, page 35) are not grayed: the lines which pop up the error message. These are the lines that must bemodified to change the “Save” button’s behavior.




if (fileName.isEmpty()){QMessageBox::warning(this,tr("No file name specified"),tr("Save first your document using ’Save As...’ from the menu"),QMessageBox::Ok );

return false;}



}

Figure 4.17: CoverageBrowser source view of the comparison of the “Save clicked” execution.

In this case, changing the fileSave() function is easy—we simply replace the QMessageBox::warning() callwith a call to the fileSaveAs() method. (see Listing 4.18, page 35)


if (fileName.isEmpty())return fileSaveAs();



}

Figure 4.18: The TextEdit’s improved fileSave() function.

4.2.2 Evaluating the impact of a hot fix

Before committing a change or starting to test a hot fix, it is possible to estimate the impact of the code modification.CoverageBrowser is able to perform an analysis on the difference between two source sets and can list the tests thatwill be affected (and those which will not).

Start CoverageBrowser and load the modified TextEdit example’s instrumentation database. Now click on"ToolsÕCompare with..." and select the original version of the TextEdit instrumentation database. Cover-ageBrowser will now display the source code like a text comparison application does (e.g., diff).

Click on "ToolsÕAnalysis of Modified Methods" to exclude all unmodified functions from the coverageanalysis. In the TextEdit case, doing this will mean that only one function, TextEdit::fileSave(), will betreated as being instrumented since that is the only method we have changed (see Figure 4.19, page 36). This also



affects the statistic calculations since execution coverage statistics will now be limited to just this function. The testexecutions whose coverage statistic is not zero are the ones that are affected by the code modifications we have made.

In our case we have:

• “Save clicked”,

• “SaveAs clicked before Save clicked” and

• “tst_saveFile” (our unit test).

The “Start and Exit” case has a coverage of 0% and so does not execute our modified code. It is for this reason nolonger visible in the execution list. All entries in the ‘Execution’ column are struck through to inform us that thesetests are not executed in the newest version and only present in the reference database.

In other words, only the two manual tests and the unit test listed above must be re-executed to ensure that noregressions have been introduced by our code changes.

Figure 4.19: List of tests affected by a code modification

4.2.3 Black-box testing/distributed testing

Up to now we have done white-box testing, i.e. testing where we have access to the source code and which makesuse of our knowledge of the code. It is also possible to use Squish Coco for black-box testing. In other words, wecan still do code coverage analysis without having access to or even knowledge of the source code. If we use thisapproach, the generated instrumentation database will, of course, contain no source code.

To use black-box testing we must create a suitable instrumentation database by clicking "FileÕGenerate Black-BoxConfiguration...". This database, along with the TextEdit executable, can be given to the test team whichcan then use them with a simplified version of CoverageBrowser (see Figure 4.20, page 37). This version of



CoverageBrowser only supports the importing and managing of execution reports since it does not have access to theapplication’s source code.

Figure 4.20: Black-box testing results as shown by CoverageBrowser

Once all the tests are finished, the black-box database can be merged into the original TextEdit instrumentationdatabase using CoverageBrowser’s merge facilities (Menu: "FileÕMerge with...").

4.2.4 Verifying if a bug fix is correctly tested

Often, when a small bug fix is made, the effects are very localized and leave most of the source code unchanged. Inview of this, it is often unnecessary to retest the entire application with the whole test suite.

Squish Coco makes it possible to avoid unnecessary testing. We can tell it to restrict itself to the source code thathas changed between the original and fixed version of the application (see Chapter 4.2.2, page 35). This allowsus to focus purely on the analysis of the fix. To achieve this, simply load the fixed application’s freshly generatedinstrumentation database (e.g., for the modified TextEdit application), and compare it with the earlier database forthe unfixed version, using Squish Coco’s facility for analyzing modified functions.


Conclusion

Figure 4.21: Coverage of the patched function

We have done just such a comparison and the results are shown in in 4.21: The two tests, “Save clicked” and “Startand Exit”, cover 85% of the TextEdit::fileSave() function, the only method that was modified for our fix. Fromthis we know exactly what additional testing is necessary to achieve 100% code coverage for our tests for the fixedversion of the application. CoverageBrowser continues to display the list of missing tests (which are only executedusing the first version of TextEdit) in strikeout style. This gives a hint of what remains for testing effort.

Using cmreport it is possible to generate a HTML report which displays the same information:

cmreport --csmes=textedit_v2/textedit.exe.csmes \--csmes-reference=textedit_v1/textedit.exe.csmes \--html=textedit.html \--section=execution

4.3 Conclusion

Squish Coco provides code coverage analysis which can be applied to all the usual testing techniques: unit, manual,and black-box testing. Squish Coco can easily be told to ignore the generated code produced by the Qt library’s tools(moc, qrc, and uic), so that only the code written by developers is instrumented. Test results can be collected intoa database and can be used to evaluate how much code coverage our tests achieve and to show which statementsare not currently tested. With this information we can target our testing efforts towards 100% test code coverage.In addition, Squish Coco makes it possible to see what effect a code modification would have in terms of test codecoverage without having to test the entire application.



Conclusion

Overall, Squish Coco can help us target our tests to ensure that our applications have as much test coverage aspossible, while avoiding or minimizing test coverage duplication. Furthermore, Squish Coco can help us see whateffects changes to our code have on test coverage, so that we can adapt our test suites accordingly.


Part III About CodeCoverage


Squish Coco’s code coverage analysis

Chapter

5Squish Coco’s code coverage analysis

This chapter describes the code analysis done by Squish Coco in detail.

In order to analyze the code coverage of e.g. a test suite, it is necessary to compile a version of the application inwhich statements are inserted that record the execution of each part of the source code. The generation of such amodified version of a program is called instrumentation.

5.1 Coverage metrics supported by Squish Coco

Several kinds of code coverage are possible with Squish Coco. The most common are:

Statement Block Coverage:With this metric, Squish Coco verifies that all statements are executed. For this it groups the statements ofthe program to blocks. Statements belong to the same block if they are always executed together. (Statementblocks roughly correspond to the “blocks” of most programming languages.) The coverage of a program is thenumber of executed statement blocks in it, divided by the total number of blocks.

Decision (or Branch) Coverage:Here Squish Coco verifies that all statements are executed and all decisions have all possible results. Thecoverage of a program is the number of executed statement blocks and decisions divided by the total number ofstatements and decisions – where each decision counts twice, once for the true case and one for the falsecase.

Condition Coverage:This is like decision coverage, except that the decisions are split into elementary subexpressions (or conditions)that are connected by “and” or “or” operators. The coverage of a program is the number of executed statementblocks and conditions divided by the total number of them.

Here each condition counts twice, which may result in a large number of possible outcomes in a complexdecision.

Modified Condition/Decision Coverage (MC/DC):This is like condition coverage, but every condition in a decision must be tested independently to reach fullcoverage. For each condition there must be two execution that differ only in the results of this conditions, while


Coverage metrics supported by Squish Coco

the other conditions have the same result. In addition, it needs to be shown that each condition independentlyaffects the decision.

The coverage of a program is the number of executed statement blocks and of conditions that were testedindependently divided by the number of statement blocks and conditions in the program.

Multiple Condition Coverage:In this coverage metric, all statements must be executed and all combinations of truth values in each decisionmust occur at least once to reach full coverage. The coverage of a program is the number of executed statementblocks and condition combinations divided by their total number in the program.

Other, simpler coverage metrics are also supported.

Function coverage:With this metric, Squish Coco counts which functions were called and how often. “Functions” includes alsothe member functions of objects, as always with Squish Coco. The coverage of a program is then the numberof functions that were called at least once, divided by the total number of functions.

Relevance for safety standards: IEC 61508 highly recommends 100% structural test coverage of entry points(i.e. functions) for SIL 1, 2, 3 and 4.

Line coverage:Here the number of executed code lines is counted and how often each one of them was executed. Only linesthat contain executable statements are instrumented, not those with pure declarations. The coverage of aprogram is then the number of executed lines divided by the number of instrumented lines.

Line coverage is however an unstable coverage measure since it depends strongly on the way the code isformatted (see Chapter 5.6, page 53). We do not recommend line coverage.

5.1.1 Coverage metrics and safety standards

In many safety standards, specific coverage levels are required.

• IEC 61508 is a standard for the functional safety of electrical and electronic programmable systems.

• ISO 26262 is a standard for the functional safety of electrical and electronic systems in cars.

• EN 50128 is a standard for safety-relevant software in railway systems.

• DO 178 is a safety standard for commercial software-based aerospace systems.

The following table shows which coverage levels are required by these safety standards.

IEC 61508 ISO 26262 EN 50128 DO 178Function Coverage (Entry Points) XStatement Block Coverage X X X XDecision Coverage (Branch Coverage) X X X XModified Condition/Decision Coverage X X X XMultiple Condition Coverage X

Depending on the security levels the coverage requirement is either just recommended, highly recommended orrequired. More detailed information can be found at the end of the descriptions of the coverage metrics in thefollowing sections.


https://en.wikipedia.org/wiki/IEC_61508

https://en.wikipedia.org/wiki/ISO_26262

https://de.wikipedia.org/wiki/EN_50128

https://en.wikipedia.org/wiki/DO-178C

Description of the coverage metrics

5.2 Description of the coverage metrics

In this section we describe the coverage metrics supported by Squish Coco in gerater detail; the code that is insertedduring the instrumentation process is described in more detail in Section D, page 273.

In the next sections, we will use the following function to illustrate the coverage metrics and the instrumentationprocess. It is written in C++; coverage with other languages it is similar.

1 void foo()2 {3 bool found=false;4 for (int i=0; (i<100) && (!found); ++i) {5 if (i==50)6 break;7 if (i==20)8 found=true;9 if (i==30)

10 found=true;11 }12 printf("foo\n");13 }

Figure 5.1: Example function for simple coverage metrics

5.2.1 Statement Block Coverage

The most elementary kind of instrumentation records the statements in a program that are executed when it runs. It ishowever not necessary to record the execution of every statement to get this information. If several statements form asequence, it is enough to record how often the last statement is executed, since they all form a block that is eitherexecuted as a whole or not at all. Squish Cocotherefore inserts instrumentation statements only at the end of eachblock, and the resulting coverage metric is called Statement Block Coverage.

In the following listing, the instrumented statements are underlined:

1 void foo()2 {3 bool found=false; // not instrumented4 for (int i=0; (i<100) && (!found); ++i) {5 if (i==50)6 break;7 if (i==20)8 found=true;9 if (i==30)

10 found=true;11 }12 printf("foo\n"); // not instrumented13 }

Figure 5.2: Code coverage at statement block level



One can see that not all lines are instrumented. The conditions of the if and for statements are not covered sincethey are no statements themselves. Some other statements need not to be instrumented because they are part of ablock. In the example, these are the statements at lines 3 and 12: they belong to the block that begins at line 2 andends with line 13. It is therefore enough to put an instrumentation point at line 13 to verify that all of them wereexecuted.

Relevance for safety standards ISO 26262 highly recommends Statement Coverage for ASIL A and B. ASIL Cand D recommend it as well but the more strict Branch Coverage and Modified Condition/Decision Coverage arehighly recommended instead.

5.2.2 Decision Coverage

A more detailed coverage metric also records the values of the Boolean conditiona in branch and loop statements,like if, while, for, etc. To reach full coverage, the decision in such a construct must have evaluated to true and tofalse at least once. This kind of code coverage is called Decision Coverage or sometimes Branch Coverage.

Decision coverage also handles switch statements. In them, full coverage is reached if every case label in the codewas reached at least once during the execution of the program.

Note that Decision coverage also include the coverage of statements, as in Statement Block Coverage. In the followinglisting, the conditions instrumented for Decision Coverage are displayed with a gray background. The instrumentedstatements are underlined as before. They are the same as in the previous listing.

1 void foo()2 {3 bool found=false;4 for (int i=0; (i<100) && (!found) ; ++i) {5 if (i==50 )6 break;7 if (i==20 )8 found=true;9 if (i==30 )


Figure 5.3: Code coverage at decision level

Relevance for safety standards ISO 26262 recommends Branch Coverage for ASIL A and highly recommendsthe method for ASIL B, C and D.

EN 50128 recommends Branch Coverage for SIL 1 and 2. For SIL 3 and 4 this level is even highly recommended.

DO 178 mandates that Decision Coverage should be satisfied with independence for Software Level A and B.

IEC 61508 recommends Branch Coverage for SIL 1 and 2 and highly recommends this level for SIL 3 and 4.



5.2.3 Condition Coverage

Often one wants to analyze the Boolean decision in the if, while, for, etc. statements in greater detail. For this onecan use Condition Coverage.

In this coverage metric, each decision is decomposed into simpler statements (or conditions) that are connected byBoolean operators like ’~’, ’||’ and ’&&’. Then for full coverage of the decision, each of the conditions must evaluateto true and to false when the program is executed.

Note that ’||’ and ’&&’ are shortcut operators and that therefore in a complex decision, not all conditions are alwaysexecuted. An example is the code fragment ‘if (a || b)return 0;’. In it, the conditions are the variables a andb. To get them both evaluate to true and false, one has to run the code fragment once with a == true and barbitrarily (since it is not evaluated), and also two times with a == false: in one run b is set to true, in the otherrun it is set to false.

This is in contrast to the situation with decision coverage, where it is enough for full coverage that the whole decision,’a || b’, evalutes to true and to false.

With Condition Coverage, our example function is instrumented in the following way. As before, the instrumentedBoolean expressions are gray :

1 void foo()2 {3 bool found=false;4 for (int i=0; (i<100) && (!found) ; ++i) {5 if (i==50 )6 break;7 if (i==20 )8 found=true;9 if (i==30 )


Figure 5.4: Code coverage at condition level

We can see here that the decision of the for statement has been split into two separately instrumented conditions.

Instrumentation of assignments CoverageScanner also instruments the assignment of Boolean expressions to avariable. Constant or static expressions are not instrumented. This is because their values are computed at compiletime or only once during program initialization, so full coverage cannot be reached anyway.

In the following listing, the instrumented Boolean expressions are again gray :



1 void foo()2 {3 static bool a = x && y;4 bool b = x && y ;5 bool c = b;6 int d = b ? 0 : 1;7 }

Figure 5.5: Code coverage of assignments

We see that the statements at lines 3 and 5 are not instrumented – the first is evaluated at compile time and the seconddoes not involve a Boolean operation.

Potential problems with Boolean overloading Since Squish Coco works on a syntactic level, it cannot distinguishbetween expressions that involve only Booleans and those with other data types that support Boolean operators – allof them are instrumented.

The instrumentation tries to have no effect on the program, but sometimes this is not possible. One example are olderversions of C#. They need to be instrumented with the option --cs-no-csharp-dynamic: Then the instrumentedcode converts the operands of a Boolean operator like ’||’ and ’&&’ first to Boolean, before the operator is applies.(Short circuit evaluation still works, of course.) This is different from the way in which an uninstrumented programwould do it.

In C# programs that are compiled with Microsoft® Visual Studio® versions before 2010, it is therefore necessary thatboth cast operators true and false are defined for the objects that are arguments of Boolean operators. For newerC# versions, the default settings of the CoverageScanner can be used and the instrumented code does not have thisproblem.

5.2.4 Multiple Condition Coverage

The properties of Multiple Condition Coverage and MC/DC become only visible when a program contains a complexdecision with many conditions. Our previous example therefore cannot be used anymore. Instead, we will use thefollowing program, which replaces some letters in a string with underscores: those between ‘a’ and ‘e’ (inclusive) inthe alphabet, and those that follow the full stop at the end of the sentence. (There is no character after the full stop inthe example text, but that is on purpose: We need some conditions that fail.)



1 #include <stdio.h>2

3 int main()4 {5 char text[] = "The quick brown dog jumps over the lazy fox.";6

7 for (char *p = text; *p; p++) {8 if ((*p >= ’a’ && *p <= ’e’) || (p != text && *(p - 1) == ’.’))9 *p = ’_’;

10 }11

12 puts(text);13 return 0;14 }

Figure 5.6: Example program for Multiple Condition Coverage and MC/DC

We will concentrate on the complex condition in line 8. When the coverage of this program is shown in theCoverageBrowser, one can move the mouse over that line and see the following table, which describes the coveragedata. (Or one can click on the line and see a similar table in the "Explanation" window.)

*p >= ’a’ *p <= ’e’ p != text *(p - 1) == ’.’ Description

TRUE FALSE TRUE TRUE never executed

FALSE TRUE TRUE never executed

TRUE FALSE TRUE FALSE executed 27 times by 1 test

FALSE TRUE FALSE executed 9 times by 1 test

TRUE FALSE FALSE never executed

FALSE FALSE executed 1 time by 1 test

TRUE TRUE executed 7 times by 1 test

Figure 5.7: Multiple Condition Coverage table for line 8

In this table, each line contains a combination of condition results. The first four columns contain the results of asingle condition. Since C uses shortcut operators, not all conditions are shown executed in each row. Rows that wereexecuted have a green background, rows that were not have a background in red.

From the current table we see that three combinations of conditions were not executed and are missing from fullcondition coverage. To execute e.g. the condition combination in the second red line, it would be necessary to makethe condition *p >= ’a’ false and the conditions p != text and *(p - 1)== ’ ’ true. To reach this, one couldput a capital letter (for the first condition) behind the full stop (for the two others) in text.

Relevance for safety standards EN 50128 recommends MCC (or Modified Condition/Decision Coverage) forSIL 1 and 2. For SIL 3 and 4 this level is even highly recommended.

5.2.5 MC/DC

Multiple Condition/Decision Coverage (MC/DC) is a variant of Multiple Condition Coverage that requires fewertests. Its goal is to make sure that for each condition in a complex decision there are two executions that differ only inthe result of that condition and the condition result.



The condition table of MC/DC is a more complex version of the table for Multiple Condition Coverage:

*p >= ’a’ *p <= ’e’ p != text *(p - 1) == ’ ’ Decision Description

TRUE FALSE TRUE TRUE TRUE never executed

FALSE TRUE TRUE TRUE never executed

TRUE FALSE TRUE FALSE FALSE executed 27 times by 1 test

FALSE TRUE FALSE FALSE executed 9 times by 1 test

FALSE FALSE FALSE executed 1 time by 1 test

TRUE TRUE TRUE executed 7 times by 1 test

TRUE FALSE FALSE FALSE never executed

Figure 5.8: MC/DC table for line 8

Its large-scale structure is the same as with Multiple Condition Coverage, but there are some new features:

• The column headers for the conditions and the decision have a colored background. If a condition has a green

background, the MC/DC conditions for this condition are already fulfilled, otherwise it is red .

• The red rows contain combinations of truth values that did not occur when the program was executed.Executing one of them would increase the code coverage.

• The light green rows contain combinations of truth values that did occur in the execution of the program. If

they contribute to the coverage of a condition, they contain fields with a dark green background.

To see which rows contributed to the coverage of a covered condition, search in the column that belongs to thisconditions for dark green fields. Any two rows with a dark green field contribute together to the coverage ofthe condition if one of the dark green fields has a TRUE entry and the other one a FALSE entry.

In the example, we see that the first condition, ‘*p >= ’a’’, is covered in two different ways: by the secondgreen row together with the fourth green row, and also by the third and fourth green row together. The secondcondition, ‘*p <= ’e’’, is covered by the first and fourth green row together. One can also see that these tworows only differ in the second column and in the decision; the other columns contain the same results or atleast one empty field.

• The light red rows contain combinations of truth valued that were not executed either, but executing a singleof them will not increase the code coverage.

You may have noted that the table in sorted in a different way from that one for Multiple Condition Coverage: Firstthere are the red rows, then the green rows and then the light red ones. This is always the case.

Relevance for safety standards ISO 26262 recommends MC/DC for ASIL A, B, C and highly recommends thismethod for ASIL D.

EN 50128 recommends MC/DC (or Multiple Condition Coverage) for SIL 1 and 2. For SIL 3 and 4 this level is evenhighly recommended.

DO 178 mandates that MC/DC should be satisfied with independence for Software Level A.

IEC 61508 recommends MC/DC for SIL 1, 2 and 3 and highly recommends this level for SIL 4.


Display of the results

5.3 Display of the results

The CoverageBrowser is a graphical user interface program to display the analyzed results of the instrumentation. Ituses a color coding to indicate the status of the statements. In this manual we use the same colors as in the program.

There are two kinds of statements in an instrumented program. Some contain an instrumentation point, i.e. a piece ofcode inserted by Squish Coco which increments a counter when it is executed. If a line contains an instrumentationpoint, it is shown one a dark-colored background by the CoverageBrowser and in the HTML reports.

For reasons of efficiency, not all statements contain instrumentation points. If a line does not contain an instrumenta-tion point but its coverage status can be inferred from other statements, it is shown on a light-colored background.

The resulting color scheme is then:

• Lines in dark green and in light green have been executed.

• Lines in dark red and in light red have not been executed.

• Lines in orange contain boolean expressions that have been partially executed. These expressions occur ingeneral either as part of control structures like while and if or as the right side of an assignment to a booleanvariable.

Boolean expressions are always instrumented, therefore a light orange background is not necessary.

The output for our example function illustrates this:

1 void foo()2 {3 bool found=false;4 for (int i=0; (i < 100) && (!found); ++i)5 {6 if (i==50)7 break;8 if (i==20)9 found=true;

10 if (i==30)11 found=true;12 }13 printf("foo\n");14 }

Figure 5.9: Code coverage output

The for statement in line 4 is shown in orange since it contains the expression ’i<100’, which is only partiallyexecuted: it always takes the value true when the function is executed. The expressions ’i==50’ and ’i==30’ werealso partially executed, and we can see from the way the statements ’break;’ and ’found=true;’ are colored thatin both cases the condition evaluates to false. (In CoverageBrowser, the value of the condition is also shown in atooltip.)

Lines 3, 5, and 13 are not directly included in the coverage measurements and are therefore shown on a lightbackground. Their coverage states are inferred by Coco from other statements that were (or were not) executed later.In our example, lines 3 and 13 must have been executed because the closing brace in line 14 has been executed, andline 3 has been executed because of line 12. Therefore all the lines with light background are shown in green.


Performance

5.4 Performance

The insertion of code during the instrumentation increases the code size and also affects performance of theinstrumented application. It will use more memory and run slower.

For non-conditional expressions, the instrumentation code is only a write instruction to a counter at a fixed memorylocation. However, for conditional expressions, more detailed analysis is needed and this is more computationallyexpensive.

Overall, an instrumented application will be 60% to 90% larger and will run 10% to 30% slower.

i Detailed measurements are available in the Appendix (see Chapter E, page 277).

5.5 Statistics

Some developers write more lines of code than others simply by using a particular coding style—for example byputting opening braces on a line of their own rather than on the same line as an if statement. Therefore Squish Cocouses by default a coverage metric that is not susceptible to such differences in coding style. Its calculations are basedon the number of executed instrumented instructions compared with the total number of instrumented instructions.

Every instrumented simple statement (like return, break, the last instruction of a function, etc.) is recorded bya single instrumentation counter. A fully instrumented condition in an IF...THEN...ENDIF block uses has twoinstrumentation counters: one for the true case and one for the false case. If the code is only partially instrumented,only one condition is recorded (either the false case or the true case).

The statistics itself depends of the type of instrumentation. It is in general not possible to compare code coverage atstatement block level with that at decision level: having 80% coverage at decision level does not tell us anythingabout the coverage at statement block level, which could be bigger or smaller. We can only be sure that reaching100% coverage is more difficult with condition coverage than with decision or statement block coverage.

In our example code, the coverage is between 60% and 75%. The following table shows the details for eachinstrumentation type.

Code coverage type Instrumented Executed Coveragestatement block (see Figure 5.10, page 51) 5 3 60%Decision partial (see Figure 5.12, page 52) 9 7 77%Decision full (see Figure 5.11, page 51) 13 9 69%Condition partial (see Figure 5.14, page 53) 12 9 75%Condition full (see Figure 5.13, page 52) 15 10 66%

In the examples below, the details of the calculations are displayed with subscripts. The first number in a subscriptshows how many instrumentated statements were executed; the second is the number of instrumented statements intotal.


Statistics

1 void foo()2 {3 bool found=false;4 for (int i=0; (i < 100) && (!found); ++i)5 {6 if (i==50)7 break;0/1 [not executed]

8 if (i==20)9 found=true;1/1 [executed]

10 if (i==30)11 found=true;0/1 [not executed]

12 }1/1 [executed]

13 printf("foo\n");14 }1/1 [executed]

Figure 5.10: Code coverage statistics for full statement block instrumentation

1 void foo()2 {3 bool found=false;4 for (int i=0; (i < 100) && (!found)2/2 [was false and true]; ++i)5 {6 if (i==50)1/2 [was false but not true]

7 break;0/1 [not executed]

8 if (i==20)2/2 [was false and true]

9 found=true;1/1 [executed]

10 if (i==30)1/2 [was false but not true]

11 found=true;0/1 [not executed]

12 }1/1 [executed]


Figure 5.11: Code coverage statistics for full decision level instrumentation


Statistics

1 void foo()2 {3 bool found=false;4 for (int i=0; (i < 100) && (!found)1/1 [was false]; ++i)5 {6 if (i==50)1/1 [was false]


8 if (i==20)1/1 [was false]


10 if (i==30)1/1 [was false]


12 }1/1 [executed]


Figure 5.12: Code coverage statistics for partial decision level instrumentation

1 void foo()2 {3 bool found=false;4 for (int i=0; (i < 100)1/2 [was true but not false] && (!found)2/2 [was false and true]; ++i)5 {6 if (i==50)1/2 [was false but not true]


8 if (i==20)2/2 [was false and true]


10 if (i==30)1/2 [was false but not true]


12 }1/1 [executed]


Figure 5.13: Code coverage statistics for full condition level instrumentation


Problems with line coverage

1 void foo()2 {3 bool found=false;4 for (int i=0; (i < 100)1/2 [was true but not false] && (!found)2/2 [was false and true]; ++i)5 {6 if (i==50)1/1 [was false]


8 if (i==20)1/1 [was false]


10 if (i==30)1/1 [was false]


12 }1/1 [executed]


Figure 5.14: Code coverage statistics for partial condition level instrumentation

5.6 Problems with line coverage

Line coverage is a natural metric which allows easily to see which lines of code are executed. But it is less accuratethan the instrumentation at statement block level and its results rely on the developer’s coding style.

The following example illustrates this problem:

int main(){if (true) return 1;foo();return 0;

}

Executing it would produce the following result:

int main(){

if (true) return 1; // Executedfoo(); // Not executedreturn 0; // Not executed

}

This execution correspond to a coverage of 33%.

Since the first line of main contains two executed statements, splitting it in two increases the number of executedlines, and so the test coverage. So if we reformat the main function as follows,

int main(){if (true)return 1;

foo();


Problems with line coverage

return 0;}

the execution of this code results in a code coverage of 66%:

int main(){

if (true) // Executedreturn 1; // Executed

foo(); // Not executedreturn 0; // Not executed

}

Another way to increase the coverage by reformatting is to hide an uncovered statement behind an executed one. Todo it it is only necessary to write the whole code of this main function in one line:

int main(){if (true) return 1; foo(); return 0;

}

This code has a line coverage of 100%:

int main(){

if (true) return 1; foo(); return 0; // Executed}

This small example illustrate how the result depends on how the source code is formatted. Therefore, Squish Cocoprovides the line coverage as additional measurement to the decision and the condition coverage and so does notallow that a source code is only instrumented at line level.


Testing Methodologies

Chapter

6Testing Methodologies

This chapter discusses the various testing strategies that can be carried out so as to make the best use of the SquishCoco tools.

6.1 Coverage Hits vs. Counts

If we want to set very stringent standards for an application to ensure that it is of the highest possible quality, executingjust portions of its code, or executing code just once, is insufficient.

To address this issue the Squish Coco tools provide two possible approaches:

Code Coverage Count:We can require that each portion of instrumented code is executed a minimum number of times. This canbe achieved by adding to the instrumented code so that it counts each execution.1 Naturally, setting suchminimums, and the extra bookkeeping itself, can make the application consume more memory and run a bitslower. An important disadvantage of this approach is that loops in the source code end up having a high countwith only a single execution, so counting this way is not always useful.

Test Coverage Count:We can require that each portion of instrumented code is executed by a minimum number of test items. Thisavoids the code coverage count disadvantage regarding loops, since by setting a minimum number of testexecutions for each portion of instrumented code, we give equal value to loops, recursions, and to instructionsthat are executed only once.

6.2 Testing Strategies

Squish Coco can be adapted to fit in with many different testing strategies, from unit tests during early developmentto complete product validation (e.g., acceptance tests). CoverageBrowser supports the merging of code coverageresults at each stage of testing and so provides a precise overview of the software’s overall quality.

1This can be done using CoverageScanner’s --cs-on command line option.


Strategies

6.2.1 Manual White Box Tests

White box testing (a.k.a. clear box testing, glass box testing or structural testing) uses an internalperspective of the system to design test cases based on internal structure. It requires programming skillsto identify all paths through the software. The tester chooses test case inputs to exercise paths throughthe code and determines the appropriate outputs.

Definition from Wikipedia

Interactive white box tests are easy to do using Squish Coco: once the application has been compiled with Cov-erageScanner, the test engineer can execute the it, and after each execution they can import the execution report intoCoverageBrowserfor analysis.

6.2.2 Manual Black Box Tests

Black box testing takes an external perspective of the test object to derive test cases. These tests canbe functional or non-functional, though usually functional. The test designer selects valid and invalidinput and determines the correct output. There is no knowledge of the test object’s internal structure.


Black box tests assume that test engineers have no knowledge of the application’s internals—they work purely interms of inputs and the corresponding expected outputs (whether of data or of error reports for invalid input).

For this kind of testing it is possible to follow the same pattern as for white box testing, but without providing thetest engineer with the instrumentation database (.csmes file) generated by CoverageScanner. The application itselfwill generate an execution report which can be analyzed after the testing cycle. However, this approach has thedisadvantage that test engineers cannot manage their own execution reports (e.g., to add comments).

Squish Coco has a facility which makes black box testing possible without disadvantaging test engineers. TheCoverageBrowser tool can generate an instrumentation database (.csmes file) which does not contain any sourcecode or source browsing information. This allows test engineers to manage the list of tests and to view the statistics,but it does not provide any source level coverage information.

6.2.3 Unit Tests

Unit testing is a method by which individual units of source code are tested to determine if they arefit for use. A unit is the smallest testable part of an application. In procedural programming a unitcould be an entire module but is more commonly an individual function or procedure. In object-orientedprogramming a unit is often an entire interface, such as a class, but could be an individual method.


CoverageScanner supports the saving of the code coverage data generated by each unit test using its library. (seeChapter 30.2, page 173)

As long as all objects are well compiled2, it is possible to merge the unit tests instrumentation database into those ofthe real application. And so see the code coverage of the unit tests on the main application.

2Well compiled means that a source file is not be compiled twice or at least it is not be compiled with different preprocessor options.


Strategies

6.2.4 Automatic Tests

Squish Coco allows us to annotate the execution report with the execution name and state. This does not requireany specific library—the information can be inserted directly by editing the execution report before and after theexecution of a test item. This makes it possible to integrate the report into an automatic testing framework or for usewith a test executing script. (see Chapter 30.1, page 172)


Part IV CoverageScannerReference


Introduction

Chapter

7Introduction

CoverageScanner is a C++ program that—in effect—replaces the usual compiler. It preprocesses the source codeusing the native preprocessor program, in the course of which it inserts instrumentation code, and at the end itcompiles and links the project file, just the same as for a normal build.

CoverageScanner’s instrumentation consists of:

1. declaring a static table that contains the of instrumentation results.

2. generating a library which produces an execution report upon program termination, a UNIX® signal or aMicrosoft® Windows event.

3. adding the instrumentation code itself for each relevant C and C++ statement or Boolean expression.

The compiler, linker and preprocessor used are from the native environments (gcc, Visual Studio, etc.). These tools areused transparently and the developer has only to prepend ’cs’ to the name of the compiler executable to activate thecode coverage1. For example, using ’csgcc’ instruments the source code and uses the ’gcc’ compiler to generateobjects.

i CoverageScanner does only support precompiled headers with Microsoft® Visual Studio® .NET tool chain.CoverageScanner automatically deactivates this feature during compilation.

7.1 Invoking CoverageScanner

CoverageScanner works like a pre-processor which calls after instrumenting a source file the native compiler(Microsoft® cl.exe, GNU g++, . . . ).

Several invocation method are possible:

1. Using the dedicated CoverageScanner executable for a specific compiler: CoverageScanner executable for aspecific compiler is called to cs+’native compiler name’ or alternatively ’native compiler name’+-cs. (example:cl.exe is the Microsoft® C++ compiler and the corresponding CoverageScanner executable is cscl.exe orcl-cs.exe).

1 For Microsoft® Visual Studio®, an additional wrapper is provided which directly uses the Microsoft® ’cl’ and ’link’ (see Chapter 34.10,page 203)


Invoking CoverageScanner

CoverageScanner aims to work then as the native compiler with the difference that the code get instrumented.

Example:

csg++ source.cpp -o application.exe

2. Using CoverageScanner compiler wrapper and activating the instrumentation using --cs-on in the commandline arguments:

Squish Coco provides also a replacement of the native compiler called wrapper which extends its command lineargument to support the code coverage feature. To use it it is necessary to add at the first position in the PATHvariable the path of CoverageScanner wrapper. The instrumentation get activated, when adding --cs-on tothe command line argument.

Example:

export PATH=/opt/SquishCoco/wrapper/bin:$PATHg++ --cs-on source.cpp -o application.exe

Or, without modifying globally the PATH variable:

PATH=/opt/SquishCoco/wrapper/bin:$PATH g++ --cs-on source.cpp -o application.exe

3. Using CoverageScanner compiler wrapper and activating the instrumentation through COVERAGESCANNER_ARGSenvironment variable:

It is also possible to set the command line arguments of the wrapper using the COVERAGESCANNER_ARGSenvironment variable (see Chapter 8.1, page 61). In this case it is no longer necessary to modify the commandline arguments.

Example:

export PATH=/opt/SquishCoco/wrapper/bin:$PATHexport COVERAGESCANNER_ARGS=’--cs-on’g++ source.cpp -o application.exe

Or, to instrument a complete project built using GNU make:

COVERAGESCANNER_ARGS=’--cs-on’ PATH=/opt/SquishCoco/wrapper/bin:$PATH make all

4. Using --cs-compiler command line option of CoverageScanner:

--cs-compiler permits to select which native compiler should be used. All other command line arguments,except those specific to CoverageScanner, are passed through to the native compiler.

Example:

coveragescanner --cs-compiler=g++ source.cpp -o application.exe

i The native compiler path needs to appear in the PATH variable, after the path to the compiler wrappers. This isnecessary for finding the location of the native compiler after insertion of the instrumentation code.

Compiler support is done with the help of a profile. It contains declarations which make it possible to adaptCoverageScanner to C, C++ and C# preprocessors, compilers and linkers.

For a description of the compilers that are currently supported see Chapter C, page 271. For a description of theprofile syntax see Chapter 12, page 90


Command Line Arguments

Chapter

8Command Line Arguments

Syntax:

cs〈compiler〉 [〈cs-options〉] 〈compiler arguments〉coveragescanner --cs-compiler=〈STRING〉 [〈cs-options〉] 〈compiler arguments〉coveragescanner --cs-profile=〈STRING〉 [〈cs-options〉] 〈compiler arguments〉

where

〈compiler〉:One of the compilers supported by Squish Coco. Calling cs〈compiler〉 results in a call of CoverageScanner asa wrapper of 〈compiler〉.

〈cs-options〉:One or more of the options that are described in the following sections.

〈compiler arguments〉:All the other options. They are copied to the command line of the compiler when it is called by Coverage-Scanner.

For the use of --cs-compiler and --cs-profile see Chapter 8.2.9, page 69.

i The command line arguments are prioritized by their order in the command line: the latest entered option hasa higher priority than the first one.Example: cscl --cs-include=foo.h --cs-exclude=foo.h will exclude the header file foo.h from thecoverage analysis.

8.1 Environment variables

CoverageScanner also reads options from the environment variable COVERAGESCANNER_ARGS. The options must beseparated by spaces or tab characters (‘ ’ or ‘\t’). If both command line arguments are present and the variableCOVERAGESCANNER_ARGS is set, the command line options take precedence over the options in the environmentvariable.

This variable should only be set temporarily, e.g. in a script, because otherwise it would influence all builds.


List of options

8.2 List of options

8.2.1 Coverage modes

For a description of the coverage modes, see Part III, page 41.

--cs-off:Disable all code coverage analysis. CoverageScanner then calls the compiler or linker unchanged.

--cs-on:Enable code coverage analysis.

This is the default if CoverageScanner is called directly or as a wrapper that starts with cs, like csgcc. If thewrapper has however the same name as the program (e.g. gcc), code coverage is by default switched off.

--cs-off=〈pattern〉:Disable all code coverage analysis when compiling a source with an absolute file name that matches thewildcard expression 〈pattern〉. CoverageScanner then calls the compiler or linker unchanged.

--cs-on=〈pattern〉:Enable all code coverage analysis when compiling a source with an absolute file name that matches the wildcardexpression 〈pattern〉.It is possible to have several --cs-on=〈pattern〉 and --cs-off=〈pattern〉 options. The last option that matchesa file decides whether code coverage analysis is enabled or disabled.

!If several sources are compiled with the same compiler command (as it occurs with C#), disablingthe coverage analysis for one source file with --cs-off=〈pattern〉 will disable the analysis forall others.

--cs-hit:Set the instrumentation mode to hit instrumentation. It then only matters whether an instrumentation point (aline, a condition or a block) was executed at least once, not how many times.

--cs-count:Set the instrumentation mode to count instrumentation. (Default)

With this mode, the number of times an instrumentation point is executed is counted.

--cs-statement-block:Enable statement block coverage.

--cs-decision:Enable statement block and decision coverage.

--cs-condition:Enable statement block, decision and condition coverage. (Default)

--cs-mcc:Enable Multiple condition coverage.

--cs-mcdc:Enable MC/DC coverage.

--cs-function:Enable function coverage. (Default)


List of options

--cs-no-function:Disable function coverage instrumentation.

--cs-line:Enable line coverage. (Default)

If this option is set, decision and condition coverage are automatically enabled too and full instrumentationmode (--cs-full-instrumentation) is switched on.

--cs-no-line:Disable line coverage. The other instrumentation modes that are automatically enabled by --cs-line are notdisabled.

8.2.2 Fine-tuning of the instrumentation

--cs-no-exceptions:Do not instrument the catch block of a try. . .catch statement.

--cs-no-assignments:Do not instrument Boolean expressions in assignments.

--cs-combine-switch-cases:With this option, decision and condition coverage do not distinguish between case labels that lead to the samecode. This means that in the following code, the cases i == 3 and i == 4 are not distinguished. If the optionis not set, the two cases are counted separately.

switch (i) {case 3:case 4:

i = 0;break;

...}

The option is valid for C++ and C#.

--cs-partial-instrumentation:Suppression of redundant conditions. Setting this option also disables line coverage.

The option influences the instrumentation of if statements without an else part, like

if (cond) {i += 1;

}

Without the option, an invisible else branch is created and its coverage is measured too. If the option isset, this is no longer done and the coverage of the case in which cond is false is no longer measured. Theinstrumented code then becomes slightly smaller, which may be necessary on small embedded systems.

--cs-full-instrumentation:No suppression of redundant conditions. (Enabled by default)

--cs-boost:Specific handling of Boost library constructs. Currently this only means that the code behind the macroBOOST_FOREACH is not instrumented. (Enabled by default.)


List of options

--cs-no-boost:No specific handling of Boost library constructs.

--cs-qt4:No instrumentation for class members generated by moc for the Qt4 and Qt5 libraries (Q_OBJECT, qt_metacall, qt_metacast, . . . ) . The code behind the macro Q_FOREACH stays also uninstrumented. Qt signals remaininstrumented. (Enabled by default.)

--cs-no-qt4:No specific handling of the Qt4 and Qt5 libraries.

--cs-qt3:No instrumentation for class members generated by moc for the Qt3 library (Q_OBJECT, className, tr, . . . ).Qt signals remain instrumented. (Enabled by default.)

--cs-no-qt3:No specific handling of the Qt3 library.

--cs-no-annotations:No extraction of annotations (comments and manual validations) from the source files.

--cs-vs2010-lambda:Instrument the expression in the return statement of a lambda function.

This brings a better support of lambda expression on Microsoft® Visual Studio® 2010. The option is only validfor C++11 compilers and Microsoft® Visual Studio® 2010. With a C compiler or an older C++ compiler, thisoption has no effect.

--cs-no-csharp-dynamic:If set, the C# dynamic type is not used in the instrumentation code generated by Squish Coco.

With the dynamic type, Squish Coco can better handle Boolean expressions that contain non-Boolean objectswith overloaded true and false functions. Dynamic types are however only available for Microsoft® VisualStudio® 2010 and later; for earlier C# versions, this option is needed.

--cs-csharp-dynamic:If set, the C# dynamic type is used in the instrumentation code generated by Squish Coco.

--cs-exit-function=〈function〉:Declare 〈function〉 as the name of an exit function.

An exit function is a command that ends the control flow in a function, like a return statement or the call ofthe exit() function, or a function that always throws an exception. Any code directly after such a statement isthen unreachable. With this option, it is possible to inform Squish Coco that there are user-defined functionsthat have the same effect as exit().

The option can be repeated; then several exit functions are declared.

--cs-no-line-directive:Remove all #line directives from the code generated by the preprocessor for C++and C#.

The preprocessor is called by the CoverageScanner before instrumenting a file; afterwards the original compileris called. On some systems, compiling a file this way causes problems that do not occur when the compiler iscalled directly. Removing the #line statements then sometimes helps.

--cs-counter-size=〈int〉:Size of the internal execution counters. Possible values are 1 (byte), 4 (32bit integer) or 8 (64bit integer). Thecounters are used inside the instrumented program to count the number of times an instrumentation point hasbeen executed. Large internal counters prevent counter overflows, smaller counters may be necessary on smallembedded systems to save space. The default value is 8.



http://doc.qt.io/qt-4.8/

http://doc.qt.io/qt-5

http://doc.qt.io/qt-4.8/

http://doc.qt.io/qt-5


http://doc.qt.digia.com/3.3/

http://doc.qt.digia.com/3.3/

List of options

--cs-mcc-max=〈size〉 | --cs-mcdc-max=〈size〉:Set the maximal size of the instrumentation tables that are used for Multiple Condition coverage or MC/DC.The default for 〈size〉 is 8000.

For each decision, a truth table is generated under Multiple Condition coverage and MC/DC. 〈size〉 is themaximal number of lines in the table. If a decision requires a truth table that is longer than 〈size〉, Squish Cocofalls back to condition coverage for that decision.

8.2.3 File and path inclusion

The following options allow to specify which files are included in the code coverage. The options apply to files onthe command line and to those that are specified on #include statements.

One can use wildcards and regular expressions. In contrast to the common usage, the ‘*’ in wildcard expressionsmatches also the path separation characters. The syntax of regular expressions is described in Section 8.4, page 71.

Options accumulate. If more than one of the file inclusion/exclusion option is present, the last one that matches agiven file determines whether it is included or excluded.

In the next options, name of the file in question is first converted to an absolute path.

--cs-include-file-abs-wildcard=〈pattern〉:Include a file in the coverage analysis if its absolute path matches the wildcard expression 〈pattern〉.

--cs-exclude-file-abs-wildcard=〈pattern〉:Exclude a file from the coverage analysis if its absolute path matches the wildcard expression 〈pattern〉.Example: --cs-exclude-file-abs-wildcard=*.h excludes all files with the extension .h.

--cs-include-file-abs-regex=〈pattern〉:Include a file in the coverage analysis if its absolute path matches the regular expression 〈pattern〉.

--cs-exclude-file-abs-regex=〈pattern〉:Exclude a file from the coverage analysis if its absolute path matches the regular expression 〈pattern〉.Example: --cs-exclude-file-abs-regex=^.*\.h$ excludes all files with the extension .h.

--cs-include-path=〈path〉:Include all files in the directory 〈path〉 and its subdirectories in the coverage analysis. Wildcards or regularexpressions cannot be used.

Example: --cs-include-path=c:\include includes all files located in c:\include.

--cs-exclude-path=〈path〉:Exclude all files in the directory 〈path〉 and its subdirectories from the coverage analysis. Wildcards or regularexpressions cannot be used.

Example: --cs-exclude-path=c:\include excludes all files located in c:\include and its subdirectories.

Examples for complex inclusion patterns:

• --cs-exclude-file-abs-wildcard=* --cs-include-file-abs-wildcard=*.cpp instruments only .cppfiles.

• --cs-exclude-file-abs-wildcard=* --cs-include-file-abs-wildcard=*.cpp--cs-include-file-abs-wildcard=*.cxx instruments only .cpp and .cxx files.

• --cs-exclude-file-abs-wildcard=*/3rdparty/* does not instrument any files from the directory3rdparty.


List of options

• --cs-exclude-file-abs-wildcard=*/3rdparty/* --cs-include-file-abs-wildcard=* instru-ments all files: The first filter rule has no effect.

Relative paths The following options refer to the name under which a file occurs in an #include statement or onthe command line.

File paths and wildcard expression are converted to a canonical form which does not contain ‘.’, ‘..’, or a duplicatedpath separation character. (A ‘..’ at the beginning is however not removed.) On Microsoft® Windows, characters arealso converted to lower case and every slash (‘/’) is replaced with a backslash (‘\’).

Regular expressions are untouched. It is e.g. necessary to ensure that they do not contain duplicated path separators.

--cs-include-file-wildcard=〈pattern〉:Include a file in the coverage analysis if its normalized path matches the wildcard expression 〈pattern〉.Example: --cs-include-file-wildcard=../include/*.h includes all files with the extension .h locatedin the directory ../include/ and its subdirectories.

--cs-exclude-file-wildcard=〈pattern〉:Exclude a file from the coverage analysis if its normalized path matches the wildcard expression 〈pattern〉.

--cs-include-file-regex=〈pattern〉:Include a file in the coverage analysis if its normalized path matches the regular expression 〈pattern〉.Example: --cs-include-file-regex=^\.\./include/[^/]*\.h$ includes all files with the extension .hlocated in ../include/.

--cs-exclude-file-regex=〈pattern〉:Exclude a file from the coverage analysis if its normalized path matches the regular expression 〈pattern〉.

Relative file names are computed relatively to the current working directory. For example, depending on the workingdirectory one has to use different parameters in order to inhibit the instrumentation of a file src/test.c: If one isoutside the directory src, one may use

$ csgcc src/test.c --cs-exclude-file-wildcard=src/test.c

but when switching to src, the parameters must change:

$ cd src$ csgcc test.c --cs-exclude-file-wildcard=test.c

!Setting inclusion/exclusion expressions on absolute paths should be preferred on complex build systems,since modern build system often change the current working directory. Working with absolute pathsmake it easier to set global rules for a project.

8.2.4 Function inclusion

--cs-include-function-wildcard=〈string〉:This command line option enables the user to include a function in the coverage analysis. The function name isspecified using a wildcard expression and must include for C++ code the class name and namespace.

Example: --cs-include-function-wildcard=C::* includes all members of the class C.


List of options

--cs-exclude-function-wildcard=〈string〉:This command line option enables the user to exclude a function from the coverage analysis. The functionname is specified using a wildcard expression and must include for C++ code the class name and namespace.

Example: --cs-exclude-function-wildcard=*::Get* excludes all members of any classes which startswith Get.

--cs-include-function-regex=〈string〉:This command line option enables the user to include a function in the coverage analysis. The function nameis specified using a regular expression and must include for C++ code the class name and namespace. (seechapter 8.4, page 71 for the syntax)

--cs-exclude-function-regex=〈string〉:This command line option enables the user to exclude a function from the coverage analysis. The functionname is specified using a regular expression and must include for C++ code the class name and namespace.(see chapter 8.4, page 71 for the syntax)

If more than one inclusion/exclusion option is present, the last one that matches a given function determines whetherit is included or excluded.

8.2.5 Debugging

--cs-verbose-source-lines=〈int〉:Maximum number of source code lines dumped in an error message.

--cs-verbose=〈string〉:Generate verbose debug messages. Possible options are:

build: During build, the instrumentation by CoverageScanner is traced. This includes e.g. the informationabout which files are instrumented and which are excluded from instrumentation.

api: When the instrumented program is run, the calls to the CoverageScanner library are traced.

By default, the output is written to stderr.

--cs-verbose-file=〈file〉:Define the file used for generating a CoverageScanner log file. This option is meaningless when --cs-verboseis not used.

--cs-keep-instrumentation-files:Do not delete temporary instrumentation files. They will be left under random names in /tmp on Unix and in%TEMP% on Windows.

--cs-warnings=〈string〉:Instrumentation warning behaviour:

none: warnings will not be displayed.

display: warnings will be displayed via stderr.

error: warnings will trigger a compilation error.

8.2.6 Execution report

--cs-output=〈string〉:Set the file name for the execution report. The extension .csexe is added automatically.


List of options

By default, 〈string〉 has the value “%F” and CoverageScanner uses the application name without the path asname for the .csexe file.

The value of 〈string〉 may be an absolute path, like “/my/dir/%F”. The directories in this path must alreadybe present when the instrumented program runs, since CoverageScanner does not create them by default.1

Otherwise, 〈string〉 is interpreted as a path relative to the working directory at the time the program starts.

〈string〉 may contain several format specifiers. The values of the following specifiers are computed at compiletime:

%T: Current local time as an integer. (Example: 163355 for 16h33 and 55s)%D: Current local date as an integer. (Example: 20140130 for January 30, 2014)%F: Application/library file name without path.%A: Absolute application/library file name.%B: Application/library file name without path and extension.%P: Application/library file path.%/: File path separator (on Unix a slash, ‘/’, and on Windows a backslash, ‘\’).

The values of the following format specifiers are computed at runtime:

%c: Current working directory. The value is is computed before the main() function is started. If thecurrent working directory changes, this path remains identical.

%p: Process identifier. If for any reason it cannot be computed, “%p” will be replaced with an emptystring.

%t: Current local time. If for any reason it cannot be computed, “%t” will be replaced with an emptystring.

%d: Current local date. If for any reason it cannot be computed, “%d” will be replaced with an emptystring.

--cs-lock-csexe:Create a lock file which lock the access of the .csexe file during its generation.

--cs-nolock-csexe:Inverse option to --cs-lock-csexe. The preprocessor output is transmitted to CoverageScanner through atemporary file.

--cs-dump-on-signal=〈signal〉:Generate an execution report when a specific signal was received. The signal can be specified by a numberfrom 1 to 63, or by the conventional signal name (SIGINT, SIGTERM, . . . ).

--cs-dump-on-event=〈string〉:Generate an execution report when a specific Microsoft® Windows event was received.

8.2.7 Command line modification

--cs-compiler-option=〈option〉:Add 〈option〉 to the end of the compiler command line when compiling the instrumented code. When codecoverage is disabled, nothing happens. The option can be repeated if more than one option is needed.

--cs-libgen=〈option〉:Put an additional command line option to the compiler call which generates the code coverage library. Theoption can be repeated if more than one option is needed.

The options are appended and become the content of $LIBGEN$, a variable that may occur in the replacementtext for the profile option COMPILER_CMD (see Chapter 12.4, page 92).

1This behaviour may change if the user provides their own output functions. (See e.g. chapters 9.1.11, page 77 and 9.2.9, page 80.)


List of options

--cs-library-after=〈option〉:Put the linker argument for the CoverageScanner library after a specified option.

If used in the form “--cs-library-after=-lmylib”, the linker parameter for the CoverageScanner librarywill be inserted directly after the option -lmylib.

--cs-library-at-start:Add the linker argument for the CoverageScanner library before all other command line options.

--cs-library-at-end:Add the linker argument for the CoverageScanner library after all other command line options.

8.2.8 Generated code

--cs-enable-coveragescanner-library-abs-wildcard=〈string〉:Enable the generation of the CoverageScanner library when the absolute file name of the generated binarymatches the given wildcard expression.

--cs-disable-coveragescanner-library-abs-wildcard=〈string〉:Disable the generation of the CoverageScanner library when the absolute file name of the generated binarymatches the given wildcard expression.

--cs-memory-pool=〈int〉:Generate a CoverageScanner library which uses a memory pool bytes instead of calling malloc()/free().The size of the pool is 〈int〉 bytes. This option is useful only for operating systems which do not supportdynamic memory allocation.

--cs-link-instrumentation-tables:Chain all instrumentation tables at link time and not at run time.

--cs-no-exit-handler-installation:Disable the automatic installation of the handler which saves the code coverage report. If this handler isnot installed, no executions report will be generated when the application exits or when a custom signal orWindows event is received. Call __coveragescanner_install() to install the handler manually.

--cs-minimum-api:With this command line option, a CoverageScanner API is generated that has fewer dependencies to externallibraries than the default API. This may be necessary for embedded systems or when compiling a C# applicationwith the command line switch /noconfig.

8.2.9 Miscellaneous

--cs-compiler=〈string〉:This command option enables the user to select a profile. This option does not normally need to be specified,due to the fact that the profile name is usually extracted from the command line name. (cscl.exe implicitlyselects the profile cl.cspro)

Example: --cs-compiler=cl selects the profile ’cl.cspro’.

--cs-profile=〈string〉:This option permits to set directly the path of CoverageScanner profile. Using this option if not compatiblewith --cs-compiler.

Example: --cs-profile=%SQUISHCOCO%\cl.cspro is equivalent to --cs-compiler=cl on Microsoft® Win-dows.


Instrumenting using preprocessor defines

--cs-architecture=〈string〉:Internal option to specify the target architecture. It sets the variable $ARCHITECTURE$ in the replacement textfor the profile parameter PREPROCESSOR_CMD (see Chapter 12.3, page 91). The target architecture is also usedto select alternate profile parameters (see Chapter 12, page 90).

--cs-parallel-build=〈int〉:Maximum number of concurrent code generation/compilation.

--cs-cpp-ext=〈list〉:List of file extensions corresponding to C++ files (ex: .cpp,.cxx)

--cs-c-ext=〈list〉:List of file extensions corresponding to C files (ex: .c)

--cs-native-toolchain=〈string〉:Adds a directory in the search path of the native compiler. This command can be repeated. In this case the firstone is the prefered location.

--cs-option-file=〈file〉:Specify command line options from a file. The file may contain all valid CoverageScanner command lineoptions and the command line options of the native tool. It must contain one argument per line. Leading andtrailing spaces are removed and empty lines are ignored, but no other input processing is done. The option canbe repeated.

--cs-no-cspch:If set, the files with the extension .cspch are not generated. This files are only used to enhance the compilationspeed when using precompiled headers.

--cs-no-abort-on-error:By default, any errors raised while instrumenting C/C++ source code will stop the compilation. This behaviourcan be overridden with the option --cs-no-abort-on-error. If it is set, the compiler wrapper will displayan error message if a file cannot be instrumented, but then compile it again without instrumentation.

8.3 Instrumenting using preprocessor defines

CoverageScanner activates also the instrumentation if some defines are present in the command line option. Thispermits to instrument code with an IDE which does not permits to add custom command line options to the compileror linker.Example: Defining COVERAGESCANNER_COVERAGE_ON is equivalent of adding --cs-on to the command line.


Regular Expressions

Define EquivalenceCOVERAGESCANNER_RUNTIME_LOG --cs-verbose=api

COVERAGESCANNER_DUMP_ON_EVENT --cs-dump-on-event=Global\COVERAGE

COVERAGESCANNER_COVERAGE_ON --cs-on

COVERAGESCANNER_COVERAGE_HIT --cs-hit

COVERAGESCANNER_COVERAGE_COUNT --cs-count

COVERAGESCANNER_COVERAGE_BASIC_BLOCK --cs-statement-block

COVERAGESCANNER_COVERAGE_DECISION --cs-decision

COVERAGESCANNER_COVERAGE_CONDITION --cs-condition

COVERAGESCANNER_COVERAGE_MCC --cs-mcc

COVERAGESCANNER_NO_CSHARP_DYNAMIC --cs-no-csharp-dynamic

COVERAGESCANNER_NO_LINE_DIRECTIVE --cs-no-line-directive

COVERAGESCANNER_CSHARP_DYNAMIC --cs-csharp-dynamic

COVERAGESCANNER_COVERAGE_LOCK_CSEXE --cs-lock-csexe

COVERAGESCANNER_COVERAGE_NOLOCK_CSEXE --cs-nolock-csexe

COVERAGESCANNER_COVERAGE_PARTIAL_INSTRUMENTATION --cs-partial-instrumentation

COVERAGESCANNER_COVERAGE_FULL_INSTRUMENTATION --cs-full-instrumentation

COVERAGESCANNER_COVERAGE_NO_EXCEPTIONS --cs-no-exceptions

COVERAGESCANNER_COVERAGE_NO_ASSIGNMENTS --cs-no-assignments

COVERAGESCANNER_MINIMUM_API --cs-minimum-api

COVERAGESCANNER_COVERAGE_COMBINE_SWITCH_CASES --cs-combine-switch-cases

8.4 Regular Expressions

Regular expressions are composed of items (see table 8.1, page 72 for the full list) and combining operators. Thesecan be:

• [. . .]: The brackets are used to define a set of characters. For example, [123] corresponds to the characters 1,2 or 3. It is possible to define a range using the minus sign: [a-z] corresponds to all characters between a andz.

• [^. . .]: The caret defines a set of characters which should not match. For example, [^0-9] corresponds to anycharacter except a digit.

• ?: The question mark after a regular expression item makes it optional.Example: ab? matches the string a or ab.

• +: The plus operator makes it possible to repeat the regular expression item.Example: ab+ matches every strings ab. . .b.

• *: The star operator after a regular expression item signifies that the item is optional and can be repeated.Example: ab* matches every string ab. . .b and a.

• |: The pipe operator specifies an alternative.Example: a|b matches the character a or b.

• (. . .): The parenthesis groups expressions into sub-expressions.Example: (true)|(false) matches the string true or false.

• Regular Expression Items: The regular expression items are ASCII or magic characters (See table 8.1, page 72for the complete list). If a magic character needs to be used as an ASCII character, it has to be escaped with abackslash (\).


Regular Expressions

Element Description Equivalence Reverse Element^ The caret marks the beginning of the string. For example, ^c:

will only match file names with an absolute path on the driveC:. If you wish to match a literal ^ you must escape it bywriting \^.

$ The dollar marks the end of the string. For example \.cpp$will match every C++ file name. If you wish to match a literal$ you must escape it by writing \$.

\< Match the start of a word.\> Match the end of a word.. Any character\ Escape character. Example: \. corresponds to the character

’.’ and not any character\m Alphanumeric character [0-9a-zA-Z] \M

\a Alphabetic character [a-zA-Z] \A

\b Blank character [ t] \B

\c Control character \C

\d Digit [0-9] \D

\g Printable character except space [] \G

\l Lower-case character [a-z] \L

\p Printable character \P

\n Printable character except space or alphanumeric character \N

\s White-space characters (space, form-feed , newline , carriagereturn , horizontal tab and vertical tab)

[ fnrtv] \S

\u Uppercase letter [A-Z] \U

\x Hexadecimal digits [0-9a-fA-F] \X

\w Alphanumeric character or underscore [0-9a-zA-Z_] \W

Table 8.1: Non ASCII Regular Expression Items


Library calls

Chapter

9Library calls

9.1 C and C++ Library

The CoverageScanner library is generated during instrumentation and enables the user

1. to initialize the measurement system and select the name of the .csexe file,

2. to install a handler which saves the execution report on exit,

3. in a unit test scenario, to pass information about the tests to the execution report, especially the name of thetest, comments and the test result,

4. to handle dynamically loaded libraries, and

5. to customize the output system for special needs.

The library functions are accessible from any instrumented file, without the need for an #include directive. When afile is instrumented, a preprocessor symbol __COVERAGESCANNER__ is defined: It can be used to exclude calls to thelibrary functions from the compilation when instrumentation is switched off.

9.1.1 __coveragescanner_install()

Syntax:

void __coveragescanner_install(const char *appname)

i This function is not needed for gcc, clang or Microsoft® Visual Studio®.

Initialize the CoverageScanner library. It should be the first function call in the main() function. The parameterappname contains the name of the executable. It is used to set the name of the measurement file to 〈appname〉.csmes.

__coveragescanner_install() registers an exit handler via atexit() in order to save the execution reportwhenever the application terminates. The handler calls the function __coveragescanner_save() to store theexecution traces at the end of the program. It is called when the program terminates regularly and also if the signalsSIGABRT, SIGTERM, SIGFPE, SIGILL, SIGINT, SIGSEGV and SIGTERM are received.


C and C++ Library

Suggested usage:

int main(argc,char *argv[]){#ifdef __COVERAGESCANNER____coveragescanner_install(argv[0]);

#endif...

}

Platform dependencies With gcc, clang or Microsoft® Visual Studio®, CoverageScanner automatically installsa minimal handler. It saves the execution in the file 〈program_name〉.csmes on a normal exit of the application.Calling __coveragescanner_install() allows to set the file name of the execution report and also saves the reportat an abnormal exit, like a crash or an interrupt by a signal.

__coveragescanner_install() is not available for a compiler if its profile parameter CUSTOM_SETUP is set toNONE (see Chapter 12, page 90).

9.1.2 __coveragescanner_testname()

Syntax:

void __coveragescanner_testname(const char *name)

Set the name of the current test. It will be saved in the execution report and be displayed in the "Executions"window (see Chapter 18.1, page 121) when the .csmes file is loaded into CoverageBrowser.

All code that is executed until the end of the program or the next call of __coveragescanner_save() is consideredpart of the test name. If tests get the same name, CoverageBrowser disambiguates them by numbers, like “MyTest(2)”.

Test names may be hierarchical, with hierarchy levels separated by slashes: If a program defines two tests, “group/test1” and “group/test2”, CoverageBrowser displays them as two tests, “test1” and “test2” under a commonheading, “group”.

i This and the following functions until __coveragescanner_save() are especially intended for the use withunit test frameworks (see Chapter 30, page 172).

9.1.3 __coveragescanner_teststate()

Syntax:

void __coveragescanner_teststate(const char *state)

Set the state of the current test. The parameter state may have the following values:

"PASSED":to indicate that the test was successful.

"FAILED":to indicate that the test was not successful.


C and C++ Library

"CHECK_MANUALLY":to indicate that it was not possible to determine whether the test was successful.

If the function is called twice, the first state is overwritten. The test state will be saved in the execution reportand be displayed in the "Executions" window (see Chapter 18.1, page 121) when the .csmes file is loaded intoCoverageBrowser.

9.1.4 __coveragescanner_add_html_comment()

Syntax:

void __coveragescanner_add_html_comment(const char *comment)

Set am HTML comment to the current execution. There is only one HTML comment active at any time, but it canbe put together with several calls of __coveragescanner_add_html_comment(). In this case, the values of thevarious comment parameters are concatenated.

The full comment must follow the HTML syntax, but only the content of the <body> tag is used. A minimal commendmight therefore be something like “<html><body>My comment</body></html>”.

9.1.5 __coveragescanner_clear_html_comment()

Syntax:

void __coveragescanner_clear_html_comment()

Remove the comment added by calls of the function __coveragescanner_add_html_comment().

9.1.6 __coveragescanner_save()

Syntax:

void __coveragescanner_save()

Save the execution report and reset the status of all instrumentations. The coverage counters for each line of code andthe execution status of the current test are reset, but the HTML comment stays unchanged.

9.1.7 __coveragescanner_clear()

Syntax:

void __coveragescanner_clear()

Reset the coverage counters for all lines of code.

This is useful in a unit testing framework. After the framework has started, __coveragescanner_reset() can becalled. Then the activity of the framework at startup does not become part of the code coverage.


C and C++ Library

9.1.8 __coveragescanner_filename()

Syntax:

void __coveragescanner_filename(const char *name)

Set the file name of the execution report. The extension ’.csexe’ is added automatically. If name is NULL, thegeneration of execution reports is disabled.

name may contain the following escape sequences:

%p: The process identifier, if it is provided by the platform, otherwise the empty string.

%w: The working directory. The value is computed when the CoverageBrowser library is initialized; it does notchange afterwards.

%t: The current time, or the empty string if it cannot be computed.

%d: The current date, or the empty string if it cannot be computed.

Platform dependencies The escape sequences are only available for a compiler if its profile parameter FILE_FORMAT_SPECIFIERis set to YES (see Chapter 12, page 90).

Date and time may not be available on some embedded platforms.

9.1.9 __coveragescanner_register_library()

Syntax:

int __coveragescanner_register_library(const char *library_name)

Register a shared library that was loaded on demand. The function must be called after a library has been loadedwith dlopen() (UNIX®) or LoadLibrary() (Microsoft® Windows). After it is registered, the code coverage of thelibrary is saved in the same coverage report file as the coverage of the main application.

It is possible to call __coveragescanner_register_library() more than once, but every call must be matchedby a corresponding call of __coveragescanner_unregister_library(). For each library there is a use counterto keep track of the registrations.

Returned values:

0: Success.

-1: Error: library could not be loaded.

-2: Error: invalid library name.

1: Success, but the library is already registered. The use count of the library has been incremented.

2: Success, but the library is not instrumented.

Platform dependencies __coveragescanner_register_library() and __coveragescanner_unregister_library()are not available for a compiler if its profile parameter PLUGIN_REGISTRATION_API is set to NO (see Chapter 12,page 90).


C and C++ Library

9.1.10 __coveragescanner_unregister_library()

Syntax:

int __coveragescanner_unregister_library(const char *library_name)

De-register a shared library that was loaded on demand. The function must be called before a library is unloaded withdlclose() (UNIX®) or CloseHandle() (Microsoft® Windows).

Returned values:

0: Success.

-1: Error: library was not registered.

-2: Error: invalid library name.

1: Success, but the library was not unregistered because its use count was not null.

2: Success, but the library is not instrumented.

9.1.11 __coveragescanner_set_custom_io()

Syntax:

void __coveragescanner_set_custom_io(char *(*csfgets)(char *s, int size, void *stream),int (*csfputs)(const char *s, void *stream),void *(*csfopenappend)(const char *path),void *(*csfopenread)(const char *path),void *(*csfopenwrite)(const char *path),int (*csfclose)(void *fp),int (*csremove)(const char *filename) )

Define your own set of I/O functions.

Parameters:

csfgets:Read at most size - 1 characters from stream and store them into the buffer pointed to by s. Reading stopsafter an EOF or a newline is read. If a newline is read, it is stored in the buffer. A ‘\0’ is stored after the lastcharacter in the buffer.

csfputs:Write the string s to stream, without its trailing ’\0’.

csfopenappend:Open the file path for appending text at the end.

csfopenread:Open the file path for reading.

csfopenwrite:Open the file path for writing.

csfclose:Flush and close the stream pointed to by fp.


C# Library

csremove:Remove the file filename from the file system.

By default, CoverageScanner writes the execution report (the .csexe file) to the file systems with help of commonC library functions. In some situations, e.g. with embedded systems, it is necessary to use other methods. Forthis reason, CoverageScanner provides __coveragescanner_set_custom_io(), which allows to replace the I/Ofunctions used by __coveragescanner_save().

Default implementation:

char *csfgets(char *s, int size, void *stream) { return fgets(s,size,(FILE *)stream);}

int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); }void *csfopenappend(const char *path) { return (void*)fopen(path,"a+"); }void *csfopenread(const char *path) { return (void*)fopen(path,"r"); }void *csfopenwrite(const char *path) { return (void*)fopen(path,"w"); }int csremove(const char *filename) { return remove(filename); }int csfclose(void *fp) { return fclose((FILE*)fp); }

Some examples are available in Appendix 32, page 178.

9.2 C# Library

CoverageScanner library is generated during the link process and enables the user to:

1. Generate the execution report.

2. Give a name to the executed tests.

3. Install a signal handler which saved the execution report on exit.

Additionally CoverageScanner sets the preprocessor define __COVERAGESCANNER__ which excludes this functionfrom a normal compilation.

9.2.1 CoverageScanner.__coveragescanner_init()

Syntax:

void CoverageScanner.__coveragescanner_init()

CoverageScanner.__coveragescanner_init() initialize explicitly the CoverageScanner library. Calling thisfunction is only necessary on C# DLL which are not calling the static module initializer automatically.

9.2.2 CoverageScanner.__coveragescanner_testname()

Syntax:

void CoverageScanner.__coveragescanner_testname(string name)


C# Library

CoverageScanner.__coveragescanner_testname() sets the name of the test which is currently being executed.It will be saved to the execution report and is displayed in the "Executions" window (see Chapter 18.1, page 121)when the loaded in CoverageBrowser.

9.2.3 CoverageScanner.__coveragescanner_teststate()

Syntax:

void CoverageScanner.__coveragescanner_teststate(string state)

CoverageScanner.__coveragescanner_teststate() sets the state of the test which is currently being executed.The string parameter state can have the following values:

"PASSED":indicates that the test was successfully executed.

"FAILED":indicates that the test was not successfully passed.

"CHECK_MANUALLY":indicates that it was not possible to determinate if the test was successfully executed.

It will be saved to the execution report and is displayed in the "Executions" window (see Chapter 18.1, page 121)when the loaded in CoverageBrowser.

9.2.4 CoverageScanner.__coveragescanner_add_html_comment()

Syntax:

void __coveragescanner_add_html_comment(string comment)

__coveragescanner_add_html_comment() append a comment to the current execution. The comments needs tofollow the HTML syntax but only the body is parsed. (The contents of the “<HEAD>” tag is ignored). This functioncan be called several times, in this case the content is appended.

9.2.5 CoverageScanner.__coveragescanner_clear_html_comment()

Syntax:

void __coveragescanner_clear_html_comment()

__coveragescanner_add_html_comment() clear comments added using void __coveragescanner_add_html_comment(string comment).

9.2.6 CoverageScanner.__coveragescanner_save()

Syntax:

void CoverageScanner.__coveragescanner_save()


C# Library

CoverageScanner.__coveragescanner_save() saves the execution report and resets the status of all instrumen-tations.

9.2.7 CoverageScanner.__coveragescanner_clear()

Syntax:

void CoverageScanner.__coveragescanner_clear()

CoverageScanner.__coveragescanner_clear() resets the status of all instrumentations.

9.2.8 CoverageScanner.__coveragescanner_filename()

Syntax:

void CoverageScanner.__coveragescanner_filename(string name)

this function sets the file name of the execution report. The extension ’.csexe’ is added automatically.

i Setting the execution report file name is necessary if the initialization function__coveragescanner_install() is not used.

9.2.9 CoverageScanner.__coveragescanner_set_custom_io()

Syntax:

void CoverageScanner.__coveragescanner_set_custom_io(__cs_fgets_delegate cs_fgets,__cs_fputs_delegate cs_fputs,__cs_fopenappend_delegate cs_fopenappend,__cs_fopenread_delegate cs_fopenread,__cs_fopenwrite_delegate cs_fopenwrite,__cs_fclose_delegate cs_fclose,__cs_remove_delegate cs_remove)

Parameters:

csfgets:Type:public delegate string __cs_fgets_delegate(System.IO.Stream stream)csfgets reads in at most one less than size characters from stream and stores them into the buffer pointed toby s. Reading stops after an EOF or a newline. If a newline is read, it is stored into the buffer. A ’\0’ is storedafter the last character in the buffer.

csfputs:Type:public delegate void __cs_fputs_delegate(string s, System.IO.Stream stream)csfputs writes the string s to stream, without its trailing ’\0’.

csfopenappend:Type:public delegate System.IO.Stream __cs_fopenappend_delegate(string path)csfopenappend open the file path for writing at the end.


C# Library

csfopenread:Type:public delegate System.IO.Stream __cs_fopenread_delegate(string path)csfopenread open the file path for reading.

csfopenwrite:Type:public delegate System.IO.Stream __cs_fopenwrite_delegate(string path)csfopenwrite open the file path for writing.

csfclose:Type:public delegate void __cs_fclose_delegate(System.IO.Stream fp)The csfclose function will flush and close the stream pointed to by fp.

csremove:Type:public delegate void __cs_remove_delegate(string n)csremove removes filename from the file system.

CoverageScanner writes the execution report (.csexe file) into file using the common C# file IO functions. In somesituations, for embedded systems for example, it is necessary to use an other way for recording it. For this reason,CoverageScanner provides CoverageScanner.__coveragescanner_set_custom_io() which permits to replacethe IO functions used by CoverageScanner.__coveragescanner_save().


Controlling the instrumentation during the compilation

Chapter

10Controlling the instrumentation during thecompilation

CoverageScanner offers the possibility to modify the coverage settings of some source code parts during thecompilation. This permits, for example, to exclude some source code from the coverage analysis.

10.1 Source Code Annotations

Source code annotations are specific comments which permits to validate some source code parts. Two kind ofannotations are existing:

Comments:comments are simple code comments which can be used for giving advices to the tester. They do not have anyimpact on the code coverage

Manual Validations:Manual validations permits to indicate that some lines of code should be considered as validated. Manualvalidations have the same effect for the code coverage as executing the source lines.

Source code annotations can directly be edited with CoverageBrowser or generated by writing specific commentsinto the source code.

10.1.1 Comments

To generate comments for a single source line it is only necessary to start a C or C++ comment with the keyword“coco:〈comment〉”. This will insert a comment for all instrumentations in the current line.

Example: In the following example, the comment “To test the failure, change the current working directory to aread-only directory” will be added to the instrumentations of the line 4.

1 void write_log( const char *text )2 {3 FILE * f = fopen( "log.txt", "a+" );4 if ( f ) /* coco: To test the failure,5 change the current working


Source Code Annotations

6 directory to a read-only directory */7 {8 fputs( text, f ):9 fclose( f );

10 }11 }

Comments can also be generated for multiples source lines using the keywords “coco begin:〈comment〉” and “cocoend”.

Example: In the following example, the comment “To test the failure, change the current working directory to aread-only directory” will be added to the instrumentations of the lines of the function write_log().

1 void write_log( const char *text )2 {3 /* coco begin: To test the failure,4 change the current working5 directory to a read-only directory */6 FILE * f = fopen( "log.txt", "a+" );7 if ( f )8 {9 fputs( text, f ):

10 fclose( f );11 }12 /* coco end */13 }

10.1.2 Manual Validation

Manual validated instrumentation can be generated using a comment starting with the keyword “coco validated:〈comment〉”.For validated a code area, “coco begin validated:〈comment〉” and “coco end” keywords can be used. In or-der to be compatible with PureCov adjustments, CoverageScanner also recognizes “purecov:” line commentswith the keywords “inspected” or “tested”. (alternatively “begin inspected”/“begin tested” and “endinspected”/“end tested” for multi-line annotations)

Example: In the following example, the comment “This function does not need to be tested, only used for debugging.”will be added to the instrumentations of the function write_log().

1 void write_log( const char *text )2 {3 // coco begin validated: This function does not need to be tested, only used for

debugging.4 FILE * f = fopen( "log.txt", "a+" );5 if ( f )6 {7 fputs( text, f ):8 fclose( f );9 }

10 // coco end11 }


C and C++ Pragma

10.2 C and C++ Pragma

CoverageScanner defines C and C++ pragmas which permit to control during the compilation the generation of theinstrumentation. All control commands are handled in a stack. The pragmas have the following syntax:#pragma CoverageScanner ( string )or: __pragma ( CoverageScanner ( string ) )or: _Pragma ( CoverageScanner ( string ) )

List of supported pragmas:

#pragma CoverageScanner(cov-on) :Enable code coverage instrumentation.

#pragma CoverageScanner(cov-off) :Disable code coverage instrumentation.

#pragma CoverageScanner(cov-hit) :Instruments using code coverage hit.

#pragma CoverageScanner(cov-count) :Instruments using code coverage count.

#pragma CoverageScanner(cov-statement-block) :Select statement block coverage as instrumentation method.

#pragma CoverageScanner(cov-decision) :Select decision coverage as instrumentation method.

#pragma CoverageScanner(cov-condition) :Select condition coverage as instrumentation method.

#pragma CoverageScanner(cov-partial-instrumentation) :Partial instrumentation of conditions and decisions.

#pragma CoverageScanner(cov-full-instrumentation) :Full instrumentation of conditions and decisions.

#pragma CoverageScanner(cov-assignment-on) :Instrument Boolean expressions in assignments.

#pragma CoverageScanner(cov-assignment-off) :Do not instrument Boolean expressions in assignments.

#pragma CoverageScanner(cov-switch-case-off) :Do not instrument all cases of a switch statement.

#pragma CoverageScanner(cov-switch-case-on) :Instrument all cases in a switch statement.

#pragma CoverageScanner(pop) :Restore the instrumentation options from before the call of the latest CoverageScanner pragma.


C# regions

10.3 C# regions

CoverageScanner defines C# extensions of the #region keyword which permits to control during the compilationthe generation of the instrumentation. The #region extension have the following syntax:#region CoverageScanner ( string )

The keyword #endregion restores the generation options.

List of supported regions:

#region CoverageScanner(cov-on) :Enable code coverage instrumentation.

#region CoverageScanner(cov-off) :Disable code coverage instrumentation.

#region CoverageScanner(cov-hit) :Instrument using code coverage hit.

#region CoverageScanner(cov-count) :Instrument using code coverage count.

#region CoverageScanner(cov-statement-block) :Select statement block coverage as instrumentation method.

#region CoverageScanner(cov-decision) :Select decision coverage as instrumentation method.

#region CoverageScanner(cov-condition) :Select condition coverage as instrumentation method.

#region CoverageScanner(cov-partial-instrumentation) :Partial instrumentation of conditions and decisions.

#region CoverageScanner(cov-full-instrumentation) :Full instrumentation of conditions and decisions.

#region CoverageScanner(cov-assignment-on) :Instrument Boolean expressions in assignments.

#region CoverageScanner(cov-assignment-off) :Do not instrument Boolean expressions in assignments.

#region CoverageScanner(cov-switch-case-off) :Do not instrument all cases in a switch statement.

#region CoverageScanner(cov-switch-case-on) :Instrument all cases in a switch statement.


Code Coverage of Libraries

Chapter

11Code Coverage of Libraries

11.1 Code Coverage of Static/Shared Libraries and DLL

During the linking operation, CoverageScanner includes all instrumentations of the shared libraries (if these arecompiled with CoverageScanner). CoverageBrowser displays the code coverage of the complete application(executable and its libraries) in one view.

i To get an analysis of the code coverage of a library only, it is necessary to compile the main application andexclude its sources from the code coverage (by adding in the command line --cs-exclude-file-regex=.*for example.)

11.2 Code Coverage of Plugins/Manually Loaded SharedLibraries

Libraries loaded dynamically can also be instrumented, but it is necessary to handle the generation of the executionreport in the main application or in the plugin code itself.

11.2.1 Generating Code Coverage Information directly from the MainApplication

Handling the plugins into the main application can be easily performed using the register/unregister mechanism ofthe CoverageScanner API: It is just necessary to call __coveragescanner_register_library() after loading alibrary and call __coveragescanner_unregister_library() just before unloading it.

Example:

#include <stdio.h>#include <stdlib.h>#include <dlfcn.h>


Code Coverage of Plugins/Manually Loaded Shared Libraries

intmain(int argc, char **argv){

void *handle;double (*cosine)(double);char *error;

handle = dlopen("libm.so", RTLD_LAZY);#ifdef __COVERAGESCANNER__

__coveragescanner_register_library("libm.so");#endif

if (!handle) {fprintf(stderr, "%s\n", dlerror());exit(EXIT_FAILURE);

}

dlerror(); /* Clear any existing error */

/* Writing: cosine = (double (*)(double)) dlsym(handle, "cos");would seem more natural, but the C99 standard leavescasting from "void *" to a function pointer undefined.The assignment used below is the POSIX.1-2003 (TechnicalCorrigendum 1) workaround; see the Rationale for thePOSIX specification of dlsym(). */

*(void **) (&cosine) = dlsym(handle, "cos");

if ((error = dlerror()) != NULL) {fprintf(stderr, "%s\n", error);exit(EXIT_FAILURE);

}

printf("%f\n", (*cosine)(2.0));#ifdef __COVERAGESCANNER__

__coveragescanner_unregister_library("libm.so");#endif

dlclose(handle);exit(EXIT_SUCCESS);

}

i Calling __coveragescanner_register_library() or __coveragescanner_unregister_library()on an non-instrumented library is allowed.

11.2.2 Generating Code Coverage Information directly from the Plugin

CoverageScanner cannot handle the instrumentation of plugins (i.e. shared libraries loaded manually) during thelinking phase. In this case, the library must initialize and store itself the executions.Therefore, the shared library needs to call __coveragescanner_filename() to set the name of the execution fileduring its initialization and __coveragescanner_save() to save the instrumentations when its becomes unloaded.



11.2.2.1 Code Coverage of Plugins Generated with Microsoft® Visual Studio®

The function DllMain is called on the initialization and the termination of the DLL generated using Microsoft® VisualStudio®. When the ’reason’ field is equal to DLL_PROCESS_ATTACH the function __coveragescanner_filename()should be called. To save the measures on exit the function __coveragescanner_save() shall be called when’reason’ is DLL_PROCESS_DETACH.

Example:

extern "C"BOOL WINAPI DllMain(HINSTANCE hInstance,

DWORD dwReason,LPVOID /*lpReserved*/)

{switch( dwReason ){case DLL_PROCESS_ATTACH:


/* Initialization of the CoverageScanner library. *//* Replace "mylib" with your filename without extension */__coveragescanner_filename("mylib");

#endif...

break;case DLL_PROCESS_DETACH:...


/* Saves the execution report */__coveragescanner_save();

#endifbreak;

}return TRUE;

}

11.2.2.2 Code Coverage of Plugins Generated with GNU gcc

The GNU compiler offers 2 attributes which permits to execute a function when a library becomes loaded or unloaded:

__attribute__ ((constructor)) my_init(void);:This attribute permits to call a function when the library is loaded. Call the function __coveragescanner_filename()in the custom initialization function of the library.

__attribute__ ((destructor)) my_fini(void);:This attribute permits to call a function when the library is unloaded. Call the function __coveragescanner_save()on the termination of the library.

Example:

static void plugin_load(void) __attribute__ ((constructor)) ;static void plugin_unload(void) __attribute__ ((destructor)) ;

static void plugin_load(void){




/* Initialization of the CoverageScanner library. *//* Replace "mylib" with your filename without extension */__coveragescanner_filename("mylib");

#endif...

}

static void plugin_unload(void){

...#ifdef __COVERAGESCANNER__

/* Saves the execution report */__coveragescanner_save();

#endif}


CoverageScanner Adaptation to a Tool Suite

Chapter

12CoverageScanner Adaptation to a Tool Suite

If Squish Coco should be used with a new kind of compiler or tool chain, a profile must be written to adapt theCoverageScanner to the compiler. This chapter describes the content of profiles; for the use of profiles with SquishCoco, see Chapter 7.1, page 59.1

The profile lets CoverageScanner:

• interpret the command line of the compiler (see Chapter 12.5, page 95).

• interpret the command line of the linker (see Chapter 12.4, page 92).

• customize the output of the coverage analysis2 (see Chapter 12.7, page 99).

The profile is a text file with a name of the form ’〈tool name〉.cspro’. It must be located in the Squish Cocoinstallation directory. A profile may contain:

Comments:Lines that start with a # character are treated as comments.

Global parameters:A profile parameter can be assigned a value with the following syntax:

〈parameter〉=〈value〉

Only one definition per line is allowed.

Parameters for a specific architecture:It is also possible to have a specific version of a parameter that is used only with certain architectures. For thisone puts the name of the architecture in brackets behind the parameter, in the following way:

〈parameter〉[〈architecture〉]=〈value〉

Such a definition must be placed after the global parameter definition.

The architecture is determined automatically, but it can also be specified with --cs-architecture.

1Note that some other commands that are not compilers, like ld or ar, are also wrapped with the help of a profile. This is because they, too, areinvolved in the compilation process.

2This can be necessary on embedded systems which do not have access to a file system.


Profile Parameters for Architecture Settings

12.1 Profile Parameters for Architecture Settings

ARCHITECTURE=〈string〉:List of command line arguments which specify the architecture used during the compilation. This parametershould be placed in the first lines of the profile.

ARCHITECTURE_APPEND=〈YES/NO/ALLOW〉:If set, the argument of the command line option, which defines the architecture, is directly appended (no spacecharacters are placed between the option and the arguments).

12.2 Profile Parameters for Instrumentation Settings

COVERAGESCANNER_DUMP_ON_EVENT=〈regular expression〉:Regular expression which activate the Windows event handler listening the global event Global\COVERAGE.

COVERAGESCANNER_RUNTIME_LOG=〈regular expression〉:Regular expression which activate the verbose mode of the CoverageScanner API.

COVERAGESCANNER_COVERAGE_ON=〈regular expression〉:Regular expression which activate the code coverage analysis (equivalent to --cs-on) if this expression matchon a command line argument.

SOURCE_IGNORE_INSTRUMENTATION=〈list〉:List of source file which are not instrumented. The list is a list of wildcard expressions which matches absolutefile paths. Environment variables can be access by placing the name between two dollars. (ex: $HOME$ toaccess to HOME environment variable)

12.3 Profile Parameters for Preprocessor Settings

PREPROCESSOR_HIDE_OPTION_NO_ARG=〈string〉:List of single options (without arguments) which should not be transmitted to the native preprocessor.

PREPROCESSOR_HIDE_OPTION_ONE_ARG=〈string〉:List of options with one argument which should not be transmitted to the native preprocessor.

PREPROCESSOR_HIDE_OPTION_ONE_ARG_APPEND=〈YES/NO/ALLOW〉:If set, the argument of the options to hide is directly appended (no space characters are placed between theoption and the arguments).

PREPROCESSOR_KEEP_OPTION_ONE_ARG=〈string〉:List of options with one argument which must be transmitted to the native preprocessor.

PREPROCESSOR_KEEP_OPTION_ONE_ARG_APPEND=〈YES/NO/ALLOW〉:If set, the argument of the options which must be transmitted is directly appended (no space characters areplaced between the option and the arguments).

PREPROCESSOR_DEFINE_OPTION=〈string〉:Preprocessor command line option which permits to enter C or C++ defines. (ex: -D)


Profile Parameters for Linker Settings

PREPROCESSOR_DEFINE_OPTION_APPEND=〈YES/NO/ALLOW〉:Allows appending the preprocessor defines just after its command line option string. (without spaces, ex: -DNDEBUG)

PREPROCESSOR_INCLUDE_OPTION=〈string〉:Preprocessor command line option which permits to include directly some source files. (ex: /FI)

PREPROCESSOR_INCLUDE_OPTION_APPEND=〈YES/NO/ALLOW〉:Allows appending the include file name defines just after its command line option string. (without spaces,ex: /FImyheader.h)

PREPROCESSOR_DEFINE_SEPARATOR_OPTION=〈string〉:Separator of the define list. For example, if set to ,, CoverageScanner will detect the defines DEF1 and DEF2when parsing the string DEF1,DEF2.

PREPROCESSOR_CMD=〈string〉:Preprocessor command line. CoverageScanner does only analyse preprocessed files. The command generatedthe C or C++ file used for the code coverage analysis.The following variables are available:

$TMP1$. . . $TMP9$: Temporary files names. The temporary files are removed on exit.

$OPTIONS$: Preprocessing options. The preprocessing option are extracted from the command line argumentsand contains the list of includes, defines, etc. . .

$TOOL$: Compiler command (TOOL variable).

$ARCHITECTURE$: Architecture detected or specified by --cs-architecture.

$PROFILE_PATH$: Path of the profile file.

$SOURCE$: Source file name.

$OUTPUT$: Preprocessor output file name. If this variable is present in the command line, it contains the nameof a temporary filename used as output for the preprocessor. If not present, it is assumed that thepreprocessor generate its output to stdout.

The preprocessor output is stdout. If this parameter is omitted in the configuration, the preprocessor step isskipped.

i The preprocessor command must set the define __COVERAGESCANNER__ to 1.

(ex: $TOOL$ $OPTIONS$ -D__COVERAGESCANNER__=1 -E $SOURCE$)

12.4 Profile Parameters for Linker Settings

LINKER_HIDE_OPTION_NO_ARG=〈string〉:List of single options (without arguments) which should not be transmitted to the native linker.

LINKER_HIDE_OPTION_ONE_ARG=〈string〉:List of options with one argument which should not be transmitted to the native linker.

LINKER_HIDE_OPTION_ONE_ARG_APPEND=〈YES/NO/ALLOW〉:If set, the argument of the options to hide is directly appended (no space characters are placed between theoption and the arguments).



OBJ_EXT=〈string〉:List of extension for object files separated with a semicolon. (ex: .obj)

DYN_LIB_EXT=〈string〉:List of extensions for dynamic libraries separated with a semicolon. (ex: .dll)

STATIC_LIB_EXT=〈string〉:List of extensions for static libraries separated with a semicolon. (ex: .lib)

UNIX_LIBRARY_NAME=〈YES/NO〉:The library name follows the Unix standard.

LINK_LIBRARY_OPTION=〈string〉:Link option which permits to select libraries (ex:-l).

LINK_LIBRARY_OPTION_APPEND=〈YES/NO/ALLOW〉:Append the library directly after the link option string (ex: -lfoo) if LINK_LIBRARY_OPTION_APPEND is set.

LINK_OUTPUT_OPTION=〈string〉:Linker output option (ex:-o)

LINK_OUTPUT_OPTION_APPEND=〈YES/NO/ALLOW〉:Append the linker output file directly after the option string (without spaces, ex: -ofoo.o) if LINK_OUTPUT_OPTION_APPENDis set.

LINK_OUTPUT_DEFAULT=〈string〉:Linker output default filename.The following variable is available:


$BASENAME_SRC$: Base name of the the first source file. The base name is the filename without extension.

(ex: $BASENAME_SRC$.exe)

LINK_ADDITIONAL_ARGUMENTS=〈string〉:Additional arguments passed to the linker when instrumenting the source code.

DEFAULT_LIB_PATH=〈string〉:List of directories which contains the shared libraries (ex: /usr/library;/usr/share/library).

DLL_OPTION=〈string〉:Linker command line option which permits to generate a shared library or a DLL (ex: /DLL).

DLL_OUTPUT_STATIC_LIB=〈string〉:Static library (associated with the DLL) output file name (ex: /IMPLIB:foo.lib).

DLL_OUTPUT_STATI_LIB_APPEND=〈YES/NO/ALLOW〉:Append the static library (associated with the DLL) output file directly after the option string (without spaces,ex: /IMPLIB:foo.lib) if DLL_OUTPUT_STATIC_LIB_APPEND is set.

STATIC_LIB_AS_DEFAULT=〈YES/NO〉:Instrumentation database name takes as base name the static library name associated with a DLL.

LIBRARY_PATH_OPTION=〈string〉:Link option which permits to add additional directories to the path of directories (ex:-L)

LIBRARY_PATH_OPTION_APPEND=〈YES/NO/ALLOW〉:Append the directory name directly after the option string (ex: -L/foo) if LIBRARY_PATH_OPTION_APPEND isset.



AR_COMMAND_FORMAT_OPTION=〈YES/NO〉:The GNU librarian (ar) has a specific command line syntax which is supported when this option is set.

STDIN_MRI_SCRIPT_OPTION=〈list〉:This option indicates that an MRI script is used to generate a library. The script is provided by the standardinput.

GENERATE_COVERAGESCANNER_LIBRARY=〈YES/NO〉:If set, the CoverageScanner library is generated during the linking operation. This should be disabled whengenerating static libraries.

SKIP_GENERATE_COVERAGESCANNER_LIBRARY_OPTION=〈string〉:Permits to skip the generation of CoverageScanner library if a specific option appears in the command line.

AUTODETECT_MS_RUNTIME=〈YES/NO〉:Detect automatically the runtime library used when generating an application using Microsoft® Visual Studio®.This option compiles the CoverageScanner library with /MT, /MD or /ML if --cs-libgen is not used to set it.If --cs-libgen set /MT, /MD or /ML, this option does not have any effect.

AUTODETECT_MS_RUNTIME_OPTION=〈list〉:List of command line option which selects the runtime library (for Microsoft® Visual Studio®: /MT;/MD;/ML;/MLd;/MTd;/MDd)

INSTRUMENTATION_TABLES_LINKED_DURING_RUNTIME=〈YES/NO〉:Instead of linking all instrumentation tables when calling the linker, link it during the runtime of the application.

PLUGIN_REGISTRATION_API=〈YES/NO〉:If set, __coveragescanner_register_library() and __coveragescanner_unregister_library() areprovided bythe CoverageScanner API to register instrumented plugins during the execution.

FILE_FORMAT_SPECIFIER=〈YES/NO〉:If set, file format specifier (see command line option --cs-output) are supported at the runtime.

INSTRUMENTATION_TABLES_OF_DLL_LINKED_DURING_RUNTIME=〈YES/NO〉:Instead of linking all instrumentation tables of DLL when calling the linker, link it during the runtime of theapplication.

COVERAGESCANNER_REGISTRATION_UID=〈YES/NO〉:Generate an unique symbol in each static and dynamic library to retrieve the code coverage counters.

INJECT_COVERAGESCANNER_LIBRARY_AT_END=〈YES/NO〉:If YES, the CoverageScanner library is always inserted at the end of the linker command line arguments.

GNU_LINKER_SCRIPT=〈YES/NO〉:Support of GNU linker scripts. CoverageScanner will then try to recognize a script by trying to parse it.

GNU_LINKER_SCRIPT_OPTION=〈string〉:Command line option which permits to specify a GNU linker script.

GNU_LINKER_SCRIPT_OPTION_APPEND=〈YES/NO/ALLOW〉:Append the gnu linker script directly after the linker script command line option option string (ex: -Tfoo.script)if GNU_LINKER_SCRIPT_OPTION_APPEND is set.

VS_DEF_FILE_OPTION=〈string〉:Command line option for specifying the module definition file (.def file).(ex: VS_DEF_FILE_OPTION=/DEF:)

VS_DEF_FILE_OPTION_APPEND=〈YES/NO/ALLOW〉:If set, the command line option for specifying the module definition file is directly appended to the file name(no space characters are placed between the option and the arguments).


Profile Parameters for Compiler Settings

12.5 Profile Parameters for Compiler Settings

COMPILER_ONLY=〈YES/NO〉:If true, the native tool is only a compiler which has no linker capabilities.

COMPILER_HIDE_OPTION_ONE_ARG=〈string〉:List of options with one argument which should not be transmitted to the native compiler.

COMPILER_HIDE_OPTION_ONE_ARG_APPEND=〈YES/NO/ALLOW〉:If set, the argument of the options to hide is directly appended (no space characters are placed between theoption and the arguments).

BYTECODE_EMULATOR=〈string〉:If set, the call of the native toolchain is made by calling a byte code intepreter.

Example: BYTECODE_EMULATOR=/math/to/mono.exe permits for force the call of mcs.exe in a specificMono emulator.

TOOL=〈string〉:Native compiler/linker command (example: gcc).The following variables are available:

$PROFILE$: Profile name without extension.

REMOVE_EXT=〈string〉:List of file extension, separated with a semicolon, of files to remove from the command line arguments.

IGNORE_EXT=〈string〉:List of file extension, separated with a semicolon, of files to ignore. Ignored files are not instrumented.

C#_EXT=〈string〉:List of extension for C# files separated with a semicolon. (ex: .cs)

C_EXT=〈string〉:List of extension for C files separated with a semicolon. (ex: .c)

CPP_EXT=〈string〉:List of extension for C++ files separated with a semicolon. (ex: .cpp;.cxx)

CPP_LANGUAGE_OPTION=〈string〉:List of command line options which are switching the compiler in C++ mode.

C_LANGUAGE_OPTION=〈string〉:List of command line options which are switching the compiler in C mode.

LANGUAGE_SELECTION_OPTION=〈string〉:List of command line options which permits to select the computing language (C, or C++).

LANGUAGE_SELECTION_OPTION_APPEND=〈YES/NO/ALLOW〉:Append the selected language directly after the option string.

LANGUAGE_SELECTION_OPTION_C_KEYWORD=〈string〉:List of arguments of the language selection command line option which force the switching in C mode.

LANGUAGE_SELECTION_OPTION_CPP_KEYWORD=〈string〉:List of arguments of the language selection command line option which force the switching in C++ mode.



COMPILER_CMD=〈string〉:Compiler command line.The compiler command line is used to generate the CoverageScanner library during the linking phase.The following variables are available:


$COMPILER$: Compiler command (COMPILER variable).

$SOURCE$: Source file name.

$DESTINATION$: Destination file name.

$LIBGEN$: Contents of the command line option --cs-libgen.

(ex: $COMPILER$ -c $SOURCE$ -o $DESTINATION$)

COMPILER_ADDITIONAL_ARGUMENTS=〈string〉:Additional arguments passed to the compiler when instrumenting the source code.

COMPILE_OPTION=〈string〉:Compile command line option. (ex: -c)

COMPILER_OUTPUT_OPTION=〈string〉:Compiler output option. (ex: -o)

COMPILER_OUTPUT_OPTION_APPEND=〈YES/NO/ALLOW〉:Append the compiler output directly after the option string (without spaces, ex: -ofoo.o) if COMPILER_OUTPUT_OPTION_APPENDis true.

COMPILER_OUTPUT_DEFAULT=〈string〉:Compiler output default filename.The following variable is available:


$BASENAME_SRC$: Base name of the first source file. The base name is the filename without extension.

(ex: $BASENAME_SRC$.obj)

CALLING_CONVENTION=〈string〉:CALLING_CONVENTION sets the calling convention (__stdcall, __cdecl or __fastcall) on Windows plat-form.

FUNCTION_ATTRIBUTE=〈string〉:FUNCTION_ATTRIBUTE sets additional compiler attribute to each function of the CoverageScanner library.

DESTRUCTOR_ATTRIBUTE=〈string〉:DESTRUCTOR_ATTRIBUTE sets additional compiler attribute which permits to the functions, which reset thecode coverage information when the application get shutdown or when a dynamic library get unloaded, to beautomatically called. This attribute is is only necessary when the instrumentation tables are registered duringthe runtime.

CONSTRUCTOR_ATTRIBUTE=〈string〉:CONSTRUCTOR_ATTRIBUTE sets additional compiler attribute which permits to the functions, which initializethe code coverage information when the application starts, to be automatically called. This attribute is is onlynecessary when the instrumentation tables are registered during the runtime.

DESTRUCTOR_PRAGMA=〈string〉:DESTRUCTOR_PRAGMA sets additional compiler pragma attribute which permits to the functions, which reset thecode coverage information when the application get shutdown or when a dynamic library get unloaded, to beautomatically called. The variable $SYMBOL$ is parsed and is replaced through the symbol on which thepragma is applyed. This attribute is is only necessary when the instrumentation tables are registered during theruntime.



CONSTRUCTOR_PRAGMA=〈string〉:CONSTRUCTOR_PRAGMA sets additional compiler attribute which permits to the functions, which initialize thecode coverage information when the application starts, to be automatically called. The variable $SYMBOL$ isparsed and is replaced through the symbol on which the pragma is applyed. This attribute is is only necessarywhen the instrumentation tables are registered during the runtime.

DLL_EXPORT=〈string〉:DLL_EXPORT contains the compiler attribute which permits to export a symbol from a DLL/shared library.

DLL_IMPORT=〈string〉:DLL_IMPORT contains the compiler attribute which permits to import a symbol from a DLL/shared library.

FORCE_DLL_EXPORT=〈YES/NO〉:FORCE_DLL_EXPORT permits to force the usage of DLL_EXPORT attribute during the linking phase instead ofDLL_IMPORT attribute.

FORCE_DLL_EXPORT_OPTION=〈string〉:FORCE_DLL_EXPORT_OPTION permits to force the usage of DLL_EXPORT attribute when a appears in thecommand line.

PARALLEL_COMPILATION_OPTION_APPEND=〈YES/NO/ALLOW〉:Indicates if the number of processors are directly appended to the command which enables the parallel build.

PARALLEL_COMPILATION_OPTION=〈list〉:Command line option which enables the parallel build. If this command line argument is followed by an integer,this value indicates the maximum number of parallel build to execute.

SYSTEM_INCLUDES=〈list〉:List of system include directories. (ex: /usr/include)

INCLUDE_PATH_OPTION=〈string〉:Compiler option which permits to add additional include directories (ex:-I)

INCLUDE_PATH_OPTION_APPEND=〈YES/NO/ALLOW〉:Append the include directory name directly after the option string (ex: -I/foo) if INCLUDE_PATH_OPTION_APPENDis set.

CUSTOM_MALLOC_INCLUDE=〈string〉:Additional include for a custom malloc()/free() function (ex: <stdlib.h>)

CUSTOM_MALLOC_FUNCTION=〈string〉:Custom body of a malloc() function. The body should return a void* and the parameter int size indicatethe number of bytes allocated. (ex: return malloc((size_t)size);)

CUSTOM_FREE_FUNCTION=〈string〉:Custom body of a free() function. The parameter void* ptr is the pointer to free. (ex: free(ptr);)

LOCALE_SYSTEM_FILE_ENCODING=〈YES/NO〉:If YES, the locale system configuration of text encoding is used for compiling a source file.

OBJECT_DIR_OPTION=〈string〉:Compiler option which permits to specify the directory on which the objects are stored. (ex: -object-dir=debug/)

OBJECT_DIR_OPTION_APPEND=〈YES/NO/ALLOW〉:Append the directory directly after the option string (without spaces, ex: -object-dir=foo)

WARNINGS_TO_DISABLE=〈list〉:List of compiler warnings to disable. Each item of the list will be used to generate a compiler pragma whichpermits to enable/disable a compilation warning.


Profile Parameters for Precompiled Headers Support

WARNING_DISABLE_PRAGMA=〈string〉:Preprocessor directive which permits to disable a compiler warning. The variable $WARNING$ will be parsedand contains the warning number which should be disabled and \n is expanded as a cariage return. (ex: #pragmawarning disable $WARNING$)

WARNING_ENABLE_PRAGMA=〈string〉:Preprocessor directive which permits to enable/restore a compiler warning. The variable $WARNING$ will beparsed and contains the warning number which should be enabled and \n is expanded as a cariage return. (ex:#pragma warning restore $WARNING$)

COVERAGESCANNER_CSHARP_DYNAMIC=〈string〉:Regular expression which enable the support of the C# dynamic type. (ex: ^/define:.*\<COVERAGESCANNER_CSHARP_DYNAMIC\>.*$)

COVERAGESCANNER_NO_CSHARP_DYNAMIC=〈string〉:Regular expression which disable the support of the C# dynamic type. (ex: ^/define:.*\<COVERAGESCANNER_NO_CSHARP_DYNAMIC\>.*$)

ACTIVATE_CSHARP_DYNAMIC=〈string〉:Command line argument which sets the define __COVERAGESCANNER_CSHARP_DYNAMIC__ to enable the C#dynamic support. (ex: /define:__COVERAGESCANNER_CSHARP_DYNAMIC__)

12.6 Profile Parameters for Precompiled Headers Support

PCH_EXT=〈string〉:List of extension for precompiled header files separated with a semicolon. (ex: .pch)

CREATE_PCH_OPTION_APPEND=〈YES/NO/ALLOW〉:Indicates if the PCH header generated is directly appended to the command line option.

CREATE_PCH_OPTION=〈list〉:List of command line options which are generating precompiled headers. (ex: /Yc)

USE_PCH_OPTION_APPEND=〈YES/NO/ALLOW〉:Indicates if the PCH header used is directly appended to the command line option.

USE_PCH_OPTION=〈list〉:List of command line options which permits to select a precompiled header. (ex: /Yu)

PCH_OUTPUT_OPTION=〈string〉:Command line option to specify the precompiled header output. (ex: /Fp)

PCH_OUTPUT_OPTION_APPEND=〈YES/NO/ALLOW〉:Append the precompiled header output directly after the option string (without spaces, ex: -Fpfoo.pch) ifPCH_OUTPUT_OPTION_APPEND is true.

PCH_OUTPUT_DEFAULT=〈string〉:Default precompiled header output file name.The following variable is available:


$BASENAME_SRC$: Base name of the first source file. The base name is the filename without extension.

(ex: $BASENAME_SRC$.pdb)


Profile Parameters for Custom IO

12.7 Profile Parameters for Custom IO

CSEXE_FOPEN=〈string〉:’fopen function used by the generation of the execution report file.

CSEXE_FCLOSE=〈string〉:’fclose function used by the generation of the execution report file.

CSEXE_FPUTS=〈string〉:fputs function used by the generation of the execution report file.

CUSTOM_SETUP=〈NONE/POSIX/MS/GNU/MSCE/C++〉: NONE: if set to NONE the function __coveragescanner_install()will not be provided by CoverageScannerThis parameter should be set to NONE on embedded systemswhich do not provide the functions signal() and atexit().The code coverage information is saved only when __coveragescanner_save() is called.

POSIX: CoverageScanner provides a function __coveragescanner_install(), which installs a handlerwhich saves the execution report when the application terminates. The handler is installed usingPOSIX functions (signal() and atexit()).If __coveragescanner_install() is not called, the code coverage information is only saved when__coveragescanner_save() is called.

GNU: This is similar to the POSIX setting except that, if __coveragescanner_install() is not called, adefault handler is used which saves the code coverage data in the file coverage.csmes on a normalapplication exit.If __coveragescanner_install() is not called, the code coverage information is not saved onabnormal exit (crash, software interruption, . . . ).

MS: Same as GNU but for Microsoft® Visual Studio® compiler.

C++: Install a handler written in C++ which saves the code coverage data in the file coverage.csmes uponnormal application exit. CoverageScanner provides a function __coveragescanner_install(),which only changes the name of the destination file.

MSCE: Same as C++ but for Microsoft® eMbedded Visual C++® compiler.

12.8 Miscellaneous Profile Parameters

HIDE_OPTION_NO_ARG=〈string〉:List of single options (without arguments) which should not be transmitted to the native tool (compiler orlinker).(ex: -Yd;/Yd for disabling the precompiled headears on Microsoft® Visual Studio®)

HIDE_OPTION_ONE_ARG=〈string〉:List of options with one argument which should not be transmitted to the native tool (compiler or linker).(ex: -Yl;-Yc;-Yu;-YX;/Yl;/Yc;/Yu;/YX for disabling the precompiled headears on Microsoft® Visual Studio®)

HIDE_OPTION_ONE_ARG_APPEND=〈YES/NO/ALLOW〉:If set, the argument of the options to hide is directly appended (no space characters are placed between theoption and the arguments).

PDB_EXT=〈string〉:List of extension for debug information files separated with a semicolon. (ex: .pdb)


Miscellaneous Profile Parameters

PDB_OUTPUT_OPTION=〈string〉:Command line option for specifying the debug information file.(ex: PDB_OUTPUT_OPTION=/Fd)

PDB_OUTPUT_OPTION_APPEND=〈YES/NO/ALLOW〉:If set, the command line option for specifying the debug information file is directly appended to the file name(no space characters are placed between the option and the arguments).

SBR_EXT=〈string〉:List of extension for symbol browser files separated with a semicolon. (ex: .sbr)

SBR_OUTPUT_OPTION=〈string〉:Command line option for specifying the symbol browser information file.(ex: SBR_OUTPUT_OPTION=/Fr;/FR)

SBR_OUTPUT_OPTION_APPEND=〈YES/NO/ALLOW〉:If set, the command line option for specifying the symbol browser information file is directly appended to thefile name (no space characters are placed between the option and the arguments).

USE_RESPONSE_FILE=〈YES/NO/ALLOW〉:Pack all compiler/linker command line option into a response file. This is generally necessary on Microsoft®

Windows due to the command line limitation length. The option ALLOW consists of using a response file onlyif a response file is used in the command line arguments of the native compiler. A response file is only used ifthe if the command line size reach a limit defined by MINIMUM_COMMAND_LINE_SIZE_FOR_RESPONSE_FILE.

MINIMUM_COMMAND_LINE_SIZE_FOR_RESPONSE_FILE=〈integer〉:If the command line arguments are lower that the specified size, no response file is used.

RESPONSE_FILE_EXT=〈string〉:List of file extensions for the response file.

RESPONSE_FILE_OPTION=〈list〉:Command line option for specifying the response file.(ex: RESPONSE_FILE_OPTION=@)

RESPONSE_FILE_OPTION_APPEND=〈YES/NO/ALLOW〉:If set, the command line option for specifying the response file is directly appended to the file name (no spacecharacters are placed between the option and the arguments).

CODE_STYLE=〈C++/C#〉:Language supported.

EXIT_FUNCTIONS=〈string〉:List of functions which causes an application exit.(ex: EXIT_FUNCTIONS=abort;exit)

ENABLE_COVERAGESCANNER_PER_DEFAULT=〈YES/NO〉:Enable code coverage analysis per default. This is equivalent to implicitly set the command line option--cs-on. This option has only an impact if the native executable has the same name as CoverageScannerwrapper.

DEACTIVATE_COVERAGESCANNER=〈YES/NO〉:Disable CoverageScanner globally if set to YES.

DEACTIVATE_COVERAGESCANNER_OPTION_NO_ARG=〈string〉:List of command line option which deactivates CoverageScanner.(ex: DEACTIVATE_COVERAGESCANNER_OPTION_NO_ARG=-M;-MM;-MD;-MMD)


Miscellaneous Profile Parameters

DEACTIVATE_COVERAGESCANNER_OPTION_ONE_ARG=〈string〉:List of command line option with one argument which deactivates CoverageScanner.

DEACTIVATE_COVERAGESCANNER_OPTION_ONE_ARG_APPEND=〈YES/NO/ALLOW〉:If set, the command line option which disables CoverageScanner is directly followed by an argument (no spacecharacters are placed between the option and the arguments).

COVERAGESCANNER_LIBRARY_OBJECT=〈string〉:Name of the CoverageScanner object containing its library code. On Unix, its default value is __cs_library.o.On Microsoft® Windows, its default value is __cs_library.obj.

EXIT_VALUE_ON_SUCCESS=〈list〉:List of exit value that are return by the native linker or compiler and which means that the command wassucessfull.

ERROR_FORMAT=〈string〉:Format of the errors. The following items are parsed:

%m: Error message.

%f: File name.

%l: Line.

%c: Column.

ERROR_FORMAT_LINE=〈string〉:Same as ERROR_FORMAT but used when no column information are available.

ADDITIONAL_ARGUMENT_SEPARATOR=〈string〉:Additional character used as separator for the command line arguments.(ex: ADDITIONAL_ARGUMENT_SEPARATOR=, for VisualDSP++ compiler)

CORRECT_FILENAME_IN_COMPILER_MESSAGE=〈YES/NO〉:If enabled, CoverageScanner replace in the compiler/linker messages the file name used for the instrumentationthrough the real source filename of the project.

SUPPRESS_APTH_ENTRY=〈YES/NO〉:If set to YES, CoverageScanner alter the PATH variable and removes its own location when calling the nativecompiler.

ATEXIT=〈string〉:Command line used to register the application exit handler which saves the execution report upon the applicationexit.

FORCE_ESCAPING_RULES_RESPONSE_FILE=〈NO/MS/POSIX〉:Rules for escaping command line arguments in response files:

POSIX: Escaping rules for UNIX® platforms

MS: Escaping rules for Microsoft® Windows.

NO: Native escaping rules for the current platform.

FILESYSTEM_8.3=〈YES/NO〉:If set, the extension of the execution report file name is .cse. If the application file name is application.exethe generated report will be application.cse in this case and not application.exe.csexe


Instrumentation Parameters

12.9 Instrumentation Parameters

RETURN_INSTRUMENTED_IN_EXPRESSION=〈YES/NO〉:If true, return statement are instrumented into the returned expression (ex:return inst[0]++, x;).


Part V QML CoverageReference


Qt QML coverage

Chapter

13Qt QML coverage

13.1 Overview

QML is a user interface markup language that is used in the Qt Framework. Squish Coco has an add-on that supportscode coverage for QML. The add-on is delivered separately and requires an additional license.

It consists of a replacement for a Qt shared library file. By default the new library acts the same way as the versionsupplied by Qt, unless QML coverage is activated by setting an environment variable.

i There is one exception from this rule. It concerns the QML disk cache, which interferes with correctinstrumentation when switched on. In Qt 5.8 and later, where a disk cache exists, it is therefore alwaysdeactivated in the replacement library, even when instrumentation is not enabled.

13.2 Installation

The coverage add-on consists of a replacement of the Qt shared library Qt5Qml and is delivered in a ZIP archive.There exist several versions of the add-on, each for a different combination of Qt version and platform. To install it,one needs to download the right version of the archive, unpack it and put its content in place of the original library.

It is useful to keep a backup version of the original library files.

13.2.1 Finding the right version of the archive

The archives have names of the form “cocoqml-〈coco-version〉-qt〈qt-version〉-〈compiler〉-〈platform〉.zip”, withthe following components:

〈coco-version〉:The version of Coco to which the add-on belongs. It has the form x.y.z, like e.g. 4.0.3.All versions of Squish Coco with a release number later than that of 〈coco-version〉 can then read the coveragedata that are generated by the add-on. However, since each Squish Coco release can read the data generated byearlier releases, it is not necessary to update the add-on when a newer version of Squish Coco is installed.



Installation

〈qt-version〉:The Qt version for which the library is compiled. It has the form a.b.c, like e.g. 5.8.0.

Note that a new version of the add-on is necessary whenever the patch level (the third number in the versionstring) of the Qt installation changes.

〈compiler〉:A string that identifies the compiler for which the Qt library is built. Examples are gcc_64 or msvc2013.

〈platform〉:Either linux, linux_64, mac or windows.

13.2.2 Linux™ installation

Under Linux™, the archive contains a library file with a name like libQt5Qml.so.5.8.0. The numbers at the endare those of the Qt version.

To find the right version of the archive to download, you can usually use the command qmake that is used for thecompilation of your project. If Qt is installed in the /opt/Qt/ directory, the full path to qmake might be /opt/Qt/5.8/gcc_64/bin/qmake. This means that the 〈compiler〉 is gcc_64 and the 〈qt-version〉 is a version number thatbegins with 5.8. To get the full version number, run qmake in the form

$ /opt/Qt/5.8/gcc_64/bin/qmake -query QT_VERSION5.8.0

This then tells us that the full 〈qt-version〉 must be 5.8.0.

The correct package is then, for a 64-bit Linux system, “cocoqml-4.1.0-qt5.8.0-gcc_64-linux_64.zip”.Unpack it and copy the file libQt5Qml.so.5.8.0 contained in it to the directory /opt/Qt/5.8/gcc_64/lib.

i Apart from libQt5Qml.so.5.8.0, there are also other files in the same directory that also begin withlibQt5Qml.so. These files are only symbolic links and need not to be changed.

13.2.3 macOS installation

Under macOS, the archive contains a library in the form of a directory, with the name QtQml.framework/.

To find the right version of the archive to download, you can usually use the command qmake that is used forthe compilation of your project. If Qt is installed in the /Users/someone/Qt/ directory, the full path to qmakemight be /Users/someone/Qt/5.8/clang_64/bin/qmake. This means that the 〈compiler〉 is clang_64 and the〈qt-version〉 is a version number that begins with 5.8. To get the full version number, run qmake in the form

$ /Users/someone/Qt/5.8/clang_64/bin/qmake -query QT_VERSION5.8.0

This tells us that the full 〈qt-version〉 must be 5.8.0.

The correct package is then “cocoqml-4.1.0-qt5.8.0-clang_64-mac.zip”. Unpack it and copy the directoryQtQml.framework/ contained in it to the directory /Users/someone/Qt/5.8/clang_64/lib/. (You need torename the original QtQml.framework/ directory.)


CocoQML Tutorial

13.2.4 Microsoft® Windows installation

Under Microsoft® Windows, the archive contains two Qt library files, Qt5Qml.dll and Qt5Qmld.dll, together withsome more DLL files that are needed by the enhanced QML libraries.

To find the right version of the archive to download, you can usually use the command qmake that is used for thecompilation of your project. If Qt is installed in the C:\Qt\ directory, the full path to qmake might be C:\Qt\5.8\msvc2013\bin\qmake. This means that the 〈compiler〉 is msvc2013 and the 〈qt-version〉 is a version number thatbegins with 5.8. To get the full version number, run qmake in the form

C:\Users\Someone>C:\Qt\5.8\msvc2013\bin\qmake -query QT_VERSION5.8.0

This tells us that the full 〈qt-version〉 must be 5.8.0.

This then means that the correct package is “cocoqml-4.1.0-qt5.8.0-msvc2013-windows.zip”. Unpack it andcopy the DLL files contained in it to the directory C:\Qt\5.8\msvc2013\bin\.

13.3 CocoQML Tutorial

The following tutorial provides examples how to use CocoQML. It also shows how to detect whether CocoQML isinstalled correctly.

i The following examples are written for a UNIX® system. Microsoft® Windows is similar.

13.3.1 Compile a Qt application

We will use the calqlatr program as our example. It is part of the Qt distribution and can be found in theExamples/Qt-5.8/quick/demos/calqlatr/ subdirectory of your Qt installation.

The easiest way to set it up is to use the Qt Creator. Start the Qt Creator, select the "Welcome" mode and then"Examples". It is then possible to search for “calqlatr” and open the project.

i Make sure that the compiler toolkit used for compiling the calqlatr project agrees with that of the CocoQMLadd-on. If you have e.g. downloaded and installed the package cocoqml-4.0.3-qt5.8.0-gcc_64-linux_64.zip, you need to select the Qt 5.8.0 GCC 64bit kit in the Qt Creator to compile it.

Compile the program and run it. Check whether it runs correctly.

13.3.2 Enable code coverage

Now you can enable code coverage by setting the COCOQML variable to 1. To do this, select "Projects" in the QtCreator and add COCOQML=1 to the "Run Environment". Then run calqlatr again.

In the "Application Output" window you now should see lines like the following:


https://doc.qt.io/qt-5/qtquick-demos-calqlatr-example.html

Environment Variables

Starting /home/user/calqlatr/calqlatr...CocoQML: Instrumentation database for ’calqlatr.qml’ written to /home/user/calqlatr/

calqlatr_qml.csmesCocoQML: Instrumentation database for ’calculator.js’ written to /home/user/calqlatr/

calqlatr_qml.csmesCocoQML: Instrumentation database for ’NumberPad.qml’ written to /home/user/calqlatr/

calqlatr_qml.csmesCocoQML: Instrumentation database for ’Button.qml’ written to /home/user/calqlatr/

calqlatr_qml.csmesCocoQML: Instrumentation database for ’Display.qml’ written to /home/user/calqlatr/

calqlatr_qml.csmes

If it does not, there is an error in the setup of CocoQML.

Now do some computations with the calqlatr, then close the program window. (Do not kill it from the Qt Creator,because then it cannot write the coverage data.) You should see an additional output similar to this:

CocoQML: Saved coverage information: /home/user/calqlatr/calqlatr_qml.csexe/home/user/calqlatr/calqlatr exited with code 0

From this output we can see that the files calqlatr_qml.csmes and calqlatr_qml.csexe have been created inthe directory /home/user/calqlatr. You can now open them with the CoverageBrowserand view the results.

13.3.3 Excluding source files from coverage

There is a common use case when some source files need to be excluded from instrumentation – maybe because theycause instrumentation errors or because coverage data is not needed for them. We will now show how this is doneand exclude the file Button.qml from the code coverage measurements.

Source code exclusion is controlled by the environment variable COCOQML_BLACKLIST, the value of which must bethe path to a text file. So we need to create he text file first.

Create a file with the name cocoqml.txt. It should contain one line of text,

Button.qml

Then add the variable COCOQML_BLACKLIST to the "Run Environment" section in the Qt Creator. Its value shouldbe the full path to the cocoqml.txt file, like “/home/user/cocoqml.txt”.

Then the file Button.qml (and in fact all files with a name that contains the string “Button.qml”) will be excludedfrom instrumentation. Run the calqlatr program again. In the "Application Output" window you should nowsee a similar output as before, but with the Button.qml line missing. Similarly, the file Button.qml will be nolonger be visible in the CoverageBrowser.

13.4 Environment Variables

When the Coco add-on is present, QML coverage is determined by environment variables. By default, instrumentationand coverage are switched off.


Environment Variables

The following variables can be used to control code coverage:

COCOQML:Set this variable to 1 to enable the instrumentation of your QML source code. If the variable is not set or set toa different value, no instrumentation will be done: The Qt installation then behaves as usual.

COCOQML_OUTPUT:This variable contains the path to the directory in which the output files are generated. If the variable is not set,the current working directory of the program is used.

Setting this variable is especially useful if you do not have sufficient permissions in the automatically determinedpaths or want to gather all intermediate instrumentation databases in a single file.

COCOQML_BLACKLIST:Set this variable to exclude certain paths from instrumentation. The value of the variable is the path to a textfile which contains in each line a regular expression. If one of the regular expressions matches a file name, thefile is excluded from the instrumentation.

Lines beginning with a “#” are ignored.

The following variables control various debug output generated by the extension, especially the log messages. All logmessages are prefixed with “CocoQML: ” (without the quotes).

COCOQML_QUIET:Set this variable to 1 if you want to disable all messages generated by the add-on. Usually all messages will beprinted to standard error, but if you rely on parsing standard error in your application you can use this to avoidinterference.

This setting overrides the verbose mode, see below.

COCOQML_VERBOSE:Set this variable to 1 if you want to enable verbose mode. If the verbose mode is active, the CocoQMLextension will print additional messages that might be helpful for debugging purposes. Usually you will notneed to enable verbose mode.

This settings can be overridden by the quiet mode, see above.

COCOQML_LOGFILE:If this variable is set, the CocoQML add-on writes its log messages to a file. The value of COCOQML_LOGFILEmust be an absolute or relative file path. A relative file path is interpreted relative to the working directory ofthe application when it starts. If the log file already exists, the new data is appended to it. If the path containsdirectories, they must already exist, otherwise no log file will be created.

Example: Under Unix, one could set COCOQML_LOGFILE=/tmp/cocoqml.log. Since the directory /tmp/exists in any normal Unix installation and is writable, all log messages are then written to the file /tmp/cocoqml.log

COCOQML_DUMPLOC:If this variable is set, the CocoQML add-on writes the instrumented versions of the QML files it instruments tothe disk.

The value of COCOQML_DUMPLOC must be the path to a directory, either absolute or relative. A relative path isinterpreted relative to the working directory at start time. If the directory does not exist, it is created.


Part VI CoverageScannerTclReference



Chapter

14Command Line Arguments

Syntax:

coveragescannertcl [〈cs-options〉] 〈application arguments〉

Where:

〈application arguments〉:Tcl-based application including optional arguments

〈cs-options〉:Any of the following options:

--cs-output=〈STRING〉: Base name of the generated .csmes and .csexe files

--cs-exclude-path=〈STRING〉: Exclude files located in specified directory

--cs-include-path=〈STRING〉: Override exclusion of files based on specified directory

--cs-exclude-file-abs-regexp=〈STRING〉: Exclude file if absolute file name matches regular expression

--cs-exclude-file-abs-regexp=〈STRING〉: Override exclusion of file if absolute name matches regularexpression

--cs-exclude-file-abs-wildcard=〈STRING〉: Exclude file if absolute name matches wildcard

--cs-exclude-file-abs-wildcard=〈STRING〉: Override exclusion of file if absolute name matches wild-card

--cs-include-code-regexp=〈STRING〉: Regular expression of code that will be instrumented even if it isexcluded

--cs-exclude-code-regexp=〈STRING〉: Regular expression of code that will not be instrumented

--cs-include-code-wildcard=〈STRING〉: Wildcard of code that will be instrumented even if it is ex-cluded

--cs-exclude-code-wildcard=〈STRING〉: Wildcard of code that will be instrumented even if it is ex-cluded

--cs-include-init-scripts: Enables code instrumentation for standard Tcl initialization scripts

--cs-include-unnamed: Instrument Tcl code even if it has no or just an ambiguous file name set. By defaultsuch code is ignored as it is typically just temporary code generated for e.g. an Tk event handler orresults from nested eval calls.



CoverageScannerTcl is a utility which permits to collect code coverage information of a Tcl-based execution. It willcollect all scripts parsed by the embedded Tcl interpreter and determine which parts were actually executed.

Once the application exits a pair of .csmes and .csexe files will be written out that contain the code instrumentationinformation and execution data, respectively.

If the --cs-output switch is omitted the base name of the output files will the same as the 〈application〉 name.

See (see Chapter 15, page 112) for a recommended set of --cs-exclude-code-regexp switches for typical Tcl/Tkapplications.


Mixed Tcl/C/C++ Code Coverage Analysis

Chapter

15Mixed Tcl/C/C++ Code Coverage Analysis

15.1 Analysis of Tcl script coverage

15.1.1 Demo application build

Copy the sample to your working diretcory

$ cp -r /opt/SquishCoco/doc/demoapp_tcl ./$ cd demoapp_tcl

Edit Makefile and adjust the following variables

TCLLIBS=-ltk8.5 -litcl3.4 -ltcl8.5TCLINCLUDES=-I/usr/include/tcl8.5

to match your system’s library names and include paths.Build the C++ executable with:

$ make

Test the simple Tk GUI with an invocation of the launcher script:

$ ./proto

15.1.2 Ad-hoc Tcl coverage run

Launch the application under test via the Tcl coverage scanning tool :

$ coveragescannertcl --cs-output=tcl ./proto

The GUI will show up as normal while code coverage is monitored. At this point the application might alreadybe shut down via the "FileÕExit" menu entry or the [X] button in the window’s title bar. Observe the two files


Analysis of Tcl script coverage

tcl.csmes and tcl.csexe being generated. The latter contains data about the code covered in the last execution.The former is a measurement database containing information about the instrumented code and possibly importedprevious executions. In this case the base name of the output files was set via the --cs-output switch. If omitted itwill be deduced from the application name.

15.1.3 Result analysis

The results can be view interactively in the GUI provided by Coco. Run

$ coveragebrowser

to launch it. Select "FileÕOpen..." to open the tcl.csmes file. You’ll see all Tcl files and procedures listed in theSources and Methods tab, respectively. When selecting either of them a source code viewer will open that showsinstrumented code with a gray background color. Note that the coloring is still limited to instrumented, i.e. seen codeonly. To check which code got actually executed (or not) select "FileÕLoad Execution Report..." and opentcl.csexe. Now red and green colors denoting positive and negative coverage will show up for files, proceduresand the code itself.

15.1.4 Filtered Tcl coverage run

As you might have noticed that above ad-hoc run produced results for several unnamed and internal files. Thesestem from internal Tcl/Tk files or e.g. mouse event handlers that result in on-the-fly eval calls. Compared to theapplication-specific Tcl files the coverage information for these internal pieces of code is of little interest.

The easiest way to hide the non-relevant pieces of code is through usage of the various --cs-exclude-code options.A recommended list of switches will be shown in the repetition of the previous run with filtering options applied:

$ coveragescannertcl --cs-output=tcl --cs-exclude-code-regex="^\s*[^\n]*\s*$"--cs-exclude-code-regex="(tcl|tk)Init" --cs-exclude-code-regex="namespace eval ::

itcl"--cs-exclude-code-regex="^# This file is generated"--cs-exclude-code-regex="tk::Menu(Motion|Leave)" ./proto

When viewing the new tcl.csmes in CoverageBrowser you’ll notice that the list of files is reduced to the relevantcanvas.tclx, input.tclx, mainwindow.tclx, point.itcl and rect.itcl scripts.

15.1.5 Report Generation

All of text, XML and HTML reports can be generated from the CoverageBrowser GUI. With future automation ofnightly test runs or Continuous Integration (CI) in mind we’ll create the report from the command line. And for thisvery purpose we’ll also pretend that the execution data has not been imported into the measurement database, yet.This step can be accomplished like this from the command line:

$ cmcsexeimport --title="First run" -m tcl.csmes -e tcl.csexe

The resulting proto.csmes file now contains everything we need to generated a report:

$ cmreport --title="Coverage Data" -m tcl.csmes --html=report.html


Analysis of C/C++ code coverage

The resulting report.html file (including a report_html/ directory) can now be visited with a Web browser. Itincludes a listing of executions, source files, functions and metrics on a global level, per file and per functions.

15.2 Analysis of C/C++ code coverage


While Tcl code can be instrumented for coverage analysis on-the-fly this step requires a rebuild for C or C++applications. But this can be accomplished with almost no change to your regular application build. Here are the twopossible approaches:

• Modify your build system to make use of the code scanning tools csgcc, csg++, etc. that will both instrumentthe code and later use the original compiler to compile the code as usual.

• Sneak Coco’s compiler wrappers named gcc, g++, etc. into the executable search PATH. They will againinstrument the code and then let the real compiler do its job.

We’ll make use of approach number 2. First, clean the existing object files:

$ make clean

Then, recompile the application with a changed PATH and a switch that enables the compiler wrappers. For brevity,we’ll do this all in a single line:

$ PATH=~/SquishCoco/wrapper/bin:$PATH COVERAGESCANNER_ARGS="--cs-on" make

As a result we’ll not only get the main executable (proto.exe) but also a corresponding instrumentation file proto.exe.csmes. Information about the instrumented code (gray background color) can optionally be reviewed at thispoint

$ coveragebrowser -m proto.exe.csmes

15.2.2 Execution

Let’s run the application

$ ./proto

and quit it via its GUI right after. The current working directory will now contain a proto.exe.csexe containinginformation about completed execution.

15.2.3 Result analysis

Same as with Tcl the results can now be reviewed in CoverageBrowser. A short cut invocation that opens both theinstrumentation and execution files is shown here:



$ coveragebrowser -m proto.exe.csmes -e proto.exe.csexe


The process of a report generation works analogous to the approach shown for Tcl code above.

15.2.5 Analysis of mixed Tcl and C/C++ code coverage

Coverage data for the Tcl and C/C++ parts of a hybrid application can be gathered independently as shown above.Either in a single or multiple executions. The only extra step needed is the merging of the data into a singlemeasurement database. Which then can be used to generate a single, combined report taking both languages intoaccount.

15.2.6 Combined coverage run

Assuming the instrumentation of the C/C++ code is still in place we can gather coverage data for both types oflanguages with a single run:

$ coveragescannertcl --cs-output=tcl ./proto

With the proto.exe.csmes file still in place from the application rebuild we’ll now have two sets of files:

1. tcl.csexe

2. tcl.csmes

3. proto.exe.csexe

4. proto.exe.csmes

At this point the coverage data for Tcl and C/C++ could be analysed independently using cmcsexeimport. But wewant to gain an overall picture and will therefore merge the results in the next section.

15.2.7 Merging measurement data

Instead of using the Coverage Browser GUI we’ll again use the command line tools. In the first step we’ll merge theinstrumentation files stemming from the two languages into one:

cmmerge -o all.csmes tcl.csmes proto.exe.csmes

To save some disk space the --append option can be used rather than creating the new all.csmes file.

In the second step the two execution files are merged using the merge policy for the 2nd language:

$ cmcsexeimport -m all.csmes -e tcl.csexe --title="Hybrid" --passed$ cmcsexeimport -m all.csmes -e proto.exe.csexe -p merge --title="Hybrid" --passed

For the merge to actually happen the title has to be identical.




An HTML report encompassing coverage for both application parts can be generated with a single command linecall:

$ cmreport --title="Hybrid Tcl/C++ Coverage" -m all.csmes --html=report.html

Select "Source Files" to see both Tcl (various files) and C++ code (main.cpp) being listed in a combined report.


Part VII CoverageBrowserReference


Introduction

Chapter

16Introduction

CoverageBrowser is a graphical user interface program which enables the user to analyze the test coverage.

CoverageBrowser is typically used in the following way:

1. Load an instrumentation database (a .csmes file) that was generated by CoverageScanner.

2. Load a corresponding execution report (a .csexe file). There may be several reports to choose from: Cover-ageBrowser displays them in a tree view where they can be selected or deselected individually for coverageanalysis.

3. Search for untested code segments.

4. Mark dead code or code which cannot be tested as “manually validated”.

5. Add comments to the instrumented code.

CoverageBrowser saves all these data (execution reports, comments, etc.) to the instrumentation database.

16.1 Command Line Arguments

Syntax:

coveragebrowser -m 〈csmes_file〉 -e 〈csexe_file〉 ...

Where:

-m 〈argument〉 | --csmes=〈argument〉:Load an instrumentation database.

-e 〈argument〉 | --csexe=〈argument〉:Open the import dialog which permits to load a specified execution report.

This command line argument requires the command line option --csmes.


Black box and white box testing

Chapter

17Black box and white box testing

CoverageBrowser can be used both for white box testing and black box testing: If no source code information isavailable in the instrumentation database (i.e., in the .csmes file), CoverageBrowser will switch to black box testingmode. In this mode, CoverageBrowser has a simpler user interface that does not provide the functionality that ispossible only with access to the source code. Nevertheless, even with this reduced functionality, it is still possible toimport and manage executions.

A black box instrumentation database can be generated by clicking on "FileÕGenerate Black-Box Configuration...".It is possible to merge such a database into a white box configuration at a later stage.


Black box and white box testing

Figure 17.1: CoverageBrowser’s simplified user interface for black box tests


The windows of CoverageBrowser

Chapter

18The windows of CoverageBrowser

18.1 The Executions Window

This window is present in both black box and white box testing mode. It shows all application executions that havebeen done, including details of their code coverage.

18.1.1 Principles

Executions of the instrumented application are displayed in a tree view in the "Executions" window. Coverage-Browser uses a slash ‘/’ as a separator for grouping measurements together.

For example, the tests shown in Figure 18.1, page 121 have the following names:

• SubTest/test1

• SubTest/test2 (12)

• SubTest/test2 (2)

Code Coverage

Statistics

Selection

Execution State

or Regular Expressions

Wildcard Expressions

Selection using

Figure 18.1: The Executions View

The checkbox next to each item can be used to select executions. The "Sources", "Functions" and "SourceViewer" windows only display the code coverage status of the selected executions.

The code coverage percentage of the tests is visualized by gray horizontal bars in the "Coverage" column.

The input field allows to filter the output with regular expressions (see Chapter 19.1, page 133). Click the button


The Executions Window

to select all the visible executions (i.e., all those that have not been filtered out). Or click the button to unselect allthe executions.

For a more finely controlled filter, click the "..." button: This will pop up a dialog by which it is possible to setfiltering depending on the execution state and the comments.

Note that the text which is filtered is a test execution’s full name—for example, SubTest/test1.

Click the -+ button to switch to the execution comparison analysis mode (see Chapter 18.1.3, page 125).

The user can set the state of the executed test by clicking into the "State" field of a test. The new state can be any ofthe following:

"Unknown":Default state.

" Passed ":This state is used to mark the test as passed. The background colour of the "State" field is then green.

" Failed ":This state is used to mark the test as failed. The background colour of the "State" field is then red.

" Need manual check ":This state is used to indicate that the test must be done manually. The background colour of the "State" fieldis then orange.

i The name of the test item and its state can also be defined by an external test suite. (see Chapter 30, page 172)

It is possible to rename, delete or merge executions, or add comments to them through the use of CoverageBrowser’scontext menus and dock windows1. One can use regular expressions to identify the executions to which thesemodifications are applied. (The regular expression syntax is described in Chapter 19.1, page 133.) Before regularexpression-driven actions are carried out, CoverageBrowser shows a preview of what effects the change would have.

To delete executions, right-click into the "Executions" window and select "Delete multiple executions..."from the context menu. A window appears into which the name of the executions can be entered. Here are someexamples:

• To delete all execution using the wildcard syntax:Execution Name: *

• To delete all executions in TESTS using the wildcard syntax:Execution Name: TESTS/*

• To delete all executions in TESTS using the regular expression syntax:Execution Name: =TESTS/.*

To rename executions, select "Rename multiple executions..." from the context menu. A window appears (seeFigure 18.2, page 123) into which expressions for the old and new names of the executions can be entered. Here aresome examples:

• To move all executions to the directory TESTS, set:Actual Execution Name: =.*New Execution Name: TESTS/&

• To move all executions in the directory TESTS to the directory OLD, set:Actual Execution Name: =TESTS/(.*)New Execution Name: OLD/\1

1After merging, the executions used as the source of the new merged execution are deleted.



• To rename all executions in all directories in testname [directory], set:Actual Execution Name: =([^/]*)/([^/]*)New Execution Name: \2 [\1]

Error Message

Error

Regular Expression

Executions to Rename

New Execution Name

Preview

Figure 18.2: Renaming with regular expressions

18.1.2 Loading an Execution Report

An execution report is produced when an instrumented application finishes execution. Its name is defined by theinitialization function __coveragescanner_install() of the CoverageScanner library (see Chapter 9.1.1, page 73)and always ends with ".csexe". The report contains the list of all executed code segments for each application thatwas run. The execution report is never overwritten; execution data are always appended.

To load an execution report, click on "FileÕLoad Execution Report..." or the icon on the toolbar. A dialoglike that in Figure 18.3, page 123 appears.

Default execution name

Execution resultfile name

Default execution status

Figure 18.3: Execution report loading dialog

The file can be loaded directly or by a script. To load it directly, click on "File" and enter the path of the .csexe fileinto the free form input box use the browse button.

The "Name" field allows the user to specify the name of the imported instrumentation if this is not already specified inthe execution report (see Chapter 9.1.2, page 74). It is also possible to set the status (“Passed”, “Failed” or “RequiresManual Checking”) of all imported executions. By default the status is set to “Unknown”.

Additional options become visible when the button "Options" is pressed.

• If the checkbox "Delete execution report after loading" is enabled, the execution report file will beautomatically deleted after it has been loaded.



• If the checkbox "Open this dialog automatically when the file is modified" is selected, the im-port dialog reappears whenever the .csmes file changes, and it can be loaded again.

A workflow to collect test results interactively is to have this and the previous checkbox selected and thenrun the instrumented program repeatedly. Each time the program ends, a .csmes file is created and can beimported in the CoverageBrowser and then be deleted. This way the test results accumulate.

• The option "Import Preprocessing" allows to select the behaviour in the case of conflicts or redundantexecutions:

"Ignore Duplicate Executions":Executions are ignored if they have executed the same code as an execution that was already imported.

"Import Duplicate Executions":Executions are imported if at least one instrumented source code line is executed.

"Import Duplicate and Empty Executions":All executions are imported.

"Merge executions with the same name":All executions with the same name are merged. The execution counts of all merged instrumentations areadded.

Invalid executions are not imported. If more than one instrumentation is imported with the same name, an indexnumber is appended to the name to make it unique. An execution with the name “Main Test” may then become“Main Test (2)”.

To see which executions will be imported, press the button "Overview", which shows a list of all executions in thefile you want to load, together with a description whether they will be imported and why. (see Figure 18.4, page 124)

i At this point, no import takes place. One still needs to press the "Import" (or "Import & Delete") button toload the .csexe data.

Figure 18.4: Import overview



Import by a script If the execution report is not accessible through the file system, a script can be used. The scriptmust print the content of the execution report to the standard output (stdout). If the program writes to the standarderror stream (stderr), the output occurs in a window and can be used for debugging. On success, the script mustreturn the value 0.

To use the script, select "Script:" in the menu at the top left at the dialog.

18.1.3 The Execution Comparison Analysis Mode

The Execution Comparison Analysis mode is activated by clicking the button -+ . In this mode, one or more executions

must be selected as references and others as those to analyze. CoverageBrowser then shows only those lines ascovered that were executed in the analyzed executions but not in the reference executions. Similarly, the coveragestatistics displayed in the "Sources" list contain only the percentage of instrumented statements that were executedonly by the executions to analyze.

With Execution Comparison Analysis mode it is therefore possible to find changes in the coverage between differentruns of a program.

To select the reference executions, use the checkboxes in the "Reference" column at the right of the "Executions"window. They must be selected first. Then select the executions to analyze as before in the "Execution" column.

i If the execution to analyse is present in the list of reference executions, it is implicitly removed from the list.So if execution A is compared to executions A and B, CoverageBrowser actually compares execution A onlywith B, since comparing A with itself will provide no useful information.

Activation

of lines which

are only coveraged

Coverage Statistic of

by the selected execution

are only coveraged

by the selected execution

are displayed

Instrumented lines which

Reference

List

Execution

Analysed

Figure 18.5: Execution Comparison Analysis Mode


The Source Browser Window

18.2 The Source Browser Window

i This feature is not available for black box testing.

The "Sources" window can be displayed by clicking on "ViewÕSources".

Statistics

per File

Source

Header and

Source Files

Compiled

Source excludedfrom the instrumentation

Figure 18.6: Source Browser Window

Each line represents a source file. For C and C++, there are sub-entries for included header files which have beeninstrumented. When an item is clicked, the "Source Window" is displayed.

The "Coverage" column displays a rudimentary code coverage statistics for each source code file. There is anunderlying horizontal bar in each field whose length represents the fraction of the code that is covered. The color ofthe main part of the bar is selected according to code coverage statistics for each file and the value of the thresholds(see Chapter 21.3, page 146).

If parts of the code have been manually validated, then there is a second, gray, horizontal bar at the left of the"Coverage" column; its width represents the number of tests that have been only manually validated. The full lengthof the column still represents the fraction of all tests that have been validated. When the window however alsocontains a "Manual Validations" column, the gray horizontal bar is only displayed there.

The checkbox in front of each source file allows to exclude or include the source file to the statistics computation. Ifexcluded, the file is treated as if it and its functions had not been instrumented.

The input field allows to filter the content of the window with regular expressions (see Chapter 19.1, page 133).The filter expression refers to the full path of the source file. (Example: c:\directory\file.cpp)

Icon Shortcut DescriptionCtrl+Shift+F Previous source file

Ctrl+F Next source file

Table 18.1: Source Browser - Shortcuts

18.3 The Function Browser Window



The Source Viewer Window

The "Functions" window can be opened by clicking on "ViewÕFunctions".

Prototype

Statistics

Namespace

or Class

Method

Figure 18.7: Function Browser Window

It displays the code coverage statistics for all functions, classes and namespaces. A click on an item in the windowshows the code for the corresponding object in the "Source Viewer" window and highlights it.

The "Coverage" column displays a rudimentary code coverage statistics for each function. There is an underlyinghorizontal bar in each field whose length represents the fraction of the code that is covered. The color of the mainpart of the bar is selected according to code coverage statistics for each file and the value of the thresholds (seeChapter 21.3, page 146).

If parts of the code have been manually validated, then there is a second, gray, horizontal bar at the left of the"Coverage" column; its width represents the number of tests that have been only manually validated. The full lengthof the column still represents the fraction of all tests that have been validated. When the window however alsocontains a "Manual Validations" column, the gray horizontal bar is only displayed there.

The input field allows to filter the output with regular expressions (see Chapter 19.1, page 133). The filter expressionrefers to the full names of the items, including the class name and the namespace. (Example: MyNamespace::MyClass::MyProc)

18.4 The Source Viewer Window


18.4.1 Source Display

The "Source Viewer" window can be displayed by clicking on "ViewÕNew Source Window".



Selected

InstrumentationFolding

Filter

Comment

Mark

Unselected

Instrumentation

Test and Code

Coverage Count

Figure 18.8: Source Window

The "Source Viewer" window displays the source file or its C or C++ preprocessed view. Clicking on C++CPP enables

the user to toggle between the 2 different views.

The source code is colored with code coverage instrumentations. The colors used are described in section 18.4.2,page 128. By selecting an area with the mouse, corresponding instrumentations are highlighted and a detaileddescription of them is displayed in the "Explanation" window (see Chapter 18.5, page 131). It is possible tonavigate between instrumentations using the navigation buttons and . Navigation buttons in yellow, blue, red andgreen permit to jump to the next or previous comments, manually validated instrumentations, non-executed codeparts or executed code parts. Clicking on the source code selects the nearest instrumentation.

If a comment is entered for an instrumentation, the icon is displayed in the margin.

On the right side, CoverageBrowser displays the test coverage count2 or the code coverage count3 for each line. If asource code line contains more than one instrumentation, CoverageBrowser display the range of their counts.

Mouse Wheel DescriptionWheel Scroll up/downCtrl+Wheel Zoom in/outShift+Wheel Next/previous instrumentation

Table 18.2: Source Display - Mouse Wheel

18.4.2 Color Convention

Instrumentations are displayed in a source window using different colors:

Green - "Executed":An instrumentation is displayed in green when the code has been executed.

Orange - "Partially Executed":An instrumentation is marked as "Partially Executed" when it is not completely executed. This occurswhen a Boolean expression was only true or false for example. In the case of a source code line which contains

2The test coverage count is obtained by generating the instrumentation with CoverageScanner using the option --cs-hit3The code coverage count is obtained by generating the instrumentation with CoverageScanner using the option --cs-count



Icon Shortcut DescriptionCtrl+Shift+B Previous comment

Ctrl+B Next commentCtrl+Shift+U Previous unexecuted code

Ctrl+U Next unexecuted codeCtrl+Shift+T Previous executed code

Ctrl+T Next executed codeCtrl+Shift+V Previous manually validated instrumentation

Ctrl+V Next manually validated instrumentationCtrl+Shift+I Previous instrumentation

Ctrl+I Next instrumentationCtrl+Shift+F Previous source file

Ctrl+F Next source fileCtrl+J Open a new source window

C++CPP Ctrl+Shift+J Switch between the preprocessor view and the original source

Ctrl+K Add/Edit CommentsCtrl+Shift+K Remove Comments

ok Ctrl+D Mark as Validatedok Ctrl+Shift+D Clear Validation Flag

Ctrl+Z UndoCtrl+Shift+Z Redo

Table 18.3: Source Display - Shortcuts

more than one instrumentation, the line is marked as "Partially Executed" when one of its instrumentationshas not been "Executed". A detailed information is displayed in the "Explanation" window (see Chapter 18.5,page 131).

Red - "Never Executed" or "Execution count too low":An instrumentation is displayed in red when the code is never executed or when the execution count is lowerthat than the execution count requested.

Magenta - "Dead-Code":An instrumentation is displayed in magenta when the code cannot be executed.

Blue - "Manually Set To Be Executed":The user has the possibility to mark an instrumentation as ’Manually Validated’. This is usually to excludedead code or code which cannot be tested for code coverage statistics. This state is only relevant if executionsare in a "Never Executed" or "Partially Executed" state.

Gray - "Unknown" or "Hidden":Gray is used when no information about instrumentation is available. This occurs when no executions areselected or when comparing executions of tests (see Chapter 18.1.3, page 125).

18.4.3 Comments




18.4.3.1 Editing Comments

It is possible to add a comment by selecting an instrumentation and clicking on the context menu entry "Add/SetComment", the main menu entry "InstrumentationÕAdd/Set Comment" or the icon on the toolbar.

The "Comment" Window 18.9, page 130 appears and allows a comment to be edited. The most recently enteredcomments can be retrieved by clicking on the "Last Comments" selection field. Basic text formatting is possibleusing the integrated toolbar buttons (see 18.4, page 130).

Text

Formating

Comment

Length

Last

Comments

Edit Field

Figure 18.9: Comment Editing

i If a minimal length for a comment is set, the comment can only be entered if this is reached (see Chapter 21.2,page 146).

The comment is printed in the explanation in a yellow box and the icon ( ) is displayed in the source window nearthe line number.

Icon Shortcut DescriptionCtrl+S StrikeoutCtrl+B BoldCtrl+I ItalicCtrl+U UnderlineCtrl+J JustifyCtrl+R Align RightCtrl+L Align LeftCtrl+M CenterCtrl+Z Undo

Ctrl+Shift+Z Redo

Table 18.4: Comments - Shortcuts

18.4.3.2 Removing Comments

It is possible to remove a comment by selecting an instrumentation and clicking on the context menu entry "ClearComments", the main menu entry "InstrumentationÕClear Comment" or the icon on the toolbar.


The Explanation Window

18.5 The Explanation Window


The "Explanation" Window 18.10, page 131 is a docking window which is automatically updated with a detaileddescription of the selected instrumentations of the source window. For each instrumentation, the following informationis displayed:

1. A short description of the instrumentation state (see Chapter 18.4.2, page 128).

2. The preprocessed source code which is concerned by the instrumentation.

3. For Boolean expressions, the truth-table which shows executed and unexecuted states.

4. The list of executions which are executing the portion of code.

5. User comments.

Preprocessed

Source

Comment

Truth-Table

State

Figure 18.10: Explanation Window

CoverageBrowser displays the truth-table in the case of a Boolean expression which is partially executed. Thetruth-table indicates which value the expression has or has not reached during execution.

Example: the truth-table 18.5, page 131 indicates that the expression was false but not true.

TRUE FALSE

no yes

Table 18.5: Truth-Table Example


The Statistics Window

18.6 The Statistics Window

The "Statistics" Window 18.11, page 132 is a docking window which is automatically updated with the codecoverage statistic for the whole project.

Figure 18.11: Statistics Window

If parts of the code are manually validated, then their percentage is also displayed in the coverage statistics. The barchart has then two regions: the percentage of the manually validated code at the left and then the percentage of thecode that is covered by the automatic tests. The numbers in the bar chart refer to all validated code, that which wasmanually validated together with that covered by tests.

By clicking the "..." button, the two kinds of validation are split into two separate bars.

Figure 18.12: Statistics Window with Manual Validation, Split


Working with CoverageBrowser

Chapter

19Working with CoverageBrowser

19.1 Filtering with wildcards or regular expressions

CoverageBrowser provides a generic filtering mechanism of rows using wildcard or regular expressions. Wildcardexpressions are activated per default and regular expressions are selected when the expression starts with an equalsign (’=’). Clicking on the filter icon converts the expression from wildcard into regular form as far as this is possibleand vice versa.

Icon Description.* The filter uses regular expression syntax.?* The filter uses wildcard syntax.! Syntax error. More information are displayed in the status bar.

Table 19.1: Filter States

19.1.1 Wildcard Expressions

Element Meaning* any characters (0 or more)? any character[...] set of character

Example: foo*bar match any tests containing the string foo followed by bar.

19.1.2 Regular Expression

The first character must be ’=’ to activate the regular expressions.


Filtering

19.1.2.1 Pattern matching

Element Meaningc Any character represents itself unless it has a special regexp meaning. Thus c matches

the character c.\c A character that follows a backslash matches the character itself except where mentioned

below. For example if you wished to match a literal caret at the beginning of a string youwould write \^.

\a This matches the ASCII bell character (BEL, 0x07).\f This matches the ASCII form feed character (FF, 0x0C).\n This matches the ASCII line feed character (LF, 0x0A, Unix newline).\r This matches the ASCII carriage return character (CR, 0x0D).\t This matches the ASCII horizontal tab character (HT, 0x09).\v This matches the ASCII vertical tab character (VT, 0x0B).\xhhhh This matches the Unicode character corresponding to the hexadecimal number hhhh

(between 0x0000 and 0xFFFF).\0ooo (i.e., zero ooo) matches the ASCII/Latin1 character corresponding to the octal number ooo (between 0

and 0377).. (dot) This matches any character (including newline).\d This matches a digit.\D This matches a non-digit.\s This matches a whitespace.\S This matches a non-whitespace.\w This matches a word character.\W This matches a non-word character.^ The caret negates the character set if it occurs as the first character, i.e. immediately after

the opening square bracket. For example, [abc] matches ’a’ or ’b’ or ’c’, but [^abc]matches anything except ’a’ or ’b’ or ’c’.

- The dash is used to indicate a range of characters, for example [W-Z] matches ’W’ or’X’ or ’Y’ or ’Z’.

E? Matches zero or one occurrence of E. This quantifier means "the previous expression isoptional" since it will match whether or not the expression occurs in the string. It is thesame as E{0,1}. For example dents? will match ’dent’ and ’dents’.

E+ Matches one or more occurrences of E. This is the same as E{1,}. For example, 0+ willmatch ’0’, ’00’, ’000’, etc. . .

E* Matches zero or more occurrences of E. This is the same as E{0,}. The * quantifier isoften used by a mistake. Since it matches zero or more occurrences it will match nooccurrences at all. For example if we want to match strings that end in whitespace anduse the regexp \s *$ we would get a match on every string. This is because we havesaid find zero or more whitespace followed by the end of string, so even strings that donot end in whitespace will match. The regexp we want in this case is \s +$ to matchstrings that have at least one whitespace at the end.

E{n} Matches exactly n occurrences of the expression. This is the same as repeating theexpression n times. For example, x{5} is the same as xxxxx. It is also the same asE{n,n}, e.g. x{5,5}.

E{n,} Matches at least n occurrences of the expression.E{,m} Matches at most m occurrences of the expression. This is the same as E{0,m}.E{n,m} Matches at least n occurrences of the expression and at most m occurrences of the

expression.() Permits to group expressions into sub-expressions.| Alternative. Example: "aaa|bbb" matches the string "aaa" or "bbb".


Code Coverage Level

19.1.2.2 String substitution

Element Meaning& Matched expression\n sub-expression number n. Example: the regular expression is "(.*):([0-9]*)" matches the string

"joe:18". The replacement string "\1 is \2" will produce the result: "joe is 18"

19.2 Code/Test Coverage Level

The menu entry "InstrumentationÕLevel:x" permits to set the targeted code coverage count or, if the compiledwith instrumentation hit support1, the targeted test coverage count.

The level is corresponding of the number of code/test coverage count necessary to consider that an instrumented codeis executed.Example: Setting the level to 10, will made necessary to execute 10 times the each line of the source code if compiledwith code coverage count. If compiled with code coverage hit, 10 execution runs need to execute each lines of thesource code. The menu entry "ToolsÕTest Coverage Count Mode" and the button 1.2.3. permits to switch betweencode coverage count and test coverage count analysis. This simulates the behaviour of the compilation with codecoverage hit support2 when the project is compiled with code coverage count support3.

19.3 Code Coverage Algorithm

CoverageBrowser displays the code coverage analysis (statement block, decision or condition) generated be Coverage-Scanner. But "InstrumentationÕCoverage MethodÕStatement Block" allows to reduce the analysis to thecode coverage of statement blocks. This produces the same result as compiling with the --cs-statement-block ofCoverageScanner. "InstrumentationÕCoverage MethodÕDecision" allows to show the code coverage analysisat the level defined at the compilation.

Here is a short overview of the command line options necessary for each code coverage analysis method:

Coverage analysis CoverageScanner command line optionStatement Block --cs-statement-block

Decision with full instrumentation --cs-decision

Decision with partial instrumentation --cs-decision --cs-partial-instrumentation

Condition with full instrumentation (default)Condition with partial instrumentation --cs-partial-instrumentation

1Command line option --cs-hit from CoverageScanner. To compile with code coverage count support, the option --cs-count must beadded in the command line.

2CoverageScanner command line option --cs-hit3CoverageScanner command line option --cs-count


Optimized execution order

19.4 Optimized execution order

CoverageBrowser can calculate an execution order for tests with which high code coverage can be reached fast with asmall number of tests.

In this execution order, the test with the highest coverage comes first. The second test is the that one which makes theadditional code coverage as high as possible, and so on.

This feature is meant for cases where the full test suite cannot be executed, e.g. because there is not enough time orthere are many manual tests. Then one can run a number of tests from the beginning of the list, say the first 20, andstill get a high coverage fast.

To calculate the execution order proceed as follows:

1. Select a set of executions in the "Executions" window.

2. Click on "ViewÕOptimized Execution Order...". The "Optimized Execution Order" window (seeFigure 19.1, page 136) is then displayed.

3. Click on the "Compute" button to start the analysis.

Execution Order

first executions

Execution name

Coverage of the 'n'

Figure 19.1: Optimized execution order

19.5 Bug Location

19.5.1 Theory

To locate a bug, CoverageBrowser simulates the behaviour of a human programmer searching for a single error inthe source code. It simplifies the programmer’s behaviour to a stochastic process that goes from source code line tosource code line. After each step, the process tries to jump to the next better error candidate. After an infinite time,we can then look on the probability that a source code line was chosen as the best location of the failure.

Because instructions in a program are have a strong dependence, CoverageBrowser forms groups from instructionsthat are always executed together, because they cannot be distinguished from code coverage data. The bug locationalgorithms works with these groups of instruction instead of individual statement.

At the beginning of the process, a covered source code line is selected at random. Then we select an other instrumentedsource code line with the following rules:

1. Select a test which covers the current line.

2. Then select the next source line as follows:


Bug Location

(a) If the test had passed, the line that caused the failure is not expected to be among the source lines executedby the selected test. Therefore select any instrumented line that is not executed by this test.

(b) If the test had failed, select any source code line that is executed by this test.

We repeat this process until a set of pertinent source code lines are identified.

An example We will use a trivial example to illustrate how the algorithm works. The following function computesthe inverse of a number:

float inv( float x ){

if ( x != 0 )x = 1 * x ; // <- here is the bug

elsex = 0;

return x;}

The bug itself is easy to understand; a multiplication is used instead of a division.

Our test suite is:Name Test StateINV(0) inv(0)== 0 PassedINV(1) inv(1)== 1 PassedINV(2) inv(2)== 0.5 FailedINV(3) inv(3)== 0.3333333 FailedINV(4) inv(4)== 0.25 Failed

We will now simulate the bug location algorithm step by step.

i The following is a simplified version of the algorithm that Squish Coco uses. It would return the same results asthe actual algorithm, but be to slow in practice. For better precision and better performance, CoverageBrowsertherefore computes the probabilities directly and does not use a sampling method as below.

First we note that it is not possible to distinguish between the lines “if ( x != 0 )” and “return x;” with a test.If one of these lines is executed, the other one is also. We group them together and view them as a single line. Thismeans that if we estimate that these lines are a good error candidate, we cannot determinate which of this line isresponsible of the bug. To simplify the explanation, we omit now the “return;” statement.

The algorithm starts by randomly selecting a source code line as an error candidate. We will use here the line“if ( x != 0 )” as our starting point. The algorithm then searches the list of the tests that execute this line andchooses one at random; assume that it selects INV(2):


Bug Location

if ( x != 0 )

x = 1 * x ;

INV(0)

INV(1)

INV(2)

INV(3)

INV(4)

else

x = 0 ;

Figure 19.2: Bug location of INV example – Step 1

The test INV(2) has failed, and we suppose that one of the source code lines executed by this test is responsible of theerror. The algorithm selects then as error candidate a line that is executed by INV(2); we assume it is “x = 1 * x”:

if ( x != 0 )

x = 1 * x ;

INV(0)

INV(1)

INV(2)

INV(3)

INV(4)

else

x = 0 ;


The algoritm then select randomly the test INV(1) in the set of tests executed by the line “x = 1 * x” (that are:INV(1), INV(2), INV(3) and INV(4)):


Bug Location

if ( x != 0 )

x = 1 * x ;

INV(0)

INV(1)

INV(2)

INV(3)

INV(4)

else

x = 0 ;


The test INV(1) was passed, so we suppose that a source code line which is not executed by this test is responsible ofthe error. We select then “x = 0;” as the next candidate:

if ( x != 0 )

x = 1 * x ;

INV(0)

INV(1)

INV(2)

INV(3)

INV(4)

else

x = 0 ;


We iterate then the process infinitely and compute the probabilities that a source line is chosen as error candidate:Line Probability

x = 1 * x 0.528if ( x != 0 ) 0.283

x = 0; 0.188As expected, the line “x = 1 * x” has the highest probability of having a bug.


Patch Analysis

19.5.2 Usage

To calculate the bug location proceed as follows:

1. Select a set of executions in the "Executions" window. At least one execution should have failed.

2. Click on "ViewÕBug Location...". The window (see Figure 19.6, page 140) will be displayed.

3. Click on the "Compute" button to start the analysis.

Rank

source code

Source code fragment

Location in the

Figure 19.6: Bug Location

19.6 Patch Analysis

With the menu entry "ToolsÕPatch File Analysis..." one can generate a report about the influence of a patchon the test coverage of a project, without running the test suite for the patched version.

Prerequisites are a project for which .csmes and .csexe files exist and are loaded into CoverageBrowser, and adiff file. Patch analysis works best with programs that have automatic tests and which are instrumented in such away that the names of the tests are known to Squish Coco (see Chapter 30.1, page 172). Line coverage (--cs-line)and statement block coverage (--cs-statement-block) should not be disabled. (They are on by default.)

The diff file must be in the “unified” format. It is generated by the Linux™ diff utility with the option -u, and isalso the default output format of several version control systems (see Chapter 33, page 187).

Clicking the menu entry "ToolsÕPatch File Analysis..." lets the dialog "Patch File Analysis" appear. Ithas the following entries:

Title:The title of the report, both for HTML and CSV.

Patch File:Path of the patch file that contains the changes to the project.

Output:To select the output file and its type:

Type: Either HTML or the Excel CSV format.

The field to the right of the "Type" field contains the name and path of the report file that is generated.

Source Code Parameter:For HTML reports, to select the display of the annotated source code:



Comparing Releases

Tooltip Maximal Size: The annotated patch file in the HTML report has a tool tip which displays the tests thathave executed a certain line of code. This parameter sets the maximal number of tests that can appearin a tooltip. With a value of 0, no tooltips appear.

CSV Parameter:To select the format of the CSV report:

Column Separator: The column separator symbol can be either a comma or a semicolon.

Sections:To choose the content of the report:

Execution Statistics: Create a table that groups the tests by their results. It shows how many of the tests havepassed, failed, need manual testing, and whose execution status is unknown.

Executions: Create a list of the tests that execute code which is affected by the patch. For each test, the nameand the execution result is shown.

Source Code Statistics: Create a table that shows the influence of the patch on the test coverage. It showshow many lines in the patch are covered by a test, how many are not, and for how many lines SquishCoco could not find out whether they are covered. These numbers are shown for the lines that wereremoved, or added, and for all lines of the patch.

Annotated Patch Source: Create an annotated version of the patch file. Each line of code in the patch isshown in red if it is removed, green if it is added, or otherwise gray. Also shown are the line numbersof the code lines, both before and after the patch is applied, and the number of tests that cover a line.This last field also has a tooltip that shows which tests cover the specific line. (The tooltip is onlyvisible if "Tooltip Maximal Size" is set to a non-zero value.)

To generate the report, one clicks either the button "OK" or "Show": The second button also opens a browser windowto show the generated report. Clicking "Apply" saves the values of the dialog entries without generating a report,while "Cancel" closes the dialog without saving anything.

19.7 Comparing Code Coverage of Two Software Releases

CoverageBrowser is able to instrumentation database together in order to:

1. check is the modified/unmodified code is correctly tested.

2. find which tests are impacted by a source code modification.

This feature is particularly adapted to compare two releases together which contains small modifications (bug fixesonly) and to limit the tests of the modified code only.

In this mode CoverageBrowser uses the following typographic rules:

Rule Source Window Method List Source List Execution ListNormal font Identical4 source

partIdentical4 methods Identical4 files Executions available

in both releasesBold Modified methods Modified filesBold+Underline New text inserted New methods New files New executionsBold+Strike Deleted text Deleted methods Deleted files Missing executions

CoverageBrowser comparison and difference algorithm is particularly designed for C and C++ source code andignore white spaces and modifications in comments.


Changing the Instrumentation Database

19.7.1 Reference Database

The reference database is the base instrumentation database which are used for the comparison. To select it clickon "ToolsÕCompare with..." and select a .csmes database. Switching the working database with the referencedatabase can be performed by clicking on "ToolsÕSwitch databases".

Once the reference file loaded, additional filter possibilities are available in the "Executions", the "Sources" andthe "Methods" window. This filters permits to show/hide, modified, new, deleted or identical procedures and sourcefiles.

The "Executions" window displays a mix between the executions of the reference and the current release:

• stroked executions are only available in the reference release. The statistic computed is corresponding to thecoverage for the reference release. This value can be interpreted as an expected code coverage when this testsget re-executed on the current release.

• underlined executions are new tests.

• executions available in both version are not highlighted.

19.7.2 Coverage analysis of modified/unmodified source code

CoverageBrowser is able to limit the code coverage analysis to the modified (resp. unmodified) functions. Whenselecting the coverage analysis on the modified (resp. unmodified) functions only, CoverageBrowser treat allunmodified (resp. modified) functions as if they are not instrumented. Limiting the code coverage analysis tomodified functions can be a practical way to verify that the new features are tested and to identify the list of testswhich have are impacted by a modification.To limit the code coverage to modified function (resp. unmodified functions) click on "ToolsÕAnalysis ofModified Methods" (resp. "ToolsÕAnalysis on Identical Methods").

19.8 Changing the Instrumentation Database

19.8.1 Merging Instrumentations

Clicking on the menu entry "FileÕMerge with..." permits to import the executions, the source code and theinstrumentations from other .csmes files. Comments and code mark as validated are merged together.

19.8.2 Importing Unit Tests

Clicking on the menu entry "FileÕImport Unit Tests..." permits to import the execution report of unit testsinto the current application. Only execution reports of source file present into the main application are imported,instrumentation of other source files (for example test code) are ignored.


Changing the Instrumentation Database

19.8.3 Importing Reviewer Comments

Clicking on the menu entry "FileÕImport Reviewer Comments..." permits to import comments and manualvalidations of a previous version of the current instrumentation database. Comments and manual validations ofunmodified functions will be imported even if if the source code is modified.


Generation of Reports

Chapter

20Generation of Reports


20.1 HTML/CSV Report

The menu entry "ReportsÕGenerate HTML/CSV Report..." allows to export code coverage statistics (per meth-ods, source files, executions, . . . ) of the selected executions in HTML format. It allows also to list the manuallyvalidated and unexecuted code parts.

20.2 EMMA-XML Report

Selecting the menu entry "ReportsÕExport Statistics in EMMA-XML Format..." allows to export code cov-erage statistics of in EMMA-XML format. The output contains global statistics in a format that is compatible with EMMA.This allows to use Squish Coco in tools that provide support for EMMA, notably to give an easy way to use SquishCoco with continuous integration servers like Hudson.

EMMA defines four categories for coverage: classes, methods, blocks, and lines. With Squish Coco they have thefollowing meaning:

EMMA category Meaningclasses A class is considered executed if one of its method is called. Code which is not in a class is

located in the class "" (empty).methods A method is covered if it was called.blocks Code coverage at statement block level.lines Line code coverage (if compiled with line coverage support).conditions Decision/Condition coverage (if compiled with Decision/Condition coverage support).






http://hudson-ci.org/



Cobertura XML Report

20.3 Cobertura XML Report

The menu entry "ReportsÕExport Statistics in Cobertura-XML Format..." allows to export code cover-age statistics in Cobertura-XML format. The output contains global statistics in a format that is compatible withCobertura. This allows to use Squish Coco in tools that provide support for Cobertura, notably giving an easy wayto use Squish Coco with continuous integration servers like Hudson and SonarQube.

The statistics in the Cobertura report is computed a little bit differently from the usual way due to the limitations ofthe report format. The report is in fact a combination of line and condition coverage.

Every statement block, decision and condition instrumentation is counted as a Cobertura-condition. The end of afunction is e.g. marked as one condition to fulfill.

The report format also requires that each line with a condition is also instrumented at line coverage level. This is notalways the case (e.g. with empty functions), so in this case we add artificially some instrumentation values. Finally, ifan instrumented statement is written in more than one line, it is necessary to generate instrumentation data for eachline.

All this has an impact on the computation of the statistics for classes, methods and sources. But the resulting valuesare comparable to those in the other report formats.

20.4 JUnit Report

The menu entry "ReportsÕExport JUnit Report..." allows to export the test result as JUnit report. This reportdoes not contain coverage data and only list the test execution result (i.e. “passed” or “failed”) for each test item.

20.5 Text Report

Clicking on the menu entry "ReportsÕGenerate Text Report..." allows to generate a small text report in theform of one line per executed/unexecuted item. A distinct line format can be specified for executed or unexecutedlines.

The following escape sequences are recognized:

%f: Absolute source code file name

%r: Relative to the current build directory source code file name

%l: Line number

%c: Column number

%m: Explanation

Example: Setting the field "Unexecuted Code Fragments" to "%f:%l: %m" will create a text file which contains allunexecuted code parts. Each line will look like as follows:

foo.cpp:55: Unexecuted: ’return;’


http://hudson-ci.org/

http://www.sonarqube.org/

Preferences

Chapter

21Preferences

21.1 Save/Load Project

"Save/Restore window position":If this option is selected, the position of all windows and toolbars will be restored upon application restart.

"Reload automatically the last project":If this option is selected, the last project opened will automatically be reloaded upon application restart.

"Saves project automatically on exit":Saves the project file automatically without asking on application exit.

21.2 Comments

"Minimum Comment Size":The minimum comment size, is the minimum length requested for a comment.

"Do not request a comment when setting an item to the ’manually validated’ state":This option is used to allow the user to manually modify the state of an instrumentation without entering acomment.

i Enabling this option should be avoided because modifying the state of an instrumentation should beperformed with a valid reason which should be recorded as a comment.

21.3 Thresholds

Thresholds are trigger values that control the background color of:

• the instrumented source files in in the "Sources" window.


Cache

• the instrumented classes or namespaces in the "Methods" window.

• the instrumented functions, methods or procedures in the "Methods" window.

Description:

"Medium/High Coverage Level":If the statistic is above this value, the background color is set to green. Otherwise, the color is orange.

"Low/Medium Coverage Level":If the statistic is below this value, the background color is set to red. Otherwise, the color is orange.

21.4 Cache

Description:

"Execution":Maximum number of executions loaded into the RAM.

"Source":Maximum number of source files loaded into the RAM.


Part VIII Other Coco Tools


cmreport – Code coverage report generation

Chapter

22cmreport – Code coverage report generation

cmreport is a utility which permits to generate Text, HTML, XML or CSV reports from an instrumentation database (a .csmes file). It generates exactly the same reports as these generated by CoverageBrowser (see Chapter 20,page 144).

Example: The simplest use case is the generation of an HTML report from a single instrumentation database. If thename of the file is project.csmes, we may write:

$ cmreport --csmes=project.csmes --html=report

This command creates a file report.html and a directory report_html. The file report.html is the main page ofthe report and can be opened with the browser, while report_html contains all the other files. Most of the othercommand line options just modify the content of such a report.

Syntax:

cmreport -m 〈csmes_file〉 ...

Where:

-m 〈file_name〉 | --csmes=〈file_name〉:Set the name of the main .csmes file (mandatory).

-p 〈file_name〉 | --patch=〈file_name〉:Enable patch/diff file analysis. 〈file_name〉 is the name of a file im "unified diff" format (see Chapter 33,page 187). The file must contain the differences between the files in the directory tree covered by the .csmesfile and a modified version of the same directory tree. Patch reports can be generated in the CSV and in theHTML format.

With the option --section (see Chapter 22.4, page 152) one can select which sections of the report aredisplayed.

--source-type=〈argument〉:Source type used for the analysis. Possible values:

preprocessed: The preprocessed code is analyzed.

original: The original source code is analyzed. C macros are ignored in this mode.

-l 〈number〉 | --level=〈number〉:Set the code coverage level. 〈number〉 must be an integer greater than 0.


File selection options

--max-threads=〈number〉:Maximal number of threads used for the report computation. By default, there as many threads as there areCPUs on the system.

--stat:Print global coverage statistics to the standard output

--max-lines-per-page=〈number〉:Maximal number of lines per page for HTML tables.

22.1 File selection options

The following options allow to specify the source files shown in the report.

--include-file-abs-regex=〈regex〉:Include the source files that match the regular expression 〈regex〉 in the report.

--exclude-file-abs-regex=〈regex〉:Exclude the source files that match the regular expression 〈regex〉 from the report.

--include-file-abs-wildcard=〈wildcard〉:Include the source files that match the wildcard pattern 〈wildcard〉 in the report.

--exclude-file-abs-wildcard=〈wildcard〉:Exclude the source files that match the wildcard pattern 〈wildcard〉 from the report.

If two or more of these options match a file, the last one decides whether it is included or excluded. For the syntax ofregular expressions and wildcard patterns see Chapter 19.1, page 133.

22.2 Execution selection options

The following options allow to specify the executions that occur in the report.

-s 〈regex〉 | --select=〈regex〉:Select executions using regular expression. If not used, all executions are implicitly selected.

-d 〈regex〉 | --deselect=〈regex〉:Deselect executions using regular expression

--deselect-to-check:Deselect all executions with the status “CHECK_MANUALLY”

--select-passed:Select all executions with the status “PASSED”

--deselect-passed:Deselect all executions with the status “PASSED”

--select-failed:Select all executions with the status “FAILED”


Release comparison

--deselect-failed:Deselect all executions with the status “FAILED”

--select-unknown:Select all executions with unknown status

--deselect-unknown:Deselect all executions with unknown status

--executions-reference-from-csmes=〈argument〉:Select all executions present in the specified .csmes file.

22.3 Release comparison

With the following options, one can compare the coverage of two releases.

-mr 〈file_name〉 | --csmes-reference=〈file_name〉:Set the name of the .csmes reference file. With this option, cmreport compares two software releases.

--release-comparison=〈argument〉:Comparison mode when comparing two software releases. Possible values:

none: Display only the differences between two software releases.

modified_functions: Display code coverage only for modified functions.

unmodified_functions: Display code coverage only for identical functions.

-sr 〈regex〉 | --select-reference=〈regex〉:Select reference executions using regular expression. This option activates the comparison of executions.

-dd 〈regex〉 | --deselect-reference=〈regex〉:Deselect reference executions using regular expression. This option activates the comparison of executions.

--deselect-to-check-reference:Deselect all reference executions with the status “CHECK_MANUALLY”

--select-passed-reference:Select all reference executions with the status “PASSED”

--deselect-passed-reference:Deselect all reference executions with the status “PASSED”

--select-failed-reference:Select all reference executions with the status “FAILED”

--deselect-failed-reference:Deselect all reference executions with the status “FAILED”

--select-unknown-reference:Select all reference executions with unknown status

--deselect-unknown-reference:Deselect all reference executions with unknown status


Options for HTML or CSV reports

22.4 Options for HTML or CSV reports

Command line arguments:

-h 〈file_name〉 | --html=〈file_name〉:Browsable HTML report output file name

--html-single=〈file_name〉:HTML report output file name (single file)

--csv-excel=〈file_name〉:CSV report (optimized for Excel and OpenOffice) output file name

--execution-level=〈argument〉:Code coverage level for executions

--css=〈file_name〉:CSS style sheet

--icon=〈file_name〉:Icon

--title=〈file_name〉:Title

--section=〈section_name〉:Select the sections of the generated report. This option can be repeated to select more than one section type. Ifnothing is selected, cmreport generates all relevant sections automatically.

The following sections are possible:

global: Global statistics

function: Function statistics

function-tree: Function tree with statistics

class: Class/namespace statistics

execution: Execution statistics

source: Source file statistics

source-tree: Source tree with statistics

directory: Directory file statistics

manually-validated: Manually validated code fragments

unexecuted: Unexecuted code fragments

dead-code: Dead code fragments

executed: Executed code fragments

If the option --patch is used, the following sections are also possible.

patch-execution-statistic: Global execution statistics of a patch

patch-source-statistic: Global source code statistics of a patch

patch-execution: Executions impacted by a patch

patch-source: Annotated patch file

For the content of these sections, see Chapter 19.6. By default, if no --section option is set, all them occur inthe report.


Options for HTML or CSV reports

--global-threshold-low-medium=〈num〉:Global threshold setting

--global-threshold-medium-high=〈num〉:Global threshold setting

--source-threshold-low-medium=〈num〉:Source/directory threshold setting

--source-threshold-medium-high=〈num〉:Source/directory threshold setting

--function-threshold-low-medium=〈num〉:Function/class/namespace threshold setting

--function-threshold-medium-high=〈num〉:Function/class/namespace threshold setting

-b | --coverage-statement-block:Code coverage of statement blocks only

--coverage-condition:Code coverage at decision/condition level

--coverage-mcc:Code coverage at multiple conditions level

--coverage-mcdc:Code coverage at MC/DC level

--line-coverage:Line coverage analysis

--function-coverage:Function coverage analysis

-t | --test-coverage:Test count mode

-D | --debug:Debug flag

--csv-field-separator=〈char〉:Field separator for a CSV file

--csv-comma=〈char〉:Separator (, or .) used for floats in a CSV file

22.4.1 Obsolete options for CSV reports


--csv-method=〈argument〉:Generate a CSV report file for each method

--csv-source=〈argument〉:Generate a CSV report file for each file


Options for text reports

--csv-field-separator=〈argument〉:Field separator for a CSV file

--csv-comma=〈argument〉:Separator (, or .) used for floats in a CSV file

22.5 Options for text reports


-t 〈file_name〉 | --text=〈file_name〉:Text report output file name

--format-executed=〈argument〉:Line format for the executed for code fragments

--format-unexecuted=〈argument〉:Line format for the unexecuted for code fragments

--format-dead-code=〈argument〉:Line format for dead code fragments

The following escape sequences are recognized:

%f: Absolute source code file name

%r: Relative to the current build directory source code file name

%l: Line number

%c: Column number

%m: Explanation

It is allowed to to set the output file to an empty string (i.e.: --text=), in this case the standard output of the consoleis used.

22.6 Options for EMMA-XML reports

The following options are used if a report in the EMMA-XML format is generated:

--emma=〈filename〉:Generate an EMMA-XML report. 〈filename〉 is the name of the XML file that is generated.

--emma-source-pattern=〈argument〉:Source file pattern for EMMA-XML reports

--emma-package-pattern=〈argument〉:Package name pattern for EMMA-XML reports

The patterns have the following format:







Options for JUnit reports

%f: file name (without path)

%b: file name without path and extension

%e: file extension

%p: file path (without file name)

%P: absolute file path (without file name)

i For Jenkins-CI users, it is not recommended to use the last two parameters. The default values are chosen towork with the EMMA-XML plugin.

22.7 Options for JUnit reports


-j 〈file_name〉 | --junit=〈file_name〉:Generate an JUnit report

JUnit reports does not contain any code coverage information. The report list only the tests executed, their status(passed or failed) and their comments.

22.8 Options for Cobertura reports


--cobertura=〈argument〉:Generate a Cobertura-XML report

--cobertura-source=〈argument〉:Root path of the source code when generating a Cobertura-XML report

--sonarqube-project=〈argument〉:Generate a Cobertura-XML report for a SonarQube project

To generate a Cobertura reports two work flows are supported:

1. Specifying directly all necessary arguments to generate the report which are:

(a) the output file name (--cobertura=〈argument〉 command line option)

(b) the root path of the source directory tree (--cobertura-source=〈argument〉 command line option)

2. Indicating the path of a SonarQube project properties file sonar-project.properties (--sonarqube-project=〈argument〉command line option): in this case, the root path of the source tree is extracted from its contents usingsonar.cxx.coverage.reportPath, sonar.cxx.coverage.itReportPath or sonar.cxx.coverage.overallReportPathproperty.





cmcsexeimport – Command Line Import Utility

Chapter

23cmcsexeimport – Command Line Import Utility

cmcsexeimport is a utility to import an execution report (.csexe file) into an instrumentation data base (.csmes file).It behaves like the menu entry "FileÕLoad Execution Report..." in the CoverageBrowser (see Chapter 18.1.2,page 123).

Syntax:

cmcsexeimport -m 〈csmes_file〉 -t 〈title〉 [〈options〉] 〈csexe_file〉...

where

-m 〈filename〉 | --csmes=〈filename〉:Set the name of the .csmes file into which the .csexe files are imported. This parameter is mandatory.

-t 〈string〉 | --title=〈string〉:Set a title for these executions. This parameter is mandatory.

〈csexe_file〉:Name of one of the .csexe files that should be imported.

〈options〉:Any of:

-e 〈filename〉 | --csexe=〈filename〉: Set the name of a .csexe file to be imported. This argument can berepeated to process more than one file.

-l 〈filename〉 | --csexe-list=〈filename〉: Read the list of .csmes files to import from a text file. The textfile must have one file name per line, without leading or trailing spaces.

-p 〈policy〉 | --policy=〈policy〉: Set the import policy. The possible import policies are:ignore_duplicates: Executions are ignored if they have executed the same code as an execution

that was already imported. (Default.)import_duplicates: Executions are imported if at least one instrumented source code line is

executed.import_duplicates_and_empty: All Exections are imported.merge: All executions with the same name are merged. The execution counts of all merged

instrumentations are added.

-P | --passed: Mark the executions in the imported files as “PASSED”.

-F | --failed: Mark the executions in the imported files as “FAILED”.


cmcsexeimport – Command Line Import Utility

-C | --check-manually: Mark the executions in the imported files as “CHECK_MANUALLY”.

-f 〈num〉 | --csmes-save-frequency=〈num〉: After every 〈num〉 exection reports have been read, thecurrent state of the instrumentation database is written to disk. Without this option, the it is writtento the disk only at the end.

-d | --delete: Delete the imported files after processing them.

-v | --verbose: Write progress messages to the standard error output (stderr).

-D | --debug: Write debugging information to the standard error output (stderr).


cmmerge – Merging Utility

Chapter

24cmmerge – Merging Utility

cmmerge is a small utility which permits to merge several instrumentation databases (.csmes file) together. It behaveslike the CoverageBrowser menu entry "FileÕMerge with..." (see Chapter 19.8.1, page 142).

Syntax:

cmmerge [〈options〉] -o 〈outfile〉〈infile〉...cmmerge --blackbox -o 〈outfile〉〈infile〉

where:

-o 〈outfile〉 | --output=〈outfile〉:.csmes output file. (Required.)

〈infile〉:.csmes input file. Only one input file is allowed if --blackbox is selected.

-b | --blackbox:Generate a .csmes file for black-box testing.

〈options〉:Any of:

-a | --append: Merge the input file(s) with the existing content of the output file. 〈outfile〉 must already exist.

-i 〈reference〉 | --instrumentation-and-execution=〈reference〉: Include only execution results that be-long to 〈reference〉 (which is a .csmes file).With this option, cmmerge merges 〈reference〉 with the 〈infile〉s, but includes the instrumentationsand executions of an 〈infile〉 only if they also occur in 〈reference〉. If --append is set, 〈outfile〉 isviewed as a part of 〈reference〉.This option is useful for unit tests, where the 〈reference〉 is the program for which the test is written,while the 〈infile〉s contain the execution results of the unit tests.

-r 〈reference〉 | --reviews-only=〈reference〉: Merge 〈reference〉 with the comments and manual vali-dations in the 〈infile〉s. Only validations and comments of unmodified functions are imported.(〈reference〉 is a .csmes file too).

--delete: Delete the input files from disk after importing them.

-v | --verbose: Verbose output.


cocolic – License activation

Chapter

25cocolic – License activation

Command line alternative to cocolicwizard. The program can be used to activate and configure the Squish Cocolicense on a computer without a graphical interface.

Syntax:

cocolic 〈arguments〉

Where:

--fetch-license-key=〈code〉:Use an activation code to retrieve the license key for a node-locked license. The activation code is provided byfroglogic.

For this, cocolic needs a network connection to froglogic. It can also be configured to connect via a proxyserver (see below).

--license-key=〈key〉:Set the key for a node-locked license. The license key is generated by froglogic from the output of cocolic --machine-identifier.

--machine-identifier:Display the machine identifier on the standard output.

--license-server=〈server〉:〈port〉:Set the address and the port of the license sever for a floating license. The port can be omitted; then the defaultport of 49344 is used.

--check:Check the license validity. The program prints a message about the status of the license to stderr. It thenexits with a return value of 0 if the license is valid, or 1 if it is invalid.

--http-proxy=〈server〉:〈port〉:Use a HTTP proxy sever to retrieve the license key.

--socks5-proxy=〈server〉:〈port〉:Use a SOCKS5 proxy sever to retrieve the license key.

--proxy-user=〈name〉:Set the user name for the proxy server.


cocolic – License activation

--proxy-password=〈password〉:Set the password for the proxy server.

--no-proxy:Disable the use of a proxy server.

--disable-node-locked:Disable the node locked license. This option is meaningful if a node-locked license and a license server use arepresent on the same machine. Squish Coco will then use the license server, but the node-locked license is stillpresent and can be enabled later.

--enable-node-locked:Re-enable a previously disabled node locked license. Do nothing if the license is already enabled.


cocolicwizard – GUI for license activation

Chapter

26cocolicwizard – GUI for license activation

cocolicwizard provides a graphical user interface to configure the license. It allows to enter a license activation code,send an activation email, to enter a license key or to configure the connection to a license server. The program alsodisplays information about the currently active license.

Syntax:

cocolicwizard [--check]

Where:

--check:check whether there is a valid license. Show the GUI window only if there is no valid license, otherwise donothing.


cocolicserver – License Server

Chapter

27cocolicserver – License Server

cocolicserver can be used to serve Squish Coco licenses from a central computer; clients can connect to the serverand cocolicserver will make sure that no more than the available number of licenses will be used concurrently.

Syntax:

cocolicserver 〈options〉

where:

-s | --server-identifier:Print the identifier of the machine on which cocolicserver should run. This identifier must then be sent tofroglogic to get a license file (see Chapter 2.2.1, page 9).

-c 〈path〉 | --config=〈path〉:Path to the configuration file (provided by froglogic).

-l 〈path〉 | --logfile=〈path〉:Write log messages to the file at 〈path〉. The file name may contain the following escape sequences:

%Y: current year, in four digits,

%M: current month number, in two digits,

%D: current day, in two digits,

%h: current hour, in two digits,

%m: current minute, in two digits,

%s: current second, in two digits.

New log files will be opened according to the values of the escape sequences. This means that if 〈path〉is e.g. cocoserver_%Y_%M_%D.log, every day a new log file is created, and the files have names likecocoserver_2016_01_01.log, cocoserver_2016_01_02.log, etc.

The following options are available only on Linux™and macOS:

--syslog:Write log messages to the UNIX® system logging service.

--daemon:On Linux™and macOS: Launch cocolicserver as a daemon background process.

The following options are available only on Microsoft® Windows:


cocolicserver – License Server

--service-install:Install cocolicserver as a Windows service that starts automatically at boot time.

For this option cocolicserver needs administrator privileges. It must be run in the form

C:\somedir> cocolicserver --service-install 〈options〉

Then the service is installed and starts automatically. It will run as if called directly in the form cocolicserver〈options〉. The visible name of the server is “Squish Coco License Server”.

If the server is already installed as a service, the 〈options〉 are updated. The new options have effect only afterthe service is restarted.

--service-uninstall:Uninstalls the cocolicserver Windows service. For this option cocolicserver needs administrator privileges.


Part IX Howtos


Creating an instrumented project

Chapter

28Creating an instrumented project

This chapter contains recipes how a software project can be set up for instrumentation. The methods depend onthe languages and the development environments. Here we show how to create a new software project so that it isprepared for instrumentation. This is another simple way to get acquainted with Squish Coco.

A description of a larger project can be found in Chapter 4, page 23.

28.1 C++ on Microsoft® Visual Studio® using the Microsoft®

Visual Studio® Add-In

There is an add-in for Microsoft® Visual Studio® (see Chapter 34.10.3, page 206) that supports the work with SquishCoco. The following description shows the setup of a project that uses this add-in.

Start Microsoft® Visual Studio® and create a new C++ application:

1. Click on "FileÕNewÕProject..." to pop up the new project wizard.

2. Choose a project type of "Visual C++ÕWin32" and the "Win32 Console Application" template.

3. Enter a project name of squishcoco_sample, then click the "OK" button.

4. When the wizard’s second page appears, click the "Finish" button.

At this stage the application is not yet instrumented. So now we will create a copy of the build and activate theinstrumentation:

1. Open the configuration manager by clicking "BuildÕConfiguration Manager...".

2. In the "Configuration" column, select "New..." in the combobox

3. In the "New Project Configuration" dialog:

(a) Enter Code Coverage in the "Name" field,

(b) Select Release or Debug in the "Copy settings from" selection dialog.

(c) Click the "OK" button.

To activate the instrumentation, use the Microsoft® Visual Studio® Add-In:


C# on Microsoft Visual Studio

1. Click "ToolsÕCode Coverage Build Mode..." to pop up the Squish Coco wizard.

2. In the "Project:" selection dialog, select squishcoco_sample.

3. In the selection dialog "Configuration:", select Code Coverage.

4. In the Configuration section at the bottom, select the radio button "Modify", and then click on the button,"Enable code coverage for C++ projects".

The Code Coverage configuration has now been modified to generate code coverage information. The "SquishCoco"output window summarizes all the modifications that have been made:

...Modifying configuration ’Code Coverage’ for the project

’squishcoco_test’ for the platform ’Code Coverage|Win32’Compiler ConfigurationAdditional command line arguments

’--cs-on --cs-condition --cs-count --cs-partial-instrumentation’are appended

Linker ConfigurationAdditional command line arguments

’--cs-on --cs-condition --cs-count --cs-partial-instrumentation--cs-libgen=/MD’ are appended

...

Build the squishcoco_sample project. This will cause the executable squishcoco_sample.exe to be builtand the code coverage instrumentation file squishcoco_sample.exe.csmes to be generated. Double click onsquishcoco_sample.exe.csmes to inspect this file in CoverageBrowser.

Right now there is no code coverage statistics visible in CoverageBrowser: This is because the application hasnot yet been executed. Click on squishcoco_sample.cpp in the source list to display the main function. All theinstrumented lines are shown grayed out, to indicate that nothing has been executed.

Now execute squishcoco_sample.exe by double clicking it. This will result in a file called squishcoco_sample.exe.csexe being generated. The file contains a code coverage snapshot which can be imported into Coverage-Browser:

1. Click "FileÕLoad Execution Report...".

2. Select the "File" item and enter the path of the squishcoco_sample.exe.csexe file.

3. Click on the "Import" button.

This will cause the code coverage statistics to be updated. Now, in the source code window, the main function’sreturn statement will be colored green to indicate that this line has been executed.

28.2 C# on Microsoft® Visual Studio®

Start Microsoft® Visual Studio® and create a new C# application:

1. Click on "FileÕNewÕProject..." to pop up the new project wizard.

2. Choose a project type of "Visual C#ÕWindows" and the "Console Application" template.

3. Enter a project name of squishcoco_sample, then click the "OK" button.


Tcl

4. When the wizard’s second page appears, click the "Finish" button.

To activate the instrumentation, use the project properties:

1. In the "Solution Explorer", double click "squishcoco_sampleÕProperties" to open the property dia-log.

2. Click on the "Build" tab.

3. Select "Conditional compilation symbols" and enter COVERAGESCANNER_COVERAGE_ON.

The code coverage analysis is enabled when the symbol COVERAGESCANNER_COVERAGE_ON is defined from thecommand line.

Build the squish_coco project. This will cause the executable squishcoco_sample.exe to be built and the codecoverage instrumentation file squishcoco_sample.exe.csmes to be generated. Double click on squishcoco_sample.exe.csmes to inspect this file in CoverageBrowser.

Right now there are no code coverage statistics to be seen in CoverageBrowser: this is because the application hasnot yet been executed. Click on Program.cs in the source list to display the main function. All the instrumentedlines are shown grayed out to indicate that nothing has been executed.

Now execute squishcoco_sample.exe by double clicking it. This will result in a file called squishcoco_sample.exe.csexe being generated. This file contains a code coverage snapshot which can be imported into Coverage-Browser:

1. Click "FileÕLoad Execution Report...".

2. Select the "File" item and enter the path of the squishcoco_sample.exe.csexe file.

3. Click on the "Import" button.

This will cause the code coverage statistics to be updated. Now, in the source code window, the main function’sreturn statement will be colored green to indicate that this line has been executed.

28.3 Tcl

i At the moment, this example only works with Tcl v8.5 or earlier. See Section 28.3.1, page 168 below fordetails.

Create a file hello.tcl with the following content:

if { $argc == 0 } {puts "Hello World!"

} else {for {set i 0} {$i<$argc} {incr i} {

if { $i == 0 } {set hello "Hello [lindex $argv $i]"

} elseif { $i == $argc - 1 } {set hello "$hello and [lindex $argv $i]"

} else {set hello "$hello, [lindex $argv $i]"

}}


Command Line Tools

puts $hello}

Execute hello.tcl using the tclsh interpreter:

$ coveragescannertcl tclsh hello.tcl Jim John James

Two files are generated:

tclsh.csmes:The instrumentation database of hello.tcl

tclsh.csexe:The execution report.

To import the execution report into its instrumentation database, execute:

$ cmcsexeimport -m tclsh.csmes --title="Jim John James" tclsh.csexe

After the import, it is possible to analyze the code coverage data directly with CoverageBrowser or generate a HTMLreport:

$ cmreport --title="Hello application" -m tclsh.csmes --html=tclsh.html

28.3.1 Using more than one Tcl version on one system

Squish Coco v3.0 does not yet support the use of tclsh under Tcl v8.6. If you want to use coveragescannertclwith tclsh, Tcl v8.5 or earlier is necessary.

In some UNIX®systems (like Debian Linux™), more than one Tcl version can be present at the same time. Theexecutables are then distinguished by their names. On such a system, the tclsh executable of Tcl v8.5 may be calledtclsh8.5.

We can then run CoverageScannerTcl such that it calls tclsh8.5. On its own, this would however change the namesof the generated files into tclsh8.5.csmes and tclsh8.5.csexe. We can correct this with another commandlineoption, --cs-output=tclsh. The first command in the example above now becomes:

$ coveragescannertcl --cs-output=tclsh tclsh8.5 hello.tcl Jim John James

Afterwards, the rest of the example can be done as before.

28.4 Command Line Tools

Open a console window (MS-DOS Prompt or Command Prompt window on Windows) and make sure that theMicrosoft® Visual Studio® compiler (cl.exe) or GCC is installed on your system.

Create a simple hello.c source file that contains the following code:


Command Line Tools

#include <stdio.h>

int main(int argc, char *argv[]){

printf("hello\n");return 0;

}

Compile this program using the CoverageScanner compiler wrapper instead of the native compiler. This is done byprepending cs to the command line compiler’s name:

With Microsoft® Visual Studio® With GCC$ cscl.exe hello.c /Fehello.exe $ csgcc hello.c -o hello.exe

Then the executable hello.exe and the file hello.exe.csmes, which contains the code coverage instrumentation,will be generated.

Execute hello.exe; this will cause the file hello.exe.csexe to be generated. It contains a code coverage snapshotwhich can be imported into hello.exe.csmes using cmcsexeimport:

$ cmcsexeimport --title="Hello execution" -m hello.exe.csmes -e hello.exe.csexe

Once imported, the hello.exe.csexe file is no longer needed and can be deleted. Now hello.exe.csmes containsa single execution record. We can generate an HTML report with cmreport:

$ cmreport --title="Hello application" -m hello.exe.csmes --html=hello.html

Here is a summary of the command line options that were used in this command:

--title="Hello application":Set a title for the report. It appears on all generated HTML pages.

-m hello.exe.csmes:Select the instrumentation database.

–html=hello.html:Specify the name of the HTML file that the report should be written to.


Generating instrumentations without modifying projects

Chapter

29Generating instrumentations without modifyingprojects

Squish Coco can also generate code coverage information for a project without the need to modify it. The principle isto prepend to the PATH variable the path of the CoverageScanner compiler wrappers and to set the instrumentationparameters with the COVERAGESCANNER_ARGS environment variable. To activate the instrumentation, --cs-on mustbe present in COVERAGESCANNER_ARGS. If this is not the case, CoverageScanner is completely deactivated.

! The variable COVERAGESCANNER_ARGS should only be set locally, e.g. in a script or on the commandline. If it is set globally, it will influence every build.

29.1 GNU Make

On a UNIX® system, proceed as follows to instrument a project which can be generated using GNU Make:

export PATH=/opt/SquishCoco/wrapper/bin:$PATHexport COVERAGESCANNER_ARGS=--cs-onmake cleanmake

For macOS, replace the first line with

export PATH=/opt/SquishCoco/wrapper:$PATH

29.2 Microsoft® NMake

Proceed as follows to instrument a project that is generated with NMake:

set PATH=%SQUISHCOCO%\visualstudio;%PATH%


Microsoft Visual Studio

set COVERAGESCANNER_ARGS=--cs-onnmake cleannmake

29.3 Microsoft® Visual Studio®

Proceed as follows to instrument a project that is generated with Microsoft® Visual Studio®:

set PATH=%SQUISHCOCO%\visualstudio;%PATH%set COVERAGESCANNER_ARGS=--cs-ondevenv /useenv myproject.sln /Rebuild

29.4 Microsoft® MSBuild

Proceed as follows to instrument a project that is generated with Microsoft® MSBuild

set PATH=%SQUISHCOCO%\visualstudio;%PATH%set COVERAGESCANNER_ARGS=--cs-onmsbuild /p:UseEnv=true myproject.sln /t:ReBuild

29.5 Mono C# XBuild

Proceed as follows to instrument a project that is generated with Mono C# XBuild

export COVERAGESCANNER_ARGS=--cs-onmsbuild \

/p:UseEnv=true \/p:UseHostCompilerIfAvailable=true \/p:CscToolPath="${SQUISHCOCO}/wrapper" \myproject.sln /t:ReBuild

The environment variable SQUISHCOCO contains the path to the CoverageScanner executable.


Test suites and Squish Coco

Chapter

30Test suites and Squish Coco

30.1 Execution comment, name and status

Test suites are able to provide to CoverageBrowser the name of the test, a comment in HTML format and also thestatus of their execution (passed or failed). Two methods are possible:

• The functions __coveragescanner_testname(), __coveragescanner_add_html_comment() and __coveragescanner_teststate()can be used to specify the test name, its comment and its status. This is particularly useful for unit tests inwhich the test framework is compiled into the application to test (see Chapter 9.1.2, page 74).

• One can write these data directly into the .csexe file. (See below.)

The .csexe file is a simple text file on which is it possible to append additional lines to extend the informationprovided to CoverageBrowser.

To set the name of a test, simply add before executing it the following line: “*〈name of the test〉\n”The character “*” must be the first character of the line. The name of the test is placed directly after it and the line isterminated with a carriage return.1

To set the status of a test, simply add after executing it the following line: “!〈status〉\n”The character “!” must be the first character of the line. The status of the test is placed directly after it and the line isterminated with a carriage return. The status can be one of the following strings:

"PASSED":to indicate that the test was successfully executed,

"FAILED":to indicate that the test was not successfully passed, and

"CHECK_MANUALLY":to indicate that it was not possible to determinate whether the test was successfully executed.

To append an execution comment, insert the contents of an HTML document just after the execution of the application.

Example: The following batch file execute the test First Test and set the status of the execution to CHECK_MANUALLY.

1It is also possible to begin the line with a paragraph sign (“§”) instead of a “*”. The “§” must be the in Latin-1 encoding and has the numericalvalue of 167. Since this may lead to internationalization problems, we strongly dicourage its use. It is only supported for compatibility with olderversions of the program.


Unit testing

echo *First Test >> myapp.csexemyappecho "<HTML><BODY>Execution of myapp</BODY></HTML>" >> myapp.csexeecho !CHECK_MANUALLY >> myapp.csexe

There may be more than one HTML comment, and the comments and the status declaration may occur in any order,but they all must be appended to the .csexe file after the test has been executed.

30.2 Unit testing

CoverageBrowser imports execution results for all object files that are part of an application. This means that, if anunit test uses the same object files as the application, CoverageBrowser can import the execution results of the unittests and merge them into the code coverage of the compiled application.

Example:

An application consists of three files:

app.cpp:This file contains the main() function of the application.

library.cpp:This file contains functions that are called by app.cpp.

testlibrary.cpp:This file contains the test code for the functions in library.cpp. It has its own main() function.

Under Microsoft® Windows, the application is compiled with the following commands:

cscl app.cpp /Foapp.objcscl library.cpp /Folibrary.objcscl library.obj app.obj /Feapp.exe

The following commands compile the unit test program:

cscl testlibrary.cpp /Fotestlibrary.objcscl library.obj testlibrary.obj /Fetestlibrary.exe

To import the execution report of the unit tests into the instrumentation database of the main application, two methodscan be used:

• The execution of testlibrary.exe generates the execution report testlibrary.exe.csexe. This report isthen loaded into the measurement database app.exe.csmes.

• The execution of testlibrary.exe generates the execution report testlibrary.exe.csexe. This report isthen loaded into the measurement databaseof the test code, testlibrary.exe.csmes. This can be done withthe function "FileÕImport Unit Tests..." of the CoverageBrowser(see Chapter 19.8.2, page 142).

In both cases, only the code coverage analysis of the file library.cpp is loaded and the execution report of the testcode ignored.


Control of execution report generation

Chapter

31Control of execution report generation

By default, the of CoverageScanner library produces an execution report when the instrumented application exits.This might not be enough in the case of unit test suites (where it is preferable to generate a report after each singletest) or for applications like daemons which never terminate.

For these purposes, the CoverageScanner library also allows to:

• embed C code into the instrumented application to control the report generation (30.2, page 173), and

• generate a coverage report when a Microsoft® Windows event or a UNIX® signal has been received.

This chapter is about the second method.

31.1 Execution report generation with UNIX® signals

To enable execution report generation with UNIX® signals, use the option --cs-dump-on-signal=〈sig〉, where〈sig〉 is the number of a signal or its common name (like e.g. SIGUSR1).

With the UNIX® kill command it is possible to trigger the report generation. (ex: kill -SIGUSR1 <pid>)

31.2 Execution report generation with Microsoft® Windowsevents

To enable execution report generation with Microsoft® Windows events, use the option --cs-dump-on-event=〈event〉, where 〈event〉 is a string which identifies the event.

Since there is no standard Windows equivalent to the UNIX® kill command, Squish Coco offers a replacementwhich is described in the following sections.


Execution report generation with Microsoft Windows events

31.2.1 A program to send events to applications

Squish Coco contains a tool to send events to Windows applications, for the use with --cs-dump-on-event. It isneeded not very often, therefore you need to compile it yourself.

The program can be found in the directory 〈Windows Coco〉\dump_on_event. It exists in two versions, one in C++and one in C#. They are called dump_on_event and dump_on_event_cs, respectively.

To compile the program and see how it works, follow the instructions below. They refer to the C++ case – the C#version is quite similar and the differences are listed at the end.

1. Copy the folder 〈Windows Coco〉\dump_on_event to some other place, say to C:\dump_on_event, since theoriginal version is write protected.

2. Double-click on the file build_cpp. This starts the compilation of dump_on_event.cpp and of the exampleapplication event_sample_cpp.cpp. When the file event_sample_cpp.cpp is compiled, it is instrumentedwith the option --cs-dump-on-event=COVERAGE.

3. After the programs have been compiled, three windows become visible.

(a) The window of the CoverageBrowser, with the contents of the file event_sample_cpp.exe.csmesloaded. It contains the source of the program event_sample_cpp, but not yet any coverage data.

(b) A window with a command prompt from which the program dump_on_event can be run later.

(c) A command window in which event_sample_cpp runs. Every second it prints a dot, and it will do sofor five minutes.

4. Now you can send the event COVERAGE to event_sample_cpp. Enter in the command prompt window:

C:\dump_on_event>dump_on_event COVERAGE

When the program event_sample_cpp receives the event, it creates a file event_sample_cpp.exe.csexe,which contains the coverage data that were generated so far.

Send the command a second time. The file event_sample_cpp.exe.csexe then grows, since the newcoverage data are appended to it. There is no danger that the same data are written twice.

5. You can import the execution report from the CoverageBrowser with "FileÕLoad Execution Report...".Each data dump appears as a separate execution, so if you have sent the event twice, you will see two executionsin the executions window of the bowser.

The same recipe also works with the C# versions of the programs. You only need to replace “build_cpp” with“build_cs”, “dump_on_event” with “dump_on_event_cs” and “event_sample_cpp” with “event_sample_cs”.

31.2.2 Global events

In general, Windows events are created for a specific account and not accessible from other accounts on the samemachine. But Windows supports also system-wide events. They have names that begin with Global\.

So to generate a code coverage report with Windows events from an application on a different account, system-wideevents are needed.

In the following example we use the program dump_on_event to generate the event:

1. Compile your application with the argument --cs-dump-on-event=Global\COVERAGE.

2. Start your application.

3. On another account, execute the command



C:\dump_on_event>dump_on_event Global\COVERAGE

The application then writes its coverage report to the disk.

31.2.3 Write your own program

You can also use dump_on_event as an example for sending events from your test setup. It is in fact only a wrapperfor the Microsoft® Windows function SetEvent():

#include <windows.h>#include <stdio.h>#include <strsafe.h>#include <iostream>

void DisplayError( LPTSTR lpszMessage ){

// Retrieve the system error message for the last-error code

LPSTR lpMsgBuf;DWORD dw = GetLastError();

FormatMessageA(FORMAT_MESSAGE_ALLOCATE_BUFFER |FORMAT_MESSAGE_FROM_SYSTEM |FORMAT_MESSAGE_IGNORE_INSERTS,NULL,dw,MAKELANGID( LANG_NEUTRAL, SUBLANG_DEFAULT ),( LPSTR ) &lpMsgBuf,0, NULL );

// Display the error message and exit the processstd::cerr << lpszMessage << " (error " << dw << "): " << lpMsgBuf << ’\n’;

LocalFree( lpMsgBuf );}

int main( int argc, char *argv[] ){

if ( argc != 2 ){

std::cerr << "Usage: " << argv[0] << " <event_name>" << ’\n’;return 1;

}

const char * eventName = argv[1] ;

HANDLE ghEvent = OpenEventA(EVENT_MODIFY_STATE,FALSE,



eventName);

if ( ghEvent == NULL ){

DisplayError( "Opening event failed" );return 1;

}

if ( ! SetEvent( ghEvent ) ){

DisplayError( "Raising event failed" );CloseHandle( ghEvent );return 1;

}

CloseHandle( ghEvent );return 0;

}


Customizing I/O of CoverageScanner library

Chapter

32Customizing I/O of CoverageScanner library

32.1 Custom I/O using C file access

The following example shows how to generate the execution report using the standard C file API.

To compile the example on Microsoft® Windows:

cscl customiofile.c

To compile the example on Linux™ or macOS:

csgcc customiofile.c -o customiofile

Source code:

#include <stdio.h>#include <string.h>#define VERBOSE 1

static int csfputs(const char *s, void *stream){#if VERBOSEfprintf(stderr,"csfputs:%s\n",s);

#endifreturn fputs(s, (FILE *)stream);

}

static void *csfopenappend(const char *path){#if VERBOSEfprintf(stderr,"csfopenappend:%s\n",path);

#endifreturn (void*)fopen(path,"a+");

}

static void *csfopenread(const char *path)


Custom I/O using C file access

{#if VERBOSEfprintf(stderr,"csfopenread:%s\n",path);

#endifreturn (void*)fopen(path,"r");

}

static void *csfopenwrite(const char *path){#if VERBOSEfprintf(stderr,"csfopenwrite:%s\n",path);

#endifreturn (void*)fopen(path,"w");

}

static char *csfgets(char *s, int size, void *stream){char * ret;ret = fgets(s, size, (FILE *)stream);

#if VERBOSEfprintf(stderr,"csfgets:%s\n",s);

#endifreturn ret;

}

static int csremove(const char *filename){#if VERBOSEfprintf(stderr,"csremove:%s\n",filename);

#endifreturn remove(filename);

}

static int csfclose(void *fp){#if VERBOSEfprintf(stderr,"csfclose\n");

#endifreturn fclose((FILE*)fp);

}

int main(){char location[256];int lg_location;

printf(".csexe file name (without extension}:"); fflush(stdout);fgets(location,sizeof(location),stdin);lg_location=strlen(location);if (lg_location)location[lg_location-1]=’\0’; // strip \n

#ifdef __COVERAGESCANNER____coveragescanner_set_custom_io( csfgets,


Custom I/O using SFTP protocol

csfputs,csfopenappend,csfopenread,csfopenwrite,csfclose,csremove);

__coveragescanner_install(location);#endif}

32.2 Custom I/O using SFTP protocol

The following example shows how to generate the execution report directly on a SFTP server. The SFTP server ispart of SSH v2 and is available on most of the Unix platforms. On Microsoft® Windows, a free SSH server can bedownloaded from http://www.freesshd.com.

To compile the example on Microsoft® Windows:

1. Download libSSH2 from http://www.libssh2.org. Generate the library and set the environment variableLIBSSH2 to the location of the libSSH2 source code.

2. To compile the example:

cscl %LIBSSH2%\win32\debug_dll\libssh2.lib -DWIN32 --cs-libgen=/MTd/MTd -I %LIBSSH2%\include ws2_32.lib customiosftp.c

3. Execute custom_io_sftp.exe.

To compile the example on Linux™:

1. Install the development package of libssh2.

2. To compile the example:

csgcc -lssh2 customiosftp.c -o customiosftp

3. Execute custom_io_sftp.

Source code:

#define VERBOSE 1

#ifdef WIN32# include <winsock2.h>#define LIBSSH2_WIN32#define LIBSSH2_API#else#include <unistd.h># include <sys/socket.h># include <netinet/in.h>#include <arpa/inet.h>


http://www.freesshd.com

http://www.libssh2.org


#endif#include <libssh2.h>#include <libssh2_sftp.h>

#include <sys/types.h>#include <fcntl.h>#include <errno.h>#include <stdio.h>#include <ctype.h>

static LIBSSH2_SESSION *session=NULL;static LIBSSH2_SFTP *sftp_session=NULL;static int sock=0;

static void extract_location(const char *location, char *server, char *user, char*passwd, char *file)

{int i,j;int lg_location;lg_location=strlen(location);for (i=0;i<lg_location;i++){if (location[i]==’\n’)break;

server[i]=location[i];}server[i]=’\0’;i++;

for (j=0;i<lg_location;i++,j++){if (location[i]==’\n’)break;

user[j]=location[i];}user[j]=’\0’;i++;

for (j=0;i<lg_location;i++,j++){if (location[i]==’\n’)break;

passwd[j]=location[i];}passwd[j]=’\0’;i++;

for (j=0;i<lg_location;i++,j++){if (location[i]==’\0’)break;

file[j]=location[i];}



file[j]=’\0’;}

static void close_sftp_session(){if (sftp_session)libssh2_sftp_shutdown(sftp_session);

sftp_session=NULL;

if (session){libssh2_session_disconnect(session, "Normal Shutdown, Thank you for playing");libssh2_session_free(session);

}session=NULL;

if (sock){

#ifdef WIN32Sleep(1000);closesocket(sock);

#elsesleep(1);close(sock);

#endif}sock=0;

}

static int open_sftp_session(const char *server,const char *user,const char *passwd){struct sockaddr_in sin;int rc;

sock = socket(AF_INET, SOCK_STREAM, 0);

sin.sin_family = AF_INET;sin.sin_port = htons(22);sin.sin_addr.s_addr = inet_addr(server);if (connect(sock, (struct sockaddr*)(&sin),

sizeof(struct sockaddr_in)) != 0){close_sftp_session();return 0;

}

/* Create a session instance

*/session = libssh2_session_init();if(!session){close_sftp_session();return 0;



}

/* ... start it up. This will trade welcome banners, exchange keys,

* and setup crypto, compression, and MAC layers

*/rc = libssh2_session_startup(session, sock);if(rc){close_sftp_session();return 0;

}

libssh2_session_set_blocking(session, 1);

if (libssh2_userauth_password(session, user, passwd)){close_sftp_session();return 0;

}

sftp_session = libssh2_sftp_init(session);

if (!sftp_session){close_sftp_session();return 0;

}return 1;

}

static int csfputs(const char *s, void *stream){#if VERBOSEfprintf(stderr,"csfputs:%s\n",s);

#endifreturn libssh2_sftp_write((LIBSSH2_SFTP_HANDLE*)stream,s,strlen(s));

}

static void *csfopenappend(const char *location){LIBSSH2_SFTP_HANDLE *handle;char server[1024];char user[1024];char passwd[1024];char file[1024];LIBSSH2_SFTP_ATTRIBUTES attrs;

extract_location(location,server,user,passwd,file);#if VERBOSEfprintf(stderr,"csfopenappend %s:%s:%s\n",server,user,file);

#endifif (open_sftp_session(server,user,passwd)){



handle = libssh2_sftp_open(sftp_session, file, LIBSSH2_FXF_CREAT|LIBSSH2_FXF_WRITE,LIBSSH2_SFTP_S_IRUSR|LIBSSH2_SFTP_S_IWUSR|LIBSSH2_SFTP_S_IRGRP|LIBSSH2_SFTP_S_IROTH );

}elsereturn NULL;

if (handle){if (libssh2_sftp_fstat(handle,&attrs)==0){ /* Go to the end of the file */libssh2_sftp_seek(handle,attrs.filesize);

}}return handle;

}

static void *csfopenread(const char *location){char server[1024];char user[1024];char passwd[1024];char file[1024];

extract_location(location,server,user,passwd,file);#if VERBOSEfprintf(stderr,"csfopenread %s:%s:%s\n",server,user,file);

#endifif (open_sftp_session(server,user,passwd))return (void*) libssh2_sftp_open(sftp_session, file, LIBSSH2_FXF_READ, 0);

elsereturn NULL;

}

static void *csfopenwrite(const char *location){char server[1024];char user[1024];char passwd[1024];char file[1024];

extract_location(location,server,user,passwd,file);#if VERBOSEfprintf(stderr,"csfopenwrite %s:%s:%s\n",server,user,file);

#endifif (open_sftp_session(server,user,passwd))return (void*) libssh2_sftp_open(sftp_session, file, LIBSSH2_FXF_CREAT|

LIBSSH2_FXF_WRITE,LIBSSH2_SFTP_S_IRUSR|LIBSSH2_SFTP_S_IWUSR|LIBSSH2_SFTP_S_IRGRP|LIBSSH2_SFTP_S_IROTH );

elsereturn NULL;



}

static char *csfgets(char *s, int size, void *stream){size_t ss;ss = libssh2_sftp_read((LIBSSH2_SFTP_HANDLE*)stream,s,size-1);if (ss){s[ss]=’\0’;

#if VERBOSEfprintf(stderr,"csfgets:%s\n",s);

#endifreturn s;

}elsereturn NULL;

}

static int csremove(const char *location){int ret;char server[1024];char user[1024];char passwd[1024];char file[1024];

extract_location(location,server,user,passwd,file);#if VERBOSEfprintf(stderr,"csremove %s:%s:%s\n",server,user,file);

#endifif (open_sftp_session(server,user,passwd)){ret = libssh2_sftp_unlink(sftp_session,file);close_sftp_session();return ret;

}elsereturn -1;

}

static int csfclose(void *fp){#if VERBOSEfprintf(stderr,"csfclose\n");

#endifreturn libssh2_sftp_close((LIBSSH2_SFTP_HANDLE*)fp);

}

int main(){char location[1024];char tmp[1024];

#ifdef WIN32



WSADATA wsadata;

WSAStartup(WINSOCK_VERSION, &wsadata);#endif

location[0]=’\0’;

printf("server IP:"); fflush(stdout);fgets(tmp,sizeof(tmp),stdin);strcat(location,tmp);

printf("user:"); fflush(stdout);fgets(tmp,sizeof(tmp),stdin);strcat(location,tmp);

printf("passwd:"); fflush(stdout);fgets(tmp,sizeof(tmp),stdin);strcat(location,tmp);

printf(".csexe file name (without extension}:"); fflush(stdout);fgets(tmp,sizeof(tmp),stdin);strcat(location,tmp);

location[strlen(location)-1]=’\0’;

#ifdef __COVERAGESCANNER____coveragescanner_set_custom_io( csfgets,

csfputs,csfopenappend,csfopenread,csfopenwrite,csfclose,csremove);

__coveragescanner_install(location);#endif}


Generation of diff files

Chapter

33Generation of diff files

Squish Coco supports patch file analysis in the CoverageBrowser (see Chapter 19.6, page 140) and in cmreport withthe option --patch (see Chapter 22, page 149). For this, a diff file in the “unified” format is needed. It is providedby the UNIX® diff tool and by most version control systems.

The following examples work both on the UNIX® shell and with a Microsoft® Windows command window.

33.1 Comparing directories

If the two versions of the source are in different directories of the file system, the diff command can be used. It isthe standard UNIX® file comparison program which gives the functionality its name.

The “unified” format must be set with the -u flag, and the result must be redirected to a file, so that the command linewill look like

$ diff -u project/version_1 project/version_2 > project.diff

Then the directories project/version_1 and projects/version_2 are compared and the output is written to thefile project.diff. Of the two directories, project/version_1 is viewed as the earlier version.

33.2 Version control systems

In a version control system, the diff functionality is used to compare two revisions of the source tree.

For most of the newer version control systems exist GUI wrappers, like Tortoise SVN, especially for Windows. If theunderlying version control system supports unified diffs by default, the same can be expected from the GUI wrappers.

git: With git, one needs to specify two revisions and to redirect the output to a file. git uses the unified format bydefault. One can compare two revisions with

$ git diff 〈rev1〉〈rev2〉 > project.diff



https://tortoisesvn.net

https://git-scm.com

Version control systems

This command creates by default a diff of the whole source tree.

For a list of ways to specify the revisions under git, see the manpage of gitrevisions (7).

Mercurial:The distributed version system Mercurial generates by default diff files in the unified format. One has to write

$ hg diff -r 〈rev1〉 -r 〈rev2〉 > project.diff

to compare the Mercury change sets 〈rev1〉 and 〈rev2〉.

Subversion:The version control system Subversion generates unified diffs by default. To generate the diff, write

$ svn diff -r 〈rev1〉 -r 〈rev2〉 > project.diff

where 〈rev1〉 and 〈rev2〉 may be tags or revision numbers.

cvs: The Concurrent Versions System is the oldest of the revision control systems described here. It does not usethe unified format by default. With it, one has to write

$ cvs diff -u -N -r 〈rev1〉 -r 〈rev2〉 > project.diff

to compare two revisions. The flag -u switches unified diffs on, and -N is necessary to track changes in which afile is added or removed. The revisions 〈rev1〉 and 〈rev2〉 must be tags or dates, since cvs does not have globalversion numbers. The command then creates a lot of additional console output, but the differences between thetwo revisions are still written to project.diff.

Note that the command only lists the differences in the current directory and its subdirectories.


https://www.mercurial-scm.org

http://subversion.apache.org

https://savannah.nongnu.org/projects/cvs

Part X Coco IntegrationHandbook


Support for IDEs and toolkits

Chapter

34Support for IDEs and toolkits

In this chapter we give examples how Squish Coco can be combined with several IDEs and build environments.

34.1 GNU Makefiles

Mostly, in makefiles, the C and C++ compiler and the linker are defined using the environment variables CC, CXX andLINK. This can be substituted by CoverageScanner by setting CC, CXX and LINK in the command arguments of make.

Example: make LINK=csg++ CXX=csg++ CC=csgcc

34.2 CygWin

To install the CoverageScanner compiler wrapper for GCC and G++ on CygWin, proceed as follows:

1. Open the “Build Environment Selection” application. (〈Windows Coco〉\toolselector.exe)

2. Select the item “CygWin - www.cygwin.com”.

3. Click on “Install CygWin Support”. The build environment selection dialog then displays the list of generatedcompiler wrappers.

Open then the CygWin console and compile your application with csgcc instead of gcc (or csg++ instead of g++).





Scratchbox

34.3 Scratchbox

If Squish Coco is installed on the root file system1, a compiler wrapper is created for each compiler supported byScratchbox. To invoke CoverageScanner, prepend cs to the name of the cross-compiler.

34.4 Kitware CMake

CMake is a platform independent build tool from Kitware which can be downloaded from http://www.cmake.org.

When Squish Coco is used with CMake, the changes are partially dependent on the tool chain that is used forcompilation. We will now first describe the addition of a new build type, which is independent from the tool chain,and then the additional changes for Microsoft® Visual Studio® and GNU GCC.

34.4.1 Adding a new build type

Create a new CMake COVERAGE build type by adding to CMakeLists.txt the following lines:

SET(COVERAGE_FLAGS "--cs-on --cs-count")SET( CMAKE_CXX_FLAGS_COVERAGE

"${CMAKE_CXX_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING"Flags used by the C++ compiler during coverage builds."FORCE )

SET( CMAKE_C_FLAGS_COVERAGE"${CMAKE_C_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING"Flags used by the C compiler during coverage builds."FORCE )

SET( CMAKE_EXE_LINKER_FLAGS_COVERAGE"${CMAKE_EXE_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING"Flags used for linking binaries during coverage builds."FORCE )

SET( CMAKE_SHARED_LINKER_FLAGS_COVERAGE"${CMAKE_SHARED_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING"Flags used by the shared libraries linker during coverage builds."FORCE )

SET( CMAKE_STATIC_LINKER_FLAGS_COVERAGE"${CMAKE_STATIC_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING"Flags used by the static libraries linker during coverage builds."FORCE )

MARK_AS_ADVANCED(CMAKE_CXX_FLAGS_COVERAGECMAKE_C_FLAGS_COVERAGECMAKE_EXE_LINKER_FLAGS_COVERAGECMAKE_SHARED_LINKER_FLAGS_COVERAGECMAKE_STATIC_LINKER_FLAGS_COVERAGECOMPILE_DEFINITIONS_COVERAGE

)

1Installation option: "1) Installation on the local machine (need to be root)"


http://en.wikipedia.org/wiki/Scratchbox

http://www.cmake.org

Kitware CMake

These commands take the compiler and linker flags from the “Release” build type and add to them the coverage flags.If you want to use instead the flags of another build type, replace the suffix “_RELEASE” in this code with the name ofanother build type, e.g. with “_DEBUG”.

The variable COVERAGE_FLAGS in the first line specifies the CoverageScanner command line options. Change itsvalue to fit you needs. Only --cs-on should always be present.

34.4.2 Microsoft® Visual Studio®

Under Microsoft® Visual Studio®, we need to make the new Coverage build type visible to the IDE. To do this, addto CMakeLists.txt the lines

if(CMAKE_CONFIGURATION_TYPES)set(CMAKE_CONFIGURATION_TYPES Debug Release MinSizeRel RelWithDebInfo Coverage)set(CMAKE_CONFIGURATION_TYPES "${CMAKE_CONFIGURATION_TYPES}" CACHE STRING"Reset the configurations to what we need" FORCE)

endif()

If necessary, customize the list of configuration types according to your needs. To make the changes visible toMicrosoft® Visual Studio®, a complex procedure is apparently needed:2

1. Compile CMakeLists.txt with Ctrl-F7. If warnings occur, they can be ignored.

2. Make a trivial change in CMakeLists.txt, like adding a space.

3. Compile again. Then a dialog, "File modification detected", should appear. Click "Reload".

Then the list of solution configurations is updated and one can compile in coverage mode.

34.4.3 Microsoft® NMake

Proceed as follows:

1. Create a toolchain definition file cl.cmake which replaces the compiler and linker with their CoverageScannerwrappers.

Example:

# this one is importantSET(CMAKE_SYSTEM_NAME Windows)

# specify the cross compilerFILE(TO_CMAKE_PATH "$ENV{SQUISHCOCO}/visualstudio" SQUISHCOCO)SET(CMAKE_C_COMPILER ${SQUISHCOCO}/cl.exe

CACHE FILEPATH "CoverageScanner wrapper" FORCE)SET(CMAKE_CXX_COMPILER ${SQUISHCOCO}/cl.exe

CACHE FILEPATH "CoverageScanner wrapper" FORCE)SET(CMAKE_LINKER ${SQUISHCOCO}/link.exe

CACHE FILEPATH "CoverageScanner wrapper" FORCE)

2This applies at least to CMake 3.2 and Microsoft® Visual Studio® 2013.


Qt

2. Create a Makefile project. Set the toolchain to the CoverageScanner wrapper and the build mode to COVERAGE.

Example:

cmake.exe -DCMAKE_TOOLCHAIN_FILE=cl.cmake -DCMAKE_BUILD_TYPE=COVERAGE \-G "NMake Makefiles" 〈path of cmake project〉

3. Build the project with nmake.

34.4.4 GNU GCC

1. Create a toolchain definition file gcc.cmake which replaces the compiler and linker with their CoverageScannerwrappers.

Example:

find_program(CODE_COVERAGE_GCC csgcc)find_program(CODE_COVERAGE_GXX csg++)find_program(CODE_COVERAGE_AR csar)

# specify the cross compilerSET(CMAKE_C_COMPILER "${CODE_COVERAGE_GCC}"

CACHE FILEPATH "CoverageScanner wrapper" FORCE)SET(CMAKE_CXX_COMPILER "${CODE_COVERAGE_GXX}"

CACHE FILEPATH "CoverageScanner wrapper" FORCE)SET(CMAKE_LINKER "${CODE_COVERAGE_GXX}"

CACHE FILEPATH "CoverageScanner wrapper" FORCE)SET(CMAKE_AR "${CODE_COVERAGE_AR}"

CACHE FILEPATH "CoverageScanner wrapper" FORCE)

2. Create a Makefile project. Set the toolchain to the CoverageScanner wrapper and the build mode to COVERAGE.

Example:

cmake -DCMAKE_TOOLCHAIN_FILE=gcc.cmake -DCMAKE_BUILD_TYPE=COVERAGE \-G "Unix Makefiles" 〈path of cmake project〉

3. Build the project with make.

34.5 Qt Framework

34.5.1 Qt Creator

The following steps are needed to add instrumentation to a Qt Creator project.

• Switch to the Projects mode using the corresponding button in the mode pane on the left side of the IDE’sinterface.

• Create a custom build mode called CodeCoverage:



Qt

– Select the Debug or Release field in the Edit build configuration combo box.

– Click on AddÕClone Selected.

– Enter CodeCoverage as new configuration name.

• Adjust the Build directory such that the CodeCoverage configuration uses a directory different from thatof the other builds.

• In the Build Environment section, expand the System Environment pane and modify the PATH variable byprepending the path of the CoverageScanner binary:

Platform new PathmacOS /Applications/SquishCoco:〈old path〉Linux™ /opt/SquishCoco/bin:〈old path〉Microsoft® Windows c:\Program Files\SquishCoco;〈old path〉

Under Linux™, Squish Coco may have been installed locally. If it has been your account name is usr, use thepath /home/usr/SquishCoco/bin:〈old path〉 instead of that one in the table.

• Select the build step qmake and add to the Additional arguments the text CONFIG+=CodeCoverage. Thisallows to specify additional code coverage settings by modifying the qmake’s profile file (34.5.2, page 195).

• Select the build step Make and enter in the Additional arguments box the following text:

Platform Additional ArgumentsmacOS LINK=csclang++ AR=csar CXX=csclang++ CC=csclangLinux™ LINK=csg++ AR=csar CXX=csg++ CC=csgccMicrosoft® Windows LINK=cscl LIB=cslib CXX=cscl CC=cscl

With these settings, a new build mode CodeCoverage has been created that creates an instrumented version of yourprogram.

If the project has unit tests, it is possible to execute them at every build and also generate a coverage report for them.In order to do this, the following additional steps are needed.

• In the Build Steps section, click on Add Build Step and select Custom Process Step. A new build stepis created to execute the unit test the code coverage of which you are interested in. Fill in the Command box thename of the program that runs the unit tests and into the Working directory box the directory in which itshould be executed. If necessary, also fill in the command line arguments of the program.

• Add another Custom Process Step. For this step enter the full path of the cmcsexeimport tool in theCommand box. In Working directory, enter the same directory as in before for the unit test program. Asfor the arguments of cmcsexeimport, make sure the csmes and csexe files are included. An entry in theArguments could therefore look like

--title="Execution" -m unit_test.csmes -P -e unit_test.csexe.

Remember to replace the word unit_test in unit_test.csexe and unit_test.csmes with the actual nameof your unit test.

• The final Custom Process Step that needs to be added is the one for the cmreport tool, in case that reportgeneration was wanted. Again set the Working directory as that of the unit test, and set cmreport’s absolutepath as the Command. Then set the arguments, e.g.:

--title="Execution" -m unit_test.csmes --html=report

Remember again to replace unit_test with the name of your unit test.


Qt

Now, every time you click on Build (not just Run) with this configuration, the project will be built, instrumentedand optionally a report will be generated. Note that in case the project has been built before this configuration wasperformed, it will be required to execute the Clean Build step before building, so that the coverage measurementdata will be generated.

Also keep in mind that when there was an error and one of the build steps exits with a non-zero exit status code,Qt Creator stops the build and displays the error message of the executable that failed, instead of continuing thebuild. This means that if a unit test fails, instead of cmcsexeimport running and importing the execution as a failedexecution, cmcsexeimport does not run at all until the test case failure was fixed.

34.5.2 qmake

qmake is a tool to generate a Makefile in a platform-independent way, using as specification a so-called “projectfile”. By default, qmake chooses the programs that are used for compilation. We can however use the Squish Cocowrappers by setting some of qmake’s variables to new values. This can be done via command line parameters or byediting the project configuration files.

Command line. In simple cases, we can do this on the command line. There are two methods.

• The first method works with older versions of Qt. In it, one passes the names of the Squish Coco wrappers asparameters to qmake. Under Windows this looks like

C:\code>qmake QMAKE_LINK=cslink QMAKE_CXX=cscl QMAKE_CC=cscl

Here we redefine the variables QMAKE_CC, QMAKE_CXX and QMAKE_LINK: They contain the name of the Ccompiler, the C++ compiler and the linker that are used in the Makefiles generated by qmake.

For other platforms, the values of these variables must be replaced with the wrappers for the compilers andlinker that are used there.

• In some newer versions of Qt there is however a bug in qmake that makes the previous approach unusable.In this case, one can overwrite the compiler versions later, when the Makefile generated by qmake is usedfor compilation. In every call of the make program, certain Makefile variables must be overwritten. UnderMicrosoft® Windows, the resulting command line is:

C:\code>nmake LINK=cslink CXX=cscl CC=cscl 〈other options〉

(Under UNIX®-like systems, make must be called instead of nmake.)

Project configuration. In more complex cases it is better to enable coverage by putting some declarations in theqmake project files. Since one needs to build the project with and without coverage, the new definitions must be putinto a scope. (A scope is a region in a qmake project file that can be activated on demand.)

The following listing shows a template for such a scope, named CoverageScanner, which should be sufficient formost most projects.

CodeCoverage {COVERAGE_OPTIONS =

QMAKE_CFLAGS += $$COVERAGE_OPTIONSQMAKE_CXXFLAGS += $$COVERAGE_OPTIONSQMAKE_LFLAGS += $$COVERAGE_OPTIONS


Qt

QMAKE_CC=cs$$QMAKE_CCQMAKE_CXX=cs$$QMAKE_CXXQMAKE_LINK=cs$$QMAKE_LINKQMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIBQMAKE_AR=cs$$QMAKE_ARQMAKE_LIB=cs$$QMAKE_LIB

}

Here we also set the variables QMAKE_LINK_SHLIB and QMAKE_AR, which contain the names of the command to linkshared libraries and that to generate archives. Furthermore, can use COVERAGE_OPTIONS to set coveragescannercommandline options (see Chapter 8.2, page 62) to customize the project. An empty value for COVERAGE_OPTIONSwill also work and results in a default instrumentation.

In a small project, the CodeCoverage scope may then be copied to all profile files of the project, i.e. those that end in“.pro”. (In fact, it is enough to insert them only into those files that actually compile code and not those that onlyinclude other files.) If the project is larger, it has very often a file with common settings that is included by all profiles:This is then the most convenient place to insert the CodeCoverage scope only once.

The new coverage scope is by default inactive. To enable code coverage in a project you want to built, just add thename of the scope to the CONFIG variable when configuring it with qmake:

$ qmake CONFIG+=CodeCoverage

34.5.3 moc

The Meta-Object Compiler moc adds automatically new methods to each classes derived from QObject. Forexample, the translation function tr, the source code for all signals, the cast operator qt_cast, . . . In order toinstrument the code using the Qt Framework and not the Qt code itself, CoverageScanner provides the command lineoptions --cs-qt4 for Qt4 (resp. --cs-qt3 for Qt3).

In this case:

• Q_OBJECT macro is no more instrumented.

• All signal are instrumented in order to track their emission.

• The glue code necessary for the signal/slot mechanism is not instrumented.

• The code of Q_FOREACH macro is not instrumented on Qt4.

34.5.4 qbs

To use CoverageScanner with the Qt Build Suite (qbs) it can be set up as a toolchain. This toolchain can then beused for all qbs projects.

In order to set up CoverageScanner as a toolchain, issue the following command:

qbs setup-toolchains --type gcc /opt/SquishCoco/bin/csgcc csgcc

For Unix-based operating systems some additional configuration steps are necessary:





SCons

qbs config profiles.csgcc.cpp.archiverPath /opt/SquishCoco/bin/csarqbs config profiles.csgcc.cpp.linkerName csg++qbs config profiles.csgcc.cpp.nmPath /opt/SquishCoco/bin/csgcc-nm

The csgcc toolchain can then also be used as base profile for Qt projects:

qbs setup-qt /opt/Qt/bin/qmake qt-csgccqbs config profiles.qt-csgcc.baseProfile csgcc

34.6 SCons

To use Squish Coco with SCons, proceed as followings:

• Prepend the path of CoverageScanner’s wrapper (csgcc, cscl, . . . ) to the PATH environment variable. ThePATH environment variable should be set to be able to execute CoverageScanner wrappers .

• Set CC, AR and LINK variable to CoverageScanner corresponding wrapper.Example: when using Microsoft® Visual Studio® compiler, set CC to cscl.exe, set LINK to cslink.exe andset AR to cslib.exe.

! Do no use absolute file paths to the compiler wrapper since that some versions of SCons do nothandle properly spaces in file names.

• Add additional code coverage settings to the variables CCFLAGS, ARFLAGS and LINKFLAGS to exclude forexample files from the instrumentation.

Here is a code snippet which can be used for Microsoft® Visual Studio® command line tools:

import osfrom os.path import pathsep

env = Environment()

# Add the path of Squish Coco compiler wrapperenv[ ’ENV’ ][ ’PATH’ ] = os.environ[ ’SQUISHCOCO’ ] + pathsep + env[ ’ENV’ ][ ’PATH’ ]# TEMP variable need to be definedenv[ ’ENV’ ][ ’TEMP’ ] = os.environ[ ’TEMP’ ]

# Set the compiler to Squish Coco wrappersenv[ ’CC’ ] = ’cs’ + env[ ’CC’ ] ;env[ ’AR’ ] = ’cs’ + env[ ’AR’ ] ;env[ ’LINK’ ] = ’cs’ + env[ ’LINK’ ] ;

# Code coverage settingscoverageflags = [ ’--cs-count’ ]env[ ’CCFLAGS’ ] = env[ ’CCFLAGS’ ] + coverageflags ;env[ ’ARFLAGS’ ] = env[ ’ARFLAGS’ ] + coverageflags ;env[ ’LINKFLAGS’ ] = env[ ’LINKFLAGS’ ] + coverageflags ;


ARM Keil

34.7 ARM® Keil® µVision

To enable the code coverage analysis it is first necessary to compile the project with Squish Coco compiler wrapperfor ARM and enable the code coverage analysis during the compilation.

Processed as followings:

1. Select the Squish Coco tool chain instead of the original ARM toolchain.

(a) On the ARM® Keil® µVision IDE click on "ProjectÕManageÕComponents, Environments, Books...".

(b) Click on "Folders/Extensions", and replace .\ARMCC\bin through .\SquishCoco\bin in the field"ARMCC Folder:".

Figure 34.1: Installation of CoverageScanner on ARM® Keil® µVision: Setting the path of CoverageScanner.

2. Activate the code coverage analysis during the compilation

(a) Click on "ProjectÕOption for Target..."

(b) On the "C/C++" tab, add --cs-on into the "Misc Controls" field.


ARM Keil

Figure 34.2: Activating the code coverage analysis for an ARM® compiler

(c) On the "Linker" tab, add --cs-on into the "Misc Controls" field.

Figure 34.3: Activating the code coverage analysis for an ARM® linker


Green Hills

34.8 Green Hills® Software MULTI Launcher

34.8.1 Installation

To install CoverageScanner compiler wrapper for the Green Hills® Software tool chain, proceed as follows:

1. Open the “Build Environment Selection” application. (〈Windows Coco〉\toolselector.exe)

2. Select the item “Green Hills”.

3. Select the directory o which the Green Hills® Software compilers are installed

4. Click on “Install Green Hills Support” and wait for the confirmation dialog that the tools are generated.

34.8.2 Command Line Tools

The command line tools for the Green Hills® Software compiler are installed in the folder squishcoco of theinstalled native tool chain. Example: if the native tool chain is installed under c:\ghs\comp_201426, the SquishCoco compiler wrapper are installed under c:\ghs\comp_201426\squishcoco.

The compiler wrappers are replacing completely the Green Hills® Software tool chain. To activate the code coverageanalysis, it is necessary to add the parameter --cs-on to the compiler arguments.

34.8.3 CoverageScanner Library

In order to save an execution report, it is necessary to:

1. provide to the CoverageScanner library a list of I/O function which permits to upload the coverage informationto the host.

2. Save manually the coverage report on a specific trigger.

To save a coverage report the following code snippet can be used (an this code in an event handler which shouldtrigger the report generation):

1 #ifdef __COVERAGESCANNER__

2 __coveragescanner_save();3 __coveragescanner_clear();4 #endif

To provide the i/O functions it is necessary to call __coveragescanner_set_custom_io() in the main() functionof the application. At least three functions need to be provided by __coveragescanner_set_custom_io():

1. An equivalent function of fopen() which opens the execution report file (with the extension .csexe) inappend mode.

2. An equivalent function of fclose()

3. An equivalent function of fputs() to transfer the contain.

Example: The following code write an execution report to the local file system using the C file API.


Visual DSP


2 static int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); }3 static void *csfopenappend(const char *path) { return (void*)fopen(path,"a+"); }4 static int csfclose(void *fp) { return fclose((FILE*)fp); }5 #endif6

7 int main()8 {9 #ifdef __COVERAGESCANNER__

10 __coveragescanner_set_custom_io( NULL,11 csfputs,12 csfopenappend,13 NULL,14 NULL,15 csfclose,16 NULL);17 #endif18 ....19 }

34.9 VisualDSP®

To enable the code coverage analysis it is first necessary to compile the project with Squish Coco compiler wrapperand enable the code coverage analysis during the compilation in the VisualDSP® configuration.

Processed as followings:

1. Click on "ProjectÕProject Options..."

2. On the "Compile" tab, add --cs-on into the "Additional options" field.


Visual DSP

Figure 34.4: Activating the code coverage analysis for the compiler

3. On the "Link" tab, add --cs-on into the "Additional options" field.

Figure 34.5: Activating the code coverage analysis for the linker

4. Rebuild your project.

Like for all embedded targets, it is necessary to provide a dedicated I/O since that in most of the cases no file systemsare available for storing the code coverage information. Also since that embedded applications are generally notexiting, it is also necessary to implement a event handler which saves the execution report upon the reception of aspecific trigger.

The simulator emulates the support of a file system. To save the coverage report on the current build directory, it isonly necessary to register in the first lines of the main() a custom file I/O which uses the standard C file API.



Example:


2 static int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); }3 static void *csfopenappend(const char *path) { return (void*)fopen(path,"a+"); }4 static int csfclose(void *fp) { return fclose((FILE*)fp); }5 #endif6

7 int main()8 {9 #ifdef __COVERAGESCANNER__

10 __coveragescanner_set_custom_io( NULL,11 csfputs,12 csfopenappend,13 NULL,14 NULL,15 csfclose,16 NULL);17 #endif18 ....19 }

To record a coverage report when a specific trigger occurs, add the following source code lines in its handler:


2 __coveragescanner_save();3 __coveragescanner_clear();4 #endif

34.10 Microsoft® Visual Studio®

Squish Coco provides a wrapper for link.exe and cl.exe located on the %SQUISHCOCO%\visualstudio directory.It behaves exactly like the corresponding Microsoft® wrapper except that the code coverage analysis becomesactivated when the option --cs-on is added to the command arguments. These wrappers call the Microsoft® tools forcompilation or for linkage.

34.10.1 Microsoft® Visual Studio® .NET C# Compiler

To activate the instrumentation of C# source code, it is only necessary to add the defineCOVERAGESCANNER_COVERAGE_ON in the properties of the Microsoft® Visual Studio® .NET project. Otherdefines can be appended to select additional instrumentation options. The full list can be found on the chapter 8.3,page 70.

34.10.2 Microsoft® Visual Studio® .NET C and C++ Compiler

To use Squish Coco with Microsoft® Visual Studio® .NET proceed as follows:



1. Add the location of the CoverageScanner wrappers to the first position in the VC++ Directories.For Microsoft® Visual Studio® 2005 or 2008:

(a) Start Microsoft® Visual Studio® 2005 or 2008.

(b) Open the option dialog: click on "ToolsÕPreferences...".

(c) Select the item "ProjectsÕVC++ Directories".

(d) Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of directories. (see Fig-ure 34.6, page 204)

Add the path of the compiler

wrapper for Visual Studio

to the first position

Figure 34.6: Installation of CoverageScanner on Microsoft® Visual Studio® 2005 and 2008: Setting the path of CoverageScanner.

For Microsoft® Visual Studio® 2010:

(a) Start Microsoft® Visual Studio® 2010.

(b) Open a C++ project.

(c) Open the project properties using the context menu of the laded project.

(d) Select the item "Configuration PropertiesÕVC++ Directories".

(e) Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of executable directories.(see Figure 34.7, page 205)






Figure 34.7: Installation of CoverageScanner on Microsoft® Visual Studio® 2010: Setting the path of CoverageScanner.

2. To activate code coverage analysis:

(a) Open a Visual C or C++ project.

(b) Edit the project settings (click on "ProjectÕProperties").

(c) Add to the option --cs-on to the additional command line arguments of the C or C++ compiler andlinker. (see Figure 34.8, page 206)

(d) In the additional arguments of the linker, add the --cs-libgen which permits to specify which libraryshould be used for the generation of the CoverageScanner library. The table 34.1, page 205 contains thelist of recommended settings.

(e) For Microsoft® Windows CE applications, append to the linker arguments the command line option--cs-architecture which permits to specify the target platform. The table 34.2, page 206 contains thelist of available architectures.

Library Library File Command line optionSingle Threaded LIBC.LIB --cs-libgen=/ML

Static MultiThread LIBCMT.LIB --cs-libgen=/MT

Dynamic Link (DLL) LIBCRT.LIB --cs-libgen=/MD

Debug Single Threaded LIBCD.LIB --cs-libgen=/MLd

Debug Static MultiThread LIBCMTD.LIB --cs-libgen=/MTd

Debug Dynamic Link (DLL) LIBCRTD.LIB --cs-libgen=/MDd

Table 34.1: CoverageScanner library settings for Microsoft® Visual Studio®



Targeted Architecture Command line optionARM Microprocessor --cs-architecture=ARM

ARM Microprocessor (Thumb code) --cs-architecture=THUMB

x86 Microprocessor --cs-architecture=IX86

MIPS16 Microprocessor --cs-architecture=MIPS16

MIPS Microprocessor --cs-architecture=MIPS

MIPS Microprocessor with FPU --cs-architecture=MIPSFPU

SH3 Microprocessor with FPU --cs-architecture=SH3

SH4 Microprocessor with FPU --cs-architecture=SH4

Table 34.2: List of target architectures

line option'--cs-on' here

Add the command

Figure 34.8: Activation of the instrumentation under Visual Studio® .NET.


Add the command

Figure 34.9: Activation of the instrumentation under Visual Studio® .NET.

34.10.3 Squish Coco Add-In for Microsoft® Visual Studio®

Squish Coco provides for Microsoft® Visual Studio® 2005 up to Microsoft® Visual Studio® 2013 an Add-In whichdoes automatically the steps described in the section 34.10.2, page 203. The Add-In itself is normally not mandatory



to build an instrumented application.

Two different workflow are supported:

• Enabling/disabling the code coverage analysis on an existing C++ or C# configuration.This method permits to quickly enable the code coverage analysis on an existing build mode. Unfortunately, acomplete rebuild is then necessary.

• Creating a new configuration which provides the coverage analysis. The Add-In duplicate a specified configu-ration and apply the code coverage settings on it.This method is more convenient for a developer since that it permits to quickly switch from the Debug/Releaseto the Coverage build mode without rebuilding the whole application. Also once the code coverage settings areapplied it is possible to save it permanently in the new configuration. Unfortunately, this method may requiresome manual adjustments on some projects which are using fixed encoded paths.

To use it:

1. Open a C++ or a C# project.

2. Click on “ToolsÕCode Coverage Build Mode...”.

3. Select the base configuration and the project which should be compiled with code coverage support.

4. And finally:

• Select “Modify” and click on “Enable code coverage” to alter the current build mode and add thecode coverage support. “Disable code coverage” permits removes all CoverageScanner commandline options.

• Select “New” and click on “Create new configuration” to generate a new build mode with codecoverage support.

Coverage options

Project and Configuration

used as template

the inline code is instrumented

Include paths in which

Creation of a new configuration

Enabling/disabling the coverageanalysis on the current configuration

Figure 34.10: Microsoft® Visual Studio® Add-In

i For Microsoft® Visual Studio® 2010, it might be necessary to add manually the path Squish Coco executablesto the search path. See section 34.10.2, page 203 for more the detailed procedure.



34.10.4 Microsoft® Visual C++ Express

To use Squish Coco with Microsoft® Visual C++ Express proceed as follows:

1. Add the location of the CoverageScanner wrappers to the first position in the VC++ Directories.

(a) Start Microsoft® Visual C++ Express.


(c) Select the item "ProjectsÕVC++ Directories".

(d) Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of directories. (see Fig-ure 34.11, page 208)




Figure 34.11: Installation of CoverageScanner on Visual C++ Express: Setting the path of CoverageScanner.

2. The activation of the code coverage analysis is similar to Microsoft® Visual Studio® .NET. (see Chapter 34.10.2,page 203)

34.10.5 Microsoft® Visual Studio® 6.0

To use Squish Coco with Microsoft® Visual Studio® 6.0 proceed as follows:

1. Add the location of the CoverageScanner wrappers to the first position in the executable directories.

(a) Start Microsoft® Visual Studio® 6.0.


(c) Select the item "Directories".

(d) Select "Executable files" in the combobox "Show directories for:".

(e) Add the path of the directory visualstudio of the Squish Coco installation3 to the first position in thelist of directories. (example: if Squish Coco is installed on c:\programme\SquishCoco, add the pathc:\programme\SquishCoco\visualstudio, (see Figure 34.12, page 209))

3Microsoft® Visual Studio® 6.0 does not handle system variables in the path list. So the %SQUISHCOCO% variable needs to be expanded.






Figure 34.12: Installation of CoverageScanner on Visual Studio® 6.0: Setting the path of CoverageScanner.

2. To activate the code coverage analysis:

(a) Open a Visual C or C++ project.


(c) Add the option --cs-on to the additional command line arguments of the C or C++ compiler and linker.(see Figure 34.13, page 209)

(d) In the additional arguments of the linker, add the --cs-libgen which permits to specify which libraryshould be used for the generation of the CoverageScanner library. The table 34.1, page 205 contains thelist of recommended settings.


Add the command

Figure 34.13: Activation of the instrumentation under Visual Studio® 6.0.


eMbedded Visual C++


Add the command

Figure 34.14: Activation of the instrumentation under Visual Studio® 6.0.

34.11 Microsoft® eMbedded Visual C++®

To use Squish Coco with Microsoft® eMbedded Visual C++®proceed as follows:

1. Add the location of the CoverageScanner wrappers to the first position in the executable directories.

(a) Start Microsoft® eMbedded Visual C++®.


(c) Select the item "Directories".

(d) Select "Executable files" in the combobox "Show directories for:".

(e) Select "Platform" and the targeted "CPUs".

(f) Add the path of the directory WinCE of the Squish Coco installation4 to the first position in the listof directories. (example: if Squish Coco is installed on c:\programme\SquishCoco, add the pathc:\programme\SquishCoco\WinCE, (see Figure 34.15, page 210))



wrapper for Windows CE

Select the Platform and the CPU

Figure 34.15: Installation of CoverageScanner on eMbedded Visual C++®: Setting the path of CoverageScanner.

2. To activate the code coverage analysis:

(a) Open a Visual C or C++ project.4Microsoft® eMbedded Visual C++®does not handle system variables in the path list. So the %SQUISHCOCO% variable needs to be expanded.


Eclipse


(c) Add the option --cs-on to the additional command line arguments of the C and C++ compiler and linker.(see Figure 34.16, page 211)


Add the command

Figure 34.16: Activation of the instrumentation under eMbedded Visual C++®.


Add the command

Figure 34.17: Activation of the instrumentation under eMbedded Visual C++®.

34.12 Eclipse™ IDE for C/C++

The Eclipse™ IDE for C/C++ relies on a GNU gcc command line compiler. Before trying to compile a project withcoverage support, it is therefore necessary to create a compiler wrapper (see Chapter C.4, page 272). Then proceed asfollows:

1. Start Eclipse™.

2. Load a C or C++ project.

3. Open the property window (ProjectÕ Properties).

4. Click on "C/C++ Build/Settings".

5. Create a new configuration by clicking on "Manage Configurations...", and select it.


Xcode

6. Click on "Tools Settings" tab.

7. Click on "GCC C++ Compiler" and prepend cs to the name of the compiler.

8. Click on "GCC C Compiler" and prepend cs to the name of the compiler.

9. Click on "C++ Linker" and prepend cs to the name of the linker.

10. If it is a library project, also click on "GCC archiver" and replace the name of the archiver command ar withcsar.

compiler/linkerexecutable name

Append 'cs' to the

Figure 34.18: Eclipse™ settings

34.13 Apple® Xcode

To use Squish Coco with Apple® Xcode proceed as follows: To activate the code coverage analysis:

1. Open a terminal window and set the CPLUSPLUS, LDPLUSPLUS, LD and CC to CoverageScanner compilerwrapper. The path of native compiler (clang, clang++, gcc or g++) need to be present in the PATH environmentvariable. Start Xcode using the open command.

If GCC is used as compiler:

SQUISHCOCO=/Applications/SquishCoco/wrapperexport CC=$SQUISHCOCO/gccexport LD=$SQUISHCOCO/gccexport CPLUSPLUS=$SQUISHCOCO/g++export LDPLUSPLUS=$SQUISHCOCO/g++


Xcode

open /Developer/Applications/Xcode.app

If clang is used as compiler:

SQUISHCOCO=/Applications/SquishCoco/wrapperexport CC=$SQUISHCOCO/clangexport LD=$SQUISHCOCO/clangexport CPLUSPLUS=$SQUISHCOCO/clang++export LDPLUSPLUS=$SQUISHCOCO/clang++export XCODE_TOOLCHAIN_DIR=/Applications/Xcode.app/Contents/Developer/Toolchainsexport XCODE_TOOLCHAIN=$XCODE_TOOLCHAIN_DIR/XcodeDefault.xctoolchain/usr/bin/

export PATH=$XCODE_TOOLCHAIN:$PATH

open /Applications/Xcode.app

2. Open a Xcode C or C++ project.

3. Edit the project settings (click on "ProjectÕEdit Project Settings").

4. Add the option --cs-on to the additional command line arguments of the C and C++ compiler (fieldsOther C Flags and Other C++ Flags) and linker (field Other Linker Flags). (see Figure 34.19,page 213)

5. Disable the usage of the precomiled header: Open the settings of the active target ("ProjectÕEdit ActiveTarget") and remove the contains of Prefix Header.

Add the compiler

command line option'--cs-on' here

command line option'--cs-on' here

Add the linker

Figure 34.19: Activation of the instrumentation under Apple® Xcode.


Support for specific test frameworks

Chapter

35Support for specific test frameworks

35.1 CppUnit

CppUnit1 is a unit test framework for C++. This environment can easily be adapted to get the code coverage fromeach unit test.

The following code is an example how this can be done:

#include <cppunit/TestListener.h>#include <cppunit/BriefTestProgressListener.h>#include <cppunit/CompilerOutputter.h>#include <cppunit/extensions/TestFactoryRegistry.h>#include <cppunit/TestResult.h>#include <cppunit/TestResultCollector.h>#include <cppunit/TestRunner.h>

class CoverageScannerListener : public CppUnit::TestListener{

public:CoverageScannerListener() {}

void startTest( CppUnit::Test *test ){

m_testFailed = false;#ifdef __COVERAGESCANNER__

int pos;// Adjusting the name of the test to display the tests// in a tree view in CoverageBrowserstd::string testname = "CppUnit/" + test->getName();while ( ( pos = testname.find( "::", 0 ) ) != std::string::npos )

testname.replace( pos, 2, "/" );

// Reset the code coverage data to get only the code coverage

1Project page of CppUnit: http://cppunit.sourceforge.net


http://cppunit.sourceforge.net

CppUnit

// of the actual unit test.__coveragescanner_clear();__coveragescanner_testname( testname.c_str() ) ;

#endif}

void addFailure( const CppUnit::TestFailure &failure ){

m_testFailed = true;}

void endTest( CppUnit::Test *test ){


// Recording the execution state in the coverage reportif ( m_testFailed )

__coveragescanner_teststate( "FAILED" );else

__coveragescanner_teststate( "PASSED" );

// Saving the code coverage report of the unit test__coveragescanner_save();__coveragescanner_testname( "" );

#endif}

private:bool m_testFailed;// Prevents the use of the copy constructor and operator.CoverageScannerListener( const CoverageScannerListener &copy );void operator =( const CoverageScannerListener &copy );

};

int main( int argc, char* argv[] ){#ifdef __COVERAGESCANNER__

__coveragescanner_install( argv[0] );#endif

// Create the event manager and test controllerCPPUNIT_NS::TestResult controller;

// Add a listener that colllects test resultCPPUNIT_NS::TestResultCollector result;controller.addListener( &result );

// Add a listener that print dots as test run.CPPUNIT_NS::BriefTestProgressListener progress;controller.addListener( &progress );

// Add a listener that saves the code coverage informationCoverageScannerListener coveragescannerlistener;controller.addListener( &coveragescannerlistener );


QTestLib

// Add the top suite to the test runnerCPPUNIT_NS::TestRunner runner;runner.addTest( CPPUNIT_NS::TestFactoryRegistry::getRegistry().makeTest() );runner.run( controller );

return result.wasSuccessful() ? 0 : 1;}

In the example, we have done the following steps:

1. We write a CppUnit listener class which records the code coverage of unit each test after it is completed.

We want to be able to run the program with and without Squish Coco. Therefore we use in the code the macro__COVERAGESCANNER__ for conditional compilation. The macro is defined in every file that is instrumented bySquish Coco, without the need to #include anything.

In the listener class, CppUnitListener, we use the following menber functions:

startTest():This function is called before each test begins.In it, we compute a test name with the information provided by CppUnit and pass it to the Squish Cocolibrary with __coveragescanner_testname().We also call the function __coveragescanner_clear(): It empties the internal database and so makessure that the coverage of the code that was executed before this test is ignored.

addFailure():This function is called after a test fails. It just sets a flag that is used by the other functions.

endTest():This function is called after a test has ended.It uses __coveragescanner_teststate() to record the execution status ("PASSED" or "FAILED")and then saves the code coverage report itself with __coveragescanner_save().

2. We call __coveragescanner_install() in the main() function.

3. We add this listener in the test manager of CppUnit, the class CPPUNIT_NS::TestResult. In the exampleabove, this is done by the following lines:

CoverageScannerListener coveragescannerlistener;controller.addListener( &coveragescannerlistener );

35.2 QTestLib

QTestLib is a unit test framework for Qt. It can easily be adapted to get the code coverage for each unit test.

Proceed as follows:

1. Call __coveragescanner_install() in the main() function.

2. Write a subclass of QObject, named TestCoverageObject. It must record the code coverage at the end ofevery unit test.

3. Instead of inheriting from QObject, let all your test cases inherit from TestCoverageObject.


http://doc.qt.io/qt-5/qtest-tutorial.html

QTestLib

4. The TestCoverageObject class provides its own init() and cleanup() slots, which use the Coverage-Scanner API to save the code coverage report. If these slots are also declared in the test case classes, it isnecessary to rename them to initTest() and cleanupTest().

5. Compile your project with code code coverage enabled.

TestCoverageObject header:

#ifndef _TEST_COVERAGE_OBJECT_H#define _TEST_COVERAGE_OBJECT_H#include <QObject>class TestCoverageObject : public QObject{Q_OBJECTpublic:virtual void initTest() {}virtual void cleanupTest() {}

protected slots:void init() ;void cleanup();

};#endif

TestCoverageObject source:

#include "testcoverageobject.h"#include <QTest>#include <QMetaObject>#include <QString>

void TestCoverageObject::init(){#ifdef __COVERAGESCANNER____coveragescanner_clear();

#endifinitTest();

}

void TestCoverageObject::cleanup(){cleanupTest();


QString test_name="unittest/";test_name+=metaObject()->className();test_name+="/";test_name+=QTest::currentTestFunction();__coveragescanner_testname(test_name.toLatin1());if (QTest::currentTestFailed())__coveragescanner_teststate("FAILED");

else__coveragescanner_teststate("PASSED") ;

__coveragescanner_save();


GoogleTest

__coveragescanner_testname("");#endif}

35.3 GoogleTest

GoogleTest2 is a unit test framework for C++. This environment can easily be adapted to get the code coverage fromeach unit test.

Simply proceed as follows:


2. Write a TestEventListenet class which records the code coverage report upon every unit test completion. Thelistener should set the name (using __coveragescanner_testname()) and clear the instrumentation (using__coveragescanner_clear()) before executing a test item (class member startTest()) to ensure to getonly the coverage data of the concerned test. When an test item is executed, the instrumentation and the execu-tion status should be saved (using __coveragescanner_teststate() and __coveragescanner_save())in the class member endTest(). The class CodeCoverageListener give an implementation example.

3. Add this listener in the Append function of the GoogleTest listener (function::testing::UnitTest::GetInstance()->listeners().Append()).

4. Compile the unit test using CoverageScanner.

#include <gtest/gtest.h>#include <direct.h>#include <stdlib.h>

class CodeCoverageListener : public ::testing::TestEventListener{public:// Fired before any test activity starts.virtual void OnTestProgramStart(const ::testing::UnitTest& unit_test){}

// Fired before each iteration of tests starts. There may be more than// one iteration if GTEST_FLAG(repeat) is set. iteration is the iteration// index, starting from 0.virtual void OnTestIterationStart(const ::testing::UnitTest& unit_test, int

iteration){}

// Fired before environment set-up for each iteration of tests starts.virtual void OnEnvironmentsSetUpStart(const ::testing::UnitTest& unit_test){}

2Project page of GoogleTest: http://code.google.com/p/googletest/


http://code.google.com/p/googletest/

GoogleTest

// Fired after environment set-up for each iteration of tests ends.virtual void OnEnvironmentsSetUpEnd(const ::testing::UnitTest& unit_test){}

// Fired before the test case starts.virtual void OnTestCaseStart(const ::testing::TestCase& test_case){}

// Fired before the test starts.virtual void OnTestStart(const ::testing::TestInfo& test_info){

#ifdef __COVERAGESCANNER____coveragescanner_clear();std::string test_name=std::string(test_info.test_case_name())+ ’/’+ std::string(test_info.name());

__coveragescanner_testname(test_name.c_str());#endif

}

// Fired after a failed assertion or a SUCCESS().virtual void OnTestPartResult(const ::testing::TestPartResult& test_part_result){}

// Fired after the test ends.virtual void OnTestEnd(const ::testing::TestInfo& test_info){

#ifdef __COVERAGESCANNER____coveragescanner_teststate("UNKNOWN");if (test_info.result()){if (test_info.result()->Passed())__coveragescanner_teststate("PASSED");

if (test_info.result()->Failed())__coveragescanner_teststate("FAILED");

}__coveragescanner_save();

#endif}

// Fired after the test case ends.virtual void OnTestCaseEnd(const ::testing::TestCase& test_case){}

// Fired before environment tear-down for each iteration of tests starts.virtual void OnEnvironmentsTearDownStart(const ::testing::UnitTest& unit_test){


CxxTest

}

// Fired after environment tear-down for each iteration of tests ends.virtual void OnEnvironmentsTearDownEnd(const ::testing::UnitTest& unit_test){}

// Fired after each iteration of tests finishes.virtual void OnTestIterationEnd(const ::testing::UnitTest& unit_test, int

iteration){}

// Fired after all test activities have ended.virtual void OnTestProgramEnd(const ::testing::UnitTest& unit_test){}

} ;

int main(int argc, char **argv){

//initialize CoverageScanner library#ifdef __COVERAGESCANNER____coveragescanner_install(argv[0]);

#endif

::testing::FLAGS_gtest_output = "xml";::testing::UnitTest::GetInstance()->listeners().Append(new CodeCoverageListener);

::testing::InitGoogleTest(&argc, argv); //init google test framework

return RUN_ALL_TESTS(); //run all tests which are listed in this project}

35.4 CxxTest

CxxTest3 is a unit test framework for C++. This environment can easily be adapted to get the code coverage fromeach unit test.

Proceed as follows:


2. Create a CxxTest TestListener class CoverageScannerListener by subclassing an existing listener.

In the example below this is ErrorPrinter. It will record the code coverage report upon every unit testcompletion. To ensure to get only the coverage data of the concerned test, the listener should set the namewith __coveragescanner_testname() and clear the instrumentation with __coveragescanner_clear()before executing a test item (class member enterTest()).

3Project page of CxxTest: http://cxxtest.com


http://cxxtest.com

CxxTest

When an test item is executed, the instrumentation and the execution status should be saved in the memberfunction leaveTest() with __coveragescanner_teststate() and __coveragescanner_save().

Finally, all test failure members of this class must be reimplemented to record the test failures.

3. In the main() function call the run() function of CoverageScannerListener instead of CxxTest::ErrorPrinter().run().

4. Compile the unit test with CoverageScanner activated.

Example:

#include <cxxtest/TestRunner.h>#include <cxxtest/TestListener.h>#include <cxxtest/TestTracker.h>#include <cxxtest/ValueTraits.h>#include <cxxtest/ValueTraits.h>#include <cxxtest/ErrorPrinter.h>

class CoverageScannerListener : public CxxTest::ErrorPrinter{public:CoverageScannerListener(std::ostream &o=std::cout,

const char *preLine = ":",const char *postLine = "")

: CxxTest::ErrorPrinter( o, preLine , postLine ) {}

int run(){return CxxTest::ErrorPrinter::run();

}

void enterTest( const CxxTest::TestDescription & desc){test_passed=true;


// Adjust the name of the test to display the tests// in a tree view in CoverageBrowserstd::string testname="CxxTest/";testname += desc.suiteName();testname += "/";testname += desc.testName();

// Reset the code coverage data to get only the code coverage// of the actual unit test.__coveragescanner_clear();__coveragescanner_testname(testname.c_str());

#endifreturn CxxTest::ErrorPrinter::enterTest( desc );

}

void leaveTest( const CxxTest::TestDescription & desc){



CxxTest

// Record the execution state in the coverage reportif (test_passed)__coveragescanner_teststate("PASSED");

else__coveragescanner_teststate("FAILED");

// Save the code coverage report of the unit test__coveragescanner_save();

#endifreturn CxxTest::ErrorPrinter::leaveTest( desc );

}

void failedTest(const char *file, int line, const char *expression){ // Only record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedTest( file, line, expression );

}

void failedAssert(const char *file, int line, const char *expression){ // Only record that the test failstest_passed = false;return CxxTest::ErrorPrinter::failedAssert( file, line, expression );

}

void failedAssertEquals(const char *file, int line,const char *xStr, const char *yStr,const char *x, const char *y)

{ // Only record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertEquals( file, line,

xStr, yStr,x, y );

}

void failedAssertSameData(const char *file, int line,const char *xStr, const char *yStr, const char *sizeStr,const void *x, const void *y, unsigned size)

{ // Only record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertSameData( file, line,

xStr, yStr, sizeStr,x, y, size );

}

void failedAssertDelta(const char *file, int line,const char *xStr, const char *yStr, const char *dStr,const char *x, const char *y, const char *d)

{ // Only record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertDelta( file, line,

xStr, yStr, dStr,x, y, d );

}


CxxTest

void failedAssertDiffers(const char *file, int line,const char *xStr, const char *yStr, const char *value)

{ // Only record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertDiffers(file, line,

xStr, yStr, value );}

void failedAssertLessThan(const char *file, int line,const char *xStr, const char *yStr,const char *x, const char *y)

{ // Only record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertLessThan(file, line,

xStr, yStr,x, y );

}

void failedAssertLessThanEquals(const char *file, int line,const char *xStr, const char *yStr,const char *x, const char *y)

{ // Only record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertLessThanEquals( file, line,

xStr, yStr,x, y );

}

void failedAssertRelation(const char *file, int line, const char *relation,const char *xStr, const char *yStr,const char *x, const char *y)

{ // Only record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertRelation( file, line, relation,

xStr, yStr,x, y);

}

void failedAssertPredicate(const char *file, int line, const char *predicate,const char *xStr, const char *x )

{ // Only record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertPredicate( file, line, predicate,

xStr, x);}

void failedAssertThrows(const char *file, int line, const char *expression,const char *type, bool otherThrown)

{ // Only record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertThrows( file, line, expression,

type, otherThrown );


NUnit

}

void failedAssertThrowsNot(const char *file, int line, const char *expression){ // Only record that the test failstest_passed=false;return CxxTest::ErrorPrinter::failedAssertThrowsNot( file, line, expression );

}

private:bool test_passed;

};

int main(){#ifdef __COVERAGESCANNER____coveragescanner_install(argv[0]);

#endif

// Use "CoverageScannerListener().run()" instead of "CxxTest::ErrorPrinter().run()"return CoverageScannerListener().run();

}

35.5 NUnit

Squish Coco provides an addin for NUnit4 version 2.4.4 and above as sample. To install it proceed as followings:

1. Build NUnitSquishCoco.dll using the Microsoft® Visual Studio® project NUnitSquishCoco.vsproj pro-vided in the sample directory.

2. Copy NUnitSquishCoco.dll to the addins folder located in the bin folder where NUnit.exe can be found.

3. Start NUnit.exe and verify that the addin “NUnit Squish Coco” is loaded.

Once installed, as soon as NUnit’s test driver is executing a C# or C++ managed unit test test.dll, it generatesautomatically a code coverage execution report test.dll.csexe automatically if test.dll is instrumented withSquish Coco. The code coverage information is organized into a tree containing the coverage and the execution statusfor each single unit test. The execution report can be then imported into the application’s instrumentation databasewith CoverageBrowser or cmcsexeimport.

35.6 Squish

It is easily possible to run the GUI testing tool Squish together with Squish Coco in order to get the C/C++coverageof a Squishtest suite. A more in-depth analysis is then possible, correlating each test case (and its results) with the

4http://nunit.org/


http://nunit.org/

Squish

respective coverage information. This is especially true since Squish Coco features the comparison of individualexecutions, including the calculation of an optimal order.

35.6.1 General Approach

The approach depicted below is based on the possibility to apply information about the name, the result and afree-form comment to each execution (stored in .csexe files). See Chapter 30.1, page 172 for the details.

As an example we’ll use Squish for Qt’s addressbook on a Unix-based system and a JavaScript test script.

1. First we’ll initialize the execution data with the name of the Squish test case that is being run.

function main(){

var currentAUT = currentApplicationContext();var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe"var testCase = squishinfo.testCase;var testExecutionName = testCase.substr(testCase.lastIndexOf(’/’) + 1);var file = File.open(execution, "a");file.write("*" + testExecutionName + "\n");file.close();var ctx = startApplication("addressbook");

...

2. Insert the main test script at this point

3. After the main test script we’ll log the result of the test for the coverage tool:

...// wait until AUT shutdownwhile (ctx.isRunning) {

snooze(1); // increase time if not enough to dump coverage data}

// test result summary and statusvar positive = test.resultCount("passes");var negative = test.resultCount("fails") + test.resultCount("errors") + test.

resultCount("fatals");var msg = "TEST RESULTS - Passed: " + positive + " | " + "Failed/Errored/

Fatal: " + negative;var status = negative == 0 ? "PASSED" : "FAILED";var file = File.open(execution, "a");file.write("<html><body>" + msg + "</body></html>\n");file.write("!" + status + "\n")file.close();

}

When you execute the scripts containing these steps, the Squish CocoExecution Report loads with the test case name,status and execution summary in the execution details and comments.


Squish

35.6.2 Simplified for Reuse

1. Create a file called squishCocoLogging.js in Test Suite Resources with the following functions:

function getExecutionPath() {var currentAUT = currentApplicationContext();var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe"return execution;

}

function logTestNameToCocoReport(currentTestCase, execution) {var testExecutionName = currentTestCase.substr(currentTestCase.lastIndexOf

(’\\’) + 1);var file = File.open(execution, "a");file.write("*" + testExecutionName + "\n");file.close();

}

function logTestResultsToCocoReport(testInfo, execution){

var currentAUT = currentApplicationContext();

// wait until AUT shuts downwhile (currentAUT.isRunning)snooze(5);

// collect test result summary and statusvar positive = testInfo.resultCount("passes");var negative = testInfo.resultCount("fails") + testInfo.resultCount("errors")

+testInfo.resultCount("fatals");

var msg = "TEST RESULTS - Passed: " + positive + " | " + "Failed/Errored/Fatal: " + negative;

var status = negative == 0 ? "PASSED" : "FAILED";

// output results and status to Coco execution report filevar file = File.open(execution, "a");file.write("<html><body>" + msg + "</body></html>\n");file.write("!" + status + "\n")file.close();

}

A Python version of this code is:

import re

def getExecutionPath():currentAUT = currentApplicationContext()execution = "%(currAUTPath)s\\%(currAUTName)s.exe.csexe" % {"currAUTPath" :

currentAUT.cwd, "currAUTName" : currentAUT.name}return execution

def logTestNameToCocoReport(currentTestCase, execution):testExecutionName = re.search(r’[^\\]\w*$’, currentTestCase)


Squish

testExecutionName = testExecutionName.group(0)file = open(execution, "a")file.write("*" + testExecutionName + "\n")file.close()

def logTestResultsToCocoReport(testInfo, execution):currentAUT = currentApplicationContext()# wait until AUT shuts downwhile (currentAUT.isRunning):

snooze(5)

# collect test result summary and statuspositive = testInfo.resultCount("passes")negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") +

testInfo.resultCount("fatals")msg = "TEST RESULTS - Passed: %(positive)s | Failed/Errored/Fatal: %(

negative)s" % {’positive’: positive, ’negative’: negative}if negative == 0:

status = "PASSED"else:

status = "FAILED"

# output results and status to Coco execution report filefile = open(execution, "a")file.write("<html><body>" + msg + "</body></html>\n")file.write("!" + status + "\n")file.close()

2. Add the following function calls after startApplication() in the main test script:

execution = getExecutionPath();

logTestNameToCocoReport(squishinfo.testCase);

In Python:

execution = getExecutionPath()

logTestNameToCocoReport(squishinfo.testCase)

3. At the end of your script, after closing the AUT (for example with steps clicking File > Exit), call the followingfunction:

logTestResultsToCocoReport(test);

In Python:

logTestResultsToCocoReport(test)

4. In the event your AUT closes unexpectedly, or a script error occurs, incorporating a try, catch, finallyensures your results still output to the Coco report file.

Your main test script should be similar to the following:


Squish

source(findFile("scripts","squishCocoLogging.JS"))

function main(){

startApplication("addressbook");execution = getExecutionPath();logTestNameToCocoReport(squishinfo.testCase, execution);

try {// body of script

}catch(e) {

test.fail(’An unexpected error occurred’, e.message)}finally {

logTestResultsToCocoReport(test, execution)}

}

Python version:

source(findFile("scripts","squishCocoLogging.py"))

def main():startApplication("addressbook")execution = getExecutionPath()logTestNameToCocoReport(squishinfo.testCase, execution)

try:try:

# body of scriptexcept Exception, e:

test.fail("test failed: ", e)finally:

logTestResultsToCocoReport(test,execution)


Atlassian Bamboo integration

Chapter

36Atlassian Bamboo integration

36.1 Introduction

Atlassian Bamboo is a continuous integration server used to build, test and release software. The Squish Coco plug-inprovides seamless Squish Coco integration with Bamboo. Users can add one or more Squish Coco tasks to a BambooJob. Each Squish Coco Task can perform the following actions:

1. Import an execution report into an instrumentation data base

2. Merge several instrumentations databases together

3. Generate code coverage report

User can choose only one action to be performed, two actions or all three actions. If more than one action is chosenthen they are executed in mentioned above order.

Figure 36.1: Squish Coco Task configuration overview

After executions finish, Artifacts for Squish Coco (like Coverage Report) can be stored to allow efficient defecttroubleshooting.


Agent configuration

36.2 Agent configuration

An Agent is a service that executes Bamboo builds, tests and deployments. The Agent Capabilities setting allowsusers to specify which agents are capable of executing Squish Coco jobs. In order to execute it , we must firstinstall Squish Coco at Agent host. Next, we need to setup Agent-specific capability called ’coco’ with Squish Cocoinstallation directory as a Value. (go to Bamboo administration|BUILD RESOURCES|Agents|AgentName|Agent-specific capabilities). Later on, during Job configuration, we need to set ’coco’ capability existence as a Jobrequirement. This way, Squish Coco jobs will be executed only on Agents where Squish Coco in installed.

36.3 Importing an execution report

This action calls cmcsexeimport to import execution report (CSExe file) into an instrumentation data base (CSMesfile). User need to provide absolute path or relative path (from Bamboo build working directory) for both CSExe fileand CSMes file and execution title. When this action is activated, the following command will be executed on Agenthost:

cmcsexeimport --debug --title=<Title> -m <CSMesFile> -e <CSExeFile>

36.4 Merging instrumentation databases

This action calls cmmerge to merge several instrumentation databases (CSMes files) together. We need to provideCSMes output file full or relative path name that will be generate as a result of merge. If we provide relative path(or just filename), then CSMes output file will be generated in Bamboo working directory or relative sub directory.CSMes input files need to be specified, multiple files can be used separated by comma. When this action is activated,the following command will be executed on Agent host:

cmmerge --verbose -o <CSMesOutputFile> <CSMesInputFile1> ... <CSMesInputFileN>

Additionally user can choose option to ’merge only instrumentations and executions present in the reference file’.This option is useful for importing CSMes files with Unit Tests. When activated, used need to provide the name ofCSMes reference file. This will cause that above cmmerge command will be executed with additional ’-i’ option.

cmmerge --verbose -o <CSMesOutputFile> -i <CSMesRefFile> <CSMesInputFile1> ... <CSMesInputFileN>

36.5 Generating coverage report

The last action calls cmreport to generate HTML Report. We need to provide CSMes file name (either absolute orrelative path). CSMes file shall contain previously imported execution report (i.e. by using first action offered byplugin).

cmreport --title=<BambooJobName> -m <CSMesFile> --debug --html=coco/report.html

Generated report will be stored in coco subdirectory, therefore we can define Artefact with ’coco/**’ as a Copy


Use Case Examples

pattern. This allows us to view the report directly in Bamboo after job execution. The report title is build usingBamboo Job name.

36.6 Use Case Examples

36.6.1 Coverage Report from single execution

We need to activate first action ’Import an execution report into an instrumentation data base’ to import executionreport (CSEXE file) into an instrumentation data base (CSMES). Finally, we need to activate third action ’Generatecode coverage report’. CSMES file defined for this action is the same as for first action.

Figure 36.2: Squish Coco Task configuration for example addressbook application


Use Case Examples

36.6.2 Coverage Report for Unit Tests

In this scenario we would like to generate coverage report for unit tests execution. To achieve this scenario in Bamboowe need to use Squish Coco task twice in one job.

After the build we have two CSMes files: one for our application and second for Unit Tests. We need to merge thoseinstrumentation data bases where CSMes file for our application will be a reference during the merge. After themerge new CSMes output file is created.

Figure 36.3: Squish Coco Task configuration for merging Unit Tests executions

Using second Squish Coco task execution report from Unit Test run need to be imported into CSMes file (which wascreated in previous task - CSMes output file). Last action ’Generate code coverage report’ is used to generate HTMLcoverage report.


Use Case Examples

Figure 36.4: Squish Coco Task configuration for importing execution and generating report


Jenkins CI integration

Chapter

37Jenkins CI integration

Jenkins CI is a popular continuous integration tool. Squish Coco can be run under Jenkins, and the changes of thecoverage over time displayed by a plugin. This chapter describes the necessary setup.

37.1 Prerequisites

37.1.1 Getting a license

The Jenkins process runs under an own account, usually called “jenkins”. When therefore Squish Coco runs underJenkins CI with a node-locked license, it needs a specific license for its account.

How to get this license is described in the following section; the section after that one describes the use of a licenseserver.

Installing a node-locked license In order to generate the license, the Squish Coco license tools need to be runfrom the “jenkins” account. There are two ways to do it, depending on whether the Jenkins server has an Internetconnection.

1. With an Internet connection.

You need to get an activation code from froglogic. With this activation code, create a new Jenkins project thatonly runs (under Microsoft® Windows) the command

"〈Windows Coco〉\cocolic" --fetch-license-key=〈activation code〉

Under Linux™ and macOS, this line becomes1

〈cocobin〉/cocolic --fetch-license-key=〈activation code〉

If the Jenkins server is connected to the Internet through a proxy, additional options are needed to specify theproxy. See Chapter 25, page 159 for the details.

1〈cocobin〉 stands for the directory with the Squish Coco binaries. Under Linux™, this is usually /opt/SquishCoco/bin, while under macOSit is /Applications/SquishCoco.






Adaption of the project to Jenkins

Now build this project. In the console output window page of Jenkins you can then see whether cocolic wassuccessful.

2. Without an Internet connection:

Here we create a machine ID offline, and from that ID froglogic creates a license key. This key is then installedon the server in a second step.

To create a machine ID, we create a Jenkins project that executes (under Windows) the following command:

"〈Windows Coco〉\cocolic" --machine-identifier

Under Linux™ and macOS, this becomes

〈cocobin〉/cocolic --machine-identifier

When this project is built, the console output of the build contains the machine ID.

Send the machine ID to froglogic. We will send you a license key. Now create a Jenkins project that onlycontains the following command, and build it:

"〈Windows Coco〉\cocolic" --license-key=〈your key〉


〈cocobin〉/cocolic --license-key=〈your key〉

Using a license server For the use of Squish Coco with a license server, one has to create a new Jenkins CI project,which only runs (under Microsoft® Windows) the following command:

"〈Windows Coco〉\cocolic" --license-server=〈host〉:〈port〉

where 〈host〉 and 〈port〉 refer to the license server (see Chapter 25, page 159). The colon and the port can be omittedif the default port is used.


〈cocobin〉/cocolic --license-server=〈host〉:〈port〉

Then the license is installed.

37.1.2 Getting an EMMA plugin

Coco uses the EMMA-XML format to communicate the coverage results to Jenkins. Jenkins therefore needs a plugin todisplay EMMA data. We use here the plugin at https://wiki.jenkins-ci.org/display/JENKINS/Emma+Plugin.Install it in Jenkins with the Jenkins plugin manager.

37.2 Adaption of the project to Jenkins

We will assume here that you already compile and run your project under Jenkins and that you can also generatecoverage information, but not yet under Jenkins.





https://wiki.jenkins-ci.org/display/JENKINS/Emma+Plugin

Adaption of the project to Jenkins

With the license installed, code coverage should now work under Jenkins.

The build process must now be configured in such a way that a single .csmes file is generated that contains all theexecution data. (This is done with cmcsexeimport.)

Now extend your build process such that at the end a report im EMMA-XML format is generated. This is done bycalling

"〈Windows Coco〉\cmreport" -m project.csmes --emma=report.xml

where project.csmes must be replaced with the name of your results file.

We then use the plugin to display the data that were written to the file report.xml. To do this, select the configurationsection of your Jenkins project and add a post-build action. Since the plugin is installed, the "Add post-build action"menu contains an entry "Record Emma coverage report". Select it, and in the field "Folders or files containing EmmaXML reports" enter "report.xml".

Then after the next build, a coverage report will be generated.

Figure 37.1: Generating a coverage report from report.html With Jenkins CI.




Part XI Frequently AskedQuestions


Installation

Chapter

38Installation

38.1 Is there a way to deploy Squish Coco on Windows withoutany user interactions?

The Squish Coco installer for Windows supports the following command line options:

/S: runs the installer in silent mode

/SD: runs the installer in silent mode and suppress all popup dialog

/D: specification of the installation directory

For example, to install Squish Coco in background, do:

SquishCocoSetup.exe /SD

For example, to install Squish Coco in background on c:\Program Files\squishcoco, do:

SquishCocoSetup.exe /SD "/D=c:\Program Files\squishcoco"

38.2 The Microsoft® Visual Studio® Add-In is not installed

To verify that Microsoft® Visual Studio® Add-In is loaded by Microsoft® Visual Studio®, do the following:

1. Open "My Document" folder.

2. Open the folder "Visual Studio 2005" (or "Visual Studio 2008")

3. Open the folder "Addins"

4. You should find the files Visual Studio 2005\SquishCocoVs2005AddIn.dll andSquishCocoVs2005AddIn.addin. If the files are missing, please reinstall Squish Coco.

5. Start Microsoft® Visual Studio®.


The Microsoft Visual Studio Add-In is not installed

6. Click on "ToolsÕOptions...". Select the entry Environment - Add-in/Macro Security. Verify that%VSMYDOCUMENTS%\addins is present in the path, and that the macros are enabled.

Figure 38.1: Microsoft® Visual Studio® Option Dialog

7. Click on "ToolsÕAdd-In Manager...". The list of loaded addins is displayed. Verify that Squish Coco ischecked and also that the startup flag is set.

Figure 38.2: Add-In Manager

8. Open a C or C++ project.

9. Click on "ToolsÕCode Coverage Build Mode...". The Microsoft® Visual Studio® Add-In window shouldappear.


Microsoft Visual Studio Add-In is crashing when "My Documents" is on a network drive

38.3 Microsoft® Visual Studio® Add-In is crashing when "MyDocuments" is on a network drive

Due to some security policy, loading a DLL in Microsoft® Visual Studio® from a network share is not allowed. Thesolution is to install the Microsoft® Visual Studio® Add-In on the local drive.

Proceed as follows:

1. Create a folder called "C:\VisualStudioPlugins"

2. Start Visual Studio. Open the option dialog and add "C:\VisualStudioPlugins":

(a) Click on Tools MenuÕOptions.

(b) Select EnvironmentÕAdd-In/Macro Security.

(c) Add the path "C:\VisualStudioPlugins".

3. Move the contents of "My Documents\Visual Studio 2010\Addins\*SquishCoco*" to"C:\VisualStudioPlugins".

4. Restart Microsoft® Visual Studio®.

38.4 Installation problems on macOS 10.12

Squish Coco is not yet considered a trusted source by macOS, therefore it is not yet possible to install it directly onnewer versions of macOSby clicking on the .pkg file.

However, there is a simple method to circumvent this problem:

1. Locate the downloaded .pgk file in the Finder.

2. Click on it with the Ctrl key pressed.

3. A menu appears. In it, select the “Open” entry.

Then a dialog box appears which allows you to override the trust settings for this file. Afterwards, the installationbegins as usual.

38.5 Code coverage for C# is no longer working

If the C# compiler is updated, the Squish Coco setup for code coverage is overwritten and code coverage no longerworks. The Squish Coco installer must then be run again (see Chapter 2.3, page 10).


Compiling

Chapter

39Compiling

39.1 Using CoverageScanner with ccache (Unix only)

ccache is a compiler cache which permits to speedup the recompilation of objects. To use CoverageScanner withccache it is necessary to set the PATH variable as followings:

export PATH=/opt/SquishCoco/wrapper/bin:/opt/ccache/bin:$PATH

Create then a ccache wrapper got g++ and gcc as follows (as root):

mkdir -p /opt/ccache/binln -s /usr/bin/ccache /opt/ccache/bin/g++ln -s /usr/bin/ccache /opt/ccache/bin/gcc

Which this configuration, calling gcc will:

1. call CoverageScanner’s gcc wrapper

2. which call ccache’s gcc wrapper

3. which finally call the real gcc binary.

39.2 Compiling issue when using the Boost library

When compiling a project using the boost library, the following errors occurs during the compilation:5>c:\boost\boost_1_47_0\boost\type_traits\type_with_alignment.hpp(294):

error : col 2:syntax error5>Error:Could not insert insert instrumentation in file test.cpp

The project is instrumenting the boost library, it is necessary to disable the instrumentation of it. Simply add--cs-exclude-file-abs-wildcard="c:\boost\boost_1_47_0\*" to the list of the compiler command linearguments.







Microsoft MSBuild Does Not Call CoverageScanner on x86 Platforms

39.3 Microsoft® MSBuild Does Not call CoverageScanner onx86 Platforms

Proceed as follows:

• Open "Visual Studio x64 Win64 Command Prompt (2010)" console.

• Add the PATH to Squish Coco Microsoft® Visual Studio® compiler wrapper for x64 platforms:

set PATH=%SQUISHCOCO%\visualstudio_x64;%PATH%

• Build your project with the property UseEnv set to true:

msbuild /p:UseEnv=true myproject.sln /t:ReBuild


Instrumentation

Chapter

40Instrumentation

40.1 Compiling Issues of Lambda Expressions with Microsoft®

Visual Studio® 2010?

Microsoft® Visual Studio® 2010 des not provide a complete support of C++11 and so expressions in the form “[](int x){ return x; }” may not compile once instrumented. To support such expressions, Squish Coco providesan experiemental instrumentation than can be activated through the command line option --cs-vs2010-lambda.

40.2 How to exclude a source file from the code coverageanalysis?

Excluding some source files from the code coverage analysis needs to be performed during the compilation. Twomethods are possible:

1. Compile using CoverageScanner with the command line option --cs-exclude-file-regex,--cs-exclude-file-wildcard and --cs-exclude-path.

2. Use CoverageScanner pragmas to deactivate the instrumentation, by adding the following lines after all#include commands.


#pragma CoverageScanner (cov-off)#endif

i Adding this portion of code at the top of the source file (and so before all #include commands), willalso deactivate the coverage analysis of the headers.


My source code contains inline functions, are they instrumented?

40.3 My source code contains inline functions, are theyinstrumented?

Inline functions are instrumented like other functions. To avoid instrumenting templates and inline functions of 3rdparty library or the standard C++ library, CoverageScanner instrument all functions and procedures of the sourcefile and the headers which are not present in the system directories. On UNIX®, this correspond to /opt, /usr and$QTDIR. On Microsoft® Windows, this correspond to the standard application installation directory and the QTDIRpath. If its header file is in these directories, it is necessary to tell CoverageScanner to instrument it too. The commandline options --cs-include-file-regex, --cs-include-file-wildcard and --cs-include-path permits toforce the instrumentation of additional files.

40.4 Is it normal that the .csmes file contains a copy of thesource code?

Yes. The .csmes file contains all information necessary for CoverageBrowser. That’s why the source code and thepreprocessed source file is also included.


CoverageBrowser

Chapter

41CoverageBrowser

41.1 Is there a way to minimize the CPU usage?

CoverageBrowser needs a lot of CPU resources and disk accesses to calculate the statistics for each execution. Thiscalculation is performed in background, but can also be deactivated. Just proceed as follows: hide the columnCoverage of the docking window "Executions" using the context menu "Show/Hide ColumnsÕCoverage".

41.2 When I’m exporting the statistics per methods I can onlysee the 65536 first entries

This is a limitation of Microsoft Excel or OpenOffice. Excel 97 can only display 16384 rows. Excel 2000 and Excel2003 can display up to 65536 rows and newer version of Excel can display up to 1048576 rows. OpenOffice Calc islimited to using 32000 rows.


Unit Tests

Chapter

42Unit Tests

42.1 How to get the code coverage of a library from my unittests?

We suppose that the project contains N libraries in the directories libprojN. Each library generated are static libraries,and they are generated in libprojN. We have N unit tests, all of them are located in testprojM directories.

So the directory tree looks as followings:libproj1| libproj1.a+-- libproj1.a.csmes

libproj2| libproj2.a+-- libproj2.a.csmes

......libprojN| libprojN.a+-- libprojN.a.csmes

testproj1| testproj1.exe+-- testproj1.exe.csmes

testproj2| testproj2.exe+-- testproj2.exe.csmes

.....testprojM| testprojM.exe+-- testprojM.exe.csmes

First we need to execute the unit tests:

for TESTDIR in testproj*do

cd $TESTDIR./$TESTDIR.exe


How to get the code coverage of a library from my unit tests?

cd ..done

After execting this bash script, a .csexe file is created for each unit test:testproj1| testproj1.exe| testproj1.exe.csexe+-- testproj1.exe.csmes

testproj2| testproj2.exe| testproj2.exe.csexe+-- testproj2.exe.csmes

.....testprojM| testprojM.exe| testprojM.exe.csexe+-- testprojM.exe.csmes

We import then the testprojM.exe.csexe file into its testprojM.exe.csmes:

for TESTDIR in testproj*do

cmcsexeimport -m $TESTDIR/$TESTDIR.exe.csmes -e $TESTDIR/$TESTDIR.exe.csexe \--delete --title="$TESTDIR"

done

Now, the files testprojM.exe.csmes are containing the coverage information of the unit tests and the librariestested. The file testprojM.exe.csexe is deleted since that it is no more used.

It is now possible to import the coverage information of the unit tests in the .csmes file of each library tested:

TESTPROJs=$(ls testproj*/testproj*.exe.csmes)for LIBRARY_DIR in libproj*do

LIBPROJ=$LIBRARY_DIR/$LIBRARY_DIR.a.csmescmmerge -a -o $LIBPROJ -i $LIBPROJ $TESTPROJs

done

This command imports all unit test results into each libprojN.a.csmes. The -i command line option permits toignore the source files which are not present into libprojN.a.csmes. This permits to ignore the test code during themerge process.


Reports

Chapter

43Reports

43.1 Is is possible to compute a code coverage report ofselection of files?

cmreport has the possibility to exclude files from a code coverage report. If we want for example to compute a reportof all files present only in mysrc directory, then proceed as followings:

1. First, exclude all sources files: the command --exclude-file-abs-wildcard="*" exclude all source files.

2. Finally, include all sources files from the directory mysrc: the command--include-file-abs-wildcard="*/mysrc/*" include all source files whose absolute path constains/mysrc/.

A complete command would be:

cmreport --csmes=project.csmes \--html=project.html \--exclude-file-abs-wildcard="*" \--include-file-abs-wildcard="*/mysrc/*"


Miscellaneous

Chapter

44Miscellaneous

44.1 Is it possible to debug the CoverageScanner API?

To debug the functions provided by the CoverageScanner API it is necessary to:

1. force CoverageScanner to keep the generated source file after compiling and linking. Therefore it is necessaryto add --cs-keep-instrumentation-files to the compiler and linker command line arguments.

2. compile the CoverageScanner library with debug information. Therefore it is necessary to add --cs-libgen=-g for GCC (resp. --cs-libgen=/Zi for Microsoft® Visual Studio®) to the linker command line option.

Once the application is recompiled, it is possible to set a breakpoint in __coveragescanner_save() and stepthrough the source code.

44.2 How to report an issue concerning CoverageScanner?

It is possible to generate a log file which allows froglogic to investigate issues concerning CoverageScanner.

Proceed as follows on Microsoft® Windows:

1. Click on StartÕProgramsÕSquish CocoÕSupportÕCoverageScanner Log Recording This will in-stall a debug version of CoverageScanner which will produce a set of log files.

2. Reproduce your issue.

3. In the console window, press enter to install the normal version of CoverageScanner.

4. The log files are located in %SQUISHCOCO%\logfiles. Zip it together and send it to Squish Coco support(k [email protected]).

Proceed as follows on Linux™:

1. In a console window:

$ cd /opt/SquishCoco/{version of Squish Coco}$ mv coveragescanner coveragescanner.bak


How to report an issue concerning CoverageScanner?

$ cp coveragescannerdbg coveragescanner

This will install a debug version of CoverageScanner which will produce a set of log files.

2. Reproduce your issue.

3. Restore the release version of CoverageScanner:

$ cd /opt/SquishCoco/bin$ mv coveragescanner.bak coveragescanner

4. The log files are located in /tmp/squishcoco or %TEMP%\SquishCoco_logfiles. Zip it together and send itto Squish Coco support (k [email protected]).

! The log file may contain parts of your source code. If this is an issue, you may remove the concernedlines with your favorite editor.


Part XII Appendix


Release Notes

Appendix

ARelease Notes


Squish Coco v4.0.3

A.1 Squish Coco v4.0.3

Changes between Squish Coco v4.0.2 and Squish Coco v4.0.3:

CocoQML:

• New Feature: An experimental add-on for coverage of QML code is now available.

CoverageScanner :

• New Feature: The option --cs-no-line-directive now also works for C#.

• Speed improvement of the execution report generation.

Squish Coco:

• Bug Fix: ARM® Keil® µVision v5.23 support. (The installation is now performed by the "Build Environ-ment Selection" tool)



CoverageScanner :

• Bug Fix: Compilation issues fixed for projects with precompiled headers and MCC or MC/DC instrumen-tation.



Microsoft® Visual Studio® Add-In:

• Bug Fix: Coverage settings of C# projects are now applied to all platforms of a specific configuration.

CoverageScanner :

• Bug Fix: Compilation issues fixed for C# projects instrumented with MCC and MC/DC metric.



Squish Coco:

• New Feature: New coverage metrics MCC (Multiple Condition Coverage) and MC/DC (ModifiedCondition/Decision Coverage)

CoverageBrowser :


Squish Coco v3.4.1

• New Feature: Difference analysis between two software versions will highlight changes in instructionsonly and ignore changes in source code comments.

• Bug Fix: Minor GUI usability improvements.

cmreport:

• New Feature: The EMMA-XML report format has been extended to support Decision, MC/DC and MCCcoverage. A new dedicated Jenkins EMMA-XML plugin is available.

• New Feature: Difference analysis between two software versions will highlight changes in instructionsonly and ignore changes in source code comments.

CoverageScanner :

• New Feature: To avoid potential overflows the execution counters are now 64 bits wide. The size can bechanged with --cs-counter-size.

• New Feature: --cs-mcc and --cs-mcdc switches activate instrumentation for all code coverage metricsup to MCC.



CoverageScanner :

• New Feature: The CoverageScanner API will propagate the test name, status and comments from anexecutable or library to all linked DLLs and subordinate DLLs.

cmcsexeimport:

• New Feature: Upon import a warning will be emitted if test meta-information (name, status and comments)is found to be incoherent.

License Server:

• Bug Fix: Crash fixed with new license key formats.



cmreport:

• New Feature: HTML Tree view of source and functions.

CoverageScanner :

• New Feature: Coverage for the cases in a switch statement is now counted separately even if the case con-tains no statements. The old (less precise) behavior can be restored with --cs-combine-switch-cases.

• New Feature: The Decision Coverage level is now available as a distinct choice without the need to turnoff Condition Coverage through --cs-decision.




Squish Coco v3.3.3

• To align naming of coverage levels with other vendors, --cs-branch is now called --cs-basic-blocks.Similarly, the C# pragma and region cov-branch have been renamed to cov-basic-block. The oldnames are still recognized for backward compatibility. If coverage analysis of ‘empty’ branches is desired,use Decision Coverage.

CoverageBrowser :

• New Feature: Reverse patch analysis. With this feature, a patch file that is already part of the code can beused to find out which tests cover the patched code.

• New Feature: Bug location: computation of the list of source code lines most likely responsible for afailure in the test suite. (see Chapter 19.5, page 136)

• New Feature: The computation of the optimized execution order now occurs in the background.

• New Feature: The dialog to import an execution report has now an integrated preview.

• New Feature: Simplified report generation dialogs.



Tool Selector :

• New Feature: Generation of compiler wrappers for CygWin and MinGW’s compilers.

cmreport:

• New Feature: Splitting long lists over several pages in the HTML report.


• Bug Fix: Installation issue on Microsoft® Visual Studio® 2005 fixed.



cmcsexeimport:

• New Feature: Wildcards are supported to specify the list of execution reports.Example: cmcsexeimport -m project.csmes --title="execution" *.csexe

cmmerge:

• New Feature: Wildcards are supported to specify the list of input databases.Example: cmmerge -o output.csmes inputs_*.csmes

CoverageScanner :

• Bug Fix: Support of C++11 shared_ptr template in expressions.

• Bug Fix: C# 6.0 auto-property initializer supported.


• New Feature: Microsoft® Visual Studio® 2015 support.


Squish Coco v3.3.1



Licenses:

• Bug Fix: Simplified handling of node-locked licenses under UNIX® and macOS.A Coco program with a node-locked license now expects that the license had been generated on the sameaccount as that one under which it runs.Previously, the account to which a license was bound was that one under which the user that started it hadinitially logged in. Changing the account with su or sudo had no influence on the license. This meansthat under some circumstances a license was generated for an account different from that under whichSquish Coco ran, but it worked nevertheless. In these cases, the license must now be renewed.

CoverageBrowser :

• Bug Fix: Patch analysis: Issue related to parsing of date formats from different time zones resolved.



CoverageBrowser :

• Speed improvement in the detection of duplicate executions.

• Speed improvements in the merge of instrumentation databases, the generation of HTML reports, thecomputation of the list of executions that execute a source line, and in the computation of the codecoverage metrics.

• New Feature: Import of comments and manual validations from projects which are checked out in differentdirectories.

• New Feature: Patch analysis.

CoverageScanner :

• Bug Fix: Full instrumentation of the C/C++ “ternary if” operator: In an expression a ? b : c, allboolean operands in the expressions a, b or c are instrumented too. Nested ternary operators are thereforeinstrumented completely.

• New Feature: Automatic exclusion of the coverage analysis of all headers which are inconsistent (forexample, built with different preprocessor options). A linker warning lists all files which are excluded.

cmcsexeimport:

• New Feature: Speed improvement of the detection of duplicate executions.

cmreport:

• New Feature: Patch analysis.

• New Feature: In the HTML report, a list of the tests which execute a source code line is displayed in thetooltip.

• New Feature: A list of the executions that cover an instrumented statement is displayed in the tooltip for asource code line.

cmmerge:


Squish Coco v3.2.3

• General speed improvements.

• New Feature: Importing comments and manual validations from projects which are checked out indifferent directories.



CoverageScanner :

• Bug Fix: Memory usage reduced. This fixes an "out-of-memory" error message during the compilation.

• New Feature: Explicit initialization of the CoverageScanner API is no longer necessary for C# DLLs.(The program InjectModuleInitializer.exe is no longer needed)



CoverageScanner :

• Bug Fix: More meaningful instrumentation of the C# keywords using, lock, unsafe and fixed.

• New Feature: If the debugger is connected, CoverageScanner runtime information is displayed in theMicrosoft® Visual Studio® console.

• New Feature: In C# files, the “dump on event” feature can be enabled through a macro definition.


• New Feature: Configuration of the debug output of Squish Coco.

• New Feature: Configuration of the Windows event handler.



CoverageScanner :

• New Feature: Support of Shift-JIS text encoding.

• New Feature: Execution report of Silverlight C# binaries is stored on an Isolated Storage when storing onthe local file system is not allowed.

• Bug Fix: Support of strong-named C# assemblies.

CoverageBrowser :

• New Feature: The Preferences dialog now allows to switch off syntax highlighting.


Squish Coco v3.2.0



CoverageScanner :

• New Feature: Precise error message when a merge failure occurs.

CoverageBrowser :

• New Feature: Thresholds for the overall statistics.

• New Feature: Statistics of manual validated instrumentation (for each function, source code and for thecomplete project).

cmreport:

• New Feature: Statistics of manual validated instrumentation (for each function, source code and for thecomplete project) for the HTML and CSV report.

cmmerge:

• Speed improvements.

• New Feature: --delete permits to delete all input files after a successful merge.

• New Feature: --blackbox permits to generate an instrumentation database for black-box testing.



Squish Coco:

• New Feature: NUnit addin.

Tool Selector :

• Bug Fix: Detection of CygWin installations for x64 systems corrected.

CoverageScanner :

• VisualDSP® support.

• Bug Fix: Support of UTF16 source files.

• New Feature: For embedded systems, CoverageScanner is able to generate execution reports in 8.3 format.In this case the report extension is .cse and not .csexe.

• New Feature: New command line switch --cs-no-abort-on-error: if used, instrumentation errorsraised while processing C/C++ source code no longer abort the build.

cmreport:

• New Feature: Refactoring of the HTML report:– New Feature: Single HTML file and full HTML report have now the same look and feel.– New Feature: Source directories statistics in a clickable tree view.– New Feature: Class directories statistics in a clickable tree view.– New Feature: Statistic tables are interactively sortable.– New Feature: Tool-tip for a detailed information on each table item.



Squish Coco v3.0.2

• New Feature: CSV table export for all informations that can be exported through a HTML report.• New Feature: The source, execution and function dialog can now be interactively switched between a list

and tree view.• New Feature: CoverageBrowser reacts automatically when the instrumentation database file is regenerated

and reload it automatically.

CoverageBrowser :

• New Feature: The source, execution and function dialog can now be interactively switched between a listand tree view.

• New Feature: CoverageBrowser reacts automatically when the instrumentation database file is regeneratedand reload it automatically.



Squish Coco:

• New Feature: Solaris support.

CoverageBrowser :

• Bug Fix: EMMA-XML Report generation corrected when excluding source files.• New Feature: Drag and drop of .csmes and .csexe files supported.

CoverageScanner :

• New Feature: Format specifier %c added to --cs-output.• Bug Fix: Issues with strings in __declspec extension fixed.• Bug Fix: Deal with non-instrumentable OpenMP code.• Bug Fix: C++11 string literal support. (example: R"delim(")delim")• Bug Fix: UTF-8 source file can now be compiled.• Bug Fix: C++ Line comments ending a source file are no more generating a parsing issue.• Bug Fix: C# linq expressions are now instrumented.• New Feature: New C# API function CoverageScanner.__coveragescanner_init().• New Feature: Dead code detection in constant boolean expressions:

– Dead code detection for if ( false ) ... or for if ( true ) ... branches.– Expression do statement while ( false ) is treated if the statement was not included into ado ... while ( ... ) construct.

– Dead code detection of while ( false ) .... loops.– Statements ’while ( true ) ...’ and ’do ... while ( true );’ are now treated as the infi-

nite loop ’for(;;) ...’. The conditions are no more instrumented to avoid false negative coverageinstrumentations.

• Bug Fix: New command line switch for C++ mixed mode: --cs-architecture=CLR• Bug Fix: Several parsing issue for C# and C++ code fixed.• Bug Fix: Support of multiple header files included with/FI when generating precompiled headers.• Bug Fix: C# support for Microsoft® Visual Studio® 2013.

Tool Selector :

• New Feature: generation of all wrappers for all installed version of GCC compilers.



Squish Coco v3.0.1



CoverageBrowser :

• New Feature: Validating manually an instrumentation validates also all depending instrumentations. Forexample, when marking a return statement as validated, all sequential lines before are also automaticallyvalidated.

CoverageScanner :

• New Feature: Source code annotation though code comments.

• Bug Fix: clang compiler support on macOS.


• Bug Fix: Excluding source directories from the instrumentation is working again.


Changes between Squish Coco v3.0.0-pre1 and Squish Coco v3.0.0:

CoverageScanner :

• Bug Fix: Line coverage information of nested compound statements are now computed correctly.

• Change: --cs-include-path/--cs-exclude-path works now recursively on all subdirectories.

• New Feature: --cs-output supports now format specifiers (%f, %P, . . . )

• Bug Fix: lib.exe wrapper support now Module-Definition (.def) files

• Bug Fix: Compilation of expression in the form CLASS c = a && b are now supported even if theoperator || or && are not logical operators.

CoverageBrowser :

• New Feature: Disabling the difference view of a source file is possible.

• New Feature: Importing reviewer comments from a previous instrumentation database.

cmmerge:

• New Feature: Importing reviewer comments (–reviews-only command line option)

A.19 Squish Coco v3.0.0-pre1

Changes between Squish Coco v2.1.8 and Squish Coco v3.0.0-pre1:

CoverageBrowser :

• New Feature: Usability enhancement of the folding functionality in the source window:– possibility to define a number of lines before and after the folded line part of the context and which

are not folded.


Squish Coco v2.1.8

– folded lines are chosen the instrumentation filter dialog.

• New Feature: For the code coverage at branch or decision/condition level, the executed (resp. notexecuted) lines which are not instrumented are displayed in pastel green (resp. pastel red).

• New Feature: Line coverage information is now also available in the preprocessed view of the sourcecode.

• New Feature: Automatic copying to the clipboard of a selected source code fragment.

• New Feature: Generation of Cobertura reports.

CoverageScanner :

• New Feature: simple support of GNU linker script for MinGW.

• New Feature: clang support on macOS.

• New Feature: Configuration of instrumentation warnings and errors.

• Bug Fix: throw statement are now instrumented in expressions. (example:int a = b ? 0 : throw 0 ;)

• Bug Fix: /Yc and /Yu can now be used simultaneously for the Microsoft® Visual Studio® compiler.

• New FeatureBETA: Tcl support.

• New FeatureBETA: C# support.

cmreport:

• New Feature: Generation of Cobertura reports.



CoverageBrowser :

• New Feature: Configuration of the color scheme.

• New Feature: Excluding source files interactively from the instrumentation.

• New Feature: Configuration of the maximum number of threads which can be used for a computation.

• New Feature: Progress information during the generation of a code coverage report.

CoverageScanner :

• New Feature: Better detection of dead code after a switch/case statement.

• New Feature: Optimized handling of C++ header which permits to reduce the size of the instrumentationdatabase and the time for generating reports.

cmreport:

• New Feature: Excluding source files from the report generation.

• New Feature: Configuration of the maximum number of threads which can be used for a computation.

• Change: Function parameter are now appended to each function in the EMMA-XML report.

cmcsexeimport:

• New Feature: cmcsexeimport is now able to import more than one execution report file.



Squish Coco v2.1.7



CoverageBrowser :

• Change: Navigation buttons are now grouped into one toolbar.

• Speed improvement of the source code viewer.

• Speed improvement of the computing of the list of tests which are executing a source line.

• Speed improvement of the of the text search in the source code function.

• New Feature: Source code viewer:– New Feature: Sources are displayed in a tabbed view.– New Feature: Preview of the instrumentation of a whole file into the scroll bar.– New Feature: Tooltip which displays the detail of an explanation of a source line.– New Feature: More intuitive text search function (search when typing into the source code window)

and interactive highlighting of the matched patterns.– New Feature: Dock/undock line number column and statistic column.

CoverageScanner :

• Bug Fix: CoverageScanner is now able to handle "Whole Program Optimization" (option /GL) and "UseLink Time Code Generation" (option /LTCG) of Microsoft® Visual Studio®.

• New Feature: Command line option --cs-vs2010-lambda which permits to instrument lambda functionsof Microsoft® Visual Studio® 2010

• New Feature: Support for "for each" C++ syntax extension of Microsoft® Visual Studio®.

• Bug Fix: Support of ccache wrapper.

• Bug Fix: CoverageScanner detects now dead code after a try/catch block.

• Bug Fix: Precompiled header issues.

• Bug Fix: Handling source code lines which are ending with a backslash.

cmreport:

• Speed improvement of the generation of EMMA-XML reports.

• Speed improvement of the generation of HTML reports.

• New Feature: HTML report displays the detailed instrumentation report in a tooltip.

• Change: HTML report is generated in a directory whose name ends with _html. This avoids a nameclash under Unix where the executable has in most of the cases no extension.



CoverageBrowser :

• New Feature: Possibility to choose the source code type used for comparing software releases (originalsource code or preprocessed source code)

CoverageScanner :



Squish Coco v2.1.5

• New Feature: New command line options which permits to specify command line arguments into a textfile (--cs-option-file).

• New Feature: New command line options which permits to specify to select directly the CoverageScannerprofile (--cs-profile).

cmreport:

• New Feature: possibility to customize the EMMA-XML report



CoverageScanner :

• New Feature: Support for precompiled headers for Microsoft® Visual Studio® compiler.

• New Feature: Xoreax IncrediBuild support (http://www.incredibuild.com/incredibuild-product-overview.html)

• Bug Fix: Wildcard expressions on relative paths are computed on the canonical path.(example: --cs-include-file-wildcard=include/* matches include/foo.h andmodule/../include/bar.h)

• New Feature: New command line options which permits to specify which extensions are used forC++ (--cs-cpp-ext) and C (--cs-c-ext) files.

• New Feature: New command line option which permits to set the tool chain path(--cs-native-toolchain).

• Change: Verbose command line option (--cs-verbose) permits to select if the build, the instrumentationor the CoverageScanner API is instrumented. The verbose output of the instrumented file is generatedinto a log file (name of the execution report with the extension.cslog).

• Bug Fix: -m32 and -m64 command line support for GCC.

• Bug Fix: Compilation issue on Microsoft® Visual Studio® for instrumented C++0x lambda functioncorrected.

• Bug Fix: Support of __stdcall as default calling convention when compiling with Microsoft® VisualStudio®.

• New Feature: New command line option for UNIX® platform which permits to generate a coverage reporton the reception of a signal (--cs-dump-on-signal).

• New Feature: New command line option for Microsoft® Windows platform which permits to generate acoverage report on the reception of an event (--cs-dump-on-event).

• Change: Instrumentation tables of libraries are chained together during the runtime whencompiling with Microsoft® Visual Studio®. The old behaviour can be restored with--cs-link-instrumentation-tables.

cmreport:

• Change: cmreport generate now per default a code coverage report at decision/condition level with linecoverage for uncovered lines.


• Change: Microsoft® Visual Studio® Add-In does not disable the precompiled headers anymore.



http://www.incredibuild.com/incredibuild-product-overview.html

http://www.incredibuild.com/incredibuild-product-overview.html

Squish Coco v2.1.4



CoverageBrowser :

• New Feature: Import summary display the imported execution comment.

• New Feature: New instrumentation mode available which permits to displays the computed line coverageof uninstrumented lines. This enables the readability of the source code but have no impact on thecoverage statistic.

CoverageScanner :

• Bug Fix: Support for /clr:safe and clr:pure command line option.

• Optimization: CoverageScanner calls now directly the native compiler without calling a second instanceof it.

• Bug Fix: The line coverage was wrongly calculated before some switch/case statements.

License Manager:

• New Feature: The license manager is now a wizard dialog.

• New Feature: Command line tool to activate a license on Windows and Unix



CoverageScanner :

• New Feature: /MP command line (parallel build) supported for Microsoft® Visual Studio® compiler.

• Bug Fix: Parsing issues of C++11 templates corrected.

• New Feature: The Build Environment Selection tool generate CoverageScanner wrapper for Cyg-Win.


• New Feature: Since that CoverageScanner instruments now all non system folders, the selection dialog ofthe list of directories to include is removed. A dialog which permits to select the list of directories toexclude replace it.

Linux™ installer:

• New Feature: Possibility to specify a custom installation directory.

cocolic:

• New Feature: Command line interface on Unix.

• Bug Fix: More verbose error message in the case of network failures.




Squish Coco v2.1.2



CoverageScanner :

• Bug Fix: Multi line #pragma supported.

• Bug Fix: ::new keyword supported.


• Bug Fix: Support of solutions with folder.

• Bug Fix: Support of solutions which mix C++ and C# projects.



CoverageScanner :

• Change: CoverageScanner instruments now per defaults all sources files except those which arein the system directory (/usr, /opt or C:\Windows) or the application installation directory(C:\Program Files).

• New Feature: Automatic detection of the runtime library necessary when compiling using Microsoft®

Visual Studio®. Adding /MD or /MT to the build command argument of the CoverageScanner library is nomore necessary.

• Bug Fix: Crash when using wildcard expression on file name on UNIX®.

• New Feature: The instrumentation table are now generated during the runtime. This permits to handlestubs, which overwrite objects from libraries.

• New Feature: Constructor which use member initialization list are instrumented even if their body is doesnot contain code.

• New Feature: C++ CLR language extension of Microsoft® Visual Studio® supported.

CoverageBrowser :

• New Feature: Navigation buttons are disables if they are irrelevant. After reaching the end of search, thenavigation buttons start from the beginning again.

• Bug Fix: Clicking on a function in the method list, show it entirely in the source window.

• Bug Fix: Statistics count was wrongly computed when the instrumentation is computed with line coveragesupport and the test count mode was activated.

• New Feature: Importing an execution report does not generate an error message any more. If someexecutions cannot be imported, the information window is automatically shown. A list of importedexecutions and errors are displayed on it.

• New Feature: Windows x64 package available.

cmreport:

• New Feature: EMMA-XML report generated now condition coverage statistics. This requires JenkinsEMMA-XML plugin version 1.29 or above.

• New Feature: JUnit report generation.




Squish Coco v2.1.0



Squish Coco:

• New Feature: Recognition of dead code inside a function.

• New Feature: Function coverage.

• New Feature: Line coverage.

CoverageScanner :

• Code coverage count is enabled per default.

cmreport:

• Bug Fix: EMMA-XML report corrected: the line field correspond now to the line coverage.



Squish Coco:

• Bug Fix: License activation issues.



cmreport:

• Improving the speed of the production of HTML reports through parallelization of the generation of thesource code view.

CoverageScanner :

• Bug Fix: __try and __finally keyword now handled on Windows for C files.

• Bug Fix: g++ compiler wrapper treats files with .c extension as C++ files.

• Bug Fix: On Windows, command line containing directory name which ends with a backslash are nowcorrectly escaped.

• Bug Fix: ar and ld wrapper are now working, and do not report that the profile cannot be found.

• New Feature: The code behind the macro BOOST_FOREACH is no more analyzed, unless the command line--cs-no-boost is used.

• New Feature: New command line option --cs-output-abs.

• Change: The support for Qt3, Qt4 and Boost is per default activated. Disabling it can be performed usingcommand line switches --cs-no-qt3, --cs-no-qt4 and --cs-no-boost.

• Change: The full instrumentation at decision and condition level is now the default setting for theinstrumentation.



Squish Coco v2.0.1



CoverageScanner :

• Bug Fix: Support setting define which contains escaped quotes per command line (ex:"-DVERSION=\"version 1.1\"")

• Bug Fix: Compilation issue of TextEdit sample when spaces are in the path corrected.

License Dialog:

• Bug Fix: Retrieving a license key is now interruptible and displays a progress bar.


Changes between TestCocoon v1.6 and Squish Coco v2.0.0:


• New Feature: Support for Microsoft® Visual Studio® 2010.• Bug Fix: x64 platform are now recognized and handled.

CoverageBrowser :

• New Feature: Supporting linking several times the same object into an application.• New Feature: Possibility to filter out execution which does not cover any code.• New Feature: Possibility to filter out source files and functions which are not instrumented.• Bug Fix: Calculation of statistics is able to saturate all available CPU.• Bug Fix: After opening a new instrumentation database, no executions become selected.• New Feature: Possibility to choose between the native file dialog and the dialog provided by Qt.• New Feature: Displaying the execution of a reference release into the list of execution. Executions of the

reference release are strikeout, new executions are underlined.• New Feature: Generation of browsable HTML output.

CoverageScanner :

• Empty functions are no more instrumented.• New Feature: C++0x support.• New Feature: Verbose output of the CoverageScanner output when using command line option--cs-verbose.

• New Feature: Saving the contents of a tree view waits until all statistics are computed.• New Feature: Q_FOREACH macro es now handled to avoid false negative code coverage statistics. Once a

look is executed once, the Q_FOREACH macro is considered as 100% covered.

cmcsexeimport:

• New Feature: Execution comments in HTML form are parsed from the execution report. This permits toadd extra log information to the execution report.

cmreport:

• Bug Fix: Correction of an EMMA-XML issue: global statistics are now ordered in EMMA-XML report asfollowing: Line, Method, Class and then Block

• New Feature: Generation of browsable HTML output.




File system and registry

Appendix

BFile system and registry

This chapter is about the usage Squish Coco makes of the file system to store permantent settings.

B.1 Location of the installed files

The location of the installed files varies according to the platform.

B.1.1 Microsoft® Windows

The location of the programs can be chosen at installation time. The installer creates and environment vari-able SQUISHCOCO that contains the path to the installation directory. By default it is either C:\Program Files\squishcoco or C:\Program Files (x86)\squishcoco, depending on the platform. In this documentation weuse the expression 〈Windows Coco〉 to refer to it.

This directory contains the following files and directories:

• The standard Squish Coco executables, like CoverageBrowser, cmcsexeimport, etc.

• The documentation.

• The prefix versions of the wrapper programs and their configuration files. The prefix version of a compilerwrapper is a file that begins with “cs”, like cscl.exe as wrapper for cl.exe. The configuration file for thisprogram is then cl.cspro.

• Directories with specific versions of the compiler wrappers.

An example are the directories visualstudio and visualstudio_x64 which contain the wrappers forMicrosoft® Visual Studio®. These directories often contain compiler wrappers without a cs prefix: There ise.g. a program cl.exe in the directory visualstudio that wraps the program cl.exe of Microsoft® VisualStudio®.

• Directories with tutorial and example programs, like parser, textedit and tutorial.


Location of the license

B.1.2 Linux™

The location can be chosen at installation time; its default value is /opt/SquishCoco/. The directory has thefollowing subdirectories:

bin/, lib/:The Squish Coco binaries, including the prefix versions of the compiler wrappers and their profile files.

wrapper/bin/:The compiler wrappers as files without the cs prefix.

doc/:Documentation files.

samples/:Example programs and files for the tutorials.

B.1.3 macOS

The programs are installed at /Applications/SquishCoco.

This directory contains the Squish Coco binaries, including the prefixed versions of the compiler wrappers and theirprofile files. It also contains the documentation. It has the following subdirectories:

wrapper/:The compiler wrappers as files without the cs prefix.

samples/:Example programs and files for the tutorials.

B.2 Location of the license

The professional and the non-commercial edition of Squish Coco use different naming schemes to locate the filesand registry keys that contain information about the license. In the following description, the professional edition isprimarily described, with the values that change in the noncommercial edition following in parentheses.

B.2.1 Node-locked licenses

When searching for a license, Squish Coco first tries to find a node-locked license.

1. If the environment variable SQUISHCOCO_LICENSEKEY_DIR is set, Squish Coco treats the content as a directoryname and tries to find there the license file. The license file is named .squishcoco-3-license.

2. If Squish Coco runs under UNIX® or macOS, it then searches for a license file in the directories $HOMEPATH,$HOME and in the root directory, ‘/’.

3. If Squish Coco runs under Microsoft® Windows, it searches instead in the registry under HKEY_CURRENT_USER\Software\squishcoco\LicenseKey.

It is possible that a license is present but that it has been deactivated. If Squish Coco finds such a license, it stopssearching for another node-locked license and tries to find a license server instead.


Location of the program settings

! Setting SQUISHCOCO_LICENSEKEY_DIR globally for all users is not recommended. It would force allusers to use the same license, but each license key is specific to user and host.

B.2.2 Address of the license server

If no node-locked license is found, Squish Coco will try to find the location of a license server.

1. First it tries to read the specification of a license server from the environment variableSQUISHCOCO_LICENSE_SERVER, if such a variable is present and its value is not the empty string.

2. If Squish Coco runs under UNIX® or macOS, it then tries to read the specification from a file .squishcoco-3-licserver.

It searches for this file in the directories $HOMEPATH, $HOME and the root directory, ‘/’.

3. If Squish Coco runs under Microsoft® Windows, then it searches in the registry under HKEY_CURRENT_USER\Software\squishcoco\LicenseServer for a specification.

The specification of the license server consists of a host name or an IP address, possibly followed by a colon and aport number, like ‘myserver.com:49344’.

B.3 Location of the program settings

If a program has other permanent settings that must be stored in a file, that file is located in the following directory:

On Windows:%APPDATA%\squishcoco

On Linux:$HOME/.config/squishcoco

B.4 Location of the installation log file

The Windows installer produces a log file at %TEMP%\SquishCoco_logfile\vsaddin.log. It can be consultedafter installation problems.


Supported compilers

Appendix

CSupported compilers

The following compilers are currently supported by CoverageScanner:

C.1 C# compilers

The command line C# compiler of Microsoft® Visual Studio® .NET and Mono C# compiler are supported.

Native Command CoverageScanner Command’mcs’ ’csmcs’

’gmcs’ ’csgmcs’’dmcs’ ’csdmcs’’csc’ ’cscsc’

C.2 Microsoft® Visual C++

The command line compiler and linker of Microsoft® Visual C++ and Microsoft® Visual C++ Toolkit 2003 aresupported.

Native Command CoverageScanner Command’cl’ ’cscl’

’lib’ ’cslib’’link’ ’cslink’


Intel C++ Compiler

C.3 Intel® C++ Compiler

The C and C++ command line compiler from Intel® is supported.

Native Command CoverageScanner Command’icl’ ’csicl’’icc’ ’csicc’

’icpc’ ’csicpc’

C.4 GNU gcc

Only the g++ and gcc command line compilers are directly supported.

Native Command CoverageScanner Command’gcc’ ’csgcc’’g++’ ’csg++’’ar’ ’csar’

C.4.1 Special compiler versions for cross-compiling

If a GNU compiler is used for cross-compiling, it often has a special name, like ‘arm-linux-gcc’ instead of ‘gcc’.

The installation script of Squish Coco tries to create automatically the compiler wrappers for every GNU compilerinstalled on the machine. In general, all necessary compiler wrappers are therefore present in the system. A cross-compiler may however also reside in a non-standard directory in the system, where it will not be found by the SquishCoco installer.

If a compiler configuration is missing, the following can be done:

1. Copy ’coveragescanner’ to the file ’cs〈compiler〉’.Example: On Microsoft® Windows, this would be something like:

C:\Program Files\squishcoco>copy coveragescanner.exe csarm-linux-gcc.exe

2. Copy the profile ’gcc.cspro’ or g++.cspro’ to ’〈compiler〉.cspro’.

Example:

C:\Program Files\squishcoco>copy gcc.cspro arm-linux-gcc.cspro

The new GNU cross-compiler ’cs〈compiler〉’ can now be used. It instruments the code and then calls ’〈compiler〉’for compilation.


Code insertion

Appendix

DCode insertion

(This chapter is a continuation of the example in Chapter 5.)

CoverageScanner inserts the instrumentation code as illustrated in the following examples. (The inserted code isdisplayed in bold blue.)

Sequential Statement:The statements are instrumented before their execution. The instrumentation consists in allocating a Booleanvariable which detects if the code was executed or not.

Example:

a=foo();a++;break;

will be changed into:

a=foo();a++;{ inst[0]=1; break; }

inst[0] is set to 1 if the ‘break’ statement is executed.

Conditional Statements and Boolean Expressions (full instrumentation):For Boolean expressions the same principle applies except that in addition to recording the execution itself, thestate (true or false) is also recorded.

Example:

if ( a<b )return 1;


if ( (a<b) ? inst[0]=1 : inst[1]=1,0 )return 1;

inst[0] is set to 1 if the Boolean expression a<b was true.

inst[1] is set to 1 if the Boolean expression a<b was false.


Code insertion

Conditional Statements and Boolean Expressions (partial instrumentation):In some cases, recording the value of a Boolean expression is unnecessary for sequential statement instrumenta-tion. For example, the statement if (b)return 0; else return 1; is completely covered by a statementcoverage—after all, recording whether b becomes true or false does not provide any extra information. Simi-larly, for the statement if (b)return 0; it is only necessary to check if b was false. By default, Squish Cocosuppresses the generation of redundant instrumentation in order to minimize the instrumented code’s size andto maximize execution speed.

Example:

if ( a<b )return 1;


if ( a<b )return 1;

else inst[0]=1 ;

inst[0] is set to 1 if the Boolean expression a<b was false. Since the statement coverage records theinstruction return 1;, it is not necessary to record if a<b was true;

The generated instrumentation code that needs to be inserted to provide statement coverage for the earlier foo()function example means that the function’s code is transformed to the code shown here:

1 char inst[5];2 void foo()3 {4 bool found=false;5 for (int i=0; (i<100) && (!found); ++i)6 {7 if (i==50 ) { inst[0]=1;break;}8 if (i==20 ) { inst[1]=1;found=true;}9 if (i==30 ) { inst[2]=1;found=true;}

10 inst[3]=1; }11 printf("foo\n");12 inst[4]=1; }

Figure D.1: Code coverage instrumentation at statement block level

If we insert the instrumentation code necessary to support decision coverage into this example, the resulting code willlook like this:


Code insertion

1 char inst[13];2 void foo()3 {4 bool found=false;5 for (int i=0; ((i<100 && !found)?inst[0]=1:inst[1]=1,0); ++i)6 {7 if ((i==50?inst[2]=1:inst[3]=1,0)) {inst[4]=1; break;}8 if ((i==20?inst[5]=1:inst[6]=1,0)) {inst[7]=1; found=true;}9 if ((i==30?inst[8]=1:inst[9]=1,0)) {inst[10]=1; found=true;}


Figure D.2: Code coverage instrumentation at decision level

If we insert the instrumentation code necessary to support condition coverage into this example, the resulting codewill look like this (except that we have wrapped one line to fit better on the page):

1 char inst[15];2 void foo()3 {4 bool found=false;5 for (int i=0;((i<100)?inst[0]=1:inst[1]=1,0) &&6 ((!found)?inst[2]=1:inst[3]=1,0); ++i) {7 if ((i==50?inst[4]=1:inst[5]=1,0)) {inst[6]=1; break;}8 if ((i==20?inst[7]=1:inst[8]=1,0)) {inst[9]=1; found=true;}9 if ((i==30?inst[10]=1:inst[11]=1,0)) {inst[12]=1; found=true;}


Figure D.3: Full code coverage instrumentation at condition level

Here is what the code would look like if we inserted the partial instrumentation code for condition coverage (again,with one line wrapped):


Code insertion

1 char inst[12];2 void foo()3 {4 bool found=false;5 for (int i=0; ((i<100)?inst[0]=1:inst[1]=1,0) &&6 ((!found)?inst[2]=1:inst[3]=1,0); ++i) {7 if (i==50 ) {inst[4]=1; break;} else inst[5]=1;8 if (i==20 ) {inst[6]=1; found=true;} else inst[7]=1;9 if (i==30 ) {inst[8]=1; found=true;} else inst[9]=1;


Figure D.4: Partial code coverage instrumentation at condition level


Code Coverage Benchmarks

Appendix

ECode Coverage Benchmarks

E.1 Test Algorithm

The sorting algorithm used for the tests is quicksort.Source code:

#define SIZE 10000000#define NB_TESTS 4#include <stdio.h>#include <stdlib.h>#include <sys/times.h>#include <time.h>

void sort (int array[],int size);

main( ){int i,t ;long duration[NB_TESTS];long duration_val;clock_t starttime,endtime ;int *array;

for (t=0;t<NB_TESTS;t++){starttime=clock();array=(int*)malloc(sizeof(int)*SIZE);for ( i = 0 ; i < SIZE ; i++ )array[i]=(i*7)%SIZE;

sort(array,SIZE);free(array);endtime=clock();duration[t]=1000*((double)(endtime-starttime))/CLOCKS_PER_SEC;fprintf(stderr," %dms",duration[t]);

}duration_val=duration[0];


Benchmarks

for (t=1;t<NB_TESTS;t++)if (duration_val>duration[t])duration_val=duration[t];

printf("%d",duration_val);}

static void quicksort ( int array[], int low, int high ){int pos ;if ( low < high ){int item, i, j, t ;item = array[low] ;i = low ;j = high ;while ( i < j ){while ( array[j] > item )j = j - 1 ;

while ( array[i] <= item && i < j )i = i + 1 ;

if ( i < j ){t = array[i] ;array[i] = array[j] ;array[j] = t ;

}}pos = j ;t = array[low] ;array[low] = array[pos] ;array[pos] = t ;

quicksort ( array, pos + 1, high ) ;quicksort ( array, low, pos - 1 ) ;

}}

void sort (int array[],int size){quicksort ( array, 0, size-1 ) ;

}

E.2 Benchmarks


Benchmarks

Compiler Normal Execution(time)

ExecutionBranch Coverageor Function Cover-age(time)

ExecutionDecision CoverageorLine Coverage(time)

ExecutionCondition Cover-age(time)

GCC without optimization 6840ms 8030ms (+17.3%) 9400ms (+37.4%) 9830ms (+43.7%)GCC with optimization -Os 4160ms 4430ms (+6.4%) 6180ms (+48.5%) 6360ms (+52.8%)GCC with optimization -O1 3530ms 4250ms (+20.3%) 5420ms (+53.5%) 5970ms (+69.1%)GCC with optimization -O2 3950ms 4040ms (+2.2%) 5080ms (+28.6%) 5320ms (+34.6%)GCC with optimization -O3 3860ms 4030ms (+4.4%) 5060ms (+31%) 5230ms (+35.4%)

Table E.1: Benchmark (sorting algorithm)

Compiler Native Compila-tion(bytes)

Statement BlockCoverageorFunction Cover-age(bytes)

Decision CoverageorLine Coverage(bytes)

Condition Cover-age(bytes)

GCC without optimization 2128 3452 (+1324) 3996 (+1868) 4052 (+1924)GCC with optimization -Os 1940 3196 (+1256) 3716 (+1776) 3760 (+1820)GCC with optimization -O1 1968 3172 (+1204) 3736 (+1768) 3792 (+1824)GCC with optimization -O2 2100 3452 (+1352) 4096 (+1996) 4160 (+2060)GCC with optimization -O3 3544 3548 (+4) 4364 (+820) 4428 (+884)

Table E.2: Sorting algorithm code size of the object


License Agreement

Appendix

FLicense Agreement

Squish Coco Commercial License AgreementAgreement version 1.1

This froglogic Squish Coco Commercial License Agreement ("Agreement")is made by and between froglogic GmbH ("froglogic"), and you (eitheran individual or a legal entity) ("Licensee").

IMPORTANT-READ CAREFULLY:

1. froglogic developed and originated Squish Coco including sourcecode, documentation and example programs ("Licensed Software"). TheLicensed Software is protected by copyright laws and internationalcopyright treaties, as well as other intellectual property laws andtreaties. froglogic retains all rights not expressly granted.

2. By installing, copying, or otherwise using the Licensed Software,Licensee agrees to be bound by the terms of this Agreement. IfLicensee does not agree to the terms of this Agreement, Licenseeshould not install, copy, or otherwise use the Licensed Software. Inaddition, by installing, copying, or otherwise using any updates orother components of the Licensed Software that Licensee receivesseparately as part of the Licensed Software ("Updates"), Licenseeagrees to be bound by any additional license terms that accompany suchUpdates. If Licensee does not agree to the additional license termsthat accompany such Updates, Licensee may not install, copy, orotherwise use such Updates.

3. Upon Licensee’s acceptance of the terms and conditions of thisAgreement, froglogic grants Licensee the right to use the LicensedSoftware solely for Licensee’s internal business purposes in themanner provided below.

4. Options(a) Evaluation License

Licensee may use the Licensed Software for a limited time(evaluation period) to be determined and stated by froglogic,solely for the purpose of determining whether the LicensedSoftware meets Licensee’s requirements.


License Agreement

After the evaluation period has finished, Licensee must either:

(i) discontinue use of the Licensed Software,(ii) acquire a commercial license (see section (b)),

Commercial use of a derived work of the Licensed Software isnot allowed under the scope of the evaluation license.

(b) Node-Locked LicenseLicensee may install and use the Licensed Software locally onone identified computer (Node), with the Node executing theLicensed Software for only one Authorized User. Licensee maytransfer a license certificate to another individual and Nodeafter notification to and approval from froglogic, but not moreoften than every 4 weeks, provided the future Licensee isemployed or contracted by the same individual or legal entityas the initial Licensee.

(c) Floating User LicenseLicensee may install the Licensed Software on an unlimitednumber of computers. All machines using the Licensed Softwaremust have the ability to communicate with a license server. Thenumber of Floating Users that may use a component of theLicensed Software concurrently at any time is limited by thenumber of Floating User Licenses purchased for suchuse. Regardless of whether such use is by persons or computersystems.

5. Licensee may modify the Licensed Software except altering orremoving any details of ownership, copyright, trademark or otherproperty right connected with the Licensed Software.

6. Licensee may not distribute the Licensed Software, modified orunmodified, separately or as part of any software package.

WARRANTY DISCLAIMER

7. The Licensed Software is licensed to Licensee "as is". To themaximum extent permitted by applicable law, froglogic on behalf ofitself and its suppliers, disclaims all warranties and conditions,either expressed or implied, including, but not limited to, impliedwarranties of merchantability, fitness for a particular purpose, titleand non-infringement with regard to the Licensed Software.

LIMITATION OF LIABILITY

8. If, froglogic’s warranty disclaimer notwithstanding, froglogic isheld liable to Licensee, whether in contract, tort or any other legaltheory, based on the Licensed Software, froglogic’s entire liabilityto Licensee and Licensee’s exclusive remedy shall be, at froglogic’soption, either (A) return of the price Licensee paid for the LicensedSoftware, or (B) repair or replacement of the Licensed Software,provided Licensee returns to froglogic all copies of the LicensedSoftware as originally delivered to Licensee. froglogic shall notunder any circumstances be liable to Licensee based on failure of the


License Agreement

Licensed Software if the failure resulted from accident, abuse ormisapplication, nor shall froglogic under any circumstances be liablefor special damages, punitive or exemplary damages, damages for lossof profits or interruption of business or for loss or corruption ofdata. Any award of damages from froglogic to Licensee shall not exceedthe total amount Licensee has paid to froglogic in connection withthis Agreement.

SUPPORT AND UPDATES

9. If this license, as specified in the license certificate, includesa support and update agreement, Licensee will be eligible to receiveemail based software support and access to Updates to the LicensedSoftware for a period not to exceed one year from the date of initialdelivery, in accordance with froglogic’s then current policies andprocedures, if any. Such policies and procedures may be changed fromtime to time.

GENERAL PROVISIONS

10. This Agreement may only be modified in writing signed byauthorized representatives of Licensee and froglogic. In case of aconflict between this Agreement and the terms of any purchase order orother ordering document, this Agreement shall prevail. If any provisionof this Agreement is found void or unenforceable, the remainder willremain valid and enforceable according to its terms. If any remedyprovided is determined to have failed for its essential purpose, alllimitations of liability and exclusions of damages set forth in thisAgreement shall remain in effect.

11. This Agreement shall be construed, interpreted and governed by thelaws of Germany, the legal venue to be Hamburg City Court. froglogicreserves all rights not specifically granted in this Agreement.


Appendix

FIndex

--cs-architecture, 70, 90, 92, 205, 206--cs-basic-blocks, 255--cs-boost, 63--cs-branch, 255--cs-c-ext, 70, 263--cs-combine-switch-cases, 63, 71, 254--cs-compiler, 60, 61, 69--cs-compiler-option, 68--cs-condition, 62, 71--cs-count, 24, 62, 71, 128, 135--cs-counter-size, 64, 254--cs-cpp-ext, 70, 263--cs-csharp-dynamic, 64, 71--cs-decision, 62, 71, 135, 254--cs-disable-coveragescanner-library-abs-wildcard,

69--cs-dump-on-event, 68, 71, 174, 175, 263--cs-dump-on-signal, 68, 174, 263--cs-enable-coveragescanner-library-abs-wildcard,

69--cs-exclude, 61--cs-exclude-code, 113--cs-exclude-code-regexp, 110, 111--cs-exclude-code-wildcard, 110--cs-exclude-file-abs-regex, 65--cs-exclude-file-abs-regexp, 110--cs-exclude-file-abs-wildcard, 24, 65, 110--cs-exclude-file-regex, 66, 86, 243--cs-exclude-file-wildcard, 18, 66, 243--cs-exclude-function-regex, 67--cs-exclude-function-wildcard, 67--cs-exclude-path, 18, 65, 110, 243, 260--cs-exit-function, 64--cs-full-instrumentation, 24, 63, 71--cs-function, 62--cs-hit, 62, 71, 128, 135--cs-include, 61--cs-include-code-regexp, 110--cs-include-code-wildcard, 110--cs-include-file-abs-regex, 65--cs-include-file-abs-wildcard, 65--cs-include-file-regex, 66, 244

--cs-include-file-wildcard, 66, 244--cs-include-function-regex, 67--cs-include-function-wildcard, 66--cs-include-init-scripts, 110--cs-include-path, 29, 65, 110, 244, 260--cs-include-unnamed, 110--cs-keep-instrumentation-files, 67, 249--cs-libgen, 68, 94, 96, 205, 209, 249--cs-library-after, 69--cs-library-at-end, 69--cs-library-at-start, 69--cs-line, 63, 140--cs-link-instrumentation-tables, 69, 263--cs-lock-csexe, 68, 71--cs-mcc, 62, 71, 254--cs-mcc-max, 65--cs-mcdc, 62, 254--cs-mcdc-max, 65--cs-memory-pool, 69--cs-minimum-api, 69, 71--cs-native-toolchain, 70, 263--cs-no-abort-on-error, 70--cs-no-annotations, 64--cs-no-assignments, 63, 71--cs-no-boost, 64, 266--cs-no-csharp-dynamic, 46, 64, 71--cs-no-cspch, 70--cs-no-exceptions, 63, 71--cs-no-exit-handler-installation, 69--cs-no-function, 63--cs-no-line, 63--cs-no-line-directive, 64, 71, 253--cs-no-qt3, 64, 266--cs-no-qt4, 24, 64, 266--cs-nolock-csexe, 68, 71--cs-off, 62--cs-on, 14, 18, 55, 60, 62, 70, 71, 91, 100, 170, 192, 200, 203,

205, 209, 211, 213--cs-option-file, 70, 263--cs-output, 24, 67, 94, 110, 111, 113, 259, 260--cs-output-abs, 266--cs-parallel-build, 70

283

INDEX

--cs-partial-instrumentation, 63, 71, 135--cs-profile, 61, 69, 263--cs-qt3, 64, 196--cs-qt4, 64, 196--cs-statement-block, 62, 71, 135, 140--cs-verbose, 67, 263, 267--cs-verbose-file, 67--cs-verbose-source-lines, 67--cs-verbose=api, 71--cs-vs2010-lambda, 64, 243, 262--cs-warnings, 67#endregion, 85#pragma, 84#region, 85_Pragma, 84__COVERAGESCANNER__, 73, 78, 92__coveragescanner_add_html_comment(), 75, 79, 172__coveragescanner_clear(), 75, 216, 218, 220__coveragescanner_clear_html_comment(), 75__coveragescanner_filename(), 76, 87, 88__coveragescanner_install(), 69, 73, 74, 80, 99, 123, 216,

218, 220__coveragescanner_register_library(), 76, 86, 87, 94__coveragescanner_reset(), 75__coveragescanner_save(), 73–75, 78, 87, 88, 99, 216, 218,

221, 249__coveragescanner_set_custom_io(), 77, 78__coveragescanner_testname(), 74, 172, 216, 218, 220__coveragescanner_teststate(), 74, 172, 216, 218, 221__coveragescanner_unregister_library(), 76, 77, 86,

87, 94__pragma, 84Condition Coverage, 45

Apple Xcode, 212, 213ar, 90, 212, 272ARM Keil uVision, 198, 253arm-linux-gcc, 272

bash, 9, 12, 13benchmark, 279Branch Coverage, 44build_cpp, 175build_cs, 175Button.qml, 107

C, 3–5, 24, 47, 59, 60, 73, 82, 84, 91, 92, 126, 128, 141, 190,195, 203, 205, 209–211, 213, 224, 239, 272

C++, 3–5, 11, 12, 15, 24, 43, 59, 60, 63, 64, 73, 82, 84, 91, 92,126, 128, 141, 175, 190, 195, 203, 205, 209–211,213, 224, 239, 259, 272

C#, 3–5, 10, 46, 60, 62–64, 69, 78, 81, 85, 95, 98, 166, 175,203, 224, 240, 253, 255, 257, 259, 261, 271

Cache, 147ccache, 241cl, 271cl.exe, 59, 268clang, 73, 74

CMake, 191CMakeLists.txt, 191, 192cmcsexeimport, 115, 194, 236cmmerge, 158Cobertura, 145, 155, 261cocolic, 235COCOQML, 106, 108cocoqml.txt, 107COCOQML_BLACKLIST, 107, 108COCOQML_DUMPLOC, 108COCOQML_LOGFILE, 108COCOQML_OUTPUT, 108COCOQML_QUIET, 108COCOQML_VERBOSE, 108Code coverage count, 55, 128, 135Code coverage hit, 55, 135Color

Blue, 129Gray, 129Magenta, 129Orange, 128Red, 128, 129

Comments, 130, 146cov-assignment-off

pragma, 84region, 85

cov-assignment-onpragma, 84region, 85

cov-conditionpragma, 84region, 85

cov-countpragma, 84region, 85

cov-decisionpragma, 84region, 85

cov-full-instrumentationpragma, 84region, 85

cov-hitpragma, 84region, 85

cov-offpragma, 84region, 85

cov-onpragma, 84region, 85

cov-partial-instrumentationpragma, 84region, 85

cov-statement-blockpragma, 84region, 85

cov-switch-case-offpragma, 84


INDEX

region, 85cov-switch-case-on

pragma, 84region, 85

coveragebrowser, 12CoverageBrowser Menu

Context MenuAdd/Set Comment, 130Clear Comments, 130

FileGenerate Black-Box Configuration. . . , 36, 119Import Reviewer Comments. . . , 143Import Unit Tests. . . , 32, 142, 173Load Execution Report. . . , 26, 123, 156, 166, 167, 175Merge with. . . , 37, 142, 158Open. . . , 25Save, 13, 17

InstrumentationAdd/Set Comment, 130Clear Comment, 130Coverage Method, 135Level:x, 135

ReportsExport JUnit Report. . . , 145Export Statistics in Cobertura-XML Format. . . , 145Export Statistics in EMMA-XML Format. . . , 144Generate HTML/CSV Report. . . , 144Generate Text Report. . . , 145

ToolsAnalysis of Modified Methods, 35, 142Analysis on Identical Methods, 142Compare with. . . , 35, 142Execution Comparison Analysis, 34Patch File Analysis. . . , 20, 140Switch databases, 142Test Coverage Count Mode, 135

ViewBug Location, 21Bug Location. . . , 140Functions, 127New Source Window, 127Optimized Execution Order. . . , 136Sources, 126

coveragescanner, 196, 272CoverageScanner (C# class)

__coveragescanner_add_html_comment, 79__coveragescanner_clear_html_comment, 79__coveragescanner_clear, 80__coveragescanner_filename, 80__coveragescanner_init, 78, 259__coveragescanner_save, 79–81__coveragescanner_set_custom_io, 80, 81__coveragescanner_testname, 78, 79__coveragescanner_teststate, 79

CoverageScanner() directive, 84, 85COVERAGESCANNER_ARGS, 18, 60, 61, 170COVERAGESCANNER_COVERAGE_BASIC_BLOCK, 71COVERAGESCANNER_COVERAGE_COMBINE_SWITCH_CASES, 71

COVERAGESCANNER_COVERAGE_CONDITION, 71COVERAGESCANNER_COVERAGE_COUNT, 71COVERAGESCANNER_COVERAGE_DECISION, 71COVERAGESCANNER_COVERAGE_FULL_INSTRUMENTATION, 71COVERAGESCANNER_COVERAGE_HIT, 71COVERAGESCANNER_COVERAGE_LOCK_CSEXE, 71COVERAGESCANNER_COVERAGE_MCC, 71COVERAGESCANNER_COVERAGE_NO_ASSIGNMENTS, 71COVERAGESCANNER_COVERAGE_NO_EXCEPTIONS, 71COVERAGESCANNER_COVERAGE_NOLOCK_CSEXE, 71COVERAGESCANNER_COVERAGE_ON, 70, 71, 167, 203COVERAGESCANNER_COVERAGE_PARTIAL_INSTRUMENTATION,

71COVERAGESCANNER_CSHARP_DYNAMIC, 71COVERAGESCANNER_DUMP_ON_EVENT, 71COVERAGESCANNER_MINIMUM_API, 71COVERAGESCANNER_NO_CSHARP_DYNAMIC, 71COVERAGESCANNER_NO_LINE_DIRECTIVE, 71COVERAGESCANNER_RUNTIME_LOG, 71coveragescannertcl, 168csar, 212, 272csc, 271cscl, 271cscl.exe, 268cscsc, 271csdmcs, 271csg++, 272csgcc, 272csgmcs, 271csicc, 272csicl, 272csicpc, 272cslib, 271cslink, 271csmcs, 271CSV, 140, 144, 149, 152–154, 258, 259cvs, 188CygWin, 190, 258, 264

Decision Coverage, 44diff, 19, 20, 35, 187dmcs, 271dump_on_event, 175, 176dump_on_event.cpp, 175dump_on_event_cs, 175

Eclipse, 211, 212EMMA, 144, 154, 155, 235, 236, 254, 259, 261–263, 265–267event_sample_cpp, 175event_sample_cpp.cpp, 175event_sample_cpp.exe.csexe, 175event_sample_cpp.exe.csmes, 175Execution Report, 123exit(), 64

free(), 69

g++, 59, 272


INDEX

gcc, 73, 74, 211, 272git, 187, 188gmcs, 271Green Hills MULTI, 200

HTML, 5, 34, 38, 49, 75, 79, 113, 116, 140, 141, 144, 149, 152,168, 169, 172, 173, 230, 232, 258, 259, 262, 266,267

Hudson, 144, 145

icc, 272icl, 272Icons

..., 122, 132, 130, 130, 130, 130, 130, 129, 130, 130, 130, 128–130, 130, 129, 130, 123, 122

! , 133.* , 133?* , 133

, 121, 126, 127, 129, 129, 129, 129, 129, 128, 129, 126, 129, 129, 130

ok , 129C++CPP , 128, 129

, 129, 129, 129, 129, 128, 129, 126, 129, 121

1.2.3., 135-

+ , 122, 125ok , 129Open, 20

icpc, 272

Jenkins, 234–236JUnit, 145, 155

kill, 174

ld, 90lib, 271link, 271Linux, 3, 9, 25, 105, 140, 162, 168, 178, 180, 194, 234, 235,

249, 264, 269

macOS, 3, 9, 11, 25, 105, 162, 170, 178, 194, 234, 235, 240,256, 260, 261, 269, 270

main(), 19make, 14, 18, 60, 190, 195malloc(), 69mcs, 271Meta-Object Compiler (moc), 4, 24, 38, 64, 196Microsoft Visual Studio, 3, 10, 15, 46, 59, 64, 73, 74, 88, 94,

165, 166, 168, 169, 171, 191, 192, 197, 203–207,224, 238–240, 242, 243, 249, 255, 257, 259, 261–265, 267, 268, 271

Microsoft Visual Studio .NET, 59, 203, 271Microsoft Visual Studio Add-In, 3, 165, 207, 238–240, 253,

255, 257, 260, 263–265, 267Microsoft Windows, 3, 9, 11, 12, 15, 18, 20, 25, 59, 66, 68, 69,

76, 77, 100, 101, 106, 162, 173, 174, 176, 178, 180,187, 194, 195, 234, 235, 244, 249, 263, 268–270,272

Microsoft Windows CE, 205Minimum Comment Size, 146moc, 4, 24, 38, 64, 196Mono C#, 171, 271MSBuild, 171, 242

nmake, 195

poppragma, 84

qbs, 196qmake, 23–25, 29–31, 105, 106, 195, 196qrc, 24, 38Qt Build Suite (qbs), 196Qt Framework, 4, 104, 193, 196

Reload, 146

SCons, 197Shortcut

Ctrl+B, 129, 130Ctrl+D, 129Ctrl+F, 126, 129Ctrl+I, 129, 130Ctrl+J, 129, 130Ctrl+K, 129Ctrl+L, 130Ctrl+M, 130Ctrl+R, 130Ctrl+S, 130Ctrl+Shift+B, 129Ctrl+Shift+D, 129Ctrl+Shift+F, 126, 129Ctrl+Shift+I, 129


INDEX

Ctrl+Shift+J, 129Ctrl+Shift+K, 129Ctrl+Shift+T, 129Ctrl+Shift+U, 129Ctrl+Shift+V, 129Ctrl+Shift+Z, 129, 130Ctrl+T, 129Ctrl+U, 129, 130Ctrl+V, 129Ctrl+Wheel, 128Ctrl+Z, 129, 130Shift+Wheel, 128Wheel, 128

SonarQube, 145, 155Statement Block Coverage, 43Status

Dead-Code, 129Executed, 128Execution count too low, 129Hidden, 129Manually Set To Be Executed, 129Never Executed, 129Partially Executed, 128Unknown, 129

su, 256sudo, 256

Tcl, 3–5, 167, 168, 261tclsh, 168tclsh8.5, 168Test coverage count, 55, 128, 135Thresholds, 146

uic, 24, 38UNIX, 11, 14, 15, 19, 59, 76, 77, 101, 106, 162, 168, 170, 174,

187, 195, 244, 256, 263, 265, 269, 270

VisualDSP, 201, 258

XBuild, 171XML, 113, 144, 145, 149, 154, 155, 235, 236


Squish Coco 4.0 - froglogic · CONTENTS Contents I Overview2 1 Squish Coco – A code coverage tool...

Documents

Transcript of Squish Coco 4.0 - froglogic · CONTENTS Contents I Overview2 1 Squish Coco – A code coverage tool...