Applying Business Intelligence to Software Engineering

Applying Business Intelligence to Software Engineering

O S C A R G Ö T H B E R G

Master’s Thesis in Computer Science (20 credits) at the School of Computer Science and Engineering Royal Institute of Technology year 2007 Supervisor at CSC was Stefan Arnborg Examiner was Stefan Arnborg TRITA-CSC-E 2007:003 ISRN-KTH/CSC/E--07/003--SE ISSN-1653-5715 Royal Institute of Technology School of Computer Science and Communication KTH CSC SE-100 44 Stockholm, Sweden URL: www.csc.kth.se

AbstractA major problem with software development is a lack of insight into howwell the project is going. Every year, a vast majority of finished softwareprojects in the US are either scrapped, or completed without meetingcost and time constraints. At the same time, analysis of performancemetrics is an area that is often underutilized in software engineeringbecause of a lack of expertise and tools.

However, this is not so in the business world, where data is modeledand analyzed along different dimensions to help make critical businessdecisions that can determine profits or loss. These techniques fall un-der the umbrella of “business intelligence”. In my thesis, I have evalu-ated using these techniques for the gathering of metrics related to theprogress of a software project.

With a list of metrics to provide as the project aim, I used the busi-ness intelligence platform developed by LucidEra to develop four analyt-ics applications, each corresponding to a different engineering supportsystem (a bug database, a testing framework, a continous integrationsystem and a revision control system) as the data source.

Through this approach, I was able to provide a majority, althoughnot all, of the metrics from my list in an automated and relativelysource-independent fashion. Based on my results, I conclude that us-ing a business intelligence platform is a viable approach to the problemof providing better analytics related to software engineering.

Referat

Att tillämpa Business Intelligence påmjukvaruutveckling

Ett stort problem inom mjukvaruutveckling ar brist pa insikt i hur valprojektet fortskrider. Varje ar ar en stor majoritet av de mjukvaruprojektsom avslutas i USA antingen nedlagda, eller avslutade utan att motatids- och kostnadsramar. Samtidigt ar analys av prestationsindikatorerett omrade som ofta ar underutnyttjat i mjukvaruutveckling pa grundav en brist pa expertis och verktyg.

Sa ar dock inte fallet i affarsvarlden, dar data modelleras och analy-seras langs olika dimensioner for att underlatta kritiska affarsbeslut somavgor vinst eller forlust. Dessa tekniker faller under begreppet “businessintelligence”. I mitt examensarbete har jag utvarderat mojligheterna attanvanda dessa tekniker for att underlatta insamlingen av matetal rela-terade till progressionen hos ett mjukvaruprojekt.

Med en lista av matetal att tillhandahalla som malsattning for pro-jektet, har jag anvant den business intelligence-plattform som utvecklasav LucidEra for att utveckla fyra analysapplikationer, var och en base-rad pa ett av utvecklingslagets stodsystem (en buggdatabas, ett testram-verk, ett system integrationsautomatisering och ett versionshanterings-system) som datakalla.

Genom detta tillvagagangssatt kunde jag tillhandahalla de flesta,om an inte alla, av matetalen fran min lista pa ett automatiserat och re-lativt kalloberoende satt. Med mina resultat som bas kan jag konstateraatt anvandandet av en business intelligence-plattform ar ett gangbartsatt att tillhandahalla battre matetal for mjukvaruutveckling.

PrefaceThe project forming the basis of this thesis was carried out at LucidEra in San Mateo,California, between March and September 2006. I would like to thank Boris Chenfor inviting me and for his invaluable support throughout the project.

I would also like to thank everyone at LucidEra for their help and input. Aspecial thanks to Fan Zhang for contributing some of his Sed&Awk magic.

Most of all, I would like to thank my parents for always supporting me.

Contents

1 Introduction 11.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.5 Report outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Background 52.1 The concept of Business Intelligence . . . . . . . . . . . . . . . . . . 52.2 The potential of engineering analytics . . . . . . . . . . . . . . . . . 52.3 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.3.1 Productivity is hard to measure . . . . . . . . . . . . . . . . . 62.3.2 Software vendors get away with it . . . . . . . . . . . . . . . 6

3 Aim 93.1 Defect tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.1.1 Open bugs over time . . . . . . . . . . . . . . . . . . . . . . . 103.1.2 Cumulative open/closed plot . . . . . . . . . . . . . . . . . . 103.1.3 Open/close activity charts . . . . . . . . . . . . . . . . . . . . 103.1.4 Bug report quality . . . . . . . . . . . . . . . . . . . . . . . . 103.1.5 Average resolution time . . . . . . . . . . . . . . . . . . . . . 10

3.2 Test result tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2.1 Total test results over time . . . . . . . . . . . . . . . . . . . . 113.2.2 Performance comparisons . . . . . . . . . . . . . . . . . . . . 113.2.3 Test execution reports . . . . . . . . . . . . . . . . . . . . . . 113.2.4 Test development reports . . . . . . . . . . . . . . . . . . . . 12

3.3 Continous integration tracking . . . . . . . . . . . . . . . . . . . . . 123.3.1 Number of runs per day . . . . . . . . . . . . . . . . . . . . . 123.3.2 Time to fix . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.3.3 Number of new submissions per build . . . . . . . . . . . . . 123.3.4 Number of changed files per build . . . . . . . . . . . . . . . 133.3.5 Number of line changes per build . . . . . . . . . . . . . . . . 13

3.4 Revision control tracking . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.4.1 Number of checkins . . . . . . . . . . . . . . . . . . . . . . . 133.4.2 Number of files changed per submission . . . . . . . . . . . . 133.4.3 Number of lines changed per submission . . . . . . . . . . . . 133.4.4 Number of integrations . . . . . . . . . . . . . . . . . . . . . 14

4 Tools 154.1 Introduction to OLAP concepts . . . . . . . . . . . . . . . . . . . . . 15

4.1.1 An example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.1.2 Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.1.3 Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.1.4 Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.1.5 Hierarchies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.1.6 Cubes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.1.7 Slicing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.1.8 MDX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.1.9 Star schema design . . . . . . . . . . . . . . . . . . . . . . . . 17

4.2 The platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.2.1 Clearview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.2.2 Mondrian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.2.3 LucidDB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.2.4 Data adaptors . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.3 Analytics application development tools . . . . . . . . . . . . . . . . 204.3.1 The Workbench . . . . . . . . . . . . . . . . . . . . . . . . . . 204.3.2 Application Lifecycle Service . . . . . . . . . . . . . . . . . . . 20

4.4 Data sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.4.1 Blackhawk . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.4.2 Cruisecontrol . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.4.3 Jira . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.4.4 Perforce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

5 Method 235.1 Introduction to application design . . . . . . . . . . . . . . . . . . . . 23

5.1.1 Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.1.2 Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . 245.1.3 Star schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

5.2 Defect tracking - Jira . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.2.1 Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.2.2 Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . 255.2.3 Star schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

5.3 Test result tracking - Blackhawk . . . . . . . . . . . . . . . . . . . . . 275.3.1 Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.3.2 Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . 275.3.3 Star schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

5.4 Continous integration tracking - Cruisecontrol . . . . . . . . . . . . . 28

5.4.1 Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285.4.2 Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . 305.4.3 Star schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

5.5 Revision control tracking - Perforce . . . . . . . . . . . . . . . . . . . 305.5.1 Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305.5.2 Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . 315.5.3 Star schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

6 Results 336.1 Example reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

6.1.1 Performance comparison between virtual java machines . . . 336.1.2 Test success monitoring for recent builds . . . . . . . . . . . . 34

6.2 Compared to the original aim . . . . . . . . . . . . . . . . . . . . . . 346.2.1 Metrics that were easy to get . . . . . . . . . . . . . . . . . . 346.2.2 Metrics that would require more work . . . . . . . . . . . . . 36

6.3 Next steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376.3.1 Possible with additional data sources . . . . . . . . . . . . . . 376.3.2 Possible with better cross-referencing . . . . . . . . . . . . . . 37

7 Conclusions 39

Bibliography 41

List of Figures

1.1 Manual metrics gathering in a spreadsheet . . . . . . . . . . . . . . . . . 2

4.1 Hierarchies example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.2 Star schema example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.3 The LucidEra application platform and ETL layer . . . . . . . . . . . . . 19

5.1 ETL layer for the Jira-based application . . . . . . . . . . . . . . . . . . 255.2 Example of normalization in the Jira source schema . . . . . . . . . . . . 255.3 ETL layer for the Blackhawk-based application . . . . . . . . . . . . . . . 275.4 ETL layer for the Cruisecontrol-based application . . . . . . . . . . . . . 295.5 ETL layer for the Perforce-based application . . . . . . . . . . . . . . . . 31

6.1 Report on execution time for different virtual java machines. . . . . . . . 346.2 Report on the percentage of passed tests for the ten most recent builds. . 35

List of Tables

5.1 Measures and dimensions acquired for defect tracking . . . . . . . . . . 265.2 Measures and dimensions acquired for test result tracking . . . . . . . . 285.3 Measures and dimensions acquired for continous integration tracking . . 305.4 Measures and dimensions acquired for revision control tracking . . . . . 32

6.1 Metrics that are easy to get from the source systems. . . . . . . . . . . . 356.2 Metrics that would require extra work to retrieve from source systems. . 36

Chapter 1

Introduction

1.1 BackgroundA major problem with software development is a lack of insight into how well theproject is going. The first problem is determining how to measure a project, thesecond is being able to obtain those measurements, and the third is how to analyzethem.

Analysis of metrics is an area that is often underutilized in software engineeringbecause of a lack of expertise and tools. However, this is not so in the businessworld, where data is modeled and analyzed along different dimensions to helpmake critical business decisions that can determine profits or loss. These techniquesfall under the umbrella of “business intelligence”.

1.2 ProblemAs my project for this thesis I will investigate the feasibility of applying a “tra-ditional” business intelligence platform to the problem of analyzing the softwareengineering process.

Today, much of the metrics gathered for analyzing the progress of softwareprojects are collected by hand (example in figure 1.1), and compiled through man-ual work in tools such as Microsoft Excel. Therefore, I will investigate the feasibilityof automating and enhancing this metrics collection process through the use ofbusiness intelligence techniques.

1.3 MethodLucidEra, Inc. is developing a business intelligence software-as-a-service to midsizeorganizations, that collects data from a company’s internal systems as well as exter-nal ones, combines, aggregates and analyzes these, and finally presents them in anaccessible way.

1

CHAPTER 1. INTRODUCTION

Figure 1.1. Manual metrics gathering in a spreadsheet

I will try and apply the service being developed by LucidEra to the gatheringand presenting of data relevant to software engineering, from sources such as bugdatabases, testing frameworks and revision control systems, in order to help themanaging of software engineering projects.

As data sources I will use the tools used in LucidEra’s internal developmentinfrastructure, including Jira (for bug tracking), Perforce (for revision control),Cruisecontrol (for continous integration) and Blackhawk (for organizing and run-ning recurring test suites).

For the design of the applications themselves, I will use standard data warehous-ing, relational database and OLAP (on-line analytical processing) techniques, suchas ETL (extract, transform, load), as well as SQL/MED connectivity features of theLucidDB database.

1.4 ScopeI will primarily study what can be done just by “plugging in” an analytics platform tothe tools and workflows already being used in a software development organization,and see what can be had “for free” or at least “for cheap”.

The reason for this is to limit the barrier of entry by keeping the amount of workadapting an organization’s current way of doing things to a minimum. Evangelicaltexts along the lines of “we could improve a lot by totally changing the way we dothings” already exist.

In my conclusions I will, however, discuss the small incremental changes, the

2

1.5. REPORT OUTLINE

low-hanging fruit, to engineering practices and conventions that could give big im-provements in performance analysis.

Also outside of the scope of this thesis is lower-level source code analysis, such asthe generating of code complexity metrics. Although this would make an interestingaddition to the performance metrics I do cover, it has been investigated extensivelyby others (including previous KTH master’s theses[16][19]), and it would detractfrom the focus of this thesis.

1.5 Report outlineIn chapter 2, I introduce the concept of business intelligence along with a back-ground as to why better analytics is needed in the software industry, while chapter 3gives a detailed specification of the specific analytics I wanted to realize. Chapter 4introduces the tools and techniques used, and chapter 5 provides in-depth examplesof how these were actually used in the project. In chapter 6 I report what metricsand analytics I was actually able to extract using this approach, and compare thatto the original aim along with some suggestions as to which next steps are possible.Finally, in chapter 7 I present my conclusions and my recommendations for furtherdevelopment in the area.

3

Chapter 2

Background

2.1 The concept of Business Intelligence“Business Intelligence” is a common term for the gathering and analysis of an or-ganization’s data in the pursuit of better business decisions[31]. Being somewhatloosely defined, the term spans many technical disciplines from data warehousingand customer relationship management, to high-end technical data analysis anddata mining. Big vendors in the field are SAS, Cognos, SAP and Business Objects.

Businesses in financial investement and trading have used business intelligence-related analytics in their competitive strategies for decades, but in recent yearsthese techniques have also started to spread into other industries such as consumerfinance, retail and travel[7].

2.2 The potential of engineering analyticsIn the past decades, software has become an essential part of business and society.In 2006, the revenues from software and software related services for the industry’s500 biggest companies totalled over $380 billion[8].

Since 1994, the Massachusetts-based Standish Group has been tracking the out-comes of software projects in the US. Their first study[27], from 1994, reportedthat only 16% of software projects were delivered on-time and on-budget, with53% over-budget and over-time, and 31% terminated without completion. In 2000the situation had improved[12] somewhat, with 28% of the projects within budgetand time constraints, 49% over-budget and over-time, and 23% scrapped.

Even so, it can be concluded that predictability in the software engineering pro-cess is still sorely lacking. Especially so considering the values at stake in the in-dustry. There is a big potential in having better ways of telling how well a softwareproject is progressing, and one possible path could be through the use of businessintelligence techniques. Having better predictability in the software engineeringprocess would lead to improvements in many areas, including:

5

CHAPTER 2. BACKGROUND

• higher accuracy in cost and time estimates.

• better utilization, and wiser allocation, of project resources.

• closer understanding of business processes and organizations.

2.3 ChallengesLike any other context, however, the software engineering organization has certaincharacteristics that are important to consider when applying analytics strategies.

2.3.1 Productivity is hard to measure

An area where business intelligence is widely used today is in sales organizations;the value of individual sales representatives, their managers, and so on, are oftenmeasured in terms of generated revenue. Selling is the major responsibility of salespersons, and revenue is the point of selling. Thus, since revenue easily can beexpressed in actual numbers, sales person performance can be measured ratherwell.

The problem with doing a direct transfer from this situation to that of, say,a programmer, is that the responsibilities of a programmer are harder to mea-sure by numbers. And worse, using bad metrics would drive the wrong type ofperformance[5].

A major problem with measuring performance on a per-individual level is that itencourages competition and discourages teamwork[5]. This may be fine in a situ-ation where a large amount of competitiveness is a good thing, but in a teamwork-dependent organization, such as an engineering group, maximizing individual per-formance metrics may actually lead to worse performance for the team as a whole.

Therefore, this thesis will primarily evaluate business intelligence techniques forperformance analysis for a project or team as a whole rather than on a per-personbasis.

2.3.2 Software vendors get away with it

Microsoft announced[10] in 2004 that it would have to cut major features from thecoming revision to it’s Windows operating system in order to make the 2006 dead-line. Still, in 2006, the update was delayed further[3] into 2007, with Microsoftleaving PC manufacturers and resellers without the expected new Windows versionin the important holiday shopping season. However, even though this is the flagshipproduct of a company with a yearly revenue of $45 billion, the reaction to the delayhas been one of sighs and shrugs, at most, suggesting that this sort of productionhickup is to be expected in the computer industry.

By comparison, when the new Airbus 380 passenger jet project hit it’s mostrecent delay, pushing back initial deliveries by 6 months (October 2006 to April

6

2.3. CHALLENGES

2007), it caused a shareholder class action lawsuit that forced the resignations ofboth Airbus’ and parent company EADS’ CEOs[2].

One possible explanation to why analytics has not caught on in the softwareindustry could be that people simply are more willing to accept problems related tocomputers than they are in the case of other types of technology[21], and that thishas lead to software vendors getting comfortable with the situation.

Considering, however, that the annual added cost of inadequate software test-ing infrastructure has been estimated[28] to $59.5 billion per year in the UnitedStates alone, the upside of this unwillingness is that there is a significant compet-itive advantage to be seized by a company that manages to handle this problembetter.

7

Chapter 3

Aim

The specific aim for this thesis is to investigate how much of the metrics and analyt-ics currently being gathered by hand for the management of a software project canbe acquired automatically through the use of common data warehousing techniquesand a business intelligence platform such as the one being developed by LucidEra.

I will start with a requirements specification; a wish list of the kind of metricsand analytics we would like to have for the development organization at LucidEra.Providing the metrics described below in an automated and accessible fashion willbe the goal for this project.

It is worth noting that this list of metrics was subject to change throughout theactual implementation effort, due to discoveries of previously unknown limitationsas well as possibilities in the source data. This is to be expected during the devel-opment of an ETL (extract, transform, load) process[6].

I have grouped the metrics by different areas of analysis. To a large extent thesegroups also correspond to the different data sources to be used:

• Defect tracking (bug report database) - section 3.1

• Test result tracking (test suite framework) - section 3.2

• Continous integration tracking (automated build system) - section 3.3

• Revision control tracking (revision control system) - section 3.4

3.1 Defect trackingFor tracking defects, the plan was to connect to a bug tracking software. These bugtracking services are usually centered around the filing of issues, concerning somepart of the product, and the assigning of these to someone for fixing. An issue canbe either open (not yet resolved) or closed (resolved), it can be of higher or lowerpriority, and so on.

For my analysis purposes, I would want to extract metadata to allow me to breakdown and/or filter these issues along at least the following dimensions:

9

CHAPTER 3. AIM

priority distinguish critical problems from minor issuescomponent which part of the project is affected

reporter who reported the bugassignee who is assigned to fix it

fixer if fixed, who fixed it

3.1.1 Open bugs over time

Track trends over time to gauge whether we are nearing “readiness” for release.When we are ready, we should see a downward slope in open bugs.

3.1.2 Cumulative open/closed plot

Also to track trend. For a given day, how many bugs have been opened to this point,and how many have been closed. (two lines). In this case, we should see a smoothconvergence. Danger signs are[4]:

1. when lines do not converge and continue to increase (Endless bug discovery)

2. if we plateau but do not converge (Bug deficit)

3. if we see plateaus and inclines alternating (blurred project vision, may indi-cate inconsistency in bug filing/closing)

3.1.3 Open/close activity charts

See “open” and “close” actions per day. Towards the end of the release, the find rateshould decrease. Close rate should also decrease (since fixing/closing bugs alsomay introduce new bugs).

3.1.4 Bug report quality

Breaking down the number of resolved bugs per their resolution (“fixed”, “will notfix”, “invalid”, “can not reproduce”), etc, gives us an idea of the quality of the bugreports.

A high percentage of “invalid” or “will not fix” could indicate that there is aperception mismatch between developers and testers regarding what is a bug andwhat is not. A high number of “can not reproduce” may indicate that the level ofdetail in bug descriptions needs to improve, and so on.

3.1.5 Average resolution time

Measure trend in ability to resolve bugs promptly. If time is lengthening, theneither bugs are getting harder, or exceeding capacity, or there is some inefficiencyin handling.

10

3.2. TEST RESULT TRACKING

3.2 Test result trackingFor an organization doing continous automated testing, it will be valuable to beable to analyze the results from these tests.

A central part of test results gathering is the management of the configurationmatrix (the possible environments that the project will be tested on), and since thatdata is inherently multidimensional, it lends itself well for business intelligence-style dimensional analysis.

For test results, we will want to be able to break down and filter data along thefollowing dimensions:

build number the number uniquely identifying a buildcomponent what software component was tested

results success, failed, skipped, etcconfiguration dimensions operating system, JVM1 used, etc

3.2.1 Total test results over time

This provides simple high level aggregate numbers of test results over time, andshould provide the project team an estimate on the quality of the product. Anincrease of tests will indicate better coverage, an increase of success percentageindicates higher known quality.

Ideally at the end of a release, we will want to have increasing test coverage,and success rate. A decrease in coverage and increase of success rate is no comfort.An increase in tests, and decrease in success is expected normally, and over time,the success rate should then increase (if an increase in tests do not result in decreaseof success rate, then this may indicate ineffective tests being developed).

3.2.2 Performance comparisons

Though functional tests are not performance tests, they can be analyzed for perfor-mance differences between different configuration values.

For instance, show tests run on solaris vs tests run on linux, or top 10 disparities(i.e. where solaris is much faster; or conversely much slower).

Another common problem is finding out why a test run is taking so long, and forthis purpose it would be useful to be able to find the top 10 longest running tests.

3.2.3 Test execution reports

Test management reports are for tracking how many tests we want to run, and howclose we are to our goal.

This requires a test management system to be put in place, which includes theability to define an expected set of tests, and match them to configurations whichthey should be run with, and at what frequency they should be run. You may forinstance want to have all tests run for a given milestone, but you may not require

11

CHAPTER 3. AIM

that all tests run for all milestones. Some tests may be run once per release. Somemay be run every day.

3.2.4 Test development reports

Test development reports are for tracking what tests should be developed and when.Tests will usually be defined in a test specification, and then be tried out manuallybefore being automated.

If tests are to be executed manually to be tracked, then development is in threestages, 1) definition, 2) implementation for manual testing (sometimes requiressome simple scripting or some dependency; or if none, then definition and imple-mentation is the same thing), 3) implementation for automated testing.

An important milestone may be to have some percentage of expected tests tobe implemented completed by some date, all to be complete by another. When atest set is frozen, then success rate should climb to an acceptable level that satisfiesrelease criteria.

3.3 Continous integration trackingDoing builds on a regular schedule, nightly/daily or some other, is a common prac-tice in software development. Analyzing data from such builds could provide impor-tant information about the complexity and matureness of the product. Interestingthings to break down by would be:

success whether the build attempt was successful or not.last submitter(s) who made submissions since the last successful build

date when the build was made

3.3.1 Number of runs per day

How many build attempts were made. Gives us an indicator of how busy the conti-nous integration system is. If the load is heavy and if there is a high percentage offailures, it might be an indicator that the code base should be branched.

3.3.2 Time to �x

Whenever we have a broken build, it is useful to measure how long it takes to get asuccessful one. Broken builds slow down development progress, so it is importantto have a quick turnaround on build failures.

3.3.3 Number of new submissions per build

How many submissions were made to the revision control system in between onebuild and the next. A high number of new submissions per build could be the causeof a lesser build success rate.

12

3.4. REVISION CONTROL TRACKING

3.3.4 Number of changed �les per build

How many files were changed between one build and the next. Compared to met-ric 3.3.3, this gives a better (albeit still not very good) indicator of how much codewas actually changed inbetween two builds.

3.3.5 Number of line changes per build

How many lines of code (where applicable) were changed between one build andthe next. Compared to metric 3.3.3 or 3.3.4, this gives a better indicator of howmuch code mass was actually changed inbetween two builds.

3.4 Revision control trackingThe revision control system manages all the changes (bug fixes, new features) sub-mitted by developers. It is a fundamental part of any software development organi-zation. Relevant breakdowns of metrics gathered from the revision control systemwould be:

change number the number identifying the changesubmitter who made the submission

affected component(s) software component(s) affected by the changeschange type type of change (add, edit, delete, etc)

date date the change was made

3.4.1 Number of checkins

The number of checkins would give an estimate of developer activity on a “logical”change level; one checkin often corresponds to a new feature, a new bug fix, andso on.

3.4.2 Number of �les changed per submission

The number of files changed per submission gives insight into the project’s file struc-ture; how tightly coupled different parts of the repository are, how many files needto be edited to make a certain kind of change. If the number is very high, we mighthave a case of spaghetti-code.

3.4.3 Number of lines changed per submission

The number of lines changed can be used as a measure for how much actual work isrequired to make a certain kind of change. If simple changes require big amounts ofcode to be edited, it could be an indicator that the code structure and developmentenvironment are not very powerful for the type of development in question.

13

CHAPTER 3. AIM

3.4.4 Number of integrations

With multiple code branches, the number of integrations (points of merging be-tween the codelines) is important to keep track of. Having integrations happen toofar apart increases the risk of having developers work with out-dated code.

Also, if the number of changes per integration is very small for an extendedperiod of time, it might be an indicator that the reason for splitting code into thesetwo branches is no longer applicable and that it could be worthwhile to permanentlymerge them into one.

14

Chapter 4

Tools

4.1 Introduction to OLAP conceptsOLAP, or Online Analytical Processing[29], is an approach to the structuring of datathat makes sense to analyze by dimensions. Fundamental concepts in OLAP, and therelations between them, are best illustrated by an example:

4.1.1 An example

Let’s say we want to analyze data describing the bug tracking database for a soft-ware project. A bug report has several properties, out of which a few could be:

reporter someone reported the bugassignee someone is assigned to fix it

create date at which point in time it was reportedpriority if the bug is critical or not

4.1.2 Measures

Among things interesting to analyze here is average priority, to see if the bugs inour database, on average, are bad ones or not. In order to do this, we take the“priority” property and couple that with an aggregation rule to calculate the averagefor several bug reports. We now have a measure called “average priority”.

4.1.3 Dimensions

That gives us an average for how severe all of our bugs are, but what we don’t seeyet is whether this measure varies depending on the other properties. For example,we could imagine that a plot of how the average priority measure changes over timecould be useful.

For that, we need dimensions. The “average priority over time” report can beachieved by taking the “create date” metadata and using that as the basis for adimension.

15

CHAPTER 4. TOOLS

In our example, we can also create “reporter” and “assignee” dimensions out ofthose respective metadata, allowing reports like “average priority for bugs reportedby Amy and assigned to Bob, over time”.

4.1.4 Members

The dimensions are populated by members, essentially all the possible values thatthe dimensions can take. For example, the report above suggests that “Amy” is amember of the “reporter” dimension, and that “Bob” is a member of “assignee”.

4.1.5 Hierarchies

Many dimensions that we want to analyze data by are hierarchical by nature. Suchis the case with “create date”, for example; we might want to analyze per year, permonth, or per day. Thus, we define “create year”, “create month” and “create day”as different levels of the “create date” dimension.

2006

2006-10-24October 2006

Ancestor/Descendant

Parent/Child

November2006

Siblings

2006-11-12

Cousins

Create Year Create Month Create Day

Figure 4.1. Hierarchies example

As shown in figure 4.1, members of the different levels are connected throughparent/child relationships: for example, “2006-10-24” of the “create day” level isthe child of “October 2006” of the “create month” level, while “2006” of the “createyear” dimension is the parent of “October 2006”. Both “2006-10-24” and “October2006” are descendants of “2006”, and the other way around, “2006” is their ancestor.

Continuing the family analogy, two members on the same dimension level shar-ing the same parent are called siblings, while two members on the same level havingdifferent parents but sharing the same grandparent are called cousins.

16

4.1. INTRODUCTION TO OLAP CONCEPTS

4.1.6 Cubes

For every combination of assignee, reporter and create date, it is possible for the“average priority” measure to take on a different value.

This relation between measures and dimensions can be seen as a cube, with ourthree dimensions as the axises X, Y and Z. For each cell (each possible combinationof values for X, Y and Z) this OLAP cube can hold different values for the differentmeasures.

4.1.7 Slicing

When looking at our cube, by deciding to keep the value of one of the dimensionsfixed (such as only analyzing bug reports reported by Amy) we limit the number ofremaining combinations to analyze.

In effect, we’re no longer looking at the whole cube, but rather just a slice. Theactivity of breaking down results by different dimensions is therefore called slicing.

The cube analogy gets difficult to picture geometrically as the number of dimen-sions increase, so these concepts are easiest to visualize with no more than two orthree dimensions.

4.1.8 MDX

Multidimensional Expressions, or MDX, is a language specially geared towardsquerying OLAP cubes, like SQL is in the case of relational databases. MDX wasinvented at Microsoft in 1997 as part of Microsoft SQL Server 7.0[30], and hassince been adopted by most major business intelligence vendors[32].

SELECT [ Measures ] . [ Average p r i o r i t y ] ON COLUMNS,[ Create date ] . [2006] . Chi ldren ON ROWS

FROM BugReportCubeWHERE ( [ Reporter ] . [Amy] , [ Ass ignee ] . [Bob] )

Above is an example query for retrieving the average priority for all bugs filedby Amy in 2006 that are assigned to Bob.

Simple MDX queries can look quite similar to SQL statements, but while SQL isadamant about seeing precisely everything as a table, MDX has powerful featuresfor actually navigating the spatial structures of an OLAP cube, which makes manythings that are very hard to express in SQL easy to do in MDX.

4.1.9 Star schema design

Normally, the multidimensional data model of an OLAP cube is an abstraction, re-alized through a relational database schema.

The two prevalent data warehousing schema designs for such cubes are snowflakeschemas and star schemas [29]. In this project I will use star schemas exclusively,and will therefore give a brief introduction to these.

17

CHAPTER 4. TOOLS

Star schemas consist of a fact table connected to separate dimension tables. Inpractice, each cell in the cube is represented by one or more rows in the fact table,and these fact table rows, or simply facts, are associated to dimension membersusing foreign keys mapping to surrogate keys in the dimension tables. See figure4.2 for what this would look like in our example. Note that measures are stored astheir own columns in the fact table and not in separate tables.

ID: integerPRIORITY: integerASSIGNEE_KEY: integerREPORTER_KEY: integerCREATE_DATE_KEY: integer

BUGREPORT_FACT

ID: integerNAME: varchar(255)

ASSIGNEE_DIMID: integerNAME: varchar(255)

REPORTER_DIM

ID: integerYEAR: varchar(4)MONTH: varchar(11)DAY: varchar(11)

CALENDAR_DIM

Figure 4.2. Star schema example

4.2 The platformThe LucidEra platform for analytics applications consists of a number of compo-nents, out of which major parts (Mondrian, LucidDB) are open-source. The num-bered steps in figure 4.3 illustrate how the different parts of the platform interactwhen responding to a report request from a user. In this section I will briefly intro-duce the different parts of the platform.

4.2.1 Clearview

Clearview is the Ajax1-based graphical user interface for the LucidEra platform andthe point of interaction with the end user. It puts advanced analytics in the handsof users without prior knowledge of databases or MDX, by allowing them to createqueries through the dragging and dropping of graphical elements.

4.2.2 Mondrian

Mondrian[26] is an on-line analytical processing (OLAP) server. In the LucidEraplatform it essentially takes the role of translator between MDX and SQL; it allows

1Asynchronous JavaScript and XML

18

4.2. THE PLATFORM

2. MDX queries

Internet

Clearview

Mondrian

1. user request

5. final report

Oracle

MySQLMySQL

adapter

Oracleadapter

Flatfileadapter

CSVfiles

Extract

IDPRIORITYASSIGNEE_KEYREPORTER_KEYCREATE_DATE_KEY

BUGREPORT_FACT

IDNAME

ASSIGNEE_DIM IDNAME

REPORTER_DIM

IDYEARMONTHDAY

CALENDAR_DIM

Star schema

IDNAME

REPORTER_DIM

IDYEARMONTHDAY

CALENDAR_DIM


BUGREPORT_FACT

IDNAME

ASSIGNEE_DIM

IDYEARMONTHDAY

CALENDAR_DIM


BUGREPORT_FACT

TransformLoad

LucidDB

5. MDX results

3. SQL queries4. SQL results

ETL layer

Figure 4.3. The LucidEra application platform and ETL layer

us to define an OLAP cube as a schema in a regular relational database.

4.2.3 LucidDB

LucidDB[20] is a relational database tailored for use in business intelligence ap-plications. It is built from the Eigenbase2 platform for modular data managementsystems, and is released as open source under the GNU Public License3.

4.2.4 Data adaptors

LucidDB implements parts of the SQL/MED standard[11] for external connectivity,which gives application developers a convenient way of accessing various kinds ofstructured data using plain SQL.

Data from foreign servers, such as structured text files, or relational databasessuch as Oracle, MySQL or Microsoft SQL Server, are connected to using SQL/MED

2Eigenbase Foundation, http://www.eigenbase.org3GNU Public License, http://www.opensource.org/licenses/gpl-license.php

19

CHAPTER 4. TOOLS

foreign data wrappers, which make them show up as foreign tables. These foreigntables can then be accessed by regular SQL statements just like any other table.

4.3 Analytics application development toolsThe above components make up the platform for the hosting and the delivery ofa business intelligence application. However, for the development of the analyticsapplications themselves, I have used a couple of additional LucidEra in-house tools,the Workbench and ALS.

4.3.1 The Workbench

The Workbench is the development environment for applications running on theplatform, and is based on the Eclipse4 platform. Creating an application entailscreating views, tables, queries and foreign data connectors, and the Workbenchallows me to do all this from a single environment.

4.3.2 Application Lifecycle Service

The Application Lifecycle Service (ALS) server manages the contact between theWorkbench and the delivery platform described in section 4.2, where the app getsdeployed. An application is created in the Workbench, and when the user decidesto deploy it to the platform, it is the ALS server that does the work.

4.4 Data sourcesFor each of the analysis areas (test results, continous integration, defects tracking,revision control), there are many different frameworks available on the market. Forsimplicity, I decided to go with the ones already in use at LucidEra. I describe thesein this section.

However, other comparable frameworks would provide more or less the samefunctionalities, and the applications I have developed in this project would be rea-sonably adaptable to other datasources of the same kind. It should not be too hardto replace Jira with Bugzilla5, for example, or Perforce with Subversion6.

4.4.1 Blackhawk

Blackhawk[9] is a pluggable test harness that allows for dynamic composition oftest suites based on test meta-data, parameterization of test cases, and multi-process

4Eclipse, an open-source development platform, http://www.eclipse.org5Bugzilla, an open-source bug tracking system, http://www.bugzilla.org/6Subversion, an open-source version control system, http://subversion.tigris.org/

20

4.4. DATA SOURCES

management. It supports junit7 and HTTP protocol testing, and has a pluggable ar-chitecture for connecting to other test harnesses. Blackhawk is open-sourced underthe Apache License8 as part of the Apache Beehive project.

LucidEra uses Blackhawk for automating the nightly execution of unit test suites,in order to see that features work as intended and to make sure that fixed bugs don’treappear.

4.4.2 Cruisecontrol

Cruisecontrol[25] is a framework for continous builds and integrations. It triggersbuilds to be performed at certain events (such as changes in the revision controlsystem), and then notifies various listeners of the results thereof. Cruisecontrol isopen-sourced under a BSD-style9 license10.

LucidEra uses Cruisecontrol for automating the build process. Whenever a builddependency (such as a source file) has changed, Cruisecontrol schedules builds forit’s dependant.

4.4.3 Jira

Jira[1] is a bug tracking, issue tracking, and project management application de-veloped by Australia-based Atlassian Software. Jira is proprietary to Atlassian Soft-ware.

LucidEra uses Jira to track bugs, feature requests, tasks, to-do items, and so on.

4.4.4 Perforce

Perforce[23] is a revision control system, developed by Perforce Software, Inc. ThePerforce system is based on a server/client architecture, with the server performingversion management for the individual files in a central repository, to which theclients submit changes. Perforce is proprietary to Perforce Software, Inc.

LucidEra uses Perforce for managing all it’s source code.

7JUnit is a unit testing framework for the Java programming language8Apache License Version 2.0, http://www.opensource.org/licenses/apache2.0.php9BSD License, http://www.opensource.org/licenses/bsd-license.php

10CruiseControl license, http://cruisecontrol.sourceforge.net/license.html

21

Chapter 5

Method

5.1 Introduction to application designDeveloping an analytics application for the LucidEra application stack is essentiallyabout solving the problem of taking information from various sources and structur-ing it in a database schema that lends itself well to answering interesting questionsusing the data.

This poses several challenges:

1. How to access the raw data to be analyzed.

Data can come from all sorts of sources and in all sorts of formats, but I needto access it without requiring too much human assistance.

2. How to design the resulting database schema.

I am building an analytics application to answer questions using data. Theuser asking the questions does this using a graphical interface (essentially anMDX query generator), that has a limited scope as to which questions arepossible to ask.

Thus, it is not enough that the data is there in the schema, the schema mustalso be designed in such a way that the answers are made available throughthese limited queries.

3. How to transform the raw data into this schema.

When the data sources to be analyzed are identified, and a schema that wouldprovide the user with answers to the questions we are interested in is defined,I need to figure out how to get from raw data to well-ordered schema.

Next, I will give a general introduction to how I have performed data extractionand transformation for my applications, and after that I will give brief descriptionsof how these were constructed in each specific case of the four analysis applications.

23

CHAPTER 5. METHOD

5.1.1 Extraction

By using SQL/MED data adaptors (see section 4.2.4), LucidDB can access a varietyof data sources as regular relational database tables. For the apps implemented inthis project, I used data adaptors for Oracle (in the cases of Jira and Blackhawk) andCSV (comma separated value) flatfiles (in the cases of Cruisecontrol and Perforce).

This allowed me to keep the transformation layer (see below) strictly in SQL,even though the data sources come from different sources.

5.1.2 Transformation

Once the connections to the data sources to be used for analysis are set up, the ETL(Extract, Transform, Load) layer takes over. The ETL layer is the set of rules fortransforming and loading the raw data into an ordered schema suited for analysis.

Creating an ETL layer is basically answering the question of how to get frompoint A to point B, with point A being the raw data to be analyzed, and point Bbeing a schema that meets a number of requirements. This proved to be the mosttime consuming part of analytics application development, by far.

Two typical patterns I used when creating the ETL layer was normalization,when creating dimensions out of a central table, and de-normalization when cre-ating measures out of a normalized relational database, for example. I will givereal-world examples of both these patterns being used in the cubes described be-low.

5.1.3 Star schema

The result of the ETL layer is the star schema. For each of the data sources used, Iwill list which measures and dimensions I was able to extract.

5.2 Defect tracking - JiraJira stores it’s information in a relational database, the particular instance used atLucidEra runs on an Oracle server. For extracting data from the database I usedLucidDB’s ready-made Oracle adapter.

5.2.1 Extraction

The Jira database schema is a fairly normalized one, with a table JIRAISSUE, whereeach row describes an issue, as the hub. The structure of the schema is alreadygeared towards analysis, with columns in the JIRAISSUE table containing keys torows in tables that can be used as OLAP dimensions without too much need foralteration.

24

5.2. DEFECT TRACKING - JIRA


Using a traditionally normalized relational database schema such as the one in Jiraas a data source for a mondrian OLAP cube makes creating dimensions easy (sincethe structure is already that of a central hub table containing keys to data in othertables), but getting the data for the actual measures in place required me to do de-normalization in the cases where that data is stored in separate tables. The resultingdata flow for the Jira application’s ETL layer is illustrated by figure 5.1.

Oracleadapter

Extract


BUGREPORT_FACT

IDNAME

ASSIGNEE_DIM

IDNAME

REPORTER_DIMIDYEARMONTHDAY

CALENDAR_DIM

Star schema

IDNAME

REPORTER_DIM

IDYEARMONTHDAY

CALENDAR_DIM

IDNAME

ASSIGNEE_DIM

IDYEARMONTHDAY

CALENDAR_DIM


BUGREPORT_FACT

Transform Load

LucidDB

Oracle

IDYEARMONTHDAY

CALENDAR_DIM

IDYEARMONTHDAY

CALENDAR_DIM

IDNAME

ASSIGNEE_DIM

IDNAME


CALENDAR_DIM

IDNAME

REPORTER_DIM

IDYEARMONTHDAY

CALENDAR_DIM

IDNAME

REPORTER_DIM IDPRIORITYASSIGNEE_KEYREPORTER_KEYCREATE_DATE_KEY

BUGREPORT_FACT

Figure 5.1. ETL layer for the Jira-based application

To give a concrete example of when denormalization was used: I wanted a measure’average time to resolve’, so for each issue (each row in the JIRAISSUE table) Ihad to get the date when the status of that issue was set to “Resolved”. However,that data is not there in the hub JIRAISSUE table, but is sitting in a table calledCHANGEITEM, where each row belongs to one row in the CHANGEGROUP table,where, in turn, each row belongs to one row in the JIRAISSUE table. See figure 5.2.

Also, a single issue may have been set to “Resolved” several times (it might havebeen re-opened if an attempted fix didn’t work), so I only want to get the latestresolution date.

ID: integerPKEY: varchar(255)PROJECT: integerREPORTER: varchar(255)...

JIRAISSUEID: integerISSUEID: integerAUTHOR: varchar(255)CREATED: date

CHANGEGROUPID: integerGROUPID: integerFIELDTYPE: varchar(255)FIELD: varchar(255)...

CHANGEITEM

Figure 5.2. Example of normalization in the Jira source schema

First, I join the CHANGEITEM table to CHANGEGROUP, putting all changes into oneand the same table, it could be done easier just for the case of latest resolution date,but this way I can reuse this view for other similar de-normalizations:

create view ”JIRACHANGES VIEW” asse lec t ”CHANGEITEM” . ” ID ” ,

25

CHAPTER 5. METHOD

” ISSUEID ” ,”CREATED” ,” FIELDTYPE ” ,” FIELD ”

from ”CHANGEGROUP” l e f t jo in ”CHANGEITEM”on ”CHANGEGROUP” . ” ID ” = ”CHANGEITEM” . ”GROUPID”

Second, I find out which is the latest resolution change for each ISSUEID:

create view ”LATEST RESOLUTION DATE VIEW” asse lec t ” ISSUEID ” ,

max( ”CREATED” ) as ”RESOLUTION DATE”from ”JIRACHANGES VIEW”where ” FIELDTYPE ” = ’ j i r a ’ and ” FIELD ” = ’ r e s o l u t i o n ’group by ” ISSUEID ”

Third, to use this extracted resolution date as a measure, I need to get it into thefact table, which calls for another join:

create view ”LE FACT JIRA ” asse lec t ” JIRAISSUE ” . ” ID ” ,

” JIRAISSUE ” . ”REPORTER” ,” JIRAISSUE ” . ”ASSIGNEE” ,” JIRAISSUE ” . ” PRIORITY ” ,”LATEST RESOLUTION DATE VIEW” . ”RESOLUTION DATE”

from ” JIRAISSUE ” l e f t jo in ”LATEST RESOLUTION DATE VIEW”on ” JIRAISSUE ” . ” ID ” = ”LATEST RESOLUTION DATE VIEW” . ” ISSUEID ”

5.2.3 Star schema

With the Jira relational database schema as the data source, I was able to constructan OLAP cube with the measures and dimensions listed in table 5.1.

MeasuresAvg. days to resolve Avg. number of days to resolve.

Avg. priority Avg. issue priority.Issue count (all) Number of issues.

Issue count (open) Number of open issues.Issue count (closed) Number of closed issues.

Closed % of total Percentage of issues closed.Dimensions

Issue type Type of issue (bug, task, etc).Reporter Person who reported the issue.Assignee Person assigned to resolve the issue.

Issuestatus Issue status (open, resolved, etc).Priority Bug priority.

Resolution Kind of resolution (fixed, invalid, etc).Issue name Name of issue.

Project Which project affected.Component Which subcomponent affected.

Creation date Date of creation.Resolution date Date of resolution.

Modify date Date of last modification.

Table 5.1. Measures and dimensions acquired for defect tracking

26

5.3. TEST RESULT TRACKING - BLACKHAWK

5.3 Test result tracking - BlackhawkBlackhawk is used for automatically running big sets of test cases. They can be unittests, testing some specific low-level functionality, or higher-level tests testing largerparts of the platform.

Individual tests are organized in suites, and each set of suites is then run withdifferent configurations (affecting operating systems, JVMs, optimization levels).For each test run, the results are gathered in a report covering pass/fail status andduration for each individual test and configuration. This report is then stored in adatabase.

5.3.1 Extraction

The Blackhawk server at LucidEra stores it’s results in an Oracle database. Thismade the extraction straightforward, all that needed to be done was to again utilizeLucidDB’s Oracle adapter to connect to this foreign server using SQL/MED and thencopying the data using normal SQL INSERTs.


The test report data in the Oracle server is stored in one single table. To createdimensions and make a star schema out of this, I had to move dimension data intoseparate tables, so a certain amount of normalization was needed.

Through this normalization process I use SQL SELECTs and INSERTs to trans-form the big all-in-one table into a star schema. Data is moved into separate tables,connected to the fact table using foreign keys, while some data, that which is to beused as measures, is left in the fact table. The resulting ETL process can be seen infigure 5.3.

Oracleadapter

Extract


BUGREPORT_FACT

IDNAME

ASSIGNEE_DIM

IDNAME

REPORTER_DIM

IDYEARMONTHDAY

CALENDAR_DIM

Star schema

IDNAME

REPORTER_DIM

IDYEARMONTHDAY

CALENDAR_DIM


BUGREPORT_FACT

IDNAME

ASSIGNEE_DIM

IDYEARMONTHDAY

CALENDAR_DIM


BUGREPORT_FACT

Transform Load

LucidDB

Oracle

IDYEARMONTHDAY

CALENDAR_DIM

IDYEARMONTHDAY

CALENDAR_DIM

OPENCNPLATFORMCNSUCCESSDURATIONTESTNAMEFOREIGNDB...

NIGHTLY_RESULTS

Figure 5.3. ETL layer for the Blackhawk-based application

27

CHAPTER 5. METHOD

5.3.3 Star schema

With the Blackhawk results table as the data source, I was able to construct an OLAPcube the measures and dimensions in table 5.2.

MeasuresTests Run Number of tests run.

Success Count Number of successful tests.Duration How long it took.

Fail Count Number of failed tests.Success % Percentage of tests successful.

DimensionsOpen Changeno Open source components build label.

Platform Changeno Closed source components build label.Known failure To filter out things known to be broken.

Foreign DB Which foreign database was used.Test Suite Which suite of tests.

Test Name Name of the test.CPU CPU architecture used.JVM JVM used, Sun Hotspot or BEA JRockit.

Optimization Compiler optimization level.Operating System Operating system used.

Test date Date of test run.

Table 5.2. Measures and dimensions acquired for test result tracking

5.4 Continous integration tracking - CruisecontrolCruisecontrol is used for scheduling and performing automatic builds whenever thecode base changes. When a build attempt is made, Cruisecontrol keeps track of thechangelists submitted since the last successful build. If the new build is successful,it is given a unique ID equal to the revision control system submission number ofthe last submission included. The logs from each build attempt are stored as XMLfiles on the build server.

An example chain of events could be where developer Amy submits change 3441to the revision control system. Cruisecontrol notices that build dependencies havechanged since the last successful build, and schedules a new build attempt to bemade. Before the build attempt starts, however, Bob submits change 3442. Thiswill have Cruisecontrol include both changes 3441 and 3442 in the build attempt.The Build is successful, so Cruisecontrol labels the build as number 3442, saves thelog from this attempt, and waits for the next submission to be made.

5.4.1 Extraction

Cruisecontrol logs are stored as XML files:,<?xml version="1.0" encoding="UTF -8"?><cruisecontrol >

<modifications ><modification type="p4">

<date>10/11/2006 14:06:07</date>

28

5.4. CONTINOUS INTEGRATION TRACKING - CRUISECONTROL

<user>ogothberg </user><revision >4072</revision ><email >[email protected]</email ><client >ogothberg.wabeeno.lucidera </client >

</modification ><modification type="p4">

<date>10/11/2006 14:14:40</date><user>ogothberg </user><revision >4073</revision ><email >[email protected]</email ><client >ogothberg.wabeeno.lucidera </client >

</modification ></modifications ><info>

<property name="projectname" value="platform" /><property name="builddate" value="10/11/2006 14:55:46" /><property name="label" value="4074" />...

</info><build time="46 min 11 sec">

...</build >

</cruisecontrol >

LucidDB currently doesn’t have a ready-made adapter for XML files, so in orderto make the Cruisecontrol logs importable, I created an XSL (extensible stylesheetlanguage) script to convert them into comma separated value (CSV) files:

,"LABEL","PROJECT","BUILDDATE","DURATION","ERROR","CHANGENO","CHANGEDATE","USERNAM.."4074","platform","10/11/2006 14:55:46","46 min 11 sec","","4073","10/11/2006 14:.."4074","platform","10/11/2006 14:55:46","46 min 11 sec","","4072","10/11/2006 14:..

As can be seen in this example piece of log data, each row in the table cor-responds to a revision control system submission, and there can be several suchsubmissions for each build.

CSV Flatfileadapter

Extract


BUGREPORT_FACT

IDNAME

ASSIGNEE_DIM

IDNAME


CALENDAR_DIM

Star schema

IDNAME

REPORTER_DIM

IDYEARMONTHDAY

CALENDAR_DIM

IDNAME

ASSIGNEE_DIM

IDYEARMONTHDAY

CALENDAR_DIM


BUGREPORT_FACT

Transform Load

LucidDB

IDYEARMONTHDAY

CALENDAR_DIM

IDYEARMONTHDAY

CALENDAR_DIM

XMLfiles

XSL

Figure 5.4. ETL layer for the Cruisecontrol-based application

29

CHAPTER 5. METHOD


Once the CSV files are imported as tables in the database, the transformation pro-cess for Cruisecontrol data from source table to star schema, as illustrated by figure5.4, is very similar to that for Blackhawk. That is, all data is stored in a single table,which requires normalization in order to create dimension tables.

5.4.3 Star schema

Cruisecontrol logs as a data source enables the construction of an OLAP cube withthe measures and dimensions listed in table 5.3.

MeasuresBuild count (all) Number of builds made.

Build count (successful) Number of successful builds.Build count (failed) Number of failed builds.

Success % Percentage of builds successful.Avg. number of changes Avg. number of included changelists.

Avg. duration Avg. time to do a build.Dimensions

Label Build label.Project Which project.

Build timestamp Time of build.Success If the build was successful or not.

Change timestamp Time of submitted Perforce change.Change author Author of Perforce change.

Table 5.3. Measures and dimensions acquired for continous integration tracking

5.5 Revision control tracking - PerforceWith Perforce, I chose to interface directly through the changelogs as generated byit’s end-user command-line utility. The advantage of this approach is that anyonewith a perforce account will be able to use the analytics application, as it doesn’trequire direct access to the central repository server.

5.5.1 Extraction

A submission to the perforce server consists of a changelist (one or more files added,edited, deleted) along with metadata such as a change number to uniquely identifythe change, the client submitting the change, a description, and a timestamp. Per-force allows clients to retrieve a description for a specific changelist using the p4describe command:

,Change 7688 by [email protected] on 2006/09/18 01:33:35

LUCIDDB: new UDP for calculating statistics for everything in a schema LER -1820

Affected files ...

... //open/lu/dev/luciddb/initsql/installApplib.ref#21 edit

30

5.5. REVISION CONTROL TRACKING - PERFORCE

... //open/lu/dev/luciddb/initsql/installApplib.sql#18 edit

... //open/lu/dev/luciddb/test/sql/udr/udp/test.xml#9 edit

Data in these change descriptions is nicely structured, but to be able to importit into LucidDB we have to transform it into a comma-separated-value (CSV) file:

,CHANGELIST_NUMBER ,FILE ,REVISION_NUMBER ,ACTION ,AUTHOR ,WHEN7688,//open/lu/dev/luciddb/initsql/installApplib.ref ,21,edit ,ogothberg ,2006 -09 -18..7688,//open/lu/dev/luciddb/initsql/installApplib.sql ,18,edit ,ogothberg ,2006 -09 -18..7688,//open/lu/dev/luciddb/test/sql/udr/udp/test.xml ,9,edit ,ogothberg ,2006 -09 -18 ..

The process of extracting descriptions from Perforce and transforming these intoCSV files as that above is automated using sed1, awk2, regular expressions and shellscripting.


As can be concluded from the CSV file format used for import, like in the casesof Blackhawk and Cruisecontrol, and in contrast to the case with Jira, the datafrom Perforce had to be normalized to adhere to the star schema required for OLAPanalysis. The data flow is shown in figure 5.5.

CSV Flatfileadapter

Extract


BUGREPORT_FACT

IDNAME

ASSIGNEE_DIM

IDNAME


CALENDAR_DIM

Star schema

IDNAME

REPORTER_DIM

IDYEARMONTHDAY

CALENDAR_DIM

IDNAME

ASSIGNEE_DIM

IDYEARMONTHDAY

CALENDAR_DIM


BUGREPORT_FACT

Transform Load

LucidDB

IDYEARMONTHDAY

CALENDAR_DIM

IDYEARMONTHDAY

CALENDAR_DIM

Textfiles

SED/AWK

Figure 5.5. ETL layer for the Perforce-based application

5.5.3 Star schema

With Perforce changelogs as the data source, I was able to construct an OLAP cubewith the measures and dimensions listed in table 5.4.

1Sed, http://en.wikipedia.org/wiki/Sed2Awk, http://en.wikipedia.org/wiki/Awk

31

CHAPTER 5. METHOD

MeasuresChange count (all) Number of file changes.

Submission count (all) Number of submissions.Avg. revision number Avg. revision number for changed files.

DimensionsChangelist ID Unique ID of submission.

Action Action performed (add, edit, etc).Author Submission author.Project Affected project.

Component Affected project subcomponent.Submit date Date of submission.

Table 5.4. Measures and dimensions acquired for revision control tracking

32

Chapter 6

Results

To give the reader a better feel for what analytics applications on the LucidErastack actually look like, and a better understanding for the real-world usage ofOLAP measures and dimensions, I will start out this results chapter with a coupleof screenshots from example reports, in section 6.1. The screenshots are from apre-beta version of Clearview (LucidEra’s graphical MDX-generating user interface,see section 4.2.1).

Regarding the actual aim of this thesis, a sizeable part of the analytics from mywish list in chapter 3 is fairly easy to obtain, while a few would require more work.In section 6.2, I will compare the original aim to what I was actually able to acquirethrough the approach described in chapter 5.

With the metrics in section 6.2 as the jumping board, there are a number ofpossible “next steps”, metrics that should be possible to get but that would requirealterations or additions to existing systems or workflows to obtain. I present a fewsuggestions in section 6.3.

6.1 Example reports6.1.1 Performance comparison between virtual java machines

With the test result tracking application developed in section 5.3, I was able to com-pare how long the same test suite runs in different server environments (number3.2.2 in my metrics list).

Figure 6.1 shows a report comparing the differences in execution time betweentwo virtual java machines. From an end-user perspective, the report was created bydragging and dropping measures and dimensions from the left-hand panel to thelarger right-hand report area. The report shows one measure, “Duration”, and twovisible dimensions: “JVM Name” on columns, and hierarchical dimension “Test” onrows. Note that dimension “Test” has two levels: “Test Suite” and “Test Name”.

Two additional dimensions are used for filtering to create this report: “OperatingSystem” (for filtering out only tests run on the RedHat 4 operating system) and“Platform Changeno” (for filtering out only the latest build available).

33

CHAPTER 6. RESULTS

Figure 6.1. Report on execution time for different virtual java machines.

6.1.2 Test success monitoring for recent builds

Again from the test result tracking application, figure 6.2 shows a bar chart overthe percentage of successful automated tests for the ten most recent builds, andthe differences between two different operating systems (RedHat 4 and Windows2003). It uses the measure “Success %”, as the X-axis and two dimensions “PlatformChangeno” and “Operating System” on the Y-axis.

Both dimensions showed in this chart are also used as filters for the data in it,“Platform Changeno” for filtering out only the 10 most recent build numbers, and“Operating System” for filtering out only RedHat 4 and Windows 2003.

6.2 Compared to the original aimIn this section, I will compare the metrics I was able to get from my applications towhat I initially aimed for with my list of metrics in chapter 3.

6.2.1 Metrics that were easy to get

Many simple but useful metrics are easy to retrieve from the source systems, requir-ing only fairly simple ETL processes and no changes to infrastructures or workflows.Basically, these are what a software engineering team could be able to expect “forfree” out of their in-place systems.

34

6.2. COMPARED TO THE ORIGINAL AIM

Figure 6.2. Report on the percentage of passed tests for the ten most recent builds.

Comparing the kind of measures and dimensions I was able to extract from thesource systems (see tables 5.1, 5.2, 5.3, 5.4) to the specific metrics wished for inchapter 3, table 6.1 lists those that can be deemed easy to get.

3.1 Defect tracking3.1.3 Open/close activity charts.3.1.4 Bug report quality.3.1.5 Average resolution time.

3.2 Test result tracking3.2.1 Total test results over time.3.2.2 Performance comparisons.

3.3 Continous integration tracking3.3.1 Number of runs per day.3.3.2 Time to fix.3.3.3 Number of new submissions per build.3.3.4 Number of changed files per build.3.3.5 Number of line changes per build.

3.4 Revision control tracking3.4.1 Number of checkins.3.4.2 Number of files changed per submission.3.4.3 Number of lines changed per submission.3.4.4 Number of integrations.

Table 6.1. Metrics that are easy to get from the source systems.

It should be noted that a couple of the metrics listed, 3.3.4 and 3.3.5, althoughthey could not be extracted only from continous integration tracking data, wouldbe easy to get through a bit of cross-referencing with the revision control system,

35

CHAPTER 6. RESULTS

and are therefore well within the “easy” group.

6.2.2 Metrics that would require more work

Some of the metrics (shown in table 6.2) we hoped to have are currently not pos-sible to retrieve due to data simply not being there (3.2.3 and 3.2.4), while othersjust proved to be harder to get than expected due to limitations in the analyticsplatform (3.1.1 and 3.1.2). Implementing the changes required to infrastructureand/or workflows in order to obtain these would be some work, but not out of thepossible.

3.1 Defect tracking3.1.1 Open bugs over time.3.1.2 Cumulative open/close plot.

3.2 Test result tracking3.2.3 Total test results over time.3.2.4 Performance comparisons.

Table 6.2. Metrics that would require extra work to retrieve from source systems.

Problems with cumulative measures

Due to the way that queries are executed against the star schema, reports suchas “Open bugs over time” (3.1.1) or “Cumulative open/close plot” (3.1.2) wouldrequire workarounds to get working.

The reason for this is that even though each row in the star schema’s fact tablehas a “create date” and a “close date”, which does enable metrics such as “Open/-close activity charts” (3.1.3), the OLAP server does not not allow joins on conditionsother than “equals”. That is, asking the schema “I would like the number of factsopened on the date D” is possible, but asking “I would like the number of factsopened before, but closed after, the date D” is currently not.

A possible, but ugly, workaround could be to have a stored procedure in thedatabase “explode” the fact table temporarily for this kind of queries (creating du-plicate rows for each value in the interval) to allow such metrics even when onlyjoining on “equals”.

Metrics that would require infrastructure improvements

Metrics such as “Test execution reports” (3.2.3) and “Test development reports”(3.2.4) are technically simple to retrieve from the source data, but since they arebased on the comparison to target data (number of tests expected to be running,number of tests expected to be developed) from user input, they would require theinfrastructure (extensions to software) to allow users to set these targets.

36

6.3. NEXT STEPS

6.3 Next stepsWith the metrics I was able to provide using my analytics applications, there area number of possible next steps. In this section, I will cover a few suggestions forfurther development.

6.3.1 Possible with additional data sources

Test development effectiveness

By incorporating data from code coverage1 systems such as Emma2, it would bepossible to measure code coverage as a function of test cases, to get a metric for theeffectiveness of the engineering team’s test development efforts.

Code complexity measures

By combining revision control data and code complexity measures (such as cy-clometric complexity), one could try and measure changes in code complexity byproject events.

For example, is there a correlation of jumps of complexity to large bugfixes? Ordo large bugfixes lower complexity (maybe large fixes indicate people simplifyinglogic to fix problems)?

Another question: does a higher rate of change result in a worse complexitythan the same change amount over a longer period of time (i.e. when the team is ina hectic “crunch” period of high workload, how much worse is the code produced)?

6.3.2 Possible with better cross-referencing

Issues affected per submission

Keeping track of how submissions relate to issues in the bug database would makefor interesting analysis of the team’s effectiveness. For example, how long it takesfrom the point when a bug is filed until a first attempt to fix it is made, or how manysubmissions are required to fix a bug.

Provided a reliable way to link revision control system submissions to issues inthe bug database, a metric like “Known issues affected per submission” this shouldbe possible to get. The simplest way would be to have developers always enter thebug database issue ID in the description of every submission they make, the problemis that having this done regularly enough to base analysis on would require a verydisciplined group of developers.

However, there is software available on the market to make this kind of linkinghappen more reliably, such as the Perforce plugin3 for Jira.

1Code coverage is a measure for to which degree the source code is being tested.2Emma is an open source tool for java code coverage.3Perforce plugin for Jira, http://confluence.atlassian.com/display/JIRAEXT/JIRA+

Perforce+Plugin

37

CHAPTER 6. RESULTS

File fragility

Getting a metric for how likely it is to break things by editing a certain file couldbe useful for identifying weak points in the source dependencies, basically a filefragility index.

By cross-referencing submission statistics from revision control data and buildbreakage data from continous integration tracking, this should be possible to get.

38

Chapter 7

Conclusions

This thesis has investigated the viability of using common business intelligence tech-niques in order to analyze the software engineering process.

Using multiple software-based tools, such as revision control systems and bugdatabases, in order to manage engineering projects, is already a widespread prac-tice. Therefore, the introduction of another one, provided it can add substantialsupport to managerial decisions with only a relatively low barrier of entry, shouldbe possible. What this would require, however, is analytics applications that aremodular enough to be hooked up to existing systems without too much tailoring.

In this thesis I have shown that developing such applications using businessintelligence techniques and platforms should be possible, and that this approachindeed can automate the gathering of many of the analytics that today are gatheredmanually by engineering management.

39

Bibliography

[1] Atlassian Software Jira - Bug tracking, issue tracking and project managementAccessed on October 23rd, 2006http://www.atlassian.com/software/jira/

[2] BBC News July 2nd, 2006. EADS and Airbus bosses both quitAccessed on October 2nd, 2006http://news.bbc.co.uk/2/hi/business/5138840.stm

[3] BBC News March 22nd, 2006. Microsoft delays launch of VistaAccessed on October 2nd, 2006http://news.bbc.co.uk/2/hi/business/4831374.stm

[4] Black, R. (1999) Managing the Testing Process. Microsoft PressISBN 0-7356-0584-X

[5] Brown, M. (1996) Keeping Score: Using the Right Metrics to Drive World-ClassPerformanceQuality Resources, ISBN 0-527-76312-8

[6] Caserta, J. and Kimball, R. (2004) The Data Warehouse ETL Toolkit: PracticalTechniques for Extracting, Cleaning, Conforming, and Delivering Data.Wiley, ISBN 0-764-56757-8

[7] Davenport, T. Competing On AnalyticsInformationWeek, February 1st, 2006.

[8] Desmond, J. 2006 Software 500: Deep Concerns Over SecuritySoftware Magazine, issue of October 2006

[9] Eigenbase Foundation BlackhawkWikiAccessed on October 23rd, 2006http://eigenbase.wikispaces.com/BlackhawkWiki

[10] eWeek Microsoft to Gut Longhorn to Make 2006 Delivery DateAccessed on October 2nd, 2006http://www.eweek.com/article2/0,1895,1640212,00.asp

41

BIBLIOGRAPHY

[11] ISO/IEC 9075-9:2003 Management of External Data (SQL/MED)http://www.iso.org/iso/en/CatalogueDetailPage.CatalogueDetail?CSNUMBER=34136

[12] Johnson, J. and Boucher K. and Connors, K. and Robinson, J. (2001) Collabo-rating on Project SuccessSoftware Magazine, issue of February/March 2001

[13] Kaner, C. (2000) Rethinking software metrics: Evaluating measurement schemesSoftware Testing & Quality Engineering. Vol. 2, Issue 2 (March) p. 50-57,2000.http://www.kaner.com/pdfs/rethinking_sw_metrics.pdf

[14] Kaner, C. (2002) A short course in metrics and measurement dysfunction.Tutorial session at 15th International Software Quality Conference (QualityWeek), San Francisco, CA, September, 2002http://www.kaner.com/pdfs/metrics_measurement_dysfunction.pdf

[15] Kaner, C. and Bond, W. (2004) Software Engineering Metrics: What do theymeasure and how do we know?presented at 10th International Software Metrics Symposium (METRICS ’04),Chicago, IL, September 14-16, 2004.http://www.kaner.com/pdfs/metrics2004.pdf

[16] Kannala, J. (2005) Predicting Defects in CodeKungliga Tekniska Hogskolan, TRITA-NA-E05082

[17] Kimball, R. and Ross, M. (2002) The Data Warehouse Toolkit: The CompleteGuide to Dimensional Modeling (Second Edition).Wiley, ISBN 0-471-20024-7

[18] Kimball, R. and Reeves, L. and Ross, M. Thornthwaite, W. (1998) The DataWarehouse Lifecycle Toolkit: Expert Methods for Designing, Developing, and De-ploying Data WarehousesWiley, ISBN 0-471-25547-5

[19] Krusko, A. (2004) Complexity Analysis of Real Time SoftwareKungliga Tekniska Hogskolan, TRITA-NA-E04032

[20] Eigenbase Foundation LucidDBWikiAccessed on October 23rd, 2006http://eigenbase.wikispaces.com/LucidDbWiki

[21] Mann, C. Why Software Is So BadTechnology Review, issue of July/August 2002

42

[22] Mills, E. (1988) Software MetricsCarnegie Mellon University, SEI Curriculum Module SEI-CM-12-1.1.ftp://ftp.sei.cmu.edu/pub/education/cm12.pdf

[23] Perforce Software, Inc. Perforce Product InformationAccessed on October 23rd, 2006http://www.perforce.com/perforce/products.html

[24] Pressman, R. (1992) Software Engineering: A practitioner’s approachMcGraw-Hill, ISBN 0-07-050814-3

[25] SourceForge.net CruiseControl OverviewAccessed on October 23rd, 2006http://cruisecontrol.sourceforge.net/overview.html

[26] SourceForge.net MondrianAccessed on October 23rd, 2006http://sourceforge.net/projects/mondrian

[27] The Standish Group International (1995) The CHAOS Report (1994)Accessed on October 28th, 2006http://www.standishgroup.com/sample_research/chaos_1994_1.php

[28] Tassey, G. (2002) The Economic Impacts of Inadequate Infrastructure for Soft-ware TestingNational Institute of Standards and Technology (NIST)Accessed on October 28th, 2006http://www.nist.gov/director/prog-ofc/report02-3.pdf

[29] Thomsen, E. and Spofford, G. and Chase, D. (1999) Microsoft OLAP Solutions.Wiley, ISBN 0-471-33258-5

[30] Whitehorn, M. and Zare, R. and Pasumansky, M. (2002) Fast Track to MDXSpringer, ISBN 1-85233-681-1

[31] Wikipedia Business IntelligenceAccessed on October 2nd, 2006http://en.wikipedia.org/wiki/Business_Intelligence

[32] Wikipedia Multidimensional ExpressionsAccessed on October 24th, 2006http://en.wikipedia.org/wiki/Multidimensional_Expressions

43

TRITA-CSC-E 2007:003 ISRN-KTH/CSC/E--07/003--SE

ISSN-1653-5715

www.kth.se

Applying Business Intelligence to Software Engineering

Documents

Transcript of Applying Business Intelligence to Software Engineering