Business Intelligence and Reporting Tools Evaluation: BIRT vs Pentaho
by
ZHI ZHANG (100985308)
WENJING ZHANG (7352330)
A Project Report
submitted in fulfillment of the requirements for CSI5112 SOFTWARE ENGINEERING
Approved by: Professor
Daniel Amyot April 17, 2015
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
– ii–
Executive Summary
The primary objective of this report is to select a BI reporting tools that satisfy our organization's need
to deliver enterprise portal applications to our customers. More specifically, we performed comparative
analysis on behalf of Ivy Technologies, an independent software vendor that provides IT management
solutions for the retail industry. The company is hoping to find a proper BI reporting tool that enables
their developers to efficiently deliver customized reporting portal for the clients, so as to improve the
performance and functionality of client's current reporting systems.
It is for this reason that we conduct comparative study on two of the most popular and powerful tools
by including a series of testing, analysis and critiques. This report adopted the criteria-based assessment,
which is a quantitative method for software selection and evaluation. We obtained our evaluation
results both from the experimentations conducted using two products, and the reviews from Innovent
Solutions [4]. In addition we made reference to documentations from the open source community,
which covers a great deal of information on specific features.
Our evaluation shows that both products satisfied Ivy Technologies’ basic requirements for a BI
reporting tool, with BIRT scores a bit higher than Pentaho by 6%. As open source tools, however, some
advanced features are not available in these community versions. Based on the evaluation results, we
made the decision to recommend BIRT to Ivy Technologies due to its superior usability and richer
functionality. Since our evaluation does not include scalability testing, further investigation needs to be
conducted if Ivy Technologies adds new requirements in terms of scalability and integration.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
– iii–
Revision History
Version Description Summary of changes Date
V 1.0 Initial Draft Created outline 2015-3-10
V 1.1 Revised Draft Experiment and results documented 2015-3-28
V 1.2 Revised Draft Content revised 2015-04-04
V 2.0 Final Document Document ready for final review 2015-04-17
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
– iv–
Table of Contents Executive Summary ...................................................................................................................................... ii
Revision History ........................................................................................................................................... iii
List of Figures ............................................................................................................................................... vi
List of Tables ............................................................................................................................................... vii
Glossary ..................................................................................................................................................... viii
1. Introduction .......................................................................................................................................... 1
1.1 Business Context ........................................................................................................................... 1
1.2 Evaluation Goals ............................................................................................................................ 1
1.3 Evaluation Scope ........................................................................................................................... 2
2. Evaluation Criteria ................................................................................................................................ 3
2.1. Overview ....................................................................................................................................... 3
2.2. Report Designer Components ....................................................................................................... 4
2.3. Data Source ................................................................................................................................... 5
2.4. Charting Criteria Evaluation .......................................................................................................... 6
2.5. Parameterization and Output Format Criteria Evaluation ............................................................ 7
2.6. Analytical Functionality Criteria Evaluation .................................................................................. 7
2.7. Usability Criteria Evaluation .......................................................................................................... 8
2.8. Installation and Implementation Criteria Evaluation .................................................................... 9
2.9. Miscellaneous Criteria Evaluation ............................................................................................... 10
3. Methodology ...................................................................................................................................... 11
4. Evaluation of Pentaho Report Designer ............................................................................................ 12
4.1. Report Designer Components Criteria Evaluation ...................................................................... 13
4.2. Data Source Criteria Evaluation .................................................................................................. 15
4.3. Charting Criteria Evaluation ........................................................................................................ 17
4.4. Parameterization and Output Format Criteria Evaluation .......................................................... 19
4.5. Analytical Functionality Criteria Evaluation ................................................................................ 20
4.6. Usability Criteria Evaluation ........................................................................................................ 21
4.7. Installation and Implementation Criteria Evaluation .................................................................. 22
4.8. Miscellaneous Criteria Evaluation ............................................................................................... 23
4.9. Evaluation Summary ................................................................................................................... 24
5. Evaluation of BIRT .............................................................................................................................. 25
5.1. Report Designer Components Criteria Evaluation ...................................................................... 27
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
– v–
5.2. Data Source Criteria Evaluation .................................................................................................. 29
5.3. Charting Criteria Evaluation ........................................................................................................ 29
5.4. Parameterization and Output Format Criteria Evaluation .......................................................... 31
5.5. Analytical Functionality Criteria Evaluation ................................................................................ 33
5.6. Usability Criteria Evaluation ........................................................................................................ 35
5.7. Installation and Implementation Criteria Evaluation .................................................................. 36
5.8. Miscellaneous Criteria Evaluation ............................................................................................... 37
5.9. Evaluation Summary ................................................................................................................... 38
6. Conclusions and Recommendations .................................................................................................. 40
Reference .................................................................................................................................................... 41
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
– vi–
List of Figures Figure 4-1 PRD UI with Welcome Window ................................................................................................. 12
Figure 4-2 Preview report in html output format ....................................................................................... 13
Figure 4-3 Data source connection in PRD ................................................................................................. 16
Figure 4-4 Chart editor in PRD .................................................................................................................... 17
Figure 4-5 Parameter editor in PRD ............................................................................................................ 19
Figure 5-1 BIRT Report Designer UI ............................................................................................................ 25
Figure 5-2 Create a new Data source .......................................................................................................... 26
Figure 5-3 Define a new Data set ................................................................................................................ 26
Figure 5-4 Preview report in web browser ................................................................................................. 27
Figure 5-5 Architecture Overview of BIRT .................................................................................................. 28
Figure 5-6 Interactive chart by adding script .............................................................................................. 30
Figure 5-7 Cascading parameter group in BIRT .......................................................................................... 32
Figure 5-8 Define Data Cube in BIRT ........................................................................................................... 34
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
– vii–
List of Tables Table 2-1 Evaluation Criteria Categories ....................................................................................................... 3
Table 2-2 Criteria Impact Levels and Numerical Weights ............................................................................. 4
Table 2-3 Description of the Report Designer Components Criteria ............................................................ 4
Table 2-4 Description of Data Source Criteria .............................................................................................. 5
Table 2-5 Description of Charting Criteria .................................................................................................... 6
Table 2-6 Paremeterization and Output Format Criteria Evaluation ............................................................ 7
Table 2-7 Analytical Functionality Criteria Evaluation .................................................................................. 7
Table 2-8 Usability Criteria Evaluation .......................................................................................................... 8
Table 2-9 Installation and Implementation Criteria Evaluation .................................................................... 9
Table 2-10 Miscellaneous Criteria Evaluation ............................................................................................. 10
Table 3-1 Criteria Evaluation Score ............................................................................................................. 11
Table 4-1 Report Designer Components Criteria Evaluation ...................................................................... 14
Table 4-2 Data Source Criteria Evaluation .................................................................................................. 16
Table 4-3 Charting Criteria Evaluation ........................................................................................................ 18
Table 4-4 Paremeterization and Output Format Criteria Evaluation .......................................................... 20
Table 4-5 Analytical Functionality Criteria Evaluation ................................................................................ 20
Table 4-6 Usability Criteria Evaluation ........................................................................................................ 21
Table 4-7 Installation and Implementation Criteria Evaluation .................................................................. 23
Table 4-8 Miscellaneous Criteria Evaluation ............................................................................................... 24
Table 5-1 Report Designer Components Criteria Evaluation ...................................................................... 28
Table 5-2 Data Source Criteria Evaluation .................................................................................................. 29
Table 5-3 Charting Criteria Evaluation ........................................................................................................ 30
Table 5-4 Parameterization and Output Format Criteria Evaluation .......................................................... 32
Table 5-5 Analytical Functionality Criteria Evaluation ................................................................................ 34
Table 5-6 Usability Criteria Evaluation ........................................................................................................ 35
Table 5-7 Installation and Implementation Criteria Evaluation .................................................................. 36
Table 5-8 Miscellaneous Criteria Evaluation ............................................................................................... 37
Table 6-1 Evaluation Scores Comparison .................................................................................................... 40
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
– viii–
Glossary
ad-hoc query The process by which a user can create a new report to answer a specific question
immediately.
BI Short for Business Intelligence. It is a set of business applications and technologies for
gathering, storing, analyzing, and converting data into information to enable future
business strategies and decisions to be made.
Dashboard It is a user interface that organizes and presents information in an easy-to-read format
and help organizations align people’s actions with strategy by tracking and analyzing key
business metrics and goals.
Data cube A set of data that is usually constructed from a subset of a data warehouse and is
organized and summarized into a multidimensional structure defined by a set of
dimensions and measures.
Filter A saved subset of information pulled from your database in a query that is based on
certain criteria.
MDX It is the query language for multidimensional databases. It is equivalent to SQL.
OLAP Short for On-Line Analytical Processing. It is an approach to building databases that are
optimized for analytical purposes.
Portal A web site that typically provides personalized capabilities to its visitors.
PRD Pentaho Report Designer
UI User Interface
XML Short for Extensible Mark-up Language. It offers a widely adopted standard way of
representing text and data in a format that can be processed without much human or
machine intelligence.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
1
1. Introduction
1.1 Business Context This project intends to evaluate business intelligence and reporting tools for Ivy Technologies. Ivy
Technologies is an independent software vendor that provides IT management solutions for the retail
industry. Since last year, Ivy Technologies has received requests from clients asking for services
regarding data analysis and online reporting. Most of these clients complain about poor performance,
lack of visualization and little customization of their current reporting system, for example, end users
were forced to manually modify report output to cope with specific queries. Given the situation, Ivy
Technologies decided to build up a new team to deliver enterprise portal applications that offer robust
reporting framework and analytics capabilities. The management team are considering the possibility of
utilizing a BI application tool to develop customized reporting portal that can access client's existing
databases, view query results from a browser, and be able to design the report framework based on
end-user needs. Currently Ivy Technologies does all its web application developments in Eclipse IDE,
therefore the preference is Eclipse-based integrated open source tools.
1.2 Evaluation Goals The purpose of this evaluation is to perform objective comparison between two open source BI
reporting tools, Pentaho and BIRT, and find the best possible choice that match the client’s requirement
and can be used in production environment.
By considering the problem mentioned in our business context above, we list the following
requirements that must be satisfied for a suitable candidate:
Provide common report designer components;
Support multiple data sources;
Support common chart types;
Provide common analytical functionality;
Support multiple platform; and
Support deployment onto application servers.
The tool should also meet several soft goals include:
Ease of use via simple user interface;
Provide sufficient documentation manuals and online repositories/FAQs/Forums;
Low product cost; and
Easy to maintain and update.
The winner must pass all mandatory requirements and get higher final score.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
2
1.3 Evaluation Scope There are a number of open source business intelligence and reporting tools available in the market,
such as BIRT, Jaspersoft, Pentaho, SpagoBI, just to name a few. After our initial filter, we narrowed down
our choice to two open source BI reporting tools, BIRT and Pentaho. The reason is that Jaspersoft focus
on nicely formatted printed output of report rather than extensive use of charting, while SpagoBI is a
100% open source BI tool with no commercial edition, which limits the choice of our client when they
can afford to upgrade to an enterprise edition with new features and better performance.
Our investigation into these two tools will be limited to a comparative evaluation of the functionalities
of community versions, namely Pentaho Report Designer 5.2 and BIRT Version: 4.4.2. The commercial
versions provide extended features based on these open source platforms, and will be mentioned but
will not get tested in this report. Community versions of two tools will be installed on windows
platforms and evaluated by two separate arbiters.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
3
2. Evaluation Criteria This section describes the set of evaluation criteria used to assess the business intelligence and
reporting tools: Pentaho Report Designer and BIRT. These criteria were selected to reflect the needs of
our client.
2.1. Overview When assessing the quality of software, it is always good to balance between hard objectivity and
subjective individual user experience. This report will adopt the criteria-based assessment which is a
quantitative method for software selection and evaluation. A criteria-based assessment will look at the
quality of software in a number of areas which stated in ISO/IEC 9126 Software engineering – Product
quality[1]. There are 27 criteria in total used for assessment and grouped into the following 8 categories
outlined in table 2-1.
Table 2-1 Evaluation Criteria Categories
Criteria Category Description
Report Designer Components Focuses on evaluating the tool’s user interface components and advanced functional components. Refer to section 2.2 and Table 2-3 for the complete list of criteria.
Data Source Focuses on evaluating the tool’s supported data source and the way used to refine that data source. Refer to section 2.3 and Table 2-4 for the complete list of criteria.
Charting Focuses on evaluating the tool’s charting functionality. Refer to section 2.4 and Table 2-5 for the complete list of criteria.
Parameterization & Output Format Focuses on evaluating the tool’s parameterization and output format which enables report readers to customize the content of the output. Refer to section 2.5 and Table 2-6 for the complete list of criteria.
Analytical Functionality Focuses on evaluating the tool’s ability to analyze multidimensional data. Refer to section 2.6 and Table 2-7 for the complete list of criteria.
Usability Focuses on evaluating the tool’s user experiences and available support offered. Refer to section 2.7 and Table 2-8 for the complete list of criteria.
Installation &Implementation Focuses on evaluating the tool’s ease of installation and implementation. Refer to section 2.8 and Table 2-9 for the complete list of criteria.
Miscellaneous Focuses on evaluating the tool’s costs and stability. Refer to section 2.9 and Table 2-10 for the complete list of criteria.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
4
According to the evaluation methodology, each criterion is assigned a relative weight (1-Nice to Have, 5-
Important, 10-Crucial), based on the relevance and importance of that specific criterion to the product
requirements and desires. The detailed description is listed in Table 2-2.
Table 2-2 Criteria Impact Levels and Numerical Weights
Criteria Impact Level Weight Description
Crucial 10 This is the minimum requirement which must be satisfied by the tool.
Important 5 This is an important or highly desired factor for potential solution.
Nice to Have 1 This is a low impact factor which will have little impact on decision.
2.2. Report Designer Components For our evaluation goals, we want to find a suitable solution which can provide flexible reporting and
printing functionality using data from multiple source and supports output to display devices, printers,
PDF, XHTML, XML, and CVS files. In that case, we want to evaluate what common and advanced
components of report designer contained, in order to fulfil the complex tasks as mentioned above.
Table 2-3 lists two evaluation criteria under report designer components category.
Table 2-3 Description of the Report Designer Components Criteria
Report Designer Components Criteria
Impact Level Description
Common Report Designer Components
10-Crucial This criteria describes the product’s user interface common components it must contained, in order to help our client to design and publish any style of report easily. This is evaluated based on testing whether the product contains the following components in order to fulfil the most common reporting requirements, such as: Report Editor, Palette, Data Explorer, Property Editor, Preview, Expression Builder, Chart Builder and Script Editor. We will create a couple of sample reports which contains different features in order to cover the above components as much as possible to see whether these reports can published without problem.
Advanced Components 5-Important This criteria describes the product’s advanced functional components that desired to have.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
5
Report Designer Components Criteria
Impact Level Description
This is evaluated based on testing whether the product contains the following components, such as: Sub-reports, Side-by-side report, Cross-tabs, Horizontal panning, Multi-column layout, CSS controlled format, etc.
2.3. Data Source For our business context, clients can have a wide variety of databases, therefore we would like the
product to support as many types of data sources as possible. This is also an important category in our
evaluation as many advanced reporting features are based on an appropriate data source.
Table 2-4 lists three evaluation criteria under data source category.
Table 2-4 Description of Data Source Criteria
Data Source Criteria Impact Level Description
Support Multiple Data Sources 10-Crucial This criteria describes whether multiple data sources and queries can be supported in one report. This evaluation is based on assessing how to include multiple data source in one report and how many types of data source can be used in the product. According to our client’s requirement, the suitable tool should cover as much as possible common data source types. In addition, does it have a (graphical) query designer is also an essential aspect to considerate.
Support for Joining Multiple Data Sources in the Designer
5-Important This criteria describes whether joining multiple data sources can achieved in the product. This evaluation is based on assessing how to join multiple data sources in the product.
Support Non-JDBC Data Sources 5-Important This criteria is asking whether the product support to connect non-JDBC data sources or any advanced data source, such as CSV, EJB, Excel, Hibernate, JNDI, JSON, Mondrian, MangoDB. This is evaluated based on how many different types of non-JDBC data source can the product supported and assessing the ease of connection.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
6
2.4. Charting Criteria Evaluation Based on the client’s expectation, charting has been considered to be one of the most important criteria
when it comes to decision.
Table 2-5 lists five evaluation criteria under data source category.
Table 2-5 Description of Charting Criteria
Charting Criteria Evaluation Score Observations and Comments
Support for Common Chart Types
10-Crucial This criteria describes the product’s common chart types it must contained. This is evaluated based on testing whether the product contains the following chart types, such as: 2D, 3D, Pie, Multi-pie, Bar, Stacked Bar, Bar XY Line, Line XY Area, Area XY, Stacked Area, Bar Line, Bubble, Scatter Plot, Multi-Axis. Besides, the ease of creating a chart is also a major factor to be assessed here.
Chart Interactivity 5-Important This criteria describes the product’s ability in term of interactivity function in charting. This is evaluated based on assessing whether the product contains interactivities, such as: mouse-over, tool tips, drill-through, hyperlinks, hide/show, etc.
Advanced Charts 5-Important This criteria describes how many types of advanced charts the product contained, in order to meet the various needs of clients. The advanced chart types which our client is looking for are: Radar, Gantt, Ring, Tube, Cone, Waterfall, Difference chart. This is evaluated based on what type of advanced charts does the product have apart from the common types mentioned above and whether these charts are satisfied the client’s needs.
Variety of Chart Theme 1-Nice to Have This criteria describes the product’s choice of chart theme. This is evaluated based on how many chart theme does it offer.
Support for Scripting 5-Important This criteria describes the product’s feasibility to support scripting language in charting. This is evaluated based on testing whether the
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
7
Charting Criteria Evaluation Score Observations and Comments
product contains the function that enable the user to directly type in code from a supported scripting language and how many scripting languages are supported, which JavaScript is a mandatory requirement here.
2.5. Parameterization and Output Format Criteria Evaluation Instead of generating multiple reports to share a same layout and probably the same data,
parameterization is used to maintain the output of report that enables the report reader be able to
change the structure or value of some of the data.
Table 2-6 lists three evaluation criteria under Parameterization and Output Format category.
Table 2-6 Parameterization and Output Format Criteria Evaluation
Parameterization & Output Format Criteria
Evaluation Score Observations and Comments
Static Parameters Selection 10-Crucial This criteria describes the product’s ability to query certain parts of data by setting parameters. This is evaluated based on testing how many value types of parameter offered by the product, whether the default value can be changed based on the formula we specified.
Dynamic Parameters Selection 5-Important This criteria describes the product’s ability to add dynamic interactivity to a published report such that when a user executes or views it, he can specify how to constrain certain parts of the query data. This is evaluated based on testing the above features in our sample reports.
Multiple Output Formats 5-Important This criteria describes which output formats are available. This is evaluated based on how many types of output formats offered in the product. The minimum requirement on output formats included: HTML, PDF, Excel and XML.
2.6. Analytical Functionality Criteria Evaluation Table 2-7 lists five evaluation criteria under analytical functionality category.
Table 2-7 Analytical Functionality Criteria Evaluation
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
8
Analytical Functionality Criteria Evaluation Score Observations and Comments
Common Aggregations 10-Crucial This criteria describes the common aggregation type each tool contains. The minimum requirement includes: sum, average count, min, max, and variance. This is evaluated based on testing whether the tool satisfies the minimum requirements and how many aggregations the tool can offer.
User-defined Aggregations 5-Important This criteria describes the product’s ability on user-defined aggregations. This is evaluated based on whether the user can define aggregations by using script language.
Slice & Dice 5-Important These analytical functional criteria is used to test whether any data mining/analysis function is built-in the produce. These criteria can be seen as an advanced functionality which our client considered to be desired to have within the product and can help our client to view the report through a dynamic approach. This is evaluated based on testing or assessing whether the followed functionalities are included in the produce: slice & dice, drill-down & roll-up, sorting and ranking.
Drill-down & Roll-up 5-Important
Sorting & Ranking 5-Important
2.7. Usability Criteria Evaluation Table 2-8 contains the list of usability criteria for this evaluation. User efficiency is always a crucial factor
to include for all evaluation. We have also include the documentation and support criteria, in order to
discuss how well or bad an open source tool can offer in term of available support.
Table 2-8 Usability Criteria Evaluation
Usability Criteria Evaluation Score Observations and Comments
User Efficiency 5-Important This criteria describes the product’s most functions and available actions are designed to be easy accessible and easy to understand. This is evaluated based on:
How long it will take to set up the environment?
Does the graphical user interface considered user-friendly?
Whether sufficient template and examples are available?
Whether prior technical knowledge is
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
9
Usability Criteria Evaluation Score Observations and Comments
required to create reports?
Does provide any preview regarding to query on data source, charting and report?
Product Documentation 5-Important This criteria describes the product’s availability of documentations as an open source tool. This is evaluated based on how many documentations offered on web, does it free and does it updated frequently, etc.
Support 1-Nice to Have This criteria describes the product’s availability of support as an open source tool. This is evaluated based on:
How well supported in terms of forum activity?
How many new posts per day in forum?
Does user need to pay for the support?
2.8. Installation and Implementation Criteria Evaluation Table 2-9 lists four evaluation criteria under installation and implementation category.
Table 2-9 Installation and Implementation Criteria Evaluation
Installation and Implementation Criteria
Evaluation Score Observations and Comments
Supports Multiple System Platforms
10-Crucial This criteria describes the product’s feasibility to run on which platform. This is evaluated based on:
Can the product be used on all three mainstream platforms: Windows, Linux and iOS?
What is the minimum system requirements to run this tool?
What is the minimum memory needed to run this tool?
Deploying Report Onto an Application Server
10-Crucial This criteria describes the product’s ability to deploy report on an application server. This is evaluated based on assessing:
Whether the product can deploy a report through an application server?
Does the application server is free of charge
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
10
Installation and Implementation Criteria
Evaluation Score Observations and Comments
Support IDE Plug-in 1-Nice to Have This criteria describes the product’s feasibility of running as IDE plug-in. This is evaluated based on whether the product support IDE plug-in.
No Compilation Before Running Report
1-Nice to Have This is evaluated based on running sample reports, in order to assess whether compilation required before running it.
2.9. Miscellaneous Criteria Evaluation Table 2-10 lists two evaluation criteria under miscellaneous category.
Table 2-10 Miscellaneous Criteria Evaluation
Miscellaneous Criteria Evaluation Score Observations and Comments
Product Cost 1-Nice to Have This is evaluated based on how much does it cost to acquire the product and other associated products.
Product Stability 5-Important This criteria describes the stability of the product. This is evaluated based on observations of the product's performance in our experimental test environment. We will test the product by playing with it in a normal way or in an unusual way to test whether it will crash or not. And we will also judge it by checking how often the product gets updated, has new features released and reacts to reported bugs.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
11
3. Methodology We will use a decision matrix to score and compare two BI tools. Each tool will be scored for evaluation
criteria listed above, from 0-3 (0-Not Available, 1-Minimal, 2-Proper, 3-Exceptional), as shown in Table 3-
1, and each criterion is assigned a relative weight (1-Nice to Have, 5-Important, 10-Crucial), based on the
view of the importance of each factor in our evaluation.
Table 3-1 Criteria Evaluation Score
Evaluation Score Score Description
Not Available 0 This element is not available in the product.
Minimal 1 This element is present but with limited features, which cannot fully satisfied the user’s requirements.
Proper 2 This element is present with comprehensive features, which satisfies the user’s requirements
Exceptional 3 This element is present with outstanding features, which is far beyond the user’s requirements.
A sample retailer database will be used to develop BI application using both tools, on which we can run
tests with regards to selected criteria. The results of the tests will be collected using a spreadsheet that
represents the evaluation matrix. The tool with a higher and satisfactory overall score will be the one
recommended for Ivy Technologies management team. For a close contest, the one scores high on
crucial and important criteria will be selected (a draw on crucial and important criteria indicates that
they are homogeneous tools, thus we need to include extra considerations such as availability of
technical support, potential features offered in the future, etc). If both tools exhibits poor performance,
alternatives will be considered and discussed.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
12
4. Evaluation of Pentaho Report Designer This section will perform the evaluation of Pentaho Report Designer (PRD) based on the evaluation
criteria and methodology we introduced in Section 3.
Pentaho is a complete BI suite, consisted of several open source projects, such as reporting, analysis,
dashboards and data mining. This evaluation report focuses on the reporting capabilities which the
project named Pentaho Report Designer. The version under evaluation is Pentaho Report Designer 5.2
which is free and open source software available under the terms of the GNU Lesser General Public
License (LGPL) version 2.1 [3].
Pentaho Report Designer provides a graphical user interface to help user design and publish any style of
report easily, which is totally different from the traditional way of creating reports programmatically or
by creating and manipulating XML. In addition, Pentaho Report Designer runs report designs against
data sources and renders report output in HTML, PDF, Excel and other formats.
When users start the journey from PRD, the first view is the welcome screen as shown in Figure 4-1. The
purpose of this window is to provide a simple and quick process for new user by clicking one of the
buttons named Report Wizard or New Report. In addition, this screen also shows a lists of sample
reports on the right side of screen, giving the user an intuitive view as to how to use a certain PRD
feature.
Figure 4-1 PRD UI with Welcome Window
Normally, the process of creating a report in PRD will include the following process [2]:
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
13
Connect to a data source;
Write a query to select the data according to your needs;
Organize data elements in the PRD workspace;
Adding aggregates based on your needs; and
Finally, publish the report, either to the Pentaho BI Server or supported output format.
Figure4-2 shows the preview report in html output format.
Figure 4-2 Preview report in html output format
4.1. Report Designer Components Criteria Evaluation Whenever using a new product, the first thing user need to learn is how to navigate the user interface,
before they move on to more complex tasks. PRD consists of several components as follows [2]:
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
14
The Welcome Screen: provides a quick, four-step process that walks the user through creating a
new report through the Report Design Wizard.
The Report Designer Main Toolbar: makes some of the most frequently used features more
accessible to users, such as: file, data source, publishing report and cut-and-paste operations.
Report Designer's Tabbed Views: views each report or sub-report, similar to text editors and
web browsers.
The Report Workspace: the place to design and customize the report. This component
dominated by the layout bands which define each individual section of the report.
The Structure Pane: shows the hierarchy of every element included in a report.
The Data Pane: enables the user to add data sources and view the individual queries, functions,
and parameters in each report.
The Style Pane: displays all of the visual and positional style options for any given item in the
Structure pane.
The Attributes Pane: displays all of the low-level properties, and input and output options for
any given item in the Structure pane.
The Palette: contains all of the elements that users can add to build a report.
Table 4-1 contains the evaluation scores, observations and comments of the report designer
components criteria for PRD.
The evaluation of this product resulted in a final weighted score of 40 out of 45 for the report designer
components criteria.
Table 4-1 Report Designer Components Criteria Evaluation
Report Designer Components Criteria
Evaluation Score Observations and Comments
Common Report Designer Components
Exceptional (3)
As listed above, Pentaho Report Designer has included all basic user interface components, in order to help users design and publish any style of report easily. We have tested a couple of sample reports, such as product catalogue, revenue by product line, top n customers, html action report, which all of them can be successfully created and generated in the web browser.
Advanced Components Proper (2)
PRD does have some advanced components to provide a wide variety of choice, such as: Geometric shapes, Barcodes, Table of Contents, Sub-reports, Side-by-Side report, Hyperlinks within a report, etc. Sub-report is an important component in PRD, as its use sub-report to support reports with multiple queries, multiple group sections and multiple data sources.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
15
Report Designer Components Criteria
Evaluation Score Observations and Comments
But some components like cross-tabs, horizontal panning and multi-column layout are not available in Pentaho.
4.2. Data Source Criteria Evaluation The second step of creating a report is to setup a data source connection. Figure 4-3 displays the
database connection wizard in PRD, which the user need to first choose a connection type, then provide
all setting information and run a test to verify the status of connection. Once the connection is valid, the
user can write a query or use a SQL query editor to refine that data source such that it only contains the
information specified by the user.
PRD supports the following data sources types [2]:
JDBC: 47 JDBC-compliant database are included in PRD, however users will probably provide the
appropriate JDBC JAR file to the Pentaho directory.
Metadata: A Pentaho Metadata XMI file.
Pentaho Data Integration (kettle): Kettle KTR files can act as a data source, but extra JAR files are
required.
OLAP: only support OLAP source created by Pentaho Analysis (Mondrian) application.
XML.
XLS: a data table which imported from excel spreadsheet file.
MangoDB: use data stored in this document-oriented non-relational database.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
16
Figure 4-3 Data source connection in PRD
Table 4-2 contains the evaluation scores, observations and comments of the data source criteria for PRD.
The evaluation of this product resulted in a final weighted score of 55 out of 60 for the data source
criteria.
Table 4-2 Data Source Criteria Evaluation
Data Source Criteria Evaluation Score Observations and Comments
Support Multiple Data Sources Exceptional (3)
Pentaho can connect with 47 data source through JDBC and other types of data source as discussed above which fully covers the databases used by the clients of Ivy Technologies. Multiple data source and queries can achieved by using sub-report. By testing the connection with our sample database, we found that the tutorial or the data source connection wizard is not straightforward.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
17
Data Source Criteria Evaluation Score Observations and Comments
Support for Joining Multiple Data Sources in the Designer
Exceptional (3)
Pentaho Data Integration (Kettle) is another tool in Pentaho BI Suite, which can provide the data source to PRD. Using Kettle with PRD allows users to specialize on either the data acquisition or presentation [4].
Support Non-JDBC Data Sources Proper (2)
Again, PRD can support most of non JDBC data source by itself, but still requires extra support from Pentaho Data Integration to provide data source as Excel, Hbase and JSON [4].
4.3. Charting Criteria Evaluation By testing PRD, the process of creating a chart is seen as the most difficult task in the whole procedure
of designing a report, which the user need to fill required property value in a large dialog box as shown
in Figure 4-4. Furthermore, no wizard to guide users through the whole process, which makes it even
more difficult. During our testing, we also found that, whenever the user fill in some values under each
category in property dialog and decide to change a chart type by clicking the type illustrated along the
top, they are likely to lose every inputs in the editor without any warning message pop-up.
Figure 4-4 Chart editor in PRD
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
18
Table 4-3 contains the evaluation scores, observations and comments of the charting criteria for PRD.
The evaluation of this product resulted in a final weighted score of 55 out of 78 for the charting criteria.
Table 4-3 Charting Criteria Evaluation
Charting Criteria Evaluation Score Observations and Comments
Support for Common Chart Types
Proper (2)
PRD supports 14 common chart types, includes: 2D, 3D, Pie, Multi-pie, Bar, Stacked Bar, Bar XY Line, Line XY Area, Area XY, Stacked Area, Bar Line, Bubble, Scatter Plot, Multi-Axis, which satisfied what we are looking for. However, one weak points of charting in PRD is does not have a charting wizard to provide step-by-step guidance to create a chart. In addition, PRD provide property editor to set value and actions to each property while BIRT is providing dialog screens with guidance and structure to the property settings. This editor is also difficult to use as it does not offer users the pick-lists instead of a pop-up window that doesn’t provide any guidance on default inputs. This makes charting become more difficult in PRD.
Chart Interactivity Proper (2)
PRD does support a few html actions such as: on-click, double-click, on-mouse-down, on-mouse-up and hyperlinks. But, the following actions are not available in PRD: drill-through, hide/show series.
Advanced Charts Proper (2)
PRD supports 11 advanced chart types as follows: Ring, Waterfall, Step Area, Step, Difference, Radar, Thermometer, Survey Scale and Bar/Line/Pie Sparkline. However, some advanced charts are not included in PRD, such as: Study, Tube, Cone, Pyramid, Time Series, Meter and Gantt chart, which some of them are required by our clients.
Variety of Chart Theme Not Available (0)
PRD do not provide any chart theme.
Support for Scripting Exceptional (3)
The script category in PRD enable users to directly type in code from a supported scripting language as follows [2]:
JavaScript
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
19
Charting Criteria Evaluation Score Observations and Comments
Groovy
Bean Shell
Netrexx
XSLT
JACL or Jython
4.4. Parameterization and Output Format Criteria Evaluation In PRD, parameters are included in data pane that enables users to add data sources and view queries,
functions and parameters in each report (see Figure 4-5). Once the query is well formed, we can also
add a parameter to the report, which enables report readers to customize the content of the output.
Right click the parameter icon in data pane, a parameter editor will pop-up. In parameter editor, users
can choose the value type from Number, Date, Time, Integer, Float, Double etc., and create a formula in
a formula editor.
Figure 4-5 Parameter editor in PRD
Table 4-4 contains the evaluation scores, observations and comments of the parameterization &output
format criteria for PRD.
The evaluation of this product resulted in a final weighted score of 55 out of 60 for the parameterization
&output format criteria.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
20
Table 4-4 Parameterization and Output Format Criteria Evaluation
Parameterization & Output Format Criteria
Evaluation Score Observations and Comments
Static Parameters Selection Exceptional (3)
PRD supports static parameters select parameter values from a hard-coded list of values During our testing, we can have an option in the report to change data values in a drop-down list. These kind of parameter are added after the data structure has been defined through a query [2].
Dynamic Parameters Selection Exceptional (3)
PRD supports dynamic parameters users select parameters from a list of values that came from a database.
One example is that we can add dynamic interactivity to a published report, which a user can specify how to constrain certain parts of the query data while viewing it. However, this requires a Pentaho Analysis (Mondrain) data source type, in order to create an OLAP-based report [2].
Multiple Output Formats Proper (2)
By using PRD, users can publish the report in a wide variety of output formats, includes: HTML, PDF, Excel, Plain Text, Rich Text and CSV. However, some common output format, such as: PowerPoint (PPT), Flash, Postscript and Open Office report types, does not available in PRD. Another reason that PRD is not excellent under this criteria is due to the “pixel positioning” of report design, which means that the report cannot adapt to different-sized displays.
4.5. Analytical Functionality Criteria Evaluation In order to perform analytical functionality in PRD, it requires the data source should arrived in an
appropriate format.
Table 4-5 contains the evaluation scores, observations and comments of the analytical functionality
criteria for PRD.
The evaluation of this product resulted in a final weighted score of 65 out of 90 for the analytical
functionality criteria.
Table 4-5 Analytical Functionality Criteria Evaluation
Analytical Functionality Criteria Evaluation Score Observations and Comments
Common Aggregations Exceptional PRD provide 27 types of mathematic
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
21
Analytical Functionality Criteria Evaluation Score Observations and Comments
(3) aggregations which fully covered the minimum requirement of clients. You can define the function to use to aggregate the attribute in attributes pane.
User-defined Aggregations Exceptional (3)
Similar like BIRT, users are allowed to define expressions based on filter conditions.
Slice & Dice Minimal (1)
Pentaho has a broad vision to provide a comprehensive solution for Data Integration and Business Analytics. This explains why Pentaho BI Suite was included a whole set of applications from reporting to data mining, in order to provide mutual support between each other. For this reason, the analytical functionality, such as: slice and dice, drill-down and roll-up, requires support from another application named Pentaho Analysis in BI Suite in order to perform. PRD needs to set up a Pentaho Analysis data source and add an MDX query to retrieve a data set from a multidimensional database[2]. This also means that when using only Pentaho Report Designer, it can be more difficult to create complex reports. That is why it only scored 1 out of 3 here.
Drill-down & Roll-up Minimal (1)
Sorting & Ranking Proper (2)
4.6. Usability Criteria Evaluation As mentioned above, PRD has a nice and user-friendly report wizard to get started with, which can be
very helpful when users are new to the product or when they want to initialize a layout of a report
quickly.
Table 4-6 contains the evaluation scores, observations and comments of the usability criteria for PRD.
The evaluation of this product resulted in a final weighted score of 22 out of 33 for the usability criteria.
Table 4-6 Usability Criteria Evaluation
Usability Criteria Evaluation Score Observations and Comments
User Efficiency Proper (2)
Pentaho Report Designer is easy and simple to learn how to create common reports. Its UI is concise and offers most commonly used functionality in a good layout. Nevertheless, charting function was cluttered with sophisticated attribute setting dialog box, which makes charting is more difficult. Also, there is no chart preview, i.e. users cannot see
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
22
Usability Criteria Evaluation Score Observations and Comments
the chart result until they run the report. In addition, the setup is very simple too, for example on setup in windows, download the zip distribution, extract PRD into any directory but recommended to "C:\Program Files\report-designer" directory, then start the application by executing (or double-clicking) either the "report-designer.bat" file or the launcher.jar file.
Product Documentation Proper (2)
Pentaho provide documentation for current and previous releases of all Pentaho products in two delivery formats: the Info Center web-based HTML and.zip archives that contain PDF files. The documentations and manuals are includes: user-guide, walk-through tutorials, tech tips, etc. However, all these materials are not categorised by product and not well organized, which means that it is takes time to dig out what users are looking for. In addition, the Javadoc HTML pages for the latest release of Pentaho Reporting are also available on web. One book named "Pentaho Reporting 3.5 for Java Developers" written by Will Gorman is also available online but it costs $16.99, which documented the reporting engine and most of the features of the report designer.
Support Proper (2)
Pentaho provides a community forum to answer all software related queries and provide a platform for users to exchange ideas. There are dozens of new posts are issued every day which indicates that the forum is quite active. Obviously, all the questions are answered by volunteers, so there is no guaranteed response time or level of service.
4.7. Installation and Implementation Criteria Evaluation
Table 4-7 contains the evaluation scores, observations and comments of the installation and
Implementation criteria for PRD.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
23
The evaluation of this product resulted in a final weighted score of 53 out of 66 for the installation and
Implementation criteria.
Table 4-7 Installation and Implementation Criteria Evaluation
Installation and Implementation Criteria
Evaluation Score Observations and Comments
Supports Multiple System Platforms
Exceptional (3)
PRD is able to run on Windows, Linux and iOS. Pre-requisites: Java 1.6 or higher Minimum hardware requirements [3]:
192MB of allocated heap-space to process reports
Additional memory and/or adjustments to the global configuration parameters of the reporting engine is required when reports with more than 400 pages or about 50 rows of data.
PRD requires a Java Runtime environment that is fully compatible to the Java Platform Specification 5.0 (JSR-176)
Deploying Report Onto an Application Server
Proper (2)
The report can be run by the Classic Engine or the Pentaho BI Server which is another application in Pentaho BI Suite [2]. The Pentaho BI Server is a Java EE web application that provides all services to manage reports in a Web-2.0 environment and also is free to use.
Support IDE Plug-in Not Available (0)
The Pentaho Report Designer is a stand-alone, desktop-installed client tool, and is not available as an Eclipse or NetBeans plug-in.
No Compilation Before Running Report
Exceptional (3)
Pentaho does not require that reports be compiled prior to running which makes publishing a report easier.
4.8. Miscellaneous Criteria Evaluation
Table 4-8 contains the evaluation scores, observations and comments of the miscellaneous criteria for
PRD.
The evaluation of this product resulted in a final weighted score of 13 out of 18 for the miscellaneous
criteria.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
24
Table 4-8 Miscellaneous Criteria Evaluation
Miscellaneous Criteria Evaluation Score Observations and Comments
Product Cost Exceptional (3)
PRD is an open source tool, however the enterprise edition with more advanced features, such as: analysis services, dashboard designer, interactive reporting, will approximate costs $11,000 per year for unlimited users.
Product Stability Proper (2)
Pentaho is performing well with none crushing during our test. Reporting bugs can be handled through discussing in the forum or reporting to JIRA system, which requires creating a JIRA login account but it is free [3]. One thing should mention here is, over the past a few years, Pentaho has shown the consistent improvement in its open source reporting tool.
4.9. Evaluation Summary PRD has much of the same functionality as the other report designers like BIRT. However, PRD does
have some strengths and weaknesses over the other report designers in the following ways:
Strengths:
PRD takes a great advantage on user experience and usability, especially provides the best build-
in report design wizard.
PRD reports are in XML format, which makes it possible to put it under revision control.
Pentaho provide a BI suite which contains several client applications for different tasks. If some
business are looking for a feature-rich BI solution, Pentaho could be a good choice.
Weakness:
The process of creating charts is one of the weak points in PRD, due to the lack of step-by-step
guidance and a user-friendly property editor.
PRD itself has limited functionality in terms of interactivity reporting. It requires extra support
from other applications in its BI Suite, such as: Pentaho Analysis and Pentaho Data Integration,
to provide data in a proper format in order to achieve analytic functionality.
In conclusion, based on our evaluation, PRD would be a good choice for creating a simple or a moderate
complexity report with less requirement on charting. Moreover, PRD integrates with other applications
of Pentaho BI Suite, in order to fulfil the requirement on data mining, data integration, OLAP analysis.
Final Score: 358
Satisfaction Percentage: 80%
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
25
5. Evaluation of BIRT
BIRT, which is short for Business Intelligence and Reporting Tools, is a top-level Eclipse project. BIRT was
first proposed and sponsored by Actuate Corporation in 2004 and has been receiving contributions from
IBM, Innovent Solutions and the community ever since [5]. It provides a platform to create data
visualizations and reports that can be embedded into rich client and web applications, especially those
based on Java and Java EE.
The version we used for evaluation is Version 4.4.2 (can be downloaded from
http://download.eclipse.org/birt/downloads), an open source community version licensed under the
Eclipse Public License. It can be either installed by All-in-One package, or using the plug-ins for the
existing Eclipse environment, or through the Update Manager [6].
Developers who are familiar with Eclipse IDE will be comfortable with BIRT Report Designer since it is
part of the IDE. Figure 5-1 displays the standard UI of BIRT Report Designer, which integrates seamlessly
with Eclipse's project management features. In the upper left corner report items in the Palette pane
can be added to report using drag and drop. Next to the Palette is Data Explorer that shows data sources
(connections), data sets (queries) and report parameters. It is worth mentioning that BIRT allows the
reuse of report objects, such as tables, data sources and styles. These reusable objects are stored in a
library file under the Library Explorer.
Figure 5-1 BIRT Report Designer UI
BIRT uses data source elements to connect to database and uses data sets to retrieve data. Common
data sources like JDBC data source can be created by specifying the JAR files, database URL, username
and password, as shown in Figure 5-2.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
26
Figure 5-2 Create a new Data source
To build a data set, users can simply open the data set editor and define SQL queries either by typing the
whole clause or using drag & drop functions to select column names from Available Items on the left
hand side. The Preview Results function allows users to verify the retrieved data set before proceeding
to report layout design. Figure 5-3 illustrates a common SQL query in the editor.
Figure 5-3 Define a new Data set
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
27
Once basic queries are defined, users will be able to add report items such as tables, grid, chart, images,
etc. Figure 5-4 displays a sample report that shows a bar chart with the top N customers by revenue,
followed by a listing of the top M% customers by revenue sorted by customer name. To verify if the
report is presenting the data as expected, users can preview the report by clicking on the View Report
button in the navigation bar. This will prompt the web browser to open the report, displayed in HTML
format. For example, Figure 5-4 is the preview of report designed in Figure 5-1.
Figure 5-4 Preview report in web browser
5.1. Report Designer Components Criteria Evaluation
BIRT consists of several components [7]:
BIRT Report Designer: It is an Eclipse perspective used to create BIRT report designs stored in an
open XML format.
Design Engine: It is responsible for producing concrete report designs and it allow modifications
to be made on these designs. The Report Designer uses Design Engine internally to construct the
XML designs.
Report Engine: The BIRT Web View uses Report Engine to generate and render reports. It is
basically a series of Java class files that can be deployed in any Java/J2EE environment. When it
is called in the context of an application, a report design is passed to it. Then it will access the
data source, retrieve the data, do the required operations such as aggregation, sorting, etc, and
then format the report for presentation as HTM or PDF or any supported output format.
Charting Engine: It is used to design charts that either can be integrated into BIRT designer or
into an application. The Design and Report Engines make use of this API to deliver Charts.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
28
BIRT Viewer: Although BIRT lacks an open source server component, it uses BIRT viewer on a
Java Application server preview reports within Eclipse.
Figure 5-5 [7] illustrates an architecture overview of BIRT. Basically it has two main components: a visual
report designer for creating BIRT Designs, and a runtime component for generating those designs that
can be deployed to any Java environment. BIRT also includes a charting engine that is both fully
integrated into the BIRT designer and can be used standalone to integrate charts into an application.
Figure 5-5 Architecture Overview of BIRT
Table 5-1 contains the evaluation scores, observations and comments of the report designer
components criteria for BIRT.
The evaluation of this product resulted in a final weighted score of 40 out of 45 for the report designer
components criteria.
Table 5-1 Report Designer Components Criteria Evaluation
Report Designer Components Criteria
Evaluation Score Observations and Comments
Common Report Designer Components
Exceptional (3)
It includes all common report designer components, making it easy to design and generating common reports. Testing reports such as product catalogue, revenue by product line, top n customers, crosstab report can be successfully created and generated in the web browser.
Advanced Components Proper (2)
We tested some advanced report functionality and BIRT proved to be capable of carrying out most of these jobs. The tool supports Barcodes/sub-reports/side-by-side/Cross-tabs functions, but not includes Survey
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
29
Report Designer Components Criteria
Evaluation Score Observations and Comments
Scale/Horizontal Panning/Multi-column layout
5.2. Data Source Criteria Evaluation For our business context, clients can have a wide variety of databases, therefore we would like the
product to support as many types of data sources as possible. The community version supports multiple
data sources, including JDBC, scripted, flat file, XML, POJO, EJB. It also provides Joint Data Set function
that combines two data sets from different sources.
Table 5-2 contains the evaluation scores, observations and comments of the data source criteria for BIRT.
The evaluation of this product resulted in a final weighted score of 55 out of 60 for the data source
criteria.
Table 5-2 Data Source Criteria Evaluation
Data Source Criteria Evaluation Score Observations and Comments
Support Multiple Data Sources Exceptional (3)
Apart from JDBC connection, BIRT also supports EJB, XML, scripted and flat file data sources, which covers the majority databases used by the clients of Ivy Technologies. It should be noted that BIRT requires users to locate and install JDBC driver. In our testing we noticed that connecting to MS SQL Server Database is not very straightforward.
Support for Joining Multiple Data Sources in the Designer
Exceptional (3)
The tool supports both heterogeneous and homogeneous data set joins. This feature is very similar to SQL table joins, except that each joint data set can only involve two tables. But it is still possible to produce multiple dataset joins by chaining joint data sets, since a joint data set may be used to define another joint data set.
Support Non-JDBC Data Sources Proper (2)
The tool supports non-JDBC data sources, including XML, Web Service, Flat Files, EJB, Scripted. However it needs Source Forge Java Excel API to use excel. Currently the community version does not support JSON, Hadoop Hive.(some clients may use them as data source)
5.3. Charting Criteria Evaluation
BIRT supports a rich type of charts, such as bar, tube, pie, that comes in both 2D and 3D effects. Charts
can be easily created using the Chart Wizard, which takes users through the process of creating a report
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
30
step-by-step. BIRT provides some advanced charting abilities with a few lines of custom code. Examples
include allowing the chart to further filter the data, use report parameters in the context of creating the
chart, and specify all kinds of interactivity for chart elements like data points, data series, and axes.
BIRT uses JavaScript for expressions, business logic, and integration with application-specific Java classes.
Chart interaction can be created by defining triggers (event-action pair) and add scripts. When the user
does a mouse click, changes focus, presses a key, and many other operations, it can trigger an action.
Actions can include going to a hyperlink, running a script, displaying a tooltip, highlighting the chart
element, or toggling the chart elements’ visibility. Figure 5-6 shows an example of defining an action
that invokes user-defined script when there's mouse clicking the chart.
Figure 5-6 Interactive chart by adding script
Table 5-3 contains the evaluation scores, observations and comments of the charting criteria for BIRT.
The evaluation of this product resulted in a final weighted score of 66 out of 78 for the charting criteria.
Table 5-3 Charting Criteria Evaluation
Charting Criteria Evaluation Score Observations and Comments
Support for Common Chart Types
Exceptional (3)
Common charts are available in BIRT and can be created easily using Chart Wizard.
Chart Interactivity Exceptional (3)
The tool supports the following events: mouse-over tool tips drill-through
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
31
Charting Criteria Evaluation Score Observations and Comments
hyperlinks hide/show series
Advanced Charts Minimal (1)
The community does support dynamic parameter selection and interactive charting, however, advanced charts like flash charts/gadgets/maps are not provided. These advanced functions are accessible through Actuate BIRT Designer Professional, which will cost approximately $400 per developer [8].
Variety of Chart Theme Minimal (1)
Same as above. The community version only provides limited selections of chart themes. Commercial version like BIRT Designer Professional provides developers with support for Flash Objects and produce rich, highly interactive and navigable reports with formatting and graphics.
Support for Scripting Exceptional (3)
Despite the limitations listed above, users will still able to leverage the power of JavaScript to add custom logic to the report. BIRT provides a rich scripting model based on the Mozilla Rhino engine. Script events are defined for Report Object, Report Elements and Data Sets. Users can create event handlers to override these script events, thus altering the report content. Selecting the Palette view while in the Script editor will display functions and variables that are available in the given event for the selected report element.
5.4. Parameterization and Output Format Criteria Evaluation BIRT provides an API for working with parameters and a default UI that prompts the user to enter
parameter values. The values will be available within the report where they can be displayed, passed to
a query, used in expressions, etc. Figure 5-7 shows a cascading parameter group allows a group of
parameters to be interlinked, where selecting a value for the first parameter affects the choices
available in the subsequent parameters.
BIRT stores a report design in XML format and has a ".rptdesign" suffix. The XML design file is created by
the BIRT designer and directly executed by the BIRT engine. Currently BIRT supports HTML, Paginated
HTML, PDF, WORD, XLS, PostScript, etc.
Table 5-4 contains the evaluation scores, observations and comments of the parameterization &output
format criteria for BIRT.
The evaluation of this product resulted in a final weighted score of 55 out of 60 for the parameterization
&output format criteria.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
32
Figure 5-7 Cascading parameter group in BIRT
Table 5-4 Parameterization and Output Format Criteria Evaluation
Parameters & Output Format Criteria
Evaluation Score Observations and Comments
Static Parameters Selection Exceptional (3)
BIRT allows parameter values to be selected from a hard-coded list of values. Using parameters to filter the data also applies to the charts, and specify all kinds of interactivity for chart elements.
Dynamic Parameters Selection Exceptional (3)
BIRT allows parameters to be selected from a list of values that came from a database. An example is to build a master report and a detail report. The master report lists orders and allows the report user to click on a particular order number to drill into a detail report for the selected order. The hyperlink property within the master report is generated dynamically based on results retrieved from a query.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
33
Parameters & Output Format Criteria
Evaluation Score Observations and Comments
Multiple Output Formats Proper (2)
No pixel positioning: Although reports can be rendered in versatile formats, sometimes developers cannot precisely control the placement of each element in the report [12]. For reports that need to be printed with precise placement of each report element on the page, a report designer with a “pixel positioning” approach to report layout is required.
5.5. Analytical Functionality Criteria Evaluation
As long as data set returns data in tabular format, BIRT can perform sorting, aggregation, grouping
operations.
Crosstab functionality within BIRT can include data cube from defined queries. By editing the Crosstab’s
properties, users will be able to add subtotals and grand totals, specify sorting, format the individual
elements of the crosstab. The data cube element within BIRT provides some basic analytical
functionality which allows the developer to build cubes based on existing data sets. These cubes are
constructed using dimensions and measures as illustrated in Figure 5-8.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
34
Figure 5-8 Define Data Cube in BIRT
Table 5-5 contains the evaluation scores, observations and comments of the analytical functionality
criteria for BIRT.
The evaluation of this product resulted in a final weighted score of 60 out of 90 for the analytical
functionality criteria.
Table 5-5 Analytical Functionality Criteria Evaluation
Analytical Functionality Criteria Evaluation Score Observations and Comments
Common Aggregations Exceptional (3)
BIRT supports a wide variety of totals: sum, running sum, average, minimum, maximum, count, count distinct, and many more. Common aggregation operation comes in an easy way. User can drag and drop an aggregation element from Quick Tools and specify the function and expression needed for a specified aggregation.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
35
Analytical Functionality Criteria Evaluation Score Observations and Comments
User-defined Aggregations Exceptional (3)
Apart from common aggregations, users are allowed to define expressions based on filter conditions. This can be done in the Expression Builder using JavaScript.
Slice & Dice Not available (0)
Despite the fact that it is possible to perform data mining by adding Data Cubes, the analytical functionality is rather limited. Currently the community version do not support advanced BI functions such as slice and dice, ad hoc query, forecasting. These extensions are available in the commercial version, Actuate OLAP tool BIRT Analytics, which is able to perform advanced and predictive analytics.
Drill-down & Roll-up Minimal (1)
Sub-reports, master & detail report, and crosstab report functionalities enable users to conduct drill-down on reports. However the function is still limited in that user cannot define dimensions or choose measures on-the-fly. Again these extensions are available in Actuate OLAP tool BIRT Analytics, the commercial version.
Sorting & Ranking Proper (2)
BIRT provides mapping and sorting features, which can be used in many places combined with each other, and turn out to be very flexible. Still these features have their limitation in the community version, since they are achieved through pre-defined parameters instead of on-the-fly.
5.6. Usability Criteria Evaluation
We give BIRT high marks on usability as users can easily design common reports through UI and Wizards.
We created a series of sample reports following the BIRT tutorial and came to have the general
impression that the tool is fairly simple to start with. We can usually figure out what to do without
resorting to the manuals. Upon mastering a few basic BIRT concepts, such as Datasets, Table Binding,
Expressions, most developers will be able to build complex reports.
Table 5-6 contains the evaluation scores, observations and comments of the usability criteria for BIRT.
The evaluation of this product resulted in a final weighted score of 27 out of 33 for the usability criteria.
Table 5-6 Usability Criteria Evaluation
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
36
Usability Criteria Evaluation Score Observations and Comments
User Efficiency Exceptional (3)
The tool runs in Eclipse environment. During our testing, the set up of this environment takes 20-30 seconds on average. Since developers at Ivy Technologies are very familiar with Eclipse IDE, the tool should be fairly easy to use. The preview of the report is very convenient, with just a click and the user will instantly see the render effect in web browser.
Product Documentation Proper (2)
Users can always resort to BIRT documentation: http://eclipse.org/birt/documentation/ BIRT documentation is consisted of a great deal of repository of samples, tutorials and FAQs. The documentation is complete, well organized and effective. It available for free as part of the product package. However we do noticed that some documentation only applies to older version. We encountered some issues when trying to applying these documentation to our designs.
Support Proper (2)
BIRT has abundant external resources for developers to find guides, share ideas, as well as download examples. Also there is a forum area to search and ask questions on the BIRT Developer Center. As an open source project, however, the support is not 7/24.
5.7. Installation and Implementation Criteria Evaluation
BIRT can be downloaded and installed through BIRT homepage, with the All-in-One package being the
fastest way to get started. As mentioned in previous section, BIRT does not have an open source server
component, instead it uses report viewer for previews and deployment.
Table 5-7 contains the evaluation scores, observations and comments of the installation and
Implementation criteria for BIRT.
The evaluation of this product resulted in a final weighted score of 56 out of 66 for the installation and
Implementation criteria.
Table 5-7 Installation and Implementation Criteria Evaluation
Installation and Implementation Criteria
Evaluation Score Observations and Comments
Supports Multiple System Platforms
Exceptional (3)
BIRT work on any platform on which Eclipse itself works. However, the product is built on
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
37
Installation and Implementation Criteria
Evaluation Score Observations and Comments
Windows, which is the best-tested platform. The package includes everything needed to start creating BIRT designs. Pre-requisites: Java 1.6 JDK/JRE. Minimum hardware requirements [10]:
1GHz processor or faster;
512 MB RAM (1 GB recommended);
352 MB of disk space is required for product package and additional disk space is required for designing reports.
Deploying Report Onto an Application Server
Proper (2)
BIRT Viewer can be deployed onto the following application server: Tomcat, Jboss, WebSphere, WebLogic, Oracle Application Server, Sun's Application Server, etc [11]. In addition, starting from release 3.7, BIRT offers POJO Runtime to deploy POJO Viewer to different application servers.
Support IDE Plug-in Exceptional (3)
The BIRT designer is a set of Eclipse plug-ins that lets users to build reports as a perspective from within Eclipse.
No Compilation Before Running Report
Exceptional (3)
The BIRT Report Engine directly executes the report design XML file. This is one reason that BIRT uses interpreted JavaScript instead of compiled Java as its scripting language.
5.8. Miscellaneous Criteria Evaluation
BIRT stores report design in a "design file." This file is in XML format and has a ".rptdesign" suffix. Like
Pentaho, since reports are in XML format, they can be effectively put under revision control.
Table 5-8 contains the evaluation scores, observations and comments of the miscellaneous criteria for
BIRT.
The evaluation of this product resulted in a final weighted score of 13 out of 18 for the miscellaneous
criteria.
Table 5-8 Miscellaneous Criteria Evaluation
Miscellaneous Criteria Evaluation Score Observations and Comments
Product Cost Exceptional Community Version is free, however if switching
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
38
Miscellaneous Criteria Evaluation Score Observations and Comments
(3) to commercial version to get access to advanced functionalities, the cost is approximately $400 per developer.
Product Stability Proper (2)
BIRT uses Bugzilla to track bugs and enhancements. Users can find BIRT-related bugs using the Bugzilla search facility. We noticed there are some reported stability issue from Eclipse Community Forums. The tool crashed once in our testing. With that said, overall the stability performance is satisfactory. It should also be noted that since 2012 BIRT has shown signs of slowdown in terms of adding new features [4]. No major changes or improvements has been made in recent two years.
5.9. Evaluation Summary
BIRT is an Eclipse plug-in that allows users to design report templates and publish them in web-ready
formats. As a BI tool, BIRT provides most of the functionalities for creating and deploying reports to
enterprise applications. It satisfies our need to deliver enterprise portal applications that offer reporting
framework, and partially fulfills analytical functionality. Here is a summary of advantage and drawbacks
of this tool:
Pros
1. BIRT Report Designer is a robust, powerful tool for creating reports. It exhibits outstanding
charting capability.
2. Since Ivy Technologies does all its web application developments in Eclipse IDE, BIRT is fairly
easy to use (although it is occasionally too complex).
3. It supports reuse of reporting components: users can create libraries of shared resources that
can be dynamically used by multiple reports.
4. It can create complex reports with minimal custom code.
5. It presents flexible and extensive interactive charting function.
Cons
1. BIRT primarily focuses on reporting instead of analytics and therefore it is not appropriate to
work with OLAP data. To address this issue, Ivy Technologies may need to consider purchasing
the commercial version if clients require fully functional analytical capability.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
39
2. It lacks a true report server, instead it uses the Viewer on a Java application server [12]. For
many applications, a report server provides functionality such as user management, report
management, security, notifications, and more.
Final Score: 372
Satisfaction Percentage: 83%
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
40
6. Conclusions and Recommendations Table 6-1 Evaluation Scores Comparison
Criteria Weighted Evaluation Score
Pentaho BIRT
Report Designer Components 40 out of 45 40 out of 45
Data Source 55 out of 60 55 out of 60
Charting 55 out of 78 66 out of 78
Parameterization & Output Format 55 out of 60 55 out of 60
Analytical Functionality 65 out of 90 60 out of 90
Usability 22 out of 33 27 out of 33
Installation &Implementation 53 out of 66 56 out of 66
Miscellaneous 13 out of 18 13 out of 18
Total 358 out of 450 372 out of 450
The final results show that Pentaho is at par with BIRT, with BIRT scores a bit higher than Pentaho by 6%.
Neither of the tools scores zero on any crucial criteria. Of all the criteria, Pentaho scores higher on
analytical functionality while BIRT has better performance on charting, usability, installation and
deployment.
To summarize, Pentaho and BIRT prove to be excellent candidates and both satisfy our requirements.
Pentaho has stronger analytical capability and BIRT is easier to use, providing more interactivity. As open
source tools, however, some advanced features like ad-hoc queries, data exploration, interactive
dashboards are not available.
For our business context, we recommend BIRT because it is more user-friendly and it provides deepest
access to reporting functionality, with better chart implementation and rich extensions. Since our
evaluation does not include scalability testing, further investigation needs to be conducted if the client
adds new requirements such as the ability to handle large heterogeneous databases.
CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation
41
Reference [1] “ISO/IEC 9126,” Wikipedia, the free encyclopedia. 09-Mar-2015. [2] Design Print-Quality Reports with Report Designer, Pentaho Report Designer User Guide, Version 5.2,
30, September 2014, https://help.pentaho.com/Documentation/5.2/Version_5.2 [3] Pentaho Report Designer Version 5.2 Readme, 26, June 2014, https://github.com/pentaho/pentaho-
reporting/blob/master/README.md
[4] Open Source Reporting Review - BIRT, Jaspersoft, Pentaho, Innovent Solutions, February 2015, http://www.innoventsolutions.com/open-source-reporting-review-birt-jasper-pentaho.html
[5] About BIRT: http://eclipse.org/birt/about/ [6] Getting started with BIRT: http://eclipse.org/birt/getting-started/ [7] Architecture Overview of BIRT: http://eclipse.org/birt/about/architecture.php [8] About Actuate BIRT Report Designer Professional: http://www.actuate.com/be/download/Actuate-
BIRT-Report-Designer-Professional.pdf [9] Open Source Business Intelligence: http://www.innoventsolutions.com/ [10] BIRT FAQs: https://wiki.eclipse.org/BIRT/FAQ [11] BIRT Documentation: http://eclipse.org/birt/documentation
[12] Showdown of BI Tools: Pentaho vs Others: http://www.edureka.co/blog/pentaho-vs-jaspersoft-vs-birt/
Top Related