The Analytics Landscape de Jasper Soft

The Analytics Landscape

An overview of analytic technologies and how Jaspersoft‘s products and ecosystem address those areas.

Jaspersoft Headquarters:

539 Bryant Street, Suite 100

San Francisco, CA 94107, USA

phone: +1 888.399.2199 or +1 415.348.2380

Email: [email protected]

Jaspersoft Europe, Middle-East, Africa:

4 St. Catherines Lane

Dublin 8, Republic of Ireland

Phone: +353 1 443 4700


Jaspersoft Asia Pacific, Japan, Australia/NZ:

36th Flr., CRC Tower, All Seasons Place

87/2 Wireless Rd., Lumpini, Phatumwan

Bangkok 10330 Thailand

Phone: +66 2 625-3165


mailto:[email protected]


© 2010 Jaspersoft

Contents

WHAT DOES “ANALYTICS” COVER? ........................................................................................ 3

ASSOCIATED TOPICS OUT OF SCOPE FOR THIS PAPER ............................................................. 4

PRESENTATION LAYER ........................................................................................................... 5

REPORTING .................................................................................................................................. 5

ANALYSIS ..................................................................................................................................... 9

DASHBOARDS, MASH-UPS, AND PORTALS ............................................................................... 14

DATA STORE ........................................................................................................................ 16

ONLINE TRANSACTION PROCESSING (OLTP) .......................................................................... 16

RDBMS WITH ANALYTIC EXTENSIONS ..................................................................................... 17

ANALYTIC DATABASES (ARDBMS) .......................................................................................... 17

NOSQL SYSTEMS ..................................................................................................................... 20

MIDDLEWARE ...................................................................................................................... 22

EXTRACT TRANSFORM LOAD (ETL) .......................................................................................... 22

ENTERPRISE INFORMATION INTEGRATION (EII) ........................................................................ 23

COMPLEX EVENT PROCESSING (CEP) ..................................................................................... 23

SEARCH / UNSTRUCTURED CONTENT ....................................................................................... 24

MAPREDUCE / HADOOP ............................................................................................................. 25

ANALYTIC APPLICATIONS ..................................................................................................... 27

BUSINESS PERFORMANCE MANAGEMENT (BPM) .................................................................... 27

OTHER ANALYTIC APPLICATIONS .............................................................................................. 28

CONCLUSION ....................................................................................................................... 29

COMMENTS? ....................................................................................................................... 30

© 2010 Jaspersoft

What Does “Analytics” Cover?

The term ―analytics‖ has evolved over the past decades into a vibrant ecosystem encompassing a wide range of technologies and products. This ecosystem spans fields traditionally associated with the word, such as statistical and predictive analytics, to ―business intelligence‖ (traditionally including at least reporting and analysis), middleware technologies designed for analytics, as well as specially designed databases for analytic workloads. The Data Warehouse Institute (TDWI) director of research Wayne Eckerson cites Webster‘s dictionary‘s definition of analytics: ―studying the nature of something or determining its essential features and their relations‖ adding that a practical view is to consider the problem at hand.

“Few veteran BI professionals equate “reporting” or “monitoring” with analytics, but it is not

entirely inaccurate to do so. Business people can use reports to understand the business, analyze

root causes, and guide future activities (and dashboards are really just graphical, multilayered

exception reports).

The question is not whether reports and dashboards are analysis tools; it’s whether these tools

and methods are best suited to the types of people doing the analysis. For certain types of users,

such as executives and managers, reports and dashboards are optimal analytical tools; but for

others, such as data-savvy analysts, we need to move beyond reporting (and spreadsheets and

desktop databases) to analytical and predictive technologies with richer functionality.” Wayne Eckerson, TDWI Research Beyond Reporting, Delivering Insights with Next-Generation Analytics, TDWI Best Practice Reports, 2009. http://www.tdwi.org/research/display.aspx?ID=9622

This paper is meant to be a broad overview for people of a technical profile, typically architects, who will actually implement analytics solutions. Therefore, it is organized around three major ways to think about how to deal with data, i.e., how data is viewed, delivered, and stored, and thus the first three corresponding categories presented are: presentation layer, middleware, and data stores. A fourth category, analytic applications, takes on a solution-oriented view. The paper explores the various categories of analytics with respect to specific types of users and their corresponding use cases. Suggestions are presented around considerations that customers should make when selecting a solution for a given analytics problem, as well as how Jaspersoft products and its ecosystem address each area.

http://www.tdwi.org/research/display.aspx?ID=9622


© 2010 Jaspersoft

Associated Topics Out of Scope for This Paper

This paper, although broad in scope, is not meant to be completely comprehensive. For example, it does not cover more detailed areas also popularly associated with, or tangentially related to, analytics including various data management topics (e.g., data quality and master data management), operational business intelligence, collaboration, and the still emerging fields of semantic technologies (semantic web oriented, or otherwise, including ontology-based, Natural Language Processing, etc.). Additionally, manifestations of how analytic solutions are acquired or deployed are not included, i.e., open source versus proprietary, on-premise, virtualized appliance, cloud, and software-as-as-service (SaaS).


© 2010 Jaspersoft

Presentation Layer

When presenting data to a user, whether an average business user or an advanced business analyst, the challenge is to present the data in the most compelling, simple – yet powerful, manner for the use case. Typically, this means delivering visually compelling charts, intuitive parameters, easy ways to interact with or analyze the data, and mechanisms to output the data for easy consumption (e.g., output to Microsoft Excel file, print-ready, send to a mobile phone, etc.). It‘s not to say that the Presentation Layer is only presenting data; there is nearly always a back-end component that processes or manipulates data. There are several main ways to consider the presentation layer: reporting, analysis, advanced analytics, and dashboards.

Reporting

Reporting is typically thought of as either static or ad hoc in nature. On one hand, static reporting means that the lay-out as well as the queries that define the data within the report are static. Ad hoc, on the other hand, allows a business user to create new reports from scratch ―on-the-fly.‖ In addition, the reports may be deployed in either an operational reporting or production reporting manner.

Gartner‘s February 2009 report entitled Critical Capabilities for Business Intelligence Reporting (by James Richardson, Kurt Schlegel, and Rita L. Sallam) delivers an excellent overview of essential aspects of reporting. Specifically, they call out the following areas:

Ease of use for end users — functionality promoting a good experience for users when they consume reports (that is, parameterization, navigation, search and so on).

Ease of use for report developers — functionality promoting a good experience for developers when they design reports, including complex specifications like highly formatted asymmetric layouts.

Data volume scalability — the proven ability to report on large data sets, expressed as either terabytes of data or millions of input rows.

User volume scalability — the proven ability to serve the needs of thousands of report consumers.


© 2010 Jaspersoft

Complex, interactive reporting — support for cross-tabs, data drilling and prompting and filtering, all of which are key to end-user adoption and satisfaction, and in reducing the report development load on IT.

Sophisticated Structured Query Language (SQL) support — being able to take advantage of the power of SQL and push processing down to the relational database management system (RDBMS), with optimized calls that aid reporting development and performance.

High-quality, portable output — creating pixel-perfect layouts, using vector graphics and the ability to create dynamic output (for example, Adobe Flash) mean that more use cases can be supported, including customer-facing reports and disconnected usage.

Integration — connectedness is critical to success with BI, so a range of native data adaptors for a broad range of sources (including online analytical processing [OLAP] databases) is important, as is the provision of a single integrative semantic layer to enable reuse within the BI platform.

Financial reporting — the often overlooked ability to correctly format, markup and present financial information automatically. This specific capability is included as it is the most common use case requiring special report formatting (rather than, say, geographic or scientific report formatting).

Mobile device support — the capability to render reports on mobile devices in order to support remote employees, ideally beyond that supplied by simply surfacing a report in a mobile Web browser.

Report Creation

Static Reporting (pre-defined reports)

Static reports are typically created by developers, and they are static in the sense of their design. Therefore, it does not mean that the report‘s data is necessarily static. In addition, the reports could certainly include pre-defined customizable parameters that the user manipulates, or else the reports could also be scheduled to be refreshed on a periodic basis.


© 2010 Jaspersoft

Audience

Jaspersoft’s Solution

Nearly every person in an organization from average business user to analyst to executive.

Jaspersoft provides several open source and commercial products to manage static reporting:

JasperReports (LGPL license) reporting library for developers enables print-ready, or ‗pixel-perfect‘ reporting and output to a variety of formats (HTML, ODF, PDF, DOC, XLS, etc.).

iReport (GPL v3 license) is a graphical desktop report designer built on JasperReports. With iReport, developers and power users can design extremely sophisticated reports and dashboards.

JasperServer Community Edition (AGPL v3) license is a report server that lets you manage reports in a secured repository, scheduled reports, and display dashboards of reports. Business users and executives can access reports, manipulate report parameters, and schedule reports.

Commercial licenses of the products listed above are also available.

Ad Hoc Reporting

Ad hoc reporting allows a business user to create new reports from scratch. To enable this, a technical person constructs pre-built queries and/or a semantic layer to shield the business user from the data structure jargon and complexity. One of the more important aspects of ad hoc reporting is to provide the ability to easily define the visualization of the data.

Audience


Nearly every person in an organization from average business user to analyst to executive.

iReport delivers very powerful and flexible ad hoc reporting to developers and power users. JasperServer Professional Edition provides all the features of JasperServer Community Edition (see above) and extends those to deliver ad hoc reporting for business users:

- Topics are pre-built queries created by a technical person. These topics may include simple objects or complex joins. Business users can then easily select


© 2010 Jaspersoft

and combine topics to create their desired report.

- Domains provide an abstraction layer of a physical data model so that business users are shielded from the technical jargon to create reports on the fly. Domains are created beforehand by a more technical person that understands SQL, and it is a more comprehensive approach than Topics in exposing the underlying data structure for a wider range of queries.

The ad hoc layout of data can be easily defined as tabular/lists, charts, and cross tabs, and provides powerful data selection capabilities.

Report Deployment

Analytics services for an organization are often centralized to provide consistent services and information access, including:

Scheduled report distribution: used for ―production reporting‖ but also required

periodic reports for an individual or group

Secured access to data: data level security based on a user‘s profile, including

their ―role‖ in the organization

Centralized point of control for information administrators and developers to

distribute to information consumers

Gartner‘s James Richardson defines two main ways to deploy reports in the following manner:

• Production reporting: the ‗production‘ in the term relates to the activity of

producing high-volumes (often tens of thousands) of pages of reports (e.g., bank statements) automatically and then distributing them to interested parties (often via a mechanism called ‗bursting‘).

• Operational reporting: the activity of reporting detailed (e.g., production

process) data often in real (or near real) time from operational or transactional systems (e.g., ERP). Organizations that are serious about this may use an Operational Data Store (ODS) trickle fed from live systems (as opposed to the batch loaded data warehouses used for most BI).


© 2010 Jaspersoft

Analysis Whereas reporting is about outputting data to answer a pre-defined question, analysis is about interacting with data and discovering trends in data. In the 1990's 'OLAP' was considered nearly synonymous with 'analysis.' One of the leading authorities on analysis is Nigel Pendse, of Business Application Research Center (BARC) which publishes the BI Verdict (www.bi-verdict.com), formerly known as The OLAP Report. His report on ―What is OLAP?‖ available from BARC‘s website clearly lays out the key features of OLAP, as well as discusses the much debated ‗12 Rules of OLAP‘ which appeared in the vendor-sponsored paper, Providing OLAP (On-line Analytical Processing) to User-Analysts: An IT Mandate, credited to E.F. Codd & Associates. Those resources are recommended reading for further background on OLAP and analysis. One of the key points of Pendse‘s article is the FASMI test, paraphrased and abbreviated to the following, which describes what OLAP is meant to achieve:

Fast: means that the system is targeted to deliver most responses to users within

about five seconds

Analysis: means that the system can cope with any business logic and statistical

analysis that is relevant for the application and the user, and keep it easy enough

for the target user.

Shared: means that the system implements all the security requirements for

confidentiality (possibly down to cell level) and, if multiple write access is needed,

concurrent update locking at an appropriate level.

Multi-dimensional: must provide a multidimensional conceptual view of the

data, including full support for hierarchies and multiple hierarchies

Information: all of the data and derived information needed, wherever it is and

however much is relevant for the application

Relational On-line Analytical Processing (ROLAP)

With OLAP, business users can ―slice-and-dice‖ data, i.e., view data with respect to different dimensions (shared or private) such as sales, geography, and time. They can change the axis of analysis, ―drill-down‖ into a specific dimension to get greater detail and discover the trends in historical data. In order to achieve OLAP in a relational database (ROLAP), a technical person maps the OLAP schema to an underlying physical data structure such as an OLTP database but more commonly a data warehouse or data mart‘s star schema or snowflake schema. With star and snowflake schema‘s, the data model is organized with respect to business user queries rather than transactions. The benefits of ROLAP include the ability to handle large volumes of data


© 2010 Jaspersoft

and take advantage of the underlying database system‘s SQL capabilities as well as the hardware resources for processing queries.

Audience


Executive management as well as business analysts working with very large sets of data, typically in data warehouses that are commonly fed by a vast range of data sources that require much data transformation.

JasperAnalysis is an extension of JasperServer and provides powerful ROLAP capabilities, on top of your choice of OLTP or analytic database, via a web browser. JasperAnalysis uses an MDX (multi-dimensional expression) query language to manipulate the data for analytic purposes. MDX is more expressive than SQL but decomposes into SQL at run-time. JasperAnalysis is available in two editions:

JasperAnalsysis Community Edition (AGPL v3 license) provides slice-and-dice of data via a web browser as well as the OLAP Workbench to map the OLAP schema to the underlying relational structure. Specific capabilities include:

Powerful and easy to use o Intuitive Web-based online analytical processing o Web management console for administrators o Metadata-based ―virtual cubes‖ map onto

RDBMS

Stand-alone and embeddable o Jaspersoft BI platform integration APIs o ―Drill-out‖ to reports & other applications o ―Drill-to‖ from reports and other applications

Modern and mature o Based on battle-tested independent open

source technology o Leverages OLAP standard: Multi-Dimensional

Expressions (MDX)

JasperAnalysis Professional Edition includes the features in the Community Edition and adds:

o Row and column (cell-level) data security o Jaspersoft ODBO Connect: Slice-and-dice data via

a specialized driver for Microsoft Excel, using XML for Analysis (XML/A), a standard analogous to SQL with JDBC.


© 2010 Jaspersoft

In-Memory Analysis

The benefits of an in-memory analysis solution are clear: speed, simplicity, and insight. Because data is pulled into memory the analysis operations are extremely fast, allowing manipulation and visualization of data in sub-second response times. In-memory analysis enables analysis of live operational data, without data getting stale as a result of batch data integration processes. In addition, the need for a dedicated and separate data integration tool (such as ETL – Extract, Load, and Transform) is lessened and in many cases removed, for example when doing analysis against one data source. The in-memory approach is often cited in conjunction with ‗visualization‘ due to its ability to allow users to interact and visualize the trends in data. JasperServer offers in-memory analysis techniques that Jaspersoft refers to as "fully-integrated" in to the reporting and dashboard server environment. In this sense, JasperServer's in-memory analysis capabilities share underlying metadata as well as ad hoc reporting and dashboard-building tools that are common within the Server architecture. So, a simple and powerful analytic environment is created. The BI Scorecard's Cindi Howson offers some crisp background on conventional OLAP tools versus in-memory products:

As the name suggests, the key difference between conventional BI tools and in-memory products is that

the former query data on disk while the latter query data in random access memory (RAM). When a

user runs a query against a typical data warehouse,for example, the query normally goes to a

database that reads the information from multiple tables stored on a server's hard disk. With in-

memory tools, all information is first loaded into memory. If the in-memory tool is server-based, an

administrator may initiate the load process; if it's a desktop analysis tool, the user may initiate the

process on his or her workstation. Users then query and interact with data loaded into the machine's

memory. Accessing data in memory is literally millions of times faster than accessing data from disk.

This is the real, "speed-of-thought" advantage that lives up to all the hyperbole.” - Cindi Howson, The BI Scorecard ―Take Advantage of In-Memory Analytics,‖ published in Intelligent Enterprise.

Furthermore, Nigel Pendse offers an interesting overview of the history and benefits of in-memory analysis on the BARC website in his commentary, What in-memory BI ‘revolution’? In this report, he also describes briefly that a hybrid approaches of ROLAP and in-memory are often an attractive option:

“Products don’t have to stick rigidly with one architecture or the other. There is nothing to stop

designers of disk-based products from also including an in-memory option, or simply using large

caches to optimize disk performance (just as disk controllers do).” - Nigel Pendse, Business Application Research Center (BARC)


© 2010 Jaspersoft

Audience


Executive management as well as business analysts working with moderate size data sets that are typically fed by one or more data sources with little or simple data transformation requirements.

JasperServer Professional Edition provides a hybrid approach to analysis called ―integrated analysis.‖ Business users can perform analysis with the same graphical front-end of JasperServer and the back-end is simply a data source choice which can be performed either in-memory or via a traditional RDBMS. In-memory capabilities provide:

Easy analysis for business users o Drag & drop dimension / measures o Pivot o Expand / collapse (―drill‖) o Slice / Dynamic Filter o Sort o Drill to detail o Partial Pivot o Undo / Redo

Builds on Web 2.0 Ad Hoc Query & Domains infrastructure

o Built on Jaspersoft ―Domains‖ metadata

No ―OLAP cube‖ required No specialized MDX query

language o Web 2.0 AJAX engine o In-memory caching engine OR push-

down to RDBMS o Architected for data source

independence

Browser based solution

Save as a report for later distribution

One may compare and contrast reporting and analysis use cases and benefits in the following way.


© 2010 Jaspersoft

Description Reporting Analysis What is it? Detailed, graphical, and textual

presentation of data, typically either printed or viewed in a dashboard

Data exploration and interaction. Summarized, statistical, drill into detail

Main use Generally used to answer, ―what is there?‖ or ―what has happened?‖ Show history and current status

Answering a question, define new metrics about history

How to use it? Used regularly or ―pushed‖ via scheduling. Little or no change of view over time.

On demand, experimental. Could be a starting point for reporting.

What are the basic requirements behind using it?

A person tends to need a limited set of reports

Once off or useful for a short time. Data sets are re-used

Does it deliver live or data?

Fixed or limited data, visualization and navigation: relatively static

Dynamic, selectable data, visualization and navigation: interactive, iterative

Audience Many information consumers One or few information producers: primarily business analysts

Key strength Easy to use and consume Powerful way to understand trends in the data.

Advanced Analytics

Advanced analytics includes the more advanced domains traditionally associated with ―analytics.‖ Whereas reporting and analysis tools are typically presenting what happened, advanced analytics tools take historical data to try to predict what will happen based upon previous data patterns. This broad category has been referred to in the past in a number of ways:

- Statistical analytics: The goal in statistical analysis is to determine cause-and-

effect patterns by analyzing data.

- Data mining: the process of extracting patterns from data, and it is used to

profile patterns in data for example: marketing, surveillance, fraud detection and

scientific discovery

- Predictive analytics: includes statistics, data mining, and game theory, and the

goal is to make predictions about future events from current and historical facts.

- Crowd-sourcing analytics: drawing on the wisdom of the crowds from a

specific business domain and/or audience to predict outcomes. For example, a

VP of Sales might poll his sales team for a prediction on next quarter and next

year‘s sales revenues.


© 2010 Jaspersoft

The two market share leaders in the advanced analytics segment are SAS and R. SAS is the largest privately held software company in the world and has specialized in statistical analytics since its inception in the 1970‘s, primarily targeting financial, retail, risk analysis, education, and government markets. R is the leading open source advanced analytics environment that rivals and soon will surpass, if it has not already, SAS in audience size. Offering proprietary competition to SAS on the proprietary side is IBM, with its recent acquisition of SPSS, as well as a variety of other players including: KXEN, Visual Analytics, CrowdCast, and Zementis. On the commercial open source side, vendors include KNIME, Rattle and Revolution Computing (both leveraging R), and Rapid-I‘s RapidMiner (formerly YALE).

“… Predictive models are employed for a full range of challenging problems, from strategy and

scenario planning to dynamic pricing. In most cases, the outcomes of a predictive modeling effort are

reviewed by experts. For example, a decision to change the product mix, or to enter a new market, is

not something that is left to algorithms alone. However, one area where predictive models are

deployed in an unattended mode is where there are a great number of small decisions and getting

most of them right is what that counts, not every one. For example, when directing a 2,500-person

sales force where to place promotional material on a given day, a certain level of error is acceptable,

but automatically pricing a popular item on a Web site 50% below your cost can have devastating

effects. Predictive models are not useful for unattended operation when even a single wrong

occurrence can have devastating effects. Alternatively, a certain level of error is acceptable when

each individual mistake has minor and non-recurring consequences. Currently, predictive modeling

is implemented as either a standalone application or embedded in each operational application, as

part of the application logic. In SOA, standard predictive modeling services can be dispatched and

will be able to easily access historical data (data warehouses principally).” - Neil Raden, Hired Brains

Audience

Jaspersoft’s Solution and Ecosystem

Typically technical analysts working with vast data sets.

JasperServer can expose the output of advanced analytic tools, for example from R, as a dashboard component.

Dashboards, Mash-Ups, and Portals

Aside from the e-mail client, dashboards are often thought of as the ―home base‖ for a business user. A dashboard means different things to different business users depending upon their profile. On one hand, a sales executive will typically want a dashboard showing the status of their lead funnel and sales revenues across geography for the current quarter. On the other hand, an average business user might want to ―mash-up‖ content that includes reports regarding their daily or weekly responsibilities as well as personalized content that could include geographically related information such as this week‘s local weather, travel information, etc., intermingled with corporate data.


© 2010 Jaspersoft

“From an analysis and BI perspective, web 2.0 technologies address interactive data exploration

and delivery more successfully than running a series of reports in a conventional BI tool. Think of

mash-ups as context-sensitive analytic applications, interactive, easy to use and self-service.

Mash-ups will be the Rip–Mix–Burn of BI as the technologies mature. People don’t want reports

for operational tasks; they want information in the context of the immediate task. This requires

building BI capabilities into applications, most of which are now deployed via web interfaces.

There are important elements required for data delivery to support repeatable mash-up

development. Mash-up and web developers have a need for more consumable data, which for

them means data available via services and in alternate formats. We need to deliver warehouse

data beyond traditional SQL access. When delivering data to application and web developers it’s

important to think SOA rather than SQL.

Doing this enables people outside the BI group to do things the BI group doesn’t have the

bandwidth to do, or never thought of doing. There’s a lot of opportunity in both of these.

These new technologies mean that databases can be web-friendly rather than the obstacle most

developers view them as today.”

- Mark Madsen, Third Nature

Therefore, mash-ups and portals, using open SOA and web services standards, can provide both flexibility of deployment from a technical perspective as well as a personalized experience from a content perspective. Very importantly, often dashboards are the access point to analytic applications, which allow people to see high-level views of relevant information and to navigate in a guided way through a set of business intelligence views. This allows personnel to find their own explanations about the details of a particular metric or key performance indicator. Ideally, an analytics environment should be able to accommodate all of these requirements.

Audience


All ranges of business users

Developers and power users can use Jaspersoft‘s iReport to create very sophisticated dashboards for business users that are deployed in JasperServer or a standard portal product. JasperServer’s open APIs enable both developers and business users to mash-up content within a dashboard using web services. And because JasperServer is JSR-168 compatible, developers can deploy reports into the portal of their choice, for


© 2010 Jaspersoft

example, IBM‘s WebSphere, Red Hat‘s JBoss Portal, and LifeRay. JasperServer Professional enables business users to drag & drop reports to create their own dashboard as well as create mash-ups with other applications and websites.

Data Store

Where the data is stored, managed, and queried typically becomes a bigger consideration with the increase in: volume of data, numbers of users, and complexity of queries. For the purpose of this paper, a ―data store‖ is defined as having:

Persistence

Metadata about the contents/structure of the store

Engine and APIs to load/update and query the data store

Online Transaction Processing (OLTP)

You may decide to use your existing OLTP operational system(s) for analytics projects due to a variety of valid reasons. For example, total-cost-of ownership is not yet an issue, there are not yet high performance requirements, you have rigid enterprise standards (eg., ―only Oracle‖), or you are in a small organization and have only that one RDBMS system. You may need a separate data store from your OLTP/application for the following reasons:

Degradation of OLTP system: Analysis operations are often large queries across

the broad database, which may negatively affect application performance and

integrity

Multiple data sources required: Data from multiple data sources and for long

historical periods may need to be integrated at different rates (daily, weekly,

monthly) to have a ―source of truth‖ for analytic needs. It may not make sense for

the single OLTP application to hold all the data, for example across different

department or niche applications, so a separate analysis specific data source

may need to be created

However, as analytic workloads increase, you may be required to design a specialized and dedicated system to accommodate those workloads. This will likely incur


© 2010 Jaspersoft

implications in modeling not only a star-schema or snowflake schema data model but also an OLAP schema for an analytical front-end. In addition, specialized indexes and optimization tools may be used to increase performance.

RDBMS with Analytic Extensions

Many OLTP systems now or will soon include analytic extensions, for example: i. Oracle: Cubes, columnar extensions

ii. Microsoft‘s acquisition of DatAllegro is expected to be incorporated into

SQL Server‘s Madison release

iii. EnterpriseDB adds analytic features to PostgreSQL

iv. MySQL pluggable analytic storage engines including Infobright, Kickfire,

Calpoint, etc.

v. VectorWise empowers Ingres and its underlying hardware with greater

efficiency for analytic workloads

The use case for an OLTP system may hit a wall for a variety of reasons, for example: complexity and people cost, hardware cost, software cost, performance of the system, or specific capabilities not provided by the OLTP system, such as an MDX language for analysis.

Analytic Databases (ARDBMS)

The Analytic database segment arose due to the realization that analytic workloads have much different requirements, i.e., primarily querying data, versus those of the traditional systems where the primary focus is transactions. Teradata pioneered the segment in the late 1970‘s and still dominates the enterprise data warehouse segment today using a mostly normalized data model and corresponding philosophy that the enterprise data warehouse is the primary source to answer all analytic questions. That philosophy of the enterprise data warehouse is still evangelized today, perhaps most notably by Bill Inmon. Another early player was RedBrick (acquired by Informix, which was subsequently acquired by IBM), founded by Ralph Kimball, and Kimball‘s philosophy took a much more decentralized approach of building data marts which may or may not evolve into a larger data warehouse. For years, the data warehouse segment revolved primarily around the Inmon versus Kimball philosophies. Despite those specialized analytic database technologies that existed, organizations more commonly used OLTP systems including Oracle, Sybase, Informix, DB2, and Microsoft SQL Server since those were the technologies already in-house and understood well by staff. In the past five years, however, the segment has seen a proliferation of new players, typically using the so-called columnar approach such as used by Sybase IQ in the late 1990‘s. (Columnar approaches were first introduced in the 1970s with products such as Model 204 and ABABAS). With a columnar database, the data is stored in columns


© 2010 Jaspersoft

rather than rows as in a traditional OLTP system. While inserts are slower that a traditional row based database, selects are many times faster, perfect for analytic workloads. Columnar databases also typically augment their speed thanks to compression and indexing techniques. A number of different approaches may be considered depending upon the business problem at hand. Curt Monash, a prolific business intelligence blogger and analyst, offers his views in a presentation called, How to Select an Analytic Database (http://www.dbms2.com/2009/02/06/final-for-now-slides-on-how-to-select-a-data-warehouse-dbms/, February 6, 2009). In it, he calls out specific analytic workload challenges that involve ―the interplay between storage, processors, and RAM,‖ and he goes on to offer the following high-level check-list in selecting a system, and it includes business considerations (e.g., SLA‘s) as well as the standard technical considerations:

Considerations Detail 1) Inventory your use cases - Current

- Known future - Wish-list/dream-list future

2) Set constraints - People and platforms - Money

3) Establish target Service Level Agreements (SLAs)

- Must-haves - Nice-to-haves

4) Database growth - As time goes by … - More detail - New data sources

5) Users (human) -

6) Users/usage (automated) -

7) Freshness (data and query results)

-

Monash points out that it is dangerous to expect one analytic database to solve every problem. Instead, he suggests to organize your requirements around specific uses cases such as: Light reporting, Diverse Enterprise Data Warehouse, Big Data (many Terabytes that grow by Terabytes over days), or Operational Analytics. In addition to both Gartner‘s Magic Quadrant report on data warehouse systems and Monash‘s report cited above, The 451 Group also offers an enlightening look at the analytic database segment as well as some profiling of the analytic database vendors. The 451 Group‘s September 2009 paper is entitled ―WAREHOUSE OPTIMIZATION, Ten considerations for choosing/building a data warehouse.‖ The following table taken from the report (Page 29, Figure 3, entitled ―The Major Data Warehousing Players,) highlights some analytic databases on the market today.

http://www.dbms2.com/2009/02/06/final-for-now-slides-on-how-to-select-a-data-warehouse-dbms/

http://www.dbms2.com/2009/02/06/final-for-now-slides-on-how-to-select-a-data-warehouse-dbms/


© 2010 Jaspersoft

Audience


All ranges of business users realize the benefits of the data store used for analytics. This segment is particularly important for power business users who have complex queries for high-volumes of data.

The Jaspersoft Business Intelligence Suite offers reporting, analysis, dashboards, and ETL and supports a wide range of OLTP and analytic databases as data sources.


© 2010 Jaspersoft

NoSQL Systems

Although NoSQL systems, in general, are not directly necessarily applicable to analytics, it is interesting to explore their use cases to provide context in today‘s information management environments. With the explosion of data we are now seeing on the internet, particularly in the e-commerce, consumer-oriented, and social networking web sites, the traditional RDBMS data stores are being found to reach practical limitations in terms of managing scalability of new data on a daily basis. New systems are emerging to address the challenges around ―Big Data,‖ typically associated with use cases of managing high transaction rates on a real-time basis. These systems, loosely grouped together under a much-debated moniker called NoSQL, break the mold of the traditional RDBMS. The term ―NoSQL‖, originally coined by Eric Evans, Systems Developer at Rackspace, was not meant to imply ―no SQL at all,‖ and it may be more appropriately defined as ―not only SQL‖, as pointed out by Emil Eifrém of Neo Technology and recognized by Evans as well. Several excellent overviews of NoSQL systems are available on the web:

Drop ACID and Think About Data, presentation by Bob Ippolito, co-founder and CTO

of Mochi Media, at Pycon 2009

http://us.pycon.org/2009/conference/schedule/event/64/

NoSQL Ecosystem, November 9, 2009, blog by Jonathan Ellis, Systems Architect at

Rackspace

http://www.rackspacecloud.com/blog/2009/11/09/nosql-ecosystem/

A NOSQL Overview And The Benefits Of Graph Databases by Emil Eifrém, CEO of

Neo Technology, at NoSQL East 2009 http://www.slideshare.net/emileifrem/nosql-

east-a-nosql-overview-and-the-benefits-of-graph-databases

Eifrém categorizes NoSQL into four major (emerging) types of data stores. The following is a combination of cited content from Eifrém and Ellis.

NoSQL Data Store Categorizations and Descriptions

Category Key Value Stores

Big Table Clones

Document databases

Graph databases

Inspiration Based on DHTs / Amazon's Dynamo specification

Based on Google's BigTable specification

Inspired by Lotus Notes

Inspired by Euler & graph theory

Data model

(global) collection of K-V pairs

Big table, column families

Collection of K-V collections

Nodes, relationships, K-V on both

Example (Query API)

Dynomite, Voldemort (get/put), Tokyo Cabinet (get/put), Sclaris (get/put)

Hbase (Thrift , REST), Hypertable, Cassandra (Thrift )

CouchDB (Map/reduce views), MongoDB (Cursor), Riak (Nested hashes)

AllegroGraph (Graph), VertexDB (Graph), Neo4J (Graph)

http://www.slideshare.net/emileifrem/nosql-east-a-nosql-overview-and-the-benefits-of-graph-databases

http://www.slideshare.net/emileifrem/nosql-east-a-nosql-overview-and-the-benefits-of-graph-databases


© 2010 Jaspersoft

Although not designed specifically for analytics, the Big Table clones are the ones most pertinent to some analytic scenarios, specifically in the context of using MapReduce. (see more on Map Reduce in the Middleware section, Hadoop and MapReduce.)

Audience


All ranges of users realize the benefits of the NoSQL, but challenges exist around analytics with those systems. The context most pertinent to analytics is when Big Table systems are using MapReduce for that purpose.

One of the strengths of Jaspersoft products is the ability to manage a wide range of data sources. Jaspersoft products can provide reporting for NoSQL and BigTable systems via JDBC connections, via a custom data source defined in Japersoft, or simply as files that are output from the NoSQL system. The following products are pertinent:

- JasperReports reporting library for developers enables print-ready, or ‗pixel-perfect‘ reporting (LGPL license)

- iReport is a graphical desktop report designer built on JasperReports (GPL v3 license). With iReport developers and power users can design extremely sophisticated reports and dashboards

- JasperServer is a report server that lets you manage reports in a secured repository, scheduled reports, and display dashboards of reports (AGPL v3 license and commercial licenses available). Business users and executives can access reports, manipulate parameters, and schedule reports.


© 2010 Jaspersoft

Middleware

The often overlooked and underestimated category of middleware serves as the crucial backbone to any serious analytics implementation, and its relevance increases particularly for mid-sized and enterprise organizations. Middleware solutions solve for requirements around data latency, data movement, data transformation, semantic interpretation of data, and management of unstructured content.

Extract Transform Load (ETL)

Extract-Transform-Load (ETL) is typically the bread-and-butter middleware for any analytics implementation. ETL handles extraction of data from source systems (RDBMS, files, etc.), transforming the multiple source data formats into the single desired target format, and loading the data to a target system such as an operational data store or data warehouse. Common industry estimates are that ETL represents 80% of the cost and effort in implementing an analytics solution. ETL is typically manifested in three major forms:

i. Hand-coding: Scripts have been historically a favorite for business

intelligence technical experts to quickly move and transform data. A

variety of languages can be used to manage the ETL jobs including Perl

and Java, and the language of choice of data management professional is

typically of course SQL. However, as the number of sources and

transformations increase, a purpose-built graphical ETL tool quickly

becomes useful, not only for ease in creating transformations but

particularly in maintaining them.

ii. ETL tools: a tool with a graphical client to design data movement and

transformations as well as a backend data transformation engine. Leading

ETL vendors include IBM and Informatica, and Talend is the leading

commercial open source vendor. Jaspersoft OEM‘s Talend technology as

JasperETL. These graphical tools are crucial for both easily creating and

maintaining transformations.

iii. E-LT tools (Extract-Load-Transform): This approach marries the familiar

hand-coding scripts of SQL and the graphical nature of an ETL tool. This

gives the best of both worlds when processing massive volumes of data in

a set inside a data warehouse. A number of RDBMS vendors include an

E-LT tool with their RDBMS including Oracle, IBM, and Microsoft. Talend

also delivers this capability as does Jaspersoft with JasperETL.

Another sort of ETL in many respects and a way to manage batch jobs of ―big data‖ (many, many terabytes) is Hadoop; see later in this section.


© 2010 Jaspersoft

Audience


Data integration and data warehouse professionals.

JasperETL, powered by Talend, is an ETL tool that provides a graphical front-end to manage either ETL or E-LT, and the languages used are a choice of Java, Perl, or SQL. Jaspersoft and its system integrator partners provide rigorous experience in ETL. http://www.jaspersoft.com/system-integrators

Enterprise Information Integration (EII)

Enterprise Information Integration (EII) is a data integration approach in which a federated query layer is a source for analytic queries. Therefore, whereas ETL is about moving and transforming data from sources in batch to a target location, EII is about leaving the data at its source locations and acquiring it only at query time. Full enterprise-grade EII tools provide a far-ranging connection of all or most data sources in an enterprise. A benefit is that they allow for real-time querying of pertinent data at a moment‘s notice. A challenge is that they can be difficult to implement and manage, and performance degrades with an increase in query complexity and source systems called. Leading EII vendors include IBM and Composite Software. The leading commercial open source vendor is Red Hat with their Teiid project (formerly MetaMatrix ).

Audience


Professionals with ad hoc query requirements

JasperServer Professional integrates well with EII systems including Red Hat‘s Teiid so that an ad hoc query solution is available enterprise-wide without having to build a data warehouse for some ad hoc query requirements.

Complex Event Processing (CEP)

Complex Event Processing (CEP) is a category of technology that has long been recognized for its promise to deliver insight from the ever-increasing volumes of data encountered. CEP specializes in transforming massive amounts of incoming real-time data. Financial, logistics, telecommunications, and government intelligence agencies are all industries that are already relying on CEP to manage their real-time analytics requirements. Some vendors in this segment include Aleri, StreamBase, Truviso, and Progress Software. Typically CEP approaches are either computation-oriented or


© 2010 Jaspersoft

detection-oriented. A computation-oriented CEP solution is focused on executing algorithms on real-time event data, for example to compute a predicted moving average. A detection-oriented solution detects combinations of events as patterns; for example, a sequence of financial transactions could lead to fraud detection. “(CEP) can process and analyze multiple event streams associated with multiple business processes.

These event streams are often called an event cloud. CEP can correlate event streams in an event cloud

looking for patterns that may not be immediately apparent to the observer. It could, for example, examine

how buying patterns are affected by news reports, advertising campaigns, product reviews, etc.” - Claudia Imhoff, Intelligent Solutions

Audience


Professionals that require analytics engrained as a part of their daily business processes.

JasperServer Professional may be used as the front-end to an operational business intelligence environment that includes any of the following: CEP, Business Activity Monitoring (BAM), business rules engine, and business process management .

Search / Unstructured Content The ability of Search products to unlock the value of unstructured content and incorporate it with other structured content is finally becoming a reality. Search products enable categorization of text via linguistic, lexical, and statistical analysis. Some search products have the ability to expose unstructured content with a familiar SQL query syntax. In combination with business intelligence, search is being heralded as a key component of Unified Information Access architectures. Some of the leading search vendors include Autonomy, Google, Microsoft (acquired Fast Search & Transfer), Endeca, Vivisimo, Attivio, and Exalead. The leading open source engine is Lucene, and its commercial open source vendor counterpart is Lucid Imagination. “Because organizations are starting to understand the value of using more varied information to make

decisions, the ability to merge analysis of textual information with data residing in databases expands

opportunities to create better marketing initiatives, improve customer relationship management (CRM)

and increase performance.” - Lyndsay Wise, Wise Analytics

Audience


Professionals with ad hoc query requirements

JasperServer Professional may be used as the front-end to a Unified Information Access architecture, to access structured and non-structured information in a unified, consistent GUI.


© 2010 Jaspersoft

MapReduce / Hadoop MapReduce is a software framework specification created by Google to support distributed computing on large data sets on clusters of computers. Hadoop, an open source project based upon MapReduce and originally created by Yahoo!, is a Java software framework that supports data-intensive distributed applications and enables applications to work with thousands of nodes and petabytes of data. Hadoop can be used for batch-oriented, data-intensive jobs. Hadoop has a file system which distributes data across multiple machines. It achieves reliability by replicating the data across multiple hosts. One of its strengths is that it enables work to be done on pieces of the data in parallel. In this way, Hadoop may be used as a sort of distributed ETL for very complex or unstructured data to manipulate it before outputting it to a data warehouse. CloudEra is a commercial open source vendor supporting Hadoop. Some vendors are embracing Hadoop in a lukewarm fashion. Anant Jhingran, CTO of IBM's information management division in the software group, says, ―Hadoop is best used to solve emerging problems such as Web analytics, fraud, and analysis of unstructured and semi-structured data, rather than the problems that relational databases have already proven to excel with.‖ In addition to managing batch jobs similar to an ETL fashion, Hadoop extensions such as HIVE and HadoopDB, may be used to manage new types of data warehouse environments. HIVE is an open source project originally developed by Facebook. Although its efficiency with hardware and availability are not as good as with traditional RDBMS technologies, Hadoop excels in being able to scale with managing large amounts of data. However, basic data manipulations that are simple in SQL could take hundreds of lines of Java or Python code. To overcome these challenges, HIVE enables Hadoop in the following ways:

Enables publishing data to schemas (metadata)

Implements a subset of SQL (rather than require programming in MapReduce)

Compiles SQL into a MapReduce configuration file

An engine that can execute the configuration file

HadoopDB, a hybrid of a traditional DBMS (currently PostgreSQL but theoretically others could be plugged in) and MapReduce, extends HIVE and is specifically designed for web scale analytic workloads. Daniel Abadi, Assistant Professor of Computer Science at Yale University, announced the start of the project on his blog in July 2009 (http://dbmsmusings.blogspot.com/2009/07/announcing-release-of-hadoopdb-shorter.html), and the paper describing it was published in August 2009

http://dbmsmusings.blogspot.com/2009/07/announcing-release-of-hadoopdb-shorter.html

http://dbmsmusings.blogspot.com/2009/07/announcing-release-of-hadoopdb-shorter.html


© 2010 Jaspersoft

(http://db.cs.yale.edu/hadoopdb/hadoopdb.pdf). In that blog, he summarizes the characteristics of HadoopDB:

1. A hybrid of DBMS and MapReduce technologies targeting analytical query

workloads

2. Designed to run on a shared-nothing cluster of commodity machines, or in the

cloud

3. An attempt to fill the gap in the market for a free and open source parallel DBMS

4. Much more scalable than currently available parallel database systems and

DBMS/MapReduce hybrid systems (see longer blog post).

5. As scalable as Hadoop, while achieving superior performance on structured data

analysis workloads

Audience


All ranges of business users realize the benefits of the data manipulated for analytics. This segment is particularly important for developers managing high-volumes of data, typically hundreds of gigabytes or terabytes of data per day.

The output of Hadoop‘s file system, HIVE, and other Hadoop extensions are files, and Jaspersoft products can easily report on those files. Jaspersoft provides several products to manage reporting:

- JasperReports reporting library for developers enables print-ready, or ‗pixel-perfect‘ reporting (LGPL license)

- iReport is a graphical desktop report designer built on JasperReports (GPL v3 license). With iReport developers and power users can design extremely sophisticated reports and dashboards

- JasperServer is a report server that lets you manage reports in a secured repository, scheduled reports, and display dashboards of reports (AGPL v3 license and commercial licenses available). Business users and executives can access reports, manipulate parameters, and schedule reports.


© 2010 Jaspersoft

Analytic Applications

Business Performance Management (BPM)

Business Performance Management (also sometimes referred to as Enterprise or Corporate Performance Management) gives organizations a top-down framework, with an iterative feedback loop built-in, by which to align planning and execution, strategy and tactics, and business unit and enterprise objectives. Various methodologies for implementing BPM include: the Six Sigma strategy, balanced scorecard, activity-based costing (ABC), Total Quality Management, economic value-add, and integrated strategic measurement. Key Performance Indicators (KPIs) are the core metrics on which a BPM strategy is based, and they are used to assess the current state of the business. BPM is usually about the budget planning and monitoring process:

Accounting focused

Workflow to develop the budget: review historical budget, set new budget at a

low level, approve/reject at a higher level

―What if‖ scenarios

Set budget including KPIs

Monitor budget and KPIs: dashboards, exception reporting, working at

appropriate views for a person controlling a budget in a particular area

Feed into next budget cycle.

A critical and often neglected aspect of BPM is a mechanism to identify the causes of shortfalls in KPI‘s, resources to overcome those challenges, and a clear procedure to fix the issues.

Audience


CEO‘s and other executive team members

JasperServer may be used as the front-end to a visualize a BPM solution. Specific expertise in one of the BPM methodologies should be applied by a Jaspersoft system integrator partner. http://www.jaspersoft.com/system-integrators


© 2010 Jaspersoft

Other Analytic Applications

Other analytic applications take a similar approach to BPM solutions, but rather than generalizing about the overall business, they focus on a specific problem domain of the business. Some examples include:

Sales: measuring performance and productivity, pipeline analysis, forecast

accuracy, upsell/cross-sell, lead conversion, etc.

Human resources: measuring employee acquisition cost, development cost,

retention rates, productivity, and compensation.

Service and Contact Center: service effectiveness, customer satisfaction,

resolution rates, service representative efficiency, service cost, etc.

Financial: measuring general ledger, accounts receivables and payables;

budgeting, forecasting, planning, profitability, etc.

Supply chain and Order Management: measuring spend and supplier

performance, inventory, fulfillment status, customer status, order cycle time, etc.

Audience


Executive and management team members responsible for specific business units, and any staff managing budgets

JasperServer may be used as the front-end to a visualize an analytic application solution. Specific expertise in a specific business domain should be applied by a Jaspersoft system integrator partner. http://www.jaspersoft.com/system-integrators Jaspersoft ISV partner Adaptive Planning provides a ―software-as-a-service‖ (SaaS) application to manage budgeting and forecasting as an alternative to the traditional, complex, and inefficient approaches with Microsoft Excel.

http://www.jaspersoft.com/system-integrators


© 2010 Jaspersoft

Conclusion

In summary, the analytics landscape has grown to cover a broad area across specialized solutions that cover the presentation layer, middleware, and data store. Despite major consolidation in the analytics space over the past five years amongst the largest independent vendors such as IBM, Oracle, SAP, and Microsoft, innovation continues to thrive. And the one thing that is predictable is that the market dynamics will continue to change as explosions in data accelerate and the market demands ever increasing insight around analyzing and revealing to customers ―what happened‖ and ―what will happen‖ based upon collected data. However, of course analytics is not about technologies; rather, it‘s about solving problems. As Eckerson of TDWI writes, the solution:

“…needs to be customized to users’ roles and personalized to their individual tastes. The best

way to start this process is to create an inventory of your users at an aggregate level... These

categories should be based on how business users consume information today.”

Combined with Eckerson‘s description of first determining ―what‖ is required of your stakeholders, this paper has provided an overview of ―how‖ those solutions may be attained.


© 2010 Jaspersoft

Comments?

In this white paper, Jaspersoft has presented its view of the analytics landscape. We would be pleased if you could share your thoughts and questions. Please send comments and questions to [email protected], and a Jaspersoft representative will follow-up with you. Jaspersoft is the market leader in open source business intelligence (BI). With millions of downloads worldwide and thousands of commercial customers in dozens of countries, Jaspersoft offers the most widely used open source business intelligence software in the world. The Jaspersoft Business Intelligence Suite is comprised of an interactive reporting server, graphical and ad hoc report design interfaces, OLAP data analysis, an ETL data integration engine, and a Java reporting library for scalable small, medium, and large enterprises, for use as either stand-alone or embedded business intelligence applications.

© Copyright 2010 Jaspersoft Corporation. All rights reserved. Jaspersoft, the Jaspersoft logo, Jaspersoft Business Intelligence Suite, JasperAnalysis, JasperServer, JasperETL, JasperReports, JasperStudio, iReport, Jasper4Salesforce, and Jasper4 products are trademarks and/or registered trademarks of Jaspersoft Corporation in the United States and in jurisdictions throughout the world. All other company and product names are or may be trade names or trademarks of their respective owners.

mailto:[email protected]

The Analytics Landscape de Jasper Soft

Documents

Transcript of The Analytics Landscape de Jasper Soft