VENDOR COMPARISONS: THE END GAME IN BUSINESS INTELLIGENCE

A great deal of innovation in the business intelligence industry, in the past, will culminate with the

emergence of players who will dominate the industry for years to come. The winners will be

companies which are able to reduce the latencies in data gathering, analysis and decision

making. Since information for decision making requires multi-dimensional data, users have to be

able to aggregate information from diverse sources; the information should include not only

quantitative information but also qualitative information such as conversations, notes, images and

videos. The patterns in the data have to be discerned and understood as quickly as possible so

that action in taken before an opportunity is lost.

Feedback from customers provides clinching evidence that ease of integration is the most valued

attribute for customers. Inter-linked transaction systems allow companies to aggregate data from

their CRM, SCM, production and financial systems. All this data has to be free from errors and

the definitions have to be consistent across all sources. The extraction of patterns of data is aided

by machine learning systems and comprehended quickly if it is vividly visualized. Decision

making and its implementation involves the broad majority of employees in a company who need

to be able to compare their data with agreed standards of performance before they can take

action. All employees have to be able to share the same data and access in user friendly form. In

order to take action, the processes of companies have to be flexible enough to respond to

situations as they happen.

The winners among vendors will have exceptional capability in implementation of large projects

pulling together capabilities in reporting and querying, multidimensional analysis, analytics, data

management, visualization and business process management. Pure play business intelligence

vendors, such as Cognos, Business Objects, Hyperion, as much as ERP vendors like

Oracle/Siebel, IBM and advanced analytics players, such as SAS and SPSS, and upstarts like

Qliktech are all looking to provide suites of business intelligence functionalities. While pure play

vendors have an edge in consolidating products, the ERP players have accumulated competence

in integration of applications, business processes and data, the advanced analytics group of

vendors has strengths in enhancing the value of data by drawing insights for decision making

while upstarts continue to tap disruptive new technologies.

Enterprise scale business intelligence suites are the preferred flavor in the business community

eager to reduce complexity and costs. Pure Play business intelligence vendors have responded

to competition from ERP companies by tightly integrating their products. They are also agnostic

about the databases and applications and are more inclined to use service-oriented architecture

to be able to access any source of data on their network. BusinessObjects, for example, has

integrated its Crystal Enterprise and BusinessObjects products as a single suite which can be

operated with a common set of administrative tools thereby lowering the costs of installation. In

addition, users are able to take advantage of the composite of business intelligence functions

including reporting, ad hoc queries, OLAP and dashboards. Cognos 8 has unified its OLAP

(PowerPlay), Visualizer, Metrics Manager and NoticeCast, into the services-oriented architecture

that under grids its operational reporting tool ReportNet in a web based environment. In addition,

Cognos has improved access to data from the entire enterprise as a result of integration with its

DecisionStream ETL tool which can now be managed by ReportNet. As a result of the

partnership with Composite software, Cognos 8 users have the ability to query data from any

database such as Oracle, DB2, etc.

On the other hand, database and ERP vendors such as Oracle, Teradata and IBM are

consolidating their products by integrating the business intelligence functions into their databases

which considerably lowers the latencies in the transfer of data for analytical purposes. Microsoft

bundles its online analytic processing (OLAP) with SQL Server and has added data mining and a

reporting server. Oracle's10g database incorporates several of the routine business intelligence

functions into its database. SAP, Oracle, Siebel and Microsoft all offer products with automated

business processes; an update on a table triggers processes within the database, sets

applications and business processes in motion, causes updates in other databases, initiates

communication with users, and even trigger remote procedures in external systems. A tight

integration of Siebel Analytics package into its CRM applications helps to steer workflow and

receive real-time information.

All this is much harder with pure play business intelligence vendors who have to partner or make

risky acquisitions to achieve the same objective. These capabilities are important to lower the

latencies between the time a decision is taken and relevant actions are executed.

The other major advantage the traditional ERP and database vendors have to offer are their

platforms that support the broad range of functions such as composite applications, business

processes and data integration technologies. In the web services and SOA environment,

platforms are particularly useful to join myriad services. Within an SOA framework like Project

Fusion of Oracle or mySAP of SAP, Microsoft’s SQL Server 2005, diversity of functionalities can

be incorporated rendering business intelligence packages irrelevant.

REAL TIME PREDICTIVE ANALYTICS AND DATABASE PROVIDERS

One instance of the advantage database providers have in the business intelligence domain is

their ability to build in predictive analytics required for operational purposes. Several decisions are

recurring in nature yet they require up-to-date data for sound decisions. Financial institutions

have to be able to make judgments about credit worthiness, increasingly in real time, before they

can accept credit card applications from retail customers. Similarly, customers are often swayed

by fads when they choose colors for their cars or clothes or specific models. Other times some

combinations of products sell well and they are better displayed next to each other. Customer

churn, cross-selling and pricing policies are other problems that need to be addressed in real

time. Sales people have to be able to make impromptu decisions about their stocking policy as

such information flows into their transaction databases.

In the world of data warehouses and cubes, any kind of data analysis is preceded by an elaborate

process of cleaning and reformatting data before it will be ready for analysis. In operational

decisions, moreover, data volumes become overwhelmingly large and data warehouses much too

clunky to cope with the pressures of real time decision making. Increasingly, database providers

are looking to build-in canned models for the analysis of data required for decisions in routine

processes.

The change has come with the advent of the Predictive Model Markup Language (PMML), an

open standards XML based language, which facilitates the transfer of models created in one

environment, such as SAS, and transfer it to a relational database. The XML tagging helps to

describe data inputs into data mining models, the transformations used in preparing data for data

mining, and the parameters defining the data mining models so that the data mining algorithms

can be transferred to any environment whether it is CRM, SCM or production data.

IBM, for example, has a partnership with SAS, to create scoring models and transfer them to the

relational context of its DB2 database. While SQL is not meant to address complex queries, it has

the ability to find answers to relatively simple questions of most operational staff. Once the

predictive models are embedded in relational databases, they can be accessed and modified

using SQL. The data for such purposes is drawn directly from transactional databases and does

not have to be processed in a data warehouse. Similarly, Microsoft is building equivalent

capabilities for its SQL Server 2005.

The pure play BI vendors, on the other hand, are at a disadvantage as they have traditionally

used cubes for their analytical routines. Microstrategy is one exception among them with its long

standing ROLAP capabilities and is using embedded DBMS models to provide predictive

modeling capabilities for report generation. Hyperion has incorporated predictive analytics in its

Essbase product where the predictive model is added as another dimension.

Intelligence in the language I understand

The ideal that customers want to achieve in integration is to find information as close to human

natural language as possible and in a form that reflects their thought processes. This implies

pulling together data in any form whether structured or unstructured. They want to be able to

search information that matches concepts rather than specific queries. XML has helped to break

some barriers by affording an ability to describe data. Web Services and SOA architecture has

helped to join information from a variety of sources. Semantic metadata has built the foundation

for natural language searches.

When information is available close to natural language, decision makers are better able to

visualize a scenario before they can make decisions. In order for these decisions to be

actionable, decision makers need levers to implement their decisions. Integration, in other words,

is not simply a question of inter-linking applications and business processes as Enterprise

Applications Integration does. Similarly, integration is more than linking all data sources as

Enterprise Information Integration does. With the integration of data sources, enterprises can

begin to use metadata and federated queries to parse data spread all around their network. Both

EAI and EII have proved to be expensive so vendors are turning increasingly to grid computing to

lower costs. In the final analysis, integration is the sum total of integration of all sources of data,

metadata, applications and business processes.

Large scale vendors with their scalable platforms are best positioned to unify the diverse

elements that can help to extract information and present it in a form that is intelligible to decision

makers. Companies like Oracle, IBM, and Microsoft have long had strengths in application

servers that can help to bring together the composite of services required for business

intelligence purposes. Pure play business intelligence vendors, on the other hand, have

collaborated with companies, such as Composite Software, which provide integration servers.

Vendors have progressively moved from exposing legacy software as services and integrate

them with software of more recent vintage to increasingly a portfolio of services or composite

applications to department wide integration of exposed services to tentative efforts at enterprise

wide services-oriented architecture. One example of the early attempts at creating tools for

enterprise scale services architecture is SAP's Enterprise Services Architecture which provides

the tools to create services for use across an enterprise, to weave business processes with the

services using the SAP Composite Application Framework and to implement those processes on

the NetWeaver application server. BPEL-based service orchestration is used to integrate

enterprise services with SAP and non-SAP applications, including their business processes. The

distinctive aspect of SAP’s SOA strategy is that it exposes its ERP applications to services thus

saving its customers a major overhaul of their architecture. Siebel has also redesigned its CRM

so that it can be exposed as a service.

Oracle is able to integrate web services with its "Oracle Fusion Middleware," centered around its

Oracle Application Server 10g, web services orchestrated on J2EE (Java 2 Enterprise Edition)

Application Server Web services infrastructure, ESBs (enterprise service buses) and integration

server. At the base of the Fusion stack lie Oracle's version of the database grid supported

clusters of several computers. Although Oracle initially expected customers to choose its own

application server, it has increasingly been willing to integrate the products of other companies

including WebSphere.

Additional tools are available for business process management and activity monitoring tools,

business intelligence tools and enterprise portals as well as Oracle's data hubs and the Oracle

Collaboration Suite. Oracle also uses BPEL based business process integration; unlike SAP, it

extends the scope to all processes. Fusion middleware is open system architecture for integrating

services including those from other vendors such as IBM.

The action plans

The acid test for competing vendors would be the ability to orchestrate business processes with

applications and data flows. Flexible business processes are pivotal to translating strategy into

actions. Business users should be able to change business processes with ease using visual

tools. They need to be also monitor business processes and the performance parameters in order

to determine where efficiencies are possible.

A more flexible approach to management of business processes is now possible with the

emergence of Business Process Execution Language (BPEL). Standards based languages, such

as BPEL, enable companies to avoid vendor lock-in and facilitate widespread adoption. This

language is intuitive enough to let business users set up a flow of business activity; a series of

processes can be automated when the first of them is initiated. So a customer could request a

ticket for travel which will trigger a series of actions such as checking for availability, selecting a

seat, issuing a ticket, receiving a payment and depositing the money in a bank. More complex

business process integration will record the revenue in accounting software. All of this can be

done as a seamless flow of activity if none of these processes are embedded in any specific

application.

While integration of a few of the business processes has happened already, the more complex

integrations are beginning to happen with the entry of the larger vendors. Comprehensive

integration enables a company to optimize and simulate to extract efficiencies. Companies can

use data from their business processes to take decisions on improving operational efficiency.

Additional benefits follow when automated business processes respond to a new event. For

example, a retail store may find that inventories are lower than expected and its business

processes will initiate action to replenish them.

The emergence of business process management tools which can be operated by business

users, without the assistance from IT, is illustrated by Microsoft’s BizTalk Server. Business

processes can be programmed with the use of Visual Studio .NET 2003 development

environment. Alternatively, the server has the ability to insert Visio for business users to alter

business processes as they see fit.

The integration of business intelligence software with business processes paves the way for

business activity monitoring as well as event based monitoring of workflows. Business activity

monitoring compares planned performance to the actual. Events management software sets up

alerts so that managers receive warnings when the exceptions are experienced. One of the

leading players in this segment is Teradata with its Active Warehouse. Oracle and IBM have also

introduced sophisticated programs for business process management.

Master data: Navigating complex information systems

Master data management provides consistent definitions of data in a services-oriented

architecture where heterogeneous applications have to co-exist. The availability of consistent

definitions helps to considerably improve efficiencies by smoother cross-flow of processes. In the

past, each application had its own way to define business process and logic. When these

applications were integrated, the data stored with these applications had inconsistent definitions.

In addition, the data was duplicated in several applications and created confusion when it was

combined. Master data management systems provide a centralized library of business operations

such as querying customer information. The availability of master data management systems

helps to solve problems of data quality. However, applications have to be able to call information

from master data management systems before they can be utilized.

The extent of standardization of definitions can vary across different business intelligence

providers. Ideally, enterprises would like to see a master data management system is

comprehensive to include both unstructured and structured data and include all types of business

processes such as CRM, supply chain management and manufacturing data. For vendors, the

cost and complexity of Master Data Management systems grows as they aggregate more

information. The MDM solution offered by SAP, for example, is focused on transaction processing

while Hyperion MDM product is oriented toward business intelligence.

The usability of master data management systems depends greatly on how well they are

integrated with transaction and business intelligence systems and the ability to use them at run-

time. SAP, for example, has integrated its master data into its Enterprise Services Architecture

which enables its applications to look up data at run time. Siebel Systems has a market leading

master data management system in its Siebel's Universal Customer Master (UCM) management

and Universal Application Networking (UAN) systems which can work together to call data

definitions at run-time. IBM is also incorporating its Master Data Management System in its

service-oriented architecture so that data definitions can be called when applications need them.

Mining structured data

Increasingly, enterprises are looking for data mining solutions that provide analysis in real time to

feed into decision here and now. The typical problems they want to solve are to anticipate

customer churn or determining the credit worthiness of their customers. Analysis in such a short

period of time implies that data has to be extracted, cleaned and prepared for analysis in a short

enough intervals for decisions to be made. Some new entrants have found an opportunity in this

largely unaddressed segment of the market. KXEN, for example, offers Analytic Framework

product which reduces the time to define, develop and run a model. KXEN's Consistent Coder

module automatically transforms raw, inconsistent data into clean, uniformly formatted data ready

for modeling.

Several other vendors have offered solutions for the improvement of business processes in real

time. BusinessObjects XI has added BusinessObjects Process Tracker and BusinessObjects

Process Analysis that embed analytics in their business processes. The data on performance

metrics are linked to alerting capabilities so that managers can take action in real time. Cognos

has software, e-Applications, for supply chain management processes such as procurement,

sales and inventory. The tool allows customers to keep track of the performance metrics of their

suppliers and respond to alerts about events.

Mining unstructured data

Unstructured data is available in much larger volumes and is of greater interest in operational

situations where qualitative information, such as the lifestyles of customers, is more relevant.

Information from text documents can be extracted in a variety of ways including categorization,

classification, information extraction, summarization, identifying themes or topics, concepts,

information visualization and responses to questions.

Information extraction looks for key phrases such as “tourists vacationing in San Francisco tend

to come from China and other East Asian countries” in order to find data on the travel behavior of

tourists in California. When conducting topic searches, text mining tools search for broad themes

such as “the Japanese stock market performance” to cull out information most relevant for the

topic, text summarization tools scours for words like “in short” or “in conclusion” to find related

information that tells the gist of the text, categorization can be accomplished by looking at the

frequency of usage of words and their synonyms; the recurrence of a word such as mining would

suggest that the document is about decision support analysis tools. Similarly, clustering methods

comb through text to find frequently occurring words and themes; a text on business intelligence

would have clusters of predictive mining, decision-making, etc. It is also possible to extract

information related to a specific concept; a doctor could be looking for information on allergies

and would like to obtain related information on weather conditions, food habits and lifestyles. Text

mining tools are also capable of visualizing information in the form of link tables that help to

understand the mass of information. Finally, text mining can be used to answer specific questions

and the words used would suggest the required information.

Vendors can be differentiated by their inclination to use text mining tools for knowledge extraction

and real time operational needs of companies. Text mining tools, such as those offered by SAS

and SPSS, have strengths in information or knowledge extraction and categorization while the

more recent entrants are focused on addressing the real time text analysis requirements which

will focus more on clustering, summarization and visualization. SAS Text Miner has capabilities in

information extraction, categorization and concept linkage. SPSS has strengths in information

extraction, categorization and information visualization. Megacomputer, canned software like

Cognos on the other hand, has capabilities in summarization, clustering, answering questions

besides categorization and information extraction.

For the broader category of unstructured data, IBM Omnifind promises to leader in the

marketplace with its capabilities in searching content including e-mails. The search engine in the

software indexes the information and lays the ground for queries.

Intelligent data cleaning

Data cleaning software has a number of tools such as data profiling which finds inconsistencies,

parsing which identifies different types of data and places them in the relevant fields,

standardization which brings consistency into data from a variety of sources and verification tools

for comparing data against a universal master such as the U.S. Postal Service, matching which

links interrelated files and consolidation which eliminates duplicate entries. The Master Data used

by most of such software does not take into account the context, the connotations, the overtone

or the undertone in much of human expression. Inevitably, a large majority of the conversions that

happen are prone to error and require inordinate human effort to make the corrections.

Increasingly, vendors are looking to semantic metadata to do the translations of data from one

source to another. Such semantic metadata is conscious of the context in which language is

used. Some of the newer technologies, such as those offered by Silver Creek Systems, are able

to improve the efficiencies in data cleaning.

Pricing pressures

The launch of suites and the entry of ERP players in the business intelligence market have

intensified the pressure to lower prices which will work to the disadvantage of pure play BI

vendors. By all accounts, the recognized price leader in the market is Microsoft SQL Server.

Microsoft's SQL Server 2005, with its Analysis and Reporting Services, is priced at $80,000 for

1,000 users while most BI suites have a list price in the range of $450,000 to $700,000. However,

pricing data about business intelligence packages has several layers of complexity as strategic

pricing is the norm and it is not always possible to make apples-to-apples comparisons.

In a review of the data revealed during the anti-trust investigation conducted when Oracle made a

bid for PeopleSoft, the many caveats that have to be added when comparing prices were

revealed. Oracle’s well kept secret that spilled out that it willing to price out Microsoft at almost

any cost—with as much as 90% discounts. Customer acquisition has a lucrative reward in the

maintenance earnings that both companies can earn estimated at 25-30% of the license

revenues.

Mass adoption

Real time decisions have to be necessarily complemented by collaboration across the enterprise.

The broad majority of employees in organizations can participate if their familiar tools, such as

spreadsheets, are embedded in business intelligence tools. In the future, however, spreadsheets

cannot be used in isolation and have to be incorporated in enterprise wide systems; they have to

be able to migrate from desktops to servers. In the past, spreadsheets also allowed individual

users, often highly educated business analysts, to manually configure their spreadsheets to suit

their analytical needs. The formulas and the data were often lost when an employee left. In a

business intelligence environment, users have to be able to share their data and analytical

techniques with the larger community. It should be possible for all users, whatever their skill level,

to reuse the formulas somebody else might have created.

In the past, the data from business intelligence tools was, at best, exported to Excel sheets where

it could be manipulated in intractable ways and the final results were not imported back.

Increasingly, customers are looking for tools that integrate Excel spreadsheets with corporate

databases, relational or multi-dimensional, consistent with the format and architecture of their

business intelligence systems. The data should be available across all information systems and

all users should be able to trace back the methods used in analysis.

The players that stand-out in their integration of excel spreadsheets into business intelligence

systems are Hyperion, Actuate, Information Builders, Business Objects Outlook Soft, SAP and

lately Oracle. Hyperion was one of the earliest among leading Business Intelligence players and

SRC Software, later acquired by Business Objects, was among the first to offer an Excel

interface.

The value of integration of Windows Office is potentially more than the usability of a familiar

interface. Much greater benefits can be reaped when the Office applications integrate with the

applications, data and business processes of enterprises. Business intelligence vendors are

increasingly trying to gain an edge over their competitors by linking inter-related processes,

applications and data with a convenient Office interface. The joint product “Mendocino”, created

by the partnership between SAP and Microsoft typifies the competitive trends in industry; the

processes that were earlier integrated by APIs is increasingly done on a SOA platform and helps

to realize much larger gains in productivity.

Reporting tools

Two main types of reporting are available with business intelligence tools and these are

production reporting and management reporting. Production reporting generates routine

documents such as invoices, bank statements which are repeatable. Management reports, on the

other hand, are ad hoc in nature and extract data to decision related questions such as how many

customers bought goods worth more than $2000 in the Christmas season. Increasingly, vendors

seek to gain competitive advantage by building in the capability to generate more intelligent

reports. Users of production reports soon begin to ask questions such as the reasons for

exceptionally high debits recorded in their bank statements. Ad hoc reporting is meant for the

business analysts in companies. Over time, vendors have discovered a larger market for static

reports, with pre-defined templates and drill-down capabilities, for a much larger client base in the

operational staff of companies which is usually satisfied with simple queries most relevant for

their roles.

The leaders among the group of vendors who focus on the reporting space are recognized to be

Cognos which was offering Cognos 7, recently upgraded to its eighth edition recently with greater

integration of multidimensional cubes and the reporting engine, and Actuate 8. Cognos has been

a strong enough player to provoke Business Objects to acquire Crystal Reports to match its

reporting capabilities. Cognos stands out for its capabilities in ad hoc queries and a web interface.

Actuate 8 has gained considerable recognition for its ability after its acquisition of Nimble

technologies improved its ability to integrate with a diversity of data sources using EII

technologies. In addition, Actuate has its eSpreadsheet interface.

The visual big pictures in the detail

Business intelligence vendors see in interactive visualization a means to gain an edge by

providing customers a means to extract insight from large data stores. Several different

approaches are available to achieve this objective including geographical data, interactivity,

animation, super-imposing objects on data and dimensionality of the graphics.

Mapping of geo-spatial data is one of the means of relating data to location to understand trends

in terms of who, where and how determined them. A typical example could be the mapping of

concentrations of population to understand the impact store location could have on the

purchasing behavior of customers. Insights can be extracted by visually estimating the time it

would take customers to reach the store location. Additional insights could be extracted if store

locations are compared with the centers of crime in the city. Vendors seek to gain a competitive

edge by integrating business intelligence and location information so that they can be juxtaposed

on graphs which can be depicted without getting bogged down in tedious processes of data

extraction. SAS, one of the leaders in combining geographical information and business data,

now offers SAS/GIS which integrate business and location data that it draws from ESRI, a long

time market leader in location information.

Interactive visuals are another means to gain insight. While exploring information, users of

business analytics software want to view data from a variety of angles and want to portray

information as their thought process evolves. They want to see not just pretty pictures but

relationships which would require them to slide, move and juxtapose components of their visuals

to compare, contrast and highlight to bring into relief patterns and trends. They want to shuffle the

visuals to ask “what if” questions. In a typical application involving balanced scorecards, they

want to compare the planned and the actual performance. Infommersion, recently acquired by

Business Objects, provides these very features relevant for decision-support analytical

presentations.

Users can gain better understanding of their data if they have the ability to conduct visual queries

which enable them to select their data and their visuals to address the specific questions they

have in their mind. Tableau, a start-up, has pioneered visual queries using business intelligence

data. Instead of slicing and dicing data, users are able to flip visuals to spot any anomalies in their

data, noticeable trends or patterns that would be elusive especially in large data sets. The same

product has been licensed and renamed as Hyperion Visual Explorer.