TOTAL DATA INDUSTRY REPORT

LEO ZHANG

DISSECTING THE TOTAL DATA LANDSCAPE

“What gets measured, gets managed” - Peter Drucker

Big Data is no longer a buzzword. It is a norm. The production of data is expanding at an astonishing pace. According to Computer Science Corp, the size of total data generated in 2012 was about 0.79ZB; in 2015, this number had grown to 7.9ZB.

This is clearly a large and lucrative market that will continue to grow as the data universe expands. It is no longer optional for companies to look for data management and analytics technologies to support important business processes.

This presentation is a summary of where the data industry stands now. The following pages serve as a roadmap of the drivers and trends in the broader landscape, the different segment, as well as the key challenges and innovations in each sub-sectors.

THE TOTAL DATA ENVIRONMENT

$7.30 $11.80

$18.60

$28.50

$38.40 $45.30

$50.10

$-

$10.00

$20.00

$30.00

$40.00

$50.00

$60.00

2011 2012 2013 2014 2015 2016 2017

Big Data Market Volume Has Grown Tremendously Over Past Few Years ($Bn)

What Are The Drivers of Big Data Growth? Key Trends To Watch

Consumer Data

Digitization of Machine

Data

Platform Vendors

Tools for Queries

BI Vendors

Big-Data Oriented

Tools

Specialist Data

Service

Established Data

Practices

Data as a Service

On Stream Data •  The increasing shift to NoSQL databases

•  More enterprise adoption of Apache Spark •  Fast growing Hadoop community •  Hadoop adding to enterprise standards •  Options expand to add speed to Hadoop •  Proliferation of Self-service data preparation tools •  Data warehouses shifting to Cloud •  IoT, Cloud, and Big Data coming together

New Databases Have Been Created To Take On The Explosion Of New Data

Traditional Relational Database

NoSQL Database

Parallel Relational Database

Hadoop

Direct Record Access or Queries MapReduce Programs

Distributed Hardware Monolithic Hardware Source: Wikibon “Big Data Vender Revenue and Market Forecast 2013-17” Source: CSC “Big Data Universe Beginning To Explode”

Source: T-Systems “Trendwatch On Big Data: Facts and Figures” Source: Tableau “Top 8 Trend For Big Data”

OPERATIONAL DATABASES

ANALYTIC DATABASES

REPORTING &ANATLYTICS

DATA MANAGEMENT

PERFORMANCE MANAGEMENT

EVENT PROCESSING

DISTRIBUTED DATA CACHE

SEARCH

HADOOP

THE TOTAL DATA LANDSCAPE Databases used to support and drive operational business applications, including relational, non-relational and NoSQL databases

Databases used to support and drive analytic workloads, including data warehouses and data marts, running analytic applications such as reporting and business intelligence tools

Business-user-facing analysis tools, including reporting, ad hoc analytics and visualization, as well as more advanced analytics tools, such as predictive and statistical analytics, used by data scientists and data analysts

Tools and applications to discover, integrate and prepare data for analysis, and master data management, data quality, as well as profiling, cleansing and governance tools

Specialist analytics tools used to monitor an organization’s performance according to key performance indicators

Engines designed to enable real-time processing of events and streams of events from multiple data sources

In-memory distributed data layers – including data caching and distributed data grid processing – used to improve the performance of dynamic applications and reduce database load

The use of search and related technologies to better structure and order corporate data in order for it to be readily searchable by staff, customers, suppliers or partners. Includes log management technologies used to search for and extract information from logs, sensor data and other machine data

The open source distributed data storage and processing framework and its associated projects and products

“If you torture the data long enough, it will confess.” - Ronald Coase

Source: 451 Research “Forecast: Total Data 2016”

TOTAL DATA SEGMENTS GROWTH Projected Annual Revenue ($Bn)

$0

$20

$40

$60

$80

$100

$120

$140

2015 2016 2017 2018 2019 2020

Performance Management Data Management BI & Reporting Data Grid/Cache Search Event Processing Hadoop Analytic Databases Operational Databases

Source: 451 Research “Forecast: Total Data 2016”

14% Estimated CAGR There are significant difference between the sectors in terms of projected growth and estimated vale creation – while the smaller startups in emerging sectors are growing the fastest, the incumbent providers are likely to generate the greatest value

DATA PLATFORMS

“There’s no better way to overpower a trickle of doubt than with a flood of naked truth.”

- Francis Underwood

The Data Platform Market Can Be Split Into Six Major Categories

Operational Databases

Analytic Databases

Hadoop

Event Processing

Search

Grid / Cache

•  Cannot deal with distributed architecture or non-relational data

•  Workloads have been slow to move to the cloud

•  Must handle more complex and larger data set •  Fierce competition in the space

•  Complex to deploy, integrate and manage •  Not reliable and secure enough for established

data management users

•  Hard to see real-time value •  Does not always help with the analysis of historical

data

•  Require human tagging to work effectively •  Enterprise search engines not as powerful as

consumer web search services

•  Not widely adopted •  In-memory database can potentially replace in-

memory data grid / cache

•  Multi-model NoSQL databases can solve scalability issue in exchange for format consistency

•  In-memory technologies increase query time •  Enabling single query to be run across several

different platforms

•  SQL-on-Hadoop enables developers to better leverage Hadoop when working with analytic databases

•  Technologies that can analyze both streaming data and historical data

•  In-memory capabilities can boost performance

•  Machine learning is being applied to search to fine tune the results based on previous searches

•  Ability to store most frequently accessed data, and data that is known to be hard to extract

•  The decision as to which to store on the grid/cache is becoming more automated

Challenges Innovations

Source: 451 Research “Data Platforms and Analytics Market Map 2016”

DATA MANAGEMENT & ANALYTICS

“The question of whether a computer can think is no more interesting than the question of whether a submarine can swim.”

- Edsger W. Dijkstra

Business Intelligence &

Reporting

Predictive Analytics

Machine Learning

Data Management

Performance Management

•  Hard to build a business around reporting •  Self-service analytics is a headache for data

management

•  Difficult for other business personnel to design advanced analytics on their own

•  Fast-growing, highly competitive industry

•  The domain of data scientists – a relatively small group of potential users

•  Difficult to meet the need for self-service data preparation

•  Providing data governance and data management for semi-structured environments

•  Persuading enterprises that Excel alone is not enough

•  Applications that support diverse data format •  Data preparation capabilities to cleanse,

transform and restructure data

•  In-database approach can obviate the need for data movement

•  The increasing use of containers enable more small machine learning applications that can be integrated into larger software

•  Searchable data catalog •  Self-service data preparation products

•  Cloud service for a wide range of functions •  Operational performance management

features that align all business units

Challenges Innovations

Five Sub-Categories

Source: 451 Research “Data Platforms and Analytics Market Map 2016”

REFERENCES

Trendwatch on Big Data: Facts and Figures | T-Systems Belgium. 2016. Trendwatch on Big Data: Facts and Figures | T-Systems Belgium. [ONLINE] Available at: http://www.t-systems.be/abouttsystems/trendwatch-on-big-data-facts-and-figures/1023624

Big Data Vendor Revenue And Market Forecast 2013-2017 - Wikibon. 2016. Big Data Vendor Revenue And Market Forecast 2013-2017 - Wikibon. [ONLINE] Available at: http://wikibon.org/wiki/v/Big_Data_Vendor_Revenue_and_Market_Forecast_2013-2017#Big_Data_Growth_Drivers

Computer Sciences Corporation. 2016. Data Universe Explosion & the Growth of Big Data | CSC. [ONLINE] Available at: http://www.csc.com/insights/flxwd/78931-big_data_universe_beginning_to_explode

Aslett, Matt. Data Platforms And Analytics Market Map 2016. 1st ed. New York City: 451 Research, LLC, 2016. PDF.

Stamper, Jason. Forecast: Total Data 2016. 1st ed. New York City: 451 Research, LLC, 2016. PDF.

Tableau. Top 8 Trends For Big Data. 1st ed. Palo Alto: Tableau, LLC, 2016. PDF.