Big Data is Only the Beginning

8/10/2019 Big Data is Only the Beginning

1/16

G00211490

'Big Data' Is Only the Beginning of Extreme

Information ManagementPublished: 7 April 2011

Analyst(s): Mark A. Beyer, Anne Lapkin, Nicholas Gall, Donald Feinberg, Valentin T. Sribar

Enterprise architects, information managers, and data management and

integration leaders often delve into the challenge of "big data" and find that

the volume of data represents only one aspect of the problem. Leaders

must understand the many extreme aspects of information management in

order to design architectures that will keep up with enterprise needs.

Key Findings Concern over big data represents the first manifestation of the extreme challenges that will

overwhelm existing information management practices and technology.

Extreme information management issues consist of a dozen different dimensions in three

categories: quantification, access and quality assurance.

The ability to manage extreme data will be a core competency of enterprises who areincreasingly using new forms of information (text, social, context) to look for patterns that

support business decisions (Pattern-Based Strategy).

In practice, multiple factors will move toward the extreme and will interact to make data

management more complicated.

Enterprises are beginning to focus on high volumes of information to the exclusion of the many

other dimensions of information management thereby leaving massive challenges to be

addressed later.

Extreme information management challenges will exacerbate the difficulty of information

sharing and will fuel the demand for an overall metadata management capability in enterprises.

Recommendations Analyze business and operational problems using the 12 dimensions of extreme information

management particularly if those problems arise suddenly or seem unusual or unexpected.

Re-examine the enterprise's five-year plans from the perspective of the 12 dimensions. Rank

the importance of the various dimensions and determine which are current pain points.


2/16

Link data management practices across dimensions. For example, to ensure the fidelity of a set

of data, apply your evaluation standard to the metadata that came along with the data, and pay

attention to all the access enablement and control dimensions to ensure the metadata hasn't

been tampered with.

Assess the ability of the current information management infrastructure to effectively handle allthe dimensions of extreme information management environments.

Focus on information governance so as to be able to manage valuable/relevant information

do not try to governallof your available assets. Your governance focus should be based on

core business processes and the data domains that communicate across and between

processes. Develop appropriate specialist competencies in the organization, such as data

scientists, to evaluate both how information is used and how it evolves.

Table of Contents

Analysis..................................................................................................................................................3

Introduction......................................................................................................................................3

The Uses and Dangers of the Term "Big Data"................................................................................. 4

Examples of How Data Volume Evolves Into Other Challenges................................................... 4

"Big Data" Starts a Conversation About Other Dimensions......................................................... 4

Extreme Aspects of Information Management.................................................................................. 5

Big Data Starts With Quantification............................................................................................. 6

Addressing Access and Qualification.......................................................................................... 8

Pattern-Based Strategy Demands Extreme Information Management.........................................9

What to Do About Extreme Information Management Issues.......................................................... 12

Recognizing the Challenge....................................................................................................... 12

Planning for Extreme Information Management.........................................................................12

Managing Data Extremes..........................................................................................................13

Summary........................................................................................................................................14

Appendix: The Economics of Data..................................................................................................14

Recommended Reading.......................................................................................................................14

List of Figures

Figure 1. "Big Data" Concepts Create an Unbalanced Data Environment................................................7

Figure 2. Dimensions of Information Management.................................................................................. 8

Figure 3. Pushing Information Initiatives to the Extreme........................................................................ 11

Page 2 of 16 Gartner, Inc. | G00211490


3/16

Analysis

Introduction

Big data has such a vast size that it exceeds the capacity of traditional data managementtechnologies; it requires the use of new or exotic technologies simply to manage the volume alone.

But processing matters, too. A complex statistical model can make a 300GB database "seem"

bigger than a 110TB database even if both are running on multicore, distributed parallel processing

platforms. Big data has quickly emerged as a significant challenge for IT leaders. The term only

became popular in 2009. By February 2011, a Google search on "big data" yielded 2.9 million hits,

and vendors now advertise their products as solutions to the big data challenge. Inquiries about big

data from Gartner clients have risen sharply as well.

This interest springs from an increase in data volumes within enterprise systems caused by

transaction volumes and other traditional data types, as well as by new types of data associated

with next-generation operational technology (OT), video streams, images, audio, social networks

and so on. Social networking alone could bring huge external datasets into the enterprise either as

actual data or metadata and links from blogs, communities, Facebook, YouTube, Twitter, LinkedIn

and others. Importantly, volume does not only refer to the management of storing data, but also the

analytics processing the information. Too much information is a storage issue, certainly, but too

much data is also a massive analysis issue. In addition, context-aware computing and next-

generation devices could bring in another huge set of data created or captured on mobile phones

and tablets (for example, images, video and audio), as well as contextual data such as location

data, previous searches, preferences, ratings, and future information types we don't know about

yet.

Enterprises often address new information management challenges with one-off solutions, and the

big-data challenge could unfortunately follow the same pattern. Certainly, big data will require new

approaches to distributed parallel processing (such as MapReduce). But the increasing velocity,

variety and complexity of data also pose a challenge, and some information managers have

deployed systems that assume users have qualified the information and have appropriate access to

it. In other words, "big data" implies other dimensions besides volume, and these dimensions

become critical for use cases such as Pattern-Based Strategy, e-discovery, information governance

and context-aware computing. Information managers may be tempted to focus on volume alone

when they are losing control of the access and qualification aspects of data at the same time. If they

do focus too narrowly, their enterprises will have to make massive reinvestments within two or three

years to address the other dimensions of big data.

Today's information management disciplines and technologies are simply not up to the task ofhandling all these dynamics. Information managers must fundamentally rethink their approach to

data by planning for all the dimensions of information management. The business's demand for

access to the vast resources of big data gives information managers an opportunity to alter the way

the enterprise uses information. IT leaders must educate their business counterparts on the

challenges while ensuring some degree of control and coordination so that the big-data opportunity

doesn't become big-data chaos, which may raise compliance risks, increase costs and create yet

more silos.

Gartner, Inc. | G00211490 Page 3 of 16


4/16

The Uses and Dangers of the Term "Big Data"

The term "big data" puts an inordinate focus on the issue of information volume (in every aspect

from storage through transform/transport to analysis). Big data is also heavily weighted toward

currentissues and can lead to short-sighted decisions that will hamper the enterprise's information

architecture as IT leaders try to expand and change it to meet changing business needs. But theterm does serve the useful purpose of starting a conversation about the challenges and benefits of

a modern information management architecture and infrastructure. This conversation will prove

especially valuable for enterprises that must operate in hyperconnected business environments.

Examples of How Data Volume Evolves Into Other Challenges

Large data volumes also bring challenges in other dimensions. For example, suppose a large

country builds a smart electrical grid with billions of meters (a form of OT) to detect power usage.

The meters would send huge amounts of data to those monitoring the health of the grid, but most

readings would amount to "noise" because nearly all would simply report normal operation. A

technology such as MapReduce can perform a statistical affinity or clustering analysis by movingthe initial processing to the meters to gather similar readings together. An analyst can then specify

that the system should filter out readings within normal parameters and display only abnormal

readings. This type of processing cuts down on the actual data volume being transmitted and can

also address the fidelity and perishability of the data for this use case managers can't adjust the

power grid to avoid problems if they don't get the right data fast enough. Other users can set

different parameters for the meter data to perform other kinds of analysis as long as this polling

doesn't prevent the smart meters from doing their main task.

Similarly, climate scientists may want to combine massive environmental datasets (such as ocean

temperature or salinity data combined with weather data). But when simply performing large-data-

volume transport, reductions or some other types of optimized query are completed in a void; it isthen possible that one source or part of the dataset is no longer valid and the entire analysis is

therefore invalidated.

"Big Data" Starts a Conversation About Other Dimensions

Thus, when clients delve into the big-data problem, they discover that it is more complex than they

realized. The term "big data" has up to a dozen different definitions. More importantly, the amount

of data that IT leaders must manage represents only one of the challenges they face. Other

challenges appear under different names, such as:

Real-time data, which focuses on what is happening right now, not on what has alreadyhappened. It enables situational awareness. Real-time data raises the issue of perishable data

(data freshness) and "orphaned" data (which no longer has valid use cases but continues in use

nonetheless; this is different form orphaned data that has lost integrity).

Shared data,which focuses on information shared across applications. To share information

effectively, enterprises must ensure the data is consistent, usable and extensible. People can

combine it with data from other sources and easily share it with other users. More importantly,

shared data complicates the task of determining the authority of information.



5/16

Linked data, which comes from various sources that have relationships with each other and

maintain this context so as to be useful to humans and computers. (Linked data often uses the

Resource Description Framework data model and Uniform Resource Identifiers to name data

objects, which can be accessed via HTTP.) Once data is linked by a user, a relationship in that

data persists from that point forward. The Linked Data group, a public organization, addresses

these concepts.

High-fidelity data, which preserves the context, detail, relationships and identities of important

business information (often via embedded metadata). Most importantly, high-fidelity data allows

new meanings to be added without destroying the previous meaning of the data.

Gartner contends that terms like "big data," "real-time data" and "linked data" signal a new era in

which the economics of data (not the economics of applications, software or hardware) will drive

competitive advantage (see Appendix). But these big-data, real-time practices and data

management theories create a pool of data dependent on external factors or otherwise stretch

conventional data management technologies and practices beyond their capacity. Traditional

information management techniques usually assume a cohesive control of both the storageprocesses and the integrity of the information then link disparate sets via metadata instructions,

which are executed in a type of application server. With big data, the process must move to the

data, instead of moving the data from its stored location into a process and then back to write out

the result. The only merit of the traditional approach is that it happens to be in place yet it

provides no real advantage and actually increases the number of hours required to maintain and

modify it.

Big data will cause traditional practices to fail, no matter how aggressively information managers

address dimensions beyond volume. Data and system architects have learned that immediate

business needs drive the information architecture along one or more dimensions of data

management sometimes to the exclusion of the other dimensions. But the moment anyinformation or data asset leaves its original process, the excluded dimensions of information

management reassert themselves.

Extreme Aspects of Information Management

IT leaders must understand all 12 of the dimensions that stress the information management

environment. This knowledge will enable information managers to make intelligent compromises

when they choose not to consider all the dimensions at once. Moreover, an awareness of these

dimensions will help to anticipate integration issues.

We have grouped the 12 dimensions of information management into three categories, each with

four dimensions:

Quantification:

Volume of data.

Velocity of data streams, access demands and record creation.

Variety of data formats.



6/16


7/16

Figure 1. "Big Data" Concepts Create an Unbalanced Data Environment

Variety Complexity

Velocity Volume

Source: Gartner (March 2011)

In Figure 1, the middle portion of the concentric circles represents traditional operational

parameters that most IT leaders are comfortable with. As the outward-pointing arrows indicate, a

data environment can become extreme along any of the four dimensions or all of them at once.

Multiple factors will move the data challenge toward the extreme and will interact to make data

management more complicated.

For example, an increase in the speed with which data changes may accompany an increase in

data volumes. Or adding a variety of new data types to the data environment (such as video) may

mean dealing with technologies for which there are no standard formats and therefore involve more

complex data types. Thus, IT leaders cannot simply focus on data volume alone or any other single

dimension. They must be aware of all the dimensions that are moving toward extremes in their

environment, then learn how these dimensions interact with each other. Information managers must

make decisions every day that balance the long-term needs of the enterprise against the immediate

pain points.

Big data suggests a new scenario for information architecture that can solve problems we cannot

address today. If data can be analyzed rapidly by moving the application to the data, instead of the

other way around, we can move multiple scenarios to the data very quickly. Instead of determining

data quality rules processing for batch applications, the same data quality rules can be moved to

the data along with the analytics model. When we remove the issue of data transport and

processing along the "route," we can analyze information under multiple scenarios very quickly.



8/16

Addressing Access and Qualification

New information sources (such as social media), easy access to them, and the lack of tools in

traditional IT departments to mine such sources have helped drive concerns over big data. A

tendency has even started to emerge among IT professionals to use the term "big data" to refer to

anything beyond traditional understanding much like the use of "structured" and "unstructured."But social media, blogs, "tweets" and chats are not always large in volume compare, for

example, astronomical data. Addressing the access and qualification dimensions allows enterprises

to plan for reusing, analyzing and sharing these information assets (see Figure 2).

Figure 2. Dimensions of Information Management

Qualification and Assurance

Access Enablement and Control

Quantification

Classification Contracts

Technology Pervasive Use

Perishability Fidelity

Validation Linking

Velocity Volume

Variety Complexity


Access Enablement and Control

These dimensions relate to making sure that people and machines can find and use information

when needed, but that unauthorized use does not occur. Access enablement and control set the



9/16

stage for who can see data, how fast it should be provided, the different delivery mechanisms and

much more. As a result, they provide the most significant opportunity to plan for context-aware

computing, data center modernization, and the convergence of OT and IT.

Classificationincludes sensitive/non-sensitive and private/public classifications and an

understanding their implications. Authorization and security fit here, as does public or privateinformation.

Contractsinvolve agreements on who will share information and how both inside and outside

the enterprise usually represented by metadata. This dimension also includes the terms of

sharing, how the records will be exposed, the intended use, how long can you use the

information and so on. These contracts are required to satisfy compliance requirements,

especially with respect to external data transfers between enterprises, such as a retailer

sending files to a data enrichment service to add demographic detail to customer information.

Pervasivenessrefers to information and data that becomes "hot" and is in great demand across

the organization. How long does data remain active? What do you do with orphaned data thathas outlived its value but for some reason keeps hanging around?

Technology-enablementinvolves specifications derived from the other 11 dimensions that guide

the design and integration infrastructure of systems such as data integration tools, data quality

tools, master data management and application middleware.

Qualification and Assurance

These dimensions relate to the levels of trust that users can place in data. Addressing qualification

and assurance can help with Pattern-Based Strategy, social networking and analysis.

Fidelitymeans the ability or inability to confidently adapt an asset for wider use.

Linkinginvolves data in combination and the uses related to this context.

Validationensures that the information was created in accordance with complete understanding

of the use cases and includes all the other aspects of data quality. Unknown future use cases

make validation a constant challenge.

Perishabilityrefers to the confidence that the data remains valid and reaches all use cases while

it remains so. What is the shelf-life of the information? How long does it remain useful? How

long should it be kept? What are the aging aspects of information?

Pattern-Based Strategy Demands Extreme Information Management

Considering all the data dimensions goes against conventional system design, which often focuses

on one or a few dimensions to solve problems quickly and elegantly. For example, OTs such as

meters and RFID systems have traditionally kept data models and attribution very narrow (they

usually meter data and do not have to manage transactions). This specialization, along the

technology dimension, provided speed and aggressive storage management so that the

technologies could fit into many repeatable, distributed devices. Back-office systems have focused



10/16


11/16

Figure 3. Pushing Information Initiatives to the Extreme

Context-AwareComputing

Search/

Mobile

SocialComputing

Social

Networking

Enterprise Systems

Transactional DataDocuments VideoTextAudio

ImagesIT/OT

Variety Complexity

Velocity Volume

Pattern-Based Strategy

OT = operational technology


The dimensions within each category interact to complicate the challenge of managing information

and data. Urgent demand for a particular set of data across the organization and higher

expectations for data validation may work against each other. Making data available to more peopleand technologies exacerbates the challenge of preventing unauthorized access. The dimensions

interactbetweencategories as well. For example, linking external data to internal sources increases

the challenge of maintaining a consistent ontology. Adding metatags to illuminate the context of

data can vastly increase the size of the data and further complicate technology issues.

Interactions between dimensions can also solve problems. For example, data validation (data

quality) can introduce bias, but the metadata of high-fidelity data can explain where that bias comes



12/16

from and how to address it. Linked data, once linked, can be qualified and the linking weighted,

based on metadata found in the other three axes of the qualification category.

What to Do About Extreme Information Management Issues

Information managers, enterprise information management leads, information architects, dataarchitects, system architects and other IT leaders must recognize when they face extreme

information management challenges, plan how to address them (including deferral if needed),

determine if more Web-enabled architecture approaches are needed, and acquire new technologies

and practices to manage them when the business needs demand it.

Recognizing the Challenge

IT leaders can easily spot extraordinary amounts of data, but other extreme dimensions of

information management may be disguised at first. Leaders should get into the habit of analyzing

business and operational problems using the 12 dimensions of extreme information management

particularly if those problems arise suddenly or seem unusual or unexpected. Determine whetherimmediate issues must be addressed in isolation or as part of a broader solution. For example, if

marketing or product development complains that security policies are too restrictive, ask whether

new business needs are driving the department to give more users (perhaps customers) access to

sensitive data. That might point to extreme demands along the classification dimension. Do you find

that right after upgrading systems to give people access to applications and data on mobile

devices, the sales department adds two new technologies to its must-have list? In that case, IT

leaders may need to plan for ongoing challenges in technology-enablement. Plan on retrofits to your

information architecture to compensate for the compromises affecting data dimensions that were

previously ignored.

Planning for Extreme Information Management

When IT leaders believe they face an extreme information management challenge, they should re-

examine their enterprise's strategy and plans for the next five years from the perspective of the 12

dimensions:

Where will business activities strain the data environment?

Will data velocity increase or the number of information asset types?

Will users need to know the context around data more than they do now, or will the life cycle of

data become critical?

Next, IT leaders should rank the importance of the various dimensions within each of the three

categories and then for all three categories together. Leaders should revise their data architecture to

reflect these priorities. The 12 dimensions can also help leaders make the trade-offs necessary to

support extreme data. For example, if linked data becomes most important, perhaps the business

can afford to lower its requirements for other dimensions such as the variety of data types

supported or the validation of data. IT leaders should work with business managers to agree on the

right set of compromises.



13/16

This kind of big data analysis can also establish patterns and solutions for managing the disparate,

unqualified and high-volume data that is available on the public cloud. Techniques for accessing

and analyzing the very largest datasets apply to data from the cloud. Public data is just as "dirty"

and unqualified as any other source. The strategies that IT leaders have developed for ensuring the

fidelity of data can work for large, publicly available datasets. The largest datasets will require the

use of public or private cloud computing resources to achieve periodic and on-demand scaling.

Managing Data Extremes

Many familiar technologies and practices will fail when they confront extreme data, so IT leaders will

need to find new ones. The profiling and cleansing of data commonly done today will not work

when qualifying large data volumes or wide varieties of data types. Search technologies work best

with unstructured data in the middle of the spectrum of velocity, volume, variety and complexity, but

falter at the extremes.

To prioritize which aspects of extreme information management to address, a new class of data

expert is emerging, sometimes referred to as the data scientist. These experts combine expertise inmathematics-based semantics in computer science with knowledge of the physics of digital

systems. The data scientist figures out when data movement (such as to warehouses) or distributed

processing (such as MapReduce) represents the best practice or an interim compromise that best

meets the service expectations of information consumers.

IT leaders will also have to link more traditional data management practices to efforts to manage

data extremes. For example, if the enterprise acquires a set of data and needs to ensure high

fidelity, leaders will have to apply their evaluation standard to the metadata that came along with the

data. Leaders will also have to pay attention to access enablement and control dimensions to

ensure the metadata hasn't been tampered with, such as by auditing the lineage of the metadata

and applying risk management calculations. Leaders will be expected to address multiple issuessimultaneously, such as considering how the technology and volume dimensions interact with each

other to complicate subsequent data validation requirements.

New Tools Will Help

Generally speaking, the further out along a given dimension a problem goes, the more specialized

the solution becomes. For example, big data will require moving affinity, scoring and cluster

analysis to the data. But this approach might assume that data perishability has been addressed

maybe it hasn't. Maybe 44% or 52% of the dataset has expired. Best-of-breed architectures and

partnerships will emerge to augment core information management capabilities with a specific

toolset for managing specific data dimensions. For instance, a database management system(DBMS) that supports MapReduce will supplement a core DBMS that optimizes analytics. A data

integration tool will incorporate statistical scoring via a best-of-breed deployment. In-memory will

replace disk management, but data architecture and modeling will enhance in-memory for complex

analytic models. Analysis tools deployed as portable services will become extremely important.

Extreme information management challenges will attract more vendors to help. Some vendors

already manage large amounts of data for custom audiences at scale (for example, managing sales



14/16

campaigns across inventory sources). In some cases, both enterprises and vendors will be able to

use venture capital to develop solutions for extreme data management.

Summary

Clients and vendors increasingly encounter a phenomenon they call "big data," but the term issometimes misleading because the challenge has many dimensions in addition to the volume of

data under management. Gartner has identified 12 dimensions in three categories: quantification,

access enablement and control, and information qualification and assurance. These dimensions

interact with each other to exacerbate the challenges of next-generation information management.

IT leaders must recognize the signs of these challenges, design information architectures and

information management strategies to address them, and deploy new technologies and practices to

manage data extremes, as traditional methods will fail. Failure to plan for all data dimensions in

systems deployed in 2011 and 2012 will probably force a massive redesign for more expansive

capabilities within two or three years. IT leaders should stop thinking about big data volumes alone

and consider all 12 dimensions of extreme information management when they develop modern

information architectures and strategies. This perspective will enable them to make intelligent

compromises.

Appendix: The Economics of Data

IT leaders must think about big data and all the dimensions it implies in order to take advantage of

the economics of data (see "How to Plan, Participate and Prosper in the Data Economy"). The

Internet openly and rapidly distributes massive amounts of content at low cost, and this widespread

access and information sharing creates an unprecedented depth and breadth of connections and

relationships.

In this hyperconnected age, where users are empowered, IT leaders may never regain the controlthey once had. They must change from being the sole owner of the technology stack to being a

business partner and technology steward. Thus, they must help their business counterparts

understand how to use new technologies and new data sources, while educating them on the risks.

In the hyperconnected age, everyone with an Internet browser can become an information

specialist. More data and more technology introduce more risk (and potentially more costs). These

challenges reach their peak in the widening array of data sources available to the business that

is, in the big-data phenomenon.

Recommended ReadingSome documents may not be available as part of your current Gartner subscription.

"Findings: 'Big Data' Is More Extreme Than Volume"

"Hadoop and MapReduce: Big Data Analytics"

"2011 Planning Guide: Data Management"



15/16

"Magic Quadrant for Data Warehouse Database Management Systems"

"The State of Data Warehousing in 2011"

"How to Plan, Participate and Prosper in the Data Economy"

"CEO Advisory: 'Big Data' Equals Big Opportunity"

More on This Topic

This is part of three in-depth collections of research. See the collections:

Pattern-Based Strategy: Getting Value From Big Data

Information 2020: Beyond Big Data

CRM Team Perspective: CRM and Big Data



16/16

GARTNER HEADQUARTERS

Corporate Headquarters

56 Top Gallant RoadStamford, CT 06902-7700

USA+1 203 964 0096

Regional Headquarters

AUSTRALIABRAZILJAPANUNITED KINGDOM

For a complete list of worldwide locations,visit http://www.gartner.com/technology/about.jsp

2011 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. This

publication may not be reproduced or distributed in any form without Gartners prior written permission. If you are authorized to accessthis publication, your use of it is subject to the Usage Guidelines for Gartner Servicesposted on gartner.com. The information contained

in this publication has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy,

completeness or adequacy of such information and shall have no liability for errors, omissions or inadequacies in such information. Thispublication consists of the opinions of Gartners research organization and should not be construed as statements of fact. The opinions

expressed herein are subject to change without notice. Although Gartner research may include a discussion of related legal issues,

Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner is a public company,and its shareholders may include firms and funds that have financial interests in entities covered in Gartner research. Gartners Board of

Directors may include senior managers of these firms or funds. Gartner research is produced independently by its research organization

without input or influence from these firms, funds or their managers. For further information on the independence and integrity of Gartnerresearch, see Guiding Principles on Independence and Objectivity.

Page 16 of 16 Gartner Inc | G00211490
http://www.gartner.com/technology/about/ombudsman/omb_guide2.jsphttp://www.gartner.com/technology/about/ombudsman/omb_guide2.jsphttp://www.gartner.com/technology/about/policies/usage_guidelines.jsphttp://www.gartner.com/technology/about.jsp

Big Data is Only the Beginning

Documents

Transcript of Big Data is Only the Beginning