Becoming (Big) Data Driven -...

29
Becoming (Big) Data Driven

Transcript of Becoming (Big) Data Driven -...

Page 1: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

Becoming (Big) Data Driven

Page 2: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

Geert Van Landeghem

3

Big Data Consultant@DataCrunchers

[email protected]

From Vision to Implementation

References

Page 3: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

13

The Ubiquity of Data Opportunities

Continuous investments in business infrastructure

Virtually every aspect of business can be measured

More and more External data is available

More interest into getting information out of data

Data mining versus data science

Page 4: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

13

From Business Problems to Data Mining

Page 5: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

13

Data mining tasks

Classification

Regression

Clustering

Profiling

Similarity matching

Co-occurence grouping

Data reduction

Link prediction

Casual modeling

Page 6: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

Data Science Context

7

Page 7: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

2

How DDD affects performance

The more data driven a firm is, the more productive it is!

One standard deviation higher on the DDD scale is associated with a

4%–6% increase in productivity.

DDD also is correlated with higher return on assets, return on equity, asset

utilization, and market value, and the relationship seems to be causal

Source: Study by Economist Erik Brynjolfsson and his colleagues from MIT (2011)

Page 8: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

2

Data Driven marketeers outperform their competition

Source: The State of Marketing 2013, IBM’s Global Survey of Marketeers

Page 9: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

Many leaders are data-driven

4

Page 10: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

What does it mean to be data-driven?

The culture and capabilities that

operationalize capturing, processing, and

utilizing data to make timely decisions

that improve products, customer

experiences, operational efficiency and

competiveness.

2

Page 11: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

Data driven hurdles

Disparate Data Sets

Data owned by different silos within the org

Lack of funding

Lack of time/resources

Legacy operating rhythm

2

Page 12: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

Embarking on your data driven journey

Transformation will not happen overnight

Support of C-Level (top-down)

Requires cross-functional collaboration

Combination of people, process, tools

Cultural champion is needed

2

Page 13: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

How does big data fit in?

Big data refers to data that is too big to fit

on a single server, too unstructured to fit

into a row-and-column database,

or too continuously flowing

to fit into a static data warehouse

6

Page 14: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

Big Data Technical Drivers

7

Page 15: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

Big Data Business Drivers

Do More (Analytics) with Less (Costs)

8

Page 16: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

Big Data Driven Roadmap

13

Page 17: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

14

Source: Bill Schmerzo (EMC)

Page 18: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

How to build the 360° view?

Two-way data integrations across

multi-channels

Data is always updated and

accessible for use

2

Page 19: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

The Enterprise Data Hub

Page 20: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

Enterprise Data Hub Definition

Contains all enterprise data

Internal & External

Processes all data in „near real-time” to generate views

Historic & Flowing

Views

Answers to business questions

2

Page 21: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

A Common Start Situation

16

Page 22: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

Our Vision #1

Focus on Data not on Derived Data

17

Page 23: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

Our Vision #2

Data is immutable

18

Page 24: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

Our Vision #3

Query = function (all data)

19

Page 25: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

Conceptually

20

Page 26: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

21

Lambda architecture

Page 27: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

View generation

- Use machine learning

Recommendations

Clustering

Classification

Regression

R

- SQL Like (Hive)

- Data Flow Programming (Pig)

25

Page 28: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

13

References

Page 29: Becoming (Big) Data Driven - download.minoc.comdownload.minoc.com/...bmit_bigdata...datacrunchers.pdf · Becoming (Big) Data Driven . Geert Van Landeghem 3 ... The more data driven

28