WHY NIELSEN COMPANY'S GLOBAL BUY PLATFORM RELIES ON COUCHBASE

DARRELL PRATTARCHITECTURE LEADER

ABOUT NIELSEN

Help our clients have the most complete understanding of consumers worldwide.

OUR MISSION

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

4

OUR ECOSYSTEMCOLLABORATIVE DATA

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

5

NIELSEN ANSWERS ON DEMAND

Flexibility• Dynamic “on-the-fly” processing engine• On-Demand products, markets, periods, buyer groups• User role-based reporting• Custom product definitions, hierarchies and characteristics

Speed to Insights• Expedited reporting• Roadmaps and guided analysis• Dynamic reporting

Integration• Consistent access channel• Internal and external data sources• Support for client business processes

Consumer foresight for faster, smarter,

more confident decisions

OUR PROMISE

to drive growth

CONSUMER DATA

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

8

SCAN, PANEL AND LOYALTY• Scan

• Point of Sales data from 1000’s of retailers• Weekly data

• Disaggregated, Anonymous data

• Billions of records per week

• Panel• Data from more than 250,000 households across 25 countries

• Similar to Nielsen Families from View

• Trip and Demographic data

• Millions of records a week

• Loyalty• Loyalty card data from retailers

• Basket level transaction data – received daily from thousands of stores

• Some demographic data

• 100’s of Millions of items weekly

WHO IS OUR CONSUMER

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

10

OUR CLIENTS• Manufacturers - Kraft, Procter & Gamble, others

• Measure product success

• Understand consumer behaviors

• Target new products or promotions

• Identify new product opportunities

• Product pricing

• Retailers – Safeway, Tesco, Walmart

• Understand consumer buying behavior

• Store performance in market

• Comparison to competitors

• Product pricing

OUR REPORTING APPLICATION

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

12

REPORT BUILDER

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

13

REPORT PLAYER

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

14

REPORT PLAYER

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

15

DATA SELECTION

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

16

FEATURES• Single front-end web application which interfaces with the disparate back-end

data sources

• Advanced BI capabilities• User expressions

• Conditional formatting

• Smart text

• Smart linking of report objects

• Very few limits to what user can request with regards to data

• Most reports to run under 2 minutes maximum

• Loading of application with most data under 5 seconds

APPLICATION SPECIFICS

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

18

HIGH LEVEL VIEW

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

19

ARCHITECTURAL VIEW

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

20

TECHNOLOGY STACK• UI built on Sencha Ext JS

• AM Charts used for charting• D3 used in some edge case chart types

• Middle tier composed of Spring MVC and Spring IOC• JSON REST endpoints through configuration

• XML to JSON conversion where needed

• SOA Tier using Tibco AMX 3.2

• Couchbase 2.1 – Storage, Caching and Search

• Hudson, Artifactory, Gradle, Jasmine, JS Duck

WHY COUCHBASE

WE STARTED WITH A SEARCH FOR A CACHING SOLUTION

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

23

SOLUTION REQUIREMENTS• Our needs

• Scalability• Shared cache/storage for separate applications

• Speed• Out of JVM process

• Support• As an enterprise, we need 24x7 support

• Our wants• Document storage

• Map/reduce views

• Full text search

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

24

WHY NO-SQL• Relation Data Model Overload

• Complexity of objects in system causes churn in DB models

• Poor performance due to complexity

• Need to get out of business of data transformations

• Flexibility of data model is near number one requirement

• Scalability with modest hardware and ease

• Data Sharding and replication for reliability

• JSON encoding• Used throughout UI, important to store as such

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

25

COUCHBASE – USAGE• Storage of native JSON data from application

• User customizations of reports

• Report definitions

• Request instances – Data selections

• BI Responses

• Metadata change management• Characteristics can and do change weekly

• Views created to track user usage and items affected by these changes

DEALING WITH REPORTING DATA

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

27

MAKE IT RESPONSIVE• Asynchronous UI

• Reporting data gets BIG• Breaking up a report into objects

• Asynchronously store chunked data in Couchbase

• UI only requests chunk needed

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

28

MOVING DATA EFFICIENTLY• Life of a request for data moves through several systems

• Web, Tibco AMX, Tibco EMS, Composite, Database

• Use Couchbase as a document storage system• Enables a pass by reference methodology

• Storage of data in format closest to what is displayed to user

• True persistent storage and in-memory performance

DEALING WITH CHANGE

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

30

DIMENSIONAL DATA CHANGES• Our data is made up dimensions (Product, Market, Period, Fact…) with each

dimension described by Characteristics

• Characteristic data is large and changing• New products introduced

• Human error on ingest

• Manufacturers change their minds

• Changes occur weekly if not daily

• Changes here create waves throughout the system

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

31

COPING WITH CHANGE• Metadata change management process

• Catch the differential changes at inception

• Send those changes through system on ESB

• User saved data needs to be updated or invalidated• Saved selections

• Saved reports

• Segment definitions

• All of these items contain this characteristic data• Before Couchbase -> Stored as CLOBS in Oracle

• Full table scans and programs to read all data and change where it was found

• With Couchbase, MapReduce Views created to easily find items with reference to characteristic data with changes• Easy to find, easy to fix. Huge time savings

LOOKING FORWARD

Copy

right

©20

12 T

he N

iels

en C

ompa

ny. C

onfid

entia

l and

pro

prie

tary

.

33

WHAT’S NEXT• Couchbase as a first class data storage application from mainframe acquisition

• Full storage of metadata in Couchbase

• Map/reduce views to capture statistics on data usage by clients

• Predictive analytics using collaborative data from Couchbase views