BIG DATA,We have a communication problem.

GINORMOUS SYSTEMSApril 30–May 1, 2013Washington, D.C.

Daniel TunkelangHead of Query Understanding, LinkedIn

BIG DATA IS EVERYWHERE

BIG DATA POWERS EVERYTHING

DATA SCIENTISTS WORRY ABOUTVOLUME, VELOCITY, VARIETY, …

BUT THE BOTTLENECKISN’T COMPUTATIONAL

IT’S COGNITIVE

TOOLS AUGMENTHUMAN INTELLECT

BIG DATA IS A TOOL

Doug Engelbart, inventor ofthe mouse, hypertext, etc.

NOT EVERYONE SUBSCRIBESTO THIS POINT OF VIEW…

Claudia Perlich, Chief Scientist of media6degrees, speaking atTTI/Vanguard 2012 Conference on Understanding Understanding:

SHE HAS A POINT

BUT PREDICTIVE MODELINGIS NOT ENOUGH

TRAININGDATA?

OBJECTIVEFUNCTION?

WE NEED APEOPLE-CENTRICAPPROACH TOBIG DATA

INTERPRETABILITYINTERACTION

INSIGHT

LET’S START WITHINTERPRETABILITY

EXAMPLE:SVMvs.

DECISION TREE

DECISION TREES HAVE FLAWS…

DISCRETE

BUT THEYCOMMUNICATE

(if they’re shallow)

early splits provide big picture…

fat leaves guidefeature engineering

…or reveal training data problems

WHI

CHSUPPORTS

ITERATION

INTERPRETABILITY DELIVERS

Key search leader favors rule-based approach for key scoring algorithms.

Replaced regression with decision tree in local search model: gained accuracy and insight.

Using trees to recognize spam, analyze search abandonment, model / quantify social proof.

GO DEEP vs INTERPRETABILITY

A KEY DATA SCIENCE TRADE-OFF

ON TOINTERACTION

DON’T OVERPAY FOR PRECISION

BE FAST, CHEAP, AND 98% RIGHT

http://metamarkets.com/2012/fast-cheap-and-98-right-cardinality-estimation-for-big-data/

ARE PEOPLE THAT IMPATIENT?

tolerable wait time for web users

0.1s increase in latency significantly reduces # of searches, ad revenue

tl;dr: YES

IMPATIENCE IS GOODSPEED MATTERS

INSIGHT

http://blog.takejune.com/archives/52334044.html

BE TRENDY AND NORMALIZE

vs

Sept. 11thAbu Ghraib

Weapons Inspectors

SOLVE FOR INTERESTINGNESS

COMPUTE POTENTIAL INSIGHTS

APPLY HUMAN INTUITION

SUMMARY: Let’s have a conversation with Big Data.

INTERPRETABILITYINTERACTION

INSIGHT