Tristan Sternson & Jeff Jonas

8/2/2019 Tristan Sternson & Jeff Jonas

1/67

2012 Infoready Pty Ltd1

Big Data,New Physics, and

Geospatial Super-FoodTristan Sternson, InfoReady Managing Director

Jeff Jonas, IBM Distinguished EngineerChief Scientist, IBM Entity Analytics


2/67


Who is InfoReady?

?Pure-Play Information Management and Business IntelligenceConsulting firm

?Team InfoReady career IMand BI Experts

One of the fastest growingconsulting firms in

Australia. IM Focused Tier One

Consulting capability. Focus - people, process and

technology Assisting companies turn

valuable information intoactionable intelligence.

Strategy, Architecture,Solution Design & Delivery


3/67


My Background Tristan Sternson

?Past 10 years focussed purely on IM / BI Solutions

?Started InfoReady in 2008

?Prior Roles Accenture Data Management & Architecture / IMLead, PWC Consulting / IBM

?Personally designed and deployed and led many large IM andDW application in Australia and UK

?Thought leader in Information Management in Australia andAPAC

?

Early adopter Data Governance, Big Data, Industry DataModels, Appliance DW solutions


4/67


My Background Jeff Jonas

?Early 80s: Founded Systems Research & Development (SRD), acustom software consultancy

?Personally designed and deployed +/- 100 systems, a number ofwhich contained multi-billions of transactions describing 100sof millions of entities

?1989 2003: Built numerous systems for Las Vegas casinosincluding a technology known as Non-Obvious RelationshipAwareness (NORA)

?2001: Funded by In-Q-Tel, the venture capital arm of the CIA

?

2005: IBM acquires SRD?Today: Primarily focused on sensemaking on streams with

special attention towards privacy and civil liberties protections


5/67


Big Data Definition

Datasets that grow so large that they become difficult to work with,including; capture, storage, search, sharing, analytics, and visualization.

Benefits of working with larger and larger datasets allowinganalysts to "spot business trends, prevent diseases, combat crime.

We havent seen anything yet, as more devices come online, eg;mobile, airborn, logs, cameras, microphones etc

Wikipedia - 2012


6/67


The Big Data Opportunity

V3


7/67


Big Data Why the hype?

?By 2015, nearly 3B people will be online, pushing the datacreated and shared to nearly 8 zettabytes.

?30 billion pieces of content were added to Facebook this pastmonth by 600M plus users.

?More than 2B videos were watched on YouTube yesterday.

?In the US mobile phone users between the ages of 18 and 24send an incredible 110 text messages per day.

?32B searches were performed last month on Twitter.

?Worldwide IP traffic will quadruple by 2015.


8/67

2012 Infoready Pty Ltd8 8

Business leaders frequently makedecisions based on information theydont trust, or dont have

1in3

83%of CIOs cited Business intelligence &Analytics as part of their visionaryplans to enhance competitiveness

Business leaders say they dont haveaccess to the information they need to

do their jobs

1in2

of CEOs recognise they need to betterunderstand information more rapidlyin order to make swift decisions

60%

Business Value


9/67


Big Data Trends

80%20%


10/67


What the Industry Analysts say

Gartner predicts Big Data to beone of the top-10 strategic initiatives

for 2012


11/67


What the Industry Analysts say

Key take-aways from Analyst perspectives Gartner TDWI

Data will grow exponentially

Fusion of structured and unstructured data

The connection between big data and advancedanalytics will get even stronger

Future users will not be able to put all useful

information into a single data warehouse

BigDatausedtob

ea

technicalprob

lem -nowitsa

businessoppo

rtunity.


12/67


Enterprise Intelligencevs. Enterprise Amnesia

Jeff Jonas, IBM Distinguished Engineer

Chief Scientist, IBM Entity Analytics


13/67


Time

C

omputingPow

erGrowth

Sensemaking

Algorithms

AvailableObservation

Space

Context

Trend: Organizations Are Getting Dumber

EnterpriseAmnesia


14/67


Time

Sensemaking

Algorithms

AvailableObservation

Space

ContextWHY?

Trend: Organizations Are Getting Dumber

C

omputingPow

erGrowth


15/67


Algorithms at Dead End.

You CantSqueeze KnowledgeOut of a Pixel.


16/67


[email protected]

No Context


17/67


Context, definition

Better understandingsomething by taking intoaccount the things around it.


18/67


Information in Context and Accumulating

Top 200Customer

JobApplicant

IdentityThief

CriminalInvestigation

[email protected]


19/67


The Puzzle Metaphor

?Imagine an ever-growing pile of puzzle pieces of varying sizes,shapes and colors

?What it represents is unknown there is no picture on hand

?Is it one puzzle, 15 puzzles, or 1,500 different puzzles?

?Some pieces are duplicates, missing, incomplete, low quality, orhave been misinterpreted

?Some pieces may even be professionally fabricated lies

?Until you take the pieces to the table and attempt assembly,you dont know what you are dealing with


20/67


Puzzling

200 pieces

66%

150 pieces

50%

270 pieces90%

6 pieces

2%(pure noise)

30 pieces10%(duplicates)


21/67



22/67



23/67


First Discovery


24/67


More Data Finds Data


25/67


Duplicates in Front Of Your Eyes


26/67


First Duplicate Found Here


27/67



28/67



29/67


Incremental Context Incremental Discovery

6:40pm START

22min Hey, this one is a duplicate!

35min I think some pieces are missing.

37min Looks like a bunch of hillbillies ona porch.

44min Hillbillies, playing guitars, sitting

on a porch, near a barber sign and a banjo!


30/67


150 pieces

50%


31/67


Incremental Context Incremental Discovery

47min We should take the sky and grassoff the table.

2hr Lets switch sides, and see if wecan make sense of this fromdifferent perspectives.

2hr10m Wait, there are three no, fourpuzzles.

2hr17m We need a bigger table.

2hr18m I think you threw in a few randompieces.


32/67



33/67



34/67



35/67


How Context Accumulates

?With each new observation one of three assertions are made:1) Un-associated; 2) placed near like neighbors; or 3) connected

?Must favor the false negative

?New observations sometimes reverse earlier assertions

?Some observations produce novel discovery

?As the working space expands, computational effort increases

?Given sufficient observations, there can come a tipping point

?Thereafter, confidence improves while computational effortdecreases!


36/67


Observations

UniqueIde

ntities

True Population

Overstated Population


37/67


Counting Is Difficult

Mark Smith

6/12/1978443-43-0000

Mark R Smith(707) 433-0000DL: 00001234

File 1

File 2


38/67


39/67


Data Triangulation

Mark Smith

6/12/1978443-43-0000

Mark R Smith(707) 433-0000DL: 00001234

File 1

File 2

Mark Randy Smith443-43-0000DL: 00001234

New Record


40/67


Big Data [in context]. New Physics.

?More data: better the predictions Lower false positives

Lower false negatives

?More data: bad data good Suddenly glad your data is not perfect

?More data: less compute


41/67


Big Data

Pile of ____ In Context


42/67


One Form of Context: Expert Counting

?Is it 5 people each with 1 account or is it 1person with 5 accounts?

?Is it 20 cases of H1N1 in 20 cities or one

case reported 20 times?

?If one cannot count one cannot estimatevector or velocity (direction and speed).

?Without vector and velocity prediction isnearly impossible.


43/67


Expert Counting: Degrees of Difficulty

Exactly

Same

Fuzzy

IncompatibleFeatures

Deceit

Bob Jones123455

Bob Jones123455

Bob Jones123455

Robert T Jonnes000123455

Bob Jones123455

bjones@hotmail

Bob Jones123455

Ken Wells550119


44/67


Key Features Enable Expert Counting

People Cars Router

Name Make Device IDAddress Model MakeDate of Birth Year ModelPhone License Plate No. Firmware Vers.

Passport VIN Asset IDNationality Owner Etc.Biometric Etc.Etc.


45/67


Consider Lying Identical Twins

#123Sue3/3/84UberstanExp 2011

PASSPORT#123Sue3/3/84UberstanExp 2011

PASSPORT

Fingerprint

DNA

Most TrustedAuthority

Sameperson

trust me.

Most TrustedAuthority


46/67


47/67


48/67


Life Arcs Are Also Telling

Bill Smith4/13/67

Salem, Oregon

Bill Smith4/13/67

Seattle, Washington

Address History

Tampa, FL 2008-2008

Biloxi, MS 2005-2008

NY, NY 1996-2005

Tampa, FL 1984-1996

Address History

San Diego, CA 2005-2009

San Fran, CA 2005-2005

Phoenix, AZ 1990-2005

San Jose, CA 1982-1990


49/67


OMG


50/67


51/67


Powerful Predictions

?Prediction with 87% certainty where you will benext Thursday at 5:35pm

?Names of the top 10 people you co-locate with,not at home and not at work

?The Uberstan intelligence service preempts thenext mass protest in real-time

?

A political opponent is crushed and resigns twodays after announcing their candidacy


52/67


53/67


Macro Trends


54/67


ValueofData

The Greater the Context, the Greater the Value

Pile of Data

Records Managed(Big) (Ludicrous Big)

Data

in Context


55/67


Willingness

toWait

The better thepredictions the

faster they will bewanted.

Why did we haveto wait until the

end of the day forthe smart answer?

Time Is Of The Essence

Relevance(Iffy) (Totally)

Day

Hour

200ms

Batch

Real-Time


56/67


Enterprise IntelligenceOne Plausible Journey

Enterprise IntelligenceOne Plausible Journey


57/67


ObservationSpace

Sense and Respond

What you know

NewObservations


58/67


ObservationSpace

Decide

?Relevance

Finds the Sensor(


59/67


Explore and Reflect

ObservationSpace

Decide

?

DirectedAttention

RelevanceFind You

DeepReflection

CuratedData

Pattern

Discovery

RelevanceFinds the Sensor

(


60/67


ObservationSpace

Decide

?

DirectedAttention

NEWINTERESTS

DeepReflection

CuratedData

Pattern

Discovery


(


61/67


ObservationSpace

Decide

?

DirectedAttention

NEWINTERESTS

DeepReflection

CuratedData

Pattern

Discovery


(


62/67


Closing Thoughts


63/67


The most competitive organizations

are going to make sense of what they are observing

fast enough to do something about it

while they are observing it.


64/67


Time

Sensemaking

Algorithms

AvailableObservationSpace

Context

Wish This On The Enemy

EnterpriseAmnesia

ComputingPowerGrowth


65/67


Time

The Way Forward: Enterprise Intelligence

Sensemaking

Algorithms

AvailableObservationSpace

Context

ComputingPowerGrowth


66/67


Related Blog Posts

Data Finds Data

Algorithms At Dead-End: Cannot Squeeze Knowledge Out Of APixel

Puzzling: How Observations Are Accumulated Into Context

Big Data. New Physics.

On A Smarter Planet Some Organizations Will Be Smarter-erThan Others

Your Movements Speak for Themselves: Space-Time Travel Datais Analytic Super-Food!


67/67

Email: [email protected] [email protected]

Twitter: http://www.twitter.com/jeffjonas http://www.twitter.com/tsternson

Blog: www.jeffjonas.typepad.com www.infoready.com.au

Questions?

Tristan Sternson & Jeff Jonas

Documents

Transcript of Tristan Sternson & Jeff Jonas