Enterprise intelligence apr2012 load - romania - 30 min

© 2012 IBM Corporation1

Enterprise Intelligence

Jeff Jonas, IBM Distinguished EngineerChief Scientist, IBM Entity Analytics

Email: [email protected]: www.jeffjonas.typepad.com

Twitter: http://www.twitter.com/jeffjonas

mailto:[email protected]

http://www.jeffjonas.typepad.com/

http://www.twitter.com/jeffjonas


My Background

Early 80‟s: Founded Systems Research & Development (SRD), a custom software consultancy

Personally designed and deployed +/- 100 systems, a number of which contained multi-billions of transactions describing 100‟s of millions of entities

1989 – 2003: Built numerous systems for Las Vegas casinos including a technology known as Non-Obvious Relationship Awareness (NORA)

2001: Funded by In-Q-Tel, the venture capital arm of the CIA

2005: IBM acquires SRD

Today: Primarily focused on „sensemaking on streams‟ with special attention towards privacy and civil liberties protections


Time

Com

puti

ng P

ower

Gro

wth

Sensemaking Algorithms

Available Observation

Space

Context

Trend: Organizations Are Getting Dumber

EnterpriseAmnesia

Every two days now we create as much information as we did from the dawn of civilization up until 2003.”

~ Eric Schmidt, CEO Google


Amnesia, definition

A defect in memory, especially resulting from brain damage.


Enterprise Amnesia, definition

A defect in memory, resulting in wasted resources, lower revenues, unnecessary fraud losses, etc.


Time



Space

ContextWHY?

Trend: Organizations Are Getting DumberC

ompu

ting

Pow

er

Gro

wth


Algorithms at Dead End.

You Can‟t Squeeze Knowledge

Out of a Pixel.


[email protected]

No Context


Context, definition

Better understanding something by taking into account the things around it.


Information in Context … and Accumulating

Top 200Customer

Job Applicant

IdentityThief

CriminalInvestigation

[email protected]


The Puzzle Metaphor

Imagine an ever-growing pile of puzzle pieces of varying sizes, shapes and colors

What it represents is unknown – there is no picture on hand

Is it one puzzle, 15 puzzles, or 1,500 different puzzles?

Some pieces are duplicates, missing, incomplete, low quality, or have been misinterpreted

Some pieces may even be professionally fabricated lies

Until you take the pieces to the table and attempt assembly, you don‟t know what you are dealing with


Puzzling

Cottage Garden

© 2010 Royce B. McClure,

Artist All Rights Reserved

© 2010 Ravensburger USA,

Inc.

Down Home Music

© Kay Lamb Shannon,

Artist

Licensed by Cypress Fine

Art Licensing

© 2011 Ravensburger USA

Inc.

Neuschwanstein Beauty

© 2009 Photo Copyright

Robert Cushman Hayes


Inc.

Vegas

Artwork provided by

Hadley House Licensing,

Minneapolis

© 2011 Giesla Hoelscher

All Rights Reserved


Inc.

270 pieces

90%200 pieces

66%

150 pieces

50%

6 pieces2%

30 pieces10% (duplicates)


First Discovery


More Data Finds Data


Duplicates in Front Of Your Eyes


First Duplicate Found Here


Incremental Context – Incremental Discovery

6:40pm START

22min “Hey, this one is a duplicate!”

35min “I think some pieces are missing.”

37min “Looks like a bunch of hillbillies ona porch.”

44min “Hillbillies, playing guitars, sittingon a porch, near a barber sign …and a banjo!”


150 pieces

50%


Incremental Context – Incremental Discovery

47min “We should take the sky and grassoff the table.”

2hr “Let‟s switch sides, and see if wecan make sense of this fromdifferent perspectives.”

2hr10m “Wait, there are three … no, fourpuzzles.”

2hr17m “We need a bigger table.”

2hr18m “I think you threw in a few randompieces.”


How Context Accumulates

With each new observation … one of three assertions are made: 1) Un-associated; 2) placed near like neighbors; or 3) connected

Must favor the false negative

New observations sometimes reverse earlier assertions

Some observations produce novel discovery

As the working space expands, computational effort increases

Given sufficient observations, there can come a tipping point

Thereafter, confidence improves while computational effort decreases!


Big Data [in context]. New Physics.

More data: better the predictions– Lower false positives

– Lower false negatives

More data: bad data good– Suddenly glad your data is not perfect

More data: less compute


Big Data

Pile of ____ In Context


One Form of Context: “Expert Counting”

Is it 5 people each with 1 account … or is it 1 person with 5 accounts?

Is it 20 cases of H1N1 in 20 cities … or one case reported 20 times?

If one cannot count … one cannot estimate vector or velocity (direction and speed).

Without vector and velocity … prediction is nearly impossible.


Entity ResolutionDemonstration


VOTERGeorge F Balston

YOB: 1951 D/L: 4801

13070 SW Karen Blvd Apt 7

Beaverton, OR 97005

Last voted: 2008

DECEASED PERSONGeorge Balston

YOB: 1951 SSN: 5598

DOD: 1995

Entity Resolution Demonstration

When it comes to best practices in voter matching, if only a name and year of birth match, this is insufficient proof of a match. Many

different people in the U.S. share a name and year of birth.

Human review is required.

Unfortunately, there are thousands and thousands of cases just like this and state election offices don‟t have the staff (or budget) to

manually review such volumes.



YOB: 1951 D/L: 4801


Beaverton, OR 97005

Last voted: 2008


YOB: 1951 SSN: 5598

DOD: 1995

Now Consider This Tertiary DMV Record

DMVGeorge F Balston

YOB: 1951 SSN: 5598 D/L: 4801

3043 SW Clementine Blvd Apt 210

Beaverton, OR 97005

The DMV record contains enough features to match both the voter (name, year of birth and driver‟s license) and/or the deceased persons record (name, year of birth and SSN). For the sake of argument, let‟s

say it matches the voter best.



YOB: 1951 D/L: 4801


Beaverton, OR 97005

Last voted: 2008

DMVGeorge F Balston

YOB: 1951 SSN: 5598 D/L: 4801


Beaverton, OR 97005


YOB: 1951 SSN: 5598

DOD: 1995

Features Accumulate

The voter/DMV record now shares a name, year of birth and SSN with the deceased person record. In voter matching best practices, this evidence would be sufficient to make a determination that this voter

is in fact deceased. This case no longer needs human review.



YOB: 1951 D/L: 4801


Beaverton, OR 97005

Last voted: 2008

DMVGeorge F Balston

YOB: 1951 SSN: 5598 D/L: 4801


Beaverton, OR 97005


YOB: 1951 SSN: 5598

DOD: 1995

Useful Insight Revealed!

As features accumulate it becomes possible to resolve previous un-resolvable identity

records.

As events and transactions accumulate –

detection of relevance improves.

Here we can see George who died in 1995 voted in

2008.


IBM InfoSphere Identity Insight V8


MoneyGram International


Enterprise IntelligenceOne Plausible Journey

Enterprise IntelligenceOne Plausible Journey


ObservationSpace

Sense and Respond

What you know

New Observations


ObservationSpace

Decide

?Relevance

Finds the Sensor(<200ms)

Data Finds Data

Sense and Respond


Explore and Reflect

ObservationSpace

Decide

?

DirectedAttention

Relevance Find You

DeepReflection

CuratedData

PatternDiscovery

RelevanceFinds the Sensor

(<200ms)

Data Finds Data

Sense and Respond


ObservationSpace

Decide

?

DirectedAttention

NEWINTERESTS

DeepReflection

CuratedData

PatternDiscovery


(<200ms)

Data Finds Data

Explore and ReflectSense and Respond


ObservationSpace

Decide

?

DeepReflection

CuratedData

PatternDiscovery


(<200ms)

Data Finds Data

InfoSphere StreamsILog

NetezzaSPSS

Watson

DirectedAttention

Cognos


InfoSphere Streams

NEWINTERESTS

SPSSSensemaking


ObservationSpace

Decide

?

DirectedAttention

NEWINTERESTS

DeepReflection

CuratedData

PatternDiscovery


(<200ms)

Data Finds Data

Report and Manage



Decide

?

DirectedAttention

NEWINTERESTS

PatternDiscovery


(<200ms)

Data Finds Data

Info Management Systems

Content ManagementCase ManagementData Warehousing

Report and Manage


Big Data Trends


Val

ue o

f D

ata

The Greater the Context, the Greater the Value

Pile of Data

Records Managed(Big) (Ludicrous Big)

Data in Context


Willing

ness

to

Wai

tThe better the

predictions … the faster they will be

wanted.

“Why did we have to wait until the

end of the day for the smart answer?”

Time Is Of The Essence

Relevance (Iffy) (Totally)

Day

Hour

200ms

Batch

Real-Time


Closing Thoughts


The most competitive organizations

are going to make sense of what they are observing

fast enough to do something about it

while they are observing it.


Time



Space

Context

Wish This On The Competitor

EnterpriseAmnesia

Com

puti

ng P

ower

Gro

wth


Time

The Way Forward: Enterprise Intelligence



Space

Context

Com

puti

ng P

ower

Gro

wth


Related Blog Posts

Algorithms At Dead-End: Cannot Squeeze Knowledge Out Of A Pixel

Puzzling: How Observations Are Accumulated Into Context

On A Smarter Planet … Some Organizations Will Be Smarter-er Than Others

G2 | Sensemaking – One Year Birthday Today. Cognitive Basics Emerging.

http://jeffjonas.typepad.com/jeff_jonas/2008/02/algorithms-at-d.html




http://jeffjonas.typepad.com/jeff_jonas/2008/11/puzzling-how-observations-are-assembled-into-context.html

http://jeffjonas.typepad.com/jeff_jonas/2010/03/on-a-smarter-planet-some-organizations-will-be-smarterer-than-others.html




http://jeffjonas.typepad.com/jeff_jonas/2012/01/g2-sensemaking-1-year-birthday-today-cognitive-basics-emerging.html






Email: [email protected]

Blog: www.jeffjonas.typepad.com


Questions?





Enterprise Intelligence

Jeff Jonas, IBM Distinguished EngineerChief Scientist, IBM Entity Analytics

Email: [email protected]: www.jeffjonas.typepad.com






Sensemaking on StreamsMy G2 Secret Little IBM Project

3+ years in the making


G2 Mission Statement

1) Evaluate each new observation against previous observations.

2) Determine if what is being observed is relevant.

3) Delivering this actionable insight to its consumer … fast enough to do something about it while it is still happening.

4) Doing this with sufficient accuracy and scale to really matter.


From Pixels to Pictures to Action

Observations

Data Finds Data

PersistentContext

Relevance Finds You

Consumer(An analyst, a system, the sensor itself, etc.)

This is G2


Uniquely G2

More scalable, faster and extensible– Designed for grid compute and sub-200ms sense and respond

Smarter– Tolerance for disagreement (no such thing as a single version of truth)

– Support for more abstract entities (e.g., locations, products, asteroids)

– Support for more exotic features (e.g., biometrics, social circles)

Crazy stuff– Detects on its own when it is confused and makes “note to self”

– Geospatial reasoning including a sense of here and now

Privacy by Design (PbD) – More privacy and civil liberties enhancing features baked-in than any other

commercial technology


PbD: Self-Correcting False Positives

Which reveals this is a FALSE POSITIVE

John T Smith Jr123 Main Street

703 111-2000DOB: 03/12/1984

John T Smith123 Main Street

703 111-2000DL: 009900991

A plausible claim these two people are the same

1

2 John T Smith Sr123 Main Street

703 111-2000DL: 009900991

Until this record comes into view

3


PbD: Self-Correcting False Positives

John T Smith Jr123 Main Street

703 111-2000DOB: 03/12/1984


703 111-2000DL: 009900991

John T Smith Sr123 Main Street

703 111-2000DL: 009900991

New Best Practice:FIXED IN REAL-TIME

(not end of month)


703 111-2000DL: 009900991

1

3

2

2


Customer Facing Systems

Data Mining

Back-of-House Accounting Systems

Fraud

This System That System

Sensemaking

Enterprise intelligence apr2012 load - romania - 30 min

Technology

Transcript of Enterprise intelligence apr2012 load - romania - 30 min