BIG DATA NEEDS BIG CLASSIFICATION Andy Carnahan 13 November 2014 Big Data needs Big Classification...

BIG DATA NEEDS BIG CLASSIFICATION

Andy Carnahan 13 November 2014

Big Data needs Big Classification

Andy CarnahanCustomer and Information Services Manager

Wingecarribee Shire Council



Context

• The “records” battle is lost• The federated states of search• A tale of two indexes• Why big classification will happen



J Brew – Cathedral Term

ite Mound

Cc Wikim

edia Comm

ons



The extinct ECM modelLouisa Anne M

eredith, 'Tasmanian Tiger',

1880 (Tasmanian Library, SLT)

CONTEXT CONTENT• Manual Record Keeping• Central registry• Human Classification

• Physical Content• Manageable Growth• Single instance



The current ECM model

• Manual Record Keeping• Single instance ECM• Human Classification

• Technology created• Huge volumes • Indexing speed/access

CONTENT

CONTEXT

Popular Science Monthly/Volume 8/February 1876



The new ECM model

CONTEXT CONTENT• Synthetic RK• Semantic Indexing• Machine Classification

• Technology created• Federated Search• Indexing speed

Aviceda Wikim

edia Comm

ons:Peregrine Falcon , Q

ld, Apr 2007



Big Data

VelocityVolumeVariety



Big Data

Velocity

Volume

Variety

Classification must be performed at the speed of information creation at equal to human quality

Data does not (necessarily) reside in the RKMS

So long as the information is digitally available it can be will be accessible in the future

Big Classification



What is big classification?

Human-competitive automatic topic indexing

Alyona MedelyanPhD thesis in Computer ScienceUniversity of WaikatoJuly 2009



How’s your math?

• Our inputs:– Central registry: 250 mail items/day, 100

classified/workflowed (40%) by 2 records staff– 200 white collar staff: 5,000 emails / day

• The Question: How many records staff do I need to manage the 5,000 daily emails?



Current Human situation

• General staff won’t do it– Don’t understand metadata/Classification schemas– Not their job/too busy– Maintain their own data sets for own productivity

• Records staff can’t do it– Sheer quantity of information– Limited access to information domain– Not their job/too busy



Fields, haystacks and needlesClaude M

onet Wheatstacks (End of Sum

mer),

1890-91, The Art Institute of Chicago



The (old) centre of the Universe

RecordsTHE Central

Registry



A small part of a bigger Universe

EnterpriseInformation

Domain(Content)

Application Data

Databases

Documents

Intranet

Extranet

CorporateIntranet

SocialMedia

Teamsites

(Sharepoint)

Multimedia

Email

ECMS



The inadvertent ECMS



The deliberate ECMSwww.youragency.gov.org.com.net.au/about

Our mission is to organise the world’s agency’s information and make it universally appropriately accessible and useful.



ECMS

What is an ECMS?



The two components

CONTEXTMetadataDatabase

CONTENTDocument

Store



Contenxt is King


CONTENTDocument

Store



Extending to the domain


CONTENTDocument

Store



Crawling the domain


CONTENTDocument

Store



Exploring Context



Concordance -> content

Concordance Index(location)



Semantic -> context

Concordance Index(location, Content)

Semantic Index(Classification, Taxonomy

Thesaurus, Context)



The big imbalance



Thesaurus, Context)

Machine mediated

Human mediated



Index (disambiguation)

The CONTENT index Concordance (location) Left Brain Literal/Algorithmic Single path Speed

The CONTEXT index Semantic (meaning) Right brain Concepts/thesaurus Back of Book Precision



Concordance Indexing




Concordance History


First recorded instance 1230 AD Hugh of St Cher 500 monks created index to location

of every word in the Versio Vulgata (common Bible)

Same methodology used by Google to index web – URL instead of book/chapter/verse



Concordance Features


Needle-in-haystack searching Index requires no human assistance

to build Index is now built at machine speed Access to results is at machine speed Mature, widely adopted Can find the needle, but we still need

the haystacks



Semantic Indexing


Thesaurus, Context)



Semantic Indexing Injects meaning into search Search on concepts Enables multiple taxonomies in virtual

views (pivot taxonomy) Disambiguates Emerging in research and commercial

software


Thesaurus, Context)



Manual Semantic Indexing


Thesaurus, Context)

Record Keeperscontinuously

classifying incoming mail

ConfidentClassificationThreshold?

Add to indexes

Get expert help

RK remembers how to classify that exception

Y

N



Automatic Semantic Indexing


Thesaurus, Context)

“robot”continuously

crawls contentdomain

ConfidentClassificationThreshold?

Add to indexes

Get human help

robot remembers how to classify that exception

Y

N



Semantic Indexing


Thesaurus, Context)

Denise Bedford AIIM

2013



Semantic Indexing


Thesaurus, Context) Denise Bedford AIIM

2013



The big balance



Thesaurus, Context)

Machine mediated

Machine mediated



Big classification will happen

Ow

en Carnahan - used with perm

ission



2008 Predictions



2014 three suggestions

• Enterprise content management can only occur if context management (classification) is performed at machine speed.

• Machine classification must be performed at a quality similar to a records officer.

• The entire enterprise information domain is the responsibility of Records and Information Management Professionals.



The e-context future is coming…

• IBM’s Watson• Wolfram Alpha (Siri)• Cyc/Wikipedia• Smartlogic• TopQuadrant




• IBM’s Watson• Wolfram Alpha (Siri)• StoredIQ• Recommind• Pingar• Cyc/Wikipedia• Smartlogic• TopQuadrant




• IBM’s Watson• Wolfram Alpha (Siri)• Cyc/Wikipedia• Smartlogic• TopQuadrant



Record Context keepers

• Most important people in the organisation• “Training” the synthetic context agent• Refining and enhancing the context engine• ICT is the maintainer of the content engine and content

store• Records and Information Management Role is much more

rewarding



Record keeping now

Add to indexes

Record Keepers

continuously classifying incoming

mail

ConfidentClassifi-cation

Threshold?

Get expert help

RK remembers how to classify that exception

N

ECMS“small”data

Y

Manual Process



Automatic Semantic Indexing

Add to indexes

“robot”continuously

crawls contentdomain

ConfidentClassifi-cation

Threshold?

Get expert help

robot remembers

how to classify that exception

N

Y

Automated Process

Manual Process



The new ECM model

CONTEXT CONTENT• Synthetic RK• Semantic Indexing• Machine Classification

• Technology created• Federated Search• Indexing speed

Aviceda Wikim

edia Comm

ons:Peregrine Falcon , Q

ld, Apr 2007



The big balance



Thesaurus, Context)

Machine mediated

Machine mediated



Big classification

1. Federated search based on concordance indexing

2. Semantic search/classification based on context engine (machine speed/human quality)



Thank you!

Keep in touch– Linked in Andy Carnahan– [email protected]– [email protected]– RIMPA list (www.rimpa.com.au)– LG IT lists

• [email protected]• [email protected]• [email protected]

Sometimes small data is good

mailto:[email protected]


http://www.rimpa.com.au/




BIG DATA NEEDS BIG CLASSIFICATION Andy Carnahan 13 November 2014 Big Data needs Big Classification...

Documents

Transcript of BIG DATA NEEDS BIG CLASSIFICATION Andy Carnahan 13 November 2014 Big Data needs Big Classification...