Tna how taxonomy applications were built

23

Transcript of Tna how taxonomy applications were built

Page 1: Tna how taxonomy applications were built
Page 2: Tna how taxonomy applications were built

Jeremie Charlet

04 08 2015

Trial-and-error experiments on Taxonomy Applications

Page 3: Tna how taxonomy applications were built

Introduction

What is Taxonomy?

To better understand what it is about,

Let’s make a search on Discovery!

3

Page 4: Tna how taxonomy applications were built

Introduction

4

Page 5: Tna how taxonomy applications were built

Introduction

Taxonomy is just about classification.

Here it concerns the applications used to apply categories (or subjects) to the records in Discovery.

Project involving several people from Taxonomy teamand Systems Development team

5

Page 6: Tna how taxonomy applications were built

Introduction

SolutionAdministration interface for taxonomists

6

Application to categorise everything once1.To do it for the first time2.to apply latest modifications from taxonomists on all documents

Application to categorise documents every day1.to categorise new documents2.to re-categorise documents when they are updated

Page 7: Tna how taxonomy applications were built

Plan

This presentation is all about how we built this categorisation system

A.Using category queries1. Get it right2. Get it fast

a) Evolution of the algorithmb) Fine tuningc) Scale out

B. Attempt using machine learningo Using a training set based algorithm

7

Page 8: Tna how taxonomy applications were built

A. Using category queriesHow to categorise a document?

Solution (from former system Autonomy):1 category = 1 search Query

8

“air force” "Air Force" OR "air forces" OR "Air Ministry" OR "Air Historical Branch" OR "Air Department" OR "Air Board" OR "Air Council" OR "Department of the Air Member" OR "air army“ …

Page 9: Tna how taxonomy applications were built

A.1. Get it rightMany parameters to take into account•Is case sensitiveness important?•Use synonyms?•Ignore stop words (of, the, a, …)?•Which attributes to use (title, description, …)? Are some more important than others?•And many others

> Iterative process

How to evaluate if our results are valid? > Use documents and categories from former system> Categorise them again and compare results

To do that quickly, created Command Line Interface

9

[jcharlet@server ~]$

./runCli.sh -EVALcategoriseEvalDataSet --lucene.index.useSynonymFilter=true

Page 10: Tna how taxonomy applications were built

A.1. Get it rightFindings1.To automate evaluation

o saved me a lot of timeo regression toolo benchmarking tool

2.Using a training set based system was not satisfactory

3.Needed to ignore case sensitiveness + punctuation in most cases

10

Page 11: Tna how taxonomy applications were built

A.2 Get it fastHow to apply our 140 categories to 22 millions records quickly?

How fast do we need our system to be?

•Former system: 10+ dayso clunkyo Have to wait months to do it againo What if categorisation goes wrong? Start again for 10 days?

•Target: ~1d1 document categorised in 4ms

11

Page 12: Tna how taxonomy applications were built

Let’s categorise 1 document at a time

Run queries in parallel

Run inverted queries

Run every query against every document one after another on the file index

Run queries against memory index

Run queries in memory to find candidates and run the candidates against the file index

A.2.a Evolution of the algorithm

12

Solution Time to categorise everything

Works

A few years

Fewer years

About 10 days

?About 10 days (60ms/doc)

Page 13: Tna how taxonomy applications were built

A.2.b Fine tuningUse the right driver for your system (NRTCacheDirectory instead of default one) > 1 line in 1 file = 20% faster on search queries

Use filter instead of query to search on only 1 document + use carefully low level api

Profile your application frequently> Identify ugly code, where to add cache, where to add concurrencySpent 7% on creating Query objects for every document: instead, create them once and store them in memory

13

Page 14: Tna how taxonomy applications were built

A.2.c Scale outRequires suitable architecture~Micro services like vs monolithic application

14

Page 15: Tna how taxonomy applications were built

A.2.c Scale out

Back to the solution…GUI for taxonomists (+ backend for GUI)•Available at all time•Do search queries•Update categories

Application to categorise everything once•Run once in a while•Needs a huge amount of instances to do the job as fast as possible•Categorise everything

Application to categorise documents every day•Run every night•Receive categorisation requests from another system

15

Page 16: Tna how taxonomy applications were built

A.2.c Scale outRequires suitable architecture~Micro services like vs monolithic application

16

Page 17: Tna how taxonomy applications were built

A.2.c Scale outOn current available platform:

2 * 24 Core CPU40 Go RAM

2 * 6 categorisation processes

Categorise 22m documents in 1d 8h = 5ms to categorise 1 doc

17

Run queries in memory to find candidates and run the candidates against the file index

About 10 days (60ms/doc)

Progress is linear

Page 18: Tna how taxonomy applications were built

A.2.c Scale outLet’s imagine that we use cloud services

Let’s suppose we already pay for something equivalent on Microsoft Azure

4 *

How much does it cost to use twice that number of servers to be twice faster (ideally)?NOTHING (* If you shut down your server once process ended)

18

INSTANCE CORES RAM DISK SIZES

PRICE

D3 4 14 GB 200 GB

£0.4179/hr 

Page 19: Tna how taxonomy applications were built

Plan

This presentation is all about how we built this categorisation system

A.Using category queries1. Get it right2. Get it fast

a) Evolution of the algorithmb) Fine tuningc) Scale out

B. Attempt using machine learningo Using a training set based algorithm

19

Page 20: Tna how taxonomy applications were built

Research on a training set based solution for 2 monthsBiggest failure, best learning

1.Take a data set of known (already classified) documents2.Split it into a test set and training set

o Train the system with the training seto Evaluate it using the test seto Iterate until satisfactory

3.Move it to productiono Classify new documents using the trained system

B. Using machine learning

20

Page 21: Tna how taxonomy applications were built

B. Using machine learningWhy it did not work1.Using category queries to create the training set

21

Page 22: Tna how taxonomy applications were built

B. Using machine learningWhy it did not work1.Using category queries to create the training set

o Highly dependent on the validity/accuracy of the category queries

2.Nature of our categorieso far too many (136)o categories too vague or too similar (“Poverty”): do not suit such a

system

3.Not the right tool? We used Lucene (search engine) built in tool

4.Nature of the data?

22

Page 23: Tna how taxonomy applications were built

B. Using machine learningWhy we should get into it•Capabilities are impressive (examples)•Enabled thanks to Cloud Computing (the computing power needed is all available) •Machine Learning As A Service> You can play with it for free (*), start prototyping

23