7/30/2019 Xapian Motor Inferencia

1/14

Introducing Xapian

Justin Finkelstein | @ilithiumPHP London, November 2011

7/30/2019 Xapian Motor Inferencia

2/14

Background and Alternatives

ReportBuyer.com235,000 reports

1.3 GB of text

Hierarchical categories

MySQL FullText

Search alternatives:

SphinxLucene, etc

Justin Finkelstein | @ilithiumPHP London, November 2011

7/30/2019 Xapian Motor Inferencia

3/14

Benefits

Justin Finkelstein | @ilithiumPHP London, November 2011

Easy to install and portableFast searching

Accurate

Powerful

7/30/2019 Xapian Motor Inferencia

4/14

Drawbacks

Justin Finkelstein | @ilithiumPHP London, November 2011

Not a databaseSingle-writer, many reader

Limited to 4.2 billion documents

OS file size limit

7/30/2019 Xapian Motor Inferencia

5/14

Installation

Justin Finkelstein | @ilithiumPHP London, November 2011

Binaries for WindowsVendor packages & PPA

Source code

Bindings

PHP

C#

Java

LuaPerl

Python, etc

7/30/2019 Xapian Motor Inferencia

6/14

Indexing

Justin Finkelstein | @ilithiumPHP London, November 2011

DatabasesDocuments

Document IDs must be unique

Terms & Stemmers

Term Generator

Values

7/30/2019 Xapian Motor Inferencia

7/14

Querying the Database

Justin Finkelstein | @ilithiumPHP London, November 2011

Simple QueriesPhrases: php development

Logical operators: OR, AND, NOT, MAYBE

Ranges: alpha..omega

NEAR: shop NEAR pubWildcards (report*)

Synonyms

Query Parser make it easy

data management AND NOT real estate AND NEAR data

7/30/2019 Xapian Motor Inferencia

8/14

Relevance and Sorting

Justin Finkelstein | @ilithiumPHP London, November 2011

BM25 Probabilistic Relevancy

Sort by rank/relevance

Sort by values

7/30/2019 Xapian Motor Inferencia

9/14

Getting Started

Justin Finkelstein | @ilithiumPHP London, November 2011

Know your data setWhat are users looking for

How will they refine their search

7/30/2019 Xapian Motor Inferencia

10/14

Report Buyer Product Data

Justin Finkelstein | @ilithiumPHP London, November 2011

item_guidtitle

subtitle

summary

table of contents

price

category

publication dateavailability

product url

7/30/2019 Xapian Motor Inferencia

11/14

Searching on Report Buyer

Justin Finkelstein | @ilithiumPHP London, November 2011

Search by:Product code

Category

Title

Price

Search text of:

Title

SubtitleSummary

Table of Contents

Refine by:Price

Availability

7/30/2019 Xapian Motor Inferencia

12/14

Mapping to Xapian

Justin FinkelsteinPHP London, November 2011

Full text with weighting:name

subtitle

summary

table of contents

Text with prefixes:

titleproduct code

category

Values:price

availability

publication date

Facets:

Category

Availability

7/30/2019 Xapian Motor Inferencia

13/14

Demo Walk-throughs

Justin Finkelstein | @ilithiumPHP London, November 2011

Indexing the data

Query parser

Sorting

MatchSpies

7/30/2019 Xapian Motor Inferencia

14/14

The End

Justin Finkelstein | @ilithiumPHP London, November 2011

http://readthedocs.org/docs/getting-started-with-xapian/

www.redwiredesign.com

blog.ilithium.com