A sneak peek at CBFT: A Full Text Search for Couchbase: Couchbase Connect 2015

A SNEAK PEEK AT CBFT

Couchbase Full-Text ServerMarty Schoch & Steve Yen, Couchbase, Inc.

©2015 Couchbase Inc. 2

about the speakers

Steve [email protected] Couchbase

mailto:[email protected]


about the speakers

Marty Schochlead contributor to bleve

the most popular, open-source

full-text indexing engine

for golang


agenda

why cbft?what’s full-text search and how’s it work?designdemostatus / roadmap / what’s next


agenda



why cbft?

couchbase connectors…

yes

yes

Lucidworks yes


why cbft?

couchbase connectors… yet another tier &

cluster to manage

yesyes

yesyes

Lucidworks yesyes


why cbft?

why cbft?

simpleintegrated

80/20 of features


agenda



what’s full-text search?


advanced search


search results


search results SpellingSuggestions



Result TextSnippets



Result TextSnippets

HighlightedSearch Terms


faceted search


JSON document in Couchbase

Key: akay1980

Document: {

“name”: “Alan Kay”, “description”: “... the wisest

engineer ...” }


Text Analysis : tokenizer + token filters

A pipeline of transformations

One Tokenizer

Zero or more Token Filters


“… the wisest engineer …”

thewises

tenginee

r• Seems like simple whitespace… but, this doesn’t work for

all languages• Unicode standard rules help (see Unicode Standard Annex

#29)• Still need to account for exceptions

• E-mail addresses and URLs don’t follow normal rules




thewises

tengineer

Stop WordRemoval the

wisest

engineer

Stemming wise engineer


Inverted Index

…

wise

…

engineer…

…

…

…, akay1980, …

…, akay1980, …

Inverted Index


Search

…

wise

…

engineer…

…

…

…, akay1980, …

…, akay1980, …

engineersInverted Index


Search

…

wise

…

engineer…

…

…

…, akay1980, …

…, akay1980, …

engineers

engineer

Apply the same analysis at search time that we used at index time.

Inverted Index


Search

…

wise

…

engineer…

…

…

…, akay1980, …

…, akay1980, …

engineers

engineer

Exact Match

Apply the same analysis at search time that we used at index time.

Inverted Index


Document Scoring

• tf/idf scoring• Term Frequency• How often does a term occur in

a doc?• More often yields a higher score

• Inverse Document Frequency• How many docs have this term?• More docs yield lower score

(because the term is more common)


Quality Results

• Getting high quality results depends on the right text analysis

• Beware: adjustments that increase precision may reduce recall (and the other way around)


agenda



cbft design / index partitioning



bucket partitions: 0, 1, 2, 3, 4, … … ,1021, 1022, 1023 (1024 vbuckets)




index partitions: A B C





(groups of vbuckets) 0-399 400-799 800-1023






cbft nodes:

X






assign to cbft nodes:

cbft nodes:

X






assign to cbft nodes:

cbft nodes:

X Y Z






assign to cbft nodes:replicas, too:

cbft nodes:

X Y Z


cbft design / indexing

couchbase couchbase couchbase

cbft cbft cbft

DCP streams


cbft design / indexing


cbft cbft cbft

DCP streams


cbft design / queries

cbft cbft

a query sentto any cbftnode…

your application

cbftR

ES

T


cbft design / queries

cbft cbft

a query sentto any cbftnode…

…is scatter / gathered

to the other cbft nodes

your application

cbftR

ES

T


agenda



project status

cbft is developer preview!

please help kick the tires

http://labs.couchbase.com/cbft



project status / roadmap / what’s next

today

bleve full-text engine yadvanced mappings yfaceted search y

incremental indexing y

index partitioning and replication y

index aliasesy


project status / roadmap / what’s next

today future

bleve full-text engine yy

advanced mappings yy

faceted search yy

incremental indexing yy

index partitioning and replication yy

index aliasesy y

integrated into Couchbase Server & N1QLy

API stabilityy

production qualityy

performance optimization / tuningy

forestdb storage & partial rollbacks y

security, SSLy

more docs, examples, SDK supporty


links & Q+A

http://labs.couchbase.com/cbftdownloads, getting started, tech docs

and, share your feedback!

THANKS! (and please do the survey!)


A SNEAK PEEK AT CBFT

couchbase full-text server

THANKS! (and please do the survey!)


cbft design


cbft cbft cbft

cfg

DCP streamsfor incrementalindex updates

a cfg bucketholds metadata

about the indexes

A sneak peek at CBFT: A Full Text Search for Couchbase: Couchbase Connect 2015

Technology

Transcript of A sneak peek at CBFT: A Full Text Search for Couchbase: Couchbase Connect 2015