Containerizing Couchbase with Swarm & Kubernetes – Couchbase Connect 2016
Using Elasticsearch and Couchbase Together to Build Large Scale Applications
-
Upload
couchbase -
Category
Technology
-
view
8.842 -
download
5
description
Transcript of Using Elasticsearch and Couchbase Together to Build Large Scale Applications
![Page 1: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/1.jpg)
Using&Elas*csearch&and&Couchbase&Together&to&Build&Large&Scale&Apps&
Uri&Boness,&Founder,&Elas*csearch&
Dip*&Borkar,&Director,&Products,&Couchbase&
![Page 2: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/2.jpg)
Introduction to Elasticsearch
![Page 3: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/3.jpg)
What is Elasticsearch?
Open source Apache 2 license•
multi-tenant, realtime anddistributed search & analytics
engine
Backed by Elasticsearch (the company)•
Proven technology in productionOver 2 million downloads
•
![Page 4: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/4.jpg)
What can Elasticsearch do?Unstructured search
find all companies in the “search” market
•
Structured searchfind all companies founded since 2000
•
Analyticsfind the average annual revenue of all companies
•
Combine allfind the average annual revenue of all companies foundedsince 2000 within the “search” market
•
![Page 5: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/5.jpg)
(near) real-time!
![Page 6: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/6.jpg)
Distributed & multi-tenantA node is single Elasticsearch instanceMultiple nodes can form a clusterA cluster can manage multiple indicesA cluster is agile & self managing
continuously ensuring the distributed characteristics of allindices are maintained and that all nodes in the cluster areefficiently & effectively utilized
••••
![Page 7: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/7.jpg)
The Index
![Page 8: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/8.jpg)
What’s in an index?An identified collection of documentsBuilt & designed for small & large scales
data volumesdata can be split and distributed between shards
loads & HAeach shard can have zero or more replicas
••
•
•
![Page 9: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/9.jpg)
starting a node
node_1
![Page 10: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/10.jpg)
creating our first index
node_1
curl -XPUT 'localhost:9200/companies' -d '{ "settings" : { "index" : { "number_of_shards" : 2, "number_of_replicas" : 1 } }}'
![Page 11: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/11.jpg)
the two shards are allocated
node_1
0 1
curl -XPUT 'localhost:9200/companies' -d '{ "settings" : { "index" : { "number_of_shards" : 2, "number_of_replicas" : 1 } }}'
![Page 12: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/12.jpg)
starting a second node
node_1 node_2
0 1
curl -XPUT 'localhost:9200/companies' -d '{ "settings" : { "index" : { "number_of_shards" : 2, "number_of_replicas" : 1 } }}'
![Page 13: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/13.jpg)
shards are relocating
node_1 node_2
0 1
curl -XPUT 'localhost:9200/companies' -d '{ "settings" : { "index" : { "number_of_shards" : 2, "number_of_replicas" : 1 } }}'
![Page 14: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/14.jpg)
replicas are allocated
node_1 node_2
0 11 0
curl -XPUT 'localhost:9200/companies' -d '{ "settings" : { "index" : { "number_of_shards" : 2, "number_of_replicas" : 1 } }}'
![Page 15: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/15.jpg)
Indexing Data
![Page 16: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/16.jpg)
the dataDocuments are typically JSON formatted•
curl -XPUT 'localhost:9200/companies/company/1' -d '{ "id" : "elasticsearch", "name" : "elasticsearch", "website" : "http://www.elasticsearch.com", "category" : "software", "overview" : "distributed search & analytics engine", "founded_year" : 2012, "location" : { "city" : "Amsterdam", "country_code" : "NL", "geo" : { "lat" : 52.370176, "lon" : 4.895008 } }}'
![Page 17: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/17.jpg)
sending req. to one of the nodes
node_3node_1 node_2
0 11 010
client
![Page 18: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/18.jpg)
sending req. to one of the nodes
node_3node_1 node_2
0 11 010
client
resolve the target shard
![Page 19: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/19.jpg)
resolve shard & index to primary
node_3node_1 node_2
0 11 010
client
![Page 20: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/20.jpg)
replicate to replicas
node_3node_1 node_2
0 11 010
client
![Page 21: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/21.jpg)
Searching
![Page 22: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/22.jpg)
unstructured searchUsing an extensive & powerful QueryDSL•
curl -XGET 'localhost:9200/companies/_search' -d '{ "query" : {, "match" : { "overview" : "search" } }}'
![Page 23: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/23.jpg)
unstructured searchUsing an extensive & powerful QueryDSL•
curl -XGET 'localhost:9200/companies/_search' -d '{ "query" : {, "match" : { "overview" : "search" } }}'
search for the term “search” in the “overview”field
![Page 24: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/24.jpg)
structured searchnarrows the “searchable” document space•
curl -XGET 'localhost:9200/companies/company/_search' -d '{ "query" : {, "filtered" : { "query" : { "match" : { "overview" : "search" } }, "filter" : { "range" : { "founded_year" : { "gte" : 2000 } } } } }}'
![Page 25: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/25.jpg)
structured searchnarrows the “searchable” document space•
curl -XGET 'localhost:9200/companies/company/_search' -d '{ "query" : {, "filtered" : { "query" : { "match" : { "overview" : "search" } }, "filter" : { "range" : { "founded_year" : { "gte" : 2000 } } } } }}'
only search companies that were founded since year 2000
![Page 26: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/26.jpg)
returned hits{ ... "hits": [ { "_index": "companies", "_type": "company", "_id": "1", "_score": 0.13424811, "_source": { "id": "elasticsearch", "name": "elasticsearch", "website": "http://www.elasticsearch.com", "category": "software", "founded_year": 2012, "overview": "distributed search & analytics engine", "location": { "city": "Amsterdam", "country_code": "NL", "geo": { "lat": 52.370176, "lon": 4.895008 } } } } ] }}
![Page 27: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/27.jpg)
returned hits{ ... "hits": [ { "_index": "companies", "_type": "company", "_id": "1", "_score": 0.13424811, "_source": { "id": "elasticsearch", "name": "elasticsearch", "website": "http://www.elasticsearch.com", "category": "software", "founded_year": 2012, "overview": "distributed search & analytics engine", "location": { "city": "Amsterdam", "country_code": "NL", "geo": { "lat": 52.370176, "lon": 4.895008 } } } } ] }}
![Page 28: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/28.jpg)
returned hits{ ... "hits": [ { "_index": "companies", "_type": "company", "_id": "1", "_score": 0.13424811, "_source": { "id": "elasticsearch", "name": "elasticsearch", "website": "http://www.elasticsearch.com", "category": "software", "founded_year": 2012, "overview": "distributed search & analytics engine", "location": { "city": "Amsterdam", "country_code": "NL", "geo": { "lat": 52.370176, "lon": 4.895008 } } } } ] }}
![Page 29: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/29.jpg)
returned hits{ ... "hits": [ { "_index": "companies", "_type": "company", "_id": "1", "_score": 0.13424811, "_source": { "id": "elasticsearch", "name": "elasticsearch", "website": "http://www.elasticsearch.com", "category": "software", "founded_year": 2012, "overview": "distributed search & analytics engine", "location": { "city": "Amsterdam", "country_code": "NL", "geo": { "lat": 52.370176, "lon": 4.895008 } } } } ] }}
![Page 30: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/30.jpg)
Query DSLQueries (unstructured)
term queries
boolean queries
phrase (proximity) queries
fuzzy/prefix/regexp/wildcards
more...
Filters (structured)term (exact match)
range
boolean
geo_* (e.g. geo_distance)
•
•
![Page 31: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/31.jpg)
Analytics(a.k.a facets)
![Page 32: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/32.jpg)
Analytics (facets)Slice & dice your dataCompute aggregations over field valuesAcross any index field/sAll in (near) realtime
••••
![Page 33: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/33.jpg)
used as navigation aid
![Page 34: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/34.jpg)
or analytics dashboards
![Page 35: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/35.jpg)
Elasticsearch is often usedpurely for analytics
(without incorporating free text search)
![Page 36: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/36.jpg)
ExampleFind the average revenue of all companies
since 2000•
curl -XGET 'localhost:9200/companies/revenues/_search' -d '{ "query" : { "match_all" : {} }, "facets" : { "revenue_stats" : { "date_histogram" : { "key_field" : "year", "value_field" : "value", "interval" : "month" } } }}'
![Page 37: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/37.jpg)
ExampleFind the average revenue of all companies
since 2000•
curl -XGET 'localhost:9200/companies/revenues/_search' -d '{ "query" : { "match_all" : {} }, "facets" : { "revenue_stats" : { "date_histogram" : { "key_field" : "year", "value_field" : "value", "interval" : "month" } } }}'
return a yearly breakdown of stats over companies revenues
![Page 38: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/38.jpg)
response"facets": { "revenue_stats": { "_type": "date_histogram", "entries": [ { "time": 956448895664, "mean": 23.0 }, { "time": 987984922557, "mean": 267.1034482758621 }, { "time": 1019520942098, "mean": 195.51724137931035 } ... ] } }
![Page 39: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/39.jpg)
response"facets": { "revenue_stats": { "_type": "date_histogram", "entries": [ { "time": 956448895664, "mean": 23.0 }, { "time": 987984922557, "mean": 267.1034482758621 }, { "time": 1019520942098, "mean": 195.51724137931035 } ... ] } }
year 2000
avg revenue
![Page 40: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/40.jpg)
Types of analyticsterms
unique value counts
rangestatistics of specific field for a set of range groups ofanother field
statisticalstats over a specific field
terms_statsstats over a specific fields for every unique field value
date_/histograma breakdown of statistics of a specific field over a
•
•
•
•
•
![Page 41: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/41.jpg)
There’s much moreFine control of how documents are treated
indexed, stored, text analysis, relations
Additional featureshighlighting
suggest API (type ahead, auto-completion)
percolator (reverse search)
support of document relations (parent/child)
extensive geo-location search & analytics
more....
•
•------
![Page 42: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/42.jpg)
Introduc)on*to*Couchbase*
![Page 43: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/43.jpg)
Couchbase*Server*NoSQL*Document*Database*
![Page 44: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/44.jpg)
Couchbase*Open*Source*Project*
• Leading(NoSQL(database(project(focused(on(distributed(database(technology(and(surrounding(ecosystem(
• Supports(both(key;value(and(document;oriented(use(cases(
• All(components(are(available(under(the(Apache*2.0*Public*License*
• Obtained(as(packaged(so?ware(in(both(enterprise(and(community(ediAons.(
Couchbase Open Source Project
![Page 45: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/45.jpg)
Easy*Scalability*
Consistent*High*Performance*
Always*On*24x365*
Grow(cluster(without(applicaAon(changes,(without(downAme(with(a(single(click(
Consistent(sub;millisecond((read(and(write(response(Ames((with(consistent(high(throughput(
No(downAme(for(so?ware(upgrades,(hardware(maintenance,(etc.(
JSONJSONJSON
JSONJSON
PERFORMANCE
Flexible*Data*Model*
JSON(document(model(with(no(fixed(schema.(
Couchbase*Server*
![Page 46: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/46.jpg)
Features*in*Couchbase*Server*2.0*
JSON*support* Indexing*and*Querying*
Cross*data*center*replica)on*Incremental*Map*Reduce*
JSONJSONJSON
JSONJSON
![Page 47: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/47.jpg)
Addi)onal*Features*
Built;in(clustering(–(All(nodes(equal((Data(replicaAon(with(auto;failover((Zero;downAme(maintenance(((Built;in(managed(cached((
((
Append;only(storage(layer((Online(compacAon((Monitoring(and(admin(API(&(UI((SDK(for(a(variety(of(languages(
![Page 48: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/48.jpg)
Couchbase*Server*2.0*Architecture*
Heartbe
at(
Process(mon
itor(
Global(singleton
(sup
ervisor(
Confi
guraAon
(manager(
on(each(node(
Rebalance(orchestrator(
Nod
e(he
alth(m
onitor(
one(per(cluster(
vBucket(state(and(replicaA
on(m
anager(
hQp*RE
ST*m
anagem
ent*A
PI/W
eb*UI*
HTTP(8091*
Erlang(port(mapper(4369*
Distributed(Erlang(21100*Y*21199*
Erlang/OTP*
storage(interface(
Couchbase*EP*Engine*
11210*Memcapable((2.0(
Moxi*
11211*Memcapable((1.0(
Memcached*
New*Persistence*Layer*
8092*Query(API(
Que
ry*Engine*
Data*Manager* Cluster*Manager*
![Page 49: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/49.jpg)
3(3( 2(
Cross*data*center*replica)on*–*Data*flow*2(
Managed(Cache(
Disk(Que
ue(
Disk(
ReplicaAon(Queue(
App(Server(
Couchbase(Server(Node(
Doc*1*Doc*1*
Doc*1*
To(other(node(
XDCR(Queue(
Doc*1*
To(other(cluster(
![Page 50: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/50.jpg)
Cross*Datacenter*Replica)on*(XDCR)*
![Page 51: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/51.jpg)
Couchbase*plugYin*for*Elas)csearch*
![Page 52: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/52.jpg)
How*does*it*work?*
Elas)cSearch*
UnidirecAonal(Cross(Data(Center(ReplicaAon(
![Page 53: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/53.jpg)
ElasAcsearch(IntegraAon((via(XDCR)(
RAM(CACHE(
Doc(1(
Doc(2(
Doc(
Doc(
Doc(
Doc(
Doc(
Doc(
Doc(
Doc(
Doc(
SERVER(1(
Doc(6(
DISK(
RAM(CACHE(
Doc(1(
Doc(2(
Doc(
Doc(
Doc(
Doc(
Doc(
Doc(
Doc(
Doc(
Doc(
SERVER(2(
Doc(6(
DISK(
RAM(CACHE(
Doc(1(
Doc(2(
Doc(
Doc(
Doc(
Doc(
Doc(
Doc(
Doc(
Doc(
Doc(
SERVER(3(
Doc(6(
DISK(
Couchbase(Cluster(West(Coast(Data(Center(
ES(SERVER(1(
ElasAc(Search(Cluster(
ES(SERVER(2( ES(SERVER(3(
Couchbase(Transport(Plugin(
Couchbase(Transport(Plugin(
Couchbase(Transport(Plugin(
![Page 54: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/54.jpg)
Install*the*Couchbase*PlugYIn*• PreYrequisite* ExisAng(Couchbase(and(ElasAcSearch(Clusters(
• Install*the*Elas)cSearch*Couchbase*Transport*PlugYin* bin/plugin(;install((
((((((((((((couchbaselabs/elasAcsearch;transport;couchbase/1.0.0;dp(
• Configure*the*PlugYin* Set(a(password( Install(the(Couchbase(Index(Template(
• Restart*Elas)cSearch*• Create*an*Elas)cSearch*index*for*your*documents*
![Page 55: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/55.jpg)
Configure*Couchbase*XDCR*(step*1)*
![Page 56: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/56.jpg)
Configure*Couchbase*XDCR*(step*2)*
![Page 57: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/57.jpg)
Documents*are*now*indexed*in*Elas)csearch*
Document(Count(Increasing(
![Page 58: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/58.jpg)
Reference*Architecture*
![Page 59: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/59.jpg)
Recommended*Usage*PaQern*
Elas)cSearch*
1.((ElasAcSearch(Query(
2.(ElasAcSearch(Result(
3.(Couchbase(MulA;GET(
4.(Couchbase(Result(
![Page 60: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/60.jpg)
Common*Couchbase*Use*Cases*Social*Gaming*
*• Couchbase(stores(player(and(game(data((
• Examples(customers(include:(Zynga(
• Tapjoy,(Ubiso?,(Tencent(
*
*Mobile*Apps*
*• Couchbase(stores(user(info(and(app(content(
• Examples(customers(include:(Kobo,(PlayAka((
*
*
Ad*Targe)ng**
• Couchbase(stores(user(informaAon(for(fast(access(
• Examples(customers(include:(AOL,(Mediamind,(Convertro((
*
Session*store**
• Couchbase(Server(as(a(key;value(store(
• Examples(customers(include:(Concur,(Sabre(
*
User*Profile*Store**
• Couchbase(Server(as(a(key;value(store(
• Examples(customers(include:(Tunewiki(
*High*availability*cache**
• Couchbase(Server(used(as(a(cache(Aer(replacement(
• Examples(customers(include:(Orbitz(
*
Content*&*Metadata*Store*
• Couchbase(document(store(with(ElasAc(Search(
• Examples(customers(include:(McGraw(Hill(
*
*3rd*party*data**aggrega)on**
*• Couchbase(stores(social(media(and(data(feeds(
• Examples(customers(include:(Sambacloud(
*
![Page 61: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/61.jpg)
Social*Gaming**
• Couchbase(stores(player(and(game(data((
• Examples(customers(include:(Zynga(
• Tapjoy,(Ubiso?,(Tencent(
*
*Mobile*Apps*
*• Couchbase(stores(user(info(and(app(content(
• Examples(customers(include:(Kobo,(PlayAka((
*
*
Ad*Targe)ng**
• Couchbase(stores(user(informaAon(for(fast(access(
• Examples(customers(include:(AOL,(Mediamind,(Convertro((
*
Session*store**
• Couchbase(Server(as(a(key;value(store(
• Examples(customers(include:(Concur,(Sabre(
*
User*Profile*Store**
• Couchbase(Server(as(a(key;value(store(
• Examples(customers(include:(Tunewiki(
*High*availability*cache**
• Couchbase(Server(used(as(a(cache(Aer(replacement(
• Examples(customers(include:(Orbitz(
*
Content*&*Metadata*Store*
• Couchbase(document(store(with(ElasAc(Search(
• Examples(customers(include:(McGraw(Hill(
*
*3rd*party*data**aggrega)on**
*• Couchbase(stores(social(media(and(data(feeds(
• Examples(customers(include:(Sambacloud(
*
![Page 62: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/62.jpg)
RealYworld*example*Couchbase*+*Elas)csearch*
![Page 63: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/63.jpg)
• Content*metadata*• Content:*Ar)cles,*text**• Landing*pages*for*website*• Digital*content:*eBooks,*magazine,*research*material**
Content*and*Metadata*Store*
Use*Case:*Content*and*Metadata*Store*
• Flexibility*to*store*any*kind*of*content*• Fast*access*to*content*metadata*(most*accessed*objects)*and*content**• FullYtext*Search*across*data*set*• Scales*horizontally*as*more*content*gets*added*to*the*system*
• Fast*access*to*metadata*and*content*via*objectYmanaged*cache*• JSON*provides*schema*flexibility*to*store*all*types*of*content*and*metadata*• Indexing*and*querying*provides*realY)me*analy)cs*capabili)es*across*dataset**• Integra)on*with*Elas)cSearch*for*fullYtext*search*• Ease*of*scalability*ensures*that*the*data*cluster*can*be*grown*seamlessly*as*the*amount*of*user*and*ad*data*grows*
Types*of*Data* Applica)on*Requirements*
Why*NoSQL*and*Couchbase**
![Page 64: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/64.jpg)
McGraw*Hill*Educa)on*Labs**Learning*portal*
*
![Page 65: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/65.jpg)
Use*Case:*Content*and*metadata*store*
Building(a(self;adapAng,(interacAve(learning(portal(with(Couchbase(and(ElasAcsearch(
![Page 66: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/66.jpg)
As learning move online in great numbers
Growing need to build interactive learning environments that
Scale!!Scale(to(millions(of(learners(
Serve(MHE(as(well(as(third;party(content(
Including(open(content(
Support(learning(apps(
010100100111010101010101001010101010(
Self;adapt(via(usage(data(
The Problem*
![Page 67: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/67.jpg)
• Allow(for(elasAc(scaling(under(spike(periods(
• Ability(to(catalog(&(deliver(content(from(many*sources*
• Consistent(lowYlatency*for(metadata(and(stats(access(
• Require(fullYtext*search*support(for(content(discovery(
• Offer(tunable(content(ranking(&(recommendaAon(
funcAons((
Backend is an Interactive Content Delivery Cloud that must:
XML(Databases(
SQL/MR(Engines(
In;memory(Data(Grids(
Enterprise(Search(Servers(
Experimented with a combination of:
The Challenge*
![Page 68: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/68.jpg)
![Page 69: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/69.jpg)
• Document(Modeling(
• Metadata(&(Content(Storage(
• View(Querying(to(support(Content(Browsing(• ElasAcsearch(IntegraAon((; Content(Updated(in(near(Real;Time(
; Search(Content(Summaries(
; Relevancy(boosted(based(on(User(Preferences(• Real;Time(Content(Updates(
• Event(Logging(for(offline(analysis(
Techniques*Used*
![Page 70: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/70.jpg)
Couchbase*2.0*****+******Elas)csearch*
Store(full-text articles(as(well(as(document metadata(for(image,(video(and(text(content(in(Couchbase(
Combine(user(preferences(staAsAcs(with(custom relevancy scoring(to(provide(personalized search results
Logs(user behavior(to(calculate(user(preference(staAsAcs((e.g.(video(>(text)(
1(
2( 4(
ConAnuously(accept updates from(Couchbase(with(new(content(&(stats(
3(
![Page 71: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/71.jpg)
Data(Model(
Content Metadata Bucket
User Profiles Bucket
Content Stats Bucket
• Stores content metadata for media objects and content for articles
• Includes tags, contributors, type information
• Includes pointer to the media
• Stores user view details per type • Updated every time a user views
a doc with running count • To be used for customizing ES
search results per user preference
• Stores content view details • Updated for every time a
document is viewed • To be used for boosting ES
search results based on popularity
![Page 72: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/72.jpg)
Couchbase Views
Top Contributors & Tagsdriven by Incremental MapReduce Views!
Calcula)ng*sta)s)cs*via*Couchbase*
![Page 73: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/73.jpg)
Tuning(content(ranking(via(
ElasticSearch
ElasticSearch-driven based on settings below!
Content popularity boost!
User preference boost!
![Page 74: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/74.jpg)
{ "_id": �4ae5be2df3122f06ba45b70753001841�,
�_rev�: �1-0013b349ffc3afc700000000068000000�, �$flags�: 0, �#expiration�: 0, �type�: �access�, �user�: �[email protected]�, �resource�: �379823�, �timestamp�: �2012-09-02T22:46:07Z�
}
{ "_id": �4ae5be2df3122f06ba45b70753001842�,
�_rev�: �1-0013b349ffc3afc700000000068000000�, �$flags�: 0, �#expiration�: 0, �type�: �create�, �user�: �[email protected]�, �resource�: �948177�, �timestamp�: �2012-09-02T22:48:32Z�
}
What?!
Who?!
Which?!
When?!
Analy)cs*and*Event*Logging*
• Store*full*event*log*for*offline*analysis*
• Stored*on*a*separate*analy)cs*cluster**
• Limit*impact*on*OLTP*
• Tuned*differently*
• Keep*an*upperYbound*on*data*size*via*TTL*(24*hrs)*
![Page 75: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/75.jpg)
{ "filter": { "term": { "type": "video� } }, "boost": USER_VIDEO_PREFERENCE * PREFERENCE_SLIDER }
User*Preference*Boost*
• Use*Elas)csearch*filter*boos)ng*
![Page 76: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/76.jpg)
"script": "_score * (((doc['popularity'].value + 1) / AVG_POPULARITY ) * POPULARITY_SLIDER)"
Document Popularity Boost*
• Use*Elas)csearch*custom*script*to*score*documents*
![Page 77: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/77.jpg)
"filters": [ { "filter": { "term": { "type": "video" } }, "boost": USER_VIDEO_PREFERENCE * PREFERENCE_SLIDER }, … image and texts filters omitted … ], "score_mode": "total" } }, "script": "_score * (((doc['popularity'].value + 1) / AVG_POPULARITY ) * POPULARITY_SLIDER)" }
Combined Algorithm in a Query*
![Page 78: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/78.jpg)
The Learning Portal*
• Designed and built as a collaboration between MHE Labs and Couchbase
• Serves as proof-of-concept and testing harness for Couchbase + Elasticsearch integration
• Available for download and further development as open source code
h"ps://github.com/couchbaselabs/learningportal5
![Page 79: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/79.jpg)
Q*&*A*
![Page 80: Using Elasticsearch and Couchbase Together to Build Large Scale Applications](https://reader033.fdocuments.in/reader033/viewer/2022042715/558d1504d8b42a7f248b45b6/html5/thumbnails/80.jpg)
Thank*you*
*******
dip)@couchbase.com*uri.boness@elas)csearch.com*
****