Gregg Donovan @Lucene Revolution 2011
-
Upload
lucidimagination -
Category
Documents
-
view
217 -
download
0
Transcript of Gregg Donovan @Lucene Revolution 2011
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
1/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
2/84
1.5 years Solr & Lucene at Etsy.com
3 years Solr & Lucene at TheLadders.com
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
3/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
4/84
8+ million members
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
5/84
9.3 million items
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
6/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
7/84
1+ billion pageviews / month
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
8/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
9/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
10/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
11/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
12/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
13/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
14/84
Maximize Solr out-of-the-box
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
15/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
16/84
Know when to do each
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
17/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
18/84
Or
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
19/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
20/84
Dont fear trunk
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
21/84
builds.apache.org/job/Solr-trunk/changes
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
22/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
23/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
24/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
25/84
{!lucene}
{!field}
{!func}
{!dismax}
{!edismax}
{!boost}
{!term}
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
26/84
Cheap ranking awesomeness
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
27/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
28/84
ExternalFileField ftw!
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
29/84
schema.xml:
/search/data/treasury/external_hotness.1306390802088:
1=2.32=1.7
3=1.1
Solr query:
sort={!func}hotness+desc
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
30/84
ExternalFileField caveats
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
31/84
More relevance: boost query
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
32/84
http://localhost:8983/solr/listings/select?q={!boost b=$rel v=$qq}
&rel=category:furniture^10+OR+((-material:acrylic)
^5)
&qq=desk
http://livepage.apple.com/http://livepage.apple.com/http://livepage.apple.com/http://livepage.apple.com/ -
8/6/2019 Gregg Donovan @Lucene Revolution 2011
33/84
Impression tracking
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
34/84
etsy.com/search?q=desk&explain=1
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
35/84
Side-by-Side testing
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
36/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
37/84
Cheap performance wins
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
38/84
Put off sharding till you must
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
39/84
cat ${indexDir}/* > /dev/null
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
40/84
Return IDs, minimize stored fields
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
41/84
RAM: $10-20 / GB
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
42/84
SSD: 0.1ms vs 10ms seek
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
43/84
Custom?
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
44/84
solr-user
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
45/84
Tools for low-level hacking
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
46/84
Continuous deployment
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
47/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
48/84
One button.So easy a dog could do it.
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
49/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
50/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
51/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
52/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
53/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
54/84
github.com/etsy/logster
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
55/84
Tracking GC
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
56/84
export GC_DEBUG="-verbose:gc -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:
+PrintGCApplicationStoppedTime -XX:+PrintAdaptiveSizePolicy -
XX:AdaptiveSizePolicyOutputInterval=1 -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -Xloggc:/var/log/search/gc.log"
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
57/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
58/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
59/84
Alerting
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
60/84
Testing
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
61/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
62/84
SaveAsFixture
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
63/84
Profiling
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
64/84
Java Primitive Libraryfastutil
trove4j
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
65/84
Know the hooks
QParserPlugin
SolrEventListener
SolrRequestHandler
SearchComponent
SolrCache
ValueSourceParser
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
66/84
SolrIndexSearcher gotchasreference countingusing it as a cache key:
WeakHashMap myCache...
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
67/84
Example:personalized collections
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
68/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
69/84
fq={!term f=id}123 OR {!term f=id}456
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
70/84
Need a map of PK to docId
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
71/84
Use custom SolrCache plus SolrEventListener
to fill it
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
72/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
73/84
i18n currency sorting and filtering
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
74/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
75/84
currency.xml:
! ! !
! !
! !
! !
...
!
!
! !
! !
! !
! !
...
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
76/84
price:[10.00USD to 50.00USD]
price:20.00EUR
price:[$10.00 to $50.00]
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
77/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
78/84
Replication gotcha
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
79/84
SOLR-2202
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
80/84
Related Searches
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
81/84
Autosuggest!
bjewlery dewelry ejewelry ejwelry ewelery ewerly ewlery fewelryfewlery fjewelery fjewelry gewerly gewlery hewelery hewelry hewerly
h l hj l i l ij l j l j l j l j l
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
82/84
hewlery hjewelry iewelry ijewelry jawelery jawlery jeawlery jeelery
jeelryjeewelery jeewelry jeewlery jeewlry jefwelry jejelry jelelry
jelery jellery jelwelery jelwelry jelwlery jemelry jemerly jemwelry
jeqwelry jerelery jerelry jerely jererly jerlery jerwelery jerwelry
jerwely jerwerlyjeselery jeselry jevelry jeverly jewalery jewdelryjewedlry jeweelrry jeweelry jeweely jeweer jeweery jeweilry jeweiry
jewejery jewejlry jewejrly jewejry jewekey jewekry jewelary jeweldy
jewele jewelee jewelelryjewelera jewelerey jewelerly jewelert
jewelerty jeweleru jeweleruy jeweleryl jewelerys jeweleryy jewelet
jewelety jeweleya jewelfry jewelfy jeweliy jewellryp jewelltry
jewelly jewelory jewelra jewelray jewelrejewelree jewelreyyjewelrfy jewelrh jewelri jewelrky jewelrly jewelrr jewelrs jewelrsy
jewelrt jewelrty jewelru jewelruy jewelrye jewelryh jewelryl
jewelrym jewelryr jewelrys jewelryt jewelryu jewelryuk
jewelryyjewelrz jewelsry jewelsy jeweltry jewelty jewelw jewelwery
jewelwey jewelwy jewelya jewelyj jewelyr jewelyry jewelyu jewelyy
jewelzry jeweory jewerey jeweriy jewerky jewerlary jewerley jewerlijewerlly jewerlsjewerlt jewerlu jewerlyh jewerlyr jewerlys jewerlyu
jewerry jeweryl jewetry jewewlry jewewly jewewrly jewewry jeweylry
jewiery jewilary jewkery jewlary jewledy jewleery jewlelery jewlely
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
83/84
-
8/6/2019 Gregg Donovan @Lucene Revolution 2011
84/84