Post on 08-Jan-2020
Improving Drupal search experience with Apache Solr and Elasticsearch
Milos Pumpalovic – Web Front-end DeveloperGene Mohr – Web Back-end Developer
Improving Drupal search experience with Apache Solrand Elasticsearch
Improving Drupal search experience with Apache Solr and Elasticsearch
About Us
2
MilosPumpalovic• FrontEndDeveloper• Drupalthemingandmoduledevelopment• mip2047@med.cornell.edu
GeneMohr• BackEndDeveloper• FullstackandDrupalmoduledevelopment• bas2019@med.cornell.edu
Improving Drupal search experience with Apache Solr and Elasticsearch
What is search engine and why we use it?
3
SearchengineinDrupal
WhatarethedifferencesfromDrupalSearchAPI?
Whyweuseit?
Improving Drupal search experience with Apache Solr and Elasticsearch
What is Solr?
4
Solr isasuperfast,opensource,standalonesearchserverbuiltonApacheLucene.Someofthefeaturesare:• AdvancedFull-TextSearch• DesignedforHighVolumetraffic• FacetedSearchandFiltering• Scalable,flexible,extensible• RichDocumentParsing– built-inabilitytoindexPDF’s,Worddocumentsandmore• Multiplesearchindexes• QuerySuggestions,SpellingandMore– advancedcapabilityforauto-complete,spell
checking,highlightingandmore
Improving Drupal search experience with Apache Solr and Elasticsearch
Apache Solr Configuration
5
• Installmodule• CreateanindexonSolr engine• Createaview• Attachfacetfilters
Improving Drupal search experience with Apache Solr and Elasticsearch
Install the Drupal modules
6
$composerconfig repositories.drupal composerhttps://packages.drupal.org/8
Downloadthemodule SearchAPISolrSearch.
$composerrequiredrupal/search_api_solr
InDrupal8,ComposerisusedtomanagePHPlibraries.TheSearchAPImoduledoesn’tincludeSolarium,sodependenciesneedtobesetviaComposerwiththefollowingcommands:
Install the following modules:
Optional:installSolr SearchDefaultsmodule
Improving Drupal search experience with Apache Solr and Elasticsearch
Add Solr server
7
Goto Configuration->Searchandmetadata->SearchAPI.Click“AddServer”andconfiguretheServer.Servernameexample:LocalSolr Server
Improving Drupal search experience with Apache Solr and Elasticsearch
Confirm Solr server is connected
8
Atthisstage,theserver’sstatuspageshouldshowmessage:"TheSolr servercouldbereached"and"TheSolr corecouldbeaccessed"
IfSolr SearchDefaultsmodulewasinstalled,itcanbeuninstalledasit’snolongerrequired.
Improving Drupal search experience with Apache Solr and Elasticsearch
Add search index
9
Goto Configuration->Searchandmetadata->SearchAPI.Click“AddIndex”
Selectingthe“Content”datasource,optionsarepresentedtoselectwhichbundlesaretobeindexed
Improving Drupal search experience with Apache Solr and Elasticsearch
Add fields to index
10
Beforesearchcanbeperformed,selectallthefieldsthatshouldbeavailabletosearch.Thatisconfiguredinthe“Fields”tab.
Improving Drupal search experience with Apache Solr and Elasticsearch
Add processors to the index
11
Laststepistoaddadditional’processors’.
Thisincludesitemssuchas:• Contentaccess• Ignorecase(case-insensitivesearch)• Tokenizer(splitintoindividualwords)
Improving Drupal search experience with Apache Solr and Elasticsearch
Verify search index is working
12
Oncefieldsandprocessorsaresetup,goingbacktothe”View”tab,willshowthestatusoftheindex,andatthispoint,thecontentisreadytobeindexedifnotalreadysettoindeximmediatelywhentheindexiscreated.
Indexingofcontentisdoneviacron andanynewcontentwillgetindexedthen.
Improving Drupal search experience with Apache Solr and Elasticsearch
Create a Faceted Solr Search page
13
Finalstepistocreateasearchpage.ThisisdonebycreatingaViewandcreateaPage.
Generalsettingstoconfigure:ViewSettings:Show>Index(createdearlier)PageSettings:• PageTitle• Path:/search/content• DisplayFormat>Unformattedlistof“RenderedEntity”
Theclick“Saveandedit”toconfiguretheview.1. UnderFormat>Show,click“Settings”forRendered
Entityandchange”viewmode”toTeaser2. UnderFilterCriteria,add“Fulltext search”fieldand
exposethefieldforfiltering
Furthersolr configurationcanbedoneintheSolr schema,suchas:
• Searchingpartsofworks,usingSolr’s EdgeNGramFilterFactory,forexample,”info”willmatch”information”
• LowerCaseTokenizer– removewhitespaceandnon-lettersandconvertallletterstolowercase
Improving Drupal search experience with Apache Solr and Elasticsearch
Add search facets
14
Withthesearchpagesetupnow,wewanttoaddfacetstoletusersfilterdowncontent.NavigatetoConfiguration>Searchandmetadata>Facetsthenclick“Addfacet”
Facetshaveanumberofsettingstoconfigure:- Widget- Showtheamountofresults- Sorting(bycount,displayvalue…)- Operator(ORandAND- ANDfiltersare
exclusiveandnarrowtheresultset.ORfiltersareinclusiveandwidentheresultset.)
- …
Improving Drupal search experience with Apache Solr and Elasticsearch
Add search facets
15
Laststep,addthenewlycreatedFacetblocksontheBlockLayoutpage
Note:Facetblocksdon’tneedtohavethepagevisibilityset,itwillautomaticallydetectifthepageasearchpage,otherwise,theywillnotbedisplayed
Improving Drupal search experience with Apache Solr and Elasticsearch
Search Page
16
Finally,viewthesearchpagecreated
Improving Drupal search experience with Apache Solr and Elasticsearch
What is Elasticsearch?
17
• Alsocapableofprovidingadvancedfull-textsearch• Easiertoinstall• RESTbasedAPIs,asimpleHTTPinterface,andusesschema-freeJSON• DistributedDocumentStore- easytouseJSONdocument-orientedstorage
platform• Loggingdataandanalysis• Visualizationandreal-timedatamonitoring whencoupleofKibana
Improving Drupal search experience with Apache Solr and Elasticsearch
Elasticsearch Configuration
18
• Installmodule• CreateanindexonElasticsearch engine• Createaview• Attachfacetfilters
Improving Drupal search experience with Apache Solr and Elasticsearch
Install the Drupal modules
19
$composerrequirenodespark/des-connector:5.x-dev$composerupdate
TheSearchAPImoduleneeds Elasticsearch PHPlibrarywhichprovidestheabstractlayerofElasticsearchConnectormoduleinDrupal. Thiscanbeinstalledthroughcomposer.
Install the following modules:
NOTE:Elasticsearch doesn’thaveSolr SearchDefaults-likemodule.Settingupandcreatingaviewpagehastobedonemanually.
Improving Drupal search experience with Apache Solr and Elasticsearch
Add Elasticsearch
20
Goto Configuration>Searchandmetadata>Elasticsearch Connector.
Click“AddCluster”andconfiguretheServer.
Improving Drupal search experience with Apache Solr and Elasticsearch
Add Elasticsearch (continue)
21
Goto Configuration>Searchandmetadata>Elasticsearch Connector.Click“AddCluster”andconfiguretheserver.
Asdefault,itis“elasticsearch.”Ifyouwanttoeditthecluster/nodeinformation,editelasticsearch.ymlfile.
Improving Drupal search experience with Apache Solr and Elasticsearch
Add search index (same as Solr)
22
Goto Configuration>Searchandmetadata>SearchAPI.Click“AddIndex”
Selectingthe“Content”datasource,optionsarepresentedtoselectwhichbundlesaretobeindexed
Improving Drupal search experience with Apache Solr and Elasticsearch
Add fields to index (same as Solr)
23
Beforesearchcanbeperformed,selectallthefieldsthatshouldbeavailabletosearch.Thatisconfiguredinthe“Fields”tab.
Improving Drupal search experience with Apache Solr and Elasticsearch
Add processors to the index (same as Solr)
24
Laststepistoaddadditional’processors’.
Thisincludesitemssuchas:• Contentaccess• Ignorecase(case-insensitivesearch)• Tokenizer(splitintoindividualwords)
Improving Drupal search experience with Apache Solr and Elasticsearch
Verify search index is working (same as Solr)
25
Oncefieldsandprocessorsaresetup,gobacktothe”View”tab.Itwillshowthestatusoftheindex,andatthispoint,thecontentisreadytobeindexedifnotalreadysettoindeximmediatelywhentheindexiscreated.
Indexingofcontentisdoneviacron andanynewcontentwillgetindexedthen.
Improving Drupal search experience with Apache Solr and Elasticsearch
Create a view page
26
1.GotoStructure>Addview
2.Provideaviewnameandselectyourindexnameastheviewsource
Improving Drupal search experience with Apache Solr and Elasticsearch
Create a view page (continue)
27
3.UnderFormat>Show,selectRenderedEntity
Or,youcanselect“Fields”andaddeachfieldyouwouldliketodisplayintheFieldssection.
4.UnderFilterCriteria,add“Fulltext search”fieldandexposethefieldforfiltering
5.AddSortCriteria:Thebestonetouseis“Relevance(desc)”
Improving Drupal search experience with Apache Solr and Elasticsearch
Add search facets (same as Solr)
28
Withthesearchpagesetupnow,wewanttoaddfacetstoletusersfilterdowncontent.NavigatetoConfiguration>Searchandmetadata>Facetsthenclick“Addfacet”
Facetshaveanumberofsettingstoconfigure:• Widget• Showtheamountofresults• Sorting(bycount,displayvalue…)• Operator(ORandAND- ANDfiltersare
exclusiveandnarrowtheresultset.ORfiltersareinclusiveandwidentheresultset.)
Improving Drupal search experience with Apache Solr and Elasticsearch
Place facet blocks (same as Solr)
29
Laststep,placethenewlycreatedFacetblocksontheBlockLayoutpage
Note:Facetblocksdon’tneedtohavethepagevisibilityset,itwillautomaticallydetectEachofthepageasearchpage,otherwise,theywillnotbedisplayed.
Improving Drupal search experience with Apache Solr and Elasticsearch
Search Page
30
Finally,viewthesearchpagecreated.
Improving Drupal search experience with Apache Solr and Elasticsearch 31
Apache Solr or Elasticsearch?
Source:AscenefromAnnieGetYourGun(1950)
Improving Drupal search experience with Apache Solr and Elasticsearch
When to use Apache Solr
32
• Iftherearefewthousandnodesofcontent(ormore,dependingonserverperformanceandwhetherhostedlocallyorexternally)
• Youdonotwanttoputstressonthedatabase
• Ifsearchingisamajorfeatureofthesite
• Ifyouneedtosearchmultiplefields
Improving Drupal search experience with Apache Solr and Elasticsearch
When to use Elasticsearch
33
• TheElasticStack(Elasticsearch,Logstash,andKibana)caninteractivelysearch,discover,andanalyzetogaininsightsthatimprovetheanalysisoftime-seriesdata.
• Noneedforupfrontschemadefinition.Schemacanbedefinedpertypeforcustomizationofindexingprocess.
• Hasanedgeinthecloudenvironment- thisisdependuponSolrCloudadvancement.
• Hasadvantagesofsearchforenterpriseorhigher-ed levelwhereanalyticsplaysabiggerrole.
Improving Drupal search experience with Apache Solr and Elasticsearch
Additional resources
34
Solr:ConfiguringtheSearchAPIserver(D8)https://www.drupal.org/node/2763137
Drupal8andSolr:Google-fastsearchonyourownwebsite|Whyandhowhttps://blog.openlucius.com/en/blog/drupal-8-and-solr-google-fast-search-your-own-website-why-and-how
SetupfacetedApacheSolr searchonDrupal8https://www.jeffgeerling.com/blog/2016/set-faceted-apache-solr-search-on-drupal-8-2016-deprecated
WhentoconsiderApacheSolrhttps://lastcallmedia.com/blog/when-consider-apache-solr
Elasticsearch:IndexingcontentfromDrupal8usingElasticsearchhttps://www.lullabot.com/articles/indexing-content-from-drupal-8-to-elasticsearch
HowtoindexAttachmentsandFilestoElasticsearchhttps://qbox.io/blog/index-attachments-files-elasticsearch-mapper
---------------
Solr vs.Elasticsearch:5FactorstoConsiderhttps://www.cmswire.com/information-management/solr-vs-elasticsearch-5-factors-to-consider/
Improving Drupal search experience with Apache Solr and Elasticsearch
Questions?
35