Post on 18-Dec-2015
VIVO 1.8SearchIndexer
http://gist.github.com/j2blake/388cbc50efb611481698
IntroductionEfficient
Configurable
Visible
Maintainable
OutlineTheory
Flow
Modularity
Configuration
Performance
Testing
Questions
Task Force
TheoryAdjust automatically to changes in the model.
User experience does not require that updates are synchronous.
When a triple is changed, several search records may require updating.
Several altered triples may affect the same search record.
Each search record can be updated independently.
TheoryRespond to search Indexer admin page: request
a rebuild.
Respond to search indexing service API: specify a list of URIs to be re-indexed.
FlowRespond to changes in the Abox
From the list of altered triples, build a list of URIs that require re-indexing.
Respond to request for rebuildBuild a list of all URIs in the model and re-index.
Respond to the search indexing serviceAccept a list of URIs for re-indexing.
Flow – asynchronous
Update URIs Task
Rebuild Index Task
Update Statements
Task
Work Unit
Work Unit
Work Unit
Task Queue Thread Pool
Search Indexing Service
Search Indexer Admin page
Change to ABox model
Flow – update URIsAccept a list of URIs to be indexed.
Determine the eligible URIs from the list.Must be an existing individual with at least one
VClass (type).Must pass the list of SearchIndexExcluders.
Remove the ineligible URIs from the index.
Build a search document for each eligible URI.Execute the list of DocumentModifiers.Add the completed document to the index.
Flow – update URIs
Flow – update URIs
Flow – rebuild indexGet a list of all individuals in the ABox.
Update these URIs.(use the existing logic to update URIs)
Remove obsolete records from the index.Anything that was indexed prior to the rebuild.
Flow – rebuild index
Flow – rebuild index
Flow – change to ABoxAccumulate changes into a batch
Delimited by a quiescent interval.Optional delimiting calls to pause() and unpause()
Produce a set of URIs to be updatedEach statement is examined by the list of
IndexingUriFinders.
Update these URIs.(use the existing logic to update URIs)
Flow – change to ABox
Flow – change to ABox
ModularityApplication.java
SearchIndexer.java
ConfigurationBeanLoader.java
[vivo-home]/config/applicationSetup.n3Read at runtime.
Application.javapublic interface Application { ServletContext getServletContext(); VitroHomeDirectory getHomeDirectory();
SearchEngine getSearchEngine(); SearchIndexer getSearchIndexer();
ImageProcessor getImageProcessor(); FileStorage getFileStorage();
ContentTripleSource getContentTripleSource(); ConfigurationTripleSource getConfigurationTripleSource();
TBoxReasonerModule getTBoxReasonerModule();
void shutdown();}
https://gist.githubusercontent.com/j2blake/388cbc50efb611481698/raw/c25ef7c0638f46473e935fec86545c689ffd4fc9/Application.java
SearchIndexer.javapublic interface SearchIndexer extends Application.Module { void startup(Application app, ComponentStartupStatus ss); void shutdown(Application app);
void pause(); void unpause();
void addListener(Listener listener); void removeListener(Listener listener);
void scheduleUpdatesForStatements(List<Statement> changes); void scheduleUpdatesForUris(Collection<String> uris); void rebuildIndex();
SearchIndexerStatus getStatus();}
https://gist.githubusercontent.com/j2blake/388cbc50efb611481698/raw/a1ffb6330dec0010cdff0ce283178a57b1fb1048/SearchIndexer.java
ConfigurationBeanLoaderpublic ConfigurationBeanLoader(Model model);
/** * Load the instance with this URI, * if it is assignable to this class. */public <T> T loadInstance(String uri, Class<T> resultClass)
throws ConfigurationBeanLoaderException;
/** * Find all of the resources with the specified class, * and instantiate them. */public <T> Set<T> loadAll(Class<T> resultClass)
throws ConfigurationBeanLoaderException;
ApplicationImpl.java@Overridepublic SearchIndexer getSearchIndexer() {
return searchIndexer;}
@Property(uri = "http://vitro.mannlib.cornell.edu/ns/vitro/ApplicationSetup#hasSearchIndexer")public void setSearchIndexer(SearchIndexer si) {
searchIndexer = si;}
@Validationpublic void validate() throws Exception {
if (searchIndexer == null) {throw new IllegalStateException("Configuration did not include a
SearchIndexer.");}
}
applicationSetup.n3@prefix : <http://vitro.mannlib.cornell.edu/ns/vitro/ApplicationSetup#> .
:application a <java:edu.cornell.mannlib.vitro.webapp.application.ApplicationImpl> , <java:edu.cornell.mannlib.vitro.webapp.modules.Application> ; :hasSearchEngine :instrumentedSearchEngineWrapper ; :hasSearchIndexer :basicSearchIndexer ; :hasImageProcessor :jaiImageProcessor ; :hasFileStorage :ptiFileStorage ; :hasContentTripleSource :sdbContentTripleSource ; :hasConfigurationTripleSource :tdbConfigurationTripleSource ; :hasTBoxReasonerModule :jfactTBoxReasonerModule .
# ...
:basicSearchIndexer a <java:edu.cornell.mannlib.vitro.webapp.searchindex.SearchIndexerImpl> , <java:edu.cornell.mannlib.vitro.webapp.modules.searchIndexer.SearchIndexer> ; :threadPoolSize "10" .
https://gist.githubusercontent.com/j2blake/388cbc50efb611481698/raw/7b009909ee8f812366cde14f3c1253ded514f85e/applicationSetup.n3
ConfigurationUriFinders
What URIs are affected by an altered triple?
ExcludersWhat URIs should not have documents in the
index?
DocumentModifiersWhat data belongs in the search document?
ConfigurationRDF in the display model
[vitro]/webapp/rdf/display/everytime/searchIndexerConfigurationVitro.n3 https://gist.githubusercontent.com/j2blake/388cbc50efb611481698/raw/499
0dc99f6a686ac501dc8f8b68808b20759657e/searchIndexerConfigurationVitro.n3
[vivo]/rdf/display/everytime/searchIndexerConfigurationVivo.n3 https://gist.githubusercontent.com/j2blake/388cbc50efb611481698/raw/904
d3457aeeaade03cb5515ef7856c010ff774f4/searchIndexerConfigurationVivo.n3
Read at startup
Configuration - examples:searchExcluder_typeExcluder a <java:edu.cornell.mannlib.vitro.webapp.searchindex.exclusions.ExcludeBasedOnType> , <java:edu.cornell.mannlib.vitro.webapp.searchindex.exclusions.SearchIndexExcluder> ; :excludes "http://www.w3.org/2002/07/owl#AnnotationProperty" , "http://www.w3.org/2002/07/owl#DatatypeProperty" , "http://www.w3.org/2002/07/owl#ObjectProperty" .
:uriFinder_forDataProperties a <java:edu.cornell.mannlib.vitro.webapp.searchindex.indexing.IndexingUriFinder> , <java:edu.cornell.mannlib.vitro.webapp.searchindex.AdditionalURIsForDataProperties> .
:documentModifier_NameFieldBooster a <java:edu.cornell.mannlib.vitro.webapp.searchindex.document.FieldBooster> , <java:edu.cornell.mannlib.vitro.webapp.searchindex.document.DocumentModifier> ; :hasTargetField "nameRaw" ; :hasTargetField "nameLowercase" ; :hasTargetField "nameUnstemmed" ; :hasTargetField "nameStemmed" ; :hasBoost "1.2"^^xsd:float .
PerformanceSimple efficiency:
Don’t ask for more information than we need.Don’t discard information we have obtained.
Multi-threading:Finding URIs for a collection of altered statements.
Need to remove duplicate URIs.Building the search records for a collection of URIs.
Performance – memoryKeeping the memory footprint low:
A list of URIs One URI is likely to be ~100 bytes.100,000 URIs is likely to be ~10 megabytes.
A list of Individuals.One Individual may be ~100 kilobytes.50,000 Individuals may be ~5 gigabytes.
Performance - timingContinuous improvement: timings from the
developer panel:
Performance - timinghttps://gist.githubusercontent.com/j2blake/388c
bc50efb611481698/raw/26000d6793bef1bdb0d2ff9a885000caf175a619/vivo.all.log
TestingAdd to the battery of Selenium tests
For example:Create a person, and a book written by that person.Search for the person’s name: both the person and
the book should be returned.Search for the title of the book: both the person and
the book should be returned.
A continuing effortSo far, not very successful.
QuestionsOther configurable document modifier classes?
Other configurable excluder classes?
What about configurable URI finder classes?
QuestionsBoost – multiplicative would make it order-
independent. Do we care?
Private data – the SPARQL-based DocumentModifiers do not use a filtered connection. Is that a problem?
At what point does it make sense to do a full rebuild instead of responding to model changes?Finding URIs can be more expensive than building
search documents.
QuestionsWhat should be the default configuration?
Task ForceDon Ellsborg and I have talked about creating a
group from the community.
Short time-frame, specific deliverable:The default configuration for the SearchIndexer,
post-release 1.8.
Questions?