Post on 17-May-2015
description
#SummitNow
Super Size Your Search6th November 2013Piergiorgio Lucidi (Sourcesense)
Fran Alvarez (Zaizi)
#SummitNow
#SummitNow
Piergiorgio Lucidi• Open Source ECM Specialist at Sourcesense• Alfresco Certified Trainer / Engineer• Alfresco Wiki Gardener / Community Star• Alfresco forum supporter• Global Moderator of the italian forum• Author and Technical Reviewer at Packt• PMC Member and Mentor at ASF• Project Leader in the JBoss Community
#SummitNow
#SummitNow
OverviewHow to build and manage your search server:
1. Scenario2. Introducing Apache
ManifoldCF3. Zaizi Integrated Search
Solution
#SummitNow
#SummitNow
ScenarioAn overview about the typical complex search architecture
#SummitNow
#SummitNow
Scenario - Alfresco limitationsAlfresco supports these search engines:• Apache Lucene (embedded)• Apache Solr (provided by Alfresco)• needs development if other
repositories must be involvedEvery other approach must be implemented (ScheduledActions, WebScripts, etc..)
#SummitNow
#SummitNow
Scenario – Embedded
Simple Search Architecture
Alfresco is the only one repository involved in the architecture using the embedded search engine:• the repository must take care of indexes
also managing index transactions
Indexes
AlfrescoFrontEnd
applications
Apache Lucene
#SummitNow
#SummitNow
Scenario – Embedded - Cluster
Embedded
Not easy to scale out with Lucene1. every cluster must have its own search
indexes2. The cluster must synchronize indexes
Indexes
Alfresco
Apache Lucene
Indexes
Alfresco
Apache Lucene
JGroups
#SummitNow
#SummitNow
Scenario – Simple Architecture
Simple search architecture
Alfresco is the only one repository involved in the architecture with an external search server1. The search server can be used for
publish contents in the front end architecture
2. The repository will stay in the logic backend
Search Engine
Indexes
Alfresco FrontEnd applications
#SummitNow
#SummitNow
Scenario – Publish with searchA search engine can be used for:• advanced management of search
indexes• scaling out• executing complex search on
contents• publishing contents in the FE
architecture
#SummitNow
#SummitNow
Scenario – Publish with search
Publish with search architecture
Alfresco is the only one repository involved in the architecture with an external search server1. The search server can be used for
publishing contents in the front end architecture (HTML)
2. The repository will stay in the logic backend
Search Engine
Indexes
Alfresco FrontEnd applications
BackEnd FrontEnd
Lucene / Solr
Indexes
#SummitNow
#SummitNow
Scenario – Simple Architecture
Simple Search Architecture
Alfresco is the only one repository involved in the architecture with an external search server1. The search server can be used for
publish contents in the front end architecture
2. The repository will stay in the logic backend
Search Engine
Indexes
Alfresco FrontEnd applications
#SummitNow
#SummitNow
Scenario – Complex Architecture1. Alfresco is only one of the platforms
that must be involved in your search architecture
2. You don’t want to increase the development effort
3. You want just something to configure
#SummitNow
#SummitNow
Scenario – Complex Architecture
Architecture with different ECM systems
Alfresco is one of the content platforms that must be involved in the indexing process
Alfresco
Search Engine
Indexes
SharePoint
FileNet
CMIS
JIRA
Google Drive
DropBox
#SummitNow
#SummitNow
Scenario – Complex Architecture
Architecture with different ECM systems
Alfresco is one of the content platforms that must be involved in the indexing process
Alfresco
Search Engine
Indexes
SharePoint
FileNet
CMIS
JIRA
Google Drive
DropBox ?
#SummitNow
#SummitNow
Scenario – Complex Architecture
Architecture with different ECM systems
Alfresco is one of the content platforms that must be involved in the indexing process
Alfresco
Search Engine
Indexes
SharePoint
FileNet
CMIS
JIRA
Google Drive
DropBox
#SummitNow
#SummitNow
Introducing Apache ManifoldCF
#SummitNow
#SummitNow
Apache ManifoldCF - HistoryManifoldCF code base was granted by MetaCarta to the Apache Software Foundation in December 2009.
The MetaCarta effort represented more than five years of successful development and testing in multiple, challenging enterprise environments.
The project was graduated as Apache Top Level Project in July 2012.
#SummitNow
#SummitNow
Apache ManifoldCF – What is?Open Source crawler• crawling model (add, change,
delete)• schedule jobs to create indexes • get contents from repositories• push contents on search servers
#SummitNow
#SummitNow
Apache ManifoldCF – What is?
Repository 1
Repository 3
Repository 4
Repository 2Apache ManifoldCF
Search Server 1
Search Server 2
Search Server 3
Search Server 4
#SummitNow
#SummitNow
Apache ManifoldCF – What is?Out-Of-The-Box it is distributed as a webapp• REST API• Authority Service• ACL indexes
• Crawler UI can be embedded in any Java application
#SummitNow
#SummitNow
Apache ManifoldCF – Why?• Reliability • Incremental• Flexible• Multi repositories• Security model• Monitoring
#SummitNow
#SummitNow
ManifoldCF – Why? - ReliabilityJobs scheduling and configuration are stored in the database to maintain the state of all the executions
Repository 1
Repository 3
Repository 4
Repository 2Apache ManifoldCF
Search Server 1
Search Server 2
Search Server 3
Search Server 4
Pull Agent Daemon
Database
#SummitNow
#SummitNow
ManifoldCF – Why? - Incrementalget content changesets obtained from the repository API
Repository 1 Apache ManifoldCF
Pull Agent Daemon
Database
query
Complete Changesets
#SummitNow
#SummitNow
ManifoldCF – Why? - FlexibleIf the repository can't supply all the changes Manifold can discover them through crawling
Apache ManifoldCF
Pull Agent Daemon
Database
queryIncomplete Changesets
Change Discovery
N N
#SummitNow
#SummitNow
ManifoldCF – Why? – Multi repoJobs can retrieve contents from the following repositories:• Google Drive• Dropbox• HDFS• CMIS-compliant• Alfresco • IBM FileNet• EMC Documentum
• Microsoft SharePoint• OpenText LiveLink• Autonomy Meridio• Memex Patriarch• Windows Share/DFS • Generic JDBC • Generic Filesystem • Generic RSS and Web
#SummitNow
#SummitNow
ManifoldCF – Why? – Multi repoJobs can ingest contents to the following search servers:
• Apache Solr• ElasticSearch • OpenSearchServ
er• MetaCarta GTS
#SummitNow
#SummitNow
ManifoldCF – Why? - SecurityRetrieve per-content ACLs
Repository 1
Repository 3
Repository 4
Repository 2
Apache ManifoldCFSearch Server 1
Search Server 2
Search Server 3
Search Server 4
Authority Service
Authority 1
Authority 2
access tokens
#SummitNow
#SummitNow
ManifoldCF – Why? - SecurityRetrieve per-content ACLs
Repository 1
Repository 3
Repository 4
Repository 2
Apache ManifoldCFSearch Server 1
Search Server 2
Search Server 3
Search Server 4
Authority Service
Authority 1
Authority 2
user access tokens
user specific search results
#SummitNow
#SummitNow
ManifoldCF – Why? – MonitoringUI Crawler allows you to:• configure jobs and connectors• monitor jobs execution• monitor contents ingestion
• status reports• document status• queue status
• history reports • simple history• maximum activity• maximum bandwidth• result histogram
#SummitNow
#SummitNow
ManifoldCF – Architecture
Repository Job Search Server
ACLs
#SummitNow
#SummitNow
ManifoldCF – Architecture
Repository Job Search Server
ACLs
Repository Connector
#SummitNow
#SummitNow
ManifoldCF – Architecture
Repository Job Search Server
ACLs
Repository Connector Output Connector
#SummitNow
#SummitNow
ManifoldCF – Architecture
Repository Job Search Server
ACLs
Repository Connector Output Connector
Authority Connector
#SummitNow
#SummitNow
ManifoldCF – Architecture
Repository Job Search Server
ACLs
Repository Connectorquery to retrieve
contentsOutput Connector
Authority Connector
#SummitNow
#SummitNow
ManifoldCF – Architecture
Repository Job Search Server
ACLs
Repository Connectorquery to retrieve
contents
Output Connectormetadata mappingcontent ingestion
Authority Connector
#SummitNow
#SummitNow
ManifoldCF – Architecture
Repository Job Search Server
ACLs
Repository Connectorquery to retrieve
contents
Output Connectormetadata mappingcontent ingestion
Authority Connectorretrieve content
ACEs
#SummitNow
#SummitNow
ManifoldCF – Architecture
Repository Job Search Server
ACLs
Repository Connectorquery to retrieve
contents
Output Connectormetadata mappingcontent ingestion
Authority Connectorretrieve content
ACEs
• verbal description
• crawling model• scheduling
#SummitNow
#SummitNow
Who is using ManifoldCF?
#SummitNow
#SummitNow
ManifoldCF - Resources
The project is available at http://manifoldcf.apache.org/
From this website you can access to the mailing lists, documentation and download links for binaries and source.
#SummitNow
#SummitNow
ManifoldCF – Resources - BookManifoldCF in Action by Karl Wright published by Manning Karl is the original developer and the principal committer of Apache ManifoldCF The book is available at
http://www.manning.com/wright
#SummitNow
#SummitNow
Zaizi Integrated Search Solution
#SummitNow
#SummitNow
Fran Alvarez• Director of Zaizi Iberia and Lead
Architect• Alfresco Certified Engineer• Responsible of large Alfresco
architectures• Semantic Consultant for Sensefy• Alfresco Meetups Organizer
#SummitNow
#SummitNow
Alfresco + Solr ApproachQuite a good architecture
• Performance issues are solved
• Different architectures depending on business requirements
However…
• It does not cover some use cases or scenarios
• It does not leverage Cloud benefits or latest technologies
• With huge data volume there are other approaches
How can we solve limitations and enhance benefits?
#SummitNow
#SummitNow
Alfresco + Solr Approach• Decouples Search solution from Alfresco•Allow to implement different Search solutions•Allow to change Search solution without changing anything in Alfresco
• Not even a property!•Provides an API to integrate it with Alfresco as search engine
• Even other repository vendors! E.g. Filesystem, Sharepoint, Documentum, Filenet, Drupal…
•And preserve security permissions in the results• Alfresco permissions are indexed and used during search
It’s included in our Semantic solution: Sensefy!
#SummitNow
#SummitNow
What we’ve done in ManifoldRepository Connector:• Alfresco Repository Connector: New implementation
• Removing dependency with Alfresco Solr APIOutput connectors:• Cloud Search Output Connector: Design & Development• Elastic Search Output Connector: Improvements• Solr Cloud Output Connector: Configuration for Alfresco
Authority Connector• Alfresco Authority Connector: Design & Development
• Similar approach to Alfresco Solr• Acl reads for Users and Groups in Alfresco
#SummitNow
#SummitNow
Scenarios
Let’s see some examples
#SummitNow
#SummitNow
I: Several Alfresco instancesCurrent Approach:
• Each Alfresco has its own Search subsystem
• They can’t share indexes
Implications:• Federated search is not an option• Results can’t be merged
• If so, what resultset should be first?
ConclusionResults could be presented to users in different tabs or “manually” merged.Not the best approach
#SummitNow
#SummitNow
I: Several Alfresco instancesZaizi Approach:
• Our solution like search box• Which manages a single index
Implications:• All documents are driven to same
index• Users can select results from
either all Alfresco instances or a subset
ConclusionSearch across Repositories
Could be based Elastic Search, Solr Cloud, Amazon Cloud, etc.
#SummitNow
#SummitNow
II: Alfresco + Other data providers
Current Approach:• Alfresco has its own Search
subsystem• Other repository may have (or
not) its own Search subsystemImplications:
• Different data providers mean different formats• E.g. Filesystem does not
support CMIS• Alfresco can’t reach external data
ConclusionNo way to merge results and
present them uniformly to end users
#SummitNow
#SummitNow
II: Alfresco + Other data providers
Zaizi Approach:
• Both Alfresco and other repositories share Search subsystem (Manifold)
Implications:• Alfresco and other providers
results will have same format in our Solution• They will speak ‘our’ language
• Alfresco reaches external data when communicating with our solution
ConclusionResults are present and accessible between data providers
#SummitNow
#SummitNow
III: Alfresco + O(TB) dataCurrent Approach:
• Alfresco has its own Search subsystem
• All data is in one (or several if cluster) Solr instance
Implications:• Every Solr node manages the
whole index• No chance to apply scale
techniques for indexing:• Sharding, Replication…
ConclusionHuge servers are required and performance might be compromised
#SummitNow
#SummitNow
III: Alfresco + O(TB) dataZaizi Approach:
• Alfresco uses our solution• Data is indexed in search solution
which better suits:• Amazon Cloud, Solr Cloud,
Elastic Search…
Implications:• Cloud Search solution manages
index• Indexing techniques can be applied
according to use cases• Sharding, Replication
ConclusionSearch strategy can be adopted and easily implemented with search solution which better fits
#SummitNow
#SummitNow
Apache Manifold: Other benefitsCan extract, index and map information from any other sources• Apache Stanbol, RedLink, any other data enricher• Our solution will gather everything in one place
• Documents, entities…Permissions are checked just once• Everything is in the same place, even user
authorization capabilities• Performance and scalability is improved• Faceted search and other search capabilities are
combined with such permission feature
#SummitNow
#SummitNow
Demo
#SummitNow
#SummitNow
ConclusionsZaizi solution allows searching and indexing in the most popular Cloud Search solutions
• Other Search solutions can be integrated as wellZaizi solution allows retrieving information from the most popular repositories
• Other Data providers can be integrated too• It solves plenty of current issues related search and
indexing in Alfresco• Can be used outside Alfresco or even with Alfresco and
any other data repositoryZaizi solution manages permissions and security from the most popular repositories and the latest Cloud search technologies Fully supported by us!
#SummitNow
#SummitNow
Conclusions
#SummitNow
#SummitNow
What’s comingPowerful User Interface• Admin functions• Wide range of
facets• UI for Share
Benchmarking
New connectors• Filesystem
authority• RedLink repository• Stanbol repository
Alfresco Search Subsystem?
#SummitNow