Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search &...

29
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Search and Analytics (using Elasticsearch) Costin Leau

Transcript of Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search &...

Page 1: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Search and Analytics

(using Elasticsearch)

Costin Leau

Page 2: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Why search?

Page 3: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Search – what’s the big deal?

Basic/Metadata retrieval

“Find banks with more then (x) accounts”

Page 4: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Search – what’s the big deal?

Basic/Metadata retrieval

“Find banks near my location”

Page 5: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Search – What we’re all about

Page 6: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Search categories

Basic/Metadata retrieval

Full-text search

Highlighting

Geolocation

Fuzzy search (“did-you-mean”)

Natural Language

Page 7: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Search categories

Basic/Metadata retrieval

Full-text search

Highlighting

Geolocation

Fuzzy search (“did-you-mean”)

Natural Language

data stores

search engines

Page 8: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

‘Players’ in the search market

Search engines

- Google/Bing/Yahoo!/Ask.com/Yandex/Baidu

Open-Source

- Sphinx

- Apache Lucene

- Elasticsearch

- Solr

- Sensei

Enterprise Search

- Oracle Endeca / MDEX

- HP Autonomy

- Exalead

- IBM Enterprise Search

Page 9: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Elasticsearch

Page 10: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Elasticsearch

Open-Source Search & Analytics engine

- Structured & Unstructured Data

- Real Time

- Analytics capabilities (facets)

- REST based

Distributed

- Designed for the Cloud

- Designed for Big Data

Page 11: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Elasticsearch

Open-Source Search & Analytics engine

- Structured & Unstructured Data

- Real Time

- Analytics capabilities (facets)

- REST based

Distributed

- Designed for the Cloud

- Designed for Big Data

Lightweight

Page 12: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Elasticsearch

Open-Source Search & Analytics engine

- Structured & Unstructured Data

- Real Time

- Analytics capabilities (facets)

- REST based

Distributed

- Designed for the Cloud

- Designed for Big Data

Lightweight

Popular: >200K downloads/month

Page 13: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Users

Page 14: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Users

Page 15: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Platform Adoption

http://www.thoughtworks.com/radar#platforms 2013

http://www.thoughtworks.com/radar
Page 16: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Platform Adoption

http://www.thoughtworks.com/radar#platforms 2013

http://www.thoughtworks.com/radar
Page 17: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Use Case - Text Search

https://github.com/blog/1381-a-whole-new-code-search

https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
https://github.com/blog/1381-a-whole-new-code-search
Page 18: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Searches 50,000,000 venues every day using

Elasticsearch

Use Case - Geolocation

Page 19: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Use Case – Support/Reporting

Page 20: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Use Case - Centralized Logging

Page 21: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Use Case - Pure Analytics

Page 22: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Search and Big Data

Page 23: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

A Holistic View of a Big Data System

ETL

Real

Time

Streams

Unstructured Data (HDFS)

RT Semi

structured

Database

(hBase,

Cassandra,

Mongo)

Big SQL (Greenplum,

AsterData,

Etc…)

Batch Processing Real-Time

Processing

(s4, storm)

Analytics

Page 24: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

A Holistic View of a Big Data System

ETL

Real

Time

Streams

Unstructured Data (HDFS)

RT Semi

structured

Database

(hBase,

Cassandra,

Mongo)

Big SQL (Greenplum,

AsterData,

Etc…)

Batch Processing

Analytics

Real-Time

Processing

(s4, storm)

Page 25: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Hadoop eco-system

Hadoop Distributed File System (HDFS)

Map Reduce Framework (MapRed)

http://hbase.apache.org/
http://hive.apache.org/
http://pig.apache.org/
http://hadoop.apache.org/
Page 26: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Hadoop eco-system

Hadoop Distributed File System (HDFS)

Map Reduce Framework (MapRed)

http://hbase.apache.org/
http://hive.apache.org/
http://pig.apache.org/
http://hadoop.apache.org/
Page 27: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Elasticsearch + Hadoop

0

10

20

30

40

50

60

M/R Pig Hive

Raw w/ ES

0

10

20

30

40

50

60

M/R Pig Hive

Raw w/ ES

Writing Reading / Querying

Page 28: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Explore data through

(Elastic)Search

Page 29: Search and Analytics (using Elasticsearch) - Costin Leau.pdf · Elasticsearch Open-Source Search & Analytics engine -Structured & Unstructured Data -Real Time -Analytics capabilities

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Thank you! @costinl

http://www.elasticsearch.org/

http://costinl/
http://www.elasticsearch.org/

Search and analyze your data with elasticsearch

Search and analyze your data with elasticsearch

Text search with Elasticsearch on AWS

Text search with Elasticsearch on AWS

Elasticsearch you know, for search

Elasticsearch you know, for search

Replace your Google Search Appliance (GSA) with Elasticsearch · PDF fileTurn Elasticsearch into a Complete Enterprise-wide Search Solution Integrate Elasticsearch and SharePoint with

Replace your Google Search Appliance (GSA) with Elasticsearch · PDF fileTurn Elasticsearch into a Complete Enterprise-wide Search Solution Integrate Elasticsearch and SharePoint with

Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Log Analytics with Amazon Elasticsearch Serviceaws-de-media.s3.amazonaws.com/images/_Munich_Loft_Slides/Elast… · Log Analytics with Amazon Elasticsearch Service Christoph Schmitter

Log Analytics with Amazon Elasticsearch Serviceaws-de-media.s3.amazonaws.com/images/_Munich_Loft_Slides/Elast… · Log Analytics with Amazon Elasticsearch Service Christoph Schmitter

Integrating Elasticsearch for Real-Time Analytics with Kibana

Integrating Elasticsearch for Real-Time Analytics with Kibana

Dell EMC Search · above Elasticsearch. Elasticsearch cluster ports NGINX TCP/ HTTPS 9300– 9400 Ports for communicating with Elasticsearch (Index data nodes). Elasticsearch cluster

Dell EMC Search · above Elasticsearch. Elasticsearch cluster ports NGINX TCP/ HTTPS 9300– 9400 Ports for communicating with Elasticsearch (Index data nodes). Elasticsearch cluster

Spark and Elasticsearch for real-time data analysis · What is Elasticsearch? Scalable, real-time search and analytics engine Open-source (on Github, Apache 2 License)

Spark and Elasticsearch for real-time data analysis · What is Elasticsearch? Scalable, real-time search and analytics engine Open-source (on Github, Apache 2 License)

Using Logstash and Elasticsearch analytics capabilities as ... · PDF fileUsing Logstash and Elasticsearch analytics capabilities as a BI tool Paschalis Korosoglou1, Pavlos Daoglou1,

Using Logstash and Elasticsearch analytics capabilities as ... · PDF fileUsing Logstash and Elasticsearch analytics capabilities as a BI tool Paschalis Korosoglou1, Pavlos Daoglou1,

Why Search? (starring Elasticsearch)

Why Search? (starring Elasticsearch)

Identity Manager 3.2.0.1 Upgrading to VMware (Linux) · Check for Elasticsearch Mapping Conflicts 2 Elasticsearch, a search and analytics engine, is embedded in VMware Identity Manager

Identity Manager 3.2.0.1 Upgrading to VMware (Linux) · Check for Elasticsearch Mapping Conflicts 2 Elasticsearch, a search and analytics engine, is embedded in VMware Identity Manager

VQ Conference Manager Unleash Awesome!€¦ · the future, using class-leading Elasticsearch and Kibana analytics Enterprise-grade Elasticsearch and Kibana analytics Benefit from

VQ Conference Manager Unleash Awesome!€¦ · the future, using class-leading Elasticsearch and Kibana analytics Enterprise-grade Elasticsearch and Kibana analytics Benefit from

Optimization of the search engine Elasticsearch

Optimization of the search engine Elasticsearch

Modernizing WordPress Search with Elasticsearch

Modernizing WordPress Search with Elasticsearch

Building search app with ElasticSearch

Building search app with ElasticSearch

Search Bugs Fast with Elasticsearch

Search Bugs Fast with Elasticsearch

Realtime Analytics With Elasticsearch [New Media Inspiration 2013]

Realtime Analytics With Elasticsearch [New Media Inspiration 2013]

Tuning ElasticSearch for multi-terabyte analytics

Tuning ElasticSearch for multi-terabyte analytics

Elasticsearch Introduction to Data model, Search & Aggregations

Elasticsearch Introduction to Data model, Search & Aggregations