Otis gospodnetic Search Analytics Lucene Eurocon 2011

Search Analytics

Business Value&

NoSQL Backend

Otis Gospodnetić – Sematext International@otisg ◦ @sematext ◦ sematext.com

sematext.com/search-analytics

http://twitter.com/otisg

http://twitter.com/sematext

http://www.sematext.com/

http://sematext.com/search-analytics/index.html

Copyright 2011 Sematext Int'l. All rights reserved.2

About Otis Gospodnetić

• ASF Member: Lucene, Solr, Nutch, Mahout

• Author: Lucene in Action 1 & 2

• Entrepreneur: Sematext, Simpy


Sematext Metrics

100% organic: no GMO, no VC 4 years old < 10 people 7 countries 3 timezones 2 continents > 100 customers


About Sematext

Products & Services

Consulting, Development, Tech Support:

Search (Lucene, Solr, ElasticSearch...) Big Data (Hadoop, HBase,

Voldemort...) Web Crawling (Nutch, Droids) Machine Learning (Mahout)


Agenda

What is Search Analytics and why it matters

Example reports and their value What we built, why, and how


Communication

twitter.com/sematext twitter.com/otisg hash tags: #stsa or #stanalytics http://sematext.com/search-analytics/index.html Raise your hand! [email protected]


The Compass

Search logs are your Map

Search Analytics is your Compass


High Level Why

searchusers

searchproviders

searchexperience


High Level Why

searchproviders

searchexperience

This search sucks!It takes 17 tries to find anything here!

F!?@#$%^&?!?

searchusers

Cool, the latest search tweaks made our site really sticky!

Awesome!


Don't Be Like This Dude


Got Clue?

Search Analytics

Performance Monitoring

Quality Assurance

Tuning UI


More Concrete Why

Measure and monitor everything. Introspection. Supports (re)design, navigation choices Helps with content acquisition & enhancement Improve search experience Mula


The Moment of Truth

Question for the audience #1

What do you use for Search Analytics?

a) Home grown stuffb) Google Analyticsc) Omnitured) Webtrendse) Otherf ) Nothing


Search Analytics Outline

Collect: queries & clicks & interactions & ... Analyze: actions / xactions / conversions Output: reports – over time Output++: feedback loop

The means, not the goal Ongoing, not one-off

remember this


Search vs. Web Analytics

User intent and information needs vs. inferring Hand in hand Ideally you can relate data from both or even

unify it


Example Core Reports

Rate & Volume, Latency (mean, avg, 90%) Click Through Rate, Mean Reciprocal Rank Top Queries by count, clicks, 0 hits... Query Trending Top Seen Docs, Top Clicked Docs (msft) Page & Click Depth Facet & Sort Usage ...


More Reports in More Detail

See Search Analytics What? Why? How?

http://blog.sematext.com/tag/analytics/


Part Dos

Switching gears... Juno digs NoSQL


What We've Built

Search Analytics SaaS Numerous reports (e.g. query volume,

rate, latency, term frequencies / comparisons, hit buckets, search origins, etc.)

Trending over time Comparisons of time periods Top N reports Filter, slice and dice


Who Needs a Compass?

We need it search-hadoop.com & search-lucene.com

Our customers need it!

You?


Sematext Search Analytics


Big Dreams

SaaS Multitenant Large Scale – Massive Data Cloud


Storage Choices

RDBMS: MySQL, PostgreSQL HDFS Hive HBase Cassandra


SaaS vs. In-House

Question for the audience #2

SaaS vs in-house Search Analytics?

a) SaaSb) in-house


Data Flow See Search Analytics with Flume and HBase

http://blog.sematext.com/2010/10/16/search-analytics-hadoop-world-flume-hbase/


Data Collection See Search Analytics with Flume and HBase

http://blog.sematext.com/2010/10/16/search-analytics-hadoop-world-flume-hbase/


Core Tech

JavaScript Beacons Metric Capture Web App aka Receiver Flume Agents, Collectors, Sinks HBase MapReduce Aggregations Search Analytics Reporting Web App


What is Flume

Distributed data/log collection service Scalable, configurable, extensible Centrally manageable, open source

Agents get data from app, Collectors save it Abstractions: Source → Decorator(s) → Sink


What is HBase

Scalable, reliable, distributed, column-oriented DB On top of HDFS MapReducable


Data Flow, Detailed


Why Flume

Reliable delivery e.g. queue msgs locally if destination unreachable

Easy, centralized management via Web UI or console

Good community, good progress, now @ASF But: more complex, more moving parts On Flume: slideshare.net/cloudera/inside-flume Alternatives: Kafka, Scribe...


Why HBase

Scalable raw & aggregate data storage MapReduce data input Fast scans for time ranges, fast key lookups Easy storage and compute power expansion Good looking roadmap, community,

progress


Open Sourcing

2 open-source projects:

github.com/sematext/HBaseWD

github.com/sematext/HBaseHUT See sematext.com/open-source/index.html

Patches for Flume and HBaseblog.sematext.com/tag/flume/


Challenges

Data size. Solutions: Compression (4-5x smaller with lzo) Data pruning (variable levels)

Query string distribution: very long-tail Lots of data to process, update, aggregate

Young tools: Flume, HBase Poor IO on EC2 Hadoop distributions


Output++

AutoComplete - $MM improvement Better DYM Spellchecker Related Searches Recommendations Relevance Feedback ...


Closing the Loop

searchusers

searchproviders

searchexperience


Resource

http://rosenfeldmedia.com/books/searchanalytics/

Search Analytics for Your SiteLouis Rosenfeld


We're Hiring

Dig Search?

Dig Analytics?

Dig Big Data?

Dig Performance?

Dig working with and in open-source?

We're hiring world-wide!

http://sematext.com/about/jobs.html


sematext.com blog.sematext.com @sematext @otisg [email protected]

Want SA? Grab me or go to: sematext.com/search-analytics

Hash tags: #stsa or #stanalytics

Contact

Otis gospodnetic Search Analytics Lucene Eurocon 2011

Technology

Transcript of Otis gospodnetic Search Analytics Lucene Eurocon 2011