PhillyCHI Site Search Analytics presentation (April 2, 2008)

50
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved. PhillyCHI Site Search Analytics: Conversations with your customers Louis Rosenfeld [email protected] Philadelphia, PA • April 2, 2008

description

Short talk on site search analytics given to PhillyCHI, April 2 2008, Philadelphia, PA, USA.

Transcript of PhillyCHI Site Search Analytics presentation (April 2, 2008)

Page 1: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

PhillyCHI Site Search Analytics: Conversations with your customers

Louis Rosenfeld [email protected] Philadelphia, PA • April 2, 2008

Page 2: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

About us

Me

You?

Page 3: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Anatomy of a search log (from Google Search Appliance) Critical elements in bold: IP address, time/date stamp, query, and # of results:

XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02

XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license+plate &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16

XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=regional+transportation+governance+commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17

Page 4: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Page 5: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Page 6: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Querying the queries: Generic questions to help get started

1.  What are the most frequent unique queries? 2.  Are frequent queries retrieving quality results? 3.  Click-through rates per frequent query? 4.  Most frequently clicked result per query? 5.  Which frequent queries retrieve zero results? 6.  What are the referrer pages for frequent queries? 7.  Which queries retrieve popular documents? 8.  What interesting patterns emerge in general?

Page 7: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Improving metadata and navigation: How SSA can help

1.  Analyze for metadata value tone, granularity, and trends

2.  Determine (and invest in) important metadata attributes

3.  Study queries that follow navigation failures/changes to create better contextual navigation

4.  Look to improve “top-down” navigation (e.g., indices, taxonomy, main page)

Page 8: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Determining metadata values through clustering

One week’s top 50 queries, clustered by topic

Represents 20% of all search activity

< 1 hour of work Can help suggest

preferred terms

Page 9: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Determining metadata values through similar queries

BBC explores similar queries (i.e., collaborative filtering)

Helps with misspellings, synonyms, best bets

Page 10: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Determining metadata attributes (AKA types, fields)

Top queries clustered and prioritized by attributes:   Place   Dept./

Program   Service   Task

1-2 hours work

Page 11: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Navigation failures/goal changes: (from Google Analytics)

Page 12: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Supporting contextual navigation through embedded queries

SLI Systems can display other queries related to a particular query…

…and queries related to a particular document

Page 13: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Hybrid A-Z index example: Michigan State University

Page 14: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Improving content: How SSA can help

1.  Verify content spidering 2.  Plug content gaps

(placeholders, new content, appropriate types, best bets)

3.  Improve content positioning (highlighting, demotion, deletion)

4.  Prioritize content development (invest in appropriate topics and types, more efficient migration and inventory)

Page 15: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Developing new content: Addressing emergent content needs

Using SSA to plug gaps  Failure analysis suggests

unmet content needs by topic (e.g., frequent queries for “iPod wristband” and “iPod holder for jogging”)

 Content typing (covered later) suggests what kind of content to create (e.g., “product comparison” and “specifications”)

0 results report (from behaviortracking.com)

Page 16: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Developing new content: Addressing emergent content needs

Use SSA to develop editorial agenda; determine relative “demand” for content topics and document types

Available content (supply) Content by query (demand)

Page 17: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Best bets: HP example (exposes content model)

Page 18: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Monitoring trends: Tuning content to meet demand

How might such trends impact your content? Search engine? Navigation?

From behaviortracking.com

Page 19: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Page 20: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Monitoring trends

Short head and long tail generally predictable

“Middle torso” is interesting area to monitor

Page 21: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Comparing short head and long tail queries (from Vanguard)

Short head: common queries Long tail: common queries

Beneficiary form 401(k) beneficiary career forms amt money market location loans calculator

403(b)(7) account asset transfer authorization automatic investing Wire transfer instructions adoption agreement international wire transfers socially responsible investing Vanguard tax identification number IRA Asset Transfer form fdic insured account early withdrawal penalties

Page 22: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

The shape of your content: Determining content types

Top queries clustered and prioritized by content types:   Application   News/

Announce-ments

  Main page   Contact info   Instructions

1-2 hours work

“What kind of page would users want when they searched this term?”

Page 23: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Content types help expose content models, improve navigation

23

artist descriptions

album reviews

album pages

artist bios discography

TV listings

Page 24: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Engaging content owners: Showing how users find their content

Connecting pages that are found through search…

Page 25: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Engaging content owners: Showing how users find their content

…with how those pages were found.

Page 26: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Engaging content owners: Comparing time periods

Showing trends

(from behaviortracking.com)

Page 27: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Improving search: How SSA can help

1.  Understand how search works (and what can be changed/fixed)

2.  Validate “the box” UI design 3.  Look for opportunities to “soup up” queries

(e.g., syntax, spell-checking) 4.  Design better search results (individual and

groups) 5.  Blow up “advanced search” and

contextualize functions 6.  Buy a better engine—or improve your own

Page 28: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Improving “the box”: Initial query entry UI

Validate width of query entry field  Use query data to calculate “high

number” (e.g., 90th percentile) for query length in characters

 Does your entry field accommodate that length?

Amazon: 65 characters

Microsoft: 25 characters

Page 29: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Improving “the box”: Search suggestions

Populate query entry with common queries (Ask)

Alternative: use controlled vocabulary (Netflix movie titles)

Page 30: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Query enhancement: Handling unique query types

Little fixes can add up  Spell-checking  Proper names (people, places)  Unique identifiers (e.g., SKUs, ISBNs)

Page 31: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Query enhancement: Handling unique query types (cont.)

Little fixes can add up  Acronyms (glossary lookup)

Page 32: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Search results design: What to show per individual result

…more descriptive information for open-ended searchers

…less information for known-item searchers

Page 33: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Search results design: SSA helps determine sorting

Financial Times found a high level of dates within queries

Solution: support chronological sorting and filtering

Page 34: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Search results design: SSA helps determine typing

Content typing can enable:   Filtering by exposing types

(HP)   Faceted classification/

navigation (WebMD)

Page 35: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Search results design: Which results where?

#10 result is clicked through more often than #s 6, 7, 8, and 9 (ten results per page)

From SLI Systems (www.sli-systems.com)

Page 36: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Rethinking advanced search: Which functions go where (or away)?

Page 37: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Rethinking advanced search: “Revise your Search” in action

Page 38: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

How we determine what users want

Qualitative research (observation)  Field studies  Focus groups  Card sorting  Usability testing and task analysis

Quantitative research (data-driven)  Help/reference desk/switchboard  Web analytics (SEO/SEM, clickstream, SSA)

Page 39: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

SSA as a user research method: How is it different?

Data that’s:  Real  High volume  Readily available

Analysis that’s:  Quantitative   Inexpensive  Scalable  Complementary to other user research methods

Page 40: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Comparing SSA with other quantitative user research methods

Data-driven methods  Clickstream/server log analysis: tells you where

more than what users wants (complementary)  Event logging: comparable but often less data

and usually at a different point in customer/user lifecycle

 SEO/SEM: Different query types (finding where the answer is versus finding the answer); often focused on commerce conversions

Page 41: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Comparing referral queries with local queries

Page 42: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

SSA and qualitative user research methods

What questions lead to Why questions Examples

 Personas: SSA supplements fictitious construct with real data (i.e., likely queries a persona might have)

 Task analysis: SSA helps identify common tasks to evaluate

 Field studies: SSA suggests issues to monitor in the field

Page 43: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Augmenting personas and audience profiles with frequent queries

Persona example (from Adaptive Path)

Frequent queries added (in green)

Page 44: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Using common queries to enhance/fuel task analyses

Use for tasks that are more what than how (i.e., testing information needs rather than functions)

Examples •  Can you find a map of the campus? •  What study abroad options are available to MSU students? •  When is the last home football game of the season?

Page 45: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Identifying trends to monitor during field studies

Example: catalog clothing retailer SSA showed:

 Preponderance of SKUs, even though they were not displayed on the site

 What were they doing there? Subsequent field study showed:

 Customers preferred print catalog to web catalog to identify products

 Customers preferred web catalog for ordering Action: add SKUs to web catalog

Page 46: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Combining quantitative and qualitative

Exploring the needs of a particular audience: users of BBC children’s content

Page 47: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

The power of data: Why UX needs SSA

1.  Balance: UX methodologies/practitioners are often quantitatively weak

2.  Legitimacy: Data-driven analyses make an impression on people hard to impress

3.  Cost: The data is available; the analysis scales well with available time

4.  Fidelity: The data is behavioral, real, and voluminous

5.  Comprehensiveness: Quantitative methods complement and help improve qualitative methods (and vice versa)

Page 48: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Please share your SSA knowledge: Visit our book in progress site Search Analytics for Your Site:

Conversations with your Customers by Louis Rosenfeld and Richard Wiggins (Rosenfeld Media, 2008)

Site URL: www.rosenfeldmedia.com/books/searchanalytics/ Feed URL: feeds.rosenfeldmedia.com/searchanalytics/

Page 49: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slides and discounts

Slides available here: http://www.slideshare.net/lrosenfeld

For a discount, use code…

PHILLYCHI

…at the Rosenfeld Media site for 10% off all of our titles

Page 50: PhillyCHI Site Search Analytics presentation (April 2, 2008)

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Contact Information

Louis Rosenfeld Rosenfeld Media, LLC 705 Carroll Street, #2L Brooklyn, NY 11215 USA

[email protected]

www.louisrosenfeld.com www.rosenfeldmedia.com