Welcome To

Director of Engineering

Search Science Recall & Spam

April 3, 2015

BRIAN JOHNSON

With more than 100 million active users globally, eBay is

the world's largest online marketplace, where practically

anyone can buy and sell practically anything. Founded in

1995, eBay connects a diverse and passionate community

of individual buyers and sellers, as well as small

businesses. Their collective impact on ecommerce is

staggering: In 2014, the total value of goods sold on eBay

was $82 billion -- more than $2,500 every second.

http://www.ebayinc.com/who

DATA DRIVEN DECISIONSMETRICS & TESTING

Why?

www.wallpapertimes.com

$’s per year in

incremental revenue

Data Collection

• Cell Tracking

• Eye Tracking

• Much Richer and more Detailed online

• Behavioral Log Files

We Are Data

Trends

Intelligence:Human Machine

Data: Small Big

Sources: Few Many

Context: Aggregate Detailed

SEARCH QUERY INTENT

Users

+

Documents

What is Special for eBay Search?

•Commercial Intentions– Both the sellers and buyers have strong and clear intention

– Transactions happen on eBay, hence more behavior data

•Listings (Supply Side)– Given by sellers

– Semi-structured data

•Buyers (Demand Side)– Relevance matters (Browser vs. Searching)

– Price matters

– Seller trust/credit matters

– 70% eBay revenue starts with Search

Fish Sticks

Demand Category

42%

Business & Industrial > Electrical & Test Equipment > Electrical Equipment & Tools >

Electrical Tools > Cable Pullers

10% Business & Industrial > Construction > Building Materials & Supplies > Electrical

9% Home & Garden > Tools > Other

7% Pet Supplies > Fish & Aquariums > Fish Pond Supplies

6% Business & Industrial > Light Equipment & Tools > Air Tools > Staplers

Fishing Sticker

Demand Category

41%

eBay Motors > Parts & Accessories > Car & Truck Parts > Decals/Emblems/License

Frames > Decals & Stickers > Graphics Decals

28% Home & Garden > Home Decor > Decals, Stickers & Vinyl Art

12% Sporting Goods > Fishing > Novelties & Gifts

11% Sporting Goods > Fishing > Fishing Equipment > Decals, Stickers & Patches

3% Collectibles > Transportation > Automobilia > Decals & Stickers

Fish Stickers

Demand Category

53% Home & Garden > Home Decor > Decals, Stickers & Vinyl Art

24% Crafts > Scrapbooking & Paper Crafts > Embellishments > Stickers

16%

eBay Motors > Parts & Accessories > Car & Truck Parts > Decals/Emblems/License

Frames > Decals & Stickers > Graphics Decals

3% Sporting Goods > Fishing > Novelties & Gifts

1% Sporting Goods > Fishing > Fishing Equipment > Decals, Stickers & Patches

Keyword Expansion Test

Key Expansion Top Query

fishing sticker fish stickers fishing sticker

diaries diary vampire diaries

baggies baggie patagonia baggies

baggies baggy patagonia baggies

cranberries cranberry the cranberries

jogging jog jogging stroller

Key Expansion Top Query

fishing sticker fish stickers fishing sticker

diaries diary vampire diaries

baggies baggie patagonia baggies

baggies baggy patagonia baggies

cranberries cranberry the cranberries

jogging jog jogging stroller

Context Matters

cowboys hats ≠ cowboy hats

Plastic toy cowboys = plastic toy cowboy

Context & Specificity

Context

ATC Armored Troop Carrier in Toys and Hobbies

ATC Artist trading card in ART

ATC Automatic Tool Change in Business and Industrial

Specificity/Directionality

Old Antique

Yoga towels/mats Yogitoes

Category Expansion

Stamps > Commonwealth/ British Colonial > Bermuda

Before After

Compound/Decompound Expansions

acidwash acid wash

Before After

German Compounds

•Syntactically, words can be combined and split in many ways

•Multiple candidatesGranitpflastersteine (granite paving stones)

Granit(granite) pflastersteine(cobblestones)

Granit(granite) pflaster(paving/band-aid) steine(stones)

•Binding characters

Hochzeitsschuhe (grammatically correct, 593 hits on ebay.de)

Hochzeitschuhe (129 hits on ebay.de)

•Some words shouldn’t be de-compounded.beiden (both) – bei(at) den(the)

Intent Preserving Query Relaxation

Hadoop Graph/Session Analysis

Bipartite Graphs

Keyword | Keyword Synonym Expansion

Keyword | Attribute Aspect Expansions

Keyword | Category Category Expansions, Related Search Diversity

Query | Query Related Search

Query | Item Related Search

Query Session Analysis

Successive Queries Synonyms, Related Search

Query Substitutions Synonyms

Same Session Correlation Related Search

Query Metrics

Click Through Rate

Purchase Attribution

Time to First Click, View Item, Purchase

Query Pair Price & Category Divergence

Query Pair Result Set Overlap

Result Count

Why we’re excited about data mining…

• We’re at an inflection point – customers are defining how they shop

– We are a data company

– 40+ Pb of data (listings, pictures, queries, clicks, sales, feedback, …)

– Many tests running orthogonally (in parallel with overlapping user slices)

– Nearly all users in one of more tests

– Many users per test, often millions

• Find patterns and insights drives our customer experience

• We’ve built successful teams of data scientists

THANKS!

QUESTIONS?

WE’RE HIRING

BJOHNSON@EBAY.COM

METRICS

•What should we optimize

–Page Views

–Time on Site

–Click Through Rate

–Normalized Discounted Cumulative Gain

–Purchases per User per Session/Day/Week

–Revenue per User per Session/Day/Week

–Net Promoter Score

•How likely would you be to recommend …?

REVENUE

Every business focuses on and measures revenue.

Every business focuses on profit and loss.

And they should.

Experimental Variation By Day

Experimental Variation By Metric

Query Rewrites at eBay

Query RewriteSearch

User Query

eBay Results

Search Query

User Query: pilzlampe {mushroom lamp}

Search Query: OR(pilzlampe, PHRASE(OR(pilz,pilze),OR(lampe,lampen)))

Example Query Services/Rewrites

• Stemming (ipod OR ipods)

• Spelling (cannon OR canon)

• Condition (new OR condition=new)

• Synonyms (boat carpet OR marine carpet)

• Space Synonyms (MarioKart OR Mario-Kart)

• Item Specifics (blue OR color=blue)

• Acronyms (hp OR hewlett-packard OR horsepower)

• Category (shoes OR Category=Shoes)

• Cross Border (site=0 AND category =123) OR (site=3 AND

category=456)

• Fitment (fits model=corolla)

• Term Removal (Harry Potter and the Order of the Phoenix (daily deal))

Acronym/Abbreviation Mining

•Acronyms/Abbreviations mined from raw text and query logs

•Look for patterns of text:

long form (short form)

short form (long form)

• Employ intelligent matching algorithms to mine candidates

• Schwartz et al: Greedy Match Algorithm

new cheap Playstation portable (PSP)

PlayStation 3 (PS3)

• Acronym discovered

PSP => PlayStation Portable

PS3 => PlayStation 3

• Candidates mined are fed to an ML classifier to remove false positives