Welcome To

Director of Engineering

Search Science Recall & Spam

April 3, 2015

BRIAN JOHNSON

With more than 100 million active users globally, eBay is

the world's largest online marketplace, where practically

anyone can buy and sell practically anything. Founded in

1995, eBay connects a diverse and passionate community

of individual buyers and sellers, as well as small

businesses. Their collective impact on ecommerce is

staggering: In 2014, the total value of goods sold on eBay

was $82 billion -- more than $2,500 every second.

EBAY OVERVIEW

http://www.ebayinc.com/who

SEARCH

Users

+

Documents

Why we’re excited about data mining…

• We’re at an inflection point – customers are defining how they shop

– We are a data company

– 40+ Pb of data (listings, pictures, queries, clicks, sales, feedback, …)

– Many tests running orthogonally (in parallel with overlapping user slices)

– Nearly all users in one of more tests

– Many users per test, often millions

• Find patterns and insights drives our customer experience

• We’ve built successful teams of data scientists

We Are Data

Trends

Intelligence:Human Machine

Data: Small Big

Sources: Few Many

Context: Aggregate Detailed

What is Special for eBay Search?

•Commercial Intentions– Both the sellers and buyers have strong and clear intention

– Transactions happen on eBay, hence more behavior data

•Listings (Supply Side)– Given by sellers

– Semi-structured data

•Buyers (Demand Side)– Relevance matters (Browser vs. Searching)

– Price matters

– Seller trust/credit matters

– 70% eBay revenue starts with Search

QUERY INTENT

Fish Sticks

Demand Category

42%

Business & Industrial > Electrical & Test Equipment > Electrical Equipment & Tools >

Electrical Tools > Cable Pullers

10% Business & Industrial > Construction > Building Materials & Supplies > Electrical

9% Home & Garden > Tools > Other

7% Pet Supplies > Fish & Aquariums > Fish Pond Supplies

6% Business & Industrial > Light Equipment & Tools > Air Tools > Staplers

Keyword Expansion Test

Key Expansion Top Query

fishing sticker fish stickers fishing sticker

diaries diary vampire diaries

baggies baggie patagonia baggies

baggies baggy patagonia baggies

cranberries cranberry the cranberries

jogging jog jogging stroller

Key Expansion Top Query

fishing sticker fish stickers fishing sticker

diaries diary vampire diaries

baggies baggie patagonia baggies

baggies baggy patagonia baggies

cranberries cranberry the cranberries

jogging jog jogging stroller

Fishing Sticker

Demand Category

41%

eBay Motors > Parts & Accessories > Car & Truck Parts > Decals/Emblems/License

Frames > Decals & Stickers > Graphics Decals

28% Home & Garden > Home Decor > Decals, Stickers & Vinyl Art

12% Sporting Goods > Fishing > Novelties & Gifts

11% Sporting Goods > Fishing > Fishing Equipment > Decals, Stickers & Patches

3% Collectibles > Transportation > Automobilia > Decals & Stickers

Fish Stickers

Demand Category

53% Home & Garden > Home Decor > Decals, Stickers & Vinyl Art

24% Crafts > Scrapbooking & Paper Crafts > Embellishments > Stickers

16%

eBay Motors > Parts & Accessories > Car & Truck Parts > Decals/Emblems/License

Frames > Decals & Stickers > Graphics Decals

3% Sporting Goods > Fishing > Novelties & Gifts

1% Sporting Goods > Fishing > Fishing Equipment > Decals, Stickers & Patches

SEARCH RECALL

Query Rewrites

Query Rewrites at eBay

Query RewriteSearch

User Query

eBay Results

Search Query

User Query: pilzlampe {mushroom lamp}

Search Query: OR(pilzlampe, PHRASE(OR(pilz,pilze),OR(lampe,lampen)))

Example Query Services/Rewrites

• Stemming (ipod or ipods)

• Spelling (cannon or canon)

• Condition (new or condition=new)

• Synonyms (boat carpet or marine carpet)

• Space Synonyms (MarioKart > Mario Kart)

• Item Specifics (blue or color=blue)

• Acronyms (os = one size in CSA | Operating Systems in Electronics)

• Category (shoes or category=63850)

• Cross Border (site=0 and category =123) or (site=3 and category=456)

• Fitment (fits model=corolla)

• Term Removal (Harry Potter and the Order of the Phoenix (daily deal))

Category Expansion

Add Category Expansion

Stamps > Commonwealth/ British Colonial > Bermuda

Before After

Context Matters

Uncontexted expansions: cowboys cowboy

cowboys hats ≠ cowboy hats Plastic toy cowboys = plastic toy cowboy

Context & Specificity

Context

ATC > Armored Troop Carrier in Toys and Hobbies

ATC > Artist trading card in ART

ATC > Automatic Tool Change in Business and Industrial

Specificity/Directionality

Old > Antique

Yoga towels/mats > Yogitoes

Specificity

• Category based expansions

• Directional expansions

• Positive and Negative expansions

horsepower

Electronics Cars and Trucks

hp

Hewlett Packard

Acronym/Abbreviation Mining

•Acronyms/Abbreviations mined from raw text and query logs

•Look for patterns of text:

long form (short form)

short form (long form)

• Employ intelligent matching algorithms to mine candidates

• Schwartz et al: Greedy Match Algorithm

new cheap Playstation portable (PSP)

PlayStation 3 (PS3)

• Acronym discovered

PSP => PlayStation Portable

PS3 => PlayStation 3

• Candidates mined are fed to an ML classifier to remove false positives

Compound/Decompound Expansions

Add “acidwash” > “acid wash”

Before After

Space Synonyms and German Compound Words

•Sellers might list products in one form

•Buyers might search in another form

•We want to show results for both forms when any of these synonymous forms are searched

Germans Love Compound Words

Damenlederhose

-Damen lederhose

-Damen leder hose

Akustikgitarrenverstärker

-Akustik gitarrenverstärker

-Akustik gitarren verstärker

All these forms are equivalent

•German compound words can be arbitrarily created and

extremely longAdidastrainingsanzug (Adidas track suit)

Rindfleischetikettierungsüberwachungsaufgabenübertragungsgesetz

(beef labeling regulation & delegation of supervision law)

•Syntactically, words can be combined and split in many ways.

•Some words shouldn’t be de-compounded.beiden (both) – bei(at) den(the)

•Too many candidates forGranitpflastersteine (granite paving stones)

Granit(granite) pflastersteine(cobblestones)

Granit(granite) pflaster(paving/band-aid) steine(stones)

•Binding characters

Hochzeitsschuhe (grammatically correct, 593 hits on ebay.de)

Hochzeitschuhe (129 hits on ebay.de).

Hadoop Graph/Session Analysis

Bipartite Graphs

Keyword | Keyword Synonym Expansion

Keyword | Attribute Aspect Expansions

Keyword | Category Category Expansions, Related Search Diversity

Query | Query Related Search

Query | Item Related Search

Query Session Analysis

Successive Queries Synonyms, Related Search

Query Substitutions Synonyms

Same Session Correlation Related Search

Query Metrics

Click Through Rate

Purchase Attribution

Time to First Click, View Item, Purchase

Query Pair Price & Category Divergence

Query Pair Result Set Overlap

Result Count

Null/Low Result Set Expansion

THANKS!

QUESTIONS?

BJOHNSON@EBAY.COM

DATA DRIVEN DECISIONSMETRICS & TESTING

Why?

www.wallpapertimes.com

$’s per year in

incremental revenue

METRICS

•What should we optimize

–Page Views

–Time on Site

–Click Through Rate

–Normalized Discounted Cumulative Gain

–Purchases per User per Session/Day/Week

–Revenue per User per Session/Day/Week

–Net Promoter Score

•How likely would you be to recommend …?

REVENUE

Every business focuses on and measures revenue.

Every business focuses on profit and loss.

And they should.

Experimental Variation By Day

Experimental Variation By Metric

FEATURES / FACTORS

Features: Mutual Information

• Rationale: The goal of this metric is determine if the co-occurrence of

the candidates in the description is significantly more than the

random chance of them co-occurring.

Features: Neighborhood Similarity

• Rationale: Two synonym candidates A and B, will tend to

have similar neighbors (viz keywords) surrounding them.

Intersection ( Neighbours(A) , Neighbours(b) )

Min (Neighbours(a), Neighbours(b))

Neighborhood similarity =

Features: KL divergence

• Rationale: Two synonym candidates will have similar

price and category distributions of their inventory.

EBAY OVERVIEW