Cross Channel Communication

at eBay Kleinanzeigen

Matthias Huttar @MattHuttarWhisky, Vienna mhuttar@ebay-kleinanzeigen.de

Manuel AldanaMusic, Football, Berlin maldana@ebay-kleinanzeigen.de

Top-1013 M 15 M

7 M2.5 M

6 M 4 M

65 %

Most Visited Site in Germany

Ads (Kleinanzeigen)

Unique Visitors / Month

Visits / Day

Messages / Day

Android Downloads

iOS Downloads

Mobile Traffic

Platform

DevelopersProduct ManagerQA Engineer

811

511

Mobile Apps

Search

Contact

Post-Ad

Selling

Wait

Buying

Meet & Trade

-- Privacy --

2011

seller-12sd43tyx@mail.ebayk.de

buyer-kjha2422@mail.ebayk.de

Seller Buyer

Bike (suuuper cool!)

WebsiteContact-Poster Form

ReplyTS

BuyerSusi@gmx.de

SellerMax@gmail.com

From: Susi@gmx.de

Hi! Would like to get this Bike.

WebsiteContact Seller Form

ReplyTS

BuyerSusi@gmx.de

SellerMax@gmail.com

From: buyer-12323@mail.ebayk.deTo: max@gmail.com

Hi! Would like to get this Bike.

WebsiteContact Seller Form

ReplyTS

BuyerSusi@gmx.de

SellerMax@gmail.comFrom: max@gmail.com

To: buyer-12323@mail.ebayk.de

Sure! Let’s meet!

WebsiteContact Seller Form

ReplyTS

BuyerSusi@gmx.de

SellerMax@gmail.comFrom: seller-slk543j@mail.ebayk.de

To: Susi@gmx.de

Sure! Let’s meet!

RFC 2822

Delivery Status Notifications

Autoresponder

Out-of-office Mails

Newsgroup Mails

SMTP

SMTP

FilterAnonymizer

HD

MySQL Cassandra

Theory is a Bastard

(2012)

SMTP

FilterAnonymizer

HD

Riak

SMTP

Riak

conversationconversation ID → Conversation Data

conversation_secretanonymous mail →

Conversation ID

AdConversation

Buyer Seller

Message

Message

Message

[ +18

]

=18

[ +18, - 4

]

=14

[ +18, - 4, -10

]

=4

[ +18, - 4, -10, +20

]

=24

[ +18, - 4, -10, +20, -10]

=14

Create Conversation13:00

Message from Buyer Received13:10

Message Sent13:11

Message from Seller Received13:22

Message from Seller Sent 13:22

Message From Seller Received14:11

A 13:00

B 13:10

C 13:11

A 13:00

B 13:10

D 13:50

A 13:00

B 13:10

C 13:11

D 13:50

A 13:00

B 13:10

C 13:11

A 13:00

B 13:10

D 13:50

A 13:00

B 13:10

C 13:11

D 13:50

A 13:00

B 13:10

C 13:11

A 13:00

B 13:10

D 13:50

A 13:00

B 13:10

C 13:11

D 13:50

A 13:00

B 13:10

C 13:11

A 13:00

B 13:10

D 13:50

A 13:00

B 13:10

C 13:11

D 13:50

A 13:00

B 13:10

C 13:11

A 13:00

B 13:10

D 13:50

A 13:00

B 13:10

C 13:11

D 13:50

Mai 2013:

1.000.000Mails a Day. Each Day.

(50 mails / sec)

BANG!

… some months later…

In-App Messaging

The Problem with E-Mail...

Mobile Growth

TargetKPIs

Welcome, CC-license https://flic.kr/p/6FP2Fu

MoreXY %

Visits / Day

speed lights 2, CC-license https://flic.kr/p/A5aTo

Reducedt1 + t2

Buyer Sellert1

t2

Smiley Face Graffiti, CC-license https://flic.kr/p/9PpYvj

Positive App Reviews

+Adoption Rate

2 in-house User-Tests later...

Screen 1:List-View

Screen 2:Detail-View

E-MailIn-App +E-MailIn-App

SMTP

FilterAnonymizer

HD

Riak

SMTP

Asynchronous

Asynchronous

Synchronous

In App Messaging

/postbox

SMTPSMTP

Problem:E-Mail Payload

?

Top / BottomPosting

1st Message2nd Message

QuotationMarks

Payload Mutation

Regular Expressions...

public static final Pattern MESSAGEBOX_OFFER_PATTERN = Pattern.compile("Angebot:\\s+[0-9\\.,]+\\s+EUR\\s+Angebot annehmen\\s+Gegenangebot"); // not used for push notifications. check PushNotificationTextShortener for that purpose. this impl is more defensive to ensure correct results. public static final List<Pattern> REPLACE_PATTERNS = ImmutableList.of( Pattern.compile("[<]http:.*?[>]", Pattern.MULTILINE), Pattern.compile("[*]", Pattern.MULTILINE), Pattern.compile("---* ?Urspr.*?-.*$", Pattern.MULTILINE), Pattern.compile("---* ?Original.*?-.*$", Pattern.MULTILINE), Pattern.compile("---* ?Reply.*?-.*$", Pattern.MULTILINE), Pattern.compile("Anfang der weitergeleiteten .*$", Pattern.MULTILINE), Pattern.compile(" hat am.*?geschrieben:", Pattern.MULTILINE Pattern.compile("(^| )wrote:", Pattern.MULTILINE), Pattern.compile("(^| )schrieb:", Pattern.MULTILINE), Pattern.compile("Am [0-9][0-9][.].*? schrieb:?(.*?ber eBay Pattern.compile("On [0-9][0-9][.].*? wrote:?(.*?ber eBay Kleinanzeigen)?", Pattern.compile("Sent:.*?$", Pattern.MULTILINE), Pattern.compile("Gesendet:.*?$", Pattern.MULTILINE), Pattern.compile("On.*?$", Pattern.MULTILINE), Pattern.compile("^.*?[<].*?@.*?[>].*?$", Pattern.MULTILINE), Pattern.compile("\\<mailto:.*?@.*?\\>"), Pattern.compile("\\[mailto:.*?@.*?\\]"), Pattern.compile("interessent-.*?@mail.ebay-kleinanzeigen.de:?"), Pattern.compile("anbieter-.*?@mail.ebay-kleinanzeigen.de:?"), Pattern.compile("[Vv]om [Ii][Pp]hone gesendet"), // no Pattern.CASE_INSENSITIVE Pattern.compile("[Vv]on meinem [Ii][Pp]hone gesendet"), // no Pattern.CASE_INSENSITIVE Pattern.compile("(eBay Kleinanzeigen [|] Kosten.*?)?Anfrage zu Ihrer.*?Ein Pattern.compile("^.*?Nachricht von:.*$", Pattern.MULTILINE), Pattern.compile("Artikel bereits verkauft.*?[|].*?Kontakt(\\[.*?\\])?", Pattern.DOTALL | Pattern.MULTILINE), Pattern.compile("Beantworten Sie diese Nachricht einfach.*?[|].*?Kontakt(\\[.*?\\])?", Pattern.DOTALL | Pattern.MULTILINE), MESSAGEBOX_OFFER_PATTERN

… andNGram based

Diff

1. Tokenize2. Seek3. Cleanup

NGram-Diff

Thunderbird:Yes, pick it up.> Still there? Am interested.

Outlook: Yes, pick it up.I’ll be there tomorrow.> Still there? Am interested.

n+1

n

Yes, pick it up.

thunderbird:Yes, pick it up.> Still there? Am interested.

Still there? Am interested.

n

Tokenize

Outlook: Yes, pick it up.I’ll be there tomorrow.> Still there? Am interested.

I’ll be there tomorrow.

n+1

Yes, pick it up.

Still there? Am interested.

Tokenize

Ja

SeekYes, pick it up.

Still there? Am interested.

I’ll be there tomorrow.

Yes, pick it up.

Still there? Am interested.

Yes,

Yes, pick

it

it

up.

up.

pick

Still

I’ll be there tomorrow.

Still

there? Am interested.

interested.there? Am

Ok bin morgen da.

Ja kannst du abholen.

Bereits verkauft? Habe Interesse.

Yes, pick it up.

I’ll be there tomorrow.

Still interested.there? Am

Cleanup

Regex +NGram Diff

Push-Notifications

Push-ServerPush-Server

Push-Server

(foo@bar.de , dev123)

Device-Registration

Push-Server

javapns gcm4j

GCMAPNS

‘hello world’ an ‘foo@bar.de’

‘foo@bar.de’ -> ‘dev123’

‘hello world’ an ‘dev123’

Push-Trigger

Push-Server

APNSGCM

‘hello world’ an ‘dev123’

Push-Receiving

Push-Server

javapns gcm4j

GCMAPNS

Rollout 1:Start Dec 2013

(~2 months)

Backend-ServerLoad

“Let’s try out new feature (swiping hell)…”Normal Traffic Pattern

Firefight Mode (CPU doubled)

Backend for Async-Requirements (Mail-Sending)

Backend for Sync-Requirements (User Facing Traffic)

Riak Read/Write

Riak Response-Times

iOSPush-Server

(APNS)

Push-Errors

Invalid Device-Tokens

Cause: Buggy Device Registration

Bug Fix

Push-Messages sent

Fix

Rollout

Android ≈ iOS.clone();

Rollout 2:

Start Feb 2014

(~2 weeks)

AndroidPush-Server

(GCM)

Push-Message Load

Push-Error Rate

GCM stability (502 Gateway)

Iteration 3:

Web Sockets?

Long Polling?

Old-School Polling!

5%

Rollout 3:Mid Feb. 2014

AB-Test

(~3 weeks)

factor 30x(-> but no impact)

Polling Bug

KPIs Revisited

MoreXY %

Visits / Day

Mobile: +15-20%Desktop:+5-10%

Reducedt1 + t2

Buyer Sellert1

t2

-50%

The Hourse, CC-license, https://flic.kr/p/eLbQcD

Positive App Reviews

+Adoption Rate

Email

iOS

Android

Desktop

...Iterate...

...Think X-Platform... ...Monitor...

...Live BizDevOps...

That’s it!

Questions?