real-time large data

@ra!deview - 17 september 2012

there are over400 million tweetsa day

a second4600 tweets

0.2 msa tweet≈

Pull Push

Targeted twitter.comhome_timeline API

User / Site StreamsMobile Push (SMS, etc.)

Queried Search API Track / Follow Streams

Write API

Fanout

Tim

elin

e C

ache

Redis

Ingester

Sear

ch C

ache

Redis

Push

Com

pute

HTTP Push

Bat

ch C

ompu

te

Hadoop

Timeline Service

Ingester

Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Redis

Social Graph Service

Timeline Service

Ingester

Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Redis

Social Graph Service

Ingester

Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Timeline Service

Redis

Bat

ch C

ompu

te

Hadoop

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

RedisEarlybirdMobile Push

STORAGE & RETRIEVAL

LOGICPRESENTATIONROUTING

Monorail

T-Bird

T-Flock + Haplo

Flock(s)

Darkwing

Macaw+Swift

Story Service

Macaw+Disco

STORAGE & RETRIEVAL

LOGICPRESENTATIONROUTING

T-Bird

T-Flock + Haplo

Flock(s)

Darkwing

Tweetypie

Gizmoduck

TLS

Social Graph Service

Monorail

Woodstar

TFE

timeline delivery statistics⇢30b deliveries / day (~20m / min)

⇢3.5 seconds @ p50 to deliver to 1m

⇢~350k deliveries / sec

#JoinTheFlock