How ask.fm was built

Valery Vishnyakov

ask.fm/balepc

Covered topics

• Scaling Database

• Scaling Application

• Scaling Infrastructure

Started as a side project

• 0 users

• 1 server

• 3 tech guys

2010

Is ask.fm big?

Users Servers Tech guys

Ask.fm 135m 800 15

Twitter* 554m ~3500 1800

StackOverflow** 4m 25 24

* https://about.twitter.com/company ** http://bit.ly/1rEl8sX

No white spots

Initial stack

Scaling UP application

Not a sponsor slide

Scaling OUT application

User uploaded content

• 97k users

• 9 servers

• 3 tech guys

2011

Scaling database

Caching

Redis

Read scalability

Decision time

Middleware Logical partitioning

App-level logical partitioning

Preparation

• Choose Key + IDs

• Avoid JOINS

• Remove DB Constraints

Sharding schema

• node0 • shard0

• questions_by_user • shard1

• questions_by_user

• node1 • shard2

• questions_by_user • shard3

• questions_by_user • …

Maneuver

Adding more servers

DB Scaling takeaways

• Divide and conquer

• MySQL is good (schema change could be painful)

• Cache on all layers

• Scale out, not just up

• Denormalize

When 256 is not enough?

user_id % 256

2012

• 2M users

• 3 tech guys

• 46 servers

More on scaling application

Middleware

End of AWS era

New data center

Bare metal

Infrastructure evolution

Public Cloud => Bare metal => Private Cloud

SPAM

SPAM fighting evolution

‣ IP addresses

‣ Captcha

‣ Links

‣ Patterns

‣ Users

‣ Beneficiary

‣…

Moderation

How to moderate 4M photos every day?

Philippines

2013

• 31M users

• 440 servers

• 6 tech guys

Exponential growth

Functionality switches

• Media content

• Likes

• Wall

Trend Driven Development

TDD

Lessons learned

• Monitoring

• Incremental rollout

• Use battle-proven techniques

Android & iOS

2014

• 105M users

• 12 tech guys

• ~700 servers

Nowadays challenges

• DC High-Availability

• Fault-tolerance

• Scaling teams

• Do not over engineer

• Stay small

• Avoid SPOF

• Performance as a feature

• Plan for scaling out

Global takeaways

What could we do differently?

• More SOA

• Plan hiring ahead

• Don’t f**k up tests

Questions?