Wongnai Engineering Story

Wongnai Engineering Story

Pattrawoot Suesatayasilp @pattrawoots

CTO & Co-founder @ Wongnai

Wongnai• A startup company

• Our mission is to help Thai people discover and connect to great local businesses.

• We believe in trials and errors.

• Time is our most valuable resource.

What It Means To Be Wongnai

WongnaiUser-Generated Content Website

CMS, SEO

Search

Location-Based Service

Social Network

Feed, Follow, Like, Comment

Commerce

Deals, Payment

Web & App

Statistics

300k sessions/day 53M page views/month

20k requests/min (3x during holidays)

Statistics

80%

20%

DesktopMobile (App/Web)

34%

66%

Mobile AppMobile Web

• During meal — Lunch, Dinner

• Push Notifications

• Social Post / Dramas

High Traffic

DinnerLunch

Push

• Weekends especially the long ones

• Special Occasion (Valentine’s)

• Holidays

High Traffic

Chinese New Year Valentine’s

Long WeekendSat - Sun

Our 1st dev server

Apple iMac Core 2 Duo 2.0GHz + DynDns

Our 1st prod server

Dell PowerEdge R210 ฿30,xxx

Move To The Cloud

2010-2013 User Growth

0

250000

500000

750000

1000000

Oct '10 Mar '11 Aug '11 Jan '12 Jun '12 Nov '12 Apr '13 Sep '13

Sep 2012 200k subscribers

Email Blast• MailChimp until it became too

expensive. (almost 200k subscribers)

• Set Up our own mail server

• OpenEMM

• No additional cost but many problems

Email Blast• Slow send rate — Takes too long to send all

• Consume too much bandwidth

• Poor open rate — Some went to Junk

Amazon SES

• Send Rate (emails/second)

• Up to 90 emails/second

• Whitelist IP Addresses

• Migrating to SES wasn’t quite smooth.

Amazon SES

SEO & Scalability• Large number of web pages (> 100k pages)

expecting Google bot to crawl

• Poor latency and not quite reliable (e.g. power outage)

• 2013’s goal is to reach 1M users. Be ready to scale.

• Cloud computing is the way to go.

Migrate to AWS• Million of photos to

move to the cloud

• Near-zero downtime

• Installed mobile apps must continue seamlessly.

Migrate to AWSAmazon Web ServicesHost in TH

www.wongnai.com photo.wongnai.com

Static Content Servers

Web Server

Database


Web Server

Upload photos gradually

Database

Update ‘synced’ flag



Web Server

Database


www.wongnai.co

m

1


Web Server

Database


photo.wongnai.

com

Identify photo’s location

2

3


Web Server

Database


Redirect

to th

e old host

4


pull not-synced-yet

images

Web Server

Database


5


Web Server

Database


www.wongnai.co

m

1


Web Server

Database


photo.wongnai.

com

2

3

Identify photo’s location


Web Server

Database


Redirect

to S3

4


Web Server

Database

pull already-synced images


5

Database & ORM

When a list of reviews are displayed

a review • review’s details

• user (reviewer) • location • photos • likes

• preview comments • and many more …

Select N+1 Problem

ReviewerReview Details

Review

PhotoPhotoPhotoPhotoPhotoPreview Comments

Location

Many tables needs to be joined to get enough info for displaying a review.

Large Table JoiningReview

500,000 rows

User

2,000,000 rows

Location

200,000 rows

Photo

3,000,000 rows

Large Table Joining

• As a result of experiments, selecting rows separately without table joining performs much better. (No bottleneck at the database) — Feel the NoSQL?

Single SQL command

Multiple SQL commandVS

Schema Changes

• Mid 2015 — We decided to combine Food and Beauty into a single app

• Tables need to be merged. Columns need to be added. Millions of rows are affected.

What we expected

large table

What actually happens

create new tablelarge table

What actually happens

create new tablelarge table

copy

• A very large number of I/O operations need to perform to update the schema

I/O Speed• Storage for Amazon RDS (Amazon EBS)

• IOPS — The number of I/O operations completed per second)

• Base Perf: 300 IOPS (for 100GB storage)

• Burst 3,000 IOPS Maximum — Non-provisioned IOPS Storage

• Burst duration is limited.

• Error 404 — almost 24hrs

• Liquibase and many other schema migration tools

• Only one column can be added at a time!

Why still SQL?• ACID Transaction

• NoSQL is not good at searching. (Compared to RDBMS)

• NoSQL was not popular yet, and also still lacked many key features that SQL has. (MongoDB was just one-year old. Cassandra is still in beta.)

Search

Types of Search

Any good restaurants near me?

Show me all branches of Sushi Hiro

Show me all restaurants in Central World

Search for “Bonchon”

Indexing• Apache Solr

• Fast read but slow write

• Our indexes get updated very often, and they affects ranking — e.g. new review/photo/check-in, create new place, edit place’s info, etc.

• Tried updating indexes real-time and Solr’s soft-commit but the performance was not satisfied.

• So, we add another layer to do batch processing.

Indexing

• Why not Amazon Elasticsearch Service?

• Too difficult for testing

• We don’t want to stick with AWS.

• Hybrid cloud computing?

Cache• EHCache

• Most of the content are location-based.

• What should be cached? article, guide

• Content keeps updating

• e.g. Your review may get liked right after posted.

• Most of our caches have very short life. (< 1 min) and never get invalidated. Shared cache is not needed.

Scheduled Task• Quartz (by Terracotta) — Job Scheduler

• Score Updater for calculating ranking

• Business, User, Review, Photo, Activity, etc.

• Leaderboard

• Statistical Data Updater

• Number of Photos/Reviews/Check-ins/…

• etc.

Web Stack

เร็วส์Java@roofimon

“ ”

Web Stack

Java Spring SolrMySQL Hibernate Cassandra

RequireJSjQuery Backbone Less Bootstrap

Infra/DevOps

Tomcat

Git

Jenkins

DockerMaven

ZookeeperHTTP Server

Architecture

Solr Cloud

HTTP/HTTPS Load Balancer

Amazon RDS

ZooKeeper Cluster

Cassandra Cluster


Monolithic Application

Servers

Amazon SES

• 6 year ago, we were new to web development.

• Default Templating System — JSP

• No pre-processor was there back then, so we

wrote our own pre-processors in Maven Lifecycle

(Phase: Prepare package).

• Minify JS, CSS files — 1 file per page.

• YUI Compressor

Web Stack (Frontend)

• Asynchronous Module Definition • RequireJS

• Why not Angular? • SEO is one of the most important thing for a

content website like Wongnai.com • JavaScript SEO (prerender.io / …)

• More code to maintain • Code refactoring takes time. Not worth for

now

Web Stack (Frontend)

http://prerender.io

• Sonar (now SonarQube) • No longer use it as we found it doesn’t help

us that much. Our dev team is relative small. Manual code review is far better.

• Jenkins • No distributed build yet. • Only used on dev environment

DevOps

What’s Next• Personalization

• Machine Learning / Scala / Spark

• NoSQL for data that doesn’t require searchability

• e.g. Feed, Notifications

• Advanced Message Queuing Protocol (AMQP)

• RabbitMQ

• Microservices

• Image Server

Process & Tools• Asana, Slack, Google Docs

• Agile

• Manual First

• TDD

• Daily Scrum

• 2-week Sprint (1-month Sprint before)

• Retrospect

Dev Team 6 Full-Stack

Web Developers

3 iOS Developers + 1 Part-timer

2 Android Developers

1 QA Engineer 1 UI/UX Designer

Dev Team

We are Hiring!

6 Full-Stack Web Developers

3 iOS Developers + 1 Part-timer

2 Android Developers

1 QA Engineer 1 UI/UX Designer

Q&A

Q&AJoin Us!

wongnai.com/careers

Wongnai Engineering Story

Engineering

Transcript of Wongnai Engineering Story