YATOTOSOCIAL CROWD FUNDING PLATFORM

THE TECHNICAL PARTYou can find the most recent version online here

http://yatoto.gudaso�.com

BUSINESSLottery - Different kind of rewards

Facebook - Pages, Events, Friends, Chat,Main/private newsfeed

Ebay - Bidding on offers

Charities - Donations

Kickstater - Projects

Gamification - A lot of tricks to engage the users

NUMBERS (JUL 2013 - SEP 2016)

More than 250 000EUR for rewards

More than 100 000 Users Watched 9 Millon ads

Launched in Bulgaria and the United States

4 Designs changed

3 Devs, 1 frontend, graphic designer

1.3M Publications, 90M Stream Entries

800 Accomplished Missions for ~ 1 month

BACKEND

Easy to use backend interface.

Google Analytics on steroids

Play with A/B Testing

Smart Feature flapping

Track every event on the website

KNOWING OUR USERS

Ruby On Rails, PostgreSQL, Elasticsearch, Redis,Memcached, Message bus implementation onSidekiq

Ansible provisioning with multistage deploymentsvia Capistrano/Docker

Monitoring via Zabbix + Uptime Robot

Zero downtime deployment within our CI/CDprocess

TECHNOLOGY

THE APPLICATIONWe started with Ruby On Rails with PostgreSQL, Elasticsearch, Redis, Memcached.

At first we use QEMU Virtual Machine with 9GB of RAM and it works great with +100k users. Living ona virtual machine allow us to know every CPU cycle and every disk IO.

We expect 1m of users, thats why we scale on 3 servers with dockers.

We have 50Milion user stream entries. For our Newsfeed stream processing we use a pipeline overSidekiq. We have to send a lot of notifications in almost real time fashion.

THE FRONTENDFor the frontend we use Twitter bootsrap, and for the backend we used Zurb Foundation. Both were

good, +1 for Zurb for its usability on mobile devices.

We monitor the usage of every html element on the website with google analytics, because of thiswe know how the users interact with the website.

We decided to convert Yatoto to SPA with React. We made a small feature with React, then weremove it. The performance, and the SEO were poor, and we don't have human power to maintain it.

We kept the real time chat.

THE MOBILE APPWith Ionic 2, we shipped simple Yatoto application.

DEVELOPMENT

Always stay on the latest versions of everything.With Rails we start at 4.0.0.rc2 and end at 5.0.1.

Adopt new technology like Nodejs/React

Learn to delete code and write it from scratch

Low Technical Depth

THE DEVELOPER HAPPINESS

2 weeks project Yatoto a�er 3+ years

YES IT IS MONOLITH

We have to be fast because we deliver entire new functionality every week or 1/3/6 months. Wechange the product a lot! It is faster to refactor than micro services architecture.

We wrote Facebook, EBay, Quora, Kickstarter, GA, Games and Meme generator functionality sadly weforgot to double LinkedIn.

Compared to the we done it easer and faster.Biggest Monolith

METHODOLOGY

Scrum with a lot of XP

The team do pair programming, have coding standard.

Collective code ownership and simple design.

Code reviews were more effective than writing tests.

BDD, CI/CD, Small releases

SPECIFICATION

Each functionality is made in mock and polished there before it enters the project

PROCESS

3400+ Tickets

Sometimes we were lucky to work with bigger than one week milestones.

The tests 50min = 36min rspec + 13min cucumber

165 rubygems, (330 with dependencies)

external services: geoip, epay/banking, sendingsms

we publish open source gems for most of ourexternal services.

The Commits from 9 developers and 3 others

THE CODE

Comits Developer3781 I. B.2303 D. N.1721 R. B. 969 A. D. 923 A. Y. 879 I. T. 463 V. S. 323 V. N. 309 N. 65 The CEO :) 8 M. 0 K.

DEPLOYMENT

Initially was hosted on one virtual machine,

4CPU, 3GB RAM, serving > 100+K users

every ActiveRecord N+1 was noticeable

For the USA we launch 3 servers, expecting a lot oftraffic

no one came...

ORCHESTRATION WITH ANSIBLE

NetworkHost (Security, Accounts, ntp, basic packages)MailDockerWebserverZabbix/ZabbixAgents

ErrbitMemcachedRedisElasticsearchPostgresSamba

EACH WEBSITE IS ORCHESTRATED WITH ANSIBLE WITH THOSE PLAYBOOKS

A�er the initial effort of setup the Ansible provisioning there were no need for Sysadmin, the"configuration" was documented in the playbooks. We are stable and fast.

OUR SYSADMIN WELCOME BROCHURE

When he saw it, he run away - true story! Here is the title part of the brochure.

generates rails_best_practices, brakeman, rdoc,states machines diagrams once a daytried jenkins, gitlabci, cruisecontrol.rb, droneA�er successful build the CI deploys on testing.fails if we introduce library with strange license.

We deploy with the Vincent Driessen's branchingmodel. The development team uses testenvironments, and the business has stablestaging environment.

CI/CD

SPEED & AVAILABILITY

DEVELOPERS KNOW THE SPEED OF THE WEBSITE IN TEST/PRODUCTION

Web pages above 100ms are considered to be slow and those urls are redirected to separateapplication server dedicated to process them without hurting the overall performance. When the

pages are fixed they are returned to the fast application servers.

We are faster in discovering problems than Newrelic services.

ERRBIT

Monitor the errors from staging and testing environment so we know when someone hits the error.

The errors are redirected to rocket chat so that instantly knew about them

MONITORING

Zabbix installed on another data center is monitoring the website availability, the speed of the webpages and alerts via push notifications/chat the developers. UptimeRobot is monitoring Zabbix

Logcheck is reporting everthing unusual

OPTIMIZATIONS

Assets - Use separate IP (assets.yatoto.com) for faster loadingCache - Heavy caching on memcached and NginxReturn HTML ASAP - later process the request in the backgroundFight SPAM - Process the mail logs to find fake email accountsDragonfly behind Nginx with big cachepage preloading

SATISFACTIONThe project is huge, and we have been able to maintain and develop it during the years. There were

no noticeable downtimes. Maintaining the project was with very little effort because of theautomation and the tests. We have played with a lot of technologies