Porting Rails Apps to High Availability Systems
-
Upload
marcelo-pinheiro -
Category
Technology
-
view
2.997 -
download
9
description
Transcript of Porting Rails Apps to High Availability Systems
Porting Rails apps for High Availability
SystemsMarcelo Correia Pinheiro
@salizzar
$ whoami
Programmed with ASP, PHP, C#, Java, Python, Ruby, etc etc
Twitter: @salizzar
http://salizzar.net/ (always under construction)
Locaweb PaaS/IaaS Engineering Team (Domain Registration)
TopicsCommon Architecture
Application Modularity
Useful Gems
Database Replication
Session Management
App Distribution and Deployment
Configuration Management
Load Balancing
Conclusion
Common Architecture
One machine running HTTP Server, App and Database (sometimes in a dedicated server)
Production config files in application GIT repo (all of us use git, alright?)
Deploy via Capistrano / Vlad / * ?
Sufficient for small apps with no perspective to grow absurdly or have monitored growing
Grants HA? Hmmm... what do you do if your database corrupts journaling? (MongoDB REAL CASE)
Common Architecture
Think about it and the possibility of a nightmare due for a bad db query, zombie processes and unexpected errors that will frozen your app
What to do? Look at your app with a Distributed System perspective
Application Modularity
Rails apps, in most cases, can be splitted to two or more small gems or apps (web services, checkout systems, asynch processing, etc)
Application Modularity
You write good code?
If yes, creating gems with it is EASY.
How to detect: first DRY while splitting a app
How to create: $ bundle gem <asdfg>
How to learn more: Creating Your Own Gem in www.rubygems.org
If not, refactoring!
Application Modularity
Breaking your Rails app in smaller apps:
increase future improvements
first step to make distribution via packaging more easy
consequently, more scalable
Look at ActiveController::Metal / sinatra / rack-api / goliath / cramp
Application Modularity
Be careful:
using multiple gems
coupling Rails features inside your models (Rails.logger)
Useful Gems
Hypermedia
Asynchronous Processing (resque)
Webservers
Monitoring
Useful Gems
JSON for the win
Hypermedia-based REST API’s
ROAR: https://github.com/apotonick/roar
RestModel: https://github.com/rodrigues/rest_model
Useful Gems# -*- encoding: UTF-8 -*-
class Domain attr_accessor :id, :creation_date, :expiration_dateend
module DomainRepresenter include Roar::Representer::JSON, Roar::Representer::Feature::Hypermedia
property :id, :creation_date, :expiration_dateend
request = RestClient.get 'http://an.app/domains/example.com'domain = Domain.new.extend DomainRepresenterdomain.from_json request.body
Useful Gems
# -*- encoding: UTF-8 -*-
class Domain < RestModel property :id, type: String property :creation_date, type: Date property :expiration_date, type: Date property :provisioning_id, type: Integerend
request = RestClient.get 'http://an.app/domains/example.com'domain = Domain.from_source(request.body).first
Useful GemsAsynchronous Processing
Resque
resque-scheduler
resque-retry
resque-lock
resque-logger
Useful Gems
resque-scheduler:
https://github.com/bvandenbos/resque-scheduler
cron-based Resque extension
Useful Gems
Incluir config de cron
# resque_scheduler.yml
appointment_notification_enqueuer: cron: "0 4 * * *" class: AppointmentNotificationEnqueuer queue: appointment_notification_enqueuer args: email: true sms: true description: Notify today appointments to customers
free_slot_monitor: cron: "0/15 * * * *" class: FreeSlotMonitor queue: free_slot_monitor description: Check for free slots in agenda
Useful Gems# -*- encoding: UTF-8 -*-
class AppointmentNotificationEnqueuer @queue = :appointment_notification_enqueuer
def self.perform(args) appointments = Appointment.today factory = NotificationFactory.new args appointments.each do |appointment| factory.enqueue appointment.id end endend
Useful Gemsresque-retry:
https://github.com/lantins/resque-retry
redis backed
retry count times until reach a limit
retry on all or specific exceptions
retry functionality on Resque Web Server
Useful Gems# -*- encoding: UTF-8 -*-
class EmailNotificationWorker extend Resque::Plugins::Retry
@queue = :email_notification_enqueuer @retry_limit = 4 @retry_delay = 300
def self.perform(args) appointment = Appointment.find id: args['id']
mailer = MailNotification.new mailer.notify_appointment appointment endend
Useful Gems
Useful Gems
resque-lock:
https://github.com/defunkt/resque-lock
Grants only one instance of a job running in a time
Useful Gems
# -*- encoding: UTF-8 -*-
class SmsNotificationWorker extend Resque::Plugins::Lock
def self.perform(args) appointment = Appointment.find id: args['id']
sender = SmsNotification.new sender.deliver_appointment appointment endend
Useful Gems
resque-logger:
https://github.com/salizzar/resque-logger
Provides a logger for each Resque worker based on queue name
Log Driven Development, please!
Useful Gems
# config/initializers/resque.rb
log_path = Rails.root.join 'log'
config = { folder: log_path, class_name: Logger, class_args: [ 'daily', 1.gigabyte ], level: Logger::INFO, formatter: Logger::Formatter.new,}
Resque.logger = config
Useful Gems# -*- encoding: UTF-8 -*-
class EmailNotificationWorker extend Resque::Plugins::Retry, Resque::Plugins::Logger
@queue = :email_notification_enqueuer @retry_limit = 4 @retry_delay = 300
def self.perform(args) appointment = Appointment.find id: args['id']
logger.info(“trying to notify #{appointment.id}”)
mailer = MailNotification.new mailer.notify_appointment appointment endend
Useful GemsWeb Servers:
Thin
https://github.com/macournoyer/thin
EventMachine powered
Killer combo with Nginx
Unicorn
http://unicorn.bogomips.org
Pre-fork based
Great with low-latency network
Useful GemsMonitoring
God
https://github.com/mojombo/god
Several memory leaks at beginning, very stable today
Largely used to manage Resque workers and Thin clusters
Bluepill
https://github.com/arya/bluepill
More simple than God, less features
Some issues with signal handling
Useful Gems
If memory usage is a problem, use Monit.
http://mmonit.com/monit
VERY stable
HIGH expansible
EASY configuration
Database ReplicationRelational Database Replication is not trivial
Be a friend of a sysadm
CAP Theorem: YOU NEED TO KNOW
No magic: look to your architecture
Traditional replication strategies:
master / slave for consistency
master for write / slave for read
Database Replication
My experience:
Relational Databases: no choice
MongoDB: replica set
Redis: master / slave
Database ReplicationFor MongoDB:
MongoMapper
https://github.com/jnunemaker/mongomapper
Largely used, stable
Mongoid
https://github.com/mongoid/mongoid
ActiveRecord-like
Database Replication
brotip #1342: check if your MongoDB client detects when master server goes down automagically (MongoMapper NOT does it)
Suggested keynote: MongoDB em Produção, dá pra confiar?
http://www.slideshare.net/deferraz/mongodb-em-producao
Session Management Given a problem, use the right tool PLEASE
memcached
http://memcached.org
distributed memory object caching system
key/value storage
stores arbitrary data
can be clustered
Session Management
repcached
http://repcached.lab.klab.org
patch inside memcached to add data replication
offers session sharing between memcached servers
Session Managementavailable gems for memcache
memcache-client
https://github.com/mperham/memcache-client
First Ruby client, written by Mike Perham
memcached
https://github.com/evan/memcached
Ruby SWIG wrapper for libmemcached C library
dalli
https://github.com/mperham/dalli
Major refactoring from memcache-client
App Distribution and Deployment
Capistrano? Hnmm... talk with a sysadm about it. It works but lacks security severely (SSH, GCC in production is a invite for attack)
Sysadms knows process monitoring tools, software distribution and server configuration better than us
Be a friend of a sysadm
Or LEARN A LOT to be one :)
App Distribution and Deployment
Build self-packaged apps
$ man dh_make (for Debian based distros)
Hard or too boring? Maybe, but effort compensates
$ dh_make --native --simple --indep --packagename <yourapp> --copyright blank
$ debuild
# dpkg -i <yourapp>_<version>_<arch>.deb
App Distribution and Deployment
Generating a distro-based package with your app grants fast update/rollback to sysadms
# apt-get update && apt-get install <yourapp>
# dpkg -i <your_app_package>
If you want to automate builds:
Setup a repository server
Use Bricklayer and be happy
Configuration Management
Keeping production configs in your app git repo is not always a good idea
Getting config files over control is VERY important to avoid a messy environment
Standardization is king in server configuration
Configuration Management
At this time, three choices:
Puppet
Chef
cfengine
Configuration Management
Puppet
https://github.com/puppetlabs/puppet
Written in Ruby
Large community adoption
Great number of recipes
Works well
Configuration Management
Chef
https://github.com/opscode/chef
Written in Ruby
Created as a Puppet alternative in order to solve some problems with it
Large community adoption
Works good
Configuration Management
cfengine
http://cfengine.com
Written in C
Largely adopted by sysadms
HARD to learn
F****** ROBUST (a single master node can manage 4K servers)
WTF is Load Balancing?
A network methodology to distribute workload across multiple computers in a transparent way for users
Can be proved by /hard|soft/ware
Alternatives:
LVS with keepalived
HAProxy with heartbeat
Load Balancing
Load BalancingLVS (Linux Virtual Server)
http://www.linuxvirtualserver.org
Robust, high scalable solution
Need to share HTTP sessions? It’s the choice
Built-in Linux Kernel module since 2.4.28*, runs in layer-4 (transport layer)
keepalived
http://www.keepalived.org
Largely used with LVS by sysadms
Uses Virtual Router Redundancy Protocol (vrrp) - RFC
Load BalancingHAProxy
http://haproxy.1wt.eu
Generic solution
Works if session sharing between servers is not required
Runs in layer-7 (application layer), computationally more expensive
heartbeat
http://www.linux-ha.org/wiki/Heartbeat
Based on message exchange between clients in a cluster to detect failures
ConclusionPorting Rails apps to a HA environment is a not trivial challenge, but nothing insanely hard
Forces you to think in application architecture, design, distribution, modularity
brotip #915435: do it before coding
Restricts you in order to make server correctly managed
Implies you to be prudent on a deploy
“it sucks at all” = bad excuse, my friend
App tunning later, without fear :)
In fact, in HA deploys are more faster
Questions?
No fear, answers are free as beer :)
Thank you!