Rubymanor - Nanite talk
-
Upload
georgio1999 -
Category
Technology
-
view
4.259 -
download
0
Transcript of Rubymanor - Nanite talk
![Page 1: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/1.jpg)
VetebraNanite
George Palmer
![Page 2: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/2.jpg)
Background processing
• Work should be moved to the background to stop application server load
• This keeps website responsive
• Useful for: images, videos, web services, slow database queries....anything that isn’t quick!
![Page 3: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/3.jpg)
![Page 4: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/4.jpg)
![Page 5: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/5.jpg)
Current background offerings
• Fork into background of rails process
• eg spawn, run_later
• Record to database (or file) and background daemon picks up from there
• eg background_job, delayed_job
• Fork onto some kind of queue
• eg backgroundRB, beanstalk, starling with workling
![Page 6: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/6.jpg)
Introducing Nanite
• Developed by EngineYard
• Gives presence - we know what’s available
• Handles failure
• Instant scalability
• Event based architecture
![Page 7: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/7.jpg)
Pre-reqs
• Erlang
• Not R12B5 (the version installed by MacPorts)
• RabbitMQ
• Untar to erlang lib directory - most likely /usr/local/lib/erlang/lib
![Page 8: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/8.jpg)
AMQP
• Advanced Message Queuing Protocol
• Enterprise quality protocol developed by several financial institutions
• Includes wire protocol to ensure language neutral
• RabbitMQ implements AMQP 0.8
![Page 9: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/9.jpg)
Installing AMQP for Ruby
$ sudo gem install eventmachine
$ git clone git://github.com/tmm1/amqp.git
$ cd amqp && rake gem && sudo gem install amqp-<version>.gem
![Page 10: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/10.jpg)
![Page 11: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/11.jpg)
NaniteThinApp
Server
AMQP
Nanite
Nanite Mappers Nanite Agents
RubyScript
Nanite
AMQP
AMQP
NaniteActor
NaniteActor
AMQP
NaniteActor
NaniteActor
m e s s a g e s
![Page 12: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/12.jpg)
Nanite Mappers
• Control and track work
• Unlimited number can be run that get updates from mapper exchange
• mapper exchange itself is just a heartbeat and registration MQ
• Run either inside Rails/MERB app (on Thin) or via command line
![Page 13: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/13.jpg)
Nanite Agents
• Do the work
• A given nanite agent can have multiple actors
• Scale by adding more agents
• Pings the mapper exchange every @ping_time seconds to report health
![Page 14: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/14.jpg)
Nanite Actors
class Manor < Nanite::Actor expose :name
def name(vars) # Do something interesting here :result => “RubyManor” endend
Nanite::Dispatcher.register(Manor.new)
![Page 15: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/15.jpg)
Agent directory structure
+ myagent
+ actors
- manor.rb
+ files
- init.rb
- config.yml
![Page 16: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/16.jpg)
Agent config.yml---
:vhost: /nanite # Allow multiple agents with different queues [compulsory]
:user: nanite # Username for queue
:pass: testing # Password for queue
:identity: barney # Can be auto-generated but useful to send work to specific
# agents
:file_root: path # where to store any transfered files
:format: marshal # or :json
# Additional options include host and port. All options can be passed into
# nanite command so can avoid config file if want
![Page 17: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/17.jpg)
![Page 18: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/18.jpg)
Getting Started...
• Start RabbitMQ
• /usr/local/lib/erlang/lib/rabbitmq/sbin/rabbitmq-server
• On first run nanite/bin/rabbitconf
• Sets up RabbitMQ with a vhost and users for that vhost (more on this later)
![Page 19: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/19.jpg)
Starting agents
• cd <agentdir> && <nanite basedir>/bin/nanite
• I’ve been using:cd <agentdir> && <nanite basedir>/bin/nanite -t <identity> &
• Could be managed better through a daemon/monitoring system though
![Page 20: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/20.jpg)
Offloading work to Nanite
• Use the following code:Nanite.request(callable, params = ‘’, options = {}) {|res| # use res to do something}
• callable is the actor and method - eg ‘/manor/name’
• params are parameters for the callable method - eg ‘2008’
• options includes timeout, target and the routing algorithm - more on this later
![Page 21: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/21.jpg)
Interfaces into the Mappers
• Via console:
• nanite/nanite-mapper -i
• Via command line:
• See nanite/examples/cli.rb
• Via Rails/MERB app:
• See next slide....
![Page 22: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/22.jpg)
Rails/MERB & Nanite# Updates the userdef update ... if (@user.save) Nanite.request(‘/updates/twitter’, ‘georgio_1999’) {|res| # This block won’t execute until the event fires @user.status = res[:status] @user.save } endend
def ajax_call # Must use database for state and not Nanite jobend
![Page 23: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/23.jpg)
![Page 24: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/24.jpg)
Allocation of work
• The pings are used by the mappers to find the healthiest nanite agents
• If a nanite agent doesn’t ping inside a window it is removed from the active list (until it does ping again)
• perfect for busy or error hit nanites
• The default routing algorithm is based on server load
![Page 25: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/25.jpg)
Routing options
• In options argument of Nanite.request you can choose:
• selector
• :least_loaded (default), :all, :random, :rr
• target - use this to target a specific agent
• timeout - override the default timeout (60s)
![Page 26: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/26.jpg)
Custom algorithms
• Nanites report state with their ping
• By default this is the server load
• Can override this by adding code to agent init.rb
• Nanite.status_proc = lambda { MyApp.some_statistic_indicating_load }
• Must be comparable
• Can use with existing routing algorithms or create own more complex ones
![Page 27: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/27.jpg)
![Page 28: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/28.jpg)
File transfers
• Nanite can handle file transmission
• Agents subscribe (for all actors) in init.rb or Actors subscribe individually:
• Nanite.subscribe_to_files(domain)
• Mappers send via:
• Nanite.broadcast_file(filepath, opts)
• where opts can contain :domain and :destination (destination filename)
![Page 29: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/29.jpg)
![Page 30: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/30.jpg)
The beauty of JSON
• Nanite is built on top of AMQP, so if the queue items are serialised using JSON...
• Then Nanite isn’t needed at the agent side
• Any AMQP implementing daemon can read message and respond
• Useful for legacy code (or legacy people)
![Page 31: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/31.jpg)
![Page 32: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/32.jpg)
Understanding Security
• Security is implemented using RabbitMQ’s vhosts and username/passwords
• The username/password is defined in config.yml for each nanite
• Need to configure RabbitMQ using the rabbitmqctl command
• Generally one vhost per application
![Page 33: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/33.jpg)
![Page 34: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/34.jpg)
Let’s play
• cd <git_resources>/nanite/examples/myagent
• ../../bin/nanite -t <identity> -h <host>
• Let’s use fullname for identity - eg georgepalmer
• Should see something like:# subscribing to file broadcasts for foobar
# loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/gems.rb
# loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/mock.rb
# "advertise_services"
# ["/gems/list", "/mock/list"]
![Page 35: Rubymanor - Nanite talk](https://reader034.fdocuments.in/reader034/viewer/2022042814/5552c1f4b4c90581158b4833/html5/thumbnails/35.jpg)
Resources
• RabbitMQ: http://www.rabbitmq.com
• Ruby AMQP with RabbitMQ tutorial: http://hopper.squarespace.com/blog/2008/7/22/simple-amqp-library-for-ruby.html
• Nanite: http://github.com/ezmobius/nanite
• Doc isn’t great, code is very readable