http://www.flickr.com/photos/booleansplit/2376359338/gareth rushgrove | morethanseven.net

You’re Going To Need ABigger Toolbox

DIBI 28th April 2010

Gareth Rushgrove

gareth rushgrove | morethanseven.net

morethanseven.net

We Used To Just Build Websites

gareth rushgrove | morethanseven.net

Probably Simple Websites

gareth rushgrove | morethanseven.net

Maybe the odd Ecommerce Site

gareth rushgrove | morethanseven.net

Then We Built Web Applications

gareth rushgrove | morethanseven.net

Now We Just Build.

gareth rushgrove | morethanseven.net

http://www.flickr.com/photos/gordonr/42555739

Your Toolbox is Growing

gareth rushgrove | morethanseven.net

Embrace Heterogeneous Environments

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/mk1971/2548492513/

1

No Development Tool Silver Bullet

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/mk1971/2548492513/

1

We Build Websites With...

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/freefoto/3436970425/

PHP, MySQL, Apache

gareth rushgrove | morethanseven.net

.NET, MSSQL, IIS

gareth rushgrove | morethanseven.net

Java, Oracle, Tomcat

gareth rushgrove | morethanseven.net

Lots of Others

gareth rushgrove | morethanseven.net

Just One Stack?

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/harrygoldenfeld/4263710200/

The Guardian Java, Python, Oracle, App Engine

gareth rushgrove | morethanseven.net

LastFM PHP, C++, Java, Hadoop, Python

gareth rushgrove | morethanseven.net

GitHub Ruby, Erlang, MySQL, Redis, Sinatra

gareth rushgrove | morethanseven.net

Twitter Ruby, Scala, Java, C/C++

gareth rushgrove | morethanseven.net

Facebook PHP, Erlang, C, MySQL, Cassandra

gareth rushgrove | morethanseven.net

2. Know When Your Stack is out of its Depth

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/dk_spook/2421009077/

2

What Should You Know About?

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/takomabibelot/220265100/

Serve Web Pages With Nginx

gareth rushgrove | morethanseven.net

wiki.nginx.org

Why Nginx

gareth rushgrove | morethanseven.net

Apache is like Microsoft Word, it has a million options but you only need six. Nginx does those six things, and it does five of them 50 times faster than Apache.Chris Lea

“

Nginx Example

gareth rushgrove | morethanseven.net

server { listen 80; server_name www.example.com;

location / { root /var/www/example.com; }}

http { upstream php { server localhost:8002; } upstream python { server localhost:8003; }}

server { server_name www.example.com; location / { proxy_pass http://python; } location ~ /basket/* { proxy_pass http://php; }}

Nginx Example

gareth rushgrove | morethanseven.net

Also See

gareth rushgrove | morethanseven.net

- Thin - http://code.macournoyer.com/thin/

- Mongrel - http://github.com/fauna/mongrel/

- Spawning - http://pypi.python.org/pypi/Spawning/

- Unicorn - http://github.com/defunkt/unicorn/

Caching with Memcached

gareth rushgrove | morethanseven.net

memcached.org

from django.core.cache import cache

key = "/about/"content = cache.get(key)if not content: # expensive query to get content cache.set(key, content, 300)

Memcached Example

gareth rushgrove | morethanseven.net

Also See

gareth rushgrove | morethanseven.net

- Squid - http://www.squid-cache.org

- Varnish - http://varnish-cache.org

Search with Solr

gareth rushgrove | morethanseven.net

lucene.apache.org/solr/

http://solr:8983/solr/products/select/ ?q=colour:red &sort=price%20desc &rows=60 &wt=json

Solr Example

gareth rushgrove | morethanseven.net

Also See

gareth rushgrove | morethanseven.net

- Xapian - http://xapian.org

- Sphinx - http://sphinxsearch.com

- Nutch - http://lucene.apache.org/nutch/

- Whoosh - http://whoosh.ca

Asynchronous Processing with RabbitMQ

gareth rushgrove | morethanseven.net

rabbitmq.com

RabbitMQ Enqueue Example

gareth rushgrove | morethanseven.net

require 'carrot'

q = Carrot.queue('carrot', :durable => true)10.times do |num| q.publish(num.to_s)end

RabbitMQ Dequeue Example

gareth rushgrove | morethanseven.net

puts "count: #{q.message_count}"

while msg = q.pop(:ack => true) puts msg q.ackend

Carrot.stop

Also See

gareth rushgrove | morethanseven.net

- Apache ActiveMQ - http://activemq.apache.org

- Beanstalk - http://kr.github.com/beanstalkd/

- Stomp Server - http://stomp.codehaus.org

- MemcacheQ - http://memcachedb.org/memcacheq/

gareth rushgrove | morethanseven.net

couchdb.apache.org

Data Storage With CouchDB

CouchDB Example

gareth rushgrove | morethanseven.net

<?php

$options['host'] = "localhost"; $options['port'] = 5984;

$couch = new CouchSimple($options); $resp = $couch->send("PUT", "/test"); $resp = $couch->send("PUT", "/test/123",

'{"_id":"123","data":"Foo"}'); ?>

<?php

$options['host'] = "localhost"; $options['port'] = 5984;

$couch = new CouchSimple($options); $resp = $couch->send("GET", "/test/_all_docs"); $resp = $couch->send("GET", "/test/123"); $resp = $couch->send("DELETE", "/test/"); ?>

CouchDB Example

gareth rushgrove | morethanseven.net

Also See

gareth rushgrove | morethanseven.net

- MongoDB - http://www.mongodb.org

- Tokyo Tyrant - http://1978th.net/tokyotyrant/

- Cassandra - http://cassandra.apache.org

- Redis - http://code.google.com/p/redis/

gareth rushgrove | morethanseven.net

hadoop.apache.org

Data Mining With Hive

CREATE TABLE u_data ( userid INT, movieid INT, rating INT, unixtime STRING)ROW FORMAT DELIMITEDFIELDS TERMINATED BY '\t'STORED AS TEXTFILE;

Cucumber DSL Example

gareth rushgrove | morethanseven.net

Hive Create Example

gareth rushgrove | morethanseven.net

LOAD DATA LOCAL INPATH 'data.txt' OVERWRITE INTO TABLE u_data;

SELECT COUNT(1) FROM u_data;

Cucumber DSL Example

gareth rushgrove | morethanseven.net

Hive Load Example

gareth rushgrove | morethanseven.net

Cucumber DSL Example

gareth rushgrove | morethanseven.net

Hive Map Reduce Example

gareth rushgrove | morethanseven.net

add FILE weekday_mapper.py;

INSERT OVERWRITE TABLE u_data_newSELECT TRANSFORM (userid, movieid, rating, unixtime) USING 'python weekday_mapper.py' AS (userid, movieid, rating, weekday)FROM u_data;

SELECT weekday, COUNT(1)FROM u_data_newGROUP BY weekday;

Also See

gareth rushgrove | morethanseven.net

- Hadoop - http://hadoop.apache.org

- Hive - http://wiki.apache.org/hadoop/Hive/

- Pig - http://hadoop.apache.org/pig/

- Dumbo - http://lastfm.com/dumbo/

- Disco - http://discoproject.org

Testing with Cucumber

gareth rushgrove | morethanseven.net

cukes.info

Feature: google.co.uk To broaden their knowledge A user should be able To search for things Scenario: Searching for things Given I visit "http://www.google.co.uk" When I fill in "q" with "wikipedia" And I press "Google Search" Then I should see "www.wikipedia.org"

Cucumber DSL Example

gareth rushgrove | morethanseven.net

Feature: google.co.uk To broaden their knowledge A user should be able To search for things Scenario: Searching for things Given I visit "http://www.google.co.uk" When I fill in "q" with "wikipedia" And I press "Google Search" Then I should see "www.wikipedia.org"

1 scenario (1 failed)4 steps (1 failed, 2 skipped, 1 passed)0m0.332s

Cucumber Results Example

gareth rushgrove | morethanseven.net

Server Provisioning with Puppet

gareth rushgrove | morethanseven.net

puppetlabs.com

Puppet Class Example

class baseclass { $packagelist = ["sudo", "openssh-server"] package { $packagelist: ensure => installed } service { sshd: name => "ssh", enable => true, ensure => running }}

gareth rushgrove | morethanseven.netgareth rushgrove | morethanseven.net

Puppet Node Example

gareth rushgrove | morethanseven.netgareth rushgrove | morethanseven.net

node 'example.com' inherits basenode { $packagelist = ["nginx"] package { $packagelist: ensure => installed } service { "nginx": ensure => running, require => Package["nginx"] }}

Also See

gareth rushgrove | morethanseven.net

- Chef - http://wiki.opscode.com/display/chef/Home/

http://www.flickr.com/photos/batega/1596898776/

Conclusions

gareth rushgrove | morethanseven.net

Know What's Possible

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/docman/36125185/

1

http://www.flickr.com/photos/kgregory/500456103/

Look for Opportunities

gareth rushgrove | morethanseven.net

2

Experiment

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/seeminglee/3967329241/

3

Manage Complexity

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/30890318@N06/3510161637/

4

Questions?

gareth rushgrove | morethanseven.net http://flickr.com/photos/psd/102332391/