You're Going To Need A Bigger Toolbox

http://www.flickr.com/photos/booleansplit/2376359338/gareth rushgrove | morethanseven.net

You’re Going To Need ABigger Toolbox

DIBI 28th April 2010

Gareth Rushgrove

gareth rushgrove | morethanseven.net

morethanseven.net

We Used To Just Build Websites

Probably Simple Websites

Maybe the odd Ecommerce Site

Then We Built Web Applications

Now We Just Build.

http://www.flickr.com/photos/gordonr/42555739

Your Toolbox is Growing

Embrace Heterogeneous Environments

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/mk1971/2548492513/

No Development Tool Silver Bullet

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/mk1971/2548492513/

We Build Websites With...

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/freefoto/3436970425/

PHP, MySQL, Apache

.NET, MSSQL, IIS

Java, Oracle, Tomcat

Lots of Others

Just One Stack?

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/harrygoldenfeld/4263710200/

The Guardian Java, Python, Oracle, App Engine

LastFM PHP, C++, Java, Hadoop, Python

GitHub Ruby, Erlang, MySQL, Redis, Sinatra

Twitter Ruby, Scala, Java, C/C++

Facebook PHP, Erlang, C, MySQL, Cassandra

2. Know When Your Stack is out of its Depth

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/dk_spook/2421009077/

What Should You Know About?

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/takomabibelot/220265100/

Serve Web Pages With Nginx

wiki.nginx.org

Why Nginx

Apache is like Microsoft Word, it has a million options but you only need six. Nginx does those six things, and it does five of them 50 times faster than Apache.Chris Lea

Nginx Example

server { listen 80; server_name www.example.com;

location / { root /var/www/example.com; }}

http { upstream php { server localhost:8002; } upstream python { server localhost:8003; }}

server { server_name www.example.com; location / { proxy_pass http://python; } location ~ /basket/* { proxy_pass http://php; }}

Nginx Example

Also See

- Thin - http://code.macournoyer.com/thin/

- Mongrel - http://github.com/fauna/mongrel/

- Spawning - http://pypi.python.org/pypi/Spawning/

- Unicorn - http://github.com/defunkt/unicorn/

Caching with Memcached

memcached.org

from django.core.cache import cache

key = "/about/"content = cache.get(key)if not content: # expensive query to get content cache.set(key, content, 300)

Memcached Example

Also See

- Squid - http://www.squid-cache.org

- Varnish - http://varnish-cache.org

Search with Solr

lucene.apache.org/solr/

http://solr:8983/solr/products/select/ ?q=colour:red &sort=price%20desc &rows=60 &wt=json

Solr Example

Also See

- Xapian - http://xapian.org

- Sphinx - http://sphinxsearch.com

- Nutch - http://lucene.apache.org/nutch/

- Whoosh - http://whoosh.ca

Asynchronous Processing with RabbitMQ

rabbitmq.com

RabbitMQ Enqueue Example

require 'carrot'

q = Carrot.queue('carrot', :durable => true)10.times do |num| q.publish(num.to_s)end

RabbitMQ Dequeue Example

puts "count: #{q.message_count}"

while msg = q.pop(:ack => true) puts msg q.ackend

Carrot.stop

Also See

- Apache ActiveMQ - http://activemq.apache.org

- Beanstalk - http://kr.github.com/beanstalkd/

- Stomp Server - http://stomp.codehaus.org

- MemcacheQ - http://memcachedb.org/memcacheq/

couchdb.apache.org

Data Storage With CouchDB

CouchDB Example

$options['host'] = "localhost"; $options['port'] = 5984;

$couch = new CouchSimple($options); $resp = $couch->send("PUT", "/test"); $resp = $couch->send("PUT", "/test/123",

'{"_id":"123","data":"Foo"}'); ?>

$options['host'] = "localhost"; $options['port'] = 5984;

$couch = new CouchSimple($options); $resp = $couch->send("GET", "/test/_all_docs"); $resp = $couch->send("GET", "/test/123"); $resp = $couch->send("DELETE", "/test/"); ?>

CouchDB Example

Also See

- MongoDB - http://www.mongodb.org

- Tokyo Tyrant - http://1978th.net/tokyotyrant/

- Cassandra - http://cassandra.apache.org

- Redis - http://code.google.com/p/redis/

hadoop.apache.org

Data Mining With Hive

CREATE TABLE u_data ( userid INT, movieid INT, rating INT, unixtime STRING)ROW FORMAT DELIMITEDFIELDS TERMINATED BY '\t'STORED AS TEXTFILE;

Cucumber DSL Example

Hive Create Example

LOAD DATA LOCAL INPATH 'data.txt' OVERWRITE INTO TABLE u_data;

SELECT COUNT(1) FROM u_data;

Hive Load Example

Hive Map Reduce Example

add FILE weekday_mapper.py;

INSERT OVERWRITE TABLE u_data_newSELECT TRANSFORM (userid, movieid, rating, unixtime) USING 'python weekday_mapper.py' AS (userid, movieid, rating, weekday)FROM u_data;

SELECT weekday, COUNT(1)FROM u_data_newGROUP BY weekday;

Also See

- Hadoop - http://hadoop.apache.org

- Hive - http://wiki.apache.org/hadoop/Hive/

- Pig - http://hadoop.apache.org/pig/

- Dumbo - http://lastfm.com/dumbo/

- Disco - http://discoproject.org

Testing with Cucumber

cukes.info

Feature: google.co.uk To broaden their knowledge A user should be able To search for things Scenario: Searching for things Given I visit "http://www.google.co.uk" When I fill in "q" with "wikipedia" And I press "Google Search" Then I should see "www.wikipedia.org"

1 scenario (1 failed)4 steps (1 failed, 2 skipped, 1 passed)0m0.332s

Cucumber Results Example

Server Provisioning with Puppet

puppetlabs.com

Puppet Class Example

class baseclass { $packagelist = ["sudo", "openssh-server"] package { $packagelist: ensure => installed } service { sshd: name => "ssh", enable => true, ensure => running }}

gareth rushgrove | morethanseven.netgareth rushgrove | morethanseven.net

Puppet Node Example

gareth rushgrove | morethanseven.netgareth rushgrove | morethanseven.net

node 'example.com' inherits basenode { $packagelist = ["nginx"] package { $packagelist: ensure => installed } service { "nginx": ensure => running, require => Package["nginx"] }}

Also See

- Chef - http://wiki.opscode.com/display/chef/Home/

http://www.flickr.com/photos/batega/1596898776/

Conclusions

Know What's Possible

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/docman/36125185/

http://www.flickr.com/photos/kgregory/500456103/

Look for Opportunities

Experiment

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/seeminglee/3967329241/

Manage Complexity

gareth rushgrove | morethanseven.net http://www.flickr.com/photos/30890318@N06/3510161637/

Questions?

gareth rushgrove | morethanseven.net http://flickr.com/photos/psd/102332391/

You're Going To Need A Bigger Toolbox

Technology

Transcript of You're Going To Need A Bigger Toolbox

London: "If you're tired of London you're tired of life!"

Matlab Installation with UdS Campus License5G Toolbox 1.1 Aerospace Blockset 4.1 Aerospace Toolbox 3.1 Antenna Toolbox 4.0 Audio Toolbox 2.0 Automated Driving Toolbox 2.0 AUTOSAR Blockset

When you're considering a faculty position, you're · When you're considering a faculty position, you're not ... on the map for connoisseurs of ... regard to race, color, national

The Design of A Fast and Flexible Internetgroups.csail.mit.edu/ana/Publications/PubPDFs/Joanna... · 2009. 7. 9. · You're the top! You're the Colosseum, You're the top! You're the

What’s New – MATLAB and Simulink · 18 Solving Bigger Problems with MATLAB Problems that take too long for one processor to solve –Growing toolbox support (parallel and GPU)

ScanGear Toolbox 2 - download.canon.itdownload.canon.it/manuali_operatore/files/Toolbox D660U.pdf · 6 Installazione Se ScanGear Toolbox CS e ScanGear Toolbox FAU sono già stati

What to Do if You'Re Behind in You'Re Property Taxes

You're hired!

Avvo Webinar: If You're Not on Facebook You're Committing Malpractice

You're My Daydream and You're Driving Me Insane

If You're Hot You're Hot

You're going to need a bigger binder.

YOU FUN YOU'RE GREAT FRIEND YOU'RE COOL YOU MAKE THE WORLD BRIGHTER EYE THINK YOU'RE SMART HAPPY BIRTHDAY… · FRIEND YOU'RE COOL YOU MAKE THE WORLD BRIGHTER EYE THINK YOU'RE SMART

If you're not first, you're last

You're Accepted

Bigger and Bigger - Community Catalyst

Installing MATLAB - Students@USN · Optimization Toolbox Parallel Computing Toolbox Partial Differential Equation Toolbox Phased Array System Toolbox RF Toolbox Robust Control Toolbox

learnenglishkids.britishcouncil.org · BRITISH COUNCIL If you're Happy and you Know it it, Clap your you're hap you're hap you're hap you're hap you're hap you're hap py and py and

Kindergarten Unit 4 Bigger Books Bigger Reading Muscles ...

You're invited!