Cowboy development with Django
-
Upload
simon-willison -
Category
Technology
-
view
8.461 -
download
0
description
Transcript of Cowboy development with Django
![Page 1: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/1.jpg)
Cowboy development with Django
Simon WillisonDjangoCon 2009
![Page 2: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/2.jpg)
http://www.youtube.com/watch?v=nZx9sNXv9h0
![Page 3: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/3.jpg)
Just one problem... we didn’t have cowboys in
England
![Page 4: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/4.jpg)
The Napoleonic Wars
![Page 5: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/5.jpg)
A Napoleonic Sea Fort
http://en.wikipedia.org/wiki/File:Alderney_-_Fort_Clonque_02.jpg
![Page 6: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/6.jpg)
Super Evil Dev Fort
![Page 7: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/7.jpg)
http://www.anotherurl.com/travel/fort_clonque/handbook.htm
![Page 8: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/8.jpg)
Photos by Cindy Li
http://www.flickr.com/photos/cindyli/sets/72157610369683426/
![Page 9: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/9.jpg)
![Page 10: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/10.jpg)
![Page 11: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/11.jpg)
![Page 12: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/12.jpg)
![Page 13: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/13.jpg)
![Page 14: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/14.jpg)
WildLifeNearYou.com(Built in 1 week and 10 months)
![Page 15: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/15.jpg)
DEMO
![Page 16: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/16.jpg)
Search uses the geospatial branch of Xapian
Species database comes from Freebase
Photos can be imported from Flickr
“Suggest changes” to our Zoo information uses model objects representing proposed changes to other model objects
![Page 17: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/17.jpg)
What is /dev/fort?
Imagine a place of no distractions, noIM, no Twitter — in fact, nointernet. Within, a group of a dozenor more developers, designers,thinkers and doers. And a lot of afood.
Now imagine that place is a fort.
The idea behind /dev/fort is to throwa group of people together, cut themoff from the rest of the world, and
/dev/fortCohort 3: Winter 2009
The tripThe third /dev/fort will run from 9th to 16th November on the KintyrePeninsula in Scotland.
Cohort 2: Summer 2009
The tripThe second /dev/fort ran from 30th May to 6th June 2009 at KnockbrexCastle in Scotland. As with the first cohort, we have a few remainingproblems still to iron out (thorny issues inside Django we were hoping toavoid, that sort of thing). We hope to have the site in alpha by the end of thesummer.
Cohort membersRyan Alexander, Steven Anderson, James Aylett, Hannah Donovan, NatalieDowne, Mark Norman Francis, Matthew Hasler, Steve Marshall, RichardPope, Gareth Rushgrove, Simon Willison.
Cohort 1: Winter 2008
http://devfort.com/
![Page 18: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/18.jpg)
Cowboy development at work
![Page 19: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/19.jpg)
MP expenses
![Page 20: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/20.jpg)
Heather Brooke
![Page 21: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/21.jpg)
January 2005The FOI request
![Page 22: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/22.jpg)
February 2008The Information Tribunal
![Page 23: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/23.jpg)
“Transparency will damage democracy”
![Page 24: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/24.jpg)
January 2009The exemption law
![Page 25: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/25.jpg)
![Page 26: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/26.jpg)
![Page 27: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/27.jpg)
March 2009The mole
![Page 28: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/28.jpg)
“All of the receipts of 650-odd MPs, redacted and unredacted, are for sale at a price of £300,000, so I am told. The price is going up because of the interest in the
subject.”Sir Stuart Bell, MP
Newsnight, 30th March
![Page 29: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/29.jpg)
8th May, 2009The Daily Telegraph
![Page 30: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/30.jpg)
At the Guardian...
![Page 31: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/31.jpg)
April: “Expenses are due out in a couple of months, is there
anything we can do?”
![Page 32: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/32.jpg)
June: “Expenses have been bumped forward, they’re out
next week!”
![Page 33: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/33.jpg)
Thursday 11th JuneThe proof-of-concept
![Page 34: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/34.jpg)
Monday 15th JuneThe tentative go-ahead
![Page 35: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/35.jpg)
Tuesday 16th JuneDesigner + client-side engineer
![Page 36: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/36.jpg)
Wednesday 17th JuneOperations engineer
![Page 37: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/37.jpg)
Thursday 18th JuneLaunch day!
![Page 38: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/38.jpg)
![Page 39: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/39.jpg)
![Page 40: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/40.jpg)
![Page 41: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/41.jpg)
![Page 42: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/42.jpg)
![Page 43: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/43.jpg)
![Page 44: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/44.jpg)
How we built it
![Page 45: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/45.jpg)
![Page 46: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/46.jpg)
![Page 47: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/47.jpg)
$ convert Frank_Comm.pdf pages.png
![Page 48: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/48.jpg)
![Page 49: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/49.jpg)
Frictionless registration
![Page 50: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/50.jpg)
![Page 51: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/51.jpg)
Page filters
![Page 52: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/52.jpg)
![Page 53: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/53.jpg)
page_filters = ( # Maps name of filter to dictionary of kwargs to doc.pages.filter() ('reviewed', { 'votes__isnull': False }), ('unreviewed', { 'votes__isnull': True }), ('with line items', { 'line_items__isnull': False }), ('interesting', { 'votes__interestingvote__status': 'yes' }), ('interesting but known', { 'votes__interestingvote__status': 'known'...)page_filters_lookup = dict(page_filters)
![Page 54: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/54.jpg)
pages = doc.pages.all() if page_filter: kwargs = page_filters_lookup.get(page_filter) if kwargs is None: raise Http404, 'Invalid page filter: %s' % page_filter pages = pages.filter(**kwargs).distinct() # Build the filters filters = [] for name, kwargs in page_filters: filters.append({ 'name': name, 'count': doc.pages.filter(**kwargs).distinct().count(), })
![Page 55: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/55.jpg)
Matching names
![Page 56: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/56.jpg)
http://github.com/simonw/datamatcher
![Page 57: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/57.jpg)
On the day
![Page 58: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/58.jpg)
![Page 59: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/59.jpg)
![Page 60: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/60.jpg)
![Page 61: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/61.jpg)
def get_mp_pages(): "Returns list of (mp-name, mp-page-url) tuples" soup = Soup(urllib.urlopen(INDEX_URL)) mp_links = [] for link in soup.findAll('a'): if link.get('title', '').endswith("'s allowances"): mp_links.append( (link['title'].replace("'s allowances", ''), link['href']) ) return mp_links
![Page 62: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/62.jpg)
def get_pdfs(mp_url): "Returns list of (description, years, pdf-url, size) tuples" soup = Soup(urllib.urlopen(mp_url)) pdfs = [] trs = soup.findAll('tr')[1:] # Skip the first, it's the table header for tr in trs: name_td, year_td, pdf_td = tr.findAll('td') name = name_td.string year = year_td.string pdf_url = pdf_td.find('a')['href'] size = pdf_td.find('a').contents[-1].replace('(', '').replace(')', '') pdfs.append( (name, year, pdf_url, size) ) return pdfs
![Page 63: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/63.jpg)
![Page 64: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/64.jpg)
![Page 65: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/65.jpg)
“Drop Everything”
![Page 66: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/66.jpg)
Photoshop + AppleScriptv.s.
Java + IntelliJ
![Page 67: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/67.jpg)
Images on our docroot (S3 upload was taking too long)
![Page 68: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/68.jpg)
Blitz QA
![Page 69: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/69.jpg)
Launch! (on EC2)
![Page 70: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/70.jpg)
![Page 71: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/71.jpg)
Crash #1: more Apache children than MySQL
connections
![Page 72: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/72.jpg)
![Page 73: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/73.jpg)
![Page 74: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/74.jpg)
unreviewed_count = Page.objects.filter( votes__isnull = True).distinct().count()
![Page 75: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/75.jpg)
SELECT COUNT(DISTINCT `expenses_page`.`id`)FROM `expenses_page` LEFT OUTER JOIN `expenses_vote` ON ( `expenses_page`.`id` = `expenses_vote`.`page_id` ) WHERE `expenses_vote`.`id` IS NULL
![Page 76: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/76.jpg)
unreviewed_count = cache.get('homepage:unreviewed_count')if unreviewed_count is None: unreviewed_count = Page.objects.filter( votes__isnull = True ).distinct().count() cache.set('homepage: unreviewed_count', unreviewed_count, 60)
![Page 77: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/77.jpg)
With 70,000 pages and a LOT of votes...
DB takes up 135% of CPU
Cache the count in memcached...
DB drops to %35 of CPU
![Page 78: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/78.jpg)
unreviewed_count = Page.objects.filter( votes__isnull = True ).distinct().count()
reviewed_count = Page.objects.filter( votes__isnull = False ).distinct().count()
![Page 79: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/79.jpg)
unreviewed_count = Page.objects.filter( is_reviewed = False ).count()
![Page 80: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/80.jpg)
Migrating to InnoDB on a separate server
![Page 81: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/81.jpg)
ssh mps-live "mysqldump mp_expenses" |sed 's/ENGINE=MyISAM/ENGINE=InnoDB/g' |
sed 's/CHARSET=latin1/CHARSET=utf8/g' |ssh mysql-big "mysql -u root mp_expenses"
![Page 82: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/82.jpg)
Reigning in the cowboy
![Page 83: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/83.jpg)
An RSS to JSON proxy service
Pair programming
Comprehensive unit tests, with mocks
Continuous integration (Team City)
Deployment scripts against CI build numbers
Reigning in the cowboy
![Page 84: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/84.jpg)
Points of embarrassment
Database required to run the test suite
Logging? What logging?
Tests get deployed alongside the code (!)
... but generally pretty smooth sailing
![Page 85: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/85.jpg)
A final thought
![Page 86: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/86.jpg)
Web development in 2005
RelationalDatabase Cache
Application Admin tools
Templates XML feeds
![Page 87: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/87.jpg)
Web development in 2009RelationalDatabase Cache
ApplicationAdmin tools
Templates XML feeds
Datastructure servers
Search index
External web services
Monitoring and reporting
API Webhooks
Message queue Offline workers
Non-relationaldatabase
![Page 88: Cowboy development with Django](https://reader031.fdocuments.in/reader031/viewer/2022013003/5552c07cb4c90581158b471b/html5/thumbnails/88.jpg)
Thank you