Anatomy of a large Django site

Andy McKay Mozilla

mozilla

mozilla

Vancouver

PythonZope and Plone...now at Mozilla

mozilla

mozilla

Using Djangohttp://www.djangoproject.com

Credit: http://www.flickr.com/photos/abiavati/3110357974/

1. About the site2. Performance3. Localisation4. Reuse

mozilla

1. About the site

mozilla

mozilla

All code is open:https://github.com/jbalogh/zamboni

mozilla

All* bugs are open:https://bugzilla.mozilla.org

mozilla

Convert from CakePHP (remora)

to Django (zamboni)

mozilla

mozillaCredit: http://www.flickr.com/photos/improbcat/4177702580/

Changing one URL at a time from CakePHP to Django

mozilla

General trend to move away from PHP and do more Python and Django

How large?250k+ addons

150 mn views month500 mn API hits day

mozilla

Lines of code

PHPPython

40k18k

mozilla

Lines of code

PHPPython

Unit tests

40k18k15k

mozilla

No pages go out until they are faster

mozilla

running both php and python side by side. a few issues on that

3 zeus load balancers24 django (and php) 1 mysql + 4 slaves3 memcached3 sphinx1 rabbitmq + 2 celeryd1 redis master + 1 slave

Credit: http://www.flickr.com/photos/tbridge/15300843/ mozilla

2. Performance

mozilla

mozilla

As usual, database bottleneck

Cache machinehttp://bit.ly/cache-machine

mozillaCredit: http://www.flickr.com/photos/mwichary/4063534688/

mozilla

from django.db import modelsimport caching.base

class Addon(caching.base.CachingMixin, models.Model): ... status = models.IntegerField() objects = caching.base.CachingManager()

available as a mixin

need to addin the custom manager

mozilla

>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13

mozilla

>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13

>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13

mozilla

Invalidation

mozillamozilla

md5(‘select... a’) [addon 3615]

mozillamozilla

md5(‘select... a’) [addon 3615]

addon 3615 md5(‘select... a’)

mozillamozilla

md5(‘select... a’) [addon 3615]

md5(‘select... b’) [addon 3615, addon 1685]

addon 3615 md5(‘select... a’)md5(‘select... b’)

mozilla

Memcachedrules = cache.get(3615)rules.add('select...')cache.set(3615, rules)

mozilla

Redisredis.SADD(3615, ‘select...’)

mozilla

Home page20+ addons

400+ sql queries

mozilla

standard answer in django is select-related

add-on version

files

version

version

files

files

mozilla

django: select_related()http://bit.ly/select-related

mozilla

Transformerhttp://bit.ly/queryset-transform

simonw

mozilla

@staticmethod def transformer(addons): addon_dict = dict((a.id, a) for a in addons) vs = filter(None, (a.current_version_id for a in addons) versions = list(Version.objects.filter(id__in=vs)) for version in versions: addon_dict[version.addon_id].current_version = version

slightly outdated example

mozilla

Home page20+ addons

~14 sql queries

big SQL statements... :(

14313 character

UpdateCalled on startup

about:config ☞ extensions.update.url

mozilla

IncomingUncached

8,000 req/sec1,600 req/sec

Im used to Plone in the bad old days

mozilla

IncomingUncached

PHP

8,000 req/sec1,600 req/sec550 req/sec

Im used to Plone in the bad old days

mozilla

PHP Python

v1Plain

Django

mozilla

PHP Python

v2Min. SQL

queries

mozilla

oh god

mozilla

PHP Python

v3Django and

raw SQLmax-requests 200, actually we hit 210

mozilla

PHP Python

v4WSGI

no Django

mozilla

PHP Python

v5Pooling,

optimised queries

Thats 700 req/secwhich translates into

mozilla

Reducing the SQL queries...doesn’t always help

mozilla

mySQL query cache is fast

mozilla

Celeryhttp://celeryproject.org

mozillaCredit: http://www.flickr.com/photos/chiotsrun/3843988392/

Push things asyncemail

image processingadd-on validation

specifically fixing data changes between php and python

mozilla

from celeryutils import task

@taskdef update_tag(tag, **kw): tag.update_stat()

mozilla

from celeryutils import task

@task(rate_limit='60/h')def update_tag(tag, **kw): tag.update_stat()

mozilla

from tasks import update_tag

update_tag.delay(tag)

mozilla

mozilla

Measurement

Timing Middlewarehttp://bit.ly/timing-ware

mozillaCredit: http://www.flickr.com/photos/wwarby/3297205226/

mozilla

3. Localization

mozilla

mozilla

40+ languagesincluding rtl

show site in arabic?

mozilla

Database strings

content translated

and

templates

mozilla

class Addon(caching.base.CachingMixin, models.Model): ... name = models.ForeignKey(Translation)

mozilla

addon.name = 'name'addon.save()

mozilla

addon.name = 'name'addon.save()

addon.name = {'fr': 'la nomme'}addon.save()

mozilla

Templates

mozilla

Django{% blocktrans with app=request.APP %}

Add-ons for {{ app }} {% endblocktrans %}

mozilla

Jinja2http://jinja.pocoo.org/

{{ _('Add-ons for {0}')|f(request.APP) }}

mozilla

Python Unicode hellUnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 16: ordinal

not in range(128)

4. Reuse

mozilla

Bleachhttp://pypi.python.org/pypi/

mozillaCredit: http://www.flickr.com/photos/maisonbisson/3350954463/

>>> bleach.clean('an <script>evil()</script> example')

'an &lt;script&gt;evil()&lt;/script&gt; example'

mozilla

>>> bleach.linkify('an http://ex.com url')

'an <a href="http://ex.com" rel="nofollow">http://ex.com</a> url'

mozilla

Javascript tests

mozilla

django-qunithttp://bit.ly/django-qunit

mozilla

kumar

mozilla

test('English', function() {    z.refreshL10n('en-us');    equals($('textarea:visible', this.sandbox).text().trim(),           'Firebug integrates with Firefox to put ' + 'a wealth of development tools...');});

mozilla

test('Japanese', function() {    z.refreshL10n('ja');    equals($('textarea:visible', this.sandbox).text().trim(),           'Firebug は、Web ページを閲覧中にクリック一つ ' +

' で使える豊富な開発ツールを Firefox' +           ' に統合します。あなたはあらゆる');});

Use HudsonJenkinshttp://bit.ly/jstestnet

mozilla

So we use hudson for CI, but haven’t got the automated tests in yet

Hoping to do this via jstestnet

Flake8http://bit.ly/flake8

mozillaCredit: http://www.flickr.com/photos/nebarnix/357779131/

pep 8py flakesMacCabe

~/sandboxes/zamboni(632719) $ flake8 apps/editors/tasks.pyapps/editors/tasks.py:1: 'datetime' imported but unusedapps/editors/tasks.py:3: 'stat' imported but unused

mozilla

Playdohhttp://bit.ly/mozilla-playdoh

mozilla

fred wenzel

Credit: http://www.flickr.com/photos/ahmee/97960570/

Basis for new projects

mozilla

Celery supportJinja2 supportSimple migrations

By default: SHA-512 password hashing X-Frame-Options: DENY secure and httponly flags on cookies

mozilla

fred wenzel

Take inspiration from...but not the best for you

mozilla

for example jinja2 which makes integration with lots of django addons possible, but a bit harder

Questions?@andymckay

andym@mozilla.comandym on irc.freenode.net, irc.mozilla.org

mozilla