ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Drupal as a Data Purveyor

Layering Drupal with emerging technologies to create a performant, scalable data purveyor.

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

ASTONISH DESIGNC H A N G E G A M E

Diana Montalion Dupuis@dianadupuis

Sam Heuck@samheuck

Andrew Elster@filmknurd

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

ASTONISH DESIGNC H A N G E G A M E

We have a new website: astonishdesign.com

@Astonish_Design

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

An open-source content management PHP/MySQL framework designed for rapid development of content-rich web applications.

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Reasons (among many)

• Growth: evolving tool

• Foundation skill set

• User features

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

http://jdhancock.com/ • JD Hancock• http://farm9.staticflickr.com/8322/8031897271_9c63e48a29_b.jpg

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

“Big” Data

"Big" in relation to the amount of content Drupal is designed

to manage

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Volume

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Velocity

BIKE WALLS• http://www.bikewalls.com

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Variety

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Content

•Every kind of data•Coming from everywhere

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Current Issues

•Bootstrapping•Bottlenecks - Normalized - Join Happy•Lack of solution architecture

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

ViewsRapid

DevelopmentPlatform

Visitors& ContentAuthors

Miners

Who is the user?

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Welcome to the future

When modularizing the architecture creates performant applications for all content and all users.

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Traditional Optimizations

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

DuplicateData

RapidDevelopment

Platform

API

DenormalizedDocument Storage for READS

NormalizedRelational for WRITES

Data Mining

Content & DataCreation

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Cost of Bootstrap• Hooks allow modules to

customize behavior

• Drupal calls hooks during each stage of bootstrap

• More modules, more hooks called, slower bootstrap

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Massive Callstacks

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Bootstrap on Every Request

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

ViewsRapid

DevelopmentPlatform

Visitors& ContentAuthors

Miners

Backbone.js

Twisted

JS

PhotonVert.xCelluloid::IO

PYTHON

PHP

JAVA

RUBY

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

ViewsRapid

DevelopmentPlatform

Visitors& ContentAuthors

Miners

Backbone.js

Twisted

JS

PhotonVert.xCelluloid::IO

PYTHON

PHP

JAVA

RUBYSources

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

API and Node.JS

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Why Proxy External Sources?

OK

your.domai

GET

OK

other.domain

OPTION

Yes

GET

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

RESTful

Robert Tadlock • http://www.flickr.com/photos/rtadlock/2716877199/

http://apigee.com/about/api-best-practices Since you are building the API, you can make it any way you want, but following some good basic REST conventions is a good idea.

It will make it easier to integrate with front end frameworks.

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Pick a FrameworkExpress or Restify

•http://expressjs.com•http://mcavage.github.com/node-restify

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Pick a Connector

Tony Hisgett • http://www.flickr.com/photos/hisgett/

http://mongoosejs.com/Mongoose

Alternatives•Mongolia•Mongo Skin•Native MongoDB Driver

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

ExampleExpress & Mongoose

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Create a Resource

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Create an Endpointhttp://domain/resources

{ [ { “someField”: “blah blah blah” “anotherField”: 2 }

...

] }

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

On the Front-end

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

ViewsRapid

DevelopmentPlatform

Visitors& ContentAuthors

Miners

Backbone.js

Twisted

JS

PhotonVert.xCelluloid::IO

PYTHON

PHP

JAVA

RUBYSources

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Isn’t JS Slow?Yes but....

1. JS and devices continue to improve

2. We've improved the overall round trip: faster data, smaller payload, no bootstrap. It's still faster overall.

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Organize Your CodeOn the front-end, you could write everything as AJAX calls and jQuery DOM manipulation, but that will get messy very quickly as your application grows in sophistication.

In the interests of maintainability and sanity, use a framework that organizes your code and abstracts away the DOM. There are many to choose from. A new one probably came out this morning.

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

We use Ember because it is straightforward to use, powerful, flexible, and still fairly light weight. Backbone has been around for a while and is quite nice, but we like how Ember handles garbage collection and the other goodies it ships with.

What Why Howhttp://www.emberjs.com Why Discourse Uses Ember •Official Guides

•Resources•Fire Up Ember•Ember Camp•Darthdeus V. Ember•

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Results fromRich Snippets

For Algebraix, all the data comes from an external source, so it makes sense to run the application on the client side, as opposed to importing the data into Drupal first.

It also makes the control panel feel more responsive. No more click and hope while waiting for the interface to receive data. Now we've created a conversation with the user.

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Essential question ...

Why use Drupal?

Don’t use Drupal if you don’t need it.

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

The role of the themer

Front engineer

ASTONISH DESIGNC H A N G E G A M E astonishdesign.com@astonishdesign #ddtx13

Questions?