Behind the Curtain

Testing This

TESTING TESTING

Thursday, April 21, 2011

GOOD MORNING!

Aaron Patterson

@tenderlove

WWFMD?Thursday, April 21, 2011

ITWFMWD?Thursday, April 21, 2011

AT&T, AT&T logo and all AT&T related marks are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies.

Miami Vice Principal Señor Facebook

Integration Engineering Manager

Inspirational Entertaining Informative Wasted

Our Time Together

KEYNOTE

Inspirational Entertaining Informative Wasted

Our Time Together

20% More Transitions

Living Behindthe Curtain 6 subjects, less

than 10minutes on each of them

I enjoy dealing with behinds the scenes information

DATABASE

APPLICATION

CLIENT

DATABASE

APPLICATION

CLIENT

DATABASE

APPLICATION

CLIENT

DATABASE

APPLICATION

CLIENT

Code as Data

Politics

"Open Sourceis a Democracy"

We don't vote on commits. It would be too inefficient.

OSS is an Oligarchy

Benevolent Dictator

Tyrannical Leader

Small projects have few people, so it is efficient

Large projects have too many people so it becomes inefficient to individually consider all feedback

Care and Feeding of Your Tyrant

Cheetos

Feed them Quality Bugs

Good reproduction steps

Feed them Patches with Tests

leave build systems

Gemspec,JewlerHoe,etc

leave test frameworks

Use +1's Carefully

320 comments

210 plus one

FFFFFFFUUUUUUUUUUUU

Superficial

Quality

How to fix?

We'll discuss this in closing.

Encodingsthe silent killer

Never a miscommunication

Data goes in, Data goes out

Until there is

`encode': "\xE9" from ASCII-8BIT to UTF-8(Encoding::UndefinedConversionError)

You can't explain that

Encoding Error?

String Tagging

"hello! <3".encoding # => #<Encoding:UTF-8>

Strings with unknown encoding are marked as binary.

socket.read(4).encoding # => #<Encoding:ASCII-8BIT>

Boundaries

To understand these bugs, we must find boundaries

App Code

Web Server

Database

NoSQL File System

SadnessBelt

#<Encoding:ASCII-8BIT>is usually a bug!

Databases know the stored encoding

Post contains encoding / spec declares default

# encoding: utf-8

name = 'たこ焼き仮面'

user = User.create!(:name => name)user.reloaduser.name + name

# encoding: utf-8

name = 'たこ焼き仮面'

user = User.create!(:name => name)user.reloaduser.name + name

incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)

>> name = "Aaron Patterson">> name.encoding=> #<Encoding:UTF-8>>> User.create!(:name => name)>> u = User.find :first>> u.name.encoding=> #<Encoding:ASCII-8BIT>

name = 'aaron patterson'user = User.create!(:name => name)user.reloaduser.name + name

Ruby attempts to convert the binary for you

name = 'aaron patterson'user = User.create!(:name => name)user.reloaduser.name + name

no error

Ruby attempts to convert the binary for you

ASCII-8BIT + othershould raise

Aaron'sOpinionCorner!

Resources Know Encoding

Prepared Statements

SELECT * FROM "users" WHERE id = 1

♥Parse the Query♥Formulate an Execution Plan♥Return results

Normal Queries

Application

Database

Application

Database

Result Set

Application

Database

Application

Database

query: 1234

Application

Database

Application

Database

Result Set

♥Parse the Query♥Formulate an Execution Plan

Preparation

Execution

♥Return Results

Data transfer decreases

Query Planner Improves

Security

SQLite3 Issues :'(

Talk about problem inserting rows with session data

Could insert session data, but could not find it

INSERT INTO "sessions"("data", "session_id")VALUES (?, ?)

['data', 'BAh7BjoIZm9vSSIIYmF6BjoGRUY=']['session_id', 'dae0fb8a9c34e6980c9d0fae33a8fff6']

db = SQLite3::Database.new ':memory:'db.trace { |sql| puts sql }

INSERT INTO "sessions"("data", "session_id")

VALUES (

'BAh7BjoIZm9vSSIIYmF6BjoGRUY=',

x'3633303337623066376536613130343132343765623763626631616439303631')

INSERT INTO "sessions"("data", "session_id")

VALUES (

'BAh7BjoIZm9vSSIIYmF6BjoGRUY=',

x'3633303337623066376536613130343132343765623763626631616439303631')

db.trace { |sql| puts sql }

stmt = db.prepare('INSERT INTO "sessions" (session_id)VALUES (?)')

stmt.bind_param 1, 'fuu'stmt.execute

Made a sample program to reproduce the problem

INSERT INTO "sessions"(session_id) VALUES ('fuu')

Resulting Query

stmt.bind_param 1, 'fuu'.encode('BINARY')stmt.execute

INSERT INTO "sessions" (session_id)VALUES (x'667575')

Why did sqlite3 do this?

Talk about column affinity

Why is it stored as Binary?

Why is the session id tagged binary?

>> x = OpenSSL::Random.random_bytes(10)=> "G\x93\xFC\xB2\xCE\xC0\xEC\xBB\xA7W">> x.encoding=> #<Encoding:ASCII-8BIT>

>> y = x.unpack('H*')=> ["4793fcb2cec0ecbba757"]>> y.first.encoding=> #<Encoding:ASCII-8BIT>

def generate_sid ActiveSupport::SecureRandom.hex(16)end

def generate_sid sid = ActiveSupport::SecureRandom.hex(16) if sid.respond_to?(:encode!) sid.encode!('UTF-8') end sidend

Should hex have tagged it?

Should have generate_sid tagged it?

Find the Source

ARel and ActiveRecord

Fear of SQL

We must learn SQL

But not specifically SQL

Working with sets

Can fetch correct data.

We must learn to work with Sets

Avoid SQL for:

code reuse

security

independence

from databases

What is ARel?

Represents SQL as a Tree

Represents the IDEA of a SQL statement

SQL Compiler

Representsan IDEA

Only until we invoke the compiler, does the AST become a query

Relationship to ActiveRecord

User.where('something')

class User scope :heart where(:name => '<3')end

User.heart.select('name')

ActiveRecord

Our Application

ARel Database

ActiveRecord

Our Application

ARel Database

ActiveRecord

Our Application

ARel Database

to_sql

ActiveRecord

Our Application

ARel Database

SELECT * ...

ActiveRecord

Our Application

ARel Database

[{: name => '<3'}]

ActiveRecord::Relation

shortens to...

ARel != ARel

SQL Compiler

Future we can have optimizers, compiler cache, etc

Interpreter?

module Arel module Visitors class Mongo < Arel::Visitors::Visitor attr_reader :db

def initialize db; @db = db; end Query = Struct.new(:collection_name, :fields, :conditions)

private

def visit_Arel_Nodes_SelectStatement o o.cores.map { |c| visit_Arel_Nodes_SelectCore c }.map { |query| collection = db.collection query.collection_name fields = query.fields selector = Hash[query.conditions] opts = {} opts[:fields] = fields unless fields.empty? opts[:limit] = o.limit.expr.to_i if o.limit collection.find(selector, opts).to_a }.flatten end

def visit_Arel_Nodes_SelectCore o fields = o.projections.map { |proj| visit(proj) }.compact conditions = o.wheres.map { |condition| visit(condition) }.flatten(1) Query.new(visit(o.source), fields, conditions) end

def visit_Arel_Nodes_And o o.children.map { |child| visit child } end

def visit_Arel_Nodes_Equality o [visit(o.left), visit(o.right)] end

def visit_Arel_Attributes_Attribute o return if o.name == '*'; o.name end

def visit_Arel_Nodes_JoinSource o visit o.left end

def visit_Arel_Table o o.name end def literal o; o; end alias :visit_String :literal alias :visit_Fixnum :literal end endend

module Arel module Visitors class Mongo < Arel::Visitors::Visitor attr_reader :db

def initialize db; @db = db; end Query = Struct.new(:collection_name, :fields, :conditions)

private

def visit_Arel_Nodes_SelectStatement o o.cores.map { |c| visit_Arel_Nodes_SelectCore c }.map { |query| collection = db.collection query.collection_name fields = query.fields selector = Hash[query.conditions] opts = {} opts[:fields] = fields unless fields.empty? opts[:limit] = o.limit.expr.to_i if o.limit collection.find(selector, opts).to_a }.flatten end

def visit_Arel_Nodes_SelectCore o fields = o.projections.map { |proj| visit(proj) }.compact conditions = o.wheres.map { |condition| visit(condition) }.flatten(1) Query.new(visit(o.source), fields, conditions) end

def visit_Arel_Nodes_And o o.children.map { |child| visit child } end

def visit_Arel_Nodes_Equality o [visit(o.left), visit(o.right)] end

def visit_Arel_Attributes_Attribute o return if o.name == '*'; o.name end

def visit_Arel_Nodes_JoinSource o visit o.left end

def visit_Arel_Table o o.name end def literal o; o; end alias :visit_String :literal alias :visit_Fixnum :literal end endend

Mongo Interpreter64 loc

gist.github.com/845782

Custom DSL

class Select < Struct.new(:columns) def self.* other other.select = new(Arel.sql('*')) other endend

Responds to *

Responds to WHERE

class From < Struct.new(:table, :conditions) def WHERE conditions self.conditions = conditions Where.new(self) endend

Responds to to_s

class Where < Struct.new(:from, :select) def to_s Arel::Table.engine = Arel::Sql::Engine.new( FakeRecord::Base.new) table = Arel::Table.new from.table table.project(select.columns).where( from.conditions.map { |k,v| table[k].eq v }).to_sql endend

Bootstrap Methods

SELECT = Selectdef FROM table From.new tableend

x = SELECT * FROM("users") .WHERE(:id => 10)puts x

Write SQL Rubyto avoid SQL

Streaming

Talked about data from the back end, talk a bit about data to the front end

Twitter Streaming API

Chunked Responses

Needed for persistent connections

Normal TimelineBuffers data, then sends a response

Processing ERb

Download Asset

Chunked TimelineSend data as soon as it is available

Processing ERb

Download Asset

Implementation

Rack API

status, headers, body = app.call(env)

status.to_i

headers.is_a?(Hash)

body.responds_to?(:each)

Inside Rack

body.each do |chunk| output(chunk)endbody.close rescue nil

Calls each,

possibly calls close

Sample Applicationclass MyApp def call(env) # some computation body = 'hello' # more computation body << ' world'

[200, { 'X-Hello' => 'World' }, [body]] endend

Delay work until eachclass FooBody def each yield "hello " sleep(10) # simulate work yield "world!" endend

Problems ☹

Database connections were lost

Middleware

Database connections are managed in middleware

"There are two hard problems in CS: cache invalidation

and naming things"-- Phil Karlton

class DbCache < Struct.new(:app) def call(env) # init cache status, headers, body = app.call(env) # clear cache [status, headers, body] endend

class BodyProxy < Struct.new(:delegate) def each delegate.each { |x| yield x } end

def close delegate.close rescue nil # clear cache endend

Update Middleware, Database Works! ☺

But Query Cache was broken. ☹

Same problem in query cache as the database handle cache.

Middleware is coupled to the callstack

~ 24 middleware

~ 5 required a proxy

Types of Middleware

♥Generators♥Filters♥Lifecycle Handlers

Rack jams these to one API.

Approach seems naive when examining usage

Tied to callstack

Solutions!

Embrace The Differences!

class Listener def created(event) end

def destroyed(event) endend

Lifecycle Listeners

Generatorsclass Resource def service(request, response) 10.times { |i| response.body.write "hello #{i}" } response.body.close endend

Synchronous or Asynchronous

Return value is ignored

Filters

class Filter def filter(request, response, chain) chain.filter(request, response) endend

♥Synchronous

♥Asynchronous

♥Freedom from the Callstack

♥Thread Safety

I stole this API.

TEE HEE!!!!!

fixing our tyrants

Facebook + Github

Get Involved

Gain Trust

Only fix the immediate problem

Then Persuade

Questions?

Behind the Curtain

Technology

Transcript of Behind the Curtain

Behind the Curtain: 2010 Auction Catalog

Behind The Curtain Winter 2014

Behind The Curtain: GIS and Floodplain Mapping

"Behind The Red Curtain" by WKR

What’s Behind the Curtain of Customer Transformation

HUGH BARRET THE MAN BEHIND THE CURTAIN

Rent Seeking Behind the Green Curtain

2010 Behind the Curtain Auction Catalog

The Magic Behind the Curtain: Marketing Automation

Behind The Curtain--The Full Monty

Mr. Perry…what’s behind the curtain?

Russian Military Expenditures: What’s Behind the Curtain

BEHIND THE CURTAIN - DCT

A LOOK BEHIND THE CURTAIN

Behind the Curtain the Wizards of API

Backstage: What's Behind the Curtain

Behind the curtain cf dc cil 2015

Behind the curtain of race & wealth

Behind the Curtain: Exposing Advanced Threats

Out from Behind the Iron Curtain