Forbidden Fruit: A Taste of Ruby's ParseTree

Post on 15-Jan-2015

7.825 views 0 download

Tags:

description

My Goruco 2008 presentation on Parse Tree and friends.

Transcript of Forbidden Fruit: A Taste of Ruby's ParseTree

Chris Wanstrathhttp://github.com/defunkt

• define_method

• Class.new

• instance_eval

• send

class Person singleton = (class << self; self end)

tons_of_class_methods.each do |name, body| singleton.send(:define_method, name, &body) endend

class Class def singleton (class << self; self end) end def define_class_method(name, &body) singleton.send(:define_method, name, &body) endend

class Person tons_of_class_methods.each do |name, body| define_class_method name, &body end end

Ruby is powerful

'rake'.sub 'r', 's'

desc 'Apply a patch directly from Pastie'task 'pastie:patch' do require "open-uri" pastie_url = "http://pastie.caboo.se/%s.txt" patch_id = (ENV['ID'] || ENV['PASTE']).gsub(/\D/, "") patch = open(pastie_url % patch_id).read File.open("patch.diff", "w+") { |f| f.puts(patch) } `patch -p0 < patch.diff && rm patch.diff` puts "Patched with pastie ##{patch_id}."end

desc 'Apply a patch directly from Pastie'task 'pastie:patch' do require("open-uri") pastie_url = " " patch_id = (ENV['ID'] || ENV['PASTE']).gsub(/\D/, "") patch = open(pastie_url % patch_id).read File.open("patch.diff", "w+") { |f| f.puts(patch) } `patch -p0 < patch.diff && rm patch.diff` puts("Patched with pastie ##{patch_id}.")end

desc 'Routes listed in a browser.'task :routes_page => :environment do require("erb") include(ERB::Util) @routes = ActionController::Routing::Routes.routes.collect do |route| name = ActionController::Routing::Routes.named_routes.routes.index(route).to_s verb = route.conditions[:method].to_s.upcase segs = route.segments.inject("") { |str, s| (str << s.to_s) } reqs = route.requirements.inspect { :name => name, :verb => verb, :segs => segs, :reqs => reqs } end template = "<html><head><title>Rails Routes</title></head>\n<body>\n<table>\n" ERB.new(template).run(binding)end

desc 'Parses git-log output and prints out summary of commits grouped by date'task('git:format_log') { ["date", "time", "rubygems", "active_support"].each { |f| require(f) } file = "git.log" lines = File.readlines(file) dates = Hash.new { |h, k| h[k] = [] } lines.each_with_index do |line, index| next unless line =~ /chris@/ _, date = lines[(index + 1)].split(":", 2) date = (Time.parse(date.strip) - 7.hours).midnight i, l = index, line end sorted_dates = dates.keys.sort.reverse sorted_dates.each do |date| puts(date.strftime("%A, %B %d, %Y")) commits = dates[date] commits.each do |commit| dcommit = commit.sub(/^(-|\*) /, "") puts(" - #{dcommit.capitalize}") end puts("") end puts("Days worked: #{sorted_dates.size}")}

“I know, I’ll use regular expressions.”

Now you havetwo problems

Ask Ruby.

>> ask_ruby = proc do ?> puts "How do we ask Ruby for method bodies?"?> end

>> require 'rubygems'>> require 'ruby2ruby'>> puts ask_ruby.to_rubyproc { puts("How do we ask Ruby for method bodies?")}=> nil

>> ask_ruby.to_ruby=> "proc {\n puts(\"How do we ask Ruby for method bodies?\")\n}"

Forbidden Fruit

A Taste of Ruby’s ParseTree

$ gem install ruby2ruby

>> require 'rubygems'>> require 'ruby2ruby'>> puts Ruby2Ruby.translate(AnyClass)(class definition)

task :hello do puts "Hello, world!"end

task = Sake::Task.new :hello do puts "Hello, world!"end

# puts task.to_rubytask 'hello' do puts("Hello, world!")end

task = Sake::Task.new :hello, :env, 'Say hello.' do puts "Hello, world!"end

desc 'Say hello.'task 'hello' => [ 'env' ] do puts("Hello, world!")end

By the way...

>>> var sayHi = function() { alert('Hello, world!') }>>> sayHi.toString()'function () { alert("Hello, world!"); }'

$ gem install sake

?

(1..100).to_a.dmap { |v| v * 2 }

module Enumerable def dmap(&block) self.each_with_index do |element,idx| ring_server.write([:dmap, Process.pid, block.to_ruby, element, idx]) end

results = [] while results.size < self.size result, i = ring_server.take([:dmap, Process.pid, nil, nil]).last(2) results[i] = result end

results endend

Client

Client

module Enumerable def dmap(&block) self.each_with_index do |element,idx| ring_server.write([:dmap, Process.pid, block.to_ruby, element, idx]) end

results = [] while results.size < self.size result, i = ring_server.take([:dmap, Process.pid, nil, nil]).last(2) results[i] = result end

results endend

Server

ring_server = RingyDingy.new(nil).ring_server

loop do msg = [:dmap, nil, nil, nil, nil] pid, block, element, i = ring_server.take(msg).last(4) begin result = eval(block).call(element) rescue Object => err result = err end puts "Got #{result} from #{element} for #{pid}." ring_server.write([:dmap, pid, result, i])end

Server

ring_server = RingyDingy.new(nil).ring_server

loop do msg = [:dmap, nil, nil, nil, nil] pid, block, element, i = ring_server.take(msg).last(4) begin result = eval(block).call(element) rescue Object => err result = err end puts "Got #{result} from #{element} for #{pid}." ring_server.write([:dmap, pid, result, i])end

?

User.find(:all, :conditions => Condition.block { |c| c.and "users.name", 'jon' c.and "users.age", 20})

Conditions Builder

EZ-Where

User.find_where(:all) do |user| user.all do name == 'jon' age == 20 end end

Ambition

User.select do |u| u.name == 'jon' && u.age == 20 end

Ambition

User.find :all, :conditions => { :name => 'jon' }

User.find :all, :conditions => { :name => 'jon' }

"SELECT * FROM users WHERE users.name = 'jon'"

User.select { |u| u.name =='jon' }

User.select { |u| u.name =='jon' }

"SELECT * FROM users WHERE users.name = 'jon'"

@people = User.select { |u| [1, 2, 3, 4].include? u.id }@people = @people.sort_by { |u| [ -u.age, u.name ] }

@people = User.select { |u| [1, 2, 3, 4].include? u.id }@people = @people.sort_by { |u| [ -u.age, u.name ] }

SELECT * FROM users WHERE users.id IN (1,2,3,4)ORDER BY users.age DESC, users.name

User.detect do |u| u.ideas.title == 'New Freezer' || u.profile.email =~ /pj/end

User.detect do |u| u.ideas.title == 'New Freezer' || u.profile.email =~ /pj/end

SELECT * FROM users ... (joins) ...WHERE ideas.title = "New Freezer " AND profiles.email ~ "pj"LIMIT 1

User.select do |u| u.profile.email.downcase =~ 'chris%'end.first(20)

User.select do |u| u.profile.email.downcase =~ 'chris%'end.first(20)

SELECT * FROM users WHERE LOWER(profiles.email) LIKE 'chris%'JOIN profiles.user_id = users.idLIMIT 20

Ambition

User.select do |u| u.name == 'jon' && u.age == 20 end

def &&(other) self + otherend

&&||!

$ gem install ParseTree

>> proc { 1 + 1 }.to_sexp=> [:proc, nil, [:call, [:lit, 1], :+, [:array, [:lit, 1]]]]

sexp of 1 + 1:[:call, [:lit, 1], :+, [:array, [:lit, 1]]]

:+

1

:call

:lit

:array:lit

1

1 + 1

>> pp proc { |u| u.name == 'jon' && u.age == 20 }.to_sexp[:proc, [:dasgn_curr, :u], [:and, [:call, [:call, [:dvar, :u], :name], :==, [:array, [:str, "jon"]]], [:call, [:call, [:dvar, :u], :age], :==, [:array, [:lit, 20]]]]]

?

Processors

class BlockToSqlProcessor def initialize(&block) @block = block end def to_sql process @block.to_sexp endend

processor = BlockToSqlProcessor.new do |u| u.name == ‘chris’end processor.to_sql

processor = BlockToSqlProcessor.new(&block)processor.to_sql

class BlockToSqlProcessor def process(node) node ||= []

if node.is_a? Symbol node elsif respond_to? method = "process_#{node.first}" send(method, node[1..-1]) elsif node.empty? '' else raise "Missing process method for sexp: #{node.inspect}" end endend

def process(node) node ||= []

if node.is_a? Symbol node elsif respond_to? method = "process_#{node.first}" send(method, node[1..-1]) elsif node.empty? '' else raise "Missing process method for sexp: #{node.inspect}" endend

[:dasgn_curr, :u], [:and, [:call, [:call, [:dvar, :u], :name], :==, [:array, [:str, "jon"]]], [:call, [:call, [:dvar, :u], :age], :==, [:array, [:lit, 20]]]]

[:dasgn_curr, :u], [:and, [:call, [:call, [:dvar, :u], :name], :==, [:array, [:str, "jon"]]], [:call, [:call, [:dvar, :u], :age], :==, [:array, [:lit, 20]]]]

class BlockToSqlProcessor def process_dasgn_curr(exp) (@receiver = exp.first).to_s end alias_method :process_dasgn, :process_dasgn_curr

def process_str(exp) exp.first end

def process_ivar(exp) eval exp.shift.to_s, @block endend

module Ambition #:nodoc: module Processors #:nodoc: class Select < Base def initialize(context, block) @context = context @block = block end

def process_call(args) # Operation (m.name == 'chris') # [[:call, [:dvar, :m], :name], :==, [:array, [:str, "chris"]]] if args.size == 3 left, operator, right = args

# params are passed as an array, even when only one element: # abc(1) # => [:fcall, :abc, [:array, [:lit, 1]] # abc([1]) # => [:fcall, :abc, [:array, [:array, [:lit, 1]]]] if right.first == :array right = process(right) right = right.is_a?(Array) ? right.first : right else right = process(right) end

translator.send(process_operator(operator), process(left), right)

# Property of passed arg: # [[:dvar, :m], :name] elsif args.first.last == @receiver translator.call(*args[1..-1])

# Method call: # [[:call, [:dvar, :m], :name], :upcase] elsif args.first.first == :call && args.first[1].last == @receiver receiver, method = args translator.chained_call(receiver.last, method)

# Deep, chained call: # [[:call, [:call, [:call, [:dvar, :m], :created_at], :something], :else], :perhaps] elsif args.flatten.include? @receiver calls = []

until args.empty? args = args.last if args.last.is_a?(Array) break if args.last == @receiver calls << args.pop end

translator.chained_call(*calls.reverse)

else raise args.inspect end end

def process_match3(exp) right, left = exp process_call [ left, :=~, right ] end

def process_and(exp) joined_expressions exp, :both end

def process_or(exp) joined_expressions exp, :either end

def joined_expressions(exp, with = nil) expressions = []

while expression = exp.shift expressions << process(expression) end

translator.send(with, *expressions) end

def process_not(args) negate { process(args.first) } end

def process_operator(operator) @negated ? negate_operator(operator) : operator end

def negate_operator(operator) case operator when :== then :not_equal when :=~ then :not_regexp else raise "Missing negated operator definition: #{operator}" end end

def negate @negated = translator.negated = true yield ensure @negated = translator.negated = false end end endend

module Ambition #:nodoc: module Processors #:nodoc: class Select < Base def initialize(context, block) @context = context @block = block end

def process_call(args) # Operation (m.name == 'chris') # [[:call, [:dvar, :m], :name], :==, [:array, [:str, "chris"]]] if args.size == 3 left, operator, right = args

# params are passed as an array, even when only one element: # abc(1) # => [:fcall, :abc, [:array, [:lit, 1]] # abc([1]) # => [:fcall, :abc, [:array, [:array, [:lit, 1]]]] if right.first == :array right = process(right) right = right.is_a?(Array) ? right.first : right else right = process(right) end

translator.send(process_operator(operator), process(left), right)

# Property of passed arg: # [[:dvar, :m], :name] elsif args.first.last == @receiver translator.call(*args[1..-1])

# Method call: # [[:call, [:dvar, :m], :name], :upcase] elsif args.first.first == :call && args.first[1].last == @receiver receiver, method = args translator.chained_call(receiver.last, method)

# Deep, chained call: # [[:call, [:call, [:call, [:dvar, :m], :created_at], :something], :else], :perhaps] elsif args.flatten.include? @receiver calls = []

until args.empty? args = args.last if args.last.is_a?(Array) break if args.last == @receiver calls << args.pop end

translator.chained_call(*calls.reverse)

else raise args.inspect end end

def process_match3(exp) right, left = exp process_call [ left, :=~, right ] end

def process_and(exp) joined_expressions exp, :both end

def process_or(exp) joined_expressions exp, :either end

def joined_expressions(exp, with = nil) expressions = []

while expression = exp.shift expressions << process(expression) end

translator.send(with, *expressions) end

def process_not(args) negate { process(args.first) } end

def process_operator(operator) @negated ? negate_operator(operator) : operator end

def negate_operator(operator) case operator when :== then :not_equal when :=~ then :not_regexp else raise "Missing negated operator definition: #{operator}" end end

def negate @negated = translator.negated = true yield ensure @negated = translator.negated = false end end endend

module Ambition #:nodoc: module Processors #:nodoc: class Select < Base def initialize(context, block) @context = context @block = block end

def process_call(args) # Operation (m.name == 'chris') # [[:call, [:dvar, :m], :name], :==, [:array, [:str, "chris"]]] if args.size == 3 left, operator, right = args

# params are passed as an array, even when only one element: # abc(1) # => [:fcall, :abc, [:array, [:lit, 1]] # abc([1]) # => [:fcall, :abc, [:array, [:array, [:lit, 1]]]] if right.first == :array right = process(right) right = right.is_a?(Array) ? right.first : right else right = process(right) end

translator.send(process_operator(operator), process(left), right)

# Property of passed arg: # [[:dvar, :m], :name] elsif args.first.last == @receiver translator.call(*args[1..-1])

# Method call: # [[:call, [:dvar, :m], :name], :upcase] elsif args.first.first == :call && args.first[1].last == @receiver receiver, method = args translator.chained_call(receiver.last, method)

# Deep, chained call: # [[:call, [:call, [:call, [:dvar, :m], :created_at], :something], :else], :perhaps] elsif args.flatten.include? @receiver calls = []

until args.empty? args = args.last if args.last.is_a?(Array) break if args.last == @receiver calls << args.pop end

translator.chained_call(*calls.reverse)

else raise args.inspect end end

def process_match3(exp) right, left = exp process_call [ left, :=~, right ] end

def process_and(exp) joined_expressions exp, :both end

def process_or(exp) joined_expressions exp, :either end

def joined_expressions(exp, with = nil) expressions = []

while expression = exp.shift expressions << process(expression) end

translator.send(with, *expressions) end

def process_not(args) negate { process(args.first) } end

def process_operator(operator) @negated ? negate_operator(operator) : operator end

def negate_operator(operator) case operator when :== then :not_equal when :=~ then :not_regexp else raise "Missing negated operator definition: #{operator}" end end

def negate @negated = translator.negated = true yield ensure @negated = translator.negated = false end end endend

module Ambition #:nodoc: module Processors #:nodoc: class Select < Base def initialize(context, block) @context = context @block = block end

def process_call(args) # Operation (m.name == 'chris') # [[:call, [:dvar, :m], :name], :==, [:array, [:str, "chris"]]] if args.size == 3 left, operator, right = args

# params are passed as an array, even when only one element: # abc(1) # => [:fcall, :abc, [:array, [:lit, 1]] # abc([1]) # => [:fcall, :abc, [:array, [:array, [:lit, 1]]]] if right.first == :array right = process(right) right = right.is_a?(Array) ? right.first : right else right = process(right) end

translator.send(process_operator(operator), process(left), right)

# Property of passed arg: # [[:dvar, :m], :name] elsif args.first.last == @receiver translator.call(*args[1..-1])

# Method call: # [[:call, [:dvar, :m], :name], :upcase] elsif args.first.first == :call && args.first[1].last == @receiver receiver, method = args translator.chained_call(receiver.last, method)

# Deep, chained call: # [[:call, [:call, [:call, [:dvar, :m], :created_at], :something], :else], :perhaps] elsif args.flatten.include? @receiver calls = []

until args.empty? args = args.last if args.last.is_a?(Array) break if args.last == @receiver calls << args.pop end

translator.chained_call(*calls.reverse)

else raise args.inspect end end

def process_match3(exp) right, left = exp process_call [ left, :=~, right ] end

def process_and(exp) joined_expressions exp, :both end

def process_or(exp) joined_expressions exp, :either end

def joined_expressions(exp, with = nil) expressions = []

while expression = exp.shift expressions << process(expression) end

translator.send(with, *expressions) end

def process_not(args) negate { process(args.first) } end

def process_operator(operator) @negated ? negate_operator(operator) : operator end

def negate_operator(operator) case operator when :== then :not_equal when :=~ then :not_regexp else raise "Missing negated operator definition: #{operator}" end end

def negate @negated = translator.negated = true yield ensure @negated = translator.negated = false end end endend

“Dude”- Giles

Translators

class SelectTranslator def <=(left, right) "#{left} <= #{sanitize right}" end

def include?(left, right) left = left.map { |element| sanitize element }.join(', ') "#{right} IN (#{left})" end

def downcase(column) "LOWER(#{owner.table_name}.#{quote_column_name column})" endend

class Query def kick owner.find(:all, to_hash) end def size owner.count(to_hash) end alias_method :length, :size end

class Query def to_s hash = to_hash

sql = [] sql << "WHERE #{hash[:conditions]}" unless hash[:conditions].blank? sql << "ORDER BY #{hash[:order]}" unless hash[:order].blank? sql << clauses[:slice].last unless hash[:slice].blank?

@@select_sql % [ owner.table_name, sql.join(' ') ] end alias_method :to_sql, :to_s end

class SelectTranslator def <=(left, right) "#{left} <= #{sanitize right}" end

def include?(left, right) left = left.map { |element| sanitize element }.join(', ') "#{right} IN (#{left})" end

def downcase(column) "LOWER(#{owner.table_name}.#{quote_column_name column})" endend

class SelectTranslator def <=(left, right) "#{left} LESS_THAN #{sanitize right}" end

def include?(left, right) left = left.map { |element| sanitize element }.join(', ') "#{right} IN (#{left})" end

def downcase(column) "LOWER(#{owner.table_name}.#{quote_column_name column})" endend

User.select { |m| m.name == 'jon' && m.age == 20 }

User.select { |m| m.name == 'jon' && m.age == 20 }

"(&(name=jon)(age=21))"

class SelectTranslator def <=(left, right) "(#{left}<=#{sanitize right})" end

def include?(left, right) bits = left.map { |item| "(#{right}=#{item})" } "(|#{bits})" endend

Roll your own

• ActiveRecord

• ActiveLDAP

• CouchDB

• Sphinx

>> puts ask_ruby.to_rubyproc { puts("How do we ask Ruby for method bodies?")}

>> proc { module Dude; end }.to_sexp=> [:proc, nil, [:module, :Dude, [:scope]]]

def process_module(exp) "module #{util_module_or_class(exp)}"end

>> proc { module Dude; end }.to_ruby=> "proc {\n module Dude\n end\n}"

def process_match2(exp) lhs = process(exp.shift) rhs = process(exp.shift) "#{lhs} =~ #{rhs}"end

• C++

• Java

• Javascript

• Python

• Erlang

Vulnerable Parse Trees

And, of course

LISP!

LISP!

(define (factorial x) (if (zero? x) 1 (* x (factorial (- x 1)))))

LISP in Ruby

Bus Scheme

Jim Weirich’s LISP.rb

http://onestepback.org/index.cgi/Tech/Ruby/LispInRuby.red

env = [ cons(:rev_shift, [:lambda, [:list, :result], [:cond, [[:null, :list], :result], [:t, [:rev_shift, [:cdr, :list], [:cons, [:car, :list], :result]]]]].sexp), cons(:reverse, [:lambda, [:list], [:rev_shift, :list, nil]].sexp), cons(:null, [:lambda, [:e], [:eq, :e, nil]].sexp), cons(:t, true), cons(nil, nil) ].sexp

http://onestepback.org/

Oil

( coming soon )

Thanks!