柴田 博志SHIBATA Hiroshi

paperboy&co.asakusa.rbpaperboy&co., Inc.

Ruby の不具合の見つけ方

発表場所 RubyConf Taiwan 2012 2012-12-8(Sat)

How to discover the Ruby's defects with web application.

你好!ni-hao.

5文字のテキストらしいですよ？大丈夫か？

Xie Xie rubytaiwan. Lai tai wan, wan hen kai xin.

SHIBATA Hiroshi(@hsbt)Wo shi cong ri ben lai de. Wo jiao Hiroshi, Shibata.My name is Hiroshi SHIBATA, please call me Hiroshi or SHIBATA. This jacket is my trademark and fighting armor. Today is same jacket.

Thanks for @ihower and @rubytaiwan accepting my proposal. I appreciate it. I’m super exciting.Taiwan is good city. I love taiwan. I decided to arrived rubyconf taiwan in next year.

asakusa.rbI’m member of asakusa.rb. asakusa.rb is inspired by Seattle.rb. This meetup start every tuesday. this meetup founder is Akira Matsuda. If you arrived tokyo, japan, please join our meetup.

I’m working at paperboy&co. My job is solving for all technical problems such as continuous deployment, security issue and gaining code quality etc... it’s company wide scrum master.

http://fanic.jpOur one of the rails service is fanic. fanic is that you can have personal music store. I’m sorry, fanic is only japanese language available.

http://musiccaptains.com/zh/2012/11/27/fanic-music-website-song-purchase/You are lucky. Some of chinese writer wrote introduction of fanic. this entry is chinese! Please join it.

tDiaryIn this talk, I’ll introduce tDiary and how tDiary can discover ruby’s defects. I don’t talk agile and how to maintain legacy code. If you interested these theme, please talk to me after session.

http://github.com/tdiarytDiary is hosted by github now, It this url.I’m sorry this site is only available for japanese and english information. if you interested tDiary, please join our organization.

ｗhat’s?

First, what’s tDiary?Do you know tDiary? Please raise up your hand?

NikkiSystem

tDiary is Nikki System. Nikki is Diary.Japanese internet has web nikki culture before blog.

Like a Blog

Nikki is a blog in 2012. We don’t divide their.

SpecificationOverview

I’ll introduce tDiary mainly specification.

3. Pluggable2. CGI/Rack1. Ruby

First, it use Ruby, and this is most important things. Second, running environments are traditional CGI and Rack. It means tDiary can run in all of application server. For example apache, passenger and unicorn. Third, it has pluggable mechanism. It mechanism is possibled by ruby meta-programming. this feature supports newbie rubyists.

HistorySecond, I talk about history.

Ruby 1.6.7

2002In 2002, tDiary began to development in the sf.net and used CVS. I joined tDiary developper team this year.First version supported Ruby-1.6.7. Please raise up your hands if you had used this version of Ruby?

Ruby 1.8.0

2003In 2003, tDiary supported Ruby-1.8.0.This version included many standard libraries.

What is Ruby needed to run

the tDiary?“What is Ruby needed to run the tDiary.”This phrase posted some contents in 2004. It means tDiary more famous lather than Ruby.Everyone knows tDiary in Japan.

Ruby 1.9.0

2008In 2008, tDiary supported 1.9.0. Big internal changing occurred in this version. All of String have Encoding. tDiary was broken by encoding changes. It was so hard for us, and we have be supporting now.

2013In 2013. it’s future.

Ruby2.0.0

Probably, tDiary is supporting 2.0. This feature is assigned me.

As you can see that tDiary is growing up with Ruby.In other hand, Ruby is growing up with tDiary.

tDiary discoveredRuby’s defects

In other words, tDiary discovered some of ruby’s defects.

why?Why tDiary be able to discover Ruby’s defects?

Background

I’m going to describe Japanese Internet environment. Nearly 2004, web programmer need to use shared hosting server for web programming.

tDiary can use plugin written by tiny ruby code. Implementation of this feature need to ruby-meta-programming.

evaltDiary’s core feature is using eval, instance_eval and class_eval.

private # loading tdiary.conf in current directory def configure_attrs @secure = true unless @secure @options = {}

eval( File::open( 'tdiary.conf' ) {|f| f.read }.untaint, b, "(tdiary.conf)", 1 )

# language setup @lang = 'ja' unless @lang begin

This is part of tDiary’s code.tDiary configuration was written by Ruby code. it’s not by yaml and json.

# It is better to add your diary's URL to @no_referer. # The value must be the Array of Ruby and Array's contents are String # of Ruby. Though these Strings are converted to Regular Expression# of Ruby when compared, you can't use Regexp of Ruby.@no_referer = [ '^' + Regexp.quote( base_url ), # Your diary '^http://localhost[:/]', '^http://192.168.', '^http://172.1[6789]', '^http://172.2[0-9]', '^http://172.3[01]', '^http://10.',]

# URLs which are only recorded into Today's Link (Regular Expression)# @only_volatile is an array of URLs to record into only Today's Link.# When a referer match to this list, it will be recoreded to volatile# list. This list will be cleared when you make new text in new day.# Specify @only_volatile same style of @no_referer.@only_volatile = []

# The rules which convert the specified URL to the word (Regular Expression)# @referer_table is configured so that readable URLs are shown in This is example of tdiary configuration file.If you try to customize tDiary behavior, you should write method definitions, variable settings and more into “tdiary.conf”.It’s very useful.

private def setup_attr_accessor_to_all_ivars names = instance_variables().collect {|ivar| ivar.to_s.sub(/@/, '') } (class << self; self; end).class_eval { attr_accessor *names } end

def configure_bot_pattern bot = ["bot", "spider", "antenna", "crawler", "moget", "slurp"] bot += @options['bot'] || [] @bot = Regexp::new( "(#{bot.uniq.join( '|' )})", true ) end

Next point is class_eval.tDiary uses classic “cgi.rb” and creates CGI instance per all requests.CGI instance have a configuration instance. It needs access mechanism for tDiary’s core and plugins.class_eval make configuration instance possible to access it.

def load_plugin( file ) @resource_loaded = false begin res_file = File::dirname( file ) + "/#{@conf.lang}/" + File::basename( file ) open( res_file.untaint ) do |src| instance_eval( src.read.untaint, "(plugin/#{@conf.lang}/#{File::basename( res_file )})", 1 ) end @resource_loaded = true rescue IOError, Errno::ENOENT end File::open( file.untaint ) do |src| instance_eval( src.read.untaint, "(plugin/#{File::basename( file )})", 1 ) end end

The i18n of tDiary use Ruby code. this is same the case of configuration files. tDiary doesn’t use i18n gem! In past, tDiary include chinese language. but we dropped it, because we have no maintainer.

One of 2.0.0 feature is “Refinements”.tDiary have many monkey patches for Ruby’s core feature. I’m going to fix these patches by using “Refinements”.

“eval” is powerful metaprogramming in ruby. but you worry about security issue.support two things. one, share hosting environment. imagine of you made cgi in shared hosting environment. that cgi can use code written by other people. If this code is exploit code. How do you protect it?

$SAFEtDiary protects with variable of $SAFE against security issue.I’m introducing $SAFE mechanism in Ruby. Did you know this feature?

def load_plugin( file ) @resource_loaded = false begin res_file = File::dirname( file ) + "/#{@conf.lang}/" + File::basename( file ) open( res_file.untaint ) do |src| instance_eval( src.read.untaint, "(plugin/#{@conf.lang}/#{File::basename( res_file )})", 1 ) end @resource_loaded = true rescue IOError, Errno::ENOENT end File::open( file.untaint ) do |src| instance_eval( src.read.untaint, "(plugin/#{File::basename( file )})", 1 ) end end

I talk overview of $SAFE mechanism. $SAFE mechanism makes all object taint. Once you use $SAFE, you can’t use tainted object in some function. If you want to use tainted objects, you should call “untaint” method.

$SAFE=1$ ruby -e '$SAFE = 1; open(ARGV[0])' foo

-e:1:in `initialize': Insecure operation - initialize (SecurityError)

from -e:1

$SAFE level 1 disallow “File.open” with tainted strings. If you want to open with ARGV[0] string. you should call “untaint”.

$SAFE=4

% ruby -e '$SAFE=4; eval("p :foo".untaint)'

-e:1:in `untaint': Insecure operation at level 4 (SecurityError)

from -e:1:in `<main>'

% ruby -e '$SAFE=1; eval("p :foo".untaint)'

:foo

$SAFE level 4. If you use untainted strings with eval, this level disallow using it.tDiary customization is handled with eval and $SAFE mechanism.

=begin== Safe module=endmodule Safe def safe( level = 4 ) result = nil if $SAFE < level then Proc.new { $SAFE = level result = yield }.call else result = yield end result end module_function :safeend

It’s protips. If you use different $SAFE level in same code. you need to use “Proc#new” and “Proc#call”.tDiary use this procedure when running with shared environment.

bugreport

http://bugs.ruby-lang.org/issues/5279fixed r33328:* encoding.c (require_enc): reject only loading from untrusted load paths. [ruby-dev:44541] [Bug #5279]* transcode.c (load_transcoder_entry): ditto. assume system default tmpdir safe. [ruby-dev:42089]

http://bugs.ruby-lang.org/issues/3733

fixed r29209: assume system default tmpdir safe. [ruby-dev:42089]

http://bugs.ruby-lang.org/issues/1115 * load.c (rb_require_safe): raises when the path to be loaded is tainted. [ruby-dev:37843]

I've never seen that other people are using $SAFE feature. therefore $SAFE feature is very buggy.

http://bugs.ruby-lang.org/issues/1115 * load.c (rb_require_safe): raises when the path to be loaded is tainted. [ruby-dev:37843]

% ruby -e '$SAFE=1;p require "zlib"'-e:1:in `require': Insecure operation - require (SecurityError) from -e:1:in `<main>'

% ruby -e '$SAFE=1;p require "English"'-e:1:in `require': Insecure operation - require (SecurityError) from -e:1:in `<main>'

This defect is that ruby raises exception when ruby call require with tainted path.

http://bugs.ruby-lang.org/issues/3733

fixed r29209: assume system default tmpdir safe. [ruby-dev:42089]

% ruby -e '$SAFE=1;File.expand_path(".".taint)'-e:1:in `expand_path': Insecure operation - expand_path (SecurityError) from -e:1:in `<main>'

This defects. if you use “tmpdir” running with $SAFE level 4, Ruby raised exception.

http://bugs.ruby-lang.org/issues/5279fixed r33328:* encoding.c (require_enc): reject only loading from untrusted load paths. [ruby-dev:44541] [Bug #5279]* transcode.c (load_transcoder_entry): ditto. assume system default tmpdir safe. [ruby-dev:42089]

% ruby -e ‘$SAFE = 3; "a".encode("UTF-16")’-e:1:in `encode': Insecure operation - encode (SecurityError) from -e:1:in `<main>'

In this report. If you run some code with $SAFE level 4, and not with calling “encode”. Ruby is raised exception.I already talked about all of String has Encoding in Ruby 1.9.This defects is important, because anyone never use String#encode with $SAFE level 4 in Ruby 1.9

5文字のテキストらしいですよ？大丈夫か？

I’m sorry, I introduce bad news. JRuby isn’t support $SAFE yet.

@shugomaeda

$SAFE users is only tDiary. I want to drop it in next version of ruby.

really!?

@hsbtI think probably dropped out $SAFE feature in 2.1. We accept it. so tDiary need to change with rubies. The Internet environment is changing now. people doesn’t use shared hosting server. they use PaaS like a sqale and heroku.

tDiary discover ruby’s defects is easy. In this case is only tDiary being?I think no.

tDiary uses Travis-CI.I build ruby-trunk every morning, and run it with tDiary. So, sometimes I get defects in ruby-trunk.

@tenderlove said that Rails-3.2 works with Ruby-2.0.It means that many web applications can work with Ruby-2.0. You can begin to use Ruby-2.0 now.

Even though, you couldn’t run code with Ruby-2.0 in production, you can run code in test environment.I think modern code have a lot of test code. you can.

https://speakerdeck.com/kakutani/above-all-make-it-fun

If it was also difficult to run your code in test environment, your hobby code should run with Ruby-2.0.I think that programmer have hobby code. tDiary is hobby code. Matz said “Ruby is hobby code!” too.

FFFFFFFFFFFFFFFFFFF.....FFFFFFFFFFFFFF....F.F...FFFFFFFFFFFFFFFFFFFFFFFFFF.....FFFFFFFFFFFFFF....F.F...FFFFFFFFFFFFFFFFFFFFFFFFFF.....FFFFFFFFFFFFFF....F.F...FFFFFFF.......FFFFFFFFFFWhen you run hobby code with Ruby-2.0, you will get error or segmentation fault. You should report it ruby core team.

bugreport

http://bugs.ruby-lang.org/issues/6781* lib/open-uri.rb: use respond_to? to test Tempfile. [ruby-dev:45995] [Bug #6781] reported by hsbt (Hiroshi SHIBATA).

http://bugs.ruby-lang.org/issues/5952* io.c (argf_next_argv): reset ARGF.next_p on ARGV.replace. r34409 breaks replacing ARGV. [ruby-dev:45160] [Bug #5952]

http://bugs.ruby-lang.org/issues/7040* ext/zlib/zlib.c (zstream_run_func): don't call inflate() when z->stream.avail_in == 0. it return Z_BUF_ERROR. but deflate() could be called with z->stream->avail_in == 0 because it has hidden buffer in z->stream->state (opaque structure). fix for gem install error. [ruby-dev:46149] [Bug #7040]

I discovered some defects in ruby-trunk.One of these defects is ruby-trunk can’t install some gems like libv8. Today’s ruby is fixed by my reports.

A few weeks ago. I discovered a new feature in cgi.rb. I thought this feature broke backward compatibility. I reported it for ruby-core.In this result, this feature reverted.You need to run your code with early version of Ruby. and report defects, behavior change and new feature.If you detect behavior change after code freeze, it’s too late.

Report it!

You shouldn’t write these report in blog or twitter. Probably, ruby-core commiters don’t read your tweets.

Are commiter only people who develop ruby? I don’t think so. this picture is one of scene in rubyconf. matz, ko1 and JRuby guys and many commiter and people discussion “refinements”Your report and proposal of new feature are making ruby!

Run your codewith 2.0.0

Let’s run your code with ruby 2.0.0

Thanks.