Zombie-chaser 0.1Going beyond skin deep
Andrew GrimmUniversity of New
South Wales
Overview
● Gratuitous violence and copyright violation● Unit testing● Mutation testing● Heckle● Chaser● Zombie-chaser● Gratuitous violence and copyright violation
Gratuitous violence and copyright violation
● See above
We don't test code
● I get a specification from my boss● I spend a couple of weeks writing on my typewriter● Fax it to the client● They scan it in● It runs first time● And does what they wanted● This DOESN'T happen
We all test code somehow
● SFTP code.php, make a few edits, SFTP it back to the production server, view it in your browser
– This DOES happen
● svn update, make a few changes, try it on a development web server, don't spot anything wrong, push to production
– Better
● git clone, add new test, add new code, check all tests pass, push to production
– Yay!!
Automated testing
● Term “Unit testing” sometimes misused● Technically means testing a small portion of your
code● Nowadays used to mean automated testing● Sometimes automated testing is the lazier option
than manual testing even in the short-term
Automated testing = exomemory
● Your code is thousands of lines long– You can't memorize your code
● Requirements change from the original– You can't write once, try it out, and never touch it
again
● You can't email past-self– The code itself needs to know what's right and wrong
An example of a test
● Given I am a fake Canadian motorcade at the 2007 APEC summit
● When I supply obviously bogus credentials to security
● Then I should not be let in to the green zone
Mutation testing
● Evaluates thoroughness of testing● If your code is modified, it ought to fail some of its
tests
Example of a mutation
● Modifying an APEC credential verification method, such that it regards an invalid pass as valid and/or vice versa
Heckle
● Main mutation tester in ruby● Name derived from Jester (Java mutation tester)● Uses ParseTree and ruby2ruby● Mutates literals (eg strings) and branches (eg swaps
“if”s with “unless”s)● Parses the program, makes changes to the parse
tree, and builds the new program
Heckle pros and cons
● Best mutator out there● Except when it's not available
– Doesn't work with default version of ruby on Windows
– ParseTree doesn't work on ruby 1.9
– Plans to replace ParseTree with RubyParser, which does work on ruby 1.9
Chaser
● Modified version of heckle● Named after the ABC comedy team, best known for
a penetration test that went horribly right● Created to deal with unavailability of heckle on
Windows (which I no longer need)● Uses metaprogramming to modify methods● Intended to run under anything that's ruby● Runs test/unit and my fork of exemplor
Chaser metaprogramming
def modify_instance_method chaser = self @mutated = true @old_method = @klass.instance_method(@method_name) @klass.class_eval do remove_method(chaser.method_name) define_method(chaser.method_name) do (*args) original_value = chaser.old_method.bind(self).call(*args) chaser.mutate_value(original_value) end endend
Chaser disadvantages
● Only modifies return values (and yielded parameters)
● Methods with side-effects will work as normal● Ability to not call a method at all could be added● Doesn't mutate every line● Currently doesn't describe the mutations being done
– May be difficult task
Availability of chaser
● Chaser (actually zombie-chaser) works on:● *nix 1.8 and 1.9● Windows 1.8 and 1.9● JRuby 1.8 only
– Haven't got to the bottom yet for 1.9 (see next slide)
● MacRuby: We broke it good! (see following slide)● Haven't tried IronRuby, Rubinius, or other platforms
Availability of chaser: JRuby 1.9Andrew-Grimms-MacBook-Pro:zombie-chaser agrimm$ jruby --1.9 -Ilib bin/zombie-chaser /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:11:in `included'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `call'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `set_attributes'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `each'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `set_attributes'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `each'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `set_attributes'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:42:in `attribute'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `each'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `attribute'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:76:in `register_fixture'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:24:in `setup'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:9:in `included'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:8:in `class_eval'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:8:in `included'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/testcase.rb:82/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/testcase.rb:31:in `require'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit.rb:1/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit.rb:31:in `require'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/ui/testrunnermediator.rb:7/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/ui/testrunnermediator.rb:31:in `require'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'/Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:4/Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:31:in `require'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'bin/zombie-chaser:4/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:11:in `included': undefined method `unregister_setup_method' for nil:NilClass (NoMethodError)
from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `call'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `set_attributes'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `each'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `set_attributes'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `each'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `set_attributes'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:42:in `attribute'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `each' ... 16 levels...from /Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:31:in `require'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'from bin/zombie-chaser:4
Availability of chaser: MacRuby
Not even close!
[1] % ruby -Ilib -Itest -rrubygems test/test_unit.rb ~/Code/zombie-chaser/Users/pete/Code/zombie-chaser/test/test_unit.rb:1:in `<main>': unrecognized runtime type `' (TypeError)[1] % ruby -v ~/Code/zombie-chaserMacRuby version 0.6 (ruby 1.9.0) [universal-darwin10.0, x86_64]
And no, we don't recognize the runtime type either!
Thanks to Peter Hollows for running zombie-chaser on his box
Zombie-chaser
● Graphic(al) user interface for mutation testing● Incorporated code from Chris Lloyd (et al)'s brains● Ryan Bigg demonstrated zombino, a brains-based
game he created because brains wasn't working● psDooM, a Doom interface to sysadmin, was also an
inspiration
Gooey metaphor
● Human running across the screen represents running of your unit tests with unmodified code
– Equivalent to a “control” in a scientific experiment
– Fail any tests, and the human dies before zombies appear (boring!)
● Zombies represent mutations to your code– Each successful test is represented with a step closer
– Kill them with failing unit tests
Interfaces available
● Nethack like representation– @ human
– Z zombie
– * dying zombie
– + dead zombie
● GUI interface– Gosu used
Disadvantages with Zombie-chaser
● Issues shared with chaser:– Only modifies return values and yielded parameters
– Doesn't describe the mutations
● Additional issues– Doesn't explicitly describe which test it failed, or
how
– Stops running after the first failed test
Possible TODOs
● Add color to console based output (make dying zombies red)
● Make it work on more platforms● Make it work with more testing frameworks● Fix performance and threading bugs● Provide information that's currently not provided● Improve GUI metaphor● Sound and music
Gratuitous violence and copyright violation
● See above