Rubish: A Quixotic Shell
Howard Yeh
Quix.ot.ic
• \kwik-’sä-tik\
• foolishly impractical especially in the pursuit of ideals ; especially : marked by rash lofty romantic ideas or extravagantly chivalrous action.
Why Do You Care?
• Closes the gap between script and shell
• Object-Oriented Meta-shell (frobbable)
• Extensible
– Namespace
– Local extensions (with singleton objects)
– Abstraction as codified memory
• Ruby you know and love
Symbiosis With Ruby
• Meta access
• Data type
• Plausibly concise syntax
• Object#instance_eval(&block)
• Rubish has no metasyntax
• Rubish uses no monkey patches
– Not strictly true, but…
Introduction to Rubish
abandon bash all ye who enters…
Overview
• Executable
– Command, Pipe
– Sed, Awk
– Batch
• Job Control
– Concurrency; Exception Mechanism
• Context
– Dynamically scoped IO; Workspace
Command
Rubish Bash
foo foo
foo :abc foo -abc
foo “a b c” foo a b c
foo.q “a b c” foo “a b c”
foo “a”, “b”, “c”
foo ["a","b","c"]
foo [["a","b"],"c","d"]
Handling weird filenamesrsh> touch("a b","c d").qrsh> wc(ls.map).q
Pipe
• Build pipe with factory
rsh> p { cmd1; cmd2; cmd3 }
• Build pipe from array of commands
rsh> @cmds = [cmd1,cmd2,cmd3]
rsh> p(@cmds)
IO Redirection
• Methods defined on the Executable class
– Executable#{i,o,err}
– Common API to all subclasses
– So far only supports stdin, stdout, stderr
• With File
• With Ruby IO object
• With a block that receives a pipe
– Good for building further abstractions
IO to File
rsh> cat.o("output")
a
b
c
^D
rsh> wc("output").first.to_i
3
IO with Block
rsh> @c = cat.i {|pipe| pipe.puts 1,2,3 }
rsh> @c
1
2
3
# ask the command to write to a pipe instead
rsh> @c.o { |pipe| ... }
IO Abstractions
• Enumerating methods
– Common API to all Executable subclasses
– Executable#{each,map}
– Executable#{head(n=1),tail(n=1),first,last}
• Implemented with the Rubish IO architecture.
– OOP for the win!
Executable#each!
def each!self.o do |pipe|pipe.each_line do |line|
line.chomp!yield(line)
endendjob = self.exec!return job
end
Processing with Ruby
# array of listed file names
ls.map
# last file as string
ls.last # == ls.tail(1).first
# extension name of last 10 files
ls.tail(10) {|f| File.extname(f) }
# ditto with pipe
p { …}.map {|line| … }
Awk & Sed
Sed “One-Liner”
• Courtesy J. Zawinsky (Unix Hater Handbook)# find *.el files that didn't have corresponding *.elc files
# only two processes per file.
sh> find . -name ’*.el’ -print \
| sed ’s/^/FOO=/’|\
sed ’s/$/; if * ! -f \ ${FOO}c ]; then \
echo \ $FOO ; fi/’ | sh
• Rubish Sedrsh> find(". -name '*.el'").sed { p if !File.exist?(line + "c") }
rsh> find(". -name '*.el'").sed { p if !File.exist?(line + "c") }.map {|f| … }
Awk & Sed
• Unix Powertools are for Powerfools
• Hard to predict complexity.
• Hard to get right.
• Hard to remember.
• Hard to extend.
• Hard to generalize.
– “weird chars” problem
Rubish Sed & Awk
• Doesn’t aim for full generality
– Since we are embedded in Ruby anyway
• Captures common usage patterns
– Common-Lisp loopesque helpers
• Be explicit
– Sed does not print by default
• Fits well
– Both are subclasses of Executable
grep with context (Sed)
# grep -A1 -B1
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h
grep with context (Rubish)
# rubish sedsed {if line =~ /regexp/p "===="# special case if the first line matchesp prev if respond_to?(:prev)pp peek(1)
endhold(:prev,1,line) # dynamically creates a method “prev”
}
Streamer
• Awk and Sed are subclasses of Streamer
• Streamer#{peek(n=1),skip(n=1)}
– No more sedistic register shufflings
• Streamer#{max,min,count,collect,…}
– Common usage patterns
• Streamer#{quit,stop}
• Streamer#{each,map,head,tail,…}
– Inherited methods from Executable
grep with context (Rubish)
rsh> cat.i {|p| p.puts((1..11).to_a)}.sed { ... }====12====91011====1011
grep with context (Rubish)
• Easy to generalize# simulates ‘grep regexp -Aa -Bb’sed {if line =~ regexp
p "===="# special case if the first line matchesp prev if respond_to?(:prev)pp peek(a)
endhold(:prev,b,line)
}
Aggregator
• Streamer#{max,min,count,collect,…}
– Capture common usage patterns
– Inspired by Common Lisp’s Loop Facility
– Method signature: helper(name,value,key)
• Aggregation partitioned by key
– nil is the special global key under which everything is aggregated.
Aggregator
# longest and shortest filename lengths
rsh> ls.awk {
max(:mx,line)
min(:mn,line)
}.end {[mx,mn]}
[25, 3]
Aggregator
rsh> ls.awk { f=a[0]; collect(:fn,f,File.extname(f))}.end { fn}{""=> ["a",...,"util"],
".gz"=>["ruby-termios-0.9.5.tar.gz"],".yml"=>["VERSION.yml"],".sh"=>["test.sh"],".output"=>["awk.output"],".5"=>["ruby-termios-0.9.5"],".textile"=>["README.textile"],".rb"=>["address.rb", "foo.rb", "my.rb"],".bar"=>["foo.bar"]nil=>["a", ...,"VERSION.yml"]}
# fn(“.rb”) contains the array of *.rb
Addressed Patterns
.sed(3) { … } # triggered for line 3
.sed(3,9) { … } # lines 3 to 9 inclusive
.sed(3,:eof) { … } # lines 3 to end of file
.sed(/a/) { … } # triggered for matching lines
.sed(/a/,/b/) # triggered for lines between
# ditto for awk
.awk(/a/,/b/)
Rubish Concurrency
Concurrency
• Coarse Grained
– For independent tasks
• No interaction between tasks
– No deadlock
– No shared memory (by abstinence)
– But is safety by policy (read: no safety)
• Ruby Green Thread
– You could always fork… I guess?
Background Jobs
• Executable#{exec!,each!,map!}– #exec! returns an instance of Job
– #each! invokes the iterator in a background thread
– #map!(acc) << into an (hopefully thread safe) accumulator.
• Job#{wait,stop}– #wait returns the result of a job after it completes
– #stop signal a job to terminate, then wait
• For Command,Pipe,Sed,Awk, and more.
Background Jobs
# returns immediatelyrsh> @job = slowcat(3).exec! rsh> jobs # == [@job]# slowcat(3) takes 3 seconds to completersh> waitall # after 3 seconds => @job.wait
rsh> @acc = [] # should use a thread-safe accrsh> ls.map!(@acc)rsh> ls.map!(@acc)
Exception Handling
• Job#wait would raise on abnormal completion
– exitstatus != 0
• Exception avoids error checking cruft
– It’s the 21st century!
• A little suprising
– grep "notfound *"
– wc a-directory
Rubish in Context
Context
• Encapsulates IOs
– Dynamically scoped
• Encapsulates Bindings
– Lexically scoped
– Namespace management
– Local extensibility
• A context defines the meaning of a closure with free bindings.
Context-Sensitive Block
• Object#instance_eval(&block)
obj.instance_eval {
binding1
binding2
}
• Local extensibility
obj.extend(module).instance_eval { … }
Context IO
with {
cmd1.exec
cmd2.exec
with { cmd3 }.o("output3").exec
}.o("output1-2").exec
Context Extension
# ad hoc, local extensions
with(derive({def foo; ...; end})) {
... # outer foo
with(derive({def foo; ...; end})) {
... #inner foo
}
... # outer foo
}
Context Extension with Modules
# prebaked (modules from ‘load’)
with(derive(mod1,mod2)) {
…
}
Batch
• A batch job is a contextualized block executed in a thread.
• Schematically :
@job = Thread.new { context.eval { … }}
@job.wait
• Similar to subshell, but in the same process
• A Batch is also an Executable!
Batch
# first extend context# then carry out a block within a batch threadbatch(derive(...)) { ... }
# all Executable methods apply@job = batch(derive(...)) { ... }.exec!batch(derive(...)) { ... }.mapbatch(derive(...)) { ... }.awk…
Structured Concurrency
# Concurrent Jobs arranged in a tree:batch {exec! cmd1, cmd2batch {exec! cmd3, cmd4batch { exec! cmd5 }
}.exec # we’ll wait till this batch completes…
}.exec!
Conclusion
• OOP is great!
– Inheritance makes code confusing
– Polymorphism is powerful
– Excellent namespace management
• Design by Symbiosis
• Singleton objects for local extensibility
• Object#instance_eval(&block)
• Metaprogramming
– Make things frobbable
– I don’t miss lisp that much…
Thank You
• http://github.com/hayeah/rubish/tree/master
– Need lots more work.
– Generalize Rubish for remote scripting.
• Looking for interesting projects
– I like weird languages.
Top Related