An introduction and future of Ruby coverage library
-
Upload
mametter -
Category
Technology
-
view
1.198 -
download
0
Transcript of An introduction and future of Ruby coverage library
![Page 1: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/1.jpg)
An introduction and future of Ruby coverage library
RubyKaigi 2017 (19th Sep. )
Yusuke Endoh (@mametter)
![Page 2: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/2.jpg)
Yusuke Endoh (@mametter)
• Ruby committer (2008—)
• Full-time Ruby committer (2017—)
• My goal: make Ruby programs robust
– Test coverage, and type system?
![Page 3: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/3.jpg)
![Page 4: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/4.jpg)
![Page 5: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/5.jpg)
Method.Put the cream cheese into the mixing bowl.Put the sour cream into the mixing bowl.Put the white sugar into the mixing bowl.Put the yogrut into the mixing bowl.Put the unsalted butter into the mixing bowl.Combine the milk.Put the cake flour into the mixing bowl.Combine the corn starch.Put the brown sugar into the mixing bowl.Combine the egg whites.Combine the egg yolks.Put the lemon juice into the mixing bowl.Stir for 7 minutes.Liquify the contents of the mixing bowl.Pour contents of the mixing bowl into the baking dish.Bake the cake mixture.Watch the cake mixture until baked.
Serves 4.
Cheese cake in Chef.
Ingredients.100 g cream cheese97 g sour cream107 g yogrut112 g white sugar11 g brown sugar37 g unsalted butter37 g cake flour3 g corn starch3 ml milk3 egg yolks3 egg whites10 ml lemon juice0 g cake mixture
Cooking time: 80 minutes.
‘d’‘a’
‘k’‘p’
11*3*3=99 ‘c’
37*3=111 ‘o’
¥n
‘o’
Push characters‘¥n’ ‘d’ ‘a’ ‘p’ ‘k’ ‘o’ ‘o’ ‘c’
into the stack
Convert them to a stringand “serve” it.
data
code
![Page 6: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/6.jpg)
Esoteric Recipe
• Polyglot of Chef andreal recipe
– Japanese versionhttps://cookpad.com/recipe/4649810
– English versionhttps://cookpad.com/us/recipes/3335222
![Page 7: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/7.jpg)
My main contributions for Ruby
• Implementation of some features:keyword arguments, deadlock detection, etc.
• Release management for Ruby 1.9.2 and 2.0.0
• Optcarrot: A NES emulator for Ruby3x3 benchmark
• Enhancement ofthe test suite of Ruby
• coverage.so: the core libraryfor coverage measurement
’06B ’07A ’07B ’08A60
70
80
90
100
covera
ge (
%)
70%
85%
line coverage
![Page 8: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/8.jpg)
Today’s theme
• An introduction of test coverage
• An improvement plan of Ruby’s coverage measurement feature towards 2.5
![Page 9: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/9.jpg)
Survey [1/3]
• Q. Do you use Ruby/RoR in production?
– Raise your hand, please!
![Page 10: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/10.jpg)
Survey [2/3]
• Q. In those, do you test your code?
![Page 11: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/11.jpg)
Survey [3/3]
• Q. In those, do you use “coverage”?
– Do you check the result of SimpleCov?
![Page 12: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/12.jpg)
Agenda
• What is coverage
• How to understand and use coverage
• The current status of Ruby coverage feature
• The future plan of Ruby coverage feature
• Conclusion
![Page 13: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/13.jpg)
Agenda
☞ What is coverage
• How to understand and use coverage
• The current status of Ruby coverage feature
• The future plan of Ruby coverage feature
• Conclusion
![Page 14: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/14.jpg)
What is coverage?
• A measure of “goodness” of a test suite– Also called “test coverage” or “code coverage”
• Allows you:– Find untested code– Decide whether your test suite is good enough
or not yet• (This is arguable, but I think we can use it as an
advice)
• Types of coverage– Function coverage, line coverage, branch
coverage, …
![Page 15: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/15.jpg)
Function coverage
• How many functions are executed by the tests# codedef foo; …; end # ✓def bar; …; end # ✗def baz; …; end # ✓
# testfoobaz
2/3(67%)
• Advantage• Easy to understand
• Easy to visualize
• Disadvantage• Too weak as a measure
![Page 16: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/16.jpg)
Line coverage
• How many lines are executed
# codedef foo(x) # ✓
if x == 0 # ✓p :foo # ✗
elsep :bar # ✓
endend
# testfoo(1)
3/4(75%)
Non-significant line isignored
• Advantage• Easy to understand• Easy to visualize
• Disadvantage• Still weak as a measure
• foo() if x == 0
![Page 17: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/17.jpg)
Branch coverage
• How many branches are taken true and false
# codedef foo(x)
p :foo if x == 0 # ✓p :bar if x < 2 # ✗
end
# testfoo(0)foo(1)
1/2(50%)
• Advantage• Relatively exhaustive
• Disadvantage• Difficult to visualize
true-case and false-case areboth executed
![Page 18: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/18.jpg)
Coverage types
Coverage type Easy tounderstand/
visualize
Exhaustive
Functioncoverage
○ ✕
Linecoverage
○ △
Branchcoverage
△ ○
Currently, Ruby supports only line coverage
![Page 19: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/19.jpg)
Other types of coverage
• Condition coverage– How many conditions
(not branches) are takenboth true and false
• Path coverage– How many paths are executed
– Combinatorial explosion
• Other advanced ones– Data-flow coverage
– MC/DC coverage
if a && b
branch
conditioncondition
![Page 20: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/20.jpg)
Trivia
• “C0/C1/C2 coverages” have difference meanings to different people
– C0 coverage = line coverage
– C1 coverage = branch coverage or path coverage?
– C2 coverage = condition coverage or path coverage?
![Page 21: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/21.jpg)
Coverage and Ruby
• In Ruby, Coverage is crucial!– A test is the only way to ensure quality– Coverage is important to measure test goodness
• Considering it, coverage is not used so much…– Coverage is not well-known?– It is not well-known how to use coverage?– Ruby’s coverage measurement feature is not
enough?
• I’d like to improve the situation with this talk
![Page 22: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/22.jpg)
Agenda
• What is coverage
☞ How to understand and use coverage
• The current status of Ruby coverage feature
• The future plan of Ruby coverage feature
• Conclusion
![Page 23: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/23.jpg)
What is a good test suite?
• Covers the code
– Coverage measures this
• Also covers the design of program
– Coverage does not measure this directly
![Page 24: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/24.jpg)
How to understand coverage
• Coverage is just a measure
• Coverage is not a goal
![Page 25: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/25.jpg)
If coverage XX% is required as a goal…
• Developers will1. Pick untested code that looks easiest to
cover
2. Write a trivial test just for the code
3. Iterate this procedure until XX% is achieved
• It will result in trivial, not-so-good test suite– It may be exhaustive for the code itself, but
– It won't be exhaustive for the design
![Page 26: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/26.jpg)
A good way to improve coverage
• Developers should1. Look at untested code
2. Consider what “test design” is insufficient
3. Write them
– In consequence of them, the untested code is executed
• It will result in good test suite– It will be exhaustive not only for the code
but also for the design
![Page 27: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/27.jpg)
How many % is needed/enough?
• It depends upon the module being tested– Aim 100% for a significant module (e.g., injure
someone)
– Don't worry too much for a non-significant module
• It also depends upon cost performance– For non-significant module, write a test only if it
is not so hard
• Again: Coverage is not a goal
![Page 28: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/28.jpg)
Agenda
• What is coverage
• How to understand and use coverage
☞ The current status of Ruby coverage feature
• The future plan of Ruby coverage feature
• Conclusion
![Page 29: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/29.jpg)
Ruby's coverage ecosystem
• SimpleCov
• coverage.so
• Concov
![Page 30: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/30.jpg)
SimpleCov
• A wrapper library for coverage.so• Visualization with HTML• Useful features: merging, filtering, for Rails app• Author: Christoph Olszowka (@colszowka)
![Page 31: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/31.jpg)
Usage of SimpleCov
• Write this at the top of test/test_helper.rb
• Run the test suite
• coverage/index.html will be produced
– Note: SimpleCov cannot measure already-loaded files before SimpleCov.start
require "simplecov"SimpleCov.start
![Page 32: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/32.jpg)
coverage.so
• The only implementation of coverage measurement for Ruby 1.9+
• SimpleCov is a wrapper for coverage.so
• Author: me
![Page 33: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/33.jpg)
Basic usage
# test.rbrequire "coverage"Coverage.start
load "target.rb"
p Coverage.result#=> {"target.rb"=># [nil,nil,1,1,1,nil,# 0,nil,nil,nil,1]}
Start measuring coverage
Load the target file
Stop measuringand get the result
Coverage data
![Page 34: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/34.jpg)
Coverage data
# target.rb
def foo(x)if x == 0
p 0else
p 1end
end
foo(1)
[nil,nil110
nil1
nilnilnil1]
nil:Non-significant line
Number:Count executed
Untested line!
![Page 35: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/35.jpg)
Method definition is code in Ruby
# target.rb
def foo(x)if x == 0
p 0else
p 1end
end
[nil,nil100
nil0
nilnil]
Method definition iscounted as an
execution
(It is not a count ofmethod invocation!)
![Page 36: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/36.jpg)
I regret the design of coverage.so
• Support only line coverage• Excuse: I introduced it just for test
enhancement of Ruby itself– I didn't deliberate the API for external
project…
• I have wanted to make the next version better
ext/coverage/coverage.c:
69 /* Coverage provides coverage measurement feature for Ruby.70 * This feature is experimental, so these APIs may be changed in future.71 *
![Page 37: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/37.jpg)
Concov
• CONtinuous COVerage– Detects temporal change (degradation) of
coverage
• Developed for monitoring Ruby's coverage
• Author: me
• Presented at RubyKaigi 2009, and then…– It has not been used by everyone (including me)
– It was based on Ramaze (old web framework)!
![Page 38: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/38.jpg)
Concov reveals reality of Ruby dev.(Enumerable#join, 2009/07/07, M***)
New feature
introduced
with no tests!
![Page 39: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/39.jpg)
Concov reveals reality of Ruby dev.(File#size, 2009/02/25, M***)
![Page 40: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/40.jpg)
Concov reveals reality of Ruby dev.(Etc::Passwd.each, 2009/02/19, N*****)
![Page 41: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/41.jpg)
Concov reveals reality of Ruby dev.(Dir.home, 2009/02/03, N*****)
![Page 42: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/42.jpg)
Concov reveals reality of Ruby dev.(Array#sort_by!, 2009/02/03, M***)
![Page 43: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/43.jpg)
Concov reveals reality of Ruby dev.(rb_to_float, 2008/12/30, M***)
![Page 44: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/44.jpg)
Coverage ecosystem for other languages
• C/C++: GCOV/LCOV– Integrated with gcc:gcc -coverage target.c
• Java: A lot of tools– Cobertura, Emma,
Clover, JaCoCo– Integrated with CI tools
and/or IDEs
• JavaScript: IstanbulJenkins Cobertura plugin
LCOV result
![Page 45: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/45.jpg)
Agenda
• What is coverage
• How to understand and use coverage
• The current status of Ruby coverage feature
☞ The future plan of Ruby coverage feature
• Conclusion
![Page 46: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/46.jpg)
A plan towards Ruby 2.5
• Support function and branch coverage
– There have been multiple requests and some PoC patches…
• To make the API better, any comments are welcome
– https://bugs.ruby-lang.org/issues/13901
![Page 47: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/47.jpg)
API: to start measuring
# compatible layerCoverage.startCoverage.result#=> {"file.rb"=>[nil, 1, 0, …], … }
# new APICoverage.start(lines: true)Coverage.result#=> {"file.rb" => { :lines => [nil, 1, 0, …] } }
![Page 48: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/48.jpg)
API: to start other types of coverage
# enable branch and function coverageCoverage.start(lines:true,
branches:true,methods:true)
Coverage.result#=> {"file.rb" => { :lines => [nil, 1, 0, …],# :branches => {…},# :methods => {…} } }
# shorthandCoverage.start(:all)
![Page 49: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/49.jpg)
Coverage.result for branch coverage
{"target1.rb"=>{:lines=>[…],
:branches=>{[:if, 0, 2]=>{
[:then, 1, 3]=>2,[:else, 2, 5]=>1
}},:methods=>{
[:test_if, 1]=>3}}}
# target1.rb1: def test_if(x)2: if x == 03: p "x == 0"4: else5: p "x != 0"6: end7: end8: 9: test_if(0)10: test_if(0)11: test_if(1)
From if at Line 2
Jumped tothen clauseat Line 3
twice
Jumped toelse clauseat Line 5
once
![Page 50: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/50.jpg)
Coverage.result for branch coverage
{"target2.rb"=>{:lines=>[1, 1, 1, nil, nil, 1],
:branches=>{[:if, 0, 2]=>{
[:then, 1, 2]=>1,[:else, 2, 2]=>0},
[:if, 3, 3]=>{[:then, 4, 3]=>0,[:else, 5, 3]=>1}},
:methods=>{[:test_if, 1]=>3
}}}
# target2.rb1: def test_if_oneline(x)2: p "x == 0" if x == 03: p "x != 0" if x != 04: end5: 6: test_if_oneline(0)
Line coverage 100%
Branch coverage tells youthere are untested cases
![Page 51: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/51.jpg)
Discussion of API design
• 100% compatible
• [<label>, <numbering>, <line-no>]– e.g., [:if, 0, 1], [:while, 1, 1], [:case, 2, 1]– <numbering> is a unique ID to avoid conflicts for the
case where there are multiple branches in one line• LCOV-style
– Other candidates:• [<label>, <line-no>, <column-no>]
– How to handle TAB character
• [<label>, <offset from file head>]– Looks good, but hard to implement (I'll try later)
![Page 52: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/52.jpg)
Overhead of coverage measurement
(just preliminary experiment)
# Example 21: foo()2: foo()…
99: foo()100: foo()
Benchmark w/o cov. w/ cov. Overhead
Example 1 0.322 μs 6.21 μs x19.3
Example 2 1.55 μs 7.16 μs x4.61
make test-all 485 s 550 s x1.13
# Example 11: x = 12: x = 1…99: x = 1100: x = 1
![Page 53: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/53.jpg)
Demo
• Applied the new coverage.so to Ruby
• Integrated with C code coverage by GCOV and Visualized by LCOV
Ruby codein stdlib
make exam withgcc -coverage
make examCOVERAGE=1
test-coverage
.dat
*.gcda gcov
my script
run test
C codeof MRI
cov. datasource aggregate
lcov HTML
![Page 54: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/54.jpg)
Jenkins Cobertura Plugin
![Page 55: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/55.jpg)
Agenda
• What is coverage
• How to understand and use coverage
• The current status of Ruby coverage feature
• The future plan of Ruby coverage feature
☞ Conclusion
![Page 56: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/56.jpg)
Acknowledgement
• @_ko1• @t_wada• @kazu_cocoa• @moro• @makimoto• @dev_shia• @tanaka_akr• @nalsh• @spikeolaf• @k_tsj
![Page 57: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/57.jpg)
Conclusion
• What is coverage, how important in Ruby, and how to understand coverage
• The current status of Ruby's coverage measurement and ecosystem
• A plan towards Ruby 2.5 and preliminary demo– Any comments are welcome!
– https://bugs.ruby-lang.org/issues/13901
![Page 58: An introduction and future of Ruby coverage library](https://reader034.fdocuments.in/reader034/viewer/2022050614/5a6478657f8b9a27568b45d9/html5/thumbnails/58.jpg)
Future work
• Determine the API
• define_method as method coverage
• &. as branch coverage
• Callsite coverage
• Block coverage
obj.foo.bar
ary.map { …… }