RAILSCLUB 2015

Testingand Software Writer a year later

Simon BagreevHead of Ruby development

Rambler&Co

RAILSCLUB 2015

whoami

status_200

semmin

Simon Bagreev

RAILSCLUB 2015

US

RAILSCLUB 2015

RailsConf 2014

RAILSCLUB 2015

TDD

● doesn’t test anything

● complicates design

● holy trinity (speed, coverage, ratio)

● adds cost, not value

RAILSCLUB 2015

Software Writer

● does the whole thing work?

● testing at the higher level (system tests)

● adequate testing

RAILSCLUB 2015

More problems

● no testing

● false confidence

● holy wars

● ignored failing tests

● endless test suite (due to units)

● “virtual” QA department

RAILSCLUB 2015

No testing

● no proof that features

work

● no way to catch

regressions

● impossible to maintain

RAILSCLUB 2015

No testing - TODO

● start with user stories

● write acceptance specs

● move to system tests

● create “visual” tests

RAILSCLUB 2015

False confidence

● many (unit) tests

● runs fast

● 100% coverage

RAILSCLUB 2015

About testing

“Testing shows the presence, not the absence of bugs.”

Edsger W. Dijkstra

NATO Software Engineering Conference, 1969

RAILSCLUB 2015

Value of unit testing

“Most software failures come from the interactions between objects rather than being a property of an object or method in isolation.”

“ Functional testing typically finds twice as many bugs as unit testing ”

James Coplien

Why Most Unit Testing is Waste

RAILSCLUB 2015

Coverageclass Event

def initialize(started_at)

@started_at = started_at

end

def today?

(Time.now.beginning_of_day..Time.now.end_of_day).cover? @started_at

end

end

RAILSCLUB 2015

100% coverageRSpec.describe Event do

subject { Event.new(start_time) }

context 'today' do

let(:start_time) { Time.now }

it 'todays event' do

expect(subject).to be_today

end

end

end

RAILSCLUB 2015

100% coverage

no expectation

RSpec.describe Event do

subject { Event.new(start_time) }

context 'today' do

let(:start_time) { Time.now }

it 'todays event' do

subject.today?

end

end

end

RAILSCLUB 2015

● mutation testing● higher level testing

● “Create system tests with good feature coverage (not code coverage)”

James Coplien

Why Most Unit Testing is Waste

False confidence TODO

RAILSCLUB 2015

Holy wars

● RSpec vs MiniTest

● Capybara vs Cucumber

● AR fixtures vs FactoryGirl

● stubs vs mocks

● PhantomJS vs Selenium

● etc...

RAILSCLUB 2015

Ignored failures

“Oh, you have to invoke Maven with this flag that turns off those tests — they are tests that no longer work!”

James Coplien

Why Most Unit Testing is Waste

RAILSCLUB 2015

“Rotten” tests

“If you want to reduce your test mass, the number one thing you should do is look at the tests that have never failed in a year and consider throwing them away.”

James Coplien

Why Most Unit Testing is Waste

RAILSCLUB 2015

rspec-rotten, FTW

“rotten” – haven’t changed status (failed/passed /pending) in extended period of time

https://github.com/rambler-digital-solutions/rspec-rotten

RAILSCLUB 2015

rspec-rotten

● uses custom rspec formatter● subscribes to rspec notifications● after every test run saves stats into json file:

{

"id": "./spec/controllers/calendar_controller_spec.rb[1:1:1]",

"state": "failed",

"date": "2015-09-21T22:04:39.063+03:00",

"location": "./spec/controllers/calendar_controller_spec.rb:13",

"description": "render the :show template"

}

RAILSCLUB 2015

rspec-rotten

Notifies you about “rotten” tests:

RAILSCLUB 2015

Endless unit testsWeinberg’s Law of Decomposition

RAILSCLUB 2015

Endless unit testsAcceptance test

RAILSCLUB 2015

Endless unit testsUnit testing

RAILSCLUB 2015

Endless unit tests

(Testing) is identifying and resolving problems that arise in the system as a whole rather than in individual components. Testing is done by groups, as an individual programmer lacks the total knowledge needed to test large sections of the system. The emphasis in testing should never be on individual modules.

Edsger W. Dijkstra

NATO Software Conference, 1969

RAILSCLUB 2015

Additionally

● hard to maintain (tests are code too, may contain bugs)

● hard to change (tends to become “binary”)● may never occur in prod

RAILSCLUB 2015

Many unit tests TODO

● mutation testing

● rspec-rotten

● “Software Writer’s” approach: “If this test fails,

what business requirement is compromised?”

RAILSCLUB 2015

“Virtual” QA dept

“My unit tests pass, QA / PM / Product Owner will take care of the rest” ... Nope

another team

RAILSCLUB 2015

“Virtual” QA TODO

● focus on acceptance specs (outside-in)

● “visual” tests for end-user regressions

RAILSCLUB 2015

rspec-visual, FTW

"visual" testing with rspec via screenshot comparison

https://github.com/rambler-digital-solutions/rspec-visual

RAILSCLUB 2015

rspec-visual step 1

# spec/features/visual/home_page_spec.rb

describe 'home page', type: :feature, visual: true do

it 'home_page' do |example|

visit '/'

take_screenshot(page, example)

should look_like example.description

end

end

Create “visual” test

RAILSCLUB 2015

rspec-visual step 2

# spec/features/visual/home_page_spec.rb

describe 'home page', type: :feature, visual: true do

it 'home_page' do |example|

visit '/'

take_screenshot(page, example)

should look_like example.description

end

end

Take a screenshot

RAILSCLUB 2015

rspec-visual step 3

# spec/features/visual/home_page_spec.rb

describe 'home page', type: :feature, visual: true do

it 'home_page' do |example|

visit '/'

take_screenshot(page, example)

should look_like example.description

end

end

Use look_like matcher to compare with “stable”

RAILSCLUB 2015

rspec-visual step 3

# rspec/visual/.../look_like_matcher.rb

RSpec::Matchers.define :look_like do |expected|

# ... set file paths

match do |actual|

# ... copy to stable folder if not found

system "compare -metric PAE -subimage-search -dissimilarity-threshold 1 \

#{expected_image_file_path} #{actual_image_file_path} #{diff_file_path}"

end

# ... customize failure message

end

look_like matcher

RAILSCLUB 2015

rspec-visual result

!=

RAILSCLUB 2015

rspec-visual diff

RAILSCLUB 2015

rspec-visual result

!=

RAILSCLUB 2015

rspec-visual diff

RAILSCLUB 2015

rspec-visual gotchas

● dependencies (Capybara, Poltergeist + PhantomJS, imagemagick)

● only supports Poltergeist● dynamic data● speed

RAILSCLUB 2015

To conclude

As a Software Writer:

● focus on what really matters● throw away tests that have no business

value● do “visual” testing● use Thought-Driven Development

RAILSCLUB 2015

Contacts

В группе компаний Rambler&Co всегда есть открытые вакансии для тех, кто

хочет профессионально расти и развиваться, занимаясь тем, что по-

настоящему нравится

hr@rambler-co.ru

www.rambler-co.ru/jobs

RAILSCLUB 2015

Links

RailsConf 2014 Keynote: Writing Software by DHH: http://bit.ly/1LwrENX

Why Most Unit Testing is Waste, James Coplien: http://bit.ly/1kBMor1

Why Most Unit Testing is Waste part 2, James Coplien: http://bit.ly/1OuzrOc

NATO Software Engineering Conference 1969: http://bit.ly/1iwANKT

rspec-visual: https://github.com/rambler-digital-solutions/rspec-visual

rspec-rotten: https://github.com/rambler-digital-solutions/rspec-rotten

RAILSCLUB 2015

Thanks

Questions?