Software Testing & Quality Assurance
Why using Specflow (Gherkin) and Selenium ?
Benjamin Baumann@ZentiltoutouOctober 21st 2016
How much costs a bug?• 1962 NASA - Mariner 1
Omission of a hyphen in instructions. Auto destruct. $18M• 1996 Ariane 5
Integer Overflow on old legacy code that is not needed. No flight simulator to spare money. $370M for satellites alone
Coder punchline #1
How much costs a bug?• 2009 Toyota
Lag in the anti-lock-brake system. Recall 9M cars. $3G estimated• 2012 Knight Capital Group
Code deployed to all but one server. This server sent out 4M+ wrong stock orders in one hour. $440M
But we’re no Spacecraft or trading firm!• 2015 Eurosport Player V6
Video lags, fps drop, overheating PC, angry clients.Due to incompatibility between technology and requirements. Detected by beta testing but gone live before the test results
Ok. Let’s test! White-box? Black-box?• White-box testing
Tests the internal structure or workings of a program :• Unit tests• Functional testing• Integration testing• Regression testing• Load & Stress testing• Sanity & Smoke testing
• Black-box testingSoftware is a black box. Tester knows nothing about how it works.• Acceptance testing• Beta testing
Unit tests are idiots.
Unit tests are idiots.
Unit tests are idiots but legion!• Think about code coverage• Assess code quality, not software’s• Think about edge cases, error handling
Unit tests are idiots, legion and hard to maintain• Each code change/writing begin with test change/writing• « I’m a developer, these tests are idiots! Why do I have to write
them? »• « The tests failed, such a pain. I’ll just comment them out »
Break #1 : That’s not me. All internet is saying it!
That’s not me. All internet is saying it!
That’s not me. All internet is saying it!
That’s not me. All internet is saying it!
Still, I want to do unit testing• Please, do unit testing only if :• You have a decent number of developers in your team• You have time to do it and to maintain it• You can run them and validate them automatically (through CI)• You run a business critical project• You have detailled technical specs• Coding unit test is still coding, treat your tests like your code• Don’t write the code AND the tests, make someone else write the tests.
• Golden rule : « Each unit test case should be very limited in scope »
Ok I have a ton of unit tests. Am I OK?
Ok I have a ton of unit tests. Am I OK?
Focus on functional,regression and acceptance tests• Testing software features above code methods• Less detailed, less to do, less to maintain• Focus on testing the requirements/specs• Safeguard for developers
Ok let’s test a feature !• Describe the feature : Who? What? Why?• « In order to watch the player
As a Eurosport Player user I want to be able to connect»• Help describing the need• Feature-centric• Technology agnostic• Need for testing scenarios
Ok let’s write a scenario !Given I go to the Eurosport Player home pageAnd I open the connection modalAnd I enter my credentialsWhen I submit the formThen I must be logged in
White-box? Black-box?
Ok let’s write a scenario!Given I go to the Eurosport Player home pageAnd I open the connection modalAnd I enter my credentialsWhen I submit the formThen I must be logged in
White-box! Describing a feature should be blackboxed.
Ok let’s write a better scenario!Given I go to the Eurosport PlayerWhen I loginThen I must be able to watch Eurosport Player
No tech. No design stuff. Just the feature.
Feature does not change when code is refactored.You should rewrite the code behind, but not the test scenario
Break #2: Understanding others
Break #2: Understanding others• « Hey, this is not the behaviour I asked for !», « But that’s what you
told me in your mail… », « You did not understand, that’s not what I wanted » • « Domain, Network, AD, I mix up all these, it’s gibberish to me »• Solutions• Learn the other language so that you understand them• Everyone learns a new language so that we understand each other
(AWS)• Don’t go into details, just do high level• Speak like you’d speak to the average man in the street
What about Gherkin?• We’ve already used Gherkin• Gherkin is the language that Specflow understands. It is a
Business Readable, Domain Specific Language that lets you describe software’s behaviour without detailing how that behaviour is implemented.• Gherkin serves two purposes — documentation and automated tests.
The third is a bonus feature — when it yells in red it’s talking to you, telling you what code you should write.
Documentation by Gherkin• List features• List testing scenarios• Readable by EVERYONE• Hard to write, easy to understand• A heaven for maintenance
• http://qa.sti.eurosport.intra/Home/SpecflowList?path=d%3A%5Csites%5CQA%5CProjects%5CEurosport%20Player%20PROD%5CScenarios&projectName=Eurosport%20Player%20PROD
Automated tests with Specflow• Each scenario line is linked to a code method• Pass scenario data to the method• Scenario Context (browser, language, environment…)• Runnable in classic test runners (NUnit, MsTest …)• Explicit errors• When failed, need to go deeper with unit tests or by hand
And what about Selenium?• Selenium WebDriver is a library that can drive a web browser• Method on the webdriver results on « real user » action
• Let’ code!
Merci!
Top Related