Download - Software testing and quality assurance

Transcript
Page 1: Software testing and quality assurance

Software Testing & Quality Assurance

Why using Specflow (Gherkin) and Selenium ?

Benjamin Baumann@ZentiltoutouOctober 21st 2016

Page 2: Software testing and quality assurance

How much costs a bug?• 1962 NASA - Mariner 1

Omission of a hyphen in instructions. Auto destruct. $18M• 1996 Ariane 5

Integer Overflow on old legacy code that is not needed. No flight simulator to spare money. $370M for satellites alone

Page 3: Software testing and quality assurance

Coder punchline #1

Page 4: Software testing and quality assurance

How much costs a bug?• 2009 Toyota

Lag in the anti-lock-brake system. Recall 9M cars. $3G estimated• 2012 Knight Capital Group

Code deployed to all but one server. This server sent out 4M+ wrong stock orders in one hour. $440M

Page 5: Software testing and quality assurance
Page 6: Software testing and quality assurance

But we’re no Spacecraft or trading firm!• 2015 Eurosport Player V6

Video lags, fps drop, overheating PC, angry clients.Due to incompatibility between technology and requirements. Detected by beta testing but gone live before the test results

Page 7: Software testing and quality assurance

Ok. Let’s test! White-box? Black-box?• White-box testing

Tests the internal structure or workings of a program :• Unit tests• Functional testing• Integration testing• Regression testing• Load & Stress testing• Sanity & Smoke testing

• Black-box testingSoftware is a black box. Tester knows nothing about how it works.• Acceptance testing• Beta testing

Page 8: Software testing and quality assurance

Unit tests are idiots.

Page 9: Software testing and quality assurance

Unit tests are idiots.

Page 10: Software testing and quality assurance

Unit tests are idiots but legion!• Think about code coverage• Assess code quality, not software’s• Think about edge cases, error handling

Page 11: Software testing and quality assurance

Unit tests are idiots, legion and hard to maintain• Each code change/writing begin with test change/writing• « I’m a developer, these tests are idiots! Why do I have to write

them? »• « The tests failed, such a pain. I’ll just comment them out »

Page 12: Software testing and quality assurance

Break #1 : That’s not me. All internet is saying it!

Page 13: Software testing and quality assurance

That’s not me. All internet is saying it!

Page 14: Software testing and quality assurance

That’s not me. All internet is saying it!

Page 15: Software testing and quality assurance

That’s not me. All internet is saying it!

Page 16: Software testing and quality assurance

Still, I want to do unit testing• Please, do unit testing only if :• You have a decent number of developers in your team• You have time to do it and to maintain it• You can run them and validate them automatically (through CI)• You run a business critical project• You have detailled technical specs• Coding unit test is still coding, treat your tests like your code• Don’t write the code AND the tests, make someone else write the tests.

• Golden rule : « Each unit test case should be very limited in scope »

Page 17: Software testing and quality assurance

Ok I have a ton of unit tests. Am I OK?

Page 18: Software testing and quality assurance

Ok I have a ton of unit tests. Am I OK?

Page 19: Software testing and quality assurance

Focus on functional,regression and acceptance tests• Testing software features above code methods• Less detailed, less to do, less to maintain• Focus on testing the requirements/specs• Safeguard for developers

Page 20: Software testing and quality assurance

Ok let’s test a feature !• Describe the feature : Who? What? Why?• « In order to watch the player

As a Eurosport Player user I want to be able to connect»• Help describing the need• Feature-centric• Technology agnostic• Need for testing scenarios

Page 21: Software testing and quality assurance

Ok let’s write a scenario !Given I go to the Eurosport Player home pageAnd I open the connection modalAnd I enter my credentialsWhen I submit the formThen I must be logged in

White-box? Black-box?

Page 22: Software testing and quality assurance

Ok let’s write a scenario!Given I go to the Eurosport Player home pageAnd I open the connection modalAnd I enter my credentialsWhen I submit the formThen I must be logged in

White-box! Describing a feature should be blackboxed.

Page 23: Software testing and quality assurance

Ok let’s write a better scenario!Given I go to the Eurosport PlayerWhen I loginThen I must be able to watch Eurosport Player

No tech. No design stuff. Just the feature.

Feature does not change when code is refactored.You should rewrite the code behind, but not the test scenario

Page 24: Software testing and quality assurance

Break #2: Understanding others

Page 25: Software testing and quality assurance

Break #2: Understanding others• « Hey, this is not the behaviour I asked for !», « But that’s what you

told me in your mail… », « You did not understand, that’s not what I wanted » • « Domain, Network, AD, I mix up all these, it’s gibberish to me »• Solutions• Learn the other language so that you understand them• Everyone learns a new language so that we understand each other

(AWS)• Don’t go into details, just do high level• Speak like you’d speak to the average man in the street

Page 26: Software testing and quality assurance

What about Gherkin?• We’ve already used Gherkin• Gherkin is the language that Specflow understands. It is a

Business Readable, Domain Specific Language that lets you describe software’s behaviour without detailing how that behaviour is implemented.• Gherkin serves two purposes — documentation and automated tests.

The third is a bonus feature — when it yells in red it’s talking to you, telling you what code you should write.

Page 27: Software testing and quality assurance

Documentation by Gherkin• List features• List testing scenarios• Readable by EVERYONE• Hard to write, easy to understand• A heaven for maintenance

• http://qa.sti.eurosport.intra/Home/SpecflowList?path=d%3A%5Csites%5CQA%5CProjects%5CEurosport%20Player%20PROD%5CScenarios&projectName=Eurosport%20Player%20PROD

Page 28: Software testing and quality assurance

Automated tests with Specflow• Each scenario line is linked to a code method• Pass scenario data to the method• Scenario Context (browser, language, environment…)• Runnable in classic test runners (NUnit, MsTest …)• Explicit errors• When failed, need to go deeper with unit tests or by hand

Page 29: Software testing and quality assurance

And what about Selenium?• Selenium WebDriver is a library that can drive a web browser• Method on the webdriver results on « real user » action

• Let’ code!

Page 30: Software testing and quality assurance

Merci!