Lecture #6. automation testing (andrey oleynik)

63
Test Automation Andrey Oleynik 2014

Transcript of Lecture #6. automation testing (andrey oleynik)

Test Automation

Andrey Oleynik

2014

Agenda

1. Testing pyramid2. Automated testing vs. manual testing3. ROI4. Test Automation tools5. Testing methodologies6. Automation testing framework7. More details8. Homework

2

About myself

Andrey Oleynik

• About 7 years of work in Quality Assurance area

• QA Automation Engineer

• PhD, Theoretical physics

3

History

Fred Brooks

4

Kent Beck

History

5

John Ferguson Smart

Aslak Hellesøy

Dan North

Testing levels

6

Testing pyramid

7

Manual Testing

• Manual testing is the process through which QA engineers run tests manually, comparing program expectations and actual outcomes in order to find software defects.

8

• A tester uses the program as an end user and checks whether or not the program acts appropriately.

• Manual testing is a good fit for smaller projects as well as companies without significant financial resources.

Automated Testing

• Test automation is the use of special software (separate from the software being tested) to control the execution of tests and the comparison of actual outcomes with predicted outcomes.

• If the program expectations and outcomes align, your project is behaving as it should, and you are likely bug free.

• If the two don’t align, however, there is an issue that needs to be investigated. You should check if there is an issue in the tested software or an issue in the testing tool or test script.

• Automated testing is good to use when the project is large, and we have enough resources.

9

General approaches to test automation

• Code-driven testingPublic (usually) interfaces to

classes, modules, libraries are tested.

• Graphical user interface testingA testing framework generates

user interface events such as

keystrokes and mouse clicks and checks

the result.

• API driven testingA testing framework that uses a programming interface to the application to validate the behavior under test. 10

Advantages

11

Automated Testing Manual Testing

• If you have to run a set of tests repeatedly automation is a huge gain

• If Test Cases have to be run a small number of times it's more likely to perform manual testing

• Helps performing "compatibility testing" - testing the software on different configurations

• It allows the tester to perform more ad-hoc (random testing)

• It gives you the ability to run automation scenarios to perform regressions in a shorter time

• Short term costs are reduced

• It gives you the ability to run regressions on a code that is continuously changing

• The more time tester spends testing a module the grater the odds to find real user bugs

• Can be run simultaneously on different machines thus decreasing testing time

• Long term costs are reduced

Disadvantages

12

Automated Testing Manual Testing

• It's more expensive to automate. Initial investments are bigger than manual testing

• Manual tests can be very time consuming

• You cannot automate everything, some tests still have to be done manually

• For every release you must rerun the same set of tests which can be tiresome

Other factors

• The performance of your test tools

• The knowledge level of your testing team

• The continuous growth of software to be tested

• Number of necessary regressions

13

False Expectations

• Immediate Test Effort Reduction

• One Test Tool Fits All

• Tool Ease of Use

• 100% Test Coverage

14

Return on InvestmentROR, Rate of return

• Automation Return-on-Investment (ROI) is a calculation of automated testing benefits in terms of: Cost Savings, Increased Efficiency and Increased Software Quality (reduced risks).

• return on investment = (gain from investment - cost of investment) / cost of investment

• gain from investment – manual testing efforts

• cost of investment – automation testing efforts

• It is a good case, if ROI >=1 15

ROI estimation(what to take into account?)

• Tool for automation

• Training

• Test framework development

• Automated test environment

• Test development

• Test execution and support

• Frequency of test execution

16

Testing tools

17

HP QTP (expert view)

18

HP QTP (object repository)

19

QTP (integration with QC)

20

QTP (reports)

21

HP Unified Functional Testing (HP QTP)

Recording or coding Flash applications Keyword driven testing Data driven testing Web Service testing

VBScript

Weak IDE Only on Windows platform Only IE Proprietary

22

Keyword-driven testing• Also known as table-driven testing or action word based

testing

• A software testing methodology suitable for both manual and automated testing. This method separates the documentation of test cases -including the data to use- from the prescription of the way the test cases are executed.

• We use a special table format to specify test cases: we define keywords (or action words) for each function we want to test. Below the keywords we set up our test data.

23

A B C D E

. Name Address Postcode City

Enter Client Jane Smith 6 High Street London SE25 6EP

Example

24

Keyword-driven testing

• Obviously, we need reusability, modularity and readability.

• Create different keywords for different set of operations and in the main script (testcase) we can just refer to these keywords. So we can separate tests from code.

• Simplify test development

• Automated testcases are readable for human

QTP

TestComplete

SilkTest

etc.

25

QTP (keyword view)

26

Data driven testing

• Data-driven testing (DDT) is a term used in the testing of computer software to describe testing done using a table of conditions directly as test inputs and verifiable outputs as well as the process where test environment settings and control are not hard-coded.

• In the simplest form the tester supplies the inputs from a row in the table and expects the outputs which occur in the same row.

QTP

TestComplete

SilkTest

TestNG!

etc.

27

Data driven testing

Sources:

Data pools

ODBC / JDBC sources

CSV files

Excel files

DAO objects

ADO objects

28

TestComplete

29

TestComplete

30

TestComplete

Recording or coding

Keyword driven testing

Data driven testing VBScript

JScript

DelphiScript

C++Script

C#Script

Internet Explorer, Firefox, Google Chrome

Only on Windows platform

Proprietary

31

Sikuli

GUI testing tool

MIT license

Jython

It is possible to compile your Java tool

JVM => cross-platform

32

Selenium

• Selenium IDE

• Selenium Remote Control

• Selenium WebDriver

• Selenium Grid

• Open source (Apache License 2.0)

• Cross-platform and Cross-browser

• Java, C#, Groovy, Perl, PHP, Python and Ruby

33

Selenium IDE

34

Selenium RC

35

Selenium WebDriver

36

HtmlUnitDriver;

FirefoxDriver;

InternetExplorerDriver;

ChromeDriver;

OperaDriver;

SafariDriver;

AndriodDriver;

IphoneDriver.

Selenium Grid

37

Selenium (locators)

• By.id;

• By.name:

• By.xpath;

• By.tagName;

• By.cssSelector;

• By.className;

• By.linkTex;

• By.partionalLinkText;

38

WebDriver (example)package org.openqa.selenium.example;

import org.openqa.selenium.By;

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.WebElement;

import org.openqa.selenium.firefox.FirefoxDriver;

import org.openqa.selenium.support.ui.ExpectedCondition;

import org.openqa.selenium.support.ui.WebDriverWait;

public class Selenium2Example {

public static void main(String[] args) {

WebDriver driver = new FirefoxDriver();

driver.get("http://www.google.com");

WebElement element = driver.findElement(By.name("q"));

element.sendKeys("Cheese!");

element.submit();

System.out.println("Page title is: " + driver.getTitle());

driver.quit();

}

} 39

Test automation framework

• Test automation framework is a set of assumptions, concepts and tools that provide support for automated software testing

Automation tool

Execution

Distributed system

Reusable components

Logging/reporting

Integration with CI

…40

BDD

41

Feature: New accountIn order to have new ready to use accountAs a userI want to get an account with balance $0

Scenario: Create new accountGiven I have no accountWhen I create an accountThen I should have an account with balance $0

BDD tools

• Jbehave (Java)

• Thucidydes

• Rspec (Ruby)

• Cucumber

• SpecFlow (C#)

• …

42

Thucydides (Java)

43

Thucydides (Structure)

44

Thucydides (PageObject)

45

Thucydides (Steps)

46

Thucydides (Story)

47

Thucydides (Story)

48

Thucydides (Story)

49

Thucydides

50

Thucydides

51

Thucydides

Crossplatform

Selenium

Java

Page object

BDD framework (JBehave)

JUnit

Jira integration

Maven

52

Cucumber

53

Crossplatform

Ruby (usually)

MIT license

BDD framework

Jenkins plugins

Usually used with Selenium / Capybara / WATIR

page-object gem

Cucumber report

54

Cucumber report

55

Cucumber test result trend

56

More details

57

• Портал об автоматизации тестирования ПО:

http://automated-testing.info

• Конференции sqadays: http://www.sqadays.com

• Про Тестинг: http://www.protesting.ru/

• http://software-testing.ru

• Official Cucumber site: http://cukes.info/

• https://github.com/jonathanchrisp/selenium-cucumber-ruby-kickstarter

• Official Thucydides site : http://www.thucydides.info

• http://www.thucydides.info/docs/thucydides-one-page/thucydides.html

• Automated testing with Selenium and Cucumber: http://www.ibm.com/developerworks/opensource/library/a-automating-ria/index.html

More details

58

• Official Selenium Documentation: http://docs.seleniumhq.org/docs

• https://code.google.com/p/selenium

• W3 Schools XPath Tutorial: http://www.w3schools.com/XPath

• XPath в примерах: http://zvon.org/xxl/XPathTutorial/General_rus/examples.html

• Selenium Tips CSS Selectors: http://saucelabs.com/resources/selenium/css-selectors

• Tutorial: Writing XPath selectors for Selenium tests: http://hedleyproctor.com/2011/05/tutorial-writing-xpath-selectors-for-selenium-tests

More details

59

• Gojko Adzic Specification by Example

• Криспин, Грегори Гибкое тестирование

• Э.Дастин, Дж. Рэшка, Дж.Пол Автоматизированное тестирование программного обеспечения. Внедрение, управление и эксплуатация

More details

60

• Matt Wynne and Aslak Hellesoy The Cucumber Book: Behaviour-Driven Development for Testers and Developers

• David Burns Selenium 2 Testing Tools: Beginners Guide

• Unmesh Gundecha Selenium Testing Tools Cookbook

Home task

61

• Find out if we need automated testing for

1. Mobile application project (2 month, 4 iterations)

2. Startup. A website (6 month, 12 iterations). Scope is not defined and will be changed a few times.

3. Enterprise application. 10 versions already released. (12 month, about 24 iterations) Core functionality is the same. Some new features added/ some features updated.

Q & A

62

The End

• Thank you!

63