Lecture #6. automation testing (andrey oleynik)
-
Upload
andrey-oleynik -
Category
Software
-
view
184 -
download
0
Transcript of Lecture #6. automation testing (andrey oleynik)
Agenda
1. Testing pyramid2. Automated testing vs. manual testing3. ROI4. Test Automation tools5. Testing methodologies6. Automation testing framework7. More details8. Homework
2
About myself
Andrey Oleynik
• About 7 years of work in Quality Assurance area
• QA Automation Engineer
• PhD, Theoretical physics
3
Manual Testing
• Manual testing is the process through which QA engineers run tests manually, comparing program expectations and actual outcomes in order to find software defects.
8
• A tester uses the program as an end user and checks whether or not the program acts appropriately.
• Manual testing is a good fit for smaller projects as well as companies without significant financial resources.
Automated Testing
• Test automation is the use of special software (separate from the software being tested) to control the execution of tests and the comparison of actual outcomes with predicted outcomes.
• If the program expectations and outcomes align, your project is behaving as it should, and you are likely bug free.
• If the two don’t align, however, there is an issue that needs to be investigated. You should check if there is an issue in the tested software or an issue in the testing tool or test script.
• Automated testing is good to use when the project is large, and we have enough resources.
9
General approaches to test automation
• Code-driven testingPublic (usually) interfaces to
classes, modules, libraries are tested.
• Graphical user interface testingA testing framework generates
user interface events such as
keystrokes and mouse clicks and checks
the result.
• API driven testingA testing framework that uses a programming interface to the application to validate the behavior under test. 10
Advantages
11
Automated Testing Manual Testing
• If you have to run a set of tests repeatedly automation is a huge gain
• If Test Cases have to be run a small number of times it's more likely to perform manual testing
• Helps performing "compatibility testing" - testing the software on different configurations
• It allows the tester to perform more ad-hoc (random testing)
• It gives you the ability to run automation scenarios to perform regressions in a shorter time
• Short term costs are reduced
• It gives you the ability to run regressions on a code that is continuously changing
• The more time tester spends testing a module the grater the odds to find real user bugs
• Can be run simultaneously on different machines thus decreasing testing time
• Long term costs are reduced
Disadvantages
12
Automated Testing Manual Testing
• It's more expensive to automate. Initial investments are bigger than manual testing
• Manual tests can be very time consuming
• You cannot automate everything, some tests still have to be done manually
• For every release you must rerun the same set of tests which can be tiresome
Other factors
• The performance of your test tools
• The knowledge level of your testing team
• The continuous growth of software to be tested
• Number of necessary regressions
13
False Expectations
• Immediate Test Effort Reduction
• One Test Tool Fits All
• Tool Ease of Use
• 100% Test Coverage
14
Return on InvestmentROR, Rate of return
• Automation Return-on-Investment (ROI) is a calculation of automated testing benefits in terms of: Cost Savings, Increased Efficiency and Increased Software Quality (reduced risks).
• return on investment = (gain from investment - cost of investment) / cost of investment
• gain from investment – manual testing efforts
• cost of investment – automation testing efforts
• It is a good case, if ROI >=1 15
ROI estimation(what to take into account?)
• Tool for automation
• Training
• Test framework development
• Automated test environment
• Test development
• Test execution and support
• Frequency of test execution
16
HP Unified Functional Testing (HP QTP)
Recording or coding Flash applications Keyword driven testing Data driven testing Web Service testing
VBScript
Weak IDE Only on Windows platform Only IE Proprietary
22
Keyword-driven testing• Also known as table-driven testing or action word based
testing
• A software testing methodology suitable for both manual and automated testing. This method separates the documentation of test cases -including the data to use- from the prescription of the way the test cases are executed.
• We use a special table format to specify test cases: we define keywords (or action words) for each function we want to test. Below the keywords we set up our test data.
23
A B C D E
. Name Address Postcode City
Enter Client Jane Smith 6 High Street London SE25 6EP
Keyword-driven testing
• Obviously, we need reusability, modularity and readability.
• Create different keywords for different set of operations and in the main script (testcase) we can just refer to these keywords. So we can separate tests from code.
• Simplify test development
• Automated testcases are readable for human
QTP
TestComplete
SilkTest
etc.
25
Data driven testing
• Data-driven testing (DDT) is a term used in the testing of computer software to describe testing done using a table of conditions directly as test inputs and verifiable outputs as well as the process where test environment settings and control are not hard-coded.
• In the simplest form the tester supplies the inputs from a row in the table and expects the outputs which occur in the same row.
QTP
TestComplete
SilkTest
TestNG!
etc.
27
Data driven testing
Sources:
Data pools
ODBC / JDBC sources
CSV files
Excel files
DAO objects
ADO objects
…
28
TestComplete
Recording or coding
Keyword driven testing
Data driven testing VBScript
JScript
DelphiScript
C++Script
C#Script
Internet Explorer, Firefox, Google Chrome
Only on Windows platform
Proprietary
31
Sikuli
GUI testing tool
MIT license
Jython
It is possible to compile your Java tool
JVM => cross-platform
32
Selenium
• Selenium IDE
• Selenium Remote Control
• Selenium WebDriver
• Selenium Grid
• Open source (Apache License 2.0)
• Cross-platform and Cross-browser
• Java, C#, Groovy, Perl, PHP, Python and Ruby
33
Selenium WebDriver
36
HtmlUnitDriver;
FirefoxDriver;
InternetExplorerDriver;
ChromeDriver;
OperaDriver;
SafariDriver;
AndriodDriver;
IphoneDriver.
Selenium (locators)
• By.id;
• By.name:
• By.xpath;
• By.tagName;
• By.cssSelector;
• By.className;
• By.linkTex;
• By.partionalLinkText;
38
WebDriver (example)package org.openqa.selenium.example;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.support.ui.ExpectedCondition;
import org.openqa.selenium.support.ui.WebDriverWait;
public class Selenium2Example {
public static void main(String[] args) {
WebDriver driver = new FirefoxDriver();
driver.get("http://www.google.com");
WebElement element = driver.findElement(By.name("q"));
element.sendKeys("Cheese!");
element.submit();
System.out.println("Page title is: " + driver.getTitle());
driver.quit();
}
} 39
Test automation framework
• Test automation framework is a set of assumptions, concepts and tools that provide support for automated software testing
Automation tool
Execution
Distributed system
Reusable components
Logging/reporting
Integration with CI
…40
BDD
41
Feature: New accountIn order to have new ready to use accountAs a userI want to get an account with balance $0
Scenario: Create new accountGiven I have no accountWhen I create an accountThen I should have an account with balance $0
Thucydides
Crossplatform
Selenium
Java
Page object
BDD framework (JBehave)
JUnit
Jira integration
Maven
52
Cucumber
53
Crossplatform
Ruby (usually)
MIT license
BDD framework
Jenkins plugins
Usually used with Selenium / Capybara / WATIR
page-object gem
More details
57
• Портал об автоматизации тестирования ПО:
http://automated-testing.info
• Конференции sqadays: http://www.sqadays.com
• Про Тестинг: http://www.protesting.ru/
• http://software-testing.ru
• Official Cucumber site: http://cukes.info/
• https://github.com/jonathanchrisp/selenium-cucumber-ruby-kickstarter
• Official Thucydides site : http://www.thucydides.info
• http://www.thucydides.info/docs/thucydides-one-page/thucydides.html
• Automated testing with Selenium and Cucumber: http://www.ibm.com/developerworks/opensource/library/a-automating-ria/index.html
More details
58
• Official Selenium Documentation: http://docs.seleniumhq.org/docs
• https://code.google.com/p/selenium
• W3 Schools XPath Tutorial: http://www.w3schools.com/XPath
• XPath в примерах: http://zvon.org/xxl/XPathTutorial/General_rus/examples.html
• Selenium Tips CSS Selectors: http://saucelabs.com/resources/selenium/css-selectors
• Tutorial: Writing XPath selectors for Selenium tests: http://hedleyproctor.com/2011/05/tutorial-writing-xpath-selectors-for-selenium-tests
More details
59
• Gojko Adzic Specification by Example
• Криспин, Грегори Гибкое тестирование
• Э.Дастин, Дж. Рэшка, Дж.Пол Автоматизированное тестирование программного обеспечения. Внедрение, управление и эксплуатация
More details
60
• Matt Wynne and Aslak Hellesoy The Cucumber Book: Behaviour-Driven Development for Testers and Developers
• David Burns Selenium 2 Testing Tools: Beginners Guide
• Unmesh Gundecha Selenium Testing Tools Cookbook
Home task
61
• Find out if we need automated testing for
1. Mobile application project (2 month, 4 iterations)
2. Startup. A website (6 month, 12 iterations). Scope is not defined and will be changed a few times.
3. Enterprise application. 10 versions already released. (12 month, about 24 iterations) Core functionality is the same. Some new features added/ some features updated.