Post on 05-Dec-2014
description
Mobiltest automation - experiences from the trenches
Let’s Talk Apple, 11. sep, 2014 Niels Frydenholm, ebay Classifieds
eBay Inc. Confidential2
Agenda
• Intro
• Tools
• Examples / Demo
• Continuous Integration / Follow up
• Challenges
• Latest improvements
• Questions?
eBay Inc. Confidential3
eBay Classifieds, Denmark
• DBA, BilBasen og Bilinfo
• DBA (a classifieds site/app) has approx. 1 mill downloads
• BilBasen (cars) has approx. 450.000 downloads
• Both brands have apps for iPhone, iPad and Android
• Traffic from mobile doubled during 2013
• Strive for rapid releases
eBay Inc. Confidential4
How we are organised
• Developers • QA
• Product Owner • UX • Other stakeholders
eBay Inc. Confidential5
Tools
Jenkins (CI)
eBay Inc. Confidential6
Cucumber -‐ more than a test tool
eBay Inc. Confidential6
Cucumber -‐ more than a test tool
eBay Inc. Confidential6
Cucumber -‐ more than a test tool
• Behaviour-‐Driven Development
• Adds “power” to Gherkin syntax
• A great way to communicate and cooperate between “the business people”, developers and QA.
• Written in (your) business/domain language
• Easy to both read and write
eBay Inc. Confidential6
Cucumber -‐ more than a test tool
• Behaviour-‐Driven Development
• Adds “power” to Gherkin syntax
• A great way to communicate and cooperate between “the business people”, developers and QA.
• Written in (your) business/domain language
• Easy to both read and write
Feature: Simple search As a user I want to be able to do simple searches
eBay Inc. Confidential6
Cucumber -‐ more than a test tool
• Behaviour-‐Driven Development
• Adds “power” to Gherkin syntax
• A great way to communicate and cooperate between “the business people”, developers and QA.
• Written in (your) business/domain language
• Easy to both read and write
Feature: Simple search As a user I want to be able to do simple searches
Scenario: I can search without any parameters Given I am on search When I perform the search Then I see the result
eBay Inc. Confidential6
Cucumber -‐ more than a test tool
• Behaviour-‐Driven Development
• Adds “power” to Gherkin syntax
• A great way to communicate and cooperate between “the business people”, developers and QA.
• Written in (your) business/domain language
• Easy to both read and write
Feature: Simple search As a user I want to be able to do simple searches
Scenario: I can search without any parameters Given I am on search When I perform the search Then I see the result
Step-definition!!Then(/^I see the result$/) do! unless query("view marked:'SearchResultCell'").count > 0! screenshot_and_raise "No result was shown"! end!end!!
eBay Inc. Confidential7
Calabash
• Enables the use of Cucumber for Native apps
• iOS
• Android
!• The bits and pieces that interact with the app
• touch, scroll, swipe etc
• Query-‐syntax (iOS examples)
• query(“label marked:’label text’”)
• query(“button marked:’Ok’”)
eBay Inc. Confidential8
Calabash -‐ architecture
eBay Inc. Confidential9
Scenario implementation example
eBay Inc. Confidential
Scenario: Check that change classification clears matrixdata with warning! Given I am logged in as "Buyer"! And I am on the SYI hub! And I select classification "Hovedtelefoner"! And I set "Type" to “Nakkekontur”! And I set price to "250"!! When I press "Kategori"! Then I wait for "Du er ved at skifte kategori" to appear!! When I touch "Fortryd"! Then I see cells containing "Nakkekontur"!! When I press "Kategori"! Then I wait for "Du er ved at skifte kategori" to appear!! When I touch "Skift"! Then I see the SYI classification search!
9
Scenario implementation example
SYI = Sell your item
eBay Inc. Confidential
Scenario: Check that change classification clears matrixdata with warning! Given I am logged in as "Buyer"! And I am on the SYI hub! And I select classification "Hovedtelefoner"! And I set "Type" to “Nakkekontur”! And I set price to "250"!! When I press "Kategori"! Then I wait for "Du er ved at skifte kategori" to appear!! When I touch "Fortryd"! Then I see cells containing "Nakkekontur"!! When I press "Kategori"! Then I wait for "Du er ved at skifte kategori" to appear!! When I touch "Skift"! Then I see the SYI classification search!
9
Scenario: Check that change classification clears matrixdata with warning Given I am logged in as "Buyer" And I am on the SYI hub And I select classification "Hovedtelefoner" And I set matrixdata "Type" to “Nakkekontur" And I set price to "250"!! When I try to change classification Then I see an alert with title "Du er ved at skifte kategori" ! When I cancel the alert Then the classification is "Nakkekontur" ! When I try to change classification Then I see an alert with title "Du er ved at skifte kategori" ! When I accept the change classification alert Then I see the SYI classification search
Scenario implementation example
eBay Inc. Confidential10
Scenario implementation example
Step from “sell your item feature”!!And I set price to "250"
eBay Inc. Confidential10
Scenario implementation example
!And /^I set price to "(.*?)"$/ do |price|! macro 'I swipe up'! sleep(1)! touch("view marked:'Price'")! sleep(0.5)! set_text("view marked:'Price'", price)! sleep(0.5)! touch("view marked:'OK'")! sleep(1)!end
Step from “sell your item feature”!!And I set price to "250"
eBay Inc. Confidential10
Scenario implementation example
!And /^I set price to "(.*?)"$/ do |price|! macro 'I swipe up'! sleep(1)! touch("view marked:'Price'")! sleep(0.5)! set_text("view marked:'Price'", price)! sleep(0.5)! touch("view marked:'OK'")! sleep(1)!end
!And /^I set price to "(.*?)"$/ do |price| @page.write_price(price) end
Step from “sell your item feature”!!And I set price to "250"
eBay Inc. Confidential10
Scenario implementation example
!And /^I set price to "(.*?)"$/ do |price|! macro 'I swipe up'! sleep(1)! touch("view marked:'Price'")! sleep(0.5)! set_text("view marked:'Price'", price)! sleep(0.5)! touch("view marked:'OK'")! sleep(1)!end
!And /^I set price to "(.*?)"$/ do |price| @page.write_price(price) end
Step from “sell your item feature”!!And I set price to "250"
SellYourItemPage (page object) def write_price(price) scroll_and_wait_for_row_with_mark("priceCell") touch("view marked:'Price'") keyboard_enter_text price close_keyboard end
eBay Inc. Confidential11
PageObject pattern
Feature
Scenario
Step
eBay Inc. Confidential11
PageObject pattern
Feature
Scenario
Step
Page object
iPad iPhone Android
eBay Inc. Confidential11
PageObject pattern
Feature
Scenario
Step
Page object
iPad iPhone Android
eBay Inc. Confidential11
PageObject pattern
Feature
Scenario
Step
Page object
iPad iPhone Android
SyiHubPage (PageObject) !• go_to_picture_center !• go_to_classification_search !• go_to_value_selection_for_matrix
data(label) !• write_price(price)
eBay Inc. Confidential11
PageObject pattern
Feature
Scenario
Step
Page object
iPad iPhone Android
SyiHubPage (PageObject) !• go_to_picture_center !• go_to_classification_search !• go_to_value_selection_for_matrix
data(label) !• write_price(price)
eBay Inc. Confidential12
DEMO
eBay Inc. Confidential13
Continuous Integration
• Jenkins on a couple of Mac Mini’s
• Runs multiple builds/tests in parallel (Master/Slave Jenkins setup)
• The tests only runs in the simulator
• Unit tests —> Integration tests —> Accept/UI tests
• A subset of the Accept tests run after each commit
• Controlled by @tags in cucumber
• All Accept tests runs once every evening (or manually)
eBay Inc. Confidential14
The “control room”
eBay Inc. Confidential14
The “control room”
eBay Inc. Confidential15
Cucumber reports
eBay Inc. Confidential15
Cucumber reports
eBay Inc. Confidential16
When it fails…
eBay Inc. Confidential17
Ensure high quality tests
• Review and/or Pair program tests with QA
• (Only) use “Custom steps”
• Test code is (also) production code
• Let the quality reflect that!
• Refactor, Refactor, Refactor!
• Make sure the tests actually covers what is expected
• Do it with facts -‐ no guessing/assumptions
eBay Inc. Confidential18
Code coverage
• Do not (necessarily) focus on the coverage percentage
• Find inspiration to new test scenarios
• Ensure every feature in the app is well covered
eBay Inc. Confidential18
Code coverage
• Do not (necessarily) focus on the coverage percentage
• Find inspiration to new test scenarios
• Ensure every feature in the app is well covered
eBay Inc. Confidential18
Code coverage
• Do not (necessarily) focus on the coverage percentage
• Find inspiration to new test scenarios
• Ensure every feature in the app is well covered
eBay Inc. Confidential18
Code coverage
• Do not (necessarily) focus on the coverage percentage
• Find inspiration to new test scenarios
• Ensure every feature in the app is well covered
eBay Inc. Confidential19
Keys to success
• Team effort
• Definition of Done (DoD) includes automated tests
• Know that it costs time -‐ it’s not a silver bullet
• Maintenance of setup -‐ new versions of iOS & Calabash
• Fix broken builds ASAP!
eBay Inc. Confidential20
Future improvements (as seen in January)
Run a subset of tests in Xamarin test cloud a couple of times each week
Do more BDD/TDD
Automatic feature builds on Jenkins
Faster feedback cycle
Select a subset (with @tags) to run after each commit
Use “backdoor” to do e.g. login
Avoid resetting after each scenario
Make all tests run stable all the time!
eBay Inc. Confidential21
Xamarin Cloud
eBay Inc. Confidential21
Xamarin Cloud
• Currently only as proof of concept on DBA
• 5 scenarios on 3 devices
• Waiting for a proxy solution to take full advantage
• Create listings (and complete payments)
• BilBasen is also running in the cloud
• Trying out “Series” for iPad and iPhone and other cloud experiments to find out what works best for us
eBay Inc. Confidential22
Calabash backdoor
eBay Inc. Confidential22
Calabash backdoor
• Invoke logic in the app “behind the scenes”
• Reduce scenario test-‐time
• Avoid testing the same UI over and over in each scenario
• Stable
• Easy to use
eBay Inc. Confidential22
Calabash backdoor
• Invoke logic in the app “behind the scenes”
• Reduce scenario test-‐time
• Avoid testing the same UI over and over in each scenario
• Stable
• Easy to use
• We use it for
• Login (and create user)
• Create listings
• Clean up data
• (Code coverage)
eBay Inc. Confidential23
Unstable tests
eBay Inc. Confidential23
Unstable tests
• UI Automation / iOS Simulator often fails
• I thought it was Calabash being unstable :-‐)
• We would start tests over and over and over…
• …and overlook real failures
• It was almost impossible to get our “All scenarios” to run green
eBay Inc. Confidential24
Stable tests
• Cucumber Rerun formatter to the rescue
eBay Inc. Confidential24
Stable tests
• Cucumber Rerun formatter to the rescue
A simple script that will….
eBay Inc. Confidential24
Stable tests
• Cucumber Rerun formatter to the rescue
Run tests - use rerun formatter
A simple script that will….
eBay Inc. Confidential24
Stable tests
• Cucumber Rerun formatter to the rescue
Run tests - use rerun formatter
If rerun.txt contains tests - run them
A simple script that will….
eBay Inc. Confidential24
Stable tests
• Cucumber Rerun formatter to the rescue
Run tests - use rerun formatter
If rerun.txt contains tests - run them
Inform jenkins how it all went
A simple script that will….
eBay Inc. Confidential24
Stable tests
• Cucumber Rerun formatter to the rescue
Run tests - use rerun formatter
If rerun.txt contains tests - run them
Inform jenkins how it all went
A simple script that will….
Replace rerun result in original result
eBay Inc. Confidential24
Stable tests
• Cucumber Rerun formatter to the rescue
Run tests - use rerun formatter
If rerun.txt contains tests - run them
Inform jenkins how it all went
A simple script that will….
Replace rerun result in original result
eBay Inc. Confidential25
Wrap up!
• Test scenarios follows the code = the truth
• Enables rapid releases / fast time to market
• A necessity to deliver high quality
• Spend less time doing (boring) trivial regression tests
• Happy QA´er
• Brave developers -‐ higher code quality
• Quality product
• Happy users
eBay Inc. Confidential26
QUESTIONS?
eBay Inc. Confidential27
Ressourcer
• The Cucumber Book
• Calabash @github (https://github.com/calabash/)
• Calabash google groups • https://groups.google.com/forum/#!forum/calabash-‐ios
• https://groups.google.com/forum/#!forum/calabash-‐android
• Calabash overview (http://docs.xamarin.com/guides/cross-‐platform/test_cloud/calabash/intro_to_calabash/)
• Jenkins (http://jenkins-‐ci.org/) • plugins: Cucumber-‐reports, Cobertura (code coverage), Build monitor view
• RubyMine (http://www.jetbrains.com/ruby/)
• Uncle Bobs Clean Code
• because it’s a great book that promotes craftsmanship