Crowd Sourced Testing - Real-World Experience Using Rainforest QA Crowd-Sourced Testing Platform

Solution Proposal 1

Crowd Sourced Testing

March 22, 2017

Marc Fasel

Technical Director

Real-World Experience Using Rainforest QA Crowd-Sourced Testing Platform


PerformanceCentre

2

•PerformanceCentre is a SaaS platform

•We calculate commissions for sales people

•Clients:

•Companies with large sales teams: 50-5000

•Commissions cannot be accurately calculated using Excel

•Example Suncorp, Optus, Bankwest …

•Headquarters: South Melbourne, team of 17

•Cater to the Australian- and US- market

•I am the technical director, lead an agile team of 4 developers, 1 tester


Problem: Our Regression is not Agile

3

•Agile development: 2 week sprints, deploy after each sprint

•Lots of functionality built over the last 10 years

•Web application with > 300 pages

•Regression test

•Before every deployment

•Written Regression test: 300 test cases, average10 test steps

•Execution time: 2 person weeks

•Not agile!


Test Automation

4

•High initial development effort; estimate 2 hours per test; Every time we added a new feature there was a new problem to overcome with Selenium

•High maintenance effort; many tests broken with each run

•Tests were flaky (timeouts etc)

•We could hire a single person only doing Selenium

•The tests never found anything (or it was drowned out by the false-positives)


The Search

5

•Add local test engineer

•Management Overhead

•Expensive

•Regression test time still too long: 1 week

•Outsourcing

•Recruit tester in the Philippines

•Testing is not on-demand: Gotta keep person busy 40hrs/wk

•Management overhead

•Regression test time still too long: 1 week


The Solution: Crowd-Sourced Testing

6

•Emerging trend in testing

•Gartner report “Market Guide for Crowdsourced Application Testing Services” Oct 2016:

•By 2018 20% of testing will be done using crowdsourcing

•Testing-as-a-Service

•Instead of using in-house test engineers for QA the testing is outsourced to a company

•Crowd: Large number of testers, parallel execution of tests

•Typically use case: web-based or mobile testing

•Web-based SaaS platform to manage the testing

•Testers can be qualified or unqualified

•Either virtual machines or use their own devices


Solution: Crowd Testing

7

•We ended up using Rainforest QA SaaS platform

•Written tests are entered into platform

•Run now button: select tests you want to run

•Tests are executed by human testers

•Testers get a virtual machine spun up with the correct OS and browser

•Tests run in parallel

•Time to run all tests is determined by the time to run the longest test

•On-demand

•Regression test time:1 hour


Platform View

8


Test Details

9

•Typical test consists of 5-10 steps

•Each step is an instruction followed by a Yes/No question

•Any qualified tester can take on the test

•Tester gets VM with correct OS and browser, and the step description shown.

•Tester executes the step and answers Yes/No question.


Individual Test

10


Test Step

11


Results

12


Individual Failure Result

13


Where do Testers come from?

14

•In our case they come from HIT market places

•Example: Amazon Mechanical Turk

•Human Intelligence Tasks (HIT): Micro-jobs that cannot be automated

•Marketplace: requestors and workers

•Platform is requestor, workers are the testers

•Each test becomes a HIT

•Workers have to earn qualifications and have reputation to be allowed to take test HITs

•Earn qualifications through training

•Earn reputation through successful HITs


Amazon Mechanical Turk

15


HIT Example

16


Cost

17

•Cost is per test step: US $0.66-$0.77 depending on pre-paid number of credits

•Example: US $0.70/ test step

•200 Tests * 10 Steps * 2 browsers * US $0.70 = US $2800 per regression test

•2 regression tests per month = US $5600

•Cheaper than an local Test Engineer, more expensive than offshore Tester


Advantages

18

•Test results in an hour: super agile!

•Maintenance is a breeze:

•1 hour to add new tests per sprint

•5 tests broken per cycle, and it takes <30mins to fix

•0 management overhead

•Easy testing against different browsers, mobile devices

•Test crowd works 24/7

•Issues easy to reproduce

•They actually found real bugs!


Disadvantages

19

•Cost

•Monthly cost US$4000 for 2 regression tests @1 hr

•You could have 200 failed tests: cost US$2000

•Tester Skill

•Testers do not know anything about your application

•The testers are not particularly skilled in testing

•They know how to use a web app

•The tests have to be simple to execute

•Testers sign NDA: how do you enforce that?

•Tests run in parallel

•The tester look only for the answer of each test step


Platform Choice

20

•Popular vendor Options

•Rainforest QA (www.rainforestqa.com)

•Applause

•Testbirds

•Lionbridge

•Mob4Hire

•Space is still very young, and vendors have different approaches

•Exploratory testing vs. functional/ regression testing

•Testers from HIT marketplaces vs. fixed test teams

•Some vendors even provide test automation for you


Conclusion

21

•Crowd Testing has been a great success for us

•Best of both worlds of test automation and manual testing

•Quick results

•No management overhead

•Easy scalability to new browsers and devices

•Well-maintained test suite

•Few false-positives

•Main drawback: Cost

•Agile QA

•Release the day after the sprint ends

•Regression-tested hot fixes in 4 hours

•Retain Small team

•Very young and dynamic space, lots of changes still happening


Contact Us

22

Measure Sales - Reward Success

11 Yarra Street, South Melbourne VIC

+61 3 8685 1200

[email protected]

linkedin.com/company/

www.performancecentre.c

Crowd Sourced Testing - Real-World Experience Using Rainforest QA Crowd-Sourced Testing Platform

Software

Transcript of Crowd Sourced Testing - Real-World Experience Using Rainforest QA Crowd-Sourced Testing Platform