Universit degli Studi dellAquila Henry Muccini DISIM, University of LAquila henry.muccini@univaq.it, @muccinihenry, www.henrymuccini.com @AST 2014, Hyderabad, India June 01, 2014

Researcher at the University of LAquila, Italy Research 2

How did I get to Crowd Testing? 3 @AST 2012 @MobileSoft2014

Acknowledgment and Disclaimer This work is conducted with the collaboration of Fabio Guaiani (University of LAquila) and the CLARITER company (Rome). This presentation represents my group initial effort towards the understanding of Software Engineering needs and opportunities in Crowd Testing. 4

Agenda 5 Software Testing Trends Crowd Testing Software Engineering for Crowd Testing

PREFACE: SOFTWARE TESTING TRENDS 6 Software Testing Trends Crowd Testing Software Engineering for Crowd Testing

Software Testing Evolution 7 Software Testing No dedicated testing team Dedicated (in-house) testing team Outsourced testing (in-house) resources resources in the cloud Software development

Investment in QA 8 [World Quality Report 2013-2014] [1191 respondents] Right first time apps > $50 billion/year on applications testing and quality assurance [Top six trends to drive market for software testing in 2012, TechJournal, citing Pierre Audoin Consultants, Nov 2011] 18% 23% 28% 0% 5% 10% 15% 20% 25% 30% 1 IT budget allocated to QA 2015 2013 2012

9 Software Testing No dedicated testing team Dedicated (in-house) testing team Outsourced testing [1500 respondents] 0% 20% 40% 60% 51% 41% Internal/In-house Testing 2013 2012 [1500 respondents] 0% 10% 20% 13% 20% External/Managed Services 2013 2012 0% 20% 40% 8% 26% "Centralized" Testing 2013 2012 [World Quality Report 2013-2014]

10 Software Testing No dedicated testing team Dedicated (in-house) testing team Outsourced testing [1500 respondents] 0% 20% 40% 60% 51% 41% Internal/In-house Testing 2013 2012 [1500 respondents] 0% 10% 20% 13% 20% Crowd Sourced Testing (External/Managed Services) 2013 2012 0% 20% 40% 8% 26% "Centralized" Testing 2013 2012 [World Quality Report 2013-2014] IDC and Forrester indicate an increase in outsourcing of 15% annually over the next several years. [Forrester]

Testing Mobile App 11 Taken from [World Quality Report 2013-2014], page 27

Testing Mobile App 12 Taken from [World Quality Report 2013-2014], page 27 Must be right first time Even for free apps Must get into the market first

Summing up CROWD TESTING QA needs Mobile and Web Outsourci ng

CROWD TESTING 14 Software Testing Trends Crowd Testing Software Engineering for Crowd Testing

(the Crowd) [Oxford Dictionaries] The mass or multitude of ordinary people

Crowdsourcing [CrowdSource] The word is a combination of the words 'crowd' and 'outsourcing'. Work Work Work Work Work Work Work Work Crowdsourcing is the process of getting work or funding, usually online, from a crowd of people.

Crowd TestingGlossary (to be refined) 17 Crowd Crowd Sourcing Outsourcing Crowd Testing

What is Crowd Testing? It is, therefore, a form of outsourcing testing tasks to a mass of internet users (the crowd). [Stein_TestExp14] If you are testing software that all kinds of strangers are going to use, then why not use a bunch of strangers to test it. (Paul Herzlich) Crowtesting puts software to test under real-world conditions by real users, allowing companies to gather real insight and feedback and to identify defects quickly and cheaply [CSTE2013] 18 HM5 HM6

Diapositiva 18 HM5 Its called the power of the people. I asked thousands of friends to spend 5 minutes testing rather than requiring five of our employees to spend thousands of hours testing. It got us quicker results on more devices than we typically consider. [Mike Sparks article] Henry Muccini; 30/05/2014 HM6 It consists in asking 1000s of crowd testers to spend 5 minutes testing rather than requiring 5 employees to spend 1000s of hours testing. Henry Muccini; 30/05/2014

Examples 19 Tested by a crowd of 9 million people 2 million valuable comments and insights [Stein_TestExp14] GAP had to scrap their logo redesign after protests on Facebook and Twitter [www.pass.ch]

20 diversity/coverage/realistic independent and test as a user payment system immediate results Principles

21 Crowd Testing

The players and the process 22 Crowd Testers Crowd Testing Company Customers

Who are the crowd testers? [RV_TestExp14] The testing crowd can be a pool of: Inter-company testers Intra/inter-project testers in a company Strategic partners Domain knowledge experts Technology experts Tool experts Network of end users Social network users 23 Certified Specialist Recreational

Crowd Testers expertise and infrastructure Crowd Testing types 24 Crowd Testing Exploratory Crowd Testing Enterprise Crowd Testing

Exploratory Crowd Testing + Effective to cover all the many configurations + Good for non critical systems + Simulating real load and testing - Outcome not predictable - Discovering basic failures To help making it more effective: Show them the big picture of what they are doing Formal process A clear report 25

Enterprise cloud infrastruct ure Enterprise Crowd Testing 26 Large enterprises utilizing their in-house excess capacity for on-demand crowdsourced testing Secured private cloud infrastructure Testing Managers Organized Crowd testers screening Project management, tracking, effort planning Professionalized Standardized Process and ReportingStandardized

Enterprise (Managed) Crowd Testing Enterprise crowd testing, although still in its nascent and emerging stage, promises to be the next wave in the social, mobility and localization era. [28+_TestExp14] 27 Crowdtesting Platform Test Scheduling Testers Profiles Knowle dge center Comm. and collaboration Test Managem ent Reporting

A typical Process 28 Preparation Initiation Execution Evaluation Completion Identify customers needs Fix price Define report template Select crowdtesters Prepare the test environment Initiate testing Supervise and monitor Validate results Prepare client deliverables Testers payment Presents results to client Crowdtesters ratings [TestExp14]

Crowd TestingGlossary 29 Crowd Crowd Sourcing Outsourcing Crowd Testing Exploratory Crowd T. Managed Crowd T. Beta Testing Enterprise Crowd T.

30

- 1.5 months of full-time job for one mobile developer novice + 0.5 months for an expert mobile developer (client side) - 13.000 products in the DB - 400 MB of storage into the device - DB update 31

- The annulla feature (when the button is visualized during the data download) does not work on IOs - In case the product image is missing, a default image needs to be loaded - The loading % screen is not fluid - On IOs, if I download a product with an already existing image, I get an error - In the product list, when tapping on a product imagine, while looking at the magnified image, I can still scroll the page (shall be disabled). 32

SOFTWARE ENGINEERING FOR CROWD TESTING 33 Software Testing Trends Crowd Testing Software Engineering for Crowd Testing

Benefits 34 Availability Coverage Cost- effectiveness Impartiality Real scenarios Speed Diversity

Most of the reports come from CrowdTesting companies, so there could be a BIAS!! 35

Still a number of big companies working on it [BenchmarkQA] 36 are 55.000 testers better than 1.600 ? What types of testing can be run with the crowd? Big companies!

Data from Utest (now, Applause) 37 Randy Perry. Trulia Leverages uTest's "Crowdsourced" Testing Solution to Speed Deployment of High-Quality Mobile Apps. Sponsored by uTest. January 2013

Goldman Sachs Bets Millions On uTest [http://www.forbes.com/sites/petercohan/2014/01/22/goldman-sachs-bets-millions-on-utest/] 38

Challenges in Crowd Testing 39 Security breaches Duplicate defect management Measurement Coverage critical features coverage No control Lack of ownership Reward mechanism Communication and mgmt. Lack of standards Continuous involvement

Opportunities for SE research in crowd testing: COMMUNITIES Crowdsourcing in SE Global Software Engineering Social Software Engineering Empirical studies Software Testing 40

1. Opportunities for SE research Effectiveness of different testing methods Whenshall we test with the crowd, when not to? 41 Crowd Testing Cloud Testing Lab Testing Which sw. domain Which type of tests Ongoing Work

2. Opportunities for SE research Complementing different testing methods 42 Crowd Testing Cloud Testing Lab Testing How crowd and lab testing can be used together? Ongoing Work Need: clients want to get the best from the combination of both Efficiency

2. Complementing different testing methods (cont.) 43 Systematic Mobile Testing Crowd Testing Solution to ?

2. Complementing different testing methods (cont.) 44 Systematic Structural Testing Crowd Testing Solution to ? Coverage Issues Test Selection Test Execution

2. Complementing different testing methods (cont.) 45 Model- based Mobml Testing Crowd Testing Model- based Mobile

Principle 46 Device independent tests (in LAB) + device dependent in crowd Test HTML 5 PhoneGap (in LAB) + test Phone Gap app in Android, iOS, Microsoft Test MobML mobile (in LAB) + test Phone Gap app in Android, iOS, Microsoft Impl. 1 Impl. With MBT 2. Complementing different testing methods (cont.)

3. Opportunities for SE research Engineering the CrowdTesting Platform 47 Crowdtesting Platform Test Scheduling Testers Profiles Knowle dge center Comm. and collaboration Test Managem ent Reporting

4. Opportunities for SE research Crowd Coverage Criteria 48 Measurement Coverage critical features coverage No control System Coverage Crowd Coverage Metrics

5. Opportunities for SE research Social Software Engineering How to keep Crowd Testers motivated How to keep them involved Appropriate reward mechanisms 49

In summary 50 Crowd Testing is growing exponentially Crowd Testing may bring big benefits to QA Still, there a number of challenges to be managed

If you are interested to this topic, please 51 Stop by after the presentation Contact me at henry.muccini@univaq.it Tweet @muccinihenry Skype me at henry.muccini Call me H. Muccini Is Crowd Testing (relevant) for Software Engineers? @AST20114

References Testing Experience, 25/2014 issue on Crowd testing [TestExp14] Testing Experience, 25/2014 issue on Crowd testing [RV_TestExp14] Rama Murari & Venkat Ramesh Atigadda, Panorama of Crowd Testing, in Testing Experience, 25/2014 [Stein_TestExp14] Markus Steinhauser Crowd Testing An Addition to Traditional Software Testing, in Testing Experience, 25/2014 [34+_TestExp14] Dieter Speidel, Mayank Mittal & Mithun Sridharan Enterprise Crowd Testing, in Testing Experience, 25/2014, page 34+ [28+_TestExp14] Saisubramanian Sivasailem & Dilipkarthik Jayamohan, The Next Wave in the Social, Mobile and Localization Era - Enterprise Crowd Testing, in Testing Experience, 25/2014, page 28+ 52

References [BenchmarkQA] http://www.benchmarkqa.com/ [CSTE2013] Crowdsourced Software Testing for Enterprises, August 2013 (by passbrains.com) [World Quality Report 2013-2014] By Capgemini, Sogeti, Hp, 5th edition Crowdsourced Software Testing for Enterprises, August 2013 (passbrain) Dagstuhl Seminar 13362, Cloud-based Software Crowdsourcing, Edited by Michael N. Huhns, Wei Li, and Wei-Tek Tsai 53