Integrating crowd testing into ci
Transcript of Integrating crowd testing into ci
QualiTest was recognized as the
only visionary pure play QA and testing Service Company in the “Gartner Magic
Quadrant”
2
Intro
3
Source: Nelson Hall Report, 2013
QualiTest is the world’s second largest testing
services specialist by revenue
Intro
Test Outsourcing Engagement Evolvement
4
Staff Augmentation
TCoE / Offshore
Managed Services
Crowd Testing
•Time & Materials
•Mainly in-house
contractors
•Cost deduction of
price per role
•Time & Materials
•Blended Rate
•Cost deduction of
average hourly
rate
•Time & Materials
and Fixed Price
•Responsibility
shitted to vendor
•Long Term
engagement
•Results Based
Testing: Service
Level Agreement /
KPIs
•Advanced pricing
models (pay per
bug/cycle)
•Test In The Wild
•Task based
•Results focus
Intro
• “Crowdsourcing is the process of obtaining needed services,ideas, or content by soliciting contributions from a large groupof people, and especially from an online community, ratherthan from traditional employees or suppliers”
• CrowdTesting is an emerging trend in software testing whichexploits the benefits, effectiveness, and efficiency ofcrowdsourcing and the Wisdom of the Crowd.
• It differs from traditional testing methods in that the testing iscarried out by a number of different testers from differentplaces, and not by hired consultants and professionals” Wikipedia
5
CrowdTesting is an approach
7
CrowdTesting
Access to over2.5 million testers
through over40 platforms
Top 5 reasons to use CrowdTesting
1. Access to specific talents
2. Ability to perform tasks quickly and efficiently
3. Test in the wild – different opinions, environments
4. The right incentive - reduce cost of xxx
5. It’s cool
8
Test Types1. Exploratory Testing – testing the entire functionality
2. Feature Testing – focus on specific feature/functionality
3. Localization Testing – focus on various languages
4. Compatibility – testing various Mobile devices
5. Usability – User Experience and Ease of use done by a mixture of real users and UI experts
6. Inter-operability Testing – running different applications to ensure proper performance
7. Configuration Testing – run device on different configuration settings
8. User specific testing – run testing on specific type of users (existing clients, designers etc.)
9. Test Data – create massive test data
10. Test Planning – convert design into test cases
11. Beta Testing – test the system using real users
9
Business Case
12
• Testers are usually paid by the bug
• Customers usually pay by cycle or subscription
• Pricing defined based on:
– Complexity of testing
– Amount of testers
– Crowd testers qualifications required
• ROI
– Cost of CrowdTesting vs. Traditional
– Cost of finding a defect
Moment of Involvement
• Before each release (Test Environment)
• After each release (Development Environment)
• Weekly basis – in parallel to internal testing
• After each sprint
13
CrowdTesting & DevOps
• DevOps is a software development method that stresses communication, collaboration, integration, automation and measurement…
• Testers becoming closer to the clients/end users and thus required to communicate with them
• CrowdTesting helps testers to get the real end-user experience and feedback
14
Example of CrowdTesting and Devops integration
• CrowdTesting cycles is performed after initial deployment of any new feature
• Companies with very frequent releases require on-going feedback from Crowd. Using Crowd as “human monitoring system”
• Communicate and get feedback from crowd before defining sprints backlog: Surveys, UI Review, Heuristic Evaluation
• Testing as a Service (TaaS):
15
1 hour
Target
• Reinforce the test team. Beta testing for specific SIM cards and locations
Achievements
• 10 private crowd testers
• Completed full cycle in South Africa within 3 days
16
Target
• Build Test Data used for regression test automation
• Repetitive work of tagging webpages and images
Achievements
• Categorize 157.000 URLs and images
• Completion within 3 weeks (compared to 6-12 months)
• Reduce the cost by over 80%
17
Target
• Testing web game in 52 countries and 23 languages
• Testing real life conditions
Achievements
• Full coverage delivered in 30 days
• Over 800 defects detected
18
Target
• Testing 20-30 mobile apps
• Running over 200 cycles each year
• Continuous integration SDLC (release each week)
• Functional, Usability and localization
Achievements
• Reduced cost per defect from 244£ to 48£
19