2010 GTAC Crowd Source Testing Mozilla Style

Crowd Source Testing Mozilla Community Style

Matt EvansMozilla QA Director

Wednesday, November 10, 2010

about:talk

• Background and Scale of Mozilla

• Crowdsourcing: Crowd vs. Community

• History of Mozilla Community testing

• How Mozilla succeeds with Community testing


• I have been in the testing biz for most my career

• 9 mos as QA Director @ Mozilla

• Manage a team of ~30 Global Test Engineers

about:me


about:mozilla


about:mozilla

Mozilla’s Mission:

is to promote openness, innovation, and opportunity on the web


I’m still getting my head around scale at Mozilla

• The scale of information flow

• The scale of the user base


Mozilla Firehose


Mozilla Firehose• Many sources of information

• bugzilla (200+bugs/day)

• forums (50+)

• blogs (50+)

• tweets (100s/day)

• wikis (1000s pages)

• irc (50+ channels)


Mozilla Scale

400+Developers

10,000+Nightly Testers

2,500,000Beta Testers

400,000,000Users

400,000,000

350,000,000

300,000,000

250,000,000

200,000,000

150,000,000

100,000,000

50,000,0000Wednesday, November 10, 2010

Story Time


Big decision: Ship or No-Ship to Beta

A Recent Firefox product team meeting


Protector of the tester community


Get feedback from tester community


We were both wrong right!


It was a big change for my perspective


Respecter of the Tester Community as QA

Team Partner


We now work much better together


Crowd Sourcing: It involves a crowdworking towards a common goal


What is new, the virtual crowd is enabled by internet technologies


• Building a global community of testers ~30k+

• Stewardship of the test community is their secret sauce

Community approach to software testing


Mozilla Labs Crowdsource Crowdsourcing Project


Key Findings• there are two types users: heavy

users and casual users. • 90-9-1 Principle

• crowd motivation is separated into two forms: intrinsic and extrinsic

• crowdsourcing relies on people, and thus works when participants are treated as people.

Mozilla Labs Crowdsource Crowdsourcing Project


VS

Community Crowd


Mozilla uses a hybrid of crowd based feedback mechanisms and community

management

“Crowd sourced testing Mozilla Style”


The History of Mozilla Testing Community


Mozilla.org is launched

by Netscape

1998 2002

Mozilla 1.0 is released

Phoenix 0.1released

2004

Firefox 1.0released

2005-2009 2010

Firefox 1.5-3.5released

Firefox 4.0expected release

Pioneer Community Testers

Community Testers, The Next Generation

1 2 3


1998 2002 2004 2005-2009 2010



1 2 3

1998 Netscape open sourced the Netscape browser source code.


History of Mozilla’s Tester Community

•A lot of passionate people without a development background wanted to get involved.

Asa Dotzler First Mozilla Community Tester


The Rise of the Nightly Build




•A great success

•The community of testers grew to the 1000s



Too much of good thing



•As tester community members increased so did the noise


1998 2002 2004 2005-2009 2010



1 2 3

• 2002+ • The rise of Firefox code base• Split from AOL/Netscape• 2003 Mozilla Foundation is formed


1998 2002 2004 2005-2009 2010



2


1998 2002 2004 2005-2009 2010



1 2 3

• 2007+ • The rise of Continuous Integration

and unit tests


Continuous Integration

~3.3 million automated tests executed for every committed code change

•The largest CI system built on Buildbot and Tinderbox technologies

• For every code change

• 11 full builds of the browser across supported platforms

•~300k unit/perf tests per build


Minefield is the result of Continuous Integration

Continuous Integration and Unit Testing is the Foundation of Today’s Mozilla

Community Testing Success


Community Feedback


Mozilla Runs On Bugzilla


Crash Data Analysis


Crash Data Analysis5 years of crash data

FF1.5 FF2.0 FF3.0 FF3.5

FiledFixedDups


Firefox 4 Beta Feedback


Firefox 4 Beta Feedback

LCD Screens of Feedback and Twitter mentions are displayed throughout the Mozilla Office


Engaging the Mozilla Tester Community


Manual TestingLitmus - Test Case Management


Test and Bug Days


Mozmill Test Automation Framework

Community Focused Automated Testing

• Javascript framework for testing gecko based applications• Includes an IDE• Installs as an Firefox

extension• Better suited for

community contribution


Mozilla Community Based Automated Testing


Overview of How we succeed with Community Testing

• Continuous Integration is the foundation of today’s community testing

• Utilizing Community Feedback

• Engage in Manual Testing efforts

• Engage Community focused automated testing in the near future


Community Stewardship


Community Stewardshipquality.mozilla.org


Reaching Out to the Communityquality.mozilla.org

quality.mozilla.org


A call out to you!


Thank you

Questions?


2010 GTAC Crowd Source Testing Mozilla Style

Documents

Transcript of 2010 GTAC Crowd Source Testing Mozilla Style