“Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to...
Transcript of “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to...
![Page 1: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/1.jpg)
Today’s Testing Innovations
Lee Copeland,Software Quality Engineering,
USA
Europe’s Premier Software Testing EventStockholmsmässan, Sweden
WWW.EUROSTARCONFERENCES.COM
“Testing For Real, Testing For Now”
![Page 3: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/3.jpg)
What Is An Innovation?
• in·no·va·tion (ĭn'ə-vā'shən)
1. Something new or different
2. Something newly introduced
or adopted
3. A creation (a new device or
process) resulting from study and
experimentation
![Page 4: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/4.jpg)
Innovations In Software
Testing
• Context-driven school
• Test-first development
• Really good books
• Open source tools
• Testing workshops
• Freedom of the press
• Virtualization
• Testing in the cloud
• Crowdsourced testing
![Page 5: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/5.jpg)
Context-Driven School
• Testing groups exist to provide
testing-related services. They do
not run the development project;
they serve
• Entirely different testing strategies could be
appropriate for these different objectives
• It is entirely proper for different test groups to have
different missions. A core practice in the service of one
mission might be irrelevant or counter-productive in
the service of another
• Metrics that are not valid are dangerous
![Page 6: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/6.jpg)
Test-First Development
Then,
TDD = TFD + Refactoring
Developers “refuse to add
even a single line of code
until a test exists for it.”
- Scott Ambler
![Page 7: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/7.jpg)
Test-First Development
• Goals
− Accurate specifications
− Useful specifications
− Focus the mind
− Write clean code
− Prevent the creation of an “inventory” of things
that don’t work
![Page 8: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/8.jpg)
Really Good Books
• A Practitioner’s Guide to Software
Test Design ─ Lee Copeland
• Secrets of a Buccaneer-Scholar
─ James Bach
![Page 9: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/9.jpg)
Other Really Good Books
• Systematic Software Testing ─ Craig and Jaskiel
• Lessons Learned in Software Testing ─ Kaner,
Bach, and Pettichord
• Testing Computer Software ─ Kaner
• How to Break Software ─ Whittaker
• The Art of Software Testing ─ Myers
• The Craft of Software Testing ─ Marick
• Software Testing Techniques ─ Beizer
• Managing the Testing Process ─ Black
![Page 10: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/10.jpg)
Open Source Tools
Development
Eclipse
Unit Testing
xUnit
Cobertura
NCover
System Testing
FIT
FitNesse
Watir
Selenium
Performance Testing
OpenSTA
JMeter
![Page 11: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/11.jpg)
Testing Workshops
• Specialized focus and Participatory style
− Attendance by application
and/or invitation
− Limited to 15-20 participants
− Focus on a small set of topics
− Participation is required; challenging presenters
is encouraged
− Learnings are collected, published, and available
to all participants
![Page 12: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/12.jpg)
Testing Workshops
• Examples are:
− Austin Workshop on Test Automation
− London Exploratory Workshop in Testing
− Los Altos Workshop on Software Testing
− Workshop on Heuristic and Exploratory
Techniques
− Workshop on Open Certification of software
Testers
− Workshop On Performance and Reliability
− Workshop on Teaching Software Testing
![Page 13: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/13.jpg)
Freedom of the Press
• “Freedom of the press is limited to those who
own one.”
─ A.J. Liebling
(American Journalist)
• Today’s “press”(and lots of people own these)
![Page 14: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/14.jpg)
Freedom of the Press
• Blogs
– James Bach www.satisfice.com/blog
– Michael Hunter blogs.msdn.com/michael
– Jonathan Kohl www.kohl.ca/blog
– Antony Marcano www.testingreflections.com
– Brian Marick www.exampler.com/blog
– Elisabeth Hendrickson
www.testobsessed.com/category/ruminations
– others …
![Page 15: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/15.jpg)
Freedom of the Press
• Open Source Training
– Black Box Software Testing (Kaner & Bach)
– www.satisfice.com/moodle
![Page 16: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/16.jpg)
Virtualization
• Definition
• Vendors include Microsoft, Surgient, VMLogix,
and VMWare
![Page 17: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/17.jpg)
Virtualization
• Benefits
− Rapid provisioning (30%-50% of testing is
environmental setup time)
− System state capture (capture/replay in its finest
form)
− Reduced costs (purchase, maintenance, space,
power)
![Page 18: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/18.jpg)
Testing in the Cloud
• Rent as many servers as you
like for as long as you like
– Need a thousand servers for
performance load generation?
– Need multiple servers, each
with a different configuration
to simultaneously test your
product?
![Page 19: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/19.jpg)
Testing in the Cloud
• Usage rates:
– $0.10/hour – small machine
– $1.00/hour – extra large machine
![Page 20: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/20.jpg)
Crowdsourced Testing
• If virtualization makes sense for hardware,
why not for test teams?
• Crowd testing is using a virtual
test team rather than, or in
addition to, an organization’s
dedicated team
• The organization defines a set of test
requirements (types of testing, scenarios,
environments, platforms, etc.)
![Page 21: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/21.jpg)
Crowdsourced Testing
• A crowd test vendor (utest.com,
mob4hire.com, …) identifies a pool of
qualified testers, creates a test project,
assigns the work to testers, monitors the
progress, and reports defects
• The crowd test vendor compensates testers
and evaluates their performance
• Testers receive “Pay Per Bug” and also “Pay
Per Valuable Feedback”
![Page 22: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/22.jpg)
Innovations In Software
Testing
• Context-driven school
• Test-first development
• Really good books
• Open source tools
• Testing workshops
• Freedom of the press
• Virtualization
• Testing in the cloud
• Crowdsourced testing
![Page 23: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/23.jpg)
Future Innovations?
?
![Page 24: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/24.jpg)
Future Innovations?
• My personal event horizon is very short
circa 1969
![Page 25: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/25.jpg)
Articles On Innovation
• “Two Keys to Innovation”
• “Three Overlooked Keys to Innovation”
• “Four Keys to Innovation”
• “Six Keys to Building New Markets Through
Innovation”
• “Seven Keys to Cultivating Magic and
Innovation”
• “Nine Keys to Innovation”
• “40 Principles: Keys to Technical Innovation”
![Page 26: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/26.jpg)
Lee’s Keys To Innovation
• Creative
• Talented
• Fearless
• Visionary
• Empowered
• Passionate
• Multiple disciplines
![Page 27: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/27.jpg)
Future Innovations?
•
•
•
•
•
•
•
•
![Page 28: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/28.jpg)
Thanks
• Thanks for joining with me today. I greatly
appreciate the invitation to speak
• If I can be of assistance, or if you’d just like to
chat, please contact me at
• And remember, … keep on innovating
My thanks to:James Bach, Jon Bach, Rex Black, Bob Galen,
Andy Glover, Dorothy Graham, Cem Kaner,
Harry Robinson, Rob Sabourin, and James
Whittaker
![Page 29: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/29.jpg)
Your Innovation Quota
Print the following list, and take it
into your next meeting. Score one
point each time a phrase is used,
plus bonus points as indicated.
Score more than 5, and you've got an
organization that is innovation-averse. Score
10 or more, and you are innovation-dead.
Score 15 or more, and you might as well close
up shop
- www.jimcarroll.com
![Page 30: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/30.jpg)
Your Innovation Quota
• “It won't work”
• “That's the dumbest thing I ever heard”
• “That's not my problem”
• “You can't do that”
• “I don't know how”
• “I don't think I can”
• “I didn't know that”
• “We've always done it this way” (3 bonus points)
• “The boss won't go for it” (5 bonus points)
• “Why should I care?” (10 bonus points)
![Page 31: “Testing For Real, Testing For Now” Today’s Testing ...€¦ · • Testing groups exist to provide testing-related services. They do not run the development project; they serve](https://reader031.fdocuments.in/reader031/viewer/2022041100/5ed7153f62136e72fb7bcd8f/html5/thumbnails/31.jpg)