Viktor Slavchev &The Testing trollSiteground

The “worst” practices of software testing

About me

• Senior QA @ Siteground Hosting• Wooden philosopher at Pragmatic

(part time)• Blogger @ mrslavchev.com(long

time ago in a galaxy far far away)• True believer that testing is a

human centric activity

• A mythical testing creature• Graduated the technical

university of Trolandia• Certified as SHGMCNGTT -

Super Hyper Giga Mega Certified Ninja Guru Testing troll

• Book author

About The testing troll

And the book:

• Absolute bestseller• Sold more copies than “50

shades of gray” in Trolandia• “Absolutely mind-blowing”

New York Times• “The troll is strong with this

one”Darth Vader

Best practice#1 - Rely on requirements and documentation

Worst practice #1: Learn about the real testing oracles instead

Testing oracles can benefit from:

Feelings and mental models

• It starts as a feeling• Not sufficient for a

bug report

Artifacts: documents, designs, requirements, etc.

Experience in:

• The domain• General testing• Software industry

Conference

• Developers• Other testers• Clients• Anyone who

matters

Example 1: Reference oracle

• External application• Other platform• Correctness presumed

Example 2: Self-verifying data as an oracle

Desired response: Credit card #:

000: Approved 4470330769941000

010: Partially Approved 4658512425423010

110: Insufficient Funds 4488282659650110

Example 3: Constraints oracle

Example: VAT number

BG123456789• Length• Country index• Spaces

Source:The Oracle Problem and the Teaching of Software Testing

Best practice#2 - Do regression testing by executing your old tests

Regression testing != repetitive testing

Repetitive testing is the reason why “others” see testing like this...

Worst practice #2: Regression checks and focus on tests that reveal new information

Regress - verb• to move backward; go back.• to revert to an earlier or less advanced

state or form

Regression bug:• Bug causing regress in

quality

Regression testing:• Testing to expose

regression bugs

Regression check:• Building part of a test• Definite answer• Small and easy to

manage

Focus on information

Best practice#3 - Automationion testing is the best, “manual” testing sucks

Quote from J. Kohl - Tap into mobile application testing

Ideas that will produce bug-free software:• C++ STL• OOP• Agile• TDD• Test automation

Quote from J. Kohl - Tap into mobile application testing

“Each of these were also used to tell me that testers would no longer be needed on software projects. So far, I am fifteen years in and I haven’t seen anything bug free yet.”

Can a testing tool produce better testers? Answer:

How to make efficient automation that provides value to testing?The answer is:

Worst practice #3: Use automation as a tool

• To extend your abilities • Not to replace you

completely• To ask the right

questions

Automation is great for:

• Confirmation / Verification

• Low level checking• Precision• Functional correctness• Solving machine

problems

But automation sucks when dealing with:• Exploration• Unanticipated risks• Qualitative evaluation• Open ended questions• Solving human

problemsSource:Things Could Get Worse- Michael Bolton

Source: The non-manual, unautomated tester

Best practice#4 - Assure quality

Quality is not a resourceQuality != testing

Worst practice #4: Provide information about risks

• Risk based testing - focus testing to explore potential risks

“Nightmare newspaper headings”

• You missed a critical bug in production

• It caused your company a huge financial loss

• What would it be?

• Pre-mortem - imagining a failure and working backwards in order to determine the possible risks.

Benefits:• Evaluation of risks• Business perspective• Prioritization

Best practice#5 - Follow best practices

Worst practice #5: Be always alert and aware of your context• What would be your

approach in 3 different contexts?

Context #1: Weather app

• You’re an outsourced testing expert

• Working for a start-up• Low on budget• Behind schedule

Context #2: Medical software

• High precision is a must• Human lives might

depend on it• Regulatory compliance

of testing

Context #3: Court case

• Bug caused an injury• You are independant

expert• On a case for negligence

in testing

Questions:

• How exhaustive will be your testing in each context?

• Will bug-hunting be your highest priority?

• Will you have the same strategy of testing?

Conclusion

• No best practices!• “We are not into the

belief business” J.Bach• In order to be a good

tester, you have to be a professional sceptic

Thank you very much!

… And don’t forget to Rock ‘n Troll

Image credits:

Too lazy to post them all. Image credits go to the almighty internet, I don’t own any of the images included in this presentation, except the ones created by me, linked to my blog.

STIONS?

ANY