Wix Automation - The False Positive Paradox
25
1 Wix Automation The False Positive Paradox Efrat Attas
Transcript of Wix Automation - The False Positive Paradox
1
Wix AutomationThe False Positive Paradox
Efrat Attas
2
What?!What is “False Positive”?
3
What?!Release cycle
• Daily / Weekly deployments of multiple products• QA verifies each version using
manual and automation tests (accompanied by automation engineers)
4
What?!
Test failure not necessarily a bugFalse positives – What is it?
5
What?! Understand!False positives – What are the consequences?• Time consuming (in case of many
failures)• Bottleneck
6
Why?!Why is it happening?
7
Why?
• Unstable test – “flaky”• Outdated test
False positives – Why is it happening?
8
Why? Invent!Reduce false positives to a minimum• Identify • Isolate• Analyze• Fix the flakiness
9
How?!How to reduce outdated tests?
10
How?!
Bonus! Can also be used for “real” issues in product – true positives
Solution• Ignore On Bug annotation
Outdated test• Ignore test on bug with a link to Jira
issue
Motivation• Outdated test must be adapted to
new behavior without causing unnecessary failures during the refactor process
11
Solution
@IgnoreOnBug(issue = ”SE-5541”)@Test public void verifyButtonText() { … }
@IgnoreOnBug
12
Solution@IgnoreOnBug
@Retention(RetentionPolicy.RUNTIME) @Target(ElementType.METHOD) public @interface IgnoreOnBug {
String issue();
}
13
Solution@IgnoreOnBug
public interface TestObserver {
default void starting(Description description) { } default void before(Description description) { } default void ending() { } default void skipped(AssumptionViolatedException e, Description description) { }default void succeeded(Description description) { } default void failed(Throwable e, Description description) { } default void finished(Description description) { }
}
14
Solution@IgnoreOnBug
@Override public void before(Description description) { if(hasAnnotation(description, IgnoreOnBug.class)) {
String issue = getJiraIssueFromMethodAnnotation(description); if(!isJiraIssueSolved(issue)) {
ignoreTest(); } } }
15
How?!How to reduce unstable tests?
16
How?!Unstable test
* What is stable enough?
• New tests are executed independently, without affecting existing test results
Motivation• New test has to prove it is stable
enough* before entering the entire test collection
Solution• Beta test mechanism
17
Solution@BetaTest
• Beta tests will run in a specific configuration, X times every night• Each test will report its run result to a dedicated server• Beta tests which passed more than Y times will be
“approved”
18
Solution@BetaTest
@BetaTest(owner = "someone") public void verifyButtonText() { … }
19
Solution@BetaTest
@Retention(RetentionPolicy.RUNTIME) @Target({ElementType.METHOD, ElementType.TYPE}) public @interface BetaTest {
String owner();
}
20
Solution@BetaTest
Collection<Class<? extends Annotation>> testMethodAnnotations = Arrays.asList(Test.class, BetaTest.class);
@Override protected List<FrameworkMethod> computeTestMethods() {
return getAnnotatedMethods(getTestClass(), testMethodAnnotations); }
21
Solution@BetaTest
public interface TestObserver {default void starting(Description description) { } default void before(Description description) { } default void ending() { } default void skipped(AssumptionViolatedException e, Description description) { }default void succeeded(Description description) { } default void failed(Throwable e, Description description) { } default void finished(Description description) { }
}
22
Solution@BetaTest
@Override public void before(Description description) {
if (shouldIgnoreBetaTest(description)) {
ignoreTest("Test in beta mode - ignored"); }}
23
Solution@BetaTest
@Override public void succeeded(Description description) {
sendStatus(description, TestStatus.TEST_PASSED); }
@Override public void failed(Throwable e, Description description) {
sendStatus(description, TestStatus.TEST_FAILED); }
24
Solution@BetaTest
25
Thank you!Questions?
