WORKS IN (X, Y, Z)Parallel Combination Testing With Selenium,JUnit, and Sauce Labs

David Ross Drake

Dynacron Group

• Surprisingly, not the evil corporation from that 80’s dystopian sci-fi movie

• Actually a Seattle consultancy that helps companies implement continuous delivery…• Integrated Testing• Custom Development• Big Data Strategies• Deployment Automation

04/11/2023 2

A Common Selenium Problem• Tests are slow• Lots of combinations need to be tested

• Browsers• Versions• Operating Systems• Languages• Skins

• Combination interfaces are very similar

04/11/2023 3

Parameter Description

04/11/2023 4

Browser

- Application- Type- Version- OS

Locale

- Language- Country

Brand

- Code- URL Component

Extant Solution: CSV

04/11/2023 5

• Simple for Manual Testers to Use

• Allows some nested structures (PSV in CSV…)

• Incapable of containing complicated data

• Hard to Read• Ugly

TYPESAFE CONFIGDemo

04/11/2023 6

But the Problem is Actually Worse…IC CP IN HI EX SB CW 6C PCR

US En X X X X X X X X

UK En X X X X X X

French X X X X X X X X

German X X X X X X

Japanese X X X

Chinese X X X X X X X

Korean X X X

Italian X X X X

Arabic X

Hebrew X

Portuguese X X

Spanish X X X X X X X X

Russian X

Turkish X

How Many Combinations?

• Browsers: A, B, C, D, E• Locales: 1, 2, 3, 4• Brands: X, Y, Z

Let S1 be the size of the largest parameter array, S2 be the next largest, etc…

04/11/2023 8

Each Once?

• Browsers: A, B, C, D, E• Locales: 1, 2, 3, 4• Brands: X, Y, Z

Number of combinations: S1

04/11/2023 9

1. (A, 1, X)2. (B, 2, Y)3. (C, 3, Z)4. (D, 4, X)5. (E, 1, Y)

All Pairs?

• Browsers: A, B, C, D, E• Locales: 1, 2, 3, 4• Brands: X, Y, Z

Number of combinations: (S1S2)

04/11/2023 10

1. (A, 1, X)2. (A, 2, Y)3. (A, 3, Z)4. (A, 4, X)5. (B, 1, Y)6. (B, 2, Z)7. (B, 3, X)8. (B, 4, Y)9. (C, 1, Z)10. (C, 2, X)

11. (C, 3, Y)12. (C, 4, Z)13. (D, 1, X)14. (D, 2, Y)15. (D, 3, Z)16. (D, 4, X)17. (E, 1, Y)18. (E, 2, Z)19. (E, 3, X)20. (E, 4, Y)

All?

• Browsers: A, B, C, D, E• Locales: 1, 2, 3, 4• Brands: X, Y, Z

Number of combinations: (S1S2S3)

04/11/2023 11

1. (A, 1, X)2. (A, 1, Y)3. (A, 1, Z)4. (A, 2, X)5. (A, 2, Y)6. (A, 2, Z)7. (A, 3, X)8. (A, 3, Y)9. (A, 3, Z)10. (A, 4, X)

11. (A, 4, Y)12. (A, 4, Z)13. (B, 1, X)14. (B, 1, Y)15. (B, 1, Z)16. (B, 2, X)17. (B, 2, Y)18. (B, 2, Z)19. (B, 3, X)20. …

Risk vs Rewards

Riskx = probability a given pair has a failure only found when togetherR = probability that “each-once” fails to reveal a defect that could be seen in “all-pairs”

R = (1 - x) ^ (S1S2 – S1)

04/11/2023 12

Rewardy = average time it takes to test a combinationP = Fraction of testing time taken testing “each-once” as compared to “all-pairs”

M = S1 / S1S2

Converting Combinations into Tests

04/11/2023 13

Read Parameters• Add overrides for class, profiles, and system properties• Read as TypeSafe Config objects

Form list of config combinations• Apply combination strategy from config

Convert configurations back into json• Simple in TypeSafe Config – prevents writing custom

object mapping

Deserialize json combination in ParameterCombination object• Uses Jackson and @FromJson annotations as necessary

Parallelization

• Easy to Implement• See Sauce Labs blogs!

• Very hard to do right• See demo

04/11/2023 14

Class Browser Reuse

b = time to start and close browsert = average time to run a test methodm = average number of methods in a classF = fraction of time spent testing when using one browser per class instead of per method

F = (b/m + t) / (b + t)

04/11/2023 15

Class Browser Reuse (small tests)

15 = time to start and close browser6 = average time to run a test method5 = average number of methods in a classF = fraction of time spent testing when using one browser per class instead of per method

F = (15/5 + 6) / (15 + 6) = 3/7 = 42.9%

04/11/2023 16

Class Browser Reuse (long tests)

15 = time to start and close browser60 = average time to run a test method5 = average number of methods in a classF = fraction of time spent testing when using one browser per class instead of per method

F = (15/5 + 60) / (15 + 60) = 21/25 = 84%

04/11/2023 17

Tracking Parameterized and Parallelized Tests

• Descriptive method naming• Descriptive logging• Demo!

04/11/2023 18

Future Work

• Remembering Tested Combinations in Coverage Reports

• Effectively displaying tested combinations• Converting to using Junit 4.11 style parameterized runner.

04/11/2023 19

Questions?

• Parallel Webtest: https://github.com/dynacron-group/parallel-webtest

• TypeSafe Config: https://github.com/typesafehub/config

• Sauce post that provided the seed: http://sauceio.com/index.php/2010/10/parallel-junit-4-and-selenium-part-three-parallelism-and-ondemand/

04/11/2023 20