How To Grade Your Selenium Tests

Let’s talk about tests

What Makes a Bad Test?

• Fails For No Good Reason

• Changes in AUT, Need to Update Multiple Tests

• Slow

• Unreliable

• Hard to Understand and Maintain

What Makes a Good Test?• Written for BDD or xUnit test framework

• Test one thing (atomic)

• Each test can be run independently (autonomous)

• Anyone can understand what it is doing

• Group similar tests together

• Centralized setup/teardown

• Uses Page Objects

A Testing Rubric (Each test starts with 100 points, deduct as necessary)

Item Belong in tests? Score

Selenium Commands No -3 per (max of -9)

Locators No -2 per (max of -8)

Selenium setup/teardown* No -20

Hard-coded sleeps No -5 per (max of -20)

Implicit wait calls No -10

Explicit Wait calls No -3

Conditionals No -5 (max of -20)

Calls to Page Objects Yes N/A

Assertion(s)* Yes N/A

Let’s talk about Page Objects

http://martinfowler.com/bliki/PageObject.html

What Makes a Bad PO?

• Gigantic in size (e.g., hundreds or thousands of lines in code)

• Contain overly complicated logic

• Return not enough or too much information to the test (leaky abstraction)

• Assertions happening in the PO instead of the test

What Makes a Good PO?• Contains State (e.g., locators)

• Contains Behavior (e.g., methods to interact with the page)

• Returns some information about the page (e.g., new page object, text from the page, a boolean result, etc. — never a WebElement)

• Verifies page ready state as part of initialization

A Page Object Rubric (Each Page Object starts with 100, deduct points as necessary)

Item Belong in POs? Score

> 200 lines of code No -5 per 50 lines over

Assertions No* -5 per (max of -20)

Hard-coded sleeps No -5 per (max of -20)

Implicit wait calls No -10

Explicit waits Yes N/A

Verify page ready state* Yes -8 if not verifying an element

Locators* Yes N/A

http://se.tips/se-waiting-jim-evans

Let’s talk about locators

Locator Strategies• Class

• CSS selectors

• ID

• Link Text

• Partial Link Text

• Tag Name

• XPath

Good locators are: • unique • descriptive • unlikely to change

That rules a few of these out

Locator Strategies• Class

• CSS selectors

• ID

• Link Text

• Partial Link Text

• Tag Name

• XPath

Good locators are: • unique • descriptive • unlikely to change

That rules a few of these out

Locator Strategies• Class

• CSS selectors

• ID

• Link Text

• Partial Link Text

• Tag Name

• XPath

Good locators are: • unique • descriptive • unlikely to change

That rules a few of these out

Start with IDs and Classes

Locator Strategies• Class

• CSS selectors

• ID

• Link Text

• Partial Link Text

• Tag Name

• XPath

Good locators are: • unique • descriptive • unlikely to change

That rules a few of these out

Start with IDs and Classes

Use CSS or XPath (with care)

Which is better CSS or XPath?

CSS vs. XPath Benchmarks

http://se.tips/se-benchmarks

A Locator Rubric (Each locator gets a grade)

Item Letter Grade E.g.,

Dynamic locators D- User account specific or tied to page render

Tied to page layout D XPath: / / / CSS: > > >, etc.

Text on the page C Link text, page copy

Reasonable traversal B Parent to child w/in an element node

Using semantic name B+ Input labels (name='username')

Semantic ID A Unique, descriptive, unlikely to change

Non-unique locator? -2 full letter grades

So…

What’s your testing GPA?Topic Weight

Tests 20%

Page Objects 30%

Locators 50%

http://seleniumconf.co.uk

http://se.tips/master-visual-testing