Appium Better PracticesLessons learned the hard way

Matt IsaacsGilt

What is Gilt

Short story

Luxe brands up to 60% off

Inventory changes daily. - Sales last 36 hours.

Sales start at noon ET. - Bonus sales at 9PM, Sundays and Wednesdays.

Gilt Mobile

Goals

Understand how users use their devices.

Understand the capabilities of each device.

Understand our users and how the want to interact with Gilt.

Put these all together to create experiences that surprise and delight our users.

A brief tour

How we roll

By hand, with rigorous QA.

Engineers focus on app features + performance + fixes.

QA crew in Jordan takes over at the end of the day.

Repeat.

Seems to be working

"Perfect App !!! Well done"

"Awesome app! Love it!"

"OMG! Amazing!"

"Your app is great and keeps getting better."

"I LOOOOOOOVE YOU GUYS AND YOUR APP!"

Or is it..

"Since your update this app runs slower than ever"

"How do you get rid of this. It is like a incurable disease."

"Never using again. Sister and mom and friends have removed also. Horrible"

Some issues aren't Gilt Mobile's fault. But still..

It gets worse

Mobile First. So hot right now

Revenue from Mobile at ~40% and growing.

Future of Gilt is in our hands.

Increasingly involved with other teams at Gilt.

Negative experience becomes more costly.

Informal Process

Doesn't scale!

Leads to questions: - Who's doing the build tonight? - What's in this build? - Which build must QA test?

Unit tests are run at the discretion of today's hand roller.

Surprise! Tests never get run.

What does it mean?

Potential for wasted QA time is high.

Expensive resource being used improperly.

Tough for outsiders to work with Gilt Mobile.

More shocking: We're not the only ones.

How can this be?

Tools have not kept up.

Apple tools lean in favour of the IDE.

xcodebuild test issues.

UIAutomation is part of intsruments.

XCode Bots is nice, but not quite ready.

Our way is easier.

Time for change

Road to automating

Change gradually.

CI - Nightly builds with Jenkins.

Internal distribution. Dogfooding!

Dust off old unit testing efforts.

UI Testing

What's the big deal with UI tests

Special time saving powers for QA.

Sanity tests are time consuming.

STR can also be time consumig.

UI tests are QA enablers.

Appium

Why?

Philosophy works for us.

Test the app you release

Language/Framework independent.

Uses popular standard API - Selenium WebDriver.

Thriving open source community.

Support for multiple mobile platforms.

Sounds great but...

Sometimes tests work. Sometimes they don't.

Reliability issues widespread enough to prompt development of Flaky tool.

UIAutomation is the root of a lot of the chaos. - "Instruments unexpectedly quit" - Random delays when searching tableviews.

Sometimes Apple breaks it altogether - Xcode 5.0.1.

Expect some misery on iOS + XCode major releases.

One test at a time. Sauce has you covered.

The setup at Gilt

Basic set of sanity tests. - New user registration. - Existing user sign on. - Product purchase flow. - Product Waitlist flow.

Scala + ScalaTest. Allows us to leverage existing Selenium infrastructure. Why reinventthe wheel?

Still a little too brittle.

Some lessons learned.

Accessibility

Not just good for your user base.

UIAutomation leverages accessibility to drive your app.

Accessibility = Testability.

Setting AccessibilityLabel and isAccessibilityElement only takes you so far.

Container views are troublesome. If marked as accessible, their contents will bemasked. UIAccessibilityContainer has you covered.

The special case

Table cells are containers and so need the aforementioned special care.

UIAccessibilityContainer might not be necessary.

Set the accessibility label of the cell, unless you like surprises.

Product Detail Page Example

PDP Navbar title AccessibilityLabel = Product Name (which is what it should be)

How can a test identify this page?

Dynamic Accessibility labels? UIAccessibilityIdentification is for you.

"Using an identifier allows you to avoid inappropriately setting or accessing anelement’s accessibility label."

Complex actions.

What if we want to search for unavailable products?

For each sale: - Wait for the product list and inventory to load, - Search elements for "Sold Out" or "Reserved" - Do some string processing to determine which of the products for a given cell isunavailable.

Or we could just use our API.

Complex navigations and searches with many moving parts can be susceptible toreliability issues.

Patterns

Action - Locator has a couple issues that are more promininent on mobile. - No state. What page am I on? - No convetion: When do I check page? Before or after? - Found ourselves fine tuning delays.

Page Objects formalize some conventions on top of Actions and Locators. - All actions return page objects - No assertions in page object actions.

Never reuse page objects. Element IDs sometimes change.

Some page actions dont always return the same page. Use page source to identifypage.

Since we dont have the luxury of navigating by URL. - Restart the driver - Recovery calls

Final nugget of sage wisdom

Be wary of "clever" UI hacks. Expect testing + accessibility misery.

Pass throughs and overlays: - Underlying views may be visible by you, but can't be acted upon because they are"hidden". - Tests become an error prone jumble of "mobile: tap" calls.

Appium needs you

Use it.

File those radar issues for Instruments and UIAutomation.

Contribute to Appium! - https://github.com/appium

IRC: #appium on freenode

Special Thanks

Sauce for hosting.

Appium community.

Everyone that joined and is still with us.

Thank you

Matt IsaacsGilthttp://gilt.com (http://gilt.com)

@gilttech (http://twitter.com/gilttech)

@haveahennessy (http://twitter.com/haveahennessy)