2011 

 

  

Mobile Testing at Google: A Marriage of Idealism and 

Pragmatism

QUEST Boston 

Thursday, April 7th, 2011 

11:00 AM – 12:00 PM 

PRESENTER: Matthew Vaughan­Vail and Dounia Berrada COMPANY: Google, Inc.   

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

This was page intentionally left blank 

Mobile Testing at Google:A Marriage of Idealism and Pragmatism

Matthew Vaughan-Vailmattv@google.com

Dounia Berradaberrada@google.com

April 8th, 2011Quest 2011 Conference

Pragmatism

Idealism

Introduction to Us

Matthew Vaughan-VailSoftware Engineering MangerTech Lead - Book Search Test Engineering

Dounia BerradaSoftware Engineer in TestMobile Testing

Agenda

Intro to Google Testing PhilosophyMobile Testing Overview and GuidelinesNative App TestingWeb App TestingConclusion

Google Test Engineering

Google's Testing Philosophy

"When writing the code, think of the test. When writing the test, think of the code. When you think of code and test as one, testing is easy and code is beautiful" - Alberto Savoia's The Way of Testivus

Who tests at Google?Software EngineersTest EngineersSoftware Engineers in Test

Testing Pragmatically

Build on existing solutionsSimple solutions are preferableUse best practices and conform to standards where applicableInvestment in testing infrastructure pays dividends

Testing Idealistically

Success measured on product impactDon't let your tools kill youInnovate, Pretotype, and Take Risks

The cost of taking a risk and failing is less than not taking a risk at all

Simplicity, Speed, Feedback

Mobile at Google

http://www.google.com/mobile

Mobile Testing Challenges

Several Operating PlatformsAndroid, Blackberry, iOS, Symbian, WinMo

Various number of inputsRange and variety of hardwareDesktop Application ParityMobile web browser compatibilitySlower developmentLack of developer tools

General Guidelines

Unit test!Use real world data in lieu of Network Simulators Abstract platform specific primitivesDesign interfaces consistently

Abstract Platform Primitives

Example: Mobile Maps C++Same code base for WinMo/SymbianIsolate and abstract platform-specific implementation

Native App Testing

Unit TestingInstrumentationImage Comparison TestingTrusted Tester (crowd sourcing)Latency Testing

Instrumentation

A set of "hooks" into the OS that control components independent of normal lifecycle.

Pros:Programmatically control the application to drive scenario based testsEncourages developers to think of testing during development

Cons:Cannot test cross-application tasks (e.g. browser hand off)May require deeper technical understanding of the OS and/or codebase

Image Based Testing

Pragmatism: Don't do it! It doesn't work!Idealism: It can work if we fix the pain points.

Sikuli (sikuli.org) Demo

Trusted Tester (crowd sourcing)

Latency Testing Native Apps

Tool FAIL :(

Testing Web Applications

Unit Testing

End-to-End Testing

Latency Testing

Unit-Testing with JsTestDriver

Unit-testing with JsTestDriver

var HelloWorldTest = TestCase('HelloWorldTest');HelloWorldTest.prototype.testHello = function() { var helloWorld = new HelloWorld(); var name = 'world'; assertEquals('hello ' + name, helloWorld.hello(name));};

HelloWorldTest.prototype.testHelloDom = function() { /*:DOC helloDom = <div id="hello"></div> */ var helloWorld = new HelloWorld(); helloWorld.paintHello(helloDom); assertEquals('hello', helloDom.innerHTML);}

Example:

Async JsTestDriver

Event-driven code (e.g. XHRs and window.setTimeout)

apps.something.SomeTest = AsyncTestCase('apps.something.SomeTest');

apps.something.SomeTest.prototype.testSomething = function(queue) { queue.defer(function(pool) { var myCallback = pool.add(function() { assertTrue(something); }); window.setTimeout(myCallback, 3000); });};

Unit-testing with JsTestDriver

Pros:Fast executionDeclarative html syntaxSimplified dependency inclusionPlatform independent

Cons:No cleanup of global state No external dependencies (extra server resources)

JsTestDriver Demo

End-To-End Tests With WebDriver

Navigation: WebDriver.get("http://www.google.com"); WebDriver.navigate().back(); WebDriver.navigate().refresh();

Getting information about the page: WebDriver.getCurrentUrl(); WebDriver.getTitle();

Interacting with different parts of the page: WebDriver.switchTo().frame(...); WebDriver.switchTo().window(String id); WebDriver.getWindowHandle() / getWindowHandles();

Architecture Model

End-To-End Tests With WebDriver

The By class:By.idBy.linkTextBy.partialLinkTextBy.nameBy.tagNameBy.xpathBy.classNameBy.cssSelector

Using the By class:WebDriver.findElement(By by);WebDriver.findElements(By by);

End-To-End Tests With WebDriver

Interrogation WebElement.getValue(); WebElement.getTagName(); WebElement.getAttribute(String attribute); WebElement.getText();

Interaction WebElement.click(); WebElement.submit(); WebElement.sendKeys(); WebElement.setSelected();

Finding sub-elements WebElement.findElement(By by); WebElement.findElements(By by);

End-To-End Tests With WebDriver

public class OneTest extends TestCase {

public void testGoogle() throws Exception { AndroidDriver driver = new AndroidDriver(); // And now use this to visit Google driver.get("http://www.google.com"); // Find the text input element by its name WebElement element = driver.findElement(By.name("q")); // Enter something to search for element.sendKeys("Android Rocks!"); // Now submit the form. WebDriver will find the form for us from the element element.submit(); // Check the title of the page assertTrue(driver.getTitle().contains("Google")); driver.quit(); }}

Android WebDriver Demo

End-To-End Tests With WebDriver

High Level Architecture

Page Object Design Pattern

Model screens of the web application as objects:

Screen / Component <=> Object

Services <=> Public methods

Navigate using other Page Objects

Page Object Design Pattern

public void testMessagesAreReceived() { Inbox inbox = new Inbox(driver);

ComposeMail compose = inbox.clickOnComposeMail();

compose.writeSubject("Hello"); compose.writeBody("I like cheese"); compose.addRecipient("cheese@gmail.com");

compose.sendMail();

assertTrue(inbox.isMessageWithSubjectIsUnread("Hello")); }

Compose Mail Example:

HTML5

BrowserConnection

LocationContext

WebStorageSessionStorageLocalStorage

DatabaseStorage

Mobile WebDriver

Pros:Mobile browsersReal/native eventsGrowing HTML5 support

Cons:Slower than desktop drivers

Mobile Web Infrastructure

Speed: build / run mobile tests in the cloudAndroid VM can run anywhere

Simplicity:Specify browser in build languageOne command can run the same tests on multiple browsers

Availability:Emulators in the cloudFarm of devices

Mobile tests become part of the continuous integration / build system

Latency Testing

Mobile WebDriver to drive the testJavaScript to instrument

<html><head><script type="text/javascript">

function onLoad() { var now = new Date().getTime(); var page_load_time = now - performance.timing.navigationStart; alert("User-perceived page loading time: " + page_load_time); }

</script></head> <body onload="onLoad()"> <!- Main page body goes from here. --> </body></html>

Questions?

Google Testing Blog - googletesting.blogspot.com

Open source tools:WebDriver - code.google.com/p/seleniumJsTestDriver - code.google.com/p/js-test-driver/

Async JsTestDriver - code.google.com/p/js-test-driver/wiki/AsyncTestCaseSikuli - sikuli.org