Post on 12-Jul-2020
2011
Mobile Testing at Google: A Marriage of Idealism and
Pragmatism
QUEST Boston
Thursday, April 7th, 2011
11:00 AM – 12:00 PM
PRESENTER: Matthew VaughanVail and Dounia Berrada COMPANY: Google, Inc.
This was page intentionally left blank
Mobile Testing at Google:A Marriage of Idealism and Pragmatism
Matthew Vaughan-Vailmattv@google.com
Dounia Berradaberrada@google.com
April 8th, 2011Quest 2011 Conference
Pragmatism
Idealism
Introduction to Us
Matthew Vaughan-VailSoftware Engineering MangerTech Lead - Book Search Test Engineering
Dounia BerradaSoftware Engineer in TestMobile Testing
Agenda
Intro to Google Testing PhilosophyMobile Testing Overview and GuidelinesNative App TestingWeb App TestingConclusion
Google Test Engineering
Google's Testing Philosophy
"When writing the code, think of the test. When writing the test, think of the code. When you think of code and test as one, testing is easy and code is beautiful" - Alberto Savoia's The Way of Testivus
Who tests at Google?Software EngineersTest EngineersSoftware Engineers in Test
Testing Pragmatically
Build on existing solutionsSimple solutions are preferableUse best practices and conform to standards where applicableInvestment in testing infrastructure pays dividends
Testing Idealistically
Success measured on product impactDon't let your tools kill youInnovate, Pretotype, and Take Risks
The cost of taking a risk and failing is less than not taking a risk at all
Simplicity, Speed, Feedback
Mobile at Google
http://www.google.com/mobile
Mobile Testing Challenges
Several Operating PlatformsAndroid, Blackberry, iOS, Symbian, WinMo
Various number of inputsRange and variety of hardwareDesktop Application ParityMobile web browser compatibilitySlower developmentLack of developer tools
General Guidelines
Unit test!Use real world data in lieu of Network Simulators Abstract platform specific primitivesDesign interfaces consistently
Abstract Platform Primitives
Example: Mobile Maps C++Same code base for WinMo/SymbianIsolate and abstract platform-specific implementation
Native App Testing
Unit TestingInstrumentationImage Comparison TestingTrusted Tester (crowd sourcing)Latency Testing
Instrumentation
A set of "hooks" into the OS that control components independent of normal lifecycle.
Pros:Programmatically control the application to drive scenario based testsEncourages developers to think of testing during development
Cons:Cannot test cross-application tasks (e.g. browser hand off)May require deeper technical understanding of the OS and/or codebase
Image Based Testing
Pragmatism: Don't do it! It doesn't work!Idealism: It can work if we fix the pain points.
Sikuli (sikuli.org) Demo
Trusted Tester (crowd sourcing)
Latency Testing Native Apps
Tool FAIL :(
Testing Web Applications
Unit Testing
End-to-End Testing
Latency Testing
Unit-Testing with JsTestDriver
Unit-testing with JsTestDriver
var HelloWorldTest = TestCase('HelloWorldTest');HelloWorldTest.prototype.testHello = function() { var helloWorld = new HelloWorld(); var name = 'world'; assertEquals('hello ' + name, helloWorld.hello(name));};
HelloWorldTest.prototype.testHelloDom = function() { /*:DOC helloDom = <div id="hello"></div> */ var helloWorld = new HelloWorld(); helloWorld.paintHello(helloDom); assertEquals('hello', helloDom.innerHTML);}
Example:
Async JsTestDriver
Event-driven code (e.g. XHRs and window.setTimeout)
apps.something.SomeTest = AsyncTestCase('apps.something.SomeTest');
apps.something.SomeTest.prototype.testSomething = function(queue) { queue.defer(function(pool) { var myCallback = pool.add(function() { assertTrue(something); }); window.setTimeout(myCallback, 3000); });};
Unit-testing with JsTestDriver
Pros:Fast executionDeclarative html syntaxSimplified dependency inclusionPlatform independent
Cons:No cleanup of global state No external dependencies (extra server resources)
JsTestDriver Demo
End-To-End Tests With WebDriver
Navigation: WebDriver.get("http://www.google.com"); WebDriver.navigate().back(); WebDriver.navigate().refresh();
Getting information about the page: WebDriver.getCurrentUrl(); WebDriver.getTitle();
Interacting with different parts of the page: WebDriver.switchTo().frame(...); WebDriver.switchTo().window(String id); WebDriver.getWindowHandle() / getWindowHandles();
Architecture Model
End-To-End Tests With WebDriver
The By class:By.idBy.linkTextBy.partialLinkTextBy.nameBy.tagNameBy.xpathBy.classNameBy.cssSelector
Using the By class:WebDriver.findElement(By by);WebDriver.findElements(By by);
End-To-End Tests With WebDriver
Interrogation WebElement.getValue(); WebElement.getTagName(); WebElement.getAttribute(String attribute); WebElement.getText();
Interaction WebElement.click(); WebElement.submit(); WebElement.sendKeys(); WebElement.setSelected();
Finding sub-elements WebElement.findElement(By by); WebElement.findElements(By by);
End-To-End Tests With WebDriver
public class OneTest extends TestCase {
public void testGoogle() throws Exception { AndroidDriver driver = new AndroidDriver(); // And now use this to visit Google driver.get("http://www.google.com"); // Find the text input element by its name WebElement element = driver.findElement(By.name("q")); // Enter something to search for element.sendKeys("Android Rocks!"); // Now submit the form. WebDriver will find the form for us from the element element.submit(); // Check the title of the page assertTrue(driver.getTitle().contains("Google")); driver.quit(); }}
Android WebDriver Demo
End-To-End Tests With WebDriver
High Level Architecture
Page Object Design Pattern
Model screens of the web application as objects:
Screen / Component <=> Object
Services <=> Public methods
Navigate using other Page Objects
Page Object Design Pattern
public void testMessagesAreReceived() { Inbox inbox = new Inbox(driver);
ComposeMail compose = inbox.clickOnComposeMail();
compose.writeSubject("Hello"); compose.writeBody("I like cheese"); compose.addRecipient("cheese@gmail.com");
compose.sendMail();
assertTrue(inbox.isMessageWithSubjectIsUnread("Hello")); }
Compose Mail Example:
HTML5
BrowserConnection
LocationContext
WebStorageSessionStorageLocalStorage
DatabaseStorage
Mobile WebDriver
Pros:Mobile browsersReal/native eventsGrowing HTML5 support
Cons:Slower than desktop drivers
Mobile Web Infrastructure
Speed: build / run mobile tests in the cloudAndroid VM can run anywhere
Simplicity:Specify browser in build languageOne command can run the same tests on multiple browsers
Availability:Emulators in the cloudFarm of devices
Mobile tests become part of the continuous integration / build system
Latency Testing
Mobile WebDriver to drive the testJavaScript to instrument
<html><head><script type="text/javascript">
function onLoad() { var now = new Date().getTime(); var page_load_time = now - performance.timing.navigationStart; alert("User-perceived page loading time: " + page_load_time); }
</script></head> <body onload="onLoad()"> <!- Main page body goes from here. --> </body></html>
Questions?
Google Testing Blog - googletesting.blogspot.com
Open source tools:WebDriver - code.google.com/p/seleniumJsTestDriver - code.google.com/p/js-test-driver/
Async JsTestDriver - code.google.com/p/js-test-driver/wiki/AsyncTestCaseSikuli - sikuli.org