Ensuring application performance in the cloud and mobile age
Ensuring quality in cloud and mobile applications
description
Transcript of Ensuring quality in cloud and mobile applications
![Page 1: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/1.jpg)
![Page 2: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/2.jpg)
1 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Ensuring Quality in Cloud and Mobile Applications
Gerard Meszaros [email protected]
![Page 3: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/3.jpg)
2 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Product & I.T.
I.T.
Embedded Telecom
My Background
Gerard Meszaros [email protected]
• Software developer
• Development manager
• Project Manager
• Software architect
• OOA/OOD Mentor
• Requirements (Use Case) Mentor
• XP/TDD Mentor
• Agile PM Mentor
• Test Automation Consultant & Trainer
• Lean/Agile Coach/Consultant
80’s ----- 90’s ----- 00’s
![Page 4: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/4.jpg)
3 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Agenda • What’s Special About Cloud and Mobile Apps? • What is Quality? • A Quality Strategy • Testing Cloud-Based Apps • Testing Mobile Apps • No DOH! • Ensuring Design for Testability • Tools, Frameworks and Patterns
![Page 5: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/5.jpg)
4 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Current Challenges
The Good Old Days • Captive Audience
– Have no real choice
• Stable requirements – Can have a single release
• Long lead times – Can have long manual test
cycle
• Relatively simple environment – Just a few inputs (user, DB) – Mostly synchronous
Current Reality
• Users Have Real Choice – Many alternatives
• Rapid change – Highly iterative; multi releases
• Rapid change – No time for extensive manual
testing
• Highly Complex Environment – Many sources of inputs – Sync and Async
![Page 6: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/6.jpg)
5 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
DB Server Desktop
Other System
Other System
User Interface
Business Logic
OS Services
Other System
Typical Desktop Application • UI Logic and
Business Logic run on desktop
• Server hosts the database.
Database
![Page 7: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/7.jpg)
6 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Desktop Browser
Server Browser
Other System
UI Logic
Business Logic Database
Container Services
Other System
Typical Cloud-Based Application • UI Logic runs in
browser • Business Logic is split
between Server and Browser
• Both use browser services
– Widgets, Cookies, SSL,DOM rendering
• Business logic may also interact with other systems.
– E.g. IP-based location • Browser-based BL
may receive data from server: – synchronous
(request-reply) – asynchronous
(event-based)
Business Logic
Browser Services
OS Services
(A.K.A. Software-as-a-Service or SaaS)
Dynamic scripting languages • won’t report compile errors until run.
![Page 8: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/8.jpg)
7 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Device Browser
Sensors GPS Acc
Gyro
OthrApps Contacts Msg’ing Gyro Batt
Server Browser
Other System
UI Logic
Business Logic Database
Container Services
Other System
Typical Web-Based Mobile Application • UI Logic runs in
browser • Business Logic is split
between Server and Browser
• Both use browser services
– Widgets, Cookies, SSL,DOM rendering
• Business logic may also interact with other systems.
– E.g. IP-based location • Browser-based BL
may receive data from server: – synchronous
(request-reply) – asynchronous
(event-based)
Business Logic
Browser Services
Platform Services
(A.K.A. Software-as-a-Service or SaaS)
![Page 9: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/9.jpg)
8 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Device
Server
Other System
UI Logic
Business Logic Database
Container Services
Other System
Typical Mobile Application • UI Logic runs in
device • Business Logic is split
between Device and Servers
• Both use Platform services
– Config, Prefs, Widgets,SSL,
• Business logic may also interact with other systems.
– E.g. IP-based location • Browser-based BL
may receive data from server: – synchronous
(request-reply) – asynchronous
(event-based)
Business Logic
Platform Services
Sensors GPS Acc
Gyro
OthrApps Contacts Msg’ing Gyro
Config Prefs
Time Prefs
Batt
![Page 10: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/10.jpg)
9 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Outputs / Controlables • Screen (LCD) • LED indicators • Vibration motor • Camera • Wifi, Bluetooth, 3G,
NFC Radios (on/off)
• Other Applications • Add event to
Calendar • Add/edit/read contact
![Page 11: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/11.jpg)
10 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Other static inputs Physical Configuration • Screen size • Screen density • Roles of Physical
buttons • Camera(s)
capabilities
User Preferences • Language / locale • Keyboard prefs • Application prefs
![Page 12: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/12.jpg)
11 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Sources of Events Sensors • Proximity • Accelerometer • GPS • Signal Strength (Wifi,
3g, BT, NFC) • Compass
Services • Send message (e-mail,
SMS, MMS, IAM) • Voice interpretation • Output Audio • IP Location • Wifi Location • GPS Location • Get/Show Advertising • In-App purchasing
![Page 13: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/13.jpg)
12 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Events & Reponses • Low Battery • Lost Data
Connection • Change in data
capabilities • Geofence crossed
• Valid Responses • Data returned sync • Data returned async • Error responses • Low resources • Can’t communicate
![Page 14: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/14.jpg)
13 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Example – Screen Layout • Large number of available screen sizes • Need to verify app uses screen space well • Four different orientations • For each screen size and orientation:
– need to verify how various screen elements are sized and placed
• May be possible to do manually for IOS – iPhone 1, iPhone 4, iPhone 5, iPad1/2, iPad3, iPod Touch
• Not possible to do well manually for Android – Automation is essential; best done at component level
![Page 15: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/15.jpg)
14 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Example – Screen Layout • What about changes to orientation of the
device? • That’s 4P2 or 12 transitions we need to test
North ->South North -> East North -> West
• And we have to run this test at various states of the app
• So, 12 tests in 10’s or 100’s of states • On 10’s to 100’s of device configurations
South -> North South -> East South -> West
East -> North East -> West East -> South
West -> North West -> East West -> South
![Page 16: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/16.jpg)
15 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Agenda • What’s Special About Cloud and Mobile Apps? • What is Quality? • A Quality Strategy • Testing Cloud-Based Apps • Testing Mobile Apps • No DOH! • Ensuring Design for Testability • Tools, Frameworks and Patterns
![Page 17: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/17.jpg)
16 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Google Quality Checklists Core App Quality Quality Criteria • Design and Interaction • Functionality • Performance and Stability • Google Play Testing • Setting Up a Test
Environment • Test Procedures You Should Also Read • Tablet App Quality Checklist • Improving App Quality
Tablet App Quality 1. Test for Core App Quality 2. Optimize your layouts 3. Use the extra screen area 4. Use assets designed for
tablets 5. Adjust fonts and touch targets 6. Adjust HomeScreen widgets 7. Offer the app's full feature set 8. Don’t require hardware
features 9. Declare tablet screen support 10. Follow best practices for
publishing in Google Play http://developer.android.com/distribute/googleplay/quality/core.html http://developer.android.com/distribute/googleplay/quality/tablet.html
![Page 18: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/18.jpg)
17 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
What Exactly is Quality?
Building the Right Thing • Fitness-for-purpose • Understand the users • And their usage env’ts
Practices: • UxD • Usability testing • Lean Startup stuff
Building the Thing Right • Good architecture • Well implemented in
code • Well tested at all levels
Practices: • Automated Unit Testing • Test-Driven
Development • ATDD • Exploratory Testing
![Page 19: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/19.jpg)
18 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Traditional Test Cycle
Development Verification & Acceptance
Software
Requirements Requirements
![Page 20: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/20.jpg)
19 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Agenda • What’s Special About Cloud and Mobile Apps? • What is Quality? • A Quality Strategy • Testing Cloud-Based Apps • Testing Mobile Apps • No DOH! • Ensuring Design for Testability • Tools, Frameworks and Patterns
![Page 21: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/21.jpg)
20 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
How to Achieve Quality?
![Page 22: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/22.jpg)
21 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
What Should We Test? • Product X consists of components A, B, and C • Components A and B are well tested but have
never been used together. • Component C is not well tested
• We need to Component-test C, • Interaction test A vs. B • Interaction test A vs C
![Page 23: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/23.jpg)
22 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
A Quality Strategy • Use automated tests at unit and component
level – to maximize confidence in quality of components.
• Use system-level tests focusing on the logic not already tested – mostly integration tests – automated if possible, manual otherwise
• Use Exploratory testing to find out what else could possibly go wrong. – Let testers follow their hunches to find problem areas
![Page 24: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/24.jpg)
23 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
A word about Terminology • Testing vs. Checking
– Checking is verifying that a behaviour occurs as expected
– Testing is seeing what happens when I do something – Automated Testing is therefor an oxymoron; it should be
automated checking. • Direct vs Indirect Inputs (and outputs)
SUT methodA( )
methodB( ) Client
Client
Client Objects (Users)
DOC
DOC
DOC
Depended-on Components
Client
Indirect Output
Input
Output
Direct Input
Indirect Input
Direct Output
System Under Test
![Page 25: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/25.jpg)
24 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Test Automation Pyramid
Unit Tests
Component Tests
System Tests
Pyramid originally proposed by Mike Cohn
Exploratory Tests • Tools to support effective
exploratory testing • A small number of tests
for the entire application & workflow – Ensure application(s) support
users’ requirements • Medium number of
functional tests for major components – Verify integration of units
• Large numbers of very small unit tests – Ensures integrity of code
![Page 26: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/26.jpg)
25 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Agenda • What’s Special About Cloud and Mobile Apps? • What is Quality? • A Quality Strategy • Testing Cloud-Based Apps • Testing Mobile Apps • No DOH! • Ensuring Design for Testability • Tools, Frameworks and Patterns
![Page 27: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/27.jpg)
26 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Server Browser
Other System
UI Logic
Business Logic Database
Container Services
Other System
Testing Cloud-Based Application • Natural interface
between Browser & Server facilitates testing – Server testing – Browser testing with
Server stubbed out • Component tests for
Browser Business Logic bypassing UI – Requires API on BL
• Also need unit tests for Business Logic code
Business Logic
Browser Services
OS Services
(A.K.A. Software-as-a-Service or SaaS)
![Page 28: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/28.jpg)
27 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Server Browser
Other System
UI Logic
Business Logic Database
Container Services
Other System
Testing Cloud-Based Application • Natural interface
between Browser & Server facilitates testing – Server testing – Browser testing with
Server stubbed out • Component tests for
Browser Business Logic bypassing UI – Requires API on BL
• Also need unit tests for Business Logic code
Business Logic
Browser Services
OS Services
(A.K.A. Software-as-a-Service or SaaS)
Server Test
![Page 29: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/29.jpg)
28 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Server Browser
Other System
UI Logic
Business Logic Database
Container Services
Other System
Testing Cloud-Based Application • Natural interface
between Browser & Server facilitates testing – Server testing – Browser testing with
Server stubbed out • Component tests for
Browser Business Logic bypassing UI – Requires API on BL
• Also need unit tests for Business Logic code
Business Logic
Browser Services
OS Services
(A.K.A. Software-as-a-Service or SaaS)
Whole Client Test
Server Test Stub
![Page 30: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/30.jpg)
29 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Server Browser
Other System
UI Logic
Business Logic Database
Container Services
Other System
Testing Cloud-Based Application • Natural interface
between Browser & Server facilitates testing – Server testing – Browser testing with
Server stubbed out • Component tests for
Browser Business Logic bypassing UI – Requires API on BL
• Also need unit tests for Business Logic code
Business Logic
Browser Services
OS Services
(A.K.A. Software-as-a-Service or SaaS)
Unit Test
Test Stub Unit
Unit Test
Test Stub Unit
Unit Test
Test Stub Unit
Unit Test
Test Stub Unit
![Page 31: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/31.jpg)
30 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Server Browser
Other System
UI Logic
Business Logic Database
Container Services
Other System
Testing Cloud-Based Application • Natural interface
between Browser & Server facilitates testing – Server testing – Browser testing with
Server stubbed out • Component tests for
Browser Business Logic bypassing UI – Requires API on BL
• Also need unit tests for Business Logic code
Business Logic
Browser Services
OS Services
(A.K.A. Software-as-a-Service or SaaS)
Business Logic Test
Server Test Stub
Platform Services Test Stub
![Page 32: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/32.jpg)
31 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Server Browser
Other System
UI Logic
Business Logic Database
Container Services
Other System
Testing Cloud-Based Application • Natural interface
between Browser & Server facilitates testing – Server testing – Browser testing with
Server stubbed out • Component tests for
Browser Business Logic bypassing UI – Requires API on BL
• Also need unit tests for Business Logic code
Business Logic
Browser Services
OS Services
(A.K.A. Software-as-a-Service or SaaS)
Server Logic Test
DB Test Stub
Platform Services Test Stub
![Page 33: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/33.jpg)
32 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Server Browser
Other System
UI Logic
Business Logic Database
Container Services
Other System
Testing Cloud-Based Application • Natural interface
between Browser & Server facilitates testing – Server testing – Browser testing with
Server stubbed out • Component tests for
Browser Business Logic bypassing UI – Requires API on BL
• Also need unit tests for Business Logic code
Business Logic
Browser Services
OS Services
(A.K.A. Software-as-a-Service or SaaS)
Whole System
Test
![Page 34: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/34.jpg)
33 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Special Considerations • Different Browsers and Browser-Versions
– Eg. Rendering behaviour, widget behavior, etc. • Different Host-OSes
– For OS-based services such as file choosers, etc. • Different languages & locales • Browser resizing
![Page 35: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/35.jpg)
34 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Server Device
Other System
UI Logic
Business Logic Database
Container Services
Other System
Testing Browser-Based Mobile Application Similar to testing
Cloud-based apps but:
• Different browsers & Screen sizes
• Need to test with events from relevant sensors – E.g. Orientations
and changes
Business Logic
Platform Services
Sensors GPS Acc
Gyro
OthrApps Contacts Msg’ing Gyro
Config Prefs
Time Prefs
Batt
![Page 36: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/36.jpg)
35 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Server Device
Other System
UI Logic
Business Logic Database
Container Services
Other System
Testing Browser-Based Mobile Application Similar to testing
Cloud-based apps but:
• Different browsers & Screen sizes
• Need to test with events from relevant sensors – E.g. Orientations
and changes
Business Logic
Platform Services
Sensors GPS Acc
Gyro
OthrApps Contacts Msg’ing Gyro
Config Prefs
Time Prefs
Batt
Business Logic Test
Server Test Stub
Platform Services Test Stub
![Page 37: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/37.jpg)
36 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Server Device
Other System
UI Logic
Business Logic Database
Container Services
Other System
Testing Native Mobile Application • To test device BL,
need to: – Provide API
bypassing UI – Trap outgoing
reqests to sensors and services
– Inject responses from same
• Need ability to: – Stub out platform
services, sensors and other apps
• Need unit tests for all code
Business Logic
Platform Services
Sensors GPS Acc
Gyro
OthrApps Contacts Msg’ing Gyro
Config Prefs
Time Prefs
Batt
![Page 38: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/38.jpg)
37 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Server Device
Other System
UI Logic
Business Logic Database
Container Services
Other System
Testing Native Mobile Application • To test device BL,
need to: – Provide API
bypassing UI – Trap outgoing
reqests to sensors and services
– Inject responses from same
• Need ability to: – Stub out platform
services, sensors and other apps
• Need unit tests for all code
Business Logic
Platform Services
Sensors GPS Acc
Gyro
OthrApps Contacts Msg’ing Gyro
Config Prefs
Time Prefs
Batt
Business Logic Test
Server Test Stub
Platform Services Test Stub
![Page 39: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/39.jpg)
38 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Server Device
Other System
UI Logic
Business Logic Database
Container Services
Other System
Testing Native Mobile Application • To test device BL,
need to: – Provide API
bypassing UI – Trap outgoing
reqests to sensors and services
– Inject responses from same
• Need ability to: – Stub out platform
services, sensors and other apps
• Need unit tests for all code
• Characterization Tests to detect changes in dependencies
Business Logic
Platform Services
Sensors GPS Acc
Gyro
OthrApps Contacts Msg’ing Gyro
Config Prefs
Time Prefs
Batt
Dependency Characterization
Tests
![Page 40: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/40.jpg)
39 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Agenda • What’s Special About Cloud and Mobile
Apps? • What is Quality? • A Quality Strategy • Testing Cloud-Based Apps • Testing Mobile Apps • No DOH! • Ensuring Design for Testability • Tools, Frameworks and Patterns
![Page 41: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/41.jpg)
40 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
DOH! • DOH! = Debugging on Hardware • Should be avoided because • Slow, tedious, non-deterministic • Complete code coverage is impossible
because we cannot cause on-demand: – Hardware errors – “Impossible” return codes or exceptions – All but most common events from sensors
You have to ask yourself the following question: Do I feel lucky?
![Page 42: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/42.jpg)
41 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
James Grenning on DOH! • Reasons why DOH is not feasible:
– There is no hardware, it’s being concurrently developed. – The hardware is expensive, so team members have to
share it. – DOH is slow, with tools that may be less capable than
the development system native tools. – The hardware has bugs, as if DOH! isn’t slow enough;
buggy hardware makes it extra difficult. – Building code for the target is slow. – Downloading to the target is slow. – The embedded processor is probably slow too.
![Page 43: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/43.jpg)
42 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Test in IDE (Automated)
Test in Emulator
(Automated)
Test on Hardware (Automated)
Test in Acceptance
(manual)
How No DOH? • How do we avoid Debugging on Hardware?
Unit Tests Component Tests
System Tests
Which tests:
When: Day 1 As soon as Available Weekly
![Page 44: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/44.jpg)
43 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Test in IDE (Automated)
Test in Emulator
(Automated)
Test on Hardware (Automated)
Test in Acceptance
(manual)
No DOH How • How do we avoid Debugging on Hardware?
Frequency: Every few minutes Daily Weekly+
Each code change Per component test Per user story
![Page 45: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/45.jpg)
44 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Agenda • What’s Special About Cloud and Mobile
Apps? • What is Quality? • A Quality Strategy • Testing Cloud-Based Apps • Testing Mobile Apps • No DOH! • Ensuring Design for Testability • Tools, Frameworks and Patterns
![Page 46: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/46.jpg)
45 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
How Do We Ensure Testability? “Design for Testability”, then code • Write tests last • And hope it’s easy
Or • Code for Testability • Via Test-Driven Development (TDD)
![Page 47: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/47.jpg)
46 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
TDD Rhythm
Test Code • First Test • Second Test • Third Test • Fourth Test
Product Code • Hard-coded method • Introduce variable • Introduce conditional • Surround with a loop
Just Like Double-Entry Booking: • An entry on the test side for each entry on
the Prod side
![Page 48: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/48.jpg)
47 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Test - After Architecture
System Under Test
• Must test through User Interface
Should we Notify? Process
Transaction
Configure Notification Threshold
Notification Rules
Transaction Interface
Configuration User
Interface
Notification Log
Workflow Test
Do Notification.
![Page 49: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/49.jpg)
48 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
System Under Test
Test-Driven Architecture • Need to provide API’s to invoke functionality
directly
• And ways to stub out dependencies
Should we Notify?
Do Notification.
Process Transaction
Configure Notification Threshold
Notification Rules
Notification Log
Transaction Interface
Configuration User
Interface Workflow
Test Test Stub
![Page 50: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/50.jpg)
49 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Test-Driven Architecture • “What kind of tests do I want to be able to automate?” • “Which component would be responsible for that part
of the behavior?”
Should we Notify?
Do Notification.
Process Transaction
Configure Notification Threshold
Notification Rules
Notification Log
Transaction Interface
Configuration User
Interface
Configuration TX Test
![Page 51: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/51.jpg)
50 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Test-Driven Architecture • “What other components would that component
depend on?” • “How can I break that dependency when component
testing this component?”
• With the right architecture, automating these tests is trivial
Should we Notify?
Do Notification.
Process Transaction
Configure Notification Threshold
Notification Rules
Notification Log
Transaction Interface
Configuration User
Interface Notification Rule Test
Notification Method Test
Notification Rules Fake
![Page 52: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/52.jpg)
51 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Unit Tests
Component Tests
System Tests
Workflow
Transactions
Behavior Specification at Right Level • Specify broad scope at minimum detail
– E.g. Use least detail when specifying workflow • Specify most detailed req’ts at narrowest scope
– E.g. Don’t use workflow when specifying business rules
Make examples / tests easy to
understand and easy to write
Broad Narrow Scope
Det
ail
High
Lo
w
Business Rules
Too vague
Too much detail Unmaintainable
Unit Tests
![Page 53: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/53.jpg)
52 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Agenda • What’s Special About Cloud and Mobile
Apps? • What is Quality? • A Quality Strategy • Testing Cloud-Based Apps • Testing Mobile Apps • No DOH! • Ensuring Design for Testability • Tools, Frameworks and Patterns
![Page 54: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/54.jpg)
53 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Tools, Frameworks & Patterns • Tools come & go
– Different ecosystems (Android, IOS, BBOS, WP, etc.) – New, better tools appear regularly
• Strategies persist – Need to define strategy first
• Then choose tools to support the strategy • And Patterns for testable designs
Avoids “Impedance Mismatch” between tools and strategy
![Page 55: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/55.jpg)
54 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Framework Issues • Many frameworks not designed for testability • May need to introduce level of indirection to
API (via Proxy or Adapter) • May need to extract Plain Ordinary Testable
Object (POTO) leaving a Humble Object that subclasses framework base class
![Page 56: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/56.jpg)
55 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Extract POTO - Before
Framework Plug-in
ActionBase
MyAction
doAction()
MyActionTest
testDoAction()
tests Rules
ActionContext
+rules
getRules()
![Page 57: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/57.jpg)
56 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Framework Plug-in
Extract POTO - After
Runs in device; Tests “wrapping”
(Optional)
Runs in IDE; Tests “business logic”
MyActionHOTest
testDoAction()
ActionBase
MyActionHO
doAction()
tests
MyActionPoto
doAction(rules)
MyActionPotoTest
testDoAction() tests
Rules
ActionContext
+rules
Humble Object
Testable Object
Rules
Fake Rules
Simple design change; Huge testability improvement!
getRules()
![Page 58: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/58.jpg)
57 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Conclusions • Mobile and Cloud-Based Applications come
with special challenges – While business logic tends to be simpler, constraints
and variations make design and testing more challenging
• Testing-in Quality is not an option – There are too many potentially interacting dimensions – Many are not controllable from outside the black box
• Need to avoid Debugging on Hardware – Test as much as possible “off device” – Only deploy code you know works correctly
![Page 59: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/59.jpg)
58 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Conclusions • Design-for-Testability is essential for success
– Otherwise, tests will be too expensive to execute and/or automate
• Beware tools/frameworks that make DfT difficult – May need to decouple from framework to ensure
testability – But extra effort is compensated for by better quality and
productivity
![Page 60: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/60.jpg)
59 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
What Can You Do? • Product Owner:
– Break work into manageble chunks – Provide examples/tests for each chunk to drive
development – Insist that team automates the tests
• Tester/BA: – Provide tests to developers Before they start dev – Insist on seeing the results of the automate tests before
you start testing – Focus manual testing on areas not covered by
automated tests
![Page 61: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/61.jpg)
60 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
What Can You Do? Developer: • Ask for tests from PO/Testers before agreeing to build • Discuss the kinds of tests that they will provide • Include the work to automate the tests in any
estimates you provide • Implement the automation before you develop the
code, not after! • Use the automated tests to drive development • Don’t recreate provided tests at unit test level; find
ways of reusing business rule tests as unit tests • Think/Do Design-for-Testability • If you find yourself in an interactive debugger, ask
yourself: “What unit tests would tell me the same information and prevent this from happening again?”
![Page 62: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/62.jpg)
61 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
References • James Grenning “Don’t Let Embedded Tool Chain
Slow You Down” – http://www.renaissancesoftware.net/blog/archives/41
• Michael Bolton “Testing vs. Checking” – http://www.developsense.com/blog/2009/08/testing-vs-checking/
• Dion Johnson “Cyber-dueling Over ‘Check’ vs. ‘Test’ and Other Semantics” – http://www.automatedtestinginstitute.com/home/index.php?
option=com_content&task=view&id=1373 • Gerard Meszaros “Humble Object”
– http://xunitpatterns.com/Humble%20Object.html
• Homer Simpson “Doh!” – http://en.wikipedia.org/wiki/D%27oh!
![Page 63: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/63.jpg)
62 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Resources • http://www.mobileapptesting.com/ • For Success, Build Record/Playback into Your
Application - StarEast 2008 – http://builtinrecordandplayback.xunitpatterns.com
– http://mobile.testAutomationPatterns.com
![Page 64: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/64.jpg)
63 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012
Thank You! Gerard Meszaros
[email protected] http://www.xunitpatterns.com
Slides:
http://infoQ2012hz.testautomationpatterns.com Call me when you: • Want to transition to Agile or Lean • Want to do Agile or Lean better • Want to teach developers how to test • Need help with test automation strategy • Want to improve your test automation
Jolt Productivity Award winner - Technical Books
Coming Soon:
![Page 65: Ensuring quality in cloud and mobile applications](https://reader033.fdocuments.in/reader033/viewer/2022052522/548fc5f3b479597e6a8b5142/html5/thumbnails/65.jpg)