www.immobilienscout24.dewww.immobilienscout24.de

Android Meetup Berlin | 29.08.2012 | Jürgen Schmidt

Android FragmentationQA lessons learned from the fragmentation hell

www.immobilienscout24.de

About me

Jürgen Schmidt

Test Engineer

Development Lifecycle Expert

Database addicted

| Android Fragmentation | Jürgen Schmidt Page 3

Fragmentation

QA must consider different levels of fragmentation

• OEMs (>27)• OS levels (16)• Display size/resolution (>100)• Further hardware features

Devices (> 400)

| Android Fragmentation | Jürgen Schmidt Page 4

Strategy

• With an effort of 20% cover 80% of all devices to test

• Source of usage statistics:• App statistics on publisher

site on Google Play Store• Reporting Frameworks like

Google Analytics

• Don‘t forget potential new devices or features

20% effort

80% coverage

Follow the famous 80/20 rule

ImmoScout App Facts…

Top30 ~80%

49%

17%8%

3%

2%1%

20%

OEM

Samsung SonyEricsson HTCLG Motorola GoogleSonstige

| Android Fragmentation | Jürgen Schmidt Page 5

Which devices are currentlyused by our users?

Find a minimum set among top devices which cover all top 80% device features ~ 92%

56%

12%

9%

8%4%

3%

3% 3% 2%2%

Display Resolution

480x800 720x1280 320x480 480x854540x960 720x1184 800x1232 800x1280240x320 Other

| Android Fragmentation | Jürgen Schmidt Page 6

Selected top devices for manual test

Wildfire S Galaxy S 2 Galaxy S 3 Nexus 7

OEM HTC Samsung Samsung Asus Google

OS level 2.2 (8) 2.3.5 (10) 4.0.3 (15) 4.1 (16)

Display Res. 320x480 480x800 720x1280 800x1280

Density ldpi mdpi hdpi mdpi

| Android Fragmentation | Jürgen Schmidt Page 7

Test automation

We consider OS level fragmentation for running unit tests and robotium system tests on virtual devices

Jenkins (CI server) with Android plugin offers multi-configuration jobs for executing tests on different virtual devices at the same time

| Android Fragmentation | Jürgen Schmidt Page 8

Further tests to help finding bugs caused by fragmentation

Session Based Tests• Unstructured exploratory test

driven by test and domain experts• Use complete device pool

Crowd-sourced Testing• Bug hunting by real users

under real conditions• Many testers with a lot of

various devices

| Android Fragmentation | Jürgen Schmidt Page 9

Monitoring a (pre-)published app

Many real-time bug reporting frameworks can be integrated into the app

Help in evaluation of reported exceptions or crashes by grouping and annotating of device features

Quality risks caused by fragmentation are managable

Experience show: Most found critical bugs are not specific for certain devices, OS level etc.

Focus only on top and new features instead reaching full coverage

Conclusion

| Android Fragmentation | Jürgen Schmidt Page 10

Limit your efforts in testing fragmentation!

Android fragmentation visualized by OpenSignalMaps:http://opensignalmaps.com/reports/fragmentation.php

TechCrunch article to show how several game developers manage fragmentationhttp://techcrunch.com/2012/06/02/android-qa-testing-quality-assurance/

About Session Based Testing by Satisfice Inc.:http://www.satisfice.com/sbtm/

Blog post about Crowd-sourced testing by Mithun Sridharanhttp://it.toolbox.com/blogs/mithuns-memoirs/crowd-sourced-software-testing-52763

Want to read more about fragmentation?

| Android Fragmentation | Jürgen Schmidt Page 11

www.immobilienscout24.de

Contact:ImmobilienScout24 Andreasstraße 1010243 Berlin

Thanks for your attention!

Fon: 030/24301-11 00Email: info@immobilienscout24.deURL: www.immobilienscout24.de