Presented by

Date

Browser Testing Framework for LHG

Trevor Woerner, Will Chen

February, 2015

Outline

● Overview of the test suite● Test category● Run the test

○ On Linux○ On Android○ Verified platforms

● Test output○ Output file○ Browser comparison

Overview of the test suite

Overview of the test suite

● Linaro Web Browser Test Suite is a benchmark tool to evaluate the performance of web browsing.

● It consists of Bash script, Python script, html, and Javascript files.

Overview of the test suite

● Test suite contain six categories (Sites, Layout, Image, Rendering, Javascript, HTML5).

● The testing result is reported in milliseconds, which is a measurement of the test suite processing time.

Overview of the test suite

● Features:○ Automatic○ Can be used on Linux and Android platform○ Easy to add test cases for future extension○ Browser options can be configured○ Test case/category/iteration number can be

configured

Overview of the test suite

● Compared to other test suite:○ High testing coverage - Combine most of the

existing browser test suite on the internet.○ Easy to compare - One score to represent the

performance.○ Has recent popular test cases - Add web album,

4k image, and web game test.

Test category

Test category

● Sites: This benchmark is an offline copy of the front page of some real and commonly used web sites.

Test category

● Layout: this benchmark performs mostly static HTML/CSS or javascript controlled dynamic HTML layout of objects on the page. This is supposed to benchmark sites that are mainly static and with little computation, few images and little user interaction.

Test category

● Image: this benchmark performs mainly image rendering and layout. This is supposed to benchmark sites with lots of images. For instance, it should demonstrate the laying out of the images in an online album.

Test category

● Rendering: this benchmark mainly consists of rendering animated vector graphics into a canvas. It is controlled by Javascript, but most of the time is spent in graphics libraries. This is supposed to benchmark sites that are graphic intensive.

Test category

● Javascript: this benchmark consists of Javascript tests, mainly focusing on the Javascript engine performance. It is supposed to reflect performance of rich internet application (RIA) sites. For instance it should demonstrate the usage of an online mail agent.

Test category

● HTML5: this benchmark performs image/canvas drawing, graphic manipulation, and 2D shape rendering specific for HTML5. For instance it should demonstrate the usage of the web game.

Run the test

● On Linux

Run the test on Linux

● We need a browser, bash script, and Python to run the test suite on Linux.

● Put test suite in the storage device and issue command, then the test suite will finish all the tests and generate testing result automatically.

Run the test on Linux

Run the test

● On Android

Run the test on Android

● We need to establish ADB connection between testing device and host computer to run the test.

● The test suite can install itself in the testing device automatically. The installed path is configured in script file.

Run the test on Android

Host computer

Target device

ADB connection

Run the test

● Verified platforms

Run the test on Android

Test output

● Output file

Test outputsuite_scores.out: Record Real Time and Process Time of each testing category. Also record the summing time of all the categories except html5. The total Process Time is the final benchmarking score (the unit is ms).

Test outputall_scores.out: Similar with suite_scores.out, but it will append each testing result in the file. So, we can check the previous result in this file.

Test outputrun.”category”.log: The log file when run the test for specific “category”.

Test outputrun.”category”.out: Record the Suite Score of each test case in the specific “category”. This file will record each case’s score and sum of the score. It will also contain the real time, user time, and system time while running this category (the unit is second).

Test outputtest_result_”category”.html: Similar with run.”category”.out file, but it is more easy to read. It shows the name and Suite Score of each case by table with html format.

Test output

● Browser comparison

Test outputComparison between chromium and firefox browser: (Platform is Intel i5 CPU + Ubuntu on Laptop)

Test output