Title

7. Bug process

Intel® 2014

BayTrail 64-bit Android

Platform Validation Test Plan

PC Client Group (PCCG) System Test & Validation (PSTV)

13 March, 2014

Revision v0.4

Copyright and Disclaimer

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined." Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained at: http://www.intel.com/design/literature.htm

All products, platforms, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. All dates specified are target dates, are provided for planning purposes only and are subject to change. This document contains information on products in the design phase of development. Do not finalize a design with this information. Revised information will be published when the product is available. Verify with your local sales office that you have the latest datasheet before finalizing a design. Code names featured are used internally within Intel to identify products that are in development and not yet publicly announced for release. Customers, licensees and other third parties are not authorized by Intel to use code names in advertising, promotion or marketing of any product or services and any such use of Intel's internal code names is at the sole risk of the user. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries.

*Other names and brands may be claimed as the property of others.

Copyright © 2013, Intel Corporation, All rights reserved.

Contents

Revision Taxonomy7

Revision History8

1. Introduction9

1.1 Purpose and Document Scope9

1.2 Terminologies9

1.3 Reference Documents12

2. Stakeholders/ Content Reviewers13

3. Product Architecture15

BayTrail Block Diagram16

BayTrail Features16

BayTrail – Valley View SOC20

Android Architecture24

Intel Optimizations to the Android Software Stack24

4. Test Scope25

Validation Overview25

System Validation Scope25

5. Test Strategy27

Test content strategy27

Test coverage strategy27

Test execution strategy29

Test automation strategy29

Test categories29

6. Weekly execution Strategy31

Acceptance Criteria31

Weekly Validation cycle31

Validation Execution Schedule34

High level milestone test execution includes:34

Priority of tests35

7. Bug process40

Bug priority levels40

Bug Severity levels41

8. Guidelines for Test case development43

9. Platform Test Overview47

Android47

Table 7.1 Android SW Stack - BSP Deliverable47

Connectivity48

Graphics/Media/Video49

Imaging / Camera49

Audio49

EM49

Sensor49

USB & Storage50

System Test Security50

System Test – Usability/Usage50

System Interoperability50

System Compatibility51

Responsiveness55

Power & Performance55

Performance Overview56

System Stress60

Power cycling methods60

System Stress tests61

ACS66

MTBF Scope68

Concurrent stress tests72

Certifications73

Compatibility Test Suite (CTS)73

GTS Overview78

Exploratory Test80

Overview80

Scope80

Planner Content80

Example of an Exploratory Session Planner for A2DP80

Planner80

Tested Area(s)81

Test Tools81

Bugs(s)81

Test Notes81

Appendix Landing Zone82

Figures

Figure 21. BayTrail Block Diagram16

Figure 22 Valley View SoC22

Figure 23 Android Architecture24

Figure 41. BayTrail Validation Execution Schedule and Milestone33

Figure 48. Power KPI Targets Table -157

Figure 47. Python Test Scripts + Xml UECmds68

Figure 61. Compatibility Test Suite (CTS) Process74

Figure 62. Google Certification Process75

Tables

Table 11. Terminologies8

Table 12. Reference Documents11

Table 21. BayTrail Features15

Table 22. BayTrail Variants for Android18

Table 42. Customer H/W Configurations21

Table 31. Platform Validation in Scope23

Table 6: Milestone Schedule33

Table 11: Tags for domains for test cases44

Table 47 Performance KPI Targets Table -256

Table 45. MTBF – Test Suite68

Table 61. Test package73

Table 62. Baylay Bay CRB – Golden Configuration80

Contents

7

BayTrail Android Validation Test Plan Intel Confidential16

BayTrail Android Validation Test Plan Intel Confidential5

Revision Taxonomy

Revision History

Document Number

Revision Number

Reason for Changes

Revision Date

0.1

Initial Test Plan creation

February 27, 2014

0.4

Updated document with all sections

March 12, 2014

Contents

Contents

1. Introduction

This Validation Test Plan document details the overall system validation strategy, methodology, scope, and process flow adopted for the validation of Intel® BayTrail 64-bit RVP Based Validation for Android Kitkat 4.4.2. The validation vehicle is for RVP. It shall cover all the POR components - including both Internal and 3rd Part, as per the POR Matrix ensuring optimum coverage on various configurations (detailed in later sections).

1.1 Purpose and Document Scope

The document describes the approaches to system test planning and execution strategy for the Intel® BayTrail RVP programs. The intended audience includes validation engineers, stake holders/PST/SW QRC team, and so on.

This document also describes requirements and customer commitments as known today. The current document will be updated as new product requirements or information becomes available and also if the test approach needs update or the existing document no longer meets the purpose. Descriptions of product functionality reside in other documents but may be provided in this document as an example to aid the reader.

This document only describes the test approaches. The actual detailed test cases reside in the Contour Requirement Management tool

1.2 Terminologies

Table 11 Terminologies

Term

Definition

ACPI

ACPI (Advanced Configuration and Power Interface) is an open industry specification co-developed by Hewlett-Packard, Intel, Microsoft, Phoenix, and Toshiba.

ACS

Automation Control System

ADB

Intel® Automatic Display Brightness

ADB

Android Debug Bridge

ALS

Ambient Light Sensor

AOSP

Android Open Source Project

ACS

Automation Control System

ATF

Android test framework

.apk

File extension of an Android application

BIOS

Basic Input / Output System

BLC

Back Light Control

BT

Bluetooth

CPU

Central Processing Unit

CTS

Compatibility Test Suite

DP

Display Panel

DPTF

Dynamic Platform Thermal Framework

DRM

Digital Rights Management

Dx

Display Power State

EC

Embedded Controller

eDP

Embedded Display Port

EFI

Extensible Firmware Interface (i.e., Tiano BIOS mechanism for adding features)

eMMC

Embedded Multimedia Card

GPIO

General Purpose IO – IO control for device management within the system

GPU

Graphics Processing Unit

GTS

GMS Test Suite

HD

High Definition

HDCP

High-bandwidth Digital Content Protection

HDD

Hard Disk Drive

HDMI

High-Definition Multimedia Interface

HID

Human Interface Device (keyboard or mouse)

HSIC

High Speed Interchip

KPI

Key Performance Indicators

KSC

Keyboard Scan Controller

LCD

Liquid Crystal Display

LED

Light Emitting Diode

LPE

Low Power Engine

MSR

Model Specific Register

MTBF

Mean Time Between Failure

OEM

Original Equipment Manufacturer

OpenGL ES

Open Graphics Library

OS

Operating System

OSC

Oscillator

OTA

Over the Air

PAT

Platform Acceptance Tests

PM

Power Management

PPI

Pixels per inch

PRL

Preferred Roaming List

PSR

Panel self-refresh

PTP

Picture Transfer Protocol

PWM

Pulse Width Modulation

RAM

Random Access Memory

RFID

Radio Frequency Identification

RTC

Real Time Clock

SATA

Serial Advanced Technology Attachment

SBIOS

System BIOS

SDK

Software Development Kit

SDRAM

Synchronous Dynamic Random Access Memory

SDcard

Secure Digital

SSD

Solid State Disk, eg: Flash memory configured to look like an HDD (has an IDE interface).

Sx

Sleep State

1080p

1080 Pixels

TIM

Thermal Interface Material

Tjmax

Maximum Junction Temperature rating

TPM

Trusted Platform Module (http://www.trustedcomputinggroup.org/)

USB

Universal Serial Bus

VBIOS

Video BIOS

VCO

Voltage Controlled Oscillator

WideVine

DRM solution Used by Google

WiDi

Wireless Display

WiFi

Wireless exchange of data

XTAL

Crystal

1.3 Reference Documents

Table 12 Reference Documents

Document Title

Revision

Revision Date

Author

Link

Intel® Pentium® Processor N3520, J2850,J2900, & Intel® Celeron® Processor N2920, N2820, N2815, N2806, J1850, J1750, J1900, J1800 – External Design Specification (EDS) – Rev. 2.5

2.5

December 2013

N/A

N/A

Baytrail Graphics and Display SAS

0.5

March 5th, 2013

Jim Bish

PSIArch URL

Baytrail-WidevineDRM-SAS-v 23

0.23

March 1st, 2013

NA

PSIArch URL

BayTrail Android MR1 Wireless Display SAS v0.5

0.31

December 30th, 2013

Prakash Iyer, J-P Giacalone

PSIArch URL

BYT 64-bit PRD Feature Dashboard

-

-

-

https://jira01.devtools.intel.com/secure/Dashboard.jspa?selectPageId=14400

2. Stakeholders/ Content Reviewers

Name

Organization

Role

Nguyen, Trung

PCCG/PSTV

PSTV Engineering Manager

Jeff Harris

PCCG/PSTV

PSTV Android Engineering Manager

Mukesh Kothari

PCCG/PSTV

PSTV Engineering Manager

Senthilvadivu Guruswamy

PCCG/PSTV

PSTV Project Manager

Adam Caplan

PCCG

PCCG Android Atom PXT Lead

Vang Sebastien

PCCG/PSTV

PSTV Android Lead

Rushdan A Abdul-khaliq

PCCG/PSTV

PSTV Android Architect

Arun Anand Pai

PCCG/PSTV

PSTV BYT Android Lead

Daniel Tsui

PCCG/PSTV

PSTV BSW Android Lead

Sagar Sarma

PCCG/CSS

CSS Android Development Manager (Core)

Srinivas Sripathi

PCCG/CSS

CSS Android Development Manager (Atom)

Daniel Hsia

PCCG/CSS

CSS Android Development Manager

Manoj Dhawan

PCCG/CSS

CSS Android Engineering Manager

Ankur Garg

PCCG/CSS

BIOS Development for Core & BYT-M&D

Robert Dower, Christian C

MCG

MCG-PSI

Pierre B, Matt G, Rob R

SSG/OTC

Android Enabling - OTC

Przekop Zbigniew

SSG/OTC

Android Audio Dev Manager

Maciej Dochtorowicz

SSG/OTC

Android Audio Validation manager

Terence Chen

SSG/OTC

Platform Validation SQA Manager SSG/OTC

Aldo Veranzo

PCCG/CSS

Security program manager

Randy Bulriss

PCCG

Android Core Planner

Laurentiu Serban

SSG/OTC

SSG Android validation lead

John Lin

PCCG/CSS

Build & Release Infra Manager (US)

Mitali Monalisa

PCCG/CSS

SEI Program Manager

M V Chandra

PCCG/CSS

Build & Release Infra Manager (BA)

Anil Sahu

PCCG/CSS

Bios Ingredient Validation (Core & BYT)

Balaji Manigandan

PCCG/CSS

Sysdebug Rep

Seokwon Yang

PCCG/CSS

Test Automation Framework

Leesa Hicks

PCCG/CSS

Build & Release Infra Engineer

Andrei Lorga

SSG/OTC

SSG/OTC CTS Expert - Andorid Core Validation (HSW & BDW)

Mihai Ceausu

SSG/OTC

Android Test Framework and CI Lead

Lonel Mitel

SSG/OTC

Android SDK and HAXM

Catalin Nitu

SSG/OTC

Andorid Core Validation

Joe Daly

SSG/OTC

Android Engineering Team Director

Dirk Hihndel

SSG/OTC

Chief Linux and OSS technologist

Anton Tchachkin

SSG/OTC

CHT/BSW SeC SW/FW VPEM

Andy Jin

PCCG/PSTV

Debug Manager

3. Product Architecture

Intel has launched the 4th generation Intel Atom processor, code-named “BayTrail”. This latest Atom processor is a multi-core system-on-chip (SoC) that integrates the next generation Intel® processor core, graphics, memory, and I/O interfaces into one solution. It is also Intel’s first SoC which is based on the 22 nm processor technology. This multi-core Atom processor provides outstanding computing power and is more power efficient compared to its predecessors. Besides latest IA core technology, it also provides extensive platform features, such as graphics, connectivity, security, and sensors, which enable developers to create software with unlimited user experiences.

BayTrail Block Diagram

Figure 21. BayTrail Block Diagram

BayTrail Features

The following table lists the RVP features and its implementation details:

Table 21. BayTrail Features

Feature

RVP Implementation

Memory

DDR3L SODIMM – 2 Channels

1333MT/s : Entry Desktop; 1066MT/s: Entry Notebook

2GB, Expandable up to 8GB

Display interfaces

HDMI/DP EV options on DDI0 (default DP)

eDP/DP options on DDI1 (default eDP)

VGA port on Back panel

SATA – 2 PORTS

One Cable Connect Port

One Direct Connect Port

Clocks

Bay Trail-M/D Supports Internal Clocking for all peripherals

USB 2.0 – 4 PORTS

Supports 1 Back panel Port and 2 Front Panel Ports

1 USB2.0 Port from USB HSIC terminated to Back panel

Supports USB Webcam through rework option

Supports MPCIE, Sensor Hub, NGFF connections through reworks

USB 3.0 – 1 PORT

Supports 1 BP port which can be used as USB2.0 port as well

Audio

On board Realtek* HD Audio Codec

PCI-e

1x4 Slot (supports 1x1 and 1x2 configuration)

2x1 inline slot

One on board Mini PCI-e Slot

On Board option for 3G SIM support through M-PCIE and NGFF

SPI

Socketed 8MB SPI chip for BIOS on Bay Trail-M/D SPI

In-system programming through Dediprog* Programmer

Sensors

Sensors supported over the Sensor AIC.

Sensor connect to Sensor Hub over I2C and Sensor Hub to SoC over USB

Sensors:

SAR proximity

Pressure/Altimeter

Ambient Light Sensor

Gyro

Magnetometer/e-Compass/Accelerometer

Onboard thermistors for temperature measurement

NGFF support

Wi-Fi + Bluetooth*

WWAN

Touch Screen

Supports Multi Touch Capacitive sensor from Atmel* mXT1664S

Embedded Controller/

Keyboard Controller

Nuvaton* NPCE985EA0DX

Serial port on board

PS/2 keyboard and mouse connectors

Scan matrix keyboard connector

Programming option through Dediprog*

SPI flash 16MB Numonyx* M25PX16-VMW6TG

EC

EC on LPC bus for Mobile Validation

LPC Slot

Allows testing of alternate KSC and SIO solutions and TPM

Modules

LDC2

LDC2 for Legacy Support on Entry Level Desktop Validation

Power

Configurable for mobile or desktop modes (AC Brick, Battery &

ATX mode)

Supports 2SXP battery

Processor IMVP7.0 regulator with SVID support for CPU Core

Support IMVP7.0 regulator (1+1 controller with Core) for Gfx Core

PORT 80

On board Port80/debug display

TPM

TPM header on board

Socket

Support Bay Trail-M/D SoC Socket

Key Validation

Features

60 Pin On board XDP

On board Port 80 display support

LED indicators for major System States

Power Measurement resistors

On board DDR3L Thermal Sensors and SODIMM thermal sensor support

4-Pin Fully Controllable Fan (MPI socket Fan) driven on EC

LPC side band header

SMBUS support

Thermal Tools supported for Bay Trail-M/D

Thermal Monitoring

Virtual Battery switch for battery mode while running in EC

Lid Switch for emulating Lid close scenario for Gfx driver validation

PCB Design

Halogen free PCB

8-Layer PCB with 6L core routing

Mini-ITX core design for Entry Desktop

ATX mounting holes for chassis support

Dimension

12.6 x 10.2 inches

BayTrail – Valley View SOC

Intel announced its latest Atom processor, BayTrail which is available in both dual and quad core and is based on Intel’s latest 22-nm processor technology. Many improvements have been made to BayTrail M/D.

The latest BYT-M C0 silicon is based on C0 specs/configuration. Top to bottom BYT-M sku will likely be on DDR3L 1333. No more DDR3L 1066. BYT-M Pentium QC (Top Bin) will support high resolution panel eDP 32x18 @ 60Hz

All below QDFs are supported in BYT platform. QDF suggested by the program to PSTV for validation, is highlighted in BOLD in the table below.

Table 22. BayTrail Variants for Android

QDF being used now is highlighted in BOLD in table below.

Segment

SKU

Processor Number

X/C-Class QDF

CPU

TDP (SDP)

LFM (MHz) /

HFM Core

Gfx Freq

Gfx Fmax

(W)

Ratio

(GHz)

(MHz)

@Vmin (MHz)

BYT-M (eNB)

Pentium QC

N3530

Y7XG / Y7CG

4

7.5 (4.5)

500 / 6

2.17 / 2.58 (B)

313 / 896 (B)

646

Mid-Celeron QC

N2930

Y7XH / Y7CH

4

7.5 (4.5)

500 / 6

1.83 / 2.17 (B)

313 / 854 (B)

646

Entry-Celeron DC

N2830

Y7XJ / Y7CJ

2

7.5 (4.5)

500 / 6

2.17 / 2.42 (B)

313 / 750 (B)

646

Fanless-10" DC (new catch-all)

N2807

Y7XK / Y7CK

2

4.3 (3.0)

500 / 6

1.58 / 2.17 (B)

313 / 750 (B)

646

BYT-D (eDT)

Pentium QC

J2900

Y6XD / Y6CD

4

10

1333 / 16

2.41 / 2.67 (B)

688 / 896 (B)

688

Celeron QC

J1900

Y6XE / Y6CE

4

10

1333 / 16

2.0 / 2.42 (B)

688 / 854 (B)

688

Celeron DC

J1800

Y6XF / Y6CF

2

10

1333 / 16

2.41 / 2.58 (B)

688 / 792 (B)

688

Figure 22 Valley View SoC

Key Enhancements:

Quad Core Atom

Graphics: Gen7 4 EU GFx DX11

Enhanced media decode

LPDDR3 1067 memory : 2 x 64b

Native support for eDP (support upto 25x14)

Enhanced audio DSP/capability

eMMC 4.5 in B0

Higher perf. IOs

1 x USB3 (xHCI)

2 x USB (HSIC) + 3 USB 2.0 (EHCI)+ 1 ULPI Port

I2C speed upto 3.4MHz

Potential Customer H/W Configurations – Different from POR

Table 42 Customer H/W Configurations

Platform Peripherals

Component Name

Vendor Name

Part Number

Possible choice of Customer

Touch

Touch controller

Atmel

mXT3243

Sharp

WWAN

Not Supported on BayTrail RVP platform

Gobi 5000 : Sierra

WiFi

WiFi+Bluetooth

Modem: Broadcom

Module: Mitsumi

BCM43241 (2x2 ABGN)

DWM-W095 (SIP)

Intel Wilkins Peak2 AC 2x2

Bluetooth 4.0 Mini PCIe M.2

Or

BCM43241 (AW-NB136NF)

Modem: Intel

Module: Intel

Intel 7260HMW (2X2 AC)

CWS

GPS

Silicon: Broadcom

BCM 4752

U-blox8

(exclusive WWAN module)

CWS

NFC

None

None

Gadget Option : TBD

Camera

Back camera

Not Supported on Bay Trail RVP platform

Module: Bison

ISP: RTS 5830

Sensor: Omnivision OV5640

Camera

Front camera

Chicony

CNFCH54

Module: AzureWave

ISP: RTS 5829

Sensor: SONY IMX-132

Camera

Front camera

Chicony

CNFCH1821

Audio

Audio Codec

RealTek

ALC5642

Conexant : CX20751

Audio

Audio Codec

RealTek

ALC262VD2

Sensor

HW Sensor Hub

ST

STM32F103CBU6 (I2C/USB)Atmel: ATUC256L4U-D3HT (I2C)

Atmel

IO

µSD Cardreader Socket

???

O2 Micro

µSD Cardreader Socket

None

NoneO2Micro : OZ620FJ1

IO

Ethernet NIC (USB)

RTL8152B-CG (Gadget Option)

RTL8253 (Dock Option)

IO

Barcode Reader

??

Intermec EA30 (Gadget)

IO

Magnetic stripe Reader

Magtek (Gadget)

IO

Super IO

SMSC : SIO1028-JZX-TR (Dock Option)

IO

RS232C

Dock Option

IO

USB 3.0 Controller

Android Architecture

Following figure illustrates the Android architecture and interfaces available generic for BayTrail RVP Platforms:

Figure 23 Android Architecture

Intel Optimizations to the Android Software Stack

Android is Google’s open source Linux-based software stack developed for mobile phones and tablets.

Intel has introduced many optimizations to the Android software stack for performance enhancement. Developers can create apps with snappy performance, smooth, and fluid user experiences.

Optimization includes:

Improvements that are made to ensure Dalvik apps run well on Intel processors, by adding Houdini layer into middleware

Tools for NDK developers to compile native code (C/C++) for x86

Optimizations to new web technologies such as HTML5 and Javascript

Performance enhancement to Dalvik VM

Optimizations to core libraries and the kernel by contributing to AOSP

Device drivers that are validated and optimized for the x86 power and memory footprint

Kernel is changed to 64-bit, user-space is not changed and remains 32-bit (Dalvik VM is maintained as default 32-bit)

4. Test ScopeValidation Overview

PSTV-Android Team owns the System Validation for BayTrail Platforms. The scope includes System test, Stress test, Performance KPIs, Power KPIs, MTBF, power cycling, Certifications (Android CTS, Android XTS, Android CTS Verifier),. The main validation vehicle for the team is Bayley Bay RVP boards with Valley View SoC on Kit Kat distribution of Android. It shall cover the POR components for Android platform.

Major Deliverables from the Organization are the following:

weekly release of Platform Validation Report (PVR)

weekly release of BKC

weekly bug scrub– Critical/High defects per ingredient

Contents and detailed delivery schedule of Platform Validation Report and BKC are discussed in sections below.

System Validation Scope

The following validation categories for any domain/ingredient at a platform level are covered in this test plan:

Table 31 Platform Validation in Scope

In-Scope

Out-of-Scope

Use Case tests that target application layer of Android

Kernel layer, middleware layer, Silicon or component level validation (SVG SV and CV teams)

Platform Acceptance Test that targets basic functionality and critical system-level functionality

Component-level ingredient validation (All Ingredient SW teams)

System functional tests based on requirements

HW validation (EV, Mechanical, Thermal)

System level certifications (Android CTS, Android XTS, CTS Verifier)

Compliance tests at device/component level (BT, HDMI, USB, Wi-Fi, Wi-Di)

Inter-op and coexistence that utilizes multiple dependencies with shared capabilities and user-experience testing

3rd Party Peripherals IOT (BT, USB, Wi-Fi, Wi-Di)

3rd-party Application Compatibility testing (top-most popular, apps known to provide issues, list not more than 200 applications)

Extensive test suite of Houdini applications

Platform power measurements for P1 workloads

Component Level power measurement (AEA)

Platform performance KPI measurements (industry-standard benchmarks and Responsiveness)

Performance KPI testing targeted for competitive Assessment, thermal measurements

Usage of Android tools and utilities to test the platform (tools such as DDMS, monitor, Hierarchy-viewer, cpueater, SOCwatch, ITP)

Coverity, LTP, gcov/lcovITP/Lauterbach

System level stress tests that overloads components, iterative testing, long-duration testing, power cycling, MTBF, concurrent stress tests that overload multiple components simultaneously under CPU and memory load conditions

Environmental Stress Testing (PRST)

End-user level security testing

Hardware-level security testing

User Experience verification - Use proper metrics to objectivity predict the perceived quality of key aspects of the user experience

Non-POR configuration

Testing on single OS (Android)

Dual-OS (Virtualized or toggled boot approach)

5. Test StrategyTest content strategy

PSTV Validation team will focus on the system level validation for applicable RVP platforms. There will be leverage of previous generation of test content for BYT platform for the applicable legacy features.

· New test cases will be added for new features specific to feature mentioned in Requirement Document for Android on Intel architecture based platform.

· Based on leverage and reviews with other groups (SSG, MCG) and stakeholders, tests will be optimized for legacy features.

· Top focus areas for system validation: for improving test content and coverage which will help in qualifying platform health: (based on analysis of BYT M/D Android 32-bit JIRA bugs and BYT M/D Android 32-bit customer bugs, existing gaps) on these area:

· Android system

· Bluetooth

· Camera

· power management

· security domain

· Wi-Di

· user experience

Test coverage strategy

Test coverage will be measured against platform requirements and PSTV categories mapped for each requirement

System is divided into following components: Android Framework, Applications, ART, Audio Codecs, Bluetooth, Browser, Camera, CTS, Widevine DRM, Ethernet, Power Management, Firmware/BIOS, Gfx, HDMI, Flashing/Installer-Bootloader, Kernel, OTA Updates, Security, Sensors-I/O, Storage, Touchscreen, USB, Video Codecs, WiDi, WiFi

Test coverage can be said to be good, if each domain has test cases in our defined set of PSTV category of system test execution, which will cover domain and all its dependent components.

Domain

Components on which domain depends (shared component)

Android Framework

Applications

Kernel

Power Management

Gfx

Security

ART

Applications

Kernel

Audio Codecs

Audio Codecs

Bluetooth

HDMI

Storage

WiDi

Bluetooth

Storage

Browser

Gfx

Kernel

Storage

WiFi

Camera

Audio Codecs

Applications

Storage

DRM

Applications

Browser

Firmware/BIOS

Security

WiFi

Power Management

Applications

Kernel

Firmware/BIOS

Kernel

Storage

Security

Gfx

Kernel

HDMI

Audio Codecs

Gfx

Security

Flashing/Installer-Bootloader

Firmware/BIOS

Security

Kernel

Flashing/Installer-Bootloader

OTA Updates

OTA Updates

Power Management

Firmware/BIOS

Flashing/Installer-Bootloader

Security

WiFi

Security

Firmware/BIOS

Flashing/Installer-Bootloader

Kernel

Sensors-I/O

Firmware/BIOS

Flashing/Installer-Bootloader

Kernel

Storage

Touchscreen

Storage

Kernel

Security

Tools

Kernel

Security

Sensors-I/O

Storage

Touchscreen

Gfx

Kernel

Sensors-I/O

USB

Applications

Power Management

Security

Sensors-I/O

Touchscreen

Video Codecs

Applications

DRM

Gfx

Kernel

Storage

WiDi

Gfx

Kernel

Storage

Video Codecs

WiFi

WiFi

Kernel

Test execution strategy

· Of total applicable tests in scope for release, P1 should be 20%, P2 should be 40% and P3 should be 40%. These values are provided for guidance, as the driving factor to optimize the time and to get efficient results in weekly validation cycle

· During any weekly release, execute not more than 60% of total applicable tests. After two weeks of execution, test execution will have covered 120% of total tests

Test automation strategy

· Improve upon automation of tests. Target is to have following set of automated tests by end of Q2:

· 100% for PAT tests

· 100% for all Concurrent stress tests

· 30% for system tests

Note: There can be semi-automation in place. Need focused approach for automation. Automation framework under use must be supported on all platforms (ABT build, MCG build)

Test categories

BYT test content is divided into following test categories:

Platform Acceptance Tests: This is the test suite to accept an ingredient to BKC. These tests focus on sanity checkout on basic functionality of the platform. PAT testing is bare minimum test to ensure the ingredient driver/FW software to meet the PSTV platform validation acceptance criteria before the platform validation starts. Large portion of the PAT will be automated tests when the system reaches minimum stability, and those tests that require human judgment will be run manually.

Ingredient tests: These tests will include:

functional tests: focused on functionality of particular ingredients/features

negative tests: testing for failure under certain conditions

System tests: These tests will include:

Compatibility/coexistence: Making sure that multiple components without any shared dependencies can function together

Interoperability: Utilizes multiple capabilities with shared dependencies. This can be classified as a feature having cross interactions that impact  Functional (impacting any attribute like Power, Performance) or Hardware or Software Areas

Usability/use-case: Verify the flow an end user would perform to achieve a certain result

Responsiveness tests: User perceived performance and is more focused on testing the latencies involved in ON/OFF transitions, App response time, IO Response times

Performance KPIs: Measure the platform performance indices industry-standard Android benchmarks and system level responsiveness/latency measurement

Power KPIs: measure the power consumption

Power tests: Measure KPIs including power consumption, tests planned around suspend-resume, reboot/shutdown

Platform Stress Tests: Under stress testing, tests will validate the platform stability for extended hours/iterations with extra load in background, power-cycling tests, MTBF tests and stress tests with added CPU load and RAM load in background. Concurrent stress tests will include multiple sub-systems being used simultaneously and in continuous loop for definite duration/iteration. Stress testing will be executed using applicable automation framework in fully automated environment.

Certifications: Certification testing includes Android CTS, Android XTS, Android CTS Verifier

Exploratory: session is a tool to uncover the bugs in platform and not by the traditional method of testing

User Experience Verification (KPI): Identify key focus areas for UX as per UX Pillars and critical use cases to verify the metrics/targets

1. Introduction

3. Product Architecture

3. Product Architecture

9 Intel ConfidentialBayTrail Android Validation Test Plan

BayTrail Android Validation Test Plan Intel Confidential19

6. Weekly execution StrategyAcceptance Criteria Entry Criteria for weekly release

List of Collaterals

· Bootable image file (live.img), matching list of BIOS FW and KSC

· Silicon Stepping changes, Re-works on BYT Fab3 board, KK versions, Kernel versions

· PAT test report as executed by Integration team

· Recently integrated features

· List of non-integrated features and planned features

· List of known issues in bug database (such as Bugzilla or HSDs or JIRA) from Integration team

· Targeted PnP numbers

Exit Criteria for weekly release

Validation reports

· Validation results adhering to Milestone QRC Criteria

· If targeted number of tests cannot be executed, then the observations of failed tests will be escalated as gatekeeping issues.

Customer Milestone release notes to be verified by PSTV, to update necessary information of release details, limitations/known issues

Weekly Validation cycle

Tests are run on the weekly basis on the latest available image posted by PCCG Integration team. The build will be provided to PSTV team by Friday IST morning 08.00 A.M.

PSTV team will perform Platform Acceptance Tests (PAT) as a part of entry criteria.

Once the build is accepted, full validation cycle will be performed. Full validation cycle contains: PAT tests, Android CTS, Android XTS, performance KPIs, power cycling, power KPIs, sylstem tests, stress tests, MTBF and exploratory testing.

BKC report

BKC report will be delivered within 2 working days from receiving the build from Integration team.

BKC reports contains execution results of PAT tests, performance KPIs, subset of stress tests and subset of power cycling tests.

Platform validation report (PVR)

Platform Validation Report will be delivered within 5 working days from receiving the build from Integration team.

PVR is superset of BKC report.

PVR contains execution results of PAT, Android CTS, Android XTS, performance KPIs, power cycling, power KPIs, system cases, stress tests, MTBF and concurrent stress tests.

Figure 41. BayTrail Validation Execution Schedule and Milestone

30 Intel Confidential BayTrail Android Validation Test Plan

Validation Execution Schedule

BayTrail execution at PSTV aligns to the Platform validation milestones, including, Alpha, Beta milestone and post Beta support.

PSTV team will procure ingredient component enablement plan from ingredient teams and integration plan from Integration team at the start of project (before pre-alpha). This will help to focused approach of validation towards the sub-systems and domains.

PSTV team will ensure that complete scope of PSTV team is added into QRC criteria for measurement.

High level milestone test execution includes:

Alpha:

Objective of release: is to allow customers to start software validation, targeted subsystem testing, and driver/FW development or validation. For Alpha, platform features are expected to be integrated in BSP (Intel & 3rd party) for validation and health status check/reporting. SW is code complete for all initial POR features: Designed, Written & Unit-Level Tested. Some important limitations documented.

Validation focus: is on targeting to provide enough confidence on integrated components and progressively achieve early targets for platform and uncover early blocking issues.

In Alpha release, we intend to cover Android CTS, CTS verifier, Android XTS, functional cases, stress tests, and system tests, Performance KPIs, Power KPIs and Power Cycling.

Beta:

Objective of release: is to enable system with all POR features. SW is feature complete for all current POR features. During Beta all features are available with aim to uncover platform issues/stress or performance issues. Target performance meets and certifications are ready.

Validation focus: is to provide measurable confidence of fully stable and fully performance-ready system, ready to deliver to external customers.

In Beta release, we intend to cover Android CTS, CTS verifier, Android XTS, functional cases, stress tests, MTBF stress tests, concurrent stress tests, and system tests, Performance KPIs, Power KPIs and Power Cycling.

The following table provides BayTrail validation execution schedule and milestone details:

Table 6: Milestone Schedule

Milestone

Kit Kat 4.4 64 bit BYT RVP

Alpha

WW12

Beta

WW15

Priority of tests

Based on validation execution phases and the features enabled in the builds, we adjust which test content to be run and how many test cases to be run. The test content will be divided in to the following execution groups:

P1 tests: Critical/high importance tests: PAT test, Android CTS, Android XTS, Performance KPIs, Power KPIs, Power cycling tests, subset of system tests, subset of stress tests, MTBF tests

P2 tests: important/medium risk tests: subset of system tests, subset of stress tests

P3 tests: low risk tests: subset of system tests, subset of stress tests

Table showing break-up of PSTV test categories into priorities:

PSTV test categories

Sub-category

P1

P2

P3

Ingredient

Functional

Functional: (critical features, buggy components)

Functional: (non-critical features)

 

Negative

 

 Negative

1x1 Acceptance (PAT)

1x1 Acceptance (PAT)

 

 

System test

Compatibility / Coexistence

 Compatibility / Coexistence

Compatibility / Coexistence

 Compatibility / Coexistence

Interoperability

 Interoperability

Interoperability

 Interoperability

Usability/Use Case

 Usability/Use Case

Usability/Use Case

Usability/Use Case

Responsiveness

 

Responsiveness

 

Performance

Performance (KPIs)

 

 

Power

Power (KPIs, cycling)

 

 

Security

Security

Security

 

System stress

Stress/Reliability

Stress/Reliability

Stress/Reliability

Stress/Reliability

Recoverability

 

 

Recoverability

Cycling

Cycling (Power cycling)

 

 

MTBF

MTBF

 

 

Out of bound testing

 

 

Out of bound testing

Fault Tolerant

 

 

Fault Tolerant

Certification

CTS, XTS

Certification: (CTS, XTS)

 

Certification: (CTS verifier)

UX

User Experience

User Experience

User experience

User Experience: (Scenario (Heuristic) Testing

Table to show break-up for execution of tests for each milestone release and release between milestones:

Test category

Priority

Releases before Alpha

Releases between Alpha and Beta

After Beta

Entry criteria

As per QRC

As per QRC

Exit criteria

As per QRC

As per QRC

PAT tests

P1

100% weekly

100% weekly

100% weekly

Automated tests

P1

100% weekly

100% weekly

100% weekly

Android CTS

P1

100% weekly

100% weekly

100% weekly

Android XTS

P1

100% weekly

100% weekly

100% weekly

CTS Verifier

P3

Every odd week

Every odd week

Every odd week

Performance KPIs

(Targets from QRC)

P1

100% weekly

100% weekly

All failed tests of previous build + bug-fixes

Power KPIs

(Targets from QRC)

P1

100% weekly

100% weekly

All failed tests of previous build + bug-fixes

Power cycling

(Targets from QRC)

P1

100% weekly

100% weekly

All failed tests of previous build + bug-fixes

Ingredient tests

P1, P2

Build wk1: P1 + All failed tests of previous build + bug-fixes + new feature

Build wk2: P1 + P2 + All failed tests of previous build + bug-fixes + new feature

Build wk1: P1 + All failed tests of previous build + bug-fixes + new feature

Build wk2: P1 + P2 + All failed tests of previous build + bug-fixes + new feature

P1 + All failed tests of previous build + bug-fixes

System tests

P1, P2, P3

Build wk1: P1 + P2 + All failed tests of previous build + bug-fixes+ new feature

Build wk2: P1 + P3 + All failed tests of previous build + bug-fixes+ new feature

Build wk1: P1 + P2 + All failed tests of previous build + bug-fixes+ new feature

Build wk2: P1 + P3 + All failed tests of previous build + bug-fixes+ new feature

P1 + All failed tests of previous build + bug-fixes

UX Verification

P1, P2

Build wk1: P1 + All failed tests of previous build + bug-fixes+ new feature

Build wk2: P1 + P2 + All failed tests of previous build + bug-fixes+ new feature

Build wk1: P1 + All failed tests of previous build + bug-fixes+ new feature

Build wk2: P1 + P2 + All failed tests of previous build + bug-fixes+ new feature

All failed tests of previous build + bug-fixes

System stress tests

P1, P2, P3

Build wk1: P1 + P2 + All failed tests of previous build + bug-fixes

Build wk2: P1 + P3 + All failed tests of previous build + bug-fixes

Build wk3: P1 + P2 + All failed tests of previous build + bug-fixes

Build wk1: P1 + P2 + All failed tests of previous build + bug-fixes

Build wk2: P1 + P3 + All failed tests of previous build + bug-fixes

Build wk3: P1 + P2 + All failed tests of previous build + bug-fixes

P1 + All failed tests of previous build + bug-fixes

Exploratory

P2

Every even week (wk2, wk4, wk6…) and every Milestone

Every even week (wk2, wk4, wk6…) and every Milestone

Every weekly build

6. Weekly execution Strategy

BayTrail Android Validation Test PlanIntel Confidential40

Intel Confidential BayTrail Android Validation Test Plan84

BayTrail Android Validation Test Plan Intel Confidential83

7. Bug process

The issues encountered during the test execution are logged into bug database (such as JIRA, Bugzilla, and HSD). Logged issue is scrubbed by the PCCG integration team and assigned to the development team based on the priority set during the Sysdebug process.

For BYT 64-bit, JIRA will be used. Following is the link to the JIRA:

https://jira01.devtools.intel.com/

Figure 4.6: Android Bug Life Cycle

Bug priority levelsP1

Resolution of this defect takes precedence over other defects and most other development activities. This level is used to focus maximum team resources to resolve a defect in the shortest possible timeframe. P1 is applied “most of the time” to Critical severity defects blocking development, internal validation or external customer validation. [System hangs, crash, general protection fault, severe security vulnerability, loss or data corruption, certification failure, factory lines down -> app failure impacting overall system behavior, no workaround currently available].

Closure of P1 bugs: timeframe to resolve P1 priority defects is always immediate and may force an unplanned release to customers to resolve the defect. Target date for closure will be 5 days.

P2

Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. P2 is applied “most of the time” to High severity defects applying to functionality, features and use cases currently being integrated and targeted for next milestone release. Should typically not be shipped to customers but if it is, it can be documented. [Defect causes severe degradation to intended feature, functionality, use case, recovery may require Soft reboot, issue can be documented, workaround possible w/o impacting user significantly, development or testing can be continued].

Closure of P2 bugs: P2 priority defects are intended to be resolved by the next planned external release of the software.

P3

Resolution of the defect has precedence over resolving other defects with lesser classifications of priority. P3 is applied “most of the time” to Medium severity defects applying to functional, features and use cases currently being integrated and targeted for next milestone release. [Product fails to meet its specifications or requirements for non-key features. Workaround exists and is easily implemented; development and testing can be continued].

Closure of P3 bugs: P3 priority defects must have a planned timeframe for a verified resolution.

P4

Resolution of the defect has the least urgency to resolve. P4 priority defects may or may not have documented plans to resolve. P4 is usually applied to Low severity defects. [Cosmetics defects can be shipped to customers w/o documentation.]

Closure of P4 bugs: P4 priority defects must be fixed before project closure.

Bug Severity levels

Critical: Any defect that will block Milestone Releases, for example: Features lose unexpected/notified power off, shutdown, re-boot, Blue screen, black screen, system hang, application hang, battery loss. Bad user experience with ID, system interfaces (keyboard, touch, touchpad) and comms, acoustic and thermal. Intermittent or 100% repeatable functionality issues which cannot be workaround or errata-ed. Performance or power issues which can impact Intel brand image and Reliability issues which may cause post-shipment failures on customer side.

High: Major loss of function: Functional, bad customer experience, performance or power issues which still allow shipment to customer with reasonable (approved) workaround or errata reflected in sighting list or Dear Customer Letter (DCL).

Medium: Issue encountered, hindering work but not significantly. Issues have no serious impact on functions, features and behaviors, and bring unexpected experience. Minor ID, user experience issues; Medium issue can be shipped if workaround, errata, DCL are provided.

Low: Minor loss of function, issues do not meet above criteria. No impact to overall product quality. It is tolerable to user experience with minimal CSF support efforts. Low issue may be shipped without a customer communication.

8. Guidelines for Test case development

The test cases are developed based on test requirements, assigning each test case with domain and test categories.

Test case development will be done in Contour.

· Test requirements are present in test content database Contour. Each requirement will have two tags. One tag to denote the project. Second tag will denote one of the domains.

· Example of Contour tags for project in requirement:

· Android_BYT_64bit_PCCG_Req

· Android_BDW_64bit_ABT_Req

· Example of Contour tags for domains in requirements:

Android_Req_Android_Framework, Android_Req_Audio_and_Codecs, Android_Req_Bluetooth,

Android_Req_Camera, Android_Req_Display, Android_Req_DRM, Android_Req_Ethernet,

Android_Req_Energy_Management, Android_Req_Flashing, Android_Req_Firmware/Bios,

Android_Req_Graphics, Android_Req_Houdini, Android_Req_Images, Android_Req_Kernel,

Android_Req_OTA_Update, Android_Req_Performance, Android_Req_Power,

Android_Req_Security, Android_Req_Sensor_I/O, Android_Req_Storage,

Android_Req_Touchscreen, Android_Req_USB, Android_Req_Video_and_Codecs,

Android_Req_WiDi, Android_Req_WiFi

Table 10: Tags for requirements

Tags for requirements

Definition

Android_Req_Android_Framework

Applications, Android OS features

Android_Req_Audio_and_Codecs

Audio Encode, Decode, Codec

Android_Req_Bluetooth

A2DP, AVRCP

Android_Req_Camera

Camera

Android_Req_Display

HDMI, eDP

Android_Req_DRM

Widevine DRM

Android_Req_Ethernet

Ethernet

Android_Req_Energy_Management

Charging, battery

Android_Req_Flashing

Flashing

Android_Req_Firmware/Bios

EC, BIOS

Android_Req_Graphics

2D/3D, Graphics

Android_Req_Houdini

3rd-party applications

Android_Req_Images

BMP, JPEG, GIF, imaging

Android_Req_Kernel

Kernel

Android_Req_OTA_Update

OTA updates, FOTA

Android_Req_Performance

Performance KPIs

Android_Req_Power

Power measurements

Android_Req_Security

Security

Android_Req_Sensor_I/O

Sensor I/O, touchpad

Android_Req_Storage

Storage, hard disk, SD card

Android_Req_Touchscreen

Touchscreen

Android_Req_USB

USB storage, logging over USB

Android_Req_Video_and_Codecs

Video encode, decode, codecs

Android_Req_WiDi

Wi-Di

Android_Req_WiFi

Wi-Fi

· Tests are present in test content database Contour. Each test will have two tags. One tag to denote the domain (Audio, BIOS, Camera, Flashing, Video, and such). Second tag will denote one of the PSTV test categories.

· Example of Contour tags for PSTV test categories in test cases:

· Android – Compatibility/Coexistence

· Android – Fault Tolerant

· Android – Functional

· Android – Negative

· Android – Performance

· Android – Power

· Android – Recoverability

· Android - Responsiveness

Example of Contour tags for domain in test cases: Android-BAT, Android_Conn_BT, Android_Conn_NFC, Android_Conn_Ethernet, Android_Conn_Wifi, Android_Kernel,

Android_MM_Audio, Android_MM_Camera,

· Android_MM_Connected&ProtectedMedia, Android_MM_Graphics,

· Android_MM_Graphics_3D, Android_MM_Graphics_Core,

· Android_MM_Graphics_Display, Android_MM_Video, Android_IO_FileSystem,

· Android_IO_PersistentStorage, Android_IO_RemovableStorage, Android_IO_Sensor,

· Android_IO_Touch, Android_IO_USB, Android_System_Android,

· Android_System_Debug&Trace, Android_System_EnergyMgmt,

· Android_System_Fault_Recovery, Android_System_Fault_Tolerant,

· Android_System_KPI, Android_System_PowerMgmt, Android_System_Responsiveness,

· Android_System_UX, Android_System_PUPDR&Flashing, Android_System_Security,

· Android_System_VolatileMemory

Table 11: Tags for domains for test cases

Tags for test cases

Definition

Android_MM_Audio

Encode, Decode, Codec,

Android_MM_Video

Encode, Decode, Codec,

Android_MM_Camera

Capture, record

Android_MM_Graphics_Display

HDMI, Wi-Di, eDP, DP, HDCP2.0

Android_MM_Graphics_3D

Game

Android_MM_Graphics_Core

GPU

Android_MM_Graphics

Benchmarks

Android_Conn_BT

A2DP, AVRCP, BLE

Android_Conn_Wifi

Wi-Direct, FTP, Streaming, RTSP, HTTP,

Android_Conn_NFC

File-Transfer

Android_IO_USB

USB HUB, Headset/Speaker, HID

Android_IO_FileSystem

EMMC, SD HD, TP, File Transfer, Pen-drive

Android_IO_Touch

Touch Pad, Touch panel

Android_IO_Sensor

All Sensor

Android_System_PowerMgmt

Power states, wake up source. Governor, clocking

Android_System_EnergyMgmt

Battery, Charging, Temp

Android-BAT

Smoke test

Android_System_KPI

Performance, Power KPI

Android_System_Responsiveness

Latency, Boot Time, Response Time

Android_System_Recovery

App crash, System crash, kernel Crash.

Android_System_UX

Android  Setting, notification, non- measureable responsiveness

Android_System_Misc

Apps install, BIOS configuration setting.

Android_System_Debug&Trace

Serial Log, DDMS, Android monitor, SD-card logging

Android_System_PUPDR&Flashing

Flashing, OTA, Fastboot, ADB-sideload, Anti-Theft, rollback.

Android_System_Security

Secure Boot, Lock/unlock, Face- detection, DRM, Wide-Vine,

Android_MM_Connected&ProtectedMedia

Wi-Di , HDCP

Android_System_VolatileMemory

Low memory Test, All memory leak test.

9. Platform Test Overview Requirement

Use case requirement for BYT 64-bit split across domains is provided in the below sections. For 64-bit Kitkat, requirements are provided by PCCG Integration team. PCCG tracks the requirement in JIRA for development purposes and in Contour for mapping test requirements to test cases.

Current decision has been made to follow MCG-baseline based tree as integration tree, for 64-bit KK.

Requirement for BYT 64-bit KK can be referred from:

https://jira01.devtools.intel.com/secure/Dashboard.jspa?selectPageId=15923

AndroidOverview

Android is an operating system based on the Linux kernel, and designed primarily for touchscreen mobile devices such as smartphones tablet computers and PC, initially, developed by Android, Incorporate. User interface of Android is based on direct manipulation, using touch inputs that loosely correspond to real-world actions, like swiping, tapping, pinching and reverse pinching to manipulate on-screen objects. Internal hardware such as accelerometers, gyroscopes and proximity sensors are used by some applications to respond to additional user actions, for example, adjusting the screen from portrait to landscape depending on how the device is oriented. Android allows users to customize their home screens with shortcuts to applications and widgets, which allow users to display live content, such as emails and weather information, directly on the home screen. Applications can further send notifications to the user to inform them of relevant information, such as new emails and text messages.

Table 7.1 Android SW Stack - BSP Deliverable

ConnectivityOverview

Connectivity is the ability to link to and communicate with other computer systems, electronic devices, software, or the Internet. WLAN and Bluetooth are major connectivity features in BYT-M/D.

WLAN and Bluetooth solution is based on Intel’s Wilkin’s Peak 2 combo device that is integrating with BT, WLAN and FM features on the same device.

Note: In this module, the FM feature is not enabled.

System validation tests for BT and Wi-Fi includes Stress scenarios and Interaction scenarios with other modules. Stress tests validate stability of connectivity module. Interaction scenarios are designed to cover the interaction of Wi-Fi and Bluetooth modules along with other modules such as Multimedia (Audio/Video/Camera), Power Management (S3/S5), Streaming and so on. Also, interaction within connectivity module, that is, BT and Wi-Fi co-existence tests are planned.

Wi-Fi

BayTrail Android platform supports 802.11 Wi-Fi feature for different bands and modes enabling advanced use cases such as Web Browsing, Video/Audio call with Skype application, FTP content upload/download and audio/video streaming ( YouTube, …), Wi-Fi Direct (Wi-Fi Display, File Transfer, …).

Standard 2.4GHz (802.11bgn) and 5GHz (802.11a/n) bands shall be supported with in addition all security (802.11i, WPA2, EAP-xxx…) and quality of service (802.11e, WMM…) required to achieve a good end user experience.

BT

BayTrail Android platform supports Specification version BT 4.0 +EDR.

As WLAN and BT are integrated into the same component, BT/Wi-Fi coexistence is managed internally.

GPS, NFC, WWAN are not supported for BYT Android platform.

Graphics/Media/VideoOverview

Graphics module is categorized into – Display, Core, 3D, and multimedia. System validation tests are designed to cover the interaction of each of these components with other modules such as BT, Wifi, Data, sensors, Widi, power management, and so on. Stress tests or long duration tests are run to validate the reliability of Graphics stack. Graphics performance tests are also run using some standard benchmarks.

Imaging / CameraOverview

Camera is integrated on the platform and mainly focus area for validation is the compatibility scenario with different chat application, covering interoperability feature with Video recording etc.

Audio Overview

Focus will be to cover the entire support Audio format with all the different interoperability, compatibility scenario with BT, USB and on board audio speaker.

EMOverview

As part of battery energy management, we check how platform behaves when it run on battery. We validate battery capacity, voltage, rated current etc. We verify whether platform detects AC charger or not when connected. We execute various use cases like running audio, video, file transfer, installation when battery is low. Also, validate battery fuel gauge information.

SensorOverview

Sensors are classified as Human Interface Devices (HID). The sensor hub currently uses STM. Intel provides sensor solution firmware. The sensors are listed below:

Physical Hardware:

Accelerometer

Gyrometer

Magnetometer

Ambient Light Sensor

Barometer(Air Pressure)

STMicro Sensor Hub

Fusion Sensors:

Simple Orientation

The sensor hub is a microcontroller containing a 32 bit embedded ARM core. The sensor hub attaches directly to the PCH up stream and to multiple physical sensors downstream.

USB & StorageOverview

BYT M/D provides extensive I/O support. Functions and capabilities include Integrated Serial ATA host controller, xHCI USB controller. Flexible I/O - allows some high speed I/O signals to be configured as PCIe*, SATA or USB 3.0.

System Test SecurityOverview

Security module is categorized into – Provisioning, Secure Boot & Update and Recovery. System validation tests are designed to cover the Functionality and interaction of each of these components with other modules such as power management, NFC, Modem, DRM, Wide-Vine etc. XTS test are run on for checking the Provisioned Data. Also recovery test are done to check the security stack

System Test – Usability/UsageOverview

Usability test will mainly focus how the end user is going to use the system and will try to cover the scenario in the same way, how user expects the system to behave. Use cases goes in line with what a person thinks, feels, and perceives, before, during, and after using a product or service.

System InteroperabilityOverview

Interoperability Test cases will Utilizes multiple capabilities on the platform with shared dependencies. This can be classified as a feature having cross interactions that impact Functional (impacting any attribute like Power, Performance) or Hardware or Software Areas

System CompatibilityOverview

Compatibility test cases will making sure that multiple components without any shared dependencies can function together.

Additionally, top 100 popular applications will tested as part of this testing. This list will contain applications which are known to give issues in Houdini layer.

Sr. No.

Test Case Name

1

Download and use Android application 0xbench

2

Download and use Android application Android Terminal Emulator

3

Download and use Android application Angry Birds

4

Download and use Android application Angry Birds Space

5

Download and use Android application Ant Killer - Best Ant Smasher

6

Download and use Android application AntiVirus Security Free

7

Download and use Android application AnTuTu Benchmark

8

Download and use Android application Benji bananas

9

Download and use Android application CF-Bench

10

Download and use Android application Chrome Browser

11

Download and use Android application Cut the Rope

12

Download and use Android application Defender

13

Download and use Android application Defender II

14

Download and use Android application Dr.Driving

15

Download and use Android application ES File Explorer File Manager

16

Download and use Android application Facebook

17

Download and use Android application Facebook Messenger

18

Download and use Android application forfone: Free Calls & Messages

19

Download and use Android application Fruit Ninja

20

Download and use Android application Go launcher EX

21

Download and use Android application Hill Climb Racing

22

Download and use Android application Kindle

23

Download and use Android application KingSoft Office

24

Download and use Android application LINE

25

Download and use Android application LinkedIn

26

Download and use Android application Linpack for Android

27

Download and use Android application MapQuest: Maps, GPS & Traffic

28

Download and use Android application MX Player

29

Download and use Android application MyTrails

30

Download and use Android application NenaMark1

31

Download and use Android application NenaMark2

32

Download and use Android application Opera Mini Browser

33

Download and use Android application Osmos Demo

34

Download and use Android application PhoneFlicks-Netflix

35

Download and use Android application Quadrant Standard Edition

36

Download and use Android application Quake 3 Engine- Zombie

37

Download and use Android application Real Racing 3

38

Download and use Android application ScummVM

39

Download and use Android application Shazam

40

Download and use Android application SketchBook Mobile Express

41

Download and use Android application Skype

42

Download and use Android application SpeedMoto

43

Download and use Android application Speedtest.net

44

Download and use Android application Spotify

45

Download and use Android application Subway Surfers

46

Download and use Android application Talking Tomcat 2

47

Download and use Android application Tango Video, Voice & Text

48

Download and use Android application Temple Run 2

49

Download and use Android application TuneIn Radio

50

Download and use Android application Twitter

51

Download and use Android application Ustream

52

Download and use Android application Vellamo Mobile Benchmark

53

Download and use Android application Viber

54

Download and use Android application WhatsApp Messenger

55

Download and use Android application Real Racing 3

56

Download and use Android application Perfect Piano

57

Download and use Android application Enemy Strike

58

Download and use Android application Castle Clash

59

Download and use Android application Tower of saviors

60

Download and use Android application LINE Pokopang

61

Download and use Android application Despicable Me

62

Download and use Android application Nun Attack Run & Gun

63

Download and use Android application Line MASS FISHING

64

Download and use Android application Toy Claw 3D Free

65

Download and use Android application Deer Hunter 2014

66

Download and use Android application LINE MapleStory Village

67

Download and use Android application LINE POP

68

Download and use Android application 3D Airplane flight simulator

69

Download and use Android application Pet Pop

70

Download and use Android application Happy Fish

71

Download and use Android application Marble Legend

72

Download and use Android application Jewels Star 2

73

Download and use Android application Pet Rescue saga

74

Download and use Android application Gun strike : shooting

75

Download and use Android application LINE I LOVE coffee

76

Download and use Android application Fishing superstars

77

Download and use Android application LINE Bubble

78

Download and use Android application Car Wash & Design

79

Download and use Android application Zombie Tsunami

80

Download and use Android application Clever Block 2

81

Download and use Android application Crazy Parking Car King 3D

82

Download and use Android application Bubble Mania

83

Download and use Android application Pancake Tower

84

Download and use Android application The Blockheads

85

Download and use Android application Fast Racing 3D

86

Download and use Android application Diamond Dash

87

Download and use Android application Cytus

88

Download and use Android application Richman4

89

Download and use Android application Clash of Clans

90

Download and use Android application Hello Kitty café

91

Download and use Android application Little commander WW2

92

Download and use Android application F18 3D Fighter

93

Download and use Android application Coin Dozer

94

Download and use Android application The Mansion:A Puzzle of Rooms

95

Download and use Android application angry piggy

96

Download and use Android application Spot The Difference

97

Download and use Android application Ninja Revenge

98

Download and use Android application Dead Trigger 2

99

Download and use Android application Adobe Reader

Responsiveness

Responsiveness is user perceived performance-. It is the specific ability of a system or functional unit to complete the assigned tasks within a given time.

Benefit of Responsiveness:

Diagnose potential problems

Diagnose inconsistent behavior.

Diagnose HW/SW Poor design.

Meeting end user expectation.

Diagnose compatibility issues of the system.

Meeting MSFT/Google/Apple certification requirements

Power & Performance

5 power KPIs are executed on BYT. We measured power numbers using NI-DAQ equipment and compared the total platform power with target values on various WW android builds. KPI details are given in the following table:

Standby

Standby as a feature is the single most indicator of battery life for a mobile system. The device is expected to enter standby when left un-attended or during a display timeout or a device "lock" by short-pressing the power button. The motivation of the OS is to make sure all components operate in lower/power down modes to achieve the least power consumption during standby. Over an average complete day use case, standby is the majority duration on the system. The OS should strive to go to the lowest power state irrespective of the Wireless Connectivity (BT, WLAN...). The power consumption of the platform is dependents on what components need to be always on in order to wake up the system and allow usage. It also depends on the frequent interruptions by various services and user space daemons requiring system sanity/stability etc. A platform can enter into various modes of standby depending upon the type of wireless connectivity (WLAN, BT, GPS) or network association and data connection activity.

We have collected power numbers when platform is in sleep (s3) mode by setting display time out to 15sec. The fan and corresponding LEDs should be off during S3.

Idle Display ON

S0-Idle-Screen-On KPI is a base KPI for all other active KPIs. So meeting the target on this base KPI is most important and prerequisite for all other active KPIs to meet the target. Also this is good KPI to see Pre silicon to post silicon SOC power correlation. Change the wallpaper to rainbow colored wallpaper and start measuring power. The brightness must be set to 200Nits when DPST is disabled. All the readings to be taken after DPST are enabled.

Local Audio Playback

This scenario is typical for MP3 audio playback from local storage with headset connected and display screen turned off.

Video Playback

Android Video Playback KPI is used to measure power for Local Video playback feature 1080p and 720p stream on Android where content is stored locally on SSD. In this UC, the display panel is always turned on where video data is shown at 60fps and audio playback is heard on headset.

Browsing over Wifi

This KPI measures browsing workload power consumption. The Browsing (5.8) workload runs for 7 minutes and contains 14 web pages. The complete workload is placed at webserver for browsing through native browser.

Figure 48. Power KPI Targets Table -1

Power KPI

Target(mW)

Static Content on Display

3887

Video playback

6246

Music playback via wired headset MP3

1230

S3 Flight mode: All Radio disabled

166

Browsing over WIFI

6473

Performance Overview

Performance Tests are categorized for – CPU, Memory, Browser & Graphics. System validation tests are designed to cover the Performance for each sub modules as categorized. Benchmark test are run in-order to assess initiator’s performance with those of target values.

Some Apps that run mathematical calculations and spit out some numerical rating of the CPU (probably measured in M-Flops or floating-point operations) are measuring the capacity of the CPU to do work. There are few apps that measure just memory performance because it is partially dependent on the chipset, but this data is often included in comprehensive benchmarks. For GPU related there is app that checks the 3D environments. This often involves tests at multiple resolutions and with different engines. GPU benches may also test 2D drawing performance. So much of what goes on inside an Android device requires the system to interact with the NAND flash storage, for which memory Apps are required which measured on I/O per-second and storage speed. Lastly we have Browser Benchmarks which test the Browser with specific website and workloads, this kind of test is heavily dependent on CPU performance and screen resolution, but it gives data about a specific use case.

Table 47 Performance KPI Targets Table -2

SL

KPI

Target

Margin

Units

%

CPU

1

EEMBC

1250

3

score

2

COREMARK

29500

3

score

3

Caffeinemark

24500

3

score

4

Dhrystone-BENC

29450

3

score

5

ISPEC 2k (SPEED)

1230

3

score

6

SpecIntRate 2k (RATE) 4T

47

3

score

7

0xBenchmark(Linpack 4T)

573

3

score

8

AnTuTu 3.3.2 CPU Int

7800

3

score

9

AnTuTu 3.3.2 CPU Float

10500

3

score

Memory

10

Stream-Copy

9000

3

MB/sec

11

Stream-Scale

9000

3

MB/sec

12

Stream-Add

9000

3

MB/sec

13

Stream-Triad

9000

3

MB/sec

14

AnTuTu 3.3.2 RAM

8800

3

score

Browser(Chrome)

15

Chrome Browsermark

225000

3

score

6

Chrome EEMBC BrowsingBench

3220

3

score

17

Chrome Sunspider

0.6

3

Sec

18

Chrome Octane

5300

3

score

19

Chrome Fish Tank - 200 Fishes

30

3

fps

Graphics

20

SmartBench2012 Productivity Index - Native resolution

8900

3

score

21

SmartBench2012 Gaming Index - Native resolution

3700

3

score

22

Quadrant 2.1 - 2D

1000

3

score

23

Quadrant 2.1- 3D

2460

3

score

24

Quadrant 2.1 - CPU

75000

3

score

25

Quadrant 2.1- Memory

12000

3

score

26

Quadrant 2.1- IO

9000

3

score

27

Basemark2.0 V1.2 Hoverjet no AA + no PP - Native resolution

60

3

fps

28

Basemark2.0 V1.2 Taiji noAA + noPP - Native resolution

60

3

fps

29

GLBenchmark 2.5.1 Ezypt HD C24Z16 Offscreen ETC1

50

3

fps

30

GLBenchmark 2.1 Ezypt Classic C16Z16 Offscreen ETC1

135

3

fps

31

GLBenchmark 2.7 T-Rex HD C24Z16 Offscreen ETC1

21

3

fps

File System

32

JIO_Sequential_Write_64KB_eMMC

85

3

MB/sec

33

JIO_Sequential_Read_64KB_eMMC

90

3

MB/sec

Response Time

34

Cold Boot Time to lock screen

30

3

Sec

System StressOverview

Stress testing will mainly focus on the Overloading the system or component and testing beyond normal operating conditions for system stability. One of the ways is to stress the system is to run the cycling test by repeating the number of iteration of a usage on the system. As part of power cycling, turn off and on the system for few iterations to check device reinitialize its configuration or recover from an unresponsive state of its critical functionality, such as in a crash or hang situation. This can be done manually were approx. 10 iteration is covered or automatically 1000 iteration/10 hrs test by running scripts or applications.

The BYT platform supports the following system states:

1.

1. S0 – Full On

1. S0i1, S0i2, S0i3 on selected SKUs

1. S3 – Suspend to RAM

1. S4 – Suspend to hard disk

1. S5 – Soft off

1.

Power cycling methods

We proposed 4 methods for S3-S0 cycling and 3 methods for S0-S5 cycling.

1.

S0-S3 cycling

Two apk files are available to run automatic suspend-resume cycles on BYT DUT. Install it on system and set the iterations as POR.

IntelAlarmClock.apk should be installed on DUT and set the alarm ring times and snooze times to 1minute. Also, set the display sleep time as 30 seconds so that system enters sleep after alarm is triggered. After 1 minute, system wakes up from sleep as alarm rings. The observations must be made during sleep state when fan should be off as well as LEDs should be turned off.

The other applications to run S0-S3 cycling is using ATT tool (Acer Test Tool). Install the tool on DUT and click “suspend-resume” tab to run cycling. The number of iterations to be defined in AATT.xml file is given along with the apk. We also need to define the sleep time xml file. The application provides information on how many cycles passed and failed.

The third method is to trigger suspend resume for few iterations by manually pressing the power button.

1.

S0-S5 cycling

The same ATT application can be used for S0-S5 cycling using reboot tab. Reboot time and number of iterations to be run should be written in xml file.

The second method is to reboot over ADB. Establish peer to peer WIFI connection with system and host PC and then connect over ADB. Run a script using adb reboot command keep in loop.

We are using a USB relay to simulate power button press and run it in loop. Wiring should be done across the power button and connect to USB relay. The platform will be rebooted based on the pre-programmed control application.

The last method is to press power button till platform shut down for a few iterations.

In all cases, the kernel logs to be analyzed for proper initialization of all components and check for any hangs or display artifacts

System Stress testsMethod 1 (with host)

In this method, the stress tests are triggered from automation framework residing on a host machine. This set-up involves executing tests over ADB. This setup helps in log collection especially in crash or hangs and aids in debugging in random hangs, automatic uploading and updating of test results on the host. This simulates the customer usage when they receive the BSP and run some tests, since the customers OEMs also use the device with host connected via USB or Ethernet.

Method 2 (without host)

In this method, the stress tests are triggered from the device itself, simulating the behavior of user with the device. This is done by launching an application that tests opens existing applications along with desired settings in the Android device. This is usually a Java application that calls the Android APIs of JB or KK.

System Stress tests are focused on Multimedia and Connectivity subsystems. Automation on I/O subsystem tests require Opto-Fidelity equipment and deployment of the same is in plan.

Stress tests based on host based framework (ACS):

Contour Test ID

Name

Priority

PSPV-PSPVTC-68867

[ACS] File transfer of 1GB file between PC and device, over ADB

P2

PSPV-PSPVTC-68868

[ACS] Consecutive - Copy file between file systems (USB pen drive -> local filesystem, local filesystem-> USB pen drive)

P2

PSPV-PSPVTC-68869

[ACS] Consecutive - Copy file between file systems (SD card -> local filesystem, local filesystem -> SD card)

P2

PSPV-PSPVTC-68870

[ACS] Consecutive - Take picture

PSPV-PSPVTC-68871

[ACS] Consecutive - Record video file

Unassigned

PSPV-PSPVTC-68872

[ACS] Consecutive - Browse web pages

PSPV-PSPVTC-68873

[ACS] Consecutive - Record audio file

PSPV-PSPVTC-68874

[ACS] Consecutive - Record video file on SATA

PSPV-PSPVTC-69714

[ACS] Consecutive - Play audio file from USB pen drive

P2

PSPV-PSPVTC-69715

[ACS] Consecutive - Play audio file from local filesystem

P2

PSPV-PSPVTC-69716

[ACS] Consecutive - Play audio file from SD card

P2

PSPV-PSPVTC-69717

[ACS] Consecutive - Play video file from USB pen drive

P2

PSPV-PSPVTC-69718

[ACS] Consecutive - Play video file from local filesystem

P2

PSPV-PSPVTC-69719

[ACS] Consecutive - Play video file from SD card

P2

PSPV-PSPVTC-69720

[ACS] Consecutive - File transfer between PC and device, over ADB

P2

Stress Tests based on non-host framework

Contour Test ID

Name

Priority

PSPV-PSPVTC-69765

Consecutive - Check WiDi detection after each S0-S3 transition

P2

PSPV-PSPVTC-75192

Consecutive - S0-S3 cycling with WiDi connected and without connecting to Wifi access point

P2

PSPV-PSPVTC-69766

Consecutive - Check HDMI detection after each S0-S3 transition

P1

PSPV-PSPVTC-69768

Consecutive - S0-S5 transition with HDMI cable plugged in.

P2

PSPV-PSPVTC-69769

Consecutive - S0-S5 transition for WiDi display

P2

PSPV-PSPVTC-69771

Consecutive - Rotate screen (physically) when Widi Miracast is connected

P2

PSPV-PSPVTC-75191

Consecutive - Widi connect/disconnect with Wifi ON but not connected to access point

P2

PSPV-PSPVTC-69762

Consecutive - Widi connect /disconnect

P2

PSPV-PSPVTC-75194

Consecutive - Hot plug and unplug HDMI cable during Video Playback

P2

PSPV-PSPVTC-75873

Consecutive - Video record and playback on Widi using CTS media stress apk.

P1

PSPV-PSPVTC-75507

Long Duration - Video playback on HDMI

P1

PSPV-PSPVTC-75511

Long Duration - 3D WebGL graphics test

P2

PSPV-PSPVTC-75513

Long Duration - Idle mode test with HDMI connected - 10 hrs

P2

PSPV-PSPVTC-75514

Long Duration - Audio playback on Widi

P2

PSPV-PSPVTC-69738

Long Duration - Video streaming from Youtube on Widi -10 hrs

P2

PSPV-PSPVTC-68875

Consecutive - Play audio file from local filesystem (e.g. SATA)

P1

PSPV-PSPVTC-68876

Consecutive - Play audio file from USB pen drive

P2

PSPV-PSPVTC-68877

Consecutive - Play audio file from SD card

P2

PSPV-PSPVTC-68878

Consecutive - Play video file from local filesystem (e.g. SATA)

P1

PSPV-PSPVTC-68892

Audio playback during alarm ringing

P1

PSPV-PSPVTC-68893

Consecutive - Take picture

P1

PSPV-PSPVTC-68894

Consectuive - Record video file

P1

PSPV-PSPVTC-68895

Consecutive - Launch viewfinder- photo

P1

PSPV-PSPVTC-68896

Consecutive - Launch viewfinder- video

P2

PSPV-PSPVTC-68897

Consecutive - Record video and playback

P1

PSPV-PSPVTC-68898

Consecutive - Switch recorder

P2

PSPV-PSPVTC-68899

Consecutive - Browse web pages

P1

PSPV-PSPVTC-68900

Consecutive - Play High resolution Youtube/video stream

P2

PSPV-PSPVTC-68903

Consecutive - Trigger Kernel Panic - system shall reboot itself

P2

PSPV-PSPVTC-75517

Consecutive - Turn ON/OFF Ethernet 10 times from UI

P1

PSPV-PSPVTC-75516

Consecutive - Remove and insert back Ethernet cable 10 times

P2

PSPV-PSPVTC-75880

Long Duration - Play Audio over BT headset (A2DP)

P2

PSPV-PSPVTC-69705

Long Duration - Idle test

P2

PSPV-PSPVTC-69707

Long duration - Graphics

P2

PSPV-PSPVTC-68901

Rotate between landscape and portrait (using application, not actual rotate H/W)

P2

PSPV-PSPVTC-68902

Monkey

P2

PSPV-PSPVTC-69680

Check huge file transfer and multiple(100) files transfer from internal memory to SD4.0 card

P2

PSPV-PSPVTC-75884

Take Picture - 5 times

P1

PSPV-PSPVTC-75885

Take Video - 5 times (10 sec)

P1

PSPV-PSPVTC-75888

open and close camera for 10 times

P1

PSPV-PSPVTC-75890

switch between camera and camcorder mode 10 times

P1

PSPV-PSPVTC-75894

scroll up/down in a gallery with 100 thumbnails

P1

ACS

ACS is an end to end system test automation framework which Support all reference platforms (with high focus on Android).it provide a way to measure platform Key performance and reliability indicators and Optimize Test Equipment utilization (Test Room with 24/7 testing).It also Provide Platform debug and logging capabilities.

ACS is extensively used for

Android developer patches gate keeping (using buildbot).

Platform daily build gate keeping in addition to TH manual testing.

FUTE and Non regression testing.

Platform reliability indicators reporting (MTBF and KRI).

Figure 47. ACS Test Scripts

Figure 47. Python Test Scripts + Xml UECmds

MTBF Scope

· Measure MTBF (mean time between failures) for Intel Android big core powered devices: Tablets and Ultra-books

· All official releases for each DUT will be subject to MTBF score calculation

Stability Indicators:

MTFF (Mean Time to First Failure)

Sum of the runtimes for all DUTs divided by number of failures encountered (only first one counted per device). It evaluates the rate of failures.

MTBF (Mean Time Between Failures)

Sum of the runtimes for all DUTs divided by number of failures encountered. It evaluates the rate of failures.

First failure definition

Android App failure:

App stops responding but recovers itself when restarted

App stops responding but works after DUT reboot

Android service failure:

Service fails and recovers itself after a time period

Service fails and only recovers after DUT reboot

Android OS failure:

OS fails to respond and recovers after DUT reboot

Test case failure:

Test over a specific feature reports failed on 1 run but passed on other runs

AT&T suggests that to stop testing when any of the following first failure types is encountered.

This should not be the case at the beginning, as this situation is expected to happen very often.

Run Time

Standard duration for MTBF calculation should be equal with the time between 2 official releases – 1 Wd(not more than 200h/DUT)

The time to iterate one stability loop shall not exceed 15 hours – this will provide valuable information about test cases reported failures over time

Lab Entry Criteria

600 hours cumulative for 6 DUT

Target values are provided as per QRC criteria

Calculation Methodology for lab entry:

Each device used for stability testing shall run a minimum time of 100 hours to be counted in the MTBF calculation.

Each device used for LE stability testing shall run a maximum time of 200 hours to be counted in the MTBF calculation.

If a reset or lockup occurs the number of hours up to the point of the reset or lock up shall be used in the LE MTBF calculation. The hours accumulated after the reset or lock up shall not be used in the LE MTBF calculation.

The 600H criteria could be summarized as 50% of the population must reach the Targeted Time of 200H without failure

Lab Exit Criteria

2000 hours cumulative for 10 DUT

Target values are provided as per QRC criteria

Calculation Methodology for lab exit:

Each device used for Final Software stability testing shall run a minimum time of 110 hours to be counted in the MTBF calculation.

Each device used for Final Software stability testing shall run a maximum time of 220 hours to be counted in the MTBF calculation.

If a reset or lockup occurs the number of hours up to the point of the reset or lock up shall be used in the Final Software MTBF calculation. The hours accumulated after the reset or lock up shall not be used in the Final Software MTBF calculation.

Reporting MTBF

Test resources

Android Automation Solution: Jenkins + ATF + Reporting tool able to display required metrics and graphs for project stakeholders

10 Android devices – DUT

Stability test suite – should contain tests able to check an equivalent number of features as the ones in AT&T stability document with an equal number of repetitive loops and specific exit criteria per test. Choose only tests that apply and add more where case.

MTBF Test Suite

MTBF stands for Mean Time between Failures. Suite of same tests is executed across devices for multiple times (Total execution time - ranging for days) and failures are recorded. MTBF value is calculated and results give overview about platform stability.

MTBF