GTAC 2014 - Test automation on an infrared set-top box

Test Automation on an infrared Set-top box

Olivier ETIENNE - Orange

What is a STB

a Set Top Box ?

Basically just a device to manage IP TV and associated services

Tuner

TNTHDD

RAM CPU

Network adapter

Drivers

Native

SDK

IR Radio

Video card

OS (Linux ou Android)

SDK

HTML

Javascript

Browser…

La STB… pour faire simple

OS (Linux ou Android)

SDK

HTML

Javascript

Browser


A Browser

(Webkit)


A Browser

(Webkit)

HTML

pages


A Browser

(Webkit)

HTML

pages

But all

interactions

made via

remote control

Many countries

FrancePoland SpainSenegal

Many versions for each countries

Maintenance Evolutive

Many devices

Sagem Samsung Airties

So many devices…

How would you test

your TV app ?

Test

Status

How would you test your TV app ?

TEST1 sprint =

Accuracy

relative to

tester

Boring

Activity

Many tests

DEV DEV DEV

• Impossible to test everything

• Not confident in what has been tested

• Non regression test are expensive

• Fear to make any change in code

How would you test your TV app ?

We need to

automate that !

Automation objectives

• Reduce test campaign duration–non regression tests

–functional tests

• Get confident in refactorings

• Use target environnement

• Make reproductible tests

Low

cost

Easy

to use

Previous experiences

• Witbe

• TAKT Engine

• STB-tester

ExpensiveHard to maintain testSensitive to UI change

• Record IR signal

• Send IR signal

• Video capture and analysis

• Expensive hardware

A different approach

Infrared

Automate, automate, but… how ?

OrangeTV

12:32

?

Raspberry pi as a mediator

IN : readIR OUT: sendIR

Infrared



Infrared

RPi



Infrared

RPi

OUT: send IR IN : regIR



HTTP

Infrared

RPi




HTTP

Infrared

RPi

IN : sendKey

IN : runSequence




HTTP

Infrared

RPi

IN : sendKey

IN : runSequence

IN : readDump




HTTP

Infrared

RPi

GET HTTP

IN : sendKey

IN : runSequence

IN : readDump




HTTP

Infrared

RPi

GET HTTP

IR Signal

IN : sendKey

IN : runSequence

IN : readDump


DOM


OrangeTV

12:32


HTTP

Infrared

RPi

GET HTTP

IR Signal

IN : sendKey

IN : runSequence

IN : readDump


DOM


OrangeTV

12:32


HTTP

Infrared

RPi

GET HTTP

IR Signal

IN : sendKey

IN : runSequence

IN : readDump


DOM

DOM


OrangeTV

12:32


HTTP

Infrared

RPi

GET HTTP

IR Signal

Synchronous

HTTP

response

IN : sendKey

IN : runSequence

IN : readDump


DOM

Pimp my Pi

Raspberry Pi

Apache

PH

P

Scripts

SH

Java

GPIO

Driver

LIRC

HT

TP

Send a key

Record a keypi4j

screen

Screenshot or dump

RPi

Start Sequence

OrangeTV

12:32

PC

(Browser)

TV

Video 1:Record an infrared sequence and replay it from firefox

http://youtube.com/v/NOeIcJZX-T0

http://youtube.com/v/NOeIcJZX-T0

From specific to

standard

Screenshot or dump

« Screenshot »

of the STB

Content

DOM

« Screenshot »

of the STB

Content

Selenium IDE

TestSuite

TestCaseSend orders

to STB

Make

assertions

on state

Video 2 :Replay a sequence from Selenium IDE

http://youtube.com/v/YwTon9peduc

http://youtube.com/v/YwTon9peduc

interne Orange44

HTTP

IN : sendKey

IN : runSequence

IN : askDump

IN : runShellCmd

IN : softReboot

IN : hardReboot

RPi

Call HTTP verbs

Make assertions

A raspberry pi as an HTTP proxy to the STB

OrangeTV

12:32

Why Selenium IDE ???

• Quite as good as Selenium

• But much simpler

– Simplified script language

– Firefox Plugin

• No need to be a developer to write a test

every night

Big pictureOn demand

Code

Repo

updateTestSuitesdeploy

updatereset

selenium

Delivery

Status

Test

Repo

Results

Results

• First prototype developed in 4 weeks

• Over 300 functional tests running every night

• VOD tests takes 3h in the night instead of 5

working days

• Team has gained

– confidence in its code

– a better image

Lessons learned

• Keep control on your test data

• Think test soon

– when you write specs

– when you design your architecture

– when you code

• Simple is better

• Everything can’t be automated

And now, …

how do you test

your TV app ?

Like this !

Questions ?

Olivier [email protected]://linkedin.com/profile/view?id=7988978@olivier_etienne

Contact

Bonus slide : What kind of interaction does the

pi provides with STB

❏ send a key

❏ send a key sequence

❏ ask a screen dump

❏ wait for a specific screen to be visible

❏ execute a shell command on the stb

❏ upload and execute a sheel script on stb

❏ put a file on stb

❏ get a file on stb

❏ get logs from stb console

❏ wait for specific log

❏ …

All these verbs are exposed through HTTP interface of the raspberry pi

Bonus slide : What can you retrieve on test failure

❏ Logs from STB Console

❏ Screen dump

Available through selenium execution report in Jenkins

Bonus slide : What is java used for

Java is used for the key sequence recorder and player.

It support a script language

KEY_PRESS

WAIT

REPEAT

:

KEY_RANDOM

DUMP_SCREEN

GTAC 2014 - Test automation on an infrared set-top box

Technology

Transcript of GTAC 2014 - Test automation on an infrared set-top box