. . . . . .

Introduction System description Building blocks Summary Discussion

”Automated Test System - build blocks”

Łukasz Majewski

Samsung R&D Institute Poland

Embedded Linux Conference EuropeDublin, 05-10-2015

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 1/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Outline

1 Introduction

2 System description

3 Building blocks

4 Summary

5 Discussion

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 2/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Section 1

Introduction

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 3/25

. . . . . .

Introduction System description Building blocks Summary Discussion

A few words about me

Embedded systems programmerUsing u-boot since 2008”Blame me” for:

Device Firmware Upgrade (DFU)Trats/Trats2 devices

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 4/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Presentation goal

Presentation goalSharing knowledge and experience after building simple and costeffective setup for automated u-boot and kernel tests with usingopen source SW and off-the-shelf HW.

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 5/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Testing our u-boot

2011: Build test infrastructure (u-boot-CI)

2014: USB DFU/USB Mass Storage test scripts in mainline2014: Boot and flashing tests - HardWare Test (HWT) v12015: HWT v2

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 6/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Testing our u-boot

2011: Build test infrastructure (u-boot-CI)2014: USB DFU/USB Mass Storage test scripts in mainline

2014: Boot and flashing tests - HardWare Test (HWT) v12015: HWT v2

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 6/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Testing our u-boot

2011: Build test infrastructure (u-boot-CI)2014: USB DFU/USB Mass Storage test scripts in mainline2014: Boot and flashing tests - HardWare Test (HWT) v1

2015: HWT v2

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 6/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Testing our u-boot

2011: Build test infrastructure (u-boot-CI)2014: USB DFU/USB Mass Storage test scripts in mainline2014: Boot and flashing tests - HardWare Test (HWT) v12015: HWT v2

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 6/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Automated Test System

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 7/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Automated Test System - overview

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 8/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Automated Test System - block diagram

Board1

BoardN

Host PCWiki

USB HUB

JenkinsTests

USB HUB USB HUBPWR IO

RelaysUSB <->UART

USB <->UART

Power

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 9/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Section 2

System description

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 10/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Architecture

N testsSetup

Instance

TestInstance

ConfigPer board:- Toolchain- HW config

- Board quirks

Odroid U3/XU3

Trats Board N

Setupscripts

N testsTest

scripts

SetupInstance

TestInstance

SetupInstance

TestInstance

SetupInstance

TestInstance

SetupInstance

TestInstance

SetupInstance

TestInstance

ConfigPer board:- Toolchain- HW config

- Board quirks

ConfigPer board:- Toolchain- HW config

- Board quirks

Jenkins and WIKIresultsvisualization

Toolchains

GIT reposTest trigger:GIT push

Test trigger:Schedule

- Build bootloader and kernel- Download rootfs- Flash target board

- Execute script on target board- Use procfs, sysfs,debugfs, etc.- Test result:0 → OKOther → FAIL

Links to proper tests/setuprepositories.Test/setup instances representedas directories.

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 11/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Key design decisions

Why: New vs. Reuse

SW: Python vs. ShellConsole: Python (pexpect) vs. expect (CLI program)Flashing: Ethernet (TFTP) vs. USB (DFU)Scheduling: JenkinsPresentation: Wiki

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 12/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Key design decisions

Why: New vs. ReuseSW: Python vs. Shell

Console: Python (pexpect) vs. expect (CLI program)Flashing: Ethernet (TFTP) vs. USB (DFU)Scheduling: JenkinsPresentation: Wiki

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 12/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Key design decisions

Why: New vs. ReuseSW: Python vs. ShellConsole: Python (pexpect) vs. expect (CLI program)

Flashing: Ethernet (TFTP) vs. USB (DFU)Scheduling: JenkinsPresentation: Wiki

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 12/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Key design decisions

Why: New vs. ReuseSW: Python vs. ShellConsole: Python (pexpect) vs. expect (CLI program)Flashing: Ethernet (TFTP) vs. USB (DFU)

Scheduling: JenkinsPresentation: Wiki

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 12/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Key design decisions

Why: New vs. ReuseSW: Python vs. ShellConsole: Python (pexpect) vs. expect (CLI program)Flashing: Ethernet (TFTP) vs. USB (DFU)Scheduling: Jenkins

Presentation: Wiki

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 12/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Key design decisions

Why: New vs. ReuseSW: Python vs. ShellConsole: Python (pexpect) vs. expect (CLI program)Flashing: Ethernet (TFTP) vs. USB (DFU)Scheduling: JenkinsPresentation: Wiki

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 12/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Section 3

Building blocks

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 13/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Power control

FTDI FT232RLNote: https://github.com/lmajewski/HWT_io_ctl.git

RelaysNote: GPIO-controlled relays module for Arduino

USB HUBNote: External and stable power supply

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 14/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Host PC

Ubuntu Long Term Support (LTS)USB udev configurationRuns:

Dash testsJenkins clientRepository mirroringMediaWiki

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 15/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Host PC - Test U-boot booting

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 16/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Host PC - Test kernel

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 17/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Host PC - Jenkins [optional for simple setup]

Responsible for scheduling testsProvides more sophisticated functionality to the test setup:

Gerrit push triggerRepository/branch change trigger

For simple test system it can be omitted - cron managed testtriggering with results uploaded to wiki

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 18/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Host PC - Wiki

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 19/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Minimal HW costs

Device Price [EUR] RemarksFT232RL 10Relays 5USB HUB 40 External powerHost PC 0 Old reusedCables/connectors 10Power supply 0 Included with target boardTotal 65

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 20/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Section 4

Summary

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 21/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Solved problems

Problem: Breaking the default boot

Solution: Use C/C++ with nanosleep()Problem: Jenkins logs readability

Solution: Concatenate logs and link them to Wiki

Problem: Target power control

Solution: HW rework to also control VUSB

Problem: Lack of persistent USB configuration

Solution: Test system connectivity managed by udev rules,which use USB device serial numbers

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Solved problems

Problem: Breaking the default bootSolution: Use C/C++ with nanosleep()

Problem: Jenkins logs readability

Solution: Concatenate logs and link them to Wiki

Problem: Target power control

Solution: HW rework to also control VUSB

Problem: Lack of persistent USB configuration

Solution: Test system connectivity managed by udev rules,which use USB device serial numbers

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Solved problems

Problem: Breaking the default bootSolution: Use C/C++ with nanosleep()

Problem: Jenkins logs readability

Solution: Concatenate logs and link them to WikiProblem: Target power control

Solution: HW rework to also control VUSB

Problem: Lack of persistent USB configuration

Solution: Test system connectivity managed by udev rules,which use USB device serial numbers

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Solved problems

Problem: Breaking the default bootSolution: Use C/C++ with nanosleep()

Problem: Jenkins logs readabilitySolution: Concatenate logs and link them to Wiki

Problem: Target power control

Solution: HW rework to also control VUSB

Problem: Lack of persistent USB configuration

Solution: Test system connectivity managed by udev rules,which use USB device serial numbers

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Solved problems

Problem: Breaking the default bootSolution: Use C/C++ with nanosleep()

Problem: Jenkins logs readabilitySolution: Concatenate logs and link them to Wiki

Problem: Target power control

Solution: HW rework to also control VUSBProblem: Lack of persistent USB configuration

Solution: Test system connectivity managed by udev rules,which use USB device serial numbers

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Solved problems

Problem: Breaking the default bootSolution: Use C/C++ with nanosleep()

Problem: Jenkins logs readabilitySolution: Concatenate logs and link them to Wiki

Problem: Target power controlSolution: HW rework to also control VUSB

Problem: Lack of persistent USB configuration

Solution: Test system connectivity managed by udev rules,which use USB device serial numbers

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Solved problems

Problem: Breaking the default bootSolution: Use C/C++ with nanosleep()

Problem: Jenkins logs readabilitySolution: Concatenate logs and link them to Wiki

Problem: Target power controlSolution: HW rework to also control VUSB

Problem: Lack of persistent USB configuration

Solution: Test system connectivity managed by udev rules,which use USB device serial numbers

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Solved problems

Problem: Breaking the default bootSolution: Use C/C++ with nanosleep()

Problem: Jenkins logs readabilitySolution: Concatenate logs and link them to Wiki

Problem: Target power controlSolution: HW rework to also control VUSB

Problem: Lack of persistent USB configurationSolution: Test system connectivity managed by udev rules,which use USB device serial numbers

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 22/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Found bugs:

u-boot: SD card timeout for writing datau-boot: Wrong error status passing from FAT to upper layersLinux: ”rootwait” problem in Trats/Trats2

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 23/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Conclusion

YES, it was worth to build this setup.

Benefits:Tizen.org: build, boot and functionality testsu-boot-dfu: build, boot (u-boot and kernel), USB: MassStorage, DFU, THOR

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 24/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Conclusion

YES, it was worth to build this setup.Benefits:

Tizen.org: build, boot and functionality testsu-boot-dfu: build, boot (u-boot and kernel), USB: MassStorage, DFU, THOR

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 24/25

. . . . . .

Introduction System description Building blocks Summary Discussion

Section 5

Discussion

Łukasz Majewski Samsung R&D Institute Poland”Automated Test System - build blocks” 25/25