Issta11

Testing EmbeddedSoftware

John Regehr

University of Utah

• “Over 15 billion ARM based chips shipped to date”– [ARM web site, 2011]

• “The microcontroller market is forecast to reach over $16 billion worldwide in 2011”– [Microcontroller Market Tracker,

2011]

2

Diverse!

4

Diverse!

5

I have 6 pins and 32 bytes

of RAM

Diverse!

6

Diverse!

7

Diverse!

8

I am quad core @ 1.5 GHz and

have a GPU

• Usually there are multiple processors–On-chip networks

– In-device networks

–Distributed systems

• Resource constraints are…– Severe – to minimize unit cost

–Hard – failure if system runs out of…

• Time

• RAM – stack or heap

• Energy9

• Continuously interact with the world through I/O devices–May be little abstraction of HW

–Probably using both interrupt handlers and threads

• Often there are fault tolerance and security requirements

10

• Sensor network → 103–105 LOC

• Modern airplane → 106–107 LOC

• Hybrid vehicle → 107–108 LOC

• How do we get these right?–Mostly testing

11

• Software on many individual processors is small–Permits aggressive analysis and testing

• Constrained domain simplifies testing– Embedded systems are (by definition)

special-purpose devices

12

The “Real System Problem”

• Many interesting embedded codes are proprietary

• Necessary tools may be expensive or nonexistent– Compilers, debuggers, simulators

• May not be able to run it in the lab

• Often lacks specifications and oracles

13

• Consequently, academic embedded work may be…– Forced to use small, contrived examples

–Out of tune with industry

14

• Consequently, academic embedded work may be…– Forced to use small, contrived examples

–Out of tune with industry

15

Solution: Ubiquitous open embedded platforms

Arduino

• Arduino Uno:–8-bit AVR processor @ 16 MHz

–2 KB RAM

–~$30

• Emphasis is on interfacing16

Arduino

• Nice IDE + libraries + C/C++–Minimal abstraction of the embedded

processor

• 18 new books in 2011

17

Arduino

• Nice IDE + libraries + C/C++–Minimal abstraction of the embedded

processor

• 18 new books in 2011

18

• Simulators and model checkers for AVR code exist

• Very few Arduino tool papers exist

• This is a big opportunity

TinyOS

• OS and middleware support for sensor networks– Sensing

–Collection and dissemination

– Localization

• Applications are in nesC, a C dialect19

TinyOS

• “Motes” based on a variety of MCUs–Cost $50 – $200

• Good simulators exist

• There are a few books20

TinyOS

• “Motes” based on a variety of MCUs–Cost $50 – $200

• Good simulators exist

• There are a few books21

• ~100 tool papers

• Many open problems

Android

• OS + middleware for smart phones / tablets–ARM based hardware running Linux

–Much less constrained than motes and Arduino

22

Android

• Application code in Java

• Great tools

• Tons of books

23

Android

• Application code in Java

• Great tools

• Tons of books

24

• < 100 tool papers• Most are very recent• This is not a scary platform

ROS – Robot Operating

System

• Linux-based infrastructure for programming robots

• Primary abstraction is graph of communicating processes– Local and distributed

25

ROS – Robot Operating

System

• Linux-based infrastructure for programming robots

• Primary abstraction is graph of communicating processes– Local and distributed

26

• Very few ROS tool papers exist

• Plenty of other open embedded platforms exist– FreeRTOS

–Contiki

–Pacemaker Challenge

– Etc.

• Embarrassment of riches

• Still, huge room for improvement–Where’s the open automobile?

27

• So, let’s test some embedded software–But what are we testing for?

28

Properties / Oracles

• Temporal safety–Deadlines

–Or just responsiveness

• Memory safety

• Contracts / assertions

• Reference implementation

29

Worst-Case Execution Time

• What is the upper bound on execution time for a piece of code?–We care because the world has deadlines

• Static analysis of WCET is extremely difficult if there is…–A cache

–Preemption

–An aggressive processor

30

31

Conservative WCET

True WCET

Longest observed ET #2

Execution time

Nu

mb

er

of

exe

cuti

on

s

Longest observed ET #1

32

+ printf()=

pthread_attr_setstacksize (&attr, &mystacksize);

HANDLE WINAPI CreateThread(

LPSECURITY_ATTRIBUTES lpThreadAttributes,

SIZE_T dwStackSize,

LPTHREAD_START_ROUTINE lpStartAddress,

LPVOID lpParameter,

DWORD dwCreationFlags,

LPDWORD lpThreadId );

Stack Overflow in TinyOS

33

4 KB


34

4 KB

main()


35

4 KB

main()

irq 0

irq 1


36

4 KB

main()

irq 0


37

4 KB

main()

irq 0

irq 1


38

4 KB

main()

irq 0

irq 1

• Not the same thing as buffer overflow!

• Type safe language doesn’t solve this problem

Eliminating Stack Overflow

• Testing is hard–Need to drive code to its WC stack depth

– Interrupt coincidences are rare

• Approach: Static analysis of compiled code–Can’t estimate stack depth of source

39

1. Estimate WC stack depth of each sequential flow, handling

– Indirect branches

– Recursion

– Loads into the stack pointer

2. Compute “interrupt preemption graph”

3. Find longest cycle in this graph

40

41

in r24, 0x3f ; r24 <- CPU status

register

cli ; disable interrupts

adc r24, r24 ; carry bit <- prev

interrupt status

eor r24, r24 ; r24 <- 0

adc r24, r24 ; r24 <- carry bit

mov r18, r24 ; r18 <- r24

... critical section ...

and r18, r18 ; test r18 for zero

breq .+2 ; if zero, skip next

instruction

sei ; enable interrupts

ret ; return from function

• Stack analysis tool deployed in the TinyOS distribution

• Results are typically much larger than worst observed stack depths–But, we validated its results by randomly

firing interrupts

42

Need… more… oracles…

43

• TinyOS applications are built using components– Interface

requirements documented but not checked

– Interface misuse often silent

44

• We augmented nesC with contracts–Dynamic checking

reasonable efficient

– Found some long-standing bugs

45

• nesC is not type safe–Memory safety bugs in TinyOS are difficult

• We ported an existing safe C dialect

• Found some otherwise-impossible bugs

• Main problem was getting overhead under control–Whole-program optimization

46

47

Code size

35%

13%

-11%

48

Array Out-of-bounds

Array Out-of-bounds

Reboot

NormalTinyOS

SafeTinyOS

RebuildSoft state

Normal TinyOS:0% average availability

Safe TinyOS:95% average

availability

Increasing Availability

• What about application-level sensornet properties?–All the interesting ones are distributed

• We adapted TOSSIM, a non-cycle-accurate simulator, to be…–A random tester

–A depth-bounded model checker

• Oracles: –Type safety checks

–Application-level properties

49

Application-Level Properties

• Eventually…– Each send buffer is unlocked

–No cycles in the routing tree

–All nodes become part of the collection tree

–All nodes have consistent values

• 6 out of 8 of these properties require global knowledge

50

• Found 12 previously unknown bugs in TinyOS 2.0–10 safety, 2 liveness

• Random testing outperformed depth-bounded model checking– Even after a lot of work on POR

–But required work to shorten long error traces

51

Conclusions

• Open embedded platforms exist– Some have steep learning curves

• Finding oracles is hard

• Generating valid input is hard

• Embedded systems are fun and important and rewarding

52

Issta11

Technology

Transcript of Issta11