Dynamic Runtime Testing for Cycle-Accurate Simulators

Saša Tomić, Adrián Cristal, Osman Unsal, Mateo Valero

Barcelona Supercomputing Center (BSC)Universitat Politecnica de Catalunya (UPC)

2

Can we trust the simulator-based evaluations?

• Typical simulator evaluation:Make a simulatorREPEAT {

DebugSimulate

} UNTIL: the results make sense (intuition!)• Discard and ignore the failed simulations• Are there any bugs left?

Verifying the simulators• Verification is important!

– Industry puts significant resources• Testing and Verification 50-70% of the costs• Mission critical application even 90% of the costs

– Academia puts less resources• Why do we have bugs?

– Simulators are complex– Proposed extensions are often complex– The extensions may uncover existing bugs

3

Simulator bugs• Timing bugs

– Incorrect estimation of the execution time– Simulation terminates without obvious errors– Needs other types of testing and verification

• Functional bugs– Incorrect implementation of functional units– Simulations may or may not terminate correctly

4

Our target

Outline• Examples of functional bugs• An overview of the Dynamic Simulator Testing

methodology• Use Cases of Dynamic Simulator Testing• Performance evaluation• Conclusions

5

Example: a bug in the cache coherence protocol

6

simulator ofmulti-level coherent caches

X=0

X=0

X=10 X=0 X=20

X=20

X += 10 X += 20

Bug: X should beX = 10+20 = 30shared memory

processor 1 processor 2

Proc 1X+=10

Proc 2X+=20

X=0

X=30

time

Example: a bug in the HTMX = 0;Atomic { X += 10;}

7

HTM simulator

X=0

X=0 X=10

nothing?

Bug: not committedX = 10

processor

shared memory

X += 10

Detecting functional bugs• The functionality of the simulators is often simple

– Can be emulated with simple emulators– The emulators can be fast and stable

• Can we take an advantage of the emulators?

8

Dynamic Testing Methodology

• Add a simple, no-timing emulator• Execute each operation in the simulator and then in

the emulator• Compare the executions

– We compared only the memory accesses• The execution must be identical during entire simulation

9

An overview of dynamic testing

10

timing simulatorsimple

no-timing emulator

input input

input inputoutput output

output output

• Use the same input• Compare the outputs• Repeat for every operation!

Dynamic testing a cache coherence protocol

11

STL maptiming simulator ofmulti-level coherent caches

X += 10 X += 20

shared memory

processor 1 processor 2

inputX=10X=10

inputX=0

outputX=0

outputX=0

outputX=20input

X=20

outputoutput input

X=0 X=0

input

X=10

Check failed:should be X=10

X=0X=10

Dynamic testing of an HTM

12

timing simulator of an HTM

STL mapper TX

TX_Begin; X += 10;TX_Commit;

Check failed:should commit X=10

input

X=0

input

X=0

processor

shared memory

input

X=10

input

X=10

output

X=0 X=0

output

output

X=??? X=10

output

X=0X=10

Other Use Cases• Out-Of-Order or pipelined processor

– With a processor emulator, e.g., QEMU• Complex memory hierarchy

– With an STL map• Incoherent multilevel memory hierarchy

– W/ multiple STL maps, one per memory hierarchy• System-On-Chip, Routing Protocols, etc.

– Simple emulators of the functionalities

13

Performance Evaluation• Implemented on 4 HTMs with lazy and eager

version management• Implemented for a directory-based cache-

coherence protocol• Baseline: M5 full-system simulator

14

Performance evaluationOS booting

15

1 2 4 8 16 320

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

OS booting with cache testing OS booting with HTM and cache testing

Norm

aliz

ed e

xecu

tion

time

Performance evaluationapplications

16

baye

s

geno

me

intrud

er

kmea

ns-hi

kmea

ns-lo

w

labyri

nthss

ca2

vaca

tion-h

i

vaca

tion-l

owya

da0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Application with cache testing Application with HTM and cache testing

Norm

aliz

ed e

xecu

tion

time

Our experience with Dynamic Testing

• Reduced the time spent on writing tests• Faster debugging

– Detects most bugs “in minutes”– Eliminating a bug takes tens of minutes instead of

hours/days/weeks/…• Shortened the total simulator development from

12-18 months to 3-4 months

17

Conclusions• Presented the Dynamic Simulator Testing• Detects the functional bugs in Cycle-Accurate

Simulators• Modest reduction of simulator performance

18

Thanks!

Sasha Tomićsasa.tomic@bsc.es

19

Dynamic Runtime Testing for Cycle-Accurate Simulators

Cache/HTM emulator implementation

20

STL map (dictionary)

addressaddressaddress

line dataline dataline data

address line data...

.

.

.