MCC: A dynamic verification tool for MCAPI user applications

1

MCC: A dynamic verification tool for MCAPI user applications

Subodh Sharma, UofU Ganesh Gopalakrishnan, UofU Eric Mercer, BYU Jim Holt, Freescale

1Supported by NSF CCF 0903408/0903491 and SRC 2009-TJ-1993/1994

FMCAD 2009

2

Concurrency Space in Multicore Era

S/W solution H/W solution

Cloud/HPC Computing MPI, PVM, Mapreduce, MSCS, Dryad

Clusters, Vector machines, Supercomputers

Desktop computing OOSD, Scripting, Pthreads, TBB, CT, OpenMP

Desktop machines

Embedded Computing Lightweight counterparts of the above.

FPGAs, DoC, SoC, etc.

https://asc.llnl.gov/computing_resources/bluegenel/images/bluegenel.jpg

3

Formalize Emerging Communications API in the Embedded Space

S/W solution H/W solution

Cloud/HPC Computing MPI, PVM, Mapreduce, MSCS

Clusters, Vector machines, Supercomputers

Server/Desktop computing

OOSD, Scripting, Pthreads, TBB, CT, OpenMP

Desktop machines

Embedded Computing Lightweight counterparts of the above.

FPGAs, DoC, SoC, etc.

Demonstrate and Evaluate

Prototype Solutions

Build Dynamic Formal Verifier for Applications

Formalize Standards, build

Query Oracle, Derive Tests

https://asc.llnl.gov/computing_resources/bluegenel/images/bluegenel.jpg

4

MCAPI (Multicore Communication API) Introduction

5

What is MCAPI (Multicore Communication API)?

• An API specification from MCA (Multicore Association) - Member companies – Freescale, Samsung, Intel, etc.

• To program embedded systems like mobile phones, PDAs, routers, servers, etc.

• Not restricted to SPMD (like MPI) or multi threaded style of programming.

http://www.linuxdevices.com/articles/AT8728350077.html




6

Taxonomy for MCA APIs

MCAPI – Communication API• Message based• Packet/Scalar Channel based

MRAPI – Resource Management API

• Semaphores, mutexes• Shared memory segment allocation, deallocation

MTAPI – Task Management API• Specification work yet to begin• Thread Pooling, Work Stealing queues e.g. CILK, TBB, TPL etc.Inter API

interactions

7

Example usage scenario of MCA APIs

Core Core

Shared Memory

H/W Accelerator

Interconnect

MRAPI

MCAPI

MTAPI Threads

8

MCAPI Messages

Node 1

Port 1 <1,1>

Port 2 <1,2>

Node 2

Port 1 <2,1>To <2,1>

• Message received at FIFO ordered receive

queue

Endpoint:Node_id, Port_id

9

MCAPI Connection Oriented Communication

Node 1

Port 1 <1,1>

Port 2 <1,2>

Node 2

Port 1 <2,1>To <2,1>

• Sent over a connected channel• <1,1> can communicate on ONE channel only

Node 3

Port 1 <3,1>

To <3,1>ERROR!!

10

Our Contributions!

11

• To promote early adoption of the API

• To promote better programming practices

• To help avoid early pit-falls – e.g. unspecified behaviors

Early Engagement of FV in the MCAPI space

12

MCC – MCAPI Checker

MCAPI C ProgramMCAPI C Program

instrumentation

Instrumented ProgramInstrumented Program

compile

MCAPI Library WrapperMCAPI Library Wrapper

Scheduler

Executable

thread 1

thread n

request/permit

request/permit

RUNTIME

….

13

Related Work

• Tools in this domain work on directly work on unmodified sources - inspired by Verisoft.

• Tools control the scheduling to achieve the goal of coverage guarantees

• Examples are CHESS from Microsoft, Inspect and ISP from Univ of Utah, etc.

14

Unique features of MCC

• External schedulers may not be able to exercise control over runtime.

• Novel way of enforcing deterministic match of transitions at runtime

• Instruments Pthread calls ( i.e. would support hybrid programs written in future using MCAPI)

15

MCAPI API calls supported by MCC

Initialize/Finalize Sets and deletes the environment. Must be called by each communicating node.

Create/Delete endpoint API calls to manage creation/deletion of endpoints on the owner node

Get_endpoint A blocking call to get the address of a remote endpoint.

Send (sndEp, rcvEp, ..) Sends a data-payload from a sending endpoint to a receiving endpoint. API provides blocking and non blocking versions.

Recv (rcvEp, ..) Receives a data payload from the receiving endpoint. API provides blocking non-blocking versions.

Wait (reqHndl, ..), Test(reqHndl,..) Checks the completion of the request. Wait – Blocking; Test – Non blocking

16

Bugs in the application space?Why Dynamic Analysis?

Why Deterministic Execution Control?

17

• The recv() call is passed with only receiving endpoint as a parameter.

• The receiving endpoint extracts message from the FIFO receive queue.

T0 T1 T2

--- --- ---

The send that matches the recv may decide the correctness !!

POTENTIAL BUG

Send(from e1 to e2, d1) Send(from e1 to e2, d2) Recv( from e2 in d3)

If( d3 == 22) Recv(from e2)Else ERROR

Non-deterministic MCAPI Receive Calls

18

Mismatched Parameters

“get_endpoint” requested for a non-existent endpoint -- ERROR

T0 T1

--- ---

Create_Ep (0,1)

R(<0,1>) Get_Ep(<0,2>)

S(1,<0,2>)Does not exist!!

19

Challenge: Exponential Interleavings

19

T0 T1 T2 T3 T4 T5

T0 T1 T2 T3 T4 T5

A B

Dependent MCAPI calls

Only these 2 are RELEVANT!!!

Possible Interleaving

2020

• Dynamic reduction• Transition dependency at

runtime• precise information

effective reduction

Dynamic Interleaving Reduction

T0 T1 T3 T0 T1 T3

Send(to E1)Send(to E1) Send(to E2)Send(to E2) Recv(E3)Recv(E3)

DYNAMICALLY:• Discover all potential senders• Match Recv with each sender• Recurse through all such configurations

E1 == E2 == E3?If yes, extra interleaving

21

Relevant interleaving exploration by MCC scheduler

T0 T1 T2

S (ep1,ep3) S(ep2,ep3) R(ep3)

R(ep3)

Intra happens-Before Edge

22

T0 T1 T2

S (ep1,ep3) S (ep2,ep3) R (ep3)

R (ep3)

Interleaving 1

Issue to the runtime

Match sets are computed at “decision points”

Set of matchingtransitions

All threads are blocked


23

T0 T1 T2


R (ep3)

Interleaving 1



24

T0 T1 T2


R (ep3)

Enabled Transitions: S(ep1, ep3), S(ep2, ep3), R (ep3)

Match Set: <S(ep2, ep3), R (ep3)>

Interleaving 2



25

T0 T1 T2


R (ep3)

Enabled Transitions: S(ep2, ep3), R (ep3)

Match Set: <S(ep1, ep3), R (ep3)>

Interleaving 2



26

T0 T1 T2


R (ep3)

Interleaving 2

Issue to the runtimeError Caught


27

Enforcing a deterministic runtime match• For non blocking requests, there is an additional problem

• Runtime communication race even after a scheduler decides deterministically.

T0 T1 T2

nbS (ep1,ep3) nbS (ep2,ep3) nbR (ep3)

nbR (ep3)

Match-set 1

Match-set 2

28

T0 T1 T2


nbR (ep3)

Match-set 1

Match-set 2

Not finished yet!!

RACE

Enforcing a deterministic runtime match

29

Solution 1 to enforce a deterministic match.

• Addition of Probing function to the MCAPI library• Probes an endpoint’s receive queue for a message• Returns TRUE if the message is found

• Scheduler issues the next match-set only when the probe function returns TRUE.


30

T0 T1 T2


nbR (ep3)

Probing Solution

Receive Queue

matched


31

T0 T1 T2


nbR (ep3)

Probing Solution

Receive Queue

matched

Runtime MCC Scheduler

Probes


32

T0 T1 T2


nbR (ep3)

Probing Solution

T1 Receive Queue

Probes



33

T0 T1 T2


nbR (ep3)

Probing Solution

T1T0 Receive Queue


34

Solution 2 to enforce a deterministic match.

• Scheduler induces a “test” call

• Scheduler spin-loops on the request handle until the

successful completion of the “test” call.

We opt Solution 2 in our work as it is non-intrusive.


35

T0 T1 T2


nbR (ep3)

Enabled Transitions: Send(ep1, ep3), Send(ep2, ep3), Recv (ep3)

Match Set: <Send(ep2, ep3), Recv (ep3)> <Send(ep1, ep3), Recv (ep3)>

Dummy “wait” Solution

matched


36

T0 T1 T2


nbR (ep3)



Test ()


37

T0 T1 T2

nbS (ep1,ep3) IS (ep2,ep3) nbR (ep3)

nbR (ep3)



Test()

matched


38

Concluding remarks and future work

• Code is currently being developed • MCC currently supports blocking and non-blocking connectionless API

constructs• Checks for safety assertion violations and Deadlocks• Porting concurrency benchmarks into MCAPI – Eg. Rodinia

Benchmarks, BSS use case• MCC will be tested on these benchmarks.

• Steadily improve MCC• Support for connection-oriented API calls, sanity checks etc.• Accommodate “non-determinism” in the shared memory space.

39

Thank You!

www.cs.utah.edu/formal_verification

40

Happens-Before Relationship – Computed Dynamically

T0 T1 T2

nbS (ep1,ep3, r1) nbS (ep2,ep3, r3) nbR (ep3, r4)

nbR (ep3, r5)nbS (ep1,ep4, r2)

R (ep3)W(r2)

41

t1: a[x] := …

t2: … := a[y] t1: a[x] := …

t2: … := a[y]

Equal ??

t2: … := a[y]

Review of POR

Dynamic POR

Dependencies based on concrete values

x=56, y=753 (e.g.) at runtime!

42

Happens-Before Relationship – Example

T0 T1 T2



R (ep3)W(r2)

43

Happens-Before Relationship – Example

T0 T1 T2



R (ep3)W(r2)

Instrumented Code

Start Start Start

End End End

W(r3)

44

Time based flow of the workings of MCC scheduler on a sequence

T0:Start

Sched T0 T1 T2

T0:Start

45

Time based flow of the workings of MCC scheduler

T0:Start

Sched T0 T1 T2T1:Start

T0:Start

T1:Start

46


T0:Start

Sched T0 T1 T2T1:Start T2:Start

Decision Point is

hit!

T0:Start

T1:Start

T2:Start

47


T0:Start


Go-aheadT0:Start

T1:Start

T2:Start

48


T0:Start


T0: IST0:Start

T1:Start

T2:Start

T0: IS (ep3)

49


T0:Start


T0: IS T2: IRT0:Start

T1:Start

T2:Start

T0: IS (ep3)

T2: IR (ep3)

50


T0:Start


T0: IS T2: IR

T1: IS

T0:Start

T1:Start

T2:Start

T0: IS (ep3)

T2: IR (ep3)

T1: IS (ep3)

51


T0:Start


T0: IS T2: IR

T1: IST1: W

T0:Start

T1:Start

T2:Start

T0: IS (ep3)

T2: IR (ep3)

T1: IS (ep3)

T1: W

52


T0:Start


T0: IS T2: IR

T1: IST1: W

T0:Start

T1:Start

T2:Start

T0: IS (ep3)

T2: IR (ep3)

T1: IS (ep3)

T1: W

T0: IS (ep4)

T0: IS (ep4)

53


T0:Start


T0: IS T2: IR

T1: IST1: W

T0:Start

T1:Start

T2:Start

T0: IS (ep3)

T2: IR (ep3)

T1: IS (ep3)

T1: W

T0: IS (ep4)

T0: W

T0: IS (ep4)

T0: W

54


T0:Start


T0: IS T2: IR

T1: IST1: W

T0:Start

T1:Start

T2:Start

T0: IS (ep3)

T2: IR (ep3)

T1: IS (ep3)

T1: W

T0: IS (ep4)

T0: W

T2: IR (ep4)

T0: IS (ep4)

T0: W

T2: IR (ep4)

55


T0:Start


T0: IS T2: IR

T1: IST1: W

T0:Start

T1:Start

T2:Start

T0: IS (ep3)

T2: IR (ep3)

T1: IS (ep3)

T1: W

T0: IS (ep4)

T0: W

T2: IR (ep4)

T2: R (ep3)

T0: IS (ep4)

T0: W

T2: IR (ep4)

T2: R (ep3)Decision Point is

hit!

56


T0:Start


T0: IS T2: IR

T1: IST1: W

T0:Start

T1:Start

T2:Start

T0: IS (ep3)

T2: IR (ep3)

T1: IS (ep3)

T1: W

T0: IS (ep4)

T0: W

T2: IR (ep4)

T2: R (ep3)

T0: IS (ep4)

T0: W

T2: IR (ep4)

T2: R (ep3)Match set:T0: IS(ep3)T2: IR(ep3)

57


T0:Start


T1: IST1: W

T0:Start

T1:Start

T2:Start

T0: IS (ep3)

T2: IR (ep3)

T1: IS (ep3)

T1: W

T0: IS (ep4)

T0: W

T2: IR (ep4)

T2: R (ep3)

T0: IS (ep4)

T0: W

T2: IR (ep4)

T2: R (ep3)

Go-ahead

58


T0:Start


T1: IST1: W

T0:Start

T1:Start

T2:Start

T0: IS (ep3)

T2: IR (ep3)

T1: IS (ep3)

T1: W

T0: IS (ep4)

T0: W

T2: IR (ep4)

T2: R (ep3)

T0: IS (ep4)

T0: W

T2: IR (ep4)

T2: R (ep3)

Match set:T0: IS(ep4)T2: IR(ep3)

59


T0:Start


T1: IST1: W

T0:Start

T1:Start

T2:Start

T0: IS (ep3)

T2: IR (ep3)

T1: IS (ep3)

T1: W

T0: IS (ep4)

T0: W

T2: IR (ep4)

T2: R (ep3)

T2: R (ep3)

Go-ahead

60


T0:Start


T1: IST1: W

T0:Start

T1:Start

T2:Start

T0: IS (ep3)

T2: IR (ep3)

T1: IS (ep3)

T1: W

T0: IS (ep4)

T0: W

T2: IR (ep4)

T2: R (ep3)

T2: R (ep3)

Match set:T1: IS(ep3)T2: R(ep3)

61


T0:Start


T0:Start

T1:Start

T2:Start

T0: IS (ep3)

T2: IR (ep3)

T1: IS (ep3)

T1: W

T0: IS (ep4)

T0: W

T2: IR (ep4)

T2: R (ep3)

Go-ahead

62

OOO completion: Why construct Happens-Before relationship?

62

T0

T0

IS (1GB to ep1)IS (1GB to ep1)

IS (1 byte to ep2)IS (1 byte to ep2)Must calls complete in

PROGRAM ORDER?

NO!!!

Happens-Before relationship helps the scheduler to precisely constructmatching transitions in recognition of the non-overtaking rules of the runtime.

T1

T1

IR (ep1)IR (ep1)

IR (ep2)IR (ep2)

63

OOO completion: Why construct Happens-Before relationship?

63

T0

T0

IS (1GB to ep1)IS (1GB to ep1)

IS (1 byte to ep1)IS (1 byte to ep1)Must calls complete in

PROGRAM ORDER?

YES!!!

NON OVERTAKING RULE

T1

T1

IR (ep1)IR (ep1)

64

Illustration of MCAPI C program

int main () {…for(t=0; t<NUM_THREADS; t++){ rc = pthread_create( .. ); } for (t = 0; t < NUM_THREADS; t++) { pthread_join( .. ); }}void* run_thread (void *t) {… mcapi_initialize( .. ); if (tid == 1) { recv_ep = mcapi_create_endpoint (P_NUM, ..); mcapi_msg_recv(recv_ep, ..); mcapi_msg_recv(recv_ep, ..);}

else {send_ep = mcapi_create_endpoint (P_NUM, ..);

recv_endpt = mcapi_get_endpoint (1,P_NUM, ..);

mcapi_msg_send(send_ep,recv_ep, ..);}mcapi_finalize( .. );return NULL;}

Potential deadlock

65

Focus of the talk!

• Dynamic Verification of MCAPI user apps (at Utah)

• MCC tool created.

• Current capabilities of MCC• Dynamic search of dependencies• Exerting a deterministic runtime control• Supports connectionless blocking and non-blocking API calls

MCC: A dynamic verification tool for MCAPI user applications

Documents

Transcript of MCC: A dynamic verification tool for MCAPI user applications