Modeling and Analyzing Periodic Distributed 

Computations

Anurag AgarwalVijay Garg (garg@ece.utexas.edu)

Vinit Ogale

The University of Texas at Austin

SSS’2010

Motivation

• Many distributed computations are infiniteo E.g., various reactive systems, servers

• Correctness Specificationso Safety: Nothing "bad" will ever happen o Liveness: Something "good" will happen eventually

• Specified using Temporal Logic [Pnueli 77]

• Runtime verification of properties

• How can one detect violation of liveness properties?

Difficulties

Can observe only a finite execution• How to exhibit the offending execution?• How to detect if a global state is reachable in an

infinite computation?

Recurrent Global State: Liveness violation

• Two identical states with P1 being hungry

• P1 does not eat in the intermediate computation

Overview

• Predicate Detection in Partial Order

• Recurrent global states• Modeling infinite computations as d-diagrams

• Vector clocks in d-diagrams• Predicate detection in d-diagrams

• Related work• Conclusions and Future Work

Predicate Detection

Predicate: A property expressed using variables on processes. e.g., more than one process is in critical section

Predicate detection: Determining if an execution trace satisfies the predicate

6

Predicate detection

trace

predicate

Yes

No

Program

Trace Model: Total Order

• Total order: interleaving of events in a trace – Temporal Rover [Drusinsky 03], – Java-MaC [Kim, Kannan, Lee, Sokolsky, and Viswanathan 04], – JPaX [Havelund and Rosu 04]

– PET [Gunter, Kurshan, Peled 00]

• Low computational complexity

7

Partial Order Traces

• Predicate Detection– Exponential time algorithm for general predicate [Cooper and Marzullo 91]– NP-complete for simple boolean expressions (2-CNF) [Mittal and Garg 01]– Efficient algorithms for linear predicates [Chase, Garg 95], relational

predicates [Tomlinson, Garg 93], Temporal Logic [Ogale Garg 07]

8

{}

{e1} {f1}

{e1, f1}

{e2, e1, f1}

{e2, e1, f2, f1}

{e1, f2, f1}

e1 e2

f1 f2

P1

P2 {e2,e1}

Computation

Corresponding computation lattice

Recurrent global states

• (Consistent) Global states which occur more than once

 

D-diagram

A d-diagram is a finite representation of an infinite periodic distributed computation ; (V,R,F,B)• V: set of vertices• R: recurrent vertices (infinite instances)• F: forward edges for all i:ei -> f i

• B: shift edges for all i: ei -> f i + 1

Examples

• Set of natural numbers under natural order

• Set of natural numbers with no order

Unrolling the d-diagram

A directed graph can be generated by "unrolling" a d-diagram ie. creating infinite instances of the recurrent vertices and generating the appropriate edges between them.

Finite Width Posets

• Lemma: A directed graph G defined by a d-diagram has finite width iff for every recurrent vertex there exists a cycle that includes a shift-edge.

Shift-of-a-cut

A consistent cut in the graph can be "shifted" forwards or backwards with respect to some recurrent vertices to generate another consistent cut. 

Overview

• Predicate Detection in Partial Order

• Recurrent global states• Modeling infinite computations as d-diagrams

• Vector clocks in d-diagrams• Predicate detection in d-diagrams

• Related work• Conclusions and Future Work

Vector Clocks in a Distributed System

e happened before f iff V(e) < V(f) [Fidge 89, Mattern 89]

P1(1,0,0) (2,1,0) (3,1,0)

P2(0,1,0) (0,2,0)

P3(0,0,1) (0,0,2) (2,1,3)

How do we timestamp infinite sets of events?

Vector Clocks for d-diagrams 

J(e) = least consistent cut that includes e

Theorem: For a recurrent vertex e, J(e) is guaranteed tostabilize after shift-diameter of the d-diagram.

J(ei+1) can be derived from J(ei) by shifting the cut.

Shift-diameter of a d-diagram: Maximum number of shift-edges in the shortest path between any two vertices

Lemma: On a d-diagram of a computation, shift-diameter is at most 2N.

Vector Clock for d-diagrams 

PV(e) = (V(e1), V(e2),..., V(en); I(e))

where V(ei) = vector timestamp of ei

           I(e) = V(ei+1 ) - V(ei )            n = shift diameter

Given p-timestamp, V(en+j) = V(en) + j * I(e)

I(a) = [2,2]

PV(a) = ([1,0], [3,0], [5,2]; [2,2]

Detecting global predicates

• Theorem: Sufficient to detect predicates on a finite part of the computation obtained by unrolling the d-diagram some number of times.

• Henceo If a predicate is never true in finite part, then it'll never be true

in the infinite computationo If a predicate becomes true on recurrent events, then it'll be true

infinitely often during the computation.

• Stopping Rule: the number of unrollings required is less than N (number of processes)

Recurrent Global State Detection

• Step 1: Ensure that the computation can be replayed (Deterministic Replay) [LM 87]

• Step 2: Compute a global state G (Global Snapshot Algorithm) [CL85]. Let the vector clock be Y.

• Step 3: Replay the computation detecting the first global state H that matches G (Conjunctive Predicate Detection [GW92] with vector clock Z)

• Step 4: Return (G,H) if Y != Z

Related Work

•  Global Predicate Detection on happened-before model (e.g. conjunctive, linear, temporal logic predicates)

– interpretation over finite traces

• Petri Nets– modeling and analyzing concurrent systems (versus a single computation)

• Message Sequence Charts (MSC)– incomparable to d-diagrams (e.g. require a message sent in a MSC to be received in the same MSC node)

Conclusions

  Recurrent global states

D-diagram as model of an infinite periodic poset suitable for distributed computation

Algorithm to timestamp events in a d-diagram

Algorithm to detect global predicates in a d-diagram

Future Work

  Minimum unrolling

Detecting general temporal logic formulas on d-diagrams

Questions?

• Recurrent global states• Modeling infinite computations as d-diagrams

• Vector clocks in d-diagrams• Predicate detection on d-diagrams

Backup Slides

Computation Model

• A distributed computation consists of N sequential processes P1, P2, ... PN

• A directed graph is used to represent the computation with the vertices corresponding to the events and the edges representing the dependencies.o Acyclic graphs can represent finite computations

• A consistent cut in the distributed graph represents a set of vertices such that if it contains a vertex e, it contains all its incoming neighbors as wello A consistent cut represents a valid global state

• The frontier of a consistent cut is the set of events who successors from the same process don't exist in the cut.o The frontier can be used to uniquely represent a consistent cut

Trace Model: Partial Order

• Partial order: Lamport’s happened-before model [Lamport 78]– suitable for concurrent and distributed programs– encodes exponential number of total orders, captures

bugs that may not be found with a total order

27

Vector Clock for d-diagrams 

We prove that after sufficient unrolling, the smallest consistent cuts containing different iteration of all recurrent vertices can be obtained simply by shifts. This iteration is called the shift-diameter.

We associate a p-timestamp with every recurrent vertex which allows us to generate the timestamp for any iteration of the vertex:

PV(e) = (V(e^1), V(e^2),..., V(e^n); I(e))

where V(e^i) = vector timestamp of e^i (the i-th iteration of e);           I(e) = V(e^(n+1)) - V(e^n)           n = shift diameter

Given p-timestamp, V(e^(n+j)) = V(e^n) + j * I(e)

Detecting global predicates

A predicate is a property defined on the states of the processes as well as channels• We show that its sufficient to detect predicates on a finite part of

the computation i.e. all possible consistent cuts that can be found in the full computation can be found in a finite subset obtained by rolling the d-diagram sufficient number of times.

• Henceo If a predicate is never true in finite part, then it'll never be true

in the infinite computationo If a predicate becomes true on recurrent events, then it'll be true

infinitely often during the computation.• We also show that the number of unrollings required are less than

N (number of processes)