Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard...

100
Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm http://www.cs.tau.ac.il/~TVLA http://www.cs.tau.ac.il/~msagiv/toplas02.pdf
  • date post

    19-Dec-2015
  • Category

    Documents

  • view

    215
  • download

    1

Transcript of Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard...

Page 1: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Compile-Time Verification of Properties of Heap Intensive Programs

Mooly SagivThomas Reps

Reinhard Wilhelm

http://www.cs.tau.ac.il/~TVLAhttp://www.cs.tau.ac.il/~msagiv/toplas02.pdf

Page 2: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

. . . and also• Tel-Aviv University

– G. Arnold– I. Bogudlov– G. Erez– N. Dor– T. Lev-Ami– R. Manevich– R. Shaham– A. Rabinovich– N. Rinetzky– G. Yorsh– A. Warshavsky

• Universität des Saarlandes– Jörg Bauer– Ronald Biber

• University of Wisconsin– F. DiMaio– D. Gopan– A. Loginov

• IBM Research– J. Field– H. Kolodner – M. Rodeh– E. Yahav

• Microsoft Research– G. Ramalingam

• University of Massachusetts– N. Immerman– B. Hesse

• The Technical University of Denmark– H.R. Nielson– F. Nielson

• Weizmann Institute/NYU– A. Pnueli

• Inria– B. Jeannet

Page 3: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Shape Analysis

• Determine the possible shapes of a dynamically allocated data structure at given program point

• Relevant questions:– Does x.next point to a shared element?– Does a variable point p to an allocated element every

time p is dereferenced– Does a variable point to an acyclic list?– Does a variable point to a doubly-linked list? – ?– Can a procedure create a memory-leak

Page 4: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Problem

• Programs with pointers and dynamically allocated data structures are error prone

• Automatically prove correctness

• Identify subtle bugs at compile time

Page 5: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Interesting Properties of Heap Manipulating Programs

• No null dereference

• No memory leaks

• Preservation of data structure invariant

• Correct API usage

• Partial correctness

• Total correctness

Page 6: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Example

rotate(List first, List last) {if ( first != NULL) {

last next = first;

first = first next;

last = last next;

last next = NULL;

}

}

lastfirst n n n

lastfirst n n n

n

lastfirst

n n n

n

lastfirst

n n n

n

lastfirst

n n

n

Page 7: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Interesting Properties

rotate(List first, List last) {if ( first != NULL) {

last next = first;

first = first next;

last = last next;

last next = NULL;

}

}

No null-de references

Page 8: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Interesting Properties

rotate(List first, List last) {if ( first != NULL) {

last next = first;

first = first next;

last = last next;

last next = NULL;

}

}

No null-de references

No memory leaks

Page 9: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Interesting Properties

rotate(List first, List last) {if ( first != NULL) {

last next = first;

first = first next;

last = last next;

last next = NULL;

}

}

No null-de references

No memory leaks

Returns an acyclic linked list

Partially correct

Page 10: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Partial CorrectnessList InsertSort(List x) { List r, pr, rn, l, pl; r = x; pr = NULL; while (r != NULL) { l = x; rn = r n; pl = NULL; while (l != r) { if (l data > r data) { pr n = rn; r n = l; if (pl = = NULL) x = r; else pl n = r; r = pr; break; } pl = l; l = l n; } pr = r; r = rn; } return x; }

typedef struct list_cell { int data; struct list_cell *n;} *List;

Page 11: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Partial CorrectnessList quickSort(List p, List q) {

if(p==q || q == NULL) return p;

List h = partition(p,q);List x = pn;p n = NULL;List low = quickSort(h, p);List high = quickSort(x, NULL);pn = high;return low;

}

Page 12: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Challenges

• Specification– Desired properties– Program Semantics

• Automatic Verification– Program Semantics Desired properties– Undecidable even for simple programs and

prooperties

Page 13: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Plan

• Concrete Interpretation of Heap

• Canonical Heap Abstraction

• Abstract Interpretation using Canonical Abstraction

• The TVLA system

• Applications

• Techniques for scaling

Page 14: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Logical Structures (Labeled Graphs)• Nullary relation symbols• Unary relation symbols• Binary relation symbols• FOTC over TC, express logical structure

properties• Logical Structures provide meaning for relations

– A set of individuals (nodes) U

– Interpretation of relation symbols in Pp0() {0,1}p1(v) {0,1}p2(u,v) {0,1}

Fixed

Page 15: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Representing Stores as Logical Structures

• Locations Individuals• Program variables Unary relations• Fields Binary relations• Example

– U = {u1, u2, u3, u4, u5}– x = {u1}, p = {u3}– n = {<u1, u2>, <u2, u3>, <u3, u4>, <u4, u5>}

u1 u2 u3 u4 u5xn n n n

p

Page 16: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Example: List Creationtypedef struct node { int val; struct node *next;} *List;

✔ No null dereferences

✔ No memory leaks

✔ Returns acyclic list

List create (…)

{

List x, t;

x = NULL;

while (…) do {

t = malloc();

t next=x;

x = t ;}

return x;

}

Page 17: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Example: Concrete Interpretation

x

tn n

t

x

n

x

t n

x

tn n

x

tn n

xtt

x

ntt

nt

x

t

x

t

xempty

return x

x = t

t =malloc(..);

tnext=x;

x = NULL

TF

Page 18: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Concrete Interpretation Rules

Statement Update formula

x =NULL x’(v)= 0

x= malloc() x’(v) = IsNew(v)

x=y x’(v)= y(v)

x=y next x’(v)= w: y(w) n(w, v)

x next=y n’(v, w) = (x(v) n(v, w)) (x(v) y(w))

Page 19: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Invariants

• No garbagev: {x PVar} w: x(w) n*(w, v)

• Acyclic list(x)v, w: x(v) n*(v, w) n+(w, v)

• Reverse (x)v, w, r: x(v) n*(v, w) n(w, r) n’(r, w)

Page 20: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Example: Abstract Interpretation

t

x

n

x

t n

x

tn n

xtt

x

ntt

nt

x

t

x

t

xempty

x

tn

n

x

tn

n

n

x

tn

t n

xn

x

tn

nreturn x

x = t

t =malloc(..);

tnext=x;

x = NULL

TF

Page 21: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

3-Valued Logical Structures

• A set of individuals (nodes) U

• Relation meaning– Interpretation of relation symbols in P

p0() {0,1, 1/2}p1(v) {0,1, 1/2}p2(u,v) {0,1, 1/2}

• A join semi-lattice: 0 1 = 1/2

Page 22: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Canonical Abstraction ()

• Partition the individuals into equivalence classes based on the values of their unary relations– Every individual is mapped into its equivalence class

• Collapse relations via

– pS (u’1, ..., u’k) = {pB (u1, ..., uk) | f(u1)=u’1, ..., f(uk)=u’k) }

• At most 2A abstract individuals

Page 23: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Canonical Abstraction

x = NULL;

while (…) do {

t = malloc();

t next=x;

x = t

}

u1x

t

u2 u3

u1x

t

u2,3

n n

n

n

Page 24: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

x

t

n nu2u1 u3

Canonical Abstraction

x = NULL;

while (…) do {

t = malloc();

t next=x;

x = t

} u1x

t

u2,3n

n

n

Page 25: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Canonical Abstraction and Equality

• Summary nodes may represent more than one element

• (In)equality need not be preserved under abstraction

• Explicitly record equality

• Summary nodes are nodes with eq(u, u)=1/2

Page 26: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Canonical Abstraction and Equality

x = NULL;

while (…) do {

t = malloc();

t next=x;

x = t

}

u1x

t

u2 u3

u1x

t

u2,3

eq

n n

n

n

eq eq

eq

eqeq

u2,3

eq

eq

eq

Page 27: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Canonical Abstraction

x = NULL;

while (…) do {

t = malloc();

t next=x;

x = t

}

u1x

t

u2 u3n n

u1x

t

u2,3n

n

Page 28: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Canonical Abstraction

• Partition the individuals into equivalence classes based on the values of their unary relations– Every individual is mapped into its equivalence class

• Collapse relations via

– pS (u’1, ..., u’k) = {pB (u1, ..., uk) | f(u1)=u’1, ..., f(u’k)=uk) }

• At most 2A abstract individuals

Page 29: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Canonical Abstraction

x = NULL;

while (…) do {

t = malloc();

t next=x;

x = t

}

u1x

t

u2 u3n n

u1x

t

u2,3n

n

Page 30: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Limitations

• Information on summary nodes is lost

Page 31: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Increasing Precision

• Global invariants– User-supplied, or consequence of the semantics

of the programming language– Naturally expressed in FOTC

• Record extra information in the concrete interpretation– Tunes the abstraction– Refines the concretization

Page 32: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Cyclicity relationc[x]() = v1,v2: x(v1) n*(v1,v2) n+(v2, v2)

c[x]()=0

c[x]()=0

u1x

t

u2 un…

u1x

t

u2..n

n

n

nn n

Page 33: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Cyclicity relationc[x]() = v1,v2: x(v1) n*(v1,v2) n+(v2, v2)

c[x]()=1

c[x]()=1

u1x

t

u2 un…

u1x

t

u2..n

n

n

nn n

n

Page 34: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Heap Sharing relation

is(v)=0

u1x

t

u2 un…

u1x

t

u2..n

n

n

is(v) = v1,v2: n(v1,v) n(v2,v) v1 v2

is(v)=0 is(v)=0

is(v)=0 is(v)=0

n n n

Page 35: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Heap Sharing relation

is(v)=0

u1x

t

u2 un…

is(v) = v1,v2: n(v1,v) n(v2,v) v1 v2

is(v)=1 is(v)=0

n n

n

n

u1x

t

u2n

is(v)=0 is(v)=1 is(v)=0

n

u3..n

n

n

Page 36: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Concrete Interpretation Rules

Statement Update formula

x =NULL x’(v)= 0

x= malloc() x’(v) = IsNew(v)

is’(v) =is(v) IsNew(v)

x=y x’(v)= y(v)

x=y next x’(v)= w: y(w) n(w, v)

x next=NULL n’(v, w) = x(v) n(v, w)

is’(v) = is(v) v1, v2: n(v1, v) x(v1) n(v2, v) x(v2) eq(v1, v2)

Page 37: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Reachability relationt[n](v1, v2) = n*(v1,v2)

u1x

t

u2 unn n n

t[n] t[n] t[n]

t[n]

t[n]

t[n]

u1x

t

u2..n

n

n

t[n]

t[n]

t[n]

...

Page 38: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

List Segments

u1

x

u2 u5nu3 u4 u6 u7 u8n n n n n n

y

u1

x

u2,3,4,6,7,8 u5n n

y

Page 39: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Reachability from a variable

• r[n,y](v) =w: y(w) n*(w, v)

u1

x

u2 u5nu3 u4 u6 u7 u8n n n n n n

y

r[n,y]=0 r[n,y]=0 r[n,y]=0 r[n,y]=1 r[n,y]=1 r[n,y]=1

u1

x

u2,3,4 u5n n n

y

u6,7,8

Page 40: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

• inOrder(v) = w: n(v, w) data(v) data(w)

• cfb(v) = w: f(v, w) b(w, v)

• tree(v)• dag(v)• Weakest Precondition

[Ramalingam, PLDI’02]• Learned via Inductive Logic Programming

[Loginov, CAV’05]• Counterexample guided refinement

Additional Instrumentation relations

Page 41: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Instrumentation (Summary)• Refines the abstraction

• Adds global invariants

• But requires update-formulas (generated automatically in TVLA2)

is(v) = v1,v2: n(v1,v) n(v2,v) v1 v2

is(v) v1,v2: n(v1,v) n(v2,v) v1 v2

(S#)={S : S , (S)= S#}

Page 42: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Abstract Interpretation

• Best Transformers

• Kleene Evaluation

• Kleene Evaluation + semantic reduction– Focus Based Transformers

Page 43: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

yx

yx ...

Evaluateupdateformulas

y

x

y

x

...

inverse canonical

y

x

y

xcanonical abstraction

xy

Best Transformer Transformer (x = x n)

Page 44: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Boolean Connectives [Kleene]

0 1/2 1

0 0 0 01/2 0 1/2 1/21 0 1/2 1

0 1/2 1

0 0 1/2 11/2 1/2 1/2 11 1 1 1

Page 45: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Boolean Connectives [Kleene]

0 1/2 1

0 0 0 01/2 0 1/2 1/21 0 1/2 1

0 1/2 1

0 0 1/2 11/2 1/2 1/2 11 1 1 1

Page 46: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Embedding

• A logical structure B can be embedded into a structure S via an onto function f (B f S) if the basic relations are preserved, i.e., pB(u1, .., uk) pS (f(u1), ..., f(uk))

• S is a tight embedding of B with respect to f if:– S does not lose unnecessary information, i.e.,– pS(u#

1, .., u#k) = {pB (u1 ..., uk) | f(u1)=u#

1, ..., f(uk)=u#k}

• Canonical Abstraction is a tight embedding

Page 47: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Embedding and Concretization

• Two natural choices– B 1(S) if B can be embedded into S via an

onto function f (B f S)

– B 2(S) if S is a tight embedding of B

Page 48: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Embedding Theorem

• Assume B f S, pB(u1, .., uk) pS (f(u1), ..., f(uk))

• Then every formula is preserved:

– If = 1 in S, then = 1 in B

– If = 0 in S, then = 0 in B

– If = 1/2 in S, then could be 0 or 1 in B

Page 49: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Embedding Theorem

u1x

t

u2,3n

n

v: x(v) 1=Yes

v: x(v)t(v) 1=Yes

v: x(v)y(v) 0=No

v1,v2: x(v1)n(v1, v2) ½=Maybe

v1,v2: x(v1)n(v1, v2) n*(v2, v1) 0=No

v1,v2: x(v1) n*(v1,v2) n+(v2, v2) 1/2=Maybe

Page 50: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

xy

Kleene Transformer (x = x n)

Page 51: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Semantic Reduction• Improve the precision of the analysis by

recovering properties of the program semantics

• A Galois connection (L1, , , L2)

• An operation op:L2L2 is a semantic reduction lL2 op(l)l (op(l)) = (l)

• Can be applied before and after basic operations

l

L1

L2 op

Page 52: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

“Focus”-Based Transformer (x = x n)

Kleene Evaluation

canonical

Focus(x n)

“Partial ”

xy

yx

yx

yx

x

x

y

y

yx

x

y

y

y

Page 53: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

The Focus Operation

• Focus: Formula(P(3-Struct) P(3-Struct))

• Generalizes materialization• For every formula

– Focus()(X) yields structure in which evaluates to a definite values in all assignments

– Only maximal in terms of embedding

– Focus() is a semantic reduction

– But Focus()(X) may be undefined for some X

Page 54: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

“Focus”-Based Transformer (x = x n)

Kleene Evaluation

canonical

Focus(x n)

“Partial ”

xy

yx

yx

yx

x

x

y

y

yx

x

y

y

y

w: x(w) n(w, v)

Page 55: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

The Coercion Principle

• Another Semantic Reduction• Can be applied after Focus or after Update or both• Increase precision by exploiting some structural

properties possessed by all stores (Global invariants)

• Structural properties captured by constraints

• Apply a constraint solver

Page 56: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Apply Constraint Solver

x

y

yr[n,y](v)=1

x

x

y

is(v)=0 x

yy

is(v)=0

Page 57: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Sources of Constraints

• Properties of the operational semantics

• Domain specific knowledge– Instrumentation predicates

• User supplied

Page 58: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Example Constraints

x(v1) x(v2)eq(v1, v2)

n(v, v1) n(v,v2)eq(v1, v2)

n(v1, v) n(v2,v)eq(v1, v2)is(v)

n*(v3, v4)t[n](v1, v2)

Page 59: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Apply Constraint Solver

y

is(v)=0x

x(v1) x(v2)eq(v1, v2)

y

is(v)=0

x

Page 60: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Apply Constraint Solver

y

is(v)=0x

n(v1, v) n(v2,v)eq(v1, v2)is(v)n(v1, v) is(v)eq(v1, v2) n(v2, v)

y

is(v)=0x

Page 61: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Summary Transformers

• Kleene evaluation yields sound solution

• Focus is statement specific implements partial concretization

• Coerce applies global constraints

Page 62: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Three Valued Logic Analysis (TVLA)T. Lev-Ami & R. Manevich

• Input (FOTC)

– Concrete interpretation rules

– Definition of instrumentation relations

– Definition of safety properties

– First Order Transition System (TVP)

• Output– Warnings (text)

– The 3-valued structure at every node (invariants)

Page 63: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

TVLA inputs

• TVP - Three Valued Program– Predicate declaration– Action definitions SOS

• Statements

• Conditions

– Control flow graph

• TVS - Three Valued Structure

Program independent

Page 64: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Null Dereferences

Demo

typedef struct element

{

int value;

struct element n;

} Element

bool search( int value, Element x)

{

Element c = x

while ( x != NULL ){

if (c val == value)

return TRUE;

c = c n;

}

return FALSE; }

276

Page 65: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Proving Correctness of Sorting Implementations (Lev-Ami, Reps, S,

Wilhelm ISSTA 2000)• Partial correctness

– The elements are sorted– The list is a permutation of the original list

• Termination– At every loop iterations the set of elements

reachable from the head is decreased

Page 66: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Sortedness

u1x

t

u2 unn n n

dle dle dle

dle

dle

dle

u1x

t

u2..n

n

n

dle

dle

dle

...

Page 67: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Example: SortednessinOrder(v) = v1: n(v,v1) dle(v, v1)

u1x

t

u2 unn n

dle dle dle

dle

dle

dle

u1x

t

u2..n

n

n

dle

dledle

inOrder = 1 inOrder = 1 inOrder = 1

inOrder = 1 inOrder = 1

n...

Page 68: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Example: InsertSort

Run Demo

List InsertSort(List x) { List r, pr, rn, l, pl; r = x; pr = NULL; while (r != NULL) { l = x; rn = r n; pl = NULL; while (l != r) { if (l data > r data) { pr n = rn; r n = l; if (pl = = NULL) x = r; else pl n = r; r = pr; break; } pl = l; l = l n; } pr = r; r = rn; } return x; }

typedef struct list_cell { int data; struct list_cell *n;} *List;

pred.tvp

actions.tvp

Page 69: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Example: InsertSort

Run Demo

List InsertSort(List x) { if (x == NULL) return NULL pr = x; r = x->n; while (r != NULL) {

pl = x; rn = r->n; l = x->n; while (l != r) {

pr->n = rn ; r->n = l;

pl->n = r; r = pr; break; }

pl = l; l = l->n;

} pr = r; r = rn;

}

typedef struct list_cell { int data; struct list_cell *n;} *List;

14

Page 70: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Example: Mark and Sweepvoid Sweep() { unexplored = Universe collected = while (unexplored ) { x = SelectAndRemove(unexplored) if (x marked) collected = collected {x} } assert(collected = = Universe – Reachset(root) )}

void Mark(Node root) { if (root != NULL) { pending = pending = pending {root} marked = while (pending ) { x = SelectAndRemove(pending) marked = marked {x} t = x left if (t NULL) if (t marked) pending = pending {t} t = x right if (t NULL) if (t marked) pending = pending {t} } } assert(marked = = Reachset(root))}

Page 71: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Example: Mark and Sweepvoid Mark(Node root) { if (root != NULL) { pending = pending = pending {root} marked = while (pending ) { x = SelectAndRemove(pending) marked = marked {x} t = x left if (t NULL) if (t marked) pending = pending {t} t = x right if (t NULL) if (t marked) pending = pending {t} } } assert(marked = = Reachset(root))}

Run Demo

pred.tvp

pred_set.tvp

actions.tvp

actions_set.tvp

Page 72: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Example: Mark and Sweepvoid Sweep() { unexplored = Universe collected = while (unexplored ) { x = SelectAndRemove(unexplored) if (x marked) collected = collected {x} } assert(collected = = Universe – Reachset(root) )}

void Mark(Node root) { if (root != NULL) { pending = pending = pending {root} marked = while (pending ) { x = SelectAndRemove(pending) marked = marked {x} t = x left if (t NULL) if (t marked) pending = pending {t}/* t = x right * if (t NULL) * if (t marked) * pending = pending {t} */ } } assert(marked = = Reachset(root))}

Run Demo

Page 73: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Lightweight Specification"correct usage" rules a client must follow

"call open() before read()"

Certificationdoes the client program satisfy the lightweight specification?

Verification of Safety Properties(PLDI’02, 04)

Componenta library with cleanly encapsulated state

Clienta program that uses

the library

The Canvas Project (with IBM Watson)(Component Annotation, Verification and Stuff)

Page 74: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Prototype Implementation

• Applied to several example programs– Up to 5000 lines of Java

• Used to verify– Absence of concurrent modification

exception – JDBC API conformance– IOStreams API conformance

Page 75: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Analysis Times

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

ISPath InputStream5 InputStream5b inputStream6db KernelBenchmark 1

JDBC Example JDBC Example2

SQLExecutor

IOStreamsIOStreamsIOStreamsIOStreamsIOStreamsCMPJDBCJDBCJDBC

Benchmark

Tim

e (

se

c)

Page 76: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Summary

• Canonical abstraction is powerful– Intuitive– Adapts to the property of interest– More instrumentation may mean more efficient

• Used to verify interesting program properties– Very few false alarms

• But scaling is an issue

Page 77: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Scaling for Larger Programs

• Staged Analyses • Represent 3-valued structures with BDDs [Manevich

SAS’02]• Coercer Abstractions [Manevich SAS’04]• Reduce static costs• Handling procedures• Assume/Guarantee Reasoning

– Use procedure specifications [Yorsh, TACAS’04]– Decision procedures for linked data structures

[Immerman, CAV’04, Lev-Ami, CADE’05, Yorsh FOSSACS06]

Page 78: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Scaling

• Staged analysis• Reduce static costs• Controlled complexity

– More coarse abstractions [Manevich SAS’04]– Counter example based refinement

• Exploit “good” program properties– Encapsulation & Data abstraction

• Handle procedures efficiently

Page 79: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Partially DisjunctiveHeap Abstraction (Manevich, SAS’04)

• Use a heap-similarity criterion– We defined similarity by universe congruence

• Merge similar heaps

• Avoid merging dissimilar heaps

• The same concrete state can belong to more than one abstract value

Page 80: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

x

t n

x

tn n

Partially Disjunctive Abstraction

x

t n

Page 81: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Running times

02,0004,0006,0008,00010,00012,00014,00016,00018,00020,000

DSW

Inpu

tStre

am5

Inpu

tStre

am5b

Inpu

tStre

am6

SQLExe

cuto

r

Kerne

lBen

ch.1

GC.mar

k

se

co

nd

s

PowersetPartial

Page 82: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Interprocedural Analysis

Noam Rinetzky

www.cs.tau.ac.il/~maon

Page 83: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

main() {

append(y,z);

}

• Procedure input/output relation– Not reachable Not effected

– proc: local (reachable) heap local heap

How to tabulate procedures?

append(List p, List q) {…

}

nx

n

t

y

n

z

n

p q p q

n

y z

p qn

xn

t

Page 84: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

main() {

append(y,z);

}

ynn

z

n

How to handle sharing?• External sharing may break the functional view

append(List p, List q) {…

}

nn

t z

x n

p qnn

p qnn

t

y

p qnn n

x

Page 85: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

append(y,z);

What’s the difference?

nn

t z

y

x

1st Example 2nd Example

append(y,z);

nx

n

t y z

Page 86: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Cutpoints

• An object is a cutpoint for an invocation– Reachable from actual parameters– Not pointed to by an actual parameter– Reachable without going through a parameter

append(y,z) append(y,z)

y

x

nn

t z

ynn

t zn n

Page 87: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Main Results(POPL’05)

• Concrete operational semantics– Sequential programs– Local heap– Track cutpoints– Storeless

• good for shape abstractions

– Observational equivalent with “standard” global store-based heap semantics

• Java and “clean” C

• Abstractions– Shape Analysis of singly-linked lists– May-alias [Deutsch, PLDI 04]

Page 88: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Introducing local heap semanticsOperational semantics

Abstract transformer

Local heap Operational semantics

~’ ’

Page 89: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Main results(SAS’05)• Cutpoint freedom• Non-standard concrete semantics

– Verifies that an execution is cutpoint-free– Local heaps

• Interprocedural shape analysis– Conservatively verifies

• program is cutpoint free • Desired properties

– Partial correctness of quicksort

– Procedure summaries

• Prototype implementation

Page 90: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Memory states

• A memory state encodes a local heap– Local variables of the current procedure

invocation– Relevant part of the heap

• Relevant Reachable

x

y

nn

t z

p q

main append

Page 91: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Abstract semantics

• Conservatively apply statements using 3-valued

logic (with the non-standard semantics)– Use canonical abstraction– Reinterpret FO formulas using Kleene value

Page 92: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

1. Verify cutpoint

freedom

2 Compute input

… Execute callee …

3 Combine output

append body

Procedure calls

p q

pn

q

y zxn

y zxn n

append(y,z)

append(p,q)

Page 93: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Interprocedural shape analysis

• Procedure input/output relation

p qn

nrqrp rp

qpn

nrp rq

n rp

q q

rqrq

qprp

qprp rq

n rprq

Input Output

rp

Page 94: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Interprocedural shape analysis• Reusable procedure summaries

– Heap modularity

qprp rq

n rp

qprp rq

g h i kn n

g h i kn

rgrgrg rh rh rirkrhrgrirk

append(h,i)

yx zn

nn n rx ry rz

yx zn

nn rz rx ryrxrxrx ryrx rx

append(y,z)

yn

zxappend(y,z)y zx

rx ry rzryrx ry

Page 95: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Prototype implementation

• TVLA based analyzer • Soot-based Java front-end• Parametric abstraction

Data structure Verified properties

Singly linked list Cleanness, acyclicity

Sorting (of SLL) + Sortedness

Unshared binary trees Cleaness, tree-ness

Page 96: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Iterative vs. Recursive (SLL)

0102030405060708090

100

Program

Sec

onds

Iterative

Recursive

585

Page 97: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Inline vs. Procedural abstraction

0

5

10

15

20

1 2 4 8Number of lists

Meg

ab

yte

s

Inline

Proc. call

020406080

100120140160

1 2 4 8Number of lists

Seco

nd

s

Inline

Proc. call

// Allocates a list of

// length 3

List create3(){

}

main() {

List x1 = create3();

List x2 = create3();

List x3 = create3();

List x4 = create3();

}

Page 98: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Call string vs. Relational vs. CPF[Rinetzky and Sagiv, CC’01] [Jeannet et al., SAS’04]

0123456789

10

insert delete rev rev8

Meg

abyt

es Call String

Relational

CPF

0

20

40

60

80

100

120

140

160

insert delete rev rev8

Sec

on

ds Call String

Relational

CPF

Page 99: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Summary

• Cutpoint freedom

• Non-standard operational semantics

• Interprocedural shape analysis– Partial correctness of quicksort

• Prototype implementation

Page 100: Compile-Time Verification of Properties of Heap Intensive Programs Mooly Sagiv Thomas Reps Reinhard Wilhelm TVLA msagiv/toplas02.pdf.

Summary

• Reasoning about the heap is challenging

• [Parametric] Abstraction is necessary

• Canonical abstraction is powerful

• Useful for programs with arrays [Gopan POPL’05]

• Information lost by canonical abstraction– Correlations between list lengths