Ymer:A Statistical Model Checker

Håkan L. S. YounesCarnegie Mellon University

Younes Ymer: A Statistical Model Checker 2

Probabilistic Model Checking

Given a model M, a state s, and a property , does hold in s for M ? Model: stochastic discrete event system Property: probabilistic temporal logic formula

Example: ≥0.1[ ≤5 full ]

Younes Ymer: A Statistical Model Checker 3

Statistical Solution Method

Use acceptance sampling to verify probabilistic properties Hypothesis: ≥ [] Observation: verify over a sample path

Bounds on probability of verification error Probability of false negative: ≤ Probability of false positive: ≤

Younes Ymer: A Statistical Model Checker 4

Error Bounds

Actual probability of holding

Pro

babi

lity

of e

rror

whe

n ve

rifyi

ng

≥ [

]

p1 p0

Indifference region

2

Younes Ymer: A Statistical Model Checker 5

Ymer at a Glance

Supports time-homogeneous generalized semi-Markov processes

Limited to time-bounded properties Distributed acceptance sampling (even

with sequential acceptance sampling) Purely statistical approach for verifying

nested probabilistic statements

Younes Ymer: A Statistical Model Checker 6

DistributedAcceptance Sampling

MasterAcceptance Sampling

Slavesimulation

Slavesimulation

Slave Masterregister

model & property

observation

observation

done

Younes Ymer: A Statistical Model Checker 7

Avoiding Sample Bias

Process observations as they come in? No, bias against observations that take a long

time to generate (long sample paths) Process observations according to a

predetermined schedule

1 2 1 1 2

1 1 2

Schedule:

Received:

Younes Ymer: A Statistical Model Checker 8

Case Study:Symmetric Polling System

Single server, n polling stations Stations are attended in cyclic order Each station can hold one message State space of size O(n·2n)

Server

…Polling stations

Younes Ymer: A Statistical Model Checker 9

ResultsP

erc

en

t of s

ing

le m

ach

ine 100

Size of state space102 104 106 108 1010 1012 1014

90

80

70

60

50

Machine 1: 733 MHz Pentium III

Machine 2: 500 MHz Pentium III

Younes Ymer: A Statistical Model Checker 10

Nested Probabilistic Statements: Robot Grid World

Probability is at least 0.9 that goal is reached within 100 seconds while periodically communicating ≥0.9[≥0.5[ ≤9 comm] ≤100 goal ]

Younes Ymer: A Statistical Model Checker 11

Statistical Verification ofNested Probabilistic Statements

Cannot verify path formula without some probability of error Probability of false negative: ≤ ′ Probability of false positive: ≤ ′

Observation error

Younes Ymer: A Statistical Model Checker 12

Performance Considerations

Verification error is independent of observation error Pick observation error to minimize effort

The same state may be visited along multiple sample paths Memoize verification results to avoid repeated

effort

Younes Ymer: A Statistical Model Checker 13

Robot Grid World (results)numericalmixedmixedstatisticalstatistical

Veri

fica

tion

tim

e (

seco

nd

s)

10−2

10−1

100

101

102

103

104

Size of state space102 104 106 108 1010 1012

≥0.9[≥0.5[ ≤9 comm] ≤100 goal ]

= 0.025

= 0.05

= = 10−2

Younes Ymer: A Statistical Model Checker 14

Robot Grid World:Effect of Memoization

Sa

mp

le s

ize

101

102

103

statisticalstatistical

statisticalstatistical

Un

iqu

e/v

isite

d s

tate

s

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

102 104 106 102 104 106

Size of state space Size of state space

Younes Ymer: A Statistical Model Checker 15

Availability

Source code is released under GPL http://sweden.autonomy.ri.cmu.edu/ymer/