Arti cial Intelligence in Theorem Proving · 2019-07-08 · Overview Last Lecture theorem proving...

Artificial Intelligence in Theorem Proving

Cezary Kaliszyk

VTSA

Overview

Last Lecturetheorem proving problems

premise selection

deep learning for theorem proving

state estimation

Today

automated reasoning

learning in classical ATPs

learning for tableaux

reinforcement learning in TP

longer proofs

Cezary Kaliszyk Artificial Intelligence in Theorem Proving 2 / 72

What about ATPs

Proof by contradiction

Assume that the conjecture does not hold

Derive that axioms and negated conjecture imply ⊥

SaturationConvert problem to CNF

Enumerate the consequences of the available clauses

Goal: get to the empty clause

RedundanciesSimplify or eliminate some clauses (contract)


Calculus

Ordered

Resolution

Aσ strictly maximal wrt Cσ and B maximal wrt Dσ.

Equality axioms?

Ordered

Paramodulation

(s = t)σ and L[s ′]σ′ maximal in their clauses.


Calculus

Ordered Resolution


Equality axioms?

Ordered

Paramodulation



Calculus

Ordered Resolution


Equality axioms?

Ordered Paramodulation



Completion


Superposition Calculus

Basis ofE, Vampire, Spass, Prover9, ≈Metis


Beyond the Calculus

Tautology Deletion

a ∨ b ∨ ¬a ∨ d

Subsumption (forward and backward)e.g. E uses Feature Vector Indexing

{C1, C2, C3, C4}

{C3, C4}0

{C2}1

{C1}

2

{C3}0

{C4}2

{C2}0

{C1}1

{C3}0

{C4}0

{C2}1

{C1}0


Still...

fof(6, axiom,![X1]:![X2]:![X4]:gg(X1,sup_sup(X1,X2,X4)),file(’i/f/1/goal_138__Q_Restricted_Rewriting.qrstepfof(32, axiom,![X1]:![X2]:gg(set(product_prod(X1,X1)),transitive_rtrancl(X1,X2)),file(’i/f/1/goal_138__Q_Refof(55, axiom,![X1]:![X19]:![X20]:(member(product_prod(X1,X1),X19,X20)=>member(product_prod(X1,X1),X19,tranfof(68, axiom,![X1]:![X5]:![X3]:![X36]:![X20]:![X37]:![X16]:(ord_less_eq(set(product_prod(X1,X3)),X36,X20)=fof(70, axiom,![X1]:![X20]:transitive_rtrancl(X1,transitive_rtrancl(X1,X20))=transitive_rtrancl(X1,X20),filfof(74, axiom,![X1]:![X24]:![X34]:![X33]:((~(member(X1,X24,X34))=>member(X1,X24,X33))=>member(X1,X24,sup_sufof(78, axiom,![X1]:![X11]:![X13]:transitive_rtrancl(X1,sup_sup(set(product_prod(X1,X1)),transitive_rtranclfof(79, axiom,![X1]:![X22]:![X39]:(member(X1,X22,collect(X1,X39))<=>pp(aa(X1,bool,X39,X22))),file(’i/f/1/gofof(85, axiom,![X1]:(semilattice_sup(X1)=>![X23]:![X24]:![X22]:(ord_less_eq(X1,sup_sup(X1,X23,X24),X22)<=>(fof(86, axiom,![X1]:![X11]:relcomp(X1,X1,X1,transitive_rtrancl(X1,X11),transitive_rtrancl(X1,X11))=transitifof(98, axiom,![X1]:![X33]:![X34]:(gg(set(X1),X34)=>(ord_less_eq(set(X1),X33,X34)<=>sup_sup(set(X1),X33,X34fof(99, axiom,![X1]:![X33]:![X34]:ord_less_eq(set(X1),X33,sup_sup(set(X1),X33,X34)),file(’i/f/1/goal_138__Qfof(100, axiom,![X3]:![X1]:supteq(X1,X3)=sup_sup(set(product_prod(term(X1,X3),term(X1,X3))),supt(X1,X3),id(fof(102, axiom,![X1]:![X34]:![X33]:ord_less_eq(set(X1),X34,sup_sup(set(X1),X33,X34)),file(’i/f/1/goal_138__fof(103, axiom,![X1]:![X33]:![X18]:![X34]:(ord_less_eq(set(X1),X33,X18)=>(ord_less_eq(set(X1),X34,X18)=>ordfof(109, axiom,![X1]:![X34]:![X33]:(gg(set(X1),X33)=>(ord_less_eq(set(X1),X34,X33)=>sup_sup(set(X1),X33,X34fof(114, axiom,![X1]:![X33]:![X18]:![X34]:![X48]:(ord_less_eq(set(X1),X33,X18)=>(ord_less_eq(set(X1),X34,X4fof(116, axiom,![X1]:![X33]:ord_less_eq(set(X1),X33,X33),file(’i/f/1/goal_138__Q_Restricted_Rewriting.qrstefof(125, axiom,![X1]:![X24]:![X33]:![X34]:(member(X1,X24,X33)=>(~(member(X1,X24,X34))=>member(X1,X24,minus_fof(127, axiom,![X1]:![X24]:![X33]:![X34]:(member(X1,X24,minus_minus(set(X1),X33,X34))=>~((member(X1,X24,X3fof(131, axiom,![X1]:![X33]:(gg(set(X1),X33)=>collect(X1,aTP_Lamp_a(set(X1),fun(X1,bool),X33))=X33),file(’ifof(134, axiom,![X1]:(order(X1)=>![X35]:![X49]:((gg(X1,X35)&gg(X1,X49))=>(ord_less_eq(X1,X35,X49)=>(ord_lesfof(136, axiom,![X1]:(preorder(X1)=>![X35]:![X49]:![X50]:(ord_less_eq(X1,X35,X49)=>(ord_less_eq(X1,X49,X50)fof(143, axiom,![X1]:![X33]:![X34]:(ord_less_eq(set(X1),X33,X34)<=>![X52]:(gg(X1,X52)=>(member(X1,X52,X33)=fof(160, axiom,![X1]:![X39]:![X35]:![X33]:(pp(aa(X1,bool,X39,X35))=>(member(X1,X35,X33)=>?[X30]:(gg(X1,X30)fof(171, axiom,![X1]:![X65]:![X66]:(pp(aa(X1,bool,aTP_Lamp_a(set(X1),fun(X1,bool),X65),X66))<=>member(X1,X6fof(186, axiom,![X67]:semilattice_sup(set(X67)),file(’i/f/1/goal_138__Q_Restricted_Rewriting.qrsteps_comp_sfof(187, axiom,![X67]:preorder(set(X67)),file(’i/f/1/goal_138__Q_Restricted_Rewriting.qrsteps_comp_supteq_sfof(188, axiom,![X67]:order(set(X67)),file(’i/f/1/goal_138__Q_Restricted_Rewriting.qrsteps_comp_supteq_subsfof(207, conjecture,ord_less_eq(set(product_prod(term(a,b),term(a,b))),relcomp(term(a,b),term(a,b),term(a,b


Still the search space is huge: What can we learn?

What has been learnedCASC: Strategies

AIM: Hints

Hammers: Premises

What can be chosen in Superposition calculus

Term ordering

(Negative) literal selection

Clause selection


E-Prover given-clause loop

Most important choice: unprocessed clause selection [Schulz 2015 ]


Learning for E: Data Collection

Mizar top-level theorems [Urban 2006 ]

Encoded in FOF

32,521 Mizar theorems with ≥ 1 proof

training-validation split (90%-10%)

replay with one strategy

Collect all CNF intermediate steps

and unprocessed clauses when proof is found


Deep Network Architectures

Clause EmbedderNegated conjecture

embedder

Concatenate

Fully Connected(1024 nodes)

Fully Connected(1 node)

Logistic loss

Clause tokens Negated conjecture tokens

Conv 5 (1024) + ReLU

Input token embeddings



Max Pooling

Overall network Convolutional Embedding

Non-dilated and dilated convolutionsCezary Kaliszyk Artificial Intelligence in Theorem Proving 12 / 72

Recursive Neural Networks

Curried representation of first-order statements

Separate nodes for apply, or, and, not

Layer weights learned jointly for the same formula

Embeddings of symbols learned with rest of network

Tree-RNN and Tree-LSTM models1

1Relation to graphsCezary Kaliszyk Artificial Intelligence in Theorem Proving 13 / 72

Model accuracy

Model Embedding Size Accuracy: 50-50% splitTree-RNN-256×2 256 77.5%Tree-RNN-512×1 256 78.1%

Tree-LSTM-256×2 256 77.0%Tree-LSTM-256×3 256 77.0%Tree-LSTM-512×2 256 77.9%

CNN-1024×3 256 80.3%?CNN-1024×3 256 78.7%

CNN-1024×3 512 79.7%CNN-1024×3 1024 79.8%

WaveNet-256×3×7 256 79.9%?WaveNet-256×3×7 256 79.9%WaveNet-1024×3×7 1024 81.0%

WaveNet-640×3×7(20%) 640 81.5%?WaveNet-640×3×7(20%) 640 79.9%

? = train on unprocessed clauses as negative examples


Improving Proof Search inside E

Overview

Processed ClausesUnprocessed

Clauses

Select oneUsing a deep neural

network

Superposition

ProblemDeep neural network evaluation is slow

Slower than combining selected clause with all processed clauses2

2State of 2016Cezary Kaliszyk Artificial Intelligence in Theorem Proving 15 / 72

Hybrid heuristic

Optimizations for performance

Batching

Combining TF with auto

102 103 104 105

Processed clause limit

0%

20%

40%

60%

80%

100%

Perc

ent

unpro

ved

Pure CNN

Hybrid CNN

Pure CNN; Auto

Hyrbid CNN; Auto

102 103 104 105

Processed clause limit

0%

20%

40%

60%

80%

100%

Perc

ent

unpro

ved

Auto

WaveNet 640*

WaveNet 256

WaveNet 256*

WaveNet 640

CNN

CNN*


Harder Mizar top-level statements

Model DeepMath 1 DeepMath 2 Union of 1 and 2

Auto 578 581 674?WaveNet 640 644 612 767?WaveNet 256 692 712 864

WaveNet 640 629 685 997?CNN 905 812 1,057

CNN 839 935 1,101

Total (unique) 1,451 1,458 1,712

Overall proved 7.4% of the harder statements

Batching and hybrid necessary

Model accuracy unsatisfactory


ENIGMA [Jakubuv,Urban 2017 ]

Evaluation on AIM

E’s auto-schedule: 261

Single best strategy: 239

Different trained models: 337

Accuracy: 97.6%

Looping and boosting

Still in 30s: best trained strategy: 318


Automated Theorem Proving

Historical dispute: Gentzen and Hilbert

Today two communities: Resolution (-style) and Tableaux

Possible answer: What is better in practice?

Say the CASC competition or ITP libraries?

Since the late 90s: resolution (superposition)

But still so far from humans?We can do learning much better for Tableaux

And with ML beating brute force search in games, maybe?


leanCoP: Lean Connection Prover [Otten 2010 ]

Connected tableaux calculusGoal oriented, good for large theories

Regularly beats Metis and Prover9 in CASC (ATP Systems Competition)

despite their much larger implementation

Compact Prolog implementation, easy to modify

Variants for other foundations: iLeanCoP, mLeanCoP

First experiments with machine learning: MaLeCoP

Easy to imitate

leanCoP tactic in HOL Light


Lean connection Tableaux

Very simple rules:

Extension unifies the current literal with a copy of a clause

Reduction unifies the current literal with a literal on the path

{}, M, PathAxiom

C , M, Path ∪ {L2}C ∪ {L1}, M, Path ∪ {L2}

Reduction

C2 \ {L2}, M, Path ∪ {L1} C , M, Path

C ∪ {L1}, M, PathExtension


Example lean connection proof

Clauses:

c1 : P(x)

c2 : R(x , y) ∨ ¬P(x) ∨ Q(y)

c3 : S(x) ∨ ¬Q(b)

c4 : ¬S(x) ∨ ¬Q(x)

c5 : ¬Q(x) ∨ ¬R(a, x)

c6 : ¬R(a, x) ∨ Q(x)

Tableau: P(a)

R(a, b)

¬R(a, b) Q(b)

¬Q(b) ¬R(a, b)

¬P(a) Q(b)

S(b)

¬S(b) ¬Q(b)

¬Q(b)


leanCoP Example [Otten’15 ]

Formula to prove:

DNF:

Matrix:

Tableaux:


leanCoP: Basic Code

1 prove([Lit|Cla],Path ,PathLim ,Lem ,Set) :-

23 (-NegLit=Lit;-Lit=NegLit) ->

4 (

567 member(NegL ,Path),unify_with_occurs_check(NegL ,NegLit)

8 ;

9 lit(NegLit ,NegL ,Cla1 ,Grnd1),

10 unify_with_occurs_check(NegL ,NegLit),

11121314 prove(Cla1 ,[Lit|Path],PathLim ,Lem ,Set)

15 ),

1617 prove(Cla ,Path ,PathLim ,Lem ,Set).

18 prove([],_,_,_,_).


leanCoP: Actual Code (Optimizations, No history)

1 prove([Lit|Cla],Path ,PathLim ,Lem ,Set) :-

2 \+ (member(LitC ,[Lit|Cla]), member(LitP ,Path),LitC==LitP),

3 (-NegLit=Lit;-Lit=NegLit) ->

4 (

5 member(LitL ,Lem), Lit==LitL

6 ;

7 member(NegL ,Path),unify_with_occurs_check(NegL ,NegLit)

8 ;

9 lit(NegLit ,NegL ,Cla1 ,Grnd1),

10 unify_with_occurs_check(NegL ,NegLit),

11 ( Grnd1=g -> true ;

12 length(Path ,K), K<PathLim -> true ;

13 \+ pathlim -> assert(pathlim), fail ),

14 prove(Cla1 ,[Lit|Path],PathLim ,Lem ,Set)

15 ),

16 ( member(cut ,Set) -> ! ; true ),

17 prove(Cla ,Path ,PathLim ,[Lit|Lem],Set).

18 prove([],_,_,_,_).


First experiment: MaLeCoP in Prolog [Tableaux 2011 ]

Select extension stepsUsing external advice

Slow implementation

1000 less inf per second

Can avoid 90% inferences!

Important: Strategies

leanCoP

cache

advisor

Otherprovers

SNoWlearningsystem


What about efficiency: FEMaLeCoP [LPAR 2015 ]

Advise the:selection of clause for every tableau extension step

Proof state: weighted vector of symbols (or terms)

extracted from all the literals on the active path

Frequency-based weighting (IDF)

Simple decay factor (using maximum)

Consistent clausificationformula ?[X]: p(X) becomes p(’skolem(?[A]:p(A),1)’)

Predictor: Custom sparse naive Bayes

association of the features of the proof states

with contrapositives used for the successful extension steps


FEMaLeCoP: Data Collection and Indexing

Extension of the saved proofs

Training Data: pairs (path, used extension step)

External Data Indexing (incremental)

te num: number of training examples

pf no: map from features to number of occurrences ∈ Qcn no: map from contrapositives to numbers of occurrences

cn pf no: map of maps of cn/pf co-occurrences

Problem Specific Data

Upon start FEMaLeCoP reads

only current-problem relevant parts of the training data

cn no and cn pf no filtered by contrapositives in the problem

pf no and cn pf no filtered by possible features in the problem


Efficient Relevance (1/2)

Estimate the relevance of each contrapositive ϕ by

P(ϕ is used in a proof in state ψ | ψ has features F(γ))

where F (γ) are the features of the current path.

Assuming the features are independent, this is:

P(ϕ is used in ψ’s proof)

·∏

f∈F (γ)∩F (ϕ)P(ψ has feature f | ϕ is used in ψ’s proof

)·∏

f∈F (γ)−F (ϕ)P(ψ has feature f | ϕ is not used in ψ’s proof

)·∏

f∈F (ϕ)−F (γ)P(ψ does not have f | ϕ is used in ψ’s proof

)


Efficient Relevance (2/2)

All these probabilities can be estimated (using training examples):

σ1 ln t +∑

f∈(f∩s)

i(f ) lnσ2s(f )

t+ σ3

∑f∈(f−s)

i(f ) + σ4

∑f∈(s−f )

i(f ) ln(1−s(f )

t)

where

f are the features of the path

s are the features that co-occurred with ϕ

t = cn no(ϕ)

s = cn fp no(ϕ)

i is the IDF

σ∗ are experimentally chosen parameters


Inference speed ... drops to about 40%

Prover Proved (%)

OCaml-leanCoP 574 (27.6%)FEMaLeCoP 635 (30.6%)together 664 (32.0%)

(evaluation on MPTP bushy problems, 60 s)

On various datasets, 3–15% problems more solved than leanCoP(run for double the time)


What about stronger learning?

Yes, but... [Michalewski 2017 ]

If put directly, huge times needed

Still improvement small

NBayes vs XGBoost on 2h timeout

Preliminary experiments with deep learning [Olsak 2017 ]

So far too slow


Is theorem proving just a maze search?

Yes and NO!The proof search tree is not the same as the tableau tree!

Unification can cause other branches to disappear.

Can we provide a tree search like interface?

Two functions sufficestart : problem→ state

action : action→ state

wherestate = 〈action list× remaining goal-paths〉


Is it ok to change the tree?

Most learning for games sticks to game dynamics

Only tell it how to do the moves

Why is it ok to skip other branches

Theoretically ATP calculi are complete

Practically most ATP strategies incomplete

In usual 30s – 300s runsDepth of proofs with backtracking: 5–7 (complete)

Depth with restricted backtracking: 7–10 (more proofs found!)

But with random playouts: depth hundreds of thousands!

Just unlikely to find a proof → learning


Monte Carlo First Try: monteCoP

Use Monte Carlo playouts to guide restricted backtracking

Improves on leanCoP, but not by a big margin

Potential still limited by depth

0 20 40 60 80 100 120 140

450

460

470

480

490

500

maxIterations

Pro

ble

ms

solv

ed

0 20 40 60 80 100 120 140100

150

200

250

300

350

smax

Pro

ble

ms

solv

ed


“Simple” learning in leanCoP

FEMaLeCoP: Speed: 40%On various datasets, 3–15% problems more solved than leanCoP

XGBoost: Speed: 8%But more precise and again small improvement

Monte CarloImproves on leanCoP, but not by a big margin

Change in game moves

More inspiration from games?


AlphaZero (1/3) [Silver et al.]


How to select the best action? [Szepesvari 2006 ]

IntuitionGiven some prior probabilities

And having done some experiments

Which action to take?

(later extended to sequences of actions in a tree)

Monte Carlo Tree Search with Upper Confidence Bounds for Trees

Select node n maximizingwi

ni+ c · pi ·

√lnN

ni

wherewi

niaverage reward pi action i prior

N number of experiments ni action i experiments









ni+ c · pi ·

√lnN

ni

where

wi











ni+ c · pi ·

√lnN

ni

wherewi




MCTS tree for WAYBEL 0:28

r=0.3099n=1182

Tableaux starting axiom

p=0.24r=0.3501n=548

p=0.21r=0.1859n=28...

p=0.10r=0.2038

n=9...

p=0.13r=0.2110n=14...

p=0.14r=0.2384n=21...

p=0.14r=0.3370n=181...

p=0.20r=0.3967n=279

p=0.30r=0.1368n=14...

p=0.15r=0.0288

n=2...

p=0.56r=0.4135n=262

p=0.66r=0.4217n=247

36 more MCTS tree levels until proved

Subset(c2, powerset(carrier(c1))p=0.18r=0.2633

n=8...

p=0.17r=0.2554

n=6...

Subset(union(c2),carrier(c1))

upper(c1)p=0.08r=0.1116

n=3...

RelStr(c1)p=0.19r=0.2289n=58...

p=0.22r=0.1783n=40...

p=0.35r=0.2889n=536...


Learn Policy and Value

Policy: Which actions to take?

Proportions predicted based on proportions in similar states

Explore less the actions that were “bad” in the past

Explore more and earlier the actions that were “good”

Value: How good (close to a proof) is a state?

Reward states that have few goals

Reward easy goals

Where to get training data?

Explore 1000 nodes using UCT

Select the most visited action and focus on it for this proof

A sequence of selected actions can train both policy and value


Mizar TPTP problems: train (29272) and test (3252) sets

BaselineSystem leanCoP playouts UCT

Train 10438 4184 7348

Test 1143 431 804

10 iterations

Iteration 1 2 3 4 5

6 7 8 9 10

Train 12325 13749 14155 14363 14403 14431 14342 14498 14481 14487

Test 1354 1519 1566 1595 1624

1586 1582 1591 1577 1621

More TimeleanCoP, 4M inferences, strategies 1396

rlCoP union 1839


Mizar TPTP problems: train (29272) and test (3252) sets

BaselineSystem leanCoP playouts UCTTrain 10438 4184 7348Test 1143 431 804

10 iterations

Iteration 1 2 3 4 5 6 7 8 9 10

Train 12325 13749 14155 14363 14403 14431 14342 14498 14481 14487Test 1354 1519 1566 1595 1624 1586 1582 1591 1577 1621

More TimeleanCoP, 4M inferences, strategies 1396

rlCoP union 1839


n0

p=1.00r=0.3944v=1386

p=0.08r=0.0621

v=3

p=0.08r=0.1048

v=5

p=0.08r=0.0544

v=3

p=0.09r=0.0712

v=4

p=0.08r=0.0443

v=3

p=0.26r=0.4016v=1356

p=0.07r=0.0504

v=2

p=0.07r=0.0628

v=3

p=0.08r=0.0544

v=3

p=0.08r=0.0660

v=3

p=0.38r=0.0518

v=1

p=0.47r=0.0774

v=1p=0.14

..58.. ..61..

p=0.37r=0.0942

v=2

p=0.47r=0.1391

v=2p=0.16

..57..p=0.03

r=0.0615v=1

..58..

p=0.02r=0.1099

v=1..60..

..62..

p=0.29r=0.0448

v=1

p=0.36r=0.0613

v=1..2..

..58.. ..61..

p=0.75r=0.0367

v=3p=0.25

p=0.02r=0.0214

v=2..60..

p=0.03 ..61..

p=0.27p=0.73

r=0.0353v=2

p=0.03r=0.0212

v=1..60..

..62..

p=0.68r=0.4067v=1322

p=0.32r=0.2054

v=33

p=0.31r=0.1136

v=28

p=0.09r=0.0297

v=2

p=0.16r=0.4148v=1288

p=0.11r=0.0595

v=3..5..

p=0.52r=0.1158

v=27..6..

p=0.89r=0.1145

v=26p=0.11

p=0.09p=0.91

r=0.1138v=25

p=0.91r=0.1131

v=24p=0.09

p=0.10p=0.90

r=0.1109v=23

p=0.88r=0.1086

v=22p=0.12

p=0.89r=0.1055

v=21p=0.11

p=0.89r=0.1021

v=20p=0.11

p=0.90r=0.0959

v=19p=0.10

p=0.11p=1.00

r=0.0838v=17

STEPS:2

p=0.07r=0.0144

v=1

p=0.50r=0.0703

v=10

p=0.23r=0.0250

v=2

p=0.21r=0.1899

v=3

..4..p=0.64

r=0.0922v=7

p=0.36r=0.0109

v=2

p=0.58p=0.42

r=0.1138v=5

..8..p=0.32

r=0.1340v=3

p=0.32r=0.1593

v=1

p=0.57r=0.1177

v=2p=0.43

p=0.38 p=0.62

..2..

..8..p=0.06

r=0.0121v=1

..7..

..3..p=0.04

r=0.0097v=1

..4..

p=0.56r=0.1736

v=2p=0.44

p=0.57 p=0.43

p=0.23r=0.0347

v=1..2..

..6..

..6..p=0.05

r=0.0376v=1

p=0.45r=0.4158v=1284

p=0.10r=0.0641

v=2

..7..p=0.13

r=0.4095v=32

p=0.87r=0.4161v=1251

p=0.09r=0.0166

v=1

p=0.91r=0.4329

v=30

..2.. ..2..p=0.09

r=0.5188v=1

p=0.71r=0.4312

v=28

..3.. ..2..p=0.80

r=0.4263v=27

p=0.10p=0.90

r=0.4194v=26

p=0.90r=0.4119

v=25p=0.10

p=0.10p=0.90

r=0.4029v=24

p=0.14p=0.86

r=0.3925v=23

p=0.87r=0.3791

v=22p=0.13

p=0.13p=0.87

r=0.3636v=21

p=0.88r=0.3429

v=20p=0.12

p=0.13p=0.87

r=0.3194v=19

p=0.27r=0.1444

v=2

p=0.73r=0.3383

v=16

p=0.50 p=0.50p=0.38

r=0.3240v=15

..8..

..6..p=0.08

r=0.4829v=1

p=0.50r=0.3151

v=13

..9.. p=0.14p=1.00

r=0.3028v=11

STEPS:2

p=0.19p=0.05

r=0.2163v=1

p=0.46r=0.3041

v=6

p=0.30r=0.3181

v=3

..4..p=0.69

r=0.2876v=5

..2..

p=0.18p=0.82

r=0.2565v=4

p=0.30r=0.3362

v=1

p=0.70r=0.1728

v=2

..58.. ..2..p=0.11

r=0.1136v=1

..58..

p=0.38p=0.62

r=0.2401v=2

p=0.62 p=0.38

p=0.10r=0.1064

v=2

p=0.90r=0.4168v=1248

..3..p=0.07

r=0.0220v=1

..4..

p=0.12r=0.0475

v=2

p=0.88r=0.4175v=1245

p=0.92p=0.08

r=0.0132v=1

..2..

p=0.10r=0.0883

v=2

p=0.90r=0.4182v=1242

p=0.05r=0.0185

v=1..3..

..4..

p=0.85r=0.4193v=1217

p=0.15r=0.3694

v=24

p=0.14r=0.3774

v=23

p=0.86r=0.4202v=1193

p=0.09r=0.0128

v=1

p=0.91r=0.4059

v=21

..2..p=0.75

r=0.4018v=20

..2..

p=0.75r=0.3887

v=19..2..

p=0.13p=0.87

r=0.3712v=18

p=0.12p=0.88

r=0.3472v=17

p=0.87r=0.3197

v=16p=0.13

p=0.27r=0.1444

v=2

p=0.73r=0.3430

v=13

p=0.50 p=0.50p=0.38

r=0.3255v=12

..8..

..6..p=0.08

r=0.4829v=1

p=0.50r=0.3141

v=10

..9..p=1.00

r=0.2971v=8

STEPS:2

p=0.14

p=0.05r=0.2163

v=1

p=0.46r=0.3041

v=6..2..

..4.. ..2..p=0.69

r=0.2876v=5

p=0.82r=0.2565

v=4p=0.18

p=0.30r=0.3362

v=1

p=0.70r=0.1728

v=2

..58..p=0.11

r=0.1136v=1

..2..

..58..

p=0.85r=0.4211v=1171

p=0.15r=0.3782

v=21

p=0.88r=0.4218v=1159

p=0.12r=0.3576

v=11

p=0.87r=0.4227v=1148

p=0.13r=0.3300

v=10

p=0.72r=0.4248v=1010

p=0.28r=0.4098

v=137

p=0.37r=0.3275

v=180

p=0.19r=0.4483

v=822

p=0.12r=0.1899

v=7..6..

..6..p=0.09

r=0.2099v=2

p=0.49r=0.3301

v=177

..8..p=0.07

r=0.1942v=1

..8..

p=1.00r=0.3316

v=173

STEPS:2

p=0.14r=0.1451

v=2

p=0.05r=0.1507

v=1

p=0.46r=0.3531

v=138

p=0.19r=0.2429

v=10

p=0.30r=0.2492

v=23

..4..p=0.70

r=0.3574v=134

p=0.16r=0.1781

v=3p=0.14

p=0.82r=0.3605

v=130

p=0.18r=0.2057

v=3

p=0.30r=0.1158

v=7

p=0.70r=0.3747

v=122

p=0.03r=0.0797

v=6..57..

..56.. p=0.03 p=1.00

STEPS:5

p=0.11r=0.1120

v=1

p=0.84r=0.3781

v=120p=0.05

..58..p=0.83

r=0.3776v=119

..2..

p=0.90r=0.3774

v=118p=0.10

p=0.90r=0.3764

v=117p=0.10

p=0.10p=0.90

r=0.3755v=116

p=0.09p=0.91

r=0.3743v=115

p=1.00r=0.3727

v=113

STEPS:2

p=0.10

p=0.05r=0.2690

v=1

p=0.46r=0.3990

v=87

p=0.22r=0.2926

v=10

p=0.27r=0.2758

v=14

..4.. p=0.13p=0.70

r=0.4046v=84

p=0.18r=0.1837

v=2

p=0.17p=0.83

r=0.4039v=83

p=0.31r=0.2517

v=7

p=0.69r=0.4182

v=75

p=0.03r=0.2353

v=6..57..

p=0.03p=0.04

r=0.2789v=4

..56..

p=0.02r=0.2712

v=3..60..

..56.. p=0.02 p=1.00

STEPS:2

p=0.11r=0.2201

v=1

p=0.84r=0.4230

v=73p=0.05

..58..p=0.86

r=0.4221v=72

..2..

p=0.90r=0.4216

v=71p=0.10

p=0.90r=0.4206

v=70p=0.10

p=0.90r=0.4196

v=69p=0.10

p=0.90r=0.4179

v=68p=0.10

p=0.10p=0.90

r=0.4161v=67

p=0.90r=0.4157

v=66p=0.10

p=0.10p=0.90

r=0.4151v=65

p=0.16p=1.00

r=0.4137v=63

STEPS:2

p=0.07r=0.3008

v=1

p=0.54r=0.4289

v=49

p=0.20r=0.4200

v=7

p=0.19r=0.2850

v=5

..4..p=0.72

r=0.4296v=48

..2..

p=0.83r=0.4283

v=47p=0.17

p=0.31r=0.3160

v=5

p=0.69r=0.4420

v=41

p=0.03r=0.2981

v=4..57..

p=0.03p=0.04

r=0.4316v=2

..56..

p=0.02r=0.5279

v=1..60..

..58..

p=0.05p=0.11

r=0.2474v=1

p=0.84r=0.4506

v=39

..58.. ..2..p=0.86

r=0.4487v=38

p=0.10p=0.90

r=0.4465v=37

p=0.90r=0.4442

v=36p=0.10

p=0.10p=0.90

r=0.4404v=35

p=0.90r=0.4364

v=34p=0.10

p=0.10p=1.00

r=0.4308v=32

STEPS:2

p=0.05r=0.3183

v=1

p=0.46r=0.4650

v=24

p=0.22r=0.3632

v=1

p=0.27r=0.3042

v=5

..4.. ..2..p=0.75

r=0.4671v=23

p=0.18p=0.82

r=0.4653v=22

p=0.29r=0.2719

v=3

p=0.71r=0.4984

v=18

..57..p=0.03

r=0.1886v=2

p=0.02 ..57..

p=0.05p=0.11

r=0.2887v=1

p=0.84r=0.5227

v=16

..58.. ..2..p=0.86

r=0.5215v=15

p=0.90r=0.5185

v=14p=0.10

p=0.10p=0.90

r=0.5161v=13

p=0.10p=0.90

r=0.5126v=12

p=0.90r=0.5043

v=11p=0.10

p=0.10p=0.90

r=0.4943v=10

p=0.91r=0.4919

v=9p=0.09

p=0.10p=1.00

r=0.4719v=7

STEPS:2

..2..p=0.06

r=0.4340v=1

p=0.52r=0.4765

v=5

..4.. ..2..p=0.71

r=0.4841v=4

p=0.20p=0.80

r=0.4556v=3

p=0.30r=0.4804

v=1

p=0.70r=0.3744

v=1

..58.. ..3..

..4..p=0.55

r=0.2420v=4

p=0.45

p=0.55p=0.45

r=0.2074v=2

..6.. p=0.12

p=0.11p=0.04

r=0.4998v=1

p=0.43r=0.4249

v=4

p=0.42r=0.3528

v=1

..4..p=0.91

r=0.4022v=3

p=0.09

p=0.04r=0.3513

v=1

p=0.43r=0.3622

v=1..2..

..4.. ..2..

..2..

p=0.39p=0.61

r=0.2249v=4

p=0.61p=0.39

r=0.1872v=2

p=0.13 ..6..

p=0.55 p=0.45

..2..p=0.05

r=0.4794v=1

p=0.53r=0.2601

v=8

..4.. p=0.09p=0.91

r=0.2298v=7

p=0.05r=0.2465

v=1

p=0.54r=0.1778

v=5..2..

..4..p=0.90

r=0.1585v=4

p=0.10

..4.. p=0.12p=0.26

r=0.1691v=2

p=0.58 p=0.42

p=0.66r=0.2021

v=9

p=0.34r=0.3890

v=4

p=0.66p=0.34

r=0.1903v=7

..5.. p=0.12p=0.34

r=0.2037v=5

..2.. p=0.10p=0.69

r=0.2328v=3

p=0.35p=0.65

r=0.1757v=2

p=0.43 p=0.57

p=0.04r=0.4636

v=1

p=0.54r=0.3893

v=2..7..

..8..p=0.83

r=0.3519v=1

p=0.17

..2..

p=0.86r=0.1706

v=2p=0.14

p=0.10r=0.1576

v=1..2..

..58..

p=0.49p=0.51

r=0.1531v=1

..10..

p=0.04r=0.0813

v=1

p=0.42r=0.2220

v=4

p=0.41r=0.2948

v=4p=0.12

..4.. p=0.09p=0.91

r=0.2154v=3

p=0.04r=0.0533

v=1

p=0.42r=0.1284

v=1..2..

..4.. ..2..

p=0.91r=0.2885

v=3p=0.09

..2..p=0.04

r=0.2072v=1

p=0.41r=0.3459

v=1

..4.. ..2..

p=0.62r=0.2172

v=15

p=0.38r=0.2867

v=7

p=0.62p=0.38

r=0.2144v=13

..5.. p=0.11p=0.35

r=0.2260v=11

p=0.10p=0.66

r=0.2410v=9

..2..

p=0.67r=0.2318

v=8p=0.33

p=0.61p=0.39

r=0.2561v=6

..7..p=0.04

r=0.2875v=1

p=0.49r=0.2169

v=4

..8.. p=0.17p=0.83

r=0.1984v=3

p=0.30r=0.2021

v=1

p=0.70r=0.1930

v=1

..58.. ..3..

..7..p=0.04

r=0.3833v=1

p=0.52r=0.2645

v=5

..8.. p=0.17p=0.83

r=0.2306v=4

p=0.31r=0.2910

v=1

p=0.69r=0.1461

v=2

..58.. ..2..p=0.11

r=0.1104v=1

..58..

..3..p=0.06

r=0.0439v=1

..4..

..3.. p=0.16p=0.11

r=0.0468v=2

p=0.20r=0.4570

v=774

p=0.10r=0.0338

v=2

p=0.18r=0.3394

v=42

p=0.28r=0.0390

v=1..2..

..6..

..6..p=0.05

r=0.0394v=1

p=0.51r=0.4594

v=759

p=0.13r=0.3648

v=13

..8..p=0.81

r=0.4580v=699

p=0.19r=0.4742

v=59

p=0.76r=0.4613

v=690

p=0.24r=0.1710

v=8

p=0.07p=0.83

r=0.4625v=686

p=0.10r=0.1745

v=3

p=0.09p=0.79

r=0.4623v=666

p=0.12r=0.4638

v=19

p=0.12p=0.88

r=0.4622v=665

p=0.88r=0.4620

v=664p=0.12

p=0.12p=0.88

r=0.4618v=663

p=0.10p=0.90

r=0.4615v=662

p=0.90r=0.4613

v=661p=0.10

..5..p=0.39

r=0.4666v=607

p=0.20r=0.3947

v=53

p=0.27r=0.4797

v=98

p=0.73r=0.4644

v=508

p=0.50p=0.50

r=0.4866v=96

p=0.23r=0.5009

v=36

p=0.36r=0.4737

v=59..6..

..7..p=0.04

r=0.6056v=1

p=0.54r=0.4924

v=34

..8.. p=0.10p=0.90

r=0.4889v=33

p=0.89r=0.4835

v=32p=0.11

p=0.09p=0.91

r=0.4863v=31

..2..p=0.68

r=0.4798v=30

p=0.89r=0.4719

v=29p=0.11

p=0.89r=0.4627

v=28p=0.11

p=0.91r=0.4631

v=27p=0.09

p=0.90r=0.4529

v=26p=0.10

p=0.43r=0.4423

v=25..8..

..4.. p=0.10p=0.30

r=0.4541v=23

p=0.65p=0.35

r=0.4681v=21

p=0.13r=0.3356

v=3

p=0.27r=0.4818

v=17..6..

p=0.02r=0.2698

v=2..35..

p=0.02 ..35..

..8..p=0.02

r=0.4200v=1

p=0.44r=0.4819

v=15

..3..p=0.08

r=0.1894v=1

p=0.92r=0.4960

v=13

..2..p=0.92

r=0.5004v=12

p=0.08

p=0.08p=0.92

r=0.4982v=11

p=0.08p=0.92

r=0.4919v=10

p=0.90r=0.4798

v=9p=0.10

p=0.90r=0.4649

v=8p=0.10

p=0.10p=0.90

r=0.4430v=7

p=0.09p=0.91

r=0.4138v=6

p=0.90r=0.3729

v=5p=0.10

p=0.59r=0.2968

v=4p=0.41

p=0.59p=0.41

r=0.2550v=2

..6.. p=0.13

p=0.05p=0.41

r=0.4980v=36

p=0.39r=0.4428

v=21..2..

p=0.52r=0.4551

v=24

p=0.26r=0.5671

v=11p=0.22

..4..p=0.10

r=0.3057v=1

p=0.42r=0.4537

v=22

..3.. ..8..p=0.49

r=0.4586v=21

..4..p=0.09

r=0.1651v=1

p=0.43r=0.4650

v=19

..3..p=0.19

r=0.5221v=15

p=0.23r=0.1725

v=3..6..

..6..p=0.21

r=0.5380v=8

p=0.26r=0.4791

v=6

p=0.01r=0.5267

v=7..35..

..7..p=0.27

r=0.5151v=6

p=0.04r=0.3664

v=1

p=0.56r=0.5148

v=4..7..

..8..p=0.90

r=0.5338v=3

p=0.10

p=0.89r=0.5478

v=2p=0.11

p=0.09p=0.91

r=0.4984v=1

..3..

p=0.03r=0.5425

v=3

p=0.32r=0.3149

v=2..5..

..7..p=0.27

r=0.5156v=2

..8..p=0.04

r=0.3664v=1

..8..

p=0.82p=0.18

r=0.0274v=1

..2..

..4.. p=0.06 p=0.43

p=0.19p=0.41

r=0.6284v=8

p=0.40r=0.3018

v=2

p=0.25r=0.7791

v=1

p=0.75r=0.5984

v=6

..6.. ..7..p=0.15

r=0.6117v=5

p=0.35r=0.5958

v=4..9..

p=0.33r=0.6144

v=3..6..

p=0.68p=0.32

r=0.5316v=2

p=0.11r=0.2808

v=1..5..

..3..

p=0.39r=0.2222

v=1..8..

..8..

..3..p=0.35

r=0.4115v=9

p=0.34r=0.4492

v=11

..3.. p=0.31p=0.29

r=0.4237v=7

p=0.37p=0.63

r=0.3672v=6

p=0.21p=0.79

r=0.3910v=4

p=0.39r=0.4036

v=3..8..

..6..p=0.10

r=0.5081v=1

p=0.53r=0.2632

v=1

..10.. ..2..

..3..p=0.26

r=0.4252v=6

p=0.28r=0.4656

v=4

p=0.37p=0.63

r=0.3578v=5

p=0.63r=0.2875

v=4..2..

p=0.23r=0.2858

v=3..6..

p=0.54p=0.46

r=0.2154v=2

p=0.54 p=0.46

p=0.63r=0.3692

v=3p=0.37

p=0.61r=0.2491

v=2..2..

p=0.23r=0.2618

v=1..6..

..2..

p=0.17r=0.3871

v=28

p=0.40r=0.4731

v=467

p=0.17r=0.2982

v=12..6..

p=0.15r=0.4362

v=13

p=0.21r=0.3464

v=13p=0.18 ..5..

..8..p=0.35

r=0.3554v=5

p=0.35r=0.4581

v=7

..6..p=0.40

r=0.3600v=4

p=0.24r=0.3528

v=1

p=0.76r=0.3870

v=2

..2.. ..8..p=0.42

r=0.2055v=1

..8..

p=0.38r=0.4185

v=6..6..

p=0.26r=0.4174

v=1

p=0.74r=0.4484

v=4

..2..p=0.41

r=0.3908v=3

..8..

..6..p=0.06

r=0.4177v=1

p=0.56r=0.3502

v=1

..8.. ..2..

..7..p=0.09

r=0.2261v=1

p=0.24r=0.3351

v=11

..3..p=0.33

r=0.3285v=2

p=0.67r=0.3064

v=8

p=0.58 p=0.42p=0.38

r=0.2979v=7

..8..

p=1.00r=0.2606

v=5

STEPS:2

..6..

p=0.05r=0.2286

v=1

p=0.51r=0.2492

v=3..2..

..4..p=0.58

r=0.0981v=2

p=0.42

p=0.59 p=0.41

p=0.50r=0.4747

v=462

p=0.10r=0.3024

v=4..6..

p=0.14r=0.2577

v=3

p=0.86r=0.4762

v=458

..2..p=0.06

r=0.2723v=1

p=0.52r=0.2484

v=1

..4.. ..2..

p=0.88r=0.4766

v=457p=0.12

p=0.88r=0.4768

v=456p=0.12

p=0.90r=0.4770

v=455p=0.10

p=0.10p=0.90

r=0.4772v=454

p=0.16r=0.2766

v=5

p=0.84r=0.4794

v=448

..2..p=0.05

r=0.3036v=1

p=0.45r=0.2648

v=3

..4..p=0.71

r=0.2583v=2

p=0.29

p=0.11r=0.2203

v=1..5..

..3..

p=0.88r=0.4796

v=447p=0.12

p=0.88r=0.4797

v=446p=0.12

p=0.90r=0.4798

v=445p=0.10

p=0.89r=0.4798

v=444p=0.11

p=0.13p=0.87

r=0.4795v=443

p=0.15r=0.4817

v=16

p=0.85r=0.4789

v=426

..2..p=0.05

r=0.4097v=1

p=0.45r=0.4884

v=14

..4.. ..2..p=0.07

r=0.4209v=1

p=0.70r=0.5033

v=12

..8..p=0.89

r=0.4805v=11

p=0.11

p=0.88r=0.4517

v=10p=0.12

p=0.62r=0.4663

v=9..2..

p=0.48r=0.4790

v=4

p=0.52r=0.3705

v=4

p=0.48p=0.52

r=0.5651v=2

p=0.05r=0.5643

v=1..8..

..8..

..8..p=0.21

r=0.3091v=3

..4.. p=0.09p=0.28

r=0.4870v=1

..10..

p=0.88r=0.4783

v=425p=0.12

p=0.88r=0.4776

v=424p=0.12

p=0.15r=0.4962

v=36

p=0.62r=0.4786

v=343

p=0.23r=0.4584

v=44

p=0.63r=0.4992

v=35..2..

p=0.48r=0.3582

v=11

p=0.52r=0.5532

v=23

p=0.48p=0.52

r=0.3486v=9

p=0.27r=0.3236

v=6

p=0.23r=0.3026

v=2..7..

p=0.09p=0.28

r=0.3753v=4

..4..

p=0.32r=0.3381

v=3..9..

p=0.09r=0.1868

v=1

p=0.29r=0.3751

v=1..4..

..3.. ..2..

p=0.09 ..4..

..7..p=0.05

r=0.5643v=1

p=0.56r=0.5504

v=21

..8.. p=0.12p=0.88

r=0.5430v=20

p=0.12p=0.88

r=0.5334v=19

p=0.10p=0.90

r=0.5315v=18

..2..p=0.66

r=0.5182v=17

p=0.12p=0.88

r=0.5033v=16

p=0.88r=0.4849

v=15p=0.12

p=0.90r=0.4754

v=14p=0.10

p=0.12p=0.88

r=0.4531v=13

p=0.65r=0.3579

v=9

p=0.35r=0.6200

v=3

p=0.48p=0.52

r=0.3481v=7

p=0.26r=0.2994

v=4

p=0.26r=0.3124

v=2..7..

p=0.09p=0.28

r=0.4084v=2

..4..

..9..p=0.32

r=0.4097v=1

..6..

p=0.09 ..4..

..7..p=0.26

r=0.6335v=2

p=0.21r=0.6346

v=1..7..

..7..

p=0.48r=0.4476

v=135

p=0.52r=0.4972

v=207

p=0.48p=0.52

r=0.4484v=133

p=0.05r=0.4632

v=5

p=0.17r=0.4672

v=14

p=0.56r=0.4444

v=113..6..

..7..p=0.15

r=0.4379v=4

..6..p=0.31

r=0.3920v=3

p=0.22r=0.2971

v=2..7..

p=0.27 ..6..

..3.. p=0.07p=0.59

r=0.4976v=12

p=0.45r=0.4610

v=7

p=0.36r=0.5306

v=4p=0.19

p=0.45r=0.3315

v=2

p=0.55r=0.4756

v=4

p=0.45 p=0.55 ..8..p=0.53

r=0.4835v=3

p=0.55p=0.45

r=0.3938v=2

p=0.45 p=0.55

p=0.37r=0.4981

v=3..2..

p=0.45r=0.4014

v=2p=0.55

p=0.41 p=0.59

p=0.88r=0.4425

v=112p=0.12

p=0.88r=0.4402

v=111p=0.12

p=0.90r=0.4393

v=110p=0.10

p=0.12p=0.66

r=0.4427v=103

p=0.22r=0.3332

v=6

p=0.88r=0.4399

v=102p=0.12

p=0.12p=0.88

r=0.4368v=101

p=0.10p=0.90

r=0.4354v=100

p=0.88r=0.4320

v=99p=0.12

p=0.65r=0.4143

v=68

p=0.35r=0.4597

v=30

p=0.48p=0.52

r=0.4150v=66

p=0.18r=0.2796

v=5

p=0.31r=0.4195

v=41

p=0.18r=0.4321

v=19..6..

..4.. p=0.09p=0.28

r=0.3391v=3

p=0.32r=0.3051

v=2..9..

p=0.09r=0.2005

v=1..5..

..3..

p=0.09p=0.39

r=0.3547v=18

p=0.24r=0.4870

v=21..2..

p=0.65r=0.3728

v=13

p=0.35r=0.2334

v=4

p=0.43p=0.57

r=0.3850v=11

p=0.24r=0.3291

v=6

p=0.24r=0.4441

v=4..6..

p=0.17r=0.3258

v=4p=0.28 ..6..

p=0.32p=0.23

r=0.3173v=2

..5..

..3.. p=0.13

..4.. p=0.05p=0.43

r=0.5692v=2

p=0.89r=0.5473

v=1p=0.11

..2..

p=0.32p=0.15

r=0.2967v=2

..8..

..4.. p=0.13

p=0.26p=0.39

r=0.5096v=19

..4..

p=0.81r=0.5073

v=18p=0.19

p=0.24r=0.4398

v=2

p=0.76r=0.5182

v=15

..57..p=0.03

r=0.4113v=1

..58..

p=0.11r=0.4318

v=1

p=0.82r=0.5337

v=13p=0.07

..58.. ..2..p=0.80

r=0.5301v=12

p=0.10p=0.90

r=0.5233v=11

p=0.88r=0.5130

v=10p=0.12

p=0.12p=0.88

r=0.4977v=9

p=0.89r=0.4709

v=8p=0.11

p=0.13p=0.87

r=0.4316v=7

p=0.50r=0.4425

v=3

p=0.50r=0.3067

v=3

p=0.56p=0.44

r=0.5536v=1

..8..

p=0.18r=0.1904

v=2..8..

p=0.12 ..5..

..8..p=0.36

r=0.4042v=7

p=0.36r=0.4354

v=11

p=0.45r=0.4313

v=3

p=0.55r=0.2543

v=3

p=0.50p=0.50

r=0.5116v=1

..8..

..8..p=0.19

r=0.1257v=2

..5.. p=0.12

p=0.48r=0.5025

v=5

p=0.52r=0.2932

v=5

p=0.54p=0.46

r=0.5626v=3

..7..p=0.23

r=0.5773v=2

p=0.19r=0.5922

v=1..7..

..7..

..7..p=0.18

r=0.2067v=2

p=0.29r=0.2670

v=2

p=0.11 ..5.. p=0.09 ..4..

p=0.26r=0.3885

v=8

p=0.39r=0.4821

v=21..6..

p=0.21r=0.3587

v=7..7..

p=0.26p=0.27

r=0.3822v=5

..5..

p=0.11p=0.68

r=0.3970v=3

..2..

p=0.25p=0.75

r=0.3033v=2

p=0.44 p=0.56

p=0.03p=0.29

r=0.4745v=5

p=0.46r=0.5120

v=14..3..

p=0.43r=0.4471

v=4..2..

p=0.47r=0.3399

v=2

p=0.53r=0.4263

v=1

p=0.53 p=0.47 ..9..

p=0.87r=0.5089

v=13p=0.13

p=0.12p=0.88

r=0.4913v=12

p=0.90r=0.4879

v=11p=0.10

p=0.68r=0.4640

v=10..2..

p=0.12p=0.88

r=0.4357v=9

p=0.12p=0.88

r=0.3975v=8

p=0.90r=0.3674

v=7p=0.10

p=0.88p=0.12

r=0.3637v=5

p=0.08r=0.4549

v=1

p=0.92r=0.2570

v=3

..2.. p=0.11 p=0.74 p=0.15

p=0.77r=0.2613

v=5p=0.23

p=0.03r=0.2015

v=4..57..

..56.. p=0.02 p=1.00

STEPS:3

..6..p=0.21

r=0.2660v=9

p=0.20r=0.5418

v=107

p=0.25r=0.4666

v=90

p=0.09p=0.28

r=0.3002v=5

p=0.25r=0.2262

v=2..3..

..9..p=0.32

r=0.2535v=4

p=0.09r=0.1868

v=1

p=0.29r=0.1876

v=2..4..

..3.. p=0.56 p=0.44

p=0.54 p=0.46

..8..p=0.35

r=0.5811v=57

p=0.35r=0.4949

v=49

p=0.43r=0.5304

v=21

p=0.57r=0.6067

v=35

p=0.43p=0.57

r=0.5480v=19

p=0.05r=0.5454

v=1

p=0.53r=0.5508

v=17..7..

..8.. p=0.12p=0.88

r=0.5441v=16

p=0.12p=0.88

r=0.5330v=15

p=0.10p=0.90

r=0.5321v=14

p=0.66r=0.5183

v=13..2..

p=0.88r=0.5024

v=12p=0.12

p=0.88r=0.4814

v=11p=0.12

p=0.10p=0.90

r=0.4699v=10

p=0.12p=0.88

r=0.4365v=9

p=0.56r=0.3855

v=4

p=0.44r=0.4045

v=4

p=0.45p=0.55

r=0.3815v=2

p=0.19r=0.2514

v=1..8..

..6..

..7..p=0.25

r=0.3706v=3

p=0.20r=0.2880

v=2..7..

..6.. p=0.27

..6..p=0.21

r=0.1182v=2

p=0.21r=0.7704

v=20

p=0.25r=0.4261

v=12

..5.. p=0.10p=0.06

r=0.5509v=1

p=0.55r=0.7935

v=18..7..

..8..p=0.54

r=0.8765v=14

p=0.46r=0.4268

v=3

p=0.39r=0.8980

v=13..9..

p=0.68r=0.9059

v=12p=0.32

p=0.63r=0.9143

v=11..2..

p=0.12p=0.88

r=0.9208v=10

p=0.86r=0.9207

v=9p=0.14

p=0.91r=0.9471

v=8p=0.09

..2..p=0.68

r=0.9487v=7

p=0.86r=0.9496

v=6p=0.14

p=0.14p=0.86

r=0.9497v=5

p=0.09p=0.91

r=0.9889v=4

p=0.11p=0.89

r=0.9996v=3

p=0.20r=0.9999

v=2..3..

p=0.62 ..2..

p=0.46p=0.54

r=0.5401v=1

..9..

..3.. p=0.09p=0.37

r=0.4607v=10

p=0.62r=0.4713

v=8

p=0.38r=0.2794

v=1

p=0.41p=0.59

r=0.5299v=6

p=0.04r=0.6329

v=1

p=0.56r=0.5331

v=4..7..

..8..p=0.89

r=0.5264v=3

p=0.11

p=0.11p=0.89

r=0.5044v=2

p=0.09p=0.91

r=0.5703v=1

..3..

..10..

p=0.46r=0.5184

v=24

p=0.54r=0.4590

v=24

p=0.46p=0.54

r=0.5284v=22

..7..p=0.05

r=0.6141v=1

p=0.51r=0.5219

v=20

..8..p=0.88

r=0.5137v=19

p=0.12

p=0.12p=0.88

r=0.5019v=18

p=0.90r=0.4980

v=17p=0.10

p=0.66r=0.4808

v=16..2..

p=0.12p=0.88

r=0.4644v=15

p=0.12p=0.88

r=0.4440v=14

p=0.90r=0.4306

v=13p=0.10

p=0.12p=0.88

r=0.3980v=12

p=0.59r=0.3268

v=7

p=0.41r=0.4100

v=4

p=0.48p=0.52

r=0.2932v=5

..7..p=0.19

r=0.2067v=2

p=0.30r=0.2670

v=2

..5.. p=0.11 ..4.. p=0.09

..7..p=0.24

r=0.3754v=3

..7..p=0.19

r=0.2916v=2

..6.. p=0.25

..7..p=0.20

r=0.2671v=4

p=0.24r=0.4948

v=19

p=0.08p=0.33

r=0.3064v=2

..4..

p=0.59 p=0.41

p=0.09p=0.38

r=0.5191v=17

..3..

p=0.65r=0.5614

v=14

p=0.35r=0.1698

v=2

p=0.44p=0.56

r=0.6002v=12

..7..p=0.04

r=0.6957v=1

p=0.55r=0.6023

v=10

..8.. p=0.11p=0.89

r=0.6070v=9

p=0.89r=0.6107

v=8p=0.11

p=0.91r=0.6342

v=7p=0.09

p=0.70r=0.6437

v=6..2..

p=0.89r=0.6526

v=5p=0.11

p=0.11p=0.89

r=0.6590v=4

p=0.09p=0.91

r=0.7128v=3

p=0.09p=0.91

r=0.7399v=2

p=0.49p=0.51

r=0.7784v=1

..2..

..9.. p=0.28

..2.. p=0.09p=0.38

r=0.4301v=47

p=0.24r=0.5173

v=41

p=0.65r=0.4512

v=39

p=0.35r=0.2848

v=7

p=0.44p=0.56

r=0.4584v=37

p=0.04r=0.5810

v=1

p=0.56r=0.4556

v=35..7..

..8..p=0.89

r=0.4548v=34

p=0.11

p=0.89r=0.4523

v=33p=0.11

p=0.09p=0.91

r=0.4536v=32

..2..p=0.70

r=0.4500v=31

p=0.11p=0.89

r=0.4454v=30

p=0.11p=0.89

r=0.4395v=29

p=0.91r=0.4388

v=28p=0.09

p=0.10p=0.90

r=0.4320v=27

p=0.55r=0.4619

v=17

p=0.45r=0.3506

v=9

p=0.57p=0.43

r=0.4738v=15

p=0.34r=0.4636

v=14..9..

p=0.35r=0.5304

v=6

p=0.65r=0.3451

v=7

p=0.40p=0.60

r=0.5728v=4

p=0.21r=0.5134

v=3..7..

p=0.16r=0.3842

v=2..7..

..6.. p=0.25

p=0.20r=0.2794

v=6..8..

..4.. p=0.11p=0.23

r=0.3131v=4

..9..p=0.31

r=0.2711v=3

..5..p=0.11

r=0.1859v=2

p=0.26r=0.1393

v=1..2..

..62..

p=0.24r=0.3234

v=6

p=0.25r=0.3098

v=2..6..

p=0.18r=0.3215

v=4p=0.26 ..6..

..5.. p=0.32p=0.23

r=0.3129v=2

..3.. p=0.12

p=0.05 ..5..

..8.. p=0.32p=0.15

r=0.3391v=5

p=0.13p=0.40

r=0.3833v=3

..3..

p=0.44r=0.3183

v=2p=0.56

p=0.57 p=0.43

..4.. p=0.26p=0.39

r=0.5295v=39

p=0.81r=0.5270

v=38p=0.19

p=0.24r=0.3417

v=3

p=0.76r=0.5424

v=34

p=0.03r=0.2369

v=2..57..

..57.. p=0.02

p=0.07p=0.10

r=0.5146v=1

p=0.83r=0.5463

v=32

..58.. ..2..p=0.80

r=0.5428v=31

p=0.10p=0.90

r=0.5396v=30

p=0.12p=0.88

r=0.5358v=29

p=0.12p=0.88

r=0.5309v=28

p=0.11p=0.89

r=0.5236v=27

p=0.87r=0.5150

v=26p=0.13

p=0.50r=0.5754

v=17

p=0.50r=0.3538

v=8

p=0.50p=0.50

r=0.5997v=15

..7..p=0.05

r=0.5662v=1

p=0.54r=0.6051

v=13

..8.. p=0.12p=0.88

r=0.6026v=12

p=0.88r=0.5970

v=11p=0.12

p=0.10p=0.90

r=0.6067v=10

..2..p=0.66

r=0.5961v=9

p=0.12p=0.88

r=0.5831v=8

p=0.12p=0.88

r=0.5631v=7

p=0.90r=0.5648

v=6p=0.10

p=0.88r=0.5304

v=5p=0.12

p=0.33p=0.67

r=0.4605v=4

p=0.50p=0.50

r=0.5287v=2

p=0.18r=0.3844

v=1..8..

..6..

p=0.20r=0.2202

v=2

p=0.27r=0.3675

v=5..7..

p=0.10 ..5.. p=0.09p=0.38

r=0.4202v=3

..3..

p=0.35p=0.65

r=0.3193v=2

p=0.44 p=0.56

p=0.17r=0.3480

v=4

p=0.38r=0.4657

v=39..6..

p=0.24r=0.2757

v=3..8..

..4.. p=0.09p=0.27

r=0.3378v=1

..10..

..2.. p=0.05p=0.30

r=0.5426v=22

p=0.23r=0.3243

v=5

p=0.28r=0.3998

v=10

..7.. p=0.03p=0.38

r=0.5688v=20

p=0.86r=0.5702

v=19p=0.14

p=0.89r=0.5736

v=18p=0.11

p=0.12p=0.88

r=0.5755v=17

p=0.09p=0.91

r=0.5744v=16

p=0.91r=0.5732

v=15p=0.09

p=0.16p=0.84

r=0.5719v=14

p=0.12p=0.88

r=0.5709v=13

p=0.12p=0.88

r=0.5691v=12

p=0.09p=0.91

r=0.5629v=11

p=0.90r=0.5533

v=10p=0.10

p=0.13p=0.87

r=0.5348v=9

p=0.85r=0.5020

v=8p=0.15

p=0.89r=0.4503

v=7p=0.11

p=0.88p=0.12

r=0.4582v=5

..2..p=0.04

r=0.5282v=1

p=0.44r=0.3586

v=3

..4.. p=0.34p=0.66

r=0.2698v=2

p=0.58 p=0.42

p=0.39r=0.3230

v=2

p=0.61r=0.1446

v=2

p=0.42 p=0.58 p=0.27 ..9..

p=0.41r=0.3842

v=9..4..

p=0.32p=0.30

r=0.4076v=7

..3..

..2..p=0.50

r=0.3654v=6

p=0.36r=0.3542

v=2

p=0.64r=0.2617

v=3

p=0.64 p=0.36p=0.30

r=0.2430v=2

..8..

..5.. p=0.09

..9..p=0.33

r=0.2461v=3

..6..p=0.45

r=0.2882v=2

p=0.86r=0.2676

v=1p=0.14

..2..

..2.. p=0.09p=0.36

r=0.2978v=7

p=0.28r=0.3034

v=3

p=0.33r=0.3285

v=2

p=0.67r=0.2133

v=4

p=0.58 p=0.42 ..8.. p=0.29p=0.27

r=0.2428v=2

p=0.09 ..5..

p=0.06r=0.2109

v=1p=0.26 ..7..

..3..

p=0.11r=0.2994

v=1

p=0.67r=0.3888

v=51..2..

..2..p=0.27

r=0.5332v=18

p=0.73r=0.3019

v=32

p=0.42p=0.58

r=0.5636v=16

..7..p=0.30

r=0.5660v=15

..7..p=0.04

r=0.4323v=1

p=0.55r=0.5621

v=13

..8.. p=0.10p=0.90

r=0.5586v=12

p=0.89r=0.5502

v=11p=0.11

p=0.91r=0.5366

v=10p=0.09

p=0.68r=0.5247

v=9..2..

p=0.11p=0.89

r=0.5064v=8

p=0.11p=0.89

r=0.4792v=7

p=0.91r=0.4308

v=6p=0.09

p=0.90r=0.3776

v=5p=0.10

..8..p=0.30

r=0.3111v=4

p=0.10p=0.30

r=0.3111v=2

..4..

p=0.65 p=0.35

..8..p=0.36

r=0.2984v=31

..6..p=0.11

r=0.3128v=2

p=0.53r=0.2924

v=28

p=0.33r=0.1606

v=1..9..

..10..

p=0.80r=0.2903

v=27p=0.20

p=0.89r=0.2920

v=26p=0.11

p=1.00r=0.2929

v=24

STEPS:2

p=0.11

p=0.04r=0.2870

v=1

p=0.42r=0.2902

v=10

p=0.41r=0.3011

v=12p=0.13

..4..p=0.46

r=0.2763v=4

p=0.54r=0.2852

v=5

p=0.46p=0.54

r=0.3663v=2

..9..p=0.35

r=0.4165v=1

..2..

..9..p=0.39

r=0.2844v=4

p=0.48r=0.2118

v=2

p=0.52r=0.2904

v=1

p=0.48 p=0.52 ..7..

p=1.00r=0.3055

v=10

STEPS:2

..2..

p=0.17p=0.04

r=0.3126v=1

p=0.34r=0.2824

v=3

p=0.46r=0.3225

v=5

..4..p=0.46

r=0.2118v=2

p=0.54

p=0.46 p=0.54

p=0.80r=0.3336

v=4..2..

p=0.19p=0.81

r=0.3243v=3

p=0.89r=0.3300

v=2p=0.11

p=1.00

STEPS:2

p=0.11

..7..p=0.53

r=0.4589v=18

p=0.05p=0.62

r=0.4852v=16

p=0.33

p=0.88r=0.4794

v=15p=0.12

p=0.88r=0.4703

v=14p=0.12

p=0.12p=0.88

r=0.4604v=13

p=0.10p=0.90

r=0.4468v=12

p=0.90r=0.4291

v=11p=0.10

..6..p=0.40

r=0.3957v=10

p=0.27r=0.3028

v=1

p=0.73r=0.4193

v=8

..2.. ..8..p=0.41

r=0.3977v=7

..6..p=0.10

r=0.4657v=1

p=0.52r=0.3791

v=5

..10.. p=0.14p=0.86

r=0.3718v=4

p=0.88r=0.3868

v=3p=0.12

p=0.12p=0.88

r=0.4018v=2

p=0.09p=0.91

r=0.4054v=1

..2..

p=0.03r=0.1194

v=2..57..

p=0.02 ..57..

p=0.03r=0.1127

v=6p=1.00

STEPS:2

..56..

..56.. p=0.02 p=1.00

STEPS:5

p=0.85r=0.4725

v=58p=0.15

p=0.10r=0.3364

v=1

p=0.83r=0.4726

v=56p=0.07

..58.. ..2..p=0.79

r=0.4721v=55

p=0.12p=0.88

r=0.4704v=54

p=0.88r=0.4679

v=53p=0.12

p=0.12p=0.88

r=0.4656v=52

p=0.10p=0.90

r=0.4625v=51

p=0.10p=0.90

r=0.4591v=50

p=0.39r=0.4628

v=39

p=0.20r=0.4148

v=10..5..

p=0.27r=0.1514

v=2

p=0.73r=0.4847

v=36

p=0.50 p=0.50 ..8..p=0.40

r=0.4823v=35

..6..p=0.10

r=0.3167v=2

p=0.50r=0.4946

v=32

p=0.33r=0.1620

v=1..9..

..7..

p=0.86r=0.4975

v=31p=0.14

p=0.12p=0.88

r=0.5031v=30

p=0.88r=0.5080

v=29p=0.12

p=0.10p=0.90

r=0.5120v=28

p=0.90r=0.5160

v=27p=0.10

p=0.84r=0.5184

v=26p=0.16

p=0.12p=0.88

r=0.5227v=25

p=0.88r=0.5271

v=24p=0.12

p=0.90r=0.5299

v=23p=0.10

p=0.89r=0.5330

v=22p=0.11

p=0.13p=0.87

r=0.5295v=21

p=0.85r=0.5206

v=20p=0.15

p=0.88r=0.5083

v=19p=0.12

p=0.88r=0.4939

v=18p=0.12

..2..p=0.62

r=0.5040v=17

p=0.48r=0.5852

v=9

p=0.52r=0.3577

v=7

p=0.48p=0.52

r=0.6401v=7

..7..p=0.05

r=0.5643v=1

p=0.56r=0.6701

v=5

..8.. p=0.12p=0.88

r=0.6714v=4

p=0.12p=0.88

r=0.6631v=3

p=0.10p=0.90

r=0.7294v=2

..2..p=0.66

r=0.7299v=1

..2..

p=0.21r=0.3449

v=4

p=0.25r=0.2848

v=2..7..

p=0.09p=0.28

r=0.4696v=2

..4..

p=0.32r=0.4523

v=1..9..

..6..

..4.. p=0.09

..2..p=0.11

r=0.2994v=1

p=0.67r=0.3821

v=8

..2..p=0.27

r=0.2901v=2

p=0.73r=0.3813

v=5

p=0.58 p=0.42p=0.36

r=0.3742v=4

..8..

..6..p=0.11

r=0.4650v=1

p=0.53r=0.2979

v=2

..10.. p=0.20p=0.80

r=0.2459v=1

..2..

p=0.37p=0.19

r=0.3564v=11

..5..

..2.. p=0.12p=0.68

r=0.3451v=9

p=0.30r=0.2872

v=2

p=0.70r=0.3279

v=6

p=0.42 p=0.58p=0.36

r=0.3423v=5

..8..

p=0.07r=0.4707

v=1

p=0.55r=0.2771

v=3..6..

..9.. p=0.20p=1.00

r=0.2435v=1

STEPS:2

..4..

..5..p=0.19

r=0.0194v=1

..3..

p=0.06r=0.1037

v=1

p=0.37r=0.3390

v=37

p=0.15r=0.3697

v=3..7..

..3.. p=0.10p=0.90

r=0.3339v=36

p=0.90r=0.3289

v=35p=0.10

p=0.10p=0.90

r=0.3235v=34

p=0.90r=0.3162

v=33p=0.10

p=0.14p=0.86

r=0.3085v=32

p=0.87r=0.2974

v=31p=0.13

p=0.87r=0.2838

v=30p=0.13

p=0.12p=0.88

r=0.2690v=29

p=0.87p=0.13

r=0.2609v=27

p=0.08r=0.1900

v=1

p=0.92r=0.2529

v=25

..2.. p=0.71p=0.14

r=0.2545v=22

p=0.15

..5.. p=0.08p=0.39

r=0.2552v=20

..2.. p=0.08p=0.68

r=0.2553v=18

p=0.65p=0.12

r=0.2659v=15

p=0.23

..8.. p=0.26p=0.25

r=0.2849v=13

..3.. p=0.10p=0.21

r=0.3350v=6

p=0.28r=0.2273

v=5

..8..p=0.54

r=0.3410v=5

p=0.09r=0.3196

v=2

p=0.29r=0.2573

v=2..4..

..2..p=0.37

r=0.2994v=1

..62..

p=0.43 p=0.57

p=0.57p=0.43

r=0.1933v=3

..6..p=0.14

r=0.0320v=1

p=0.23r=0.2604

v=1

..36.. ..10..

p=0.30r=0.5445

v=1

p=0.70r=0.0838

v=1

..2.. ..9..

p=0.10p=0.37

r=0.2067v=5

..3..

p=0.30r=0.4085

v=1

p=0.70r=0.0481

v=3

..2.. ..8.. p=0.25p=0.26

r=0.0463v=1

..6..

p=0.48p=0.52

r=0.4143v=135

..6..p=0.30

r=0.4105v=77

p=0.25r=0.4189

v=57

p=0.05r=0.1446

v=1

p=0.47r=0.4160

v=75..7..

..8..p=0.90

r=0.4137v=74

p=0.10

p=0.11p=0.89

r=0.4105v=73

p=0.09p=0.91

r=0.4115v=72

p=0.66r=0.4196

v=65

p=0.25r=0.2813

v=6p=0.09

p=0.11p=0.89

r=0.4153v=64

p=0.11p=0.89

r=0.4106v=63

p=0.91r=0.4101

v=62p=0.09

p=0.10p=0.90

r=0.4050v=61

..8..p=0.39

r=0.3984v=60

..2.. p=0.10p=0.28

r=0.4193v=48

p=0.16r=0.3520

v=9p=0.19

p=0.65p=0.35

r=0.4242v=46

..5..p=0.13

r=0.2911v=3

p=0.28r=0.4391

v=41p=0.19

p=0.02r=0.2692

v=2..35..

p=0.02 ..35..

p=0.02r=0.4187

v=1

p=0.44r=0.4608

v=32

p=0.25r=0.3259

v=7..7..

..3..p=0.08

r=0.0542v=1

p=0.92r=0.4700

v=30

..2.. p=0.08p=0.92

r=0.4710v=29

p=0.08p=0.92

r=0.4692v=28

p=0.08p=0.92

r=0.4658v=27

p=0.10p=0.90

r=0.4622v=26

p=0.90r=0.4568

v=25p=0.10

p=0.10p=0.90

r=0.4501v=24

p=0.10p=0.90

r=0.4429v=23

p=0.90r=0.4350

v=22p=0.10

p=0.58r=0.3075

v=9

p=0.42r=0.5105

v=12

p=0.58p=0.42

r=0.2988v=7

..5.. p=0.13p=0.36

r=0.3165v=5

..2.. p=0.10p=0.68

r=0.3537v=3

p=0.69r=0.2659

v=2p=0.31

p=0.40 p=0.60

p=0.04r=0.5645

v=1

p=0.56r=0.5052

v=10..7..

..7..p=0.92

r=0.5109v=9

p=0.08

p=0.08p=0.92

r=0.5169v=8

p=0.92r=0.5127

v=7p=0.08

p=0.92r=0.5075

v=6p=0.08

p=0.10p=0.90

r=0.5005v=5

p=0.10p=0.90

r=0.4709v=4

p=0.90r=0.4252

v=3p=0.10

p=0.10p=0.90

r=0.3189v=2

p=0.90 p=0.10

p=0.62r=0.2735

v=6p=0.38

p=0.62p=0.38

r=0.2499v=4

..5.. p=0.27p=0.32

r=0.2506v=2

p=0.12 ..3..

p=0.31p=0.28

r=0.3647v=7

..5..

..2.. p=0.12p=0.66

r=0.3871v=5

p=0.09r=0.3487

v=2

p=0.32r=0.3096

v=2..4..

p=0.37r=0.2580

v=1..2..

..62..

p=0.62 p=0.38

p=0.80r=0.2138

v=5p=0.20

..57..p=0.03

r=0.1371v=4

..56.. p=0.02 p=1.00

STEPS:3

..2.. p=0.05p=0.40

r=0.4399v=32

p=0.40r=0.3964

v=23

p=0.54r=0.4314

v=24

p=0.23r=0.4427

v=7p=0.22

..4..p=0.10

r=0.1949v=1

p=0.42r=0.4313

v=22

..3.. ..8..p=0.45

r=0.4451v=21

p=0.08r=0.1305

v=1

p=0.47r=0.4523

v=19..4..

..3.. ..6..p=0.16

r=0.5161v=15

p=0.24r=0.1730

v=3

p=0.20r=0.5430

v=6

p=0.28r=0.4791

v=8..6..

p=0.01r=0.5272

v=5..35..

..7..p=0.28

r=0.5086v=4

p=0.04r=0.3769

v=1

p=0.56r=0.4931

v=2..7..

..8..p=0.90

r=0.5171v=1

p=0.10

..2..

..5..p=0.03

r=0.5499v=3

p=0.32r=0.3902

v=4

..7..p=0.28

r=0.5240v=2

p=0.04r=0.3769

v=1..8..

..8..

p=0.18r=0.0166

v=1

p=0.82r=0.4682

v=2

..2..p=0.91

r=0.5030v=1

p=0.09

..2..

p=0.06 p=0.44 ..4..

p=0.42r=0.4670

v=5

p=0.39r=0.1447

v=1p=0.19

p=0.26r=0.7836

v=1

p=0.74r=0.2994

v=3

..6..p=0.18

r=0.2650v=2

..7..

..8..p=0.03

r=0.0100v=1

..8..

..9..

..3..p=0.35

r=0.3900v=10

p=0.33r=0.4177

v=12

..3.. p=0.31p=0.29

r=0.3934v=8

p=0.61r=0.3713

v=6

p=0.39r=0.2209

v=1

p=0.21p=0.79

r=0.3868v=4

..8..p=0.36

r=0.4444v=3

p=0.07r=0.5024

v=1

p=0.54r=0.4031

v=1..6..

..9.. ..2..

..8..

..3..p=0.26

r=0.4085v=6

p=0.28r=0.4255

v=5

p=0.39p=0.61

r=0.3550v=5

p=0.62r=0.2738

v=4..2..

..6..p=0.16

r=0.2710v=3

p=0.37p=0.63

r=0.2042v=2

p=0.59 p=0.41

p=0.39p=0.61

r=0.3374v=4

..2..p=0.60

r=0.2417v=3

p=0.16r=0.2486

v=2..6..

p=0.64r=0.2486

v=1p=0.36

..2..

p=0.11r=0.0603

v=1

p=0.89r=0.3809

v=8

..2.. ..2..p=0.71

r=0.3464v=7

p=0.25r=0.2615

v=1

p=0.75r=0.3745

v=5

..2.. ..8..p=0.38

r=0.3496v=4

p=0.09r=0.4333

v=1

p=0.49r=0.3791

v=2..6..

..9.. p=1.00

STEPS:2

p=0.14

p=0.09r=0.0228

v=1

p=0.91r=0.4129

v=9

..2.. ..2..p=0.71

r=0.3935v=8

p=0.87r=0.3464

v=7p=0.13

p=0.25r=0.2615

v=1

p=0.75r=0.3745

v=5

..2.. ..8..p=0.38

r=0.3496v=4

p=0.09r=0.4333

v=1

p=0.49r=0.3791

v=2..6..

..9.. p=0.14 p=1.00

STEPS:2

p=0.09r=0.0128

v=1

p=0.91r=0.4098

v=19

..2.. ..2..p=0.74

r=0.4019v=18

..2..p=0.75

r=0.3864v=17

p=0.12p=0.88

r=0.3618v=16

p=0.87r=0.3334

v=15p=0.13

p=0.27r=0.1444

v=2

p=0.73r=0.3621

v=12

p=0.50 p=0.50 ..8..p=0.38

r=0.3448v=11

..6..p=0.08

r=0.4829v=1

p=0.50r=0.3364

v=9

..9.. p=0.14p=1.00

r=0.3233v=7

STEPS:2

p=0.05r=0.2163

v=1

p=0.46r=0.3422

v=5..2..

..4.. ..2..p=0.69

r=0.3310v=4

p=0.82r=0.3041

v=3p=0.18

p=0.30r=0.3362

v=1

p=0.70r=0.2320

v=1

..58.. ..3..

p=0.10r=0.0131

v=1

p=0.90r=0.3989

v=22

..2.. ..2..p=0.76

r=0.4052v=21

p=0.75r=0.4037

v=20..2..

p=0.13p=0.87

r=0.3887v=19

p=0.13p=0.87

r=0.3712v=18

p=0.88r=0.3472

v=17p=0.12

p=0.87r=0.3197

v=16p=0.13

p=0.27r=0.1444

v=2

p=0.73r=0.3430

v=13

p=0.50 p=0.50p=0.38

r=0.3255v=12

..8..

..6..p=0.08

r=0.4829v=1

p=0.50r=0.3141

v=10

..9.. p=0.14p=1.00

r=0.2971v=8

STEPS:2

p=0.05r=0.2163

v=1

p=0.46r=0.3041

v=6..2..

..4.. ..2..p=0.69

r=0.2876v=5

p=0.18p=0.82

r=0.2565v=4

p=0.30r=0.3362

v=1

p=0.70r=0.1728

v=2

..58..p=0.11

r=0.1136v=1

..2..

..58..

p=0.08 ..4..

..3.. p=0.06p=0.50

r=0.0985v=1

..2..

p=0.21p=0.79

r=0.2156v=31

..5..p=0.22

r=0.2637v=22

p=0.13r=0.0392

v=2

p=0.21r=0.1267

v=6

..7..p=0.06

r=0.0320v=1

p=0.46r=0.2857

v=20

..8.. p=0.13p=0.87

r=0.2910v=19

p=0.11p=0.89

r=0.2966v=18

p=0.10p=0.90

r=0.3095v=17

p=0.09p=0.56

r=0.3231v=13

p=0.35r=0.1771

v=3

p=0.12p=0.88

r=0.3040v=12

p=0.11p=0.89

r=0.2788v=11

p=0.10p=0.90

r=0.2970v=10

p=0.11p=0.89

r=0.2485v=9

..8..p=0.37

r=0.1795v=8

..3.. p=0.10p=0.19

r=0.1188v=1

p=0.27r=0.1712

v=5

..7.. p=0.61p=0.39

r=0.1168v=3

p=0.13r=0.0386

v=1

p=0.27r=0.1806

v=1..6..

..36.. ..10..

p=0.19p=0.81

r=0.0301v=2

..57..p=0.03

r=0.0365v=1

..58..

..7..p=0.39

r=0.0362v=1

..9..

..2.. p=0.06p=0.38

r=0.1479v=1

p=0.41r=0.1610

v=3

..3..p=0.33

r=0.2246v=2

..4..

p=0.29 ..4..

p=0.06r=0.0760

v=1..26..

..61..

..26..p=0.06

r=0.0656v=2

..60..p=0.02

r=0.0410v=1

..62..

p=0.37r=0.0448

v=1

p=0.47r=0.0613

v=1p=0.16

..58.. ..61..

p=0.29r=0.0486

v=1

p=0.37r=0.1091

v=1..2..

..58.. ..61..


RL-CoP setup summary

1. Representation:a search in the treeshould correspondto a tableaux

<=

>

3. Explore thenode and backupthe found rewardto all nodes above

6. Repeat100 times

9. Repeat!

2. Playout: followmaximum UCT untilunexplored node

5. Focus onmost visitednode

4. Repeat1000 times

7. Do this for alltheorems. We getmany sequencesof focused steps

8. Train newpredictors forpolicy and valueusing the seqs.

Theorem proving requiring significant hardware


RL-CoP setup summary

1. Representation:a search in the treeshould correspondto a tableaux

<=

>

3. Explore thenode and backupthe found rewardto all nodes above

6. Repeat100 times

9. Repeat!

2. Playout: followmaximum UCT untilunexplored node

5. Focus onmost visitednode

4. Repeat1000 times

7. Do this for alltheorems. We getmany sequencesof focused steps

8. Train newpredictors forpolicy and valueusing the seqs.

Theorem proving requiring significant hardwareCezary Kaliszyk Artificial Intelligence in Theorem Proving 45 / 72

ATP versus learned ATP

ATPs tend to find short proofs.

Learning helps only minimally

Graph Representations for Higher-Order Logic andTheorem Proving [A. Paliwal et.al., 2019 ]

Cumulative proof lengths of rlCoP on the MizarMathematical Library [NeurIPS 2018 ]


Main aims of FLoP [Zombori’2019 ]

Build an internal guidance system that can find long proofs

Find a domain that is simple enough to analyse the inner workings of the prover

At first, try to learn from very few problems (with given or without given proofs)

Try to generalize to harder problems (longer proofs) with a similar structure


Domain: Robinson Arithmetic

Prove simple ground equalities

Proofs are non trivial, but have a strong shared structure

Proof lengths can get very long as numbers increase

See how little supervision is required to learn some proof types


Challenges for RL for TP

Theorem proving as a 1 person game

Meta-Learning task: each problem isa new maze: train on some, evaluateon others

Sparse, binary rewards

Defining good features

Action space not fixed:different across steps and acrossproblems


FLoP

External guidance based on RL

Theorem Prover encapsulated as an environment

Use curriculum learning

Applicable when we know the proof of a problem

More efficient use of training signals

Start learning from the end of the proof

Gradually move starting step towards the beginning of proof

Proximal Policy Optimization (PPO)

Actor learns a policy (what steps to take)

Critic learns a value (how promising is a proof state)

Actor is confined to change slowly to increase stability


Datasets


Evaluation


Learning for ATPs: Summary and next steps

For some calculi major improvement

Learning for Resolution-style systems open

Learn features

RL prefers shorter proofs

but they may not be the ones that generalize best

Evaluate with backtracking

Scale to more interesting domains

Bolzano–Weierstrass


Communication with a Proof Assistant

The prover does not get what I mean

Completely clear things need to be fully expanded

Even if I said it 100 times, I have to say it again

(or implement the expansion)

Compared to a student

Proof assistant does not get what I mean

Cannot repeat a simple action


Proof assistant – assistant

Given some text, the assistant can say

What you wrote

What you wanted to write

(What I think you meant)

Does it make sense

Can I be convinced of this(Can I prove this∗)

TasksUnderstand LATEX formulas, as well as some text

Translate it to logic (of a/the proof assistant)

Report on the success

Questions

Can we (a computer) learn how to state lemmas formally?

Can we (a computer) learn to prove?


Demo


Why don’t we have this? (1/2)

Claus Zinn and others tried and have not arrived very far because:

lack of background knowledge

lack of powerful automated reasoning

lack of self-adapting translation

But huge machine learning progress


Why don’t we have this? (2/2)

Controlled languages

Ranging from Naproche and MathLang to Mizar

Easy start but huge number of patterns100 most frequent patterns cover half of 42,931 ProofWiki sentences [CICM’14 ]

5829 Let $?$ be [?].

2688 Let $?$.

774 Then $?$ is [?].

736 Let $?$ be [?] of $?$.

724 Let $?$ and $?$ be [?].

578 Let $?$ be the [?] of $?$.

555 Let $?$ be the [?].

But can go very far

Thousands of manually entered patterns [Matsuzaki+’16,’17 ]

Better than humans on university entrance exams (some domains) [Arai+’18 ]


Learning data: Aligned corpora

Dense Sphere Packings: A Blueprint for Formal Proofs

400 theorems and 200 concepts mapped [Hales13 ]

IsaFoR [SternagelThiemann14 ]

most of “Term Rewriting and All That” [BaaderNipkow ]

Compendium of Continuous Lattices (CCL)

60% formalized in Mizar [BancerekRudnicki02 ]

high-level concepts and theorems aligned

Feit-Thompson theorem by Gonthier [Gonthier13 ]

Two graduate books

detailed proofs and symbol linking in Wikipedia, ProofWiki, PlanetMath, ...


Aligned corpora: Kepler ExampleDocument:

Informal Formal

Definition of [fan, blade] DSKAGVP (fan) [fan FAN]

Let be a pair consisting of a set and a set of unordered pairs of distinct elementsof . The pair is said to be a fan if the following properties hold.

(CARDINALITY) is finite and nonempty. [cardinality fan1]1.(ORIGIN) . [origin fan2]2.(NONPARALLEL) If , then and are not parallel. [nonparallel fan6]3.(INTERSECTION) For all , [intersection fan7]4.

When , call or a blade of the fan.

basic properties

The rest of the chapter develops the properties of fans. We begin with a completely trivialconsequence of the definition.

Informal Formal

Lemma [] CTVTAQA (subset-fan)

If is a fan, then for every , is also a fan.

Proof

This proof is elementary.

Informal Formal

Lemma [fan cyclic] XOHLED

[ set_of_edge] Let be a fan. For each , the set

is cyclic with respect to .

Proof

If , then and are not parallel. Also, if , then

Article Raw Log in

↔

(V , E) V ⊂ R3 EV

V ↔0 ∉ V ↔

{v, w} ∈ E v w ↔ε, ∈ E ∪ {{v} : v ∈ V }ε′ ↔

C(ε) ∩ C( ) = C(ε ∩ ).ε′ ε′

ε ∈ E (ε)C0 C(ε)

(V , E) ⊂ EE ′ (V , )E ′

E(v) ↔ (V , E) v ∈ V

E(v) = {w ∈ V : {v, w} ∈ E}

(0, v)

w ∈ E(v) v w w ≠ ∈ E(v)w′

Document:

Informal Formal

#DSKAGVP?

let FAN=new_definition`FAN(x,V,E) <=> ((UNIONS E) SUBSET V) /\ graph(E) /\ fan1(x,V,E) /\ fan2(x,V,E)/\fan6(x,V,E)/\ fan7(x,V,E)`;;

basic properties

The rest of the chapter develops the properties of fans. We begin with a completely trivial consequence ofthe definition.

Informal Formal

let CTVTAQA=prove(`!(x:real^3) (V:real^3->bool) (E:(real^3->bool)->bool) (E1:(real^3->bool)->bool).FAN(x,V,E) /\ E1 SUBSET E==>FAN(x,V,E1)`,

REPEAT GEN_TACTHEN REWRITE_TAC[FAN;fan1;fan2;fan6;fan7;graph]THEN ASM_SET_TAC[]);;

Informal Formal

let XOHLED=prove(`!(x:real^3) (V:real^3->bool) (E:(real^3->bool)->bool) (v:real^3).FAN(x,V,E) /\ v IN V==> cyclic_set (set_of_edge v V E) x v`,

MESON_TAC[CYCLIC_SET_EDGE_FAN]);;

Informal Formal

Remark [easy consequences of the definition] WCXASPV (fan)

Let be a fan.

The pair is a graph with nodes and edges . The set

is the set of edges at node . There is an evident symmetry: if and only if .

1.

[ sigma_fan] [ inverse1_sigma_fan] Since is cyclic, each has an azimuthcycle . The set can reduce to a

2.

singleton. If so, is the identity map on . To make the notation less cumbersome, denotes the value of the map at .

The property (NONPARALLEL) implies that the graph has no loops: .1.

The property (INTERSECTION) implies that distinct sets do not meet. This property of fansis eventually related to the planarity of hypermaps.

2.

Article Raw Log in

(V , E)

(V , E) V E

{{v, w} : w ∈ E(v)}

v w ∈ E(v) v ∈ E(w)σ ↔ σ(v)−1 ↔ E(v) v ∈ V

σ(v) : E(v) → E(v) E(v)

σ(v) E(v) σ(v, w)σ(v) w

{v, v} ∉ E

(ε)C0


Aligned corpora: Kepler Example

596 formulas from the Flyspeck book extracted with LATEXML

Translation to HOL Light based on a small table

17% same as formal ones

Too hardmake more precise examples or [ongoing ]

start with simpler ones [ITP’15 +]


Informalization

22000 Flyspeck statements informalized

72 overloaded instances like “+” for vector add

108 infix operators

forget all “prefixes”

real , int , vector , nadd , hreal , matrix , complex

ccos, cexp, clog, csin, ...

vsum, rpow, nsum, list sum, ...

Deleting all brackets, type annotations, and casting functors

Cx and real of num (which alone is used 17152 times).


CYK and parsing — just a little

Induce PCFG (probabilistic context-free grammar) from term trees

inner nodes → rules frequencies → probabilities

Binarize PCFG grammar for efficiency

CYK parses ambiguous sentences

outputs most probable parse trees

tweak: small probability for each symbol to be a variable

Pruning

Compatible types for free variables in subtrees

HOL type-checking

Hammer


Example tree inducing grammar

"(Type bool)"

! "(Type (fun real bool))"

Abs

"(Type real)" "(Type bool)"

Var

A0

"(Type real)" = "(Type real)"

real_neg "(Type real)"

real_neg "(Type real)"

Var

A0

Var

A0


Statistics

Just PFG [ITP’15 ]

39.4% of the Flyspeck sentences parsed correctly

average rank: 9.34

Problems with PCFG and CYK

1 ∗ x + 2 ∗ x

Use deeper trees [ITP 2017 ]

semantic pruning + subtree depth 4-8 + substitution trees

83% sentences parsed correctly

average rank: 1.93


Types helped us - what about no types?

MizarDeveloped by mathematicians for mathematicians

Many features significantly different from the usual

How would you formalize:


Mizar Statistics


Sequence-to-sequence models: decoder/encoder RNN

[Luong et al’15 ]


Neural Auto-formalization [CICM’18 ]

IdenticalStatements

0

≤ 1 ≤ 2 ≤ 3

Best Model- 1024 Units

69179 (total)22978 (no-overlap)

65.73%47.77%

74.58%59.91%

86.07%70.26%

88.73%74.33%

Top-5 Greedy Cover- 1024 Units- 4-Layer Bi. Res.- 512 Units- 6-Layer Adam Bi. Res.- 2048 Units


74.50%59.68%

82.07%70.85%

87.27%78.84%

89.06%81.76%

Top-10 Greedy Cover- 1024 Units- 4-Layer Bi. Res.- 512 Units- 6-Layer Adam Bi. Res.- 2048 Units- 2-Layer Adam Bi. Res.- 256 Units- 5-Layer Adam Res.- 6-Layer Adam Res.- 2-Layer Bi. Res.


76.89%63.25%

83.91%73.74%

88.60%81.07%

90.24%83.68%

Union of All 39 Models 83321 (total)32083 (no-overlap)

79.17%66.70%

85.57%76.39%

89.73%82.88%

91.25%85.30%


Neural Auto-formalization [CICM’18 ]

IdenticalStatements

0 ≤ 1 ≤ 2 ≤ 3

Best Model- 1024 Units


65.73%47.77%

74.58%59.91%

86.07%70.26%

88.73%74.33%

Top-5 Greedy Cover- 1024 Units- 4-Layer Bi. Res.- 512 Units- 6-Layer Adam Bi. Res.- 2048 Units


74.50%59.68%

82.07%70.85%

87.27%78.84%

89.06%81.76%

Top-10 Greedy Cover- 1024 Units- 4-Layer Bi. Res.- 512 Units- 6-Layer Adam Bi. Res.- 2048 Units- 2-Layer Adam Bi. Res.- 256 Units- 5-Layer Adam Res.- 6-Layer Adam Res.- 2-Layer Bi. Res.


76.89%63.25%

83.91%73.74%

88.60%81.07%

90.24%83.68%

Union of All 39 Models 83321 (total)32083 (no-overlap)

79.17%66.70%

85.57%76.39%

89.73%82.88%

91.25%85.30%


Machine Learning applied to informal LaTeX

[Karpathy’16 ]

Linguistic methodsWord2Vec on the Arxiv [Alemi’16,...]


Final Summary / Take Home

Proofs are hard

Machine learning key to most powerful proof assistant automation

Older but very efficient algorithms with significant adjustments

Many other learning problems and scenarios

Not coveredLearning strategy selection [Jakubuv,Urban]

Kernel methods [Kuhlwein]

Deep Prolog [Rocktaschel ]

Semantic Features, Conecturing

Tactic selection [Nagashima,...]

SVM [Holden]

Adversarial Networks [Szegedy ]

Human proof optimization

Theory exploration [Bundy+]

Concept Alignment [Gauthier ]

...


Arti cial Intelligence in Theorem Proving · 2019-07-08 · Overview Last Lecture theorem proving...

Documents

Transcript of Arti cial Intelligence in Theorem Proving · 2019-07-08 · Overview Last Lecture theorem proving...