Verified Runtime Monitoring:From Foundations to Practice

Presenter: Brandon Bohrer1, based on works with:Yong Kiam Tan1, Stefan Mitsch1, Andrew Sogokon1,

Edward Ahn1, David Held1, John Dolan1, Aman Khurana1,Magnus O. Myreen2, and Andre Platzer1

Carnegie Mellon University1

Chalmers University of Technology2

CMU V&V Workshop, Dec 12 2018

A Real Cyber-Physical System 2

A Scary Cyber-Physical System 2

VeriPhy: Automatic, Verified EXEs from Controllers 3

VeriPhy: Automatic, Verified EXEs from Controllers 3

VeriPhy: Automatic, Verified EXEs from Controllers 3

VeriPhy: Automatic, Verified EXEs from Controllers 3

VeriPhy: Automatic, Verified EXEs from Controllers 3

VeriPhy: Automatic, Verified EXEs from Controllers 3

VeriPhy: Automatic, Verified EXEs from Controllers 3

VeriPhy: Automatic, Verified EXEs from Controllers 3

HPs Model Control and Environment 4

α ≡(

(

drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪

stop︷ ︸︸ ︷v := 0); t := 0;

{env .︷ ︸︸ ︷

d ′ = −v , t ′ = 1 & t ≤ ε})∗

FarEnough? Velocity

Envelope

Fallback

PhysicsConstraint

HPs Model Control and Environment 4

α ≡(

(

drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪

stop︷ ︸︸ ︷v := 0); t := 0;

{env .︷ ︸︸ ︷

d ′ = −v , t ′ = 1 & t ≤ ε})∗

FarEnough?

VelocityEnvelope

Fallback

PhysicsConstraint

HPs Model Control and Environment 4

α ≡(

(

drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪

stop︷ ︸︸ ︷v := 0); t := 0;

{env .︷ ︸︸ ︷

d ′ = −v , t ′ = 1 & t ≤ ε})∗

FarEnough? Velocity

Envelope

Fallback

PhysicsConstraint

HPs Model Control and Environment 4

α ≡(

(

drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪

stop︷ ︸︸ ︷v := 0); t := 0;

{env .︷ ︸︸ ︷

d ′ = −v , t ′ = 1 & t ≤ ε})∗

FarEnough? Velocity

Envelope

Fallback

PhysicsConstraint

HPs Model Control and Environment 4

α ≡(

(

drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪

stop︷ ︸︸ ︷v := 0); t := 0;

{env .︷ ︸︸ ︷

d ′ = −v , t ′ = 1 & t ≤ ε})∗

FarEnough? Velocity

Envelope

Fallback

Physics

Constraint

HPs Model Control and Environment 4

α ≡(

(

drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪

stop︷ ︸︸ ︷v := 0); t := 0;

{env .︷ ︸︸ ︷

d ′ = −v , t ′ = 1 & t ≤ ε})∗

FarEnough? Velocity

Envelope

Fallback

PhysicsConstraint

KeYmaera X Enables Model Verification 5

ModelPlex: Provably Correct Monitors 6

Monitor whether transitions from previous state ~x to next state ~x+

are consistent with control, environment models.

α ≡(

(

drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪

stop︷ ︸︸ ︷v := 0); t := 0;

{env .︷ ︸︸ ︷

d ′ = −v , t ′ = 1 & t ≤ ε})∗

ModelPlex: Provably Correct Monitors 6

Monitor whether transitions from previous state ~x to next state ~x+

are consistent with control, environment models.

α ≡(

(

drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪

stop︷ ︸︸ ︷v := 0); t := 0;

{env .︷ ︸︸ ︷

d ′ = −v , t ′ = 1 & t ≤ ε})∗

ModelPlex: Provably Correct Monitors 6

Monitor whether transitions from previous state ~x to next state ~x+

are consistent with control, environment models.

α ≡(

(

drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪

stop︷ ︸︸ ︷v := 0); t := 0;

{env .︷ ︸︸ ︷

d ′ = −v , t ′ = 1 & t ≤ ε})∗

Provable Monitor Provable Sandbox 7

Sandboxed controller uses external controller when decision is safe,else uses verified fallback. Detects non-compliant plants.

V := ∗; ε := ∗; d := ∗; t := ∗; // ~x := ∗?d ≥ 0 ∧ V ≥ 0 ∧ ε ≥ 0; // ?φ(

t+ := ∗; v+ := ∗; d+ := d ; // ~x+ := extCtrl( ?ctrlMon(d , t, v , d+, t+, v+)

∪ t+ := 0; v+ := 0 ); // ~x+ := fallbackt := t+; v := v+; // ~x :=~x+

d+ := ∗; t+ := ∗; // ~x+ := ∗?plantMon(~x , ~x+);

d := d+; t := t+ // ~x :=~x+)∗

Intervals Make ctrlMon and plantMon Computable 8

Example: Check whether π < e, efficiently.Solution: Conservative interval approximation

ExampleLet νI = {pi 7→ [3, 4], e 7→ [2, 3]}, then

• pi <w e is false (⊥)

• pi <w e + 3 is true (>)• pi <w e + 1 is

Intervals Make ctrlMon and plantMon Computable 8

Example: Check whether π < e, efficiently.Solution: Conservative interval approximation

ExampleLet νI = {pi 7→ [3, 4], e 7→ [2, 3]}, then

• pi <w e is false (⊥)

• pi <w e + 3 is true (>)

• pi <w e + 1 is

Intervals Make ctrlMon and plantMon Computable 8

Example: Check whether π < e, efficiently.Solution: Conservative interval approximation

ExampleLet νI = {pi 7→ [3, 4], e 7→ [2, 3]}, then

• pi <w e is false (⊥)

• pi <w e + 3 is true (>)• pi <w e + 1 is ???

Intervals Make ctrlMon and plantMon Computable 8

Example: Check whether π < e, efficiently.Solution: Conservative interval approximation

ExampleLet νI = {pi 7→ [3, 4], e 7→ [2, 3]}, then

• pi <w e is false (⊥)

• pi <w e + 3 is true (>)

• pi <w e + 1 is a known unknown (U)When truth values can be unknown, resulting logic is 3-valued

Interval dL is 3-Valued 9

∧ > U ⊥> > U ⊥U U U ⊥⊥ ⊥ ⊥ ⊥

∨ > U ⊥> > > >U > U U⊥ > U ⊥

ωI [(θ1 + θ2)] = [l1+w l2, u1+w u2] where ωI [(θi )] = [li , ui ]

ωI [(θ1<θ2)] =

> if ωI [(θi )] = (li , ui ) and u1 < l2⊥ if ωI [(θi )] = (li , ui ) and l1 ≥ u2

U otherwise

(ωI , νI) ∈ [(α ∪ β)] iff (ωI , νI) ∈ [(α)] or (ωI , νI) ∈ [(β)]

Interval dL is a Sound Approximation 10

Theorem (Interval Soundness for Formulas)

• If ω ∈ ωI and ωI [(φ)]=> then ω ∈ [[φ]]

• If ω ∈ ωI and ωI [(φ)]=⊥ then ω /∈ [[φ]]

• No claims when ωI [(φ)]=U

Generalizes naturally to programs, but CakeML sandbox only runssimpler formula case

Sandbox HP Already Verified 11

V := ∗; ε := ∗; d := ∗; t := ∗; // ~x := ∗?d ≥ 0 ∧ V ≥ 0 ∧ ε ≥ 0; // ?φ(

t+ := ∗; v+ := ∗; d+ := d ; // ~x+ := extCtrl( ?ctrlMon(d , t, v , d+, t+, v+)

∪ t+ := 0; v+ := 0 ); // ~x+ := fallbackt := t+; v := v+; // ~x :=~x+

d+ := ∗; t+ := ∗; // ~x+ := ∗?(0≤t+≤ε ∧ d+≥v(ε− t+)

); // ?plantMon(~x , ~x+)

d := d+; t := t+ // ~x :=~x+)∗

Verified CakeML Source is Generated 11

CakeML source incorporates external control, actuation, sensing

fun cmlSandboxBody state =if not (stop ()) then

state.ctrl+:= extCtrl state;state.ctrl := if intervalSem ctrlMon state = >

then state.ctrl+

else fallback state;actuate state.ctrl;state.sensors+:= sense ();if intervalSem plantMon state = > then

Runtime.fullGC ();state.sensors := state.sensors+;cmlSandboxBody state

else violation "Plant Violation"

CakeML Sandbox is Sound 12

Theorem (Soundness for CakeML Sandbox, Main Case)If([{ω}], [{ν}]

)∈ [{cmlSandbox}] then ([(ω)], [(ν)]) ∈ [(sandbox)]

CakeML Compiler Preserves Guarantees 13

Code Executed on Sim, Soon Bot 14

Speed Ctrl Fail. Phys Fail. CollideWorld Sim Human Sim Human Sim Human Sim Human1 6.69 17.4 .431 .913 .045 .377 0 02 5.78 10.7 .632 .890 .011 .417 0 03 7.89 29.9 1 .996 .01 .151 0 0

Table : Average speed, Monitor failure rates, safety violation rates, for AirSim,F1/10, and human driver in Rectangular World, NeighborHood, and Free-Rangefor Patrol and Goto missions

Proof Chain Justifies Transformations 15

ν |= ψ

⇑

(ω, ν) ∈ [[sandbox]]

dL (KeYmaera X)

Real arithmetic,nondeterministic

⇑(ωI , νI

)∈ [(sandbox)]

dL (Isabelle/HOL)

Interval word arithmetic,nondeterministic

⇑([{ω}], [{ν}]

)∈ [{cmlSandbox}]

CakeML (HOL4)

Interval word arithmetic,deterministic

⇑({|ω|}, {|ν|}

)∈ {|CML(cmlSandbox)|}

ARM/x64

Interval word arithmetic,machine-executable

Takeaway Metaphor 16

Takeaway Metaphor 16

References I 17

Brandon Bohrer, Vincent Rahli, Ivana Vukotic, Marcus Volp,and Andre Platzer, Formally verified differential dynamic logic,Certified Programs and Proofs - 6th ACM SIGPLANConference, CPP 2017, Paris, France, January 16-17, 2017(Yves Bertot and Viktor Vafeiadis, eds.), ACM, 2017,pp. 208–221.

Joe Hurd, The OpenTheory standard theory library, NFM(Mihaela Gheorghiu Bobaru, Klaus Havelund, Gerard J.Holzmann, and Rajeev Joshi, eds.), LNCS, vol. 6617, Springer,2011, pp. 177–191.

Magnus O. Myreen and Scott Owens, Proof-producingsynthesis of ML from higher-order logic, ICFP (PeterThiemann and Robby Bruce Findler, eds.), ACM, 2012,pp. 115–126.

Isabelle/HOL Cross-Checks KeYmaera X 18

Problem: Later pipeline stages need understanding of dLsemantics, which KeYmaera X lacks

Solution: Import soundly into Isabelle/HOL from KeYmaera X

• Proof term exported from KeYmaera X, serialized• Proof checker verified in Isabelle/HOL, extending [BRV+17]• Executable checker code-generated [MO12]• Scales to 100K’s of proof steps (≈6 seconds)• Eliminates KeYmaera X core from trusted base!

Isabelle/HOL Cross-Checks KeYmaera X 18

Problem: Later pipeline stages need understanding of dLsemantics, which KeYmaera X lacksSolution: Import soundly into Isabelle/HOL from KeYmaera X

• Proof term exported from KeYmaera X, serialized• Proof checker verified in Isabelle/HOL, extending [BRV+17]

• Executable checker code-generated [MO12]• Scales to 100K’s of proof steps (≈6 seconds)• Eliminates KeYmaera X core from trusted base!

Isabelle/HOL Cross-Checks KeYmaera X 18

Problem: Later pipeline stages need understanding of dLsemantics, which KeYmaera X lacksSolution: Import soundly into Isabelle/HOL from KeYmaera X

• Proof term exported from KeYmaera X, serialized• Proof checker verified in Isabelle/HOL, extending [BRV+17]• Executable checker code-generated [MO12]• Scales to 100K’s of proof steps (≈6 seconds)• Eliminates KeYmaera X core from trusted base!

Isabelle/HOL → HOL4 Translation is Trusted 19

Isabelle/HOL Strength: Library Access• Analysis libraries (absolute must for dL soundness)• Machine word libraries (must for interval arithmetic)

Isabelle/HOL Weakness: Weaker Verified Compiler Support• This is a problem: need to generate source code!

We jump to HOL4 for access to verified CakeML compiler:• Manually translate Isabelle/HOL definitions to HOL4• Justification: Similar logical foundation• Could be automated in principle, see OpenTheory [Hur11]

Isabelle/HOL → HOL4 Translation is Trusted 19

Isabelle/HOL Strength: Library Access• Analysis libraries (absolute must for dL soundness)• Machine word libraries (must for interval arithmetic)

Isabelle/HOL Weakness: Weaker Verified Compiler Support• This is a problem: need to generate source code!

We jump to HOL4 for access to verified CakeML compiler:• Manually translate Isabelle/HOL definitions to HOL4• Justification: Similar logical foundation• Could be automated in principle, see OpenTheory [Hur11]

Isabelle/HOL → HOL4 Translation is Trusted 19

Isabelle/HOL Strength: Library Access• Analysis libraries (absolute must for dL soundness)• Machine word libraries (must for interval arithmetic)

Isabelle/HOL Weakness: Weaker Verified Compiler Support• This is a problem: need to generate source code!

We jump to HOL4 for access to verified CakeML compiler:• Manually translate Isabelle/HOL definitions to HOL4• Justification: Similar logical foundation• Could be automated in principle, see OpenTheory [Hur11]

Future Work 20

Improve pipeline components:• Reduce trusted base: OpenTheory, arithmetic witnesses in

KeYmaera X• Floating-point, mixed precision interval arithmetic• Generalize proof-driven monitor synthesis

Exploit pipeline in case studies:• UAVs• High-speed robots• Your favorite CPS