Verified Runtime Monitoring: From Foundations to Practice · Verified Runtime Monitoring: From...
Transcript of Verified Runtime Monitoring: From Foundations to Practice · Verified Runtime Monitoring: From...
Verified Runtime Monitoring:From Foundations to Practice
Presenter: Brandon Bohrer1, based on works with:Yong Kiam Tan1, Stefan Mitsch1, Andrew Sogokon1,
Edward Ahn1, David Held1, John Dolan1, Aman Khurana1,Magnus O. Myreen2, and Andre Platzer1
Carnegie Mellon University1
Chalmers University of Technology2
CMU V&V Workshop, Dec 12 2018
A Real Cyber-Physical System 2
A Scary Cyber-Physical System 2
VeriPhy: Automatic, Verified EXEs from Controllers 3
VeriPhy: Automatic, Verified EXEs from Controllers 3
VeriPhy: Automatic, Verified EXEs from Controllers 3
VeriPhy: Automatic, Verified EXEs from Controllers 3
VeriPhy: Automatic, Verified EXEs from Controllers 3
VeriPhy: Automatic, Verified EXEs from Controllers 3
VeriPhy: Automatic, Verified EXEs from Controllers 3
VeriPhy: Automatic, Verified EXEs from Controllers 3
HPs Model Control and Environment 4
α ≡(
(
drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪
stop︷ ︸︸ ︷v := 0); t := 0;
{env .︷ ︸︸ ︷
d ′ = −v , t ′ = 1 & t ≤ ε})∗
FarEnough? Velocity
Envelope
Fallback
PhysicsConstraint
HPs Model Control and Environment 4
α ≡(
(
drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪
stop︷ ︸︸ ︷v := 0); t := 0;
{env .︷ ︸︸ ︷
d ′ = −v , t ′ = 1 & t ≤ ε})∗
FarEnough?
VelocityEnvelope
Fallback
PhysicsConstraint
HPs Model Control and Environment 4
α ≡(
(
drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪
stop︷ ︸︸ ︷v := 0); t := 0;
{env .︷ ︸︸ ︷
d ′ = −v , t ′ = 1 & t ≤ ε})∗
FarEnough? Velocity
Envelope
Fallback
PhysicsConstraint
HPs Model Control and Environment 4
α ≡(
(
drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪
stop︷ ︸︸ ︷v := 0); t := 0;
{env .︷ ︸︸ ︷
d ′ = −v , t ′ = 1 & t ≤ ε})∗
FarEnough? Velocity
Envelope
Fallback
PhysicsConstraint
HPs Model Control and Environment 4
α ≡(
(
drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪
stop︷ ︸︸ ︷v := 0); t := 0;
{env .︷ ︸︸ ︷
d ′ = −v , t ′ = 1 & t ≤ ε})∗
FarEnough? Velocity
Envelope
Fallback
Physics
Constraint
HPs Model Control and Environment 4
α ≡(
(
drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪
stop︷ ︸︸ ︷v := 0); t := 0;
{env .︷ ︸︸ ︷
d ′ = −v , t ′ = 1 & t ≤ ε})∗
FarEnough? Velocity
Envelope
Fallback
PhysicsConstraint
KeYmaera X Enables Model Verification 5
ModelPlex: Provably Correct Monitors 6
Monitor whether transitions from previous state ~x to next state ~x+
are consistent with control, environment models.
α ≡(
(
drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪
stop︷ ︸︸ ︷v := 0); t := 0;
{env .︷ ︸︸ ︷
d ′ = −v , t ′ = 1 & t ≤ ε})∗
ModelPlex: Provably Correct Monitors 6
Monitor whether transitions from previous state ~x to next state ~x+
are consistent with control, environment models.
α ≡(
(
drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪
stop︷ ︸︸ ︷v := 0); t := 0;
{env .︷ ︸︸ ︷
d ′ = −v , t ′ = 1 & t ≤ ε})∗
ModelPlex: Provably Correct Monitors 6
Monitor whether transitions from previous state ~x to next state ~x+
are consistent with control, environment models.
α ≡(
(
drive︷ ︸︸ ︷?d ≥ εV ; v := ∗; ?0 ≤ v ≤ V ∪
stop︷ ︸︸ ︷v := 0); t := 0;
{env .︷ ︸︸ ︷
d ′ = −v , t ′ = 1 & t ≤ ε})∗
Provable Monitor Provable Sandbox 7
Sandboxed controller uses external controller when decision is safe,else uses verified fallback. Detects non-compliant plants.
V := ∗; ε := ∗; d := ∗; t := ∗; // ~x := ∗?d ≥ 0 ∧ V ≥ 0 ∧ ε ≥ 0; // ?φ(
t+ := ∗; v+ := ∗; d+ := d ; // ~x+ := extCtrl( ?ctrlMon(d , t, v , d+, t+, v+)
∪ t+ := 0; v+ := 0 ); // ~x+ := fallbackt := t+; v := v+; // ~x :=~x+
d+ := ∗; t+ := ∗; // ~x+ := ∗?plantMon(~x , ~x+);
d := d+; t := t+ // ~x :=~x+)∗
Intervals Make ctrlMon and plantMon Computable 8
Example: Check whether π < e, efficiently.Solution: Conservative interval approximation
ExampleLet νI = {pi 7→ [3, 4], e 7→ [2, 3]}, then
• pi <w e is false (⊥)
• pi <w e + 3 is true (>)• pi <w e + 1 is
Intervals Make ctrlMon and plantMon Computable 8
Example: Check whether π < e, efficiently.Solution: Conservative interval approximation
ExampleLet νI = {pi 7→ [3, 4], e 7→ [2, 3]}, then
• pi <w e is false (⊥)
• pi <w e + 3 is true (>)
• pi <w e + 1 is
Intervals Make ctrlMon and plantMon Computable 8
Example: Check whether π < e, efficiently.Solution: Conservative interval approximation
ExampleLet νI = {pi 7→ [3, 4], e 7→ [2, 3]}, then
• pi <w e is false (⊥)
• pi <w e + 3 is true (>)• pi <w e + 1 is ???
Intervals Make ctrlMon and plantMon Computable 8
Example: Check whether π < e, efficiently.Solution: Conservative interval approximation
ExampleLet νI = {pi 7→ [3, 4], e 7→ [2, 3]}, then
• pi <w e is false (⊥)
• pi <w e + 3 is true (>)
• pi <w e + 1 is a known unknown (U)When truth values can be unknown, resulting logic is 3-valued
Interval dL is 3-Valued 9
∧ > U ⊥> > U ⊥U U U ⊥⊥ ⊥ ⊥ ⊥
∨ > U ⊥> > > >U > U U⊥ > U ⊥
ωI [(θ1 + θ2)] = [l1+w l2, u1+w u2] where ωI [(θi )] = [li , ui ]
ωI [(θ1<θ2)] =
> if ωI [(θi )] = (li , ui ) and u1 < l2⊥ if ωI [(θi )] = (li , ui ) and l1 ≥ u2
U otherwise
(ωI , νI) ∈ [(α ∪ β)] iff (ωI , νI) ∈ [(α)] or (ωI , νI) ∈ [(β)]
Interval dL is a Sound Approximation 10
Theorem (Interval Soundness for Formulas)
• If ω ∈ ωI and ωI [(φ)]=> then ω ∈ [[φ]]
• If ω ∈ ωI and ωI [(φ)]=⊥ then ω /∈ [[φ]]
• No claims when ωI [(φ)]=U
Generalizes naturally to programs, but CakeML sandbox only runssimpler formula case
Sandbox HP Already Verified 11
V := ∗; ε := ∗; d := ∗; t := ∗; // ~x := ∗?d ≥ 0 ∧ V ≥ 0 ∧ ε ≥ 0; // ?φ(
t+ := ∗; v+ := ∗; d+ := d ; // ~x+ := extCtrl( ?ctrlMon(d , t, v , d+, t+, v+)
∪ t+ := 0; v+ := 0 ); // ~x+ := fallbackt := t+; v := v+; // ~x :=~x+
d+ := ∗; t+ := ∗; // ~x+ := ∗?(0≤t+≤ε ∧ d+≥v(ε− t+)
); // ?plantMon(~x , ~x+)
d := d+; t := t+ // ~x :=~x+)∗
Verified CakeML Source is Generated 11
CakeML source incorporates external control, actuation, sensing
fun cmlSandboxBody state =if not (stop ()) then
state.ctrl+:= extCtrl state;state.ctrl := if intervalSem ctrlMon state = >
then state.ctrl+
else fallback state;actuate state.ctrl;state.sensors+:= sense ();if intervalSem plantMon state = > then
Runtime.fullGC ();state.sensors := state.sensors+;cmlSandboxBody state
else violation "Plant Violation"
CakeML Sandbox is Sound 12
Theorem (Soundness for CakeML Sandbox, Main Case)If([{ω}], [{ν}]
)∈ [{cmlSandbox}] then ([(ω)], [(ν)]) ∈ [(sandbox)]
CakeML Compiler Preserves Guarantees 13
Code Executed on Sim, Soon Bot 14
Speed Ctrl Fail. Phys Fail. CollideWorld Sim Human Sim Human Sim Human Sim Human1 6.69 17.4 .431 .913 .045 .377 0 02 5.78 10.7 .632 .890 .011 .417 0 03 7.89 29.9 1 .996 .01 .151 0 0
Table : Average speed, Monitor failure rates, safety violation rates, for AirSim,F1/10, and human driver in Rectangular World, NeighborHood, and Free-Rangefor Patrol and Goto missions
Proof Chain Justifies Transformations 15
ν |= ψ
⇑
(ω, ν) ∈ [[sandbox]]
dL (KeYmaera X)
Real arithmetic,nondeterministic
⇑(ωI , νI
)∈ [(sandbox)]
dL (Isabelle/HOL)
Interval word arithmetic,nondeterministic
⇑([{ω}], [{ν}]
)∈ [{cmlSandbox}]
CakeML (HOL4)
Interval word arithmetic,deterministic
⇑({|ω|}, {|ν|}
)∈ {|CML(cmlSandbox)|}
ARM/x64
Interval word arithmetic,machine-executable
Takeaway Metaphor 16
Takeaway Metaphor 16
References I 17
Brandon Bohrer, Vincent Rahli, Ivana Vukotic, Marcus Volp,and Andre Platzer, Formally verified differential dynamic logic,Certified Programs and Proofs - 6th ACM SIGPLANConference, CPP 2017, Paris, France, January 16-17, 2017(Yves Bertot and Viktor Vafeiadis, eds.), ACM, 2017,pp. 208–221.
Joe Hurd, The OpenTheory standard theory library, NFM(Mihaela Gheorghiu Bobaru, Klaus Havelund, Gerard J.Holzmann, and Rajeev Joshi, eds.), LNCS, vol. 6617, Springer,2011, pp. 177–191.
Magnus O. Myreen and Scott Owens, Proof-producingsynthesis of ML from higher-order logic, ICFP (PeterThiemann and Robby Bruce Findler, eds.), ACM, 2012,pp. 115–126.
Isabelle/HOL Cross-Checks KeYmaera X 18
Problem: Later pipeline stages need understanding of dLsemantics, which KeYmaera X lacks
Solution: Import soundly into Isabelle/HOL from KeYmaera X
• Proof term exported from KeYmaera X, serialized• Proof checker verified in Isabelle/HOL, extending [BRV+17]• Executable checker code-generated [MO12]• Scales to 100K’s of proof steps (≈6 seconds)• Eliminates KeYmaera X core from trusted base!
Isabelle/HOL Cross-Checks KeYmaera X 18
Problem: Later pipeline stages need understanding of dLsemantics, which KeYmaera X lacksSolution: Import soundly into Isabelle/HOL from KeYmaera X
• Proof term exported from KeYmaera X, serialized• Proof checker verified in Isabelle/HOL, extending [BRV+17]
• Executable checker code-generated [MO12]• Scales to 100K’s of proof steps (≈6 seconds)• Eliminates KeYmaera X core from trusted base!
Isabelle/HOL Cross-Checks KeYmaera X 18
Problem: Later pipeline stages need understanding of dLsemantics, which KeYmaera X lacksSolution: Import soundly into Isabelle/HOL from KeYmaera X
• Proof term exported from KeYmaera X, serialized• Proof checker verified in Isabelle/HOL, extending [BRV+17]• Executable checker code-generated [MO12]• Scales to 100K’s of proof steps (≈6 seconds)• Eliminates KeYmaera X core from trusted base!
Isabelle/HOL → HOL4 Translation is Trusted 19
Isabelle/HOL Strength: Library Access• Analysis libraries (absolute must for dL soundness)• Machine word libraries (must for interval arithmetic)
Isabelle/HOL Weakness: Weaker Verified Compiler Support• This is a problem: need to generate source code!
We jump to HOL4 for access to verified CakeML compiler:• Manually translate Isabelle/HOL definitions to HOL4• Justification: Similar logical foundation• Could be automated in principle, see OpenTheory [Hur11]
Isabelle/HOL → HOL4 Translation is Trusted 19
Isabelle/HOL Strength: Library Access• Analysis libraries (absolute must for dL soundness)• Machine word libraries (must for interval arithmetic)
Isabelle/HOL Weakness: Weaker Verified Compiler Support• This is a problem: need to generate source code!
We jump to HOL4 for access to verified CakeML compiler:• Manually translate Isabelle/HOL definitions to HOL4• Justification: Similar logical foundation• Could be automated in principle, see OpenTheory [Hur11]
Isabelle/HOL → HOL4 Translation is Trusted 19
Isabelle/HOL Strength: Library Access• Analysis libraries (absolute must for dL soundness)• Machine word libraries (must for interval arithmetic)
Isabelle/HOL Weakness: Weaker Verified Compiler Support• This is a problem: need to generate source code!
We jump to HOL4 for access to verified CakeML compiler:• Manually translate Isabelle/HOL definitions to HOL4• Justification: Similar logical foundation• Could be automated in principle, see OpenTheory [Hur11]
Future Work 20
Improve pipeline components:• Reduce trusted base: OpenTheory, arithmetic witnesses in
KeYmaera X• Floating-point, mixed precision interval arithmetic• Generalize proof-driven monitor synthesis
Exploit pipeline in case studies:• UAVs• High-speed robots• Your favorite CPS