Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods
description
Transcript of Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods
![Page 1: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/1.jpg)
Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods
Erik SeligmanCS 510, Lecture 13, February 2009
![Page 2: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/2.jpg)
Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments
![Page 3: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/3.jpg)
Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments
![Page 4: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/4.jpg)
Bounded FPV Majority of FPV work done at low bound
• Around 50-100 cycles• Many modules testable this way• Limits of engines: exponential blowup
Good in many (most?) cases• Verification of isolated modules• Tricky bugs often don’t need much depth
– with right input & constraints
• Cases where deep queues/counters not critical Double-check with coverage points!
![Page 5: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/5.jpg)
Problem: FPV Bounds Bounded FPV is ignoring possibilities
• Known industry cases of “deep” errors Deep bugs suspected in some cases
• Example: 2 independent 13-bit ctrs– How many combinations possible?– How many cycles needed?
• Bugs may be lurking beyond FPV bounds• Maybe too much logic to solve with pruning
Strategies desired for hi-bound FPV
![Page 6: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/6.jpg)
Compromise: Abstraction Counters, state machines: free outputs
• As we saw in memory controller exampleAll values possible
• Useful in many cases• But at significant cost
• Realism of cases: values jump around• No coverage for abstracted logic
![Page 7: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/7.jpg)
Compromise: Initial State Set initial state to something ‘interesting’
• Fill queues, set high counters, etc. FPV no longer exhaustive
• Errors from different initial state not covered• Focus is bug hunting in suspicious area
Can we do better?
![Page 8: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/8.jpg)
Limits of Pruning Pruning often helps complexity issues
• In best case, bounded full proof• Or can deepen bounds
But it can’t help in worst cases• Maybe long, complex interactions of parts
– Think about 2-counter example again
• Lots of logic may be involved
![Page 9: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/9.jpg)
Key Insight: Leverage Simulation Simulation env has design knowledge
• Usually set up to create interesting states• Wide variety of states possible in sim
Can we leverage simulation for FPV?• Simple example: Snapshot a simulation
state, use as FPV input• Are there bigger opportunities?
![Page 10: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/10.jpg)
Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments
![Page 11: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/11.jpg)
0in Dynamic Formal FPV Monitor simulation runs
• Tool runs as plugin to simulator Identify “interesting” states
• Hit cover point• Transition counter, state machine, etc.
Launch many low-bound FPV runs
![Page 12: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/12.jpg)
Ordinary FPV & State Space
• Small proof radius covered after reset
![Page 13: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/13.jpg)
Dynamic Formal Approach
• Many mini-FPV runs launched from sim• Smaller proof radius for each FPV run
![Page 14: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/14.jpg)
Advantages/Disadvantages Advantages
• Based on real tests likely good examples• Produce realistic counterexamples since
most cycles are from simulation• Finds deep bugs “almost” tested in sim
Disadvantages?
![Page 15: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/15.jpg)
Advantages/Disadvantages Advantages
• Based on real tests likely good examples• Produce realistic counterexamples since most
cycles are from simulation• Finds deep bugs “almost” tested in sim
Disadvantages?• Depends on setting up simulation env
– Not usable really early or arbitrary hierarchy• Can’t find counterexamples distant from tests
– Misses major selling point of FPV: corner cases not conceived by designer
• Tool pain: FPV & sim tools interact?– Harder than it sounds!
![Page 16: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/16.jpg)
Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments
![Page 17: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/17.jpg)
Synopsys Magellan Approach Similar motivations to 0in Recognize problem: simulation env
• Few real designers use ‘raw’ sim tool• Many layers of scripts & wrappers• Really painful to enable 0in-like method
Solution: Random Simulation• Still uses simulation engine• Randomly tries to simulate to get cover pts
– No use of real tests like 0in
![Page 18: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/18.jpg)
Hybrid Formal Approach
• Lots of random simulation paths• Many mini-FPV runs launched from sim• Smaller proof radius for each FPV run
![Page 19: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/19.jpg)
Advantages/Disadvantages Advantages
• Not limited by tests from validation team• Can theoretically find very deep bugs• Independent of simulation environment
Disadvantages?
![Page 20: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/20.jpg)
Advantages/Disadvantages Advantages
• Not limited by tests from validation team• Can theoretically find very deep bugs• Independent of simulation environment
Disadvantages?• Very random: need to get lucky for bug
– Loses both comprehensiveness & test-guidance– Good cover points may mitigate
• Still have sim/fpv tool integration issues– Many “simulation-synthesis mismatches”: time delays,
system functions, non-det behavior, X/Z vals, etc.
![Page 21: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/21.jpg)
Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments
![Page 22: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/22.jpg)
Can We Use A Non-Simulation Approach? FPV user suspects deep error
• But proof bounds not enough• Standard complexity techniques not working
What does FPV user know?• Probably has likely scenarios in mind• But FPV tool isn’t getting there
So why not use user-guided FPV?• Try to manually get FPV tool close to error• Then launch bounded, comprehensive run
![Page 23: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/23.jpg)
User-Guided FPV Create cover points for reset state
• Suspected intermediate state on way to err Ask FPV tool to reach cover point
• CEX at cover == reset state for next• Save state manually
– Nicer if tool help, but not seen in current tools
After cover pt, use real bounded FPV• Run isn’t full formal proof of properties• But is comprehensive bug hunt from there
![Page 24: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/24.jpg)
User Guided Approach
• Use FPV engine to visit cover point• Launch bounded FPV from targeted pt
![Page 25: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/25.jpg)
Advantages/Disadvantages Advantages
• Completely eliminates sim engine issues• Can theoretically find very deep bugs• Takes advantage of user knowledge
Disadvantages?
![Page 26: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/26.jpg)
Advantages/Disadvantages Advantages
• Completely eliminates sim engine issues• Can theoretically find very deep bugs• Takes advantage of user knowledge
Disadvantages?• Highly dependent on user intuition
– Loses ‘automatic search’ aspect of 0in/Synopsys
• Problem areas must be specifically targeted
![Page 27: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/27.jpg)
Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments
![Page 28: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/28.jpg)
Deep Bounds: Why ‘Sexy’? Much effort in industry/academia Recognition of limits of bounded FPV Potential for fame and fortune
• Really cool to claim find of 10000-cycle-deep error nobody noticed!
• Common view: One such error justifies cost of full deployment + BMW for the validator
But is this really the key FPV problem?
![Page 29: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/29.jpg)
Where Do FPV users Spend Their Time? Intuitive explanation, take with grain of salt 70% “Wiggling”
• Early attempts with quick counterexamples• Root-causing CEXs, adding assumptions, & finding occasional
shallow bugs• Many FPV efforts never get beyond this stage
20% “Solidifying”• Have bounded proofs on most assertions• Incrementally deepening bounds, checking covers
10% “Heroics”• Extra, expert efforts to get full proofs or deep proofs in
suspected problem areas • Trying exotic FPV technologies
![Page 30: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/30.jpg)
Where Should FPV Research Be Concentrated? Vast majority of FPV effort is “wiggling”
• Assuming previous slide roughly accurate! Then next largest chunk is “solidifying”
• Understaning, improving low bounds• Basic complexity issues
What are FPVers actually doing?• Running low-bound proof attempts• Root-causing failures to add assumptions• Attacking basic complexity issues
(IMHO) The real problem is Usability
![Page 31: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/31.jpg)
What Would An FPV Breakthrough Look Like? Need quantum leaps in productivity of
“wiggling” stage• Secondary priority: addressing complexity
Key functionality I want to see• Viewing and understanding counterexamples• More & better live what-if experiments• Intuitive user interaction• Really fast incremental responsiveness • Support for interactive complexity analysis
![Page 32: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/32.jpg)
My Advice To Academia Better engines & deeper proofs are nice
• But only leveraged by small group of engineers who progress to “heroic” FPV stage
Best impact: where engineers spend time• Wiggling stage: quickly understanding cex• Solidifying stage: root-causing & fixing complexity
Focus on usability, interactivity, and incremental resposiveness!
![Page 33: Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167d9550346895ddd346d/html5/thumbnails/33.jpg)
References / Further Reading http://www.edacafe.com/Vision/200208/d
esign3.html http://drona.csa.iisc.ernet.in/~deepakd/tal
ks/formal-iisc-0306.pdf http://www.te.rl.ac.uk/europractice/vendo
rs/magellan_ds.pdf