Exploratory Analysis of the Performance of a Configurable CEGAR Framework
-
Upload
akos-hajdu -
Category
Science
-
view
103 -
download
3
Transcript of Exploratory Analysis of the Performance of a Configurable CEGAR Framework
![Page 1: Exploratory Analysis of the Performance of a Configurable CEGAR Framework](https://reader034.fdocuments.in/reader034/viewer/2022051507/5899a47c1a28ab30688b5d6d/html5/thumbnails/1.jpg)
Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems
Exploratory Analysis of the Performance of a Configurable CEGAR Framework
Ákos Hajdu1,2, Zoltán Micskei1
1Budapest University of Technology and Economics,
Department of Measurement and Information Systems2MTA-BME Lendület Cyber-Physical Systems Research Group
24th Minisymposium of DMIS, 31.01.2017.
1
![Page 2: Exploratory Analysis of the Performance of a Configurable CEGAR Framework](https://reader034.fdocuments.in/reader034/viewer/2022051507/5899a47c1a28ab30688b5d6d/html5/thumbnails/2.jpg)
Background – Formal verification
2
Real-life system
Formal model Formal requirement
Verification: explore states
CEGAR
Safe Counterexample
Abstraction Refinement
¬(Red Ʌ Green)
![Page 3: Exploratory Analysis of the Performance of a Configurable CEGAR Framework](https://reader034.fdocuments.in/reader034/viewer/2022051507/5899a47c1a28ab30688b5d6d/html5/thumbnails/3.jpg)
Motivation
Configurable CEGAR framework
o Different algorithm configurations
o Different kinds of models
Which is the “best” configuration?
Preliminary experiment and evaluation
3
Á. Hajdu, T. Tóth, A. Vörös, and I. Majzik, “A configurable CEGAR framework withinterpolation-based refinements,” in Formal Techniques for Distributed Objects,Components and Systems, ser. LNCS. Springer, 2016, vol. 9688, pp. 158–174.
![Page 4: Exploratory Analysis of the Performance of a Configurable CEGAR Framework](https://reader034.fdocuments.in/reader034/viewer/2022051507/5899a47c1a28ab30688b5d6d/html5/thumbnails/4.jpg)
Variables of the problem
Input variables: model
o System type (Hardware/PLC)
o Name
o Number of variables
o Size
Input variables: configuration
o Domain of abstraction (Pred./Expl.)
o Refinement strategy (Craig itp./Seq. itp./Unsat core)
o Initial precision (Empty/Prop.)
o Search strategy (BFS/DFS)
4
![Page 5: Exploratory Analysis of the Performance of a Configurable CEGAR Framework](https://reader034.fdocuments.in/reader034/viewer/2022051507/5899a47c1a28ab30688b5d6d/html5/thumbnails/5.jpg)
Variables of the problem
Output variables
o Is the model safe
o Execution time
o Number of refinement iterations
o Size of the ARG (Abstract Reachability Graph)
o Depth of the ARG
o Length of the counterexample (cex)
5
![Page 6: Exploratory Analysis of the Performance of a Configurable CEGAR Framework](https://reader034.fdocuments.in/reader034/viewer/2022051507/5899a47c1a28ab30688b5d6d/html5/thumbnails/6.jpg)
Measurement procedure
18 input models
o 12 hardware (benchmarks from HWMCC)
o 6 PLC (from a particle accelerator)
20 algorithm configurations
Repeated 5 times
Timeout 480 s
1800 measurement points, 1120 successful
6
![Page 7: Exploratory Analysis of the Performance of a Configurable CEGAR Framework](https://reader034.fdocuments.in/reader034/viewer/2022051507/5899a47c1a28ab30688b5d6d/html5/thumbnails/7.jpg)
Research questions
RQ1: Overall, high level properties
RQ2: Effect of individual input parameters
RQ3: Influence of input parameters on output
Validity
o External: representative input models
o Internal: repetitions, dedicated machine
7
![Page 8: Exploratory Analysis of the Performance of a Configurable CEGAR Framework](https://reader034.fdocuments.in/reader034/viewer/2022051507/5899a47c1a28ab30688b5d6d/html5/thumbnails/8.jpg)
RQ1: Overall, high level properties
8
Many outliers
Small IQR
![Page 9: Exploratory Analysis of the Performance of a Configurable CEGAR Framework](https://reader034.fdocuments.in/reader034/viewer/2022051507/5899a47c1a28ab30688b5d6d/html5/thumbnails/9.jpg)
RQ1: Overall, high level properties
9
Average execution time (ms, log scale)
Easy problems Varying difficulty
High success rate
Single configuration, but short time
PredSeq. Itp.Prop.DFS
![Page 10: Exploratory Analysis of the Performance of a Configurable CEGAR Framework](https://reader034.fdocuments.in/reader034/viewer/2022051507/5899a47c1a28ab30688b5d6d/html5/thumbnails/10.jpg)
RQ2: Effect of individual input parameters
10
Explicit value abstraction more efficient for PLCs
Execution time (ms)
![Page 11: Exploratory Analysis of the Performance of a Configurable CEGAR Framework](https://reader034.fdocuments.in/reader034/viewer/2022051507/5899a47c1a28ab30688b5d6d/html5/thumbnails/11.jpg)
RQ2: Effect of individual input parameters
11
Number of iterations
Less iterations with seq. itp.
Large difference for some PLCs
![Page 12: Exploratory Analysis of the Performance of a Configurable CEGAR Framework](https://reader034.fdocuments.in/reader034/viewer/2022051507/5899a47c1a28ab30688b5d6d/html5/thumbnails/12.jpg)
RQ3: Influence of input parameters on output
12
Predicate domain bad for PLCs
Predicate domain good for hardware
Explicit domain with Craig itp. good in general
![Page 13: Exploratory Analysis of the Performance of a Configurable CEGAR Framework](https://reader034.fdocuments.in/reader034/viewer/2022051507/5899a47c1a28ab30688b5d6d/html5/thumbnails/13.jpg)
Conclusions CEGAR framework
o Different configurations
o Different systems
Preliminary results
o Different configurations are moresuitable for different tasks
o Connections between input andoutput variables
Future work
o Improving the framework
o Further analysis, heuristics
13
inf.mit.bme.hu/en/members/hajdua