Thoughts on Metrics & Benchmarking Randal E. Bryant.
-
Upload
ashlynn-fox -
Category
Documents
-
view
213 -
download
1
Transcript of Thoughts on Metrics & Benchmarking Randal E. Bryant.
![Page 1: Thoughts on Metrics & Benchmarking Randal E. Bryant.](https://reader036.fdocuments.in/reader036/viewer/2022083005/56649f295503460f94c41b90/html5/thumbnails/1.jpg)
Thoughts on Metrics & Thoughts on Metrics & BenchmarkingBenchmarking
Thoughts on Metrics & Thoughts on Metrics & BenchmarkingBenchmarking
Randal E. Bryant
![Page 2: Thoughts on Metrics & Benchmarking Randal E. Bryant.](https://reader036.fdocuments.in/reader036/viewer/2022083005/56649f295503460f94c41b90/html5/thumbnails/2.jpg)
– 2 –
Program ComplexityProgram Complexity
Complexity SourcesComplexity Sources Control
Concurrency Error handling & mitigation
Data Complex data structures Data structure invariants
Data
Control
EmbeddedSystems
My Systems
![Page 3: Thoughts on Metrics & Benchmarking Randal E. Bryant.](https://reader036.fdocuments.in/reader036/viewer/2022083005/56649f295503460f94c41b90/html5/thumbnails/3.jpg)
– 3 –
Role of BenchmarksRole of Benchmarks
GoodGood Provides objective standard for comparison Competition drives progress
BadBad Can become goal rather than means to achieve goal Poorly selected set can thwart progress
E.g., DIMACS SAT benchmarks
![Page 4: Thoughts on Metrics & Benchmarking Randal E. Bryant.](https://reader036.fdocuments.in/reader036/viewer/2022083005/56649f295503460f94c41b90/html5/thumbnails/4.jpg)
– 4 –
Benchmarking DesiderataBenchmarking Desiderata
Organic ProcessOrganic Process E.g., SPEC, SAT Continually update Large community involved
SelectionSelection Range of sizes From different sources (applications, organizations)
Desirable, but Uncommon:Desirable, but Uncommon: Parameterized Access to originating application
![Page 5: Thoughts on Metrics & Benchmarking Randal E. Bryant.](https://reader036.fdocuments.in/reader036/viewer/2022083005/56649f295503460f94c41b90/html5/thumbnails/5.jpg)
– 5 –
Some IdeasSome Ideas
Select Small Set of Driving ApplicationsSelect Small Set of Driving Applications Control intensive embedded system(s) Data intensive
Create Supporting BenchmarksCreate Supporting Benchmarks Scaled down versions or components of driving applications
IssuesIssues Language Ancilliary code: Libraries, OS Specs.