EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark...
Transcript of EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark...
![Page 1: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/1.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench, a tool to help you benchmark and bisectthe Graphics Stack’s performance
Martin Peres
Intel Open Source Technology Center Finland
September 23, 2016
![Page 2: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/2.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
Summary
1 Introduction
2 Graphics Continuous Integration
3 EzBench
4 Conclusion
![Page 3: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/3.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
Introduction
Introduction
Current situation
Complex games/benchmarks are available and used on Linux;
Drivers are getting more complex as performance improves;
Users now rely on Open Source drivers for games/apps...
Risks when merging new code
Break previous functionalities / rendering;
Break the performance of a game inadvertly;
Improve the performance of one app but slow down others.
⇒ Need to test and benchmark all the platforms and games ofinterest.
![Page 4: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/4.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
Introduction
Introduction
Current situation
Complex games/benchmarks are available and used on Linux;
Drivers are getting more complex as performance improves;
Users now rely on Open Source drivers for games/apps...
Risks when merging new code
Break previous functionalities / rendering;
Break the performance of a game inadvertly;
Improve the performance of one app but slow down others.
⇒ Need to test and benchmark all the platforms and games ofinterest.
![Page 5: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/5.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
Introduction
Introduction
Current situation
Complex games/benchmarks are available and used on Linux;
Drivers are getting more complex as performance improves;
Users now rely on Open Source drivers for games/apps...
Risks when merging new code
Break previous functionalities / rendering;
Break the performance of a game inadvertly;
Improve the performance of one app but slow down others.
⇒ Need to test and benchmark all the platforms and games ofinterest.
![Page 6: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/6.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
Summary
1 Introduction
2 Graphics Continuous Integration
3 EzBench
4 Conclusion
![Page 7: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/7.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
Objectives
CI: Objectives and chalenges
Objectives
Catch changes in unit tests, rendering, performance or power;
Pin-point the change, to help bug-reporting and fixing;
Guarantee reproducibility of the results;
Warn the relevant developers of changes.
Challenges
Unit tests, performance, metrics or rendering can be unstable;
Multiple components interacting with each-other;
Avoid false positives and false negatives;
Impossible to test every commit.
![Page 8: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/8.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
Objectives
CI: Objectives and chalenges
Objectives
Catch changes in unit tests, rendering, performance or power;
Pin-point the change, to help bug-reporting and fixing;
Guarantee reproducibility of the results;
Warn the relevant developers of changes.
Challenges
Unit tests, performance, metrics or rendering can be unstable;
Multiple components interacting with each-other;
Avoid false positives and false negatives;
Impossible to test every commit.
![Page 9: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/9.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
Current solutions
CI: Current tools
Current solutions
Unit testing: Piglit, dEQP, gl-CTS, vk-CTS, more...;
Performance: Phoronix Test Suite, Sixonix;
Rendering: Phoronix Test Suite, Anholt’s trace-db;
Job scheduling: Phoronix Test Suite, Jenkins, ...
Issue: Great for reporting, not for bisecting
No feedback loop to address variance issues;
Environment may have changed;
Unit tests may flip/flop;
Rendering may be unstable (yes, it does happen);
Solution: external runner for them to take care of this!
![Page 10: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/10.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
Current solutions
CI: Current tools
Current solutions
Unit testing: Piglit, dEQP, gl-CTS, vk-CTS, more...;
Performance: Phoronix Test Suite, Sixonix;
Rendering: Phoronix Test Suite, Anholt’s trace-db;
Job scheduling: Phoronix Test Suite, Jenkins, ...
Issue: Great for reporting, not for bisecting
No feedback loop to address variance issues;
Environment may have changed;
Unit tests may flip/flop;
Rendering may be unstable (yes, it does happen);
Solution: external runner for them to take care of this!
![Page 11: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/11.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
Summary
1 Introduction
2 Graphics Continuous Integration
3 EzBench
4 Conclusion
![Page 12: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/12.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: General architecture
General flow graph
Acquiredata
Generatereport
Enhancereport
Scheduleenhance-
ments
Blocks
Acquire data: Compile/deploy, run tests and collect data/env;
Generate report: Read from the disk, create a python IR;
Enhance report: Analyse the data, find changes, report events;
Schedule enhancements: Request more data (bisect!).
![Page 13: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/13.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: General architecture
General flow graph
Acquiredata
Generatereport
Enhancereport
Scheduleenhance-
ments
Blocks
Acquire data: Compile/deploy, run tests and collect data/env;
Generate report: Read from the disk, create a python IR;
Enhance report: Analyse the data, find changes, report events;
Schedule enhancements: Request more data (bisect!).
![Page 14: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/14.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Code and license
MIT-licensed code
Available at https://cgit.freedesktop.org/ezbench/
Blocks
runner: bash-based, handles:
compilation and deployment of the component;setting up the environment (X, compositor);running the test.
env-dump.so: LD PRELOADed C library:
dump the environments and loaded libs;hook interesting calls (GLX, EGL, GL, X);dump metrics (RAPL, GPU temperature and power usage).
Report generation, enhancing and scheduling: python daemon;
Reporting: python script generating an HTML file.
![Page 15: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/15.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Code and license
MIT-licensed code
Available at https://cgit.freedesktop.org/ezbench/
Blocks
runner: bash-based, handles:
compilation and deployment of the component;setting up the environment (X, compositor);running the test.
env-dump.so: LD PRELOADed C library:
dump the environments and loaded libs;hook interesting calls (GLX, EGL, GL, X);dump metrics (RAPL, GPU temperature and power usage).
Report generation, enhancing and scheduling: python daemon;
Reporting: python script generating an HTML file.
![Page 16: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/16.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Code and license
MIT-licensed code
Available at https://cgit.freedesktop.org/ezbench/
Blocks
runner: bash-based, handles:
compilation and deployment of the component;setting up the environment (X, compositor);running the test.
env-dump.so: LD PRELOADed C library:
dump the environments and loaded libs;hook interesting calls (GLX, EGL, GL, X);dump metrics (RAPL, GPU temperature and power usage).
Report generation, enhancing and scheduling: python daemon;
Reporting: python script generating an HTML file.
![Page 17: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/17.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Code and license
MIT-licensed code
Available at https://cgit.freedesktop.org/ezbench/
Blocks
runner: bash-based, handles:
compilation and deployment of the component;setting up the environment (X, compositor);running the test.
env-dump.so: LD PRELOADed C library:
dump the environments and loaded libs;hook interesting calls (GLX, EGL, GL, X);dump metrics (RAPL, GPU temperature and power usage).
Report generation, enhancing and scheduling: python daemon;
Reporting: python script generating an HTML file.
![Page 18: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/18.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Features
Features
Supports:
Unit tests: Piglit, dEQP, IGT (WIP);Benchmarks: GPUTest, Unigine, GFX Bench (corporate), ...;Rendering: Apitrace.
Acquires environment information, for catching changes;
Analyses variance on data and reproduces changes;
Auto-bisecting on data, metrics are WIP.
Profiles
Mesa: No limitations;
xf86-video-intel: No limitations;
Linux: may require an external watchdog.
![Page 19: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/19.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Features
Features
Supports:
Unit tests: Piglit, dEQP, IGT (WIP);Benchmarks: GPUTest, Unigine, GFX Bench (corporate), ...;Rendering: Apitrace.
Acquires environment information, for catching changes;
Analyses variance on data and reproduces changes;
Auto-bisecting on data, metrics are WIP.
Profiles
Mesa: No limitations;
xf86-video-intel: No limitations;
Linux: may require an external watchdog.
![Page 20: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/20.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
Examples of variance
(a) Bad FPS distribution
(b) Good FPS distribution
Figure: Examples of variance
![Page 21: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/21.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
Examples of variance
(a) Bad FPS distribution (b) Good FPS distribution
Figure: Examples of variance
![Page 22: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/22.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Handling variance
Student-T test
Check if two data sets belong to the same normal distribution.
Source: http://serc.carleton.edu/introgeo/teachingwdata/Ttest.html
![Page 23: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/23.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Image comparaison
Overview
Contributed by Pekka Jylha-Ollila (Intel);
Comparaison done using RMSE and requires 3 steps.
Step 1: Comparing the output of 2 versions
RMSE != 0
Version A Version B
![Page 24: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/24.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Image comparaison
Overview
Contributed by Pekka Jylha-Ollila (Intel);
Comparaison done using RMSE and requires 3 steps.
Step 1: Comparing the output of 2 versions
RMSE != 0
Version A Version B
![Page 25: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/25.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Image comparaison
Step 2: Acquire more data and generate averages
Version A Version B
A
B
C
D
Averageimage
Step 3: Use the student-t test on the RMSEs
0 A B C D RMSE
![Page 26: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/26.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Image comparaison
Step 2: Acquire more data and generate averages
Version A Version B
A
B
C
D
Averageimage
Step 3: Use the student-t test on the RMSEs
0 A B C D RMSE
![Page 27: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/27.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Demo time
Demo 1: running loads with the simple runner
Listing tests;
Running gtkperf in different environments;
Showing the generated report;
Start compiling a new version of mesa.
Demo 2: Actual reports
Auto-bisected rendering change (5k commits, 7 months);
Running gtkperf in different environments;
Showing the generated report;
Start compiling a new version of mesa.
![Page 28: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/28.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Demo time
Demo 1: running loads with the simple runner
Listing tests;
Running gtkperf in different environments;
Showing the generated report;
Start compiling a new version of mesa.
Demo 2: Actual reports
Auto-bisected rendering change (5k commits, 7 months);
Running gtkperf in different environments;
Showing the generated report;
Start compiling a new version of mesa.
![Page 29: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/29.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Needed features for CI
Randomized testing
Not all tests can be run every day;
Tests should be added randomly (as time permits);
Support changing multiple components at the same time
EzBench needs to find the component that made the change;
It thus needs to group data per environment;
It needs to merge data from similar environments;
It needs to be able to re-deploy environments;
It needs to be able to recompile important components.
![Page 30: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/30.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench
EzBench: Needed features for CI
Randomized testing
Not all tests can be run every day;
Tests should be added randomly (as time permits);
Support changing multiple components at the same time
EzBench needs to find the component that made the change;
It thus needs to group data per environment;
It needs to merge data from similar environments;
It needs to be able to re-deploy environments;
It needs to be able to recompile important components.
![Page 31: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/31.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
Summary
1 Introduction
2 Graphics Continuous Integration
3 EzBench
4 Conclusion
![Page 32: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/32.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
Conclusion
Conclusion
Ezbench’s Goals
Automatically annotate a git tree with:
Unit test results;Power and performance results;Rendering changes.
Require as little human intervention as possible;
Provide reproducible results (environment).
EzBench tries to take care of the pitfalls of benchmarking
Environment dumping and diffing;
Reproduces results and tries to handle variance;
Is reactive to changes, and self-improving;
Handles most of the testing automatically.
![Page 33: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/33.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
Conclusion
Conclusion
Ezbench’s Goals
Automatically annotate a git tree with:
Unit test results;Power and performance results;Rendering changes.
Require as little human intervention as possible;
Provide reproducible results (environment).
EzBench tries to take care of the pitfalls of benchmarking
Environment dumping and diffing;
Reproduces results and tries to handle variance;
Is reactive to changes, and self-improving;
Handles most of the testing automatically.
![Page 34: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/34.jpg)
Questions?
![Page 35: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/35.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
Current features
Modular architecture (profiles, tests and user hooks);
Automates the acquisition of benchmark data;
Knows how long it is going to take;
Generates a report that is usable by developers;
Bisects performance changes automatically;
Provides python bindings to acquire data and parse reports;
Be crash-resistant by storing the expected goal and comparingit to the current state;
Collect the environment information and diff it;
Detect the variance and peformance changes;
Automatically schedule more work to improve the report.
![Page 36: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/36.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
Current features
Modular architecture (profiles, tests and user hooks);
Automates the acquisition of benchmark data;
Knows how long it is going to take;
Generates a report that is usable by developers;
Bisects performance changes automatically;
Provides python bindings to acquire data and parse reports;
Be crash-resistant by storing the expected goal and comparingit to the current state;
Collect the environment information and diff it;
Detect the variance and peformance changes;
Automatically schedule more work to improve the report.
![Page 37: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/37.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
Current features
Modular architecture (profiles, tests and user hooks);
Automates the acquisition of benchmark data;
Knows how long it is going to take;
Generates a report that is usable by developers;
Bisects performance changes automatically;
Provides python bindings to acquire data and parse reports;
Be crash-resistant by storing the expected goal and comparingit to the current state;
Collect the environment information and diff it;
Detect the variance and peformance changes;
Automatically schedule more work to improve the report.
![Page 38: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/38.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
Current features
Modular architecture (profiles, tests and user hooks);
Automates the acquisition of benchmark data;
Knows how long it is going to take;
Generates a report that is usable by developers;
Bisects performance changes automatically;
Provides python bindings to acquire data and parse reports;
Be crash-resistant by storing the expected goal and comparingit to the current state;
Collect the environment information and diff it;
Detect the variance and peformance changes;
Automatically schedule more work to improve the report.
![Page 39: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/39.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
Current features
Modular architecture (profiles, tests and user hooks);
Automates the acquisition of benchmark data;
Knows how long it is going to take;
Generates a report that is usable by developers;
Bisects performance changes automatically;
Provides python bindings to acquire data and parse reports;
Be crash-resistant by storing the expected goal and comparingit to the current state;
Collect the environment information and diff it;
Detect the variance and peformance changes;
Automatically schedule more work to improve the report.
![Page 40: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/40.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
Current features
Modular architecture (profiles, tests and user hooks);
Automates the acquisition of benchmark data;
Knows how long it is going to take;
Generates a report that is usable by developers;
Bisects performance changes automatically;
Provides python bindings to acquire data and parse reports;
Be crash-resistant by storing the expected goal and comparingit to the current state;
Collect the environment information and diff it;
Detect the variance and peformance changes;
Automatically schedule more work to improve the report.
![Page 41: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/41.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
Current features
Modular architecture (profiles, tests and user hooks);
Automates the acquisition of benchmark data;
Knows how long it is going to take;
Generates a report that is usable by developers;
Bisects performance changes automatically;
Provides python bindings to acquire data and parse reports;
Be crash-resistant by storing the expected goal and comparingit to the current state;
Collect the environment information and diff it;
Detect the variance and peformance changes;
Automatically schedule more work to improve the report.
![Page 42: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/42.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
Current features
Modular architecture (profiles, tests and user hooks);
Automates the acquisition of benchmark data;
Knows how long it is going to take;
Generates a report that is usable by developers;
Bisects performance changes automatically;
Provides python bindings to acquire data and parse reports;
Be crash-resistant by storing the expected goal and comparingit to the current state;
Collect the environment information and diff it;
Detect the variance and peformance changes;
Automatically schedule more work to improve the report.
![Page 43: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/43.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
Current features
Modular architecture (profiles, tests and user hooks);
Automates the acquisition of benchmark data;
Knows how long it is going to take;
Generates a report that is usable by developers;
Bisects performance changes automatically;
Provides python bindings to acquire data and parse reports;
Be crash-resistant by storing the expected goal and comparingit to the current state;
Collect the environment information and diff it;
Detect the variance and peformance changes;
Automatically schedule more work to improve the report.
![Page 44: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/44.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
Current features
Modular architecture (profiles, tests and user hooks);
Automates the acquisition of benchmark data;
Knows how long it is going to take;
Generates a report that is usable by developers;
Bisects performance changes automatically;
Provides python bindings to acquire data and parse reports;
Be crash-resistant by storing the expected goal and comparingit to the current state;
Collect the environment information and diff it;
Detect the variance and peformance changes;
Automatically schedule more work to improve the report.
![Page 45: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/45.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
TODO
Watchdog support;
Handle kernel boot failures (need the watchdog);
Add support for PTS as a backend;
Better integrate the build process;
React to HW events such as throttling;
Reset the environment to a previous state;
Integrate with patchwork to test patch series;
Support sending emails to the authors of changes.
![Page 46: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/46.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
TODO
Watchdog support;
Handle kernel boot failures (need the watchdog);
Add support for PTS as a backend;
Better integrate the build process;
React to HW events such as throttling;
Reset the environment to a previous state;
Integrate with patchwork to test patch series;
Support sending emails to the authors of changes.
![Page 47: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/47.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
TODO
Watchdog support;
Handle kernel boot failures (need the watchdog);
Add support for PTS as a backend;
Better integrate the build process;
React to HW events such as throttling;
Reset the environment to a previous state;
Integrate with patchwork to test patch series;
Support sending emails to the authors of changes.
![Page 48: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/48.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
TODO
Watchdog support;
Handle kernel boot failures (need the watchdog);
Add support for PTS as a backend;
Better integrate the build process;
React to HW events such as throttling;
Reset the environment to a previous state;
Integrate with patchwork to test patch series;
Support sending emails to the authors of changes.
![Page 49: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/49.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
TODO
Watchdog support;
Handle kernel boot failures (need the watchdog);
Add support for PTS as a backend;
Better integrate the build process;
React to HW events such as throttling;
Reset the environment to a previous state;
Integrate with patchwork to test patch series;
Support sending emails to the authors of changes.
![Page 50: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/50.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
TODO
Watchdog support;
Handle kernel boot failures (need the watchdog);
Add support for PTS as a backend;
Better integrate the build process;
React to HW events such as throttling;
Reset the environment to a previous state;
Integrate with patchwork to test patch series;
Support sending emails to the authors of changes.
![Page 51: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/51.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
TODO
Watchdog support;
Handle kernel boot failures (need the watchdog);
Add support for PTS as a backend;
Better integrate the build process;
React to HW events such as throttling;
Reset the environment to a previous state;
Integrate with patchwork to test patch series;
Support sending emails to the authors of changes.
![Page 52: EzBench, a tool to help you benchmark and bisect the ... · EzBench, a tool to help you benchmark and bisect the Graphics Stack’s performance Martin Peres Intel Open Source Technology](https://reader030.fdocuments.in/reader030/viewer/2022040605/5eab2e4a7cebb26ffc43ba0e/html5/thumbnails/52.jpg)
Introduction Graphics Continuous Integration EzBench Conclusion Backup slides
EzBench - Features
TODO
Watchdog support;
Handle kernel boot failures (need the watchdog);
Add support for PTS as a backend;
Better integrate the build process;
React to HW events such as throttling;
Reset the environment to a previous state;
Integrate with patchwork to test patch series;
Support sending emails to the authors of changes.