Post on 22-Jan-2017
Big Button 2.0Big Button 2.0
Quality Assurance & Related Software Development for Innovative Trading SystemsTel: +7 495 640 2460, +1 415 830 38 49www.exactpro.com
A new approach to automated regressionA new approach to automated regression
Alexey Zverev, CEO ExactproAlexey Zverev, CEO Exactpro11 November 201511 November 2015
2
Goal: to execute a test regression and make it:
•as extensive as possible (A)•as efficient as possible (B)
How do people usually achieve that?
•They develop software which performs the tests and ensure that it requires an adequate amount of maintenance when the software under test changes
Test Automation
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com
3 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com
Test Automation Classics
1. Design Test Library 2. Develop Test Harness• Script execution framework• GUI injectors• DPI injectors• Upstream / Downstream
systems stubs
3. Develop Test Case 4. Develop Many Test Cases
5. Team Of QA Engineers Runs Regression
Login ALogin BA send Order (…)B Send Order (…)A verify Ask (…)B verify Trade (…)A verify Trade (…)
Login ALogin BA send Order (…)B Send Order (…)A verify Ask (…)B verify Trade (…)A verify Trade (…)
4 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com
The Job is not finished
The next task is to make test execution more efficient:
•We need a “Big Button”, i.e. software allowing executing a large number of automated test scripts with minimum human attendance
•When successfully implemented it can reduce up to 80% effort (in man days) spent on regression
5 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com
Big Button 1.0 concept
1. We have many automated test cases
2. Prepare multiple test environment
3. Prepare reference data 4. Redesign test library multiple threads
5. Develop orchestrating test harness
Login ALogin BA send Order (…)B Send Order (…)A verify Ask (…)B verify Trade (…)A verify Trade (…)
6. Run regression
- SHELL- CRON- ...
Login A Login B Login C
6 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com
Limitations of Big Button 1.0
• Complete control over test environments is required• Automated GUI tests are not stable• Dependent test cases are not stable• Independent test cases require overhead• A lot of copy/paste code• Executing tests consecutively, one by one, takes significant time• Big Button conditions are not similar to production (discussed later)
7 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com
Alternative approach - “Load with Trades”
• This was a success• We have learned that it is called HiVAT (High volume automated
testing)• Unfortunately it is still not a regression
8 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com
Meanwhile - Robots!
• The Trading Industry lives and breathes High Transaction Volumes• High Volume is created by Trading Algos and we have created a test
tool called Minirobots to simulate that• The tool good for simulation of production conditions• Classical test scenarios are still not validated
9 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com
Big Button 2.0
• A set of Minirobots Algorithms allowing to perform a complete test regression of a trading system within a few hours
• All Robots work concurrently against the test environment• Robots are independent from each other, and their actions are based on
current conditions on the order books• Robots use an abstract Test Library and their main goal is to cover all
test cases within the library
10 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com
How it will work
1. Random load “Pace Maker” 2. Variety of Passive Liquidity
3. “Agressors” try to interact with the market to increase number of test cases happening in the environment
4. Analyze situation in retrospective Make sure that what happened is correct Tick checkboxes against test that was actually executed
Updated test
library
11 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com
Example
1. Read book 2. Search for appropriate test case
3. Try to test the scenario
4. Wait for full trade or auto cancel
5. Iterate through actual event orders:
6. Conclusion
•We hit only 2 orders•Another robot cancelled 3rd order•Ticking box against this test case
IOC hits 3 passive orders at exact price
PFILL PFILL
CANCEL ASK
12 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com
Discussion
+ It can work really fast+ Maintenance is limited (with adequate robot design)+ A challenging and motivating task for QA- May be too complex to implement- It is not clear how to incorporate DLC related scenarios and manual
tests
13 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com
Q&A
Thank You!