EXTENT-2015: Big Button 2.0

Big Button 2.0Big Button 2.0

Quality Assurance & Related Software Development for Innovative Trading SystemsTel: +7 495 640 2460, +1 415 830 38 49www.exactpro.com

A new approach to automated regressionA new approach to automated regression

Alexey Zverev, CEO ExactproAlexey Zverev, CEO Exactpro11 November 201511 November 2015

http://exactpro.com/

2

Goal: to execute a test regression and make it:

•as extensive as possible (A)•as efficient as possible (B)

How do people usually achieve that?

•They develop software which performs the tests and ensure that it requires an adequate amount of maintenance when the software under test changes

Test Automation

Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com

http://www.exactpro.com/

3 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com

Test Automation Classics

1. Design Test Library 2. Develop Test Harness• Script execution framework• GUI injectors• DPI injectors• Upstream / Downstream

systems stubs

3. Develop Test Case 4. Develop Many Test Cases

5. Team Of QA Engineers Runs Regression

Login ALogin BA send Order (…)B Send Order (…)A verify Ask (…)B verify Trade (…)A verify Trade (…)



The Job is not finished

The next task is to make test execution more efficient:

•We need a “Big Button”, i.e. software allowing executing a large number of automated test scripts with minimum human attendance

•When successfully implemented it can reduce up to 80% effort (in man days) spent on regression


Big Button 1.0 concept

1. We have many automated test cases

2. Prepare multiple test environment

3. Prepare reference data 4. Redesign test library multiple threads

5. Develop orchestrating test harness


6. Run regression

- SHELL- CRON- ...

Login A Login B Login C


Limitations of Big Button 1.0

• Complete control over test environments is required• Automated GUI tests are not stable• Dependent test cases are not stable• Independent test cases require overhead• A lot of copy/paste code• Executing tests consecutively, one by one, takes significant time• Big Button conditions are not similar to production (discussed later)


Alternative approach - “Load with Trades”

• This was a success• We have learned that it is called HiVAT (High volume automated

testing)• Unfortunately it is still not a regression


Meanwhile - Robots!

• The Trading Industry lives and breathes High Transaction Volumes• High Volume is created by Trading Algos and we have created a test

tool called Minirobots to simulate that• The tool good for simulation of production conditions• Classical test scenarios are still not validated


Big Button 2.0

• A set of Minirobots Algorithms allowing to perform a complete test regression of a trading system within a few hours

• All Robots work concurrently against the test environment• Robots are independent from each other, and their actions are based on

current conditions on the order books• Robots use an abstract Test Library and their main goal is to cover all

test cases within the library


How it will work

1. Random load “Pace Maker” 2. Variety of Passive Liquidity

3. “Agressors” try to interact with the market to increase number of test cases happening in the environment

4. Analyze situation in retrospective Make sure that what happened is correct Tick checkboxes against test that was actually executed

Updated test

library


Example

1. Read book 2. Search for appropriate test case

3. Try to test the scenario

4. Wait for full trade or auto cancel

5. Iterate through actual event orders:

6. Conclusion

•We hit only 2 orders•Another robot cancelled 3rd order•Ticking box against this test case

IOC hits 3 passive orders at exact price

PFILL PFILL

CANCEL ASK


Discussion

+ It can work really fast+ Maintenance is limited (with adequate robot design)+ A challenging and motivating task for QA- May be too complex to implement- It is not clear how to incorporate DLC related scenarios and manual

tests


Q&A

Thank You!

EXTENT-2015: Big Button 2.0

Technology

Transcript of EXTENT-2015: Big Button 2.0