Non-Functional Testing at London Stock Exchange

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 1

Non-Functional Testing at London Stock Exchange29 November 2016Iosif Itkin, Exactpro CEO & Co-FounderLondon Stock Exchange Group

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 2460, +1 415 830 38 49www.exactpro.com

http://www.exactpro.com/

http://exactpro.com/


London Stock Exchange Group

Exactpro - Open Access Software Testing

Testing Execution Venues

Efficient Load Generation

Latency Measurement

Non-Functional Testing Principles

Other Ways to Produce Load

Approach to Test Automation

Failover Testing Approach

Passive Software Testing

Research in Tools and Methods of Program Analysis

ContentsContents



London Stock Exchange Group

• A leading diversified international exchange and infrastructure group with assets across the entire exchange value chain

• The Group can trace its history back to 1698• Headquartered in the United Kingdom, with significant global operations, the Group employs approximately 3,500

people• Post trade and risk management services are a significant part of the Group’s business operations• Through FTSE Russell, the Group is a global leader in financial indexing, benchmarking and analytic services with

approximately $10 trillion benchmarked to its indexes



Open Access Software Testing

Exactpro is:

• A specialist firm operating an open access model with focus on functional and operational testing of securities data distribution, trading systems, risk management, market surveillance and post-trade infrastructures.

• A UK company with operations in the US and four QA & software development centres in Russia

• Part of London Stock Exchange Group as of May 29, 2015

• Incorporated in 2009 with 10 people, our company has experienced significant growth as satisfied clients require

more services; now employing over 420 specialists

Clients Worldwide:



Open Access Software Testing

Toronto

New YorkTysons Corner

Montreal

Portland

Seattle

Dubai

Colombo

Johannesburg

Sydney

Singapore

Hong Kong

Moscow

Oslo

ParisMilan

Athens

Vienna

London Amsterdam

A specialist firm operating an open access model with focus on functional and operational testing for global financial markets worldwide.

A UK company with operations in the US and four QA & software development centres in Russia

Part of LSEG Since May 29, 2015

Incorporated in 2009 with 10 people, our company has experienced significant growth as satisfied clients require more services;

Now employing over 400 specialists



Types of Trading Systems

Algo TradingHFT

Brokerageand IDB

Quality Assurance:test planning and

managementProcess audit and test

coverage analysis

Intelligent Management

of Large Data Sets

Test data management

Protocol level testing using FIX/FAST, SOAP, HTTP, ITCH, SWIFT, MQ, SQL, proprietary

binary and text baseddata formats, etc.

Automated regressiontesting

Gathering requirementsand test scenario creation

(human, message & reporting interfaces)

Creating and productizing state-of-the-arttest harnesses

Test automation

Intelligent functional andexploratory testing

Latency & capacitytesting

Market Infrastructures



Execution Venues Testing

Our experience in market infrastructures testing embraces all components and operational processes. Unique tools have been used to test a wide variety of vendor platforms. A holistic integrated automated test solution for trading, clearing, settlement and other systems



Performance Testing Cost Structure

System Under Test

ProductionConnectivity

Test Tools

Infrastructure: ≈ 10 - 100 servers

Infrastructure: thousands of servers

Infrastructure: 1 - 4 servers



The Idea Behind Efficient Load Generation



Latency Measurement



Latency Distribution



Question Everything

Do Not Trust:• Test Tools

• Average Metrics and Percentiles

• That Load Shape is Right

• System Requirements

• Anything



Keep Calm

Have No Fear:• Annihilate the System Under Test

• Repeat the Same Tests Many Times

• Load Levels that Exceed Presumed Production

• Make no Assumptions About

Extrapolating Test Results into

More Scalable Environments



Be Confident

Do Not Beg For:• Test Tools

• Confirmation that PROD is Configured as TEST

• Max Load Figure

• Load Scenarios for Existing Production System

• Constant Support



Two Load Generator Types

1.

2. Simulate real trading participants

Adopt to changing market conditions

Check data consistency from member’s point of view



Achieving Diverse and Relevant Load

1. Random load “Pace Maker” 2. Variety of Passive Liquidity

3. “Aggressors” try to interact with the market to increase the number of test cases happening in the environment

4. Analyze the situation in retrospect. Make sure that what happened is correct. Tick checkboxes against tests that were actually executed

Updated test

library



Types of Non-Functional Tests



Failover Tests and Catching In-Flight Transactions



Use Passive Testing Tools to Collect Audit Trails



Questions & Answers

• Competing conference papers go through a review by a number of leading software development and analysis experts and scientists

• The best papers are recommended for their inclusion in Russia’s leading professional and scientific publications

The Fourth TMPA Conference Find out more: www.tmpaconf.org

Renowned experts in program engineering speaking on:

• software test automation• static program analysis• verification• dynamic methods of program analysis• analysis of parallel and distributed systems• testing and analysis of high-availability systems• analysis of hardware and software systems• methods of building quality software• tools for software analysis, testing and verification


http://www.tmpaconf.org/


Questions & Answers

https://ru.linkedin.com/in/iosifitkinThe seventh EXTENT conference will take place in London, UK.

Find out more: www.extentconf.com

– LSEG Technology and Quality Assurance– Risk controls and FPGA– Market Surveillance systems– Blockchain and trading technology trends– Artificial intelligence


https://ru.linkedin.com/in/iosifitkin

http://www.extentconf.com/

Non-Functional Testing at London Stock Exchange

Technology

Transcript of Non-Functional Testing at London Stock Exchange