MySQL Empowers Mission Critical Financial Application ~ MySQL Case study in financial industry ~

MySQL Empowers Mission

Critical Financial Application ~ MySQL Case study in financial industry ~

Ryusuke KajiyamaMySQL Senior Evangelist, APACSun Microsystems / MySQL

2

Agenda

1. The project background 　

2. System requirements 　

3. System architecture 　

4. DBMS selection

5. Detailed design and benchmark

6. Post project review 　

3




4. DBMS selection



4

Project Overview

Target systemTarget system 　　Front office support application for asset management

companies (investment trusts/investment advisors, etc.)System for fund managerssoftware-as-a-service model application

The purposes of systematizationThe purposes of systematization 　　Timely front positions understanding and fund planningsTo enhance the ability to manage financial products in

accordanceTo integrate information from various sources, both in

and outside the companyTo focus on core operations as business efficiency

improves

5

The main functions of the system

Front-office position managementFront-office position managementFund and capital managementFund and capital managementContract managementContract managementUser managementUser management

Application functions 　

Data integration via messaging infrastructureData integration via messaging infrastructure　　 (used for linkage with systems of other companies)(used for linkage with systems of other companies)Client application development frameworkClient application development framework　　 (tailored for the individual needs of fund managers)(tailored for the individual needs of fund managers)

Infrastructure functions

6




4. DBMS selection



7

2. System requirements

Data VolumeData VolumeThe number of funds: 2,500+The number of balances reported by account/brand: 250,000+Transaction data volume: 1TB+

Function requirements Function requirements 　　Advanced user interface

Graphs and other GUI components as well as a high operability of drag and drop, and other functionsPublishing infrastructure for data to be reflected in real-time

Real-time/batch data linkage between systemsMessaging/file transfer

Basic requirementsBasic requirements 　　Small start & high scalability

In terms of costIn terms of system configuration

Consideration for multi-customers

8

2. System requirements

Fault-tolerance requirementsFault-tolerance requirements 　　Avioidng single failure point (Server/network duplexing)If server down: recovery time within 10 minutes

Operational requirementsOperational requirements 　　Weekdays: 6:00-20:00 (online)Other days: application batch/infrastructure batchAll maintenance operations are automated

Performance requirements (online)Performance requirements (online)Number of transactions per second: 200/secondResponse time: 　　　　　 3 seconds

9




4. DBMS selection



10

System configuration overview

Orchestration InfrastructureOrchestration InfrastructureOrchestration InfrastructureOrchestration Infrastructure

Workflow definitionWorkflow definition Flow controlFlow control

Presentation InfrastructurePresentation InfrastructurePresentation InfrastructurePresentation Infrastructure

Messaging infrastructureMessaging infrastructureMessaging infrastructureMessaging infrastructure

Service component groupService component group

Position manageme

nt

Position manageme

nt

Fund manageme

nt

Fund manageme

ntRisk

management

Risk manageme

nt

OMSOMS

Compliance

Compliance

Database service groupDatabase service group

BrandBrand

CharacteristicsCharacteristics MarketMarket

BenchmarkBenchmarkAccountAccount

CompanyCompany

RoutingRouting Database linkageDatabase linkagePublishingPublishing

Screen layout definition

Screen layout definition

Screen - data map

Screen - data map

Data set definitionData set

definition Data loadData load Data cacheData cache

------------

------------

------------

System evidence / system auditSystem evidence / system audit

11

Servers supporting the architecture are as follows:Usage Role

Presentation infrastructure

Aggregation server ・ Accepting client requests, and via the EMS server requesting the service group to deal with them

・ Delivering real data on the server side to clients

Messaging infrastructure

EMS server ・ Serving as a messaging infrastructure to enable smooth communications between servers which are loosely coupled

Service group (In this project, DB function is available in each service server.)

Position management / fund management server

・ Providing business logic relating to position management/fund management

・ Using same configuration for business logic server and DB server

Authentication server ・ Providing an authentication function

Other (key usages)

File linkage server ・ A server used for file linkage with external systems

External EMS server ・ A server used for real-time linkage with external systems

Shared disk ・ Storage for sharing business DB, authentication DB and others

Server configuration overview

12

Hardware configuration

This program exclusively employs IA-based servers.Usage CPU Memory HDD capacity

Presentation infrastructure

Aggregation server Dual core Xeon 5160 (3.00GHz) 4GB 146GB(RAID1)

Messaging infrastructure

EMS server Dual core Xeon 5160 (3.00GHz) 4GB 146GB(RAID1)

Service group (In this project, DB function is available in each service server.)

Position management / fund management server

Quad core Xeon X535 (2.66GHz) 8GB 146GB(RAID1)

Authentication server Dual core Xeon 5110 (1.60GHz) 4GB 146GB(RAID1)

Others

File linkage server Dual core Xeon 5110 (1.60GHz) 4GB 146GB(RAID1)

External EMS server Dual core Xeon 5160 (3.00GHz) 4GB 146GB(RAID1)

Shared disk The 2C2D (C: controller, D: disk enclosure) configuration with a capacity of 146GB×16 HDDs. Possible to increase capacity up to 25TB (146GB×28 HDDs). By adding enclosures, can be up to 50TB with 2C4D (56 HDDs)

1238GB (RAID 5)

SAN switch - - -

13

Software configuration

Messaging infrastructureMessaging infrastructure

Presentation infrastructurePresentation infrastructure

RedHat Linux ES 4.0

Apache2.0.54

JBoss AP Server 4.0.5

JDK 5.0

EMSClient API

Strus In-company F/W

Application

Aggregation server

RedHat Linux ES 4.0

Cluster middleware

EMS serverExternal EMS server

TIBCO EMS 4.4.0

Scale-out Scale-out

configurationconfiguration

Scale-up + HA Scale-up + HA


14

Software configuration

Service group infrastructureService group infrastructure

Cluster middleware

MySQL5.0.40

JBoss AP Server 4.0.5

JDK 5.0

EMSClient API

In-company F/W

Application

Position/fund management serverAuthentication server

RedHat Linux ES 4.0

Scale-up + HA Scale-up + HA


15

Screen shots

User registration screenUser registration screen

16

Screen shots

Front position management screenFront position management screen

17

Screen shots

Transaction progress management screenTransaction progress management screen

18




4. DBMS selection



19

First thoughts

Commercial product ? 　

MySQL ? 　

20

Thoughts on “commercial product”

Points: consPoints: consCost (1)

Software license fees/software maintenance fees are required.Cost (2)

Higher spec hardware may be needed to run heavy DBMS. Possible higher hardware costs

Cost (3)While existing know-how can be applied, designing and implementation tend to be time consuming

Points: prosPoints: prosBoth in and outside the company it has produced proven results:

Stable product qualityMany engineers with know-how are in SIerMore know-how of application and system design in SIer

The product architecture can accommodate a large-scale project

21

Thoughts on MySQL

Points: consPoints: cons Proven results are relatively limited in business systems

Main proven results are in reference systems.No proven results with NRI’s internal use in update systems.It is uncertain whether the requirements and practical use conditions of this project would be met.

Limited design know-howNot much experience in HA designDifficult to find technical experts in other div of NRI

Points: prosPoints: prosExpectations from simple architecture of MySQL:

Easier in designing and implementations phaseFewer errors in designing and implementationsLess time for designing and implementations

Low costsNo software license fee; only an inexpensive maintenance feeEasy designing means SE cost reductions can be expected.

22

Final decision “MySQL”

Reasons behind adopting MySQLReasons behind adopting MySQL

MySQL meets basic requirements of this project.MySQL enables a small start both in investment costs and

systems scale.MySQL is flexible to expand both in terms of investment costs

and scale. * Cost advantage is outstanding, including preparing dev

environment.Rapid system design and implementation

Ease of expansion meets requirements in this multi-user model system.

Simplicity of MySQL’s implementation process is impressive.Advanced technical support from NRI’s OSS support team

Quick trouble shooting can be expected.Many experienced engineers went through number of tough

projectsChallenging something new!!

Using only proven technology will rust knowledge and solution capability

23

Comments in executive review

Response measures based on the suggestionsResponse measures based on the suggestionsFor (1)

The lack of design know-how was complemented by support from NRI’s OSS Support team and MySQL. Followed by a DBMS-related professional review

For (2)A simple prototype application was developed to confirm that at least there was no problem with function/performance.

For (3)As a backup plan, in the case of the system being found infeasible, replace with a commercial product.

Suggestions in internal design review meetingSuggestions in internal design review meeting

(1) Review by MySQL professional for system design is required, because team did not have much experience of using MySQL in mission critical and high transactional environment.

(2) Feasibility test should be conducted using real environment.(3) If the system malfunctioned, backup plans should be

discussed for worst case scenarios.

24




4. DBMS selection

5. Detailed design and benchmark 　


25

System design/operation design

Adopting the InnoDB storage engine to handle transaction dataSingle MySQL server instance to be shared for multiple companies, not running multiple instances for each companies on the single server

To reduce daily operational tasks, use partitions (backup, monitoring)Minimizing downtime through server

Storing data/index in the common tablespace file

Tablespace(single file viewed from the OS)

Tablespace(single file viewed from the OS)

DB for Company A

DB for Company B

DB for Company N

MySQL

Multi-users supportMulti-users support

26

Backup with “Snapshot”

Adopting “Snapshot” feature of the storage device【 Processing steps 】　 (1) Storage management server issues a Snapshot command.　　 A new tablespace is cloned in storage device.

TablespaceTablespace

DB for Company A

Shared storage

DB for Company B

TablespaceTablespaceSnapshotSnapshot

Position/fund management

server

Storage management

serverNFS media server

Snapshot commandSnapshot commandUnmountUnmount

Snapshot operation performedSnapshot operation performed

Backup operations (Step 1)Backup operations (Step 1)

27


(2) When Snapshot operation is done, the position/fund management server mounts the shared storage, in preparation for making service available; in addition, in preparation for backup, the NFS media server mounts the copy area.

Shared storage

Copied tablespaceCopied tablespace

Position/fund management

server

Storage management

serverNFS media server

MountMount MountMount

TablespaceTablespace

DB for Company A

DB for Company B

Available


28


(3)The backup server issues a backup command to the NFS media server and executes “LAN-free backup” to tape library is performed.

Shared storage

NFS media server Backup server

MountMount

Copied tablespaceCopied tablespace

Tape library

Backup command


29

Infrastructure test

The basic features as a DBMS meets the conditions of practical use.New features of MySQL 5.0 was not used in this project. e.g. view, stored procedure and trigger

Avoiding the use of brand-new features to reduce risks: the application can be implemented without such featuresPreparing for performance tuning: View will not be usedFacilitating operations and maintenance: Avoiding using stored procedure and trigger

After series of tests, stability of products, ease of restore + recovery and simplicity of restore + recovery process met our requirements.TIPS: When configuring HA cluster, do NOT use `mysqld_safe` shell script to start the server.　 -> In case of failure , both mysqld_safe and clustering tool will try to restart MySQL server and won’t make proper fail-over or cause unstable condition.

Evaluation of featuresEvaluation of features

30

Performance tests

Doing benchmark tests to find out basic performance of MySQL server.　 Tests included Import, Export and Create Index performance.Using production environment to find “real” performance.

Evaluation of performanceEvaluation of performance

Tests Purposes

Select (FULL Scan)

- To obtain basic performance values for full-table scan in MySQL

- To find the difference of execution time and resources usage on:

data size; concurrency; existence of cache in InnoDB buffer pool.

Select (Index Scan)

- To obtain basic performance values in index search in MySQL



Insert - To obtain basic performance values in data insert in MySQL



31

Performance tests

Data size: 1GB, 5GBConcurrency: 1, 4, 8, and 16Rebooting OS before each test caseIssuing two sets of queries in a row per concurrency

1st : Right after booting the MySQL server2nd: After a 60-second interval following the first query

The SQL statement issued is shown below. Full table scan is forced by the HINT

SELECT SUM(gid) FROM http_auth IGNORE INDEX (primary key) ;

To assess the execution time, the date command was executed before and after processing and the difference was measured.

Select (FULL Scan) performanceSelect (FULL Scan) performance

32

Performance tests

Data transfer rate of storage : 14.5[MB/sec].With 5GB, data transfer rate was accelerated to 59.7[MB/sec]

with smart storage controller.No difference was found relating to concurrency, probably

because of one-table access. The effectiveness of the cache mechanism was confirmed.

Select (FULL Scan) performance : resultsSelect (FULL Scan) performance : resultsData size Concurrenc

yTrial Execution

time (hh:mm:dd)

1GB 1 1st 0:01:372nd 0:00:00

4 1st 0:01:372nd 0:00:00

8 1st 0:01:362nd 0:00:00

16 1st 0:01:392nd 0:00:00

33

Performance test

CPU utilization (1GB, 1 concurrency pattern)In the 1st trial, the process proved to be I/O bound. In the 2nd trial, CPU utilization proved to be close to zero.

The 2nd trialThe 1st trial

Select (FULL Scan) performance : resultsSelect (FULL Scan) performance : results

34

Performance test

Available memory size (1GB, 1 concurrency pattern)In the 1st trial, approximately 1.4 GB memory was used, probably because the retrieval data was cached.In the 2nd trial, no change was found.

The 2nd trial

The 1st trial


35

Performance test

Disk read size: 1GB, 1 concurrency patternIn the 1st trial, a disk read was found to have occurred with an

average of approx 15[MB/s].In the 2nd trial, no disk read was found.

The 2nd trial

The 1st trial


36

Performance test

Data size: 1GB, 5GBConcurrency: 1, 4, 8, and 16Rebooting OS before each test caseIssuing two sets of queries in a row per concurrency

1st : Right after booting the MySQL server2nd: After a 60-second interval following the first query

The SQL statement issued is shown below. Executing the program by specifying a unique and random number for the WHERE statement

SELECT * FROM http_auth WHERE uid ＝ ‘ xxx’ ;


Select (Index Scan) performanceSelect (Index Scan) performance

37

Performance test

Retrieval performance is processed at the high speed of 10[ms/sec]. A 5GB item of data is processed at a speed of 20-30 [ms/sec].An increase in concurrency results in almost no decrease in retrieval speeds.It was confirmed that the cache mechanism functions effectively.

Select (Index Scan) performance : resultsSelect (Index Scan) performance : resultsData size

Concurrency Trial Execution time (hh:mm:dd)

Ave. throughputs (TPS)

Ave. response time (ms/process)

1GB 1 1st 0:00:01 100 10

2nd 0:00:00 - -

4 1st 0:00:01 400 10

2nd 0:00:00 - -

8 1st 0:00:01 800 10

2nd 0:00:00 - -

16 1st 0:00:01 1600 10

2nd 0:00:00 - -

38

Performance test

CPU utilization (1GB, 1 concurrency pattern)In the 1st trial, it was found that only a minimal amount of resources was required to complete the process.In the 2nd trial, CPU utilization proved to be close to zero.

The 2nd Trial

The 1st Trial

Select (Index Scan) performance : resultsSelect (Index Scan) performance : results

39

Performance test

Available memory size (1GB, 1 concurrency pattern)In the 1st trial, a memory of approx. 11MB was utilized. Index and retrieved data are considered to have been cached.In the 2nd trial, no reduction in free memory and no increase in cache was found.

The 2nd trial

The 1st trial


40

Performance test

Disk read size (1GB, 1 concurrency pattern)In the 1st trial, a disk read was found to have occurred (approx. 13[KB/s] on average).In the 2nd trial, no disk read was found.

The 2nd trialThe 1st trial


41

Performance test

Data size: 1GB, 5GBConcurrency: 1, 5, and 10Rebooting OS before each test caseInserting one record = 1KBIn the case of 5 or 10 multiplexes, a specified data size is to be built in throughout the entire number of processes operated in parallel.

Example) in the case of a 5 GB data size　 1 concurency ： 1 thread (5GB/thread) Insert　 5 concurency ： 5 threads (1GB/thread) Inserts　 10 concurency ： 10 threads (500MB/thread) Inserts


Inserts performanceInserts performance

42

Performance test

Insert performance of 450-480[processes/second] with single thread.In 5GB test case, 500+ [processes/second]

Particularly noteworthy was that regardless of multiplex or DB size, there was no change in throughputs. Although in this test, commit processing was performed for each insert, the performance in this method is considered to be reaching the processing limit.

Inserts performance : resultsInserts performance : resultsData size

Concurrency Execution time (sec.)

Throughputs/thread

Ave. throughputs per threads (TPS)

Total throughputs (processes/second)

1GB 1 2212 452.1 452.1 452.1

5 2089 96.0 95.9 478.796.095.895.895.8

10 2095 47.8 47.8 477.247.847.847.847.847.847.747.747.747.7

43

Performance test

CPU utilization (1GB, 1 concurrency pattern)An average CPU utilization is 13% with a maximum utilization of

37%.It was confirmed that programmatically, a delayed write occurred

even after the completion of the Insert process. It can be assumed that the MySQL system performs commit processing.

Delayed write

Inserts performance : resultsInserts performance : results

44

Performance test

Available memory size: 1GB, 1 concurrency patternIt is considered that through the Insert processing, the data for Insert was cached.

A reduction of approx. 1GB due to

caching


45

Performance test

Disk read/write size: 1GB, 1 concurrency patternFew disk reads occurred. An average disk read speed was 14[MB/s] with maximum 17[MB/s]. The bottleneck can be caused by the disk I/O to the Binlog.It was confirmed that a delayed write occurred in disk I/O.

Actual DB is updated in bulk?


46




4. DBMS selection



47

Benefits of using MySQL

No license fees for MySQLNo license fees for MySQLMinimum investment: Low initial investment costs in this service

delivery business model is important In most of projects, license fee of DBMS is relatively large

portion in hardware and software costs.No additional fees: No fee for options, No upliftLower cost of expansion: No additional fee for more CPUs,

memory or clients

Stable and high product qualityStable and high product qualityMySQL showed high data processing performance not only in data

reference application, but also in writing application. With no product deficiency detected during development and

production phase.Easy to confirm “real” performance in early stage, because MySQL

does not require high spec servers and testing on dev server shows similar results on production server

48

Conclusion

There were no technical problems in functions of MySQLStable and high product qualityHigh quality technical support from Sun

Now we are seeing more use of MySQL in multiple industryincluding Telco, Finance, Manufacturing, Healthcare and Government.

MySQL is now better than commercial products in both quality and performance.

With the accumulation of best-practice and know-how, better professional services and technical support are available.

“You can find many systems which you don’t haveto pay expensive “DBMS Tax” and you can get a lot of

benefits with understanding MySQL more.”

“Do feasibility tests before jump into project with MySQL.”

MySQL is more than cheap & easy-to-use!MySQL is more than cheap & easy-to-use!

Time to use MySQL in mission criticalTime to use MySQL in mission critical

MySQL Empowers Mission

Critical Financial Application ~ MySQL Case study in financial industry ~

Ryusuke [email protected] Senior Evangelist, APACSun Microsystems / MySQL

MySQL Empowers Mission Critical Financial Application ~ MySQL Case study in financial industry ~

Documents

Transcript of MySQL Empowers Mission Critical Financial Application ~ MySQL Case study in financial industry ~