Computer Measurement Group, India 1 1 ARCHITECTURE AND DESIGN FOR PERFORMANCE OF A LARGE EUROPEAN...

Computer Measurement Group, India 1Computer Measurement Group, India 1

www.cmgindia.org

ARCHITECTURE AND DESIGN FOR PERFORMANCE OF A LARGE EUROPEAN

BANK PAYMENT SYSTEM

R Harikumar, TCSNityan Gulati, TCS

http://www.cmgindia.org/

Computer Measurement Group, India 2

Outline

• System Context and Architecture• Design for performance - parameters– Tuning Web Page response– Tuning Straight through process throughput– Batch process optimization– Tuning Database Parameters

• Key Benefits Realized• Key Lessons Learnt

Computer Measurement Group, India 3Computer Measurement Group, India

Performance Engineering: Typical Challenges

• Large system characteristics – huge volumes, complex eco system, several unknowns

• Understanding application user patterns

• Varying Load conditions and increasing volumes

• Understanding the business and its growth

• Performance planned only as a testing phase !!


System Context and Architecture

• It is a Corporate banking payment system

• Huge daily volumes. Two million txs / day, peak of 300k per hour

• 2000 branch users across Europe

• End of day completion under 2 hours

• About 20 upstream systems


Key Design Considerations

• Web page response o Search screen page response under 2-3 secondso Blank Searcheso Graceful stopping of long running querieso Handling network latency

• STP throughouto Non uniformity in the sizes of file received (~1 to 100KB -> Small, 100 to

1000KB -> medium, 2 to 60 MB -> Large files)o Non uniformity in the type of transaction in a fileo STP design

• Batch jobso Huge volume of transactions o Parallel processingo Managing Concurrency and hot spots


Web page response tuning – Key considerations#1: Gracefully stopping the long running SQLs that can impact the system stability.

Configure CPU_PER_CALL threshold post which the query is terminated with ORA-02393 error. Configure this for a separate oracle user and exercise control on when it needs to be invoked

#2: Network latency: Avoid HTTP 304 calls from the browser. It is a quick change and provides significant gains in browser performance. HTTPWATCH and DYNATRACE tools provide great insight into the page performance

Configuration required in HTTPD to prevent 304 calls Configuration to reload static content during software upgrade


#3:

• Control on the search performance on the most frequently used screens

• Identify the most used search combinations along with business and tune them using indexes for best performance

• For other non popular combinations, limit the data display to a weeks data by appending the date filter automatically

• Blank Search requests: it means retrieving and sorting millions of rows and then presenting possibly the first few rows

• Guide the query plan with a hint such that an index with column order matching the sort order is picked .

• Additionally restrict the size of the result set with a specified period. E.g. one week of data

• Use ROWNUM to restrict the size of the required result set

Web page response tuning – Key considerations


Straight Through Processing - scalable designo 90% of the payments received

are file based

o No uniformity in the sizes of file received

o Non homogeneity in the type of transaction in a file

o ~1 to 100 -> Small, 100 to 1000KB -> medium, 2 to 60 MB -> Large files

The above design helped us to process 300,000 transactions per hour

• Payment systems are pre-dominantly file based

• Design of the system has to be extremely flexible and robust

• The component needs to be configurable for varying load conditions


Batch performance considerationsBatch processing Characteristics• Large volume of transactions to be processed at the EOD as batches leading to a

surge of parallel activity• Large variation in Transaction to Account ratio resulting in deadlocks

file with payment

txns

Modulus (Account)

Txn 1 - Account 1Txn 2 - Account 2Txn 3 - Account 3Txn 4 - Account 1Txn 5 - Account 2Txn 6 - Account 3Txn 7 - Account 1Txn 8 - Account 2Txn 9 - Account 1

Thread 1 ( Account 1 > 4 Txns)

Thread 2 ( Account 2 3 Txns)

Thread 3 ( Account 3 2 Txns)

Allocation of transaction to threads

Distinct Accounts are processed by separate threads avoiding contention

#1: Proper allocation of transaction to threads to avoid contention to ensure we have minimal contention amongst parallel paths of execution


Batch performance considerations#2:Group the components into logically related clusters and execute several groups in parallel

Parallelism in batch profile

Group the jobs logically into clusters and run them in parallel to squeeze the elapsed time

#3:Using appropriate collection objects in the code. E.g. concurrent hash map reduces the possibility of transactions entering into dead lock situations.#4:Apply sequence caching. It reduces the overhead of ID generation and leads to improvement of performance .


Database considerationsThe following are the key tuning considerations that helped to tune the database performance

Sno Tuning parameter Remarks

1 Reduce the quantum of Redo-logging and commit less frequently

• Avoid repeated updates in the same commit cycle• Place the redo logs on faster storage and separating the log

members / groups on different disks [PERF003]

2 Connection Pool library • Use specialized connection pool libraries e.g. BoneCP• Reduces application wait times for connection

3 Bulking of inserts / updates

• Use Add batch and execute Batch JDBC methods• Saves on network round trips

4 Controlling the JDBC Fetch Size

The fetch size selected should be optimal to balance the network traffic and the available heap available on the client side

5 Table Partitioning It reduces the hot spots such as index last block, where we have sequentially increasing keys for the index.

6 Using prepared statements having bind variables

Promotes statement caching, reduces memory footprint in the shared pool and reduces the parsing overhead.


Key Achievements

The picture below shows the response times of the most frequently used screensSLA : under 3 seconds


Key AchievementsSLA was to process around 300k transactions under 60 minutes

The Graph below shows payment files and the time taken to process 300k transactions

Sno Type

Time Taken ( minutes)

No of TX’s

1 Debit card Txs 59 54615

2Accounting entries-1 44 54615

3 Payments-1 17 25040

4 Payments-2 14 25040

5 Payments-3 9 25040

6 Trade settlements 58 59250

7 MT202 44 2400

8 MT103 37 3060

9 SEPA messages 60 5130

10 DD- remittances 12 35550

11Accounting entries-2 22 10260

TOTAL 300,000

Debit card Txs

Accounting entries-1

Payments-1

Payments-2

Payments-3

Trade settlements

MT202

MT103

SEPA messages

DD- remittances

Accounting entries-2

0 10 20 30 40 50 60 70

Transaction Type versus Time Taken in Minutes


Lessons Learnt• Performance is a key component to be planned and worked upon right from the

requirement stage till the performance testing stage. It needs to be monitored in production through appropriate KPIs

• Keep the of SQLs fired to the minimum per unit of output. This should be controlled during the development phase

• IE 8 / IE9 give significantly better results as compared to IE 7 on account of efficient rendering APIs in these versions

• Eliminating HTTP 304 calls from the browser provides quick gains. It can be done through a simple configuration in web server

• Application performance should always be measured considering the actual volume of historical data that resides in the database.

• Caching features need to be exploited to the maximum across all layers in all high volume applications


Q & A


Appendix


HTTP 304 calls- TCS.COM BACK


NETWORK

PERFORMANCE

DYNATRACE TOOL

BACK


Batch Profile - parallelism

10 20 15 15 15 10 5

Timeline in minutes, SLA :120 minutes

BACK

Computer Measurement Group, India 1 1 ARCHITECTURE AND DESIGN FOR PERFORMANCE OF A LARGE EUROPEAN...

Documents

Transcript of Computer Measurement Group, India 1 1 ARCHITECTURE AND DESIGN FOR PERFORMANCE OF A LARGE EUROPEAN...