IBM Technical BriefFILE/Async_CF_Lock_Duplexing_withSAP.pdf · Keep in mind that the SAP Day...

© Copyright IBM Corp. Async CF Lock Duplexing with SAP Performance Report of 35

IBM Technical Brief

Asynchronous CF Lock Duplexing with SAP®

Performance Report from IBM z Systems®

Version 1.0

June 27, 2017

Authors: Brenda Beane Seewah Chan Dr. Paul Lekkas Document Owner: Brenda Beane IBM SAP on z Systems Performance Version: 1.00 Date: June 27, 2017 Filename: Async_CF_Lock_Duplexing_withSAP.pdf


Table of Contents

Table of Contents..................................................................................................................................... 2 Tables ...................................................................................................................................................... 3 Figures ..................................................................................................................................................... 3 Disclaimers .............................................................................................................................................. 3 Trademarks .............................................................................................................................................. 4 Acknowledgements .................................................................................................................................. 4 Feedback ................................................................................................................................................. 4 1.0 Introduction ........................................................................................................................................ 5 2.0 Executive Summary ........................................................................................................................... 7 3.0 Workload Description ......................................................................................................................... 8 4.0 Test Environment ............................................................................................................................... 9

4.1 Hardware ..................................................................................................................................... 10 4.2 Software ....................................................................................................................................... 11 4.3 Data Sharing Configuration .......................................................................................................... 12

4.3.1 Duplexed CF Lock Structure with Virtual Links ...................................................................... 12 4.3.2 Simplex CF Lock Structure with Virtual Links ........................................................................ 14 4.3.3 Duplexed CF Lock with Shared External CF links .................................................................. 15 4.3.4 Simplex CF Lock Structure with Shared External CF links ..................................................... 16

4.4 Db2 Buffer Pool Configuration ...................................................................................................... 17 5.0 Measurement Results and Analysis ................................................................................................. 18

5.1 No Distance (Virtual CF Links) ..................................................................................................... 18 5.2 Short Distance (150 meters) CF Links .......................................................................................... 23 5.3 Long Distance (10 Km) CF Links .................................................................................................. 27

5.4 CF Failure with Asynchronous CF Lock Duplexing ........................................................................... 31 7.0 Conclusion ....................................................................................................................................... 35 8.0 References ...................................................................................................................................... 35


Tables

Table 1: SAP Day Posting Workload ........................................................................................................ 8 Table 2: Buffer Pool Assignments .......................................................................................................... 17 Table 3: Measurement Results - Virtual Links ........................................................................................ 19 Table 4: Measurement Results - Short Distance .................................................................................... 24 Table 5: Measurement Results - Long Distance ..................................................................................... 28 Table 6: Measurement Results - CF Failure ........................................................................................... 31

Figures

Figure 1: Test Environment - Async CF Lock Duplexing with SAP ........................................................... 9 Figure 2: Duplexing - Virtual Links ......................................................................................................... 12 Figure 3: Simplex - Virtual Links ............................................................................................................. 14 Figure 4: Duplexing - External Links....................................................................................................... 15 Figure 5: Simplex - External Links .......................................................................................................... 16 Figure 6: Virtual Links - ITR ................................................................................................................... 20 Figure 7: Virtual Links - CF Utilization .................................................................................................... 22 Figure 8: Short Distance - ITR................................................................................................................ 25 Figure 9: Short Distance - CF Utilization ................................................................................................ 26 Figure 10: Long Distance - ITR .............................................................................................................. 29 Figure 11: Long Distance - CF Utilization ............................................................................................... 30 Figure 12: Recovery Scenario - Before Configuration ............................................................................ 32 Figure 13: Recovery Scenario - After Configuration ............................................................................... 32 Figure 14: CF Lock Structure Takeover - ETR ....................................................................................... 33 Figure 15: CF Lock Structure Takeover - CF Utilization ......................................................................... 33

Disclaimers

Neither this document nor any part of it may be copied or reproduced in any form or by any means or translated into another language, without the prior consent of the IBM Corporation. IBM makes no warranties or representations with respect to the content hereof and specifically disclaims any implied warranties of merchantability or fitness of any particular purpose. IBM assumes no responsibility for any errors that may appear in this document. The information contained in this document is subject to change without any notice. IBM reserves the right to make any such changes without obligation to notify any person of such revision or changes. IBM makes no commitment to keep the information contained herein up to date. Performance data and customer experiences for IBM and non-IBM products and services contained in this document were derived under specific operating and environmental conditions. The actual results obtained by any party implementing any such products or services will depend on a large number of factors specific to such party’s operating environment and may vary significantly. IBM makes no representation that these results can be expected or obtained in any implementation of any such products or services. The information in this document is provided “as-is” without any warranty, either express or implied. IBM expressly disclaims any warranties of merchantability, fitness for a particular purpose or infringement.


Trademarks

© IBM Corporation 1994-2017. All rights reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. AIX, Db2, DFSMSdfp, DFSMShsm, DFSMSrmm, DS8800, FICON, FICON Express, IBM, IBM eServer, IBM logo, Language Environment, OSA, OSA Express, OS/390, Power7, RACF, RMF, S/390, System Storage, System z, VTAM, z Systems, zEC12, z13, z/Architecture, z/OS, and zSeries are trademarks or registered trademarks of the International Business Machines Corporation in the United States and other countries. SAP, the SAP logo, and all other SAP product and service names mentioned herein are trademarks or registered trademarks of SAP SE in Germany and in several other countries all over the world. Unicode is a trademark of Unicode, Inc. Other company, product, or service names may be trademarks or service marks of others.

Acknowledgements

The authors of this document would like to acknowledge the following individuals for their contributions to this project:

Dave Surman for his Parallel Sysplex analysis support

Barbara Weiler for her Parallel Sysplex analysis support

Akiko Hoshikawa for her Db2 analysis support

Andrea Fuga for her z/OS systems programming support

Nick Crimmins for his z/OS systems programming support

Tom Litty for his z/OS systems programming support

Feedback

Please send comments or suggestions for changes to [email protected]

mailto:[email protected]


1.0 Introduction Asynchronous CF Lock Duplexing is a new enhancement to IBM’s parallel sysplex technology that was made generally available in October 2016. It is an alternative to the synchronous system managed duplexing of coupling facility (CF) lock structures that has been available for many years. Duplexing a structure means that two copies of the structure are maintained in separate CFs. There is a primary structure and a secondary structure. So, if one structure is lost due to a failure, the other structure is still available. Duplexing of the CF lock and SCA structures eliminates the need for a failure-isolated CF. A failure-isolated CF means that it is isolated from all the Db2 members of the data sharing group. One example of a failure isolated CF is a stand-alone CF. Another example is an internal CF that doesn’t share a CEC with any connected Db2 member. Failure-isolated CFs eliminate a single point of failure. But, they require additional hardware and that comes at an additional cost. When the CF lock and SCA structures are duplexed, two CECs are enough to provide a highly available solution. Each CEC can have an internal CF and one or more Db2 members of the data sharing group. There are many benefits to duplexing. However, the performance overhead is high with synchronous system managed duplexing of the lock structure, especially as the distance between the primary and secondary structures grows and especially with the SAP workloads. For this reason, IBM has never recommended the use of synchronous system managed CF lock duplexing to our SAP customers. The new Asynchronous CF Lock Duplexing feature was designed to be a viable alternative to synchronous system managed duplexing. The goal was to provide the benefits of lock duplexing without the high performance penalty. It eliminates the synchronous mirroring between the CFs to keep the primary and secondary structures in sync. With Asynchronous CF Lock Duplexing, the secondary lock structure updates are performed asynchronously with respect to the primary lock updates. Db2 (or any exploiter of this feature) can consider the command complete after the primary lock structure has been updated. Before Db2 commits a transaction, it checks that the necessary requests have been successfully written to the secondary structure. Most of the time this will be the case. If it isn’t then the Db2 log write process will be suspended until the updates to the secondary structure are complete. This “sync-up” protocol ensures that the secondary structure contains all the necessary updates (i.e. those that have been committed by Db2) even though at any given moment the updates to the secondary structure are lagging behind those to the primary structure by some amount. The IBM SAP on z Systems Performance Team, located in Poughkeepsie, NY, conducted a number of tests on an IBM z13 to evaluate the performance effects of using the Asynchronous CF Lock Duplexing protocol. We used the SAP Banking Services (SBS) Day Posting workload. It is a good representation of a customer online transaction processing (OLTP) workload. It is dominated by random database accesses and its read/write ratio is roughly 70:30. It has high data sharing costs due to the high amount of inter-DB2 read/write interest. It is also a “pure Db2 workload”. Only Db2 is running on the z13, the application processing is done on another machine. See section 3.0 Workload Description for more information about this workload. We ran tests with simplex CF lock structures (i.e. no lock duplexing), synchronous system managed CF lock structures, and asynchronous duplexed CF lock structures with various distances between our CFs. We started our testing using virtual CF links (i.e. no distance between the CFs) to verify our


configurations and the various locking protocols. We then moved on to testing with 150 meter CF links (short distance) and then with 10 Km CF links (long distance). We also tested a recovery scenario while running the SAP Day Posting workload using asynchronous duplexed CF lock structures where we failed the CF with the primary lock structure. We wanted to confirm that the new protocol was essentially keeping the secondary structure in sync with the primary. We needed the following to use Asynchronous CF Lock Duplexing. But, always check the appropriate product documentation for the latest requirements.

IBM z13 or later hardware

Coupling Facility Control Code (CFCC) Level 21 with Service Level 02.16 or later

z/OS 2.2 SPE with PTFs for APARs OA47796 and OA49148

Db2 12 on z/OS at Function Level V12R1M500 with PTFs for APAR PI66689

IRML 2.3 with PTFs for APAR PI68378

CFRM Couple Dataset (CDS) formatted with the new ASYNCDUPLEX keyword

CFRM Policy with the new ASYNCONLY keyword for the Db2 lock structure

This paper documents our tests and findings. It is not intended to be a guide for implementing Asynchronous CF Lock Duplexing or setting up a highly available simplex. The measurements that we did were stress tests, not certified benchmarks.


2.0 Executive Summary The Asynchronous CF Lock Duplexing feature that was made generally available in October 2016 is a great alternative to the traditional synchronous system managed lock duplexing that has been available for many years. Our measurements with the SAP Day Posting workload showed that there are still some costs associated with asynchronous CF lock duplexing compared to simplex (i.e. not duplexing the lock structure). But, it performs much better than synchronous system managed lock duplexing. Given the benefits of duplexing the CF lock structure, using the asynchronous CF lock duplexing feature should be considered. Keep in mind that the SAP Day Posting workload is a “pure Db2 workload”. Only Db2 is running on the IBM z13, the application processing is done on another machine. The Day Posting workload is also a data sharing intensive workload because of the large amount of inter-Db2 read/write interest that it generates. These facts result in a large amount of CF lock activity, which, in turn, contributes to the overhead of both synchronous and asynchronous duplexing. Depending on your configuration and workload, your results with asynchronous CF lock duplexing may be better or worse. In this document, the terms External Throughput Rate (ETR) and Internal Throughput Rate (ITR) are used. ETR is the transaction rate. ITR is the ETR normalized to 100% CP utilization and it gives a relative CPU time per transaction. See reference [2] on page 35 for more of an explanation of these terms. Comparing the two CF lock duplexing protocols to simplex, our measurements showed a 39% degradation in ITR with synchronous system managed duplexing and an 8% or 9% degradation with asynchronous CF lock duplexing. Our measurements also showed over a 100% increase in CF utilization with both lock duplexing protocols compared to simplex. However, the increase with asynchronous lock duplexing was a little less than with synchronous system managed duplexing. Comparing the two lock duplexing protocols in terms of DB request time, asynchronous CF lock duplexing is the clear winner. And, even better, it is comparable to that of simplex. The DB request time is the time it takes the SAP application server to get a request back from the SAP database server. It consists of the network time and the database processing time. It does not include the SAP application server processing time. The key improvement in the asynchronous CF lock duplexing protocol compared to the synchronous system managed duplexing protocol is the drastic reduction in the number of requests between the CFs to keep the primary and secondary lock structures in sync. Our measurements showed that there were two thirds fewer CF-to-CF requests with asynchronous lock duplexing compared to synchronous system managed duplexing and these requests were asynchronous instead of synchronous. This is significant especially when there is distance between the CFs. As the distance between CECs in a parallel simplex increases, the time it takes for a signal to travel between them increases. This is unavoidable speed-of-light latency due to distance. The asynchronous lock duplexing protocol doesn’t change this. But, by drastically reducing the number of requests it sends between the remote CFs, the asynchronous lock duplexing protocol provides a big performance advantage over synchronous system managed duplexing.


Since the asynchronous lock duplexing protocol updates the secondary lock structure asynchronously, there were some concerns about whether all the required updates would be in the secondary structure when it was needed in a recovery scenario. So, we did a test where we failed the CF with the primary lock structure while the SAP Day Posting workload was running. The test was successful. When the CF failed, the secondary lock structure took over along with all the other secondary structures without any noticeable impact to the system and the Day Posting workload continued to run without any errors.

3.0 Workload Description

The SAP Banking Services Day Posting workload was used in these tests. It is an online transaction processing (OLTP) workload. We have been running this workload for many years. We have quite a bit of experience with it. See reference [1] on page 35. In this workload, a posting is a deposit or a withdrawal from a customer’s account. Typical examples of a posting are a payment out of the account or a deposit into the account. This workload was developed by SAP to simulate customer environments. The workload consists of interactive “users” going through repetitive cycles of 15 dialogue steps.

Step Operation

1 Create a total of 150 postings via five BAPI calls

2 Create five postings

3 Create bank statement

4 Read postings for account

5 Read details of postings


7 Create one bank statement for account



10 Create five payment orders

11 Read balances of account



14 Read balances for account

15 Read master data for account

Table 1: SAP Day Posting Workload


4.0 Test Environment

This section documents the test environment that was used for our tests, including the hardware, the software, and some of the key configuration information. Our test environment simulates a multiple z System and multiple data center configuration that a client could have for high availability. Due to time, resource, and budget constraints, we did not set up a full highly available environment. However, it was not necessary to do this to showcase IBM’s new Asynchronous CF Lock Duplexing feature. A physical 3-tier environment was used. Each of the three layers of the SAP on z Systems solution resided on separate machines. The SAP Database Server was on an IBM z13 system running z/OS. The SAP application servers were IBM Power7 Blade Servers running AIX. The presentation server was an IBM Power5 p55A running AIX. The focus of our tests was the Database Server running on an IBM z13. The following figure provides an overview of our test environment.

Figure 1: Test Environment - Async CF Lock Duplexing with SAP


4.1 Hardware

z Systems Database Server An IBM 2964-N96 (z13) with 12 dedicated CPs, 8 internal coupling facility (ICF) processors, and 3 TB of real storage configured online was used for these tests. Two z/OS simplex partitions and two internal coupling facilities were running on this system. Each z/OS image was configured with 6 dedicated CPs and 1 TB of real storage. Each internal CF was configured with 4 ICFs and 512 GB of real storage. External coupling facility links were used to simulate two separate CECs, each configured with one z/OS image and one internal CF. See section 4.3 Data Sharing Configuration on page 12 for more information on the system configuration.

Coupling Facility Links Various types of coupling facility links were used in these tests.

Internal coupling links (channel type ICP) were used in initial testing. And, they continued to be used for the local connections between the z/OS image and the internal CF running on the same CEC.

Four 12x Parallel Sysplex InfiniBand (PSIFB) HCA3-O fanout links (channel type CIB) were used for the “short distance” tests. These are 150 meter links (or 492 feet).

Four 1x Parallel Sysplex InfiniBand (PSIFB) HCA3-O LR fanout links (channel type CIB) were used for the “long distance” tests. These are 10 Km links (or 6.2 miles).

Database DASD A SBS 7.0 database with 60M accounts was used for these tests. It resided on a dual frame IBM System Storage DS8870 (2107-961) server with 60 ranks of 15K-RPM 146 GB Hard Disk Drives (HDDs) configured as RAID 5. The effective total capacity was 896 3390-mod54 volumes, about 46 TB. The unit has 256 GB regular cache, 8 GB non-volatile storage (NVS), and 16 long wave FICON Express8S attachments. The Db2 subsystem, the database, and two flash copies were contained on 834 emulated 3390-mod54 volumes. The Db2 active logs resided on a separate DASD unit from the database. The active logs were striped across four 3390-mod54 volumes on separate ranks of a single frame IBM System Storage DS8700 (2107-941) server. HyperPAV support provided on the DS8000 family of DASD was used to reduce disk I/O queueing. High Performance FICON for System z (zHPF) with multi-track support was used to improve the efficiency of I/O resources.

Application Servers Twenty-five IBM PS701 8406-71Y Blade Servers running AIX were used for the SAP application servers. Each of these Power7 blades had eight 3.0 GHz processor cores and 128 GB of memory. The SAP central instance resided on one of these blades, along with a stand-alone SAP enqueue server. Four SAP dialog instances were installed on each of the remaining blades.

Presentation Server One IBM 9133-55A server with four 2.1 GHz processor cores and 32 GB of memory running AIX was used as the presentation server to drive the workload.


Network A dedicated 10 Gb Ethernet network was used to connect the presentation server, the applications servers, and the database server. The application servers were connected via 10 Gb Ethernet adapters through a 10 Gb Ethernet switch to the zEC12 via two OSA-Express4S adapters. The Optimized Latency Mode (OLM) option of the OSA-Express4S adapters was used to improve the elapsed time of this communication.

4.2 Software

z/OS z/OS release 2.2 with PTFs for APARs OA47796 and OA49148 Coupling Facility Control Code CFCC Level 21 with Service Level 02.17

Db2 on z/OS Db2 12 at function level V12R1M500 with PTFs for APAR PI66689 IRLM 2.3 with PTFs for APAR PI68378

Db2 Connect IBM Data Server Driver for CLI that is shipped as part of Db2 Connect 11.1 SB35550

AIX AIX 7.1.0

SAP SAP NetWeaver 7.1 Enhancement Package 1 SAP kernel level 721 EXT, 64-bit, patch number 500


4.3 Data Sharing Configuration

This section describes the 2-way data sharing configurations that we used on the IBM z13 and what we were simulating in our tests.

4.3.1 Duplexed CF Lock Structure with Virtual Links

Initially, our test environment used internal coupling peer (ICP) channels (ie. virtual links) to communicate with the CFs. This setup was sufficent to functionally test the various CF locking protocols. ICP channels are the easiest to use in a test environment since no external cables are needed and they provide the best performance. However, this configuration doesn’t reflect a highly available customer enviroment. This same configuration was used for both our synchronous system managed duplexing and our asynchronous CF lock duplexing tests with virtual links. The following figure shows our data sharing configuration with CF lock duplexing using virtual links.

Figure 2: Duplexing - Virtual Links


The SAP Database Server had two z/OS LPARs, named S30 and S32. Each of these LPARs were configured with 6 dedicated general purpose CPs and 1 TB of real storage. Member 1 (BD01) of the Db2 data sharing group was on S30 and member 2 (BD02) of the data sharing group was on S32. Each Db2 member had local buffer pools defined with about 230 GB of storage. There also were two internal coupling facilities, named CF30 and CF32. Each of these CFs were configured with 4 dedicated ICF processors and 512 GB of real storage. The group buffer pools were defined with about 203 GB of storage. The lock structure was defined with 8 GB of storage. It gets populated with 2 G lock entries with the SAP Day Posting workload. The SCA (Shared Communication Area) structure was defined with 25M of storage. When the lock structure is duplexed, the SCA structure should also be duplexed. However, Asynchronous CF Duplexing is currently only available for lock structures. So, in both our duplexing test scenarios, the SCA structure was synchronously system managed duplexed. This is not an issue since the volume of activity to the SCA is very low. In our duplexing tests, the primary lock structure and SCA structure are on CF30. The secondary lock and SCA structures are on CF32. With Asynchronous CF Lock Duplexing, the primary lock structure is the most heavily used structure. Group buffer pool duplexing was enabled for all the measurements. Group buffer pool duplexing costs very little in terms of performance overhead and it is highly recommended. The primary GBP structures were spread across the two CFs. The GBP structures with the greatest amount of activity had their primary structures on CF32 and their secondary structures on CF30. The GBP structures with the least amount of activity had their primary structures on CF30 and their secondary structures on CF32. The volume of requests to the secondary GBP structures is much less than that to the primary GBP structures.


4.3.2 Simplex CF Lock Structure with Virtual Links

Our test environment for testing without CF lock duplexing was the same as with duplexing, except the CF lock and SCA structures were not duplexed. Therefore, there were no secondary lock or SCA structures in CF32. The following figure shows our data sharing configuration without CF lock duplexing using virtual links.

Figure 3: Simplex - Virtual Links


4.3.3 Duplexed CF Lock with Shared External CF links

After verifying the various CF lock protocols using virtual links, we added Parallel Sysplex Infiniband (PSIFB) coupling facitliy links to our test environment. Although we had only one CEC in our test environment, we used four short distance external Infiniband (IFB) CF links to make it look like we had two CECs, each with a z/OS LPAR and an internal CF, physically separated by meters within a single data center. Each CF link was configured with two channel paths. We had eight channel paths defined between S30 and CF32, eight between S32 and CF30, and eight channel paths between CF30 and CF32. We also tested with 10 Km IFB links between the two “CECs” to make it look like they were at different data centers separated by kilometers. The local CF was connected to the local z/OS using four dedicated Internal Coupling Peer (ICP) channels. The following figure shows our data sharing configuration with CF lock duplexing using shared external CF links.

Figure 4: Duplexing - External Links


4.3.4 Simplex CF Lock Structure with Shared External CF links

Our test environment for testing without CF lock duplexing was the same as with duplexing, except the CF lock and SCA structures were not duplexed. Therefore, there are no secondary lock or SCA structures in CF32. The following figure shows our data sharing configuration without CF lock duplexing using shared external CF links.

Figure 5: Simplex - External Links


4.4 Db2 Buffer Pool Configuration

Thirteen buffer pools were defined. The SAP Day Posting workload generates GBP interest for nine of these buffer pools. The following table shows how objects were assigned to buffer pools. In addition to the buffer pools listed in the table, BP1 is also used. It contains 4K sort work files, however, there is no sorting in the Day Posting workload.

Buffer Pool Objects

BP0 4K Db2 catalog tables / indexes

BP2 4K tables

BP3 4K indexes

BP40 LOBs

BP8K0 8K Db2 catalog tables

BP8K1 8K tables, including BCA_PAYMITEM

BP8K2 8K indexes

BP16K0 16K Db2 catalog tables

BP16K1 16K tables

BP32K 32K Db2 Catalog tables

BP32K1 32K tables

BP32K2 32K indexes

Table 2: Buffer Pool Assignments


5.0 Measurement Results and Analysis

First, we executed a set of measurements with a simplex CF lock structure (i.e. no CF lock duplexing), a synchronous system managed lock structure, and an asynchronous duplexed lock structure using virtual links to communicate with our CFs. Virtual links mean there was no distance or external links to the CFs. These measurements established a baseline and verified the various locking protocols with our SAP Day Posting workload. Then, we executed two sets of measurements with simplex lock structures and asynchronous duplexed lock structures using external CF links, one set was with short distance (150 meter) links and one set was with long distance (10 Km) links. We also tested a recovery scenario using an asynchronous duplexed lock structure where we failed the CF with the primary lock structure. We wanted to confirm that the new protocol was essentially keeping the secondary structure in sync with the primary structure and it could take over in the case of a loss of the primary structure. For all these measurements, we ran 2-way data sharing on an IBM z13 with two z/OS LPARs and two internal CFs. Each z/OS LPAR was configured with 6 dedicated general purpose processors and 1 TB of real memory. Each internal CF was configured with 4 dedicated ICFs and 512 GB of real memory. See section 4.0 Test Environment on page 9 for additional details of the test environment. The measurements and the results are documented and discussed in this section.

5.1 No Distance (Virtual CF Links)

Using virtual CF links, we executed three measurements, each with a different CF locking protocol. The first measurement was without duplexing the CF lock and SCA structures (i.e. simplex mode). The second measurement was with using the traditional synchronous system managed duplexing for the CF lock and SCA structures. And, the third measurement was with using the new asynchronous duplexing for the CF lock structure and the traditional synchronous system managed duplexing for the SCA structure. The new asynchronous CF duplexing protocol only currently supports CF lock structures. There was very little activity to the SCA structure so using synchronous system managed duplexing for this structure is not a concern. For these measurements, the user load was held constant at 6432 users. See section 4.3.2 Simplex CF Lock Structure with Virtual Links on page 14 and 4.3.1 Duplexed CF Lock Structure with Virtual Links on page 12 for details of the test configurations. These measurements provided a good performance comparison between the three locking protocols using a realistic customer-like workload. The details of these measurements are summarized in the following table.


Simplex

Virtual Links

Sync System Managed Duplexing

Virtual Links

Async CF Lock Duplexing

Virtual Links

Run id S70126B1 S70130B1 S70131B1

z/OS Level 2.2 2.2 2.2

CF Level 21 2.16 21 2.16 21 2.16

Db2 Level Db2 12 Db2 12 Db2 12

Db2 Connect Level 11.1 11.1 11.1

Local z/OS to Local CF Links 3 ICPs 3 ICPs 3 ICPs

Local z/OS to Remote CF Links 3 ICPs 3 ICPs 3 ICPs

CF to CF Links 2 ICPs 2 ICPs 2 ICPs

Number of Users 6432 6432 6432

DB Request Time (secs) 0.314 0.735 0.332

ETR (DS/sec) 603.37 534.17 601.53

z/OS CP Utilization 60.18% 86.68% 65.08%

ITR (DS/sec) 1002.61 616.26 924.29

Average CF Utilization 18.7% 40.8% 37.5%

CF30 CF32 CF30 CF32 CF30 CF32

CF utilization 22.6% 14.7% 39.2% 42.3% 40.6% 34.4%

Total Requests /sec 217,632 82,913 192,108 207,479 217,054 83,136

Sync Requests /sec 158,664 79,566 129,617 193,386 157,557 78,778

Average Service Time (Sync Requests) 3.5 7.5 18.8 15.5 4.6 8.1

Async Requests /sec 58,882 3,446 62,450 14,168 59,467 4,129

Average Service Time (Async Requests) 13.9 97.0 47.7 158.2 34.4 121.5

LOCK1 Requests /sec 157,000 n/a 136,400 135,800 156,900 1,446

Sync Requests /sec 156,667 n/a 128,889 127,778 155,556 1,446

Average Service Time (Sync Requests) 3.4 n/a 18.9 19.4 4.6 19.9

Async Requests /sec 3 n/a 7,992 7992 917 0

Average Service Time (Async Requests) 62.8 n/a 156.5 159.5 65.3 0.0

ASYNC DUPLEX CF Ops Total /sec n/a n/a n/a n/a n/a 198,438

Average Service Time n/a n/a n/a n/a n/a 8.5

SYNC_UP REQs - Total /sec n/a n/a n/a n/a n/a 1,900

#SUSPEND /sec n/a n/a n/a n/a n/a 0.7

Average Suspend Time n/a n/a n/a n/a n/a 420.3

SYNC CF-to-CF Requests /sec 0 0 308,714 308,603 11.5 11.5

Average Service Time 0 0 1.4 0.2 7.3 5.1

Table 3: Measurement Results - Virtual Links


The following chart compares the ITRs for the three measurements with the different locking protocols.

Figure 6: Virtual Links - ITR

These results show that Asynchronous CF Lock Duplexing is a great alternative to Synchronous System Managed Duplexing. Even though there is an 8% degradation in ITR with Async CF Lock Duplexing compared to Simplex, this is much better than the 39% degradation in ITR with the traditional Synchronous System Managed Duplexing compared to Simplex. In the synchronous system managed duplexing measurement, the ETR dropped significantly and the z/OS CP utilization drastically increased. This caused the big drop in ITR with synchronous system managed duplexing. On the other hand, in the asynchronous duplexing measurement, the ETR is basically equivalent to that in the simplex measurement and the z/OS CP utilization increased slightly. This caused the slight drop in ITR with asynchronous duplexing. There was a 134% increase in DB request time with synchronous system managed duplexing compared to simplex. With asynchronous duplexing, there was just a 5% increase in DB request time compared to simplex. This is a huge improvement. The DB request time is the time it takes the SAP application server to get a request back from the SAP database server. It consists of the network time and the database processing time. It does not include the SAP application server processing time. The SAP Day Posting workload is a “pure Db2 workload”, only Db2 is running on the z13. It is also a data sharing intensive workload because of the large amount of inter-Db2 read/write interest that it generates. Because of these, there is a high amount of activity to the CF lock structure. This contributes to the overhead of CF lock duplexing, both synchronous and asynchronous.


In these measurements, the majority of the requests to the CF are requests to the lock structures. As seen from Table 3: Measurement Results - Virtual Links, 52% of the total CF requests are lock requests in the simplex and the asynchronous lock duplexing measurements. In the synchronous system managed duplexing measurement, 68% of the total CF requests are lock requests. Most of the other CF requests are to group buffer structures that were user managed duplexed. Asynchronous CF lock duplexing drastically reduced the number of lock requests to the secondary lock structure. The secondary lock structure was on CF32 for these tests. The average service times for both the synchronous and asynchronous CF lock requests are extremely high in the synchronous system managed duplexing measurement. However, with the asynchronous lock duplexing measurement, they are more in line with the simplex measurement. With asynchronous duplexing, a command to the CF completes without having to wait for the secondary structure to be updated. The secondary update is done asynchronously. High CF service times can elongate transaction response times and possibly aggravate global lock contention. A key improvement in the asynchronous CF lock duplexing protocol compared to the synchronous system managed duplexing protocol is the drastic reduction in the number of requests between the CFs to keep the primary and secondary lock structures in sync. In our synchronous system managed duplexing measurements, there were over 600,000 synchronous requests per second. In our asynchronous lock duplexing measurement, there were less than 200,000 asynchronous requests per second. This is significant especially when there is distance between the CFs. For this set of measurements, we used virtual links so we don’t see the impact caused by distance in the overall measurement results. In the asynchronous lock duplexing measurement, Db2 only had to send 1900 sync-up requests per second to the secondary lock structure at commit time to verify that the lock entry had been hardened into the structure. And, Db2 only had to suspend its log write process on average 0.7 times per second to wait for the lock entry to be hardened into the secondary lock structure. This is extremely good. Beware that your results may differ depending on your workload and configuration. Both synchronous system managed duplexing and asynchronous lock duplexing come with a cost of additional CF utilization. Our measurements show a 118% increase in average CF utilization with synchronous system managed duplexing compared to simplex and a 101% increase with asynchronous lock duplexing compared to simplex. So, asynchronous lock duplexing did improve on the CF utilization cost compared to synchronous system managed duplexing. The chart below shows the details. CF30 contained the primary lock structure and CF32 contained the secondary lock structure for these measurements.


Figure 7: Virtual Links - CF Utilization


5.2 Short Distance (150 meters) CF Links

Using short distance Parallel Sysplex Infinband (PSIFB) coupling facitliy links, we executed two measurements. The first measurement was without duplexing the CF lock and SCA structures (i.e. simplex mode). The second measurement was using the new asynchronous duplexing for the CF lock structure and the traditional synchronous system managed duplexing for the SCA structure. The new asynchronous duplexing protocol only currently supports CF lock structures. There was very little activity to the SCA structure so using synchronous system managed duplexing for this structure is not a concern. We did not include a synchronous system managed duplexing measurement in this set of measurements since it becomes more challenging to get an optimal measurement with the SAP Day Posting workload with this configuration as we add external CF links and distance. With synchronous system managed duplexing, every update request to the CF requires synchronous CF-to-CF communication. And, with the Day Posting workload, there are a lot of update requests to the CF. Since we do not recommend synchronous system managed duplexing to our SAP customers and the focus of this study is on the new Asynchronous CF Lock Duplexing feature, it was not necessary to do this test. For these measurements, we held the z/OS CP utilization constant at about 80%. To do this, we varied the user load, which affects the throughput (or ETR). Holding the CP utilization constant between the measurements equalizes the CP queuing effects. See section 4.3.4 Simplex CF Lock Structure with Shared External CF links on page 16 and 4.3.3 Duplexed CF Lock with Shared External CF links on page 15 for details of the test configurations. The details of these measurements are summarized in the following table.


Simplex at 150 meters

Async CF Lock Duplexing

at 150 meters

Run id S70502B1 S70428B1

z/OS Level 2.2 2.2

CF Level 21 2.17 21 2.17

Db2 Level Db2 12 Db2 12

Db2 Connect Level 11.1 11.1

Local z/OS to Local CF Links 4 dedicated ICP 4 dedicated ICP

Local z/OS to Remote CF Links 8 shared CIB 8 shared CIB

CF to CF Links 8 shared CIB 8 shared CIB

Number of Users 7968 7200

DB Request Time (secs) 0.569 0.532

ETR (DS/sec) 677.57 627.60

z/OS CP Utilization 79.14% 80.14%

ITR (DS/sec) 856.17 783.18

Average CF utilization 20.9% 49.4%

CF30 CF32 CF30 CF32

CF utilization 26.0% 15.7% 51.3% 47.4%

Total Requests /sec 239,260 90,774 225,851 87,462

Sync Requests /sec 171,713 85,837 159,042 82,073

Average Service Time (Sync Requests) 5.0 10.3 6.7 12.0

Async Requests /sec 68,388 5,277 66,310 5,521

Average Service Time (Async Requests) 20.9 68.0 45.2 123.8

LOCK1 Requests /sec 170,200 n/a 160,400 1,685

Sync Requests /sec 170,000 n/a 157,778 1,684

Average Service Time (Sync Requests) 4.9 n/a 6.6 22.8

Async Requests /sec 9 n/a 2,821 0

Average Service Time (Async Requests) 45.1 n/a 95.9 0

ASYNC DUPLEX CF Ops Total /sec n/a n/a n/a 202,766

Average Service Time n/a n/a n/a 18.2

SYNC_UP REQs - Total /sec n/a n/a n/a 2,001

#SUSPEND /sec n/a n/a n/a 12

Average Suspend Time n/a n/a n/a 360.7

SYNC CF-to-CF Requests /sec 0 0 11.9 11.9

Average Service Time 0 0 14.6 12.6

Table 4: Measurement Results - Short Distance


The following chart compares the ITRs for the two measurements.

Figure 8: Short Distance - ITR

The measurement results show an 8.5% degradation in ITR with Asynchronous CF Lock Duplexing compared to Simplex. However, given the benefits that lock duplexing provides, Asynchronous CF Lock Duplexing is a viable solution and should be considered. This was not the case with the traditional synchronous system managed duplexing in the SAP environment. The performance overhead of synchronous system managed duplexing was such that it was not recommended for use for SAP. See section 5.1 No Distance (Virtual CF Links) on page 18 for a direct comparison of the three CF locking protocols. These measurements also show a 136% increase in average CF utilization with asynchronous CF lock duplexing compared to simplex. The chart below shows the details. CF30 contained the primary lock structure and CF32 contained the secondary lock structure for these measurements.


Figure 9: Short Distance - CF Utilization


5.3 Long Distance (10 Km) CF Links Using 10 Km long distance Parallel Sysplex Infinband (PSIFB) coupling facitliy links, we executed two measurements. The first measurement was without duplexing the CF lock and SCA structures (i.e. simplex mode). The second measurement was using the new asynchronous duplexing for the CF lock structure and the traditional synchronous system managed duplexing for the SCA structure. The new asynchronous CF duplexing protocol only currently supports CF lock structures. There was very little activity to the SCA structure so using synchronous system managed duplexing for this structure is not a concern. Again, we did not include a synchronous system managed duplexing measurement in this set of measurements since it becomes even more challenging to get an optimal measurement with the SAP Day Posting workload as we add distance between the CFs. With synchronous system managed duplexing, every update request to the CF requires synchronous CF-to-CF communication. And, when the CFs are separated by a long distance, the time to service the request is elongated at least by the time it takes the signal to travel the distance between the CFs. Since we do not recommend synchronous system managed duplexing to our SAP customers and the focus of this study is on the new Asynchronous CF Lock Duplexing feature, it was not necessary for us to do this test. Again, for these measurements, we held the z/OS CP utilization constant at about 80%. To do this, we varied the user load, which affects the throughput (or ETR). Holding the CP utilization constant between the measurements equalizes the CP queuing effects. See section 4.3.4 Simplex CF Lock Structure with Shared External CF links on page 16 and 4.3.3 Duplexed CF Lock with Shared External CF links on page 15 for details of the test configurations. The details of these measurements are summarized in the following table.


Simplex at 10 Km

Async CF Lock Duplexing at 10 Km

Run id S70403B2 S70403B1

z/OS Level 2.2 2.2

CF Level 21 2.17 21 2.17

Db2 Level Db2 12 Db2 12

Db2 Connect Level 11.1 11.1

Local z/OS to Local CF Links 4 dedicated ICP 4 dedicated ICP

Local z/OS to Remote CF Links 8 shared CIB 8 shared CIB

CF to CF Links 8 shared CIB 8 shared CIB

Number of Users 6432 5664

DB Request Time (secs) 0.561 0.541

ETR (DS/sec) 571.24 517.37

z/OS CP Utilization 81.29% 81.13%

ITR (DS/sec) 702.76 637.70

Average CF Utilization 19.5% 40.9%

CF30 CF32 CF30 CF32

CF Utilization 24.9% 14.1% 45.8% 35.9%

Total Requests /sec 201,585 76,230 184,983 70,446

Sync Requests /sec 75,051 31,024 65,104 30,720

Average Service Time (Sync Requests) 5.3 24.7 6.8 27.5

Async Requests /sec 126,482 44,954 120,239 39,876

Average Service Time (Async Requests) 121.6 194.6 130.0 202.48

LOCK1 Requests /sec 144,000 n/a 133,200 1,466

Sync Requests /sec 74,444 n/a 64,444 1,466

Average Service Time (Sync Requests) 5.2 n/a 6.7 84.6

Async Requests /sec 70,000 n/a 68,889 0

Average Service Time (Async Requests) 144.3 n/a 150.3 0

ASYNC DUPLEX CF Ops Total /sec n/a n/a n/a 167,100

Average Service Time n/a n/a n/a 76.1

SYNC_UP REQs - Total /sec n/a n/a n/a 1,772

#SUSPEND /sec n/a n/a n/a 1

Average Suspend Time n/a n/a n/a 254.1

SYNC CF-to-CF Requests /sec 0 0 11.3 11.3

Average Service Time 0 0 142.3 139.2

Table 5: Measurement Results - Long Distance


The following chart compares the ITRs for the two measurements.

Figure 10: Long Distance - ITR

The measurement results show a 9.2% degradation in ITR with Asynchronous CF Lock Duplexing compared to Simplex. This is very similar to what was seen in our measurement with short distance CF links. So, again, given the benefits that lock duplexing provides, Asynchronous CF Lock Duplexing is a viable solution and should be considered. This was not the case with the traditional synchronous system managed duplexing in the SAP environment. The performance overhead of synchronous system managed duplexing was such that it was not recommended for use for SAP. See section 5.1 No Distance (Virtual CF Links) on page 18 for a direct comparison of the three CF locking protocols. As the distance between CECs in a parallel simplex increases, the time it takes for a signal to travel between them increases. This is unavoidable speed-of-light latency due to distance. The asynchronous lock duplexing protocol doesn’t change this. However, the asynchronous duplexing protocol drastically reduces the number of requests that are sent between the remote CFs compared to the synchronous system managed duplexing protocol. This reduction in requests provides a significant performance benefit. These measurements also show a 110% increase in average CF utilization with asynchronous CF lock duplexing compared to simplex. The chart below shows the details. CF30 contained the primary lock structure and CF32 contained the secondary lock structure for these measurements.


Figure 11: Long Distance - CF Utilization


5.4 CF Failure with Asynchronous CF Lock Duplexing

We tested a failure scenario where the CF with the primary lock structure failed while using asynchronous CF lock duplexing and 10 Km long distance links. We caused a CF failure while the SAP Day Posting workload was running. We wanted to confirm that the new asynchronous locking protocol was essentially keeping the secondary structure in sync with the primary structure and would be ready for takeover. The details of this measurement are summarized in the following table.

Async CF Lock Duplexing at 10 Km

with CF failure

Run id S70408B1

z/OS Level 2.2

CF Level 21 2.17

Db2 Level Db2 12

Db2 Connect Level 11.1

Local z/OS to Local CF Links 4 dedicated ICP

Local z/OS to Remote CF Links 8 shared CIB

CF to CF Links 8 shared CIB

Start of Measurement Interval 19:33:32

Time of CF Failure 19:38:49

Number of Benchmark Users in Error 0

Table 6: Measurement Results - CF Failure For this measurement, the location of our primary and secondary structures were switched from our other measurements. The primary lock and SCA structures were on CF32 and their secondary structures were on CF30. The GBP structures with the greatest amount of activity had their primary structures on CF30 and their secondary structures on CF32. The GBP structures with the least amount of activity had their primary structures on CF32 and their secondary structures on CF30. The procedure for this test was pretty simple. We started the SAP Day Posting workload and got it running at a steady state. After about five minutes, we deactivated CF32 to simulate a CF failure. Messages in the z/OS syslogs showed the paths to the failing CF were no longer operational, duplexing was stopped for the lock, SCA, and GBP structures, and simplex mode was established. Message IXC577I was displayed at 19:38:49.78.

Without any noticeable interruption or impact to the system, all the structures in CF30 became active in simplex mode. The workload continued to run without any errors.

IXC577I SYSTEM-MANAGED DUPLEXING REBUILD HAS BEEN COMPLETED FOR

STRUCTURE BAP_LOCK1

STRUCTURE NOW IN COUPLING FACILITY CF30

PHYSICAL STRUCTURE VERSION: D25AE330 5EF9DD4C

LOGICAL STRUCTURE VERSION: D25AE32F BE65A54C

AUTO VERSION: D25AE330 459D8B4C


The following figure shows the configuration before the CF32 failure.

Figure 12: Recovery Scenario - Before Configuration

The following figure shows the configuration after the CF32 failure.

Figure 13: Recovery Scenario - After Configuration


The following chart shows the ETR during the 10 minute measurement interval from 19:33 to 19:42. The CF failure occurred at 19:38. The ETR is relatively consistent throughout the interval.

Figure 14: CF Lock Structure Takeover - ETR

The following chart shows the CF utilization during the 10 minute measurement inteval. It shows the CF32 utilization going to zero at 19:39 after the failure.

Figure 15: CF Lock Structure Takeover - CF Utilization


With the asynchronous CF lock duplexing protocol, at any given point in time, updates to the secondary structure are lagging behind the primary structure by some amount. The key point of this test was to verify that if there was a CF or structure failure that required a switch to the secondary copy of the lock structure then the secondary structure would have all the required updates. This test proved that this was the case. The process to re-establish duplexing after the failing CF is fixed and brought back online is the same for both synchronous system managed duplexing and asynchronous CF lock duplexing. This test case did not look at this.


7.0 Conclusion

Asynchronous CF lock duplexing is a great alternative to synchronous system managed duplexing. It provides all the benefits of lock duplexing at a fraction of the cost of the traditional synchronous system managed duplexing. It is a feature worth considering for a highly available environment to ensure business continuity. However, keep in mind, that additional coupling facility capacity will be required if you are not currently duplexing the CF lock structures. Also, asynchronous duplexing of the lock structures doesn’t help with the unavoidable speed-of-light latency due to distance. So, if you are adding distance between your coupling facilities, as well as implementing asynchronous duplexing of lock structures then you will need to take into account the overhead incurred due to the distance. The asynchronous lock duplexing protocol does minimize the number of requests that are sent between the remote CFs to keep the primary and secondary lock structures in sync so this is a significant performance benefit. IBM’s parallel simplex technology is top of the line in providing scalability, availability, system management, and total cost of ownership. And, IBM continues to invest in it.

8.0 References

1. IBM Corp. 2012. IBM System zEnterprise, System Storage, and DB2 10 for z/OS: SAP Banking Services 7.0 150 Million Accounts Measurements http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101978

2. IBM Corp. 2016. Large Systems Performance Reference, Document Number SC28-1187-19 https://www-304.ibm.com/servers/resourcelink/lib03060.nsf/pages/lsprindex/$file/SC28118719.pdf

3. IBM Corp. 2016. DB2 12 for z/OS Technical Overview http://www.redbooks.ibm.com/redbooks/pdfs/sg248383.pdf

4. SAP SE 2016. Scaling Progressive SAP Solutions with DB2 12; Johannes Schuetzner, Bernd Kohler https://assets.cdn.sap.com/sapcom/docs/2016/10/cecef6df-917c-0010-82c7-eda71af511fa.pdf

5. IBM extends its z Systems Parallel Sysplex Leadership with Asynchronous CF Duplexing for Lock Structures; IBM US Hardware Announcement 116-093; 10/4/2016 https://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=an&subtype=ca&appname=gpateam&supplier=897&letternum=ENUS116-093

6. Mainframe Insights Blog: z/OS Parallel Sysplex support for Asynchronous CF Duplexing for Lock Structures; David Surman; 12/15/2016 http://mainframeinsights.com/asynchronous-coupling-facility-cf-duplexing-for-lock-structures/

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101978

https://www-304.ibm.com/servers/resourcelink/lib03060.nsf/pages/lsprindex/$file/SC28118719.pdf

http://www.redbooks.ibm.com/redbooks/pdfs/sg248383.pdf

https://assets.cdn.sap.com/sapcom/docs/2016/10/cecef6df-917c-0010-82c7-eda71af511fa.pdf

https://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=an&subtype=ca&appname=gpateam&supplier=897&letternum=ENUS116-093

https://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=an&subtype=ca&appname=gpateam&supplier=897&letternum=ENUS116-093

IBM Technical BriefFILE/Async_CF_Lock_Duplexing_withSAP.pdf · Keep in mind that the SAP Day...

Documents

Transcript of IBM Technical BriefFILE/Async_CF_Lock_Duplexing_withSAP.pdf · Keep in mind that the SAP Day...