Post on 12-Mar-2018
© Copyright IBM Corporation All Rights Reserved 2011
Information Management Performance and
Benchmarks Toronto Lab Information Management Development Boeblingen
Products & Solutions Support Center Montpellier
SAP® TRBK Banking Benchmark
using IBM DB2 pureScale® for Linux/Intel
Benchmark report
Document information
Name DB2 PS SAP TRBK Benchmark Report
Version V1.3.2
Date 14/12/11
Authors Ref to chap 1.3
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
2/168
Table of content
1 Introduction ........................................................................................................................ 5
1.1 Document purpose ...................................................................................................... 5
1.2 Document content ....................................................................................................... 5
1.3 Document authors ....................................................................................................... 6
1.4 Reminders ................................................................................................................... 7
1.5 Trademarks .................................................................................................................. 7
2 Executive Summary ............................................................................................................. 9
3 Project Overview & Requirements ..................................................................................... 11
3.1 Objectives .................................................................................................................. 11
3.2 Tests scenario description .......................................................................................... 11
3.2.1 Day processing scenario ..................................................................................... 11
3.2.2 Night processing scenario ................................................................................... 11
3.2.3 Scalability of the architecture ............................................................................. 11
3.2.4 High Availability ................................................................................................. 11
3.2.5 Disaster Recovery ............................................................................................... 12
4 Architecture – system configuration .................................................................................. 13
4.1 Overall layout ............................................................................................................ 13
4.2 Servers overview........................................................................................................ 14
4.2.1 Application servers - HS22 .................................................................................. 14
4.2.2 Database servers - x3850 x5 ............................................................................... 15
4.3 Storage overview ....................................................................................................... 18
4.3.1 Assumptions for DB2 / GPFS / Storage design ..................................................... 21
4.3.2 LUN and file system for DB2 with 768 disks FC .................................................. 23
4.3.3 Detailed of the LUNs configuration ..................................................................... 24
4.4 Network fabric overview ............................................................................................ 25
4.4.1 Infiniband network ............................................................................................. 25
4.4.2 Fiber Channel network ....................................................................................... 26
4.4.3 Ethernet network ............................................................................................... 26
4.5 Operating Systems ..................................................................................................... 28
4.5.1 Suse Enterprise Linux 10 SP3 .............................................................................. 28
4.5.2 I/O device drivers ............................................................................................... 28
4.5.3 Settings tuned .................................................................................................... 30
4.6 DB2 pureScale ............................................................................................................ 30
4.6.1 DB2 concepts ..................................................................................................... 30
4.6.2 Integrated Technologies ..................................................................................... 31
4.7 SAP ............................................................................................................................ 33
4.7.1 TRBK overview ................................................................................................... 33
4.7.2 Benchmark Configuration ................................................................................... 34
4.7.3 System Installation ............................................................................................. 35
5 Test Results : Performance, Scalability & Availability .......................................................... 43
5.1 Overall test configuration .......................................................................................... 43
5.3 TRBK Day Run ............................................................................................................ 43
5.3.1 Day run settings + results achieved .................................................................... 43
5.3.2 Scalability ........................................................................................................... 44
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
3/168
5.3.3 Specific Day Run Tuning ..................................................................................... 44
5.3.4 Conclusion........................................................................................................ 46
5.4 TRBK Night Run .......................................................................................................... 47
5.4.1 Night run settings + results achieved .................................................................. 47
5.5 Experiments with packagesize and number of work processes ................................... 49
5.5.1 Resource usage ............................................................................................... 49
5.5.2 Conclusion........................................................................................................ 50
5.6 High Availability Test .................................................................................................. 51
5.6.1 Results ............................................................................................................... 51
5.6.2 GPFS IO fencing .................................................................................................. 51
5.6.3 Measuring the outage time ................................................................................ 52
5.6.4 Methods to generate the failure ......................................................................... 52
5.6.5 Behaviors in different failure scenarios ............................................................... 52
5.6.6 Reduce the client reroute time ........................................................................... 53
5.6.7 Reduce the number of SAP® errors for read only transactions ........................... 53
6 Disaster Recovery tests ...................................................................................................... 54
6.1 Layout change............................................................................................................ 54
6.2 Results ....................................................................................................................... 54
6.3 Site Failure – Storage replication ................................................................................ 55
6.3.1 Site Failure Implementation ............................................................................... 56
6.3.2 Disaster recovery mechanism ............................................................................. 56
6.3.3 High level process of recovery. ........................................................................... 57
6.3.4 GPFS configuration ............................................................................................. 60
6.3.5 Recovery Procedure on DR Site .......................................................................... 63
6.3.6 Going back to the production configuration ....................................................... 74
6.4 Storage interconnect failure and catchup ................................................................... 83
6.4.1 Implementation ................................................................................................. 83
6.4.2 Storage Interconnect test mechanism ................................................................ 86
6.4.3 High level process .............................................................................................. 86
6.4.4 Storage Interconnect test results ........................................................................ 89
6.5 Queue Replication to logical copy .............................................................................. 94
6.5.1 Methodology ..................................................................................................... 95
6.5.2 Results ............................................................................................................... 96
6.6 Site failure with Storage Replication and QRep .......................................................... 98
6.6.1 QRep & Site Failure Implementation .................................................................. 98
6.6.2 Results ............................................................................................................... 98
6.7 Backup during on-line operations............................................................................... 98
6.7.1 On line backup Implementation ......................................................................... 98
6.7.2 On-line Backup mechanism .............................................................................. 100
6.7.3 Online Backup Procedure ................................................................................. 104
6.7.4 Create Online backup using Remote FlashCopy ................................................ 107
6.7.5 Refresh Online backup clone using Remote FlashCopy ..................................... 115
6.7.6 Accessing FlashCopy database Clone (from DR site) .......................................... 121
6.7.7 Going back to the production configuration ..................................................... 126
6.7.8 Conclusion ....................................................................................................... 127
7 SAP® TRBK Certification results ....................................................................................... 128
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
4/168
7.1 Architecture – system configuration ........................................................................ 128
7.1.1 Overall layout ................................................................................................... 128
7.1.2 Application servers - HS22 ................................................................................ 129
7.1.3 Database servers - x3690 x5 ............................................................................. 129
7.1.4 Storage overview ............................................................................................. 132
7.1.5 Network fabric overview .................................................................................. 138
7.1.6 Operating Systems ........................................................................................... 140
7.2 Certification results .................................................................................................. 141
7.3 Tips & recommendations ........................................................................................ 142
7.3.1 Database Configuration .................................................................................... 142
7.3.2 SAP® Application Server .................................................................................. 143
7.3.3 Operating System ............................................................................................. 143
7.3.4 Network Topology ............................................................................................ 144
8 Best Practices for DB2 pureScale® and SAP® TRBK ......................................................... 145
8.1 Tablespace Definitions ............................................................................................. 145
8.1.1 Extent Size........................................................................................................ 145
8.1.2 Increase Size .................................................................................................... 145
8.2 Table Compression .................................................................................................. 146
8.3 Data and Index Percent Free .................................................................................... 146
8.4 Data Partitioning ...................................................................................................... 146
9 Conclusions ..................................................................................................................... 147
10 Appendix ..................................................................................................................... 148
10.1 db2dsdriver.cfg ........................................................................................................ 148
10.2 Malloc Bomb............................................................................................................ 150
10.3 DB2 Registry Settings ............................................................................................... 151
10.4 DB Manager Configuration Settings ......................................................................... 151
10.5 DB2 Configuration Settings ...................................................................................... 153
10.6 enable_dr_nsd.sh .................................................................................................... 156
10.7 disable_dr_nsd.sh .................................................................................................... 157
10.8 disable_dr_nsd_and_enable_fc_nsd.sh .................................................................... 160
10.9 suspend_or_resume_gpfs_fs.sh ............................................................................... 162
10.10 remote_flash_copy.sh .......................................................................................... 162
10.11 disable_repl_nsd.sh ............................................................................................. 165
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
5/168
1 Introduction
1.1 Document purpose
This document describes IBM testing of the SAP® TRBK solution using DB2 pureScale® on
x86/Linux, based on a predefined set of tests scenario.
It was IBM managed under the leadership of the IBM Information Management Performance
and Benchmark team based in Toronto, with the support of the Information Management
Development teams based in Boëblingen, and in San Jose, and carried out in the IBM
Montpellier benchmarking centre (PSSC).
The project has been performed between July and early December 2010 for the performance
and HA tests, then between February and early May 2011 for the DR tests and finally from June
to early August 2011 for the certification test campaign.
The project is referred to as the SAP® TRBK Benchmark using DB2 pureScale® on x86/Linux.
1.2 Document content
The content of this document corresponds to the results achieved during the benchmark
activities. It covers both the SAP® results as well as the system behaviour for the most
representative runs achieved.
This paper is split in various chapters.
Chapter 2 is dedicated to the Executive Summary.
Chapter 3 is related to the overall project description and the various test scenario description
with the associated targets.
Chapter 4 covers the architecture used for this benchmark. It includes details about the level
and the configuration of the software components used (such as DB2 pureScale® and SAP® ), as
well as a description of the various hardware components configuration (servers, storage and
network).
Chapter 5 focuses on the performance results achieved, for the various batch processes
exercised as well as the results of the availability tests executed.
Chapter 6 is a summary of the outcome achieved with the DR test cases
Chapter 7 covers the certification test campaign.
Chapter 8 provides some further remarks and best practices about the DB2 pureScale®
implementation
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
6/168
Chapter 9 draws our overall conclusion based on the test cases executed.
The Appendix chapter contains the various profiles and key parameters used to achieved the
previous described results.
1.3 Document authors
During almost 12 months of intense activity, many different organizations and people have been
involved across IBM to perform this project.
Here are listed the authors & contributors of this document:
Berni Schiefer DE, Information Management Performance and Benchmarks
Russ Stocker Information Management Performance Benchmarks and Solutions Dev. Kostas Rakopoulos Information Management DB2 Performance Quality Assurance
Wallace Chen Information Management Software Developer
Pandu Mutyala Information Management DB2 System Verification
Aruna De Silva IM Solutions - QA Specialist Software Test Specialist
Karl Fleckenstein STSM, Integration IBM Data Server solutions into SAP®
Thomas Matthae IM Senior Development Engineer, SAP® Certified Consultant
Serge Bourbonnais IM STSM, Database Replication, Software Architect
Ping Li IM, DB2 Replication Software Performance Analyst
Anand Krishniyer IM Replication Center of Competency, Software Developer: Java
Aslam Nomani DB2 LUW Quality Assurance Test and Tooling
Brent Beardsley DE, Enterprise Storage Subsystems, Master Inventor, HW Designer
Christian Hugues Performance Benchmark System x
Gael Marlin Benchmark System x
Alain Fournil Performance Benchmark DB2 UDB
Philippe Jackimczyk Certified Senior IT specialist - Storage Networking
Alexis Gausach Storage specialist
Hervé Guerin SAP® benchmark specialist – SAP® Certified consultant
Francoise Alabiso Certified Executive IT Architect – PSSC Technical Director
Hervé Sabrié Certified Senior Project Manager – PMP
Beside the people listed here, many other specialists from different teams have been involved at
some point in time to support this project. Our thanks go to all of them also.
We would like also to thank our sponsors for this project as well as the various managers who
helped to make this happen.
Vijay Lund SWG, Information management – Vice President Cross IBM Offerings
Ian Hurst S&D, General Manager, Global Financial Services Sector
Erich Baier STG, HW Development, Vice President, System x and RSS Development
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
7/168
David Gelardi STG, Vice President, STG Competitive Labs, STG Technical Sales Centers
1.4 Reminders
Neither this document nor any part of it may be copied or reproduced in any form or by any
means or translated into another language without the prior consent of the IBM Corporation.
IBM makes no warranties or a representation with respect to the content hereof and specifically
disclaims any implied warranties of merchantability or fitness of any particular purpose.
IBM assumes no responsibility for any errors that may appear in this document. The information
contained in this document is subject to change without any notice. IBM reserves the right to
make any such changes without obligation to notify any person of such revision or changes. IBM
makes no commitment to keep the information contained herein up to date.
The results shown are based on specific workloads run in a controlled environment. The actual
throughput that any user will experience will vary considerably from these results. Therefore, no
assurance can be given that an individual user will achieve throughput equivalent to the
performance stated here.
Actual environmental costs and performance characteristics will vary depending on individual
customer configurations and conditions.
Consult your local IBM business contact for information on the product or services available in
your area.
Information about non-IBM products is obtained from the manufacturers of those products or
their published announcements. IBM has not tested those products and cannot confirm the
performance, compatibility, or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
1.5 Trademarks
IBM, the IBM logo, and ibm.com® are trademarks or registered trademarks of International
Business Machines Corporation in the United States, other countries, or both. If these and other
IBM trademarked terms are marked on their first occurrence in this information with a
trademark symbol (® or ™), these symbols indicate US registered or common law trademarks
owned by IBM at the time this information was published. Such trademarks may also be
registered or common law trademarks in other countries. A current list of IBM trademarks is
available on the web at “Copyright and trademark information” at
www.ibm.com/legal/us/en/copytrade.shtml.
SAP® , SAP® R/3, SAP® ERP, mySAP® , mySAP® .com, xApps, xApp, ABAP, BAPI, SAP®
NetWeaver and all
SAP® product and service names mentioned herein are trademarks or registered trademarks of
SAP® AG in Germany and in several other countries all over the world.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
8/168
Adobe is a registered trademark of Adobe Systems Incorporated in the United States, and/or
other countries.
InfiniBand is a registered trademark of the InfiniBand Trade Association.
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.
Windows and Microsoft are registered trademarks of Microsoft Corporation in the United
States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle
and/or its affiliates.
Unicode is a trademark of Unicode, Inc.
Other product and service names might be trademarks of IBM or other companies.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
9/168
2 Executive Summary
The objective of this proof-of-concept was to demonstrate to a large customer the value of
IBM’s proposed architecture based on DB2 pureScale® running SAP® TRBK on Intel servers and
IBM DS8K storage systems.
Designed for organizations that run online transaction processing (OLTP) applications on
distributed systems, IBM® DB2® pureScale™ offers clustering technology that helps deliver high
availability and exceptional scalability transparent to applications. DB2 pureScale® is
conceptually based on clustering technology from IBM DB2 for z/OS® and is available as an
option on IBM DB2 9.7 Enterprise Server Edition.
The Transaction Banking (TRBK) Benchmark used in this PoC, is divided into two business
scenarios. The first scenario, day processing, reflects usage by users and integrated systems that
generally occurs during the daytime. The second scenario, night processing, reflects account
balancing that generally occurs overnight in batch mode.
The main goals of this project were to achieve some specific performance figures that were
viewed by the customer as representative of their future production workload, while
demonstrating the high-availability of the solution. In a second stage, we covered some Disaster
Recovery scenarios to further demonstrate the strength and value of the solution for a
production system.
During several months of proof-of-concept, we went through the following phases:
- Environment setup and database growth up to 90 Mio customer accounts
- Performance tests based on on-line transaction processing (‘Day runs’) and batch
processing (‘Night runs’).
- Tests to ensure the scalability of the solution
- High Availability tests
- Reconfiguration of the environment to get ready for DR test
- Disaster Recovery Test campaign.
- Setup and configuration of new platform
- Certification Test campaign.
The challenge was to demonstrate that beside the HA and DR conditions, we could also achieve
very high throughput in terms of OLTP or batch process, up to 40Mio of postings processed per
hour for the “Day run” and 22.5Mio of accounts processed per hour for the “Night run’.
In terms of day processing, we achieved 56.9Mio postings/hour, 42% higher than the initial
target. Using the same workload, we also demonstrated a linear scalability from 1 to 4
members.
For the night processing, we managed to balance 90Mio accounts within 158 minutes which
correspond to a throughput of 38.4 Mio accounts/hour. For that scenario, we also clearly over-
achieve the initial target by 65%.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
10/168
In terms of high availability of the solution, whatever the type of failure encountered (member
failure, coupling facility failure, malloc bomb failure, Infiniband adapter loss, Infiniband switch
failure, … ), we achieved an outage time less than the 60 second target time, and usually much
lower.
Finally, through the various test cases executed during the DR Test campaign, we stressed the
DB2 pureScale® solution, and demonstrated that in case of Site Failure or Storage interconnect
failure, we were able to recover without much impact on the production activity.
We also evaluated successfully the impact of a Backup during online activity. And we tested the
Q-Replication as an extra option to ensure double security at the table level.
During this project, we overachieved our initial targets, and demonstrated that DB2 pureScale®
is capable of delivering excellent performance when running a very challenging workload in an
environment that effectively guarantees 100% availability, with the further guarantee of
supporting the required production recovery mechanism in the event of a complete site failure.
We ended this effort with the official Certification by SAP® of our metrics. We achieved 56.5M
postings to bank accounts processed per hour and demonstrated the ability to balance 90Mio
accounts during 4 hours as a proof of stability and requirement for the certification.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
11/168
3 Project Overview & Requirements
3.1 Objectives
4 main objectives have been assigned to this project from a technical stand point.
- Performance
- Scalability
- High Availability
- Disaster Recovery
Based on SAP® TRBK benchmark suite, the workload was based on generic data with 90 Mio
accounts loaded in the database as a starting point.
3.2 Tests scenario description
3.2.1 Day processing scenario
During the day processing scenario, the benchmark program simulates typical daily volumes of a
retail bank, including the quantity of payment transaction operations received from external
payment transaction systems, single real-time payment transactions received from ATM
machines, real-time account statements, as well as the quantity of account information read
from the system. The scenario was constructed to mirror a distribution of typical banking tasks.
Since the usual integration into the bank IT infrastructure takes place via real-time interfaces
(BAPIs) rather than SAP® GUI, the simulation consists of a sequence of BAPI calls.
The goal of that test was to achieve a throughput of 40Mio postings per hour.
3.2.2 Night processing scenario
In the night processing scenario, account balancing is performed at regular intervals to settle an
account. The benchmark simulates account balancing by calculating interest and charges for
each account across 20 value date days.
The goal of that test was to balance the 90 Mio accounts within 4 hours, which corresponds to a
throughput of 22.5Mio accounts per hour.
3.2.3 Scalability of the architecture
The goal of that test is to ensure that we can scale from 1 member to 4 members
3.2.4 High Availability
Many tests have been identified to ensure the High Availability of the global solution. They are
listed in the table below :
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
12/168
Test # Scenarios Customer requirement (outage time)
1 Member shutdown failure (dual member failure part 1) <= 60 seconds
2 Malloc bomb member (dual member failure part 2) <= 60 seconds
3 Primary CF shutdown failure <= 60 seconds
4 Secondary CF shutdown failure <= 60 seconds
5 Malloc bomb primary CF <= 60 seconds
6 Member IB adapter failure <= 60 seconds
7 Soft kill member <= 30 seconds
8 IB switch failure <= 60 seconds
3.2.5 Disaster Recovery
The Disaster recovery test campaign covered the following test cases:
Test# DR test case Description
1 Site Failure – storage replication TRBK Night workload can be completed on site2 in case of a failure on site1
(DWDM switched off) Based on Night Run.
2 Storage interconnect – catchup test Once the link for the storage interconnect is lost (10min delay)
monitor the performance and the time to resync. Based on Day run.
3 Database Q-Replication with 15 min replication delay
Maintain a shadow db with Qrep:
• Measure qrep impact on primary
• Demonstrate ability to keep up with primary
• Maintain shadow db 15 mins behind primary
Based on Night Run 4 Qrep + Storage replication combined Site failure test with impact of QRep
Based on Night Run
5 Backup duing on-line operations Evaluate impact on the throughput of a flashcopy on primary site or secondary
site. Based on Day Run
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
13/168
4 Architecture – system configuration
4.1 Overall layout
This is the target Architecture used for the Performance tests, the scalability tests and the high
availability tests. Please refer to the Disaster Recovery chapter for details about the architecture
used in the DR tests.
Hardware Involved
Machine type Quantity Role
x3850 x5 (Type 7145) 6 Database servers (DB2PS)
HS22 (Type 7870)
BladeCenter H
24
2
Application servers (SAP® )
DS8700 2 Database storage
DS5300 1 Benchmark results storage
Mellanox IB switches IS5030 2 Interconnect Infiniband network (provide HA)
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
14/168
4.2 Servers overview
4.2.1 Application servers - HS22
4.2.1.1 Technical characteristics
Element Quantity Description
Processor 2 Intel Xeon 4 Cores X5570 95W 2.93GHz/1333MHz/8MB L2
Memory 12
Express 4GB Dual Rank PC3-10600 CL9 ECC DDR3 VLP RDIMM
1333MHz
Internal disks 2 Express IBM 146 GB 2.5in SFF Slim-HS 15K 6Gbps SAS HDD
Additional cards 1 Qlogic 1 Gb Ethernet dual port CIO-v
Note: the internal disks have been configured as a RAID1 array.
Remark: Among all the HS22, one has been chosen to host the TRBK driver. This blade is also
responsible of collecting all the monitoring data (SAP® & system related) from all the machines.
Thus, this blade is attached to a DS5300 via a daughter card CFF-h Combo LAN +SAN Qlogic 8Gb.
4.2.1.2 Firmware levels
Module Version Release date
FW/BIOS P9E146A 03/01/2010 (1.07)
Diagnostics DSYT60K 02/25/2010 (3.01)
Blade Sys Mgmt Processor YUOO57H (1.10)
4.2.1.3 CPU information
Architecture: x86_64 Model: 26
Logical CPU(s): 16 Stepping: 5
Thread(s) per core: 2 CPU MHz: 2926.000
Core(s) per socket: 4 Virtualization: VT-x
CPU socket(s): 2 L1d cache: 32K
NUMA node(s): 2 L1i cache: 32K
Vendor ID: GenuineIntel L2 cache: 256K
CPU family: 6 L3 cache: 8192K
4.2.1.4 RAM configuration
HS22 are equipped with 12 DIMMs of 4GB DDR-3, resulting 48 GB total. All the slots are
populated, and the 2 nodes have an equivalent amount of memory. This is part of the Best
Practices recommended by IBM. Here is a dump of the Linux numactl command, which shows
the NUMA topology of the server.
available: 2 nodes (0-1)
node 0 size: 24196 MB
node 1 size: 24240 MB
node distances:
node 0 1
0: 10 21
1: 21 10
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
15/168
4.2.2 Database servers - x3850 x5
4.2.2.1 Technical characteristics
Element Quantity Description
Processor 4 Intel Xeon 8 cores X7560 130W 2.26 GHz/1066MHz/24MB L3
Memory 64
4GB (1x4GB, Quad Rankx8) PC3-8500 CL7 ECC DDR3 1066MHz
LP RDIMM
Internal disks 2 IBM 146 GB 10K SAS 2.5" SFF Slim-HS HDD
Additional cards
1 Emulex 10Gbps Ethernet Dual ports PCI-express
2 (1) Qlogic 8 Gb FC Dual ports PCI-express
1 (2) Mellanox 40 Gbps QDR Infiniband Dual ports PCI-Express
Note: the internal disks have been configured as a RAID1 array.
4.2.2.2 Firmware levels
Module Version Release date
IMM YUOO73K 06/16/2010
FPGA G0UD29C 06/16/2010
UEFI G0E122DUS 06/29/2010
DSA DSYT72C 06/16/2010
4.2.2.3 CPU information
Architecture: x86_64 Model: 46
Logical CPU(s): 64 Stepping: 6
Thread(s) per core: 2 CPU MHz: 2261.041
Core(s) per socket: 8 Virtualization: VT-x
CPU socket(s): 4 L1d cache: 32K
NUMA node(s): 4 L1i cache: 32K
Vendor ID: GenuineIntel L2 cache: 256K
CPU family: 6 L3 cache: 24576K
4.2.2.4 RAM configuration
All x3850 x5 are equipped with 8 Memory cards, each containing 8 DIMMs of 4GB DDR-3.
Overall, the amount of memory is 256 GB of DDR3 RAM. As all the slots are used, the memory is
equally split between the 4 NUMA nodes. The use of 64 DIMM (8 DIMMS per memory cards)
ensures that the memory throughput is optimal, as recommended in IBM Best practices.
Here is a dump of the Linux numactl command, which shows the NUMA topology of the server.
available: 4 nodes (0-3)
node 0 size: 64592 MB
node 1 size: 64640 MB
node 2 size: 64640 MB
node 3 size: 64640 MB
node distances:
node 0 1 2 3
0: 10 11 11 11
1: 11 10 11 11
2: 11 11 10 11
3: 11 11 11 10
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
16/168
4.2.2.5 uEFI Configuration
Here is an uEFI dump extract using the ASU (Advanced Setup Utility – tool provided by IBM).
The configuration has been done using the recommendations of Toronto & Intel Lab.
uEFI.OperatingMode=Performance Mode
uEFI.QuietBoot=Enable
uEFI.TurboModeEnable=Enable
uEFI.TurboBoost=Traditional
uEFI.ProcessorEistEnable=Disable
uEFI.ProcessorCcxEnable=Disable
uEFI.ProcessorC1eEnable=Disable
uEFI.HyperThreading=Enable
uEFI.EnableCoresInSbsp=All
uEFI.ExecuteDisableBit=Enable
uEFI.ProcessorVmxEnable=Enable
uEFI.ProcessorDataPrefetch=Enable
uEFI.DDRspeed=Max Performance
uEFI.CkeLowPolicy=Disable
uEFI.MirrorMigration=NonMirrored
uEFI.MapperPolicy=Closed
uEFI.SchedulerPolicy=Adaptive
uEFI.PatrolScrub=Disable
uEFI.PagePolicy=Closed
uEFI.MemoryScrambling=Disable
uEFI.Socket0Branch0SpareEn=Disable
uEFI.IdeMode=Compatibility mode
uEFI.ThunkSupport=Enable
uEFI.TpmEnable=Disable
uEFI.TpmStateUserSelection=Deactivate
uEFI.QPISpeed=Max Performance
4.2.2.6 I/O card layout for x3850 x5
Database servers require a suitable connection to the Storage Area Network (SAN) to access the
disks where the tables are stored. Moreover, DB2 pureScale® allows the different database
servers to communicate directly using RDMA, thanks to the use of Infiniband stack and uDAPL
upper protocol.
However, all the servers are not equivalent: among 6 servers, 4 are “members” (i.e. regular
database servers) and 2 are “coupling facilities” (used to manage shared data structures
between members – there are two CF’s for redundancy). Thus, these 2 kinds of servers have
different needs in terms of communication: that’s why 2 layouts have been designed for the PCI-
express cards.
Note: In both cases, the card used for the 10 Gb Ethernet is the Emulex card shipped with the
server, which is located in the slot 7.
Naming convention:
This outline illustrates the internal organization of the x3850 x5 components. The PCI slots are
represented on the right. The speed of these buses is also specified (x4 to x16).
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
17/168
For regular members:
Card type Quantity Slot number
Infiniband (HCA) 1 6
Fiber Channel (HBA) 2 3 – 4
10 Gb Ethernet 1 7 (default)
Explanation:
- Objective: priority is given to the IB bandwidth
- Assumption: 10 Gb Eth is not resource consuming
- Load balancing on the 2 I/O Hubs:
I/O Hub 1: IB and 10 Gb Eth
I/O Hub 2: 2x FC
- Advantage : IB can benefit from « full » bandwidth of an I/O Hub
- Drawback : 2 sockets must do a additional « hop » to access FC cards
For coupling facilities:
Card type Quantity Slot number
Infiniband (HCA) 4 3 – 4 – 5 - 6
Fiber Channel (HBA) 1 2
10 Gb Ethernet 1 7 (default)
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
18/168
Explanation:
- Hypothesis 1 : CF might be able to use 4 active IB cards
- Hypothesis 2 : CF will not have intensive SAN I/O
- All IB cards have homogeneous configuration (PCI-e 8x slots)
- Two IB on each I/O Hub with another type of interface
- Load balancing on the 2 I/O Hubs:
I/O Hub 1: IB0, IB2 and FC
I/O Hub 2: IB1, IB3 and 10 Gb Eth
- Advantage : maximize bandwidth for both IB cards
Note: the slot 1 (16x) has not been used because the Mellanox card is not currently supporting
16x speed. Thus, to have a homogeneous configuration between all the cards, all the x8 slots
have been used for the IB cards. The FC card (Qlogic) has been placed in slot 2, as this slot is
compatible with any 8x card (IBM documentation) and that the I/O on the CF is very low.
4.3 Storage overview
The storage subsystem is made of 2 DS8700 with for each DS8700 the following characteristics:
• 2 frames
• 256 GB Processor Memory (4-Way)
• 384 disks of 450 GB 15K FC
• 16 4Gb SW FCP ports connections to the SAN.
Each server (member or coupling facility) have access to the two DS8700 through a SAN (Storage
Area Network)
Some description of the two DS8700 is detailed below:
dscli showsi IBM.2107-75TX861 -cfg C:\Progra~1\IBM\dscli\profile\DB2_PRA_Pure_1.profile
Date/Time: 08 December 2010 16:25:50 CET IBM DSCLI Version: 6.5.1.203 DS: IBM.2107-
75TX861
ID IBM.2107-75TX861
Storage Unit IBM.2107-75TX860
Model 941
WWNN 500507630AFFC2BC
Signature 4d2b-1929-cad1-dee3
State Online
ESSNet Enabled
Volume Group V0
os400Serial 2BC
NVS Memory 8.0 GB
Cache Memory 235.9 GB
Processor Memory 253.4 GB
MTS IBM.2421-75TX860
numegsupported 0
ETAutoMode on
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
19/168
ETMonitor all
dscli showsi IBM.2107-75TX901 -cfg C:\Progra~1\IBM\dscli\profile\DB2_PRA_pure_2.profile
Date/Time: 08 December 2010 18:06:06 CET IBM DSCLI Version: 6.5.1.203 DS: IBM.2107-
75TX901
ID IBM.2107-75TX901
Storage Unit IBM.2107-75TX900
Model 941
WWNN 500507630AFFC2C1
Signature 8c7d-cd9d-1e17-ae88
State Online
ESSNet Enabled
Volume Group V0
os400Serial 2C1
NVS Memory 8.0 GB
Cache Memory 235.9 GB
Processor Memory 253.4 GB
MTS IBM.2421-75TX900
numegsupported 0
ETAutoMode on
ETMonitor all
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
20/168
SAN
x16
x16
x2 x4 x4 x4 x4x2
X3850CF0
X3850CF1
X3850M0
X3850M1
X3850M2
X3850M3
2 X 384 FC
PRA110.15.8.5375TX860WWN=2bc
PRA210.15.8.5475TX900WWN=2c1
db2PS-member010.3.67. 500x2100001b3283b04b0x2101001b32a3b04b0x21000024ff28538e0x21000024ff28538f
db2PS-member110.3.67. 510x2100001b3283d54c0x2101001b32a3d54c0x21000024ff2087100x21000024ff208711
db2PS-member210.3.67. 520x21000024ff2087180x21000024ff2087190x21000024ff2087ae0x21000024ff2087af
db2PS-member310.3.67. 530x21000024ff2087480x21000024ff2087490x21000024ff20872a0x21000024ff20872b
db2PS-cf010.3.67. 540x21000024ff2086f80x21000024ff2086f9
db2PS-cf110.3.67. 550x21000024ff2087400x21000024ff208741
DS8700 DS8700
Overview of the storage configuration
DA 2
DA 0
DA 6
DA 4
DA 7
DA 5
6PS A0
6PS A2
6PS A1
6PS A3
6PS A8
6PS A10
6PS A9
6PS A11
7P A4 7P A5
7P A77P A6
7P A12
EXP 1
7P A13
7P A14 7P A15
6PS A16 6PS A17
6PS A18 6PS A19
7P A20 7P A21
7P A22 7P A23
6PS A24 6PS A25
6PS A26 6PS A27
7P A28 7P A29
7P A30 7P A31
6PS A32 6PS A33
6PS A34 6PS A35
7P A36 7P A37
7P A38 7P A39
6PS A40 6PS A41
6PS A42 6PS A43
7P A44 7P A45
7P A46 7P A47
BASE
DS8000 arrays with 384 disks FC 450 GB
48 FC arrays
2374 GB each
24 arrays for pool1
24 arrays for pool2
All arrays for data and log
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
21/168
The configuration of the RAID5 arrays of one DS8700 with 384 FC disks. There are 16 RAID5
arrays in the first frame and 32 in the second frame.
4.3.1 Assumptions for DB2 / GPFS / Storage design
� Assumptions:
– Shared pool of FC disks for Data/Index/Temp and logs
– Storage Pool Striping (1 GB stripe size) for Data/Index/Temp and logs
– No GPFS striping (each file system is defined / mapped on only one LUN).
– Easy configuration with only one level of stripping at DS8700 level.
� Design:
– Each file system (SAP® data1 to SAP® adata 16) is mapped on one LUN of 388 GB
size
– Each file system (log0 to log3) is mapped on one LUN of 2 TB size.
– Each LUN is stripped on 24 RAID5 arrays (192 FC disks) with a block size of 1 GB
– LUN is created with the option Rotate Extent in an Extent Pool
– 2 Extents Pool per DS8700 which consist in 24 FC RAID5 arrays each
– Each RAID5 array has 6 datas, 1 Parity and 1 spare or 7 datas and 1 parity disks.
– A total of 768 disks for the two DS8700.
DA 2
DA 0
DA 6
DA 4
DA 7
DA 5
6PS A0
6PS A2
6PS A1
6PS A3
6PS A8
6PS A10
6PS A9
6PS A11
7P A4 7P A5
7P A77P A6
7P A12
EXP 1
7P A13
7P A14 7P A15
6PS A16 6PS A17
6PS A18 6PS A19
7P A20 7P A21
7P A22 7P A23
6PS A24 6PS A25
6PS A26 6PS A27
7P A28 7P A29
7P A30 7P A31
6PS A32 6PS A33
6PS A34 6PS A35
7P A36 7P A37
7P A38 7P A39
6PS A40 6PS A41
6PS A42 6PS A43
7P A44 7P A45
7P A46 7P A47
BASE
Extent pool mapping with DS8000 RAID5 arrays
48 FC arrays
2374 GB each
24 arrays for extent pool1
24 arrays for extent pool2
All for data and log
Pool 2Pool 1
Pool 1 Pool 2
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
22/168
An extent pool is made of 24 RAID5 arrays.
DA 2
DA 0
DA 6
DA 4
DA 7
DA 5
6PS A0
6PS A2
6PS A1
6PS A3
6PS A8
6PS A10
6PS A9
6PS A11
7P A4 7P A5
7P A77P A6
7P A12
EXP 1
7P A13
7P A14 7P A15
6PS A16 6PS A17
6PS A18 6PS A19
7P A20 7P A21
7P A22 7P A23
6PS A24 6PS A25
6PS A26 6PS A27
7P A28 7P A29
7P A30 7P A31
6PS A32 6PS A33
6PS A34 6PS A35
7P A36 7P A37
7P A38 7P A39
6PS A40 6PS A41
6PS A42 6PS A43
7P A44 7P A45
7P A46 7P A47
BASE
LUN definition in the extent pools
Pool 2Pool 1
Pool 1 Pool 2
D1
D2
D3
D4
L1
D5
D6
D7
D8
L2
Per Extent Pool:
4 LUNs of 388 GB for Data/index/Temp
1 LUN of 2 TB for logs
LUN is created with the option Rotate Extent in an Extent Pool. For each extent pool:
• 4 LUNs of 388 GB for Data/index/Temp
• 1 LUN of 2 TB for logs
For example the LUN 2000 is distributed on 24 different RAID5 arrays (R0 to R46 evenly).
This LUN of 388 GB size consists in 16 or 17 extents of 1 GB for each array.
dscli showfbvol -rank 2000 -cfg C:\Progra~1\IBM\dscli\profile\DB2_PRA_pure_2.profile
Date/Time: 08 December 2010 19:52:46 CET IBM DSCLI Version: 6.5.1.203 DS: IBM.2107-75TX901
Name PROD2_XXXX
ID 2000
accstate Online
datastate Normal
configstate Normal
deviceMTM 2107-900
datatype FB 512
addrgrp 2
extpool P0
exts 388
captype DS
cap (2^30B) 388.0
cap (10^9B) -
cap (blocks) 813694976
volgrp V0,V1
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
23/168
ranks 24
dbexts 0
sam Standard
repcapalloc -
eam rotateexts
reqcap (blocks) 813694976
realextents 388
virtualextents 0
migrating 0
perfgrp PG0
migratingfrom -
==============Rank extents==============rank extents============
R0 16
R2 16
R4 16
R6 16
R8 16
R10 16
R12 17
R14 17
R16 17
R18 17
R20 16
R22 16
R24 16
R26 16
R28 16
R30 16
R32 16
R34 16
R36 16
R38 16
R40 16
R42 16
R44 16
R46 16
4.3.2 LUN and file system for DB2 with 768 disks FC
File system capacity techno NumDisks Num
LUNs
LUN size Storage Pool
Striping
/db2/D3D/ 2000 GB FC 192 1 2000 GB Yes
/db2/D3D_log0
/db2/D3D_log1
/db2/D3D_log2
/db2/D3D_log3
2000 GB
2000 GB
2000 GB
2000 GB
FC
FC
FC
FC
192 1
1
1
1
2000 GB Yes
/db2/D3D_SAP®
data1
…………………..
388 GB
388 GB
FC
FC
192
192
1
1
388 GB
388 GB
Yes
Yes
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
24/168
/db2/D3D_SAP®
data16
4.3.3 Detailed of the LUNs configuration
SAP® fs DS8700 ID + LUN ID EP DS8700 cap
SAP® data1 2bc0000000000002100) P1 PRA1 388 GB
SAP® data2 2c10000000000002100) P1 PRA2 388 GB
SAP® data3 2c10000000000002003) P0 PRA2 388 GB
SAP® data4 2c10000000000002002) P0 PRA2 388 GB
SAP® data5 2c10000000000002001) P0 PRA2 388 GB
SAP® data6 2bc0000000000002103) P1 PRA1 388 GB
SAP® data7 2c10000000000002000) P0 PRA2 388 GB
SAP® data8 2bc0000000000002102) P1 PRA1 388 GB
SAP® data9 2bc0000000000002101) P1 PRA1 388 GB
SAP® data10 2bc0000000000002003) P0 PRA1 388 GB
SAP® data11 2bc0000000000002000) P0 PRA1 388 GB
SAP® data12 2bc0000000000002002) P0 PRA1 388 GB
SAP® data13 2c10000000000002103) P1 PRA2 388 GB
SAP® data14 2bc0000000000002001) P0 PRA1 388 GB
SAP® data15 2c10000000000002102) P1 PRA2 388 GB
SAP® data16 2c10000000000002101) P1 PRA2 388 GB
SAP® fs DS8700 ID + LUN ID EP DS8700 cap
log0 2c10000000000002004 P0 PRA2 2000 GB
log1 2c10000000000002104 P1 PRA2 2000 GB
log2 2bc0000000000002004 P0 PRA1 2000 GB
log3 2bc0000000000002104 P1 PRA1 2000 GB
tiebreaker 2bc0000000000002005 P0 PRA1 2000 GB
db2fs1 2c10000000000002105 P1 PRA2 2000 GB
notused 2c10000000000001005
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
25/168
db2d3d 2c10000000000002005 P0 PRA2 2000 GB
4.4 Network fabric overview
4.4.1 Infiniband network
4.4.1.1 Topology
For HA test purposes, 2 Mellanox switches (IS5030) have been set up for providing switch failure
tolerance. Here is the topology:
In case of a switch failure, half on the cluster is brought offline.
On the outline above, 2 types of cables have been used:
- From member/switch and coupling/switch : 5m Copper cables QSFP
- Between switches : 10m Optic cables QSFP
Having four links between the 2 switches ensure an optimal throughput, as each coupling facility
has 4 HCAs.
4.4.1.2 Subnet Manager configuration
Among the 2 switches, one has been chosen to act as the Subnet Manager. It is keeping the
identifier of each HCAs and is aware of the global Infiniband topology. The switch running as a
Subnet Manager is a licensed Mellanox feature, and has been activated in mlx-switch1. The
other switch (mlx-switch2) is considering as passive, as it is only routing the packets without
having the responsibility of maintaining the IB network.
4.4.1.3 Physical connections
Here is the list of the ports that are used on the 2 switches:
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
26/168
Port number mlx-switch-1 mlx-switch-2
1 DB2PS-member0 HCA-1 DB2PS-member1 HCA-1
3 DB2PS-member2 HCA-1 DB2PS-member3 HCA-1
7 DB2PS-cf0 HCA-2 DB2PS-cf1 HCA-2
9 DB2PS-cf0 HCA-3 DB2PS-cf1 HCA-3
11 DB2PS-cf0 HCA-4 DB2PS-cf1 HCA-1
13 DB2PS-cf0 HCA-1 DB2PS-cf1 HCA-4
19 mlx-switch-2:IS5030 mlx-switch-1:IS5030
21 mlx-switch-2:IS5030 mlx-switch-1:IS5030
23 mlx-switch-2:IS5030 mlx-switch-1:IS5030
25 mlx-switch-2:IS5030 mlx-switch-1:IS5030
4.4.2 Fiber Channel network
Each database server is connected to the storage systems (DS8700) via a storage area network.
This network is created using dedicated SAN switches in the infrastructure. As described in the
server part, each member has 4 connections as they perform intensive I/O while coupling
facilities have only 2 links on the SAN.
Another smaller SAN connection is used between the SAP® injector and a DS5000 system (peer-
to-peer connection), to store all the data coming from the run.
4.4.3 Ethernet network
4 networks are used to ensure the communication through the DB2 pureScale® architecture
components:
Network Type Machines
connected
Subnet MTU
Administration 1 Gb Ethernet All 10.3.67.0/24 1500
Application 1 Gb Ethernet SAP® servers 10.1.1.0/24 1500
Database 10 Gb Ethernet SAP® + DB2
servers
10.1.2.0/24 9000
Interconnect 10 Gb QDR
Infiniband
DB2 servers 10.9.9.0/24
(IPoIB)
65520
The administration network is reachable from the Internet thanks to a Virtual Private Network.
All other networks are isolated and dedicated to this architecture.
Remark: the gateway of the 10.3.67.0/24 network is also used for NTP (Network Time Protocol)
synchronization.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
27/168
For more information concerning the IP addressing map, please refer to the file pureScale – IP
Addressing.xls
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
28/168
4.5 Operating Systems
4.5.1 Suse Enterprise Linux 10 SP3
All servers are using Suse Enterprise Linux (SLES) 10 SP3. Kernel has been upgraded on the DB2
servers from the original version to allow the newest OFED driver (for Infiniband) to be installed.
Machine Type Kernel version
Database Servers (x3850x5) 2.6.16.60-0.66.1-smp (x86_64)
Application Servers (HS22) 2.6.16.60-0.54.5-smp (x86_64)
4.5.2 I/O device drivers
4.5.2.1 Multipathing driver
The multipathing driver used for this benchmark is DM-MPIO that comes with the SLES 10 SP3
distribution. The configuration file (multipath.conf) follows IBM support recommendations. The
policy is round-robin based (rr_min_io = 100), and paths are grouped by priority. Here is an
extract of the multipath.conf file:
defaults {
polling_interval 30
failback immediate
no_path_retry 5
rr_min_io 100
path_checker tur
# features "1 queue_if_no_path"
user_friendly_names yes
# These are the default settings for 2107 (IBM DS8000)
# Uncomment them if needed on this system
device {
vendor "IBM"
product "2107900"
path_grouping_policy group_by_serial
}
4.5.2.2 Linux IO scheduler parameter
Here are Linux parameters concerning the IO scheduling, for each accessible disk:
- IO Queue size : 128
- Scheduler used : Completely Fair Queuing
At a lower level, the use of NCQ (Native Command Queuing) also comes with sets of parameters
- NCQ queue size : 32
For deeper details about the LUN mapping please refer to multipath command display or
storage configuration documentation.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
29/168
4.5.2.3 Infiniband driver
For the database servers, the OFED stack is used to provide the driver for Infiniband HCA.
On one node, here is the result of the infiniband driver self test command:
---- Performing Infiniband HCA Self Test ----
Number of HCAs Detected ................ 1
PCI Device Check ....................... PASS
Kernel Arch ............................ x86_64
Host Driver Version .................... OFED-1.5.2-20100824-0600: 1.5.2-2.6.16.60_0.66.1_smp
Host Driver RPM Check .................. PASS
HCA Firmware on HCA #0 ................. v2.7.0
Host Driver Initialization ............. PASS
Number of HCA Ports Active ............. 1
Port State of Port #1 on HCA #0 ........ UP 4X QDR
Port State of Port #2 on HCA #0 ........ DOWN
Error Counter Check on HCA #0 .......... PASS
Kernel Syslog Check .................... PASS
------------------ DONE ---------------------
4.5.2.4 Major on-board drivers (including Emulex and firmware)
Here are some of the major drivers used in the database servers, including the 10 Gb Ethernet
Emulex NIC. The updates have been done and checked using the UpdateXpress pack provided
on the IBM support site.
BC-Emulex Device Driver 2.101.411.0 for 10Gb CNA for SLES10
Installed Version : 2.101.411.0
Broadcom bnx2 sles10 driver
Installed Version : 2.0.1-suse
IBM and LSI Basic or Integrated RAID SAS Controller Driver
Installed Version : 3.04.07_suse
Installed Version : 3.04.07_suse
Installed Version : 3.04.07_suse
Installed Version : 3.04.07_suse
IBM uEFI Flash Update
Installed Version : 1.23 (G0E122D)
IBM Dynamic System Analysis
Installed Version : 3.13 (DSYT72C)
IBM HBA/LSI Onboard 3Gb SAS/SATA BIOS/FW/UEFI Update
Installed Version : 1.30.00.00
IBM Online SAS/SATA Hard Disk Drive Update Program
Installed Version : B536
Installed Version : B536
Online Broadcom NetXtreme and NetXtreme II Firmware Utility
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
30/168
Installed Version : 5.2.2
Installed Version : 5.2.2
Integrated Management Module Update
Installed Version : 1.14 (YUOO73K)
IBM x3850 / x3950 X5 FPGA Flash Update
Installed Version : 83.0.56.1
4.5.3 Settings tuned
4.5.3.1 Kernel parameters
For performance improvement, the use of several HCA (Host Channel Adapter – Infiniband
Adapter) has been recommended on the coupling facilities. As the uDAPL needs IPoIB
configuration for each HCA, several IP addresses from the same subnet are utilized on the same
physical machine.
The Linux kernel doesn’t handle this configuration by default. Thus, the kernel parameter
net.ipv4.conf.all.arp_ignore has been modified to the value “2”, as recommended in
the OFED stack documentation.
4.6 DB2 pureScale®
4.6.1 DB2 concepts DB2 pureScale® is a DB2 feature that allows the ability to scale out a database over a
number of servers in an “active-active” configuration and delivers a high level of availability and
scalability. DB2 pureScale® offers virtually unlimited capacity for growth as servers can be
dynamically added to a DB2 pureScale® instance with no disruption to cluster operation. Upon
adding additional servers, DB2 pureScale® ’s load balancing logic will automatically begin to
transparently reroute transactions to make use of the additional resources. Applications require
no knowledge of the underlying database system topology and do not require any modifications
to run on DB2 pureScale® . Thus DB2 pureScale® offers true application transparency.
Applications running on DB2 pureScale® need not be “cluster aware” in order to scale, as can be
the case with other scale-out architectures that rely on a heavy network-based messaging
infrastructure for sharing data in a cluster. As we describe below, by exploiting the latest
network and hardware architectures as well as a centralized server for locking and caching, DB2
pureScale® is able to transparently scale applications. Lastly, DB2 pureScale® features
continuous availability by being able to tolerate multiple component failures while continuing to
provide full access to data that does not need recovery. The following sections describe the
main components that make up a DB2 pureScale® instance.
A DB2 pureScale® instance consists of one or more DB2 servers (referred to as “members”)
where each member runs a full DB2 engine with its own buffer pools, agent threads, log files
and so on. DB2 pureScale® is a shared-data distributed database architecture meaning each
member has access to the same data for read and write operations. Synchronizing access to this
data is required for database consistency and being able to do so efficiently is essential for good
database performance. The solution for ensuring database consistency while maintaining high
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
31/168
levels of performance in DB2 pureScale® is derived from DB2 for z/OS Parallel Sysplex, a shared-
data DBMS known for its exceptional availability and scalability.
DB2 for z/OS Parallel Sysplex achieves the recognition it does in large part due to its use of the
“Coupling Facility”. The Coupling Facility provides a centralized location for managing locks as
well as a global shared cache for data. Part of every DB2 pureScale® instance is the pureScale
Cluster Caching Facility (or CF) server which provides the same centralized locking and global
shared cache architecture. Whenever SQL processing on a member requires a lock such as for
updating a row on a page, it will request the lock from the CF. The CF will determine if the lock is
available and if so, the lock will be granted and execution can continue on the member.
Requests to the CF are also made when a member wants to read a page from the CF into its
local buffer pool (LBP) or commit changes to a page. During a read request, if the CF has the
page cached, it will send the page to the member, thus saving the member from having to read
the page from disk. When the CF receives a page write request, the modified page is sent to the
CF as part of the request and is cached in the Group Buffer Pool (GBP). At this point the CF has
the most recent version of the page and proceeds to “invalidate” (mark as stale) any other
copies of the page that exist in the LBPs of other members. To eliminate the CF as a single point
of failure, the DB2 pureScale® instance is generally configured to use two CFs. In a two CF
configuration, all requests are sent to the CF designated as being the “primary CF” and are also
synchronously duplexed to the “secondary CF”. In doing so, the secondary CF can quickly
takeover as the primary in case of a primary CF failure. Optimum performance requires low CF
response times, a large part of which is the cost of communicating with the CF. This
communication overhead is minimized through the use of a high performance network
architecture.
4.6.2 Integrated Technologies
The CF is connected to each member in the DB2 pureScale® instance via a low latency, high
speed Remote Direct Memory Access (RDMA) network. InfiniBand and 10 Gigabit Ethernet
support interrupt free RDMA, which plays a critical role in the scalability of DB2 pureScale® .
With RDMA, each member is able to directly access memory on the CF and the CF is able to
directly access memory on each member, without consuming any CPU at the supplying side. For
example, during commit processing all the pages that were modified during the transaction will
be sent to the CF. Upon receiving these pages, the CF will cache them and determine which (if
any) of the other members have (now stale) copies of these pages in their local buffer pools.
Once this information is obtained, the CF will directly access the pages on the remote member
machines via RDMA and mark them as “invalid”. The ability to mark these pages as stale with no
interruption to the remote member is critical for the scalability of write intensive applications.
All servers in a DB2 pureScale® instance are connected to the same disk storage via a Storage
Area Network (SAN). The IBM General Parallel File System (GPFS) is used to provide access to
the data with high scalability, high availability and seamless capacity expansion. GPFS is shipped
as part of DB2 pureScale® and is installed and configured seamlessly during DB2 pureScale®
installation. The db2cluster command is used to create and manage your GPFS file systems
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
32/168
and will automatically configure GPFS with known best practices for settings such as file system
block size and file system cache size.
Ensuring high availability in a clustered environment like DB2 pureScale® where there are
dependencies on various resources is non-trivial. DB2 pureScale® uses IBM Tivoli Systems
Automation (TSA) and IBM Reliable Scalable Cluster Technology (RSCT) to handle the
automation of failure detection and recovery. With the ability to provide continuous high
availability and reduced duration of service disruptions, TSA and RSCT offer a way to monitor the
state of each of the components in a DB2 pureScale® environment. Upon failure detection, TSA
and RSCT will take the appropriate action to bring the failed component back to an operational
state. Examples of actions that could be taken include remounting file systems, restarting a DB2
member and rebooting a server. TSA and RSCT are shipped as part of DB2 pureScale® and are
installed and configured during DB2 pureScale® installation. Easy post installation management
of TSA and RSCT can be done through the db2cluster command.
The following diagram illustrates the topology of a DB2 pureScale® instance with two DB2
members and two DB2 pureScale® CF servers. One CF server is designated as primary (P) and the
other as secondary (S).
DB2Member 0
GPFS
TSARSCT
LBP
DB2Engine
DB2Member 1
GPFS
TSARSCT
pureScaleCF (S)
GPFS
TSARSCT
SwitchSwitch
Storage Area Network
RDMA Network
Disk Storage System
pureScaleCF (P)
GPFS
TSARSCT
Local AreaNetwork
EthernetNetwork
LBP
DB2Engine
CF Server
GBP
Locks
CF Server
GBP
Locks
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
33/168
4.7 SAP
4.7.1 TRBK overview
The Transaction Banking Benchmark (TRBK) reflects standard business scenarios within the SAP®
Banking 7.0 application and is used to measure
all performance relevant parameters.
It is divided into two different types of workload. The first scenario, day processing,
reflects usage by users and integrated systems that generally occurs during the
daytime. The second scenario, night processing, reflects account balancing that
generally occurs overnight in batch mode.
Day Processing:
The first scenario, day processing, reflects usage by users and integrated
systems that generally occurs during the daytime.
In this scenario multiple user logins are simulated and in the “High Load Phase” a
defined number of users are working concurrently on the system.
The figure below shows the workload in day processing over time:
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
34/168
Night Processing
In the night processing scenario, account balancing is performed at in order to settle an account.
The benchmark simulates account balancing by calculating interest and charges for each account
across 20 value date days.
In the night processing scenario, a defined number of batch processes are running in parallel
and the total amount of work (accounts to be balanced) is divided into packages that are
processed separately by the several work processes of the SAP® application servers.
4.7.2 Benchmark Configuration
Installed Software Components
Here is a list of prerequisite SAP® software components and their support package levels that
have been installed to build the SAP® Banking 7.0 System that has been used for this
benchmark:
SAP® Kernel Release: SAP® Kernel 711 Patch Level 80
Additionally the following SAP® notes have to be applied (SNOTE)
1441931 BW extraction from Bank analyzer taking long time for custom
1449022 Memory leak in contract
1449296 Performance: BAdI for BW extraction
1452459 Reordering the number assignment in benchmark tool
1452640 Performance improvement in mass generation/account migration
1461898 BCST: Synchronization of global buffer missing
1478386 Error during totals update
and manually 494171 Section to adapt DB Indexes on DFKKCOH and DFKKCOHI
3 tier SAP® Layout
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
35/168
The figure below shows the over all topology and architecture of the
SAP® TRBK benchmark environment:
4.7.3 System Installation
The following chapter reflects the procedure that was performed in this POC due to special
situations/restrictions (e.g. hardware availability). Usually the GPFS file systems are created first
and then the system installation is performed directly on the target hardware.
For a description of the SAP® recommended general procedure to migrate an existing DB2 9.7
system to DB2 pureScale®, see also the following SDN article :
http://www.sdn.SAP® .com/irj/scn/go/portal/prtroot/docs/library/uuid/10517fd9-e2bc-2d10-
cfa1-b2c182149798?QuickLink=index&overridelayout=true
The steps that were performed to build the TRBK System on DB2 pureScale® for the POC project
are enumerated below
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
36/168
4.7.3.1 Build SAP® TRBK system (in Walldorf)
The SAP® system with all required components was installed and prepared
in Walldorf according to the listed requirements in 4.2.7.
4.7.3.2 Build DB2 9.7 System on preliminary hardware (SAP® system copy)
The benchmark system was created by performing a SAP® system copy
with the backup/restore method on a preliminary hardware so be able to
start the data generation process as soon as possible. The final hardware was
not yet available at that point in time.
4.7.3.3 Data generation (90 Million accounts)
The TRBK Master data was generated in a staged approach. The customers
request was to generate 90 Million accounts overall. Here are the steps that
describe the data generation process:
a. generate 10 Million accounts
b. perform a “runstats with detailed indexes all” on tables BCA_CONTRACT,
BCA_CNSP_ACCT, BCA_PAYMITEM, BCA_COUNTER, , BCA_TRANSFIG
c. backup the database
d. generate another 10 Million accounts
e. backup again
f. generate 50 Million accounts (over weekend)
g. backup again
h. generate remaining 20 Million accounts
i. perform full runstats on all tables
j. final backup
k. check number of rows for the following tables (system should have 90 Million
accounts generated)
90 Million records in BCA_CONTRACT - 1 per account
90 Million records in BCA_CNSP_ACCT - 1 per account
1800 Million records in BCA_PAYMITEM - 20 per account
1800 Million records in BCA_COUNTER - 20 per account
1800 Million records in BCA_TRANSFIG - 20 per account
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
37/168
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
38/168
4.7.3.4 Create system on final hardware with DB2 9.7
After the target hardware was available, the database server of the SAP®
system was installed on the target hardware by a SAP® system copy with
backup/restore. To be able to migrate that system later to DB2 pureScale®,
the container paths have to exist on GPFS file systems. Therefore, on the
respective disks on the DS8700 storage system 21 GPFS file systems were
created:
/db2/D3D database directory
/db2/D3D_log_0 logfiles member 0
/db2/D3D_log_1 logfiles member 1
/db2/D3D_log_2 logfiles member 2
/db2/D3D_log_3 logfiles member 3
/db2/D3D_SAP® data1 |
/db2/D3D_SAP® data2 |
/db2/D3D_SAP® data3 |
/db2/D3D_SAP® data4 |
/db2/D3D_SAP® data5 |
/db2/D3D_SAP® data6 |
/db2/D3D_SAP® data7 |
/db2/D3D_SAP® data8 |----- 16 SAP® data paths
/db2/D3D_SAP® data9 |
/db2/D3D_SAP® data10 |
/db2/D3D_SAP® data11 |
/db2/D3D_SAP® data12 |
/db2/D3D_SAP® data13 |
/db2/D3D_SAP® data14 |
/db2/D3D_SAP® data15 |
/db2/D3D_SAP® data16 |
Two additional disks of the shared DS8700 storage were used for the
following purposes:
1 device as instance shared device “INSTANCE_SHARED_DEV”
1 device as tiebreaker “TIEBREAKER_DEV”
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
39/168
4.7.3.5 Prepare GPFS file systems
First DB2 pureScale® software was installed on the member 0 of target hardware:
db2_install –p ese_dsf –b /opt/sd
The instance shared device INSTANCE_SHARED_DEV was prepared by the following
command:
/opt/sd/instance/db2cluster_prepare –instance_shared_dev
/dev/<devicefilename of instance_shared_dev>
All GPFS filesystems (see 4.7.3.4) were created – here is an example
db2cluster –cfs –create –filesystem db2d3d
–disk /dev/<devicefilename> – mount /db2/D3D
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
40/168
4.7.3.6 SAP® system copy with backup/restore
The database server of the SAP® system was then installed on the host which
became member 0 later by performing a SAP® system copy using the backup/
restore method. The DB2 9.7 database was created with automatic storage
using the storage paths listed in 4.7.3.4.
4.7.3.7 Migrate to DB2 pureScale® with 1 member and 1 CF
Here is a description of what has to be done to migrate the existing DB2 9.7
database to DB2 pureScale® . This step consists of 2 parts, first migrating to
a DB2 9.8 ESE instance and finally change the instance type to SD (Shared
Data).
a. Migrating to a DB2 9.8 ESE instance
as db2d3d
db2 update dbm cfg using numdb 1
/opt/sd/bin/db2checkSD d3d => no problems reported
db2stop
as root
/opt/sd/instance/db2iupgrade –d –k db2d3d
as db2d3d
db2start
db2 upgrade db d3d
db2 connect to d3d
db2 terminate
db2stop
b. Change instance type to sd (data sharing instance) by performing
an instance update:
as root
/opt/sd/instance/db2iupdt –d –tbdev <device file for
tiebreaker> -cf DB2PS-cf0:DB2PS-cf0-ib0 –m DB2PS-
member0:DB2PS-member0-ib0 db2d3d
4.7.3.8 Add 1 additional CF and 3 additional members
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
41/168
as root
# add second CF
/opt/sd/instance/db2iupdt -d -add -cf DB2PS-cf1:DB2PS-
cf1-ib0 db2d3d
# add member1
/opt/sd/instance/db2iupdt -d -add -m DB2PS-
member1:DB2PS-member1-ib0 db2d3d
# add member2
/opt/sd/instance/db2iupdt -d -add -m DB2PS-
member2:DB2PS-member2-ib0 db2d3d
# add member3
/opt/sd/instance/db2iupdt -d -add -m DB2PS-
member3:DB2PS-member3-ib0 db2d3d
4.7.3.9 Backup final configuration
After some backup and restore tuning, the time for backup and restore could
be reduced significantly (the restore time for example was reduced from
~5 hours to ~2 ¼ hours).
Here is an example of the tuned restore command:
db2 RESTORE DATABASE D3D FROM /db_backup_10, /DB_BACKUP_11,
/DB_BACKUP12, /DB_BACKUP13 TAKEN AT 20101108143425 ON
/db2/D3D_SAP® data1, /db2/D3D_SAP® data2, /db2/D3D_SAP® data3,
/db2/D3D_SAP® data4, /db2/D3D_SAP® data5, /db2/D3D_SAP® data6,
/db2/D3D_SAP® data7, /db2/D3D_SAP® data8, /db2/D3D_SAP® data9,
/db2/D3D_SAP® data10, /db2/D3D_SAP® data11, /db2/D3D_SAP® data12,
/db2/D3D_SAP® data13, /db2/D3D_SAP® data14, /db2/D3D_SAP® data15,
/db2/D3D_SAP® data16 DBPATH ON /db2/D3D WITH 128 BUFFERS
BUFFER 16384 PARALLELISM 32
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
42/168
Here is an example for the backup command:
db2 BACKUP DATABASE D3D to /db_backup_10, /DB_BACKUP_11,
/DB_BACKUP12, /DB_BACKUP13 WITH 128 BUFFERS
BUFFER 16384 PARALLELISM 32COMPRESS
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
43/168
5 Test Results : Performance, Scalability & Availability
The TRBK benchmark used for the performance tests during the PoC consists of 2 distinct
phases.
The Day phase is a simulation of the creation of payment items against accounts such as might
occur from, for example, ATM’s during the normal day time operation of a transactional banking
system. This workload is simulated by a configurable number of users running a 15 step script of
transactions a configurable number of times. The accounts against which each payment items
are processed are generated randomly. This means that the database activity generated by the
Day run hits values across the entire range of accounts in the database randomly.
The Night phase is a simulation of the balancing of all accounts in the system. As initially loaded
the database contains 20 payment items per account, i.e. for this PoC, the total number of
payment items reconciled was 1.8 billion. The workload consists of a configurable number of
SAP® batch processes that work on sequential ranges of accounts until the entire range is
balanced. The database activity generated by the workload is sequential in nature.
The follow-up tests (scalability, availability and Disaster recovery) were either based on the Day
or the Night workload.
5.1 Overall test configuration
5.2 Performance test results were performed using 4 DB2 pureScale® members, 2 CF’s
(primary and secondary), and 1 enqueue server/CI and 20 application servers.
5.3 TRBK Day Run
The high water mark Day run was performed using 11000 benchmark users running on 20
application server instances, each with 120 dialog work processes. TRBK allows the configuration
of benchmark think time, i.e. a time delay between transaction steps of the benchmark
workload that emulates the delay that might be seen in an interaction from an actual user. A
short time delay will cause users to do more work in any time interval. A think time of 7 seconds
gave an optimal balance between number of users and workload.
5.3.1 Day run settings + results achieved The measurement metric for the Day run is payment items per hour. Each cycle of 15 dialog
steps run by each user generates 175 payment items.
56916893 Postings per hour
Duration of high load phase in seconds: 981
Average response time per dialog step (ms): 1119.8
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
44/168
5.3.2 Scalability Tests were done to see how well the system scaled from 1 to 4 members.
As shown in the graph below, the results indicated effectively linear scaling.
Scalability
0
10000000
20000000
30000000
40000000
50000000
60000000
1 2 3 4
Members
Pi/hr
5.3.3 Specific Day Run Tuning
5.3.3.1 BCA_RCN_SUMS_IN/OUT
These tables contain accumulation values and are updated very heavily during both the Day and
Night runs. Because these are very small tables, they are potentially the source of contention
between the many work processes doing simultaneous updates. To eliminate this contention,
we reorganized the table with PCTFREE 99 in order to place one row on each page, and used an
SAP® report to assign each work process to a specific row.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
45/168
Database Servers
System Summary DB2PS-member 12/4/2010
0
10
20
30
40
50
60
70
80
90
100
22:0
3
22:0
4
22:0
5
22:0
6
22:0
7
22:0
8
22:0
9
22:1
0
22:1
1
22:1
2
22:1
3
22:1
4
22:1
5
22:1
6
22:1
7
22:1
8
22:1
9
22:2
0
22:2
1
22:2
2
22:2
3
22:2
4
22:2
5
22:2
6
22:2
7
22:2
8
22:2
9
22:3
0
22:3
1
22:3
2
22:3
3
22:3
4
22:3
5
22:3
6
22:3
7
22:3
8
22:3
9
usr%
+sy
s%
CPU%
Each of the 4 DB2 database members consumed 62% user and 3% system CPU during the high
load phase.
Disk total KB/s DB2PS-member - 12/4/2010
0
100
200
300
400
500
600
700
800
900
1000
22:0
3
22:0
4
22:0
5
22:0
6
22:0
7
22:0
8
22:0
9
22:1
0
22:1
1
22:1
2
22:1
3
22:1
4
22:1
5
22:1
6
22:1
7
22:1
8
22:1
9
22:2
0
22:2
1
22:2
2
22:2
3
22:2
4
22:2
5
22:2
6
22:2
7
22:2
8
22:2
9
22:3
0
22:3
1
22:3
2
22:3
3
22:3
4
22:3
5
22:3
6
22:3
7
22:3
8
22:3
9
Thou
sand
s
KB
/sec
0
5000
10000
15000
20000
25000
30000
35000
IO/s
ec
Disk Read KB/s Disk Write KB/s IO/sec
Sustained IOPS rate during high load was 26000 per server.
Application Servers
System Summary sapservn 12/4/2010
0
10
20
30
40
50
60
70
80
90
100
22:0
3
22:0
4
22:0
5
22:0
6
22:0
7
22:0
8
22:0
9
22:1
0
22:1
1
22:1
2
22:1
3
22:1
4
22:1
5
22:1
6
22:1
7
22:1
8
22:1
9
22:2
0
22:2
1
22:2
2
22:2
3
22:2
4
22:2
5
22:2
6
22:2
7
22:2
8
22:2
9
22:3
0
22:3
1
22:3
2
22:3
3
22:3
4
22:3
5
22:3
6
22:3
7
22:3
8
22:3
9
usr%
+sy
s%
CPU%
Each of the 20 application servers consumed 65% CPU (62% user, 3% system).
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
46/168
Disk total KB/s sapservn - 12/4/2010
0
50
100
150
200
250
22:0
3
22:0
4
22:0
5
22:0
6
22:0
7
22:0
8
22:0
9
22:1
0
22:1
1
22:1
2
22:1
3
22:1
4
22:1
5
22:1
6
22:1
7
22:1
8
22:1
9
22:2
0
22:2
1
22:2
2
22:2
3
22:2
4
22:2
5
22:2
6
22:2
7
22:2
8
22:2
9
22:3
0
22:3
1
22:3
2
22:3
3
22:3
4
22:3
5
22:3
6
22:3
7
22:3
8
22:3
9
KB
/sec
0
5
10
15
20
25
30
IO/s
ec
Disk Read KB/s Disk Write KB/s IO/sec
5.3.4 Conclusion The final Day processing number was constrained more by the amount of ime the hardware was
available than by any machine resource constraint. The final result was achieved with a well
balanced system, with approximately equal and optimal CPU usage on the DB server and the
application servers. The IO rate of 104000 IOPS gave average read times of 12ms and minimal
write times.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
47/168
5.4 TRBK Night Run
The high water mark Night run was performed using 720 batch work processes on 20 application
servers each with one SAP® instance. The number of accounts processed in a single transaction
by a work process (a package) is configurable. We found that the default of 50 accounts per
package worked well.
5.4.1 Night run settings + results achieved The measurement metric for the Night run is accounts per hour. The entire loaded range of 90
million accounts is processed during the Night run.
Night run duration: 2 hours, 38 minutes
34300233 accounts per hour
5.4.1.1 Night Run processing in detail
During the night run, the total number of packages (in our case1.8 million packages with
packagesize 50) is distributed over the available application servers. So each of the 20
application servers gets a range of package numbers to process.
The following figure shows the distribution of packages over application servers
1-
90000
90001-
180000
180001-
270000
270001-
360000
…… 1620001-
1710000
1710001-
1800000 app server 1 app server 2 app server 3 app server 4 …… app server 19 app server 20
All the work processes of a respective application server then pick a free package out the range,
process it and write back status information into the table BANK_WORKL_PACK.
Package is not processed ‘ ‘
Package has been picked – processing started ‘X’
Package is completed ‘R’
The distribution of package number ranges over the involved application servers can be
checked in table BANK_WL_SRVREL.
Generally speaking, that account master data that is used to balance a package is aligned
with the package numbers. So neighboring packages refer to neighboring data.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
48/168
5.4.1.2 Problem in night run end processing
A default behavior of the SAP® benchmarking application was that work processes on
application servers that have completed their amount of work (number range of packages
assigned to that application server) take over unprocessed packages of other ranges.
That caused serious contention problems in the DB2 Pure Scale architecture, especially because
after some time, a massive number of work processes connected to all 4 members are working
on packages of the same range (and therefore need to access the same – or at least – colocated
data).
The throughput went down to nearly 0 within the last 0.5 % of the packages to process.
So solve this problem, SAP® provided a fix at the on ABAP level, to avoid having application
servers that have finished their work taking over packages of other application servers. The
ABAP modification is officially available and is described in SAP® note 1929954.
5.4.1.3 UUID Generation
Within the project we observed that neither the application servers nor the database servers
were heavily utilized (app server ~ 60%, database server ~25%). This picture also did not change
when trying to increase the number of work processes on the application servers. But during
this test the problem was showing up more clearly. Almost all work processes were stuck in a
ABAP class called CL_SYSTEM_UUID. The SAP® kernel uses an OS functionality to determine
unique IDs. It turned out that the SLES 10 implementation of this feature is the bottleneck. In
SAP® note 1391070 there is a description how to solve this problem (use the UUID deamon).
The result of the new implementation was that the UUID generation speeded up dramatically,
our CPU utilization on the application servers jumped to 99% and the throughput went up to the
final result values.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
49/168
5.5 Experiments with packagesize and number of work processes
As described above, during the project there was a problem with the night run end processing.
Increasing the package size (reducing the over all number of packages) first seemed to be a valid
method to reduce the end processing problem.
But it turned out, that the actual problem could not be removed, even though the point in time
when the problem occured, could be moved to 0.05% of left packages. Database parameters like
LOCKLIST had to be adapted when using higher package sizes because the processing of one
package is done within one database transaction. Over all, we experimented with the following
package sizes:
Packagesize 50 � default at the beginning, also used for the golden run
Packagesize 100
Packagesize 300 � LOCKLIST had to be increased
Packagesize 600
In the night run scenario we also experimented with several settings for the number of used
work processes. But all these tests were bottlenecked by the described problem with UUID
generation, so the changes in number of work processes did not really have big effects. Here is
an overview of which work process configurations were used:
640 batch work processes � initial setting
720 batch work processes � setting for the golden run
800 batch work processes � setting used to try to increase workload
1280 batch work processes � roll in / roll out problems
5.5.1 Resource usage
Database Servers
Each DB member consumed 33% CPU and drove ~8000 IOPS
System Summary DB2PS-member 12/5/2010
0
10
20
30
40
50
60
70
80
90
100
22:4
6
22:5
0
22:5
4
22:5
8
23:0
2
23:0
6
23:1
0
23:1
4
23:1
8
23:2
2
23:2
6
23:3
0
23:3
4
23:3
8
23:4
2
23:4
6
23:5
0
23:5
4
23:5
8
00:0
2
00:0
6
00:1
0
00:1
4
00:1
8
00:2
2
00:2
6
00:3
0
00:3
4
00:3
8
00:4
2
00:4
6
00:5
0
00:5
4
00:5
8
01:0
2
01:0
6
01:1
0
01:1
4
01:1
8
01:2
2
01:2
6
01:3
0
01:3
4
usr%
+sy
s%
0
2000
4000
6000
8000
10000
12000D
isk
xfer
s
CPU% IO/sec
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
50/168
Disk total KB/s DB2PS-member - 12/5/2010
0
50
100
150
200
250
300
350
400
450
500
22:4
6
22:5
0
22:5
4
22:5
8
23:0
2
23:0
6
23:1
0
23:1
4
23:1
8
23:2
2
23:2
6
23:3
0
23:3
4
23:3
8
23:4
2
23:4
6
23:5
0
23:5
4
23:5
8
00:0
2
00:0
6
00:1
0
00:1
4
00:1
8
00:2
2
00:2
6
00:3
0
00:3
4
00:3
8
00:4
2
00:4
6
00:5
0
00:5
4
00:5
8
01:0
2
01:0
6
01:1
0
01:1
4
01:1
8
01:2
2
01:2
6
01:3
0
01:3
4
Tho
usan
ds
KB
/sec
0
2000
4000
6000
8000
10000
12000
IO/s
ec
Disk Read KB/s Disk Write KB/s IO/sec
Application Servers
System Summary sapservn 12/5/2010
0
10
20
30
40
50
60
70
80
90
100
22:4
6
22:5
0
22:5
4
22:5
8
23:0
2
23:0
6
23:1
0
23:1
4
23:1
8
23:2
2
23:2
6
23:3
0
23:3
4
23:3
8
23:4
2
23:4
6
23:5
0
23:5
4
23:5
8
00:0
2
00:0
6
00:1
0
00:1
4
00:1
8
00:2
2
00:2
6
00:3
0
00:3
4
00:3
8
00:4
2
00:4
6
00:5
0
00:5
4
00:5
8
01:0
2
01:0
6
01:1
0
01:1
4
01:1
8
01:2
2
01:2
6
01:3
0
01:3
4
usr%
+sy
s%
0
2
4
6
8
10
12
14
16
18
20
Dis
k xf
ers
CPU% IO/sec
5.5.2 Conclusion
At the end that night run CPU usage on the database servers was ~33 % while all 20 application
servers were at 99%. So the conclusion is that we were application server bound in this
situation, and higher throughput might be reached by adding additional application servers.
DB2 Pure Scale performs very well in the TRBK night run scenario especially because of the
nature of the workload. The packages that are processed separately on the several servers allow
nearly full parallelization.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
51/168
5.6 High Availability Test
5.6.1 Results
The table below contains the results of the HA scenarios performed:
Test
# Scenarios Customer requirement
(outage time) DB2 pureScale® result
(outage time) 1 Member shutdown failure (dual
member failure part 1) <= 60 seconds 30 seconds
2 Malloc bomb member (dual member failure part 2)
<= 60 seconds 40 seconds
3 Primary CF shutdown failure <= 60 seconds 20 seconds
4 Secondary CF shutdown failure <= 60 seconds 10 seconds
5 Malloc bomb primary CF <= 60 seconds 20 seconds
6 Member IB adapter failure <= 60 seconds 30 seconds
7 Soft kill member <= 30 seconds 30 seconds
8 IB switch failure <= 60 seconds 30 seconds
The high availability (HA) tests were done using the same settings as the day/night run except
the following:
- Single HCA was used (HA support with multiple HCA will be in db2_v98fp4)
- The (primary/secondary) CFs’ memory were set to 70% of the amount of physical
memory available on the CF machines to match the HA test minimum requirement
of running the TRBK workload on half the throughput level of the day run
- 13 app servers with 6000 users were used to match the HA test minimum
requirement of running the TRBK workload on half the throughput level of the day
run
5.6.2 GPFS IO fencing
All the day/night/HA runs were run with the following TSA/GPFS settings which reduced the IO
fencing time when a hard failure occur. The settings below reduced the IO fencing time from 50
seconds to about 10 seconds.
hostfailuredetectiontime = 2
gpfs failureDectionTime = 10
leaseRecoveryWait =10
SCSI 3 PR support for IBM storage DS8700 series is now available in db2_v98fp4 and it will not
be necessary to alter the TSA/GPFS settings above anymore. With this change a much improved
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
52/168
IO fencing time approximately in a range of 2 to 5 seconds will be seen, with a corresponding
improvement in outage times.
5.6.3 Measuring the outage time
The outage time is defined by measuring the zero throughput period from the SAP®
simstat data.
5.6.4 Methods to generate the failure
Shutdown failure: echo o > /proc/sysrq-trigger Software failure: kill -9 on an actively running db2sysc, or ca-server process Malloc bomb: malloc.pl 999999 (see appendix for the malloc.pl code)
IB adapter failure: ifdown <ib-interface> IB switch failure: Manually unplug the power cable of the IB switch
5.6.5 Behaviors in different failure scenarios
Hard member failure: applications connected to the failed member will be rerouted to
one of the active members as defined in the db2dsdriver.cfg file (see the appendix for the
db2dsdriver.cfg content used in all the HA scenarios). SQL30108 (i.e. “A connection failed but
has been re-established”) will be returned to the applications which have in-flight transactions
running against the failing member, and the applications should handle this error gracefully,
resubmit the work which have not yet been committed and continue to proceed. Notice that
the TRBK benchmark program would actually treat the SQL30108 as errors, so SQL30108 will
show up as errors in the simstat data. The behavior of the benchmark program is to continue
the remaining of the 15 steps loop -- the first remaining step will get SQL30108, saying the
connection has been rerouted. And the remaining of the 15 steps will get SAP® logical errors.
Once the 15-step loop ended and a new loop started, the applications will stop generating
errors.
On the DB2 pureScale® server side, all the in-flight transactions running against the
failing member will be rolled back and resources (e.g. locks) held by the failing member will be
released by member crash recovery which will be done on one of the live members. Once the
machine that failed comes back, it will rejoin the cluster. Connections which got rerouted to
other live members due to the hard member failure will get rerouted back to the newly joined
member.
Soft member failure: The behavior of soft member failure will be the same as hard
member failure except DB2 will be able to restart the failing member locally (instead of one of
the live members) and perform member crash recovery. Once member crash recovery is
completed, the member will be able to start serving connections immediately. Connections
which previously got rerouted to other live members due to the soft kill failure will get rerouted
back.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
53/168
Primary/Secondary CF failure: CF failure will be transparent to the applications, i.e.
simstat data will show no errors returned. In the case of primary CF failure, the secondary CF
will take over as the primary CF in a short period of time. In the case of secondary CF failure, the
primary CF will stay as the primary. Once the failed CF comes back online, reintegration and
catchup will occur and the CF will be marked as peer and will act as the secondary CF.
Malloc bomb: This is to simulate a machine to become unresponsive due to all the
physical memory and paging space being used up. A program (malloc.pl) listed in the appendix
was used to simulate this situation. In this PoC a TSA policy was set on the DB2 pureScale®
servers such that when 90% of the physical memory is used, a reboot will be triggered by TSA.
This is to avoid the situation when the machine becomes unresponsive and processes starting to
get killed randomly by the OS due to running out of memory. Once the machine is rebooted by
TSA, the behavior of the clients will be the same as a member failure scenario with clients
getting rerouted to the active members.
IB Switch failure: Powering off one of the IB switches will result in (soft) killing of half
the members and a CF (in our case, they are member 1, member 3, and the secondary CF)
simultaneously – essentially half of the cluster will be lost. Member failovers will occur (i.e.
member 1 failover to db2ps-member2, member 3 failover to db2ps-member0). Applications
connected to members 1 and 3 will get rerouted accordingly.
5.6.6 Reduce the client reroute time
The following parameters were adjusted on the client side to reduce the client reroute
time:
db2dsdriver.cfg:
<parameter name="keepAliveTimeout" value="5"/>
/etc/sysctl.conf:
net.ipv4.tcp_retries2 = 3
net.ipv4.tcp_syn_retries=2
5.6.7 Reduce the number of SAP® errors for read only transactions
A new DBSL driver was provided by SAP® to handle the SQL30108 gracefully on the
DBSL level without propagating the errors back to the user level for read only
transactions.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
54/168
6 Disaster Recovery tests
6.1 Layout change
In order to execute the various DR test cases identified, the existing equipment was divided into
2 sites, each with 2 DB2 pureScale® members and 1 CF, as described the following architecture
diagram.
6.2 Results
The DR tests performed were defined from customer requirements for the Proof of Concept. For
the main disaster tests, the workload was specified to be 50% of the performance target for the
Night run (approximately 12m accounts per hour. For the Storage Interconnect interruption, the
workload was defined to be the Day run running at 50% of the performance target
(approximately 20m payitems/hour). For the QReplication test (maintenance of a logical copy 15
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
55/168
minutes “behind” the primary system), there was no requirement to use the logical copy to
restart the primary, and so this test was not done. However, the logical copy was consistent, and
could have been used for restart.
All the tests were completed successfully. The following table summarizes the tests and results
with more detailed descriptions in the later sections of this chapter.
6.3 Site Failure – Storage replication
The objective of this test was to show that the TRBK Night run could be restarted and completed
on the secondary site in the event of a complete failure of the primary site.
Methodology
• DS8800 Storage replication is in operation between the DS8800’s on the primary and
secondary sites
• TRBK Night executing on the primary site, with workload of approximately 17.5m
accounts/hour
• Connection to the ADVA device physically severed
• Consistency copy taken, primary workload continues until manually terminated
• Database started on secondary site
• Group crash recovery processed – 10 minutes
• SAP® started on secondary application servers
• Previous in-progress run terminated using RBANK_DELETE_RUNS
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
56/168
• New run started, and run to completion
• Rate 18m accounts/hour, slightly faster due to quicker completion of already
processed accounts
6.3.1 Site Failure Implementation
Primary site : Primary SAP® transaction banking running normally
Disaster recovery site: Devices are down except storage devices receiving data replication
(DS8800 Metro Mirror)
6.3.2 Disaster recovery mechanism
DB2 storage devices are replicated to the Recovery site using DS8000 Metro Mirror.
(Tivoli Storage productivity Center for replication is used to manage the advanced copy
services).
Configuration for DR
l ll l
Storage replication
l
DB : D3D DB : D3D
Member0 Member1 cf0 Member2 Member3 cf1
QcQ
l
m
m
m
m
SAP®
50 51 54 52 53 55
SAP®
DS8800
Storage
10.7.21.0
10.1.2.0 10.1.21.0
10.1.3.0 10.1.3.0
Latency
Latency
10.9.9.0 10.9.9.0
DS8800
Admin network attached to all pieces
DB : D3D
fenced
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
57/168
6.3.3 High level process of recovery.
� DR Simulation
• SAP® TRBK night run workload running on primary site
• To simulate loss of primary site, ADVA (DWDM) will be switched off.
� RPO measurement
Number of processed accounts versus targeted accounts (measurement of accounts that were
not processed). The test specification required RPO=0. Running the restarted Night run to
completion demonstrated that RPO=0 was achieved.
� Process to restart on DR site
• Recover metro Mirror session to re-enable access to target devices ( Using TPCR Web
GUI)
• Replicated devices are re-mapped to gpfs devices (see procedure in appendix )
• gpfs started
• DB2 database recataloged on dbpath
• DB2 instance started
• DB2 crash recovery started
• SAP® restarted
• Cleanup of SAP® tables
• Restart on TRBK night run
� Prerequisites on DR site
Disaster recovery site must have DB2 pureScale® installed at same level as on primary site,
DS8000 Metro Mirror
• DB2 instance created ( see procedure in appendix )
• DB2 path created ( see procedure in appendix )
• DS8000 metro mirror activated and source-target volume pairs are fully synchronized
(i.e. in “Prepared” state)
• GPFS meta-data definitions must be imported from primary cluster (see procedure in
appendix).
o Note: This is a one-time process unless NDS definitions got changed in the
primary site ( more details in section 2.2.2 Exporting the filesystem definitions )
• Imported GPFS meta-data is masked to avoid GPFS daemons from accessing those
metro-mirror replicated devices until we fail-over to the DR site ( see procedure in
appendix )
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
58/168
� List of storage devices replicated
Running df command should return the following information
The replicated are the ones which are grey shaded.
Filesystem 1K-blocks Used Available Use% Mounted
/dev/sda3 138195432 29875488 101299944 23% / Internal disk
devtmpfs 132266056 736 132265320 1% /dev Internal disk
tmpfs 132266056 0 132266056 0% /dev/shm Internal disk
/dev/sda1 72102 28760 43342 40% /boot/efi Internal disk
/dev/d3d 2147483648 1777664 2145705984 1% /db2/D3D DB2 home
/dev/db2fs1 2147483648 117846016 2029637632 6% /db2sd_20110311231413
DB2 shared instance
/dev/log0 2147483648 1930240 2145553408 1% /db2/D3D_log0 DB2 logs member0
/dev/log1 2147483648 1883136 2145600512 1% /db2/D3D_log1 DB2 logs member1
/dev/SAP® data1 1073741824 346840064 726901760 33% /db2/D3D_SAP® data1
DB2 container
/dev/SAP® data10 1073741824 346840064 726901760 33% /db2/D3D_SAP® data10
DB2 container
/dev/SAP® data11 1073741824 346839040 726902784 33% /db2/D3D_SAP® data11
DB2 container
/dev/SAP® data12 1073741824 346840064 726901760 33% /db2/D3D_SAP® data12
DB2 container
/dev/SAP® data13 1073741824 346840064 726901760 33% /db2/D3D_SAP® data13
DB2 container
/dev/SAP® data14 1073741824 346840064 726901760 33% /db2/D3D_SAP® data14
DB2 container
/dev/SAP® data15 1073741824 346840064 726901760 33% /db2/D3D_SAP® data15
DB2 container
/dev/SAP® data16 1073741824 346839040 726902784 33% /db2/D3D_SAP® data16
DB2 container
/dev/SAP® data2 1073741824 346840064 726901760 33% /db2/D3D_SAP® data2
DB2 container
/dev/SAP® data3 1073741824 346840064 726901760 33% /db2/D3D_SAP® data3
DB2 container
/dev/SAP® data4 1073741824 346840064 726901760 33% /db2/D3D_SAP® data4
DB2 container
/dev/SAP® data5 1073741824 346840064 726901760 33% /db2/D3D_SAP® data5
DB2 container
/dev/SAP® data6 1073741824 346840064 726901760 33% /db2/D3D_SAP® data6
DB2 container
/dev/SAP® data7 1073741824 346840064 726901760 33% /db2/D3D_SAP® data7
DB2 container
/dev/SAP® data8 1073741824 346840064 726901760 33% /db2/D3D_SAP® data8
DB2 container
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
59/168
/dev/SAP® data9 1073741824 346840064 726901760 33% /db2/D3D_SAP® data9
DB2 container
These devices are mapped to GPFS nsd ( 1 to 1 )
List of gpfs nsd
DB2PS-member0:~ # mmlsnsd
File system Disk name NSD servers
--------------------------------------------------------------------------
d3d gpfs2nsd (directly attached) DB2 base directory Replicated
db2fs1 gpfs1nsd (directly attached) DB2 shared instance Not replicated
log0 gpfs22nsd (directly attached) Logs for member0 Replicated
log1 gpfs21nsd (directly attached) Logs for member1 Replicated
SAP®
data1 gpfs16nsd (directly attached) DB2 container
Replicated
SAP®
data10 gpfs8nsd (directly attached) DB2 container
Replicated
SAP®
data11 gpfs6nsd (directly attached) DB2 container
Replicated
SAP®
data12 gpfs4nsd (directly attached) DB2 container
Replicated
SAP®
data13 gpfs3nsd (directly attached) DB2 container
Replicated
SAP®
data14 gpfs18nsd (directly attached) DB2 container
Replicated
SAP®
data15 gpfs17nsd (directly attached) DB2 container
Replicated
SAP®
data16 gpfs15nsd (directly attached) DB2 container
Replicated
SAP®
data2 gpfs14nsd (directly attached) DB2 container
Replicated
SAP®
data3 gpfs13nsd (directly attached) DB2 container
Replicated
SAP®
data4 gpfs12nsd (directly attached) DB2 container
Replicated
SAP®
data5 gpfs11nsd (directly attached) DB2 container
Replicated
SAP®
data6 gpfs10nsd (directly attached) DB2 container
Replicated
SAP®
data7 gpfs9nsd (directly attached) DB2 container
Replicated
SAP® gpfs7nsd (directly attached) DB2 container Replicated
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
60/168
data8
SAP®
data9 gpfs5nsd (directly attached) DB2 container
Replicated
6.3.4 GPFS configuration
GPFS configuration must be propagated to the remote site.
� Prepare remote nodes file
This file will be used as input for the mmfsctl command .
add the remote nodes into this file .
vi remotenodefile
DB2PS-member2
DB2PS-member3
DB2PS-cf1
� Propagating GPFS filesystem definitions to the DR cluster
Prerequisites:
• Configure password-less rsh for root on machines between the primary site and the
target site.
• All source-target volume pairs in the Metro-Mirror session have reached “Prepared”
State (fully synchronized).
For each filesystem
Run the following command
/usr/lpp/mmfs/bin/mmfsctl gpfs_filesystem_name syncFSconfig -n remotenodefile
Example:
/usr/lpp/mmfs/bin/mmfsctl SAP® data9 syncFSconfig - n remotenodefile
Note : whenever a GPFS NSDfilesystem definition changes in the primary site, this command
must be run again .
All commands are prepared in the /root/DR/fs_sync.sh on DB2PS-member0.
This script must be run as root.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
61/168
o fs_sync.sh DB2PS-member0:~/DR # cat fs_sync.sh mmfsctl d3d syncFSconfig -n remotenodefile mmfsctl log0 syncFSconfig -n remotenodefile mmfsctl log1 syncFSconfig -n remotenodefile mmfsctl log2 syncFSconfig -n remotenodefile mmfsctl log3 syncFSconfig -n remotenodefile mmfsctl SAP® data1 syncFSconfig -n remotenodefile mmfsctl SAP® data10 syncFSconfig -n remotenodefile mmfsctl SAP® data11 syncFSconfig -n remotenodefile mmfsctl SAP® data12 syncFSconfig -n remotenodefile mmfsctl SAP® data13 syncFSconfig -n remotenodefile mmfsctl SAP® data14 syncFSconfig -n remotenodefile mmfsctl SAP® data15 syncFSconfig -n remotenodefile mmfsctl SAP® data16 syncFSconfig -n remotenodefile mmfsctl SAP® data2 syncFSconfig -n remotenodefile mmfsctl SAP® data3 syncFSconfig -n remotenodefile mmfsctl SAP® data4 syncFSconfig -n remotenodefile mmfsctl SAP® data5 syncFSconfig -n remotenodefile mmfsctl SAP® data6 syncFSconfig -n remotenodefile mmfsctl SAP® data7 syncFSconfig -n remotenodefile mmfsctl SAP® data8 syncFSconfig -n remotenodefile mmfsctl SAP® data9 syncFSconfig -n remotenodefile
o Execution of fs_sync.sh
DB2PS-member0:~/DR # ./fs_sync.sh > fs_sync.out 2>&1
Survey of output in a different window
DB2PS-member0:~/DR # tail -f fs_sync.out mmexportfs: Processing file system d3d ... mmfsctl: Importing file system information into th e target cluster on node DB2PS-member2 . . . mmimportfs: Processing file system d3d ... mmimportfs: Processing disk gpfs2nsd mmimportfs: Committing the changes ... mmimportfs: The following file systems were success fully imported: …/…
Note:
• We propagated the GPFS meta-data into the Secondary Clusters as part of the initial
configuration.
• Since NSDs on the primary site remain unchanged from the time we did the initial GPFS
DR configuration on the standby side, there was no need to re-sync the NSD meta-data
again. Hence we skipped this step during the DR fail-over testing.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
62/168
• When NSD meta-data is propagated to the DR cluster via mmfsctl .. sysFSconfig , all the block devices that map to the meta-data definitions (which will
be imported from the primary site via mmfsctl ) must be “visible” to the GPFS
daemons . Hence, in-order to do the meta-data definition we need to ensure that step
2.2.7.2 Remap GPFS nsd is executed before hand.
• At a later time, if there is ever a need to propagate any NSD changes to the DR site, for
example due to a new NSD/GPFS filesystem being added as a storage path to D3D db,
you need to ensure all NSDs are mapped (unmasked) – i.e. execute step 2.2.7.2 Remap
GPFS nsd , before executing mmfsctl .. sysFSconfig ..
• Note that this type of resynchronization of GPFS meta-data between primary and
secondary clusters are required only for those actions that modify the file system
information maintained in the local GPFS configuration data, which includes:
o Additions, removals, and replacements of disks (mmadddisk, mmdeldisk, mmrpldisk )
o Modifications to disk attributes (mmchdisk )
o Changes to the file system's mount point (mmchfs -T )
o Changing the file system device name (mmchfs -W )
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
63/168
6.3.5 Recovery Procedure on DR Site
Initial status
• Verify GPFS daemons are up
DB2PS-member2:~/scripts # mmgetstate -a -L Node number Node name Quorum Nodes up Tot al nodes GPFS state Remarks --------------------------------------------------- --------------------------------- 1 DB2PS-member2 1* 3 3 active quorum node 2 DB2PS-cf1 1* 3 3 active quorum node 3 DB2PS-member3 1* 3 3 active quorum node
• Uncatalog DB D3D used for Qrep
db2d3d@DB2PS-Member2:~> db2 uncatalog db d3d DB20000I The UNCATALOG DATABASE command completed successfully.
• Filesystems already mounted Filesystem 1K-blocks Used Available Use% Mo unted on /dev/sda3 138195432 21604884 109570548 17% / devtmpfs 132266056 1056 132265000 1% /de v tmpfs 132266056 0 132266056 0% /dev/sh m /dev/sda1 72102 28762 43340 40% /boot/efi /dev/db2fs1 524288000 1528832 522759168 1% /db2sd_20110401115043
Qrep FS still mounted.
db2d3d@DB2PS-cf1:~> df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda3 138195432 19466016 111709416 15% / devtmpfs 132266056 740 132265316 1% /dev tmpfs 132266056 0 132266056 0% /dev/shm /dev/sda1 72102 28768 43334 40% /boot/efi /dev/db2fs1 524288000 1623040 522664960 1% /db2sd_20110401115043 /dev/qrlog0 524288000 14642176 509645824 3% /QR_log0 /dev/qrlog1 524288000 585728 523702272 1% /QR_log1 /dev/qrdbpath 52428800 1273856 51154944 3% /db2/QR_D3D /dev/qrSAP® data3 524288000 346584064 17770393 6 67% /QR_SAP® data3 /dev/qrSAP® data7 524288000 346584064 17770393 6 67% /QR_SAP® data7 /dev/qrSAP® data2 524288000 345648640 17863936 0 66% /QR_SAP® data2 /dev/qrSAP® data6 524288000 346584064 17770393 6 67% /QR_SAP® data6 /dev/qrSAP® data5 524288000 346584064 17770393 6 67% /QR_SAP® data5 /dev/qrSAP® data8 524288000 346584064 17770393 6 67% /QR_SAP® data8 /dev/qrSAP® data4 524288000 346584064 17770393 6 67% /QR_SAP® data4 /dev/qrSAP® data1 524288000 345648640 17863936 0 66% /QR_SAP® data1 /dev/qrSAP® data14 524288000 346584064 17770393 6 67% /QR_SAP® data14 /dev/qrSAP® data15 524288000 346584064 17770393 6 67% /QR_SAP® data15 /dev/qrSAP® data13 524288000 346584064 17770393 6 67% /QR_SAP® data13 /dev/qrSAP® data12 524288000 345648640 17863936 0 66% /QR_SAP® data12
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
64/168
/dev/qrSAP® data11 524288000 346584064 17770393 6 67% /QR_SAP® data11 /dev/qrSAP® data10 524288000 346584064 17770393 6 67% /QR_SAP® data10 /dev/qrSAP® data16 524288000 346584064 17770393 6 67% /QR_SAP® data16 /dev/qrSAP® data9 524288000 346584064 17770393 6 67% /QR_SAP® data9
• DB2 installed and configured As user db2d3d check existence of instance
db2d3d@DB2PS-member2:~> db2instance -list ID TYPE STATE HOM E_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NET NAME -- ---- ----- --- ------ ------------ ----- ---------------- ------------ --- ---- 0 MEMBER STOPPED DB2 PS-member2 DB2PS-member2 NO 0 0 DB2PS- member2-ib0 1 MEMBER STOPPED DB2 PS-member3 DB2PS-member3 NO 0 0 DB2PS- member3-ib0 128 CF STOPPED DB2 PS-cf1 DB2PS-cf1 NO - 0 DB2PS- cf1-ib0 HOSTNAME STATE INS TANCE_STOPPED ALERT -------- ----- --- ------------- ----- DB2PS-cf1 ACTIVE NO NO DB2PS-member3 ACTIVE NO NO DB2PS-member2 ACTIVE NO NO
Check that node state is active.
• Replication
Replication is suspended (loss of PPRC paths)
Recover Metro-Mirror session, prior configuring storage devices.
o start a Remote Desktop Connection to the machine on which TPCR (Tivoli Productivity
Centre for Replication) is installed. Note that TPCR should be not be installed on either
the primary or secondary systems, for obvious reasons.
o On the Desktop there is a shortcut to "Tivoli Storage Productivity Center for
Replication"
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
65/168
o go to Sessions - there is only 1 session ( DB2PS_PPRC)
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
66/168
o Select DP2PS_PPRC
o Check that storage replication is in status Normal
Primary Site Failure
� Start TRBK night run and leave it running 1 hour
� Disconnect interconnect fibre between the 2 ADVA
o Check that storage replication is in status Suspended
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
67/168
� Recover session
Recover H2 (DR site) to make the MM replicated LUNs host-addressible from DR cluster
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
68/168
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
69/168
� TPCR Console log
Now devices are ready to be re-mapped to gpfs
� Prepare restart of recovery site
� Check accessibility of storage devices
Multipath –l should return information similar to the following.
Check presence of all replicated devices.
DB2PS-member2:~ # multipath –l SAP® data14 (3600507630bffc09c0000000000001113) dm- 41 IBM,2107900 size=1.0T features='1 queue_if_no_path' hwhandler=' 0' wp=rw `-+- policy='round-robin 0' prio=-1 status=active |- 4:0:0:39 sdao 66:128 active undef running |- 5:0:0:39 sdcl 69:144 active undef running `- 6:0:0:39 sdei 128:160 active undef running
• Remap (Unmask) storage replicated GPFS nsds
Make Metro-Mirror replicated GPFS NSDs visible to the DR cluster and mount those replicated
file systems
- On DB2PS-member2 as root, issue /root/scripts/enable_dr_nsd.sh
See contents of enable_dr_nsd.sh in appendix.
DB2PS-member2:~ # /root/scripts/enable_dr_nsd.sh sda generic sda1 generic sda2 generic …/…
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
70/168
GPFS NSDs visible to the cluster ... Disk name NSD volume ID Device Nod e name Remarks --------------------------------------------------- ------------------------------------ gpfs1001nsd 0A0102344D963C72 /dev/dm-0 DB2 PS-member2.localdomain gpfs1002nsd 0A0102344D963C73 /dev/dm-11 DB2 PS-member2.localdomain gpfs1006nsd 0A0102344D9656A6 /dev/dm-16 DB2 PS-member2.localdomain gpfs1007nsd 0A0102344D9656A7 /dev/dm-19 DB2 PS-member2.localdomain gpfs10nsd 0A0102324D80C4A2 /dev/dm-37 DB2 PS-member2.localdomain gpfs11nsd 0A0102324D80C4BC /dev/dm-26 DB2 PS-member2.localdomain …/… gpfs8nsd 0A0102324D80C475 /dev/dm-40 DB2 PS-member2.localdomain gpfs9nsd 0A0102324D80C489 /dev/dm-27 DB2 PS-member2.localdomain Tue Apr 26 19:43:35 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:43:37 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:43:38 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:43:44 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:43:47 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:43:53 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:43:55 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:00 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:02 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:04 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:11 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:12 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:14 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:21 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:23 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:25 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:32 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:34 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:36 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:43 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:45 CEST 2011: mmmount: Mounting fi le systems ... Mount status of all GPFS filesystems defined in the cluster ... File system d3d is mounted on 3 nodes: 10.1.2.52 DB2PS-member2 10.1.2.55 DB2PS-cf1 10.1.2.53 DB2PS-member3 File system db2fs1 is mounted on 3 nodes: 10.1.2.52 DB2PS-member2 10.1.2.55 DB2PS-cf1 10.1.2.53 DB2PS-member3 File system log0 is mounted on 3 nodes: 10.1.2.55 DB2PS-cf1 10.1.2.52 DB2PS-member2 10.1.2.53 DB2PS-member3
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
71/168
…/… File system SAP® data9 is mounted on 3 nodes: 10.1.2.55 DB2PS-cf1 10.1.2.53 DB2PS-member3 10.1.2.52 DB2PS-member2
Issuing mmlsnsd –m should now return all the MM replicated GPFS NSDs on secondary/DR site
DB2PS-member2:~/scripts # mmlsnsd -m Disk name NSD volume ID Device Nod e name Remarks --------------------------------------------------- ------------------------------------ gpfs1001nsd 0A0102344D963C72 /dev/dm-0 DB2 PS-member2.localdomain gpfs1002nsd 0A0102344D963C73 /dev/dm-11 DB2 PS-member2.localdomain gpfs1006nsd 0A0102344D9656A6 /dev/dm-16 DB2 PS-member2.localdomain gpfs1007nsd 0A0102344D9656A7 /dev/dm-19 DB2 PS-member2.localdomain gpfs10nsd 0A0102324D80C4A2 /dev/dm-37 DB2 PS-member2.localdomain gpfs11nsd 0A0102324D80C4BC /dev/dm-26 DB2 PS-member2.localdomain gpfs12nsd 0A0102324D80C4D9 /dev/dm-36 DB2 PS-member2.localdomain gpfs13nsd 0A0102324D80C4F5 /dev/dm-25 DB2 PS-member2.localdomain gpfs14nsd 0A0102324D80C512 /dev/dm-35 DB2 PS-member2.localdomain gpfs15nsd 0A0102324D80C530 /dev/dm-42 DB2 PS-member2.localdomain gpfs16nsd 0A0102324D80C54F /dev/dm-24 DB2 PS-member2.localdomain gpfs17nsd 0A0102324D80C570 /dev/dm-31 DB2 PS-member2.localdomain gpfs18nsd 0A0102324D80C594 /dev/dm-41 DB2 PS-member2.localdomain gpfs19nsd 0A0102324D80C61E /dev/dm-44 DB2 PS-member2.localdomain gpfs1nsd 0A0102344D959FDD /dev/dm-8 DB2 PS-member2.localdomain gpfs20nsd 0A0102324D80C641 /dev/dm-33 DB2 PS-member2.localdomain gpfs21nsd 0A0102324D80C665 /dev/dm-43 DB2 PS-member2.localdomain gpfs22nsd 0A0102324D80C68B /dev/dm-32 DB2 PS-member2.localdomain gpfs23nsd 0A0102344D95E0ED /dev/dm-1 DB2 PS-member2.localdomain gpfs24nsd 0A0102344D95E0FF /dev/dm-12 DB2 PS-member2.localdomain gpfs25nsd 0A0102344D95E112 /dev/dm-2 DB2 PS-member2.localdomain gpfs26nsd 0A0102344D95E126 /dev/dm-13 DB2 PS-member2.localdomain gpfs27nsd 0A0102344D95E139 /dev/dm-3 DB2 PS-member2.localdomain gpfs28nsd 0A0102344D95E150 /dev/dm-14 DB2 PS-member2.localdomain gpfs29nsd 0A0102344D95E166 /dev/dm-4 DB2 PS-member2.localdomain gpfs2nsd 0A0102324D80C2BA /dev/dm-34 DB2 PS-member2.localdomain gpfs30nsd 0A0102344D95E180 /dev/dm-15 DB2 PS-member2.localdomain gpfs31nsd 0A0102344D95E19A /dev/dm-10 DB2 PS-member2.localdomain gpfs33nsd 0A0102344D95E1D3 /dev/dm-5 DB2 PS-member2.localdomain gpfs34nsd 0A0102344D95E1F1 /dev/dm-17 DB2 PS-member2.localdomain gpfs35nsd 0A0102344D95E210 /dev/dm-6 DB2 PS-member2.localdomain gpfs36nsd 0A0102344D95E230 /dev/dm-18 DB2 PS-member2.localdomain gpfs37nsd 0A0102344D95E253 /dev/dm-7 DB2 PS-member2.localdomain gpfs39nsd 0A0102344D95E29C /dev/dm-9 DB2 PS-member2.localdomain gpfs3nsd 0A0102324D80C412 /dev/dm-30 DB2 PS-member2.localdomain gpfs4nsd 0A0102324D80C424 /dev/dm-39 DB2 PS-member2.localdomain gpfs5nsd 0A0102324D80C438 /dev/dm-29 DB2 PS-member2.localdomain gpfs6nsd 0A0102324D80C449 /dev/dm-28 DB2 PS-member2.localdomain gpfs7nsd 0A0102324D80C45F /dev/dm-38 DB2 PS-member2.localdomain gpfs8nsd 0A0102324D80C475 /dev/dm-40 DB2 PS-member2.localdomain gpfs9nsd 0A0102324D80C489 /dev/dm-27 DB2 PS-member2.localdomain
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
72/168
Issuing mmlsmount all_local will display all the mounted gfs filesystems, which should
now show MM replicated mounts as well.
DB2PS-member2:~/scripts # mmlsmount all_local File system d3d is mounted on 3 nodes. File system db2fs1 is mounted on 3 nodes. File system log0 is mounted on 3 nodes. File system log1 is mounted on 3 nodes. File system log2 is mounted on 3 nodes. File system log3 is mounted on 3 nodes. File system qrdbpath is mounted on 3 nodes. File system qrlog0 is mounted on 3 nodes. File system qrlog1 is mounted on 3 nodes. File system qrSAP® data1 is mounted on 3 nodes. File system qrSAP® data10 is mounted on 3 nodes. File system qrSAP® data11 is mounted on 3 nodes. File system qrSAP® data12 is mounted on 3 nodes. File system qrSAP® data13 is mounted on 3 nodes. File system qrSAP® data14 is mounted on 3 nodes. File system qrSAP® data15 is mounted on 3 nodes. File system qrSAP® data16 is mounted on 3 nodes. File system qrSAP® data2 is mounted on 3 nodes. File system qrSAP® data3 is mounted on 3 nodes. File system qrSAP® data4 is mounted on 3 nodes. File system qrSAP® data5 is mounted on 3 nodes. File system qrSAP® data6 is mounted on 3 nodes. File system qrSAP® data7 is mounted on 3 nodes. File system qrSAP® data8 is mounted on 3 nodes. File system qrSAP® data9 is mounted on 3 nodes. File system SAP® data1 is mounted on 3 nodes. File system SAP® data10 is mounted on 3 nodes. File system SAP® data11 is mounted on 3 nodes. File system SAP® data12 is mounted on 3 nodes. File system SAP® data13 is mounted on 3 nodes. File system SAP® data14 is mounted on 3 nodes. File system SAP® data15 is mounted on 3 nodes. File system SAP® data16 is mounted on 3 nodes. File system SAP® data2 is mounted on 3 nodes. File system SAP® data3 is mounted on 3 nodes. File system SAP® data4 is mounted on 3 nodes. File system SAP® data5 is mounted on 3 nodes. File system SAP® data6 is mounted on 3 nodes. File system SAP® data7 is mounted on 3 nodes. File system SAP® data8 is mounted on 3 nodes. File system SAP® data9 is mounted on 3 nodes.
• Restart database
• Catalog MM replicated copy of D3D database
o On DB2PS-member2 as DB2 instance ID (db2d3d), catalog D3D db
db2d3d@DB2PS-Member2:~> db2 catalog db D3D on /db2/D3D DB20000I The CATALOG DATABASE command completed su ccessfully.
• Start instance db2d3d
On DB2PS-member2 as DB2 instance ID (db2d3d), start DB2
db2d3d@DB2PS-member2:~> db2start 04/26/2011 19:51:56 0 0 SQL1063N DB2START processing was successful.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
73/168
04/26/2011 19:51:56 1 0 SQL1063N DB2START processing was successful. SQL1063N DB2START processing was successful.
� Verify status of instance db2d3d
db2d3d@DB2PS-member2:~> db2instance -list ID TYPE STATE HOM E_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME -- ---- ----- --- ------ ------------ ----- ---------------- ------------ ------- 0 MEMBER STARTED DB2 PS-member2 DB2PS-member2 NO 0 0 DB2PS-member2-ib0 1 MEMBER STARTED DB2 PS-member3 DB2PS-member3 NO 0 0 DB2PS-member3-ib0 128 CF PRIMARY DB2 PS-cf1 DB2PS-cf1 NO - 0 DB2PS-cf1-ib0 HOSTNAME STATE INS TANCE_STOPPED ALERT -------- ----- --- ------------- ----- DB2PS-cf1 ACTIVE NO NO DB2PS-member3 ACTIVE NO NO DB2PS-member2 ACTIVE NO NO
• Restart database
On DB2PS-member2 as DB2 instance ID (db2d3d),
o db2 restart db D3D Check the status of the Group Crash Recovery (GCR) by
db2d3d@DB2PS-member2:~/sqllib/db2dump> db2pd -db d3d -recovery Database Member 0 -- Database D3D -- Active -- Up 0 days 00:04:40 Recovery: Recovery Status 0x410A8015 Current Log Current LSN 000003E9DC81A8CC Current LRI 00000000000000010000000003324C0 4000003E9DC81A8CC Current LSO 4432765186522 Job Type GROUP CRASH RECOVERY Job ID 1 Job Start Time (1303840522) Tue Apr 26 19:55:2 2 2011 Job Description Group Crash Recovery Invoker Type User Total Phases 2 Current Phase 1 Progress: Address PhaseNum Description StartTime CompletedWork TotalWork 0x0000000205E4F8A8 1 Forward Tue Apr 26 19:55:22 182194440 bytes 1735886880 bytes 0x0000000205E4FA30 2 Backward NotStarted 0 bytes 1735886880 bytes Check DB2 diag log to verify GCR completed successfully
2011-04-26-20.06.04.718193+120 E215808E450 LEVEL: Info PID : 15930 TID : 4800789452569 6 PROC : db2sysc 0 INSTANCE: db2d3d NODE : 000 DB : D3D APPHDL : 0-52 APPID: *N0.DB2.11042 6175157
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
74/168
EDUID : 32 EDUNAME: db2agnti (D 3D ) 0 FUNCTION: DB2 UDB, recovery manager, sqlpresr, prob e:3110 MESSAGE : ADM1528I Group crash recovery has comple ted successfully.
2011-04-26-20.06.04.718801+120 I216259E462 LEVEL: Info PID : 15930 TID : 4800789452569 6 PROC : db2sysc 0 INSTANCE: db2d3d NODE : 000 DB : D3D APPHDL : 0-52 APPID: *N0.DB2.11042 6175157 EDUID : 32 EDUNAME: db2agnti (D 3D ) 0 FUNCTION: DB2 UDB, recovery manager, sqlpresr, prob e:3170 DATA #1 : <preformatted> Crash recovery completed. Next LSN is 000003E9DCB12 427.
Re-Start the SAP® run on the DR site
• Restart the SAP® application servers
• Use transaction SM37 to clean up the status of the batch jobs that were running on the
primary
• Run report RBANK_DELETE_RUN to reset the status of the interrupted run
• Start a new TRBK Reconciliation (Night) run in the normal way
• Elapsed time from the time the “disaster” (severing of the link between the primary and
secondary systems) until the start of the new Night run was approximately 40 minutes
• The new Night run reprocesses all accounts. For those accounts already processed by
the original run, the workload is minimal, leading to a quicker completion of the new
Night run.
6.3.6 Going back to the production configuration
Initial status All the GPFS file systems are mounted, i.e. both Q-replicated filesystems and Metro-Mirror replicated filesystems are mounted on the DR GPFS cluster.
Restore the replication link Power on the ADVA ( DWDM ) equipment and validate the FC links used for the Metro-Mirror copy session is functional. You can use DSCLI to query the Metro-Mirror copy-session path status using:
lspprcpath -dev storage_image_ID -l Source_LSS_ID Failed reason should indicate System Reserved Path. See following IBM System Storage Information Centre link for more details on the command and the path status codes:
http://publib.boulder.ibm.com/infocenter/dsichelp/ds8000ic/topic/com.ibm.storage.ssic.help.doc/f2c_clilspprcpath_1kz8kv.html
For, example in our test configuration, running DSCLI from TPCR desktop:
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
75/168
Note : If the link is not online, when you try to re-start the MM session it will fail.
Reset DB2 and GPFS back to production configuration � Update the DB2 Configuration on DR site
Before we switch over to the production configuration: o Update the GPFS Configuration on DR site Stop SAP® run o Deactivate D3D database:
o db2 terminate; force applications all; db2 deactiva te db D3D o Uncatalog D3D database:
o db2 uncatalog db D3D
� Before we switch over to the production configuration, unmount the Metro-Mirror
replicated filesystems and unmap (mask) those replicated NSDs on DR cluster.
- On DB2PS-member2 as root, issue /root/scripts/disable_dr_nsd.sh DB2PS-member2:~/scripts # ./disable_dr_nsd.sh Wed Apr 27 16:01:31 CEST 2011: mmumount: Unmounting file systems ... Wed Apr 27 16:01:32 CEST 2011: mmumount: Unmounting file systems ... Wed Apr 27 16:01:33 CEST 2011: mmumount: Unmounting file systems ... …/… Wed Apr 27 16:01:41 CEST 2011: mmumount: Unmounting file systems ... Mount status of all GPFS filesystems defined in the cluster ... File system d3d is not mounted. File system db2fs1 is mounted on 3 nodes: 10.1.2.52 DB2PS-member2 10.1.2.55 DB2PS-cf1 10.1.2.53 DB2PS-member3 File system log0 is not mounted. File system log1 is not mounted. File system log2 is not mounted. File system log3 is not mounted.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
76/168
File system qrdbpath is mounted on 3 nodes: 10.1.2.52 DB2PS-member2 …/… File system qrSAP® data9 is mounted on 3 nodes: 10.1.2.52 DB2PS-member2 10.1.2.55 DB2PS-cf1 10.1.2.53 DB2PS-member3 File system SAP® data1 is not mounted. File system SAP® data10 is not mounted. …/… dm-0 dmm dm-1 dmm dm-10 dmm dm-11 dmm ../… GPFS NSDs visible to the cluster ... Disk name NSD volume ID Device Nod e name Remarks --------------------------------------------------- ------------------------------------ gpfs1001nsd 0A0102344D963C72 /dev/dm-0 DB2 PS-member2.localdomain gpfs1002nsd 0A0102344D963C73 /dev/dm-11 DB2 PS-member2.localdomain gpfs1006nsd 0A0102344D9656A6 /dev/dm-16 DB2 PS-member2.localdomain gpfs1007nsd 0A0102344D9656A7 /dev/dm-19 DB2 PS-member2.localdomain gpfs10nsd 0A0102324D80C4A2 - DB2 PS-member2.localdomain (not found) directly attached gpfs11nsd 0A0102324D80C4BC - DB2 PS-member2.localdomain (not found) directly attached gpfs12nsd 0A0102324D80C4D9 - DB2 PS-member2.localdomain (not found) directly attached gpfs13nsd 0A0102324D80C4F5 - DB2 PS-member2.localdomain (not found) directly attached gpfs14nsd 0A0102324D80C512 - DB2 PS-member2.localdomain (not found) directly attached gpfs15nsd 0A0102324D80C530 - DB2 PS-member2.localdomain (not found) directly attached gpfs16nsd 0A0102324D80C54F - DB2 PS-member2.localdomain (not found) directly attached gpfs17nsd 0A0102324D80C570 - DB2 PS-member2.localdomain (not found) directly attached gpfs18nsd 0A0102324D80C594 - DB2 PS-member2.localdomain (not found) directly attached gpfs19nsd 0A0102324D80C61E - DB2 PS-member2.localdomain (not found) directly attached gpfs1nsd 0A0102344D959FDD /dev/dm-8 DB2 PS-member2.localdomain gpfs20nsd 0A0102324D80C641 - DB2 PS-member2.localdomain (not found) directly attached gpfs21nsd 0A0102324D80C665 - DB2 PS-member2.localdomain (not found) directly attached gpfs22nsd 0A0102324D80C68B - DB2 PS-member2.localdomain (not found) directly attached gpfs23nsd 0A0102344D95E0ED /dev/dm-1 DB2 PS-member2.localdomain gpfs24nsd 0A0102344D95E0FF /dev/dm-12 DB2 PS-member2.localdomain gpfs25nsd 0A0102344D95E112 /dev/dm-2 DB2 PS-member2.localdomain
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
77/168
gpfs26nsd 0A0102344D95E126 /dev/dm-13 DB2 PS-member2.localdomain gpfs27nsd 0A0102344D95E139 /dev/dm-3 DB2 PS-member2.localdomain gpfs28nsd 0A0102344D95E150 /dev/dm-14 DB2 PS-member2.localdomain gpfs29nsd 0A0102344D95E166 /dev/dm-4 DB2 PS-member2.localdomain gpfs2nsd 0A0102324D80C2BA - DB2 PS-member2.localdomain (not found) directly attached gpfs30nsd 0A0102344D95E180 /dev/dm-15 DB2 PS-member2.localdomain gpfs31nsd 0A0102344D95E19A /dev/dm-10 DB2 PS-member2.localdomain gpfs33nsd 0A0102344D95E1D3 /dev/dm-5 DB2 PS-member2.localdomain gpfs34nsd 0A0102344D95E1F1 /dev/dm-17 DB2 PS-member2.localdomain gpfs35nsd 0A0102344D95E210 /dev/dm-6 DB2 PS-member2.localdomain gpfs36nsd 0A0102344D95E230 /dev/dm-18 DB2 PS-member2.localdomain gpfs37nsd 0A0102344D95E253 /dev/dm-7 DB2 PS-member2.localdomain gpfs39nsd 0A0102344D95E29C /dev/dm-9 DB2 PS-member2.localdomain gpfs3nsd 0A0102324D80C412 - DB2 PS-member2.localdomain (not found) directly attached gpfs4nsd 0A0102324D80C424 - DB2 PS-member2.localdomain (not found) directly attached gpfs5nsd 0A0102324D80C438 - DB2 PS-member2.localdomain (not found) directly attached gpfs6nsd 0A0102324D80C449 - DB2 PS-member2.localdomain (not found) directly attached gpfs7nsd 0A0102324D80C45F - DB2 PS-member2.localdomain (not found) directly attached gpfs8nsd 0A0102324D80C475 - DB2 PS-member2.localdomain (not found) directly attached gpfs9nsd 0A0102324D80C489 - DB2 PS-member2.localdomain (not found) directly attached
Re-start the Metro-Mirror replication session (DP2_PS) from Primary (H1) to DR site (H2) Login to the TPCR Web GUI and Start the replication session in the original direction:
o select session DP2PS_PPRC -> Start H1 ---> H2
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
78/168
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
79/168
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
80/168
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
81/168
o Once the MM session complete synchronizing all the copy-pairs defined in the session, the replication Session state will change from “Preparing” to “Prepared ” and Copy progress will be 100%.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
82/168
For details about the the script used to re-map the replicated devices back to the NSD meta-
data that was imported as part of DR site setup process, please refer to the appendix section of
this document.
Same for the script used to mask those Metro-Mirror replicated NSD definitions from the DR site
GPFS cluster by creating a nsd devices user-exit and initiating mmdevdiscover to rescan the
devices from GPFS.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
83/168
6.4 Storage interconnect failure and catchup
The objective of this test was to measure the effect on the TRBK Day run of a storage
interconnect failure and the further impact of the storage replication resynchronization
Methodology
• DS8800 Storage replication is in operation between the DS8800’s on the primary and
secondary sites
• TRBK Day run executing on the primary site, with workload of approximately 20m
payitems/hour
• Connection to the ADVA device physically severed
• Consistency copy taken, primary workload continues
• After 10 minutes, the connection is manually reestablished
• Time measured until TPCR shows storage replication shows catchup
6.4.1 Implementation
Primary site: Primary SAP® transaction banking. The TRBK day run is running.
Disaster recovery site: Storage mirroring in Synchronous mode (DS8800 Metro Mirror )
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
84/168
Configuration for Storage Interconnect
l ll l
Storage replication
l
DB : D3D DB : D3D
Member0 Member1 cf0 Member2 Member3 cf1
QcQ
l
m
m
m
m
SAP®
50 51 54 52 53 55
SAP®
DS8800
Storage
10.7.21.0
10.1.2.0 10.1.21.0
10.1.3.0 10.1.3.0
Latency
Latency
10.9.9.0 10.9.9.0
DS8800
Admin network attached to all pieces
DB : D3D
fenced
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
85/168
Configuration on Production site
ADVA FSP 3000
ADVA Optical Networking’s FSP 3000 is a scalable, modular optical transport solution based on
Dense Wavelength Division Multiplexing (DWDM) technology, providing both Optical and
Ethernet end-to-end connectivity (from the access to the metro and on to long haul).
The DWDM technology offers almost unlimited bandwidth (up to 120 wavelengths, each
carrying 10G) and it also enables native transport of all relevant protocols in the data center
space: Ethernet, ESCON, Fibre Channel, InfiniBand.
The systems were configured so that all communication between the 2 sites passed through the
ADVA switches.
ADVA FSP 3000 product webpage: http://www.advaoptical.com/en/products/scalable-optical-
transport/fsp-3000.aspx
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
86/168
6.4.2 Storage Interconnect test mechanism
DB2 storage devices are replicated toward Recovery site using DS8000 metro mirror.
(Tivoli Storage Productivity Center for Replication is used to manage the advanced copy
services).
Once the link for the storage interconnect is lost, monitor the performance and the time to re-
sync.
6.4.3 High level process
� Procedure
• SAP® TRBK benchmark runs with the day load at 50% of performation target (20m
payitems/hour)
• Once the load is in the high load phase, remove the link between the two ADVA
equipment to stop the mirror for 10 minutes
• After the 10 minutes reconnect the link between the two ADVA equipment and
restart the metro-mirror
• For completeness, we also did a Storage Replication Disconnect/Reconnect test with
the Night run running at 12m accounts/hour
• Success Criteria
• Time to re-sync, (no more than 30 minutes, check this with business and experts
confirming the feasibility to re-sync in this time window given the amount of postings)
• No downtime, no aborts, no coredumps, no timeouts from the application side
• After re-sync, check that the secondary site is in sync without any data loss
Prerequisites
Storage mirror in Synchronous mode – all copy sets in the Metro Mirror consistency group
are in state “Prepared”
List of storage devices replicated
Running df command should return the following information
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
87/168
The replicated are highlighted in grey.
Filesystem 1K-blocks Used Available Use% Mounted
/dev/sda3 138195432 29875488 101299944 23% / Internal disk
devtmpfs 132266056 736 132265320 1% /dev Internal disk
tmpfs 132266056 0 132266056 0% /dev/shm Internal disk
/dev/sda1 72102 28760 43342 40% /boot/efi Internal disk
/dev/d3d 2147483648 1777664 2145705984 1% /db2/D3D DB2 home
/dev/db2fs1 2147483648 117846016 2029637632 6% /db2sd_20110311231413
DB2 shared instance
/dev/log0 2147483648 1930240 2145553408 1% /db2/D3D_log0 DB2 logs member0
/dev/log1 2147483648 1883136 2145600512 1% /db2/D3D_log1 DB2 logs member1
/dev/SAP® data1 1073741824 346840064 726901760 33% /db2/D3D_SAP® data1 DB2 container
/dev/SAP® data10 1073741824 346840064 726901760 33% /db2/D3D_SAP® data10 DB2 container
/dev/SAP® data11 1073741824 346839040 726902784 33% /db2/D3D_SAP® data11 DB2 container
/dev/SAP® data12 1073741824 346840064 726901760 33% /db2/D3D_SAP® data12 DB2 container
/dev/SAP® data13 1073741824 346840064 726901760 33% /db2/D3D_SAP® data13 DB2 container
/dev/SAP® data14 1073741824 346840064 726901760 33% /db2/D3D_SAP® data14 DB2 container
/dev/SAP® data15 1073741824 346840064 726901760 33% /db2/D3D_SAP® data15 DB2 container
/dev/SAP® data16 1073741824 346839040 726902784 33% /db2/D3D_SAP® data16 DB2 container
/dev/SAP® data2 1073741824 346840064 726901760 33% /db2/D3D_SAP® data2 DB2 container
/dev/SAP® data3 1073741824 346840064 726901760 33% /db2/D3D_SAP® data3 DB2 container
/dev/SAP® data4 1073741824 346840064 726901760 33% /db2/D3D_SAP® data4 DB2 container
/dev/SAP® data5 1073741824 346840064 726901760 33% /db2/D3D_SAP® data5 DB2 container
/dev/SAP® data6 1073741824 346840064 726901760 33% /db2/D3D_SAP® data6 DB2 container
/dev/SAP® data7 1073741824 346840064 726901760 33% /db2/D3D_SAP® data7 DB2 container
/dev/SAP® data8 1073741824 346840064 726901760 33% /db2/D3D_SAP® data8 DB2 container
/dev/SAP® data9 1073741824 346840064 726901760 33% /db2/D3D_SAP® data9 DB2 container
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
88/168
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
89/168
6.4.4 Storage Interconnect test results
Step 1
The TRBK day run is running and the Metro Mirror session is synchronized (prepared)
Step 2
We remove the link between the two ADVA equipments by physically disconnecting the cable
between the two switches.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
90/168
The Metro Mirror session is now in Suspended state.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
91/168
Step 3
The TRBK day run continues to run and after 10 minutes 524GB of data has been modified
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
92/168
Step 4
The fiber is reconnected and the Metro Mirror session re-started (start H1-H2).
Step 5
The session if full synchronized 10 minutes after the fiber reconnection (and re-start of the
Metro Mirror session).
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
93/168
TPCR Console Error Apr 14, 2011 5:12:19 PM : Server : IWNR2052E : The pair in session DP2PS _PPRC for copy set
DS8000:2107.WR541:VOL:1107 with source DS8000:2107.WR541:VOL:1107(SAP® data8) and target DS8000:2107.WR821:VOL:1107(SAP® data8_dr) in role pair H1-H2 was suspended with a reason code of 8 and is consistent.
normal Apr 14, 2011 5:12:30 PM : Server : IWNR6002I : Suspending all pairs in role pair H1-H2 ... normal Apr 14, 2011 5:12:30 PM : Server : IWNR1041I : The command freeze was successfully issued to all pairs
under role pair H1-H2 for session DP2PS _PPRC. normal Apr 14, 2011 5:12:30 PM : Server : IWNR6020I : Releasing I/O for all pairs in role pair H1-H2... normal Apr 14, 2011 5:12:30 PM : Server : IWNR1041I : The command thaw was successfully issued to all pairs
under role pair H1-H2 for session DP2PS _PPRC. normal Apr 14, 2011 5:12:30 PM : Server : IWNR1950I : Session DP2PS _PPRC changed from the Prepared state to
the Suspended state. normal Apr 14, 2011 5:12:30 PM : Server : IWNR6019I : Verifying the consistency of role pair H1-H2... normal Apr 14, 2011 5:26:24 PM : administrator : IWNR1028I : The command Start H1->H2 in session DP2PS
_PPRC has been run. normal Apr 14, 2011 5:26:24 PM : administrator : IWNR6000I : Starting all pairs in role pair H1-H2 ... normal Apr 14, 2011 5:26:24 PM : Server : IWNR1950I : Session DP2PS _PPRC changed from the Suspended state
to the Preparing state. normal Apr 14, 2011 5:26:25 PM : administrator : IWNR6006I : Waiting for all pairs in role pair H1-H2 to reach a
state of Prepared ... normal Apr 14, 2011 5:26:25 PM : administrator : IWNR1026I : Command Start H1->H2 in session DP2PS _PPRC
has completed successfully.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
94/168
warning Apr 14, 2011 5:26:25 PM : Server : IWNR1959W : Session DP2PS _PPRC has changed from Severe status to Warning status.
normal Apr 14, 2011 5:36:28 PM : Server : IWNR1950I : Session DP2PS _PPRC changed from the Preparing state to the Prepared state.
Test Results
• Time to resynchronize: 10 minutes
• No downtime, no aborts, no coredumps, no timeouts from the application side
• After re-sync no data loss in secondary site
• Day run disconnect time – 10 minutes
• Catchup time – 9 minutes
• Night run disconnect time – 30 minutes
• Catchup time – 10 minutes
6.5 Queue Replication to logical copy
The requirement for this test was to use Queue Replication software to maintain a logical copy
of the database that represented the state 15 minutes before the current primary state. Storage
replication propagates all changes to the primary database. The existence of a delayed logical
copy provides a mechanism to cope with and recovery from mass changes to the primary
system introduced in error.
Software levels (MQ): Websphere MQ 7.0.1-3
Queue Replication software works by monitoring the DB2 transaction log with a Capture
process, and inserting the harvested information about committed transactions to an MQ
queue. MQ causes this data to be moved to the secondary system, where it is read from the MQ
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
95/168
queue by the Apply process, and inserted/updated to the logical copy database using regular
SQL statements.
DB2 pureScale® implements a new Log Read API interface that provides a merged view of the
transaction logs of all members. This allows the Capture process to read transaction log records
in commit order (across all members), and maintains the logical consistency of the updates to
the target database.
The tables that should be maintained by Queue Replication in the logical copy are registered.
For our tests, we registered all tables referenced during our target TRBK workloads – about 2000
in total.
6.5.1 Methodology
To perform the Queue Replication test, we:
• created a third database (QREP) on the secondary system (the other 2 being the primary
database, and the storage replicated secondary database).
• The QREP database was synchronized with the primary database using a common
restore image
• Queue Replication was initialized so that all changes to the primary database would be
replicated to the logical copy
• The Night run was executed at a rate of 12m accounts per hour
• The Night run completes
Log
Member0
Member1
Log0
Log1
QRep Capture
MQ Queues
Site 1
Site 2
Membe
Membe DB
MQ Queues
QRep Apply
DB
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
96/168
6.5.2 Results
To achieve the necessary throughput we defined 2 queues (ASN1 and ASN2). Having 2 queues
implies also having 2 MQ queues, 2 Capture processes on the source system and 2 Apply
processes on the target system. As the workload starts, the Capture processes read transactions
from the DB2 transaction log via the Read Log API, and inserting these to the MQ queues. MQ
moves the messages to the target system. Due to the requirement to have a delay on the logical
copy, the Apply processes on the target do not start to remove messages from the MQ queues
for 15 minutes. This means that the MQ queues build to a considerable size (approximately
90GB). Once the Apply processes start their work, the MQ queue size stabilized for the
remainder of the run.
Qdepth (MQ messages)
020000400006000080000
100000120000140000160000
23.3
6.56
23.5
8.31
00.2
0.07
00.4
1.42
01.0
3.17
01.2
4.52
01.4
6.27
02.0
8.02
02.2
9.37
02.5
1.13
03.1
2.48
03.3
4.23
03.5
5.58
04.1
7.33
04.3
9.08
05.0
0.43
05.2
2.19
05.4
3.54
06.0
5.29
06.2
7.04
06.4
8.39
asn1
asn2
Qlatency (ms)
0
200000
400000
600000
800000
1000000
23.3
6.56
23.5
8.46
00.2
0.37
00.4
2.27
01.0
4.17
01.2
6.07
01.4
7.57
02.0
9.47
02.3
1.37
02.5
3.28
03.1
5.18
03.3
7.08
03.5
8.58
04.2
0.48
04.4
2.38
05.0
4.29
05.2
6.19
05.4
8.09
06.0
9.59
06.3
1.49
06.5
3.39
asn1
asn2
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
97/168
Rows Applied
0
200000400000
600000
800000
10000001200000
1400000
23.3
6.56
23.5
8.41
00.2
0.27
00.4
2.12
01.0
3.57
01.2
5.42
01.4
7.27
02.0
9.12
02.3
0.57
02.5
2.43
03.1
4.28
03.3
6.13
03.5
7.58
04.1
9.43
04.4
1.28
05.0
3.14
05.2
4.59
05.4
6.44
06.0
8.29
06.3
0.14
06.5
1.59
asn2
asn1
The disparity between the work processed between queues ASN1 and ASN2 is due to an
imbalance in the work done on the tables registered in each queue.
Primary Site:
• Member 0 CPU: 40%, Member 1 CPU: 28% (difference is Capture overhead)
• 1.8 billion rows inserted/member
• 115 million rows updated/member
• 1.8 million rows deleted/member
• 75k statements/member/sec
• 33.9 mb/sec/member logged
On the logical copy site, all work was applied on one member, using 100 Apply threads.
Logical copy site:
• Member 2 CPU: 55%
• 3.6 billion rows inserted
• 406 million rows updated
• 5.3 million rows deleted
• 134k statements/sec
• 66.7mb logged/sec
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
98/168
6.6 Site failure with Storage Replication and QRep
6.6.1 QRep & Site Failure Implementation
For this test, a Site Failure test was done when both Storage Replication and Queue Replication
was running. As the details of both these tests have been described individually earlier in this
document, they will not be described here, other than where this test differed
• Due to resource limitations, the QREP database was on the secondary system. In a real
world configuration, the QREP and secondary would be different systems
• The devices used by the QREP database were not Storage Replicated
• After the Site Failure was triggered by disconnecting the link between the ADVA devices,
the Apply processes were stopped and the QREP database was deactivated and
uncataloged
6.6.2 Results
The Night run was successfully restarted and ran to completion. The presence of the Queue
Replication workload and database did have any perceivable impact on the recovery process.
6.7 Backup during on-line operations
6.7.1 On line backup Implementation
Primary site : Production SAP® transaction banking
Secondary / Disaster recovery site : All Devices are up, i.e. storage devices receiving data
replication ( DS8800 Metro Mirror ) and devices used as remote FlashCopy targets.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
99/168
This document applies only to storage replication
Note:
The presence of the QREP database volumes on the Site 2 complex left us with insufficient space
on that system to accommodate the Flash Copy targets. Due to this storage space issue, we had
to reverse the roles of the sites from the original DR configuration we used for previous DR
tests., i.e. the workload was run on Site2, and the FlashCopy backup was taken on Site 1.
Configuration for Metro -Mirror replication and Remote FlashCopy
l l
Storage replication DB : D3D
Member0 Member1 cf0 Member2 Member3 cf1
m
m
SAP®
52 53 55 50 51 54
SAP®
Storage
10.7.21.0
10.1.2.0 10.1.2.0
10.1.3.0 10.1.3.0
L a t e n c y
10.9.9.0 10.9.9.0
DS8800
Admin network attached to all pieces
l lm
m
DB : D3D
l l
DB : D3D
fenced
F l a s h C o p y
DS8800
Primary / Production Site
Secondary / DR Site
FlashCopy Targets
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
100/168
6.7.2 On-line Backup mechanism
DB2 storage devices are replicated toward Recovery site using DS8000 metro mirror.
(Tivoli Storage Productivity Center for Replication is used to manage the advanced copy
services).
From the primary site, we take an online backup of the D3D database at the remote DR site
using Remote FlashCopy via the existing metro mirror session (DP2PS_PPRC). That is, the Metro
Mirror target volumes on the recovery site’s DS8K frame are used as the source volumes for the
FlashCopy targets on the same frame.
High level process of taking an On-line backup
The following diagram describes the process of taking an online backup via PPRC link directly on
the remote frame.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
101/168
� On-line Backup Methodology
• SAP® TRBK day run workload running on primary site
• All the changes to primary volumes are replicated over to recovery site via long-
distance Fibre link through ADVA (DWDM).
• Issue DB2 Suspend I/O and freeze GPFS filesystems on Primary site and issue Remote
FlashCopy create operation (mkremoteflash) to create a persistent FlashCopy
relationship with the PPRC target volumes (which is also the source for FC targets) on
the recovery site’s DS8K frame.
• Issue DB2 Resume I/O and thaw GPFS filesystems on Primary site and access the
consistent Point-In-Time (PIT) FlashCopy backup clone on the recovery site.
• This Remote FlashCopy session can be re-synched (resyncremoteflash) to refresh the
Clone volumes to create any number of PIT backups as required.
� Process to access the Clone copy on DR site
• Stop DB2, uncatalog DR copy/Q-rep copy of D3D database and shutdown GPFS daemons
• Unmap the Metro-Mirror replicated copy of the gpfs devices and map the FlashCopy
target devices (with the same NSD meta-data) on the new block devices assigned to FC
targets (see procedure in appendix )
• DB2 database recataloged on dbpath
• DB2 instance started
• DB2 crash recovery started (via activate db)
• Connect to the database and do a row count on key tables (Clone read test)
� Prerequisites on DR site
Disaster recovery site must have DB2 pureScale® installed at same level as on primary site and
DS8000 Metro Mirror
• DB2 instance created
• DB2 path created
• DS8000 metro mirror activated and all source-target copy sets are fully synchronized
(i.e. in “Prepared” state)
• Gpfs meta-data definitions are already propagated from the primary site (See Disaster
Recovery procedure doc)
• FlashCopy Target LUNs are mapped to hosts in DR site
� List of storage devices replicated (via Metro-Mirror)
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
102/168
Running df command should return the following information -- the replicated are the ones
highlighted in grey.
Filesystem 1K-blocks Used Available Use% Mounted
/dev/sda3 138195432 29875488 101299944 23% / Internal disk
devtmpfs 132266056 736 132265320 1% /dev Internal disk
tmpfs 132266056 0 132266056 0% /dev/shm Internal disk
/dev/sda1 72102 28760 43342 40% /boot/efi Internal disk
/dev/d3d 2147483648 1777664 2145705984 1% /db2/D3D DB2 home
/dev/db2fs1 2147483648 117846016 2029637632 6% /db2sd_20110311231413
DB2 shared instance
/dev/log0 2147483648 1930240 2145553408 1% /db2/D3D_log0 DB2 logs member0
/dev/log1 2147483648 1883136 2145600512 1% /db2/D3D_log1 DB2 logs member1
/dev/SAP® data1 1073741824 346840064 726901760 33% /db2/D3D_SAP® data1
DB2 container
/dev/SAP® data10 1073741824 346840064 726901760 33% /db2/D3D_SAP® data10
DB2 container
/dev/SAP® data11 1073741824 346839040 726902784 33% /db2/D3D_SAP® data11
DB2 container
/dev/SAP® data12 1073741824 346840064 726901760 33% /db2/D3D_SAP® data12
DB2 container
/dev/SAP® data13 1073741824 346840064 726901760 33% /db2/D3D_SAP® data13
DB2 container
/dev/SAP® data14 1073741824 346840064 726901760 33% /db2/D3D_SAP® data14
DB2 container
/dev/SAP® data15 1073741824 346840064 726901760 33% /db2/D3D_SAP® data15
DB2 container
/dev/SAP® data16 1073741824 346839040 726902784 33% /db2/D3D_SAP® data16
DB2 container
/dev/SAP® data2 1073741824 346840064 726901760 33% /db2/D3D_SAP® data2
DB2 container
/dev/SAP® data3 1073741824 346840064 726901760 33% /db2/D3D_SAP® data3
DB2 container
/dev/SAP® data4 1073741824 346840064 726901760 33% /db2/D3D_SAP® data4
DB2 container
/dev/SAP® data5 1073741824 346840064 726901760 33% /db2/D3D_SAP® data5
DB2 container
/dev/SAP® data6 1073741824 346840064 726901760 33% /db2/D3D_SAP® data6
DB2 container
/dev/SAP® data7 1073741824 346840064 726901760 33% /db2/D3D_SAP® data7
DB2 container
/dev/SAP® data8 1073741824 346840064 726901760 33% /db2/D3D_SAP® data8
DB2 container
/dev/SAP® data9 1073741824 346840064 726901760 33% /db2/D3D_SAP® data9
DB2 container
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
103/168
These devices are mapped to GPFS nsd ( 1 to 1 )
List of gpfs nsd
DB2PS-member0:~ # mmlsnsd
File system Disk name NSD servers
--------------------------------------------------------------------------
d3d gpfs2nsd (directly attached) DB2 base directory Replicated
db2fs1 gpfs1nsd (directly attached) DB2 shared instance Not replicated
log0 gpfs22nsd (directly attached) Logs for member0 Replicated
log1 gpfs21nsd (directly attached) Logs for member1 Replicated
SAP®
data1 gpfs16nsd (directly attached) DB2 container
Replicated
SAP®
data10 gpfs8nsd (directly attached) DB2 container
Replicated
SAP®
data11 gpfs6nsd (directly attached) DB2 container
Replicated
SAP®
data12 gpfs4nsd (directly attached) DB2 container
Replicated
SAP®
data13 gpfs3nsd (directly attached) DB2 container
Replicated
SAP®
data14 gpfs18nsd (directly attached) DB2 container
Replicated
SAP®
data15 gpfs17nsd (directly attached) DB2 container
Replicated
SAP®
data16 gpfs15nsd (directly attached) DB2 container
Replicated
SAP®
data2 gpfs14nsd (directly attached) DB2 container
Replicated
SAP®
data3 gpfs13nsd (directly attached) DB2 container
Replicated
SAP®
data4 gpfs12nsd (directly attached) DB2 container
Replicated
SAP®
data5 gpfs11nsd (directly attached) DB2 container
Replicated
SAP®
data6 gpfs10nsd (directly attached) DB2 container
Replicated
SAP®
data7 gpfs9nsd (directly attached) DB2 container
Replicated
SAP®
data8 gpfs7nsd (directly attached) DB2 container
Replicated
SAP® gpfs5nsd (directly attached) DB2 container Replicated
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
104/168
data9
GPFS configuration
GPFS configuration must be updated to map FlashCopy target volumes when accessing the clone
copy and un-map when not using them on the remote site.
Note:
• GPFS configuration is already propagated from the Primary site to the Secondary / DR
site before online backup process is started ( see Disaster Recovery procedure
document for how to propagate GPFS meta-data using mmfsctl ... syncFSconfig )
6.7.3 Online Backup Procedure
Initial status
� Check accessibility of FlashCopy target storage devices DB2PS-member0:~/alain # multipath -l | grep 0000004 fSAP® data11 (3600507630bffc06c0000000000004010) dm -40 IBM,2107900 fSAP® data10 (3600507630bffc06c0000000000004109) dm -57 IBM,2107900 fSAP® data9 (3600507630bffc06c0000000000004008) dm- 39 IBM,2107900 fdb2d3d (3600507630bffc06c0000000000004022) dm-56 I BM,2107900 fSAP® data8 (3600507630bffc06c0000000000004107) dm- 55 IBM,2107900 fSAP® data7 (3600507630bffc06c0000000000004006) dm- 45 IBM,2107900 fSAP® data6 (3600507630bffc06c0000000000004105) dm- 49 IBM,2107900 fSAP® data5 (3600507630bffc06c0000000000004004) dm- 43 IBM,2107900 fSAP® data4 (3600507630bffc06c0000000000004103) dm- 46 IBM,2107900 fSAP® data16 (3600507630bffc06c0000000000004115) dm -59 IBM,2107900 fSAP® data3 (3600507630bffc06c0000000000004002) dm- 41 IBM,2107900 flog3 (3600507630bffc06c0000000000004119) dm-60 IBM ,2107900 fSAP® data15 (3600507630bffc06c0000000000004014) dm -52 IBM,2107900 fSAP® data2 (3600507630bffc06c0000000000004101) dm- 47 IBM,2107900 flog2 (3600507630bffc06c0000000000004018) dm-50 IBM ,2107900 fSAP® data14 (3600507630bffc06c0000000000004113) dm -58 IBM,2107900 flog1 (3600507630bffc06c0000000000004117) dm-54 IBM ,2107900 fSAP® data1 (3600507630bffc06c0000000000004000) dm- 42 IBM,2107900 fSAP® data13 (3600507630bffc06c0000000000004012) dm -44 IBM,2107900 flog0 (3600507630bffc06c0000000000004016) dm-48 IBM ,2107900 fSAP® data12 (3600507630bffc06c0000000000004111) dm -51 IBM,2107900
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
105/168
� Verify GPFS daemons are up (on DR site) DB2PS-member2:~/scripts # mmgetstate -a -L Node number Node name Quorum Nodes up Tot al nodes GPFS state Remarks --------------------------------------------------- --------------------------------- 1 DB2PS-member0 1* 3 3 active quorum node 2 DB2PS-cf0 1* 3 3 active quorum node 3 DB2PS-member1 1* 3 3 active quorum node
� Filesystems already mounted (on DR site) Filesystem 1K-blocks Used Available Use% Mo unted on /dev/sda3 138195432 21604884 109570548 17% / devtmpfs 132266056 1056 132265000 1% /de v tmpfs 132266056 0 132266056 0% /dev/sh m /dev/sda1 72102 28762 43340 40% /boot/efi /dev/db2fs1 524288000 1528832 522759168 1% /db2sd_20110401115043
� DB2 installed and configured As user db2d3d check existence of instance
db2d3d@DB2PS-member0:~> db2instance -list ID TYPE STATE HOM E_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME -- ---- ----- --- ------ ------------ ----- ---------------- ------------ ------- 0 MEMBER STARTED DB2 PS-member0 DB2PS-member0 NO 0 0 DB2PS-member0-ib0 1 MEMBER STARTED DB2 PS-member1 DB2PS-member1 NO 0 0 DB2PS-member1-ib0 128 CF PRIMARY DB2 PS-cf0 DB2PS-cf0 NO - 0 DB2PS-cf0-ib0 HOSTNAME STATE INS TANCE_STOPPED ALERT -------- ----- --- ------------- ----- DB2PS-cf0 ACTIVE NO NO DB2PS-member0 ACTIVE NO NO DB2PS-member1 ACTIVE NO NO
Check that node state is active.
� Replication
Replication is in full-duplex state (i.e. source and target copy sets are in full sync)
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
106/168
o Check that storage replication is in status Normal
o Query the pairing status of all the copy sets in the PPRC Consistency Group /opt/ibm/dscli/dscli [-cfg Primary_Storage_profile] lspprc -l -dev Primary_Storage_Image_ID -remotedev Standby_Storage _Image_ID Range_of_Source_volume_IDs
o Login to DB2PS-member2 as root and run the following DSCLI command:
DB2PS-member2:~/scripts # ./PR_run_dscli.sh lspprc -l -dev IBM.2107-75WR821 -remotedev IBM.2107-75WR541 0000-FFFF Date/Time: May 6, 2011 9:36:26 PM CEST IBM DSCLI Ve rsion: 6.6.0.305 DS: IBM.2107-75WR821 ID State Reason Type Out Of Sync Tracks Tgt R ead Src Cascade Tgt Cascade Date Suspended SourceLSS Timeou t (secs) Critical Mode First Pass Status Incremental Resync Tgt Write GMIR CG PPRC CG isTgtSE DisableAutoResync =================================================== ================================================================================ ================================================================================ ========================= 1000:1000 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1002:1002 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1004:1004 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1006:1006 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1008:1008 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1010:1010 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1012:1012 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1014:1014 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1016:1016 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1018:1018 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1022:1022 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1101:1101 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1103:1103 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown -
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
107/168
1105:1105 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1107:1107 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1109:1109 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1111:1111 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1113:1113 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1115:1115 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1117:1117 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1119:1119 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown –
o PPRC CG column in the above table for all the SOURCE:TARGET volume pairs (copy sets)
should be set to ‘Enable’.
6.7.4 Create Online backup using Remote FlashCopy
Do the following steps on the primary production site.
� Start SAP® TRBK day run Start SAP® TRBK day run performance benchmark workload and leave it running till SAP®
throughput enters the high load phase
� Create Remote FlashCopy relationship On the primary production site
� Issue DB2 suspend I/O On DB2PS-member2 as db2 instance owner (db2d3d), issue DB2 suspend I/O on the D3D
database :
db2d3d@DB2PS-member2:~/sqllib/db2dump> db2 -v conne ct to d3d db2d3d@DB2PS-member2:~/sqllib/db2dump> db2 -v set w rite suspend for database
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
108/168
� Issue GPFS I/O suspend Issue GPFS I/O suspend on the file systems used as dbpath and database storage paths
mmfsctl <gpfs filesystem name> suspend
- Issue the above command on all GPFS file systems used by D3D db
- The following script with ‘suspend’ option was used to execute GPFS filesystem freeze ( see
appendix ):
/root/scripts/suspend_or_resume_gpfs_fs.sh suspend
� Create Remote FlashCopy Create a permanently attached (persistent) Remote FlashCopy session
dscli mkremoteflash -dev storage_image_ID -conduit source_storage_image_ID/LSS_ID -freeze
-record -persist source_volume_ID:target_volume_ID
- The following script with ‘mkremoteflash’ option was used to create the Remote FlashCopy
relationship ( see appendix ):
/root/scripts/remote_flash_copy.sh mkremoteflash
Note: To prevent TPC-R from suspending the PPRC session, immediately issue ‘unfreezeflash’
command on the remote frame. Refer to the notes under appendix for the above script for more
details.
� Issue GPFS I/O resume Resume I/O to suspended GPFS file systems
mmfsctl <gpfs filesystem name> resume
- Issue the above command on all GPFS file systems previously frozen
- The following script with ‘resume’ option was used to execute GPFS filesystem thaw (see
appendix):
/root/scripts/suspend_or_resume_gpfs_fs.sh resume
Issue DB2 resume I/O Resume I/O to D3D database
db2d3d@DB2PS-member2:~/sqllib/db2dump> db2 -v set w rite resume for database
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
109/168
In order to measure the performance impact of this process on the primary site, steps 3.2.2.1-
3.2.2.5 were grouped into a single script and issued as db2 instance owner “db2d3d” from
DB2PS-member2: db2d3d@DB2PS-member2:~/scripts> ./get_snapshot.sh Contents of the above script:
# We start the snapshot copy here outfile="./get_snapshot.out.`date +%F-%H:%M:%S`" echo "Starting snapshotcopy process at `date` ..." > $outfile db2 -v connect to d3d >> $outfile date >> $outfile db2 -v set write suspend for db >> $outfile date >> $outfile ssh root@DB2PS-member2 ". /root/.profile; /root/scripts/suspend_or_resume_gpfs_fs.sh suspend; date; /root/scripts/remote_flash_copy.sh mkremoteflash; d ate; /root/scripts/suspend_or_resume_gpfs_fs.sh resume" >> $outfile date >> $outfile db2 -v set write resume for db >> $outfile date >> $outfile db2 -v connect reset >> $outfile echo "Ending snapshotcopy process at `date` ..." >> $outfile
Note:
• Password-less SSH into root from db2d3d ID is required to run the above script.
From another window we examined the output of the above script
db2d3d@DB2PS-member2:~/scripts> tail -f get_snapshot.out.2011-05-07-00:19:35 Starting snapshotcopy process at Sat May 7 00:19:3 5 CEST 2011 ... connect to d3d Database Connection Information Database server = DB2/LINUXX8664 9.8.4 SQL authorization ID = DB2D3D Local database alias = D3D Sat May 7 00:19:36 CEST 2011 set write suspend for db DB20000I The SET WRITE command completed successfu lly. Sat May 7 00:20:39 CEST 2011 suspend Issuing GPFS I/O suspend to filesystems used by dat abase D3D ... Writing dirty data to disk Quiescing all file system operations Writing dirty data to disk again …/…
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
110/168
Quiescing all file system operations Writing dirty data to disk again Sat May 7 00:20:42 CEST 2011 mkremoteflash Executing the following DSCLI command for user opti on mkremoteflash: /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile mkremoteflash -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/10 -free ze -record -persist 1000:4000 1002:4002 1004:4004 1006:4006 1008:4008 1 010:4010 1012:4012 1014:4014 1016:4016 1018:4018 1022:4022 Date/Time: May 7, 2011 12:20:43 AM CEST IBM DSCLI V ersion: 6.6.0.305 DS: IBM.2107-75WR541 CMUC00173I mkremoteflash: Remote FlashCopy volume p air 1000:4000 successfully created. Use the lsremoteflash command to determine copy completion. …/… CMUC00173I mkremoteflash: Remote FlashCopy volume p air 1022:4022 successfully created. Use the lsremoteflash command to determine copy completion. /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile mkremoteflash -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/11 -free ze -record -persist 1101:4101 1103:4103 1105:4105 1107:4107 1109:4109 1 111:4111 1113:4113 1115:4115 1117:4117 1119:4119 Date/Time: May 7, 2011 12:20:46 AM CEST IBM DSCLI V ersion: 6.6.0.305 DS: IBM.2107-75WR541 CMUC00173I mkremoteflash: Remote FlashCopy volume p air 1101:4101 successfully created. Use the lsremoteflash command to determine copy completion. …/… CMUC00173I mkremoteflash: Remote FlashCopy volume p air 1119:4119 successfully created. Use the lsremoteflash command to determine copy completion. Date/Time: May 7, 2011 12:20:48 AM CEST IBM DSCLI V ersion: 6.6.0.305 DS: IBM.2107-75WR541 CMUC00172I unfreezeflash: FlashCopy consistency gro up for logical subsystem 10 successfully reset. CMUC00172I unfreezeflash: FlashCopy consistency gro up for logical subsystem 11 successfully reset. Sat May 7 00:20:50 CEST 2011 resume Issuing GPFS I/O resume to filesystems used by data base D3D ... Resuming operations. …/… Resuming operations. Sat May 7 00:20:53 CEST 2011 set write resume for db DB20000I The SET WRITE command completed successfu lly. Sat May 7 00:20:53 CEST 2011 connect reset DB20000I The SQL command completed successfully. Ending snapshotcopy process at Sat May 7 00:20:53 CEST 2011 ...
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
111/168
Accessing FlashCopy database Clone (from DR site) After successfully creating the Remote FlashCopy relationship, do the following steps on the
Secondary / DR site to access the FlashCopy database clone.
� Shutdown DB2 On DB2PS-member0 as DB2 instance ID (db2d3d), stop DB2:
db2d3d@DB2PS-member0:~/sqllib/db2dump> db2stop force
� Shutdown GPFS Unmount all the GPFS file systems seen to the cluster (including the filesystem used by
db2instance)
DB2PS-member0:~/scripts # mmumount -a Note: Make sure that above unmount command is successful. Any open file handles can be
cleaned by using fuser –kmu /dev/<fs name>
Shutdown GPFS daemons
DB2PS-member0:~/scripts # mmshutdown -a
Check all GPFS daemons are down in the cluster by issuing:
DB2PS-member0:~/scripts # mmgetstate -a –L
Note:
• Since, the FlashCopy Targets are identical (including GPFS meta-data content) to the
source, it is recommended to shutdown GPFS before unmapping and remapping block
devices ( /dev/dm-# ) backed by GPFS nsds to avoid hosts using a cached copy or any
stale data – i.e. flush hosts file system cache before accessing FC target volumes.
� Unmap PPRC targets and map FlashCopy Targets Mask (unmap) the PPRC’d NSDs (FlashCopy source) and map FlashCopy target volumes as the
NSDs.
FC target volumes will have all the tacks copied from the source including NSD header (first 10
sectors). As long as the underlying block device contains all the NSD meta-data known to the
cluster, GPFS does not have an ability to detect that the block device paths were remapped to a
different set of DM MP devices. Key thing to remember is, we do this masking using an
nsddevices user-exit which is file local to each node. Hence this remap process needs to be done
on all nodes in the cluster.
DB2PS-member0:~/scripts # /scripts/disable_dr_nsd_a nd_enable_fc_nsd.sh
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
112/168
DB2PS-member0:~/scripts # ssh root@DB2PS-member1 ". /root/.profile; /root/scripts/disable_dr_nsd_and_enable_fc_nsd.sh" DB2PS-member0:~/scripts # ssh root@DB2PS-cf0 ". /ro ot/.profile; /root/scripts/disable_dr_nsd_and_enable_fc_nsd.sh"
See Appendix for the contents of this script.
� Start GPFS Start GPFS daemons
DB2PS-member0:~/scripts # mmstartup -a Sat May 7 00:41:28 CEST 2011: mmstartup: Starting GPFS ... Check the state of the GPFS daemons
DB2PS-member0:~/scripts # mmgetstate -a -L Node number Node name Quorum Nodes up Tot al nodes GPFS state Remarks --------------------------------------------------- --------------------------------- 1 DB2PS-member0 1* 3 3 active quorum node 2 DB2PS-cf0 1* 3 3 active quorum node 3 DB2PS-member1 1* 3 3 active quorum node
List the NSD-to-block device (DM MP device) mappings to verify that GPFS has now picked up the FlashCopy target
LUNs
DB2PS-member0:~/scripts # mmlsnsd -X Disk name NSD volume ID Device Dev type Node name Remarks --------------------------------------------------- ------------------------------------------------ gpfs10nsd 0A0102324D80C4A2 /dev/dm-52 dmm DB2PS-member0.localdomain gpfs11nsd 0A0102324D80C4BC /dev/dm-41 dmm DB2PS-member0.localdomain gpfs12nsd 0A0102324D80C4D9 /dev/dm-51 dmm DB2PS-member0.localdomain gpfs13nsd 0A0102324D80C4F5 /dev/dm-40 dmm DB2PS-member0.localdomain gpfs14nsd 0A0102324D80C512 /dev/dm-50 dmm DB2PS-member0.localdomain gpfs15nsd 0A0102324D80C530 /dev/dm-57 dmm DB2PS-member0.localdomain gpfs16nsd 0A0102324D80C54F /dev/dm-39 dmm DB2PS-member0.localdomain gpfs17nsd 0A0102324D80C570 /dev/dm-46 dmm DB2PS-member0.localdomain gpfs18nsd 0A0102324D80C594 /dev/dm-56 dmm DB2PS-member0.localdomain
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
113/168
gpfs19nsd 0A0102324D80C61E /dev/dm-59 dmm DB2PS-member0.localdomain gpfs1nsd 0A0102324D7A9E9F /dev/dm-33 dmm DB2PS-member0.localdomain gpfs20nsd 0A0102324D80C641 /dev/dm-48 dmm DB2PS-member0.localdomain gpfs21nsd 0A0102324D80C665 /dev/dm-58 dmm DB2PS-member0.localdomain gpfs22nsd 0A0102324D80C68B /dev/dm-47 dmm DB2PS-member0.localdomain gpfs2nsd 0A0102324D80C2BA /dev/dm-49 dmm DB2PS-member0.localdomain gpfs3nsd 0A0102324D80C412 /dev/dm-45 dmm DB2PS-member0.localdomain gpfs4nsd 0A0102324D80C424 /dev/dm-55 dmm DB2PS-member0.localdomain gpfs5nsd 0A0102324D80C438 /dev/dm-43 dmm DB2PS-member0.localdomain gpfs6nsd 0A0102324D80C449 /dev/dm-44 dmm DB2PS-member0.localdomain gpfs7nsd 0A0102324D80C45F /dev/dm-53 dmm DB2PS-member0.localdomain gpfs8nsd 0A0102324D80C475 /dev/dm-54 dmm DB2PS-member0.localdomain gpfs9nsd 0A0102324D80C489 /dev/dm-42 dmm DB2PS-member0.localdomain
Notes:
• NSDs highlighted in gray are not part of the FlashCopy relationship and hence the block
device mappings remain unchanged.
• Since GPFS file systems are configured to Auto-Mount (during GPFS startup) on the
primary site, the replicated NSD meta-data in the FC target volumes are also set with
Auto-Mount. Therefore there is no need to manually mount the GPFS file systems on FC
target NSDs.
� Bring FlashCopy clone of D3D database online
o Catalog FlashCopy clone of D3D database On DB2PS-member0 as DB2 instance ID (db2d3d), catalog D3D db
db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 catalog db d3d on /db2/D3D DB20000I The CATALOG DATABASE command completed su ccessfully.
db2d3d@DB2PS-member0:~> db2 list db directory System Database Directory Number of entries in the directory = 1 Database 1 entry: Database alias = D3D Database name = D3D
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
114/168
Local database directory = /db2/D3D Database release level = e.00 Comment = Directory entry type = Indirect Catalog database partition number = 0 Alternate server hostname = Alternate server port number =
o Start instance db2d3d On DB2PS-member0 as DB2 instance ID (db2d3d), start DB2 db2d3d@DB2PS-member0:~> db2start ADM12026W The DB2 server has detected that a valid license for the product "DB2 Enterprise Server Edition" has not been registered. 05/07/2011 00:47:26 0 0 SQL1063N DB2START processing was successful. 05/07/2011 00:47:27 1 0 SQL1063N DB2START processing was successful.
o Initialize the FlashCopy clone of D3D database On DB2PS-member0 as DB2 instance ID (db2d3d), initialize the FC copy of D3D database as a
snapshot
db2d3d@DB2PS-member0:~> db2inidb d3d as snapshot
The above ‘db2inidb’ command would trigger an implicit GCR operation, status of which can be monitored
via,
db2d3d@DB2PS-member0:~/sqllib/db2dump> db2pd -db d3d -recovery Database Member 0 -- Database D3D -- Active -- Up 0 days 00:04:35 -- Date 05/07/2011 00:52:02 Recovery: Recovery Status 0x410A8015 Current Log Current LSN 000003E9CF6608EA Current LRI 000000000000000100000000041F648 A000003E9CF6608EA Current LSO 4401095066827 Job Type GROUP CRASH RECOVERY Job ID 1 Job Start Time (1304722286) Sat May 7 00:51:2 6 2011 Job Description Group Crash Recovery Invoker Type User Total Phases 2 Current Phase 1 Progress: Address PhaseNum Description StartTime CompletedWork TotalWork 0x0000000204A6FBC8 1 Forward Sat May 7 00:51:26 132901367 bytes 5474720160 bytes 0x0000000204A6FD50 2 Backward NotStarted 0 bytes 5474720160 bytes
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
115/168
The DB2 diaglog should indicate when the above GCR completes
2011-05-07-01.02.25.640751+120 E213622E450 LEVEL: Info PID : 34391 TID : 4699686174080 0 PROC : db2sysc 0 INSTANCE: db2d3d NODE : 000 DB : D3D APPHDL : 0-52 APPID: *N0.DB2.11050 6224727 EDUID : 35 EDUNAME: db2agnti (D 3D ) 0 FUNCTION: DB2 UDB, recovery manager, sqlpresr, prob e:3110 MESSAGE : ADM1528I Group crash recovery has comple ted successfully.
� Clone read test After successful ‘db2inidb’ operation, activate and connect to the FlashCopy clone of D3D
database and perform a row-count of few key tables.
db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 activate db d3d DB20000I The ACTIVATE DATABASE command completed s uccessfully.
db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 connect to d3d Database Connection Information Database server = DB2/LINUXX8664 9.8.4 SQL authorization ID = DB2D3D Local database alias = D3D db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 "select count(*) from SAP® d3d.BCA_PAYMITEM with ur" 1 ----------- 1807164585 1 record(s) selected.
6.7.5 Refresh Online backup clone using Remote FlashCopy
� Replication
Verify Metro-Mirror (PPRC) replication session is in sync via TPC-R GUI or by issuing DSCLI
command ‘lspprcpath’, and do the following steps on the primary production site.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
116/168
� Start TRBK day run an leave it running till SAP® throughput is stable
� Refresh Remote FlashCopy targets On the primary production site
� Issue DB2 suspend I/O On DB2PS-member2 as db2 instance owner (db2d3d), issue DB2 suspend I/O on the D3D
database :
db2d3d@DB2PS-member2:~/sqllib/db2dump> db2 -v connec t to d3d db2d3d@DB2PS-member2:~/sqllib/db2dump> db2 -v set w rite suspend for database
� Issue GPFS I/O suspend Issue GPFS I/O suspend on the file systems used as dbpath and database storage paths
mmfsctl <gpfs filesystem name> suspend
- Issue the above command on all GPFS file systems used by D3D db
- The following script with ‘suspend’ option was used to execute GPFS filesystem freeze (see
appendix):
/root/scripts/suspend_or_resume_gpfs_fs.sh suspend
� Re-sync Remote FlashCopy Create a permanently attached (persistent) Remote FlashCopy session
dscli resyncremoteflash -dev storage_image_ID -cond uit source_storage_image_ID/LSS_ID -freeze
-record -persist source_volume_ID:target_volume_ID
- The following script with ‘remoteflash’ option was used to re-sync the Remote FlashCopy
relationship (see appendix):
/root/scripts/remote_flash_copy.sh remoteflash
Note: To prevent TPC-R from suspending the PPRC session, immediately issue ‘unfreezeflash’
command on the remote frame. Refer to the notes under appendix for the above script for more
details.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
117/168
� Issue GPFS I/O resume Resume I/O to suspended GPFS file systems
mmfsctl <gpfs filesystem name> resume
- Issue the above command on all GPFS file systems used by D3D db
- The following script with ‘resume’ option was used to execute GPFS filesystem thaw (see
appendix):
/root/scripts/suspend_or_resume_gpfs_fs.sh resume
� Issue DB2 resume I/O Resume I/O to D3D database
db2d3d@DB2PS-member2:~/sqllib/db2dump> db2 -v set w rite resume for database In order to measure the performance impact of this process on primary site, steps 3.4.3.1-
3.4.3.5 were grouped into a single script and issued as db2 instance owner “db2d3d” from
DB2PS-member2 : db2d3d@DB2PS-member2:~/scripts> ./refresh_snapshot.sh Contents of the above script:
# We start the snapshot copy here outfile="./refresh_snapshot.out.`date +%F-%H:%M:%S` " echo "Starting snapshotcopy process at `date` ..." > $outfile db2 -v connect to d3d >> $outfile date >> $outfile db2 -v set write suspend for db >> $outfile date >> $outfile ssh root@DB2PS-member2 ". /root/.profile; /root/scripts/suspend_or_resume_gpfs_fs.sh suspend; date; /root/scripts/remote_flash_copy.sh remoteflash; dat e; /root/scripts/suspend_or_resume_gpfs_fs.sh resume" >> $outfile date >> $outfile db2 -v set write resume for db >> $outfile date >> $outfile db2 -v connect reset >> $outfile echo "Ending snapshotcopy process at `date` ..." >> $outfile Note:
• Password-less SSH into root from db2d3d ID is required to run the above script.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
118/168
From another window we examined the output of the above script
db2d3d@DB2PS-member2:~/scripts> tail -f refresh_snapshot.out.2011-05-07-16:31:27 Starting snapshotcopy process at Sat May 7 16:31:2 7 CEST 2011 ... connect to d3d Database Connection Information Database server = DB2/LINUXX8664 9.8.4 SQL authorization ID = DB2D3D Local database alias = D3D Sat May 7 16:31:28 CEST 2011 set write suspend for db DB20000I The SET WRITE command completed successfu lly. Sat May 7 16:42:25 CEST 2011 suspend Issuing GPFS I/O suspend to filesystems used by dat abase D3D ... Writing dirty data to disk Quiescing all file system operations Writing dirty data to disk again …/… Writing dirty data to disk Quiescing all file system operations Writing dirty data to disk again Sat May 7 16:42:28 CEST 2011 remoteflash Executing the following DSCLI command for user opti on remoteflash: /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile resyncremoteflash -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/10 -free ze -record -persist 1000:4000 1002:4002 1004:4004 1006:4006 1008:4008 1 010:4010 1012:4012 1014:4014 1016:4016 1018:4018 1022:4022 Date/Time: May 7, 2011 4:42:29 PM CEST IBM DSCLI Ve rsion: 6.6.0.305 DS: IBM.2107-75WR541 CMUC00175I resyncremoteflash: Remote FlashCopy volu me pair 1000:4000 successfully resynchronized. Use the lsremoteflash command to determine copy completion. …/… CMUC00175I resyncremoteflash: Remote FlashCopy volu me pair 1022:4022 successfully resynchronized. Use the lsremoteflash command to determine copy completion. /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile resyncremoteflash -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/11 -free ze -record -persist 1101:4101 1103:4103 1105:4105 1107:4107 1109:4109 1 111:4111 1113:4113 1115:4115 1117:4117 1119:4119 Date/Time: May 7, 2011 4:42:32 PM CEST IBM DSCLI Ve rsion: 6.6.0.305 DS: IBM.2107-75WR541 CMUC00175I resyncremoteflash: Remote FlashCopy volu me pair 1101:4101 successfully resynchronized. Use the lsremoteflash command to determine copy completion. …/…
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
119/168
CMUC00175I resyncremoteflash: Remote FlashCopy volu me pair 1119:4119 successfully resynchronized. Use the lsremoteflash command to determine copy completion. Date/Time: May 7, 2011 4:42:34 PM CEST IBM DSCLI Ve rsion: 6.6.0.305 DS: IBM.2107-75WR541 CMUC00172I unfreezeflash: FlashCopy consistency gro up for logical subsystem 10 successfully reset. CMUC00172I unfreezeflash: FlashCopy consistency gro up for logical subsystem 11 successfully reset. Sat May 7 16:42:36 CEST 2011 resume Issuing GPFS I/O resume to filesystems used by data base D3D ... Resuming operations. …/… Resuming operations. Sat May 7 16:42:39 CEST 2011 set write resume for db DB20000I The SET WRITE command completed successfu lly. Sat May 7 16:42:39 CEST 2011 connect reset DB20000I The SQL command completed successfully. Ending snapshotcopy process at Sat May 7 16:42:39 CEST 2011 ...
� Query status of Clone copy refresh You can query the status of the FlashCopy progress to figure out when the targets achieve the
full copied state. Note that, since FC will copy the tracks in background, and will use Copy-On-
Access (COA), you do not need to wait till targets are fully synced to access the data on them.
However, there may be slight performance degradation whenever a track that is not yet copied
is accessed.
DB2PS-member2:~/scripts # ./remote_flash_copy.sh ls remoteflash lsremoteflash Executing the following DSCLI command for user opti on lsremoteflash: /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile lsremoteflash -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/10 -l 1 000:4000 1002:4002 1004:4004 1006:4006 1008:4008 1010:4010 1012:4012 1014:4014 1 016:4016 1018:4018 1022:4022 Date/Time: May 7, 2011 5:01:52 PM CEST IBM DSCLI Version: 6.6.0.305 DS: IBM.2107-75WR541 ID SrcLSS SequenceNum ActiveCopy Recording P ersistent Revertible SourceWriteEnabled TargetWriteEnabled BackgroundCop y OutOfSyncTracks DateCreated DateSynced State isTgtSE Pmir =================================================== ================================================================================ ================================================================================ ======== 1000:4000 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1002:4002 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1004:4004 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
120/168
1006:4006 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1008:4008 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1010:4010 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1012:4012 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1014:4014 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1016:4016 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1018:4018 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1022:4022 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile lsremoteflash -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/11 -l 1 101:4101 1103:4103 1105:4105 1107:4107 1109:4109 1111:4111 1113:4113 1115:4115 1 117:4117 1119:4119 Date/Time: May 7, 2011 5:01:55 PM CEST IBM DSCLI Ve rsion: 6.6.0.305 DS: IBM.2107-75WR541 ID SrcLSS SequenceNum ActiveCopy Recording P ersistent Revertible SourceWriteEnabled TargetWriteEnabled BackgroundCop y OutOfSyncTracks DateCreate d DateSynced State isTgtSE Pmir =================================================== ================================================================================ ================================================================================ ======== 1101:4101 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1103:4103 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1105:4105 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1107:4107 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1109:4109 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1111:4111 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1113:4113 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1115:4115 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
121/168
1117:4117 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1119:4119 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No
Notes:
• From output above:
o DateSynced : Indicated the timestamp of either a create FC command or a
resync FC command was issued ( May 07 16:43:54 CEST 2011 )
o OutOfSyncTracks : Indicate how many tracks are copy pending. This
column value needs to reach ‘0’ to achieve fully sync status. We reached this
fully sync state at May 7, 2011 5:01:52 PM CEST.
• Therefore time for FC target to get 100% copied is ~ 17min.
6.7.6 Accessing FlashCopy database Clone (from DR site)
After successfully re-syncing the Remote FlashCopy relationship, do the following steps on the
Secondary / DR site to access the clone copy.
� Shutdown DB2 On DB2PS-member0 as DB2 instance ID (db2d3d), stop DB2:
db2d3d@DB2PS-member0:~/sqllib/db2dump> db2stop force
� Shutdown GPFS Unmount all the GPFS file systems seen to the cluster (including the filesystem used by
db2instance)
DB2PS-member0:~/scripts # mmumount -a Note: Make sure that above unmount command is successful. Any open file handles can be
cleaned by using fuser –kmu /dev/<fs name>
Shutdown GPFS daemons
DB2PS-member0:~/scripts # mmshutdown -a
Check all GPFS daemons are down in the cluster by issuing:
DB2PS-member0:~/scripts # mmgetstate -a -L
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
122/168
Note:
• It is recommended to flush hosts file system cache before accessing refreshed FC target
volumes to avoid accessing any stale data – i.e. stop/start GPFS.
� Unmap PPRC targets and map FlashCopy Targets Mask (unmap) the PPRC’d NSDs (FlashCopy source) and map FlashCopy taget volumes as the
NSDs.
FC target volumes will have all the tacks copied from the source including NSD header (first 10
sectors). As long as the underlying block device contains all the NSD meta-data known to the
cluster, GPFS does not have an ability to detect that the block device paths were remapped to a
different set of DM MP devices. Key thing to remember is, we do this masking using an
nsddevices user-exit which is file local to each node. Hence this remap process needs to be done
on all nodes in the cluster.
DB2PS-member0:~/scripts # disable_dr_nsd_and_enable _fc_nsd.sh DB2PS-member0:~/scripts # ssh root@DB2PS-member1 ". /root/.profile; /root/scripts/disable_dr_nsd_and_enable_fc_nsd.sh" DB2PS-member0:~/scripts # ssh root@DB2PS-cf0 ". /ro ot/.profile; /root/scripts/disable_dr_nsd_and_enable_fc_nsd.sh"
See Appendix for the contents of this script.
Note:
o This step should already have been done when the remote FC was created. Unless there
was a change that required reverting back to the source volumes (for example like a
disaster recovery operation), there is no need to execute this step again.
� Start GPFS Start GPFS daemons
DB2PS-member0:~/scripts # mmstartup -a
Check the state of the GPFS daemons
DB2PS-member0:~/scripts # mmgetstate -a -L Node number Node name Quorum Nodes up Tot al nodes GPFS state Remarks --------------------------------------------------- --------------------------------- 1 DB2PS-member0 1* 3 3 active quorum node 2 DB2PS-cf0 1* 3 3 active quorum node
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
123/168
3 DB2PS-member1 1* 3 3 active quorum node
List the NSD-to-block device (DM MP device) mappings to verify that GPFS has now picked up
the FlashCopy target LUNs
DB2PS-member0:~/scripts # mmlsnsd -X Disk name NSD volume ID Device Dev type Node name Remarks --------------------------------------------------- ------------------------------------------------ gpfs10nsd 0A0102324D80C4A2 /dev/dm-52 dmm DB2PS-member0.localdomain gpfs11nsd 0A0102324D80C4BC /dev/dm-41 dmm DB2PS-member0.localdomain gpfs12nsd 0A0102324D80C4D9 /dev/dm-51 dmm DB2PS-member0.localdomain gpfs13nsd 0A0102324D80C4F5 /dev/dm-40 dmm DB2PS-member0.localdomain gpfs14nsd 0A0102324D80C512 /dev/dm-50 dmm DB2PS-member0.localdomain gpfs15nsd 0A0102324D80C530 /dev/dm-57 dmm DB2PS-member0.localdomain gpfs16nsd 0A0102324D80C54F /dev/dm-39 dmm DB2PS-member0.localdomain gpfs17nsd 0A0102324D80C570 /dev/dm-46 dmm DB2PS-member0.localdomain gpfs18nsd 0A0102324D80C594 /dev/dm-56 dmm DB2PS-member0.localdomain gpfs19nsd 0A0102324D80C61E /dev/dm-59 dmm DB2PS-member0.localdomain gpfs1nsd 0A0102324D7A9E9F /dev/dm-33 dmm DB2PS-member0.localdomain gpfs20nsd 0A0102324D80C641 /dev/dm-48 dmm DB2PS-member0.localdomain gpfs21nsd 0A0102324D80C665 /dev/dm-58 dmm DB2PS-member0.localdomain gpfs22nsd 0A0102324D80C68B /dev/dm-47 dmm DB2PS-member0.localdomain gpfs2nsd 0A0102324D80C2BA /dev/dm-49 dmm DB2PS-member0.localdomain gpfs3nsd 0A0102324D80C412 /dev/dm-45 dmm DB2PS-member0.localdomain gpfs4nsd 0A0102324D80C424 /dev/dm-55 dmm DB2PS-member0.localdomain gpfs5nsd 0A0102324D80C438 /dev/dm-43 dmm DB2PS-member0.localdomain gpfs6nsd 0A0102324D80C449 /dev/dm-44 dmm DB2PS-member0.localdomain gpfs7nsd 0A0102324D80C45F /dev/dm-53 dmm DB2PS-member0.localdomain gpfs8nsd 0A0102324D80C475 /dev/dm-54 dmm DB2PS-member0.localdomain gpfs9nsd 0A0102324D80C489 /dev/dm-42 dmm DB2PS-member0.localdomain
Notes:
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
124/168
• NSDs highlighted in gray are not part of the FlashCopy relationship and hence the block
device mappings remain unchanged.
• Since GPFS file systems are configured to Auto-Mount (during GPFS startup) on the
primary site, the replicated NSD meta-data in the FC target volumes are also set with
Auto-Mount. Therefore there is no need to manually mount the GPFS file systems on FC
target NSDs.
� Bring FlashCopy clone of D3D database online
o Catalog FlashCopy clone of D3D database On DB2PS-member0 as DB2 instance ID (db2d3d), catalog D3D db
db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 catalog db d3d on /db2/D3D DB20000I The CATALOG DATABASE command completed su ccessfully.
db2d3d@DB2PS-member0:~> db2 list db directory System Database Directory Number of entries in the directory = 1 Database 1 entry: Database alias = D3D Database name = D3D Local database directory = /db2/D3D Database release level = e.00 Comment = Directory entry type = Indirect Catalog database partition number = 0 Alternate server hostname = Alternate server port number =
o Start instance db2d3d On DB2PS-member0 as DB2 instance ID (db2d3d), start DB2 db2d3d@DB2PS-member0:~> db2start ADM12026W The DB2 server has detected that a valid license for the product "DB2 Enterprise Server Edition" has not bee n registered. 05/07/2011 17:05:02 0 0 SQL1063N DB2START processing was successful. 05/07/2011 17:05:03 1 0 SQL1063N DB2START processing was successful. SQL1063N DB2START processing was successful.
o Initialize the FlashCopy clone of D3D database On DB2PS-member0 as DB2 instance ID (db2d3d), initialize the FC copy of D3D database as a
snapshot
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
125/168
db2d3d@DB2PS-member0:~> db2inidb d3d as snapshot
The above ‘db2inidb’ command would trigger an implicit GCR operation, status of which can be
monitored via,
db2d3d@DB2PS-member0:~> db2pd -db d3d -recovery Database Member 0 -- Database D3D -- Active -- Up 0 days 00:03:54 -- Date 05/07/2011 17:08:56 Recovery: Recovery Status 0x410A8015 Current Log Current LSN 000003E9D3E2309B Current LRI 0000000000000001000000000471911 9000003E9D3E2309B Current LSO 4416510133036 Job Type GROUP CRASH RECOVERY Job ID 1 Job Start Time (1304780916) Sat May 7 17:08:3 6 2011 Job Description Group Crash Recovery Invoker Type User Total Phases 2 Current Phase 1 Progress: Address PhaseNum Description StartTime CompletedWork TotalWork 0x0000000204A6FBC8 1 Forward Sat May 7 17:08:36 137987677 bytes 20162993760 bytes 0x0000000204A6FD50 2 Backward NotStarted 0 bytes
The DB2 diaglog should indicate when the above GCR completes
2011-05-07-17.41.36.954133+120 E212632E450 LEVEL: Info PID : 40764 TID : 4743261637196 8 PROC : db2sysc 0 INSTANCE: db2d3d NODE : 000 DB : D3D APPHDL : 0-52 APPID: *N0.DB2.11050 7150502 EDUID : 35 EDUNAME: db2agnti (D 3D ) 0 FUNCTION: DB2 UDB, recovery manager, sqlpresr, prob e:3110 MESSAGE : ADM1528I Group crash recovery has comple ted successfully.
� Clone read test After successful execution of ‘db2inidb’ operation, activate and connect to the FlashCopy clone
of D3D database and perform a row-count of a key table to demonstrate that the flashcopy
clone of the database is usable.
db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 activate db d3d DB20000I The ACTIVATE DATABASE command completed s uccessfully. db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 connect to d3d
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
126/168
Database Connection Information Database server = DB2/LINUXX8664 9.8.4 SQL authorization ID = DB2D3D Local database alias = D3D db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 "select count(*) from SAP® d3d.BCA_PAYMITEM with ur" 1 ----------- 1805273916 1 record(s) selected.
6.7.7 Going back to the production configuration
While this section is not part of the POC’s Online backup or DR test cases, it is useful to
understand what got changed in the production configuration after adding FC targets on the DR
site – since those new NSDs have changed the GPFS configuration on the DR site. In addition, it
would be useful to briefly address how to operate with this new configuration.
Initial status Some of the GPFS file systems are mounted, i.e. both Q-replicated filesystems and FlashCopy
Target clone filesystems (created via Metro-Mirror) are mounted on the DR GPFS cluster.
Reset DB2 and GPFS back to production configuration The “new” production configuration would have only filesystem with db2 instance / shared
sqllib and Q-rep file systems mounted – i.e. both PPRC target volumes and FC target volumes
must be unmapped (i.e. made invisible) from GPFS.
� Update the DB2 Configuration on DR site Before we switch over to the production configuration:
o Stop SAP® run
o Deactivate FC clone copy of D3D database:
o db2 terminate; force applications all; db2 deactiva te db D3D o Stop Db2
o db2stop force
o Uncatalog FC clone copy of D3D database:
o db2 uncatalog db D3D
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
127/168
� Update the GPFS Configuration on DR site Before switching to the production configuration, unmount the FC clone target filesystems and
unmap (mask) both Metro-Mirror and FC replicated NSDs on the DR cluster.
- From a node on DR site, as root issue: /root/scripts/disable_repl_nsd.sh - See Appendix for script details
6.7.8 Conclusion
No impact to the primary system was observed during the flashcopy backup.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
128/168
7 SAP® TRBK Certification results
7.1 Architecture – system configuration
7.1.1 Overall layout
Here is the target Architecture used for the Certification test campaign.
Hardware Involved
Machine type Quantity Role
x3690 x5 ( 7147-H1G) 5 Database servers (DB2PS)
HS22 (Type 7870)
BladeCenter H
22
2
Application servers (SAP® )
DS8800 (2421-951 ) 1 Database storage
DS5300 1 Benchmark results storage
Mellanox IB switches IS5030 2 Interconnect Infiniband network (provide HA)
10 Gb Eth
Uplink 10G
Uplink 10G
HS22 x 10
HS22 x 10
X3690 eX5
1Infiniban
SA
1 Gb Eth
SAP® TRBK Certification
8Gb
CF 0
Member 0
Member 1
Member 2
Member 3
SA
DS8800 4 way 384 GB cache
90 Ranks Drives 146GB
15k RPM 48 ports FC
SLES 11 SP 1
SLES 11 SP 1 DB2PS FP4
Micro-code 6.1
s2&3 s5
s1
s3 s1
s2
s2&3 s5
s1
s2&3 s
5
s1
s2&3
s5
s1
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
129/168
7.1.2 Application servers - HS22
7.1.2.1 Technical characteristics
Element Quantity Description
Processor 2 Intel Xeon 4 Cores X5570 95W 2.93GHz/1333MHz/8MB L2
Memory 12
Express 4GB Dual Rank PC3-10600 CL9 ECC DDR3 VLP RDIMM
1333MHz
Internal disks 2 Express IBM 146 GB 2.5in SFF Slim-HS 15K 6Gbps SAS HDD
Additional cards 1 Qlogic 1 Gb Ethernet dual port CIO-v
Note: the internal disks have been configured as a RAID1 array.
Remark: Among all the HS22, one has been chosen to host the TRBK driver. This blade is also
responsible of collecting all the monitoring data (SAP® & system related) from all the machines.
Thus, this blade is attached to a DS5300 via a daughter card CFF-h Combo LAN +SAN Qlogic 8Gb.
7.1.2.2 Firmware levels
Module Version Release date
FW/BIOS P9E146A 03/01/2010 (1.07)
Diagnostics DSYT60K 02/25/2010 (3.01)
Blade Sys Mgmt Processor YUOO57H (1.10)
7.1.2.3 CPU information
Architecture: x86_64 Model: 26
Logical CPU(s): 16 Stepping: 5
Thread(s) per core: 2 CPU MHz: 2926.000
Core(s) per socket: 4 Virtualization: VT-x
CPU socket(s): 2 L1d cache: 32K
NUMA node(s): 2 L1i cache: 32K
Vendor ID: GenuineIntel L2 cache: 256K
CPU family: 6 L3 cache: 8192K
7.1.2.4 RAM configuration
HS22 are equipped with 12 DIMMs of 4GB DDR-3, resulting 48 GB total. All the slots are
populated, and the 2 nodes have an equivalent amount of memory. This is part of the Best
Practices recommended by IBM. Here is a dump of the Linux numactl command, which shows
the NUMA topology of the server.
available: 2 nodes (0-1)
node 0 size: 24196 MB
node 1 size: 24240 MB
node distances:
node 0 1
0: 10 21
1: 21 10
7.1.3 Database servers - x3690 x5
7.1.3.1 Technical characteristics
Element Quantity Description
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
130/168
Processor 2 Intel Xeon Processor E7-2870 10C (2.40GHz 30MB L3 130w 2S)
Memory 32 8 GB DDR3 technology and operates at memory speeds of 1066 MHz.
Internal disks 8 IBM 200 GB SDD
Additional cards
1 SAS Raid
1 Emulex 10Gbps Ethernet Dual ports PCI-express
1 Qlogic 8 Gb FC Dual ports PCI-express
1 (2) Mellanox 40 Gbps QDR Infiniband Dual ports PCI-Express
Note: the internal disks have been configured as a RAID5 array.
7.1.3.2 RAM configuration
All x3690 x5 are equipped with 2 Memory cards, each containing 16 DIMMs of 8GB DDR-3.
7.1.3.3 Overall, the amount of memory is 256 GB of DDR3 RAM. uEFI Configuration
Here is an uEFI dump extract using the ASU (Advanced Setup Utility – tool provided by IBM).
The configuration has been done using the recommendations of Toronto & Intel Lab.
uEFI.OperatingMode=Performance Mode
uEFI.QuietBoot=Enable
uEFI.TurboModeEnable=Enable
uEFI.TurboBoost=Traditional
uEFI.ProcessorEistEnable=Disable
uEFI.ProcessorCcxEnable=Disable
uEFI.ProcessorC1eEnable=Disable
uEFI.HyperThreading=Enable
uEFI.EnableCoresInSbsp=All
uEFI.ExecuteDisableBit=Enable
uEFI.ProcessorVmxEnable=Enable
uEFI.ProcessorDataPrefetch=Enable
uEFI.DDRspeed=Max Performance
uEFI.CkeLowPolicy=Disable
uEFI.MirrorMigration=NonMirrored
uEFI.MapperPolicy=Closed
uEFI.SchedulerPolicy=Adaptive
uEFI.PatrolScrub=Disable
uEFI.PagePolicy=Closed uEFI.MemoryScrambling=Disable
uEFI.Socket0Branch0SpareEn=Disable
uEFI.IdeMode=Compatibility mode
uEFI.ThunkSupport=Enable
uEFI.TpmEnable=Disable
uEFI.TpmStateUserSelection=Deactivate
uEFI.QPISpeed=Max Performance
7.1.3.4 I/O card layout for x3690 x5
Database servers require a suitable connection to the Storage Area Network (SAN) to access the
disks where the tables are stored. Moreover, DB2 pureScale® allows the different database
servers to communicate directly using RDMA, thanks to the use of Infiniband stack and uDAPL
upper protocol.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
131/168
However, all the servers are not equivalent: among 5 servers, 4 are “members” (i.e. regular
database servers) and 1 is “coupling facility” (used to manage shared data structures between
members). Thus, these 2 kinds of servers have different needs in terms of communication: that’s
why 2 layouts have been designed for the PCI-express cards.
Naming convention:
This outline illustrates the internal organization of the x3690 x5 components. The PCI slots are
represented on the right. The speed of these buses is also specified (x4 to x16).
For regular members:
Card type Quantity Slot number
Infiniband (HCA) 1 1
Fiber Channel (HBA) 2 2 - 3
Server RAID 1 4
10 Gb Ethernet 1 5
Explanation:
- Objective: priority is given to the IB bandwidth
- Assumption: 10 Gb Eth is not resource consuming
- Advantage : IB can benefit from « full » bandwidth of an PCI bus
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
132/168
For coupling facilities:
Card type Quantity Slot number
Infiniband (HCA) 1 1
Fiber Channel (HBA) 2 2
Infiniband (HCA) 1 3
Server RAID 1 4
10 Gb Ethernet 1 5
Explanation:
- Hypothesis 1 : CF might be able to use 2 active IB cards
- Hypothesis 2 : CF will not have intensive SAN I/O
- All IB cards have homogeneous configuration (PCI-e 8x slots)
- Advantage : maximize bandwidth for both IB cards
-
Note: the slot 1 (16x) has not been used because the Mellanox card is not currently supporting
16x speed. Thus, to have a homogeneous configuration between all the cards, all the x8 slots
have been used for the IB cards. The FC card (Qlogic) has been placed in slot 2, as this slot is
compatible with any 8x card (IBM documentation) and that the I/O on the CF is very low.
7.1.4 Storage overview
The storage subsystem is made of 1 DS8800 following characteristics:
• 3 frames
• 384 GB Processor Memory (4-Way)
• 768 disks of 450 GB 15K FC
• 16 x 8Gb SW FCP ports connections to the SAN.
Each server (member or coupling facility) have access to the DS8800 through a SAN (Storage
Area Network)
Some description of the two DS8800 is detailed below:
dscli showsi IBM.2107-75TX861 -cfg C:\Progra~1\IBM\dscli\profile\DB2_PRA_Pure_1.profile
Date/Time: 08 December 2010 16:25:50 CET IBM DSCLI Version: 6.5.1.203 DS: IBM.2107-
75TX861
ID IBM.2107-75TX861
Storage Unit IBM.2107-75TX860
Model 941
WWNN 500507630AFFC2BC
Signature 4d2b-1929-cad1-dee3
State Online
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
133/168
ESSNet Enabled
Volume Group V0
os400Serial 2BC
NVS Memory 8.0 GB
Cache Memory 235.9 GB
Processor Memory 253.4 GB
MTS IBM.2421-75TX860
numegsupported 0
ETAutoMode on
ETMonitor all
Overview of the storage configuration
DS8800 Layout
Ran
k RAID Extentpo
ol Arra
y Device Adapter
ExtentPool ID
ExtentPool Name
Lun 1000
Lun1101
Lun1002
Lun1103
R0 RAID5 6+P+S 0 A0 0 P0 pure-pool1 x
R1 RAID5 6+P+S 1 A1 0 P1 pure-pool2 x
R2 RAID5 6+P+S 0 A2 0 P2 pure-pool3 x
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
134/168
R3 RAID5 6+P+S 1 A3 0 P3 pure-pool4 x
R4 RAID5 7+P 0 A4 0 P0 pure-pool1 x
R5 RAID5 7+P 1 A5 0 P1 pure-pool2 x
R6 RAID5 7+P 0 A6 0 P2 pure-pool3 x
R7 RAID5 7+P 1 A7 0 P3 pure-pool4 x
R8 RAID5 7+P 0 A8 0 P0 pure-pool1 x
R9 RAID5 7+P 1 A9 0 P1 pure-pool2 x
R10 RAID5 7+P 0 A10 0 P2 pure-pool3 x
R11 RAID5 7+P 1 A11 0 P3 pure-pool4 x
R12 RAID5 6+P+S 0 A12 1 P0 pure-pool1 x
R13 RAID5 6+P+S 1 A13 1 P1 pure-pool2 x
R14 RAID5 6+P+S 0 A14 1 P2 pure-pool3 x
R15 RAID5 6+P+S 1 A15 1 P3 pure-pool4 x
R16 RAID5 7+P 0 A16 1 P0 pure-pool1 x
R17 RAID5 7+P 1 A17 1 P1 pure-pool2 x
R18 RAID5 7+P 0 A18 1 P2 pure-pool3 x
R19 RAID5 7+P 1 A19 1 P3 pure-pool4 x
R20 RAID5 7+P 0 A20 1 P0 pure-pool1 x
R21 RAID5 7+P 1 A21 1 P1 pure-pool2 x
R22 RAID5 7+P 0 A22 1 P2 pure-pool3 x
R23 RAID5 7+P 1 A23 1 P3 pure-pool4 x
R24 RAID5 6+P+S 0 A24 2 P0 pure-pool1 x
R25 RAID5 6+P+S 1 A25 2 P1 pure-pool2 x
R26 RAID5 6+P+S 0 A26 2 P2 pure-pool3 x
R27 RAID5 6+P+S 1 A27 2 P3 pure-pool4 x
R28 RAID5 7+P 0 A28 2 P0 pure-pool1 x
R29 RAID5 7+P 1 A29 2 P1 pure-pool2 x
R30 RAID5 7+P 0 A30 2 P2 pure-pool3 x
R31 RAID5 7+P 1 A31 2 P3 pure-pool4 x
R32 RAID5 7+P 0 A32 2 P0 pure-pool1 x
R33 RAID5 7+P 1 A33 2 P1 pure-pool2 x
R34 RAID5 7+P 0 A34 2 P2 pure-pool3 x
R35 RAID5 7+P 1 A35 2 P3 pure-pool4 x
R36 RAID5 6+P+S 0 A36 3 P0 pure-pool1 x
R37 RAID5 6+P+S 1 A37 3 P1 pure-pool2 x
R38 RAID5 6+P+S 0 A38 3 P2 pure-pool3 x
R39 RAID5 6+P+S 1 A39 3 P3 pure-pool4 x
R40 RAID5 7+P 0 A40 3 P0 pure-pool1 x
R41 RAID5 7+P 1 A41 3 P1 pure-pool2 x
R42 RAID5 7+P 0 A42 3 P2 pure-pool3 x
R43 RAID5 7+P 1 A43 3 P3 pure-pool4 x
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
135/168
R44 RAID5 7+P 0 A44 3 P0 pure-pool1 x
R45 RAID5 7+P 1 A45 3 P1 pure-pool2 x
R46 RAID5 7+P 0 A46 3 P2 pure-pool3 x
R47 RAID5 7+P 1 A47 3 P3 pure-pool4 x
R48 RAID5 6+P+S 0 A48 4 P0 pure-pool1 x
R49 RAID5 6+P+S 1 A49 4 P1 pure-pool2 x
R50 RAID5 6+P+S 0 A50 4 P2 pure-pool3 x
R51 RAID5 6+P+S 1 A51 4 P3 pure-pool4 x
R52 RAID5 7+P 0 A52 4 P0 pure-pool1 x
R53 RAID5 7+P 1 A53 4 P1 pure-pool2 x
R54 RAID5 7+P 0 A54 4 P2 pure-pool3 x
R55 RAID5 7+P 1 A55 4 P3 pure-pool4 x
R56 RAID5 7+P 0 A56 4 P0 pure-pool1 x
R57 RAID5 7+P 1 A57 4 P1 pure-pool2 x
R58 RAID5 7+P 0 A58 4 P2 pure-pool3 x
R59 RAID5 7+P 1 A59 4 P3 pure-pool4 x
R60 RAID5 6+P+S 0 A60 5 P0 pure-pool1 x
R61 RAID5 6+P+S 1 A61 5 P1 pure-pool2 x
R62 RAID5 6+P+S 0 A62 5 P2 pure-pool3 x
R63 RAID5 6+P+S 1 A63 5 P3 pure-pool4 x
R64 RAID5 7+P 0 A64 5 P0 pure-pool1 x
R65 RAID5 7+P 1 A65 5 P1 pure-pool2 x
R66 RAID5 6+P+S 0 A66 6 P2 pure-pool3 x
R67 RAID5 6+P+S 1 A67 6 P3 pure-pool4 x
R68 RAID5 6+P+S 0 A68 6 P0 pure-pool1 x
R69 RAID5 6+P+S 1 A69 6 P1 pure-pool2 x
R70 RAID5 7+P 0 A70 6 P2 pure-pool3 x
R71 RAID5 7+P 1 A71 6 P3 pure-pool4 x
R72 RAID5 7+P 0 A72 6 P0 pure-pool1 x
R73 RAID5 7+P 1 A73 6 P1 pure-pool2 x
R74 RAID5 7+P 0 A74 6 P2 pure-pool3 x
R75 RAID5 7+P 1 A75 6 P3 pure-pool4 x
R76 RAID5 7+P 0 A76 6 P0 pure-pool1 x
R77 RAID5 7+P 1 A77 6 P1 pure-pool2 x
R78 RAID5 6+P+S 0 A78 7 P2 pure-pool3 x
R79 RAID5 6+P+S 1 A79 7 P3 pure-pool4 x
R80 RAID5 6+P+S 0 A80 7 P0 pure-pool1 x
R81 RAID5 6+P+S 1 A81 7 P1 pure-pool2 x
R82 RAID5 7+P 0 A82 7 P2 pure-pool3 x
R83 RAID5 7+P 1 A83 7 P3 pure-pool4 x
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
136/168
R84 RAID5 7+P 0 A84 7 P0 pure-pool1 x
R85 RAID5 7+P 1 A85 7 P1 pure-pool2 x
R86 RAID5 7+P 0 A86 7 P2 pure-pool3 x
R87 RAID5 7+P 1 A87 7 P3 pure-pool4 x
R88 RAID5 7+P - A88 7 - Not used
R89 RAID5 7+P - A89 7 - Not used
The configuration of the RAID5 arrays of one DS8800 with 720 FC disks.
• 90 RAID5 arrays.
• 4 extent pools
• 22 ranks / extent pool
• Luns created with rotateexts option
7.1.4.1 Assumptions for DB2 / GPFS / Storage design
� Assumptions:
– Shared pool of FC disks for Data/Index/Temp and logs
– Storage Pool Striping (1 GB stripe size) for Data/Index/Temp and logs
– No GPFS striping (each file system is defined / mapped on only one LUN).
– Easy configuration with only one level of stripping at DS8800 level.
� Design:
– Each file system (SAP® data1 to SAP® adata 16) is mapped on one LUN of 388 GB
size
– Each file system (log0 to log3) is mapped on one LUN of 2 TB size.
– Each LUN is stripped on 22 RAID5 arrays (176 FC disks) with a block size of 1 GB
– LUN is created with the option Rotate Extent in an Extent Pool
– 4 Extents Pools on DS8800 which consist in 22 FC RAID5 arrays each
– Each RAID5 array has 6 data, 1 parity and 1 spare or 7 datas and 1 parity disks.
– A total of 720 disks.
LUN is created with the option Rotate Extent in an Extent Pool. For each extent pool:
• 4 LUNs of 388 GB for Data/index/Temp
• 1 LUN of 2 TB for logs
For example the LUN 2000 is distributed on 22 different RAID5 arrays
This LUN of 388 GB size consists in 16 or 17 extents of 1 GB for each array.
7.1.4.2 LUN and file system for DB2 with 720 disks FC
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
137/168
File system capacity Techno NumDisks Num
LUNs
LUN size Storage Pool
Striping
/db2/D3D/ 2000 GB FC 176 1 2000 GB Yes
/db2/D3D_log0
/db2/D3D_log1
/db2/D3D_log2
/db2/D3D_log3
2000 GB
2000 GB
2000 GB
2000 GB
FC
FC
FC
FC
176 1
1
1
1
2000 GB Yes
/db2/D3D_SAP® data1
…………………..
/db2/D3D_SAP® data16
388 GB
388 GB
FC
FC
176
176
1
1
388 GB
388 GB
Yes
Yes
7.1.4.3 Detailed of the LUNs configuration
SAP® fs DS8800 ID + LUN ID EP cap
SAP® data1 2bc0000000000002100) P0 388 GB
SAP® data2 2c10000000000002100) P1 388 GB
SAP® data3 2c10000000000002003) P2 388 GB
SAP® data4 2c10000000000002002) P3 388 GB
SAP® data5 2c10000000000002001) P0 388 GB
SAP® data6 2bc0000000000002103) P1 388 GB
SAP® data7 2c10000000000002000) P0 388 GB
SAP® data8 2bc0000000000002102) P3 388 GB
SAP® data9 2bc0000000000002101) P0 388 GB
SAP® data10 2bc0000000000002003) P1 388 GB
SAP® data11 2bc0000000000002000) P2 388 GB
SAP® data12 2bc0000000000002002) P3 388 GB
SAP® data13 2c10000000000002103) P0 388 GB
SAP® data14 2bc0000000000002001) P1 388 GB
SAP® data15 2c10000000000002102) P2 388 GB
SAP® data16 2c10000000000002101) P3 388 GB
SAP® fs DS8800 ID + LUN ID EP cap
log0 2c10000000000002004 P0 2000 GB
log1 2c10000000000002104 P1 2000 GB
log2 2bc0000000000002004 P2 2000 GB
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
138/168
log3 2bc0000000000002104 P3 2000 GB
tiebreaker 2bc0000000000002005 P0 2000 GB
db2fs1 2c10000000000002105 P1 2000 GB
notused 2c10000000000001005 P2
db2d3d 2c10000000000002005 P3 2000 GB
7.1.5 Network fabric overview
7.1.5.1 Infiniband network Topology
For HA test purposes, 2 Mellanox switches (IS5030) have been set up for providing switch failure
tolerance. Here is the topology:
In case of a switch failure, half on the cluster is brought offline.
On the outline above, 2 types of cables have been used:
- From member/switch and coupling/switch : 5m Copper cables QSFP
- Between switches : 10m Optic cables QSFP
Having four links between the 2 switches ensure an optimal throughput, as each coupling facility
has 4 HCAs.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
139/168
7.1.5.2 Infiniband Subnet Manager configuration
Among the 2 switches, one has been chosen to act as the Subnet Manager. It is keeping the
identifier of each HCAs and is aware of the global Infiniband topology. The switch running as a
Subnet Manager is a licensed Mellanox feature, and has been activated in mlx-switch1. The
other switch (mlx-switch2) is considering as passive, as it is only routing the packets without
having the responsibility of maintaining the IB network.
7.1.5.3 Fiber Channel network
Each database server is connected to the storage system (DS8800) via a storage area network.
This network is created using dedicated SAN switches in the infrastructure. As described in the
server part, each member has 4 connections as they perform intensive I/O while coupling
facilities have only 2 links on the SAN.
Another smaller SAN connection is used between the SAP® injector and a DS5000 system (peer-
to-peer connection), to store all the data coming from the run.
7.1.5.4 Ethernet network
4 networks are used to ensure the communication through the DB2 pureScale® architecture
components:
Network Type Machines
connected
Subnet MTU
Administration 1 Gb Ethernet All 10.7.21/24 1500
Application 1 Gb Ethernet SAP® servers 10.1.1.0/24 1500
Database 10 Gb Ethernet SAP® + DB2
servers
10.1.2.0/24 1500
Interconnect 10 Gb QDR
Infiniband
DB2 servers 10.9.9.0/24
(IPoIB)
65520
The administration network is reachable from the Internet thanks to a Virtual Private Network.
All other networks are isolated and dedicated to this architecture.
Remark: the gateway of the 10.7.21.0/24 network is also used for NTP (Network Time Protocol)
synchronization.
For more information concerning the IP addressing map, please refer to the file pureScale – IP
Addressing.xls
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
140/168
7.1.6 Operating Systems
7.1.6.1 Suse Entreprise Linux 11 SP1
All servers are using Suse Enterprise Linux (SLES) 11 SP1. Kernel has been upgraded on the DB2
servers from the original version to allow the newest OFED driver (for Infiniband) to be installed.
Machine Type Kernel version
Database Servers (x3690 x5) 2.6.32.24-0.2-default
Application Servers (HS22) 2.6.32.16-0.54-default
7.1.6.2 I/O device drivers
• Multipathing driver
The multipathing driver used for this benchmark is DM-MPIO that comes with the SLES 11 SP1
distribution. The configuration file (multipath.conf) follows IBM support recommendations. The
policy is round-robin based (rr_min_io = 100), and paths are grouped by priority. Here is an
extract of the multipath.conf file:
defaults {
polling_interval 30
failback immediate
no_path_retry 5
rr_min_io 100
path_checker tur
# features "1 queue_if_no_path"
user_friendly_names yes
# These are the default settings for 2107 (IBM DS8000)
# Uncomment them if needed on this system
device {
vendor "IBM"
product "2107900"
path_grouping_policy group_by_serial
}
• Linux IO scheduler parameter
Here are Linux parameters concerning the IO scheduling, for each accessible disk:
- IO Queue size : 128
- Scheduler used : Completely Fair Queuing
At a lower level, the use of NCQ (Native Command Queuing) also comes with sets of parameters
- NCQ queue size : 32
For deeper details about the LUN mapping please refer to multipath command display or
storage configuration documentation.
• Infiniband driver
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
141/168
For the database servers, the OFED stack is used to provide the driver for Infiniband HCA.
On one node, here is the result of the infiniband driver self test command:
---- Performing Infiniband HCA Self Test ----
Number of HCAs Detected ................ 1
PCI Device Check ....................... PASS
Kernel Arch ............................ x86_64
Host Driver Version .................... OFED-1.5.2-20100824-0600: 1.5.2-2.6.16.60_0.66.1_smp
Host Driver RPM Check .................. PASS
HCA Firmware on HCA #0 ................. v2.7.0
Host Driver Initialization ............. PASS
Number of HCA Ports Active ............. 1
Port State of Port #1 on HCA #0 ........ UP 4X QDR
Port State of Port #2 on HCA #0 ........ DOWN
Error Counter Check on HCA #0 .......... PASS
Kernel Syslog Check .................... PASS
------------------ DONE ---------------------
7.1.6.3 Settings tuned
• Kernel parameters
For performance improvement, the use of several HCA (Host Channel Adapter – Infiniband
Adapter) has been recommended on the coupling facilities. As the uDAPL needs IPoIB
configuration for each HCA, several IP addresses from the same subnet are utilized on the same
physical machine.
The Linux kernel doesn’t handle this configuration by default. Thus, the kernel parameter
net.ipv4.conf.all.arp_ignore has been modified to the value “2”, as recommended in
the OFED stack documentation.
7.2 Certification results
On September 12, 2011, SAP® published the following press release which summarize the key
results achieved on the previously described configuration.
The SAP® TRBK (Transaction Banking) Standard Application Benchmark performed on August 09,
2011 by IBM in Montpellier, France, has been certified with the following data:
The scenario contained 90,000,000 accounts and 1,800,000,000 postings.
Part I: Day Processing
No. of postings to bank accounts/hour: 56,518,000
CPU utilization database servers: 89% (CF: 59%, Member 0: 97%,
Member 1: 97%, Member 2: 97%,
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
142/168
Member 3: 96%)
CPU utilization application servers: 62%
Part II: Night Processing
No. of balanced accounts/hour: 22,382,000
CPU utilization database servers: 36% (CF: 75%, Member 0: 26%,
Member 1: 26%, Member 2: 26%,
Member 3: 25%)
CPU utilization application servers: 47%
The software configuration for all steps of the SAP® TRBK benchmark:
Operating system, all servers: SUSE Linux Enterprise Server 11 SP1
RDBMS database servers: IBM DB2 9.7 ESE with pureScale
feature
SAP® Business Suite software: Banking Services from SAP® 7.0
Configuration:
5 Database Servers:
(1 CF, 4 Members)
IBM x3690 X5, 2 processors / 20 cores /
40 threads, Intel Xeon Processor E7-
2870, 2.40 GHz, 64 KB L1 cache and 256
KB L2 cache per core, 30 MB L3 cache
per processor, 256 GB main memory
21 Application Servers:
1 Dialog/Batch/Message/Enq. server:
IBM BladeCenter HS22 7870, 2
processors / 8 cores /
16 threads, Intel Xeon Processor X5570,
2.93 GHz,
64 KB L1 cache and 256 KB L2 cache per
core, 8 MB L3 cache per processor, 48
GB main memory
20 Dialog/Batch server: IBM BladeCenter HS22 7870, 2
processors / 8 cores /
16 threads, Intel Xeon Processor X5570,
2.93 GHz,
64 KB L1 cache and 256 KB L2 cache per
core, 8 MB L3 cache per processor, 48
GB main memory
Certification number: 2011035
7.3 Tips & recommendations
7.3.1 Database Configuration
Single CF
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
143/168
Only a single CF was used for the certification runs. We were unable to measure the impact of
running with a secondary CF in the benchmark configuration. However, single versus duplexed
CF tests on the PoC configuration showed only about a 1% overhead.
BCA_BCAS_EVBST Table
Space Map Page (SMP) contention on the BCA_BCAS_EVBST table caused INSERT and UPDATE
performance on this table to suffer due to reclaims. After moving this table from a tablespace
with extent size 2 to a tablespace with an extent size 256, SMP contention was eliminated since
there was no need to frequently extend the table. As an additional tuning step we enabled
APPEND mode for this table which provided an 8.5% improvement to Day run throughput.
DFKKCOHARC Table
Contention on DFKKCOHARC data pages was resolved by partitioning this table into a 16-arm
union-all-view. INSERT times improved by about 10x.
7.3.2 SAP® Application Server
SAP® Enqueue Server Bottleneck
The number of SAP® Enqueue work processes on the SAP® Central Instance needed to be
increased from 1 to 4 in order to address poor Enqueue Server performance. This bottleneck
initially limited Day run throughput to 36 Million Postings per Hour.
7.3.3 Operating System
SLES11 SP1 CPU Scheduler
We made use of the sched_compat_yield kernel tunable to revert to the SLES10 version of the
CPU scheduler. As a result of this we observed a significant reduction in system CPU time from
about 40% to 15-20% and Day run throughput increased by about 10%.
SLES11 SP1 IO Wait Accounting
This is more of an observation. During this certification effort we observed 10-15% IO wait time
during the Day run. This was a change from the earlier performance tests we did on SLES10,
where reported IO wait time was minimal. We observed that it is only the castout engines that
are in iowait and looking at their kernel stack, they are collecting IOs in the "get_events" kernel
function. This function was changed in the SLES11 kernel such that it now accounts for iowait,
thus we now see threads in this function in an iowait state. In the SLES10 kernel, this function
did not account for iowait.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
144/168
7.3.4 Network Topology
Three InfiniBand Cards in CF
Three IB cards were used in the CF. Moving from 2 IB cards to 3 IB cards reduced CF response
times by about 50%. Because of the limited number of PCI-X slots in the x3690s, the 10 Gigabit
Ethernet card was removed and replaced by the third IB card.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
145/168
8 Best Practices for DB2 pureScale® and SAP® TRBK
DB2 pureScale® considerations
DB2 pureScale® facilitates the linear growth of workloads across multiple systems in a fashion
not previously achievable with DB2 for Linux, Unix, Windows. The existence of an extremely
fast-access shared memory system like the Coupling Facility enables one data image to be
effectively shared across many systems. The more update intensive a workload is, however, the
more likelihood there is of contention for data pages between members. This contention can be
controlled and tuned, and this tuning is the key to achieving excellent performance with TRBK.
There is a distinct difference in this respect between the Day and Night processing of TRBK.
Night processing can be segmented to allow difference application servers to process different
ranges of accounts in parallel. If the work coming from an application server aligns with the
partitioning of certain key tables in TRBK, there need be no sharing of table data between
database members. This facilitates isolated parallel processing of accounts and near-perfect
workload scaling with the number of members.
The Day workload is somewhat different. The payitem processing being performed by an
arbitrary benchmark user on an arbitrary application server can reference any account number.
With a busy system processing millions of payitems per hour, there is inevitably going to be
some degree of contention on account data. This is particularly true for index pages, due to their
tree structure. This contention can be controlled with the use of table partitioning and table and
index freespace values.
8.1 Tablespace Definitions
8.1.1 Extent Size
Members maintain an extent sized space map buffer for local space allocation. For tables where
there is a high volume of inserts or page allocation changing updates, increasing extent size of
the tablespace containing the table to the maximum value (255) is strongly recommended.
Examples of TRBK tables that fall into this category and benefit from a large extent size are
BCA_PAYMITEM, BCA_GL_PAYMITEM, and BCA96.
8.1.2 Increase Size
Extending tablespaces is an expensive operation with DB2 pureScale®, and has negative impacts
on performance. For this reason, a reasonable INITIALSIZE should be set, and INCREASESIZE
should not be left at the default value. By default, we found INCREASESIZE to be a relatively
small value of around 1GB. With heavy insert activity such as is seen during the TRBK Night
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
146/168
processing, INCREASESIZE should be set to at least 50GB in order to keep the number of increase
actions to a minimum.
8.2 Table Compression
Traditionally, from the performance perspective, tables benefit from the use of DB2’s row
compression feature when the cpu cost of decompressing a row for use is outweighed by the
overall savings flowing from a reduced memory footprint and from reduced transfer times from
disk. Tables that benefited from table compression in the PoC fell into 3 categories:
� Tables with high ongoing IO requirements, such as PAYMITEM
� Tables whose data are not frequently re-referenced after insert, such as DFKKCOH
� Tables that are effectively read-only, like BUT000
8.3 Data and Index Percent Free
Allocating tables with a portion of data and index pages unused (achieved by use of the PCTFREE
table and index attribute) is beneficial in allowing subsequent row growth (update) and insert
activity to proceed without reference to space map pages.
Tables benefiting from use of PCTFREE fell into different categories
� For tables inserted randomly, like PAYMITEM, PCTFREE changes on both indexes and
data were found to be effective
� Tables that are heavily inserted and rarely re-referenced, like DFKKCOH benefit from
index (leaf and level2) compression
� Tables that are effectively read-only, i.e. heavily referenced but rarely updated, such as
BUT000 are candidates for minimal PCTFREE (i.e, zero).
8.4 Data Partitioning
A key technique that can be used to minimize contention between members is table
partitioning. For tables that have column data that is uniform, and where the activity against
that column data is evenly spread by member, the table can be divided into segments and
logically joined with a view that unions all the pieces. An example of such a column in TRBK is
CONTRACT_INT. This column contains an effectively random GUID value. It exists (with some
variation in the name) in most of the large TRBK tables.
SAP® code contains an optimization for shared data systems that allows the allocation of ranges
of data to the batch processes running on specific application servers. As SAP® application
server work processes connect to specific members, this means that, with the right partitioning
schema, Night processing activity is predominantly local, with very little or no contention for
table data or index pages between members.
Day processing cannot be set up in the same way, as each benchmark user on any appserver can
reference or insert to part of the major tables. However, having partitioned tables reduces the
incidence of data contention between members.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
147/168
9 Conclusions The combined set of results (Day, Night, scalability and HA) demonstrate that DB2 pureScale® is
capable of delivering excellent performance when running a very challenging workload in an
environment that effectively guarantees 100% availability.
The very large scale of database (90 million accounts) coupled with the realistic nature of the
SAP® TRBK benchmark makes this exercise a very concrete proof point of the exceptional
robustness and capability of the DB2 pureScale® solution.
Some of the metrics achieved give an indication of the scale of the workload:
Statements per second (Day run): 169000
Statements per second (Night run): 444500
Transactions per second (Day run): 4889
Log writes (Day run): 53 megabytes per second
Log writes (Night run): 200 megabytes per second
The high availability testing done demonstrates that the DB2 pureScale® solution can suffer the
complete failure of major components such as a database member of a CF, and recover to the
same transaction rate with minimal transaction loss and with complete data integrity.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
148/168
10 Appendix
10.1 db2dsdriver.cfg
<configuration>
<DSN_Collection>
<dsn alias="D3D" name="D3D" host="DB2PS-member0" port="5912" />
</DSN_Collection>
<databases>
<database name="D3D" host="DB2PS-member0" port="5912">
<parameter name="keepAliveTimeout" value="5"/>
<parameter name="TcpipConnectTimeout" value="2"/>
<acr>
<parameter name="enableAcr" value="true"/>
<parameter name="enableSeamlessAcr" value="true"/>
<parameter name="affinityFailbackInterval" value="180"/>
<parameter name="maxAcrRetries" value="1"/>
<alternate_server_list>
<server name="server1" hostname="DB2PS-member0" port="5912" />
<server name="server2" hostname="DB2PS-member1" port="5912" />
<server name="server3" hostname="DB2PS-member2" port="5912" />
<server name="server4" hostname="DB2PS-member3" port="5912" />
</alternate_server_list>
<affinity_list>
<list name="list1" serverorder="server1,server2,server3,server4" />
<list name="list2" serverorder="server1,server2,server4,server3" />
<list name="list3" serverorder="server1,server3,server2,server4" />
<list name="list4" serverorder="server1,server3,server4,server2" />
<list name="list5" serverorder="server1,server4,server2,server3" />
<list name="list6" serverorder="server1,server4,server3,server2" />
<list name="list7" serverorder="server2,server1,server3,server4" />
<list name="list8" serverorder="server2,server1,server4,server3" />
<list name="list9" serverorder="server2,server3,server1,server4" />
<list name="list10" serverorder="server2,server3,server4,server1" />
<list name="list11" serverorder="server2,server4,server3,server1" />
<list name="list12" serverorder="server2,server4,server3,server1" />
<list name="list13" serverorder="server3,server1,server2,server4" />
<list name="list14" serverorder="server3,server1,server4,server2" />
<list name="list15" serverorder="server3,server2,server1,server4" />
<list name="list16" serverorder="server3,server2,server4,server1" />
<list name="list17" serverorder="server3,server4,server1,server2" />
<list name="list18" serverorder="server3,server4,server2,server1" />
<list name="list19" serverorder="server4,server1,server2,server3" />
<list name="list20" serverorder="server4,server1,server3,server2" />
<list name="list21" serverorder="server4,server2,server1,server3" />
<list name="list22" serverorder="server4,server2,server3,server1" />
<list name="list23" serverorder="server4,server3,server1,server2" />
<list name="list24" serverorder="server4,server3,server2,server1" />
</affinity_list>
<client_affinity_defined>
<client name="client1" hostname="SAP® serv1" listname="list1" />
<client name="client2" hostname="SAP® serv2" listname="list3" />
<client name="client3" hostname="SAP® serv3" listname="list5" />
<client name="client4" hostname="SAP® serv4" listname="list5" />
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
149/168
<client name="client5" hostname="SAP® serv5" listname="list7" />
<client name="client6" hostname="SAP® serv6" listname="list9" />
<client name="client7" hostname="SAP® serv7" listname="list11" />
<client name="client8" hostname="SAP® serv8" listname="list13" />
<client name="client9" hostname="SAP® serv9" listname="list15" />
<client name="client10" hostname="SAP® serv10" listname="list17" />
<client name="client11" hostname="SAP® serv11" listname="list19" />
<client name="client12" hostname="SAP® serv12" listname="list21" />
<client name="client13" hostname="SAP® serv13" listname="list23" />
</client_affinity_defined>
<client_affinity_roundrobin>
</client_affinity_roundrobin>
</acr>
</database>
</databases>
<parameters>
<parameter name="CommProtocol" value="TCPIP"/>
<parameter name="enableAcr" value="true"/>
<parameter name="enableSeamlessAcr" value="true"/>
</parameters>
</configuration>
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
150/168
10.2 Malloc Bomb
malloc.pl:
#!/usr/bin/perl -w # # Very simple memory filling mechanism # # Usage: muse <size_to_fill> (in mb) # # The script will fork processes if the amount of m emory is # too large (to cope with the 2GB process limits) # my $MAX_SINGLE_PROC = 100; my $LONG_TIME = 300; if( $#ARGV != 0 ) { print STDERR "Usage: $0 <size_to_al loc_in_mb>\n"; exit( 1 ); } my $size = shift @ARGV; my $procs = int( $size / $MAX_SINGLE_PROC ); my $remain = $size - ($procs * $MAX_SINGLE_PROC ); if( $remain == 0 ) { $procs--; $remain = $MAX_SINGLE_PROC; } print "Will allocate approximately $size mb.\n"; print "Forking $procs children at $MAX_SINGLE_PROC mb each.\n"; print "And allocating $remain mb in parent.\n"; for( my $i=0; $i < $procs; $i++ ) { my $p = fork(); if( $p == 0 ) { # Child my $a = malloc( $M AX_SINGLE_PROC ); sleep $LONG_TIME; exit( 0 ); } } my $a = malloc( $remain ); sleep $LONG_TIME; exit 0; sub malloc { my $mb = shift @_; $mb--; return( "." x (1024*512*$mb)); }
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
151/168
10.3 DB2 Registry Settings [e] DB2DBDFT=D3D
[i] DB2_CA_INITIAL_CONNECTION_POOL_SIZE=2
[i] DB2_INDEX_BRANCH_LOCKING=ON
[i] DB2_CA_PORTS=56001,56002,56003,56004
[i] DB2_BLOCKING_WITHHOLD_LOBLOCATOR=NO [DB2_WORKLOAD]
[i] DB2_AGENT_CACHING_FMP=OFF [DB2_WORKLOAD]
[i] DB2_SD_PAGE_REUSE_THRESHOLD=100000
[i] DB2_CA_CONNECTION_POOL_SIZE=120
[i] DB2_NUM_NOTIFY_BUFFERS=256
[i] DB2_TRUST_MDC_BLOCK_FULL_HINT=YES [DB2_WORKLOAD]
[i] DB2_CREATE_INDEX_COLLECT_STATS=YES [DB2_WORKLOAD]
[i] DB2_ATS_ENABLE=YES [DB2_WORKLOAD]
[i] DB2_RESTRICT_DDF=YES [DB2_WORKLOAD]
[i] DB2_DUMP_SECTION_ENV=YES [DB2_WORKLOAD]
[i] DB2_OPT_MAX_TEMP_SIZE=10240 [DB2_WORKLOAD]
[i] DB2_WORKLOAD=SAP®
[i] DB2_TRUNCATE_REUSESTORAGE=IMPORT [DB2_WORKLOAD]
[i] DB2_MDC_ROLLOUT=DEFER [DB2_WORKLOAD]
[i] DB2RSHCMD=/usr/bin/ssh
[i] DB2_ATM_CMD_LINE_ARGS=-include-manual-tables [DB2_WORKLOAD]
[i] DB2_SKIPINSERTED=YES [DB2_WORKLOAD]
[i] DB2_VIEW_REOPT_VALUES=YES [DB2_WORKLOAD]
[i] DB2_OBJECT_TABLE_ENTRIES=65532 [DB2_WORKLOAD]
[i] DB2_OPTPROFILE=YES [DB2_WORKLOAD]
[i] DB2_IMPLICIT_UNICODE=YES [DB2_WORKLOAD]
[i] DB2STMM=APPLY_HEURISTICS:YES [DB2_WORKLOAD]
[i] DB2_INLIST_TO_NLJN=YES [DB2_WORKLOAD]
[i] DB2_MINIMIZE_LISTPREFETCH=YES [DB2_WORKLOAD]
[i]
DB2_REDUCED_OPTIMIZATION=4,INDEX,JOIN,NO_TQ_FACT,NO_HSJN_BUILD_FACT,STARJN_CARD_SKEW,NO_SORT_
MGJOIN,CART OFF,CAP OFF [DB2_WORKLOAD]
[i] DB2NOTIFYVERBOSE=YES [DB2_WORKLOAD]
[i] DB2_INTERESTING_KEYS=YES [DB2_WORKLOAD]
[i] DB2_EVALUNCOMMITTED=YES [DB2_WORKLOAD]
[i] DB2_EXTENDED_OPTIMIZATION=NLJOIN_KEYCARD [DB2_WORKLOAD]
[i] DB2_ANTIJOIN=EXTEND [DB2_WORKLOAD]
[i] DB2COMPOPT=327685,704 [DB2_WORKLOAD]
[i] DB2ATLD_PORTS=60000:65000
[i] DB2ENVLIST=INSTHOME SAP® SYSTEMNAME dbs_db6_schema DIR_LIBRARY LD_LIBRARY_PATH
[i] DB2COUNTRY=1
[i] DB2COMM=TCPIP [DB2_WORKLOAD]
[i] DB2AUTOSTART=NO
[g] DB2SYSTEM=DB2PS-member0
[g] DB2INSTDEF=db2d3d
10.4 DB Manager Configuration Settings
Database Manager Configuration
Node type = Enterprise Server Edition with local and remote clients
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
152/168
Description Parameter Current Value
------------------------------------------------------------------------------------ Database manager configuration release level = 0x0e00
CPU speed (millisec/instruction) (CPUSPEED) = 4.133012e-07
Communications bandwidth (MB/sec) (COMM_BANDWIDTH) = 1.000000e+02
Max number of concurrently active databases (NUMDB) = 1
Federated Database System Support (FEDERATED) = NO
Transaction processor monitor name (TP_MON_NAME) =
Default charge-back account (DFT_ACCOUNT_STR) =
Java Development Kit installation path (JDK_PATH) = /db2/db2d3d/sqllib/java/jd
Diagnostic error capture level (DIAGLEVEL) = 3
Notify Level (NOTIFYLEVEL) = 3
Diagnostic data directory path (DIAGPATH) = /db2/db2d3d/sqllib/db2dump
Size of rotating db2diag & notify logs (MB) (DIAGSIZE) = 0
Default database monitor switches
Buffer pool (DFT_MON_BUFPOOL) = ON
Lock (DFT_MON_LOCK) = ON
Sort (DFT_MON_SORT) = ON
Statement (DFT_MON_STMT) = ON
Table (DFT_MON_TABLE) = ON
Timestamp (DFT_MON_TIMESTAMP) = ON
Unit of work (DFT_MON_UOW) = ON
Monitor health of instance and databases (HEALTH_MON) = OFF
SYSADM group name (SYSADM_GROUP) = DBD3DADM
SYSCTRL group name (SYSCTRL_GROUP) = DBD3DCTL
SYSMAINT group name (SYSMAINT_GROUP) = DBD3DMNT
SYSMON group name (SYSMON_GROUP) =
Client Userid-Password Plugin (CLNT_PW_PLUGIN) =
Client Kerberos Plugin (CLNT_KRB_PLUGIN) =
Group Plugin (GROUP_PLUGIN) =
GSS Plugin for Local Authorization (LOCAL_GSSPLUGIN) =
Server Plugin Mode (SRV_PLUGIN_MODE) = UNFENCED
Server List of GSS Plugins (SRVCON_GSSPLUGIN_LIST) =
Server Userid-Password Plugin (SRVCON_PW_PLUGIN) =
Server Connection Authentication (SRVCON_AUTH) = NOT_SPECIFIED
Cluster manager = TSA
Database manager authentication (AUTHENTICATION) = SERVER_ENCRYPT
Alternate authentication (ALTERNATE_AUTH_ENC) = NOT_SPECIFIED
Cataloging allowed without authority (CATALOG_NOAUTH) = NO
Trust all clients (TRUST_ALLCLNTS) = YES
Trusted client authentication (TRUST_CLNTAUTH) = CLIENT
Bypass federated authentication (FED_NOAUTH) = NO
Default database path (DFTDBPATH) = /db2/D3D
Database monitor heap size (4KB) (MON_HEAP_SZ) = 32768
Java Virtual Machine heap size (4KB) (JAVA_HEAP_SZ) = 2048
Audit buffer size (4KB) (AUDIT_BUF_SZ) = 0
Size of instance shared memory (4KB) (INSTANCE_MEMORY) = 62914560
Instance memory for restart light (%) (RSTRT_LIGHT_MEM) = 3
Backup buffer default size (4KB) (BACKBUFSZ) = 1024
Restore buffer default size (4KB) (RESTBUFSZ) = 1024
Agent stack size (AGENT_STACK_SZ) = 1024
Sort heap threshold (4KB) (SHEAPTHRES) = 0
Directory cache support (DIR_CACHE) = NO
Application support layer heap size (4KB) (ASLHEAPSZ) = 16
Max requester I/O block size (bytes) (RQRIOBLK) = 65000
Query heap size (4KB) (QUERY_HEAP_SZ) = 1000
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
153/168
Workload impact by throttled utilities(UTIL_IMPACT_LIM) = 10
Priority of agents (AGENTPRI) = SYSTEM
Agent pool size (NUM_POOLAGENTS) = 100
Initial number of agents in pool (NUM_INITAGENTS) = 5
Max number of coordinating agents (MAX_COORDAGENTS) = AUTOMATIC(200)
Max number of client connections (MAX_CONNECTIONS) = AUTOMATIC(MAX_COORDAGENTS)
Keep fenced process (KEEPFENCED) = NO
Number of pooled fenced processes (FENCED_POOL) = MAX_COORDAGENTS
Initial number of fenced processes (NUM_INITFENCED) = 0
Index re-creation time and redo index build (INDEXREC) = RESTART
Transaction manager database name (TM_DATABASE) = 1ST_CONN
Transaction resync interval (sec) (RESYNC_INTERVAL) = 180
SPM name (SPM_NAME) =
SPM log size (SPM_LOG_FILE_SZ) = 256
SPM resync agent limit (SPM_MAX_RESYNC) = 20
SPM log path (SPM_LOG_PATH) =
TCP/IP Service name (SVCENAME) = SAP® db2D3D
Discovery mode (DISCOVER) = SEARCH
Discover server instance (DISCOVER_INST) = ENABLE
SSL server keydb file (SSL_SVR_KEYDB) =
SSL server stash file (SSL_SVR_STASH) =
SSL server certificate label (SSL_SVR_LABEL) =
SSL service name (SSL_SVCENAME) =
SSL cipher specs (SSL_CIPHERSPECS) =
SSL versions (SSL_VERSIONS) =
SSL client keydb file (SSL_CLNT_KEYDB) =
SSL client stash file (SSL_CLNT_STASH) =
Maximum query degree of parallelism (MAX_QUERYDEGREE) = ANY
Enable intra-partition parallelism (INTRA_PARALLEL) = NO
Maximum Asynchronous TQs per query (FEDERATED_ASYNC) = 0
No. of int. communication buffers(4KB)(FCM_NUM_BUFFERS) = AUTOMATIC(9182)
No. of int. communication channels (FCM_NUM_CHANNELS) = AUTOMATIC(2048)
Node connection elapse time (sec) (CONN_ELAPSE) = 10
Max number of node connection retries (MAX_CONNRETRIES) = 5
Max time difference between nodes (min) (MAX_TIME_DIFF) = 1
db2start/db2stop timeout (min) (START_STOP_TIME) = 10
CF Server Configuration:
Memory size (4KB) (CF_MEM_SZ) = 62000128
Number of worker threads (CF_NUM_WORKERS) = 48
Number of connections (CF_NUM_CONNS) = AUTOMATIC(14)
Diagnostic error capture level (CF_DIAGLEVEL) = 2
Diagnostic data directory path (CF_DIAGPATH) =
10.5 DB2 Configuration Settings
Database Configuration for Database d3d
Description Parameter Current Value
------------------------------------------------------------------------------
Database configuration release level = 0x0e00
Database release level = 0x0e00
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
154/168
Database territory = en_US
Database code page = 1208
Database code set = UTF-8
Database country/region code = 1
Database collating sequence = IDENTITY_16BIT
Alternate collating sequence (ALT_COLLATE) =
Number compatibility = OFF
Varchar2 compatibility = OFF
Date compatibility = OFF
Database page size = 16384
Dynamic SQL Query management (DYN_QUERY_MGMT) = DISABLE
Statement concentrator (STMT_CONC) = OFF
Discovery support for this database (DISCOVER_DB) = ENABLE
Restrict access = NO
Default query optimization class (DFT_QUERYOPT) = 5
Degree of parallelism (DFT_DEGREE) = 1
Continue upon arithmetic exceptions (DFT_SQLMATHWARN) = NO
Default refresh age (DFT_REFRESH_AGE) = 0
Default maintained table types for opt (DFT_MTTB_TYPES) = SYSTEM
Number of frequent values retained (NUM_FREQVALUES) = 10
Number of quantiles retained (NUM_QUANTILES) = 20
Decimal floating point rounding mode (DECFLT_ROUNDING) = ROUND_HALF_EVEN
Backup pending = NO
All committed transactions have been written to disk = NO
Rollforward pending = NO
Restore pending = NO
Multi-page file allocation enabled = YES
Log retain for recovery status = NO
User exit for logging status = NO
Self tuning memory (SELF_TUNING_MEM) = OFF
Size of database shared memory (4KB) (DATABASE_MEMORY) = 60293120
Database memory threshold (DB_MEM_THRESH) = 10
Max storage for lock list (4KB) (LOCKLIST) = 200000
Percent. of lock lists per application (MAXLOCKS) = 97
Package cache size (4KB) (PCKCACHESZ) = 120000
Sort heap thres for shared sorts (4KB) (SHEAPTHRES_SHR) = 2400000
Sort list heap (4KB) (SORTHEAP) = 2048
Database heap (4KB) (DBHEAP) = AUTOMATIC(10000)
Catalog cache size (4KB) (CATALOGCACHE_SZ) = 30000
Log buffer size (4KB) (LOGBUFSZ) = 4096
Utilities heap size (4KB) (UTIL_HEAP_SZ) = 10000
Buffer pool size (pages) (BUFFPAGE) = 10000
SQL statement heap (4KB) (STMTHEAP) = AUTOMATIC(38563)
Default application heap (4KB) (APPLHEAPSZ) = AUTOMATIC(256)
Application Memory Size (4KB) (APPL_MEMORY) = AUTOMATIC(241024)
Statistics heap size (4KB) (STAT_HEAP_SZ) = AUTOMATIC(4384)
Interval for checking deadlock (ms) (DLCHKTIME) = 10000
Lock timeout (sec) (LOCKTIMEOUT) = 3600
Changed pages threshold (CHNGPGS_THRESH) = 40
Number of asynchronous page cleaners (NUM_IOCLEANERS) = 12
Number of I/O servers (NUM_IOSERVERS) = 20
Index sort flag (INDEXSORT) = YES
Sequential detect flag (SEQDETECT) = NO
Default prefetch size (pages) (DFT_PREFETCH_SZ) = AUTOMATIC
Track modified pages (TRACKMOD) = NO
Default number of containers = 1
Default tablespace extentsize (pages) (DFT_EXTENT_SZ) = 2
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
155/168
Max number of active applications (MAXAPPLS) = AUTOMATIC(4747)
Average number of active applications (AVG_APPLS) = AUTOMATIC(3)
Max DB files open per application (MAXFILOP) = 61440
Log file size (4KB) (LOGFILSIZ) = 32760
Number of primary log files (LOGPRIMARY) = 100
Number of secondary log files (LOGSECOND) = 154
Changed path to log files (NEWLOGPATH) =
Path to log files = /db2/D3D/logs/DBPART
Overflow log path (OVERFLOWLOGPATH) =
Mirror log path (MIRRORLOGPATH) =
First active log file =
Block log on disk full (BLK_LOG_DSK_FUL) = YES
Block non logged operations (BLOCKNONLOGGED) = NO
Percent max primary log space by transaction (MAX_LOG) = 0
Num. of active log files for 1 active UOW(NUM_LOG_SPAN) = 0
Group commit count (MINCOMMIT) = 1
Percent log file reclaimed before soft chckpt (SOFTMAX) = 1000
Log retain for recovery enabled (LOGRETAIN) = OFF
User exit for logging enabled (USEREXIT) = OFF
HADR database role = STANDARD
HADR local host name (HADR_LOCAL_HOST) =
HADR local service name (HADR_LOCAL_SVC) =
HADR remote host name (HADR_REMOTE_HOST) =
HADR remote service name (HADR_REMOTE_SVC) =
HADR instance name of remote server (HADR_REMOTE_INST) =
HADR timeout value (HADR_TIMEOUT) = 120
HADR log write synchronization mode (HADR_SYNCMODE) = NEARSYNC
HADR peer window duration (seconds) (HADR_PEER_WINDOW) = 0
First log archive method (LOGARCHMETH1) = OFF
Options for logarchmeth1 (LOGARCHOPT1) =
Second log archive method (LOGARCHMETH2) = OFF
Options for logarchmeth2 (LOGARCHOPT2) =
Failover log archive path (FAILARCHPATH) =
Number of log archive retries on error (NUMARCHRETRY) = 5
Log archive retry Delay (secs) (ARCHRETRYDELAY) = 20
Vendor options (VENDOROPT) =
Auto restart enabled (AUTORESTART) = ON
Index re-creation time and redo index build (INDEXREC) = SYSTEM (RESTART)
Log pages during index build (LOGINDEXBUILD) = OFF
Default number of loadrec sessions (DFT_LOADREC_SES) = 1
Number of database backups to retain (NUM_DB_BACKUPS) = 12
Recovery history retention (days) (REC_HIS_RETENTN) = 60
Auto deletion of recovery objects (AUTO_DEL_REC_OBJ) = OFF
TSM management class (TSM_MGMTCLASS) =
TSM node name (TSM_NODENAME) =
TSM owner (TSM_OWNER) =
TSM password (TSM_PASSWORD) =
Automatic maintenance (AUTO_MAINT) = OFF
Automatic database backup (AUTO_DB_BACKUP) = OFF
Automatic table maintenance (AUTO_TBL_MAINT) = ON
Automatic runstats (AUTO_RUNSTATS) = ON
Automatic statement statistics (AUTO_STMT_STATS) = ON
Automatic statistics profiling (AUTO_STATS_PROF) = OFF
Automatic profile updates (AUTO_PROF_UPD) = OFF
Automatic reorganization (AUTO_REORG) = OFF
Auto-Revalidation (AUTO_REVAL) = DEFERRED
CF Resource Configuration:
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
156/168
CF database memory size (4KB) (CF_DB_MEM_SZ) = 61380000
Group buffer pool size (4KB) (CF_GBP_SZ) = 55994112
Global lock memory size (4KB) (CF_LOCK_SZ) = 2000128
Shared communication area size (4KB) (CF_SCA_SZ) = 2337792
Catchup target for secondary CF (mins)(CF_CATCHUP_TRGT) = AUTOMATIC(15)
Currently Committed (CUR_COMMIT) = ON
CHAR output with DECIMAL input (DEC_TO_CHAR_FMT) = NEW
Enable XML Character operations (ENABLE_XMLCHAR) = YES
WLM Collection Interval (minutes) (WLM_COLLECT_INT) = 0
Monitor Collect Settings
Request metrics (MON_REQ_METRICS) = BASE
Activity metrics (MON_ACT_METRICS) = BASE
Object metrics (MON_OBJ_METRICS) = BASE
Unit of work events (MON_UOW_DATA) = NONE
Lock timeout events (MON_LOCKTIMEOUT) = NONE
Deadlock events (MON_DEADLOCK) = WITHOUT_HIST
Lock wait events (MON_LOCKWAIT) = NONE
Lock wait event threshold (MON_LW_THRESH) = 5000000
Number of package list entries (MON_PKGLIST_SZ) = 32
Lock event notification level (MON_LCK_MSG_LVL) = 1
SMTP Server (SMTP_SERVER) =
SQL conditional compilation flags (SQL_CCFLAGS) =
Section actuals setting (SECTION_ACTUALS) = NONE
10.6 enable_dr_nsd.sh
The purpose of this script is to re-map the replicated devices back to the NSD meta-data that
was imported as part of DR site setup process.
DB2PS-member2:~/scripts # vi enable_dr_nsd.sh
#!/bin/ksh # Remove nsddevices userexit rm -f /var/mmfs/etc/nsddevices # Run GPFS device discovery /usr/lpp/mmfs/bin/mmdevdiscover
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
157/168
# List the direct-attached nsds echo "\nGPFS NSDs visible to the cluster ...\n" mmlsnsd -m # Mount the GPFS filesystems cat > /tmp/drfs.out <<EOF d3d log0 log1 log2 log3 SAP® data1 SAP® data10 SAP® data11 SAP® data12 SAP® data13 SAP® data14 SAP® data15 SAP® data16 SAP® data2 SAP® data3 SAP® data4 SAP® data5 SAP® data6 SAP® data7 SAP® data8 SAP® data9 EOF for f in `cat /tmp/drfs.out` do mmmount $f -a done # List all mounted GPFS file systems echo "\nMount status of all GPFS filesystems define d in the cluster ...\n" mmlsmount all –L
10.7 disable_dr_nsd.sh
GPFS daemons during startup find NSDs by reading an NSD ID (in first two sectors) from each
block device visible to it. However, unless a site-failure is initiated LUNs replicated via Metro-
Mirror is not host-addressible on the DR site. Therefore we need to ensure that GPFS is not able
to see those Metro-Mirror replicated NSD definitions (we imported during DR site setup), during
the non-site failure times.
The following script masks those Metro-Mirror replicated NSD definitions from the DR site GPFS
cluster by creating a nsddevices user-exit and initiating mmdevdiscover to rescan the devices
from GPFS.
DB2PS-member2:~/scripts # vi disable_dr_nsd.sh
#!/bin/ksh # Unmount the PPRC'd GPFS filesystems cat > /tmp/drfs.out <<EOF d3d
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
158/168
log0 log1 log2 log3 SAP® data1 SAP® data10 SAP® data11 SAP® data12 SAP® data13 SAP® data14 SAP® data15 SAP® data16 SAP® data2 SAP® data3 SAP® data4 SAP® data5 SAP® data6 SAP® data7 SAP® data8 SAP® data9 EOF for f in `cat /tmp/drfs.out` do mmumount $f -a done # List all mounted GPFS file systems echo "\nMount status of all GPFS filesystems define d in the cluster ...\n" mmlsmount all -L # Create nsddevices script cat > /var/mmfs/etc/nsddevices <<"EndofUserExit" #!/bin/ksh # @(#)53 1.5 src/avs/fs/mmfs/ts/config/nsdd evices.sample, mmfs, avs_rche, rche0933a 8/9/04 16:51:23 ################################################### ########################### # # When properly installed, this script is invoked b y the # /usr/lpp/mmfs/bin/mmdevdiscover script. # # INSTALLATION GUIDELINES FOR THIS SCRIPT: # # a) edit this script using the configuration gui delines below # b) copy this script to /var/mmfs/etc/nsddevices # c) ensure this script is executable (chmod +x / var/mmfs/etc/nsddevices) # # DESCRIPTION OF NSD DEVICE DISCOVERY: # # The mmdevdiscover script and conversely this sc ript are invoked by # the mmfsd daemon when it tries to discover or v erify physical # devices previously defined to GPFS with the mmc rnsd command. These # scripts identify devices found in the /dev file system on the local # machine that may correlate to NSDs defined to G PFS. # # GPFS uses the list of devices output by these s cripts in mapping # the NSD name listed in the configuration databa se to a local device # in the /dev file system. When an NSD is create d via the mmcrnsd # command it is marked with a unique identifier w ritten to sector # two of the device. This unique identifier is r ecorded in the # configuration database along with the user reco gnizable NSD name.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
159/168
# # During GPFS disk discovery each device name out put by mmdevdiscover # and nsddevices is opened in turn and sector two of each device is # read. If a match between an NSD identifier on the device and an # identifier recorded in the configuration databa se is found, then # this machine has local access to the NSD device . I/O is thus # subsequently performed via this local /dev inte rface. # # CONFIGURATION AND EDITING GUIDELINES: # # If this script is not installed then disk disco very is done # only via the commands listed in mmdevdiscover. # # If this script is installed and returns a NON Z ERO return code # then the disk discovery commands listed in mmde vdiscover will ALSO # be run. # # If this script is installed and returns a ZERO return code # then the disk discovery commands listed in mmde vdiscover will NOT # be run. # # The output from both this script and nsddevices is a number # of lines in the following format: # # deviceName deviceType # # where (deviceName) is a device name such as (hd isk1) # and (deviceType) is a set of known disk types. Consult # # /usr/lpp/mmfs/bin/mmdevdiscover # # for a list of currently known deviceTypes # # Example output: # # hdisk1 hdisk # hdisk2 hdisk # ################################################### ########################### osName=$(/bin/uname -s) if [[ $osName = Linux ]] then # Add function to discover disks in the Lin ux environment. multipath -l | awk '$3 ~/dm-/ && $1~/[_qr]$ / || $1~/^[mq_ backup]/ {print $3}' | sort | xargs -I {} echo "{} dmm" fi if [[ $osName = AIX ]] then :# Add function to discover disks in the AI X environment. fi # To bypass the GPFS disk discovery (/usr/lpp/mmfs/ bin/mmdevdiscover), return 0 # To continue with the GPFS disk discovery steps, # return 1 EndofUserExit
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
160/168
# Set executable permissions chmod +x /var/mmfs/etc/nsddevices # Run GPFS device discovery /usr/lpp/mmfs/bin/mmdevdiscover # List the direct-attached nsds echo "\nGPFS NSDs visible to the cluster ...\n" mmlsnsd -m
10.8 disable_dr_nsd_and_enable_fc_nsd.sh
The purpose of this script is to unmap block devices backed by PPRC replicated NSDs and map
block devices backed by FC target devices.
DB2PS-member0:~/scripts # cat disable_dr_nsd_and_enable_fc_nsd.sh #!/bin/ksh # Create nsddevices script cat > /var/mmfs/etc/nsddevices <<"EndofUserExit" #!/bin/ksh # @(#)53 1.5 src/avs/fs/mmfs/ts/config/nsdd evices.sample, mmfs, avs_rche, rche0933a 8/9/04 16:51:23 ################################################### ########################### # # When properly installed, this script is invoked b y the # /usr/lpp/mmfs/bin/mmdevdiscover script. # # INSTALLATION GUIDELINES FOR THIS SCRIPT: # # a) edit this script using the configuration gui delines below # b) copy this script to /var/mmfs/etc/nsddevices # c) ensure this script is executable (chmod +x / var/mmfs/etc/nsddevices) # # DESCRIPTION OF NSD DEVICE DISCOVERY: # # The mmdevdiscover script and conversely this sc ript are invoked by # the mmfsd daemon when it tries to discover or v erify physical # devices previously defined to GPFS with the mmc rnsd command. These # scripts identify devices found in the /dev file system on the local # machine that may correlate to NSDs defined to G PFS. # # GPFS uses the list of devices output by these s cripts in mapping # the NSD name listed in the configuration databa se to a local device # in the /dev file system. When an NSD is create d via the mmcrnsd # command it is marked with a unique identifier w ritten to sector # two of the device. This unique identifier is r ecorded in the # configuration database along with the user reco gnizable NSD name. # # During GPFS disk discovery each device name out put by mmdevdiscover # and nsddevices is opened in turn and sector two of each device is # read. If a match between an NSD identifier on the device and an # identifier recorded in the configuration databa se is found, then # this machine has local access to the NSD device . I/O is thus # subsequently performed via this local /dev inte rface. # # CONFIGURATION AND EDITING GUIDELINES:
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
161/168
# # If this script is not installed then disk disco very is done # only via the commands listed in mmdevdiscover. # # If this script is installed and returns a NON Z ERO return code # then the disk discovery commands listed in mmde vdiscover will ALSO # be run. # # If this script is installed and returns a ZERO return code # then the disk discovery commands listed in mmde vdiscover will NOT # be run. # # The output from both this script and nsddevices is a number # of lines in the following format: # # deviceName deviceType # # where (deviceName) is a device name such as (hd isk1) # and (deviceType) is a set of known disk types. Consult # # /usr/lpp/mmfs/bin/mmdevdiscover # # for a list of currently known deviceTypes # # Example output: # # hdisk1 hdisk # hdisk2 hdisk # ################################################### ########################### osName=$(/bin/uname -s) if [[ $osName = Linux ]] then # Add function to discover disks in the Lin ux environment. multipath -l | awk '($1 ~/^f/ || $1 ~/^db2f s1/) && $3 ~/dm-/ {print $3}' | sort | xargs -I {} echo "{} dmm" fi if [[ $osName = AIX ]] then :# Add function to discover disks in the AI X environment. fi # To bypass the GPFS disk discovery (/usr/lpp/mmfs/ bin/mmdevdiscover), return 0 # To continue with the GPFS disk discovery steps, # return 1 EndofUserExit # Set executable permissions chmod +x /var/mmfs/etc/nsddevices # Run GPFS device discovery /usr/lpp/mmfs/bin/mmdevdiscover # List the direct-attached nsds echo "\nGPFS NSDs visible to the cluster ...\n" mmlsnsd -m
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
162/168
exit 0
10.9 suspend_or_resume_gpfs_fs.sh
This script allows, to:
a) suspend/freeze
b) resume/thaw
GPFS file systems replicated via PPRC link from Primary site to the DR site. The script must be
run from one of the cluster nodes on the primary site.
DB2PS-member2:~/scripts # cat suspend_or_resume_gpfs_fs.sh #! /bin/ksh if [ $# -ne 1 ] then echo "Usage $0 <suspend|resume>" exit fi option=$1 case $1 in suspend|resume ) echo $option ;; *) echo "Wrong option : $option"; exit ;; esac echo "Issuing GPFS I/O $option to filesystems used by database D3D ..."; mmlsnsd | awk '$1 ~/^[^qr File -]/ {print $1}' | se d '/db2fs1/'d | xargs -I {} mmfsctl {} $option
10.10 remote_flash_copy.sh
This script allows:
a) Creating a Remote FlashCopy relationship
b) Refreshing / re-syncing an already established Remote FC relationship
c) Listing the status of a Remote FC relationship / copy-session
d) Deleting an already established Remote FC relationship
The script should be run from a node on primary site with DSCLI installed and has a network
connection to the remote DS8K frame as well.
DB2PS-member2:~/scripts # cat remote_flash_copy.sh #target disk = source disk + 3000 if [ $# -ne 1 ] then echo "Usage $0 <mkremoteflash|rmremoteflash|remotef lash|lsremoteflash>" exit fi option=$1 case $1 in
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
163/168
lsremoteflash|remoteflash|rmremoteflash|mkremotefla sh ) echo $option ;; *) echo "Wrong option : $option" exit ;; esac list="1000 1002 1004 1006 1008 1010 1012 1014 1016 1018 1022" fc_list1="" for source in $list do (( target = $source + 3000 )) fc_list1="$fc_list1 $source:$target" done list="1101 1103 1105 1107 1109 1111 1113 1115 1117 1119" fc_list2="" for source in $list do (( target = $source + 3000 )) fc_list2="$fc_list2 $source:$target" done case $option in mkremoteflash) cmd=mkremoteflash; opt="-freeze -record -persist" ;; remoteflash) cmd=resyncremoteflash; opt="-freeze -record -persist" ;; rmremoteflash) cmd=rmremoteflash; opt="-quiet" ;; lsremoteflash) cmd=lsremoteflash; opt="-l";; esac echo "Executing the following DSCLI command for use r option $option:" # Execute DSCLI command echo "/opt/ibm/dscli/dscli -cfg /root/scripts/PR_ds cli.profile $cmd -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/10 $opt $fc_list1" /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile $cmd -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/10 $opt $fc_list1 echo "/opt/ibm/dscli/dscli -cfg /root/scripts/PR_ds cli.profile $cmd -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/11 $opt $fc_list2" /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile $cmd -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/11 $opt $fc_list2 # Unfreeze the FlashCopy session to prevent TPCR ti mingout and suspending copy-pairs in the PPRC consistency group case $option in mkremoteflash|remoteflash) ssh root@DB2PS-member0 ". /root/.profile; /opt/ibm/dscli/dscli -cfg /root/scripts/DR_dscli.pr ofile unfreezeflash -dev IBM.2107-75WR541 10 11";; esac
Notes:
Work-around to avoid PPRC session being suspended when TPC-R is managing the copy
session
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
164/168
• Both mkremoteflash and resyncremoteflash are issued using '-freeze ' option in
our script to ensures that volumes on the target LSSs are consistent with the source LSSs
volumes.
• The -freeze option creates a Freeze Consistency Group condition -- i.e. it'll trigger a queue
full condition for the source volumes,
o All writes to the source volumes are queued by the host and are written after the queue
full condition is reset.
o This queue full condition is reset by an extended long busy timeout condition -- default
for the timeout is 2 min.
o The timeout condition affects all FlashCopy source volumes that are contained within a
respective logical subsystem (LSS) and that are established or modified with the -freeze parameter.
• When -freeze option is issued to a PPRC target which is the case when using Remote FC,
the following is the behaviour:
o The Freeze Consistency Group condition stops further I/O to PPRC target volumes (which
in turn act as the source for FC).
o Until extended long busy timeout condition is exceeded, those volumes will be frozen.
o Note that TPC-R monitoring timeout (default is 30 sec) is less than long busy timeout
(default is 2min).
o We issued Remote FC commands (mkremoteflash / resyncremoteflash )
directly on the remote from using DSCI, which is outside of TPC-R – i.e. TPC-R did not
issue the I/O freeze on the PPRC pipe.
o Hence, that will cause TPC-R to detect this I/O freeze on the remote frame as an error
condition, which will in turn result in TPC-R cutting the PPRC link and suspending the
replication copy sets.
o The I/O freeze will be reset after extended long busy timeout condition is exceeded.
However, that will not automatically re-start/re-sync PPRC session managed by TPC-R.
• To avoid having to manually restart the PPRC session every time a Remote FC resync is done,
the following work-around is done:
o Issue the FC command with consistency group to all of the FC pairs
o Issue a resume to each of those LSSes once all of the FC commands finished -- a FC
'unfreezeflash' command
o Assuming this was done quickly enough (i.e. before TPC-R detects), then non of the
PPRC copy sets would be suspended
• This is the reason for the unfreezeflash command issued after, either a mkremoteflash or a resyncremoteflash command in our script.
Work-around to use Remote FlashCopy when the PPRC copy-set spans over multiple LSSes
• Remote FC command –conduit option limits source:target pairs to one LSS.
• Hence if your PPRC copy-set spans over multiple LSSes, then you need to issue Remote FC
commands grouped by copy-sets within a single LSS. For example, since our test
configuration had copy sets in two LSSes, (10 and 11) we had to issue two Remote FC
commands in our script, one for each LSS.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
165/168
• Note that the FC Freeze Consistency is only applicable for copy-sets within that LSS. Hence,
this would mean we are staggering our consistency over a period of time when more than
one LSS is involved.
• Hence to guarantee the integrity of the FC target clone, we suspend both DB2 and GPFS I/O
before issuing either a mkremoteflash or a resyncremoteflash command.
The two profile scripts that describe the configuration used to connect to each Storage frame, is
as follows: DB2PS-member2:~/scripts # cat DR_dscli.profile # # DS CLI Profile for ATS_MOP # # # created and modified by easy_dscli hmc1: 10.15.8.70 hmc2: username: db2 password: xxxxxxxx devid: IBM.2107-75WR541 remotedevid: IBM.2107-WR541 fullid: off banner: on verbose: off paging: off olc:off # End of Profile DB2PS-member2:~/scripts # cat PR_dscli.profile # # DS CLI Profile for ATS_MOP # # # created and modified by easy_dscli hmc1: 10.15.8.71 hmc2: username: db2 password: xxxxxxxx devid: IBM.2107-75WR821 remotedevid: IBM.2107-75WR821 fullid: off banner: on verbose: off paging: off olc:off # End of Profile
10.11 disable_repl_nsd.sh
An nsddevices user-exit to unmap (mask) both Metro-Mirror and FC replicated NSDs on DR
cluster. This script must be run on all DR cluster nodes.
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
166/168
GPFS daemons during startup find NSDs by reading an NSD ID (in first two sectors) from each
block device visible to it. However, unless a site-failure is initiated LUNs replicated via Metro-
Mirror is not host-addressable on the DR site. Similarly, unless there is a need to access the
clone copy, the FC targets should be accessed as well. Therefore we need to ensure that GPFS is
not able to see both Metro-Mirror replicated NSD definitions (we imported during DR site
setup), and FC target NSD definitions, during the normal operational state.
/root/scripts/ disable_repl_nsd.sh #!/bin/ksh # Unmount the PPRC'd/FC GPFS filesystems (note both PPRC and FC targets have same NSD meta) cat > /tmp/dr_or_fc_fs.out <<EOF d3d log0 log1 log2 log3 SAP® data1 SAP® data10 SAP® data11 SAP® data12 SAP® data13 SAP® data14 SAP® data15 SAP® data16 SAP® data2 SAP® data3 SAP® data4 SAP® data5 SAP® data6 SAP® data7 SAP® data8 SAP® data9 EOF for f in `cat /tmp/dr_or_fc_fs.out` do mmumount $f -a done # List all mounted GPFS file systems echo "\nMount status of all GPFS filesystems define d in the cluster ...\n" mmlsmount all -L # Create nsddevices script cat > /var/mmfs/etc/nsddevices <<"EndofUserExit" #!/bin/ksh # @(#)53 1.5 src/avs/fs/mmfs/ts/config/nsdd evices.sample, mmfs, avs_rche, rche0933a 8/9/04 16:51:23 ################################################### ########################### # # When properly installed, this script is invoked b y the # /usr/lpp/mmfs/bin/mmdevdiscover script. # # INSTALLATION GUIDELINES FOR THIS SCRIPT: #
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
167/168
# a) edit this script using the configuration gui delines below # b) copy this script to /var/mmfs/etc/nsddevices # c) ensure this script is executable (chmod +x / var/mmfs/etc/nsddevices) # # DESCRIPTION OF NSD DEVICE DISCOVERY: # # The mmdevdiscover script and conversely this sc ript are invoked by # the mmfsd daemon when it tries to discover or v erify physical # devices previously defined to GPFS with the mmc rnsd command. These # scripts identify devices found in the /dev file system on the local # machine that may correlate to NSDs defined to G PFS. # # GPFS uses the list of devices output by these s cripts in mapping # the NSD name listed in the configuration databa se to a local device # in the /dev file system. When an NSD is create d via the mmcrnsd # command it is marked with a unique identifier w ritten to sector # two of the device. This unique identifier is r ecorded in the # configuration database along with the user reco gnizable NSD name. # # During GPFS disk discovery each device name out put by mmdevdiscover # and nsddevices is opened in turn and sector two of each device is # read. If a match between an NSD identifier on the device and an # identifier recorded in the configuration databa se is found, then # this machine has local access to the NSD device . I/O is thus # subsequently performed via this local /dev inte rface. # # CONFIGURATION AND EDITING GUIDELINES: # # If this script is not installed then disk disco very is done # only via the commands listed in mmdevdiscover. # # If this script is installed and returns a NON Z ERO return code # then the disk discovery commands listed in mmde vdiscover will ALSO # be run. # # If this script is installed and returns a ZERO return code # then the disk discovery commands listed in mmde vdiscover will NOT # be run. # # The output from both this script and nsddevices is a number # of lines in the following format: # # deviceName deviceType # # where (deviceName) is a device name such as (hd isk1) # and (deviceType) is a set of known disk types. Consult # # /usr/lpp/mmfs/bin/mmdevdiscover # # for a list of currently known deviceTypes # # Example output: # # hdisk1 hdisk # hdisk2 hdisk # ################################################### ########################### osName=$(/bin/uname -s) if [[ $osName = Linux ]]
IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier
© Copyright IBM Corporation All Rights Reserved 2011
DB2 PS SAP® TRBK
168/168
then # Add function to discover disks in the Lin ux environment.
multipath -l | awk '($1 ~/^db2fs1/ || $1~/[_qr]$/ | | $1~/^[mq_ backup]/ && $3 ~/^dm-/ {print $3})' | sort | xargs -I {} ech o "{} dmm"
fi if [[ $osName = AIX ]] then :# Add function to discover disks in the AI X environment. fi # To bypass the GPFS disk discovery (/usr/lpp/mmfs/ bin/mmdevdiscover), return 0 # To continue with the GPFS disk discovery steps, # return 1 EndofUserExit # Set executable permissions chmod +x /var/mmfs/etc/nsddevices # Run GPFS device discovery /usr/lpp/mmfs/bin/mmdevdiscover # List the direct-attached nsds echo "\nGPFS NSDs visible to the cluster ...\n" mmlsnsd -m exit 0