IBM Tivoli System Automation for Multiplatforms (TSA MP)

Rolling Upgrade Procedure for a TSAMP Automated HADR environment.

This whitepaper is written for a DB2 v9.5/9.7 Automated Failover HADR environment. The goal is to show how to keep the database available on one server while maintenance of any sort is carried out on the other server.

October 2011

Authors: Gareth Holl, IBM Tivoli Support (gholl@us.ibm.com)

Table of Contents

Table of Contents ............................................................................................................................................. 2

Introduction ..................................................................................................................................................... 3

High Level Overview ........................................................................................................................................ 4

A. Display Initial System State .......................................................................................................................... 5

B. Save the automation policy to an XML file .................................................................................................. 6

C. Disable Critical Resource Protection ............................................................................................................ 6

D. Deactivate the Standby Database (on node02) ........................................................................................... 6

E. Shutdown the DB2 Instance on the Standby Node (node02) ....................................................................... 7

F. Offline the Standby Node (node 02) ............................................................................................................ 7

G. Maintenance time for the Standby node (node02) ..................................................................................... 7

H. Bring the Standby Node (node02) back Online ........................................................................................... 8

I. Start the DB2 instance on the Standby node (node02) ................................................................................. 8

J. Perform a controlled failover (to move online resources off current primary node) .................................. 9

K. Repeat steps D through F where the new standby node is now the original primary node (node01). ........ 9

L. Maintenance Time for the New Standby node (node01) ........................................................................... 10

M. Online the node and restart the current Standby DB2 instance ............................................................... 10

N. [OPTIONAL] – Perform a Controlled Failover (Failback) ............................................................................ 11

O. Post Maintenance Task – Activating new RSCT code level (if RSCT was upgraded by any means) ............ 12

P. Post Maintenance Task - Activate new TSA MP code level (if TSAMP was upgraded, including Fixpack upgrade) ........................................................................................................................................................ 13

Q. Post Maintenance Task - Check each component has been migrated completely (if RSCT or TSAMP upgrades performed) ..................................................................................................................................... 14

R. Post Maintenance Task - Re-enable Critical Resource Protection ............................................................. 15

S. Check the state of your HADR environment .............................................................................................. 15

T. [OPTIONAL] Restore/Activate the automation policy previously saved ..................................................... 16

Appendix - References and more information .............................................................................................. 17

2

Introduction

This whitepaper provides a step-by-step approach to preparing each server for planned maintenance activities in a “rolling upgrade” fashion. It is tailored for a DB2 v9.5/9.7 High Availability Disaster Recovery (HADR) environment. This procedure keeps the database accessible throughout the upgrade process, however automated failover and other automated recovery actions would unavailable until all steps were completed.

This document assumes that an Automated Failover HADR environment has been created via the db2haicu tool (for details on constructing such an Automated Failover HADR environment, please consult the white paper “Automated cluster controlled HADR configuration setup using the IBM DB2 high availability instance configuration utility” available at www.ibm.com/developerworks/data/library/long/dm-0907hadrdb2haicu/index.html).

The rolling upgrade procedure is documented in a step by step fashion, beginning with step A. and concluding with step T.; ensure that each step is carefully considered before potentially ignoring. Finally, an Appendix is included for addition information.

The environment used to build this step-by-step guide was a two node cluster using TSAMP v3.2.1.2 and RSCT v3.1.0.4. TSAMP was being used to manage/automate a DB2 v9.7.0.4 HADR environment. However, the same procedure applies to other combinations of the same software set.

In the following text, these conventions apply:

Bold indicates commands that you type.

The prompt indicates the userid that will issue the command. For example,

root@node01:# indicates that the command is issued by the 'root' user on node 'node02'

db2inst1@node02% indicates that the command is issued by the ‘db2inst1’ user on 'node02'

3

High Level Overview

The rolling upgrade process consists of the following steps:1) Stop DB2 and TSAMP services on the current standby node2) Perform required maintenance on the current standby node3) Bring the TSAMP and DB2 services back online on the current standby node4) Perform a takeover so that maintenance can be repeated on the other node5) Repeat steps 1 to 36) [Optional] Perform a takeover to move primary database back to original node

The purpose of doing it this way is to maintain access to the DB2 database with only a minimal interruption to service during a necessary “takeover” step. It is recommended that a planned maintenance period be schedule in case of unforeseen problems.

Please note if your maintenance activities are to include upgrading the TSAMP software from v2.2 directly to v3.2 (or some later fixpack of v3.1), you will not be able to use the rolling upgrade procedure documenting in this guide. You will need to use “Entire domain migration” technique (whitepaper available via TSAMP Support team).

4

A. Display Initial System State

Use the lsrpdomain, lsrpnode, and lssam commands to list and verify the current cluster state and resource states.

Verify the state of the HADR pair using native DB2 commands, for example:

5

root@node01:# export CT_MANAGEMENT_SCOPE=2root@node01:# lsrpdomainName OpState RSCTActiveVersion MixedVersions TSPort GSPorthadrdom Online 3.1.0.4 No 12347 12348

root@node01:# lsrpnodeName OpState RSCTVersionnode02 Online 3.1.0.4node01 Online 3.1.0.4

root@node01:# lssamOnline IBM.ResourceGroup:db2_db2inst1_node02_0-rg Nominal=Online '- Online IBM.Application:db2_db2inst1_node02_0-rs '- Online IBM.Application:db2_db2inst1_node02_0-rs:node02Online IBM.ResourceGroup:db2_db2inst1_node01_0-rg Nominal=Online '- Online IBM.Application:db2_db2inst1_node01_0-rs '- Online IBM.Application:db2_db2inst1_node01_0-rs:node01Online IBM.ResourceGroup:db2_db2inst1_db2inst1_HADRDB-rg Nominal=Online |- Online IBM.Application:db2_db2inst1_db2inst1_HADRDB-rs |- Offline IBM.Application:db2_db2inst1_db2inst1_HADRDB-rs:node02 '- Online IBM.Application:db2_db2inst1_db2inst1_HADRDB-rs:node01 '- Online IBM.ServiceIP:db2ip_10_20_30_40-rs |- Offline IBM.ServiceIP:db2ip_10_20_30_40-rs:node02 '- Online IBM.ServiceIP:db2ip_10_20_30_40-rs:node01

db2inst1@node01% db2pd -hadr -db hadrdbDatabase Partition 0 -- Database HADRDB -- Active -- Up 0 days 00:00:05

HADR Information:Role State SyncMode HeartBeatsMissed LogGapRunAvg (bytes)Primary Peer Sync 0 0

ConnectStatus ConnectTime TimeoutConnected Sun Aug 8 11:00:58 2010 (1241544058) 120

….

B. Save the automation policy to an XML file

Save the current DB2/HADR resources, groups, equivalencies, and relationships (the “automation policy”) to an XML file in case of problems that might require restoration/activation of the policy. The TSAMP command “sampolicy” is used for this task :

C. Disable Critical Resource Protection

Disable Critical Resource Protection so as to prevent unexpected server rebooting :

Verify change in setting as shown :

D. Deactivate the Standby Database (on node02)

As the instance owner, deactivate the database on the standby node only:

This should cause the HADR Resource Group to be automatically locked as shown via the 'lssam' output. This is expected … the HADR Resource Group is locked by the DB2 engine whenever the HADR pair are not in a peer state, by design, to prevent TSAMP from attempting any failover activity.

6

root@node01:# chrsrc -c IBM.PeerNode CritRsrcProtMethod=5

root@node01:# lsrsrc -c IBM.PeerNode CritRsrcProtMethodResource Class Persistent Attributes for IBM.PeerNoderesource 1: CritRsrcProtMethod = 5

root@node01:# sampolicy -s hadr_policy_YYYYMMDD.xml

db2inst1@node02:% db2 deactivate database hadrdb

E. Shutdown the DB2 Instance on the Standby Node (node02)

As the instance owner, stop the DB2 instance on the standby node only :

This should cause the DB2 instance Resource Group on the standby node to be locked so that TSAMP does not attempt to restart the DB2 instance.

F. Offline the Standby Node (node 02)

From the standby node, run the following as the root user:

Wait until the OpState of the domain shows Offline (from the standby node), as shown above, before proceeding to the next step. Note that the domain will still show online if you check from the other node … the 'lsrpnode' command (from the online node) will show the opposite node as Offline.

G. Maintenance time for the Standby node (node02)

At this point, the standby node is in a state where maintenance activities can be performed on the OS, DB2, or TSAMP/RSCT, including any necessary reboots.

7

db2inst1@node02:% db2stop force

root@node02:# stoprpnode –f node02root@node02:# lsrpdomainName OpState RSCTActiveVersion MixedVersions TSPort GSPorthadrdom Offline 3.1.0.4 No 12347 12348

H. Bring the Standby Node (node02) back Online

The standby node needs to be brought back Online after you've completed all maintenance activities on that node. This can be done with the startrpdomain command from the standby node :

Re-issue the “lsrpdomain” command until the domain changes from “Pending online” to “Online” … must each Online before you proceed to the next step.

I. Start the DB2 instance on the Standby node (node02)

As the instance owner, start the DB2 instance on the standby node :

Activate the database if it is not automatically activated:

Check that the database comes up in “Standby” mode and that it reaches a “Peer” state :

8

root@node02:# startrpdomain hadrdomroot@node02:# lsrpdomainName OpState RSCTActiveVersion MixedVersions TSPort GSPorthadrdom Online 3.1.0.4 No 12347 12348

db2inst1@node02% db2pd -hadr -db hadrdbDatabase Partition 0 -- Database HADRDB -- Active -- Up 0 days 00:00:05

HADR Information:Role State SyncMode HeartBeatsMissed LogGapRunAvg (bytes)Standby Peer Sync 0 0

ConnectStatus ConnectTime TimeoutConnected Fri Feb 25 11:21:30 2011 (1241544058) 120

db2inst1@node02:% db2 activate database hadrdb

db2inst1@node02:% db2start

J. Perform a controlled failover (to move online resources off current primary node)

For a DB2 v9.5/9.7 HADR environment, a controlled failover is actually performed by issuing the DB2 takeover command on the current standby node.

Confirm the HADR roles on each node swaps and Peer state is maintained:

K. Repeat steps D through F where the new standby node is now the original primary node (node01).

As the instance owner, deactivate the database and stop the instance on the “new” standby node only:

From the “new” standby node, run the following as the root user:

Wait until the OpState of the domain shows Offline, as shown above, before proceeding to the next step. Note that the domain will still show online if you check from the other node ... the 'lsrpnode' command (from the online node) will show the opposite node as Offline.

9

db2inst1@node01:% db2 deactivate database hadrdbdb2inst1@node01:% db2stop force

db2inst1@node02:% db2 takeover hadr on database hadrdb

db2inst1@node02% db2pd -hadr -db hadrdbDatabase Partition 0 -- Database HADRDB -- Active -- Up 0 days 00:00:05

HADR Information:Role State SyncMode HeartBeatsMissed LogGapRunAvg (bytes)Primary Peer Sync 0 0

ConnectStatus ConnectTime TimeoutConnected Fri Feb 25 11:20:58 2011 (1241544058) 120

….

root@node01:# stoprpnode –f node01root@node01:# lsrpdomainName OpState RSCTActiveVersion MixedVersions TSPort GSPorthadrdom Offline 3.1.0.4 No 12347 12348

L. Maintenance Time for the New Standby node (node01)

At this point, the node is in a state where maintenance activities can be performed on the OS, DB2, or TSAMP/RSCT, including any necessary reboots.

M. Online the node and restart the current Standby DB2 instanceOnce maintenance activities have been completed, bring the current standby node (node01) back online within the domain, restart the DB2 instance, and where necessary activate the database.

Issue the “lsrpdomain” command until the domain changes from “Pending online” to “Online”.

As the instance owner, start the DB2 instance on the standby node :

Activate the database if it is not automatically activated:

Check that the database comes up in “Standby” mode and that it reaches a “Peer” state :

10

root@node01:# startrpdomain hadrdomroot@node01:# lsrpdomainName OpState RSCTActiveVersion MixedVersions TSPort GSPorthadrdom Online 3.1.0.4 No 12347 12348

db2inst1@node01% db2pd -hadr -db hadrdbDatabase Partition 0 -- Database HADRDB -- Active -- Up 0 days 00:00:05

HADR Information:Role State SyncMode HeartBeatsMissed LogGapRunAvg (bytes)Standby Peer Sync 0 0

ConnectStatus ConnectTime TimeoutConnected Fri Feb 25 11:21:30 2011 (1241544058) 120

db2inst1@node01:% db2 activate database hadrdb

db2inst1@node01:% db2start

N. [OPTIONAL] – Perform a Controlled Failover (Failback)

If you prefer that your primary HADR database resides on the node it was originally on (node01) prior to this whole rolling upgrade procedure, issue a DB2 takeover command from the current standby node (node01) :

Check that the roles swap and Peer state is maintained :

11

db2inst1@node01:% db2 takeover hadr on database hadrdb

db2inst1@node02% db2pd -hadr -db hadrdbDatabase Partition 0 -- Database HADRDB -- Active -- Up 0 days 00:00:05

HADR Information:Role State SyncMode HeartBeatsMissed LogGapRunAvg (bytes)Standby Peer Sync 0 0

ConnectStatus ConnectTime TimeoutConnected Fri Feb 25 11:21:30 2011 (1241544058) 120

db2inst1@node01% db2pd -hadr -db hadrdbDatabase Partition 0 -- Database HADRDB -- Active -- Up 0 days 00:00:05

HADR Information:Role State SyncMode HeartBeatsMissed LogGapRunAvg (bytes)Primary Peer Sync 0 0

ConnectStatus ConnectTime TimeoutConnected Fri Feb 25 11:20:58 2011 (1241544058) 120

….

O. Post Maintenance Task – Activating new RSCT code level (if RSCT was upgraded by any means)

If you've patched the AIX operating system, its possible the RSCT software has also been upgraded. If you've applied a Fixpack to the TSAMP software, its also very likely the RSCT software has been upgraded. If this is the case, the online domain will show MixedVersions (lsrpdomain) as “Yes”, and RSCTVersion (lsrpnode) will show the new RSCT level. Assuming you’ve installed the new code on each server, RSCTVersion should be the same on each server. RSCTVersion would be different to the RSCTActiveVersion (lsrpdomain) since the newly installed RSCT level has not been activated yet.

If you’re expecting a change to the RSCT level but MixedVersions is still set to “No”, wait a couple minutes and re-check using “lsrpdomain” and “lsrpnode”.

To activate the new RSCT level, issue the following commands :

12

root@node01:# export CT_MANAGEMENT_SCOPE=2root@node01:# runact -c IBM.PeerDomain CompleteMigration Options=0Resource Class Action Response for CompleteMigration

P. Post Maintenance Task - Activate new TSA MP code level (if TSAMP was upgraded, including Fixpack upgrade)

Before activating the new TSA MP code level, ensure TSAMP (IBM.RecoveryRM) has finished initializing by checking “In Config State” is set to TRUE :

Now activate the new TSA MP level :

13

root@node01:# samctrl -mReady to Migrate! Are you Sure? [Y|N]:.

Y

root@node01:# lssrc -ls IBM.RecoveryRM | grep “In Config State” In Config State : TRUE

Q. Post Maintenance Task - Check each component has been migrated completely (if RSCT or TSAMP upgrades performed)

Ensure that MixedVersions is no longer ‘Yes’ for the Cluster component (RSCT), and that the RSCTActiveVersion shows the same level as the RSCTVersion on each server :

Ensure that the Active Version Number (AVN) matches the Installed Version Number (IVN) for TSA MP :

14

root@node01:# lsrpdomainName OpState RSCTActiveVersion MixedVersions TSPort GSPorthadrdom Online 3.1.0.4 No 12347 12348

root@node01:# lsrpnodeName OpState RSCTVersionnode02 Online 3.1.0.4node01 Online 3.1.0.4

root@node01:# lssrc –ls IBM.RecoveryRM |grep VN Our IVN : 3.2.1.2 Our AVN : 3.2.1.2

R. Post Maintenance Task - Re-enable Critical Resource Protection

Turn critical resource protection (CritRsrcProtMethod) back on :

Verify the change in setting as shown:

S. Check the state of your HADR environment

At this point your DB2 resources should be showing Online ... the ‘lssam’ output should look similar to the following :

15

root@node01:# chrsrc -c IBM.PeerNode CritRsrcProtMethod=1

root@node01:# lsrsrc -c IBM.PeerNodeResource Class Persistent Attributes for IBM.PeerNoderesource 1: CritRsrcProtMethod = 1

root@node01:# lssamOnline IBM.ResourceGroup:db2_db2inst1_node02_0-rg Nominal=Online '- Online IBM.Application:db2_db2inst1_node02_0-rs '- Online IBM.Application:db2_db2inst1_node02_0-rs:node02Online IBM.ResourceGroup:db2_db2inst1_node01_0-rg Nominal=Online '- Online IBM.Application:db2_db2inst1_node01_0-rs '- Online IBM.Application:db2_db2inst1_node01_0-rs:node01Online IBM.ResourceGroup:db2_db2inst1_db2inst1_HADRDB-rg Nominal=Online |- Online IBM.Application:db2_db2inst1_db2inst1_HADRDB-rs |- Offline IBM.Application:db2_db2inst1_db2inst1_HADRDB-rs:node02 '- Online IBM.Application:db2_db2inst1_db2inst1_HADRDB-rs:node01 '- Online IBM.ServiceIP:db2ip_9_26_124_22-rs |- Offline IBM.ServiceIP:db2ip_9_26_124_22-rs:node02 '- Online IBM.ServiceIP:db2ip_9_26_124_22-rs:node01

Again confirm HADR role and state on each server to be sure all is in good order:

T. [OPTIONAL] Restore/Activate the automation policy previously saved

If the automation policy is incomplete for some reason, you can resolve by simply re-activating the automation policy from the XML file saved back in section B, using the TSAMP command “sampolicy” as follows :

16

db2inst1@node01% db2pd -hadr -db hadrdbDatabase Partition 0 -- Database HADRDB -- Active -- Up 0 days 00:00:05

HADR Information:Role State SyncMode HeartBeatsMissed LogGapRunAvg (bytes)Primary Peer Sync 0 0

ConnectStatus ConnectTime TimeoutConnected Fri Feb 25 11:20:58 2011 (1241544058) 120

….

db2inst1@node02% db2pd -hadr -db hadrdbDatabase Partition 0 -- Database HADRDB -- Active -- Up 0 days 00:00:05

HADR Information:Role State SyncMode HeartBeatsMissed LogGapRunAvg (bytes)Standby Peer Sync 0 0

ConnectStatus ConnectTime TimeoutConnected Fri Feb 25 11:21:30 2011 (1241544058) 120

root@node01:# sampolicy -a hadr_policy_YYYYMMDD.xml

Appendix - References and more information

Automated Cluster Controlled HADR (High Availability Disaster Recovery) Configuration Setup using the IBM DB2 High Availability Instance Configuration Utility (db2haicu), by Steve Raspudic, Malaravan Ponnuthurai (IBM Canada Ltd./IBM Toronto Software Lab), June 2009http://www.ibm.com/developerworks/data/library/long/dm-0907hadrdb2haicu/

IBM Red Book: High Availability and Disaster Recovery Options for DB2 on Linux, UNIX, and Windows, by Whei-Jen Chen, Masafumi Otsuki, Paul Descovich, Selvaprabhu Arumuggharaj, Toshihiko Kubo and Yong Jun Bi (IBM Corporation), February 2009.http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg247363.html

IBM Tivoli® System Automation for Multiplatforms (Version 3 Release 1) product/technical documentation:http://publib.boulder.ibm.com/tividd/td/IBMTivoliSystemAutomationforMultiplatforms3.1.html

IBM Tivoli® System Automation for Multiplatforms (Version 3 Release 2) product/technical documentation:http://www.ibm.com/developerworks/wikis/display/tivolidoccentral/Tivoli+System+Automation+for+Multiplatforms

Reliable Scalable Cluster Technology (RSCT) Administration Guidehttp://publib.boulder.ibm.com/infocenter/clresctr

IBM DB2 9.5 and DB2 9.7 for Linux, UNIX, and Windows Information Centers on the Webhttp://publib.boulder.ibm.com/infocenter/db2luw/v9r7/index.jsphttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp

17

© Copyright IBM Corporation, 2011IBM Corporation Software GroupRoute 100Somers, NY 10589U.S.A.Produced in the United States of AmericaFebruary 2010All Rights Reserved

Neither this document nor any part of it may be copied or reproduced in any form or by any means or translated into another language, without the prior consent of the above-mentioned copyright owner.

IBM makes no warranties or representations with respect to the content hereof and specifically disclaims any implied warranties of merchantability or fitness for any particular purpose. IBM assumes no responsibility for any errors that may appear in this document. The information contained in this document is subject to change without any notice. IBM reserves the right to make any such changes withoutobligation to notify any person of such revision or changes. IBM makes no commitment to keep the information contained herein up to date.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

References in this publication to IBM products or services do not imply that IBM intends to make them available in all countries in which IBM operates.

18

Additional Notices Language

This information was developed for products and services offered in the U.S.A.Information about non-IBM products is based on information available at the timeof first publication of this document and is subject to change.

IBM may not offer the products, services, or features discussed in this document inother countries. Consult your local IBM representative for information on theproducts and services currently available in your area. Any reference to an IBMproduct, program, or service is not intended to state or imply that only that IBMproduct, program, or service may be used. Any functionally equivalent product,program, or service that does not infringe any IBM intellectual property right maybe used instead. However, it is the user’s responsibility to evaluate and verify theoperation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matterdescribed in this document. The furnishing of this document does not grant youany license to these patents. You can send license inquiries, in writing, to:IBM Director of LicensingIBM CorporationNorth Castle DriveArmonk, NY 10504-1785U.S.A.

For license inquiries regarding double-byte character set (DBCS) information,contact the IBM Intellectual Property Department in your country or sendinquiries, in writing, to:Intellectual Property LicensingLegal and Intellectual Property LawIBM Japan, Ltd.3-2-12, Roppongi, Minato-ku, Tokyo 106-8711 Japan

Any performance data contained herein was determined in a controlledenvironment. Therefore, the results obtained in other operating environments mayvary significantly. Some measurements may have been made on development-levelsystems, and there is no guarantee that these measurements will be the same ongenerally available systems. Furthermore, some measurements may have beenestimated through extrapolation. Actual results may vary. Users of this documentshould verify the applicable data for their specific environment.

This information could include technical inaccuracies or typographical errors.Changes are periodically made to the information herein; these changes will beincorporated in new editions of the publication. IBM may make improvementsand/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

All statements regarding IBM’s future direction or intent are subject to change orwithdrawal without notice, and represent goals and objectives only.

19