1

May 21, 2008 • 10:00 a.m. – 11:00 a.m.Platform: DB2 for Linux, Unix, Windows

Jessica Escott-RockwoodIBM Toronto Lab

Session: D09

DB2 HADR – Case Study of Implementation

An in-depth look at using DB2 High Availability Disaster Recovery (HADR) in a large enterprise. This presentation will review the HADR technology and walk through the end-to-end process of implementing HADR and automation technology to manage failovers, including: architecture, setup and configuration, testing, and monitoring.

2

2

Session objectives• High Availability Disaster Recovery (HADR)

• Functional overview • Restrictions/limitations • Set up

• How to plan and implement your HADR solution• Best Practices

• HADR operations• Monitoring & alerting• Troubleshooting• DB2 administration in an HADR database

3

3

Goals of HADR• Ultra-fast failover• Easy administration• Handling of site failures • Negligible impact on performance• Configurable degree of consistency • Protection against errant transactions• Software upgrades without interruption• Very easy integration with HA-software • Eventually, no need for HA-software at all• Transparent failover and failback for applications

(combined with “client re-route”)

4

4

Basic principles of HADR• Two active machines

• Primary• Processes transactions• Ships log entries to the other machine

• Standby (or Secondary)• Cloned from the primary• Receives and stores log entries from the primary• Re-applies the transaction

• If the primary fails, the standby can take over the transactional workload• The standby becomes the new primary

• If the failed machine becomes available again, it can try to be resynchronized• The old primary becomes the new standby

5

5

Limitations/Restrictions• No CONNECT access to standby database• Not supported with Data Partitioning Feature (DPF), raw I/O for

transaction logs, or redirected restore• Primary and standby databases must have the same name, the same

operating system and DB2 levels and the same bit-ness (32-bit vs 64-bit)

• Rolling upgrades refer to Fix Pack upgrades only• Non-logged operations are not replicated:

• Updates to DB/DBM cfg and registry• LOBs > 1GB or LOBs marked as NOT LOGGED• NOT LOGGED INITIALLY operations

• Log archiving and self-tuning memory manager (STMM) can only be done on the primary database

With no CONNECT access, utilities cannot be run on the standby database. If the standby database is deactivated, an offline backup can be taken but during the time of deactivation and the subsequent time to “catch-up” to peer state, the standby is not valid for failover.

HADR is also not supported with Data Links in V8, however, note that Data Links is deprecated functionality in DB2 9.

6

6

Scope of activity• HADR replication takes place at the database level

Database E Database E

HADR HADRTCP/IP

Primary Standby

Database A

HADR HADRTCP/IP

Standby Primary

Database A

Database B

Database C

Database D

7

7

HADR implementation details

LogsOld

Logs

Log Writer Log Reader

Tables

IndexesLogs

Old

Logs

Log Reader

Tables

Indexes

TCPIP

DB2 EngineDB2 EnginePRIMARY SERVER

STANDBY SERVER

Log Writer

HADR HADR Shredder Replay Master

Redo SlavesRedo SlavesRedo SlavesReplay Slaves

Log PagesLog Pages

Log Pages

Log RecordsLog Pages

Log

Records

Client ReroutePrimary Connection

8

8

HADR State FlowHADR/DB2 Startup

Local catch-up

Remote catch-up Pending

Remote catch-up

Peer State

ConnectC

onne

ctio

n lo

st

9

9

HADR synchronization modes

Logs

Log writer

Logs

TCP/IP

Commit Succeeded

Synchronous

Near

- Syn

chro

nousAsynchronous

receive()send()

HADR HADR

Log writer

10

10

Planning• Three components to examine:

• Client• Network• Server

• Goal is to identify the expected behavior in the case of a component failure

TCPIP

TSA detects failure of Site A within 5 seconds and invokes forced HADR takeover on Site B

e.g. AIX server failure

Site BSite A

Expected resultFailure description

Failure location

Location operating as primary

11

11

Planning• Client

• Uses DB2 Client or DB2 application driver (Java, CLI)?- With application driver, the Java or CLI client will pick up the alternate

client from the server on first connect. If you need to support the clients being able to reroute without having successfully connected to the “primary” site, then need to harden the information in the client.

• Type of workload? Constant or spikes of activity?- Spikes of utilization may affect the ability of the standby to keep up so

should consider NEARSYNC (or ASYNC if only using the solution for disaster recovery and not HA)

• Availability requirements? Hours of operation?- What’s the priority on the system? Is it availability or ensuring no data

loss - each have a tradeoff and can influence the choice between syncmodes as well as options like HADR_TIMEOUT

- Need to understand if/when there are down-times on the system

TCPIP

Client reroute:For CLI, it’s an option on the definition of the data source. For JCC driver, need to implement the <TBC>

If there are down-times on the system:-Could do off-line maintenance activities or those with high impact to the standby (like offline reorg) during these periods and even consider deactivating the standby, doing maintenance, and then refreshing the standby with a new backup image of the primary

12

12

Planning• Network

• What is the network topology.. same network used between clients & server as between HADR primary & standby?

- Recommended to isolate HADR communication on a separate network from regular database communication; if shared, then ensure optimal network tuning to handle the peak data flow

• Is bandwidth greater than the database log generation rate• Redundant network adapters to avoid single point of failure?• Virtual IP for database servers?

- In the event of primary failure, the IP ownership moves to standby; no clients will be able to connect to the primary box

- Use client reroute by defining the same IP for the alternate client server as for the initial catalog entry

• Server• DB2 and operating system release levels?• Storage … How robust? Where will logs be archived?• Automation software?

TCPIP

Network topology:-Note that HADR requests tcp_nodelay for its connection for optimal performance; if ignored by the OS, it can negatively affect HADR performance-Increasing LOGBUFSZ can increase the amount of log flushed to the standby in each request… but more likely to need to increase DB2_HADR_BUF_SZ registry variable (to more than the default of 2xLOGBUFSZ) in order to handle peaks of log data flushed to standbyStorage:-Storage is a possible point of failure on both the primary and the standby… disk mirroring or some other sort of HA strategy for the disks can avoid requiring a failover due to disk failure; there is no interruption of service if a failover is not required, so always want to look to repair in place when possible-If archived logs are on external storage or something like TSM, then both primary and standby need to be able to access the location; also, delays in retrieving logs can result in significant delays in catch-up by the standby so you may want to retrieve archived logs to the standby before starting HADR to avoid that network overhead (but bear in mind that you need to manage the deletion of those retrieved logs)Server:-Run recommended operating system levels prescribed by IBM ensuring that all OS patches are applied in a timely fashion-Tivoli System Automation for Multiplatforms is available for Linux and AIX. HACMP is available for AIX.

13

13

Case A: Bright Yellow Truck Co.

•p5-575 AIX 5.2 64-bit ML6 with DB2 Viper 2•Tivoli Storage Manager (TSM) 5.3 is used to archive logs•HACMP 5.3 is the incumbent automation software

Server

Single public GB ethernet for all network traffic, heartbeat RSA connection between primary and disaster site

Network

24x7 proprietary application using DB2 clientClient

Alternate Case B: Little Blue Box JewelersClient-Java application running on WebSphere Application Server (WAS) using the JCC Type-4 driver-Application must be operational 8am-5pm weekdays and SaturdayNetwork- Public GB ethernet for client-server traffic, private GB ethernet for HADR connection between primary and disaster siteServer- x3650 Linux SLES9 SP3 with DB2 9- Logs are archived to shared disk- Tivoli System Automation for Multiplatforms 2.1 (TSA) for automation

14

14

Bright Yellow Truck – Planning Analysis

… and so on, including network failure, DB2 failure, filesystem damage/corruption, TSM server failure

Local disk is mirrored, no impact to DB2; storage team to replace failed disk

Disk failureSite ASite A

No automatic action; need to repair server on site B, restart, and regain peer state

AIX server failure

Site BSite A

HACMP invokes rc.hadr.start script on Site B to takeover on standby database; clients reconnect to Site B primary through client reroute

AIX server failure

Site ASite A

Expected ResultFailure description

Failure location

Primary location

15

15

Planning - HADR configuration• Two decisions to make about HADR connection:

• hadr_timeout • Time (in seconds) that HADR waits before considering a

communication attempt as failed• Heartbeat interval will be ¼ of hadr_timeout or 30 seconds,

whichever is shorter• hadr_syncmode

• SYNC, NEARSYNC, or ASYNC• For disaster recovery, most choose between SYNC and

NEARSYNC -> performance testing is valuable to quantify the difference between the two

Note that, for maximal availability, as soon as the connection is closed (because the other end closed the connection, a network error is detected, or timeout is reached), the primary drops out of peer state to avoid blocking transactions.A recommended timeout value depends on the capacity of the data server and of the HADR network connection. If either are busy, then it can result in a delay in response between the HADR pair and that will start the countdown to the HADR_TIMEOUT value.In most enterprise implementations, an HADR_TIMEOUT between 30 and 60 seconds provides the best balance of a short timeout without risking falling out of peer due to too short a timeout. An HADR_TIMEOUT less than 30 seconds should be thoroughly tested to ensure the primary does not drop out of peer state as part ofregular processing.

16

16

Big Yellow Truck Decisions• After testing SYNC and NEARSYNC, no significant performance

impact was detected between the two so set hadr_syncmode=SYNC

• Highest priority at Big Yellow Truck was data integrity – no transactions could be lost without a ripple effect through the organization• Risk even in SYNC mode; if the TCP/IP socket closed on the

primary, the primary could drop out of Peer mode immediately and if the primary failed and a forced takeover occurred before the standby reconnected and regained peer state there was the possibility of data lossChose to implement new Viper 2 feature: HADR peer window

17

17

HADR Peer Window• HADR_PEER_WINDOW can be used to cause the HADR

pair (primary and standby databases) to behave as though in peer state even if the primary loses connection with the standby database• Once connection to standby is lost, primary waits the

number of seconds specified by HADR_PEER_WINDOW before dropping out of Disconnected Peer state and continuing to process transactions

• Peer window end (meaning the time at which the primary is no longer guaranteed to be in peer state) is sent to the standby through the heartbeat - timestamp can be checked by automation software to determine whether or not to do forced takeover

• If standby does not reconnect with primary by the end of the peer window, the primary drops from Disconnected Peer state to Remote Catchup Pending

Note: important to synchronize clocks between primary and standby for effective usage of End of peer window

18

18

Setting HADR_PEER_WINDOW• DB2_HADR_PEER_WINDOW should be set to a value that fits the

following equation:

DB2_HADR_PEER_WINDOW >= response_time + fudge_factor + heartbeat_interval

response_time = estimated time for automation software to detect failure and invoke HADR takeover

fudge_factor = 5 sec, small delta time delay because detection/response is not instantaneous

heartbeat_interval = MIN(HADR_TIMEOUT/4, DB2_HADR_PEER_WINDOW/4, 30 sec)

Big Yellow Trucks determined their response time as a minimum of 45secsHACMP detection of failure: 5 secondsHACMP processing time until HADR takeover invoked: up to 40 seconds

19

19

HADR State Flow with Peer WindowHADR/DB2 Startup

Local catch-up

Remote catch-up Pending

Remote catch-up

Peer State

Connect

Con

nect

ion

re-e

stab

lishe

d

Con

nect

ion

lost

Disconnected Peer State

Connection lost

Peer window expires

20

20

Set-up• Prepare and clone the primary database and

configure for HADR• Create the standby database from the clone• Configure the standby database for HADR• Configure client reroute • Start-up HADR• Set-up automation process

21

21

Prepare and clone the primary DB• Ensure archive logging is enabled on the primary database

Archive log location must be accessible by both primary and standby database

• Sample database configuration for Case A: LOGFILSIZ 16380LOGSECOND 12LOGARCHMETH1 TSM:DB2

• Sample database configuration for Case B:LOGFILSIZ 8192LOGSECOND 2LOGARCHMETH1 DISK:/shared/db2inst1/logarchives

• Take backup or split mirror copy of primary database and move the backup image to the standby location

22

22

Code snippet - Configure primaryexport $dbname=TRUCKERSexport $pri=yellow1.bigtruck.comexport $sec=yellow2.bigtruck.comexport $svc=7000db2 connect to $dbnamedb2 update db cfg for $dbname using LOGFILSIZ 16380db2 update db cfg for $dbname using LOGSECOND 12db2 update db cfg for $dbname using LOGARCHMETH1 TSM:DB2db2 update db cfg for $dbname using INDEXREC RESTARTdb2 update db cfg for $dbname using LOGINDEXBUILD ONdb2 update db cfg for $dbname using HADR_TIMEOUT 30db2 update db cfg for $dbname using HADR_PEER_WINDOW 90db2 update db cfg for $dbname using HADR_LOCAL_HOST $pridb2 update db cfg for $dbname using HADR_LOCAL_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_HOST $secdb2 update db cfg for $dbname using HADR_REMOTE_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_INST sitebdb2 update db cfg for $dbname using HADR_SYNCMODE SYNCdb2 deactivate db $dbnamedb2 activate db $dbname

Configuration for Case B would be:export $dbname=DIAMONDexport $pri=bluebox.jeweler.comexport $sec=bluebox2.jeweler.comexport $svc=7000

db2 connect to $dbnamedb2 update db cfg for $dbname using LOGFILSIZ 8192db2 update db cfg for $dbname using LOGSECOND 2db2 update db cfg for $dbname using LOGARCHMETH1 DISK:/shared/mydb/archivelogsdb2 update db cfg for $dbname using INDEXREC RESTARTdb2 update db cfg for $dbname using LOGINDEXBUILD ONdb2 update db cfg for $dbname using HADR_TIMEOUT 30db2 update db cfg for $dbname using HADR_LOCAL_HOST $pridb2 update db cfg for $dbname using HADR_LOCAL_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_HOST $secdb2 update db cfg for $dbname using HADR_REMOTE_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_INST sitebdb2 update db cfg for $dbname using HADR_SYNCMODE SYNCdb2 deactivate db $dbnamedb2 activate db $dbname

23

23

Create the standby DB and configure• Restore the backup image or db2inidb the split mirror copy

database is left in roll forward pending state

db2 update db cfg for $dbname using LOGFILSIZ 16380db2 update db cfg for $dbname using LOGSECOND 12db2 update db cfg for $dbname using LOGARCHMETH1 TSM:DB2db2 update db cfg for $dbname using INDEXREC RESTARTdb2 update db cfg for $dbname using LOGINDEXBUILD ONdb2 update db cfg for $dbname using HADR_TIMEOUT 30db2 update db cfg for $dbname using HADR_PEER_WINDOW 90db2 update db cfg for $dbname using HADR_LOCAL_HOST $secdb2 update db cfg for $dbname using HADR_LOCAL_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_HOST $pridb2 update db cfg for $dbname using HADR_REMOTE_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_INST siteadb2 update db cfg for $dbname using HADR_SYNCMODE SYNCdb2 deactivate db $dbname

Strict symmetry of table space and container configuration is required on the standby. Name, path, size all must match. Relative container paths are allowed, and the full path may differ in this case.

If strict symmetry does not exist, HADR may fail to replicate a container operation on the standby. In such a case replication of the affected table space stops and the table space will be left in "roll forward in progress"

24

24

Configure automatic client reroute• On both the primary and standby database servers issue:

db2 update alternate server for db $dbname using hostname <host> port <port_num>

• Note: Alternate server is flowed to the client and cached on first connect; without a connection, the client will not know where to re-route• To protect against this on a DB2 client, the same

command can be run on the client to ‘harden’ the alternate server

• On a JCC-Type 4 client with JNDI, use the clientRerouteServerListJNDIName DataSource property to persist alternate server information across JVMs and to ‘harden’ the alternate server

For JDBC and SQLJ clients:http://publib.boulder.ibm.com/infocenter/db2luw/v9/index.jsp?topic=/com.ibm.db2.udb.apdv.java.doc/doc/r0021822.htm

You need to set properties forcom.ibm.db2.jcc.DB2BaseDataSource.clientRerouteAlternatePortNumbercom.ibm.db2.jcc.DB2BaseDataSource.clientRerouteAlternateServerName

25

25

Start-up HADRSTANDBY

db2 start hadr on db $dbname as standby

db2 get snapshot for db on $dbname |grep –A 15 “HADR Status”

ORdb2 select * from

sysibmadm.snaphadr

PRIMARY

db2 start hadr on db $dbname as primary

db2 get snapshot for db on $dbname |grep –A 15 “HADR Status”

Start local catch-up

Validate connection between primary & standby and start remote catch-up

Upon starting the standby, the Local Catchup phase can begin:-Replay locally available log files (if any)-Replay log files that may be retrieved at the primary and shipped to the standby

Once the primary is started (which is an asynchronous operation), remote catchup can begin:-Replay log pages from the primary's archived logs-Replay log pages from the primary's active log to the standby until the standby is caught up to the tail of the log.-Replay log pages from the primary's in-memory log buffer to the standby whenever the primary flushes those pages to disk

At the conclusion of remote catchup, the standby is now in Peer state.

26

26

Start-up on standby – Expected ResultsDB20000I The START HADR ON DATABASE command completed successfully.

HADR StatusRole = StandbyState = Remote catchup pendingSynchronization mode = SyncConnection status = Disconnected, 03/22/2007 09:50:12.423522Peer window end = Null (0)Peer window (seconds) = 65Heartbeats missed = 0Local host = bbird007Local service = HADR_test2Remote host = bbird006Remote service = HADR_test1Remote instance = db2hdatimeout(seconds) = 30Primary log position(file, page, LSN) = S0000115.LOG, 0, 0000000748858000Standby log position(file, page, LSN) = S0000114.LOG, 1, 000000074485D0BCLog gap running average(bytes) = 67088191

27

27

Start-up on primary – Expected Results

DB20000I The START HADR ON DATABASE command completed successfully.

HADR StatusRole = PrimaryState = PeerSynchronization mode = SyncConnection status = Connected, 03/22/2007 09:50:50.622363Peer window end = 03/22/2007 09:51:56.000000 (1174571516)Peer window (seconds) = 65Heartbeats missed = 0Local host = bbird006Local service = HADR_test1Remote host = bbird007Remote service = HADR_test2Remote instance = db2hdatimeout(seconds) = 30Primary log position(file, page, LSN) = S0000115.LOG, 0, 0000000748858000Standby log position(file, page, LSN) = S0000114.LOG, 1, 000000074485D0BCLog gap running average(bytes) = 67088191

28

28

Configure automation software (before DB2 9.5)• For TSA, use DB2-provided scripts hadr_start.ksh,

hadr_stop.ksh, and hadr_monitor.ksh• Reference whitepaper: “Automating DB2 HADR

Failover on Linux using Tivoli System Automation on Multiplatforms”(ftp://ftp.software.ibm.com/software/data/pubs/papers/hadr_tsa.pdf)

• For HACMP, use DB2-provided scripts rc.hadr.start, rc.hadr.stop, rc.hadr.monitor• Reference whitepaper: “Automating IBM DB2 UDB

HADR with HACMP”(ftp://ftp.software.ibm.com/software/data/pubs/papers/hadr-hacmp.pdf)

29

29

Configure automation software (in DB2 9.5)

• For Linux and AIX, Tivoli System Automation for Multiplatforms (TSA MP) is now bundled with and installed by DB2

• db2haicu utility is used to specify the cluster manager configuration• Create a cluster domain

• Define local DB2 instance resource groups on each node• Note that the default behavior is to restart the instance in place by default

• Define quorum device for the domain• Define DB2 HADR database resource across the cluster, online on

the primary and offline on the standby • Define the network interface to the cluster domain

• Optionally, define network resource for a floating virtual IP

• db2pd –ha shows the cluster manager status

Requires a sysadmin step to initialize the TSA environment, but the majority is DBA driven.Vendor interfaces have been provided, but it’s TSA that’s bundled. Work underway with VERITAS, but no immediate plans to include HACMP.

30

30

Next steps• Yippee! HADR is now up and running…

… but now you need to test it and develop procedures for the HADR environment

• Testing matrix should cover all the failure scenarios outlined during the planning phase to validate expected results

• Measurements to capture:• Time to detect failure, time for failover, and time until new

primary is open for business (any open transactions at time of failure rolled back and database ready to process new transactions)

• Performance of workload• Server utilization statistics• Consistency check between primary and standby data

31

31

Best Practices – HADR operations• Database start-up

• STANDBY: db2start; db2 activate db $dbname• PRIMARY: db2start; db2 activate db $dbname• Verify success with snapshot on PRIMARY

• If standby not available, PRIMARY: db2start; db2 start hadr on db $dbname as primary by force

• Database shut-down• PRIMARY: db2 deactivate db $dbname; db2stop force• STANDBY: db2 deactivate db $dbname; db2stop

Recommended aliases:alias gethadrhost=’db2 get db cfg for $dbname |grep HADR_LOCAL_HOST’alias getdbrole=’db2 get db cfg for $dbname |grep “HADR database role”’alias hadrstate=’snaphadr | grep State’alias snaphadr=’db2 get snapshot for db on $dbname |grep –A 15 “HADR Status”’

32

32

Best Practices – HADR operation• Database role reversal

• Verify peer state on primary with snapshot• STANDBY: db2 takeover hadr on db $dbname

• Verify success with snapshot on new PRIMARY

Primary Standby

33

33

Best Practices – HADR operations• Manual database failover

• With peer window configuration, issue on STANDBY: db2 takeover hadr on db $dbname by force peer window only

• Without, issue on STANDBY: db2 takeover hadr on db $dbname

• If this fails, need to evaluate ‘safe-ness’ of doing the forced takeover – risk of split-brain or losing transactions

34

34

Is the primary server (and its disk) available?

YES, OK to run a planned

takeover

On STANDBY, check the HADR state of the database, use alias:

hadrstate

On PRIMARY, run db2pdlog –l <outfile> from the database directory (where SQLOGCTL.LFH exists)

No

UNPLANNED EVENT: Task flow for determining ‘safe-ness’ to failover

Yes

Risk of data loss with

forced HADR takeover

Any other state

Remote catchuppending

state

Peer state

Did the primary server fail while the databases were still in peer (by comparing timestamps from automation software (AS)

logs and db2diag.log on standby)?Get timestamp from previous step’s message.Get timestamp from automation log for when the primary server failed (or whatever event

caused the failover)

On STANDBY, check diaglog. Was db last in Peer

before changing to remote catchup pending (“HADR state set to S-

RemoteCatchupPending(was S-Peer)”)?

Yes

No

YesNo

Compare db2diag timestamp (DT) and AS log timestamp (AT). Is the

DT within a “safe” (duration of failure detection) timespan?

YesNo

In <outfile>, is hdrPrimaryLastState

= 0x10 (Peer)

Note that the values returned from the db2pdlog command for hdrLastPrimaryState are internal values. The possible values are:0x01 – Booting0x02 – Waiting to enter remote catchup0x04 – Remote catchup0x08 – Nearly peer0x10 – Peer

35

35

Best Practices – HADR operations• Failed primary reintegration

• On old PRIMARY (now new STANDBY): db2 start hadr on db $dbname as standby

• HADR and reintegration starts asynchronously so success must be monitored

• If reintegration is not possible, details will be logged in the db2diag.log as to why it failed to synchronize

36

36

Best Practices – HADR operations• Rolling fix pack upgrade

1. On site B, upgrade the database• Deactivate standby database and stop DB2• Upgrade software, run db2iupdt• Restart DB2 and activate standby database• Wait for HADR pair to regain Peer state

2. Switch primary/standby role• Issue “TAKEOVER HADR ON DB $DBNAME” on site B

3. On site A, upgrade the database• Deactivate new standby database and stop DB2• Connect to site B, run BIND, db2updv8, etc on the new primary to complete the

upgrade on site B• Upgrade software, run db2iupt• Restart DB2 and activate new standby database• Wait for HADR pair to regain Peer state which completes the upgrade on site A

4. Optionally, switch primary/standby role

37

37

Best Practices – Alerts• HADR state

• Monitor hadr_state element for ‘Peer’ on the primary• Not logged objects exist

• On the primary, check SYSCAT.COLUMNS for any LOGGED=‘N’select tabschema, tabname, colname from syscat.columns

where logged=’N’

• Not logged initially table operations• Scan the administration notification log on the primary for

ADM5530W which indicates that a table has committed activity that was not logged (either created NOT LOGGED INITIALLY or altered to activate NOT LOGGED)

38

38

Best practices - Alerts• Non-replicated load

• Monitor the diagnostic log on the standby• Load copy cannot be accessed on the standby:

FUNCTION: DB2 UDB, database utilities, sqluMCCheckDevType, probe:10MESSAGE : Media controller -- invalid device path:

/db2fs/db2hda/HDA.4.db2hda.NODE0000.CATN0000.20060406045810.001

FUNCTION: DB2 UDB, recovery manager, sqlpRecDbRedo, probe:2129MESSAGE : Tablespace 7 in restore pending state.

• Non-recoverable loadFUNCTION: DB2 UDB, High Availability Disaster Recovery,

hdrTakeoverCheckTablespaces, probe:51010 MESSAGE : Warning: Tablespace is not in NORMAL state following

takeover TSID:DATA #1 : Hexdump, 8 bytes0x0000007FBFFBA2B0 : 0700 0000 0000

000 ........

39

39

Best practices - Alerts• Non-normal table space state

• If any load operation has an inaccessible load copy• If there’s a disk error on the standby• On the standby, check the State column in the

Tablespace Statistics report generated from db2pd -db <dbname> -tablespaces

• Remember, some non-zero states are okay

40

40

Trouble-shooting• db2diag.log is the primary tool for trouble-shooting• Example: Identifying which role a system should be

• Worst-case scenario where maybe automation failed to takeover and you have two standby databases .. or maybe two primary databases

db2diag -gi "funcname:=hdr" -fmt "%{tsmonth}-%{tsday}-%{tshour}.%{tsmin}.%{tssec} %{process} %{changeevent}\t %{message}"

04-11-13.09.57 db2hadrp (HDA) 0 HADR state set to P-Peer (was P-NearlyPeer)04-11-13.25.08 db2hadrp (HDA) 0 Info: Primary has started non-forced takeover request.04-11-13.25.08 db2hadrp (HDA) 0 Info: Primary is switching to standby role....04-11-13.59.16 db2hadrs (HDA) 0 HADR state set to S-Peer (was S-NearlyPeer)04-11-14.16.36 db2hadrs (HDA) 0 Info: Standby has initiated a takeover.04-11-14.16.36 db2hadrs (HDA) 0 Info: Standby switching roles to primary.04-11-14.16.40 db2hadrp (HDA) 0 HADR state set to P-Peer (was S-Peer)04-11-14.16.41 db2hadrp (HDA) 0 Info: Standby has completed takeover (now primary)....04-27-15.11.26 db2agent (HDA) 0 Info: HADR Startup has begun.04-27-15.11.26 db2hadrp (HDA) 0 Info: Primary Started.04-27-15.24.39 db2hadrs (HDA) 0 HADR state set to None (was None)04-27-15.24.39 db2hadrs (HDA) 0 HADR state set to S-Boot (was None)

On Unix, when an HADR database is operating as a primary database, the process name in diagnostic log entries is “db2hadrp”. When operating as a standby database, the process name is “db2hadrs”. On Windows, the state description for each HADR state set message uses “P-“ to indicate primary and “S-“ to indicate standby.

41

41

Trouble-shooting

• Congestion• Can result in delay of transaction commits on

the primary if in SYNC or NEARSYNC• Due to competition for network or log receiving

blocked at standby• If due to peaks in workload and see an increase

in log gap, consider increasing DB2_HADR_BUF_SIZE to 4x or 8x LOGBUFSZ (default setting is 2xLOGBUFSZ)

42

42

Administration considerations• Restore:

• Table space level restore is not allowed• Database level restore can reset HADR role to STANDARD

• Table and index reorganization:• Online reorganization logs at fine granularity so replicated frequently;

cannot restart an online reorg started on site A after failover to site B• Offline reorganization can have a significant impact on HADR as a

single log record could encompass many updates .. worst case, the standby repeats the offline reorg after the primary completes

• Not logged initially (NLI) operations• NLI operations are not replicated; may not be an issue if it’s for

temporary or staging operations, but should monitor this scenario

Re-clustering REORGOperations logged per hundreds or thousands of affected rowsSecondary may fall behind in this case.

Non re-clustering REORGEntire operation is a single record generated after REORG on PrimaryGreatest impact on secondary to keep up with Primary

43

43

Administration considerations• Load operations:

• Must be run with COPY YES option to be replicated• All other options result in standby becoming inconsistent with primary

and no longer valid for failover• User DB2_LOAD_COPY_NO_OVERRIDE registry variable to avoid non-

copied loads

• Not-logged large objects:• LOBs greater than 1GB or those defined as NOT LOGGED will not be

replicated to the standby• If these objects are required in DB, then will need regular refreshes of

standby from primary backup to keep in sync

• Configuration updates and file system changes:• Recommended to keep configuration synchronized between primary and

standby by applying updates to both

When loading data into tables, a LOAD ... COPY YES ... is run on the primary and the data is replicated on the standby if the standby can access the copy through the path or device specified in the load command.

If the load copy cannot be accessed on the standby, the table space in which the table is stored is marked bad on the standby and the standby will skip future log records regarding this table space. In this situation, the standby database has to be rebuilt from a backup image taken after the LOAD operation was issued on the primary.

A LOAD ... NONRECOVERABLE is run on the primary as usual. On the standby, the table will be marked bad and the standby will skip future log records regarding this table. If the table is not going to be dropped as part of the load processing, the table can be recovered by issuing a LOAD COPY YES REPLACE for the table on the primary, or the standby can be rebuilt from a backup image taken after the LOAD ... NONRECOVERABLE was issued.

If a LOAD ... COPY NO is run on the primary, it is converted to a LOAD NONRECOVERABLE by default. However, this conversion can be overridden to convert to a COPY YES operation by setting DB2_LOAD_COPY_NO_OVERRIDE=”COPY YES TO <shared directory path>”. When the registry variable is used, all LOAD … COPY NO operations will be converted to and behave according to the LOAD … COPY YES behavior outlined above.

44

44

Summary• HADR is an easy-to-use technology• Early planning will lead to reaping rewards with a

straight-forward implementation• Pack of procedures will keep things simple and

enable smooth operation of HADR

45

45

Jessica Escott-RockwoodIBM Toronto Lab

jessicae@ca.ibm.com

Session D09

DB2 HADR – Case Study of Implementation