DB2 HADR – Case Study of Implementation
Transcript of DB2 HADR – Case Study of Implementation
1
06 November 2007 • 14:00 – 15:00
Platform: Linux, Unix, Windows
DB2 HADR – Case Study of Implementation
Jessica RockwoodIBM Toronto Lab
Session: D06
An in-depth look at using DB2 High Availability Disaster Recovery (HADR) in a large enterprise. This presentation will review the HADR technology and walk through the end-to-end process of implementing HADR and automation technology to manage failovers, including: architecture, setup and configuration, testing, and monitoring.
2
2
Session Objectives• A thorough understanding of the capabilities of High
Availability Disaster Recovery (HADR) • Impart my lessons learned in implementing HADR solutions
• How to build a plan for deploying HADR successfully in an enterprise environment
• An understanding of automation software and how it interacts with HADR
• How to build a test plan to ensure a complete availability solution before production deployment
• How to monitor and administer HADR databases (including “My Pack of Procedures”)
At the end of this presentation, the audience should be able to implement HADR in their environment, understanding how to build the project plan, assess risks and benefits, implement and thoroughly test the solution, and build monitoring capabilities.
3
3
Goals of HADR• Ultra-fast failover• Easy administration• Handling of site failures • Negligible impact on performance• Configurable degree of consistency • Protection against errant transactions• Software upgrades without interruption• Very easy integration with HA-software • Eventually, no need for HA-software at all• Transparent failover and failback for applications
(combined with “client re-route”)
Limitations/restrictions:No CONNECT access to standby databaseNot supported with Data Partitioning Feature (DPF), raw I/O for transaction logs, or redirected restorePrimary and standby databases must have the same name, the same operating system and DB2 levels and the same bit-ness (32-bit vs 64-bit)Rolling upgrades refer to Fix Pack upgrades onlyNon-logged operations are not replicated:
Updates to DB/DBM cfg and registryLOBs > 1GB or LOBs marked as NOT LOGGEDNOT LOGGED INITIALLY operations
Log archiving and self-tuning memory manager (STMM) can only be done on the primary database
4
4
Basic principles of HADR• Two active machines
• Primary• Processes transactions• Ships log entries to the other machine
• Standby (or Secondary)• Cloned from the primary• Receives and stores log entries from the primary• Re-applies the transaction
• If the primary fails, the standby can take over the transactional workload• The standby becomes the new primary
• If the failed machine becomes available again, it can try to be resynchronized• The old primary becomes the new standby
5
5
Scope of activity• HADR replication takes place at the database level
Database E Database E
HADR HADRTCP/IP
Primary Standby
Database A
HADR HADRTCP/IP
Standby Primary
Database A
Database B
Database C
Database D
6
6
HADR implementation details
LogsOld
Logs
Log Writer Log Reader
Tables
IndexesLogs
Old
Logs
Log Reader
Tables
Indexes
TCPIP
DB2 Engine DB2 EnginePRIMARY SERVER STANDBY SERVER
Log Writer
HADR HADR Shredder Replay Master
Redo SlavesRedo SlavesRedo SlavesReplay Slaves
Log Pages
Log Pages
Log Pages
Log RecordsLog Pages
Log
Records
Client ReroutePrimary Connection
7
7
HADR State Flow
HADR/DB2 Startup
Local catch-up
Remote catch-up Pending
Remote catch-up
Peer State
Connect
Con
nect
ion
lost
These states are as seen from the standby; on the primary only remote catch-up and peer state are valid states. All others are represented as ‘Disconnected”from the perspective of the primary.
8
8
HADR synchronization modes
Logs
Log writer
Logs
TCP/IP
Commit Succeeded
Synchronous
Near
- Syn
chro
nousAsynchronous
receive()send()
HADR HADR
Log writer
Near-Sync sends back the acknowledgement once data is received in memory.
9
9
Planning• Three components to examine:
• Client• Network• Server
• Goal is to identify the expected behavior in the case of a component failure
TCPIP
TSA detects failure of Site A within 5 seconds and invokes forced HADR takeover on Site B
e.g. AIX server failure
Site BSite A
Expected resultFailure description
Failure location
Location operating as primary
10
10
Planning stage - Client• Type of client?
• Uses DB2 Client or DB2 application driver (Java, CLI)?
• Type of workload?• Availability requirements?• Hours of operation?
TCPIP
Type of client:-When using an application driver, need additional attention to how to handle client reroute. When you specify the alternate client on the server, the Java or CLI client will pick up the alternate server information on first connect. If you need to support the clients being able to reroute without having successfully connected to the “primary” site, then need to harden the information in the client. For CLI, it’s an option on the definition of the data source. For JCC driver, need to implement the Type of workload:-Need to understand whether workload is constant or will have spikes-With spikes of utilization, it may affect the ability of the standby to keep up and could influence the choice of SYNC mode. With spikes in workload, likely you should consider NEARSYNC (or ASYNC if only using the solution for disaster recovery and not HA)Availability requirements:-What’s the priority on the system? Is it availability or ensuring no data loss…each have a tradeoff and can influence the choice between sync modes as well as options like HADR_TIMEOUTHours of operations:-Need to understand when and if there are down-times on the system-If there’s a period of little to no activity, then could do off-line maintenance activities or those with high impact to the standby (like offline reorg) during these periods and even consider deactivating the standby, doing maintenance,
d th f hi th t db ith b k i f th i
11
11
Planning stage - Network• What is the network topology?
• Same network used between clients & server as between HADR primary & standby?
• Bandwidth?• Must have bandwidth greater than the
database log generation rate• Redundancy of network adapters?
• Single network adapter is a possible point of failure
• Virtual IP?
TCPIP
Network topology:-It is recommended to isolate HADR communication on a network separate from the rest of the application communications. This ensures that spikes in application workload or other activity in the environment will not affect the HADR communication.-If the network is going to be shared, then may need to ensure optimal tuning of the network to handle all the data flow-Note that HADR requests tcp_nodelay for its connection for optimal performance; if ignored by the OS, it can negatively affect HADR performance-Increasing LOGBUFSZ can increase the amount of log flushed to the standby in each request… but more likely to need to increase DB2_HADR_BUF_SZ registry variable (to more than the default of 2xLOGBUFSZ) in order to handle peaks of log data flushed to standbyVirtual IP:- Using a virtual IP can simplify management of the HADR pair in cluster management software… in the event of a failure, the IP ownership moves from primary to standby and so no clients will be able to connect to the primary box (which can lead to split-brain if the primary was not shut down properly)- With a virtual IP, can still use client reroute by defining the same IP for the alternate client server as for the initial catalog entry
12
12
Planning stage - Server• DB2 levels (recommend minimum V8.1 FP12 or
V9 FP2)• Storage
• robustness of database storage?• location of archived logs?
• Server• operating system? release levels?• automation software?
TCPIP
Storage:-Storage is a possible point of failure on both the primary and the standby…disk mirroring or some other sort of HA strategy for the disks can avoid requiring a failover due to disk failure; there is no interruption of service if a failover is not required, so always want to look to repair in place when possible-If archived logs are on external storage or something like TSM, then both primary and standby need to be able to access the location; also, delays in retrieving logs can result in significant delays in catch-up by the standby so you may want to retrieve archived logs to the standby before starting HADR to avoid that network overhead (but bear in mind that you need to manage the deletion of those retrieved logs)Server:-Look to run the operating system levels prescribed by IBM and ensure that all OS patches are applied in a timely fashion-Tivoli System Automation for Multiplatforms is available for Linux and AIX. HACMP is also available for AIX.b
13
13
Case A: Bright Yellow Truck Company
•p5-575 AIX 5.2 64-bit ML6 with DB2 Viper 2•Tivoli Storage Manager (TSM) 5.3 is used to archive logs•HACMP 5.3 is the incumbent automation software
Server
Single public GB ethernet for all network traffic, heartbeat RSA connection between primary and disaster site
Network
24x7 proprietary application using DB2 clientClient
14
14
Bright Yellow Truck – Planning Analysis
… and so on, including network failure, DB2 failure, filesystem damage/corruption, TSM server failure
Local disk is mirrored, no impact to DB2; storage team to replace failed disk
Disk failureSite ASite A
No automatic action; need to repair server on site B, restart, and regain peer state
AIX server failure
Site BSite A
HACMP invokes rc.hadr.start script on Site B to takeover on standby database; clients reconnect to Site B primary through client reroute
AIX server failure
Site ASite A
Expected ResultFailure description
Failure location
Primary location
15
15
Case B: Little Blue Box Jewelers
•x3650 Linux SLES9 SP3 with DB2 9•Logs are archived to shared disk•Tivoli System Automation for Multiplatforms 2.1 (TSA) for automation
Server
Public GB ethernet for client-server traffic, private GB ethernet for HADR connection between primary and disaster site
Network
Java application running on WebSphere Application Server (WAS) using the JCC Type-4 driverApplication must be operational 8am-5pm weekdays and Saturday
Client
16
16
Decide on HADR configuration• Two decisions to make about HADR connection:
• hadr_timeout • Time (in seconds) that HADR waits before considering a
communication attempt as failed• Heartbeat interval will be ¼ of hadr_timeout or 30 seconds,
whichever is shorter• hadr_syncmode
• SYNC, NEARSYNC, or ASYNC• For disaster recovery, most choose between SYNC and
NEARSYNC -> performance testing is valuable to quantify the difference between the two
Note that, for maximal availability, as soon as the connection is closed (because the other end closed the connection, a network error is detected, or timeout is reached), the primary drops outof peer state to avoid blocking transactions.A recommended timeout value depends on the capacity of the data server and of the HADR network connection. If either are busy, then it can result in a delay in response between the HADR pair and that will start the countdown to the HADR_TIMEOUT value.In most enterprise implementations, an HADR_TIMEOUT between 30 and 60 seconds provides the best balance of a short timeout without risking falling out of peer due to too short a timeout. An HADR_TIMEOUT less than 30 seconds should be thoroughly tested to ensure the primary does not drop out of peer state as part of regular processing.
17
17
Big Yellow Truck Decisions• After testing SYNC and NEARSYNC, no significant performance
impact was detected between the two so set hadr_syncmode=SYNC
• Highest priority at Big Yellow Truck was data integrity – no transactions could be lost without a ripple effect through the organization• Risk even in SYNC mode; if the TCP/IP socket closed on the
primary, the primary could drop out of Peer mode immediately and if the primary failed and a forced takeover occurred before the standby reconnected and regained peer state there was the possibility of data lossChose to implement new Viper 2 feature: HADR peer window
18
18
HADR Peer Window• HADR_PEER_WINDOW can be used to cause the HADR pair
(primary and standby databases) to behave as though in peer state even if the primary loses connection with the standby database• Once connection to standby is lost, primary waits the number
of seconds specified by HADR_PEER_WINDOW before dropping out of Disconnected Peer state and continuing to process transactions
• Peer window end (meaning the time at which the primary is no longer guaranteed to be in peer state) is sent to the standby through the heartbeat - timestamp can be checked by automation software to determine whether or not to do forced takeover
• If standby does not reconnect with primary by the end of the peer window, the primary drops from Disconnected Peer state to Remote Catchup Pending
Note: important to synchronize clocks between primary and standby for effective usage of End of peer window
19
19
Setting HADR_PEER_WINDOW• DB2_HADR_PEER_WINDOW should be set to a value that fits the following
equation:
DB2_HADR_PEER_WINDOW >= response_time + fudge_factor + heartbeat_interval
response_time = estimated time for automation software to detect failure and invoke HADR takeover
fudge_factor = 5 sec, small delta time delay because detection/response is not instantaneous
heartbeat_interval = MIN(HADR_TIMEOUT/4, DB2_HADR_PEER_WINDOW/4,30 sec)
• Big Yellow Trucks determined their response time as a minimum of 45secs• HACMP detection of failure: 5 seconds• HACMP processing time until HADR takeover invoked: up to 40
seconds
20
20
HADR State FlowHADR/DB2 Startup
Local catch-up
Remote catch-up Pending
Remote catch-up
Peer State
Connect
Con
nect
ion
re-e
stab
lishe
d
Con
nect
ion
lost
Disconnected Peer State
Connection lost
Peer window expires
21
21
Implementing the HADR solution• Prepare and clone the primary database and
configure for HADR• Create the standby database from the clone• Configure the standby database for HADR• Configure client reroute • Start-up HADR• Set-up automation process
22
22
Prepare and clone the primary DB• Ensure archive logging is enabled on the primary database
Archive log location must be accessible by both primary and standby database
• Sample database configuration for Case A: LOGFILSIZ 16380LOGSECOND 12LOGARCHMETH1 TSM:DB2
• Sample database configuration for Case B:LOGFILSIZ 8192LOGSECOND 2LOGARCHMETH1 DISK:/shared/db2inst1/logarchives
• Take backup or split mirror copy of primary database and move the backup image to the standby location
23
23
Code snippet - Configure primary for HADRexport $dbname=TRUCKERSexport $pri=yellow1.bigtruck.comexport $sec=yellow2.bigtruck.comexport $svc=7000
db2 connect to $dbnamedb2 update db cfg for $dbname using LOGFILSIZ 16380db2 update db cfg for $dbname using LOGSECOND 12db2 update db cfg for $dbname using LOGARCHMETH1 TSM:DB2db2 update db cfg for $dbname using INDEXREC RESTARTdb2 update db cfg for $dbname using LOGINDEXBUILD ONdb2 update db cfg for $dbname using HADR_TIMEOUT 30db2 update db cfg for $dbname using HADR_PEER_WINDOW 90db2 update db cfg for $dbname using HADR_LOCAL_HOST $pridb2 update db cfg for $dbname using HADR_LOCAL_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_HOST $secdb2 update db cfg for $dbname using HADR_REMOTE_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_INST sitebdb2 update db cfg for $dbname using HADR_SYNCMODE SYNCdb2 deactivate db $dbnamedb2 activate db $dbname
Configuration for Case B would be:export $dbname=DIAMONDexport $pri=bluebox.jeweler.comexport $sec=bluebox2.jeweler.comexport $svc=7000
db2 connect to $dbnamedb2 update db cfg for $dbname using LOGFILSIZ 8192db2 update db cfg for $dbname using LOGSECOND 2db2 update db cfg for $dbname using LOGARCHMETH1 DISK:/shared/mydb/archivelogsdb2 update db cfg for $dbname using INDEXREC RESTARTdb2 update db cfg for $dbname using LOGINDEXBUILD ONdb2 update db cfg for $dbname using HADR_TIMEOUT 30db2 update db cfg for $dbname using HADR_LOCAL_HOST $pridb2 update db cfg for $dbname using HADR_LOCAL_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_HOST $secdb2 update db cfg for $dbname using HADR_REMOTE_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_INST sitebdb2 update db cfg for $dbname using HADR_SYNCMODE SYNCdb2 deactivate db $dbnamedb2 activate db $dbname
24
24
Create the standby DB and configure HADR• Restore the backup image or db2inidb the split mirror copy
database is left in roll forward pending state
db2 update db cfg for $dbname using LOGFILSIZ 16380db2 update db cfg for $dbname using LOGSECOND 12db2 update db cfg for $dbname using LOGARCHMETH1 TSM:DB2db2 update db cfg for $dbname using INDEXREC RESTARTdb2 update db cfg for $dbname using LOGINDEXBUILD ONdb2 update db cfg for $dbname using HADR_TIMEOUT 30db2 update db cfg for $dbname using HADR_PEER_WINDOW 90db2 update db cfg for $dbname using HADR_LOCAL_HOST $secdb2 update db cfg for $dbname using HADR_LOCAL_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_HOST $pridb2 update db cfg for $dbname using HADR_REMOTE_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_INST siteadb2 update db cfg for $dbname using HADR_SYNCMODE SYNCdb2 deactivate db $dbname
Strict symmetry of table space and container configuration is required on the standby. Name, path, size all must match. Relative container paths are allowed, and the full path may differ in this case.
If strict symmetry does not exist, HADR may fail to replicate a container operation on the standby. In such a case replication of the affected table space stops and the table space will be left in "roll forward in progress"
25
25
Configure automatic client reroute• On both the primary and standby database servers issue:
db2 update alternate server for db $dbname using hostname <host> port <port_num>
• Note: Alternate server is flowed to the client and cached on first connect; without a connection, the client will not know where to re-route• To protect against this on a DB2 client, the same
command can be run on the client to ‘harden’ the alternate server
• On a JCC-Type 4 client, use the clientRerouteServerListJNDIName DataSource property to persist alternate server information across JVMs and to ‘harden’ the alternate server
26
26
Start-up HADRSTANDBY
db2 start hadr on db $dbname as standby
db2 get snapshot for db on $dbname |grep –A 15 “HADR Status”
ORdb2 select * from
sysibmadm.snaphadr
PRIMARY
db2 start hadr on db $dbname as primary
db2 get snapshot for db on $dbname |grep –A 15 “HADR Status”
Start local catch-up
Validate connection
between primary & standby and start remote
catch-up
Upon starting the standby, the Local Catchup phase can begin:-Replay locally available log files (if any)-Replay log files that may be retrieved at the primary and shipped to the standby
Once the primary is started (which is an asynchronous operation), remote catchup can begin:-Replay log pages from the primary's archived logs-Replay log pages from the primary's active log to the standby until the standby is caught up to the tail of the log.-Replay log pages from the primary's in-memory log buffer to the standby whenever the primary flushes those pages to disk
At the conclusion of remote catchup, the standby is now in Peer state.
27
27
Start-up on standby – Expected ResultsDB20000I The START HADR ON DATABASE command completed successfully.
HADR StatusRole = StandbyState = Remote catchup pendingSynchronization mode = SyncConnection status = Disconnected, 03/22/2007 09:50:12.423522Peer window end = Null (0)Peer window (seconds) = 65Heartbeats missed = 0Local host = bbird007Local service = HADR_test2Remote host = bbird006Remote service = HADR_test1Remote instance = db2hdatimeout(seconds) = 30Primary log position(file, page, LSN) = S0000115.LOG, 0, 0000000748858000Standby log position(file, page, LSN) = S0000114.LOG, 1, 000000074485D0BCLog gap running average(bytes) = 67088191
28
28
Start-up on primary – Expected ResultsDB20000I The START HADR ON DATABASE command completed successfully.
HADR StatusRole = PrimaryState = PeerSynchronization mode = SyncConnection status = Connected, 03/22/2007 09:50:50.622363Peer window end = 03/22/2007 09:51:56.000000 (1174571516)Peer window (seconds) = 65Heartbeats missed = 0Local host = bbird006Local service = HADR_test1Remote host = bbird007Remote service = HADR_test2Remote instance = db2hdatimeout(seconds) = 30Primary log position(file, page, LSN) = S0000115.LOG, 0, 0000000748858000Standby log position(file, page, LSN) = S0000114.LOG, 1, 000000074485D0BCLog gap running average(bytes) = 67088191
29
29
Configure automation software• For TSA, use DB2-provided scripts hadr_start.ksh,
hadr_stop.ksh, and hadr_monitor.ksh• Reference whitepaper: “Automating DB2 HADR Failover
on Linux using Tivoli System Automation on Multiplatforms”(ftp://ftp.software.ibm.com/software/data/pubs/papers/hadr_tsa.pdf)
• For HACMP, use DB2-provided scripts rc.hadr.start, rc.hadr.stop, rc.hadr.monitor• Reference whitepaper: “Automating IBM DB2 UDB
HADR with HACMP”(ftp://ftp.software.ibm.com/software/data/pubs/papers/hadr-hacmp.pdf)
30
30
Next steps• Yippee! HADR is now up and running…
… but now you need to test it and develop procedures for the HADRenvironment
• Testing matrix should cover all the failure scenarios outlined during the planning phase to validate expected results
• Measurements to capture:• Time to detect failure, time for failover, and time until new primary
is open for business (any open transactions at time of failure rolled back and database ready to process new transactions)
• Performance of workload• Server utilization statistics• Consistency check between primary and standby data
31
31
My Pack of Procedures• Database start-up
• STANDBY: db2start; db2 activate db $dbname• PRIMARY: db2start; db2 activate db $dbname• Verify success with snapshot on PRIMARY
• If standby not available, PRIMARY: db2start; db2 start hadr on db $dbname as primary by force
• Database shut-down• PRIMARY: db2 deactivate db $dbname; db2stop force• STANDBY: db2 deactivate db $dbname; db2stop
Recommended aliases:alias gethadrhost=’db2 get db cfg for $dbname |grep HADR_LOCAL_HOST’alias getdbrole=’db2 get db cfg for $dbname |grep “HADR database role”’alias hadrstate=’snaphadr | grep State’alias snaphadr=’db2 get snapshot for db on $dbname |grep –A 15 “HADR Status”’
32
32
My Pack of Procedures• Database role reversal
• Verify peer state on primary with snapshot• STANDBY: db2 takeover hadr on db $dbname
• Verify success with snapshot on new PRIMARY
Primary Standby
33
33
My Pack of Procedures• Manual database failover
• With peer window configuration, issue on STANDBY: db2 takeover hadr on db $dbname by force peer window only
• Without, issue on STANDBY: db2 takeover hadr on db $dbname
• If this fails, need to evaluate ‘safe-ness’ of doing the forced takeover – risk of split-brain or losing transactions
34
34
Is the primary server (and its disk) available?
YES, OK to run a planned
takeover
On STANDBY, check the HADR state of the database, use alias:
hadrstate
On PRIMARY, run db2pdlog –l <outfile> from the database directory (where SQLOGCTL.LFH exists)
No
UNPLANNED EVENT: Task flow for determining ‘safe-ness’ to failover
Yes
Risk of data loss with
forced HADR takeover
Any other state
Remote catchuppending
state
Peer state
Did the primary server fail while the databases were still in peer (by comparing timestamps from automation software (AS)
logs and db2diag.log on standby)?Get timestamp from previous step’s message.Get timestamp from automation log for when the primary server failed (or whatever event
caused the failover)
On STANDBY, check diaglog. Was db last in Peer
before changing to remote catchup pending (“HADR state set to S-
RemoteCatchupPending(was S-Peer)”)?
Yes
No
YesNo
Compare db2diag timestamp (DT) and AS log timestamp (AT). Is the
DT within a “safe” (duration of failure detection) timespan?
YesNo
In <outfile>, is hdrPrimaryLastState
= 0x10 (Peer)
Note that the values returned from the db2pdlog command for hdrLastPrimaryState are internal values. The possible values are:0x01 – Booting0x02 – Waiting to enter remote catchup0x04 – Remote catchup0x08 – Nearly peer0x10 – Peer
35
35
My Pack of Procedures• Failed primary reintegration
• On old PRIMARY (now new STANDBY): db2 start hadr on db $dbname as standby
• HADR and reintegration starts asynchronously so success must be monitored
• If reintegration is not possible, details will be logged in the db2diag.log as to why it failed to synchronize
36
36
My Pack of Procedures• Rolling fix pack upgrade
1. On site B, upgrade the database• Deactivate standby database and stop DB2• Upgrade software, run db2iupdt• Restart DB2 and activate standby database• Wait for HADR pair to regain Peer state
2. Switch primary/standby role• Issue “TAKEOVER HADR ON DB $DBNAME” on site B
3. On site A, upgrade the database• Deactivate new standby database and stop DB2• Connect to site B, run BIND, db2updv8, etc on the new primary to complete the
upgrade on site B• Upgrade software, run db2iupt• Restart DB2 and activate new standby database• Wait for HADR pair to regain Peer state which completes the upgrade on site A
4. Optionally, switch primary/standby role
37
37
My pack of procedures - Alerts• HADR state
• Monitor hadr_state element for ‘Peer’ on the primary• Not logged objects exist
• On the primary, check SYSCAT.COLUMNS for any LOGGED=‘N’select tabschema, tabname, colname from syscat.columns
where logged=’N’
• Not logged initially table operations• Scan the administration notification log on the primary for
ADM5530W which indicates that a table has committed activity that was not logged (either created NOT LOGGED INITIALLY or altered to activate NOT LOGGED)
38
38
My pack of procedures - Alerts• Non-replicated load
• Monitor the diagnostic log on the standby• Load copy cannot be accessed on the standby:
FUNCTION: DB2 UDB, database utilities, sqluMCCheckDevType, probe:10MESSAGE : Media controller -- invalid device path:
/db2fs/db2hda/HDA.4.db2hda.NODE0000.CATN0000.20060406045810.001
FUNCTION: DB2 UDB, recovery manager, sqlpRecDbRedo, probe:2129MESSAGE : Tablespace 7 in restore pending state.
• Non-recoverable loadFUNCTION: DB2 UDB, High Availability Disaster Recovery,
hdrTakeoverCheckTablespaces, probe:51010 MESSAGE : Warning: Tablespace is not in NORMAL state following takeover TSID:DATA #1 : Hexdump, 8 bytes0x0000007FBFFBA2B0 : 0700 0000 0000 000 ........
39
39
My pack of procedures - Alerts• Non-normal table space state
• If any load operation has an inaccessible load copy• If there’s a disk error on the standby• On the standby, check the State column in the
Tablespace Statistics report generated from db2pd -db <dbname> -tablespaces
• Remember, some non-zero states are okay
40
40
My pack of procedures• Identifying which role a system should be
• Worst-case scenario where maybe automation failed to takeover and you have two standby databases .. or maybe two primary databases
db2diag -gi "funcname:=hdr" -fmt "%{tsmonth}-%{tsday}-%{tshour}.%{tsmin}.%{tssec} %{process} %{changeevent}\t %{message}"
04-11-13.09.57 db2hadrp (HDA) 0 HADR state set to P-Peer (was P-NearlyPeer)04-11-13.25.08 db2hadrp (HDA) 0 Info: Primary has started non-forced takeover request.04-11-13.25.08 db2hadrp (HDA) 0 Info: Primary is switching to standby role....04-11-13.59.16 db2hadrs (HDA) 0 HADR state set to S-Peer (was S-NearlyPeer)04-11-14.16.36 db2hadrs (HDA) 0 Info: Standby has initiated a takeover.04-11-14.16.36 db2hadrs (HDA) 0 Info: Standby switching roles to primary.04-11-14.16.40 db2hadrp (HDA) 0 HADR state set to P-Peer (was S-Peer)04-11-14.16.41 db2hadrp (HDA) 0 Info: Standby has completed takeover (now primary)....04-27-15.11.26 db2agent (HDA) 0 Info: HADR Startup has begun.04-27-15.11.26 db2hadrp (HDA) 0 Info: Primary Started.04-27-15.24.39 db2hadrs (HDA) 0 HADR state set to None (was None)04-27-15.24.39 db2hadrs (HDA) 0 HADR state set to S-Boot (was None)
On Unix, when an HADR database is operating as a primary database, the process name in diagnostic log entries is “db2hadrp”. When operating as a standby database, the process name is “db2hadrs”. On Windows, the state description for each HADR state set message uses “P-“to indicate primary and “S-“ to indicate standby.
41
41
Trouble-shooting
• Congestion• Can result in delay of transaction commits on
the primary if in SYNC or NEARSYNC• Due to competition for network or log receiving
blocked at standby• If due to peaks in workload and see an increase
in log gap, consider increasing DB2_HADR_BUF_SIZE to 4x or 8x LOGBUFSZ (default setting is 2xLOGBUFSZ)
42
42
Administration considerations• Restore:
• Table space level restore is not allowed• Database level restore can reset HADR role to STANDARD
• Table and index reorganization:• Online reorganization logs at fine granularity so replicated
frequently; cannot restart an online reorg started on site A after failover to site B
• Offline reorganization can have a significant impact on HADR as a single log record could encompass many updates .. the standby repeats the offline reorg after the primary completes
• Not logged initially (NLI) operations• NLI operations are not replicated; may not be an issue if it’s for
temporary or staging operations, but should monitor this scenario
43
43
Administration considerations• Load operations:
• Must be run with COPY YES option to be replicated• All other options result in standby becoming inconsistent with
primary and no longer valid for failover• User DB2_LOAD_COPY_NO_OVERRIDE registry variable to
avoid non-copied loads• Not-logged large objects:
• LOBs greater than 1GB or those defined as NOT LOGGED will not be replicated to the standby
• If these objects are required in DB, then will need regular refreshes of standby from primary backup to keep in sync
• Configuration updates and file system changes:• Recommended to keep configuration synchronized between
primary and standby by applying updates to both
When loading data into tables, a LOAD ... COPY YES ... is run on the primary and the data is replicated on the standby if the standby can access the copy through the path or device specified in the load command.
If the load copy cannot be accessed on the standby, the table space in which the table is stored is marked bad on the standby and the standby will skip future log records regarding this table space. In this situation, the standby database has to be rebuilt from a backup image taken after the LOAD operation was issued on the primary.
A LOAD ... NONRECOVERABLE is run on the primary as usual. On the standby, the table will be marked bad and the standby will skip future log records regarding this table. If the table is not going to be dropped as part of the load processing, the table can be recovered by issuing a LOAD COPY YES REPLACE for the table on the primary, or the standby can be rebuilt from a backup image taken after the LOAD ... NONRECOVERABLE was issued.
If a LOAD ... COPY NO is run on the primary, it is converted to a LOAD NONRECOVERABLE by default. However, this conversion can be overridden to convert to a COPY YES operation by setting DB2_LOAD_COPY_NO_OVERRIDE=”COPY YES TO <shared directory path>”. When the registry variable is used, all LOAD … COPY NO operations will be converted to and behave according to the LOAD … COPY YES behavior outlined above.
44
44
Summary• HADR is an easy-to-use technology• Early planning will lead to reaping rewards during
implementation• Pack of procedures will keep things simple and
enable smooth operation of HADR
45
45
Jessica RockwoodIBM Toronto Lab
Session D06DB2 HADR – Case Study of Implementation