Configuring High Availability for the PMG DB...Configuring High Availability for the PMG DB...
Transcript of Configuring High Availability for the PMG DB...Configuring High Availability for the PMG DB...
Configuring High Availability for the PMG DB
This chapter describes the process of configuring high availability for the PMG DB. It provides theprerequisites and procedures required to configure and test the hot standby and cold standby for Cisco RMS.It also includes procedures required to back up and restore standby and primary configurations, troubleshootData Guard on PMG DB, and upgrade Cisco RMS.
The Oracle Data Guard tool is used to ensure high availability and disaster recovery for PMG DB in CiscoRMS. This tool is used to configure one primary database and one or more standby databases. These databasesin a Data Guard configuration are connected by Oracle Net andmay be dispersed geographically. The primaryand standby databases can be managed using the SQL CLIs or the Data Guard Broker interfaces, includinga CLI and a GUI that is integrated in the Oracle Enterprise Manager.
The Oracle Enterprise Manager GUI or the Data Guard (DGMGRL) CLI is used to enable fast-start failoverto fail over automatically when the primary database becomes unavailable. When a fast-start failover is
High Availability for Cisco RAN Management Systems 1
enabled, the Data Guard Broker determines if a failover is necessary and initiates the failover to the specifiedtarget standby database automatically, without a need for DBA intervention.
A failover occurs when the primary database is unavailable. Failover is performed only in the event of afailure of the primary database, and the failover results in a transition of a standby database to the primaryrole.
A switchover is a role reversal between the primary database and one of its standby databases. This is typicallydone for planned maintenance of the primary system. During a switchover, the primary database transitionsto a standby role, and the standby database transitions to the primary role.
The following sections describes the prerequisites, process for configuring high availability for the PMGDB, and Cisco RMS upgrade information.
The output of some of the commands may slightly differ based on the installation setup and time of theexecution.
Note
• Prerequisites, page 2
• Configuration Workflow, page 3
• Configuring the Standby Server Setup, page 4
• Setting Up the Oracle Data Guard Broker, page 16
• Enabling Flashback, page 20
• Configuring Hot Standby for PMG DB, page 21
• Configuring Cold Standby , page 34
• Testing Hot Standby, page 35
• Testing Cold Standby, page 45
• Rolling Back and Cleaning Up Standby and Primary Configurations, page 55
• Troubleshooting Data Guard on PMG DB, page 65
Prerequisites• Primary server should have a running instance.
• Standby server should have a software only installation.
• Operator or user of this procedure should have the following experience:
◦Knowledge of basic database/DBA know-how
◦Knowledge of basic Linux/shell commands
◦Ability to edit files with vi or vim
◦Ability to view files (cat, tail, more, less)
High Availability for Cisco RAN Management Systems2
Configuring High Availability for the PMG DBPrerequisites
• ORACLE_HOME directory path should be /u01/app/oracle/product/11.2.0/dbhome_1. If this path isdifferent, respective file paths (for example, data files) would differ.
• To enable the failover feature for Cisco RMS applications connecting to PMGDB, the Cisco RMSCentralnode should be configured for PMG DB as per the install guide (see Cisco RAN Management SystemInstallation Guide).
The installation can be performed on either UCS 5108 Blade Server or UCS 240 Server.There is no specific dependency based type of hardware.
Note
Configuration WorkflowThe following table provides the general flow which can be followed to perform tasks for different types ofredundancy setup for PMGDB using the Oracle Data Guard tool.
ProcedureTaskSl. No.
Configuring the Standby Server Setup, onpage 4
Configure the Standby Server1
Configuring the Hot Standby Server, on page10
Add Only Hot Standby to Primary Server2
Configuring Primary With Only ColdStandby, on page 34
Add Only Cold Standby to Primary Server3
Configuring Primary With Hot and ColdStandby, on page 34
Add Both, Hot and Cold Standby toPrimary Server
4
Testing Hot Standby, on page 35Test Hot Standby Setup5
• Testing Site Failure, on page 46
• Recovering Original Primary After SiteFailure, on page 48
Test Cold Standby Setup6
Troubleshooting Data Guard on PMG DB,on page 65
Troubleshooting PMG DB7
Rolling Back and Cleaning Up Standby andPrimary Configurations, on page 55
Rollback and Clean-up8
High Availability for Cisco RAN Management Systems 3
Configuring High Availability for the PMG DBConfiguration Workflow
Configuring the Standby Server SetupThe following procedure describes how to configure a standby server for the primary server. If the PMG DBredundancy setup involves primary with only cold standby, this standby server can be configured as coldstandby server. If the redundancy setup involves primary with only hot standby or primary with both hot andcold standby, this standby can be configured as a hot standby.
• Configuring the Primary Server, on page 4
• Configuring the Hot Standby Server, on page 10
Configuring the Primary Server
Step 1 Log in to the primary server with the username, “oracle”.Step 2 Log in to the sql prompt.
Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 17:51:51 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL>
Step 3 Create the pfile as a backup vSphere client. This pfile can be used later when rollback is needed.The specific path for initPMGDB_pre_dgsetup.ora file can be specified if the file is to be stored in some backup directory(for example, /backup/initPMGDB_pre_dgsetup.ora). If the path is not specified the file gets created under the defaultdirectory, that is, $ORACLE_HOME/dbs.
Example:SQL> CREATE PFILE='initPMGDB_pre_dgsetup.ora' FROM SPFILE;File created.
Logging
Step 1 Ensure that the primary database is in archivelog mode.Step 2
Example:SQL> SELECT log_mode FROM v$database;LOG_MODE------------NOARCHIVELOG
High Availability for Cisco RAN Management Systems4
Configuring High Availability for the PMG DBConfiguring the Standby Server Setup
If it is noarchivelog mode, switch to archivelog mode:SQL> SHUTDOWN IMMEDIATE;Database closed.Database dismounted.ORACLE instance shut down.
SQL> STARTUP MOUNT;ORACLE instance started.
Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.
SQL> ALTER DATABASE ARCHIVELOG;Database altered.
SQL> ALTER DATABASE OPEN;Database altered.
SQL> SELECT log_mode FROM v$database;LOG_MODE------------ARCHIVELOG
Step 3 Enable forced logging by using the following command.
Example:SQL> ALTER DATABASE FORCE LOGGING;Database altered.
Initializing Parameters
Step 1 Check the setting for the DB_NAME and DB_UNIQUE_NAME parameters. In this case they are both set to "PMGDB"on the primary database.
Example:SQL> show parameter db_nameNAME TYPE VALUE------------------------------------ ----------- -----------------db_name string PMGDB
SQL> show parameter db_unique_nameNAME TYPE VALUE------------------------------------ ----------- -----------------db_unique_name string PMGDB
Step 2 Enter the DB_NAME of the standby database. It should be the same as that of the primary, but it must have a differentDB_UNIQUE_NAME value. The DB_UNIQUE_NAME values of the primary and standby database should be used inthe DG_CONFIG setting of the LOG_ARCHIVE_CONFIG parameter. For this example, the standby database has thevalue "PMGDB_STBY".
High Availability for Cisco RAN Management Systems 5
Configuring High Availability for the PMG DBConfiguring the Primary Server
Example:SQL> ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(PMGDB,PMGDB_STBY)';System altered.
Step 3 Set the suitable remote archive log destinations. In this case, flash recovery area for the local location is used; however,other location can be specified. Note the SERVICE and the DB_UNIQUE_NAME for the remote location reference thestandby location.
Example:SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=pmgdb_stby NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PMGDB_STBY';LOG_ARCHIVE_CONFIG='DG_CONFIG=(PMGDB,PMGDB_STBY)';System altered.
SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;System altered.
Step 4 Set the LOG_ARCHIVE_FORMAT and LOG_ARCHIVE_MAX_PROCESSES parameters to appropriate values andthe REMOTE_LOGIN_PASSWORDFILE must be set to exclusive.
Example:SQL> ALTER SYSTEM SET LOG_ARCHIVE_FORMAT='%t_%s_%r.arc' SCOPE=SPFILE;System altered.
SQL> ALTER SYSTEM SET LOG_ARCHIVE_MAX_PROCESSES=30;System altered.
SQL> ALTER SYSTEM SET REMOTE_LOGIN_PASSWORDFILE=EXCLUSIVE SCOPE=SPFILE;System altered.
Step 5 Set the following parameters (in addition to the previous setting) to ensure that the primary server is ready to switch tostandby mode. (This setting is recommended.)Adjust the *_CONVERT parameters to account for filename and path differences between the servers.
Example:SQL> ALTER SYSTEM SET FAL_SERVER=PMGDB_STBY;System altered.
SQL> ALTER SYSTEM SET DB_FILE_NAME_CONVERT='PMGDB_STBY','PMGDB' SCOPE=SPFILE;System altered.
SQL> ALTER SYSTEM SET LOG_FILE_NAME_CONVERT='PMGDB_STBY','PMGDB' SCOPE=SPFILE;System altered.
SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO;System altered.
Step 6 Restart the database to implement the modifications made to the parameters.
Example:SQL> shutdown immediateDatabase closed.Database dismounted.ORACLE instance shut down.
SQL> startupORACLE instance started.Total System Global Area 3340451840 bytes
High Availability for Cisco RAN Management Systems6
Configuring High Availability for the PMG DBConfiguring the Primary Server
Fixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.Database opened.
Step 7 Exit from sql prompt.
Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
Setting Up the Service
Step 1 Provide the values for the primary and standby databases in the "$ORACLE_HOME/network/admin/tnsnames.ora" fileson both the servers. The following values are required. The values of database server hostname or IP address, ports, andoracle home should be specified as per the installation setup.
Example:PMGDB =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = <primary-server-host-address>)(PORT = <port number>)(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PMGDB)
))
PMGDB_STBY =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = <standby-server-host-address>)(PORT = <port number>))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PMGDB)
))
Step 2 Edit "$ORACLE_HOME/network/admin/listener.ora" to add the entry that the Data Guard Broker will refer to. IfSID_LIST is already present, add SID_DESC to the list.
Example:SID_LIST_LISTENER =(SID_LIST =(SID_DESC =(GLOBAL_NAME = PMGDB_DGMGRL)(ORACLE_HOME = <oracle-home-directory-path>)(SID_NAME = PMGDB)
))
Starting the ListenerEnsure that the listener is started on the primary server.
High Availability for Cisco RAN Management Systems 7
Configuring High Availability for the PMG DBConfiguring the Primary Server
All of the output is not displayed.Note
$ lsnrctl statusIf not running:LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 02-JUL-2014 09:35:37Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<host address>)(PORT=<port number>))TNS-12541: TNS:no listenerTNS-12560: TNS:protocol adapter errorTNS-00511: No listenerLinux Error: 111: Connection refused
If running:LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 02-JUL-2014 09:43:23Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<host address>)(PORT=<port number>)))STATUS of the LISTENER------------------------…Services Summary...……The command completed successfully
1 If listener is not running (that is, TNS-12541: TNS:no listener), start it.$ lsnrctl startLSNRCTL for Linux: Version 11.2.0.1.0 - Production on 17-JUN-2014 19:10:31Copyright (c) 1991, 2009, Oracle. All rights reserved.Starting /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr:please wait...……The command completed successfully
2 If listener is running, reload it.$ lsnrctl reloadLSNRCTL for Linux: Version 11.2.0.1.0 - Production on 17-JUN-2014 19:12:23Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to…The command completed successfully
Backing Up the Primary Database
Step 1 Back up the primary database, which will be restored on the standby database. To do this, open another console for theprimary database server and log in as an oracle user.
Step 2 Set the ORACLE_SID to PMGDB and take a backup.The complete output is notdisplayed.
Note
Example:$ export ORACLE_SID=PMGDB$ rman target=/
Recovery Manager: Release 11.2.0.1.0 - Production on Thu Jul 3 18:53:35 2014
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
High Availability for Cisco RAN Management Systems8
Configuring High Availability for the PMG DBConfiguring the Primary Server
connected to target database: PMGDB (DBID=122120137)
RMAN> BACKUP DATABASE PLUS ARCHIVELOG;Starting backup at 03-JUL-14current log archivedusing target database control file instead of recovery catalogallocated channel: ORA_DISK_1channel ORA_DISK_1: SID=97 device type=DISKchannel ORA_DISK_1: starting archived log backup setchannel ORA_DISK_1: specifying archived log(s) in backup setinput archived log thread=1 sequence=15 RECID=93 STAMP=851365692input archived log thread=1 sequence=16 RECID=95 STAMP=851367216………………channel ORA_DISK_1: starting full datafile backup setchannel ORA_DISK_1: specifying datafile(s) in backup setinput datafile file number=00005 name=/u01/app/oracle/oradata/PMGDB/mapinfo_ts.dbfinput datafile file number=00006 name=/u01/app/oracle/oradata/PMGDB/pmgdb_ts.dbfinput datafile file number=00001 name=/u01/app/oracle/oradata/PMGDB/system01.dbfinput datafile file number=00002 name=/u01/app/oracle/oradata/PMGDB/sysaux01.dbfinput datafile file number=00003 name=/u01/app/oracle/oradata/PMGDB/undotbs01.dbfinput datafile file number=00004 name=/u01/app/oracle/oradata/PMGDB/users01.dbfpiecehandle=/u01/app/oracle/flash_recovery_area/PMGDB/backupset/2014_07_03/o1_mf_annnn_TAG20140703T185830_9vbpmykl_.bkptag=TAG20140703T185830 comment=NONEchannel ORA_DISK_1: backup set complete, elapsed time: 00:00:01Finished backup at 03-JUL-14
RMAN> exitRecovery Manager complete.
To check if the backup is completed successfully, check the last statement in the output. For example, “Finishedbackup at 03-JUL-14”.
Note
Creating Standby Control File and PFILE
Step 1 Login to the sql prompt.
Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 17:51:51 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL>
Step 2 Create a control file for the standby database using the following command on the primary database.
High Availability for Cisco RAN Management Systems 9
Configuring High Availability for the PMG DBConfiguring the Primary Server
Example:SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/tmp/pmgdb_stby.ctl';Database altered.
Step 3 Create a parameter file for the standby database.
Example:SQL> CREATE PFILE='/tmp/initPMGDB_stby.ora' FROM SPFILE;File created.
Step 4 Exit from the sql prompt.
Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production…
Step 5 Edit the PFILE generated, that is, /tmp/initPMGDB_stby.ora, making the entries relevant for the standby database.Because this is a replica of the original server, only the following parameters should be modified. Modify the parameterif it exists, otherwise add the parameter with the specified value.
Example:*.db_unique_name='PMGDB_STBY'*.fal_server='PMGDB'*.log_archive_dest_2='SERVICE=pmgdb ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)DB_UNIQUE_NAME=PMGDB'
Configuring the Hot Standby Server
Step 1 Log in to the secondary or standby server with the username “oracle”.Step 2 Add the tnsnames entries to "$ORACLE_HOME/network/admin/tnsnames.ora".
Example:PMGDB =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = <primary-server-host-address>)(PORT = <port number>))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PMGDB)
))
PMGDB_STBY =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = <standby-server-host-address>)(PORT = <port number>))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PMGDB)
))
Step 3 Edit "$ORACLE_HOME/network/admin/listener.ora" to add the entry that the Data Guard Broker will refer to.
High Availability for Cisco RAN Management Systems10
Configuring High Availability for the PMG DBConfiguring the Hot Standby Server
Example:SID_LIST_LISTENER =(SID_LIST =(SID_DESC =(GLOBAL_NAME = PMGDB_STBY_DGMGRL)(ORACLE_HOME = <oracle-home-directory-path>)(SID_NAME = PMGDB)
))
Copying Files
Step 1 Create the necessary directories on the standby server.
Example:$ mkdir -p /u01/app/oracle/oradata/PMGDBThe system responds with a command prompt.
$ mkdir -p /u01/app/oracle/flash_recovery_area/PMGDBThe system responds with a command prompt.
$ mkdir -p /u01/app/oracle/admin/PMGDB/adumpThe system responds with a command prompt.
Step 2 Copy the files from the primary to the standby server. Copy the standby control file to all locations.
Example:$ scp oracle@<primary-server-host-address>:/tmp/pmgdb_stby.ctl/u01/app/oracle/oradata/PMGDB/control01.ctl<scp-output>
$ cp /u01/app/oracle/oradata/PMGDB/control01.ctl/u01/app/oracle/flash_recovery_area/PMGDB/control02.ctlThe system responds with a command prompt.
Step 3 Copy archivelogs and backups.
Example:$ scp -r oracle@<primary-server-host-address>:/u01/app/oracle/flash_recovery_area/PMGDB/archivelog /u01/app/oracle/flash_recovery_area/PMGDB<scp-output>
$ scp -r oracle@<primary-server-host-address>:/u01/app/oracle/flash_recovery_area/PMGDB/backupset/u01/app/oracle/flash_recovery_area/PMGDB<scp-output>
Step 4 Copy the parameter file.
Example:$ scp oracle@<primary-server-host-address>:/tmp/initPMGDB_stby.ora /tmp/initPMGDB_stby.ora<scp-output>
Step 5 Copy the remote login password file.
High Availability for Cisco RAN Management Systems 11
Configuring High Availability for the PMG DBConfiguring the Hot Standby Server
Example:$ scp oracle@<primary-server-host-address>:$ORACLE_HOME/dbs/orapwPMGDB $ORACLE_HOME/dbs<scp-output>
The backups are copied across to the standby server as part of the flash recovery area (FRA) copy. Ifbackups are not held within the FRA, ensure that those are copied to the standby server and make themavailable from the same path as used on the primary server.
Note
Starting the Listener on Standby ServerEnsure that the listener is started on the standby server.
All of the output is not displayed.Note
$ lsnrctl status
If not running:
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 02-JUL-2014 09:35:37Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<primary-server-host-address>)(PORT=<port number>))TNS-12541: TNS:no listenerTNS-12560: TNS:protocol adapter errorTNS-00511: No listenerLinux Error: 111: Connection refused
If running:
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 02-JUL-2014 09:43:23Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<primary-server-host-address>)(PORT=<port number>)))STATUS of the LISTENER------------------------…Services Summary...……The command completed successfully
1 If listener is not running (that is, TNS-12541: TNS:no listener), start it.$ lsnrctl startLSNRCTL for Linux: Version 11.2.0.1.0 - Production on 17-JUN-2014 19:10:31Copyright (c) 1991, 2009, Oracle. All rights reserved.Starting /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr:please wait...……The command completed successfully
High Availability for Cisco RAN Management Systems12
Configuring High Availability for the PMG DBConfiguring the Hot Standby Server
2 If listener is running, reload it.$ lsnrctl reloadLSNRCTL for Linux: Version 11.2.0.1.0 - Production on 17-JUN-2014 19:12:23Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to…The command completed successfully
Restoring Backup
Step 1 Create the SPFILE from the modified PFILE.
Example:$ export ORACLE_SID=PMGDBThe system responds with a command prompt.
$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 12:24:01 2014
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to an idle instance.
SQL> CREATE SPFILE FROM PFILE='/tmp/initPMGDB_stby.ora';File created.
Step 2 Exit from the sql prompt.
Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release11.2.0.1.0 - 64bit Production
Step 3 Restore the backup files. Depending on database size, restore time will vary.All of the output is notdisplayed.
Note
Example:$ export ORACLE_SID=PMGDB$ rman target=/Recovery Manager: Release 11.2.0.1.0 - Production on Thu Jul 3 19:23:50 2014
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
connected to target database (not started)
RMAN> STARTUP MOUNT;Oracle instance starteddatabase mounted
Total System Global Area 3340451840 bytes
Fixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytes
RMAN> RESTORE DATABASE;Starting restore at 03-JUL-14
High Availability for Cisco RAN Management Systems 13
Configuring High Availability for the PMG DBConfiguring the Hot Standby Server
Starting implicit crosscheck backup at 03-JUL-14using target database control file instead of recovery catalogallocated channel: ORA_DISK_1………channel ORA_DISK_1: piece handle=/u01/app/oracle/flash_recovery_area/PMGDB/backupset/2014_07_03/o1_mf_nnndf_TAG20140703T185812_9vbpmf2x_.bkp tag=TAG20140703T185812channel ORA_DISK_1: restored backup piece 1channel ORA_DISK_1: restore complete, elapsed time: 00:00:26Finished restore at 03-JUL-14
RMAN> exitRecovery Manager complete.
Creating Redo Logs
Step 1 Exit from the sqlplus prompt if already logged in and re-login.
Example:SQL> exitDisconnected
$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 12:24:01 2014
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to an idle instance.
Step 2 Create online redo logs for the standby.
Example:SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=MANUAL;System altered.
SQL> ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/PMGDB/online_redo01.log') SIZE 50M;Database altered.
SQL> ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/PMGDB/online_redo02.log') SIZE 50M;Database altered.
SQL> ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/PMGDB/online_redo03.log') SIZE 50M;Database altered.
SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO;System altered.
Create standby redo logs on both the standby and the primary database (in case of switchovers). The standby redo logs shouldbe at least as big as the largest online redo log and there should be one extra group per thread compared the online redo logs.In this case, the following standby redo logs must be created on both servers.SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo01.log') SIZE 50M;Database altered.
SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo02.log') SIZE 50M;
High Availability for Cisco RAN Management Systems14
Configuring High Availability for the PMG DBConfiguring the Hot Standby Server
Database altered.
SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo03.log') SIZE 50M;Database altered.
SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo04.log') SIZE 50M;Database altered.
Step 3 Create standby redo logs on both the standby and the primary database (in case of switchovers). The standby redo logsshould be at least as big as the largest online redo log and there should be one extra group per thread compared the onlineredo logs. In this case, the following standby redo logs must be created on both servers.
Example:SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo01.log') SIZE 50M;Database altered.
SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo02.log') SIZE 50M;Database altered.
SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo03.log') SIZE50M;Database altered.
SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo04.log') SIZE50M;Database altered.
Step 4 Repeat steps 1 to 3 to create standby redo logs on the primary server. After this is complete, the apply process can bestarted.
Starting the Apply ProcessStart the apply process on the standby server.SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;Database altered.
Checking Status
Step 1 Check status on primary database.The status LOG SWITCH GAP or RESOLVABLE GAP will change after the Data Guard Broker is setup.
Note
Example:SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE,SWITCHOVER_STATUS from v$database
NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS--------- -------------------- ------------------------------ ---------------- --------------------PMGDB READ WRITE PMGDB PRIMARYLOG SWITCH GAP
Step 2 Check status on secondary database.The switchover status could be TO PRIMARYor SESSIONSACTIVATE.Note
High Availability for Cisco RAN Management Systems 15
Configuring High Availability for the PMG DBConfiguring the Hot Standby Server
Example:SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE,SWITCHOVER_STATUS from v$database
NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS--------- -------------------- ---------------------------- ---------------
--------------------PMGDB MOUNTED PMGDB_STBYPHYSICAL STANDBY TO PRIMARY
Setting Up the Oracle Data Guard BrokerThe Oracle Data Guard Broker is used to create a broker configuration that allows the broker to manage andmonitor primary and standby databases together as an integrated unit.
Step 1 Log in to the primary server with the username “oracle”.Step 2 Check the Data Guard Broker process on the primary database.
Example:SQL> sho parameter dg_broker
NAME TYPE VALUE------------------------------------ ----------- ------------------------------dg_broker_config_file1 string /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1PMGDB.dat
dg_broker_config_file2 string /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2PMGDB.dat
dg_broker_start boolean FALSE
Step 3 Start the Data Guard Broker process on the primary database.
Example:SQL> alter system set dg_broker_start=true scope=both;System altered.
Step 4 Check the DG_BROKER on the standby database and start it.
Example:SQL> sho parameter dg_brokerNAME TYPE VALUE----------------- ---------- ----------dg_broker_start boolean FALSE
SQL> alter system set dg_broker_start=true scope=both ;System altered.
Step 5 Verify the "$ORACLE_HOME/network/admin/listener.ora" file.If required, edit the file that includes the db_unique_name_DGMGRL.db_domain values for the GLOBAL_DBNAMEin both primary and standby database. To set the value, check the db_domain value on the primary and standby database.
High Availability for Cisco RAN Management Systems16
Configuring High Availability for the PMG DBSetting Up the Oracle Data Guard Broker
Example:SQL> show parameter db_domainNAME TYPE VALUE-------------- ----------- --------------db_domain stringBecause the value of the db_domain is null so is the value of GLOBAL_DBNAME = PMGDB_DGMGRL for primarydatabase and for standby GLOBAL_DBNAME = PMGDB_STBY_DGMGRL. The primary"$ORACLE_HOME/network/admin/listener.ora" file is as.
Example:SID_LIST_LISTENER =
(SID_DESC =(GLOBAL_NAME = PMGDB_DGMGRL)(ORACLE_HOME = <oracle-home-directory-path>)(SID_NAME = PMGDB)
))
Step 6 Verify the "$ORACLE_HOME/network/admin/listener.ora" file on the standby database.Step 7 Configure the Data Guard configuration. Open another console on primary or standby server to open the DGMGRLCLI.
DGMGRLCLI can be accessed from any server where Oracle Client Administrator is installed.Note
Example:$ dgmgrlDGMGRL for Linux: Version 11.2.0.1.0 - 64bit Production
Copyright (c) 2000, 2009, Oracle. All rights reserved.
Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys@PMGDBPassword:.Connected.
DGMGRL> create configuration 'dgPMGDB' as primary database is 'PMGDB' connect identifier is PMGDB ;Configuration "dgPMGDB" created with primary database " PMGDB" .
Step 8 Check the status of the configuration after the configuration is created.
Example:DGMGRL> show configurationConfiguration - dgPMGDBProtection Mode : MaxPerformanceDatabases : PMGDB - Primary databaseFast-Start Failover : DISABLEDConfiguration Status : DISABLED
Step 9 Add the standby database to the Data Guard Broker configuration.
Example:DGMGRL> add database 'PMGDB_STBY' as connect identifier is PMGDB_STBY maintained as physical ;Database " PMGDB_STBY " added
DGMGRL> show configurationConfiguration - dgPMGDBProtection Mode : MaxPerformanceDatabases : PMGDB - Primary database
: PMGDB_STBY - Physical standby database
High Availability for Cisco RAN Management Systems 17
Configuring High Availability for the PMG DBSetting Up the Oracle Data Guard Broker
Fast-Start Failover : DISABLEDConfiguration Status : DISABLED
Step 10 Enable the configuration.
Example:DGMGRL> enable configurationEnabled.
DGMGRL> show configurationConfiguration - dgPMGDBProtection Mode : MaxPerformanceDatabases : PMGDB - Primary database
: PMGDB_STBY - Physical standby databaseFast-Start Failover : DISABLEDConfiguration Status : SUCCESS
Step 11 View the primary and standby database properties.While copying the command from the document, the single quotes (for example, as in 'PMGDB') may not becopied correctly, the command may have to be entered at the prompt.
Note
Example:DGMGRL> show database verbose 'PMGDB'Database - PMGDBRole: PRIMARYIntended State: TRANSPORT-ONInstance(s):PMGDB
Properties:DGConnectIdentifier = 'pmgdb'ObserverConnectIdentifier = ''LogXptMode = 'ASYNC'DelayMins = '0'Binding = 'optional'MaxFailure = '0'MaxConnections = '1'ReopenSecs = '300'NetTimeout = '30'RedoCompression = 'DISABLE'LogShipping = 'ON'PreferredApplyInstance = ''ApplyInstanceTimeout = '0'ApplyParallel = 'AUTO'StandbyFileManagement = 'AUTO'ArchiveLagTarget = '0'LogArchiveMaxProcesses = '30'LogArchiveMinSucceedDest = '1'DbFileNameConvert = 'PMGDB_STBY, PMGDB'LogFileNameConvert = 'PMGDB_STBY, PMGDB'FastStartFailoverTarget = ''StatusReport = '(monitor)'InconsistentProperties = '(monitor)'InconsistentLogXptProps = '(monitor)'SendQEntries = '(monitor)'LogXptStatus = '(monitor)'RecvQEntries = '(monitor)'HostName = 'oracle-vm-primary'SidName = 'PMGDB'
StaticConnectIdentifier ='(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oracle-vm-primary)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=PMGDB_DGMGRL)(INSTANCE_NAME=PMGDB)(SERVER=DEDICATED)))'
StandbyArchiveLocation = 'USE_DB_RECOVERY_FILE_DEST'AlternateLocation = ''LogArchiveTrace = '0'LogArchiveFormat = '%t_%s_%r.arc'TopWaitEvents = '(monitor)'
High Availability for Cisco RAN Management Systems18
Configuring High Availability for the PMG DBSetting Up the Oracle Data Guard Broker
Database Status:SUCCESS
DGMGRL> show database verbose 'PMGDB_STBY'Database - PMGDB_STBYRole: PHYSICAL STANDBYIntended State: APPLY-ONTransport Lag: 0 secondsApply Lag: 0 secondsReal Time Query: OFFInstance(s):PMGDB
Properties:DGConnectIdentifier = 'pmgdb_stby'ObserverConnectIdentifier = ''LogXptMode = 'ASYNC'DelayMins = '0'Binding = 'OPTIONAL'MaxFailure = '0'MaxConnections = '1'ReopenSecs = '300'NetTimeout = '30'RedoCompression = 'DISABLE'LogShipping = 'ON'PreferredApplyInstance = ''ApplyInstanceTimeout = '0'ApplyParallel = 'AUTO'StandbyFileManagement = 'AUTO'ArchiveLagTarget = '0'LogArchiveMaxProcesses = '30'LogArchiveMinSucceedDest = '1'DbFileNameConvert = 'PMGDB_STBY, PMGDB'LogFileNameConvert = 'PMGDB_STBY, PMGDB'FastStartFailoverTarget = ''StatusReport = '(monitor)'InconsistentProperties = '(monitor)'InconsistentLogXptProps = '(monitor)'SendQEntries = '(monitor)'LogXptStatus = '(monitor)'RecvQEntries = '(monitor)'HostName = 'blr-oracle2-standby'SidName = 'PMGDB'
StaticConnectIdentifier ='(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=blr-oracle2-standby)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=PMGDB_STBY_DGMGRL)(INSTANCE_NAME=PMGDB)(SERVER=DEDICATED)))'
StandbyArchiveLocation = 'USE_DB_RECOVERY_FILE_DEST'AlternateLocation = ''LogArchiveTrace = '0'LogArchiveFormat = '%t_%s_%r.arc'TopWaitEvents = '(monitor)'
Database Status:SUCCESS
Verifying Log Synchronization on Standby Server
Step 1 Identify the existing archived redo log files on the standby server. Note the latest sequence number.Sequence number and timestamp in the output will vary depending on the installationsetup.
Note
High Availability for Cisco RAN Management Systems 19
Configuring High Availability for the PMG DBVerifying Log Synchronization on Standby Server
Example:SQL> ALTER SESSION SET nls_date_format='DD-MON-YYYY HH24:MI:SS';Session altered.
SQL> SELECT sequence#, first_time, next_time, appliedFROM v$archived_logORDER BY sequence#;SEQUENCE# FIRST_TIME NEXT_TIME APPLIED---------- -------------------- -------------------- ---------
41 10-JUL-2014 09:06:27 10-JUL-2014 09:06:53 YES42 10-JUL-2014 09:06:53 10-JUL-2014 09:36:09 YES43 10-JUL-2014 09:36:09 10-JUL-2014 09:51:38 IN-MEMORY
Step 2 Force a log switch to archive the current online redo log file on the primary server.
Example:SQL> ALTER SYSTEM SWITCH LOGFILE;System altered.
Step 3 Verify on the standby, the new redo data that was archived on the standby database. That is, new sequence number withlatest timestamp of redo apply is displayed.
The value of the APPLIED column for the most recently received log file will either be IN-MEMORY or YESif that log file has been applied.
Note
Example:SQL> SELECT sequence#, first_time, next_time, appliedFROM v$archived_logORDER BY sequence#;SEQUENCE# FIRST_TIME NEXT_TIME APPLIED---------- -------------------- -------------------- ---------
41 10-JUL-2014 09:06:27 10-JUL-2014 09:06:53 YES42 10-JUL-2014 09:06:53 10-JUL-2014 09:36:09 YES43 10-JUL-2014 09:36:09 10-JUL-2014 09:51:38 YES44 10-JUL-2014 09:51:38 10-JUL-2014 10:03:48 IN-MEMORY
Enabling FlashbackFlashback database is needed to enable fast-start failover for hot standby. In case of cold standby also it isadvisable to enable Flashback database because it is helpful in recovering the database.
Step 1 Check if flashback is on from the primary database sql prompt.
Example:SQL> select flashback_on from v$database;FLASHBACK_ON------------------NO
Step 2 Set flashback "on" if it is not already on.
High Availability for Cisco RAN Management Systems20
Configuring High Availability for the PMG DBEnabling Flashback
Example:Database altered.
Step 3 Check if the flashback is on from the standby database sql prompt.
Example:SQL> select flashback_on from v$database;FLASHBACK_ON------------------NO
Step 4 Cancel standby apply process if flashback mode has to be made on.
Example:SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;Database altered.
Step 5 Set flashback “on” if it is not already on.
Example:SQL> alter database flashback on;Database altered.
Step 6 Start back apply process.
Example:SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;Database altered.
Configuring Hot Standby for PMG DBHot standby refers to a standby server that becomes active and acts as primary when the original primaryserver fails. The failover is automatic without manual intervention and with a little or no downtime. Typicallyhot standby servers are placed on different hosts in the same site as that of the primary server. The followingprocedure describes how to add a hot standby. Skip this section if there is no need to configure a hot standby.
Setting Up the Hot Standby
Before You Begin
Ensure that you have completed the Configuring the Standby Server Setup, on page 4 to create a standbyserver.
High Availability for Cisco RAN Management Systems 21
Configuring High Availability for the PMG DBConfiguring Hot Standby for PMG DB
Enabling Failover
DETAILED STEPS
PurposeCommand or Action
To enable failover, return to the console where the DGMGRL tool is running andchange the LogXptMode property for the PMGDB.
Step 1
Example:DGMGRL> edit database "PMGDB" set property LogXptMode='SYNC';Property "logxptmode" updated
Change the LogXptMode PMGDB_STBY.Step 2
Example:DGMGRL> edit database "PMGDB_STBY" set property LogXptMode='SYNC';Property "logxptmode" updated
Check the LogXptStatus.Step 3
Example:DGMGRL> show database 'PMGDB' 'LogXptStatus';LOG TRANSPORT STATUSPRIMARY_INSTANCE_NAME STANDBY_DATABASE_NAME STATUS
PMGDB PMGDB_STBY
Set the protection mode to MaxAvailabilityStep 4
Example:DGMGRL> edit configuration set protection mode as MaxAvailability;Succeeded.
Return to the DGMGRL console and enable failover. If not connected, connect assys@PMGDB.
Step 5
Example:DGMGRL> enable fast_start failover;Enabled
It is recommendedrunning the Observer ona server host that isseparate from the primaryand standby servers. Thishost needs Oracle ClientAdministrator installed torun DGMGRL tool andthe Observer.
Note
Otherwise, run the Observer onthe standby server. In case of
Open another console for the standby server and login as an oracle user. Start theObserver as a background process. The Observer monitors both primary and standbydatabases and detects failures, if any.
Example:$ nohup dgmgrl -silent sys/<password>@PMGDB "start observer" &$ cat nohup.outObserver started
Step 6
failure, if primary and standbyroles are switched, ensure that theobserver is running on the newstandby server.
High Availability for Cisco RAN Management Systems22
Configuring High Availability for the PMG DBSetting Up the Hot Standby
PurposeCommand or Action
Return to the previous DGMGRL console and verify the configuration.Step 7
Example:DGMGRL> show configuration verboseConfiguration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby databaseWarning: ORA-16826: apply service state is inconsistent with the
DelayMins property
(*) Fast-Start Failover target
Fast-Start Failover: ENABLEDThreshold: 30 secondsTarget: PMGDB_STBYObserver: oracle-vm-primaryLag Limit: 30 seconds (not in use)Shutdown Primary: TRUEAuto-reinstate: TRUE
Configuration Status:WARNING
Start recovery from the current log file on the standby server if the above error(mentioned in the previous step) is observed.
Step 8
Example:SQL> alter database recover managed standby database cancel;Database altered.
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENTLOGFILE DISCONNECT FROM SESSION;Database altered.
Return to the DGMGRL console to check the configuration again.Step 9
Example:DGMGRL> show configuration verboseConfiguration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby database
(*) Fast-Start Failover target
Fast-Start Failover: ENABLED
Threshold: 30 secondsTarget: PMGDB_STBYObserver: oracle-vm-primaryLag Limit: 30 seconds (not in use)Shutdown Primary: TRUEAuto-reinstate: TRUE
Configuration Status:SUCCESS
High Availability for Cisco RAN Management Systems 23
Configuring High Availability for the PMG DBSetting Up the Hot Standby
Checking Status
Step 1 Check status on primary database.The status LOG SWITCH GAP or RESOLVABLE GAP will change after the Data Guard Broker is setup.
Note
Example:SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE,SWITCHOVER_STATUS from v$database
NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS--------- -------------------- ------------------------------ ---------------- --------------------PMGDB READ WRITE PMGDB PRIMARYLOG SWITCH GAP
Step 2 Check status on secondary database.The switchover status could be TO PRIMARYor SESSIONSACTIVATE.Note
Example:SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE,SWITCHOVER_STATUS from v$database
NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS--------- -------------------- ---------------------------- ---------------
--------------------PMGDB MOUNTED PMGDB_STBYPHYSICAL STANDBY TO PRIMARY
Initializing Parameters for Standby Server
Step 1 Add db_unique_name to log_archive_config
Example:SQL> ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(PMGDB,PMGDB_STBY,PMGDB_STBY2)';System altered.
Step 2 Set and enable LOG_ARCHIVE_DEST_3.
Example:SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_3='SERVICE=PMGDB_stby2 NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PMGDB_STBY2';System altered.
Step 3 Set the parameters to ensure that the primary is ready to switch roles to become a standby.
Example:SQL> ALTER SYSTEM SET FAL_SERVER=PMGDB_STBY,PMGDB_STBY2;System altered.
SQL> ALTER SYSTEM SET DB_FILE_NAME_CONVERT='PMGDB','PMGDB_STBY', 'PMGDB', 'PMGDB_STBY2' SCOPE=SPFILE;System altered.
High Availability for Cisco RAN Management Systems24
Configuring High Availability for the PMG DBSetting Up the Hot Standby
SQL> ALTER SYSTEM SET LOG_FILE_NAME_CONVERT ='PMGDB','PMGDB_STBY', 'PMGDB', 'PMGDB_STBY2' SCOPE=SPFILE;System altered.
Step 4 Restart the database to implement the modifications made to the parameters.
Example:SQL> shutdown immediateDatabase closed.Database dismounted.ORACLE instance shut down.
SQL> startupORACLE instance started.Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.Database opened.
Setting Up the Service for Standby ServerProvide the values for the new standby databases in the "$ORACLE_HOME/network/admin/tnsnames.ora"files on both the servers. The following values are used during the setup.PMGDB_STBY2 =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = <standby2-server-host-address>)(PORT = <port number>))
(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PMGDB)
))
Backing Up the Primary Database
Step 1 Back up the primary database, which will be restored on the standby database. To do this, open another console for theprimary database server and log in as an oracle user.
Step 2 Set the ORACLE_SID to PMGDB and take a backup.The complete output is notdisplayed.
Note
Example:$ export ORACLE_SID=PMGDB$ rman target=/
Recovery Manager: Release 11.2.0.1.0 - Production on Thu Jul 3 18:53:35 2014
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
connected to target database: PMGDB (DBID=122120137)
High Availability for Cisco RAN Management Systems 25
Configuring High Availability for the PMG DBSetting Up the Hot Standby
RMAN> BACKUP DATABASE PLUS ARCHIVELOG;Starting backup at 03-JUL-14current log archivedusing target database control file instead of recovery catalogallocated channel: ORA_DISK_1channel ORA_DISK_1: SID=97 device type=DISKchannel ORA_DISK_1: starting archived log backup setchannel ORA_DISK_1: specifying archived log(s) in backup setinput archived log thread=1 sequence=15 RECID=93 STAMP=851365692input archived log thread=1 sequence=16 RECID=95 STAMP=851367216………………channel ORA_DISK_1: starting full datafile backup setchannel ORA_DISK_1: specifying datafile(s) in backup setinput datafile file number=00005 name=/u01/app/oracle/oradata/PMGDB/mapinfo_ts.dbfinput datafile file number=00006 name=/u01/app/oracle/oradata/PMGDB/pmgdb_ts.dbfinput datafile file number=00001 name=/u01/app/oracle/oradata/PMGDB/system01.dbfinput datafile file number=00002 name=/u01/app/oracle/oradata/PMGDB/sysaux01.dbfinput datafile file number=00003 name=/u01/app/oracle/oradata/PMGDB/undotbs01.dbfinput datafile file number=00004 name=/u01/app/oracle/oradata/PMGDB/users01.dbfpiecehandle=/u01/app/oracle/flash_recovery_area/PMGDB/backupset/2014_07_03/o1_mf_annnn_TAG20140703T185830_9vbpmykl_.bkptag=TAG20140703T185830 comment=NONEchannel ORA_DISK_1: backup set complete, elapsed time: 00:00:01Finished backup at 03-JUL-14
RMAN> exitRecovery Manager complete.
To check if the backup is completed successfully, check the last statement in the output. For example, “Finishedbackup at 03-JUL-14”.
Note
Creating the Standby Control File and PFILE for Standby Server
Step 1 Return to the sqlplus prompt of the primary database and create a control file for the standby database using the followingcommand on the primary database.
Example:SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/tmp/pmgdb_stby2.ctl';Database altered.
Step 2 Create a parameter file for the standby database.
Example:SQL> CREATE PFILE='/tmp/initPMGDB_stby2.ora' FROM SPFILE;File created.
Step 3 Modify the PFILE “/tmp/initPMGDB_stby2.ora” making the entries relevant for the standby database. Because this is areplica of the original server, only the following parameters need to be modified, added, or updated. Modify the parameterif it exists otherwise add the parameter with specified value.
High Availability for Cisco RAN Management Systems26
Configuring High Availability for the PMG DBSetting Up the Hot Standby
Example:*.db_unique_name='PMGDB_STBY2'*.fal_server='PMGDB'*.log_archive_dest_2='SERVICE=PMGDB ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)DB_UNIQUE_NAME=PMGDB'*.log_archive_config='DG_CONFIG=(PMGDB,PMGDB_STBY2)'*.db_file_name_convert='PMGDB_STBY2','PMGDB'*.log_file_name_convert='PMGDB_STBY2','PMGDB'Also, remove the following parameters:
Example:*.log_archive_dest_3='ENABLE'*.log_archive_dest_state_3='ENABLE'
Configuring the Standby Server
Step 1 Log in to the standby 2 server (to be used as cold standby) with the username “oracle”.Step 2 Add tnsnames entries to "$ORACLE_HOME/network/admin/tnsnames.ora".
Example:PMGDB =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = <primary-server-host-address>)(PORT = <port number>))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PMGDB)
))
PMGDB_STBY2 =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = <standby2-server-host-address>)(PORT = <port number>))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PMGDB)
))
Step 3 Edit "$ORACLE_HOME/network/admin/listener.ora” to add a entry that the Data Guard Broker will refer to.
Example:SID_LIST_LISTENER =
(SID_DESC =(GLOBAL_NAME = PMGDB_STBY2_DGMGRL)(ORACLE_HOME = <oracle-home-directory-path>)(SID_NAME = PMGDB)
))
High Availability for Cisco RAN Management Systems 27
Configuring High Availability for the PMG DBSetting Up the Hot Standby
Copying Files to the Standby Server
Step 1 Create the necessary directories on the standby server.
Example:$ mkdir -p /u01/app/oracle/oradata/PMGDBThe system responds with a command prompt.
$ mkdir -p /u01/app/oracle/flash_recovery_area/PMGDBThe system responds with a command prompt.
$ mkdir -p /u01/app/oracle/admin/PMGDB/adumpThe system responds with a command prompt.
Step 2 Copy the files from the primary to the standby server. Copy the standby control file to all locations.
Example:$ scp oracle@<primary-server-host-address>:/tmp/pmgdb_stby.ctl/u01/app/oracle/oradata/PMGDB/control01.ctl<scp-output>
$ cp /u01/app/oracle/oradata/PMGDB/control01.ctl/u01/app/oracle/flash_recovery_area/PMGDB/control02.ctlThe system responds with a command prompt.
Step 3 Copy archivelogs and backups.
Example:$ scp -r oracle@<primary-server-host-address>:/u01/app/oracle/flash_recovery_area/PMGDB/archivelog /u01/app/oracle/flash_recovery_area/PMGDB<scp-output>
$ scp -r oracle@<primary-server-host-address>:/u01/app/oracle/flash_recovery_area/PMGDB/backupset/u01/app/oracle/flash_recovery_area/PMGDB<scp-output>
Step 4 Copy the parameter file.
Example:$ scp oracle@<primary-server-host-address>:/tmp/initPMGDB_stby.ora /tmp/initPMGDB_stby.ora<scp-output>
Step 5 Copy the remote login password file.
Example:$ scp oracle@<primary-server-host-address>:$ORACLE_HOME/dbs/orapwPMGDB $ORACLE_HOME/dbs<scp-output>
Starting the Listener on Standby ServerEnsure that the listener is started on the standby server.
High Availability for Cisco RAN Management Systems28
Configuring High Availability for the PMG DBSetting Up the Hot Standby
All of the output is not displayed.Note
$ lsnrctl status
If not running:
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 02-JUL-2014 09:35:37Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<primary-server-host-address>)(PORT=<port number>))TNS-12541: TNS:no listenerTNS-12560: TNS:protocol adapter errorTNS-00511: No listenerLinux Error: 111: Connection refused
If running:
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 02-JUL-2014 09:43:23Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<primary-server-host-address>)(PORT=<port number>)))STATUS of the LISTENER------------------------…Services Summary...……The command completed successfully
1 If listener is not running (that is, TNS-12541: TNS:no listener), start it.$ lsnrctl startLSNRCTL for Linux: Version 11.2.0.1.0 - Production on 17-JUN-2014 19:10:31Copyright (c) 1991, 2009, Oracle. All rights reserved.Starting /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr:please wait...……The command completed successfully
2 If listener is running, reload it.$ lsnrctl reloadLSNRCTL for Linux: Version 11.2.0.1.0 - Production on 17-JUN-2014 19:12:23Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to…The command completed successfully
Restoring Backup on the Standby Server
Step 1 Create the SPFILE from the modified PFILE.
Example:$ export ORACLE_SID=PMGDBThe system responds with a command prompt.
$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 12:24:01 2014
High Availability for Cisco RAN Management Systems 29
Configuring High Availability for the PMG DBSetting Up the Hot Standby
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to an idle instance.
SQL> CREATE SPFILE FROM PFILE='/tmp/initPMGDB_stby.ora';File created.
Step 2 Exit from the sql prompt.
Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release11.2.0.1.0 - 64bit Production
Step 3 Restore the backup files. Depending on database size, restore time will vary.All of the output is notdisplayed.
Note
Example:$ export ORACLE_SID=PMGDB$ rman target=/Recovery Manager: Release 11.2.0.1.0 - Production on Thu Jul 3 19:23:50 2014
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
connected to target database (not started)
RMAN> STARTUP MOUNT;Oracle instance starteddatabase mounted
Total System Global Area 3340451840 bytes
Fixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytes
RMAN> RESTORE DATABASE;Starting restore at 03-JUL-14Starting implicit crosscheck backup at 03-JUL-14using target database control file instead of recovery catalogallocated channel: ORA_DISK_1………channel ORA_DISK_1: piece handle=/u01/app/oracle/flash_recovery_area/PMGDB/backupset/2014_07_03/o1_mf_nnndf_TAG20140703T185812_9vbpmf2x_.bkp tag=TAG20140703T185812channel ORA_DISK_1: restored backup piece 1channel ORA_DISK_1: restore complete, elapsed time: 00:00:26Finished restore at 03-JUL-14
RMAN> exitRecovery Manager complete.
High Availability for Cisco RAN Management Systems30
Configuring High Availability for the PMG DBSetting Up the Hot Standby
Creating Redo Logs for Standby Server
Step 1 Exit from the sqlplus prompt if already logged in and re-login.
Example:SQL> exitDisconnected
$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 12:24:01 2014
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to an idle instance.
Step 2 Create online redo logs for the standby.
Example:SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=MANUAL;System altered.
SQL> ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/PMGDB/online_redo01.log') SIZE 50M;Database altered.
SQL> ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/PMGDB/online_redo02.log') SIZE 50M;Database altered.
SQL> ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/PMGDB/online_redo03.log') SIZE 50M;Database altered.
SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO;System altered.
Create standby redo logs on both the standby and the primary database (in case of switchovers). The standby redo logs shouldbe at least as big as the largest online redo log and there should be one extra group per thread compared the online redo logs.In this case, the following standby redo logs must be created on both servers.SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo01.log') SIZE 50M;Database altered.
SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo02.log') SIZE 50M;Database altered.
SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo03.log') SIZE 50M;Database altered.
SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo04.log') SIZE 50M;Database altered.
Step 3 Create standby redo logs on both the standby and the primary database (in case of switchovers). The standby redo logsshould be at least as big as the largest online redo log and there should be one extra group per thread compared the onlineredo logs. In this case, the following standby redo logs must be created on both servers.
Example:SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo01.log') SIZE 50M;Database altered.
SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo02.log') SIZE 50M;
High Availability for Cisco RAN Management Systems 31
Configuring High Availability for the PMG DBSetting Up the Hot Standby
Database altered.
SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo03.log') SIZE50M;Database altered.
SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo04.log') SIZE50M;Database altered.
Step 4 Repeat steps 1 to 3 to create standby redo logs on the primary server. After this is complete, the apply process can bestarted.
Starting the Apply Process on Standby ServerStart the apply process on the standby server.SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;Database altered.
Configuring the Data Guard Broker on Standby Server
Step 1 Start the Data Guard Broker Process on the standby server.
Example:SQL> alter system set dg_broker_start=true scope=both;System altered.
Step 2 Configure the Data Guard Broker by opening another console on the primary or standby server to invoke the DGMGRLCLI.
DGMGRLCLI can be accessed from any server where the Oracle Client Administrator is installed.Note
Example:$ dgmgrlDGMGRL for Linux: Version 11.2.0.1.0 - 64bit ProductionCopyright (c) 2000, 2009, Oracle. All rights reserved.Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys@PMGDBPassword:.Connected.
Step 3 Add the standby database to the Data Guard Broker configuration.DGMGRL> add database 'PMGDB_STBY2' as connect identifier is PMGDB_STBY2 maintainedas physical ;Database " PMGDB_STBY2 " added
Step 4 Update the primary database property and enable the database.
Example:DGMGRL> edit database 'PMGDB' set property DbFileNameConvert='PMGDB,PMGDB_STBY,PMGDB,PMGDB_STBY2';Property "dbfilenameconvert" updated
High Availability for Cisco RAN Management Systems32
Configuring High Availability for the PMG DBSetting Up the Hot Standby
DGMGRL> edit database 'PMGDB' set property LogFileNameConvert='PMGDB,PMGDB_STBY,PMGDB,PMGDB_STBY2';Property "logfilenameconvert" updated
DGMGRL> enable database 'PMGDB';Enabled.
Step 5 Enable the standby database.
Example:DGMGRL> enable database 'PMGDB_STBY2';Enabled.
Step 6 Verify the configuration.
Example:DGMGRL> show configurationConfiguration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby databasePMGDB_STBY2 - Physical standby database
Fast-Start Failover: ENABLEDConfiguration Status:SUCCESS
Verifying Log Synchronization on Standby Server
Step 1 Identify the existing archived redo log files on the standby server. Note the latest sequence number.Sequence number and timestamp in the output will vary depending on the installationsetup.
Note
Example:SQL> ALTER SESSION SET nls_date_format='DD-MON-YYYY HH24:MI:SS';Session altered.
SQL> SELECT sequence#, first_time, next_time, appliedFROM v$archived_logORDER BY sequence#;SEQUENCE# FIRST_TIME NEXT_TIME APPLIED---------- -------------------- -------------------- ---------
41 10-JUL-2014 09:06:27 10-JUL-2014 09:06:53 YES42 10-JUL-2014 09:06:53 10-JUL-2014 09:36:09 YES43 10-JUL-2014 09:36:09 10-JUL-2014 09:51:38 IN-MEMORY
Step 2 Force a log switch to archive the current online redo log file on the primary server.
Example:SQL> ALTER SYSTEM SWITCH LOGFILE;System altered.
Step 3 Verify on the standby, the new redo data that was archived on the standby database. That is, new sequence number withlatest timestamp of redo apply is displayed.
High Availability for Cisco RAN Management Systems 33
Configuring High Availability for the PMG DBSetting Up the Hot Standby
The value of the APPLIED column for the most recently received log file will either be IN-MEMORY or YESif that log file has been applied.
Note
Example:SQL> SELECT sequence#, first_time, next_time, appliedFROM v$archived_logORDER BY sequence#;SEQUENCE# FIRST_TIME NEXT_TIME APPLIED---------- -------------------- -------------------- ---------
41 10-JUL-2014 09:06:27 10-JUL-2014 09:06:53 YES42 10-JUL-2014 09:06:53 10-JUL-2014 09:36:09 YES43 10-JUL-2014 09:36:09 10-JUL-2014 09:51:38 YES44 10-JUL-2014 09:51:38 10-JUL-2014 10:03:48 IN-MEMORY
Enabling Flashback on Standby Server
Follow the steps in Enabling Flashback, on page 20 to enable flashback. Read the standby server as coldstandby server.
Configuring Cold StandbyCold standby refers to a standby server that is made active and switched as a primary when the original primaryserver or site fails due to any unforeseen event. The switchover needs manual intervention and there is somedowntime involved. Typically cold standby servers are placed in a different site other than the primary serversite. There are two options available to configure cold standby:
• Configuring Primary With Only Cold Standby, on page 34
• Configuring Primary With Hot and Cold Standby, on page 34
Configuring Primary With Only Cold StandbyEnsure that you complete the Configuring the Standby Server Setup, on page 4 to create a standby server.No additional steps are required to configure for primary with only cold standby setup.
Configuring Primary With Hot and Cold Standby
Setting up Hot Standby
Complete the Configuring Hot Standby for PMG DB, on page 21 to set up the hot standby server.
Setting Up Additional Standby Server as Cold Standby
This will be an additional standby database added to the primary and should be used as a cold standby serverfor disaster recovery such as site failure recovery. Following configuration steps mentioned assume hot standbyis already created.
High Availability for Cisco RAN Management Systems34
Configuring High Availability for the PMG DBConfiguring Cold Standby
Testing Hot StandbyYou can test the following two hot standby processes for the database:
• Testing Failover Process, on page 35
• Testing Switchover Process, on page 42
Testing Failover ProcessTo test the following failover processes, follow these procedures.
• Testing Failover From Primary Database to Standby Database, on page 35
• Testing Failover Revert From New Primary to Original Primary Database, on page 39
Testing Failover From Primary Database to Standby Database
Step 1 Log in to the primary server with the username “oracle”. If you are already logged in, proceed to Step 2.Step 2 Shutdown the primary database and check if standby is implemented.
Example:SQL> select db_unique_name from v$database;DB_UNIQUE_NAME------------------------------PMGDB
SQL> shut abortORACLE instance shut down.
Step 3 Check alert logs on the standby server.TheOracle base directory pathmay vary based on your installation.Note
Example:$ cd /u01/app/oracle/diag/rdbms/pmgdb_stby/PMGDB/trace$ tail -f alert_PMGDB.log -n100…Failover succeeded. Primary database is now PMGDB_STBY.……
Step 4 Check the log status.
Example:SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database;DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS
FSFO_CURR_TARGET------------------------------ ---------------- ----------- -------------------- ----------------------------------------------------
High Availability for Cisco RAN Management Systems 35
Configuring High Availability for the PMG DBTesting Hot Standby
PMGDB_STBY PRIMARY 1204588 READ WRITE REINSTATE REQUIREDPMGDB
Step 5 Synchronize failure status by starting up and mounting the database on the original primary server.
Example:SQL> startup mountORACLE instance started.
Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.
Step 6 Open a new console on the original primary server, and keep tailing the log file drcPMGDB.log. Wait until the tail outputstops at "DMON: >> DMON Process Shutdown <<".
Example:$cd /u01/app/oracle/diag/rdbms/pmgdb/PMGDB/trace$tail -f drcPMGDB.log<tail output>……014-07-03 21:41:43.825 01001000 787692441 DMON: status from posting instances for Database QUIESCE= ORA-000002014-07-03 21:41:43.825 INSV: Received message for inter-instance publication2014-07-03 21:41:43.825 req ID 1.1.787692441, opcode CTL_QUIESCE, phaseTEARDOWN, flags 10052014-07-03 21:41:43.825 DMON: Releasing (convert to NULL) Health Check Masterlock2014-07-03 21:41:43.825 DMON: Releasing FSFP HOME lock2014-07-03 21:41:43.826 INSV: Reply received for message with2014-07-03 21:41:43.826 req ID 1.1.787692441, opcode CTL_QUIESCE, phaseTEARDOWN2014-07-03 21:41:43.826 01001000 787692441 DMON: Entered rfm_release_chief_lock() for CTL_QUIESCE2014-07-03 21:41:45.827 Fore: FSFO shutting down2014-07-03 21:41:49.879 DMON: Data Guard Broker shutting down2014-07-03 21:41:49.879 DMON: Terminating RSM processes2014-07-03 21:41:49.880 RSM0: delete state object for RSM02014-07-03 21:41:50.879 DMON: RSM0 successfully terminated2014-07-03 21:41:50.880 DMON: Terminating NetSlave processes2014-07-03 21:41:50.880 DMON: Freeing all task elements.2014-07-03 21:41:50.880 DMON: Terminating Instance Slave process2014-07-03 21:41:50.880 INSV: Shutting down2014-07-03 21:41:51.880 DMON: INSV successfully terminated2014-07-03 21:41:51.881 DMON: Zeroing metadata root pointer2014-07-03 21:41:51.881 DMON: Clearing Primary State2014-07-03 21:41:51.881 DMON: Freeing Broker SGA heap2014-07-03 21:41:51.881 DMON: Freeing PGA heap2014-07-03 21:41:51.881 DMON: Removing DMON's state object2014-07-03 21:41:51.881 DMON: Resetting DMON context structure2014-07-03 21:41:51.881 DMON: >> DMON Process Shutdown <<
Step 7 Return to the sql prompt of the original primary server, exit the sql prompt, relogin to the sql prompt, and redo the startupmount.
Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing options
$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Thu Jul 3 21:57:39 2014
Copyright (c) 1982, 2009, Oracle. All rights reserved.
High Availability for Cisco RAN Management Systems36
Configuring High Availability for the PMG DBTesting Failover Process
Connected to an idle instance.
SQL> startup mountORACLE instance started.
Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.
Step 8 Connect the DGMGRL to PMGDB_STBY, which is the primary now and reinstate the failover status.
Example:DGMGRL> connect sys@PMGDB_STBYPassword:Connected
Step 9 View the configuration. Note that the database roles are changed, however, a warning to reinstate standby is displayed.
Example:DGMGRL> show configurationIn case of primary with only hot standby:Configuration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB_STBY - Primary databaseWarning: ORA-16817: unsynchronized fast-start failover configuration
PMGDB - (*) Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated
Fast-Start Failover: ENABLED
Configuration Status:WARNING
In case of primary with hot and cold standby:Configuration - dgPMGDB
Protection Mode: MaxAvailabilityDatabases:PMGDB_STBY - Primary databaseWarning: ORA-16817: unsynchronized fast-start failover configuration
PMGDB - (*) Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated
PMGDB_STBY2 - Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated
Fast-Start Failover: ENABLED
Configuration Status:WARNING
Step 10 Reinstate the standby database.If Error: ORA-16653: failed to reinstate database is observed in the output, go to the original primary database,exit the sql prompt, re-login to the sql prompt, startup, and mount the database again.
Note
Example:DGMGRL> REINSTATE DATABASE 'PMGDB'Reinstating database "PMGDB", please wait...Reinstatement of database "PMGDB" succeeded
Step 11 View the configuration again to check if the configuration status is a "SUCCESS" and with no errors.
High Availability for Cisco RAN Management Systems 37
Configuring High Availability for the PMG DBTesting Failover Process
In case of primary with hot and cold standby, the error or warning of cold standby (PMGDB_STBY2) can beignored.
Note
Example:DGMGRL> show configurationIn case of primary with only hot standby:Configuration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB_STBY - Primary databasePMGDB - (*) Physical standby database
Fast-Start Failover: ENABLEDConfiguration Status:SUCCESS
In case of primary with hot and cold standby:Configuration - dgPMGDB
Protection Mode: MaxAvailabilityDatabases:PMGDB_STBY - Primary databasePMGDB - (*) Physical standby databasePMGDB_STBY2 - Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated
Fast-Start Failover: ENABLED
Configuration Status:SUCCESS
Step 12 Check the status on the original primary server.
Example:SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database;DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS
FSFO_CURR_TARGET------------------------------ ---------------- ----------- -------------------- ----------------------------------------------------PMGDB PHYSICAL STANDBY 1205137 MOUNTED SYNCHRONIZED
PMGDB
Step 13 Check the status on the new primary server.
Example:SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database;DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS
FSFO_CURR_TARGET------------------------------ ---------------- ----------- -------------------- ----------------------------------------------------PMGDB_STBY PRIMARY 1206642 READ WRITE SYNCHRONIZED
PMGDB
High Availability for Cisco RAN Management Systems38
Configuring High Availability for the PMG DBTesting Failover Process
Testing Failover Revert From New Primary to Original Primary Database
Step 1 Log in to the primary server with the username “oracle”. If you are already logged in, proceed to Step 2Step 2 Shutdown the new primary (original hot standby) database and check if failover reverts to the original primary database.
Example:SQL> select db_unique_name from v$database;DB_UNIQUE_NAME------------------------------PMGDB_STBY
SQL> shut abortORACLE instance shut down
Step 3 Check the alert logs on the original primary server.TheOracle base directory pathmay vary based on your installation.Note
Example:$ cd /u01/app/oracle/diag/rdbms/pmgdb/PMGDB/trace$ tail -f alert_PMGDB.log -n100…Failover succeeded. Primary database is now PMGDB.……
Step 4 Check the log status on the original primary server.
Example:SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database;DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS
FSFO_CURR_TARGET------------------------------ ---------------- ----------- -------------------- ----------------------------------------------------PMGDB PRIMARY 1204588 READ WRITEREINSTATE REQUIRED PMGDB_STBY
Step 5 Synchronize the failover status on the original hot standby server, start up, and mount the database.
Example:SQL> startup mountORACLE instance started.
Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.
Step 6 Open a new console on the original hot standby server and keep tailing the log file drcPMGDB.log. Wait until the tailoutput stops at "DMON: >> DMON Process Shutdown <<".
TheOracle base directory pathmay vary based on your installation.Note
Example:$cd /u01/app/oracle/diag/rdbms/pmgdb_stby/PMGDB/trace$tail -f drcPMGDB.log
High Availability for Cisco RAN Management Systems 39
Configuring High Availability for the PMG DBTesting Failover Process
<tail output>…… 2014-07-03 22:27:54.467 02001000 354428689 DMON: status from posting instances for DatabaseQUIESCE = ORA-000002014-07-03 22:27:54.467 DMON: Releasing (convert to NULL) Health Check Masterlock2014-07-03 22:27:54.467 DMON: Releasing FSFP HOME lock2014-07-03 22:27:54.467 INSV: Reply received for message with2014-07-03 22:27:54.468 req ID 2.1.354428689, opcode CTL_QUIESCE, phaseTEARDOWN2014-07-03 22:27:54.469 02001000 354428689 DMON: Entered rfm_release_chief_lock() for CTL_QUIESCE2014-07-03 22:27:56.485 Fore: FSFO shutting down2014-07-03 22:27:59.600 DMON: Data Guard Broker shutting down2014-07-03 22:27:59.601 DMON: Terminating RSM processes2014-07-03 22:27:59.601 RSM0: delete state object for RSM02014-07-03 22:28:00.601 DMON: RSM0 successfully terminated2014-07-03 22:28:00.601 DMON: Terminating NetSlave processes2014-07-03 22:28:00.601 DMON: Freeing all task elements.2014-07-03 22:28:00.601 DMON: Terminating Instance Slave process2014-07-03 22:28:00.602 INSV: Shutting down2014-07-03 22:28:01.602 DMON: INSV successfully terminated2014-07-03 22:28:01.602 DMON: Zeroing metadata root pointer2014-07-03 22:28:01.602 DMON: Clearing Primary State2014-07-03 22:28:01.602 DMON: Freeing Broker SGA heap2014-07-03 22:28:01.603 DMON: Freeing PGA heap2014-07-03 22:28:01.603 DMON: Removing DMON's state object2014-07-03 22:28:01.603 DMON: Resetting DMON context structure2014-07-03 22:28:01.603 DMON: >> DMON Process Shutdown <<
Step 7 Return to the sql prompt of the original hot standby server, exit the sql prompt, relogin to the sql prompt, and redo thestartup mount.
Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing options
$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Thu Jul 3 21:57:39 2014
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup mountORACLE instance started.
Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.
Step 8 Connect the DGMGRL to the PMGDB, which is the primary now and reinstate the failover status.
Example:DGMGRL> connect sys@PMGDBPassword:Connected.
Step 9 View the configuration. Note that the database roles are changed, however, a warning to reinstate standby is displayed.
Example:DGMGRL> show configuration
In case of primary with only hot standby:Configuration - dgPMGDB
High Availability for Cisco RAN Management Systems40
Configuring High Availability for the PMG DBTesting Failover Process
Protection Mode: MaxAvailabilityDatabases:PMGDB - Primary databaseWarning: ORA-16817: unsynchronized fast-start failover configuration
PMGDB_STBY - (*) Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated
Fast-Start Failover: ENABLED
Configuration Status:WARNING
In case of primary with hot and cold standby:Configuration - dgPMGDB
Protection Mode: MaxAvailabilityDatabases:PMGDB - Primary databaseWarning: ORA-16817: unsynchronized fast-start failover configuration
PMGDB_STBY - (*) Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated
PMGDB_STBY2 - Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated
Fast-Start Failover: ENABLED
Configuration Status:WARNING
Step 10 Reinstate the standby database.If Error: ORA-16653: failed to reinstate database is observed in the output, go to the original hot standbydatabase, exit the sql prompt, relogin to the sql prompt, start up, and mount the database again.
Note
Example:DGMGRL> REINSTATE DATABASE 'PMGDB_STBY'Reinstating database "PMGDB_STBY", please wait...Reinstatement of database "PMGDB_STBY" succeeded
Step 11 View the configuration again to check if the configuration status is a "SUCCESS" and with no errors.Note • Sometimes processes from the previous commandmay not be complete, therefore the “show configuration”
may show a different output. Allow a couple of minutes and execute the command again.
• In case of primary with hot and cold standby, an error or warning of cold standby (PMGDB_STBY2) canbe ignored.
Example:DGMGRL> show configuration
In case of primary with only hot standby:Configuration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby database
Fast-Start Failover: ENABLEDConfiguration Status:SUCCESS
In case of primary with hot and cold standby:Configuration - dgPMGDB
Protection Mode: MaxAvailabilityDatabases:
High Availability for Cisco RAN Management Systems 41
Configuring High Availability for the PMG DBTesting Failover Process
PMGDB - Primary databasePMGDB_STBY - (*) Physical standby databasePMGDB_STBY2 - Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated
Fast-Start Failover: ENABLED
Configuration Status:SUCCESS
Step 12 Check the status on the original hot standby server.
Example:SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database;DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS
FSFO_CURR_TARGET------------------------------ ---------------- ----------- -------------------- ----------------------------------------------------PMGDB_STBY PHYSICAL STANDBY 1205137 MOUNTED SYNCHRONIZED
PMGDB_STBY
Step 13 Check the status on the original primary server.
Example:SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database;DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS
FSFO_CURR_TARGET------------------------------ ---------------- ----------- -------------------- ----------------------------------------------------PMGDB PRIMARY 1206642 READ WRITESYNCHRONIZED PMGDB_STBY
Testing Switchover ProcessSwitchover from the primary to secondary or standby database and vice versa is a manual process typicallycarried out during plannedmaintenance or disaster recovery. The following procedures describe the switchoverfrom PMGDB (primary) to PMGDB_STBY (hot standby).
• Testing Switchover From Primary to Standby Database, on page 42
• Testing Switchover Revert From New Primary to Original Primary Database, on page 44
Testing Switchover From Primary to Standby Database
Step 1 Start the DGMGRL utility from the primary or standby server (whichever is up) or from another host where the Oracleclient is installed.
Example:$ dgmgrlDGMGRL for Linux: Version 11.2.0.1.0 - 64bit Production
High Availability for Cisco RAN Management Systems42
Configuring High Availability for the PMG DBTesting Switchover Process
Copyright (c) 2000, 2009, Oracle. All rights reserved.Welcome to DGMGRL, type "help" for information.
Step 2 Connect to the PMG DB.
Example:DGMGRL> connect sys@PMGDBPassword:Connected.
Step 3 Switchover to PMGDB_STBY.
Example:DGMGRL> SWITCHOVER TO 'PMGDB_STBY'Performing switchover NOW, please wait...New primary database "PMGDB_STBY" is opening...Operation requires shutdown of instance "PMGDB" on database "PMGDB"Shutting down instance "PMGDB"...ORA-01109: database not open
Database dismounted.ORACLE instance shut down.Operation requires startup of instance "PMGDB" on database "PMGDB"Starting instance "PMGDB"...Unable to connect to databaseORA-12514: TNS:listener does not currently know of service requested in connect descriptorFailed.Warning: You are no longer connected to ORACLE.
Please complete the following steps to finish switchover:start up and mount instance "PMGDB" of database "PMGDB"
Step 4 Exit from the previous sql prompt to start up and mount the PMGDB.
Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing options
Step 5 Connect to sqlplus.
Example:$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 16:50:03 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to an idle instance.
Step 6 Start up and mount the database.
Example:SQL> startup mountORACLE instance started.Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.
Step 7 Verify that the roles are switched.a) Verify on the original primary (PMGDB).
Example:SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database;
High Availability for Cisco RAN Management Systems 43
Configuring High Availability for the PMG DBTesting Switchover Process
DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE------------------------------ ---------------- ----------- --------------------FS_FAILOVER_STATUS FSFO_CURR_TARGET---------------------- ------------------------------PMGDB PHYSICAL STANDBY 1208044 MOUNTEDSYNCHRONIZED PMGDB
b) Verify on the new primary (PMGDB_STBY).
Example:SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database;DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS
FSFO_CURR_TARGET------------------------------ ---------------- ----------- ------------------------------------------ ------------------------------PMGDB_STBY PRIMARY 1222493 READ WRITE SYNCHRONIZED
PMGDB
Testing Switchover Revert From New Primary to Original Primary Database
Before You Begin
Step 1 Start the DGMGRL utility from the primary or standby server or from another host where the Oracle client is installed.
Example:$ dgmgrlDGMGRL for Linux: Version 11.2.0.1.0 - 64bit ProductionCopyright (c) 2000, 2009, Oracle. All rights reserved.Welcome to DGMGRL, type "help" for information.
Step 2 Connect to PMGDB_STBY.
Example:DGMGRL> connect sys@PMGDB_STBYPassword:Connected.
Step 3 Switchover to PMGDB.
Example:DGMGRL> SWITCHOVER TO 'PMGDB'Performing switchover NOW, please wait...New primary database "PMGDB" is opening...Operation requires shutdown of instance "PMGDB" on database "PMGDB_STBY"Shutting down instance "PMGDB"...ORA-01109: database not open
Database dismounted.ORACLE instance shut down.Operation requires startup of instance "PMGDB" on database "PMGDB_STBY"Starting instance "PMGDB"...Unable to connect to databaseORA-12514: TNS:listener does not currently know of service requested in connect descriptor
Failed.Warning: You are no longer connected to ORACLE.
High Availability for Cisco RAN Management Systems44
Configuring High Availability for the PMG DBTesting Switchover Process
Please complete the following steps to finish switchover:start up and mount instance "PMGDB" of database "PMGDB_STBY"
Step 4 Exit from previous sql prompt to start up and mount the PMGDB_STBY.
Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing options
Step 5 Connect to sqlplus.
Example:$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 16:50:03 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to an idle instance.
Step 6 Start up and mount the database.
Example:SQL> startup mountORACLE instance started.Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.
Step 7 Verify that the roles are switched.a) Verify on the hot standby (PMGDB_STBY).
Example:SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database;DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE------------------------------ ---------------- ----------- --------------------FS_FAILOVER_STATUS FSFO_CURR_TARGET---------------------- ------------------------------PMGDB_STBY PHYSICAL STANDBY 1938653 MOUNTEDSYNCHRONIZED PMGDB_STBY
b) Verify on the original primary (PMGDB).
Example:SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database;DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE------------------------------ ---------------- ----------- --------------------FS_FAILOVER_STATUS FSFO_CURR_TARGET---------------------- ------------------------------PMGDB PRIMARY 1938910 READ WRITESYNCHRONIZED PMGDB_STBY
Testing Cold StandbyUse these procedures to test cold standby on PMG DB.
High Availability for Cisco RAN Management Systems 45
Configuring High Availability for the PMG DBTesting Cold Standby
• Testing Site Failure, on page 46
• Recovering Original Primary After Site Failure, on page 48
Testing Site FailureUse the following steps to test unscheduled failure of primary and first hot standby (if available) servers. Toprevent applications (PMG, OpsTools) connection failures without any manual intervention, ensure that thePMGDB configuration script (/rms/app/rms/install/pmgdb_configure.sh) for the Central node is executedwith the standby server configuration as per the instructions in the installation guide.Usage:
pmgdb_configure.sh <Pmgdb_Enabled> <Pmgdb_Primary_Dbserver_Address><Pmgdb_Primary_Dbserver_Port> [<Pmgdb_Stby1_Dbserver_Address>][<Pmgdb_Stby1_Dbserver_Port>] [<Pmgdb_Stby2_Dbserver_Address>][<Pmgdb_Stby2_Dbserver_Port>]If not configured earlier, execute the script again with the appropriate values for database servers and ports.For more details, see the Cisco RAN Management System Installation Guide.
Step 1 To test site failure, verify if the applications connecting to the PMGDB database are able to connect. For example, theOpsTools script getAreas.sh can be executed. Log in to Central node as admin1 user and execute the script.
Example:$ getAreas.sh -key 1001Config files script-props/private/GetAreas.properties or script-props/public/GetAreas.properties notfound.Continuing with default settings.Execution parameters:
key=1001GetAreas processing can take some time please do not terminate.Received areas, total areas 1Writing to file: /home/admin1/getAreas.csvThe report captured in csv file: /home/admin1/getAreas.csv**** GetAreas End Script ***
Step 2 Log in to the hot standby server as an oracle user and shut down the database server.Perform this step only if the hot standby server is configured. Else, proceed to the nextstep.
Note
Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaSQL> shutdown immediateDatabase closed.Database dismounted.ORACLE instance shut down.
Step 3 Log in to the primary server an oracle user and shut down the database server.
Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaSQL> shutdown immediateDatabase closed.
High Availability for Cisco RAN Management Systems46
Configuring High Availability for the PMG DBTesting Site Failure
Database dismounted.ORACLE instance shut down.
Step 4 Verify that the applications connecting to the PMGDB database are failing considering that the primary and hot standbyare down. Log in to the Central node as admin1 user and execute the getAreas.sh script.
Example:$ getAreas.sh -key 1001Config files script-props/private/GetAreas.properties orscript-props/public/GetAreas.properties not found. Continuing with default settings.Execution parameters:
key=1001GetAreas processing can take some time please do not terminate.Jul 01, 2014 1:45:14 PM org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable to create initial connections of pool.....Failed to Get Areas : Error while connecting to PmgDb, java.sql.SQLException: Listener refused theconnection with the following error:ORA-12514, TNS:listener does not currently know of service requested in connect descriptor....Alternate Output:……Failed to Get Areas : Error while connecting to PmgDb, java.sql.SQLRecoverableException: Io exception:
The Network Adapter could not establish the connection
Step 5 Log in to the cold standby server as an oracle user and log in to sql prompt.
Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaConnected to database
Step 6 Stop the redo apply process.
Example:SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;Database altered.
Step 7 Finish applying all received redo data.
Example:SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH;Database altered.
SQL> ALTER DATABASE ACTIVATE PHYSICAL STANDBY DATABASE;Database altered.
Step 8 Switch the physical standby database to the primary role.
Example:SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN;ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN*ERROR at line 1:ORA-01109: database not open
Step 9 Open the new primary database. This may take some time to complete.
High Availability for Cisco RAN Management Systems 47
Configuring High Availability for the PMG DBTesting Site Failure
Example:SQL> ALTER DATABASE OPEN;Database altered.
Step 10 Check the status of the database. The switchover status could be FAILED DESTINATION or NOT ALLOWED.
Example:SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE, SWITCHOVER_STATUS from v$database;NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS--------- -------------------- ------------------------------ ---------------- --------------------PMGDB READ WRITE PMGDB_STBY2PRIMARY FAILED DESTINATION
Step 11 Verify that applications connecting to the PMGDB database are getting connected now that the cold standby is up as anew primary. Log in to the Central node as admin1 user and execute the getAreas.sh script.
Example:$ getAreas.sh -key 1001Config files script-props/private/GetAreas.properties or script-props/public/GetAreas.properties notfound. Continuing with default settings.Execution parameters:
key=1001GetAreas processing can take some time please do not terminate.Received areas, total areas 1Writing to file: /home/admin1/getAreas.csvThe report captured in csv file: /home/admin1/getAreas.csv**** GetAreas End Script ***
Recovering Original Primary After Site FailureThe database can be recovered using either the RMAN backup or Flashback database. Following steps describerecovery using RMAN backup.
Converting a Failed Primary into a Standby Database Using RMAN Backups
DETAILED STEPS
PurposeCommand or Action
Determine the SCN at which the old/cold standby database became the primary database.Step 1
Run the following query on the new primary database to determine the SCN at which the original/cold standby database became the newprimary database:
Step 2
Example:SQL> SELECT TO_CHAR(STANDBY_BECAME_PRIMARY_SCN) FROM V$DATABASE;TO_CHAR(STANDBY_BECAME_PRIMARY_SCN)----------------------------------------1351900
Restore the database with a backup taken before the original primaryhad reached the SCN at which the standby became the new primary
Restore and recover the entire database on the original primary.Step 3
High Availability for Cisco RAN Management Systems48
Configuring High Availability for the PMG DBRecovering Original Primary After Site Failure
PurposeCommand or Action
(standby_became_primary_scn). Then, perform a point-in-time recoveryto recover the original primary to that same point.
Find out the value of the <recovery_scn> from the output from theprevious command. It will be adding 1 to the value ofstandby_became_primary_scn. For example, if previous commandreturns standby_became_primary_scn value as 1351900, <recovery_scn>will be 1351900 + 1 = 1351901.
If the output of command “RMAN> RUN…” shows error“RMAN-20208”, see the Troubleshooting Data Guard on PMGDB, on page 65 to resolve the issue.
Note
Run the following RMAN commands on the original primary.Step 4
Example:$ export ORACLE_SID=PMGDB$ rman target=/Recovery Manager: Release 11.2.0.1.0 - Production on Tue Jul 8 19:30:41 2014
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
connected to target database (not started)
RMAN> startup mountOracle instance starteddatabase mounted
Total System Global Area 1068937216 bytesFixed Size 2220200 bytesVariable Size 289410904 bytesDatabase Buffers 771751936 bytesRedo Buffers 5554176 bytes
RMAN> RUN{SET UNTIL SCN <recovery_scn> ;RESTORE DATABASE;RECOVER DATABASE;}
executing command: SET until clause
Starting restore at 08-JUL-14allocated channel: ORA_DISK_1channel ORA_DISK_1: SID=11 device type=DISK
channel ORA_DISK_1: starting datafile backup set restorechannel ORA_DISK_1: specifying datafile(s) to restore from backup setchannel ORA_DISK_1: restoring datafile 00001 to /u01/app/oracle/oradata/PMGDB/system01.dbfchannel ORA_DISK_1: restoring datafile 00002 to /u01/app/oracle/oradata/PMGDB/sysaux01.dbfchannel ORA_DISK_1: restoring datafile 00003 to /u01/app/oracle/oradata/PMGDB/undotbs01.dbfchannel ORA_DISK_1: restoring datafile 00004 to /u01/app/oracle/oradata/PMGDB/users01.dbfchannel ORA_DISK_1: restoring datafile 00005 to /u01/app/oracle/oradata/PMGDB/mapinfo_ts.dbfchannel ORA_DISK_1: restoring datafile 00006 to /u01/app/oracle/oradata/PMGDB/pmgdb_ts.dbfchannel ORA_DISK_1: reading from backup piece/u01/app/oracle/flash_recovery_area/PMGDB/backupset/2014_07_08/o1_mf_nnndf_TAG20140708T162646_9vqmmgsv_.bkpchannel ORA_DISK_1: piecehandle=/u01/app/oracle/flash_recovery_area/PMGDB/backupset/2014_07_08/o1_mf_nnndf_TAG20140708T162646_9vqmmgsv_.bkptag=TAG20140708T162646channel ORA_DISK_1: restored backup piece 1channel ORA_DISK_1: restore complete, elapsed time: 00:00:25
High Availability for Cisco RAN Management Systems 49
Configuring High Availability for the PMG DBRecovering Original Primary After Site Failure
PurposeCommand or Action
Finished restore at 08-JUL-14
Starting recover at 08-JUL-14using channel ORA_DISK_1
starting media recovery
archived log for thread 1 with sequence 24 is already on disk as file/u01/app/oracle/flash_recovery_area/PMGDB/archivelog/2014_07_08/o1_mf_1_24_9vqmn043_.arcarchived log for thread 1 with sequence 25 is already on disk as file/u01/app/oracle/flash_recovery_area/PMGDB/archivelog/2014_07_08/o1_mf_1_25_9vqv5j39_.arcarchived log for thread 1 with sequence 26 is already on disk as file/u01/app/oracle/flash_recovery_area/PMGDB/archivelog/2014_07_08/o1_mf_1_26_9vqw8v1v_.arcarchived log file name=/u01/app/oracle/flash_recovery_area/PMGDB/archivelog/2014_07_08/o1_mf_1_24_9vqmn043_.arcthread=1 sequence=24media recovery complete, elapsed time: 00:00:00Finished recover at 08-JUL-14
RMAN> exitRecovery Manager complete.
Convert original primary database to a physical standby database. Perform the following steps on the original primary database.Step 5
Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 17:51:51 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL>
SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY;Database altered.
Shut down and start up the original primary.Step 6
Example:SQL> SHUTDOWN IMMEDIATE;ORA-01507: database not mounted
ORACLE instance shut down.
SQL> STARTUP MOUNT;ORACLE instance started.
Total System Global Area 1068937216 bytesFixed Size 2220200 bytesVariable Size 289410904 bytesDatabase Buffers 771751936 bytesRedo Buffers 5554176 bytesDatabase mounted.
The goal of this step is to synchronize the control file with the databaseby using a dictionary check. After this command, check the alert log
Open the database as read-only.
Example:SQL> ALTER DATABASE OPEN READ ONLY;Database altered.
Step 7
(/u01/app/oracle/diag/rdbms/pmgdb/PMGDB/trace/alert_PMGDB.log)for any actions suggested by the dictionary check. Typically, no useraction is needed if the original primary was not in the middle of addingor dropping data files during the failover.
High Availability for Cisco RAN Management Systems50
Configuring High Availability for the PMG DBRecovering Original Primary After Site Failure
PurposeCommand or Action
The log directory may vary based on the installationsetup.
Note
Mount the new standby (original primary) again.Step 8
Example:SQL> SHUTDOWN IMMEDIATE;Database closed.Database dismounted.ORACLE instance shut down.
SQL> STARTUP MOUNT;ORACLE instance started.
Total System Global Area 1068937216 bytesFixed Size 2220200 bytesVariable Size 289410904 bytesDatabase Buffers 771751936 bytesRedo Buffers 5554176 bytesDatabase mounted.
Log in to the sql prompt on the new primary (old cold standby) and make sure the archive destination is enabled.Step 9
Example:SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;System altered.
Now the failed primary database is restored and is running in the standbyrole.
Start Redo Apply on the new standby (original primary) database.
Example:SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;Database altered.
Step 10
Verify status, it could be SWITCHOVER PENDING or SWITCHOVER LATENT or TO PRIMARY.Step 11
Example:SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE;SWITCHOVER_STATUS-----------------SWITCHOVER PENDING
Verify the status on the new primary as well. A value of TO STANDBY or SESSIONS ACTIVE indicates that the primary database canbe switched to the standby role.
Step 12
Example:SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE;SWITCHOVER_STATUS-----------------TO STANDBY
High Availability for Cisco RAN Management Systems 51
Configuring High Availability for the PMG DBRecovering Original Primary After Site Failure
Converting Failed Primary Into a Standby Database Using RMAN Backups
DETAILED STEPS
PurposeCommand or Action
Issue the following SQL statement on the primary database to switch it to thestandby role:
Step 1
Example:SQL> select DATABASE_ROLE from v$database;DATABASE_ROLE----------------PRIMARY
SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITHSESSION SHUTDOWN;Database altered.
Shut down and then mount this current primary database.Step 2
Example:SQL> SHUTDOWN IMMEDIATE;ORA-01507: database not mounted
ORACLE instance shut down.
SQL> STARTUP MOUNT;ORACLE instance started.Total System Global Area 1068937216 bytesFixed Size 2220200 bytesVariable Size 289410904 bytesDatabase Buffers 771751936 bytesRedo Buffers 5554176 bytesDatabase mounted.
Check the switchover status on the current primary which now has become coldstandby again. A value of TO PRIMARY or SESSIONS ACTIVE indicates that
Step 3
the standby database is ready. If neither of these values is returned, continue toquery this column until the value returned is either TO PRIMARY or SESSIONSACTIVE.
Example:SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE;SWITCHOVER_STATUS-----------------TO_PRIMARY
Return to the original primary to revert its database role to the primary role.Step 4
Example:SQL> RECOVER MANAGED STANDBY DATABASE FINISH;Media recovery complete.
SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSIONSHUTDOWN;Database altered.
High Availability for Cisco RAN Management Systems52
Configuring High Availability for the PMG DBRecovering Original Primary After Site Failure
PurposeCommand or Action
Now open the original primary database.Step 5
Example:SQL> ALTER DATABASE OPEN;Database opened.
Ignore and continue if error"ORA-01153: an incompatiblemedia recovery is active" isdisplayed.
NoteReturn to the new physical standby (cold standby) database and start redo apply.
Example:SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENTLOGFILE DISCONNECT FROM SESSION;Database altered.
Step 6
On original primary:Check status on the original primary and cold standby tomake sure that the databaserole is correct.
Step 7
On cold standby:
Example:SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE,SWITCHOVER_STATUS from v$database;NAME OPEN_MODE DB_UNIQUE_NAMEDATABASE_ROLE--------- -------------------- ----------------------------------------------SWITCHOVER_STATUS--------------------PMGDB READ WRITE PMGDB PRIMARYFAILED DESTINATION
Example:SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE,SWITCHOVER_STATUS from v$database;NAME OPEN_MODE DB_UNIQUE_NAMEDATABASE_ROLE--------- -------------------- ----------------------------------------------SWITCHOVER_STATUS--------------------PMGDB MOUNTED PMGDB_STBY2 PHYSICALSTANDBYSESSIONS ACTIVE
Start up and mount the hot standby if hot standby is present in the setup.Step 8
Example:$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Thu Jul 10 11:39:54 2014
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup mountORACLE instance started.
Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1845495840 bytesDatabase Buffers 1476395008 bytes
High Availability for Cisco RAN Management Systems 53
Configuring High Availability for the PMG DBRecovering Original Primary After Site Failure
PurposeCommand or Action
Redo Buffers 16343040 bytesDatabase mounted.
The status should be a SUCCESS,which indicates reverting to the original
Verify the dgbroker status, log in to DGMGRL either from primary or standby.Note that in case of hot and cold standby to primary setup,WARNING is displayed
Step 9
roles is complete for hot and coldstandby to primary setup.
as hot standby server is not enabled. For only cold standby to primary setup, itshould show SUCCESS which means reverting to the original role is complete.
Example:$ dgmgrlDGMGRL for Linux: Version 11.2.0.1.0 - 64bit Production
Note • If the output of commandshows ERROR orWARNING status, seethe TroubleshootingDataGuard on PMG DB, onpage 65 to resolve theissues.
• Sometimes processesfrom the previouscommand may not becomplete, therefore the“show configuration”command may showdifferent output. Allow acouple of minutes andexecute the commandagain.
In case of primary and cold standbysetup:
Copyright (c) 2000, 2009, Oracle. All rights reserved.Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys@PMGDBPassword:.Connected.
DGMGRL> show configuration
Example:Configuration - dgPMGDBProtection Mode : MaxPerformanceDatabases : PMGDB - Primary database
: PMGDB_STBY - Physical standby databaseFast-Start Failover : DISABLEDConfiguration Status : SUCCESS
Example:Configuration - dgPMGDBProtection Mode: MaxAvailability In case of primary hot and cold standby
setup:Databases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby databasePMGDB_STBY2 - Physical standby database
Fast-Start Failover: ENABLEDConfiguration Status:SUCCESS
In case of primary hot and cold standbysetup:
Enable cold standby if cold standby (PMGDB_STBY2) is showing disabled orwith warning.
Step 10
Example:DGMGRL> enable database 'PMGDB_STBY2'Enabled.
If the output of commandshows ERRORorWARNINGstatus, see the TroubleshootingData Guard on PMG DB, onpage 65 to resolve the issues.
Note
DGMGRL> show configuration
Example:Configuration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby databasePMGDB_STBY2 - Physical standby database
Fast-Start Failover: ENABLED
High Availability for Cisco RAN Management Systems54
Configuring High Availability for the PMG DBRecovering Original Primary After Site Failure
PurposeCommand or Action
Configuration Status:SUCCESS
Rolling Back and Cleaning Up Standby and PrimaryConfigurations
This section covers common scenarios for rollback. For specific scenarios, like database corruption and soon, it is recommended that you refer to standard DBA practices or Oracle documentation.
Before starting the rollback, it is recommended that the primary database is backed up.Note
The following procedures describe how to roll back the standby and primary configurations.
• Removing Data Guard Broker Configuration, on page 55
• Removing the Standby Database, on page 60
• Removing the Additional Standby Database, on page 61
Removing Data Guard Broker Configuration
Step 1 Connect to the primary database from DGMGRL.
Example:DGMGRL> connect sys@PMGDBPassword:Connected.
Step 2 Disable FAST_START FAILOVER.Perform this step only if the setup has hot standbyconfigured.
Note
Example:DGMGRL> DISABLE FAST_START FAILOVER FORCEDisabled.
Step 3 Change the protection mode.Perform this step only if the setup has hot standbyconfigured.
Note
High Availability for Cisco RAN Management Systems 55
Configuring High Availability for the PMG DBRolling Back and Cleaning Up Standby and Primary Configurations
Example:DGMGRL> edit configuration set protection mode as MaxPerformance;Succeeded.
Step 4 Remove the configuration.
Example:DGMGRL> remove configurationRemoved configuration
Step 5 Check if the configuration is removed.
Example:DGMGRL> show configurationORA-16532: Data Guard broker configuration does not exist
Configuration details cannot be determined by DGMGRL
Alternate OutputORA-16596: database not part of the Data Guard broker configuration
Configuration details cannot be determined by DGMGRL
Removing Configuration Files from Primary Server
Step 1 Log in to the primary server as an oracle user and connect to sqlplus.
Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Wed Jul 2 20:17:31 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL>
Step 2 Stop the Data Guard Broker process.
Example:SQL> alter system set dg_broker_start=FALSE;System altered.
Step 3 Check the path of the Data Guard Broker configuration files.
Example:SQL> select name , value from v$parameter where name like '%dg_broker%';NAME--------------------------------------------------------------------------------VALUE------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------dg_broker_startFALSE
dg_broker_config_file1
High Availability for Cisco RAN Management Systems56
Configuring High Availability for the PMG DBRemoving Data Guard Broker Configuration
/u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1PMGDB.dat
dg_broker_config_file2/u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2PMGDB.dat
Step 4 Exit from the sql prompt.
Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
Step 5 Remove dg_broker_config_file1 and dg_broker_config_file2.TheOracle base directory pathmay vary based on your installation.Note
Example:$ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1PMGDB.dat
$ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2PMGDB.datThe system responds with a command prompt.
Step 6 Remove the standby control files if they still exist in /tmp directory. Ignore the error if the standby control files do notexist.
Example:$ rm /tmp/pmgdb_stby.ctl
$ rm /tmp/pmgdb_stby2.ctlThe system responds with a command prompt.
Removing Configuration Files From Standby Server
Note • If the setup is only cold standby added to the primary, then this standby is cold standby.
• If the setup is only hot standby added to the primary, then this standby is hot standby.
• If the setup is both hot and cold standby added to the primary, this standby is hot standby.
DETAILED STEPS
PurposeCommand or Action
Log in to the first standby server as an oracle user and connect to sqlplus.Step 1
Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Wed Jul 2 20:17:31 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL
High Availability for Cisco RAN Management Systems 57
Configuring High Availability for the PMG DBRemoving Data Guard Broker Configuration
PurposeCommand or Action
Stop the redo apply process.Step 2
Example:SQL> alter database recover managed standby database cancel;Database altered.
Stop Data Guard Broker process.Step 3
Example:SQL> alter system set dg_broker_start=FALSE;System altered.
Check the path of the Data Guard Broker configuration files.Step 4
Example:SQL> select name , value from v$parameter where name like '%dg_broker%';NAME--------------------------------------------------------------------------------VALUE------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------dg_broker_startFALSE
dg_broker_config_file1/u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1PMGDB_STBY.dat
dg_broker_config_file2/u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2PMGDB_STBY.dat
Open the database in read-only mode.Step 5
Example:SQL> alter database open read only;alter database open read only;
Exit from the sql prompt.Step 6
Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
TheOraclebasedirectorypathmayvarybasedon yourinstallation.
NoteRemove the dg_broker_config_file1 and dg_broker_config_file2.
Example:$ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1PMGDB_STBY.datThe system responds with a command prompt.
$ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2PMGDB_STBY.datThe system responds with a command prompt.
Step 7
High Availability for Cisco RAN Management Systems58
Configuring High Availability for the PMG DBRemoving Data Guard Broker Configuration
Removing Configuration Files From Additional Standby Server
This section is applicable only to the setup with both hot and cold standby added to primary. This additionalstandby is the cold standby in the setup.
Note
DETAILED STEPS
PurposeCommand or Action
Log in to the standby server as an oracle user and connect to sqlplus.Step 1
Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Wed Jul 2 20:17:31 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL>
Stop the redo apply process.Step 2
Example:SQL> alter database recover managed standby database cancel;Database altered.
Stop the Data Guard Broker process.Step 3
Example:SQL> alter system set dg_broker_start=FALSE;System altered.
Check the path of the Data Guard Broker configuration files.Step 4
Example:SQL> select name , value from v$parameter where name like '%dg_broker%';NAME--------------------------------------------------------------------------------VALUE------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------dg_broker_startFALSE
dg_broker_config_file1/u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1PMGDB_STBY2.dat
dg_broker_config_file2/u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2PMGDB_STBY2.dat
Open the database in read-only mode.Step 5
Example:SQL> alter database open read only;alter database open read only;
Exit from the sql promptStep 6
High Availability for Cisco RAN Management Systems 59
Configuring High Availability for the PMG DBRemoving Data Guard Broker Configuration
PurposeCommand or Action
Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
TheOraclebasedirectorypathmayvarybasedon yourinstallation.
NoteRemove the dg_broker_config_file1 and dg_broker_config_file2.
Example:$ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1PMGDB_STBY2.datThe system responds with a command prompt.
$ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2PMGDB_STBY2.datThe system responds with a command prompt.
Step 7
Removing the Standby Database
Note • If the setup is only cold standby added to the primary, then this standby is cold standby.
• If the setup is only hot standby added to the primary, then this standby is hot standby.
• If the setup is both hot and cold standby added to the primary, this standby is hot standby.
Step 1 From first standby server console, execute the dbca command.
Example:$ dbca -silent -deleteDatabase -sourceDB PMGDBConnecting to database…100% completeLook at the log file "/u01/app/oracle/cfgtoollogs/dbca/PMGDB.log" for further details.
Step 2 Move the respective directories as backup directories.TheOracle base directory pathmay vary based on your installation.Note
Example:$ mv /u01/app/oracle/oradata/PMGDB /u01/app/oracle/oradata/PMGDB-bakThe system responds with a command prompt.
$ mv /u01/app/oracle/flash_recovery_area/PMGDB /u01/app/oracle/flash_recovery_area/PMGDB-bakThe system responds with a command prompt.
$ mv /u01/app/oracle/flash_recovery_area/PMGDB_STBY /u01/app/oracle/flash_recovery_area/PMGDB-STBY-bakThe system responds with a command prompt.
High Availability for Cisco RAN Management Systems60
Configuring High Availability for the PMG DBRemoving the Standby Database
$ mv /u01/app/oracle/admin/PMGDB /u01/app/oracle/admin/PMGDB-bakThe system responds with a command prompt.
Removing the Additional Standby DatabaseThis section is applicable only to the setup with both hot and cold standby added to primary. This additionalstandby is the cold standby in this setup.
DETAILED STEPS
PurposeCommand or Action
From the standby server console, execute the dbca command.Step 1
Example:$ dbca -silent -deleteDatabase -sourceDB PMGDBConnecting to database…100% completeLook at the log file "/u01/app/oracle/cfgtoollogs/dbca/PMGDB.log" for further details.
Move the respective directories as backup directories.Step 2
Example:$ mv /u01/app/oracle/oradata/PMGDB /u01/app/oracle/oradata/PMGDB-bakThe system responds with a command prompt.
$ mv /u01/app/oracle/flash_recovery_area/PMGDB /u01/app/oracle/flash_recovery_area/PMGDB-bakThe system responds with a command prompt.
$ mv /u01/app/oracle/flash_recovery_area/PMGDB_STBY2/u01/app/oracle/flash_recovery_area/PMGDB-STBY2-bakThe system responds with a command prompt.
$ mv /u01/app/oracle/admin/PMGDB /u01/app/oracle/admin/PMGDB-bakThe system responds with a command prompt.
Cleaning Up the Primary Database
Cleaning Up the Redo Log Files
Step 1 Drop the standby redo log files from the primary database.
Example:SQL> ALTER DATABASE DROP STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo01.log');Database altered.
High Availability for Cisco RAN Management Systems 61
Configuring High Availability for the PMG DBCleaning Up the Primary Database
SQL> ALTER DATABASE DROP STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo02.log');Database altered.
SQL> ALTER DATABASE DROP STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo03.log');Database altered.
SQL> ALTER DATABASE DROP STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo04.log');Database altered.
Step 2 Remove the standby redo log files from the directory.
Example:$ rm /u01/app/oracle/oradata/PMGDB/standby_redo01.logThe system responds with a command prompt.
$ rm /u01/app/oracle/oradata/PMGDB/standby_redo02.logThe system responds with a command prompt.
$ rm /u01/app/oracle/oradata/PMGDB/standby_redo03.logThe system responds with a command prompt.
$ rm /u01/app/oracle/oradata/PMGDB/standby_redo04.logThe system responds with a command prompt.
Cleaning Up Initialization ParametersThe initialization parameters can be reset using either one of these methods:
Note • (Recommended) Create spfile from backup pfile that was generated during initial steps. For moreinformation, see Using the Backup Pfile, on page 62.
• If the backup pfile is not available, the parameters can be reset using sql statements. For moreinformation, see Using SQL Statements, on page 63.
Using the Backup Pfile
Use this option to restore the spfile from the backup pfile.
Step 1 Shutdown the database.
Example:SQL> shutdown immediateDatabase closed.Database dismounted.ORACLE instance shut down.
Step 2 From the sql prompt, create the spfile from the backup pfile.f the specific path for the initPMGDB_pre_dgsetup.ora file (for example, /backup/initPMGDB_pre_dgsetup.ora)was specified when the file was backed up during the initial steps, specify the full path (for example,/backup/initPMGDB_pre_dgsetup.ora). If the path is not specified, the file will be picked up from the defaultdirectory, that is, $ORACLE_HOME/dbs.
Note
High Availability for Cisco RAN Management Systems62
Configuring High Availability for the PMG DBCleaning Up the Primary Database
Example:SQL> CREATE SPFILE FROM PFILE='initPMGDB_pre_dgsetup.ora';File created.
Step 3 Start up the database.
Example:SQL> startupORACLE instance started.
Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.Database opened.
Using SQL Statements
If the backup pfile is not available, the parameters can be reset through sql statements. This section can beskipped if spfile is already restored from the backup pfile in the previous section.
From the sql prompt, reset the initialization parameters.SQL> ALTER SYSTEM RESET LOG_ARCHIVE_CONFIG;System altered.
SQL> ALTER SYSTEM RESET LOG_ARCHIVE_DEST_2;System altered.
SQL> ALTER SYSTEM RESET LOG_ARCHIVE_DEST_STATE_2;System altered.
SQL> ALTER SYSTEM RESET LOG_ARCHIVE_FORMAT SCOPE=SPFILE;System altered.
SQL> ALTER SYSTEM RESET LOG_ARCHIVE_MAX_PROCESSES;System altered.
SQL> ALTER SYSTEM RESET FAL_SERVER;System altered.
SQL> ALTER SYSTEM RESET DB_FILE_NAME_CONVERT SCOPE=SPFILE;System altered.
SQL> ALTER SYSTEM RESET LOG_FILE_NAME_CONVERT SCOPE=SPFILE;System altered.
SQL> ALTER SYSTEM RESET LOG_ARCHIVE_DEST_3;System altered.
SQL> ALTER SYSTEM RESET LOG_ARCHIVE_DEST_STATE_3;System altered.
High Availability for Cisco RAN Management Systems 63
Configuring High Availability for the PMG DBCleaning Up the Primary Database
Verifying the Database
Before You Begin
Step 1 From the sql prompt, check if the database role is ‘Primary’.
Example:SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE, SWITCHOVER_STATUS from v$database;NAME TYPE VALUE------------------------------------ ----------- ------------------------------service_names string PMGDB_PRIMARY
Step 2 Ensure that the database service PMGDB_PRIMARY is present.
Example:SQL> show parameter serviceNAME TYPE VALUE------------------------------------ ----------- ------------------------------service_names string PMGDB_PRIMARY
Step 3 Verify that applications connecting to the PMGDB database are getting connected. For example, the OpsTools scriptgetAreas.sh can be executed. Log in to the Central node as admin1 user and execute the script.
Example:$ getAreas.sh -key 1001Config files script-props/private/GetAreas.properties or script-props/public/GetAreas.properties notfound.Continuing with default settings.Execution parameters:
key=1001GetAreas processing can take some time please do not terminate.Received areas, total areas 1Writing to file: /home/admin1/getAreas.csvThe report captured in csv file: /home/admin1/getAreas.csv**** GetAreas End Script ***
Recreating Standby ServersNow that the standby server and primary server configurations are cleaned up, reconfigure the Data GuardBroker. For more information, see the “Setting Up the Oracle Data Guard Broker” procedure on page 58.
Deleting the Primary DatabaseThe primary database is now rolled back to previous state using steps in the previous sections. However, ifthere is a need to clean up all configurations, including the primary database, ensure that you do the following:
• Delete the primary database and create it again using the standard DBA process (for example, usingOracle DBCA utility, and so on).
• Install the PMG DB schema and populate data as per steps mentioned in the install guide. For moreinformation, see, Cisco RAN Management System Installation Guide.
High Availability for Cisco RAN Management Systems64
Configuring High Availability for the PMG DBRecreating Standby Servers
• To create the standby servers again, follow the tasks listed in the Configuration Workflow, on page 3.
Troubleshooting Data Guard on PMG DBThis section provides solutions to some of the errors that may be encountered while configuring Data Guardon the PMG DB.
• Reverting Original Primary Database After Site Failure , on page 65
• Verifying the Data Guard Broker Configuration, on page 73
• Reverting From Disk Space Issues, on page 73
Reverting Original Primary Database After Site FailureIf the Data Guard Broker "show configuration" status is not a SUCCESS, check the error code and try toresolve using steps under respective errors. If not connected to DGMGRL tool, connect as sys@PMGDB.
The outputs of the steps are from the setup with primary, hot standby, and cold standby.Note
ORA-16825
ORA-16825: multiple errors or warnings, including fast-start failover-related errors or warnings, detected forthe database
If DGMGRL> show configuration is showing this error for primary database, check verbose view to seeexact errors related to the database.DGMGRL> show database verbose 'PMGDB'Database - PMGDB
Role: PRIMARYIntended State: TRANSPORT-ONInstance(s):PMGDB
Database Error(s):ORA-16628: broker protection mode inconsistent with the database setting
Database Warning(s):ORA-16817: unsynchronized fast-start failover configuration
………ORA-16628
ORA-16628: broker protection mode inconsistent with the database setting
IfDGMGRL> show configuration is showing this error for the primary database, change toMaxAvailabilitymode. Disable “Fast-Start Failover”, if configuration shows it as “ENABLED”.DGMGRL> disable fast_start failoverDisabled.
DGMGRL> edit configuration set protection mode as MaxAvailability;Succeeded.
High Availability for Cisco RAN Management Systems 65
Configuring High Availability for the PMG DBTroubleshooting Data Guard on PMG DB
Check configuration; if it is a SUCCESS enable fast_start failover, otherwise troubleshootfor the errordisplayed.DGMGRL> show configuration………Fast-Start Failover: DISABLED
Configuration Status:SUCCESS
DGMGRL> enable fast_start failoverEnabled.
ORA-16810
ORA-16810: multiple errors or warnings detected for the database
If DGMGRL> show configuration is showing this error for standby database, check verbose view to seeexact errors related to the database.DGMGRL> show database verbose 'PMGDB_STBY'Database - PMGDB_STBY
Role: PHYSICAL STANDBYIntended State: APPLY-ONTransport Lag: (unknown)Apply Lag: (unknown)Real Time Query: OFFInstance(s):PMGDB
Database Error(s):ORA-16700: the standby database has diverged from the primary databaseORA-16766: Redo Apply is stopped
………ORA-16700
ORA-16700: the standby database has diverged from the primary database
If DGMGRL> show database verbose is showing this error for the standby database it indicates that the theprimary and standby server files have a mismatch. View alertPMGDB.log of the standby database to see if itis due to ORA-19909 error. If ORA-19909 is in the log, follow the trouble steps for ORA-19909. Followingis sample output of the logs.$ cd /u01/app/oracle/diag/rdbms/pmgdb_stby/PMGDB/trace$ tail -f alert_PMGDB.log -n100Or$ vi alert_PMGDB.log
………MRP0 started with pid=66, OS id=24771MRP0: Background Managed Standby Recovery process started (PMGDB)Serial Media Recovery startedManaged Standby Recovery starting Real Time ApplyWarning: Recovery target destination is in a sibling branchof the controlfile checkpoint. Recovery will only recoverchanges to datafiles.MRP0: Detected orphaned datafiles!Recovery will possibly be retried after flashback...Errors in file /u01/app/oracle/diag/rdbms/pmgdb_stby/PMGDB/trace/PMGDB_mrp0_24771.trc:ORA-19909: datafile 1 belongs to an orphan incarnationORA-01110: data file 1: '/u01/app/oracle/oradata/PMGDB/system01.dbf'
High Availability for Cisco RAN Management Systems66
Configuring High Availability for the PMG DBReverting Original Primary Database After Site Failure
Managed Standby Recovery not using Real Time Apply………
The log directory may vary based on the installation setup.Note
ORA-19909
ORA-19909: datafile 1 belongs to an orphan incarnation
If alert_PMGDB.log shows ORA-19909 error, it indicates that the primary database was opened withRESETLOGS after the database recovery that causes the database incarnation to be different between theprimary and standby database. The following steps describe how to avoid re-creating a physical standbydatabase to resolve this issue.
These steps require FLASHBACK enabled. FLASHBACK is enabled as part of configuring the standbyserver setup and is enabled for standby databases. However, for any reason if it is not enabled, the standbydatabase needs to be recreated. To do this, see “ORA-38729”.
Note
1 Before proceeding ensure that the fast_start failover in DGMRL configuration is DISABLED. If notdisabled, disable it.DGMGRL> disable fast_start failoverDisabled.
2 Determine the SCN before the RESETLOGS operation occurred.
• On the primary database, use the following query to obtain the value of the system change number(SCN) that is two SCNs before the RESETLOGS operation occurred on the primary database.SQL> SELECT TO_CHAR(RESETLOGS_CHANGE# - 2) FROM V$DATABASE;TO_CHAR(RESETLOGS_CHANGE#-2)----------------------------------------1245964
3 Obtain the current SCN on the standby database.SQL> SELECT TO_CHAR(CURRENT_SCN) FROM V$DATABASE;TO_CHAR(CURRENT_SCN)----------------------------------------1249692
4 If the value of CURRENT_SCN is larger than the value of resetlogs_change# - 2, issue the followingstatement on the standby database to flash back the standby database.
If the value of CURRENT_SCN is less than the value of the resetlogs_change# - 2, skip to next step.Note
SQL> FLASHBACK STANDBY DATABASE TO SCN 1245964;Flashback complete.
5 Restart Redo Apply on the standby server.
High Availability for Cisco RAN Management Systems 67
Configuring High Availability for the PMG DBReverting Original Primary Database After Site Failure
This step may have to be repeated if following step shows configuration status Error or Warning.Note
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;Database altered.
6 Connect to DGMGRL as sys@PMGDB if not connected and view the DGMGRL show configuration; ifthe status is error with error ORA-16766, repeat previous step to restart redo apply and continue with thisstep till no error is displayed.DGMGRL> show configurationConfiguration - dgPMGDB
Protection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - Physical standby databaseError: ORA-16766: Redo Apply is stopped
PMGDB_STBY2 - Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated
Fast-Start Failover: DISABLED
Configuration Status:ERROR
7 If the previous step, shows “Warning: ORA-16826: apply service state is inconsistent with the DelayMinsproperty”, use this step on the standby server sql prompt. Else continue with the next step.SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;Database altered.
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECTFROM SESSION;Database altered.
DGMGRL> show configurationConfiguration - dgPMGDB
Protection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - Physical standby databasePMGDB_STBY2 - Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated
Fast-Start Failover: DISABLED
Configuration Status:SUCCESS
8 Now enable fast_start failover and if cold standby is present in the setup, enable cold standby database.DGMGRL> enable fast_start failoverEnabled.
DGMGRL> enable database 'PMGDB_STBY2'Enabled.
9 Verify that errors are resolved and the setup is reverted to the original state successfully.DGMGRL> show configurationConfiguration - dgPMGDB
Protection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby databasePMGDB_STBY2 - Physical standby database
High Availability for Cisco RAN Management Systems68
Configuring High Availability for the PMG DBReverting Original Primary Database After Site Failure
Fast-Start Failover: ENABLED
Configuration Status:SUCCESS
RMAN-20208
RMAN-20208: UNTIL CHANGE is before RESETLOGS change
The following command may throw error as shown.RMAN> RUN
{SET UNTIL SCN <recovery_scn> ;RESTORE DATABASE;RECOVER DATABASE;}
Starting restore at 17-JUL-14using target database control file instead of recovery catalogRMAN-00571: ===========================================================RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============RMAN-00571: ===========================================================RMAN-03002: failure of restore command at 07/17/2014RMAN-20208: UNTIL CHANGE is before RESETLOGS changeSolution:
The error RMAN-20208 occurred because the command is trying to go to an SCN (<recovery_scn>) beforethe current incarnation of the database. Follow the steps to resolve this.
1 List the incarnation of the original primary database in RMAN.$ export ORACLE_SID=PMGDB$ rman target=/RMAN> list incarnation of database;using target database control file instead of recovery catalog
List of Database IncarnationsDB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time------- ------- -------- ---------------- --- ---------- ----------1 1 PMGDB 2481454463 PARENT 1 15-AUG-092 2 PMGDB 2481454463 PARENT 945184 17-JUL-143 3 PMGDB 2481454463 PARENT 1243235 17-JUL-144 4 PMGDB 2481454463 CURRENT 1243996 17-JUL-14
2 Nowmark the incarnation having SCN before the <recovery_scn>. In this example, if value of recovery_scnis 1243233, select incarnation 2 with SCN 945184. Reset the database to this incarnation.RMAN> reset database to incarnation 2;database reset to incarnation 2
3 Now execute the RUN command which was failed; it should show output as expected.RMAN> RUN
{SET UNTIL SCN <recovery_scn> ;RESTORE DATABASE;RECOVER DATABASE;}
executing command: SET until clause
………media recovery complete, elapsed time: 00:00:00Finished recover at 08-JUL-14
ORA-38729
ORA-38729: Not enough flashback database log data to do FLASHBACK.
High Availability for Cisco RAN Management Systems 69
Configuring High Availability for the PMG DBReverting Original Primary Database After Site Failure
This error may occur while resolving error RMAN-20208. If FLASHBACK DATABASE is run when thetarget SCN is outside the flashback window, then FLASHBACK DATABASE fails with an ORA-38729error. In this case the database will not be changed. The possible reason to get this error could be intentionallyor unintentionally removing/moving some flashback database logs.
To overcome this error, recreate the standby again, that is, clean up as described in Rolling Back and CleaningUp Standby and Primary Configurations, on page 55 and create the standby as described in Configuring thePrimary Server, on page 4 and Configuring the Hot Standby Server, on page 10. Though most of the stepsare same, some additional changes are required. Hence those steps are repeated here.
1 Log in to primary sql prompt and defer LOG_ARCHIVE_DEST_STATESQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=DEFER;System altered.
2 Log in to DGMGRL from primary or hot standby; Disable FAST_START FAILOVER is not disabledalready.DGMGRL> DISABLE FAST_START FAILOVER FORCEDisabled.
3 Change protection mode in Data Guard Broker configuration.DGMGRL> edit configuration set protection mode as MaxPerformance;Succeeded.
4 Remove hot standby from Data Guard Broker configuration.DGMGRL> REMOVE DATABASE 'PMGDB_STBY'Removed database "PMGDB_STBY" from the configuration
5 To remove the configuration files from the hot standby, see .
Redo apply process may have already stopped for the hot standby. Ignore the error if the step "Stop redoapply process" throws any error.
Note
6 To remove the standby database, see Removing the Standby Database, on page 60.
7 Return and log in to the primary sql prompt and enable LOG_ARCHIVE_DEST_STATE.SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;System altered.
8 Back up the primary database, see Backing Up the Primary Database, on page 8.
9 Move existing file as backup if pfile and control file already exist in the /tmp file.$ mv /tmp/pmgdb_stby.ctl /tmp/pmgdb_stby.ctl_back$ mv /tmp/initPMGDB_stby.ora /tmp/initPMGDB_stby.ora_backSystem responds with command prompt.
10 Create the standby control file and PFILE, see the Creating Standby Control File and PFILE, on page 9.
11 Stop the observer by logging in to DGMGRL and connecting as sys@PMGDB if the observer is running.DGMGRL> stop observerDone.
12 Create a hot standby, see Configuring the Hot Standby Server, on page 10.
ORA-16820
ORA-16820: fast-start failover observer is no longer observing this database
This error may be displayed in DGMGRL show configuration output, if the observer is not running or thedrcPMGDB.log (under /u01/app/oracle/diag/rdbms/pmgdb/PMGDB/trace) on primary shows warning as“ObserverHB indicates lack of Observer pings at Standby”.
High Availability for Cisco RAN Management Systems70
Configuring High Availability for the PMG DBReverting Original Primary Database After Site Failure
To overcome this error, stop the observer and run the observer from a different host (primary or standby)other than the one that is running currently.
1 Log in to DGMGRL and connect as [email protected]> stop observerDone.
2 Open another console for primary or standby server and log in as oracle user. Start observer as a backgroundprocess.$ nohup dgmgrl -silent sys/<passoword>@PMGDB "start observer" &$ cat nohup.outObserver started
3 Stop observer again if error persists and start observer from another host.
ORA-16631
ORA-16631: operation requires shutdown of database or instance ""
This error may occur when enabling cold standby database while recovering original primary after site failure:DGMGRL> enable database 'PMGDB_STBY2'Error: ORA-16631: operation requires shutdown of database or instance ""To overcome this error:
1 Shut down and start up mount the cold standby.SQL> SHUTDOWN IMMEDIATE;ORA-01507: database not mounted
ORACLE instance shut down.
SQL> STARTUP MOUNT;ORACLE instance started.Total System Global Area 3290345472 bytesFixed Size 2217832 bytesVariable Size 1811941528 bytesDatabase Buffers 1459617792 bytesRedo Buffers 16568320 bytesDatabase mounted.
2 Attempt to enable the database again from the DGMGRL. If successful, skip the next steps.
3 Verify database parameter FAL_SERVER on cold standby sql prompt if error still persists. If the valueis not “pmgdb”, correct it using the alter system command.SQL> show parameter fal_serverNAME TYPE VALUE------------------------------------ ----------- ------------------------------fal_server string pmgdb
SQL> ALTER SYSTEM SET FAL_SERVER=pmgdbSystem altered.
4 Verify LOG_ARCHIVE_DEST_3 on the primary database. If it is not pointing to PMGDB_stby2 asshown in the output, Set LOG_ARCHIVE_DEST_3 and enable.SQL> show parameter LOG_ARCHIVE_DEST_STATE_3NAME TYPE VALUE-------------------- ----- ------------------------------log_archive_dest_3 string SERVICE=PMGDB_stby2 NOAFFIRM A
SYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PMGDB_STBY2
log_archive_dest_30 stringlog_archive_dest_31 string
SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_3='SERVICE=PMGDB_stby2 NOAFFIRM ASYNC
High Availability for Cisco RAN Management Systems 71
Configuring High Availability for the PMG DBReverting Original Primary Database After Site Failure
VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PMGDB_STBY2';System altered.
SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=ENABLE;System altered.
5 Shut down and startup mount the cold standby database.SQL> SHUTDOWN IMMEDIATE;ORA-01507: database not mountedORACLE instance shut down.
SQL> STARTUP MOUNT;ORACLE instance started.Total System Global Area 3290345472 bytesFixed Size 2217832 bytesVariable Size 1811941528 bytesDatabase Buffers 1459617792 bytesRedo Buffers 16568320 bytesDatabase mounted.
6 Remove the cold standby database from DGMGRL. Add standby database to the Data Guard Brokerconfiguration.DGMGRL> remove database 'PMGDB_STBY2';Removed database "PMGDB_STBY2" from the configuration
DGMGRL> add database 'PMGDB_STBY2' as connect identifier is PMGDB_STBY2 maintained asphysical ;Database " PMGDB_STBY2 " added
7 Enable cold standby database and verify that the configuration is showing that it is a SUCCESS.DGMGRL> enable database 'PMGDB_STBY2';Enabled.
DGMGRL> show configurationConfiguration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby databasePMGDB_STBY2 - Physical standby database
Fast-Start Failover: ENABLEDConfiguration Status:SUCCESS
ORA-12543
ORA-12543: TNS:destination host unreachable
This error may occur while adding or viewing the database from the Data Guard Broker configuration. Thepossible reason could be that the entries in tnsnames.ora, listener.ora are wrong or with syntax error ornetwork/firewall is not allowing the source host to connect to the destination host.
1 Verify if tnsnames.ora on source host (primary/standby) has correct entries as mentioned in this document.The command tnsping can be used to verify:$ tnsping <connection_name>where <connection_name> is the name of an entry in tnsnames.ora file. i.e. either PMGDB, PMGDB_STBYor PMGDB_STBY2.
If connection is working, the output would be OK as shown in following example.$ tnsping PMGDB_STBY2TNS Ping Utility for Linux: Version 11.2.0.1.0 - Production on 07-SEP-2014 19:48:44Copyright (c) 1997, 2009, Oracle. All rights reserved.Used parameter files:
High Availability for Cisco RAN Management Systems72
Configuring High Availability for the PMG DBReverting Original Primary Database After Site Failure
/u01/app/oracle/product/11.2.0/dbhome_1/network/admin/sqlnet.oraUsed TNSNAMES adapter to resolve the aliasAttempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = pmg-db-ha3)(PORT= 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = PMGDB)))OK (0 msec)
2 Verify if listener.ora on the destination host (primary/standby) has correct entries as mentioned in thisdocument. Try to restart listener.$ lsnrctl stop
$ lsnrctl start
3 Verify if any iptables rules on either source or destination host are blocking the connection. Try tnspingby stopping iptables rules, that is, log in as a root and execute:$ service iptables stop
Log in as an oracle user and verify tnsping$ tnsping <connection_name>
ORA-16737
ORA-16737: the redo transport service for standby database "PMGDB_STBY" has an error
The error may be displayed in DGMGRL show configuration. This may be an intermediate error which maygo away if subsequent steps are executed or underlying errors could be different which can be viewed usingthe show database command.DGMGRL> show database verbose 'PMGDB'.
Verifying the Data Guard Broker ConfigurationThe DGMGRL> show configuration command configuration status sometimes shows errors ORA-16610& DGM-17017.
For example:Configuration Status:ORA-16610: command "ENABLE DATABASE PMGDB_STBY2" in progressDGM-17017: unable to determine configuration status
Configuration Status:ORA-16610: command "Broker automatic health check" in progressDGM-17017: unable to determine configuration status
Solution:
Wait for a few minutes and execute the command again.
Reverting From Disk Space IssuesAfter PMG DB is installed and configured, disk space issues may be encountered due to archiving of logsand DB content. This may cause the PMG DB to stop functioning. To verify if PMG DB has stoppedfunctioning, check the PMG DB alert log "alert.PMGDB.log" for the following error.ORA-19815: WARNING: db_recovery_file_dest_size of 4070572032 bytes is 99.56% used, and has17721344remaining bytes available.To revert from the error, increase the DB recovery file destination size to a higher value using the followingcommand:SQL> ALTER SYSTEM SET DB_RECOVERY_FILE_DEST_SIZE=8G;
High Availability for Cisco RAN Management Systems 73
Configuring High Availability for the PMG DBVerifying the Data Guard Broker Configuration
In some cases, following error could be observed.ORA-00845: MEMORY_TARGET not supported on this systemThe possible reason is that the tmpfs directory may be full and it needs to be increased to a higher value usingthe following commands.
• To view tmpfs size:# df -k
• Change the size# mount -t tmpfs shmfs -o size=8g /dev/shm
• Verify the size# df -k
• To apply the above settings permanently to prevent the value from resetting to the default (after reboot),add the following entry in the /etc/fstab:shmfs /dev/shm tmpfs size=8g 0
High Availability for Cisco RAN Management Systems74
Configuring High Availability for the PMG DBReverting From Disk Space Issues