Post on 03-Sep-2020
Oracle Optimized Solution for Disaster Recovery
Oracle E-Business Suite 12.1.3 Using Oracle Solaris Cluster Geo Edition – Implementation Guide
AU G U S T 2 0 1 8 | E 9 8 6 1 8 - 0 1
Table of Contents
1. Overview ......................................................................................................................................................... 1
2. Architecture ..................................................................................................................................................... 2
Sites Configuration with Oracle E-Business Suites 12.1.3 ........................................................................ 2
Hostnames and IP Address Information .................................................................................................... 3
2.2.1. The /etc/hosts File .................................................................................................................... 3
2.2.2. The FND_NODES Table .................................................................................................................... 4
Sites Configuration with Oracle Solaris Cluster Geo Edition ..................................................................... 6
3. Configure the Oracle E-Business Suite 12.1.3 DB Context Files ................................................................... 7
Edit the /etc/hosts Files ....................................................................................................................... 8
Set the s_virtual_hostname Variable ................................................................................................. 8
Run AutoConfig on the Primary Cluster ..................................................................................................... 9
Run AutoConfig on the Secondary Cluster .............................................................................................. 23
Verify the Contents of the FND_NODES Table .......................................................................................... 25
4. Configure Flashback on the Database .......................................................................................................... 26
5. Configure Oracle Solaris Cluster Geo Edition ............................................................................................... 30
Install Geographic Edition Software ........................................................................................................ 30
Prepare Each Zone Cluster for Partner Membership .............................................................................. 31
5.2.1. Set Up the Network Address for Each Zone Cluster ..................................................................... 31
5.2.2. Set Up Security on the Primary Cluster ........................................................................................ 33
5.2.3. Set Up Security on the Secondary Cluster ................................................................................... 34
Enable Geographic Edition Software ....................................................................................................... 36
Configure Partnerships ............................................................................................................................ 42
5.4.1. Configure Trust Between Partner Clusters ................................................................................... 42
5.4.2. Create Partnerships ...................................................................................................................... 52
5.4.3. Configure the Secondary Cluster to Join the Partnership ............................................................. 55
Configure Protection Groups ................................................................................................................... 56
5.5.1. Create the First Protection Group (opmn-ebs-pg) ........................................................................ 56
Overview ii
5.5.2. Create the Second Protection Group (cm-ebs-pg) ....................................................................... 60
5.5.3. Create the Third Protection Group (cm-odg-pg) ........................................................................... 64
5.5.4. Display the Oracle Solaris Cluster Listings ................................................................................... 67
Configure Sites and Multigroups ............................................................................................................. 74
5.6.1. Create and Configure the santaclara Site ..................................................................................... 75
5.6.2. Create and Configure the hillsboro Site ........................................................................................ 77
6. Create and Configure a DNS Server ............................................................................................................ 78
Configure the DNS Service ..................................................................................................................... 79
Configure the DNS Update Scripts .......................................................................................................... 82
6.2.1. Create the dns_update.sh Script .............................................................................................. 82
6.2.2. Verify the DNS Update Scripts on the DNS Server ...................................................................... 84
Configure the DNS Client ........................................................................................................................ 84
Configure the Protection Group Role Change Script ............................................................................... 86
7. Load Testing with Oracle Application Testing Suite (OATS) .......................................................................... 86
The Test Configuration ............................................................................................................................ 86
OLTP Load Generator: the OATS Machine ............................................................................................. 88
OATS Testing Information ........................................................................................................................ 89
Concurrent Manager Reports .................................................................................................................. 90
8. Switchover ebs-mg Multigroup to the Secondary Cluster ............................................................................. 90
Switchover Operation .............................................................................................................................. 91
Switchover Timing Information ................................................................................................................ 95
9. Takeover ebs-mg Multigroup from the Secondary Cluster ............................................................................ 96
Takeover Operation ................................................................................................................................. 96
Takeover Timing Information ................................................................................................................... 98
10. Restore the Primary Site ............................................................................................................................... 99
11. Apply SuperCluster Quarterly Maintenance Updates ................................................................................. 106
Overview ii
Overview 1
1. Overview
This document provides a step-by-step guide to set up disaster recovery for an Oracle E-Business Suite 12.1.3 installation on Oracle SuperCluster. The configuration uses Oracle Solaris Cluster Geo Edition to switch Oracle E-Business Suite 12.1.3 to a standby Oracle SuperCluster in the event of a failure. The primary and standby SuperClusters can be in different geographical locations.
The document assumes that you have installed Oracle E-Business Suite 12.1.3 and configured Oracle Data Guard and Oracle’s ZFS Storage Appliance by following the instructions provided in the following guides (available on My Oracle Support):
Oracle Optimized Solution for Oracle E-Business Suite on Oracle SuperCluster Example Implementation Guide (Doc ID 1557489.1)
Oracle Optimized Solution for Disaster Recovery Oracle E-Business Suite Example Implementation Guide (Doc ID 1566368.1)
The guides contain procedures for duplicating database content and applications between primary and standby Oracle SuperClusters.
This document builds on the configurations from the implementation guides by deploying Oracle Cluster Geo Edition to automatically orchestrate the stopping and starting of Oracle E-Business Suite 12.1.3 services after a failover to the standby SuperCluster. Furthermore, in the configuration of the primary and standby SuperClusters, logical hostnames are used for the APP Tiers. Likewise, an alias hostname for each Oracle RAC VIP is used as a virtual hostname for each Oracle RAC database host. Thus, in the event of a switchover, you do not need to repeatedly run AutoConfig beyond initial configuration.
The topics for a disaster recovery solution are presented as follows:
1. Description of the reference architecture.
2. Configuration of Oracle E Business Suite 12.1.3 in both the primary and secondary SuperClusters.
3. Installation and configuration of Oracle Solaris Cluster Geo Edition.
4. Description of the switchover process between the primary and standby SuperClusters which ends with the standby SuperCluster taking subsequent control.
Architecture 2
2. Architecture
This section describes the reference architecture for the implementation of the disaster recovery program.
Sites Configuration with Oracle E-Business Suites 12.1.3
The configuration of Oracle E-Business Suites 12.1.3 in the primary and standby sites is illustrated in the following figure:
Figure 1- Starting Configuration of Primary and Standby Sites
Note: In this document, the terms “standby” and “secondary” are used interchangeably.
As shown in Figure 1, each site consists of four Oracle VM SPARC Server domains.
Two clustered database domain containing Oracle RAC
Two clustered application domains containing Oracle Solaris Cluster, which is configured with the following zone clusters:
o Zone cluster containing a Web Entry Point (Web) and 2 Oracle Process Manager and Server (OPMN1 and OPMN2) services
o Zone cluster containing Concurrent Manager (CM1) services
Architecture 3
Additionally, redo log shipping and ZFSSA continuous replication are configured for both primary and standby sites.
Db_unique_names are assigned for the two sites:
VIS at the primary site
VISION at the standby site
In this configuration, Oracle Solaris Cluster Geo Edition is not yet installed on either site. Only the primary site is active.
Note: In the configuration on which this document is based, a symlink is set up to an ENV file.
This is in place for each database node. Below is an example.
root@etc18cn1dbiodom1:~# su – oracle
Oracle Corporation SunOS 5.11 11.3 December 2015
oracle@etc18cn1dbiodom1:~$
oracle@etc18cn1dbiodom1:~$ ls –l VIS.env
lrwxrwxrwx 1 oracle oinstall 67 Oct 2 2015 VIS.env ->
/u01/app/oracle/product/11.2.0.3/dbhome_1/VIS1_etc18cn1dbiodom1.env
The file enables you to access the SQL and the DGMGRL prompts. Accessing either prompt is part of the subsequent procedures in this document.
When accessing the DGMGRL prompt, make sure that you use the correct DB sys user password. For example, after sourcing the ENV file, type:
oracle@etc18cn1dbiodom1:~$ dgmgrl sys/<password>
Hostnames and IP Address Information
The Oracle SuperCluster provides an external 10 gigabit Ethernet (10GbE) network and internal Internet Protocol over InfiniBand (IPoIB) network. The latter is used as a private non-routable network (192.168/16 prefix) that is only accessible from within the Oracle SuperCluster.
One of the tasks in this document to implement disaster recovery is to map an alias hostname for each Oracle RAC VIP as a virtual hostname for each Oracle RAC database host.
After you complete the task, the mapping of hostnames, IP addresses, and the aliases appears in both the
/etc/hosts files of the DB and APP tier nodes as well as FND_NODES table for both sites.
2.2.1. The /etc/hosts File
For example, on the primary site, the /etc/hosts file on each DB and APP Tier contains the following entries:
192.168.28.101 etc11-01-ib-vip.us.oracle.com etc11-01-ib-vip idb1-vip.us.oracle.com idb1-vip
192.168.28.102 etc11-02-ib-vip.us.oracle.com etc11-02-ib-vip idb2-vip.us.oracle.com idb2-vip
Architecture 4
On the standby site, the /etc/hosts file on each DB and APP Tier contains the following entries:
192.168.28.101 etc18db01-ibvip.us.oracle.com etc18db01-ibvip idb1-vip.us.oracle.com idb1-vip
192.168.28.102 etc18db02-ibvip.us.oracle.com etc18db02-ibvip idb2-vip.us.oracle.com idb2-vip
In these sample /etc/hosts files, the alias hostnames idb1-vip and idb2-vip map to the same IP
addresses at the primary and standby sites, respectively, 192.168.28.101 and 192.168.28.102.
2.2.2. The FND_NODES Table
The mapping also appears in the FND_NODES table within the APPS schema as shown in this example.
root@etc18cn1dbiodom1:~# su – oracle
Oracle Corporation SunOS 5.11 11.3 December 2015
oracle@etc18cn1dbiodom1:~$
oracle@etc18cn1dbiodom1:~$ . VIS.env
oracle@etc18cn1dbiodom1:~$ sqlplus “/as sysdba”
SQL *Plus: Release 11.2.0.3.0 Production
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage
Management, OLAP,
Data Mining and Real Application Testing options
SQL>
SQL> select node_name, server_address, virtual_ip from fnd_nodes;
NODE_NAME SERVER_ADDRESS VIRTUAL_IP
---------------- -------------- ----------
EWEB-LH m.m.m.68
IOPMN2-LH 192.168.28.34
ETC11-CN01 idb1-vip
ETC18CN1DBIODOM1 idb1-vip
ICM1-LH 192.168.28.43
ETC11-CN02 idb2-vip
ETC18CN2DBIODOM1 idb2-vip
AUTHENTICATION *
IOPMN1-LH 192.168.28.33
9 rows selected.
SQL>
Architecture 5
Note: In all the examples in this documentation, some IP addresses have been masked on the first 3 octets to indicate that these should reflect valid addresses that are actually used at your company’s configuration. For the primary cluster, the masked address is n.n.n.#, such as n.n.n.27. For the secondary cluster, the masked address is m.m.m.#, such as m.m.m.68.The distinction also indicates the typical configuration where primary and secondary clusters are on different subnets.
The FND_NODES table provides the following information:
Node names for the nodes in the Cluster configuration. Oracle Solaris Cluster manages the location of the logical hostname for the node names. Consequently, the logical hostname is hosted only on one node at any one time.
The hostnames are prefixed with either E or I. The E prefix designates that the logical hostname is an external address on the 10GbE network. The I prefix indicates that the hostname is in an internal address on the IPoIB network.
The same logical hostnames for the services are used at the primary and standby sites, specifically EWEB-LH, IOPMN1-LH, IOPMN2-LH and ICM1-LH.
Two Oracle SuperCluster DB hostname pairs exist:
o ETC11-CN01 and ETC11-CN02 on the primary site. o ETC18CN1DBIODOM1 and ETC18CN2DBIODOM1 on the standby site.
Each pair of Oracle SuperCluster DB hostnames is mapped to corresponding virtual IP names in the internal IPoIB network as follows:
o ETC11-CN01 and ETC18CN1DBIODOM1 are mapped to idb1-vip. o ETC11-CN02 and ETC18CN2DBIODOM1 are mapped to idb2-vip.
Architecture 6
Sites Configuration with Oracle Solaris Cluster Geo Edition
The following figure shows the primary and standby sites with Oracle Solaris Cluster Geo Edition installed and configured.
Figure 2- Intended Final Configuration of Primary and Standby Sites
Figure 2 still reflects the basic architecture as illustrated in Figure 1. However, this figure also shows how partnerships are configured through Oracle Solaris Cluster Geo Edition:
Partnership 1 consists of the OPMN zone clusters of both sites
Partnership 2 consists of the CM zone clusters of both sites.
Likewise, protection groups are also configured for the partnerships:
Protection Group 1 consists of the WEB and OPMN services.
Protection Group 2 consists of the CM services and ZFS Storage Appliance
Protection Group 3 consists of the Oracle Data Guard
Note that Protection Groups 2 and 3 are set for Partnership 2. The steps to configure partnerships and protection groups are provided later in the section Configure Oracle Solaris Cluster Geo Edition.
With this configuration, Oracle Solaris Cluster Geo Edition provides the capability that orchestrates the stopping and starting of Oracle E-Business Suite 12.1.3 services after a switchover or failover from the primary site to the
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 7
standby site.
In summary, the architecture resembles the configurations described in the Oracle Optimized Solution for Oracle E-Business Suite on Oracle SuperCluster Example Implementation Guide (Doc ID 1557489.1) and Oracle Optimized Solution for Disaster Recovery Oracle E-Business Suite Example Implementation Guide (Doc ID 1566368.1).
3. Configure the Oracle E-Business Suite 12.1.3 DB Context
Files
This section provides the procedure to configure the Oracle E-Business Suite 12.1.3. DB context files to prepare for disaster recovery setup.
Based on the reference architecture described previously, the following DB and APP Tier nodes exist in the current configuration. Refer to this table while performing the configurations as instructed:
On the Primary Cluster
Tier Node RAC VIP
DB (RAC) etc11-cn01 etc11-01-ib-vip
DB (RAC) etc11-cn02 etc11-02-ib-vip
APP (opmn-zc) eopmn1
APP (opmn-zc) eopmn2
APP (cm-zc) ecm1
APP (cm-zc) ecm2
On the Secondary Cluster
Tier Node RAC VIP
DB (RAC) etc18cn1dbiodom1 etc18db01-ibvip
DB (RAC) etc18cn2dbiodom1 etc18db02-ibvip
APP (dr-opmn-zc) deopmn1
APP (dr-opmn-zc) deopmn2
APP (dr-cm-zc) decm1
APP (dr-cm-zc) decm2
In the procedure that follows, you would perform the steps on all 12 nodes as indicated.
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 8
Edit the /etc/hosts Files
This step maps the hostnames and their addresses to the virtual IP names idb1-vip and idb2-vip. The
revisions are in bold.
On the primary cluster, edit the /etc/hosts files to reference the virtual IP names.
192.168.28.101 etc11-01-ib-vip.us.oracle.com etc11-01-ib-vip idb1-
vip.us.oracle.com idb1-vip
192.168.28.102 etc11-02-ib-vip.us.oracle.com etc11-02-ib-vip idb2-
vip.us.oracle.com idb2-vip
On the secondary cluster, edit the /etc/hosts files to reference the virtual IP names.
192.168.28.101 etc18db01-ibvip.us.oracle.com etc18db01-ibvip idb1-
vip.us.oracle.com idb1-vip
192.168.28.102 etc18db02-ibvip.us.oracle.com etc18db02-ibvip idb2-
vip.us.oracle.com idb2-vip
Set the s_virtual_hostname Variable
This step populates the FND_NODES table and subsequent TNS entries at each DB and APP Tier.
On the primary cluster, perform the following steps.
a. On etc11-cn01, set s_virtual_hostname to idb1-vip.
oracle@etc11-cn01:~$ . /u01/app/oracle/product/11.2.0.3/dbhome_1/VIS1_etc11-
cn01.env
oracle@etc11-cn01:~$ echo $CONTEXT_FILE
/u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/VIS1_etc11-cn01.xml
oracle@etc11-cn01:~$
oracle@etc11-cn01:~$ grep s_virtual_hostname $CONTEXT_FILE
<host oa_var="s_virtual_hostname">idb1-vip</host>
oracle@etc11-cn01:~$
b. On etc11-cn02, set s_virtual_hostname to idb2-vip.
oracle@etc11-cn02:~$ . /u01/app/oracle/product/11.2.0.3/dbhome_1/VIS2_etc11-
cn02.env
oracle@etc11-cn02:~$
oracle@etc11-cn02:~$ echo $CONTEXT_FILE
/u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/VIS2_etc11-cn02.xml
oracle@etc11-cn02:~$
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 9
oracle@etc11-cn02:~$ grep s_virtual_hostname $CONTEXT_FILE
<host oa_var="s_virtual_hostname">idb2-vip</host>
oracle@etc11-cn02:~$
On the secondary cluster, perform the following steps:
a. On etc18cn1dbiodom1, set s_virtual_hostname to idb1-vip.
oracle@etc18cn1dbiodom1:~$ .
/u01/app/oracle/product/11.2.0.3/dbhome_1/VIS1_etc18cn1dbiodom1.env
oracle@etc18cn1dbiodom1:~$ echo $CONTEXT_FILE
/u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/VIS1_etc18cn1dbiodom1.xml
oracle@etc18cn1dbiodom1:~$
oracle@etc18cn1dbiodom1:~$ grep s_virtual_hostname $CONTEXT_FILE
<host oa_var="s_virtual_hostname">idb1-vip</host>
oracle@etc18cn1dbiodom1:~$
b. On etc18cn2dbiodom1, set s_virtual_hostname to idb2-vip.
oracle@etc18cn2dbiodom1:~$ .
/u01/app/oracle/product/11.2.0.3/dbhome_1/VIS2_etc18cn2dbiodom1.env
oracle@etc18cn2dbiodom1:~$ echo $CONTEXT_FILE
/u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/VIS2_etc18cn2dbiodom1.xml
oracle@etc18cn2dbiodom1:~$
oracle@etc18cn2dbiodom1:~$ grep s_virtual_hostname $CONTEXT_FILE
<host oa_var="s_virtual_hostname">idb2-vip</host>
oracle@etc18cn2dbiodom1:~$
Run AutoConfig on the Primary Cluster
On etc11-cn01, run the adautocfg.sh script.
oracle@etc11-cn01:~$ . /u01/app/oracle/product/11.2.0.3/dbhome_1/VIS1_etc11-
cn01.env
oracle@etc11-cn01:~$ echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0.3/dbhome_1
oracle@etc11-cn01:~$
oracle@etc11-cn01:~$ cd $ORACLE_HOME/appsutil/scripts/VIS1_etc11-cn01
oracle@etc11-cn01:~$ ./adautocfg.sh
Enter the APPS user password:
The log file for this session is located at:
/u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/log/
VIS1_etc11-cn01/10210119/adconfig.log
AutoConfig is configuring the Database environment...
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 10
AutoConfig will consider the custom templates if present.
Using ORACLE_HOME location : /u01/app/oracle/product/11.2.0.3/dbhome_1
Classpath :
:/u01/app/oracle/product/11.2.0.3/dbhome_1/
jdbc/lib/ojdbc5.jar:/u01/app/oracle/product/11.2.0.3/dbhome_1/
appsutil/java/xmlparserv2.jar:/u01/app/oracle/product/11.2.0.3/
dbhome_1/appsutil/java:/u01/app/oracle/product/11.2.0.3/dbhome_1/
jlib/netcfg.jar:/u01/app/oracle/product/11.2.0.3/dbhome_1/jlib/
ldapjclnt11.jar
Using Context file :
/u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/VIS1_etc11-cn01.xml
Context Value Management will now update the Context file
Updating Context file...COMPLETED
Attempting upload of Context file and templates to
database...COMPLETED
Updating rdbms version in Context file to db112
Updating rdbms type in Context file to 64 bits
Configuring templates from ORACLE_HOME ...
AutoConfig completed successfully.
oracle@etc11-cn01:~$
On etc11-cn02, run the adautocfg.sh script.
oracle@etc11-cn02:~$ . /u01/app/oracle/product/11.2.0.3/dbhome_1/VIS2_etc11-
cn02.env
oracle@etc11-cn02:~$ echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0.3/dbhome_1
oracle@etc11-cn02:~$
oracle@etc11-cn02:~$ cd $ORACLE_HOME/appsutil/scripts/VIS2_etc11-cn02
oracle@etc11-cn02:~$ ./adautocfg.sh
Enter the APPS user password:
The log file for this session is located at: /u01/app/oracle/product/11.2.0.3/
dbhome_1/appsutil/log/VIS2_etc11-cn02/10210118/adconfig.log
AutoConfig is configuring the Database environment...
AutoConfig will consider the custom templates if present.
Using ORACLE_HOME location : /u01/app/oracle/product/11.2.0.3/dbhome_1
Classpath :
:/u01/app/oracle/product/11.2.0.3/dbhome_1/jdbc/lib/ojdbc5.jar:/u01/app/
oracle/product/11.2.0.3/dbhome_1/appsutil/java/xmlparserv2.jar:/
u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/java:/u01/app/oracle/
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 11
product/11.2.0.3/dbhome_1/jlib/netcfg.jar:/u01/app/oracle/product/11.2.0.3/
dbhome_1/jlib/ldapjclnt11.jar
Using Context file :
/u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/VIS2_etc11-cn02.xml
Context Value Management will now update the Context file
Updating Context file...COMPLETED
Attempting upload of Context file and templates to
database...COMPLETED
Updating rdbms version in Context file to db112
Updating rdbms type in Context file to 64 bits
Configuring templates from ORACLE_HOME ...
AutoConfig completed successfully.
oracle@etc11-cn02:~$
On etc11-cn01, run the adautocfg.sh script.
This second run of AutoConfig on the node updates the $ORACLE_HOME/VIS1_etc11-
cn01/tnsnames.ora file to make its entries for the RAC instance on etc11-cn02 reflect the idb2-vip alias.
The file can only be updated after AutoConfig is run on etc11-cn02 in the preceding step.
In the following example, the output is intentionally truncated.
oracle@etc11-cn01:~$ . /u01/app/oracle/product/11.2.0.3/dbhome_1/VIS1_etc11-
cn01.env
oracle@etc11-cn01:~$ echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0.3/dbhome_1
oracle@etc11-cn01:~$
oracle@etc11-cn01:~$ cd $ORACLE_HOME/appsutil/scripts/VIS1_etc11-cn01
oracle@etc11-cn01:~$ ./adautocfg.sh
Enter the APPS user password:
The log file for this session is located at:
/u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/log/VIS1_etc11-
cn01/10210119/adconfig.log
AutoConfig is configuring the Database environment...
On eopmn1 (eweb-lh), run the adautocfg.sh script.
bash-4.1$ . /ebsfs/apps/VIS/apps/apps_st/appl/APPSVIS_eweb-lh.env
-bash-4.1$
-bash-4.1$ echo $ADMIN_SCRIPTS_HOME
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 12
/ebsfs/apps/VIS/inst/apps/VIS_eweb-lh/admin/scripts
-bash-4.1$
-bash-4.1$ echo $CONTEXT_FILE
/ebsfs/apps/VIS/inst/apps/VIS_eweb-lh/appl/admin/VIS_eweb-lh.xml
-bash-4.1$
-bash-4.1$ $ADMIN_SCRIPTS_HOME/adautocfg.sh
Enter the APPS user password:
The log file for this session is located at: /ebsfs/apps/VIS/inst/apps/
VIS_eweb-lh/admin/log/10070650/adconfig.log
AutoConfig is configuring the Applications environment...
AutoConfig will consider the custom templates if present.
Using CONFIG_HOME location : /ebsfs/apps/VIS/inst/apps/VIS_eweb-lh
Classpath :
/ebsfs/apps/VIS/apps/apps_st/comn/java/lib/appsborg2.zip:/ebsfs/apps/VIS/
apps/apps_st/comn/java/classes
Using Context file : /ebsfs/apps/VIS/inst/apps/VIS_eweb-lh.xml
Context Value Management will now update the Context file
Updating Context file...COMPLETED
Attempting upload of Context file and templates to database...COMPLETED
Configuring templates from all of the product tops...
Configuring AD_TOP........COMPLETED
Configuring FND_TOP.......COMPLETED
Configuring ICX_TOP.......COMPLETED
Configuring MSC_TOP.......COMPLETED
Configuring IEO_TOP.......COMPLETED
Configuring BIS_TOP.......COMPLETED
Configuring AMS_TOP.......COMPLETED
Configuring CCT_TOP.......COMPLETED
Configuring WSH_TOP.......COMPLETED
Configuring CLN_TOP.......COMPLETED
Configuring OKE_TOP.......COMPLETED
Configuring OKL_TOP.......COMPLETED
Configuring OKS_TOP.......COMPLETED
Configuring CSF_TOP.......COMPLETED
Configuring IGS_TOP.......COMPLETED
Configuring IBY_TOP.......COMPLETED
Configuring JTF_TOP.......COMPLETED
Configuring MWA_TOP.......COMPLETED
Configuring CN_TOP........COMPLETED
Configuring CSI_TOP.......COMPLETED
Configuring WIP_TOP.......COMPLETED
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 13
Configuring CSE_TOP.......COMPLETED
Configuring EAM_TOP.......COMPLETED
Configuring FTE_TOP.......COMPLETED
Configuring ONT_TOP.......COMPLETED
Configuring AR_TOP........COMPLETED
Configuring AHL_TOP.......COMPLETED
Configuring OZF_TOP.......COMPLETED
Configuring IES_TOP.......COMPLETED
Configuring CSD_TOP.......COMPLETED
Configuring IGC_TOP.......COMPLETED
AutoConfig completed successfully.
-bash-4.1$
On eopmn1 (iopmn1-lh), run the adautocfg.sh script.
-bash-4.1$ . /ebsfs/apps/VIS/apps/apps_st/appl/APPSVIS_iopmn1-lh.env
-bash-4.1$
-bash-4.1$ echo $ADMIN_SCRIPTS_HOME
/ebsfs/apps/VIS/inst/apps/VIS_iopmn1-lh/admin/scripts
-bash-4.1$
-bash-4.1$ echo $CONTEXT_FILE
/ebsfs/apps/VIS/inst/apps/VIS_eweb-lh/appl/admin/VIS_iopmn1-lh.xml
-bash-4.1$
-bash-4.1$ $ADMIN_SCRIPTS_HOME/adautocfg.sh
Enter the APPS user password:
The log file for this session is located at:
/ebsfs/apps/VIS/inst/apps/VIS_iopmn1-lh/admin/log/10070648/adconfig.log
AutoConfig is configuring the Applications environment...
AutoConfig will consider the custom templates if present.
Using CONFIG_HOME location : /ebsfs/apps/VIS/inst/apps/VIS_iopmn1-lh
Classpath :
/ebsfs/apps/VIS/apps/apps_st/comn/java/lib/appsborg2.zip:/ebsfs/apps/VIS/
apps/apps_st/comn/java/classes
Using Context file : /ebsfs/apps/VIS/inst/apps/VIS_iopmn1-lh/
appl/admin/VIS_iopmn1-lh.xml
Context Value Management will now update the Context file
Updating Context file...COMPLETED
Attempting upload of Context file and templates to database...COMPLETED
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 14
Configuring templates from all of the product tops...
Configuring AD_TOP........COMPLETED
Configuring FND_TOP.......COMPLETED
Configuring ICX_TOP.......COMPLETED
Configuring MSC_TOP.......COMPLETED
Configuring IEO_TOP.......COMPLETED
Configuring BIS_TOP.......COMPLETED
Configuring AMS_TOP.......COMPLETED
Configuring CCT_TOP.......COMPLETED
Configuring WSH_TOP.......COMPLETED
Configuring CLN_TOP.......COMPLETED
Configuring OKE_TOP.......COMPLETED
Configuring OKL_TOP.......COMPLETED
Configuring OKS_TOP.......COMPLETED
Configuring CSF_TOP.......COMPLETED
Configuring IGS_TOP.......COMPLETED
Configuring IBY_TOP.......COMPLETED
Configuring JTF_TOP.......COMPLETED
Configuring MWA_TOP.......COMPLETED
Configuring CN_TOP........COMPLETED
Configuring CSI_TOP.......COMPLETED
Configuring WIP_TOP.......COMPLETED
Configuring CSE_TOP.......COMPLETED
Configuring EAM_TOP.......COMPLETED
Configuring FTE_TOP.......COMPLETED
Configuring ONT_TOP.......COMPLETED
Configuring AR_TOP........COMPLETED
Configuring AHL_TOP.......COMPLETED
Configuring OZF_TOP.......COMPLETED
Configuring IES_TOP.......COMPLETED
Configuring CSD_TOP.......COMPLETED
Configuring IGC_TOP.......COMPLETED
AutoConfig completed successfully.
-bash-4.1$
On eopmn2 (iopmn2-lh), run the adautocfg.sh script.
-bash-4.1$ . /ebsfs/apps/VIS/apps/apps_st/appl/APPSVIS_iopmn2-lh.env
-bash-4.1$
-bash-4.1$ echo $ADMIN_SCRIPTS_HOME
/ebsfs/apps/VIS/inst/apps/VIS_iopmn2-lh/admin/scripts
-bash-4.1$
-bash-4.1$ echo $CONTEXT_FILE
/ebsfs/apps/VIS/inst/apps/VIS_eweb-lh/appl/admin/VIS_iopmn2-lh.xml
-bash-4.1$
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 15
-bash-4.1$ $ADMIN_SCRIPTS_HOME/adautocfg.sh
Enter the APPS user password:
The log file for this session is located at:
/ebsfs/apps/VIS/inst/apps/VIS_iopmn2-
lh/admin/log/10070649/adconfig.log
AutoConfig is configuring the Applications environment...
AutoConfig will consider the custom templates if present.
Using CONFIG_HOME location : /ebsfs/apps/VIS/inst/apps/VIS_iopmn2-lh
Classpath :
/ebsfs/apps/VIS/apps/apps_st/comn/java/lib/appsborg2.zip:/ebsfs/apps/VIS/apps/
apps_st/comn/java/classes
Using Context file : /ebsfs/apps/VIS/inst/apps/VIS_iopmn2-
lh/appl/admin/VIS_iopmn2-lh.xml
Context Value Management will now update the Context file
Updating Context file...COMPLETED
Attempting upload of Context file and templates to database...COMPLETED
Configuring templates from all of the product tops...
Configuring AD_TOP........COMPLETED
Configuring FND_TOP.......COMPLETED
Configuring ICX_TOP.......COMPLETED
Configuring MSC_TOP.......COMPLETED
Configuring IEO_TOP.......COMPLETED
Configuring BIS_TOP.......COMPLETED
Configuring AMS_TOP.......COMPLETED
Configuring CCT_TOP.......COMPLETED
Configuring WSH_TOP.......COMPLETED
Configuring CLN_TOP.......COMPLETED
Configuring OKE_TOP.......COMPLETED
Configuring OKL_TOP.......COMPLETED
Configuring OKS_TOP.......COMPLETED
Configuring CSF_TOP.......COMPLETED
Configuring IGS_TOP.......COMPLETED
Configuring IBY_TOP.......COMPLETED
Configuring JTF_TOP.......COMPLETED
Configuring MWA_TOP.......COMPLETED
Configuring CN_TOP........COMPLETED
Configuring CSI_TOP.......COMPLETED
Configuring WIP_TOP.......COMPLETED
Configuring CSE_TOP.......COMPLETED
Configuring EAM_TOP.......COMPLETED
Configuring FTE_TOP.......COMPLETED
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 16
Configuring ONT_TOP.......COMPLETED
Configuring AR_TOP........COMPLETED
Configuring AHL_TOP.......COMPLETED
Configuring OZF_TOP.......COMPLETED
Configuring IES_TOP.......COMPLETED
Configuring CSD_TOP.......COMPLETED
Configuring IGC_TOP.......COMPLETED
AutoConfig completed successfully.
-bash-4.1$
On ecm1 (icm1-lh), run the adautocfg.sh script.
-bash-4.1$ . /ebsfs/apps/VIS/apps/apps_st/appl/APPSVIS_icm1-lh.env
-bash-4.1$
-bash-4.1$ echo $ADMIN_SCRIPTS_HOME
/ebsfs/apps/VIS/inst/apps/VIS_icm1-lh/admin/scripts
-bash-4.1$
-bash-4.1$ echo $CONTEXT_FILE
/ebsfs/apps/VIS/inst/apps/VIS_eweb-lh/appl/admin/VIS_icm1-lh.xml
-bash-4.1$
-bash-4.1$ $ADMIN_SCRIPTS_HOME/adautocfg.sh
Enter the APPS user password:
The log file for this session is located at: /ebsfs/apps/VIS/inst/apps/VIS_icm1-
lh/admin/log/10070641/adconfig.log
AutoConfig is configuring the Applications environment...
AutoConfig will consider the custom templates if present.
Using CONFIG_HOME location : /ebsfs/apps/VIS/inst/apps/VIS_icm1-lh
Classpath :
/ebsfs/apps/VIS/apps/apps_st/comn/java/lib/appsborg2.zip:/ebsfs/apps/VIS/apps/
apps_st/comn/java/classes
Using Context file : /ebsfs/apps/VIS/inst/apps/VIS_icm1/appl/
admin/VIS_icm1-lh.xml
Context Value Management will now update the Context file
Updating Context file...COMPLETED
Attempting upload of Context file and templates to database...COMPLETED
Configuring templates from all of the product tops...
Configuring AD_TOP........COMPLETED
Configuring FND_TOP.......COMPLETED
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 17
Configuring ICX_TOP.......COMPLETED
Configuring MSC_TOP.......COMPLETED
Configuring IEO_TOP.......COMPLETED
Configuring BIS_TOP.......COMPLETED
Configuring AMS_TOP.......COMPLETED
Configuring CCT_TOP.......COMPLETED
Configuring WSH_TOP.......COMPLETED
Configuring CLN_TOP.......COMPLETED
Configuring OKE_TOP.......COMPLETED
Configuring OKL_TOP.......COMPLETED
Configuring OKS_TOP.......COMPLETED
Configuring CSF_TOP.......COMPLETED
Configuring IGS_TOP.......COMPLETED
Configuring IBY_TOP.......COMPLETED
Configuring JTF_TOP.......COMPLETED
Configuring MWA_TOP.......COMPLETED
Configuring CN_TOP........COMPLETED
Configuring CSI_TOP.......COMPLETED
Configuring WIP_TOP.......COMPLETED
Configuring CSE_TOP.......COMPLETED
Configuring EAM_TOP.......COMPLETED
Configuring FTE_TOP.......COMPLETED
Configuring ONT_TOP.......COMPLETED
Configuring AR_TOP........COMPLETED
Configuring AHL_TOP.......COMPLETED
Configuring OZF_TOP.......COMPLETED
Configuring IES_TOP.......COMPLETED
Configuring CSD_TOP.......COMPLETED
Configuring IGC_TOP.......COMPLETED
AutoConfig completed successfully.
-bash-4.1$
Verify that the contents of the tnsnames.ora file are correct.
Ensure that the file’s entries correctly reference the idb1-vip and idb2-vip aliases. Confirm also that
FNDFS_APPLTOP_AT_eweblh references all the other APP Tiers.
If an address entry is missing on an APP Tier node entry, rerun AutoConfig on that node.
-bash-4.1$ cat $TNS_ADMIN/tnsnames.ora
###############################################################
#
# This file is automatically generated by AutoConfig. It will be read and
# overwritten. If you were instructed to edit this file, or if you are not
# able to use the settings created by AutoConfig, refer to Metalink Note
# 387859.1 for assistance.
#
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 18
#$Header: NetServiceHandler.java 120.19.12010000.6 2010/03/09 08:11:36 jmajumde ship $
#
###############################################################
VIS=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=idb1-vip.us.oracle.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=VIS)
(INSTANCE_NAME=VIS1)
)
)
VIS1=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=idb1-vip.us.oracle.com)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=idb1-vip.us.oracle.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=VIS)
(INSTANCE_NAME=VIS1)
)
)
VIS1_FO=
(DESCRIPTION_LIST=
(LOAD_BALANCE=NO)
(FAILOVER=YES)
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=idb1-vip.us.oracle.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=VIS)
(INSTANCE_NAME=VIS1)
)
)
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=idb1-vip.us.oracle.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=VIS)
(INSTANCE_NAME=VIS1)
)
)
)
VIS2=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=idb2-vip.us.oracle.com)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=idb2-vip.us.oracle.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=VIS)
(INSTANCE_NAME=VIS2)
)
)
VIS2_FO=
(DESCRIPTION_LIST=
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 19
(LOAD_BALANCE=NO)
(FAILOVER=YES)
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=idb2-vip.us.oracle.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=VIS)
(INSTANCE_NAME=VIS2)
)
)
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=idb2-vip.us.oracle.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=VIS)
(INSTANCE_NAME=VIS2)
)
)
)
VIS_FO=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=idb1-vip.us.oracle.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=VIS)
(INSTANCE_NAME=VIS1)
)
)
VIS_BALANCE=
(DESCRIPTION=
(ADDRESS_LIST=
(LOAD_BALANCE=YES)
(FAILOVER=YES)
(ADDRESS=(PROTOCOL=tcp)(HOST=idb2-vip.us.oracle.com)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=idb1-vip.us.oracle.com)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=idb2-vip.us.oracle.com)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=idb1-vip.us.oracle.com)(PORT=1521))
)
(CONNECT_DATA=
(SERVICE_NAME=VIS)
)
)
FNDFS_IOPMN1-LH=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN1-LH)(PORT=1626))
(CONNECT_DATA=
(SID=FNDFS)
)
)
FNDFS_IOPMN1-LH.us.oracle.com=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN1-LH)(PORT=1626))
(CONNECT_DATA=
(SID=FNDFS)
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 20
)
)
FNDFS_VIS_IOPMN1-LH=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN1-LH)(PORT=1626))
(CONNECT_DATA=
(SID=FNDFS)
)
)
FNDFS_VIS_IOPMN1-LH.us.oracle.com=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN1-LH)(PORT=1626))
(CONNECT_DATA=
(SID=FNDFS)
)
)
FNDFS_IOPMN2-LH=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN2-LH)(PORT=1626))
(CONNECT_DATA=
(SID=FNDFS)
)
)
FNDFS_IOPMN2-LH.us.oracle.com=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN2-LH)(PORT=1626))
(CONNECT_DATA=
(SID=FNDFS)
)
)
FNDFS_VIS_IOPMN2-LH=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN2-LH)(PORT=1626))
(CONNECT_DATA=
(SID=FNDFS)
)
)
FNDFS_VIS_IOPMN2-LH.us.oracle.com=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN2-LH)(PORT=1626))
(CONNECT_DATA=
(SID=FNDFS)
)
)
FNDFS_EWEB-LH=
(DESCRIPTION=
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 21
(ADDRESS=(PROTOCOL=tcp)(HOST=EWEB-LH.us.oracle.com)(PORT=1627))
(CONNECT_DATA=
(SID=FNDFS)
)
)
FNDFS_EWEB-LH.us.oracle.com=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=EWEB-LH.us.oracle.com)(PORT=1627))
(CONNECT_DATA=
(SID=FNDFS)
)
)
FNDFS_VIS_EWEB-LH=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=EWEB-LH.us.oracle.com)(PORT=1627))
(CONNECT_DATA=
(SID=FNDFS)
)
)
FNDFS_VIS_EWEB-LH.us.oracle.com=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=EWEB-LH.us.oracle.com)(PORT=1627))
(CONNECT_DATA=
(SID=FNDFS)
)
)
FNDFS_ICM1-LH=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ICM1-LH)(PORT=1627))
(CONNECT_DATA=
(SID=FNDFS)
)
)
FNDFS_ICM1-LH.us.oracle.com=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ICM1-LH)(PORT=1627))
(CONNECT_DATA=
(SID=FNDFS)
)
)
FNDFS_VIS_ICM1-LH=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ICM1-LH)(PORT=1627))
(CONNECT_DATA=
(SID=FNDFS)
)
)
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 22
FNDFS_VIS_ICM1-LH.us.oracle.com=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ICM1-LH)(PORT=1627))
(CONNECT_DATA=
(SID=FNDFS)
)
)
FNDSM_IOPMN1-LH_VIS=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN1-LH)(PORT=1626))
(CONNECT_DATA=
(SID=FNDSM)
)
)
FNDSM_IOPMN1-LH.us.oracle.com_VIS=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN1-LH)(PORT=1626))
(CONNECT_DATA=
(SID=FNDSM)
)
)
FNDSM_IOPMN2-LH_VIS=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN2-LH)(PORT=1626))
(CONNECT_DATA=
(SID=FNDSM)
)
)
FNDSM_IOPMN2-LH.us.oracle.com_VIS=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN2-LH)(PORT=1626))
(CONNECT_DATA=
(SID=FNDSM)
)
)
FNDSM_EWEB-LH_VIS=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=EWEB-LH.us.oracle.com)(PORT=1627))
(CONNECT_DATA=
(SID=FNDSM)
)
)
FNDSM_EWEB-LH.us.oracle.com_VIS=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=EWEB-LH.us.oracle.com)(PORT=1627))
(CONNECT_DATA=
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 23
(SID=FNDSM)
)
)
FNDSM_ICM1-LH_VIS=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ICM1-LH)(PORT=1627))
(CONNECT_DATA=
(SID=FNDSM)
)
)
FNDSM_ICM1-LH.us.oracle.com_VIS=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=ICM1-LH)(PORT=1627))
(CONNECT_DATA=
(SID=FNDSM)
)
)
FNDFS_APPLTOP_AT_eweblh=
(DESCRIPTION=
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=tcp)(HOST=EWEB-LH.us.oracle.com)(PORT=1627))
(ADDRESS=(PROTOCOL=tcp)(HOST=ICM1-LH)(PORT=1627))
(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN1-LH)(PORT=1626))
(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN2-LH)(PORT=1626))
)
(CONNECT_DATA=
(SID=FNDFS)
)
)
IFILE=/ebsfs/apps/VIS/inst/apps/VIS_eweb-lh/ora/10.1.2/network/admin/VIS_eweb-lh_ifile.ora
-bash-4.1$
Run AutoConfig on the Secondary Cluster
Repeat the step to run AutoConfig on the secondary cluster to populate the FND_NODES table as well as update
the relevant tnsnames.ora entries for the DB and APP nodes.
Access Oracle Data Guard broker.
root@etc18cn1dbiodom1:~# su – oracle
Oracle Corporation SunOS 5.11 11.3 December 2015
oracle@etc18cn1dbiodom1:~$
oracle@etc18cn1dbiodom1:~$ . VIS.env
oracle@etc18cn1dbiodom1:~$ dgmgrl sys/<password>
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 24
DGMGRL for Solaris Version 11.2.0.3.0 – 64bit Production
Copyright (c) 2000, 2009, Oracle. All Rights reserved.
Welcome to DGMGRL, type “help” for information.
Connected.
DGMGRL>
Switchover the database.
This step causes VISION to become the primary database.
DGMGRL> switchover to 'VISION'
Performing switchover NOW, please wait...
New primary database "VISION" is opening...
Operation requires shutdown of instance "VIS2" on database "VIS"
Shutting down instance "VIS2"...
ORACLE instance shut down.
Operation requires startup of instance "VIS2" on database "VIS"
Starting instance "VIS2"...
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "VISION"
Display database configuration.
DGMGRL> show configuration
Configuration - DR Orchestration for EBS
Protection Mode: MaxAvailability
Databases:
VISION - Primary database
VIS - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS
DGMGRL>
Configure the Oracle E-Business Suite 12.1.3 DB Context Files 25
Run the adautocfg.sh script on the nodes.
a. etc18cn1dbiodom1
b. etc18cn2dbiodom1
c. etc18cn1dbiodom1
d. deopmn1 (eweb-lh)
e. deopmn1 (iopmn1-lh)
f. deopmn2 (iopmn2-lh)
g. deopmn2 (icm1-lh)
Verify that the contents of the tnsnames.ora file are correct.
Verify the Contents of the FND_NODES Table
The table within the APPS schema should contain relevant hostnames for the primary and secondary clusters as follows:
root@etc18cn1dbiodom1:~# su – oracle
Oracle Corporation SunOS 5.11 11.3 December 2015
oracle@etc18cn1dbiodom1:~$
oracle@etc18cn1dbiodom1:~$ . VIS.env
oracle@etc18cn1dbiodom1:~$ sqlplus “/as sysdba”
SQL *Plus: Release 11.2.0.3.0 Production
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage
Management, OLAP,
Data Mining and Real Application Testing options
SQL>
SQL> select node_name, server_address, virtual_ip from fnd_nodes;
NODE_NAME SERVER_ADDRESS VIRTUAL_IP
---------------- -------------- ----------
EWEB-LH m.m.m.68
IOPMN2-LH 192.168.28.34
ETC11-CN01 idb1-vip
Configure Flashback on the Database 26
ETC18CN1DBIODOM1 idb1-vip
ICM1-LH 192.168.28.43
ETC11-CN02 idb2-vip
ETC18CN2DBIODOM1 idb2-vip
AUTHENTICATION *
IOPMN1-LH 192.168.28.33
9 rows selected.
SQL>
4. Configure Flashback on the Database
You can configure the database with flashback retention. The flashback logs can be used to recover the failed primary database to a point in time before the switchover to the standby database occurred and convert that database to a physical standby database.
By default, flashback retention is set to 1440 minutes or 1 day. You can set it to a different value depending on how much flashback data is kept in the fast recovery area.
You define the flashback retention time through the db_flashback_retention_target parameter.
Do the following steps:
Check the status of the databases.
The output shows that as a consequence of Step 2 in Section 3.4, VISION is the primary database while VIS is the standby database.
oracle@etc18cn1dbiodom1:~$ srvctl status db –d vision –v
Instance VIS1 is running on node etc18cn1dbiodom1. Instance status: Open.
Instance VIS1 is running on node etc18cn2dbiodom1. Instance status: Open.
oracle@etc11-cn01:~$ srvctl status db –d vision –v
Instance VIS1 is running on node etc11-cn01. Instance status: Mounted
(Closed).
Instance VIS1 is running on node etc11-cn02. Instance status: Mounted
(Closed).
Access the primary database.
root@etc18cn1dbiodom1:~# su – oracle
Oracle Corporation SunOS 5.11 11.3 December 2015
oracle@etc18cn1dbiodom1:~$
oracle@etc18cn1dbiodom1:~$ . VIS.env
oracle@etc18cn1dbiodom1:~$ sqlplus “/as sysdba”
Configure Flashback on the Database 27
SQL *Plus: Release 11.2.0.3.0 Production
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage
Management, OLAP,
Data Mining and Real Application Testing options
SQL>
Configure the flashback retention as shown.
SQL> alter database flashback on;
SQL> select flashback_on from v$database;
FLASHBACK_ON
------------------
YES
SQL>
SQL> alter system set db_flashback_retention_target=240 scope=both;
System altered.
SQL> show parameters db_flashback_retention_target
NAME TYPE VALUE
------------------------------------ ----------- -----------------------------
-
db_flashback_retention_target integer 240
SQL>
Access Oracle Data Guard broker.
root@etc18cn1dbiodom1:~# su – oracle
Oracle Corporation SunOS 5.11 11.3 December 2015
oracle@etc18cn1dbiodom1:~$
oracle@etc18cn1dbiodom1:~$ . VIS.env
oracle@etc18cn1dbiodom1:~$ dgmgrl sys/<password>
DGMGRL for Solaris Version 11.2.0.3.0 – 64bit Production
Configure Flashback on the Database 28
Copyright (c) 2000, 2009, Oracle. All Rights reserved.
Welcome to DGMGRL, type “help” for information.
Connected.
DGMGRL>
Display standby database information.
DGMGRL> show database ‘VIS’
Database – VIS
Role: PHYSICAL STAND BY
Intended State: APPLY-ON
Transport Lag: 0 seconds
Apply Lag: 0 seconds
Real Time Query: OFF
Instance(s):
VIS1 (apply instance)
VIS2
Database Status:
SUCCESS
Switchover the database.
This step causes VIS to become the primary database.
DGMGRL> switchover to 'VIS'
Performing switchover NOW, please wait...
New primary database "VIS" is opening...
Operation requires shutdown of instance "VIS2" on database "VISION"
Shutting down instance "VIS2"...
ORACLE instance shut down.
Operation requires startup of instance "VIS2" on database "VISION"
Starting instance "VIS2"...
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "VIS"
Configure Flashback on the Database 29
Display database configuration.
DGMGRL> show configuration
Configuration - DR Orchestration for EBS
Protection Mode: MaxAvailability
Databases:
VIS - Primary database
VISION - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS
DGMGRL>
Display both VIS and VISION database information.
DGMGRL> show database ‘VIS’
Database – VIS
Role: Primary
Intended State: TRANSPORT-ON
Instance(s):
VIS1
VIS2
Database Status:
SUCCESS
DGMGRL> show database ‘VISION’
Database – VISION
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds
Apply Lag: 0 seconds
Real Time Query: OFF
Instance(s)
VIS1 (apply instance)
VIS2
Database Status
SUCCESS
Configure Oracle Solaris Cluster Geo Edition 30
Check the status of the databases.
oracle@etc11-cn01:~$ srvctl status db –d vis –v
Instance VIS1 is running on node etc11-cn01. Instance status: Open
Instance VIS2 is running on node etc11-cn02. Instance status: Open
oracle@etc18cn1dbiodom1:~$ srvctl status db –d vision –v
Instance VIS1 is running on node etc18cn1dbiodom1. Instance status: Mounted
(Closed).
Instance VIS2 is running on node etc18cn2dbiodom1. Instance status: Mounted
(Closed).
On the new primary database, repeat Steps 2 and 3 to turn on flashback retention.
5. Configure Oracle Solaris Cluster Geo Edition
Oracle Solaris Cluster Geo Edition manages the disaster recovery orchestration by using a cluster partnership, protection groups, multigroups and sites.
Install Geographic Edition Software
Reference documentation:
How to Install Geographic Edition Software
Install the Oracle Solaris Cluster Geo Edition software on each Oracle Solaris Cluster node.
root@eopmn1:~# pkg install ha-cluster/group-package/ha-cluster-geo-full
Verify that the package installed successfully on each Oracle Solaris Cluster node.
root@eopmn1:~# pkg info ha-cluster/group-package/ha-cluster-geo-full
Name: ha-cluster/group-package/ha-cluster-geo-full
Summary: Oracle Solaris Cluster Geographic Edition full group package
Description: Oracle Solaris Cluster Geographic Edition full group package
Category: Meta Packages/Group Packages
State: Installed
Publisher: ha-cluster
Version: 4.3 (Oracle Solaris Cluster Geographic Edition 4.3.0.24.0)
Build Release: 5.11
Branch: 0.24.0
Configure Oracle Solaris Cluster Geo Edition 31
Packaging Date: August 26, 2015 11:33:37 PM
Size: 5.53 kB
FMRI: pkg://ha-cluster/ha-cluster/group-package/ha-cluster-geo-
full@4.3,5.11-0.24.0:20150826T233337Z
root@eopmn1:~#
Prepare Each Zone Cluster for Partner Membership
Reference documentation:
How to Prepare a Zone Cluster for Partner Membership
5.2.1. Set Up the Network Address for Each Zone Cluster
Check first that the correct entries are in the /etc/hosts file before you set the address for the
zone clusters.
On the primary cluster, set the network address for opmn-zc and cm-zc.
a. Verify zone cluster entries on the /etc/hosts file.
root@eopmn1:~# grep opmn-zc /etc/hosts
n.n.n.115 opmn-zc.us.oracle.com opmn-zc
root@eopmn1:~#
root@eopmn2:~# grep opmn-zc /etc/hosts
n.n.n.115 opmn-zc.us.oracle.com opmn-zc
root@eopmn2:~#
root@ecm1:~# grep cm-zc /etc/hosts
n.n.n.116 cm-zc.us.oracle.com cm-zc
root@ecm1:~#
root@ecm2:~# grep cm-zc /etc/hosts
n.n.n.116 cm-zc.us.oracle.com cm-zc
root@ecm2:~#
b. Set the zone clusters’ network address.
root@etc11cn01-appiodom1:~# clzc configure opmn-zc
clzc:opmn-zc> add net
clzc:opmn-zc:net> set address=opmn-zc
clzc:opmn-zc:net> end
clzc:opmn-zc> verify
clzc:opmn-zc> commit
Configure Oracle Solaris Cluster Geo Edition 32
clzc:opmn-zc> exit
root@etc11cn01-appiodom1:~#
root@etc11cn01-appiodom1:~# clzc configure cm-zc
clzc:cm-zc> add net
clzc:cm-zc:net> set address=cm-zc
clzc:cm-zc:net> end
clzc:cm-zc> verify
clzc:cm-zc> commit
clzc:cm-zc> exit
root@etc11cn01-appiodom1:~#
On the secondary cluster, set the network address for dr-opmn-zc and dr-cm-zc.
a. Verify zone cluster entries on the /etc/hosts file.
root@deopmn1:~# grep dr-opmn-zc /etc/hosts
m.m.m.81 dr-opmn-zc.us.oracle.com dr-opmn-zc
root@deopmn1:~#
root@deopmn2:~# grep dr-opmn-zc /etc/hosts
m.m.m.81 dr-opmn-zc.us.oracle.com dr-opmn-zc
root@deopmn2:~#
root@decm1:~# grep dr-cm-zc /etc/hosts
m.m.m.82 dr-cm-zc.us.oracle.com dr-cm-zc
root@decm1:~#
root@decm2:~# grep dr-cm-zc /etc/hosts
m.m.m.82 dr-cm-zc.us.oracle.com dr-cm-zc
root@decm2:~#
b. Set the zone clusters’ network address.
root@etc18app01:~# clzc configure dr-opmn-zc
clzc:dr-opmn-zc> add net
clzc:dr-opmn-zc:net> set address=dr-opmn-zc
clzc:dr-opmn-zc:net> end
clzc:dr-opmn-zc> verify
clzc:dr-opmn-zc> commit
clzc:dr-opmn-zc> exit
root@etc18app01:~# clzc configure dr-cm-zc
clzc:dr-cm-zc> add net
clzc:dr-cm-zc:net> set address=dr-cm-zc
clzc:dr-cm-zc:net> end
clzc:dr-cm-zc> verify
clzc:dr-cm-zc> commit
Configure Oracle Solaris Cluster Geo Edition 33
clzc:dr-cm-zc> exit
root@etc18app01:~#
5.2.2. Set Up Security on the Primary Cluster
In this step, you copy the security files for the common agent container to all zone cluster nodes on the primary cluster.
For the opmn-zc cluster nodes, perform the following steps:
a. On eopmn1, issue the commands as shown.
root@eopmn1:~# cacaoadm status
default instance is DISABLED at system startup.
default instance is not running.
root@eopmn1:~# cacaoadm stop
root@eopmn1:~# cacaoadm create-keys --force
root@eopmn1:~# cd /etc/cacao/instances/default
root@eopmn1:/etc/cacao/instances/default# tar cf /tmp/SECURITY.tar ./security
root@eopmn1:/etc/cacao/instances/default# scp /tmp/SECURITY.tar iopmn2:/tmp
Password:
SECURITY.tar 100% |*****************************| 984 KB 00:00
root@eopmn1:/etc/cacao/instances/default# cd
root@eopmn1:~# cacaoadm set-param network-bind-address=0.0.0.0
root@eopmn1:~# cacaoadm enable
root@eopmn1:~# cacaoadm start
b. On eopmn2, issue the commands as shown.
root@eopmn2:~# cacaoadm status
default instance is DISABLED at system startup.
default instance is not running.
root@eopmn2:~# cacaoadm stop
root@eopmn2:~# cd /etc/cacao/instances/default
root@eopmn2:/etc/cacao/instances/default# tar xf /tmp/SECURITY.tar
root@eopmn2:/etc/cacao/instances/default# cd
root@eopmn2:~# cacaoadm set-param network-bind-address=0.0.0.0
root@eopmn2:~# cacaoadm enable
root@eopmn2:~# cacaoadm start
Configure Oracle Solaris Cluster Geo Edition 34
For the cm-zc nodes, perform the following steps:
a. On ecm1, issue the commands as shown.
root@ecm1:~# cacaoadm status
default instance is DISABLED at system startup.
default instance is not running.
root@ecm1:~# cacaoadm stop
root@ecm1:~# cacaoadm create-keys --force
root@ecm1:~# cd /etc/cacao/instances/default
root@ecm1:/etc/cacao/instances/default# tar cf /tmp/SECURITY.tar ./security
root@ecm1:/etc/cacao/instances/default# scp /tmp/SECURITY.tar icm2:/tmp
Password:
SECURITY.tar 100% |*****************************| 984 KB 00:00
root@ecm1:/etc/cacao/instances/default# cd
root@ecm1:~# cacaoadm set-param network-bind-address=0.0.0.0
root@ecm1:~# cacaoadm enable
root@ecm1:~# cacaoadm start
b. On ecm2, issue the commands as shown.
root@ecm2:~# cacaoadm status
default instance is DISABLED at system startup.
default instance is not running.
root@ecm2:~# cacaoadm stop
root@ecm2:~# cd /etc/cacao/instances/default
root@ecm2:/etc/cacao/instances/default# tar xf /tmp/SECURITY.tar
root@ecm2:/etc/cacao/instances/default# cd
root@ecm2:~# cacaoadm set-param network-bind-address=0.0.0.0
root@ecm2:~# cacaoadm enable
root@ecm2:~# cacaoadm start
5.2.3. Set Up Security on the Secondary Cluster
In this step, you copy the security files for the common agent container to all zone cluster nodes on the secondary cluster.
For the dr-opmn-zc cluster nodes, perform the following steps:
a. On deopmn1, issue the commands as shown.
root@deopmn1:~# cacaoadm status
default instance is ENABLED at system startup.
Smf monitoring process:
15282
Configure Oracle Solaris Cluster Geo Edition 35
15284
Uptime: 0 day(s), 22:56
root@deopmn1:~# cacaoadm stop
root@deopmn1:~# cacaoadm create-keys --force
root@deopmn1:~# cd /etc/cacao/instances/default
root@deopmn1:/etc/cacao/instances/default# tar cf /tmp/SECURITY.tar
./security
root@deopmn1:/etc/cacao/instances/default# cd
root@deopmn1:~# scp /tmp/SECURITY.tar deopmn2:/tmp
Password:
SECURITY.tar 100% |*****************************| 985 KB 00:00
root@deopmn1:/etc/cacao/instances/default# cd
root@deopmn1:~# cacaoadm set-param network-bind-address=0.0.0.0
root@deopmn1:~# cacaoadm enable
root@deopmn1:~# cacaoadm start
b. On deopmn2, issue the commands as shown.
root@deopmn2:~# cacaoadm status
default instance is ENABLED at system startup.
Smf monitoring process:
29308
29309
Uptime: 0 day(s), 22:56
root@deopmn2:~# cacaoadm stop
root@deopmn2:~# cd /etc/cacao/instances/default
root@deopmn2:/etc/cacao/instances/default# tar xf /tmp/SECURITY.tar
root@deopmn2:/etc/cacao/instances/default# cd
root@deopmn2:~# cacaoadm set-param network-bind-address=0.0.0.0
root@deopmn2:~# cacaoadm enable
root@deopmn2:~# cacaoadm start
For the dr-cm-zc cluster nodes, perform the following steps:
a. On decm1, issue the commands as shown.
root@decm1:~# cacaoadm status
default instance is ENABLED at system startup.
Smf monitoring process:
Smf monitoring process:
18667
18668
Uptime: 0 day(s), 22:55
root@decm1:~# cacaoadm stop
root@decm1:~# cacaoadm create-keys --force
Configure Oracle Solaris Cluster Geo Edition 36
root@decm1:~# cd /etc/cacao/instances/default
root@decm1:/etc/cacao/instances/default# tar cf /tmp/SECURITY.tar ./security
root@decm1:/etc/cacao/instances/default# scp /tmp/SECURITY.tar decm2:/tmp
Password:
SECURITY.tar 100% |*****************************| 984 KB 00:00
root@decm1:/etc/cacao/instances/default# cd
root@decm1:~# cacaoadm set-param network-bind-address=0.0.0.0
root@decm1:~# cacaoadm enable
root@decm1:~# cacaoadm start
b. On decm2, issue the commands as shown.
root@decm2:~# cacaoadm status
default instance is ENABLED at system startup.
Smf monitoring process:
2668
2670
Uptime: 0 day(s), 22:55
root@decm2:~# cacaoadm stop
root@decm2:~# cd /etc/cacao/instances/default
root@decm2:/etc/cacao/instances/default# tar xf /tmp/SECURITY.tar
root@decm2:/etc/cacao/instances/default# cd
root@decm2:~# cacaoadm set-param network-bind-address=0.0.0.0
root@decm2:~# cacaoadm enable
root@decm2:~# cacaoadm start
Enable Geographic Edition Software
Reference documentation:
How to Enable the Geographic Edition Framework
This step enables the Geographic Edition infrastructure. You must enable it on all the clusters of the geographically separated clusters.
Enable the Geographic Edition framework on the primary cluster.
a. For the zone cluster opmn-zc, issue the commands on eopmn1 as shown.
root@eopmn1:~# geoadm start
... checking for management agent ...
... management agent check done ....
... starting product infrastructure ... please wait ...
Registering resource type <SUNW.HBmonitor>...done.
Registering resource type <SUNW.SCGeoInitSvc>...done.
Configure Oracle Solaris Cluster Geo Edition 37
Resource type <SUNW.scmasa> has been registered already
Registering resource type <SUNW.SCGeoZC>...done.
Creating scalable resource group <geo-clusterstate>...done.
Creating service tag management resource <geo-servicetag>...
Service tag management resource created successfully ....
Creating failover resource group <geo-infrastructure>...done.
Creating logical host resource <geo-clustername>...
Logical host resource created successfully ....
Creating resource <geo-hbmonitor> ...done.
Creating resource <geo-failovercontrol> ...done.
Bringing RG <geo-clusterstate> to managed state ...done.
Bringing resource group <geo-infrastructure> to managed state ...done.
Enabling resource <geo-clustername> ...done.
Enabling resource <geo-hbmonitor> ...done.
Enabling resource <geo-failovercontrol> ...done.
Node eopmn1: Bringing resource group <geo-infrastructure> online ...done.
Oracle Solaris Cluster Geographic Edition infrastructure started successfully.
root@eopmn1:~#
root@eopmn1:~# geoadm show
--- CLUSTER LEVEL INFORMATION ---
Oracle Solaris Cluster Geographic Edition software is active on : node eopmn1, cluster
opmn-zc
Command ran successfully.
root@eopmn1:~#
root@eopmn1:~# clrg status geo-clusterstate geo-infrastructure
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- ------
geo-clusterstate eopmn1 No Online
eopmn2 No Online
geo-infrastructure eopmn1 No Online
eopmn2 No Offline
root@eopmn1:~# clrs status -g geo-clusterstate, geo-infrastructure
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
geo-zc-sysevent eopmn1 Online_not_monitored Online_not_monitored
eopmn2 Online_not_monitored Online_not_monitored
geo_servicetag eopmn1 Online_not_monitored Online_not_monitored
eopmn2 Online_not_monitored Online_not_monitored
geo-failovercontrol eopmn1 Online Online - Service is online.
eopmn2 Offline Offline
Configure Oracle Solaris Cluster Geo Edition 38
geo-hbmonitor eopmn1 Online Online - Daemon OK
eopmn2 Offline Offline
geo-clustername eopmn1 Online Online - LogicalHostname
online.
eopmn2 Offline Offline
root@eopmn1:~#
b. For the zone cluster cm-zc, issue the commands on ecm1 as shown.
root@ecm1:~# geoadm start
... checking for management agent ...
... management agent check done ....
... starting product infrastructure ... please wait ...
Registering resource type <SUNW.HBmonitor>...done.
Registering resource type <SUNW.SCGeoInitSvc>...done.
Resource type <SUNW.scmasa> has been registered already
Registering resource type <SUNW.SCGeoZC>...done.
Creating scalable resource group <geo-clusterstate>...done.
Creating service tag management resource <geo-servicetag>...
Service tag management resource created successfully ....
Creating failover resource group <geo-infrastructure>...done.
Creating logical host resource <geo-clustername>...
Logical host resource created successfully ....
Creating resource <geo-hbmonitor> ...done.
Creating resource <geo-failovercontrol> ...done.
Bringing RG <geo-clusterstate> to managed state ...done.
Bringing resource group <geo-infrastructure> to managed state ...done.
Enabling resource <geo-clustername> ...done.
Enabling resource <geo-hbmonitor> ...done.
Enabling resource <geo-failovercontrol> ...done.
Node ecm1: Bringing resource group <geo-infrastructure> online ...done.
Oracle Solaris Cluster Geographic Edition infrastructure started successfully.
Smf monitoring process:
7175
7180
Uptime: 0 day(s), 23:11
root@ecm1:~#
root@ecm1:~# geoadm show
--- CLUSTER LEVEL INFORMATION ---
Oracle Solaris Cluster Geographic Edition software is active on : node ecm1, cluster dr-cm-
zc
Command ran successfully.
root@ecm1:~#
root@ecm1:~# clrg status geo-clusterstate geo-infrastructure
Configure Oracle Solaris Cluster Geo Edition 39
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- ------
geo-clusterstate ecm1 No Online
ecm2 No Online
geo-infrastructure ecm1 No Online
ecm2 No Offline
root@ecm1:~# clrs status -g geo-clusterstate, geo-infrastructure
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
geo-zc-systevent ecm1 Online_not_monitored Online_not_monitored
ecm2 Online_not_monitored Online_not_monitored
geo-servicetag ecm1 Online_not_monitored Online_not_monitored
ecm2 Online_not_monitored Online_not_monitored
geo-failovercontrol ecm1 Online Online - Service is online.
Ecm2 Offline Offline
geo-hbmonitor ecm1 Online Online - Daemon OK
ecm2 Offline Offline
geo-clustername ecm1 Online Online - LogicalHostname
online.
Ecm2 Offline Offline
root@ecm1:~#
Enable the Geographic Edition framework on the secondary cluster.
a. For the zone cluster dr-opmn-zc, issue the commands on deopm1 as shown.
root@deopmn1:~# geoadm start
... checking for management agent ...
... management agent check done ....
... starting product infrastructure ... please wait ...
Registering resource type <SUNW.HBmonitor>...done.
Registering resource type <SUNW.SCGeoInitSvc>...done.
Resource type <SUNW.scmasa> has been registered already
Registering resource type <SUNW.SCGeoZC>...done.
Creating scalable resource group <geo-clusterstate>...done.
Creating service tag management resource <geo-servicetag>...
Service tag management resource created successfully ....
Creating failover resource group <geo-infrastructure>...done.
Creating logical host resource <geo-clustername>...
Logical host resource created successfully ....
Configure Oracle Solaris Cluster Geo Edition 40
Creating resource <geo-hbmonitor> ...done.
Creating resource <geo-failovercontrol> ...done.
Bringing RG <geo-clusterstate> to managed state ...done.
Bringing resource group <geo-infrastructure> to managed state ...done.
Enabling resource <geo-clustername> ...done.
Enabling resource <geo-hbmonitor> ...done.
Enabling resource <geo-failovercontrol> ...done.
Node deopmn1: Bringing resource group <geo-infrastructure> online ...done.
Oracle Solaris Cluster Geographic Edition infrastructure started successfully.
root@deopmn1:~#
root@deopmn1:~# geoadm show
--- CLUSTER LEVEL INFORMATION ---
Oracle Solaris Cluster Geographic Edition software is active on : node deopmn1, cluster dr-
opmn-zc
Command ran successfully.
root@deopmn1:~#
root@deopmn1:~# clrg status geo-clusterstate geo-infrastructure
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- ------
geo-clusterstate deopmn1 No Online
deopmn2 No Online
geo-infrastructure deopmn1 No Online
deopmn2 No Offline
root@deopmn1:~# clrs status -g geo-clusterstate, geo-infrastructure
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
geo-zc-sysevent deopmn1 Online_not_monitored Online_not_monitored
deopmn2 Online_not_monitored Online_not_monitored
geo-servicetag deopmn1 Online_not_monitored Online_not_monitored
deopmn2 Online_not_monitored Online_not_monitored
geo-failovercontrol deopmn1 Online Online - Service is online.
deopmn2 Offline Offline
geo-hbmonitor deopmn1 Online Online - Daemon OK
deopmn2 Offline Offline
geo-clustername deopmn1 Online Online - LogicalHostname
online.
deopmn2 Offline Offline
Configure Oracle Solaris Cluster Geo Edition 41
root@deopmn1:~#
b. For the zone cluster dr-cm-zc, issue the commands on decm1 as shown.
root@decm1:~# geoadm start
... checking for management agent ...
... management agent check done ....
... starting product infrastructure ... please wait ...
Registering resource type <SUNW.HBmonitor>...done.
Registering resource type <SUNW.SCGeoInitSvc>...done.
Resource type <SUNW.scmasa> has been registered already
Registering resource type <SUNW.SCGeoZC>...done.
Creating scalable resource group <geo-clusterstate>...done.
Creating service tag management resource <geo-servicetag>...
Service tag management resource created successfully ....
Creating failover resource group <geo-infrastructure>...done.
Creating logical host resource <geo-clustername>...
Logical host resource created successfully ....
Creating resource <geo-hbmonitor> ...done.
Creating resource <geo-failovercontrol> ...done.
Bringing RG <geo-clusterstate> to managed state ...done.
Bringing resource group <geo-infrastructure> to managed state ...done.
Enabling resource <geo-clustername> ...done.
Enabling resource <geo-hbmonitor> ...done.
Enabling resource <geo-failovercontrol> ...done.
Node decm1: Bringing resource group <geo-infrastructure> online ...done.
Oracle Solaris Cluster Geographic Edition infrastructure started successfully.
root@decm1:~#
root@decm1:~# geoadm show
--- CLUSTER LEVEL INFORMATION ---
Oracle Solaris Cluster Geographic Edition software is active on : node decm1, cluster dr-
cm-zc
--- LOCAL NODE INFORMATION ---
Node decm1 does not host active product module.
Command ran successfully.
root@decm1:~#
root@decm1:~# clrg status geo-clusterstate geo-infrastructure
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- ------
geo-clusterstate decm1 No Online
decm2 No Online
Configure Oracle Solaris Cluster Geo Edition 42
geo-infrastructure decm1 No Online
decm2 No Offline
root@decm1:~# clrs status -g geo-clusterstate, geo-infrastructure
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
geo-zc-sysevent decm1 Online_not_monitored Online_not_monitored
decm2 Online_not_monitored Online_not_monitored
geo-servicetag decm1 Online_not_monitored Online_not_monitored
decm2 Online_not_monitored Online_not_monitored
geo-failovercontrol decm1 Online Online - Service is online.
decm2 Offline Offline
geo-hbmonitor decm1 Online Online - Daemon OK
decm2 Offline Offline
geo-clustername decm1 Online Online - LogicalHostname
online.
decm2 Offline Offline
root@decm1:~#
Configure Partnerships
The procedures in this section cover the following topics:
Configuration of partnership trusts
Creation of partnerships
Joining of partner clusters into the partnership
5.4.1. Configure Trust Between Partner Clusters
Reference documentation:
How to Configure Trust Between Two Clusters
Before creating a partnership between two clusters, you must first configure Geographic Edition software for secure communication between the two clusters. The configuration must be reciprocal.
On the primary cluster, configure partnership trusts for opmn-zc.
a. On eopmn1, create trust between opmn-zc and the remote dr-opmn-zc.
root@eopmn1:~# geops add-trust -c dr-opmn-zc
Configure Oracle Solaris Cluster Geo Edition 43
Local cluster : opmn-zc
Local node : eopmn1
Cleaning up certificate files in /etc/cacao/instances/default/security/jsse
on eopmn1
Retrieving certificates from dr-opmn-zc ... Done
New Certificate:
Owner: CN=deopmn1_agent
Issuer: CN=deopmn1_ca
Serial number: 701b5d1c
Valid from: Wed May 14 23:36:48 PDT 1969 until: Sun Oct 14 23:36:48 PDT 2035
Certificate fingerprints:
MD5: 0E:D6:7A:73:47:8D:7F:7F:7A:5A:38:AA:1B:5C:2D:62
SHA1: 57:62:C6:F2:AF:84:8A:AE:B0:18:2E:AE:B5:D8:AB:31:E3:87:7C:59
SHA256:
7C:16:92:01:94:0F:80:39:D3:56:30:86:5C:56:EF:69:97:70:EA:60:85:2F:FB:
6E:6C:EF:C0:72:28:0B:27:26
Signature algorithm name: SHA1withRSA
Version: 3
Extensions:
#1: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:false
PathLen: undefined
]
Do you trust this certificate? [y/n] y
Adding certificate to truststore on eopmn1 ... Done
Adding certificate to truststore on eopmn2 ... Done
New Certificate:
Owner: CN=deopmn1_ca
Issuer: CN=deopmn1_ca
Serial number: 3b569267
Valid from: Wed May 14 23:36:47 PDT 1969 until: Sun Oct 14 23:36:47 PDT 2035
Certificate fingerprints:
MD5: 8C:3C:9E:A9:4B:7E:E6:40:96:B5:E0:4F:29:64:29:0F
SHA1: 8B:D1:8D:0C:01:9D:11:75:E9:65:CE:C9:BA:B1:68:55:F4:96:78:0B
SHA256:
ED:58:C4:42:C9:55:12:37:5C:64:C1:34:4F:D3:E9:FB:37:68:7E:6D:87:7F:C2:8F:96:
Configure Oracle Solaris Cluster Geo Edition 44
06:FF:D3:6B:15:41:1A
Signature algorithm name: SHA1withRSA
Version: 3
Extensions:
#1: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:1
]
Do you trust this certificate? [y/n] y
Adding certificate to truststore on eopmn1 ... Done
Adding certificate to truststore on eopmn2 ... Done
Operation completed successfully. All certificates are added to truststore on
nodes of cluster opmn-zc
root@eopmn1:~#
b. On both eopmn1 and eopmn2, verify that the trust was created.
root@eopmn1:~# geops verify-trust -c dr-opmn-zc
Local cluster : opmn-zc
Local node : eopmn1
Retrieving information from cluster dr-opmn-zc ... Done
Verifying connections to nodes of cluster dr-opmn-zc:
eopmn1 -> deopmn2 / m.m.m.64 ... OK
eopmn1 -> deopmn1 / m.m.m.63 ... OK
Operation completed successfully, able to establish secure connections from
eopmn1 to all nodes of cluster dr-opmn-zc
root@eopmn1:~#
root@eopmn2:~# geops verify-trust -c dr-opmn-zc
Local cluster : opmn-zc
Local node : eopmn2
Retrieving information from cluster dr-opmn-zc ... Done
Verifying connections to nodes of cluster dr-opmn-zc:
Configure Oracle Solaris Cluster Geo Edition 45
eopmn2 -> deopmn2 / m.m.m.64 ... OK
eopmn2 -> deopmn1 / m.m.m.63 ... OK
Operation completed successfully, able to establish secure connections from
eopmn2 to all nodes of cluster dr-opmn-zc.
root@eopmn2:~#
On the primary cluster, configure partnership trusts for cm-zc.
a. On ecm1, create trust between cm-zc and the remote dr-cm-zc.
root@ecm1:~# geops add-trust -c dr-cm-zc
Local cluster : cm-zc
Local node : ecm1
Cleaning up certificate files in /etc/cacao/instances/default/security/jsse on
ecm1
Retrieving certificates from dr-cm-zc ... Done
New Certificate:
Owner: CN=decm1_agent
Issuer: CN=decm1_ca
Serial number: 63ee75ca
Valid from: Wed May 14 23:40:29 PDT 1969 until: Sun Oct 14 23:40:29 PDT 2035
Certificate fingerprints:
MD5: BD:4C:9A:55:BB:68:FB:61:95:A9:3F:8E:80:6A:B9:DE
SHA1: 71:62:6E:DF:E0:CA:A2:52:D8:D2:CC:63:5E:D3:2F:1B:6C:A1:BF:AB
SHA256: 19:87:53:BE:E8:93:F7:2D:36:D5:C7:04:17:45:70:0F:1F:A4:32:
47:39:F6:80:1E:52:0A:95:35:B0:96:78:1E
Signature algorithm name: SHA1withRSA
Version: 3
Extensions:
#1: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:false
PathLen: undefined
]
Do you trust this certificate? [y/n] y
Adding certificate to truststore on ecm1 ... Done
Configure Oracle Solaris Cluster Geo Edition 46
Adding certificate to truststore on ecm2 ... Done
New Certificate:
Owner: CN=decm1_ca
Issuer: CN=decm1_ca
Serial number: 1249454
Valid from: Wed May 14 23:40:28 PDT 1969 until: Sun Oct 14 23:40:28 PDT 2035
Certificate fingerprints:
MD5: 32:8D:3B:3F:83:8E:20:B1:55:95:5F:FC:87:76:57:55
SHA1: CE:18:C4:0D:AD:39:E4:04:A0:9D:1D:7B:03:E4:2B:C4:8A:9E:9D:89
SHA256: 52:0D:B8:C4:0E:BD:3E:D5:E6:8B:49:DB:66:9D:38:22:11:
0F:35:4D:B9:8B:BA:78:B1:26:C8:4F:21:0B:AD:67
Signature algorithm name: SHA1withRSA
Version: 3
Extensions:
#1: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:1
]
Do you trust this certificate? [y/n] y
Adding certificate to truststore on ecm1 ... Done
Adding certificate to truststore on ecm2 ... Done
Operation completed successfully. All certificates are added
to truststore on nodes of cluster cm-zc
root@ecm1:~#
b. On both ecm1 and ecm2, verify that the trust was created.
root@ecm1:~# geops verify-trust -c dr-cm-zc
Local cluster : cm-zc
Local node : ecm1
Retrieving information from cluster dr-cm-zc ... Done
Verifying connections to nodes of cluster dr-cm-zc:
ecm1 -> decm1 / m.m.m 65 ... OK
ecm1 -> decm2 / m.m.m.66 ... OK
Configure Oracle Solaris Cluster Geo Edition 47
Operation completed successfully, able to establish secure connections from
ecm1 to all nodes of cluster dr-cm-zc
root@ecm1:~#
root@ecm2:~# geops verify-trust -c dr-cm-zc
Local cluster : cm-zc
Local node : ecm2
Retrieving information from cluster dr-cm-zc ... Done
Verifying connections to nodes of cluster dr-cm-zc:
ecm2 -> decm1 / m.m.m 65 ... OK
ecm2 -> decm2 / m.m.m 66 ... OK
Operation completed successfully, able to establish secure connections from
ecm2 to all nodes of cluster dr-cm-zc
root@ecm2:~#
On the secondary cluster, configure partnership trusts for dr-opmn-zc.
a. On deopmn1, create trust between dr-opmn-zc and the remote opmn-zc.
root@deopmn1:~# geops add-trust -c opmn-zc
Local cluster : dr-opmn-zc
Local node : deopmn1
Cleaning up certificate files in /etc/cacao/instances/default/security/jsse
on deopmn1
Retrieving certificates from opmn-zc ... Done
New Certificate:
Owner: CN=eopmn1_agent
Issuer: CN=eopmn1_ca
Serial number: 4855faab
Valid from: Wed May 14 23:32:46 MeDT 1969 until: Sun Oct 14 22:32:46 MeST
2035
Certificate fingerprints:
MD5: 90:02:22:E5:06:FB:9C:6A:6C:BA:1C:EE:1F:55:14:6A
SHA1: 4A:8D:FD:7B:7D:29:54:06:70:A1:52:27:63:5A:13:C7:56:EA:AA:62
SHA256:
24:6D:88:5F:E6:AD:B5:06:00:84:96:FE:B9:C0:2E:B2:C3:B0:26:23:66:
D6:7A:33:13:F5:06:0C:21:EE:FE:2C
Signature algorithm name: SHA1withRSA
Version: 3
Configure Oracle Solaris Cluster Geo Edition 48
Extensions:
#1: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:false
PathLen: undefined
]
Do you trust this certificate? [y/n] y
Adding certificate to truststore on deopmn1 ... Done
Adding certificate to truststore on deopmn2 ... Done
New Certificate:
Owner: CN=eopmn1_ca
Issuer: CN=eopmn1_ca
Serial number: 5adf1784
Valid from: Wed May 14 23:32:45 MeDT 1969 until: Sun Oct 14 22:32:45 MeST
2035
Certificate fingerprints:
MD5: AB:9F:B8:79:A0:4B:66:48:26:64:62:43:1D:AD:00:47
SHA1: 18:5E:E8:C8:A2:FC:3C:B1:D2:AD:92:CF:1B:2F:AB:D4:12:44:04:5E
SHA256: 52:E6:97:A9:22:16:98:71:43:21:10:43:97:39:45:05:05:36:77:
99:AB:0D:65:0E:C6:55:7D:19:AB:26:9A:4A
Signature algorithm name: SHA1withRSA
Version: 3
Extensions:
#1: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:1
]
Do you trust this certificate? [y/n] y
Adding certificate to truststore on deopmn1 ... Done
Adding certificate to truststore on deopmn2 ... Done
Operation completed successfully. All certificates are added to truststore on
nodes of cluster dr-opmn-zc
root@deopmn1:~#
Configure Oracle Solaris Cluster Geo Edition 49
b. On both deopmn1 and deopmn2, verify that the trust was created.
root@deopmn1:~# geops verify-trust -c opmn-zc
Local cluster : dr-opmn-zc
Local node : deopmn1
Retrieving information from cluster opmn-zc ... Done
Verifying connections to nodes of cluster opmn-zc:
deopmn1 -> eopmn1 / n.n.n.22 ... OK
deopmn1 -> eopmn2 / n.n.n.23 ... OK
Operation completed successfully, able to establish secure connections from
deopmn1 to all nodes of cluster opmn-zc
root@deopmn1:~#
root@deopmn2:~# geops verify-trust -c opmn-zc
Local cluster : dr-opmn-zc
Local node : deopmn2
Retrieving information from cluster opmn-zc ... Done
Verifying connections to nodes of cluster opmn-zc:
deopmn2 -> eopmn1 / n.n.n.22 ... OK
deopmn2 -> eopmn2 / n.n.n.23 ... OK
Operation completed successfully, able to establish secure connections from
deopmn2 to all nodes of cluster opmn-zc
root@deopmn2:~#
On the secondary cluster, configure partnership trusts for dr-cm-zc.
a. On decm1, create trust between dr-cm-zc and the remote cm-zc.
root@decm1:~# geops add-trust -c cm-zc
Local cluster : dr-cm-zc
Local node : decm1
Cleaning up certificate files in /etc/cacao/instances/default/security
/jsse on decm1
Retrieving certificates from cm-zc ... Done
New Certificate:
Configure Oracle Solaris Cluster Geo Edition 50
Owner: CN=ecm1_agent
Issuer: CN=ecm1_ca
Serial number: 3e192ee1
Valid from: Wed May 14 23:13:50 MeDT 1969 until: Sun Oct 14 22:13:50 MeST 2035
Certificate fingerprints:
MD5: AB:25:0D:95:56:6B:14:DB:49:98:D2:D0:E1:30:71:E2
SHA1: DF:44:CC:6A:C5:3C:E7:D6:3B:DC:16:BB:BE:47:1C:F5:61:79:AB:00
SHA256: B8:87:52:7F:B4:32:2E:5E:9D:35:D1:22:75:44:FC:08:35:19:05:
E6:6A:D5:97:69:22:86:56:21:DA:1E:F7:A9
Signature algorithm name: SHA1withRSA
Version: 3
Extensions:
#1: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:false
PathLen: undefined
]
Do you trust this certificate? [y/n] y
Adding certificate to truststore on decm1 ... Done
Adding certificate to truststore on decm2 ... Done
New Certificate:
Owner: CN=ecm1_ca
Issuer: CN=ecm1_ca
Serial number: 19e8a71e
Valid from: Wed May 14 23:13:48 MeDT 1969 until: Sun Oct 14 22:13:48 MeST 2035
Certificate fingerprints:
MD5: B7:31:32:F3:7B:D3:6F:73:41:46:FE:88:B1:37:69:AB
SHA1: 1B:14:98:BC:C0:62:EE:21:31:2D:34:02:68:FC:A7:6F:68:D0:AC:06
SHA256: 3B:52:5E:47:10:FE:DB:E0:F1:B7:BA:42:A4:FB:AB:48:0F:43:
4A:A4:D7:E5:9C:6B:11:06:CF:0F:71:18:CE:E4
Signature algorithm name: SHA1withRSA
Version: 3
Extensions:
#1: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:1
]
Configure Oracle Solaris Cluster Geo Edition 51
Do you trust this certificate? [y/n] y
Adding certificate to truststore on decm1 ... Done
Adding certificate to truststore on decm2 ... Done
Operation completed successfully. All certificates are added to
truststore on nodes of cluster dr-cm-zc
root@decm1:~#
b. On both decm1 and decm2, verify that the trust was created.
root@decm1:~# geops verify-trust -c cm-zc
Local cluster : dr-cm-zc
Local node : decm1
Retrieving information from cluster cm-zc ... Done
Verifying connections to nodes of cluster cm-zc:
decm1 -> ecm1 / n.n.n.24 ... OK
decm1 -> ecm2 / n.n.n.25 ... OK
Operation completed successfully, able to establish secure connections from
decm1 to all nodes of cluster cm-zc
root@decm1:~#
root@decm2:~# geops verify-trust -c cm-zc
Local cluster : dr-cm-zc
Local node : decm2
Retrieving information from cluster cm-zc ... Done
Verifying connections to nodes of cluster cm-zc:
decm2 -> ecm1 / n.n.n.24 ... OK
decm2 -> ecm2 / n.n.n.25 ... OK
Operation completed successfully, able to establish secure connections from
decm2 to all nodes of cluster cm-zc
:~#
Configure Oracle Solaris Cluster Geo Edition 52
5.4.2. Create Partnerships
Reference documentation:
How to Create a Partnership
A partnership establishes heartbeat monitoring between two clusters that are running the Geographic Edition framework. Partnered clusters exchange heartbeats to monitor each other's presence and health. A partnership can involve only two clusters, and only one partnership can be defined between the clusters.
The steps that follow create partnerships between the two clusters as reflected in Figure 2:
“Partnership 1” involves the primary zone cluster opmn-zc and the standby zone cluster dr-opmn-zc. This partnership is created as opmn-ps.
“Partnership 2” involves the primary zone cluster cm-zc and the standby zone cluster dr-cm-zc. This partnership is created as cm-ps.
The procedure includes verification that the partnerships are reflected in both the primary and secondary clusters.
On the primary cluster, configure the partnerships.
a. On eopmn1, create a partnership between zone clusters opmn-zc and dr-opmn-zc.
root@eopmn1:~# geops create -c dr-opmn-zc opmn-ps
Partnership between local cluster "opmn-zc" and remote cluster "dr-opmn-zc"
successfully created.
root@eopmn1:~# geops status
Partnership "opmn-ps" : OK
Partner clusters : dr-opmn-zc
Synchronization : OK
ICRM Connection : OK
Heartbeat "hb_opmn-zc~dr-opmn-zc" monitoring "dr-opmn-zc": OK
Plug-in "ping_plugin" : Inactive
Plug-in "tcp_udp_plugin" : OK
root@eopmn1:~#
b. On eopmn2, verify the created partnership.
root@eopmn2:~# geops status
Partnership "opmn-ps" : OK
Partner clusters : dr-opmn-zc
Synchronization : OK
ICRM Connection : OK
Heartbeat "hb_opmn-zc~dr-opmn-zc" monitoring "dr-opmn-zc": OK
Configure Oracle Solaris Cluster Geo Edition 53
Plug-in "ping_plugin" : Inactive
Plug-in "tcp_udp_plugin" : OK
root@eopmn2:~#
c. On ecm1, create a partnership between zone clusters cm-zc and dr-cm-zc.
root@ecm1:~# geops create -c dr-cm-zc cm-ps
Partnership between local cluster "cm-zc" and remote cluster "dr-cm-zc"
successfully created.
root@ecm1:~# geops status
Partnership "cm-ps" : OK
Partner clusters : dr-cm-zc
Synchronization : OK
ICRM Connection : OK
Heartbeat "hb_cm-zc~dr-cm-zc" monitoring "dr-cm-zc": OK
Plug-in "ping_plugin" : Inactive
Plug-in "tcp_udp_plugin" : OK
root@ecm1:~#
d. On ecm2, verify the created partnership.
root@ecm2:~# geops status
Partnership "cm-ps" : OK
Partner clusters : dr-cm-zc
Synchronization : OK
ICRM Connection : OK
Heartbeat "hb_cm-zc~dr-cm-zc" monitoring "dr-cm-zc": OK
Plug-in "ping_plugin" : Inactive
Plug-in "tcp_udp_plugin" : OK
root@ecm2:~#
On the secondary cluster, verify the partnerships.
a. On deopmn1 and deopmn2, verify the partnership for the zone cluster dr-opmn-zc.
root@deopmn1:~# geops status
Configure Oracle Solaris Cluster Geo Edition 54
Partnership "opmn-ps" : OK
Partner clusters : opmn-zc
Synchronization : OK
ICRM Connection : OK
Heartbeat "hb_dr-opmn-zc~opmn-zc" monitoring "opmn-zc": OK
Plug-in "ping_plugin" : Inactive
Plug-in "tcp_udp_plugin" : OK
root@deopmn1:~#
root@deopmn2:~# geops status
Partnership "opmn-ps" : OK
Partner clusters : opmn-zc
Synchronization : OK
ICRM Connection : OK
Heartbeat "hb_dr-opmn-zc~opmn-zc" monitoring "opmn-zc": OK
Plug-in "ping_plugin" : Inactive
Plug-in "tcp_udp_plugin" : OK
root@deopmn2:~#
b. On decm1 and decm2, verify the partnership for dr-cm-zc.
root@decm1:~# geops status
Partnership "cm-ps" : OK
Partner clusters : cm-zc
Synchronization : OK
ICRM Connection : OK
Heartbeat "hb_dr-cm-zc~cm-zc" monitoring "cm-zc": OK
Plug-in "ping_plugin" : Inactive
Plug-in "tcp_udp_plugin" : OK
root@decm1:~#
root@decm2:~# geops status
Partnership "cm-ps" : OK
Partner clusters : cm-zc
Synchronization : OK
ICRM Connection : OK
Configure Oracle Solaris Cluster Geo Edition 55
Heartbeat "hb_dr-cm-zc~cm-zc" monitoring "cm-zc": OK
Plug-in "ping_plugin" : Inactive
Plug-in "tcp_udp_plugin" : OK
root@decm2:~#
5.4.3. Configure the Secondary Cluster to Join the Partnership
Reference documentation:
How to Join a Partnership
In this step, you complete the partnership configuration by defining the secondary cluster to become the second member of the partnership. You perform this step on the secondary cluster.
On deopmn1, configure the local cluster to join the cluster opmn-zc in the partnership opmn-ps.
root@deopmn1:~# geops join-partnership opmn-zc opmn-ps
Local cluster "dr-opmn-zc" is now partner of cluster "opmn-zc".
root@deopmn1:~#
Verify the partnership’s status.
root@deopmn1:~# geops show opmn-ps
Partnership: opmn-ps
Description :
Local heartbeat : hb_dr-opmn-zc~opmn-zc
Remote heartbeat : hb_opmn-zc~dr-opmn-zc
Notification_ActionCmd :
Notification_EmailAddrs :
root@deopmn1:~#
On decm1, configure the local cluster to join the cluster cm-zc in the partnership cm-ps.
root@decm1:~# geops join-partnership cm-zc cm-ps
Local cluster "dr-cm-zc" is now partner of cluster "cm-zc".
root@decm1:~#
Verify the partnership’s status.
root@decm1:~# geops show cm-ps
Configure Oracle Solaris Cluster Geo Edition 56
Partnership: cm-ps
Description :
Local heartbeat : hb_dr-cm-zc~cm-zc
Remote heartbeat : hb_cm-zc~dr-cm-zc
Notification_ActionCmd :
Notification_EmailAddrs :
root@decm1:~#
Configure Protection Groups
Protection groups enable a set of clusters to tolerate and recover from disaster by managing the resource groups for services. Protection groups can exist only in a partnership. Thus, you must first create partnerships before you can define protection groups.
In a protection group for a specific partnership, one partner cluster becomes the primary cluster while the other partner becomes the secondary cluster.
A protection group contains application resource groups and properties for managing data replication for those application resource groups.
The configuration for a protection group is identical on partner clusters. Therefore, partner clusters must have the application resource groups of the protection group defined in their configuration. The Geographic Edition framework propagates protection group configurations between partners.
5.5.1. Create the First Protection Group (opmn-ebs-pg)
Reference documentation:
Creating a Protection Group That Does Not Require Data Replication
Based on Figure 2, Protection Group 1 manages Web Entry Point (WEB) and Oracle Process Manager and Server (OPMN1 & OPMN2) services. Thus, the group is called opmn-ebs-pg. This group is set up for Partnership 1 (opmn-ps).
Ensure that the required resource groups are online.
root@eopmn1:~# clrg online -eM opmn1-lh-rg opmn2-lh-rg
root@eopmn1:~# clrg online -eM web-rg opmn1-rg opmn2-rg
root@eopmn1:~# clrg status
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- ------
web-rg eopmn1 No Online
eopmn2 No Offline
Configure Oracle Solaris Cluster Geo Edition 57
opmn1-lh-rg eopmn1 No Online
eopmn2 No Offline
opmn2-lh-rg eopmn2 No Online
eopmn1 No Offline
opmn1-rg eopmn1 No Online
opmn2-rg eopmn2 No Online
scalmnt-rg eopmn1 No Online
eopmn2 No Online
geo-clusterstate eopmn1 No Online
eopmn2 No Online
geo-infrastructure eopmn1 No Online
eopmn2 No Offline
On the primary cluster, perform the following steps:
a. Create the group.
root@eopmn1:~# geopg list
*** No protection group configured ***
root@eopmn1:~#
root@eopmn1:~# geopg create -s opmn-ps -o primary opmn-ebs-pg
Protection group "opmn-ebs-pg" successfully created.
root@eopmn1:~# geopg validate opmn-ebs-pg
Protection group "opmn-ebs-pg" successfully validated.
b. Configure the group.
root@eopmn1:~# clrg set -p Auto_start_on_new_cluster=false web-rg opmn1-rg
opmn2-rg
root@eopmn1:~# geopg set-prop -p External_Dependency_Allowed=true opmn-ebs-pg
Protection group "opmn-ebs-pg" successfully modified.
root@eopmn1:~#
root@eopmn1:~# geopg status opmn-ebs-pg
Configure Oracle Solaris Cluster Geo Edition 58
Protection group "opmn-ebs-pg" : OK
Partnership : opmn-ps
Synchronization : OK
Cluster opmn-zc : OK
Role : Primary
Activation state : Activated
Configuration : OK
Data replication : None
Resource groups : None
Cluster dr-opmn-zc : OK
Role : Secondary
Activation state : Activated
Configuration : OK
Data replication : None
Resource groups : None
c. Add resource groups to the protection group.
root@eopmn1:~# geopg add-resource-group web-rg,opmn1-rg,opmn2-rg opmn-ebs-pg
Following resource groups successfully added:
"web-rg,opmn1-rg,opmn2-rg".
root@eopmn1:~# geopg validate opmn-ebs-pg
Protection group "opmn-ebs-pg" successfully validated.
root@eopmn1:~#
root@eopmn1:~# geopg show opmn-ebs-pg
Protection Group: opmn-ebs-pg
Partnership name : opmn-ps
Local Role : Primary
Deployed on clusters : opmn-zc, dr-opmn-zc
Data Replication type : NONE
Configuration status : OK
Synchronization status : OK
Creation signature : opmn-zc Oct 16, 2015 5:42:31 PM PDT
Last update : Oct 16, 2015 4:54:19 PM MeST
Local properties : None
Global properties :
Configure Oracle Solaris Cluster Geo Edition 59
Description :
Timeout : 3600 seconds
RoleChange_ActionCmd :
External_Dependency_Allowed : true
RoleChange_ActionArgs :
Protected resource groups:
opmn2-rg
web-rg
opmn1-rg
*** No device groups defined in protection group "opmn-ebs-pg" ***
On the secondary cluster, create the same protection group by retrieving the configuration from the partner cluster.
root@deopmn1:~# geopg get -s opmn-ps opmn-ebs-pg
Protection group "opmn-ebs-pg" successfully created.
root@deopmn1:~# geopg validate opmn-ebs-pg
Protection group "opmn-ebs-pg" successfully validated.
On the primary cluster, start the protection group.
root@eopmn1:~# geopg start -e global opmn-ebs-pg
Processing operation... The timeout period for this operation on each cluster
is 3600 seconds...
Protection group "opmn-ebs-pg" successfully started.
root@eopmn1:~#
root@eopmn1:~# geopg validate opmn-ebs-pg
Protection group "opmn-ebs-pg" successfully validated.
root@eopmn1:~#
root@eopmn1:~# geopg status opmn-ebs-pg
Protection group "opmn-ebs-pg" : OK
Partnership : opmn-ps
Synchronization : OK
Configure Oracle Solaris Cluster Geo Edition 60
Cluster opmn-zc : OK
Role : Primary
Activation state : Activated
Configuration : OK
Data replication : None
Resource groups : OK
Cluster dr-opmn-zc : OK
Role : Secondary
Activation state : Activated
Configuration : OK
Data replication : None
Resource groups : OK
5.5.2. Create the Second Protection Group (cm-ebs-pg)
Reference documentation:
How to Create and Configure an Oracle ZFS Storage Appliance Protection Group
Based on Figure 2, Protection Group 2 manages Concurrent Manager (CM1) services as well as providing data replication for the Oracle ZFS Storage Appliance. Thus, the group is called cm-ebs-pg. This group is set up for Partnership 2 (cm-ps).
You create this protection group by creating and configuring the Oracle ZFS Storage Appliance replication.
On the primary cluster, perform the following steps:
a. Create a resource of SUNW.ScalMountPoint for the Concurrent Manager Logs.
root@ecm1:~# grep cmlogs /etc/vfstab
etc11-storIB:/export/ebslogs/ebsfs_cmlogs - /ebsfs_cmlogs nfs - no
rw,bg,hard,nointr,nosuid,rsize=131072,wsize=131072,proto=tcp,vers=3
root@ecm2:~# grep cmlogs /etc/vfstab
etc11-storIB:/export/ebslogs/ebsfs_cmlogs - /ebsfs_cmlogs nfs - no
rw,bg,hard,nointr,nosuid,rsize=131072,wsize=131072,proto=tcp,vers=3
root@ecm1:~# mount /ebsfs_cmlogs
root@ecm1:~# df -h /ebsfs_cmlogs
Filesystem Size Used Available Capacity Mounted on
etc11-storIB:/export/ebslogs/ebsfs_cmlogs
13T 31K 13T 1% /ebsfs_cmlogs
Configure Oracle Solaris Cluster Geo Edition 61
root@ecm2:~# mount /ebsfs_cmlogs
root@ecm2:~# df -h /ebsfs_cmlogs
Filesystem Size Used Available Capacity Mounted on
etc11-storIB:/export/ebslogs/ebsfs_cmlogs
13T 31K 13T 1% /ebsfs_cmlogs
root@ecm1:~# clrg create -S scalmnt2-rg
root@ecm1:~# clrs create -d -g scalmnt2-rg -t ScalMountPoint -x
MountPointDir=/ebsfs_cmlogs -x FileSystemType=nas -x TargetFileSystem=etc11-
storIB:/export/ebslogs/ebsfs_cmlogs cm-logs
root@ecm1:~#
root@ecm1:~# mkdir -p /var/geo/config/
root@ecm1:~# echo "ebs-apps-cmlogs|any|ecm2,ecm1" >
/var/geo/config/geozfssaconfig
root@ecm1:~# grep vnic1 /etc/hosts
n.n.n.90 etc11-vnic1.us.oracle.com etc11-vnic1
m.m.m.35 etc18-vnic1.us.oracle.com etc18-vnic1
root@ecm1:~#
root@ecm1:~# mkdir -p /var/tmp/geo
root@ecm1:~# cp /opt/ORCLscgrepzfssa/etc/zfssa_geo_config /var/tmp/geo/
root@ecm1:~# grep -v ^# /var/tmp/geo/zfssa_geo_config | grep -v ^$
PS=cm-ps
PG=cm-ebs-pg
REPCOMP=ebs-apps-cmlogs
REPRS=zfssa-rep-rs
REPRG=zfssa-rep-rg
DESC="Geo ZFSSA"
APPRG=scalmnt2-rg,cm-rg
CONFIGFILE=/var/geo/config/geozfssaconfig
LOCAL_CONNECT_STRING=geouser@etc11-vnic1
REMOTE_CONNECT_STRING=geouser@etc18-vnic1
CLUSTER_DGS=
root@ecm1:~#
root@ecm1:~# clrg offline cm-rg
root@ecm1:~# clrg offline scalmnt2-rg
root@ecm1:~# /opt/ORCLscgrepzfssa/util/zfssa_geo_register -f
/var/tmp/geo/zfssa_geo_config
sourcing /var/tmp/geo/zfssa_geo_config
Creating protection group "cm-ebs-pg"
This operation might take a while...
Protection group "cm-ebs-pg" successfully created.
Configure Oracle Solaris Cluster Geo Edition 62
Adding the replication component "ebs-apps-cmlogs" to the protection group
"cm-ebs-pg"
Password for property local_service_password :
Password for property remote_service_password :
Script-based plug-in configuration "ebs-apps-cmlogs" successfully added to
the protection group "cm-ebs-pg".
Run "geopg get --partnership cm-ps cm-ebs-pg" on the partner cluster if this
has not been done.
root@ecm1:~#
b. Add resource groups to the protection group.
root@ecm1:~# geopg add-resource-group cm-rg cm-ebs-pg
Following resource groups successfully added:
"cm-rg".
root@ecm1:~#
root@ecm1:~# geopg show cm-ebs-pg
Protection Group: cm-ebs-pg
Partnership name : cm-ps
Local Role : Primary
Deployed on clusters : cm-zc, dr-cm-zc
Data Replication type : SBP
Configuration status : OK
Synchronization status : OK
Creation signature : cm-zc Oct 16, 2015 9:14:17 AM PDT
Last update : Oct 16, 2015 9:31:10 AM PDT
Local properties : None
Global properties :
create_config_script :
/opt/ORCLscgrepzfssa/bin/zfssa_geo_control
Timeout : 3600 seconds
Description : Geo ZFSSA
add_app_rg_args :
remove_app_rg_args :
RoleChange_ActionCmd :
remove_config_script :
/opt/ORCLscgrepzfssa/bin/zfssa_geo_control
RoleChange_ActionArgs :
switchover_script :
/opt/ORCLscgrepzfssa/bin/zfssa_geo_control
Configure Oracle Solaris Cluster Geo Edition 63
takeover_script :
/opt/ORCLscgrepzfssa/bin/zfssa_geo_control
start_replication_script :
/opt/ORCLscgrepzfssa/bin/zfssa_geo_control
remove_app_rg_script : /bin/true
stop_replication_script :
/opt/ORCLscgrepzfssa/bin/zfssa_geo_control
add_app_rg_script :
/opt/ORCLscgrepzfssa/bin/zfssa_geo_control
External_Dependency_Allowed : true
configuration_file : /var/geo/config/geozfssaconfig
Protected resource groups:
cm-rg
scalmnt2-rg
SBP Script-based plug-in configurations:
ebs-apps-cmlogs
ebs-apps-cmlogs_create_config_args : -O ebs-apps-cmlogs -r zfssa-rep-
rs -g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-vnic1
ebs-apps-cmlogs_start_replication_args : -O ebs-apps-cmlogs -r zfssa-
rep-rs -g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-
vnic1
ebs-apps-cmlogs_switchover_args : -O ebs-apps-cmlogs -r zfssa-rep-rs
-g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-vnic1
ebs-apps-cmlogs_local_service_password : ********
ebs-apps-cmlogs_stop_replication_args : -O ebs-apps-cmlogs -r zfssa-
rep-rs -g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-
vnic1
ebs-apps-cmlogs_remote_service_password : ********
ebs-apps-cmlogs_remove_config_args : -O ebs-apps-cmlogs -r zfssa-rep-
rs -g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-vnic1
ebs-apps-cmlogs_takeover_args : -O ebs-apps-cmlogs -r zfssa-rep-rs -g
zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-vnic1
root@ecm1:~#
root@ecm1:~# geopg validate cm-ebs-pg
Protection group "cm-ebs-pg" successfully validated.
On the secondary cluster, create the same protection group by retrieving the configuration from the partner cluster.
root@decm1:~# geopg get -s cm-ps cm-ebs-pg
Protection group "cm-ebs-pg" successfully created.
Configure Oracle Solaris Cluster Geo Edition 64
root@decm1:~#
root@decm1:~# geopg validate cm-ebs-pg
Protection group "cm-ebs-pg" successfully validated.
On the primary cluster, start the protection group.
root@ecm1:~# geopg start -e global cm-ebs-pg
Processing operation... The timeout period for this operation on each cluster
is 3600 seconds...
Protection group "cm-ebs-pg" successfully started.
root@ecm1:~#
root@ecm1:~# geopg validate cm-ebs-pg
Protection group "cm-ebs-pg" successfully validated.
root@ecm1:~#
root@ecm1:~# geopg status cm-ebs-pg
Protection group "cm-ebs-pg" : OK
Partnership : cm-ps
Synchronization : OK
Cluster cm-zc : OK
Role : Primary
Activation state : Activated
Configuration : OK
Data replication : OK
Resource groups : OK
Cluster dr-cm-zc : OK
Role : Secondary
Activation state : Activated
Configuration : OK
Data replication : OK
Resource groups : OK
5.5.3. Create the Third Protection Group (cm-odg-pg)
Reference documentation:
How to Create and Configure an Oracle Data Guard Protection Group
Configure Oracle Solaris Cluster Geo Edition 65
Based on Figure 2, Protection Group 3 provides data replication for Oracle Data Guard. Thus, the group is called cm-odg-pg. This group is also set up for Partnership 2 (cm-ps).
On the primary cluster, perform the following steps:
a. Create the group.
root@ecm1:~# geopg create -s cm-ps -o primary -d odg cm-odg-pg
Protection group "cm-odg-pg" successfully created.
root@ecm1:~# geopg validate cm-odg-pg
Protection group "cm-odg-pg" successfully validated.
root@ecm1:~#
b. Configure the protection group.
This step creates the following for Oracle Solaris Cluster:
EBS-oracle-external-proxy-shadow-rg resource group
EBS-oracle-external-proxy-shadow-rs resource
root@ecm1:~# cat /var/cluster/geo/odg/EBS_config ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1
root@ecm1:~#
root@ecm1:~# geopg add-replication-component \
-p local_database_name=VIS \
-p local_db_service_name=VIS_DGMGRL \
-p remote_database_name=VISION \
-p remote_db_service_name=VISION_DGMGRL \
-p standby_type=physical \
-p local_oracle_svr_rg_name=oep-rg \
-p remote_oracle_svr_rg_name=oep-rg \
-p sysdba_username=sys \
-p sysdba_password= \
-p replication_mode=maxavailability EBS cm-odg-pg
Password for property sysdba_password :
Oracle Data Guard configuration "EBS" successfully added to the protection group "cm-odg-pg".
c. Add resource groups to the protection group.
root@ecm1:# geopg add-resource-group EBS-oracle-external-proxy-shadow-rg cm-odg-pg
Following resource groups successfully added:
"EBS-oracle-external-proxy-shadow-rg".
root@ecm1:#
root@ecm1:~# geopg validate cm-odg-pg
Protection group "cm-odg-pg" successfully validated.
Configure Oracle Solaris Cluster Geo Edition 66
root@ecm1:~#
root@ecm1:~# geopg show cm-odg-pg
Protection Group: cm-odg-pg
Partnership name : cm-ps
Local Role : Primary
Deployed on clusters : cm-zc, dr-cm-zc
Data Replication type : ODG
Configuration status : OK
Synchronization status : OK
Creation signature : cm-zc Oct 16, 2015 7:31:09 AM PDT
Last update : Oct 20, 2015 4:42:00 AM MeST
Local properties : None
Global properties :
Description :
Timeout : 3600 seconds
RoleChange_ActionCmd :
External_Dependency_Allowed : true
RoleChange_ActionArgs :
Protected resource groups:
EBS-oracle-external-proxy-shadow-rg
ODG Oracle Data Guard configurations:
EBS
EBS_remote_db_service_name : VIS_DGMGRL
EBS_remote_rac_proxy_svr_rg_name : oep-rg
EBS_replication_mode : maxavailability
EBS_sysdba_username : sys
EBS_local_db_service_name : VISION_DGMGRL
EBS_local_rac_proxy_svr_rg_name : oep-rg
EBS_sysdba_password : ********
EBS_local_database_name : VISION
EBS_standby_type : physical
EBS_remote_database_name : VIS
Configure Oracle Solaris Cluster Geo Edition 67
On the secondary cluster, create the same protection group by retrieving the configuration from the partner cluster.
root@decm1:~# geopg get -s cm-ps cm-odg-pg
Protection group "cm-odg-pg" successfully created.
root@decm1:~#
root@decm1:~# geopg validate cm-odg-pg
Protection group "cm-odg-pg" successfully validated.
On the primary cluster, start the protection group.
root@ecm1:~# geopg start -e global cm-odg-pg
Processing operation... The timeout period for this operation on each cluster
is 3600 seconds...
Protection group "cm-odg-pg" successfully started.
root@ecm1:~#
root@ecm1:~# geopg validate cm-odg-pg
Protection group "cm-odg-pg" successfully validated.
root@ecm1:~#
root@ecm1:~# geopg status cm-odg-pg
Protection group "cm-odg-pg" : OK
Partnership : cm-ps
Synchronization : OK
Cluster cm-zc : OK
Role : Primary
Activation state : Activated
Configuration : OK
Data replication : OK
Resource groups : OK
Cluster dr-cm-zc : OK
Role : Secondary
Activation state : Activated
Configuration : OK
Data replication : OK
Resource groups : OK
5.5.4. Display the Oracle Solaris Cluster Listings
After you have completed the configuration of Oracle Solaris Cluster Geo Edition, display the
listings.
Configure Oracle Solaris Cluster Geo Edition 68
Display the listings on the primary cluster.
a. Check the status of the opmn-zc zone cluster.
root@eopmn1:~# clrg status
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- -------
opmn1-lh-rg eopmn1 No Online
eopmn2 No Offline
opmn2-lh-rg eopmn2 No Online
eopmn1 No Offline
scalmnt-rg eopmn1 No Online
eopmn2 No Online
geo-clusterstate eopmn1 No Online
eopmn2 No Online
geo-infrastructure eopmn1 No Online
eopmn2 No Offline
web-rg eopmn1 No Online
eopmn2 No Offline
opmn1-rg eopmn1 No Online
opmn2-rg eopmn2 No Online
root@eopmn1:~#
root@eopmn1:~# clrs status
=== Cluster Resources ===
Resource Name Node State Status Message
------------- ----- ----- --------------
ebs-web-lsr eopmn1 Online Online - Service is online.
eopmn2 Offline Offline
ebs-web eopmn1 Online Online - Service is online.
eopmn2 Offline Offline
ebs-web-lh eopmn1 Online Online - LogicalHostname online.
eopmn2 Offline Offline - LogicalHostname offline.
ebs-opmn1-lh eopmn1 Online Online - LogicalHostname online.
eopmn2 Offline Offline - LogicalHostname offline.
ebs-opmn2-lh eopmn2 Online Online - LogicalHostname online.
eopmn1 Offline Offline
ebs-opmn1 eopmn1 Online Online
Configure Oracle Solaris Cluster Geo Edition 69
ebs-opmn1-lsr eopmn1 Online Online - Service is online.
ebs-opmn2 eopmn2 Online Online
ebs-opmn2-lsr eopmn2 Online Online - Service is online.
ebs-storage eopmn1 Online Online
eopmn2 Online Online
geo-zc-sysevent eopmn1 Online_not_monitored Online_not_monitored
eopmn2 Online_not_monitored Online_not_monitored
geo-servicetag eopmn1 Online_not_monitored Online_not_monitored
eopmn2 Online_not_monitored Online_not_monitored
geo-failovercontrol eopmn1 Online Online - Service is online.
eopmn2 Offline Offline
geo-hbmonitor eopmn1 Online Online - Daemon OK
eopmn2 Offline Offline
geo-clustername eopmn1 Online Online - LogicalHostname online.
eopmn2 Offline Offline
b. Check the status of the cm-zc zone cluster.
root@ecm1:~# clrg status
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
--------------- --------- --------- ------
oep-rg ecm1 No Online
ecm2 No Online
scalmnt-rg ecm1 No Online
ecm2 No Online
cm-rg ecm1 No Online
ecm2 No Offline
scalmnt2-rg ecm1 No Online
ecm2 No Online
geo-clusterstate ecm1 No Online
ecm2 No Online
geo-infrastructure ecm1 No Online
ecm2 No Offline
Configure Oracle Solaris Cluster Geo Edition 70
cm-odg-pg-odg-rep-rg ecm No Online
ecm2 No Offline
EBS-oracle-external-proxy-shadow-rg ecm1No Online
ecm2 No Online
zfssa-rep-rg ecm1 No Online
ecm2 No Offline
root@ecm1:~#
root@ecm1:~# clrs status
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------------- -------------- ------ ------------------
oep-rs ecm Online Online - Service VIS is UP [Instance is
OPEN]
ecm2 Online Online - Service VIS is UP [Instance is
OPEN]
ebs-storage ecm1 Online Online
ecm2 Online Online
ebs-cm ecm1 Online Online - Service is online.
ecm2 Offline Offline
ebs-cm-lsr ecm1 Online Online - Service is online.
ecm2 Offline Offline
ebs-cm-lh ecm1 Online Online - LogicalHostname online.
ecm2 Offline Offline - LogicalHostname offline.
cm-logs ecm1 Online Online
ecm2 Online Online
geo-zc-sysevent ecm1 Online_not_monitored
Online_not_monitored
ecm2 Online_not_monitored
Online_not_monitored
geo-servicetag ecm1 Online_not_monitored
Online_not_monitored
ecm2 Online_not_monitored
Online_not_monitored
geo-failovercontrol ecm1 Online Online - Service is online.
ecm2 Offline Offline
geo-hbmonitor ecm1 Online Online - Daemon OK
ecm2 Offline Offline
geo-clustername ecm1 Online Online - LogicalHostname online.
ecm2 Offline Offline
Configure Oracle Solaris Cluster Geo Edition 71
EBS-odg-rep-rs ecm1 Online Online - Data protection mode is
MaxAvailability
ecm2 Offline Offline
EBS-oracle-external-proxy-shadow-rs ecm1 Online Online - Service VIS is UP
[Instance is OPEN] (Data Guard Primary)
ecm2 Online Online - Service VIS is UP [Instance is
OPEN] (Data Guard Primary)
zfssa-rep-rs ecm1 Online Online - Sending update
ecm2 Offline Offline
Display the listings on the secondary cluster.
a. Check the status of the dr-opmn-zc zone cluster.
root@deopmn1:~# clrg status
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- ------
web-rg deopmn1 No Unmanaged
deopmn2 No Unmanaged
opmn1-lh-rg deopmn1 No Online
deopmn2 No Offline
opmn2-lh-rg deopmn2 No Online
deopmn1 No Offline
scalmnt-rg deopmn1 No Online
deopmn2 No Online
opmn1-rg deopmn1 No Unmanaged
opmn2-rg deopmn2 No Unmanaged
geo-clusterstate deopmn1 No Online
deopmn2 No Online
geo-infrastructure deopmn1 No Online
deopmn2 No Offline
root@deopmn1:~#
root@deopmn1:~# clrs status
=== Cluster Resources ===
Resource Name Node Name State Status Message
Configure Oracle Solaris Cluster Geo Edition 72
------------- --------- ----- --------------
ebs-web deopmn1 Offline Offline
deopmn2 Offline Offline
ebs-web-lsr deopmn1 Offline Offline
deopmn2 Offline Offline
ebs-web-lh deopmn1 Offline Offline
deopmn2 Offline Offline
ebs-opmn1-lh deopmn1 Online Online - LogicalHostname online.
deopmn2 Offline Offline
ebs-opmn2-lh deopmn2 Online Online - LogicalHostname online.
deopmn1 Offline Offline
ebs-storage deopmn1 Offline Online
deopmn2 Offline Online
ebs-opmn1 deopmn1 Offline Offline
ebs-opmn1-lsr deopmn1 Offline Offline
ebs-opmn2 deopmn2 Offline Offline
ebs-opmn2-lsr deopmn2 Offline Offline
geo-zc-sysevent deopmn1 Online_not_monitored Online_not_monitored
deopmn2 Online_not_monitored Online_not_monitored
geo-servicetag deopmn1 Online_not_monitored Online_not_monitored
deopmn2 Online_not_monitored Online_not_monitored
geo-failovercontrol deopmn1 Online Online - Service is online.
deopmn2 Offline Offline
geo-hbmonitor deopmn1 Online Online - Daemon OK
deopmn2 Offline Offline
geo-clustername deopmn1 Online Online - LogicalHostname online.
deopmn2 Offline Offline
b. Check the status of the dr-cm-zc zone cluster.
root@decm1:~# clrg status
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status
---------- --------- --------- ------
Configure Oracle Solaris Cluster Geo Edition 73
scalmnt-rg decm1 No Online
decm2 No Online
oep-rg decm1 No Online
decm2 No Online
cm-rg decm1 No Unmanaged
decm2 No Unmanaged
geo-clusterstate decm1 No Online
decm2 No Online
geo-infrastructure decm1 No Online
decm2 No Offline
cm-odg-pg-odg-rep-rg decm1 No Online
decm2 No Offline
EBS-oracle-external-proxy-shadow-rg decm1 No Unmanaged
decm2 No Unmanaged
zfssa-rep-rg decm1 No Online
decm2 No Offline
scalmnt2-rg decm1 No Unmanaged
decm2 No Unmanaged
root@decm1:~#
root@decm1:~# clrs status
=== Cluster Resources ===
Resource Name Node Name State Status Message
------------- --------- ----- --------------
ebs-storage decm1 Online Online
decm2 Online Online
oep-rs decm1 Online Online - Service VIS
is UP [Instance is MOUNTED]
decm2 Online Online - Service VIS
is UP [Instance is MOUNTED]
geo-zc-sysevent decm1 Online_not_monitored
Online_not_monitored
decm2 Online_not_monitored
Online_not_monitored
geo-servicetag decm1 Online_not_monitored
Configure Oracle Solaris Cluster Geo Edition 74
Online_not_monitored
decm2 Online_not_monitored
Online_not_monitored
geo-failovercontrol decm1 Online Online - Service is
online.
decm2 Offline Offline
geo-hbmonitor decm1 Online Online - Daemon OK
decm2 Offline Offline
geo-clustername decm1 Online Online -
LogicalHostname online.
decm2 Offline Offline
EBS-odg-rep-rs decm1 Online Online - Data
protection mode is MaxAvailability
decm2 Offline Offline
zfssa-rep-rs decm1 Online Online - Receiving
update
decm2 Offline Offline
ebs-cm decm1 Offline Offline
decm2 Offline Offline
ebs-cm-lsr decm1 Offline Offline
decm2 Offline Offline
ebs-cm-lh decm1 Offline Offline -
LogicalHostname offline.
decm2 Offline Offline -
LogicalHostname offline.
EBS-oracle-external-proxy-shadow-
rs decm1 Offline Offline
decm2 Offline Offline
cm-logs decm1 Offline Offline
decm2 Offline Offline
Configure Sites and Multigroups
A multigroup is a set of protection groups that is preconfigured for group migration between predetermined sets of clusters called sites. A switchover or takeover of all protection groups in a multigroup is performed in a single operation.
Configure Oracle Solaris Cluster Geo Edition 75
A protection group in a multigroup can be configured in a dependency chain, which defines which protection groups have a dependency on another protection group.
Likewise, a multigroup can contain multiple dependency chains. The dependency chain configuration determines the sequence protection groups must be taken offline and brought online. This sequence is based on the groups' roles in the dependency chain.
For example, an application that depends on a database needs to be stopped first before the database is shut down. In contrast, during restart operations after a switchover, the database takes precedence over the application.
Based on the Figure 1 and Figure 2, the two sites to be created are santaclara and hillsboro.
Reference documentation:
How to Create a Site
How to Create a Multigroup
5.6.1. Create and Configure the santaclara Site
Perform these steps on the primary cluster.
Create the site.
On this site, the zone cluster opmn-zc is the controller while cm-zc is a member.
root@eopmn1:~# geosite create -m cm-zc santaclara
Processing operation...might take several minutes to complete.
Site "santaclara" successfully created.
root@eopmn1:~#
root@ecm1:~# geosite join opmn-zc santaclara
Processing operation...might take several minutes to complete.
Local cluster "cm-zc" successfully joined site "santaclara" on cluster "opmn-
zc".
root@ecm1:~#
Create the multigroup at the site.
root@eopmn1:~# geomg create -s santaclara ebs-mg
Processing operation...might take several minutes to complete.
Multigroup "ebs-mg" successfully created.
root@eopmn1:~#
Add the protection groups to the multigroup.
This syntax means that later on, during multigroup switchover or takeover, opmn-ebs-pg and cm-ebs-pg can execute in parallel, but after the execution of cm-odg-pg. This arrangement makes sure that the database
Configure Oracle Solaris Cluster Geo Edition 76
services are recovered before the application services.
root@eopmn1:~# geomg add-protection-group "(opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-
pg)/cm-zc:cm-odg-pg" ebs-mg
The following protection group list was successfully added to multigroup "ebs-
mg":
"(opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-pg)/cm-zc:cm-odg-pg".
root@eopmn1:~#
Validate the multigroup.
root@eopmn1:~# geomg validate ebs-mg
Multigroup "ebs-mg" successfully validated.
root@eopmn1:~#
Display the multigroup’s status.
root@eopmn1:~# geomg status ebs-mg
Multigroup: ebs-mg
Description :
Site name : santaclara
Creation signature : opmn-zc Oct 16, 2015 6:14:52 PM PDT
Last update : Oct 16, 2015 6:15:50 PM PDT
Configuration status : OK
Synchronization status with :
cm-zc : OK
Timeout : 10920 seconds
Protection group list : (opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-pg)/cm-
zc:cm-odg-pg
Status of protection group "opmn-zc:opmn-ebs-pg" :
Global status : OK
Role : Primary
Activation state : Activated
Status of protection group "cm-zc:cm-ebs-pg" :
Global status : OK
Role : Primary
Activation state : Activated
Status of protection group "cm-zc:cm-odg-pg" :
Global status : OK
Role : Primary
Configure Oracle Solaris Cluster Geo Edition 77
Activation state : Activated
root@eopmn1:~#
5.6.2. Create and Configure the hillsboro Site
Perform these steps on the secondary cluster.
Create the site.
On this site, the zone cluster dr-opm-zc is the controller while the dr-cm-zc is a member.
root@deopmn1:~# geosite create -m dr-cm-zc hillsboro
Processing operation...might take several minutes to complete.
Site "hillsboro" successfully created.
root@deopmn1:~#
root@decm1:~# geosite join dr-opmn-zc hillsboro
Processing operation...might take several minutes to complete.
Local cluster "dr-cm-zc" successfully joined site "hillsboro" on cluster "dr-
opmn-zc".
root@decm1:~#
Create the multigroup at the site.
root@deopmn1:~# geomg create -s hillsboro ebs-mg
Processing operation...might take several minutes to complete.
Multigroup "ebs-mg" successfully created.
root@deopmn1:~#
Add the protection groups to the multigroup.
root@deopmn1:~# geomg add-protection-group "(dr-opmn-zc:opmn-ebs-pg,dr-cm-
zc:cm-ebs-pg)/dr-cm-zc:cm-odg-pg" ebs-mg
The following protection group list was successfully added to multigroup
"ebs-mg":
"(dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs-pg)/dr-cm-zc:cm-odg-pg".
root@deopmn1:~#
Create and Configure a DNS Server 78
Validate the multigroup.
root@deopmn1:~# geomg validate ebs-mg
Multigroup "ebs-mg" successfully validated.
root@deopmn1:~#
Display the multigroup’s status.
root@deopmn1:~# geomg status ebs-mg
Multigroup: ebs-mg
Description :
Site name : hillsboro
Creation signature : dr-opmn-zc Oct 16, 2015 5:19:33 PM MeST
Last update : Oct 16, 2015 5:29:16 PM MeST
Configuration status : OK
Synchronization status with :
dr-cm-zc : OK
Timeout : 10920 seconds
Protection group list : (dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs-
pg)/dr-cm-zc:cm-odg-pg
Status of protection group "dr-cm-zc:cm-odg-pg" :
Global status : OK
Role : Secondary
Activation state : Activated
Status of protection group "dr-cm-zc:cm-ebs-pg" :
Global status : OK
Role : Secondary
Activation state : Activated
Status of protection group "dr-opmn-zc:opmn-ebs-pg" :
Global status : OK
Role : Secondary
Activation state : Activated
root@deopmn1:~#
6. Create and Configure a DNS Server
This section provides steps in preparation for a switchover and takeover from the primary cluster to the secondary
Create and Configure a DNS Server 79
cluster.
Note: This section is a simplified illustration of how you can remap network host names by using a dedicated DNS server that can be remotely updated. However, you must choose a method appropriate to the network administration of your own data center.
Before a switchover and takeover, the client address eweb-lh needs to be changed from the IP address that is used by the primary cluster to the IP address that is used by the secondary cluster. A script is provided that modifies the DNS server so that the client address eweb-lh refers to the correct address.
Whenever a role change occurs, modifying the DNS server is required to ensure that the primary cluster always refers to the correct IP address for eweb-lh. For example, if the primary cluster is running at the Santa Clara site, the DNS server resolves the client address eweb-lh to the IP address n.n.n.27. If the primary cluster is running at the Hillsboro site, the DNS server resolves eweb-lh to the IP address m.m.m.68.
In the procedures that follow, the examples use ebsdns as the name of the DNS server.
Configure the DNS Service
Install the DNS package.
root@ebsdns:~# pkg install service/network/dns/bind
Creating Plan (Solver setup): |
Creating Plan (Evaluating mediators): \
Verify the installed packages.
root@ebsdns:~# pkg list |grep -i bind
network/dns/bind 9.6.3.11.1-0.175.2.6.0.3.0 i--
service/network/dns/bind 9.6.3.11.1-0.175.2.6.0.3.0 i--
Create the /etc/named.conf file.
root@ebsdns:~# cat /etc/named.conf
options {
directory "/var/named";
dump-file "/var/named/dump/named_dump.db";
statistics-file "/var/named/named.stats";
};
zone "." in {
type hint ;
file "named.root" ;
};
zone "us.oracle.com" in {
Create and Configure a DNS Server 80
type master ;
file "domain.us.oracle.com";
};
zone "in-addr.arpa" in {
type master ;
file "rdomain.us.oracle.com";
};
zone "0.0.127.in-addr.arpa" in {
type master ;
file "loopback";
};
logging {
category "default" { "debug"; };
category "general" { "debug"; };
category "database" { "debug"; };
category "security" { "debug"; };
category "config" { "debug"; };
category "resolver" { "debug"; };
category "xfer-in" { "debug"; };
category "xfer-out" { "debug"; };
category "notify" { "debug"; };
category "client" { "debug"; };
category "unmatched" { "debug"; };
category "network" { "debug"; };
category "update" { "debug"; };
category "queries" { "debug"; };
category "dispatch" { "debug"; };
category "dnssec" { "debug"; };
category "lame-servers" { "debug"; };
channel "debug" {
file "/tmp/nameddbg" versions 2 size 50m;
print-time yes;
print-category yes;
};
};
Create a forward lookup file for the DNS server.
In this file, the client address eweb-lh resolves to n.n.n.27.
root@ebsdns:~# cat /var/named/domain.us.oracle.com
; Forward map for us.oracle.com
$TTL 3h
Create and Configure a DNS Server 81
@ in soa ebsdns.us.oracle.com. root.ebsdns.us.oracle.com. (
20151019
43200
3600
604800
86400 )
in ns ebsdns.oracle.com.
in mx 10 mail.us.oracle.com.
ebsdns in A n.n.n.20
localhost in A 127.0.0.1
mail in CNAME ebsdns
www in CNAME ebsdns
ftp in CNAME ebsdns
eweb-lh in A n.n.n.27
root@ebsdns:~#
Create a reverse lookup file for the DNS server.
root@ebsdns:~# cat /var/named/loopback
; Reverse map for 0.0.127.in-addr.arpa.
$TTL 1h
@ in soa ebsdns.oracle.com. root.ebsdns.oracle.com. (
20151020
43200
3600
604800
86400 )
in ns ebsdns.oracle.com.
1 in ptr localhost.oracle.com.
root@ebsdns:~#
Enable the DNS server.
root@ebsdns:~# svcadm enable dns/server
root@ebsdns:~# svcs dns/server
STATE STIME FMRI
online 1:09:58 svc:/network/dns/server:default
root@ebsdns:~#
Create and Configure a DNS Server 82
Configure the DNS Update Scripts
Perform this configuration on each node in the zone clusters opmn-zc and dr-opmn-zc.
6.2.1. Create the dns_update.sh Script
On the primary cluster, the script remotely runs the dns_change_to_primaryip.sh script in the DNS server.
On the secondary cluster, the same script runs the dns_change_to_drip.sh script. Passwordless SSH has
already been configured for nodes of the opmn-zc and dr-opmn-zc clusters to access the DNS server.
On the primary cluster, create the scripts with execute permissions on the eopmn1 and eopmn2 nodes.
root@eopmn1:~# cat /var/geo/scripts/dns_update.sh
#!/bin/ksh
ssh ebsdns /var/geo/scripts/dns_change_to_primaryip.sh
root@eopmn1:~#
root@eopmn2:~# cat /var/geo/scripts/dns_update.sh
#!/bin/ksh
ssh ebsdns /var/geo/scripts/dns_change_to_primaryip.sh
root@eopmn2:~#
On the secondary cluster, create the scripts with execute permissions on the deopmn1 and deopmn2 nodes.
root@deopmn1:~# cat /var/geo/scripts/dns_update.sh
#!/bin/ksh
ssh ebsdns /var/geo/scripts/dns_change_to_drip.sh
root@deopmn1:~#
root@deopmn2:~# cat /var/geo/scripts/dns_update.sh
#!/bin/ksh
ssh ebsdns /var/geo/scripts/dns_change_to_drip.sh
root@deopmn2:~#
On the DNS server ebsdns, create two scripts with execute permissions as indicated.
a. Create the dns_change_to_drip.sh script.
root@ebsdns:~# cat /var/geo/scripts/dns_change_to_drip.sh
#!/bin/ksh
#
# This script sets eweb-lh hostname to m.m.m.68
#
Create and Configure a DNS Server 83
DB_hosts=/var/named/domain.us.oracle.com
DB_ip2host=/var/named/rdomain.us.oracle.com
/usr/bin/ed ${DB_ip2host} << EOF
/^27.n.n.n/s/27.n.n.n/68.m.m.m/
w
q
EOF
/usr/bin/ed ${DB_hosts} << EOF
/^eweb-lh/s/n.n.n.27/m.m.m.68/
w
q
EOF
svcadm restart dns/server
root@ebsdns:~#
b. Create the dns_change_to_primaryip.sh script.
root@ebsdns:~# cat /var/geo/scripts/dns_change_to_primaryip.sh
#!/bin/ksh
#
# This script sets eweb-lh hostname to n.n.n.27
#
DB_hosts=/var/named/domain.us.oracle.com
DB_ip2host=/var/named/rdomain.us.oracle.com
/usr/bin/ed ${DB_ip2host} << EOF
/^68.m.m.m/s/68.m.m.m/27.n.n.n/
w
q
EOF
/usr/bin/ed ${DB_hosts} << EOF
/^eweb-lh/s/m.m.m.68/n.n.n.27/
w
q
EOF
svcadm restart dns/server
root@ebsdns:~#
Create and Configure a DNS Server 84
6.2.2. Verify the DNS Update Scripts on the DNS Server
Run these commands.
root@ebsdns:~# cd /var/geo/scripts
root@ebsdns:/var/geo/scripts# ls -l
total 4
-rwxr-xr-x 1 root root 497 Oct 20 2015 dns_change_to_drip.sh
-rwxr-xr-x 1 root root 496 Oct 20 2015
dns_change_to_primaryip.sh
root@ebsdns:/var/geo/scripts#
root@ebsdns:/var/geo/scripts# ./dns_change_to_drip.sh
270
270
369
369
root@ebsdns:/var/geo/scripts#
root@ebsdns:/var/geo/scripts# nslookup eweb-lh
Server: n.n.n.20
Address: n.n.n.20#53
Name: eweb-lh.us.oracle.com
Address: m.m.m.68
root@ebsdns:/var/geo/scripts# ./dns_change_to_primaryip.sh
270
270
369
369
root@ebsdns:/var/geo/scripts#
root@ebsdns:/var/geo/scripts# nslookup eweb-lh
Server: n.n.n.20
Address: n.n.n.20#53
Name: eweb-lh.us.oracle.com
Address: n.n.n.27
root@ebsdns:/var/geo/scripts#
Configure the DNS Client
In the two zone clusters opmn-zc and dr-opmn-zc, their respective nodes have a constant mapping of eweb-lh to
the IP address in their /etc/hosts file. This mapping ensures that the eweb-lh within the cluster resolves to the
correct IP address for that cluster.
Create and Configure a DNS Server 85
On each zone cluster node, add the DNS server to /etc/hosts.
The nodes are eopmn1, eopmn2, deopmn1, and deopmn2.
root@eopmn1:~# grep ebsdns /etc/hosts
n.n.n.20 ebsdns
root@eopmn1:~#
root@eopmn2:~# grep ebsdns /etc/hosts
n.n.n.20 ebsdns
root@eopmn2:~#
root@deopmn1:~# grep ebsdns /etc/hosts
n.n.n.20 ebsdns
root@deopmn1:~#
root@deopmn2:~# grep ebsdns /etc/hosts
n.n.n.20 ebsdns
root@deopmn1:~#
On each DNS client machine, configure the name server.
Note: In your own company configuration, multiple name servers might exist for redundancy.
root# svccfg -s network/dns/client setprop config/nameserver = net_address: n.n.n.20
root# svccfg -s network/dns/client setprop config/domain = astring: us.oracle.com
root# svccfg -s network/dns/client setprop config/search = astring: us.oracle.com
root# svccfg -s name-service/switch setprop config/ipnodes = astring: '"files dns"'
root# svccfg -s name-service/switch setprop config/host = astring: '"files dns"'
root# svccfg -s network/dns/client listprop config
config application
config/value_authorization astring solaris.smf.value.name-service.dns.client
config/nameserver net_address n.n.n.20
config/domain astring us.oracle.com
config/search astring us.oracle.com
root#
Enable the DNS client on each client machine.
root# svcadm enable dns/client
root# svcs dns/client
STATE STIME FMRI
online Nov_08 svc:/network/dns/client:default
root#
Load Testing with Oracle Application Testing Suite (OATS) 86
On each DNS client machine, verify a name server lookup for eweb-lh.
root# nslookup eweb-lh
Server: n.n.n.20
Address: n.n.n.20#53
Name: eweb-lh.us.oracle.com
Address: n.n.n.27
root# nslookup n.n.n.27
Server: n.n.n.20
Address: n.n.n.20#53
27.n.n.n.in-addr.arpa name = eweb-lh.us.oracle.com.
root#
Configure the Protection Group Role Change Script
Add the dns_update.sh script to the RoleChange_ActionCmd property for the opmn-ebs-pg
protection group.
The system sets this property on the local cluster, and then propagates the value to the partner cluster.
root@eopmn1:~# geopg set-prop -p RoleChange_ActionCmd=/var/geo/scripts/dns_update.sh opmn-ebs-pg
Protection group "opmn-ebs-pg" successfully modified.
root@eopmn1:~#
7. Load Testing with Oracle Application Testing Suite (OATS)
This section describes a test performed using the disaster recovery configuration implemented in this documentation.
In this testing scenario, 600 virtual users perform online transactions using Oracle E-Business Suite 12.1.3. The test and subsequent graphs show the time disruption users experience during an outage and through the disaster recovery implementation, the recovery speed for all 600 virtual users to resume normal operations. There is a separate SQL script to submit batch jobs to the concurrent managers.
The Test Configuration
The test configuration is shown in the following figure:
Load Testing with Oracle Application Testing Suite (OATS) 87
Figure 3 - Primary and Standby Sites Configuration with Oracle Application Testing Suites
The test configuration consists of a Windows server that is using the Oracle Application Testing Suite (OATS) as a load generator. In its DNS configuration, the Windows server is set up to use the DNS Server ebsdns.
In a disaster recovery operation, the Oracle Solaris Cluster Geo Edition software modifies the eweb-lh entry on the DNS Server ebsdns. After that modification, the Oracle E-Business Suite 12.1.3 virtual users are directed to the correct SuperCluster where Oracle E-Business Suite 12.1.3 services are running.
Load Testing with Oracle Application Testing Suite (OATS) 88
OLTP Load Generator: the OATS Machine
Figure 4 shows that OATS is configured with 4 OATS scripts. Each script simulates 150 users across 4 different types of transactions for a total of 600 virtual users.
Figure 4 - Load Testing Configuration
The following list shows the parameters of the scripts:
Create_Query_Cash_Expenses 150 Virtual Users and OLT Server
Create_Query_Credit_Expenses 150 Virtual Users and OLT Server
TC_Entry 150 Virtual Users and OLT Server
View_Payslip 150 Virtual Users and OLT Server
Load Testing with Oracle Application Testing Suite (OATS) 89
OATS Testing Information
The test was run for approximately 15 minutes to measure the performance of Oracle E-Business Suite 12.1.3 in handling the workload of sustained usage by 600 virtual users.
Figure 5 shows the efficiency with which Oracle E-Business Suite 12.1.3 performed in supporting the transactions of 600 virtual users before and after disaster recovery.
Figure 5- OATS Test Performance Graph
Switchover ebs-mg Multigroup to the Secondary Cluster 90
Concurrent Manager Reports
In the test, an additional task was added to run concurrent to the constant OLTP transactions of the 600. The task simulated a DBA user logged to the SuperCluster DB node and repeatedly running a script that submits an Order-to-Cash job.
Figure 6 - Concurrent Manager Reports
Figure 6 shows the Oracle E-Business Suite System Administration screen for completed jobs. The screen displays the report name, status and links to the report details and output.
The Output column links to access the output files. In the test, part of the recovery verification was actual accessing of the output links. Overall, the entire test demonstrates the continuous replication feature of the ZFS SA for the Concurrent Manager output files based on the configuration prescribed in the architecture.
8. Switchover ebs-mg Multigroup to the Secondary Cluster
This section describes procedures for performing a switchover of the ebs-mg multigroup to the standby. The section first describes the steps for the switchover. Then timing information regarding the switchover operation is provided.
Switchover ebs-mg Multigroup to the Secondary Cluster 91
Switchover Operation
Check the status first before performing the switchover.
On the primary cluster, perform the following steps:
a. Check the status of the ebs-mg multigroup.
The command output shows that the santaclara site is activated and has the primary role for the three protection groups. Consequently, the hillsboro site has the secondary role for the protection groups.
root@eopmn1:~# geomg status ebs-mg
Multigroup: ebs-mg
Description :
Site name : santaclara
Creation signature : opmn-zc Oct 16, 2015 6:14:52 PM PDT
Last update : Oct 16, 2015 6:15:50 PM PDT
Configuration status : OK
Synchronization status with :
cm-zc : OK
Timeout : 10920 seconds
Protection group list : (opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-pg)/cm-
zc:cm-odg-pg
Status of protection group "opmn-zc:opmn-ebs-pg" :
Global status : OK
Role : Primary
Activation state : Activated
Status of protection group "cm-zc:cm-ebs-pg" :
Global status : OK
Role : Primary
Activation state : Activated
Status of protection group "cm-zc:cm-odg-pg" :
Global status : OK
Role : Primary
Activation state : Activated
root@eopmn1:~#
Switchover ebs-mg Multigroup to the Secondary Cluster 92
b. Check the status of the standby site.
root@deopmn1:~# geosite show -v hillsboro
Site: hillsboro
Description :
Creation signature : dr-opmn-zc Oct 16, 2015 5:18:02 PM MeST
Last update : Oct 16, 2015 5:19:33 PM MeST
Timeout : 30 seconds
Query_interval : 120 seconds
Controller "dr-opmn-zc"
Configuration : OK
Member "dr-cm-zc"
Configuration : OK
Local heartbeat : hb_dr-opmn-zc~hillsboro~dr-cm-zc
Multigroup: ebs-mg
Protection group list : (dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs-pg)/dr-cm-
zc:cm-odg-pg
The command output shows that for this site, the dr-opmn-zc is the
controller.
On the secondary cluster, perform the following steps:
a. Check the status of the ebs-mg multigroup.
This step shows that prior to the switchover, the ebs-mg multigroup on the standby site has the secondary role.
root@deopmn1:~# geomg status ebs-mg
Multigroup: ebs-mg
Description :
Site name : hillsboro
Creation signature : dr-opmn-zc Oct 16, 2015 5:19:33 PM MeST
Last update : Oct 16, 2015 5:29:16 PM MeST
Configuration status : OK
Synchronization status with :
dr-cm-zc : OK
Timeout : 10920 seconds
Switchover ebs-mg Multigroup to the Secondary Cluster 93
Protection group list : (dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs-
pg)/dr-cm-zc:cm-odg-pg
Status of protection group "dr-cm-zc:cm-odg-pg" :
Global status : OK
Role : Secondary
Activation state : Activated
Status of protection group "dr-cm-zc:cm-ebs-pg" :
Global status : OK
Role : Secondary
Activation state : Activated
Status of protection group "dr-opmn-zc:opmn-ebs-pg" :
Global status : OK
Role : Secondary
Activation state : Activated
root@deopmn1:~#
b. Switch the ebs-mg multigroup over to the standby site.
root@deopmn1:~# time geomg switchover -s hillsboro ebs-mg
Are you sure you want to switch over multigroup ‘ebs-mg’ to site ‘hillsboro’?
(yes|no) > yes
Processing operation... The timeout period for this multigroup operation is
10920 seconds...
Beginning "Switchover" operation on protection group "dr-cm-zc:cm-odg-pg"
Beginning "Switchover" operation on protection group "dr-opmn-zc:opmn-ebs-pg"
Beginning "Switchover" operation on protection group "dr-cm-zc:cm-ebs-pg"
"Switchover" operation on protection group "dr-cm-zc:cm-odg-pg" completed
successfully
"Switchover" operation on protection group "dr-opmn-zc:opmn-ebs-pg" completed
successfully
"Switchover" operation on protection group "dr-cm-zc:cm-ebs-pg" completed
successfully
"Switchover" operation succeeded for the multigroup "ebs-mg".
real 8m12.937s
user 0m3.078s
sys 0m2.151s
root@deopmn1:~#
Switchover ebs-mg Multigroup to the Secondary Cluster 94
c. Check the status of the multigroup.
This step shows that after the switchover, the protection groups in the standby site assume the primary role.
root@deopmn1:~# geomg status ebs-mg
Multigroup: ebs-mg
Description :
Site name : hillsboro
Creation signature : dr-opmn-zc Oct 16, 2015 5:19:33 PM MeST
Last update : Oct 16, 2015 5:29:16 PM MeST
Configuration status : OK
Synchronization status with :
dr-cm-zc : OK
Timeout : 10920 seconds
Protection group list : (dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs-
pg)/dr-cm-zc:cm-odg-pg
Status of protection group "dr-cm-zc:cm-odg-pg" :
Global status : OK
Role : Primary
Activation state : Activated
Status of protection group "dr-cm-zc:cm-ebs-pg" :
Global status : OK
Role : Primary
Activation state : Activated
Status of protection group "dr-opmn-zc:opmn-ebs-pg" :
Global status : OK
Role : Primary
Activation state : Activated
root@deopmn1:~#
Switchover ebs-mg Multigroup to the Secondary Cluster 95
Switchover Timing Information
The following figure shows timing information about the switchover operation.
Figure 7 - Oracle SuperCluster - E-Business Suite Disaster Recovery Orchestration demo – Services Availability Timing
Figure 7 provides the following data:
Site switchover time <10 minutes including validations at the start and end of the operation.
Actual service interruption ~6 minutes.
Takeover ebs-mg Multigroup from the Secondary Cluster 96
9. Takeover ebs-mg Multigroup from the Secondary Cluster
This section describes procedures for performing a takeover of the ebs-mg multigroup by the standby site. The section first describes the steps for the takeover. Then timing information regarding the takeover operation is provided.
Takeover Operation
On the primary (santaclara) site, issue a stop command at each SuperCluster compute node’s service processor.
The command causes an outage by powering off all the site’s nodes.
This step simulates an unplanned outage at the site.
Stop /SYS –f -script
On the standby site, issue the command to execute a takeover.
root@deopmn1:~# time geomg takeover -f -s hillsboro ebs-mg
Processing operation... The timeout period for this multigroup operation is
10920 seconds...
Beginning "Takeover" operation on protection group "dr-cm-zc:cm-odg-pg"
Beginning "Takeover" operation on protection group "dr-opmn-zc:opmn-ebs-pg"
Beginning "Takeover" operation on protection group "dr-cm-zc:cm-ebs-pg"
"Takeover" operation on protection group "dr-cm-zc:cm-odg-pg" completed
successfully
"Takeover" operation on protection group "dr-cm-zc:cm-ebs-pg" completed
successfully
"Takeover" operation on protection group "dr-opmn-zc:opmn-ebs-pg" completed
successfully
"Takeover" operation succeeded for the multigroup "ebs-mg".
real 10m19.878s
user 0m3.759s
sys 0m2.756s
root@deopmn1:~#
Access Oracle Data Guard broker.
root@etc18cn1dbiodom1:~# su – oracle
Oracle Corporation SunOS 5.11 11.3 December 2015
oracle@etc18cn1dbiodom1:~$
oracle@etc18cn1dbiodom1:~$ . VIS.env
oracle@etc18cn1dbiodom1:~$ dgmgrl sys/<password>
Takeover ebs-mg Multigroup from the Secondary Cluster 97
DGMGRL for Solaris Version 11.2.0.3.0 – 64bit Production
Copyright (c) 2000, 2009, Oracle. All Rights reserved.
Welcome to DGMGRL, type “help” for information.
Connected.
DGMGRL>
Check and verify that the standby database has become primary.
DGMGRL> show configuration
Configuration - DR Orchestration for EBS
Protection Mode: MaxAvailability
Databases:
VISION - Primary database
Warning: ORA-16629: database reports a different protection level from
the protection mode
VIS - Physical standby database (disabled)
ORA-16661: the standby database needs to be reinstated
Fast-Start Failover: DISABLED
Configuration Status:
WARNING
DGMGRL>
Check the status of the multigroup.
geomg status ebs-mg
Takeover ebs-mg Multigroup from the Secondary Cluster 98
Takeover Timing Information
The following figure shows timing information about the takeover operation.
Figure 8 - Oracle SuperCluster – E-Business Suite Disaster Recovery Orchestration demo – Services Availability Timing
Figure 8 provides the following data:
Site takeover time ~10 minutes including validations at the start and end of the operation.
Actual service interruption ~15 minutes, includes ~5 minutes of decision making time.
Restore the Primary Site 99
10. Restore the Primary Site
This section provides instructions for the restoration of the failed primary site that has been fixed.
Issue the following command.
The command powers up the server, and as a consequence, also boots up all the domains. However the software detects that the site has been taken over by the standby site. Thus, application and database services will remain offline.
Make sure to issue the command at each of the SuperCluster compute node’s service processor.
Start /SYS –f -script
When the primary site is ready for use again, reinstate its database as an active standby database.
a. Access Oracle Data Guard broker.
root@etc11-cn01:~# su – oracle
Oracle Corporation SunOS 5.11 11.3 December 2015
oracle@etc11-cn01:~$
oracle@etc11-cn01:~$ . VIS.env
oracle@etc11-cn01:~$ dgmgrl sys/<password>
DGMGRL for Solaris Version 11.2.0.3.0 – 64bit Production
Copyright (c) 2000, 2009, Oracle. All Rights reserved.
Welcome to DGMGRL, type “help” for information.
Connected.
DGMGRL>
b. Reinstate the database.
DGMGRL> reinstate database 'VIS'
Reinstating database "VIS", please wait...
Operation requires shutdown of instance "VIS1" on database "VIS"
Shutting down instance "VIS1"...
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "VIS1" on database "VIS"
Starting instance "VIS1"...
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.
Restore the Primary Site 100
Database mounted.
Continuing to reinstate database "VIS" ...
Reinstatement of database "VIS" succeeded
DGMGRL> show configuration
Configuration - DR Orchestration for EBS
Protection Mode: MaxAvailability
Databases:
VISION - Primary database
VIS - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS
DGMGRL>
Display the status of the multigroup at santaclara.
In the following example, the global status shows that the ebs-mg multigroup is in an error state. This state is expected after the former primary cluster of the protection groups is up and running after a takeover by a secondary cluster. The next three steps in this procedure remedy this error state.
root@eopmn1:~# geomg status ebs-mg
Multigroup: ebs-mg
Description :
Site name : santaclara
Creation signature : opmn-zc Oct 16, 2015 6:14:52 PM PDT
Last update : Oct 16, 2015 6:15:50 PM PDT
Configuration status : OK
Synchronization status with :
cm-zc : OK
Timeout : 10920 seconds
Protection group list : (opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-pg)/cm-zc:cm-odg-pg
Status of protection group "opmn-zc:opmn-ebs-pg" :
Global status : Error
Role : Primary
Activation state : Deactivated
Status of protection group "cm-zc:cm-ebs-pg" :
Global status : Error
Role : Primary
Activation state : Deactivated
Restore the Primary Site 101
Status of protection group "cm-zc:cm-odg-pg" :
Global status : Error
Role : Primary
Activation state : Deactivated
Update and start the protection group opmn-ebs-pg.
root@eopmn1:~# geopg update opmn-ebs-pg
WARNING:This operation overwrites the existing local configuration
for protection group "opmn-ebs-pg" with the configuration of the
remote cluster.
Check the configuration for "opmn-ebs-pg" on the
remote cluster before proceeding.
Do you want to continue (yes|no) > yes
Protection group "opmn-ebs-pg" successfully updated.
root@eopmn1:~#
root@eopmn1:~# geopg start -e global opmn-ebs-pg
Processing operation... The timeout period for this operation on each cluster
is 3600 seconds...
Protection group "opmn-ebs-pg" successfully started.
root@eopmn1:~# geopg status opmn-ebs-pg
Protection group "opmn-ebs-pg" : OK
Partnership : opmn-ps
Synchronization : OK
Cluster opmn-zc : OK
Role : Secondary
Activation state : Activated
Configuration : OK
Data replication : None
Resource groups : OK
Cluster dr-opmn-zc : OK
Role : Primary
Activation state : Activated
Configuration : OK
Data replication : None
Resource groups : OK
root@eopmn1:~#
Update and start the protection group cm-odg-pg.
root@ecm2:~# geopg update cm-odg-pg
WARNING:This operation overwrites the existing local configuration
Restore the Primary Site 102
for protection group "cm-odg-pg" with the configuration of the remote
cluster.
Check the configuration for "cm-odg-pg" on the
remote cluster before proceeding.
Do you want to continue (yes|no) > yes
Protection group "cm-odg-pg" successfully updated.
root@ecm2:~#
root@ecm2:~# geopg start -e global cm-odg-pg
Processing operation... The timeout period for this operation on each cluster
is 3600 seconds...
Protection group "cm-odg-pg" successfully started.
root@ecm2:~# geopg status cm-odg-pg
Protection group "cm-odg-pg" : OK
Partnership : cm-ps
Synchronization : OK
Cluster cm-zc : OK
Role : Secondary
Activation state : Activated
Configuration : OK
Data replication : OK
Resource groups : OK
Cluster dr-cm-zc : OK
Role : Primary
Activation state : Activated
Configuration : OK
Data replication : OK
Resource groups : OK
root@ecm2:~#
Update and start the protection group cm-ebs-pg.
The cm-ebs-pg group involves the ZFS SA replication. This step performs a cleanup on the ZFS SA to destroy the ebs-app-cmlogs share.
a. On the standby site’s ZFS SA, issue the commands as shown.
e tc18sn1:shares ebs-apps-cmlogs replication> ls
Actions:
TARGET STATUS NEXT
action-000 etc11-vnic1 idle Sync now
etc18sn1:shares ebs-apps-cmlogs replication> select action-000
Restore the Primary Site 103
etc18sn1:shares ebs-apps-cmlogs action-000> sendupdate
etc18sn1:shares ebs-apps-cmlogs action-000> set continuous=true
continuous = true (uncommitted)
etc18sn1:shares ebs-apps-cmlogs action-000> set enabled=true
enabled = true (uncommitted)
etc18sn1:shares ebs-apps-cmlogs action-000> commit
etc18sn1:shares ebs-apps-cmlogs action-000> done
etc18sn1:shares ebs-apps-cmlogs replication> ls
Actions:
TARGET STATUS NEXT
action-000 etc11-vnic1 sending Continuous
b. On the primary site’s ZFS SA, issue the commands as shown.
etc11-sn1:shares> ls
Properties:
pool = ssc
Projects:
QuorumProject
default
ebs-apps
ebs-apps-cmlogs
ebs-appzones
ebs-backup
ebs-db
iodomrpool
osc-backup
ssc
Children:
replication => Manage remote replication
schema => Define custom property schema
etc11-sn1:shares> destroy ebs-apps-cmlogs
This will destroy all data in "ebs-apps-cmlogs"! Are you sure? (Y/N) Y
etc11-sn1:shares> ls
Properties:
pool = ssc
Projects:
QuorumProject
default
ebs-apps
ebs-appzones
ebs-backup
Restore the Primary Site 104
ebs-db
iodomrpool
osc-backup
ssc
Children:
replication => Manage remote replication
schema => Define custom property schema
etc11-sn1:shares>
c. Update and start the cm-ebs-pg group.
root@ecm2:~# geopg update cm-ebs-pg
WARNING:This operation overwrites the existing local configuration
for protection group "cm-ebs-pg" with the configuration of the remote
cluster.
Check the configuration for "cm-ebs-pg" on the
remote cluster before proceeding.
Do you want to continue (yes|no) > yes
Protection group "cm-ebs-pg" successfully updated.
root@ecm2:~#
root@ecm2:~# geopg start -e global cm-ebs-pg
Processing operation... The timeout period for this operation on each cluster
is 3600 seconds...
Protection group "cm-ebs-pg" successfully started.
root@ecm2:~# geopg status cm-ebs-pg
Protection group "cm-ebs-pg" : OK
Partnership : cm-ps
Synchronization : OK
Cluster cm-zc : OK
Role : Secondary
Activation state : Activated
Configuration : OK
Data replication : OK
Resource groups : OK
Cluster dr-cm-zc : OK
Role : Primary
Activation state : Activated
Configuration : OK
Data replication : OK
Resource groups : OK
root@ecm2:~#
Restore the Primary Site 105
Verify the status on both clusters to ensure that normal operations have resumed.
a. Check the status of the primary site.
root@eopmn1:~# geomg status
Multigroup: ebs-mg
Description :
Site name : santaclara
Creation signature : opmn-zc Oct 16, 2015 6:14:52 PM PDT
Last update : Oct 16, 2015 6:15:50 PM PDT
Configuration status : OK
Synchronization status with :
cm-zc : OK
Timeout : 10920 seconds
Protection group list : (opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-pg)/cm-
zc:cm-odg-pg
Status of protection group "opmn-zc:opmn-ebs-pg" :
Global status : OK
Role : Secondary
Activation state : Activated
Status of protection group "cm-zc:cm-ebs-pg" :
Global status : OK
Role : Secondary
Activation state : Activated
Status of protection group "cm-zc:cm-odg-pg" :
Global status : OK
Role : Secondary
Activation state : Activated
root@eopmn1:~#
b. Check the status of the standby site.
root@deopmn1:~# geomg status
Multigroup: ebs-mg
Description :
Site name : hillsboro
Creation signature : dr-opmn-zc Oct 16, 2015 5:19:33 PM MeST
Apply SuperCluster Quarterly Maintenance Updates 106
Last update : Oct 16, 2015 5:29:16 PM MeST
Configuration status : OK
Synchronization status with :
dr-cm-zc : OK
Timeout : 10920 seconds
Protection group list : (dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs-
pg)/dr-cm-zc:cm-odg-pg
Status of protection group "dr-cm-zc:cm-odg-pg" :
Global status : OK
Role : Primary
Activation state : Activated
Status of protection group "dr-cm-zc:cm-ebs-pg" :
Global status : OK
Role : Primary
Activation state : Activated
Status of protection group "dr-opmn-zc:opmn-ebs-pg" :
Global status : OK
Role : Primary
Activation state : Activated
root@deopmn1:~#
(Optional): If the status of both sites indicates OK, switch over the ebs-mg multigroup to the original primary site.
The sample output is intentionally truncated.
root@deopmn1:~# time geomg switchover -s santaclara ebs-mg
Are you sure you want to switch over multigroup ‘ebs-mg’ to site
‘santaclara’? (yes|no) > yes
Processing operation... The timeout period for this multigroup operation is
10920 seconds...
11. Apply SuperCluster Quarterly Maintenance Updates
Periodically the SuperCluster needs to be patched with a Quarterly Maintenance Update (QMU). Every quarter, Oracle provides a SuperCluster Quarterly Full Stack Download Patches (QFSDP).
QFSDP is the only mechanism that delivers the proactive patching policy for all SuperClusters. Systems should not be kept out of date for more than 6 months (2 QFSDPs). Oracle makes no guarantees that SuperCluster patches older than 6 months will be maintained.
Apply SuperCluster Quarterly Maintenance Updates 107
The QFSDP should be applied as a whole unit in a single operation. Partial or selective patching is not
recommended. Reactive patching policy does allow individual patches to be applied to individual components to address critical issues with no reasonable workaround. However, before applying patches selectively, seek advice from Oracle Support first. Refer to the README files within the QFSDP for further details.
For example, for the project that is the basis of this document’s implementation, the Jan 2016 QFSDP was applied to upgrade the OS release from Solaris 11.2 to Solaris 11.3 and Oracle Solaris Cluster from 4.2 to 4.3.
Refer to the following MOS notes:
Contents of each SuperCluster Quarterly Full Stack Download Patch (QFSDP) (Doc ID 2056975.1)
SuperCluster QFSDP Installation Checklists (Doc ID 2090692.1)
The following is an extract from the QFSDP Installation checklist for T4-4 SuperCluster, January 2016 that is referenced from Doc ID 2090692.1.
The intent of this document is to provide a high-level overview of the Quarterly Full Stack Download Patch (QFSDP) installation. This document does not replace the requirement to read & follow the instructions in the QFSDP README files and the QFSDP SuperCluster Patching Best Practices document (Oracle Support Document 1569461.1), this document is meant as a navigation help and for tracking progress during the update process.
When planning a QFSDP, follow this recommended sequence.
Update the standby site first.
For example, in this document, the patch update is applied to the hillsboro site first.
After the update of the standby site, perform a planned switchover of the multigroup to the standby site.
For example, in this project, the ebs-mg multigroup is switched over to the hillsboro site as follows:
root@deopmn1:~# geomg switchover -s hillsboro ebs-mg
Are you sure you want to switch over multigroup ‘ebs-mg’ to site ‘hillsboro’?
(yes|no) > yes
Processing operation... The timeout period for this multigroup operation is
10920 seconds...
Beginning "Switchover" operation on protection group "dr-cm-zc:cm-odg-pg"
Beginning "Switchover" operation on protection group "dr-opmn-zc:opmn-ebs-pg"
Beginning "Switchover" operation on protection group "dr-cm-zc:cm-ebs-pg"
"Switchover" operation on protection group "dr-cm-zc:cm-odg-pg" completed
successfully
"Switchover" operation on protection group "dr-opmn-zc:opmn-ebs-pg" completed
successfully
"Switchover" operation on protection group "dr-cm-zc:cm-ebs-pg" completed
successfully
"Switchover" operation succeeded for the multigroup "ebs-mg".
root@deopmn1:~#
Apply SuperCluster Quarterly Maintenance Updates 108
After the standby site takes over the services, apply the same patch update to the primary site.
As an option, you can perform a load test during the switchover between the sites to monitor any performance changes after the QFSDP application.
The following figure shows the load testing graph when OATS was used to perform load testing after the standby site has been updated. No performance degradation observed.
Figure 9 – E-Business Suite Disaster Recovery Orchestration demo – Services Availability Timing
ORACLE CORPORATION
Worldwide Headquarters
500 Oracle Parkway, Redwood Shores, CA 94065 USA
Worldwide Inquiries
TELE + 1.650.506.7000 + 1.800.ORACLE1 FAX + 1.650.506.7200 oracle.com
CONNECT WITH US
Call +1.800.ORACLE1 or visit oracle.com. Outside North America, find your local office at oracle.com/contact.
blogs.oracle.com/oracle facebook.com/oracle twitter.com/oracle
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the
contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other
warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability
or fitness for a particular purpose. We specifically disclaim any liability with respect to this document, and no contractual
obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any
form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle and Java are
registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license
and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron
logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.
0818
Oracle E-Business Suite 12.1.3 Using Oracle Solaris Cluster Geo Edition – Implementation Guide
August 2018
Author: Neil Garthwaite, Gia-Khanh Nguyen