Installing a Cisco Prime Network Gateway and Embedded ... · Cisco Prime Network 3.8 Installation...

OL-24641-01

C H A P T E R 6
Installing a Cisco Prime Network Gateway and Embedded Database in a Veritas High Availability Configuration
This chapter explains how to install, upgrade, or roll back a Cisco Prime Network 3.8 gateway in a Veritas local redundancy and geographical redundancy high availability (HA) configuration.

This chapter includes:

• Veritas Gateway HA System Requirements, page 6-1

• Installing Cisco Prime Network 3.8 in a Veritas Gateway HA Environment, page 6-9

• Upgrading to Cisco Prime Network 3.8 in a Veritas Gateway HA Environment, page 6-84

• Rolling Back to an Earlier Cisco ANA Version in a Veritas Gateway HA Environment, page 6-89

Note The Oracle filenames, paths, and directories listed in this chapter are specific to Oracle 11g. Cisco Prime Network 3.8 also supports Oracle 10g. If you are using Oracle 10g, the filenames and paths will differ. The mount points are identical for Oracle 10g and 11g.

Note A Cisco Prime Network base license must be registered and activated within 120 days of installation or upgrade for each server in the Cisco Prime Network gateway HA configuration. If there are no valid license files on the gateway server, Cisco Prime Network acts as an evaluation version. This means that it has full functionality for 120 days after installation, and then it expires and UI connections will be disallowed. For Cisco Prime Networklicensing information, see the Cisco Prime Network 3.8 Administrator Guide.

Veritas Gateway HA System RequirementsThis section explains the system requirements for a Veritas-based HA configuration.

Veritas HA Gateway RequirementsAll gateways that will be configured for Veritas-based HA must meet the general Cisco Prime Network gateway requirements specified in Table 1-1 on page 1-3.

6-1Cisco Prime Network 3.8 Installation Guide

http://www.cisco.com/en/US/products/ps11879/prod_maintenance_guides_list.html

http://www.cisco.com/en/US/products/ps11879/prod_maintenance_guides_list.html

Chapter 6 Installing a Cisco Prime Network Gateway and Embedded Database in a Veritas High Availability Veritas Gateway HA System Requirements

Veritas Gateway HA Hardware Requirements for Local RedundancyFigure 6-1 illustrates the hardware requirements for the Cisco Prime Network gateway in a dual-node, local redundancy HA solution.

Figure 6-1 Cisco Prime Network Local Redundancy HA Hardware Configuration

The gateway requires:

• 2 servers of identical configuration that meet the Cisco Prime Network gateway hardware requirements, plus the following additional hardware requirements:

– 2 x internal hard disks with at least 73 GB storage.

– 4 x Gigabit Ethernet ports.

– 2 to 4 x SCSI/SAS/fiber connectors (depending on the external disk configuration).

• External disk storage that:

– Is supported by Veritas Cluster Server (VCS).

– Contains enough physical disks to provide the following volumes:

— 1 volume for Cisco Prime Network.

— 1 volume for the Oracle application and data file(s).

— 1 volume for Oracle redo logs.

— 1 volume for Oracle archives.

Cisco Prime Networkgateway

2 internal disks(1 OS + 1 mirror)

Oracle database server2 internal disks

(1 OS + 1 mirror )

Dual Gigabit Ethernet crossover

connections for heartbeat

Dual Gigabit Ethernet connections to different switches onLAN backbone for network and backup heartbeat

Dual connections from each server to the external disk storage unit

2828

60

External storage

Database:1 or more data volumes

1 archive volume1 redo log volume

1 backup volume (embedded database only)1 SRL volume, all with mirroring/RAID protection

Cisco Prime Network:1 ANA volume with mirroring/RAID protection1 SRL volume with mirroring/RAID protection


OL-24641-01


— 1 volume for Oracle backups (only if you are using an embedded database).

Note The database administrator determines the size needed for Oracle redo logs and archives.

— 1 or more volumes for additional data files (optional).

Note Oracle disk requirements depend on how much event history the operator wants to retain. See Chapter 4, “Creating and Configuring an Oracle Database.”

– Contains enough physical disks to implement mirroring or hardware RAID protection for all data and log disks.

– Contains enough physical disks to ensure optimal performance. It is recommended that Oracle data and logs reside on separate physical disks.

• Enough SCSI/SAS/fiber connections and cables to implement two connections between each server and each disk external storage unit.

• 2 x Gigabit Ethernet cables (for LAN connections).

• 2 x Gigabit Ethernet crossover cables (for the heartbeat).

Servers

The local cluster consists of two servers. Because Cisco Prime Network3.8 supports running the Oracle database application on a separate server from the gateway, both servers in the cluster are active during normal operation. One server runs the Cisco Prime Network gateway application; the other server runs the Oracle database application. Each server provides backup for the other in case of failure.

Connections

The Veritas Cluster Server uses Group Membership Services/Atomic Broadcast (GAB) and Low Latency Transport (LLT) to determine cluster membership and maintain communication via heartbeats. The two servers that form the cluster exchange heartbeats over the dual crossover Ethernet cables that connect them. If the connection between the servers fails, the heartbeat is sent across the LAN backbone. For added redundancy, the two servers can be connected to the LAN on separate segments.

Storage

The local redundancy HA solution requires internal and external storage for the two servers. Internally, each server must contain dual mirrored internal disks with at least 73 GB of storage. The Veritas Volume Manager performs the mirroring.

For external disk storage, the dual-node configuration requires five or more shared volumes (depending on how many Oracle data files you want to keep). The volumes must be available for mounting on either of the two servers. The external storage unit housing the disks for these volumes must be supported by the VCS. Beyond these minimum requirements, the data must be protected from disk failure. The five or more shared volumes can be provided to the servers in various ways; the physical implementation is transparent. See the Veritas product documentation for the latest list of supported devices.


OL-24641-01


If you use Just a Bunch of Disks (JBOD), redundancy requires that the external storage consist of six to ten separate disks with capacities large enough to provide the necessary volumes. The Veritas Volume Manager (VxVM) provides mirroring between each pair of disks. Oracle data and logs must reside on separate physical disks. The need to create additional Oracle data volumes on a separate disk is determined by the size and performance of the disks.

If you use external storage that supports hardware RAID, five or more shared volumes are required; the disk configuration is transparent. Oracle performance requirements can be met with the hardware and without having to place the data and logs on separate physical disks. The need to create additional Oracle data volumes is determined by performance considerations; the size of the initial data volume is not limited by physical disk sizes.

Veritas Gateway HA Software Requirements for Local RedundancyFor local redundancy, the cluster consists of two nodes. Each node has the following software requirements:

• 1 x Prime Network Starter Kit.

• 1 x Veritas Storage Foundation Standard HA Solaris v5.1 (licensing is either per server or per CPU), including:

– Veritas Cluster Server (VCS) Release 5.1.

– Veritas Volume Manager (VxVM) Release 5.1.

– Veritas Cluster Server Oracle Agent Release 5.1.

• Solaris 10 operating system that meets the Cisco Prime Network gateway software requirements, plus all recommended Solaris 10 patches.

Veritas Gateway HA Network Requirements for Local RedundancyTwo additional IP addresses are required for the cluster logical addresses of Cisco Prime Network and Oracle.

Veritas Gateway HA Hardware Requirements for Geographical RedundancyFigure 6-2 illustrates the hardware requirements for the Cisco Prime Network gateway in a geographical redundancy HA solution.


OL-24641-01


Figure 6-2 Cisco Prime Network Geographical Redundancy HA Secondary Site Hardware

Configuration

The hardware requirements for a single-node, geographical redundancy gateway at the secondary site are in addition to the hardware requirements for the primary site.

The secondary gateway requires:

• 1 server that meets the Cisco Prime Network gateway hardware requirements, plus 2 Gigabit Ethernet ports.

• External disk storage that:

– Is supported by VCS.

– Contains enough physical disks to provide the following volumes:

— 1 volume for Cisco Prime Network.

— 1 volume for the Oracle application and data file(s).

— 1 volume for Oracle redo logs.

— 1 volume for Oracle archives.

— 1 volume for Oracle backups (only if you are using an embedded database).

Note The database administrator determines the size needed for Oracle redo logs and archives.

— 1 or more volumes for additional data files (optional).

Cisco Prime Network gateway1 to 2 internal disks

(1 OS + 0 or 1 mirror)

External storage

Database:1 or more data volumes

1 archive volume1 redo log volume

1 backup volume (embedded database only)1 SRL volume, all with mirroring/RAID protection

Cisco Prime Network:1 ANA volume with mirroring/RAID protection1 SRL volume with mirroring/RAID protection

Dual Gigabit Ethernet connections to different switches on LAN backbone for network and heartbeat

2828

61

Dual connections from the server to the external disk storage unit


OL-24641-01


Note Oracle disk requirements depend on how much event history the operator wants to retain. See Chapter 4, “Creating and Configuring an Oracle Database.”

— 4 volumes for storage replicator logs (SRLs; two per site) that meet the SRL sizing requirements. (See SRL, page 6-7 for the recommended sizes.)

– Contains enough physical disks to implement mirroring or hardware RAID protection for all data and log disks, including the SRLs at both sites (if you are using Veritas Volume Replicator [VVR]). It is recommended that SRLs (for VVR) reside on separate physical disks. For more information, see the VVR documentation.

– Contains enough physical disks to ensure optimal performance. It is recommended that Oracle data and logs reside on separate physical disks.

– Contains disks with enough space that they can be divided into volumes with sizes identical to the volumes at the local site.

– Is supported by VCS Global Cluster, if you are using storage-based replication instead of VVR.

• Hardware infrastructure for storage-based data replication, if you are using storage-based replication instead of VVR.

• Enough SCSI/SAS/fiber connections and cables to implement two connections between each server and each disk external storage unit.

• 2 x Gigabit Ethernet cables (for LAN connections).

Server

The secondary site requires only a single server, which runs both the Cisco Prime Network gateway and the Oracle database.

Connections

For the global cluster, VCS uses Low Latency Transport (LLT) to determine cluster membership and maintain communication via heartbeats. The two clusters that form the global cluster exchange heartbeats over the IP network. For redundancy, the servers are connected to the LAN on separate segments.

Storage

The geographical redundancy HA solution requires internal and external storage for the remote server. Internally, the server must contain a single internal disk with at least 73 GB of storage.

For external disk storage, the single-node configuration requires five or more shared volumes (depending on how many Oracle data files you want to keep). The volumes must be available for mounting on the single server. To allow data replication, the volume sizes at the two sites must be identical. If you are using VVR, each site also requires an additional two volumes (one each for the Prime-Network and Oracle disk groups) for SRLs. The external storage unit housing the disks for these volumes must be supported by VCS.

Even though the secondary site is considered temporary, if the primary site fails, the storage at the secondary site becomes the primary copy of the Cisco Prime Network and Oracle data. Therefore, the data must be protected from disk failure. According to the VVR documentation, SRLs must be mirrored. As with the primary site, you determine the level of protection or redundancy for the external disks.


OL-24641-01


If you use JBOD disks, the SRLs must reside on separate physical disks. If you use external storage that supports hardware RAID, SRL performance requirements can be met with the hardware and without having to place the SRL on a separate physical disk.

You can implement storage-based data replication (either SAN or NAS) between the sites. The requirements for storage-based data replication are in addition to the requirements discussed previously.

Replication

You can implement data replication between the primary and secondary sites in one of the following ways:

• Software-based replication (VVR).

• Storage-based replication (NAS or SAN)—External storage supporting replication and the accompanying hardware infrastructure must be in place. This method is more expensive than software-based replication, but provides better performance.

SRL

A storage replicator log is a circular buffer of write for a replicated volume group (RVG). SRLs store outstanding write until system and network availability sends them across the network to the secondary site. The required size for an SRL is calculated based on the rate of disk writes, network availability, and the length of potential network outages.

Cisco Prime Network uses two RVGs—one for the Cisco Prime Network data and one for the Oracle data. The following are the recommended SRL sizes, assuming a 24-hour outage:

• Prime-Network RVG SRL volume size: 20 GB

• Oracle RVG SRL volume size: 50 GB

It is recommended that you reserve extra disk space for future growth. For example, if data reservation is required for more than a 24-hour outage, the SRL size increases linearly. The Oracle SRL volume size might also change according to the Oracle archiving mode. If ARCHIVELOG mode is used, the amount of data being written increases substantially, and the Oracle SRL volume must be adjusted accordingly.

After the system is installed and activated, see the Veritas Volume Replicator Advisor User’s Guide from Symantec publications to determine the specific SRL log size requirements.

Redundancy

The single-node cluster is intended to be used only if the primary site fails. Assuming the single-node cluster will be in use for a short time until the primary site is restored, there is no requirement for redundancy, other than mirroring the disk that contains the Cisco Prime Network and Oracle data. However, if the single-node cluster might operate for an extended period of time, or if you want to add redundancy, the cluster can be augmented at the secondary site as follows:

• Use a dual-node cluster to protect against server failure.

• Mirror the internal disk to protect against internal disk failure.

• Add a second connection to each external disk unit to protect against connector failure.


OL-24641-01


Veritas Gateway HA Software Requirements for Geographical RedundancyFor geographical redundancy, the cluster consists of two nodes at the local site and a single node at the remote site, for a total of three nodes. Each node has the following software requirements:

• 1 x Prime Network Starter Kit.

• 1 x Storage Foundation Enterprise HR/DR for Oracle, Solaris v5.1 (licensing is either per server or per CPU), including:

– Veritas Cluster Server (VCS) Release 5.1.

– Veritas Volume Manager (VxVM) Release 5.1.

– Veritas Cluster Server Oracle Agent Release 5.1.

• 3 x Veritas Cluster Server Global Cluster option, Solaris.

• 3 x Veritas Volume Replicator (VVR) Release 5.1 (if you are using software-based replication), or software licenses for storage-based data replication (SAN or NAS).

• Solaris 10 operating system that meets the Cisco Prime Network gateway software requirements, plus all recommended Solaris 10 patches.

Veritas Gateway HA Network Requirements for Geographical Redundancy

IP Addresses

Two additional IP addresses are required for the wide area connection (WAC) resources used to implement the VCS global cluster—one IP address for the local cluster, and one IP address for the remote cluster. An additional IP address is required for a cluster logical address on the remote cluster.

Cisco Prime Network Ports and Protocols

Ports and protocols must be open between the primary and secondary sites to allow communication during a site-to-site failover. Communication between the primary and secondary sites includes:

• Northbound applications <—> gateway

• Gateway <—> units

• Units <—> units

• Units <—> network elements

For more information, see Cisco Prime Network Server, HTTP, TCP, and UDP Ports, page 1-12.

VCS Ports and Protocols

To implement the global cluster, VCS uses the WAC to maintain contact between the clusters. By default, the WAC uses TCP 14155, and the VCS service uses TCP 14141. (See Table 6-1.) These ports must remain open between the primary and secondary sites. See the VCS documentation before changing these ports.


OL-24641-01

Chapter 6 Installing a Cisco Prime Network Gateway and Embedded Database in a Veritas High Availability Installing Cisco Prime Network 3.8 in a Veritas Gateway HA Environment

VVR Ports and Protocols

VVR transmits heartbeats and data between the primary and secondary sites. Heartbeats always use UDP, while data uses UDP (default) or TCP. Table 6-2 lists the default port for each protocol, which must be open between the sites if VVR is used for data replication. You can change the default ports as described in the VVR documentation.

Installing Cisco Prime Network 3.8 in a Veritas Gateway HA Environment

This section describes how to install Cisco Prime Network in the following HA configurations:

• For local redundancy, a dual-node cluster is installed at the primary site (see Figure 6-3).

• For geographical redundancy, an additional single-node cluster is installed at the remote site (see Figure 6-4) and with units (Figure 6-5).

Table 6-1 Open Ports Between Primary and Secondary Sites

Port Number Description

TCP 14155 WAC

TCP 14141 VCS service

Table 6-2 Default Ports for UDP and TCP

Port Number Description

Data Replication Using UDP

UDP 4145 IPrime Network-approved port for heartbeat communication between the primary and secondary.

TCP 8199 IPrime Network-approved port for communication between the vradmind daemons on the primary and secondary.

TCP 8989 Communication between the in.vxrsyncd daemons, which are used for differences-based synchronization.

UDP anonymous ports (OS-dependent)

Ports used for each primary-to-secondary connection for data replication between the primary and secondary. One data port is required on each host.

Data Replication Using TCP

UDP 4145 IPrime Network-approved port for heartbeat communication between the primary and secondary.

TCP 4145 IPrime Network-approved port for the TCP listener port.

TCP 8199 IPrime Network-approved port for communication between the vradmind daemons on the primary and secondary.

TCP 8989 Communication between the in.vxrsyncd daemons, which are used for differences-based synchronization.

TCP anonymous ports

Ports used for each primary-to-secondary connection for data replication between the primary and secondary. One data port is required on each host.


OL-24641-01


In the following figures, the servers represent:

• Primary site:

– P1—Primary Cisco Prime Network gateway server/backup Oracle server.

– P2—Primary Oracle server/backup Cisco Prime Network gateway server.

• Secondary site:

– S1—Cisco Prime Network gateway and Oracle server.

Figure 6-3 Veritas HA Local Redundancy Setup

GUI clients

Web clients

WWW

Customer OSS/BSS

Cisco Prime Networkgateways

Heartbeat

External storage

Server P1(primary

Cisco Prime Network)

Server P2(primary Oracle)

2828

64

Cisco PrimeNetwork units

Dual-node cluster


OL-24641-01


Figure 6-4 Veritas HA Geographical Redundancy Setup

Replication

Primary site/ local clusterdual -node cluster

GUI clients

Web clients

WWW

Customer OSS/BSS



External storage External storage

Global heartbeat

Server P1(primary

Cisco PrimeNetwork)


Server S1

Secondary site/remote clustersingle-node cluster

2828

62

Cisco Prime Network

units


OL-24641-01


Figure 6-5 Veritas HA Geographical Redundancy Setup with Units

Veritas Gateway HA Installation RolesThe Cisco Prime Network installation includes the hardware configuration, OS installation, and installation of the Oracle and Cisco Prime Network applications. In addition to these tasks, an HA solution includes installing the Veritas Software Foundation, configuring Veritas clusters, and configuring additional hardware and OS components.

An HA installation requires these roles:

• Veritas consultant—Responsible for installing the Veritas applications (Veritas Volume Manager and Veritas Cluster Server for local redundancy; Veritas Volume Replicator for geographical redundancy). This role requires a high level of Veritas expertise. The Veritas consultant might be asked to assist with setting up external disks and configuring the OS.

GUI clients

Web clients

WWW

Customer OSS/BSS

Cisco PrimeNetwork units

Replication



External storage External storage

Global heartbeat

Server P1(primary

Cisco PrimeNetwork)


Server S1

Secondary site/remote clustersingle-node cluster + units

2828

63

Primary site/local clusterdual-node cluster + units


OL-24641-01


• Veritas administrator—Once the Veritas software installation is complete, the Veritas administrator performs ongoing configuration and administration of the cluster, including adding service groups and resources.

This chapter highlights the portions of the installation that require the skills of the Veritas consultant or administrator.

Veritas Gateway HA Local Dual-Node Cluster Unless noted otherwise, all steps for the local dual-node cluster pertain to both local redundancy and geographical redundancy HA environments.

Hardware

Tip The steps in this section might require assistance from the Veritas consultant.

Configure the two servers as shown in Figure 6-1, including:

• Dual Gigabit Ethernet connections for the heartbeat.

• Dual connections to each external storage unit.

Configure the external storage so that all disks or logical unit numbers (LUNs) are accessible from both servers in the cluster. The disk or LUN configuration depends on the type of storage. If you are using JBOD disks, provide enough physical disks to create the following volumes to satisfy the Oracle (and VVR) performance requirements. If you are using storage that supports hardware RAID, divide the physical disks into LUNs so that the following volumes can be created, configured to satisfy the Oracle (and VVR) performance requirements, and protected with RAID5, RAID1, or RAID10. You can create the Oracle volumes on a single LUN. The SRL volumes are relevant only if you are using geographical redundancy with VVR.

Table 6-3 Size Requirements for Oracle Volumes

Volume Minimum Size Comments

Prime Network 50 GB —

Prime Network SRL — See the SRL requirements.

Oracle application and data files 10 GB —

Oracle redo logs 6.4 GB —

Oracle archives — Contact your Cisco account representative to obtain the Cisco Prime Network Capacity Planning Guide.

Additional Oracle data files (if used) — Based on Cisco Prime Network alarm history needs; see the Capacity Planning Guide.


OL-24641-01


Operating System


Step 1 Install the operating system and all recommended patches on both servers in the cluster. The installation on both servers must be identical. Once the operating system is installed, verify access to all external disks.

Step 2 Divide the internal disks as shown in Table 6-4. It is recommended—but not required—that you place the individual directories in separate partitions.

Step 3 The Cisco Prime Network installation normally requires 1024 MB of additional free space under the root partition. With HA, a temporary copy of Cisco Prime Network is installed under the root partition, which requires an additional 5120 MB of free space, bringing the total free space requirement to 6144 MB. The Veritas software is normally installed under /opt, which requires a minimum of 3072 MB. You can add this amount to the root partition instead of creating a separate partition for /opt. The Veritas logs are normally saved under /var, which you can set up as a separate partition.

Step 4 Verify that neither server has a local partition mounted under /export/home. (During normal cluster operation, the server where Cisco Prime Network is running has the external Prime Network data volume mounted under /export/home; the server where Oracle is running has nothing mounted under /export/home.)

Step 5 Verify that the time is synchronized on both servers.

Step 6 As the root user, enter the following commands on both servers to create the mount points:

# mkdir -p /export/home# cd /opt# mkdir db dbarch dbbackup dbdata dblogs

Oracle backup

Note The backup partition is required only if you are using an embedded database. If you are not using an embedded database, it is up to the administrator to back up the database.

— See the Capacity Planning Guide.

Oracle SRL — See the SRL requirements.

Table 6-4

Partition Space (in MB)

swap Standard amount of space, as per system configuration

/ Standard amount of space plus 6144 MB extra

/opt Minimum 3072 MB

/var (optional) Space for Veritas logs

others (optional) Remaining space

Table 6-3 Size Requirements for Oracle Volumes (continued)

Volume Minimum Size Comments


OL-24641-01


The following mount points are created:

• /export/home

• /opt/db

• /opt/dbarch

• /opt/dbbackup (required only if you are using an embedded database)

• /opt/dbdata

• /opt/dblogs

Veritas Software and Local Cluster Creation

Tip The Veritas consultant should use the product documentation to perform the steps in this section.

Step 1 Install the Veritas Storage Foundation HA or Enterprise HA with global cluster option (if you are using disaster recovery) on both servers in the local cluster. Veritas Volume Manager is used to manage the internal storage. It can also be used to manage the external storage if the external storage does not have its own volume management solution.

Step 2 Install the latest Veritas maintenance pack that contains the Veritas patches for Solaris.

Step 3 Install the Veritas High Availability Agent for Oracle.

Step 4 Use VCS to create and configure the local cluster:

a. Create a new cluster.

b. Set up the LLT heartbeat between the two servers.

Disk Groups, Volumes, and Mirroring


Step 1 Use Veritas Volume Manager (VxVM) to create two disk groups (one for Prime Network; one for Oracle or the embedded database) as defined in Hardware, page 6-13. Use the volumes listed in Table 6-5. If the disks are mirror-protected instead of RAID-protected, include a duplicate set of disks for the mirroring.

Table 6-5 Disk Groups, Volumes, and Mount Points

Disk Group Volume(s) Mount Point Comments

Prime Network

Prime Network /export/home —

Prime Network SRL (VVR only) — —


OL-24641-01


Step 2 Use VxVM to set up mirroring between the two internal disks on both servers.

Step 3 If you are not using hardware that supports hardware RAID for the external disks, use VxVM to set up mirroring for the external disks (including SRLs). (This task takes a long time and can be deferred to a convenient time during the installation process.)

Local Redundancy Dual-Node Cluster Worksheet

Use the following worksheet to record all parameters necessary to configure the local dual-node cluster. The parameter names shown in italics will be requested during the actual HA installation.

Oracle or embedded database

Oracle application and data files /opt/db Create the volumes using a block size of 8192.

Oracle redo logs /opt/dblogs Mount the redo logs volume with Convert osync policy=direct.

Oracle archives /opt/dbarch —

Additional Oracle data files (if used) /opt/dbdata Create the volumes using a block size of 8192.

Oracle backup files /opt/dbbackup Required only when using an embedded database.

Oracle SRL (VVR only) — —

Table 6-5 Disk Groups, Volumes, and Mount Points (continued)

Disk Group Volume(s) Mount Point Comments

Table 6-6 Parameters Required for the Local Dual-Node Cluster

Parameter Name Description Value

Local-Prime Network-SG-IP Logical IP address of the local cluster Prime Network service group.

Local-Prime-Network-SG-netmask Netmask of the logical interface of the local cluster Prime-Network service group.

Local-Oracle-SG-IP Logical IP address of the local cluster Oracle service group.

Local-Oracle-SG-netmask Netmask of the logical interface of the local cluster Oracle service group.

Remote-single-node-IP Logical IP address of the remote single-node cluster.

Network-interface Name of network interface to which the logical IP address is added. The name must be identical on all servers.

Example: bge0.

Prime-Network-disk-group Name of the disk group created with Veritas Volume Manager for the Prime-Network data.

Example: anadg.

Prime-Network-volume-block-device Block device of the volume created for the Prime-Network data.

Example: /dev/vx/dsk/anadg/anavol.


OL-24641-01


Oracle-disk-group-name Name of the disk group created with Veritas Volume Manager for the Oracle data.

Example: oradg.

Oracle-application-volume-block-device Block device of the volume created for the Oracle application.

Example: /dev/vx/dsk/oradg/oravol.

Oracle-archive-file-volume-block-device Block device of the volume created for the Oracle archive logs.

Example: /dev/vx/dsk/oradg/oraarchvol.

Oracle-data- volume-block-device Block device of the volume created for the Oracle data files.

Example: /dev/vx/dsk/oradg/oradatavol.

Oracle-backup-volume- block-device Block device of the volume created for the Oracle backup files for the embedded database.

Example: /dev/vx/dsk/oradg/orabackupvol.

Oracle-log-volume-blockdevice Block device of the volume created for the Oracle redo log files.

Example: /dev/vx/dsk/oradg/oralogsvol.

ORACLE_HOME Value of the $ORACLE_HOME environment variable, which is the root of the Oracle installation (not the home directory of the user oracle).

Example: /opt/db/oracle/Ora920.

Netlsnr-resource-name Name of the Netlsnr resource in the Oracle service group.

Example: Netlsnr.

Prime-Network-IP-resource-name Name of the IP resource in the Prime-Network service group.

Example: ana_IP.

Oracle-IP-resource-name Name of the IP resource in the Oracle service group.

Example: ora_IP.

Local-LDAP-URL Local LDAP server URL (if you are using LDAP authentication and geographical redundancy HA).

Example: ldap://10.2.3.4:389.

Local-LDAP-prefix Local LDAP distinguished name prefix (if you are using LDAP authentication and geographical redundancy HA).

Example: CN.

Table 6-6 Parameters Required for the Local Dual-Node Cluster (continued)



OL-24641-01


Local Cluster Configuration

Tip The Veritas administrator should perform the steps in this section.

Step 1 Use Cluster Manager to add two new service groups to the cluster: ana_sg for Prime-Network and oracle_sg for Oracle. You must use the exact names ana_sg and oracle_sg. Use the following settings:

• Use “failover” as the service group type for both service groups.

• Add both servers to each service group.

• Determine the server priority according to which server is the primary server for each service group:

• Set the resources in the service groups to critical. Note, however, that the resources are initially set to non-critical during the installation process and are made critical once the service groups are built.

Step 2 Add the following aliases to the /etc/hosts file on P1 and P2:

Local-Prime-Network-SG-IP ana-cluster-anaLocal-Oracle-SG-IP ana-cluster-oracle

Note If the IP address that you enter is not the default one, the database installation updates the hostname in the listener’s files. Verify that /etc/hosts is updated with the correct IP address and hostname; the order is important. If more than one hostname is attached to the selected IP address, the first hostname is used.

Local-LDAP-suffix Local LDAP distinguished name suffix (if you are using LDAP authentication and geographical redundancy HA).

Example: CN=Users,DC=,DC=myldapserver,DC=local.

Local-LDAP-SSL-usage Local Prime-Network-LDAP SSL protocol usage (if you are using LDAP authentication and geographical redundancy HA). The boolean value is true if the SSL protocol is used between Prime-Network and the LDAP server. The boolean value is false (the default) if SSL is not used.

Table 6-6 Parameters Required for the Local Dual-Node Cluster (continued)


Service Group P1 Server Priority P2 Server Priority

Prime-Network 0 1

Oracle 1 0


OL-24641-01


Step 3 To configure the Prime-Network service group, do the following:

a. Add a resource of type “NIC” to the service group with the following configuration:

b. Enable the NIC resource.

c. Add a resource of type “DiskGroup” to the service group with the following configuration:

d. Enable the DiskGroup resource.

e. Add a resource of type “IP” with the following configuration:

f. Enable the IP resource.

g. Verify that no partitions from the internal disk are mounted under /export/home on either server. If a partition is mounted, unmount it and comment out the corresponding entry in /etc/vfstab.

Resource Type: NIC

Key Attributes

Critical False

Type-Specific Attributes

Device <network interface>

Overridden Attributes

None —

Resource Type: DiskGroup

Key Attributes

Critical False


DiskGroup <Prime-Network disk group>


None —

Resource Type: IP1

1. In addition to the simple IP agent, VCS provides the IPMultiNIC and IPMultiNICB agents, which can be used in place of the IP agent if necessary. This functionality also applies to the NIC agents in the other service groups.

Key Attributes

Critical False



Address <local Prime-Network SG IP>

NetMask <local Prime-Network SG netmask>


None —


OL-24641-01


h. Add a resource of type “Mount” with the following configuration:

i. Enable the Mount resource.

j. Create dependency links between the following resources. (The direction of the arrow indicates the dependency.)

– IP —> NIC

– Mount —> Disk group

Step 4 To configure the Oracle service group, do the following:

a. Add a resource of type “NIC” to the Oracle service group with the following configuration:

b. Enable the NIC resource.

c. Add a resource of type “DiskGroup” to the Oracle service group with the following configuration:

Resource Type: Mount

Key Attributes

Critical False


MountPoint /export/home

BlockDevice <Prime-Network volume block device>

FSType vxfs

FsckOpt -y


None —

Resource Type: NIC

Key Attributes

Critical False




None —


Key Attributes

Critical False


DiskGroup <Oracle disk group>


None —


OL-24641-01


d. Enable the DiskGroup resource.

e. Add a resource of type “IP” to the Oracle service group with the following configuration:

f. Enable the IP resource.

g. Verify that no partitions from the internal disk are mounted under /opt/db, /opt/dbarch, /opt/dbbackup, /opt/dbdata, or /opt/dblogs on either server. If a partition is mounted, unmount it and comment out the corresponding entry in /etc/vfstab.

h. Add four or five resources of type “Mount” to the Oracle service group with the following configurations:

Note The Oracle service group is shown here with five mount resources. Oracle and the embedded database require one volume for binaries and data, one volume for archive files, and one volume for redo logs. The embedded database requires an additional volume for backup files. Adding a fifth volume (or more) for additional data files is optional.

Resource Type: IP

Key Attributes

Critical False



Address <local Oracle SG IP>

NetMask <local Oracle SG netmask>


None —


Key Attributes

Critical False


MountPoint /opt/db

BlockDevice <Oracle application volume block device>

FSType vxfs

FsckOpt -y


None —


Key Attributes

Critical False


OL-24641-01


i. Enable the Mount resource.


MountPoint /opt/dbarch

BlockDevice <Oracle archive volume block device>

FSType vxfs

FsckOpt -y


None —



Key Attributes

Critical False


MountPoint /opt/dbdata

BlockDevice <Oracle data volume block device>

FSType vxfs

FsckOpt -y


None —


Key Attributes

Critical False


MountPoint /opt/dblogs

BlockDevice <Oracle logs volume block device>

FSType vxfs

MountOpt convosync=direct

FsckOpt -y


None —


OL-24641-01


j. If you are using an embedded database, add an additional resource of type “Mount” to the Oracle service group with the following configurations:

k. Enable the Mount resource.

l. Create dependency links between the following resources. (The direction of the arrow indicates the dependency.)

– IP —> NIC

– Mount (1) —> Disk group




– Mount (5) —> Disk group (only if you are using an embedded database)

Oracle Database Software

This section describes how to install the Oracle database software. If you are using an embedded database, skip this section and go to Cisco Prime Network Software, page 6-26.

Install the Oracle database according to the Oracle product documentation and Chapter 4, “Creating and Configuring an Oracle Database.” An HR/DR scenario has additional requirements as described in this section.

Tip Throughout this section, the Oracle user is oracle and the Oracle group is dba.

Some administrators choose to create the user oracle with the default group oinstall and then create an additional group dba with the user oracle as a member. The group dba can be replaced by oinstall where appropriate.

If you use network address translation (NAT) with the Cisco Prime Network Events client, update the database host in the Prime Network registry to contain the hostname instead of the IP address. For updating the database host, see Updating the Database Host in the Registry for NAT, page 7-12.


Key Attributes

Critical False


MountPoint /opt/dbbackup

BlockDevice <Oracle backup volume block device>

FSType vxfs

FsckOpt -y


None —


OL-24641-01


Oracle Preinstallation Requirements in an HR/DR Environment

In HR/DR environment, the Oracle database installation requires that you:

• Create a software owner user and group(s). When doing so:

– Create the user and group(s) on P2.

– Place the user’s home directory on the root file system of P2. Give the directory the correct ownership.

– Do not place the user’s home directory under /export/home, because the Prime-Network shared volume is mounted there.

• Set up the Oracle user’s environment. Set up the required environment variables in the user’s home directory on P2.

• Configure system parameters in /etc/system on P2.

Oracle Preinstallation Tasks in an HR/DR Environment

The following preinstallation tasks are specific to gateway server high availability:

1. Verify that you can connect from:

– P1 to P2

– P1 to P1

– P2 to P2

– P2 to P1

2. Verify that the Oracle listener port (1521 by default) is available on both servers.

3. Verify that the Oracle service group is online on P2 (which is the default for this service group). The Oracle external volumes will be mounted on P2 under /opt/db, /opt/dbarch, and /opt/dbdata.

4. Create the following directories with oracle:dba ownership:

– /opt/db/oracle

– /opt/db/oraInventory

– /opt/dbarch/oraarch

– /opt/dbdata/oradata2

– /opt/dblogs/oralogs

Oracle Database Installation and Database Creation

Step 1 Install the Oracle database software on P2 according to the Oracle product documentation, the guidelines in Chapter 4, “Creating and Configuring an Oracle Database,” any additional performance guidelines provided for Cisco Prime Network, and the following HA-specific requirements:

• Set the Oracle base path to /opt/db/oracle. Based on this value, the software location is set automatically to /opt/db/oracle/product/11.2.0/dbhome_1.

• Set the inventory directory to /opt/db/oraInventory.

• Save the archive logs in /opt/dbarch/oraarch.

• Save the redo logs in /opt/dblogs/oralogs.


OL-24641-01


Step 2 If you are using the netca utility to create a listener, edit the /opt/db/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora file on P2 immediately after you create the listener, and before you create the database. If you create the listener and database during the initial Oracle database installation, edit the file at the end of the process. Wherever the hostname of the P2 IP address appears, replace it with the hostname alias of the Oracle service group logical IP address, as follows:

# listener.ora Network Configuration File: /opt/db/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora# Generated by Oracle configuration tools.

LISTENER = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = ana-cluster-oracle)(PORT = 1521)) ) )

ADR_BASE_LISTENER = /opt/db/oracle

Step 3 At the end of the database creation process, edit the /opt/db/oracle/product/11.2.0/dbhome_1/network/admin/tnsnames.ora file on P2. Wherever the hostname of the P2 IP address appears, replace it with the hostname alias of the Oracle service group logical IP address, as follows:

# tnsnames.ora Network Configuration File: /opt/db/oracle/product/11.2.0/dbhome_1/network/admin/tnsnames.ora# Generated by Oracle configuration tools.

LISTENER_MCDB = (ADDRESS = (PROTOCOL = TCP)(HOST = ana-cluster-oracle)(PORT = 1521))

MCDB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = ana-cluster-oracle)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = MCDB) ) )

Oracle Postinstallation Tasks in an HA/DR Environment

Complete the following HA/DR postinstallation tasks to synchronize the Oracle environment on P1 and P2:

Step 1 Synchronize the changes in the following files from P2 to P1:

• /etc/passwd

• /etc/shadow

• /etc/group

• /etc/system

Step 2 On P1, create the same home directory for user oracle on the root file system as was created for user oracle on P2. Give the directory oracle:dba ownership.

Step 3 On P1, set up the same environment variables for user oracle as were set up for user oracle on P2.


OL-24641-01


Step 4 Copy (overwrite) the following directory and all its files from P2 to P1:

• /var/opt/oracle

• Home directory of user oracle

Step 5 If necessary, change the owner/mode of all files copied to P1 to match the corresponding files on P2.

Step 6 Reboot P1 and P2.

Step 7 When P2 restarts, verify that the Oracle service group is online on P2. If the Oracle service group is not online, bring it online on P2.

Step 8 If the database and listener are not running, start them manually on P2 and verify that they start and run correctly. To start the database and listener, enter the following commands as user oracle:

% sqlplus /nologSQL> connect /as sysdbaSQL> startupSQL> quit% lsnrctl start

Step 9 If you are using raw devices for the redo logs, add three new redo logs on the designated volumes, one redo log per volume. Force Oracle to switch log files three times so that it uses the first of the new files on the raw devices. You can delete the original temporary log files that were created when the database was created.

Step 10 Shut down the database and listener manually.

Step 11 Switch the Oracle service group to P1.

Step 12 Start the database and listener manually on P1 and verify that they start correctly.

Cisco Prime Network Software

Cisco Prime Network must be installed on both servers in the cluster because it affects files on the root partition and on the /export/home partition. This section uses the default names:

• network-user—network38. This is the user-defined Cisco Prime Network OS user. The username must start with a letter and contain only the following characters: [A-Z a-z 0-9 -].

• Prime Network Group—network

• Prime Network Home directory—network38

If you are not using the default names, substitute the correct values.

To install Cisco Prime Network:

Step 1 Verify that the Prime-Network service group is online on P1 (which is the default for this service group).

Step 2 Verify that the Oracle service group is online on P1 and that the Oracle database application and listener are running on P1.

Step 3 Verify that /etc/passwd, /etc/shadow, and /etc/group files are identical on both servers (and on the server in the remote cluster, if applicable).

Step 4 Enter the following command to install Cisco Prime Network on P1:

perl install.pl -user network38 -ha


OL-24641-01


Note Make sure that the port 2148 is free before starting the Veritas HA installation.

Step 5 As the root user, enter the following command to switch users to network38:

# su - network38

Step 6 Run the network-conf configuration script. Follow the local database instructions and note:

• When choosing the IP address of the Prime-Network back-end services, choose the logical IP address of the Prime-Network service group.

• When choosing the IP address of the Oracle database, choose the logical IP address of the Oracle service group.

• If you are using Oracle, use /opt/db/oracle/oradata for the location of the Oracle data files.

• If you are using an embedded database, use:

– /opt/db/oracle for the home directory of the database user.

– /opt/db/oracle/oradata/anadb for the location of the general data files.

– /opt/dblogs/oralogs for the location of the redo logs.

Step 7 As the root user, enter the following command to change the password for user network38:

# passwd network38

Step 8 While the Prime-Network service group is still online on P1, log into P2 as the root user.

Step 9 Enter one of the following commands to install Cisco Prime Network on P2:

• If you are not using an embedded database, enter:

perl install_ha.pl -user network38

• If you are using an embedded database, enter:

perl install_ha.pl -user network38 -embedded

During the installation, you will be asked to enter the following information:

• Group ID (gid) of group network and the user ID (uid) of the network-user on the primary server. To find these values, look for the following line in the /etc/passwd file on P1:

network-user:x:uid:gid::/export/home/network38:/bin/tcsh

The uid and gid are shown in the line corresponding to the networkuser, as indicated in bold in the following example:

root:x:0:0:Super-User:/root:/bin/tcshdaemon:x:1:1::/:bin:x:2:2::/usr/bin:sys:x:3:3::/:adm:x:4:4:Admin:/var/adm:nobody4:x:65534:65534:SunOS 4.x Nobody:/:admin:x:101:1::/local/admin:/bin/tcshnetwork38:x:1310:1310::/export/home/network38:/bin/tcshbos:x:1310:1310:Bos Admin:/export/home/sheer/Main/bosconfig/bos_shell_scripts:/export/home/sheer/Main/bosconfig/bos_shell_scripts/run.cshoracle:x:1311:502::/opt/db/oracle:/bin/tcsh

In the preceding output, the network-user is network38, the uid of the network-user is 1310, and the gid of group network is 1310.


OL-24641-01


• The oinstall group gid and primary Cisco Prime Network gateway dba. To find these values, look for the following lines in the /etc/group file on P1:

oinstall::1311:oracledba::1312:

• The oracle user uid on the primary Cisco Prime Network gateway. To find these values, enter the following commands as the oracle user on P1:

oracle@sh-v240-053 [~]# iduid=1311(oracle) gid=1312(dba)

Step 10 If you are using geographical redundancy HA, On P1, enter the following command as user network38:

# ~/Main/scripts/add_secondary_host.pl

Step 11 When you are requested for the IP address of the secondary host, enter the remote-single-node-IP.

Step 12 Switch the Prime-Network service group to P2. The external Prime-Network volume is now mounted on P2 under /export/home.

Step 13 On P2, enter the following command to change the password for user network38. Use the same password as on P1:

# passwd network38

Step 14 Verify that Cisco Prime Network starts and runs correctly on P2.

Step 15 On P2, enter the following command as user network38:

# ~/Main/scripts/edit_registry_for_ha.pl

Step 16 Launch the Cisco Prime Network Administration client application and connect using the logical IP address of the Cisco Prime Network service group.

Step 17 Stop Cisco Prime Network on P2 using the normal stop method and switch the Prime-Network service group to P1. Then, start Prime-Network manually using the normal Cisco Prime Network start script on P1 and verify that Cisco Prime Network starts and runs correctly on P1.

Step 18 Using the Cisco Prime Network Administration client application, reconnect using the logical IP address of the Cisco Prime Network service group.

Step 19 If desired, add additional data files to the Oracle database. See Adding Data Files to the Tablespace, page 4-16. Create the additional data files under /opt/db/oradata and /opt/dbdata/oradata2 (if used).

Caution If you are using geographical redundancy HA, skip the remaining steps in this section and proceed to VCS Configuration for Oracle and Cisco Prime Network, page 6-29.

Step 20 With Cisco Prime Network running on P1, as user network38, install any relevant service pack(s) on P1 according to the service pack installation instructions.

Step 21 Add the unit servers to the setup. For the gateway IP address, use the logical IP address of the Cisco Prime Network service group.

Step 22 Install any relevant point patches according to the patch documentation.

Step 23 If you are using LDAP to authenticate Cisco Prime Network users, configure the LDAP settings in Cisco Prime Network Administration. You must set the LDAP values in Cisco Prime Network Administration and on the Prime Network Gateway agent.


• /etc/passwd


OL-24641-01


• /etc/shadow

• /etc/group

• /etc/system

VCS Configuration for Oracle and Cisco Prime Network


Once Oracle and Cisco Prime Network have been installed, complete the VCS configuration:

Step 1 From the Cluster Manager GUI, choose File > Import Types and import the Oracle and Cisco Prime Network resource types. By default, the Oracle resource types file resides in /etc/VRTSagents/ha/conf/Oracle/OracleTypes.cf; the Cisco Prime Network resource types file resides in /etc/VRTSvcs/conf/Prime-NetworkTypes.cf. When you are finished, exit the Cluster Manager GUI and restart it.

Step 2 Add a resource of type “Oracle” to the Oracle service group with the following configuration. Note the overridden attribute:

Step 3 Enable the Oracle resource.

Step 4 Add a resource of type “Netlsnr” to the Oracle service group with the following configuration:

Resource Type: Oracle

Key Attributes

Critical False


SID • MCDB—For Oracle

• anadb—For an embedded database

Owner oracle

Home <ORACLE_HOME>


OnlineTimeout 1200

Resource Type: Netlsnr

Key Attributes

Critical False


Owner oracle

Home <ORACLE_HOME>


OL-24641-01


Step 5 Enable the Netlsnr resource.

Step 6 Add a resource of type “Prime-NetworkGateway” to the Prime Network service group with the following configuration. The four LDAP parameters are required only if you are using LDAP authentication and geographical redundancy HA. (For information on configuring the Prime-NetworkGateway agent, see Using and Configuring the Prime-NetworkGateway Agent, page 6-65.)

Step 7 Enable the Prime-NetworkGateway resource.

Step 8 Create dependency links between the following resources in the Oracle service group. (The direction of the arrow indicates the dependency.)

• Oracle —> IP

• Oracle —> Mount (1)




• Oracle —> Mount (5) (only if you are using an embedded database)

• Netlsnr —> Oracle

Step 9 Create dependency links between the following resources in the Prime-Network service group. (The direction of the arrow indicates the dependency.)

• Prime-NetworkGateway —> IP

• Prime-NetworkGateway —> Mount


RestartLimit 1

Resource Type: Prime-NetworkGateway

Key Attributes

Critical False


NetlsnrResourceName <Netlsrn resource name>

Prime-NetworkIPResourceName

<Prime-Network IP resource name>

OraIPResourceName <Oracle IP resource name>

LDAPUrl <Local LDAP URL>

LDAPPrefix <Local LDAP prefix>

LDAPSuffix <Local LDAP suffix>

LDAPUseSSL <Local LDAP SSL usage>


OnlineTimeout 1200

RestartLimit 1



OL-24641-01


Step 10 Manually switch each service group to the other server and back again to confirm the service groups work correctly.

Step 11 Change all resources in the two service groups to critical.

Notifications


You can add notifiers (resources of type “NotifierMngr”) to send SMTP or SNMP notifications for faults. Notifiers are cluster-level resources and should be added to a separate ClusterService service group, which is created automatically. If the cluster is selected in the GUI, the notifier is created as the ClusterService’s resource. Use the Notifier Wizard to create the notifier.

Final Cluster Configuration

This section marks the end of the configuration process for local redundancy HA. At this point, the cluster resource views should resemble the following figures. The resource names might vary.

Figure 6-6 Local Redundancy HA Prime-Network Service Group


OL-24641-01


Figure 6-7 Local Redundancy HA Oracle Service Group


OL-24641-01


Figure 6-8 Local Redundancy HA Service Groups

In Figure 6-8, the ClusterService service group can be used for notifier resources (but is not required).

See Sample main.cf File for Local Redundancy Veritas Gateway HA, page 6-70 for a sample VCS main.cf configuration file after all cluster resources have been created.

Veritas Gateway HA Remote Single-Node Cluster To implement the geographical redundancy HA (DR) solution, a single-node cluster is created at the remote site and a global cluster is created with the existing local dual-node cluster.

Hardware


Configure the server as shown in Figure 6-2, including:

• Dual Gigabit Ethernet connections for the heartbeat.

• Single or dual connections to each external storage unit.

Configure the external storage so that all disks or LUNs are accessible from the server. The disk or LUN configuration depends on the type of storage. If you are using JBOD disks, provide enough physical disks to create volumes identical in size to those in the local cluster. If you are using storage that supports hardware RAID, divide the physical disks into LUNs so that the necessary volumes can be created,


OL-24641-01


configured to satisfy the Oracle (and VVR) performance requirements, and protected with RAID5, RAID1, or RAID10. You can create the Oracle volumes on a single LUN. The SRL volumes are relevant only if you are using geographical redundancy with VVR.

Operating System


Step 1 Install and configure the operating system on the server in the remote cluster in exactly the same way as it was installed and configured on the local cluster servers.

Step 2 Verify that the time is synchronized with both servers in the local cluster.

Step 3 As the root user, enter the following commands to create the mount points:

# mkdir -p /export/home# cd /opt# mkdir db dbarch dbbackup dbdata dblogs

The following mount points are created:

• /export/home

• /opt/db

• /opt/dbarch

• /opt/dbbackup (required only if you are using an embedded database)

• /opt/dbdata

• /opt/dblogs

Veritas Software

Tip The Veritas administrator should perform the steps in this section according to the product documentation.

Step 1 Install the Veritas Storage Foundation Enterprise HA with global cluster option on the server in the remote cluster. Veritas Volume Manager is used to manage the internal storage. It can also be used to manage the external storage if the external storage does not have its own volume management solution.

Step 2 Install the latest Veritas maintenance pack that contains the Veritas patches for Solaris.

Step 3 Install the Veritas High Availability Agent for Oracle.


OL-24641-01


Disk Groups, Volumes, SRLs, and Mirroring


Step 1 Use Veritas Volume Manager (VxVM) to create disk groups and volumes that are identical in size and configuration to those in the local cluster. If the disks are mirror-protected instead of RAID-protected, include a duplicate set of disks for the mirroring.

Step 2 Use VxVM to set up mirroring between the two internal disks on both servers.

Step 3 If you are not using hardware that supports hardware RAID for the external disks, use VxVM to set up mirroring for the external disks (including SRLs). (This task takes a long time and can be deferred to a convenient time during the installation process.)

Geographical Redundancy Single-Node Cluster Worksheet

Use the following worksheet to record all parameters necessary to create and configure the remote single-node cluster. The parameter names shown in italics will be requested during the actual HA installation. These parameters are in addition to the parameters used in the local dual-node cluster (see Table 6-6).

Table 6-7 Parameters Required for the Remote Single-Node Cluster


Local-ClusterService-SG-IP IP address of the local cluster WAC.

Local-ClusterService-SG- netmask

Netmask of the interface of the local cluster WAC.

Example: csgnic.

Remote-ClusterService-SG-IP IP address of the remote cluster WAC.

Remote-ClusterService-SG- netmask

Netmask of the interface of the remote cluster WAC.

Remote-single-node-IP Logical IP address of the remote single-node cluster.

Remote-single-node-netmask Netmask of the logical interface of the remote single-node cluster.

Prime-Network-RVG Name of the RVG created for the Prime-Network data (VVR only).

Example: anarvg.

Oracle-RVG Name of the RVG created for the Oracle data (VVR only).

Example: orarvg.

Prime-Network-volume Name of the volume for the Prime-Network application or data.

Example: anavol.


OL-24641-01


Prime-Network-SRL-volume Name of the volume for the Prime-Network SRL.

Example: anavol.

Oracle-application-volume Name of the volume for the Oracle application.

Example: oravol.

Oracle-archive-volume Name of the volume for the Oracle archive logs.

Example: oraarchvol.

Oracle-data-volume Name of the volume for the additional Oracle data files.

Example: oravol.

Oracle-backup-volume Name of the volume for the Oracle backup files.

Example: oravol.

Oracle-logs-volume Name of the volume for the Oracle redo logs.

Example: oravol.

Oracle-SRL-volume Name of the volume for the Oracle SRL.

Example: orasrlvol.

Oracle-application-volume- block-device

Block device of the volume created for the Oracle application.

Example: /dev/vx/dsk/oradg/oravol.

Oracle-archive-file-volume- block-device

Block device of the volume created for the Oracle archive logs.

Example: /dev/vx/dsk/oradg/oraarchvol.

Oracle- data-volume-blockdevice

Name of the volume for the additional Oracle data file.

Example: oradatavol.

Oracle-backup-volume-block-device

Block device of the volume created for the Oracle backup files for the embedded database.

Example: /dev/vx/dsk/oradg/orabackupvol.

Oracle-logs-volume-blockdevice

Block device of the volume created for Oracle redo log files.

Example: /dev/vx/dsk/oradg/oralogsvol.

Prime-Network-RVG-resource-name

Name of the RVG resource for the Prime-Network data (VVR only).

Example: ana_RVG.

Oracle-RVG-resource-name Name of the RVG resource for the Oracle data (VVR only).

Example: oracle_RVG.

Oracle-SG-Netlsnr-resource- name

Name of the Netlsnr resource in the Oracle service group.

Example: Netlsnr.

Table 6-7 Parameters Required for the Remote Single-Node Cluster (continued)



OL-24641-01


Remote Cluster Creation

Tip The Veritas consultant should perform the steps in this section.

Use VCS to enable the global cluster option (WAC):

Step 1 Create a new cluster for the remote site.

Step 2 Create a global cluster that includes the local and remote clusters.

Step 3 As part of the creation process, add a ClusterService service group to the local and remote clusters (if it does not already exist) to manage the global cluster, including all necessary resources.

a. Use local-ClusterService-SG-IP and local-ClusterService-SG-netmask for the ClusterService IP resource in the local cluster.

b. Use remote-ClusterService-SG-IP and remote-ClusterService-SG-netmask for the ClusterService IP resource in the remote cluster.

Single-node-IP-resource-name For a single-node cluster, the name of the IP resource in the VVR service group.

Example: single_node_IP.

Domain Domain served by the DNS server to be updated with the current IP address of the gateway in case of a site-to-site failover or switchover (if you are configuring a DNS resource).

Remote-LDAP-URL Remote LDAP server URL (if you are using LDAP authentication).

Example: ldap://10.2.3.4:389.

Remote-LDAP-prefix Remote LDAP distinguished name prefix (if you are using LDAP authentication).

Example: CN.

Remote-LDAP-suffix Remote LDAP distinguished name suffix (if you are using LDAP authentication).

Example: CN=Users,DC=,DC=myldapserver,DC=local.

Remote-LDAP-SSL-usage Remote Prime-Network-LDAP SSL protocol usage (if you are using LDAP authentication). The boolean value is true if the SSL protocol is used between Prime-Network and the LDAP server. The boolean value is false (the default) if SSL is not used.

Table 6-7 Parameters Required for the Remote Single-Node Cluster (continued)



OL-24641-01


Remote Cluster Configuration


Step 1 Use Cluster Manager to add three new service groups to the remote cluster: ana_sg for Prime-Network, oracle_sg for Oracle, and a replication service group called vvr_sg (if you are using VVR) or replication_sg (if you are not using VVR). You must use the exact names ana_sg, oracle_sg, and vvr_sg or replication_sg. Use the following settings:

• Use “failover” as the service group type for all three service groups.

• Add the single server to each service group.

• You do not have to set the resources as critical in the service groups for the remote single-node cluster. There is no local failover option within the remote single-node cluster.

Step 2 For the ana_sg and oracle_sg service groups in the local and remote clusters, use Cluster Manager to make the service groups global. Right-click each service group and use the Configure as global option with the following settings:

• On the local cluster, add the remote cluster to both service groups and connect to the remote cluster using its ClusterService SG IP address and cluster manager username/password (if in use).

• On the remote cluster, add the local cluster to both service groups and connect to the local cluster using its ClusterService SG IP address and cluster manager username/password (if in use).

Step 3 Add a resource of type “NIC” to the remote cluster’s replication (VVR) service group with the following configuration:

Step 4 Enable the NIC resource.

Step 5 Add a resource of type “IP” to the remote cluster’s replication (VVR) service group with the following configuration:

Resource Type: NIC

Key Attributes

Critical False




None —

Resource Type: IP

Key Attributes

Critical False



Address <remote single-node IP>

NetMask <remote single-node netmask>


OL-24641-01


Step 6 Enable the IP resource.

Step 7 Add two resources of type “DiskGroup” to the remote cluster’s replication (VVR) service group with the following configurations:

Step 8 Enable the DiskGroup resources.

Step 9 Create a dependency link between the following resources in the remote cluster’s replication service group. (The direction of the arrow indicates the dependency.)

• IP —> NIC

Step 10 Verify that no partitions are mounted under /export/home on the server in the remote cluster. If a partition is mounted, unmount it and comment out any corresponding entry in /etc/vfstab.

Step 11 Add a resource of type “Mount” to the remote cluster’s Prime-Network service group with the following configuration:


None —


Key Attributes

Critical False




None —


Key Attributes

Critical False




None —

Resource Type: IP


Key Attributes

Critical False


MountPoint /export/home


OL-24641-01


Step 12 Enable the Mount resource.

Step 13 Verify that no partitions are mounted under /opt/db, /opt/dbarch, or /opt/dbdata on either server. If a partition is mounted, unmount it and comment out any corresponding entries in /etc/vfstab.

Step 14 Add four or five resources of type “Mount” to the service group with the following configurations:

Note The Oracle service group is shown here with five mount resources. Oracle and the embedded database require one volume for binaries and data, one volume for archive files, and one volume for redo logs. The embedded database requires an additional volume for backup files. Adding a fifth volume (or more) for additional data files is optional.

BlockDevice <Prime-Network volume block device>

FSType vxfs

FsckOpt -y


None —



Key Attributes

Critical False


MountPoint /opt/db

BlockDevice <Oracle application volume block device>

FSType vxfs

FsckOpt -y


None —


Key Attributes

Critical False


MountPoint /opt/dbarch

BlockDevice <Oracle archive volume block device>

FSType vxfs

FsckOpt -y


None —


OL-24641-01


Step 15 Enable the Mount resources.

Step 16 If you are using an embedded database, add an additional resource of type “Mount” to the service group with the following configuration:

Step 17 Enable the Mount resource.


Key Attributes

Critical False


MountPoint /opt/dbdata

BlockDevice <Oracle data volume block device>

FSType vxfs

FsckOpt -y


None —


Key Attributes

Critical False


MountPoint /opt/dblogs

BlockDevice <Oracle logs volume block device>

FSType convosync=direct

FsckOpt -y


None —


Key Attributes

Critical False


MountPoint /opt/dbbackup

BlockDevice <Oracle backup volume block device>

FSType vxfs

FsckOpt -y


None —


OL-24641-01


Step 18 Add the following aliases to the /etc/hosts file on the server in the remote cluster:

remote-single-node-IP ana-cluster-ana ana-cluster-oracle

Note If the IP address that you enter is not the default one, the database installation updates the hostname in the listener’s files. Verify that /etc/hosts is updated with the correct IP address and hostname; the order is important. If more than one hostname is attached to the selected IP address, the first hostname is used.

Replication Using VVR

Tip The Veritas consultant should perform the steps in this section.

This section describes how to use VVR to set up data replication between the local and remote clusters. The initial replication involves copying the entire contents of the two volumes. It is recommended that you perform the initial replication with the local and remote servers connected to the same LAN. After the initial replication is complete, move the server for the remote cluster to its permanent location.

If you are not using VVR, skip to Replication Without Using VVR, page 6-49.

Step 1 Add the following aliases to the /etc/hosts file on both servers in the local dual-node cluster:

local-Prime-Network-SG-IP ana-cluster-ana vvr-local-analocal-Oracle-SG-IP ana-cluster-oracle vvr-local-oracleremote-single-node-IP vvr-remote-single

Step 2 Add the following entries to the /etc/hosts file on the server in the remote single-node cluster:

local-Prime-Network-SG-IP vvr-local-analocal-Oracle-SG-IP vvr-local-oracleremote-single-node-IP ana-cluster-ana ana-cluster-oracle vvr-remote-single

Step 3 Create two replicated data sets (RDSs) with the following values:

RDS Name Values

Prime-Network RDS

RVG name <Prime-Network RVG>

Disk group <Prime-Network disk group>

Data volumes <Prime-Network volume>

SRL volume <Prime-Network SRL volume>

Oracle RDS

RVG name <Oracle RVG>

Disk group <Oracle disk group>


OL-24641-01


Step 4 For each RDS, add the secondary with the following hostnames:

Step 5 If you are using VCS version 5.0, from the Cluster Manager GUI, choose File > Import Types and import the VVR resource types. By default, the VVR resource types file resides in /etc/VRTSvcs/conf/VVRTypes.cf. Import the resource types for both the local and remote clusters. When you are finished, exit the Cluster Manager GUI and restart it.

Cluster Configuration for VVR


Remote Cluster Service Groups

Step 1 Add two resources of type “RVG” to the remote cluster’s VVR service group with the following configuration:

Data volumes <Oracle application volume>

<Oracle logs volume>

<Oracle data volume>

<Oracle archive volume>

<Oracle backup volume> (if you are using an embedded database)

SRL volume <Oracle SRL volume>

RDS Name Values

Prime-Network RDS

Primary vvr-local-ana

Secondary vvr-remote-single

Oracle RDS

Primary vvr-local-oracle

Secondary vvr-remote-single

Resource Type: RVG

Key Attributes

Critical False


RVG <Prime-Network RVG>


RDS Name Values


OL-24641-01


Step 2 Enable the RVG resource.

Step 3 Create dependency links between the following resources in the remote cluster’s VVR service group. (The direction of the arrow indicates the dependency.)

• RVG (Prime-Network) —> IP

• RVG (Oracle) —> IP

• RVG (Prime-Network) —> Disk group (Prime-Network)

• RVG (Oracle) —> Disk group (Oracle)

Step 4 Add a resource of type “RVGPrimary” to the remote cluster’s Prime-Network service group with the following configuration:

Step 5 Enable the RVGPrimary resource.

Step 6 Create a dependency link between the following resources in the remote cluster’s Prime-Network service group. (The direction of the arrow indicates the dependency.)

• Mount —> RVGPrimary


None —

Resource Type: RVG

Key Attributes

Critical False


RVG <Oracle RVG>



None —

Resource Type: RVG

Resource Type: RVGPrimary

Key Attributes

Critical False


RVGResourceName <Prime-Network RVG resource name>

AutoTakeover 1

AutoResync 0


None —


OL-24641-01


Step 7 Add a resource of type “RVGPrimary” to the remote cluster’s Oracle service group with the following configuration:


Step 9 Create dependency links between the following resources in the remote cluster’s Oracle service group. (The direction of the arrow indicates the dependency.)

• Mount (1) —> RVGPrimary




• Mount (5) —> RVGPrimary (if you are using an embedded database)

Note If you are using raw devices for redo logs, there are no volume resources for the three redo log volumes. Because the RDS includes the three volumes, bringing the RVG online automatically ensures that the redo log volumes are online.

Local Cluster Service Groups

Step 1 Add two new service groups: ana_vvr_sg and oracle_vvr_sg (if you are using VVR) or ana_replication_sg and oracle_replication_sg (if you are not using VVR). Use the following settings:

• Use “failover” as the service group type for both service groups.

• Add both servers to each service group.

• Determine the server priority according to which server is the primary server for each service group:


Key Attributes

Critical False


RVGResourceName <Oracle RVG resource name>

AutoTakeover 1

AutoResync 0


None —

Service Group P1 Server Priority P2 Server Priority

Prime-Network 0 1

Oracle 1 0


OL-24641-01


• Set the resources in these service groups to critical. Note, however, that the resources are initially set to non-critical during the installation process and are made critical once the service groups are built.

Step 2 Remove the dependency links between the following resources in the local cluster’s Prime-Network service groups:

• Mount —> Disk group

• Prime-NetworkGateway —> IP

Step 3 Remove the dependency links between the following resources in the local cluster’s Oracle service group:

• Mount (1) —> Disk group




• Mount (5) —> Disk group (if you are using an embedded database)

• Oracle —> IP

Step 4 Move the DiskGroup resource from the ana_sg service group to ana_vvr_sg. To move a resource between groups, do the following from the Cluster Manager:

a. When moving the first resource, the target service group does not contain any resources. Therefore, create any type of resource as a placeholder in the target service group.

b. Select the resource that you want to move.

c. Right-click and choose Copy—> Self.

d. Right-click and choose Delete.

e. Select any resource in the target service group.

f. Right-click and choose Paste—> Paste.

g. Once the final resource has been removed, remove the placeholder resource.

Step 5 Move the IP resource from the ana_sg service group to ana_vvr_sg.

Step 6 Move the NIC resource from the ana_sg service group to ana_vvr_sg.

Step 7 Move the DiskGroup resource from the oracle_sg service group to oracle_vvr_sg.

Step 8 Move the IP resource from the oracle_sg service group to oracle_vvr_sg.

Step 9 Move the NIC resource from the oracle_sg service group to oracle_vvr_sg.

Step 10 Add a resource of type “RVG” to the local cluster’s Prime-Network VVR service group with the following configuration:

Resource Type: RVG

Key Attributes

Critical False


RVG <Prime-Network RVG>



OL-24641-01



Step 12 Add a resource of type “RVG” to the local cluster’s Oracle VVR service group with the following configuration:


Step 14 Create dependency links between the following resources in the local cluster’s Prime-Network and Oracle VVR service groups. (The direction of the arrow indicates the dependency.)

• RVG —> Disk group

• RVG —> IP

• IP —> NIC

Step 15 Bring the ana_vvr_sg and oracle_vvr_sg service groups online on their primary servers.

Step 16 Add a resource of type “RVGPrimary” to the local cluster’s Prime-Network service group with the following configuration:


Step 18 Add a resource of type “RVGPrimary” to the local cluster’s Oracle service group with the following configuration:


None —

Resource Type: RVG

Key Attributes

Critical False


RVG <Oracle RVG>



None —

Resource Type: RVG


Key Attributes

Critical False


RVGResourceName <Prime-Network RVG resource name>

AutoTakeover 1

AutoResync 0


None —


OL-24641-01



Step 20 Create a dependency link between the following resources in the local cluster’s Prime-Network service group. (The direction of the arrow indicates the dependency.)

• Mount —> RVGPrimary

Step 21 Create dependency links between the following resources in the local cluster’s Oracle service group. (The direction of the arrow indicates the dependency.)





• Mount (5) —> RVGPrimary (if you are using an embedded database)

Step 22 Bring the new RVGPrimary resources online.

Step 23 Use the VVR management application to start replication for both RDSs.

Replication Without Using VVR


This section describes how to set up storage-based data replication (without using VVR) between the local and remote clusters. The initial replication involves copying the entire contents of the two volumes. It is recommended that you perform the initial replication with the local and remote servers connected to the same LAN. After the initial replication is complete, move the server for the remote cluster to its permanent location.

Step 1 For the local cluster, add the resources and create dependency links between the cluster and the storage-based replication system.

Step 2 For the remote cluster, add the resources and create dependency links between the cluster and the storage-based replication system.


Key Attributes

Critical False


RVGResourceName <Oracle RVG resource name>

AutoTakeover 1

AutoResync 0


None —


OL-24641-01


Step 3 If you are using raw devices for the redo logs, determine whether the storage-based replication requires volume resources for the redo logs volumes. If volume resources are required, include them in the replication service group.

Step 4 Bring the remote cluster’s replication service group online.

Step 5 Start replication between the disks that contain the Cisco Prime Network and Oracle data at the local and remote sites. Do not continue to the next section until replication of all disks is complete.

Oracle Database Software

This section describes how to install the Oracle database software. If you are using an embedded database, skip this section and go to Installing Cisco Prime Network Software on Geographical Redundancy Nodes, page 6-50.

Step 1 Synchronize the changes in the following files from P2 to S1:

• /etc/passwd

• /etc/shadow

• /etc/group

• /etc/system

Step 2 On S1, create the same home directory for user oracle on the root file system as was created for user oracle on P2. Give the directory oracle:dba ownership.

Step 3 On S1, set up the same environment variables for user oracle as were set up for user oracle on P2.

Step 4 Copy (overwrite) the following directory and all its files from P2 to S1:

• /var/opt/oracle

• Home directory of user oracle

Step 5 If necessary, change the owner/mode of all files copied to S1 to match the corresponding files on P2.

Step 6 Reboot S1.

Installing Cisco Prime Network Software on Geographical Redundancy Nodes

Cisco Prime Network must be installed on all servers in the global cluster because it affects files on the root partition and on the /export/home partition. Before installing Cisco Prime Network on S1, verify that a full initial data replication was completed successfully and that the secondary site’s data is up-to-date.

Step 1 Take the Prime-Network service group offline on P1 (and do not bring it online anywhere else).

Step 2 Verify that the Oracle service group is offline on all servers.

Step 3 Enter one of the following commands to install Cisco Prime Network on S1:

• If you are not using an embedded database, enter:

perl install_ha.pl -user network38

• If you are using an embedded database, enter:


OL-24641-01


perl install_ha.pl -user network38 -embedded

Step 4 Bring the Prime-Network service group online on S1. The remote cluster’s external Prime-Network volume is now mounted on S1 under /export/home.

Step 5 On S1, enter the following command as user root to change the password for user network38. Use the same password as on P1 and P2:

# passwd network38


• /etc/passwd

• /etc/shadow

• /etc/group

• /etc/system

VCS Remote Cluster Configuration for Oracle and Prime-Network


Step 1 Bring the Oracle and Prime-Network service groups online on S1.

Step 2 From the Cluster Manager GUI, choose File > Import Types and import the Oracle and Prime-Network resource types. By default, the Oracle resource types file resides in /etc/VRTSagents/ha/conf/Oracle/OracleTypes.cf and the Prime-NetworkGateway resource types file resides in /etc/VRTSvcs/conf/Prime-NetworkTypes.cf. When you are finished, exit the Cluster Manager GUI and restart it.

Step 3 Add a resource of type “Oracle” to the remote cluster’s Oracle service group with the following configuration. Note the overridden attribute:

Step 4 Enable the Oracle resource.

Step 5 Add a resource of type “Netlsnr” to the remote cluster’s Oracle service group with the following configuration:

Resource Type: Oracle

Key Attributes

Critical False


SID • MCDB—For Oracle

• anadb—For an embedded database

Owner oracle

Home <ORACLE_HOME>


OnlineTimeout 1200


OL-24641-01


Step 6 Enable the Netlsnr resource.

Step 7 Add a resource of type “Prime-NetworkGateway” to the remote cluster’s Prime-Network service group with the following configuration. The four LDAP parameters are required only if you are using LDAP authentication. (For information on configuring the Prime-NetworkGateway agent, see Using and Configuring the Prime-NetworkGateway Agent, page 6-65.)

Step 8 Enable the Prime-NetworkGateway resource.

Step 9 Create dependency links between the following resources in the remote cluster’s Oracle service group. (The direction of the arrow indicates the dependency.)






Key Attributes

Critical False


Owner oracle

Home <ORACLE_HOME>


RestartLimit 1

Resource Type: Prime-NetworkGateway

Key Attributes

Critical False


NetlsnrResourceName <Netlsrn resource name>


<Single-node IP resource name>

Note For a single-node cluster with a single IP resource, the Prime-Network IP resource name is the same as the Oracle IP resource name.

OraIPResourceName <Single-node IP resource name>

LDAPUrl <Remote LDAP URL>

LDAPPrefix <Remote LDAP prefix>

LDAPSuffix <Remote LDAP suffix>

LDAPUseSSL <Remote LDAP SSL usage>


OnlineTimeout 1200

RestartLimit 1


OL-24641-01


• Oracle —> Mount (5) (if you are using an embedded database)

• Netlsnr —> Oracle

Step 10 Create a dependency link between the following resources in the remote cluster’s Prime-Network service group. (The direction of the arrow indicates the dependency.)

• Prime-NetworkGateway —> Mount

VVR/Replication Service Group

Step 1 If you are using VVR, do the following:

a. In the local cluster, create a dependency link from the Prime-Network service group to the Prime-Network VVR service group, and from the Oracle service group to the Oracle VVR service group. (The direction of the arrow indicates the dependency.)

– Prime-Network service group —> Prime-Network VVR service group

Relationship: Online local

Dependency type: Hard

– Oracle service group —> Oracle VVR service group


Dependency type: Hard

b. In the remote cluster, create a dependency link from the Oracle service group to the VVR service group. (The direction of the arrow indicates the dependency.)

– Oracle service group —> VVR service group


Dependency type: Firm

Note Normally, dependencies from application service groups to the VVR service group have dependency type “hard.” The single-node cluster does not need a hard dependency; using type “firm” lets you link both the Oracle and Prime-Network service groups to the single VVR service group.

– Prime-Network service group —> VVR service group


Dependency type: Firm

Step 2 If you are not using VVR, create any necessary links between service groups in the remote cluster.

Prime-Network Postinstallation Tasks

Step 1 Take the Oracle and Prime-Network service groups offline; then, bring them online on S1 to verify that the gateway starts correctly.


OL-24641-01


Step 2 Use the Cisco Prime Network Administration client to reconnect using the logical IP address of the remote cluster’s Prime-Network service group.

Step 3 As user network38, install any relevant service pack(s) on S1 according to the service pack installation instructions.

Step 4 Add the unit servers to the setup. For the gateway IP address, use the logical IP address of the single-node remote cluster.

Step 5 If you are using LDAP to authenticate Prime-Network users, use Cisco Prime Network Administration to connect to S1 and configure the LDAP settings for the remote site LDAP server. You must set the values once in Cisco Prime Network Administration. To configure the Prime-NetworkGateway resources for the local and remote clusters, use the local LDAP server values for the Prime-NetworkGateway resource on the local side and the remote LDAP server values for the Prime-NetworkGateway resource on the remote side.

Step 6 Switch the Oracle service group to P2 and the Prime-Network service group to P1. For VVR, see Failing Back from the Secondary Site to the Primary Site, page 6-68.

Step 7 Verify that the gateway starts correctly.

Notifications


Notifications require a notifier resource in each cluster. As with the local cluster, the notifier must be created as a resource in the ClusterService service group. For a global cluster, this service group already exists in both clusters.

If the operator wants to receive notifications, add a notification resource to the ClusterServer service group in the remote cluster. This setup ensures that notifications continue even after failover.

DNS


In the event of a site-to-site failover or switchover, VCS updates the DNS server with the new address of the gateway. This way, applications that know the gateway by its hostname alias can reconnect without knowing the new IP address. To add a DNS resource, do the following:

Step 1 Verify that all servers are configured as a DNS client.

Step 2 Add a resource of type “DNS” to the Prime-Network service group in the local cluster with the following configuration:

Resource Type: DNS

Key Attributes

Critical False



OL-24641-01


Step 3 Enable the DNS resource.

Step 4 Add a resource of type “DNS” to the Prime-Network service group in the remote cluster with the following configuration:

Step 5 Enable the DNS resource.

Final Cluster Configuration

At the end of the configuration process for geographical redundancy HA with VVR, the cluster resource views should resemble the following figures. The resource names might vary. The ClusterService service groups are not shown.

Domain <Domain>

Alias <Local cluster Prime-Network SG IP>

Hostname ana-cluster-ana


None —

Resource Type: DNS

Key Attributes

Critical False


Domain <Domain>

Alias <Local cluster Prime-Network SG IP>

Hostname ana-cluster-ana


None —

Resource Type: DNS


OL-24641-01


Figure 6-9 Geographical Redundancy Local Cluster Prime-Network Service Group


OL-24641-01


Figure 6-10 Geographical Redundancy Local Cluster Oracle Service Group


OL-24641-01


Figure 6-11 Geographical Redundancy Local Cluster Prime-Network VVR Service Group


OL-24641-01


Figure 6-12 Geographical Redundancy Local Cluster Oracle VVR Service Group


OL-24641-01


Figure 6-13 Geographical Redundancy Local Cluster Service Group


OL-24641-01


Figure 6-14 Geographical Redundancy Remote Cluster Prime-Network Service Group


OL-24641-01


Figure 6-15 Geographical Redundancy Remote Cluster Oracle Service Group


OL-24641-01


Figure 6-16 Geographical Redundancy Remote Cluster VVR Service Group


OL-24641-01


Figure 6-17 Geographical Redundancy Remote Cluster Service Group

See the following sample VCS main.cf configuration files (one for each cluster) after all cluster resources have been created:

• Sample main.cf File for Geographical Redundancy Veritas Gateway HA Using VVR—Local Cluster, page 6-74

• Sample main.cf File for Geographical Redundancy Veritas Gateway HA Using VVR—Remote Cluster, page 6-79

Veritas Gateway HA User Guidelines

Starting and Stopping Applications

Once VCS begins monitoring Cisco Prime Network and Oracle, you should start or stop those applications by using the VCS CLI commands or GUI to start or stop the cluster resource. If you use the operating system to start or stop a monitored application, it might cause a failover to another server.

Coordination Between Cisco Prime Network and the Oracle Database or the Embedded Database

The Oracle listener should be running before Cisco Prime Network so the Prime-Network gateway process (AVM11) can connect to the database. The regular (non-HA) startup procedure is to start the database and listener (if they are not already running) and then start Prime-Network. The Prime-NetworkGateway agent can delay startup of the Prime-Network processes while it waits for the listener to start. This lets you start the Prime-Network and Oracle service groups in parallel without


OL-24641-01


worrying about the startup order. Under normal conditions, the Prime-NetworkGateway agent waits for the listener to start and then starts Prime-Network. Occasionally, the listener might not start on time, which causes the Prime-NetworkGateway agent to abort startup, and the Prime-NetworkGateway resource faults. The resource is normally configured with a retry limit of 1, which allows a retry in this scenario. It is also possible to bring the service groups online serially, instead of in parallel: first the Oracle service group, then the Prime-Network service group. However, the cluster does not enforce this behavior.

When failing over or failing back between the local and remote clusters, it might take a long time to bring up the RVGPrimary resources (when using VVR) in the Prime-Network and Oracle service groups if there is a large amount of data to synchronize between the old and new primary servers. In this case, it is advisable to wait for the Oracle service group to come online before starting the Prime-Network service group. Doing so avoids a fault of the Prime-NetworkGateway resource due to the listener having delayed startup. (The Netlsnr resource is above the RVGPrimary resource in the dependency tree.) If a fault occurs because the Oracle service group is not online on time, once it comes online, clear the fault and start the Prime-NetworkGateway resource.

The global cluster does not prevent the Prime-Network service group from being brought online on the local cluster while the Oracle service group is online on the remote cluster (or vice versa). This situation is not supported and causes the Prime-NetworkGateway resource to fault, because it cannot find the Oracle listener online on its local cluster. Verify that the two service groups are always online on the same cluster.

Using and Configuring the Prime-NetworkGateway Agent

Table 6-8 lists the Prime-NetworkGateway agent properties.

Table 6-8 Prime-NetworkGateway Agent Properties

Property Name Default Value Description

NetlsnrResourceName — Name of the Netlsnr resource in the local cluster; used to check if the Oracle listener is online. This property is required.


— Name of the IP resource used by Prime-Network; used during a site-to-site failover or failback to reconfigure the gateway and units. This property is required.

OraIPResourceName — Name of the IP resource used by Oracle; used during a site-to-site failover or failback to reconfigure the gateway. This property is required.

NetlsnrOfflineRecheckInterval 30 Interval, in seconds, to wait between checks for the Netlsnr if the Netlsnr shows offline and is not waiting to come online. This property is required.

NetlsnrOfflineRecheckLimit 4 Maximum number of times to check for the Netlsnr if the Netlsnr shows offline and is not waiting to come online. This property is required.

NetlsnrWaitingRecheckInterval 15 Interval, in seconds, to wait between checks for the Netlsnr if the Netlsnr shows offline but is waiting to come online. This property is required.

NetlsnrWaitingRecheckLimit 80 Maximum number of times to check for the Netlsnr if the Netlsnr shows offline but is waiting to come online. This property is required.

LDAPURL — URL of the LDAP server in use for the local or remote site. This property is required only if you are using LDAP for authentication and geographical redundancy HA.


OL-24641-01


The default values for the intervals and limits ensure that the Prime-NetworkGateway resource does not fault when the Prime-Network and Oracle service groups fail over locally within a dual-node cluster. In this scenario, if the Prime-NetworkGateway resource comes online before the Oracle listener is up, the two internal wait loops (one for when the listener is down and not waiting to come online and another for when the listener is down but waiting to come online) allow the entire Oracle service group to come online before the Prime-NetworkGateway resource times out.

The Prime-NetworkGateway resource behavior is governed by a number of standard VCS resource attributes, which are normally set to default values. In particular, the OnlineTimeout attribute specifies the amount of time the Prime-NetworkGateway resource’s online script has to finish before execution is halted forcibly. The default value of 300 seconds is enough to avoid preempting the Prime-NetworkGateway resource’s online script internal wait loops and prematurely halt the script, assuming the wait loop values remain unchanged. If, however, the wait loop values (any of the RecheckInterval or RecheckLimit attributes) are increased, the OnlineTimeout could halt the script even though the wait loops have not run out. Therefore, if the Prime-NetworkGateway resource’s wait loop values are changed such that the combined wait time of both loops is greater than 300 seconds, the OnlineTimeout attribute must be overridden and its value increased to a number of seconds greater than or equal to the combined wait time of both loops.

In general, the wait loops are not designed to avoid a fault in a site-to-site failover if the Prime-Network and Oracle service groups are started in parallel. In this situation, depending on how much data must be synchronized from the local site to the remote site (assuming the RVGs are still connected), the startup of the Oracle service group could take a long time. The safest course is to wait until the Oracle service group’s RVG resource comes online (indicating that synchronization is complete) before starting the Prime-Network service group. For a single-node cluster, a fault in the Prime-NetworkGateway is not critical, because it does not trigger a failover; if a fault occurs, simply clear the fault and restart the resource.

LDAP

If you are using LDAP authentication in a geographical redundancy HA environment, it is assumed that the gateway communicates with two LDAP servers—one at the local site and one at the remote site. For this purpose, the Prime-NetworkGateway resources in each cluster can be configured with independent sets of LDAP settings. After a failover or failback, the appropriate LDAP setting is loaded. In addition to configuring the two Prime-NetworkGateway resources, the LDAP setting must be set once in the Cisco Prime Network Administration GUI; the LDAP settings in the Prime-NetworkGateway resources then take effect.

LDAPPrefix — Prefix of the LDAP server in use for the local or remote site. This property is required only if you are using LDAP for authentication and geographical redundancy HA.

LDAPSuffix — Suffix of the LDAP server in use for the local or remote site. This property is required only if you are using LDAP for authentication and geographical redundancy HA.

LDAPUseSSL False Boolean variable that indicates whether SSL is the Prime-Network-LDAP protocol for the local or remote site. This property is required only if you are using LDAP for authentication and geographical redundancy HA.

Table 6-8 Prime-NetworkGateway Agent Properties (continued)

Property Name Default Value Description


OL-24641-01


Connecting to the Gateway

Any applications that use an IP address to connect to the gateway must use either the logical IP address of the Prime-Network service group (when the gateway is running on a dual-node cluster), the logical IP address of the single-node cluster (when the gateway is running on a single-node cluster), or the hostname. Specifically:

• While setting up a new unit, when the network-conf.pl script requests the IP address of the gateway, use the Prime-Network service group/single-node cluster IP address.

• When connecting to the gateway via any of the Prime-Network clients, use the Prime-Network service group IP address.

Applications that use a hostname to connect to the gateway can potentially reconnect without changes, assuming a DNS resource has been configured for the cluster. For more information, see DNS, page 6-54.

Adding Units to the Secondary Site

For complete geographical redundancy, redundant units are located at the remote site in case the units at the local site are unavailable. Do not use the Prime-Network n+m HA feature to make units available at the secondary site. Instead, add units to the gateway as regular (not standby) units.

Starting Units

The process for starting or restarting a unit is the same as for a non-HA environment: enter the anactl start command on the unit.

Forcibly Changing the Gateway IP Address on the Gateway and on All Units

If the gateway IP address did not change on a specific unit or set of units during a site-to-site failover or switchover, complete the following steps to forcibly switch the IP address. This procedure changes the IP address on the gateway and on all reachable units.

Step 1 Log into the gateway as user network38.

Step 2 Change directories to the /export/home/network38/Main/ha directory.

Step 3 Enter the following commands, where all IP addresses are logical IP addresses:

% perl changeSite.pl -force -newgwip new-gateway-IP -newdbip new-database-IP -oldgwip old-gateway-IP -olddbip old-database-IP[-newldapurl new-LDAP-URL -oldldapurl old-LDAP-URL][-newldapprefix new-LDAP-prefix -oldldapprefix old-LDAP-prefix][-newldapsuffix new-LDAP-suffix -oldldapsuffix old-LDAP-suffix][-newldapisssl new-LDAP-is-ssl -oldldapisssl old-LDAP-is-ssl]

• For a dual-node cluster, use the IP addresses of the Prime-Network and Oracle service groups.

• For a single-node cluster, use the lone IP address (used for the gateway and the database) of the single-node cluster.

• The old address is that of the cluster where the gateway and database were running before the failover or switchover.

• The new address is that of the cluster where the gateway and database will run.


OL-24641-01


• The new and old LDAP parameters (which appear within square brackets) are optional. For each new/old LDAP parameter pair, the value changes from the old value to the new value, assuming the comparable LDAP parameter was changed from the default value in Cisco Prime Network Administration.

Forcibly Changing the Gateway IP Address on a Single Unit

If the gateway or database IP address of a unit did not change during a site-to-site failover or switchover, complete the following steps to forcibly switch the IP address. This procedure changes the IP address on the unit only.

Step 1 Log into the unit as user network38.

Step 2 Change directories to the /export/home/network38/Main/ha directory.

Step 3 Enter the following command, where all IP addresses are logical IP addresses:

% perl switchUnit.pl new-gateway-IP old-gateway-IP new-database-IP old-database-IP

• For a dual-node cluster, use the IP addresses of the Prime-Network and Oracle service groups.

• For a single-node cluster, use the lone IP address (used for the gateway and the database) of the single-node cluster.

• The old address is that of the cluster where the gateway and database were running before the failover or switchover.

• The new address is that of the cluster where the gateway and database will run.

Failing Back from the Secondary Site to the Primary Site

Data replication normally runs from the primary site to the secondary site. When a failover to the secondary site occurs, the secondary replication site takes over as the new primary site. This can happen because of:

• An orderly migration as part of a planned site-to-site failover.

• An unplanned takeover due to a primary site failure.

A migration can occur if the primary site is up and running during the failover. After the failover is complete, the primary site can be taken offline. An example of this scenario is a failover due to planned maintenance. A takeover occurs when the replication link between the sites is down at the time of the failover. The secondary site must take over as the primary server, either automatically as part of bringing the replication resources (RVGs in the case of VVR) at the secondary site online (if auto-takeover is enabled), or manually. The failback process differs depending on whether a migration or a takeover occurred.

Failing Back After a Migration

As part of the process of failing back to the primary site, the data from the new primary must be replicated in the reverse direction to the original primary.


OL-24641-01


For VVR, the failback process is as follows:

Step 1 If the VVR service groups at the primary site are offline, bring them online as soon as possible. Doing so allows the data to replicate from the remote site (acting as primary) to the local site (acting as secondary).

Step 2 If possible, wait for all data from the remote site to replicate to the local site; this ensures the shortest failback time.

Step 3 Take the Prime-Network and Oracle service groups in the remote cluster offline.

Step 4 Bring the Oracle service group online on the desired server (normally P2).

Step 5 Bring the Prime-Network service group online on the desired server (normally P1).

For storage-based replication, the process is analogous to the preceding process, except that the process reflects the resources, service groups, and configuration for the specific storage used.

Failing Back After a Takeover

As part of the process of failing back to the primary site, the data at the original primary must be resynchronized with the data from the new primary.

For VVR, the failback process is as follows:

Step 1 Take the Prime-Network and Oracle service groups in the remote cluster offline.

Step 2 If fast failback is enabled, use the Cluster Manager GUI to perform failback synchronization:

a. Right-click the desired RVG resource.

b. Choose Actions.

c. Run the fbsync command on the appropriate server.

Step 3 Bring the Oracle service group online on the desired server (normally P2).

Step 4 Bring the Prime-Network service group online on the desired server (normally P1).

For storage-based replication, the process is analogous to the preceding process, except that the process reflects the resources, service groups, and configuration for the specific storage used.

Adding Prime Network Integrations in a Veritas Gateway HA/DR Environment

Interfaces

All integrations that run on an HA/DR-enabled Cisco Prime Network gateway should have a well-defined and documented:

• Command for starting the application.

• Command for stopping the application.

• List of dependencies.


OL-24641-01


• Method for determining application status, such as:

– Status command that returns 110 if the application is up and 100 if the application is down.

– Output of the ps command.

– pid file.

Due to the current Cisco Prime Network design, if the integration application runs as user network38, the VCS start, stop, and status commands must run as user root. The start command can be a script that switches users to user network38. The stop and status scripts must run without ever switching to user network38. (The stop and status commands must be able to run even when the Prime-Network volume is not mounted and the home directory for user network38 is not available. Without an available home directory, switching to user network38 in the script fails.) Do not use the MonitorProcesses (the output of the ps command) method to determine status, because VCS searches for processes owned by user root, not user network38. If the application runs as a different user, the start, stop, and status commands can be designed to run as that user, as long as the user’s home directory is placed directly under the root file system and not under /export/home. In this case, you can use the MonitorProcesses method to determine status.

Installation

All files under the /export/home/network38 directory are shared or replicated. Any application binaries or scripts installed under /export/home/network38 must be installed only once on the cluster, whereas files installed in a different location must be installed once per server. Similarly, any system changes or user directories created under the root partition must be made once per server in the cluster.

If the application consists of system files, user directories, binaries, and data, there should be a way to install one or both parts. If not, a procedure must be developed to install the application in an HA environment, such that the data is installed only once for the cluster, while the other files are installed once per server.

Connecting to the Gateway

All external applications or processes that connect to any of the gateway processes should use the hostname alias of the Prime-Network service group IP interface when possible. In a DR environment, a DNS resource can be set up in VCS to automatically change the mapping of the alias to the current gateway IP address. The machine running the external application must be set up to use this DNS to resolve the hostname.

If the process or application cannot use a hostname, the IP address of the gateway must be updated whenever the gateway fails over or fails back between sites. In this case, the startup script must use VCS CLI commands to determine the current IP address and use that address to configure the process or application.

If the gateway fails over within the local cluster where the IP address does not change, all existing BQL connections from client applications to the gateway break. The client must reconnect to the gateway after the gateway restarts.

Sample main.cf File for Local Redundancy Veritas Gateway HAinclude "Prime-NetworkTypes.cf"include "types.cf"include "OracleTypes.cf"

cluster ana-dual-local (


OL-24641-01


UserNames = { admin = bIJbIDiFJeJJhRJdIG }ClusterAddress = "10.56.22.66"Administrators = { admin })

system sh-v240-045 ()


group ClusterService (SystemList = { sh-v240-045 = 0, sh-v240-053 = 1 }AutoStartList = { sh-v240-045, sh-v240-053 }OnlineRetryLimit = 3OnlineRetryInterval = 120)

Application wac (StartProgram = "/opt/VRTSvcs/bin/wacstart"StopProgram = "/opt/VRTSvcs/bin/wacstop"MonitorProcesses = { "/opt/VRTSvcs/bin/wac" }RestartLimit = 3)

IP gcoip (Device = bge0Address = "10.56.22.66"NetMask = "255.255.255.0")

NIC gconic (Device = bge0)

gcoip requires gconicwac requires gcoip

// resource dependency tree//// group ClusterService// {// Application wac// {// IP gcoip// {// NIC gconic// }// }// }

group ana_sg (SystemList = { sh-v240-045 = 0, sh-v240-053 = 1 }AutoStartList = { sh-v240-045, sh-v240-053 })

Prime-NetworkGateway anaGW (Critical = 0NetlsnrResourceName = netlsnrPrime-NetworkIPResourceName = ana_ipOraIPResourceName = oracleLDAPURL = "ldap://10.56.22.122:389"


OL-24641-01


LDAPPrefix = CNLDAPSuffix = "CN=Users,DC=local-ldap,DC=ldap")

DiskGroup ana_dg (Critical = 0DiskGroup = anadg)

IP ana_ip (Critical = 0Device = bge0Address = "10.56.22.22"NetMask = "255.255.255.0")

Mount ana_mount (Critical = 0MountPoint = "/export/home"BlockDevice = "/dev/vx/dsk/anadg/anavol"FSType = vxfsFsckOpt = "-y")

NIC ana_nic (Critical = 0Device = bge0)

anaGW requires ana_ipanaGW requires ana_mountana_ip requires ana_nicana_mount requires ana_dg

// resource dependency tree//// group ana_sg// {// Prime-NetworkGateway anaGW// {// Mount ana_mount// {// DiskGroup ana_dg// }// IP ana_ip// {// NIC ana_nic// }// }// }

group oracle_sg (SystemList = { sh-v240-053 = 0, sh-v240-045 = 1 }AutoStartList = { sh-v240-053, sh-v240-045 })

DiskGroup ora_dg (Critical = 0DiskGroup = oradg)

IP ora_ip (


OL-24641-01


Critical = 0Device = bge0Address = "10.56.22.21"NetMask = "255.255.255.0")

Mount mount_db (Critical = 0MountPoint = "/opt/db"BlockDevice = "/dev/vx/dsk/oradg/oravol"FSType = vxfsFsckOpt = "-y")

Mount mount_dbarch (Critical = 0MountPoint = "/opt/dbarch"BlockDevice = "/dev/vx/dsk/oradg/oraarchvol"FSType = vxfsFsckOpt = "-y")

Mount mount_dbbackup (Critical = 0MountPoint = "/opt/dbbackup"BlockDevice = "/dev/vx/dsk/oradg/orabackupvol"FSType = vxfsFsckOpt = "-y")

Mount mount_dbdata (Critical = 0MountPoint = "/opt/dbdata"BlockDevice = "/dev/vx/dsk/oradg/oradatavol"FSType = vxfsFsckOpt = "-y")

Mount mount_dblogs (Critical = 0MountPoint = "/opt/dblogs"BlockDevice = "/dev/vx/dsk/oradg/oralogsvol"FSType = vxfsFsckOpt = "-y")

NIC oracle_nic (Critical = 0Device = bge0)

Netlsnr netlsnr (Critical = 0Owner = oracleHome = "/opt/db/oracle/product/11.2.0/dbhome_1")

Oracle oracle (Critical = 0Sid = MCDBOwner = oracleHome = "/opt/db/oracle/product/11.2.0/dbhome_1")

mount_db requires ora_dg


OL-24641-01


mount_dbarch requires ora_dgmount_dbbackup requires ora_dgmount_dbarch requires ora_dgmount_dblogs requires ora_dgnetlsnr requires oracleora_ip requires oracle_nicoracle requires mount_dboracle requires mount_dbarchoracle requires mount_dbbackuporacle requires mount_dbdataoracle requires mount_dblogsoracle requires ora_ip

// resource dependency tree//// group oracle_sg// {// Netlsnr netlsnr// {// Oracle oracle// {// Mount mount_db// {// DiskGroup ora_dg// }// Mount mount_dbarch// {// DiskGroup ora_dg// }// Mount mount_dbbackup// {// DiskGroup ora_dg// }// Mount mount_dbdata// {// DiskGroup ora_dg// }// Mount mount_dblogs// {// DiskGroup ora_dg// }// IP ora_ip// {// NIC oracle_nic// }// }// }// }

Sample main.cf File for Geographical Redundancy Veritas Gateway HA Using VVR—Local Cluster

include "Prime-NetworkTypes.cf"include "types.cf"include "OracleTypes.cf"

cluster ana-dual-local (UserNames = { admin = bIJbIDiFJeJJhRJdIG }ClusterAddress = "10.56.22.66"Administrators = { admin }


OL-24641-01


)

remotecluster ana-single-remote (ClusterAddress = "10.56.58.17")

heartbeat Icmp (ClusterList = { ana-single-remote }Arguments @ana-single-remote = { "10.56.58.17" })



group ClusterService (SystemList = { sh-v240-045 = 0, sh-v240-053 = 1 }AutoStartList = { sh-v240-045, sh-v240-053 }OnlineRetryLimit = 3OnlineRetryInterval = 120)



NIC gconic (Device = bge0)



group ana_sg (SystemList = { sh-v240-045 = 0, sh-v240-053 = 1 }ClusterList = { ana-dual-local = 0, ana-single-remote = 1 }AutoStartList = { sh-v240-045, sh-v240-053 })


OL-24641-01


Prime-NetworkGateway GW (Critical = 0NetlsnrResourceName = netlsnrPrime-NetworkIPResourceName = ana_ipOraIPResourceName = ora_ipLDAPURL = "ldap://10.56.22.122:389"LDAPPrefix = CNLDAPSuffix = "CN=Users,DC=local-ldap,DC=ldap")

Mount ana_mount (MountPoint = "/export/home"BlockDevice = "/dev/vx/dsk/anadg/anavol"FSType = vxfsFsckOpt = "-y")

RVGPrimary ana_rvgprimary (RvgResourceName = ana_rvg)

requires group ana_vvr_sg online local hardGW requires ana_mountana_mount requires ana_rvgprimary

// resource dependency tree//// group ana_sg// {// Prime-NetworkGateway GW// {// Mount ana_mount// {// RVGPrimary ana_rvgprimary// }// }// }

group ana_vvr_sg (SystemList = { sh-v240-045 = 0, sh-v240-053 = 1 }AutoStartList = { sh-v240-045, sh-v240-053 })

DiskGroup ana_dg (DiskGroup = anadg)

IP ana_ip (Device = bge0Address = "10.56.22.22"NetMask = "255.255.255.0")

NIC ana_nic (Device = bge0)

RVG ana_rvg (RVG = anarvgDiskGroup = anadg)


OL-24641-01


ana_ip requires ana_nicana_rvg requires ana_dgana_rvg requires ana_ip

// resource dependency tree//// group ana_vvr_sg// {// RVG ana_rvg// {// IP ana_ip// {// NIC ana_nic// }// DiskGroup ana_dg// }// }

group oracle_sg (SystemList = { sh-v240-053 = 0, sh-v240-045 = 1 }ClusterList = { ana-dual-local = 0, ana-single-remote = 1 }AutoStartList = { sh-v240-053, sh-v240-045 })

Mount mount_db (MountPoint = "/opt/db"BlockDevice = "/dev/vx/dsk/oradg/oravol"FSType = vxfsFsckOpt = "-y")

Mount mount_dbarch (MountPoint = "/opt/dbarch"BlockDevice = "/dev/vx/dsk/oradg/oraarchvol"FSType = vxfsFsckOpt = "-y")

Mount mount_dbbackup (MountPoint = "/opt/dbbackup"BlockDevice = "/dev/vx/dsk/oradg/orabackupvol"FSType = vxfsFsckOpt = "-y")

Mount mount_dbdata (MountPoint = "/opt/dbdata"BlockDevice = "/dev/vx/dsk/oradg/oradatavol"FSType = vxfsFsckOpt = "-y")

Mount mount_dblogs (MountPoint = "/opt/dblogs"BlockDevice = "/dev/vx/dsk/oradg/oralogsvol"FSType = vxfsFsckOpt = "-y")

Netlsnr netlsnr (Critical = 0


OL-24641-01


Owner = oracleHome = "/opt/db/oracle/product/11.2.0/dbhome_1")


RVGPrimary ora_rvgprimary (RvgResourceName = ora_rvg)

requires group oracle_vvr_sg online local hardmount_db requires ora_rvgprimarymount_dbarch requires ora_rvgprimarymount_dbbackup requires ora_rvgprimarymount_dbdata requires ora_rvgprimarymount_dblogs requires ora_rvgprimarynetlsnr requires oracleoracle requires mount_dboracle requires mount_dbarchoracle requires mount_dbbackuporacle requires mount_dbdataoracle requires mount_dblogs

// resource dependency tree//// group oracle_sg// {// Netlsnr netlsnr// {// Oracle oracle// {// Mount mount_db// {// RVGPrimary ora_rvgprimary// }// Mount mount_dbarch// {// RVGPrimary ora_rvgprimary// }// Mount mount_dbbackup// {// RVGPrimary ora_rvgprimary// }// Mount mount_dbdata// {// RVGPrimary ora_rvgprimary// }// Mount mount_dblogs// {// RVGPrimary ora_rvgprimary// }// }// }// }

group oracle_vvr_sg (SystemList = { sh-v240-053 = 0, sh-v240-045 = 1 }


OL-24641-01


AutoStartList = { sh-v240-053, sh-v240-045 })

DiskGroup ora_dg (DiskGroup = oradg)

IP ora_ip (Device = bge0Address = "10.56.22.21"NetMask = "255.255.255.0")

NIC oracle_nic (Device = bge0)

RVG ora_rvg (RVG = orarvgDiskGroup = oradg)

ora_ip requires oracle_nicora_rvg requires ora_dgora_rvg requires ora_ip

// resource dependency tree//// group oracle_vvr_sg// {// RVG ora_rvg// {// DiskGroup ora_dg// IP ora_ip// {// NIC oracle_nic// }// }// }

Sample main.cf File for Geographical Redundancy Veritas Gateway HA Using VVR—Remote Cluster

include "Prime-NetworkTypes.cf"include "types.cf"include "OracleTypes.cf"

cluster ana-single-remote (UserNames = { admin = fMNfMHmJNiNNlVNhMK }ClusterAddress = "10.56.58.17"Administrators = { admin })

remotecluster ana-dual-local (ClusterAddress = "10.56.22.66")

heartbeat Icmp (ClusterList = { ana-dual-local }Arguments @ana-dual-local = { "10.56.22.66" }


OL-24641-01


)

system sh-v215-0k0 ()

group ClusterService (SystemList = { sh-v215-0k0 = 0 }AutoStartList = { sh-v215-0k0 }OnlineRetryLimit = 3OnlineRetryInterval = 120)



NIC gconic (Device = bge0Protocol = IPv4)



group ana_sg (SystemList = { sh-v215-0k0 = 0 }ClusterList = { ana-dual-local = 0, ana-single-remote = 1 }Authority = 1AutoStartList = { sh-v215-0k0 })

Prime-NetworkGateway GW (NetlsnrResourceName = netlsnrPrime-NetworkIPResourceName = single_node_ipOraIPResourceName = single_node_ipLDAPURL = "ldap://10.56.20.110:389"LDAPPrefix = CNLDAPSuffix = "CN=Users,DC=remote-ldap,DC=local")


OL-24641-01


Mount ana_mount (Critical = 0MountPoint = "/export/home"BlockDevice = "/dev/vx/dsk/anadg/anavol"FSType = vxfsFsckOpt = "-y")

RVGPrimary ana_rvgprimary (Critical = 0RvgResourceName = ana_rvg)

requires group vvr_sg online local firmGW requires ana_mountana_mount requires ana_rvgprimary

// resource dependency tree//// group ana_sg// {// Prime-NetworkGateway GW// {// Mount ana_mount// {// RVGPrimary ana_rvgprimary// }// }// }

group oracle_sg (SystemList = { sh-v215-0k0 = 0 }ClusterList = { ana-dual-local = 0, ana-single-remote = 1 }Authority = 1AutoStartList = { sh-v215-0k0 })

Mount mount_db (Critical = 0MountPoint = "/opt/db"BlockDevice = "/dev/vx/dsk/oradg/oravol"FSType = vxfsFsckOpt = "-y")

Mount mount_dbarch (Critical = 0MountPoint = "/opt/dbarch"BlockDevice = "/dev/vx/dsk/oradg/oraarchvol"FSType = vxfsFsckOpt = "-y")

Mount mount_dbbackup (Critical = 0MountPoint = "/opt/dbbackup"BlockDevice = "/dev/vx/dsk/oradg/orabackupvol"FSType = vxfsFsckOpt = "-y")

Mount mount_dbdata (


OL-24641-01


Critical = 0MountPoint = "/opt/dbdata"BlockDevice = "/dev/vx/dsk/oradg/oradatavol"FSType = vxfsFsckOpt = "-y")

Mount mount_dblogs (Critical = 0MountPoint = "/opt/dblogs"BlockDevice = "/dev/vx/dsk/oradg/oralogsvol"FSType = vxfsFsckOpt = "-y")

Netlsnr netlsnr (Critical = 0Owner = oracleHome = "/opt/db/oracle/product/11.2.0/dbhome_1")


RVGPrimary ora_rvgprimary (Critical = 0RvgResourceName = ora_rvg)

requires group vvr_sg online local firmmount_db requires ora_rvgprimarymount_dbarch requires ora_rvgprimarymount_dbbackup requires ora_rvgprimarymount_dbdata requires ora_rvgprimarymount_dblogs requires ora_rvgprimarynetlsnr requires oracleoracle requires mount_dboracle requires mount_dbarchoracle requires mount_dbbackuporacle requires mount_dbdata

oracle requires mount_dblogs

// resource dependency tree//// group oracle_sg// {// Netlsnr netlsnr// {// Oracle oracle// {// Mount mount_db// {// RVGPrimary ora_rvgprimary// }// Mount mount_dbarch// {// RVGPrimary ora_rvgprimary// }// Mount mount_dbbackup// {


OL-24641-01


// RVGPrimary ora_rvgprimary// }// }// Mount mount_dbdata// {// RVGPrimary ora_rvgprimary// }// }// Mount mount_dblogs// {// RVGPrimary ora_rvgprimary// }// }// }// }

group vvr_sg (SystemList = { sh-v215-0k0 = 0 }AutoStartList = { sh-v215-0k0 })

DiskGroup ana_dg (Critical = 0DiskGroup = anadg)

DiskGroup ora_dg (Critical = 0DiskGroup = oradg)

IPMultiNIC single_node_ip (Critical = 0Address = "10.56.58.20"NetMask = "255.255.255.0"MultiNICResName = single_node_multinic)

MultiNICA single_node_multinic (Critical = 0Device = { bge0 = "10.56.58.140", bge1 = "10.56.58.140" })

RVG ana_rvg (Critical = 0RVG = anarvgDiskGroup = anadg)

RVG ora_rvg (Critical = 0RVG = orarvgDiskGroup = oradg)

ana_rvg requires ana_dgana_rvg requires single_node_ipora_rvg requires ora_dgora_rvg requires single_node_ipsingle_node_ip requires single_node_multinic

// resource dependency tree


OL-24641-01

Chapter 6 Installing a Cisco Prime Network Gateway and Embedded Database in a Veritas High Availability Upgrading to Cisco Prime Network 3.8 in a Veritas Gateway HA Environment

//// group vvr_sg// {// RVG ana_rvg// {// IPMultiNIC single_node_ip// {// MultiNICA single_node_multinic// }// DiskGroup ana_dg// }// RVG ora_rvg// {// IPMultiNIC single_node_ip// {// MultiNICA single_node_multinic// }// DiskGroup ora_dg// }// }

Upgrading to Cisco Prime Network 3.8 in a Veritas Gateway HA Environment

This section explains how to upgrade to Cisco Prime Network 3.8 in a high availability (HA), disaster recovery environment.

Before You BeginBefore upgrading to Cisco Prime Network 3.8 in an HA environment, complete the following steps:

Step 1 Familiarize yourself with the instructions for upgrading in a standard, non-HA environment (Chapter 12, “Upgrading to Cisco Prime Network 3.8”).

Step 2 Back up all Oracle data and the Cisco Prime Network installation in:

• /export/home/sheer4—Cisco Prime Network 3.6.7 or earlier.

• /export/home/network38—Cisco Prime Network 3.7 or later.

Step 3 Verify that the Veritas processes are error-free.

Understanding the Upgrade ProcessCisco Prime Network 3.8 includes an upgrade process for 3.7. For versions earlier than 3.7, you must first upgrade to 3.7. Additionally, if you are running 3.6.7 or earlier, the upgrade from 3.6.7 to 3.7 requires Oracle 10g. (Cisco Prime Network 3.6.7 does not support Oracle 11g.) If your installation uses Oracle 9i, you must upgrade to Oracle 10g before upgrading to Cisco Prime Network 3.7. If you are upgrading to Cisco Prime-Network 3.6.7 and Oracle 10g, complete the 3.6.7 upgrade before upgrading


OL-24641-01


to Oracle 10g. Cisco Prime-Network 3.7 supports Oracle 10g and Oracle 11g. When the upgrade to 3.7 is complete, you can optionally upgrade to Oracle 11g, which requires Veritas 5.0 with Maintenance Pack 3.

Differences Between an Upgraded Veritas Gateway HA Setup and a Newly Installed Veritas Gateway HA Setup

Due to the process of upgrading Oracle and Cisco Prime Network, a Cisco Prime Network 3.8 gateway Veritas HA setup that was upgraded from a version prior to 3.7.2 differs from a newly installed Veritas gateway HA setup. The differences, which do not affect HA operation or performance, Cisco Prime Network are described in this section.

Oracle Volumes and Directories

Compared to releases prior to 3.7.2, the Cisco Prime Network 3.8 Veritas HA gateway includes changes to the Oracle mount points and installation directories. For existing HA users, no changes are required to the mount points or directories beyond those changes required by the Oracle upgrade process. See Table 6-9.

Cisco Prime Network User and Home Directory

The Cisco Prime Network user and home directory names do not change during the upgrade. See Table 6-10.

Upgrading to Cisco Prime Network 3.8 in a Veritas Gateway HA EnvironmentDepending on your current Cisco Prime Network version, upgrading to Cisco Prime Network 3.8 might be a multistep process. Refer to the following section(s) in conjunction with Chapter 12, “Upgrading to Cisco Prime Network 3.8.”

Table 6-9 Differences in Oracle Volumes and Directories

Item Upgraded HA Setup New 3.8 HA Setup

Volume Mount Point Owner Oracle File Subdirectory Owned by User oracle

Mount Point Owner Oracle File Subdirectory Owned by User oracle

Data /opt/db oracle /opt/db/oracle /opt/db root /opt/db/oracle

Archives /opt/dblogs oracle — /opt/dbarch root /opt/dbarch/oraarch

Logs /opt/dblogs oracle — /opt/dlogs root /opt/dbarch/oralogs

Additional Data /opt/dbdata oracle — /opt/dbdata root /opt/dbdata/oradata2

Table 6-10 Differences in Prime-Network User and Prime-Network Home Directory Names

Item Upgraded HA Setup New 3.8 HA Setup

Prime-Network user sheer network38

Prime-Network home directory

/export/home/sheer4 /export/home/network38


OL-24641-01


Upgrading from Cisco Prime Network 3.6.6 or Earlier to 3.6.7

Step 1 Bring the Prime-Network service group online on P1. See the upgrade instructions in the Cisco Prime Network 3.7.1 Installation Guide and run all commands on P1 only.

Step 2 Verify that the following files are the same on P1, P2, and S1 (if you are using geographical redundancy):

• /etc/passwd

• /etc/group

• /etc/shadow

Separating the Cluster for Geographical Redundancy in a Veritas Gateway HA Environment

Complete the following steps to separate the geographical cluster into separate local clusters:

Step 1 Verify that the Prime-Network service group is online on P1.

Step 2 Remove the Prime-Network and Oracle Reliable Datagram Sockets (RDSs):

a. Verify that the servers in the local cluster are designated as primary on both the Prime-Network and Oracle RDSs.

b. Stop replication on both RDSs.

c. Remove the secondary server from both RDSs.

d. Disassociate all volumes on both RDSs.

e. Remove the primary servers from both RDSs

Step 3 Use the Veritas Cluster Server (VCS) GUI to make the Prime-Network and Oracle service groups (ana_sg and oracle_sg) local.

Step 4 In the VCS GUI resource view, remove all RVG and RVGPrimary resources from the local and remote clusters.

Step 5 Use Cisco Prime Network Administration to connect to Cisco Prime Network and stop all VNEs.

Step 6 Use the VCS agent to stop Cisco Prime Network.

Step 7 Run the UNIX fsck utility on all local volumes.

Step 8 Bring the Oracle and Prime-Network service groups online on S1.

Step 9 Point any applications to the Cisco Prime Network instance that is running on S1.

Upgrading from Cisco ANA 3.6.7 to 3.7

Step 1 If the Prime-Network service group is not online on P1, bring it online.

Step 2 If Cisco Prime Network is still running on P1, use Cisco Prime Network Administration to connect to Cisco Prime Network and stop all VNEs.

Step 3 Use the Prime-NetworkGateway VCS agent to take Cisco Prime Network offline.


OL-24641-01

http://www.cisco.com/en/US/partner/products/ps6776/prod_installation_guides_list.html


Step 4 Remove the Prime-NetworkGateway resource from the Prime-Network service group on the local cluster.

Step 5 Log into P1 as the root user and enter the following commands to remove the package for the Prime-NetworkGateway VCS agent and the HA scripts:

# pkgrm CANAtyp# pkgrm CANAha

Step 6 Log into P2 as the root user and enter the following command to remove the package for the Prime-NetworkGateway VCS agent:

# pkgrm CANAtyp

Step 7 As the root user, enter the following command on P1:

# /opt/VRTS/bin/hatype -delete ANAGateway

Step 8 If you are using Oracle 9i:

a. Bring the Oracle service group online on P2.

b. Use the Veritas agent to take the Oracle listener and database offline.

c. As the oracle user, connect to P2 and upgrade to Oracle 10g, which Cisco ANA 3.7 requires. After the upgrade, you might need to reconfigure the Oracle agents.

Step 9 Complete all preupgrade steps in the Cisco Active Network Abstraction 3.7.1 Installation Guide, section “Upgrading Cisco Prime Network.”

Step 10 Upgrade to Cisco ANA 3.7:

a. Bring the Oracle service group online on P1. (The Prime-Network service group should already be online on P1.) Verify that the listener and database are online.

b. Install Cisco ANA 3.7 according to the instructions in the Cisco Active Network Abstraction 3.7.1 Installation Guide, section “Upgrading to Cisco Prime Network 3.7.” Be sure to do the following (assuming that 3.6.7 was installed as the user sheer; if 3.6.7 was installed as a different user, substitute that username):

– Install Cisco ANA 3.7 by logging in as the root user and entering the following command:

perl install.pl -migrate sheer

– Change the password for the user sheer.

– Switch to the user sheer and enter the following command:

network-conf

c. The installation process must now run on P2, which requires an /export/home/sheer4 directory. During the original HA installation on P2, a directory named /export/home/sheer4 was created on the root partition; it was subsequently renamed to /export/home/local.unused.sheer4. If that directory still exists, enter the following commands to rename it to /export/home/sheer4:

# cd /export/home# mv local.unused.sheer4 sheer4

If the directory does not exist, recreate it by going to P1 (where the shared Prime-Network volume is still mounted under /export/home). In the /export/home directory, use the tar command to create an archive of the sheer4 directory. Copy the archive to P2 and open the archive under the locally mounted /export/home directory. Note that the contents of this directory are not used, but it must be available for the upgrade procedure to work.


OL-24641-01





d. With the Prime-Network and Oracle service groups still online on P1, as the root user, rerun the Cisco ANA 3.7 installation process on P2. Start from the install.pl command as documented and stop before entering the network-conf command.

e. Switch to the sheer user and enter the following command:

% network-conf -ha_redundant

The -ha_redundant option completes the migration without migrating the registry or the database, which were already migrated when network-conf ran on P1 with the Prime-Network and Oracle service groups connected.

f. As the root user, enter the following commands to move the local Prime-Network installation home directory (which is /export/home/sheer4 by default). If you are using a different home directory, substitute the appropriate value.

# cd /export/home# mv sheer4 local.unused.sheer4

Upgrading from Cisco ANA 3.7, 3.7.1, 3.7.2, and 3.7.3 to Cisco Prime Network 3.8

Step 1 Complete the Veritas cluster preupgrade tasks:

a. If the Prime-Network service group is not already online on P1, bring it online.

b. Remove the Prime-NetworkGateway resource from the Prime-Network service group in the local cluster.

c. If the Oracle installation directory changed because of an upgrade, update the directory names for both the oracle and netlsnr resources in the local cluster.

Step 2 Upgrade Cisco Prime Network:

a. As the Prime-Network user, log into P1.

b. Follow the instructions in Upgrading from Cisco ANA 3.7.x to Cisco Prime Network 3.8, page 12-9, running all commands on P1. When running the upgrade.pl script, include the -ha flag.

c. On P1, use the CLI command anactl stop to stop Cisco Prime Network.

d. Switch the Prime-Network service group to P2. (If the local cluster does not contain server P2, proceed to Step 3.)

e. Log into P2 as the Prime-Network user.

f. Follow the instructions in Upgrading from Cisco ANA 3.7.x to Cisco Prime Network 3.8, page 12-9, running all commands on P2. When running the upgrade.pl script, include the -ha_redundant flag.

Step 3 Complete the Veritas cluster postupgrade tasks:

a. From the Cluster Manager GUI, choose File > Import Types and import the Prime-NetworkGateway resource type for the local cluster. The Prime-Network resource types file resides in /etc/VRTSvcs/conf/Prime-NetworkTypes.cf.

a. When finished, exit the Cluster Manager GUI, then restart it.

b. Reinstall the Prime-NetworkGateway agent in the Prime-Network service group in the local cluster.

c. If an upgrade changed the Oracle installation directory, update the directory names for the oracle and netlsnr resources. Then restart the Oracle database and listener using the VCS agent.

d. If the Prime-Network service group on P1 is not online, bring it online.


OL-24641-01

Chapter 6 Installing a Cisco Prime Network Gateway and Embedded Database in a Veritas High Availability Rolling Back to an Earlier Cisco ANA Version in a Veritas Gateway HA Environment

e. If the Oracle service group on P2 is not online, bring it online.

f. Use the VCS agent to start Cisco Prime Network on P1.

Upgrading the Remote Server (S1) in a Geographical Redundancy Veritas Gateway HA Environment

Step 1 Complete the Veritas cluster preupgrade tasks:

a. Switch both the Prime-Network and Oracle service groups to be online on S1.

b. In the remote cluster, remove the Prime-NetworkGateway resource from the Prime-Network service group.

c. If an upgrade changed the Oracle installation directory, update the directory names for both the oracle and netlsnr resources in the remote cluster.

Step 2 Upgrade Cisco Prime Network:

a. Log into S1 as the Prime-Network user.

b. Follow the instructions in Upgrading from Cisco ANA 3.7.x to Cisco Prime Network 3.8, page 12-9, running all commands on S1. When running the upgrade.pl script, include the -ha_redundant flag.


a. From the Cluster Manager GUI, choose File > Import Types and import the Prime-NetworkGateway resource type for the remote cluster. The Prime-Network resource types file resides in /etc/VRTSvcs/conf/Prime-NetworkTypes.cf. When you are finished, exit the Cluster Manager GUI and restart it.

b. In the remote cluster, reinstall the Prime-NetworkGateway agent in the Prime-Network service group.

c. Start Prime-Network on S1 using the VCS Prime-NetworkGateway agent and check that it is possible to connect using Cisco Prime Network Administration.

d. Switch the Prime-Network and Oracle service groups back to the local cluster (the default is Prime-Network on P1 and Oracle on P2) using VCS and check that it is possible to connect using Cisco Prime Network Administration.

Rolling Back to an Earlier Cisco ANA Version in a Veritas Gateway HA Environment

Step 1 Complete the following Veritas cluster preupgrade tasks:

a. Bring the Prime-Network service group online on P1.

b. Use Cisco Prime Network Administration to connect to Cisco Prime Network and stop all VNEs.

c. Use the Veritas agent in the local cluster to take Cisco Prime Network offline.

d. Remove the Prime-NetworkGateway resource from the Prime-Network service group on the local cluster.


OL-24641-01


e. If you are using a geographical redundancy HA environment, remove the Prime-NetworkGateway resource from the Prime-Network service group on the remote cluster.

f. As the root user, enter the following command on P1:

# /opt/VRTS/bin/hatype -delete Prime-NetworkGateway

g. If you are using a geographical redundancy HA environment, repeat substeps d to f for S1.

Step 2 Depending on the Cisco Prime Network version that you want to roll back to, complete one of the following sections on P1 only:

• Rolling Back from Cisco Prime Network 3.8 to Cisco ANA 3.7.x, page 12-18

• Rolling Back from Cisco Prime Network 3.8 to Cisco ANA 3.6.x, page 12-19

Step 3 If you are rolling back to a version older than Cisco Prime Network 3.7.2, reinstall the CANAtyp and CANAha packages for the HA version that was previously installed. If you are rolling back to Cisco ANA 3.7.2, continue with the next step.


a. From the Cluster Manager GUI, choose File > Import Types and import the Prime-NetworkGateway resource type for the local and remote clusters (if applicable). The Prime-Network resource types file resides in /etc/VRTSvcs/conf/Prime-NetworkTypes.cf. When you are finished, exit the Cluster Manager GUI and restart it.

b. Reinstall the ANAGateway agent in the ANA service group in the local and remote clusters (if applicable) for the HA version that was previously installed.

c. Use the VCS agent to restart Cisco ANA.


OL-24641-01



OL-24641-01

Installing a Cisco Prime Network Gateway and Embedded ... · Cisco Prime Network 3.8 Installation...

Documents

Transcript of Installing a Cisco Prime Network Gateway and Embedded ... · Cisco Prime Network 3.8 Installation...