Aix Hacmp Cluster

© Copyright IBM Corporation 2004Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

Welcome to:

3.1

AU54 - HACMP System Administration I: AU54 - HACMP System Administration I: Planning and ImplementationPlanning and Implementation

© Copyright IBM Corporation 2004

Course Objectives

After completing this unit, you should be able to: Explain what High Availability isOutline the capabilities of HACMP for AIXDesign and plan a highly available clusterCorrectly configure networks and shared disks for a highly available clusterInstall and configure HACMP in the following modes of operation:Primary node with an idle standby nodeTwo applications on two nodes in a Mutual Takeover configurationSingle application running concurrently on two nodes (optional)Perform basic system administration of an HACMP clusterPerform basic customization of an HACMP clusterCarry out basic problem determination and recovery


Course AgendaDay 1

09:00 - 09:30 Course Introduction09:30 - 10:20 Unit 1 - Introduction to High Availability10:20 - 10:30 Break10:30 - 12:00 Unit 2 - Introduction to HACMP for AIX12:00 - 13:00 Lunch Break13:00 - 14:00 Unit 3 Topics 1,2 - Shared Storage Concepts,

Technologies14:00 - 14:10 Break14:10 - 14:50 Unit 3 Topic 3 - Shared Storage From AIX 14:50 - 16:30 Exercise 1: Cluster Design

Exercise 2: Planning Storage Exercise 3: Setup Storage


Course AgendaDay 2

09:00 - 10:00 Unit 4 Topics 1,2 - Networking Considerations 10:00 - 10:10 Break

10:10 - 11:10 Unit 4 Topics 3,4 - Networking Considerations11:10 - 11:20 Break11:20 - 12:00 Unit 5 Topic 1 - HACMP Installation12:00 - 13:00 Lunch Break13:00 - 16:30 Exercise 4: Network Planning, Setup and Test Exercise 5: HACMP Software Installation Exercise 6: Client Setup


Course AgendaDay 3

09:00 - 10:00 Unit 5 Topics 2,3 - HACMP Architecture10:00 - 10:10 Break10:10 - 11:00 Unit 6 Topic 1 - Cluster Configuration11:00 - 11:10 Break11:10 - 12:00 Unit 6 Topic 2 - Other Configuration Scenarios12:00 - 13:00 Lunch Break13:00 - 16:30 Exercise 7: Cluster Configuration Exercise 8: Application Integration Exercise 9: Mutual Takeover


Course AgendaDay 4

09:00 - 10:00 Unit 7 - Cluster Single Point of Control10:00 - 10:10 Break10:10 - 11:00 Unit 8 - Dynamic Reconfiguration11:00 - 11:10 Break11:10 - 12:00 Unit 9 - Integrating NFS Into HACMP12:00 - 13:00 Lunch Break13:00 - 16:30 Exercise 10: HACMP Extended Features Exercise 11: Resource Group Options Exercise 12: Network File System (NFS)


Course AgendaDay 5

09:00 - 09:50 Unit 10 - Cluster Customization09:50 - 10:00 Break10:00 - 11:00 Unit 11 - Problem Determination and Recovery11:00 - 11:10 Break11:10 - 12:00 Unit 12 - Documenting Your System12:00 - 13:00 Lunch Break13:00 - 13:45 Exercise 13: Error Notification13:45 - 16:30 Open Lab


Lab ExercisesPoints to note :

Work as a team and split the workload.Manuals are available online.HACMP software has been loaded and may have already been installed.TCP/IP and LVM have not been configured.Each lab must be completed successfully before continuing on to the next lab, as each lab is a prerequisite for the next one.Any questions, ask your instructor.


Course Summary

Having completed this unit, you should understand that: There is ample time for the lab exercises.Thorough design, planning and teamwork are essential.Prior AIX, LVM, Storage Management and TCP/IP experience is assumed and required.


Welcome to:

3.0.23.0.3

Introduction to High-AvailabilityIntroduction to High-Availability


Unit Objectives

After completing this unit, you should be able to: Understand what high availability isUnderstand why you might need high availabilityOutline the various options for implementing high availabilityCompare and contrast the high availability optionsState the benefits of using highly available clustersUnderstand the key considerations when designing and implementing a high availability clusterBe familiar with the basics of risk analysis


So, What Is High Availability?High Availability is...

The masking or elimination of both planned and unplanned downtime.The elimination of single points of failure (SPOFs).Fault resilience, but NOT fault tolerance.

Workload Fallover

Production Standby

Client

WAN


Planned downtime:

Hardware upgradesRepairsSoftware updatesBackupsTestingDevelopment

So Why Is Planned Downtime Important?

High availability solutions should reduce both planned and unplanned downtime.

Unplanned downtime:

Administrator ErrorApplication failureHardware faultsEnvironmental Disasters

1.0%14.0%

85.0%

Hardware Failure (1%)Other unplanned downtime (14%)Planned downtime (85%)


Continuous Availability Is the Goal

Continuous Availability

Continuous Operations

High Availability

Elimination of Downtime

Masking or elimination of planned downtime

Masking or elimination of unplanned downtime


Eliminating Single Points of Failure

Cluster Object Eliminated as a single point of failure by . . .

Node Using multiple nodes

Power Source Using multiple circuits or uninterruptible power supplies

Network adapter Using redundant network adapters

Network Using multiple networks to connect nodes

TCP/IP Subsystem Using serial networks to connect adjoining nodes and clients

Disk adapter Using redundant disk adapters

Disk Using redundant hardware and disk mirroring and/or striping

Application Assigning a node for application takeover; configuring an application monitor

A fundamental design goal of (successful) cluster design isthe elimination of single points of failure (SPOFs).


Availability - from Simple to Complex

Stand-alone

Enhanced

High AvailabilityCluster

FaultTolerant


The Stand-alone SystemThe stand-alone system may offer limited availability benefits:

Journaled FilesystemDynamic CPU DeallocationService ProcessorRedundant PowerRedundant CoolingECC MemoryHot Swap AdaptersDynamic KernelDisk mirroring

Example single points of failure:

Disk Adapter/ Data PathsNo Hot Swap StoragePower for Storage ArraysCooling for Storage ArraysHot Spare Storage Node/Operating SystemNetworkNetwork AdapterApplicationSite Failure (SAN distance)Site Failure (via mirroring)


The Enhanced SystemThe enhanced system may offer increased availability benefits:

Journaled FilesystemDynamic CPU DeallocationService ProcessorRedundant PowerRedundant CoolingECC MemoryHot Swap AdaptersDynamic Kernel Disk MirroringRedundant Disk adapters/multiple pathsHot Swap StorageRedundant Power for Storage ArraysRedundant Cooling for Storage ArraysHot Spare Storage

Example single points of failure:Node/Operating SystemNetwork AdapterNetworkApplicationSite Failure (SAN distance)Site Failure (via mirroring)


High-Availability Clusters (HACMP)Clustering technologies offer high-availability:

Journaled FilesystemDynamic CPU DeallocationService ProcessorRedundant PowerRedundant CoolingECC MemoryHot Swap AdaptersDynamic KernelRedundant Data PathsData MirroringHot Swap StorageRedundant Power for Storage ArraysRedundant Cooling for Storage ArraysHot Spare StorageDual Disk AdaptersRedundant nodes (operating system)Redundant Network AdaptersRedundant NetworksApplication MonitoringSite Failure (SAN distance)

Example single points of failure:Site Failure (via mirroring)


Fault-Tolerant ComputingFault-tolerant solutions should not fail:

Lock Step CPUsHardened Operating SystemHot Swap StorageContinuous Restart

Example single points of failure:

Site Failure (SAN distance)Site Failure (via mirroring)


Availability Solutions

Stand-alone Enhanced Standalone

High Availability Clusters

Fault-tolerant Computers

Solutions

Availability benefits

Journaled FilesystemDynamic CPU DeallocationService ProcessorRedundant PowerRedundant CoolingECC MemoryHot Swap AdaptersDynamic Kernel

Redundant Data PathsData MirroringHot Swap StorageRedundant Power for Storage Arrays

Redundant Cooling for Storage Arrays

Hot Spare Storage

Redundant ServersRedundant NetworksRedundant Network Adapters

Heartbeat MonitoringFailure DetectionFailure DiagnosisAutomated FalloverAutomated Reintegration

Lock Step CPUsHardened Operating SystemRedundant MemoryContinuous Restart

Downtime Couple of days Couple of hours Depends, but typically 3 mins In theory, none!

Data Availability Good as your last full backup Last transaction Last transaction No loss of Data

Relative Cost* 1 1.5 2-3 10+

* All other parameters being equal.

Simple Complex


So, What About Site Failure?

Toronto London

Data Replication

Near distance (using SAN) supported by HACMP 5.2Far distance, (requires data mirroring) invest in a Geographic Clustering Solution (for example, HACMP XD*)

Distance unlimitedData replication across a geographyApplication, disk and network independentAutomated site failover and reintegrationA single cluster across two sites

*The HACMP XD feature of HACMP contains IBM's HAGEO product and PPRC support .


Why Might I Need High Availability?60% of all large companies now operate round the clock (7x24)

Losses on failure:330,000 $US per hour (industry average)Peak losses: 130,000 $US per minute (telephone network)Loss of customer loyaltyLoss of customer confidence

And, if there is no disaster recovery:50% of affected companies will never reopen90% of affected companies are out of business in less than two years

Note: High Availability is NOT a Disaster Recovery solution.

$ £

050

100150200Lose of Revenue $M

E


Benefits of High-Availability SolutionsHigh-availability solutions offer the following benefits:

Standard components (no specialized hardware)Can be built from existing hardware (no need to invest in new kit)Work with just about any applicationWork with wide range of disk and network typesNo specialized operating system or microcodeExcellent availability at low cost

+ =

Standard Components High Availability Solution


High availability

Continuous operation

Continuous availability

SystemsManagement

People

Data

Hardware

Software

Environment

Networking

Other Considerations for High-AvailabilityHigh-availability solutions require the following:

Thorough design and detailed planningElimination of single points of failureSelection of appropriate hardwareCorrect implementationDisciplined system administration practicesDocumented operational proceduresComprehensive testing


A Philosophical View of High AvailabilityThe goal of an HA cluster is to make a service highly available.

Users aren't interested in highly available hardware.Users aren't even interested in highly available software.Users are interested in the availability of services.

Therefore, use the hardware and the software to make the services highly available.Cluster design decisions should be judged on the basis of whether or not they:

Contribute to availability (for example, eliminate a SPOF)Detract from availability (for example, gratuitous complexity)

Since it is impractical if not impossible to truly eliminate all SPOFs, be prepared to use risk analysis techniques to determine which SPOFs are tolerated and which must be eliminated


Classic Risk Analysis1. Identify relevant policies

What existing risk tolerance policies are available?2. Study the current environment

Understand what strengths (for example, server room is on a properly sized UPS) and weaknesses (for example, no disk mirroring) exist today

3. Perform requirements analysisJust how much availability is required?What is the acceptable likelihood of a long outage?

4. Hypothesize vulnerabilitiesWhat can possibly go wrong?

5. Identify and quantify risksThe statistical probability of something going wrong over the life of the project (or the likely number of times something will go wrong over the life of the project) multiplied by the cost of an occurrence

6. Evaluate countermeasuresWhat does take to reduce the risk (by reducing the likelihoodor consequences of an occurrence) to an acceptable level

7. Make decisions, create a budget and plan the cluster


What Do We Plan to Achieve This Week?

A

B

A

B

Your mission this week is to build a two-node highly available cluster using two previously separate pSeries systems, each of which has an application which needs to be made highly available.


Checkpoint1. Which of the following is a characteristic of high availability?

a. High availability always requires specially designed hardware components.b. High availability solutions always require manual intervention to ensure recovery following

failover. c. High availability solutions never require customization.d. High availability solutions offer excellent price performance when compared with Fault

Tolerant solutions.

2. True or False?High availability solutions never fail.

3. True or False?A thorough design and detailed planning is required for all high availability solutions.

4. True or False?The cluster shown on the foil titled "What We Plan to Achieve This Week" has no obvious single points of failure.

5. A proposed cluster with a two year life (for planning purposes) has a vulnerability which is likely to occur twice per year at a cost of $10,000 per occurrence. It costs $25,000 in additional hardware costs to eliminate the vulnerability. Should the vulnerability be eliminated?

a. yesb. no


Checkpoint Answers1. Which of the following is a characteristic of high availability?

a. High availability always requires specially designed hardware components.b. High availability solutions always require manual intervention to ensure recovery following

failover. c. High availability solutions never require customization.d. High availability solutions offer excellent price performance when compared with Fault

Tolerant solutions.

2. True or False?High availability solutions never fail.

3. True or False?A thorough design and detailed planning is required for all high availability solutions.

4. True or False? (the local area network is a SPOF)The cluster shown on the foil titled "What Will We Achieve This Week" has no obvious single points of failure.

5. A proposed cluster with a two year life (for planning purposes) has a vulnerability which is likely to occur twice per year at a cost of $10,000 per occurrence. It will cost $25,000 in additional hardware costs to eliminate the vulnerability. Should the vulnerability be eliminated?

a. yes ($25,000 is less than $10,000 times four)b. no


Unit Summary

Having completed this unit, you should be able to: Understand what high availability isUnderstand why you might need high availabilityOutline the various options for implementing high availabilityCompare and contrast the high-availability optionsState the benefits of using highly available clustersUnderstand the key considerations when designing and implementing a high-availability clusterBe familiar with the basics of risk analysis


Welcome to:

3.13.0.3

Introduction to HACMP for AIXIntroduction to HACMP for AIX


Unit Objectives

After completing this unit, you should be able to: Outline the features and benefits of HACMP for AIXDescribe the physical and logical components of an HACMP clusterUnderstand how HACMP operates in typical configurationsDescribe the evolution of the HACMP 5.2 product family


HACMP BasicsAfter completing this topic, you should be able to: Outline the features and benefits for HACMP for AIXDescribe the HACMP concepts of topology and resourcesGive examples of topology components and resourcesProvide a brief description of the software and hardware components of a typical HACMP cluster


IBM's HA Solution for AIXHigh Availability Cluster MultiprocessingBased on cluster technologyProvides two environments:

High Availability: the process of ensuring an application is available for use through the use of serially accessible shared data and duplicated resourcesCluster MultiProcessing: concurrent access to shared data


A Highly Available Cluster

Clusters based upon HACMP 5.2 can contain between 2 and 32 nodes.

A cluster comprises physical components (topology) and logical components (resources).

Application Fallover

Resource

Group


HACMP's Topology Components

The Topology components consist of a cluster, nodes and the technology which connects them together.

Node Node

IP

Network Non-IP

Network

Communication

Interface

Cluster

Communication

Device


HACMP's Resource Components

Volume G

roup

Appli

catio

n Se

rver

start

scrip

tst

op sc

ript

Resources

Service IP

Address

File S

ystem

Resource Group Policiesstartupfalloverfallback

Resource Group Node list Policies


Solution Components


AIX's Contribution to High Availability

Object Data Manager (ODM) System Resource Controller (SRC)Logical Volume Manager (LVM)Journaled File System (JFS)Online JFS Backup (splitlvcopy) Work Load Manager (WLM)Quality of Service (Qos) External Boot Software Installation Management (installp)Reliable Scalable Cluster Technology (RSCT)


High-end

pSeries 680

pSeries 690

IBM

IBM

IBM

Hardware Prerequisites

All pSeries systems work with HACMP in any combination of nodes within a cluster. However, a minimum of four free adapter slots is recommended.

IBM Eserver pSeries

pSeries

IBM

server

pSeries

pSeries 610 Model 6E1

pSeries 620Models 6F0, 6F1

IBM

pSeries 610 Model 6C1

IBMserver

pSeries pSeries 640 Model B80

Entry Deskside

Entry Rack

IBM

pSeries 630 Model 6E4

pSeries 630 Model 6C4

IBM

H C R U6

IBM

server

pSeries

Midrange

pSeries 660

pSeries 670

pSeries 650 pSeries 655

IBM

IBM

IBM

IBM

server

pSeries

pSeries 520 pSeries 550 pSeries 570


Most HACMP clusters require shared storage. Disk technologies which support multihost connections include: SCSI, SSA and FC-AL (with or without RAID).

Supported Storage Environments

DiskDrive

SCSIModule

SCSIController T

Host System

Maximum 25m

DiskDrive

SCSIModule

DiskDrive

SCSIModule

DiskDrive

SCSIModule

SAN

RS/6000

IBM

RS/6000

RS/6000

ESS Fibre

FAStT

SSA Adapter


Supported IP network interfaces include: Ethernet (10 MB,100 MB, 1 GB), Etherchannel, Token-Ring, FDDI, ATM, Fibre Channel and the SP switches. Supported non-IP network devices include: RS232/422, SSA adapters, (target Mode SSA) and Enhanced Concurrent Volume Group disks.

Supported Networking Environments

Traffic Flow

PC

Traffic Flow Server

Server

Server

Token Ring

FDDI

Non -IP

Server Server

Ethernet

Server

ServerServer

PC


Some Clusters Do Not Have Shared Disks

Clusters providing firewall services do not usually have shared disks.Can you think of any other examples?


So What Is HACMP Really?

HACMP is an application which:Monitors cluster components, Detects status changes, Diagnoses and recovers from failures and...Reintegrates previous failed components back in to the cluster upon recovery.Provides tools to create cluster wide definitions and to synchronize

http://www.ibm.com/servers/eserver/pseries/solutions/ha/

cspocRecovery

programs event scripts

CLSTRMGR

rgmove RSCT, RMC

dareclverify clsmuxpd


Additional Features of HACMP

HACMP is shipped with utilities to simplify configuration, monitoring, customization and cluster administration.

Configuration assistantplanning worksheetsclstat

clsmuxpdsnmp mib

clinfo smit via Web

auto testsTivoli

snmp

Application Monitoring


Some Assembly Required

HACMP is not an out of the box solution.HACMP's flexibility allows for complex customization in order to meet availability goals.

Customized Pre-event scripts

HACMP core events

Customized post-event scripts

Application start scriptApplication stop script

ApplicationServer


Overview of the Implementation ProcessPlan for network, storage, and application

Eliminate single points of failureDefine and configure the AIX environment

Storage (adapters, LVM volume group, filesystem)Networks (ip interfaces, /etc/hosts, non-ip networks and devices)Application start and stop scripts

Install the HACMP filesets and rebootConfigure the HACMP environment

TopologyCluster, node names, HACMP ip and non-ip networks

Resources:Application Server Service labels

Resource group:Identify name, nodes, policiesResources: Application Server, service label, VG, filesystem

Synchronize then start HACMP


Hints to Get Started

Create a cluster diagram.Use the planning sheets.Try to reduce Single Points of Failure (SPOFs). Always include a non IP network. Mirror across power and buses. Document test plan.Be methodical.Execute the test plan prior to placing the cluster into production!

hints

Public Network

Resourse Group databaserg containsVolume Group = dbvghdisk3, hdisk4, hdisk5, hdisk6, hdisk7Major # = 51JFS Log = dblvlogLogical Volume = dblv1, dblv2FS Mount Point = /db, /dbdata

Node Name = nodea Resource group = dbrg Applications = database Resources = cascadingA-B Priority = 1,2 CWOF = yes

Label = a_tmssa Device = /dev/tmssa1

Label = a_tty Device = /dev/tty1

Node Name =nodeb Resource group = httprg Applications = http Resources = cascadingB-A Priority = 2,1 CWOF = yes

Label = b_tmssa Device = /dev/tmssa2

Label = a_tty Device = /dev/tty1

tmssa network

serial network

VG = dbvgRaid5100GB

VG =httpvgRaid19GB

rootvgraid1

9.1GB

rootvgraid1

9.1GB

usercommunity

HACMP Clusterfor

the ABC company

Resourse Group httprg containsVolume Group = httpvghdisk2,hdisk8Major # = 50JFS Log = httplvlogLogical Volume = httplvFS Mount Point = /http

Node A IP Label IP Address NetmaskService webserv 192.168.9.5 255.255.255.0Boot nodebboot 192.168.9.6 255.255.255.0Standby nodebstand 192.168.254.3 255.255.255.0

Node A IP Label IP Address NetmaskService database 192.168.9.3 255.255.255.0Boot nodeaboot 192.168.9.4 255.255.255.0Standby nodeastand 192.168.254.3 255.255.255.0


Checkpoint1. Which of the following are examples of topology components in

HACMP 5.2 (select all that apply)?a. Nodeb. Networkc. Service IP labeld. Hard disk drive

2. True or False?All clusters require shared disk for storage of HACMP log files.

3. True or False?All nodes in an HACMP cluster must have roughly equivalent performance characteristics.


Checkpoint Answers1. Which of the following are examples of topology components in

HACMP 5.1 (select all that apply)?a. Nodeb. Networkc. Service IP label*d. Hard disk drive

2. True or False?All clusters require shared disk for storage of HACMP log files.

3. True or False?All nodes in an HACMP cluster must have roughly equivalent performance characteristics.

*Service IP labels were considered topology components in HACMP 4.5.


What Does HACMP Do?After completing this topic, you should be able to: Describe the failures which HACMP detects directlyDescribe how HACMP deals with network adapter and node failuresProvide an overview of typical two node cluster configurationsDescribe some of the considerations and limits of an HACMP cluster


Just What Does HACMP Do?

HACMP moves resource group resources based upon detecting three failures by design, others can be dealt with by customization.

A node failureA network adapter failure A network failure

HACMP can also monitor applications, processor load and available disk capacity.


What Happens When Something Fails?

How the cluster responds to a failure depends on what has failed, what the resource group's fallover policy is, and if there are any resource group dependencies The cluster's configuration is determined by the application's requirementsTypically another equivalent component takes over duties of failed component (for example, another node takes over from a failed node)


What Happens When a Problem Is Fixed?

?How the cluster responds to the recovery of a failed component depends on what has recovered, what the resource group's fallback policy is, and what resource group dependencies there are.The cluster's configuration is determined by the application's requirements.Cluster administrator may need to indicate/confirm that the fixed component is approved for use.


Primary Node With a Standby Node

Multiple layers of backup nodes are possible--fallover policy determines which node .For example: primary -> secondary -> tertiary -> quaternary -> quinary -> senary -> septenary -> octonary -> nonary -> denary ...

node Vancouver fails, node Halifax takes no action

node Halifax fails, Vancouver takes over resource group A

Fallback policy = "Fallback to higher priority node"

Vancouver returns

Halifax Vancouver

Halifax Vancouver

Halifax Vancouver

Halifax Vancouver

Halifax Vancouver

Fallover policy = "Fallover to nextpriority node"

A

A

A

A

A

Halifax returns

Start policy = "Home node"


Minimizing Downtime

Downtime is minimized by avoiding fallbacks.Multiple resource groups tend to gather together on the node which has been up the longest.

Halifax fails, Vancouver takes over resources

Halifax joins the cluster

Halifax Vancouver

Halifax Vancouver

Halifax VancouverHalifax Vancouver

Fallback policy = "Never Fallback "

Fallover policy = "Fallover to next priority node"

A

AA

A

node Vancouver fails, node Halifax takes no action

Start policy = "Home node"


Two-Node Mutual Takeover Scenario

This is a very common HACMP cluster configuration.

Halifax fails, Vancouver takes over resources from Halifax

Halifax joins the cluster, Vancouver releases resource group A

Vancouver joins the cluster, Halifax releases resource group B

Vancouver fails, Halifax takes over resources from Vancouver

Halifax Vancouver

BA

Halifax Vancouver

BA

Halifax Vancouver

B A

Halifax Vancouver

BA

Fallback policy = "Fallback to higher priority node"

Start policy = "Home node"Fallover policy = "Fallover to nextpriority node"


Multiple Active Nodes

Application must be designed to run simultaneously onmultiple nodes.

This has the potential for essentially zero downtime.

A A ARegina VancouverHalifax

A A A

Halifax, Regina and Vancouver are all running Application A, each using a separate Service IP Address

If nodes fail, the application remains continuously available as long as there are surviving nodes to run on.

Fixed nodes resume running their copy of the application.

Regina VancouverHalifax


Fundamental HACMP ConceptsA cluster's topology is the cluster from a networking components perspective

A cluster's resources are the entities which are being made highly available (for example, volume groups, filesystems, service IP labels, applications)

A resource group is a collection of resources which HACMP strives to keep available as a single unit according to policies specified by the cluster designer / implementer

A given resource may only appear in at most one resource group

A startup policy determines which node the resource group is activated on

A fallover is the movement of a resource group to another node in response to a failure. A fallover policy controls the resource group's target node.

A fallback is the movement of a resource group to a more preferrednode, typically in response to the reintegration of the previouslyfailed node. A fallback policy determines when fallback occurs.

Customization is the process of augmenting HACMP, typically via implementing scripts which HACMP invokes at appropriate times


Points to PonderEach resource group must be serviced by at least two nodes.Each resource group can have different policies.Resource groups can be migrated (manually or automatically) to rebalance loads.Fallover policy (that is, node ranking) can be static or dynamic.Every cluster must have at least one IP network and one non-IP network.HACMP does not require that a cluster have any shared storage.Any combination of supported nodes may appear in a cluster*.Cluster may be split across two sites which may or may not require making a copy of the data (HACMP XD is required for data copy).HACMP/ES 4.5 and later can be customized to automatically restart failed applications.The applications must be capable of being restarted

automatically if high availability is to be achieved.

* Application performance requirements and other operational issues almost certainly impose practical constraints on the size and complexity

of a given cluster.


HACMP 5.2 LimitsHACMP 5.2 supports a maximum of:

64 resource groups per cluster32 nodes per cluster256 IP addresses known to HACMP (for example, service and boot IP labels)16 physical IP networks known to HACMP


Checkpoint1. True or False?

Resource groups may be manually moved from node to node.

2. True or False?Resources may be shared between resource groups.

3. Which of the following statements are true (select all that apply):a. A resource group always returns to the primary node when the primary node recovers from

a failure.b. All resource groups have a primary node.c. Except during fallovers and fallbacks, resources are always dedicated to a particular node

at any given time.d. The priority ordering of nodes within a resource group can be changed dynamically.

4. True of False?All nodes associated with a resource group must have roughly equivalent performance characteristics.


Checkpoint Answers1. True or False?

Resource groups may be manually moved from node to node.

2. True or False?Resources may be shared between resource groups.

3. Which of the following statements are true (select all that apply):a. FALSE A resource group always returns to the primary node when the primary node

recovers from a failure.b. FALSE All resource groups have a primary node.c. FALSE Except during fallovers and fallbacks, resources are always be dedicated to a

particular node at any given time.d. TRUE The priority ordering of nodes within a resource group can be changed dynamically.

4. True of False?All nodes associated with a resource group must have roughly equivalent performance characteristics.


HACMP Packaging and PositioningAfter completing this topic, you should be able to: Describe the packaging of the HACMP 4.x and 5.x family of products.Give examples of applications for which HACMP is not appropriate.Describe the evolution of the HACMP product.Describe where to find information on HACMP.


HACMP 4.x Product FamilyA family of products providing a full spectrum of high availability solutions:

GeoRM

RemoteMirroring

Concurrent Resource Manager

Concurrent Clusters

HACMP "classic"

Fault Resilience HAGEO

Global Availability

Fault Resilience

and 32 way

Scalability

HACMP ES

HACWS

High Availability

CWS


HACMP 4.x to 5.x Repackaging

(gone)

(combined)

(replaced by)

(still available)

(still available)

GeoRM

Concurrent Resource Manager

HACMP "classic"

HAGEO

HACMP ES

HACWS

HACMP forAIX

GeoRM

HACMP XD

HACWS

HAGEOPPRC


High Availability

CWSHACWS

HACMP 5.x Product FamilySimplified product packaging with an enhanced feature set:

Fault resilience,32-way scalabilityand concurrent

clusters

Remote MirroringGeoRM

Global AvailabilityHACMP

XD

HACMP forAIX


Things HACMP Does Not Do

Backup and restoration. Time synchronization.Application specific configuration.System Administration tasks unique to each node.

TSM


When Is HACMP Not the Correct Solution?HACMP can dramatically improve availability, although thereare situations where it may not be the appropriate solution:

Zero downtime required Maybe a fault tolerant system is the correct choice.7x24x365, HACMP occasionally needs to be shutdown for maintenance.Life Critical environments

HACMP is designed to handle one failure.A second failure could be catastrophic.

Security IssuesToo little security, lots of people with the ability to change the environment.Too much security, C2 and B1 environments may not allow HACMP to function as designed.

Unstable EnvironmentsHACMP cannot make an unstable and poorly managed environment stable. HACMP tends to reduce the availability of poorly managed systems:

Lack of change controlFailure to treat the cluster as a single entityToo many cooksUnskilled administratorsLack of documented operational procedures


HACMP's Evolution HACMP is a mature product evolving to meet customer's needs. Some of the key recent feature changes have been:

HACMP version 4.4.xIntegration with Tivoli, application monitoring, cascading with out fallback, C-SPOC enhancements, improved migration support, integration of HANFS functionality, soft copy documentation (html and pdf).

HACMP version 4.5Requires AIX 5L, Automated configuration discovery, Support for multiple Service labels on each Network Adapter through the use of IP aliasing, Persistent IP address support, 64-bit-capable APIs, Monitoring and recovery from loss of volume group quorum.

HACMP version 5.1Standard and extended configuration procedures, improved automatedconfiguration discovery, custom resource groups, heartbeating over disks,fast disk fallover, elimination of "HACMP classic".

HACMP version 5.2Only policy-based resource groups, two-node configuration assistant, auto correction with verify, auto verify, cluster test tool, enhanced online worksheets, cluster wide password change, application startup monitoring, multiple application monitors, improved security (key distribution), file collections, dependent resource groups, RMC replaces Event Manager, Web interface to the HACMP smit menus, lpar support for CUoD (PTF1).


Sources of HACMP InformationThe HACMP manuals that come with the product!

/usr/lpp/doc/release_notesIBM courses:

HACMP Administration I: Implementation (AU54)HACMP Administration II: Maintenance and Migration (AU57)HACMP Administration III: Problem Determination and Recovery (AU59)HACMP Administration IV: Master Class (AU56)Implementing High Availability on EServer Cluster 1600 (AU58)IBM HAGEO Implementation (AU52)

IBM Web Site:http://www-1.ibm.com/servers/aix/products/ibmsw/high_avail_network/hacmp.html

Non-IBM sources (not endorsed by IBM but probably worth a look):http://www.matilda.com/hacmp/http://groups.yahoo.com/group/hacmp/

BEFORE all else fails, read the manuals!



Support for the Concurrent Resource Management feature was dropped in HACMP 5.1.

2. True or False?HACMP XD is a complete solution for building geographically distributed clusters.

3. Which of the following capabilities does HACMP not provide (select all that apply):

a. Backup and restoration. b. Time synchronization.c. Automatic recovery from node and network adapter failure.d. System Administration tasks unique to each node.



Support for the Concurrent Resource Management feature was dropped in HACMP 5.x.

2. True or False?HACMP XD is a complete solution for building geographically distributed clusters.

3. Which of the following capabilities does HACMP not provide (select all that apply):

a. Backup and restoration.b. Time synchronization.c. Automatic recovery from node and network adapter failure.d. System Administration tasks unique to each node.


Unit Summary

Having completed this unit, you should be able to: Outline the features and benefits of HACMP for AIXDescribe the physical and logical components of an HACMP clusterUnderstand how HACMP operates in typical configurationsExplain the HACMP 5.x product family


Welcome to:

3.13.0.3

Shared Storage Considerations forShared Storage Considerations forHigh-AvailabilityHigh-Availability


Unit Objectives

After completing this unit, you should be able to: Understand the fundamental shared storage concepts as they apply within an HACMP clusterUnderstand the capabilities of various disk technologies as they related to HACMP clustersUnderstand the shared storage related facilities of AIX and how to use them in an HACMP cluster


Fundamental Shared Storage ConceptsAfter completing this topic, you should be able to: Understand the distinction between shared storage and private storageUnderstand how shared storage is used within an HACMP clusterUnderstand the importance of controlled access to an HACMP cluster's shared storageUnderstand how access to shared storage is controlled in an HACMP cluster


What Is Shared Storage?

SCSI

SSA

ESS


What Is Private Storage?

SCSI

SSA

ESS


Where Should The Data Go?

Operating system componentsalways on private storageDynamic application data

for example, databases, Web server contentalways on shared storage

Static application datavarious categories:

configuration files - usually on shared storage(easier to keep consistent)license keys - it depends

private storage if node lockedusually shared storage otherwise

truly static data - wherever it is convenientApplication binaries

it depends:avoid version mismatches by placing on shared storage

often easier to upgrade if they're on private storage


Shared Storage QuestionsSome questions to ask your user or customer:

For each highly available application in the cluster:How much shared storage is required?

Upon deployment of the cluster?In six months?In one year?In two years?

How is data organized?Files within file systems versus production database storage

Is the application:I/O bandwidth intensive?Random I/O intensive?

What's required to validate application data after a crash?How important is REALLY fast recovery from failures?How will it be backed up?How much private storage is required?

Usually not enough to be a concern


Access to Shared Data Must Be ControlledConsider:Data is placed in shared storage to facilitate access to the datafrom whichever node the application is running onThe application is typically running on only one node at a time*Updating the shared data from another node (that is, not the node that the application is running on) could result in data corruptionViewing the shared data from another node could yield an inconsistent view of the data

Therefore, only the node actually running the application should be able to access the data.


Who Owns the Storage?

varyonvg,varyoffvg used to control ownership in normal operations

varyonvg/varyoffvg uses one of the following mechanisms: Reserve or release-based shared storage protection

Used with non- enhanced concurrent volume groupsRSCT-based shared storage protection

Used with Enhanced Concurrent Volume Groups

ODMODM ODMODM

A B

C D


Reserve or Release-based Protection

Reserve or release-based shared storage protection relies on hardware support for disk reservation.Disks are physically reserved to a node when varied on.Disks are released when varied off.LVM is unable to varyon a volume group whose disks are reserved to another node.Not all shared storage systems support disk reservation.

varyonvg

varyonvg

ODMODM ODMODM

A B

C D


Reserve or Release Voluntary Disk Takeover

varyoffvg httpvgdbvgvaryonvg

ODMODM ODMODM

A B

C D

dbvgvaryonvg

varyonvg

ODMODM ODMODM

A B

C D

httpvg

dbvgvaryonvg

varyonvg

ODMODM ODMODM

A B

C D

httpvg

varyonvg httpvg


Reserve or Release Involuntary Disk Takeover

varyonvg

varyonvg

ODMODM ODMODM

A B

C D

varyonvg

varyonvg

ODMODM ODMODM

A B

C D


hdisk0hdisk1hdisk2hdisk3hdisk4hdisk5hdisk6hdisk7hdisk8hdisk9

Ghost Disks

varyonvg

varyonvg

ODMODM ODMODM

A B

C D


RSCT-based Shared Storage Protection

ODMODM ODMODM

active varyonpassive varyon

active varyon passive varyon

A B

C D

Requires Enhanced Concurrent Volume GroupIndependent of disk type


RSCT-based Voluntary Fast Disk Takeover

ODMODM ODMODM

active varyon

passive varyon

active varyon

passive varyondbvg

A B

C D

httpvg

2. Right node releases active varyon of httpvg

ODMODM ODMODM

passive varyon

active varyon

passive varyondbvg

A B

C D

httpvg

3. Left node obtains active varyon of httpvg

ODMODM ODMODM

active varyon

passive varyon

active varyon

passive varyondbvg

A B

C D

httpvg

1. A decision is made to move httpvg from the right node to the left


RSCT-based Involuntary Fast Disk Takeover

ODMODM ODMODM

active varyon

passive varyon

active varyon

passive varyondbvg

A B

C D

httpvg

ODMODM ODMODM

passive varyon

passive varyon

active varyon dbvg

A B

C D

httpvg

ODMODM ODMODM

dbvgpassive varyon

active varyon

passive varyon

active varyon

A B

C D

httpvg

1. Right node fails2. Left node realizes that right node has

failed

Active varyon state and passive varyon state are concepts which don't apply to failed nodes

3. Left node obtains active mode varyon of httpvg


Enabling RSCT-based Fast Disk TakeoverFast disk takeover is enabled automatically for a VolumeGroup if all of the following are true:

The cluster is running AIX 5.2 on all nodesHACMP 5.x is installed on all nodesThe volume group is an enhanced concurrent mode volume group*

*Enhanced concurrent mode volume groups will be discussed shortly


Fast Disk Takeover Additional DetailsFast disk takeover is faster than reserve or release-based disk takeover.Ghost disks do not occur when fast disk takeover is enabled.Since fast disk takeover is implemented by RSCT, it is independent of the disk technology supported by HACMP.

The gsclvmd subsystem which uses group services provides the protection. The distinction between active varyon and passive varyon is private to each node (that is, it isn't recorded anywhere on the shared disks).


Checkpoint1. Which of the following statements is true (select all that apply)?

a. Static application data should always reside on private storage.b. Dynamic application data should always reside on shared storage.c. Shared storage must always be simultaneously accessible to all cluster nodes.d. Regardless of the size of the cluster, all shared storage must always be accessible,

subject to access control, by all cluster nodes.

2. True or False?Using RSCT-based shared disk protection results in slower fallovers.

3. True or False?Ghost disks must be checked for and eliminated immediately after every cluster fallover or fallback.

4. True or False?The fast disk takeover facility is a risk free performance improvement in HACMP 5.1.


Checkpoint Answers1. Which of the following statements is true (select all that apply)?

a. Static application data should always reside on private storage.b. Dynamic application data should always reside on shared storage.c. Shared storage must always be simultaneously accessible to all cluster nodes.d. Regardless of the size of the cluster, all shared storage must always be accessible

(subject to access control) by all cluster nodes.

2. True or False?Using RSCT-based shared disk protection results in slower fallovers.

3. True or False?Ghost disks must be checked for and eliminated immediately after every cluster fallover or fallback.

4. True or False?The fast disk takeover facility is a risk free performance improvement in HACMP 5.1.


Shared Disk TechnologyAfter completing this topic, you should be able to: Understand the capabilities of various disk technologies in an HACMP environmentUnderstand the installation considerations of a selected disk technology when combined with HACMPUnderstand the issue of PVID consistency within an HACMP cluster


SCSI Technology and HACMPHACMP-related issues with SCSI disk architecture:SCSI buses require termination at each endIn HACMP environments the terminators have to be external to ensure that the bus is still terminated properly after a failed system unit has been removed.SCSI buses are ID-based. All devices must have a unique ID number.

The default for all SCSI adapters at initial power-on is ID 7.SCSI adapters on shared SCSI busses must be configured to not use ID 7 in order to ensure that there isn't an ID conflict when some other SCSI adapter powers on.

DiskDrive

SCSIModule

TSCSIController

Host System

Maximum 25m

DiskDrive

SCSIModule

DiskDrive

SCSIModule

DiskDrive

SCSIModule

SCSIController

Host System

T5 6

4 3 2 1


SCSI Continued Different SCSI bus types have different maximum cable lengths forthe buses (maximum is 25 meters for differential SCSI)Certain SCSI subsystems support hot swappable drives.SCSI cables are not hot pluggable (power must be turned off on all devices attached to the SCSI bus before a SCSI cable connection is made or severed).Clusters using shared SCSI disks often experience ghost disks.

For additional information see:IBM course AU20, AIX 5L System Administration IV: Storage Management.

http://www.ibm.com/redbooks


SSA Technology and HACMPSSA is an open standard disk technology and has the following characteristics which are relevant in HACMP environments:SSA is based on a loop technology which offers multiple data paths to disk.There are restrictions on the number of adapters and types of adapters on eachloop. For example:

SSA loops can support eight adapters per loop.Adapters used in RAID mode are limited to two per loop.

Shared SSA disks never appear as ghost disks.For additional information see:

IBM course AU20, AIX 5L System Administration IV: Storage Management.Redbook, Understanding SSA Subsystems in Your Environment, SG24-5750-00.http://www.storage.ibm.com/hardsoft/products/ssa/docs/index.html

SSA Adapter


Configuring SSA for Maximum Availability

SSAAdapter

A2

A1

B1

B2

SSAAdapter

A2

A1

B1

B2

SSAAdapter

A2

A1

B1

B2

SSAAdapter

A2

A1

B1

B2

7133

12 9

8

16

1

13

4

5

Left Node Right Node


SSA Adapters The capabilities of SSA adapters have improved over time:

Only 6215, 6219, 6225 and 6230 adapters support Target Mode SSA and RAID5.Only the 6230 adapter with 6235 Fast Write Cache Option feature code supports enabling the write cache with HACMP.Compatible adapters:

6214 + 6216 or 6217 + 6218 or 6219 + 6215 + 6225 + 6230 For more information and microcode updates: http://www.storage.ibm.com/hardsoft/products/ssa/Features and functionality of otherwise identical adapters and drives can vary depending upon the level of microcode installed on the devices so be careful!

FC Adapter Name Adapters/loop RAID/JBOD

RAID5 Cache

TMSSA

6214 SSA 4-port adapter (MCA) -/2 N N6216 Enhanced SSA 4-port adapter (MCA) -/8 N N6217 SSA 4-port RAID adapter (MCA) 1/1 N N6218 SSA 4-port adapter (PCI) 1/1 N N6219 SSA Multi-Initiator/RAID EL Adapter (MCA) 2/8 Not for HA Y6215 SSA Multi-Initiator/RAID EL Adapter (PCI) 2/8 Not for HA Y6225 Advanced SerialRAID Adapter (PCI) 2/8 Not for HA Y6230 Advanced SerialRaid Adapter Plus (PCI) 2/8 Not for HA Y6230 + 6235 Fast Write Cache Option 2/8 Yes for HA Y

Note: AIX 5.2 does not support the MCA 6214, 6216, 6217 and 6219 SSA adapters.


ESS TechnologyThe ESS is an example of smart storage devices. They provide highly available storage centrally managed by the storage manager. The inner workings of the storage device are masked from AIX. Basic implementations are transparent to HACMP.The optional HACMP XD add-on can be used to coordinate the fallover of ESS PPRC based remote data mirrors.

Disk Adapters

Physical Diskspartitioned intoLogical Volumes

HA HA HAHA HA HA HAHA HA HA HAHAHA HA HAHA

Cache

NVS

DA

Cache

NVS

DA

Switch

Cluster2Cluster1

SSA Loops

Host Adapters

Dedicated 4-way SMP Systems

Up to 32 Connection ports + online upgrades8,16 or 32 GB CacheNonvolatile backup of write data in cache64 Internal disk pathsAll disks can communicate at the same timeHot-swap disks with redundant spares +online upgradesRAID5 providing performance and availability

Full Duplication - No Single Point Of Failure


ESS ContinuedAdvanced features of the Storage unit may be supported by HACMP.

Subsystem Device Driver (SDD) is supported by HACMP with appropriate PTFs.

For additional information refer to:IBM course AU20, AIX 5L System Administration IV: Storage Management.Implementing the Enterprise Storage Server in Your Environment, SG24-5420-01.

IBM TotalStorage Enterprise Storage Server Model 800, SG24-6424-00.



Fibre Channel Technology Fibre channel is supported by AIX and HACMP:

The gigabit fibre channel adapters (FC6228 and FC6239) are supported by HACMP.The IBM fibre channel raid storage server is supported for HACMP configurations. The FAStT disk technology is supported with restrictions in AIX and HACMP.

For more information refer to the following Redbooks: Planning and Implementing an IBM SAN, SG24-6116-00Designing an IBM Storage Area Network, SG24-5758-00Implementing Fibre Channel Attachment on the ESS, SG24-6113-00



Fibre Channel Continued

An example of a redundant fabric fibre channel implementation:

FC-ALswitches

Fibre ChannelRaid StorageServer

HACMP node HACMP node


Physical Volumes IDs

# lspvhdisk0 000206238a9e74d7 rootvghdisk1 00020624ef3fafcc Nonehdisk2 00206983880a1580 Nonehdisk3 00206983880a1ed7 Nonehdisk4 00206983880a31a7 Nonehdisk5 0002051036e6bf76 codevg#

ODMODM

A B

C D


hdisk Inconsistency

Neither HACMP nor AIX are affected by having a physical disk known by different hdisk numbers on different systems.Humans are, unfortunately, more easily confused.

ODMODM ODMODM

A B

C D

# lspvhdisk0 000206238a9e74d7 rootvghdisk1 00020624ef3fafcc None Ahdisk2 00206983880a1580 None Bhdisk3 00206983880a1ed7 None C

hdisk4 00206983880a31a7 None D

# lspvhdisk0 000206238a9e74d7 rootvghdisk1 000206238beef264 rootvghdisk2 00206983880a1ed7 None Chdisk3 00206983880a31a7 None Dhdisk4 00020624ef3fafcc None Ahdisk5 00206983880a1580 None B


Removing hdisk Inconsistencies

The two systems will now have consistent hdisk PVIDs.

# lspvhdisk0 000206238a9e74d7 rootvghdisk1 000206238beef264 rootvghdisk2 00020624ef3fafcc None Ahdisk3 00206983880a1580 None Bhdisk4 00206983880a1ed7 None Chdisk5 00206983880a31a7 None D

ODMODM ODMODM

A B

C D

# rmdev -d -l hdisk1 ; rmdev -d -l hdisk2# rmdev -d -l hdisk3 ; rmdev -d -l hdisk4# mkdev -c disk -t 160mb -s scsi -p scsi0 -w 6,1 -d# cfgmgr# lspvhdisk0 000206238a9e74d7 rootvghdisk2 00020624ef3fafcc None Ahdisk3 00206983880a1580 None Bhdisk4 00206983880a1ed7 None Chdisk5 00206983880a31a7 None D

"Fake" hdisk1 will exist in a defined state and will not appear in lspv output (use lscfg to see hdisk1).


Checkpoint1. Which of the following disk technologies are supported by

HACMP?a. SCSI. b. SSA.c. FC-AL.d. All of the above.

2. True or False?SSA disk subsystems can support RAID5 (cache-enabled) with HACMP.

3. True or False?Compatibility must be checked when using different SSA adapters in the same loop.

4. True or False? SSA can be configured for no single point of failure.

5. True or False? hdisk numbers must map to the same PVIDs across an entire HACMP cluster.


Checkpoint Answers1. Which of the following disk technologies are supported by

HACMP?a. SCSI. b. SSA.c. FC-AL.d. All of the above.

2. True or False?SSA disk subsystems can support RAID5 (cache-enabled) with HACMP (although certain limitations apply).

3. True or False?Compatibility must be checked when using different SSA adapters in the same loop.

4. True or False? SSA can be configured for no single point of failure.

5. True or False? hdisk numbers must map to the same PVIDs across an entire HACMP cluster.


Shared Storage from the AIX PerspectiveAfter completing this topic, you should be able to: Understand how LVM aids cluster availabilityUnderstand the quorum issues associated with HACMPSet up LVM for maximum availabilityConfigure a new shared volume group, filesystem, and jfslog


Logical Volume ManagerLVM is one of the major enhancements that AIX brings totraditional UNIX disk management. LVM's capabilities are exploited by HACMP.LVM is responsible for managing logical disk storage. Physical volumes are organized into volume groups, are identified by a unique physical volume ID (PVID), and are mapped to logical hdisks.

VolumeGroup

PhysicalPartitions

PhysicalVolumes

LogicalVolume

LogicalPartitions

PVID

PVID

hdisk1

hdisk0


LVM Relationships LVM manages the components of the disk subsystem. Applicationstalk to the disks through LVM.

This example shows an application writing to a filesystem which has its LVs mirrored in a volume group physically residing on separate hdisks.

MirroredLogicalVolume

write to /filesystem

Application

LogicalPartitions

PhysicalPartitions

Volu

me

Gro

up

LVM


LVM Mirroring LVM mirroring has some key advantages over other types of mirroring:

Up to three-way mirroring of all logical volume types, including concurrent logical volumes, sysdumpdev, paging space, and raw logical volumes.Disk type and disk bus independence.Optional parameters for maximizing speed or reliability.Changes to most LVM parameters can be done while the affected components are in use. The splitlvcopy command can be used to perform online backups.

MirroredLogicalVolume

write to /filesystem

Application

LogicalPartitions

PhysicalPartitions

Volu

me

Gro

up

LVM


LVM Configuration Options (1 of 2)# smit mklv

Add a Logical Volume

Type or select values in entry fields.Press Enter AFTER making all desired changes.

[TOP] [Entry Fields] Logical volume NAME [sharedlv]* VOLUME GROUP name sharedvg* Number of LOGICAL PARTITIONS [100] # PHYSICAL VOLUME names [] + Logical volume TYPE [] POSITION on physical volume middle + RANGE of physical volumes minimum + MAXIMUM NUMBER of PHYSICAL VOLUMES [] # to use for allocation Number of COPIES of each logical 2 + partition Mirror Write Consistency? active + [MORE...12]

F1=Help F2=Refresh F3=Cancel F4=ListF5=Reset F6=Command F7=Edit F8=ImageF9=Shell F10=Exit Enter=Do


LVM Configuration Options (2 of 2) # smit mklv

Add a Logical Volume


[MORE...12] [Entry Fields] Allocate each logical partition copy yes + on a SEPARATE physical volume? RELOCATE the logical volume during yes + reorganization? Logical volume LABEL [] MAXIMUM NUMBER of LOGICAL PARTITIONS [512] # Enable BAD BLOCK relocation? yes + SCHEDULING POLICY for reading/writing sequential + logical partition copies Enable WRITE VERIFY? no + File containing ALLOCATION MAP [] Stripe Size? [Not Striped] +[BOTTOM]



Steps to Creating a Mirrored FilesystemThese are the steps to creating a properly mirrored filesystemfor HA environments:

Step Description Options

1 create shared volume group Name the VG something meaningful like shared_vg1

2 change the autovary on flag chvg -an shared_vg1

3 create a jfslog lv "sharedlvlog"

Type=jfslog, size=1pp, separate physical volumes = yes, scheduling = sequential, copies=2

4 initialize the jfslog logform /dev/sharedlvlog

5 create a data lv "sharedlv" type= jfs, size=??, separate physical volumes= yes, copies=2, scheduling = sequential, write verify = ??

6 create a filesystem on a previously created lv

pick the lv = sharedlv to create the file system on, automount = no, assign desired mount point

7 verify the log file is in use mount filesystem, lsvg -l shared_vg1 should show 1 lv type jfslog, 1 lp, 2pp.

Note: HACMP supports both JFS and the newer JFS2 filesystems. JFS2 filesystems are not supported for NFS export from an HACMP cluster unless an external log area is used. It would probably be best to always use an external JFS2 log logical volume as one never knows which filesystems will need to be NFS exported someday.


Split Off a Mirrorchfs can be used to remove a copy of an LVM mirror and make it available to be remounted somewhere else for backup.

# mklvcopy sharedlv 3# syncvg -l sharedlv

# chfs -a splitcopy=backuplv -a copy=3 \ /sharedlv

(splits off a new logical volume and reduces the original to 2 copies)

==

=


Adding a Shared Volume Group

#2unmountvaryoffvg

ODMVGDA ODM

#1mkvgchvg mklv (log)logformmklv (data) crfs

#3cfgmgrimportvgchvg

#4varyoffvg


Quorum Checking

VG status Quorum checking ENabled for volume group Quorum checking DISabled for volume group

Running >50% VGDA's >1 VGDAs

varyonvg >50% VGDA's100% VGDA's

or if MISSINGPV_VARYON=TRUE>50% VGDA's

AIX performs quorum checking on volume groups in order toensure that the volume group remains consistentThe quorum rules are intended to ensure that structural changes to the volume group (for example, adding or deleting a logical volume) are consistent across an arbitrary number of varyon-varyoff cyclesOverriding the quorum rules can be VERY DANGEROUS!


Disabling Quorum CheckingCluster designers or implementers are often tempted to disable quorum checking. Although often necessary/desirable, there are risks if quorum is disabled or if a volume group varyon is forced:

It may be possible for each side of a two-node cluster to have different parts of the same volume group vary'd online.

It is possible that volume group structural changes (for example, add or delete of a logical volume) made during the last varyon are unknown during the current varyon.

It is possible that volume group structural changes are made to one part of the volume group which are inconsistent with a different set of structural changes which are made to another part of the volume group.


Eliminating Quorum ProblemsThe following points help minimize the quorum problems:

Avoid volume groups with less than three disks.Generally not an issue with HA clusters.

Distribute hard disks across more than one bus. Use three adapters per node in SCSI.Use two adapters per node, per loop in SSA.Use VPATHs and two adapters with Fibre Channel.

Use different power sources.Connect each power supply in a drawer to a different power source.

Use RAID arrays or Enterprise Storage Solutions.


The Quorum BusterIn some conditions, loss of quorum may lead to an unplannedsystem downtime. The quorum buster can help eliminate this possibility.

sharedvg


HACMP 5.x's Forced Varyon FeatureHACMP 5.x provides a new approach to dealing with quorum issues:

Each resource group has a flag which can be set to cause HACMP to perform a careful forced varyon of the resource group's volume groups if necessaryIf normal varyonvg fails and this flag is set:

HACMP verifies that at least one complete copy of each logical volume is availableIf verification succeeds, HACMP forces the volume group online

This is not a complete and perfect solution to quorum issues:If the cluster is partitioned then the rest of the volume group might still be online on a node in the other partition.


Recommendations for Forced VaryonBefore enabling HACMP's forced varyon feature for a volume group or the HACMP_MIRROR_VARYON variable for the entire cluster, ensure that:

The resource group's volume groups are mirrored across disk enclosuresThe resource group's volume groups are set to super-strict allocationThere are redundant heartbeat networks between all nodes

Use disk heartbeating with at least one disk per enclosureAdministrative policies are in effect to prevent volume group structural changes when the cluster is running degraded (that is, failed over or with disks missing)


Enhanced Concurrent Volume GroupsIntroduced in AIX 5.1Supported for all HACMP-supported disk technologiesSupports JFS and JFS2 filesystems

May be mounted by at most one node at any given timeDefault type when using C-SPOC to create Concurrent Volume Groups in AIX 5.1Replaces old style SSA Concurrent Volume Groups

C-SPOC can not be used to create SSA Concurrent Volume Groups on AIX 5.2

C-SPOC can be used to convert SSA Concurrent Volume Groups to Enhanced Concurrent Volume GroupsRequired in order to use:

Heartbeat over disk for a non-ip network* fast disk takover

* Covered in the network unit


Active Varyon versus Passive VaryonActive Varyon (varyonvg -a, lsvg -o)

Behaves like normal varyon (listed with lsvg -o)Allows all of the usual operations like:

Operations on filesystems (for example, mounts and opening, reading or writing files)Execution of applications resident within the volume groupCreating, changing and deleting logical volumesSynchronizing volume groups

RSCT responsible for ensuring that only one node has VG actively varied on

Passive Varyon (varyonvg -p, lsvg <vg_name>)Volume group is available in a very limited read-only modeOnly certain operations allowed:

Reading volume group configuration information (for example, lsvg)Reading logical volume configuration information (for example, lslv)

Most operations are prohibited:Any operations on filesystems and logical volumes (for example, mounts, open, create, modify, delete, and so forth)Modifying,synchronizing the volume group's configurationAny operation which changes the contents or hardware state of the disks

HACMP uses the appropriate varyonvg commands with enhanced concurrent volume groups


lsvg <vg_name>ON ACTIVE NODE

ON PASSIVE NODE:

toronto # lsvg ecmvgVOLUME GROUP: ecmvg VG IDENTIFIER: 0009314700004c00000000fe2eaa2d6dVG STATE: active PP SIZE: 8 MBVG PERMISSION: passive-only TOTAL PPs: 537 (4296 MB) ... ... ...Concurrent: Enhanced-Capable Auto-Concurrent: Disabled

halifax # lsvg ecmvgVOLUME GROUP: ecmvg VG IDENTIFIER: 0009314700004c00000000fe2eaa2d6dVG STATE: active PP SIZE: 8 MBVG PERMISSION: read/write TOTAL PPs: 537 (4296 MB) ... ... ...Concurrent: Enhanced-Capable Auto-Concurrent: Disabled


LVM and HACMP ConsiderationsFollowing these simple guidelines helps keep theconfiguration easier to administer:

All LVM constructs must have unique names in the cluster.For example, httplv, httploglv, httpfs and httpvg.

Mirror or otherwise provide redundancy for critical logical volumes.Don't forget the jfslog.If it isn't worth mirroring then consider deleting it now rather than having to wait to lose the data when the wrong disk fails someday!Even data which is truly temporary is worth mirroring as it avoids an application crash when the wrong disk fails.RAID-5 and ESS-based storage are alternative ways to provide redundancy.

The VG major device numbers should be the same.Mandatory for clusters exporting NFS filesystems, but it is a good habit for any cluster.

Shared data on internal disks is a bad idea.

Focus on the elimination of single points of failure.



Lazy update keeps VGDA constructs in sync between cluster nodes.

2. Which of the following commands will bring a volume group online?a. getvtg <vgname>b. mountvg <vgname>c. attachvg <vgname>d. varyonvg <vgname>

3. True or False? Quorum should always be disabled on shared volume groups.

4. True or False?filesystem and logical volume attributes cannot be changed while the cluster is operational.

5. True or False?An enhanced concurrent volume group is required for the heartbeat over disk feature.


Having completed this topic, you should be able to: Understand the fundamental shared storage concepts as they apply within an HACMP clusterUnderstand the capabilities of various disk technologies as they related to HACMP clustersUnderstand the shared storage related facilities of AIX and how to use them in an HACMP cluster

Topic Summary


Welcome to:

3.13.0.3

Networking Considerations forNetworking Considerations forHigh-AvailabilityHigh-Availability


Unit Objectives

After completing this unit, you should be able to: Understand how HACMP uses networksDescribe the HACMP networking terminologyExplain and set up IP Address Takeover (IPAT)Configure an IP network for HACMPConfigure a non-IP networkExplain how client systems are likely to be affected by failure recoveryMinimize the impact of failure recovery on client systems


How HACMP Uses NetworksAfter completing this topic, you should be able to: Explain how HACMP uses networks to:

Provide client access to the clusterDetect failuresDiagnose failuresCommunicate with other nodes in the cluster

Explain why a non-IP network is an essential part of any HACMP clusterDescribe what a persistent node IP label is and what they are typically used forProvide an overview of IP Address Takeover


How Does HACMP Use Networks?HACMP uses networks to:

Provide clients with highly available access to the cluster's applicationsDetect and diagnose node, network and network interface card (NIC) failuresCommunicate with other HACMP daemons on other nodes in the cluster


Providing HA Client Access to the ClusterProviding clients with highly available access to the cluster's applications requires:

Multiple NICs per network per node(Possibly) multiple networks per nodeCareful network design and implementation all the way out to the client's systems


What HACMP Detects and DiagnosesRemember, HACMP only handles the following failures directly:

Network Interface Card (NIC) failureNode failureNetwork failure

en0 en1 en0 en1

bondar hudson


Failure Detection Requires MonitoringHACMP must monitor the cluster's components in order to detect failures of these components.Let's look at what monitoring HACMP does . . .

en0 en1 en0 en1

bondar hudson


Heartbeat PacketsHACMP sends heartbeat packets across networksHeartbeat packets are sent and received by every NICThis is sufficient to detect all NIC, node and network failuresHeartbeat packets are not acknowledged

en0 en1 en0 en1

bondar hudson


Failure Detection versus Failure DiagnosisFailure Detection is realizing that something is wrong

For example, realizing that packets have stopped flowing between bondar's en1 and hudson's en1

Failure Diagnosis is figuring out what is wrongFor example, figuring out that bondar's en1 NIC has failed

en0 en1 en0 en1

bondar hudson


Failure DiagnosisWhen a failure is detected, HACMP (RSCT topology services) uses specially crafted packet transmission patterns to determine (that is, diagnose) the actual failure by ruling out other alternativesExample:1. RSCT on bondar notices that heartbeat packets are no longer

arriving via en1 and notifies hudson (which has also noticed that heartbeat packets are no longer arriving via its en1)

2. RSCT on both nodes send diagnostic packets between various combinations of NICs (including out via one NIC and back in via another NIC on the same node)

3. The nodes soon realize that all packets involving bondar's en1 are vanishing but packets involving hudson's en1 are beingreceived

4. DIAGNOSIS: bondar's en1 has failed. ?


What If All Heartbeat Packets Stop?A node might notice that heartbeat packets are no longerarriving on any NIC.In the configuration below, it's impossible for either node to distinguish between failure of the network and failure of the other node.Each node concludes that the other node is down!

en0 en1 en0 en1

bondar hudson


All Clusters REQUIRE a Non-IP Network!Distinguishing between the failure of the other node and the failure of the network requires a second network.Distinguishing between failure of the other node's IP subsystem and the total failure of the other node requires a non-IP network.

Therefore, ALL CLUSTERS SHOULD HAVE A NON-IP NETWORK!!!

en0 en1 en0 en1

non-IP network

bondar hudson


An Important Implementation DetailHACMP must ensure that heartbeats are sent out via all NICs and know which NIC is used.If a node has multiple NICs on the same logical subnet then AIXcan rotate which NIC is used to send packets to the network.

Therefore, each NIC on each physical IP network on any given node must have an IP address on a different logical subnet.

en0192.168.1.1

en0192.168.1.2

en1192.168.2.1

en1192.168.2.2

non-IP network

bondar hudson


HACMP continues to monitor failed components in order todetect their recoveryRecovered components are reintegrated back into the clusterReintegration might trigger significant actions

For example, recovery of primary node will optionally trigger fallback of resource group to primary node

Failure Recovery

en0 en1 en0 en1

bondar hudson

en0 en1 en0 en1

bondar hudson


IP Address Takeover (IPAT)Each highly available application is likely to require its own IP address (called a service IP address)This service IP address would usually be placed in the application's resource group

HACMP would then be responsible for ensuring that the service IP address was available on the node currently responsible for the resource group

Resource

Group

Volume

Group

File

System

Application ServerSe

rvice

IP

labelNFS exports

NFS mounts


IPAT After a Node FailureIf the application's current node fails, HACMP moves the application's resource group to the other node.If IPAT is configured for the resource group then the application's service IP address is associated with the application's new node.

192.168.25.12

192.168.25.12


192.168.25.12

IPAT After a NIC FailureIf the NIC associated with the application's service IP address fails, HACMP moves the service IP address to another NIC.From HACMPs perspective, NIC failures include anything which prevent the NIC from sending and receiving packets (for example, a damaged or disconnected cable, a failed switch port, and so forth).

192.168.25.12


Checkpoint1. How does HACMP use networks (select all which apply)?

a. Provide client systems with highly available access to the cluster's applicationsb. Detect failuresc. Diagnose failuresd. Communicate between cluster nodese. Monitor network performance

2. True or False?HACMP detects the loss of volume group quorum.

3. True or False?Heartbeat packets must be acknowledged or a failure is assumed to have occurred.

4. True or False?Clusters are required to include a non-IP network.

5. True or False?Each NIC on each physical IP network on each node is required to have an IP address on a different logical subnet.


Checkpoint Answers1. How does HACMP use networks (select all which apply)?

a. Provide client systems with highly available access to the cluster's applicationsb. Detect failuresc. Diagnose failuresd. Communicate between cluster nodese. Monitor network performance

2. True or False?*HACMP detects the loss of volume group quorum.

3. True or False?Heartbeat packets must be acknowledged or a failure is assumed to have occurred.

4. True or False?Clusters are required to include a non-IP network.

5. True or False?Each NIC on each physical IP network on each node is required to have an IP address on a different logical subnet.

*HACMP responds to the loss of quorum but the loss is detected by the Logical Volume Manager.


HACMP Concepts and Configuration RulesAfter completing this topic, you should be able to: List the networking technologies supported by HACMPDescribe the purpose of public and private HACMP networksDescribe the topology components and their naming rulesDefine key networking related HACMP termsDescribe the basic HACMP network configuration rules


HACMP Networking SupportSupported IP networking technologies:

EthernetAll speedsNot the IEEE 802.3 frame type which uses et0, et1 ...

FDDIToken-RingATM and ATM LAN EmulationEtherchannelSP Switch

Supported non-IP network technologies:Heartbeat over Disks (diskhb)

Requires Enhanced Concurrent Volume Group and HACMP 5.xRS232/RS422 (rs232)Target Mode SSA (tmssa)


HACMP categorizes all networks:

IP:

ethernet, token ring, fiddi, atm, sp switch (hps)

non-IP:

rs232, tmssa, diskhb

Network Types


HACMP Topology ComponentsHACMP uses some unique terminology to describe the type and function of topology (as in, network) components under its control.

SerialPort

bondar

Serial Port

hudson

Net

wor

kIn

terfa

ceC

ard

IP Network

IP label

non-IP network

non IP - tmssa

non IP - rs232

Communication Interface

Communication Device

TCP/IP network - Internalnet

vancouver_service 192.168.5.2IP address

Net

wor

kIn

terfa

ceC

ard

Net

wor

kIn

terfa

ceC

ard

node name

Net

wor

kIn

terfa

ceC

ard

non IP - diskhb

non-IP networknon-IP network


IP label # netstat -i Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll lo0 16896 link#1 5338 0 5345 0 0 lo0 16896 127 loopback 5338 0 5345 0 0 lo0 16896 ::1 5338 0 5345 0 0 tr0 1500 link#2 0.4.ac.49.35.58 76884 0 61951 0 0 tr0 1500 192.168.1 vancouver_boot1 76884 0 61951 0 0 tr1 1492 link#3 0.4.ac.48.22.f4 476 0 451 13 0 tr1 1492 192.168.2 vancouver_boot2 476 0 451 13 0

On the Naming of NodesThere are several names a node can be known by, including the AIX hostname, the HACMP node name and IP label. These concepts should not be confused.

AIX hostname

# hostname gastown # uname -a AIX gastown 2 5 004FD0CD4C00

HACMP node name

# /usr/es/sbin/cluster/utlities/get_local_nodename vancouver


HACMP 5.x Network Terms (1 of 2)Communication Interface: A TCP/IP network interface usedby HACMP for communication with clients and/or with other cluster nodes. It's IP address is defined to the AIX odm (via smit chinet) and is assigned to the interface at AIX boot time.

Communication Device: A physical device representing one end of a point-to-point non-IP network connection, such as /dev/tty1, /dev/hdisk1 and /dev/tmssa1.

Communication Adapter: An X.25 adapter used to support a Highly Available Communication Link.


HACMP 5.x Network Terms (2 of 2)Service IP Label / Address: Address configured by HACMP to support client traffic. It is kept highly available by HACMP.

Configured on an interface by either replacement or by alias.

Non-service IP Label / Address: An IP label / address defined to HACMP for communication interfaces and is not used by HACMP for client traffic. Two types:

interface (stored in AIX odm) persistent (see below).

Service Interface: A communications interface configured with a Service IP Label / Address (either by alias or replacement).

Non-Service Interface: A communications interface not configured with a Service IP Label / Address. Used as a backup for a Service IP Label / Address.

Persistent IP label / Address: An IP label / address, defined as an alias to an interface IP Label / Address which stays on a single node and is kept available on that node by HACMP.


IP Network Configuration RulesNon-service interface IP labels must be on different logical subnets.

Should use the same subnet maskIf heartbeat over alias used then same subnet may be used

There must be at least one common IP subnet made up of non-service IP labels among the nodes in the resource group

For example, each node must have an interface IP address in the 192.168.5.0 IP subnet.

Suggestion: Minimize the number of different IP subnets that you use.


IP Network Configuration Examples

IP addresses on first node

IP addresses on second node

Is this configuration valid?Assume subnet mask of 255.255.255.0

192.168.5.1192.168.6.1

192.168.5.2192.168.5.3

No (the IP addresses on the second node are all in the same logical IP subnet)

192.168.5.1192.168.6.1

192.168.7.1192.168.8.1

No (there is no logical IP subnet which is shared by all the nodes)

192.168.5.1 192.168.5.2Sort-of (it might be possible to configure HACMP to use this configuration but there are not enough NICs on each node to avoid the NICs being single points of failure)

192.168.5.1192.168.6.1

192.168.5.2192.168.6.2 Yes

192.168.5.1192.168.6.1

192.168.5.2192.168.7.1

Yes (although fewer logical IP subnets are possible)

192.168.5.1192.168.6.1192.168.7.1192.168.8.1102.168.9.1

192.168.5.2192.168.6.2192.168.7.2

Yes (each node can have a different number of NICs)


Non-IP Network Configuration RulesNon-IP networks are strongly recommended in order to provide an alternate communication path between cluster nodes in the event of an IP network failure or IP subsystem failureWith more than two nodes you can configure the non-IP network topology using one of the following layouts:

Mesh: each node connected to all other nodes Star: one node is connected to all other nodesRing (or Loop): each node connected to its two adjacent neighbors


Persistent Node IP LabelsAn IP label associated with a particular nodeUseful for administrative purposes:

Provides always on IP address associated with a particular nodeAllows external monitoring tools (for example, Tivoli) and administrative scripts to reach a particular node

Assigned, via IP aliasing, after node synchronization, to a communications interface on the node HACMP will strive to keep the persistent node IP label available on that node -- never moved to another node. Maximum of one persistent node IP label per network per nodepersistent node IP labels must adhere to subnet rules:

Persistent node IP labels must not be in any non-service interface subnet



Clusters must always be configured with a private IP network for HACMP communication.

2. Which of the following are true statements about communication interfaces (select all that apply)?

a. Has an IP address assigned to it using the AIX TCP/IP smit screensb. Might have more than one IP address associated with itc. Sometimes but not always used to communicate with clientsd. Always used to communicate with clientse. Similar but not identical to the HACMP 4.x adapter concept

3. True or False? Persistent node IP labels are not supported for IPAT via IP replacement.

4. True or False?There are no exceptions to the rule that each NIC on each physical network on each node must have an IP address in a different subnet.



Clusters must always be configured with a private IP network for HACMP communication.

2. Which of the following are true statements about communication interfaces (select all that apply)?

a. Has an IP address assigned to it using the AIX TCP/IP smit screensb. Might have more than one IP address associated with itc. Sometimes but not always used to communicate with clientsd. Always used to communicate with clients*e. Similar but not identical to the HACMP 4.x adapter concept

3. True or False? Persistent node IP labels are not supported for IPAT via IP replacement.

4. True or False?**There are no exceptions to the rule that each NIC on each physical network on each node must have an IP address in a different subnet.

*Communication interfaces on private IP networks are not intended to be used by clients.**The HACMP 5.1 heartbeat over IP aliases feature can be used to "rescind" this rule.


After completing this topic, you should be able to: Describe IPAT via IP aliasing and IPAT via IP replacement including:

How to configure a network to support themWhat happens when

There are no failed componentsA communication interface failsA communication interface recoversA node failsA node recovers

Know how to select which style of IPAT is appropriate in a given contextDescribe how the AIX boot sequences changes when IPAT is configured in a clusterUnderstand the importance of consistent IP addressing and labeling conventions

Implementing IP Address Takeover (IPAT)


Two Ways to Implement IPATIPAT via IP Aliasing:

HACMP adds the service IP address to an (AIX) interface IP address using AIX's IP aliasing feature:

ifconfig en0 alias 192.168.1.2

IPAT via IP Replacement:HACMP replaces an (AIX) interface IP addresses with the service IP addresses:

ifconfig en0 192.168.1.2


192.168.10.1 (odm) 192.168.11.1 (odm) 192.168.10.2 (odm) 192.168.11.2 (odm)

Network: IPAT via IP Aliasing Define IP interface address in the AIX odm to each network interface.

Each interface IP address must be in a different logical IP subnet*Define these address in the /etc/host file and configure them in HACMP topology as communication interfaces

Define service addresses in /etc/hosts and in HACMP resourcesThey must not be in the same logical IP subnet as any of the interface IP addressesHACMP will configure them to AIX when needed

* Refer to earlier discussion of heartbeating and failure diagnosis for explanation of why

Before starting the application resource group


IPAT via IP Aliasing in OperationWhen the resource group comes up on a node, HACMP aliases the service IP label onto one of the node's available (that is, currently functional) interfaces (odm).

* See earlier discussion of heartbeating and failure diagnosis for explanation of why

192.168.10.1 (odm) 192.168.11.1 (odm) 192.168.10.2 (odm) 192.168.11.2 (odm)

192.168.5.1 (alias)

After starting the application resource group


IPAT via IP Aliasing After an Interface FailsIf the communication interface being used for the service IP label fails, HACMP aliases the service IP label onto one of the node's remaining available (for example, currently functional) non-service (odm) interfacesThe eventual recovery of the failed boot adapter makes it available again for future use


192.168.10.1 (odm) 192.168.11.1 (odm) 192.168.10.2 (odm) 192.168.11.2 (odm)

192.168.5.1 (alias)


IPAT via IP Aliasing After a Node FailsIf the resource group's node fails, HACMP moves the resource group to a new node and aliases the service IP label onto one of the new node's available (for example, currently functional) non-service (odm) communication interfaces


192.168.10.2 (odm) 192.168.11.2 (odm)

192.168.5.1 (alias)


IPAT via IP Aliasing SummaryConfigure each node's communication interfaces with IP addresses (each on a different subnet)Assign service IP labels to resource groups as appropriate

There is no limit on the number of resource groups with service IP labelsThere is no limit on the number of service IP labels per resource group

HACMP assigns service IP labels to communication interfaces (NICs) using IP aliases as appropriateIPAT via IP aliasing requires that hardware address takeover is not configuredIPAT via IP aliasing requires gratuitous arp support


Network: IPAT via IP Replacement Define each network interface IP addresses in the AIX odm.

Each interface IP address on a given node must be in a different logical IP subnet* and there must be a common subnet among the nodesDefine these address in the /etc/host file and configure them in HACMP topology

Define service IP addresses in /etc/hosts and HACMP resources The address must be in the SAME subnet as a common interface subnetHACMP configures them to AIX as required


192.168.10.1 (odm) 192.168.11.1 (odm) 192.168.10.2 (odm) 192.168.11.2 (odm)

Before starting the application resource group


IPAT via IP Replacement in OperationWhen the resource group comes up on a node, HACMP replaces an interface (odm) IP label with the service IP label

It replaces the interface IP label on the same subnet if the resource group is on its startup node or if the distribution fallover policy is used.It replaces an interface IP label on a different subnet otherwise

192.168.10.7 (service) 192.168.11.1 (odm) 192.168.10.2 (odm) 192.168.11.2 (odm)

After starting the application resource group


IPAT via IP Replacement After an I/F FailsIf the communication interface being used for the service IP label fails, HACMP swaps the service IP label with an interface (odm) IP label on one of the node's remaining available (that is, currently functional) communication interfacesThe IP labels remain swapped when the failed interface recovers

NIC A 192.168.11.1 (odm)

NIC B192.168.10.7 (service) 192.168.10.2 (odm) 192.168.11.2 (odm)


IPAT via IP Replacement After a Node FailsIf the resource group's node fails, HACMP moves the resource group to a new node and replaces an interface IP label with the service IP label

If the resource group is on its startup node or if the fallover policy is distributed, then replaces the interface (odm) IP label in the same subnet. Else it replaces an interface (odm) IP label in a different subnet or fail if there isn't an available interface

192.168.10.2 (odm) 192.168.10.7 (service)


IPAT via IP Replacement SummaryConfigure each node with up to eight communication interfaces (each on a different subnet)Assign service IP labels to resource groups as appropriate

Each node can be the most preferred node for at most one resource groupNo limit on number of service IP labels per resource group but each service IP label must be on a different physical network

HACMP replaces interface IP labels with service IP labels on the same subnet as the service IP label when the resource group is running on its most preferred node or if the fallover policy is distributedHACMP replaces interface IP labels with service IP labels on a different subnet from the service IP label when the resource group is moved to any other node IPAT via IP replacement supports hardware address takeover


IPAT Is OptionalAlthough practically all clusters use IPAT, it is an optionalfeatureIf you don't need/want IPAT, you must still decide between IPAT via IP aliasing style networking and IPAT via IP replacement style networking:

If IPAT via IP aliasing style:Define persistent node IP labels and configure clients to use them

If IPAT via IP replacement:Configure (using SMIT tcp configuration, that is, smit chinet) service IP labels and standby IP labels on communication interfacesConfigure clients to use the service IP labels to access cluster nodes or define persistent node IP labels for the clients to use


Changes to AIX Start Sequence The startup sequence of AIX networking is changed whenIPAT is enabled.

/etc/inittab/sbin/rc.boot cfgmgr /etc/rc.net cfgif

/etc/rc mount all

/etc/rc.tcpip daemons start

/etc/rc.nfs daemons start exportfs

/etc/inittab/sbin/rc.boot cfgmgr /etc/rc.net (modified for ipat) exit 0 /etc/rc mount all

/usr/sbin/cluster/etc/harc.net /etc/rc.net -boot cfgif

< HACMP startup > clstmgr event node_up node_up_local get_disk_vg_fs acquire_service_addr telinit -a /etc/rc.tcpip daemons start /etc/rc.nfs daemons start exportfs IPAT changes the init

sequence


init:2:initdefault:brc::sysinit:/sbin/rc.boot 3 >/dev/console 2>&1 # Phase 3 of system bootpowerfail::powerfail:/etc/rc.powerfail 2>&1 | alog -tboot > /dev/console # Power failure detection . . .srcmstr:2:respawn:/usr/sbin/srcmstr # System Resource Controllerharc:2:wait:/usr/es/sbin/cluster/etc/harc.net # HACMP for AIX network startuprctcpip:a:wait:/etc/rc.tcpip > /dev/console 2>&1 # Start TCP/IP daemonsrcnfs:a:wait:/etc/rc.nfs > /dev/console 2>&1 # Start NFS Daemonscron:2:respawn:/usr/sbin/cronpiobe:2:wait:/usr/lib/lpd/pio/etc/pioinit >/dev/null 2>&1 # pb cleanupcons:0123456789:respawn:/usr/sbin/getty /dev/consoleqdaemon:a:wait:/usr/bin/startsrc -sqdaemonwritesrv:a:wait:/usr/bin/startsrc -swritesrvuprintfd:23456789:respawn:/usr/sbin/uprintfd . . .ctrmc:2:once:/usr/bin/startsrc -s ctrmc > /dev/console 2>&1httpdlite:23456789:once:/usr/IMNSearch/httpdlite/httpdlite -r /etc/IMNSearch/httpdlite/httpdlite.conf & >/dev/console 2>&1clcomdES:2:once:startsrc -s clcomdES >/dev/console 2>&1clinit:a:wait:/bin/touch /usr/es/sbin/cluster/.telinit # HACMP for AIX These must be the last entries of run level a in inittab!pst_clinit:a:wait:/bin/echo Created /usr/es/sbin/cluster/.telinit > /dev/console # HACMP for AIX These must be the last entries of run level a in inittab!

Changes to /etc/inittab


Talk to Your Network AdministratorExplain how HACMP uses networksAsk for what you need:

IPAT via IP Aliasing:Service IP labels in the production network that the cluster will be attached toAdditional subnets for interface (odm) labels

One per network interface on the node with the most network adaptersThese do not need to be routable

IPAT via IP Replacement:Service labelInterface IP label for each network adapter (one must be in the same subnet as the service label)A different subnet for each interface

One per adapter on the node with the most adaptersOnly the subnet containing the service label need be routable

Persistent node IP label for each node on at least one network (very useful but optional)

Ask early (getting subnets assigned may take some time)


Adopt IP Address Numbering ConventionsHACMP cluster tend to have quite a few IP addressesassociated with themIf at all possible, adopt an IP address numbering conventionRequirements imposed by corporate IT policies or the network administrators may make it impractical to follow any sort of convention (do the best you can)


Adopt Labeling/Naming ConventionsHACMP cluster also tend to have quite a few IP labels andother names associated with themAdopt appropriate labeling and naming conventions:

For example: Node-resident labels should include the node's name

bondar-if1, bondar-if2, hudson-if1, hudson-if2Service IP labels that move between nodes should describe the application rather than the node

web1, infodb

Why?Conventions avoid mistakesAvoided mistakes improve availability!

hudson-if1


An IPAT via IP Aliasing ConventionHere's one possible IP label number convention for IPAT via IP aliasing networks:

IP address is of the form AA.BB.CC.DDAA.BB is assigned by network administratorCC indicates which interface, service IP label on each node:

15,16 indicates non-service/interface IP labels5 chosen for service labelsetc (as required)

DD indicates which node29 indicates an IP address on bondar31 indicates an IP address on hudson

Be flexible. For example, this convention uses DD=29 for bondar and DD=31 for hudson because the network administrator assigned bondar-if1 to be 192.168.15.29 and hudson-if1 to be 192.168.15.31. Fortunately, the network administrator could be convinced to use .29 and .31 for the other bondar andhudson interface IP addresses.

bondar-if1 192.168.15.29bondar-if2 192.168.16.29hudson-if1 192.168.15.31hudson-if2 192.168.16.31xweb 192.168.5.92yweb 192.168.5.70


An IPAT via IP Replacement ConventionHere's one possible IP label number convention for IPAT via IP replacement networks:

IP address is of the form AA.BB.CC.DDAA.BB is assigned by network administratorCC indicates which adapter on each node:

15,16 indicate non-service/interface IP labels (defined by network administrator)15 also chosen for service labelsAnd so forth (as required)

DD indicates which node29 indicates bondar31 indicates hudson

bondar-if1 192.168.15.29bondar-if2 192.168.16.29hudson-if1 192.168.15.31hudson-if2 192.168.16.31xweb 192.168.15.92yweb 192.168.15.70


The /etc/hosts file \All of the cluster's IP labels must be defined in each clusternode's /etc/hosts file:

IPAT via IP Aliasing Convention IPAT via IP Replacement Convention

127.0.0.1 loopback localhost

# cluster explorers# netmask 255.255.255.0

# bondar node192.168.15.29 bondar-if1192.168.16.29 bondar-if2

# hudson node192.168.15.31 hudson-if1192.168.16.31 hudson-if2

# persistent node IP labels192.168.5.29bondar192.168.5.31hudson

# Service IP labels192.168.5.92xweb192.168.5.70yweb

# test client node192.168.5.11 test

127.0.0.1 loopback localhost

# cluster explorers# netmask 255.255.255.0

# bondar node192.168.15.29 bondar-if1192.168.16.29 bondar-if2

# hudson node192.168.15.31 hudson-if1192.168.15.31 hudson-if2

# persistent node IP labels192.168.5.29bondar192.168.5.31hudson

# Service IP labels192.168.15.92 xweb192.168.15.70 yweb

# test client node192.168.15.11 test


Service IP Address Examples

IP addresses on first node

IP addresses on second node

Valid service IP addresses for IPAT via

IP aliasing

Valid service IP addresses for IPAT via IP replacement

192.168.5.1192.168.6.1

192.168.5.2192.168.6.2

192.168.7.1192.168.183.57198.161.22.1

192.168.5.3 and 192.168.5.97OR

192.168.6.3 and 192.168.6.97

192.168.5.1192.168.6.1

192.168.5.2192.168.7.1

192.168.8.1192.168.183.57198.161.22.1

192.168.5.3192.168.5.97

192.168.5.1192.168.6.14

192.168.5.98192.168.6.171

192.168.7.1192.168.183.57198.161.22.1

192.168.5.3 and 192.168.5.97OR

192.168.6.3 and 192.168.6.97

192.168.5.1192.168.6.1192.168.7.1192.168.8.1102.168.9.1

192.168.5.2192.168.6.2192.168.7.2

192.168.4.1192.168.10.1

192.168.183.57198.161.22.1

192.168.5.3 and 192.168.5.97OR

192.168.6.3 and 192.168.6.97OR

192.168.7.3 and 192.168.7.97


Common TCP/IP Configuration ProblemsSubnet masks are not consistent for all HA network adapters.Interface IP labels are placed on the same logical network.Service and interface IP labels are placed in the same logical networks in IPAT via IP aliasing networks.Service and interface IP labels are placed in different logical networks in IPAT via IP replacement networks.Ethernet frame type is set to 802.3. This includes etherchannel.Ethernet speed is not set uniformly or is set to autodetect.The contents of /etc/hosts is different on the cluster nodes.


Single IP Adapter NodesSingle IP Adapter nodes may appear attractive as they appearto reduce the cost of the clusterThe cost reduction is an illusion:1. A node with only a single adapter on a network is a node with a

single point of failure - the single adapter.2. Clusters with unnecessary single points of failure tend to suffer

more outages3. Unnecessary outages cost (potentially quite serious) moneyOne of the fundamental cluster design goals is to reduce unnecessary outages by avoiding single points of failure.HACMP requires at least two NICs per IP network for failure diagnosis.Clusters with less than two NICs per IP network are not supported*.

* Certain Cluster 1600 SP Switch-based clusters are supported with only one SP Switch adapter per network.



A single cluster can use both IPAT via IP aliasing and IPAT via IP replacement.

2. True or False?All networking technologies supported by HACMP support IPAT via IP aliasing.

3. True or False?All networking technologies supported by HACMP support IPAT via IP replacement.

4. If the left node has NICs with the IP addresses 192.168.20.1 and 192.168.21.1 and the right hand node has NICs with the IP addresses 192.168.20.2 and 192.168.21.2 then which of the following are valid service IP addresses if IPAT via IP aliasing is being used (select all that apply)?

a. (192.168.20.3 and 192.168.20.4) OR (192.168.21.3 and 192.168.21.4)b. 192.168.20.3 and 192.168.20.4 and 192.168.21.3 and 192.168.21.4c. 192.168.22.3 and 192.168.22.4d. 192.168.23.3 and 192.168.24.3

5. If the left node has NICs with the IP addresses 192.168.20.1 and 192.168.21.1 and the right hand node has NICs with the IP addresses 192.168.20.2 and 192.168.21.2 then which of the following are valid service IP addresses if IPAT via IP replacement is being used (select all that apply)?

a. (192.168.20.3 and 192.168.20.4) OR (192.168.21.3 and 192.168.21.4)b. 192.168.20.3, 192.168.20.4, 192.168.21.3 and 192.168.21.4c. 192.168.22.3 and 192.168.22.4d. 192.168.23.3 and 192.168.24.3


The Impact of IPAT on ClientsAfter completing this topic, you should be able to: Explain how user systems are affected by IPAT related operationsDescribe what the ARP cache issue isExplain how gratuitous ARP usually deals with the ARP cache issueExplain three ways to deal with the ARP cache issue if gratuitous ARP does not provide a satisfactory resolution to the ARP cache issue:

Configure clinfo on the client systemsConfigure clinfo within the clusterConfigure Hardware Address Takeover within the cluster


IP address moves/swaps within a node result in a short outageLong-term connection oriented sessions typically recover seamlessly (TCP layer deals with packet retransmission)

Resource group fallovers to a new node result in a longer outage and sever connection oriented services (long term connections must be reestablished, short term connections retried)In either case:

Short-lived TCP-based services like http and SQL queries experience short server down outageUDP-based services must deal with lost packets

How Are Users Affected?


What About the Users's Computers?An IPAT operation renders ARP cache entries on clientsystems obsoleteClient systems must (somehow) update their ARP caches

192.168.10.1 (odm) 192.168.11.1 (odm)00:04:ac:62:72:49 00:04:ac:48:22:f4

192.168.5.1 (alias) xweb

xweb (192.168.5.1) 00:04:ac:48:22:f4

xweb (192.168.5.1) 00:04:ac:62:72:49

192.168.10.1 (odm) 192.168.11.1 (odm)xweb 192.168.5.1 (alias)

00:04:ac:62:72:49 00:04:ac:48:22:f4

xweb (192.168.5.1) 00:04:ac:62:72:49xweb (192.168.5.1) 00:04:ac:62:72:49

xweb (192.168.5.1) 00:04:ac:48:22:f4

xweb (192.168.5.1) 00:04:ac:62:72:49

this space over here in this corner intentionally left blank


Local or Remote Client? If the client is remotely connected through a router, it is therouter's ARP cache which must be corrected.

192.168.10.1 (odm) 192.168.11.1 (odm)xweb 192.168.5.1 (alias)

00:04:ac:62:72:49 00:04:ac:48:22:f4

ARP:router (192.168.8.1) 00:04:ac:42:9c:e2

ARP:xweb (192.168.5.1) 00:04:ac:62:72:49client (192.168.8.3) 00:04:ac:27:18:09

192.168.8.100:04:ac:42:9c:e2

192.168.5.9900:04:ac:29:31:37

192.168.8.300:04:ac:27:18:09

192.168.10.1 (odm) 192.168.11.1 (odm)00:04:ac:62:72:49 00:04:ac:48:22:f4

192.168.5.1 (alias) xweb

ARP:router (192.168.8.1) 00:04:ac:42:9c:e2

ARP:xweb (192.168.5.1) ???client (192.168.8.3) 00:04:ac:27:18:09

192.168.8.100:04:ac:42:9c:e2

192.168.8.9900:04:ac:29:31:37

192.168.8.300:04:ac:27:18:09


Gratuitous ARPAIX 5L supports a feature called gratuitous ARP

AIX sends out a gratuitous (that is, unrequested) ARP update whenever an IP address is set or changed on a NIC

Other systems on the local physical network are expected to update their ARP caches when they receive the gratuitous ARP packetRemember: only systems on the cluster's local physical network must respect the gratuitous ARP packetSo arp update problems have been minimizedRequired if using IPAT via aliasing


Gratuitous ARP Support IssuesGratuitous ARP is supported by AIX on the following network technologies:

Ethernet (all types and speeds)Token-RingFDDISP Switch 1 and SP Switch 2

Gratuitous ARP is not supported on ATMOperating systems are not required to support gratuitous ARP packets

Practically every operating system does support gratuitous ARPSome systems (for example, certain routers) can be configured to respect or ignore gratuitous ARP packets


What if Gratuitous ARP is Not Supported?If the local network technology doesn't support gratuitous ARPor there is a client system on the local physical network whichmust communicate with the cluster and which does not support gratuitous ARP packets:

Clinfo can used on the client to receive updates of changes.Clinfo can be used on the servers to ping a list of clients, forcing an update to their ARP caches.HACMP can be configured to perform Hardware Address Takeover (HWAT).

Suggestion: Do not get involved with using either clinfo or HWAT to deal with ARP cache issues until you've verified that there actually are ARP issues which need to be dealt with.


Option 1: clinfo on the ClientThe cluster information daemon (clinfo) provides a facility to automatically flush the ARP cache on a client system.

In this option, clinfo must execute on the client platformclinfo executables are supplied for AIXclinfo source code is provided with HACMP to facilitate porting clinfo to other platforms

clinfo uses SNMP for communications with HACMP nodes/usr/es/sbin/cluster/etc/clhosts on the client system must contain a list of persistent node IP labels (one for each cluster node)clinfo.rc is invoked to flush the local arp cache

192.168.10.1 (boot) 192.168.11.1 (boot)00:04:ac:62:72:49 00:04:ac:48:22:f4

192.168.5.1 (alias) xweb

clstrmgr

clsmuxpd clinfo.rc

clinfosnmpd


Option 2: clinfo From Within the ClusterClinfo may also be used on the cluster's nodes to force an ARP cache update.

In this option, clinfo runs on every cluster nodeIf clinfo is only run on one cluster node then that node become a single point of failure!

clinfo flushes local ARP cache (on the cluster node) then pings a defined list of clients listed in /usr/es/sbin/cluster/etc/clinfo.rcClients pick up the new IP address to hardware address relationship as a result of the ping request

192.168.10.1 (boot) 192.168.11.1 (boot)00:04:ac:62:72:49 00:04:ac:48:22:f4

192.168.5.1 (alias) xweb

clstrmgr

clsmuxpd

snmpd

clinfo.rc

clinfo

ping!


clinfo.rc script (extract)This script is located under /usr/es/sbin/cluster/etc and is present on anAIX system if the cluster.client fileset has been installed.

A separate file /etc/cluster/ping_client_list can also contain a list of client machines to ping.

# Example: # # PING_CLIENT_LIST="host_a host_b 1.1.1.3" # PING_CLIENT_LIST=""

TOTAL_CLIENT_LIST="${PING_CLIENT_LIST}"

if [[ -s /etc/cluster/ping_client_list ]] ; then # # The file "/etc/ping_client_list" should contain only a line # setting the variable "PING_CLIENT_LIST" in the form given # in the example above. This allows the client list to be # kept in a file that is not altered when maintenance is # applied to clinfo.rc. # . /etc/cluster/ping_client_list

TOTAL_CLIENT_LIST="${TOTAL_CLIENT_LIST} ${PING_CLIENT_LIST}" fi

# # WARNING!!! For this shell script to work properly, ALL entries in # the TOTAL_CLIENT_LIST must resolve properly to IP addresses or hostnames # (must be found in /etc/hosts, DNS, or NIS). This is crucial.


Option 3: Hardware Address TakeoverHACMP can be configured to swap a service IP label'shardware address between network adapters.HWAT is incompatible with IPAT via IP aliasing because each service IP address must have its own hardware address and a NIC can support only one hardware address at any given time.Cluster implementer designates a Locally Administered Address (LAA) which HACMP assigns to the NIC which has the service IP label

xweb 192.168.5.1 (service)40:04:ac:62:72:49 00:04:ac:48:22:f4

192.168.20.1 (odm) 192.168.20.1 (odm)00:04:ac:62:2e:4c 40:04:ac:62:72:49

192.168.5.1 (service) xweb

xweb (192.168.5.1) 40:04:ac:62:72:49 xweb (192.168.5.1) 40:04:ac:62:72:49


Hardware Address Takeover (1 of 2)HACMP can move LAAs between nodes in conjunction with anIPAT fallover.

The hardware address is "moved" with the service IP

address

xweb192.168.5.1255.255.255.0

40:04:ac:62:72:49

hudsonboot192.168.5.2255.255.255.0

00:04:ac:48:22:f6

xweb192.168.5.1255.255.255.0

40:04:ac:62:72:49

bondarstandby192.168.9.1255.255.255.0

00:04:ac:48:22:f4

HudsonBondar


Hardware Address Takeover (2 of 2)

Bondar

xweb192.168.5.1

255.255.255.040:04:ac:62:72:49

hudsonboot192.168.5.2

255.255.255.000:04:ac:48:22:f6

Hudson

bondarstandby192.168.9.1255.255.255.0

00:04:ac:48:22:f4tr1

bondarboot192.168.5.3255.255.255.0

00:04:ac:48:22:f4

tr1

tr0 tr0

When a failed node comes back to life, the burned in ROM Address is used on the service network

adapter.

hudsonstandby192.168.9.2255.255.255.0

00:04:ac:62:72:61

hudsonboot192.168.5.2255.255.255.0

00:04:ac:48:22:f6

bondarstandby192.168.9.1255.255.255.0

00:04:ac:48:22:f4tr1

xweb192.168.5.1255.255.255.0

40:04:ac:62:72:49

tr1

tr0 tr0

After HACMP is started the node

reintegrates according to its resource group

parameters

Bondar Hudson


Selecting the LAAThe LAA must be unique on the cluster's physical networkThe MAC address based technologies (ethernet, tokenring and FDDI) use six byte hardware addresses of the form:xx.xx.xx.xx.xx.xx

The factory-set MAC address of the NIC will start with 0, 1, 2 or 3A MAC address that starts with 0, 1, 2 or 3 is called a Globally Administered Address because it is assigned to the NIC's vendor by a central authority

Incrementing this first digit by 4 transforms the GAA into a LAA which will be unique worldwide (unless someone has already used the same GAA to create an LAA which isn't likely since GAAs are unique worldwide)


Selecting the LAA ExampleUse netstat -in to get the GAAs of the NICs from one of yourcluster nodes:# netstat -inName Mtu Network Address Ipkts Ierrs Opkts Oerrs Collen0 1500 link#2 0.60.94.9.36.6b 40032 0 22471 0 0en0 1500 192.168.15 192.168.15.31 40032 0 22471 0 0en1 1500 link#3 2.7.1.20.a3.73 811733 0 413441 0 0en1 1500 192.168.16 192.168.16.31 811733 0 413441 0 0

Make sure each number is two digits long by prepending 0s as necessary:02.07.01.20.a3.73

Verify that the first digit is 0, 1, 2 or 3 (if it isn't then pick a different NIC's hardware address and start over)Add 4 to the first digit:42.07.01.20.a3.73

Use this as the LAA for one of your service IP addresses(repeat using a different vendor-assigned GAA for eachservice IP address that needs an LAA)


Do not enable the ALTERNATE hardware address field in smit devices.

Causes the adapter to boot on your chosen LAA rather than the burned in ROM address.Causes serious communications problems and puts the cluster in to an unstable state.Correct method is to enter your chosen LAA in to the smit HACMP menus (remove the periods or colons before entering it into the field).

The Token-Ring documentation states that the LAA must start with 42The FDDI documentation states that the first octet (digit) of the first byte of the LAA must be 4, 5, 6 or 7 (which is compatible with the method for creating LAAs described earlier)Token-Ring adapters do not release the LAA if AIX crashes.

AIX must be set to reboot automatically after a system crash (see smit chgsys).

Hardware Address Takeover Issues



Clients are required to exit and restart their application after a cluster outage.

2. True or False?All client systems are potentially directly affected by the ARP cache issue.

3. True or False?clinfo must not be run both on the cluster nodes and on the client systems.

4. Use the LAA generation technique described earlier to generate an LAA for each of the following GAA addresses (all but one of these are taken from real ethernet cards):00.20.ed.76.fb.15 ____________________0.4.ac.17.19.64 ____________________0.6.29.ac.46.8 ____________________12.7.1.71.1.6 ____________________



Clients are required to exit and restart their application after a cluster outage.

2. True or False?All client systems are potentially directly affected by the ARP cache issue.

3. True or False?clinfo must not be run both on the cluster nodes and on the client systems.

4. Use the LAA generation technique described earlier to generate an LAA for each of the following GAA addresses (all but the last one of these are taken from real ethernet cards):00.20.ed.76.fb.15 40.20.ed.76.fb.150.4.ac.17.19.64 40.04.ac.17.19.640.6.29.ac.46.8 40.06.29.ac.46.0812.7.1.71.1.6 52.07.01.71.01.06


Unit Summary

Having completed this unit, you should be able to: Understand how HACMP uses networksDescribe the HACMP networking terminologyExplain and set up IP Address Takeover (IPAT)Configure an IP network for HACMPConfigure a non-IP networkExplain how client systems are likely to be affected by failure recoveryMinimize the impact of failure recovery on client systems


Welcome to:

3.13.0.3

HACMP ArchitectureHACMP Architecture


Unit Objectives

After completing this unit, you should be able to: Describe the installation process for HACMP 5.2List and explain the purpose of the major HACMP 5.2 componentsDescribe the HACMP events conceptDescribe the resource group definition process in HACMP 5.2


Preparing for Cluster ConfigurationAfter completing this topic, you should be able to: Outline the steps required to implement a clusterKnow how to install HACMP 5.2Be aware of the prerequisites for HACMP 5.2Know when you are ready to start actual cluster configuration


Steps for Successful ImplementationHACMP should not be installed upon a system which is in production.

Step Step Description Comments1 Plan Use planning worksheets and documentation.2 Assemble hardware Install adapters, connect shared disk and network.3 Install AIX Ensure you update to the latest maintenance level.4 Configure networks Requires detailed planning.5 Configure shared storage Set up shared volume groups and filesystems.6 Install HACMP Install on all nodes in the cluster (don't forget to install latest fixes).7 Reboot each node Required after installing or patching HACMP.8 Define/discover the cluster topology Review what you end up with to make sure that it is what you expected.9 Configure application servers You will need to write your start and stop scripts.10 Configure cluster resources Refer to your planning worksheets.11 Synchronize the cluster Ensure you "actually" do this.12 Start HACMP Watch the console for messages.13 See comment Skip this step if you are superstitious.14 Test the cluster Document your tests and results.


Where Are We in the Implementation?Plan for network, storage, and applicationEliminate single points of failure

Define and configure the AIX environment Storage (adapters, LVM volume group, filesystem)Networks (ip interfaces, /etc/hosts, non-ip networks and devices)Application start and stop scripts

Install the HACMP filesets and rebootConfigure the HACMP environment

TopologyCluster, node names, HACMP IP and non-IP networks





Before All Else Fails . . .

Study the appropriate HACMP manuals:

HACMP for AIX Planning and Installation Guide SC23-4861Contains Planning Worksheets in Appendix A

Release notes: /usr/lpp/cluster/docs


What Is In the Box? HACMP 5.2 (product number 5765-F62)HACMP 5.1 (product number 5765-F62)

Only one variant which is roughly equivalent to HACMP/ES CRM in HACMP 4.5 terminologyMarketing ends: 31 Mar 05, Service ends: 1 Sep 06

HACMP 4.5 (product number 5765-E54)Is available as four distinct variants, each of which comes on its own CD-ROM:

HACMP classicHACMP classic CRMHACMP/ESHACMP/ES CRM

Make sure that you've ordered the correct variant and that you've got the correct variant!Marketing ends: 31 Dec 03, Service ends: 1 Sep 05


Install the HACMP Filesets (1 of 2)Here's a listing of the HACMP 5.2 CD:

bos.clvm.enh.5.2.0.31.U

bos.rte.lvm.5.2.0.31.U

cluster.adt.es.5.2.0.0.I

cluster.doc.en_US.es.5.2.0.0.I

cluster.es.5.2.0.0.I

cluster.es.cfs.5.2.0.0.I

cluster.es.clvm.5.2.0.0.I

cluster.es.cspoc.5.2.0.0.I

cluster.es.plugins.5.2.0.0.I

cluster.es.worksheets.5.2.0.0.I

cluster.hativoli.5.2.0.0.I (requires Tivoli)

cluster.haview.4.5.0.0.I (requires Netview)

cluster.license.5.2.0.0.I

cluster.man.en_US.es.5.2.0.0.I

cluster.msg.en_US.cspoc.5.2.0.0.I

cluster.msg.en_US.es.5.2.0.0.I

cluster.msg.en_US.hativoli.5.2.0.0.I (requires Tivoli)

cluster.msg.en_US.haview.4.5.0.0.I (requires Netview)

Your requirements may vary!

rsct.basic.hacmp.2.3.3.0.U

rsct.basic.hacmp.2.3.3.1.U

rsct.basic.rte.2.3.3.0.U

rsct.basic.rte.2.3.3.1.U

rsct.basic.sp.2.3.3.0.U

rsct.compat.basic.hacmp.2.3.3.0.U

rsct.compat.basic.rte.2.3.3.0.U

rsct.compat.basic.sp.2.3.3.0.U

rsct.compat.clients.hacmp.2.3.3.0.U

rsct.compat.clients.rte.2.3.3.0.U

rsct.compat.clients.sp.2.3.3.0.U

rsct.core.auditrm.2.3.3.0.U

rsct.core.errm.2.3.3.0.U

rsct.core.errm.2.3.3.1.U

ALSO cluster.msg.En*, Ja*, ja*


Install the HACMP Filesets (2 of 2)cluster.es:5.2.0.0 ES Base Server Runtime cluster.es.server.rte 5.2.0.0

5.2.0.0 ES Client Libraries cluster.es.client.lib 5.2.0.0

5.2.0.0 ES Client Runtime cluster.es.client.rte 5.2.0.0

5.2.0.0 ES Client Utilities cluster.es.client.utils 5.2.0.0

5.2.0.0 ES Cluster Test Tool cluster.es.server.testtool 5.2.0.0

5.2.0.0 ES Server Diags cluster.es.server.diag 5.2.0.0

5.2.0.0 ES Server Events cluster.es.server.events 5.2.0.0

5.2.0.0 ES Server Utilities cluster.es.server.utils 5.2.0.0

5.2.0.0 ES Two-Node Configuration Assistant cluster.es.server.cfgast 5.2.0.0

5.2.0.0 Web based Smit cluster.es.server.wsm 5.2.0.0


Don't Forget the PrerequisitesInstall the correct version of AIX:

AIX 5L 5.1 ML5 with RSCT 2.2.1.36 or higherAIX 5L 5.2 ML3 with RSCT 2.3.3.1 or higherAIX 5.3

Install the correct level of PSSP (on a PSSP managed cluster)Install the otherwise optional AIX filesets:bos.adt.lib bos.adt.libm bos.adt.syscallsbos.net.tcp.clients bos.net.tcp.server bos.rte.SRCbos.rte.libc bos.rte.libcfg bos.rte.libcurbos.rte.libpthreads bos.rte.odm

Other prerequisites may exist depending on your plans:Clusters with (enhanced) concurrent volume groups require bos.rte.lvm 5.1.0.25 or higherFast disk takeover requires AIX 5.2 and bos.clvm.enh 5.2.0.11 or higher


Verify That You Have the Required APARsPTF1The following APARs are required as noted:

APAR IY42782 (AIX 5.1)APAR IY55542 (snmp v3)APAR IY55105, IY55648 (interoperablility with 4.5, 5.1)APAR IY55069 (AIX 5.1) or IY55594 (AIX 5.2)

Enhanced concurrent volume group supportAPAR IY60340 (AIX 5.3)

The above lists are almost certainly out of date. Check

http://www-1.ibm.com/servers/eserver/support/pseries/fixes/

for the latest APARs.


Some Final Things to CheckThe same versions of AIX and HACMP (including patch levels) are installed on allnodes.

Each node must be rebooted once HACMP has been installed.Correct filesets and prerequisites have been installed.Documentation is installed and accessible through a Web browser or Adobe's Acrobat Reader (acroread on UNIX systems) without requiring a cluster node to be running.

That the /etc/hosts file is configured on all nodes and contains all IP labels for all nodes.

Name resolution works for all IP labels on all nodes.Use the host command to test this.Ensure name to IP address translation is the same for all IP labels on all nodes.

IP and non-IP networks are configured.The subnet mask must be identical on all interfaces known to the cluster.All interfaces on each node are configured to be in different subnets See earlier IPAT discussion for more subnet requirements.Check that a route exists to all logical networks from all nodes.Check that nodes can ping between all interfaces on the same logical subnet.

Shared storage is configured and recognized the same from all nodes (some folks disagree).You have a written plan describing what you will configure and how you will test it!


Checkpoint1. What is the first step in implementing a cluster?

a. Order the hardwareb. Plan the clusterc. Install AIX and HACMPd. Install the applicationse. Take a long nap

2. True or False?HACMP 5.2 is compatible with any version of AIX 5.x.

3. True or False?Each cluster node must be rebooted after the HACMP software is installed.

4. True or False?You should take careful notes while you install and configure HACMP so that you know what to test when you are done.


Checkpoint Answers1. What is the first step in implementing a cluster?*

a. Order the hardwareb. Plan the clusterc. Install AIX and HACMPd. Install the applicationse. Take a long nap

2. True or False?HACMP 5.2 is compatible with any version of AIX 5.x.

3. True or False?Each cluster node must be rebooted after the HACMP software is installed.

4. True or False?You should take careful notes while you install and configure HACMP so that you know what to test when you are done.

*There is some dispute about whether the correct answer is b or e although a disconcerting number of clusters are implemented in the order a, b, c, d, e (how can you possibly order the hardware if you do not yet know what you are going to build?) or even just a, c, d (cluster implementers who skip step b rarely have time for long naps).


HACMP 5.2 ComponentsAfter completing this topic, you should be able to: Describe the major components of HACMP 5.2Describe the role played by RSCTDescribe how heartbeat rings are organized


The Layered LookHere are the layers of software on an HACMP 5.2 cluster node:

Application LayerContains the highly available applications that use HACMP services

HACMP LayerProvides highly available services to applications

RSCT, RMC LayerProvides monitoring, event management and coordination of subsystems for HACMP clusters

AIX LayerProvides operating system services

LVM LayerManages disk space at the logical level

TCP/IP LayerManages communication at the logical level


HACMP Components and FeaturesThe HACMP software has the following components:

Cluster Manager (+)Cluster Secure Communication Subsystem (*)IBM RS/6000 Reliable Scalable Cluster Technology Availability Services (RSCT)RMC (%)Cluster SMUX Peer and SNMP Monitoring Programs (+)Cluster Information Program (+)Highly Available NFS Server (+)Shared External Disk Access (+)Concurrent Resource Manager (**+)Cluster Lock Manager removed %

% new in HACMP 5.2*new in HACMP 5.1**optional in HACMP 4.x+common to HACMP/ES 4.5, HACMP "classic" 4.5 and HACMP 5.1


Cluster ManagerA daemon which runs on each cluster nodePrimarily responsible for responding to unplanned events:

Recovering from software and hardware failuresResponding to user-initiated events:

Request to online/offline a nodeRequest to move/online/offline a resource groupAnd so forth

A client to RSCTKeeps Cluster SMUX Peer daemon aware of cluster statusImplemented by the subsystem clstrmgrES


Cluster Secure Communication SubsystemIntroduced in HACMP 5.1.Provides a single, common communication infrastructure for all HACMP related communication between nodes.HACMP 5.2 provides two authentication security options:

Connection Authentication Standard

Default security level.Implemented directly by cluster communication daemon (clcomd).Uses /usr/es/sbin/cluster/rhosts file to determine legitimate partners.

Kerberos (SP only)Kerberos used with systems managed by PSSP (that is, SP).

Virtual Private Networks (VPN) using persistent labels.VPNs are configured within AIX.HACMP is then configured to use VPNs for all inter-node communication.

Message Authentication and/or Message EncryptionHACMP provides methods for key distribution.

Implemented using a new subsystem -- clcomdES


Cluster Communication Daemon (clcomd)Completely replaces various adhoc inter-node communication mechanisms used by earlier versions of HACMP (for example, rsh).

/.rhosts files are no longer required.Caches coherent copies of other nodes' ODMs.Maintains long term socket connections (avoids repeated socket startup/shutdown overheads).Implements the principle of least privilege:

Provides other nodes with just as much privilege as required.Nodes no longer have or require root access on each other.

Started via /etc/inittab entry.Managed by the SRC (startsrc, stopsrc, refresh all work).


clcomd Standard Connection AuthenticationSource IP addresses for each incoming session are checkedagainst:

/usr/es/sbin/cluster/etc/rhostsHACMP adapter ODMHACMP node ODM

General Approach:Block all communication if /usr/es/sbin/cluster/etc/rhosts is missingIf /usr/es/sbin/cluster/etc/rhosts is empty then assume cluster is being configured

Accept connection and add entry to the rhosts fileAt synchronization use HACMP ODM to build set of entries for the rhosts fileIn general connection authentication is done as follows:

Connect back and ask for the hostnameConnection is considered authentic if the hostname matches else connection is rejected


clcomd Authentication at Initial Configuration

/usr/es/sbin/cluster/etc/rhosts is the key:Nodes with an empty /usr/es/.../rhosts can be asked to join a clusterOnce the cluster is formed, participating nodes will not have an empty /usr/es/.../rhosts fileNodes with a non-empty /usr/es/.../rhosts file will refuse all HACMP related communication with nodes not listed in their rhosts file

To reconfigure a cluster:Empty /usr/es/sbin/.../rhosts on all cluster nodes

Not usually necessary unless IP addresses have changedInitiate configuration from one of the cluster nodes

To improve security prior to initial configuration:List the IP addresses of future cluster nodes in each nodes /usr/es/sbin/.../rhosts file


RSCTOriginally part of PSSP in the RS/6000 SP environmentProvides:

Scalability to large clustersNotification of software failures to remote subsystemsCoordinate recovery and synchronization of changes across the cluster

Key components:Event Management

Creates events by matching knowledge of cluster's state with expressions of interest by RSCT clients

Group ServicesCoordinates/monitors state changes of an application in the cluster

Topology ServicesUses heartbeats to monitor nodes, networks and network adaptersDiagnoses failuresCoordinates reintegration of recovered components

RMC: Resource Monitoring and ControlHACMP's Cluster Manager is an RSCT client/application


HACMP from an RSCT Perspective

RSCT Topology Services

RSCT Group

Services

RSCTRMC

(ctrmc)

HA Recovery Driver~

ClusterManager

Recovery Commands

~HACMP Event

Scripts

Recovery Programs

heartbeats messages

Switch Resource Monitor

AIX Process Monitor

Database Resource Monitor

processor, LANmembershipinformation

HACMP 5.2


Heartbeat Rings

25.8.60.5

25.8.60.4

25.8.60.3

25.8.60.2

25.8.60.6

Heartbeat

Heartbeat goes to next lower node in order of IP address only.


HACMP's SNMP SupportHACMP uses SNMP to provide:

Notification of cluster eventsCluster configuration/state information

SNMP support provided by HACMP's SMUX Peer daemon (clsmuxpd)clsmuxpd's role:

Kept aware of cluster's configuration and state by Cluster ManagerGenerates SNMP traps when notified of events by Cluster ManagerResponds to SNMP queries for HACMP informationA client (smux peer) of AIX's snmpdv3 in HACMP 5.2

HACMP 5.2 supports snmpd v1 and v3 in AIX 5.2 Implemented as the subsystem clsmuxpdES


Cluster Information Daemon (clinfo)An SNMP-aware client to clsmuxpdUsed by clstatProvides a cluster information API

Focused on providing HACMP cluster informationEasier to work with than the SNMP APIsSee HACMP for AIX Programming Client Applications manual for more information

Can be run within the clusterAs part of strategy for dealing with ARP cache issuesAs API provider to customer-written utility programs

Can be run on non-cluster nodesAs part of strategy for dealing with ARP cache issuesAs API provider to customer-written cluster monitoring tools

Source code provided on the HACMP product CD-ROMImplemented as the clinfoES subsystem


Highly Available NFS ServerReplaces now withdrawn HANFS stand-alone productPreserves file locks and dupcache across falloversCluster administrator can:

Specify a network to be used when a node is acting as an NFS client (typically to another node in the same cluster)Define NFS exports and mounts at the directory levelSpecify export options for NFS-exported directories and filesystems

NFS server support is limited to two node clustersMay be used with any resource group policy except start on all nodes


Shared External Disk AccessProvides two styles of shared disk support:

Serially reusable shared disks:Varied on by one node at a timeHACMP coordinates handover (during controlled RG moves) and takeover (during recovery from node failures)LVM or RSCT responsible for ensuring disks are not shared in real timeSupported with nonconcurrent mode or enhanced concurrent mode volume groups (running in nonconcurrent mode)

concurrent access shared disks:Typically used by concurrent applications writing to raw logical volumesTwo variants:

SSA Concurrent Mode Volume GroupsEnhanced Concurrent Mode Volume Groups running in concurrent mode

Concurrent volume groups requires the clvm fileset in HACMP 5.x


Start Sequence

Run rc.cluster script to start all

Start Topology Services

Start Group Services

Start Cluster Manager

Join Group Services

Run Recovery Program

Start node in SMIT

Node up

machines.lst

NodesAdapters

.Network Name SPetherNetwork Type ether**Node Type Address 0 en 192.168.3.37 1 en0 192.168.3.1 2 en0 192.168.3.2 3 en0 192.168.3.3 .Network Name SPswitchNetwork Type HPS**Node Type Address 1 css0 192.168.13.1 2 css0 192.168.13.2 3 css0 192.168.13.3 .

ODMNode namecss0

ODM

Cluster Startup

Start event mgmt services

/etc/inittab: rmc clcomd


run_clappmond

HACMP ES Cluster Manager

clappmond

Application Application

User-defined MonitoringMethod

RSCT - EventManagement

Application Monitoring

Process Monitoring Custom Monitoring

SAP

Startup or long-runningone or many


# smit cl_app_AAA.dialog

Application Availability Analysis


[Entry Fields]* Select an Application [] +* Begin analysis on YEAR (1970-2038) [] #* MONTH (01-12) [] #* DAY (1-31) [] #* Begin analysis at HOUR (00-23) [] #* MINUTES (00-59) [] #* SECONDS (00-59) [] #* End analysis on YEAR (1970-2038) [2003] #* MONTH (01-12) [12] #* DAY (1-31) [31] #* End analysis at HOUR (00-23) [20] #* MINUTES (00-59) [54] #* SECONDS (00-59) [02] #

F1=Help F2=Refresh F3=Cancel F4=List Esc+5=Reset F6=Command F7=Edit F8=Image F9=Shell F10=Exit Enter=Do

Application Availability Analysis Tool


Migrating to HACMP 5.2 (Overview)Node-by-node migration supported from:

HACMP 4.5 (classic or ES)HACMP 5.2

Steps to migrate from older versions (HACMP 4.4.0 or later):1. Take a snapshot, store in non HACMP directory.2. De-install HACMP.3. Upgrade AIX (and anything else that needs upgrading).4. Install HACMP 5.2.5. Run clconvert_snapshot, apply snapshot.6. Add custom scripts.7. Test the cluster.clconvert_snapshot usage gets complicated if existing cluster is old enough. It may be easiest to configure cluster using HACMP smit menus based on (up-to-date) cluster documentation.HACMP 4.5 is oldest interoperable release with HACMP 5.2

This is an OVERVIEW . . . refer to the HACMP 5.2 manuals and consider attending AU57 - HACMP Administration II: Maintenance and Migration



HACMP 5.2 allows multiple application monitors for the same application.

2. Which of the following are true about HACMP 5.2 (select all that apply):a. RMC used in HACMP only at release 5.2b. Cluster Lock Manager enhancedc. Cluster Information Daemon removedd. Enhanced Concurrent Volume Group supported in nonconcurrent modee. Clcomd provides message authentication in HACMP only in release 5.2

3. True or False?Migrating from older versions of HACMP is a routine task which generally requires little planning.

4. True or False?The cluster communication daemon (clcomd) eliminates the need for /.rhosts files.



HACMP 5.1 is derived from the HACMP/ES variant of HACMP for AIX 4.5.

2. Which of the following are part of the Concurrent Resource Manager functionality (select all that apply):

a. RMC used in HACMP only at release 5.2.b. Cluster Lock Manager enhanced.c. Cluster Information Daemon removed.d. Enhanced Concurrent Volume Group supported in nonconcurrent mode.e. clcomd provides message authentication in HACMP only in release 5.2.

3. True or False?Migrating from older versions of HACMP is a routine task which generally requires little planning.

4. True or False?The cluster communication daemon (clcomd) eliminates the need for /.rhosts files.


HACMP EventsAfter completing this topic, you should be able to: Describe what an HACMP event isExplain what happens when HACMP processes an eventDescribe the sequence of events when:

The first node starts in a clusterA new node joins an existing clusterA node leaves a cluster voluntarily


What is an HACMP Event?An HACMP event is an incident of interest to HACMP:

A node joins the clusterA node crashesA NIC failsA NIC recoversCluster administrator requests a resource group moveCluster administrator requests a configuration change (synchronization)

An HACMP event script is a script invoked by a recovery program to perform the recovery function required.

node_upnode_downfail_interfacejoin_interfacerg_movereconfig_topology_start


HACMP Event

RecoveryPrograms

Recovery CommandRecovery Command __ __ __

HACMP Cluster Manager

Group Services/ES

Topology Services/ES

# ## Beginning of Event Definition Node Up ### # TE_JOIN_NODE 0 /usr/sbin/cluster/events/node_up.rp 2 0 # 6) Resource variable only used for event manager events

# 7) Instance vector, only used for event manager events

rules.hacmprd file

RMC(ctrmc)

Event Script


Event Processing

Notify Command

Notify Command

Pre-Event Script (1)

Recovery CommandHACMP Event

Pre-Event Script (n)

RC=0 Counter>0

No

Yes

Yes

Post-Event Script (1) Post-Event Script (n)

No Boom!


Types of HACMP EventsSecondary events(called by other events)

node_up_localnode_up_remotenode_down_localnode_down_remotenode_up_local_completenode_up_remote_completenode_down_local_completenode_down_remote_completeacquire_aconn_serviceacquire_service_addracquire_takeover_addrstart_serverstop_serverget_disk_vg_fsget_aconn_rsrelease_service_addrrelease_takeover_addrrelease_vg_fsrelease_aconn_rsswap_aconn_protocolsreleasingacquiringrg_uprg_downrg_errorrg_temp_error_staterg_acquiring_secondaryrg_up_secondaryrg_error_secondaryresume_appmonsuspend_appmon

Primary events (called by clstmgrES recovery programs)

site_up, site_up_completesite_down, site_down_completesite_merge, site_merge_completenode_up, node_up_completenode_down, node_down_completenetwork_up, network_up_completenetwork_down, network_down_completeswap_adapter, swap_adapter_completeswap_address, swap_address_completefail_standbyjoin_standbyfail_interfacejoin_interfacerg_move, rg_move_completerg_onlinerg_offlineevent_errorconfig_too_longreconfig_topology_startreconfig_topology_completereconfig_resource_releasereconfig_resource_acquirereconfig_resource_completereconfig_configuration_dependency_acquirereconfig_configuration_dependency_completereconfig_configuration_dependency_releasenode_up_dependencynode_up_dependency_completenode_down_dependencynode_down_dependency_completemigrate, migrate_complete


First Node Starts HACMP

EventManager

1) node_upnode_up_local

acquire_service_addracquire_takeover_addrget_disk_vg_fs

2) node_up_completenode_up_local_complete

start_server run start script

Start

RCStart

RC

Star

t


Another Node Joins the Cluster

EventManager

EventManager

1) node_upnode_up_remote

stop_server run stop script

release_takeover_addressrelease_vg_fs

3) node_up_completenode_up_remote_complete

2) node_upnode_up_local

acquire_service_addressacquire_takeover_addressget_disk_vg_fs

4) node_up_completenode_up_local_complete

start_server run start script

Messages

Startrunning

Start

Start

Start

Start

RCRC

RC

RC


Node Leaves the Cluster (Stopped)

EventManager

EventManager Messages

Stoprunning

3) node_downnode_down_remote

acquire_service_addracquire_takeover_addrget_disk_vg_fs

4) node_down_completenode_down_remote_complete

start_server

1) node_downnode_down_local

stop_server run stop script

release_takeover_addrrelease_vg_fsrelease_service _addr

2) node_down_completenode_down_local_complete

3) cluster manager exits

Start

Start

Start

Start

RC

RC

RC

RC Stop



HACMP 5.x supports a maximum of five pre and five post events per HACMP event.

2. Which of the following are examples of primary HACMP events(select all that apply)?

a. node_upb. node_up_localc. node_up_completed. start_servere. rg_up

3. When a node joins an existing cluster, what is the correct sequence for these events?

a. node_up on new node, node_up on existing node, node_up_complete on new node, node_up_complete on existing node

b. node_up on existing node, node_up on new node, node_up_complete on new node, node_up_complete on existing node

c. node_up on new node, node_up on existing node, node_up_complete on existing node, node_up_complete on new node

d. node_up on existing node, node_up on new node, node_up_complete on existing node, node_up_complete on new node

4. True or False?Checkpoint questions are boring.


Defining Resource Group BehaviorAfter completing this topic, you should be able to: Define the Resource Group behavior policiesDefine startup, fallover and fallback policiesDefine run-time choicesExplain how the policies differDescribe Dependent behaviorExplain why you might use each typeDescribe how HACMP 5.2 policies relate to previous releases of HACMP


Resource Group PoliciesStartup Policy

Additional run-time settling time policyFallover Policy

Choosing a run-time dynamic node policyFallback Policy

Additional run-time fallback timer policy


Startup PolicyOnline on Home Node Only Online on First Available Node

Run time Settling Time may be setOnline Using Distribution Policy Online on All Available Nodes


Online On All Available Nodes Referred to as concurrent mode in previous HACMP releasesResource group runs simultaneously on as many nodes as are currently available

Node failure results in loss of node's processing powerNode recovery results in addition of node's processing power

Resource group restrictions: No JFS or JFS2 filesystems (only raw logical volumes)No service IP Labels / Addresses (which means no IPAT)

Requires applications which have been explicitly designed to operate in concurrent access mode (provide own lock management)Quite uncommon but have the potential to provide essentially zero downtime as the likelihood that at least one node is up at any given time is (presumably) quite high


Fallover PolicyFallover To Next Priority Node In The List

Cannot be used with All nodes startup policyFallover Using Dynamic Node Priority

Cannot be used with All nodes startup policyBring Offline (On Error Node)

Normally used with startup policy Online On All Available NodesCannot be used with Distribution startup policy


Fallback PolicyFallback To Higher Priority Node In The List

Can use a run time Delayed Fallback Timer preferenceCannot be used with Distribution startup policy

Never Fallback


Dependent Resource Groups

One resource group can be the parent of another resource groupParent will be brought online before childParent will be brought offline after childParent and child may be on same or different nodesThree levels of dependency supportedAll resource groups processed serially if dependency usedCannot manually move parent if child online

Parent RG

Child RG

Parent/Child RG

Child/Parent RG


Startup Fallover Fallback

Cascading defaults

On Home To Next To Higher

Cascading CWOF

On Home To Next Never

Cascading Inactive takeover

On First To Next To Higher

Cascading Inactive takeover CWOF

On First To Next Never

Rotating Distribution To Next Never

Concurrent On All Offline on error Never

Migrating Resource Groups


Checkpoint1. What startup, fallover, fallback policy would be the best to use for a 2 node

mutual takeover cluster using IPAT assuming that there are performance problems if both applications are running on the same node?

a. home, next, neverb. first, next, higherc. distribution, next, neverd. all, error, nevere. home, next, higher

2. True or False?HACMP 5.2 does not support choosing Cascading as a resource group type.

3. Custom resource groups were introduced in:a. HACMP/ES 4.5b. HACMP "classic" 4.5c. HACMP 5.1

4. True or False?Resource groups support IPAT via IP replacement in HACMP 5.2.


Checkpoint Answers1. What startup, fallover, fallback policy would be the best to use for a 2 node

mutual takeover cluster using IPAT assuming that there are performance problems if both applications are running on the same node?

a. home, next, neverb. first, next, higherc. distribution, next, neverd. all, error, nevere. home, next, higher

2. True or False?HACMP 5.2 does not support choosing Cascading as a resource group type.

3. Custom resource groups were introduced in:a. HACMP/ES 4.5b. HACMP "classic" 4.5c. HACMP 5.1

4. True or False?Resource groups support IPAT via IP replacement in HACMP 5.2.


Unit Summary

Having completed this unit, you should be able to: Describe the installation process for HACMP 5.2List and explain the purpose of the major HACMP 5.2 componentsDescribe the HACMP events conceptDescribe the resource group definition process in HACMP 5.2


Welcome to:

3.13.0.3

Cluster Installation and ConfigurationCluster Installation and Configuration


Unit Objectives

After completing this unit, you should be able to: Configure HACMP 5.2

Use Standard and Extended Configuration pathsTwo-Node Cluster Configuration Assistant

Configure HACMP Topology to include:IP-based networks enabled for address takeover via both alias and replacementNon-IP networks (rs232, tmssa, diskhb)Hardware Address Takeover

Configure HACMP Resources: Create resource groups using startup, fallover, and fallback policiesAdd and remove resource groups and nodes on an existing clusterTake a snapshotRemove a clusterStart and stop the cluster on one or more cluster nodes


After completing this topic, you should be able to: Configure a two-node HACMP cluster in a primary/standby configuration using the Two-Node Configuration Assistant:

A single resource group in primary/standby configuration usingAn application server with an enhanced concurrent mode volume groupAn IP network using IPAT via aliasingA non-IP network via heartbeat over disk

Configure a two node HACMP cluster in mutual takeover configurationStart and stop HACMP on one or more cluster nodes

Configuring a Two-Node Cluster


What We Are Going to Achieve?Our aim is to configure a two-node cluster with two resourcegroups in a mutual takeover configuration.We'll start by creating a two-node standby configuration with one resource group called xwebserver_group which uses bondar as its home (primary) node and hudson as its backup nodeWe're going to use the two-node configuration assistant path for this configuration.

bondar hudson

D D



Define and configure the AIX environment Storage (adapters, LVM volume group, filesystem)Networks (IP interfaces, /etc/hosts, non-IP networks and devices)Application start and stop scripts

Install the HACMP filesets and reboot

Configure the HACMP environmentTopology

Cluster, node names, HACMP ip and non-ip networksResources:

Application Server Service labels




The Topology ConfigurationHere's the key portion of the /etc/hosts file that we'll be using in this unit:

192.168.15.29 bondar-if1 # bondar's first interface IP label192.168.16.29 bondar-if2 # bondar's second interface IP label192.168.5.29 bondar-per # persistent node IP label on bondar192.168.15.31 hudson-if1 # hudson's first interface IP label192.168.16.31 hudson-if2 # hudson's second interface IP label192.168.5.31 hudson-per # persistent node IP label on hudson192.168.5.92 cxweb # the IP label for the application # normally resident on bondar

Hostnames: bondar, hudson

bondar's network configuration (defined via smit chinet) :en0 - 192.168.15.29

en1 - 192.168.16.29

hudson's network configuration:en0 - 192.168.15.31en1 - 192.168.16.31

These network interfaces are all connected to the same physical network.The subnet mask is 255.255.255.0 on all networks/NICs.An enhanced concurrent mode volume group "ecmvg" has been created to support the xweb application and will be used for a disk non-ip heartbeat network


Configuration MethodsHACMP provides two menu paths to configure topology and resources:

Standard Path Standard configurationTwo-Node Cluster Configuration Assistant (HACMP 5.2)

Extended Path (the traditional path) Use whichever method suits your needs

The Two-node Assistant may be all you needThe other Standard path options are similar to the Extended path but easier with less optionsThe Extended path is more flexible but with all the options


Plan Two-Node Configuration AssistantPlan configuration path to other node = hudson_if1

This node will be the home (primary) node

Plan Application Server name = xwebserver Used to name the cluster and the resource group

Ensure Application Server start and stop scripts exist and are put on Bondar (where 2 node assistant will be run from):

/mydir/xweb_start/mydir/xweb_stop

Plan service IP Label = xweb

Recommended: an enhanced concurrent vg


Starting at the Very Beginning . . .

System Management

Move cursor to desired item and press Enter.

Software Installation and Maintenance Software License Management Devices System Storage Management (Physical & Logical Storage) Security & Users

Communications Applications and Services Print Spooling Problem Determination Performance & Resource Scheduling System Environments Processes & Subsystems Applications Installation Assistant Cluster System Management Using SMIT (information only)

F1=Help F2=Refresh F3=Cancel F8=ImageF9=Shell F10=Exit Enter=Do


Almost There . . .

Communications Applications and Services


TCP/IP NFS

HACMP for AIX



The Top-Level HACMP smit Menu# smit hacmp

HACMP for AIX


Initialization and Standard Configuration Extended Configuration System Management (C-SPOC) Problem Determination Tools



The Standard Configuration Menu

Initialization and Standard Configuration


Two-Node Cluster Configuration Assistant Add Nodes to an HACMP Cluster Configure Resources to Make Highly Available Configure HACMP Resource Groups Verify and Synchronize HACMP Configuration

HACMP Cluster Test Tool Display HACMP Configuration

F1=Help F2=Refresh F3=Cancel F8=Image F9=Shell F10=Exit Enter=Do

hacmp 5.2

hacmp 5.2


Two-Node Cluster Configuration Assistant

Two-Node Cluster Configuration Assistant


[Entry Fields]

* Communication Path to Takeover Node [hudson_if1] +* Application Server Name [xwebserver]* Application Server Start Script [/mydir/xweb_start]* Application Server Stop Script [/mydir/xweb_stop]* Service IP Label [xweb] +



Let's See What We've Done

Cluster Name: xwebserver_clusterCluster Connection Authentication Mode: StandardCluster Message Authentication Mode: NoneCluster Message Encryption: NoneUse Persistent Labels for Communication: NoThere are 2 node(s) and 2 network(s) definedNODE bondar: Network net_ether_01 bondar-if1 192.168.15.29 bondar-if2 192.168.16.29 Network net_diskhb_01 bondar_hdisk5_01 /dev/hdisk5NODE hudson: Network net_ether_01 hudson-if1 192.168.15.31 hudson-if2 192.168.16.31 Network net_diskhb_01 hudson_hdisk5_01 /dev/hdisk5Resource Group xwebserver_group Startup Policy Online On Home Node Only Fallover Policy Fallover To Next Priority Node In The List Fallback Policy Fallback To Higher Priority Node In The List Participating Nodes bondar hudson

# /usr/es/sbin/cluster/utilities/cltopinfo



Define and configure the AIX environment Storage (adapters, LVM volume group, filesystem)Networks (IP interfaces, /etc/hosts, non-IP networks and devices)Application start and stop scripts

Install the HACMP filesets and rebootConfigure the HACMP environmentTopology

Cluster, node names, HACMP ip and non-ip networksResources:

Application Server Service labels




Starting HACMP (1 of 4)

HACMP for AIX


Initialization and Standard Configuration Extended Configuration

System Management (C-SPOC) Problem Determination Tools




System Management (C-SPOC)


Manage HACMP Services HACMP Communication Interface Management HACMP Resource Group and Application Management HACMP Log Viewing and Management HACMP Security and Users Management HACMP Logical Volume Management HACMP Concurrent Logical Volume Management HACMP Physical Volume Management

Open a SMIT Session on a Node




Manage HACMP Services


Start Cluster Services Stop Cluster Services Show Cluster Services



Starting HACMP (4 of 4)# smit clstart

Start Cluster Services


[Entry Fields]* Start now, on system restart or both now + Start Cluster Services on these nodes [bondar,hudson] + BROADCAST message at startup? true + Startup Cluster Information Daemon? true + Reacquire resources after forced down ? false +



Stopping HACMP# smit clstop

Stop Cluster Services


[Entry Fields]* Stop now, on system restart or both now + Stop Cluster Services on these nodes [bondar] + BROADCAST cluster shutdown? true +* Shutdown mode graceful +

+--------------------------------------------------------------------------+ ¦ Shutdown mode ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦ ¦ graceful ¦ ¦ takeover ¦ ¦ forced ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦F1¦ F8=Image F10=Exit Enter=Do ¦F5¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+


Shutdown Modes ExplainedHACMP has three modes for stopping HACMP on one or more nodes.These are known as Graceful, Takeover and Forced.

Mode Is the application available on the stopped node?

Is the application available on another active node in the cluster?

Does takeover of the resources occur?

Graceful No Only if online on all nodes

No

Takeover No Yes, if a node that services that resource group remains active in the cluster.

Yes, if a node that services that resource group remains active in the cluster.

Forced Yes Only if online on all nodes

No

AIX Shutdown No* Only if online on all nodes

No

* Technically, HACMP does a Forced so there is no takeoverbut then AIX stops so the application is not available


Are We There Yet? We've configured a two-node cluster with a single resourcegroup called xwebserver_group which uses bondar as the primary node and hudson as the backupNow let's add a second resource group called adventure, which uses hudson as the primary node and bondar as the backup.We're going to use the standard configuration path this time and move a bit faster*.

bondar hudson

D

A

D

A

*The smit menu screens are left out this time.


Adding the Second Resource Group

Configure HACMP Resource Groups


Add a Resource Group Change/Show a Resource Group Remove a Resource Group Change/Show Resources for a Resource Group (standard)



Setting Name, Participating Nodes

Add a Resource Group

* Resource Group Name [Adventure]* Participating Nodes (Default Node Priority) [hudson bondar] Startup Policy Online On Home Node O> + Fallover Policy Fallover To Next Prio> + Fallback Policy Fallback To Higher Pr> +



Choose Startup Policy



[Entry Fields]* Resource Group Name [adventure]* Participating Node Names (Default Node Priority) [hudson bondar] +

Startup Policy Online On Home Node O> + Fallover Policy Fallover To Next Prio> + +--------------------------------------------------------------------------+ ¦ Startup Policy ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦

¦ Online On Home Node Only ¦ ¦ Online On First Available Node ¦ ¦ Online Using Distribution Policy ¦ ¦ Online On All Available Nodes ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦F1¦ F8=Image F10=Exit Enter=Do ¦F5¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+


Choose Fallover Policy




Startup Policy Online On Home Node O> + Fallover Policy Fallover To Next Prio> + +--------------------------------------------------------------------------+ ¦ Fallover Policy ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦

¦ Fallover To Next Priority Node In The List ¦ ¦ Fallover Using Dynamic Node Priority ¦ ¦ Bring Offline (On Error Node Only) ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦F1¦ F8=Image F10=Exit Enter=Do ¦F5¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+


Choose Fallback Policy




Startup Policy Online On Home Node O> + Fallover Policy Fallover To Next Prio> + Fallback Policy Fallback To Higher Pr> + +--------------------------------------------------------------------------+ ¦ Fallback Policy ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦

¦ Fallback To Higher Priority Node In The List ¦ ¦ Never Fallback ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦F1¦ F8=Image F10=Exit Enter=Do ¦F5¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+


Ready to Add the Resource Group




Startup Policy Online On Home Node O> + Fallover Policy Fallover To Next Prio> + Fallback Policy Fallback To Higher Pr> +



Prepare for Adventure RG Resources

Extended Configuration


Discover HACMP-related Information from Configured Nodes Extended Topology Configuration Extended Resource Configuration Extended Event Configuration Extended Performance Tuning Parameters Configuration Security and Users Configuration Snapshot Configuration Export Definition File for Online Planning Worksheets

Extended Verification and Synchronization HACMP Cluster Test Tool


Create start, stop scripts for ywebCreate ywebvg, filesystem, add yweb service label to /etc/hostsRun discovery


Configure the Resources

Configure Resources to Make Highly Available


Configure Service IP Labels/Addresses Configure Application Servers Configure Volume Groups, Logical Volumes and Filesystems Configure Concurrent Volume Groups and Logical Volumes



Configuring Second Service IP Label

Configure Service IP Labels/Addresses


Add a Service IP Label/Address Change/Show a Service IP Label/Address Remove Service IP Label(s)/Address(es)



Adding Adventure Service Label (1 of 3)

Add a Service IP Label/Address (standard)


[Entry Fields]

* IP Label/Address [] +* Network Name [] +

+--------------------------------------------------------------------------+ ¦ IP Label/Address ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦ ¦ (none) ((none)) ¦ ¦ bondar (192.168.5.29) ¦ ¦ hudson (192.168.5.31) ¦

¦ yweb (192.168.5.70) ¦

¦ xweb (192.168.5.92) ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦F1¦ F8=Image F10=Exit Enter=Do ¦F5¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+





[Entry Fields]* IP Label/Address [yweb] +

* Network Name [] +

+--------------------------------------------------------------------------+ ¦ Network Name ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦

¦ net_ether_01 (192.168.15.0/24 192.168.16.0/24) ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦F1¦ F8=Image F10=Exit Enter=Do ¦F5¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+





[Entry Fields]* IP Label/Address [yweb] +* Network Name [net_ether_01] +



Add Adventure Application Server (1 of 2)

Configure Application Servers


Add an Application Server Change/Show an Application Server Remove an Application Server



Add Adventure Application Server (2 of 2)

Add Application Server


[Entry Fields]* Server Name [ywebserver]* Start Script [/usr/local/scripts/startyweb]* Stop Script [/usr/local/scripts/stopyweb]



Configure HACMP Resource Groups


Add a Resource Group Change/Show a Resource Group Remove a Resource Group Change/Show Resources for a Resource Group (standard)

+--------------------------------------------------------------------------+ ¦ Select a Resource Group ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦ ¦ xwebserver_group

| adventure ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F8=Image F10=Exit Enter=Do ¦F1¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+

Adding Resourcesto the Adventure RG (1 of 2)


Change/Show All Resources and Attributes for a Resource Group


[Entry Fields] Custom Resource Group Name adventure Participating Node Names (Default Node Priority) hudson bondar

Startup Behavior Online On First Avail> Fallover Behavior Fallover To Next Prio> Fallback Behavior Fallback To Higher Pr>

Service IP Labels/Addresses [yweb] + Application Servers [ywebserver] + Volume Groups [ywebvg] + Use forced varyon of volume groups, if necessary false + Filesystems (empty is ALL for VGs specified) [] +


Adding Resourcesto the Adventure RG (2 of 2)


Verify and Synchronize the Changes



Two-Node Cluster Configuration Assistant Add Nodes to an HACMP Cluster Configure Resources to Make Highly Available Configure HACMP Resource Groups

Verify and Synchronize HACMP Configuration HACMP Cluster Test Tool Display HACMP Configuration



Extended Configuration Menu



Discover HACMP-related Information from Configured Nodes

Extended Topology Configuration Extended Resource Configuration Extended Event Configuration Extended Performance Tuning Parameters Configuration Security and Users Configuration Snapshot Configuration Export Definition File for Online Planning Worksheets



HACMP 5.2

HACMP 5.2


Extended Topology Configuration Menu

Extended Topology Configuration


Configure an HACMP Cluster Configure HACMP Nodes Configure HACMP Sites Configure HACMP Networks

Configure HACMP Communication Interfaces/Devices Configure HACMP Persistent Node IP Label/Addresses Configure HACMP Global Networks Configure HACMP Network Modules Configure Topology Services and Group Services Show HACMP Topology



Communication Interfaces and Devices

Configure HACMP Communication Interfaces/Devices


Add Communication Interfaces/Devices Change/Show Communication Interfaces/Devices Remove Communication Interfaces/Devices Update HACMP Communication Interface with Operating System Settings



Defining a Non-IP Network (1 of 3) Configure HACMP Communication Interfaces/Devices



+--------------------------------------------------------------------------+ ¦ Select a category ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦

¦ Add Discovered Communication Interface and Devices ¦ ¦ Add Predefined Communication Interfaces and Devices ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F8=Image F10=Exit Enter=Do ¦F1¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+

Don't risk a potentially catastrophic partitioned cluster by using cheap rs232 cables!


Defining a Non-IP Network (2 of 3)




+--------------------------------------------------------------------------+ ¦ Select a category ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦ ¦ # Discovery last performed: (Feb 12 18:20) ¦ ¦ Communication Interfaces ¦

¦ Communication Devices ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F8=Image F10=Exit Enter=Do ¦F1¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+


Defining a Non-IP Network (3 of 3)



Add Communication Interfaces/Devices Change/Show Communication Interfaces/Devices Remove Communication Interfaces/Devices +--------------------------------------------------------------------------+ ¦ Select Point-to-Point Pair of Discovered Communication Devices to Add ¦ ¦ ¦ ¦ Move cursor to desired item and press F7. Use arrow keys to scroll. ¦ ¦ ONE OR MORE items can be selected. ¦ ¦ Press Enter AFTER making all selections. ¦ ¦ ¦ ¦ # Node Device Device Path Pvid ¦ ¦ bondar hdisk5 /dev/hdisk5 000b4a7cd1...¦ ¦ hudson hdisk5 /dev/hdisk5 000b4a7cd1...¦

¦ > bondar tty1 /dev/tty1 ¦

¦ > hudson tty1 /dev/tty1 ¦ ¦ bondar tmssa1 /dev/tmssa1 ¦ ¦ hudson tmssa2 /dev/tmssa2 ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F7=Select F8=Image F10=Exit ¦F1¦ Enter=Do /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+

Press Enter and HACMP defines a new non-IP network with these communication devices.


Defining Persistent Node IP Labels (1 of 3)

Configure HACMP Persistent Node IP Label/Addresses


Add a Persistent Node IP Label/Address Change / Show a Persistent Node IP Label/Address Remove a Persistent Node IP Label/Address




Configure HACMP Persistent Node IP Label/Addresses


Add a Persistent Node IP Label/Address Change / Show a Persistent Node IP Label/Address Remove a Persistent Node IP Label/Address

+--------------------------------------------------------------------------+ ¦ Select a Node ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦

¦ bondar ¦

¦ hudson ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F8=Image F10=Exit Enter=Do ¦F1¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+



Add a Persistent Node IP Label/Address


[Entry Fields]* Node Name bondar* Network Name [net_ether_01] +* Node IP Label/Address [bondar-per] +


Press Enter and then repeat for the hudson persistent IP label on the hudson node.


Synchronize Your Changes

HACMP Verification and Synchronization


[Entry Fields]* Verify, Synchronize or Both [Both] +* Automatically correct errors found during [No] + verification?

* Force synchronization if verification fails? [No] +* Verify changes only? [No] +* Logging [Standard] +


The extended configuration path provides verification and synchronization options.

Don't forget to verify that you actually implemented what was planned by executing your test plan.


Take a Snapshot

Add a Cluster Snapshot


[Entry Fields]* Cluster Snapshot Name [] / Custom Defined Snapshot Methods [] + Save Cluster Log Files in snapshot No +* Cluster Snapshot Description []


Snapshot default directory is /usr/es/sbin/cluster/snapshots


We're There!We've configured a two-node cluster with two resource groups:

Each node is the home (primary) node for one of the resource groups.Each resource group falls back to its home node on recovery

This is called a two-node mutual takeover cluster.

bondar hudson

D

A

D

A

Each resource group is also configured to use IPAT via IP aliasing.This particular style of cluster (mutual takeover with IPAT) is, by far, the most common style of HACMP cluster.



It is possible to configure a recommended simple two-node cluster environment using just the standard configuration path.

2. In which of the top level HACMP menu choices is the menu for starting and stopping cluster nodes?

a. Initialization and Standard Configurationb. Extended Configurationc. System Management (C-SPOC)d. Problem Determination Tools

3. An orderly shutdown of AIX while HACMP is running is equivalent to which of the following:

a. Graceful shutdown of HACMP followed by an orderly shutdown of AIX.b. Takeover shutdown of HACMP followed by an orderly shutdown of AIX.c. Forced shutdown of HACMP followed by an orderly shutdown of AIX.d. None of the above.

4. True or False? It is possible to configure HACMP faster by having someone help you on the other node.

5. True or False?You must specify exactly which filesystems you want mounted when you put resources into a resource group.


Checkpoint Answers1. True or False?*

It is possible to configure a recommended simple two-node cluster environment using just the standard configuration path.

2. In which of the top level HACMP menu choices is the menu for starting and stopping cluster nodes?

a. Initialization and Standard Configurationb. Extended Configurationc. System Management (C-SPOC)d. Problem Determination Tools

3. An orderly shutdown of AIX while HACMP is running is equivalent to which of the following:

a. Graceful shutdown of HACMP followed by an orderly shutdown of AIX.b. Takeover shutdown of HACMP followed by an orderly shutdown of AIX.c. Forced shutdown of HACMP followed by an orderly shutdown of AIX.d. None of the above.

4. True or False?**It is possible to configure HACMP faster by having someone help you on the other node.

5. True or False?You must specify exactly which filesystems you want mounted when you put resources into a resource group.

*This was False in previous releases as it was not possible to configure the recommended non-IP network using the standard path. However the Two-Node configuration assistant can.**Whoever synchronizes first will cause their changes to take effect and result in the other person's changes to made prior to the time of the first synchronization to be thrown away.


Break Time!

Please don't paddle too far as we'll be resuming shortly . . .


After completing this topic, you should be able to: Configure a third resource group to minimize downtime.Add a new node to an existing cluster.Remove a node from an existing cluster.Remove a resource group from a cluster.Configure hardware address takeover using cascading resource groups and IPAT via IP replacement.Configure a target-mode SSA non-IP heartbeat network.Configure a non-IP disk heartbeat network

Other Configuration Scenarios


Yet Another Resource GroupThe users have asked that a third application be added to the cluster.The application uses very little CPU or memory and there's money in the budget for more disk drives in the disk enclosure.Minimizing downtime is particularly important for this application.The resource group is called ballerina (nobody seems to know why).

bondar hudson

D

A

D

A

B B


Adding a Third Resource Group



[Entry Fields]* Resource Group Name [ballerina]* Participating Node Names (Default Node Priority) [bondar hudson] +

Startup Policy Online On First Avail> + Fallover Policy Fallover To Next Prio> + Fallback Policy Never Fallback +


avoid startup delay by starting on first available node

avoid fallback outage by never falling back

Does the order in which the node names are specified matter?

We'll change the startup policy to "Online On First Available Node" so thatthe resource group comes up when bondar is started when hudson is down.


Adding a Third Service IP Label (1 of 2)The extended configuration path screen for adding a service IP label provides more options. We choose those which mimic the standard path.

Configure HACMP Service IP Labels/Addresses



+--------------------------------------------------------------------------+ ¦ Select a Service IP Label/Address type ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦

¦ Configurable on Multiple Nodes ¦ ¦ Bound to a Single Node ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F8=Image F10=Exit Enter=Do ¦F1¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+


Adding a Third Service IP Label (2 of 2)The Alternate Hardware Address ... field is used for hardware address takeover (which we'll configure later).

Add a Service IP Label/Address configurable on Multiple Nodes (extended)


[Entry Fields]* IP Label/Address [zweb] +* Network Name net_ether_01 Alternate HW Address to accompany IP Label/Address []



Adding a Third Application Server

Add Application Server


[Entry Fields]* Server Name [zwebserver]* Start Script [/usr/local/scripts/startzweb]* Stop Script [/usr/local/scripts/stopzweb]


The Add Application Server screen is identical in both configuration paths.




[TOP] [Entry Fields] Resource Group Name adventure Resource Group Management Policy custom Inter-site Management Policy ignore Participating Node Names (Default Node Priority) hudson bondar

Startup Behavior Online On First Avail> Fallover Behavior Fallover To Next Prio> Fallback Behavior Fallback To Higher Pr> Fallback Timer Policy (empty is immediate) [] +

Service IP Labels/Addresses [zweb] + Application Servers [zwebserver] +

Volume Groups [zwebvg] + Use forced varyon of volume groups, if necessary false + Automatically Import Volume Groups false + Filesystems (empty is ALL for VGs specified) [] + Filesystems Consistency Check fsck +[MORE...17]


The extended path's smit screen for updating the contents of a resource group is MUCH more complicated!

Adding Resources to the Third RG (1 of 2)




[MORE...17] [Entry Fields] Filesystems Consistency Check fsck + Filesystems Recovery Method sequential + Filesystems mounted before IP configured false + Filesystems/Directories to Export [] + + Filesystems/Directories to NFS Mount [] + Network For NFS Mount [] +

Tape Resources [] + Raw Disk PVIDs [] +

Fast Connect Services [] + Communication Links [] +

Primary Workload Manager Class [] + Secondary Workload Manager Class [] +

Miscellaneous Data [][BOTTOM]


Even more choices!Fortunately, only a handful tend to be used in any given context.

Adding Resources to the Third RG (2 of 2)





[Entry Fields]* Verify, Synchronize or Both [Both] + Force synchronization if verification fails? [No] +* Verify changes only? [No] +* Logging [Standard] +


The extended configuration path provides verification and synchronization options.

Don't forget to verify that you actually implemented what was planned by executing your test plan.


Expanding the ClusterThe Users "find" money in the budget and decide to "invest" itto improve the availability of the adventure and discovery applications.Nobody seems to be too worried about the ballerina application.

bondar hudson

D

A

D

A

B B

jones

A

D


Adding a New Cluster NodePhysically connect the new node to the appropriate networksand to the shared storage subsystem.

Configure non-IP networks to create a ring encompassing all nodes.Configure the shared volume groups on the new node.Add the new node's IP labels to /etc/hosts on one of the existing nodes.Copy /etc/hosts from the existing node to all other nodes.Install AIX, HACMP and the application software on the new node:

Install patches required to bring the new node up to the same level as the existing cluster nodes.Reboot the new node (always reboot after installing or patching HACMP).

Add the new node to the existing cluster's topology (from one of the existing nodes) and synchronize your changes.Start HACMP on the new node.Add the new node to the appropriate resource groupsand synchronize your changes again.Run through your (updated) test plan.


Add Node -- Standard Path

Configure Nodes to an HACMP Cluster (standard)


[Entry Fields]* Cluster Name [xwebserver_cluster] New Nodes (via selected communication paths) [jones-if1] + Currently Configured Node(s) bondar hudson



Add Node -- Standard Path (In Progress)Here's the output shortly after pressing Enter:

COMMAND STATUS

Command: OK stdout: yes stderr: no

Before command completion, additional instructions may appear below.

[TOP]Communication path jones-if1 discovered a new node. Hostname is jones. Adding it to the configuration with Nodename jones.

Discovering IP Network Connectivity

Retrieving data from available cluster nodes. This could take a few minutes....

F1=Help F2=Refresh F3=Cancel F6=CommandF8=Image F9=Shell F10=Exit /=Findn=Find Next


Add Node -- Extended Path

Add a Node to the HACMP Cluster


[Entry Fields]* Node Name [jones] Communication Path to Node [jones_if1] +



Define the Non-IP rs232 Networks (1 of 2)


+--------------------------------------------------------------------------+ ¦ Select Point-to-Point Pair of Discovered Communication Devices to Add ¦ ¦ ¦ ¦ Move cursor to desired item and press F7. Use arrow keys to scroll. ¦ ¦ ONE OR MORE items can be selected. ¦ ¦ Press Enter AFTER making all selections. ¦ ¦ ¦ ¦ # Node Device Device Path Pvid ¦ ¦ bondar tty0 /dev/tty0 ¦ ¦ hudson tty0 /dev/tty0 ¦ ¦ jones tty0 /dev/tty0 ¦ ¦ bondar tty1 /dev/tty1 ¦ ¦ hudson tty1 /dev/tty1 ¦

¦ > jones tty1 /dev/tty1 ¦

¦ > bondar tty2 /dev/tty2 ¦ ¦ hudson tty2 /dev/tty2 ¦ ¦ jones tty2 /dev/tty2 ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F7=Select F8=Image F10=Exit ¦F1¦ Enter=Do /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+

We've added (and tested) a fully wired rs232 null modem cable between jones' tty1 and bondar's tty2 so we define that as a non-IP rs232 network.


Define the Non-IP rs232 Networks (2 of 2)


+--------------------------------------------------------------------------+ ¦ Select Point-to-Point Pair of Discovered Communication Devices to Add ¦ ¦ ¦ ¦ Move cursor to desired item and press F7. Use arrow keys to scroll. ¦ ¦ ONE OR MORE items can be selected. ¦ ¦ Press Enter AFTER making all selections. ¦ ¦ ¦ ¦ # Node Device Device Path Pvid ¦ ¦ bondar tty0 /dev/tty0 ¦ ¦ hudson tty0 /dev/tty0 ¦ ¦ jones tty0 /dev/tty0 ¦ ¦ bondar tty1 /dev/tty1 ¦ ¦ hudson tty1 /dev/tty1 ¦ ¦ jones tty1 /dev/tty1 ¦ ¦ bondar tty2 /dev/tty2 ¦

¦ > hudson tty2 /dev/tty2 ¦

¦ > jones tty2 /dev/tty2 ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F7=Select F8=Image F10=Exit ¦F1¦ Enter=Do /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+

We've also added (and tested) a fully wired rs232 null-modem cable between hudson's tty2 and jones' tty2 so we define that as a non-IP rs232 network.


Start HACMP on the New Node# smit clstart

Start Cluster Services


[Entry Fields]* Start now, on system restart or both now + Start Cluster Services on these nodes [jones] + BROADCAST message at startup? true + Startup Cluster Lock Services? false + Startup Cluster Information Daemon? false + Reacquire resources after forced down ? false +



Add the Node to a Resource Group

Change/Show a Resource Group


[Entry Fields] Resource Group Name adventure New Resource Group Name [] Participating Node Names (Default Node Priority) [hudson bondar jones] +

Startup Policy Online On Home Node O> + Fallover Policy Fallover To Next Prio> + Fallback Policy Fallback To Higher Pr> +


Repeat for the discovery resource group.







Synchronize the changes and run through the test plan.


Shrinking the ClusterThe Auditors aren't impressed with the latest investmentand force the removal of the jones node from the cluster so that it can be transferred it to a new project (some users suspect that political considerations may have been involved).

bondar hudson

D

A

D

A

B B

jones

X


Removing a Cluster NodeUsing any cluster node, remove the departing node from all resource groups (ensure that each resource group is left with at least two nodes) and synchronize your changes.Stop HACMP on the departing node.Using one of the other cluster nodes which is not being removed:

Remove the departing node from the cluster's topology (using the Remove a Node from the HACMP Cluster smit screen in the extended configuration path) and synchronize your change.Once the synchronization is completed successfully, the departing node is no longer a member of the cluster.

Remove the departed node's IP addresses from /usr/es/sbin/cluster/etc/rhosts on the remaining nodes (prevents the departed node from interfering with HACMP on the remaining nodes).Physically disconnect the (correct) rs232 cables.Disconnect the departing node from the shared storage subsystem (strongly recommended as it makes it impossible for the departednode to screw up the cluster's shared storage).Run through your (updated) test plan.


Removing an ApplicationThe zwebserver application has been causing problems and a decision has been made to move it out of the cluster.

bondar hudson

D

A

D

A

B B


Removing a Resource Group (1 of 3)Take the resource group offlineUsing any cluster node and either configuration path:

Remove the departing resource group using the Remove a Resource Group smit screen.Remove any service IP labels previously used by the departing resource group using the Remove Service IP Labels/Addresses smit screen.Synchronize your changes (this will shutdown the resource group's applications using the application server's stop script and release any resources previously used by the resource group).

Clean out anything that is no longer needed by the cluster:Export any shared volume groups previously used by the application.Consider deleting service IP labels from the /etc/hosts file.Uninstall the application.

Run through your (updated) test plan.


Removing a Resource Group (2 of 3)

HACMP Extended Resource Group Configuration


Add a Resource Group Change/Show a Resource Group Change/Show Resources and Attributes for a Resource Group

Remove a Resource Group Show All Resources by Node or Resource Group

+--------------------------------------------------------------------------+ ¦ Select a Resource Group ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦ ¦ adventure ¦

¦ ballerina ¦ ¦ discovery ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F8=Image F10=Exit Enter=Do ¦F1¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+


Removing a Resource Group (3 of 3)

Press enter (if you are sure).

HACMP Extended Resource Group Configuration


Add a Resource Group Change/Show a Resource Group Change/Show Resources and Attributes for a Resource Group

Remove a Resource Group Show All Resources by Node or Resource Group

+--------------------------------------------------------------------------+ ¦ ARE YOU SURE? ¦ ¦ ¦ ¦ Continuing may delete information you may want ¦ ¦ to keep. This is your last chance to stop ¦ ¦ before continuing. ¦ ¦ Press Enter to continue. ¦ ¦ Press Cancel to return to the application. ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦F1¦ F8=Image F10=Exit Enter=Do ¦F9+--------------------------------------------------------------------------+


Implementing Hardware Address TakeoverSomeone just got a great deal on a dozen used FOOL-97x computers for the summer students to use.They run some strange proprietary operating system which refuses to update its ARP cache in response to either ping or gratuitous ARP packets.

bondar hudson

D

A

D

A


Our Plan for Implementing HWATStop HACMP on both cluster nodes (use graceful shutdown option tobring down the resource groups and their applications).Remove the alias service labels from the Resources (they are in the wrong subnet for replacement). They are automatically removed from an RG.Convert the net_ether_01 ethernet network to use IPAT via IP replacement:

Disable IPAT via IP aliasing on the ethernet network.Update /etc/hosts on both cluster nodes to describe service IP labels and addresses on the 192.168.15.0 subnet.Use the procedure described in the networking to select the LAA addresses.Configure new service IP labels with these LAA addresses in the HACMP smit screens.

Define resource groups to use the new service IP labels.Synchronize the changesRestart HACMP on the two nodes.


Stopping HACMP# smit clstop

Stop Cluster Services


[Entry Fields]* Stop now, on system restart or both now + Stop Cluster Services on these nodes [bondar,hudson] + BROADCAST cluster shutdown? true +* Shutdown mode graceful +



Removing a Service IP Label

Configure HACMP Service IP Labels/Addresses



+--------------------------------------------------------------------------+ ¦ Select Service IP Label(s)/Address(es) to Remove ¦ ¦ ¦ ¦ Move cursor to desired item and press F7. ¦ ¦ ONE OR MORE items can be selected. ¦ ¦ Press Enter AFTER making all selections. ¦ ¦ ¦ ¦ xweb ¦ ¦ yweb ¦ ¦ zweb ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F7=Select F8=Image F10=Exit ¦F1¦ Enter=Do /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+

Repeat for both service IP labels.

Press Enter here and you will be prompted to confirm the removal.


Change Network to Disable IPAT via Aliases

Change/Show an IP-Based Network in the HACMP Cluster


[Entry Fields]* Network Name net_ether_01 New Network Name []* Network Type [ether] +* Netmask [255.255.255.0] +* Enable IP Address Takeover via IP Aliases [No] + IP Address Offset for Heartbeating over IP Aliases []


Set the "Enable IP Address Takeover via IP Aliases" setting to "No" and press Enter.


The Updated /etc/hostsHere's the key portion of the /etc/hosts file with the service IPlabels moved to the 192.168.15.0 subnet:

192.168.5.29 bondar # persistent node IP label on bondar192.168.15.29 bondar-if1 # bondar's first boot IP label192.168.16.29 bondar-if2 # bondar's second boot IP label192.168.5.31 hudson # persistent node IP label on hudson192.168.15.31 hudson-if1 # hudson's first boot IP label192.168.16.31 hudson-if2 # hudson's second boot IP label192.168.15.92 xweb # the IP label for the application normally # resident on bondar192.168.15.70 yweb # the IP label for the application normally # resident on hudson

Note that neither bondar or hudson's network configuration (as defined with the AIX TCP/IP smit screens) needs to be changed.Note that we are not renaming the interface IP labels to something like bondar_boot and bondar_standby as changing IP labels in an HACMP cluster can be quite a bit of work (it is often easier to delete the cluster definition and start over)


Selecting LAA AddressesHere are two Globally Administered Addresses (GAAs) takenfrom ethernet adapters in the cluster:

0.4.ac.17.19.640.6.29.ac.46.8

First we make sure that each number is two digits long by adding leading zeros as necessary:

00.04.ac.17.19.6400.06.29.ac.46.08

Verify that the first digit is 0, 1, 2 or 3:Yep!

Add 4 to the first digit of each GAA:40.04.ac.17.19.6440.06.29.ac.46.08

Done! The two addresses just above are now LAAs.


Redefining the Service IP Labels for HWAT

Add a Service IP Label/Address configurable on Multiple Nodes (extended)


[Entry Fields]* IP Label/Address [xweb] +* Network Name net_ether_01 Alternate HW Address to accompany IP Label/Address [4004ac171964]


Redefine the two service IP labels. Note that the periods are stripped out before the LAA is entered into the HW Address field.

You probably shouldn't use the particular LAAs shown on these foils in your cluster. Select your own LAAs using the procedure described in the networking unit.

Don't forget to specify the second LAA for the second service IP label.


Implementing Target Mode SSAThe serial cable being used to implement the rs232 non-IP network has been borrowed by someone and nobody noticed.A decision has been made to implement a target mode SSA (tmssa) non-IP network as it won't fail unless complete access to the shared SSA disks is lost by one of the nodes (and someone is likely to notice that).

bondar hudson

D

A

D

A


Setting the SSA Node NumberThe first step is to give each node a unique SSA node number.We'll set bondar's ssa node number to 1 and hudson's to 2.

Use the "smit ssaa" fastpath to get to AIX's SSA Adapters menu.

Change/Show the SSA Node Number For This System


[Entry Fields] SSA Node Number [1] +#



Configuring the tmssa DevicesThis is a three-step process for a two-node cluster as eachnode needs tmssa devices which refer to the other node:1. run cfgmgr on one of the nodes (bondar).

bondar is now ready to respond to tmssa queries.2. run cfgmgr on the other node (hudson).

hudson is now ready to respond to tmssa queries.hudson also knows that bondar supports tmssa and has created the tmssa devices (/dev/tmssa1.im and /dev/tmssa1.tm) which refer to bondar.

3. run cfgmgr again on the first node (bondar).bondar now also knows that hudson supports tmssa and has created the tmssa devices (/dev/tmssa2.im and /dev/tmssa2.tm) which refer to hudson.

bondar now has /dev/tmssa2.im /dev/tmssa2.tm devices which refer to hudson


Rediscover the HACMP InformationNext we need to get HACMP to know about the new communicationdevices so we run the auto-discovery procedure again on one of the nodes.



Discover HACMP-related Information from Configured Nodes Extended Topology Configuration Extended Resource Configuration Extended Event Configuration Extended Performance Tuning Parameters Configuration Security and Users Configuration Snapshot Configuration

Extended Verification and Synchronization



Defining a Non-IP tmssa Network (1 of 3)




+--------------------------------------------------------------------------+ ¦ Select a category ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦ ¦ Add Discovered Communication Interface and Devices ¦ ¦ Add Predefined Communication Interfaces and Devices ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F8=Image F10=Exit Enter=Do ¦F1¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+

This should look very familiar as it is the same procedure that was used to define the non-IP rs232 network earlier.


Defining a Non-IP tmssa Network (2 of 3)




+--------------------------------------------------------------------------+ ¦ Select a category ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦ ¦ # Discovery last performed: (Feb 12 18:20) ¦ ¦ Communication Interfaces ¦ ¦ Communication Devices ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F8=Image F10=Exit Enter=Do ¦F1¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+


Defining a Non-IP tmssa Network (3 of 3)Now we need to define the tmssa network using a process



Add Communication Interfaces/Devices +--------------------------------------------------------------------------+ ¦ Select Point-to-Point Pair of Discovered Communication Devices to Add ¦ ¦ ¦ ¦ Move cursor to desired item and press F7. Use arrow keys to scroll. ¦ ¦ ONE OR MORE items can be selected. ¦ ¦ Press Enter AFTER making all selections. ¦ ¦ ¦ ¦ # Node Device Device Path Pvid ¦ ¦ > hudson tmssa1 /dev/tmssa1 ¦ ¦ > bondar tmssa2 /dev/tmssa2 ¦ ¦ bondar tty0 /dev/tty0 ¦ ¦ hudson tty0 /dev/tty0 ¦ ¦ bondar tty1 /dev/tty1 ¦ ¦ hudson tty1 /dev/tty1 ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F7=Select F8=Image F10=Exit ¦F1¦ Enter=Do /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+


Removing a Cluster

Configure an HACMP Cluster


Add/Change/Show an HACMP Cluster

Remove an HACMP Cluster Reset Cluster Tunables


Use Extended Topology Configuration.

Make /usr/es/sbin/cluster/etc/rhosts a null file:cat "" > /usr/es/sbin/cluster/etc/rhosts



It is impossible to add a node while HACMP is running.

2. Which of the following are not supported by HACMP 5.1? (select all that apply)

a. Cascading resource group with IPAT via IP aliasing.b. Custom resource group with IPAT via IP replacement.c. HWAT in a resource group which uses IPAT via IP aliasing.d. HWAT in a custom resource group.e. More than three custom resource groups in a two node cluster.

3. Which of the following sequences of steps implement HWAT in a cluster currently using custom resource groups?

a. Delete custom RGs, define cascading RGs, places resources in new RGs, disable IPAT via IP aliasing on network, delete old service IP labels, define new service IP labels, synchronize

b. Delete custom RGs, define cascading RGs, places resources in new RGs, delete old service IP labels, disable IPAT via IP aliasing on network, define new service IP labels, synchronize

c. Delete custom RGs, disable IPAT via IP aliasing on network, delete old service IP labels, define new service IP labels, define cascading RGs, places resources in new RGs, synchronize

d. Delete custom RGs, delete old service IP labels, disable IPAT via IP aliasing on network, define new service IP labels, define cascading RGs, places resources in new RGs, synchronize



It is impossible to add a node while HACMP is running.

2. Which of the following are not supported by HACMP 5.1? (select all that apply)

a. Cascading resource group with IPAT via IP aliasing.b. Custom resource group with IPAT via IP replacement.c. HWAT in a resource group which uses IPAT via IP aliasing.d. HWAT in a custom resource group.e. More than three custom resource groups in a two node cluster.

3. Which of the following sequences of steps implement HWAT in a cluster currently using custom resource groups? *

a. Delete custom RGs, define cascading RGs, places resources in new RGs, disable IPAT via IP aliasing on network, delete old service IP labels, define new service IP labels, synchronize

b. Delete custom RGs, define cascading RGs, places resources in new RGs, delete old service IP labels, disable IPAT via IP aliasing on network, define new service IP labels, synchronize

c. Delete custom RGs, disable IPAT via IP aliasing on network, delete old service IP labels, define new service IP labels, define cascading RGs, places resources in new RGs, synchronize

d. Delete custom RGs, delete old service IP labels, disable IPAT via IP aliasing on network, define new service IP labels, define cascading RGs, places resources in new RGs, synchronize

*Old service IP labels must be deleted before disabling IPAT via IP aliasing and new service IP labels must exist before they can be placed into the resource groups.


Unit Summary

Having completed this unit, you should be able to: Configure HACMP 5.2

Use Standard and Extended Configuration pathsTwo-Node Cluster Configuration Assistant

Configure HACMP Topology to include:IP-based networks enabled for address takeover via both alias and replacementNon-IP networks (rs232, tmssa, diskhb)Hardware Address Takeover

Configure HACMP Resources: Create resource groups using startup, fallover, and fallback policiesAdd and remove resource groups and nodes on an existing clusterTake a snapshotRemove a clusterStart and stop the cluster on one or more cluster nodes


Welcome to:

3.13.0.3

Cluster Single Point of ControlCluster Single Point of Control


Unit Objectives

After completing this unit, you should be able to: Understand the need for change management when using HACMPUnderstand the benefits and capabilities of C-SPOCPerform routine administrative changes using C-SPOC

LVMUsersDisk

Perform RGmove operations


Administering a High Availability ClusterAdministering a HA cluster is different from administering a stand-alone server:

Changes made to one node need to be reflected on the other nodePoorly considered changes can have far reaching implications

Beware the law of unintended consequencesAspects of the clusters configuration could be quite subtle and yet criticalScheduling downtime to install and test changes can be challengingSaying oops while sitting at a cluster console could get you fired!


RecommendationsImplement and adhere to a change control/managementprocessWherever possible, use HACMP's C-SPOC facility to make changes to the cluster (details to follow).Document routine operational procedures in a step-by-step list fashion (for example, shutdown, startup, increasing size of a filesystem).Restrict access to the root password to trained High Availability cluster administrators.Always take a snapshot (explained later) of your existing configuration before making a change.


Change Control or ManagementA real change control or management process requires a serious commitment on the part of the entire organization:

Every change must be carefully consideredThe onus should be on the requester of the change to demonstrate that it is necessary

Not on the cluster administrators to demonstrate that it is unwise.management must support the process

Defend cluster administrators against unreasonable request or pressureNot allow politics to affect a change's priority or schedule

Every change, even the minor ones, must follow the processThe cluster administrators must not sneak changes past the processThe notion that a change might be permitted without following the process must be considered to be absurd

The alternative is that the process rapidly becomes a farce


Change ConsiderationsEvery change must be carefully considered:

Is the change necessary?How urgent is the change?How important is the change? (not the same thing as urgent)What impact does the change have on other aspects of the cluster?What is the impact if the change is not allowed to occur?Are all of the steps required to implement the change clearly understood and documented?How is the change to be tested?What is the plan for backing out the change if necessary?Is the appropriate expertise be available should problems develop?When is the change scheduled?Have the users been notified?Does the maintenance period include sufficient time for a full set of backups prior to the change and sufficient time for a fullrestore afterwards should the change fail testing?


Masking or Eliminating Planned Downtime

Continuous Availability

Continuous Operations

High Availability

Elimination of Downtime

Masking or elimination of planned downtime throughchange management

Masking or elimination of unplanned downtime


Cluster Single Point of Control (C-SPOC)C-SPOC provides facilities for performing common cluster-wideadministration tasks from any node within the cluster.

HACMP 4.x requires either /.rhosts or kerberos to be configured on all nodes HACMP 5.x uses the clcomdES socket based subsystem.C-SPOC operations fail if any target node is down at the time of execution or selected resource is not available.Any change to a shared VGDA is synchronized automatically if C-SPOC is used to change a shared LVM component.C-SPOC uses a script parser called the command execution language

Initiating node

Target node

Target node

Target node

Target node


The Top-Level C-SPOC Menu

System Management (C-SPOC)


Manage HACMP Services HACMP Communication Interface Management HACMP Resource Group and Application Management HACMP Log Viewing and Management HACMP File Collection Management HACMP Security and Users Management HACMP Logical Volume Management HACMP Concurrent Logical Volume Management HACMP Physical Volume Management



HACMP 5.2 --


Adding a User to the Cluster (1 of 2)

Add a User to the Cluster

Type or select a value for the entry field.Press Enter AFTER making all desired changes.

[Entry Fields] Select nodes by Resource Group [] + *** No selection means all nodes! ***



Adding a User to the Cluster (2 of 2)

Add a User to the Cluster


[TOP] [Entry Fields] Select nodes by Resource Group adventure *** No selection means all nodes! ***

* User NAME [danny] User ID [500] # ADMINISTRATIVE USER? false + Primary GROUP [] + Group SET [] + ADMINISTRATIVE GROUPS [] + Another user can SU TO USER? true + SU GROUPS [ALL] + HOME directory [/home/danny] Initial PROGRAM [][MORE...34]



Removing a User from the Cluster

Remove a User from the Cluster


[Entry Fields] Select nodes by Resource Group *** No selection means all nodes! ***

* User NAME [paul] + Remove AUTHENTICATION information? Yes +



Passwords in an HACMP Cluster

Passwords in an HACMP cluster


Change a User's Password in the Cluster Change Current Users Password Manage List of Users Allowed to Change Password Modify System Password Utility

F1=Help F2=Refresh F3=Cancel Esc+8=ImageEsc+9=Shell Esc+0=Exit Enter=Do

HACMP 5.2-- HACMP 5.2-- HACMP 5.2--


Adding a Physical Disk to a Cluster

Add an SSA Logical Disk


[Entry Fields] Node Name(s) to which disk is attached bondar,hudson + Device type disk + Disk Type hdisk Disk interface ssar Description SSA Logical Disk Driv> Parent ssar* CONNECTION address [] + Location Label [] ASSIGN physical volume identifier yes + RESERVE disk on open yes + Queue depth [] + Maximum Coalesce [] +



Managing Shared LVM Components

HACMP Logical Volume Management


Shared Volume Groups Shared Logical Volumes Shared File Systems Synchronize Shared LVM Mirrors Synchronize a Shared Volume Group Definition



Creating a Shared Volume Group

Create a Shared Volume Group


[Entry Fields] Node Names bondar,hudson PVID 00055207bbf6edab 0000> VOLUME GROUP name [bernhardvg] Physical partition SIZE in megabytes 64 + Volume group MAJOR NUMBER [207] # Enable Cross-Site LVM Mirroring Verification false +

Warning : Changing the volume group major number may result in the command being unable to execute successfully on a node that does not have the major number currently available. Please check for a commonly available major number on all nodes before changing this setting.


HACMP 5.2--


Discover, Add VG to a Resource Group



Discover HACMP-related Information from Configured Nodes Extended Topology Configuration

Extended Resource Configuration Extended Event Configuration Extended Performance Tuning Parameters Configuration Security and Users Configuration Snapshot Configuration Export Definition File for Online Planning Worksheets




Creating a Shared File System (1 of 2)

Add a Shared Logical Volume


[TOP] [Entry Fields] Resource Group Name adventure VOLUME GROUP name bernhardvg Reference node* Number of LOGICAL PARTITIONS [200] # PHYSICAL VOLUME names Logical volume NAME [norbertfs] Logical volume TYPE [jfs] POSITION on physical volume middle + RANGE of physical volumes minimum + MAXIMUM NUMBER of PHYSICAL VOLUMES [] # to use for allocation Number of COPIES of each logical 2 + partition[MORE...11]


The volume group must be online somewhere and listed in a resource group or it does not appear in the pop-up list.

First create mirrored logical volumes for the filesystem and jfslog.Do not forget to logform the jfslog logical volume.


Creating a Shared File System (2 of 2)

Add a Standard Journaled File System


[Entry Fields] Node Names bondar,hudson LOGICAL VOLUME name norbertfs* MOUNT POINT [/norbert] PERMISSIONS read/write + Mount OPTIONS [] + Start Disk Accounting? no + Fragment Size (bytes) 4096 + Number of bytes per inode 4096 + Allocation Group Size (MBytes) 8 +


Then create the filesystem in the now "previously defined logical volume".


LVM Change ManagementHistorically, lack of LVM change management has been a major cause of cluster failure during fallover. There are several methods available to ensure LVM changes are correctly synced across the cluster.

Manual updates to each node to synchronize the ODM records.Lazy update.C-SPOC synchronization of ODM records.C-SPOC LVM operations - cluster enabled equivalents of the standard SMIT LVM functions. RSCT for Enhanced Concurrent Volume Groups

VGDA = ODM


LVM Changes, Manual To perform manual changes the Volume Group must be varied onto one of the nodes.

1. Make necessary changes to the volume group or filesystem.2. Unmount filesystems and varyoff the vg.

On all the other nodes that share the volume group. 1. Export the volume group from the ODM. 2. Import the information from the VGDA. 3. Change the auto vary on flag. 4. Correct the permissions and ownership's on the logical volumes as required.5. Repeat to all other nodes.

##importvg -V123 -L sharedvg hdisk3 #chvg -an sharedvg#varyoffvg sharedvg

#chfs -a size=+8192 /sharedfs #unmount /sharedfs#varyoffvg sharedvg


LVM Changes, Lazy Update At fallover time, lazy update compares the time stamp value in theVGDA with one stored in /usr/sbin/cluster/etc/vg/<vgname>. If the time stamps are the same, then the varyonvg proceeds. If the timestamps do not agree, then HACMP does the export/import cycle similar to a manual update.

NOTE: HACMP does change the VG auto vary on flag AND it preserves permissions and ownership of the logical volumes.

112

23

4567

89

1011

VGDA timestamp

112

23

4567

89

1011

timestamp in /usr/sbin/cluster/etc/vg/<vgname>


LVM Changes, C-SPOC SynchronizationManually make your change to the LVM on one node.Use C-SPOC to propagate the changes to all nodes in the resource group.

This is only available for volume groups that are inactive everywhere in the cluster (not VARYed on)Downtime is experienced for the volume group.

C-SPOC updates ODM and the time stamp file

update vg constructs use C-SPOC syncvg


The Best Method: C-SPOC LVM Changes

Journaled File Systems


Add a Journaled File System Add a Journaled File System on a Previously Defined Logical Volume List All Shared File Systems Change / Show Characteristics of a Shared File System Remove a Shared File System



LVM Changes, Select Your Filesystem

Journaled File Systems


Add a Journaled File System Add a Journaled File System on a Previously Defined Logical Volume List All Shared File Systems Change / Show Characteristics of a Shared File System Remove a Shared File System

+--------------------------------------------------------------------------+ ¦ File System Name ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦ ¦ # Resource Group File System ¦ ¦ adventure /norbert ¦ ¦ discovery /ron ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F8=Image F10=Exit Enter=Do ¦F1¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+


Update the Size of a Filesystem

Change/Show Characteristics of a Shared File System in the Cluster


[Entry Fields] Resource Group Name discovery File system name /ron NEW mount point [/ron] SIZE of file system [4000000] Mount GROUP [] Mount AUTOMATICALLY at system restart? no + PERMISSIONS read/write + Mount OPTIONS [] + Start Disk Accounting? no + Fragment Size (bytes) 4096 Number of bytes per inode 4096 Compression algorithm no



HACMP Resource Group Operations

HACMP Resource Group and Application Management


Bring a Resource Group Online Bring a Resource Group Offline Move a Resource Group to Another Node

Suspend/Resume Application Monitoring Application Availability Analysis



Priority Override Location (POL)Assigned during a resource group move operation.

The destination node for a resource group online, offline or move request becomes the resource group's POL.

Remains in effect until:A move to "Restore_Node_Priority_Order" is doneCluster is restarted (unless option chosen to persist)

HACMP 4.x has the notion of a sticky location which is similar to the notion of a persistent POL.Can be viewed with the command:

/usr/es/sbin/cluster/utilities/clRGinfo -p

*This foil describes how priority override locations work for nonconcurrent resource groups. See the HACMP 5.2 Administration and Troubleshooting Guide (SC-23-4862-03) for information on how priority override locations work for concurrent access resource groups


Moving a Resource Group

Move a Resource Group


[Entry Fields] Resource Group to be Moved adventure Destination Node hudson Persist Across Cluster Reboot? false +



Selecting Destination

HACMP Resource Group and Application Management

Move cursor to desired item and press Enter. lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x Select a Destination Node x x x x Move cursor to desired item and press Enter. Use arrow keys to scroll. x x x x # To choose the highest priority available node for the x x # resource group, and to remove any Priority Override Location x x # that is set for the resource group, select x x # "Restore_Node_Priority_Order" below. x

x Restore_Node_Priority_Order x

x x x # To choose a specific node, select one below. x x # x x # Node Site x x # x

x halifax x x x x F1=Help F2=Refresh F3=Cancel x x Esc+8=Image Esc+0=Exit Enter=Do xF1x /=Find n=Find Next xEsmqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj


Taking a Resource Group Offline

Bring a Resource Group Offline


[Entry Fields] Resource Group to Bring Offline adventure Node On Which to Bring Resource Group Offline bondar Persist Across Cluster Reboot? false +



Bring a Resource Group Back Online

Bring a Resource Group Online


[Entry Fields] Resource Group to Bring Online adventure Destination Node bondar Persist Across Cluster Reboot? false +



Log Files Generated by HACMP/usr/es/adm/cluster.log "High level view" of cluster activity.

/usr/es/sbin/cluster/history/cluster.mmddyyyy Cluster history files generated daily./tmp/cspoc.log Generated by C-SPOC commands./tmp/dms_loads.out Stores log messages every time HACMP loads

the deadman switch kernal extension./var/hacmp/clverify/clverify.log Contains verbose messages from clverify

(cluster verification utility)./tmp/emuhacmp.out Output of emulated events./tmp/hacmp.out /tmp/hacmp.out.<1-7> Output of today's HACMP event scripts.AIX error log All sorts of stuff!/var/ha/log/topsvcs Tracks execution of topology services daemon./var/ha/log/grpsvcs Tracks execution of group services daemon./var/ha/log/grpglsm Tracks execution of grpglsm daemon./tmp/clstrmgr.debug Tracks internal execution of the cluster

manager./var/hacmp/clcomd/clcomd.log Tracks activity of clcomd./var/hacmp/clcomd/clcomddiag.log Tracks more detailed activity of clcomd when

tracing is turned on./var/adm/clavan.log Output of application availability analysis tool.HACMP 5.2 /var/hacmp/log/ clconfigassist.log clutils.log cl_testtool.log

-Two-Node Cluster Configuration Assistant-Generated by utilities and file propagation -Generated by test tool



Using C-SPOC reduces the likelihood of an outage by reducing the likelihood that you will make a mistake.

2. True or False?C-SPOC reduces the need for a change management process.

3. C-SPOC cannot do which of the following administration tasks?a. Add a user to the cluster.b. Change the size of a filesystem.c. Add a physical disks to the cluster.d. Add a shared volume groups to the cluster.e. Synchronize existing passwords.f. None of the above.

4. True or False? It does not matter which node in the cluster is used to initiate a C-SPOC operation.

5. Which log file provides detailed output on HACMP event script execution?a. /tmp/clstrmgr.debugb. /tmp/hacmp.outc. /var/adm/cluster.log



Using C-SPOC reduces the likelihood of an outage by reducing the likelihood that you will make a mistake.

2. True or False?C-SPOC reduces the need for a change management process.

3. C-SPOC cannot do which of the following administration tasks?a. Add a user to the cluster.b. Change the size of a filesystem.c. Add a physical disks to the cluster.d. Add a shared volume group to the cluster.e. Synchronize existing passwords.f. None of the above (e was the correct answer in previous releases)

4. True or False? It does not matter which node in the cluster is used to initiate a C-SPOC operation.

5. Which log file provides detailed output on HACMP event script execution?a. /tmp/clstrmgr.debugb. /tmp/hacmp.outc. /var/adm/cluster.log


Unit Summary

Having completed this unit, you should be able to: Understand the need for change management when using HACMPUnderstand the benefits and capabilities of C-SPOCPerform routine administrative changes using C-SPOC

LVMUsersDisk

Perform RGmove operations


Welcome to:

3.13.0.3

Dynamic ReconfigurationDynamic Reconfiguration


Unit Objectives

After completing this unit, you should be able to: Understand Dynamic Automatic Reconfiguration Events (DARE)Make changes to cluster topology and resources in an active clusterUse snapshot to change a cluster configuration


HACMP provides a facility that allows changes to clustertopology and resources to be made while the cluster is active. This facility is known as DARE or to give it it's full name Dynamic Automatic Reconfiguration Event. This requires three copies of the HACMP ODM.

Dynamic Reconfiguration

DCD

SCD

ACD

Default Configuration Directorywhich is updated by SMIT/command line/etc/objrepos

Staging Configuration Directorywhich is used during reconfiguration/usr/es/sbin/cluster/etc/objrepos/staging

Active Configuration Directory from whichclstrmgr reads the cluster configuration/usr/es/sbin/cluster/etc/objrepos/active

rootvg


DARE allows changes to be made to most cluster topologyand nearly all resource group components without the need to stop HACMP, take the application offline or reboot a node. All changes must be synchronized in order to take effect.Here are some examples of the tasks that DARE can complete for Topology and Resources without having to bring HACMP down.

What Can DARE Do?

Topology ChangesAdding or removing cluster nodesAdding or removing networksAdding or removing communication interfaces or devicesSwapping a communication interface's IP addressResource ChangesAll resources can be changed


What Limitations Does DARE Have?DARE cannot change all cluster topology and resource group components without the need to stop HACMP, take the application offline or reboot a node.Here are some examples that require a stop and restart of HACMP for the change to be made.

Topology ChangesChange the name of the clusterChange the cluster ID*Change the name of a cluster nodeChange a communication interface attributeChanging whether or not a network uses IPAT via IP aliasing or via IP replacementChange the name of a network module*Add a network interface module*Removing a network interface module*Resource ChangesChange the name of a resource groupChange the name of an application serverChange the node relationship

DARE cannot run if two nodes are not at the same HACMP level


DARE uses the three separate copies of the ODM in order toallow changes to be propagated to all nodes whilst the cluster is active.

So How Does DARE Work?

SCDDCDSCD

ACD

change topologyor resources in smit

synchronize topologyor resources in smit

1 2 4cluster manager readsACD and refreshes

3

ACD

snapshot taken of the current ACD

5

SCD

SCD is deleted

SCD

HACMP


Cluster Configuration Cluster Services Cluster System Management Cluster Recovery Aids RAS Support


HACMP


Cluster Configuration Cluster Services Cluster System Management Cluster Recovery Aids RASfdsfsfsafsafsfsfsafsfdsafdsafdsafdsfsdafsdadafsdafsdf Support



Verifying and Synchronizing (Standard)



Two-Node Cluster Configuration Assistant Add Nodes to an HACMP Cluster Configure Resources to Make Highly Available Configure HACMP Resource Groups Verify and Synchronize HACMP Configuration Display HACMP Configuration



Verifying and Synchronizing (Extended)

HACMP Verification and Synchronization (Active Cluster on a Local Node)


(When NODE DOWN -- HACMP 5.2) [Entry Fields] * Verify, Synchronize or Both [Both] +

* Automatically correct errors found during [No] +

verification?


(When NODE UP)* Emulate or Actual [Actual] + Force synchronization if verification fails? [No] +* Verify changes only? [No] +* Logging [Standard] +



Discarding Unwanted Changes

Problem Determination Tools


HACMP Verification View Current State HACMP Log Viewing and Management Recover From HACMP Script Failure

Restore HACMP Configuration Database from Active Configuration Release Locks Set By Dynamic Reconfiguration Clear SSA Disk Fence Registers HACMP Cluster Test Tool HACMP Trace Facility HACMP Event Emulation HACMP Error Notification




Rolling Back from a DARE Operation

Apply a Cluster Snapshot


[Entry Fields] Cluster Snapshot Name jami Cluster Snapshot Description Cuz -- he did the lab> Un/Configure Cluster Resources? [Yes] + Force apply if verify fails? [No] +



If a dynamic reconfiguration should fail due to an unexpectedcluster event, then the staging configuration directory might still exist. This prevents further changes being made to the cluster.

What If DARE Fails?

SCDDCDSCD

ACD

change topologyor resources in smit

synchronize topologyor resources in smit

1 2 4cluster manager readsACD and refreshes

3

ACD

snapshot taken of the current ACD

5

SCD

SCD is deleted

SCD

HACMP


Cluster Configuration Cluster Services Cluster System Management Cluster Recovery Aids RAS Support


HACMP


Cluster Configuration Cluster Services Cluster System Management Cluster Recovery Aids RASfdsfsfsafsafsfsfsafsfdsafdsafdsafdsfsdafsdadafsdafsdf Support


BANG!


Dynamic Reconfiguration Lock



HACMP Verification View Current State HACMP Log Viewing and Management Recover From HACMP Script Failure Restore HACMP Configuration Database from Active Configuration

Release Locks Set By Dynamic Reconfiguration Clear SSA Disk Fence Registers HACMP Cluster Test Tool HACMP Trace Facility HACMP Event Emulation HACMP Error Notification





DARE operations can be performed while the cluster is running.

2. Which operations can DARE not perform (select all that apply)?a. Changing the name of the cluster.b. Removing a node from the cluster.c. Changing a resource in a resource group.d. Change whether a network uses IPAT via IP aliasing or via IP replacement.

3. True or False?It is possible to roll back from a successful DARE operation using an automatically generated snapshot.

4. True or False?Running a DARE operation requires three separate copies of the HACMP ODM.

5. True or False?Cluster snapshots can be applied while the cluster is running.

6. What is the purpose of the dynamic reconfiguration lock?a. To prevent unauthorized access to DARE functions.b. To prevent further changes being made until a DARE operation has completed.c. To keep a copy of the previous configuration for easy rollback.


Unit Summary

Having completed this unit, you should be able to: Understand Dynamic Automatic Reconfiguration Events (DARE)Make changes to cluster topology and resources in an active clusterUse snapshot to change a cluster configuration


Welcome to:

3.13.0.3

Integrating NFS into HACMPIntegrating NFS into HACMP


Unit Objectives

After completing this unit, you should be able to: Explain the concepts of Network File System (NFS)Configure HACMP to support NFSUnderstand why Volume Group major numbers must be unique when using NFS with HACMPOutline the NFS configuration parameters for HACMP


So, What Is NFS?

NFS Server

The Network File System (NFS) is a client/server application that lets a computer user view and optionally store and update files on a remote computer as though they were on the user's own computer.

JFS mount

shared_vg

NFS Client and Server

NFS Client

NFS mount

NFS mount

NFS mount

read-only

read-write

read-only


NFS Background Processes

NFS Server

NFS uses TCP/IP and a number of background processes to allow clients to access disk resource on a remote server.Configuration files are used on the client and server to specify export and mount options.

NFS Client and Servern x biodn x nfsd and mountd

NFS Client

n x biod

n x nfsd and mountd

/etc/exports/etc/filesystems


Combining NFS with HACMPNFS exports can be made highly available by using the HACMP resource group to specify NFS exports and mounts.

A

export /fsa

client system# mount aservice:/fsa /a

Bondar Hudson

# mount /fsa

/fsa

aservice

client system sees /fsa as /aThe A resource group specifies:aservice as a service IP label resource/fsa as a filesystem resource/fsa as a NFS filesystem to export


NFS Fallover with HACMPIn this scenario, the resource group moves to the surviving node in the cluster, which exports /fsa. Clients see NFS server not responding during fallover.

A

client system

Bondar Hudson

# mount /fsa

/fsa

client system "sees" /fsa as /aThe A resource group specifies:aservice as a service IP label resource/fsa as a filesystem resource/fsa as a NFS filesystem to export

# mount aservice:/fsa /a

export /fsaaservice


Configuring NFS for High Availability



[MORE...10] [Entry Fields]

Volume Groups [aaavg] + Use forced varyon of volume groups, if necessary false + Automatically Import Volume Groups false +

Filesystems (empty is ALL for VGs specified) [/fsa] + Filesystems Consistency Check fsck + Filesystems Recovery Method sequential +

Filesystems mounted before IP configured true +

Filesystems/Directories to Export [/fsa] +

Filesystems/Directories to NFS Mount [] + Network For NFS Mount [] +

[MORE...10]



Cross-mounting NFS Filesystems (1 of 3)A filesystem configured in a resource group can be madeavailable to all the nodes in the resource group:

One node has the resource group and acts as an NFS serverMounts the filesystem (/fsa)Exports the filesystem (/fsa)

All nodes act as NFS clientsMount the NFS filesystem (aservice:/fsa) onto a local mount point (/a)

acts as an NFS clientacts as an NFS server


(exports /fsa)

/a /a/fsa

aservice


Cross-mounting NFS Filesystems (2 of 3)

acts as an NFS client

acts as an NFS server


(exports /fsa)

/a /a/fsa

When a fallover occurs, the role of NFS server moves with the resource group.All (surviving) nodes continue to be NFS clients.

aservice


Here's a more detailed look at what is going on:

Cross-mounting NFS Filesystems (3 of 3)

A

client system

Bondar Hudson

# mount /fsa


/fsa

client system "sees" /fsa as /a

aservice



The A resource group specifies:aservice as a service IP label resource/fsa as a filesystem resource/fsa as a NFS filesystem to export/fsa as a NFS filesystem to mount on /a

export /fsa


In a cluster with multiple IP networks, it may be useful to specify which network should be used by HACMP for cross-mounts.This is usually done as a performance enhancement.

Choosing the Network for Cross-mounts

A

export /fsa

Bondar Hudson

# mount /fsa# mount aservice:/fsa /a

/fsa

aservice


aGservice

net_ether_02

net_ether_01

The A resource group specifies:aservice as a service IP label resource/fsa as a filesystem resource/fsa as a NFS filesystem to export/fsa as a NFS filesystem to mount on /anet_ether_01 is the network for NFS mounts


Configuring HACMP for Cross-mounting



[MORE...10] [Entry Fields]

Volume Groups [aaavg] + Use forced varyon of volume groups, if necessary false + Automatically Import Volume Groups false +

Filesystems (empty is ALL for VGs specified) [/fsa] + Filesystems Consistency Check fsck + Filesystems Recovery Method sequential + Filesystems mounted before IP configured true + Filesystems/Directories to Export [/fsa] +

Filesystems/Directories to NFS Mount [/a;/fsa] +

Network For NFS Mount [net_ether_01] +

[MORE...10]



Syntax for Specifying Cross-mounts


Where the filesystem should be mounted over

What the filesystem is exported as

/a;/fsa

What HACMP does(on each node in the resource group)


Ensuring the VG Major Number Is UniqueAny Volume Group which contains a filesystem that isoffered for NFS export to clients or other cluster nodes mustuse the same VG major number on every node in the cluster.

To display the current VG major numbers, use:# ls -l /dev/*webvgcrw-rw---- 1 root system 201, 0 Sep 04 23:23 /dev/xwebvgcrw-rw---- 1 root system 203, 0 Sep 05 18:27 /dev/ywebvgcrw-rw---- 1 root system 205, 0 Sep 05 23:31 /dev/zwebvg

The command 'lvlstmajor' will list the available major numbers for each node in the cluster. For example:

# lvlstmajor43,45...99,101...

The VG major number may be set at the time of creating the VG using smit mkvg or by using the -V flag on the importvg command. For example:

# importvg -V100 -y shared_vg_a hdisk2

C-SPOC will "suggest" a VG major number which is unique across the nodes when it is used to create a shared volume group.


NFS with HACMP Considerations

1 Resource groups which export NFS filesystems MUST implement IPAT.

2 The Filesystems mounted before IP configured resource group attribute must be set to true.

3 HACMP does not use /etc/exports and the default is to export filesystems rw to the world. Specify NFS export options in /usr/es/sbin/cluster/etc/exports if you want better control (AIX 5.2 provides an option to specify this path)

4 HACMP only preserves NFS locks if the NFS exporting resource group has no more than two nodes.

Some points to note...



HACMP supports all NFS export configuration options.

2. Which of the following is a special consideration when using HACMP to NFS export filesystems? (select all that apply)

a. NFS exports must be read-write.b. Secure RPC must be used at all times.c. A cluster may not use NFS Cross-mounts if there are client systems accessing the NFS

exported filesystems.d. A volume group which contains filesystems which are NFS exported must have the

same major device number on all cluster nodes in the resource group.

3. What does [/abc;/xyz] mean when specifying a directory to cross-mount?a. /abc is the name of the filesystem which is exported and /xyz is where it should be

mounted atb. /abc is where the filesystem should be mounted at and /xyz is the name of the filesystem

which is exported

4. True or False?HACMP's NFS exporting feature only supports clusters of two nodes.

5. True or False?IPAT is required in resource groups which export NFS filesystems.


Checkpoint Answers1. True or False?*

HACMP supports all NFS export configuration options.

2. Which of the following is a special consideration when using HACMP to NFS export filesystems? (select all that apply)

a. NFS exports must be read-write.b. Secure RPC must be used at all times.c. A cluster may not use NFS Cross-mounts if there are client systems accessing the NFS

exported filesystems.d. A volume group which contains filesystems which are NFS exported must have the

same major device number on all cluster nodes in the resource group.

3. What does [/abc;/xyz] mean when specifying a directory to cross-mount?a. /abc is the name of the filesystem which is exported and /xyz is where it should be

mounted atb. /abc is where the filesystem should be mounted at locally and /xyz is the name of the

filesystem which is exported

4. True or False?**HACMP's NFS exporting feature only supports resource groups with two nodes.

5. True or False?IPAT is required in resource groups which export NFS filesystems.

*/usr/es/sbin/cluster/exports must be used to specify NFS export options if the default of "read-write to the world" is not acceptable.**Resource groups larger than two nodes which export NFS filesystems do not provide full NFS functionality (for example, NFS file locks are not preserved across a fallover).


Unit Summary

Having completed this unit, you should be able to: Explain the concepts of Network File System (NFS)Configure HACMP to support NFSUnderstand why Volume Group major numbers must be unique when using NFS with HACMPOutline the NFS configuration parameters for HACMP


Welcome to:

3.13.0.3

Cluster CustomizationCluster Customization


Unit Objectives

After completing this unit, you should be able to: Understand the requirements for application server start and stop scriptsPerform basic cluster customizationsChange HACMP tuning parametersMonitor other devices outside the control of HACMP


HACMP is not an out of the box solution to availability.All clusters require some degree of customization. Here are some examples of the customization you may need to perform:

Create application start and stop scripts.Create pre- and post-event scripts.Tune AIX for increased availability.Extend your snapshot report to document application specific information.Extend cluster verification to test things beyond HA configuration.Configure event notification to monitor devices beyond the control of HACMP.

What Customization Is Necessary?


All application start and stop scripts must meet the followingbasic requirements:

Start and stop scripts should have all required environment variables set.The scripts must be present on all nodes used by the application server.Start scripts must be able to handle abnormal termination.

If startup fails, the scripts should not leave the cluster in an unstable state.

Start scripts must check for any dependent processes.Start scripts must be able to start any required dependent processes

Scripts must declare shell on first line (ie #!/bin/ksh)Notes: 1. Start and Stop scripts do not have to contain the same commands on all nodes, thus allowing for different application start up and shutdown sequences on a node by node basis.2. HACMP 5.2 provides a file collection facility that can be used to keep your Start and Stop scripts in sync across the nodes in the resource group.

Application Start and Stop Scripts


Pre- / Post-Events and Notify CommandsHACMP allows a pre- and post-event script to be defined for each of the HACMP event scripts. These execute immediately before (pre) and after (post) the HACMP event.

Event Manager

HACMP Eventclcallev

ODMHACMP-Classes

Notify Command

Notify Command

Pre-Event Script (1)


Pre-Event Script (n)

RC=0 Counter>0

No

Yes

Yes

Post-Event Script (1) Post-Event Script (n)

Boom!


Extended Event Configuration


Configure Pre/Post-Event Commands Change/Show Pre-Defined HACMP Events Configure User-Defined Events Configure Pager Notification Methods Change/Show Time Until Warning


Adding/Changing Cluster Events (1 of 3)


Add a Custom Cluster Event


[Entry Fields]* Cluster Event Name [stop_printq]* Cluster Event Description [stop the print queues]* Cluster Event Script Filename [/usr/local/cluster/events/stop_printq]




Change/Show Cluster Events


[Entry Fields]

Event Name node_down_local_complete

Description Script run after the >

* Event Command [/usr/es/sbin/cluster/>

Notify Command [] Pre-event Command [] + Post-event Command [stop_printq] + Recovery Command []* Recovery Counter [0] #




Recovery CommandsIf an event script should fail to exit 0, Recovery commands canbe executed if an event script does not exit 0.


RC=0 Counter>0

No

Yes

Yes


Change/Show Cluster Events


[Entry Fields]

Event Name start_server

Description Script run to start a>

* Event Command [/usr/es/sbin/cluster/>

Notify Command [] Pre-event Command [] + Post-event Command [] + Recovery Command [/usr/local/bin/recover]* Recovery Counter [3] #


Adding/Changing Recovery Commands


The HACMP 5.2 EventsPrimary events (called by clstmgrES recovery programs)

site_up, site_up_completesite_down, site_down_completesite_merge, site_merge_completenode_up, node_up_completenode_down, node_down_completenetwork_up, network_up_completenetwork_down, network_down_completeswap_adapter, swap_adapter_completeswap_address, swap_address_completefail_standbyjoin_standbyfail_interfacejoin_interfacerg_move, rg_move_completerg_onlinerg_offlineevent_errorconfig_too_longreconfig_topology_startreconfig_topology_completereconfig_resource_releasereconfig_resource_acquirereconfig_resource_completereconfig_configuration_dependency_acquirereconfig_configuration_dependency_completereconfig_configuration_dependency_releasenode_up_dependencynode_up_dependency_completenode_down_dependencynode_down_dependency_completemigrate, migrate_complete

Secondary events(called by other events)

node_up_localnode_up_remotenode_down_localnode_down_remotenode_up_local_completenode_up_remote_completenode_down_local_completenode_down_remote_completeacquire_aconn_serviceacquire_service_addracquire_takeover_addrstart_serverstop_serverget_disk_vg_fsget_aconn_rsrelease_service_addrrelease_takeover_addrrelease_vg_fsrelease_aconn_rsswap_aconn_protocolsreleasingacquiringrg_uprg_downrg_errorrg_temp_error_staterg_acquiring_secondaryrg_up_secondaryrg_error_secondaryresume_appmonsuspend_appmon


Points to NoteThe execute bit must be set on all pre-, post-, notify andrecovery scripts.Synchronization does not copy pre- and post-event script content from one node to another. You need to copy all your pre- and post-event scripts to all nodes.Your pre- and post-event scripts must handle non-zero exit codes.All scripts must have a header like:

#!/bin/ksh

Test your changes very carefully as a mistake is likely to cause a fallover to abort.


Editing an HACMP Event Script (1 of 2)It is not recommended that you modify an HACMP event script.If you do, please note the following:

All HACMP event scripts are written in the Korn Shell.All scripts are located in /usr/es/sbin/cluster/events.HACMP event scripts are VERY complex as they must operate in a wide variety of circumstances.

Be particularly careful about the event emulation mechanismDo not interfere with itMake sure your changes emulate it or do it as required

Consider changing the location of the edited event script as this prevents the modified script from being overwritten by an HACMP patch

Refer to Change/Show Cluster Event screen a few foils back


Editing an HACMP Event Script (2 of 2)When changing an HACMP event script :

1. Copy the source event script to a different directory.2. Edit the Event Script path in the "Change/Show Cluster Events"

HACMP smit panel.3. Ideally, put any new code into separate script which is called from

within the HACMP event, rather than edit the HACMP event script directly.

4. Thoroughly document any changes that you make to the HACMP event script.

5. Thoroughly test the HACMP event script behavior in all fallover scenarios.


Performance Tuning HACMP

Extended Performance Tuning Parameters Configuration


Change/Show I/O pacing Change/Show syncd frequency



Enabling I/O Pacing

Change/Show I/O pacing


[Entry Fields] HIGH water mark for pending write I/Os per file [33] +# LOW water mark for pending write I/Os per file [24] +#


The HACMP documentation recommends a high water mark of 33 and a low water mark of 24.


Changing the Frequency of syncd

Change/Show syncd frequency


[Entry Fields] syncd frequency (in seconds) [10] #


The HACMP documentation recommends a value of 10.


Other HACMP CustomizationsDynamic node priorityCustom disk methodsCustom snapshot methodsCustom verification methodsApplication monitoringApplication availability analysis toolFile CollectionConfigure Pager Notification MethodsChange/Show Time Until WarningError notification


Extending Protection to Other DevicesHACMP provides smit screens for managing the AIX errorlogging facility's error notification mechanism.

Disk adapters

CPU

Disk subsystems

Disks

Other shared devices


Error Notification within smit

HACMP Error Notification


Configure Automatic Error Notification Add a Notify Method Change/Show a Notify Method Remove a Notify Method Emulate Error Log Entry



Configuring Error Notification



Configure Automatic Error Notification Add a Notify Method Change/Show a Notify Method Remove a Notify Method Emulate Error Log Entry



Configuring Automatic Error Notification

Configure Automatic Error Notification


List Error Notify Methods for Cluster Resources Add Error Notify Methods for Cluster Resources Remove Error Notify Methods for Cluster Resources



Listing Automatic Error Notification

COMMAND STATUS

Command: OK stdout: yes stderr: no

Before command completion, additional instructions may appear below.

[TOP]bondar:bondar: HACMP Resource Error Notify Methodbondar:bondar: hdisk0 /usr/es/sbin/cluster/diag/cl_failoverbondar: scsi0 /usr/es/sbin/cluster/diag/cl_failoverbondar: hdisk11 /usr/es/sbin/cluster/diag/cl_logerrorbondar: hdisk5 /usr/es/sbin/cluster/diag/cl_logerrorbondar: hdisk9 /usr/es/sbin/cluster/diag/cl_logerrorbondar: hdisk7 /usr/es/sbin/cluster/diag/cl_logerrorbondar: ssa0 /usr/es/sbin/cluster/diag/cl_logerrorhudson:hudson: HACMP Resource Error Notify Method[MORE...9]

F1=Help F2=Refresh F3=Cancel F6=CommandF8=Image F9=Shell F10=Exit /=Findn=Find Next


Adding Error Notification Methods

Add a Notify Method


[Entry Fields]* Notification Object Name []* Persist across system restart? No + Process ID for use by Notify Method [] +# Select Error Class None + Select Error Type None + Match Alertable errors? None + Select Error Label [] + Resource Name [All] + Resource Class [All] + Resource Type [All] +* Notify Method []



Emulating Errors (1 of 2)


Mo+--------------------------------------------------------------------------+ ¦ Error Label to Emulate ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦ ¦ [TOP] ¦ ¦ SSA_DISK_ERR3 SSA_DISK_DET_ER ¦ ¦ LVM_SA_QUORCLOSE bernhardvg ¦ ¦ LVM_SA_QUORCLOSE xwebvg ¦ ¦ LVM_SA_QUORCLOSE rootvg ¦ ¦ SERVICE_EVENT diagela_SE ¦ ¦ FCP_ARRAY_ERR6 fcparray_err ¦ ¦ DISK_ARRAY_ERR2 ha_hdisk0_0 ¦ ¦ DISK_ARRAY_ERR3 ha_hdisk0_1 ¦ ¦ DISK_ARRAY_ERR5 ha_hdisk0_2 ¦ ¦ DISK_ERR2 ha_hdisk0_3 ¦ ¦ [MORE...39] ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F8=Image F10=Exit Enter=Do ¦F1¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+


Emulating Errors (2 of 2)

Emulate Error Log Entry


[Entry Fields] Error Label Name LVM_SA_QUORCLOSE Notification Object Name xwebvg Notify Method /usr/es/sbin/cluster/>



What Will This Cause?# errpt -a---------------------------------------------------------------------------LABEL: LVM_SA_QUORCLOSEIDENTIFIER: CAD234BE

Date/Time: Fri Sep 19 13:58:05 MDTSequence Number: 469Machine Id: 000841564C00Node Id: bondarClass: HType: UNKNResource Name: LVDDResource Class: NONEResource Type: NONELocation:

DescriptionQUORUM LOST, VOLUME GROUP CLOSING

Probable CausesPHYSICAL VOLUME UNAVAILABLE

Detail DataMAJOR/MINOR DEVICE NUMBER00C9 0000QUORUM COUNT 0ACTIVE COUNT 0SENSE DATA0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000---------------------------------------------------------------------------

... and a fallover of the discovery resource group to hudson.


Selective Fallover is an automatically launched function ofHACMP which attempts to selectively move only the resource group that has been affected by an individual resource failure to another node in the cluster, rather than moving all resource groups.Selective Fallover allows you to selectively provide recovery for individual resource groups that are affected by failures of specific resources.Selective fallover caters to the following failures:

Service IP labelsNetwork Interface FailuresLocal Network Failures

ApplicationsCommunication LinksVolume groups

Selective Fallover for Resource Groups



HACMP event scripts are binary executables and cannot be easily modified.

2. Which of the following runs if an HACMP event script fails? (select all that apply)

a. Pre-event scripts.b. Post-event scripts.c. error notification methods.d. recovery commands.e. notify methods.

3. What are the recommended values for I/O pacing high and low water marks?

a. 33,48b. 48,33c. 33,24d. 24,33

4. True or False?All clusters must be tuned for high availability.

5. True or False?Writing error notification methods is a normal part of configuring a cluster.



HACMP event scripts are binary executables and cannot be easily modified.

2. Which of the following runs if an HACMP event script fails? (select all that apply)

a. Pre-event scripts.b. Post-event scripts.c. error notification methods.d. recovery commands.e. notify methods.

3. What are the recommended values for I/O pacing high and low water marks?

a. 33,48b. 48,33c. 33,24d. 24,33

4. True or False? *All clusters must be tuned for high availability.

5. True or False?Writing error notification methods is a normal part of configuring a cluster.

*The HACMP documentation recommends that you tune the I/O pacing and syncd parameters. You may experience "difficulties" getting support until you do this.


Unit Summary

Having completed this unit, you should be able to: Understand the requirements for application server start and stop scriptsPerform basic cluster customizationsChange HACMP tuning parametersMonitor other devices outside the control of HACMP


Welcome to:

3.13.0.3

Problem DeterminationProblem Determinationand Recoveryand Recovery


Unit Objectives

After completing this unit, you should be able to: Understand why HACMP can failIdentify configuration and administration errorsUnderstand why the Dead Man's Switch invokesKnow when the System Resource Controller kills a nodeIsolate and recover from failed event scriptsCorrectly escalate a problem to IBM support


Planned downtime:

Hardware upgradesRepairsSoftware updatesBackupsTestingDevelopment

Remember This?

High-availability solutions should reduce both planned and unplanned downtime.

Unplanned downtime:

Administrator Error - #1 cause!Application failureHardware faultsEnvironmental Disasters

1.0%14.0%

85.0%

Hardware Failure (1%)Other unplanned downtime (14%)Planned downtime (85%)

User error is the primary cause of unplanned downtime.


Common reasons why HACMP fails:A poor cluster design and lack of thorough planning.Basic TCP/IP and LVM configuration problems.HACMP cluster topology and resource configuration problems.Absence of change management discipline in a running cluster.Lack of training for staff administering the cluster.

Why Do Good Clusters Turn Bad?

Halifax

AXVancouver

BX


Test Your Cluster before Going Live!Careful testing of your production cluster before going live reduces the risk of problems later.An example test plan might include:

Test Item Checked

Node FalloverNetwork Adapter SwapIP Network FailureSSA Adapter FailureDisk FailureClstrmgr KilledSerial Network FailureSCSI Adapter for rootvg FailureApplication FailurePartitioned Cluster


Tools to Help You Diagnose a ProblemThe vast majority of problems that you encounter withHACMP are related to IP, LVM and cluster configuration errors.Automatic Cluster Configuration Monitoring %Automatic Error Correction during Verify %HACMP Cluster Test Tool %Emulation ToolsHACMP Administration and Troubleshooting manualHACMP log files

hacmp.out, cluster.log, clverify.log, clustrmgr.debugSimple AIX and HACMP commands:

df -k mount lsfs netstat -ino -a lsdev lsvg [<ecmvg>] lsvg -olslv lspv clshowres clfindresclRGinfo* cltopinfo* ifconfig

% HACMP 5.2 * HACMP 5.x


HACMP Verification


Verify HACMP Configuration Configure Custom Verification Method

Automatic Cluster Configuration Monitoring




[Entry Fields]* Automatic cluster configuration verification Enabled + Node name Default +* HOUR (00 - 23) [00] +#




[Entry Fields]* Verify, Synchronize or Both [Both] +

* Automatically correct errors found during [No] + verification?


F1=Help F2=Refresh F3=Cancel F4=ListEsc+5=Reset Esc+6=Command Esc+7=Edit

Automatic Error Correction During Verify


HACMP Cluster Test Tool


Execute Automated Test Procedure Execute Custom Test Procedure


HACMP Cluster Test Tool

THESE TESTS ARE DISRUPTIVE


HACMP provides tools to emulate common cluster events.Only certain events are emulated.Multiple events cannot be emulated.Each event runs in isolation; results do not impact upon the next emulated event.The results are logged in /tmp/emuhacmp.out.If an event fails when emulated, it's not going to work when it happens for real.

Event Emulation Tools

VancouverHalifax

A

B

Swap AdapterFailed/Joined Network

Failed/Joined Standby

Failed/Joined Node


Emulating a Network Down Event

Emulate Network Down Event


[Entry Fields]* Network Name [net_ether_01] + Node Name [] +



Emulating Common Events

HACMP Event Emulation


Node Up Event Node Down Event Network Up Event Network Down Event Fail Standby Event Join Standby Event Swap Adapter Event



Emulating a Node Down Event

Emulate Node Down Event


[Entry Fields]* Node Name [hudson] +* Node Down Mode graceful +



Checking Cluster Processes (1 of 2)

# lssrc -g clusterSubsystem Group PID Status clstrmgrES cluster 21032 active clsmuxpdES cluster 17196 active clinfoES cluster 21676 active

Cluster Components clinfo

clstrmgr

clsmuxpd

Mandatory

Mandatory Optional


Checking Cluster Processes (2 of 2)

#

# lssrc -g topsvcsSubsystem Group PID Status topsvcs topsvcs 12230 active# lssrc -g grpsvcsSubsystem Group PID Status grpsvcs grpsvcs 11736 active grpglsm grpsvcs 12742 active# lssrc -g emsvcsSubsystem Group PID Status emsvcs emsvcs 12934 active emaixos emsvcs 13184 active# lssrc -s clcomdESSubsystem Group PID Status clcomdES clcomdES 13420 active # lssrc -s ctrmcSubsystem Group PID Status ctrmc rsct 2954 active#

Check rsct, clcomd, rmc subsystems


Testing Your Network ConnectionsTo test your IP network:

Ping from service to service and standby to standbyCheck the entries in the routing table on each nodenetstat -rn

Use the host command to resolve each IP label to its addressFor example, host vancouver_service

Use netstat -i and ifconfig to check addresses and mask

To test your non-IP networks:RS232

On one node type stty < /dev/tty#This will hang at the command line

Move to the other node which shares the RS232 cable and type stty < /dev/tty#This causes the tty settings to be displayed on both nodes

Target mode SSA network:On one node which shares the TMSSA network type cat < /dev/tmssa#.tm

The value of # is the node ID of the target (or receiving) SSA logical routerOn the other node type echo test > /dev/tmssa#.im

The value of # in this case is the node ID of the source (or sending) SSA logical router

Heartbeat over disk:On one node /usr/sbin/rsct/bin/dhb_read -p hdiskx -r (receive is done first)On another node /usr/sbin/rsct/bin/dhb_read -p hdiskx -t

Do not perform these tests while HACMP is running.


Dead Man's Switch (DMS Timeout)If one of your cluster nodes crashes with an 888 LED code,then you may have experienced a DMS timeout.

Under what circumstances will a DMS timeout occur?Excessive I/O traffic caused the clstrmgr to be starved of CPU.

Proving that DMS timeout crashed a node.Copy the system dump to a file.Run kdb on the dump file.Run the stat subcommand and look for 'HACMP dms timeout halting...'


Avoiding Dead Man's Switch TimeoutsTo avoid DMS timeout problems in the first place, carry out the

following in order:

1. Isolate the cause of excessive I/O traffic and fix it, and if that does not work...

2. Turn on I/O pacing, and if that does not work...3. Increase the frequency of the syncd, and if that does not work...4. Reduce the failure detection rate for the slowest network, and if that

does not work...5. Buy a bigger machine


SRC Halts a NodeIf one of your nodes halts for no apparent reason then youprobably need to change your root password.

Under what circumstances does the SRC halt a node?The cluster manager was killed or has crashed.

Proving that SRC halted a node:Check the AIX error log

Look for abnormal termination of clstrmgr daemonTo avoid SRC halts in the first place:

The cluster manager is not prone to abnormal termination so the most likely cause of SRC halts is an error made by a user of the root account.Don't give untrained staff access to the root password.


Partitioned Clusters and Node IsolationIf you get the message Diagnostics Group Shutting DownPartition then you have suffered either a partitioned cluster or node isolation.

Sent when a partitioned cluster or node isolation is detectedOccurs when heartbeats are received from a node that was diagnosed as failedAlso occurs when HACMP ODM configuration is not the same on a joining node as nodes already active in the clusterAlso occurs when two clusters with the same ID appear in the same logical networkA surviving node sends the DGSP message to the rogue recovering nodeThe rogue recovering or joining node is halted

Proving that DGSP caused a node to halt:Look in the /tmp/hacmp.out file


Avoiding Partitioned ClustersInstall and configure a non IP (serial) network in your cluster.Consider installing a second non-IP network

Partitioned clusters can lead to data divergenceYou do NOT want to experience data divergence

Check your non-IP networks before going liveDisable the non-IP network and verify that HACMP noticesReconnect the non-IP network and verify that HACMP notices

Watch for non-IP network failures in HACMP log filesDo not segment your cluster's IP networks

Avoid multiple switchesExcept in carefully designed highly available network configurations

Avoid bridges


Please check event status MessageWatch for the message

Cluster <clustername> has been running event <eventname> for # seconds. Please check event status.

It means that an event script has failed, hung or is taking too long. HACMP stops processing events until you resolve this issue


How Long is Too Long?Timeout value for fast events defaults to 3 minutes (180 seconds)Timeout value for slow events also defaults to 3 minutes (those which acquire/release resource groups) The timeout actually used for slow events is the SUM of fast + slowUse smit cm_time_before_warning to change these defaults


Changing the Timeouts

Change/Show Time Until Warning


[Entry Fields] Max. Event-only Duration (in seconds) [180] # Max. Resource Group Processing Time (in seconds) [180] #

Total time to process a Resource Group event 6 minutes and 0 secon> before a warning is displayed

NOTE: Changes made to this panel must be propagated to the other nodes by Verifying and Synchronizing the cluster



Recovering From an Event Script Failure1. Make a note of the time at which the message first appears

in the /usr/es/adm/cluster.log file.

2. Open the /tmp/hacmp.out file and move to the point in time recorded in step 1.

3. Work backwards through the /tmp/hacmp.out file until you find an AIX error message.

4. Go a little further back in the /tmp/hacmp.out file as the first message you encounter might not be the most important one.

5. Manually correct the problem so that you complete the event that failed.

6. Use cluster recovery aids to "Recover from Script Failure".7. Verify that the "Cluster <name> has been running ..." message is

no longer appearing.8. Verify that the cluster is now working properly.


Recovering From an Event Failure



HACMP Verification View Current State HACMP Log Viewing and Management Recover From HACMP Script Failure Restore HACMP Configuration Database from Active Configuration Release Locks Set By Dynamic Reconfiguration Clear SSA Disk Fence Registers HACMP Trace Facility +--------------------------------------------------------------------------+ ¦ Select a Node ¦ ¦ ¦ ¦ Move cursor to desired item and press Enter. ¦ ¦ ¦ ¦ bondar ¦ ¦ hudson ¦ ¦ ¦ ¦ F1=Help F2=Refresh F3=Cancel ¦ ¦ F8=Image F10=Exit Enter=Do ¦F1¦ /=Find n=Find Next ¦F9+--------------------------------------------------------------------------+


A Troubleshooting MethodologySave the log files from every available cluster node while theyare still availableAttempt to duplicate the problemApproach the problem methodicallyDistinguish between what you know and what you assumeKeep an open mindIsolate the problemGo from the simple to the complexMake one change at a timeStick to a few simple troubleshooting toolsDo not neglect the obviousWatch for what the cluster is not doingKeep a record of the tests you have completed


Contacting IBM for SupportBefore contacting IBM about a support issue, collect thefollowing information:

Item Checked

EXACT error messages that appear in HACMP logs or on the consoleYour cluster diagram (updated)A snapshot of your current cluster configuration (not a photo)Details of any customization performed to HACMP eventsDetails of current AIX, HACMP and application software levelsDetails of any PTFs applied to HACMP or AIX the clusterThe adapter microcode levels (especially for SSA adapters)Cluster planning worksheets, with all components clearly labeledA network topology diagram for the network as far as the usersCopies of all HACMP log files


Checkpoint1. What is the most common cause of cluster failure?

a. Bugs in AIX or HACMPb. Cluster administrator errorc. Marauding space aliens from another galaxyd. Cosmic rayse. Poor/inadequate cluster design

2. True or False?Event emulation can emulate all cluster events.

3. If the cluster manager process should die, what will happen to the cluster node?

a. It continues running but without HACMP to monitor and protect it.b. It continues running AIX but any resource groups will fallover.c. Nobody knows because this has never happened before.d. The System Resource Controller sends an e-mail to root and issue a "halt -q".e. The System Resource Controller sends an e-mail to root and issue a "shutdown -F".

4. True or False?A non-IP network is strongly recommended. Failure to include a non-IP network can cause the cluster to fail or malfunction in rather ugly ways.

5. (bonus question) my favorite graphic in the lower right hand corner of a foil was: ____________________________________


Checkpoint Answers1. What is the most common cause of cluster failure?

a. Bugs in AIX or HACMPb. Cluster administrator error*c. Marauding space aliens from another galaxyd. Cosmic rayse. Poor/inadequate cluster design*

2. True or False?Event emulation can emulate all cluster events.

3. If the cluster manager process should die, what will happen to the cluster node?

a. It continues running but without HACMP to monitor and protect it.b. It continues running AIX but any resource groups will fallover.c. Nobody knows because this has never happened before.d. The System Resource Controller sends an e-mail to root and issue a "halt -q".e. The System Resource Controller sends an e-mail to root and issue a "shutdown -F".

4. True or False?A non-IP network is strongly recommended. Failure to include a non-IP network can cause the cluster to fail or malfunction in rather ugly ways.

*The correct answer is almost certainly "cluster administrator error" although "poor/inadequate cluster design" would be a very close second.


Unit Summary

Having completed this unit, you should be able to: Understand why HACMP can failIdentify configuration and administration errorsUnderstand why the Dead Man's Switch invokesKnow when the System Resource Controller will kill a nodeIsolate and recover from failed event scriptsCorrectly escalate a problem to IBM support


Welcome to:

3.13.0.3

Documenting Your ClusterDocumenting Your Cluster


Unit Objectives

After completing this unit, you should be able to: Explain the importance of cluster planningDescribe the key cluster planning deliverables

Requirements documentDesign documentTest planDocumented operational procedures

Explain how the requirements, design and test plan documents should be linked togetherUse the export to planning worksheets feature of HACMP 5.2


Cluster PlanningThe purpose of cluster planning is to:

Ensure that the cluster implementer understands what the users want or need the cluster to doEnsure that the cluster does what the users want/need it to doEnsure that the cluster does what you intended it to do

The deliverables of the cluster planning process should be:A requirements document describing what the users want/need the cluster to doA design document describing how the cluster is configured to allow it to do what the users want or need it to doA test plan describing how to verify that the cluster Operational documentation


Describes what the users want or need the cluster to do:Applications and services that the cluster must provideUptime requirements

How many 9sTime-to-recovery requirements

Performance and resource requirements (as appropriate)Transactions per secondSimultaneous usersResponse timeData volumeNetwork bandwidth and response time requirementsRequirements today, in six months, in a year, and so forth

Software licensing requirementsAre applications node locked?Does vendor offer HA cluster discounts?

Budgetary goals, limits, expectations, and so forth

The Cluster Requirements Document


Describes how the cluster does what the users want or need the cluster to do:

How each highly available application is configuredHow the environment is configured

Power, cooling, physical firewalls, physical securityHow the hardware is configured

DiagramsHow AIX is configuredHow the storage is configured

DiagramsHow the network is configured

DiagramsHow HACMP is configuredHow the cluster notifies relevant people when a failure occursWho (job titles, not names) the relevant people arewho should be notifiedEven more diagrams

The Cluster Design Document (1 of 2)


Some additional considerations:Explain why decisions were madeExplain the cluster from the perspective of potential points of failure:

Which single points of failure (for example, extended loss of building power) are not addressed

Why they are not addressedHow, if known, they might be addressed

Which single points of failure are addressed by the cluster design and how

Explain which parts of the cluster design are rigid (difficult to change) and which parts are flexibleRelate design decisions back to cluster requirements

The Cluster Design Document (2 of 2)


How does the application recover automatically fromcrashes

Issues to watch for include:Startup passwords (for example, SSL certificates on Web servers)Applications which require GUI buttons to be clickedApplications which can not reliably recover from a wide range of crashes without human intervention

Location of:Configuration filesApplication binariesStatic dataDynamic data

License keysIf node locked, how does the node locking work?

In sufficient detail to be sure that the license keys won't be an obstacle after a fallover

Should be written in terms that both the application-support and the cluster-implementation or support folks can understand

Application Configuration Description


What hardware?Exact model numbers, features, adapter placement, and so forth.

Where will it go?Which rack and where in the rack? (Is there room and power?)

What will it be connected to?Cable distances, conduit routing, cable tray capacity requirements and availability, and so forth.Which ports?What kinds of cable? (length, connectors, model numbers, and so forth)

Detailed shared storage descriptionESS (or other bulk storage box) requirementsMirroring, RAIDBackup requirements, plans, procedures

Hardware Configuration Description


Version, patch level, and so forthRequired filesets (beyond the basic OS installation)Daemons/services

Which ones to runHow they are configured

If default configuration then say soWhich ones to turn off

Tuning parameters

AIX Configuration Description


For each volume group:Statement of purpose (why does it exist?)Name / size / physical locationConnectivity (which nodes need access to it?)

For each logical volume:Statement of purposeName / size / volume groupSpecial considerations (for example, high traffic volume)

For each filesystem:Statement of purposeMount point / size / type (JFS or JFS2)Parameters (for example, blocks per inode for JFS filesystems)LV name / JFS log LV nameMount sequencing

Don't forget about each node's rootvg requirements (mirroring, sizing, traffic volume, and such)

Storage Configuration Description


Which networks does the cluster connect to?How many IP addresses and switch ports you'll need

Switch and router configurationDisable auto-negotiation

Detailed network configuration for each network:IP addresses and labelsNetmasksDiagrams

How do the users get to the cluster?Any firewalls in the way?Should there be firewalls in the way?Are the links fast enough?

Network Configuration Description


Version, patch levelFilesets to be installedCluster topology

In detailDiagram and words

Cluster resourcesResources:

Service IP labels, filesystems, volume groups, application servers, and so forth

Resource groupsStatement of purposeTypeOptionsResources

Cluster parametersRG processing order, dynamic node priorities, WLM policies, fallback timer policies, startupsettling time policy, resource group dependencies, and so forth

HACMP Configuration Description


What do you test?For example, disk mirroring

How do you test it?For example, remove a physical volume (which one?)

In what context do you perform the test?For example, while the application is running on node X while doing Y with an intensity (load) of Z

How do you know if it worked?Log messages, HACMP falls over or swaps an adapter, and so forth

What it means if it fails?Is it mission-critical or can cluster run degraded for a while (how long)?What is likely to be wrong if it fails

Under what conditions the test should be performedIn the ideal world, you would run through the entire testplan after every cluster configuration change . . .

The Cluster Test Plan


What to record and who to call when something goes wrong?Names and phone numbers (kept up-to-date)Answers to questions that will be asked:

Cluster nameVendor support contract numberManager responsible

Escalation proceduresHow to recover from whatever is likely to go wrong (and what is unlikely to go wrong but delicate/critical if it does go wrong)

Detailed commands and optionsPut it in a shell script

smit screens paths (how to get there) and field valuesDetails (for example, exactly how to identify a failed disk drive and exactly how to replace it)

Backup procedures

Operational Documentation


Document LinkagesThe requirements, design and test plan documents producedduring the planning process should all be linked together:

The design document should demonstrate completeness by referring each design feature back to a requirement

The design document is not complete if there are requirements which aren't reflected in the design documentThe requirements document is not complete if there are design points which aren't backed up by requirements

The test plan document should demonstrate completeness by referring each test point back to a design point

The test plan is not complete if there are design points without test points

1. blah blah blah but not blah

2. blah must blah except when blah.

3. blah blah 92.7% blah blay blok bloop snog.

4. blah blah and blah!5. max 3,126

simultaneous users.

1. 17 blah slots with blah x27 feature.

2. blah in slots cabled around blah pipes.

3. install coffee machine near cluster.

4. custom blah with snaggle option disabled.

1. disconnect power.2. cut cables

(scissors in left hand drawer).

3. repair cables (glue in right hand drawer).

4. buy coffee (cream and sugar).

5. release x27s.

Requirements Document Design Document Test Plan


Document ReviewDocument review sessions should be scheduled during whicheach document is reviewed on a point by point basis.Representatives of the interested parties MUST participate.Hardware and software should not be ordered and the cluster implementation should not begin until documents have been reviewed and accepted.

Unless, of course, you have lots and lots of time and money available to reconfigure the cluster and purchase new equipment as the requirements and the design changes!

Properly conducted review meetings are quite intensive.Don't schedule more than about three hours per day.


Tools to Help You Plan Your ClusterpSeries eConfigurator.

Builds hardware and software configuration files.Only available to IBMers and IBM BPs.Downloaded from ehone.ibm.com.This tool does not tell you if the hardware and software components you have chosen will address the business problem at hand.

Use the Online Planning Worksheets.These are Java-based tools which are supported on Windows and AIX (with an appropriate Java Virtual Machine (JVM)).These create a cluster snapshot which can be applied to HACMP.Can also be created from current configuration.

Draw a diagram of your cluster.Create a pictorial representation of your cluster.Show how the clients connect (including routers).Label all cluster components.Keep the diagram up to date if your cluster changes.


Reference ManualsUse the cluster planning information in the appropriate HACMP manual:

HACMP classic version 4.5:HACMP for AIX Planning Guide SC23-4277-04HACMP for AIX Installation Guide SC23-4278-04

HACMP/ES version 4.5:HACMP for AIX Enhanced Scalability Installation and Administration Guide SC23-4306-03

HACMP version 5.2:HACMP Concepts and Facilities Guide SC23-4864-03HACMP for AIX Planning and Installation Guide SC23-4861-03HACMP for AIX Administration and Troubleshooting Guide SC23-4862-03/usr/lpp/cluster/doc/release_notes (all releases of HACMP)

www.ibm.com/servers/eserver/pseries/library/hacmp_docs.html


Export to Planning Worksheets

Export Definition File for Online Planning Worksheets


[Entry Fields]* File Name <hacmp/log/cluster.haw] / Cluster Notes []




Each design element needs to a documented test in the test plan document.

2. True or False?Each requirement in the requirements document needs a design element to explain how it will be satisfied and a documented test to show how it is verified.

3. True or False?Each test should test one and only one design feature.

4. Which of the following cluster planning documents is optional?a. Requirements documentb. Design documentc. Test plan documentd. Operational procedures documente. None of them

5. True or False? Proper cluster design documentation is a waste of time because nobody keeps it up-to-date.

6. True or False?The aspect of cluster design which generally receives the most attention is understanding and then documenting how the application operates within the cluster.



Each design element needs to a documented test in the test plan document.

2. True or False?Each requirement in the requirements document needs a design element to explain how it will be satisfied and a documented test to show how it is verified.

3. True or False?Each test should test one and only one design feature.

4. Which of the following cluster planning documents is optional?a. Requirements documentb. Design documentc. Test plan documentd. Operational procedures documente. None of them

5. True or False?*Proper cluster design documentation is a waste of time because nobody keeps it up-to-date.

6. True or False?The aspect of cluster design which generally receives the most attention is understanding and then documenting how the application operates within the cluster.

*Even if it is not kept up-to-date, proper cluster design documentation will be very useful in ensuring that the cluster is at least initially configured correctly. Failure to keep the cluster documentation up-to-date will probably eventually result in "accidental" outages.


Unit Summary

Having completed this unit, you should be able to: Explain the importance of cluster planningDescribe the key cluster planning deliverables

Requirements documentDesign documentTest planDocumented operational procedures

Explain how the requirements, design and test plan documents should be linked togetherUse the export to planning worksheets feature of HACMP 5.2

HACMP SystemsAdministration I: Planning and Implementation (Course Code AU54)

Instructor Exercises GuideERC 5.0

IBM Certified Course Material

V3.1.0.1

cover

�� Front cover

Instructor Exercises Guide

December 2004 Edition

The information contained in this document has not been submitted to any formal IBM test and is distributed on an “as is” basis withoutany warranty either express or implied. The use of this information or the implementation of any of these techniques is a customerresponsibility and depends on the customer’s ability to evaluate and integrate them into the customer’s operational environment. Whileeach item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results willresult elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk.

© Copyright International Business Machines Corporation 1998, 2004. All rights reserved.This document may not be reproduced in whole or in part without the prior written permission of IBM.Note to U.S. Government Users — Documentation related to restricted rights — Use, duplication or disclosure is subject to restrictionsset forth in GSA ADP Schedule Contract with IBM Corp.

Trademarks

IBM® is a registered trademark of International Business Machines Corporation.

The following are trademarks of International Business Machines Corporation in the United States, or other countries, or both:

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Other company, product and service names may be trademarks or service marks of others.

AFS AIX AIX 5LCross-Site DB2 DB2 Universal DatabaseDFS Enterprise Storage Server HACMPNetView POWERparallel pSeriesRedbooks Requisite RS/6000SP Tivoli TMETME 10 Versatile Storage Server WebSphere

Instructor Exercises GuideV3.1.0.1

TOC
Contents
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

Instructor Exercises Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Exercise Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Exercise 1. Cluster Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1

Exercise 2. Cluster Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1

Exercise 3. LVM Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1

Exercise 4. Network Setup and Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1

Exercise 5. HACMP Software Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1

Exercise 6. Client Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-1

Exercise 7. Cluster Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-1

Exercise 8. Application Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-1

Exercise 9. Mutual Takeover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-1

Exercise 10. HACMP Extended Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-1

Exercise 11. IPAT via Replacement and HWAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-1

Exercise 12. Network File System (NFS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-1

Exercise 13. Error Notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-1

Appendix A. Cluster Diagrams. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1

Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

© Copyright IBM Corp. 1998, 2004 Contents iii

Instructor Exercises GuideV3.0

TMK
Trademarks
The reader should recognize that the following terms, which appear in the content of this training document, are official trademarks of IBM or other companies:







AFS® AIX® AIX 5L™Cross-Site® DB2® DB2 Universal Database™DFS™ Enterprise Storage Server® HACMP™NetView® POWERparallel® pSeries®Redbooks™ Requisite® RS/6000®SP™ Tivoli® TME®TME 10™ Versatile Storage Server™ WebSphere®


© Copyright IBM Corp. 1998, 2004 Trademarks v


pref
Instructor Exercises Overview
The exercises for this course are based on a case study that is introduced in Exercise 1. Each student team builds their own cluster and resource groups. They have the freedom to choose their own names for resources or follow the guidelines given in the exercises. The objective is to build a mutual takeover environment.

In general the exercises depend on successfully completing the previous exercises.


© Copyright IBM Corp. 1998, 2004 Instructor Exercises Overview vii


pref
Exercise Description
Exercise instructions - This section contains what it is you are going to accomplish. See the Lab Setup Guide and the course Lab Guide for instructions and details pertaining to the labs. You are given the opportunity to work through each exercise given what you learned in the unit presentation.


© Copyright IBM Corp. 1998, 2004 Exercise Description ix


EXempty
Exercise 1. Cluster Design
What This Exercise Is About

This exercise is a high-level design of a cluster. It is scenario-based. This reinforces the lecture material.

What You Should Be Able to Do

At the end of the exercise, you should be able to:

• Create a high-level design of a cluster • Interpret the business requirements into a diagram suitable for

creating further HACMP configuration information • Describe how HACMP will assist in creating the design

Introduction

The scenario that the exercises are based on is a company which is amalgamating its computer sites to a single location. It is intended to consolidate computer sites from two cities into one situated roughly in the middle of the original two. The case study has been designed around five randomly chosen countries in the world. These countries and city configurations have been tested in our environment but we offer the choice to use your own. On to the scenario.

Required Materials

Your imagination.

Paper or a section of a white board.


© Copyright IBM Corp. 1998, 2004 Exercise 1. Cluster Design 1-1


Exercise Instructions

Preface

For this example we use the Canada cluster. The original configuration was one computer located in Halifax and one in Calgary. The systems have been named by their city designation to keep them straight. The corporate Web server resides on Halifax,. Currently the systems are running on internal disks, on systems too small for the task. As part of the consolidation new systems are used. These new systems are to be configured in such a manner as to provide as close to 7x24x365 access to the Web server as possible with pSeries technology. Corporate marketing is about to launch a major initiative to promote a new product solely available on the Web. The corporate management has insisted that this project is successful, and that the new computer center in Regina resolves all of the issues of reliability that thus far have caused great corporate embarrassment. All eyes are focused on this project.

A project briefing has been called by the senior executive to get an overview of how the funds for the equipment are applied.

Your task is to prepare for that meeting to present a solution.

Exercise Steps

__ 1. Draw each of the computer systems as described.

__ 2. Add the applications to the nodes.

__ 3. Add a network connection to each system for access to the outside world.

__ 4. Evaluate the lack of high availability of the initial drawing of the two separate systems.

__ 5. Combine the services of the existing networks resulting in a single network.

__ 6. Add new SSA disks to your drawing, showing cable connections.

__ 7. Make the disks highly available, RAID/mirror, redundant disks.

__ 8. Define the resources as described in the text.

__ 9. Define the characteristics of the resources.

__ 10. Indicate how the resources fail and recover.

__ 11. Make the diagram simple to understand.

END OF LAB


1-2 HACMP Implementation © Copyright IBM Corp. 1998, 2004


EXempty
Exercise 2. Cluster Planning

This exercise is going to build on the high-level design. You continue to build upon the cluster. The next step is to document your hardware to create an inventory of materials to work with. You use a cluster planning worksheets and a generic cluster diagram to design and document your cluster topology. The design is based on either the country scenario provided or the high-level design you created in the prior exercise.


At the end of the lab, you should be able to:

• Create component worksheets and a diagram showing your cluster topology in detail

• Identify the hardware configuration of the classroom equipment

Introduction

There may be differences in the documentation and the real machines in the classroom environment. The CPUs, network type, and type of disk units have been selected to provide a consistent experience but a variety of equipment may be used. Please ask if you have any questions.

Note: Throughout this lab the terms shared volume group, shared file system, node and client refer to components of your HACMP cluster. The convention of <name> is to be substituted with the appropriate thing. The example references a generic cluster’s naming of these components. Some names in your cluster may be different from that indicated in the notes.

Below is a picture of the generic cluster for this lab. The communications path may be Ethernet, Token-Ring, FDDI, or any other network supported by HACMP. There must also be a non IP serial network-- either RS232, target mode SSA or heartbeat over disk. The minimum requirement is that there are at least four shared disks (SCSI, Fiber Channel or SSA) connected to a shared bus so that two volume groups may be created and passed between nodes. If adequate disks can be provided for the purposes of mirroring and


© Copyright IBM Corp. 1998, 2004 Exercise 2. Cluster Planning 2-1


quorum, then a more realistic configuration can be built. However, this is not a requirement of the lab exercises.

The systems provided are to be the new systems for the consolidated computer center. You must prepare these systems to be the replacements of the production systems. It is time to check out the equipment to find out what is available to create our highly available solution on.

Instructor Exercise Overview

Ensure that a team number (or letter) has been assigned to each cluster team before starting this lab (see step 1). This number must be a single character. The idea is that everyone can use the lab hints now -- not just Canada as in the past. For names there will be a convention like Canada# (where # is the team number). For IP addresses the format of the third octet is #X where #, is the team number and X is the subnet number.

Point out that sample entries are shown in the component worksheet tables.

Note that the diagram below also appears in the appendix. The appendix has the cheat sheet version as well as two blank templates so the student can take one blank one home.




EXempty

AU545.0


LAB Reference Cluster

Home Node Name =Resource Group =Startup Policy =Fallover Policy =Fallback Policy =Service IP Label =Application server =

Label =Device =

Label =Device =

rootvg

4.8 GB

user

community

Network = ________________ (netmask = 255.255.255.0)

Home Node Name =Resource Group=Startup Policy =Fallover Policy =Fallback Policy =Service IP Label =Application server =

Label =Device =

Label =Device =

VG =

rootvg

4.8 GB

VG =

disk or tmssa

tty





Part 1: Examine the Cluster Environment and Complete the Cluster Component Worksheets with Storage Information

Using the cluster component worksheets (located at the end of this exercise), record the information as listed in the following steps.

__ 1. Write down your team number here: ____. In these lab exercises you must replace the symbol # with your team number unless otherwise noted.

__ 2. Log in as root on both of your cluster nodes. The root password will be provided by your instructor.

__ 3. Identify and record in the cluster components worksheet the device names and location codes of the disk adapters.

__ 4. Identify and record in the cluster components worksheet the device names and location codes of the external disks (hdisks and pdisk). Note: The external disks may not have PVIDs on them at this time.

__ 5. Identify and record in the cluster components worksheet the device names and location codes of the internal disks.

__ 6. The storage needs to be divided into two volume groups. Size of the volume groups is not important. In a real environment, disks should be mirrored and quorum issues addressed. Here the emphasis is on the operation of HACMP not how the storage is organized. You should have four disks so feel free to set up a mirror on one of the volume groups. Different methods of configuring the disks are going to be used through out the exercises. Decide on the organization but only create the volume groups when directed to.

__ 7. Identify and update the cluster planning worksheets with the names of 2 shared volume groups. Use the following names or choose your own.

» » shared_vg_a » shared_vg_b

__ 8. Identify and update the cluster component worksheets the LV component names to have a shared file system in each of the two volume groups. Select names for the logical volumes, jfs logs and filesystems. Use the following names or choose your own.

» data lv’s shared_jfslv_a, shared_jfslv_b » jfslog lv’s shared_jfslog_a, shared_jfslog_b » file systems shared_fs_a, shared_fs_b




EXempty
__ 9. Now add the just the storage information to the generic cluster diagram of your cluster. This diagram can be found in Appendix A (there are two blank ones after the filled in one. One is for in class and the other is to take home). On the other hand you may want to just compare the information on your component worksheets to the filled in worksheet at the beginning of Appendix A.
• Only fill in what you know -- the LVM information-- at the bottom of the diagram.

GO NOW TO EXERCISE 3. You return to Part 2 after the lecture for the unit on network planning.

Part 2: Examine the Cluster Environment and Complete the Cluster Component Worksheets with Networking Information

__ 10. Identify and record in the cluster components worksheet the device names (entX) and location codes of the Network Adapters.

__ 11. Identify and record in the cluster components worksheets the IP addresses that will be used for the cluster communication interfaces using the following guidelines:

• Ensure that the logical subnet rules are complied with. Each communication interface must be on a different logical subnet.

• Ensure that all of the communication interfaces have the same subnet mask. • You can use the following names/addresses or select your own. If you choose to

use your own please verify with the instructor that they will not conflict with another team. » » In the following, replace # with your team number. » » communication interface halifax#-if1 192.168.#1.1 » communication interface halifax#-if2 192.168.#2.1 » communication interface toronto#-if1 192.168.#1.2 » communication interface toronto#-if2 192.168.#2.2 » netmask is 255.255.255.0

__ 12. Identify and update the cluster components worksheets the names (IP Labels) and addresses for the service and persistent labels using the following guidelines.

• Ensure that the logical subnet rules are complied with. Assume IPAT via alias which means that the Service Labels/addresses and persistent addresses may not be on the same logical subnet as any one of the communication interfaces.

• The following names/addresses may be used or select your own. If you choose to use your own please verify with the instructor that they will not conflict with another team. » » halifax#-per 192.168.#3.1




» toronto#-per 192.168.#3.2 » appA#-svc 192.168.#3.10 » appB#-svc 192.168.#3.11

__ 13. The IP network name is generated by HACMP.

__ 14. Identify and update the cluster components worksheets the name for your cluster (any string without spaces, up to 32 characters) using the following or choose your own.

» cluster name is canada#

__ 15. Identify and update the cluster components worksheet the device names and location codes of the serial ports.

__ 16. The non-IP network name is generated by HACMP.

__ 17. At this point in time most of the names for the various cluster components should have been selected and populated on the cluster component worksheets. It is important to have a clear picture of the various names of these components as you progress through the exercises.

__ 18. Now add the networking information to the generic cluster diagram of your cluster. This diagram can be found in Appendix A (there are two blank ones after the filled in one. One is for in class and the other is to take home). On the other hand you may want to just compare the information on your component worksheets to the filled in worksheet at the beginning of Appendix A.

• Only fill in what you know -- cluster name, node names (halifax#, toronto#), and IP information at the top.




EXempty
Cluster Component Worksheets
Table 1: Non-shared Components Worksheet: FIRST NodeNon-shared Components Description Value

Node Name ------------------N/A-----------------------

***Network Adapter*** **IBM 10/100 Mbps Ethernet *** entX 10-60Network Adapter IF1Network Adapter IF2Network Adapter IF3Network Adapter IF4

***Ext. Disk Adapter*** ***SSA 160 SerialRAID Adapter*** ssaX 10-90Ext. Disk Adapter 1Ext. Disk Adapter 2

***Serial port*** ***Standard I/O Serial Port*** saX 01-S1Serial port 1Serial port 2

***TTY device *** ***Asynchronous Terminal*** ttyX 01-S1-00-00TTY device 1TTY device 2

***Internal Disk *** 16 Bit LVD SCSI Disk Drive hdiskX 10-80-00-4,0Internal Disk 1Internal Disk 2Internal Disk 3

Persistent Address ------------------N/A-----------------------

***IF IP Label/address*** myname 192.168.#x.yyIF1 IP Label/addressIF2 IP Label/addressIF3 IP Label/address

TMSSA device ------------------N/A-----------------------




Table 2: Non-shared Components Worksheet: SECOND NodeComponent Description Value

Node Name ------------------N/A-----------------------







***IF IP Label/address*** myname# 192.168.#x.yyIF1 IP Label/addressIF2 IP Label/addressIF3 IP Label/address

TMSSA device ------------------N/A-----------------------




EXempty
Table 3: Shared Components Worksheet
Component Description Value

Cluster Name ---------------------N/A-----------------Cluster ID ---------------------N/A-----------------

Cluster Subnet mask ---------------------N/A-----------------

Network Name ---------------------N/A-----------------Network Name ---------------------N/A-----------------

***Shared Disk *** *P1.1-I3/Q1-W4AC50A84400D* hdiskX pdiskYShared Disk 1Shared Disk 2Shared Disk 3Shared Disk 4

Shared vg 1 ---------------------N/A-----------------Shared jfs log 1 --------------------N/A------------------Shared jfs lv 1 --------------------N/A------------------Shared filesystem 1 --------------------N/A-------------------mount point --------------------N/A------------------

Shared vg 2 --------------------N/A------------------Shared jfs log 2 --------------------N/A------------------Shared jfs lv 2 --------------------N/A------------------Shared filesystem 2 --------------------N/A-------------------mount point --------------------N/A------------------

ALIAS: myname# 192.168.#x.yyService Label/addressService Label/addressService Label/addressService Label/addressREPLACEMENT node1:Service Label/addressHardware Address ---------------------N/A-----------------

REPLACEMENT node2:Service Label/addressHardware Address ---------------------N/A-----------------




EXempty
Exercise 3. LVM Components

This exercise reinforces the steps involved in creating a shared volume group with a filesystem to be used as an HACMP resource.



• Create a Volume Group suitable for use as an HACMP resource

• Create a filesystem suitable for use as an HACMP resource

• Manually perform the function of passing a filesystem between nodes in a cluster

Introduction

The next phase in our scenario is to provide the storage for the highly available application. We require a filesystem to store the Web pages on that can be accessed by each machine when that machine is the active node.

To support the passing of a filesystem between nodes there must be a volume group, logical volume, and a logical volume for the jfs log.

There are several methods to accomplish this task. Two are going to be explored during the exercises. First, a manual creation to emphasize the necessary steps in the process and second, in a later exercise, an automated cluster aware method will be explored during the C-SPOC exercise.

Required Materials

• Cluster Planning Worksheets and cluster diagram from the previous exercise.

• Shared disk storage connected to both nodes.


This exercise should only configure one filesystem. Save the second one for C-SPOC.


© Copyright IBM Corp. 1998, 2004 Exercise 3. LVM Components 3-1



Configure Volume Group

__ 1. With your cluster planning sheets available, begin the configuration.

__ 2. Log in to both nodes as root.

__ 3. Verify that both nodes have the same number of disks.

__ 4. Identify the internal and shared disks from the cluster worksheet. These disks might or might not have PVIDs on them.

If they match between the two systems, then you can skip to step 10.

__ 5. On both systems delete only the external hdisks.

__ 6. On one system add all of PVIDs back in.

__ 7. On the other system update the PVIDs.

__ 8. Verify the PVIDs were updated.

__ 9. The hdisks and PVIDs should match on both systems.

__ 10. Find a VG major number not used on either node __________.

__ 11. Go to your halifax# node. Create an Enhanced Concurrent Volume Group called shared_vg_a. This will be the volume group for appA#’s shared data.

__ 12. Vary on the volume group and create a jfslog logical volume with a name of shared_jfslog_a. The type is to be jfslog. Only one lp is required.

__ 13. Format the jfslog logical volume.

__ 14. Create a logical volume for data called shared_jfslv_a.

__ 15. Create a filesystem called shared_fs_a using the Add a Journaled File System on a previously defined logical volume. The mount point should be /shared_fs_a and the filesystem should not be automatically activated on system restart.

__ 16. Verify the filesystem can be mounted manually.

__ 17. Check the correct log file is active. If you have a loglv00 then you might not have formatted the jfs log before you created the jfs.

__ 18. Umount the filesystem.

__ 19. Vary off the volume group.

__ 20. On your Toronto# node, import the volume group using the major number, hdisk and volume group information. The VG name must be the same as the system it was created on.

__ 21. Set the autovaryon flag to “off” for the volume group.

__ 22. Mount the filesystem on the second node and verify it functions.

__ 23. Check the correct log file is active.




EXempty
__ 24. Unmount the filesystem.

END OF LAB




EXempty
Exercise 4. Network Setup and Test

This exercise guides you through the set up and testing of the networks required for HACMP.



• Configure TPC/IP networking suitable for HACMP • Test the TCP/IP configuration • Configure non IP communications for HACMP • Test the non IP communications for HACMP • Configure and test name resolution and authentication

Introduction

This section establishes the communication networks required for implementing HACMP. Networking is an important component of HACMP, so all related aspects are configured and tested. The information used in this exercise is derived from the previous exercise.


diskhb is not included here on purpose. It is automatically configured by the 2node config assist feature.

This lab should be done after the students complete part 2 of exercise 2.

The client ip information is deliberately left out of the host file in this exercise. It is added as part of exercise 6.


© Copyright IBM Corp. 1998, 2004 Exercise 4. Network Setup and Test 4-1


Figure 4-1. Lab Reference Cluster AU545.0

Required Materials

• Cluster Planning Worksheets and cluster diagram from exercise 2.




Label =Device =

Label =Device =

rootvg

4.8 GB

user

community

Network = ________________ (netmask = 255.255.255.0)


Label =Device =

Label =Device =

VG =

rootvg

4.8 GB

VG =

disk or tmssa

tty




EXempty
Part 1: Configure TCP/IP Interfaces and Name Resolution


__ 2. Log in as root to both of the cluster nodes.

__ 3. Check the UNIX Hostname (both the host command and the uname -n command should give you the same desired answer).

__ 4. Using the component worksheets or configuration diagram for values, configure two network adapters for use as communication interfaces, remember that each communication interfaces must use a separate logical subnet.

Note: Do NOT use the minimum config and setup option in smit. It changes the name of the node. Use smit chinet instead.

__ 5. Recheck the hostname.

__ 6. Verify the netmasks are specifically set in smitty chinet. The default could cause errors later depending on what your network address was.

__ 7. Check the configuration against the cluster worksheets.

__ 8. Repeat for other node. When both nodes are configured, test the communications between the nodes. Use the ping command to verify connection between each set of communication interfaces.

__ 9. Update the /etc/hosts file on both nodes (update one and ftp it to the other node).

__ 10. Verify name resolution and connectivity on BOTH nodes for all IP labels.

Part 2: Configure Non IP Interface



__ 13. On both nodes check if you can use a tty connection or SSA or both as a non IP network. You may have to ask your instructor for details.

If not using tty for your non IP network then skip to step __ 16.

Using tty

__ 14. On both nodes check the device configuration of the unused tty device. If the tty device does not exist, create it. If it does exist, ensure that a getty is not spawned, or better still, delete it and redefine.

__ 15. Test the non IP communications:

i. On one node execute stty < /dev/tty# where # is your tty number.

ii. The screen appears to hang. This is normal.




iii. On the other node execute “stty </dev/tty#” where # is your tty number.

iv. If the communications line is good, both nodes return their tty settings.

Using SSA

__ 16. If using target-mode SSA for your non IP network, then check if the prerequisites are there. A unique node number must be set and the device driver must be installed. If not add it.

__ 17. Test the non IP communication using SSA.

END OF LAB




EXempty
Exercise 5. HACMP Software Installation

This exercise installs the components of HACMP for AIX to support all resource group policies.



• Verify the node is prepared for the installation of HACMP

• Identify and install the packages to run HACMP


© Copyright IBM Corp. 1998, 2004 Exercise 5. HACMP Software Installation 5-1



Preface

• This exercise is composed of two parts, system capacity checks and software installation.

Part 1: System Capacity Checks

__ 1. Log into halifax# as root (this part can be done in parallel by a second person working on the other node, toronto#).

__ 2. Verify the following disk space requirements:

i. 140 MB free space in /usr, although the installation of the software will automatically increase the size if required.

ii. 100 MB free in /tmp, /var and /

__ 3. For most lab environments, check to see that the system paging space is set to twice the size of main memory. This is the default recommendation for small memory machines.

__ 4. Ensure Part 1 is performed for your other node, toronto#

Part 2: HACMP Node Installation

__ 5. Log in to halifax# as root (this part can be done in parallel by a second person working on the other node toronto#).

__ 6. Verify the AIX prerequisites are installed. If any of these are not installed notify your instructor. RSCT filesets must be at a minimum, version 2.2.1.

- bos.adt.lib - bos.adt.libm - bos.adt.syscalls - bos.data - rsct.compat.basic - rsct.compat.clients - devices.ssa.tm - devices.scsi.tm

__ 7. Change directory to the location of the filesets. In most classes they can be found in a subdirectory of the /usr/sys/inst.images directory. If there are questions, ask the Instructor.

__ 8. Install preview the following HACMP filesets:

• HACMP - cluster.adt.es - cluster.doc.en_US.es - cluster.es




EXempty
- cluster.es.clvm - cluster.es.cspoc - cluster.license - cluster.man.en_US.es - cluster.msg.en_US.cspoc (lower case en) - cluster.msg.en_US.es
__ 9. If the HACMP packages pass the prerequisite check, set preview to no and install the HACMP filesets. If there is a prerequest failure, notify your Instructor.

__ 10. Install HACMP maintenance. Check the /usr/sys/inst.images directory for an HA updates directory (in many classes it will be the subdirectory ./ha52/ptf1). If you have questions, ask the instructor.

__ 11. Reboot the nodes.

__ 12. Verify the SMIT menus. Check to see if the HACMP screens are available.

__ 13. (Optional) It would be a good idea to set up your /.profile to include paths to the HACMP commonly used commands so that you don’t have to keep entering full path names in the later lab exercises.

__ 14. (Very Optional) If the nodes have a tape subsystem attached, now would be a good time for a mksysb backup.

__ 15. Ensure Part 2 is also performed for your other node toronto#.

END OF LAB




Exercise Review/Wrapup

This is a good place to stop for a backup.




EXempty
Exercise 6. Client Setup

This exercise sets up the client for access to the HACMP system. It is used to demonstrate how the outside world views the highly available system.



• Ascertain if the client has been set up to access the HACMP cluster

• Verify the communication between the client and nodes is functioning correctly

Introduction

Our scenario has a Web server to be made highly available. We are required to test the availability traits of the Web server. This exercise creates a client to test from.

Required Materials

HACMP planning sheets.

AIX bonus pack

Client machine


This exercise assumes httpdlite is installed and running on the client machine.


© Copyright IBM Corp. 1998, 2004 Exercise 6. Client Setup 6-1



Preface

• All exercises of this chapter depend on the availability of specific equipment in your classroom.

• Replace the symbol # with your team number.

Part 1: Setting Up the Client Communications

__ 1. This exercise requires that there is a third AIX node for your team. If you have only a PC then you can (after the application integration lab) add clstat.cgi to the /usr/HTTPServer/cgi-bin directory on both cluster nodes and then use the PC browser to go to the service address of the HTTP resource group -- your instructor can help you with this.

__ 2. Log in to the client (your third machine) as root. If CDE is used on this machine then leave CDE for now. CDE comes back again after the reboot later in this exercise.

__ 3. Execute smit mktcpip to set the hostname and ipaddress of this machine for an enX interface (that has a cable in it). The IP address must be on the same subnet as one of the node interfaces. The suggested hostname is regina# and the suggested address is 192.168.#1.3. Do not set default route or DNS.

__ 4. Create an alias for the interface above to be on the same subnet as the service labels. The suggested value is 192.168.#3.30 and the subnet mask is 255.255.255.0.

__ 5. Acquire the /etc/host file from halifax# and ensure that the information in this file agrees with what you did in the previous two steps.

__ 6. Test to ensure that TCP/IP functions correctly.

__ 7. Test name resolution of the client and the nodes.

Part 2: HACMP Client Install and setup

__ 8. Install the HACMP client filesets:

cluster.adt.escluster.es (choose only the three client filesets)cluster.licensecluster.man.en_US.escluster.msg.en_US.es (choose only the client fileset)

__ 9. Install the ptf1 updates

__ 10. In order to use clstat.cgi, verify that httpdlite is running and that Netscape is available on this machine. If not ask your instructor.

__ 11. Verify Netscape starts and can display a URL, like file:///usr/lpp/bos.sysmgt/mkcd.README.html/




EXempty
The next three steps prepare you to use clinfoES from the client machine after HACMP is started in the next exercise.
__ 12. Copy the clstat.cgi script from /usr/es/sbin/cluster to the /var/docsearch/cgi-bin directory.

__ 13. Verify that the file /var/docsearch/cgi-bin/clstat.cgi is world-executable (755 or rwxr-xr-x)

__ 14. Test access to clstat.cgi using the URL http://localhost:49213/cgi-bin/clstat.cgi <-- you should get a window with the message “Could not initialize clinfo connection”.

__ 15. Put the cluster nodes ip address (that is, halifax#-per and toronto#-per) into the /usr/es/sbin/cluster/etc/clhosts file. Make sure you can ping these addresses.

__ 16. Reboot and do the ping tests to verify that this client machine functions as expected.

END OF LAB





We have the client all set and ready to go with communication checked, and name resolution.




EXempty
Exercise 7. Cluster Configuration
What This Exercise is About

This lab covers the configuration and testing of a one sided custom resource group. The cluster planning worksheets continue to be updated as the capabilities of the cluster grow.



• Use the Initialization and Standard Configuration menu to

• Discover nodes and networks

• Discover Volume groups and Filesystems

• Add a custom resource group to the cluster

• Verify the correct operation of a custom resource group

• Perform failover testing on the configured resource group

Introduction

The scenario is expanding you now create a custom resource group. This is the beginning of making an application highly available.

Required Materials

Cluster planning worksheets.


© Copyright IBM Corp. 1998, 2004 Exercise 7. Cluster Configuration 7-1



Remember this?

We are now ready to Configure the HACMP environment.

First we must set up the application environment so that we can do the configuration all at once using the Two-Node Cluster Configuration Assistant.

Note: These steps can only be done on one node. You should choose one of your nodes to be the administration node. We will assume it is halifax#

Where are We in the Implementation

Plan for network, storage, and application

eliminate single points of failure

Define and configure the AIX environment

storage (adapters, LVM volume group, filesystem)

networks IP interfaces, /etc/hosts, non-ip networks and devices)

application start and stop scripts


Configure the HACMP environment

Topologycluster, node names, HACMP ip and non-ip networks







EXempty
Part 1: Setting up the application environment
We use a dummy application for now to see how the Two-Node Cluster Configuration Assistant works.

__ 1. Log in to halifax# as root

__ 2. Execute the commands:echo ‘date +starting:%H%M >> /tmp/appA.log’>/tmp/appA_startecho 'date +stopping:%H:%M >> /tmp/appA.log' > /tmp/appA_stop

THE FOLLOWING WAS reported by an instructor testing this la b for this step:

I think the application displays starting:time and put it into the log. However, I get the screen display, but no log entry. So I vi the file and added a tee.

"echo +starting:%H%M | tee /dev/pts/0 >> /tmp/appA.log".

__ 3. Execute the commands: chmod +x /tmp/appA_startchmod +x /tmp/appA_stop

__ 4. Log in to toronto# as root and execute the command exportvg shared_vg_a (This is so that you can see that the 2-node assistant automatically imports the vg on the other node).

__ 5. Return to your halifax# node.

Part 2: Configuring HACMP

With your cluster planning sheets available begin the configuration.

__ 6. Run the Two-Node Cluster Configuration Assistant. You need an ipaddress(label) for the second node, an application server name unique to your team, start and stop script names, and a service label.

__ 7. If you encountered an error then do the cluster remove procedure (see lecture or ask instructor) on both nodes before retrying.

Lets now look at what happened to you as a result of this command.

__ 8. Look at the smit output to see what the Assistant did. You can also find this output in the /var/hacmp/log/clconfigassist.log file.

__ 9. Log on (go) to your other node (toronto#) to prove that the cluster was created on both nodes. Use the command cldisp | more to answer the following questions:

• Were the application start and stop scripts copied over? ________________ • Was the volume group imported to the other node? ____________________Use the command cldisp | more to answer the following questions: • What is the cluster name? ______________________________________ • What is the resource group name? _______________________________ • What is the startup policy? ______________________________________ • What is the fallback policy?______________________________________




• What is the vg resource name (if any)? _____________________________ • What is the non-IP network name (if any)? ___________________________ • On what enX is halifax#-if1? _____________________________________ • What is the ip network name? ____________________________________ • Were the start/stop scripts copied over? ____________________________

__ 10. So were you impressed? _________________________________

__ 11. You can now add the ip network and non-IP network names, that we promised would be generated by HACMP, to your component work sheets and/or the cluster diagram if you want to.

__ 12. Return to your administrative node (halifax#).

__ 13. Define an additional Non-IP RS232 or a TMSSA network. The lab environment may help you decide. Note that a network is automatically created when you choose the pair of devices that form the endpoints of the network.

__ 14. Execute the command cltopinfo and see that the additional non-IP network was configured. Add this name to the worksheet and/or diagram.

__ 15. Add a persistent node address for each node in the cluster -- select ‘Configure HACMP Persistent Node IP Label/Addresses’ from the ‘Extended Topology Configuration’ menu,

__ 16. Synchronize the changes -- Using the F3 key, traverse back to the Extended Configuration smit screen.

Review the output upon completion looking for any Errors or Warnings. Errors must be corrected before continuing, warnings should simply be reviewed and noted.

__ 17. Check to see that your persistent addresses were created. If not then wait until the cluster is started in Part 3 below and then check again.

__ 18. Take about 10 minutes to review the Startup, Fallover, and Fallback policies using the F1 key on the Add a Resource Group menu. When you are ready, proceed to Part 3.

Part 3: Starting HACMP

__ 19. With your cluster planning sheets available as reference documentation, it is time to start the cluster just on your administrative node (halifax#).

__ 20. Observe the output on one of the logs.

__ 21. Check that all resources were acquired successfully on the halifax# node.

__ 22. Go to your client machine (regina#).

__ 23. Start the clinfoES subsystem and verify that the /usr/es/sbin/cluster/clstat -a command works.




EXempty
__ 24. There is another option on the clstat command, the - r# option. This option sets the refresh rate of the information. For the lab environment “-r 10 “may be a more appropriate value. Restart clstat with the -r 10 option.
__ 25. Now start Netscape and make sure that the URL to clstat.cgi is working properly.

• The URL is http://localhost:49213/cgi-bin/clstat.cgi • You should now see a window with cluster information displayed. Be patient if

this window shows that the cluster is unstable. • Take a moment to familiarize yourself with what you are looking at. Click on the

resource group name app# • You will use this session to monitor the failover testing that comes next (or you

can run clstat on one of your cluster nodes)

__ 26. Now go to your administrative node (halifax#) and stop it graceful. Watch what happens in the clstat browser (be patient -- it may take 2 minutes).

__ 27. Now start HACMP and clinfo on BOTH nodes

__ 28. Use the lsvg command to see that the shared vg is varied on in passive mode on the other node (toronto#).

Part 4: Failover testing

__ 29. Return to your administrative node (halifax#) with your cluster planning sheets available for reference.

It is time to test the cluster. Although the failover testing is a function of planning to eliminate single points of failure, some basic tests should be performed on any cluster.

__ 30. On both nodes verify the IP labels used on each interface. “netstat -i” Notice which interface has which IP label.

__ 31. On the toronto# node telnet to the appA service address (appA#-svc).

__ 32. Run the tail -f /tmp/hacmp.out. There should not be any scrolling of the log file.

On your halifax# node, fail the adapter (enX) that the appA#-svc address is running on by executing the command ifconfig enX down (or disconnect the cable from the enX adapter card).

Watch the reaction on both nodes in the /tmp/hacmp.out file. Also monitor the clstat window. Notice that the telnet session from the toronto# node was not interrupted and that the log information scrolled by during the event processing.

Instructor note: ifconfig down is no longer corrected by HACMP at least for IPAT via alias.

__ 33. When swap adapter has completed, verify that the location of the appA#-svc service address is now on another ethernet adapter.

__ 34. Restore the failed adapter. The interface should now be in an “UP” state.

» ifconfig enX up or » connect the network cable




__ 35. (Optional) - You may wish to swap the service address (and/or) persistent address back by using C-SPOC.

__ 36. Using the console rather than a telnet session (because you will lose it), monitor the hacmp.out file on the halifax#-if1x (left) node and disconnect both network cables at the same time.

__ 37. There should be a network down event executed after a short period of time. What happens to the resource group on the halifax (left) node, and why?

__ 38. Check the /tmp/hacmp.out file on the toronto# node, it should also have detected a network failure.

__ 39. Restore both the network connections for the halifax# node. What event do you observe happens?

__ 40. Where is the resource Group at this time? Verify that the IP labels, volume groups, and file systems and application are available on that node.

__ 41. You are now going to move resources back from one node to the other. On the halifax# node monitor the log. On the toronto# node execute smit clstop and stop the cluster services with the mode of takeover. Leave the default value for the other fields.

__ 42. The clstat.cgi should change colors from green to yellow (substate unstable, toronto# leaving) and the state of the toronto# node and interfaces should change to red (down).

__ 43. All of the components in the resource group should move over to the halifax# node. Verify the IP labels, volume groups, and file systems on the halifax# node.

__ 44. On the toronto# node restart HACMP. Observe the /tmp/hacmp.out file on the halifax# node and, of course, the clstat session. The resource group stays put.

END OF LAB




EXempty
You have a running cluster. Congratulations, now the fun really begins. Make sure clstat shows the cluster as stable with the tcp/ip and non-ip networks up.




EXempty
Exercise 8. Application Integration

The HACMP cluster is now functional with a highly available filesystem and IP label. Adding the Web server to the scenario is the next step.



• Add the IBM Web server powered by Apache to the nodes • Adjust the configuration of the Web server to acknowledge the

highly available IP label • Introduce a minor configuration change to the Web server to use

the shared storage • Add an application start and stop script to HACMP • Test the application functionality

Introduction

The intention is not to become Web server programers but to simply add an existing application to the HACMP environment. This is to demonstrate one way to add an application to the HACMP environment.

Required Materials

A running cluster

The AIX 5L Expansion pack


© Copyright IBM Corp. 1998, 2004 Exercise 8. Application Integration 8-1



Preface

• As part of this exercise, C-SPOC and DARE are used to enable the addition of filesystems, applications and resource changes to the cluster while it is running. If all things function as designed, no system reboots or HACMP restarts are required.

Part 1: Install the IBM Web server file system


__ 2. Log in as root on the halifax# node.

__ 3. Create a new filesystem for the Web documents. Enter smit hacmp.

__ 4. Verify both nodes know about the new file system.

__ 5. Continue on the halifax# node. Check to see that the filesystem is mounted on the system that currently owns the resource group (should be halifax#).

Part 2: Install the IBM Web server software

__ 6. Check if http filesets listed below are installed. If not, ask your instructor. On many class images they may be found in the directory /usr/sys/inst.images/web-appl. Otherwise you may need the AIX 5L Expansion Pack CD.

» http_server.base » http_server.admin » http_server.html

__ 7. On the other node (toronto#), repeat the previous step. Once installed, delete all of the information in the directory /usr/HTTPServer/htdocs (only on this node!).

__ 8. Go back to the halifax# node. In the directory /usr/HTTPServer/conf/, edit httpd.conf and change the “ServerName” variable to be the same as the service IP label (appA#-svc).

Note: The hostname must be resolvable, that is, host hostname should return a good answer. If the hostname is not resolvable, add the hostname to the 127.0.0.1 address as an alias. If in doubt, ask the Instructor. Remember to do this on both nodes otherwise successful takeover does not happen.

__ 9. Use ftp to put a copy of the /usr/HTTPServer/conf/httpd.conf file on the toronto# node.

Part 3: Configure HACMP for the Application

__ 10. Add the Application Server to HACMP.

__ 11. Change the appA_group to use the Application Server http_server.




EXempty
__ 12. While the synchronizing takes place, monitor the HACMP logs until you see the message start server http_server. Check to see that the Apache server started ok.
__ 13. From the client, start a new window in Netscape and connect to the URL http://appA#-svc. The Web screen Welcome to the IBM HTTP Server window should pop up.

__ 14. Perform a failover test by halting the Halifax# node in your favorite manner (for example, “halt -q” or “echo bye > /dev/kmem”).

__ 15. Wait for takeover to complete and verify what happens to the Web server. Use the page reload button on your Web browser to see if the Web server is really there.

__ 16. Bring up the Halifax# node again and start HACMP.

__ 17. What has happened to the Resource Group, and why?

END OF LAB




Optional Exercises

For the Web-enabled Candidates

__ 1. Change the Web server pages on the shared disk to prove the location of the data elements.

END OF LAB




EXempty
Exercise 9. Mutual Takeover

This lab exercise expands the capabilities of the cluster. The intent is to completely add the resource group and all of its components while the cluster is running.



• Use the C-SPOC functionality and DARE capabilities of HACMP to make changes to the cluster while it is running

• Add a new volume group while the cluster is running

• Add all of the components of a shared file system while the system is running

• Add a resource group to the cluster and activate it while the cluster is running

• Test a mutual takeover configuration

Introduction

In the scenario there are two resource groups to be made highly available. The addition of the second resource group is done with the C-SPOC commands with the cluster running.


© Copyright IBM Corp. 1998, 2004 Exercise 9. Mutual Takeover 9-1



Preface

• Add the shared_res_grp_b resource group components according to the scenarios. This will require a second filesystem.

Part 1: Add a Second Resource Group and Filesystem to the Cluster

__ 1. Ensure HACMP is running on both nodes and that the HTTP application is running on halifax#

__ 2. Using the lspv command on BOTH nodes, verify that there is a shared disk hdiskX available with the same PVID. If so skip to step 8.

__ 3. On the halifax# node make sure that the hdiskX has no PVID

__ 4. Create a new PVID for hdiskX

__ 5. On the toronto# node delete the hdisk.

__ 6. Add the disk back in.

__ 7. Verify the hdisk number and PVID agree between the two nodes.

__ 8. On the administrative node (halifax#) create a shared volume group called shared_vg_b using C-SPOC.

__ 9. Verify the Volume Group exists on both nodes.

Now that the volume group is created it must be discovered a resource group must be created and finally the volume group must be added to the resource group before any further C-SPOC utilities will access it.

__ 10. Discover the volume group using Extended Configuration in smitty hacmp.

__ 11. Create a resource group called appB_group with the toronto# node as the highest priority and halifax# node as the next priority.

__ 12. Add the volume group to the resource group

__ 13. Synchronize the Cluster.

__ 14. Once synchronized, the Volume Group is varied online, on the owning node (toronto#). Wait for this to happen. Then on your administrative node halifax# use C-SPOC to add a jfs log shared logical volume to the shared_vg_b. The name should be shared_jfslog_b, the LV type should be jfslog, and use 1 PP.

__ 15. Format the jfslog so that it can be used by the filesystem that is created in the next few steps. If the log is not formatted, it is not used.

__ 16. Back on halifax#, add a second shared logical volume with number of LOGICAL PARTITIONS=10, NAME= shared_jfslv_b, TYPE=jfs.




EXempty
__ 17. Add a shared file system on the previously created Logical volume called /shared_fs_b. Using the F3 key traverse back to Configure Volume Groups, Logical Volumes and Filesystems. Select Shared File Systems
__ 18. The filesystem should be available on node toronto# in a few minutes.

The following was observed during additional testing and may or may not be repeatable: a message on the smit panel saying that shared_fs_b is not a known file system, and the failed response was posted. However, when I looked at /etc/filesystems it was there and a manual mount from the Toronto node worked. I was then able to move the resource group from one node to another and back using the system management (C-SPOC) menu.

Part 2: Create the application and service label resources


__ 20. Create the application start script:echo ‘hostname>>/shared_fs_b/appB.log’ >/tmp/appB_startecho ‘date +” starting:%r” >> /shared_fs_b/appB.log’>>/tmp/appB_start

__ 21. Create the application stop script echo ‘hostname>>/shared_fs_b/appB.log’ >/tmp/appB_stopecho ‘date +” stopping:%r” >> /shared_fs_b/appB.log’>>/tmp/appB_stop

__ 22. ftp the scripts to the other node

__ 23. Make the scripts executable on both nodes:

__ 24. On halifax#, create the Service IP label resource.

__ 25. Create the application server resource

__ 26. Add the resources to the resource group

__ 27. Synchronize the Cluster. Using the F3 key, traverse back to Initialization and Standard Configuration.

__ 28. Test that the toronto# service IP label is available.

__ 29. Test the new resource group on the toronto# node for network adapter swap/failure and node failure.

__ 30. OPTIONAL -- If you have an extra disk execute the mirrorvg and chfs commands to test splitting off a copy as presented in the unit 3 lecture. Note that this step could be done using C-SPOC to create the mirror. Also note that one purpose of this step is to show how to undo the backup copy.

END OF LAB





The first part of the exercise looked at using C-CSPOC to add a new resource to the cluster.




EXempty
Exercise 10. HACMP Extended Features

This lab exercise expands the capabilities provided in the extended features option.



• Create a Cluster ‘Snapshot’

• Use the C-SPOC functions and DARE capabilities of HACMP to make changes to the cluster while it is running

• Add additional Resource Groups while the cluster is running

• Add additional Service aliases

• Modify Resource Group Behavior Policies

• Configure Settling and Fallback timers

Introduction

To enhance the scenario create two additional resource groups to be made highly available. The addition of these resource groups and their behavior modification is done with the C-SPOC commands with the cluster running.


© Copyright IBM Corp. 1998, 2004 Exercise 10. HACMP Extended Features 10-1



Preface

• Add additional service aliases, and create an additional custom resource group for each node.

• Modify the default start and fallback policies of the new resource groups to examine the resource behavior during cluster startup and reintegration event processing.

• Create

Part 1: Create a Cluster Snapshot

__ 1. In the last exercise we made the goal of the class so let’s save our environment before continuing by creating a Cluster Snapshot

Notice:There are two snapshot files <snapshot>.odm and <snapshot>.infoThe directory for the snapshot is /usr/es/sbin/cluster/snapshotsThe clsnapshotinfo command was run on both nodes (output in the “.info” file)

__ 2. Read the mutual_takover.info file. Go on to the next step when you are ready.

Part 2: Add an additional Service alias and Resource Group to each Cluster Node

__ 3. Log in to the halifax# node as root.

__ 4. Add two additional service labels to the /etc/hosts file

__ 5. Discover these new addresses in HACMP using the Extended Configuration menu from smit hacmp.

__ 6. Configure the two additional HACMP Service IP Labels/Addresses as resources in HACMP.

We now add two resource groups appC_group and appD_group with different startup policies. The first (appC_group) behaves like the old inactive takeover, and the second (appD_group) behaves like the old rotating.

__ 7. Add an additional Resource Group called appC_group with a home node of halifax# and a startup policy of Online on First Available node.

__ 8. Add another Resource Group called appD_group with a home node of toronto# and a startup policy of Online Using Distribution Policy.

__ 9. Now add the Service IP Labels created in step 1 to the Resource Groups just created. Using the F3 key, traverse back to the Extended HACMP Resource Group Configuration and select Change/Show Resources and Attributes for a Resource Group




EXempty
__ 10. In order to mimic the old rotating we need to change the distribution policy to network. This is done using the smit extended runtime menu Configure Distribution Policy for Resource Groups. The cluster must be stopped on both nodes first.
__ 11. Synchronize the cluster. Using the F3 key, traverse back to the ‘Extended Configuration’ smit screen.

Part 3: Test Resource Group Behavior

__ 12. Start HACMP only on your toront# node.

__ 13. Once the node is stable check the status of the Resource Groups. Does this look normal? If not, what is wrong -- Should appC_group be online on toronto#?

__ 14. Start HACMP on the Halifax (left) node.

__ 15. Once the node is stable check the status of the Resource Groups. Does everything look correct now (check the appC_group)? If so, what changed? Why?

__ 16. OPTIONAL: To understand better the distribution policy, stop the nodes and bring up halifax# first and see what happens to the appD_group. Then stop halifax# with takeover. Then restart halifax# and see what happens to the appD_group.

Let’s have a look at configuring a settling timer which allows you to modify the behavior of the Fallback To Higher Priority Node In The List fallback policy so that there are not two online operations if you bring up the secondary node first.

Part 4: Add a Settling Timer

__ 17. Ensure that you are on your administration node (halifax#) and configure a Settling Timer (can only be used if startup policy is ‘Online On First Available Node’ ).

__ 18. Synchronize the Cluster. Using the F3 key, traverse back to the ‘Extended Configuration’ smit screen. Notice in the smit output the messages about the settling timer value.

Part 5: Testing Cluster Behavior using a Settling Timer

__ 19. On your administrative node (halifax#), stop Cluster processing on both nodes.

__ 20. Wait 2 minutes and then start HACMP -- only on the toronto# node.

__ 21. Wait until you can see that the appB_group is online then verify that appC_group is still offline on toronto# using the clRGinfo command (note that the clRGinfo command can be run from either node as long as HACMP is started on any one of the nodes)

__ 22. Start HACMP on the halifax# node.




__ 23. Verify that the appC_group comes online on halifax# (without first being online on toronto#). As you can see the purpose of the settling timer is to prevent the resources from being immediately acquired by the first active node.

__ 24. OPTIONAL -- repeat this part but wait for settling time to expire after starting the cluster on toronto#. Verify that appC_group comes online on toronto#. Stop the cluster manager on both nodes, wait 2 minutes, start the cluster manager on both nodes.

Part 6: Configure Delayed Fallback Timer

__ 25. Cluster should be started on both nodes and appC_group should be online on halifax#.

__ 26. On your administrative node (halifax#), create a delayed fallback timer policy for 30 minutes from now (instructor may modify this time)

__ 27. Add the fallback timer policy the resource group appC_group

__ 28. Synchronize

Part 7: Testing Cluster Behavior using a Delayed Fallback Timer

__ 29. Verify that appC_group is online on halifax# using the clRGinfo command

__ 30. Stop the cluster manager only on halifax# with takeover

__ 31. Verify that appC_group is now online on toronto# (clRGinfo).

__ 32. Wait 2 minutes (required before a restart)

__ 33. Start the cluster manager on halifax#

__ 34. Monitor the cluster from toronto#. In /tmp/hacmp.out at the event summary for check_for_site_up_complete halifax#, there is now a message stating the fallback time. Make sure appC_group is still on toronto# before the fallback then tail -f the hacmp.out file and wait for the fallback to occur.

__ 35. At the time set for the Delayed Fallback Timer, appC_group should move back to halifax# (you should see activity from tail command)

__ 36. On your administrative node (halifax#), remove the name of the Delayed Fallback Timer (my_delayfbt) from the resource group appC_group (you can keep the policy definition if you want).

__ 37. Reset the Settling time to 0 (from the menu ‘Configure Resource Group Run-Time Policies’

__ 38. Synchronize.

END OF LAB




EXempty




EXempty
Exercise 11. IPAT via Replacement and HWAT

This lab explores the options of removing a cluster and creating an IPAT via replacement environment.

This lab also examines Gratuitous Arp, and the use of Hardware Address Takeover (HWAT), functionality for environments where Gratuitous Arp may not be the best solution.



• Describe how to set up IPAT via replacement

• Describe to behavior of Arp updates/refreshes using gratuitous Arp or Hardware Address Takeover where required.

• Describe how to set up HWAT


© Copyright IBM Corp. 1998, 2004 Exercise 11. IPAT via Replacement and HWAT 11-1



Preface

• The first part shows how to remove a cluster • The second part of this lab looks at setting up IPAT via replacement and using the

standard configuration path to build a cluster. • The third part of this lab looks at Gratuitous arp. • The fourth part of this exercise adds hardware address concepts. HWAT or MAC

address takeover would be used in situations where gratuitous arp may not be supported, as in older hardware, or non-standard operating systems.

Part 1: Remove and add a new Cluster

__ 1. On your administration node (halifax#), stop both the cluster nodes.

__ 2. Snapshot

__ 3. remove cluster

__ 4. Add a replacement service address to your /etc/hosts file (must be on the same subnet as one of the if interfaces.

__ 5. Configure a new cluster on halifax#. Go to the HACMP for AIX smit panel and select Initialization and Standard Configuration.

__ 6. Use Extended Configuration to set the network to turn off IPAT via aliases.

__ 7. Use Extended Configuration to configure a non-IP network by choosing the pair of devices that will make up the network (Using the F3 key, traverse back to the Extended Topology Configuration smit screen.

__ 8. Redo the Persistent Addresses from your planning worksheet. (Using the F3 key, traverse back to the Extended Topology Configuration smit screen).

__ 9. Create the Service IP Label resource. Using the F3 key, traverse back to the ‘Extended Configuration’ smit screen. Select ‘Extended Resource Configuration’.

__ 10. Create a resource groups. Using the F3 key, traverse back to the Extended Resource Configuration smit screen.

__ 11. Add Resources to the Resource Group. Using the F3 key, traverse back to the HACMP extended Resource Group Configuration smit screen.

__ 12. Synchronize the cluster. Using the F3 key, traverse back to the Extended Configuration smit screen. Select Extended Verification and Synchronization

__ 13. Start HACMP on the toronto# node.

__ 14. Verify the appR_group did not come online because of the startup policy.


__ 16. Verify that the appR_group is online on halifax#




EXempty
Part 2: Gratuitous ARP
From the AIX 5L Version 5.1 commands reference for ifconfig. “Gratuitous ARP is supported for ethernet, token-ring, and FDDI interfaces. This means when an IP address is assigned, the host sends an ARP request for its own address (the new address) to inform other machines of its address so that they can update their ARP entry immediately. It also lets hosts detect duplicate IP addresses.”

This will make it a little difficult to create a failure with AIX clients but the tests are valid.

__ 17. Log on the client machine. Verify that clinfo has not been started.

__ 18. Use the ping command to test the service IP Label of appR_group on halifax#).

__ 19. Check the contents of the arp cache.

__ 20. On the halifax# node generate a swap adapter event. Be aware that you need to do this fairly quickly before the arp cache times out.

__ 21. Check the contents of the arp cache on the client, compare the results with the previous iteration of the command.

__ 22. The hardware address should have updated in the arp cache on the client without any intervention.

Note: If the entry is not in the arp cache when the Gratuitous arp is broadcast it is ignored.

Part 3: Hardware Address Takeover

In this scenario the router in Regina is a bit of an antique and does not support gratuitous ARP. It was highlighted as a problem since the ARP cache retention is 15 minutes. This problem was discovered during the preliminary cluster testing.

__ 23. On the halifax# node log in as root.

__ 24. Identify the interface that is reconfigured with the appR-repl service address and write the mac address here._____________________________________

__ 25. Identify the alternate mac address. To specify an alternate hardware address for an Ethernet interface, change the 1st byte xx to 4x j_________________________________________

__ 26. Change the appR-repl service IP label to add an alternate hardware address in the field.

__ 27. Synchronize the cluster. Using the F3 key, traverse back to the ‘Extended Configuration’ smit screen. Notice the following message in the smit log: cldare: Detected changes to service IP label appR1-repl. Please note that changing parameters of service IP label via a DARE may result in releasing resource group appR_group.




__ 28. Bring the appR_group online using the C-SPOC menu. If, on the client, there is no arp cache entry for the appR-repl service address, then ping the appR-repl service address.

__ 29. Verify that the alternate hardware address is now configured on the interface for the appR#-repl service address.

__ 30. Fail the halifax# node in your favorite manner.

__ 31. Check that the halifax service address is on the toronto# node and observe the hardware address associated with that service address

Part 4: Re-create a Cluster from a Snapshot

__ 32. Ensure the cluster manager is stopped on both clusters nodes.

__ 33. Apply the snapshot that contains all the cluster definitions you made in exercise 10.

__ 34. Start HACMP

__ 35. For each resource group, verify to yourself that you understand how the online node was chosen.


__ 37. Restart the failed node and observe the re-integration. Verify that you understand how the online node was chosen for each of the resource groups

END OF LAB




EXempty
This exercise looked at cascading resource groups, and how to configure both cascading without Fallback and Inactive Takeover. It also covered setting up and testing Hardware Address Takeover.

This exercise also looked at rotating resource groups.




EXempty
Exercise 12. Network File System (NFS)

This lab covers a couple of different methods for configuring network filesystems with HACMP. It also demonstrates how to set various NFS options in HACMP exported filesystems.



• Have HACMP export a filesystem as part of a resource group

• Have HACMP import a filesystems as part of resource group

• Modify the NFS export options for the exported filesystem

• Add an NFS cross-mount

• Modify the NFS cross-mount for performance and flexibility


© Copyright IBM Corp. 1998, 2004 Exercise 12. Network File System (NFS) 12-1



Preface


Part 1: NFS exports in a resource group

__ 1. Assumptions: You need to start this exercise off with HACMP up on both nodes, and identify two resource groups -- one whose home node is halifax# (that is, appA_group) and the other whose home node is toronto# (that is, appB_group). Each group should have a shared filesystem defined to it (that is, shared_fs_a and shared_fs_b). On each node, verify that nfs is running (lssrc -g nfs) after HACMP is started.

__ 2. Modify the resource group appA_group to add /shared_fs_a as a filesystem/directory to NFS export and set to true the option ‘Filesystems mounted before IP configured’

__ 3. Modify the resource group appB_group to add /shared_fs_b as a filesystem/directory to NFS export. Using the F3 key, traverse back to the ’HACMP Extended Resource Group Configuration’.

__ 4. Synchronize the resources. Using the F3 key, traverse back to ‘Extended Configuration’ smit screen.

__ 5. When the reconfiguration of resources has completed on each node, check the directories are exported through NFS.

__ 6. Log in on the client as root.

__ 7. Create a directory /halifax and /toronto.

__ 8. On the client, using the service address for the appA_group, mount the nfs exported directory /shared_fs_a on the local directory /halifax.

__ 9. On the client, using the service address for the appB_group, mount the nfs exported directory /shared_fs_b on the local directory /toronto.

__ 10. Verify the nfs directories are mounted where intended.

__ 11. Back on a cluster node -- fail one of the nodes in your favorite manner. Verify that the nfs directories are still exported on the remaining node and mounted on the client system.

__ 12. Try to create a file in the /halifax directory. It should not work. Lets see how this can be addressed.

Part 2: Modifying the NFS Export Options




EXempty
__ 13. The output of the lsnfsexp command on the nodes explains that only the cluster nodes can use user root. To change this we create an override file. Its name is /usr/es/sbin/cluster/etc/exports. HACMP uses this file to update the /etc/xtabs file used by NFS.
__ 14. On the running node, use the lsnfsexp command to copy the current /etc/xtabs file to the HACMP file and then modify the HACMP file using the following commands:

- lsnfsexp > /usr/es/sbin/cluster/etc/exports - Edit /usr/es/sbin/cluster/etc/exports and add the client to the list of hosts - Save the file - ftp the file to the other node

__ 15. Restart the failed node.

__ 16. From the client try to create a file in the nfs directory on the client of the node you have just restarted.

Part 3: NFS Cross-mount within the Cluster

__ 17. On both nodes create a directory /hanfs.

__ 18. Edit the resource group appA_group and add the following to the option ‘Filesystems/Directories to NFS mount” /hanfs;/shared_fs_a. This will mount the /shared_fs_a nfs filesystem on the mount point /hanfs for all systems in that resource group.

__ 19. Synchronize the resources and verify this is true on both nodes.

__ 20. Fail the toronto# node in your favorite manner.

__ 21. Confirm that halifax# node has all the resource groups, and that the NFS mounts are OK.

END OF LAB





This exercise looked at various methods of implementing NFS in an HACMP cluster.




EXempty
Exercise 13. Error Notification

This lab covers the adding of error notifications into AIX through the HACMP smit screens.



• Add an error notification for the loss of quorum on a volume group

• Emulate the error condition and test the error notification method

• Optionally add another error notification based on filesystems full


© Copyright IBM Corp. 1998, 2004 Exercise 13. Error Notification 13-1



Preface

• This exercise looks at Automatic Error Notification. Before you configure Automatic Error Notification, you must have a valid HACMP configuration. Using the SMIT options, you can use the following methods: - Configure Automatic Error Notification - List Automatic Error Notification - Remove Automatic Error Notification.

• Remember that Error Notification is a function of AIX - HACMP just gives you the smit screens that make it easier to enter error notification methods.

Setting Up the automatic error notifications on the halifax# node.


__ 2. Stop the Cluster. The cluster must be down to configure Automatic Error Notification.

__ 3. Configure Automatic Error Notification.

When you run automatic error notification, it assigns two error methods for all the error types noted:

cl_failover is assigned if a disk or network interface card is determined to be a single point of failure, and that failure would cause the cluster to fall over. If there is a failure of one of these devices, this method logs the error in hacmp.out and shuts the cluster node down. A graceful stop is attempted first, if this is unsuccessful, cl_exit is called to shut down the node.

cl_logerror is assigned for any other error type. If there is a failure of a device configured with this method, they are logged in hacmp.out.

__ 4. List Error Notification Methods. Use the F3 key to traverse back to the ‘Configure Automatic Error Notification’ smit screen.

__ 5. To see the AIX odm file, execute the command odmget errnotify | more. The HACMP generated stanzas will be at the bottom.

END OF LAB




Cluster Planning Diagram

AU54 lab teams

REPLACE # with team number

team number= ______

_______ IP Label IP Address HW Address

if1 halifax#-if1 192.168.#1.1 _____________

if2 halifax#-if2 192.168.#2.1 _____________

Persist halifax#-per 192.168.#3..1

Resource Group appA_group contains

Volume Group= shared_vg_a

hdisks = ______________

Major # = ______________

JFS Log =shared_jfslog_a

Logical Volume =shared_jfslv_a

FS Mount Point =/shared_fs_a

Home Node Name toronto#Resource Group appB_groupStartup Policy =OHNOFallover Policy =FONPFallback Policy =FBHPService IP Label =appB#-svc 192.168.#3.20Application server =appB

Label =toronto#_hiskY_01Device =/dev/hdiskX

Label =toronto_tty0_01Device =/dev/tty0

rootvg

4.8 GB

user

community

Network = ________________ (netmask = ___.___.___.___)

Home Node Name halifax#Resource Group = appA_groupStartup Policy =OHNOFallover Policy =FONPFallback Policy =FBNFService IP Label =appA#-svc 192.168.#3.10Application server =appA

Label =halifax#_hiskX_01Device =/dev/hdiskX

Label =halifax#_tty0_01Device =/dev/tty0

VG =

rootvg

4.8 GB

VG =

serial

serial


if1 toronto#-if1 192.168.#1.2 _____________

if2 toronto#-if2 192.168.#2.2 _____________

Persist toronto#-per 192.168.#3.2

Resource Group appB_group contains

Volume Group= shared_vg_b

hdisks = ______________

Major # = ______________

JFS Log = shared_jfslog_b

Logical Volume = shared_jfslv_b

FS Mount Point = /shared_fs_b

client

regina#

if1192.168.#1.3

alias 192.168.#3.30


A-2 HACMP Implementation © Copyright IBM Corp. 1998, 2004


AP


_______ IP Label IP Address Hardware Address

if1 _______ _________ _______________

if2 _______ _________ _______________

Persist _______ _________

Resource Group __________ contains

Volume Group= ______________

hdisks = ______________

Major # = ______________

JFS Log = ______________

Logical Volume = ______________

FS Mount Point = ______________

Home Node Name = Resource Group = Startup Policy = Fallover Policy = Fallback Policy =

Service IP Label =

Application server =

Label = Device =

Label = Device =

rootvg

4.8 GB

user

community

Network = _____________netmask=___.___.___.___

Home Node Name =Resource Group =Startup Policy =Fallover Policy =Fallback Policy =

Service IP Label =

Application server=

Label =Device =

Label =Device =

VG =

rootvg

4.8 GB

VG =

serial

serial


if1 _______ _________ _______________

if2 _______ _________ _______________

Persist _______ _________


Volume Group= ______________

hdisks = ______________

Major # = ______________

JFS Log = ______________


FS Mount Point = ______________

client

hostname ___________

if1 _________________

svc alias ____________






if1 _______ _________ _______________

if2 _______ _________ _______________

Persist _______ _________


Volume Group= ______________

hdisks = ______________

Major # = ______________

JFS Log = ______________


FS Mount Point = ______________


Service IP Label =


Label = Device =

Label = Device =

rootvg

4.8 GB

user

community

Network = _____________netmask=___.___.___.___


Service IP Label =

Application server=

Label =Device =

Label =Device =

VG =

rootvg

4.8 GB

VG =

serial

serial


if1 _______ _________ _______________

if2 _______ _________ _______________

Persist _______ _________


Volume Group= ______________

hdisks = ______________

Major # = ______________

JFS Log = ______________


FS Mount Point = ______________

client


if1 _________________

svc alias ____________



V1.2.2

backpg

HACMP SystemsAdministration I: Planning and Implementation (Course Code AU54)

Instructor Exercises Guidewith HintsERC 5.0

IBM Certified Course Material

V3.1.0.1

cover

�� Front cover

Instructor Exercises Guide with Hints

December 2004 Edition

The information contained in this document has not been submitted to any formal IBM test and is distributed on an “as is” basis withoutany warranty either express or implied. The use of this information or the implementation of any of these techniques is a customerresponsibility and depends on the customer’s ability to evaluate and integrate them into the customer’s operational environment. Whileeach item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results willresult elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk.

© Copyright International Business Machines Corporation 1998, 2004. All rights reserved.This document may not be reproduced in whole or in part without the prior written permission of IBM.Note to U.S. Government Users — Documentation related to restricted rights — Use, duplication or disclosure is subject to restrictionsset forth in GSA ADP Schedule Contract with IBM Corp.

Trademarks







AFS AIX AIX 5LCross-Site DB2 DB2 Universal DatabaseDFS Enterprise Storage Server HACMPNetView POWERparallel pSeriesRedbooks Requisite RS/6000SP Tivoli TMETME 10 Versatile Storage Server WebSphere

Instructor Exercises Guide with HintsV3.1.0.1

TOC
Contents
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

Instructor Exercises Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Exercise Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Exercise 1. Cluster Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1

Exercise 2. Cluster Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1

Exercise 3. LVM Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1

Exercise 4. Network Setup and Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1

Exercise 5. HACMP Software Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1

Exercise 6. Client Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-1

Exercise 7. Cluster Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-1

Exercise 8. Application Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-1

Exercise 9. Mutual Takeover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-1

Exercise 10. HACMP Extended Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-1

Exercise 11. IPAT via Replacement and HWAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-1

Exercise 12. Network File System (NFS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-1

Exercise 13. Error Notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-1

Appendix A. Cluster Diagrams. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1


© Copyright IBM Corp. 1998, 2004 Contents iii

Instructor Exercises Guide with HintsV3.0

TMK
Trademarks
The reader should recognize that the following terms, which appear in the content of this training document, are official trademarks of IBM or other companies:







AFS® AIX® AIX 5L™Cross-Site® DB2® DB2 Universal Database™DFS™ Enterprise Storage Server® HACMP™NetView® POWERparallel® pSeries®Redbooks™ Requisite® RS/6000®SP™ Tivoli® TME®TME 10™ Versatile Storage Server™ WebSphere®


© Copyright IBM Corp. 1998, 2004 Trademarks v


pref
Instructor Exercises Overview
The exercises for this course are based on a case study that is introduced in Exercise 1. Each student team builds their own cluster and resource groups. They have the freedom to choose their own names for resources or follow the guidelines given in the exercises. The objective is to build a mutual takeover environment.

In general the exercises depend on successfully completing the previous exercises.


© Copyright IBM Corp. 1998, 2004 Instructor Exercises Overview vii


pref
Exercise Description
Exercise instructions - This section contains what it is you are going to accomplish. See the Lab Setup Guide and the course Lab Guide for instructions and details pertaining to the labs. You are given the opportunity to work through each exercise given what you learned in the unit presentation.


© Copyright IBM Corp. 1998, 2004 Exercise Description ix


EXempty
Exercise 1. Cluster Design
(with Hints)


This exercise is a high-level design of a cluster. It is scenario-based. This reinforces the lecture material.



• Create a high-level design of a cluster • Interpret the business requirements into a diagram suitable for

creating further HACMP configuration information • Describe how HACMP will assist in creating the design

Introduction

The scenario that the exercises are based on is a company which is amalgamating its computer sites to a single location. It is intended to consolidate computer sites from two cities into one situated roughly in the middle of the original two. The case study has been designed around five randomly chosen countries in the world. These countries and city configurations have been tested in our environment but we offer the choice to use your own. On to the scenario.

Required Materials

Your imagination.

Paper or a section of a white board.


© Copyright IBM Corp. 1998, 2004 Exercise 1. Cluster Design 1-1


Exercise Instructions with Hints

Preface

(All hints are marked by a » sign)

For this example we use the Canada cluster. The original configuration was one computer located in Halifax and one in Calgary. The systems have been named by their city designation to keep them straight. The corporate Web server resides on Halifax,. Currently the systems are running on internal disks, on systems too small for the task. As part of the consolidation new systems are used. These new systems are to be configured in such a manner as to provide as close to 7x24x365 access to the Web server as possible with pSeries technology. Corporate marketing is about to launch a major initiative to promote a new product solely available on the Web. The corporate management has insisted that this project is successful, and that the new computer center in Regina resolves all of the issues of reliability that thus far have caused great corporate embarrassment. All eyes are focused on this project.

A project briefing has been called by the senior executive to get an overview of how the funds for the equipment are applied.

Your task is to prepare for that meeting to present a solution.

Exercise Steps

__ 1. Draw each of the computer systems as described.

__ 2. Add the applications to the nodes.

__ 3. Add a network connection to each system for access to the outside world.

__ 4. Evaluate the lack of high availability of the initial drawing of the two separate systems.

__ 5. Combine the services of the existing networks resulting in a single network.

__ 6. Add new SSA disks to your drawing, showing cable connections.

__ 7. Make the disks highly available, RAID/mirror, redundant disks.

__ 8. Define the resources as described in the text.

__ 9. Define the characteristics of the resources.

__ 10. Indicate how the resources fail and recover.

__ 11. Make the diagram simple to understand.

END OF LAB




EXempty
Exercise 2. Cluster Planning
(with Hints)


This exercise is going to build on the high-level design. You continue to build upon the cluster. The next step is to document your hardware to create an inventory of materials to work with. You use a cluster planning worksheets and a generic cluster diagram to design and document your cluster topology. The design is based on either the country scenario provided or the high-level design you created in the prior exercise.



• Create component worksheets and a diagram showing your cluster topology in detail

• Identify the hardware configuration of the classroom equipment

Introduction

There may be differences in the documentation and the real machines in the classroom environment. The CPUs, network type, and type of disk units have been selected to provide a consistent experience but a variety of equipment may be used. Please ask if you have any questions.

Note: Throughout this lab the terms shared volume group, shared file system, node and client refer to components of your HACMP cluster. The convention of <name> is to be substituted with the appropriate thing. The example references a generic cluster’s naming of these components. Some names in your cluster may be different from that indicated in the notes.

Below is a picture of the generic cluster for this lab. The communications path may be Ethernet, Token-Ring, FDDI, or any other network supported by HACMP. There must also be a non IP serial network-- either RS232, target mode SSA or heartbeat over disk. The minimum requirement is that there are at least four shared disks (SCSI, Fiber Channel or SSA) connected to a shared bus so that




two volume groups may be created and passed between nodes. If adequate disks can be provided for the purposes of mirroring and quorum, then a more realistic configuration can be built. However, this is not a requirement of the lab exercises.

The systems provided are to be the new systems for the consolidated computer center. You must prepare these systems to be the replacements of the production systems. It is time to check out the equipment to find out what is available to create our highly available solution on.


Ensure that a team number (or letter) has been assigned to each cluster team before starting this lab (see step 1). This number must be a single character. The idea is that everyone can use the lab hints now -- not just Canada as in the past. For names there will be a convention like Canada# (where # is the team number). For IP addresses the format of the third octet is #X where #, is the team number and X is the subnet number.

Point out that sample entries are shown in the component worksheet tables.

Note that the diagram below also appears in the appendix. The appendix has the cheat sheet version as well as two blank templates so the student can take one blank one home.




EXempty

AU545.0




Label =Device =

Label =Device =

rootvg

4.8 GB

user

community

Network = ________________ (netmask = 255.255.255.0)


Label =Device =

Label =Device =

VG =

rootvg

4.8 GB

VG =

disk or tmssa

tty





Preface

All hints are marked by a » sign.

Part 1: Examine the Cluster Environment and Complete the Cluster Component Worksheets with Storage Information

Using the cluster component worksheets (located at the end of this exercise), record the information as listed in the following steps.

__ 1. Write down your team number here: ____. In these lab exercises you must replace the symbol # with your team number unless otherwise noted.

__ 2. Log in as root on both of your cluster nodes. The root password will be provided by your instructor.

__ 3. Identify and record in the cluster components worksheet the device names and location codes of the disk adapters.

» lsdev -Cc adapter

__ 4. Identify and record in the cluster components worksheet the device names and location codes of the external disks (hdisks and pdisk). Note: The external disks may not have PVIDs on them at this time.

» lspv » lsdev -Cc disk » smitty ssadlog

__ 5. Identify and record in the cluster components worksheet the device names and location codes of the internal disks.

» lsdev -Cc disk » lsdev -Cc pdisk

__ 6. The storage needs to be divided into two volume groups. Size of the volume groups is not important. In a real environment, disks should be mirrored and quorum issues addressed. Here the emphasis is on the operation of HACMP not how the storage is organized. You should have four disks so feel free to set up a mirror on one of the volume groups. Different methods of configuring the disks are going to be used through out the exercises. Decide on the organization but only create the volume groups when directed to.

__ 7. Identify and update the cluster planning worksheets with the names of 2 shared volume groups. Use the following names or choose your own.

»




EXempty
» shared_vg_a » shared_vg_b
__ 8. Identify and update the cluster component worksheets the LV component names to have a shared file system in each of the two volume groups. Select names for the logical volumes, jfs logs and filesystems. Use the following names or choose your own.

» data lv’s shared_jfslv_a, shared_jfslv_b » jfslog lv’s shared_jfslog_a, shared_jfslog_b » file systems shared_fs_a, shared_fs_b

__ 9. Now add the just the storage information to the generic cluster diagram of your cluster. This diagram can be found in Appendix A (there are two blank ones after the filled in one. One is for in class and the other is to take home). On the other hand you may want to just compare the information on your component worksheets to the filled in worksheet at the beginning of Appendix A.

• Only fill in what you know -- the LVM information-- at the bottom of the diagram.

GO NOW TO EXERCISE 3. You return to Part 2 after the lecture for the unit on network planning.

Part 2: Examine the Cluster Environment and Complete the Cluster Component Worksheets with Networking Information

__ 10. Identify and record in the cluster components worksheet the device names (entX) and location codes of the Network Adapters.

» lsdev -Cc adapter » lsdev -Cc if

__ 11. Identify and record in the cluster components worksheets the IP addresses that will be used for the cluster communication interfaces using the following guidelines:

• Ensure that the logical subnet rules are complied with. Each communication interface must be on a different logical subnet.

• Ensure that all of the communication interfaces have the same subnet mask. • You can use the following names/addresses or select your own. If you choose to

use your own please verify with the instructor that they will not conflict with another team. » » In the following, replace # with your team number. » » communication interface halifax#-if1 192.168.#1.1 » communication interface halifax#-if2 192.168.#2.1 » communication interface toronto#-if1 192.168.#1.2




» communication interface toronto#-if2 192.168.#2.2 » netmask is 255.255.255.0

__ 12. Identify and update the cluster components worksheets the names (IP Labels) and addresses for the service and persistent labels using the following guidelines.

• Ensure that the logical subnet rules are complied with. Assume IPAT via alias which means that the Service Labels/addresses and persistent addresses may not be on the same logical subnet as any one of the communication interfaces.

• The following names/addresses may be used or select your own. If you choose to use your own please verify with the instructor that they will not conflict with another team. » » halifax#-per 192.168.#3.1 » toronto#-per 192.168.#3.2 » appA#-svc 192.168.#3.10 » appB#-svc 192.168.#3.11

__ 13. The IP network name is generated by HACMP.

__ 14. Identify and update the cluster components worksheets the name for your cluster (any string without spaces, up to 32 characters) using the following or choose your own.

» cluster name is canada#

__ 15. Identify and update the cluster components worksheet the device names and location codes of the serial ports.

» lsdev -C | grep - i serial

__ 16. The non-IP network name is generated by HACMP.

__ 17. At this point in time most of the names for the various cluster components should have been selected and populated on the cluster component worksheets. It is important to have a clear picture of the various names of these components as you progress through the exercises.

__ 18. Now add the networking information to the generic cluster diagram of your cluster. This diagram can be found in Appendix A (there are two blank ones after the filled in one. One is for in class and the other is to take home). On the other hand you may want to just compare the information on your component worksheets to the filled in worksheet at the beginning of Appendix A.

• Only fill in what you know -- cluster name, node names (halifax#, toronto#), and IP information at the top.




EXempty
Cluster Component Worksheets
Table 1: Non-shared Components Worksheet: FIRST NodeNon-shared Components Description Value

Node Name ------------------N/A-----------------------







***IF IP Label/address*** myname 192.168.#x.yyIF1 IP Label/addressIF2 IP Label/addressIF3 IP Label/address

TMSSA device ------------------N/A-----------------------




Table 2: Non-shared Components Worksheet: SECOND NodeComponent Description Value

Node Name ------------------N/A-----------------------







***IF IP Label/address*** myname# 192.168.#x.yyIF1 IP Label/addressIF2 IP Label/addressIF3 IP Label/address

TMSSA device ------------------N/A-----------------------




EXempty
Table 3: Shared Components Worksheet
Component Description Value

Cluster Name ---------------------N/A-----------------Cluster ID ---------------------N/A-----------------

Cluster Subnet mask ---------------------N/A-----------------

Network Name ---------------------N/A-----------------Network Name ---------------------N/A-----------------

***Shared Disk *** *P1.1-I3/Q1-W4AC50A84400D* hdiskX pdiskYShared Disk 1Shared Disk 2Shared Disk 3Shared Disk 4

Shared vg 1 ---------------------N/A-----------------Shared jfs log 1 --------------------N/A------------------Shared jfs lv 1 --------------------N/A------------------Shared filesystem 1 --------------------N/A-------------------mount point --------------------N/A------------------

Shared vg 2 --------------------N/A------------------Shared jfs log 2 --------------------N/A------------------Shared jfs lv 2 --------------------N/A------------------Shared filesystem 2 --------------------N/A-------------------mount point --------------------N/A------------------

ALIAS: myname# 192.168.#x.yyService Label/addressService Label/addressService Label/addressService Label/addressREPLACEMENT node1:Service Label/addressHardware Address ---------------------N/A-----------------

REPLACEMENT node2:Service Label/addressHardware Address ---------------------N/A-----------------




EXempty
Exercise 3. LVM Components
(with Hints)


This exercise reinforces the steps involved in creating a shared volume group with a filesystem to be used as an HACMP resource.



• Create a Volume Group suitable for use as an HACMP resource

• Create a filesystem suitable for use as an HACMP resource

• Manually perform the function of passing a filesystem between nodes in a cluster

Introduction

The next phase in our scenario is to provide the storage for the highly available application. We require a filesystem to store the Web pages on that can be accessed by each machine when that machine is the active node.

To support the passing of a filesystem between nodes there must be a volume group, logical volume, and a logical volume for the jfs log.

There are several methods to accomplish this task. Two are going to be explored during the exercises. First, a manual creation to emphasize the necessary steps in the process and second, in a later exercise, an automated cluster aware method will be explored during the C-SPOC exercise.

Required Materials

• Cluster Planning Worksheets and cluster diagram from the previous exercise.

• Shared disk storage connected to both nodes.





This exercise should only configure one filesystem. Save the second one for C-SPOC.




EXempty
Preface

• All hints are marked by a » sign.

Configure Volume Group


__ 2. Log in to both nodes as root.

__ 3. Verify that both nodes have the same number of disks.

» lspv » lsdev -Cc disk

__ 4. Identify the internal and shared disks from the cluster worksheet. These disks might or might not have PVIDs on them.

» lscfg | grep hdisk

If they match between the two systems, then you can skip to step 10.

__ 5. On both systems delete only the external hdisks.

» hint assumes hdisk0 and hdisk1 are internal disks and that hdisk 2-5 are external shared disks

» rmdev -l hdisk2 -d » rmdev -l hdisk3 -d » rmdev -l hdisk4 -d » rmdev -l hdisk5 -d » ALTERNATE METHOD: » for i in $(lspv|grep -v hdisk0|grep -v hdisk1|awk '{print $1}";do » >rmdev -l $i -d » >done

__ 6. On one system add all of PVIDs back in.

» hint assumes hdisk0 and hdisk1 are internal disks and that hdisk2-5 are external shared disks

» cfgmgr -l ssar » chdev -a pv=yes -l hdisk2 » chdev -a pv=yes -l hdisk3 » chdev -a pv=yes -l hdisk4 » chdev -a pv=yes -l hdisk5

__ 7. On the other system update the PVIDs.

» cfgmgr

__ 8. Verify the PVIDs were updated.

» lspv




__ 9. The hdisks and PVIDs should match on both systems.

__ 10. Find a VG major number not used on either node __________.

» on both nodes execute lvlstmajor (the numbers listed are available)

__ 11. Go to your halifax# node. Create an Enhanced Concurrent Volume Group called shared_vg_a. This will be the volume group for appA#’s shared data.

» smitty vg (or smitty ->system storage -> logical volume manager ->volume groups)

» -> add a volume group) » Fill in VG name » Set partition size (if default value won’t work) <-- pop-up list is available » Using F4 select a single physical volume (you may select another volume if

you have three or more shared disks) » Set activate volume group automatically at system restart to ‘NO’ » Set the VG major number (use previous step) » Set “Create VG Concurrent Capable?” to enhanced concurrent

__ 12. Vary on the volume group and create a jfslog logical volume with a name of shared_jfslog_a. The type is to be jfslog. Only one lp is required.

» varyonvg shared_vg_a » smitty lv » Select Logical Volumes -> Add a Logical Volume » Select VG just created, a list is provided (F4) » Fill in the NAME=shared_jfslog_a, number of lp = 1,TYPE = jfslog » Fill in your favorite options

__ 13. Format the jfslog logical volume.

» logform /dev/shared_jfslog_a » answer yes to delete all the information.

__ 14. Create a logical volume for data called shared_jfslv_a.

» smitty lv » Add logical volume » Enter your favorite options (lp=10 should be enough)

__ 15. Create a filesystem called shared_fs_a using the Add a Journaled File System on a previously defined logical volume. The mount point should be /shared_fs_a and the filesystem should not be automatically activated on system restart.

» smitty jfs » Add/Change/Show/Delete File Systems » Journaled File Systems » Add a Journaled File System on a previously defined logical volume » Add a standard Journaled File System » F4 list, must be used to select the logical volume just created » Use /shared_fs_a as the mount point




EXempty
» Set mount automatically at system restart to ‘NO’
__ 16. Verify the filesystem can be mounted manually.

» mount /shared_fs_a

__ 17. Check the correct log file is active. If you have a loglv00 then you might not have formatted the jfs log before you created the jfs.

» lsvg -l shared_vg_a » ALTERNATE METHOD: mount command with no arguments.

__ 18. Umount the filesystem.

» umount /shared_fs_a


» varyoff vg shared_vg_a

__ 20. On your Toronto# node, import the volume group using the major number, hdisk and volume group information. The VG name must be the same as the system it was created on.

» importvg -V ?? -y sharev_vg_a hdiskX

__ 21. Set the autovaryon flag to “off” for the volume group.

» chvg -an shared_vg_a

__ 22. Mount the filesystem on the second node and verify it functions.

» mount /shared_fs_a

__ 23. Check the correct log file is active.

» lsvg -l shared_vg_a » ALTERNATE METHOD: mount command with no arguments.

__ 24. Unmount the filesystem.

» umount /shared_fs_a


» varyoffvg shared_vg_a

END OF LAB




EXempty
Exercise 4. Network Setup and Test
(with Hints)


This exercise guides you through the set up and testing of the networks required for HACMP.



• Configure TPC/IP networking suitable for HACMP • Test the TCP/IP configuration • Configure non IP communications for HACMP • Test the non IP communications for HACMP • Configure and test name resolution and authentication

Introduction

This section establishes the communication networks required for implementing HACMP. Networking is an important component of HACMP, so all related aspects are configured and tested. The information used in this exercise is derived from the previous exercise.


diskhb is not included here on purpose. It is automatically configured by the 2node config assist feature.

This lab should be done after the students complete part 2 of exercise 2.

The client ip information is deliberately left out of the host file in this exercise. It is added as part of exercise 6.




Figure 4-1. Lab Reference Cluster AU545.0

Required Materials

• Cluster Planning Worksheets and cluster diagram from exercise 2.




Label =Device =

Label =Device =

rootvg

4.8 GB

user

community

Network = ________________ (netmask = 255.255.255.0)


Label =Device =

Label =Device =

VG =

rootvg

4.8 GB

VG =

disk or tmssa

tty




EXempty
Preface


Part 1: Configure TCP/IP Interfaces and Name Resolution



__ 3. Check the UNIX Hostname (both the host command and the uname -n command should give you the same desired answer).

» To display: hostname and uname -n » To change: smitty hostname and/or uname -S ‘hostname’

__ 4. Using the component worksheets or configuration diagram for values, configure two network adapters for use as communication interfaces, remember that each communication interfaces must use a separate logical subnet.

Note: Do NOT use the minimum config and setup option in smit. It changes the name of the node. Use smit chinet instead.

» smitty chinet - or » smitty -> communication ->tcpip -> further configuration -> network interfaces

-> network interface selection -> change/show network interface -> edit each interface (one at a time) with tcp/ip address, netmask and state (up).

__ 5. Recheck the hostname.

» hostname » uname -n

__ 6. Verify the netmasks are specifically set in smitty chinet. The default could cause errors later depending on what your network address was.

» smitty chinet -- select each interface, verify netmask settings are the same., For this class the netmask should be 255.255.255.0

» or ifconfig -a

__ 7. Check the configuration against the cluster worksheets.

» netstat -i » netstat -in » ifconfig -a

__ 8. Repeat for other node. When both nodes are configured, test the communications between the nodes. Use the ping command to verify connection between each set of communication interfaces.

__ 9. Update the /etc/hosts file on both nodes (update one and ftp it to the other node).




» vi /etc/hosts » 127.0.0.1 loopback localhost halifax# toronto#

Note: Assumes using suggested hostnames --client are covered in exercise 6)

» 192.168.#1.1 halifax#-if1 » 192.168.#2.1 halifax#-if2 » 192.168.#3.1 halifax#-per » 192.168.#3.10 appA#-svc » 192.168.#1.2 toronto#-if1 » 192.168.#2.2 toronto#-if2 » 192.168.#3.2 toronto#-per » 192.168.#3.20 appB#-svc »

__ 10. Verify name resolution and connectivity on BOTH nodes for all IP labels.

» host halifax#-if1 » ping halifax#-if1 » host halifax#-if2 » ping halifax#-if2 » host toronto#-if1 » ping toronto#-if1 » host toronto#-if2 » ping toronto#-if2

Part 2: Configure Non IP Interface



__ 13. On both nodes check if you can use a tty connection or SSA or both as a non IP network. You may have to ask your instructor for details.

If not using tty for your non IP network then skip to step __ 16.

Using tty

__ 14. On both nodes check the device configuration of the unused tty device. If the tty device does not exist, create it. If it does exist, ensure that a getty is not spawned, or better still, delete it and redefine.

» smitty tty -> change/show tty or add tty » Enable login = disable » baud rate = 9600 » parity= No » bits = 8 » stop bits = 1




EXempty
__ 15. Test the non IP communications:
i. On one node execute stty < /dev/tty# where # is your tty number.

ii. The screen appears to hang. This is normal.

iii. On the other node execute “stty </dev/tty#” where # is your tty number.

iv. If the communications line is good, both nodes return their tty settings.

Using SSA

__ 16. If using target-mode SSA for your non IP network, then check if the prerequisites are there. A unique node number must be set and the device driver must be installed. If not add it.

» lsdev -C | grep ssa » lsattr -El ssar » lscfg -vl ssa0 » lslpp -L devices.ssa.tm.rte (If not installed on both nodes, ask your

instructor if it can be added and where are the needed information or resources.) • Install missing software. • Install usable microcode (if required).

» chdev -l ssar -a node_number=<a unique number> (different on each node)

» run cfgmgr -- must be run on the first node, then the second node, then the first node.

» ls -l /dev | grep ssa -- verify the existence of the .tm and .im device files

__ 17. Test the non IP communication using SSA.

» Node A: cat < /dev/tmssa<number>.tm <-- node number of node B » Node B: cat <filename> > /dev/tmssa<number>.im <-- node number of

node A

END OF LAB




EXempty
Exercise 5. HACMP Software Installation
(with Hints)


This exercise installs the components of HACMP for AIX to support all resource group policies.



• Verify the node is prepared for the installation of HACMP

• Identify and install the packages to run HACMP





Preface

• This exercise is composed of two parts, system capacity checks and software installation.


Part 1: System Capacity Checks

__ 1. Log into halifax# as root (this part can be done in parallel by a second person working on the other node, toronto#).

__ 2. Verify the following disk space requirements:

i. 140 MB free space in /usr, although the installation of the software will automatically increase the size if required.

ii. 100 MB free in /tmp, /var and /

» Use df -k to check the filesystem sizes.

__ 3. For most lab environments, check to see that the system paging space is set to twice the size of main memory. This is the default recommendation for small memory machines.

» lsps -a to see the paging file space » lsattr -E -l sys0 -a realmem to see the amount of memory

__ 4. Ensure Part 1 is performed for your other node, toronto#

Part 2: HACMP Node Installation

__ 5. Log in to halifax# as root (this part can be done in parallel by a second person working on the other node toronto#).

__ 6. Verify the AIX prerequisites are installed. If any of these are not installed notify your instructor. RSCT filesets must be at a minimum, version 2.2.1.

- bos.adt.lib - bos.adt.libm - bos.adt.syscalls - bos.data - rsct.compat.basic - rsct.compat.clients - devices.ssa.tm - devices.scsi.tm » lslpp -L bos.adt* » lslpp -L bos.data » lslpp -L rsct.compat* » lslpp -L devices.ssa.tm.rte




EXempty
» lslpp -L devices.scsi.tm.rte
__ 7. Change directory to the location of the filesets. In most classes they can be found in a subdirectory of the /usr/sys/inst.images directory. If there are questions, ask the Instructor.

» ls -l /usr/sys/inst.images

__ 8. Install preview the following HACMP filesets:

• HACMP - cluster.adt.es - cluster.doc.en_US.es - cluster.es - cluster.es.clvm - cluster.es.cspoc - cluster.license - cluster.man.en_US.es - cluster.msg.en_US.cspoc (lower case en) - cluster.msg.en_US.es » Use smitty --> Software Installation and Maintenance

--> Install and Update Software-->Install and Update Software from ALL Available Software

» Enter “.” for the directory » Press F4 to see the list of filesets. » After choosing the filesets and returning to the install menu, set “PREVIEW”

to yes and set “ACCEPT new license agreements” to yes. Then execute the preview install.

__ 9. If the HACMP packages pass the prerequisite check, set preview to no and install the HACMP filesets. If there is a prerequest failure, notify your Instructor.

__ 10. Install HACMP maintenance. Check the /usr/sys/inst.images directory for an HA updates directory (in many classes it will be the subdirectory ./ha52/ptf1). If you have questions, ask the instructor.

» Change directory to the maintenance directory. » smitty install -> Install and Update software

--> Update installed software to latest level (Update All) » Enter “.” for the directory » Change the commit updates field from ‘yes’ to ‘no’ » Change the save replaced files from ‘no’ to ‘yes’

__ 11. Reboot the nodes.

» shutdown -Fr

__ 12. Verify the SMIT menus. Check to see if the HACMP screens are available.

» smitty hacmp




__ 13. (Optional) It would be a good idea to set up your /.profile to include paths to the HACMP commonly used commands so that you don’t have to keep entering full path names in the later lab exercises.

» PATH=$PATH:/usr/es/sbin/cluster:/usr/es/sbin/cluster/utilities » PATH=$PATH:/usr/es/sbin/cluster/etc » PATH=$PATH:/usr/es/sbin/cluster/diag

__ 14. (Very Optional) If the nodes have a tape subsystem attached, now would be a good time for a mksysb backup.

__ 15. Ensure Part 2 is also performed for your other node toronto#.

END OF LAB




EXempty
This is a good place to stop for a backup.




EXempty
Exercise 6. Client Setup
(with Hints)


This exercise sets up the client for access to the HACMP system. It is used to demonstrate how the outside world views the highly available system.



• Ascertain if the client has been set up to access the HACMP cluster

• Verify the communication between the client and nodes is functioning correctly

Introduction

Our scenario has a Web server to be made highly available. We are required to test the availability traits of the Web server. This exercise creates a client to test from.

Required Materials

HACMP planning sheets.

AIX bonus pack

Client machine


This exercise assumes httpdlite is installed and running on the client machine.





Preface


• Replace the symbol # with your team number. • All hints are marked by a » sign.

Part 1: Setting Up the Client Communications

__ 1. This exercise requires that there is a third AIX node for your team. If you have only a PC then you can (after the application integration lab) add clstat.cgi to the /usr/HTTPServer/cgi-bin directory on both cluster nodes and then use the PC browser to go to the service address of the HTTP resource group -- your instructor can help you with this.

__ 2. Log in to the client (your third machine) as root. If CDE is used on this machine then leave CDE for now. CDE comes back again after the reboot later in this exercise.

__ 3. Execute smit mktcpip to set the hostname and ipaddress of this machine for an enX interface (that has a cable in it). The IP address must be on the same subnet as one of the node interfaces. The suggested hostname is regina# and the suggested address is 192.168.#1.3. Do not set default route or DNS.

__ 4. Create an alias for the interface above to be on the same subnet as the service labels. The suggested value is 192.168.#3.30 and the subnet mask is 255.255.255.0.

» smit inet --> configure Aliases --> Add an IPV4 Network Alias

__ 5. Acquire the /etc/host file from halifax# and ensure that the information in this file agrees with what you did in the previous two steps.

» ftp halifax#-if1 » get /etc/hosts » quit » vi /etc/hosts

__ 6. Test to ensure that TCP/IP functions correctly.

» ping halifax#-if1 » ping toronto#-if1

__ 7. Test name resolution of the client and the nodes.

» Use the host command to test name resolution.

Part 2: HACMP Client Install and setup

__ 8. Install the HACMP client filesets:




EXempty
cluster.adt.escluster.es (choose only the three client filesets)cluster.licensecluster.man.en_US.escluster.msg.en_US.es (choose only the client fileset)
» Change to the directory with the filesets » smitty install_all

__ 9. Install the ptf1 updates

» Change to the directory with the filesets » smitty update_all

__ 10. In order to use clstat.cgi, verify that httpdlite is running and that Netscape is available on this machine. If not ask your instructor.

» ps -ef | grep httpdlite » netscape file:///usr/lpp/bos.sysmgt//mkcd.README.html

__ 11. Verify Netscape starts and can display a URL, like file:///usr/lpp/bos.sysmgt/mkcd.README.html/

The next three steps prepare you to use clinfoES from the client machine after HACMP is started in the next exercise.

__ 12. Copy the clstat.cgi script from /usr/es/sbin/cluster to the /var/docsearch/cgi-bin directory.

» cd /var/docsearch/cgi-bin » cp /usr/es/sbin/cluster/clstat.cgi ./

__ 13. Verify that the file /var/docsearch/cgi-bin/clstat.cgi is world-executable (755 or rwxr-xr-x)

» chmod +x clstat.cgi » ls -al clstat.cgi

__ 14. Test access to clstat.cgi using the URL http://localhost:49213/cgi-bin/clstat.cgi <-- you should get a window with the message “Could not initialize clinfo connection”.

__ 15. Put the cluster nodes ip address (that is, halifax#-per and toronto#-per) into the /usr/es/sbin/cluster/etc/clhosts file. Make sure you can ping these addresses.

__ 16. Reboot and do the ping tests to verify that this client machine functions as expected.

END OF LAB





We have the client all set and ready to go with communication checked, and name resolution.




EXempty
Exercise 7. Cluster Configuration
(with Hints)


This lab covers the configuration and testing of a one sided custom resource group. The cluster planning worksheets continue to be updated as the capabilities of the cluster grow.



• Use the Initialization and Standard Configuration menu to

• Discover nodes and networks

• Discover Volume groups and Filesystems

• Add a custom resource group to the cluster

• Verify the correct operation of a custom resource group

• Perform failover testing on the configured resource group

Introduction

The scenario is expanding you now create a custom resource group. This is the beginning of making an application highly available.

Required Materials

Cluster planning worksheets.





Remember this?

We are now ready to Configure the HACMP environment.

First we must set up the application environment so that we can do the configuration all at once using the Two-Node Cluster Configuration Assistant.

Note: These steps can only be done on one node. You should choose one of your nodes to be the administration node. We will assume it is halifax#


Where are We in the Implementation

Plan for network, storage, and application

eliminate single points of failure

Define and configure the AIX environment

storage (adapters, LVM volume group, filesystem)

networks IP interfaces, /etc/hosts, non-ip networks and devices)

application start and stop scripts


Configure the HACMP environment

Topologycluster, node names, HACMP ip and non-ip networks







EXempty
Part 1: Setting up the application environment
We use a dummy application for now to see how the Two-Node Cluster Configuration Assistant works.


__ 2. Execute the commands:echo ‘date +starting:%H%M >> /tmp/appA.log’>/tmp/appA_startecho 'date +stopping:%H:%M >> /tmp/appA.log' > /tmp/appA_stop

THE FOLLOWING WAS reported by an instructor testing this la b for this step:

I think the application displays starting:time and put it into the log. However, I get the screen display, but no log entry. So I vi the file and added a tee.

"echo +starting:%H%M | tee /dev/pts/0 >> /tmp/appA.log".

__ 3. Execute the commands: chmod +x /tmp/appA_startchmod +x /tmp/appA_stop

__ 4. Log in to toronto# as root and execute the command exportvg shared_vg_a (This is so that you can see that the 2-node assistant automatically imports the vg on the other node).

__ 5. Return to your halifax# node.

Part 2: Configuring HACMP

With your cluster planning sheets available begin the configuration.

__ 6. Run the Two-Node Cluster Configuration Assistant. You need an ipaddress(label) for the second node, an application server name unique to your team, start and stop script names, and a service label.

» smitty hacmp --> Initialization and Standard Configuration--> Two-Node Cluster Configuration Assistant

» For the Communication Path to Takeover Node, select (F4) toronto#-if1 » For the Application Server Name, type appA » For the Application Server Start script, type /tmp/appA_start » For the Application Server Stop Script, type /tmp/appA_stop » For the Service IP Label, select (F4) appA#-svc.

__ 7. If you encountered an error then do the cluster remove procedure (see lecture or ask instructor) on both nodes before retrying.

Lets now look at what happened to you as a result of this command.

__ 8. Look at the smit output to see what the Assistant did. You can also find this output in the /var/hacmp/log/clconfigassist.log file.

__ 9. Log on (go) to your other node (toronto#) to prove that the cluster was created on both nodes. Use the command cldisp | more to answer the following questions:




• Were the application start and stop scripts copied over? ________________ • Was the volume group imported to the other node? ____________________Use the command cldisp | more to answer the following questions: • What is the cluster name? ______________________________________ • What is the resource group name? _______________________________ • What is the startup policy? ______________________________________ • What is the fallback policy?______________________________________ • What is the vg resource name (if any)? _____________________________ • What is the non-IP network name (if any)? ___________________________ • On what enX is halifax#-if1? _____________________________________ • What is the ip network name? ____________________________________ • Were the start/stop scripts copied over? ____________________________

__ 10. So were you impressed? _________________________________

__ 11. You can now add the ip network and non-IP network names, that we promised would be generated by HACMP, to your component work sheets and/or the cluster diagram if you want to.

__ 12. Return to your administrative node (halifax#).

__ 13. Define an additional Non-IP RS232 or a TMSSA network. The lab environment may help you decide. Note that a network is automatically created when you choose the pair of devices that form the endpoints of the network.

» smitty hacmp » Select, ‘Extended Configuration’ » Select Extended Topology Configuration » Select ‘Configure HACMP Communication Interfaces/Devices’ » Add Communication Interfaces/Devices » Select ‘Add Discovered Communication Interface and Devices’ » Select ‘Communication Devices’ from the list » Select, using F7, the Point-to-Point Pair of Discovered Communication

Devices (either a /dev/tty# pair or a TMSSA# pair).

__ 14. Execute the command cltopinfo and see that the additional non-IP network was configured. Add this name to the worksheet and/or diagram.

__ 15. Add a persistent node address for each node in the cluster -- select ‘Configure HACMP Persistent Node IP Label/Addresses’ from the ‘Extended Topology Configuration’ menu,

» Configure a Persistent Node IP Label/Address » Select Add a Persistent Node IP Label/Address » Select a node form the list, press enter » Select (using F4) the network name and IP Label/Address -- the Network

Name will be the same Network Name that the interfaces and service labels belong too. The suggested IP labels are names of the form XXX-per.

» Repeat this step for the other node.




EXempty
__ 16. Synchronize the changes -- Using the F3 key, traverse back to the Extended Configuration smit screen.
» Select Verify and Synchronize HACMP Configuration’

Review the output upon completion looking for any Errors or Warnings. Errors must be corrected before continuing, warnings should simply be reviewed and noted.

__ 17. Check to see that your persistent addresses were created. If not then wait until the cluster is started in Part 3 below and then check again.

» netstat -i

__ 18. Take about 10 minutes to review the Startup, Fallover, and Fallback policies using the F1 key on the Add a Resource Group menu. When you are ready, proceed to Part 3.

» smitty hacmp --> Initialization and Standard Configuration-->Configure HACMP Resource Groups -->Add a Resource Group

Part 3: Starting HACMP

__ 19. With your cluster planning sheets available as reference documentation, it is time to start the cluster just on your administrative node (halifax#).

» On the halifax# node, enter smitty clstart (or smit hacmp » -> System Management (C-SPOC) » -> Mange HACMP Service » -> Start Cluster Services <-- choose halifax# and start clinfo.

__ 20. Observe the output on one of the logs.

» execute “tail -f /usr/es/adm/cluster.log” for overview log » execute “tail -f /tmp/hacmp.out” for detail log

__ 21. Check that all resources were acquired successfully on the halifax# node.

» lsvg -o » mount » df -k » netstat -i or netstat -in (also verify persistent addresses are there) » There is a “starting” message in the /tmp/appA.log file.

__ 22. Go to your client machine (regina#).

__ 23. Start the clinfoES subsystem and verify that the /usr/es/sbin/cluster/clstat -a command works.

» startsrc -s clinfoES » clstat -a

__ 24. There is another option on the clstat command, the - r# option. This option sets the refresh rate of the information. For the lab environment “-r 10 “may be a more appropriate value. Restart clstat with the -r 10 option.




__ 25. Now start Netscape and make sure that the URL to clstat.cgi is working properly.

• The URL is http://localhost:49213/cgi-bin/clstat.cgi • You should now see a window with cluster information displayed. Be patient if

this window shows that the cluster is unstable. • Take a moment to familiarize yourself with what you are looking at. Click on the

resource group name app# • You will use this session to monitor the failover testing that comes next (or you

can run clstat on one of your cluster nodes)

__ 26. Now go to your administrative node (halifax#) and stop it graceful. Watch what happens in the clstat browser (be patient -- it may take 2 minutes).

» smitty clstop <-- stop halifax# graceful

__ 27. Now start HACMP and clinfo on BOTH nodes

» smitty clstart <-- start halifax# and toronto# and clinfo

__ 28. Use the lsvg command to see that the shared vg is varied on in passive mode on the other node (toronto#).

» lsvg shared_vg_a

Part 4: Failover testing

__ 29. Return to your administrative node (halifax#) with your cluster planning sheets available for reference.

It is time to test the cluster. Although the failover testing is a function of planning to eliminate single points of failure, some basic tests should be performed on any cluster.

__ 30. On both nodes verify the IP labels used on each interface. “netstat -i” Notice which interface has which IP label.

__ 31. On the toronto# node telnet to the appA service address (appA#-svc).

__ 32. Run the tail -f /tmp/hacmp.out. There should not be any scrolling of the log file.

On your halifax# node, fail the adapter (enX) that the appA#-svc address is running on by executing the command ifconfig enX down (or disconnect the cable from the enX adapter card).

Watch the reaction on both nodes in the /tmp/hacmp.out file. Also monitor the clstat window. Notice that the telnet session from the toronto# node was not interrupted and that the log information scrolled by during the event processing.

Instructor note: ifconfig down is no longer corrected by HACMP at least for IPAT via alias.

__ 33. When swap adapter has completed, verify that the location of the appA#-svc service address is now on another ethernet adapter.

» Use netstat -i

__ 34. Restore the failed adapter. The interface should now be in an “UP” state.




EXempty
» ifconfig enX up or » connect the network cable
__ 35. (Optional) - You may wish to swap the service address (and/or) persistent address back by using C-SPOC.

» execute smitty hacmp. » Select Cluster System Management (C-SPOC) » Select HACMP Communication Interface Management » Select Swap IP Addresses between Communication Interfaces » Select appA#-svc from the ‘Available Service/Communication Interfaces’ smit

screen. » Select halifax#-if1 from the Swap onto Communication Interface smit screen

(This hint assumes that appA#-svc is currently an alias to halifax#-if2). » confirm the information displayed and press Enter)

__ 36. Using the console rather than a telnet session (because you will lose it), monitor the hacmp.out file on the halifax#-if1x (left) node and disconnect both network cables at the same time.

__ 37. There should be a network down event executed after a short period of time. What happens to the resource group on the halifax (left) node, and why?

» The resource group should have moved to the toronto# node » The reason for this is because selective fallover is invoked when a network

down event was detected on halifax#. HACMP moves the resource group to maximize it’s availability

__ 38. Check the /tmp/hacmp.out file on the toronto# node, it should also have detected a network failure.

__ 39. Restore both the network connections for the halifax# node. What event do you observe happens?

» network up

__ 40. Where is the resource Group at this time? Verify that the IP labels, volume groups, and file systems and application are available on that node.

» The resource group should still be on the toronto# node because the Fallback policy is Never Fallback.

» netstat -i » lsvg -o » /usr/es/sbin/cluster/utilities/clRGinfo » cat /tmp/appA.log

__ 41. You are now going to move resources back from one node to the other. On the halifax# node monitor the log. On the toronto# node execute smit clstop and stop the cluster services with the mode of takeover. Leave the default value for the other fields.

» execute “tail -f /usr/es/adm/cluster.log” for overview log




» execute “tail -f /tmp/hacmp.out” for detail log » smit clstop

__ 42. The clstat.cgi should change colors from green to yellow (substate unstable, toronto# leaving) and the state of the toronto# node and interfaces should change to red (down).

__ 43. All of the components in the resource group should move over to the halifax# node. Verify the IP labels, volume groups, and file systems on the halifax# node.

» netstat -i » lsvg -o » /usr/es/sbin/cluster/utilities/clRGinfo » cat /tmp/appA.log

__ 44. On the toronto# node restart HACMP. Observe the /tmp/hacmp.out file on the halifax# node and, of course, the clstat session. The resource group stays put.

END OF LAB




EXempty
You have a running cluster. Congratulations, now the fun really begins. Make sure clstat shows the cluster as stable with the tcp/ip and non-ip networks up.




EXempty
Exercise 8. Application Integration
(with Hints)


The HACMP cluster is now functional with a highly available filesystem and IP label. Adding the Web server to the scenario is the next step.



• Add the IBM Web server powered by Apache to the nodes • Adjust the configuration of the Web server to acknowledge the

highly available IP label • Introduce a minor configuration change to the Web server to use

the shared storage • Add an application start and stop script to HACMP • Test the application functionality

Introduction

The intention is not to become Web server programers but to simply add an existing application to the HACMP environment. This is to demonstrate one way to add an application to the HACMP environment.

Required Materials

A running cluster

The AIX 5L Expansion pack





Preface

• As part of this exercise, C-SPOC and DARE are used to enable the addition of filesystems, applications and resource changes to the cluster while it is running. If all things function as designed, no system reboots or HACMP restarts are required.


Part 1: Install the IBM Web server file system



__ 3. Create a new filesystem for the Web documents. Enter smit hacmp.

» Select System Management (C-SPOC) » Select HACMP Logical Volume Management » Select Shared Logical Volumes » Select Add a Shared Logical Volume » Select the resource group. » Select shared_vg_a » Select Auto-select. » Enter the menu information as follows: » number of Logical Partitions (LP)s -- 10 » The logical volume name is, shared_httplv » The lv type is jfs. » Press Enter to create the lv. » Using F3 key, traverse back to ‘HACMP Logical Volume Management’ » Select Shared File systems » Select Journaled File Systems » Select Add a Journaled File System on a Previously Defined Logical Volume » Add a standard JFS » Pick the shared_httplv entry from the pop-up menu » Enter /usr/HTTPServer/htdocs as the mount point. Leave the defaults for the

other values and press enter.

__ 4. Verify both nodes know about the new file system.

» cat /etc/filesystems (on both nodes)

__ 5. Continue on the halifax# node. Check to see that the filesystem is mounted on the system that currently owns the resource group (should be halifax#).

» df

Part 2: Install the IBM Web server software




EXempty
__ 6. Check if http filesets listed below are installed. If not, ask your instructor. On many class images they may be found in the directory /usr/sys/inst.images/web-appl. Otherwise you may need the AIX 5L Expansion Pack CD.
» http_server.base » http_server.admin » http_server.html

» lslpp -L http_server* » http_server.man (OPTIONAL man pages)

__ 7. On the other node (toronto#), repeat the previous step. Once installed, delete all of the information in the directory /usr/HTTPServer/htdocs (only on this node!).

» cd /usr/HTTPServer/htdocs » rm -r ./* » This is because when the filesystem fails over this will be covered by the

shared filesystem /usr/HTTPServer/htdocs.

__ 8. Go back to the halifax# node. In the directory /usr/HTTPServer/conf/, edit httpd.conf and change the “ServerName” variable to be the same as the service IP label (appA#-svc).

Note: The hostname must be resolvable, that is, host hostname should return a good answer. If the hostname is not resolvable, add the hostname to the 127.0.0.1 address as an alias. If in doubt, ask the Instructor. Remember to do this on both nodes otherwise successful takeover does not happen.

__ 9. Use ftp to put a copy of the /usr/HTTPServer/conf/httpd.conf file on the toronto# node.

» ftp toronto# » put /usr/HTTPServer/conf/httpd.conf /usr/HTTPServer/conf/httpd.conf.

Part 3: Configure HACMP for the Application

__ 10. Add the Application Server to HACMP.

» smitty hacmp » Select ‘Initialization and Standard Configuration’ » Select ‘Configure Resources to Make Highly Available’ » Select ‘Configure Application Servers’ » Select ‘Add an Application Server’ » Enter http_server as the application name. » Enter “/usr/HTTPServer/bin/apachectl start “as the application start script. » Enter “/usr/HTTPServer/bin/apachectl stop” as the application stop script.

__ 11. Change the appA_group to use the Application Server http_server.

» Use the F3 key to go back to the Initialization and Standard Configuration’ screen




» Select Configure HACMP Resource Groups » Select Change/Show Resources for a resource Group (Standard) » Select the ‘appA_group’ resource group. » Change (use F4) the Application Servers field to the value http_server. » Leave the Filesystems field blank, as the default is All » Using the F3 key, traverse back to the Initialization and Standard

Configuration smit screen. Select Verify and Synchronize HACMP Configuration

__ 12. While the synchronizing takes place, monitor the HACMP logs until you see the message start server http_server. Check to see that the Apache server started ok.

» ps -ef | grep http should show a number of httpd daemons running.

__ 13. From the client, start a new window in Netscape and connect to the URL http://appA#-svc. The Web screen Welcome to the IBM HTTP Server window should pop up.

__ 14. Perform a failover test by halting the Halifax# node in your favorite manner (for example, “halt -q” or “echo bye > /dev/kmem”).

__ 15. Wait for takeover to complete and verify what happens to the Web server. Use the page reload button on your Web browser to see if the Web server is really there.

__ 16. Bring up the Halifax# node again and start HACMP.

__ 17. What has happened to the Resource Group, and why?

» /usr/es/sbin/cluster/utilities/clRGinfo » cllsres -g appA_group » ps -ef | grep http » lsvg -o » » df -k » netstat -i

END OF LAB




EXempty
Optional Exercises
For the Web-enabled Candidates

__ 1. Change the Web server pages on the shared disk to prove the location of the data elements.

END OF LAB




EXempty
Exercise 9. Mutual Takeover
(with Hints)


This lab exercise expands the capabilities of the cluster. The intent is to completely add the resource group and all of its components while the cluster is running.



• Use the C-SPOC functionality and DARE capabilities of HACMP to make changes to the cluster while it is running

• Add a new volume group while the cluster is running

• Add all of the components of a shared file system while the system is running

• Add a resource group to the cluster and activate it while the cluster is running

• Test a mutual takeover configuration

Introduction

In the scenario there are two resource groups to be made highly available. The addition of the second resource group is done with the C-SPOC commands with the cluster running.





Preface

• Add the shared_res_grp_b resource group components according to the scenarios. This will require a second filesystem.


Part 1: Add a Second Resource Group and Filesystem to the Cluster

__ 1. Ensure HACMP is running on both nodes and that the HTTP application is running on halifax#

__ 2. Using the lspv command on BOTH nodes, verify that there is a shared disk hdiskX available with the same PVID. If so skip to step 8.

» lspv

__ 3. On the halifax# node make sure that the hdiskX has no PVID

» lspv » chdev -a pv=clear -l hdiskX

__ 4. Create a new PVID for hdiskX

» chdev -a pv=yes -l hdiskX

__ 5. On the toronto# node delete the hdisk.

» rmdev -dl hdiskX

__ 6. Add the disk back in.

» cfgmgr

__ 7. Verify the hdisk number and PVID agree between the two nodes.

» lspv on both nodes

__ 8. On the administrative node (halifax#) create a shared volume group called shared_vg_b using C-SPOC.

» Smitty hacmp » Select Initialization and Standard Configuration » Select Configure Resources to make Highly Available. » Select Configure Concurrent Volume Groups and Logical Volumes. » Select Concurrent Volume Groups » Select Create a Concurrent Volume Group » Select Select ALL (both) the Node Names that share the Volume Group » Select the PVID that you identified in step 2. » Fill out the volume group menu:

-Name= group shared_vg_b -Using F4 select the single physical volume you identified above




EXempty
-Check the Physical partition SIZE and major number (C-SPOC chooses a valid major number); set enhanced concurrent is true)
» Create the volume group. (on the development system there were some “ok” error messages after the successful “has been imported” message)

__ 9. Verify the Volume Group exists on both nodes.

» lspv » lsvg

Now that the volume group is created it must be discovered a resource group must be created and finally the volume group must be added to the resource group before any further C-SPOC utilities will access it.

__ 10. Discover the volume group using Extended Configuration in smitty hacmp.

__ 11. Create a resource group called appB_group with the toronto# node as the highest priority and halifax# node as the next priority.

» smitty hacmp » Select Initialization and Standard Configuration » Select Configure HACMP Resource Groups » Select Add a Resource Group » Enter the resource group name appB_group, from the planning worksheets.

The participating node names must also be entered -- enter toronto# first Take the defaults for the policies.

__ 12. Add the volume group to the resource group

» Return (F3) to the menu Configure HACMP Resource Groups then » Select Change/Show Resources for a Resource Group (standard) » Select appB_group » Enter the volume group name using F4.

__ 13. Synchronize the Cluster.

» smitty hacmp » Select Initialization and Standard Configuration » Select Verify and Synchronize HACMP Configuration

__ 14. Once synchronized, the Volume Group is varied online, on the owning node (toronto#). Wait for this to happen. Then on your administrative node halifax# use C-SPOC to add a jfs log shared logical volume to the shared_vg_b. The name should be shared_jfslog_b, the LV type should be jfslog, and use 1 PP.

» smitty hacmp » Select Initialization and Standard Configuration » Select Configure Resources to make Highly Available » Select Configure Volume Groups, Logical Volumes and Filesystems » Select Shared Logical Volumes » Select Add a Shared Logical Volume » From the list provided, choose the entry for appB_group shared_vg_b




» From the list provided, choose Auto-select » Set LP=1, NAME = shared_jfslog_b, LV TYPE = jfslog

__ 15. Format the jfslog so that it can be used by the filesystem that is created in the next few steps. If the log is not formatted, it is not used.

» On toronto#, execute logform /dev/shared_jfslog_b (answer yes to the destroy data question

__ 16. Back on halifax#, add a second shared logical volume with number of LOGICAL PARTITIONS=10, NAME= shared_jfslv_b, TYPE=jfs.

» see step 11 for details

__ 17. Add a shared file system on the previously created Logical volume called /shared_fs_b. Using the F3 key traverse back to Configure Volume Groups, Logical Volumes and Filesystems. Select Shared File Systems

» Return (F3) to Configure Volume Groups, Logical Volumes and Filesystems » Select Shared File Systems » Select Journaled File Systems » Select Add a Journaled File System on a Previously Defined Logical Volume » Select Add a Standard Journaled File System » Select the Logical Volume created in the previous step, press Enter. Fill in the

File System Mount Point (/shared_fs_b), hit enter. This information should be in the cluster planning worksheets.

__ 18. The filesystem should be available on node toronto# in a few minutes.

The following was observed during additional testing and may or may not be repeatable: a message on the smit panel saying that shared_fs_b is not a known file system, and the failed response was posted. However, when I looked at /etc/filesystems it was there and a manual mount from the Toronto node worked. I was then able to move the resource group from one node to another and back using the system management (C-SPOC) menu.

» tail -f /tmp/hacmp.out or clstat and wait for the cluster to become stable.

Part 2: Create the application and service label resources


__ 20. Create the application start script:echo ‘hostname>>/shared_fs_b/appB.log’ >/tmp/appB_startecho ‘date +” starting:%r” >> /shared_fs_b/appB.log’>>/tmp/appB_start

__ 21. Create the application stop script echo ‘hostname>>/shared_fs_b/appB.log’ >/tmp/appB_stopecho ‘date +” stopping:%r” >> /shared_fs_b/appB.log’>>/tmp/appB_stop

__ 22. ftp the scripts to the other node

» ftp toronto#




EXempty
» put /tmp/appB_start /tmp/appB_start » put /tmp/appB_stop /tmp/appB_stop
__ 23. Make the scripts executable on both nodes:

» chmod +x /tmp/appB_start » chmod +x /tmp/appB_stop » REPEAT on the other node

__ 24. On halifax#, create the Service IP label resource.

» smitty hacmp --> Initialization and Standard Configuration » Select Configure Resources to make Highly Available » Select Configure Service IP Labels/Addresses » Select Add a Service IP Label/Address » Enter the Service IP Label appB#-svc (use F4) and the network (use F4)

__ 25. Create the application server resource

» Return (F3) to the menu Configure Resources to make Highly Available » Select Configure Application Servers » Select Add an Application Server » Enter appB for the server name and the full path names of the start and stop

scripts.

__ 26. Add the resources to the resource group

» Return (F3) to the menu Initialization and Standard Configuration » Select Configure HACMP Resource Groups » Select Change/Show Resources for a Resource Group (standard) » Select appB_group » Enter the service label and application server name using F4.

__ 27. Synchronize the Cluster. Using the F3 key, traverse back to Initialization and Standard Configuration.

» select Verify and Synchronize HACMP configuration

__ 28. Test that the toronto# service IP label is available.

» netstat -i

__ 29. Test the new resource group on the toronto# node for network adapter swap/failure and node failure.

» Fail the toronto# node in your favorite manner.

__ 30. OPTIONAL -- If you have an extra disk execute the mirrorvg and chfs commands to test splitting off a copy as presented in the unit 3 lecture. Note that this step could be done using C-SPOC to create the mirror. Also note that one purpose of this step is to show how to undo the backup copy.

» stop cluster on toront# » smit vg (set characteristics, add disk to vg)




» smit vg (mirrorvg) » ensure /shared_fs_b is mounted » chfs (see man page example) » lsvg -l shared_vg_b (see new logical volume/filesystem) » lslv -p hdiskX | grep USED (on both disks -- look for stale) » umount the new file system » rmfs » on disk with stale redeo the lslv command (see stale removed) » start cluster on toront#.

END OF LAB




EXempty




EXempty
Exercise 10. HACMP Extended Features
(with Hints)


This lab exercise expands the capabilities provided in the extended features option.



• Create a Cluster ‘Snapshot’

• Use the C-SPOC functions and DARE capabilities of HACMP to make changes to the cluster while it is running

• Add additional Resource Groups while the cluster is running

• Add additional Service aliases

• Modify Resource Group Behavior Policies

• Configure Settling and Fallback timers

Introduction

To enhance the scenario create two additional resource groups to be made highly available. The addition of these resource groups and their behavior modification is done with the C-SPOC commands with the cluster running.





Preface

• Add additional service aliases, and create an additional custom resource group for each node.

• Modify the default start and fallback policies of the new resource groups to examine the resource behavior during cluster startup and reintegration event processing.

• Create • All hints are marked by a » sign.

Part 1: Create a Cluster Snapshot

__ 1. In the last exercise we made the goal of the class so let’s save our environment before continuing by creating a Cluster Snapshot

» smitty hacmp » Select, Extended Configuration » Select, Snapshot Configuration » Select, Add Cluster Snapshot » Enter snapshot name and description. Example: name= exercise 10,

description = mutual takeover It is helpful to use a meaningful name. When you refer back to these it can help identify why/when it was taken. This could be helpful should a cluster restore become necessary.

Notice:There are two snapshot files <snapshot>.odm and <snapshot>.infoThe directory for the snapshot is /usr/es/sbin/cluster/snapshotsThe clsnapshotinfo command was run on both nodes (output in the “.info” file)

__ 2. Read the mutual_takover.info file. Go on to the next step when you are ready.

Part 2: Add an additional Service alias and Resource Group to each Cluster Node

__ 3. Log in to the halifax# node as root.

__ 4. Add two additional service labels to the /etc/hosts file

» 192.168.#3.21 appC#-svc » 192.168.#3.22 appD#-svc » ftp /etc/hosts file to your other node.

__ 5. Discover these new addresses in HACMP using the Extended Configuration menu from smit hacmp.

__ 6. Configure the two additional HACMP Service IP Labels/Addresses as resources in HACMP.

» smitty HACMP » Select Extended Configuration




EXempty
» Select Extended Resource Configuration » Select HACMP Extended Resources Configuration » Select Configure HACMP Service IP Labels/Addresses » Select Add a Service IP Label/Address » Select Configurable on Multiple Nodes » Select the network name. » Enter the Service IP Label
Note: An Alterable Hardware Address may not be selected. Remember that HWAT is not supported for networks using IPAT via IP Aliasing.

We now add two resource groups appC_group and appD_group with different startup policies. The first (appC_group) behaves like the old inactive takeover, and the second (appD_group) behaves like the old rotating.

__ 7. Add an additional Resource Group called appC_group with a home node of halifax# and a startup policy of Online on First Available node.

» Return (F3) to the menu Extended Resource Configuration » Select HACMP Extended Resource Group Configuration » Select Add a Resource Group » Enter the Resource Group Name, nodes (halifax# first), and change the start

policy to be Online on First Available Node

__ 8. Add another Resource Group called appD_group with a home node of toronto# and a startup policy of Online Using Distribution Policy.

» Return (F3) to the menu HACMP Extended Resource Group Configuration » Select Add a Resource Group » Enter the Resource Group Name appD_group, the nodes with toronto# first,

and the startup policy of Online Using Distribution Policy.

__ 9. Now add the Service IP Labels created in step 1 to the Resource Groups just created. Using the F3 key, traverse back to the Extended HACMP Resource Group Configuration and select Change/Show Resources and Attributes for a Resource Group

» Return (F3) to the menu HACMP Extended Resource Group Configuration » Select, Change/Show Resources and Attributes for a Resource Group » Select the correct Resource Group form the list (appC_group). » Add the correct Service IP Label and press Enter (appC1-svc). » Repeat this step adding the appD1-svc Service IP Label to the appD_group

Resource Group.

__ 10. In order to mimic the old rotating we need to change the distribution policy to network. This is done using the smit extended runtime menu Configure Distribution Policy for Resource Groups. The cluster must be stopped on both nodes first.

» Exit smit » hacmp clstop » Select both nodes




» Exit smit » Verify cluster stopped (lssrc -g cluster on both nodes) » smit hacmp » Select Extended Configuration » Select Extended Resource Configuration » Select Configure Resource Group Run-Time Policies » Select Configure Distribution Policy for Resource Groups » Change the value to network (notice the deprecated message).

__ 11. Synchronize the cluster. Using the F3 key, traverse back to the ‘Extended Configuration’ smit screen.

» Select, Extended Verification and Synchronization » Notice the menu option Automatically correct errors found during verification.

You only see this option when the cluster is down.

Part 3: Test Resource Group Behavior

__ 12. Start HACMP only on your toront# node.

» smitty clstart -- or use C-SPOC menu ‘Manage HACMP Services’ » Select toronto# only.

__ 13. Once the node is stable check the status of the Resource Groups. Does this look normal? If not, what is wrong -- Should appC_group be online on toronto#?

» /usr/es/sbin/cluster/utilities/clRGinfo » /usr/es/sbin/cluster/utilities/clRGinfo -v appC_group

__ 14. Start HACMP on the Halifax (left) node.

» Refer to step 10 hints

__ 15. Once the node is stable check the status of the Resource Groups. Does everything look correct now (check the appC_group)? If so, what changed? Why?

__ 16. OPTIONAL: To understand better the distribution policy, stop the nodes and bring up halifax# first and see what happens to the appD_group. Then stop halifax# with takeover. Then restart halifax# and see what happens to the appD_group.

Let’s have a look at configuring a settling timer which allows you to modify the behavior of the Fallback To Higher Priority Node In The List fallback policy so that there are not two online operations if you bring up the secondary node first.

Part 4: Add a Settling Timer

__ 17. Ensure that you are on your administration node (halifax#) and configure a Settling Timer (can only be used if startup policy is ‘Online On First Available Node’ ).

» smitty hacmp » Select, Extended Configuration




EXempty
» Select, Extended Resource Configuration » Select, Configure Resource Group Run-Time Policies » Select, Configure Settling Time for Resource Groups » Set a value (seconds) for the settling time. For this lab use 360.
__ 18. Synchronize the Cluster. Using the F3 key, traverse back to the ‘Extended Configuration’ smit screen. Notice in the smit output the messages about the settling timer value.

» Select Extended Verification and Synchronization

Part 5: Testing Cluster Behavior using a Settling Timer

__ 19. On your administrative node (halifax#), stop Cluster processing on both nodes.

» smitty clstop -- or use C-SPOC menu ‘Manage HACMP Services’ (choose both nodes)

__ 20. Wait 2 minutes and then start HACMP -- only on the toronto# node.

» smitty clstart -- or use C-SPOC menu ‘Manage HACMP Services’ (choose only toronto# )

__ 21. Wait until you can see that the appB_group is online then verify that appC_group is still offline on toronto# using the clRGinfo command (note that the clRGinfo command can be run from either node as long as HACMP is started on any one of the nodes)

» /usr/es/sbin/cluster/utilities/clRGinfo appB_group -- wait until you see online » clRGinfo appC_group -- should be offline


__ 23. Verify that the appC_group comes online on halifax# (without first being online on toronto#). As you can see the purpose of the settling timer is to prevent the resources from being immediately acquired by the first active node.

» /usr/es/sbin/cluster/utilities/clRGinfo (compare appA_group and appC_group -- both should come online eventually)

__ 24. OPTIONAL -- repeat this part but wait for settling time to expire after starting the cluster on toronto#. Verify that appC_group comes online on toronto#. Stop the cluster manager on both nodes, wait 2 minutes, start the cluster manager on both nodes.

Part 6: Configure Delayed Fallback Timer

__ 25. Cluster should be started on both nodes and appC_group should be online on halifax#.

» lssrc -g cluster » clRGinfo




__ 26. On your administrative node (halifax#), create a delayed fallback timer policy for 30 minutes from now (instructor may modify this time)

» Write down the current time ______________ » Make sure both nodes are using the same time (setclock toronto#-if1). » smitty hacmp » Extended Configuration... Extended Resource Configuration » --> Configure Resource Group Run-Time Policies » --> --> Configure Delayed Fallback Timer Policies » --> --> --> Add a Delayed Fallback Timer Policy » use the following values:

daily, name=my_delayfbt, hour/min=30 min from current time

__ 27. Add the fallback timer policy the resource group appC_group

» smitty hacmp » Extended Configuration... Extended Resource Configuration » ... HACMP Extended Resource Group Configuration » ... ... Change/Show Resources and Attributes for a Resource Group » Select appC_group » Fill in the field below as indicated: » Fallback Timer Policy [my_delayfbt] (use F4)

__ 28. Synchronize

Part 7: Testing Cluster Behavior using a Delayed Fallback Timer

__ 29. Verify that appC_group is online on halifax# using the clRGinfo command

» clRGinfo appC_group

__ 30. Stop the cluster manager only on halifax# with takeover

» smitty clstop or use the C-SPOC menu -->‘Manage HACMP Services’

__ 31. Verify that appC_group is now online on toronto# (clRGinfo).

__ 32. Wait 2 minutes (required before a restart)

__ 33. Start the cluster manager on halifax#

» smitty clstart -- or use the C-SPOC menu -->‘Manage HACMP Services’

__ 34. Monitor the cluster from toronto#. In /tmp/hacmp.out at the event summary for check_for_site_up_complete halifax#, there is now a message stating the fallback time. Make sure appC_group is still on toronto# before the fallback then tail -f the hacmp.out file and wait for the fallback to occur.

» vi /tmp/hacmp.out -- look for the fallback time message-- should be very near the bottom.

» - clRGinfo (verify appC_group is online on toronto# -- before the fallback time)

» - tail -f /tmp/hacmp.out and wait




EXempty
__ 35. At the time set for the Delayed Fallback Timer, appC_group should move back to halifax# (you should see activity from tail command)
» Execute clRGinfo to verify that appC_group is online on halifax#.

__ 36. On your administrative node (halifax#), remove the name of the Delayed Fallback Timer (my_delayfbt) from the resource group appC_group (you can keep the policy definition if you want).

» smitty hacmp » Extended Configuration... Extended Resource Configuration » ... HACMP Extended Resource Group Configuration » ... ... Change/Show Resources and Attributes for a Resource Group

__ 37. Reset the Settling time to 0 (from the menu ‘Configure Resource Group Run-Time Policies’

» smitty hacmp » Select, Extended Configuration » Select, Extended Resource Configuration » Select, Configure Resource Group Run-Time Policies

__ 38. Synchronize.

END OF LAB




EXempty
Exercise 11. IPAT via Replacement and HWAT
(with Hints)


This lab explores the options of removing a cluster and creating an IPAT via replacement environment.

This lab also examines Gratuitous Arp, and the use of Hardware Address Takeover (HWAT), functionality for environments where Gratuitous Arp may not be the best solution.



• Describe how to set up IPAT via replacement

• Describe to behavior of Arp updates/refreshes using gratuitous Arp or Hardware Address Takeover where required.

• Describe how to set up HWAT





Preface

• The first part shows how to remove a cluster • The second part of this lab looks at setting up IPAT via replacement and using the

standard configuration path to build a cluster. • The third part of this lab looks at Gratuitous arp. • The fourth part of this exercise adds hardware address concepts. HWAT or MAC

address takeover would be used in situations where gratuitous arp may not be supported, as in older hardware, or non-standard operating systems.


Part 1: Remove and add a new Cluster

__ 1. On your administration node (halifax#), stop both the cluster nodes.

» smitty clstop -- choose both nodes

__ 2. Snapshot

» smitty hacmp » Select, Extended Configuration » Select, Snapshot Configuration » Select, Add Cluster Snapshot » Enter snapshot name and description. Example: name= exercise 11,

description = added resource groups.

__ 3. remove cluster

» smitty hacmp » ... Select Extended Configuration » ... Select Extended Topology Configuration » ... Select Configure an HACMP Cluster » ... Select Remove an HACMP Cluster » REPEAT on the other node » echo ““ >/usr/es/sbin/cluster/etc/rhosts (double quotes with no space) » REPEAT on the other node

__ 4. Add a replacement service address to your /etc/hosts file (must be on the same subnet as one of the if interfaces.

» vi /etc/hosts » add 192.168.#1.10 appR#-repl (# is your team number - remember?) » document this address in you component worksheets (exercise 2). » REPEAT on (or send to) your other node

__ 5. Configure a new cluster on halifax#. Go to the HACMP for AIX smit panel and select Initialization and Standard Configuration.

» Initialization and Standard Configuration




EXempty
» Select, Add Nodes to an HACMP Cluster » Enter a cluster name (for example, team#). » Using the F4 List option, select the appropriate communications paths for
BOTH the nodes (that is, the two “-if1” interfaces).

__ 6. Use Extended Configuration to set the network to turn off IPAT via aliases.

» From the main menu, select Extended ConfigurationSelect » Select, Extended Topology Configuration » Select, Configure HACMP Networks » Select, Change/Show a Network in the HACMP Cluster » Select the IP network, press Enter » Change the option Enable IP Address Takeover via IP Aliases to No, hit

enter.

__ 7. Use Extended Configuration to configure a non-IP network by choosing the pair of devices that will make up the network (Using the F3 key, traverse back to the Extended Topology Configuration smit screen.

» Select, Configure HACMP Communications Interfaces/Devices » Select, Add Communication Interfaces/Devices » From the list, select, Add Discovered Communication Interface and Devices » From the list select, Communication Devices » Select the appropriate pair of devices, hdisk/hdisk (or tty/tty or tmssa/tmssa)

__ 8. Redo the Persistent Addresses from your planning worksheet. (Using the F3 key, traverse back to the Extended Topology Configuration smit screen).

» Configure HACMP Persistent Node IP Label/Addresses » Select, Add a Persistent Node IP Label/Address » Select the appropriate node from the list, press Enter » The F4 List option is available to select both the Network Name and

Persistent IP Label/Address. » REPEAT for other node

__ 9. Create the Service IP Label resource. Using the F3 key, traverse back to the ‘Extended Configuration’ smit screen. Select ‘Extended Resource Configuration’.

» Select, HACMP Extended Resources Configuration » Select, Configure HACMP Service IP Labels/Addresses » Select, Add a Service IP Label/Address » Select Configurable on Multiple Nodes, press Enter. » Select the Network Name. » Finally, select the Service IP Label (appR#-repl) to be used.

__ 10. Create a resource groups. Using the F3 key, traverse back to the Extended Resource Configuration smit screen.

» Select HACMP Extended Resource Group Configuration » Select Add a Resource Group




» Enter the Resource Group name (appR_group) and set the participating nodes (use F4 to choose the nodes). Remember the priority order, the first node listed, is considered the ‘home or owner’ node. Use the default policies

__ 11. Add Resources to the Resource Group. Using the F3 key, traverse back to the HACMP extended Resource Group Configuration smit screen.

» Select, Change/Show Resources and Attributes for a Resource Group » In the appropriate fields, use F4 5o choose a Service IP Label and a Volume

Group. For the purposes of this lab, application servers are not required. You may add them if you wish.

__ 12. Synchronize the cluster. Using the F3 key, traverse back to the Extended Configuration smit screen. Select Extended Verification and Synchronization

» Review the results for any errors.

__ 13. Start HACMP on the toronto# node.

» smitty clstart (choose only toronto# and start clinfo) » monitor the /tmp/hacmp.out during startup.

__ 14. Verify the appR_group did not come online because of the startup policy.

» /usr/es/sbin/cluster/clRGinfo


» smitty clstart (choose only halifax# and start clinfo)

__ 16. Verify that the appR_group is online on halifax#

» /usr/es/sbin/cluster/clRGinfo

Part 2: Gratuitous ARP

From the AIX 5L Version 5.1 commands reference for ifconfig. “Gratuitous ARP is supported for ethernet, token-ring, and FDDI interfaces. This means when an IP address is assigned, the host sends an ARP request for its own address (the new address) to inform other machines of its address so that they can update their ARP entry immediately. It also lets hosts detect duplicate IP addresses.”

This will make it a little difficult to create a failure with AIX clients but the tests are valid.

__ 17. Log on the client machine. Verify that clinfo has not been started.

» ps -ef | grep -i clinfo | grep -v grep

__ 18. Use the ping command to test the service IP Label of appR_group on halifax#).

» ping -c 1 appR#-svc

__ 19. Check the contents of the arp cache.

» arp -a




EXempty
__ 20. On the halifax# node generate a swap adapter event. Be aware that you need to do this fairly quickly before the arp cache times out.
» ifconfig enX down (the interface on which the service label is configured).

__ 21. Check the contents of the arp cache on the client, compare the results with the previous iteration of the command.

» arp -a

__ 22. The hardware address should have updated in the arp cache on the client without any intervention.

Note: If the entry is not in the arp cache when the Gratuitous arp is broadcast it is ignored.

Part 3: Hardware Address Takeover

In this scenario the router in Regina is a bit of an antique and does not support gratuitous ARP. It was highlighted as a problem since the ARP cache retention is 15 minutes. This problem was discovered during the preliminary cluster testing.

__ 23. On the halifax# node log in as root.

__ 24. Identify the interface that is reconfigured with the appR-repl service address and write the mac address here._____________________________________

» netstat -i » There should be 12 digits and no periods. When using the netstat command

leading 0s before each period are omitted. You must put them back in.

__ 25. Identify the alternate mac address. To specify an alternate hardware address for an Ethernet interface, change the 1st byte xx to 4x j_________________________________________

__ 26. Change the appR-repl service IP label to add an alternate hardware address in the field.

» smitty hacmp » Select Extended Configuration » Select Extended Resource Configuration » Select HACMP Extended Resource Configuration » Select Configure Service IP Labels/Addresses » Select Change/Show a Service IP Label/Address » Select appR-repl. » Create the Alternate Hardware Address using the answer to the previous

step and the description in the statement of this step or see the example from the configuration Lecture.

__ 27. Synchronize the cluster. Using the F3 key, traverse back to the ‘Extended Configuration’ smit screen. Notice the following message in the smit log: cldare: Detected changes to service IP label appR1-repl. Please note that changing




parameters of service IP label via a DARE may result in releasing resource group appR_group.

» Select “Extended Verification and Synchronization’ » clRGinfo (shows appR_group offline.

__ 28. Bring the appR_group online using the C-SPOC menu. If, on the client, there is no arp cache entry for the appR-repl service address, then ping the appR-repl service address.

» Select C-SPOC from the main smit hacmp menu » Select HACMP Resource Group and Application Management » Select Bring a Resource Group Online » Select ‘appR_group offline halifax#’ » BE CAREFUL -- select Restore_Node_Priority_Order » Accept the next menu

__ 29. Verify that the alternate hardware address is now configured on the interface for the appR#-repl service address.

» netstat -i


__ 31. Check that the halifax service address is on the toronto# node and observe the hardware address associated with that service address

» netstat -i

Part 4: Re-create a Cluster from a Snapshot

__ 32. Ensure the cluster manager is stopped on both clusters nodes.

» smitty clstop

__ 33. Apply the snapshot that contains all the cluster definitions you made in exercise 10.

» smitty hacmp » Select Extended Configuration » Select Snapshot Configuration

__ 34. Start HACMP

» smitty clstart (choose both nodes)

__ 35. For each resource group, verify to yourself that you understand how the online node was chosen.

» clRGinfo


» smitty clstop, select takeover.

__ 37. Restart the failed node and observe the re-integration. Verify that you understand how the online node was chosen for each of the resource groups




EXempty
END OF LAB




This exercise looked at cascading resource groups, and how to configure both cascading without Fallback and Inactive Takeover. It also covered setting up and testing Hardware Address Takeover.

This exercise also looked at rotating resource groups.




EXempty
Exercise 12. Network File System (NFS)
(with Hints)


This lab covers a couple of different methods for configuring network filesystems with HACMP. It also demonstrates how to set various NFS options in HACMP exported filesystems.



• Have HACMP export a filesystem as part of a resource group

• Have HACMP import a filesystems as part of resource group

• Modify the NFS export options for the exported filesystem

• Add an NFS cross-mount

• Modify the NFS cross-mount for performance and flexibility





Preface



Part 1: NFS exports in a resource group

__ 1. Assumptions: You need to start this exercise off with HACMP up on both nodes, and identify two resource groups -- one whose home node is halifax# (that is, appA_group) and the other whose home node is toronto# (that is, appB_group). Each group should have a shared filesystem defined to it (that is, shared_fs_a and shared_fs_b). On each node, verify that nfs is running (lssrc -g nfs) after HACMP is started.

__ 2. Modify the resource group appA_group to add /shared_fs_a as a filesystem/directory to NFS export and set to true the option ‘Filesystems mounted before IP configured’

» smitty hacmp » Select Extended Configuration » Select Extended Resource Configuration » Select HACMP Extended Resource Group Configuration » Select Change/Show Resources and Attributes for a Resource Group » Select the resource group from the list, press Enter » Modify the options Filesystems mounted before IP configured and

Filesystems/Directories to Export.

__ 3. Modify the resource group appB_group to add /shared_fs_b as a filesystem/directory to NFS export. Using the F3 key, traverse back to the ’HACMP Extended Resource Group Configuration’.

» See the previous step.

__ 4. Synchronize the resources. Using the F3 key, traverse back to ‘Extended Configuration’ smit screen.

» Select Extended Verification and Synchronization

__ 5. When the reconfiguration of resources has completed on each node, check the directories are exported through NFS.

» lsnfsexp You see only what is exported from THIS node » cat /etc/xtab Should look like the output from lsnfsexp. Note /etc/exports is

not used by HACMP-- see what happens when you try to cat /etc/exports

__ 6. Log in on the client as root.

__ 7. Create a directory /halifax and /toronto.




EXempty
» mkdir /halifax » mkdir /toronto
__ 8. On the client, using the service address for the appA_group, mount the nfs exported directory /shared_fs_a on the local directory /halifax.

» smitty mknfsmnt

__ 9. On the client, using the service address for the appB_group, mount the nfs exported directory /shared_fs_b on the local directory /toronto.

» smitty mknfsmnt

__ 10. Verify the nfs directories are mounted where intended.

» mount » df -k

__ 11. Back on a cluster node -- fail one of the nodes in your favorite manner. Verify that the nfs directories are still exported on the remaining node and mounted on the client system.

» lsnfsexp on the remaining cluster node » df on the client system

__ 12. Try to create a file in the /halifax directory. It should not work. Lets see how this can be addressed.

Part 2: Modifying the NFS Export Options

__ 13. The output of the lsnfsexp command on the nodes explains that only the cluster nodes can use user root. To change this we create an override file. Its name is /usr/es/sbin/cluster/etc/exports. HACMP uses this file to update the /etc/xtabs file used by NFS.

__ 14. On the running node, use the lsnfsexp command to copy the current /etc/xtabs file to the HACMP file and then modify the HACMP file using the following commands:

- lsnfsexp > /usr/es/sbin/cluster/etc/exports - Edit /usr/es/sbin/cluster/etc/exports and add the client to the list of hosts - Save the file - ftp the file to the other node

__ 15. Restart the failed node.

» The exports file should be used when NFS remounts the directory.

__ 16. From the client try to create a file in the nfs directory on the client of the node you have just restarted.

» Use the touch command




Part 3: NFS Cross-mount within the Cluster

__ 17. On both nodes create a directory /hanfs.

__ 18. Edit the resource group appA_group and add the following to the option ‘Filesystems/Directories to NFS mount” /hanfs;/shared_fs_a. This will mount the /shared_fs_a nfs filesystem on the mount point /hanfs for all systems in that resource group.

__ 19. Synchronize the resources and verify this is true on both nodes.

__ 20. Fail the toronto# node in your favorite manner.

__ 21. Confirm that halifax# node has all the resource groups, and that the NFS mounts are OK.

» /usr/es/sbin/cluster/utilities/clRGinfo » df -k

END OF LAB




EXempty
This exercise looked at various methods of implementing NFS in an HACMP cluster.




EXempty
Exercise 13. Error Notification
(with Hints)


This lab covers the adding of error notifications into AIX through the HACMP smit screens.



• Add an error notification for the loss of quorum on a volume group

• Emulate the error condition and test the error notification method

• Optionally add another error notification based on filesystems full


© Copyright IBM Corp. 1998, 2004 Exercise 13. Error Notification 13-1



Preface

• This exercise looks at Automatic Error Notification. Before you configure Automatic Error Notification, you must have a valid HACMP configuration. Using the SMIT options, you can use the following methods: - Configure Automatic Error Notification - List Automatic Error Notification - Remove Automatic Error Notification.

• Remember that Error Notification is a function of AIX - HACMP just gives you the smit screens that make it easier to enter error notification methods.


Setting Up the automatic error notifications on the halifax# node.


__ 2. Stop the Cluster. The cluster must be down to configure Automatic Error Notification.

__ 3. Configure Automatic Error Notification.

» smitty hacmp » Select Problem Determination Tools » Select Hacmp Error Notification » Select Automatic Error Notification » Select Add Error Notify Methods for Cluster Resources. The error notification

methods are automatically configured on all relevant cluster nodes.

When you run automatic error notification, it assigns two error methods for all the error types noted:

cl_failover is assigned if a disk or network interface card is determined to be a single point of failure, and that failure would cause the cluster to fall over. If there is a failure of one of these devices, this method logs the error in hacmp.out and shuts the cluster node down. A graceful stop is attempted first, if this is unsuccessful, cl_exit is called to shut down the node.

cl_logerror is assigned for any other error type. If there is a failure of a device configured with this method, they are logged in hacmp.out.

__ 4. List Error Notification Methods. Use the F3 key to traverse back to the ‘Configure Automatic Error Notification’ smit screen.

__ 5. To see the AIX odm file, execute the command odmget errnotify | more. The HACMP generated stanzas will be at the bottom.

END OF LAB




AP
Appendix A. Cluster Diagrams




AU54 lab teams

REPLACE # with team number

team number= ______


if1 halifax#-if1 192.168.#1.1 _____________

if2 halifax#-if2 192.168.#2.1 _____________

Persist halifax#-per 192.168.#3..1

Resource Group appA_group contains

Volume Group= shared_vg_a

hdisks = ______________

Major # = ______________

JFS Log =shared_jfslog_a

Logical Volume =shared_jfslv_a

FS Mount Point =/shared_fs_a

Home Node Name toronto#Resource Group appB_groupStartup Policy =OHNOFallover Policy =FONPFallback Policy =FBHPService IP Label =appB#-svc 192.168.#3.20Application server =appB

Label =toronto#_hiskY_01Device =/dev/hdiskX

Label =toronto_tty0_01Device =/dev/tty0

rootvg

4.8 GB

user

community

Network = ________________ (netmask = ___.___.___.___)

Home Node Name halifax#Resource Group = appA_groupStartup Policy =OHNOFallover Policy =FONPFallback Policy =FBNFService IP Label =appA#-svc 192.168.#3.10Application server =appA

Label =halifax#_hiskX_01Device =/dev/hdiskX

Label =halifax#_tty0_01Device =/dev/tty0

VG =

rootvg

4.8 GB

VG =

serial

serial


if1 toronto#-if1 192.168.#1.2 _____________

if2 toronto#-if2 192.168.#2.2 _____________

Persist toronto#-per 192.168.#3.2

Resource Group appB_group contains

Volume Group= shared_vg_b

hdisks = ______________

Major # = ______________

JFS Log = shared_jfslog_b

Logical Volume = shared_jfslv_b

FS Mount Point = /shared_fs_b

client

regina#

if1192.168.#1.3

alias 192.168.#3.30




AP



if1 _______ _________ _______________

if2 _______ _________ _______________

Persist _______ _________


Volume Group= ______________

hdisks = ______________

Major # = ______________

JFS Log = ______________


FS Mount Point = ______________


Service IP Label =


Label = Device =

Label = Device =

rootvg

4.8 GB

user

community

Network = _____________netmask=___.___.___.___


Service IP Label =

Application server=

Label =Device =

Label =Device =

VG =

rootvg

4.8 GB

VG =

serial

serial


if1 _______ _________ _______________

if2 _______ _________ _______________

Persist _______ _________


Volume Group= ______________

hdisks = ______________

Major # = ______________

JFS Log = ______________


FS Mount Point = ______________

client


if1 _________________

svc alias ____________






if1 _______ _________ _______________

if2 _______ _________ _______________

Persist _______ _________


Volume Group= ______________

hdisks = ______________

Major # = ______________

JFS Log = ______________


FS Mount Point = ______________


Service IP Label =


Label = Device =

Label = Device =

rootvg

4.8 GB

user

community

Network = _____________netmask=___.___.___.___


Service IP Label =

Application server=

Label =Device =

Label =Device =

VG =

rootvg

4.8 GB

VG =

serial

serial


if1 _______ _________ _______________

if2 _______ _________ _______________

Persist _______ _________


Volume Group= ______________

hdisks = ______________

Major # = ______________

JFS Log = ______________


FS Mount Point = ______________

client


if1 _________________

svc alias ____________



V1.2.2

backpg

Aix Hacmp Cluster

Documents

Transcript of Aix Hacmp Cluster