Kraemer Gpfs Aix Faq
Transcript of Kraemer Gpfs Aix Faq
![Page 1: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/1.jpg)
GPFS for AIX 5L
Questions and
Answers ���
Overview
The General Parallel File System (GPFS) for AIX 5L™ is a high-performance shared-disk file system that
can provide data access from nodes in an AIX® cluster environment. Parallel and serial applications can
readily access shared files using standard UNIX® file system interfaces, and the same file can be accessed
concurrently from multiple nodes. GPFS is designed to provide high availability through logging and
replication, and can be configured for failover from both disk and server malfunctions.
GPFS has been widely accepted on the IBM® RS/6000® SP™ system to scale file system I/O which can
help meet the requirements of a wide range of applications including seismic data processing, digital
library file serving, and data mining in business intelligence.
GPFS for AIX 5L supports the IBM ERserver Cluster 1600 as well as clusters of IBM Eserver pSeries®
nodes or IBM Eserver p5 nodes.
It is suggested you always check GPFS common FAQ question What is the current service information for
GPFS?
Updates to this FAQ include:
Date Question
October 2004 What disk support guidelines must be followed when running GPFS in an sp cluster type?
What disk support guidelines must be followed when running GPFS in an rpd cluster type?
What are the disk support guidelines that must be followed when running GPFS in an hacmp
cluster type?
What disks support hardware fencing?
What versions of GPFS support EtherChannel?
IBM TotalStorage® FAStT has been renamed IBM TotalStorage DS4000 series
DS4100 formerly FAStT100
DS4300 formerly FAStT600
DS4300 Turbo formerly FAStT600 Turbo
DS4400 formerly FAStT700
DS4500 formerly FAStT900
||||
![Page 2: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/2.jpg)
Questions & Answers
1. What is an sp cluster type?
2. When running GPFS in an sp cluster type, what are the software requirements?
3. What disk support guidelines must be followed when running GPFS in an sp cluster type?
4. What is an rpd cluster type?
5. When running GPFS in an rpd cluster type, what are the software requirements?
6. What disk support guidelines must be followed when running GPFS in an rpd cluster type?
7. What is an hacmp cluster type?
8. When running GPFS in an hacmp cluster type, what are the software requirements?
9. What are the disk support guidelines that must be followed when running GPFS in an hacmp
cluster type?
10. How can I implement disaster recovery for my GPFS cluster?
11. What are the coexistence rules for GPFS clusters?
12. What type of disk fencing mechanisms does GPFS support?
13. How can the fencing technique in operation for a cluster be verified?
14. What is AIX MPIO and does GPFS support it?
15. What disks support hardware fencing?
16. What should I do if commands such as mmcrlv, mmcrfs, mmadddisk, mmrpldisk, or mount fail to
complete due to problems with non-IBM disks?
17. What cluster specific single-node quorum restrictions are there?
18. Why would I want to use EtherChannel in my GPFS cluster?
19. What versions of GPFS support EtherChannel?
20. What’s the difference between using an ESS with or without SDD or SDDPCM installed on the host?
21. Is a Fibre Channel Switch required for all Fibre Channel disk attachments which are used by GPFS?
22. What Fibre Channel Switches are qualified for GPFS usage and is there a FC Switch support chart
available?
23. What performance tuning suggestions are there for AIX nodes?
24. What configuration and performance tuning suggestions are there for GPFS when used primarily for
Oracle databases?
25. What is the largest file system size supported by GPFS?
26. Are there any special concerns regarding the use of mksysb images?
27. Are there any considerations when using NIS automount?
28. How can I ask a more specific question about GPFS?
2
![Page 3: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/3.jpg)
Q1: What is an sp cluster type?
A1: The GPFS cluster type sp is based on the IBM Parallel System Support Programs (PSSP) licensed
product and the shared disk concept of the IBM Virtual Shared Disk component of PSSP. In the
GPFS cluster type sp (a PSSP environment), the nodes which are members of the GPFS cluster
depend on the network switch type being used. In a system with an SP Switch, the GPFS cluster
is equal to all of the nodes in the corresponding SP partition which have GPFS installed. In a
system with an SP Switch2, the GPFS cluster is equal to all of the nodes in the system which
have GPFS installed. That is, the cluster definition is implicit and there is no need to run the
GPFS cluster commands. Within the GPFS cluster, you define one or more nodesets within which
your file systems operate. Disk requirements within your GPFS sp cluster require that all disks
must be:
1. Attached to a virtual shared disk server within the nodeset.
2. Processed via either the mmcrvsd command or the directions in the Parallel System Support
Programs for AIX: Managing Shared Disks book to create virtual shared disk useable by GPFS.
3. Conform to the disk support guidelines listed in this FAQ.
Q2: When running GPFS in an sp cluster type, what are the software requirements?
A2: In an sp cluster type, GPFS requires the Parallel System Support Programs (PSSP) licensed
product and its IBM Virtual Shared Disk and IBM Recoverable Virtual Shared disk components
for uniform disk access and recovery.
Table 1. Supported software levels for an sp cluster type
GPFS Version AIX Level PSSP Level GPFS End of Service
GPFS v2.2.1 AIX 5L v5.2 PSSP v3.5
GPFS v2.1 AIX 5L v5.1 or AIX 5L v5.2 PSSP v3.5 30April2006
Note: Ensure you are at the current service level for each software level listed.
Q3: What disk support guidelines must be followed when running GPFS in an sp cluster type?
A3: In an sp cluster type, a typical GPFS cluster configuration uses recoverable virtual shared disks
(RVSD). Deviations from the typical case are GPFS clusters which use concurrent virtual shared
disks (CVSD). Use of CVSD may have an impact on the types of disks supported. This distinction
is noted in the Cluster Properties section in Table 2 on page 4, Table 3 on page 5, Table 4 on page 5,
and Table 5 on page 6. There are no specific disk limitations in a two-node sp cluster running
single-node quorum (see the question What considerations are there for two node nodesets?).
For additional disk configuration information and examples, please see
ftp.software.ibm.com/storage/fastt/fastt500/PSSP-GPFS_config_info.pdf
3
![Page 4: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/4.jpg)
Table 2. GPFS v2.2.1 and PSSP v3.5 running AIX 5L v5.2
Cluster Properties: Typical GPFS cluster configuration
Disks Supported: IBM TotalStorage® DS4100 with Serial Advanced Technology Attachment (SATA)
disk drives (SM8.42 microcode and APAR IY62752)
IBM TotalStorage FAStT500 (SM7.10, SM8.21, and SM8.3 microcode)
IBM DS4300 (SM8.33 microcode)
IBM DS4300 Turbo (SM8.40 microcode)
IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit
(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR
IY62752)
IBM DS4400 (SM8.21and SM8.3 microcode)
IBM DS4500 (SM8.3 microcode)
IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit
(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR
IY62752)
IBM TotalStorage® Enterprise Storage Server® (ESS) (2105-F20 or 2105-800 with
Subsystem Device Driver (SDD) - disk leasing mode)
IBM TotalStorage ESS (2105-F20 or 2105-800 using AIX Multiple Path I/O (MPIO)
and Subsystem Device Driver Path Control Module (SDDPCM) - persistent reserve
mode)
IBM SAN Volume Controller (APARs IY47267 and IY42355)
EMC Symmetrix Direct Matrix Architecture (DMX) Storage Subsystems (FC attach
only)
IBM Serial Storage Architecture (SSA) (all disk sizes)
Hitachi Lightning 9900™(9910, 9960, 9970V and 9980V (Hitachi Dynamic Link
Manager™) (HDLM) required)
Restrictions: Fibre Channel disks must be FC switch attached.
No disk can be larger than 1 TB.
4
||
|
|
|||
|
|
|||
![Page 5: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/5.jpg)
Table 3. GPFS v2.2.1 and PSSP v3.5 running AIX 5L v5.2
Cluster Properties: CVSD required
Disks Supported: IBM TotalStorage® DS4100 with Serial Advanced Technology Attachment (SATA)
disk drives (SM8.42 microcode and APAR IY62752)
IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit
(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR
IY62752)
IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit
(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR
IY62752)
IBM TotalStorage Enterprise Storage Server (ESS) (2105-F20 or 2105-800 using AIX
MPIO and SDDPCM - persistent reserve mode)
IBM SAN Volume Controller (APARs IY47267 and IY42355)
EMC Symmetrix DMX Storage Subsystems (FC attach only)
IBM SSA (all disk sizes)
Restrictions: Fibre Channel disks must be FC switch attached.
No disk can be larger than 1 TB.
Table 4. GPFS v2.1 and PSSP v3.5 running AIX 5L v5.1
Cluster Properties: Typical GPFS cluster configuration
Disks Supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR IY61688)
IBM FAStT500 (SM7.10, SM8.21, and SM8.3 microcode)
IBM DS4300 (SM8.33 microcode)
IBM DS4300 Turbo (SM8.40 microcode)
IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit
(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR
IY61688)
IBM DS4400 (SM8.21and SM8.3 microcode)
IBM DS4500 (SM8.3 microcode)
IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit
(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR
IY61688)
IBM TotalStorage ESS (2105-F20 or 2105-800 with SDD)
IBM TotalStorage ESS (2105-F20 or 2105-800 without SDD - persistent reserve mode)
IBM SAN Volume Controller (APARs IY47267 and IY42355)
EMC Symmetrix DMX Storage Subsystems (FC attach only)
IBM SSA (all disk sizes)
Hitachi Lightning 9900 (9910, 9960, 9970V and 9980V ) (HDLM required)
Restrictions: Fibre Channel disks must be FC switch attached.
No disk can be larger than 1 TB.
5
||
|||
|||
|
|
|
|||
|
|
|||
![Page 6: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/6.jpg)
Table 5. GPFS v2.1 and PSSP v3.5 running AIX 5L v5.1
Cluster Properties: CVSD required
Disks Supported: IBM TotalStorage® DS4100 with Serial Advanced Technology Attachment (SATA)
disk drives (SM8.42 microcode and APAR IY62752)
IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit
(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR
IY62752)
IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit
(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR
IY62752)
IBM TotalStorage ESS (2105-F20 or 2105-800 without SDD - persistent reserve mode)
IBM SAN Volume Controller (APARs IY47267 and IY42355)
EMC Symmetrix DMX Storage Subsystems (FC attach only)
IBM SSA (all disk sizes)
Restrictions: Fibre Channel disks must be FC switch attached.
No disk can be larger than 1 TB.
Q4: What is an rpd cluster type?
A4: The GPFS cluster type rpd is based on the Reliable Scalable Cluster Technology (RSCT)
subsystem of AIX 5L. The GPFS cluster is defined within an existing RSCT peer domain. The
nodes which are members of the GPFS cluster are defined via the mmcrcluster, mmaddcluster,
and mmdelcluster commands. With an RSCT peer domain, all nodes in the GPFS cluster have the
same view of the domain and share the resources within the domain. Within the GPFS cluster,
you define one or more nodesets within which your file systems operate.
Disk requirements within your GPFS rpd cluster utilizing an IP network, require that all disks
must be:
1. Logical volumes:
v SAN-attached to all nodes in the cluster.
v Processed via the mmcrlv command to create logical volumes useable by GPFS.2. Virtual shared disks:
v Accessed via the IBM Virtual Shared Disk component of RSCT.
v Processed via the mmcrvsd command to create or register disks for use by GPFS.3. Conform to the disk support guidelines listed in this FAQ.
Disk requirements within your GPFS rpd cluster utilizing the IBM Eserver pSeries High
Performance Switch (HPS), require that all disks must be:
1. Accessed via the IBM Virtual Shared Disk component of RSCT.
2. Processed via the mmcrvsd command to create or register disks for use by GPFS.
6
||
|||
|||
![Page 7: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/7.jpg)
Q5: When running GPFS in an rpd cluster type, what are the software requirements?
A5: In an rpd cluster type, GPFS requires the RSCT component of AIX. The GPFS cluster is defined
within an existing RSCT peer domain via the mmcrcluster command.
Table 6. Supported levels of AIX in an rpd cluster type
GPFS Version AIX Level GPFS End of Service
GPFS v2.2.1 AIX 5L Version 5 Release 2
For use of the Eserver p5, APAR IY58249, or later
modifications
For use of the pSeries HPS and IBM Virtual Shared Disks,
AIX 5L Version 5 Release 2 with APAR IY56532, or later
modifications.
GPFS v2.1 AIX 5L Version 5 Release 2 30April2006
For use of the pSeries HPS and IBM Virtual Shared Disks,
AIX 5L Version 5 Release 2 with APAR IY47306, or later
modifications.
30April2006
AIX 5L v5.1 5100-03 Recommended Maintenance package
(APAR IY30258), or later
30April2006
Note: Ensure you are at the current service level for each software level listed.
Q6: What disk support guidelines must be followed when running GPFS in an rpd cluster type?
A6: In an rpd cluster type, a typical GPFS cluster configuration uses multi-node quorum. Deviations
from the typical case are two-node GPFS clusters which have single-node quorum enabled.
Single-node quorum may have an impact on the types of disks supported. This distinction is
noted in the Cluster Properties section in Table 7 on page 8, Table 8 on page 8, Table 9 on page 9,
and Table 10 on page 9. For additional disk configuration information and examples, please see
ftp.software.ibm.com/storage/fastt/fastt500/PSSP-GPFS_config_info.pdf
7
![Page 8: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/8.jpg)
Table 7. GPFS v2.2.1 running AIX 5L v5.2
Cluster properties: Typical GPFS cluster configuration (2 - 32 nodes)
Disks supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR
IY62752)
IBM FAStT500 (SM7.10, SM8.21, and SM8.3 microcode)
IBM DS4300 (SM8.33 microcode)
IBM DS4300 Turbo (SM8.40 microcode)
IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage
Expansion Unit (1742-60U and 1742-60X ) with SATA disk drives
(SM8.42 microcode and APAR IY62752)
IBM DS4400 (SM8.21 and SM8.3 microcode)
IBM DS4500 (SM8.3 microcode)
IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion
Unit (1742-60U and 1742-60X ) with SATA disk drives (SM8.42
microcode and APAR IY62752)
IBM TotalStorage ESS (2105-F20 or 2105-800 with SDD - disk leasing
mode)
IBM TotalStorage ESS (2105-F20 or 2105-800 using AIX MPIO and
SDDPCM - persistent reserve mode)
IBM SAN Volume Controller (APARs IY47267 and IY42355)
EMC Symmetrix DMX Storage Subsystems (FC attach only)
IBM SSA (all disk sizes)
Hitachi Lightning 9900 (9910, 9960, 9970V and 9980V ) (HDLM
required)
Restrictions: Fibre Channel disks must be FC switch attached
Maximum cluster size is 8 nodes if using SSA disks
Table 8. GPFS v2.2.1 running AIX 5L v5.2
Cluster properties: Single-node quorum (two node cluster only)
Disks supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR
IY62752)
IBM SSA (all disk sizes)
IBM TotalStorage Enterprise Storage Server (ESS) (2105-F20 or 2105-800
using AIX MPIO and SDDPCM - persistent reserve mode)
IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage
Expansion Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR
IY62752)
IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion
Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR IY62752)
Restrictions: For single-node quorum - no disk leasing mode disks can be present.
Fibre Channel disks must be FC switch-attached.
8
||
|
|
|||
|
|
|||
||
|||
||
![Page 9: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/9.jpg)
Table 9. GPFS v2.1 running AIX 5L v5.1
Cluster properties: Typical GPFS cluster configuration (2 - 32 nodes)
Disks supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR
IY61688)
IBM FAStT500 (SM7.10, SM8.21, and SM8.3 microcode)
IBM DS4300 (SM8.33 microcode)
IBM DS4300 Turbo (SM8.40 microcode)
IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage
Expansion Unit (1742-60U and 1742-60X ) with SATA disk drives
(SM8.42 microcode and APAR IY61688)
IBM DS4400 (SM8.21 and SM8.3 microcode)
IBM DS4500 (SM8.3 microcode)
IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion
Unit (1742-60U and 1742-60X ) with SATA disk drives (SM8.42
microcode and APAR IY61688)
IBM TotalStorage ESS (2105-F20 or 2105-800 with SDD - disk leasing
mode)
IBM TotalStorage ESS (2105-F20 or 2105-800 without SDD - persistent
reserve mode)
IBM SAN Volume Controller (APARs IY47267 and IY42355)
EMC Symmetrix/DMX Storage Subsystems (FC attach only)
IBM SSA (all disk sizes)
Hitachi Lightning 9900 (9910, 9960, 9970V and 9980V ) (HDLM
required)
Restrictions: Fibre Channel disks must be FC switch attached.
Maximum cluster size is 8 nodes if using SSA disks
Table 10. GPFS v2.1 running AIX 5L v5.1
Cluster properties: Single-node quorum (two node cluster only)
Disks supported: IBM SSA (all disk sizes)
IBM TotalStorage ESS (2105-F20 or 2105-800 without SDD) - persistent
reserve mode
IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage
Expansion Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR
IY61688)
IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion
Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR IY61688)
Restrictions: For single-node quorum - no disk leasing mode disks can be present.
Fibre Channel disks must be FC switch-attached.
9
||
|
|
|||
|
|
|||
|||
||
![Page 10: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/10.jpg)
Notes:
1. You may have up to 1024 external shared disks, disk arrays, or logical units (LUNs) with the
adapters configured to allow each disk, connectivity to each node in the GPFS cluster. No disk
can be larger than 1 TB.
2. You cannot protect your file system against disk failure by mirroring data at the LVM level.
You must use GPFS replication or RAID devices to protect your data
1.
Q7: What is an hacmp cluster type?
A7: The GPFS cluster type hacmp is based on IBM’s High Availability Cluster Multi-Processing for
AIX (HACMP™) licensed product. The GPFS cluster is defined over an existing HACMP cluster.
The nodes which are members of the GPFS cluster are defined via the mmcrcluster,
mmaddcluster, and mmdelcluster commands.
Within the GPFS cluster, you define one or more nodesets within which your file systems operate.
Disk requirements within your GPFS hacmp cluster require that all disks must be:
1. SAN-attached to all nodes in the cluster.
2. Processed via the mmcrlv command.
3. Follow the disk support guidelines listed in this FAQ.
Q8: When running GPFS in an hacmp cluster type, what are the software requirements?
A8: In an hacmp cluster type, GPFS requires IBM’s HACMP licensed product over which the GPFS
cluster is defined via the mmcrcluster command.
Table 11. Supported software levels in an hacmp cluster type
GPFS Version AIX Level HACMP Level GPFS End of
Service
GPFS v2.2.1 AIX 5L v5.2 HACMP v5.1
For use of the Eserver p5, APAR IY58249, or later modifications
GPFS v2.1 AIX 5L v5.2 HACMP v4.5 / v5.1 30April2006
GPFS v2.1 AIX 5L v5.1 HACMP/ES v4.4.1/4.5 30April2006
Notes:
1. Ensure you are at the current service level for each software level listed
2. If running with HACMP/ES v4.5, the minimum APAR level required is IY32874
Q9: What are the disk support guidelines that must be followed when running GPFS in an hacmp
cluster type?
A9: In an hacmp cluster type, a typical GPFS cluster configuration uses multi-node quorum.
Deviations from the typical case are two-node GPFS clusters which have single-node quorum
enabled. Single-node quorum may have an impact on the types of disks supported. This
distinction is noted in the Cluster Properties section in Table 12 on page 11, Table 13 on page 11,
Table 14 on page 12, and Table 15 on page 12. For additional disk configuration information and
examples, please see ftp://ftp.software.ibm.com/storage/fastt/fastt500/PSSP-GPFS_config_info.pdf
1. IBM General Parallel File System for AIX 5L in an RSCT Peer Domain: Concepts, Planning, and Installation Guide Number GA22-7974
(July 2004), International Business Machines Corporation, 2455 South Road, Poughkeepsie, NY 12601-5400, USA.
10
![Page 11: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/11.jpg)
Table 12. GPFS v2.2.1 running AIX 5L v5.2
Cluster properties: Typical GPFS cluster configuration (2- 32 nodes)
Disks supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR
IY62752)
IBM FAStT500 (SM7.10, SM8.21, and SM8.3 microcode)
IBM DS4300 (SM8.33 microcode)
IBM DS4300 Turbo (SM8.40 microcode)
IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage
Expansion Unit (1742-60U and 1742-60X ) with SATA disk drives (SM8.42
microcode and APAR IY62752)
IBM DS4400 (SM8.21 and SM8.3 microcode)
IBM DS4500 (SM8.3 microcode)
IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion
Unit (1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode
and APAR IY62752)
IBM TotalStorage ESS (2105-F20 or 2105-800 with SDD - disk leasing
mode)
IBM TotalStorage ESS (2105-F20 or 2105-800 using AIX MPIO and
SDDPCM - persistent reserve mode)
IBM SAN Volume Controller (APARs IY47267 and IY42355)
EMC Symmetrix DMX Storage Subsystems (FC attach only)
IBM SSA (all disk sizes)
Hitachi Lightning 9900 (9910, 9960, 9970V and 9980V ) (HDLM required)
Restrictions: Fibre Channel disks must be FC switch attached.
Maximum cluster size is 8 nodes if using SSA disks
Table 13. GPFS v2.2.1 running AIX 5L v5.2
Cluster properties: Single-node quorum (two node cluster only)
Disks supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR
IY62752)
IBM SSA (all disk sizes)
IBM TotalStorage Enterprise Storage Server (ESS) (2105-F20 or 2105-800
using AIX MPIO and SDDPCM - persistent reserve mode)
IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage
Expansion Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR
IY62752)
IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion
Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR IY62752)
Restrictions: For single-node quorum no disk leasing mode disks can be present.
Fibre Channel disks must be FC switch attached.
11
||
|
|
|||
|
|
|||
||
|||
||
![Page 12: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/12.jpg)
Table 14. GPFS v2.1 running AIX 5L v5.1
Cluster properties: Typical GPFS cluster configuration (2- 32 nodes)
Disks supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR
IY61688)
IBM FAStT500 (SM7.10, SM8.21, and SM8.3 microcode)
IBM DS4300 (SM8.33 microcode)
IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage
Expansion Unit (1742-60U and 1742-60X ) with SATA disk drives (SM8.42
microcode and APAR IY61688)
IBM DS4400 (SM8.21 and SM8.3 microcode)
IBM DS4500 (SM8.3 microcode)
IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion
Unit (1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode
and APAR IY61688)
IBM TotalStorage ESS (2105-F20 or 2105-800 with SDD - disk leasing
mode)
IBM TotalStorage ESS (2105-F20 or 2105-800 without SDD - persistent
reserve mode)
IBM SAN Volume Controller (APARs IY47267 and IY42355)
EMC Symmetrix DMX Storage Subsystems (FC attach only)
IBM SSA (all disk sizes)
Hitachi Lightning 9900 (9910, 9960, 9970V and 9980V ) (HDLM required)
Restrictions: Fibre Channel disks must be FC switch attached.
Maximum cluster size is 8 nodes if using SSA disks.
Table 15. GPFS v2.1 running AIX 5L v5.1
Cluster properties: Single-node quorum (two node cluster only)
Disks supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR
IY61688)
IBM SSA (all disk sizes)
IBM TotalStorage ESS (2105-F20 or 2105-800 without SDD) - persistent
reserve mode
IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage
Expansion Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR
IY61688)
IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion
Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR IY61688)
Restrictions: For single-node quorum no disk leasing mode disks can be present.
Fibre Channel disks must be FC switch attached.
12
||
|
|||
|
|
|||
||
|||
||
![Page 13: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/13.jpg)
Notes:
1. You may have up to 1024 external shared disks, disk arrays, or LUNs with the adapters
configured to allow each disk, connectivity to each node in the GPFS cluster. No disk can be
larger than 1 TB.
2. You cannot protect your file system against disk failure by mirroring data at the LVM level.
You must use GPFS replication or RAID devices to protect your data
2.
Q10: How can I implement disaster recovery for my GPFS cluster?
A10: Beginning with GPFS 2.2.1 and APAR IY59339, disaster recovery for your GPFS cluster can be
implemented as described in Disaster Recovery with General Parallel File System located at
http://www.ibm.com/servers/eserver/pseries/software/whitepapers/gpfs_recovery.html
Q11: What are the coexistence rules for GPFS clusters?
A11: You must be concerned with the coexistence rules for each type of GPFS cluster. Updates to the
coexistence rules for GPFS clusters listed in this FAQ supersede the product documentation.
Table 16 and Table 17 identify release levels of additional clustering software which may coexist
on the same node. This does not imply interaction with GPFS if the software is not required for
the particular GPFS cluster type. Please note that PSSP and Cluster System Management (CSM)
may not coexist and therefore the information is presented in two separate tables. There are also
special considerations noted in Table 18 on page 14.
Table 16. Coexistence of clustering software with PSSP
GPFS cluster type GPFS level AIX level PSSP level HACMP level
rpd (IP network) 2.2.1 5.2 3.5 5.1/4.5
rpd (IP network) 2.1 5.2/5.1 3.5 5.1/4.5/4.4.1
hacmp 2.2.1 5.2 3.5 5.1/4.5
hacmp 2.1 5.1 3.5 4.5/4.4.1
sp 2.2.1 5.2 3.5 5.1/4.5
sp 2.1 5.1 3.5 4.5/4.4.1
Table 17. Coexistence of clustering software with CSM
GPFS cluster type GPFS level AIX level CSM level HACMP level
rpd (IP network) 2.2.1 5.2 1.4 / 1.3.x.x 5.1/4.5
rpd pSeries HPS 2.2.1 5.2 1.4 / 1.3x.x 5.1/4.5
rpd (IP network) 2.1 5.2/5.1 1.1.x.x 5.1/4.5/4.4.1
rpd pSeries HPS 2.1 5.2 1.4 / 1.3.x.x 5.1/4.5/4.4.1
hacmp 2.2.1 5.2 1.4 / 1.3.x.x 5.1/4.5
hacmp 2.1 5.2/5.1 1.1. x.x 4.5/4.4.1
Notes:
1. The fact that GPFS and CSM may coexist does not imply that there is interaction between
GPFS and CSM.
2. For the CSM levels, x.x refers to the current modification and fix levels.
3. PSSP does not support the pSeries HPS, therefore a GPFS rpd cluster utilizing the pSeries HPS
may not coexist with PSSP
2. IBM General Parallel File System for AIX 5L in an HACMP Cluster: Concepts, Planning, and Installation Guide Number GA22-7971-00
(December 2003), International Business Machines Corporation, 2455 South Road, Poughkeepsie, NY 12601-5400, USA.
13
![Page 14: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/14.jpg)
Table 18. Special considerations which must be adhered to when the products coexist
GPFS Cluster Type CSM RSCT peer domain PSSP HACMP
rpd 2 2 1,2 2
sp 2 2 2 2
hacmp 2 2 1,2 2
1. When running in either a GPFS rpd or hacmp cluster type that contains nodes that are also
within an sp cluster, you need to be concerned with the daemon to daemon authentication
method for the common nodes. GPFS, as an SP trusted service, can have its daemon to
daemon authentication method set by the system administrator. The authentication method
for all nodes in your GPFS nodeset must be compatible. If nodes within a cluster are not
configured for a compatible authentication method, an error condition will be written
to/var/adm/ras/mmfs.log.latest when there is an attempt to make a connection between them.
See the section on Security in the GPFS for AIX 5L in a PSSP Cluster: Clusters Concepts,
Planning, and Installation Guide and search on Security.
2. The level of RSCT is determined by your AIX service level.
Q12: What type of disk fencing mechanisms does GPFS support?
A12: In order to preserve data integrity in the event of certain system failures, GPFS is designed to
prevent a node that is down from accessing the file system until it returns to the available state.
GPFS is said to fence the failing node from the disks.
There are two types of fencing mechanisms:
v Fencing that takes advantage of hardware assist
v Fencing that is done in GPFS software without hardware assist
Depending upon the types of disks that are being using, there are three possible ways for the
fencing to occur:
SSA fencing
An SSA disk subsystem specific fencing mechanism. SSA fencing takes advantage of
hardware assistance.
persistent reserve
SCSI-3 standard reserve mechanism. Persistent reserve takes advantage of hardware
assistance. See the question Does GPFS support AIX MPIO?
Disk leasing
A GPFS specific fencing mechanism for disks which do not support either SSA fencing or
persistent reserve. Disk leasing does not rely on any hardware assistance.
GPFS can use different types of disks in the same cluster. GPFS will determine the disks
capability and prepare the disks for proper use during the setup phase. This preparation is
performed by the GPFS command mmcrlv and mmcrvsd, and can also be performed by the
RVSD command createvsd. No options need to be provided to direct the GPFS codes to perform
this function.
GPFS will always attempt to use fencing with hardware assistance if possible.
There can only be one fencing mechanism active in a cluster, hardware assisted or disk leasing.
If all the disks in a configuration support fencing with hardware assistance, that configuration
will be running in a useDiskLeasing no environment. In other words, disk leasing is not being
used in this environment to perform fencing, the fencing is being done with the assistance of the
disk subsystem hardware and firmware.
If any disk in a GPFS configuration does not support hardware fencing, all disks in that same
environment revert to using disk leasing. If any or all disks in a configuration do not support
14
![Page 15: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/15.jpg)
hardware fencing, the cluster will be running in a useDiskLease yes environment. In other
words, disk leasing is being used and none of the capabilities of the disk subsystem and
firmware are being exploited for fencing.
Single-node quorum is not supported when disk leasing is in effect.
The only manual action that needs to be performed by the administrator is in the SSA disk types,
the ODM attribute ssar needs to be setup on each node in the cluster. For directions on using the
mmlsfenceid and mmsetfenceid commands:
v For the GPFS cluster type rpd, please see the General Parallel File System for AIX 5L in an RSCT
Peer Domain: Problem Determination Guide .
v For a GPFS cluster type of hacmp, please the GPFS for AIX 5L in an HACMP Cluster General
Parallel File System for AIX 5L in an HACMP Cluster: Problem Determination Guide .
Q13: How can the fencing technique in operation for a cluster be verified?
A13: The current fencing mode for a cluster is either useDiskLease yes or useDiskLease no. The
current state of disk leasing in a cluster is determined by using the mmlsconfig command.
Q14: What is AIX MPIO and does GPFS support it?
A14: Beginning with AIX 5.2, GPFS can use the Multiple Path I/O (MPIO) feature of AIX to define
alternate paths to a device for failover purposes. Failover is a path-management algorithm that
improves the reliability and availability of a device because the system automatically detects
when one I/O path fails and re-routes I/O through an alternate path. All SCSI SCSD disk drives
are automatically configured as MPIO devices. Other devices can be supported, providing the
device driver is compatible with the MPIO implementation in AIX. For more information about
MPIO, see the:
v AIX 5L Version 5.2 System Management Concepts: Operating System and Devices book and search
on Multi-path I/O.
v AIX 5L Version 5.2 System Management Guide: Operating System and Devices book and search on
Multi-path I/O.
As GPFS supports disk subsystems exploiting the AIX MPIO function, this FAQ will be updated
and the new support statements will be communicated to the IBM Sales representatives. Please
see:
v Q3: What disk support guidelines must be followed when running GPFS in an sp cluster type?
v Q6: What disk support guidelines must be followed when running GPFS in an rpd cluster
type?
v Q9:What are the disk support guidelines that must be followed when running GPFS in an
hacmp cluster type?
Q15: What disks support hardware fencing?
A15: The current set of disks that support hardware fencing are:
v IBM SSA – all models
v IBM TotalStorage ESS using AIX MPIO and SDDPCM
v Selected members of the IBM TotalStorage DS4000 family of subsystems with microcode at
versions 8.40 or greater. See the individual cluster types (sp, rpd, hacmp) for details.
Q16: What should I do if commands such as mmcrlv, mmcrfs, mmadddisk, mmrpldisk, or mount
fail to complete due to problems with non-IBM disks?
A16: Certain GPFS disk commands issue multiple AIX varyonvg -u commands in parallel to the nodes
in the cluster. For some non-IBM disks, when many varyonvg -u commands are issued in
parallel, some of the AIX varyonvg -u invocations do not complete and the GPFS disk command
hangs. This is recognized by the GPFS disk command not completing after a long period of time
and the persistence of the varyonvg processes as shown by the output of the ps -ef command on
some of the nodes of the cluster. In such cases, kill the varyonvg processes issued by the disk
15
||
![Page 16: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/16.jpg)
command on the nodes of the cluster to allow the disk command to complete. Before mounting
the file system in question on any node where a varyonvg process was killed, issue the varyonvg
-u vgname command on the node to make the disk available to GPFS. Do this on each of the
nodes in question one by one until all of the GPFS volume groups are varied on.
Q17: What cluster specific single-node quorum restrictions are there?
A17: Restrictions when using single-node quorum include:
1. Single-node quorum is not supported when:
v The GPFS cluster type lc consists of only AIX nodes.
v Disk leasing is in effect. Disk leasing is a GPFS-specific fencing mechanism for disks which
do not support either SSA fencing or persistent reserve. Disk leasing is activated if any disk
in any file system in the nodeset is not using SSA fencing or hardware assisted persistent
reserve. See the question What type of disk fencing mechanisms does GPFS support?
v SDD (the non-MPIO version) is installed.2. If a disk is using persistent reserve, this needs to be both supported at both the disk
subsystem level and the AIX device driver level.
3. When adding nodes to a nodeset:
v The GPFS daemon must be stopped on all of the nodes.
v If after adding the nodes, the number of nodes in the nodeset exceeds two, the quorum
algorithm is automatically changed to the multi-node quorum algorithm.4. When deleting nodes from a nodeset:
v A node cannot be deleted from the nodeset without stopping the GPFS daemon on both
nodes.
v If the number of nodes remaining in the nodeset falls below three, you must run the
mchconfig command if you want to change the quorum algorithm to single-node quorum.
See the question, What considerations are there for two node nodesets?
Q18: Why would I want to use EtherChannel in my GPFS cluster?
A18: The main benefit of EtherChannel and IEEE 802.3ad Link Aggregation is increased bandwidth:
the aggregated adapter has the network bandwidth of all its adapters in a single network
presence. In addition, if an adapter fails, the packets are automatically sent on the next available
adapter without disruption to existing user connections. The adapter is automatically returned to
service on the EtherChannel or Link Aggregation when it recovers.
EtherChannel and IEEE802.3ad each requires support within the Ethernet switch. Refer to the
product documentation for your switch to detemine if EtherChannel is supported. To determine if
your version of GPFS supports EtherChannel, see the question, What versions of GPFS support
EtherChannel?
For details on how to configure EtherChannel and IEEE 802.3ad Link Aggregation:
1. Go to
publib16.boulder.ibm.com/pseries/en_US/aixbman/commadmn/tcp_etherchannel.htm#yu528frokferg
2. Search on Configuring EtherChannel
Hint: Make certain that the switch ports are configured for LACP (the default is PAGP).
For details on how to verify whether the adapter and the switch are operating with the correct
protocols For IEEE 802.3ad:
1. Go to
publib16.boulder.ibm.com/pseries/en_US/aixbman/commadmn/tcp_etherchannel.htm#yu528frokferg
2. Search on Troubleshooting IEEE 802.3ad.
For additional service updates regarding the use of EtherChannel:
1. Go to www.ibm.com/support
2. Click Search technical support
3. Enter the search term EtherChannel
16
![Page 17: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/17.jpg)
4. Click Submit
Hint: A useful command for troubleshooting, where device is the Link Aggregation device, is:
entstat -d device
Q19: What versions of GPFS support EtherChannel?
A19: GPFS support for EtherChannel includes:
v GPFS 2.2 on cluster types rpd or hacmp on AIX 5.2 and maintenance package 5.2.0.13 with
APAR IY48170 and APAR IY44290
v GPFS 2.1 on cluster types rpd or hacmp on AIX 5.1 and maintenance package 5.1.0.50 with
APAR IY48331
For the latest service level, go to www-912.ibm.com/eserver/support/fixes/fcgui.jsp
1. For your server type, select pSeries family.
2. For product or fix type select AIX OS, Java™, compilers.
3. For your option, select the type of fixes you are searching for.
4. Select your OS level.
5. Click continue.
6. If choosing Latest fixes or Critical fixes, the GPFS fixes will be bundled with all of the file
sets.
7. If you choose Specific fixes, enter mmfs as your search string
Q20: What’s the difference between using an ESS with or without SDD or SDDPCM installed on
the host?
A20: The use of SDD or SDDPCM gives the AIX host the ability to access multiple paths to a single
LUN within an ESS. This ability to access a single LUN on multiple paths allows for a higher
degree of data availability in the event of a path failure. Data can continue to be accessed within
the ESS as long as there is at least one available path. Without one of these installed, you will
lose access to the LUN in the event of a path failure.
However, your choice of whether to use SDD or SDDPCM impacts your ability to use single-node
quourm:
v Single-node quorum is not supported if SDD is installed.
v Single-node quorum is support if SDDPCM is installed.
To determine the GPFS disk support guidelines for SDD and SDDPCM for your cluster type, see
v Q3: What disk support guidelines must be followed when running GPFS in an sp cluster type?
v Q6: What disk support guidelines must be followed when running GPFS in an rpd cluster
type?
v Q9:What are the disk support guidelines that must be followed when running GPFS in an
hacmp cluster type?
Q21: Is a Fibre Channel Switch required for all Fibre Channel disk attachments which are used by
GPFS?
A21: Yes, all Fibre Channel disks require attachment to AIX hosts using a SAN FC switch when used
by GPFS.
Q22: What Fibre Channel Switches are qualified for GPFS usage and is there a FC Switch support
chart available?
A22: There are no special requirements for FC switches used by GPFS other than the switch must be
supported by AIX. For further information see www.storage.ibm.com/ibmsan/index.html
Q23: What performance tuning suggestions are there for AIX nodes?
A23: Performance tuning suggestions for AIX nodes include:
ipqmaxlen
In a GPFS cluster type sp or rpd (utilizing the pSeries HPS), the ipqmaxlen network
17
||||
![Page 18: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/18.jpg)
option controls the number of incoming packets that can exist on the IP interrupt queue.
Since both GPFS and the IBM Virtual Shared Disk use IP, the default value of 128 is often
insufficient. This is especially important if your virtual shared disks are configured over
IP rather than KLAPI. The recommended setting is 512:
no -o ipqmaxlen=512
Buddy buffers
In a GPFS cluster type sp or rpd (utilizing the pSeries HPS), the buddy buffer is pinned
kernel memory. The virtual shared disk server node uses the buddy buffer to temporarily
store data for I/O operations originating at a client node. The data in a buddy buffer is
purged immediately after the I/O operation completes. The values associated with the
buddy buffer are:
v Minimum buddy buffer size allocated to a single request
v Maximum buddy buffer size allocated to a single request
v Total number of maximum-size buffers
Total number of maximum-size buddy buffers that the system will attempt to
dynamically allocate
IBM suggests using the following values:
v Minimum buddy buffer size: 4096 (4 KB)
v Maximum buddy buffer size: 262144 (256 KB)
If your application uses the fastpath option of asynchronous I/O, the maximum buddy
buffer size must be greater than or equal to 128 KB. Otherwise, you will receive
EMSGSIZE Message too long errors.
v Total number of maximum-size buffers:
– 2000 for a virtual shared disk server node
– One for a virtual shared disk client-only node
You can either set these values using the vsdnode command, or update the values using
the updatevsdnode command. IBM suggests, however, that you configure the
recommended amount of buddy buffer space, unless your system is memory-constrained
and you want to restrict the amount of buddy buffer space.
When the device driver is configured, the total buddy buffer space is not pinned; instead,
approximately one-quarter of the total space requested is pinned when the device driver
is configured. This initial amount of spaced pinned is limited to a maximum of 64 MB for
a 32-bit kernel, or 128 MB for a 64-bit kernel. After configuration, the device driver
attempts to dynamically expand and contract additional buddy buffer space up to the
maximum specified, or until AIX can no longer satisfy the memory request. If a buddy
buffer cannot be obtained, then the request is queued at the virtual shared disk server
until a buddy buffer is available.
max_coalesce
The max_coalesce (SSA) parameter defines the maximum number of bytes that the SSA
disk device driver attempts to transfer to or from an SSA logical disk in one operation.
This parameter is very important when dealing with writes on SSA RAID systems
3. The
suggested setting is 256KB for a RAID-5 4+P configuration:
chdev -l hdiskx -a max_coalesce=0x40000
KLAPI
In the GPFS cluster type sp, the default IBM Virtual Shared Disk communication protocol
is IP. KLAPI is an alternate high-performance communication interface that can be used
to avoid some data copies (which would have been in the IP path) in handling the I/O
request. Avoiding the data copies reduces CPU utilization on both the client and server
nodes. In order to switch from IP to KLAPI, the RVSD system has to be stopped and IBM
3. Dominique Heger, Gautam Shah: General Parallel File System (GPFS v1.4) for AIX Architecture and Performance, November 2001
4. Dominique Heger, Gautam Shah: General Parallel File System (GPFS v1.4) for AIX Architecture and Performance, November 2001
18
![Page 19: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/19.jpg)
Virtual Shared Disk component has to be unconfigured. In the case that IBM Virtual
Shared Disk over KLAPI encounters a communication problem, the system will switch to
IBM Virtual Shared Disk over IP. Choosing KLAPI will not affect the number of
user-space SP Switch2 windows that are available to user applications
4:
ctlvsd -l on
SP Switch, SP Switch2, and pSeries HPS subsystem parameters
Depending upon which switch you are using, the communications subsystem parameters
should be configured differently. These are especially important if you configure your
virtual shared disks over IP.
If you are using an SP Switch, the rpoolsize and spoolsize values should be set to
16777216:
chgcss -l css0 -a rpoolsize=16777216 -a spoolsize=16777216
If you are using an SP Switch2, the rpoolsize and spoolsize values should be set to
33554432:
chgcss -l css0 -a rpoolsize=33554432 -a spoolsize=33554432
If you are using the pSeries HPS, the rpoolsize and spoolsize values should be set to
67108864:
chgsni -l sn0 –a rpoolsize=67108864 –a spoolsize 67108864
For further performance information on the SP Switch or the SP Switch2, see the GPFS
white papers at www.ibm.com/servers/eserver/clusters/library/wp_aix_lit.html
Q24: What configuration and performance tuning suggestions are there for GPFS when used
primarily for Oracle databases?
A24: Oracle uses the Asynchronous I/O (AIO) and Direct I/O (DIO) features of AIX to do their own
scheduling of I/O directly to disks, bypassing most of GPFS caching and prefetching facilities.
Most of these settings are to make sure GPFS is out of the way once a database has been
formatted, so Oracle can get the best performance:
v When setting up your LUNS it is important to:
1. Create the logical volumes such that they map one to one with a volume group and a
volume group be one to one with a LUN which is a single RAID device.
2. Not stripe logical volumes across multiple RAIDs for I/O parallelism when using raw
logical volumes because:
– GPFS puts each block of a file on different LUNs spread across all LUNs
– Logical volume striping makes removing a bad RAID more difficultv For file systems holding large Oracle databases, set the GPFS file system block size via the
mmcrfs command using the -B option, to a large value:
– 512 KB is generally suggested.
– 256 KB is suggested if there is activity other than Oracle using the filesystem and many
small files exist which are not in the database.
– 1 MB is suggested for file systems 100 TB or larger.
The large block size makes the allocation of space for the databases manageable and has no
affect on performance when Oracle is using AIO/DIO.
v Set the GPFS worker threads via the mmchconfig - worker1Threads command to allow the
maximum parallelism of the Oracle AIO threads:
– On a 64-bit AIX kernel, the setting can be as large as 548.
The GPFS prefetch threads must be adjusted accordingly via the mmchconfig -
prefetchThreads command as the sum of those two classes of threads must be 550 or less.
– On a 32-bit kernel, the setting can be as large as 162.
The GPFS prefetch threads must be adjusted accordingly via the mmchconfig -
prefetchThreads command as the sum of those two classes of threads must be 164 or less.
19
![Page 20: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/20.jpg)
When requiring GPFS sequential I/O, set the prefetch threads between 50 and 100 (the default
is 64), and set the worker threads to have the remainder.
Note: These changes via the mmchconfig command take effect upon restart of the GPFS
daemon.
v The number of AIX AIO kprocs to create should be approximately the same as the GPFS
worker1Threads setting. For the AIO maxservers setting:
– On AIX 5.1 systems, it is the total number of AIO kprocs.
– On AIX 5.2 systems, it is the number of kprocs PER CPU. It is suggested to set is slightly
larger than worker1Threads divided by the number of CPUs.
For example if worker1Threads is set to 500 on a 32-way SMP:
– On an AIX 5.1 system, set maxservers to 640
– On an AIX 5.2 system, set maxservers to 20v Setting the GPFS striping method to balanced random via the mmcrfs -s balancedRandom or
the mmchfs -s balancedRandom command may prove to make disk access patterns slightly
more random than the system default. This may avoid creating hotspots thus creating less
work for the GPFS administrative operations that rebalance data across all LUNs such as
mmdeldisk, mmadddisk, and mmrestripefs.
v Set the Oracle database block size equal to the LUN segment size or a multiple of the LUN
pdisk segment size.
v Set the Oracle read-ahead value to prefetch one or two full GPFS blocks. For example, if your
GPFS block size is 512 KB, set the Oracle blocks to either 32 or 64 16 KB blocks.
v Do not use the dio option on the mount command as this forces DIO when accessing all files.
Oracle automatically uses DIO to open database files on GPFS.
v There are fixes in the GPFS PTF stream necessary for correct AIO/DIO operation for Oracle. If
using GPFS release 2.1, the mmfs.base.rte fileset should be at least 3.5.0.11.
Q25: What is the largest file system size supported by GPFS?
A25: On 32- bit AIX machines, the largest supported file system size depends on the blocksize of your
file system:
v If your file system has a 16 KB block size, you may have one or more file systems with a total
size of 1 TB mounted.
v If your file system has a 64 KB block size, you may have one or more file systems with a total
size of 10 TB mounted
v If your file system has a 256 KB or greater block size, you may have one or more file systems
mounted with a total size of not greater than 200 TB where no single file system exceeds 100
TB mounted.
The maximum supported file system size of 100 TB is a testing limitation rather than an
architectural limitation of the GPFS licensed product.
Q26: Are there any special concerns regarding the use of mksysb images?
A26: Using a mksysb image to install or reinstall the IBM Virtual Shared Disk licensed product on a
node presents some challenges to be aware of.
In a GPFS cluster type sp:
v The RVSD version number that the post-installation scripts would normally save in the System
Data Repository can not be saved. This may prevent the RVSD subsystem from starting
because it thinks that there are back-level nodes within the SP partition. This problem can be
prevented by using the rvsdrestrict command to explicitly set the restriction level before
starting RVSD. For example, on the control workstation issue:
rvsdrestrict -s RVSD3.5
v The RVSD subsystem expects that the underlying disks and volume groups are available to
AIX. Therefore, if you are reinstalling an existing server node from a mksysb image, you may
need to use the AIX importvg command to make AIX aware of the pre-existing volume groups.
20
![Page 21: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/21.jpg)
Note: All virtual shared disk volume groups must have the auto varyon attribute turned off. To
accomplished this, after the importvg, issue:
chvg -a -n volume_group_name
In a GPFS cluster type rpd, duplicate node ids may be created. The lsnodeid command (available
in /usr/sbin/rsct/bin) has been provided for you to verify whether or not node ids are duplicated
within the cluster. If a duplicate node id is found, see the General Parallel File System for AIX 5L in
an RSCT Peer domain: Administration and Programming Reference and search on Changing IP
addresses, node names, node numbers and reliable hostnames
Q27: Are there any considerations when using NIS automount?
A27: The default file system type when using the automounter daemon is NFS. When the -fstype
option is not specified, and the server is the local node, a soft-mount of the local directory is
done at the desired mount point. JFS is assumed as the only handler of local directories. A GPFS
file system local soft-mount does not work implicitly, since the mount request is passed to JFS
which then produces an error. When specifying -fstype mmfs the local soft-mount works because
the mount is then passed to GPFS instead of JFS.
A GPFS soft-mount does not automatically unmount. Setting -fstype nfs3 causes the local server
mounts to always go through NFS. This allows you to have the same auto.map file on all nodes
whether the server is local or not and the automatic unmount will occur. If you want local
soft-mounts of GPFS file systems while other nodes perform NFS mounts, you should have
different auto.map files on the different classes of nodes. This should improve performance on
the GPFS nodes as they will not have to go through NFS.
Q28: How can I ask a more specific question about GPFS?
A28: If you have a question that can benefit other GPFS users, then please post it to the public GPFS
community mailing list ([email protected]). You can sign up for this list at
oss.software.ibm.com/developerworks/mail/?group_id=91
You can also send a note directly to the GPFS development team at [email protected]. However,
this mailing list is informally monitored as time permits and should not be used for priority
messages to the GPFS team.
If your query concerns a potential software error in GPFS and you have an IBM software
maintenance contract, then please contact 1-800-IBM-SERV in the United States or your local IBM
Service Center in other countries. IBM Scholars Program users should notify the GPFS
development team of potential software bugs via [email protected].
Note: This FAQ is continually being enhanced. Please contribute questions or answers by sending
them to [email protected]
21
![Page 22: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/22.jpg)
Notices
This information was developed for products and
services offered in the U.S.A.
IBM may not offer the products, services, or
features discussed in this document in other
countries. Consult your local IBM representative
for information on the products and services
currently available in your area. Any reference to
an IBM product, program, or service is not
intended to state or imply that only IBM’s
product, program, or service may be used. Any
functionally equivalent product, program, or
service that does not infringe any of IBM’s
intellectual property rights may be used instead.
However, it is the user’s responsibility to evaluate
and verify the operation of any non-IBM product,
program, or service.
IBM may have patents or pending patent
applications covering subject matter described in
this document. The furnishing of this document
does not give you any license to these patents.
You can send license inquiries, in writing, to:
IBM Director of Licensing
IBM Corporation
North Castle Drive
Armonk, NY 10594-1785
USA
For license inquiries regarding double-byte
(DBCS) information, contact the IBM Intellectual
Property Department in your country or send
inquiries, in writing, to:
IBM World Trade Asia Corporation
Licensing
2-31 Roppongi 3-chome, Minato-ku
Tokyo 106, Japan
The following paragraph does not apply to the
United Kingdom or any other country where such
provisions are inconsistent with local law:
INTERNATIONAL BUSINESS MACHINES
CORPORATION PROVIDES THIS PUBLICATION
“AS IS” WITHOUT WARRANTY OF ANY KIND,
EITHER EXPRESS OR IMPLIED, INCLUDING,
BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE. Some states do not
allow disclaimer of express or implied warranties
in certain transactions, therefore, this statement
may not apply to you.
This information could include technical
inaccuracies or typographical errors. Changes are
periodically made to the information herein; these
changes will be incorporated in new editions of
the publication. IBM may make improvements
and/or changes in the product(s) and/or the
program(s) described in this publication at any
time without notice.
Any references in this information to non-IBM
Web sites are provided for convenience only and
do not in any manner serve as an endorsement of
those Web sites. The materials at those Web sites
are not part of the materials for this IBM product
and use of those Web sites is at your own risk.
IBM may use or distribute any of the information
you supply in any way it believes appropriate
without incurring any obligation to you.
Licensees of this program who wish to have
information about it for the purpose of enabling:
(i) the exchange of information between
independently created programs and other
programs (including this one) and (ii) the mutual
use of the information which has been exchanged,
should contact:
IBM Corporation
Intellectual Property Law
2455 South Road,P386
Poughkeepsie, NY 12601-5400
USA
Such information may be available, subject to
appropriate terms and conditions, including in
some cases, payment or a fee.
The licensed program described in this document
and all licensed material available for it are
provided by IBM under terms of the IBM
Customer Agreement, IBM International Program
License Agreement or any equivalent agreement
between us.
Any performance data contained herein was
determined in a controlled environment.
Therefore, the results obtained in other operating
environments may vary significantly. Some
22
![Page 23: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/23.jpg)
measurements may have been made on
development-level systems and there is no
guarantee that these measurements will be the
same on generally available systems.
Furthermore, some measurements may have been
estimated through extrapolation. Actual results
may vary. Users of this document should verify
the applicable data for their specific environment.
This information contains examples of data and
reports used in daily business operations. To
illustrate them as completely as possible, the
examples include the names of individuals,
companies, brands, and products. All of these
names are fictitious and any similarity to the
names and addresses used by an actual business
enterprise is entirely coincidental.
COPYRIGHT LICENSE:
This information contains sample application
programs in source language, which illustrates
programming techniques on various operating
platforms. You may copy, modify, and distribute
these sample programs in any form without
payment to IBM, for the purposes of developing,
using, marketing or distributing application
programs conforming to the application
programming interface for the operating platform
for which the sample programs are written. These
examples have not been thoroughly tested under
all conditions. IBM, therefore, cannot guarantee or
imply reliability, serviceability, or function of
these programs. You may copy, modify, and
distribute these sample programs in any form
without payment to IBM for the purposes of
developing, using, marketing, or distributing
application programs conforming to IBM’s
application programming interfaces.
If you are viewing this information softcopy, the
photographs and color illustrations may not
appear.
Trademarks
The following terms are trademarks of the
International Business Machines Corporation in
the United States or other countries or both:
v AFS
v AIX
v AIX 5L
v Eserver
v Enterprise Storage Server
v HACMP
v IBM
v IBMLink
v LoadLeveler
v Netfinity
v pSeries
v POWER
v PowerPC
v RS/6000
v Redbooks
v SANergy
v Scalable POWERParallel Systems
v ServerProven
v SP
v Tivoli
v TotalStorage
v xSeries
Dynamic Link Manager, Lightning 9900, Thunder
9200, and Thunder 9500 are trademarks of Hitachi
Data Systems Corporation.
Intel, Intel Inside (logos), MMX and Pentium are
trademarks of Intel Corporation in the United
States, other countries, or both.
Java and all Java-based trademarks and logos are
trademarks or registered trademarks of Sun
Microsystems, Inc. in the United States, other
countries, or both.
Linux is a trademark of Linus Torvalds in the
United States, other countries, or both.
Microsoft, Windows, Windows NT, and the
Windows logo are trademarks of Microsoft
Corporation in the United States, other countries,
or both.
Red Hat, the Red Hat ″Shadow Man″ logo, and
all Red Hat-based trademarks and logos are
trademarks or registered trademarks of Red Hat,
Inc., in the United States and other countries.
UNIX is a registered trademark of the Open
Group in the United States and other countries.
Other company, product, and service names may
be the trademarks or service marks of others.
23
![Page 24: Kraemer Gpfs Aix Faq](https://reader031.fdocuments.in/reader031/viewer/2022022401/545134c5af795919308b45fa/html5/thumbnails/24.jpg)
October 2004
© Copyright International Business Machines Corporation 2003, 2004. All rights reserved.
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.