sapnote_0001077887.pdf

11.04.2013 of 14

SAP Note 1077887 - SAP on HP-UX:mount & filesystem optionsfor best performance

Note Language: English Version: 41 Validity: Valid Since 04.10.2011

Summary

SymptomProvide a guide for setting up filesystem, cache sizing and I/Oconfigurations on HP-UX.

Other termsHP-UX, 11.11, 11.23, 11.31, 11iv1, 11iv2, 11iv3, vxfs, direct I/O, bufferedI/O, IO, buffercache, filecache, UFC, mount options, dbc_max_pct,filecache_max, block size, blocksize, DIO, CIO, JFS, async I/O, UnifiedFile Cache, concurrent I/O

Reason and PrerequisitesSlow I/O performance, unknown or unusual mount options, incorrect blocksize

SolutionThe most common filesystem on HP-UX is VxFS (the Veritas Filesystem) - alsoknown as JFS (Journal Filesystem).

For more information on HP filesystems in general see also link:http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01915880/c01915880.pdf ("Supported File and File System Sizes for HFS and JFS")

HFS is not recommended for SAP use.

Summary & WARNING

This SAP note discusses the following configuration topics regarding theuse VxFS filesystems on HP-UX for SAP Database Instances:

- Setting the correct Blocksize - Concurrent I/O (CIO) - Direct I/O (DIO) - Buffered I/O - Other I/O issues which can affect performance - Additional Information on Database specific mount options

RAW device usage is not discussed here.

Other configuration variations are possible and may in some cases beappropriate. Since performance depends a lot on the behaviour of theindividual SAP and I/O subsystem, please test the chosen options beforeusing them in a productive environment.

This note also briefly discusses some other HP-UX I/O related customizingwhich can also help improve performance.

Setting the correct blocksizes

It is extremely important for Database performance that the filesystems usethe correct blocksize. This is equally true for direct I/O and bufferedI/O.

11.04.2013 of 14


Using the wrong blocksize causes unnecessary I/O overhead since blocksizeused by the database processes does not match the filesystem blocksize.

o Sybase Adaptive Server Enterprise

Use 8kb blocksize for all database filesystems.

o Oracle

For sapdata's use 8kb blockize

For redologs (origlog and mirrlog) use 1kb blocksize.

o MaxDB

Use 8kb blocksize for all database filesystems.

o DB2 for Linux, Unix and Windows

For sapdata's:

If DB2 Page Size equals 16kb for all Tablespaces use 8kbblocksize.

otherwise use 4kb blocksize.

For transaction logs use 4kb blocksize.

DB2 Page Size may be determined with the following command as userdb2<SID>:

db2 list tablespaces show detail | grep "Page size" |sort -u

o Other filesystems (non-DB, executables etc.)

The default blocksize may be used.

o How to check filesystem blocksize on HP-UX:

fstyp -v <Logical Volume> | grep f_frsize)

(e.g. fstyp -v /dev/vg00/lvol3 | grep f_frsize)

o How to create filesystems:

(For 8kb blocksize)

newfs -F vxfs -b 8192 -o largefiles /dev/<VG Name>/r<LV Name>

11.04.2013 of 14


(For 4kb blocksize)


(For 1kb blocksize)


Concurrent I/O (CIO)

Concurrent I/O bypasses filesystem caches and therefore avoids doublebuffering for databases, but also resolves problems of inode contention dueto available read-shared mode on write exclusive lock. As a resultConcurrent I/O can offer near RAW device performance for databases and isusually faster than Direct I/O or buffered I/O for Database instances.

For further information on the potential performance benefits see the HPWhite Paper "Performance improvements using Concurrent I/O on HP-UX 11i v3with OnlineJFS 5.0.1 and the HP-UX 11i Logical Volume Manager"http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA1-5719ENW.pdf

How to use Concurrent I/O:

o Check Concurrent I/O is available on the HP-UX system:

- Concurrent I/O (CIO) is available from VxFS 5.0 onwards.

- To enable CIO one of the following is required:

- VxFS 5.0 with an appropriate HP Storage Management Bundle,e.g. "hp Serviceguard Storage Management Suite"

- VxFS 5.0.1 (HP-UX 11.31 only) with an OnlineJFS License (Formore details seehttp://h20341.www2.hp.com/enterprise/w1/en/os/hpux11i-fsvm-whats-new.html )

The following command can be used to check VxFS 5.0.1 isinstalled:

# /usr/sbin/swlist -l bundle | grep -iE 'vxfs|vxvm'

The following commands can be used to check OnlineJFS isinstalled/enabled:

# vxlicrep | grep -i onlinejfs

or

# swlist -l product |grep -i onlinejfs

o Make sure the correct blocksizes are being used, see the section"setting the correct blocksizes" above.

o For systems with >=40GB Memory set the buffercache/filecache to 1%

11.04.2013 of 14


of Physical Memory (since the cache is not used by direct I/O).e.g. dbc_max_pct=dbc_min_pct=1 (HP-UX 11.11/11.23) orfilecache_min=filecache_max=1 (HP-UX 11.31)

o For Sybase Adaptive Server Enterprise:

If Concurrent I/O is supported by the file system, then SybaseASE automatically enables Concurrent I/O on the data files andtransaction log files.

It is NOT recommended to enable Concurrent I/O on file systemlevel using the CIO mount option as this would interfere withSybase ASE opening files with the correct options.

Therefore the following (default) mount options are recommendedin this case: ioerror=mwdisable,delaylog,largefiles

e.g.

/dev/vg01/lvol1 /test vxfs ioerror=mwdisable,delaylog,largefiles 0 2

o For Oracle:

- Set the Oracle Parameter "FILESYSTEMIO_OPTIONS" to "SETALL".See SAP Note 830576 (Oracle 10.2) and 1431798 (Oracle 11.2)

- Use the following mount options in /etc/fstab:cio,delaylog,nodatainlog

e.g.

/dev/vg01/lvol1 /test vxfs cio,delaylog,nodatainlog 0 2

- Do this for sapdata's, redologs (origlog's and mirrlog's) andarchive logs. Other filesystems should keep their defaultoptions (i.e. buffered I/O).

- Note: From Oracle Version 11.2.0.2 onwards the "cio" mountoption is no longer required for the database itself, but otherapplications/utilities accessing these filesystems may stillrequire the mount option. Therefore it is still recommended touse the "cio" mount option.

o For MaxDB:

- Use the following mount options in /etc/fstab:cio,delaylog,nodatainlog

e.g.

/dev/vg01/lvol1 /test vxfs cio,delaylog,nodatainlog 0 2

11.04.2013 of 14


- Do this for database and redolog filesystems (e.g. sapdata andsaplog). Other filesystems (e.g. MaxDB binaries) should keeptheir default options (i.e. buffered I/O).

o For DB2 for Linux, Unix and Windows (DB2 V9.5 andnewer):

Data Containers

If Concurrent I/O is supported by the file system, then DB2database Concurrent I/O should be enabled on the databasetablespace level by modifying the file system caching policy to "nofile system caching" via db2 command 'alter tablespace'.

It is NOT recommended to enable Concurrent I/O on file system levelusing the CIO mount option as this would interfere with DB2 openingfiles with the correct options.

Therefore the following mount options are recommended in this case:delaylog,nodatainlog

e.g.

/dev/vg01/lvol1 /test vxfs delaylog,nodatainlog 0 2

The file system caching policy option can be set as follows:

# db2 -v "alter tablespace tablespace_name no file systemcaching"

Note: The 'tablespace_name' is the name of the tablespace forwhich you want to have enabled for Concurrent I/O.

You may verify the set 'no file system caching' policy option for aparticular DB2 tablespace:

1. Use the DB2 get snapshot option to check for Concurrent I/O.

# db2 -v "get snapshot for tablespaces on dbname"

Note: 'dbname' is the database name.

2. Find the tablespace you want to check and look for the Filesystem caching attribute. If you see File system caching = No,then Concurrent I/O is enabled.

Note: As an alternative you may also use db2pd tool by usingthe command 'db2pd -db dbname -tablespaces'. Check for column'FSC' in the 'tablespace configuration' section of the output.The value 'OFF' in column 'FSC' means that Concurrent I/O isenabled for the related tablespace containers.

Transaction log files

If Concurrent I/O is supported by the file system, please note thatthe method for enabling Concurrent I/O for DB2 transaction logs

11.04.2013 of 14


also depends on the DB2 version in use.

- Enabling Concurrent I/O for DB2 transaction logs for DB2 V9.5:

- Use dedicated filesystems

- Please note, that there is NO need to set Concurrent I/O viafilesystem mount options. The "delaylog, nodatainlog" mountoptions should be used as described for DB2 data containersabove.

- Set the DB2 registry variable DB2_LOGGER_NON_BUFFERED_IO tovalue ON by using command

'db2set DB2_LOGGER_NON_BUFFERED_IO=ON'

Note: The DB2 internal default as of V9.5 FP2 is set to value'OFF' if the registry variable DB2_LOGGER_NON_BUFFERED_IO isNOT set.

All log file handles will be opened with Concurrent I/O. Thismay require further tuning for rollback activities and logarchiving.

- Enabling Concurrent I/O for DB2 transaction logs for DB2 V9.7:


- Please note, that you should NOT set Concurrent I/O viafilesystem mount options. The "delaylog, nodatainlog" mountoptions should be used as described for DB2 data containersabove.

- Unset the DB2 registry variable DB2_LOGGER_NON_BUFFERED_IO inorder to have set the DB2 default by using the followingcommand:

'db2set DB2_LOGGER_NON_BUFFERED_IO='

Note: This is equal to have set DB2 registry variableDB2_LOGGER_NON_BUFFERED_IO to value AUTOMATIC.

The log files within the active log space will be opened withConcurrent I/O. For Concurrent I/O, the inode lock is acquiredin read-shared mode for both read and write accesses. Theread-shared mode helps to improve the rollback performance. Allother log files will be opened in buffered mode, which alsohelps to improve the file copy performance during logarchiving.

SAP Temporary Tablespaces

Per default the DB2 temporary tablespaces are created as SMStablespaces using multiple container for striping. See also SAPnote 1471717 "DB6: Tempspace Performance" for details.

The SMS tablespace containers should always use buffered I/O

11.04.2013 of 14


(i.e. Do not use the "no file system caching" tablespace policyoption). See also SAP note 1451361 "DB2 Best Practices" fordetails.

Direct I/O (DIO)

The Direct I/O bypasses filesystem caches and therefore avoids doublebuffering for databases. In case Concurrent I/O is not available, butDirect I/O method is available, the Direct I/O is typically the preferredI/O method.

How to use direct I/O:

o Check Direct I/O is available on the HP-UX system:

- VxFS 5.0.1 (HP-UX 11.31 only) enables direct I/O by default.

The following command can be used to check VxFS 5.0.1 isinstalled:


- Prior to VxFS 5.0.1 the "OnlineJFS" product is required toenable direct I/O:

The following commands can be used to check OnlineJFS isinstalled/enabled:

# vxlicrep | grep -i onlinejfs

or

# swlist -l product |grep -i onlinejfs


o For systems with >=40GB Memory set the buffercache/filecache to 1%of Physical Memory (since the cache is not used by direct I/O).e.g. dbc_max_pct=dbc_min_pct=1 (HP-UX 11.11/11.23) orfilecache_min=filecache_max=1 (HP-UX 11.31)

o

o For Sybase Adaptive Server Enterprise:

If Direct I/O is supported by the file system, then Sybase ASEautomatically enables Direct I/O on the data files andtransaction log files unless Concurrent I/O is supported by thefile system as well.

It is NOT recommended to enable Direct I/O on file system levelusing the Direct I/O mount options as this would interfere with

11.04.2013 of 14


Sybase ASE opening files with the correct options.

Therefore the following (default) mount options are recommendedin this case: ioerror=mwdisable,delaylog,largefiles

e.g.

/dev/vg01/lvol1 /test vxfs ioerror=mwdisable,delaylog,largefiles 0 2

o For Oracle:

- Set the Oracle Parameter "FILESYSTEMIO_OPTIONS" to "SETALL".See SAP Note 830576 (Oracle 10.2) and 1431798 (Oracle 11.2)

- Use the following mount options in /etc/fstab:convosync=direct,mincache=direct,delaylog,nodatainlog

e.g.

/dev/vg01/lvol1 /test vxfsconvosync=direct,mincache=direct,delaylog,nodatainlog 0 2

- Do this for sapdata's, redologs (origlog's and mirrlog's) andarchive logs. Other filesystems should keep their defaultoptions (i.e. buffered I/O).

- Note: From Oracle Version 11.2.0.2 onwards the"convosync=direct,mincache=direct" mount options are no longerrequired for the database itself, but otherapplications/utilities accessing these filesystems may stillrequire the mount options. Therefore it is still recommended touse the "convosync=direct,mincache=direct" mount options.

o For MaxDB:

- Use the following mount options in /etc/fstab:

convosync=direct,mincache=direct,delaylog,nodatainlog

e.g.


- Do this for database and redolog filesystems (e.g. sapdata andsaplog). Other filesystems (e.g. MaxDB binaries) should keeptheir default options (i.e. buffered I/O).

o For DB2 for Linux, Unix and Windows:

DB2 Data Containers

Direct I/O should be enabled on the database tablespace level by

11.04.2013 of 14


modifying the file system caching policy to 'no file systemcaching' via command 'alter tablespace'. This is exactly the samesetting on the DB2 tablespace level as used for Concurrent I/O.However, this depends on the availability of the I/O method on thefile system.

Please note, that it is NOT recommended to use the Direct I/O mountoptions on file system level as this would interfere with DB2opening files with the correct options.

Therefore the following mount options are recommended in this case:delaylog,nodatainlog

e.g.

/dev/vg01/lvol1 /test vxfs delaylog,nodatainlog 0 2

The 'no file system caching' policy option can be set as follows:

# db2 -v "alter tablespace tablespace_name no file systemcaching"

Note: The 'tablespace_name' is the name of the tablespace forwhich associated containers you want to have enabled DirectI/O.

To verify that the "no file system caching" policy option has beenset for a particular DB2 tablespace:

1. Use the DB2 get snapshot option to check for Concurrent I/O.

# db2 -v "get snapshot for tablespaces on dbname"

Note: 'dbname' is the database name.

Note: As an alternative you may also use db2pd tool by usingthe command 'db2pd -db dbname -tablespaces'. Check for column'FSC' in the 'tablespace configuration' list of the output.The value 'OFF' in column 'FSC' means that Direct I/O isenabled for the related tablespace containers, if thefilesystem does not support Concurrent I/O.

2. Find the tablespace you want to check and look for the Filesystem caching attribute.

Transaction log files

Compared with Concurrent I/O the Direct I/O method does NOT supporta read-shared mode during write when flushing log buffer. As aconsequence, a write on an inode at file system level will set anexclusive lock, which may cause wait situations for concurrentlyrunning reads (due to concurrent rollback activity).

The method for enabling Direct I/O for DB2 transaction logs dependson the DB2 version, but is similiar to the method if you want toget enabled Concurrent I/O.

- Enabling Direct I/O for DB2 transaction logs for DB2 V9.1:

11.04.2013 of 14



- Set Direct I/O with the following mount options in/etc/fstab:convosync=direct,mincache=direct,delaylog,nodatainlog

e.g.


- Enabling Direct I/O for DB2 transaction logs for DB2 V9.5:


- Please note, that there is NO need to set Direct I/O viafilesystem mount options. The "delaylog, nodatainlog" mountoptions should be used as described for DB2 data containersabove.

- Set the DB2 registry variable DB2_LOGGER_NON_BUFFERED_IO tovalue ON by using command'db2set DB2_LOGGER_NON_BUFFERED_IO=ON'

Note: The DB2 internal default as of V9.5 FP2 is set to value'OFF' if the registry variable DB2_LOGGER_NON_BUFFERED_IO isNOT set.

All log file handles will be opened with Direct I/O.Concurrently running reads during write against log files mayresult into inode contention / locking on file system level,which might impact the performance for rollback activities. Inaddition, the rollback and log archiving cannot benefit fromcaching data on file system level.

- Enabling Direct I/O for DB2 transaction logs for DB2 9.7:


- Please note, that there is NO need to set Direct I/O viafilesystem mount options. The "delaylog, nodatainlog" mountoptions should be used as described for DB2 data containersabove.

- Unset the DB2 registry variable DB2_LOGGER_NON_BUFFERED_IO inorder to have set the DB2 default by using the followingcommand:

'db2set DB2_LOGGER_NON_BUFFERED_IO='

Note: This is equal to have set DB2 registry variableDB2_LOGGER_NON_BUFFERED_IO to value AUTOMATIC.

The log files within the active log space will be opened with

11.04.2013 of 14


Direct I/O. For Direct I/O, the inode lock situation may causea serialization of concurrent reads during write accesses dueto missing read-shared mode. You may tune to improve rollbackperformance and log archiving is recommended.

ATTENTION: Error in DB2 V9.5 and V9.7 with VxFS 5.x

Older patch levels of DB2 V9.5 and V9.7 do not enable DIO asdescribed. In this case the DB2 patches described in SAP Note1586523 should be installed or the follow mount options used asa workaround:

/etc/fstab:convosync=direct,mincache=direct,delaylog,nodatainlog

e.g.


SAP Temporary Tablespaces

Per default the DB2 temporary tablespaces are created as SMStablespaces using multiple container for striping. See also SAPnote 1471717 "DB6: Tempspace Performance" for details.

The SMS tablespace containers should always use buffered I/O(i.e. Do not use the "no file system caching" tablespace policyoption). See also SAP note 1451361 "DB2 Best Practices" fordetails.

Technical notes:

o The "convosync=direct" option handles any reads or writes with theO_SYNC or O_DSYNC flags

o The "mincache=direct" option handles the reads or writes withoutthe O_SYNC flag.

o (For Oracle) Oracle opens files with the "O_DSYNC" flag, so forthis the "convosync=direct" option is needed.

o The "mincache=direct" option is needed to force other applicationsto use direct I/O, e.g. a backup program. As a result this optionhas the biggest impact when used on the sapdata filesystems.

Oracle and DBVERIFY:If you use DBVERIFY (which you're probably doing if not using RMAN forbackups) then it is not recommended to use the "mincache=direct" option,since the many small logical reads from DBVERIFY typically benefit morefrom the caching and read-ahead features of buffered I/O. However, with

11.04.2013 of 14


careful configuration it is possible to combine direct I/O for backup andbuffered I/O for DBVERIFY by making "discovered_direct_iosz" entries forthe relevant filesystems in /etc/vx/tunefstab.

e.g. # cat /etc/vx/tunefstab /dev/vgXX/lvolXX discovered_direct_iosz=128K ... ..

NOTE: In this case it is important that the read request size of the backupsoftware is on or above the limit defined in tunefstab.

Please contact your backup software support representative for advice onconfiguring backup with direct I/O.

For more information on the above options please see "man tunefstab" and"man mount_vxfs"

Buffered I/O

Buffered I/O is the HP-UX Operating System default configuration.

How to use buffered I/O:


o Use the following mount options in /etc/fstab:delaylog,nodatainlog

o Do this for sapdata's and redologs (origlog's and mirrlog's) Usethe buffercache/filecache recommendations from note 172747

Technical notes:

o In some cases a BW/BI system might benefit from using buffered I/Ofor the sapdata filesystems. If this is the case then the"read_pref_io" parameter in /etc/vx/tunefstab should also beinvestigated. Please contact your HP support representative if youneed advice on configuring this option.

Other I/O issues which can affect performance

o Optimize the size of HP-UX kernel parameter vx_ninode (VXFS inodecache size)

- For VXFS Version >= 4.1 set vx_ninode to the value in note172747 (or use the formula "number CPU * 1024" for more exacttuning).

- For VXFS Version 3.x set vx_ninode to 40,000.

- To check VxFS version use one of the following commands:

11.04.2013 of 14


# vxlicrep | grep -i version

or


If no version number is displayed in swlist see the followingdefault list:

- HP-UX <= 11.23 VxFS Version 3.x is default

- HP-UX 11.31 VxFS Version 4.1 is default

o If vx_ninode has been changed to a fixed value as above, then it ispossible to reduce CPU load by making the vx_ninode cache static. Astatic vx_ninode cache means that the HP-UX OS no longer consumesCPU time trying to free memory by removing old inode cache entries.To make the vx_ninode static the HP-UX kernel parameter"vxfs_ifree_timelag" must be set to -1 (NOTE: a static vx_ninodecache should only be used when the size of vx_ninode has beencarefully defined to avoid extreme/unwanted memory consumption).

Additional Information on Database specific mount options

o For more information on mount options and Oracle see:

- HP White Paper "HP-UX VxFS tuning and performance"http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01919408/c01919408.pdf

- SAP note 793113 "FAQ: Oracle I/O configuration"

- HP White Paper "Performance improvements using Concurrent I/Oon HP-UX 11i v3 with OnlineJFS 5.0.1 and the HP-UX 11i LogicalVolume Manager"http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA1-5719ENW.pdf

- Oracle Metalink DocID 1231869.1: "How to use Concurrent I/O onHP-UX and improve throughput on an Oracle single-instancedatabase"

- HP White Paper "HP-UX JFS mount options for Oracle Databaseenvironments"http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA1-9839ENW.pdf

o For more information on mount options and MaxDB/LiveCache see:

- SAP note 993848

o For more information on DB2 configuration see:

- SAP note 147634 (in particular the "page size" recommendationsin section 4)

11.04.2013 of 14


Header Data

Release Status: Released for CustomerReleased on: 17.10.2011 10:04:39Master Language: EnglishPriority: Recommendations/additional infoCategory: PerformancePrimary Component: BC-OP-HPX HP-UX

The Note is release-independent

Related Notes

Number Short Text

1586523 DB6 on HP: DIO is not being set by DB2 with VxFS 5.x

1538899 DB6: SCHED_NOAGE on HP-UX

1471717 DB6: Tempspace Performance

1451361 DB6: Best Practices for HP-UX systems running DB2 LUW

1431798 Oracle 11.2.0: Database Parameter Settings

1075118 SAP on HP-UX: FAQ

999524 Oracle 10.2 platform support for Async IO

993848 Direct I/O mount options for liveCache/MaxDB

830576 Parameter recommendations for Oracle 10g

793113 FAQ: Oracle I/O configuration

172747 SAP on HP-UX: OS kernel parameter recommendations

147634 DB6: Tips and tricks for creating DB2 tablespaces

sapnote_0001077887.pdf

Documents

Transcript of sapnote_0001077887.pdf