IBM® PowerLinux™Open Source Infrastructure ServicesImplementation and Tuning Guide

April 23, 2012(Version 9)

Paul Clarke and Jason FurmanekLab Services Power Systems

pacman@us.ibm.co m furmanek@us.ibm.co m

SummaryThe Open Source Infrastructure Services (OSIS) Implementation and Tuning Guide provides an overview of implementation and tuning of open source applications on an IBM PowerLinux system. The document first introduces a popular set of open source applications in support of web, mail, file, print and network serving. It shows solution scenarios based on open source applications from the Red Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES) Linux distributions. It then shows considerations for the implementation and tuning of these solutions.

The intended audiences for this guide are professionals planning for open source applications on IBM PowerLinux. It is one of two OSIS solution guides. For information on OSIS reference architecture and sizing, refer to the second solution guide, IBM PowerLinux ™ Open Source Infrastructure Services Reference Architecture and Sizing Guide .

Table of Contents

TABLE OF CONTENTS............................................................................................................................................3

1 INTRODUCTION ......................................................................................................................................................5

2 SOLUTION GUIDE USE...........................................................................................................................................6

3 OSIS REFERENCE ARCHITECTURE..................................................................................................................7

3.1 VIRTUALIZATION WITH IBM POWERVM..........................................................................................................................93.2 OPEN SOURCE INFRASTRUCTURE SERVICES (OSIS)...........................................................................................................9

4 SOFTWARE COMPONENTS.................................................................................................................................11

5 INSTALLATION AND CONFIGURATION..........................................................................................................11

5.1 INSTALL PREREQUISITES................................................................................................................................................125.2 INSTALLING VIRTUAL I/O SERVER (VIOS)....................................................................................................................12

5.2.1 Accessing FSP Using Advanced System Management (ASM).............................................................135.2.2 Installing VIOS.....................................................................................................................................145.2.3 Reconfiguring FSP Post VIOS Installation..........................................................................................155.2.4 Configuring VIOS.................................................................................................................................155.2.5 Accessing IBM Virtualization Manger (IVM)......................................................................................15

5.3 CREATING VIRTUAL SERVERS........................................................................................................................................155.3.1 Preparing VIOS for Virtual Servers.....................................................................................................165.3.2 Creating Virtual Disks for Virtual Server Storage...............................................................................215.3.3 Creating Virtual Servers.......................................................................................................................235.3.4 Preparing a Virtual Server for Activation............................................................................................265.3.5 Installing Linux on a Virtual Server’s Partition...................................................................................285.3.6 Configuring a Linux Software RAID Device........................................................................................33

5.4 CONFIGURING OSIS WORKLOADS.................................................................................................................................345.4.1 Configuring webmin for Administration..............................................................................................345.4.2 Configuration of Mail Application Server...........................................................................................345.4.3 Configuring Postfix Anti-spam.............................................................................................................365.4.4 Configure Sending Mail for Performance Testing...............................................................................365.4.5 Configure Retrieving Mail for Performance Testing............................................................................36

5.5 MONITORING SOFTWARE...............................................................................................................................................375.5.1 Configuring Ganglia on Monitored Virtual Servers............................................................................385.5.2 Configuring Ganglia on Collection System.........................................................................................39

6 PERFORMANCE TUNING....................................................................................................................................40

6.1 REFERENCE ARCHITECTURE TUNING PERFORMED ............................................................................................................406.2 PHYSICAL AND VIRTUAL PROCESSOR SETTINGS................................................................................................................416.3 ADDITIONAL TUNING CONSIDERATIONS...........................................................................................................................426.4 IBM JAVA PERFORMANCE TUNING................................................................................................................................43

7 APPENDIX................................................................................................................................................................44

7.1 INSTALL LINUX DIRECTLY USING VNC ........................................................................................................................447.2 CREATING PARTITIONS WITH LVM DURING DIRECT LINUX INSTALLATION...........................................................................457.3 CREATING PARTITIONS WITH LVM DURING IBM INSTALLATION TOOLKIT INSTALLATIONS.....................................................46

8 REFERENCES..........................................................................................................................................................49

DISCLAIMER AND TRADEMARKS...................................................................................................................51

1 Introduction A survey by OpenLogic reveals that many enterprises consider using open source and adoption of open source is on the rise. The IBM PowerLinux Open Source Infrastructure Services (OSIS) reference architecture consists of open source software solutions available from Linux distributions provided by Red Hat and SUSE and the open source community. IBM PowerLinux servers with PowerVM™ offer a highly virtualized, workload optimized, cloudready platform that can support more workloads per server and greater throughput per virtual server – saving up to 16% or more in acquisition costs as compared to HP ProLiant systems with VMware1. It is an ideal platform for replacing more expensive infrastructure applications with robust open source offerings. Open source applications for PowerLinux servers are included and supported with commercial Linux distributions from Red Hat and Novell.

These open source applications provide a highly flexible environment for IBM PowerLinux servers, which means choices in architecture, sizing and individual server configurations.

Combining IBM PowerLinux 7R2 with PowerVM's superior virtualization capabilities enables up to 160 simultaneously active virtual servers on a single 16core system. Such a capable system provides an excellent reference platform for a consolidated open source solution.

This paper describes a study IBM has performed with several of the open source applications. Based on experiences and results from installation, configuration and performance testing, this paper describes architectures, results from tested configurations, along with suggested approaches to scaling, sizing, migration, consolidation, and availability.

1 Achieve 16% lower total cost of acquisition over years with 2 IBM POWER7, two socket, 16core, 3.55GHz servers instead of 4 HP DL380 G7 two socket, 12core, 3.3 GHz servers, leveraging the higher utilization and virtualization efficiencies of PowerVM.

Performance based on IBM analysis of the SPECint_rate benchmark on HP DL380 G7 two socket, 12core server (3.33 GHz Intel® Xeon® 5680 processor) and on the IBM PowerLinux 7R2 two socket, 16core server (3.55GHz processors).

Prices from www.hp.com (link resides outside of ibm.com).

Assumption is that the throughput impact of VMware is 10%, based on SAP Note 1409608 (10% overhead) and VMware whitepaper “Virtualizing Business Critical Applications”, 2010 (“keeping virtualization overhead at a very limited at 210 percent “) at http://www.vmware.com/files/pdf/VMW_10Q1_WP_vSPHERE_USLET_EN_R6_proof.pdf

2 Solution Guide UseWhen reviewing the potential use IBM PowerLinux systems with open source applications, OSIS solution guides can be part of an overall assessment process with a customer. Figure 1 below outlines several phases and activities that are appropriate in working on an OSIS proposal with a client.

1. Discover client’s technical requirements and usage (hardware, software, data center)2. Analyze their requirements and current environment3. Exploit with proposals based on IBM PowerLinux servers and OSIS

Figure 1 Client Technical Discovery, Analysis and Exploitation

3 OSIS Reference ArchitectureThe IBM PowerLinux Open Source Infrastructure Services (OSIS) solution landscape consists of IBM PowerLinux 7R2 hardware, virtualization software, Linux OS, and open source applications. Refer to Figure 2 below for details.

Figure 2 OSIS Reference Architecture Overview

NOTE: IBM provides the IBM Installation Toolkit with simplified setup tools for several commonly used applications for mail, web, print, file, and network serving.

The OSIS reference architecture landscape includes the following software:1. IBM provided:

IBM PowerVM Enterprise Edition – provides virtualization of IBM PowerLinux servers, allowing up to 160 virtual servers configured per system. PowerVM includes: Virtual I/O Server (VIOS) manages system CPU, memory and network resource sharing Integration Virtualization Manager (IVM) provides a web based virtualization manager

IBM Installation Toolkit a nocharge offering that provides installation of Linux and configuration of open source applications for web, mail, file, print and network serving

Optional management consoles (for solutions with multiple systems) Hardware Management Console (HMC) includes both hardware and software used to perform a variety of

system management tasks for all IBM Power Systems, including IBM PowerLinux. In particular, HMC may be used to create or change logical partitions, including dynamically assigning hardware to a partition.

IBM Systems Director Management Console (SDMC) - provides server hardware management and virtualization management.

Optional Cloud Foundation (advanced virtualization with image control) IBM® Systems Director for IBM PowerLinux™ provides the integrated tools needed to efficiently visualize

and communicate the relationships of physical and virtual systems that are discovered, monitor their health, define and receive threshold alerts, and update system firmware and operating environments. Builtin automation capabilities enable IT administrators to schedule updates and configuration changes to proactively avoid problems, and reduce the administrative burden of routine maintenance.

IBM® Systems Director VMControl™ for IBM PowerLinux™provides creations and modification of system pools using PowerLinux virtual workloads, make dynamic virtual workload adjustments and move work

loads within system pools, resulting in an optimized virtual environment with increased resilience to cope with planned or unplanned downtime.

2. Linux Distributions from Red Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES) provide both a Linux operating system and the following open source applications of the OSIS landscape: Postfix provides mail serving by receiving and delivery of mail. Postfix is an alternative to the sendmail server,

which is also included with Linux distributions. Dovecot and Cyrus provide interoperability with other mail servers and clients using POP, IMAP4 and MSS in

dustry standard protocols fetchmail provides mail clients with mail retrieval procmail , SpamAssassin provide mail filtering and antispam protection smtp-source provides mail generation for performance testing Apache HTTP Server provides HTTP web serving MySQL provides database serving PHP provides PHP scripting Berkeley Internet Naming Domain (BIND) provides domain name services (DNS) Squid caching proxy for the web supporting HTTP, HTTPS, FTP, and more mpstat , nmon, iostat provide system utilization information; CPU, memory, network, disk

3. Other optional (nocharge) open source applications: Ganglia provides a system resource utilization monitoring system that is highly scalable and customizable webmin provides a web based administration interface for managing Linux applications and services associated

with the OSIS reference architecture Drupal provide webbased content management

This solution guide focuses on the following mail and web serving applications that are included with Linux distributions, optionally installed and configured using the IBM Installation Toolkit. The web serving applications includes the Apache Web Server (Apache), which typically runs PHP/Perl scripts with a

database such as MySQL. The combination of Linux, Apache, PHP and MySQL is commonly referred to as the “LAMP” architecture or stack.

The mail serving applications include Postfix, Dovecot and Cyrus. The Postfix mail server supports the Simple Mail Transfer Protocol, while both Dovecot and Cyrus integrate with Postfix to provide Post Office Protocol (POP) and Internet Message Access Protocol (IMAP). A comparison of mail servers is available on Wikipedia.

Typical x86based landscapes for open source mail and web servers consist of multiple application servers and possibly storage servers. As shown in below, a typical end user request flows as follows:

For web serving,1. A browser submits an enduser request that the web server corresponding to the web page processes.2. Web server retrieves static and/or dynamic data from a storage device and returns it to the browser. 3. Programs; such has PHP scripts retrieve dynamic content from a database and return the content to the browser.

For mail serving,1. Mail clients send enduser emails to the

mail server in their mail domain.2. If necessary, the mail server forwards

emails to the mail server of the recipient’s mail domain.

3. Mail filtering is applied (open source examples include Procmail Mail Filter and SpamAssassin Mail Filter).

4. The mail server stores mail.5. The recipient requests mail.

IBM information center provides additional information about running Linux open source applications on IBM Power systems.

3.1 Virtualization with IBM PowerVMUtilizing IBM PowerVM for virtualization is fundamental to the OSIS landscape. Notice how each of the following goals for virtualization aligns with requirements for a OSIS reference architecture: Consolidate multiple environments, including underutilized servers and systems with varied and dynamic resource re

quirements. Grow and shrink resources dynamically, derive energy efficiency, save space, and optimize resource utilization. Deploy new workloads by provisioning virtual machines or new systems rapidly to meet changing business demands. Develop and test applications in secure, independent domains while production can be isolated to its own domain on the

same system. Transfer live workloads to support server migrations, balance system load, or avoid planned downtime. Control server sprawl and thereby reduce system management costs.

IBM PowerLinux systems deployed for open source mail and web applications may consist of either addon systems or replacements of existing systems. Using IBM PowerVM virtualization technology, a single system can run multiple virtual servers, each running one or more applications. Virtual servers can share physical resources such as processors, memory, and networking devices. For proper deployment, sizing of these IBM PowerLinux systems and virtual servers is required. The term IBM Power Systems virtual server is used throughout this paper, instead of the equivalent term Logical Partition (LPAR) or term Virtual Machine (VM) used with x86based virtualization.

PowerVM is the industryleading virtualization solution for AIX, IBM i, and Linux environments on IBM POWER technology. PowerVM offers a secure virtualization environment, built on the advanced RAS features and leadership performance of the Power Systems platform. It features leading technologies such as Power Hypervisor, MicroPartitioning, Dynamic Logical Partitioning, Shared Processor Pools, Shared Storage Pools, Integrated Virtualization Manager, PowerVM Lx86, Live Partition Mobility, Active Memory Sharing, N Port ID Virtualization, and Suspend/Resume. PowerVM is a combination of hardware enablement and valueadded software. Additional information is available in the IBM Redbooks publication IBM PowerVM Virtualization Introduction and Configuration, SG247940.

By applying PowerVM virtualization technology, it is possible to configure multiple virtual web, database, and mail servers utilizing a pool of system resources with one IBM PowerLinux system. Avoid network load from affecting the physical network by utilizing a high capacity virtual network between these virtual servers. The benefits of virtualizing workloads with PowerVM in this way include: Higher resource utilization: Promoting high resource utilization by virtualizing resources including processors,

memory, and I/O across multiple virtual machines. Consolidation: Hosting diverse workloads on the same server. Reduced costs: Saving system administrator time and IT infrastructure costs. Scalability: Simplifying deployment of multiple copies of the same workload type. PowerVM supports virtual servers

as small as 1/10 of a processor core, allowing up to 160 virtual servers on an IBM PowerLinux system with 16 cores. Recoverability: Bringing a workload back online after an outage, quickly and reliably. Rapid provisioning: Deploying the readytorun workload, quickly and easily. Availability: Eliminating planned downtime by moving a running partition to another server, with Live Partition Mo

bility, while upgrading or maintaining hardwarewithout interrupting productive work.

Note that a key difference between the sharing of the physical system resources using PowerVM versus x86based virtualization technologies is the handling of virtual server CPU and memory commitments. Unlike x86based virtualization technologies that only add CPU and memory resources to active virtual servers, PowerVM can also remove both CPU and memory resources, reallocating these resources transparently to other virtual servers when and where the resources are needed. For example, PowerVM dynamically removes resources from an underutilized virtual mail server and adds the resource to a virtual web server when needed. This feature, commonly referred to as Dynamic Logical Partitioning (DLPAR), allows for configuration of a larger number of virtual severs per system, up to 160 virtual servers for a 16 core Power System. For additional information on higher system utilization and performance using PowerVM, refer to A Comparison of PowerVM and x86Based Virtualization Performance.

3.2 Open Source Infrastructure Services (OSIS)With the adoption of virtualization using IBM PowerVM, it is possible to consolidate both web and mail to one or more IBM PowerLinux systems using virtual servers as shown in Figure 3. Migration of multiple physical servers to virtual servers is possible without changing the network flow or application handling of requests. However, there are several key differences to point out:

1. It is possible to configure Virtual Ethernet connections directly between web and database virtual servers, and between mail virtual servers. These high speed (in memory) virtual Ethernet connections reduce IP traffic on network switches, improving the overall network performance and network reliability.2. Virtual servers dynamically share CPU and memory. Since web and mail workloads vary throughout the day, a more efficient utilization of system resources occurs. PowerVMs efficient and dynamic sharing of physical resources by the virtual servers enables adding additional virtual servers to leverage any underutilized systems resources. When sizing open source workloads, proper selection of hardware, monitoring of system resources and a careful mix of workloads can result in a highly utilized system that consolidates a significant number of workloads on a single server.

Figure 3 OSIS Web & Mail Virtual Servers

PowerLinux Highlight: Consolidate up to 160 physical servers to a single PowerLinux 16 core system using PowerVM and virtual servers.

16 cores per systemx max 10 vCPUs per corex max 4 threads per vCPU= max 640 threads (or logical CPUs per system)

4 Software ComponentsTable 1 below provides a summary of the software components used in the tested OSIS configuration.

Software Description Software Source DocumentationLinux Distribution(supported at the time of this writing:• RHEL 5.7, 6.1,

and 6.2• SLES 11 SP1)

Red Hat Enterprise Linux (RHEL) or Novell SUSE distributions of Linux software

Linux distributors (Red Hat or Novell). If ordered from IBM, IBM provides a key to allow registration and download from the distributor’s website. Red Hat and SUSE evaluation software is available.

SLES 11RHEL

IBM Installation Toolkit v5.1

Provides a UI for install Linux on Power Systems, including services for installing and configuring OSIS.

IBM Installation Toolkit website, refer to Download for ISO file.

IBM Installation Toolkit website, refer to Install & Use for documentation files.IBM Information Center

Postfix, Dovecot, Cyrus, procmail, fetchmail, SpamAssassin

Linux open source applications for mail services

Included with Linux distribution Linux manual (man pages)IBM Information CenterPostfix documentation

Apache HTTP server, MySQL, PHP

Open source LAMP stack for web services

Included with Linux distribution Linux manual (man pages)IBM Information Center

webmin Open source utility providing a UI for managing Linux services and applications

webmin website, downloads webmin website

Ganglia Monitoring System

Graphic UI and monitoring system for use with multiple systems including IBM PowerLinux and virtual servers.

Ganglia for IBM Power website Ganglia website

mpstat, nmon, iostat Open source utilities for monitoring a virtual server’s CPU, memory, disk I/O and network I/O

Included with Linux distribution Linux manual (man pages)

Drupal v7 Content management system written in PHP for hosting web sites with both static and dynamic content.

Available from Extra Packages for Extra Packages for Enterprise Linux 6 (EPEL6)

Drupal website

Advanced System Management (ASM)

Provides an interface into the FSP (service processor)

Provided with IBM PowerLinux firmware

IBM Information Center

IBM PowerVM Virtual I/O Server (VIOS)

Provides virtualization services Included with IBM PowerVM Editions

IBM Information Center

Virtual I/O Server provides information and fixes in Fix Central

Integrated Virtualization Manager (IVM)

Provides UI for managing virtualization

Included with VIOS IBM I nformation Center Getting started with virtualization

Table 1 VIOS Software

5 Installation and ConfigurationThe following sections cover several areas of software installation, including the installation of the PowerVM Virtual I/O Server, Linux operating system and open source software.

5.1 Install PrerequisitesBelow are prerequisites for installation of VIOS and Linux:1. Hardware

NOTE: A 220v PDU is required for systems ordered with the default 220v power supply (feature code 5603) and power cords (feature code 6671). The 6671 power cord looks similar to a standard 110v power cord and does allow the FSP to power on with 110v, but the virtual servers will not start until 220v is used. The system will produce service code 11002613 when attached to an 110V power source. More information is available at http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/topic/area7/11002613.htm

a. Ethernet cables for each adapter Ethernet port and an additional cable for the HMC portb. One 9 pin serial cable (with all pins wired) for connecting a PC to the system used during VIOS installs. c. A RJ45 to serial adapter (IBM Feature code 3930) is required to convert from a DB9 to RJ45 style serial port.

NOTE: This cable is a 'nullmodem' communication cable. If an incorrect (partially wired) cable is used, then the VIOS install process may fail (terminal window stops responding). NOTE: For personal computers without a serial port, a USB to Serial adapter may be used.

d. Install Fibre Channel wrap plugs on any open ports. Make sure it is not a Fibre Channel protective covers/dust boots.

NOTE: If an orange bracket used in shipping is connected to the back of the power supply, remove it and push the power supplies in, then connect the power cords.

2. Mediaa. PowerVM VIOS installation DVDb. IBM Installation Toolkit v5.1 as an ISO file stored on a CD/DVD or on a PC’s disk drivec. Linux distribution as an ISO file stored on a CD/DVD or on a PC’s disk drive

3. Networkinga. Static IP address for accessing the field service processor (FSP) via the Advanced Management Service (ASM) in

terfaceb. Static IP address for accessing VIOS via Integrated Virtualization Manager (IVM) webbased interfacec. Static IP addresses for each of the virtual servers

4. Internet Connectivity to access a. Linux repositories containing additional open source softwareb. IBM.com for IBM Installation Toolkit updatesc. Linux distribution updates

5.2 Installing Virtual I/O Server (VIOS)This section covers the installation of VIOS without the use of a Hardware Management Console (HMC). This would be typical of OSIS solutions with a few IBM PowerLinux systems. Larger solutions could utilize HMC or System Director Management Console (SDMC).

The following sections cover the configuration required to access the Field Service Processor (FSP), changing the system boot device and the installation of VIOS.

5.2.1 Accessing FSP Using Advanced System Management (ASM)Access to the Field Service Processor (FSP) is performed using either the CLI or GUI of Advanced System Management (ASM), or both. The ASM GUI is more user friendly for overall configuration of the FSP, while the ASM CLI is necessary to change the system’s boot configuration. The following sections cover the configuration of both interfaces.

5.2.1.1 Configuring ASM Command Line Interface The ASM Command Line Interface (CLI) is used for configuring the FSP to boot from the VIOS installation media. A PC can access the ASM CLI over a serial connection using serial terminal emulator software such as Microsoft HyperTerminal or PuTTY.For a Linux based PC, minicom or other terminal emulator software may be used.

The following section covers PuTTY setup only. Note PuTTY is also very useful for providing remote terminal sessions into Linux virtual servers.1. Connect the power cords to the PDU and wait until the control panel displays

“01” (a series of program codes appear first).NOTE: Do not connect an Ethernet cable to HMC1 or HMC2 ports yet (done later).NOTE: The system is powered on if the light on the control panel is green. NOTE: To view the control panel, press the blue switch to the left, then pull out the control panel all the way, and then pull it down.

2. Install PuTTY on a personal computer3. Connect the serial cable from the top RJ45 connector found next to the PowerLin

ux power supply.4. Connect the serial cable directly to the PC’s serial connector or use a serial to

USB adapter to connect to a USB port.5. Run PuTTY 6. From the PuTTY interface

a. For Connection type click Serialb. For Serial line type COMx, where x is COM port associated with the serial connector. If not known, try different

COMx values until a connection is made.c. For Speed, enter 19200 (parity is:none and data/stop bits are 8/1)d. Click Open to open a terminal emulation window over the serial connection to ASM.e. Press Enter to request a response from ASM. If a connection is not made, try a different COMx port.f. Log in (default ID is admin and password is admin)

5.2.1.2 Configuring ASM Graphical User Interface Access the ASM Graphical User Interface (GUI) using a web browser and a private Ethernet connection. By default, the IBM PowerLinux system ships with a default IP address as indicated below. Optionally configure ASM to use a different static address that would provide additional network connectivity.

NOTE: For previously configured systems where the FSP IP address may not still be the default value, use ASM CLI to display the IP address of the FSP by clicking on; Network Services, Network Configuration, IPv4, and Configure interface Eth0. If the FSP IP address has changed, then use the new values in the next steps.First, use the system’s default IP address to access the ASM GUI by1. connecting an Ethernet cable from a personal computer (PC) to either HMC port 1 or 2 identified with labels in the

back of the PowerLinux system2. setting the PC’s TCP/IP connection properties to

a. an address in the same subnet as the current FSP IP address. For example when using FSP factory default address: 169.254.2.148 (when using HMC port 1) or 254.3.148 (when using HMC port 2)

b. subnet mask of 255.255.255.03. Open a browser to the FSP IP address. For example when using a factory default address use either

a. https://169.254.2.147 (HMC port 1) or b. https://169.254.3.147 (HMC port 2)

Once connected to ASM using the default IP address, ASM can be configured with a static IP address as follows:1. from the ASM interface, open the Network

Configuration panel as follows:a. click Expand all menusb. click Network Servicesc. click Network Configuration

2. Update the Network interface eth0 with the information for the new static IP address to be used by ASM

3. Save the configuration4. Move the Ethernet cable from the PC to an Ethernet switch associated with the new static IP address for ASM5. Connect a personal computer to the same network that as the IP address for ASM6. Open a browser to the IP address for ASM

5.2.2 Installing VIOSThe OSIS architecture requires the installation of VIOS for virtualizing resources. The following sections cover the installation of VIOS from the PowerVM media and configuring VIOS with a TCP/IP address, allowing access of the Integrated Virtualization Manager (IVM) from a web browser.

Installation of the PowerVM Virtual I/O Server (VIOS) from the PowerVM product DVD media as follows:1. Insert the DVD with VIOS into the PowerLinux DVD drive.2. Access the ASM console using a web browser (see previous sec

tion for URL)3. Authenticate with ID and password, which are by default admin

and admin 4. Configure system to boot to system from PowerVM VIOS

CDROMa. Select Power/Restart Control b. Select Power On/Off Systemc. Select Running (Auto-Start Always) for Server firmware

start policyd. Select Boot to SMS menu for AIX/Linux partition mode

boote. Click Save settings and power on

5. Use PuTTY to connect to the ASM CLI (see previous section for serial connection information).6. PowerVM Firmware boot prompts

a. optionally select a language, press Enter to continueb. press 2 and Enter to select Continue Bootc. press 1 and Enter to Accept the licensed. Press 1 to select SMS Menu

7. Use SMS Menu to change the boot ordera. Press 2 to Continue Password Entryb. Enter SMS password (admin by default)c. From Main Menu, press 5 for Select Boot Optionsd. From Multiboot, press 1 for Select Install/Boot Devicee. From Select Device Type menu, press 7 for List All Devicesf. From List All Devices menu, select the CD/DVD deviceg. Select 2 for Normal Boot Modeh. Select 1 to exit SMS

8. After a few minutes, a prompt appears requesting selection of the device to use for the console. Select the current device to use as the console.

9. After a few more minutes, VIOS will start installing10. After a few more minutes, select the language to be used in the install11. To install PowerVM Enterprise Edition

a. from the Installation and Maintenance menu, select 2 for Change / Show Installation Settings and Installb. From the settings menu,

i. Select 5 for Select Edition to change the edition to from the Express default value to standard. Select option 5 again to change the value to Enterprise.

ii. select 0 to install with the settings listed12. Installation takes about 30 minutes. Status is displayed during the install13. After installation, system reboots back to the SMS menu14. Reset the boot device to use the disk used during VIOS install (hdisk0 by default). Use the procedures listed in step 7

above to change the boot device.

5.2.3 Reconfiguring FSP Post VIOS InstallationDuring VIOS installation to provide control over the boot order, configuration of FSP occurred to boot to SMS. With VIOS installed, use one of the ASM interfaces to reconfigure FSP to boot to VIOS and not stop at SMS. The following covers the configuration via the ASM GUI (use of the ASM CLI would require restarting the system);1. Click Power On/Off System2. For AIX/Linux partition mode boot, select Continue to operation

system3. Click Save settings (see below)4. At this point, if the system reboots, VIOS will automatically start.

5.2.4 Configuring VIOSAfter configuring VIOS, continue to use the same PuTTY terminal session to configure the date and TCP/IP settings for accessing IVM from a browser.1. Sign onto VIOS for the first time, accept terms:

a. When prompted, login in with using the “padmin” IDb. Enter a new passwordc. Press Enter to accept the VIOS maintenance terms and

conditionsd. Enter license –accept to accept the license terms

2. Use cfgassist command to change the date/time and configure TCP/IP VIOS.

3. Optionally configure TCP/IP for VIOS from the command line as followsa. Enter lstcpip –adapters to display a list of Ethernet adaptersb. If you are unsure of which physical adapter the interface you want to use refers to, enter the following command

to get the physical location code: lsdev -dev entx -field name description physloc status -fmt ,

where entx is a particular interface. Notice the space before the comma.

The output will look similar to the screenshot above. The physical location code in this example is:U78AB.001.WZSGRE5-P1-C7-T1

The pertinent information is “P1C7T1”, which refers to a physical slot (P1), the slot number C7 (which can be located by finding the numbered slots on the rear of the physical frame), and port number 1 (T1).

c. Configure TCP/IP using the mktcpip command by providing IP address information and the interface, enx for the entx adapter (see screenshot to right).

d. Examplemktcpip -hostname powerlinuxivm -inetaddr 9.5.110.171 -interface en0 -start -netmask 255.255.255.0 -gateway 9.5.110.1 -nsrvaddr 9.10.244.100 -nsrvdomain rchland.ibm.com

e. Enter lstcpip -stored to verify the configuration f. Enter lstcpip –routtable to verify routing

5.2.5 Accessing IBM Virtualization Manger (IVM)

With VIOS installed, visit the IVM web interface from a web browser:1. Enter a URL containing the VIOS TCP/IP address configured above, i.e.

(http://9.5.110...).2. Login using the User ID of “padmin” and the password configured while in

stalling VIOS.

5.3 Creating Virtual ServersAfter installing the Virtual I/O Server (VIOS) and accessing the Integrated Virtualization Manager (IVM) through its web interface, configure virtual servers for running Linux and the OSIS applications.

The following sections cover additional general VIOS configurations and how to configure a virtual server.

5.3.1 Preparing VIOS for Virtual ServersThe following sections cover several VIOS configurations to consider before creating virtual servers.

5.3.1.1 Optional: Updating the VIOS NameTo change the system name:1. Click View/Modify System Properties2. Click on the General tab3. After System name, type the new name4. Click OK to save the change

5.3.1.2 Mirroring VIOS StorageMirroring the hard disk containing VIOS provides redundancy in case of disk failure or maintenance. VIOS is installed on hdisk0 by default, which is included in the rootvg storage pool. To mirror the VIOS hdisk:1. Add a second disk to the same storage pool containing the disk with VIOS:

a. Click on View/Modify Virtual Storageb. Click on the Physical Volumes tabc. Add a check mark in the box next to the second disk

(For example, to add hdisk1 as a second disk, click the box next to hdisk1.)d. From the More Tasks drop down list, select the Add to storage pool taske. For the Storage pool drop down, select the rootvg storage poolf. Click OK

(The second hdisk will be added to the rootvg storage pool.)

2. Enable the second disk as a mirror of the first: a. Open a terminal window:

i. Click on View/Modify Partitionsii. Check the box for the VIOS partition

iii. From the More Tasks drop down list, select Open ter-minal window

NOTE: If the terminal window does not open or closes immediately (the browser applet does not work), then open a Secure Shell (SSH) terminal session to the VIOS, using the IP address for VIOS configured earlier. (See References for SSH tools.)

b. From the terminal window, enter the following command to mirror the VIOS disk and force a reboot:

mirrorios –f -defer hdisk1

c. If “-f” option is not used, enter y to continue and reboot the systemNOTE: If the VIOS does not automatically restart, review the power settings as seen in 5.2.2 step 7 (normal boot mode) and 5.2.3 (boot to OS).NOTE: The option “-defer” instructs the command not to reboot the VIOS. VIOS 1.5 or later does not require a reboot after the mirrorios command, so you should use the -defer option for these levels.

5.3.1.3 Optional: Updating VIOS PropertiesTo update the VIOS partition properties (name, memory, processor, and network) using IVM:1. Click on View/Modify Partitions in the left navigation area2.3. Click on the VIOS partition's name link to display its properties 4. Update the VIOS partition's name (which by default is based on the serial

number of the system):a. Click on the General tabb. For Partition name, enter a new name

5. Update Assigned memory:a. Click on the Memory tabb. Enter the desired Assigned memory in the field for Assigned memory

(VIOS 2.2 uses about 4 GB of memory when running IVM)c. Increase or decrease Minimum memory and Maximum memory to ac

commodate. Assigned memory must be greater than Minimum memo-ry and less than Maximum memory.

d. Leave the default values for Processing Units and Virtual Processors, but monitor utilization.

6. To provide additional security, update the Ethernet ports used by VIOS to use a dedicated port instead of a port shared with other virtual servers:a. Click on the Ethernet tab.b. Check the first two ports and uncheck the last

two ports. The last two ports are configured later as shared ports for use by the virtual servers running Linux.

7. Click OK when done to update the VIOS properties

5.3.1.4 Creating a Storage Pool for Virtual ServersVirtual servers running Linux should not use the same system physical disk drive used by VIOS. Therefore, virtual servers running Linux require a separate storage pool containing their disks.

Create a new storage pool with the remaining internal disks:1. Click on View/Modify Virtual Storage 2. Click on the Storage Pools tab3. Click on Create Storage Pool 4. Enter a name (1 to 15 characters, no spaces)5. Leave Logical volume based for the type6. Select the option to have this storage pool to be used as the default7. Select the remaining hard disks8. Click OK to create the storage pool with the selected disks

5.3.1.5 Creating a Virtual Ethernet Bridge for Virtual ServersThere are 2 ways to give network connectivity to Virtual Servers:

1. Dedicated Networking: Native adapter ownership, which consists of assigning an entire adapter to a virtual server.2. Shared Networking: Virtual Ethernet bridging using a Shared Ethernet Adapter

For this paper, we have more virtual servers than physical adapters, so a Shared Ethernet Adapter (SEA) needs to be configured. Create a SEA using an Ethernet adapter managed by VIOS as follows:1. Decide upon a virtual networking layout

a. A virtual network bridge will be associated with a particular virtual network by VLAN ID.b. Virtual network adapters need to have the same VLAN ID in order to communicate with each other.c. The Integrated Virtualization Manager defines 4 virtual Ethernet adapters in the VIOS partition by default, with

VLAN IDs of 1, 2, 3, and 4 respectively.d. Pick an available VLAN ID, or create a new virtual adapter

on the VIOS with a new VLAN ID.

2. Define the Virtual Ethernet Bridge:a. Click on View/Modify Virtual Ethernet in the left navigation

areab. Click on the Virtual Ethernet Bridge tabc. For Virtual Ethernet ID 1, select the same physical adapter

from the list to use as the backing device. You can determine which interface is which by looking at the location codes provided in the list.

d. Note that selecting two different physical adapters results in two SEAs being created, not one with two ports)

e. Click on Apply

3. Verify that the SEA was created:a. Click on View/Modify TCP/IP Settings in the left navigation areab. Click on the 'Interfaces' tabc. Look for a new enx interface with the description:

Shared Ethernet Adapter - Standard Ethernet Network Interface

4. Optionally, you can verify that the SEA was created on the command VIOS line:a. Use PuTTY to connect to VIOSb. Enter lsmap –all –net to view the virtual network mappings

The example on the right shows ent2 as an SEA using ent0 using the virtual adapter ent0 and the physical location

NOTE: An interface thats in use by the VIOS cannot be designated as the backing device in an Shared Ethernet Adapter. We originally configured and interface for use by the VIOS. To convert this to an SEA, the configured interface must first be unconfigured using the serial terminal and the VIOS command, 'rmtcpip enx', for interface enx. You can then proceed to create the SEA as above using that interface as the backing device, followed by a reconfigure of the networking on the VIOS (see section 5.2.4) using the SEA adapter interface.

5.3.1.6 Creating a Virtual Optical Library for Linux Installation MediaA virtual optical library allows ISO files to be stored on the server, for later use by the virtual servers. For example, the ISO files for the Linux distribution and the IBM Installation Toolkit can be stored in the library and used at any time for the installation and configuration of Linux.

Create a virtual medial library:1. Click on View/Modify Virtual Storage in the left navigation area 2. Click on the Optical/Tape tab3. Click on Create Library

a. For Storage pool name, select the second storage pool created earlier, not the rootvg storage pool which is recommend for VIOS use only

b. For Media library size, 8 GB is sufficient for both the Linux distribution and the IBM Installation Toolkit. If necessary, later extend the library to a larger size.

4. Add the ISO files:a. Click Add Media

NOTE: When installing Linux using the IBM Installation Toolkit, first add the Linux ISO file to the library first, and then add the IBM Installation Toolkit ISO file. This avoids a problem later where the Installation Toolkit cannot find the Linux media during the install.b. To add media using ISO files transferred using a web browser:

i. Select Upload mediaii. For Media type, select Read only

iii. For Optical media file to upload, click Browse and select the ISO file from the workstation directory

iv. Click OK to copy the file into the library (this may take some time)NOTE: Because of a 2 GB limitation on the media file size for media uploaded using a web browser, select from one of the other options listed below for the Linux distribution ISO file that is >2 GB.

c. To add media using FTP:i. Sign into VIOS using padmin

ii. Use FTP to transfer the ISO images to a local VIOS directoryiii. To copy the ISO file into the optical library, either:

i. From the VIOS command line:mkvopt <filename> -file <source filename> -roor,

ii. From the IVM Virtual Optical Media panel:1. Select Add existing file2. For Media type, select Read only3. Specify the name of the ISO file, for example: /home/padmin/RHEL6.0.iso4. Click OKNOTE: The name of the ISO must be all alphanumeric and not longer than 30 characters. Rename the ISO file accordingly.

d. To add media using the system’s DVD drivei. Select Import from physical optical device

ii. For Media type, select Read onlyiii. For Media name, enter cd0iv. Select the media device from the table shownv. Click OK (this may take some time)

5.3.2 Creating Virtual Disks for Virtual Server StorageVirtual Servers require a virtual disk. For OSIS there are several considerations:

1. Virtual disks associated with internal storage2. Virtual disks associated with external storage3. Use of Linux software RAID support (redundancy and performance)

The following sections discuss configuration of virtual disks for internal storage, including Linux software RAID. External storage is dependent on the type of external storage being used and is not covered in this paper.

5.3.2.1 Creating Virtual Disks for use with Linux Software RAIDWhen using Linux software RAID support for a virtual server, create multiple virtual disks from different physical disks. Creating multiple virtual disks on the same physical disk causes performance degradation in a RAID configuration. Linux software RAID support involves the following operational steps:

Operation Comment User InterfaceCreating multiple virtual disks Each virtual disk should be created from a different physical disk for performance and redundan

cy. Virtualization Managers (2) do not support this.VIOS CLI(1,4)

Assigning the virtual disks to a virtual server

Details are provided in section 5.3 Creating Virtual Servers on page 15 Virtualization Managers(2)

VIOS CLI(1)

Creating Linux Software RAID device

Details are provided in section 5.3.6 Configuring a Linux Software RAID Device on page 32 Linux CLI(3)

Table notes: 1. VIOS CLI is the Virtual I/O Server command line interface accessible from a remote terminal such as PuTTY.2. Virtualization Mangers include

a. Integrated Virtualization Manager (IVM)b. Hardware Management Console (HMC)c. Systems Director Management Console (SDMC). For information on SDMC, see IBM Systems Director Management

Console: Introduction and Overview.3. Linux CLI is a Linux command line interface accessible from a remote terminal such as PuTTY.

5.3.2.2 Creating Virtual Disks using the VIOS Command LineSince the available virtualization managers do not support specifying physical disks when creating a virtual disk, we can use the VIOS command line to create virtual disks when preparing for a software RAID configuration. Use the following VIOS command to create a virtual disk on a specific hard disk:

mklv -lv <logical volume / virtual disk name> <volume group / Storage Pool name> <size in GB>G <hdisk to use> Example: mklv -lv mail2.2 Clients 30G hdisk2

o mail2.2 is the virtual disk name

o Clients is the existing Storage Pool to use

o 30G is the size of the virtual disk (G is used instead of GB)

o hdisk2 is the existing physical disk drive to create the virtual disk (file) on

Continue creating as many virtual disks in the RAID as needed, limited by the number of physical disks available.

5.3.2.3 Creating Virtual Disks Using IVMVirtual disks can be created during the creation of a virtual server or independently.

To use IVM to create a virtual disk independent of the virtual server:1. Click on View/Modify Virtual Storage in the left navigation area2. Click on the Virtual Disks tab 3. Click on Create Virtual Disk

4. Update the Create Virtual Disks page as follows:a. For Virtual disk name, consider a name that associates this disk with the corresponding virtual server

b. For Storage pool name, use the storage pool that was created for the Linux virtual servers, not the rootvg storage pool used by VIOS

c. For Virtual disk size, at least 5 GB is required to install Linux, but a larger size is likely required for application data requirements.

d. For Assigned partition, select a virtual server from the list or None when the assignment is to be performed later

e. Click OK to create the virtual disk5. Additional virtual disks can be created now or later

5.3.3 Creating Virtual ServersVirtual servers are created and managed by a virtualization manager (IVM, HMC). This section covers an example of creating a virtual server of use with Linux and OSIS.

Since a typical OSIS implementation would have multiple virtual servers for the various workloads of web, mail, file, print and networking, it is initially helpful collect information into a reference table. Below is an example used while developing the OSIS solution guides.

The following is an example of creating a virtual server using IVM:1. To launch the wizard for creating a partition for the virtual server

a. Click on View/Modify Partition in the left navigation areab. Click on Create Partition

2. On the Create Partition: Name panela. For Partition ID, optionally change the ID (not very significant

or used very often)b. Partition Name is the name of the virtual server and should be

unique and meaningfulc. For Environment, select AIX or Linux from the drop down listd. Click on Next to continue with the wizard

3. On the Create Partition: Memory panela. For Assigned Memory, specify at least 2 GB initially, this val

ue can be reconfigured later b. Click on Next to continue with the wizard

Name Description Parti-tion ID

Memory Processors (physical and virtual)

Networking Storage Virtual Optical Media

Mail Runs the primary mail server

(default) 2 to 3 GB 1 to 4 pCPU1 to 4 vCPU

Shared Ethernet 3 Virtual Disks (MailDisk13)

Linux ISOToolkit ISO

MailWL Runs mail workload generation utilities

(default) 1 GB 0.5 pCPU1 vCPU

Dedicated Ethernet

1 Virtual Disk (MailWLDisk)

Linux ISOToolkit ISO

Table : Virtual Server Reference Table2

4. On the Create Partition Processors panela. For Assigned processors, select at least 1 from the list to indicate

the number of virtual processors to be used. b. For Processing Mode, select Shared to allowing dynamic sharing of

processors. Shared vs. Dedicated mode can not be dynamically changed, however a virtual server’s processor sharing priority can be configured later, including no sharing, which is effectively the same as Dedicated.

c. Click on Next to continue with the wizardNOTE: Reconfigured of these values can be done later.

5. On the Create Partition: Ethernet panela. For Ethernet networking there are several possible configurations depending on whether sharing of Ethernet ports

between the virtual servers. Below are two example configurations used for the OSIS Solution Guides b. For shared networking managed by VIOS, use the Shared Ethernet

Adapter (SEA ) as follows (see example to right)i. For Virtual Ethernet Adapters, select the SEA adapter shown

in the drop down listii.

c. For dedicated networking, assign an available Ethernet Adapter natively to the partition. (example not shown) This is done after the partition has been created by going to the View/Modify Physical Adapters window and changing the partition assignment of the desired adapter.i. For Virtual Ethernet Adapters, select None for all of the

adaptersd. Click on Next to continue with the wizard

6. On the Create Partition: Storage Type panela. For Storage Type, select either Create virtual disk or Assign ex-

isting virtual disks and physical volumes (if previously created).NOTE: Creating a virtual disk now is similar the process covered in 5.3.2.3 Creating Virtual Disk on page 20.NOTE: Using a physical volume dedicates a disk for using by only one virtual server, limiting storage options for other virtual servers.NOTE: Virtual servers can have more than one virtual disk assigned. For example when planning for a Linux software RAID configuration

b. Click on Next to continue with the wizard

7. On the Create Partition: Optical/Tape panela. Click on Modify to show the contents of the media libraryb. Select the IBM Installation Toolkit media from the listc. Click on OK to save this device definition

d. Repeat the steps above to create a 2nd device for the Linux distribution mediae. Click on Next to continue with the wizard

8. Click on Next to view a summary9. To make changes, click links in the left navigation area10. Click on Finish to create the partition definition for the virtual server

5.3.4 Preparing a Virtual Server for ActivationAfter creating a partition for a virtual server using the IVM Partition Creation wizard, consider the following changes to the configuration before installing Linux:

1. Click View/Modify Partitions in the left navigation area2. Click the name of the partition in the list for the virtual server’s partition to modify3. On the General tab:

a. For Boot mode, consider setting to Systems Management Services (SMS) to stop the initial boot at the SMS menu, which is necessary (later) to change the boot device to a virtual optical device associated with the Linux media.

4. On the Memory tab:a. For Minimum memory, set the minimal amount of memory required to start the virtual server. Determining this

value is dependent on the Linux applications used. Recommendations:i. Initially set this to 2 GB while installing Linux and the applications.

ii. After configuring the virtual server, use the nmon utility to monitor for required memory and reset this value accordingly. NOTE: changing this value will require a restart of the virtual server

b. For Assigned memory, set to the amount of memory the virtual server should run with.c. For Maximum memory, set the value to match the upper memory range for the virtual server. An administrator

can then later change the virtual server’s Assigned memory up to this Maximum value. NOTE: changing this value will require a restart of the virtual server

d. The example to the right provides a virtual server configuration with (pending) changes for:

i. At least 0.5 GB of memory during startup ii. Up to 1 GB of memory dynamically while running if needed

and availableiii. Up to 2 GB of memory that an administrator can later assign to

the virtual server. The assigned value can be set to a value of from 0.5 GB to 2 GB. The virtual server memory will then grow to this new value if memory is needed and available (from the virtual server shared memory pool).

iv. Note that pending changes will occur when the virtual server’s partition is started, or dynamically if the virtual server is running with a Linux configured for dynamic LPAR operations (DLPAR), which can be determined using Retrieve Capabilities button on the partition properties General tab

5. On the Processor tab:a. Processing Units relates to the number of active physical processors in

the PowerLinux system to be assigned to the virtual server, specified in units of 1/10. Examples are 1.0 and 1.1.

i. For Minimum, set the minimal amount of physical CPU (pCPU) required to start the virtual server. Determining this value is dependent on the Linux applications used. Recommendations:

1. Initially set this to 1 while installing Linux and the applications.

2. After the virtual server is configured, use the nmon or mpstat utilities to monitor for required pCPU to start the virtual server and reset this value accordingly. NOTE: changing this value will require a restart of the virtual server

ii. For Assigned, set to the amount of pCPU the virtual server should run with. This value can be changed dynamically while the virtual server is running.

iii. For Maximum, set the value to match the upper pCPU range for the virtual server. Later, an administrator can change the virtual server’s Assigned pCPU up to this Maximum value. NOTE: changing this value will require a restart of the virtual server

b. Virtual Processors relates to the processors as seen by Linux, assigned in whole units.

NOTE: By default on POWER7, Simultaneous Multithreading (SMT) of 4 threads causes Linux to see 4x CPUs as specified by the vCPU value. For example, running a virtual server with 1 vCPU causes Linux to perceive there are really 4 CPUs. NOTE: Increasing the number of vCPUs (and thus threads) may in some cases help, depending on the number of applications and their ability to utilize multiple CPUs effectively. NOTE: When generating workloads for performance testing, as the number of vCPUs increases, it becomes more difficult to drive workloads across all vCPUs,NOTE: Creating a high number of vCPUs with a low number of pCPUs can create unnecessary overhead within PowerVM.

i. For Minimum, start with a value of 1 for the initial Linux install and configuration of the workload. Later, this value can be changed to align with changes to the Minimum pCPU.

ii. For Assigned, specify a value of at least one. Refer to the note about not setting this value too high. This value can be changed while the virtual server is running, up to the value specified for Maximum. Consider changing this value if the Assigned pCPU value changes.

iii. For Maximum, specify a value for the highest number of vCPUs expected.iv. The example to the right provides a virtual server configura

tion with (pending) changes for:1. At least 1 pCPU and 1 vCPU during startup 2. Up to 2 pCPUs and 2 vCPUs dynamically

allocated while running if needed and available

3. Up to 4 pCPUs and 4 vCPUs that an administrator can later assign to the virtual server. The assigned vCPU and pCPU values can each be set to a value of from 1 to 4. The CPU values grow to this new value if CPU resources are needed and available (from the virtual server shared processor pool).

v. Note that pending changes will occur when the virtual server’s partition is started, or dynamically if the virtual server is running with a Linux configured for dynamic LPAR operations (DLPAR), which can be determined using Retrieve Capabilities button on the partition properties General tab.

vi. When sharing CPU resources between virtual servers (that is running uncapped), the pCPUs assigned to a virtual server can dynamically increase to the value specified for the Assigned vCPU. Generally this is desired, however to prevent dynamic changes of the pCPU, an administrator can change the priority to None – Uncapped while the server is running. This was necessary during VIOS Solution Guide testing to measure running a virtual server with 0.1 pCPU (by default, the pCPU dynamically would changed to match the vCPU of 1.0).

NOTE: When modifying both CPU and memory settings of a virtual server that is running, perform this as two separate operations instead of one.

NOTE: When modifying memory settings of a virtual server that is running, consider 4 GB or less changes to allow for dynamic memory adjustments with reduced paging by Linux

NOTE: When monitoring CPU, utilities such as vmstat will report CPU cycles shared with other virtual servers as “stolen”. Refer to Measuring stolen CPU cycles for more details. Run “mpstat P ALL” to show CPU utilization for all logical CPUs.

5.3.5 Installing Linux on a Virtual Server’s PartitionOnce a virtual server’s partition is defined with the system resources to be used, it is ready for activation and installation of Linux. The following scenario covers an example of starting the partition and installing Linux using IBM Installation Toolkit v5.1 to install Linux from Linux distribution media.

5.3.5.1 Activating the PartitionSince Linux is being installed, the partition requires booting using the IBM Installation Toolkit media that was configured in the Virtual Optical Library and assigned to the partition in section 5.3.1.6. Temporarily change the partition’s boot process to boot from the corresponding virtual optical device as follows:1. To access SMS before the partition is activated establish a terminal

window. The following are two methods for creating a terminal session:a. IVM supports opening a terminal window for a partition:

i. Click on View/Modify Partitions in the left navigation areaii. Select the check box for the partition to activate

iii. From the More Tasks drop down list select Open terminal windowNOTE: There is a known problem with some browser configurations causing a terminal window to open and immediately close. If this occurs, use the second method below for establishing a terminal window.

b. A terminal window to a partition can also be opened from the VIOS command line as follows:

i. Using PuTTY, open a terminal window to the VIOS command line (covered in previous sections of this paper)

ii. Enter the following VIOS command:mkvt –id <partition ID>NOTE: <partition ID> is found in the second column of the IVM list found in View/Modify Partitions panel.NOTE: If an error indicates a terminal session is already open, enter the following VIOS command to remove the existing session (and try again): rmvt –id <partition ID>.

iii. Authenticate with the VIOS ID / password (padmin / padmin by default)2. With a terminal window open to the partition, activate the partition as follows:

a. From the IVM View/Modify Partitions panel, select the partition b. Click on Activatec. Click on OK to confirm that activation

3. Monitor the partition’s boot process from the terminal window and watch for the SMS menu to display.

4. Once the SMS menu displays press 1 to enter start using SMS. a. Note that if the partition was previously defined to stop at the SMS menu, this is not necessary

5. From the SMS menu, to change the boot device to the virtual optical device associated with the IBM Installation Toolkit, follow these steps (similar to the process used earlier when installing VIOS):a. From Main Menu, press 5 for Select Boot Optionsb. From Multiboot, press 1 for Select Install/Boot De-

vicec. From Select Device Type menu, press 7 for List All

Devicesd. From the List All Devices menu, select the SCSI CD

ROM device associated with the virtual device for the

IBM Installation Toolkit. The virtual device’s location ID corresponds to the virtual media assignment order for the partition. For example, the ID containing L82 is the first media assigned and L83 is the second media assigned.NOTE: Based on the assignment of the IBM Installation Toolkit to the second optical device earlier in IVM, boot from device 2 associated with LUN 83.

e. Select 2 for Normal Boot Modef. Select 1 to exit SMS

NOTE: At this point, the system will begin the installation process for the partition. Worth noting here is that IBM PowerLinux systems will not permit running AIX or IBM i. When you try to deploy using VMControl an AIX or IBM i virtual appliance to an IBM PowerLinux server , the task completes with errors. The following error message appears deploy task job log:DNZIMN882E The deploy task is not progressing and has timed out.

5.3.5.2 Installing Linux using IBM Installation ToolkitAs mentioned earlier, use of the IBM Installation Toolkit to install Linux is important since it configures IBM’s value add software. This includes1. Performing dynamic resource operations on virtual servers, such as adding and removing both CPU and memory

resources.2. For VIOS, provides installation and configuration of web (LAMP), mail, print, file and networking services.

From a terminal window of a partition booting from the IBM Installation Toolkit v5.1, use the following steps to install the toolkit and Linux.1. When prompted, select the install language and keymap to use. English is the default and used when not provid

ing a response.2. IBM Installation Toolkit v5.1 asks for a networking configuration up front. This is necessary to check for certain package updates.3. Select 1 for Wizard mode to start the toolkit wizard’s for install Linux. While in the toolkit wizard, use the tab

key to navigate to menu options displayed in red.NOTE: Some menu lines may not be display until enlarging the terminal window and/or navigating to the nondisplayed line. That is, to display some options, keep pressing the tab key.

4. Select Accept (at the last line of the menu) to accept the license. As mentioned above, use the tab key to navigate to the menu options.

5. Select Install to start the install process 6. Select the correct version of Linux to be installed as follows:

a. Tab to Linux distribution b. Press enter to display the selection windowc. Use up/down arrow keys to highlight the correct selection d. Press Enter to select the value

7. Select the type of Linux install to perform. For OSIS, consider selecting Full since various open source packages are used.a. Tab to Installation profile b. Press Enter to display the field selection window c. Press downarrow to highlight your selectiond. Press Enter to select the value

8. Optionally, select an automatic or manual disk partitioning schemea. Tab to Installation profile b. Press Enter to display the field selection window c. Use the arrow keys to highlight your selectiond. Press Enter to select the value

9. With the Installation settings complete:a. tab to Next b. hit Enter to proceed with the wizard

10. Select the open source workload(s) to configure during the installationa. Tab to one or more workloadsb. Press Enter to select the workloadc. Select Next when selections are completed

11. Verify the installation sources.a. Distro source and IBMIT source should show:

[CD/DVDROM]

NOTE: Optionally, you can enter a valid URL to provide access to installation mediab. Select Next when selections are completed

12. To configure networkinga. tab to each of the fields and hit enter to change their value at the

bottom of the window, enter the new valueb. select Configure to change the virtual server’s IP address infor

mationc. enter a DNS server to allow IBMIT to look for package updates

13. Update the networking informationa. tab to each field to be updatedb. at the bottom of window, enter the new value and press Enterc. when done, select Save to save the network IP information

14. Select Next to complete the network configuration and continue with the wizard.

15. Update the Linux settings for keyboard, mouse, password, etc.16. Select additional IBM Repositories to enable for the Linux installa

tionThe ibmpowerrepo repository contains IBM valueadd productivity tools that enable dynamic partitioning among other features. The repository will allow the installation to easily keep these tools up to date.

17. Select packages to installa. all base required packages are selected by defaultb. scroll down to select additional packages if required c. select Next to continue (need to scroll down to see this

option)NOTE: for RHEL 6, do not install the powerutils package since installation of the correct version occurs automatically from the RHEL 6 distribution media

18. Accept the license and click Next to continue19. Review the summary

a. Tab to menu options are at bottom of the displayb. select Previous to make changesc. select Next to continue with the wizard

20. The IBM Installation Toolkit continues with the install of LinuxNOTE: if prompted for the Linux install media (could not find a 2nd virtual device with the Linux distribution), use IVM to change the partition's virtual media library to use the Linux .iso instead of the IBM Installation toolkit as describe in section 5.3.1.6.a. when ready, select Next in the terminal window to contin

ue the IBM Installation Toolkit wizardb. the IBM Installation Toolkit automatically reboots the par

titionc. press enter when the boot prompt displays to continue the

Linux installd. installation of the Linux packages then occurs next

21. For Red Hat RHEL installations, a menu provides final configuration options. Consider,a. in a testing environment with a private network, initially

disabling the firewall until applications are installed and configured

b. verify the network IP and DNS settings

22. Consider using a PuTTY terminal window instead of the VIOS or IVM terminal windows for functionality.

23. Finally, sign into the Linux console.

24. If you did not select the ibmpowerrepo to be installed, you can install the latest IBM Service and Productivity tools for Linux on Power manually. For example, to configure the repository and install the tools for a virtual server running RHEL6

wget ftp://public.dhe.ibm.com/software/server/POWER/Linux/yum/download/ibmpowerrepo1.1.65.ppc.rpm

rpm ivh ibmpowerrepo1.1.65.ppc.rpm yum install ibmpowermanagedrhel6

NOTE: Requires firewall access to public.dhe.ibm.com (9.17.248 .112) and linuxpatch.ncsa.uiuc.edu (141.142.192.67)NOTE: Requires repo for Linux distribution to be available for prerequisite softwareNOTE: optionally install the Advanced Toolchain compilers and optimized system libraries using yum install advancetoolchainat5*

25. If the partition’s Boot mode was configured to stop at SMS, from the IVM GUI change the boot to NOT stop at SMS as followsa. open the partition’s properties panelb. on the General table, change Boot mode to Normal c. click Ok to save the changes

5.3.6 Configuring a Linux Software RAID DeviceRAID configurations can help address VIOS workloads that are disk intensive. Hardware RAID is generally preferred over software RAID to provide better performance, but hardware RAID requires additional planning since before VIOS is installed RAID configuration occurs.

To configure Linux software RAID using virtual disks, consider the following:1. If not previously done, create additional virtual disks assigned to independent physical disks in the storage pool.

For details, refer to section 5.3.2.1 Creating Virtual Disks for use with Linux Software RAID on page 20.2. Activate the virtual server’s partition with all of the virtual disks assigned to it.3. If multiple virtual disks are used during the initial Linux installation

a. The IBM Installation Toolkit will only allow an install on one disk (no software RAID support)b. Linux automatically creates a RAID device, which can be displayed using the following Linux com

mand:i. mdadm detail –scan

ii. the RAID device name should be: /dev/md/0_0c. Additional virtual disks can be added to the RAID device using the following Linux command:

i. msadd add /dev/md/0_0 /dev/sdd4. Install does not create a RAID device using only one virtual disk. To create the RAID device for a nonLVM con

figuration, use the mdadm Linux command. For example, a. mdadm create v /dev/md0 l raid1 n 3 /dev/sdb /dev/sdc /dev/sddwhere /dev/md0 is the RAID device

nameraid1 indicates a RAID1 device–n 3 indicates there are three devices to include in the RAID, which are then listed

NOTE: To create a RAID using Logical Volume Manager (LVM), refer to Basic Linux LVM Striping.5. Create a file system on the RAID device, for example:

a. mkfs.ext3 /dev/md06. Create a RAID configuration file as follows:

a. mdadm detail scan > /etc/mdadm.conf7. Create a directory for mounting the RAID device

a. mkdir <raid directory>8. Configure a mount of the RAID device at boot time

a. Add the following line to /etc/fstabi. /dev/md0 /<raid directory> ext3 defaults 1 2

9. Mount the RAID device nowa. mount –a

10. To configure Postfix to use the RAID based file system for mail filesa. Use a symbolic link to link the default mail directory to the directory for the RAID based file system. b. For example to create the symbolic link and assign the mail group to the RAID directory.

i. ln s /raid1/spool/mail /var/spool/mailii. chown :mail /raid1/spool/mail

5.4 Configuring OSIS WorkloadsIf web, mail, file, print, and networking applications were installed earlier using the IBM Installation Toolkit (covered in the section 5.3.5.2 on page 28), a quick configuration of these workloads can be performed using the IBM Installation Toolkit Simplified Setup Tool as follows:

1. Use IVM or a terminal window to ensure activation of the virtual server associated with the installed workloads and Linux is running

2. Open a browser to https://<virtual server IP address>:6060/ to visit the IBM Installation Toolkit Simplified Setup Tool web page

3. Log on using the Linux root ID and password (set earlier during Linux installation)4. Click OK

NOTE: If the virtual server is configured with less than 16GB of memory, an Unsupported System warning may be displayed.

5. Click Yes to continue6. Click Accept to accept the license7. If the virtual server has Internet access and updates are avail

able, install the updates8. Click OK to continue9. A list of workloads installed earlier by the IBM Installation

Toolkit is displayedNOTE: Use the IBM Installation Toolkit later to install workloads after installing Linux. Refer to the documentation available on the IBM Installation Toolkit website.NOTE: At the time of this writing, using the IBM Installation Toolkit version 5.1 to perform a “full” install of RHEL 6.2 produces errors when configuring some workloads with the Simplified Setup Tool. A workaround is to remove the “augeaslibs” package:rpm e augeaslibs

10. After clicking on a workload to configure, update the configuration panels and complete the wizard. The IBM Installation Toolkit Simplified Setup Tool updates the required workload configuration files, and provides information regarding the changes.

5.4.1 Configuring webmin for Administrationwebmin provides an easy to use GUI for administering Linux and controlling many services, including MySQL, web, and mail.

Download the webmin RPM from the webmin site downloads page and install: yum install webmin* nogpgcheck disablerepo=*

5.4.2 Configuration of Mail Application ServerThis section covers the configuration changes used for setting up the Postfix and Dovecot mail applications.

Use the IBM Installation Toolkit Simplified Setup Tool for the initial configuration as follows1. For Postfix configuration (refer to the example on the right)

a. Use recommended values in parenthesis for most valuesb. Use a fully qualified hostname to help with routing of mailc. For My Origin, use the virtual server’s hostname for better

identification of the senderd. Use default values for directories to make Dovecot configu

ration easier. In the case of using Linux software RAID, use a symbolic link to redirect mail to alternative directory on the RAID device (covered in the RAID configuration section of this paper)NOTE: IBM Installation Toolk Simple Setup Tool updates /etc/postfix/main.cf

2. For Dovecot configuration (refer to the example on the right) a. For Protocol, the configuration tool supports only imap.b. The following Mail location value provides indexing in memory

for better performance: mbox:~/mail:INBOX=/var/mail/%u:INDEX=MEMORY

NOTE: Setup tool updates /etc/dovecot/dovecot.conf

3. For SUSE, the Cyrus Configuration changes are:a. In /etc/imapd.conf, add the following:

configdirectory: /var/lib/imappartitiondefault: /var/spool/imapsievedir: /var/lib/sieveadmins: cyrus rootallowanonymouslogin: noautocreatequota: 10000reject8bit: noquotawarn: 90timeout: 30poptimeout: 10dracinterval: 0drachost: localhostsasl_pwcheck_method: saslauthdlmtp_overquota_perm_failure: nolmtp_downcase_rcpt: yesallowplaintext: yessasl_mech_list: PLAIN LOGINauth_mech: unixservername: node1

b. In /etc/sysconfig/mail set:SMTPD_LISTEN_REMOTE="yes"

c. In /etc/postfix/main.cf change mailbox_transport to cyrus to enable it:mailbox_transport = cyrus

d. After changing configuration files, restart the cyrus and saslauthd:service cyrus restartservice saslauthd restart

4. After the IBM Installation Toolkit Simplified Setup Tool updates the configuration files, the following changes were also made:a. For Postfix

i. In /etc/postfix/main.cf , uncomment the following lines: header_checks = regexp:/etc/postfix/header_checks default_privs = nobody

ii. Added these lines: disable_dns_lookups = yes smtp_host_lookup = native smtp_bind_address = 0.0.0.0

NOTE: When using procmail for antispam filtering, add the following line:mailbox_command = /usr/bin/procmail

iii. After changing configuration files, restart the Postfix: service postfix restart

b. For Dovecoti. In /etc/dovecot/conf.d/authdeny.conf.ext, change yes to no in the following line:

deny = noii. In /etc/dovecot/conf.d/10auth.conf, uncomment the following line

disable_plaintext_auth = noiii. Dovecot configuration can be verified using the following:

dovecot –niv. After changing configuration files, restart the Dovecot as follows

service dovecot restart

5.4.3 Configuring Postfix Anti-spamSpamAssassin provides Postfix with antispam mail protection. Using procmail, Postfix passes emails to procmail, which invokes SpamAssassin.

Configuration of Procmail is made easy using webmin as follows:1. Visit webmin using this URL: https://<virtual server IP

address>:10000NOTE: If you cannot connect to webmin then follow these instructions http://www.webmin.com/firewall.html

2. Login in with the Linux root ID and password3. Click Servers4. Click Procmail Mail Filter5. Add an action to invoke SpamAssassin as follows:

a. Click Add actionb. For Delivery mode, select Feed to programc. Set the program /usr/bin/spamassassind. Check Wait for action program to finish, and check resulte. Check Action program is a filterf. Click Save to save this action

6. Add an action to handle the result, for example to append the spam mail to default mail file (refer to screenshot below)

7. When done, two actions should in the list

5.4.4 Configure Sending Mail for Performance TestingTo enable testing of the throughput of the mail application server for the OSIS solution guides, use the open source utility smtpsource to send email to the server.

To simulate multiple mail clients sending emails: smtpsource was run on a separate virtual server specifically to generate a mail workload of sending receiving

emails smtpsource was run multiple times in the background to simulate multiple users sending mail at the same

time. For example, the following command sends in the background 5000 5k emails over 100 sessions from user root to user mailtest1 on mail server PowerLinuxMail.example.com:25

o smtpsource s100 l 5120 m 5000 c f root@PowerLinuxMailW.example.com t mailtest1@PowerLinuxMail.rchland.ibm.com PowerLinuxMail.example.com:25 &

5.4.5 Configure Retrieving Mail for Performance TestingTo enable testing of the throughput of the mail server for delivery of email, open source utility fetchmail retrieved email.

To simulate multiple mail clients receiving emails: Fetchmail was run on a separate virtual server specifically to generate a mail workload of receiving emails Received email from multiple user mailboxes to ensure a workload across all CPUs. For example, 16 mail

users were used to receive emails when running with 4 CPUs, utilizing all 16 logical CPUs. fetchmail was run as a daemon to receive emails in the background for each of these mail users

Configured fetchmail as follows:1. fetchmail is provided with RHEL distribution and installed with the following Linux yum command:

yum install fetchmail2. Create multiple Linux users3. For each user, create the file ~/.fetchmailrc 4. Modify /home/mailtestx/.fetchmailrc to retrieve emails every 5 seconds using the IMAP protocol for the user mail

testx sequentially from two different mail servers:set daemon 5defaults protocol IMAP authenticate password no keep fetchallpoll powerlinuxmail.rchland.ibm.com username "mailtestx" password "test"poll powerlinuxmail2.rchland.ibm.com username "mailtestx" password "test"

5. Set the owner of the configuration file using the following command: sudo u mailtestx chown mailtestx:mailtestx /home/mailtestx/.fetchmailrc

6. Run fetchmail in the background as a daemon for user mailtestx sudo u mailtestx fetchmail

NOTE: On RHEL 5.7 this command fails with the following error fetchmail: couldn't timecheck the runcontrol file fetchmail: lstat: /root/.fetchmailrc: Permission deniedSolution : login to maitestx

7. The following command terminates fetchmail when it is running in the background as a daemon sudo u mailtestx fetchmail quit

NOTE: fetchmail requires postfix to be running on the local client system to route mail.

NOTE : ON SLES11 SP1, if you get the following error :miz11 postfix/pipe[2562]: B34206FCC1: to=<test123@miz11.austin.ibm.com>, relay=cyrus, delay=1.3, delays=0.42/0.03/0/0.88, dsn=5.6.0, status=bounced (data format error. Command output: test123: Mailbox does not exist )

Consider creating the Mailbox for mailtestx as follows:miz06:~ # passwd cyrus Changing password for cyrus. New Password: Reenter New Password: Password changed. miz06:~ # cyradm user cyrus localhost Password: localhost> cm user.mailtest1 #create a mailbox for mailtest1 localhost> lm #verify user.mailtest1 (\HasNoChildren) localhost> exit

5.5 Monitoring softwareSeveral monitoring tools are available for Linux that run locally on the virtual server to monitor utilization.

mpstat provides CPU utilizations. For example, “mpstat 2 10” displays CPU metrics every 2 seconds for 10 iterations

iostat provides disk utilizations. For example, “iostat z d x 2 10” displays disk utilizations every 2 seconds for 10 iterations. Utilizations include average disk utilization (last column).

top provides memory utilization. nmon provides disk and memory utilization. nmon can be installed using the IBM Installation Toolkit.

Consider using Ganglia to monitor several virtual servers over the network. Ganglia is an open source monitoring application that is very useful monitoring resource utilization of both individual and multiple virtual servers.

For the development of the OSIS solution guides, the Ganglia configuration collected performance metrics from each of the virtual application servers. At a centralized collection point that received the data, a Ganglia web application displayed the results.

By utilizing Ganglia’s gmetric utility, one can customize Ganglia by adding additional displayable metrics. For OSIS related metric extensions, refer to the Ganglia Gmetric repository and Ganglia Gmetric Script Repository for open source metrics for mail, web, file sharing, MySQL, and networking. For example, a Linux script is available for obtaining the Postfix mail queue size and passing the information to gmetric to add to the database of displayable metrics.

IBM Power customers in highly virtualized environments have used ganglia to ensure efficient system utilization, without reaching limits,. The University of Pittsburgh Medical Center is an example of using Ganglia with customization to ensure 60 virtual servers utilize system resources appropriately.

The following sections cover the installation and configuration of the Ganglia software.

5.5.1 Configuring Ganglia on Monitored Virtual ServersWhen monitoring virtual servers, install and configure the Ganglia monitoring software the same on each virtual server as follows:1. Install Ganglia software updated for PowerLinux (download from Ganglia for IBM Power website ). For example, on

RHEL6: yum install libconfuse* nogpgcheck yum install ganglialib* nogpgcheck yum install gangliagmond* nogpgcheck yum install gangliamod_ibmpower* nogpgcheck yum install gangliamod_ibmrperf* nogpgcheck yum install gangliamod_netif* nogpgcheck

2. Update the Ganglia configuration file /etc/ganglia/gmond.conf: In cluster section, set the cluster name value, for example "PowerLinux".

NOTE: A cluster is a grouping of systems to be monitored. For example, a cluster could be all virtual servers within a OSIS configuration.

the following may need to updated (if not already)udp_send_channel {

mcast_join = 239.2.11.71port = 8649ttl = 1

}udp_recv_channel {

mcast_join = 239.2.11.71port = 8649bind = 239.2.11.71

}

NOTE: The frequency of collecting Collection Group metrics can be modified by changing collect_every = <fre-quency in seconds>

3. Start the Ganglia service: service gmond start

4. To ensure Ganglia starts with each virtual server restart, use this command: chkconfig gmond on

5. To check to see if Ganglia is communicating with the collection system, use the following command telnet <hostname of collection system> 8649 | grep <ip address of monitored system> Verify the monitored system is listed in the output

5.5.2 Configuring Ganglia on Collection SystemOn the virtual server used for collecting the monitored information, the following is done to install and configure the collection system.1. Ganglia software for use with IBM PowerLinux is included with EPEL6 distribution

yum install gangliagmetad2. Update the Ganglia configuration file /etc/ganglia/gmetad.conf:

the name of the cluster needs to be specifieda cluster is a grouping of systems to be monitored. For example, all of the virtual servers within a VIOS configuration

for “data_source <cluster_name> localhost”, set the cluster name value, for example "PowerLinux"NOTE: An optional polling interval (in seconds) can be specified on the data_source command.

the name of the Grid needs to be specified.3. Start the Ganglia service:

service gmetad start4. Ensure Ganglia starts with each virtual server restart:

chkconfig gmetad on

5.5.2.1 Configuring Ganglia GUI on Collection SystemConfigure one of the two versions of the Ganglia web interface as follows:

1. For original Ganglia web interface a. Ganglia web software is included with EPEL6 distribution

i. yum install gangliaweb b. Create a symbolic link from the web server's html directory to the ganglia directory

i. ln s /usr/share/ganglia /var/www/html/gangliac. Visit http://<hostname>/ganglia

2. For Ganglia web interface version 2a. Download software from http://ganglia.info/downloads/testing/gweb1.9.9.2607M.tar.gz (or latest version) into

/var/www/htmlb. cd /var/www/htmlc. untar: tar xvf gweb1.9.9.2607M.tar.gzd. mv gweb1.9.9.2607M ganglia2e. run these commands

i. APACHE_USER="apache"ii. DEST_DIR="/var/www/html/ganglia2"

iii. sed e "s|@varstatedir@|/var/lib|" conf_default.php.in > conf_default.phpiv. install o $APACHE_USER d /var/lib/ganglia/dwoov. install o $APACHE_USER d /var/lib/ganglia/conf

vi. chown R $APACHE_USER $DEST_DIR/confvii. rsync a $DEST_DIR/conf/ /var/lib/ganglia/conf/NOTE: The web interface refresh frequency can be modified by editing /var/www/html/ganglia2/conf.php and modifying $conf['default_refresh'] = <refresh rate in seconds>;

3. visit http://<hostname>/ganglia24. Creating Aggregated Graphs

a. click Aggregate Graphs tabb. selection of virtual servers to include in an aggregated view requires the use of regular expressions, documenta

tion: http://www.zytrax.com/tech/web/regex.htm

c. For example, to include powerlinuxmail, powerlinuxmail2, powerlinuxweb but not powerlinuxwebwl, a $ was added to the end of the expression as a terminatori. powerlinuxmail|powerlinuxweb$

d. Add the new graph to an existing ganglia View for later use.NOTE: View definitions are stored in files in /var/lib/ganglia/conf by default. Optionally delete files for views no longer needed or edit a file to modify a view definition.

6 Performance TuningPerformance tuning needs to be considered throughout the planning, initial configuration, testing, and production phases. Tuning can affect hardware, application, virtual server, and Linux configurations.

6.1 Reference Architecture Tuning Performed By using the IBM Installation Toolkit Simplified Setup Tool to configure both web and mail application servers, minimal performance tuning of the application is necessary. The setup tool automatically configures appropriate application performance for the Power hardware.

However, tuning of virtual server configurations is necessary to address the changes in workloads, testing and production requirements.

The following performance tuning activities performed during the testing for the OSIS solution guides: Virtual I/O Server

1. Installed VIOS initially using 8GB of the system’s 128 GB of memory and one physical CPU. 2. After installing VIOS, monitored memory usage using nmon, which showed that 5 GB of memory and 0.5

physical CPUs was sufficient.

NOTE: The default restricted VIOS CLI shell environment does not support nmon. To use nmon, first use the VIOS command oem_setup_env to exit the restricted shell environment, and then run nmon.

Virtual Application Servers1. For more accurate monitoring, tuning and sharing of resources, multiple virtual servers were used for each

workload. Each virtual server was dedicated to a specific workload. All virtual application servers shared system resources as needed.

NOTE: To replicate an enduser environment, web and mail client workload generation was performed on dedicated virtual servers, using dedicated network and CPU resources.

2. The nmon utility was used to monitor and ensure sufficient processor and memory was allocated for virtual servers at startup time and while testing of workloads.

3. To keep the number of Linux CPUs at a minimum, a limited number of virtual CPUs (vCPUs) were allocated. Sufficient workloads to ensure multithreaded application servers utilized all available CPUs were generated. When sufficient workload was generated across all available Linux CPUs, varying the number of vCPUs from 1 to 4 did not change the mail or web throughput significantly.

4. A recommendation for the OSIS solution is to run with SMT on. Letting PowerVM optimize the usage of the processor cores is usually a good thing. For example, whenever a thread is temporarily stalled on the processor waiting for a memory fetch, the other threads can continue execution.NOTE: To change the number of threads (logical CPUs), SMT can be changed using the ppc64_cpu command. The ppc64_cpu command allows dynamic setting of the SMT settings, but changes are not persistent across a reboot.

5. Since all virtual servers were configured to share CPU resources (uncapped), at times during testing it was necessary to temporarily override the sharing by setting the virtual server’s CPU priority to None – uncapped to prevent the sharing of the CPU with other virtual servers. For example when configuring assigned pCPUs of 0.1 with 1.0 vCPU to prevent the dynamic change to 1.0 pCPUs (to match a setting of 1 vCPU) caused by dynamic CPU sharing.

6. When tuning mail with antispam filter configured, significant CPU resources are required. Match the number of physical CPUs with the number of virtual CPUs. That is, having a virtual CPU is backed by a physical CPU prevents over committing the virtual server’s CPU resources. The following statement taken from this PowerVM quick start guide provides more detail.

NOTE: The ratio of virtual CPUs in a partition to the actual amount of desired / entitled capacity is a "statement" on the partitions ability to be virtualized. A minimal backing of actual (physical) CPU entitlement to a virtual CPU suggests that the LPAR will most likely not be using large amounts of CPU and will relinquish unused cycles back to the shared pool the majority of the time. This is a measure of how overcommitted the partition is.

7. Virtual servers for the application servers shared sufficient network resources through the virtual Ethernet network bridge and were not changed during testing. nmon and Ganglia were used to monitor and verify network utilization, which remained below the adapter’s capacity.

NOTE: When reaching network throughput capacity, additional adapter ports (from the same adapter or an additional adapter) can be added to the virtual network bridge by using the VIOS “mkvdev –lnagg” command to aggregate additional ports.

6.2 Physical and Virtual Processor SettingsThe following provides information concerning configuration of both physical and virtual processors for the OSIS reference architecture.

The following table defines terms used by PowerVM and then applies these to recommended virtual server settings for the OSIS reference architecture for a PowerLinux system.

Table 3 Virtualization Definitions and Recommendations

PowerVM OSIS Reference ArchitectureVirtualization

TermIVM Term

Definition Value Recommendation / Comment

Physical Processors Processing Units

number of physical cores in the system

16 PowerLinux 7R2 has 16 processor cores

Virtual Shared Pro-cessor Pool

Collection of Physical Processors that can be shared amongst virtual servers

11.5 0.5 Physical Processor is recommend for VIOS virtual server, with remaining processors shared by all other virtual servers for optimal processor utilization

Minimum Entitled Processors

Minimum Processing Units

The number of Physical Processors a virtual server is guaranteed from the Virtual Shared Processor Pool

0.1 to 1.0 minimal amount to boot the virtual server

Desired Entitled Processors

Assigned Processing Units

The number of Physical Processors a virtual server may get, if needed and available from the Virtual Shared Processor Pool

= average number of Physical Processors required by the virtual server’s workload

Sum of all Virtual Server’s Desired Entitled Proces-sors should equal size of Virtual Shared Processor Pool to minimize context switches.

Desired Virtual Processors

Assigned Virtual Processors

Determines amount of CPU a virtual server is allowed to consume

= maximum sustained number of Physical Pro-cessors required by the virtual server’s workload

Increase by one if the virtual server workload peak CPU utilization is ~100%.Reduce by one if the virtual server workload average CPU utilization is less than 50%. Check that the ratio to Desired Entitled Processors is 1 to reduce context switches.

Uncapped Parti-tions

allows virtual servers to run with more than their Desired Entitled Processors, up to the number available from the Shared Processor Pool

Use for all virtual servers

Customer Example:A client has existing web and mail x86based Linux servers and they would like to consolidate them to PowerLinux, while adding some additional workloads.

Table 4 Virtualization Processor Settings Example

Virtual Server Processing Units1 Virtual ProcessorsMinimal2 Assigned3 Maximum4 Minimal5 Assigned6 Maximum7

VIOS 0.1 0.5 1.0 1 1 1Web_1 0.1 2.5 8.0 1 3 8Web_2 0.1 3.5 8.0 1 4 8Web_3 0.1 1.0 8.0 1 1 8Mail_1 0.5 1.5 6.0 1 2 6Mail_2 0.5 1.0 6.0 1 1 6DB_1 0.1 1.5 4.0 1 2 4DNS_1 0.1 0.5 1.0 1 1 1Table Notes:

1. Processing Units are the number of physical processors allocated for each virtual server. IBM PowerLinux 7R2 with 16 cores has 16 processing units available.

2. Minimal Processing Units is the number of physical processors required to provide a reasonable boot time for VIOS and Linux. This value was initially set to 1.0 and then modified by monitoring the startup time of each virtual server.

3. Assigned Processing Units is the average number of physical processors required by the workload. This value was initially set based on sizing estimates or measured values of existing systems. Adjustments were made by monitoring CPU utilization over time using Ganglia or nmon utilities. Note that the total of all the virtual server Assigned values can not exceed the number of physical processors, which is 16.

4. Maximum Processing Units is the maximum number of physical processors that would need to be configured for the virtual server while it is running. This value was initially set relatively high and was only changed if a virtual server unexpectedly needed more physical processors.

5. Minimal Virtual Processors is the number of virtual processors required during boot time. It was determined by using a whole number of Minimal Processing Units.

6. Assigned Virtual Processors is the number of virtual processors required during an average workload. It was determined by using a whole number of Assigned Processing Units.

7. Maximum Virtual Processors is the number of virtual processors required during an average workload. It was determined by using a whole number of Assigned Processing Units.

6.3 Additional Tuning Considerations

The following are additional performance tuning considerations applicable to the IBM PowerLinux system and OSIS reference architecture:

1. IBM PowerLinux systems utilize IBM Power 7 processor architecture, which supports Simultaneous Multithreading (SMT) of up to 4 threads per virtual CPU, providing a performance boost when there are many processes and/or threads running at the same time. The example to the right shows 4 vCPUs with SMT set to 4, with Linux running one single threaded application that utilizes just one of the 16 available CPUs. CPU utilization could be improved by reducing the number of vCPUs or the number of SMT threads, running more Linux processes, or processes with more threads. Additional information is available in Nigel Griffiths’ blog article Local, Near & Far Memory part 3 Scheduling processes to SMT & Virtual Processors.

NOTE: During sizing, configuration, and testing of virtual servers ensure the workload leverages the number of threads available for the virtual server. Adjust the number of threads per CPU (1, 2, or 4), the number of virtual CPUs and/or the workload configuration accordingly.

NOTE: Reducing the number of vCPUs assigned to a virtual server, frees these vCPUs up for other virtual servers to use. In addition, setting the number of assigned (entitled) physical processors too low will reduce

the processor cache efficiency, refer to Nigel Griffiths’ blog article Local, Nar & Far Memory part 5 Low Entitlement has a Bad Side Effect.

2. Multiple virtual switches can also be implemented, to provide additional redundancy and improved networking throughput using multiple network adapters. For details, refer to Using Virtual Switches in PowerVM to Drive Maximum Value of 10 Gb Ethernet, http://www03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101752

3. As the number of virtual servers and overall system utilization increases, utilize Ganglia, an open source monitoring utility to monitor all of the virtual servers. The importance of doing this is covered in Nigel Griffiths’ blog article Whole Power Server + Virtual Server Monitoring Part 5 via Ganglia.

6.4 IBM Java Performance TuningThe following is a summary of Java tuning recommendations for running Java applications using IBM’s Java implementation based on the article Java Performance on POWER7, which also provides implementation details.

1. Use 32bit Java unless the memory requirement of the application requires running in 64bit mode. Use of 64bit mode requires more memory and memory bandwidth. POWER7 provides the same number of registers for both 32bit and 64bit modes.

2. Long running applications accessing vast amounts of memory frequently may benefit from using 16M page sizes vs. the default 64K page size for the Java heap and code cache (-Xlp option and Linux command echo NNN > /proc/sys/vm/nr_hugepages, where nnn is # of 16M pages).

3. For workloads that frequently allocate objects, such as transactional workloads, leverage onchip caches to reduce latency by using aggressive Java prefetching of threadlocal heap memory (-XtlhPrefetch option). Note that for most applications hardware data prefetching is not recommended (Linux command ppc64_cpu –dscr=1)

4. When running applications in 64bit mode, have the JVM use 32bit references to objects instead of 64bit to reduce memory and improve cache efficiencies (-Xcompressedrefs option).

5. When using Java JIT, monitor the runtime compilation to ensure the code cache is sufficient. If not, increase the size of the JIT code cache.

6. For applications that do not scale with an increase in the number of threads, consider reducing the number of threads by running with lower SMT setting or when using virtualization, running with less virtual CPUs.

7. Use Java lock reservations for workloads using many threads and many shared resources that are infrequently shared in practice (-XlockReservation option).

8. Consider reducing garbage collection (GC) overhead by using four GC threads for the JVM. If not specified, the number of GC threads equals to the number of logical CPUs on the system (-Xgcthreads4 option).

9. For workloads that have many shortlived objects, such as transactional workloads, run with the generational garbage collection scheme that attempts to deal with many varying workloads and memory usage patterns (-Xgcpolicy:gencon option).

10. Run with no heap space for very large objects (Xnoloa option).

7 Appendix

7.1 Install Linux Directly Using VNC As an alternative to using the IBM Installation Toolkit to install Linux and the IBM provided tools, you can install Linux directly and later install the IBM tools as follows:1. Configure a virtual server as described earlier, with both the Linux

distribution media and IBM Installation Toolkit optical media assigned.

2. From IVM start the virtual server and use SMS to configure a boot from the Linux distribution media.

3. During the initial boot, when boot: appears, enter linux vnc [ip=<ip address> gateway=<gateway address> netmask=<network mask>]

4. Linux will attempt to configure TCP/IP automatically or use the static address you supplyNOTE: If an error occurs because no DHCP provided dynamic address is available, press F12, select Skip, and then manually configure the IP address.

5. The Linux installer will start the VNC server and display the IP address used for a VNC connection.

6. Use a VNC client to open a session using the IP address specified and session ID of 1.

7. Continue the installation using Linux installer.8. Reboot the system at the end of install process9. From a remote terminal, login10. Install the latest IBM Service and Productivity tools for Linux on Power from an IBM provided repository. For exam

ple, to configure the repository and install the tools for a virtual server running RHEL6 Create a repo file for the Linux distribution media, for example /etc/yum.repos.d/RHEL6.repo containing

[RHEL6]name=RHEL6 DVD in SR0baseurl=file:///media/linux/Serverenabled=1gpgcheck=0

Install the tools as follows wget ftp://public.dhe.ibm.com/software/server/POWER/Linux/yum/download/ibmpowerrepo1.1.6

5.ppc.rpm rpm ivh ibmpowerrepo1.1.65.ppc.rpm yum install ibmpowermanagedrhel6

7.2 Creating Partitions with LVM during Direct Linux InstallationDuring installation of Linux, you can manually partition a virtual disk with logical volumes. The only partitioning difference from an x86based system is the requirement for an PPC PReP partition required for booting Linux on a Power system.

The following outlines creating 3 physical partitions (prep, boot and LVM) and creating a Volume Group with 2 logical volumes (swap and root) using RHEL 6 installer (reference figure to the right)1. From the RHEL installer, for the installation type select Create Custom Layout 2. Create the PPC PReP Boot partition

a. Click Create to create storageb. Select Standard Partition and click

Createc. For mount point, leave blankd. For File System Type, select PPC PReP

Boote. For Size, enter 4 (MB)f. Select Fixed Sizeg. Select Force to be primary partitionh. Click Create to create this partition

3. Create the boot partitiona. Click Create to create storageb. Select Standard Partition and click Createc. For mount point, enter /bootd. For File System Type, select Ext4e. For Size, enter a value sufficient for the swap area. For example, set the size to 100 MB. The swap area should

not be used with virtual memory sharing available.f. Select Fixed Sizeg. Select Force to be primary partitionh. Click Create to create this partition

4. Create partition for LVMa. Click Create to create storageb. Select LVM Physical Volume and click Createc. For File System Type, use the physical volume (LVM) default valued. For Additional Size Options, select Fill to maximum allowable sizee. Select Force to be primary partitionf. Click Create to create this partition

5. Create LVM Volume Group and Volumesa. Click Create to create storageb. Select LVM Volume Group and click Createc. For Volume Group Name, use the default or enter a new named. For Physical Extend, select a value or use the default valuee. Create Logical Volumes

i. Create a logical volume for the swap area1. Click Add2. For File System Type, select swap3. For Logical Volume Name, enter a meaningful name such as swap4. For Size, enter a value based on memory size5. Click Ok to create the logical volume

ii. Create a logical volume for the root directory 1. Click Add2. For Mount Point, select /3. For File System Type, select ext44. For Logical Volume Name, enter a meaningful name such as root5. For Size, enter the maximum value (shown by default)6. Click Ok to create the logical volume

iii. Click OK to create the Volume Group6. Continue with the Linux installation

7.3 Creating Partitions with LVM during IBM Installation Toolkit Installations

When using the IBM Installation Toolkit to install Linux, manual configuration of the disk is possible. The following example provides the details

1. On the installation settings menu, for Disk partitioning select Manual.

2. After continuing the installation, the Disk partitioning menu displays.

3. When the storage device is new, an error requesting a msdos partition table is displayed. Select Create msdos par-tition table to create the missing partition.

4. Otherwise, if there any existing partitionsIf a ‘prep’ partition exists, do not delete it. The “prep” partition is required for installing Linux on a Power system.

a. Delete any remaining partitions by b. moving the cursor to the () field for the partitionc. pressing enter to selectd. tab and select DeleteNOTE: Ignore the error message concerning missing partitions; these are created in the following steps.

5. Select the blk1 partition in the list by moving to blk1 entry and pressing enter.6. Select Add to create a new partition.7. If necessary, create a ‘prep’ partition as follows

a. for Partition type, select primaryb. for File System, select prepc. for Size, hit enter and type 8, press enterd. for Mount point, leave blanke. select Add to create the partition

8. Repeat the steps above to create a new primary partition with an ext3 file system of 100MB mounted at /boot.

9. Repeat the steps again to create another primary partition with a lvm file system using All available storage (no mount point)

10. At this point, there should be three primary partitions.

11. At the top of the screen, tab and select Man-age VolGroups

12. On the LVM volume groups management menu, select Add to add the LVM volume group.

13. For the volume groupa. enter a volume group nameb. select the physical volume by pressing enter on the volume namec. select Add to create the volume groupd. select Done to complete the creating of volume groups

14. To create the volumes, at the top of the screen, a. for Working on select LVM VolGroups…b. tab and select Change

15. Click Add to add a volume

16. Create a swap volume partition by a. for File System, select swap b. specify a sizec. click Add to create the volume partition

17. Create a second volume partition for the root directory bya. select blk1 from the list and select Addb. specifying a namec. specifying a file system of ext3 d. selecting All available for the sizee. specifying / for the mount pointf. selecting Add to create the volume partition

18. At this point, there should be two volumes in the volume group.19. Select Next to continue with the install wizard.

8 References

IBM PowerVM Virtualization

1. A Comparison of PowerVM and VMware Virtualization Performance, http://www.ibm.com/common/ssi/cgibin/ssialias?infotype=SA&subtype=WH&appname=STGE_PO_PO_USEN&htmlfid=POW03041USEN&attachment=POW03041USEN.PDF

2. IBM PowerVM Virtualization Introduction and Configuration, http://www.redbooks.ibm.com/abstracts/sg247940.html?Open

3. PowerVM Managing & Monitoring, http://www.redbooks.ibm.com/abstracts/sg247590.html?Open4. IBM System p Advanced POWER Virtualization (PowerVM) Best Practices, http://www.redbooks.ibm.com/abstracts/red

p4194.html 5. PowerVM QuickStart, http://www.tablespace.net/quicksheet/powervmquickstart.html6. IBM PowerVM Live Partition Mobility, http://www.redbooks.ibm.com/abstracts/sg247460.html 7. IBM Integrated Virtualization Manager Lowering the cost of entry into PowerVM Virtualization,

ftp://public.dhe.ibm.com/common/ssi/ecm/en/pow03073usen/POW03073USEN.PDF 8. Achieve technical and business benefits through processor virtualization, http://www.ibm.com/developerworks/aix/library/au

provirt/index.html 9. Using Virtual Switches in PowerVM to Drive Maximum Value of 10 Gb Ethernet, http://www.ibm.com/support/techdocs/ats

mastr.nsf/WebIndex/WP101752 10. IBM System Planning Tool, http://www.ibm.com/systems/support/tools/systemplanningtool/index.shtml 11. Measuring Stolen CPU Cycles, http://www.ibm.com/developerworks/wikis/display/LinuxP/Measuring+stolen+CPU+cycles

Virtual I/O Server (VIOS)

1. IBM Infocenter, http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/iphb1/iphb1kickoff.htm 2. Virtual I/O Server Support for Power Systems, http://www.software.ibm.com/webapp/set2/sas/f/vios/documentation/home

.html 3. Virtual I/O Server and Integrated Virtualization Manager commands,

http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/topic/iphcg/iphcg.pdf

IBM Systems Director

1. IBM Systems Director Management Console: Introduction and Overview, http://www.redbooks.ibm.com/abstracts/sg247860.html

IBM PowerLinux

1. Installing the IBM Power 710 and Power 730, http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/topic/p7ed9/p7ed9.pdf

2. IBM Installation Toolkit (website), http://www14.software.ibm.com/webapp/set2/sas/f/lopdiags/installtools/ 3. Performance FAQ – Exploiting POWER7,

https://www.ibm.com/developerworks/wikis/display/LinuxP/Performance+FAQs#PerformanceFAQsCanRedHatVersion5exploitPOWER7%3F

4. Think Power Linux (website), http://www.ibm.com/developerworks/group/tpl 5. SAS RAID controllers for Linux, http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/topic/p7ebk/p7ebk.pdf 6. Service and productivity tools For Linux on POWER systems, http://www14.software.ibm.com/webapp/set2/sas/f/lopdi

ags/yum.html 7. Java Performance on POWER7, https://www.ibm.com/developerworks/wikis/display/LinuxP/Java+Performance+on+POW

ER7

Open Source

1. Red Hat Software & Download Center (RHEL evaluation software), https ://access.redhat.com/downloads/ 2. Secure Shell tools: Putty for Windows or Linux (http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html3. ) or OpenSSH (http://www.OpenSSH.com) which is almost always included with Linux distributions by default4. Postfix Basic Configuration (website), http://www.postfix.org/BASIC_CONFIGURATION_README.html 5. Setting Up Email: A Postfix/Dovecot HOWTO, http://rimuhosting.com/support/settingupemail.jsp?mta=postfix 6. Postfix Standard Configuration Examples, http://www.postfix.org/STANDARD_CONFIGURATION_README.html 7. Postfix Performance Tuning (article), http://www.postfix.org/TUNING_README.html 8. Dovecot Test Installation, http://wiki.dovecot.org/TestInstallation 9. Ganglia for IBM Power Systems (website), http://www.perzl.org/aix/index.php?n=Main.Gangliafiles317

10. Drupal Installation Tutorial, https://www.linux.com/learn/tutorials/267051installingdrupal 11. Extra Packages for Enterprise Linux (EPEL website), http://fedoraproject.org/wiki/EPEL 12. nmon for Linux, http://nmon.sourceforge.net/pmwiki.php

Disclaimer and Trademarks

The information in this Guide is intended to provide guidance for those implementing IBM PowerLinux Open Source Infrastructure Services (OSIS). It discusses findings based on a solution that was created and tested under laboratory conditions. These findings may not be realized in all customer environments, and implementation in such environments may require additional steps, configurations, and performance analysis.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION

PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication.

IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Implementation and certification of the solution rests on the implementation team. The users of this guide should always check the latest release information in the product Readme file(s) and check the product web pages for the latest updates and findings.

Any references in this information to nonIBM web sites are provided for convenience only and do not in any manner serve as an endorsement of those web sites. The materials at those web sites are not part of the materials for this IBM product and use of those web sites is at your own risk.

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries.

Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any nonIBM product, program, or service.

Linux is a trademark of Linus Torvalds in the United States, other countries or both.

The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a worldwide basis.

Intel, Itanium and Xeon are registered trademarks and MMX and Pentium are trademarks of Intel Corporation in the United States and/or other countries

Java and all Javabased trademarks and logos are trademarks of Sun Microsystems, Inc. In the United States and/or other countries.

All performance information was determined in a controlled environment. Actual results may vary. Performance information is provided “AS IS” and no warranties or guarantees are expressed or implied by IBM. Buyers should consult other sources of information, including system benchmarks, to evaluate the performance of a system they are considering buying.

SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and SPECsfs are trademarks of the Standard Performance Evaluation Corporation (SPEC).

Photographs show engineering and design models. Changes may be incorporated in production models.

Copying or downloading the images contained in this document is expressly prohibited without the written consent of IBM.

© IBM Corporation 2010IBM CorporationSystems and Technology GroupRoute 100Somers, New York 10589

Produced in the United States of AmericaNovember 2011All Rights Reserved

This document was developed for products and/or services offered in the United States. IBM may not offer the products, features, or services discussed in this document in other countries.

The information may be subject to change without notice. Consult your local IBM business contact for information on the products, features and services available in your area.

All statements regarding IBM future directions and intent are subject to change or withdrawal without notice and represent goals and objectives only.

IBM, the IBM logo, ibm.com, IBM Systems Director VMControl, POWER, POWER Hypervisor, Power Systems, PowerLinux, Power Architecture, PowerVM, MicroPartitioning, are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml

Java and all Javabased trademarks and logos are trademarks of Sun Microsystems, Inc. In the United States and/or other countries.

Other company, product, and service names may be trademarks or service marks of others.

IBM hardware products are manufactured from new parts, or new and used parts. In some cases, the hardware product may not be new and may have been previously installed. Regardless, our warranty terms apply.

This equipment is subject to FCC rules. It will comply with the appropriate FCC rules before final delivery to the buyer.

Information concerning nonIBM products was obtained from the suppliers of these products or other public sources. Questions on the capabilities of the nonIBM products should be addressed with those suppliers.

When referring to storage capacity, 1 TB equals total GB divided by 1000; accessible capacity may be less.

The IBM home page on the Internet can be found at: http://www.ibm.co m .

The IBM Power Systems home page on the Internet can be found at: http://www.ibm.com/systems/power/

xxxxxxxxUSEN00

End of Document