ESX3.x VC2 Upgrade Guide

8/6/2019 ESX3.x VC2 Upgrade Guide

1/123Pa e 1

VMware ESX 3.x Serverand

VirtualCenter 2.x

(GA Build Eval)Upgrade Guide

Document Version 1.3

By Mike Laverick RTFM Education

For Errors/Corrections please contact:[email protected]

This guide does change. Have you

got the latest copy?


2/123Pa e 2

Table of Contents

Introduction.............................................................................................4Module 1: Upgrading Virtual Infrastructure .............................................7

Executive Summary..................................................................................7Technically Specific Issues.........................................................................7

Module 2: In Place Upgrade of VC ..........................................................10Executive Summary................................................................................ 10In Place Upgrade of VC 1.2 to 2.x ............................................................. 10Post-Upgrade Tasks ................................................................................ 14Installing the VI Client ............................................................................ 15

Module 3: In Place Upgrade of ESX ........................................................18Executive Summary................................................................................ 18Confirming you can upgrade .................................................................... 20Upgrading with CD-ROM.......................................................................... 21Licensing ESX Server .............................................................................. 22Upgrading VMFS-2 to VMFS-3 .................................................................. 24Relocate the Virtual Machines .................................................................. 27Update VMFS Volume Labels & DataStore Name for Uniqueness................... 27Custom Upgrading with a Tar-Ball (Optional) ............................................. 28Upgrading to VMFS-3 from the Service Console.......................................... 31Fixing the Service Console Network Problems ............................................ 34Clean Installation of ESX 3.x (Optional)..................................................... 37Enabling SSH/SCP from your ESX host to other hosts ................................. 42Cold Migration from ESX 2.x to ESX 3.x..................................................... 43

Module 4: Upgrading Virtual Machines ...................................................45Executive Summary................................................................................ 45Upgrading Virtual Machine Hardware (GUI)................................................ 45Upgrading VM Tools (GUI) ....................................................................... 46Upgrading VMXnet to Flexible Network Adapter.......................................... 47Upgrading VM Hardware & Tools for Bulk (CLI) .......................................... 47

Module 5: Networking ............................................................................49Executive Summary................................................................................ 49Creating an Internal Only Switch .............................................................. 51Creating a NIC Team vSwitch................................................................... 52Creating a VMKernel Switch for VMotion.................................................... 52Increasing the Number of Ports on vSwitch................................................ 55Setting Speed & Duplex .......................................................................... 56Managing Service Console Networking ...................................................... 57Creating a vSwitch with VLAN Support ...................................................... 60Conclusion............................................................................................. 61

Module 6: Storage (VMFS, SANs, NFS, iSCSI) ........................................64Executive Summary................................................................................ 64Setting up NFS/NAS on RedHat Linux........................................................ 66Setting up NFS/NAS on Windows with SFU ................................................ 66Adding a NFS Mount Point ....................................................................... 69Setting iSCSI (Software Emulation) .......................................................... 71Setting up iSCSI with Fedora Core 5 and iSCSI Target ................................ 72Setting up a Windows iSCSI Target with Stringbean WinTarget .................... 74

Module 7: Deployment and Migration .....................................................81Executive Summary................................................................................ 81Creating a Folder Holding Templates......................................................... 82Using Clone to a Template....................................................................... 83Using Convert to a Template.................................................................... 85Importing and Exporting Virtual Disks ....................................................... 86Manage VMs using Web Access................................................................ 86


3/123Pa e 3

Generating Remote Console URLs for VMs................................................ 87Modifying a VM: Hot Pluggable Disks......................................................... 89Using Topology Maps .............................................................................. 91VMotion and CPU Compatibility ................................................................ 94Checking your ESX CPU Attributes with CPU Boot CD .................................. 94Enabling and Disable CPU Attributes on a VM............................................. 95Performing a Cold Migration..................................................................... 96Performing a Hot Migration (VMotion) ....................................................... 97

Module 8: Resource Management.........................................................100Executive Summary .............................................................................. 100Creating Resource Pools on an ESX Host ................................................. 101Distributed Resource Service (DRS) Cluster ............................................. 105Creating a DRS Cluster.......................................................................... 106Creating VM Affinities and Anti-Affinity Rules ........................................... 112Modifying Virtual Machine Options .......................................................... 116Removing an ESX Host from a DRS Cluster.............................................. 117

Module 9: High Availability & Backup...................................................119Executive Summary .............................................................................. 119Enabling the VMware HA Cluster............................................................. 119Trigging VMware High Availability ........................................................... 121

Appendix A: Reverting back to SQL Authentication ..............................123


4/123Pa e 4

Introduction

Purpose of this guide

This guide is designed for people who already know ESX 2.x and VC 1.x very well.

It based on the delta two-day training that can be optionally attended for thosepeople with prior experience. It also contains additional information beyond that

course.

This guide is certainly NOT for novices or new-users!

Although Ive chosen to call this an upgrade guide, its by no means a definitive

statement on upgrading. For that you need to read VMware documentation. I alsoview this guide as upgrade of skills as well as software.

It is not a comprehensive guide to ALL the differences just the primary ones. I

hope to make this guide gradually more comprehensive, and cover all new

features. As you might gather theres a lot marginal GUI like changes which arenot included.

The emphasis here is on the Virtual Infrastructure Client used with VC.

Command-line options are introduced when it is quicker/easier or the only way tocarry out a task. Its not intended as a comprehensive guide to the Service

Console

It is highly recommended to attend the Whats New Course course, and readVMware documentation. Try to view this guide as a quick how to do guide with

quick Executive Summaries at the beginning of module.

Please email at the email address at the beginning of this document if spot any

errors or screen grabs that need updating

Hardware

This guide assumes you have at least 2 servers with 2CPUs at 1.4mgzeach

2GB of RAM 2x36GB Hard Drives (SCSI Internal) 4x Network Cards (production in a bond, vmotion and Service Console

eth0)

This is the specification of my server an old Dell 1650 PowerEdge usingPIII processors!

My hardware isnt officially supported by VMware anymore. Anyway it stillruns. But at some point I am going to buy two DL380, and re-use thishardware as my VC box and a NAS box

I have VC set-up with a SQL Database. The layout of my VC looks like thisbefore the upgrade:


5/123Pa e 5

Two Servers, Two Virtual Machine Groups (for me and my mate, Trevor)The domain name is rtfm-ed.co.uk

lavericm-admin is the VC Administrator set at Server Farms

instructor is a Virtual Machine Admin from the RTFM Farm (used when I

teach with my hardware)

baverstockt is a Virtual Machine User with rights only for Trevors VMsGroup

These are very much dummy permissions I use to test my migrationSoftware

This guide was started using ESX 3.x Beta and VC 2.x Beta As time progressed I moved on to ESX 3.x RC1, and VC 2.x RC1 This version has been upgraded and is now based on the GA Eval released

on 22nd June 2006 In my case VC Server 2.x and SQL 2000 (SP3) all runs on Windows 2003

with Service Pack 1. This was the most current release of Windows at the

time of writing this document

Warning: As ever my documents are released as is and without warrantyConventions in this Guide

Blue usually represents a hyperlink in the Virtual Infrastructure Client(VI Client)

I tend to use to represent radio buttons, as shortcut rather thananother character like this Yes

I tend use X to represent check off boxes, shortcut rather than anothercharacter like this X Connected at Power On I use PuTTy to get Service Console sessions I use nano rather than vi to edit and save files mainly because Im no

Linux guru, and I find nano more user friendly. Vi text editor is popular in

the Linux community because it pretty much standard amongst every

Linux distribution

Any Major titles marked with Redindicates this sectionis broken, released as is, and I intend to return to fix it. But got boredbanging my head against a brick wall.


6/123Pa e 6

Change Log from 1.1 to 1.2

General changes based on GA Eval Release of ESX 3.x and VirtualCenter2.x

Added how to set-up Fedora Core 5 with iSCSI Target, and then allowingESX access to the iSCSI LUNs for use as VMFS or RAW LUNs with RDM

Files. The observant amongst you will notice slight differences in some ofmy screen dumps as the documentation for iSCSI was retro-fitted to this

guide after most of it had been written. Resolved upgrade problems with VirtualCenter Database Added a the new method of exporting virtual disks with vmkfstools i Added using the preupgrade.pl script which flags up potential problems

prior to an in place upgrade

Added using cupid.iso which shows cpu attributes such as Family,Stepping, SSE Version, Execute Denied/No Execute, and 64-bit support

Correct problems in Tar Ball Upgrade of ESX 3.x Added how to upgrade VMFS from the Service Console Added the procedure for Relocating VM Files

Change Log from 1.2. to 1.3 Added enabling the SSH Client on the ESX host to allow ssh and secure

copy from just one ESX host to all others

Corrected some basic typographical and spelling errors


7/123Pa e 7

Module 1: Upgrading Virtual InfrastructureNote:

I use the term VI-2 to refer collectively to ESX 2.x and VC 1.x I use the term VI-3 to refer collectively ESX 3.x and VC 2.x This module is NOT practical. But a quick summary of the upgrade issues

Executive Summary There is NO upgrade path to VC 1.3.1 to VC 2.x. Due this the guide was

written with an upgrade from VC 1.2 to VC 2.x As it is have to report that VC 1.x to 2.x in place upgrades are not very

reliable (at least in my limited experience). ESX upgrades have beensurprisingly painless. Still you cannot set much store by one guys

experiences Lots of people on the forum have done an upgrade of VC 2.x

without an error. You can run VC 1.x and VC 2.x within an appropriately resourced VM,

VMware now suggest you that is possible to put the VC server in a VMware

High Availability clusters (VHA was previously called DAS) so should theESX host it is running on fails, it would get restarted on another ESX hostin the cluster. This is because VC does not have to be running to HA to

work

The scariest aspect of an in place upgrade is the migration of VMFS-2 toVMFS-3. As more than one ESX server can use the SAME VMFS volume

you need to know where you storage is, where your virtual disks arelocated, and which ESX server can see the LUNs. This because ESX 2.x

cannot read/write to a VMFS-3 volume, and ESX 3.x has only read-onlyaccess to VMFS-2

As with all updates of software the big question is in place upgrade orre-install. I imagine most people will use a twin-track approach of running

both VI-2 and VI-3 side-by-side. Critical to this is how long VI-2 will be

supported by VMware. There is already healthy debate about when VMware thinks this will, and

what its customers think it should be!

VMware has many strategies to choose from for the upgrade path. Thebest one is the one that minimizes your VM downtime

The order of the upgrade is important, and this guide obeys that order.Each stage of the upgrade process is not undoable, so backups are

absolutely essential These are VMware recommend stages:

o Backupo Upgrade VC 2.xo Upgrade ESX 3.0o

Upgrade to VMFS-3o Upgrade Virtual Machine hardwareo Upgrade VMware Toolso Leverage new features

Technically Specific Issues

ESX 2.x Partition Scheme and Boot Processes

When ESX 2.x was released VMware recommended 50MB for /boot and2048MB for /

These recommendations were made in good faith at the time and werefit-for-purpose


8/123Pa e 8

However, you may experience problems if you have followed theserecommendations to the letter

With 50MB boot you will not have space for all the boot menu options and you will loose the Debug Menu. Basically, there is not enough spacefor the IMG files. I am working on a work-around on this issue by deleting

the old IMG from the old ESX 2.x LILO menu

The upgrade expect around 2048MB of free-space in the / partition I have followed ESX 2.x recommendations, and have done a successful

upgrade. But I did loose the Debug Menu Ive experimented briefly with repartition tools but not found a tool that

can move and resize a partition(s) within an extended partition, AND movean extended partition so then the primary partitions can be moved to

make the first partition /boot bigger.

FYI. The boot loader is now GRUB, and the ESX Service Console is basedon Redhat Linux AS 3.0

VC Authentication

VC 1.x was geared up to use NT4/AD/Workgroup users in Windows

If you have a hot-backup/hot-replacement/standby of VC 1.x usingWorkgroup users its recommended to switch to a NT4/AD model In VC 1.x the users were stored by name (this was not fun if you rename

users!), they are reference by UID.

This means two VC 2.x servers pointing to the same SQL database wouldhave a problem because the accounts would have different UID values inthe Workgroup/SAM file

Basically with out the move to NT4/AD you would have authenticationproblems

This is not an issue if you only have one VC server and you have no hot-backup/hot-replacement/standby

Clustering VMs across boxes

There is no shared mode anymore hurrah! There only support is for RDMs to a LUN Any clusters-across-boxes where the Quorum and Shared disks are virtual

disks, need converting via cloning software to a raw LUN

Clusters-in-a-box remain unchanged, as do physical2virtual clusters.However, LSIlogic and vmxnet are only supported that mean switching

from BusLogic/Vlance if you used the default VMX options in VI-2 with

Windows 2000 Advanced Server

Other Issues

Commit your redos!o ESX 2.x redos do not work in ESX 3.x!

New Processor Compatibility issueso Intel and AMD support NX/XD (No Execute/Execute Denied bit)o Prevents buffer overflows and other attackso VI-3 supports ito By default you cant VMotion from a non-NX/XD host to a NX/XD

enable host. I imagine there maybe a weakening this

constraint/requirement or masking these bits between hostso Check out the CPU Compatibility Boot Disk to confirm you

processor attributes

DNSo Some new features require DNS name resolutiono Some new features work less easily out-of-the-box without DNS


9/123Pa e 9

o If previously you have managed your VI-2 environment withoutDNS, using IP address or hosts files I would recommend

integrating them into current DNS environment


10/123Pa e 10

Module 2: In Place Upgrade of VC

Executive Summary According to the GA documentation you can only upgrade VC 1.2 and

1.3. Anything older than VC 1.2 needs upgrading to 1.2 or 1.3 before you

begin

Make sure you do NOT have IIS or any other software which claims port80 or 443 as new VirtualCenter Web Service will now want to use those

ports. Previously the VC Web-Service used different port numbers Installer detects old VC, install new VC, and de-installs the old version An upgrade wizard upgrades the SQL/Oracle database There is no support for MDB, and no upgrade path for MDB There is support for MSDE Serial numbers dont exist, instead they use a FlexNet License server and

License files

VMware recommend the new license server be install on the same box asthe VC server

You can have stand-alone/workgroup model for ESX with a LIC file oneach, or served license from the license server to a group of ESX hosts

no prizes for guess which is easier to manage!

If the license server fails things carry on running on a cached license. Farms dont exist anymore, the upgrade converts them to DataCenters

the principle is the same, its just the terminology has changed A single client called the Virtual Infrastructure Client (VI Client) used

manage either an ESX host or VC

The MUI no longer exists (Hurrah!) A single client called the Virtual Infrastructure Client (VI Client) used

manage either an ESX host or VC

The MUI no longer exists (Hurrah!) VC 2.x now asks during the install for a service account for running the VC

Server. VMware recommend using the System Account. You do have the

option of specifying an account

In Place Upgrade of VC 1.2 to 2.xNote:

My guide assumes you going for in-place upgrade of all the software. This mightnot be best for your environment. Do the research first before steam-rolling in!

What you need to complete this task:

The VC 2.x CD/Software 1 Served License Password/Username of the DB user which is currently used with the VC

1.x database

My VC was set-up as member server on W2K3+SP1, using SQLAuthentication

IMPORTANT:Actually, I would normally set it up with Windows Authentication. I had

problems with this so I reverted by to SQL Authentication. If you want to

know how I did this see the Appendix at the end of this document

1. Insert the VC 2.x CD


11/123Pa e 11

Note:

You may need to confirm the Microsoft-Nanny-State, are you sure youwant to run this application style messages

2. Choose VC/Web AccessNote:

After a short while the MSI runs, and the installer detects your old version

of VC 1.2 like so:

3. Click Next, the Confirm the EULA, and Accept the Location for theinstallation

4. Choose Custom5. Click Next for the Program Features

Note:

Remember you only need the VC Web Service if you creating customapplication with the VC SDK

6. Choose Using Existing database serverNote:

The installer should detect your previous VC ODBC and authentication

settings. Leaving you with just a space for inputting the password. If not review your ODBC settings as appropriate and confirm your user accounts

and passwords:


12/123Pa e 12

7. Click Next, and you should get a pop-warning:

Note:

This is why you should backup before you begin. My VC/SQL is runningwithin VMs in undoable mode in case this upgrade fails!

Click OK

8. Following VMwares own recommendations select Install a localVMware License Server and Configure VMware VC to use it

Warning:If you set-up the SQL database for VC with Windows Authentication, you

must specify your account here. Otherwise the upgrade of the database

will fail due to lack of permissions. In my case this was true:


13/123Pa e 13

9. Click the Browse button, and select a Served License for use with thelicense server, and click Next

Note:Dont select an unserved, ESX host only LIC file. Otherwise (shock/horror)

it doesnt work!

10.Following VMware recommendation, click Next to accept that VC2.x willuse the built-in System Account

11.Accept the default port numbers for VC Web Service12.In the VC Web Service setup dialog enable:

X Set the Apache Tomcat Service for Automatic Start-up

X Start the Apache Tomcat Service

13.Click InstallNote:At some stage you will presented with this dialog below. Click No

Read that bit again your choosing NO because you do want to

retain your DB so it can be upgraded!

Note:At the end of this file copy you will be prompted to upgrade the

database. This utility can be run as a separate entity if you so wish it is

called VCDatabaseUpgrade.exe and is located in C:\ProgramFiles\VMware\VMware VC 2.x

14.Choose Finish, but allow X Launch the VMware VC Server 2.xdatabase upgrade wizard

15.Choose Next


14/123Pa e 14

Note:

This is far as I got in my upgrade. From this point onwards, horrible thingshappened to me, which I would rather not talk about.

16.Choose Next to this dialog box:

Note:The database upgrade will then proceed to update the VC DB with status

messages:

Creating the new database schema

Renaming Clashing VirtualCenter 1.x tables

Creating VirtualCenter 2.0 Tables and ProceduresCreating upgrade proceduresUpgrading Hosts

Upgrading Events and TasksUpgrading AlarmsInitializing access control lists

Dropping Temporary Tables

Dropping VC1.x tables and proceduresNow starting the VMware VirtualCenter Service

Note:You will be probably desperate to see whether this has really worked Sogo on install the VI client point it at your VC server and see whats there!

When I first logged on I found my ESX 2.x hosts were disconnected soright-clicked them, and choose Connect and that fixed that problem

Post-Upgrade TasksNote:

Open up the Services program in Administrative Tools, and confirm thatyour VMware VC Service has started properly sometimes it doesnt and

you have to start it manually After the VC upgrade you will notice you will still have a directory left over

from the VC 1.x install This sysprep directory held at C:\Program Files\VMware\VMware

VC\resources\windows

The location of the Resources\Windows\Sysprep has been change toC:\Documents and Settings\All Users\Application Data\VMware\VMware

VirtualCenter\sysprep.


15/123Pa e 15

Installing the VI ClientNote:

You can install the VI client to VC Server or Workstation You can get the VI client from the VC CD You can download the VI Client from an ESX 3.x server As we have yet to upgrade ESX 2.x lets do it from the CD I guess you

will want to get into the VC Server ASAP to see how well the upgrade went

1. Insert the VC 2.x CD

2. Under other choose Virtual Infrastructure Client3. Click Next and Accept the EULA4. Run the Client and Login

5. After the upgrade your VC Inventory will look like this:


16/123Pa e 16

Note:

You will notice that our VM Groups seem to have disappeared. Theyhavent they are in different view. This is the Hosts & Clusters view if youswitch the inventory to the view Virtual Machines and Templates you

will find that the Virtual Machine groups (now referred to as merelyfolders) are still there

I am going to rename RTFM Education, to be called RTFM DataCenter just

so were clear that term farm is not used anymore

6. Click the Arrow next to the Inventory button:

7.

Choose Virtual Machines and Templates

Note:At this stage we cannot manage the ESX 2.x host via the VI Client,because they are yet to be upgraded. You can however, manage the VMs

that reside on them.

Instead you are stuck with the MUI. Once ESX has been upgrade you willget a Configuration tab to manage the physical host until then to

access the MUI from the VI Client do the following:

Select the ESX Host

Click the Summary Tab


17/123Pa e 17

Click

VMotion settings are accessible just above this option


18/123Pa e 18

Module 3: In Place Upgrade of ESX

Executive Summary

Downtime and VMFS Storage

You must be running on ESX 2.1.1 or later with the exception of ESX2.3.0 and ESX 2.5.0

Upgrading ESX requires a reboot which means VM down-time isinevitable

It also means upgrade of VMFS-2 to VMFS-3 during this time the VMsmust be powered off

ESX 2.x cannot read VMFS-3. VMFS-3 volumes appear as unformatted orunavailable in the Storage Management Page of the MUI

ESX 3.x can read VMFS-2 but cannot write to it Critically, the choice is between upgrade or migrate VMFS Volume Labels need to be globally unique to a Farm/DataCenter. So

if every ESX server has a local VMFS volume called /vmfs/local, then theVC will serialise them to keep them unique. You can observe this

phenomena on the properties of ESX and the Summary Tab

esx1.rtfm-ed.co.uk:

esx2.rtfm-ed.co.uk:


19/123Pa e 19

Note:

You can also this in the DataStores view in the VI Client:

The only disappointing thing I feel about the upgrade is the state yournetworking is left in afterwards. Theres lots of references to legacy this n

that and I have (under the beta/rc1) experience situations where I havelost COS connectivity after an upgrade. This said all of these problems Iwas able to work around

Upgrade Criteria

Few virtual disks VMs can be brought down altogether Downtime works out cheaper than provisioning new storage for a Migrate How:

o Shutdown all VMso In Place Upgrade ESX/VMFS-3o Power on VMs on ESX 3.x server

Migrate Criteria

Many virtual disks Impossible to shutdown all VMs Cost of provisioning new storage is cheaper than downtime, or new

environment when storage can be use temporarily during migrate


20/123Pa e 20

How:o Clean install a new ESX 3.x servero Shutdown VMs from ESX 2.xo Copy VMDKs/VMX from ESX 2.x to 3.xo Power on VMs on ESX 3.xo Repeat until VMFS-2 partition is emptyo Replace old VMFS-2 partition with VMFS-3

Templates & ISOs are different

You can still use the old format of virtual disk for templates (COW orwhats they are now referred to as sparse format)

You can still use vmkfstools i and e to import and export There is no GUI tool currently to import/export them into a VMFS partition

because of the end of the MUI

In my experience, with the advent of directories in a VMFS, VC seems tohandle putting all the VMs files in one location more easily than doing itmanually via the Service Console and VI Client

It is no longer recommended to use Service Console anymore for thestorage for ISOs You can store templates on VMFS Shared LUN as you did in VC 1.x

You can put ISOs there as well alternatively you might want to takeadvantage of NAS based storage for ISOs and Templates as this is

cheaper per MB of storage For this reason a vmimages partition is no longer required. I would

recommend off-load your vmimages storage to NAS and then think of

different usage for this storage or keep your vmimages as a backup ofstuff you would normally use on your NAS

This said, you will still find a /vmimages directoryon your server even ifyou do a clean install and dont create a /vmimagespartition. It is now

used to store the ISO images for VMware Tools in the /vmimages/tools-isoimages for various operating systems. There is also /vmimages/floppies

which current contains a copy of the vmscsi-1.2.0.4.flp file There are two ways to initiate an in place upgrade either from booting

from the ESX 3.x CD or using a tar-ball method

Changes to ROOT Security

An upgrade ESX server has different security settings to a clean install A clean install of ESX 3.x denies root access using SSH, whereas an

upgrade inherits the security of the ESX 2.x The default in ESX 2.x allowed access to root for SSH, unless you changed

/etc/sshd/sshd_config from #PermitRootLogin yes to PermitRootLogin no In a clean install, point the VI client to the ESX host, and create new user.

Then connect to the Service Console with this new user, and then use the

su command to elevate yourself to root or else configure and use thesudo command

This change was introduce to enforce auditing/traceability in the ServiceConsole

Confirming you can upgrade1. Insert your ESX 3.x CD into the ESX 2.x server2. Type:

perl /mnt/cdrom/scripts/preupgrade.pl

3.

This should give you a print out report of the status of the ESX server


21/123Pa e 21

Upgrading with CD-ROMWhat you need for this task

Erm, the ESX 3.x CD would be a good start!Note:In this upgrade I chose not to make any revisions to /boot or /root from the

recommended partition table for ESX 2.x. I would recommend a text mode option

if you working through an ILO/DRAC card, as not all of them handle mouseactivity equally well.

1. Shutdown your VMs on the specific host, and reboot2. Insert the CD3. At the boot prompt press [Enter]

Note:At this stage drivers are then loaded in the blue background text mode.

Ive noticed Ive sometime had to wait quite a bit this makes the

installer look like its hung but its just busy

4. Choose Skip to the CD-ROM media testNote:Ive done a media test once. I guess you should your not missinganything exciting

5. Click Next6. Choose your keyboard type, in my case United Kingdom

Note:The mouse pointer question are only relevant if you were installing X-Windows to the Service Console, which you never, never, never do. So as

in ESX 2.x mouse selection is largely academic

The installer will then search for an existing ESX installation

7. Choose Upgrade8. Accept the EULA9. In the Advanced Options page choose From a drive (Install on the

MBR of the drive)

Note:

This should be the hard-drive/LUN you originally install ESX to, in my case

it was /dev/sda

Note:

The installer will briefly search for packages to upgrade

10.Click NextNote:

The installer will transfer install image to hard drive; prepare RPMtransaction; start install processors, and then the main file copy begins

Note:


22/123Pa e 22

Now go to your kitchen, make a cup of tea, have a biscuit. Or perhaps eaton of those 5 portions of fruit you have promising to consume

Licensing ESX ServerNote:

Licenses served from the license server are in a pool, like in things likeCitrix servers draw from this pool, and can give back their licenses whenthey are decommissioned

Confirm you see the FQDN name of the license server in the VI Clientpointed at the VC Server

Confirm you have DNS resolution from the ESX host to this server When you license ESX you set its Host Edition as well, this indicates if

your using a Developer Edition (VMTN) or Enterprise Edition or whetherthe ESX host is to be unlicensed and its license added back into the pool

Licensing ESX

1. Before you begin, confirm the following:In your DataCenterSelect your upgraded ESX Host

Select the Configuration TabIn the Software Pane choose the Licensing Features link

Confirm the name of your license server

If this field is blank. Chances are there is a lack of DNS name resolution.Confirm that DNS is working from the ESX host and VirtualCenter server

2. Next to Host Edition, click Edit3. Set your Host Edition (dependant on your license file


23/123Pa e 23

Note:This should complete the window with Host Edition features (iSCSI, SAN,

NAS) and number of CPU licenses the ESX host will take (in my case justtwo)

4. Next to Add-ons, click Edit and add any licenses for add-on features


24/123Pa e 24

Note:

This doesnt have to be done now, and could be done later. This allowsyou to consume the add-on licenses as you need them. You might have astand-alone ESX host that is not part of VMotion or isnt suitable for Virtual

SMP.

Cost 1 License per CPU, mean a dual-box will take 2, and quad box will

take 4. Remember dual-cores are licensed like the Xeons on per-socketbasis.

The window will refresh VirtualCenter DRS and HA will be marked as

Not Used as we havent created any DRS or HA clusters yet.

Viewing your License Pool

Note:You will probably want to track your license usage/pool to see how many youhave left over. To do this:

1. Click the Admin button

2. Select the Licenses Tab3. To see each ESX hosts licenses used, select your host under the

Name section

Note:See how we only have 1 license for VirtualCenter 1 used and 0

remaining. That my single ESX host has taken 2 licenses from a pool of 32because of my 2 physical CPUs. The DRS and HA columns are unchangedbecause I have yet to configure DRS or HA

Upgrading VMFS-2 to VMFS-3Note:

Until you do this the effected VMs will not power-on


25/123Pa e 25

This is because ESX 3.x only had read-only access to a VMFS-2 volume

Another reason VMs might not power-on is that you have not acquired alicense effectively

Before upgrading to VMFS-3 you will have to place the ESX host inMaintenance Mode. This is management state which stops background

processes affecting the ESX host.

1. In the Inventory, select your ESX Host2. In the Summary Tab, Click Enter Maintenance Mode

3. and ChooseYes, to the dialog promptNote:

This can take sometime, afterwards the ESX host status changes to reflect

maintenance mode

4. Select the Configuration Tab5. In the Hardware Pane, choose Storage (SCSI, SAN, NFS)

6. Under Storage, select your VMFS-2 volume(s)7.

Underneath Storage in the Details Pane, select Upgrade to VMFS-3


26/123Pa e 26

8. Choose Yes, to confirm the Upgrade

Note:You do not currently see a status bar during the upgrade. You do get a

notification in the Recent Tasks view at the bottom of the VI Client:

This does seem to take quite sometime so be patient it is reliable.Once it has completed the Recent Tasks bar updates like so:

9. Exit Maintenance Mode withIn the Summary Tab, Click Exit Maintenance Mode

Note:Sometimes Ive found that the GUI still show the Upgrade to VMFS-3 link.

Generally a few refreshes with F5 will clear these

IMPORTANT:At this stage you can now power on a legacy ESX 2.x VM. So it not

mandatory to do a VMware Virtual Hardware/Tools upgrade to get your

VMs back online.

However, until you do the VMware Virtual Hardware Upgrade you will not

be able to edit the definition of your virtual machines.


27/123Pa e 27

If you wish you can skip the remainder of this Module, and proceed to

Module 4, where I cover the upgrade of existing VMs originally created in

the ESX 2.x platform

Relocate the Virtual MachinesNote:

Currently the Virtual Machine VMX file are still on Service Console storage This is either /home/vmware if it was created in VC 1.x or

/home/username/vmware if created via the ESX MUI

Until they are relocated they will not power on and VC will give you thiserror:

The VMs configuration files must reside on either SAN, iSCSI or NASStorage

1. Right-click your ESX Host2. Choose Relocate VM Files

Note:

You will see this status progress:

Update VMFS Volume Labels & DataStore Name forUniqueness

VMFS have a couple of IDs there Volume Label, the DataStore Label(normally derived from the volume label) and UUID value as well (whichnever changes)

The primary label used by VC is the DataStore label. It can be differentfrom the Volume Label, and most usefully as VMs VMX point to it ratherthan the volume label you can change a DataStore Label without having

to update the VMX file as we did under ESX 2.x

As I mentioned earlier in this guide VMFS DataStore names to be globallyunique to a DataCenter. So if every ESX server has a local VMFS volumecalled /vmfs/local, then the VC will use this name as the basis for the

Datastore name serialise them to keep them unique. You can observe thisphenomena in the DataStores view in the VI Client and elsewhere in the VIClient


28/123Pa e 28

I intend to label my local storage like this /vmfs/local-esxN and checkthat the VMX files on each of my VMs is updated so they will know wherethe VMFS partition is located

This was something I couldnt have done before the upgrades of the VMsVirtual Hardware and Tools, as the VMX file is locked until the upgrade

1. Select the ESX host and the Configuration Tab2. Under the Hardware Pane, choose Storage (SCSI, SAN, and NFS)3. Under the Storage heading, select your VMFS and choose Properties

4. In the dialog click Change button

5. Change the DataStore label to be something more unique [thanlocal, local(1), local(2)] such as local-esx1

Custom Upgrading with a Tar-Ball (Optional)What you need for this task:


29/123Pa e 29

Erm, the ESX 3.x tar-ball would be start!Note:

Why do your upgrades this way? Well, it allows for unattended upgrades without having to be physically present at the server or having to run

around from one server to another with CD!

This is a two-stage process involving two scripts and reboot betweenscript1 and 2 I copied the Tar-Ball to area of free-space this happened to be my

/vmimages location (I was concerned my /tmp partition might be too

small having set the Vmware recommended size of 512MB in ESX 2.x) Upgrading ESX in this way didnt seem to help with my virtual networking

I still had some machine which thought they were on the production

switch and others thought they were connected to a Legacy Bond0 even though I never had a bond.

Run First Script & Set ESX Boot Option

1. Transfer the Tar file to the ESX 2.x Host and using WinSCP2.

Logon to the Service Console with PuTTY, as ROOT

3. Untar the tar.gz file with:tar -xvzf esx-3.?.?-?????-upgrade.tar

Note:

This will take some time!

4. Change the boot option and Reboot ESXlilo R linux-upreboot

Note:When you reconnect to the Service Console, confirm you have not loadedthe Vmkernel. An easy check is to ls /vmfs. No Vmkernel, No VMFS

5. cd in to the esx-3.?.?-?????-upgrade directory6. Run the Perl script called

./upgrade.pl

Note:The system will verify files, and then ask you to read & accept an EULA!

7. Choose Q, to quit reading the EULA, and8. Type Yes [Enter] to accept the EULA

Note:You will receive status messages like so:

Upgrading packages (this may take a while) ....... done

Removing obsolete packages .... done

Upgrading system configuration ...


30/123Pa e 30

Running vmware-config.pl ... done

then lastly

*** You must now reboot the system into Service Console mode

and run upgrade2.pl ***

Reboot the server now?[y/n]

9. At the Reboot the Server now[y/n] choose N and [Enter]10.Now edit the boot.conf file to make sure the server boots to the Service

Console with:

nano w /boot/grub/grub.conf

change default=0 to default=1

11.Now do the reboot with the command:reboot

Note:You might find you loose connectivity to the Service Console. I did on my

first attempt. The second time I didnt get a script error and my Service

Console networking was intact. The interesting thing is the upgrade wasdone on an identical ESX 2.x host. I know it was identical as it was the

same physical machine cloned with ghost which contain an image of ESX.Perhaps these network errors were related to the boot errors?

Run Second Script & Revert to ESX Boot Option

1. Logon to the Service Console with PuTTY, as ROOTNote:You will know you have boot to do the Service Console because of thismessage.

ALERT [Wed May 17 21:19:45 GMT 2006]:This VMware ESX Server machine is currently booted into troubleshootingmode.

2. cd in to the esx-3.?.?-?????-upgrade directory3. Run the Perl script called

./upgrade2.pl

Note:

You will receive this status messages like so:

This script will complete upgrading your ESX server to version3.x.x

Verifying Files don

Upgrading Packages (this may take a while).

INIT: Version X.x.x reloading


31/123Pa e 31

done

*** You must now reboot the system into the ESX Server mode tofinish the upgrade ***

4. At the Reboot the Server now[y/n] choose N and [Enter]5. Now edit the boot.conf file to make sure the server boots to the ServiceConsole with:

nano w /boot/grub/grub.conf

change default=1 to default=0

6. Now do the reboot with the command:reboot

Note:

This will boot the ESX 3.x Vmkernel for the first time. The remote upgradeis complete.

Note:If you wish to automate the change to grub.conf, this script is from IBM iscalled bootcontrol.pl you can download the zip file directly here. The very

first time you run this command it returns an error which states "Use of

uninitialized value in numeric eq (==) at bootcontrol.pl line 99".

It does actually shift the boot option - and the error only occurs once. I'veemailed the guys who wrote it - but I have a feeling its cause by our

partially completed upgrade process. This command re-compiles the grubloader so that the Service Console Only (troubleshooting only) menu is

chosen. It does that by changing the value in grub.conf called Default 0 to

being Default 1

Upgrading to VMFS-3 from the Service ConsoleAcknowledgments:

I would like to thank Mostafa Khalil, VCP (SE) for his Troubleshootingpresentation at VMware TSX in Paris 2006.

Note: From man pages of vmkfstools.

VMFS-2 to VMFS-3 file system upgrade is a two step process. Before file

system upgrade can begin the vmfs2 and vmfs3 driver must be unloaded

and the auxiliary file system driver, fsaux, should be loaded. The first stepof upgrade uses the -T option. Once the first step completes, the auxiliary

file system driver, fsaux, is unloaded and, vmfs2 and vmfs3 drivers arereloaded. The second step of file system upgrade makes use of the uoption.

-T, --tovmfs3

converts a VMFS-2 file system on the specified partition to VMFS-3 format,

preserving all files on the file system. The conversion is in-place and theauxiliary file system driver (fsaux) module must be loaded. The ESXServer file system locking mechanisms will try to ensure that no local

process or remote ESX Server is currently accessing the VMFS file system


32/123Pa e 32

to be converted. The conversion is a one-way operation and once the

VMFS-2 file system is converted to VMFS-3 file system, it cannot be rolled

back to VMFS-2.

-u, --upgradefinish /vmfs/volumes//

once the first step of file system upgrade has completed (using -T), thevmfs2 and vmfs3 modules are reloaded and the -u option is used to

complete the upgrade

1. Logon to the Service Console as ROOT2. Unload the vmfs2 driver with:

vmkload_mod -u vmfs2

3. Unload the vmfs3 driver with:vmkload_mod -u vmfs3

4. Load the FS Auxiliary Driver with the upgrade functionvmkload_mod fsaux fsauxFunction=upgrade

5. Run the first stage of the upgrade withvmkfstools -T /vmfs/volumes/local -x zeroedthick

Note:

-x zeroedthick (default) . Retains the properties of VMFS-2 thick files. Withthe zeroedthick file format, disk space is allocated to the files for future

use and the unused data blocks are not zeroed out.

-x eagerzeroedthick . Zeroes out unused data blocks in thick files during

conversion. If you use this sub-option, the upgrade process might takemuch longer than with the other options.

-x thin . Converts the VMFS-2 thick files into thin-provisioned VMFS-3

files. As opposed to thick file format, the thin-provisioned format doesn't

allow files to have extra space allocated for their future use, but insteadprovides the space on demand. During this conversion, unused blocks of

the thick files are discarded.

Note:This will give you the following message

I got /vmfs/volumes/44a38c72-156b2590-be15-00065bec0eb7

VMware ESX Server Question:Please make sure that the VMFS-2 volume /vmfs/volumes/44a38c72-

156b2590-be15-00065bec0eb7 is not in use by any local process or anyremote ESX server.

We do recommend the following:

1. Back up data on your volume as a safety measure.2. Take precautions to make sure multiple servers aren't accessing this

volume.


33/123Pa e 33

3. Please make sure you have approximately 1200MB of free space on

your volume. Note that the number is an upper bound, as the actual

requirements depend on physical layout of files.

Continue converting VMFS-2 to VMFS-3?

0) Yes

1) No

6. Type 0, and press [Enter]Note:Currently, you get no status information while this is proceeding. It should

complete with this message:

Filesystem upgrade step one completed. Step two must be completed after

the vmfs2 and vmfs3 modules are reloaded. When ready, run 'vmkfstools-u /vmfs/volumes/local' to complete the upgrade.

7. Once this part has completed confirm that your format is VMFS-3vmkfstools -P /vmfs/volumes/local

Note:

This should report something like this:

VMFS-3.21 file system spanning 1 partitions.

File system label (if any): localMode: public

Capacity 36238786560 (34560 file blocks * 1048576), 5511315456 (5256blocks) avail

UUID: 44a38c72-156b2590-be15-00065bec0eb7

Partitions spanned:vmhba0:1:0:1

8. Confirm you files havent disappeared in the process with:ls l /vmfs/volumes/local

9. Next, unload the Auxiliary Files System Driver and reload youVMFS2 and VMFS3 driver with:

vmkload_mod -u fsaux

vmkload_mod vmfs2vmkload_mod vmfs3

Note:

You should get Module load of vmfs2succeeded and Module load of

vmfs3succeeded

10.Restart the hostd service for these changes to be reflected in the VI clientwith:

service mgmt-vmware restart


34/123Pa e 34

Fixing the Service Console Network Problems Sometimes after an upgrade you find you have lost Service Console

networking

Why does this error occur? I dont know I think the Service Console is unwell But sometime its as

if the upgrade miss-assigns the NICs to the switch other time its simplynot present in the system at all!

Normally, any deviation from common defaults is the primary cause in anyOS upgrade. But my upgrades of ESX 2.x are standard, and do NOTdeviate from defaults in ESX 2.x

Anyway to fix this you must be at the physical console (for obviousreasons)

At the recent Paris TSX there was slide on this, but I found the recoveryinstructions didnt work for me. So I had to go in and fix it myself

First Example Legacy eth0 absent

Note:

In this example I discovered I had no switch with Legacy eth0. There was no swift either In some respects this was easier all I had to do from the command-

line was create a new switch and swift interface

1. Logon to the Service Console as ROOT (using the standard ESX bootoptions)

2. To list your current switch configuration typeesxcfg-vswitch l

Note:This will give you a print-out something like this

Switch Name Num Ports Used Ports Configured Ports Uplinks

vSwitch0 32 2 32 vmnic0

PortGroup Name Internal ID VLAN ID Used Ports UplinksLegacy vmnic0 portgroup1 0 0 vmnic0

vmotion portgroup0 0 0 vmnic0


vSwitch1 32 0 32

PortGroup Name Internal ID VLAN ID Used Ports UplinksLegacy vmnet_0 portgroup3 0 0

internal portgroup2 0 0


vSwitch2 32 2 32 vmnic2

PortGroup Name Internal ID VLAN ID Used Ports UplinksLegacy vmnic2 portgroup5 0 0 vmnic2

production portgroup4 0 0 vmnic2

Note:

OK, so what is this telling me. Well, I have 3 switches. One for vmotion

and one of production. There is an internal switch called internal. There is


35/123Pa e 35

no legacy eth0

I can see what Port Group vswif thinks it is using with the command:

esxcfg-vswif l

Name PortGroup

IPAddress

Netmask Broadcast Enabled DHC

To my horror, I discover this is no vswif0. To correct this I did the

following

3. Create a new vSwitch (switch3 in my case) and patch a spare physicalNIC (vmnic1 in my case) which was available

esxcfg-vswitch -add vSwitch3

esxcfg-vswitch -link=vmnic1 vSwitch3

4. Next create a Port Group within the vSwitchesxcfg-vswitch -add-pg=Service Console vSwitch3

5. Next create vswif interface mapped to the Port Group called ServiceConsole and set it IP and Subnet Mask

esxcfg-vswif -add vswif0 -portgroup Service Console-ip=192.168.2.102 --netmask=255.255.255.0

6. Restart hostd to make VI Client reflect new configurationservice mgmt-vmware restart

Note:This will result in a response like this

Stopping VMware ESX Server Management services:

VMware ESX Server Host Agent Services [OK]

VMware ESX Server Host Agent Watchdog [OK]VMware ESX Server Host Agent [OK]

Starting VMware ESX Server Management services:VMware ESX Server Host Agent (background) [OK]

Availability report startup (background) [OK]

Second Example Legacy eth0 is presentNote:In this case I had 4 nics, but the first wasnt recognised by ESX 2.x, but was

recognised by ESX 3.x. I believe location of legacy eth0 became confused. Byswapping the NICs about I was able to get Legacy eth0 up

1. Logon to the Service Console as ROOT (using the standard ESX bootoptions)

2. To list your current switch configuration typeesxcfg-vswitch l


36/123Pa e 36

Note:

This will give you a print-out something like this

Switch Name Num Ports Used Ports Configured Ports UplinksvSwitch0 32 2 32 vmnic0

PortGroup Name Internal ID VLAN ID Used Ports UplinksLegacy vmnic0 portgroup1 0 0 vmnic0vmotion portgroup0 0 0 vmnic0

Switch Name Num Ports Used Ports Configured Ports UplinksvSwitch1 32 0 32

PortGroup Name Internal ID VLAN ID Used Ports Uplinks

Legacy vmnet_0 portgroup3 0 0internal portgroup2 0 0



Legacy vmnic2 portgroup5 0 0 vmnic2production portgroup4 0 0 vmnic2



Legacy vmnic3 portgroup6 0 0 vmnic3


vSwitch4 32 1 32

PortGroup Name Internal ID VLAN ID Used Ports UplinksLegacy eth0 portgroup7 0 1 vmnic2

Note:

OK, so what is this telling me. Well I have four switches. This is quite odd.Actually originally only had 3 (production, internal, vmotion) and threeNICs one assigned to the Service Console, production, vmotion. My 4th

network card was a new one not recognised by ESX 2.x but is recognised

by ESX 3.x. interestingly, it looks as if ESX 3.x has automatically assignedit to a vSwitch. Something I wasnt expecting

Anyway originally the Service Console was using eth0 now it looks likevSwitch4, with the port group of Legacy eth0 is using vmnic2. I think itshould be using vmnic0 (the first NIC found in the PCI bus?)

I can see what Port Group vswif thinks it is using with the command:

esxcfg-vswif l

Name PortGroup

IPAddress

Netmask Broadcast Enabled DHC

vswif0 Legacy

eth0

192.168.2.102 255.255.255.0 192.168.2.255 true false


37/123Pa e 37

3. I think I will try moving the NICs aroundesxcfg-vswitch unlink=vmnic0 vSwitch0esxcfg-vswitch unlink=vmnic2 vSwitch4

esxcfg-vswitch link=vmnic2 vSwitch0

esxcfg-vswitch link=vmnic0 vSwitch4

Note:If all else fails assign all your NICs to the vSwitch containing the

Legacy eth0 Port Group, and that will get you back into the VI Client

4. Restart hostd to make VI Client reflect new configurationservice mgmt-vmware restart

Clean Installation of ESX 3.x (Optional)Note:

For this guide, Ive not bothered to cover a clean install of VC 2.x, thatsvery much like an VC 1.x install

On the other hand an ESX 3.x install is significant different to a ESX 2.xinstall

Installing the ESX Operating System

1. Boot to the ESX 3.x CD2. Skip the CD Media Test

Note:

You should MD5Sum the ISO before you burn, and double-check themedia. Alternatively, you can jump in with both feet and face the

consequence later. I will let you be the judge. If your in a test & devenvironment its not so very thrilling to do! If it is successful is says It isOK to install from this media which is reassuring.

3. Click Next4. Select your Keyboard Type5. Select your Mouse Type6. Agree the EULA

Note:Confirm the warning about the /dev/sda being unreadable. It should be

blank already on a clean install

7. Confirm that under AdvancedNote:If you have wiped the Service Console of ESX 2.x but kept the VMFSvolumes, enable X Keep virtual machines and the VMFS that contains

them. This is one upgrade strategy of re-install Service Console but keep

your VMFS.

8. This is my recommendation for manual partitioning my deviationsfrom VMware are in red and are combination of what I think is best from

the documentation, automatic partitioning and recommendations from ESX2.x.


38/123Pa e 38

The black and bolded numbers come from VMwares guide

Notice there is no reference to /vmimages. As stated earlier in the guide itis no longer required. Space required without VMFS is about 12GB

File

System

Fixed

Size

Size

in MB

Fill

To

Force to Primary

/boot ext3 X100/250 X

n/a swap X 544/1600 X

/ ext3 X2560(min)/5120 ??? X

/var/log ext3 X2048/2048

/tmp ext3 X NA/2048

NA vmkcore 100/100

/storage1 vmfs X

Note: ???VMware suggest allowing / to fill to the maximum of the disk. If you

choose this option you will have to use external storage for your VMFS and

vmkcore. Plus this might make creating separate partitions for /var/log

Note: /tmpIn the past VMware have recommend using a separate partition for /tmp

which I have always done in ESX 2.x as well. For some reason thisrecommendation was dropped in the final GA documentation

Note: vmimages

If you are doing an upgrade your /vmimages partition is retained is

browse-able from with the virtual machine.

If you have done a clean install even if you havent created a /vmimages(VMware state is no longer required) you will still have directory called

/vmimages which contains the ISO for VMware Tools and Floppiesdirectory for the VMware SCSI Driver (replacement of BusLogic)

Note: /homesVMware no longer state you need to create a separate partition for /home.This is most likely due to the fact that VMX and other VM configuration

files are no longer stored here but on a VMFS volume

Note: VMFS

Storage1 is a default volume label if you do not set one yourself. A VMFS

volume is no longer required for a swapfile - the per-ESX host swapfile hasbeen replaced with a per-VM swapfile. Your only need for local VMFS is ifyou don't have iSCSI/SAN/NAS based external storage

I would not recommend creating VMFS volumes here - they areautomatically labelled storage1, 2 and so on. For this reason you might

prefer creating VMFS partitions until later when you will have more

control. One thing I have noticed if you do have free space at the end ofthe LUN to where you installed the Service Console, you can't VMFS upthis free space with the VI Client. It's almost like the VI client expects to

see an empty LUN within which to put VMFS. So you either do it in the

install or resort to fdisk to get this space.

Note: Swap/Var/Tmp

I decided to over-allocate the Swap Partition, using the old values of max

Service Console memory of 800, and doubling the Swap Partition. I


39/123Pa e 39

decided to make the /tmp the same size as /var for consistency. Ive

never allowed /tmp to just be directory of /. I think this over-allocation is

fine, as we have lost requirement for vmimages, it has left me with lots offree disk space.

Note: Automatic Vs Manual

Automatic partitioning creates 100MB /boot not a 250MB as isrecommended in the Beta/RC1 guides.

They make root 5GB not 2560 which is recommended only in the Vmware

guides as workable minimum.

Automatic partitioning creates separate partition for /var/log of 2GB.

There doesnt seem to be any recommendation on /var/log and /tmp in

the Beta PDF guides

9. In the Advanced OptionsChoose SCSI Disk sda, only choose SAN LUN if you doing SAN basedbooting.

Note:

The other options arent relevant to us. Force LBA32 would allow /boot to

above 1024 cylinder limit normally enabled on legacy hardware if yourBIOS supports it. The From a Partition again is for legacy hardware that

stores BIOS information in the MBR in a partition like some old Compaqshave a EISE partition

10.Network ConfigurationSelect your NIC which will be used by the Service ConsoleSet your IP/SN/DG/DNS and FQDN

Set the VLAN ID, if your using VLANDisableX Create a default network for virtual machines

Note:

If you enable X Create a default network for virtual machines, this creates

a Port Group called VM Network attached to NIC0, the same NIC used bythe Service Console. You probably want to dedicated vSwitch/NIC to the

Service Console, and use your other NICs for the Virtual Machines

Remember the Service Console is just patched to a vSwitch Port Groupnow, technically it doesnt really use eth0 as did ESX 2.x. More about this

in the next module.

11.Set your Time Zone, and if appropriate UTC SettingsNote:

If your in the UK like me. We dont strictly obey UTC. We obey GMT andBST. If you enable UTC in and select the Europe/London you system clock

will be 1hr a drift (depending on the time of the year)

12.Set the root passwordNote:

There is no method to create ESX users within the installer this has to be


40/123Pa e 40

done with the VI Client pointed not at VirtualCenter but the ESX 3.x Host

itself

Adding an ESX Server into VirtualCenter

1. Right-Click the DataCenter name, in my case RTFM DataCenter2. Choose Add Host or click this icon

3. Complete the dialog as appropriate

4. Next your way through the dialog boxesNote:

These dialog boxes dont do much for us. As a blank ESX box has no

virtual machines, and we dont have VHA or DRS Clusters created just yet

Note:

You are not asked to set VMotion settings here. It is enabled elsewhere

Note:

Remember to set the Host Edition and License Add-ons.

Creating local ESX User Account for SSH/PuTTy Access

Note: In clean install, by default there is no access to the ESX Service Console

for root via SSH However during the installation you no longer have the ability to create

users. So this is appears like catch-22 situation

Heres how you should give root access to the Service Console1. Close the VI client if opened on your VirtualCenter2. Re-open the VI client and login to the ESX Host

Server: type the name of your ESX HostUsername: rootPassword: **********

Note:You may receive a notification that the VirtualCenter server is also

managing this host. Just click OK, to this warning:


41/123Pa e 41

3. Click the User & Groups tab4. Right-Click the Window and Choose Add5. Fill in the dialog box like so:

Note:You must enable X Grant shell access to this user for this user to have

access to the Service Console via Shell/PuTTy. Otherwise they will only

access to the VI Client pointed at this ESX host.

6. Click OK7. Open a SSH/PuTTy Sessionto the ESX Host8.

Login with you under-privileged account


42/123Pa e 42

9. To levitate to higher plain of privileges type:su -

Note:

This Switches User and assumes root, unless otherwise specified. The

takes roots environmental settings (very important if you want to run anycommands properly)

Note:

Some applications do not support levitation to a higher plain for exampleyou WinSCP. Sure you could use WinSCP to gain access as an ordinary

user, but then you might lack permission to copy the files you need. If you

try to logon as root, WinSCP will give you access denied.

If this upsets you and causes your brain to melt out of your ears you could(gasp/horror) lower the security to allow root access. This also has the net

effect of undermining auditing and traceability so important to our friendsSobarnes & Oxley. If only the governments of the world were so

concerned with devaluation of pension funds and the fact that my pensioncompany put 10 years on my working life. I guess Sorbarnes & Oxleywould protect me from a company going into administration and theadministration a billion pound whole in a pension fund. Anyway, I digress

Anyway, security/auditing/traceability is important so proceed withextreme caution on this one

nano w /etc/ssh/sshd_config

Locate: PermitRootLogin noPlace a # in front of PermitRootLogin no like so: #PermitRootLogin no

Exit Nano & Save the file

Restart sshd with service sshd restart

Enabling SSH/SCP from your ESX host to otherhostsNote:

One of the thing I like to do is connect to one ESX host, and then use theSSH at the Service Console to get to my other servers this saves mehaving open repeated puTTy sessions

This is not allow in ESX 3.x as the firewall denies the client (althoughevery ESX host is an SSH server

Also you will unable to use the SCP command to copy files to ESX serversfrom an ESX server

ssh: connect to host esx2.rtfm-ed.co.uk port 22: Connection refused

To enable this kind of access we need to adjust the firewall settings1. In the Inventory, select your ESX Host2. Click the Configuration Tab3. Under the Software Pane, select Security Profile


43/123Pa e 43

4. Click the Properties link in the far right-hand corner5. Enable X SSH Client and Click OK

Cold Migration from ESX 2.x to ESX 3.x

Note: If you an ESX 2.x server you can cold migrate the VMs from it to an ESX

3.x host

You could use the move option to take the VMs files off a VMFS2 toVMFS3 partition as part of a migration strategy

So you upgrade VC, and install/upgrade an ESX 3.x server, format a VMFSvolume and cold migrate VMs from ESX 2.x to 3.x start the upgrade on

ESX 2.x and cold migrate them back to the original server so if you like

the ESX 3.x server and it storage is a helper in the migrate processWhen you have two ESX 3.x server looking at the same storage you

could be VMotioning them back

1. Right-click a VM on a ESX 2.x host in the VC 2.x Inventory2. Choose Migrate3. Select an ESX 3.x host from the list

Note:The system will try validate the ESX host and give you this dialog box:


44/123Pa e 44

4. Click Next and Next again5. Choose Move virtual machine configuration files and virtual disk6. Select a VMFS3 datastore from the list and Finish

Note:

After the move I was able to power on the VM without having to gothrough the next stage of upgrading the virtual hardware and Vmware

Tools


45/123Pa e 45

Module 4: Upgrading Virtual Machines

Executive Summary Upgrading Virtual Machine hardware depends on whether you have chosen

an Upgrade or Migrate Strategy

If you Upgraded from VMFS-2 to VMFS-3 the VMs hardware isautomatically upgraded from VM-2 to VM-3

If you Migrated from VMFS-2 to VMFS-3 or import old ESX 2.x virtual diskyou will have to upgrade their virtual hardware from VM-2 to VM-3

This can be done individually, or in bulk using a command-line tool(W2K/3/XP Only)

Whats new VM-3?

Snapshots of VMs state, kind of saving as your go along, discard inreverse order method.

Think of it as multiple undo levels like you would have in Word or Excel,but on an entire VMs system state (disk and memory)

16 GB of RAM 4-CPUs (with the license) Improved network performance especially for W2K3 Hot-plug virtual disks for OSes that support it such as W2K3 Support for silencing the file system within a VM for VMware Consolidated

Backup (VCB)

Whats new in VMware Tools

Upgrade can be done manually or by CLI as mentioned earlier Vlance NIC can metamorphosis into vmxnet, which make KB ??? a thing

of the past

Driver improvements for W2K/3

Drivers will be signed by Microsoft by time the product goes into GA.Currently, they are not and this stops a bulk upgrade of VMware Tools

from being successful.

Consequences of not upgrading VM Hardware & Tools

Snapshot feature does not work VM-2 limits persist (3.6GB RAM/2-vCPUs) No hot-plug disks Important:

Can edit a VMs properties from VC2.x but can manage it from the MUI

Upgrading Virtual Machine Hardware (GUI)Note: This upgrade must be done, for you to be able to modify the properties of

the VMs VMX file

You can do this by multiple selection of VM within the Virtual Machine tabof an ESX host. But you would still have to do a Vmware Tools upgrade

and interact with the guest OS. So if bulk upgrades and automation is yourgoal you need to look at the vmware-vmupgrade tool on the hard-drive

of the VC, which covered later in this module

Warning: After the upgrade of virtual hardware I found my VMs VMX fileshad lost their switch, and it was set to nothing. I had to manually tell each

VM which switch to use. This happened in the RC1 but not in the Beta.


46/123Pa e 46

1. Right-Click a VM in the Inventory2. Choose Upgrade Virtual Hardware3. Choose Yes to the warning:

Note:This operation is very quick, nonetheless you do get status information

Upgrading VM Tools (GUI)1. Power on your VM

2. and Open a Console Session

Note:

If this is the first time you have powered the VM, and you have not changethe virtual disk extension from the old format (.dsk circa ESX 2.0) to vmdk(circa ESX 2.5), then you will get this dialog:

This message ONLY appears AFTER you have upgraded your VirtualHardware and if you are using the older .DSK extension

Click OK!

3. Logon on to your VM with Administrative/Supervisor/Root privileges depending on your guest OS


47/123Pa e 47

4. From the usual places, choose Install VMware Tools and confirmthe pop-up dialog box:

5. Next your way through the dialog boxes, and choose Yes to Reboot theVM

Upgrading VMXnet to Flexible Network AdapterNote:This was a procedure in the ESX Beta. But has now been incorporated into the

upgrade of virtual hardware

Upgrading VM Hardware & Tools for Bulk (CLI)Note:

This bulk process upgrades the virtual hardware, powers on the VM,injects VMware Tools silently, and then shutsdown the VM

I would recommend monitoring the whole process from the CMD prompt If you have kept some disks using the old .DSK extension. This bulk-

upgrade will stall asking you if its OK, to rename the .DSK file to .VMDK.

There does seem to be a time out on this question and the VMs do

power-up.

If you have user questions like my one about the DSK/VMDK file the toolwill take the default answer, and processes it to make sure it doesnt stall

for lack of user input. The kind of response you get is

RTFM DataCenter/esx1.rtfm-ed.co.uk:dc1:

Question -msg.disk.ConvertExtension.ask:

VMware ESX Server uses the new file extension '.vmdk' for virtual disks to

avoid conflicts with files from other programs. Your configuration currently

has at least one virtual disk with the older extension. We highlyrecommend that you allow it to be updated automatically. Select OK to

update all configured disks or Cancel to keep the current file extensions.If you select Cancel you will not see this dialog again.

I recommend not having any remote console windows open as I haveseen VirtualCenter crash when answering questions

This tool upgrades the virtual hardware, then powers on and upgradesVmware Tools

W2K/3/XP only, no support for NT4 or any other guest OS currently You can do the upgrade per-VM or per-Host, I will demo a per-host

method

You have options to limit the number VMs powered on at any given time;how long a VM is allowed to be powered on if the tools fails to power off

the VM in a timely fashion; you can also ask to skip the Vmware Toolsupdate


48/123Pa e 48

Lastly, I tried this on an upgraded ESX host from 2.5.2. After the upgrademy VMs all believed they were on legacy bond0. Trouble is I never had

a bond! So I had to manually tell them they were on the productionnetwork. Additionally, I found this error was NOT consistent. Out of my 5VMs one believed it was on legacy vmnic0

1. Power-down all your VMs2. Connect to your VirtualCenter Server Desktop3. Open a CMD prompt and navigate to C:\Program

Files\VMware\VMware VirtualCenter 2.0

4. Type the following command (replacing variables where appropriate,command is word-wrapped for readability)

vmware-vmupgrade

-u rtfm-ed\lavericm-admin-p ********

-h "RTFM DataCenter/esx1.rtfm-ed.co.uk"

Note:Note that you forward slashes to indicate the host /

Typical Errors:

Logon errors

will be marked with this error message Failed to connect to VirtualCenterserver: vim.fault.InvalidLogin

Path errors such as

Failed to upgrade: failed to find object at Hosts & Clusters/RTFMDataCenter/esx1.rtfm-ed.co.uk. In this case, I was specifying the built-in

container called Hosts & Clusters when I should have just specified the

DataCenter as in the example

Failed to upgrade: failed to find object at RTFM DataCenter\esx1.rtfm-ed.co.uk. In this case I had the slash \ the wrong way round /

Note:

I was going document what happened. But it got too lengthy. Basically,

the CLI connect VC, enumerates VMs, upgrades virtual hardware, powerson and installs Vmware Tools and then Powers Off the VM


49/123


50/123Pa e 50

After clearing house, my networking looked a bit nicer to look at:

Important

I am done with the in place upgrade. From now on, I will assume you have clean installation

General Overview

Improvements in Networking open a new door of possibilities in ESX, andmake certain network tasks much simpler than they have been in the past

We used to think of Switch as vmnet/vmnic/bond you can still havethese formats for networking

But now conceptualise this based on which component uses that switchand what for. This also breaks up into three types:

oService Console Traffic

o VMkernel (for VMotion/IP Storage) Traffico Virtual Machine Traffic

The Service Console is increasingly becoming a virtual machine (albeit it isnot shown as such in the VI Client)

There is no vmkpcidivy, as ESX 3.x owns all the hardware. Hurrah! Service Console no longer hogs NIC0 from the PCI Bus Instead it uses a vmkernel driver, and can be used by vmkernel as well.

This is very much like the use of vmxnet_console in ESX 2.x used by some

people who have blades with a limited number of network interfaces Virtual Switches still exist in there vmnet, vmnic, and bond formats. Their

configuration is no longer in the files that you knew in ESX 2.x in

/etc/vmware


51/123Pa e 51

If you wish to make vSwitches from the Service Console use the new esx-cfg command

As the vmkernel now has a full IP stack vSwitches are used for IP storagesuch as NAS and iSCSI.

You can still create special vmkernel switches for such things as VMotion It is still recommended that you use a dedicated NIC for the Service

Console, VMotion, and now IP Storage ESX now has a IP firewall which is configurable through the GUI or

command-line. This is because there is now a full IP-stack on ESX, andbecause VMwares customers demanded one

Whats new in vSwitches

32 ports default, configurable for up to 1016 ports Load-balancing options such as out-mac, out-ip and standby are

configurable through the GUI

Port Groups are not just used for VLAN, but allow you break up a vSwitchinto smaller management units. These management units can within avSwitch can have different rules and regulations imposed on them so

think of Port Groups as a way of applying different network policiesimposed on which VM is patch to the port group Theoretically, one vSwitch can have many NICs with many Port Groups

each port group configured for the Service Console, vmkernel IP Storage,

VMotion, and Virtual Machines. Although this is possible we are muchlikely to want to maintain physically separate vSwitch by NIC card

Remember all this work can be done with the VI client throughVirtualCenter, you no longer have point to the ESX server to do this oraccess a MUI

Creating an Internal Only Switch1. Select your ESX Host in the Inventory2.

Select the Configuration Tab3. In the Hardware Pane, select Networking

4. Click the Add Networking link5. Choose Virtual Machine and Click Next6. Remove the X next to any network adapters

Note:

Notice when you do this the preview window updates and states under

physical adapters the words No Adapters. Notice also how the ESXkernel does some clever sniffing to tell you about IP ranges to assist youin choosing the correct NIC for the right subnet/vlan


52/123Pa e 52

7. Click Next,8. In the Port Groups Properties dialog, type a friendly name for this

connection, such as internal

Note:Notice the option to specify a port group not significant in this case

9. Click FinishCreating a NIC Team vSwitch

1. Select your ESX Host in the Inventory2. Select the Configuration Tab3. In the Hardware Pane, select Networking4. Click the Add Networking link5. Choose Virtual Machine and Click Next6. Select two or more NICs

7. Click Next,8. In the Port Groups Properties dialog, type a friendly name for this

connection, such as production

Creating a VMKernel Switch for VMotionNote:

Configuring a VMKernel switch for IP based storage will be covered in thenext chapter


53/123Pa e 53

1. Select your ESX Host in the Inventory2. Select the Configuration Tab3. In the Hardware Pane, select Networking4. Click the Add Networking link5. Choose VMKernel and Click Next

Note:

In this case 3 of my 4 NICs are assigned (1 to Service Console, 2 toProduction, 0 to Internal) The dialog gives me the option to take NICs

assigned to other switches if I wish

6. Click Next7. In the Port Groups Properties dialog, type a friendly name for this

connection, such as vmotion8. Enable X Use this port group for VMotion9. Set an IP Address and Subnet mask for VMotion

Note:I am going to use 192.168.2.201/24 for esx1.rtfm-ed.co.uk, and

192.168.2.202/24 for esx2.rtfm-ed.co.uk. I have one big gigabit switchinto which everything is plugged. In the ideal world VMotion and VMKernelIP Storage and Service Console would all be on separate discrete networks

for performance and security

10.You will then get this message:

Note:At this stage Ive been told VMware have no intention of allowing VMotion

across routers. However, a VMKernel port like this could be used to access

iSCSI and NFS devices. In that case you might need to cross a router. Forpeace of mind, I always choose Yes and enter my details.


54/123Pa e 54

11.In my simple lab network I use the same router for ALL of my networktraffic, so I make the VMkernel and Service Console default gateway the

same value.

Note:

You might like to confirm the DNS configuration tab. DNS is required forVMware High Availability and if the settings here are incorrect thenyou could have problems later. This is what my dialog box shows

Note:Confirm that VMotion is now enabled (this a combination of a license and

VMkernel vSwitch enabled for VMotion) in the Summary tab of you ESXhost(s)


55/123Pa e 55

Increasing the Number of Ports on vSwitchNote:

ESX 2.x were only allow 32 virtual machines per vSwitch The default is now 56 The maximum you ports you can have is 1016 You can increase the number of fixed numbers from 24, 56, 120, 248,504

and 1016

Warning: Changing this value requires a reboot of ESX for it to take effect1. Select your ESX Host in the Inventory2. Select the Configuration Tab3. In the Hardware Pane, select Networking4. click Properties for you next to switch which contains the Port Group

labelled production (in my case this is vSwitch2)

5. In the dialog box click the Edit button

6. Under the General Tab, click the pull down list for the number of ports:


56/123Pa e 56

7. Click OKNote:The message behind reads, Changes will not take effect until the system

is restarted

Setting Speed & Duplex1. Select the Configuration Tab2. In the Hardware Pane, select Networking3. click Properties of vSwitchN4. Click the Network Adapters Tab

5. Select the NIC adapter and Click the Edit button

6. Select the Speed & Duplex and click OKNote:

You can now do this with the Service Console NIC without need to edit/etc/modules.conf


57/123Pa e 57

Note:

Remember IEEE Standards now recommend the gigabit-to-gigabit

connects be set to auto-negotiate

Managing Service Console NetworkingImportant:

OK, well on my 4-NIC box all my NICs are fully allocated If I want to show/configure other possibilities, I might have to free up

NICs (say removing one nic from production nic-team or by removing the

VMotion switch I just created to free up its NIC)

This is what my cleanly install ESX server looks like now from anetworking perspective

Creating a Switch for the Service Console

Note: Theres always been a single point of failure on the Service Console

network in ESX

Unless you were prepared to edit files and make the Service Console partof your virtual machine network

If you have free network adapters, its very easy to protect the ServiceConsole network by adding another Service Console switch

1. Select your ESX Host in the Inventory2. Select the Configuration Tab3. In the Hardware Pane, select Networking4. Click the Add Networking link


58/123Pa e 58

5. Select Service ConsoleNote:As we already have Service Console port, the system labels this secondservice console port as Service Console 2

Click Next

6. Complete the IP Settings as befits your network infrastructure

7. Click NextNote:This appears as another vSwitch

Confirm you can connect to this new port with ping/PuTTy and by pointingthe VI client to this IP/Name.

Note:As another way of protecting the Service Console network try justadding an additional vmnic to it by doing so you will create bond for the

Service Console. However, this configuration is set as Standby switch, not


59/123Pa e 59

for MAC or IP load-balancing. This is also a safe procedure to switch the

NIC on the Service Console switch. Add in the 2nd NIC and remove the 1st

NIC and shouldnt loose connectivity to the Service Console.

Changing the Service Consoles Network Settings

Note:

Were going to modify the primary connection for the Service Console This is likely to cause a disconnection If this change goes horribly wrong, it is not the end of the world we will

be able to use the backup Service Console 2 network connection

You havent got a backup connection and proceed with this task youwill get this warning message:

The safest method is to connect with your backup Service Console 2network to the ESX host, and then change Service Console Properties (this

way you dont get disconnected in the process of changing the IP!)

1. Close your VI Client2. Open the VI Client using the Backup Service Console IP

3. Select the Configuration Tab4. In the Hardware Pane, select Networking5. Select Properties ofvSwitch0/vswif0 with the Port Group name of

Service Console6. In the dialog box, select the Port Group of Service Console, and click

Edit


60/123Pa e 60

7. In the Service Console Properties Dialog box you can change your IPsettings

Creating a vSwitch with VLAN SupportNote:

As in ESX 2.x Port Group can be used for VLAN support Where multiple VMs connect to a single switch, divided into Port Group

which represents the different VLANs available

1. Select the Configuration Tab2. In the Hardware Pane, select Networking3. click Properties ofvSwitchNhost your production Port Group,

my case this is vSwitch24. In the vSwitchN Properties dialog, click the Production Port Group,

and click the Remove button

5. Then click Add, and in the Add Network Wizard, choose VirtualMachine

6. In the Port Group Properties dialog, type a friendly name like AccountsNetwork

7. In the VLAN ID (Optional) field type: 95Note:

Of course, this is specific to your implementation


61/123Pa e 61

8. Click Next and Finish, repeat this for other VLAN IDsNote:Confirm your VMs are patched into the correct network. For illustrationpurposes I patched some different VMs into these VLANs for the screendump below:

Note:

This VLAN system was setup on an upgrade ESX system. Im worried thatdespite setting the VLAN number the UI show VLAN ID *. This normallyindicates no VLAN tagging has been set. In a clean install this didnt

happen

Conclusion This section on network is by no means definitive Its just meant to get you up and running


62/123Pa e 62

Ive not decided to go through ALL options (and dialog box settings) thatwould take too long, as for another guide designed for newbie

As you have seen earlier with my tar-ball upgrade example you canmodify switches through the command-line using the esxcfg-vswitch andesxcfg-vswif utilities

But as one last illustration of the flexibility introduced to networking inESX. Heres a single vSwitch set-up with ALL features demod above withits companion internal switch. This is not recommended but if you had a

development machine with only one NIC this would be possible

At the end of this module I reset my systems back to 4 vSwitches (One forthe Service Console, Production, VMotion, Internal) This leaves 1 NIC left

out of my 4 (production not in a bond) so I can a have NIC free for aVMKernel IP Storage vSwitch covered in the next module (Im trying to

obey a separate NIC/vSwitch for each compo

ESX3.x VC2 Upgrade Guide

Documents

Transcript of ESX3.x VC2 Upgrade Guide