SUN Logical Domain Quick Start Guide
Transcript of SUN Logical Domain Quick Start Guide
Part No 821-0316-11Revision 1.1 1/13/10
SUN™ LogicaL DomaiNSQuick Start GuidePeter A. Wilson, Systems Technical MarketingGary Combs, Systems Technical Marketing
Sun Microsystems, Inc.
Table of contents
Sun™ Logical Domains Quick Start guide ...........................................................1
Straightforward use case ..................................................................................... 2
How this article is organized ................................................................................ 3
initial Server Setup .........................................................................................5
Before you begin ................................................................................................. 5
Gather documentation .................................................................................... 5
Assign IP addresses ............................................................................................. 5
Set up the service processor ................................................................................ 6
Obtaining the DHCP-assigned address .............................................................. 6
Assigning a static IP address ............................................................................ 7
Check firmware revision ...................................................................................... 8
Update system firmware ..................................................................................... 8
Configure the Solaris OS .................................................................................... 10
Command-line setup ..................................................................................... 10
Browser user interface setup ......................................................................... 10
Apply operating system patches ......................................................................... 11
Download the Logical Domains software ............................................................ 11
Install Logical Domains Software ....................................................................... 12
control and i/o Domain Setup ....................................................................... 13
Initial server state ............................................................................................. 13
Create a virtual disk server ................................................................................ 14
Create a virtual switch ...................................................................................... 14
Create a virtual console device .......................................................................... 15
Free CPU, MAU, and memory resources .............................................................. 15
Save the configuration in the service processor ............................................... 16
Virtual Disk Backend Setup ............................................................................ 17
Virtual disk options ........................................................................................... 17
Locate or create a disk partition ......................................................................... 17
guest Domain creation ................................................................................. 20
Allocate resources to a guest domain ................................................................. 20
Create the guest and allocate resources ............................................................. 21
Boot from the guest domain’s console ................................................................ 22
Use Solaris ZFS snapshots and clones ................................................................. 23
Housekeeping .............................................................................................. 25
Save configurations to the service processor ....................................................... 25
Use proper shutdown procedures ....................................................................... 25
Removing guest domains .................................................................................. 25
Removing Logical Domains software .................................................................. 26
Sun Microsystems, Inc.
Summary ..................................................................................................... 27
About the authors ............................................................................................. 28
Acknowledgments ............................................................................................ 28
Ordering Sun documents ................................................................................... 28
Accessing Sun documentation online ................................................................. 29
Sun Microsystems, Inc.1 Sun Logical Domains Quick Start Guide
.
Chapter 1
Sun™ Logical Domains Quick Start guide
Sun™ Logical Domains, or Sun LDoms, is Sun’s built-in, no-cost hardware
virtualization technology for servers and blade systems powered by UltraSPARC®
T1, T2, and T2 Plus processors. These systems are equipped for massive scalability
using chip multithreading (CMT) technology in which each of up to eight processor
cores support up to eight concurrent execution threads each for a total of up to
64 threads per processor and up to 256 threads per server. The systems based on
these processors deliver excellent performance for highly threaded applications,
along with the lowest power consumption per thread of the current generation of
processors.
LDoms allows server resources to be partitioned and allocated to virtual machines.
Resources that can be partitioned include individual CPU threads (virtual CPUs),
the processor’s cryptographic acceleration modular arithmetic units (MAUs),
memory, PCI root nexus nodes (or “PCI buses”), and network interface units (NIUs).
Architecturally, LDoms is a Type 1 hypervisor because the technology runs on a bare
machine and provides each domain with the illusion that it is running on its own
dedicated hardware — in this case hardware having the SPARC® sun4v architecture.
With LDoms, the control domain manages the hypervisor. I/O is performed through
virtual devices to one or more I/O domains that actually perform physical I/O on
behalf of a domain (these are sometimes referred to as service domains). Figure
1 illustrates a server with a control domain that manages the hypervisor, two
I/O domains that perform physical I/O, and three guest domains. In a typical
installation, a combined control and I/O domain performs both functions .
Hypervisor
Sun CMT ServerPCI A PCI B
LogicalDomain 1
Guest OSImage
ControlDomain
LogicalDomain
Manager
LogicalDomain 2
Guest OSImage
LogicalDomain 3
Guest OSImage
I/ODomain 1
Solaris™10 OS
I/ODomain 2
Solaris10 OS
Figure 1. Logical Domains supports multiple guest domains each having their own secure partition of server resources. Physical I/O is performed by I/O domains that ‘own’ one or more PCI root nexus nodes, or buses.
When this document was prepared, Sun™
Logical Domains 1.2 was the current version
and the document applies to versions 1.2
and greater. Take care to use the latest
recommended versions of LDoms, the
Solaris™ Operating System, and firmware.
Recommended sets of software can be
found in the Logical Domains Release Notes
for the current version.
Sun Microsystems, Inc.2 Sun Logical Domains Quick Start Guide
Where Logical Domains differs significantly from other Type 1 hypervisors on the
market today is that it allows physical server resources (such as CPUs, memory,
MAUs, and even PCI root nexus nodes) to be partitioned, not time shared, between
domains. This provides each domain with dedicated resources and increases isolation
between domains. Partitioning helps to avoid issues such as one domain depriving
other domains of needed resources. Partitioning potentially increases cache
performance by fixing the domain’s relationship to a CPU in contrast to timesharing.
While separating CPU and memory resources is prudent from a resource allocation
standpoint, LDoms also allows I/O resources to be shared in flexible ways to allow
both sharing and separation of resources to improve performance, reliability, and
availability.
Straightforward use caseThe purpose of this Sun BluePrints™ article is to provide a simple set of steps for
setting up and running a server with Logical Domains software. This guide is not
intended to be a comprehensive beginner’s guide to LDoms. Instead, it intends to
cover the use case of using Logical Domains in a laboratory environment for the first
time. Thus, we assume a minimal amount of supporting infrastructure, specifically a
desktop or laptop computer connected via network to the server. We do not require,
for example, the use of a network install server.
This article was originally written to describe Logical Domains 1.2 software. The
procedures shown in this document apply to version 1.2 and later, and they may
show references to Logical Domains 1.2 software. We recommend using the latest
version of Logical Domains software for your platform together with the latest
recommended versions of Solaris, patches, and firmware as described in the
applicable Logical Domains Release Notes.
Documentation for all Logical Domains versions is available from http://docs.
sun.com/app/docs/prod/ldoms.
We assume that your environment contains the following:
Server• . A Sun SPARC Enterprise® server based on UltraSPARC T2 or T2 Plus
processors and at least one internal disk drive.
Operating system• . The server runs the version of the Solaris™ Operating System
recommended in the Logical Domains Release Notes for the current LDoms
version.
Network environment• . Both the server’s lights-out management port and its
primary (NET 0) interfaces are connected to a network. The server and its service
processor ports are assigned static IP address.
Serial port• . The server’s serial port should be accessible from the workstation or
laptop either directly or through a terminal concentrator.
Sun Microsystems, Inc.3 Sun Logical Domains Quick Start Guide
Workstation or laptop• . A workstation or laptop is needed to access the server. It
should have a Web browser with a suitable Java™ software plugin.
These assumptions do not cover UltraSPARC T1 processor-powered servers such as the Note – Sun SPARC Enterprise T1000 and T2000 servers. This guide focuses on administering servers through Sun’s Integrated Lights-Out Management (ILOM) software using both its command-line interface and browser user interface (BUI). For UltraSPARC T1 server platforms that use Sun’s older lights-out management software, ALOM commands will need to be used instead.
How this article is organizedAs this article will demonstrate, it takes only seven commands to set up a guest
domain using Sun™ Logical Domains software using a given virtual disk image:
# ldm create guest1# ldm set-vcpu 4 guest1# ldm set-mem 4g guest1# ldm add-vnet vnet0 primary-vsw0 guest1# ldm add-vdsdev /ldompool/domain1/diskimagefile vol1@primary-vds0# ldm add-vdisk vdisk0 vol1@primary-vds0 guest1# ldm bind guest1
There is, however, some preparation needed to set up a new server, update firmware,
install Logical Domains software, and set up virtual devices and disks. That is the
bulk of this article that takes you through the steps from providing a server’s initial
network identity to booting your first logical domain. Although many of these steps
can be condensed by using the Logical Domains Configuration Assistant mentioned
in the sidebar, the purpose of this document is to convey an understanding of the
process and mechanics of setting up a system to run Logical Domains software and
creating guest domains on the system.
Chapter 2, “Initial Server Setup,” walks you through the tasks that you’ll need •
to perform just once: giving your server a network identity, updating the system
firmware, initializing the Solaris OS, and downloading and installing the Logical
Domains Manager software.
Chapter 3, “Control and I/O Domain Setup,” describes how to turn the OS •
running on the server into a control and I/O domain in the context of Logical
Domains software. The chapter shows how to set up the virtual disk and network
infrastructure used by guest domains.
Chapter 4, “Virtual Disk Backend Setup,” describes the benefits of using Solaris •
ZFS™ software to support virtual disks, and it walks you through the steps involved
in carving out a spare disk partition to use as a Solaris ZFS file system.
Chapter 5, “Guest Domain Creation,” shows how to create a logical domain and •
connect it to the virtual infrastructure created in Chapters 3 and 4.
Even these seven steps can be shortend by
using the Logical Domains Configuration
Assistant described in Appendix D of the
Logical Domains Administration Guide.
In a single command, the ldmconfig
command can set up a virtual disk backend
file, create, and start one or more guest
domains.
Sun Microsystems, Inc.4 Sun Logical Domains Quick Start Guide
Chapter 6, “Housekeeping,” describes how to store your current configuration in •
the service processor and to follow proper shutdown procedures. It explains how
to remove the guest domain you’ve created and restore your server to its initial
state.
Chapter 7, “Summary,” summarizes the seven commands that can be used to •
create new logical domains and suggests directions for additional exploration.
Sun Microsystems, Inc.5 Sun Logical Domains Quick Start Guide
Chapter 2
initial Server Setup
This chapter covers the steps that you need to take in order to get your server
connected to the network with current firmware installed.
Before you beginBefore you begin, make sure that your server is set up so that it meets the basic
assumptions under “Straightforward use case” in the previous chapter. This guide
assumes that you have a new, out-of-the-box server with current firmware and the
version of the Solaris OS required by the Logical Domains Release Notes is installed.
gather documentationEven though this guide is intended to give you a complete, self-contained set of
instructions for setting up a first domain using Logical Domains software, it is still
useful to have the basic LDoms documentation on hand to answer questions and
provide more background. Visit http://docs.sun.com/app/docs/prod/
ldoms to download appropriate versions of the following documents:
Logical Domains Release Notes• . This document will give you any last-minute news
on the release and any unresolved issues. You should not run into any issues
following the steps in this guide.
Logical Domains Administration Guide• . This document explains how to administer
Logical Domains software, illustrating a large number of scenarios that are beyond
the scope of this document.
Logical Domains Reference Manual• . This document provides all of the options for
all Logical Domains-related commands in a concise format.
assign iP addressesYou will need three IP addresses as you work through this guide, one for each of the
following uses:
Service processor Network Port. This is the port labeled NET MGT.•
Control domain. This is the address of the primary domain on the server, the one •
that boots with no Logical Domains software installed. This is typically the port
labeled NET 0.
Guest domain. This is the address that the guest domain will use when it connects •
to the network, and for the purpose of this example is the same NET 0 port as the
control domain.
If you need to upgrade your operating
system image, perform these initial setup
steps before performing the installation.
Sun Microsystems, Inc.6 Sun Logical Domains Quick Start Guide
Obtain these addresses, along with the appropriate netmasks, and router IP
addresses, from your network administrator. Putting the control domain and guest
domain IP addresses on the same subnet makes it easy to set up communication
between the control domain and the guest domain.
Set up the service processorThe first step is to connect to the service processor serial port and assign an IP
address to the service processor’s network port. This procedure shows you how to
accomplish this using ILOM commands. With power applied to the server, connect to
the serial port and press Return to obtain a login prompt.
obtaining the DHcP-assigned addressBy default, the service processor attempts to obtain an address from a DHCP
server, if one is available. If your network is supported with DHCP, you will need to
determine the address that was assigned so that you can connect via the browser
user interface later on.
Log in with the user root and the initial password changeme. Then use the show
/SP/network command to view the assigned address. In this example, the
address is 10.6.162.25.
For help connecting to the server’s serial
port, please refer to the administration
guide for your server available at
http://docs.sun.com.
Sun Microsystems, Inc.7 Sun Logical Domains Quick Start Guide
SUNSP00144F4641BD login: rootPassword: changemeWaiting for daemons to initialize...
Daemons ready
Sun(TM) Integrated Lights Out Manager
Version 3.0.2.50
Copyright 2008 Sun Microsystems, Inc. All rights reserved.Use is subject to license terms.
Warning: password is set to factory default.
-> show /SP/network
/SP/network Targets:
Properties: commitpending = (Cannot show property) dhcp_server_ip = none ipaddress = 10.6.162.25 ipdiscovery = static ipgateway = 10.6.163.254 ipnetmask = 255.255.252.0 macaddress = 00:14:4F:46:EE:A1 pendingipaddress = 10.6.162.25 pendingipdiscovery = static pendingipgateway = 10.6.163.254 pendingipnetmask = 255.255.252.0 state = enabled
Commands: cd set show
assigning a static iP addressIf you wish to assign a static IP address to the service processor, or override the
address allocated by DHCP, follow these steps.
Log in with the user root and the initial password changeme. Then type the
following commands to set the service console’s IP address and other parameters:
.
Sun Microsystems, Inc.8 Sun Logical Domains Quick Start Guide
.
SUNSP00144F4641BD login: rootPassword: changemeWaiting for daemons to initialize...
Daemons ready
Sun(TM) Integrated Lights Out Manager
Version 3.0.2.50
Copyright 2008 Sun Microsystems, Inc. All rights reserved.Use is subject to license terms.
Warning: password is set to factory default.
-> set /SP/network state=enabled-> set /SP/network pendingipdiscovery=static-> set /SP/network pendingipaddress=your-ip-address-> set /SP/network pendingipnetmask=your-netmask-> set /SP/network pendingipgateway=your-router-address-> set /SP/network commitpending=true
Now that the service processor has been set up to have a network identity, you can
choose to make further configuration steps through the browser user interface or
through the serial port.
check firmware revisionAt the service processor’s command prompt, type the following command line to
obtain the system software version:
-> show /HOST sysfw_version
You can determine your firmware revision from the command output. Firmware
revision numbers are sequential for each server type, so consult http://www.
sun.com/bigadmin/patches/firmware to determine the most current revision.
Locate your server model and see if the listed system firmware revision is greater
than the one installed on your server. If it is, download the latest firmware onto your
workstation and uncompress it. The result should be a package file whose name
ends in .pkg. If your have the latest firmware installed, skip the next section.
Update system firmwareYou can both check firmware revisions and update firmware through the service
console’s browser user interface (BUI) by directing your browser to http://
service-processor-address. Click the System Information tab, then the
Versions tab, and you’ll see a display similar to that shown below.
Sun Microsystems, Inc.9 Sun Logical Domains Quick Start Guide
.
The server power must be turned off before you update the firmware. To turn off
the power, click the Remote Control tab, then the Remote Power Control tab. Select
Graceful Shutdown and Power Off from the pop-up menu and click Save.
To update the service processor firmware, click the Maintenance tab, then click
Firmware Upgrade. Click the Enter Upgrade Mode button.
Now browse to where you have stored the firmware package on your desktop or
laptop, and then click the Upload button.
Sun Microsystems, Inc.10 Sun Logical Domains Quick Start Guide
.
The upgrade takes several minutes. Do not interrupt the process.
configure the Solaris oSIf you haven’t already initialized the Solaris OS that is installed on the server, now
is the time. You will need to go through a sequence of steps to give the operating
system its identity, including its name, time zone, and network settings. A new
Solaris OS instance will ask you for this information the first time that it boots.
You need to first power on the server, and then execute the initialization steps
through a command-line interface (via the service console) or through the BUI.
command-line setupTo work from the command line, you can use an ssh connection and log in to the
service processor’s network port. Use the ILOM start /SYS command to turn
on the system’s power, and then connect to the server console through the service
processor.
-> start /SYSAre you sure you want to start /SYS (y/n)? yStarting /SYS-> start /SP/consoleAre you sure you want to start /SP/console (y/n)? y
Serial console started. To stop, type #.
Note that the power-on sequence takes some time due to the power-on self tests
that are conducted. Once the Solaris OS boots for the first time, it will prompt you
for information including your language, time zone, and network settings.
Browser user interface setupThe other way to manage power and connect to the operating system console is
through the BUI. First power on the server by selecting the Remote Control tab and
then the Remote Power Control tab. Power on the server by selecting Power On from
the pop-up menu and clicking Save.
Sun Microsystems, Inc.11 Sun Logical Domains Quick Start Guide
.
To connect to the server console, select the Remote Control tab and then select
Launch Redirection. You can bring up a serial console by selecting I Want To
See Serial Redirection. If you choose the serial console, you will need to use the
sequence of commands under “Command-line setup” above to turn the server power
on. You can bring up a graphical user interface by leaving the option unchecked.
Either approach will allow you to go through the sequence of setting up the system
identity.
Now that you have the Solaris OS initialized and on the network, you can connect to
the server via ssh, through the service console (serial port or ssh), or through the
BUI (graphical or serial console)
apply operating system patchesRefer to the Logical Domains Release Notes for a list of required operating system
patches. Download and apply the patches.
Download the Logical Domains softwareNow that your operating system and system firmware is up to date, download the
Logical Domains software. Visit http://www.sun.com/ldoms, click the Get It tab,
and download the current software. For Logical Domains 1.2 the file downloaded was
LDoms_Manager-1_2.zip.
Sun Microsystems, Inc.12 Sun Logical Domains Quick Start Guide
.
install Logical Domains SoftwareNow connect to the server console to get a login prompt from the Solaris OS. You
can do this by ssh, or by connecting to the service processor (via serial port or ssh)
and then typing start /SP/console to connect to the server console.
Transfer the Logical Domains software to the server, and uncompress the zip file.
# unzip LDoms_Manager-1_2.zip
Navigate to the location of the install-ldm script and run it to install the Logical
Domains Manager software.
# cd LDoms_Manager-1_2# cd Install# ./install-ldm
The Logical Domains installation script will ask if you want to harden the server
operating system using the Solaris Security Toolkit. This is an excellent step to follow
for production systems as security of the control domain is important for protecting
guest domains. For the purpose of this quick install exercise, skip the hardening step
proceed with a standard Solaris OS configuration.
At this point you have a server with current firmware, an initialized Solaris operating
system instance, and running Logical Domains software. The next step is to convert
that OS instance into a control and I/O domain.
It is good security practice to prevent the
root user from directly logging into a server
via ssh, which is the default behavior.
In a laboratory setting, it is sometimes
expedient to override this default. To do so,
edit /etc/ssh/sshd_config. Change
the PermitRootLogin property from no
to yes. Then, restart the ssh daemon with
the command svcadm restart ssh.
Sun Microsystems, Inc.13 Sun Logical Domains Quick Start Guide
.
Chapter 3
control and i/o Domain Setup
initial server stateA Sun CMT server supports the sun4v architecture, which is implemented in part
by a thin hypervisor layer that is always interposed between the operating system
and the hardware. The hypervisor is always present regardless of whether Logical
Domains software is used. The initial server state is illustrated in Figure 2.
Internal Storage
PCIeSlots
CPU Threads,MAUs, Memory
32 GB Memory
Built-inEthernet
1 128
Control & I/O Domain
PCI Buses
Primary
Hypervisor
Sun CMT Server
Solaris 10 OS andLogical Domains
Manager
128 CPU threads
16 MAUs
Figure 2. In the server’s initial state, a single control domain owns all of the system resources including CPU threads, MAUs, memory, and PCI buses. The control domain is also referred to as the primary domain.
The example server has two eight-core UltraSPARC T2 Plus processors. The server
has two PCI buses that connect to built-in Ethernet interfaces, internal disk drives,
and PCI Express slots for additional peripherals. Each CPU has 64 threads and eight
modular arithmetic units (MAUs) that accelerate encryption operations.
Verify the server’s initial state using the ldm list command. You’ll note that a
single domain primary owns 128 virtual CPUs (threads) and 64 GB of memory
# ldm listNAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIMEprimary active -n-c-- SP 128 65312M 0.2% 3d 15h 41m
The end state of the next several steps is illustrated in Figure 3 and consists of the
following:
Sun Microsystems, Inc.14 Sun Logical Domains Quick Start Guide
.
A virtual disk server • primary-vds0 is ready to connect virtual disks in guest
domains to a service in the primary domain that serves data to them.
A virtual switch • primary-vsw0 is ready to connect virtual network devices in
guest domains to a virtual switch supported in the control and I/O domain. The
virtual switch is connected to a physical Ethernet port.
The control and I/O domain has given up most of its CPU, MAU, and memory •
resources into an unallocated pool that will provide resources to guest domains.
Built-inEthernet
Disk Storage Internal Resources
nxge0
4 CPU Threads
4 GB Memory
124 CPU Threads
16 MAUs
24 GB Memory
Unallocated Resources
Control & I/O Domain
Primary
Hypervisor
Sun CMT Server
Solaris 10 OS & LDM
primary-vsw0
primary-vsw0
primary-vds0
primary-vds0
1 124
Figure 3. The end state of this set of steps frees resources and creates virtual device infrastructure to support guest domains.
create a virtual disk serverThe virtual disk service primary-vds0 will support disk I/O in the guest domain;
actual sources of data will be attached to it later.
# ldm add-vdiskserver primary-vds0 primary
create a virtual switchUse the ifconfig command to determine the name of the physical network device
through which the primary domain communicates to the network. In this example,
the device is nxge0.
Sun Microsystems, Inc.15 Sun Logical Domains Quick Start Guide
.
# ifconfig -alo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000nxge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 10.6.162.45 netmask fffffc00 broadcast 10.6.163.255 ether 0:14:4f:46:ee:98
Now use the knowledge of that interface to create a virtual switch primary-vsw0
that is connected to the physical interface.
# ldm add-vswitch net-dev=nxge0 primary-vsw0 primary
create a virtual console deviceA virtual console device allows the control and I/O domain to connect to the console
port of the guest domain. When connecting to the guest’s console, you will use the
command telnet localhost port, where port is in the range of 5000-5100
that the virtual console device will assign.
# ldm add-vconscon port-range=5000-5100 primary-vcc0 primary
Free cPU, maU, and memory resourcesThe following commands free CPU, MAU, and memory resources, which are currently
allocated to the primary domain. These resources are released into a free pool from
which the resources can be allocated to guest domains as we create them. The
commands leave the primary domain with four virtual CPUs, no MAU, and 4 GB of
main memory. Changing these resources puts the Logical Domains Manager into
delayed reconfiguration mode where the actual resources are changed only after the
primary domain is rebooted.
# ldm set-mau 0 primaryLDom primary is unable to dynamically reconfigure crypto units.Initiating delayed reconfigure operation on LDom primary. All configurationchanges for other LDoms are disabled until the LDom reboots, at which timethe new configuration for LDom primary will also take effect.# ldm set-vcpu 4 primary------------------------------------------------------------------------------Notice: LDom primary is in the process of a delayed reconfiguration.Any changes made to this LDom will only take effect after it reboots.------------------------------------------------------------------------------# ldm set-mem 4g primary------------------------------------------------------------------------------Notice: LDom primary is in the process of a delayed reconfiguration.Any changes made to this LDom will only take effect after it reboots.------------------------------------------------------------------------------
Sun Microsystems, Inc.16 Sun Logical Domains Quick Start Guide
.
Save the configuration in the service processor
When making configuration changes, always remember to save the final state in the Note – service processor.
For the hypervisor to choose this new hardware configuration when the system
is powered on, the configuration must be stored in the service processor. In this
example, we store the changed state in the named location my-initial.
# ldm add-spconfig my-initial# ldm list-spconfigfactory-defaultmy-initial [current]
Now reboot the primary and I/O domain so that the delayed configuration takes
effect.
# shutdown -i6 -g0 -y
Sun Microsystems, Inc.17 Sun Logical Domains Quick Start Guide
.
Chapter 4
Virtual Disk Backend Setup
Virtual disk optionsA guest domain needs to boot from a virtual disk, and the storage for that disk
is most easily allocated from an internal disk drive on the server. The storage for
a virtual disk is called a virtual disk backend, and it can store data in any of the
following locations:
A flat file stored on a UFS, Solaris ZFS, or other suitable file system•
A physical disk with or without underlying hardware RAID•
A partition on a physical disk•
There are significant advantages to using a flat file stored on a Solaris ZFS file
system. You can use snapshots to create a point-in-time image of a virtual disk for
use in backups or roll back a guest domain to a previous, known state. You can use
clones as a quick and space-efficient way to create multiple guest domains from the
same “golden master” Solaris OS installation. When more than one disk device is
dedicated to a ZFS file system pool, various levels of redundancy can be established
to further protect data.
This set of steps illustrates how you can create a ZFS pool from a single disk
partition. After creating the guest domain we will demonstrate how easy it is to
make a clone of your first guest domain for use in creating subsequent guests.
Locate or create a disk partitionWe have assumed that you have at least one internal disk drive on your server, and
that you have a factory-installed OS that is on a UFS file system. This sequence of
steps locates available disk space on one disk and creates a partition for use as a
Solaris ZFS file system.
Use the format command and choose the disk on which you wish to allocate a
partition as a virtual disk backend.
This example repartitions disk 0, where the Solaris OS is usually installed. Take Note – care when allocating unused space on this volume not to destroy your running Solaris OS installation for the control domain.
Depending on the underlying storage, this
configuration may create a single point of
failure. Techniques for increasing reliability
are discussed in the Sun BluePrints articles
“Data Reliability with Logical Domains” and
“Data Availability with Logical Domains.”
Sun Microsystems, Inc.18 Sun Logical Domains Quick Start Guide
.
# formatSearching for disks...done
AVAILABLE DISK SELECTIONS: 0. c1t0d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848> /pci@400/pci@0/pci@8/scsi@0/sd@0,0 1. c1t1d0 <SEAGATE-ST914602SSUN146G-0603-136.73GB> /pci@400/pci@0/pci@8/scsi@0/sd@1,0
Specify disk (enter its number): 0
The format command will present a menu. Choose p to edit the partition table.
FORMAT MENU: disk - select a disk type - select (define) a disk type partition - select (define) a partition table current - describe the current disk format - format and analyze the disk repair - repair a defective sector label - write label to the disk analyze - surface analysis defect - defect list management backup - search for backup labels verify - read and display labels save - save new disk/partition definitions inquiry - show vendor, product and revision volname - set 8-character volume name !<cmd> - execute <cmd>, then return quitformat> p
The format command presents the partition table for the selected disk. The swap
partition uses disk cylinders 0–402. The root partition uses cylinders 403–1912. The
backup partition represents the whole disk, which is 14087 cylinders. We can create
a partition from any of the unassigned partitions. We choose partition 6, starting the
partition at cylinder 1913 (one after the root partition) and make it 12174 cylinders
in size (cylinders 1913–14087). Select 6 to begin this partitioning process.
Current partition table (original):Total disk cylinders available: 14087 + 2 (reserved cylinders)
Part Tag Flag Cylinders Size Blocks 0 root wm 403 - 1912 14.65GB (1510/0/0) 30731520 1 swap wu 0 - 402 3.91GB (403/0/0) 8201856 2 backup wm 0 - 14086 136.71GB (14087/0/0) 286698624 3 unassigned wm 0 0 (0/0/0) 0 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 0 0 (0/0/0) 0 6 unassigned wm 0 0 (0/0/0) 0 7 unassigned wm 0 0 (0/0/0) 0
partition> 6
Sun Microsystems, Inc.19 Sun Logical Domains Quick Start Guide
.
Allocate the remaining space with a partition that begins after the root partition
and extends to the end of the disk. Then choose l to label the disk and q to quit the
partition command.
Part Tag Flag Cylinders Size Blocks 6 unassigned wm 0 0 (0/0/0) 0
Enter partition id tag[unassigned]:Enter partition permission flags[wm]:Enter new starting cyl[0]: 1913Enter partition size[0b, 0c, 1913e, 0.00mb, 0.00gb]: 12174cpartition> pCurrent partition table (unnamed):Total disk cylinders available: 14087 + 2 (reserved cylinders)
Part Tag Flag Cylinders Size Blocks 0 root wm 403 - 1912 14.65GB (1510/0/0) 30731520 1 swap wu 0 - 402 3.91GB (403/0/0) 8201856 2 backup wm 0 - 14086 136.71GB (14087/0/0) 286698624 3 unassigned wm 0 0 (0/0/0) 0 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 0 0 (0/0/0) 0 6 unassigned wm 1913 - 14086 118.14GB (12174/0/0) 247765248 7 unassigned wm 0 0 (0/0/0) 0
partition> lReady to label disk, continue? y
partition> q
Create a ZFS file system pool ldompool from the c1t0d0s6 partition, and create
the domain1 file system within the pool. By default, ZFS pools are mounted in the
root directory by the name of the pool, and ZFS file systems appear underneath
as directories. Create an empty file with the mkfile command to hold the guest
domain’s disk image. Using the -n option creates the file but does not allocate all of
its data. This example creates a 20 GB file.
# cd / # zpool create -f ldompool c1t0d0s6# zfs create ldompool/domain1# mkfile -n 20g /ldompool/domain1/diskimagefile
Sun Microsystems, Inc.20 Sun Logical Domains Quick Start Guide
.
Chapter 5
guest Domain creation
allocate resources to a guest domainThe previous chapters have freed up and created resources that now can be allocated
to a guest domain easily. The end state reached in this chapter is illustrated in
Figure 4, and it has the following characteristics:
A guest domain • guest1 has four virtual CPUs and 4 GB memory allocated to it.
The domain in which it runs has the following:•
A virtual network device connecting it to the network•
A virtual disk connected to the virtual disk backend we created in •
the previous chapter
A virtual • cdrom device connected to a downloaded Solaris OS ISO image
Once the guest domain is started, it boots from the virtual cdrom device and begins
an installation of the Solaris OS.
Built-inEthernet
Disk Storage Internal Resources
nxge0
4 CPU Threads
4 GB Memory
4 CPU Threads
120 CPU Threads
16 MAUs
24 GB Memory4 GB Memory
Control & I/O Domain
Unallocated Resources
Solaris 10 OS
Control & I/O Domain
Primary Guest 1
Hypervisor
Sun CMT Server
Solaris 10 OS & LDM
/solaris-s10u6-dvd.iso
primary-vsw0
primary-vsw0
primary-vds0
cdromvdisk0
primary-vds0
ldompool/domain1/diskimagefile
vnet0
1 24
Figure 4. The end state of this quick-start guide is a guest domain with both physical and virtual resources connected to it.
Sun Microsystems, Inc.21 Sun Logical Domains Quick Start Guide
.
create the guest and allocate resourcesCreate the guest domain with the ldm create command and allocate physical
resources including CPU threads (virtual CPUs) and memory. In this example, four
virtual CPUs and 4 GB of memory is allocated to the domain guest1.
# ldm create guest1# ldm set-vcpu 4 guest1# ldm set-mem 4g guest1
Create a virtual network device vnet0 in guest1 and connect it to the virtual
switch that was created in Chapter 3.
# ldm add-vnet vnet0 primary-vsw0 guest1
Attach the flat file /ldompool/domain1/diskimagefile to the virtual disk
service primary-vds0 with the name vol1. Then make that volume available to
the guest1 domain with the name vdisk0.
# ldm add-vdsdev /ldompool/domain1/diskimagefile vol1@primary-vds0# ldm add-vdisk vdisk0 vol1@primary-vds0 guest1
The Logical Domains Manager does not check to be sure that a file name you provide Note – is valid. The only indication that you have typed an invalid file name is a virtual disk that does not work in the guest domain.
Download a Solaris OS ISO image from http://www.sun.com/solaris/get.
Connect the downloaded ISO image to the virtual disk service primary-vds0 with
the name cdrom. Then make the volume available to the guest1 domain with the
name cdrom.
# ldm add-vdsdev /solaris-s10u6-dvd.iso cdrom@primary-vds0# ldm add-vdisk cdrom cdrom@primary-vds0 guest1
Each domain runs its own instance of OpenBoot™ software. You can set the domain’s
boot device so that it first tries to boot from a disk, and then from the virtual cdrom
device. On the first boot attempt, booting from the disk will fail and the domain
will boot from the cdrom device. Once the Solaris OS is installed, the next attempt
to boot from the disk will be successful. Note that you must boot from the cdrom
device’s f partition as shown in the following command.
# ldm set-variable boot-device="disk cdrom:f" guest1
Commit the resources that have been allocated using the bind command.
# ldm bind guest1
Sun Microsystems, Inc.22 Sun Logical Domains Quick Start Guide
.
You can verify that the resources have been allocated using the
ldm list-bindings command.
# ldm list-bindings guest1NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIMEguest1 bound ------ 5000 4 4G
MAC 00:14:4f:fb:e3:4a
HOSTID 0x84fbe34a
VCPU VID PID UTIL STRAND 0 4 100% 1 5 100% 2 6 100% 3 7 100%
MEMORY RA PA SIZE 0xe000000 0x10e000000 4G
VARIABLES boot-device=cdrom:f
NETWORK NAME SERVICE DEVICE MAC MODE PVID VID vnet0 primary-vsw0@primary network@0 00:14:4f:fa:c0:ad 1 PEER MAC MODE PVID VID primary-vsw0@primary 00:14:4f:f9:6a:97 1
DISK NAME VOLUME TOUT DEVICE SERVER MPGROUP vdisk0 vol1@primary-vds0 disk@0 primary cdrom cdrom@primary-vds0 disk@1 primary
VCONS NAME SERVICE PORT guest1 primary-vcc0@primary 5000
Boot from the guest domain’s consoleUse the ldm list command to determine the port number for guest1’s console.
Observe that the console port is 5000.
# ldm listNAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIMEprimary active -n-cv- SP 4 4G 0.8% 5mguest1 bound ------ 5000 4 4G
Enable the virtual console service (you only need to do this one time), and then start
the guest domain.
Sun Microsystems, Inc.23 Sun Logical Domains Quick Start Guide
.
# svcadm enable vntsd# ldm start guest1LDom guest1 started
Now telnet to guest1’s console. The domain should be in the process of booting
from the Solaris ISO image, and it should begin an interactive Solaris OS installation.
If you do not have automatic booting enabled, you can use commands such as
show-devs and show-nets at the OpenBoot software’s ok prompt to verify that
the devices you created are available to the guest domain.
A good practice is to use a separate ssh connection to the control domain for each Note – guest domain console connection you make. When you connect to the control domain through the service processor, only one connection can own the console, which limits the number of concurrent telnet sessions you can observe. When you create a new ssh connection to the control domain for each guest domain console connection, you don’t use a limited resource. Thus, response time is better because the connection is not limited by the console serial port speed.
# telnet localhost 5000Trying 127.0.0.1...Connected to localhost.Escape character is ‘^]’.
Connecting to console “guest1” in group “guest1” ....Press ~? for control options ..
Once the installation is complete, you have a guest domain that runs using its
own partition of the server’s physical resources, supported by the virtual resources
provided through the control and I/O domain. You can boot the domain, shut it
down, and interact with the OpenBoot software via the guest’s console.
Be sure to read on to the next chapter and save the current configuration to the
service processor.
Use Solaris ZFS snapshots and clonesSince you’ve created the guest’s virtual disk on a Solaris ZFS file system, you can take
advantage of the ability to take snapshots and make clones of the file system.
For example, suppose that you don’t want to go through the Solaris OS installation
process for every new guest domain that you create. You can create a golden master
virtual disk image that you can clone as many times as needed for each new logical
domain that you create. One of the benefits of using snapshots and clones in Solaris
ZFS file systems is that they are very space efficient. Every disk block in a clone is a
reference to the original golden master, and space is allocated only as needed using
copy-on-write semantics.
Note that, by default, the control domain
cannot send network traffic to guest
domains. Therefore, you cannot ping or
ssh from the control domain to the guest.
To enable networking between the control
and guest domains, refer to the instructions
“Enabling Networking Between the Control/
Service Domain and Other Domains” in the
Logical Domains Administration Guide.
Details on using Solaris ZFS technology
with Logical Domains are available in
Chapter 6 of the Sun BluePrints article
“Data Reliability with Logical Domains.”
The Logical Domains System Administration
Guide also makes suggestions in the section
“Using ZFS with Virtual Disks” in Chapter 6,
“Using Virtual Disks.”
Sun Microsystems, Inc.24 Sun Logical Domains Quick Start Guide
.
To create a clone, first snapshot a Solaris ZFS file system and then make a clone of it.
The cloning process makes a permanent copy of the snapshot. The clone uses copy-
on-write semantics so that the clone is an independent, read/write copy.
Solaris ZFS file system snapshots operate on file systems, not files, so it is a good
practice to put only one virtual disk backend file in each Solaris ZFS file system.
In the following example, we create a snapshot of guest1’s file system named
initial. Then we clone the snapshot and create a new file system golden at the
same time.
# zfs snapshot ldompool/domain1@initial# zfs clone ldompool/domain1@initial ldompool/golden
The file /ldompool/golden/diskimagefile is a copy of /ldompool/
domain1/diskimagefile except that it occupies almost no disk space until blocks
in the file are changed. If you attach this golden master virtual disk backend to a
logical domain, however, it will have the identity and network settings of the guest1
logical domain you have already created.
You can remove the golden master’s identity with the sys-unconfig command,
however note that this will not remove many useful changes you can prepare in the
golden image, such as patches, applications, and configuration files. This will cause
the Solaris OS to go through the system identity setting prompts upon first boot so
that each clone of the golden master obtains its own identity. To accomplish this,
perform the following steps:
Shut down the 1. guest1 logical domain.
Prepare a new logical domain 2. guest2 using /ldompool/golden/
diskimagefile as its virtual disk backend.
Boot 3. guest2 and run the sys-unconfig command. The command will
unconfigure the system and halt it.
Snapshot the unconfigured state of the golden master file system.4.
# zfs snapshot ldompool/golden@goldenmaster
Now you can start 5. guest1 without any identity conflicts on the network. You
can start guest2 and give it an identity. To create new logical domains based
on the unconfigured golden master, use the zfs clone command using the
snapshot ldompool/golden@goldenmaster as the source.
Sun Microsystems, Inc.25 Sun Logical Domains Quick Start Guide
.
Chapter 6
Housekeeping
Save configurations to the service processorWhenever you change a Logical Domains configuration, be sure to save the
configuration to the service processor. If you don’t do this, and you reboot the
control domain or cycle the power on the server, the service processor will only
re-instate the last saved configuration. This configuration may be as old as the
factory-default configuration. You can save up to eight named configurations
in the service processor. The last one saved is always the one used at the next boot
unless you explicitly set the desired configuration with the ldm set-spconfig
command. One way to save the configuration is to delete and re-create the
configuration that you have already saved, for example:
# ldm remove-spconfig my-initial# ldm add-spconfig my-initial
Logical Domains software includes a configuration auto-recovery mechanism in case
the configuration was not explicitly saved to the service processor.
Use proper shutdown proceduresThe Logical Domains Release Notes contain important information about how
to properly shut down a server with multiple logical domains. Note that it is not
sufficient to simply shut down the control domain as guest domains will continue to
run. The procedure is to do the following:
Shut down and unbind all non-I/O domains. In this example, you could use the 1.
command ldm stop guest1 to gracefully shutdown the guest domain, then
unbind with ldm unbind guest1.
Shut down and unbind any active I/O domains. In this example, there is no 2.
secondary I/O domain supporting redundant I/O paths. If there were, you would
use the two commands from Step 1.
Shut down and halt the primary domain. You can use the command 3. shutdown
-i5 -g0 -y. Following the shutdown the server will power off since there are
no longer any active domains running.
Removing guest domainsYou can use the following sequence of commands to shut down, unbind, and destroy
a logical domain if you no longer need it.
Sun Microsystems, Inc.26 Sun Logical Domains Quick Start Guide
.
# ldm stop guest1LDom guest1 stopped# ldm unbind-domain guest1# ldm destroy guest1
Removing Logical Domains softwareIf you wish to restore your server to its original state, not running any logical
domains, you can execute the following sequence of commands.
# ldm rm-config my-initial# ldm stop-domain -aLDom guest1 stopped# ldm unbind-domain guest1# ldm destroy guest1# ldm set-config factory-default# svcadm disable ldmd# svcadm disable vntsd# pkgrm SUNWldm# shutdown -i5 -g0 -y
If you have enabled networking between the control domain and guests, you
may have to manually restore the previous hostname files (for example /etc/
hostname.nxge0) and remove the hostname files that plumbed virtual switches
into the control domain (for example /etc/hostname.vsw0).
Sun Microsystems, Inc.27 Sun Logical Domains Quick Start Guide
.
Chapter 7
Summary
Once you have installed the correct server firmware and the Logical Domains
Manager software, it is simple and straightforward to create new logical domains
given a virtual disk image. As demonstrated in Chapter 1, starting with a populated
virtual disk image file, only seven commands are needed to create a guest domain.
The ldm create command creates the guest. The ldm set-vcpu and ldm set-
mem commands allocate virtual CPUs and memory to the guest. The ldm add-
vnet command gives the guest a network connection. The ldm-add-vdsdev
command attaches a virtual disk to the virtual disk service, and the ldm add-
vdisk command attaches the service to the guest. The ldm bind command
activates the resource allocation.
# ldm create guest1# ldm set-vcpu 4 guest1# ldm set-mem 4g guest1# ldm add-vnet vnet0 primary-vsw0 guest1# ldm add-vdsdev /ldompool/domain1/diskimagefile vol1@primary-vds0# ldm add-vdisk vdisk0 vol1@primary-vds0 guest1# ldm bind guest1
Sun Logical Domains provides an enterprise-class virtualization environment that can
provide many more powerful security, availability, and management tools than are
available on physical machines:
Security features such as virtual local-area networks (VLANs) translate directly into •
logical domains. In addition, virtual switches can be created to manage each of
the physical interfaces on a server, increasing security and network throughput.
Data reliability techniques, such as maintaining multiple copies of business-•
critical data, translate easily to Logical Domains environments.
Availability can be enhanced in the virtual world in a way similar to how it is done •
in the physical world: with multiple redundant paths to networks and disk storage
through independent I/O domains.
I/O performance can be enhanced by allowing guest domains to access I/O •
devices directly by assigning ownership of individual PCI buses to them (on servers
with more than one PCI bus). Application performance can be enhanced through
the ability to dynamically change the number of virtual CPUs and amount of
memory assigned to a domain without requiring a reboot.
Guest domains can be created quickly and easily using Solaris ZFS file system •
features including snapshots and clones, while using minimal space for new
virtual disks.
Sun Microsystems, Inc.28 Sun Logical Domains Quick Start Guide
.
Guest domains can be moved between servers via warm or cold migration, •
simplifying datacenter workload management, and simplifying the process of
taking a physical server down for upgrades or maintenance.
Many of these features are described in Sun BluePrints articles available at
http://www.sun.com/blueprints, or in the LDoms Community Cookbook
available at http://wikis.sun.com/display/SolarisLogicalDomains/
LDoms+Community+Cookbook. With an initial guest domain up and running
on a Sun CMT server, an abundance of resources is available to support further
exploration and enterprise deployment.
about the authorsPeter A. Wilson has more than 16 years of industry experience, 14 of which have
been with Sun, serving in a wide variety of hardware, software, systems and product
marketing roles. Peter moved from the United Kingdom to the US in 2000 to lead
the customer tests of Sun’s Netra™ and fault-tolerant servers. Peter is currently a
technical marketing manager responsible for a team of Product Specialists covering
all of Sun’s hardware product portfolio. Peter holds a M.Eng (Master of Engineering)
degree in Microelectronics and Software Engineering from the University of
Newcastle-upon-Tyne, U.K.
Gary Combs is currently a SPARC Specialist in the Systems Technical Marketing Group.
He specializes in mid-range high-end SPARC servers, which includes the popular
M-Series as well as the higher end CMT servers, such as the Sun SPARC Enterprise
T5440 server. Gary also covers virtualization technologies that are implemented on
these platforms: Dynamic Domains, Logical Domains, and Solaris Containers. Gary
has over 15 years direct sales support experience as a systems engineer. For the
last 10 years, Gary has held marketing positions in product management, product
definition, and now technical marketing.
acknowledgmentsThe author would like to thank Steve Gaede, an independent technical writer and
engineer, for preparing this article. Thanks also to Alexandre Chartre, Cathleen
Reiher, and Pierre Reynes for giving the document a thorough review.
ordering Sun documentsThe SunDocsSM program provides more than 250 manuals from Sun Microsystems,
Inc. If you live in the United States, Canada, Europe, or Japan, you can purchase
documentation sets or individual manuals through this program.
Sun Microsystems, Inc.29 Sun Logical Domains Quick Start Guide
.
accessing Sun documentation onlineThe docs.sun.com web site enables you to access Sun technical documentation
online. You can browse the docs.sun.com archive or search for a specific book title or
subject. The URL is
http://docs.sun.com/
To reference Sun BluePrints OnLine articles, visit the Sun BluePrints OnLine Web site:
http://www.sun.com/blueprints/online.html
Sun Microsystems, Inc.
Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN (9786) Web sun.com
Logical Domains Quick Start Guide
© 2008-2010 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, the Sun logo, BluePrints, Java, OpenBoot, Solaris, SunDocs, and ZFS are trademarks or registered trademarks of Sun Microsystems, Inc. or its subsidiaries in the United States and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the US and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. Information subject to change without notice. Printed in USA 1/10