xenserver-drbd

1DeployingDRBDwithCitrixXenServerFlorian Haas

Copyright 2009, 2010, 2011 LINBIT HA-Solutions GmbH

Trademarks used in this document

DRBD and LINBIT are trademarks or registered trademarks of LINBIT Information Technologies GmbH in Austria, theUnited States, Japan, Australia and other countries.

Citrix, XenServer, XenEssentials, XenMotion and XenCenter are trademarks or registered trademarks of Citrix, Inc.

Other names may be trademarks or registered trademarks of their respective owners.

License information

The text and illustrations in this document are licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported license ("CC BY-NC-ND").

A summary of CC BY-NC-ND is available at http://creativecommons.org/licenses/by-nc-nd/3.0/.

The full license text is available at http://creativecommons.org/licenses/by-nc-nd/3.0/legalcode.

In accordance with CC BY-NC-ND, if you distribute this document, you must provide the URL for the original version.

1. Introduction ............................................................................................................ 12. Installation .............................................................................................................. 2

2.1. DRBD Installation on XenServer 5.0 ................................................................. 22.2. DRBD Installation on XenServer 5.5 ................................................................. 2

3. Initial Configuration .................................................................................................. 23.1. DRBD Replication Network Configuration ......................................................... 23.2. LVM configuration ......................................................................................... 4

4. Using DRBD in a XenServer setup without XenMotion support ....................................... 44.1. DRBD Configuration ...................................................................................... 54.2. Xen Storage Repository Configuration .............................................................. 64.3. Using the DRBD backed Storage Repository ...................................................... 6

5. Using DRBD in a XenServer setup with XenMotion support ............................................ 85.1. DRBD Configuration ...................................................................................... 85.2. Xen Storage Repository Configuration .............................................................. 95.3. Xen Resource Pool Configuration ..................................................................... 9

6. Limitations and Caveats .......................................................................................... 116.1. DRBD disconnect during pool master/slave transition ....................................... 116.2. Split brain potential in shared SR configurations ............................................... 11

7. Remarks ................................................................................................................ 12

1.IntroductionCitrix XenServer 5.0 is a commercial product from Citrix, Inc. containing the Xen 3.2 hypervisor.Citrix makes XenServer 5.0 available for free; this includes a Windows based management GUI(XenCenter).

Deploying DRBDwith Citrix XenServer

2

Citrix XenServer 5.5 is a commercial product from Citrix, Inc. containing the Xen 3.3 hypervisor.Citrix makes XenServer 5.5 available for free; this includes a Windows based management GUI(XenCenter).

High availability features are not included in the free offering; they are included in XenEssentialswhich Citrix markets for a license fee. XenEssentials' high availability features are, for all practicalpurposes, inferior to the features provided by the OpenAIS/Pacemaker cluster stack.

In XenServer, storage made available to virtual domains is referred to as Storage Repositories (SRs),which may be either local or shared. This document outlines the process to create

DRBD backed, unshared storage repositories which may be utilized in a warm standby fashion,where virtual machines may be manually taken over when one node fails;

DRBD backed, shared storage repositories which may be utilized in a hot standby fashionincluding uninterrupted virtual machine migration.

2.Installation2.1.DRBDInstallationonXenServer5.0

Current DRBD packages are best installed via LINBIT's official YUM repositories for XenServer 5.Detailed instructions for this are provided at http://packages.linbit.com/hash/xenserver5/i386(where hash is the your individual download hash issued when you purchase a DRBD supportcontract).

You will need to make sure that you download and install DRBD kernel packages for the correctXenServer 5.0 Domain-0 kernel.

2.2.DRBDInstallationonXenServer5.5Current DRBD packages are best installed via LINBIT's official YUM repositories for XenServer5. Detailed instructions for this are provided at http://packages.linbit.com/hash/xenserver5.5/i386 (where hash is the your individual download hash issued when you purchase a DRBDsupport contract).

You will need to make sure that you download and install DRBD kernel packages for the correctXenServer 5.5 Domain-0 kernel.

3.InitialConfiguration3.1.DRBDReplicationNetworkConfiguration

In order to be able to use one of the devices available on the XenServer host for DRBD replication,you must configure one network interface device on each host with a static IP address.

3.1.1.ReplicationNetworkConfigurationusingXenCenter

To assign a network interface for DRBD replication with XenCenter, highlight the server in theXenCenter resource pane (on the left of the screen). Right-click to open the context menu andselect Management Interfaces. Then, click the button New Interface:


3

Figure1.XenCenter management interface configuration

Select the network you wish to use for DRBD replication. Assign a name to the replicationinterface, such as DRBD Replication as shown in the example above. Then, add an IP addressand subnet mask. You may leave the gateway address blank. Finally, click OK to enable thereplication interface.

Repeat this step on the peer node. Remember to use a different IP address in the same subnet.

3.1.2.ReplicationNetworkConfigurationusingtheXenServerCLI

To use eth1 as your DRBD replication interface, you must first issue the following command tofind out the physical interface's UUID:

xe pif-list device=eth1 params=uuiduuid ( RO) : afc01480-ca70-ee45-5de0-09602f2ba52c

Then, issue these commands to reconfigure the interface with a static IP configuration:

xe pif-reconfigure-ip mode=static \ uuid=afc01480-ca70-ee45-5de0-09602f2ba52c IP=192.168.10.20 netmask=255.255.255.0xe pif-param-set disallow-unplug=true \ other-config:management_purpose="DRBD Replication" \ uuid=afc01480-ca70-ee45-5de0-09602f2ba52c

Note

You may subsequently verify your IP configuration using the xe pif-param-listuuid=[interface UUID]command.


4

3.1.3.FirewallconfigurationThe iptables firewall configuration installed by default on XenServer blocks DRBD replicationtraffic. In order to enable DRBD replication between hosts, you will have to take one of thefollowing steps:

Specifically allow DRBD replication.Assuming you are using the default TCP ports forDRBD replication, you may introduce firewall rules to enable DRBD network traffic betweenhosts:

iptables -A INPUT -i interface -p tcp --dport 7788:7799 -j ACCEPTiptables -A OUTPUT -o interface -p tcp --sport 7788:7799 -j ACCEPT/etc/init.d/iptables save

Replace interface with the network interface you are using for DRBD replication traffic(usually xenbr1)

Disable firewalling.If your host is on a secure, walled-off network and does not need localfirewalling, you may disable iptables on startup:

chkconfig iptables off

Whichever approach you select, you will have to repeat the relevant step on both DRBD nodes.

3.2.LVMconfigurationAs the Storage Repository we are about to create is of the LVM type, we must instruct LVM toread PV signatures from our DRBD devices we are about to create, rather than the underlyingbacking block devices. The easiest approach for doing this is to mask the underlying block devicefrom the list of devices LVM scans for PV signatures.

NoteModifying the LVM configuration on XenServer hosts must always be doneby modifing the LVM configuration file. There are no equivalent commands inXenCenter.

To do so, open the LVM configuration file (/etc/lvm/lvm.conf) and edit the following entries:

filter = [ "r|/dev/sdb.*|" ]In addition, you should disable the LVM cache by setting:

write_cache_state = 0

After disabling the LVM cache, make sure you remove any stale cache entries by deleting /etc/lvm/cache/.cache.

You must repeat the above steps on the peer node.

4.UsingDRBDinaXenServersetupwithoutXenMotionsupport

This section outlines the recommended DRBD/XenServer configuration if requirements do notinvolve any necessity to support virtual machine live migration using XenMotion.

While foregoing live migration, this setup has the advantage that DRBD need not be configuredin dual-Primary mode. Instead, you may run DRBD in single-Primary mode, which is extremelysimple and effective to configure, and is a lot less prone to split brain than dual-Primary mode is.


5

This type of setup may be configured on the free XenServer, and does not require CitrixXenEssentials.

4.1.DRBDConfigurationTo configure DRBD, create /etc/drbd.conf on xenserver1 and xenserver2:

global { usage-count yes; }common { protocol C; net { shared-secret "xenserver"; after-sb-0pri discard-zero-changes; after-sb-1pri consensus; after-sb-2pri disconnect; } disk { max-bio-bvecs 1; } handlers { split-brain "/usr/lib/drbd/notify-split-brain.sh"; } syncer { rate 12M; }}resource drbd-sr1 { device /dev/drbd1; disk /dev/sdb1; meta-disk internal; on xenserver1 { address 192.168.10.20:7788; } on xenserver2 { address 192.168.10.21:7788; }}

resource drbd-sr2 { device /dev/drbd2; disk /dev/sdb2; meta-disk internal; on xenserver1 { address 192.168.10.20:7789; } on xenserver2 { address 192.168.10.21:7789; }}

Then, proceed as outlined in the DRBD User's Guide [http://www.drbd.org/users-guide/s-first-time-up.html] to initialize and synchronize your resource.

When the initial synchronization has completed, check the output of chkconfig --list drbdto make sure that the drbd init script is executed on system startup.


6

4.2.XenStorageRepositoryConfiguration

Note

There are currently no XenCenter equivalents to the CLI commands listed in thissection. Configuring a DRBD backed Storage Repository must always be done usingthe XenServer CLI.

In order to create Storage Repositories from our DRBD resources, issue the following commands:

xe sr-create host-uuid=[local host UUID] \ name-label="drbd-sr1" type="lvm" \ device-config:device="/dev/drbd/by-res/drbd-sr1"xe sr-create host-uuid=[local host UUID] \ name-label="drbd-sr2" type="lvm" \ device-config:device="/dev/drbd/by-res/drbd-sr2"

Tip

You can easily find out your local host UUID by issuing xe host-listhostname=`uname -n` params=uuid. Or, you may simply use tab completionwith the xe sr-create command (host-uuid=).

If successful, the xe sr-create command will return a newly generated UUID for each SR.Next, you can check its status with the following commands:

xe sr-listxe pbd-list

4.3.UsingtheDRBDbackedStorageRepositoryThis section describes the steps required to install and start virtual machines using DRBD backedstorage repositories.

4.3.1.ManagingDRBD-backedVirtualMachinesusingXenCenter

To install a virtual machine using XenCenter, click the New VM icon and follow the installationwizard.


7

Figure2.XenCenter virtual machine installation

In the Virtual Disks step, you will be presented with the shared storage repository to create virtualblock devices from.

When installation is completed, you may highlight the virtual machine in the resource tree, right-click and select Start to start the virtual machine.

4.3.2.ManagingDRBD-backedVirtualMachinesusingtheXenServerCLI

This example is for the installation of a Debian 4.0 virtual machine:

xe vm-install template="Debian Etch 4.0" \ new-name-label=[label] sr-name-label=drbd-sr1306c3e0b-cfbf-282a-1daa-4de7334ad3d2

Once installation is complete, you may start the virtual machine:

xe vm-start vm=[label]

In order to be able to start the virtual machine on the peer node in case the current host shouldfail in the future, issue this command to backup the virtual machine metadata:

xe-backup-metadata -c -u [SR UUID]

You should repeat this command every time you create a new virtual machine, or modify a virtualmachine's configuration.

If a host fails and you want the peer to take over its resources, you may do so with the followingcommands:

drbdadm primary resource


8

vgscanxe sr-introduce uuid=[SR UUID] name-label=DRBD type=lvmxe pbd-create sr-uuid=[SR UUID] host-uuid=[host UUID] \ device-config:device=/dev/drbd/by-res/resourcexe pbd-plug uuid=[device UUID]xe-restore-metadata -u [SR UUID]

TipIf you do not have any record of your storage repository UUID, you can find it outeasily by running the pvs command after you switch your DRBD resource to thePrimary role. XenServer names LVM Volume Groups using the naming conventionVG_XenStorage-[SR UUID].

Then, you will have to reset XenServer's information about the machine's power state before youcan restart them:

xe vm-reset-powerstate vm=[label]xe vm-start vm=[label]

Repeat these last two commands for every virtual machine you wish to restart.

5.UsingDRBDinaXenServersetupwithXenMotionsupport

This section outlines the recommended DRBD/XenServer configuration if live migration support(using XenMotion) is a requirement.

While supporting live migration, this setup has the disadvantage that DRBD must be configuredin dual-Primary mode.

This type of setup may be configured on the free XenServer, and does not require CitrixXenEssentials.

5.1.DRBDConfigurationTo configure DRBD, create /etc/drbd.conf on xenserver1 and xenserver2:

global { usage-count yes; }common { protocol C; net { allow-two-primaries; shared-secret "xenserver"; after-sb-0pri discard-zero-changes; after-sb-1pri consensus; after-sb-2pri disconnect; } disk { max-bio-bvecs 1; } startup { # leave this disabled for now # become-primary-on both; } handlers { split-brain "/usr/lib/drbd/notify-split-brain.sh";


9

} syncer { rate 12M; }}resource shared-sr { device /dev/drbd1; disk /dev/sdb1; meta-disk internal; on xenserver1 { address 192.168.10.20:7788; } on xenserver2 { address 192.168.10.21:7788; }}

Then, proceed as outlined in the DRBD User's Guide [http://www.drbd.org/users-guide/s-first-time-up.html] to initialize and synchronize your resource.

When the initial synchronization has completed, you may enable become-primary-on both;in the startup section of drbd.conf. Also, check the output of chkconfig --list drbdto make sure that the drbd init script is executed on system startup.

5.2.XenStorageRepositoryConfigurationNoteThere are currently no XenCenter equivalents to the CLI commands listed in thissection. Configuring a DRBD backed Storage Repository must always be done usingthe XenServer CLI.

In order to create a shared Storage Repository which provides less data protection during normaloperation, but supports XenMotion live migration, issue the following command:

xe sr-create host-uuid=[local host UUID] \ name-label="DRBD" type="lvm" \ device-config:device="/dev/drbd/by-res/shared-sr" \ shared=true

If successful, this command will return a newly generated UUID for this SR. Next, you can checkits status with the following commands:

xe sr-list name-label="DRBD"xe pbd-list device-config=device:\ /dev/drbd/by-res/shared-sr

5.3.XenResourcePoolConfigurationIn order to have both Xen hosts use the same shared storage and enable live migration betweenthen, we must join the hosts to form a resource pool.

When creating a new resource pool, you will select one of your servers to be the pool master. Theother host joins the freshly-created pool as a slave. It is important that the slave cannot containany shared storage at the moment it joins the pool. Thus, if you created your shared storagerepository on xenserver1, you would select it as your pool master.

5.3.1.ResourcePoolConfigurationusingXenCenterTo create a new resource pool using the XenCenter GUI, click on Pool in the menu bar, then selectNew Pool and follow the wizard's instructions. Be sure to add both your hosts to the pool andselect a resource pool master.


10

Figure3.XenCenter resource pool configuration

After you have formed your new pool, the shared storage repository will appear in the XenCentertree as no longer belonging to a specific server, but to the pool as a whole instead:

Figure4.XenCenter view of shared storage repository


11

5.3.2.ResourcePoolConfigurationusingtheXenServerCLI

To create a resource pool whose master is xenserver1, and to join xenserver2 to this poolas a slave, issue the following command on xenserver2:

xe pool-join master-address=xenserver1 \ master-username=root master-password=[password]

The pool thus created is by default unnamed. To name it, issue the following command:

xe pool-param-set name-label=[label] uuid=[pool UUID]

Tip

You may use tab completion to enter the pool UUID. Simply type uuid= andthe pool UUID will appear.

After you have created your pool, the DRBD backed storage repository will appear in the list ofshared SRs, as you may verify with the following command:

xe sr-list host="" name-label="DRBD"uuid ( RO) : 62232bcf-ce22-b3e2-1e37-9e69738d25d2 name-label ( RW): DRBD name-description ( RW): Replicated storage repository host ( RO): type ( RO): lvm content-type ( RO):

6.LimitationsandCaveats6.1.DRBDdisconnectduringpoolmaster/slavetransition

In configurations supporting XenMotion and using resource pools, as a node changes its rolein a resource pool, XenServer stops and restarts the Host Agent. This causes all managementinterfaces to be temporarily disconnected. Such a a transition occurs in the following situations:

Upon creation of the resource pool, as one node relinquishes control of its own resources,transfers them to the pool, and becomes a pool slave.

After an administrative role reversal, initiated by the xe pool-designate-new-mastercommand.

After the pool master unexpectedly leaves the cluster, and you promote the slave by issuingthe xe pool-emergency-transition-to-master command.

In any of these situations, DRBD temporarily loses its replication interface, and thus entersthe StandAlone connection state on the affected node. It must then be reconnected usingdrbdadm connect.

6.2.SplitbrainpotentialinsharedSRconfigurationsIn configurations involving resource pools, shared SRs, and dual-Primary DRBD, an interruptionof the replication network causes DRBD split brain. It is thus highly recommended that a passive,direct, back-to-back network connection is used for DRBD replication to minimize the risk ofsuch outages.


12

DRBD split brain, when it occurs, must be resolved using the appropriate procedure as outlined inthe DRBD User's Guide [http://www.drbd.org/users-guide/s-resolve-split-brain.html].

7.RemarksThis document is written and maintained by LINBIT in the hope that it will be useful, but LINBITissues no warranty about it of any kind, not even merchantibility or fitness for a particularpurpose.

LINBIT offers commercial, enterprise grade DRBD support for XenServer. For more information,please contact .

Feedback about this document is greatly encouraged. Please contact the author directly at.

Deploying DRBD with Citrix XenServerTable of Contents1.Introduction2.Installation2.1.DRBD Installation on XenServer 5.02.2.DRBD Installation on XenServer 5.5

3.Initial Configuration3.1.DRBD Replication Network Configuration3.1.1.Replication Network Configuration using XenCenter3.1.2.Replication Network Configuration using the XenServer CLI3.1.3.Firewall configuration

3.2.LVM configuration

4.Using DRBD in a XenServer setup without XenMotion support4.1.DRBD Configuration4.2.Xen Storage Repository Configuration4.3.Using the DRBD backed Storage Repository4.3.1.Managing DRBD-backed Virtual Machines using XenCenter4.3.2.Managing DRBD-backed Virtual Machines using the XenServer CLI

5.Using DRBD in a XenServer setup with XenMotion support5.1.DRBD Configuration5.2.Xen Storage Repository Configuration5.3.Xen Resource Pool Configuration5.3.1.Resource Pool Configuration using XenCenter5.3.2.Resource Pool Configuration using the XenServer CLI

6.Limitations and Caveats6.1.DRBD disconnect during pool master/slave transition6.2.Split brain potential in shared SR configurations

7.Remarks

xenserver-drbd

Documents

Transcript of xenserver-drbd