HPE Matrix Operating Environment 7.5 Recovery Management User Guide

HPE Matrix Operating Environment7.5 Recovery Management UserGuide

AbstractThe HPE Matrix Operating Environment Recovery Management user guide contains information on installation, configuration,testing, and troubleshooting HPE Matrix Operating Environment Recovery Management (Matrix recovery management).

Part Number: 5900-4385RPublished: November 2015Edition: 2

© Copyright 2015 Hewlett Packard Enterprise Development LP

The information contained herein is subject to change without notice. The only warranties for Hewlett Packard Enterprise products and servicesare set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constitutingan additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.

Confidential computer software. Valid license from Hewlett Packard Enterprise required for possession, use, or copying. Consistent with FAR12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensedto the U.S. Government under vendor's standard commercial license.

Links to third-party websites take you outside the Hewlett Packard Enterprise website. Hewlett Packard Enterprise has no control over and is notresponsible for information outside the Hewlett Packard Enterprise website.

Acknowledgments

Microsoft®, Windows®, Windows® XP, and Windows Server® are trademarks of Microsoft group of companies.

Java is a registered trademark of Oracle and/or its affiliates.

VMware, VMware Server, GSX Server, ESX Server, and VMotion are trademarks of VMware, Inc.

Adobe and Acrobat are trademarks of Adobe Systems Incorporated.

Revision history

For supported operating systems, see the HPE Insight Management Support Matrix available at Hewlett Packard Enterprise Information Library.

Publication dateDocumentation editionSoftware versionDocument part number

August 201517.55900-4385

December 201317.3.05900-3777

March 201317.2.05900-2606

September 201247.1.05900–2276

August 201237.1.05900–2276

July 201227.1.05900–2276

June 201217.1.05900–2276

February 201217.0.05900–2035

http://www.hpe.com/info/matrixoe/docs

Contents1 Matrix recovery management overview...............................................................62 Installing and configuring Matrix recovery management.....................................9

Installation and configuration overview.................................................................................................9Installation and configuration prerequisites..........................................................................................9Installing and licensing Matrix recovery management........................................................................10

Uninstalling Matrix recovery management....................................................................................10Setting up Networking.........................................................................................................................10Setting up Storage..............................................................................................................................11

General storage setup notes.........................................................................................................12P6000 storage setup notes............................................................................................................13P9000 storage setup notes............................................................................................................133PAR StoreServ Storage setup notes...........................................................................................14Creating and installing a User Defined storage adapter................................................................15

Setting up Local Site logical servers...................................................................................................17Setting up Remote Site logical servers...............................................................................................17Configuring Matrix recovery management..........................................................................................18

Matrix recovery management GUI overview.................................................................................18Matrix recovery management configuration steps.........................................................................20Matrix recovery management export and import operations.........................................................21Importing Recovery Groups with physical IO services, physical logical servers, and virtual logicalservers with RAW disks.................................................................................................................25

Import Job status......................................................................................................................25Importing of Recovery Groups with virtual IO services............................................................27

DR Protection for IO services.............................................................................................................28DR Protection of IO services configuration overview....................................................................28Configure IO properties.................................................................................................................29DR protecting IO services underlying logical servers....................................................................31Configure OO workflow for optional email notification...................................................................32Network configuration....................................................................................................................33

3 Testing and failover operations..........................................................................34Testing Recovery Groups...................................................................................................................34Enabling and disabling Maintenance Mode........................................................................................35Failover operations.............................................................................................................................36

Planned failover.............................................................................................................................36Unplanned failover.........................................................................................................................37

Target selection and parallelism during an activation operation.........................................................394 Command Line Tools.........................................................................................40

drsync tool..........................................................................................................................................40CMS configuration requirements for use with drsync.........................................................................40Configuring the dr.properties file for drsync........................................................................................41Globbing Expressions.........................................................................................................................43Using the drsync command line tool...................................................................................................43

5 Dynamic workload movement with CloudSystem Matrix...................................45Capabilities and limitations.................................................................................................................46

Supported platforms......................................................................................................................49Overview of physical to virtual cross-technology configuration..........................................................49

Configuring logical servers for movement between physical and virtual targets...........................49Configuring logical servers for movement between dissimilar physical servers...........................52

Configuring and managing portable OS images.................................................................................52Portable Images Storage Assistant (PISA)...................................................................................52

Contents 3

Portable Images Network Tool (PINT)...........................................................................................53Configuring and managing cross-technology logical servers.............................................................53

Portability groups...........................................................................................................................53Defining cross-technology logical servers.....................................................................................55

Placing a logical server into a portability group........................................................................55Storage definition.....................................................................................................................57

Moving between technologies.......................................................................................................58Target attributes.............................................................................................................................59Moving between blade types.........................................................................................................60

Managing DR Protected cross-technology logical servers in a Matrix recovery managementconfiguration.......................................................................................................................................60

Setting a failover target type preference.......................................................................................606 Issues, limitations, and suggested actions........................................................62

Limitations...........................................................................................................................................62Hyper-V support limitation for bidirectional configuration..............................................................62No automatic synchronization of configuration between sites.......................................................62Matrix recovery management job information is not preserved in certain scenarios....................62

Minor issues........................................................................................................................................62Firefox browser cannot be used for site export operations...........................................................62ESX configuration setting required for VMFS datastores of Matrix recoverymanagement managedlogical servers to be visible at Remote Site...................................................................................63Activation or deactivation job hangs..............................................................................................63Identical configuration of logical servers between sites................................................................63One RAID Manager instance per P9000 Storage Management Server and One RAID Managerinstance per P9000 device group..................................................................................................64CLX/ P9000 software must be installed on a separate Windows system.....................................64One active Matrix recovery management configuration operation at any point in time.................64Site delete operation in Matrix recovery management does not remove HPE SIM tools..............64Using SPM with Remote Copy Group requirement.......................................................................65

7 Troubleshooting.................................................................................................66Configuration troubleshooting.............................................................................................................66Configuration error messages............................................................................................................69Warning messages.............................................................................................................................72Matrix recovery management Job troubleshooting.............................................................................73Failover error messages.....................................................................................................................76Matrix recovery management log files................................................................................................77DR Protected IO services troubleshooting..........................................................................................77

DR Protected IO services configuration troubleshooting...............................................................78DR Protected IO services failover troubleshooting........................................................................80

Hyper-V based IO service or logical server fails activation................................................................808 Support and other resources ............................................................................82

Accessing Hewlett Packard Enterprise Support.................................................................................82Accessing updates..............................................................................................................................82Security bulletin and alert policy for non HPE owned software components......................................82Registering for software technical support and update service..........................................................83

How to use your software technical support and update service..................................................83HPE authorized resellers....................................................................................................................83Websites.............................................................................................................................................84Customer self repair...........................................................................................................................84Remote support..................................................................................................................................84Related information.............................................................................................................................84

Matrix recovery management documentation...............................................................................85Documentation feedback....................................................................................................................85

4 Contents

A Recover a CSV from an online pending state...................................................86Glossary...............................................................................................................87Index.....................................................................................................................90

Contents 5

1 Matrix recovery management overviewMatrix recovery management is a component of the HPE Matrix Operating Environment thatprovides disaster recovery protection for logical servers and for Matrix infrastructure orchestrationservices. Logical servers and Matrix infrastructure orchestration services (IO services) that areincluded in a Matrix recovery management configuration are referred to as DR Protected logicalservers and IO services. A DR Protected logical server can run on a physical machine (c-Classblade) or on a virtual machine hosted by a hypervisor. When a DR Protected logical server isrunning on a c-Class blade equipped with Virtual Connect, it is referred to as a VC-hosted logicalserver. When a DR Protected logical server is running on a virtual machine under the control ofa hypervisor, it is known as VM-hosted logical server.A Matrix recovery management configuration consists of two sites, each managed by the MatrixOperating Environment.The site where the Central Management Server (CMS) you are loggedinto is located is referred to as the Local Site, and the other site in the Matrix recovery managementconfiguration is referred to as the Remote Site. Matrix recovery management pairs symmetricallyconfigured logical servers or IO services across the two sites. One logical server or IO servicein the pair is activated at one site, while the other (peer logical server or IO service) is deactivatedat the other site. The boot images of these DR Protected logical servers and IO services, includingapplications code and data, reside on disk array volumes. The source volumes at the site whereone of the logical servers or IO services in the pair is activated are replicated at the other sitewhere its peer logical server or IO service is deactivated. These volumes are part of a StorageReplication Group that uses storage array supported replication. One or more logical servers orIO services can be associated with a single Storage Replication Group — this is referred to asa Recovery Group.A Matrix recovery management configuration consists of two sites, each managed by the MatrixOperating Environment. The site where the Central Management Server (CMS) you are loggedinto is located is referred to as the Local Site, and the other site in the Matrix recovery managementconfiguration is referred to as the Remote Site. Matrix recovery management pairs symmetricallyconfigured logical servers or IO services across the two sites. One logical server or IO servicein the pair is activated at one site, while the other (peer logical server or IO service) is deactivatedat the other site. The boot images of these DR Protected logical servers and IO services, includingapplications code and data, reside on disk array volumes. The source volumes at the site whereone of the logical servers or IO services in the pair is activated are replicated at the other sitewhere its peer logical server or IO service is deactivated. These volumes are part of a StorageReplication Group that uses storage array supported replication. One or more logical servers orIO services can be associated with a single Storage Replication Group -- this is referred to as aRecovery Group.Recovery Group Sets can be failed over from one site to the other site in a Matrix recoverymanagement configuration. For example, if a disaster occurs at the Local Site, the administratorat the Remote Site can trigger a failover for all of the Recovery Groups that were activated atthe Local Site by activating them at the Remote Site. This prepares the storage associated withthe deactivated DR Protected logical servers and IO services at the Remote Site for read-writeaccess, and activates those logical servers and IO services.

6 Matrix recovery management overview

Figure 1 Recovery Group Sets

Features and benefits of Matrix recovery management

• Provides an automated failover mechanism for DR Protected logical servers, DR ProtectedIO services, and associated storage.

• Supports standalone Hyper-V.

• Supports Microsoft Hyper-V VMs in a clustered configuration with Cluster Shared Volumes(CSV).

• Supports online configuration using the Matrix recovery management import process.

• Supports ESXi VMs.

• Supports flexible cross-technology logical servers. VC-hosted logical servers can fail overto become VM-hosted logical servers, and VM-hosted logical servers can fail over to VChosts.

NOTE: Cross-technology servers are limited to logical servers created using Matrix OEvisualization manager. It does not support Matrix infrastructure orchestration services.

• Supports multiple logical servers and IO services in separate Recovery Groups. However,a single Recovery Group cannot have both logical servers and IO services.

• Supports bidirectional failover of Recovery Group Sets between two sites, allowing bothactivated and deactivated Recovery Groups to exist at the same site.

• Supports HPE P6000 Continuous Access Software storage replication in synchronous andasynchronous modes.

• Supports HPE P9000 Continuous Access Software storage replication in synchronous,asynchronous, and asynchronous journal modes.

• Supports HPE 3PAR StoreServ synchronous and asynchronous data replication.

• Supports integration with the remote failover features of installed storage replication productssupported by Matrix OE, other than P6000,P9000, or 3PAR StoreServ. Integration of thesestorage replication products is supported through User Defined storage adapter interface.

• Includes Recovery Group startup order settings that let you determine which RecoveryGroups are recovered first during a site failover.

7

• Includes a Copy feature that makes it easy to create multiple Storage Replication Groupswith the same configuration parameters.

• Includes drsync, a command line tool that synchronizes logical servers between Local andRemote Sites. For more information about the tool, see “Command Line Tools” (page 40).

• Integrates with Storage Replication Support (provided by the HPE Storage ProvisioningManager) and Matrix OE storage pool entry configuration (which includes specific flags toindicate the storage pool entry is enabled for disaster recovery, and individual volumes areconfigured for disaster recovery).

• Includes support for automatic configuration of storage during logical server creation and/orsynchronization while using SPM and 3PAR StoreServ Storage replication feature.

• Supports a maximum of 3500 DR protected logical servers, out of which you can configurea combination of may be 500 virtual connect (physical) logical servers, 1500 Hyper-V VMlogical servers, or up to 3500 ESXi VM logical servers, but you must not exceed the total of3500 logical servers.Examples:1. 3000 ESX VM hosted logical servers and 500 VC hosted logical servers2. 3200 ESX VM hosted logical servers and 300 VC hosted logical servers3. 3500 ESX VM hosted logical servers4. 1500 Hyper-V VM hosted logical servers5. 750 Hyper-V IO services each configured with 2 logical servers, 750 VMware IO services

each configured with 2 logical servers, and 500 VC hosted logical servers.

• Supports multiple datastores within the same replication group.By reading this Matrix operating environment recovery management user guide,, you will gain abetter understanding of Matrix recovery management concepts and configuration testing. TheMatrix recovery management graphical user interface (GUI), online help, and tooltips providetask-specific guidance.

8 Matrix recovery management overview

2 Installing and configuring Matrix recovery managementThis chapter contains sections on Matrix recovery management installation prerequisites,networking setup, storage setup, logical server setup, Matrix recovery management configuration,export and import operations, and DR Protection for IO services.

IMPORTANT: If you intend to create DR Protected IO services, see “DR Protection for IOservices” (page 28) before starting the Matrix recovery management installation and configurationprocess.

Installation and configuration overviewThe following Matrix recovery management installation and configuration overview includes linksto information on each step in the process.1. Confirm that all Matrix recovery management installation and configuration prerequisites

have been met. See “Installation and configuration prerequisites” (page 9).2. Install and license Matrix recovery management. See “Installing and licensing Matrix recovery

management” (page 10).3. Confirm that you have a supported networking configuration. See “Setting up Networking”

(page 10).4. Configure storage. See “Setting up Storage” (page 11).5. Configure for DR Protection. See “Setting up Local Site logical servers” (page 17).6. Configure logical servers at the Remote Site. See “Setting up Remote Site logical servers”

(page 17).7. Configure Matrix recovery management. See “Configuring Matrix recovery management”

(page 18).8. Configure DR Protection for IO services. See “DR Protection for IO services” (page 28)

Installation and configuration prerequisitesMatrix recovery management is the Matrix Operating Environment component that provides therecovery management capability. The Matrix Operating Environment and dependent softwaremust be installed on the Central Management Server at the Local Site and the Remote Sitebefore Matrix recovery management can be installed. For more information, see the InsightManagement installation and configuration guide available at Hewlett Packard EnterpriseInformation Library.Confirm that both the Local Site and the Remote Site meet the support requirements specifiedfor Matrix recovery management in the Insight management support matrix, including supportedservers, storage, browsers, operating systems, databases, and hypervisors. The Insightmanagement support matrix is available at Hewlett Packard Enterprise Information Library.

NOTE: Following are the assumptions:• Networking and storage replication links are present between the Local Site and the Remote

Site.• When planning for a disaster recovery solution using Matrix recovery management with

VMware hypervisor based IO services, you must ensure that the hosts at both the primarydatacenter and the DR datacenter match. For example, the hosts with visibility to thereplicated volume that the IO service uses can be selected to activate the replica servicesat the DR datacenter and must be running a version equal to or higher than the primary hoston which the original IO service is created. For example, if primary datacenter IO servicewas created on an ESX 5.5 host, the DR site ESX servers that have the replicated datastoremapped must be running ESX 5.5 or later.

Installation and configuration overview 9




Installing and licensing Matrix recovery management1. Install theMatrix Operating Environment and dependent software on the Central Management

Server (CMS) at the Local Site and the Remote Site.2. Discover the managed infrastructure at each site from HPE System Insight Management.3. Apply the license for Matrix recovery management using the insight managed system setup

wizard.For more information, refer to the Matrix operating environment getting started guide and theinsight managed system setup wizard getting started available at HPE Information Library.

Uninstalling Matrix recovery managementUse the Windows Add/Remove Programs feature as follows:1. Select recovery management, then click Remove.2. Wait until the Matrix recovery management product no longer appears in the list.

Setting up NetworkingWith the assumption that networking links are present between the Local Site and the RemoteSite. You can use Matrix recovery management in a variety of networking configurations, but itis important that you take note of the following Matrix recovery management networkingconfiguration parameters:• In a Matrix recovery management configuration, workloads can be running on DR Protected

logical servers at both sites. A Recovery Group can only be activated (workloads running)on one site at any time, but it can be activated at either site such that workloads associatedwith different Recovery Groups can be running at both sites simultaneously. For this reason,network services, such as DNS, DHCP, WINS, and AD must be available locally at bothsites. If one site becomes inoperative due to a disaster, network services continue to beavailable at the other site based on the native disaster recovery capability in these services.This ensures that workloads can be failed over from the failed site to the other site in theMatrix recovery management configuration. Matrix recovery management must not be usedto failover network services.

NOTE: The Matrix recovery management “startup order” feature is intended to start upcritical applications first, not to ensure that startup dependencies are met between applicationsand infrastructure services such as networking.

• Matrix recovery management does not perform DNS updates or update the IP configurationof recovered logical servers during a failover operation. Your Network Administrator isresponsible for making the necessary modifications to ensure that network services areavailable if you configure a logical server to use a different IP or subnet at each site in theMatrix recovery management configuration.

• When running on physical targets (VC-hosted) , Matrix recovery management does notensure that logical servers use the same MAC addresses at both sites. When running onVMware ESXi hosted virtual targets, Matrix recovery management ensures that logicalservers use the same MAC address at both sites. Your Network Administrator needs to planfor this in the networking configuration for DR Protected logical servers, if you are usingDHCP for fixed IP addresses.

NOTE: For MAC address details for cross-technology logical servers (logical servers thatare capable of running on both VC hosts and VM hosts), see Dynamic workload movementwith CloudSystem Matrix.

• When running on Virtual Connect hosted physical targets, the Portable Images NetworkTool (PINT) must be used to prepare the server image to execute on targets with different

10 Installing and configuring Matrix recovery management


network interface configurations and MAC addresses. To use PINT, the Local and RemoteSites must be on the same network, and the OS image must be a Windows or Linux versionthat is supported by Matrix recovery management and PINT. PINT ensures that the staticnetwork configuration from the source server is successfully transferred to the destinationserver network interfaces, despite the different environment. The executables and READMEare in the C:\Program Files\HP\Insight Control service migration\PI\PINTfolder, where <SMP> is the folder where Insight Control server migration is installed. Copythe executable cp011231.exe to the physical server where the image is currently running.Run cp011231.exe to install PINT and start the PINT service.For more information about supportedWindows or Linux versions and Insight Control servermigration, see the Insight Management getting started guide at HPE Information Library .

• If the Local Site and corresponding Remote Site managed servers share a common subnet,you must ensure that there is no conflict between MAC addresses assigned by HPE VirtualConnect Enterprise Manager (VCEM). For example, if the default address range providedby VCEM is used at both sites, conflict can be avoided by using the VCEM “exclusion ranges”feature. For example, on the Local Site CMS, exclude addresses from 00-21-5A-9B-00-00to 00-21-5A-9B-FF-FF, and on the Remote Site CMS, exclude addresses from00-21-5A-9C-00-00 to 00-21-5A-9C-FF-FF.

• For DR protected logical servers and IO services, ESX port group names and Hyper-V virtualnetwork names must be identical at the Local Site and the Remote Site.

Setting up StorageMatrix recovery management depends on storage array replication to enable failover of logicalservers. The storage replication links are present between the Local Site and the Remote Site.To set up Matrix recovery management Storage Replication Groups:1. Boot and data LUNs for VC-hosted or VM-hosted logical servers that are to be DR Protected

must be replicated using supported storage replication methods, for example, P6000Continuous Access, 9000 Continuous Access, or 3PAR StoreServ remote copy groups.. Ifyou are using Matrix OE supported storage other than P6000, P9000, or 3PAR StoreServ,you can useUser Defined storage adapter interface to integrate storage with Matrix recoverymanagement GUI. The integration results in the Storage server type drop-down list displayingthe newly added storage.

NOTE: You have to use the specific storage solutions replication software to set up storagereplication groups. For more information, see “Creating and installing a User Defined storageadapter” (page 15).

• If a DR Protected logical server at the Local Site is VC-hosted, the replicated boot anddata LUNs on the array at the Remote Site must be presented to the correspondingrecovery logical server.

• If a DR Protected logical server at the Local Site is VM-hosted, the replicated boot anddata LUNs on the array at the Remote Site must be presented to the VM host (forexample, ESX or Hyper-V) at the Remote Site that is targeted to run the recovery logicalserver (for example, ESX guest or Hyper-V).

Setting up Storage 11


NOTE:• Each Recovery Group has a single Storage Replication Group that is used by the

logical servers in that Recovery Group only.

IMPORTANT: All boot and data LUNs used by these logical servers must be includedin the same Storage Replication Group.

• A Storage Replication Group is a set of storage LUNs on a particular disk array that arereplicated with write order preserved. This corresponds to the P6000 Continuous AccessDR Group concept, the P9000 Continuous Access consistency group concept, and the3PAR StoreServ remote copy replication concepts.

2. Create one Storage Replication Group for each set of logical servers that will be includedin a Recovery Group.

3. Record the following details about the Storage Replication Group configuration. Thisinformation is required when you configure Matrix recovery management for replicatedstorage:• Local and Remote Site storage identifiers, for example, a P6000 storage array WWN

or a P9000 array serial number or the 3PAR StoreServ Storage array serial number.

NOTE: In the same way that conflicts in the configuration of MAC addresses at theLocal and Remote Sites are avoided in “Setting up Networking” (page 10), conflict mustalso be avoided in the configuration of WWNs, if the WWNs are not private to theirrespective sites. The same technique using VCEM exclusion ranges is available forarray WWN configuration.

• Storage Management Server FQDN names and credentials for the Local and RemoteSites, for example, P6000 Command View server name and credentials to access theCommand View server

• Storage Replication Group name given for the boot and data LUNs of the logical serversthat will be part of the same Recovery Group, for example, the P6000 DR Group name

NOTE:

◦ P6000, P9000, and User Defined Storage Replication Groups must use the sameStorage Replication Group name at the Local Site and the Remote Site.

◦ 3PAR StoreServ remote copy Storage Replication Groups will have different namesat the Local Site and the Remote Site.

• Storage port WWN and LUN for the replicated volume(s), in the case of raw LUNs usedby DR Protected logical servers

General storage setup notes• For information on storage setup of cross-technology logical servers (logical servers capable

of being VC-hosted or VM-hosted), see: Dynamic workload movement with CloudSystemMatrix.

• For a list of supported storage, see the Insight Management Support Matrix at HPEInformation Library .

• Hyper-V virtual machines in clustered environments must be stored on cluster sharedvolumes.




P6000 storage setup notes• With 6000 Continuous Access Software storage replication, when an asynchronous replication

group is used in a Recovery Group and an unplanned Recovery Group failover occurs, afull copy of the new source vDisks is automatically duplicated on the new destination vDiskswhen the failed site recovers. If a failure occurs in the middle of this full copy operation, thedata on the new destination vDisks could be corrupted. To protect the new destination vDisks,you must enable the P6000 Command View auto-suspend setting to prevent an automaticfull copy operation from occurring. To protect the new destination vDisks, you must back upthe data on them before you manually run a full copy operation.To prevent a full copy of the new source vDisks to the new destination vDisks after thefailover of asynchronous Storage Replication Groups, Matrix recovery managementautomatically sets the mode of all asynchronous Replication Groups to synchronous priorto storage failover and then resets the mode to asynchronous after the storage failover iscompleted. The storage link must be up and both the Local Site and Remote Site arraysmust be managed by the same Command View server for Matrix recovery management toperform this operation.Under rare failure conditions, the mode of a Storage Replication Group can be left insynchronous mode requiring a manual intervention to reset the mode to asynchronous.For more information, see the following:

◦ Click Manuals tab to view the P6000 Continuous Access Software documentationavailable at http://www.hpe.com/portal/site/hpesc.

◦ ClickManuals tab to view the P6000 Command View Software documentation availableat http://www.hpe.com/portal/site/hpesc.

• If the password for a Storage Management Server is changed, take the following actions torefresh the Storage Management Server password on the CMS and in the Matrix recoverymanagement configuration:1. From the tools menu, select Discover to discover the Storage Management Server

using the changed password.2. Go to the Matrix recovery management user interface Storage Management Servers

tab.3. Select the Storage Management Server that has the changed password, and click Edit.4. Select the Refresh SIM Password box and click Save.

P9000 storage setup notes• When P9000 Continuous Access Software storage replication is used, Matrix recovery

management depends on the P9000 Cluster Extension Software command-line interface(CLI) to manage storage replication. The P9000 Cluster Extension Software CLI must beinstalled on a standalone Windows system. P9000 Cluster Extension Software depends onP9000 RAIDManager Software to manage P9000 storage replication. P9000 RAIDManagerSoftware instances and configuration files must be configured to manage various device


http://www.hpe.com/portal/site/hpesc


groups that are configured in Matrix recovery management. For more information, see thefollowing:

◦ ClickManuals tab to view the P9000Cluster Extension Software documentation availableat http://www.hpe.com/portal/site/hpesc

◦ ClickManuals tab to view the P9000 RAID Manager Software documentation availableat http://www.hpe.com/portal/site/hpesc

• If the password for a Storage Management Server is changed, take the following actions torefresh the Storage Management Server password on the CMS and in the Matrix recoverymanagement configuration:From the tools menu, select Discover to discover the Storage Management Server usingthe changed password.1. Go to the Matrix recovery management user interface Storage Management Servers

tab.2. Select the Storage Management Server that has the changed password, and click Edit.3. Select the Refresh SIM Password box and click Save.

3PAR StoreServ Storage setup notes• When 3PAR StoreServ remote copy replication is used, Matrix recovery management

depends on the 3PAR StoreServ Cluster Extension Software command-line interface (CLI)to manage storage replication. The 3PAR StoreServ Cluster Extension CLI in turn dependson the 3PAR StoreServ InForm Command Line Software. Both must be installed on theCentral Management Servers where Matrix recovery management is installed. For moreinformation, see the following:

◦ Click Manuals tab to view the 3PAR StoreServ Cluster Extension Softwaredocumentation available at http://www.hpe.com/portal/site/hpesc

◦ Click Manuals tab to view the 3PAR StoreServ InForm Command Line Softwaredocumentation available at http://www.hpe.com/portal/site/hpesc

• You need an encrypted password file to manage storage replication on an 3PAR StoreServStorage system. An encrypted password file can be created by running 3PAR StoreServInForm Command Line Software commands. For more information, see the setpasswordcommand in the 3PARStoreServ InFormCommand Line Reference. The encrypted passwordfor each storage array must be present on both CMSs in the C\Program Files\HP\Insight Recovery\STORAGE\3PAR\conf directory whereMatrix recovery managementis installed. The encrypted password file replaces the need for a user name, domain name,and password required with other types of storage management servers.

• The encrypted password file for both the Local Site and Remote Site Inserv storage serversmust be available on the CMS at each site, and the name of the password file must be thesame on the CMS at each site.

• If you upgrade the 3PAR StoreServ InForm Command Line Software on the CMS with asoftware version that is supported by Matrix OE and 3PAR StoreServ Cluster ExtensionSoftware, you must modify a property in the Matrix recovery management properties file.Change the INFORM_CLI_VERSION property in conf\hp_ir.properties where Matrixrecovery management is installed on the CMS. The default value of the property is set to3.1.2.






• If you upgrade the 3PAR StoreServ clx/cli, you have to execute cli.exe andclx3parconfig.exe to accept or add the certificates back to the exclusion list. For moreinformation about adding certificates, see “Troubleshooting” (page 66).

• Starting with version 7.5, the Matrix recovery management supports automatic storageconfiguration with Storage Provisioning Manager (SPM) templates with Remote Copyrequirements when using 3PAR StoreServ Storage systems. For more information on SPMtemplates, see the Storage Provisioning Manager (SPM) user guide.

NOTE: For 3PAR periodic (asynchronous) remote copy, the manual failover action will notsynchronize the data in the remote copy group volumes. To prevent data loss, synchronize theremote copy groups before performing stop and failover operations on the remote copy groups.

Creating and installing a User Defined storage adapterMatrix recovery management provides a User Defined storage adapter interface specification toenable one-step Matrix recovery management failover capability for storage types that aresupported by Matrix OE but not yet integrated with Matrix recovery management.OverviewMatrix recovery management can be configured to invoke storage replication managementcommands for non-integrated storage when various storage configuration or failover operationsare invoked. The User Defined storage adapter specification for non-integrated storage definescommands to:• Validate Storage Management Server information when Storage Management Servers for

non-integrated storage are configured using the Matrix recovery management GUI.• Validate Storage Replication Group information when Storage Replication Groups that use

non-integrated storage are configured using the Matrix recovery management GUI.• Failover Storage Replication Groups when logical servers that use non-integrated storage

are failed over using the Matrix recovery management Activate operation.Managing non-integrated storage with Matrix recovery managementIf your DR Protected logical servers use a non-integrated storage system that is supported byMatrix OE, and you want Matrix recovery management to automatically invoke storage failoverfor the non-integrated storage using the Matrix recovery management Activate operation, youmust:1. Implement and thoroughly test the three commands defined in the “User Defined storage

adapter interface specification”, then perform testing at the Local Site and at the RemoteSite.

2. Create a new subdirectory under the STORAGE directory where Matrix recovery managementis installed. Give the subdirectory a name that identifies the storage type being managed.This name will appear in drop-downmenus in the Matrix recovery management GUI. Performthis step at the Local Site and at the Remote Site.

NOTE: The name of the subdirectory must be exactly the same at the Local Site and atthe Remote Site.

3. Place your implementation of the User Defined storage adapter commands in the subdirectorycreated for the storage type. Perform this step at the Local Site and at the Remote Site.

4. Define local and remote Storage Management Servers for the non-integrated storage byselecting the storage type from the drop-down menu in the Storage Management Serverstab in the Matrix recovery management GUI. The storage type in the drop-down menu hasthe same name as the subdirectory created in step 2.

5. Using management tools for the non-integrated storage system, create a Storage ReplicationGroup for the storage used by the logical servers that will be DR Protected.


6. Create a Storage Replication Group for the non-integrated storage by using the Matrixrecovery management GUI. The storage type in the drop-down menu has the same nameas the subdirectory created in step 2.

7. Create a Recovery Group by using the Matrix recovery management GUI, and associatethat Recovery Group with the Storage Replication Group for the non-integrated storage.

8. Perform a Matrix recovery management Export operation at the Local Site to generate anexportconfig file, then perform an Import operation to import that exportconfig fileat the Remote Site.

User Defined storage adapter interface specificationThe following commands are defined in the User Defined storage adapter interface specification:• validatesms.cmd—Validates a Storage Management Server during configuration

• validatesrg.cmd—Validates a Storage Replication Group during configuration

• failoversrg.cmd—Fails over a StorageReplicationGroupwhile RecoveryGroup activationoccurs

Command-line arguments

• For validatesms.cmd:sms_name=<name of a Storage Management Server>sms_username=<login name for a Storage Management Server>

• For validatesrg.cmd:sms_name=<name of the Storage Management Server at the Local Site>sms_username=<login name for the Storage Management Server at the Local Site>srg_name=<name of the Storage Replication Group to be validated>local_storage_id=<unique identifier of the storage system at the Local Site>remote_storage_id=<unique identifier of the storage system at the Remote Site>

NOTE: Volumes in the Storage Replication Group (identified by srg_name) are replicatedbetween the local storage system (identified by local_storage_id) and the remote storagesystem (identified by remote_storage_id).

• For failoversrg.cmd:local_sms_name=<name of the Storage Management Server at the Local Site>local_sms_username=<user name for the Storage Management Server at the Local Site>local_storage_id=<unique identifier of the storage system at the Local Site>remote_sms_name=<name of the Storage Management Server at the Remote Site>remote_sms_username=<user name for the Storage Management Server at the Remote Site>remote_storage_id=<unique identifier of the storage system at the Remote Site>srg_name=<name of the Storage Replication Group to be failed overuse_non_current_data=<yes or no>

NOTE: A value of yes requires the Storage Replication Group to be failed over even incases where the data at the destination site may not be current. A value of no requiresstorage failover to fail if the data at the destination is not current.

Example invocations of Matrix recovery management User Defined adapterimplementation

• During Storage Management Server configuration in Matrix recovery management:<Matrix recovery management installed directory>/STORAGE/EMC/validatesms.cmd sms_name=EMC_SMS1 sms_username=admin

• During Storage Replication Group configuration in Matrix recovery management:<Matrix recovery management installed directory>/STORAGE/EMC/validatesrg.cmd sms_name=EMCSE1 sms_username=admin local_storage_id=emc_id1 remote_storage_id=emc_id2 srg_name=SRG1

• During an Activate operation in Matrix recovery management:<Matrix recovery management installed directory>/STORAGE/EMC/failoversrg.cmd local_sms_name=EMC_SMS1 local_sms_username=admin local_storage_id=emc_id1 remote_sms_name=EMC_SMS2 remote_user_name=admin remote_storage_id=emc_id2 srg_name=SRG1 use_non_current_data=yes


Command return codeCommands must return 0 on successful completion, and a nonzero error code indicates failure.Multiple User Defined storage adaptersMatrix recovery management supports multiple User Defined storage adapters to coexist in aMatrix recovery management configuration. For each User Defined storage adapter type, youcan create a new subdirectory and place your implementation of the three commands for thatstorage type. For example, to add a storage adapter named EMC, create an EMC subdirectoryunder <Matrix recovery management installed directory>/STORAGE and copy allthree commands to the newly created directory:• <Matrix recovery management installed

directory>/STORAGE/EMC/validatesms.cmd

• <Matrix recovery management installeddirectory>/STORAGE/EMC/validatesrg.cmd

• <Matrix recovery management installeddirectory>/STORAGE/EMC/failoversrg.cmd

If your implementation of the storage adapter commands requires passwords to manage storagereplication, the storage adapter command implementation must handle passwords securely. Itis your responsibility to encrypt/decrypt passwords while saving and retrieving them.

Setting up Local Site logical serversThe following conditions must be met before you can configure a logical server for DR protection.1. You must ensure that the logical server is associated with SAN based storage.2. A logical server must have been activated at least once.3. An operating system and applications must be installed on a logical server.Logical servers that meet these conditions appear in the Available LS(s) list in the Local SiteMatrix recovery management configuration GUI, even if they are deactivated after they havebeen activated for the first time.For sites with a large number of logical servers, partitioning logical servers into portability groupscan reduce activation time during failover. To improve activation time, it is recommended to createportability groups based on clusters. For example, if you have 5 ESXi clusters each with 10 hosts,then you can create a portability group for each of the 5 clusters, resulting in improved activationtime. Hewlett Packard Enterprise recommends that the portability group associated with logicalservers on both the Local Site and the Remote Site be limited to a subset of virtual machinehosts and virtual connect blades that are capable of hosting these logical servers. For additionalinformation on configuring portability groups, see Logical servers→Menus & screens→Manageportability groups in the Matrix OE Visualization and Logical Servers online help.

NOTE: You cannot change the datastore of a VM-hosted logical server while it is managedby Matrix recovery management. To change the datastore, first remove the VM-hosted logicalserver from the Matrix recovery management configuration, then use the Logical ServersActivate operation in the Tools menu of the Visualization tab to change the datastore. Afterthe datastore is changed, follow the steps in “Setting up Storage” (page 11), “Setting upNetworking” (page 10), and “Setting up Remote Site logical servers” (page 17) to readd thelogical server to the Matrix recovery management configuration.

Setting up Remote Site logical serversStarting with 7.5 release, the import process of Matrix recovery management automatically createsRemote Site (peer) logical servers, without the need to perform manual deactivation of the LocalSite and manual failover of the storage replication group. It is recommended that you perform aDR rehearsal periodically to test logical server and IO service activation including availability of

Setting up Local Site logical servers 17

resources at the recovery site. For more information, see “Testing and failover operations”(page 34).Logical server names are searched to determine if they match with any of the logical servernames included in the Recovery Groups that are imported.

• If any matches are found, those logical servers are automatically associated with the logicalservers which has the same name.

• If no match is found for an imported logical server name, you can choose another logicalserver name from the drop-down list to be associated with the logical server from the exportfile.

• If IMPORT_STRATEGIES is set to COPY_LS in the dr.properties file, you can select theempty option from the drop-down list and Matrix recovery management will create a logicalserver with the same name at the Remote Site.

NOTE:• To avoid confusion, Hewlett Packard Enterprise recommends adopting a best practice of

using the same logical server name at the Remote Site as the one that was used for theassociated logical server at the Local Site.

• For information on cross-technology logical servers (logical servers capable of beingVC-hosted or VM-hosted), see Dynamic workload movement with CloudSystem Matrix.

IMPORTANT: For sites with a large number of logical servers, partitioning logical serversinto portability groups can reduce activation time during failover. Hewlett Packard Enterpriserecommends that the portability group associated with logical servers on both the Local Siteand the Remote Site be limited to a subset of virtual machine hosts and virtual connectblades that are capable of hosting these logical servers. To improve activation time, it isadvisable to create portability groups based on clusters. For example, if you have 5 ESXiclusters each with 10 hosts, then you can create a portability group for each of the 5 clusters,resulting in improved activation time. For additional information on configuring portabilitygroups, see Logical servers→Menus & screens→Manage portability groups in theMatrix OE Visualization and Logical Servers online help.

Configuring Matrix recovery managementAfter you install Matrix recovery management, you can launch the Matrix recovery managementGUI from the Matrix Operating Environment home page by selecting Tools and then selectingMatrix recovery management... from the drop-down menu.Use the Matrix recovery management GUI to configure Matrix recovery management, manageDR Protected logical servers and DR Protected IO services, and test failover capability.

Matrix recovery management GUI overviewThe home screen for the Matrix recovery management user interface includes tabs forconfiguration and administration tasks—see Figure 2 (page 19).


Figure 2 Matrix recovery management home screen

Matrix recovery management user interface tabs

• HomeInformation on the most recent Matrix recovery management Job is displayed at the top ofthe Matrix recovery management Home screen, including the Latest Job status:, the JobId:, the Start Time: (if the job is in progress), or the End Time: (if the job has completed).This is followed by a list of the other Matrix recovery management configuration tabs withConfigured or Not configured icons to indicate if the configuration tasks for each tab havebeen completed.

• SitesConfigure Preferred and Secondary Sites. Activate or Deactivate Recovery Groups. Edit ordelete existing Site configurations. Export or import Site configurations.

• Storage Management ServersDefine new Storage Management Servers. View, edit, or delete Storage ManagementServers.

• Storage Replication GroupsCreate new Storage Replication Groups. View configuration details for Storage ReplicationGroups. Copy, edit, or delete Storage Replication Groups.

• Recovery GroupsCreate new Recovery Groups. View configuration details for Recovery Groups. Import, editor delete Recovery Groups. Enable or Disable Maintenance Mode on Recovery Groups.

• JobsMonitor Job progress and view Job details. Cancel Jobs in progress. Restart failed Jobs.View logs for Jobs and Sub Jobs. Delete Job information.

The Matrix recovery management online help and tooltips provide answers to questions you mayhave while using the GUI.

Configuring Matrix recovery management 19

Matrix recovery management configuration stepsFigure 3 (page 20) illustrates the six-step Matrix recovery management configuration process.After the Matrix recovery management configuration process is completed at the Local Site,Matrix recovery management must be configured at the Remote Site. To simplify the RemoteSite configuration process and to ensure that both sites have synchronized configuration, theymust be configured from a single CMS. You can export the configuration to a file and copy it tothe remote CMS, from where you can perform an import operation.

Figure 3 Configuration steps

Table 1 (page 20) lists the six steps in the Matrix recovery management configuration process.

Table 1 The Matrix recovery management configuration process

DescriptionStepnumber

Local and Remote Sites are defined in this step, including naming the sites, designating CentralManagement Servers, and designating preferred target types (physical servers or virtual machines).

1

Local and remote Storage Management Servers are configured in this step. These servers managethe Storage Replication groups at the Local and Remote Site respectively.

2

Storage Replication Group information is configured in this step. In Matrix recovery management, aStorage Replication Group is a generic term for what is called as DR Group in P6000 ContinuousAccess terminology, or a Device Group in P9000 terminology.

3

Recovery Groups are configured in this step. A Recovery Group is a Matrix recovery managementconcept that pairs one or more logical servers or IO services with a single Storage Replication Group.

4

Recovery Group Sets can be failed over between Local and Remote Sites. A Recovery Group Setincludes all Recovery Groups that share the same Preferred and Secondary Sites.

Export the Matrix recovery management configuration to a file at the Local Site.5

Import.6


1. From the Sites tab, configure the Local Site.2. From the Storage Management Servers tab, configure Storage Management Servers at

the Local Site.3. From the Storage Replication Groups tab, configure Storage Replication Groups at the

Local Site.4. From the Recovery Groups tab, configure Recovery Groups at the Local Site.

NOTE: Matrix recovery management allows failover of the logical servers or IO servicesin a Recovery Group, independent of the associated Storage Replication Groups. Thiscapability is called Storage Decoupling.

5. From the Sites tab, create an export file at the Local Site. For information on export andimport parameters, see “Matrix recovery management export and import operations” (page21).

6. From the Sites tab at the Remote Site, import the Local Site Matrix recovery managementconfiguration. For more information, see “Matrix recovery management export and importoperations” (page 21).

7. Hewlett Packard Enterprise recommends testing your recovery logical servers after import.For more information, see “Testing Recovery Groups” (page 34).

Matrix recovery management export and import operationsThis section lists key points about Matrix recovery management export and import behavior.Export

• The Matrix recovery management configuration at the exporting site is included in theexportconfig file generated at the exporting site. The Matrix recovery managementconfiguration export file is named exportconfig by default. The exportconfig file issaved to a default location specified by the browser on the Administrator's system. You havethe option to change the default export configuration file name, for example, if you want tosave multiple Matrix recovery management configuration export files. You can also changethe location that the export configuration file is saved to, before completing the save operation.

• All of the Recovery Groups that you want to export from the Local Site and import to theRemote Site must be activated at the Local Site. Recovery Groups that are deactivatedwhen the export operation is performed at the Local Site are not imported at the RemoteSite. If there are no activated Recovery Groups at the exporting site, the generatedexportconfig file cannot be used at the importing site.


• DR Protected IO services belonging to Recovery Groups are exported, and they can beimported at a Remote Site, however, a replica IO service cannot be exported or imported.

• At the Secondary Site, when using Storage Provisioning Manager (SPM) storage templateswith Remote Copy requirements, the following considerations are applicable:

◦ Add disk operation and storage synchronization using Matrix recovery managementexport/import or drsync command

– Is supported only from the Preferred Site: You can add a disk to the Storagepool entry of a logical server at the Preferred Site, and propagate the change tothe peer logical server on the Secondary Site using Matrix recovery managementexport/import operation or by executing the drsync command. You cannot performan add disk operation at the Secondary Site and propagate the change to a peerlogical server at the Preferred Site.

◦ Requires the storage to be failed over to the Preferred Site for every step of the followingoperations:– Adding disk to the logical server at the Preferred Site.

– Exporting at the Preferred Site and importing at the Secondary Site.

– Synchronizing logical servers using the drsync command.

◦ Requires SPM resynchronization and refresh of logical server at both Preferred andSecondary Sites, prior to adding a disk or performing storage synchronization, but onlyafter the storage failover.– Perform SPM volume resynchronization from the Volumes GUI of SPM using

Resync→Volume option.– Perform logical server refresh fromMatrix OE Visualization using Tools→Logical

Servers→Refresh option to select the Storage Pool Entries check box and clickRefresh.

ImportThe Matrix recovery import operation is of two types:

• Site import— Done during initial configuration.

• Single Recovery Group import—Done after the initial configuration to import newRecoveryGroups and reimport modified Recovery Groups.

NOTE: Before attempting a reimport, you must delete the existing Recovery Groups.

• Only Recovery Groups that are activated at the exporting site are included in the export file.

• The Matrix recovery management import process searches the local system to determineif they match any of the logical server names included in the Recovery Groups that areimported. If any matches are found, those logical servers are automatically associated withthe logical servers which have the same name. If no match is found for an imported logicalserver name, you must choose another logical server name from the drop-down list to beassociated with the logical server from the export file. If IMPORT_STRATEGIES is set toCOPY_LS in the dr.properties file, you can select the empty option from the drop-downlist and Matrix recovery management will create a logical server with the same name at theRemote Site. For more information on dr.properties, see dr.properties (page 30).

• TheMatrix recovery management import operation automatically creates a replica IO servicefor each IO service belonging to each IO services Recovery Group being imported.


NOTE:

◦ A replica IO service from a Remote Site cannot be imported using a Matrix recoverymanagement import operation. As a result, all edits to an IO service must be done fromthe Preferred Site, and a new export file must be created and imported at the secondarysite.

• Importing of virtual (VM-hosted) IO services created from a Recoverable IO template requiresa site deactivation after the export file is generated and manual failover of the storagereplication group to the Remote Site prior to import.

• If deactivation is prohibited at the Local (Preferred) Site, you can consider using anonrecoverable IO template for provisioning and then DR protect the underlying logicalserver, which can be imported and created automatically during the import process withoutrequiring storage failover. For more information, see “DR protecting IO services underlyinglogical servers” (page 31).

NOTE: If the Matrix recovery management configuration includes Hyper-V logical serversor IO services, you must bring the cluster disk resource used for the logical server or IOservice storage offline prior to failing over storage to the Remote Site. For more informationif CSV is not offline, see “Recover a CSV from an online pending state” (page 86) onhow to recover.

• Site Import: Storage Management Server information is imported if Storage ManagementServers are not configured at the importing site. If the importing site already contains StorageManagement Server information:

◦ The site locality of each Storage Management Server configured at the importing sitemust match the site locality of a Storage Management Server in the exportconfigfile.

◦ The type of Storage Management Server must match if the name matches.

◦ For P6000, the user name and port number fields must match.

◦ For P9000, the user name and RAID instance number must match.

◦ For 3PAR StoreServ Storage system, the password file name must match.

If any one of the above items is not matched between the exporting site and the importingsite, the import operation fails.

NOTE: To manage 3PAR StoreServ remote copy, the encrypted password file for boththe Local Site and Remote Site Inserv storage servers must be available on the CMS ateach site, and the name of the password file must be the same on the CMS at each site.

• Storage Replication Group information associated with activated Recovery Groups in theexportconfig file is imported if they do not exist already at the Remote Site.

• It is recommended not to have Recovery Group with the same name in different RecoveryGroup sets as import fails.

• Storage Management Servers and Storage Replication Groups that are not referenced bythe imported Recovery Groups are not imported.

• If the Recovery Groups in the exportconfig file already exist at the importing site (RecoveryGroups with the same name are at the importing site), they are deleted and replaced withthe imported Recovery Groups. The Storage Replication Groups referenced by RecoveryGroups that are deleted are also deleted.


Single Recovery Group importIf you add or modify (edit) a Recovery Group at a Matrix recovery management site, there is asingle Recovery Group import feature that saves time by propagating those changes to the othersite without performing a site import operation. It is recommended to use this method once theinitial site import is completed. Ensure that the Storage Replication Group associated with thenew Recovery Group already exists, or you must manually create it at the DR site beforeperforming the import operation.For example, if you add or modify (edit) a Recovery Group at the Local Site, you can import thenew or modified Recovery Group to establish its peer Recovery Group at the Remote Site withoutimporting the entire Local Site configuration. The procedure at the Local Site is the same as thesite export procedure. From the Sites tab at the Local Site you export the Matrix recoverymanagement configuration to an exportconfig file and move that file to the Remote Site. Atthe Remote Site, browse to the Storage Replication Groups tab and ensure that the StorageReplication Group already exists, or create it manually. From the Recovery Groups tab at theRemote Site, click import...→Select import file.... The ImportWizardwindow appears allowingyou to select a Recovery Group to import from the exportconfig file. See the Matrix recoverymanagement online help for more information on the import procedure for single RecoveryGroups.

NOTE:• Recovery Groups can be imported one at a time only. Youmust repeat the import...→Select

import file... procedure for each Recovery Group that you import.• If a Recovery Group in the exportconfig file has the same name as a Recovery Group

at the importing site, it is not imported. If you are importing a Recovery Group that alreadyexists in the Matrix recovery management configuration but modified already, you have todelete the Recovery Group of the same name at the importing site before you can importthe modified version of the Recovery Group.

• If a Recovery Group in the exportconfig file references a Storage Replication Group thatis being used by another Recovery Group at the importing site, the import fails.

During a reimport of a Recovery Group that contains physical IO services with deleted VC-hostedlogical servers, the physical logical severs are not deleted at the Remote Site and orphan logicalservers are created in the Matrix OE visualization. These orphan logical servers must be deletedfrom the Matrix OE visualization.

NOTE: An orphan logical server is unaware that it does not belong to IO service. To deletethe orphan logical server, you must confirm by entering YES in the text field as shown in Figure 4(page 25). This confirmation is required despite the warning message that the server is managedby Matrix OE infrastructure orchestration.


Figure 4 Orphan Logical Server

Importing Recovery Groups with physical IO services, physical logical servers, andvirtual logical servers with RAW disks

After successfully importing a Recovery Group with physical IO services, physical logical servers,or virtual logical servers with physical volumes at the recovery site, the status in the Jobs tabdisplays either Completed or Completed – Storage Configuration Required. As aresult of an import job, a new replica IO service or logical server is created, or a new physicalstorage is added.

Import Job statusAfter an import operation is complete, the Jobs tab displays either of the following status:

• Import job status Completed

• Import job status Completed — Storage Configuration Required

Import Job status CompletedThis status is displayed when the storage configuration is performed at the Preferred Site usinga Storage Pool Entry (SPE), SPM template with Remote Copy requirements, and with the SPMjob having no issues.In this scenario, the required storage configuration is automatically performed by SPM, the SPEfor Disaster Recovery Enabled and volumes Disaster Recovery Ready check boxes areautomatically selected.In theRecovery Groups tab, the imported Recovery Groups will be in Maintenance Mode, whenall the logical servers in the Recovery Group with storage pool entries have theDisaster RecoveryEnabled flag checked, and all their volumes have the Disaster Recovery Ready flag checked.


Import job status Completed - Storage Configuration RequiredThis status is displayed when the storage configuration is performed at the Preferred Site underthe following conditions:

• With a SAN Storage Entry type SPEDisaster Recovery Enabled is checked but volumes will not have Disaster RecoveryReady flag checked.

• With a SAN Catalog Storage Entry type SPE using SPM template without Remote CopyrequirementsDisaster Recovery Enabled is checked and volumes have Disaster Recovery Ready flagchecked.

• With a SAN Catalog Storage Entry type SPE using SPM template with Remote Copyrequirements, but with the SPM job at the recovery site having issuesDisaster Recovery Enabled is checked but volumes will not have Disaster RecoveryReady flag checked. Also, an error (red X) is displayed in SPE.

In the Recovery Groups tab, the imported Recovery Groups state is Maintenance Mode –Storage Configuration Required.After an import job, a Recovery Group is in Maintenance Mode when all of its logical serversSPEs have theDisaster Recovery Enabled flag checked, and at least one SPE has one volumewith the Disaster Recovery Ready flag unchecked.In the above scenarios, check the Matrix recovery management Jobs tab for the IO services orlogical servers, which require storage configuration.For each member of the Recovery Group that require storage configuration, perform the followingsteps:

• For SAN Storage Entry and for SAN Catalog Storage Entry type SPE using SPMtemplate without Remote Copy requirements1. Go to the Recovery Group member storage pool entries and manually configure the

volumes.2. After configuring each volume, select the volume Disaster Recovery Ready check

box.3. Ensure that the above steps are complete before you check the storage pool entry for

Disaster Recovery Enabled flag.

• For SAN Catalog Storage Entry type SPE using SPM template with Remote Copyrequirements• Identify the failure.

1. If related to SPM failure, then perform the following steps:a. Fix the SPM configuration issues.b. Activate the storage service using SPM.c. Perform the storage pool entries refresh using Matrix OE

Visualization→Tools→Logical Server→Activate to select the Storage PoolEntries and click Refresh.

2. If SPM cannot find the volume to import, fix theRemote Copy Group configurationand redo the import.

• Disable Maintenance ModeAfter the storage configuration is completed for all Recovery Group members, click Refreshto update the status of the storage configuration of the Recovery Group(s) inside RecoveryGroups tab. When the Recovery Group status changes from Maintenance Mode –Storage Configuration Required toMaintenance Mode, clickDisableMaintenance


Mode to disable Maintenance Mode for that Recovery Group. For more information, see“Enabling and disabling Maintenance Mode” (page 35).

NOTE:• It is not required to selectDisaster Recovery Enabled andDisaster Recovery Ready flags

in the storage pool entries at the Preferred Site.• When all SPEs haveDisaster Recovery Enabled flag not checked for all the logical servers

of a Recovery Group, the Matrix recovery management ignores the SPE volumes DisasterRecovery Ready flag status, and the Recovery Group is always in Maintenance Mode.

• Starting with release 7.5, after an upgrade from a version lower than 7.5, all preexistentSPEs will have the new Disaster Recovery Enabled flag unchecked.

• After an import job is completed, it is recommended to perform a DR rehearsal to verify ifall the imported logical servers and IO services are correctly activated.

• You cannot disable Maintenance Mode until all storage pool entries for all logical servers orIO services inside a Recovery Group are checked forDisaster Recovery Enabled and theircorresponding volumes checked for Disaster Recovery Ready.

• After storage configuration is completed, click Refresh to refresh the status of the storageconfiguration of the Recovery Group(s) inside Recovery Groups tab.

• IO provides information about storage configuration inside IO create replica log in the IORequests tab and through automatic e-mail notification when it is configured in IO.

Importing of Recovery Groups with virtual IO servicesUnlike physical IO services or physical logical servers and virtual logical servers, virtual IO servicesrequire storage failover prior to import operation. When importing Recovery Groups that containvirtual IO services, perform the following steps after exporting the Matrix recovery managementconfiguration file at the Preferred Site:1. Deactivate the Preferred Site.2. Manually failover storage to the Recovery Site.

NOTE: If the Matrix recovery management configuration includes Hyper-V logical serversor IO services, bring the cluster disk resource used for the logical server or IO service storageoffline prior to failing over storage to the Remote Site. If CSV is not offline, see “Recover aCSV from an online pending state” (page 86) on how to recover.

3. Rescan storage using VM host management tools, such as VMware Virtual Center orMicrosoft Hyper-V Management Console, to ensure that the VM host recognizes the failedover storage.The replicated disk on the Remote Site Hyper-V host must be configured with the samedrive letter that is assigned to the Local Site disk, from where it is replicated.

• In the case of a cluster shared volume, the replicated disk on the Remote Site Hyper-Vhost must be configured with the same volume path that is assigned to the Local Sitedisk, from which it is replicated.

• In the case of a cluster shared volume or shared cluster disks, the replicated disk onthe Remote Site Hyper-V host must be configured with the same cluster resource namethat is assigned to the Local Site disk, from where it is replicated.

4. Refresh Virtual Machine resources using the Logical ServersRefresh operation under Toolsof the Visualization tab in Matrix OE visualization.

5. Import a Site or a Recovery Group.6. Perform DR rehearsal by activating and deactivating the replica IO services.7. Disable Maintenance Mode.


8. Activate the Preferred Site.

DR Protection for IO servicesMatrix infrastructure orchestration (IO) provides rapid provisioning and repurposing of infrastructureservices from shared resource pools using a self-service portal. IO delivers advancedtemplate-driven design, provisioning, and ongoing operations for multi-node, multi-tier infrastructureservices.Beginning with Matrix Operating Environment 7.1, IO is integrated with Matrix recoverymanagement to provide DR Protection for IO services. IO services are capable of DR Protectionin a Matrix recovery management configuration when they are:• Deployed from an IO template.

• Associated with storage that is supported by Matrix recovery management (this includesUser Defined storage—for more information see “Creating and installing a User Definedstorage adapter” (page 15)).

IO services that are capable of being DR Protected in aMatrix recovery management configurationare called as recoverable IO services.Recoverable IO services can be selected from a drop-downmenu in Matrix recovery managementwhen you configure IO services Recovery Groups. After IO services Recovery Groups areconfigured, the IO services they contain will be DR Protected. OO workflows provide systemadministrators with automatic email notification when IO operations (for example, create service,delete service, add disk, add server, change lease) are performed on DR Protected IO servicesthat require configuration changes in Matrix recovery management.When Matrix recovery management and IO are installed on the same CMS and recoverable IOservices have been configured, you can create DR Protected IO services Recovery Groups aspart of the initial Matrix recovery management configuration process, or you can add DR ProtectedIO services Recovery Groups to an existing Matrix recovery management configuration. Theconfiguration procedure is similar.

DR Protection of IO services configuration overviewThe steps to configure Recoverable IO services are:1. Configure IO properties on Local Site and Remote Site CMS. For more information, see

“Configure IO properties” (page 29).2. Configure the OOworkflows. For more information, see “Configure OOworkflow for optional

email notification” (page 32).3. Configure network for DR Protected IO services. For more information, see “Network

configuration” (page 33).4. Create recoverable IO services templates. For more information, see the Matrix Operating

Environment Infrastructure Orchestration user guide available at Hewlett Packard EnterpriseInformation Library.

5. Configure recoverable IO services using IO. For more information, see the Matrix OperatingEnvironment Infrastructure Orchestration user guide available at Hewlett Packard EnterpriseInformation Library.






NOTE:• When you use Matrix recovery management to DR protect VMware ESX based IO

services that are deployed from an IO template using an ICVirt template or a VM templateon the vCenter server at the Local Site, an ICVirt or VM template with same name mustbe available at the Remote Site before you perform a Matrix recovery managementimport operation to import the site configuration. If an ICVirt or VM template with thesame name is not available at the Remote Site, DR protected IO services will not beimported.

• When you use Matrix recovery management to DR protect Microsoft Hyper-V based IOservices that are deployed from an IO template using an ICVirt template on the CMSat the Local Site, an ICVirt template with same name must be available at the RemoteSite before you perform a Matrix recovery management import operation to import thesite configuration. If an ICVirt template with the same name is not available at the RemoteSite, DR protected IO services will not be imported.

• When you use Matrix recovery management to DR protect IO services deployed froman IO template using an ICVirt template, VM template, or SCVMM template, the templatemay contain only a single disk.

• When you use Matrix recovery management to DR protect physical IO services that aredeployed from an IO template using automatic software deployment, a deploymentserver of the same type used at the Local Site must be available at the Remote Sitebefore you perform a Matrix recovery management import operation. If the samedeployment server type and the software is not available at the Remote Site, DRprotected IO services must be imported.

6. Add IO services to Matrix recovery management Recovery Groups. The added IO servicesmust be present in the Available Services box in the Recovery Groups tab, when youselect to create a New Recovery Group.

7. Export site configuration at the Local Site. For more information, see “Matrix recoverymanagement export and import operations” (page 21).

8. Import site configuration at Remote Site. For more information, see “Matrix recoverymanagement export and import operations” (page 21).

Configure IO propertiesTo enable DR Protection for IO services, edit the files located at C:\Program Files\HP\Matrix infrastructure orchestration\conf\hpio.properties directory and C:\Program Files\HP\Insight Recovery\conf\dr.properties.hpio.properties

• Enable DR Protection for IO services: enable.dr.protection = true

• For DR protection in a multi-tenancy environment, all IO services must have unique namesacross all organizations. Enforce this restriction by setting theforce.unique.service.names.across.organizations=true property.

• Specify the datastore used for provisioning DR Protected virtual IO services:

NOTE: DR protection of physical IO services is not supported in a multitenancyenvironment.

◦ At the Local Site, specify the volumes to use for DR Protected IO services, for example:volume.dr.list = /vmfs/volumes/ds_1;C:\\ClusterStorage\\Volume3;C:\\ClusterStorage\\Volume4

◦ At the Remote Site, specify the volumes to use for replica IO services, for example:volume.dr.replica.list = /vmfs/volumes/ds_1;C:\\ClusterStorage\\Volume3;C:\\ClusterStorage\\Volume4

DR Protection for IO services 29

NOTE: If one datastore is specified in volume.dr.list, the DR Protected virtual IOservices are provisioned on the datastore specified.If multiple datastores are specified in volume.dr.list, the DR Protected virtual IO servicesare provisioned on the datastore in volume.dr.list that is both available for that serverpool and also has the most free disk space.If multiple datastores are specified in volume.dr.list, and the IO template specifies oneof the datastores in volume.dr.list, the DR Protected virtual IO services are provisionedon the datastore specified in the IO template.

• Set the static.ip.replica.list property to reserve static IP addresses for replica IOservices at the Remote Site. The replica IP range can be a subset of the IP range configuredon IO networks UI. The IP range specified in the property file, reserved for replica list cannotbe used when local services are created at the DR site. For example, if the networksconfigured in IO at the Remote Site are subnetX and subnetY, one or more IP addressescan be reserved for replica services from those subnets. For example, reserve a new IPrange for DR protected servers by setting the property in hpio.properties file, similarto the following:static.ip.replica.list =subnetX:192.16.0.10;subnetX:192.16.0.15-192.16.0.20;subnetY:192.16.0.30-192.16.0.40;

subnetX:1024:3143::c425:b0e1:e1c0-1024::3143:c425:b0e1:e1cf;

dr.properties

• On the Local Site and Remote Site CMS specify a name identifying the associated site whereIO is running in the dr.properties file, for example: local.site = siteA at the LocalSite, and local.site = siteB at the Remote Site. The name in the dr.propertiesfile can be different than the site name configured in Matrix recovery management.

• If the service owner domain name and username on the Local Site and Remote Site are notthe same, set the owner.username.<site_a>.<owner_domain_a\\username_a>= owner.username.<site_b>.<owner_domain_b\\username_b> property to specifythe service owner domain and usernamemapping. For example, if the service owner domainname and username configured for IO on the Local and Remote Sites are domainA\userAand domainB\userB:owner.username.siteA.domainA\\userA = owner.username.siteB.domainB\\userB

Specify additional owner-user name mapping by adding to the dr.properties file. Forexample:owner.username.siteA.ARCMS9671v\\Administrator = owner.username.siteB.ARCMS9771V\\Administrator

owner.username.siteA.ARCMS9671v\\svcuser = owner.username.siteB.ARCMS9771V\\svcuser

• If the domain names on the Local Site and Remote Site are not the same, set theowner.domain.<site_a>.<owner_domain_a> =owner.domain.<site_b>.<owner_domain_b> property to specify the domain namemapping. For example, if the domain name configured for IO on the Local and Remote Sitesis domainA and domainB:owner.domain.siteA.domainA = owner.domain.siteB.domainB

• Map the organization administrator in the dr.properties file by setting theowner.username.<site_a>.<domain_a>\\<org_admin> =owner.username.<site_b>.<domain_b>\\<org_admin> property. For example, ifthe organization administrator for IO on the Local and Remote Sites is userA:owner.username.siteA.domainA\\userA = owner.username.siteB.domainB\\userA

• If the HPIO property federated.io is set to true in the hpio.props file, the dr.propsfile must have a corresponding mapping entry:


federated.siteA.<primary_CMS_siteA> = federated.siteB.<primary_CMS_siteB>

For example, if the CMS name at the Local Site was hostA.test.net and at the Remote Siteit was hostB.test.net, the mapping line would be:federated.siteA.hostA.test.net = federated.siteB.hostB.test.net

NOTE:

◦ Use this mapping whenever federated.io is set to true, even if federation is notused.

◦ Matrix recovery management mappings in the dr.properties file must not containdots while specifying the properties values (for example, test.user). Use of "." in amapping causes IO to map incorrectly and import operations on the secondary site fails.

For more information, see the Matrix Operating Environment Infrastructure Orchestration userguide available at Hewlett Packard Enterprise Information Library.

DR protecting IO services underlying logical serversUnder following instances, you may want to DR protect the underlying logical servers createdfrom an IO template:

• When an IO service is created from a nonrecoverable IO template, for which you may requireDR protection.

• When configuring virtual IO services, even a short period of downtime to configure theRemote Site is prohibited. In this scenario, you can create a nonrecoverable IO templateand DR protect the underlying logical server.

• When you use Matrix recovery management to DR protect IO services deployed from an IOtemplate using an ICVirt template, VM template, or SCVMM template having more than asingle disk.



NOTE:• IO does not support SPM templates with Remote Copy requirements when using a

nonrecoverable IO template.• Nonrecoverable IO templates do not use datastore(s) listed in the dr.volumes.list

property from the hpio.properties file. To reserve one or more datastore for use bynonrecoverable IO templates, you can add the remaining datastores to thevolumes.exclusion.list inside the same properties file. In other words, add the volumesnot listed in dr.volumes.list that are not required to be used by the new IO services tothe volumes.exclusion.list. After the volumes.exclusion.list is updated, restartthe IO service. Optionally, for virtual IO services, you can specify the desired datastore(s)in the IO template, by double clicking the Volume in the IO template and entering the StorageVolume Names(s) field as shown in Figure 5 (page 32).

Figure 5 Storage Volume Names

• Do not select the Disaster Ready Enabled flag inside the logical servers SPEs for an IOservice, which was created using a nonrecoverable IO template.

• It is not required to have IO services installed at the Remote Site when protecting underlyinglogical servers.

Configure OO workflow for optional email notificationYou have the option to configure OO workflows to provide system administrators with automaticemail notification when IO operations are performed on DR Protected IO services that requireconfiguration changes in Matrix recovery management (for example, create service, deleteservice, add disk, add server, change lease).IR delivers the IRWorkflow.zip and Send DR Config Email.xsl files to enable automaticemail notification. These two files can be found in the C:\Program Files\HP\InsightRecovery\conf\OO\repo folder on the CMS. The IRWorkflow.zip file is the OO repositoryexport zip file containing the IR workflow and the new OO system property namedHpioDrServiceActionRecipients. The Send DR Config Email.xsl file is used by theIR workflow to build the body of the DR email notification.1. On the Local and Remote Site CMS, import the workflow packaged in the IRWorkflow.zip

file using OO Studio. The workflow is imported as DR Global Service End Action


under the Library/Hewlett-Packard/Infrastructure orchestration/ServiceActions/DR folder in the OO repository.

2. Copy the Send DR Config Email.xsl file to the C:\Program Files\HP\Matrixinfrastructure orchestration\conf\OO directory on the CMS.

3. Configure OO to set up email notification. For information on setting up email notificationsee the Matrix Operating Environment Infrastructure Orchestration user guide available atHPE Information Library.

4. Edit the HpioDrServiceActionRecipients OO system property to specify emailrecipients for notification of changes to DR Protected services.

5. Modify the following property in the C:\Program Files\HP\Matrix infrastructureorchestration\conf\hpio.properties file to point to the IR workflow:oo.global.service.end.action.path = Service Actions/DR/DR GlobalService End Action

6. Restart Matrix infrastructure orchestration.

Network configurationTo allow the same IP addresses for the Preferred Site IO service and the Secondary Site replicaIO services, when the subnet is spanned between the Local and Remote Site:

• The Local Site and the Remote Site must define the same static IP address range and theIO administrator must specify an IP exclusion list in the hpio.properties file to avoid IPaddress conflicts. These IP addresses are reserved only for replica IO services and cannotbe used by any other IO service.

If the subnet is not spanned between the Local and the Remote Site, the Preferred Site IO serviceand the Secondary Site replica IO services can be assigned with IP addresses specific to eachsite.

NOTE: Matrix recovery management does not perform DNS updates or update the IPconfiguration of logical servers associated with IO services during a failover operation. YourNetwork Administrator is responsible for making the necessary modifications to ensure thatnetwork services are available if you configure a logical server to use a different IP address orsubnet at each site in the Matrix recovery management configuration. Thus, Matrix recoverymanagement does not alter the IP address in the NIC in the OS to match that. You must modifythe IP address of the OS manually. For example, if the IP address of the VM at the PreferredSite is configured to 10.10.10.1, but the Secondary Site subnet is 12.12.12.0, Matrix IO allocates12.12.12.X IP address to the replica service. But when the OS boots, you must manually configurethe IP address of the NIC to 12.12.12.X, as its Preferred Site IP address still refers to 10.10.10.X.



3 Testing and failover operationsThis chapter describes Recovery Group testing, planned failovers, and unplanned failovers usingthe Matrix recovery management Activate and Deactivate operations.

Testing Recovery GroupsThere are two ways to test Recovery Groups:• Using Maintenance Mode to test individual logical server(s) in a specific Recovery Group.

NOTE: When importing Recovery Groups containing VC-hosted logical server(s) at theRemote Site, the logical servers must be first activated and then deactivated using MatrixOE visualization, before maintenance mode can be disabled in the Matrix recoverymanagement.

• Performing a planned failover to test all Recovery Groups. See “Planned failover” (page 36)for more information.

This section focuses on using Maintenance Mode to test individual Recovery Groups.For testing purposes, you can manually place a Recovery Group into Maintenance Mode andthen activate the deactivated logical servers or inactive services belonging to that RecoveryGroup using the Matrix Operating Environment, provided the Storage Replication Group for theRecovery Group is in RW mode at the site you are testing.Maintenance Mode temporarily stops Matrix recovery management frommanaging DR Protectedlogical servers or DR Protected IO services.At the Remote Site, the logical servers being managed by Matrix recovery management areknown as recovery logical servers. Normally, a recovery logical server cannot be activated exceptby invoking an Activate operation from the Matrix recovery management user interface at theRemote Site.To use Maintenance Mode to test a Recovery Group(s) or logical server(s):1. At the Local Site where the Recovery Group(s) were first configured, move the Recovery

Group(s) to Maintenance Mode and, use the Logical Servers Deactivate operation in theTools menu of the Visualization tab in the Matrix Operating Environment to properly shutdown the logical servers in the Recovery Group. For IO services, use theDeactivate ServersIO service operation in the Matrix infrastructure service to shut down the IO services in theRecovery Group.

NOTE: If the Matrix recovery management configuration includes Hyper-V logical serversor IO services, bring the cluster disk resource used for the logical server or IO service storageoffline prior to failing over storage to the Remote Site. If you did not bring the CSV offline,see “Recover a CSV from an online pending state” (page 86) for how to recover.

2. Log in to the Remote Site where the import operation was performed, use the appropriatestorage management tools, (for example, P6000 Continuous Access Software), to fail overthe storage corresponding to the Recovery Group to the Remote Site.

3. If there are logical servers or IO services in the Storage Replication Group that will beactivated on VM hosts during the test:a. Rescan storage using VM host management tools, for example, VMware Virtual Center,

to ensure that the VM host recognizes the failed over storage.b. Refresh Virtual Machine resources using the Logical Servers Refresh operation in the

Tools menu of the Visualization tab in Matrix OE visualization.

34 Testing and failover operations

4. Place the Recovery Group into Maintenance Mode at the Remote Site using the EnableMaintenance Mode button in the Matrix recovery management Recovery Groups tab.For logical servers, use the Logical Servers Activate operation in the Tools menu of theVisualization tab in Matrix OE visualization to activate the logical servers in the RecoveryGroup at the Remote Site. Depending on the type of logical server, the activation may beon VC blades, VM hosts, or both. For IO services, select the IO service from the list ofservices in the upper portion of the IO Services tab, and then select Activate Servers fromthe Server Actions drop-downmenu to activate the IO services in the Recovery group.Whenthe test is complete, properly shut down the operating systems, then deactivate the logicalservers or IO services.

NOTE: If the Matrix recovery management configuration includes Hyper-V logical serversor IO services, bring the cluster disk resource used for logical server or IO service storageoffline prior to failing over storage. If you did not bring the CSV offline, see “Recover a CSVfrom an online pending state” (page 86) for how to recover.

5. Log in to the Remote Site CMS, take the Recovery Group out of MaintenanceMode by usingthe Disable Maintenance Mode button in the Matrix recovery management RecoveryGroups tab.

6. Log in again to the Local Site CMS and either repeat the storage failover, rescan and refresh(if needed) sequences, then activate the logical servers in the Recovery Group using MatrixOE visualization or do a site activation, if you want to activate the full recovery group set.

When you import a Matrix recovery management configuration at the Remote Site, all RecoveryGroups that are imported have Maintenance Mode enabled by default, or the status ofMaintenance Mode – Storage Configuration Required. In the later scenario, it isrequired to complete the storage configuration prior to testing or disabling the Maintenance Mode.Similarly, a Recovery Group created at the Remote Site has Maintenance Mode enabled bydefault. After testing is completed for all imported Recovery Group, logical servers, and/or replicaIO services at the Remote Site, deactivate the recovery logical servers or replica IO servicesbelonging to each Recovery Group, then disable Maintenance Mode on each Recovery Group.Recovery Groups that haveMaintenanceMode enabled are not included in anActivate operation.After an import, Maintenance Mode can be disabled without testing. However, Hewlett PackardEnterprise recommends you to test the logical server and IO service activation as described inthe above steps before disabling the Maintenance Mode.

Enabling and disabling Maintenance ModeBy default, all the imported Recovery Groups are in Maintenance Mode or Maintenance Mode– Storage Configuration Required.

• You canmanually activate the logical server(s) or IO services belonging to imported RecoveryGroups in Maintenance Mode. If a Recovery Group displays Maintenance Mode —Storage Configuration Required, you must first finish the storage configurationbefore testing the logical server or IO service activation.

• You can manually activate the imported Recovery Group in Maintenance Mode orMaintenance Mode – Storage Configuration Required only for testing of logicalservers or IO services, which belong to the Recovery Group.

• You can enable Maintenance Mode only on Recovery Groups that are deactivated.

• You can disable Maintenance Mode only on Recovery Groups that have all logical serversand IO services deactivated. If the Recovery Group status is Maintenance Mode –Storage Configuration Required, it is also required to complete the storageconfiguration of the Recovery Group members prior to disabling Maintenance Mode.

Use the Refresh button to update and reload the storage configuration status. The last updatedstorage configuration status timestamp is displayed at the bottom left of the Recovery Group

Enabling and disabling Maintenance Mode 35

table, which can be used to verify when the last reload was completed. Depending on how muchstorage is updated, refresh may take an extended period of time.

NOTE: You must resolve any issues displayed within the Matrix OE visualization storage poolentry screen and/or complete the storage configuration (such as creating replicas, performingpresentation, and zoning), and edit the storage pool entry to indicate whether the volumes areDisaster Recovery Ready. If the Recovery Group is configured as a Matrix IO Service, youmust check the Matrix Infrastructure Orchestration Requests tab for details on the CreateReplica request and for the required actions.

For more information about Matrix storage configuration, including automated configuration ofreplication for certain environment, see the Storage Provisioning Manager (SPM) user guide,Matrix Operating Environment Infrastructure Orchestration user guide, Matrix operatingenvironment logical server management user guide, and Storage Provisioning Manager userguide available at Hewlett Packard Enterprise Information Library.

Failover operationsThis section explains the difference between planned and unplanned failovers, and provides theprocedure to follow in either case.

A note about Microsoft Hyper-V Server failover:Matrix recovery management only supports VM failover between equivalent versions of MicrosoftHyper-V Server, or from earlier versions to current versions.Failover from a current version to an earlier version, such as failover from Hyper-V Server 2012to Hyper-V Server 2008 R2, is not supported.

Planned failoverA planned failover typically involves an expected outage. For example, a planned failover mightbe necessary to perform scheduled maintenance, or to react to a severe weather forecast. Thefollowing section describes the steps necessary to perform a planned failover using Matrixrecovery management. A planned failover includes a series of steps performed at both the LocalSite and the Remote Site.A planned failover includes a series of steps performed first at the site you plan to shut down,and then at the other site in the Matrix recovery management configuration.At the site you want to shut down:1. Shut down the applications and operating system on each Matrix recovery management DR

Protected logical server and each server associated with DR Protected IO services.2. From the Matrix recovery management GUI Sites tab, click Deactivate button and the

Deactivate Recovery Groups at the site name window will appear.For more information about the Recovery Groups contained in a Recovery Group Set, selectthe Recovery Group Set and click View Recovery Group. A window will appear displayingthe following parameters for each Recovery Group in the Recovery Group Set:• Name• Status• Type• Preferred Site• Secondary Site• Storage Replication Group Name



• Start Order• Power-Up Delay

3. Select the Recovery Group sets that you want to deactivate, which under an expected fulloutage will be both sets, and then click Deactivate.

4. ClickDeactivate Recovery Groups to start the deactivation operation. A window will appearasking if it is OK to proceed. Click OK and you will be directed to the Jobs tab where youcan monitor the progress of the deactivation Job.

NOTE: If the Matrix recovery management configuration includes Hyper-V logical serversor IO services, bring the cluster disk resource used for logical server or IO service storageoffline prior to activating operation at the Remote Site. If you did not take the CSV offline,see “Recover a CSV from an online pending state” (page 86) for how to recover.

To fail over to the Remote Site:1. Log in to the remote CMS and ensure that enough resources are available to run the logical

servers that will be activated at this site.2. Click Activate... button and the Activate Recovery Groups at the site name window will

appear.For more information about the Recovery Groups contained in a Recovery Group Set, selectthe Recovery Group Set and click View Recovery Group. A window will appear displayingthe following parameters for each Recovery Group in the Recovery Group Set:• Name• Status• Type• Preferred Site• Secondary Site• Storage Replication Group Name• Start Order• Power-Up Delay

3. Select each Recovery Group Set that you want to activate or click check box on the left sideof the banner of the Activate Recovery Groups at the site name window to select all ofthe Recovery Group Sets for activation.

4. Click Activate Recovery Groups to start the activation operation. A window will appearasking if it is OK to proceed. Click OK and you will be directed to the Jobs tab where youcan monitor the progress of the activation Job.

NOTE: A successfulActivate orDeactivate operation ensures that all of the Recovery Groupswithin a Recovery Group Set are in the same state (activated or deactivated). However, certainoperations (for example, a Recovery Group edit to change site preference) may result in someRecovery Groups within a Recovery Group Set being activated and others being deactivated.You must run an Activate or Deactivate operation on a Recovery Group Set to ensure that allof the Recovery Groups in that Recovery Group Set are in the same state ( all activated or alldeactivated).

Unplanned failoverAn unplanned failover typically involves the occurrence of a site-wide event, without prior warning.This event may be a regional disaster (earthquake and massive flood), or a local problem (powerloss or water main leak in the data center).

Failover operations 37

An unplanned failover includes a series of steps performed at both sites in the Matrix recoverymanagement configuration. The following procedure assumes that the CMS and managedresources running DR Protected logical servers survived an unplanned local event (for example,a power loss). If the event is more severe, resulting in the permanent loss of the CMS or managedresources, reconstruction of the site may be necessary.At the site where the site-wide event occurred:1. Ensure that the DR Protected logical servers are no longer running to prevent a split-brain

situation. As long as the DR Protected logical servers have stopped running, Matrix recoverymanagement prevents them from automatically powering up when power is restored.Matrix recovery management is able to prevent split-brain from occurring during an unplannedfailover by regulating the auto-power configuration of managed nodes (whether virtual orphysical) that are assigned to DR Protected logical servers so they do not automaticallypower-up after an outage. If, for example, a site loses power and site failover is invoked, thesite where the power outage occurred will not resume running the DR Protected logicalservers when power is restored. Themanaged nodes (whether VC blades or virtual machines)assigned to DR Protected logical servers stay powered down (and resources remainunassigned) until an Activate operation is invoked.

At the failover destination (recovery) site:1. Ensure that enough resources are available to run the recovery logical servers.2. From theMatrix recovery management Sites tab, click theActivate... button and theActivate

Recovery Groups at the Local Site window will appear.For more information about the Recovery Groups contained in a Recovery Group Set, selectthe Recovery Group Set and click View Recovery Group. A window will appear displayingthe following parameters for each Recovery Group in the Recovery Group Set including:• Name• Status• Type• Preferred Site• Secondary Site• Storage Replication Group Name• Start Order• Power-Up Delay

3. Select each Recovery Group Set that you want to activate at the recovery site. The objectiveis for all Recovery Group Sets that were previously activated at the site where the site-wideevent occurred to be activated at the recovery site.

4. Click Activate Recovery Groups to start the activation operation. A window will appearasking if it is OK to proceed. Click OK and you will be directed to the Jobs tab where youcan monitor the progress of the activation Job.


NOTE:• If the Matrix recovery management configuration has been changed since the failover

occurred (for example, a new Recovery Group was created), the sites must be brought intosync by making appropriate configuration changes. The Matrix recovery management Siteconfiguration export and import operations can be used for this purpose.

• A successful Activate or Deactivate operation ensures that all of the Recovery Groupswithin a Recovery Group Set are in the same state (enabled or disabled). However, certainoperations (for example, a Recovery Group edit to change site preference) may result insome Recovery Groups within a Recovery Group Set being enabled and others beingdisabled. You must run an Activate or Deactivate operation on a Recovery Group Set toensure that all Recovery Groups in that Recovery Group Set are in the same state (enabledor disabled).

Target selection and parallelism during an activation operationThe Matrix OE logical server management (logical server management) component in the MatrixOperating Environment supports the concept of targets that are most suitable for activating alogical server based on various criteria, for example, an application may need to run on VC-hostedlogical servers only, to meet a performance requirement.DR Protected logical servers that can run on both physical and virtual targets (cross technologylogical servers) will be placed on the target type specified as preferred in the site configuration(P for physical or V for virtual), based on availability. If the preferred target type is not available,Matrix recovery management will ignore the target type preference and place cross technologylogical servers on available supported targets.Matrix OE logical server management allows logical servers to be activated in parallel, exploitingparallelism that is available in the managed infrastructure. Matrix recovery management exploitsthe parallelism of activation available from logical server management when performing anActivate operation, to reduce failover time. Two user accessible settings are available to influenceMatrix recovery management behavior in this area:• Use Recovery Group Start Order values to determine the workloads that will be brought up

first during the failover process.If there are workloads that you want to bring up first during the failover process, theRecoveryGroup Start Order values for the associated Recovery Groups can be set lower than thevalues you set for workloads that can be brought up later in the failover process. Matrixrecovery management ensures that logical servers in a Recovery Group with a lowerRecovery Group Start Order value are activated before logical servers in a Recovery Groupwith a higher Recovery Group Start Order value are activated.

• Use the Recovery Group Power-Up Delay parameter to ensure that the logical servers ina Recovery Group boot in a staggered fashion during the failover process. The RecoveryGroup Power-Up Delay parameter sets a minimum delay between the time when one logicalserver in the Recovery Group begins its boot process and the time when the next logicalserver in the Recovery Group begins its boot process.

Target selection and parallelism during an activation operation 39

4 Command Line ToolsStarting with 7.5 release, a new command line tool drsync is supported, which enablesadministrators to synchronize changes made to the logical servers between Local and RemoteSites without the need to failover storage. It reads information from one Matrix instance andpushes the information to a second targeted Matrix instance of the samemajor version, as definedby the command line syntax.

drsync toolThe key features of drsync command line tool are:

• Creates a peer logical server at the Remote Site, in the event that one has not been createdpreviously without the need to failover the storage and manually create the peer logicalserver saving time and preserving continuous service at the Local Site.

• Can be run at any site—preferred or secondary site.

• Changes to logical servers can originate at either site and can be synchronized in eitherdirection, regardless of where the command is being executed.

• Although you can use drsync tool to create peer logical servers, it is highly recommendedto use the Matrix recovery management import process to create peer logical servers. Youcan use drsync to synchronize the changes in existing peer logical servers. Matrix recoverymanagement import synchronizes Recovery Groups where as drsync does not. As a result,any logical servers created by drsync are not added to a Recovery Group.

CMS configuration requirements for use with drsyncFor cross communication of drsync between CMSs, an x509 certificate exchange must becompleted before running the tool. To accomplish this exchange, the HPE SIM Federated CMSConfigure option is used as shown in the following example using a local CMS — CMS A anda remote CMS — CMS B.

40 Command Line Tools

Example 1 CMS A configuration

The steps to configure CMS A at the Local Site are:1. Execute the following command using the command line prompt:

c:\ mxglobalsettings -ld managed_cms_list

2. The command output displays the FQDN of CMS A.3. Using the browser, navigate to the HPE SIM GUI and select the Options→Federated CMS

configuration…a. Click Add CMS… and enter the IP address or FQDN of the target/remote CMS and

click Next.b. Enter the username and password for a user with administrator rights on the

target/remote CMS and click Finish.4. Return to the command prompt on the local CMS and execute the following command again:

c:\ mxglobalsettings -ld managed_cms_list

5. The command output must display both local (CMS A) and remote (CMS B) CMSs.6. From the command line execute:

c:\mxglobalsettings -s -fmanaged_cms_list=<CMS_LISTED_IN_OUTPUT_FROM_STEP_ONE

NOTE: The FQDN for the local CMS is C:\mxglobalsettings -s -fmanaged_cms_list=cmsa.x.y.com

7. Edit the lsaclient.properties file located in the VSE install directory to specify the IPaddress of the local host.HPSIM_CMS_TCPIP_HOSTNAME=<IP_ADDRESS_THAT_DRSYNC_WILL_USE_TO_TALK_TO_THIS_CMSFor example, HPSIM_CMS_TCPIP_HOSTNAME=10.0.0.10110.0.0.101 is the IP address of CMS A.

8. You can either reboot the CMS or restart all HPE services.

Example 2 CMS B configuration

The steps to configure CMS B at the Remote Site are:1. Edit the lsaclient.properties file located in the VSE install directory to specify the IP

address of the local host.HPSIM_CMS_TCPIP_HOSTNAME=<IP_ADDRESS_THAT_DRSYNC_WILL_USE_TO_TALK_TO_THIS_CMSFor example, HPSIM_CMS_TCPIP_HOSTNAME=10.0.0.10210.0.0.102 is the IP address of CMS B.

2. You can either reboot the CMS or restart all HPE services.

Configuring the dr.properties file for drsyncThe dr.properties file, which is the default file to modify the configuration parameters of thedrsync tool is available at <Matrix recovery management install directory>\confdirectory. Optionally, you can define additional configuration files and call a specific file usingthe option configFile with drsync.Table 2 (page 42) lists the properties that can be set in the dr.properties file.

Configuring the dr.properties file for drsync 41

Table 2 dr.properties options

ExampleValuesDescriptionProperty Name

SYNC_STRATEGIES=COPY_LS,COPY_CPU, COPY_MEMORY,COPY_NETWORK, COPY_STORAGE

COPY_LS

COPY_NETWORK

COPY_STORAGE

Logical Server SynchronizationStrategies for drsync.This allows you to define whatis synchronized for logical

SYNC_STRATEGIES

COPY_MEMORYservers between source andtarget CMS. When you specify COPY_CPUCOPY option, the remote logicalserver CPU, memory, network,or storage properties areupdated with the source logicalserver properties.

SYNC_EXCLUSION_LIST= LS01,LS?2, LS*3, LS?[Aa]*

This allows you to define a listof logical servers (separated bycommas) from the source CMS

SYNC_EXCLUSION_LIST

that is excluded from thesynchronization process.This property supports the useof globbing patterns (wildcards)to match the logical servernames. For more information,see “Globbing Expressions”(page 43).

SYNC_LS_MAPPING=LS_source_1:LS_target_1,

This property allows you tomaplogical servers with different

SYNC_LS_MAPPING

\LS_source_2:LS_target_2,\LS_source_3:LS_target_3

names across sites. If thelogical server names are thesame at both sites, it is notnecessary to add the logicalserver to the mapping.Property syntax: Theelements of each pair must beseparated by colons. The pairsmust be separated by commas.This property can have thecontent split in multiple linesusing a backslash (\) at the endof each line.


Globbing ExpressionsThe following characters are supported while using globbing expressions:

• Supported characters

◦ A question mark represents any single character.For example, ls? results in drsync looking for all logical servers beginning with lsand ending with a single character, lsa, lsb, ls1, and so on.

◦ An asterisk represents any number of characters.For example, ls* results in drsync looking for all logical servers beginning with lsand ending with any number of characters including a logical server without any additionalcharacters, such as ls, lsa, lsbc, ls123, and so on.

◦ Square brackets allow you to set a range.For example, Entering my[123]ls results in drsync looking for my1ls, my2ls,andmy3ls.

The following rules are applicable while using globbing expressions:

• The property is case-sensitive.

• Globbing expressions accept special characters such as ?, *, [ ].It is not recommended to use special characters as literal characters and not as operators.For example,* must be written as \\* in drsync.properties file, and as \* in command line.To include logical server named LS?, you must use –ls LS\?To include logical servers LS1,LS2 you must use -ls LS?.

• Commas are not acceptable for use within globbing patterns.

Using the drsync command line toolTo use the drsync command line tool, you need to have both the source and target CMSs thatneed to synchronize on the same major version of the Matrix Operating Environment. By default,drsync filters out all the logical servers. While running the tool, you will have to do a minimumdefine -virtual and/or -physical options with the command. Additionally, drsync alsosimulates synchronization. To commit synchronization, you can use the -execute option.Table 3 (page 43) lists the various drsync options:

Table 3 drsync Options

DescriptionOption

FQDN or IP address of the source CMS.-s

FQDN or IP address of the target CMS.-t

By default, only a simulation of the changes takes place. Use this option tocommit the changes.

-execute

Include virtual logical servers in the synchronization process.-virtual

Include physical logical servers in the synchronization process.-physical

Provides the names of the logical servers to be considered for synchronization.If no names are specified, all the DR protected logical servers are considered

-ls

for synchronization. This option relies on the –physical and –virtual optionsto determine which of the considered logical servers are part of the

Globbing Expressions 43

Table 3 drsync Options (continued)

DescriptionOption

synchronization process. This option supports the use of wildcards to match thelogical server names.

Configuration file used to define which properties are to be synchronized betweenlogical servers. If not specified, the tool uses the default configuration file located

-configFile <filename>

at <Matrix recovery management install directory>\conf\dr.properties.

Provides the number of logical servers to be synchronized simultaneously. Bydefault the value is 1. Increasing this number can increase performance but it

-c <concurrency number>

also increases the load on the CMS. You must carefully choose the number ofCPUs of the CMS and network bandwidth. A maximum value of 25 is supported.

Displays the following additional information:-verbose

• Name of the loaded configuration file.

• Values loaded for the SYNC_STRATEGIES property.

• Reason as to why a logical server was skipped.

• Number of logical servers loaded.

Return ValuesThe return values of drsync can be either 0 or –1.

Table 4 Return Values

DescriptionOption

Operation completed successfully.0

An error occurred during the operation.–1

drsync file and locationsThe following are the drsync files with their respective location:

• drsync executable file located at <Matrix recovery management installdirectory>\bin\drsync.bat

• drsync configuration file located at <Matrix recovery management installdirectory>\conf\dr.properties

• drsync log file located at <Matrix recovery management installdirectory>\logs\lsdt_sync.log


5 Dynamic workload movement with CloudSystem MatrixThis chapter explains how you can configure cross-technology logical servers that can bemanagedwith Matrix recovery management.

NOTE: Cross-technology logical servers are not supported for IO services.

The Matrix Operating Environment facilitates the fluid movement of workloads between dissimilarservers within a site and across sites. Workloads can be moved between physical servers andvirtual machines and between dissimilar physical servers.A major trend today in IT data center management is the push toward greater efficiency in theuse of computing, network, and storage resources in the datacenter by treating them as a sharedpool fromwhich the resource requirements of various applications, departments, and organizationsare met. Central to this concept of a converged infrastructure is the ability to rapidly andautomatically create, move, and remove workloads on demand.In a typical converged infrastructure implementation, a customer may use the Matrix OperatingEnvironment running on a Central Management Server (CMS) to create, move, and removeworkloads as needed. The workload, which includes the operating system (OS) on which theuser application runs, can run directly on a blade or it can run in a virtual machine managed bya hypervisor , for example, VMware ESXi or HP Integrity Virtual Machines.The capabilities of the Matrix Operating Environment discussed in this chapter allow fluidmovement of workloads in this type of heterogeneous environment. These capabilities include:• Tools that allow the workload OS to be prepared as a portable system image that can run

in different server environments.• Fine-grained user control over the set of specific physical servers and virtual machine hosts

on which the Matrix Operating Environment can run the workload.The fluid, two-way movement of workloads across dissimilar servers described in this chapter isdifferent from the movement enabled by traditional migration tools. Those tools are orientedtowards enabling a one-way, permanent or semi-permanent migration, between physical andvirtual or between dissimilar physical servers. Themovement typically requires manual interventionand a relatively long period of time to complete.The importance of the ability to fluidly move a workload from a physical server to a virtual machineand back can be understood from the following examples:• You want to move your online workload running on a physical server during daily off-peak

hours to a virtual machine host, to free up the physical server to run a batch workload. Whenthe off-peak period is over, the batch workload is retired and the online workload is movedback to its original execution environment.During the time the online workload is “parked” on a virtual machine host, it has minimalresource requirements; hence it has minimal impact on other workloads that may be runningon that host. Because this pattern repeats daily, the physical to virtual and virtual to physicalmoves must be achieved quickly (in minutes, rather than hours) and automatically.

• You have two data centers located at two different sites. The production workloads run onphysical servers at one site and are configured to be failed over to the other (recovery) site,in case of a disaster. The recovery site is equipped with a set of servers configured as virtualmachine hosts. In this use case, planned or unplanned failovers require physical to virtualand virtual to physical moves across sites.The configuration of the recovery site as a set of virtual machine hosts may be driven by theneeds of test and development activities that are carried out at that site. Or, it may be drivenby a need to reduce the cost of disaster recovery by running the workloads on virtualmachines hosted by a smaller set of servers. Recovery time objectives require that themoves be achieved quickly and automatically, as in the previous example.

45

Capabilities and limitationsUsing the tools and procedures described in this chapter you can:• Configure and manage a logical server that can perform physical to virtual cross-technology

movements within the datacenter.• Configure and manage a DR Protected logical server that can be failed over across data

centers in a cross-technology movement.The following limitations must be noted:• Matrix recovery management extends support to only ESX virtual machines for P2V. For

this, ensure that all source and targets are in the same portability group.• Configuration of a cross-technology logical server requires additional steps (however, no

additional steps are required at the time of the move, within or across sites).• Cross-technology logical servers are not supported for either physical or virtual IO services.

• Virtual machines running Windows 2008 and later are supported. For more information, seethe Insight Management Support Matrix available at Hewlett Packard Enterprise InformationLibrary.Virtual machines must be configured to emulate either the LSI Logic Parallel or LSILogic SAS storage type.

• VMware ESX guest tools are not automatically installed.

• There is no explicit support of NPIV.

• Preparation of the portable system image so that it can run on both physical and virtualservers is done from an OS installed on a physical server. You cannot prepare the portablesystem image from an OS installed on a virtual machine.

• Virtual machines must be configured to use RDM Fibre Channel SAN storage presented tothe VM host for boot and data. This is the same storage that a logical server uses whenrunning on a physical server.

• Each boot and data LUN must use the same LUN number across the physical and virtualtargets for a logical server as illustrated in Figure 6 (page 47).

46 Dynamic workload movement with CloudSystem Matrix



Figure 6 Same LUN number across physical and virtual targets

• The target WWN values used to present the Logical Unit must be the same across virtualand physical targets.

NOTE: The recovery logical server that provides DR protection at the Remote Site hasits own set of target WWN/LUN values that differ from the target WWN/LUN values for thelogical server at the Local Site.

• The network name used by an ESX Host must match the network name used in the VirtualConnect Enterprise Manager configuration, as displayed in Figure 7 (page 48) andFigure 8 (page 48).

Capabilities and limitations 47

Figure 7 ESX host network name

Figure 8 Virtual Connect Enterprise Manager network name

• Whenmoving a logical server between physical and virtual servers within a site, the followingserver IDs are not preserved:

◦ Network MAC addresses

◦ Server/Initiator WWNs (On a virtual machine, the storage adapter is a virtual SCSIcontroller.)

◦ Logical Serial Number

◦ Logical UUID

• In a DR configuration, the site that you configure first must have both physical servers andvirtual machine hosts available so a logical server can be configured to run and be tested


on both types of servers. The recovery site can have a physical/virtual combination also, orhave only virtual machine hosts.

Supported platformsThe procedures for enabling movement between physical and virtual servers described in thischapter apply to physical servers, hypervisors, and workload operating systems supported byMatrix recovery management. For more information, see the Insight Management Support Matrixat Hewlett Packard Enterprise Information Library.

• For supported hypervisors, see VMware hypervisor versions specified as supported onmanaged systems by the Matrix Operating Environment.

• For supported workload operating systems, see Windows 2008 versions specified assupported on managed systems by the Matrix Operating Environment available at HewlettPackard Enterprise Information Library.

The Matrix recovery management procedures for enabling movement between physical andvirtual servers are not supported on Integrity managed nodes.

• Matrix recovery management, the component of the Matrix Operating Environment thatprovides disaster recovery across sites, does not support Integrity managed nodes.

The procedures for enabling movement across different physical servers documented in thischapter are supported for managed systems specified as supported by the HPEMatrix OperatingEnvironment, with the following restriction:A physical server target configured for cross-technology movements must be an HPE c-Classblade with Virtual Connect.

Overview of physical to virtual cross-technology configurationThis section provides an outline of the steps involved in configuring cross-technology logicalservers for movement between physical and virtual targets, and for movement between dissimilarphysical servers.

Configuring logical servers for movement between physical and virtual targets1. Prepare a logical server with a portable image

Start with a logical server configured to run on a physical server, and prepare its systemimage for movement between physical and virtual servers.a. Storage configuration

The Portable Images Storage Assistant (PISA) tool prepares the storage configurationof the server image so it can be booted in both physical and virtual environments. PISAis part of the HPE Insight Control server migration product on the Insight ManagementDVD. The executable and README are in the <SMP>\PI\PISA folder, where <SMP> isthe directory where Insight Control server migration is installed (the default installdirectory is C:\Program Files\HP\Insight Control server migration).i. Copy the executable hppisa.exe (under PI\PISA) to the physical server where

the image is currently running.ii. In the command-line window, type: > hppisa –eFor more information, see “Portable Images Storage Assistant (PISA)” (page 52).

b. Network configurationPortable Images Network Tool (PINT) prepares the image to execute on targets withdifferent network interface configurations and MAC addresses. It ensures that the staticnetwork configuration from the source server is successfully transferred to destinationserver network interfaces despite the differing environment. The executables and READMEare in the folder <SMP>\PI\PINT.

Overview of physical to virtual cross-technology configuration 49




i. Copy the executable cp011231.exe to the physical server where the image iscurrently running.

ii. Run cp011231.exe to install PINT and start the PINT service.For more information, see “Configuring and managing portable OS images” (page 52).

2. Create a portability group that includes all potential physical and VM host targetsThis step sets up the portability group that defines the list of potential targets for the logicalserver. The group must include both physical servers and VM hosts as targets. SeeFigure 9 (page 50).

Figure 9 Creating a portability group

For more information, see “Portability groups” (page 53).3. Configure the logical server for activation on both physical and VM host targets

Modify the logical server configuration as follows:• In the Create logical server: identity screen, set the portability group of the logical

server to the portability group created in Step 2.• In the Create logical server: storage screen, select a VM datastore (this datastore

will be used to store VM configuration information).For more information, see “Defining cross-technology logical servers” (page 55).

• Present the boot/data storage to the VM hosts in the portability group using the sameLUN and WWN values. For more information, see “Storage definition” (page 57).

4. Performphysical to virtual and virtual to physicalmovements to verify OS configurationMove the logical server from physical to virtual and back again. After each move, ensurethat the logical server has booted successfully and the static network configurations areapplied as expected to the network interfaces present in the new environment. For moreinformation, see “Moving between technologies” (page 58).


NOTE: When the logical server is first moved to a virtual machine, you may want to addadditional tools to the server, for example, VMware tools. In theMatrix Operating Environment,the VM configuration created does not include a virtual CD/DVD drive. You can use the VMmanagement console to modify the VM configuration to include a virtual CD/DVD drive.

5. Configure inter-site movement between physical and virtual targets (disaster recoveryuse case)In this step, Matrix recovery management capabilities are used to set up inter-site movementbetween physical and virtual targets.a. At the Local Site, create an array replication group containing the LUNs used by the

logical server for boot/data and the LUN containing the VM datastore as illustrated inFigure 10 (page 51). The VM datastore can be used exclusively to store the VMconfiguration of the logical server.

Figure 10 Creating an array replication group

b. At the Local Site, create a Recovery Group containing the logical server. Export theMatrix recovery management configuration to a file.

c. Deactivate the logical server at the Local Site and fail over the array Replication Groupto the Remote Site.

d. Perform VM host rescan and Matrix OE visualization refresh procedures to ensure thatthe VM configuration datastore is accessible to the logical server configuration.

e. Create a portability group at the Local Site. The portability group can contain physicalservers, VMs, or a combination of both.

f. Create the recovery logical server with the same configuration values used at the LocalSite, but adjusted to point to the remote storage LUN/WWN values. Do not activate thelogical server at this time.

g. Using the exported configuration from the Local Site, import the Recovery Group thatwas created at the Local Site to contain the recovery logical server.

h. The Recovery Group is in Maintenance Mode. Activate the logical server on a VM host.If one or more physical servers are available in the portability group, perform virtual tophysical and physical to virtual movements. At each stage, check for successful bootand confirm correct network configuration, as in step 4. Deactivate the logical serverand clear Maintenance Mode. Fail the array replication group back to the Local Site.

i. At the Local Site, perform rescan and refresh procedures, and activate the logical server.

Overview of physical to virtual cross-technology configuration 51

NOTE: TheMatrix recovery management Site configuration can be set up to preferentiallyactivate the logical server on a physical server at one site and a VM host at the other site.For more information, see “Setting a failover target type preference” (page 60).

Configuring logical servers for movement between dissimilar physical serversThe Matrix Operating Environment provides the ability to fine tune the list of failover targets thatare considered most suitable for a DR Protected logical server to be activated on. The ability tomodify target attributes is useful to ensure a successful failover. Target attributes are includedin the data transferred by the Matrix recovery management export/import sequence. An expansionin the target list at the exporting site is reflected at the importing site. For more information, see“Target attributes” (page 59).

Configuring and managing portable OS imagesMobility of server workloads is hampered by the fact that most operating systems are configuredat install time for the specific platform they are installed on. Examples include:• Only the device drivers necessary for the target platform are installed and configured;

attempting to boot the same OS instance on a different server may result in device errorsor system failures.

• Configuration settings, such as IP addresses or storage identifiers, may be bound to specificdevices whose names may change. For example, the same subnet may be attached to thefirst NIC port on one server, while it is attached to the third NIC port on another.

The Matrix Operating Environment supports mechanisms to prepare a server image (logicalserver) so it continues to work when moved to a different physical or virtual server. The followingareas are key to achieving this objective:• Driver installation

• HBA configuration

• NIC configurationThe tools developed for this purpose are as follows:• Portable Images Storage Assistant (PISA)

• Portable Images Network Tool (PINT)

Portable Images Storage Assistant (PISA)When Windows is installed on a SAN LUN attached to a server, it installs a driver that is specificto the HBA controller on that server. If that LUN is subsequently reassigned to a virtual machineon a server running VMware ESX, ESX presents only SCSI direct-attached devices to the virtualmachine. Because Windows is still configured to use the HBA controller for the original server itis unable to start up. PISA enables the appropriate Windows drivers so Windows can start in thevirtual machine.PISA requires that the virtual machine is configured to use the Raw Device Mapping feature inESX to configure a LUN on a SAN as a disk drive is available to the virtual machine. The virtualmachine must also be configured to emulate either the LSI Logic Parallel or LSI Logic SASstorage type.In all versions of Windows, the driver needed to properly operate the virtual version of the LSIcontroller is installed, but disabled.PISA is used to enable LSI support in the Windows image.PISA is a simple command-line tool that accepts only a few command-line options. It needs tobe executed only once after Windows is installed on a physical server. The changes it makes


are persistent and do not need to be repeated or reversed. However, repeatedly running PISAhas no negative impact. PISA can also be used to disable the driver used by the virtual machine.The command-line interface for PISA is described below. The options are mutually exclusive.PISA runs on supported versions of Windows only, and it requires that the user be a member ofthe Administrator user group.Usage: hppisa -h, -?, -help Show this information-e, -enable Enable the LSI driver-d, -disable Disable the LSI driver

After these changes are made, the OS image can be moved back and forth between physicalservers and virtual machines.For more information on PISA, including installation, supported operating systems and operatinginstructions, see the readme available on the Insight Control DVD at C:\Program Files\HP\Insight Control server migration\PI\PISA.

Portable Images Network Tool (PINT)PINT is used to resolve networking issues when youmove an OS image from one physical serverto another, or from a physical server to a VMware ESX virtual machine. However, when an OSis booted after a VC-hosted logical server activation at the DR site, PINT might not assign thecorrect IP address to the intended NIC. If this happens, manually configure the NIC IP address.PINTmaintains a network configuration file where it gathers information on each network interfaceand its configuration on a server. PINT remains in a suspended state, running only when itreceives one of the following events:• IP configuration change event

PINT considers any changes made while the server is up and running to be intentionalchanges made by the user. PINT records the changes and updates its configuration file.

• Stop eventIf the user stops the PINT service PINT receives an event that lets it know to shut down.

• User command eventIf a user makes any changes through the PINT command-line PINT is notified and actsaccordingly.

NOTE: If an NIC in the destination server requires a different set of drivers than those on thesource server, you must install the new drivers before using PINT on the destination server.

For more information on PINT, including installation and operating instructions, see the PortableImages Network Tool (PINT) Windows readme or the Portable Images Network Tool (PINT)Linux readme available in the PINT directory on the Insight Control DVD at C:\Program Files\HP\Insight Control server migration\PI\PINT.

Configuring and managing cross-technology logical serversThis section explains configuration tasks and management of cross-technology logical serversin an Matrix Operating Environment.

Portability groupsWhen you create a logical server, you must specify the portability of the logical server. You dothis by selecting a portability group from within the identity page of the logical server configurationwizard. After a logical server is associated with a particular portability group, it can be moved toany target system ( Virtual Connect physical server or virtual machine hypervisor) within thatportability group. Logical server resource constraints, such as CPU/memory requirements and

Configuring and managing cross-technology logical servers 53

network/SAN connectivity, are evaluated solely within the context of the portability group that thelogical server is associated with. Portability groups come in two classes: Default and User Defined.TheMatrix Operating Environment provides default portability groups depending on the resourcesfound within your data center. The Default portability groups include:• ESX—All ESX Hypervisors

• HYPERV—All Hyper-V Hypervisors

• Each Virtual Connect Domain Group—Each VCDG has its own Default portability group.You can also create User Defined portability groups that extend the portability of a logical serverto unlike technologies. For example, moving logical servers between a Virtual Connect physicalserver and a VMware ESX virtual machine host.User Defined portability groups are defined by selecting Modify→Logical Server PortabilityGroups in Matrix OE visualization as illustrated in Figure 11 (page 54).

Figure 11 Modifying logical server portability groups

If you have selected one or more targets in Matrix OE visualization, they are presented as potentialtargets. Otherwise, all resources are presented. Figure 12 (page 55) provides an example of theModify Portability Group screen.


Figure 12 Selecting group members and targets

Provide a name and optional description for the portability group. The name will be used fordefining logical servers. The set of Group Types is selected automatically based on the targetsinserted into the portability group. Valid combinations of targets include:• A single Virtual Connect Domain Group (VCDG).

• A set of ESX Hypervisors.

• A set of Hyper-V Hypervisors.

• A set consisting of a single VCDG plus a set of ESX Hypervisors.

Defining cross-technology logical serversTo define a cross-technology logical server, you must place a logical server into a portabilitygroup, and then define the storage for that logical server.

Placing a logical server into a portability groupPlacing a logical server into a portability group is accomplished on the logical server identitypage. This can be done during the creation of the logical server or in a subsequent logical servermodification. Select from the Portability Group list as shown in Figure 13 (page 56). This listincludes both the Default portability groups and any User Defined portability groups.


Figure 13 Selecting a portability group

To view the portability group for any logical server, click the Viewmovable logical server detailsicon in Matrix OE visualization as illustrated in Figure 14 (page 56).

Figure 14 View movable logical server details icon

The details for this logical server are displayed as illustrated in Figure 15 (page 56).

Figure 15 View logical server details


Logical servers can be made portable through techniques described in “Portability groups” (page53).

NOTE: You must determine whether the provisioned operating system within a logical serverperforms as desired on a variety of platforms. If a logical server has never been active on aplatform type, the Matrix Operating Environment shows a warning for each target of that type inthe Target Selection page during moves and activations. You must determine whether the targetis valid.

Storage definitionStorage can be defined through Storage Pool Entries or Storage Entries tied directly to a logicalserver. The storage for cross-technology logical servers must be SAN-based. This approachuses the normal SAN-boot approach within Virtual Connect and leverages ESX RDM technologywhich presents boot and data LUNs directly to the virtual machine. For examples, seeFigure 16 (page 57) and Figure 17 (page 57).

Figure 16 Storage Pool Entries

Figure 17 Storage Entries


When defining storage for a portable logical server, you must select SAN Storage Entry. Forflexibility and movement between underlying technology types, storage must be presented tothe WWNs tied to the Virtual Connect server profile, and storage must also be presented to anyESX VM hosts that are potential targets for the logical server. Storage Pool Entries must becreated within the same portability group associated with the logical servers that will use thestorage as illustrated in Figure 18 (page 58).

Figure 18 Managing Storage Pools

Moving between technologiesActivation and movement of cross-technology logical servers is accomplished in the same wayas with standard logical servers. However, theUnlikeMove operation is used for cross-technologylogical servers when an Activate or Move operation is about to be performed on a server witha different underlying technology from its previous target host. An Unlike Move operation isillustrated in Figure 19 (page 58).

Figure 19 Assigning logical servers to target hosts


Targets for a logical server are selected from that logical server's portability group. The portabilitygroup members are then further filtered based on the resource availability, including CPU andmemory resources as well as network and SAN connectivity.

NOTE: Networks in Virtual Connect must be named identically to their corresponding networks(port groups) on ESX Hypervisors. Differences in names prevent the Unlike Move operationfrom identifying networks with similar connectivity.

Target attributesYou can track where a logical server is successfully activated or moved by using logical servertarget attributes. Target attributes provide a greater number of “most suitable” targets where youcan activate or move a logical server. To view or modify target attributes on a logical server,select the logical server and then click Modify→Logical Server Target Attributes→Manageas illustrated in Figure 20 (page 59).

Figure 20 Modifying logical server target attributes

Types of targets can be selected and added or removed from the logical server's target attributes.Selecting a server from the list below and clicking Add adds that type of server with associatedresources to the list of “most suitable” targets for the logical server. Selecting a type of serverfrom the Target Attributes Available to Remove list and clicking Remove causes the type ofserver specified to no longer be listed as “most suitable.” Figure 21 (page 59) illustrates thescreen where you can manage logical server target attributes.

Figure 21 Managing logical server target attributes


Moving between blade typesFor logical servers with target attributes, the logical server management software can identifymore possible targets when moving or activating a server. As with all cross-technology logicalservers, you must ensure that the logical server can function appropriately on various platforms.If a particular target is proven to be unsuitable, it is easy to remove that type of target to moreaccurately describe the logical server's portability.

Managing DR Protected cross-technology logical servers in a Matrixrecovery management configuration

This section explains how to specify failover target type preferences for DR Protectedcross-technology logical servers in a Matrix recovery management configuration.

Setting a failover target type preferenceDuring a site failover, for every logical server that is configured with disaster protection, Matrixrecovery management activates a similarly configured peer logical server at the recovery site.For this purpose, Matrix recovery management interacts with the Matrix OE logical servermanagement capability to determine a list of appropriate available targets and chooses the mostsuitable target to activate the logical server.For a cross-technology logical server, the appropriate list of available targets may include bothphysical servers and VM hosts. Matrix recovery management includes a Target type preferredsetting in the Sites configuration tab, where you can specify the type of target preferred for eachsite, as illustrated in Figure 22 (page 60).

Figure 22 Matrix recovery management Sites configuration screen


You must specify the target type preferred for all sites on the CMS at each site:• If you specify Virtual as the target type preferred for a site, all cross-technology logical

servers whose Recovery Groups prefer that site are activated on VM hosts during anActivateoperation at that site. A physical server is chosen only if no VM hosts are available.

• If you specify Physical as the target type preferred for a site, all cross-technology logicalservers whose Recovery Groups prefer that site are activated on physical servers duringan Activate operation at that site. A VM host is chosen only if no physical servers areavailable.

By specifying appropriate target preferences for sites, you can set up a configuration where DRProtected cross-technology logical servers run on physical servers at one site and on VM hostsat the other site. For example, you might choose to have DR Protected cross-technology logicalservers run on physical servers at the Preferred Site, but run on VM hosts when they fail over tothe secondary site.

Managing DR Protected cross-technology logical servers in a Matrix recovery management configuration 61

6 Issues, limitations, and suggested actionsThis chapter lists issues and limitations for this release, categorized as follows:Limitations Limitations of the implemented functions and features of this releaseMajor issues Issues that may significantly affect functionality and usability in this releaseMinor issues Issues that may be noticeable but do not have a significant impact on

functionality or usability

Limitations• Import of virtual (VM-hosted) IO services requires manual storage failover prior to import.

• A Recovery Group can contain logical servers only, or IO services only, but not a mix oflogical servers and IO services.

• IO services with a combination of physical (VC-hosted) and virtual (VM-hosted) logicalservers are not supported.

• Replica IO services cannot be flexed (add disk, add server, remove server).

• Replica IO services cannot be imported to create primary IO services at the Preferred Sitein a Matrix recovery management configuration.

Hyper-V support limitation for bidirectional configurationFor bidirectional failover configurations, the logical servers and IO services configured in RecoveryGroups with the Local Site as the Preferred Site are required to have read-write access to theassociated storage on the Local Site. The logical servers and IO services configured in RecoveryGroups with the Remote Site as the Preferred Site may have read-only access to the associatedstorage on the Local Site. Microsoft requires that all of the storage (cluster shared volume disksor cluster disks) in the clusters where the logical servers will be activated must have read-writeaccess.Suggested actionMicrosoft has provided a hotfix. See the Microsoft Knowledge Base article and download thehotfix at http://support.microsoft.com/?id=2720218.

No automatic synchronization of configuration between sitesThe configuration of Matrix recovery management at two separate sites is not automaticallysynchronized, but Matrix recovery management provides configuration export and import featuresthat simplify this task.Suggested actionsTheMatrix recovery management online help guides you through the export and import operations.

Matrix recovery management job information is not preserved in certain scenariosInformation about previously executed Matrix recovery management Jobs is not preserved whenthe Matrix recovery management configuration is restored using the Insight mxsync utility.

Minor issuesFirefox browser cannot be used for site export operations

Firefox browser cannot be used to performMatrix recovery management site configuration exportoperations, due to a bug in Adobe Flash Player 10.3. For more information, see Bug 2980517on the Adobe website.

62 Issues, limitations, and suggested actions

http://support.microsoft.com/?id=2720218

Suggested actionsUse Internet Explorer 8 or later to perform Matrix recovery management site configuration exportoperations.

ESX configuration setting required for VMFS datastores of Matrix recoverymanagement managed logical servers to be visible at Remote Site

Under the following conditions, Matrix recovery management requires a specific ESX configurationsetting to retain the signature of a VMFS datastore so it will be visible at the Remote Site:• You have asymmetric P6000 Continuous Access Software array models at the Local and

Remote Site.• You are using P9000 Continuous Access Software storage arrays at the Local and Remote

Site.• You are using 3PAR StoreServ Storage systems.Suggested actions

• For ESX3.X hosts, set Lvm.DisallowSnapshotLun to 0 using VirtualCenter→Configuration→Advanced Settings.

• For ESX4.X hosts, to mount the Local Site datastore with an existing signature, see “Mounta VMFS Datastore with an Existing Signature” in the ESX Configuration Guide Update 1,ESX 4.0, vCenter Server 4.0 available at http://www.vmware.com/pdf/vsphere4/r40_u1/vsp_40_u1_esx_server_config.pdf. For additional information, see the VMware KnowledgeBase at http://kb.vmware.com/kb/1015986.

Activation or deactivation job hangsIf Matrix OE logical server management is not able to complete a task initiated by aMatrix recoverymanagementActivate orDeactivate operation due to underlying stack issues, theMatrix recoverymanagement operation hangs.Suggested actions:Restart the Logical Server Automation service from the CMS (Windows AdministrativeTools→Services and Applications→Services) and perform the Matrix recovery managementoperation again.

Identical configuration of logical servers between sitesLocal and recovery logical servers must be configured with identical parameters (except for thelogical server name) using Matrix OE visualization. There is a potential for discrepancies, betweenthe Local Site and the Remote Site, in the values of attributes that are not included in the Matrixrecovery management user interface configuration screens.The examples of identical configuration of logical servers between sites are:• MAC address—For VC-hosted logical servers, the MAC address is assigned from Virtual

Connect or Virtual Connect Enterprise Manager. Because disjointed address ranges mustbe used at the Local and Remote Sites, the MAC address for a Local Site logical server andits corresponding Recovery Site logical server are different.

• HBA WWN—For a VC-hosted logical server, the HBA WWN address is assigned by VirtualConnect or Virtual Connect Enterprise Manager. Because disjointed address ranges mustbe used at the Local and Remote Sites, the HBA WWN address for a Local Site logicalserver and its corresponding Recovery Site logical server are different.

• BIOSUUID—There is no supported mechanism to preserve the UUID for a VC-hosted logicalserver as it moves across sites (as there is when it moves within a site).

Minor issues 63

http://www.vmware.com/pdf/vsphere4/r40_u1/vsp_40_u1_esx_server_config.pdf

http://www.vmware.com/pdf/vsphere4/r40_u1/vsp_40_u1_esx_server_config.pdf

http://kb.vmware.com/kb/1015986

• BIOS serial number—There is no supported mechanism to preserve the serial number of aVC-hosted logical server as it moves across sites (as there is when it moves within a site).

• Array LUNs—On a VC-hosted logical server, the Windows or Linux OSmust map presentedLUNs to the volumes and file systems configured for the OS. For a VM-hosted logical server,the ESX OS on the host must map the presented LUNs to the VMFS used by the VM-hostedlogical server. Although the operating systems have built-in mechanisms to do this, HewlettPackard Enterprise recommends as a best practice that you keep LUN numbers the samefor corresponding disks across sites.

Suggested actionsAssess the impact of these discrepancies on any licensing arrangements in use for the operatingsystem and applications running on DR Protected logical servers.

One RAID Manager instance per P9000 Storage Management Server and OneRAID Manager instance per P9000 device group

Each P9000 Continuous Access Software Storage Management Server configured in Matrixrecovery management has only one P9000 RAID Manager Software instance managing theP9000 device groups. Each P9000 Continuous Access Software Storage Replication Groupconfigured in Matrix recovery management is managed by only one P9000 RAID ManagerSoftware instance at each site.Suggested actions:There is no workaround for this issue.

CLX/ P9000 software must be installed on a separate Windows systemA separateWindows system other than the Central Management Server must be configured withCLX/ P9000 and compatible P9000 RAIDManager Software to manage the various P9000 devicegroups included in a Matrix recovery management configuration.Suggested actionsThere is no workaround for this issue.

One active Matrix recovery management configuration operation at any point intime

If multiple users attempt to run configuration operations in Matrix recovery management, onlyone operation succeeds. All other configuration operations receive an error message indicatingthat some other configuration operation is in progress.Suggested actionsThere is no workaround for this issue.

Site delete operation in Matrix recovery management does not remove HPE SIMtools

If P9000 Continuous Access Storage Management Servers are configured and if a site deleteoperation is performed, the HPE SIM tools used to manage P9000 storage replication on thelocal Storage Management Server are not deleted.Suggested actionsThe mxtool command can be run manually to remove the HPE SIM tools. The names of thetools are Insight Recovery_Failover and Insight Recovery_Group validation.For example, if the name of the Storage Management Server is stgmgmtA.cup.hp.com, thenames of the tools are STGMGMTA_CUP_HP_COM_Insight Recovery_Failover andSTGMGMTA_CUP_HP_COM_Insight Recovery_Group validation.

64 Issues, limitations, and suggested actions

Using SPM with Remote Copy Group requirementWhen using SPM with Remote Copy Group requirement to create storage, storage can only beedited on the Local Site when it is primary (storage Read/Write). Storage can only be exportedfrom the Local Site when it is primary, and can only be imported into the Remote Site when it isa backup (disks are Read-Only).

Minor issues 65

7 TroubleshootingThis chapter provides troubleshooting information in the following categories:• “Configuration troubleshooting” (page 66)

• “Configuration error messages” (page 69)

• “Warning messages” (page 72)

• “Matrix recovery management Job troubleshooting” (page 73)

• “Failover error messages” (page 76)

• “Matrix recovery management log files” (page 77)

• “DR Protected IO services troubleshooting” (page 77)

Configuration troubleshootingTo troubleshoot Matrix recovery management configuration operations, take note of any errormessages that appear, then review this section for relevant information. You can also view thelsdt_web.log log files for additional information.The following troubleshooting issues are addressed in this section:• Unable to add or edit Site information

Possible causes include:

◦ The local or remote CMS name provided is not valid (not a fully qualified domain nameor locatable in the DNS).

◦ The local or remote CMS name does not include a fully qualified name associated withthe local host.

• Unable to add or edit Storage Management Server informationPossible causes include:

◦ The Storage Management Server is not discovered in the Matrix Operating Environmentuser interface.

◦ The credentials associated with the StorageManagement Server in the Matrix OperatingEnvironment user interface do not include the user name provided as part of a StorageManagement Server configuration that is added or edited.

◦ The Matrix Operating Environment user interface is unable to communicate with theStorage Management Server.

Communication Failure with Storage Management Servers may result if Open SSH is notbeing installed or configured on the target system.

• Unable to add or modify P6000 Storage Management ServerPossible causes include:

◦ The CIMOM Server is not running on the Storage Management Server.

◦ The CIMOM Server is configured to use a different port than the one specified for addor edit operations.

◦ The specified user does not have valid login credentials on the Storage ManagementServer with appropriate privileges.

66 Troubleshooting

• Unable to add or edit P6000 Storage Replication GroupPossible causes include:

◦ Matrix recovery management is unable to obtain Storage Replication Group informationfrom Command View servers to validate the Storage Replication Group informationprovided by the user.

• Unable to add or edit P9000 Storage Replication GroupPossible causes include:

◦ The Storage Replication Group is not configured to be managed by the RAID managerinstances.

◦ The RAID manager service is not running on the local Storage Management Server.

◦ The parameters entered for the Storage Replication Group (local array, remote array,serial number, and replication mode) do not match the Storage Replication Groupinformation on the disk array.

• Unable to add or modify 3PAR StoreServ Storage Management ServerPossible causes include:

◦ The password file for the 3PAR StoreServ Storage system is missing from the/storage/3par/conf directory where Matrix recovery management is installed.

◦ The password for the 3PAR StoreServ Storage system has changed since the passwordfile was created.

◦ The hostname in the Storage Management Server tab in Matrix recovery managementis not the FQDN.

• Matrix recovery management hangs when adding Storage Management Servers afteran upgrade to 3PAR StoreServ CLI and/or CLX versions.Possible cause includes:If a new certificate has not been accepted after an upgrade, the CLI.exe orCLX3PARCONFIG.exe may hang.

For 3PAR StoreServ CLITo resolve the above issue in 3PAR CLI, perform these steps:1. Kill the cli process in Windows.2. Open the 3PAR CLI manually in Windows.3. Accept and add the certificate to be included to the CLI exception list.

For 3PAR StoreServ CLX:Open the 3PAR CLX CLI manually and add the storage arrays. To add all the arrays at once,use the following command:CLX3PARCONFIG ARRAY <options>

where <options> =/ADD

[{NAME=<array network name|array IP address> PWF=<password filepath>}] [{NAME=<array network name|array IP address> PWF=<passwordfile path>}...]

/REMOVE

Configuration troubleshooting 67

[{NAME=<array network name|array IP address>}] [{NAME=<array networkname|array IP address>}...]

/MODIFY

[{NAME=<array network name|array IP address>}] [PWF=<modifiedpassword file location>]

• Unable to add or edit 3PAR StoreServ Storage Replication GroupPossible causes include:

◦ The Storage Replication Group is not configured to be managed by 3PAR StoreServremote copy.

◦ The following information entered for the Storage Replication Group cannot be validatedwith the 3PAR StoreServ Storage system configuration:– Volume group name on the local 3PAR StoreServ Storage system

– Volume group name on the remote 3PAR StoreServ Storage system

– Local array remote copy group name

– Remote array remote copy group name

– Local 3PAR StoreServ Storage system serial number

– Remote 3PAR StoreServ Storage system serial number

– Replication mode

• Matrix recovery management logical server configuration is inconsistent with MatrixOE logical server management logical server configurationPossible causes include:

◦ The Logical Server Automation service was not running when Recovery Groups wereconfigured in Matrix recovery management.

◦ The logical server was being actively managed when the Matrix recovery managementconfiguration was invoked (logical server modification, logical server activation, or logicalserver deactivation was in progress).

• No configuration operation can be runPossible causes include:

◦ An Activate, Deactivate, or Import operation is in progress.

◦ Another configuration operation may be in progress.

• Unable to import Storage Management Servers as part of an import operationPossible causes include:

◦ The Storage Management Server was not discovered in the Matrix OperatingEnvironment user interface.

◦ The credentials associated with the Storage Management Server do not include theuser name specified in the Matrix recovery management configuration at the Local Site.

◦ The Matrix Operating Environment user interface is unable to communicate with theStorageManagement Server specified in the Matrix recovery management configurationat the Local Site.

68 Troubleshooting

• Import operation failedPossible causes include:

◦ The import file is not valid.

◦ The Logical Server Automation service is not running.

◦ There were recovery logical servers in an active state at the time of the import.

• Flex-up Server/Flex-up disk/Add Disk requests fail and pause, unable to complete theoperation when using nonrecoverable IO servicesPossible cause includes:You may have manually set Disaster Recovery Enabled flag on auto-generated SPEs.Disable the Disaster Recovery Enabled flag and retry.

Configuration error messagesThis section lists Matrix recovery management configuration error messages.

Error: Matrix recovery management is being quiesced. Currently runningoperations will be allowed to complete. No configuration changes areallowed at this time.

Error message

The user attempts to change the configuration of Matrix recovery management when the systemis being quiesced.

Cause

Wait for Matrix recovery management to be unquiesced and retry the configuration change.Action

Error: Matrix recovery management is quiesced. No configuration changesare allowed at this time.

Error message

The user attempts to change the configuration of Matrix recovery management when the systemhas been quiesced.

Cause

Wait for Matrix recovery management to be unquiesced and retry the configuration change.Action

You are not authorized to access this page.Error message

The user is not authorized to use Matrix recovery management, or the local CMS hostname cannotbe resolved by the DNS.

Cause

Check the HPE SIM andMatrix recovery management log files for details, then contact the networkadministrator to add the local CMS hostname to the DNS.

Action

Cannot verify the host name specified.Error message

The hostname specified for the CMS for either the Local Site or the Remote Site is not locatablein the DNS.

Cause

Verify that a valid DNS entry with a fully qualified domain name exists for each CMS.Action

Cannot create/edit the site information.Error message

The hostname specified for the CMS does not include a fully qualified domain name associatedwith the local CMS.

Cause

Ensure that the CMS for the Local Site has the fully qualified domain of the local host.Action

Configuration error messages 69

Storage Management Server not discovered by CMS. Discover server andretry.

Error message

Each Storage Management Server to be configured in Matrix recovery management must bediscovered in the Matrix Operating Environment user interface with appropriate credentialsspecified.

Cause

Ensure that the Storage Management Server is discovered in the Matrix Operating Environmentuser interface. If not, discover the server by using Options→Discovery in the Matrix OperatingEnvironment user interface.

Action

Error: Invalid storage manager username and/or domain name.Error message

The login credentials for the server stored in the Matrix Operating Environment user interface donot include the user name specified as part of the Storage Management Server configurationoperation in Matrix recovery management.

Cause

For the server specified, ensure that the login credentials stored in the Matrix OperatingEnvironment user interface include the credentials for the user name specified.

Action

Error: Failed to add/modify EVA Storage Management Server. Credentialfor Storage Management Server does not exist. Check input and retry.

Error message

The hostname, port number, and user name specified cannot be validated.Cause

Ensure that the server identified by the hostname is an P6000 Command view server. Ensurethat CIMOM on the Command View server is configured to use the port number specified andthat the user name specified is a valid user on that server.

Action

Unable to add/edit Storage Management ServerError message

The server is not properly configured to be managed by the Matrix Operating Environment userinterface.

Cause

Ensure that Open SSH is installed and configured on the server, and ensure that the managednodes and CMS are trusted. This can be achieved by running Configure→Configure or RepairAgents in the Matrix Operating Environment user interface.

Action

Unable to add/edit EVA Storage Replication GroupError message

Unable to obtain information about the Storage Replication Group from the P6000 CommandView Server.

Cause

Confirm that the Storage Replication Group exists on the arrays that were listed when the P6000Storage Replication Group configuration was added or edited in Matrix recovery management.Ensure that the P6000 arrays are managed by P6000 Command View servers.

Action

1. Confirm that the SIM-S component of P6000 Command View is installed on the P6000Command View server. For information on installing SMI-S, see the P6000 Command Viewsoftware installation guide available at http://www.hpe.com/portal/site/hpesc.

2. Confirm that the port number specified during Storage Management Server configuration inMatrix recovery management is the same as the WBEM port number configured on the P6000command view server (for example, 5989). For more information, see the “CIMOM” serverconfiguration section in the P6000 command view software installation guide.

3. Refresh the P6000 Command View server CIM database by running the refresh command.For details on the Discover command, see the “Configuring HPE SMI-S EVA to DiscoverP6000 Command View Software Installation Guide Command View EVA Arrays” section inthe P6000 command view software installation guide.

4. Ensure that the P6000 Command View Software service and CIMOM service are running.Start or restart these services if necessary.

70 Troubleshooting


Unable to add/edit XP Storage Replication GroupError message

Matrix recovery management is unable to obtain information about the Storage Replication Groupfrom the RAID manager, or the information obtained does not match the information provided

Cause

when the P9000 Storage Replication Group configuration was added or edited in Matrix recoverymanagement.

Ensure that the RAID manager instance on the local Storage Management Server is running.From the install directory of the RAID Manager, run pairdisplay –g <group name> (where

Action

<group name> is the name of the Storage Replication Group being added or edited) to see ifthe Storage Replication Group is configured and the local array serial number, remote array serialnumber, and storage replication type matches the data provided when the P9000 StorageReplication Group configuration was added or edited in Matrix recovery management.

Unable to add CLX credentials for this 3PAR Storeserv Storage system.Error message

The encrypted password file for the corresponding 3PAR StoreServ Storage system is incorrector the password for the storage system has changed and cannot be authenticated using theexisting password file.

Cause

Ensure the correct password file is present in the /storage/3par/conf directory where Matrixrecovery management is installed for both the local and remote 3PAR StoreServ Storage systems.If necessary, regenerate the password files. Retry the StorageManagement ServerAdd operation.

Action

Error: Unable to locate CLX/3PAR install path. Check if CLX/3PAR isinstalled on the system.

Error message

The required software 3PAR StoreServ cluster extension software is not installed, or the CLIversion does not match the one in the hp_ir.properties file.

Cause

Install only the CLI component of the 3PAR StoreServ cluster extension software by using thecustom install option, or update the INFORM_CLI_VERSION property in conf\hp_ir.properties where Matrix recovery management is installed on the CMS.

Action

Unable to validate this 3PAR Storage Replication Group.Error message

The parameters entered cannot be validated against those configured on the 3PAR StoreServStorage system.

Cause

Verify and correct the parameters (including volume group name on the local 3PAR StoreServStorage system, volume group name on the remote 3PAR StoreServ Storage system, local array

Action

remote copy group name, remote array remote copy group name, local and remote 3PARStoreServStorage system serial numbers, and replication mode) and retry the operation.

Unable to run Matrix recovery management operations because Matrixrecovery management Job is in progress or another Matrix recoverymanagement configuration operation is in progress.

Error message

If an Activate or Deactivate operation is in progress, no configuration operation is allowed,because the Job is in progress. If a Matrix recovery management configuration operation is inprogress, no other Matrix recovery management configuration operations are allowed.

Cause

Ensure that the failover process is not taking longer than usual and that the backend job processhp_lsdt_automation.exe is running. Wait for the Activate or Deactivate operation to finish.

Action

Unable to import Storage Management Servers.Error message

A Storage Management Server is not properly configured in the Matrix Operating Environmentuser interface.

Cause

Ensure that the Storage Management Server is discovered in the Matrix Operating Environmentuser interface. Ensure that Open SSH is installed and configured on the Storage Management

Action

Server. If not, runConfigure→Configure or Repair Agents from theMatrix Operating Environment

Configuration error messages 71

user interface. Ensure that the credentials for the user are also the login credentials for the serverin the Matrix Operating Environment user interface.

Import Failed.Error message

Possible causes include an invalid import file, HPE Logical Server Automation service is notrunning, or one or more recovery logical servers are in an active state.

Cause

Ensure that a valid file exported from the Local Site is used to import the Matrix recoverymanagement configuration at the Remote Site. Confirm that the Logical Server Automation service

Action

is running on the CMS. Ensure that all recovery logical servers to be managed by Matrix recoverymanagement are in a deactivated state at the time of theMatrix recovery management configurationimport.

Import succeeded but not all storage managers have been fully configured.Check Matrix recovery management log files for details.

Error message

Credentials were not configured for one or more remote Storage Management Servers.Cause

Action 1. Look at the lsdt.log and lsdt_web.log file to find the remote Storage ManagementServers with credentials that were not configured.

2. Make sure that the Storage Management Servers are up and healthy.3. Discover the Storage Management Servers on the CMS.4. Click Matrix recovery management user interface Storage Management Servers tab.5. Select the Storage Management Server that you want to configure, and click Edit.6. Select the Refresh SIM Password box and click Save.

Warning messagesThis section lists Matrix recovery management warning messages.

Warning: Matrix recovery management is being quiesced. Currently runningoperations will be allowed to complete. No new operations can be started.

Warningmessage

Matrix recovery management is being quiesced. All configuration buttons (Create, Edit, Delete,etc…) are disabled.

Cause

Wait for Matrix recovery management to be unquiesced.Action

Warning: Matrix recovery management is quiesced. No new operations willbe allowed.

Warningmessage

Matrix recovery management has been quiesced. All configuration buttons (Create, Edit, Delete,etc…) are disabled.

Cause

Wait for Matrix recovery management to be unquiesced.Action

Warning: Unable to remove CLX credentials for <storage_management_server_name> (these server credentials may not exist in CLX). Matrix

Warningmessage

recovery management will continue to delete <storage_management_server_name>. HP recommends that you check and manually remove the CLXcredentials.

Failure to clean a CLX credential after successfully deleting a Storage Management Server.Cause

To check and clean a CLX credential, open a command prompt window on the CMS:Action• Change to the CLX bin directory. For example, cd c:\Program

Files\Hewlett-Packard\Cluster Extension 3PAR\bin

• ExecuteCLX3PARCONFIG array /remove name=<storage_management_server_name>

• Check the command output message.

72 Troubleshooting

Matrix recovery management Job troubleshootingFailure of a Matrix recovery management Activate, Deactivate, Recovery Group import, orSite import Job, may intermittently occur due to a transient error condition between the softwarecomponents.By default, Matrix recovery management automatically retries failed activation and deactivationoperations on logical servers. If a Recovery Group import or site import Job fails, there is noautomatic retry, and the restart option is not available.The details of each failed Job are logged to the lsdt.log file under the Matrix recovery managementinstalled directory, for example, C:\Program Files\HP\Insight Recovery\logs. If thelsdt.log indicates that a site import Job failed due to one or more Recovery Groups that werenot successfully imported, you can attempt to complete the site import by importing the individualRecovery Groups that were not successfully imported in the site import operation. IndividualRecovery Groups can be imported from the Recovery Groups tab. If the cause of the site importJob failure is more extensive, you will need to return to the Sites tab and redo the entire siteimport. If you choose to disable automatic retry, you can edit the MAX_RETRY_ATTEMPTS propertyin the hp_ir.properties file in the conf directory where Matrix recovery management isinstalled. To disable automatic retry, set the MAX_RETRY_ATTEMPTS property to 0. To re-enablethe automatic retry of failed operations by Matrix recovery management, set theMAX_RETRY_ATTEMPTS property to 1.

NOTE:• Failed activate and deactivate Jobs will be retried only after the operation has been attempted

on all of the logical servers in the configuration.• Job states for logical server and Recovery Group Jobs will not be marked failed when a retry

of the operation is pending.

To identify the subsystem where the failure occurred, review the remainder of this section forinformation on which log files you need to look at and what action you need to take. For additionalinformation, you can view the lsdt.log file in the logs directory where Matrix recoverymanagement is installed on the system.A Job in Matrix recovery management is an automated, multistep process that activates ordeactivates selected Recovery Group Sets on a site, imports a specific Recovery Group, orimports a Site configuration. For example, the Job with a Job Id of 3288 shown in Figure 23 (page73) is an Activate Job. It has an Entity of type site and an Operation of type activate. You willalso notice the Failed icon in the Status column indicating that Job 3288 has failed.

Figure 23 Jobs screen

For a failed Job, click the check box next to the Job Id to get detailed information about theassociated Sub Jobs.A site Job contains a Sub Job for each Recovery Group. Similarly, each Recovery Group hasSub Jobs for its Storage Replication Group and logical server, respectively.

Matrix recovery management Job troubleshooting 73

To troubleshoot a site Job and identify the source of the error, drill down to each of the associatedfailed Sub Jobs to determine what operation failed and the reason for the failure, as shown inFigure 24 (page 74).

Figure 24 Expanded Jobs screeen

For additional information on troubleshooting, see the Matrix Operating Environment LogicalServer management user guide and the Matrix operating environment infrastructure orchestrationuser guid available at Hewlett Packard Enterprise Information Library.After the problem is corrected, from the Jobs screen you can restart the Job by clicking restartin the Status column for the failed Job, as illustrated in Figure 25 (page 75).

74 Troubleshooting


Figure 25 Restarting a failed job

NOTE: Restarting the Job retries only Sub Jobs that previously failed; servers associated withcompleted Jobs or Sub Jobs are not impacted.

IMPORTANT: If correcting the problem that caused the Job to fail included reconfiguration oflogical servers, before you restart the Job, go to the Recovery Groups tab and delete theRecovery Groups that contain the reconfigured logical servers. Recovery Groups that have beendeleted due to the reconfiguration of logical servers can be recreated with the reconfigured logicalservers after the Job is successfully completed.

The following troubleshooting issues are addressed in this section:• Failover job failed because storage failover failed.

Possible causes include:

◦ Storage Management Servers were not available at the time of the failover.

• Failover job succeeded but recovery logical servers are not activated.Possible causes include:

◦ Recovery Groups that contain logical servers that are in Maintenance Mode at theRemote Site.

• Failover job failed because there are no sufficient licensed physical servers or virtualmachines to host the logical servers.Possible causes include:

◦ Physical servers are running other workloads.

◦ Hypervisor management software (for example, VMware vCenter Server) is not runningon the Remote Site.

Matrix recovery management Job troubleshooting 75

• Matrix recovery management job failed because of unlocatable logical server inMatrix OE logical server management.Possible causes include:

◦ A logical server managed by Matrix recovery management was removed from MatrixOE logical server management before it was unmanaged inMatrix recoverymanagement.

• Matrix recovery management job failed because an operation failed in Matrix OElogical server management for the logical server.Possible causes include:

◦ Power on of the logical server may have failed.

Failover error messagesThis section contains Matrix recovery management failover error messages.

Failover job fails because storage failover of Storage Replication Groupsfailed

Error message

Possible causes include: The Storage Management Server was not being accessible during thetime of failover, or the status of the Storage Replication Groups does not permit a storage failover.

Cause

Ensure that at least the local Storage Management Server is accessible and actively managingone of the arrays. To further triage the problem:

Action

• If you are using P6000 disk arrays, view the clxevarun.log in STORAGE/EVA/log locatedwhere Matrix recovery management is installed on the CMS.

• If you are using P9000 disk arrays, view clxrun.log in the logs directory where CLX forP9000 is installed on the local Storage Management Server.

• If you are using 3PAR StoreServ Storage systems, view clx3parrun.log in the logs directorywhere 3PAR StoreServ Cluster Extension Software is installed on the CMS.

Failover job fails because of unlocatable logical server in LSMError message

A logical server may have been deleted using Matrix OE visualization without removing theRecovery Group containing the logical server from Matrix recovery management.

Cause

Remove the logical server from the Recovery Group in Matrix recovery management and rerunthe failover job. Also, look at the lsdt.log file located in the logs directory where Matrix recoverymanagement is installed.

Action

Failover job succeeds but certain logical servers are not activatedError message

Recovery Groups containing those logical servers may be in Maintenance Mode.Cause

Disable Maintenance Mode on the Recovery Groups and rerun the failover operation. Also lookat the lsdt.log file located in the logs directory where Matrix recovery management is installed.

Action

Failover job fails because of insufficient serversError message

There may not be enough licensed physical resources capable of hosting the logical servers.Cause

Ensure that there are enough physical servers licensed for the Matrix Operating Environmentavailable to host the logical servers managed by Matrix recovery management. Also look at thelsdt.log file located in the logs directory where Matrix recovery management is installed.

Action

76 Troubleshooting

Matrix recovery management log filesThere are several log files available with detailed information that you can view to help identifythe sources of Matrix recovery management failover or failback problems:• For errors that occur during the initial Matrix recovery management configuration steps, view

the lsdt.log and lsdt_web.log file located in the lsdt directory where HPE SystemInsight Manager is installed on the system.

• For errors that occur during a failover, check the lsdt.log file in the logs directory whereMatrix recovery management is installed for details on the specific operation that failed.

• P6000 Storage failover related errors are written to the clxevarun.log file in the STORAGE/EVA/log directory located where Matrix recovery management is installed on the system.For a list of log files to use for troubleshooting P9000 storage failover issues, see the P9000Cluster Extension software administrator guide.

• P9000 Storage failover related errors are written to the clxrun.log file located in theCluster Extension XP\log directory of the storage management server accessed byMatrix recovery management to perform the storage failover operation.Additional information on logical server activation and deactivation issues initially reportedin the Matrix recovery management Jobs screen can be found in the Matrix OE visualizationLogical Server Job Status screen. Select a logical server job based on its Job Title andview its Job Details.

• For a list of log files to use for troubleshooting 3PAR StoreServ Storage failover issues, seethe 3PAR StoreServ Cluster Extension software administrator guide available at http://www.hpe.com/portal/site/hpesc.

DR Protected IO services troubleshootingFor failures that occur during Matrix recovery management operations on IO services RecoveryGroups, the following general troubleshooting steps apply:1. Check the lsdt.log file in the logs directory where Matrix recovery management is installed

for details on the specific operation that failed.2. Check the status of the IO services in the Matrix IO Services page to see if the IO services

are in the correct state for the specific operation that failed.3. Check the hpio.properties files and the dr.properties files on the Local and Remote

Site CMS to make sure they are set correctly. For more information see “DR Protection forIO services” (page 28).

4. If an activate, deactivate, or import operation failed, go to the Matrix recovery managementJobs screen and review the Job details.

5. If necessary, go to the IO Requests screen to view the requests on the IO services involvedin the failed Matrix recovery management operation.

6. If necessary, check the hpio-controller.log file in the logs directory where IO is installedfor details on the specific IO operation that failed.

7. For additional information on troubleshooting issues related to IO services, see theTroubleshooting section in the Matrix Operating Environment Infrastructure Orchestrationuser guide available at Hewlett Packard Enterprise Information Library.

Matrix recovery management log files 77




DR Protected IO services configuration troubleshootingApart from the configuration issues addressed in this User Guide that are common to both logicalservers and IO services, the following configuration issues apply to IO services only:• Failed to get a list of IO services that can be included in a recovery group

Possible Causes:

◦ Matrix infrastructure orchestration Windows service is not running.

◦ There are no IO services that are DR protection enabled.

◦ All DR protection enabled IO services are already configured in IO services RecoveryGroups.

◦ Communication failure with IO.

• Failed to import an IO servicePossible causes:

◦ The import file is invalid.

◦ The replica service to be created already exists and is running locally.

◦ A primary service with the same name already exists.

◦ The values of the properties in the hpio.properties and dr.properties files arenot correctly set.– DR protection is not enabled (enable.dr.protection = false).

– The datastores specified in volume.dr.replica.list do not match thedatastores for the IO service at the Remote Site.

– User name and domain name mapping is not specified.

◦ Resources (network and storage) are not available.

◦ The Matrix infrastructure orchestration Windows service is not running.

◦ There is a communication failure with IO.

• One or more Recovery Groups in the Matrix recovery management configuration areinconsistent with the Matrix OE logical server management or IO configurationPossible causes include:

◦ TheMatrix OE logical server management service, the Matrix infrastructure orchestrationWindows service, or both of these services were not running when Recovery Groupswere configured in Matrix recovery management.

◦ The logical sever or IO service was actively managed when the Matrix recoverymanagement configuration was invoked (modification, activation, or deactivation wasin progress).

78 Troubleshooting

IO services configuration error messages

Unable to get the IO service.Error message

The IO service does not exist or Matrix recovery management failed to get the IO serviceinformation from the Matrix infrastructure.

Cause

Check the Matrix recovery management and IO log files for more information on the failure. If theIO service does not exist in IO, it is possible that the IO service was removed. If the IO serviceexists, restart IO and retry the operation.

Action

Error: HP Matrix infrastructure orchestration is down, causing eitheran empty list of services, or an unfiltered list of available logical

Error message

servers (which includes the ones managed by Matrix IO services). RestartHP Matrix infrastructure orchestration and retry.

When an IO services Recovery Group was created or modified using the Matrix recoverymanagement GUI, Matrix recovery management was not able to communicate with IO to obtaina list of available IO services to be included in the Recovery Group.

Cause

Restart the Matrix infrastructure orchestration Windows service and retry the operation.Action

Error: Failed to filter out logical servers managed by Matrix IOservices. Restart Matrix IO and retry.

Error message

When an IO services Recovery Group was created or modified using the Matrix recoverymanagement GUI, Matrix recovery management was not able to communicate with IO to obtain

Cause

a list of IO services to filter out the VM logical servers that are not already part of an IO servicethat has DR Protection enabled.

Restart the Matrix infrastructure orchestration Windows service and retry the operation.Action

Error: Failed to import Matrix IO service. A primary service with thesame name already exists.

Error message

During import, a primary IO service with the same name as the imported replica IO service existsin IO.

Cause

Either delete the primary IO service at the Local Site or recreate the primary IO service with adifferent name.

Action

Error: One or more Recovery Groups in the Matrix recovery managementconfiguration were found to be inconsistent with the LSM or Matrix

Error message

infrastructure orchestration configuration. Please delete them. Checkthe Matrix recovery management log files to identify the inconsistencythat caused the error.

Matrix recovery management was not able to communicate with Matrix Operating Environmentor IO to confirm that a logical server or IO service is managed by Matrix recovery management.

Cause

Delete the Recovery Group containing the logical server or IO service from Matrix recoverymanagement. Confirm that the Matrix OE logical server management service and the IO service

Action

are running. Ensure that the logical server and IO service are not actively modified by anotheruser. Retry adding the Recovery Group in Matrix recovery management.

DR Protected IO services troubleshooting 79

DR Protected IO services failover troubleshootingApart from the failover issues addressed in this User Guide that are common to both logicalservers and IO services, the following failover issues apply to IO services only:• Failed to activate IO service in a Recovery Group

Possible Causes:

◦ Storage resources are not available.

◦ The IO service is in an invalid state for activation.

◦ The IO service does not exist.



• Failed to deactivate IO service in a Recovery GroupPossible Causes:


◦ The IO service does not exist.

◦ The IO service is in an invalid state for deactivation.


Matrix recovery management error messages specific to IO services

Failed to get the IO service.Error message

Matrix recovery management was not able to get the IO service information from IO.Cause

Check the Matrix recovery management and IO log files for more details on the failure. Restartthe Matrix infrastructure orchestration Windows service and retry the operation.

Action

Service activation/deactivation cannot be performed at this time sincethe service is in IN_PROGRESS state.

Error message

The IO service was operated on when the Matrix recovery management activation or deactivationoperation was performed (modification, activation, or deactivation was in progress).

Cause

Wait for the IO service to exit the IN_PROGRESS state and retry the activation or deactivationoperation.

Action

Hyper-V based IO service or logical server fails activationSymptom: Hyper-V based Matrix service or logical server fails activation with the following errorin Matrix infrastructure orchestration or Matrix OE visualization jobs screen:Unable to bring resource online. (Virtual Machine Management error code: -1)Possible cause: The Device Manager in the Hyper-V cluster node lists the disk devices previouslyfailed over to the Remote Site as pseudo devices. This can cause Windows disk managementnot to see the devices that are made available as read-write as a result of storage failover. Oneof the reasons for this symptom may be the way multi-path I/O (MPIO) represents the same LUNafter a failover and failback.

80 Troubleshooting

Suggested Actions: The following are two workarounds:

• Perform a reboot of the cluster nodes that are failing to see the disks, even after the storagereplication group has failed over to the Local Site (as part of Matrix RM failover) to fix theissue. After the reboot is complete, check the cluster disks that are visible in the MicrosoftFailover Cluster Manager UI and restart the Matrix RM activate operation.

• Uninstall the ghosted pseudo devices that are left behind as a result of storage failover toRemote Site, followed by a rescan for storage. These pseudo devices can be seen in DeviceManager’s disk drives (ensure that the hidden devices are seen only when View option isselected.

NOTE: The uninstall step must be done cautiously to ensure that only the ghosted disksare uninstalled and not the other valid disks.

Hyper-V based IO service or logical server fails activation 81

8 Support and other resourcesAccessing Hewlett Packard Enterprise Support

• For live assistance, go to the Contact Hewlett Packard Enterprise Worldwide website:www.hpe.com/assistance

• To access documentation and support services, go to the Hewlett Packard Enterprise SupportCenter website:www.hpe.com/support/hpesc

Information to collect• Technical support registration number (if applicable)

• Product name, model or version, and serial number

• Operating system name and version

• Firmware version

• Error messages

• Product-specific reports and logs

• Add-on products or components

• Third-party products or components

Accessing updates• Some software products provide a mechanism for accessing software updates through the

product interface. Review your product documentation to identify the recommended softwareupdate method.

• To download product updates, go to either of the following:

Hewlett Packard Enterprise Support Center Get connected with updates page:www.hpe.com/support/e-updates

◦

◦ Software Depot website:www.hpe.com/support/softwaredepot

• To view and update your entitlements, and to link your contracts, Care Packs, and warrantieswith your profile, go to the Hewlett Packard Enterprise Support Center More Informationon Access to Support Materials page:www.hpe.com/support/AccessToSupportMaterials

IMPORTANT: Access to some updates might require product entitlement when accessedthrough the Hewlett Packard Enterprise Support Center. You must have an HP Passportset up with relevant entitlements.

Security bulletin and alert policy for non HPE owned software componentsOpen source software (such as OpenSSL) or third-party software (such as Java) are sometimesincluded in HPE products. Hewlett Packard Enterprise discloses that the non HPE owned softwarecomponents listed in the Insight Management end user license agreement (EULA) are includedwith Insight Management. The EULA is included with the Insight Management Installer on InsightManagement DVD #1.

82 Support and other resources

http://www.hpe.com/assistance

http://www.hpe.com/support/hpesc

www.hpe.com/support/e-updates

http://www.hpe.com/support/softwaredepot

http://www.hpe.com/support/AccessToSupportMaterials

Hewlett Packard Enterprise addresses security bulletins for the software components listed inthe EULA with the same level of support afforded HPE products. Hewlett Packard Enterprise iscommitted to reducing security defects and helping you mitigate the risks associated with securitydefects when they do occur.When a security defect is found, Hewlett Packard Enterprise has a well defined process thatculminates with the publication of a security bulletin. The security bulletin provides you with ahigh level description of the problem and explains how to mitigate the security defect.

Registering for software technical support and update serviceInsight Management components and software products include one year of 24 x 7 HPE SoftwareTechnical Support and Update Service. This service provides access to HPE technical resourcesfor assistance in resolving software implementation or operations problems.The service also provides access to software updates and reference manuals, either in electronicform or on physical media, as they are made available from Hewlett Packard Enterprise.Customers who purchase an electronic license are eligible for electronic updates only.With this service, Insight Management customers benefit from expedited problem resolution andproactive notification and delivery of software updates. For more information about this service,see the following website: http://www.hpe.com/services/insight.Registration for this service takes place following online redemption of the license certificate.There are two methods for registering:• If you received a license entitlement certificate, automated registration for this service takes

place upon online redemption of the license certificate/key.• If the license information you received for your product instructs you to register for Software

Technical Support and Update Service, you must follow the instructions to be eligible fortelephone support and product updates.

How to use your software technical support and update serviceAs Hewlett Packard Enterprise releases updates to software, the latest versions of the softwareand documentation are made available to you. The Software Updates and Licensing portal givesyou access to software, documentation, and license updates for products on your HPE softwaresupport agreement.You can access this portal from the Hewlett Packard Enterprise Support Center :www.hpe.com/support/hpescAfter creating your profile and linking your support agreements to your profile, see the SoftwareUpdates and Licensing portal at http://www.hpe.com/info/hpesoftwareupdatesupport to obtainsoftware, documentation, and license updates.

HPE authorized resellersFor the name of the nearest HPE authorized reseller, see the following sources:• In the United States, see the HPE U.S. service locator website:

http://www.hpe.com/service_locator

• In other locations, see the Contact HPE worldwide website:http://www.hpe.com/us/en/contact-hpe/ww-contact

Registering for software technical support and update service 83

http://www.hpe.com/services/insight


http://www.hpe.com/info/hpesoftwareupdatesupport

http://www.hp.com/service_locator

http://www.hpe.com/us/en/contact-hpe/ww-contact

Websites

LinkWebsite

www.hpe.com/info/enterprise/docsHewlett Packard Enterprise Information Library

www.hpe.com/support/hpescHewlett Packard Enterprise Support Center

www.hpe.com/assistanceContact Hewlett Packard Enterprise Worldwide

www.hpe.com/support/e-updatesSubscription Service/Support Alerts

www.hpe.com/support/softwaredepotSoftware Depot

www.hpe.com/support/selfrepairCustomer Self Repair

www.hpe.com/info/insightremotesupport/docsInsight Remote Support

www.hpe.com/info/hpux-serviceguard-docsServiceguard Solutions for HP-UX

www.hpe.com/storage/spockSingle Point of Connectivity Knowledge (SPOCK)nl

Storage compatibility matrix

www.hpe.com/storage/whitepapersStorage white papers and analyst reports

Customer self repairHewlett Packard Enterprise customer self repair (CSR) programs allow you to repair your product.If a CSR part needs to be replaced, it will be shipped directly to you so that you can install it atyour convenience. Some parts do not qualify for CSR. Your Hewlett Packard Enterprise authorizedservice provider will determine whether a repair can be accomplished by CSR.For more information about CSR, contact your local service provider or go to the CSR website:www.hpe.com/support/selfrepair

Remote supportRemote support is available with supported devices as part of your warranty, Care Pack Service,or contractual support agreement. It provides intelligent event diagnosis, and automatic, securesubmission of hardware event notifications to Hewlett Packard Enterprise, which will initiate afast and accurate resolution based on your product’s service level. Hewlett Packard Enterprisestrongly recommends that you register your device for remote support.For more information and device support details, go to the following website:www.hpe.com/info/insightremotesupport/docs

Related informationFor support, software updates, and additional information on Matrix recovery management andother products used with Matrix recovery management, see the following websites:• Matrix recovery management website at http://www.hpe.com/info/insightrecovery

• Insight Orchestration website at http://www.hpe.com/info/insightorchestration

• Matrix Operating Environment website at http://www.hpe.com/info/matrixoe

• Insight Control website at http://www.hpe.com/info/insightcontrol

• Insight Control virtual machine management website at http://www.hpe.com/info/vmmanage

• Insight Control server deployment website at http://www.hpe.com/info/rdp

• VMware Documentation at http://www.vmware.com/support/pubs

84 Support and other resources

http://www.hpe.com/info/enterprise/docs


http://www.hpe.com/assistance

http://www.hpe.com/support/e-updates

http://www.hpe.com/support/softwaredepot

http://www.hpe.com/support/selfrepair

http://www.hpe.com/info/insightremotesupport/docs

http://www.hpe.com/info/hpux-serviceguard-docs

http://www.hpe.com/storage/spock

http://www.hpe.com/storage/whitepapers

http://www.hpe.com/support/selfrepair

http://www.hpe.com/info/insightremotesupport/docs

http://www.hpe.com/info/insightrecovery

http://www.hpe.com/info/insightorchestration

http://www.hpe.com/info/matrixoe

http://www.hpe.com/info/insightcontrol

http://www.hpe.com/info/vmmanage

http://www.hpe.com/info/rdp

http://www.vmware.com/support/pubs

Matrix recovery management documentationFor more information on Matrix recovery management, see the following sources:

• Insight Management Support MatrixProvides Matrix recovery management support information along with other Insight hardware,software, and firmware support information. Available at Hewlett Packard EnterpriseInformation Library.

• Matrix Operating Environment Release NotesInsight Control Getting Started GuideProvides information on what is new with this release, features, and change notifications forMatrix recovery management and other Matrix Operating Environment components. Availableat Hewlett Packard Enterprise Information Library.

• Matrix Operating Environment Getting Started GuideProvides installation information for Matrix recovery management and other Matrix OperatingEnvironment components. Available at Hewlett Packard Enterprise Information Library.

• Matrix Operating Environment Recovery Management User GuideProvides information on Matrix recovery management installation, configuration, testing, andtroubleshooting. Available at Hewlett Packard Enterprise Information Library.

• Matrix recovery management white papersMatrix recovery management white papers are available at Hewlett Packard EnterpriseInformation Library.

• Matrix recovery management online helpThe Matrix recovery management online help provides information on operations that areperformed from the Matrix recovery management user interface. It is accessible from theMatrix recovery management user interface and from the Help menu on the Matrix OperatingEnvironment home page.

Documentation feedbackHewlett Packard Enterprise is committed to providing documentation that meets your needs. Tohelp us improve the documentation, send any errors, suggestions, or comments to DocumentationFeedback ([email protected]). When submitting your feedback, include the documenttitle, part number, edition, and publication date located on the front cover of the document. Foronline help content, include the product name, product version, help edition, and publication datelocated on the legal notices page.

Documentation feedback 85








mailto:[email protected]

A Recover a CSV from an online pending stateIf you perform an activate operation at the Remote Site without taking the CSV offline at the LocalSite, you might see the following symptoms at the Local Site:• No storage information is available when navigating to the storage view in the Windows

Failover Cluster Management tool.• The CSV is in the online pending state.To recover from these symptoms, perform the following steps:1. Using the storage management tool of your choice, perform the following steps:

1. Unexport the CSV virtual volume from the Hyper V cluster. Note the LUN number. Leavethe virtual volume unexported until an opportune maintenance period.

2. During the maintenance period, deactivate the recovery group on the other site whereit might be currently active and take the CSV offline.

3. Perform storage failover to place the CSV virtual volume in read-write mode.4. Export the CSV virtual volume to the Hyper V cluster using the same LUN number as

when you originally exported the LUN.2. Bring the CSV online using the Hyper V cluster manager.3. Rename the CSV folder name to the original folder name.4. Perform Logical Server refresh resources on the Matrix OE visualization GUI.5. Activate the recovery group on the CMS.

86 Recover a CSV from an online pending state

Glossarybidirectionalfailover

A Matrix recovery management feature that allows Recovery Group Sets to be activated ordeactivated at either the Local Site or the Remote Site. At any point in time there can be activatedand deactivated Recovery Group Sets at both sites. In the event of a disaster, or to accommodatesite maintenance, all of the Recovery Group Sets in the Matrix recovery managementconfiguration can be deactivated at one site, and activated at the other site.

CMS HPE System Insight Manager (HPE SIM) Central Management Server—A system in themanagement domain that executes the HPE SIM software. All central operations within HPESIM are initiated from this system.

consistencygroup

Consistency groups are an important property of asynchronous mode volumes. A consistencygroup is a group of LUNs that need to be treated the same from the perspective of dataconsistency (I/O ordering). A consistency group is equal to a device group in the Raid Managerconfiguration file.

cross-technologylogical servers

Logical servers that can be moved back and forth between being VC-hosted (hosted by aphysical server) or VM-hosted (hosted by a virtual machine). See VC-hosted logical server andVM-hosted logical server for more information.

CSV Cluster shared volumes.discovery A feature within a management application that finds and identifies network objects. In HPE

management applications, discovery finds and identifies all the HPE systems within a specifiednetwork range.

DR Group (Disaster Recovery Group) The P6000 Continuous Access Software term for a StorageReplication Group.

DR Protected Logical servers and Matrix infrastructure orchestration services that are managed by Matrixrecovery management are called as DR Protected (disaster recovery protected) logical serversand IO services.

HPE SIM HPE System Insight ManagerIO services Matrix operating environment infrastructure orchestration services.IO serviceslogical server

A logical server that is associated with IO services.

Job An automated, multistep process associated with:

• A Matrix recovery management Activate or Deactivate operation.

• A Matrix recovery management Site import operation.

• A Matrix recovery management Recovery Group import operation.

Local Site In a Matrix recovery management configuration, this is the set of managed nodes andcorresponding CMS that your browser is interacting with.

logical server A logical server is a management abstraction that simplifies and optimizes the provisioning,management, and movement of a server, whether it is a physical server or a virtual machine.A logical server is the complete software image of a server, including operating system,applications, configuration data, and metadata that you create and assign to operate on aphysical server or virtual machine. Logical servers are managed by Matrix OE visualization.

MaintenanceMode

Maintenance Mode is used to test Recovery Groups to ensure they will function properly whenan Activate operation is performed. Maintenance mode temporarily stops Matrix recoverymanagement from managing a DR Protected logical server or IO service. One use ofmaintenancemode is to perform failover rehearsal. By setting MaintenanceMode on a RecoveryGroup, all of the logical servers and Matrix infrastructure orchestration services in that RecoveryGroup can be activated through the Matrix Operating Environment. Once you are satisfied withthe failover rehearsal, the Recovery Group and its corresponding logical servers and IO servicescan be brought back under the control of Matrix recovery management.

87

Matrixinfrastructureorchestrationservices

Matrix infrastructure orchestration services (IO services) quickly provision infrastructure toautomatically activate physical and virtual servers, storage, and networking from pools of sharedresources. More information on Matrix infrastructure orchestration is available at http://www.hpe.com/info/insightorchestration.

Matrix OE logicalservermanagement

A component of the Matrix Operating Environment software that manages and automatesoperations associated with logical servers, including resource provisioning, startup, andshutdown.

NPIV N_Port ID Virtualizationplanned failover A failover of all Recovery Group Sets from one site to another site, initiated in anticipation of

an imminent disaster or for planned downtime.portability group A portability group is a group of virtual machines or c-Class blades equipped with Virtual Connect

(or a combination virtual machines and c-Class blades) that a logical server can be movedbetween.

Power-Up Delay A Recovery Group configuration setting that determines the minimum time delay (in minutes)between the power-up of any two logical servers in a Recovery Group. The actual time delaymay be more than the minimum time delay specified.

Preferred Site The Preferred Site is the site where you prefer the Recovery Group to be activated unlesscircumstances require activation of the Recovery Group at the Secondary Site.

primary IOservice

A primary IO service is the first instance of an IO service that you select for DR Protection in aMatrix recovery management configuration. A replica IO service is a service Matrix recoverymanagement created during an import operation at a recovery site based on the primary IOservice definition in the Matrix recovery management export file.

private A subnet that is not routed outside the data center and typically contains addresses only in the192.x.x.x or 10.x.x.x address ranges. A subnet that is accessible to the Internet and cannotcontain IP addresses in the 192.x.x.x or 10.x.x.x address ranges.

RDM Raw Device Mappingrecoverable IOservices

Matrix infrastructure orchestration services that have been configured so they can be includedin a DR Protected Matrix recovery management IO services Recovery Group.

Recovery Group A pairing of one or more logical servers and a single Storage Replication Group. A RecoveryGroup has a Recovery Group Start Order number associated with it. During a site failover,Recovery Groups are failed over from one site to another site in the order specified.

Recovery GroupSet

A set of Recovery Groups that share the same Preferred and Secondary sites. Recovery Groupscannot be activated or deactivated individually. Instead, all Recovery Groups that share thesame Preferred and Secondary site must be activated or deactivated as a set. Recovery GroupSets can be selected for activation or deactivation at the Local Site.

Recovery GroupStart Order

An optional number that specifies the order in which a Recovery Group is to be started duringa site failover. Recovery Groups without a start order number are started after all RecoveryGroups that have an associated start order number.

recovery logicalservers

Logical servers at the Remote Site that are included in a Recovery Group. They are associatedwith logical servers at the Local Site that are included in the same Recovery Group. Normally,they are deactivated and disabled. They are enabled and activated upon site failover.

redundant SAN The duplication of components to prevent failure of the SAN solution.Remote Site A site in a Matrix recovery management configuration that is not the Local Site.replica IO service A replica IO service is a service Matrix recovery management created during an import operation

at a recovery site based on the exported IO service definition in the Matrix recovery managementexport file.

replica IO servicelogical servers

Logical servers that are associated with replica IO services.

SAN A Storage Area Network (or subnetwork) that connects data storage devices with associateddata servers. A SAN is typically part of an overall network of computing resources.

Secondary Site The Secondary Site is the site where you prefer the Recovery Group to be on “standby” (in adeactivated state), unless circumstances require activation of the Recovery Group at theSecondary Site.

88 Glossary



split-brain Split-brain occurs when two or more instances of the same application are active simultaneously,which might lead to data corruption.

StorageDecoupling

A Matrix recovery management feature that allows failover of the logical servers or IO servicesin a Recovery Group without failing over the Storage Replication Groups that are associatedwith those logical servers or IO services.

StorageManagementServer

As part of the Matrix recovery management configuration process, servers that manage P6000,P9000, 3PAR StoreServ, or User Defined storage devices must be defined. These servers arecalled as Storage Management Servers.

StorageReplicationGroup

A set of LUNs across which storage replication preserves write-order at the replication targetstorage array. In P6000 Continuous Access Software terminology, this is known as a DRGroup.In P9000 Continuous Access Software terminology this is known as a consistency group.

Sub Job A Sub Job is component of a Matrix recovery management activate, deactivate, Recovery Groupimport or Site import Job. For example, a Site activate Job would include a Recovery Groupactivate Sub Job for each Recovery Group that is activated on that site, and each RecoveryGroup activate Sub Job would include a logical server activate Sub Job for each logical serverin that Recovery Group.

unplannedfailover

A failover of all Recovery Group Sets from one site to another site, initiated in response to anunforeseen event that caused an outage at the site where the Recovery Group Sets wereactivated.

User Definedstorage

Matrix recovery management provides a User Defined storage adapter interface specificationto enable one-step Matrix recovery management failover capability for storage types that aresupported by Matrix OE, but not yet integrated with Matrix recovery management.

User DefinedStorageManagementServers

When you install a storage adaptor in your Matrix recovery management configuration for astorage type other than 3PAR StoreServ, P6000, or P9000, you can define (and edit) StorageManagement Servers based on that storage adaptor. These are called as User Defined StorageManagement Servers in a Matrix recovery management configuration. For example, if youcreate and install a storage adaptor named EMC, the Storage server type drop-down menufor configuring a Storage Management Server will allow you to select EMC as the storage servertype. For more information, see the Matrix Operating Environment getting started guide availableat Hewlett Packard Enterprise Information Library.

VC-hostedlogical server

A logical server running on a c-Class blade equipped with Virtual Connect.

VM-hostedlogical server

A logical server running on a virtual machine under the control of a hypervisor.

89


Index

Aaccessingupdates, 82

CCommand Linedrsync, 40

contacting Hewlett Packard Enterprise, 82Cross-technology logical serversPhysical, 49Virtual, 49

customer self repair, 84

Ddocumentationproviding feedback on, 85

DR ProtectedConfigure IO services, 29Cross-technology logical servers , 60IO services, 28

drsyncdr.properties, 41Globbing expressions, 43

IImportJob, 25Recovery Groups, 25

MMatrix recovery managementConfiguration, 9Configuration steps, 20Cross-technology logical servers , 45Documentation, 85Export, 21Import, 21Installation, 9Licensing, 10Log files, 77Overview, 6Supported platforms, 49Uninstall, 10

RRecovery GroupDisabling Maintenance Mode, 35Enabling Maintenance Mode, 35Failover, 34Planned Failover, 36Testing, 34Unplanned Failover, 37

remote support, 84

SSet up

3PAR StoreServ Storage, 14Local Site Logical Servers, 17Network, 10P6000, 13P9000, 13Remote Site Logical Servers, 17Storage, 11

supportHewlett Packard Enterprise, 82

TToolsPINT , 53PISA , 52

Troubleshooting, 66DR Protected IO services, 77Error messages, 69Failover, 76Warning messages, 72

TroubleshotingJobs, 73

Uupdatesaccessing, 82

User DefinedCreate, 15Install, 15

Wwebsites, 84customer self repair, 84

90 Index

HPE Matrix Operating Environment 7.5 Recovery Management User Guide

Documents

Transcript of HPE Matrix Operating Environment 7.5 Recovery Management User Guide