with IBM Corp.€¦ · A r unning migration may be paused at any time. Any in-flight copy...

IBM Cloud Object Storage System™

Version 3.14.3

Data Migration Service

IBM

This edition applies to IBM Cloud Object Storage System and is valid until replaced by new editions.

© Copyright IBM Corporation 2016, 2019.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.

Contents

Document information . . . . . . . . vIntended purpose and audience . . . . . . . . v

Chapter 1. Overview . . . . . . . . . 1

Chapter 2. Purpose . . . . . . . . . . 3

Chapter 3. Terminology and components 5

Chapter 4. Workflow . . . . . . . . . 7Initiation. . . . . . . . . . . . . . . . 7Source vault listing . . . . . . . . . . . . 8Object copy operations . . . . . . . . . . . 8Vault proxy . . . . . . . . . . . . . . . 8

Chapter 5. Data migration using themanager . . . . . . . . . . . . . . 9Setup . . . . . . . . . . . . . . . . . 9

Prerequisites . . . . . . . . . . . . . 9Select source and target vault . . . . . . . . 9Vault proxy with rename . . . . . . . . . 10User permissions . . . . . . . . . . . 10Vault deployment . . . . . . . . . . . 10

Initiation . . . . . . . . . . . . . . . 10Operation . . . . . . . . . . . . . . . 11

Pause . . . . . . . . . . . . . . . 11Resume. . . . . . . . . . . . . . . 11Abort . . . . . . . . . . . . . . . 11Change the data migration rate . . . . . . . 12Monitoring . . . . . . . . . . . . . 12

Event console. . . . . . . . . . . . . . 14Vault migration summary page . . . . . . . . 15

Chapter 6. Performance implicationsduring migration . . . . . . . . . . 17

Chapter 7. Blocked state . . . . . . . 19

Chapter 8. Halted with failures state . . 21

Chapter 9. Limitations . . . . . . . . 23

Notices . . . . . . . . . . . . . . 25Trademarks . . . . . . . . . . . . . . 27Homologation statement . . . . . . . . . . 27

© Copyright IBM Corp. 2016, 2019 iii

iv Data Migration Service

Document information

Intended purpose and audienceThe purpose of the data migration service is to migrate named and simple objects from one vault toanother vault that exists in the same system.

© Copyright IBM Corp. 2016, 2019 v

vi Data Migration Service

Chapter 1. Overview

The data migration service allows for vault level data migration. Objects are copied from a source vaultto a target vault. Objects on the source vault are not deleted. The vault proxy feature enables seamlessaccess to objects on both source and destination during migration.

© Copyright IBM Corp. 2016, 2019 1

2 Data Migration Service

Chapter 2. Purpose

The data migration service can be used to:v Reconfigure IDAv Change storage typev Enable/disable Secure Slicev Move off of an unhealthy source poolv Move to a storage pool with increased capacityv Refresh existing hardware


Chapter 3. Terminology and componentsMigration Work Queue

Contains a list of objects in need of migration. This data is persisted to the target vault.

Listing AgentPopulates the migration work queue by listing the source vault. These agents run on all sourcevault Slicestors.

Migration AgentQueries migration work queue and performs data copy operations. These agents run on all targetvault Slicestors.

Vault ProxyEnables seamless access to objects during migration by forwarding requests to the source vault asneeded. This request routing is performed by Accesser devices as needed.


Chapter 4. Workflow

Data Migration follows a simple workflow.

To improve the bucket listing performance on an existing bucket when using an old vault format, youmust migrate data from the existing vault to the newly created standard vault using IBM COS DataMigration Service (DMS). Follow these steps for improving bucket listing performance on a standardvault:1. Ensure a standard vault format exists with the “Name Index Enabled” checked.2. Choose “Create new Vault” and “Rename Destination Vault” options when setting up a vault A1

migration.

Note: only under these conditions when IBM COS performs DMS does it apply better index andcopies bucket and object ACL, IP Access Control, Hard/Soft Quota, metadata, eTag, timestamp to the

target vault.

InitiationA data migration is initiated using the Manager UI and/or Manager REST API.

The user may opt to rename the target vault and set up a proxy configuration to redirect traffic to thetarget vault. See Chapter 5, “Data migration using the manager,” on page 9 for more detailed informationabout setup and configuration.

Figure 1. Data migration workflow


Source vault listingThe listing agents begin to generate a source vault listing after the migration has been initiated. Objectswritten to the source vault after the listing has started may not be added to the migration work queueand consequently will not be copied to the target vault. To ensure that all objects are copied, configure avault proxy, rename the target vault, verify all new client IO is passing through the proxy, and allow anyin-progress writes to finish before starting the migration.

Object copy operationsObject copy operations run in parallel with source vault listing operations. The migration agents will startto copy objects as soon as there are items found in the migration work queue. If a new version of anobject is written to the target vault before the object has been migrated the older version on the sourcewill not be migrated. If an object is deleted by the user during migration, it will not be migrated.User-defined object metadata, last modified time, object ACLs, and all S3 versions are preserved. If a S3multipart upload is completed during a migration, the object will be migrated.

Vault proxyThe Vault Proxy feature enables seamless access to objects on both source and destination duringmigration.v Write: Object write operations are performed against the target vault only.v Read: Object read operations are performed first to the target vault, then to the source vault if not

found.v Object Listing: Object listing operations are performed against both vaults, and the results are merged.v Delete: Object delete operations are performed against both vaults.v S3 Multipart Upload: For S3 multipart uploads, PUT PART, COMPLETE, and ABORT are performed first

to the target vault, then to the source vault if not found.


Chapter 5. Data migration using the manager

Data Migrations and Vault Proxies are configured using the Manager GUI and/or REST API. This sectionsummarizes the main configuration steps for a configuration that includes a vault proxy and source vaultrename.

Consult the Manager Administration Guide for detailed information regarding setup and configurationusing either the Manager GUI or REST API.

Setup

Prerequisites

Data migration requires:v An existing source vaultv A storage pool with sufficient capacity to store source vault datav A system user granted System Administrator or Super User rights

Select source and target vaultDuring the setup phase, the source and target vaults are identified.

The source and target vaults must reside in the same system, and must have the same versioning. Thetarget vault may be new or existing. There can be a maximum of one active migration per vault. Vaultsthat reside on expanded/merged storage pools or belong to a mirror are not supported. If an existingvault is a suitable target for the selected source vault, it will appear in the pull-down menu. To beginsetup of a vault migration, navigate to a vault configuration page and click on the Setup Vault Migrationbutton.

Figure 2. Setup vault migration


Vault proxy with renameDuring the setup phase, the user may choose to enable the vault proxy with the rename option to allowseamless access to data during migration.

This will configure a vault proxy from target vault to source vault and rename the vaults soreconfiguration of user applications is not required. The source vault name will be given to the targetvault and the source vault name is changed to its original name with source appended. To configure thisoption, check the Rename Destination Vault check box during the setup phase.>

User permissionsThe source vault ACL is automatically applied to the target vault.

Vault deploymentThe source vault deployments are automatically applied to the target vault. To prevent applications fromwriting new objects to the source vault using its new name, all Accesser deployments for the vault areautomatically removed. In-flight client I/O to the source vault will continue and succeed and new clientI/O will be directed to the target vault.

InitiationThe user manually initiates the start of migration following the setup phase. This ensures that new writesare directed to the target vault before the source vault listing phase begins, and gives the user theopportunity to allow any in-progress writes to complete. Immediately following the setup phase, theStart Migration button will be grayed-out for several minutes. This is to ensure that the configurationinformation has propagated to all of the devices in the system when the migration is started.

Figure 3. Vault rename with proxy


Operation

PauseA running migration may be paused at any time. Any in-flight copy operations will finish.

ResumeA paused migration may be resumed at any time.

AbortA running or paused migration may be aborted at any time. Any in-flight copy operations will finish.Copied data is left in its current location. The migration cannot be resumed following abort.

Figure 4. Start migration

Figure 5. Abort data migration

Chapter 5. Data migration using the manager 11

Change the data migration rateData Migration can be throttled. A per-device data transfer (MB/s) rate and/or OPs/s (objects copied persecond) rate can be set at any time. Throttling can limit the latency impact of migration, but will notimpact the latency associated with proxying requests.

MonitoringA progress bar is displayed on the target vault page in the Manager UI once the listing phase iscompleted.

This percentage complete is based on object count. For each source device, a graph showing the numberof objects scanned per second (OPs/s) is displayed. For each target device, graphs showing the datatransfer rate (bytes/sec) and objects copied per second (OPs/s) are displayed.

Note: If a vault is proxying another vault, the Accesser-Slicstor I/O for the proxied vault will appear inthe proxying vault's graphs.

Figure 6. Data migration rate limiting


Figure 7. Data migration progress bar

Figure 8. Source vault graph


Event consoleAudit messages will appear in the Manager UI Event Console for setup, start, pause, resume, throttle,and abort for each data migration.

An event for the completion of each data migration will also appear in the Manager UI Event Console.When reviewing migration activity using the event console, be sure to enable the Show Audits filter sothat all of the audit messages are displayed. This filter is enabled by default.

Figure 9. Target vault graphs


Vault migration summary pageIf there are any active migrations in a system, a Vault Migrations tab will appear on the Vault Summarypage. The user may select this tab to display information about each migration. The table can be filteredby status, vault health, and storage pool.

Figure 10. Event console

Figure 11. Vault migration summary


Chapter 6. Performance implications during migration

Latency increases due to migration processing and proxying of client requests. Throttling may be used tolimit the impact of migration, but will increase the time required to complete the migration. Throttlingwill not impact the additional latency incurred for proxied read and delete operations because thisdepends on the network between the source and target pools.


Chapter 7. Blocked state

A migration may enter a blocked state. This occurs when all of the attempts to list the source vault fail(default is five attempts).

While in the blocked state no listing requests are sent to the source vault and no data is copied to thetarget vault. The user must manually resume the migration once the health of the source vault has beenrestored. For system troubleshooting see the Manager Administration Guide.

Figure 12. Data migration blocked state


Chapter 8. Halted with failures state

A migration may halt with failures. This occurs when all of the copy attempts (default is five copyattempts for each object) for at least one object fail.

In this state no further attempts to copy the failed object(s) are made. While in this state, you may chooseto initiate retry of failed object(s) after health of source/target vaults has been restored, or to complete themigration with failures. There is no limit on the number of retry attempts. The migration will re-enterthis state if any subsequent attempts to complete without failures fail. If you choose to complete themigration with failures, the option to retry failed object(s) will no longer be available. To retry amigration that has halted with failures click the Retry button. To mark the migration as complete withfailures click the Complete With Failures button.

Figure 13. Data migration halted with failures state


Chapter 9. Limitationsv Vaults that reside on merged storage pools are not supported.v Vaults that belong to a mirror are not supported.v Simple Object data written before release 3.0 will not be migrated.v To allow seamless access to data with S3 CORS configuration enabled, the existing S3 CORs

configuration on the source vault must be manually applied to the target vault prior to migrationconfiguration.


Notices

This information was developed for products and services offered in the US. This material might beavailable from IBM® in other languages. However, you may be required to own a copy of the product orproduct version in that language in order to access it.

IBM may not offer the products, services, or features discussed in this document in other countries.Consult your local IBM representative for information on the products and services currently available inyour area. Any reference to an IBM product, program, or service is not intended to state or imply thatonly that IBM product, program, or service may be used. Any functionally equivalent product, program,or service that does not infringe any IBM intellectual property right may be used instead. However, it isthe user's responsibility to evaluate and verify the operation of any non-IBM product, program, orservice.

IBM may have patents or pending patent applications covering subject matter described in thisdocument. The furnishing of this document does not grant you any license to these patents. You can sendlicense inquiries, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle DriveArmonk, NY 10504-1785U.S.A.

For license inquiries regarding double-byte character set (DBCS) information, contact the IBM IntellectualProperty Department in your country or send inquiries, in writing, to:

Intellectual Property LicensingLegal and Intellectual Property LawIBM Japan, Ltd.19-21, Nihonbashi-Hakozakicho, Chuo-kuTokyo 103-8510, Japan

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS"WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOTLIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY ORFITNESS FOR A PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express orimplied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodicallymade to the information herein; these changes will be incorporated in new editions of the publication.IBM may make improvements and/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Any references in this information to non-IBM websites are provided for convenience only and do not inany manner serve as an endorsement of those websites. The materials at those websites are not part ofthe materials for this IBM product and use of those websites is at your own risk.

IBM may use or distribute any of the information you provide in any way it believes appropriate withoutincurring any obligation to you.


Licensees of this program who wish to have information about it for the purpose of enabling: (i) theexchange of information between independently created programs and other programs (including thisone) and (ii) the mutual use of the information which has been exchanged, should contact:

IBM Director of LicensingIBM CorporationNorth Castle Drive, MD-NC119Armonk, NY 10504-1785US

Such information may be available, subject to appropriate terms and conditions, including in some cases,payment of a fee.

The licensed program described in this document and all licensed material available for it are providedby IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement orany equivalent agreement between us.

The performance data discussed herein is presented as derived under specific operating conditions.Actual results may vary.

Information concerning non-IBM products was obtained from the suppliers of those products, theirpublished announcements or other publicly available sources. IBM has not tested those products andcannot confirm the accuracy of performance, compatibility or any other claims related to non-IBMproducts. Questions on the capabilities of non-IBM products should be addressed to the suppliers ofthose products.

Statements regarding IBM's future direction or intent are subject to change or withdrawal without notice,and represent goals and objectives only.

All IBM prices shown are IBM's suggested retail prices, are current and are subject to change withoutnotice. Dealer prices may vary.

This information is for planning purposes only. The information herein is subject to change before theproducts described become available.

This information contains examples of data and reports used in daily business operations. To illustratethem as completely as possible, the examples include the names of individuals, companies, brands, andproducts. All of these names are fictitious and any similarity to the names and addresses used by anactual business enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrate programmingtechniques on various operating platforms. You may copy, modify, and distribute these sample programsin any form without payment to IBM, for the purposes of developing, using, marketing or distributingapplication programs conforming to the application programming interface for the operating platform forwhich the sample programs are written. These examples have not been thoroughly tested under allconditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of theseprograms. The sample programs are provided "AS IS", without warranty of any kind. IBM shall not beliable for any damages arising out of your use of the sample programs.

If you are viewing this information softcopy, the photographs and color illustrations may not appear.


TrademarksIBM, the IBM logo, and ibm.com® are trademarks or registered trademarks of International BusinessMachines Corp., registered in many jurisdictions worldwide. Other product and service names might betrademarks of IBM or other companies. A current list of IBM trademarks is available on the web atCopyright and trademark information at www.ibm.com/legal/copytrade.shtml.

Accesser®, Cleversafe®, ClevOS™, Dispersed Storage®, dsNet®, IBM Cloud Object Storage Accesser®, IBMCloud Object Storage Dedicated™, IBM Cloud Object Storage Insight™, IBM Cloud Object StorageManager™, IBM Cloud Object Storage Slicestor®, IBM Cloud Object Storage Standard™, IBM Cloud ObjectStorage System™, IBM Cloud Object Storage Vault™, SecureSlice™, and Slicestor® are trademarks orregistered trademarks of Cleversafe, an IBM Company and/or International Business Machines Corp.

Other product and service names might be trademarks of IBM or other companies.

Homologation statementThis product may not be certified in your country for connection by any means whatsoever to interfacesof public telecommunications networks. Further certification may be required by law prior to making anysuch connection. Contact an IBM representative or reseller for any questions.

Notices 27

http://www.ibm.com/legal/copytrade.shtml

IBM®

Printed in USA

with IBM Corp.€¦ · A r unning migration may be paused at any time. Any in-flight copy...

Documents

Transcript of with IBM Corp.€¦ · A r unning migration may be paused at any time. Any in-flight copy...