Service Overview - support.huaweicloud.com

Cloud Data Migration

Service Overview

Issue 01

Date 2021-07-22

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2021. All rights reserved.

No part of this document may be reproduced or transmitted in any form or by any means without priorwritten consent of Huawei Technologies Co., Ltd. Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei andthe customer. All or part of the products, services and features described in this document may not bewithin the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,information, and recommendations in this document are provided "AS IS" without warranties, guaranteesor representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.

Issue 01 (2021-07-22) Copyright © Huawei Technologies Co., Ltd. i

Contents

1 What Is CDM?...........................................................................................................................1

2 Product Advantages................................................................................................................ 3

3 Supported Data Sources.........................................................................................................5

4 Application Scenarios........................................................................................................... 30

5 Basic Concepts........................................................................................................................32

6 CDM Migration Principles................................................................................................... 37

7 Related Services.....................................................................................................................40

8 Constraints.............................................................................................................................. 44

9 Quotas......................................................................................................................................49

10 Permissions Management................................................................................................. 50

Cloud Data MigrationService Overview Contents

Issue 01 (2021-07-22) Copyright © Huawei Technologies Co., Ltd. ii

1 What Is CDM?

Product OverviewCloud Data Migration (CDM) implements data mobility by enabling batch datamigration among homogeneous and heterogeneous data sources. It supports on-premises and cloud file systems, relational databases, data warehouses, NoSQL,big data, and object storage.

Based on the distributed computing framework and the parallel processingtechnology, CDM helps you migrate massive sets of data stably and efficiently. Youcan migrate data online and quickly construct a desired data structure.

Figure 1-1 CDM positioning

Functions● Table/file/entire DB migration

Tables or files can be migrated in batches. An entire database can bemigrated between homogeneous and heterogeneous databases. A job canmigrate hundreds of tables.

● Incremental data migrationCDM supports incremental migration of files, relational databases, andHBase/CloudTable, as well as with WHERE clauses and macro variables ofdate and time.

● Migration in transaction modeWhen a CDM job fails to be executed, CDM rolls back the data to the statebefore the job starts and automatically deletes data from the destinationtable.

Cloud Data MigrationService Overview 1 What Is CDM?

Issue 01 (2021-07-22) Copyright © Huawei Technologies Co., Ltd. 1

● Field conversionCDM supports field conversion functions, such as anonymization, characterstring operations, and date operations.

● File encryptionWhen files are migrated to a file system, CDM can encrypt the files written tothe cloud.

● MD5 verificationMD5 verification is supported to check the file consistency from end to endand output verification result.

● Dirty data archivingCDM can archive the data that fails to be processed during migration, hasbeen filtered out, or is not compliant with conversion or cleaning rules todirty data logs. The threshold for dirty data ratio can be set to determinewhether a task is successful.

Cloud Data MigrationService Overview 1 What Is CDM?


2 Product Advantages

Data migration is involved when you consolidate or back up data, or develop newapplications on the cloud. Generally, if you want to migrate data, you may developdata migration scripts to read data from the source and write data to thedestination. Compared with this method, CDM has the following advantages:

Table 2-1 CDM advantages

Item User-Developed Script CDM

Ease ofuse

You need to prepare serverresources, and install andconfigure software, which istime-consuming.Because the data source typesare different, the programuses different accessinterfaces, such as JDBC andnative APIs, to read and writedata. In this case, variouslibraries and SDKs are requiredwhen you write datamigration scripts, resulting inhigh development andmanagement costs.

CDM provides a web-basedmanagement console for enablingservices on web pages in real time.You can migrate data byconfiguring data sources andmigration jobs on the GUI andCDM will manage and maintainthe data sources and migrationjobs for you. In other words, youonly need to focus on the datamigration logic without worryingabout the environment, whichgreatly reduces development andmaintenance costs.CDM also provides RESTful APIs tosupport third-party system callingand integration.

Real-timemonitoring

You need to select specificversions to develop asrequired.

You can use Cloud Eye toautomatically monitor CDMclusters in real time and managealarms and notifications, so thatyou can keep track of CDM clusterperformance metrics.

Cloud Data MigrationService Overview 2 Product Advantages


Item User-Developed Script CDM

O&M free You need to develop andoptimize O&M functions,especially alarm andnotification functions, toensure system availability.Otherwise, manual attendanceis required.

With CDM, you do not need tomaintain resources such as serversand VMs. CDM has the log,monitoring, and alarm functions,which send notifications to relatedpersonnel in a timely manner toavoid 24/7 hours of manual O&M.

Highefficiency

During data migration, theread and write process iscompleted in one job. Limitedby available resources, theperformance is poor andcannot meet the requirementsof scenarios where massivesets of data need to bemigrated.

Based on the distributedcomputing framework, CDM jobsare split into independent sub-jobsand executed concurrently, whichdrastically improves datamigration efficiency. In addition,efficient data import interfaces areprovided to import data from Hive,HBase, MySQL databases, andData Warehouse Service (DWS).

Variousdatasources

Different tasks must bedeveloped for different datasources, generating a numberof scripts.

Various data sources such asdatabases, Hadoop, NoSQL, datawarehouses, and files aresupported. For details, seeSupported Data Sources.

Differentnetworkenvironments

As the cloud computingtechnology develops, userdata may be stored indifferent environments, suchas public clouds, on-premisesor hosted Internet datacenters (IDCs), and hybridscenarios. In heterogeneousenvironments, data migrationis subject to various factors,for example, networkconnectivity, which causesinconvenience fordevelopment andmaintenance.

CDM helps you easily cope withvarious data migration scenarios,including data migration to thecloud, data exchange on the cloud,and data migration to on-premisesservice systems, regardless ofwhether the data is stored on on-premises IDCs, cloud services,third-party clouds, or self-builtdatabases or file systems onElastic Cloud Servers (ECSs).

Cloud Data MigrationService Overview 2 Product Advantages


3 Supported Data Sources

Cloud Data Migration supports the following migration modes for different datasources:

● Table/file migration is used for data migration to the cloud, data exchange onthe cloud, and data migration to on-premises service systems. For details, seeSupported Data Sources in Table/File Migration.

● Entire DB migration. For details about how to migrate an entire DB, seeSupported Data Sources in Entire DB Migration.

● In addition, this section provides the data types supported in databasemigration. For details, see Data Types Supported in Dameng DatabaseMigration, Data Types Supported in Open-Source MySQL DatabaseMigration, Data Types Supported in Oracle Database Migration, and DataTypes Supported in SQL Server Database Migration.

Supported Data Sources in Table/File Migration

Table 3-1 describes the supported data sources.

Table 3-1 Supported data sources during table/file migration

Category Data Source Read Write Description

Datawarehouse

Data WarehouseService (DWS)

Supported

Supported

-

Data Lake Insight(DLI)

Supported

Supported

-

FusionInsight LibrA Supported

Notsupported

-

Hadoop(supportedonly by localstorage)

MRS HDFS Supported

Supported

Recommendedversions:● 2.8.X● 3.1.X

Cloud Data MigrationService Overview 3 Supported Data Sources



MRS HBase Supported

Supported


MRS Hive Supported

Supported

Recommendedversions:● 1.2.X● 2.3.X● 3.1.X

FusionInsight HDFS Supported

Notsupported


FusionInsight HBase Supported

Notsupported


FusionInsight Hive Supported

Notsupported


Apache HBase Supported

Notsupported


Apache Hive Supported

Notsupported


Apache HDFS Supported

Notsupported


Object storage Object StorageService (OBS)

Supported

Supported

-




Alibaba CloudObject StorageService (OSS)

Supported

Notsupported

Migration fromother objectstorage servicesto OBS is notsupported. Toperform such amigration, youare advised touse the ObjectStorageMigration Service(OMS).

Qiniu Cloud ObjectStorage (KODO)

Supported

Notsupported

Amazon SimpleStorage Service(Amazon S3)

Supported

Notsupported

Tencent CloudObject Storage(COS)

Supported

Notsupported

File system FTP Supported

Notsupported

-

SFTP Supported

Notsupported

-

HTTP Supported

Notsupported

-

Relationaldatabase

RDS for MySQL Supported

Supported

Migration fromOLTP to OLTP isnot supported. Inthis scenario, youare advised touse the DataReplicationService (DRS).

RDS for PostgreSQL Supported

Supported

Dameng database Only toDWSand Hive

Notsupported

-

RDS for SQL Server Supported

Supported

Migration fromOLTP to OLTP isnot supported. Inthis scenario, youare advised touse the DataReplicationService (DRS).

MySQL Supported

Notsupported

PostgreSQL Supported

Notsupported




Microsoft SQLServer

Supported

Notsupported

Oracle Supported

Notsupported

IBM Db2 Supported

Notsupported

SAP HANA Supported

Notsupported

-

MYCAT Supported

Notsupported

-

NoSQL Distributed CacheService (DCS)

Notsupported

Supported

Only migrationfrom MRS to DCSis supported.

Document DatabaseService (DDS)

Supported

Supported

Only migrationbetween DDSand MRS issupported.

CloudTable Service(CloudTable)

Supported

Supported

-

CloudTableOpenTSDB

Supported

Supported

-

Redis Supported

Notsupported

Only migrationfrom Redis toMRS issupported.

MongoDB Supported

Notsupported

Only migrationfrom MongoDBto MRS issupported.

Cassandra Supported

Notsupported

-

Search Elasticsearch Supported

Notsupported

-




Cloud SearchService (CSS)

Supported

Supported

-

Messagesystem

Data IngestionService (DIS)

Supported

Notsupported

-

Apache Kafka Supported

Notsupported

-

MRS Kafka Supported

Notsupported

-

DMS Kafka Supported

Notsupported

-

NO TE

● In the preceding table, the non-cloud data sources, such as MySQL, include on-premisesMySQL, MySQL built on ECSs, or MySQL on the third-party cloud.

● When DIS, Apache Kafka, or DMS Kafka functions as the migration source, CDM canonly migrate data to CSS:

Supported Data Sources in Entire DB Migration

Entire DB migration is used when an on-premises data center or a databasecreated on an ECS needs to be synchronized to a database service or big dataservice on the cloud. It is suitable for offline database migration but not onlinereal-time migration.

Table 3-2 lists the data sources supporting entire DB migration using Cloud DataMigration.

Table 3-2 Supported data sources in entire DB migration

Source DataType

Destination Data Type

MRS(HIVE)

DWS CSS OBS CloudTable

MySQL √ √ × √ ×

PostgreSQL √ √ × √ ×

Microsoft SQLServer

√ √ × × ×

Oracle √ √ × √ ×



Elasticsearch × × √ × ×

MongoDB × × × × ×

HBase × × × × √

IBM Db2 √ √ × √ ×

SAP HANA √ √ × √ ×

DWS √ √ × × ×

Hive × √ × × ×

Data Types Supported in Dameng Database MigrationWhen the source end is a Dameng database and the destination end is a Hive orDWS database, the following data types are supported.

Table 3-3 Data types supported by the Dameng database functioning as thesource end

Category

Type Description StorageFormatExample

Hive DWS

Character

CHAR Stores specifiedfixed-lengthcharacter strings.

'a' or 'aaaaa' CHAR CHAR

CHARACTER

Same as CHAR 'a' or 'aaaaa' CHAR CHAR

VARCHAR

Stores specifiedvariable-lengthcharacter strings.

'a' or 'aaaaa' VARCHAR VARCHAR

VARCHAR2

Same as VARCHAR 'a' or 'aaaaa' VARCHAR VARCHAR

Value

NUMERIC

Stores zero,positive, andnegative fixedpoint numbers.

52.36 DECIMAL NUMERIC

DECIMAL

Similar toNUMERIC

52.36 DECIMAL NUMERIC

DEC Same as DECIMAL 52.36 DECIMAL NUMERIC

NUMBER

Same as NUMERIC 52.36 DECIMAL NUMERIC



Category


Hive DWS

INTEGER

Stores signedintegers. Integerpart: 10 digits;decimal part: 0digits

5236 INT INTEGER

INT Same as INTEGER 5236 INT INTEGER

BIGINT Stores signedintegers. Integerpart: 19 digits;decimal part: 0digits

5236 BIGINT BIGINT

TINYINT Stores signedintegers. Integerpart: 3 digits;decimal part: 0digits

100 TINYINT SMALLINT

SMALLINT

Stores signedintegers. Integerpart: 5 digits;decimal part: 0digits

9999 SMALLINT

SMALLINT

BYTE Similar to TINYINT.Integer part: 3digits; decimal part:0 digits


BINARY Stores fixed-lengthbinary data.

0x2A3B4058 BINARY(NULL)

BYTEA(NULL)

VARBINARY

Stores variable-length binary data.

0x2A3B4058 BINARY(NULL)

BYTEA(NULL)

FLOAT Stores floating-point numbers withbinary precision.

52.36 FLOAT FLOAT8

DOUBLE

Similar to FLOAT 52.36 DOUBLE FLOAT8

REAL Stores binaryfloating-pointnumbers.

52.3 FLOAT FLOAT4

DOUBLEPRECISION

Stores double-precision floating-point numbers.

52.3 DOUBLE FLOAT8



Category


Hive DWS

Bitstring

BIT Stores 1, 0, orNULL.

1, 0, orNULL

TINYINT(1 0NULL)

BOOLEAN(true falseNULL)

Timeanddate

DATE Stores informationabout the year,month, and day.

'1999-10-01','1999/10/01', or'1999.10.01'

DATE TIMESTAMP

TIME Stores informationabout the hour,minute, andsecond.

'09:10:21' or'9:10:21'

Notsupported (string)

TIME

TIMESTAMP

Stores informationabout the year,month, day, hour,minute, andsecond.

2002-12-1209:10:21','2002-12-129:10:21''2002/12/1209:10:21' or'2002.12.1209:10:21'

TIMESTAMP

TIMESTAMP

TIMEWITHTIMEZONE

Stores a TIME valuewith a time zone.Add the time zoneinformation to theend of the TIMEtype.

'09:10:21+8:00','09:10:21+8:00' or'9:10:21+8:00'


TIME WITHTIME ZONE

TIMESTAMPWITHTIMEZONE

Stores aTIMESTAMP valuewith a time zone.Add the time zoneinformation to theend of theTIMESTAMP type.

2002-12-1209:10:21+8:00','2002-12-129:10:21+8:00''2002/12/1209:10:21+8:00' or'2002.12.1209:10:21+8:00'

TIMESTAMP

TIMESTAMPWITH TIMEZONE



Category


Hive DWS

TIMESTAMPWITHLOCALTIMEZONE

Stores theTIMESTAMP valueof a local timezone. The standardtime zone type(TIMESTAMP WITHTIME ZONE) canbe converted to thelocal time zonetype.

2002-12-1209:10:21+8:00','2002-12-129:10:21+8:00''2002/12/1209:10:21+8:00' or'2002.12.1209:10:21+8:00'


Notsupported(TEXT)

DATETIMEWITHTIMEZONE

Same asTIMESTAMP WITHTIME ZONE

2002-12-1209:10:21+8:00','2002-12-129:10:21+8:00''2002/12/1209:10:21+8:00' or'2002.12.1209:10:21+8:00'

TIMESTAMP

TIMESTAMPWITH TIMEZONE

INTERVAL YEAR

Interval of years.The leadingprecision specifiesthe range of years.

INTERVAL'0015' YEAR


Notsupported(VARCHAR)

INTERVAL YEARTOMONTH

Interval of monthsand years. Theleading precisionspecifies the rangeof years.

INTERVAL'0015-08'YEAR TOMONTH



INTERVALMONTH

Interval of months.The leadingprecision specifiesthe range ofmonths.

INTERVAL'0015'MONTH



INTERVAL DAY

Interval of days.The leadingprecision specifiesthe range of days.

INTERVAL'150' DAY





Category


Hive DWS

INTERVAL DAYTOHOUR

Interval of hoursand days. Theleading precisionspecifies the rangeof days.

INTERVAL '923' DAY TOHOUR



INTERVAL DAYTOMINUTE

Interval of minutes,hours, and days.The leadingprecision specifiesthe range of days.

INTERVAL'09 23:12'DAY TOMINUTE



INTERVAL DAYTOSECOND

Interval of seconds,minutes, hours, anddays. The leadingprecision specifiesthe range of days.

INTERVAL'0923:12:01.1'DAY TOSECOND



INTERVALHOUR

Interval of hours.The leadingprecision specifiesthe range of hours.

INTERVAL'150' HOUR



INTERVALHOURTOMINUTE

Interval of minutesand hours. Theleading precisionspecifies the rangeof hours.

INTERVAL'23:12'HOUR TOMINUTE



INTERVALHOURTOSECOND

Interval of seconds,minutes, and hours.The leadingprecision specifiesthe range of hours.

INTERVAL'23:12:01.1'HOUR TOSECOND



INTERVALMINUTE

Interval of minutes.The leadingprecision specifiesthe range ofminutes.

INTERVAL'150'MINUTE



INTERVALMINUTETOSECOND

Interval of secondsand minutes. Theleading precisionspecifies the rangeof minutes.

INTERVAL'12:01.1'MINUTE TOSECOND





Category


Hive DWS

INTERVALSECOND

Interval of seconds.The leadingprecision specifiesthe range of theinteger part of thesecond.

INTERVAL'51.1'SECOND



Multimedia

IMAGE IMAGE specifiesthe image type inthe multimediainformation.An image consistsof a pixel latticewith a maximumlength of 2 GB. Inaddition to storingimage data, otherbinary data canalso be stored.

0x2A3B4058(binarydata)

Notsupported

Notsupported

LONGVARBINARY

Same as IMAGE 0x2A3B4059(binarydata)

Notsupported

Notsupported

TEXT Stores the longstring type.The maximumlength of a string is2 GB.

0x5236(binarydata)

Notsupported

Notsupported

LONGVARCHAR

Similar to TEXT 0x5236(binarydata)

Notsupported

Notsupported

BLOB Stores variable-length binary largeobjects with amaximum lengthof 2 GB.

0x5236(binarydata)

Notsupported

Notsupported

CLOB Stores variable-length binary largeobjects with amaximum lengthof 2 GB.

0x5236(binarydata)

Notsupported

Notsupported



Category


Hive DWS

BFILE Specified the binaryfiles stored in theoperating systems.Files are stored inthe operatingsystems instead ofthe databases.They can be readonly.

- Notsupported

Notsupported

Data Types Supported in Open-Source MySQL Database MigrationWhen the source end is an open-source MySQL database and the destination endis a Hive or DWS database, the following data types are supported:

Table 3-4 Data types supported by the open-source MySQL database functioningas the source end

Category


Hive DWS

Characterstring

CHAR(M)

A fixed-length stringof 1 to 255characters, forexample, CHAR(5).The length limit isnot mandatory. It isset to 1 by default.

'a' or'aaaaa'

CHAR CHAR

VARCHAR(M)

A variable-lengthstring consists of 1 to255 characters (morethan 255 charactersfor MySQL of a laterversion). Example:VARCHAR(25).When creating a fieldof the VARCHARtype, you must definethe length.

'a' or'aaaaa'

VARCHAR VARCHAR



Value DECIMAL(M,D)

Uncompressedfloating-pointnumbers cannot beunsigned. Inunpacking decimals,each decimalcorresponds to abyte.Defining the numberof display lengths(M) and decimals (D)is required. NUMERICis the synonym ofDECIMAL.

52.36 DECIMAL When D is 0,itcorrespondsto BIGINT.When D isnot 0, itcorrespondstoNUMBERIC.

NUMBERIC

Same as DECIMAL - DECIMAL NUMBERIC

INTEGER

An integer of normalsize that can besigned. If the value issigned, it ranges from-2147483648 to2147483647.If the value isunsigned, the valueranges from 0 to4294967295. Up to11-bit width can bespecified.

5236 INT INTEGER

INTEGERUNSIGNED

Unsigned form ofINTEGER

- BIGINT INTEGER

INT Same as INTEGER 5236 INT INTEGER

INTUNSIGNED

Same as INTEGERUNSIGNED

- BIGINT INTEGER



BIGINT

A large integer thatcan be signed. If thevalue is signed, itranges from-9223372036854775808 to9223372036854775807. If the value isunsigned, the valueranges from 0 to18446744073709551615. Up to 20-bitwidth can bespecified.

5236 BIGINT BIGINT

BIGINTUNSIGNED

Unsigned form ofBIGINT

- BIGINT BIGINT

MEDIUMINT

A medium-sizedinteger that can besigned. If the value issigned, it ranges from-8388608 to8388607.If the value isunsigned, it rangesfrom 0 to 16777215,and you can specify amaximum of 9-bitwidth.

-128, 127 INT INTEGER

MEDIUMINTUNSIGNED

Unsigned form ofMEDIUMINT

- BIGINT INTEGER

TINYINT

A very small integerthat can be signed. Ifsigned, the valueranges from -128 to127.If unsigned, the valueranges from 0 to 255,and you can specify amaximum of 4-bitwidth.




TINYINTUNSIGNED

Unsigned form ofTINYINT

- TINYINT SMALLINT

BOOL The bool of MySQL istinyint(1).

-128, 127 SMALLINT

BYTEA

SMALLINT

A small integer thatcan be signed. If thevalue is signed, itranges from -32768to 32767.If unsigned, the valueranges from 0 to65535, and you canspecify a maximumof 5-bit width.

9999 SMALLINT

SMALLINT

SMALLINTUNSIGNED

Unsigned form ofSMALLINT

- INT SMALLINT

REAL Same as DOUBLE - DOUBLE -

FLOAT(M,D)

Unsigned floating-point numberscannot be used. Thedisplay length (M)and number ofdecimal places (D)can be specified. Thisis not mandatory,and the default valueis 10,2. In theprecedinginformation, 2indicates the numberof decimal places and10 indicates the totalnumber of digits(including decimalplaces). The decimalprecision can reach24 floating points.

52.36 FLOAT FLOAT4



DOUBLE(M,D)

Unsigned double-precision floating-point numberscannot be used. Thedisplay length (M)and number ofdecimal places (D)can be specified. Thisis not mandatory.The default value is16,4, where 4 is thenumber of decimalplaces. The decimalprecision can reach53-digit. REAL is asynonym of DOUBLE.

52.36 DOUBLE FLOAT8

DOUBLEPRECISION

Similar to DOUBLE 52.3 DOUBLE FLOAT8

Bit BIT(M)

Stored bit type value.BIT(M) can store upto M bits of values,and M ranges from 1to 64.

B'1111100'B'1100'

TINYINT BYTEA

Timeanddate

DATE The value is in theYYYY-MM-DD formatand ranges from1000-01-01 to9999-12-31. Forexample, December30, 1973 will bestored as1973-12-30.

1999-10-01 DATE TIMESTAMP

TIME Stores informationabout the hour,minute, and second.

'09:10:21'or '9:10:21'


TIME



DATETIME

The date and timeare in the YYYY-MM-DD HH:MM:SSformat and rangefrom 1000-01-0100:00:00 to9999-12-31 23:59:59.For example, 3:30p.m. on December30, 1973 will bestored as 1973-12-3015:30:00.

'1973-12-3015:30:00'

TIMESTAMP

TIMESTAMP

TIMESTAMP

Timestamp type.Timestamp betweenmidnight on January1, 1970 and a timepoint in 2037. Similarto the DATETIMEformat(YYYYMMDDHHMMSS), except that nohyphen is required.For example, 3:30p.m. December 30,1973 will be storedas 19731230153000.

19731230153000

TIMESTAMP

TIMESTAMP

YEAR(M)

The year is stored in2-digit or 4-digitnumber format. If thelength is specified as2 (for example,YEAR(2)), the yearranges from 1970 to2069 (70 to 69). Ifthe length is specifiedas 4, the year rangesfrom 1901 to 2155.The default length is4.

2000 Notsupported (string)

Notsupported

Multimedia(binary)

BINARY(M)

The number of bytesis M. The length of avariable-lengthbinary string rangesfrom 0 to M. M is thevalue length plus 1.

0x2A3B4058 (binarydata)

Notsupported

BYTEA

VARBINARY(M)

The number of bytesis M. A fixed binarystring with a lengthof 0 to M.

0x2A3B4059 (binarydata)

Notsupported

BYTEA



TEXT The maximum lengthof the field is 65535characters. TEXT is a"binary large object"and is used to storelarge binary data,such as images orother types of files.

0x5236(binarydata)

Notsupported

Notsupported

TINYTEXT

A binary string of 0to 255 bytes in shorttext

- - Notsupported

MEDIUMTEXT

A binary string of 0to 167772154 bytesin medium-lengthtext

- - Notsupported

LONGTEXT

A binary string of 0to 4294967295 bytesin large-length text

- - Notsupported

BLOB The maximum lengthof the field is 65535characters. BLOB is a"binary large object"and is used to storelarge binary data,such as images orother types of files.BLOB is case-sensitive.

0x5236(binarydata)

Notsupported

BYTEA

TINYBLOB

A binary string of 0to 255 bytes in shorttext

- - BYTEA

MEDIUMBLOB

A binary string of 0to 167772154 bytesin medium-lengthtext

- - BYTEA

LONGBLOB

A binary string of 0to 4294967295 bytesin large-length text

0x5236(binarydata)

Notsupported

BYTEA



Specialtype

SET SET is a string objectthat can have no ormultiple values. Thevalues come from theallowed column ofvalues specified whenthe table is created.When specifying theSET column valuesthat contain multipleSET members,separate themembers withcommas (,). The SETmember value cannotcontain commas (,).

- - Notsupported

JSON - - Notsupported

Notsupported(TEXT)

ENUM

When an ENUM isdefined, a list of itsvalues is created,which are the itemsthat must be used forselection (or NULL).For example, if youwant a field tocontain "A", "B", or"C", you can definean ENUM ("A", "B",or "C"). Only thesevalues (or NULL) canbe used to fill in thefield.

- Notsupported

Notsupported

Data Types Supported in Oracle Database MigrationWhen the source end is an Oracle database and the destination end is a Hive orDWS database, the following data sources are supported:

Table 3-5 Data types supported by the Oracle database

Category

Type Description Hive DWS

Characterstring

char Fixed-length character string,which is padded with spaces toreach the maximum length.

CHAR CHAR



Category


nchar Fixed-length character stringcontains data in Unicodeformat.

CHAR CHAR

varchar2

Synonym of VARCHAR. It is avariable-length string, unlikethe CHAR type, which does notpad the field or variable toreach its maximum length withspaces.

VARCHAR VARCHAR

nvarchar2

Variable-length character stringcontains data in Unicodeformat.

VARCHAR VARCHAR

Value number Stores numbers with a precisionof up to 38 digits.

DECIMAL NUMERIC

binary_float

2-bit single-precision floatingpoint number

FLOAT FLOAT8

binary_double

64-bit double-precision floatingpoint number

DOUBLE FLOAT8

long A maximum of 2 GB characterdata can be stored.

Notsupported

Notsupported

Timeanddate

date 7-byte date/time data type,including seven attributes:century, year in the century,month, day in the month, hour,minute, and second.

DATE TIMESTAMP

timestamp

7-byte or 11-byte fixed-widthdate/time data type thatcontains decimals (seconds)

TIMESTAMP

TIMESTAMP

timestampwithtimezone

3-byte timestamp, whichsupports the time zone.

TIMESTAMP

TIME WITHTIME ZONE

timestampwithlocaltimezone

7-byte or 11-byte fixed-widthdate/time data type. Time zoneconversion occurs when data isinserted or read.

TIMESTAMP

Notsupported(TEXT)

intervalyear tomonth

5-byte fixed-width data type,which is used to store a timesegment.

Notsupported

Notsupported(TEXT)



Category


intervalday tosecond

11-byte fixed-width data type,which is used to store a timesegment. The time segment isstored in days/hours/minutes/seconds. The value can alsocontain nine decimal places(seconds).

Notsupported

Notsupported(TEXT)

Multimedia(binary)

raw A variable-length binary datatype. Character set conversionis not performed for datastored in this data type.

Notsupported

Notsupported

longraw

Stores up to 2 GB binaryinformation.

Notsupported

Notsupported

blob A maximum of 4 GB data canbe stored.

Notsupported

Notsupported

clob In Oracle 10g and laterversions, a maximum of (4 GB)x (database block size) bytes ofdata can be stored. CLOBcontains the information forwhich character set conversionis to be performed. This datatype is ideal for storing plaintext information.

Notsupported

Notsupported

nclob This type can store a maximumof 4 GB data. When thecharacter set is converted, thistype is affected.

Notsupported

Notsupported

bfile An Oracle directory object anda file name can be stored in thedatabase column, and the filecan be read through the Oracledirectory object and file name.

Notsupported

Notsupported

Others rowid In fact, it is the address of arow in the database table. It is10 bytes long.

Notsupported

Notsupported

urowid It is a common row ID anddoes not have a fixed rowidtable.

Notsupported

Notsupported



Data Types Supported in SQL Server Database MigrationWhen the source end is a SQL Server database and the destination end is a Hive,Oracle or DWS database, the following data sources are supported:

Table 3-6 Data types supported by the SQL Server database functioning as thesource end

Category Type

Description Hive DWS Oracle

String datatype

char

Fixed-length character string, whichis padded with spaces to reach themaximum length.

CHAR CHAR CHAR

nchar

Fixed-length character stringcontains data in Unicode format.

CHAR CHAR CHAR

varchar

A variable-length string consists of1 to 255 characters (more than 255characters for MySQL of a laterversion). Example: VARCHAR(25).When creating a field of theVARCHAR type, you must define thelength.

VARCHAR

VARCHAR

VARCHAR

nvarchar

Stores variable-length Unicodecharacter data, similar to varchar.

VARCHAR

VARCHAR

VARCHAR

Numericdata type

int int is stored in four bytes, whereone binary bit represents a sign bit,and the other 31 binary bitsrepresent a length and a size, andmay represent all integers rangingfrom –231 to 231 – 1.

INT INTEGER

INT

bigint

bigint is stored in eight bytes, whereone binary bit represents a sign bit,and the other 63 binary bitsrepresent a length and a size, andmay represent all integers rangingfrom –263 to 263 – 1.

BIGINT

BIGINT

NUMBER

smallint

Data of the smallint type occupiestwo bytes of storage space. Onebinary bit indicates a positive ornegative sign of an integer value,and the other 15 binary bitsindicate a length and a size, andmay represent all integers rangingfrom –215 to 215.

SMALLINT

SMALLINT

NUMBER



Category Type


tinyint

Tinyint data occupies one byte ofstorage space and can represent allintegers ranging from 0 to 255.

TINYINT

TINYINT

NUMBER

real

The value can be a positive ornegative decimal number.

DOUBLE

FLOAT4

NUMBER

float

The number of digits (in scientificnotation) of the mantissa of a floatvalue, which determines theprecision and storage size

FLOAT FLOAT8

binary_float

decimal

Numeric data type with fixedprecision and scale

DECIMAL

NUMERIC

NUMBER

numeric

Stores zero, positive, and negativefixed point numbers.

DECIMAL

NUMERIC

NUMBER

Date andtime datatype

date

Stores date data represented bystrings.

DATE TIMESTAMP

DATE

time

Time of a day, which is recorded inthe form of a character string.

Notsupported(string)

TIME Notsupported

datetime

Stores time and date data. TIMESTAMP

TIMESTAMP

Notsupported

datetime2

Extended type of datetime, whichhas a larger data range. By default,the minimum precision is thehighest, and the user-definedprecision is optional.

TIMESTAMP

TIMESTAMP

Notsupported

smalldatetime

The smalldatetime type is similar tothe datetime type. The difference isthat the smalldatetime type storesdata from January 1, 1900 to June6, 2079. When the date and timeprecision is low, the smalldatetimetype can be used. Data of this typeoccupies 4-byte storage space.

TIMESTAMP

TIMESTAMP

Notsupported



Category Type


timestamp

Timestamp data type TIMESTAMP

TIMESTAMP

TIMESTAMP

datetimeoffset

A time that uses the 24-hour clockand combined with date and thetime zone.


TIMESTAMP

Notsupported

Multimedia datatypes(binary)

text

Stores text data. Notsupported(string)


Notsupported

netxt

The function of this type is thesame as that of the text type. It isnon-Unicode data with variablelength.



Notsupported

image

Variable-length binary data used tostore pictures, catalog pictures, orpaintings.



Notsupported

binary

Binary data with a fixed length of nbytes, where n ranges from 1 to8,000.



Notsupported

varbinary

Variable-length binary data Notsupported(string)


Notsupported

Currencydata type

money

Stores currency values. Notsupported(string)


Notsupported

smallmoney

Similar to the money type, acurrency symbol is prefixed to theinput data. For example, thecurrency symbol of CNY is ¥.



Notsupported



Category Type


Data type bit Bit data type. The value is 0 or 1.The length is 1 byte. A bit value isoften used as a logical value todetermine whether it is true(1) orfalse(0). If a non-zero value isentered, the system replaces it with1.

Notsupported

Notsupported

Notsupported

Other datatypes

rowversion

Each piece of data has a counter.The value of the counter increaseswhen an insert or update operationis performed on a table thatcontains the rowversion column inthe database.

Notsupported

Notsupported

Notsupported

uniqueidentifier

A 16-byte globally unique identifier(GUID) is a unique numbergenerated by the SQL Server basedon the network adapter addressand host CPU clock. Each GUID is ahexadecimal number ranging from0 to 9 or a to f.

Notsupported

Notsupported

Notsupported

cursor

Cursor data type Notsupported

Notsupported

Notsupported

sql_variant

Stores any valid SQL Server dataexcept the text, image, andtimestamp data, which facilitatesthe development of the SQL Server.

Notsupported

Notsupported

Notsupported

table

Stores the result set after a table orview is processed.

Notsupported

Notsupported

Notsupported

xml

Data type of the XML data. XMLinstances can be stored in columnsor variables of the XML type. Thestored XML instance size cannotexceed 2 GB.

Notsupported

Notsupported

Notsupported



4 Application Scenarios

Migrating Local Data to the Cloud

Local data is stored in the IDC that you have built or rent, or on the third-partycloud, including data stored in relational databases, NoSQL databases, OLAPdatabases, and file systems.

In this scenario, if you want to use the cloud computing and storage resources,you must migrate local data to the cloud in advance. Ensure that the localnetwork can communicate with the cloud network.

Figure 4-1 Migrating local data to the public cloud

Migrating Data Between Cloud Services

In this scenario, you are allowed to exchange data between the following cloudservices:● OBS● Relational Database Service (RDS)● MapReduce Service (MRS)● DWS● DDS● DCS● CSS● DIS

Cloud Data MigrationService Overview 4 Application Scenarios


● CloudTable● DLI● SFS● Databases or file systems deployed on the ECSs

Migrating Cloud Data to On-Premises EnvironmentsA local environment is a data storage system in the IDC that you have built orrent, or on the third-party cloud, including relational databases and file systems.

In this scenario, after data is processed using the cloud computing and storageresources, the processed data can be returned to on-premises service systems,specifically relational databases and file systems. Ensure that the local networkcan communicate with the cloud network.

Figure 4-2 Migrating public cloud data to on-premises environments

Cloud Data MigrationService Overview 4 Application Scenarios


5 Basic Concepts

CDM Cluster

A CDM cluster is a CDM instance. It consists of one or more VMs. You can createmultiple CDM clusters for different purposes. For example, you can create a CDMcluster for the financial department and the procurement department respectivelyto isolate data access permissions.

Local Environment

A local environment is a data storage system in the IDC that you have built orrent, or on the third-party cloud, including relational databases and file systems.

Local Data

Local data is stored in the IDC that you have built or rent, or on the third-partycloud, including data stored in relational databases, NoSQL databases, OLAPdatabases, and file systems.

Connector

A connector is a built-in object template used for connecting to a data source.Currently, CDM uses connectors to connect to OBS, MRS, and databases. Newconnectors can be added to CDM as well.

Link

A link is an object set up based on a connector and used to connect to a specificdata source.

To create a link, you must specify the link name, connector, data source address,and authentication information. For example, to connect to a MySQL database,you must set the host IP address, port number, username, and password.

After a link is set up, it can be used by multiple jobs as either a source or adestination link.

Cloud Data MigrationService Overview 5 Basic Concepts


JobA job is a data migration task that you have created to migrate data from aspecific data source to another. To create a job, you must specify a source link,destination link, and data mapping rules.

Source Job ConfigurationDuring job creation, the source link specifies the data source from which data isextracted. The job parameters of different source links vary. For example, the tableor directory from which data is exported is specified in the job configuration at thesource end.

Destination Job ConfigurationDuring job creation, the destination link specifies the data source to which data isloaded. The job parameters of different destination links vary. For example, thetable or directory to which data is imported is specified in the job configuration ofthe destination end.

Field MappingDuring job creation, especially jobs of migrating data between heterogeneous datasources, you must configure the mapping between the source and destinationdata sources, such as field mapping and field type mapping.

AccountThe account registered with the cloud owns your cloud resources and has fullaccess permissions for the resources. You can use the account to reset userpasswords and assign permissions. Your account receives and pays all billsgenerated by your IAM users' use of resources. To log in to the managementconsole using an account, choose Account Login.

IAM UserIAM users are created by an account to use cloud services. Each IAM user has theirown password and access key to access cloud services using the console or APIs.The users manage cloud resources for the account based on assigned permissions.IAM users do not own resources or make payments. It is the account that controlsuser permissions and pays the bills. To log in to the management console as anIAM user, choose IAM User Login.



Figure 5-1 Relationship between an account and its IAM users

User GroupUser groups facilitate centralized user management and streamlined permissionsmanagement. IAM users must be added to a user group to obtain the permissionsrequired for accessing specified resources or cloud services in the account. A usercan be added to multiple groups, which allows them to inherit differentpermissions.

The default user group admin has all of the permissions required to use all of thecloud resources. Users in this group can perform operations on all the resources,including but not limited to creating user groups and users, assigning permissions,and managing resources.

Policy and AuthorizationA policy is a set of permissions defined in JSON format. It defines whichoperations on which cloud resources are allowed. There are two types of policies:system policy and custom policy.

● System policies are pre-defined by IAM and cannot be modified.● If system policies do not meet your requirements, you can create custom

policies for fine-grained access control.

Authorization is the process of granting required permissions for a user to performa task. After a system or custom policy is assigned to a user group, users in thegroup inherit the permissions defined by the policy to manage resources.

For example, the content of the CDM Administrator policy defining allpermissions of CDM is as follows:{ "Version": "1.1",



"Statement": [ { "Action": [ "cdm:*:*", "ecs:*:*", "vpc:*:*", "evs:*:*", "bss:*:*", "CTS:*:*", "eps:*:*", "obs:*:*", "CES:*:*" ], "Effect": "Allow" } ]}

Project

A project corresponds to a service region. Default projects are defined to groupand physically isolate resources (including computing, storage, and networkresources) across regions.

● Users can be granted permissions in a default project to access all resources inthe region associated with the project.

● If you need more refined access control, you can create subprojects under adefault project and create resources in subprojects. Then you can assignrequired permissions for users to access only the resources in specificsubprojects.

Figure 5-2 Project isolation model

Identity Credentials

Identity credentials are used for authentication when you or your IAM users accesscloud services through the console or APIs. Identity credentials include thepassword and access keys, which can be managed in IAM.

● Password: A common identity credential for logging in to the managementconsole or calling cloud service APIs.

● Access key: An access key ID/secret access key (AK/SK) pair, which is used onlyfor calling cloud service APIs. Each access key provides a signature for



cryptographic authentication to ensure that access requests are secret,complete, and correct.



6 CDM Migration Principles

Migration PrinciplesWhen a tenant uses CDM, the CDM system provisions a fully-managed CDMinstance in the tenant's VPC. The instance allows only console and RESTful APIaccess. Therefore the tenant cannot access the instance through other interfaces(such as SSH). This ensures data isolation between CDM tenants, prevents dataleakage, and ensures transmission security during data migration betweendifferent cloud services in a VPC. Tenants can also use the VPN to migrate datafrom the on-premises data center to cloud services to ensure migration security.

CDM works in push-pull mode. CDM pulls data from the migration source andpushes the data to the migration destination. Data access operations are initiatedby CDM. SSL will be used if the data source (such as RDS) supports it. During themigration, the usernames and passwords of the migration source and destinationare required. Such information is stored in the database of the CDM instance.Protecting such information is critical to ensure CDM security.

Figure 6-1 Migration principles

Cloud Data MigrationService Overview 6 CDM Migration Principles


Security Boundary and Risk Mitigation

Figure 6-2 Risk mitigation

As shown in Figure 6-2, CDM may have the following threats:

1. Threats from the Internet: Malicious tenants may attack CDM through theCDM console.

2. Threats from the data center: Malicious CDM administrators obtain tenants'data source access information (usernames and passwords).

3. Threats from malicious tenants: Malicious tenants steal data from othertenants.

4. Data exposure to the public network: Data is exposed when it is migratedfrom the public network.

CDM offers the following mechanisms to prevent potential security risks:

1. Threats from the Internet: Tenants cannot log in to the CDM console throughthe public network. CDM provides a two-layer security mechanism.

a. On the one hand, the cloud console framework requires userauthentication when tenants access management consoles of cloudservices.

b. On the other hand, Web Application Firewall (WAF) filters requests fromall consoles and stops request attack code or content.

2. Threats from the data center: Tenants must provide the usernames andpasswords of the migration source and destination to complete datamigration. To prevent the CDM administrators from obtaining suchinformation and attacking important data sources of tenants, CDM provides athree-level protection mechanism.

a. CDM stores passwords encrypted by AES-256 in the database of theinstance to ensure tenant isolation. The database is run by user Ruby andlistens to only 127.0.0.1. Therefore, tenants cannot remotely access thedatabase.

b. After the instance is provisioned, CDM changes the passwords of usersroot and Ruby to random passwords and does not store them in anyplace. This prevents the CDM administrators from accessing tenants'instances and databases containing password information.



c. CDM instances work in push-pull mode. Therefore, the instances do nothave any listening port enabled in the VPC, and tenants cannot accessthe local database or operating system from the VPC.

3. Threats from malicious tenants: CDM runs instances on independent VMs, sothat tenants' instances are completely isolated and secure. Malicious tenantscannot access instances of other tenants.

4. Data exposure to the public network: In push-pull mode, even if elastic IPaddresses (EIPs) are bound to the CDM clusters, no port is enabled for theEIPs. In this way, attackers cannot access and attack CDM using the EIPs.However, when data is migrated from the public network, tenants' datasources are exposed to the public network and threatened by third-partyattacks. Therefore, tenants are advised to use ACLs or firewalls on the datasource server for security. In this case, for example, only the access requestsfrom the EIPs bound to the CDM clusters are allowed.



7 Related Services

IAM

Your registered cloud account has full access to its resources and cloud services. Ifyou need to assign different permissions to employees in your enterprise to accessyour CDM resources, Identity and Access Management (IAM) is a good choice forfine-grained permissions management. IAM provides identity authentication,permissions management, and access control, helping you secure access to yourcloud resources.

VPC

CDM clusters are created in the subnets of a Virtual Private Cloud (VPC). VPCsprovide a secure, isolated, and logical network environment for CDM clusters.

MRS

CDM supports data import and export using MRS.

OBS

CDM supports data import and export using OBS, which also stores backup filesand logs of CDM clusters.

Cloud Eye

CDM uses Cloud Eye to monitor cluster performance metrics, delivering statusinformation in a concise and efficient manner, as shown in Table 7-1.

Table 7-1 CDM performance metrics

Metric Description ValueRange

MonitoredObject

Bytes In Measures the network inboundrate of the monitored object.Unit: byte/s

≥ 0 bytes/s Cloud DataMigration

Cloud Data MigrationService Overview 7 Related Services


Metric Description ValueRange

MonitoredObject

Bytes Out Measures the network outboundrate of the monitored object.Unit: byte/s

≥ 0 bytes/s Cloud DataMigration

CPU Usage Measures the CPU usage of themonitored object.Unit: %

0% to100%

Cloud DataMigration

MemoryUsage

Measures the memory usage ofthe monitored object.Unit: %

0% to100%

Cloud DataMigration

CTSCDM uses Cloud Trace Service (CTS) to record operations for later query, audit,and backtrack operations. Table 7-2 displays the recorded CDM operations.

Table 7-2 CDM operations recorded by CTS

Operation Resource Type Trace Name

Creating a cluster cluster createCluster

Deleting a cluster cluster deleteCluster

Modifying clusterconfigurations

cluster modifyCluster

Starting a cluster cluster startCluster

Stopping a cluster cluster stopCluster

Restarting a cluster cluster restartCluster

Importing a job cluster clusterImportJob

Binding an EIP cluster bindEip

Unbinding an EIP cluster unbindEip

Creating a link link createLink

Modifying a link link modifyLink

Deleting a link link deleteLink

Creating a job job createJob

Modifying a job job modifyJob

Deleting a job job deleteJob



Operation Resource Type Trace Name

Starting a job job startJob

Stopping a job job stopJob

DWSCDM allows you to import data to and export data from DWS.

RDSCDM allows you to import data to and export data from RDS, including RDS forMySQL, RDS for PostgreSQL, and RDS for SQL Server.

DDSCDM allows you to export data from DDS, but it does not allow you to importdata to DDS.

DCSCDM allows you to import data to DCS, but it does not allow you to export datafrom DDS.

CSSCDM allows you to import data to and export data from CSS.

DISCDM allows you to import data to DIS, and export data from DIS to CSS, ApacheKafka, or DMS Kafka.

CloudTableCDM allows you to import data to and export data from CloudTable.

DLICDM allows you to import data to DLI, but it does not allow you to export datafrom DLI.

SFSCDM allows you to import data to and export data from SFS.

Distributed Message Service (DMS)CDM allows you to import data to DMS, and export data from DMS to CSS,Apache Kafka, DIS, or DMS Kafka.



Data Lake Factory (DLF)CDM can be orchestrated and scheduled as a node task of DLF.



8 Constraints

Due to various factors such as technology and cost, CDM has constraints on datamigration.

CDM System Constraints1. You cannot modify the flavor of an existing cluster. If you require a higher

flavor, create a cluster with your desired flavor.2. CDM does not support the function of controlling the data migration speed.

Therefore, do not perform data migration during peak hours.3. Theoretically, the migration capability of a CDM cluster is half of the baseline

bandwidth. For example, if the baseline bandwidth of the cdm.xlarge flavor is4 Gbit/s, the data transfer speed is 2 Gbit/s (4/2 = 2 Gbit/s). Multiple CDMinstances can be used if you want faster data transfer.The preceding data volume is the theoretical value. The actual data volume isrestricted by the data source type, read and write performance of the sourceand destination data sources, and bandwidth. The actual data volume canreach about 8 TB per day (large file migration to OBS). It is recommendedthat you test the speed with a small amount of data before migration.

4. CDM supports incremental file migration (by skipping repeated files), butdoes not support resumable transfer.For example, if three files are to be migrated and the second file fails to bemigrated due to the network fault. When the migration task is started again,the first file is skipped. The second file, however, cannot be migrated from thepoint where the fault occurs, but can only be migrated again.

5. During file migration, a single task supports millions of files. If there are toomany files in the directory to be migrated, you are advised to split the filesinto different directories and create multiple tasks.

6. The number of tasks executed by a single CDM instance at a time is 30(cdm.large), 20 (cdm.medium), or 10 (cdm.small). The number of queuedjobs (in the pending state) to be executed is 10,000, 5,000, and 2,000respectively.In database migration, a job is equivalent to migrating a table. In filemigration, multiple files can be migrated in a job.

7. You can export links and jobs configured on CDM to a local directory. Toensure password security, CDM does not export the link password of the

Cloud Data MigrationService Overview 8 Constraints


corresponding data source. Therefore, before importing job configurations toCDM, you need to manually input the password in the exported JSON file.

8. The cluster cannot automatically upgrade to a new version. You need to usethe job export and import functions to upgrade the cluster to the new version.

9. CDM does not automatically back up users' job configurations. You need toexport and back up configuration data using the export function.

10. If VPC peering connection is configured, the peer VPC subnet may overlapwith the CDM management network. As a result, data sources in the peerVPC cannot be accessed. You are advised to use the public network for cross-VPC data migration, or contact the administrator to add specific routes to theVPC peering connection in the CDM background.

General Constraints on Database Migration1. CDM is mainly used for batch migration. It supports only limited incremental

migration but does not support real-time incremental migration. You areadvised to use Data Replication Service (DRS) to migrate the incrementaldata of the database to RDS.

2. The entire DB migration of CDM supports only data table migration but notmigration of database objects such as stored procedures, triggers, functions,and views. Views are migrated as tables.CDM applies only to scenarios where databases are migrated to the cloud at atime, including homogeneous and heterogeneous database migrations. CDMis not applicable to data synchronization, for example, disaster recovery andreal-time synchronization.

3. When CDM fails to migrate the entire database or data table, the data thathas been imported to the target table will not be rolled back automatically. Ifyou want to perform migration in transaction mode, configure the Import toStaging Table parameter to roll back data when the migration fails.In extreme cases, the created stage table or temporary table cannot beautomatically deleted. You need to manually clear the table (the table nameof the stage table ends with _cdm_stage), for example, cdmtet_cdm_stage).

4. If CDM needs to access data sources in the on-premises data center (forexample, the on-premises MySQL database), the data sources must supportInternet access and the CDM instances must be bound with elastic IPaddresses. In this case, the security practice is to configure the firewall orsecurity policies to allow only the EIPs of the CDM instances to access thelocal data sources.

5. Only common data types are supported, including character strings, digits,and dates. Object types are limited. If objects are too large, migration cannotbe performed.

6. Only the GBK and UTF-8 character sets are supported.7. A field name cannot contain & and %.

Constraints on MRS

Each CDM cluster supports data import and export of only one MRS data source.To import and export data of different MRS data sources, create multiple CDMclusters.



https://support.huaweicloud.com/intl/en-us/usermanual-cdm/cdm_01_0106.html


Permissions Configuration for Relational Database MigrationCommon minimum permissions required by relational database migration:

● MySQL: You need to have the read permission on theINFORMATION_SCHEMA database and data tables.

● Oracle: You need to have the resource role and have the select permissionson the data table in the tablespace.

● Dameng: You need to have the select any table permission in the schema.● DWS: You need to have the schema usage permission and the query

permission on the data tables.● SQL Server: You need to have the sysadmin permission.● PostgreSQL: You need to have the select permission on schema tables in the

database.

Constraints on FusionInsight HD and Apache HadoopIf the FusionInsight HD and Apache Hadoop data sources are deployed in the on-premises data center, CDM must access all nodes in the cluster for reading andwriting the Hadoop files. Therefore, the network access must be enabled for eachnode.

You are advised to use Direct Connect to improve the migration speed whileensuring network access.

Constraints on DWS and FusionInsight LibrA1. If the DWS primary key or table contains only one field, the field type must be

a common character string, value, or date. When data is migrated fromanother database to DWS, if automatic table creation is selected, the primarykey must be of the following types. If no primary key is set, at least one of thefollowing fields must be set. Otherwise, the table cannot be created and theCDM job fails.– INTEGER TYPES: TINYINT, SMALLINT, INT, BIGINT, NUMERIC/DECIMAL– CHARACTER TYPES: CHAR, BPCHAR, VARCHAR, VARCHAR2, NVARCHAR2,

TEXT– DATA/TIME TYPES: DATE, TIME, TIMETZ, TIMESTAMP, TIMESTAMPTZ,

INTERVAL, SMALLDATETIME2. In DWS, the character string '' is null. A null character string cannot be

inserted into a field with non-null constraints. This is inconsistent with theMySQL behavior. MySQL does not consider that '' is null. Migration fromMySQL to DWS may fail due to the preceding reason.

3. When the Gauss Data Service (GDS) mode is used to quickly import data toDWS, you need to configure a security group or firewall policy to allowDataNodes of DWS or FusionInsight LibrA to access port 25000 of the CDM IPaddress.

4. When data is imported to DWS in GDS mode, CDM automatically creates aforeign table for data import. The table name ends with a universally uniqueidentifier (UUID), for example, cdmtest_aecf3f8n0z73dsl72d0d1dk4lcir8cd.If a job fails, it will be automatically deleted. In extreme cases, you may needto manually delete it.



https://www.huaweicloud.com/intl/en-us/product/dc.html

Constraints on OBS1. During file migration, the system automatically transfers the files

concurrently. In this case, Concurrent Extractors in the task configuration isinvalid.

2. Resumable transfer is not supported. If CDM fails to transfer files, OBSfragments are generated. You need to clear fragments on the OBS console toprevent space occupation.

3. CDM does not support the versioning control function of OBS.

4. During incremental migration, the number of files or objects in the sourcedirectory of a single job depends on the CDM cluster flavor. A cdm.largecluster supports a maximum of 300,000 files; a cdm.medium cluster supportsa maximum of 200,000 files; and a cdm.small cluster supports a maximum of100,000 files.

If the number of files or objects in a single directory exceeds the upper limit,split the files or objects into multiple migration jobs based on subdirectories.

Constraints on Oracle

Real-time incremental data synchronization is not supported for Oracle databases.

Constraints on DCS and Redis1. Because DCS restricts the commands for obtaining keys, it cannot serve as the

migration source but can be the migration destination. The Redis service ofthe third-party cloud cannot serve as the migration source. However, theRedis set up in the on-premises data center or on the ECS can be themigration source and destination.

2. Only the hash and string data formats are supported.

Constraints on DDS and MongoDB

When you migrate data from MongoDB or DDS to a relational database, CDMreads the first row of the collection as an example of the field list. If the first rowof data does not contain all fields of the collection, you need to manually addfields.

Constraints on CSS and Elasticsearch1. CDM supports automatic creation of indexes and field types. The index and

field type names can contain only lowercase letters.

2. You cannot modify the field type under an index after it is created, but onlycreate another field.

If you need to modify the field type, you need to create an index or run theElasticsearch command on Kibana to delete the existing index and createanother index (the data is also deleted).

3. When the field type of the index created by CDM is date, the data formatmust be yyyy-MM-dd HH:mm:ss.SSS Z. For example, 2018-08-0808:08:08.888 +08:00.



During data migration to CSS, if the original data of the date field does notmeet the format requirements, you can use the expression conversionfunction of CDM to convert the data to the preceding format.

Constraints on DIS and Kafka1. The data in the message body is a record in CSV format that supports

multiple delimiters. Messages cannot be parsed in binary or other formats.2. If a job is set to run for a long time, the job will fail if the DIS system is

interrupted.

Constraints on CloudTable and HBase1. When you migrate data from CloudTable or HBase, CDM reads the first row

of the table as an example of the field list. If the first row of data does notcontain all fields of the table, you need to manually add fields.

2. Because HBase is schema-less, CDM cannot obtain the data types. If the datais stored in binary format, CDM cannot parse the data.

Constraints on HiveWhen Hive serves as the migration destination, if the storage format is TEXTFILE,delimiters must be explicitly specified in the statement for creating Hive tables.The following gives an example:

CREATE TABLE csv_tbl(smallint_value smallint,tinyint_value tinyint,int_value int,bigint_value bigint,float_value float,double_value double,decimal_value decimal(9, 7),timestmamp_value timestamp,date_value date,varchar_value varchar(100),string_value string,char_value char(20),boolean_value boolean,binary_value binary,varchar_null varchar(100),string_null string,char_null char(20),int_null int)ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'WITH SERDEPROPERTIES ("separatorChar" = "\t","quoteChar" = "'","escapeChar" = "\\")STORED AS TEXTFILE;




9 Quotas

CDM uses the following infrastructure resources:

● ECS● VPC● EIP● Simple Message Notification (SMN)● IAM

For details about how to view and modify the quota, see Quotas.

Cloud Data MigrationService Overview 9 Quotas


https://support.huaweicloud.com/intl/en-us/usermanual-iaas/en-us_topic_0040259342.html

10 Permissions Management

If you need to assign different permissions to employees in your enterprise toaccess your CDM resources, IAM is a good choice for fine-grained permissionsmanagement. IAM provides identity authentication, permissions management,and access control, helping you secure access to your cloud resources.

With IAM, you can use your cloud account to create IAM users, and assignpermissions to the users to control their access to specific resources. For example,some employees in your enterprise need to use CDM resources but should not beallowed to delete CDM clusters or perform any other high-risk operations. In thisscenario, you can create IAM users for the employees and grant them only thepermissions required for using CDM resources.

If your cloud account does not require individual IAM users for permissionsmanagement, skip this section.

IAM can be used free of charge. You pay only for the resources in your account.For more information about IAM, see IAM Service Overview.

CDM PermissionsBy default, new IAM users do not have permissions assigned. You need to add auser to one or more groups, and attach permissions policies or roles to thesegroups. Users inherit permissions from the groups to which they are added andcan perform specified operations on cloud services based on the permissions.

CDM is a project-level service deployed and accessed in specific physical regions.Therefore, CDM permissions are assigned to users in specific regions and only takeeffect for these regions. If you want the permissions to take effect for all regions,you need to assign the permissions to users in each region. When accessing CDM,the users need to switch to a region where they have been authorized to use theCDM service.

You can grant users permissions by using roles and policies.● Roles: A type of coarse-grained authorization mechanism that defines

permissions related to user responsibilities. This mechanism provides only alimited number of service-level roles for authorization. When using roles togrant permissions, you need to also assign other roles on which thepermissions depend to take effect. However, roles are not an ideal choice forfine-grained authorization and secure access control.

Cloud Data MigrationService Overview 10 Permissions Management


https://support.huaweicloud.com/intl/en-us/productdesc-iam/iam_01_0026.html

● Policies: A type of fine-grained authorization mechanism that definespermissions required to perform operations on specific cloud resources undercertain conditions. This mechanism allows for more flexible policy-basedauthorization, meeting requirements for secure access control. For example, aspecific user group is not allowed to delete a cluster. Only basic CDMoperations (such as creating and querying jobs) are allowed.

Table 1 lists all the system-defined roles and policies supported by CDM.

Table 10-1 System-defined roles and policies supported by CDM

Role/PolicyName

Description Type

CDMAdministrator

Permissions:● Administrator permissions for all

operations on CDM resources. Usersgranted these permissions must alsobe granted permissions of the TenantGuest and Server Administratorpolicies.

● Users granted permissions of the VPCAdministrator policy can create VPCsand subnets.

● Users granted permissions of theCloud Eye Administrator policy canview monitoring information of CDMclusters.

System role

CDM FullAccess Administrator permissions for CDM. Usersgranted these permissions can perform alloperations on CDM resources.

System-definedpolicy

CDMFullAccessExceptUpdateEIP

Users granted these permissions canperform all operations on CDM resourcesexcept binding and unbinding EIPs.


CDMCommonOperations

Users granted these permissions canoperate CDM jobs and links.


CDMReadOnlyAccess

Read-only permissions for CDM. Usersgranted these permissions can only viewCDM clusters, links, and jobs.


Table 10-2 lists the common operations supported by each system-defined policyor role of CDM. Select the policies or roles as required.



Table 10-2 Common operations supported by each system-defined policy or roleof CDM

Operation CDM FullAccess CDMFullAccessExceptUpdateEIP

CDMCommonOperations

CDMReadOnlyAccess

Creating clusters √ √ × ×

Binding orunbinding EIPs

√ × × ×

Querying thecluster list

√ √ √ √

Querying clusterdetails

√ √ √ √

Restartingclusters

√ √ × ×

Modifyingclusterconfigurations

√ √ × ×

Deleting clusters √ √ × ×

Creating links √ √ √ ×

Querying links √ √ √ √

Modifying links √ √ √ ×

Deleting links √ √ √ ×

Creating jobs √ √ √ ×

Querying jobs √ √ √ √

Modifying jobs √ √ √ ×

Starting jobs √ √ √ ×

Stopping jobs √ √ √ ×

Querying jobstatus

√ √ √ √

Querying jobexecutionhistory

√ √ √ √

Deleting jobs √ √ √ ×



Service Overview - support.huaweicloud.com

Documents

Transcript of Service Overview - support.huaweicloud.com