Distributed Database Middleware · 2020. 11. 26. · Distributed Database Middleware User Guide...

Distributed Database Middleware

User Guide

Issue 01

Date 2021-01-15

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2021. All rights reserved.

No part of this document may be reproduced or transmitted in any form or by any means without priorwritten consent of Huawei Technologies Co., Ltd. Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei andthe customer. All or part of the products, services and features described in this document may not bewithin the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,information, and recommendations in this document are provided "AS IS" without warranties, guaranteesor representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.

Issue 01 (2021-01-15) Copyright © Huawei Technologies Co., Ltd. i

Contents

1 Permissions Management..................................................................................................... 11.1 Creating a User and Granting Permissions.....................................................................................................................11.2 Creating a Custom Policy..................................................................................................................................................... 2

2 Instance Management............................................................................................................42.1 Buying a DDM Instance........................................................................................................................................................ 42.2 Changing Class of a DDM Instance.................................................................................................................................. 62.3 Scaling Out a DDM Instance............................................................................................................................................... 82.4 Scaling In a DDM Instance...................................................................................................................................................92.5 Changing Billing Mode of a DDM Instance................................................................................................................. 112.6 Renewing a DDM Instance................................................................................................................................................ 122.7 Restarting a DDM Instance............................................................................................................................................... 122.8 Unsubscribing from a DDM Instance............................................................................................................................. 132.9 Deleting a DDM Instance................................................................................................................................................... 142.10 Reloading Table Data........................................................................................................................................................ 152.11 Parameter Management.................................................................................................................................................. 162.12 Parameter Template Management.............................................................................................................................. 212.12.1 Creating a Parameter Template................................................................................................................................. 212.12.2 Selecting a Parameter Template................................................................................................................................222.13 Account Management...................................................................................................................................................... 232.13.1 Creating an Account...................................................................................................................................................... 232.13.2 Modifying an Account................................................................................................................................................... 252.13.3 Deleting an Account...................................................................................................................................................... 262.14 Read/Write Splitting.......................................................................................................................................................... 272.15 Task Center........................................................................................................................................................................... 29

3 Schema Management.......................................................................................................... 303.1 Creating a Schema............................................................................................................................................................... 303.2 Exporting Schema Information........................................................................................................................................ 313.3 Importing Schema Information....................................................................................................................................... 323.4 Deleting a Schema............................................................................................................................................................... 333.5 Configuring SQL Blacklist...................................................................................................................................................333.6 Scaling Out a Schema......................................................................................................................................................... 353.6.1 Rebalancing Shards...........................................................................................................................................................35

Distributed Database MiddlewareUser Guide Contents

Issue 01 (2021-01-15) Copyright © Huawei Technologies Co., Ltd. ii

3.6.2 Doubling Shards and Resharding Data Records..................................................................................................... 37

4 Data Management................................................................................................................ 394.1 Data Migration...................................................................................................................................................................... 394.1.1 Overview...............................................................................................................................................................................394.1.2 Migration Evaluation........................................................................................................................................................ 404.1.3 Scenario 1: Migrating Data from HUAWEI CLOUD RDS to DDM.................................................................... 424.1.4 Scenario 2: Migrating Data from On-Premises RDS MySQL Databases to DDM....................................... 514.1.5 Scenario 3: Migrating Data from Third-Party RDS MySQL Databases to DDM.......................................... 594.1.6 Scenario 4: Migrating Data from Self-Built RDS MySQL Databases to DDM.............................................. 674.1.7 Scenario 5: Migrating Data from Heterogeneous Databases to DDM........................................................... 744.1.8 Scenario 6: Exporting Data from a DDM Instance.................................................................................................744.2 Restoring from Current Instance..................................................................................................................................... 754.3 Creating a Consistent Backup...........................................................................................................................................764.3.1 Creating a Backup............................................................................................................................................................. 764.3.2 Restoring from a Backup................................................................................................................................................ 78

5 Monitoring Management.................................................................................................... 795.1 Metrics...................................................................................................................................................................................... 795.2 Viewing Metrics..................................................................................................................................................................... 80

6 SQL Syntax.............................................................................................................................. 836.1 Introduction............................................................................................................................................................................ 836.2 DDL............................................................................................................................................................................................ 856.2.1 Creating a Table................................................................................................................................................................. 866.2.1.1 Database and Table Sharding....................................................................................................................................866.2.1.1.1 Sharding Algorithms.................................................................................................................................................. 876.2.1.1.2 Sharding Algorithms.................................................................................................................................................. 891. MOD_HASH............................................................................................................................................................................... 892. UNI_HASH..................................................................................................................................................................................903. RIGHT_SHIFT............................................................................................................................................................................. 924. YYYYMM..................................................................................................................................................................................... 935. YYYYDD....................................................................................................................................................................................... 946. YYYYWEEK..................................................................................................................................................................................967. HASH........................................................................................................................................................................................... 978. Range...........................................................................................................................................................................................999. MM............................................................................................................................................................................................ 10110. DD............................................................................................................................................................................................ 10211. MMDD.................................................................................................................................................................................... 10312. WEEK...................................................................................................................................................................................... 1046.3 DML......................................................................................................................................................................................... 1056.3.1 INSERT................................................................................................................................................................................ 1056.3.2 REPLACE............................................................................................................................................................................. 1066.3.3 DELETE................................................................................................................................................................................ 106


Issue 01 (2021-01-15) Copyright © Huawei Technologies Co., Ltd. iii

6.3.4 UPDATE.............................................................................................................................................................................. 1066.3.5 SELECT................................................................................................................................................................................ 1076.3.6 SELECT JOIN Syntax....................................................................................................................................................... 1086.3.7 SELECT UNION Syntax.................................................................................................................................................. 1086.3.8 SELECT Subquery Syntax.............................................................................................................................................. 1096.4 Functions............................................................................................................................................................................... 1106.5 Unsupported Items............................................................................................................................................................ 1126.6 Supported SQL Statements............................................................................................................................................. 1136.6.1 CHECK TABLE....................................................................................................................................................................1136.6.1.1 Checking DDL Consistency of Table Shards in All Logical Tables............................................................... 1136.6.1.2 Checking DDL Consistency of Table Shards in One Logical Table..............................................................1146.6.2 SHOW RULE...................................................................................................................................................................... 1166.6.3 SHOW TOPOLOGY..........................................................................................................................................................1176.6.4 SHOW DATA NODE........................................................................................................................................................1176.6.5 TRUNCATE TABLE........................................................................................................................................................... 1176.6.5.1 HINT-DB.......................................................................................................................................................................... 1176.6.5.2 HINT-TABLE.................................................................................................................................................................... 1186.6.5.3 HINT-DB/TABLE............................................................................................................................................................ 1186.6.5.4 HINT- ALLOW_ALTER_RERUN................................................................................................................................. 1186.6.6 LOAD DATA....................................................................................................................................................................... 1186.6.7 Customized Hints for Read/Write Isolation........................................................................................................... 1206.6.8 Setting a Hint to Skip the Cached Execution Plan.............................................................................................. 1206.7 Global Sequence................................................................................................................................................................. 1206.7.1 Using NEXTVAL and CURRVAL to Query Global Sequence Numbers.......................................................... 1226.7.2 Using Global Sequences in INSERT or REPLACE Statements...........................................................................1246.8 Database Management Syntax..................................................................................................................................... 1266.9 Advanced SQL Functions................................................................................................................................................. 127

A Change History....................................................................................................................128


Issue 01 (2021-01-15) Copyright © Huawei Technologies Co., Ltd. iv

1 Permissions Management1.1 Creating a User and Granting Permissions

You can use IAM to implement refined permission control for DDM resources. Tobe specific, you can:

● Create IAM users for employees based on the organizational structure of yourenterprise. Each IAM user has their own security credentials, providing accessto DDM resources.

● Grant only the permissions required for users to perform a specific task.● Entrust an account or cloud service to perform professional and efficient O&M

on your DDM resources.

If your account does not need individual IAM users, then you may skip over thissection.

This section describes the procedure for assigning permissions. Figure 1-1 showsthe operation process.

PrerequisitesBefore assigning permissions to user groups, you should learn about DDM systempolicies and select the policies based on service requirements. For details aboutsystem permissions supported by DDM, see Permissions Management. For thesystem policies of other services, see System Permissions.

Distributed Database MiddlewareUser Guide 1 Permissions Management

Issue 01 (2021-01-15) Copyright © Huawei Technologies Co., Ltd. 1

https://support.huaweicloud.com/intl/en-us/productdesc-iam/iam_01_0026.htmlhttps://support.huaweicloud.com/intl/en-us/productdesc-ddm/ddm_01_0010.htmlhttps://support.huaweicloud.com/intl/en-us/usermanual-permissions/iam_01_0001.html

Process Flow

Figure 1-1 Flowchart for granting DDM permissions

1. Create a user group and assign permissions.Create a user group on the IAM console, and assign the DDMReadOnlyAccess policy to the group.

2. Create an IAM user.Create a user on the IAM console and add the user to the group created instep 1.

3. Log in as an IAM user and verify permissions.Log in to the DDM console by using the created user, and verify that the useronly has read permissions for DDM.– Choose Service List > Distributed Database Middleware and click Buy

DDM Instance to buy a DDM instance. If you cannot buy a DDMinstance, the DDM ReadOnlyAccess permission has taken effect.

– Choose any other service in the Service List (assuming that there is onlythe DDM ReadOnlyAccess policy). If a message appears indicatinginsufficient permissions to access the service, the DDM ReadOnlyAccesspolicy has already taken effect.

1.2 Creating a Custom PolicyCustom policies can be created as a supplement to the system policies of DDM.

You can create custom policies in either of the following ways:

● Visual editor: Select cloud services, actions, resources, and request conditions.This does not require knowledge of policy syntax.

● JSON: Edit JSON policies from scratch or based on an existing policy.

For details, see Creating a Custom Policy. The following section containsexamples of common DDM custom policies.



https://support.huaweicloud.com/intl/en-us/usermanual-iam/iam_03_0001.htmlhttps://support.huaweicloud.com/intl/en-us/usermanual-iam/iam_02_0001.htmlhttps://support.huaweicloud.com/intl/en-us/usermanual-iam/iam_01_0552.htmlhttps://support.huaweicloud.com/intl/en-us/usermanual-iam/iam_01_0605.html

Example Policies● Example: Denying DDM instance deletion

A deny policy must be used together with other policies. If the permissionsassigned to a user contain both "Allow" and "Deny", the "Deny" permissionstake precedence over the "Allow" permissions. The following is an example ofa deny policy:{ "Version": "1.1", "Statement": [ { "Effect": "Deny", "Action": [ "ddm:instance:delete" ] } ] }



2 Instance Management2.1 Buying a DDM Instance

Prerequisites

You have logged in to the DDM console.

Procedure

Step 1 On the displayed page, in the upper right corner, click Buy DDM Instance.

Step 2 On the displayed page, configure the required parameters.

Figure 2-1 Buying a DDM Instance

Distributed Database MiddlewareUser Guide 2 Instance Management


Table 2-1 Parameter description

Parameter

Description

BillingMode

DDM instance billing mode, which can be set to Yearly/Monthly orPay-per-use.● Yearly/Monthly: Specify a required duration. The system will

deduct the fees incurred from your account based on the serviceprice.

● Pay-per-use: Do not specify a required duration because thesystem deducts the fees incurred from your account every hourbased on how much the service is used.

Region Region where the DDM instance is located. Select the requiredregion.

AZ Availability zone where the DDM instance is deployed.Currently, some regions support cross-AZ deployment to enhanceavailability of DDM instances.When you create a DDM instance, you can select one or more AZsand then the DDM instance will be deployed across different AZs.NOTE

Deploy your application, DDM instance, and required RDS DB instances inthe same AZ to reduce network latency. Cross-AZ deployment may increasenetwork latency.

InstanceName

Name of the DDM instance, which:● Cannot be left blank.● Must start with a letter.● Must be 4 to 64 characters long.● Contain only letters, digits, and hyphens (-).

InstanceClass

Class of the DDM instance. You can select x86-based general-enhanced or ARM-based Kunpeng general computing-plus.NOTE

Estimate compute and storage requirements of your application and selectan appropriate instance class based on your service type and scale so thatthe DDM instance you will buy can better meet your needs. The instanceclass includes vCPUs and memory.

InstanceNodes

Number of nodes for deploying the DDM instance. You can applyfor up to 32 nodes at a time.

VPC VPC that the DDM instance belongs to. This VPC isolates networksfor different services. It allows you to manage and configure privatenetworks, simplifying network management.Click View VPC to show more details and security group rules.NOTE

The DDM instance should be in the same VPC as the required MySQL DBinstances.



Parameter

Description

SecurityGroup

Select an existing security group.Selecting the same security group is recommended for your DDMinstance, application, and required MySQL DB instances so thatnetwork access is not restricted. If different security groups areselected, add security group access rules to enable such networkaccess.

ParameterTemplate

A parameter template acts as a container for engine configurationvalues that can be applied to one or more DDM instances. You canuse parameter templates to manage DB engine configurations.

RequiredDuration

Duration of the purchased DDM instance. This parameter isavailable only if Billing Mode is set to Yearly/Monthly.You can select 1 month, 2 months, 3 months, 4 months, 5 months,6 months, 7 months, 8 months, 9 months, 1 year, 2 years, or 3years.If you select Auto-renew, the renew cycle is the same as theselected duration.

Step 3 After the configuration is complete, click Next at the bottom of the page.

Step 4 Perform subsequent operations based on the billing mode you select:● For pay-per-use, click Submit.● For yearly/monthly, click Pay Now.

----End

2.2 Changing Class of a DDM Instance

Prerequisites● You have logged in to the DDM console.● The DDM instance is in the Running state.

NO TICE

Change instance class during off-peak hours because services will be temporarilyinterrupted during class changing.

Procedure

Step 1 In the navigation pane, choose Instances; in the instance list, locate the DDMinstance whose instance you want to change, and choose More > ChangeInstance Class in the Operation column.



Figure 2-2 Changing Class of a DDM Instance

Step 2 On the displayed page, view the current instance configuration and select therequired instance class in the lower part of the page.

Figure 2-3 Page for selecting an instance class

Step 3 Click Next and proceed with the following step based on the billing mode of yourinstance:

Step 4 Perform subsequent operations based on the billing mode you select:● For pay-per-use, click Submit.● For yearly/monthly, click Pay Now.

Step 5 Switch back to the instance list, check whether the status of the instance changesto Changing class.



NO TE

● Once the change operation is performed, it cannot be undone. To change the classagain, submit another request after the class change is complete.

● Instance class can be upgraded or downgraded.

----End

2.3 Scaling Out a DDM Instance

Scenarios

As service data increases, you can scale out a DDM instance by adding nodes toimprove service stability.

NO TE

● Scale out your DDM instance during off-peak hours.

● Make sure that the associated DB instances are normal and not undergoing otheroperations.

Procedure

Step 1 Log in to the DDM console; in the instance list, locate the DDM instance that youwant to scale out, and choose More > Scale Out in the Operation column.

Step 2 On the displayed page, view the current instance configuration and specify thenumber of new nodes.

Figure 2-4 Specifying the number of nodes to be added



NO TE

Each DDM instance supports up to 32 nodes.

Step 3 Click Next.

Step 4 On the displayed page, click Submit if all configurations are correct.

Figure 2-5 Confirming the instance information

NO TE

After the scaling task is submitted, the instance status changes to Scaling out.

----End

2.4 Scaling In a DDM Instance

ScenariosThis section describes how to scale in a DDM instance as service data volumedecreases.

NO TE

● Scaling in yearly/monthly DDM instances is not supported.● Scale in your DDM instance during off-peak hours. For pay-per-use instances, nodes are

removed after the scale-in request is submitted.● Make sure that the associated DB instance is normal and not undergoing other

operations.

Procedure

Step 1 Log in to the DDM console, in the instance list, locate the DDM instance that youwant to scale in, and choose More > Scale In in the Operation column.

Step 2 On the displayed page, view the current instance configuration and set thenumber of nodes to be removed.



Figure 2-6 Specifying the number of nodes to be reduced

NO TE

At least one node should be left for a DDM instance.

Step 3 Click Next.

Step 4 On the displayed page, click Submit if all configurations are correct.

Figure 2-7 Confirming the instance information

NO TE

After the scaling task is submitted, the instance status changes to Scaling in.

----End



2.5 Changing Billing Mode of a DDM Instance


Changing to Yearly/Monthly

Step 1 In the instance list, locate the instance whose class you want to change andchoose More > Change to Yearly/Monthly in the Operation column.

The Billing Center-Subscriptions-Change Subscription page is displayed.

Step 2 Select a renewal duration and decide if you want to enable Auto-renew and clickPay.

Figure 2-8 Changing to yearly/monthly

----End

Changing to Pay-Per-Use

Step 1 In the instance list, locate the instance whose billing mode you want to changeand choose More > Change to Pay-per-use in the Operation column.

Step 2 In the displayed dialog box, click Yes.

Step 3 On the displayed page, click Submit.



NO TE

Once a request for changing to pay-per-use is issued, the pay-per-use billing mode takeseffect after the yearly/monthly subscription expires. Fees will be deducted by the hour.

----End

2.6 Renewing a DDM Instance

Prerequisites● You have logged in to the DDM console.● There are DDM instances whose billing mode is yearly/monthly.

Procedure

Step 1 In the instance list, locate the instance that you want to renew and choose More >Renew in the Operation column.

The Unsubscriptions and Returns/Exchanges page is displayed.

Step 2 On the displayed page, select a renewal duration. You can select 1 month, 2months, 3 months, 4 months, 5 months, 6 months, 7 months, 8 months, 9months, or 1 year.

Step 3 Click Pay Now.

Step 4 Confirm the information and click Pay.

----End

2.7 Restarting a DDM Instance


NO TICE

The DDM instance is not available during restart, and the restart operationcannot be undone. Exercise caution when performing this operation.

Procedure

Step 1 In the instance list, locate the DDM instance that you want to restart and chooseMore > Restart in the Operation column.




Figure 2-9 Confirming the restart

Step 3 Wait until the instance is restarted.

----End

2.8 Unsubscribing from a DDM Instance

Prerequisites● You have logged in to the DDM console.● There are DDM instances whose billing mode is yearly/monthly.

Procedure

Step 1 In the instance list, locate the instance that you want to unsubscribe from andchoose More > Unsubscribe in the Operation column.


Figure 2-10 Confirming the unsubscription



The Unsubscriptions and Returns/Exchanges page is displayed.

Step 3 Click Unsubscribe from In-Use Resources and locate the instance.

Step 4 Select the DDM instance that you want to unsubscribe from. Click Unsubscribefrom Resource in the Operation column.

Figure 2-11 Unsubscribe from Resource

Step 5 Confirm the instance information and select your reason.

Step 6 View the unsubscription information, select I have confirmed that a handling feewill be charged for this unsubscription, and click Unsubscribe.

NO TE

After a resource is unsubscribed from, its data will be deleted immediately and cannot berestored. Ensure that you have backed up the data or the data will not be used any longer.


----End

2.9 Deleting a DDM Instance

Prerequisites


NO TICE

Deleted DDM instances cannot be recovered. Exercise caution when performingthis operation.

Procedure

Step 1 In the instance list, locate the DDM instance that you want to delete and chooseMore > Delete in the Operation column.




Figure 2-12 Confirming the deletion

NO TE

● To delete data stored on associated MySQL DB instances, select Delete data onassociated DB instances.

● A monthly/yearly instance cannot be deleted. If you no longer need the instance, switchto Billing Center > My Orders Unsubscriptions and Changes > Unsubscriptions andunsubscribe the instance.

----End

2.10 Reloading Table Data

Prerequisites


Data has been migrated to a new DDM instance using DRS.

Scenarios

If you want to deploy a DDM instance cross regions for DR, use DRS to migrateservice data and then reload table data after the migration is complete so thatDDM can detect where logical table information is located.

Procedure

Step 1 In the instance list, locate the DDM instance whose table data you want to reload.

Step 2 Choose More > Reload Table Data in the Operation column.

A message is displayed, indicating that table data of instance XXX has beenreloaded.



Figure 2-13 Example message

----End

2.11 Parameter Management

ScenariosConfigure parameters of a DDM instance based on your needs to keep theinstance running well.

PrerequisitesThere is a DDM instance available and running properly.

Procedure

Step 1 Log in to the DDM console using the required account.

Step 2 In the navigation pane, choose Instances.

Figure 2-14 Instance management page

Step 3 In the instance list, locate the DDM instance whose parameters you want toconfigure and click its name.

Step 4 In the left pane, click Parameters and modify parameter values as needed.

Figure 2-15 Parameter management page



Table 2-2 Parameters of a DDM instance

Parameter DefaultValue

Value Range Description

bind_table - The value shouldbe in format[{tb.col1,tb2.col2},{tb.col2,tb3.col1},...].tb.col1,tb2.col2indicates a tablename.columnname pair, andthe value maycontain multiplepairs.The versionshould be:DDM 2.3.2.7 orlater

Data association among multiplesharded tables. The optimizerprocesses JOIN operations at theMySQL layer based on theseassociations. For details aboutparameter examples, see thedescription below the table.





max_connections

20000 10–40000 Maximum number of concurrentclient connections allowed perDDM instance.This value depends onspecifications and processingcapabilities of the target RDSMySQL DB instance. Too manyconnections may cause connectionwaiting, affecting performance. Theconsumption of DDM connectionsis related to the number of shardsand SQL design.For example, If a SQL statementcontains a sharding key, each DDMconnection consumes one RDSMySQL instance connection. If theSQL statement contains nosharding key and the number ofshards is N, N RDS MySQLconnections are consumed.If SQL design is appropriate andDDM and RDS processingcapabilities are good enough, youcan set this parameter to a valueslightly smaller than "backend RDSMySQL DB instances" x "maximumconnections supported by each RDSMySQL DB instance".Carry out pressure tests on yourservices and then select a propervalue.

connection_idle_timeout

28800 60–86400 (s) Number of seconds the server waitsfor activity on a connection beforeclosing it.

long_query_time

1 0.1–3600 (s) Minimum duration of a query to belogged as slow.

sql_execute_timeout

28800 100–28800 (s) Number of seconds to wait for aSQL statement to execute beforetiming out.

connection_failed_threshold

50 1–10000 Number of consecutive failedconnection attempts allowedbefore your account and IP addressare locked.





connection_failed_delay

1200 1–86400 Number of seconds before a lockedaccount and IP address areunlocked.

max_allowed_packet

16777216

1024–1073741824

Maximum size of one packet or anygenerated intermediate string. Thepacket message buffer is initializedto net_buffer_length bytes, butcan grow up tomax_allowed_packet bytes whenneeded. The value is small bydefault. You need to set a largervalue to catch large (and possiblyincorrect) packets. The value mustbe a multiple of 1024.

character_set_server

utf8 gbk, utf8,utf8mb4

Character set configured on theDDM instance. To store emoticons,set both this parameter and thecharacter set on the RDS MySQLDB instance to utf8mb4.NOTE

The character sets of the DDM instanceand RDS MySQL DB instance must beconsistent, including values ofparameters character_set_client,character_set_connection,character_set_database,character_set_results, andcharacter_set_server.

collation_server

Theparameter valuerange isbasedon thecharacter_set_servervalue.

The parametervalue range isbased on thecharacter_set_server value.

Collation on the DDM server.You need to configure thisparameter after configuringcharacter_set_server as follows:● If character_set_server is set to

gbk, collation_server can be setto gbk_chinese_ci or gbk_bin.

● If character_set_server is set toutf8, collation_server can beset to utf8_unicode_ci orutf8_bin.

● If character_set_server is set toutf8mb4, collation_server canbe set to utf8mb4_unicode_cior utf8mb4_bin.

sql_audit OFF OFF or ON Whether to enable or disable SQLaudit.





ddm_instance_type

- SINGLE,MASTER,SLAV

SINGLE: indicates that the DDMinstance is a single instance.SLAVE: indicates that the DDMinstance is the standby instance foracross-region DR.MASTER: indicates that the DDMinstance is the primary instance foracross-region DR.

transaction_policy

XA XA, FREE,NO_DTX

XA: XA transaction that attempts toensure atomicity and visibility.FREE: Best-effort committransaction that allows data to bewritten to multiple shards, withoutimpacting performance. Thissetting does not ensure atomicity.NO_DTX: With this setting, single-shard transactions are performed.

By default, DDM only allows you to modify the preceding instance parameters. Ifyou need to modify other parameters for some special scenarios, such as datamigration, contact customer service.

Parameter configuration examples:

Figure 2-16 Result if bind_table is not used

Figure 2-17 Result if bind_table is used

Step 5 Click Save in the upper left corner and then Yes in the displayed dialog box.



Figure 2-18 Confirming the parameter modification

NO TE

● Modifying DDM parameters may affect access to the instance. Exercise caution whenperforming this operation.

● It takes 20s to 60s to have the modifications to take effect.

----End

2.12 Parameter Template Management

2.12.1 Creating a Parameter Template

ScenariosA maximum of 100 parameter templates can be created under one account. AllDB engines in the current project share the same parameter template quota.

Create a Parameter Template

Step 1 In the navigation pane, choose Parameter Templates.

Figure 2-19 Parameter templates

Step 2 In the upper right corner, click Create Parameter Template.

Step 3 In the displayed dialog box, configure the required parameters.



Figure 2-20 Creating a parameter template

Step 4 Select the Custom Template tab, click the name of the newly-created template,and modify the assigned values of parameters as required.

Figure 2-21 Parameter settings

Step 5 Click Save and Yes on the displayed page.

Step 6 Click the Template History tab to view change history.

Figure 2-22 History of a template

----End

2.12.2 Selecting a Parameter Template

Prerequisites




ScenariosAll DB engines in the current project share the same parameter template quota.

ProcedureStep 1 In the instance list, locate the DDM instance that you want to select a parameter

template for and choose More > Modify Parameter Template in the Operationcolumn.

The Modify Parameter Template dialog box is displayed.

Step 2 Select the required parameter template. Click OK.

Figure 2-23 Configure Parameter Template

----End

2.13 Account Management

2.13.1 Creating an Account

Prerequisites● You have logged in to the DDM console.● There are schemas available in the DDM instance that you want to create an

account for.

ProcedureStep 1 In the instance list, locate the required DDM instance and click its name.

Step 2 In the navigation pane, choose Accounts.

Figure 2-24 Account management



Step 3 On the displayed page, click Create Account and configure the requiredparameters.

Figure 2-25 Creating a DDM account

Table 2-3 Required parameters

Parameter Description

Name Account name. The name:● Must start with a letter.● Is case-insensitive.● Must be 6 to 32 characters long.● Contains only letters, digits, and underscores (_).

Password Password of the account. The password:● Must contain 8 to 32 characters.● Must be a combination of any three of the following

character types: letters, digits, spaces, and special characters`~!@#$%^&*()-_=+\\|[{}];:'\"/?

● Cannot be a weak password that is overly simple and easilyguessed.



Parameter Description

ConfirmPassword

None

Permissions Permissions of a DDM account, including:● Basic Permissions: CREATE, DROP, ALTER, INDEX, INSERT,

DELETE, UPDATE, and SELECT● Extended Permissions: Table query, Table delete, and

Table updateWhen you configuring permissions, select basic permissionsbased on your needs.After you select a basic permission, the correspondingextended permissions are selected automatically. Not everybasic permission corresponds to an extended permission.The extended permissions for each basic permission are asfollows:– Table query corresponds to SELECT.– Table delete corresponds to DELETE.– Table update corresponds to UPDATE.

Schema The schema to be associated with the account. You can selectan existing schema from the drop-down list.The account can access only the associated schemas.

Description A description of the account, which cannot exceed 256characters.

Step 4 Confirm the settings and click OK.

----End

2.13.2 Modifying an Account

PrerequisitesYou have logged in to the DDM console.

Procedure

Step 1 In the instance list, locate the DDM instance that you want to modify and click itsname.


Step 3 In the account list, locate the required account and click Modify in the Operationcolumn.



Figure 2-26 Modifying an account

Step 4 In the displayed dialog box, modify the associated schemas, permissions, anddescription.

Figure 2-27 Parameters for modifying a DDM account

Step 5 Click OK.

----End

2.13.3 Deleting an Account

Prerequisites


NO TE

Deleted accounts cannot be recovered. Exercise caution when performing this operation.



ProcedureStep 1 In the instance list, locate the DDM instance with the account that you want to

delete and click the instance name.


Step 3 In the account list, locate the account that you want to delete and choose More >Delete in the Operation column.

Figure 2-28 Deleting a DDM account


----End

2.14 Read/Write SplittingPrerequisites

● You have logged in to the DDM console.● You have logged in to the RDS console.● You have bought an RDS DB instance with no read replicas and a DDM

instance.

ProcedureStep 1 In the RDS DB instance list, locate the DB instance whose read/write requests you

want to split and choose More > Create Read Replica in the Operation column.

Step 2 On the displayed page, specify the billing mode, instance class, and other requiredparameters and click Next.

Figure 2-29 Creating a read replica



Step 3 On the displayed page, confirm the selected information and click Submit.

Step 4 Wait 3 to 5 minutes for the read replica to be created.

Step 5 Switch to the DDM console, locate the DDM instance you created and click itsname.

Step 6 On the displayed page, in the left navigation pane, choose Schemas and clickCreate Schema. On the displayed page, specify Sharding Mode and Shards perDB Instance and click Next: Select DB Instance.

Figure 2-30 Creating a schema

Step 7 On the displayed page, select the RDS DB instance you created. Click Preview. Onthe displayed page, enter the administrator and password of the created RDS DBinstance and click Finish.

Figure 2-31 Selecting the required RDS DB instances

Step 8 Wait 1 to 3 minutes for the schema to be created.

Step 9 In the left pane, click Basic Information. At the Associated DB Information area,view the primary RDS DB instance and its read replicas in the list.



Figure 2-32 Associated DB instance information

Step 10 Click Modify Read Policy in the Operation column of the primary instance.

----End

2.15 Task Center

ScenariosThis topic describes how to locate a task by specifying a start date and end date.

Procedure

Step 1 Log in to the DDM console, and in the navigation pane, choose Task Center.

Step 2 Specify a start date, end date, task type, or task status. Tasks that meet thespecified conditions are displayed.

Step 3 Locate the required task.

Figure 2-33 Task Center

----End



3 Schema Management3.1 Creating a Schema

Prerequisites● You have logged in to the DDM console.● The DDM instance is in the Running state.● DDM supports RDS and GaussDB DB instances. The following uses an RDS

MySQL DB instance as an example.

You can create a schema in two ways: on the Instances page or on the Schemaspage. This section uses the DDM instance page as an example to describe how tocreate a schema.

Figure 3-1 On the Instances page

Figure 3-2 On the Schemas page

Procedure

Step 1 In the navigation pane, choose Instances. In the instance list, locate the DDMinstance that you want to create a schema for and click Create Schema in theOperation column.

Step 2 On the displayed page, specify a sharding mode, enter a schema name, select thenumber of shards per DB instance, and click Next: Select DB Instance.

Distributed Database MiddlewareUser Guide 3 Schema Management


Figure 3-3 Create Schema

NO TE

Sharding mode:● Sharded: One schema corresponds to multiple DB instances.● Unsharded: One schema corresponds only to one DB instance, and only one database

shard is created for that instance.The schema name contains 2 to 24 characters and must start with a letter. Only letters,digits, and underscores (_) are allowed.The shards per DB instance ranges from 1 to 64.

Step 3 On the RDS DB Instances tab page, select the required RDS DB instances. You canchoose to configure the instances as the unsharded table storage nodes. ClickPreview.

Step 4 On the displayed page, enter the administrator password and click Finish.

----End

3.2 Exporting Schema Information

ScenariosExport schema information from the primary DDM instance for cross-region DR.

PrerequisitesThere are schemas available in the DDM instance that you want to export schemainformation from.

Procedure

Step 1 Log in to the DDM console; in the instance list, locate the required DDM instanceand click its name.

Step 2 On the displayed page, in the navigation pane, choose Schemas.



Figure 3-4 Schema list

Step 3 On the displayed page, click Export Schema Information. All schema informationof the current DDM instance is exported as a JSON file.

----End

3.3 Importing Schema Information

Scenarios

Create a schema in the standby DDM instance for cross-region DR.

Prerequisites

The standby DDM instance has no schemas.

Procedure

Step 1 Log in to the DDM console; in the instance list, locate the DDM instance that youwant to import schema information into and click its name.


Figure 3-5 Schema list

Step 3 On the displayed page, click Import Schema Information.

NO TE

● More than one JSON file can be imported into a DDM instance in the premise that theDDM instance does not have schemas with the same names as those in the JSON file.

Step 4 On the displayed page, click Select File to select the JSON file exported inExporting Schema Information, choose the required DB instances, enter therequired password, and click Finish.

NO TE

● The number of selected DB instances must be the same as the number of DB instancesassociated with the imported schemas.

----End



3.4 Deleting a Schema

Prerequisites● You have logged in to the DDM console.● You have created a schema.

NO TICE

Deleted schemas cannot be recovered. Exercise caution when performing thisoperation.

Procedure

Step 1 In the instance list, locate the DDM instance that you want to delete and click itsname.

Step 2 In the navigation pane, choose Schemas.

Step 3 In the schema list, locate the schema that you want to delete and click Delete inthe Operation column.


NO TE

● Your schema will become faulty if its associated RDS DB instances are deleted on theRDS console.

● To delete data stored on associated RDS MySQL DB instances, select Delete data onassociated RDS DB instances.

● Before you delete a schema, check whether there are DB instances associated with thisschema. If the associated DB instances have been deleted, click Synchronize DBInstance Data and delete the schema.

● If the associated RDS DB instances are not deleted but their information is modified,such as the instance name, engine, engine version, maximum connections, port number,or IP address, click Synchronize DB Instance Data and delete the schema.

----End

3.5 Configuring SQL Blacklist

OverviewTo prohibit executing some SQL statements, you can configure a blacklist and addthose statements to it.

Prerequisites● You have logged in to the DDM console.● A DDM instance is running properly and has available schemas.



Procedure

Step 1 In the instance list, locate the instance that contains schemas you require and clickthe instance name.

Step 2 On the displayed page, choose Schemas.

Step 3 In the schema list, locate the schema that you want to configure a blacklist forand click Configure SQL Blacklist in the Operation column.

Figure 3-6 Configure SQL Blacklist

Step 4 In the displayed dialog box, click Edit, enter the required SQL statements orregular expressions in prefix match, full-text, and regular expression match boxes,and click OK.



NO TE

● Prefix Match: Enter SQL statements that contain keywords such as DROP XXXX orDELETE XXX and are not allowed by the current schema.

● Full-text Match: Enter full-text SQL statements that are not allowed by the currentschema.

● Separate SQL statements in the blacklist with commas (,). The total size of SQLstatements for prefix match and full-text match cannot exceed 1 KB.

● If you want to clear all the SQL statements in prefix match and full-text match areas,clear them separately and click OK.

----End

3.6 Scaling Out a SchemaService growth requires schemas to provide more storage space and supporthigher concurrency. To expand storage and improve concurrency, you can scale outa schema by adding DB instances.

3.6.1 Rebalancing Shards

Overview

One scaling method is rebalancing, with which, database shards are rebalancedbetween original and new DB instances.

Prerequisites● There are schemas available in a running DDM instance.● You have purchased a MySQL DB instance. The DB instance and your DDM

instance are in the same VPC, and the DB instance is not in use by other DDMinstances.

Procedure



Step 3 In the schema list, locate the schema that you want to scale out and click ScaleOut the Operation column.

Figure 3-7 Scale Out

Step 4 On the displayed page, select Rebalance for Scaling Method and the DBinstances you want to add, enter the required password, and click Next: ConfirmInformation.



Figure 3-8 Rebalance

Figure 3-9 How rebalancing works

Step 5 Click Next: Pre-check. After all check items are complete, click Scale out.

NO TE

If the DDM instance version is earlier than 2.4.1.3, do not set sql_mode to ANSI_QUOTES.If you do so, do not use double quotation marks to quote text strings because thequotation marks will be translated into an identifier that makes the quoted string invalid.

For example, in the command select * from test where tb = "logic", logic cannot beparsed correctly.

Step 6 If the schema status changes to Running, the scaling out succeeds, and thenewly-added MySQL DB instance is displayed in the Associated DB Instancecolumn.

If the schema status changes to Scaling failed, the schema failed to be scaled out.Click Retry or Roll Back.

NO TE

● Scale out a schema during off-peak hours.

● Storage scaling does not support tables without primary keys.

● Only schemas whose status is Running can be scaled out.

● Only one schema can be scaled out in a DDM instance at a time.

● A maximum of 256 MySQL DB instances can be selected for scaling.

● You can view the scaling progress on the Schemas page. The whole scaling processtakes about 5 to 30 minutes depending on the amount of data to be migrated.

● During scaling, DDM automatically enables the LOAD DATA function. After scaling, youcan disable the function in parameter template settings of the MySQL DB instance asneeded.

----End



3.6.2 Doubling Shards and Resharding Data Records

OverviewThe other method is resharding, with which, physical databases corresponding tothe schema are doubled and rebalanced to each DB instance as much as possible.During this process, data records are redistributed.

Prerequisites● There are schemas available in a running DDM instance.● You have purchased a MySQL DB instance. The DB instance and your DDM

instance are in the same VPC, and the DB instance is not in use by other DDMinstances.

Procedure



Step 3 In the schema list, locate the schema that you want to scale out and click ScaleOut the Operation column.

Figure 3-10 Scale Out

Step 4 On the displayed page, select Reshard for Scaling Method and the DB instancesyou want to add, enter the required password, and click Next: ConfirmInformation.

Figure 3-11 Reshard



Figure 3-12 How resharding works

Step 5 Click Next: Pre-check. After all check items are complete, click Scale out.

NO TE

If the DDM instance version is earlier than 2.4.1.3, do not set sql_mode to ANSI_QUOTES.If you do so, do not use double quotation marks to quote text strings because thequotation marks will be translated into an identifier that makes the quoted string invalid.For example, in the command select * from test where tb = "logic", logic cannot beparsed correctly.

Step 6 If the schema status changes to Running, the scaling out succeeds, and thenewly-added MySQL DB instance is displayed in the Associated DB Instancecolumn.

If the schema status changes to Scaling failed, the schema failed to be scaled out.Click Retry or Roll Back.

NO TE

● Scale out a schema during off-peak hours.● Storage scaling does not support tables without primary keys.● Only schemas whose status is Running can be scaled out.● Only one schema can be scaled out in a DDM instance at a time.● A maximum of 256 MySQL DB instances can be selected for scaling.● You can view the scaling progress on the Schemas page. The whole scaling process

takes about 5 to 30 minutes depending on the amount of data to be migrated.● During scaling, DDM automatically enables the LOAD DATA function. After scaling, you

can disable the function in parameter template settings of the MySQL DB instance asneeded.

----End



4 Data Management4.1 Data Migration

4.1.1 OverviewData migration is a process of migrating data from databases to DDM orexporting data from DDM to other database systems. You can use the MySQL toolmysqldump to export data. If both full and incremental migrations are required,use Data Replication Service (DRS).

NO TE

Services may be interrupted during migration. The duration of the interruption depends onthe amount of data to be migrated and on network conditions.

DDM supports the following migration methods:

● Using official MySQL clients. The following passages will use an RDS MySQLDB instance as an example to describe this method.

● Using DRS

Data migration is involved in the following scenarios:

1. Scenario 1: Migrating Data from HUAWEI CLOUD RDS to DDM2. Scenario 2: Migrating Data from On-Premises RDS MySQL Databases to

DDM3. Scenario 3: Migrating Data from Third-Party RDS MySQL Databases to

DDM4. Scenario 4: Migrating Data from Self-Built RDS MySQL Databases to DDM5. Scenario 5: Migrating Data from Heterogeneous Databases to DDM6. Scenario 6: Exporting Data from a DDM Instance

Distributed Database MiddlewareUser Guide 4 Data Management


NO TE

● Data migration is complicated and should be done during off-peak hours. Thisguide is for reference only. Design a proper migration solution based on yourservice scenario, data volume, and downtime requirements.

● To migrate a large amount of data or a large data table, submit a service ticket orcontact technical support to perform a rehearsal before formal migration.

● You can access DDM only using an ECS. Therefore, export databases as files,upload the files to the ECS, and import the data in the files to DDM from the ECS.

4.1.2 Migration EvaluationEvaluate and verify data migration before migrating your database to a cloudplatform.

Based on the status of the data to be migrated and the future service scale,evaluate migration scenarios separately and make the required preparations.

Table 4-1 Evaluation and preparation before data migration

Item Description

Amount of the data tobe migrated and classof the required DDMinstance and RDSMySQL DB instance

● Use vertical sharding, then horizontal sharding toshard a source RDS MySQL DB instance.

● Execute the following SQL statement to evaluatestorage space occupied by the source RDS MySQLDB instance:select concat(round(sum(DATA_LENGTH/1024/1024),2),'MB')as data from information_schema.TABLES;

● Partition a table with more than 10 million rows (orexpected to exceed 10 million rows) into tableshards.

● Ensure that data stored on a single RDS DB instancedoes not exceed 500 GB.

Information aboutschemas and logicaltables mapped toeach table in thesource database

● Specify the information about logical tables mappedto each source table, including the number of datarecords, logical table type, schema, DDM instance,and associated RDS DB instances.

● If there is an RDS MySQL DB instance available inthe destination DDM instance and the targetschema is unsharded, associate the DB instance withthe schema without the need to migrate tablestructures and data.



Item Description

Compatibility ● Check whether the source database uses the sameMySQL version as the target MySQL DB instanceassociated with the destination DDM instance.

● The class and storage space of the destinationinstance cannot be less than those of the sourcedatabase.

● Check whether the table structure and character setof the source database are the same as those of thedestination instance.

● For a logical table that uses hash sharding, ensurethat the number of data records to be migratedeach time does not exceed 10 million. For a logicaltable using range sharding, the number cannotexceed 5 million.

● If a table contains a huge number of data records,export the data in batches. Specify the WHEREcondition on mysqldump to limit the number ofdata records to be exported at a time.

● SQL text files imported into DDM can only containstandard DML INSERT statements.

● Evaluate the compatibility of SQL statements inyour application with DDM.

Before migrating data, collect the required information listed in Table 4-2.

Table 4-2 Information to be collected

Source/Destination Information Item

Source RDS MySQLDB instance

Connection address

Listening port

Database account

Database name

Table name

Destination DDMinstance

Connection address

Listening port

Username

Name of a shard on the target DB instance

Connection address of the target DB instance

Listening port of the target RDS DB instance



Source/Destination Information Item

Administrator of the RDS MySQL DB instance

Database name

ECS EIP

Login credentials (username and password)

4.1.3 Scenario 1: Migrating Data from HUAWEI CLOUD RDS toDDM

Scenarios

As an enterprise user, you have purchased and started using an RDS MySQL DBinstance on HUAWEI CLOUD and now expect to store data in a distributedmanner.

NO TE

Services may be interrupted during migration. The duration of the interruption depends onthe amount of data to be migrated and on network conditions.

Migration Process

Figure 4-1 Migrating data from an RDS MySQL DB instance to DDM



NO TE

The source RDS MySQL DB instance, destination DDM instance, and target RDS MySQL DBinstance must be in the same VPC and have the same security group rules.

The migration procedure is as follows:

1. Prepare for the migration.2. Export data from the source RDS MySQL DB instance to an ECS.3. Import the data into the destination DDM instance.4. Verify the consistency of data before and after migration.5. Verify services.

Constraints● The source DB instance, destination DDM instance, and target DB instance,

and the required ECS must be in the same VPC and security group.● Before data migration, you have to stop your workloads to ensure data

integrity.● You have to export data from an existing DB instance and import the data

into DDM. Associating DDM with the DB instance to store existing data in adistributed manner is not supported.

● Before migrating data, you have to prepare schemas, database shards, tableshards, and broadcast tables with the same names and structures as thesource. Importing data into new databases, table shards, or broadcast tablesduring migration is not supported. Methods for creating various types oflogical tables are described in Table 4-4.

● The target DB instance must be of the same MySQL version as the source DBinstance.

Prepare for the Migration● Prepare an ECS that can access the source DB instance.

a. Ensure that source and target DB instances and destination DDMinstance are in the same VPC.

b. Configure the same security group for the source and target DB instancesand the destination DDM instance. If they belong to different securitygroups, configure security group rules to allow them to access each other.

c. Install an official MySQL (5.6 or 5.7) client on the ECS and the followingOSs by running the required commands:

▪ Red Hat Linux: yum install mysql mysql-devel▪ Debian Linux: apt install mysql-client-5.7 mysql-client-core-5.7

d. Ensure that there is enough disk space and memory on the ECS to storeand compare dump files.

● Prepare a DDM instance and create the required DDM accounts, schemas, andlogical tables.

a. Apply for a DDM instance and create a DDM account and schema on theDDM console.



Determine the number of DDM instances based on MigrationEvaluation.

b. Export table structures of the source DB instance to a SQL text file.mysqldump -h {DB_ADDRESS} -P {DB_PORT} -u {DB_USER} -p --skip-add-locks --add-locks=false --no-data {DB_NAME} {TABLE_NAME} > {mysql_table_schema.sql}

Table 4-3 describes the parameters in the command.


Parameter Description Remarks

DB_ADDRESS Indicates theconnection address ofthe target database.

This parameter ismandatory.

DB_PORT Indicates the listeningport of the targetdatabase.


DB_USER Indicates the databaseaccount.


--skip-lock-tables

Indicates that data isexported when tablesare not locked.

The declaration for tablelocking is enabled for someparameters by default.Therefore, you are advisedto add this parameter tothe end of a data exportstatement.

--add-locks=false

Indicates that no locksare added to tables.

-

--no-data Indicates that onlydatabase tablestructures areexported.

This parameter is used toexport table structures.

DB_NAME Indicates the databasename.


TABLE_NAME Indicates the tablename.

Separate every two tablenames of the same typeusing a space.Exporting only service-related table structures isrecommended.




mysql_table_schema.sql

Indicates the name ofthe generated tablestructure file.

The file varies depending onthe table whose structure isexported.You can name the file in theformat of schemaname_logical tablename_schema to preventdata from beingoverwritten, for example,mysql_table_schema.sql.

NO TE

The parameters listed above are commonly used for data export. It is impossibleto list all mysqldump parameters. To optimize certain parameters, visit the officialMySQL website or contact DDM technical support.

c. Create a logical table.Ensure that structures of the logical table are consistent with thoseexported in b. Map source tables to the logical table of the destinationDDM instance, and specify migration policies for different table structuresand data.

NO TE

Before creating a logical table, execute SHOW CREATE TABLE TABLE_NAME toquery structures of tables in the source DB instance.

Table 4-4 Table migration policies

SchemaType

LogicalTableType

Table StructureMigration Policy

Table DataMigration Policy

Sharded Sharded Create logical tableson the DDM console.

Connect to DDMand import tabledata.Sharded Global

Sharded Unsharded

Connect to the targetDB instance andimport table creationstatements.

Connect to thetarget DB instanceand import tabledata.Unsharded Common

d. Delete test data in the destination DDM instance to prevent conflicts withthe data to be migrated.

● Prepare a new RDS MySQL DB instance.



Export DataConnect to the source DB instance using an IP address and export data from theinstance using mysqldump.

● Prerequisites– The destination DDM instance is in the same subnet and VPC as the

required ECS.– Security group rules allow DDM access.

● ProcedurePerform the following operations to export table data from the source DBinstance:

a. Stop the service system of the source DB instance to ensure that exporteddata is up to date.

b. Open your MySQL client and run the following command to connect tothe destination DDM instance and export data as a SQL text file:mysqldump -h {DB_ADDRESS} -P {DB_PORT} -u {DB_USER} -p --single-transaction --hex-blob --complete-insert --set-gtid-purged=OFF --quick --no-create-info --skip-comments --skip-add-locks --add-locks=false [--where=""] {DB_NAME}{TABLE_NAME} > {mysq_table_data.sql}



DB_ADDRESS Indicates theconnection addressof the targetdatabase.


DB_PORT Indicates thelistening port of thetarget database.


DB_USER Indicates thedatabase account.


--complete-insert Uses a completeINSERT statement(including columnnames).

-




--single-transaction After this parameteris configured, aBEGIN SQLstatement issubmitted beforedata is exported.The BEGIN SQLstatement does notblock anyapplication andensures dataconsistency whenthe data is exported.It applies only to themulti-versionstorage engine,InnoDB.

-

--quick Directly exportsdata to standardoutput files withoutbuffering queries.

This avoids sharplyincreasing memoryusage if there ismassive volumes ofdata.

--hex-blob Exports binary stringfields inhexadecimal format.This parameter ismandatory if thereis binary data to beexported.

-

--no-create-info Exports datawithout adding anyCREATE TABLEstatements.

This parameter is usedfor data export.

--skip-comments Disables additionalcomments.

-

--skip-lock-tables Indicates that datais exported whentables are notlocked.

The declaration fortable locking is enabledfor some parametersby default. Add thisparameter to the endof each data exportstatement.

--add-locks=false Indicates that nolocks are added totables.

-




--set-gtid-purged=OFF If the MySQLversion is 5.6 or 5.7,configure thisparameter.

If the MySQL version isearlier than 5.6, do notconfigure thisparameter.

--skip-add-locks Controls lockoperations duringdata export to avoidperformance issues.

-

--where Dumps only therecords selectedbased on a specifiedWHERE condition.

If the conditioncontains commandcomment symbols suchas special spaces orcharacters, put thecondition in quotes.

DB_NAME Indicates thedatabase name.



Separate every twotable names of thesame type using aspace.Exporting only service-related table structuresis recommended.

mysq_table_data.sql Indicates the nameof the generatedtable data file.

The file name variesdepending on the tablewhose data is exported.You can name the filein the format ofschema name_logicaltable name_data toprevent data frombeing overwritten, forexample,mysq_table_data.sql.

NO TE

The parameters listed above are generally used for data export. Not allmysqldump parameters are described here. Log in to the official MySQL websiteor contact the DDM administrator to optimize certain parameters.

c. Check the size of the SQL text file and check whether data is successfullyexported.

▪ If the file size is not 0 bytes, data export is successful.



▪ If the file size is 0 bytes, data export failed. Contact DDMadministrator.

Import Data

Step 1 Enable read-only access to DDM databases on your application.

Step 2 Delete test data in the destination DDM instance to prevent conflicts with thedata to be migrated.

Step 3 Proceed with subsequent operations depending on the types of tables:

● For unsharded or common tables, use a MySQL client to connect to the targetDB instance and run the following commands to import table structure anddata files:mysql -f -h {RDS_ADDRESS} -P {RDS_PORT} -u {RDS_USER} -p {DB_NAME} < {mysql_table_schema.sql}Enter password: **********mysql -f -h {RDS_ADDRESS} -P {RDS_PORT} -u {RDS_USER} -p {DB_NAME} < {mysq_table_data.sql}Enter password: **********

– RDS_ADDRESS indicates the address of the target DB instance.

– RDS_PORT indicates the port number of the target DB instance.

– RDS_USER indicates the username of the target DB instance.

– DB_NAME indicates the name of the target database. If data ofunsharded tables is to be imported, DB_NAME indicates the name of thefirst shard in the DB instance.

– mysql_table_schema.sql indicates the name of the table structure file tobe imported.

– mysq_table_data.sql indicates the name of the table data file to beimported.

NO TE

Before importing data of unsharded tables or common tables, delete the last line(for example, Dump completed on 2018-06-28 19:53:03) in the table structurefile. Otherwise, data import may fail.

● For sharded tables or broadcast tables, use a MySQL client to connect to DDMand import data files.mysql -h {DDM_ADDRESS} -P {DDM_PORT} -u {DDM_USER} -p {DB_NAME} < {mysq_table_data.sql}Enter password: **********

– DDM_ADDRESS indicates the address of the DDM instance into whichdata is imported.

– DDM_PORT indicates the listening port.

– DDM_USER indicates the DDM account.

– DB_NAME indicates the name of the target schema.

– mysq_table_data.sql indicates the name of the table data file to beimported.



NO TE

▪ Import data during off-peak hours. Performance of the destination DDMinstance and target DB instance may be affected during data import.

▪ If an interruption or exception occurs during data import, execute TRUNCATETABLE {TABLE_NAME} to clear and import data again. Executing thisstatement will clear all table data. Exercise caution when executing it.

▪ Ensure that the number of data records to be imported into a broadcast tableis less than 5 million.

----End

Verify Data

Step 1 Back up logical information of the destination DDM instance on the ECS.

● Export table structures:mysqldump -h {DDM_ADDRESS} -P {DDM_PORT} -u {DDM_USER} -p --single-transaction --set-gtid-purged=OFF --no-data{DB_NAME} {TABLE_NAME} > {mysql_table_schema_new.sql}

● Export table data:mysqldump -h {DDM_ADDRESS} -P {DDM_PORT} -u {DDM_USER} -p --single-transaction --hex-blob --complete-insert --set-gtid-purged=OFF --quick --no-create-info --skip-comments [--where=""] {DB_NAME}{TABLE_NAME} > {mysq_table_data_new.sql}

Step 2 Check data consistency.

1. Check whether the number of data records in each table is the same on thesource DB instance and destination DDM instance: {TABLE_NAME} indicatesthe table name.select count(*) from {TABLE_NAME};

2. On the ECS, check whether table structures and data records are consistentbefore and after the export:diff -B -w -q -i {mysql_table_schema.sql} {mysql_table_schema_new.sql};echo $?diff -B -w -q -i {mysq_table_data.sql} {mysq_table_data_new.sql};echo $?

– If yes, the data is successfully migrated.

– If no, contact DDM technical support.

Step 3 Verify in an E2E manner whether your application's read-only access is normal.

Step 4 Disable read-only access.

----End

Verify Services1. Switch the service data source to DDM.

2. Verify whether you can read and write data from and to DDM normally.

– If yes, data migration is completed.

– If no, switch services to the source RDS MySQL DB instance and contactDDM technical support.



4.1.4 Scenario 2: Migrating Data from On-Premises RDSMySQL Databases to DDM

Scenarios

You are using an on-premises RDS MySQL instance and want to use DDM to storedata in a distributed manner.

NO TICE

Services may be interrupted during migration. The duration of the interruptiondepends on the amount of data to be migrated and on network conditions.

Migration Process

Figure 4-2 Migrating data from an on-premises MySQL instance to DDM

Constraints● The destination DDM instance can communicate with the ECS where the

target RDS MySQL DB instance is deployed.● Before data migration, you have to stop your workloads to ensure data

integrity.● Before migrating data, you have to prepare schemas, database shards, table

shards, and broadcast tables with the same names and structures as thesource. Importing data into new databases, table shards, or broadcast tablesduring migration is not supported. Methods for creating various types oflogical tables are described in Table 4-7.



● The target RDS MySQL DB instance is of the same MySQL version as thesource on-premises DB instance.

Prepare for the Migration● Prepare an ECS that can access the source DB instance.

a. Ensure that the source on-premises DB instance can communicate withthe destination DDM instance and target DB instance.

b. Install an official MySQL (5.6 or 5.7) client on the ECS and the followingOSs by running the required commands:

▪ Red Hat Linux: yum install mysql mysql-devel▪ Debian Linux: apt install mysql-client-5.7 mysql-client-core-5.7

c. Ensure that there is enough disk space and memory on the ECS to storeand compare dump files.

● Prepare a DDM instance and create the required DDM accounts, schemas, andlogical tables.

a. Apply for a DDM instance and create a DDM account and schema on theDDM console.

b. Export table structures of the source DB instance to a SQL text file.mysqldump -h {DB_ADDRESS} -P {DB_PORT} -u {DB_USER} -p--no-data --skip-add-locks --add-locks=false {DB_NAME} {TABLE_NAME} > {mysql_table_schema.sql}

Table 4-6 describes the parameters in the command.



DB_ADDRESS Indicates theconnection addressof the targetdatabase.


DB_PORT Indicates thelistening port of thetarget database.


DB_USER Indicates thedatabase account.


--skip-lock-tables Indicates that datais exported whentables are notlocked.

The declaration fortable locking is enabledfor some parametersby default. Add thisparameter to the endof each data exportstatement.

--add-locks=false Indicates that nolocks are added totables.

-




--no-data Indicates that onlydatabase tablestructures areexported.

This parameter is usedto export tablestructures.

DB_NAME Indicates thedatabase name.



Separate every twotable names of thesame type using aspace. Exporting onlyservice-related tablestructures isrecommended.

mysql_table_schema.sql

Indicates the nameof the generatedtable structure file.

The file variesdepending on the tablewhose structure isexported.You can name the filein the format ofschema name_logicaltable name_schema toprevent data frombeing overwritten, forexample,mysql_table_schema.sql.

NO TE

The parameters listed above are generally

Distributed Database Middleware · 2020. 11. 26. · Distributed Database Middleware User Guide...

Documents

Transcript of Distributed Database Middleware · 2020. 11. 26. · Distributed Database Middleware User Guide...