Database Sharding with Oracle RDBMS - First Impressions · –Troubleshooting/Performance Tuning...

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA

HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

Database Sharding with Oracle RDBMS First Impressions

Robert Bialek Principal Consultant

Who Am I

DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions

Principal Consultant and Trainer at Trivadis GmbH in Munich

– MSc in Computer Engineering

Focus:

– Oracle Database High Availability

– Database Architecture/Internals

– Backup/Recovery

– Troubleshooting/Performance Tuning

– Linux Operating System

Trainer for the following Trivadis courses:

– Oracle Grid Infrastructure, RAC, Data Guard

17.11.2016 2

Our company.

3

Trivadis is a market leader in IT consulting, system integration, solution engineering

and the provision of IT services focusing on and

technologies

in Switzerland, Germany, Austria and Denmark. We offer our services in the following

strategic business fields:

Trivadis Services takes over the interacting operation of your IT systems.

O P E R A T I O N

DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions 16.11.2016

COPENHAGEN

MUNICH

LAUSANNE

BERN

ZURICH

BRUGG

GENEVA

HAMBURG

DÜSSELDORF

FRANKFURT

STUTTGART

FREIBURG

BASEL

VIENNA

With over 600 specialists and IT experts in your region.

4

14 Trivadis branches and more than

600 employees

200 Service Level Agreements

Over 4,000 training participants

Research and development budget:

CHF 5.0 million

Financially self-supporting and

sustainably profitable

Experience from more than 1,900

projects per year at over 800

customers


5

Technology on its own won't help you. You need to know how to use it properly.


Database Scalability – Limits?


For some business requirements:

– extreme scalability/elasticity, global data distribution,…

know database techniques might not be sufficient.

The solution: Database Sharding (horizontal scaling)

– Not really a new concept: Cassandra, MongoDB, MySQL, …

One Database

Split one

into many

16.11.2016 6

Only 1/N

components

…

Agenda


1. Sharded Database Architecture

2. Data Sharding Methods

3. Data High Availability/Disaster Recovery

4. Session/Statement Routing

5. Conclusions

16.11.2016 7


Sharded Database Architecture

16.11.2016 8

SDB Architecture – Distributed Partitioning


Horizontally distributed data partitioning using shared-nothing database architecture

– Sharded tables: data exist in one shard

– Duplicated tables: data exist in all shards

Un

sh

ard

ed D

B (

Ta

ble

s)

Server B

Shard 1 Shard 2 Shard 3

Server C Server D

Sharded Database (SDB)

Server A

Distributed

Partitioning

16.11.2016 9

Don’t share physical resources

No cluster software

Up to 1000

non-CDB shards

SDB Architecture – Big Picture


Shard Director 1 Shard Director 2 Shard Director 3 Shard Director 4

Global Services: OLTP_RW, OLTP_RO, …

Shard Catalog Shard Catalog

Shards Shards

Data Center (Region): Germany Data Center (Region): Switzerland

16.11.2016 10


All tables in one table family must be equi-partitioned (sharding key)

– Two methods to create a table family: reference partitioning or the PARENT clause

16.11.2016 11

SDB Architecture – Sharded Tables

CUSTID FIRSTNAME …

100

101

102

ORDERID CUSTID ORDERDATE …

300 100

301 101

302 102

LINEITEMID CUSTID ORDERID …

1000 100 300

1001 100 300

1002 101 301

1003 101 301

1004 102 302

CUSTOMERS

ORDERS

ORDERITEMS

CREATE SHARDED TABLE customers (...

CREATE SHARDED TABLE orders (...

Ta

ble

Fa

mily

SDB Architecture – Duplicated Tables


Used to avoid cross-shard operations

Use materialized view replication

– The master table is created in the Shard Catalog database

– R/O materialized views in all shards

– All supporting objects created automatically in SDB

– SHRD_DUPL_TABLE_REFRESH_RATE initialization

parameter controls refresh frequency

Cannot coexist in a tablespace with sharded tables

16.11.2016 12

Shard 1 Shard 2 Shard 3

CREATE DUPLICATED TABLE products(

productid INTEGER PRIMARY KEY,

...

) TABLESPACE PRODUCTS_TBS;

Shard Catalog

SDB Architecture – DDL Execution


The application schema name as well as object names are identical in all shards

DDLs are executed in the Shard Catalog database or using GDSCTL

– Automatically propagated to all shards

16.11.2016 13

SQL> CONNECT SYS@SH_CATALOG

SQL> ALTER SESSION ENABLE SHARD DDL;

SQL> CREATE USER <app_name>...

SQL> GRANT CREATE TABLE TO <app_name>...

SQL> CREATE DUPLICATED TABLE <name>...

SQL> CREATE SHARDED TABLE <name>...

Shard Catalog

GDSCTL> sql "CREATE USER ..."

GDSCTL> sql "CREATE TABLESPACE

SET ..."

...

SDB Architecture – Sharded Tables (Chunks)


Physical data distribution based on chunks

– Each table partition is stored in different tablespace

16.11.2016 14

Chunk #1 CUSTOMERS_P1

ORDERS_P1 TBS_P1

Shard 1


ORDERS_P2 TBS_P2

…


ORDERS_P7 TBS_P7 Shard 2


ORDERS_P8 TBS_P8

Chunks: 1-6

Chunks: 7-12

357913941

0

715827882

2147483646

2505397587

2863311528

SDB Architecture – Resharding (Hotspot Elimination)

Database Sharding with Oracle RDBMS 12c Release 2 - First Impressions

Adding/Removing shards or hotspot elimination requires chunk movement

– Automatically or manually

A chunk can also be spit manually within a shard

Hotspot

09.09.2016 15

Chunk R/O on

source

Chunk level 1

RMAN backup

Chunk level 0

RMAN backup

Copy files and

RMAN restore

FAN R/O

notification to

clients

File move/

restore

FAN chunk

up/down events

Drop the chunk

on the source

GDSCTL> MOVE CHUNK -CHUNK 12 -SOURCE sh01 -TARGET sh12

GDSCTL> SPLIT CHUNK -CHUNK 12


Data Sharding Methods

16.11.2016 16

Data Sharding Methods – System-Managed Sharding


Data is automatically distributed across shards using partitioning by consistent hash

– Expression: MOD(ORA_HASH('<SHARDING_KEY'),4294967295)

– Useful for data distribution

Automatic resharding in case of cardinality change

The number of chunks is defined during CREATE SHARDCATALOG and cannot be

changed later

A tablespace set needs to be created in the Shard Catalog database

SQL> CREATE TABLESPACE SET TBS_SET USING TEMPLATE

(DATAFILE SIZE 1G AUTOEXTEND ON NEXT 512M MAXSIZE UNLIMITED

EXTEND MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO);

16.11.2016 17

Data Sharding Methods – System-Managed Sharding


CREATE SHARDED TABLE customers

(

custid varchar2(20) CONSTRAINT

pk_customers PRIMARY KEY,

firstname varchar2(50),

lastname varchar2(50),

...

) TABLESPACE SET tbs_set

PARTITION BY CONSISTENT HASH (custid)

PARTITIONS AUTO;

16.11.2016 18

C001TS_SET

C002TS_SET

C003TS_SET

C004TS_SET

customers_P1

customers_P2

customers_P3

customers_P4

#Chunks=12

#Shards=3

C005TS_SET

C006TS_SET

C007TS_SET

C008TS_SET

customers_P5

customers_P6

customers_P7

customers_P8

C009TS_SET

C00ATS_SET

C00BTS_SET

C00CTS_SET

customers_P9

customers_P10

customers_P11

customers_P12

Tab

lesp

ace

Set

Shard Catalog

Shards

Data Sharding Methods – Composite Sharding


Data is first partitioned by range or list method across multiple shardspaces (set of

shards), then by consistent hash in all shards within a shardspace

16.11.2016 19

GDSCTL> ADD SHARDSPACE germany;

GDSCTL> ADD SHARDSPACE switzerland;

GDSCTL> ADD SHARD –CONNECT shdb01 –SHARDSPACE germany;

GDSCTL> ADD SHARD –CONNECT shdb02 –SHARDSPACE germany;

GDSCTL> ADD SHARD –CONNECT shdb02 –SHARDSPACE switzerland;

...

SQL> CREATE TABLESPACE SET de_tbs IN SHARDSPACE germany;

SQL> CREATE TABLESPACE SET ch_tbs IN SHARDSPACE switzerland;

Data Sharding Methods – Composite Sharding


CREATE SHARDED TABLE customers

(

custid varchar2(20),

country varchar2(2),

...

)

PARTITIONSET BY LIST (country)

PARTITION BY CONSISTENT HASH (custid)

PARTITIONS AUTO (

PARTITIONSET germany VALUES ('DE')

TABLESPACE SET de_tbs

PARTITIONSET switzerland VALUES ('CH')

TABLESPACE SET ch_tbs

...

16.11.2016 20

C001DE_TBS

C002DE_TBS

customers_P1

customers_P2

customers_P3

customers_P4

de_t

bs

Shardspace

GERMANY

Shards

C003DE_TBS

C004DE_TBS

C005CH_TBS

C006CH_TBS

customers_P5

customers_P6

customers_P7

customers_P8

ch_t

bs

Shardspace

SWITZERLAND C007CH_TBS

C008CH_TBS


Data High Availability

Disaster Recovery

16.11.2016 21

Data High Availability / Disaster Recovery – Overview


Data replication with Data Guard is a crucial component in SDB environment

– High availability, disaster recovery, read offloading

– Replication deployment performed fully automatically

– The logical unit of data replication is a shardgroup

Configuration

– Data Guard Broker with FSFO

– Local replica (HA), remote replica (DR)

– With or without ADG

16.11.2016 22

Shard Director 1 Shard Director 2

Primary Standby

Observers

Data Replication – For System-Managed Sharding


Server B

Shard 1 Shard 2 Shard 3 shgrp1

Server B

Shard 7 Shard 8 Shard 9 shgrp3

DC1

DC2

shgrp2 Shard 4 Shard 5 Shard 6

16.11.2016 23

HA

DR



GDSCTL> create shardcatalog -database shd01:1521:repo

-chunks 12 -user mygdsadmin/<pwd> -sdb shdb

-region germany,switzerland –repl DG –sharding system

-protectmode maxavailability

...

GDSCTL> add shardgroup -shardgroup shgrp1 -deploy_as primary

-region germany

GDSCTL> add shardgroup -shardgroup shgrp2 -deploy_as active_standby

-region germany

GDSCTL> add shardgroup -shardgroup shgrp3 -deploy_as active_standby

-region switzerland

16.11.2016 24

HA

DR



GDSCTL> create shard -shardgroup shgrp1 -destination sh01

-credential oracle_cred



...



...

GDSCTL> deploy

16.11.2016 25

Data Replication – For Composite Sharding


Server B

Shard 1 Shard 2 DC1

Server B

Shard 3 Shard 4

Server B

Shard 5

Shardspace GERMANY Shardspace SWITZERLAND

Shard 6

shgrp1

shgrp2

shgrp3

Shard 9 Shard 10

Shard 7 Shard 8 shgrp4

shgrp5

DC2

DC3

16.11.2016 26



GDSCTL> create shardcatalog -database shd01:1521:repo

-user mygdsadmin/<pwd> -sdb shdb –chunks 12

-region germany,switzerland

–repl DG –sharding COMPOSITE

...

GDSCTL> add shardspace –shardspace germany -PROTECTMODE

maxavailability

GDSCTL> add shardspace –shardspace switzerland -PROTECTMODE

maxavailability

16.11.2016 27



GDSCTL> add shardgroup -shardgroup shgrp1 -shardspace germany

-deploy_as primary -region germany

GDSCTL> add shardgroup -shardgroup shgrp2 -shardspace switzerland

-deploy_as active_standby -region switzerland

...

GDSCTL> add shardgroup -shardgroup shgrp4 -shardspace switzerland

-deploy_as primary -region switzerland



...

GDSCTL> deploy

16.11.2016 28


Session/Statement Routing

16.11.2016 29

Session Routing


Session routing is done based on a sharding key (super sharding key)

– SQL operations are related to the specified sharding key (shard)

– To work on another sharding key value, a new session needs to be created

OLTP_RW.TRIVADIS.COM =

(DESCRIPTION=

(CONNECT_TIMEOUT=5)(RETRY_COUNT=5)(TRANSPORT_CONNECT_TIMEOUT=3)

(ADDRESS_LIST=

(LOAD_BALANCE=ON)

(ADDRESS=(PROTOCOL = TCP)(HOST=gsm01.trivadis.com)(PORT=1530))

(ADDRESS=(PROTOCOL = TCP)(HOST=gsm02.trivadis.com)(PORT=1530))

...

(CONNECT_DATA=

(SERVICE_NAME=oltp_rw.shdb.trivadis)

([email protected])(SUPER_SHARDING_KEY=germany)

(REGION=germany))) Required

Shard Directors

16.11.2016 30

Only Composite

Sharding

Session Routing (GDS)


Shard

Director 1

Region: GERMANY Region: SWITZERLAND

Data Replication

Shard Catalog

1

16.11.2016 31

Shard

Director 2

Shard

Director 3

Shard

Director 4

[email protected]

2

3

4

Shard Catalog

Connection

Pool

1 2

3

Statement Routing/Cross-Shard Query


Client connection to the Coordinator (Catalog) Database is required

– No sharding key necessary in the connect descriptor

SQL excuted via DB-Link on Shards

– Partition and Shard pruning

Shard Catalog

16.11.2016 32

Data Replication

@<host>:<port>/GDS$CATALOG.trivadis

Shard

Director 1


Shard

Director 2

Shard

Director 3

Shard

Director 4

Global Services (GDS)

Affinity to a Local Region

– Connection requests are routed to the least-loaded database only in a local region

Affinity to a Local Region with Interregion Failover

– Connection requests are routed only to the least-loaded database in a local region

– If the global service cardinality drops to 0, client is routed to other region

DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions 16.11.2016 33

GDSCTL> add service -service oltp_ro –role physical_standby

-locality LOCAL_ONLY -region_failover –lag 20

Data Replication


Global Services: OLTP_RW, OLTP_RO, …


Conclusions

16.11.2016 34

Conclusions


Use case

– Custom OLTP applications which require extreme scalability (global data

distribution)

Existing database applications need to be adopted for database sharding

– Primarily single shard access

– With cross-shard operations performance may suffer

Existing database frameworks/features have been extended to allow for massive

data scalability and fault isolation

– Special business requirements can now also be addressed with Oracle RDBMS

– But sharding is complex and requires a lot of knowledge!

16.11.2016 35

Trivadis @ DOAG 2016

Booth: 3rd Floor – next to the escalator

Know how, T-Shirts, Contest and more

We look forward to your visit

Because with Trivadis you always win !

36 DOAG Conference 2016: Database Sharding with Oracle RDBMS - First Impressions 17.11.2016

Database Sharding with Oracle RDBMS - First Impressions · –Troubleshooting/Performance Tuning...

Documents

Transcript of Database Sharding with Oracle RDBMS - First Impressions · –Troubleshooting/Performance Tuning...