OracleDatabaseNaveSharding...Oracle Database Native Sharding: a Customer Perspective ©2016...

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Oracle Database NaBve Sharding

Nagesh BaEula [email protected] Oracle Sharding Product Development Feb 16, 2017

Technical Overview


Safe Harbor Statement The following is intended to outline our general product direcBon. It is intended for informaBon purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funcBonality, and should not be relied upon in making purchasing decisions. The development, release, and Bming of any features or funcBonality described for Oracle’s products remains at the sole discreBon of Oracle.

2


Program Agenda

Sharding Overview

Sharded Database and Schema CreaBon

Data Dependent RouBng

MAA Benchmark Results -‐ Sharding on Bare Metal Cloud

Customers’ PerspecBve

CompeBBve

1

2

3

4

3

5

6


Program Agenda

Sharding Overview





CompeBBve

1

2

3

4

4

5

6


• Horizontal parBBoning of data across up to 1000 independent Oracle Databases (shards)

•  Shared-‐nothing hardware architecture – Each shard runs on commodity server – No shared storage – No clusterware

• Data is parBBoned using a sharding key (i.e. account_id)

5

Oracle Sharding – An ElasBc Database Architecture

A single logical DB sharded into N physical Databases

Shard01: Table 1

ParBBons

B A

D C

ParBBons

F E

H G

Shard02: Table 1

Shardnn: Table 1

ParBBons

… …

… …

Server A

Server B

Server N

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 6

Oracle Database Sharding – Benefits

Linear Scalability

Add shards online to increase database size and throughput. Online split and rebalance.

Extreme Availability

Shared-‐nothing hardware architecture. Fault of one shard

has no impact on others.

Geographic DistribuJon

User defined data placement for performance, availability, DR or to meet regulatory requirements.

…

…

…

…


Shard Catalog/Coordinator

App Tier

RouBng Tier

Data Tier

7

CapabiliBes of Oracle NaBve Sharding

ConnecBon Pool

Shard Directors

Sharding key

…

…

•  Auto deployment of up to 1000 shards –  Supports Data Guard and Oracle GoldenGate

•  MulBple sharding methods –  System Managed (Consistent Hash)

•  Uniform data distribuBon across shards à linear scalability

–  Composite (Range/List-‐Consistent Hash) •  Geo-‐distribuBon plus linear scalability

•  Centralized schema maintenance –  NaBve SQL for sharded and duplicated tables

•  RouBng –  Direct rouBng from ConnecBon pools –  Proxy rouBng for MulB-‐shard queries

•  Geo-‐distribuBon for data proximity and sovereignty •  Supports on-‐premises, cloud or hybrid deployments •  Online scale-‐out with automaBc rebalance or scale-‐back •  Shards may run different Oracle releases


Oracle Database 12c Release 2 Offers Choice

Oracle Sharding

…

• Single logical database • Supports OLTP applicaBons designed to shard • Single physical database

• Supports any applicaBon

N Physical Databases

Oracle RAC

…


Sharding – a Different Way to Scale FricJonless linear scaling due to zero shared hardware or soPware

# of Shards

TPS

9

0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

50 100 150 200


Fault IsolaJon Sharding for Extreme Data Availability

The porBon of users who undergo brown-‐out at one Bme by an unplanned outage or database upgrade in a

sharded database with 100 shards

1% 10


Sharding – a Superior Way to Distribute Data

11

Data Sovereignty and Data Proximity

Customers Americas Customers Asia Customers Europe

Sharded Database Linear Scalability & GeodistribuBon


Sharding with Exadata

12

Data Sovereignty Sharded Database

GeodistribuBon for Data Sovereignty & Privacy Compliance

Ø  Fast, direct access from within each region

Ø  Queries access data from all regions as a single logical database

Customers Americas Customers Asia Customers Europe


Sharding – a Superior Way to Deploy Database

13

Flexible On-‐Premises or Cloud Deployment

On-‐Premises Cloud Hybrid


Automated Data Availability ConfiguraBon

14

AcBve Data Guard with Fast-‐Start Failover

GoldenGate ‘chunk-‐level’ acBve-‐acBve replicaBon with automaBc conflict detecBon/resoluBon (OGG 12.3)

ParBBon_F

ParBBon_F

ParBBon_A ParBBon_C

OpBonally – complement replicaBon with Oracle RAC for server HA


AcBve-‐AcBve Sharded Database – Oracle GoldenGate 12.3

15

•  AutomaBc GoldenGate deployment –  Chunk-‐level replicaBon within and across shardgroups

•  AutomaBc CDR (conflict detecBon & resoluBon) to simplify acBve-‐acBve replicaBon –  Including insert-‐delete conflicts, JSON columns, LOB updates

•  Shardgroups can have different number of shards, replicaBon factor, database versions and H/W plaqorms

#Shards = 50; ReplicaBon Factor = 3 Shardgroup 1: Availability_Domain1

#Shards = 20; ReplicaBon Factor = 2

......

Shardgroup 2: Availability_Domain2

......

Oracle GoldenGate replicaJon within and across Shardgroups


Data Modeling Requirements

• Hierarchical table family that consists of a root table and many child and grandchild tables

•  Every table in the table family must contain the sharding _key – Sharding _key must be the leading column of the primary key of the root table

•  The sharding method and sharding_key are determined based upon applicaBon and business requirements

•  IdenBfy tables that need to be duplicated on all shards – Common reference data (e.g. Products)

16

Table Family and Duplicated Tables


Direct RouBng •  OLTP workloads that require linear scale, high performance, and fault isolaBon must specify a sharding_key during connect – All data required for a transacBon is contained within a single shard

Proxy RouBng •  The following workloads must be able to accept lesser SLAs for linear scale, performance, and fault isolaBon than OLTP transacBons – ReporBng workloads – MulB-‐shard queries – ApplicaBons unable to specify a sharding_key

Database ConnecBons •  Must use Oracle integrated connecBon pools (UCP, OCI, ODP.NET, JDBC)

•  Must be able to separate workloads that use Direct RouBng from those that use Proxy RouBng – Each uses separate connecBon pools

ApplicaBon Requirements

17


Program Agenda

Sharding Overview





CompeBBve

1

2

3

4

18

5

6


Deployment of a System-‐Managed SDB with Data Guard

19

…

…

Clients

Data Guard Fast-‐Start Failover

Region Availabilty_Domain1

Shard Catalog shardcat_stdby

Shard Director shdir3,4

Region Availability_Domain2

Shardgroup shgrp2

Shardgroup shgrp1 Shard Director

shdir1,2 Shard Catalog

shardcat

ConnecBon Pools

ConnecBon Pools

Primaries

HA Standbys


DeclaraBve SpecificaBon of SDB ConfiguraBon

20

Primary components create shardcatalog -‐database <shardcat host>:1521:shardcat -‐user sdb_admin/passwd_sdb_admin add gsm –gsm sharddirector1 -‐listener 1571 -‐pwd passwd_gsmcatuser -‐catalog <shardcat host>:1521:shardcat .. add credenBal -‐credenBal oracle_cred -‐osaccount oracle -‐ospassword < > add shardgroup -‐shardgroup shgrp1 -‐deploy_as primary -‐region Availability_Domain1 create shard -‐shardgroup shgrp1 -‐desJnaJon <host1> -‐credenJal oracle_cred …

Duplicated components for HA deployed in separate availability domain add gsm –gsm sharddirector2 -‐listener 1571 -‐pwd passwd_gsmcatuser -‐catalog <shardcat host>:1521:shardcat .. add shardgroup -‐shardgroup shgrp2 -‐deploy_as acBve_standby -‐region Availability_Domain2 create shard -‐shardgroup shgrp2 -‐desJnaJon <host11> -‐credenJal oracle_cred … deploy add service -‐service oltp_rw_srvc -‐role primary


Sharded Tables

Shard 1 Shard 2 Shard N

… … …

21

Schema CreaBon – Sharded and Duplicated Tables

Customers Orders Line Items

Products

Database Tables

Sets of related parBBons of customers, orders & line items placed on a given tablespace[s]

Duplicated Tables


Map Related ParBBons to a Chunk

•  In simplest form, each chunk is a tablespace containing related parBBons of the sharded tables, i.e. Chunk#1 contains Customers_P1, Orders_P1, etc – A chunk can also be a set of tablespaces that store related parBBons – a separate tablespace for each parBBon of the table

• All data perBnent to a given sharding key resides in a single chunk

22

Chunk is the Unit of Data Movement in a Sharded Database

Customers_P1 Orders_P1 Lineitems_P1 Chunk #1 Related ParBBons of Sharded Tables


Each Shard Contains MulBple Chunks

Lineitems_P6

23


System Managed Sharding • Based on parBBoning by CONSISTENT HASH • Range of hash values assigned to each chunk • Data is uniformly sharded / re-‐sharded automaBcally • Many relaBvely small equally sized chunks

+ Linear Scalability + AutomaBc balanced data distribuBon -‐ User has no control on locaBon of data

24

Ø  Customer facing applicaBons sharded by customer_id

Ø  For linear scalability


Tablespace Set

25

Shard1

Tablespace tbs1-‐120


Shard2


Tablespace tbs1-‐121 …. ….

Tablespace Set tbs1

….



Shard3

•  CREATE TABLESPACE SET tbs1;!–  Automates creaBon of tablespaces and distributes them among shards –  Number of tablespaces = Number of chunks

•  Supports all tablespace clauses


CreaBng a Sharded Table Family with ReferenBal Integrity

CREATE DUPLICATED TABLE Products ( !ProductId INTEGER PRIMARY KEY, !Name VARCHAR2(128), !LastPrice NUMBER(19,4),!… !)!TABLESPACE products_tsp ;!

Execute DDLs on the Shard Catalog !CREATE TABLESPACE SET tbs1 ;!!CREATE SHARDED TABLE Customers!( CustId VARCHAR2(60) NOT NULL, ! FirstName VARCHAR2(60), ! LastName VARCHAR2(60),! …! CONSTRAINT pk_customers ! PRIMARY KEY(CustId)!)!PARTITION BY CONSISTENT HASH (CustId) !PARTITIONS AUTO !TABLESPACE SET tbs1 ;!

CREATE SHARDED TABLE Orders ( ! OrderId INTEGER,! CustId VARCHAR2(60),! OrderDate TIMESTAMP,! …! CONSTRAINT pk_orders ! PRIMARY KEY (CustId, OrderId),! CONSTRAINT fk_orders_parent ! FOREIGN KEY (CustId) REFERENCES Customers(CustId) !)!PARTITION BY REFERENCE (fk_orders_parent) ;!

AutomaBcally maps parBBons to the tablespaces that were created


Oracle Sharding for Document Store ApplicaBons

27

Supports Schemaless ApplicaJon Development with JSON

CREATE TABLESPACE SET tbs1 ;!!CREATE SHARDED TABLE Customers!( CustId VARCHAR2(60) NOT NULL, ! CustProfile CLOB, ! CONSTRAINT pk_customers PRIMARY KEY(CustId),! CONSTRAINT cons_json CHECK (CustProfile IS JSON)!)!PARTITION BY CONSISTENT HASH (CustId) !PARTITIONS AUTO !TABLESPACE SET tbs1 ;!

• Full power of SQL to JSON document stores plus linear scalability of SDB


Composite Sharding

•  Provides two-‐levels of sharding •  The sharded database is divided into N sets of shards called shardspaces –  Data is parBBoned across shardspaces by LIST or RANGE on super-‐sharding key (e.g. geography)

– Within each shardspace, data is parBBoned across shards by CONSISTENT HASH using sharding key (e.g. customer id)

+ Best pracBce for geo-‐distribuBon or hybrid clouds with linear scalability

-‐ Requires two sharding keys: super_sharding_key and sharding_key

28

Ø  Billing system sharded by geo then by account_id

Ø  For geo-‐distribuBon and linear scaling

Customers Asia

Sharded Database Linear Scalability in each Geography

Customers Americas


Concept: Shardspace • A set of shards that stores data corresponding to a range or list of key values – The super_sharding key is a column other than the sharding_key (e.g. Geo), that defines the number of sets

– The sharding_key is used to maintain automaBc data distribuBon across shards within a group using consistent hash

•  Each shardspace includes 1 or more replicated shardgroups for HA/DR

29


Composite Sharding – Example

30

sh3

Tablespace tbs2-‐60 P_60


…. ….

sh4



sh5



…. ….

sh1





…. ….

sh2 ADD SHARDSPACE –SHARDSPACE Americas –CHUNKS 120; CREATE SHARD –SHARDSPACE Americas -DESTINATION <host1> -CREDENTIAL oracle_cred; CREATE SHARD –SHARDSPACE Americas -DESTINATION <host2> -CREDENTIAL oracle_cred; DEPLOY; CREATE TABLESPACE SET tbs1 IN SHARDSPACE Americas; ADD SHARDSPACE –SHARDSPACE Asia –CHUNKS 180; !CREATE SHARD –SHARDSPACE Asia -DESTINATION <host3> -CREDENTIAL oracle_cred; CREATE SHARD –SHARDSPACE Asia -DESTINATION <host4> -CREDENTIAL oracle_cred; CREATE SHARD –SHARDSPACE Asia -DESTINATION <host5> -CREDENTIAL oracle_cred; DEPLOY; CREATE TABLESPACE SET tbs2 IN SHARDSPACE Asia;

Tablespace Set tbs1

Tablespace Set tbs2

Shardspace Americas

Shardspace Asia


Map Ranges of Sharding Keys to ParJJons Composite Sharding -‐ Example

sh3



…. ….

sh4



sh5



…. ….

sh1





…. ….

sh2

Tablespace Set tbs1

Tablespace Set tbs2

Shardspace Americas

Shardspace Asia

CREATE SHARDED TABLE customers ( CustId VARCHAR2(60) NOT NULL, .. Geo VARCHAR2(8), CONSTRAINT cust_pk PRIMARY KEY(CustId) ) PARTITIONSET BY LIST (Geo) PARTITION BY CONSISTENT HASH (CustId) PARTITIONS AUTO (PARTITIONSET Americas VALUES (‘Americas’) TABLESPACE SET tbs1, PARTITIONSET Asia VALUES (‘Asia’) TABLESPACE SET tbs2) ;


Program Agenda

Sharding Overview





CompeBBve

1

2

3

4

32

5

6


RouBng in an Oracle Sharded Environment • Direct RouBng based on sharding_key

– For OLTP workloads that specify sharding_key (e.g. customer_id) during connect •  A transacBon happens on single shard

– Enabled by enhancements to mid-‐Ber connecBon pools and drivers •  JDBC/UCP, OCI, and ODP.NET recognize sharding keys

– PHP, Python, Perl, and Node.js leverage OCI support for sharding

• Proxy RouBng via coordinator (shard catalog) – For workloads that cannot specify sharding_key (as part of connecBon) – Applies to reporBng, batch jobs – Queries can span 1 or more or all shards

•  Performed in parallel across shards (e.g. aggregates on sales data)

33


UCP: ConnecBon Request with a Shard Key

•  Pool creaBon: no Java applicaBon code change •  Shard-‐aware applicaBon gets a connecBon, using UCP Sharding APIs OracleShardingKey keyMaryEmail =

! ! ! pds.createShardingKeyBuilder() ! ! ! .subkey("[email protected]", OracleType.VARCHAR2)

.build(); !

Connection connection = ! ! ! pds.createConnectionBuilder()

.shardingKey(keyMaryEmail) .build(); !

34

An Example of New Sharding APIs


Session-‐Based RouBng via Sharding Key

•  Sharding key is provided by the applicaBons at connecBon checkout

•  Shard Director looks up the sharding key and redirects client to the correct shard containing the data

•  Client executes SQL directly at shard

•  A{er all transacBons for the given sharding key are executed, applicaBon must check-‐in the connecBon to the pool and check-‐out a connecBon for another key

ApplicaBon Server

Shard Directors


Fast Path for key-‐based access App Tier

RouBng Tier

Data Tier

Direct RouBng via Sharding Key

•  ConnecBon pool maintains the shard topology cache •  Upon first connecBon to a shard

–  ConnecBon pool retrieves all sharding key ranges in the shard

–  ConnecBon pool caches the key range mappings

•  DB request for a key that is in any of the cached key ranges goes directly to the shard (i.e., bypasses shard director)

36

ApplicaBon Server

Shard Directors


Coordinator (shard catalog)

Proxy RouBng

•  ApplicaBons connect to Query coordinator/Shard Catalog •  Single Shard Queries

–  SELECT CUSTID, FIRSTNAME, LASTNAME FROM CUSTOMERS WHERE CUSTID=1001;!

–  Coordinator parses SQL, extracts sharding key and routes query to the correct shard

37

For Workloads that cannot pass sharding key during connecJon check-‐out

ApplicaBon Server

Shard Directors


UCP Sample Code

38

1.  Create a connecBon pool by passing DB_URL, DB_USER, DB_PASSWORD for a sharded database.

2.  Build and Pass the sharding key for a connecBon

3.  Perform any Query/DML on the customer record

UCP transparently handles failover and runBme load balancing

Make sure to include ojdbc8.jar, ucp.jar and ons.jar in the classpath


ConnecBon URL for a sharded database

39

jdbc:oracle:thin:@ (DESCRIPTION=(FAILOVER=on)(CONNECT_TIMEOUT=5) (TRANSPORT_CONNECT_TIMEOUT=3)(RETRY_COUNT=3) (ADDRESS_LIST=(LOAD_BALANCE=ON) (ADDRESS=(HOST=orclgsm1.oracle.com)(PORT=1571)(PROTOCOL=tcp)) (ADDRESS=(HOST=orclgsm2.oracle.com)(PORT=1571)(PROTOCOL=tcp))) (ADDRESS_LIST=(LOAD_BALANCE=ON) (ADDRESS=(HOST=orclgsm3.oracle.com)(PORT=1572)(PROTOCOL=tcp)) (ADDRESS=(HOST=orclgsm4.oracle.com)(PORT=1572)(PROTOCOL=tcp))) (CONNECT_DATA=(SERVICE_NAME=oltp_rw_srvc.cust_sdb.oradbcloud)(REGION=region1)))

In the URL, specify the Shard Director listener endpoints


Step 1: Create a connecBon pool

40

// Get the PoolDataSource for UCP PoolDataSource pds = PoolDataSourceFactory.getPoolDataSource(); Connection connection; // Set the connection factory first before all other properties pds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource"); pds.setURL(DB_URL); pds.setUser(DB_USER); pds.setPassword(DB_PASSWORD); pds.setConnectionPoolName("UCP_SHARDING_POOL"); // Default is 0. Set the initial number of connections created when UCP is started. pds.setInitialPoolSize(5); // Default is 0. Set the minimum number of connections maintained by UCP at runtime. pds.setMinPoolSize(5); // Default is Integer.MAX_VALUE (2147483647). Set the maximum number of connections allowed on the connection pool. pds.setMaxPoolSize(20); // set max connection per shard pds.setMaxConnectionsPerShard(10);!

Set the ConnecBon Pool parameters


Step 2: Build Sharding Key for a connecBon

41

try { // Loop through this for 5 customers for (int i=0 ; i < 5 ; i++ ) { String email = emailIds[i]; // Build the Sharding key by passing the email id, which is the sharding key OracleShardingKey shardKey = pds.createShardingKeyBuilder() .subkey(email, OracleType.VARCHAR2) .build(); // Get the connection by passing the sharding key connection = pds.createConnectionBuilder()

! ! ! !.shardingKey(shardKey) .build(); // Perform a database operation doSQLWork(connection, email); } } catch (SQLException e) { System.out.println("UCPShardingSample - SQLException occurred : " + e.getMessage()); }!

Build the Sharding key

Check out a connecBon to a given shard by specifying the sharding key


Step 3: Perform Query/DML on the customer record

42

/* This method shows the customer id, first name and last name */ public static void doSQLWork(Connection conn, String email) { try { // Prepare a statement to execute the SQL Queries. PreparedStatement prepStatement = conn.prepareStatement("SELECT CUSTID, FIRSTNAME, LASTNAME FROM CUSTOMERS WHERE CUSTID=?"); prepStatement.setString(1, email); ResultSet rsCust = prepStatement.executeQuery(); while (rsCust.next()) { System.out.print("Customer Id" + rsCust.getString("CUSTID")); System.out.print("First Name:" + rsCust.getString("FIRSTNAME")); System.out.println("Last Name:" + rsCust.getString("LASTNAME")); } } catch (SQLException e) { System.out.println("UCPShardingSample - “ + "doSQLWork()- SQLException occurred : " + e.getMessage()); } }!

Once connected, session scope is the shard


Coordinator (shard catalog)

Proxy RouBng

•  ApplicaBons connect to Query coordinator/Shard Catalog •  Single Shard Queries

–  E.g. SELECT CUSTID, FIRSTNAME, LASTNAME FROM CUSTOMERS WHERE CUSTID=1001;!

–  Coordinator parses SQL, extracts sharding key and routes query to the correct shard

•  MulB-‐Shard Queries –  E.g. SELECT GEO, CLASS, COUNT(*) FROM CUSTOMERS GROUP BY GEO, ROLLUP(CLASS);

–  Coordinator rewrites the query to do most processing on the shards

–  Supports shard pruning and scaEer-‐gather –  Final aggregaBon performed on the coordinator

43

For Workloads that cannot pass sharding key during connecJon check-‐out

ApplicaBon Server

Shard Directors


Example : MulB-‐shard Query •  ROLLUP query to get region-‐wise number of customers in each class

•  SELECT geo, class, COUNT(*) FROM Customers GROUP BY geo, ROLLUP(class); !•  Query is executed on all shards

–  First level aggregaBon and group by is pushed on all the shards –  Coordinator does the second and final level of aggregaBon on the rows returned by all shards before returning results to client


Shard Director

Centralized Schema Management

45

Shard 1

Shard 2

Shard n

Shard Catalog

connect to GDS$CATALOG service alter session enable shard ddl; create tablespace set … create user ... create sharded table … .. Create duplicated table …


Duplicated Table Maintenance

Read-‐Only Materialized Views Master Table

Shard 1

Shard 2

Shard 3

Shard Catalog


Online AddiBon and Rebalancing of Shards

47

•  A chunk is the unit of data movement when resharding

•  Chunks are automaBcally rebalanced when a new shard is added

•  Uses RMAN Incremental Backup & Transportable Tablespace

Add shard


Online AddiBon and Rebalancing of Shards Auto

Rebalance •  A chunk is the unit of data movement when resharding

•  Chunks are automaBcally rebalanced when a new shard is added

•  Uses RMAN Incremental Backup & Transportable Tablespace


Lifecycle Management of SDB •  DBA can manually move or split a chunk from one shard to another • When a new shard is added, chunks are automaBcally rebalanced

–  Uses RMAN Incremental Backup & Transportable Tablespace

•  ConnecBon pools get noBfied (via ONS) about split, move, add/remove shards, auto-‐resharding, read-‐only –  ApplicaBon can either reconnect or access read-‐only

•  All MAA pracBces apply for BUR of shards and shard catalog with special consideraBons

•  Can patch all shards with one command via opatchauto

•  EM supports monitoring & management of SDB

49


Program Agenda

Sharding Overview





CompeBBve

1

2

3

4

50

5

6


•  ObjecBve –  ElasBcally scale-‐out the Shared Database

•  Demonstrate linear scalability of RelaBonal transacBons

–  Highest Availability with MAA on Oracle Bare Metal Cloud

•  Infrastructure –  Each shard hosted on dedicated server

51

Oracle Sharding Demo on Oracle Bare Metal Cloud

Oracle Bare Metal Cloud

Availability Domain 1 (100 Primary Shards)

Availability Domain 2 (100 HA Standby Shards)

Shard-‐level AcBve Data Guard (Max Availability) Component Resources per shard

CPU 36 Cores

Memory 512 GB

Flash 12.8 TB NVMe

Network 10 GbE

Demo Topology


Sharding – a Different Way to Scale FricJonless linear scaling due to zero shared hardware or soPware

# of Shards

TPS

52

0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

50 100 150 200


Oracle Sharding Demo on Oracle Bare Metal Cloud

•  ElasBcally scaled-‐out to 200 shards on Oracle Bare Metal Cloud – Demonstrated linear scalability of RelaBonal transacBons – Demonstrated 11 Million transacBons per sec that includes:

•  4.5 Million Read-‐Write TransacBons per sec across all 100 Primary shards •  6.5 Million Read-‐Only TransacBons per sec across all 100 AcBve Standby shards

• MAA Sharding provides highest availability – Each shard is protected by Data Guard Fast-‐Start Failover across Availability Domains – Single Shard Failure resulted in 100% availability for 99% of the applicaBon

•  1% of the applicaBon experienced only 15 seconds blackout

Conclusions


Shard Catalog Outage TesBng

54

Shard Catalog Outage Has Zero Impact on Availability for OLTP

•  Outage of shard catalog has no effect on applicaBon performance (Direct RouBng)

•  Ranges of sharding keys are cached within the connecBon pools

•  OLTP TransacBons use direct rouBng, completely bypassing the shard catalog

•  MAA Best PracBce is to protect catalog with Data Guard Maximum Availability


Shard Directors Outage TesBng

55

Shard Director Outage Has Zero Impact on Availability

•  Outage of shard directors does not affect a running connecBon pool

•  ConnecBon pool caches ranges of sharding keys / shards

•  MAA best pracBce to have 3 shard directors per region


Each Shard is a Physically Separate Oracle Database Shard Outage has Zero Impact on Surviving Shards

ApplicaBon Throughput

ApplicaBon remains available on other shards

Shard Outage

AutomaBc failover quickly restores service for the shard impacted by outage


•  Order Entry Service runs on Primary

•  ReporBng Service runs on Standby

Role Based Global Services for AcBve Data Guard

57

AD 1 AD 2

Standby ReporJng Service

Primary Order Entry Service

AcBve Data Guard


•  Order Entry Service runs on Primary

•  ReporBng Service runs on Standby

•  Upon Data Guard role change, Shard Directors failover services based on Role

Role Based Global Services for AcBve Data Guard

Data Center 2

58

AD 1 AD 2

Primary Order Entry Service

Standby ReporJng Service

AcBve Data Guard

GDSCTL>add service -service order_entry_service –preferred_all –role PRIMARY GDSCTL>add service -service reporting_service –preferred_all –role PHYSICAL_STANDBY


MiBgate Unplanned Shard Outage

59

AutomaJc Failover of Client Workload to another Availability Domain

ApplicaJons using …

Oracle pools or drivers : UCP, ODP.NET, OCI, WebLogic AcJve GridLink

3rd party App Servers using UCP: IBM WebSphere, Apache Tomcat

ApplicaBon config

•  Subscribe to FAN events (published by Shard Directors via ONS) by enabling Fast ConnecBon Failover (FCF)

•  TNS entry to include RETRY_COUNT, CONNECT_TIMEOUT and TRANSPORT_CONNECT_TIMEOUT

Unplanned events For AcBve Data Guard: Data Guard role change

Sessions Drain

FAN posts unplanned downBme event and FCF ensues.

•  New work is redirected by Shard Directors immediately

•  Idle sessions are released immediately

•  Checked out connecBons receive invalid connecBon; ApplicaBon closes the connecBon and gets new one from another database in the pool

AD 2 AD 1

Oracle Integrated Clients

FAN Events (RLB, Service Up/Down) via ONS


Program Agenda

Sharding Overview





CompeBBve

1

2

3

4

60

5

6

Oracle Database Native Sharding: a Customer Perspective

©2016 PayPal Inc. Confidential and proprietary.

John Kanagaraj, Sr. Member of Technical Staff, PayPal Core Data Platform

61

©2016 PayPal Inc. Confidential and proprietary. 62

Sharding Principles

•  Sharding is mostly a Scalability play •  ”Key is Key” – Choose one STRONG key and align. E.g. Account/User ID •  Data model changes (drastically!) – Normalization is relaxed •  Cross-key transaction boundaries are broken •  Non-shard key access can be expensive; forces “scatter-gather” pattern •  CAP Theorem: 2 of 3 for (C)onsistency, (A)vailability and Network (P)artition •  “Lookup” requirement for Common data elements •  Scheme for mapping logical to physical is critical for future scale-out •  Joins and ACID principles usually not available in Sharded systems (e.g. NoSQL)

•  Oracle Sharding supports the sharding principles while providing consistency, transactions, and relational capabilities such as Joins

©2016 PayPal Inc. Confidential and proprietary. 63

Oracle Sharding – Feedback from Beta software

•  What we liked •  Auto-configuration (Create and Manage Shards and Routing) •  Out of box support for Shard Catalog, Routing layer, Duplicated Tables •  Cross-shared query support (Not available with custom sharding) •  Familiar SQL for sharding management •  Two level sharding (not available in other technologies such as NoSQL) •  Consistent hashing to prevent large data movement as new shards are added •  Dynamically changing the number of shards

•  Things to be careful of •  Sharding requires major rework in Data Model and Data Design •  Duplicated Table – Make sure not to over-use; Not for write-heavy use cases •  Enable connection pool limits per Shard


Sharding Use Case – LinkedIn

•  Target applicaBon –  LinkedIn uses home-‐grown sharding with Oracle Database to scale customer facing applicaBons

•  Challenges faced/key business driver – Manual re-‐configuraBon during rebalancing and new table addiBons –  No mulB-‐shard queries or composite sharding capabiliBes

•  How does Oracle Sharding addresses LinkedIn’s requirement? –  Online addiBon of shards and automaBc rebalance with no impact to applicaBon –  Architectural simplicity –  Composite sharding and mulB-‐shard queries –  Efficient use of CAPEX

64

Business-‐oriented Social Networking with 380 Million Members in Over 200 Countries

ORACLE SHARDING

Home Grown Sharding Architecture: LinkedIn

©2015 LinkedIn Corporation. All Rights Reserved. 65

SHARD1 SHARD2 SHARD3 SHARD4 SHARD5

ORACLE - DB01

ORACLE - DB03

ORACLE - DB02

……..

1-2,000,000 2 MM – 4 MM 4 MM – 6 MM 6 MM – 8 MM 8 MM – 10 MM

…..

txMgr txMgr txMgr txMgr txMgr

APPLICATION

ShardLocator shard.config

ORACLE SHARDING

Where LinkedIn Wants To Be - Oracle 12c Sharding

©2015 LinkedIn Corporation. All Rights Reserved. 66

….. …..

Shard Group 1 - Rep factor - 1

Data Guard Standby

…..



APP

APP

APP

…..

…..

Oracle Database Native Sharding

Sudhi Vijayakumar Database Systems Architect, Facebook Inc

Target Sharded Architecture

Various Data Sources

Shard Group 1 Shard Group 2

Primary DC

Shard Group 3

Secondary DC

Benefits of Oracle Sharding •  Full power of Oracle RDBMS

•  Deployments can be made automatic

•  Open Compute Platform can be used for deployments

•  Horizontal scaling based on demand/growth

•  Reduce SPOF

•  Reduce tech debt (apply patches in rolling manner, etc.)

•  Uses familiar technology (Data Guard, GoldenGate)

Key Metrics for Oracle Sharding POC CPU

REG

ION

•  Support Facebook’s existing and growing ingestion rate

•  Response times for queries

•  Ease of Deployment

•  Ease of Maintenance

•  Iterative design changes


Program Agenda

Sharding Overview





CompeBBve

1

2

3

4

71

5

6


Oracle NaBve Database Sharding

• MulB-‐modal database – RelaBonal, JSON, XML, etc. •  SQL and programmaBc interfaces -‐ PL-‐SQL, OCI, JDBC, etc. •  Schema flexibility & developer agility with JSON within Oracle Database • BeEer consistency than NoSQL databases •  Easier applicaBon maintenance – schema in database instead of applicaBon

•  Enterprise-‐class features of a mature DBMS •  Leverage in-‐house and world-‐wide Oracle DBA skillset • Plus superlinear scalability & extreme availability

72

Combines the best of RDBMS and NoSQL


Scalability and Fault IsolaJon with Enterprise QualiJes of Oracle RDBMS Oracle Sharding vs. NoSQL Data Stores

NoSQLs Oracle Sharding

Linear scalability and fault isolaBon

Deploy on commodity hardware on premises or on Cloud

Developer agility (e.g. flexible schema, JSON)

ReplicaBon

RelaBonal/SQL – joins, transacBons, no pre-‐defined queries, mulB-‐shard queries

ACID within shard, MVCC & strong consistency

Enterprise class features of mature DBMS (parBBoning, compression, security)

Supports MulB-‐core over SMP (Shard-‐level)

Leverage in-‐house and world-‐wide Oracle DBA skillset


Scalability and Fault IsolaJon with Enterprise QualiJes of Oracle RDBMS

Oracle Sharding vs MySQL Fabric

MySQL Fabric Oracle Sharding

System Managed Sharding

Range based Sharding

List Based Sharding

Composite Sharding

ACID within shard

MulB-‐shard queries

AcBve-‐AcBve replicaBon between shards (OGG 12.3)

GUI support for Management


NaJve Sharding Capability vs Client Library and External Tools Oracle Sharding vs Azure ElasBc Scale

Azure Oracle Sharding

System Managed Sharding

User Defined Sharding

Composite Sharding

Deployment on-‐premises

NaBve integraBon with database automates all DDLs operaBons

ACID within shard

MulB-‐shard queries support parBal results

Shards remain online while chunks are being moved

Cloud tooling for shard deployment

AcBve-‐AcBve replicaBon between shards (OGG 12.3)


Oracle Sharding 12.2.0.1 Core Features

76

Support Matrix

Sharding Method

Supported in 12.2.0.1?

Direct RouJng Proxy RouJng AcJve Data Guard

Oracle GoldenGate

Enterprise Manager

System Managed

Yes Yes Yes Yes Yes (Requires Oracle GoldenGate 12.3)

Yes (EM 13.2.2.0 Database Plug-‐in)

Composite Yes Yes No Yes No (Planned for Oracle Database 12.2.0.2)

Yes (EM 13.2.2.0 Database Plug-‐in)

User-‐defined No


Oracle Sharding 12.2.0.1 Licensing -‐ Summary

77

Licensing Availability

On-‐Premises For a sharded database (SDB) with 3 or fewer primary shards, Sharding is included with EE (includes Data Guard). No limit on number of standby shards For an SDB with more than 3 primary shards, in addiBon to EE, all shards must be licensed for either AcBve Data Guard, GoldenGate or RAC

March 15, 2017 Ref: External link

Oracle Cloud (PaaS) DBCS EE and DBCS EE-‐HP: Use is limited to three primary shards; No limit on number of standby shards DBCS EE-‐EP and ECS: No limit on number of primary shards or standby shards Ref: Licensing InformaBon Manual

Automated Deployment -‐ 2nd Half of CY2017 Till then – PaaS customers will be able to deploy Sharding on DBCS instances using a White Paper

Oracle Cloud (IaaS) BYOL March 15, 2017


Summary • Complete plaqorm for sharding an Oracle database •  Ideal for mission-‐criBcal transacBon processing that require extreme level of scalability and availability than possible with a single physical database

78

OracleDatabaseNaveSharding...Oracle Database Native Sharding: a Customer Perspective ©2016...

Documents

Transcript of OracleDatabaseNaveSharding...Oracle Database Native Sharding: a Customer Perspective ©2016...

Oracle*Database*Nave*Sharding*...Oracle Database Native Sharding: a Customer Perspective ©2016...

Documents

Transcript of Oracle*Database*Nave*Sharding*...Oracle Database Native Sharding: a Customer Perspective ©2016...

OracleDatabaseNaveSharding...Oracle Database Native Sharding: a Customer Perspective ©2016...

Transcript of OracleDatabaseNaveSharding...Oracle Database Native Sharding: a Customer Perspective ©2016...