Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics
Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics
-
Upload
continuent -
Category
Technology
-
view
1.199 -
download
2
description
Transcript of Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics
©Continuent 2014
Replicate Oracle to Oracle, Oracle to MySQL ,
and Oracle to Analytics
Linas Virbalas, Senior Software Engineer Robert Hodges, CEO
©Continuent 2014
Introducing Continuent
���2
• The leading provider of clustering and replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance scaling and data management for MySQL
• Replication - Flexible, high-performance data movement
©Continuent 2014
What is Tungsten Replicator?
���3
Tungsten Replicator: a high-performance
open source database replication engine
!GPL V2 license - 100% open source
Download from https://code.google.com/p/tungsten-replicator/ Annual support subscription available from Continuent
“GoldenGate without the Price Tag”®
©Continuent 2014
Tungsten Replicator Overview
���4
Master
(Transactions + Metadata)
Slave
THL
DBMS Logs
Replicator
(Transactions + Metadata)
THLReplicator
Download transactions via network
Apply using JDBC
©Continuent 2014 ���5
Extract Filter Apply
StageExtract Filter Apply
StageExtract Filter Apply
Stage
Pipeline
Master DBMS
Transaction History Log
QueueSlave
DBMS
Tungsten Replication Service
©Continuent 2014
Filters and Parallel Apply
���6
Extract Filter Apply
StageExtract Filter Apply
StageStage
Pipeline
Master DBMS
Transaction History Log
In-Memory Queue
Slave DBMS
Extract Filter ApplyExtract Filter ApplyExtract Filter Apply
©Continuent 2014
Multiple Services per Replicator
Service frommysql
Service fromoracle
frommysql
fromoracle
Replicator
Replicator
Replicator
Replicator
Aggregated
©Continuent 2014
star
master-slave Heterogeneous
fan-in slave all-masters
MySQL
Oracle
Oracle
MySQL Oracle
Oracle
MySQL MySQL
©Continuent 2014©Continuent 2014
Heterogeneous Replication
©Continuent 2014©Continuent 2014
Steps to Heterogeneous Replication
1. Prepare (translate) schema for the slave DBMS
2. Set up replication
3. Provision initial data
4. Continue real-time replication
ddlscan
tpm
Parallel ApplyMySQL Blackhole
Tungsten Replicator
Parallel Extract
©Continuent 2014©Continuent 2014
Replicating from MySQL to Oracle
©Continuent 2014©Continuent 2014
Use Case: Real-Time Sales/Order Fulfillment
Real-Time Replication
Web-based Sales Purchase Ordering
3M Order Items
~1M Xacts/ Day
©Continuent 2014©Continuent 2014
1. Translating schema for the slave
©Continuent 2014©Continuent 2014
Translating Schema
���14
• Goal - set up heterogeneous replication
Replication
©Continuent 2014©Continuent 2014
Translating Schema
���15
• Beginning - how to convert tables?
empty
Tables
©Continuent 2014©Continuent 2014
Translating Schema
���16
• Beginning - how to convert tables?
empty
Tables
•Data types? •Column lengths? •Naming conventions? •Reserved words?
©Continuent 2014©Continuent 2014
Translating Schema
���17
• ddlscan automates these conversions
empty
Tables
ddlscan
©Continuent 2014©Continuent 2014
ddlscan
���18
• Part of Tungsten Replicator, GPL v2
• Translates schema with replication in mind
• Provides errors and warnings
• Can rename schema/tables/columns
©Continuent 2014©Continuent 2014
Usage (MySQL to Oracle Example)
���19
$ cd tungsten-replicator/bin !
$ ./ddlscan \ -db test \ -template ddl-mysql-oracle.vm \ -user tungsten \ -pass secret
©Continuent 2014©Continuent 2014
Translating Schema
���20
• ddlscan looks into source schema
empty
Tables
©Continuent 2014©Continuent 2014
Translating Schema
���21
• ddlscan translates and renders DDL commands
empty
Tables
oracle-ddl.sql
©Continuent 2014©Continuent 2014
Result of ddlscan
���22
CREATE TABLE 3colors /* ERROR: table starts with a number, use rename option and a filter */
( id NUMBER(10, 0), color VARCHAR2(1) /* ENUM('R','G','B') */, enabled NUMBER(1) /* BIT(1) - no constraints for 0 or 1 */, acolumnthatdoesntfittooracleat NUMBER(10, 0)
/* WARN: truncated column name exceeding 30 characters (acolumnthatdoesntfittooracleatall) */
! /* ERROR: table must have a primary key! */ ); !CREATE TABLE talks ...
©Continuent 2014©Continuent 2014
Translating Schema
���23
• You run resulting SQL file on Oracle
empty
Tables
oracle-ddl.sql
©Continuent 2014©Continuent 2014
Translating Schema
���24
• Tables are ready!
Tables Translated tables (empty)
©Continuent 2014©Continuent 2014
2. Set Up Replication
©Continuent 2014©Continuent 2014
Replicator Installation
���26
./tools/tpm configure my_ora \ --enable-heterogenous-service=true \ --install-directory=/opt/mysql_to_ora/continuent \ --members=alpha,bravo \ --master=alpha !./tools/tpm configure my_ora --hosts=alpha \ --replication-user=tungsten \ --replication-password=secret !./tools/tpm configure my_ora --hosts=bravo \ --replication-user=tungsten_frommysql \ --replication-password=secret \ --datasource-type=oracle \ --datasource-oracle-service=ORCL \ --svc-applier-filters=dropstatementdata !./tools/tpm install my_ora !./tools/tpm start my_ora
!
©Continuent 2014©Continuent 2014
Replication Under the Hood
���27
MySQL Binlog
binlog_format=row
Service my_ora
Tungsten Master Replicator
MySQLExtractor Special Filters •Transform
ENUM to string •Transform SET
to string •Column names
and signed flag
Service my_ora
Tungsten Slave Replicator
Special Filters •Drop DDL •Map names to upper case •Rename too long or reserved
objects •Optimize updates to remove
unchanged columns •Ignore extra tables
OracleApplierMySQLExtractor
©Continuent 2014©Continuent 2014
3. Provisioning an Oracle slave
©Continuent 2014©Continuent 2014
table structure
table data
table structure
Provisioning (2)
���29
table data
table structuretable data
table structure table structuretable structureddlscan
©Continuent 2014©Continuent 2014
table structure
Provisioning (3)
���30
table structuretable structure
MySQL Sandbox
CONVERT to BLACKHOLE
table data
table structuretable data
table structuretable data
table structure
©Continuent 2014©Continuent 2014
Provisioning (4)
���31
MySQL Sandbox
table data
table structuretable data
table structuretable data
table structure
not replicating
backup
©Continuent 2014©Continuent 2014
Provisioning (5)
���32
MySQL Sandbox
backup
bin logs
THL
1
(ROW)
Master Tungsten
Replicator
Slave Tungsten
Replicator
2 3 4
©Continuent 2014©Continuent 2014
4. Continue Real-Time Replication
©Continuent 2014©Continuent 2014
Provisioning (6)
���34
MySQL Sandbox
table data
table structuretable data
table structuretable data
table structure
bin logs
THL
Master Tungsten
Replicator
Slave Tungsten
Replicator
©Continuent 2014
Replicating from Oracle to MySQL
©Continuent 2014
Use Case: Web Content Publishing
Real-Time Publication
Backend Office Web-Based Catalog
©Continuent 2014
How Does Tungsten Do That?
���37
Service ora_my
Tungsten Master Replicator
OracleCDCExtractor No Special Filters
Service ora_my
Tungsten Slave Replicator
Special Filters •Map names to lower case •Ignore extra tables •Heartbeat table renaming
demo schema
demo_pub schema
Oracle CDC (Synchronous or
Async Hotlog)
MySQLApplierOracleCDCExtractor
©Continuent 2014
Replicating from Oracle to Oracle
©Continuent 2014
How Does Tungsten Do That?
���39
Service ora_ora
Tungsten Master Replicator
OracleCDCExtractor No Special Filters
Service ora_ora
Tungsten Slave Replicator
Special Filters No Special Filters
demo schema
demo_pub schema
Oracle CDC (Synchronous or
Async Hotlog)
Tungsten Slave Replicator
OracleApplierOracleCDCExtractor
©Continuent 2014©Continuent 2014
DEMO
DEMO.* demo.*
DEMO2.* demo2.*
©Continuent 2014
Replicating from Oracle to Vertica
©Continuent 2014
Use Case: Web Content Publishing
Near Real-Time Publication
Backend Office Analytic Reports
Feeds from other data sources
©Continuent 2014
How Does Tungsten Do That?
���43
Service ora_ver
Tungsten Master Replicator
OracleCDCExtractor No Special Filters
Service ora_ver
Tungsten Slave Replicator
Special Filters •Map names to lower case •Ignore extra tables •Rename data tables (if required)
demo schema
demo_pub schema
Oracle CDC (Synchronous or
Async Hotlog)
SimpleBatchApplierOracleCDCExtractor
©Continuent 2014
Batch Loading--The Gory Details
���44
Replicator
Transactions from master
CSV FilesCSV FilesCSV Files
Staging TablesStaging TablesStaging Tables
Base Tables
Base Tables
Base Tables
Merge Script
(or) COPY
directly to base tables
COPY to stage tables SELECT to
base tables
Service ora_ver
©Continuent 2014
DEMO
©Continuent 2014
Coming Attractions
©Continuent 2014
New Replication Features in 2014
• Parallel extraction for fast provisioning
• Select data as of specific SCN
• Pull data across in multiple streams
• Matches parallel apply on slave side
!
• Hadoop data loading
• Direct loading from MySQL/Oracle to HDFS
• View data using Hive
• Generation of materialized views
���47
©Continuent 2014 ���48
Continuent Web Page: http://www.continuent.com
!Documentation:
https://docs.continuent.com !
Tungsten Replicator: http://code.google.com/p/tungsten-replicator
Our Blogs: http://flyingclusters.blogspot.com http://scale-out-blog.blogspot.com http://datacharmer.blogspot.com http://continuent-tungsten.blogspot.com
560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: [email protected]