Post on 30-Sep-2014
<Insert Picture Here>
Oracle PTS GoldenGate Workshop for partners
GoldenGate is Strategic to Oracle
2
• Best-in-class leader in real-time data solutions
• Complements existing Oracle products
• Continuous Availability for heterogeneous systems
• Real-time data access for Reporting
• Real-time data for BI, EPM, Exadata
• Zero-downtime migrations / upgrades to Oracle Database and Applications
• Over 400 customers with 4,000+ implementations across Fortune 500 companies: Financial Services,
Communications, Healthcare, Public Sector, Retail & Utilities industries
• Top 3 of 5 largest commercial banks
• Top 3 of 3 busiest ATM networks
• Top 7 of 10 financial data services companies
• Top 4 of 5 telecommunications providers
• Top 3 of 5 largest food & drug stores
Industry Proof Points
“Oracle will become more competitive in data integration tools and may enhance its high-availability products by buying GoldenGate.”
Ted Friedman, Gartner
“Oracle will become more competitive in data integration tools and may enhance its high-availability products by buying GoldenGate.”
Ted Friedman, Gartner
“This is a critical requirement to enable high volume data extraction to support real-time data warehousing and BI, as well as master data management and other operational use cases.”
Rob Karel, Forrester
“This is a critical requirement to enable high volume data extraction to support real-time data warehousing and BI, as well as master data management and other operational use cases.”
Rob Karel, Forrester
“It's quite a catch for Oracle in that it's a significant addition to its data warehousing and BI portfolio.”
James Kobielus, Forrester
“It's quite a catch for Oracle in that it's a significant addition to its data warehousing and BI portfolio.”
James Kobielus, Forrester
Oracle GoldenGate: Databases and Platforms
Databases O/S and Platforms
Capture:
� Oracle
� IBM DB2
� Microsoft SQL Server
� Sybase ASE
� Ingres
� Teradata
� Enscribe
� SQL/MP
� SQL/MX
Delivery:
� All listed above
� MySQL, HP Neoview, Netezza, and any ODBC compatible databases
� ETL products
� JMS message queues or topics
Windows 2000, 2003, XP
Linux
Sun Solaris
HP NonStop
HP-UX
HP TRU64
HP OpenVMS
IBM AIX
IBM z/OS
Movement Management Integration
� Speed – Subsecond Latency
� Volume – Thousands of TPS
� Log-based Capture
� Native, Local Apply
� Efficient IO and Bandwidth Usage
� Bidirectional
� Group Transactions
� Bulk Operations
� Compression
� One-to-Many, Many-to-One
� Cascade
� Transaction Integrity
� Transparent Capture
� Guaranteed Delivery
� Conflict Detection, Resolution
� Dynamic Rollback
� Incremental TDM
� Initial Data Load
� GUI-based Monitoring and Configuration
� Proactive Alerts
� Encryption
� Real-Time Deferred or Batch
� Event Markers
� Heterogeneous Data Sources
� Mapping
� Transformation
� Enrichment
� Decoupled Architecture
� Table, Row, Column Filtering
� XML, ASCII, SQL Formats
� Queue Interface
� Stored Procedures
� User Exits
� ETL Integration
� Java/JMS Integration
Oracle GoldenGate Advantages
6
Active Data Guard
Direct
Memory Access
Read
On-Disk LogsFast SQL
Direct
Write to Logs
Oracle Products for Data MovementComparing How They Work
Redo in
Memory
Buffer
GoldenGate
Data Integrator
Disaster Recovery & Data Protection
Real Time Data Integration & High Availability
SQL Query Set-based,
Complex SQL
Data Integration for Data Warehouse & SOAHE
TE
RO
GE
NE
OU
S
Real-Time Information Access for
Intelligence & Decisioning
• Instantly available Data for Reporting
• Access to Key Information for Analysis
• Across Heterogeneous Systems
Real-Time
Business
Operations
Customers Demand Continuous Real Time DataWhile Lowering IT Costs and Gaining Operational Efficiencies
Continuous Availability of Critical
Applications and Data
• Continuous Uptime in event of disaster
• No downtime during planned outage
• Load balancing & Data distribution
Lower IT Spend & Improve Efficiencies
• Choose best system for need
• Modernize Legacy Systems
• Offload for reporting
Real-time Business Delivers Value & ROI
90% reduced recovery time for
mission-critical transactional
applicationBank of America
World’s Busiest ATM Network Remains Continuously
AvailableVocaLINK
6X productivity gains in
manufacturing plant with real-time
dataAMD
100% availability during phased upgrade for 24x7 critical CRM
applicationHSN
100% paperless environmentdrives efficiency and improves
patient care across hospitalMontefiore
Processing > 1 Billion Transactions each year, and zero
outagesSwedbank
Oracle GoldenGate Technology Differentiators
9
9
Oracle GoldenGate provides low-impact capture, routing, transformation, and delivery of database transactions across heterogeneous environments in real-time
Performance
Extensible &Flexible
Reliable
Key Differentiators:
Non-intrusive, Low Impact, Sub-second latency
Open, modular architecture - Heterogeneous
sources & Targets
Maintains Transactional Integrity - Resilient against interruptions and failures
Oracle GoldenGate Use CasesEnterprise-wide Solution for Real Time Data Needs
Log Based, Real-Time Change Data Capture
Heterogeneous Source Systems
EDWODS
EDW
Disaster Recovery, Data Protection
Zero Downtime Migration and Upgrades
Operational Reporting
Real-time BI
Standby(Open & Active)
ReportingDatabase
OGG
ETL
ETL
Query Offloading
Data Distribution
• Standardize on Single
Technology for Multiple Needs
• Deploy for Continuous
Availability and Real-time Data
Access for Reporting / BI
• Highly Flexible
• Fast Deployments
• Lower TCO & Improved ROI
Data Integration: Live Reporting
When you need:
• Solution for reporting’s performance impact on production application
• Real-time, current data for reporting demands
• Unlimited scalability as user demands and data volumes grow
Real-Time and Lower Total Cost of Ownership
Query OffloadingInfrastructure Cost Reduction
Query Offloading
12
Non-Oracle or
Legacy PlatformOracle Platform
OLTP
TransactionsRead Only
Users
OGG
Cost Reduction
Move query processing to standard, lower cost DB & Hardware infrastructure
with no disruption to legacy critical transaction processing systems
Solution• Oracle GoldenGate
• Create a real-time replica of existing expensive operational system (e.g. legacy or big iron) on Oracle
with RAC on Linux.
• Sync replica in real-time and route all read-only traffic
Benefits• Reduce/Eliminate query load on transactional system
• Directly reduce cost of legacy, mainframe, non-oracle
Real-Time Data IntegrationEnables Real-time BI
Real Time Data Integration
13
Solution• Oracle GoldenGate with Oracle Data Integrator
• Populate an operational data store for operational BI with real-time data
• Run ELT from the ODS to load data warehouse with transformed data
Benefit
• Improves business insight
• Combines intelligence on current business status with
historical context
• Eliminates load on source systems
• Reduces latencies and batch windows
Data Mart
Enterprise Data Warehouse
Oracle Database,
Oracle Exadata
ODS
Real-time Integration + Bulk
Data Transformation
OGG
ODI
Better business insight with the most current data with no
impact to your critical systems
Data Integration: Real-Time Data Warehousing
When you need:
• Real-time data for Business Intelligence
• Eliminate batch window dependency
• Data integration that is:– Lower impact & overhead
– More flexible (transformations and user exits, heterogeneity, topologies)
– Easier to implement and maintain
Continuous AvailabilityZero Down-time Migration and Upgrade
Continuous Availability
15
Solution• Oracle GoldenGate
• Migrate from any heterogeneous database to Oracle11g
with minimal downtime
• Migrate hardware or OS platforms with no downtime
• Upgrade Siebel CRM with no downtime
• Allow active users to move to new system gradually while both systems are live
Benefit• Standardize your data center and modernize platforms
• Zero down-time for mission critical applications
• Safely migrate users from one system to the next, while keeping both systems running.
• Lower risk by enabling failback
Previous Version New Version
OGG
Upgrade critical applications or migrate your database without any
down time or impacting critical business processes
Old DB Version New DB Version
Continuous Data AvailabilityData Distribution / Synchronization
Continuous Availability
16
Solution• Oracle GoldenGate
• Replicate data for distributed applications to drive global operations and improve operational efficiency
• Distribute data in real-time across Oracle and Non-Oracle
environments
• Active-Active configuration keeps multiple data centers in
sync
Benefit• High Availability for mission critical data
• Ability to scale, handle large & growing data volumes
• Access to consistent data across geographically dispersed data centers
Oracle 9i Oracle 11g
Oracle 10g SQL Server
OGG
Keep up with the demands of a 24x7 global business. While all business units
access the same, up-to-data information at any time, continuously
Oracle® GoldenGate Architecture
How Oracle GoldenGate Works: Modular “Building Blocks”
SourceDatabase(s)
TargetDatabase(s)
Network(TCP/IP)
Route: Data is compressed, encrypted for routing to targets.
Capture: Committed changes are captured (and can be filtered) as they occur by reading the transaction logs.
Capture
Trail files: Stages and queues data for routing.
SourceTrail
Delivery: Applies data with transaction integrity, transforming the data as required.
TargetTrail
Delivery
Bi-directional
SourceTrail
TargetTrail
Delivery Capture
Oracle GoldenGate Data Capture and Delivery
Oracle GoldenGate Transactional Data Management:
• Primarily used for change data capture and delivery from database
transaction logs
• Can optionally be used for initial load directly from database tables
– Especially useful for synchronizing heterogeneous databases
– Database-specific methods may be preferable for homogeneous
configurations
Change Data Capture & Delivery
Transaction Log
Extract ServerCollector
ReplicatTrail
Manager Manager
SourceDatabase
TargetDatabase
Network(TCP/IP)
Change Data Capture & Delivery using a Data Pump
Transaction Log
Extract ServerCollector
ReplicatRemoteTrail
Manager Manager
SourceDatabase
TargetDatabase
Data Pump
LocalTrail
Network(TCP/IP)
Manager - Overview
• Performs system management and monitoring tasks
– Starting GoldenGate processes
– Starting dynamic Server Collector, Replicat, or GGSCI processes
– Error and lag reporting
– GoldenGate trail management
• Parameter file
– mgr.prm file in GGS ./dirprm directory
• Event information written to ggserr.log file
Change Capture - Extract Overview
Extract can be configured to:
• Capture changed data from database logs
• Distribute data from local trails to remote systems (data pump)
• Capture data directly from source tables for initial
data load
Change Capture - Tasks
On the source system:
• Add a primary Extract (reading from source transaction logs) with an associated parameter file
• Optionally, add a local trail and a data pump Extract (reading from the local trail) with an associated parameter file
• Add a remote trail
• Start the Extract(s)
Data Pumps - Overview
• Data is stored in a local trail on the source system
• A second Extract, the data pump:
– Reads this trail and sends it to one or more targets
– Manipulates the data or passes it through without change
• Reasons for using
– A safeguard against network and target failures
– To break complex data filtering and transformation into phases
– To consolidate data from many sources
– To synchronize one source with multiple targets
Data Pumps – One to Many Trails
Primary Extract
TrailData Pump
Trail
Trail
Trail
Data Pumps – One to Many Target Systems
PrimaryExtract
TrailData
Pump 2Trail
Trail
Trail
Data Pump 1
Data Pump 3
Change Delivery - Replicat Overview
Replicat can:
• Read data out of GoldenGate trails
• Perform data filtering
– Table, row, operation
• Perform data transformation
• Perform database operations just as your application performed them
Change Delivery - Tasks
On the target system:
– Create a checkpoint table in the target database (best practice)
• DBLOGIN
• ADD CHECKPOINTTABLE
– Create a parameter file for Replicat
• EDIT PARAMS
– Add your initial Replicat checkpoint into GoldenGate trails
• ADD REPLICAT
– Start the Replicat process
• START REPLICAT
30
Extract Trails and Files - Overview
• Extract writes data to any of:– Remote trail (RMTTRAIL)
– Remote file (RMTFILE)
– Local trail (EXTTRAIL)
– Local file (EXTFILE)
• Extract trails and files are unstructured, with variable length records– I/O performed using large block writes
• Extract writes checkpoints for trails during change capture:– Guarantees no data lost during restart– Multiple Replicat processes may process the same trail
• Extract does not write checkpoints for files
31
Extract Trails and Files - Distribution
– Extract can write:
� To local trails, then distribute over IP with a data pump to remote trails
� To multiple trails
• For distribution multiple systems/disk storage devices
• For parallel processing by downstream processes
• Trails and files can be transported online using TCP/IP or sent in batch using any file transfer method
Extract Trails and Files - Contents
– Each record in the trail contains an operation that has
been committed in the source database
• Transactions are output in commit order
– Operations in a transaction are grouped together, in the
order they were applied
• By default, only the primary key and changed columns are
recorded
– Flags indicate the first and last records in each
transaction
Extract Trails and Files - Cleanup
• Trail files can be purged once consumed
– Temporary storage requirement is small if processes keep pace
– Configure Manager to purge used trail data (best practice)
Initial Load
Extract Replicat
Manager Manager
SourceDatabaseTables
TargetDatabase
Network(TCP/IP)
ServerCollector Files
Or DB Bulk Load Utility
GoldenGate initial load methods:� Direct Load (Extract sends data directly to Replicat to apply using SQL)� Direct Bulk Load (Replicat uses Oracle SQL*Loader API)� File to Replicat (Extract writes to a file that Replicat applies using SQL)� File to database utility (Extract writes to a file formatted for a DB bulk load utility)
Initial Load: File to Replicat
ADD EXTRACT <name>Extract parameters:SOURCEISTABLERMTTRAIL<name>
Files
Manager
Extract Replicat
SourceDatabase
TargetDatabase
Initial Load: File to Database Utility
ADD EXTRACT <name>Extract parameters:SOURCEISTABLERMTFILE <name>FORMATASCII BCP or SQLLOADER
SQL*Loader
BCP
SSIS
File
File
File
Manager
Extract
SourceDatabase
TargetDatabase
Initial Load: Direct Load
Manager
Extract Replicat
Manager
ADD EXTRACT <name>, SOURCEISTABLEExtract parameters:RMTTASK REPLICAT, GROUP <name>
ADD REPLICAT <name>, SPECIALRUN
SourceDatabase
TargetDatabase
Initial Load: Direct Bulk Load (to Oracle)
Manager
SQL*Loader API
Manager
ADD EXTRACT <name>, SOURCEISTABLEExtract parameters:RMTTASK REPLICAT, GROUP <name>
ADD REPLICAT <name>, SPECIALRUNReplicat parameters:BULKLOAD
Extract ReplicatSourceDatabase
SourceDatabase
OracleTarget
Online versus Batch
• Change data capture & delivery can be run either continuously (online)
or as a special run (batch run) to capture changes for a specific period
of time.
• Initial load is always a special run (batch run).
Begin, TX 1
Insert, TX 1
Begin, TX 2
Update, TX 1
Insert, TX 2
Commit, TX 2
Begin, TX 3
Insert, TX 3
Begin, TX 4
Commit, TX 3
Delete, TX 4
Capture, Pump, and Delivery save positions to a checkpoint file so they can
recover in case of failure
Capture Commit Ordered Source TrailSource
Database
Begin, TX 2
Insert, TX 2
Commit, TX 2
Begin, TX 3
Insert, TX 3
Commit, TX 3
Start of Oldest Open (Uncommitted) Transaction
Current Read Position
Capture Checkpoint
Current Write Position
Checkpointing
Capture, Pump, and Delivery save positions to a checkpoint file so they can
recover in case of failure
Capture PumpCommit Ordered Source Trail
Commit Ordered Target TrailSource
Database
Begin, TX 1
Insert, TX 1
Begin, TX 2
Update, TX 1
Insert, TX 2
Commit, TX 2
Begin, TX 3
Insert, TX 3
Begin, TX 4
Commit, TX 3
Delete, TX 4
Begin, TX 2
Insert, TX 2
Commit, TX 2
Begin, TX 3
Insert, TX 3
Commit, TX 3
Begin, TX 2
Insert, TX 2
Commit, TX 2
Start of Oldest Open (Uncommitted) Transaction
Current Read Position
Capture Checkpoint
Current Write Position
Current Read Position
Pump Checkpoint
Current Write Position
Checkpointing
Checkpointing
Capture, Pump, and Delivery save positions to a checkpoint file so they can
recover in case of failure
CaptureDeliveryPumpCommit Ordered
Source Trail
Commit OrderedTarget TrailSource
Database Target
Database
Begin, TX 1
Insert, TX 1
Begin, TX 2
Update, TX 1
Insert, TX 2
Commit, TX 2
Begin, TX 3
Insert, TX 3
Begin, TX 4
Commit, TX 3
Delete, TX 4
Begin, TX 2
Insert, TX 2
Commit, TX 2
Begin, TX 3
Insert, TX 3
Commit, TX 3
Begin, TX 2
Insert, TX 2
Commit, TX 2
Start of Oldest Open (Uncommitted) Transaction
Current Read Position
Capture Checkpoint
Current Write Position
Current Read Position
Pump Checkpoint
Current Write Position
Current Read Position
Delivery Checkpoint
September 8, 2010
Configuring Oracle® GoldenGate
Configuring Oracle GoldenGate
Oracle GoldenGate can be deployed quickly and easily in four steps:
1. Prepare the Environment
2. Change Capture
3. Initial Load
4. Change Delivery
Configuring Oracle GoldenGate
3. Initial Load (various methods)
1. Prepare the Environment
TargetDatabase
TransactionLog
Source Database
2. Change Capture
Extract RemoteTrail
LocalTrail
Data Pump
4. Change Delivery
Replicat
Parameters, Process Groups and Commands
• GoldenGate processes are configured by ASCII
parameter files.
• A process group consists of:
– An Extract or Replicat process
– Associated parameter file
– Associated checkpoint file
– Any other files associated with that process
Each process group on a system must have a unique group
name.
• Processes are added and started using the
GoldenGate Software Command Interface (GGSCI)
with the group name.
• GGSCI commands also add trails, check process status, etc.
September 8, 2010
Configuring Oracle GoldenGate:
Step 1. Prepare the Environment
Step 1. Prepare the Environment
3. Initial Load (various methods)
4. Change Delivery
Replicat
1. Prepare the Environment
2. Change Capture
Extract RemoteTrail
TargetDatabase
TransactionLog
Source Database
LocalTrail
Data Pump
Step 1. Prepare the Environment
Set up each system:
• Install Oracle GoldenGate software on source and target
• Configure and start GoldenGate Manager on source and
target
• If heterogeneous source/target, generate source definitions
and copy to target
• Prepare the database. For example:
– Ensure database access by GoldenGate
– Enable transaction logging
Prepare the Environment:
Install Oracle GoldenGateSoftware
Prepare Environment: Installation – Access the Media Pack
• Access the product media pack (software and documentation) at edelivery.oracle.com
• Identify the proper release of GoldenGate for your source and target environments
– Database and version
– Operating system and version
Prepare Environment: Installation - UNIX, Linux or z/OS
• Download .gz file to /ggs
• gzip –d {filename}.tar.gz
• tar -xvof {filename}.tar
• GGSCI> CREATE SUBDIRS
Prepare Environment: Installation - Windows
• Download .zip file to C:\GGS
• Unzip .zip file into C:\GGS folder
• Configure a Windows Service Name for Manager process in a GLOBALS parameter file (required only if multiple Managers on the server)
• C:\GGS> INSTALL ADDSERVICE ADDEVENTS
• GGSCI> CREATE SUBDIRS
Prepare Environment: Installation – Windows INSTALL Program
• On Windows, an INSTALL program performs the following
functions:
– Installs GoldenGate event messages into the system registry
– Installs the Manager as a Windows service
• Syntax:
INSTALL <item> [<item> …]
• Example:
C:\GGS> INSTALL ADDEVENTS ADDSERVICE
Note: The uninstall command is:
INSTALL DELETESERVICE DELETEEVENTS
Prepare Environment: Installation - Multiple Manager Services
• GoldenGate supports running multiple Manager services on Windows
– For two or more GoldenGate instances, or
– GoldenGate with a Veridata C Agent (which uses a Manager)
• Each Manager service must be assigned a unique name
• Before installing the service, you can specify the name
– Create a GLOBALS parameter file for each Manager
– Specify the one-word name of the service using the MGRSERVNAME <name> parameter
• INSTALL ADDSERVICE
– Reads the GLOBALS MGRSERVNAME for the service name
– If no GLOBALS setting, uses default service name GGSMGR
Prepare Environment: Installation - GoldenGateDirectories
Directory Contents
dirchk GoldenGate checkpoint files
dirdat GoldenGate trail and extract files
dirdef Data definitions produced by DEFGEN and used to
translate heterogeneous data
dirpcs Process status files
dirprm Parameter files
dirrpt Process report files
dirsql SQL scripts
dirtmp Temporary storage for transactions that exceed
allocated memory
GGSCI Examples
• Start a Manager process
GGSCI> START MGR
• Add an Extract group
GGSCI> ADD EXTRACT myext, TRANLOG, BEGIN NOW
• Add a local trail
GGSCI> ADD EXTTRAIL /ggs/dirdat/rt, EXTRACT myext
• Start an Extract group
GGSCI> START EXTRACT myext
Editing Parameter Files
• Edit parameter files to configure GoldenGate processes
• The GLOBALS parameter file is identified by its file path
GGSCI> EDIT PARAMS ./GLOBALS
• Manager and utility parameter files are identified by keywords
GGSCI> EDIT PARAMS MGR
GGSCI> EDIT PARAMS DEFGEN
• Extract and Replicat parameter files are identified by the
process group name
GGSCI> EDIT PARAMS <group name>
GLOBALS versus Process Parameters
• GLOBALS parameters apply to all processes
– Set when Manager starts
– Reside in <GoldenGate install directory>/GLOBALS
• Process parameters apply to a specific process (Manager, Extract, Server Collector, Replicat, Utilities)
– Set when the process starts
– Override GLOBALS settings
– Reside by default in the dirprm directory in files named
<processname>.prm
– Most apply to all tables processed but some can be specified at the
table level
Functions - Data Transformation
• GoldenGate provides the capability to transform
columns by using a set of built-in functions
• Transformation functions can be applied either for
Extract or Replicat
• If you require more, you also have the ability to call
your own logic through user exits
Functions - Overview
• Using column conversion functions, you can:
– Perform string and number conversion
– Extract portions of strings or concatenate columns
– Compare strings or numbers
– Perform a variety of date mappings
– Use single or nested IF statements to evaluate numbers,
strings, and other column values to determine the
appropriate value and format for target columns
• Functions are identified with the @ prefix
Prepare Environment: Source Definitions - Overview
• The problem
– Understanding source and target layouts across disparate systems and
databases
• The solution – the DEFGEN utility program
– DEFGEN produces a file containing layout definitions of the source files and tables
– This source definition file is used to interpret layouts for data stored in
GoldenGate trails
• At start up Replicat reads the definition file specified with the
SOURCEDEFS parameter
• Server Collector uses the –d argument to specify which definition file
to read at startup
• Can also capture target definitions on target system and copy to
source system for Extract to use