Oracle Golden Gate - nlOUG · Oracle Golden Gate. . . from AIX to Exadata . . . Even voorstellen...

01/06/2018

1

Oracle Golden GateOracle Golden GateOracle Golden GateOracle Golden Gate

. . . from AIX to Exadata . . .. . . from AIX to Exadata . . .. . . from AIX to Exadata . . .. . . from AIX to Exadata . . .

Even voorstellen ......Even voorstellen ......Even voorstellen ......Even voorstellen ......

1-6-20182

Rob LasonderIn dienst bij Qualogy als Exadata SpecialistExadata SpecialistExadata SpecialistExadata Specialist

... Sinds 2010 bezig met Exadata (V2)

... Momenteel project X2-6 bij int. verzekeraar

... Laatste tijd vooral bezig met Golden Gate

OCP: Oracle 8, 8i, 9i, 10g, 11g, 12cOCE: Exadata X3, X4, RAC11g, ZFS Storage Appliance, Big Data & GG 11g Impl. specialist

Email: [email protected]

01/06/2018

2

AgendaAgendaAgendaAgenda

1-6-2018Agenda3

• 1 Golden Gate Use Case• 2 Golden Gate introduction• 3 Golden Gate customer architecture• 4 Golden Gate installation & configuration• 5 Golden Gate findings & troubleshooting• 6 Golden Gate Monitoring• 7 Golden Gate High Availability• 8 Lessons learned

1 Golden Gate Use Case1 Golden Gate Use Case1 Golden Gate Use Case1 Golden Gate Use Case

01/06/2018

3

1 Golden Gate Use Case1 Golden Gate Use Case1 Golden Gate Use Case1 Golden Gate Use Case

1-6-20185

• The customer purchased an Exadata 1/8 Rack and already migrated some of their databases to Exadata

• De core legacy database is too complex to migrate at this stage, the migration is planned for 2019.

• In order to already use Exadata for this core legacy database, the database will be replicated to Exadata.

• This Golden Gate Clone on Exadata will be used to build data services, mainly for reporting services.

2 Golden Gate introduction2 Golden Gate introduction2 Golden Gate introduction2 Golden Gate introduction

01/06/2018

4

2 Golden Gate Introduction : What is Golden Gate?2 Golden Gate Introduction : What is Golden Gate?2 Golden Gate Introduction : What is Golden Gate?2 Golden Gate Introduction : What is Golden Gate?

1-6-20187

Oracle GoldenGate provides lowlowlowlow----impact captureimpact captureimpact captureimpact capture, routingroutingroutingrouting, transformationtransformationtransformationtransformation, and deliverydeliverydeliverydelivery of transactional data across heterogeneousheterogeneousheterogeneousheterogeneous environments in real timereal timereal timereal time

• Golden Gate replicates data or subsets of data, in real time

• Golden Gate can transform data during the replication

• Golden Gate supports heterogenous environments• Different platforms (Linux, Windows, etc.)• Different databases (Oracle, MySQL, SQLServer, etc)• Different database versions

2 GG Introduction: Golden Gate core functionality2 GG Introduction: Golden Gate core functionality2 GG Introduction: Golden Gate core functionality2 GG Introduction: Golden Gate core functionality

1-6-20188

1 Database transactions are captured in trail files

2 Trail files are transported to target

3 Transactions are applied in the target

Real time (asynchronous) processing, Decoupled ArchitectureReal time (asynchronous) processing, Decoupled ArchitectureReal time (asynchronous) processing, Decoupled ArchitectureReal time (asynchronous) processing, Decoupled Architecture

01/06/2018

5

2 GG Introduction: available GG architectures2 GG Introduction: available GG architectures2 GG Introduction: available GG architectures2 GG Introduction: available GG architectures

1-6-20189

2 GG Introduction: heterogenous support2 GG Introduction: heterogenous support2 GG Introduction: heterogenous support2 GG Introduction: heterogenous support

1-6-201810

01/06/2018

6

2 GG Introductie: Golden Gate 12.2 New Features2 GG Introductie: Golden Gate 12.2 New Features2 GG Introductie: Golden Gate 12.2 New Features2 GG Introductie: Golden Gate 12.2 New Features

1-6-201811

• Self describing trail files: Metadata records used to interpret DML

records instead of SOURCEDEFS or ASSUMETARGETDEFS

• Built-in Automatic Heartbeat table (GGSCI> ADD HEARTBEATTABLE)

• Transparent integration with Oracle Clusterware & ASM

• Integrated Replicat (and Extract)

• Graphical real-time instance monitoring (OGG Performance Tool Kit)

• Oracle Data Pump Integration for Table Instantiation

• Support for Invisible Columns (Oracle Only)

• New parameter – MAPINVISIBLECOLUMNS (Integrated only)

2 GG Introductie: Golden Gate 12.2 New Features2 GG Introductie: Golden Gate 12.2 New Features2 GG Introductie: Golden Gate 12.2 New Features2 GG Introductie: Golden Gate 12.2 New Features

1-6-2018Agenda12

Integrated Capture (GG 11.2 NF) & Replicat (GG 12.2 NF): Integrated Capture (GG 11.2 NF) & Replicat (GG 12.2 NF): Integrated Capture (GG 11.2 NF) & Replicat (GG 12.2 NF): Integrated Capture (GG 11.2 NF) & Replicat (GG 12.2 NF): Extract & Replicat processes interact with Oracle Log Mining Services

• Only applicable to Oracle Databases.• STREAMS_POOL_SIZE 1250 MB per Extract/Replicat process !

01/06/2018

7

2 GG Introduction: Golden Gate 12.2 New Features2 GG Introduction: Golden Gate 12.2 New Features2 GG Introduction: Golden Gate 12.2 New Features2 GG Introduction: Golden Gate 12.2 New Features

1-6-2018Agenda13

Benefits Integrated Capture (11.2 NF) & Replicat (12.2 NF)

Benefits Integrated Capture (Extract)Benefits Integrated Capture (Extract)Benefits Integrated Capture (Extract)Benefits Integrated Capture (Extract)• More datatypes natively supported (LOB's, XML)• Compression Supported• Easier configuration• Transparent support for ASM, RAC, RMAN, DataGuard and TDE

Benefits Advantages Integrated ReplicatBenefits Advantages Integrated ReplicatBenefits Advantages Integrated ReplicatBenefits Advantages Integrated Replicat• Automatic parallel processing• Support for 12c Multitenant databases• Performance

Very easy to configure:

GGSCI> DBLOGIN USERIDALIAS ALIAS_GGADMINGGSCI> REGISTER EXTRACT ext1 DATABASEGGSCI> ADD EXTRACT ext1 INTEGRATED TRANLOG, BEGIN NOW


1-6-201814

Reference: Oracle GoldenGate Performance Tuning for Oracle Database Integration CON7773

Doc in HandoutDoc in HandoutDoc in HandoutDoc in Handout

01/06/2018

8


1-6-201815

References integrated capture & replicatReferences integrated capture & replicatReferences integrated capture & replicatReferences integrated capture & replicat• Extracting Data in Oracle GoldenGate Integrated Capture Mode• Oracle GoldenGate Performance Best Practices (May 2017)

Oracle by Example:Oracle by Example:Oracle by Example:Oracle by Example:

Docs in HandoutDocs in HandoutDocs in HandoutDocs in Handout

2 GG Introduction: Golden Gate versus Dataguard2 GG Introduction: Golden Gate versus Dataguard2 GG Introduction: Golden Gate versus Dataguard2 GG Introduction: Golden Gate versus Dataguard

1-6-201816

Oracle ships 2 seperate products for database replication: Dataguard and Golden Gate

Oracle Dataguard:Oracle Dataguard:Oracle Dataguard:Oracle Dataguard:

• Optimized for disaster recovery anddisaster recovery anddisaster recovery anddisaster recovery and data protectiondata protectiondata protectiondata protection.

• (physical) standby DB is a physical block-level copy of the database

• Part of Oracle Maximum Availability Architecture (MAA)

Oracle Golden Gate:• Optimized for database replicationreplicationreplicationreplication and datatransformation in datatransformation in datatransformation in datatransformation in

heterogenous environments.heterogenous environments.heterogenous environments.heterogenous environments.

01/06/2018

9

2 GG Introduction: Golden Gate versus Dataguard2 GG Introduction: Golden Gate versus Dataguard2 GG Introduction: Golden Gate versus Dataguard2 GG Introduction: Golden Gate versus Dataguard

1-6-201817

2 GG Introduction: Golden Gate versus ODI2 GG Introduction: Golden Gate versus ODI2 GG Introduction: Golden Gate versus ODI2 GG Introduction: Golden Gate versus ODI

1-6-201818

Oracle Data Integrator and Golden Gate complement each other

Oracle ODI: Oracle ODI: Oracle ODI: Oracle ODI: is based on a new technological concept called E –LT

(Extract – Load, Transform), as opposed to traditional ETL tools

Disadvantages ETL architecture: Disadvantages ETL architecture: Disadvantages ETL architecture: Disadvantages ETL architecture: • CPU/RAM / IO overhead• Data transform process at

application level

Advantages EAdvantages EAdvantages EAdvantages E----LT architecture: LT architecture: LT architecture: LT architecture: • Less CPU/RAM/IO overhead• Data transform processes at

DB level

01/06/2018

10

2 GG Introduction: Golden Gate versus ODI2 GG Introduction: Golden Gate versus ODI2 GG Introduction: Golden Gate versus ODI2 GG Introduction: Golden Gate versus ODI

1-6-201819

Best of both worlds:• Oracle Golden Gate Oracle Golden Gate Oracle Golden Gate Oracle Golden Gate provides a cross-platform data replication and

changed data capture. (Extract)• Oracle ODI is used for the transformation and loading

ExtractExtractExtractExtract

Load & Load & Load & Load & TransformTransformTransformTransform

3 Golden Gate customer architecture3 Golden Gate customer architecture3 Golden Gate customer architecture3 Golden Gate customer architecture

01/06/2018

11

3 Golden Gate customer architecture: the “old” world3 Golden Gate customer architecture: the “old” world3 Golden Gate customer architecture: the “old” world3 Golden Gate customer architecture: the “old” world

1-6-201821

Typical characteristics:• Maintenance silo’s: network, sysadmin, storage, DBA• DBA has no control over (v)CPU & shared storage• DBA lacks required privileges by default (root, and even oracle for production)

Oracle 12cR1 RDBMSOracle 12cR1 RDBMSOracle 12cR1 RDBMSOracle 12cR1 RDBMSProduction Database : PRODProduction Database : PRODProduction Database : PRODProduction Database : PROD

SGA SGA SGA SGA : 38 GB: 38 GB: 38 GB: 38 GBPGA PGA PGA PGA : 21 GB: 21 GB: 21 GB: 21 GBAvg Redo per dayAvg Redo per dayAvg Redo per dayAvg Redo per day : 73,4 GB: 73,4 GB: 73,4 GB: 73,4 GBSize PROD_SCHEMASize PROD_SCHEMASize PROD_SCHEMASize PROD_SCHEMA : 2.111 GB: 2.111 GB: 2.111 GB: 2.111 GB

IBM Power8 series (up to 8 threads per Power8 CPU) & IBM PowerVM IBM Power8 series (up to 8 threads per Power8 CPU) & IBM PowerVM IBM Power8 series (up to 8 threads per Power8 CPU) & IBM PowerVM IBM Power8 series (up to 8 threads per Power8 CPU) & IBM PowerVM

Exadata X6-2 1/8 Rack compute nodes: 2 x 22-core Intel Xeon E5 v4 processors

3 Golden Gate customer architecute : the “new” world3 Golden Gate customer architecute : the “new” world3 Golden Gate customer architecute : the “new” world3 Golden Gate customer architecute : the “new” world

1-6-201822

DB(M

)A in F

ull C

ontr

ol, T

ota

l Sta

ck M

ain

tenance

DB(M

)A in F

ull C

ontr

ol, T

ota

l Sta

ck M

ain

tenance

DB(M

)A in F

ull C

ontr

ol, T

ota

l Sta

ck M

ain

tenance

DB(M

)A in F

ull C

ontr

ol, T

ota

l Sta

ck M

ain

tenance

DB +

Sysadam

in +

Sto

rage +

Netw

ork

DB +

Sysadam

in +

Sto

rage +

Netw

ork

DB +

Sysadam

in +

Sto

rage +

Netw

ork

DB +

Sysadam

in +

Sto

rage +

Netw

ork

01/06/2018

12

3 Golden Gate customer architecture: new versus old3 Golden Gate customer architecture: new versus old3 Golden Gate customer architecture: new versus old3 Golden Gate customer architecture: new versus old

1-6-201823

• Sample of 10 report in old (AIX) and new (Exadata) DB• All 10 reports were faster on Exadata (“out-of-the-box”)• Performance gains between 1 and 181 (!) times faster on Exadata

(especially with "big volume data reports")

New versus old: realNew versus old: realNew versus old: realNew versus old: real----life comparison in customer landscape.........life comparison in customer landscape.........life comparison in customer landscape.........life comparison in customer landscape.........

3 Golden Gate customer architecture: GG Setup3 Golden Gate customer architecture: GG Setup3 Golden Gate customer architecture: GG Setup3 Golden Gate customer architecture: GG Setup

1-6-201824

GoldenGate 12.2GoldenGate 12.2GoldenGate 12.2GoldenGate 12.2• 1 technical test environment• 1 functional test environment• 1 production environment

old world (AIX)old world (AIX)old world (AIX)old world (AIX)

new world (Exadata)new world (Exadata)new world (Exadata)new world (Exadata)

01/06/2018

13

4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration4 Golden Gate installation & configuration


1-6-201826

Analyse & Analyse & Analyse & Analyse & Architecture GGArchitecture GGArchitecture GGArchitecture GG

Install GGInstall GGInstall GGInstall GGPrepare the Prepare the Prepare the Prepare the

sourcesourcesourcesourceInstantiate Instantiate Instantiate Instantiate

GGGGGGGG

Prepare the Prepare the Prepare the Prepare the targettargettargettarget

• Golden Gate installation & configuration is a multi step approach

• Plan time for every step & substep

• If possible, implement incrementally. No Big Bang approach !!!

01/06/2018

14


1-6-201827

This presentation shares some insight on the GoldenGate implementation. It is not a step by step recipe to follow... For that.....


1-6-201828


Analyse phase :Analyse phase :Analyse phase :Analyse phase :• Which data needs to be replicated (schemas, object types)• What is the replication architecture: unilateral / bilateral, etc.• Is any filtering / transformation required• Type of data changes (DML only or DML and DDL)• What is the allowed latency• Golden Gate usage, DR & High Availability requirements• Are any non supported mechanisms in place, like nologging

operations, transportable tablespaces (TTS), etc

Use Oracle CHECK scripts to investigate source dataUse Oracle CHECK scripts to investigate source dataUse Oracle CHECK scripts to investigate source dataUse Oracle CHECK scripts to investigate source data• Oracle GoldenGate database Schema Profile check script for

Oracle DB (Doc ID 1296168.1). • Oracle GoldenGate database Complete Database Profile check

script for Oracle DB (All Schemas) (Doc ID 1298562.1)

01/06/2018

15


1-6-201829


Results analysis I : Results analysis I : Results analysis I : Results analysis I : many non supported objects, all AQ$ tables

Solution: after discussion with data architects => AQ$ tables are not required in the Golden Gate clone.

Exclude the AQ$ tables from the extract processes:TABLE HR.*;TABLEEXCLUDE HR.*_QUEUE_*

hence the importance of Naming conventions ...


1-6-201830


Results analysis II : Results analysis II : Results analysis II : Results analysis II : Many tables without PMK/UK: 650+ tables !!

Problem with tables without PMK/UK:Problem with tables without PMK/UK:Problem with tables without PMK/UK:Problem with tables without PMK/UK:• On the source DB: Storage & I/O overhead PMK/UK required for

supplemental logging (alternatively all columns will be used )• On the target: Performance overhead when processing updates &

deletes => Full tablescans

Oracle has a solution for this problemOracle has a solution for this problemOracle has a solution for this problemOracle has a solution for this problemHow to Handle Tables Without Primary Keys or Unique Indexes With Oracle GoldenGate (Doc ID 1271578.1)

01/06/2018

16


1-6-201831


Oracle Solution for tables without PMK/UK : Oracle Solution for tables without PMK/UK : Oracle Solution for tables without PMK/UK : Oracle Solution for tables without PMK/UK : How to Handle Tables Without Primary Keys or Unique Indexes With Oracle GoldenGate (Doc ID 1271578.1)

on the source database: add a column & autofill/backfill with unique valueon the source database: add a column & autofill/backfill with unique valueon the source database: add a column & autofill/backfill with unique valueon the source database: add a column & autofill/backfill with unique valueSQL> alter table table_1 add column OGG_KEY_ID raw(16);SQL> alter table table_1 modify OGG_KEY_ID default sys_guid();

Configure this column at GG Level as the unique identifier:Configure this column at GG Level as the unique identifier:Configure this column at GG Level as the unique identifier:Configure this column at GG Level as the unique identifier:GGSCI> add trandata <owner>.<table_name>, COLS (OGG_ KEY_ID), nokey

on the target database: on the target database: on the target database: on the target database: Create a UNIQUE index on this columnCreate a UNIQUE index on this columnCreate a UNIQUE index on this columnCreate a UNIQUE index on this columnSQL> create unique index ix_table_1 on table_1 (OGG_KEY_ID);


1-6-201832


Problem with this solution:

Compilation in many stored procedures: ORA-00947: not enough valuesWe did not even try the Forms & other applications.

This error occurs when you do not specify all the columns in your coding:Bad coding: INSERT INTO TABLE_1 VALUES ('A','B');Good coding: INSERT INTO TABLE_1 (COL1, COL2) VALUE S ('A','B');

SQL> alter table HR.TABLE_1 modify OGG_KEY_ID invis ible

GGSCI> add trandata HR.TABLE_1, COLS (OGG_KEY_ID), nokey

ERROR: Column OGG_KEY_ID not found in table HR.TABLE_1

Oracle Support did not provide a solution. An enhancement request was rejected ....

01/06/2018

17


1-6-201833

Installation Golden Gate: as of version 12.2 via OUI,

Install GGInstall GGInstall GGInstall GG


1-6-201834

• Local software installation, recovery related files on shared storage (HA). Installation binaries about 700 GB.

• After base (12.2) install: look for latest GG patches and apply with Opatch utililty.

• Every GG installation has its own software installation. So on consolidated platforms (Exadata): multiple gg software installations.

• Housekeeping: backup job for software + rotate job for ggserr.log (copy_truncate). GG report files get rotated via parameter configuration. (REPORTROLLOVER)


01/06/2018

18


1-6-201835

Install Golden Gate: recovery related files on shared storage


• Recovery related files are softlinked to shared storage

• Report files & ggserr.log file can grow quite large and are therefore also softlinked to shared storage (with much larger storage capacity)


1-6-201836

Install Golden Gate: multiple software installations on Exadata for GG • GG_HOME 1: /u01/app/oracle/product/12.2/oggcore_1• GG_HOME 2: /u01/app/oracle/product/12.2/oggcore_2

Create a script (gg) to set the GG environment


[oracle@exa01dbadm01 ~]$ gg 1setting gg environment to /u01/app/oracle/product/12.2/oggcore_1

[oracle@exa01dbadm01 ~]$ echo $GG_HOME/u01/app/oracle/product/12.2/oggcore_1

01/06/2018

19


1-6-2018Agenda37



1-6-201838

Prepare the Prepare the Prepare the Prepare the targettargettargettarget

1 Create the target database

2 Prepare database settings for Golden Gate

3 Create GGADMIN user and related objects

4 Start GG Manager process

5 Configure & start the GG Manager process

01/06/2018

20


1-6-201839

Prepare the Prepare the Prepare the Prepare the sourcesourcesourcesource

1 Size archive log destination (increased due to Supplemental Logging

2 Create local trail file (/oracle/trail) destination, ++ sufficient size

3 Prepare database settings for Golden Gate

4 Create GGADMIN user and related objects

5 Enable Supplemental Logging

6 Configure & Start GG Manager process

7 Configure & Start GG Extract process

8 Configure & Start GG Pump process

potential DB Impactpotential DB Impactpotential DB Impactpotential DB Impact

potential DB Impactpotential DB Impactpotential DB Impactpotential DB Impact


1-6-201840


Some details step 3: Prepare settings for source databaseSome details step 3: Prepare settings for source databaseSome details step 3: Prepare settings for source databaseSome details step 3: Prepare settings for source database

SQL> alter database force logging;SQL> alter database add supplemental log data;

SQL> alter system set undo_retention=86400 scope=bo th;SQL> alter system set "_log_read_buffers"=64 scope= spfile; SQL> alter system set "_log_read_buffer_size" = 128 scope=spfile; SQL> alter system set enable_goldengate_replication = true;SQL> alter system set shared_pool_size = <current_v alue + 1250M>;SQL> alter system set sga_target = <current_value + 1250M>;

Database in force logging !! May have effects on batch operations. Check for unrecoverable actions (v$datafile.unrecoverable_change#)

Several parameter changes. More memory required for integrated extract (logminer) processes.

01/06/2018

21


1-6-2018Aenda41


Some details step 5: Supplemental loggingSome details step 5: Supplemental loggingSome details step 5: Supplemental loggingSome details step 5: Supplemental logging

The data stored in the online redo logfiles contain the ROWIDROWIDROWIDROWID of the row being changed and the new values for the column(s) that are changed. The ROWID is used to identify the changed row.

The ROWID is a mapping to a physical block address and cannot be used by Golden Gate to identify the changed row. Therefore, the PMK or UK are added to the online redo logfile as supplemental logging.

Supplemental logging can take place at the table level or at the schema level:

GGSCI> add schematrandata HRGGSCI> add trandata HR.table_1

A DDL lock is required on the table when Supplemental Logging is added. In busy systems: SQL> alter session set ddl_lock_timeout=7200;


1-6-2018Agenda42


Some details step 5: Supplemental loggingSome details step 5: Supplemental loggingSome details step 5: Supplemental loggingSome details step 5: Supplemental logging

Redo size increased by 60% due to Supplemental Logging. Reason is the many tables without PMK / UK

Supplemental logging was (initially) implemented at the SCHEMA level

01/06/2018

22


1-6-201843


Advantages of enabling supplemental logging at the schema levelAdvantages of enabling supplemental logging at the schema levelAdvantages of enabling supplemental logging at the schema levelAdvantages of enabling supplemental logging at the schema level

• Automatically enables supplemental logging for new tables created with a CREATE TABLE DDL command.

• Updates supplemental logging for tables affected by an ALTER TABLE DDL command that adds or drops columns

• Updates supplemental logging for tables affected by RENAME TABLE command

• Updates supplemental logging for tables affected by adding or dropping of unique or primary key constraints


1-6-201844


Some details step 7: Configure the Extract process parameter fileSome details step 7: Configure the Extract process parameter fileSome details step 7: Configure the Extract process parameter fileSome details step 7: Configure the Extract process parameter file

-- configure integrated extract in extract parameter fileTRANLOGOPTIONS INTEGRATEDPARAMS (MAX_SGA_SIZE 1024)

-- include DDL-- MAPPED means we only replicat DDL from the mapped objects-- and we only want tables and indexes, no other obj ect typesDDL &INCLUDE MAPPED OBJTYPE TABLE &INCLUDE MAPPED OBJTYPE INDEX

-- we want all the tables from user HRtable HR.*

-- etc. More specifications & options.

01/06/2018

23


1-6-201845


Some details step 7: Configure & start the Extract ProcessSome details step 7: Configure & start the Extract ProcessSome details step 7: Configure & start the Extract ProcessSome details step 7: Configure & start the Extract Process

GGSCI> dblogin useridalias alias_ggadminSuccessfully logged into database.

GGSCI> register extract X1PROD database2017-09-14 12:43:29 INFO OGG-02003 Extract X1P ROD successfully registered with database at SCN 672577 3753243.

GGSCI> add extract X1PROD, integrated tranlog, begi n now

GGSCI> add exttrail /oracle/trail/PROD/x1, extract X1PROD, megabytes 200

GGSCI> start extract x1prod


1-6-201846


Some details step 7: Local trail files Some details step 7: Local trail files Some details step 7: Local trail files Some details step 7: Local trail files afterafterafterafter Extract is started....Extract is started....Extract is started....Extract is started....

01/06/2018

24


1-6-201847


Some details Step 8, remote trail files Some details Step 8, remote trail files Some details Step 8, remote trail files Some details Step 8, remote trail files afterafterafterafter Pump is configured & started:Pump is configured & started:Pump is configured & started:Pump is configured & started:


1-6-201848


Status of the Golden Gate processes on the source:Status of the Golden Gate processes on the source:Status of the Golden Gate processes on the source:Status of the Golden Gate processes on the source:

GGSCI> info allProgram Status Group Lag at Chkpt T ime Since ChkptMANAGER RUNNINGJAGENT STOPPEDEXTRACT RUNNING P1PROD 00:12:18 0 0:00:04EXTRACT RUNNING X1PROD 00:00:04 0 0:00:15

01/06/2018

25


1-6-201849


After the prepare source operation part of the Golden Gate configuration is up and running:• MGR, Extract & Pump processes running on source• Local trail files are being written• Local trail files are being transferred to remote trail


1-6-201850

Instantiation: inital copy of the source database + replication startInstantiation: inital copy of the source database + replication startInstantiation: inital copy of the source database + replication startInstantiation: inital copy of the source database + replication start

• Several methods available: GG, SQL*Loader, Oracle Datapump• We use Oracle Datapump because of performance• Instantiation needs to be a consistent, automated and well

performing operation:• Functional test environment (source DB) is being cloned from

Production every 3 months, and GG needs to be re-instantiated.• In case of very serious issues, we want to be able to re-

instantiate the Golden Gate Production target quickly

Instantiate Instantiate Instantiate Instantiate GGGGGGGG

ObjectivesObjectivesObjectivesObjectives: • (re)instantiate the target database in maximum 8 hours.

(Source DB is 2.111 GB)• Impact on Production should be as little as possible.• (re)instantiation should run as an automated batch job

No DDL allowed during instantiation !! Prevent error below:ORA-01466: Unable to read data -- Table definition h as changed" (during the export)

01/06/2018

26


1-6-201851

Instantiation architecure: NO (!) impact on source databaseInstantiation architecure: NO (!) impact on source databaseInstantiation architecure: NO (!) impact on source databaseInstantiation architecure: NO (!) impact on source database


EXPDPEXPDPEXPDPEXPDPflashback_scn=12345678flashback_scn=12345678flashback_scn=12345678flashback_scn=12345678

IMPDPIMPDPIMPDPIMPDP

GGSCI> start replicat r2symp, GGSCI> start replicat r2symp, GGSCI> start replicat r2symp, GGSCI> start replicat r2symp, aftercsn aftercsn aftercsn aftercsn 12345678 12345678 12345678 12345678


1-6-2018Agenda52

Datapump Export ImportDatapump Export ImportDatapump Export ImportDatapump Export Import• Optimized for speed• 2 seperate export jobs, parallel 24 (regular & large tables)• Import jobs on 2 seperate Exadata compute nodes• Import data only !• Rebuild indexes & enable constraints (novalidate) afterwards• During import operation: implement Advanced Compression.

(size source: 2.111 MB, size target 1.446 MB)


01/06/2018

27


1-6-201853

Export & import parameter filesExport & import parameter filesExport & import parameter filesExport & import parameter files



1-6-201854

Configure the Replicat parameter file Configure the Replicat parameter file Configure the Replicat parameter file Configure the Replicat parameter file afterafterafterafter the IMPORT operationthe IMPORT operationthe IMPORT operationthe IMPORT operation


01/06/2018

28


1-6-201855

Start the replicat Start the replicat Start the replicat Start the replicat afterafterafterafter the IMPORT operationthe IMPORT operationthe IMPORT operationthe IMPORT operation


GGSCI> register replicat rprod database2017-08-23 13:08:44 INFO OGG-02528 REPLICAT RPROD successfully registered with database as inbound server OGG$RPROD.

GGSCI> add replicat rprod, integrated, exttrail /zs01sn01/gg_prod_trail/prod/x1REPLICAT (Integrated) added.

GGSCI> start replicat rprod, aftercsn 12345678Sending START request to MANAGER ... REPLICAT RPROD starting


1-6-201856

Everything is scripted and automated runs in Batch SchedulerEverything is scripted and automated runs in Batch SchedulerEverything is scripted and automated runs in Batch SchedulerEverything is scripted and automated runs in Batch Scheduler


Batchjobs on source

01/06/2018

29


1-6-201857


Batchjobs on target

5 Findings & Troubleshooting5 Findings & Troubleshooting5 Findings & Troubleshooting5 Findings & Troubleshooting

01/06/2018

30

1-6-201859

5 Findings & troubleshooting5 Findings & troubleshooting5 Findings & troubleshooting5 Findings & troubleshooting

. . . Mitigate the whole chain when issues occur. . . Mitigate the whole chain when issues occur. . . Mitigate the whole chain when issues occur. . . Mitigate the whole chain when issues occur

1-6-2018Agenda60


. . . Document findings & solutions !!!. . . Document findings & solutions !!!. . . Document findings & solutions !!!. . . Document findings & solutions !!!

01/06/2018

31


1-6-201861

Major issue 1: Issues with supplemental logging on the sourceMajor issue 1: Issues with supplemental logging on the sourceMajor issue 1: Issues with supplemental logging on the sourceMajor issue 1: Issues with supplemental logging on the source

ORA-22328: object "HR"."GENERIC_OTC_QUEUE_TABLE" ha s errors. ORA-62009: An attempt to encode invalid XML charact er 0x4 in supplemental log has occurred.

Even though we excluded the Advanced Queuing tables from the Extract process, this error still occurred, because supplemental logging was enabled at the SCHEMA level, including these AQ tables inside the HR schema.

There is NO solution for this problem. You cannot exclude objects when you add supplemental logging at the SCHEMA Level.⇒ Confirmed by Oracle (Service Request)⇒ Enhancement request rejected. (works as designed)


1-6-201862

Major issue 1: Issues with supplemental logging on the sourceMajor issue 1: Issues with supplemental logging on the sourceMajor issue 1: Issues with supplemental logging on the sourceMajor issue 1: Issues with supplemental logging on the source

The best solution would be to move the AQ$ tables to another schema. However, this requires code & application changes & testing. Impact analysis planned for next year.

For now: move from SCHEMA level logging to TABLE level logging. GGSCI> add trandata HR.table_1 (instead of add sche matrandata HR)

In Extract parameter file: map every single table. In stead of HR.*;TABLE HR.table_1;TABLE HR.table_2;.......

Disadvantate of this approach:Disadvantate of this approach:Disadvantate of this approach:Disadvantate of this approach:1. GG extract configuration much more complex. 2000+ tables listed.2. New tables are not automatically added to the GG replication.3. Certain DDL operations are not supported with table level logging:

RENAME table, changes to PMK.

01/06/2018

32


1-6-201863

Major issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UK

• Tables without PMK/UK on source were initially ignored. (No adequate solution with invisible OGG_KEY_ID column)

• However, they cause very bad performance on the target, during updates and deletes: => full tablescans for every (grouped) transaction.


1-6-201864


01/06/2018

33


1-6-201865



1-6-201866


Subsequent table fragmentation causes performance to degrade even worse !!!

01/06/2018

34


1-6-201867


Permanent solution (impact analysis planned for next years program) ⇒ create PMK/UK or implement visible OGG_KEY_ID⇒ Requires code changes and extensive testing

Temporary workarounds 1: exclude tables from the GG replicationTemporary workarounds 1: exclude tables from the GG replicationTemporary workarounds 1: exclude tables from the GG replicationTemporary workarounds 1: exclude tables from the GG replication=> Discussion with data architects.=> Discussion with data architects.=> Discussion with data architects.=> Discussion with data architects.

Temporary workarounds 2: define logical keysTemporary workarounds 2: define logical keysTemporary workarounds 2: define logical keysTemporary workarounds 2: define logical keys=> Discussion with data architect to identify tables=> Discussion with data architect to identify tables=> Discussion with data architect to identify tables=> Discussion with data architect to identify tables

Temporary workarounds 3: use IGNOREDELETES for “snapshot tables”Temporary workarounds 3: use IGNOREDELETES for “snapshot tables”Temporary workarounds 3: use IGNOREDELETES for “snapshot tables”Temporary workarounds 3: use IGNOREDELETES for “snapshot tables”=> Discussion with data architect to identify tables=> Discussion with data architect to identify tables=> Discussion with data architect to identify tables=> Discussion with data architect to identify tables

Temporary workarounds 4: create all column index on remaining tables Temporary workarounds 4: create all column index on remaining tables Temporary workarounds 4: create all column index on remaining tables Temporary workarounds 4: create all column index on remaining tables (on target) (on target) (on target) (on target)


1-6-2018Agenda68


Table Analasis sheet, with colour coded “solutions”/workarounds

01/06/2018

35


1-6-201869

Major issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UK..... Workaround with logical keys..... Workaround with logical keys..... Workaround with logical keys..... Workaround with logical keys

On the source:GGSCI> add trandata HR.TABLE_1 COLS (A,B) NOKEY

In extract parameter file: table HR.TABLE_1 (A,B);

On target:SQL> Create unique index hr.ix_table_1 on hr.table_1 (A,B);

In replicat parameter file:MAP HR.TABLE_1, target HR.TABLE_1, KEYCOLS (A,B);


1-6-201870

Major issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UKMajor issue 2: Very bad performance due to tables without PMK/UK..... Workaround with ignoredeletes..... Workaround with ignoredeletes..... Workaround with ignoredeletes..... Workaround with ignoredeletes

• Only updates and deletes are affected negatively. Inserts are fine• Customer has ‘snapshot’ tables which only contains inserts and

periodic deletes (overnight, cleaning up old entries)

Solution: ignore the deletes and perform them on the target outside of the Golden Gate scope, in a nightly batch. Configure this in the source extract parameter file:

IGNOREDELETESTABLE HR.TABLE_2;TABLE HR.TABLE_3;

01/06/2018

36


. . . Several minor issues, some examples. . .. . . Several minor issues, some examples. . .. . . Several minor issues, some examples. . .. . . Several minor issues, some examples. . .

1-6-201871

Example 1: Performance optimization

Example 2: DDL Error, table with added column

WARNING OGG-01431 Aborted grouped transaction on 'HR.TBBU_REPROCESS_BUYER_STATUSES', Mapping error

Example 3: DDL Error, system named constraints

ERROR OGG-00519 Fatal error executing DDL replication: error [Error code [2443], ORA-02443: Cannot drop constraint -nonexistent constraint SQL alter table "


. . . Several minor issues: example 1 performance optimization. . . Several minor issues: example 1 performance optimization. . . Several minor issues: example 1 performance optimization. . . Several minor issues: example 1 performance optimization

1-6-201872

Replication lag 10+

01/06/2018

37



1-6-201873

Changing parameters

EXTRACT-- to minimize the latency during low volume changesEOFDELAYCSECS 1FLUSHCSECS 1

PUMP-- to minimize the latency during low volume changesEOFDELAYCSECS 1FLUSHCSECS 1

REPLICAT-- to minimize the latency during low volume changesEOFDELAYCSECS 1

Reference: How can I optimize (reduce delay, lag ) throughput from NSK to a target platform in a low volume situation. (Doc ID 1489540.1)



1-6-201874

Replication lag 1+ seconds

01/06/2018

38


. . . Several minor issues: example 2 new column added. . . Several minor issues: example 2 new column added. . . Several minor issues: example 2 new column added. . . Several minor issues: example 2 new column added

1-6-201875

Normally, new columns can be added to a table, and this DDL is captured and applied by Golden Gate.

However, when it concerns a table with no PMK or UK:• as a consequence, all columns are required as supplemental logging. • In those cases, the new column is not automatically added to the

supplemental log group.

Solution:⇒ Temporarily exclude the table from the Replicat (tableexclude

HR.TABLE_1;) and restart the replicat. ⇒ Process all the “bad” trails with the missing column information.⇒ Add the new column as Supplemental Logging on the source⇒ Reinstatiate the new table, remove the tableexclude from the

Replicat and restart the Replicat

WARNING OGG-01431 Aborted grouped transaction on 'HR.TBBU_REPROCESS_BUYER_STATUSES', Mapping error


. . . Several minor issues: example 3 system generated names. . . Several minor issues: example 3 system generated names. . . Several minor issues: example 3 system generated names. . . Several minor issues: example 3 system generated names

1-6-201876

ERROR OGG-00519 Fatal error executing DDL replication: error [Error code [2443], ORA-02443: Cannot drop constraint - nonexistent constraint SQL alter table "

Try to avoid these type of errors by naming your constraints !!

Use Replicat Error Handling to handle this type of errors:-- DDL ErrorsDDLERROR DEFAULT ABENDDDLERROR 2443 IGNORE

Monitor your Replicat report files and discard files errors !!

Oracle claims in an Oracle note that you can treat DML & DDL errors with the same error handling clause (REPERROR). I did not get it to work. In stead:DDL Errors => DDLERROR clauseDML Errors => REPERROR clause

01/06/2018

39


. . . Several minor issues: example 3 system generated names. . . Several minor issues: example 3 system generated names. . . Several minor issues: example 3 system generated names. . . Several minor issues: example 3 system generated names

1-6-201877

Sample section in Replicat parameter file to handle DML errors

6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring6 Golden Gate monitoring

01/06/2018

40


1-6-201879

Standard Oracle toolsStandard Oracle toolsStandard Oracle toolsStandard Oracle tools• GGSCI • Oracle AWR & SPADV reports • Oracle Healthcheck script • Oracle Enterprise Manager & GG Plug-in• Oracle Performance Monitoring Toolkit

Customized tools & reportsCustomized tools & reportsCustomized tools & reportsCustomized tools & reports• Custom process monitoring scripts• Custom heartbeat tables• Custom gg table stats report

1-6-201880

Seperate section in AWR for GoldenGate


. . . . . . . . . . . . Standard Oracle tools: AWR report (local only!)Standard Oracle tools: AWR report (local only!)Standard Oracle tools: AWR report (local only!)Standard Oracle tools: AWR report (local only!)

01/06/2018

41

1-6-201881



1-6-201882



01/06/2018

42


. . . . . . . . . . . . Standard Oracle tools: OGG Healthcheck scriptStandard Oracle tools: OGG Healthcheck scriptStandard Oracle tools: OGG Healthcheck scriptStandard Oracle tools: OGG Healthcheck script

1-6-201883

• Mos Note: GoldenGate Integrated Capture and Integrated Replicat Healthcheck Script (Doc ID 1448324.1)

• Script did not run at first. Script hang. Comment out alert log check:

Whenever you log an SR with Oracle they often ask you to run the healthcheck script and upload the results (html file).



1-6-201884

header information

database information

process state information

sys checks

configuration checks

performance checks & advice

rep errors, conflicts & collisions

table statistics

01/06/2018

43



1-6-201885

Output above based on view gv$goldengate_table_stats


. . . . . . . . . . . . Oracle Enterprise Manager and GoldenGate plugOracle Enterprise Manager and GoldenGate plugOracle Enterprise Manager and GoldenGate plugOracle Enterprise Manager and GoldenGate plug----inininin

1-6-201886

Reference: https://docs.oracle.com/goldengate/em1311/gg-emplugin/EMGGP/toc.htm

Golden Gate has it’s own agent (JAGENT), which communicates with OEM Agent.Every Golden Gate configuration has it’s own JAGENT installed.

Requires GG Management Pack license

01/06/2018

44



1-6-201887

Reference: https://docs.oracle.com/goldengate/em1311/gg-emplugin/EMGGP/toc.htm

Step 1: Download, Install & Deploy GG Plugin to OMS and GG targets

Step 2: Install & configure GG agents (JAGENT) on target nodes

Step 3: Discover GG plugins in OEM Cloud Control



1-6-201888

01/06/2018

45


. . . . . . . . . . . . GoldenGate Monitoring Performance ToolkitGoldenGate Monitoring Performance ToolkitGoldenGate Monitoring Performance ToolkitGoldenGate Monitoring Performance Toolkit

1-6-201889



1-6-201890

01/06/2018

46



1-6-201891


. . . . . . . . . . . . Custom tooling & reportsCustom tooling & reportsCustom tooling & reportsCustom tooling & reports

1-6-201892

Challenge:• Report on current Golden Gate status• Alert if the replication lag increases or GG processes are down• No access (as Oracle) to source database server. Only PUID access

Detailed work instruction in HandoutDetailed work instruction in HandoutDetailed work instruction in HandoutDetailed work instruction in Handout

Customization 1: making reports accessible for PUIDCustomization 2: remote monitoringCustumization 3: monitoring replication lagCustomization 4: reporting on Golden Gate Statistics

01/06/2018

47


. . . . . . . . . . . . Customization 1: making reports availableCustomization 1: making reports availableCustomization 1: making reports availableCustomization 1: making reports available

1-6-201893

-- parameter setting in ./GLOBALS on source -- allows user read access to the report files-- overriding the default (026)OUTPUTFILEUMASK 022


. . . . . . . . . . . . Customization 1: making reports availableCustomization 1: making reports availableCustomization 1: making reports availableCustomization 1: making reports available

1-6-201894

01/06/2018

48


. . . . . . . . . . . . Customization 2: remote monitoring Customization 2: remote monitoring Customization 2: remote monitoring Customization 2: remote monitoring

1-6-201895

AIX

/home/oracle/scripts/gg_status2.sh

Exadata

/home/oracle/scripts/gg_status2.sh

OS scripts that run GGSCI commands

--

--PROD

--

--prodgg

External tables call OS scripts:preprocessor GG_DIR2:'gg_status_02.sh '

SQL> querySQL> querySQL> querySQL> query

Advantage of this setup: we do not need access (as oracle) the source system to view the Advantage of this setup: we do not need access (as oracle) the source system to view the Advantage of this setup: we do not need access (as oracle) the source system to view the Advantage of this setup: we do not need access (as oracle) the source system to view the current GG statuscurrent GG statuscurrent GG statuscurrent GG status

DB LINK GG_SYMPUser GGADMIN_READ with Read Acces to External Table



. . . . . . . . . . . . Customization 2 : remote monitoringCustomization 2 : remote monitoringCustomization 2 : remote monitoringCustomization 2 : remote monitoring

1-6-201896

01/06/2018

49


. . . . . . . . . . . . Customization 2 : remote Customization 2 : remote Customization 2 : remote Customization 2 : remote monitoring, part of reportmonitoring, part of reportmonitoring, part of reportmonitoring, part of report

1-6-201897


. . . . . . . . . . . . Customization 3 : heartbeat tables Customization 3 : heartbeat tables Customization 3 : heartbeat tables Customization 3 : heartbeat tables

1-6-201898

01/06/2018

50


. . . . . . . . . . . . Customization 3 : heartbeat tables Customization 3 : heartbeat tables Customization 3 : heartbeat tables Customization 3 : heartbeat tables

1-6-2018Agenda99

On the target: • The INSERT_ON_TARGET Date/Time is recorded as part of the heartbeat• This allows queries to see the Replication lag:


. . . . . . . . . . . . Customization 3 : heartbeat tables Customization 3 : heartbeat tables Customization 3 : heartbeat tables Customization 3 : heartbeat tables , part of report, part of report, part of report, part of report

1-6-2018100

01/06/2018

51


. . . . . . . . . . . . Customization 3 : heartbeat tables implementationCustomization 3 : heartbeat tables implementationCustomization 3 : heartbeat tables implementationCustomization 3 : heartbeat tables implementation

1-6-2018101

ON THE SOURCE• Create Heartbeat tables on source with an automatically updated

insert_on_source date/time column• Create some dependant objects (sequences). • Create DBMS_SCHEDULER_JOBS to insert records every minute

ON THE TARGET• Create a trigger to update the insert_on_target date/time column• Enable the trigger (triggers are by default disabled by GoldenGate)• Create some reports based on the hearbeat tables


. . . . . . . . . . . . Customization 4 : GoldenGate table statisticsCustomization 4 : GoldenGate table statisticsCustomization 4 : GoldenGate table statisticsCustomization 4 : GoldenGate table statistics

1-6-2018102

• Good table update statistics are hard to find

• GGSCI interfrace quite primitive and insufficient

• Oracle ships the licensed Veridata product for data reporting and comparison, but very expensive

• Currently I am working on a customized solution using view GV$GOLDENGATE_TABLE_STATS. This view is also used by the Oracle Healthcheck Script.

01/06/2018

52

7 Golden Gate High Availability7 Golden Gate High Availability7 Golden Gate High Availability7 Golden Gate High Availability


. . . . . . . . . . . . OverviewOverviewOverviewOverview

1-6-2018104

HA Source System : with startup script

HA Remote Trail file System : clustered ZFS

HA Target System : Oracle Clusterware

01/06/2018

53


. . . Relevant Golden Gate processes for High Availability. . . Relevant Golden Gate processes for High Availability. . . Relevant Golden Gate processes for High Availability. . . Relevant Golden Gate processes for High Availability

1-6-2018105

• The only process we have to care about (when configured correctly) is the ManagerManagerManagerManager process

• The Extract, Pump and Replicat Extract, Pump and Replicat Extract, Pump and Replicat Extract, Pump and Replicat processes are managed by the Manager process (via AUTOSTART & AUTORESTART parameter)

• But … the manager process should be managed by some external mechanism

• On AIX: startup scripts• On Exadata: by Oracle Clusterware


. . . Relevant Golden Gate processes for High Availability. . . Relevant Golden Gate processes for High Availability. . . Relevant Golden Gate processes for High Availability. . . Relevant Golden Gate processes for High Availability

1-6-2018106

• The only process we have to care about (when configured correctly) is the ManagerManagerManagerManager process

• The Extract, Pump and Replicat Extract, Pump and Replicat Extract, Pump and Replicat Extract, Pump and Replicat processes are managed by the Manager process

• But … the manager process should be managed by some external mechanism

• On AIX: startup scripts• On Exadata: by Oracle Clusterware

01/06/2018

54


. . . Golden Gate extract and replicat processes. . . Golden Gate extract and replicat processes. . . Golden Gate extract and replicat processes. . . Golden Gate extract and replicat processes

1-6-2018107

• The GG Manager (can) start / restart the Extract and Replicat processes.

• This is configured with parameters in the MANAGER parameter file:

AUTOSTART ER *-- Starts Extract and Replicat processes when Manage r starts

AUTORESTART ER *, retries 12, WAITMINUTES 5, RESETM INUTES 60-- restarts Extract and Replicat processes again aft er abnormal termination.

HA events are recorded in the ggserr.log file and m anager report file:


. . . Golden Gate High Availability on the target (Exadata). . . Golden Gate High Availability on the target (Exadata). . . Golden Gate High Availability on the target (Exadata). . . Golden Gate High Availability on the target (Exadata)

1-6-2018108

Application VIPApplication VIPApplication VIPApplication VIP• Separate IP Address on public network (10 Gb/s)• Added to DNS as dm1db01-appvip

Application resource ggate_prod• hard dependency on Application VIP (pullup,

shutdown)• action script to stop/start/check gg processes

01/06/2018

55


. . . Clusterware configuration steps . . . Clusterware configuration steps . . . Clusterware configuration steps . . . Clusterware configuration steps

1-6-2018109

Step 1: Make sure certain data (like parameter files, checkpoint files, process files, etc) is on a shared location.

Step 2: Create & Start Application VIP (IP address & name must be entered in DNS)

Step 3: Create & Test GG Action Script (to be used by GG Application Resource

Step 4: Create & Start GG Application Resource

Step 5: Reconfigure GG Pump Parameter:

RMTHOST dm1db01-appvip, MGRPORT 7810, TCPBUFSIZE 10485760, TCPFLUSHBYTES 10485760



. . . Golden Gate Failover steps. . . Golden Gate Failover steps. . . Golden Gate Failover steps. . . Golden Gate Failover steps

1-6-2018110

1 Check current status of cluster resources

2 Check current status of GG processes

01/06/2018

56



1-6-2018111

3 Perform the failover

4 Check the GG processes

The Golden Gate PUMP process on the source will temporarily ABEND, but will be restarted automatically by the Manager process on the source.


. . . Golden Gate Failover, confusing ...... . . Golden Gate Failover, confusing ...... . . Golden Gate Failover, confusing ...... . . Golden Gate Failover, confusing .....

1-6-2018112

MGR and Replicat processes appear to be running on both nodes simultaneously ?????

01/06/2018

57


. . . Golden Gate Failover, confusing ...... . . Golden Gate Failover, confusing ...... . . Golden Gate Failover, confusing ...... . . Golden Gate Failover, confusing .....

1-6-2018113

The GGSCI Interface uses the shared checkpoint and process locations on the ZFS-SA to display this information. The ACTUAL GG processes are only running on only one node

Ps –ef|grep mgr on first node

Ps –ef|grep mgr on second node



1-6-2018114

3 Perform the failover

4 Check the GG processes

The Golden Gate PUMP process on the source will temporarily ABEND, but will be restarted automatically by the Manager process on the source.

01/06/2018

58

8 Lessons Learned8 Lessons Learned8 Lessons Learned8 Lessons Learned

Lessons Learned 1: DBMA not for every DBALessons Learned 1: DBMA not for every DBALessons Learned 1: DBMA not for every DBALessons Learned 1: DBMA not for every DBA

The Exadata Platform requires a scaling up and scaling out of the

traditional Oracle DBA activities:

• Scaling up: RAC, RMAN, +ASM, Dataguard, OEM++

• Scaling out:

□ Exadata (& optional ZFS-SA) Storage Management

□ Exadata network management (Private, Public, InfiniBand)

□ Exadata platform management (sysadmin skills)

Not every DBA is capable to adapt those changes.

1-6-2018116

01/06/2018

59

Lessons Learned 2: Make use of existing structuresLessons Learned 2: Make use of existing structuresLessons Learned 2: Make use of existing structuresLessons Learned 2: Make use of existing structures

Make use of existing structures & procedures already in place at

the customer site.

• It will save you work

• It will change “opponents” into supporters

Examples applied in this project:

• Existing snapshot & cloning procedures were used for GG

database instantiation

• Batch scheduling team was asked to run & monitor GG batches

1-6-2018117

Lessons Learned 3: Make use of Oracle Support !!Lessons Learned 3: Make use of Oracle Support !!Lessons Learned 3: Make use of Oracle Support !!Lessons Learned 3: Make use of Oracle Support !!

• When working on technical issues, make use of Oracle Support

to increase your project team.

• Once in place, it only takes about 20 minutes to create a

Service Request.

• Quality of Oracle Support analysists varies heavily.

(example: one SR (not being able to discover GG in OEM) took 2 months

with hardly any progress. Then support analyst went on holiday and

colleague took over. Within a few days the SR was resolved.

1-6-2018118

01/06/2018

60

Lessons Learned 4: Document the solution !!Lessons Learned 4: Document the solution !!Lessons Learned 4: Document the solution !!Lessons Learned 4: Document the solution !!

1-6-2018119

Preferably on a WIKI

Lessons Learned 5: Data analysis of source DB !!Lessons Learned 5: Data analysis of source DB !!Lessons Learned 5: Data analysis of source DB !!Lessons Learned 5: Data analysis of source DB !!

1-6-2018120

Perform the data analysis of the source DB first, with Oracle provided scripts. Reports on Golden Gate challenges:• Non supported objects (like AQ$ tables)• Tables without PMK/UK• Non supported datatypes (like IDENTITY columns)• Etc.

Try and avoid making promises up front, do a POC

Contact the data administrators who know their data and discuss the outcome and possible solutions with them.

References:Oracle GoldenGate database Schema Profile check script for Oracle DB (Doc ID 1296168.1). ScriptOracle GoldenGate database Complete Database Profile check script for Oracle DB (All Schemas) (Doc ID 1298562.1) GoldenGate Integrated Capture and Integrated Replicat Healthcheck Script (Doc ID 1448324.1

01/06/2018

61

Lessons Learned 6: Incremental approachLessons Learned 6: Incremental approachLessons Learned 6: Incremental approachLessons Learned 6: Incremental approach

1-6-2018121

If possible, choose an incremental approach and evaluate every step over time:• Place source DB in archive log & force logging mode• Add supplemental logging to the source DB• Configure and start the extract process• Configure and start the pump process• Instantiate the target database• Automate the instantiation• Configure High Availability• Configure monitoring & reporting

• Configure the TESTTESTTESTTEST environment first, then PRODUCTIONPRODUCTIONPRODUCTIONPRODUCTION

Lessons Learned 7: Take time to work out the solutionLessons Learned 7: Take time to work out the solutionLessons Learned 7: Take time to work out the solutionLessons Learned 7: Take time to work out the solution

1-6-2018122

Take time to work out the solution.

As soon as you are outside the “Golden Gate sweetspot”, you are much on your own, with little to no help from Oracle Support.

Documentation on internet/My Oracle Support hard to find. Available documentation just touches the surface.

01/06/2018

62

Lessons Learned 8: Supersize if possibleLessons Learned 8: Supersize if possibleLessons Learned 8: Supersize if possibleLessons Learned 8: Supersize if possible

1-6-2018123

Try and supersize your Golden Gate trail file locations, both the local trail and the remote trail.

This will allow you room to manage Golden Gate issues. While you fix the issues, there is still sufficient storage for Golden Gate to keep on shipping trail files.

In this Customer setup:• GG local trail file location with 200 GB of storage. Enough for +10

days of data changes• GG remote trail file location of “unlimited” storage capacity (hosted

on ZFS Storage Appliance).

.

Lessons Learned 9: GoldenGate stabilityLessons Learned 9: GoldenGate stabilityLessons Learned 9: GoldenGate stabilityLessons Learned 9: GoldenGate stability

1-6-2018124

GoldenGate Extract and Pump processes turned out to be VERY stable.

GoldenGate Manager process monitors other processes VERY succesfully. (For example AUTORESTART Pump after network issue)

Most problems occur with the GoldenGate Replicat processes.⇒ Important to configure CDR (Conflict Detection & Resolution) in

Replicat parameter file⇒ Important to have sufficient remote trail file storage. While error

is being analyzed and resolved, remote trail file processing stalls and new trails keep coming !

01/06/2018

63

Lessons Learned 10: Reporting is importantLessons Learned 10: Reporting is importantLessons Learned 10: Reporting is importantLessons Learned 10: Reporting is important

1-6-2018125

Customers like to know the status of their GG replication. Take time to build some custom reports:• Golden Gate availability• Golden Gate lag (latency) => use the Heartbeat tables• Golden Gate table stats => use GV$GOLDENGATE_TABLE_STATS

Do not promise the impossible! For real up to date reporting, products like GoldenGate Veridata need to be purchased.

Lessons Learned 11: develop GG standardsLessons Learned 11: develop GG standardsLessons Learned 11: develop GG standardsLessons Learned 11: develop GG standards

1-6-2018126

DBA and developers should be aware of the consequences of GoldenGate and adopt their standards:• No NOLOGGING operations allowed• No Transportable Tablespace methods allowed• Always create tables with PMK or Unique Key• Make sure all new objects within the replicated schema(s) are

supported by GoldenGate• Do not use objects that are not supported by GoldenGate, like AQ$

tables and IDENTITY columns• Name your constraints and indexes, avoid system generated names.• Etc.

01/06/2018

64

End of PresentationEnd of PresentationEnd of PresentationEnd of Presentation

1-6-2018Agenda127

Oracle Golden Gate - nlOUG · Oracle Golden Gate. . . from AIX to Exadata . . . Even voorstellen...

Documents

Transcript of Oracle Golden Gate - nlOUG · Oracle Golden Gate. . . from AIX to Exadata . . . Even voorstellen...