Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

86
Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies

Transcript of Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

Page 1: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

Experiences and Lessons Learned

6-Node Active-Active Oracle GoldenGate

Luke Davies

Page 2: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Why Pythian• Recognized Leader:

• Global industry leader in data infrastructure managed services and consulting with expertise in Oracle, Oracle Applications, Microsoft SQL Server, MySQL, big data and systems administration

• Work with over 200 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments

• Expertise:

• One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 8 Oracle ACEs/ACE Directors

• Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC

• Global Reach & Scalability:

• 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response

Page 3: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

About Luke Davies• Team Technical Lead• Oracle DBA since V6 (1990)• BSc Microbiology• With Pythian since 2008• Main Focus

• Oracle Core• Oracle Performance Tuning

Page 4: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

Set Up

6-Node Active-Active Oracle GoldenGate

Page 5: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Set Up• Business requirements

• High availability

• Easily Scalable

• Application releases with zero downtime

• Database patches/upgrades with zero downtime

• Real-time reporting database with no impact to Online users

• Other application entry points with no impact to Online users

• Original Solution based on • Oracle Standard Edition 10gR2

• GoldenGate V10.4

Page 6: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Set Up

Page 7: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Set Up

Page 8: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

Experiences and Lessons Learned

6-Node Active-Active Oracle GoldenGate

Page 9: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Topics• Active/Active considerations

• Supplemental Logging• DDL Replication• Conflict Avoidance• Conflict Detection• Release Management• Data comparison

• General considerations• Trail file deletion• Archive log deletion

Page 10: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging• Required at the database level (Minimum Level)

• ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

• Minimal Supplemental Logging

“logs the minimal amount of information needed for LogMiner to identify, group, and merge the redo operations associated with DML changes”

• Identification key logging (PK, UK, FK)• Table Level • Database Level• Method

• SQL e.g. ALTER TABLE <> ADD SUPPLEMENTAL LOG DATA …

• GoldenGate e.g. ADD TRANDATA <Table Name>

• Cost of Omission

Changes may fail

Page 11: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental LoggingSource Tg

t

create table repuser2.tbl4 (id number, fname varchar2(100), lname varchar2(100), issued date);

2012-11-14 09:20:57 INFO OGG-00487 Oracle GoldenGate Capture for Oracle, ext_nb.prm: DDL operation included [INCLUDE ALL OBJNAME REPUSER2.*], optype [CREATE], objtype [TABLE], objowner [REPUSER2], objname [TBL4].

2012-11-14 09:20:57 INFO OGG-00497 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Writing DDL operation to extract trail file.

2012-11-14 09:21:03 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN DATA for table [REPUSER2.TBL4], operation [ALTER TABLE "REPUSER2"."TBL4" ADD SUPPLEMENTAL LOG GROUP "GGT_61642" ("ID","FNAME","LNAME","ISSUED") ALWAYS /* GOLDENGATE_DDL_REPLICATION */ (size 141)].

2012-11-14 09:21:04 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN DATA for table [REPUSER2.TBL4], operation [ALTER TABLE "REPUSER2"."TBL4" ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS /* GOLDENGATE_DDL_REPLICATION */ (size 103)].

2012-11-14 09:21:04 INFO OGG-00479 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully deleted TRAN DATA for [CREATE] DDL operation, table [REPUSER2.TBL4], operation [ALTER TABLE "REPUSER2"."TBL4" DROP SUPPLEMENTAL LOG GROUP "GGT_61642" /* GOLDENGATE_DDL_REPLICATION */ (size 103)].

select * from dba_log_groups where table_name='TBL4';

OWNER LOG_GROUP_NAME TABLE_NAME LOG_GROUP_TYPE ALWAYS GENERATED----- -------------- ---------- -------------- ------ ---------REPUSER2 GGS_61642 TBL4 USER LOG GROUP ALWAYS USER NAME

Drop SL for all namedcolumn

s

Page 12: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental LoggingSrc Target

2012-11-14 09:21:05 INFO OGG-00482 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL found, operation [create table repuser2.tbl4 (id number, fname varchar2(100), lname varchar2(100), issued date) (size 95)].

2012-11-14 09:21:05 INFO OGG-00489 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL is of

mapped scope, after mapping new operation [create table REPUSER2.TBL4 (id number, fname varchar2(100), lname varchar2(100), issued date) (size 95)].

2012-11-14 09:21:05 INFO OGG-01407 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Setting current schema for DDL operation to [SYS].

2012-11-14 09:21:05 INFO OGG-00484 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Executing DDL operation.

2012-11-14 09:21:06 INFO OGG-00483 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL operation successful.

2012-11-14 09:21:06 INFO OGG-01408 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Restoring current schema for DDL operation to [ggsdb].

desc repuser2.tbl4

Name Null? TypeID NUMBERFNAME VARCHAR2(100)LNAME VARCHAR2(100)ISSUED DATE

No SLIn

Target

TBL4in

Target

Page 13: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental LoggingSource Tg

t

alter table repuser2.tbl4 add primary key(id);

2012-11-14 09:51:07 INFO OGG-01487 Oracle GoldenGate Capture for Oracle, ext_nb.prm: DDL found, operation [alter table repuser2.tbl4 add primary key(id) (size 47)], start SCN [9690796], commit SCN [9690813] instance [orcl1123 (1)], DDL seqno [1008], marker seqno [1008].

2012-11-14 09:51:07 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN DATA for table [REPUSER2.TBL4], operation [ALTER TABLE "REPUSER2"."TBL4" ADD SUPPLEMENTAL LOG GROUP "GGT_61642" ("ID") ALWAYS /* GOLDENGATE_DDL_REPLICATION */ (size 116)].

2012-11-14 09:51:07 INFO OGG-00479 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully deleted TRAN DATA for [ALTER] DDL operation, table [REPUSER2.TBL4], operation [ALTER TABLE "REPUSER2"."TBL4" DROP SUPPLEMENTAL LOG DATA (ALL) COLUMNS /* GOLDENGATE_DDL_REPLICATION */ (size 104)].

2012-11-14 09:51:07 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN DATA for table [REPUSER2.TBL4], operation [ALTER TABLE "REPUSER2"."TBL4" ADD SUPPLEMENTAL LOG GROUP "GGS_61642" ("ID") ALWAYS /* GOLDENGATE_DDL_REPLICATION */ (size 116)].

2012-11-14 09:51:07 INFO OGG-00479 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully deleted TRAN DATA for [ALTER] DDL operation, table [REPUSER2.TBL4], operation [ALTER TABLE "REPUSER2"."TBL4" DROP SUPPLEMENTAL LOG GROUP "GGT_61642" /* GOLDENGATE_DDL_REPLICATION */ (size 103)].

select * from dba_log_group_columns where table_name='TBL4';

OWNER LOG_GROUP_NAME TABLE_NAME COLUMN_NAME POSITION LOGGING_PROPERTY----- -------------- ---------- ----------- -------- ----------------REPUSER2 GGS_61642 TBL4 ID 1 LOG

SL for all

columns is

dropped

SL for IDis

added(PK)

Page 14: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental LoggingSource

Target

2012-11-14 09:51:11 INFO OGG-00482 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL

found, operation [alter table repuser2.tbl4 add primary key(id) (size 47)].

2012-11-14 09:51:11 INFO OGG-00489 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL is of

mapped scope, after mapping new operation [alter table REPUSER2.TBL4 add primary key(id) (size 47)].

2012-11-14 09:51:11 INFO OGG-01407 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Setting current schema for DDL operation to [SYS].

2012-11-14 09:51:11 INFO OGG-00484 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Executing DDL operation.

2012-11-14 09:51:14 INFO OGG-00483 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL operation successful.

2012-11-14 09:51:14 INFO OGG-01408 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Restoring current schema for DDL operation to [ggsdb].

select * from dba_log_groups where table_name='TBL4';

no rows

NO SLIn

TargetYET !!

Page 15: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

T Source Target

t1 insert into repuser2.tbl4 values (1,'Test','TestL',sysdate);insert into repuser2.tbl4 values (2,'Test','TestL2',sysdate);insert into repuser2.tbl4 values (3,'Test','TestL3',sysdate);

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12 AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12 AAAPDKAAEAAAAIOAAC 3 Test TestL3 14/11/12

t2

What if we keep running replication this way

Page 16: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

T Source Target

t1 insert into repuser2.tbl4 values (1,'Test','TestL',sysdate);insert into repuser2.tbl4 values (2,'Test','TestL2',sysdate);insert into repuser2.tbl4 values (3,'Test','TestL3',sysdate);

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12 AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12 AAAPDKAAEAAAAIOAAC 3 Test TestL3 14/11/12

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestL3 14/11/12

t2

What if we keep running replication this way

Page 17: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

T Source Target

t1 insert into repuser2.tbl4 values (1,'Test','TestL',sysdate);insert into repuser2.tbl4 values (2,'Test','TestL2',sysdate);insert into repuser2.tbl4 values (3,'Test','TestL3',sysdate);

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12 AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12 AAAPDKAAEAAAAIOAAC 3 Test TestL3 14/11/12

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestL3 14/11/12

t2 update repuser2.tbl4 set lname='TestL4' where fname='Test' and rowid='AAAar/AAEAAAE5HAAC‘;

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestL4 14/11/12

What if we keep running replication this way

Page 18: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

T Source Target

t1 insert into repuser2.tbl4 values (1,'Test','TestL',sysdate);insert into repuser2.tbl4 values (2,'Test','TestL2',sysdate);insert into repuser2.tbl4 values (3,'Test','TestL3',sysdate);

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12 AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12 AAAPDKAAEAAAAIOAAC 3 Test TestL3 14/11/12

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestL3 14/11/12

t2 UPDATE "REPUSER2"."TBL4" SET "LNAME"=:a1 WHERE "ID"=:b0OCI Error ORA-01403: no data found, SQL <UPDATE "REPUSER2"."TBL4" SET "LNAME" = :a1 WHERE "ID" = :b0>Operation failed at seqno 5 rba 4598Discarding record on action DISCARD on error 1403Problem replicating REPUSER2.TBL4 to REPUSER2.TBL4Record not foundMapping problem with compressed update record (target format)...*ID =LNAME = TestL4

update repuser2.tbl4 set lname='TestL4' where fname='Test' and rowid='AAAar/AAEAAAE5HAAC‘;

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestL4 14/11/12

What if we keep running replication this way

MissingValue!

Page 19: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

T Source Target

t1

insert into repuser2.tbl4 values (1,'Test','TestL',sysdate);insert into repuser2.tbl4 values (2,'Test','TestL2',sysdate);insert into repuser2.tbl4 values (3,'Test','TestL3',sysdate);

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12 AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12 AAAPDKAAEAAAAIOAAC 3 Test TestL3 14/11/12

t2

What if supplemental logging is enabled on target

Page 20: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

T Source Target

t1

insert into repuser2.tbl4 values (1,'Test','TestL',sysdate);insert into repuser2.tbl4 values (2,'Test','TestL2',sysdate);insert into repuser2.tbl4 values (3,'Test','TestL3',sysdate);

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12 AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12 AAAPDKAAEAAAAIOAAC 3 Test TestL3 14/11/12

add trandata repuser2.tbl4

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestL3 14/11/12

t2

What if supplemental logging is enabled on target

Page 21: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

T Source Target

t1

insert into repuser2.tbl4 values (1,'Test','TestL',sysdate);insert into repuser2.tbl4 values (2,'Test','TestL2',sysdate);insert into repuser2.tbl4 values (3,'Test','TestL3',sysdate);

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12 AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12 AAAPDKAAEAAAAIOAAC 3 Test TestL3 14/11/12

add trandata repuser2.tbl4

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestL3 14/11/12

t2

update repuser2.tbl4 set lname='TestLB' where fname='Test' and rowid='AAAar/AAEAAAE5HAAC‘;

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestLB 14/11/12

What if supplemental logging is enabled on target

Page 22: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

T Source Target

t1

insert into repuser2.tbl4 values (1,'Test','TestL',sysdate);insert into repuser2.tbl4 values (2,'Test','TestL2',sysdate);insert into repuser2.tbl4 values (3,'Test','TestL3',sysdate);

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12 AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12 AAAPDKAAEAAAAIOAAC 3 Test TestL3 14/11/12

add trandata repuser2.tbl4

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestL3 14/11/12

t2

UPDATE "REPUSER2"."TBL4" SET "LNAME"=:a1 WHERE "ID"=:b0a1 : TestLBb0 : 3

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12 AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12 AAAPDKAAEAAAAIOAAC 3 Test TestLB 14/11/12

update repuser2.tbl4 set lname='TestLB' where fname='Test' and rowid='AAAar/AAEAAAE5HAAC‘;

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestLB 14/11/12

What if supplemental logging is enabled on target

Page 23: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

T Source Target

t1

insert into repuser2.tbl4 values (1,'Test','TestL',sysdate);insert into repuser2.tbl4 values (2,'Test','TestL2',sysdate);insert into repuser2.tbl4 values (3,'Test','TestL3',sysdate);

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------- --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12AAAPDKAAEAAAAIOAAC 3 Test TestL3 14/11/12

t2

What if supplemental logging is enabled on DB level

Page 24: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

T Source Target

t1

insert into repuser2.tbl4 values (1,'Test','TestL',sysdate);insert into repuser2.tbl4 values (2,'Test','TestL2',sysdate);insert into repuser2.tbl4 values (3,'Test','TestL3',sysdate);

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------- --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12AAAPDKAAEAAAAIOAAC 3 Test TestL3 14/11/12

delete trandata repuser2.tbl4alter database add supplemental log data (primary key) columns;alter system switch logfile;

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------- ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12 AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestL3 14/11/12

t2

What if supplemental logging is enabled on DB level

Page 25: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

T Source Target

t1

insert into repuser2.tbl4 values (1,'Test','TestL',sysdate);insert into repuser2.tbl4 values (2,'Test','TestL2',sysdate);insert into repuser2.tbl4 values (3,'Test','TestL3',sysdate);

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------- --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12AAAPDKAAEAAAAIOAAC 3 Test TestL3 14/11/12

delete trandata repuser2.tbl4alter database add supplemental log data (primary key) columns;alter system switch logfile;

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------- ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12 AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestL3 14/11/12

t2

update repuser2.tbl4 set lname='TestLC’ where fname='Test' and rowid=‘AAAar/AAEAAAE5HAAC’;

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------- ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12 AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestLC 14/11/12

What if supplemental logging is enabled on DB level

Page 26: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

T Source Target

t1

insert into repuser2.tbl4 values (1,'Test','TestL',sysdate);insert into repuser2.tbl4 values (2,'Test','TestL2',sysdate);insert into repuser2.tbl4 values (3,'Test','TestL3',sysdate);

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------- --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12AAAPDKAAEAAAAIOAAC 3 Test TestL3 14/11/12

delete trandata repuser2.tbl4alter database add supplemental log data (primary key) columns;alter system switch logfile;

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------- ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12 AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestL3 14/11/12

t2

UPDATE "REPUSER2"."TBL4" SET "LNAME"=:a1 WHERE "ID"=:b0a1 : TestLCb0 : 3

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------- --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12AAAPDKAAEAAAAIOAAC 3 Test TestLC 14/11/12

update repuser2.tbl4 set lname='TestLC’ where fname='Test' and rowid=‘AAAar/AAEAAAE5HAAC’;

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------- ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12 AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestLC 14/11/12

What if supplemental logging is enabled on DB level

Page 27: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

Supplemental loggingSupplemental logging

New object

Existing

object

Supplemental logging is critical to healthy replication specially for tables with update/delete changes

PK

UK

KEYCOLs

ALL

Page 28: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

Supplemental loggingSupplemental logging

New object

Existing

object

PK

UK

KEYCOLs

DDLOPTIONS &ADDTRANDATA &GETREPLICATES &

REPORT

To add SL to source DB

To replicat add SL to target

ALL

Page 29: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental LoggingTime

Source

t1 create table repuser2.tbl11 (id number, name varchar2(10));

2012-11-09 14:58:58 INFO OGG-01487 Oracle GoldenGate Capture for Oracle, ext_nb.prm: DDL found, operation

[create table repuser2.tbl11 (id number, name varchar2(10))2012-11-09 14:58:59 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN

DATA for table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS /* GOLDENGATE_DDL_REPLICATION */ (size 104)].

t2

Page 30: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental LoggingTime

Source

t1 create table repuser2.tbl11 (id number, name varchar2(10));

2012-11-09 14:58:58 INFO OGG-01487 Oracle GoldenGate Capture for Oracle, ext_nb.prm: DDL found, operation

[create table repuser2.tbl11 (id number, name varchar2(10))2012-11-09 14:58:59 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN

DATA for table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS /* GOLDENGATE_DDL_REPLICATION */ (size 104)].

t2 alter table repuser2.tbl11 add primary key (id);

2012-11-09 15:00:04 INFO OGG-01487 Oracle GoldenGate Capture for Oracle, ext_nb.prm: DDL found, operation

[alter table repuser2.tbl11 add primary key (id) (2012-11-09 15:00:04 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN

DATA for table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG GROUP "GGT_61549" ("ID") ALWAYS /* GOLDENGATE_DDL_REPLICATION */ (size 117)].

2012-11-09 15:00:04 INFO OGG-00479 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully deleted TRAN

DATA for [ALTER] DDL operation, table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" DROP SUPPLEMENTAL LOG DATA (ALL) COLUMNS /* GOLDENGATE_DDL_REPLICATION */ (size 105)].

2012-11-09 15:00:04 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN

DATA for table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG GROUP "GGS_61549" ("ID") ALWAYS /* GOLDENGATE_DDL_REPLICATION */ (size 117)].

2012-11-09 15:00:04 INFO OGG-00479 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully deleted TRAN

DATA for [ALTER] DDL operation, table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" DROP SUPPLEMENTAL LOG GROUP "GGT_61549" /* GOLDENGATE_DDL_REPLICATION */ (size 104)].

Page 31: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental LoggingSource

Target

2012-11-14 09:51:11 INFO OGG-00482 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL

found, operation [alter table repuser2.tbl11 add primary key(id) (size 47)].

2012-11-14 09:51:11 INFO OGG-00489 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL is of

mapped scope, after mapping new operation [alter table REPUSER2.TBL11 add primary key(id) (size 47)].

2012-11-14 09:51:11 INFO OGG-01407 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Setting current schema for DDL operation to [SYS].

2012-11-14 09:51:11 INFO OGG-00484 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Executing DDL operation.

2012-11-14 09:51:14 INFO OGG-00483 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL operation successful.

2012-11-14 09:51:14 INFO OGG-01408 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Restoring current schema for DDL operation to [ggadmin].

No SLoperatio

ns

Page 32: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental LoggingSource

Target

2012-11-14 09:51:11 INFO OGG-00482 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL

found, operation [alter table repuser2.tbl11 add primary key(id) (size 47)].

2012-11-14 09:51:11 INFO OGG-00489 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL is of

mapped scope, after mapping new operation [alter table REPUSER2.TBL11 add primary key(id) (size 47)].

2012-11-14 09:51:11 INFO OGG-01407 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Setting current schema for DDL operation to [SYS].

2012-11-14 09:51:11 INFO OGG-00484 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Executing DDL operation.

2012-11-14 09:51:14 INFO OGG-00483 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL operation successful.

2012-11-14 09:51:14 INFO OGG-01408 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Restoring current schema for DDL operation to [ggadmin].

select * from dba_log_groups where table_name='TBL11';

no rows

No SLoperatio

ns

Page 33: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

Extract USER ID and Replicat USER ID should be different

• Problem documented in MOS 1472420.1“If your TRANSLOGOPTIONS EXCLUDEUSER specified in the Extract is the same as the EXTRACT USERID, the DDL to add supplemental logging is not captured and sent to the target.”

• Our extract settings were• USERID ggadmin

• TRANLOGOPTIONS EXCLUDEUSER ggadmin

Page 34: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental LoggingTime

Source

t1 create table repuser2.tbl11 (id number, name varchar2(10));

2012-11-09 14:58:58 INFO OGG-01487 Oracle GoldenGate Capture for Oracle, ext_nb.prm: DDL found, operation

[create table repuser2.tbl11 (id number, name varchar2(10))2012-11-09 14:58:59 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN

DATA for table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS /* GOLDENGATE_DDL_REPLICATION */ (size 104)].

t2 alter table repuser2.tbl11 add primary key (id);

2012-11-09 15:00:04 INFO OGG-01487 Oracle GoldenGate Capture for Oracle, ext_nb.prm: DDL found, operation

[alter table repuser2.tbl11 add primary key (id) (2012-11-09 15:00:04 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN

DATA for table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG GROUP "GGT_61549" ("ID") ALWAYS /* GOLDENGATE_DDL_REPLICATION */ (size 117)].

2012-11-09 15:00:04 INFO OGG-00479 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully deleted TRAN

DATA for [ALTER] DDL operation, table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" DROP SUPPLEMENTAL LOG DATA (ALL) COLUMNS /* GOLDENGATE_DDL_REPLICATION */ (size 105)].

2012-11-09 15:00:04 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN

DATA for table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG GROUP "GGS_61549" ("ID") ALWAYS /* GOLDENGATE_DDL_REPLICATION */ (size 117)].

2012-11-09 15:00:04 INFO OGG-00479 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully deleted TRAN

DATA for [ALTER] DDL operation, table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" DROP SUPPLEMENTAL LOG GROUP "GGT_61549" /* GOLDENGATE_DDL_REPLICATION */ (size 104)].

Page 35: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental LoggingTime

Target

t1,t2 2012-11-09 14:59:01 INFO OGG-00489 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL is of mapped scope,

after mapping new operation [create table REPUSER2."TBL11" (id number, name varchar2(10)) (size 62)].

2012-11-09 15:00:06 INFO OGG-00482 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL found, operation

[alter table repuser2.tbl11 add primary key (id) (size 49)].

2012-11-09 15:02:53 INFO OGG-00482 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL found, operation

[ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS (size 72)].

2012-11-09 15:02:53 INFO OGG-00483 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL operation successful.

2012-11-09 15:03:27 INFO OGG-00482 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL found, operation

[ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG GROUP "GGT_61549" ("ID") ALWAYS (size 85)].

2012-11-09 15:03:27 INFO OGG-00483 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL operation successful.

2012-11-09 15:04:59 INFO OGG-00482 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL found, operation

[ALTER TABLE "REPUSER2"."TBL11" DROP SUPPLEMENTAL LOG GROUP "GGT_61549" (size 72)].

2012-11-09 15:04:59 INFO OGG-00483 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL operation successful.

2012-11-09 15:04:59 INFO OGG-01408 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Restoring current schema for DDL operation to [ggsdb].

GETREPLICATES does not work as it is expected

Page 36: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental LoggingTime

Source

t1 Create table

t2 ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS

t3 Create PK index

t4 add supplemental log data (id) (temp)

t5 drop SUPPLEMENTAL LOG DATA (ALL)

T6 add supplemental log data (id)

T7 drop supplemental log data (id) (temp)

Time

Target

t1 Create table

t2 Create PK index

t3 ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS

t4 add supplemental log data (id) (temp)

t5 drop supplemental log data (id) (temp)

1. ADDTRANDATA on source for new object.

2. Monitor replicated object on target and add SL manually before any DML change

Page 37: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

SCHEMATRANDATAIn GoldenGate 11.1

OGG-01783 Cannot verify existence of table function that is required to enable schema

level supplemental logging, failed to find function

DB Patches needed to support SCHEMATRANDATA in OGG (Oracle GoldenGate), OGG-01783 Cannot verify existence

of table function that is required to enable schema [MOS 1426440.1]

To use SCHEMATRANDATA, please apply the fix in bug 13794550 Only on some

platforms

Page 38: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental Logging

• Supplemental logging is critical for proper function

• Different users needed for extract and replicat

• Add trandata manually for replicated nodes

• Need to monitor schema for tables without trandata

• Use SCHEMATRANDATA if you can

Lessons Learned

Page 39: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Topics• Active/Active considerations

• Supplemental Logging• DDL Replication• Conflict Avoidance• Conflict Detection• Release Management• Data comparison

• General considerations• Trail file deletion• Archive log deletion

Page 40: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

DDL ReplicationTime Source Target

T1 create table luke ( a number ,b date ,c varchar2(30) );

alter table luke add primary key ( a );

T2

Page 41: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

DDL ReplicationTime Source Target

T1 create table luke ( a number ,b date ,c varchar2(30) );

alter table luke add primary key ( a );

desc luke Name Null? Type ---- ----- ---- A NUMBER B DATE C VARCHAR2(30)

add trandata luke

T2

Page 42: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

DDL ReplicationTime Source Target

T1 create table luke ( a number ,b date ,c varchar2(30) );

alter table luke add primary key ( a );

desc luke Name Null? Type ---- ----- ---- A NUMBER B DATE C VARCHAR2(30)

add trandata luke

T2 insert into luke values ( 1, sysdate, 'Luke Test 1' );

Page 43: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

DDL ReplicationTime Source Target

T1 create table luke ( a number ,b date ,c varchar2(30) );

alter table luke add primary key ( a );

desc luke Name Null? Type ---- ----- ---- A NUMBER B DATE C VARCHAR2(30)

add trandata luke

T2 insert into luke values ( 1, sysdate, 'Luke Test 1' );

select * from luke;

A B C- --------- -------------------1 19-NOV-12 Luke Test 1

insert into luke values ( 2, sysdate, 'Luke Test 2' );

Page 44: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

DDL ReplicationTime Source Target

T3 select * from luke;

A B C- --------- -------------------2 19-NOV-12 Luke Test 21 19-NOV-12 Luke Test 1

update luke set c = 'Luke Test 1 updated' where a = 1;

T4

Page 45: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

DDL ReplicationTime Source Target

T3 select * from luke;

A B C- --------- -------------------2 19-NOV-12 Luke Test 21 19-NOV-12 Luke Test 1

update luke set c = 'Luke Test 1 updated' where a = 1;

select * from luke;

A B C- --------- -------------------2 19-NOV-12 Luke Test 21 19-NOV-12 Luke Test 1 updated

update luke set c = 'Luke Test 2 updated' where a = 2;

T4

Page 46: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

DDL ReplicationTime Source Target

T3 select * from luke;

A B C- --------- -------------------2 19-NOV-12 Luke Test 21 19-NOV-12 Luke Test 1

update luke set c = 'Luke Test 1 updated' where a = 1;

select * from luke;

A B C- --------- -------------------2 19-NOV-12 Luke Test 21 19-NOV-12 Luke Test 1 updated

update luke set c = 'Luke Test 2 updated' where a = 2;

T4 select * from luke;

A B C- --------- -------------------2 19-NOV-12 Luke Test 2 updated1 19-NOV-12 Luke Test 1 updated

alter table luke add ( d timestamp );

Page 47: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

DDL ReplicationTime Source Target

T3 select * from luke;

A B C- --------- -------------------2 19-NOV-12 Luke Test 21 19-NOV-12 Luke Test 1

update luke set c = 'Luke Test 1 updated' where a = 1;

select * from luke;

A B C- --------- -------------------2 19-NOV-12 Luke Test 21 19-NOV-12 Luke Test 1 updated

update luke set c = 'Luke Test 2 updated' where a = 2;

T4 select * from luke;

A B C- --------- -------------------2 19-NOV-12 Luke Test 2 updated1 19-NOV-12 Luke Test 1 updated

alter table luke add ( d timestamp );

desc luke Name Null? Type ---- ----- ---- A NUMBER B DATE C VARCHAR2(30) D TIMESTAMP(6)

Page 48: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

DDL ReplicationTime Source Target

T5 insert into luke values ( 3, sysdate, 'Luke Test 3', systimestamp );

T6

Page 49: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

DDL ReplicationTime Source Target

T5 insert into luke values ( 3, sysdate, 'Luke Test 3', systimestamp );

select a,d from luke;

A D - ---------------------------- 2 3 22-NOV-12 09.40.30.622017 AM1

insert into luke values ( 4, sysdate, 'Luke Test 4', systimestamp );

T6

Page 50: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

DDL ReplicationTime Source Target

T5 insert into luke values ( 3, sysdate, 'Luke Test 3', systimestamp );

select a,d from luke;

A D - ---------------------------- 2 3 22-NOV-12 09.40.30.622017 AM1

insert into luke values ( 4, sysdate, 'Luke Test 4', systimestamp );

T6 select a,d from luke;

A D - ---------------------------- 3 22-NOV-12 09.40.30.622017 AM2 4 1 Missing

Value!

Page 51: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

DDL ReplicationTime Source Target

T5 insert into luke values ( 3, sysdate, 'Luke Test 3', systimestamp );

select a,d from luke;

A D - ---------------------------- 2 3 22-NOV-12 09.40.30.622017 AM1

insert into luke values ( 4, sysdate, 'Luke Test 4', systimestamp );

T6 select a,d from luke;

A D - ---------------------------- 3 22-NOV-12 09.40.30.622017 AM2 4 <<<*** MISSING ***>>>1

Stop extractStart extract

insert into luke values ( 5, sysdate, 'Luke Test 5', systimestamp );

Page 52: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Supplemental LoggingSource

Discard -> Process Abending : 2012-11-22 09:53:07Source Context : SourceModule : [ggstd.conv.endian] SourceID : [/scratch/pradshar/view_storage/pradshar_bugdbrh40_12927937/oggcore/OpenSys/src/gglib/ggstd/lecnv.c] SourceFunction : [convCompSQL] SourceLine : [530] ThreadBacktrace : [9] elements : [/ggs/replicat(CMessageContext::AddThreadContext()+0x26) [0x5f1e56]] : [/ggs/replicat(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x7b2) [0x5e88f2]] : [/ggs/replicat(_MSG_ERR_MAP_COL_INDEX_INVALID(CSourceContext*, DBString<777> const&, int, int, CMessageFactory::MessageDisposition)+0xa6) [0x5bee16]] : [/ggs/replicat [0x900e6c]] : [/ggs/replicat(ggConvRecLE(char*, file_def*, int, char, char)+0x6c) [0x900fac]] : [/ggs/replicat [0x8f4d6e]] : [/ggs/replicat(main+0x84b) [0x507a0b]] : [/lib64/libc.so.6(__libc_start_main+0xf4) [0x34cd81d994]] : [/ggs/replicat(__gxx_personality_v0+0x1e2) [0x4e3ffa]]

2012-11-22 09:53:07 ERROR OGG-01161 Bad column index (3) specified for table FEXLINK.LUKE, max columns = 3.

Page 53: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

DDL ReplicationTime Source Target

T7 Restart replicat

select a,d from luke;

A D - ---------------------------- 3 22-NOV-12 09.40.30.622017 AM2 4 5 22-NOV-12 09.53.02.177904 AM1

• GoldenGate parameters (Replicat only)• DDLOPTIONS UPDATEMETADATA

• Automatically updates GoldenGate metadata cache

Page 54: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

DDL Replication• UPDATEMETADATA doesn’t work both ways

1. After DDL ALTER must bounce REPLICAT on the local node and PRIMARY EXTRACT on other nodes

Page 55: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Topics• Active/Active considerations

• Supplemental Logging• DDL Replication• Conflict Avoidance• Conflict Detection• Release Management• Data comparison

• General considerations• Trail file deletion• Archive log deletion

Page 56: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Conflict Avoidance• 2 Columns

• LAST_UPDATED_AT (DATE)

• LAST_UPDATING_BY (VARCHAR2(30))

• TriggerCREATE OR REPLACE TRIGGER <Owner>.<Table Name>_BIU BEFORE INSERT OR UPDATE ON <Owner>.<Table Name> FOR EACH ROW WHEN ( user != 'GGADMIN' ) DECLARE l_username VARCHAR2(50); BEGIN SELECT NVL(SYS_CONTEXT('USERENV','CLIENT_IDENTIFIER'),USER) INTO l_username FROM DUAL; :NEW.last_updated_at := SYSDATE; :NEW.last_updating_user := l_username; END; /

Page 57: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Conflict Avoidance

Sequences• 16 digit sequence number• Each silo has a unique 2 digit starting number e.g. 21, 31 etc• Ability to find originating silo for management reports• Guarantee uniqueness• Extra maintenance for new sequences• Need GG parameter

• DDL EXCLUDE OBJTYPE SEQUENCE

Page 58: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Topics• Active/Active considerations

• Supplemental Logging• DDL Replication• Conflict Avoidance• Conflict Detection• Release Management• Data comparison

• General considerations• Trail file deletion• Archive log deletion

Page 59: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Conflict Detection

Discards• Check discard files

• Use a cron script or other scheduler for every 5 mins

• Discard reader built in PL/SQL• PIPELINE function• Parses the discard file for common errors• Condenses the output to one line with the main error information• Check blog for source code

• http://www.pythian.com/news/37703/read-goldengate-discard-files/

Page 60: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Conflict Detection

Discards• Usage of READ_DISCARD

• The discard had 567 lines• READ_DISCARD reduced output to 6 lines

select LINE_NUMBER , ORACLE_ERROR , SOURCE_OBJECT_NAME , ERROR_OPERATION , ERROR_COLUMN , ERROR_VALUE , PK_TABLE_NAMEfrom table(read_discard(‘repname’));

LINE_NUMBER ORA_ERROR SOURCE_OBJECT_NAME ERROR_OPERATION ERROR_COLUMN ERROR_VALUE PK_TABLE_NAME----------- --------- ------------------ --------------- ----------------- ---------------- ------------- 1 ORA-02291 CHILD_TABLE1 INSERT PARENT_ID 4100000000360956 PARENT 44 ORA-02291 CHILD_TABLE2 INSERT CHILD1_ID 6100000000041100 CHILD_TABLE1 63 ORA-02291 CHILD_TABLE3 INSERT PARENT_ID 4100000000360956 PARENT 263 ORA-01403 TABLE1 UPDATE TABLE1_ID 4100000000562453 311 ORA-01403 PARENT UPDATE PARENT_ID 4100000000360956 392 ORA-01403 CHILD_TABLE3 UPDATE CHILD3_ID 6100000000264611

6 rows selected.

Page 61: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Topics• Active/Active considerations

• Supplemental Logging• DDL Replication• Conflict Avoidance• Conflict Detection• Release Management• Data comparison

• General considerations• Trail file deletion• Archive log deletion

Page 62: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

Major Release scriptMajor Release script

App code

DBchang

e

Pre-DDL

DDL

DML

Post-DDL

DDLDDLPost-

DDL

Post-DDL

Pre-DDL

Pre-DDL

Statements which can not be replicated OR

We do not want to be replicatedRun these statements outside

replication cycleCREATE SEQUENCE

CREATE/ALTER MATERIALIZED VIEWCREATE SYNONYM on other user

Due to GG limitations

DMLDML

Statements which are replicated

via GoldenGate

Page 63: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

ERROR OGG-01161 Oracle GoldenGate Delivery for Oracle, resb01sa.prm: Bad column index (nn) specified for table mytab, max columns = 59.

ERROR OGG-01668 Oracle GoldenGate Delivery for Oracle, resb01sa.prm: PROCESS ABENDING.

Challenge – Replication from superset structure to minor set

Version N

Version N+1

Page 64: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

T Source Target

t1

desc mytab

ID not null numberName varchar2(10)

desc mytab

ID not null numberName varchar2(10)

t2

t3

t4

Drop Column

Page 65: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

T Source Target

t1

desc mytab

ID not null numberName varchar2(10)

desc mytab

ID not null numberName varchar2(10)

t2

alter table mytab drop column name;

t3

t4

Drop Column

Page 66: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

T Source Target

t1

desc mytab

ID not null numberName varchar2(10)

desc mytab

ID not null numberName varchar2(10)

t2

alter table mytab drop column name; Generate defgen and copy to source./defgen paramfile ./dirprm/defgen.prm

insert into mytab values(1,’Test01’);

t3

t4

Drop Column

Page 67: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

T Source Target

t1

desc mytab

ID not null numberName varchar2(10)

desc mytab

ID not null numberName varchar2(10)

t2

alter table mytab drop column name; Generate defgen and copy to source./defgen paramfile ./dirprm/defgen.prm

insert into mytab values(1,’Test01’);

t3

-- ASSUMETARGETDEFSSOURCEDEFS ./dirsql/targetdefgen.sql

MAP mytab, TARGET mytab,COLMAP (ID=ID);

start replicat

t4

Drop Column

Page 68: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

T Source Target

t1

desc mytab

ID not null numberName varchar2(10)

desc mytab

ID not null numberName varchar2(10)

t2

alter table mytab drop column name; Generate defgen and copy to source./defgen paramfile ./dirprm/defgen.prm

insert into mytab values(1,’Test01’);

t3

-- ASSUMETARGETDEFSSOURCEDEFS ./dirsql/targetdefgen.sql

MAP mytab, TARGET mytab,COLMAP (ID=ID);

start replicat

start pump extract

t4

Drop Column

Page 69: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

T Source Target

t1

desc mytab

ID not null numberName varchar2(10)

desc mytab

ID not null numberName varchar2(10)

t2

alter table mytab drop column name; Generate defgen and copy to source./defgen paramfile ./dirprm/defgen.prm

insert into mytab values(1,’Test01’);

t3

-- ASSUMETARGETDEFSSOURCEDEFS ./dirsql/targetdefgen.sql

MAP mytab, TARGET mytab,COLMAP (ID=ID);

start replicat

start pump extract

t4

select * from mytab

ID--------1

Select * from mytab

ID NAME---- -----------1 Test01

Drop Column

Page 70: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

T Source Target

t1

desc mytab

ID not null numberName varchar2(10)

desc mytab

ID not null numberName varchar2(10)

t2

t3

t4

Modify Column

Page 71: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

T Source Target

t1

desc mytab

ID not null numberName varchar2(10)

desc mytab

ID not null numberName varchar2(10)

t2

alter table mytab modify column name (varchar2(6));

t3

t4

Modify Column

Page 72: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

T Source Target

t1

desc mytab

ID not null numberName varchar2(10)

desc mytab

ID not null numberName varchar2(10)

t2

alter table mytab modify column name (varchar2(6));

Generate defgen and copy to source./defgen paramfile ./dirprm/defgen.prm

insert into mytab values(1,’Test011’);

t3

t4

Modify Column

Page 73: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

T Source Target

t1

desc mytab

ID not null numberName varchar2(10)

desc mytab

ID not null numberName varchar2(10)

t2

alter table mytab modify column name (varchar2(6));

Generate defgen and copy to source./defgen paramfile ./dirprm/defgen.prm

insert into mytab values(1,’Test011’);

t3

-- ASSUMETARGETDEFSSOURCEDEFS ./dirsql/targetdefgen.sql

MAP mytab, TARGET mytab,COLMAP (USEDEFAULTS,NAME=@STREXT(NAME,1,6));

start replicat

t4

Modify Column

Page 74: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

T Source Target

t1

desc mytab

ID not null numberName varchar2(10)

desc mytab

ID not null numberName varchar2(10)

t2

alter table mytab modify column name (varchar2(6));

Generate defgen and copy to source./defgen paramfile ./dirprm/defgen.prm

insert into mytab values(1,’Test011’);

t3

-- ASSUMETARGETDEFSSOURCEDEFS ./dirsql/targetdefgen.sql

MAP mytab, TARGET mytab,COLMAP (USEDEFAULTS,NAME=@STREXT(NAME,1,6));

start replicat

start pump extract

t4

Modify Column

Page 75: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

T Source Target

t1

desc mytab

ID not null numberName varchar2(10)

desc mytab

ID not null numberName varchar2(10)

t2

alter table mytab modify column name (varchar2(6));

Generate defgen and copy to source./defgen paramfile ./dirprm/defgen.prm

insert into mytab values(1,’Test011’);

t3

-- ASSUMETARGETDEFSSOURCEDEFS ./dirsql/targetdefgen.sql

MAP mytab, TARGET mytab,COLMAP (USEDEFAULTS,NAME=@STREXT(NAME,1,6));

start replicat

start pump extract

t4

Select * from mytab

ID NAME---- -----------1 Test01

Select * from mytab

ID NAME---- -----------1 Test011

Modify Column

Page 76: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Release Management

ASSUMETARGETDEFSASSUMETARGETDEFS

Old Version: Create metadata for New Version DB ( defgen )New Version: Use SOURCEDEFS & MAP for pulling data from superset structure

SOURCEDEFS & MAPSOURCEDEFS & MAP

Page 77: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Topics• Active/Active considerations

• Supplemental Logging• DDL Replication• Conflict Avoidance• Conflict Detection• Release Management• Data comparison

• General considerations• Trail file deletion• Archive log deletion

Page 78: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Data ComparisonUser developed scripts• Master shell scripts calls PL/SQL package• Compares VARCHAR2, NUMBER, DATE and TIMESTAMP

• Large VARCHAR2, BLOB, CLOB fields are hashed for comparison

• Requires a database link• Uses LAST_UPDATED_AT timestamps to compare a subset of

each table e.g. SYSDATE – 7• Saves the ID of the offending row in a custom table• Inconsistencies are handled manually• Can be time consuming

Page 79: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Data ComparisonVeridata• Need to set up a server with a database component• Each database in the replication cluster needs an agent• There can be significant initial configuration for tables to

excluded columns, if necessary.• Hashes the entire row for less network traffic• All comparison is done on the veridata server• Reports are good and clear• Need to write an XML parser to create the insert / update /

delete statements• May need some clean up operations for report files on the

server

Page 80: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Topics• Active/Active considerations

• Supplemental Logging• DDL Replication• Conflict Avoidance• Conflict Detection• Release Management• Data comparison

• General considerations• Trail file deletion• Archive log deletion

Page 81: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Trail file deletion• PURGEOLDEXTRACTS in mgr.prm

• PURGEOLDEXTRACTS ./dirdat/*, USECHECKPOINTS, MINKEEPDAYS 6

• Problems can occur if trail files have the same name (MOS 1460097.1)

• Best practice use unique trail file names (MOS 1324890.1)

• Use a script to check primary extracts, datapumps and replicats

• Script links Primary Extract -> DataPump(s) -> Replicat• Only delete once have been used by the next level

• Used by Replicat if you want to be really safe

• Parsing output from -> info <ER Name>, showch

Use unique trail file names

Page 82: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Trail file deletionInfo extract, showch output

Read Checkpoint #1

GGS Log Trail

Startup Checkpoint (starting position in the data source): Sequence #: 8 RBA: 1139 Timestamp: 2012-11-08 18:57:11.595452 Extract Trail: ./dirdat/wb

Current Checkpoint (position of last record read in the data source): Sequence #: 11 RBA: 1139 Timestamp: 2012-11-12 11:51:02.933110 Extract Trail: ./dirdat/wb

Write Checkpoint #1

GGS Log Trail

Current Checkpoint (current write position): Sequence #: 9 RBA: 5099 Timestamp: 2012-11-12 12:17:18.995401 Extract Trail: ./dirdat/wb

Page 83: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Topics• Active/Active considerations

• Supplemental Logging• DDL Replication• Conflict Avoidance• Conflict Detection• Release Management• Data comparison

• General considerations• Trail file deletion• Archive log deletion

Page 84: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Archive Log Deletion• GoldenGate only sends transaction on commit• Need to keep archive logs until transaction

completes• Either user a script for archive log deletion• Or since 11.1.1.1 can register the extract (EE only)

• REGISTER EXTRACT <Extract Name> LOGRETENTION• DBA_CAPTURE -> OGG2$_EXT01UPL88C29FC8

• Get • RMAN-08137: WARNING: archived log not deleted, needed for

standby or upstream capture process

• Beware of bugs! (MOS 1487374.1)

Page 85: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Lessons Learned• Supplemental Logging is critical - get it right!• Structural changes need to be managed carefully• Parts of the database structure needs special attention• Sequences, Materialized Views, Extra columns

• Need to set up monitoring• Discards• Row comparison• GG processes and function

• Trail files need to be cleaned up• Archive log cleanup must be integrated with GG

Page 86: Experiences and Lessons Learned 6-Node Active-Active Oracle GoldenGate Luke Davies.

© 2012 – Pythian

Thank you and Q&A

http://www.pythian.com/news/

http://www.facebook.com/pages/The-Pythian-Group/163902527671

@pythian

http://www.linkedin.com/company/pythian

1-877-PYTHIAN

[email protected]

To contact us…

To follow us…

@pythianjobs