Upgrade and Migrate on Multi Terabyte Mission Critical RAC with Near Zero Downtime

Upgrade & Migrate on Multi-Terabyte Mission Critical RAC with Near Zero Downtime

Jane Brown Senior Database Administrator, Oracle Certified Professional (OCP)

Database Management Practice

Purpose

• Minimize planned down time

• Meet the 7x24 availability requirement for Dell Mission Critical Applications

• Streamline Dell Mission Critical databases upgrades to 11g

2


Current Versions

• DB Version: 10.2.0.4

• OS Version: Suze Linux 64-bit

3

Target Upgrade Versions• Database Version: 11.1.0.7

• OS Version: OEL 64-bit


Other Key Data:Data Files: ASMStorage: EMC SymmetricDatabase Flashback: on

High Availability Data Guard Architecture

4

Redo Apply

drdb01-06LTST_DRPhysical Standby

prddb01-06LTSTPrimary

lsdb01-06 LTST_LS Logical Standby

SQL

Ap

ply

Data Center 1 Data Center 2


I. Reimage Logical Standby

5

Other Key Data:Data Files: ASMStorage: EMC SymmetricDatabase Flashback: on

Redo Apply




SQL

Ap

ply



Backup Essential Files

1) Backup spfile to pfile› lsts1> create pfile='/u01/app/oracle/product/10.1.0/db_1/dbs/initltst.ora' from spfile;

2) Backup Oracle Home db_1 and save it on a different host› cd $ORACLE_HOME/..

› /u01/app/oracle/product/10.2.0> tar cvf db_1.tar db_1

› /u01/app/oracle/product/10.2.0> scp db_1.tar prdb01:/dbscratch/preReImange/lsdb01/.

3) Capture admin directories› /u01/app/oracle/admin/ltst> ls

adump bdump cdump udump

4) Capture cluster name› grep -i name $ORA_CRS_HOME/install/cluster.ini

cluster_name= sdr2ppramercrs

5) Capture LUN Mapping – by Linux System Engineer

6


Reimage

• Shutdown database

• Re-image the host cluster servers to OEL 11g image by Linux System Engineer

› Deploy Oracle Enterprise Linux 64-bit OS

› Restore EMC LUN mapping

› Deploy Oracle 11g CRS RPM

› Deploy Oracle 11g Software RPM

7


Start 11g ASM

1) Verify that CRS and Nodeapps are all up› crs_stat.sh

2) Create 11g ASM Initialization file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/init+ASM1.ora /u01/app/oracle/product/11.1.0/asm_1/dbs/.

› remove *_dump_dest from the file

3) Restore password file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapw+ASM1 /u01/app/oracle/product/11.1.0/db_1/dbs/.

4) Source ASM environment› export ORACLE_HOME=/u01/app/oracle/product/11.1.0/asm_1

› export PATH=$ORACLE_HOME/bin:/usr/bin:.:/bin:/usr/kerberos/bin:/usr/local/bin:/sbin:/usr/sbin:./

› export LD_LIBRARY_PATH=$ORACLE_HOME/lib

› export ORACLE_SID=+ASM1

5) Startup 11g ASM› sqlplus / as sysasm

› SQL> startup

8


Register ASM to CRS

1. Add ASM to CRS› srvctl add asm -n sdr2prddb01 -i +ASM1 -o /u01/app/oracle/product/11.1.0/asm_1

› # Repeat to add all 6 nodes

2. Copy Initialization and Password files to the rest of RAC nodes › init+ASM1.ora

› orapw+ASM1

3. Start ASM on nodes via srvctl› +ASM1 SQL>shutdown

› srvctl start asm –n lsprddb01 #repeat for node 2-6

9


Start 10g DB

1) Restore 10g Oracle Home to a temporary directory on node1 only› /dbscratch/> tar xvf db_1.tar

2) Create a symbolic link for the original 10g Oracle Home› mkdir –p /u01/app/oracle/product/10.2.0/

› cd /u01/app/oracle/product/10.2.0/

› ln –s /dbscratch/db_1 db_1

3) Create admin directories › mkdir -p /u01/app/oracle/admin/ltst/

› cd /u01/app/oracle/admin/ltst/

› mkdir adump bdump cdump udump

4) Source 10g environment

› . 10gR2.env

5) Startup 10g Database› sqlplus / as sysdba

› SQL> startup

10


Verify Logical Standby logs1. From Logical Standby: Startup 10g Listener

› lsnrctl start LISTENER_LSPRDDB01

2. From Primary: Query the latest online logs › SQL> select THREAD#, SEQUENCE# from v$LOG where status='CURRENT';

THREAD# SEQUENCE#---------- ----------

1 55922 40763 40894 41495 40916 4092

3. From Primary: Force log archive› alter system archive log current;

4. From Logical Standby: Verify that the latest logs are applied› SQL> select THREAD#, MAX(SEQUENCE#), APPLIED from dba_logstdby_log where applied!='YES';

THREAD# MAX(SEQUENCE#) APPLIED---------- -------------- --------

1 5592 YES2 4076 YES3 4089 YES4 4149 YES5 4091 YES6 4092 YES

11

II. Upgrade Logical Standby Database


Pre-Upgrade

1) Run pre-upgrade information tools › SQL> @/u01/app/oracle/product/11.1.0/db_1/utlu111i.sql

2) Resolve any outstanding issue

3) Create a guaranteed restore point› SQL> create restore point pre_upgrade guarantee flashback database;

› SQL> select * from v$restore_point;

4) Stop 10g listener › .~ 10gR2.env

› lsnrctl stop

5) Stop SQL Apply and shutdown 10g database› SQL> alter database stop logical standby apply;

› SQL> shutdown

13


Start 11g Database

1) Create 11g DB Initialization file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/initltst.ora /u01/app/oracle/product/11.1.0/db_1/dbs/.

› Edit initialization file to

– remove *_dump_dest

– set cluster_database=false

– set dg_broker_start=false

2) Restore password file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapwltst1 /u01/app/oracle/product/11.1.0/db_1/dbs/.

3) Restore tnsnames.ora› cp /u01/app/oracle/product/10.2.0/db_1/network/admin/tnsnames.ora

/u01/app/oracle/product/11.1.0/asm_1/network/admin/.

4) Startup 11g database › . .bash_profile

› cd $ORACLE_HOME/rdbms/admin

› SQL> startup upgrade

14


Upgrade Database

1. Upgrade database› SQL> spool /tmp/upgrade.log

› SQL> @catupgrd.sql #40 min, db shutdown at then end of upgrade

2. Run post update scripts› ltst1 SQL> startup

› ltst1 SQL> @utlu111s.sql

› ltst1 SQL> @catuppst.sql

› ltst1 SQL> @utlrp.sql

3. Enable cluster database› ltst1 SQL> alter syste set cluster_database=true

4. Restart Instance 1

15


Register Database to CRS

1. Add Database and Instances to CRS › srvctl add database -d ltst_ls -o /u01/app/oracle/product/11.1.0/db_1

› srvctl add instance -d ltst_ls -i ltst1 -n sdrseprddb01 # repeat to add all 6 instance

2. Copy following files to the rest of RAC nodes › tnsnames.ora

› initltst1.ora

› orapwltst1

3. Restart DB› ltst1 SQL> shutdown

› srvctl start database –d ltst_ls

16


III. Synchronize Archive log (10g->11g)

17

Redo Apply




SQL

Ap

ply


11g


Start Apply

1. Query outstanding log in Logical Standby› col file_name for a70

› select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= ‘YES'

2. Start log Apply › alter database start logical standby apply immediate;

3. Repeat step 1 until all logs are applied

18


IV. Switchover to Logical Standby

19

Redo Apply

drdb01-06LTST_DRBystand Physical Standby

prddb01-06LTSTLogical Standby

lsdb01-06 LTST_LS Primary

SQL

Ap

ply


11g


Transition Primary -> Logical Standby

1. Verify Switchover status› SQL> select switchover_status from v$database;

SWITCHOVER_STATUS --------------------TO STANDBY

If the query returns “SESSIONS ACTIVE’, then ensure it’s ok to shut them down

› SQL > SELECT SID, PROCESS, PROGRAM FROM V$SESSION WHERE TYPE = 'USER' AND SID <> (SELECT DISTINCT SID FROM V$MYSTAT);

2. Disable all RAC instances except instance 1› srvctl stop instance –d ltst –i ltst2; #repeat for lsts2 - 6

› alter database disable thread 2; ; #repeat for thread 2 - 6

3. Switchover› SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO LOGICAL STANDBY;;

20


Transition Logical Standby -> Primary

1. Defer Redo log transfer in Logical Standby› SQL> alter system set

log_archive_dest_3='service="(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=sdrseprddb01-vip.us.dell.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ltst_XPT.prd.amer.dell.com)(INSTANCE_NAME=ltst1)(SERVER=dedicated)))"',' LGWR ASYNC NOAFFIRM delay=0 OPTIONAL max_failure=0 max_connections=1 reopen=30 db_unique_name="ltst" register net_timeout=15 valid_for=(online_logfile,primary_role)' scope=both

› SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=DEFER;

2. Disable all RAC instances except instance 1› srvctl stop instance –d ltst_ls –i ltst2; #repeat for instnace 2-6

› alter database disable thread 2; #repeat for thread 2-6

3. Commit Switchover› SQL> select switchover_status from v$database;

SWITCHOVER_STATUS --------------------TO PRIMARY

› SQL> alter database commit to switchover to primary;

21


Start Rest of RAC Instances

1. New Primary LTST_LS: Enable threads and start RAC instances 2-6› alter database enable thread 2

› srvctl start instance –d ltst_ls –i ltst2;

2. New Logical Standby LTST: Enable threads and start RAC instances 2-6› alter database enable thread 2;

› srvctl start instance –d ltst –i ltst2;

22


V. Reimage Original Primary Hosts

23

Redo Apply




SQL

Ap

ply


11g

10g


Reimage Original Primary Host

• Follow the process used in reimaging logical standby hosts (slice 7 – 10)

• Do not sync archived log from 11g –> 10g (skip slice 11)

24


VI. Upgrade Original Primary to 11g

• Follow the process used in upgrading Logical Standby Database (slice 13–16)

25


VII.Synchronize Archive log (11g->11g)

1. Query outstanding log in Logical Standby› col file_name for a70

› select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= ‘YES'

2. Start log Apply› alter database start logical standby apply immediate;

3. Repeat step 1 until all logs are applied

26


VIII.Reimage Physical Standby Hosts

27

Redo Apply




SQL

Ap

ply


11g

11g 10g


Reimage Physical Standby Hosts

• Follow the process used in reimaging Logical Standby Database (slice 7 –10, skip step 5 in slice 10)

• Mount 10g Database as standby › sqlplus / as sysdba

› SQL> startup monut

› SQL> alter database mount standby database

28


Start 11g Physical Standby Database

1) Create 11g DB initialization file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/initltst.ora /u01/app/oracle/product/11.1.0/db_1/dbs/.

› Edit initialization file to remove *_dump_dest

2) Create spfile from pfile

3) Restore password file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapwltst1 /u01/app/oracle/product/11.1.0/db_1/dbs/.

4) Restore tnsnames.ora› cp /u01/app/oracle/product/10.2.0/db_1/network/admin/tnsnames.ora

c\/u01/app/oracle/product/11.1.0/asm_1/network/admin/.

5) Mount 11g standby database› . .bash_profile

› SQL> startup nomount

› SQL> alter database mount standby database

Note: wait for dg broker configuration to apply log

29


Register Physical Standby Database to CRS

1. Add Database and Instances to CRS › srvctl add database -d ltst_dr -o /u01/app/oracle/product/11.1.0/db_1 –s mount –s physical_standby -m

dr.amer.dell.com

› srvctl add instance -d ltst_dr -i ltst1 -n sdrseprddb01 # repeat to add all 6 instance

2. Copy following files to the rest of RAC nodes › tnsnames.ora

› initltst1.ora

› orapwltst1

3. Restart DB› ltst1 SQL> shutdown

› srvctl start database –d ltst_dr

30


IX. Switchover to Original Primary

31

Redo Apply




SQL

Ap

ply


11g

11g 11g


Switchover via Data Guard Broker

I. Remove old Data Guard Broker configuration› dgmgrl > remove configuration

II. Create new Data Guard Broker configuration› dgmgrl > create configuration dg_ltst as primary database is ltst_ls connect identifier is ltst_ls;

› dgmgrl > add database ltst as connect identifier is ltst_prd maintained as logical ;

III. Switchover to Logical Standby› dgmgrl > switchover to ltst

IV. Add Physical Standby to Data Guard Broker

32


X. Verify Data Guard Status

1) Verify Database Status › dgmgrl > show configuration

2) Verify Log Applying Status in Physical Standby› Run from primary

› @dg_standby_log.sql

3) Verify Log Applying Status in Logical Standby› select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= 'YES'

33


Summary I. Reimage Logical Standby Hosts to 11g – Linux System Engineer

II. Upgrade Logical Standby DB to 11g –Database Administrator

III. Sync Archive log from 10g Primary to 11g Logical Standby - Database Administrator

IV. Switchover to Logical Standby Manually –Database Administrator

V. Reimage original Primary Hosts to 11g – Linux System Engineer

VI. Upgrade the original Primary DB to 11g – Database Administrator

VII. Sync Archive log from 11g Primary to 11g Logical Standby - Database Administrator

VIII. Reimage Physical Standby Hosts to 11g – Linux System Engineer

IX. Switchover to original Primary via DG Broker- Database Administrator

X. Verify Data Guard Status

34


Reference

• Oracle® Database Upgrade Guide 11g R1 (11.1) B28300-02

• Oracle® Data Guard Concepts and Administration 11g R1 (11.1) B28294-03

• Oracle Data Guard Broker 11g Release 1 (11.1) B28295)

• Database Rolling Upgrade Using Data Guard SQL Apply - Oracle Database 11g and 10gR2

35

Global Marketing36

For more information:

• Visit Dell at Oracle OpenWorld in Moscone South booth 521

• Visit www.dell.com/oracle for more information

• Dell Services Oracle Practice information at http://content.dell.com/us/en/enterprise/oracle-solutions-services.aspx

• Dell presentations at OOW 2010 at http://www.slideshare.net/dellenterprise

• Dell | Oracle white papers at http://content.dell.com/us/en/enterprise/oracle-solutions.aspx#facets=whitepaper&p=1

• Dell | Oracle Tested and Validated documentation at http://content.dell.com/us/en/enterprise/d/solutions/product-configurations.aspx


Questions ?

37


Versions - Production

38

Upgrade and Migrate on Multi Terabyte Mission Critical RAC with Near Zero Downtime

Technology

Transcript of Upgrade and Migrate on Multi Terabyte Mission Critical RAC with Near Zero Downtime