Database Availability

and

Rolling Upgrades

Yavor IvanovYavor Ivanov

Stemo Ltd

Who is Yavor

� 9 years experience with Oracle Database

� Oracle Database & Oracle RAC fanatic

� Senior database expert and Manager of “Databases” division in Stemo Ltd“Databases” division in Stemo Ltd

� Oracle Certified Master� OCP DBA 9i/10g

� OCE RAC

� OCE SQL

Agenda

� What is Availability

� Causes of downtime

� Solutions for avoiding downtime

� Rolling upgrades� Rolling upgrades

� CRS rolling upgrade

� Database rolling upgrade options

� Rolling Database Upgrades using Data Guard SQL Apply

� References

What is availability

� The degree to which an application, service, or function is accessible on demand

� It is measured by the perception of an end user

Is measured for a given period (e.g. year)� Is measured for a given period (e.g. year)

Availability target Total downtime per year

95% 18 days 6 hrs

98% 7 days 8 hrs

99% 3 days 16 hrs

99.9% 8 hrs 45 min

99.99% 53 min

99.999% 5 min 18 sec

99.9999% 31.5 sec

Who cares about downtime?

� Cumulus Research Partners, September 2008 : Downtime costs European businesses € 5 billion a year

� Infonetics Research, March 2006: Medium businesses (101 to 1,000 employees) are losing an average of 1% of their annual revenue, or $867,000, to downtimerevenue, or $867,000, to downtime

� Forrester, December 2005: The average cost of downtime for e-commerce sites at $8,000 per hour - at larger sites like eBay, Intel and Amazon, the costs soar to hundreds of thousands of dollars per hour.

� Creative Networks, 2005: Unplanned downtime costs an average company employing 3,000 staff over € 132,000 a year

Causes of downtime (1)

� Unplanned downtime

� Hardware failure (server, storage)

� Network failure

� Human Error

Data corruption� Data corruption

� Hang or slow down

� Site failure

Causes of downtime (2)

� Planned downtime

� Hardware changes

� System changes (OS, CRS, RDBMS)

� Data changes (redefinition)

Application Changes� Application Changes

Solutions for unplanned downtime (1)

� Hardware failures:

� RAC, ASM, RMAN

� Network failure

� RAC, other

� Human error

� Constraints, Flasback *

� Data corruption

� RMAN, Hardware Assisted Resilient Data (H.A.R.D), Metalink Note:28814.1

Solutions for unplanned downtime (2)

� Hang or slow down

� The best RDBMS engine

� Site failure

� Data guard, Stretch clusters

Solutions for planned downtime

� Hardware changes

� RAC, Data guard

� System changes

� Rolling upgrades (RAC, Data guard)

� Data changes

� Online redefinition

� Application changes

CRS rolling upgrades

� Possible for upgrades from 10g to 11g

� Metalink note 338706.1: “Oracle Clusterware Rolling Upgrades”

� 10.1.0.x to 10.1.0.4/5

� 10.1.0.x to 10.2.0.1

� 10.2.0.1 to 10.2.0.2/x

� 10.1.0.x / 10.2.0.x to 11.1.0.x.

� 11.1.0.6 to 11.1.0.7/x

� http://jarneil.wordpress.com/2008/01/31/upgrading-to-oracle-11g-clusterware/

Database rolling upgrades

� Oracle MAA http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm

� Using Data Guard SQL Apply Possible after 10.1.0.3� Possible after 10.1.0.3

� Using Physical Standby Databases and Transient Logical Standby 11g

� Possible after 11g

Rolling Database Upgrades using Data Guard SQL Apply

� Prepare for rolling upgrade

� Upgrade the logical standby database

� Restart SQL Apply on the upgraded logical standby database

� Monitor events on the upgraded standby database

Perform switchover� Perform switchover

� Import any tables that were modified during the upgrade

� Activate user applications

� Upgrade the old primary database

� Start SQL Apply on the old primary database

� Optionally, raise the compatibility level on both databases

� Monitor events on the new logical standby database

� Optionally, perform another switchover

Prepare for rolling upgrade

Upgrade the logical standby database

Restart SQL Apply on the upgraded logical standby database

Perform switchover

Upgrade the old primary database

Start SQL Apply on the old primary database

Warnings

� Works after 10.1.0.3

� Be careful with the datatypes (not all supported by SQL Apply)

� Data Guard Broker not supportedData Guard Broker not supported

� Not supported by OEM Grid Control

� Data Guard protection cannot be “maximum protection”

� LOG_ARCHIVE_DEST_n for the logical standby must be OPTIONAL

� COMPATIBLE must be the lower version

Fallback

� Backups� Don’t forget ORACLE_HOME and oraInventory

backups

� Flashback database

Best suited for the standby database before � Best suited for the standby database before switchover

� Create guaranteed restore point

� Downgrade

� Maintains all transactional changes

� Slowest method

Availability benefits of rolling database upgrade

� Clients don’t wait for the upgrade to happen

� If something goes wrong with the upgrade, you are safe

� You switch the clients to the newer version when you know it is ready and testedYou switch the clients to the newer version when you know it is ready and tested

� You eliminate application downtime due to PL/SQL recompilation.

� Proof-of-concept test by Oracle: downtime as low as 1 minute for single switchover

References

� Oracle Maximum Availability Architecture (MAA): http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm

� Oracle documentation:� Oracle documentation:� High Availability Overview

� High Availability Best Practices

� Data Guard Concepts and Administration

� Oracle Database 10g High Availability with RAC, Flashback, and Data Guard (Matthew Hart, Scott Jesse)

� Google, blogs, etc…

Wrap up

� Availability is measured by the perception of an end user

� High availability is possible on database level, but…

� Do not forget all other levels.

You need solution for the system, not only for You need solution for the system, not only for the database

� The more nines you add to the availability requirements, the more complex and expensive system you will be

� Almost everything can be done, if you are ready to pay for it.

� There is no 100% (except in an advertisement)

Q&A

Yavor Ivanov

Stemo LtdEmail: yavor_ivanov [at] stemo [dot] bg

http://blog.yavor.info/