Successful Upgrade Secrets: Preventing · • Comprehensive testing of DB stack, e.g, memory, RAC,...

Successful Upgrade Secrets: Preventing Performance Problems with Database ReplayPrabhaker Gongloor (GP),Leonidas Galanis,Karl DiasDatabase ManageabilityOracle Corporation

Oracle’s Complete Enterprise Software StackBuilt-in & Integrated Manageability

• Leader in the complete enterprise application stack

• Built-in manageability in every tier

• Integrated manageability across the entire stack

• Manage applications top-down, from the business perspective by understanding user experiences and business impact of IT issues

• Manage entire application lifecycle to increase business agility with comprehensive application quality management and compliance solutions

• Reduce operational coststhrough intelligent diagnostics and automated IT processes

Oracle Enterprise ManagerIncreases Business Efficiency

Agenda

• Why Upgrade?• Upgrade Steps and Challenges• Real Application Testing and Database Replay • Database Replay for 9.2/10.2 DB Releases• Upgrade Steps: EM Support• Using 11g Snapshot Standby• Performance Analysis• Conclusion

Please visit us at the OOW Demogrounds Booth Moscone West: L52/L51 : Database Replay/SPA

New!

Why Upgrade?Benefits: Oracle Database 11g vs. 10g*

Java Just-In-TimeCompiler

0

2

4

6

8

1 0

1 2

S p e e d

Up to11 x Faster

Database ResidentConnection Pool

Up to 20 x connections

RAC PerformanceEnhancements

0 %

2 0 %

4 0 %

6 0 %

8 0 %

1 0 0 %

1 2 0 %

1 4 0 %

1 6 0 %

1 8 0 %

Q u e r y I n t e n s i v e

Up to70% Faster

Query ResultCaching

0

2 0

4 0

6 0

8 0

1 0 0

1 2 0

1 4 0

Q u e r i e s

Up to25% Faster

TimesTenCache

Up to10 x Faster

Oracle SecureBackup

0

2 0

4 0

6 0

8 0

1 0 0

1 2 0

1 4 0

B a c k u p

Up to25% Faster

Oracle StreamsEnhancements

0

5 0

1 0 0

1 5 0

2 0 0

R e p l i c a t i o n

Up to 2 x Faster

0 %

5 0 0 %

1 0 0 0 %

1 5 0 0 %

2 0 0 0 %

N b r C o n n e c t i o n s

Optimizer StatsCollection

0

200

400

600

800

1000

Stats Collection

Up to 10 x Faster

0

200

400

600

800

1000

TimesTen

* Quantify the benefit of upgrade/new feature for YOURworkload using methodology discussed today

t2008

2003

2004

2005

2006

2007

2002

2009

2010

Today

Why Upgrade?Lifetime Support Policy

2011

2012

2013

2014

Sustaining SupportPremier Support

R2 July 2007

Extended Support

July 2010

January 2009 January 2012

R2 July 2010 July 2013

http://www.oracle.com/support/library/data-sheet/oracle-lifetime-support-policy-datasheet.pdf

2015

2016

August 2012 August 2015

Jul 2008

Database Upgrade Steps and Challenges

Database Upgrade Process: Steps

1. Gather information about PROD environment2. Determine the upgrade path & method3. Prepare B/R strategy, clone and setup Test DB4. Establish performance baseline before upgrade 5. Develop tests for database & applications6. Test upgraded database & applications7. Check performance before & after upgrade8. Remediate regressions9. Go Live!

Majority of effort spent in developing tests, detec ting and remediating regressions

Real Application Testing

RemediateRemediate

TestTestTest

DeployDeploy

• Reduces risk of new technology adoption through higher testing quality

• Reduces time for effective testing from weeks to days

• Features• Database Replay

• SQL Performance Analyzer (SPA)

Change

Real Application Testing provides realistic testing for real-world systems

Real Application Testing

• SQL Performance Analyzer• Predicts impact of change on

SQL response time• Assess change by executing

query part of SQL in isolation without concurrency

• Unit testing of SQL, e.g., optimizer stats/config, drop index, etc.

• Can use for upgrades from • 9.2/10.1 ��10.2 or 11g• 10.2 ��10.2 or 11g

• SQL Performance Analyzer• Predicts impact of change on

SQL response time• Assess change by executing

query part of SQL in isolation without concurrency

• Unit testing of SQL, e.g., optimizer stats/config, drop index, etc.

• Can use for upgrades from • 9.2/10.1 ��10.2 or 11g• 10.2 ��10.2 or 11g

• Database Replay• Predicts impact of change on

workload throughput

• Assess change by replaying workload with concurrency

• Comprehensive testing of DB stack, e.g, memory, RAC, concurrency related

• Can use for upgrades from

• 9.2 or 10.2 ��11g

• Database Replay• Predicts impact of change on

workload throughput

• Assess change by replaying workload with concurrency

• Comprehensive testing of DB stack, e.g, memory, RAC, concurrency related

• Can use for upgrades from

• 9.2 or 10.2 ��11g

� Real Application Testing = SPA + Database Replay� SPA + Database Replay are complementary � Together, provide comprehensive & flexible testing solution

Database Replay

Testing TodayProduction – 1,000s of Real Online Users

PRODUCTION

Testing TodayTest – 1-2 testers trying to be 1,000s of users

PRODUCTION TEST

Database ReplayWorkload for 1,000s of online users captured

PRODUCTION

Capture

Workload

Database Replay Workload for 1,000s of online users replayed

PRODUCTION TEST

Capture Replay

Workload

Database ReplayTest your system changes at production levels

PRODUCTION TEST

Capture Replay

Workload

Database Replay Overview

• Replay actual production database workload in test environment

• Identify, analyze and fix potential instabilities before making changes to production

• Capture Workload in Production• Capture full production workload with real load, timing &

concurrency characteristics• Move the captured workload to test system

• Replay Workload in Test• Make the desired changes in test system• Replay workload with full production characteristics• Honor commit ordering

• Analyze & Report• Errors• Data divergence • Performance divergence

Analysis & Reporting

Supported Changes

Changes Unsupported

Changes Supported•Database Upgrades, Patches

•Schema, Parameters

•RAC nodes, Interconnect

•OS Platforms, OS Upgrades

•CPU, Memory

•Storage

•Etc.

ClientClient

…Client

Middle Tier

Storage

Recording of External Client

Requests

…

…

Database Replay WorkflowProduction ( ≥≥≥≥ 9.2.0.8) Test ( ≥≥≥≥ 11.1)

Capture ReplayAnalysis &

ReportingProcess

StorageStorage

Mid-Tier

Replay Driver *Clients

* No middle/client tier setup required

Step1 :Workload Capture

Backup

File 1

File 2

…

ClientClient

…Client

Middle Tier

Server 1 Server 2 Server N…

Capture Infrastructure

Server 1 Server 2 Server N

File NBackground

OS Directory

• Minimal overhead

• Platform and protocol independent

• Workload filters

• Capture interesting workload

Production System

Workload Capture Overhead

• Performance overhead • Workload dependent• Proportional to the data sent from the client • TPC-C throughput degradation about 4.5%

• Workload capture size• TPC-C 20min 100 users 10 warehouses: 1.2G• Enable capture for few minutes to assess size

LOW Overhead HIGH

Long Running SQL Short SQL/DML Insert Intensive Large LOBs

DSS OLTP

Step 2: Process Workload Files

File 1

File 2

File n

…

Metadata

Replay Files

Test System• Setup test system

• Application data should be same as production system as of capture start time

• Use RMAN, Snapshot Standby, imp/exp, Data Pump, etc. to create test system

• Make change: upgrade db and/or OS, change storage, migrate platforms, etc.

• Processing makes captured data into replay ready format

• Once processed, workload can be replayed many times

• For RAC copy all capture files to single location for processing

File 1

File 2

File n

…

Capture Files

File 1

File 2

…

File N

Metadata

Replay Files

…Server Server Server

Replay Clients

…

Replay System

• Replay captured workload• Replayed operations see

the same data and perform the same work

• Preserve timing and concurrency characteristics

• Same number of user connections

• Replay Client• Multithreaded OCI Client• Drives multiple captured

processes• Scalable Architecture• Interprets capture into

sequence of OCI calls• Functional replay

Background

Timing Preservation

Commit Order Synchronization

Sequence Replay

Timing Preservation

Re-mapping

Replay Options

• Synchronized Replay (Default)• Workload is replayed in full synchronized mode• Same concurrency and timing as production workload• Transaction commit order is honored• Ensures minimal data divergence

• Synchronization controls• Workload can be replayed in unsynchronized mode • Useful for load/stress testing• High data divergence• Parameters for controlling synchronization

• Commit order synchronization: SYNCHRONIZATION

• Think time synchronization: THINK_TIME_SCALE• Connect (logon) time synchronization: CONNECT_TIME_SCALE• Request rate preservation: THINK_TIME_AUTO_CORRECT

Replay Options

• Connection Remapping• Capture-time connection string can

be remapped for replay time

• Number of Replay Clients• Configurable by user• Calibration mode recommends

number of replay clients needed for specific workload

• Replay clients are multithreaded clients that can drive multiple workload sessions

0

20

40

60

80

Time Taken (Days)

Install & Setup Understand

Application

Usage

Identify Key

Transactions

Generate

Workload

Run Test

DB Replay

LoadRunner

2

205

4

80

24 20

DB Replay: ½ month

LoadRunner: 7 ½ months

Total Testing Time

500

Comparison of LoadRunner & DB ReplayTesting e-Business Suite

15X

Database Replay for Prior Releases

Real Application Testing for Prior Releases

Helps Smooth Transition to Newer Releases

• Database Replay: Capture on older release; Replay on 11.1 and above• SQL Performance Analyzer: Execute tests on 10.2 and above

† ML 560977.1: Real Application Testing for Earlier R eleases

10g R2 or 11g9i R2

10g R2 or 11g10g R1

10g R2 or 11g10g R2SQL Performance Analyzer

11g9i R2

11g10g R2Database Replay

Upgrade ToUpgrade FromFeature

New!

Database Upgrade Steps: EM Support



1,2: Session S298732: A Hitchhiker's Guide to Database Upgrades: Wednesday 09/24/2008 13:00 - 14:00 Moscone South

3. Backup/Recovery, Clone and Setup Test DB

� End-to-End Automation of Database Replay in EMGC 10.2.0.5 (when available)

� Capture production workload & performance data� Setup Test DB : Provision, deploy software, clone database,

RAC conversion� Deploy Replay Clients� Orchestrate Replay through GC� Copy Workload and performance data to staging

Database Replay End-to-End Automation

Setup Test Database Workflow



Use Snapshot Standby for Database Replay !!

Using Snapshot Standby DB with Database Replay

• Increase ROI through better utilization of standby DBs for Test/Dev• Provides full scale test environment• Simultaneously serves as DR environment• Zero Data loss but no real-time query/fast fail-over• Redo shipped but not applied• Uses single copy like Storage snapshots but provides DR at

the same time

Session S298770: Beat Up Your Oracle Data Guard Standby with Oracle Real Application Testing: It’s Payback Time! 09/24/2008 17:00 - 18:00 Moscone South 102

Using Snapshot Standby DB with Database Replay

Physical Standby Apply Redo

Snapshot Standby Perform Testing

Open Database

Back out Changes

* Continuous Redo Shipping

1. ALTER DATABASE CONVERT TO SNAPSHOT STANDBY;

3. BACK OUT CHANGES

4. ALTER DATABASE CONVERT TO PHYSICAL STANDBY;

2. PERFORM REAL APPLICATION TESTING!

Performance Analysis Workflow & Tools

1. CAPTURE INTERESTING WORKLOAD (PROD)

4. MAKE CHANGE, REPLAY (TEST)

2. ESTABLISH REPLAY BASELINE WITHOUT CHANGE (TEST)

5. ANALYZE REPLAY PERFORMANCE, REMEDIATE

• Capture Profile

• Capture Report

• AWR / ASH/ ADDM Report

• EM interface

• Replay Profile

• Replay Report

• AWR / ASH/ Compare Period / ADDM Report

• Compare to Capture or to other Replays

• EM interface

DONE?NoYes

6. DEPLOY CHANGE (PROD)

Database Replay and Performance Baseline and Analysis

• AWR infrastructure can be used during/after capture and replay as before…!!!• ASH/ADDM/Compare Period Analysis• Advisors

• Automatic snapshots at begin & end of capture and replay

• Transportable AWR enables comparing performance across databases • Use Datapump API and requires staging schema• Exposed through EM and command line interfaceSession: S298703: Performance Fundamentals for Or acle Database 10g and Oracle Database 11g, 09/22/2008 14 :30 - 15:30 MS 305

Database Replay: Workflow

Database Replay: Workload Capture

Database Replay: Workload Replay

Database Replay: Analysis

Replay Report:Replay Options, Statistics, Divergence

Replay Report: Error & Divergence: Which Sessions and SQL?

Sessions

ORA-947: Not enough values

SQL: Errors SQL: Divergence

Database Replay: Performance Analysis

AWR Compare Period Report:Load Profile

1. Capture Vs Replay 2. Replay Vs Replay With Change

AWR Compare Period Report:Top SQL by Elapsed Time, #Unique Plans Seen

Capture Vs Replay

Replay Vs Replay With Change

“Oracle Real Application Testing reduces the time required to test changes by as much as 80%, lower testing costs by as much as 70%, mitigate risks by reducing the number of unexpected outages, and improve the quality of service for their IT operations.”

David MitchellSenior Vice President, OVUM

Source: Oracle Real ApplicationTesting – business agility through superior testing, Jan 2008

“Oracle Database 11g will be welcome relief to DBAsresponsible for testing changes against a production workload. The way that Data Guard and Real Application Testing work together is sheer poetry, and no doubt most shops will quickly come to rely on the combination”

Sean McCown Contributing Editor, Infoworld

Source: Oracle Real ApplicationTesting – business agility through superior testing, Jan 2008

Resources on OTN

• Real Application Testing for Earlier Releases• Database Replay: Technical White Paper• Testing Performance Impact of an Oracle Database 9i/10g

Release 1 to Oracle Database 10g Release 2 Upgrade with SQL Performance Analyzer

• Migration to Cost-Based Optimizer

• Upgrading from Oracle 9i to 10g: What to expect from the Optimizer

• Upgrade Companion: Metalink Note: 466181.1: One-stop shop for Upgrades

• Oracle By Example Series: Manageability on OTN

• FAQ

Conclusion• Database Replay provides comprehensive and easy to use

solution for assessing impact of changes on workload• Higher quality testing• Full coverage of workflows

• Database Replay can be used to test many changes• Upgrade testing including 9.2, 10.2 to 11g releases• Enabling new features, RAC, schema, parameter, OS/HW changes

• Helps adopt technology faster by cutting down testing time from months for days

• With Database Replay and Real Application Testing businesses can• Stay competitive• Improve profitability• Be compliant

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Recommended Campground Demos

Moscone West

Exhibit Hall

Change Management & Data Masking for DBAs

Moscone West

Exhibit Hall

Complete Data Center Management

Moscone West Exhibit Hall

Application Quality Management

Moscone West

Exhibit Hall

Self-Managing Database: Automatic Application & SQL Tuning


Self-Managing Database: Automatic Fault Diagnostics


Self-Managing Database: Automatic Performance Diagnostics

Moscone West

Exhibit Hall

Oracle Real Application Testing: Database Replay and

SQL Performance Analyzer

LocationDemo

Recommended Sessions (Monday sessions)

Moscone South 304

2:30 pm

Monday, Sept. 22

First-Failure Fault Diagnosability and Diagnostics: Oracle Database 11g Features and Novel Approaches

Moscone West 2003

4:00 pm

Monday, Sept. 22

Oracle Enterprise Manager: Oracle's Management Solution for Your Enterprise

Moscone South 302

2:30 pm

Monday, Sept. 22

Performance Fundamentals for Oracle Database 10g and Oracle Database 11g

Moscone West 2001

1:00 pm

Monday, Sept. 22

Optimizing Application Performance: Application Testing Suite to the Rescue

LocationTimeDateSession Title

Recommended Sessions (Tuesday sessions)

Moscone South 303

1:00 pm

Tuesday, Sept. 23

Demystifying SQL Tuning: Tips and Techniques for SQL Experts

Moscone West 2003

11:30 am

Tuesday, Sept. 23

Advanced Performance Diagnostics: What the GUI Doesn't Tell You


Moscone West 2003

9:00 am

Tuesday, Sept. 23

Application Upgrade Secrets: Avoid Surprises While Making Database Changes

Moscone South 304

9:00 am

Tuesday, Sept. 23

Using Oracle Database 11g Real Application Testing to Simulate Production System Patching

Recommended Sessions (Wednesday sessions)

Moscone West 2001

5:00pm

Wednesday, Sept. 24

Deploying Oracle Enterprise Manager in a Secure Maximum Availability Architecture

Moscone West 2001

1:00 pm

Wednesday, Sept. 24

SQL Tuning Roundtable with the Experts


Moscone South 303

9:00 am

Wednesday, Sept. 24

Storage Monitoring Made Easy: Diagnosing I/O Performance Problems

Moscone West 2001

11:30 am

Wednesday, Sept. 24

Applications Data Privacy: An Expert Panel Discussion

Recommended Sessions (Thursday sessions)

Moscone South 303

3:00pm

Thursday, Sept. 25

Managing Oracle Grid Computing: Oracle Real Application Clusters, Oracle Automatic Storage Management, Oracle Data Guard

Moscone South 303

1:30 pm

Thursday, Sept. 25

Proactive Performance Monitoring with Baselines and Adaptive Thresholds

Moscone South 303

12:00 pm

Thursday, Sept. 25

Application Testing Best Practices: Real-World Customer Testimonials


Moscone South 301

9:00am

Thursday, Sept. 25

The Danish Experiment: Oracle Database 11g Shock Upgrades and Massive Workload Reduction via COBS

Moscone South 303

10:30 am

Thursday, Sept. 25

Integrating 40 Data Centers in Three Years: How Oracle'sDBAs Control the Data Center Explosion

Successful Upgrade Secrets: Preventing · • Comprehensive testing of DB stack, e.g, memory, RAC,...

Documents

Transcript of Successful Upgrade Secrets: Preventing · • Comprehensive testing of DB stack, e.g, memory, RAC,...