Real Application Testing: Consolidated Database Replay Feature

Applies to:

Oracle Server - Enterprise Edition - Version: 11.2.0.2.0 and above

Information in this document applies to any platform.

What is being announced?

Oracle Database 11g introduced Real Application Testing database option to enable businesses to assess

impact of system changes with minimal effort and proactively identify and remediate any issues that

were identified in testing. Real Application Testing option is comprised of two components – SQL

Performance Analyzer (SPA) and Database Replay. SPA helps predict the impact of a system change on

unit SQL response time and Database Replay provides the best load testing solution using real

production workloads.

Oracle Database Release 11.2.0.2.0 and above supports now supports "Consolidated Database Replay" –

a feature that helps business assess the impact of database and workload consolidation to a single

database. Consolidated Database Replay supports execution of multiple replays on a single database.

The goal of this note is to provide information on how to use the Consolidated Database Replay

functionality. This note contains information on mandatory patches and also includes usage examples,

a FAQ, and syntax documentation.

Consolidated Database Replay allows replay of multiple captured production workloads from the same

or different systems. The captured workload information includes concurrency, think time, and

transaction dependencies, etc. The captured workloads can then be replayed on a single database

maintaining the production characteristics so that any potential issues can be identified and remediated

in test before deploying the change to production. This will help customers assess how their database

consolidation is going to work and accurately size their infrastructure.

Note: The use of Real Application Testing thru API/EM interfaces including the Consolidated Database

Replay functionality requires licensing of Real Application Testing. Please check the Licensing

Information manual on OTN for details.

What do you need to do?

Apply the following mandatory patches to the replay system to use the Consolidated Database Replay

feature.

This information applies to all platforms except Windows.

Replay database

Version

What patch you need to

apply?

Download Information Comments

11.2.0.2.0 13947480 One-off patch can be

downloaded or

requested from MOS

For capture side

patches see Doc ID

560977.1

For Replay side you do

not need additional

patches from 560977.1.

Patch 13947480

includes patches

recommended from

560977.1

11.2.0.3.0 13947480 One-off patch can be

downloaded or

requested from MOS

For capture side

patches see Doc ID

560977.1

Note 1: The consolidated database replay feature is only supported on Oracle Database 11.2.0.2.0 and

higher.

Note 2: Consolidated Database Replay functionality is only supported through API in 11.2.0.2.0 and

11.2.0.3.0. EM Cloud Control will support this functionality in a future release.

1) Step 1: Capture: Capturing a Database Workload.

Please see Oracle documentation on how to capture a database workload.

Consolidated Database Replay feature does not require any special steps for capturing a database

workload, the steps for capturing the database workload are exactly the same as before for a single

capture. Consolidated Database Replay can use multiple workload captures based on any of the

following: Oracle Database Release 9.2.0.8.0 and above, and multiple platforms (OS). You can capture

on any number of databases and any or different platform supported for Oracle Database and do one

single consolidated database replay starting with 11.2.0.2.0 Database on any Oracle Database supported

platform.

For example: You can capture workload on 9.2.0.8.0 HP-UX and 10.2.0.4.0 AIX. Then you can perform

consolidated replay on Oracle Exadata 11.2.0.2.

Test System Setup.

Before a workload can be replayed, the application data state should be logically equivalent to that of

the capture systems at the start time of each workload capture. This minimizes replay divergence during

replay. Since we are consolidating different databases into one single database and server, you should

restore each schema that was captured through an Oracle supported method such as Data pump

export/import, RMAN, etc. For example, if you captured:

1) Workload from CRM schema on 10.2.0.4.0 on Solaris server

2) Workload from HR schema on 10.2.0.5.0 on Linux server

3) Workload from SALES schema on 11.2.0.2.0 on Solaris server

and are doing a Consolidated Database Replay on 11.2.0.2.0 database, then you would restore the

following on the 11.2.0.2 test database:

1) CRM schema into CRM schema

2) HR schema into HR schema

3) SALES schema into SALES schema

You cannot restore everything into one single schema and use the Consolidated Database Replay. You

cannot remap the schema to a different schema and use the Consolidated Database Replay

1) Preprocessing Consolidated Database Workload.

For Consolidated Replay, you have to preprocess each captured workload into its own directory.

Please see Oracle documentation on how to preprocess a database workload.

Consolidated Database Replay feature does not require any special steps for preprocessing a

database workload, the steps for preprocessing the database workload is exactly the same as

previously used before the release of Consolidated Replay. Do not mix different workloads into

one directory. For example if you captured:

1) Workload from CRM schema on 10.2.0.4.0 on Solaris server

2) Workload from HR schema on 10.2.0.5.0 on Linux server

3) Workload from SALES schema on 11.2.0.2.0 on Solaris server

and you are doing a Consolidated Database Replay on 11.2.0.2.0, then you would do the

preprocessing on 11.2.0.2.0 as follows:

A) CRM Workload

• create or replace directory crm as '/u01/test/cap_crm';

• ensure your captured workload is in directory ‘/u01/test/cap_crm'

• execute dbms_workload_replay.process_capture('CRM');

B) HR Workload

• create or replace directory hr as '/u01/test/cap_hr';

• ensure your captured workload is in directory ‘/u01/test/cap_hr'

• execute dbms_workload_replay.process_capture(‘HR');

C) SALES Workload

• create or replace directory sales as '/u01/test/cap_sales';

• ensure your captured workload is in directory ‘/u01/test/cap_sales'

• execute dbms_workload_replay.process_capture(‘SALES');

Note: - You need to do the preprocess only once and the preprocessed workload can be

re used again.

2) Replaying a Consolidated Database Workload.

1) Create a root directory where all the preprocessed workload will be available for replay.

mkdir '/u01/test/cons_dir';

create or replace directory cons_workload as '/u01/test/cons_dir

2) Copy each of the preprocessed workload directories into the root directory.

cp –r /u01/test/cap_crm /u01/test/cons_dir

cp –r /u01/test/cap_hr /u01/test/cons_dir

cp –r /u01/test/cap_sales /u01/test/cons_dir

3) Create database directory for each workload.

create or replace directory crm as '/u01/test/cons_dir/cap_crm';

create or replace directory hr as '/u01/test/cons_dir/cap_hr';

create or replace directory sales as '/u01/test/cons_dir/cap_sales';

4) Set root replay directory as created in step 1

execute dbms_workload_replay.set_replay_directory('CONS_WORKLOAD');

5) Create a replay schedule and add individual captures.

execute dbms_workload_replay.begin_replay_schedule('CONS_SCHEDULE');

select dbms_workload_replay.add_capture('CRM') from dual;

select dbms_workload_replay.add_capture('HR') from dual;

select dbms_workload_replay.add_capture('SALES') from dual;

execute dbms_workload_replay.end_replay_schedule;

6) Initialize Consolidated Replay.

exec dbms_workload_replay.initialize_consolidated_replay('CONS_REPLAY','CONS_SCHEDULE');

7) Remap connections strings.

During workload capture, connection strings used to connect to the production system are

captured. In order for the replay to succeed, you need to remap these connection strings to the

replay system. The replay clients can then connect to the replay system using the remapped

connections. After the Initialize step, query the dba_workload_connection_map to find out

what connection string needs to be remapped.

DBA_WORKLOAD_CONNECTION_MAP displays the connection mapping information for

workload replay. Each row defines one connection mapping for a particular workload replay.

The column schedule_cap_id shows which capture a particular connection belongs to.

schedule_cap_id identifies a workload capture added to a replay schedule. It starts with 1.

Please check Oracle documentation. Use DBMS_WORKLOAD_REPLAY.REMAP_CONNECTION

procedure to remap the connections.

For example

- Query dba_workload_connection_map for capture connections

select conn_id,schedule_cap_id,capture_conn,replay_conn from

dba_workload_connection_map;

exec DBMS_WORKLOAD_REPLAY.REMAP_CONNECTION (connection_id =>

1,SCHEDULE_CAP_ID=>1,replay_connection =>'CRM');

exec DBMS_WORKLOAD_REPLAY.REMAP_CONNECTION (connection_id =>

1,SCHEDULE_CAP_ID=>2,replay_connection =>'HR');

exec DBMS_WORKLOAD_REPLAY.REMAP_CONNECTION (connection_id =>

1,SCHEDULE_CAP_ID=>3,replay_connection =>'SALES');

-- Verify connecting remapping is done.

select conn_id,schedule_cap_id,capture_conn,replay_conn from

dba_workload_connection_map;

Note: - for replay_connection CRM, HR, SALES are services that you have defined on the test

system.

8) Prepare Consolidated Replay.

exec dbms_workload_replay.prepare_consolidated_replay(synchronization=>'OBJECT_ID');

9) Start replay clients.

wrc system/oracle mode=replay replaydir=/u01/test/cons_dir

Note: - The replaydir path is pointing to the root directory holding the consolidated captures.

You will need to start multiple replay clients. You would need at least 1 replay client per capture.

To estimate the total number of replay clients you need to start, execute the following

command for each workload in your replay

(a) wrc mode=calibrate replaydir=/u01/test/cons_dir/cap_crm'

(b) wrc mode=calibrate replaydir=/u01/test/cons_dir/cap_hr'

(c) wrc mode=calibrate replaydir=/u01/test/cons_dir/cap_sales'

To estimate the replay clients for the consolidated replay, add the output of the above (a+b+c)

and execute the following replay client command (a+b+c) times in background

wrc system/oracle mode=replay replaydir=/u01/test/cons_dir & (repeat a+b+c)

10) Start consolidated replay.

execute dbms_workload_replay.start_consolidated_replay;

11) Once the replay is finished, generate the replay report.

sqlplus / as sysdba

-- replay report

set echo off head off feedback off linesize 200 pagesize 1000

set long 1000000 longchunksize 10000000

VARIABLE rep_id number;

BEGIN

SELECT max(id) INTO :rep_id FROM dba_workload_replays;

END;

/

spool report_cons.html

select dbms_workload_replay.report( :rep_id, 'HTML') from dual;

spool off

12) Generate AWR and ADDM reports for the Replay

sqlplus / as sysdba

set linesize 200 pagesize 1000

column name format a20

spool awr_id_usecase1.txt

select id,name,AWR_BEGIN_SNAP,AWR_END_SNAP from dba_workload_replays order by id;

spool off

@$ORACLE_HOME/rdbms/admin/awrrpt

Frequently Asked Questions (FAQ)

1) How do I preprocess different workloads?

See the heading above “Preprocessing Consolidated Database Workload.”

2) How many WRC replay clients should I start?

To estimate the total number of replay clients you need to start, execute the following

Command for each workload in your replay

wrc mode=calibrate replaydir=/u01/test/cons_dir/cap_crm' wrc mode=calibrate replaydir=/u01/test/cons_dir/cap_hr' wrc mode=calibrate replaydir=/u01/test/cons_dir/cap_sales'

Add together the recommended number of clients from the above. This will tell you how many

WRC replay clients you have to start at the minimum.

3) Can I use EM interface (DB Console, Cloud Control, Grid Control) when replaying consolidated

workloads?

No, at this time we only have API support for replay consolidation. However, workload capture

can be done using the supported EM interfaces. A future release of EM Cloud Control 12c will

support the replay consolidated feature.

4) Is there a Replay Compare Period Report for Consolidated Replay?

No. There is no Replay Compare Period Report for Consolidated Replay. Replay Compare Period

Report is only available for non consolidated replays.

5) How do I do performance analysis?

This is a very generic recommendation.

While the Consolidated Replay is running we recommend using Enterprise Manager’s

performance page and Top Activity page to monitor the consolidated replay. Enterprise

Manager’s performance page and Top Activity page provides details for key database, and

system performance statistics plus helpful insights to any potential contention. After the replay

has completed, generate AWR, ADDM reports for the consolidated replay period. For RAC and

Exadata be sure to use the Global AWR and ADDM reports. On a high level look at DB Time for

consolidated replay. Compare it with the total DB Time for all the replays. Look at CPU

utilization, memory utilization, I/O metrics, and network metrics. Ensure that these meet your

business requirements. Keep the changes that were done including hardware, storage,

database version and interpret the results in context with the changes and consolidation that

was done.

6) What synchronization modes are supported for Consolidated Database Replay?

Only synchronization = object_id or false is supported for Consolidated Replay feature.

7) What are the best practices to do Consolidated Database Replay?

A) Before doing the consolidated Database Replay ensure that each individual replay

completes as expected.

B) Follow best practices for Database Replay in NOTE:560977.1 which includes using SPA first to do unit SQL response time testing

8) How do I debug Database Replay

NOTE: 1287620.1 - Database Replay Diagnostic information

Details about new procedures and functions.

1) SET_REPLAY_DIRECTORY Procedure.

This procedure sets a directory that contains multiple workload captures as the current replay directory.

Syntax

DBMS_WORKLOAD_REPLAY.SET_REPLAY_DIRECTORY (

replay_dir IN VARCHAR2);

Parameters

Parameter Description

replay_dir Name of the OS directory containing the captures for a workload consolidation

2) BEGIN_REPLAY_SCHEDULE Procedure.

This procedure initiates the creation of a reusable replay schedule.

Syntax

DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE (

replay_dir_obj IN VARCHAR2,

schedule_name IN VARCHAR2);

Parameter Description

replay_dir_obj Directory object that points to the replay directory that contains all the capture

directories involved in the schedule

schedule_name Name of the schedule to be replayed

Usage Notes

• Only one schedule can be in creation mode at a time. Calling the subprogram again

before end_replay_schedule will raise an error.

• Prerequisites:

o The workload capture was already processed using the PROCESS_CAPTURE Procedure in

the same database version.

o The user must have copied the capture directory appropriately.

o The database is not in replay mode.

o The SET_REPLAY_DIRECTORY Procedure has already been called.

3) ADD_CAPTURE Function.

This function adds the given capture to the current schedule. The directory has to be a valid capture

processed in the current database's version. It returns a unique ID that identifies this capture within this

schedule.

Syntax

DBMS_WORKLOAD_REPLAY.ADD_CAPTURE (

capture_dir_name IN VARCHAR2)

RETURN NUMBER;

Parameters

Parameter Description

capture_dir_name Name of the OS directory containing the capture under the replay top-level

Parameter Description

directory

Note: - use of any other parameter in the function is not supported in 11.2 releases.

4) END_REPLAY_SCHEDULE Procedure.

This procedure wraps up the creation of the current schedule. The schedule is now saved and associated

with the replay directory and can be used for a replay.

Syntax

DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;

Usage Notes

The BEGIN_REPLAY_SCHEDULE Procedure must have already been called.

5) INITIALIZE_CONSOLIDATED_REPLAY Procedure.

This procedure puts the database state in INIT for a multiple-capture replay. It uses the replay_dir which

has already been defined by the SET_REPLAY_DIRECTORY Procedure, pointing to a directory that

contains all the capture directories involved in the schedule. It reads data about schedule

schedule_name from the directory, and loads required connection data into the replay system.

Syntax

DBMS_WORKLOAD_REPLAY.INITIALIZE_CONSOLIDATED_REPLAY (

replay_name IN VARCHAR2,

schedule_name IN VARCHAR2);

Parameters

Parameter Description

replay_name Name of the workload replay. Every replay of a processed workload capture can be

given a name. (Mandatory)

schedule_name Name of the schedule to be replayed. It must have been created through

the BEGIN_REPLAY_SCHEDULE Procedure for the replay directory replay_dir

Usage Notes

Prerequisites

• Workload capture was already processed using the PROCESS_CAPTURE Procedure in the same

database version.

• Database state has been logically restored to what it was at the beginning of the original

workload capture.

• The SET_REPLAY_DIRECTORY Procedure has been called.

6) PREPARE_CONSOLIDATED_REPLAY Procedure.

Similar to the PREPARE_REPLAY Procedure, this procedure puts the database in a special "Prepare"

mode for a multiple-capture replay.

Syntax

DBMS_WORKLOAD_REPLAY.PREPARE_CONSOLIDATED_REPLAY (

synchronization IN BOOLEAN,

connect_time_scale IN NUMBER DEFAULT 100,

think_time_scale IN NUMBER DEFAULT 100,

think_time_auto_correct IN BOOLEAN DEFAULT TRUE,

capture_sts IN BOOLEAN DEFAULT FALSE,

sts_cap_interval IN NUMBER DEFAULT 300);

Only synchronization = OBJECT_ID or FALSE is supported for Consolidated Replay feature.

When synchronization is OBJECT_ID, a more advanced synchronization scheme is used. Every action that

is replayed will be executed only after the relevant COMMITs have finished executing. The

relevant COMMITs are the ones that were issued before the given action in the original workload

capture and that had modified at least one of the database objects the given action is referencing

(either implicitly or explicitly). This OBJECT_ID scheme has the same logical property of making sure that

any action will see the same data it saw during capture. It ensures that SCN’s for different captures will

not clash. OBJECT_ID synchronization allows more concurrency during replays for the actions that do

not touch the same objects/tables.

7) START_CONSOLIDATED_REPLAY Procedure.

This procedure starts the replay of a multiple-capture capture.

Syntax

DBMS_WORKLOAD_REPLAY.START_CONSOLIDATED_REPLAY;

Usage Notes

Prerequisites:

• The call to the PREPARE_REPLAY Procedure was already issued.

• Enough number of external replay clients (WRC) that can faithfully replay the captured workload

has already been started. The status of such external replay clients can be monitored

using V$WORKLOAD_REPLAY_CLIENTS.

Who should I contact for more information?

Please create a Service Request with Oracle Support.

If any of these patches are not available for the specific platform or database version (11.2.0.2 and

above) or if it conflicts with existing applied patch, please create an SR with Oracle Support, provide the

details of the operating system platform, database version, current list of patches applied using opatch

lsinventory.