Proof of Concept with Real Application Testing 12c
-
Upload
luis-marques -
Category
Technology
-
view
136 -
download
0
Transcript of Proof of Concept with Real Application Testing 12c
Me
Name: Luís Marques
http://lcmarques.com / @drune / [email protected]
Luís Marques - @drune - http://lcmarques.com
Luís Marques - @drune - http://lcmarques.com
We will discuss
Database Replay features on 12c version and how we can apply stress test and workload scale-up techniques to ensure a real world proof of concept
SQL Perfomance Analyzer
Actual PoC results
We will not to discuss
Luís Marques - @drune - http://lcmarques.com
PoC – What really is?
An huge amount of work to plan and execute (and re-execute until the end of available time period)
Most work is for DBA if Real Application Testing is used.
Luís Marques - @drune - http://lcmarques.com
The (real) way it was…
Hardware arrivesSetup required
Recent Backup required
Restore your DB into new hardware
Flashback Guaranteed Restore PointSetup Application:
- New hardware - Install and Configure:- AS, Clients, Network, etc
We are ready!
2 weeks of infinite fun and infinite problems
Luís Marques - @drune - http://lcmarques.com
The (real) way it was…
Database scheduler Jobs created on-the-fly
Small reference in docs(not even in restrictions area)
Luís Marques - @drune - http://lcmarques.com
The way it should be
Working with the vendor to provide support for Real Application Testing
Client requested a simple and effective way to test new platform and software releases
Luís Marques - @drune - http://lcmarques.com
The way it should be
Hardware arrivesSetup required
Recent Backup required
Restore your DB into new hardware using START_SCN from Capture
Flashback Guaranteed Restore Point
DB ReplayDB Capture
Luís Marques - @drune - http://lcmarques.com
Before start any capture…RAT Patches on Production and PoC database (Doc ID 560977.1)
Capture Restrictions: direct path load external files with SQL*Loader, scheduler jobs, flashback querys, distributed trx, etc..
Workload is important: sysdate use,avoid maintenance windows or EM activity
No ongoing transactions during capture - Plan you database shutdown and start capture immediately after startup
RAC – Parallel SQL tunning set capture not supported (capture_sts)
Luís Marques - @drune - http://lcmarques.com
Workload Capture
Payment processing starts at 19:00:
18:35 – SQL> shutdown immediate18:45 – SQL> startup restrict18:50 – $ sqlplus / as sysdba18:50 – SQL> @capture_payment_day.sql
DBMS_WORKLOAD_CAPTURE.ADD_FILTER ('PaymentUser', 'USER’, 'PAY_A');
DBMS_WORKLOAD_CAPTURE.START_CAPTURE(name =>'capture_payment_day', dir => 'DB_REPLAY_DIR', auto_unrestrict => TRUE, default_action => 'EXCLUDE', sts_cap_interval => 300);
Luís Marques - @drune - http://lcmarques.com
Workload Capture
Populate DBA_WORKLOAD_CAPTURES / FILTERS
SQL> exec dbms_workload_capture.get_capture_info('DB_REPLAY_DIR');
Save the START_SCN to use with RMAN UNTIL SCN
to restore your database on PoC database.
Luís Marques - @drune - http://lcmarques.com
Workload Capture Export AWR data to compare performance between
Production and PoC database:
SQL> exec dbms_workload_capture.export_awr(2);
Dump of AWR will be used after replay to compare AWR on
PoC database
AWR data
Luís Marques - @drune - http://lcmarques.com
Workload Capture – The intelligent stuff
Workload Intelligence – Java programs to analyze Captured data
Don’t run it on production system (Heavy memory consumption)
Allow you to know more about the capture by patterns
Create user wi
Create Workload
Job
Create Workload
Model
Identify Patterns
Generate Report
Java –classpath … oracle.dbreplay.workload.intelligence.LoadInfo -job wi_job1 -cdir /home/oracle/scripts/rat/db_replay_dir -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi
Java –classpath … oracle.dbreplay.workload.intelligence.BuildModel -job wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi
Java –classpath … oracle.dbreplay.workload.intelligence.FindPatterns -job wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi
Java –classpath … oracle.dbreplay.workload.intelligence.GenerateReport -job wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi –top 5 –out intel_report.html
Luís Marques - @drune - http://lcmarques.com
How an intelligence report looks… Pattern is a sequence of transactions that run under “certain order”.
Allow you to identify mostly used “code paths” – Example: create a customer or process a visa card “code paths”
Patterns are ordered by different columns (DB Time, Nr. Executions and Length)
Pattern id = 2 Create a customer run 2333 times during the capture with a DB Time 9 seconds (0,78% of total DB Time)
Pattern id = 2
Pattern id =1
Luís Marques - @drune - http://lcmarques.com
Before start any replay…Setup guarantee restore points before any replay
Run java utility “Workload Analyzer” – Check the quality
of capture (insufficient data, errors, etc)
Run a small test replay (~ 20 min) before full replay - Avoid potential issues with full replay
Set up your system date to the same time as time of capture start if application relies on sysdate
Before performance analysis perform a replay analysis and review divergences
Luís Marques - @drune - http://lcmarques.com
Before start replay - sysdate
Set system clock on PoC system to the start of time of capture
Make sure that NTP daemon is disabled
Make sure that AWR snapshots are still created automatically after system clock changed
Restart PoC database after change in the clock.
Luís Marques - @drune - http://lcmarques.com
Before start replay – The Workload Analyzer
Java utility provided by Oracle to analyze the quality of
your capture.
$ java -classpath $ORACLE_HOME/jdbc/lib/ojdbc6.jar:$ORACLE_HOME/rdbms/jlib/dbrparser.jar:$ORACLE_HOME/rdbms/jlib/dbranalyzer.jar: oracle.dbreplay.workload.checker.CaptureChecker /home/oracle/scripts/rat/db_replay_dir jdbc:oracle:thin:@baco.localdomain:1521:prodpay2
Almost all sessions have been captured in-flight.
PL/SQL blocks or functions have 'complicated' logic or
multiple commits in them, they are hard to synchronize and they behavior might change during replay.
wcr_cap_analysis.html
Luís Marques - @drune - http://lcmarques.com
Captured workload: Sequential payment processing by Card type
Goal: Parallelized payment processing by Card Type
Scenario #1
Luís Marques - @drune - http://lcmarques.com
Consolidate Replay – Workload folding
Consolidate Database Replay: Slicing an existing captured
workload into subsets by specifying a point in time
Allows scale-in testing replaying workload subsets together
Our Goal: Parallelized payment processing by Card Type
Start of Day Processing
VISA Payment Process
MasterCard Payment Process
OthersCardPayment Process
WorkloadTimeWL Subset
#1WL Subset #2
WL Subset #3
Luís Marques - @drune - http://lcmarques.com
Consolidate Replay – Workload folding
Workload from 1 day (7pm to 6am) – capture_payment_day
3 subsets of workload derived from 1 capture:
Start of Day Processing
VISA Payment Process
MasterCard Payment Process
OthersCard Payment Process
WorkloadTime
Day Finish
Replay this!
Luís Marques - @drune - http://lcmarques.com
Consolidate Replay – Workload folding
BEGIN_TIME and END_TIME are offset in seconds from the start of the capture (tricky)
Incomplete calls are discarded to avoid problematic results
Example (VISA workload subset #1):• 4500 seconds after start of capture (1 hour and 15 minutes)• Ends after 8100 seconds from the start of capture (1 hour of duration)
SQL> EXEC DBMS_WORKLOAD_REPLAY.GENERATE_CAPTURE_SUBSET ( INPUT_CAPTURE_DIR => 'DB_REPLAY_DIR', OUTPUT_CAPTURE_DIR => 'DB_REPLAY_VISA', NEW_CAPTURE_NAME => 'CardVisa_wkld’, BEGIN_TIME => 4500, BEGIN_INCLUDE_INCOMPLETE=> FALSE, END_TIME => 8100, END_INCLUDE_INCOMPLETE=> FALSE, PARALLEL_LEVEL=>1)
Luís Marques - @drune - http://lcmarques.com
Consolidate Replay – Workload folding
Preprocess capture from all directories:
EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_VISA');EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_MCARD');EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_OTHERS');
Create a consolidate replay schedule
EXEC DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE ('visa_mcard_others_schedule');
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_VISA') FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_MCARD') FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_OTHERS') FROM dual;
EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;
Luís Marques - @drune - http://lcmarques.com
Consolidate Replay – Workload folding
Remap connections (DBA_WORKLOAD_CONNECTION_MAP) – You know how to do it
Prepare your consolidation replay:
SQL> EXEC DBMS_WORKLOAD_REPLAY.PREPARE_CONSOLIDATED_REPLAY(synchronization=>’OBJECT_ID’, think_time_auto_correct => TRUE )
• Each capture subset have a different SCN. SCN-based sync is not possible
• Only OBJECT_ID and OFF are supported as syncronization options
• Connect_time_scale and think_time_scale were left at default
• Best Results avoiding connect time modifications or think time modifications
Luís Marques - @drune - http://lcmarques.com
Consolidate Replay – OBJECT_ID A “more advanced” synchronization scheme is used
Ensures that SCN’s for different captures do not clash
Better dealing with concurrency when actions do not touch the same object
Bugs happen:
If you find problems with OBJECT_ID sync you are left to no syncronization
Monitor the progress of your replay. A collection of scripts here help - Scripts to Debug Slow Replay (Doc ID 760402.1)
Luís Marques - @drune - http://lcmarques.com
Consolidate Replay – Workload folding
Start replay clients – At least one per subset
wrc system/oracle mode=replay replaydir=/home/oracle/scripts/rat/db_replay_dir/
sleep 1 wrc system/oracle mode=replay replaydir=/home/oracle/scripts/ rat/db_replay_dir/
…
Start database replay
SQL> EXEC DBMS_WORKLOAD_REPLAY.START_CONSOLIDATED_REPLAY;
Luís Marques - @drune - http://lcmarques.com
Review divergences after replayApplication Level divergences:
• Example: Check if all the VISA cards were processed (manual SQL Script)
Data Level divergences:• Different number of rows returned by DML or SQL queries – Don’t match with capture
Error Level divergences:• Errors on Capture that are not on replay• Errors on Replay that were not captured• Different errors on capture and replay
select DBMS_WORKLOAD_REPLAY.REPORT(replay_id => 1, format => 'HTML') from dual;
Luís Marques - @drune - http://lcmarques.com
Sequential vs Parallel: Compare it!
Import AWR from previous sequential replay to compare to parallel replay
dbms_workload_replay.import_awr(replay_id => 12, staging_schema => 'PAY_A');
Generate a compare replays report
spool report_compare_consolidate.htmlVAR v_clob CLOBBEGIN dbms_workload_replay.compare_period_report (replay_id1 => 11, replay_id2 => 12,format => DBMS_WORKLOAD_REPLAY.TYPE_HTML, result => :v_clob );END;/PRINT v_clobspool off
Luís Marques - @drune - http://lcmarques.com
Sequential vs Parallel: Compare it!
Compare DB Time between periods (Sequential TP1 vs Parallel TP2);
Easier than read an AWR
Luís Marques - @drune - http://lcmarques.com
Workload: Parallelized payment processing by Card Type
Goal: Parallelized payment processing by Card Type and ATM machines payment processing from another database capture
Scenario #2
Luís Marques - @drune - http://lcmarques.com
Consolidate Replay – Time shifting
Allows shifting a replay to a time in the future and align with other replays
Stress testing by align peak loads
Peak number of sessionsAdd another workload here!
Luís Marques - @drune - http://lcmarques.com
Consolidate Replay – Time shifting
Align “Multibanco ATM processing” and Card Payment Processing to start exactly at same time
VISA Payment Process
MasterCard Payment Process
OthersCard Payment Process
ValidationsMultibanco
ATM processing
Workload time
Workload time
Delay time
Luís Marques - @drune - http://lcmarques.com
Consolidate Replay – Time shifting
Preprocess and delay the workloads to match ATM replay
EXEC DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE('payment_peak_align');SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_ATM') FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_VISA', 600) FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_MCARD', 650) FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_OTHERS', 660) FROM dual;EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;
EXEC DBMS_WORKLOAD_REPLAY.INITIALIZE_CONSOLIDATED_REPLAY ('payment_peak_replay', 'payment_peak_align');
Delaysecs
Luís Marques - @drune - http://lcmarques.com
Workload: Single workload capture on ATM machines payment
Goal: Stress test ATM machine payment processing on a very busy day – December 24th
Scenario #3
Luís Marques - @drune - http://lcmarques.com
Consolidate Replay – Schema Remapping
A single workload consolidated into two different instances of the application
Original capture schema: MULTIBANCO_ATM (original)
Remapped schema: MULTIBANCO_ATM_B (remapped)
Validations
Multibanco ATM processing
Multibanco ATM processing
Validations
Fees Calc
Fees Calc
MULTIBANCO_ATM_B
Workload time
Luís Marques - @drune - http://lcmarques.com
Consolidate Replay – Schema Remapping
Add the same workload twice to a consolidate replay
SQL> EXEC DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE(’multibanco_stress_schedule');
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_ATM') FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_ATM') FROM dual;
EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;EXEC DBMS_WORKLOAD_REPLAY.INITIALIZE_CONSOLIDATED_REPLAY ('payment_peak_replay', ‘multibanco_stress_schedule');
EXEC DBMS_WORKLOAD_REPLAY.SET_USER_MAPPING (2, ’MULTIBANCO_ATM', ’MULTIBANCO_ATM_B');
Luís Marques - @drune - http://lcmarques.com
Workload: Applied to all previous scenarios
Goal: Get a “even” more realist parallelized payment processing
Scenario #4
Luís Marques - @drune - http://lcmarques.com
Query Only – Warm up my buffer cache
When you start your replay, remember that your cache is "cold”
Replay may not be realistic or take too much time – capture is done with already some blocks in buffer cache.
Warm-up the buffer cache (if needed) with Query Only Replay before the real workload.
Query Only Replay replays read-only statements
Luís Marques - @drune - http://lcmarques.com
Query Only – Warm up my buffer cache example
EXEC DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE ('visa_mcard_others_schedule');
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_VISA’, query_only => TRUE) FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_MCARD’, query_only => TRUE) FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_OTHERS’,query_only => TRUE) FROM dual;
EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;
Luís Marques - @drune - http://lcmarques.com
Final Notes
Consolidate Replay can be really helpful on stress testing and scaling up on “brand new” PoC system
Comparing periods is important: Capture vs Replay; Consolidate Replay vs Sequential Replays, etc
Follow capture and replay recommendations. Make sure that you know your capture and his subsets and start your replay with the “defaults”
Try different synchronization options if you find in trouble with OBJECT_ID. Remember that OBJECT_ID and OFF are the only options for Consolidated Replay