Findfixandvalidate 140221013443-phpapp01

277
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 1 Maximize your Database Performance Find, Fix and Validate Jesús Leonardo Robles DB & O Business Development Manager Valentín Leonard Tabacaru Presales Technology Consultant

Transcript of Findfixandvalidate 140221013443-phpapp01

Page 1: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 1

Maximize your Database Performance Find, Fix and Validate

Jesús Leonardo Robles DB & O Business Development Manager

Valentín Leonard Tabacaru Presales Technology Consultant

Page 2: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 2

Agenda

Database Performance Analysis

– Challenges and Solutions

1.- Find

2.- Fix

3.- Validate

Resources

Page 3: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 3

Database Performance Analysis Challenges

•Reported by end users

•Discovered by daily checks

•Reactive to end-users

Find?

•Relate symptom to possible cause

•Relate possible cause to probable metric

•Implement change

Fix? •Did it work?

•End-user feedback?

•Run reports on metric

Validate?

•Will it occur again?

•Custom script to watch for the metric

•Complex alerting and notification

Track?

1

2

3

4

Page 4: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 4

• For problem diagnosis use ADDM, EM Performance

• ADDM reveals significant problems including problematic SQL details and recommendations improve to performance

• ADDM family – Compare Period, Real-Time ADDM

1 – Find: Diagnostics

Page 5: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 5

Boost Administrator Productivity

Eliminate Performance Bottlenecks

Lower IT Management Costs

Deliver Higher Quality of Service

Automatic Performance Diagnostics Maximize Performance With Self-Managing Database

• Sustain maximum performance with self-managing database

• Top-down analysis using Automatic Workload Repository snapshots

• Real-time performance analysis with Automatic Database Diagnostic Monitor

• Resolve performance issues faster with drill-down root-cause analysis

• Classification tree based on Oracle performance tuning expertise

• Performance expert; now a RAC specialist too in Oracle Database 11g

Automatic Workload

Repository Snapshots

Performance &

Management

Advisors

High

Load

SQL

RAC

Issues

IO / CPU

Issues

Self-Diagnostic

Engine

Page 6: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 6

Performance Issues Automatically Diagnosed by ADDM

Top SQL I/O Issues

Parsing Configuration

Issues

Application Usage

Excessive Logon/Logoff

Undersized Memory

Hot Blocks and Objects

RAC Service Issues

Locks & ITL contention

Checkpoint causes

PL/SQL, Java Time

Page 7: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 7

Real-Time ADDM—Architecture

• Makes a lightweight connection without acquiring additional locks and

resources, bypassing the SQL layer through the agent

• Also attempts to initiate standard JDBC connection

• Data returned by either connection is analyzed by Real-Time ADDM

EM Agent

JDBC Connection

Diagnostic Connection

ADDM

Analysis Database

Resource Constraints

Hangs

Memory Issues

Resource Limits Reached

Deadlocks

Top Issues Identified by

Real-Time ADDM

Page 8: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 8

Real-Time ADDM

Page 9: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 9

Compare Period ADDM

• Full ADDM analysis across two AWR snapshot periods

• Detects causes, measure effects, then correlates them

• Causes: workload changes, configuration changes

• Effects: regressed SQL, reach resource limits (CPU, I/O, memory, interconnect)

• Makes actionable recommendations along with quantified impact

AWR Snapshot

Period 1

AWR Snapshot

Period 2 Analysis Report Compare

Period ADDM

SQL Commonality

Regressed SQL

I/O Bound

Undersized SGA

Page 10: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 10

Compare Period ADDM: Method

• Identify what changed

• DB configurations, workload changes

• Quantify performance differences

• Uses DB Time as basis for measuring performance

• Identify root cause

• Correlate performance differences with changes

Did the Buffer cache get smaller?

Why is there 10% new SQL?

Why Top SQL increased by 45%?

Read I/O are up by 55%, why?

Did a buffer cache reduction

cause a read I/O increase?

Page 11: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 11

Compare Period – 3 Modes

One snapshot offset

System moving window

Customized period

Page 12: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 12

Compare Period: Report

Page 13: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 13

Active Session History (ASH)

All ‘Active’ sessions captured every second

• Foregrounds and backgrounds are sampled

• Active foregrounds contribute to DB Time

In-memory: V$ACTIVE_SESSION_HISTORY

• Sampling interval = 1 second

On-disk: DBA_HIST_ACTIVE_SESS_HISTORY

• Sampling interval = 10 seconds

ASH is a system-wide record of database activity

• A FACT table with multiple dimensions that help diagnose performance issues

Page 14: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 14

Drilling into Logical Dimensions

ASH

SQL

PL/SQL

Resource Usage

Session Identifiers

Session Attributes

SQL

SQL ID

Plan Hash

Operation

OpCode

PL/SQL

PL/SQL

Top Level PL/SQL

Resources

Wait Class

Wait Event

Object

Blocking Session

Identifiers

Instance

Services

User Session

Parallel Process

Program

Session Type

Attributes

Cons. Group

Module

Action

Client

Trans. ID

Page 15: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 15

ASH Analytics

Flexible Time Picker

Flexible Activity Chart

Flexible Top Chart Flexible Top Chart

Page 16: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 16

• Seamless integration between Find and Fix

• In-depth analysis and recommendation of the fix

• Gather statistics for this example…

• How would we validate this changes?

2 – Fix: Tuning

Page 17: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 17

SQL Problems

Bind-sensitive SQL with bind peeking

Literal usage

CURSOR SHARING

Cursor Sharing

Issues

Hardware resource crunch

Contention (row lock contention, block update contention)

Data fragmentation

Resource &

Contention

Issues

RESOURCES

Stale/Missing statistics

Incomplete statistics

Improper optimizer configuration

Upgraded Database: new optimizer

Changing statistics

Rapidly changing data

STATISTICS

Optimizer

Statistics Issues

Missing access structures

Poorly written SQL statements Application

Issues

APPLICATIONS

Not parallelized (no scaling to large data)

Improperly parallelized (partially parallelized, skews) Parallelism

Issues

PARALLELISM

Page 18: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 18

SQL Tuning Advisor

SQL Tuning Advisor

Identifies alternate execution plans using

real-time and historical performance data

Recommends parallel profile if it will improve SQL performance significantly (2x or more)

SQL Profiling

Statistics Analysis

Access Path Analysis

SQL Restructure Analysis

Alternative Plan Analysis

Parallel Query Analysis

Automatic Tuning Optimizer

Administrator

SQL Tuning

Recommendations

Gather Missing or Stale Statistics

Create a SQL Profile

Add Missing Access Structures

Modify SQL Constructs

Adopt Alternative

Execution Plan

Create Parallel SQL Profile

SQL Tuning Advisor

Page 19: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 19

Application Tuning Automatic SQL Tuning

Automatic SQL Tuning

•Captures high-load SQL

•Tunes SQL using SQL Profiles

•Implements improved SQL plans (optional)

Performance benefit of advice

provided

SQL Profiling tunes execution

plan without changing SQL

text

Enables transparent tuning for packaged

applications

Applications

High-Load

Packaged Apps + SQL Profile

Customizable Apps + SQL Advice

Customizable Apps + Indexes & MVs + Partitions

Tuned SQL

Automatic Tuning Optimizer

Page 20: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 20

SQL Access Advisor

Automatic Tuning

Optimizer

SQL Cache

SQL Tuning

sets SQL Access Advisor

Indexes

Materialized Views

Materialized Views

Logs

Partitioned Objects

SQL Access Advisor

• Capture SQL Workload from multiple sources

Set filters on the Workload

SQL Access Recommendations

Page 21: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 21

Real-Time SQL Monitoring Looking Inside SQL Execution

Guides tuning efforts Automatically

monitors

long running SQL

Enabled

out-of-the-box

with no performance impact

Monitors

SQL execution

Exposes monitoring statistics

Global execution level

Plan operation level

Parallel Execution level

Page 22: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 22

SQL Monitoring Details Expensive Hash Join

Page 23: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 23

Real-Time PL/SQL Monitoring

PL/SQL execution no longer a “black box”

•Answers questions like “why did my DBMS_STATS job take twice as long this time?”

Shows global (PL/SQL) and SQL level statistics

Each SQL called by PL/SQL recursively

monitored

Drill-down to slow SQL for diagnosing

unexpected PL/SQL behavior

Page 24: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 24

Performance Diagnostics Active Reports

Interactive reports for

off-line analysis

Drill-downs for detailed analysis

Can be emailed

for analysis

Does not require Oracle Enterprise

Manager installation

Page 25: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 25

D E M O N S T R A T I O N

Automatic SQL Tuning

Page 26: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 26

Page 27: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 27

Page 28: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 28

Page 29: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 29

Page 30: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 30

Page 31: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 31

Page 32: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 32

Page 33: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 33

Page 34: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 34

Page 35: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 35

Page 36: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 36

Page 37: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 37

Page 38: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 38

Page 39: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 39

Page 40: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 40

Page 41: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 41

Page 42: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 42

Page 43: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 43

Page 44: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 44

Page 45: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 45

Page 46: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 46

Page 47: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 47

Page 48: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 48

Page 49: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 49

Page 50: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 50

Page 51: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 51

Page 52: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 52

Page 53: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 53

Page 54: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 54

Page 55: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 55

Page 56: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 56

Page 57: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 57

Page 58: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 58

Page 59: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 59

Page 60: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 60

Page 61: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 61

Page 62: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 62

Page 63: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 63

Page 64: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 64

Page 65: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 65

Page 66: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 66

Page 67: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 67

Page 68: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 68

Page 69: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 69

Page 70: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 70

Page 71: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 71

Page 72: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 72

Page 73: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 73

Page 74: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 74

Page 75: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 75

Page 76: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 76

Page 77: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 77

Page 78: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 78

Page 79: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 79

Page 80: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 80

Page 81: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 81

Page 82: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 82

Page 83: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 83

Page 84: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 84

Page 85: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 85

Page 86: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 86

Page 87: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 87

Page 88: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 88

Page 89: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 89

Page 90: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 90

Page 91: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 91

Page 92: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 92

Page 93: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 93

Page 94: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 94

Page 95: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 95

Page 96: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 96

D E M O N S T R A T I O N

Monitoring SQL Statements

Page 97: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 97

Page 98: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 98

Page 99: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 99

Page 100: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 100

Page 101: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 101

Page 102: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 102

Page 103: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 103

Page 104: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 104

Page 105: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 105

Page 106: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 106

Page 107: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 107

Page 108: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 108

Page 109: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 109

Page 110: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 110

Page 111: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 111

Page 112: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 112

Page 113: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 113

Page 114: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 114

Page 115: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 115

Page 116: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 116

Page 117: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 117

Page 118: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 118

Page 119: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 119

Page 120: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 120

Page 121: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 121

Page 122: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 122

Page 123: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 123

Page 124: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 124

Page 125: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 125

Page 126: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 126

Page 127: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 127

Page 128: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 128

Page 129: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 129

Page 130: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 130

Page 131: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 131

Page 132: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 132

Page 133: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 133

Page 134: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 134

Page 135: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 135

Page 136: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 136

Page 137: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 137

Page 138: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 138

Page 139: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 139

Page 140: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 140

Page 141: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 141

Page 142: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 142

Page 143: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 143

Page 144: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 144

Page 145: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 145

Page 146: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 146

Page 147: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 147

Page 148: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 148

Page 149: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 149

Page 150: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 150

Page 151: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 151

Page 152: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 152

Page 153: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 153

Page 154: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 154

Page 155: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 155

Page 156: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 156

Page 157: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 157

Page 158: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 158

Page 159: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 159

Page 160: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 160

Page 161: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 161

Page 162: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 162

Page 163: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 163

Page 164: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 164

Page 165: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 165

Page 166: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 166

Page 167: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 167

Page 168: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 168

Page 169: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 169

Page 170: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 170

Page 171: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 171

Page 172: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 172

Page 173: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 173

• End-to-end SPA workflow using Optimizer Statistics

• Seamless integration from Find to Fix to Validate to Implement with confidence

• No more guesswork

3 – Validate: SPA

Page 174: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 174

Real Application Testing SQL Performance Analyzer (SPA)

SQL Workload

STS

SQL Plans + Run-time Stats

Pre-Change Trial

SQL Plans + Run-time Stats

Post-Change Trial

Compare SQL

Performance Analysis Report

• Test and predict impact of system changes on SQL query performance

• Analyze performance changes for improvements and regressions

• Comprehensive performance analysis and reporting

• Re-execute SQL queries in the given environment

• End-to-end solution: STS, SQL Plan Baselines, and SQL Tuning Advisor

Page 175: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 175

SPA Analysis Report Example

4

1

3 2

5

Page 176: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 176

SPA Report Regressed SQL Statements

Page 177: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 177

Usage Scenario: Evaluating Changes On Production

Page 178: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 178

+Parameter

Change

Fix Regression thru

SPM

+Add

indexes

Using SPA For Changes in Production: Example

Fix Regression

SQL Profile

Prod

+SQL

Profile

+New Stats

Change

+Partitioning

+Validate

Tuning

Parameter change

was bad in this case

And so on…

+Index

Unusable

• Bubble following the arrow indicates the

delta change on Production

• SPA is used for testing every change

Page 179: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 179

When to use SPA?

• Optimizer Statistics

• Database parameter changes

• Database schema changes (e.g., add/drop indexes)

• Implementation of tuning recommendations

• I/O subsystem changes (e.g., ASM, Database Machine)

• For consolidation (schema, CDB/PDB)

• On test, standby, and production environments

• Extended to home-grown scripts, application specific database changes, third-party testing tools using STS compare analysis

• Supports most applications - EBS, SAP, Siebel, home-grown, etc

SPA Provides Broad Testing Coverage

• My Oracle Support (MOS) Note: 560977.1

• 9.2/10.1 10.2 or 11g or 12 releases

• 10.2.0.x 10.2.0.y or 11g or 12 releases

SPA Provides Testing in Various Database Releases

Page 180: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 180

D E M O N S T R A T I O N

SQL Access Advisor & SQL Performance Analyzer

Page 181: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 181

Page 182: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 182

Page 183: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 183

Page 184: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 184

Page 185: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 185

Page 186: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 186

Page 187: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 187

Page 188: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 188

Page 189: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 189

Page 190: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 190

Page 191: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 191

Page 192: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 192

Page 193: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 193

Page 194: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 194

Page 195: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 195

Page 196: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 196

Page 197: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 197

Page 198: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 198

Page 199: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 199

Page 200: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 200

Page 201: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 201

Page 202: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 202

Page 203: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 203

Page 204: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 204

Page 205: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 205

Page 206: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 206

Page 207: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 207

Page 208: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 208

Page 209: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 209

Page 210: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 210

Page 211: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 211

Page 212: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 212

Page 213: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 213

Page 214: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 214

Page 215: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 215

Page 216: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 216

Page 217: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 217

Page 218: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 218

Page 219: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 219

Page 220: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 220

Page 221: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 221

Page 222: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 222

Page 223: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 223

Page 224: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 224

Page 225: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 225

Page 226: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 226

Page 227: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 227

Page 228: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 228

Page 229: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 229

Page 230: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 230

Page 231: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 231

Page 232: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 232

Page 233: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 233

Page 234: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 234

Page 235: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 235

Page 236: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 236

Page 237: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 237

Page 238: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 238

Page 239: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 239

Page 240: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 240

Page 241: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 241

Page 242: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 242

Page 243: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 243

Page 244: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 244

Page 245: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 245

Page 246: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 246

Page 247: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 247

Page 248: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 248

Page 249: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 249

Page 250: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 250

Page 251: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 251

Page 252: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 252

Page 253: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 253

Page 254: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 254

Page 255: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 255

Page 256: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 256

Page 257: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 257

Page 258: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 258

Page 259: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 259

Page 260: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 260

Page 261: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 261

Page 262: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 262

Page 263: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 263

Page 264: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 264

Page 265: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 265

Page 266: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 266

Page 267: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 267

Page 268: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 268

Page 269: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 269

Page 270: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 270

Page 271: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 271

Page 272: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 272

Page 273: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 273

Page 274: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 274

Advanced Databases Performance Analysis Solution

• Built-in, self-diagnostics engine: Automatic Database Diagnostics Monitor (ADDM) (Diagnostics Pack)

Find

• Automates complex and time consuming task of application tuning (Tuning Pack)

Fix • Validates tuning and “what-if” activities (Real Application Testing)

Validate

• Extend EM out-of-box metrics - Can be defined for any target type, including applications (Metric Extensions)

Track

Page 275: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 275

Resources

Oracle Enterprise Manager 12c:

http://www.oracle.com/us/products/enterprise-manager

Oracle Enterprise Manager 12c Database Management

http://www.oracle.com/technetwork/oem/db-mgmt

Database Manageability and Real Application Testing:

http://www.oracle.com/technetwork/database/manageability

Oracle Enterprise Manager on Beehive Online

Downloadable Technical Assets

[email protected] mailing list for support

Page 276: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 276

Page 277: Findfixandvalidate 140221013443-phpapp01

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 277