Best Practices For Large Oracle Apps R12 Implementations

Ajith NarayananDell IT, Bangalore

Hyderabad , 9th Nov 2013

Disclaimer

The views/contents in this slides are those of the author and do not necessarily

reflect that of Oracle Corporation and/or its affiliates/subsidiaries.

The material in this document is for informational purposes only and is published

with no guarantee or warranty, express or implied..

Who Am I?

Ajith Narayanan

Software Development Advisor

Dell IT, Bangalore.

� 9 years of Oracle [APPS] DBA experience.

� Blogger :- http://oracledbascriptsfromajith.blogspot.com

� Website Chair: - OracleRACSIG (2011 -2013)

� Member:- IOUG, OAUG & AIOUG, RACSIG (http://www.oracleracsig.org)

Agenda

Why do we need to implement best practices?

Application Tier=================1. R12 Tech Stack2. Forms3. Response Time/CPU Issues4. Web Applications5. Concurrent Manager 6. Workflow Engine 7. Taking Diagnostics Help 8. CBO Statistics9. Application Patching Best practices 10. Network & Latency Check

Agenda

Database Tier ============== 1. CPU Utilization Observations and analysis 2. High DB_IO observation and correlation with CPU usage. 3. Taking Diagnostics Help 4. Identify Top SQLs 5. Effective usage of AWR, ADDM Reports 6. Fragmented Interface tables 7. Object to be pinned – (Frequently used database objects) 8. RAC Check (From Oracle RAC assurance team) 9. Cluster Callout scripts to capture cluster events and notification through e-mail. 10. ADR Command Interpreter (ADRCI)11. ADG for offloading reporting workload

� Looking At Issues Proactively & From Greater Heights� References� Q & A

Why do we need to implement best practices?

• Oracle Applications R12 project with 8 Node middle tiers and 16 Node RAC database tiers. Database Size 40+ TB size and with a yearly database growth approximately 2 TB

• Makes our environment proactive & simple in terms of manageability.

• Helps in environment support & maintenance cost reduction.

• Time & effort saving .• And Of Course!! Makes everybody’s life easier

Application Tier

� R12 Tech Stack: -Make sure to upgrade to the latest certified technology stack

� OC4J: Oracle AS 10g 10.1.3.3.0

� Forms: Oracle AS 10.1.2.2

� E-Business Suite - Technology Stack : Navigation (landing) Page (MOS Note - 1480550.1)

� Useful Articles for the Oracle E-Business Suite Technology Stack for Technical Consultant / DBA (MOS Note - 828157.1)

� Oracle E-Business Suite Technology Stack Release Notes for Release 12.1.3 (MOS Note - 1098650.1)

Application Tier

Application Tier

� Forms: - Few checks for improved performance are � Socket Mode for Internal users (R12: Refer to Note 384241.1.)

� Servlet mode is default in R12 and recommended for external access.

� Forms dead client detection (DCD) and Abnormal termination handler.� Value specified in minutes : FORMS_TIMEOUT=10� Terminates fwebmx processes for dead clients.� Do not set FORMS_CATCHTERM

� Disable Cancel Query� Cancel Query increases middle-tier CPU as well as DB CPU� Refer to MetaLink Note 138159.1 on how to enable and tune Cancel

query related parameters� To Disable Cancel Query� Set the Profile “FND: Enable Cancel Query” to ‘No’

Application Tier

� Response Time/CPU Issues: -When users complain about response time, configure Apache to log the time it takes to service a request. Access logs monitoring should be good way to keep a tab on the response times of user requests.

� Configure Apache to log the time it takes to service a request

� Edit: $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf

� LogFormat "%h %T� Logs: $LOG_HOME/ora/10.1.3/Apache/access_log*

Application Tier

� Web Applications:-

� Ensure the users are trained to use the Logout or Home global buttons when completing their transactions.

� Do not train them to use the browser close (“x”) link.

� Logging out gracefully releases the memory and corresponding resources (i.e. connections, etc.) Releases process resources sooner

� First, rule out any SQL issues:- Note 357597.1 on how to enable SQL trace for OA Framework applications , If there are no database related issues, then you need to analyze the JVM (GC logs: Configure JVM sizing).

� Pool Monitor can be used to monitor the Framework Applications as well as the JVM utilization

� Navigation :- Application Module Pool -> Memory Utilization -> JVM Properties -> Invoke with -> Login-> Diagnostics -> Show Pool Monitor

� Concurrent Manager – (EBS Concurrent Processing Analyzer)The EBS Concurrent Processing Analyzer is a self-service health-check script that reviews the overall Concurrent Processing footprint and analyzes the current configurations and settings for the environment, providing feedback and recommendations on best practices.

� APPS Workload (Concurrent, FORM, ICX) observations.

� Mandatory Concurrent Programs.

� Best Practices for Performance for Concurrent Managers in E-Business Suite [Doc ID- 1057802.1]

� Concurrent Processing - CP Analyzer for E-Business Suite (Doc ID 1411723.1)

Application Tier

� Workflow Engine (EBS Workflow Analyzer)

The EBS Workflow Analyzer is a script that reviews the current Workflow Footprint, analyzes the configurations environment, provides feedback and makes recommendations on best practices and areas of concern. It provides immediate analysis and output of the EBS Workflow environment.

� Workflow Process Stuck & Work Flow Checks

� How to run EBS Workflow Analyzer Tool as a Concurrent Request (Doc ID 1425053.1)

� 11i-12 Workflow Analyzer script for E-Business Suite Workflow Monitoring and Maintenance [Video] (Doc ID 1369938.1)

� How To Use Concurrent Program "Purge Order Management Workflow" (Doc ID 878032.1)

Application Tier

� Taking Diagnostics Help

� Oracle Application Object Library Best Practices: Database Security Tests Health Check Test

� Oracle Application Object Library Best Practices: E-Business Suite Security Tests Health Check Test

� Oracle Application Object Library Best Practices: E-Business Suite Diagnostic Tests Health Check Test

� CBO Statistics

� Best Practices for Gathering Statistics with Oracle E-Business Suite (Doc ID - 1586374.1)

� bde_last_analyzed.sql - Verifies CBO Statistics (Doc ID - 163208.1)

Application Tier

� Application Patching Best practices

� Through patch analysis with timing reports on sand box & non-prods environments before proceeding with the Prod patching.

� Patching Best Practices And Reducing Downtime (Doc ID 225165.1)

� See Note 9766881 "My server does not have internet connection Can I Still use Patch Wizard?“

� How to Find E-Business Suite Recommended Patches (Note 1400757.1)

https://blogs.oracle.com/stevenChan/entry/release_12_patch_wizard_improv

� Network & Latency Check

� Oracle E-Business Suite Network Utilities: Best Practices (Doc ID 556738.1)

Application Tier

� CPU Utilization Observations and analysis

� For more than 30 minutes continuously is a candidate for RCA.

� High DB_IO observation and correlation with CPU usage.

� I/O Thresholds: Logical RDS > 300K/sec, Physical RDS> 3000/Sec are candidate for RCA.

LIO is Expensive?

How many of you think RAM access is 10,000 times faster than Physical disk access?In real world, LIO is only 25-100 times cheaper than PIO

- …not 1,000s or 10,ooosReason – Internal locks & latch serialization mechanisms involved.

Targeting only PIO counts(or high cache hit ratios) during SQL optimization is an important pitfall to avoid.Even with no PIOs, a query can still be outrageously inefficient

-LIO are a critical component of query cost

Database Tier

� Taking Diagnostics Help

� Oracle Application Object Library Best Practices: E-Business Suite Database Diagnostic Tests.

� Effective usage of AWR, ADDM Reports

� Identify Top SQL, Latch Spin Counts etc., Memory Advisors� Identify top wait events (System, User IO, Cluster, Concurrency,

Configuration, Applications, Network)

� Fragmented Interface tables

� Object to be pinned – (Frequently used database objects)

� RAC Check (From Oracle RAC assurance team) RACcheck - RAC Configuration Audit Tool (Doc ID 1268927.1)

Database Tier

� Cluster Callout scripts to capture cluster events and notification through e-mail.

� ADR Command Interpreter (ADRCI)

� ADRCI has multiple features such as� Analyze diagnostic data within the Automatic Diagnostic Repository (ADR).� Create a package with incident / problem information for Oracle Support� Run the Health Checks� Helps in purging old diagnostics trace & dump filesNote: - I’ve heard many DBAs saying that they can understand shell scripts, but that’s not enough, DBAs should be creative enough to write beautiful shell scripts to automate their environments and utilize the Oracle tools & utilities in efficient manner.(It actually develops DBAs general problem solving capabilities)

� ADG for offloading reporting workload

� https://blogs.oracle.com/stevenChan/entry/adg_ebs12� Business Continuity for Oracle E-Business Release 12 Using Oracle 11g Physical Standby Database (Doc ID 1070033.1)

� Using Active Data Guard Reporting with Oracle E-Business Suite Release 12.1 and Oracle Database 11g(Doc ID 1070491.1)

Database Tier

ADRCI Commands

ajithpathiyil1:/home/oracle[RAC2]$ adrci

ADRCI: Release 11.2.0.1.0 - Production on Sat Nov 9 00:36:43 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.

ADR base = "/u01/app/oracle"

adrci> show home

ADR Homes:

diag/tnslsnr/ajithpathiyil1/listener_scan1

diag/tnslsnr/ajithpathiyil1/listener

diag/rdbms/rac/RAC2

diag/asm/+asm/+ASM2

adrci>

adrci> show problem

ADR Home = /u01/app/oracle/diag/rdbms/rac/RAC2:

*************************************************************************

PROBLEM_ID PROBLEM_KEY LAST_INCIDENT LASTINC_TIME

-------------------- ----------------------------------------------------------- -------------------- ----------------------------------------

1 ORA 7445 [qcstda()+720] 17097 2013-11-09 00:42:09.740000 +05:30

1 rows fetched

adrci> show incident

ADR Home = /u01/app/oracle/diag/rdbms/rac/RAC2:

*************************************************************************

INCIDENT_ID PROBLEM_KEY CREATE_TIME

-------------------- ----------------------------------------------------------- ----------------------------------------

17097 ORA 7445 [qcstda()+720] 2013-11-09 00:42:09.740000 +05:30

1 rows fetched

ADRCI Commands

adrci> show incident -mode detail -p "incident_id=17097"

adrci> show trace /u01/app/oracle/diag/rdbms/rac/RAC2/incident/incdir_17097/RAC2_ora_9226_i17097.trc

adrci> show control

ADR Home = /u01/app/oracle/diag/rdbms/rac/RAC2:

*************************************************************************

ADRID SHORTP_POLICY LONGP_POLICY LAST_MOD_TIME LAST_AUTOPRG_TIME LAST_MANUPRG_TIME ADRDIR_VERSION ADRSCHM_VERSION ADRSCHMV_SUMMARY ADRALERT_VERSION CREATE_TIME

-------------------- -------------------- -------------------- ---------------------------------------- ---------------------------------------- ---------------------------------------- -------------------- -------------------- -------------------- -------------------- ----------------------------------------

4067289953 720 8760 2013-06-28 05:37:42.235895 +05:30 1 2 76 1 2013-06-28 05:37:42.235895 +05:30

1 rows fetched

adrci> show tracefile -rt

09-NOV-13 00:26:32 diag/rdbms/rac/RAC2/trace/RAC2_vktm_6761.trc

09-NOV-13 00:20:15 diag/rdbms/rac/RAC2/trace/alert_RAC2.log

09-NOV-13 00:20:13 diag/rdbms/rac/RAC2/trace/RAC2_diag_6772.trc

09-NOV-13 00:20:12 diag/rdbms/rac/RAC2/incident/incdir_17097/RAC2_ora_9226_i17097.trc

09-NOV-13 00:20:12 diag/rdbms/rac/RAC2/trace/RAC2_ora_9226.trc

..

.adrci> purge -age 2880 -type trace

ADRCI Commands

1. Create a package (A logical container for the diagnostic data)2. Add diagnostic data into the package (from an incident or manually by adding trace files)3. Generate a 'physical' package (file on the filesystem)4. Finalize the package

Step 1.adrci> ips create package problem 1 correlate all

Created package 1 based on problem id 1, correlation level all

adrci> ips generate package 2 in "/home/oracle"

Step 2.adrci>ips add incident 32578 package 3;Added incident 32578 to package 3

To manually add a tracefile into the packageadrci ips add file <path to tracefile> package 3

Step 3.adrci> ips generate package 1 in "/home/oracle"

Generated package 1 in file /home/oracle/ORA7445qc_20131109004602_COM_1.zip, mode complete

Step 4.adrci> ips finalize package 3When reviewing a database incident, even if you may not forward it on to Oracle Support it is helpful to generate the package zip file then unzipping if for your own review.

ADRCI Commands

� Oracle Configuration Manager helps us with lesser headaches, Since the details of our configuration changes, Accurate planning & upgrade are proactively foreseen by Oracle support. Also, by pairing My oracle support with OCM, we can leverage our Oracle support, with latest support information, Online communities, Security recommendations & Product alerts.

� Oracle Configuration Manager is used to collect client configuration information and upload it to the Oracle repository. When client configuration data is uploaded on a regular basis, customer support representatives can analyze this data and provide better service to the customers.

� If you want to install OCM in an Oracle E-Business Suite Release 12 installation, you must use Rapid Wizard and / or AutoConfig to install and configure Oracle Configuration Manager. This process, along with known issues and other considerations for E-Business Suite users, is described in the (DOC ID 406369.1)

� Information Center: Configuration Management (Doc ID 1457157.2)� OCM (Oracle Configuration Management)� RDA (Remote Diagnostic Agent)� EM (Enterprise Manager Cloud Control 12c)� MOS (My Oracle Support)

OCM(Oracle Configuration Manager)

Looking At Issues Proactively & From Greater Heights

� No time for implementing Oracle recommended best practices?

� Implemented Oracle recommended best practices in phased manner.

References

References

1. My Oracle Support2. Steven Chan’s blog (https://blogs.oracle.com/stevenChan)

Q&A

Thank You

Blog – http://oracledbascriptsfromajith.blogspot.comE-Mail - ajithpathiyil@gmail.comTwitter - @ajithpathiyilFacebook - ajithpathiyil