Imagination at work.

Ajith NarayananTechnical Lead-Oracle ERP Configuration ManagementGE Healthcare, Bangalore, India8th Dec-2014, Liverpool, U.K.

Best Practices For Large Oracle Apps R12 Implementations

Disclaimer

The views/contents in this slides are those of the author and do not

necessarily reflect that of General Electric Company or Oracle

Corporation and/or their respective affiliates/subsidiaries.

The material in this document is for informational purposes only, and is

provided on an AS-IS basis.

11/14/2014

3

Who Am I?Ajith Narayanan

Technical Lead – Oracle ERP Configuration Management

GE Healthcare, Bangalore, India

• 10 years of Oracle [APPS] DBA/Technical Architect experience.

• Oracle ACE Associate

• Blogger :- http://oracledbascriptsfromajith.blogspot.com

• Member:-OAUG, AIOUG

• Website Chair (2011-2013) – Oracle RACSIG http://www.oracleracsig.org

• AIOUG Real Application Clusters SIG Leader

11/14/2014

4

AgendaWhy do we need to implement best practices?

• Application Tier

• R12 Tech Stack

• Forms

• Response Time

• Web Applications

• Concurrent Manager

• Workflow Engine

• Taking Diagnostics Help

• CBO Statistics

• Application Patching

• Network & Latency Check

11/14/2014

5

Agenda• Database Tier

• CPU Utilization Observations and analysis • High DB_IO observation and correlation with CPU usage. • Identify Top SQLs • Effective usage of AWR, ADDM Reports • ORACHK Check (From Oracle RAC assurance team)

• ADR Command Interpreter (ADRCI)

• Cluster Callout scripts example.

• Looking At Issues Proactively & From Greater Heights

• References

• Q & A

11/14/2014

6

Why do we need to implement best practices?

11/14/2014

7

Oracle Applications R12

• Huge infrastructure

– Oracle Applications R12

– 8 Node middle tiers

– 16 Node RAC database tier.

– Database Size 40+ TB

– Annual DB growth of 2 TB

– Daily Orders: 45 K –3 M order lines

– 1500 online users

– 1M+ XML messages/day

Best Practices

• Benefits

– Enables us being proactive

– simple in terms of manageability

– Helps in support cost reduction.

– Time & effort saving

Why do we need to implement best practices?

11/14/2014

8

Upgrade Tech Stack

• R12 Tech Stack

• OC4J: Oracle AS 10g 10.1.3./ Weblogic Managed Servers (FMW)

• Forms: Oracle AS 10.1.2

• JAVA, JRE Plugin

Useful MOS Articles

� Oracle E-Business Suite Technology Stack Release Notes for Release 12.1.3 (MOS Note - 1098650.1)

� Oracle E-Business Suite Technology Stack Release Notes for Release 12.2*

� Oracle E-Business Suite - Technology Stack : Navigation (landing) Page (MOS Note - 1480550.1)

� Troubleshooting Assistant: EBS Technology Stack (Doc ID 1607365.2)

Application Tier

11/14/2014

9

Application Tier

11/14/2014

10

Forms

• Socket Mode for Internal users (MOS 384241.1)

• Servlet mode is default in R12 and recommended for external access.– cat $FORMS_WEB_CONFIG_FILE|grep serverURL=

If the serverURL parameter has no value then Forms is implemented in socket mode else it is servlet

• Forms dead client detection (DCD) and Abnormal termination handler.– Value specified in minutes : FORMS_TIMEOUT=10

env | grep -i FORMS_TIMEOUT

– Terminates fwebmx processes for dead clients.

– Enable Forms Abnormal Termination Handle

– Do not set FORMS_CATCHTERM

– SQLNET.EXPIRE_TIME – Probes client connection at regular interval.

Useful MOS Articles

Application Tier

11/14/2014

11

� Forms Process (FRMWEB) Consumes 100% of CPU in Oracle Applications R12 (Doc ID 745711.1)

Response Time/CPU Issues

• When users complain about response time, configure Apache to log the time it takes to service a request.

• Access logs monitoring should be good way to keep a tab on the response times of user requests.

– Configure Apache to log the time it takes to service a request

– Logs: $LOG_HOME/ora/10.1.3/Apache/access_log*

– Edit: $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.confLogFormat "%h %T

(Or)– Enable Execution Content ID's (ECID) for the access_log in Release 12. ECID's allow correlation of

requests across log files so that one can map the flow of a given request across the various components of iAS.

LogFormat "%h %l %u %t [ecid: %{Oracle-ECID}i] \"%r\" %>s %b [%T (secs)]" combined

Refer to http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for getting http return code definitions

Note - R12.2 replaces Oracle 10gAS OC4J containers with Weblogic managed servers.

Application Tier

11/14/2014

12

Web Applications

• Train users to logout when transactions are complete.– Logging out gracefully releases the memory and corresponding resources (i.e. connections, etc.)

Releases process resources sooner

• JVM/OC4J Sizing + Heap Size– OACORE, OAFORMS, OAFM etc..

– Start with no more than 100 active users per JVM/OC4J instance.

– Start with 1 active JVM/OC4J instance per CPU core

Note:- Each 'core' in a multi-core system is considered to be a separate CPU. For example, a dual quad-core chip would be the same as stating 2 x 4 x CPU or 8 CPUs.

Useful MOS Articles

� JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite 11i and R12 (Doc ID 362851.1)

� MOS Note 357597.1 on how to enable SQL trace for OA Framework applications

Application Tier

11/14/2014

13

EBS Concurrent Processing Analyzer

• The EBS Concurrent Processing Analyzer is a self-service health-check script

• Reviews the overall Concurrent Processing

• Analyzes the current configurations & settings for the environment.

• Providing feedback and recommendations on best practices.

• APPS Workload (Concurrent, FORM, ICX) observations.

• Mandatory Concurrent Programs fro purging.

Sample Output

Useful MOS Articles

Application Tier

2DEMO_cp_analyzer.html

11/14/2014

14

� Best Practices for Performance for Concurrent Managers in E-Business Suite [Doc ID- 1057802.1]

� Concurrent Processing - CP Analyzer for E-Business Suite (Doc ID 1411723.1)

EBS Workflow Analyzer

• The EBS Workflow Analyzer is a script that reviews the current Workflow Footprint.

• Analyzes the workflow configurations & Provides feedback and makes recommendations on best practices and areas of concern.

• Provides immediate analysis and output of the EBS Workflow environment.

• Workflow Process Stuck & Work Flow Checks

Sample Output

Useful MOS Articles

Application Tier

2DEMO_wf_analyzer.html

11/14/2014

15

� How to run EBS Workflow Analyzer Tool as a Concurrent Request (Doc ID 1425053.1)

� 11i-12 Workflow Analyzer script for E-Business Suite Workflow Monitoring and Maintenance [Video] (Doc ID 1369938.1)

� How To Use Concurrent Program "Purge Order Management Workflow" (Doc ID 878032.1)

OAM Diagnostics Help

• Oracle Application Object Library Best Practices: Database Security Tests Health Check Test

• Oracle Application Object Library Best Practices: E-Business Suite Security Tests Health Check Test

• Oracle Application Object Library Best Practices: E-Business Suite Diagnostic Tests Health Check Test

• And many more……

Application Tier

11/14/2014

16

CBO Statistics

• FND_STATS uses the standard the DBMS_STATS package, but enhances the functionality as it provides several additional PL/SQL procedures for gathering statistics at the table, schema, or database level.

• The FND_STATS package also supports histograms, table exclusions, extended stats, incremental stats gathering for partitioned tables, and concurrent statistics gathering.

Useful MOS Articles

Application Tier

11/14/2014

17

� Best Practices for Gathering Statistics with Oracle E-Business Suite (Doc ID - 1586374.1)

� bde_last_analyzed.sql - Verifies CBO Statistics (Doc ID - 163208.1)

Application Patching

• Thorough patch analysis with timing reports on sand box & non-prods environments before proceeding with the Prod patching.

• In Release 12, you register your customized files in the Register Flagged Files tool in Oracle Applications Manager (OAM), which replaces the $APPL_TOP/admin/applcust.txt file.

Useful MOS Articles

Application Tier

11/14/2014

18

� See Note 9766881 "My server does not have internet connection Can I Still use Patch Wizard?“

� Patching Best Practices And Reducing Downtime (Doc ID 225165.1)

• There are two diagnostic tools available in the Oracle E-Business Suite:

The Network test Form – This form can be added to user’s menu , So that they can run the tests when they feel a performance degradation

Note:- Network latency should be always low (<3ms) and bandwidth should be high.

Application TierNetwork & Latency

11/14/2014

19

Useful MOS Articles

The Client Analyzer

Application TierNetwork & Latency

11/14/2014

20

� Oracle E-Business Suite Network Utilities: Best Practices (Doc ID 556738.1)

CPU Utilization Observations and analysis

• CPU spikes for more than 30 minutes continuously is a candidate for RCA.

• High DB_IO observation and correlation with CPU usage.

• I/O Thresholds: Logical RDS > 300K/sec, Physical RDS> 3000/Sec are candidate for RCA.

– How many of you think RAM access is 10,000 times faster than Physical disk access?

– In real world, LIO is only 25-100 times cheaper than PIO

– Internal locks & latch serialization mechanisms involved.

– Targeting only PIO counts(or high cache hit ratios) important pitfall to avoid. Even with no PIOs, a query can still be outrageously inefficient

– LIO are a critical component of query cost

Database Tier

11/14/2014

21

AWR, ASH, ADDM Reports

• Identify Top SQL, Latch Spin Counts etc., Memory Advisors

• Identify top wait events (System, User IO, Cluster, Concurrency, Configuration, Applications, Network)

• Fragmented Interface tables

• Object to be pinned – (Frequently used database objects)

• RAC Check (From Oracle RAC assurance team) RACcheck - RAC Configuration Audit Tool (Doc ID 1268927.1)

• ORAchk - Health Checks for the Oracle Stack (Doc ID 1268927.2)

• Sample ORACHK report

• ORAchk-Collection Manager

– ORAchk daemon auto-start mode after node reboot (init integration)

– Merge multiple ORAchk collection reports

– Exclude checks based on profile

– Upload of installed patches to database

– Collection Manager for ORAchk, RACcheck and Exachk (Doc ID 1602329.1)

– ORAchk signature file in /tmp on all nodes to verify last ORAchk run

– New checks and bug fixes, including o 30 Oracle Ebusiness AP module data integrity checks

– 12 new Database checks

– 8 new Solaris system checks

Database Tier

ORAchk-Report-Example.html

11/14/2014

22

• ADRCI has multiple features such as

– Analyze diagnostic data within the Automatic Diagnostic Repository (ADR).

– Create a package with incident / problem information for Oracle Support.

– Diagnostic data includes incident and problem descriptions, trace files, dumps, health monitor reports, alert log entries, and more.

– Run the Health Checks

– Helps in purging old diagnostics trace & dump files

– Simple to use a rich command set

Set correct ORACLE_HOME and just enter adrci

The utility starts and displays the following prompt: adrci>

Database Tier

11/14/2014

23

ADR Command Interpreter (ADRCI)

• At the ADRCI prompt, enter the following command: adrci> HELP

To get help for a specific ADRCI command: adrci> HELP command

• For example, to get help on the SHOW TRACEFILE command, enter the following: adrci> HELP SHOW TRACEFILE

adrci> set homepath diag/rdbms/orclbi/orclbi2 adrci> show homes ADR Homes: diag/rdbms/orclbi/orclbi2

• To view the alert log: 1. Start ADRCI in interactive mode, and ensure that the homepath points to the correct directory within the ADR base directory hierarchy.

Database Tier

11/14/2014

24

Useful MOS Articles

2. At the ADRCI prompt, enter the following command: SHOW ALERT

The following are variations on the SHOW ALERT command: SHOW ALERT -TAIL

SHOW ALERT -TAIL 50

SPOOL /tmp/MYALERT.LOG SHOW ALERT SPOOL OFF This outputs the alert log, without XML tags, to the file /tmp/MYALERT.LOG.

SHOW ALERT -P "MESSAGE_TEXT LIKE '%ORA-600%'"

Database Tier

11/14/2014

25

� 11G ADR DOCUMENTATION (Doc ID 445022.1)

� How to Invoke ADR Command Interpreter (adrci) in FMW home? (Doc ID 1669923.1)

Database Tier#!/bin/ksh# # Author: Ajith Narayanan## http://oracledbascriptsfromajith.blogspot.com## Version 1.0## This callout script is extended to report/mail the affected weblogic services when any Oracle cluster event occurs.##umask 022FAN_LOGFILE=$ORACLE_HOME/racg/usrco/`hostname`_uptime.logEVENTLINE=$ORACLE_HOME/racg/usrco/`hostname`_eventline.logEVENTLINE_MID=$ORACLE_HOME/racg/usrco/`hostname`_eventline_mid.logMAIL_CONT=$ORACLE_HOME/racg/usrco/`hostname`_mail.logWEBLOGIC_DS=$ORACLE_HOME/racg/usrco/weblogic_dsecho $* "reported="`date` >> $FAN_LOGFILE &tail -1 $FAN_LOGFILE > $EVENTLINEawk '{for (f = 1; f <= NF; f++) { a[NR, f] = $f } } NF > nf { nf = NF } END {for (f = 1; f <= nf; f++) { for (r = 1; r <= NR; r++) { printf a[r, f] (r==NR ? RS : FS) } }}' $EVENTLINE > $EVENTLINE_MIDSER=`grep "service=" $EVENTLINE_MID|awk -F= '{print $2}'`DB=`grep "database=" $EVENTLINE_MID|awk -F= '{print $2}'`INST=`grep "instance=" $EVENTLINE_MID|awk -F= '{print $2}'`HOST=`grep "host=" $EVENTLINE_MID|awk -F= '{print $2}'`STAT=`grep "status=" $EVENTLINE_MID|awk -F= '{print $2}'`if [ "$SER" != " " | "$DB" != " " | "$INST" != " " | "$HOST" != " " | "$STAT" != " " ]; thenif [ $STAT = nodedown ]; thencat $EVENTLINE_MID > $MAIL_CONT

11/14/2014

26

Cluster callout scripts. – Quickly get alerted on any cluster events.

echo "**============================SERVICES AFFECTED===============================**" >> $MAIL_CONTgrep -i "$DB_" $WEBLOGIC_DS >> $MAIL_CONTelif [ $STAT = up ]; thencat $EVENTLINE_MID > $MAIL_CONTecho "**============================SERVICES RESTORED===============================**" >> $MAIL_CONTgrep -i "$DB_" $WEBLOGIC_DS|grep "SERVICE_NAME=$SER" >> $MAIL_CONTelsecat $EVENTLINE_MID > $MAIL_CONTecho "**============================SERVICES AFFECTED===============================**" >> $MAIL_CONTgrep -i "$DB_" $WEBLOGIC_DS|grep "SERVICE_NAME=$SER" >> $MAIL_CONT#ficat $MAIL_CONT| /bin/mail -s "Cluster $STAT event: $DB $INST $SER $HOST" ajithpathiyil@gmail.comfi#cat $MAIL_CONT| /bin/mail -s "Cluster $STAT event: $DB $INST $SER $HOST" ajithpathiyil@gmail.comfirm $EVENTLINE $EVENTLINE_MID $MAIL_CONT

Database Tier

11/14/2014

27

Looking At Issues Proactively & From Greater Heights

11/14/2014

28

References

• My Oracle Support

• Steven Chan’s blog (https://blogs.oracle.com/stevenChan)

11/14/2014

29

Q&A

11/14/2014

30

11/14/2014

31

Thank You!