Understanding the OTM Analyzer (v1.04)

Created by: Keith.Wenyon@oracle.com

Version: 1.0

Contents Introduction .................................................................................................................................................. 3

Downloading the OTM Analyzer ................................................................................................................... 3

Running the OTM Analyzer ........................................................................................................................... 4

General Information ..................................................................................................................................... 5

System Settings ............................................................................................................................................. 6

Scalability Settings ........................................................................................................................................ 9

Agent Health Checks ................................................................................................................................... 12

System Health Checks ................................................................................................................................. 15

System Transaction Checks ......................................................................................................................... 18

Process Control Request/History Check ..................................................................................................... 19

qdlogs.pl support for the OTM Analyzer..................................................................................................... 21

Introduction From a historical perspective the role of an application system administrator has become more and more difficult throughout the years due to the numerous applications they need to provide support for. Each administrator and their team are responsible for supporting specialized applications each with their own set of config files, logs, processes and unique troubleshooting requirements that make maintaining enterprise software applications more and more challenging. You can almost hear them say to themselves, “Wouldn’t it be nice to be able to run a script on my OTM instance and generate a detailed report. A report that tells me at a glance what version of OTM we have running, what version of software we have installed on the database? A report that would provide recommendations to help avoid known problems and collect some of the information requested when investigating complex issues like performance or scalability issues.” Until recently this was only a dream, but now, with the help of the OTM Development and the Oracle Proactive Support Group we have released the OTM Analyzer script.

Downloading the OTM Analyzer The script can be downloaded from the MOS note along with a sample report, which we have used for this review. 1579683.1 - OTM Analyzer script for Analysis and Performance Monitoring

Running the OTM Analyzer Once you have the script downloaded log into the OTM database as GLOGOWNER and run the script via the following command. @otm_analyzer.sql; When the script completes an html file is generated in the same directory where the script was run from with the following naming convention. otm_analyzer_<domain>_<SID>_<DATE>_<TIME>.html Note: The analyzer is designed to work with OTM 6.1 and above and can be viewed using IE and Firefox.

General Information

The analyzer report is broken down into a number of sections, System Settings, Scalability Settings,

Agent and System Health Checks, General Information about integration, functional table and recurring

processes that are scheduled or have already been run recently. You can quickly access any of these

sections by clicking on the links provided at the top of the report.

Let’s review some of the features of the report and then review each of the sections in detail.

In each of the main sections of the report you should see a Hide Details option. This allows you to

minimize the information in this section if it has already been reviewed or not needed for a specific

problem type. When this is pressed the text changes to Show Details and section of the report is

collapsed.

Each minor section has a “Show Details” button that provides information about what each of the

checks that are being run. Each section should have the following, Item Name ( Typically SQL ), Item

Value ( the SQL being run ) and Information ( letting you know what the check is doing ).

Many of the sections will also generate warning messages with specific actions when certain conditions

are found.

System Settings

In this section of the report we look to see what version of OTM you have installed, using that

information we compare the settings of some database parameters against the recommended settings

in the sample init.ora file we provide with each release.

Running the OTM Analyzer is a great way to check and see if you are using the required settings for key

database parameters. It’s also a good idea to run the analyzer when you migrate from one version to

see if you need to make any changes to support the new version of OTM you are migrating to.

Note: You may be able to ignore some warnings, like this one regarding the SGA target and max size

settings if you are using this DB for a test instance.

The information in the connection pool section allows us to easily determine if these settings need to be

changed, especially on instances where scalability has been enabled. We have several checks in this

section to look for connection pools with no refresh interval or set longer than 1 minute and another

that verifies the test connections type. The following is a sample of the warning that would be

generated if problems were found.

Use the scroll bar to see the TEST_CONNECTION_TYPE column in this section.

Anyone familiar with OTM knows users have the ability to turn on quite a bit of logging, some of which is

very verbose and could cause performance issues. One of the first things we ask any client that reports

problems with performance is to verify what logging your users have on. You can see the log file GID, the

type of logging enabled, user, system or adhoc, the logging options that are enabled in that log file and

the domain the log is from.

Notice this warning is different as it references an existing MOS note.

Note: Since this section can contain a number of lines of data we are only showing 5 on the screen.

Press the Show More link to display up to 200 lines of data. The Show More option is also available on

other sections of the report.

Scalability Settings

The next section of the report collects information about the application scalability setup as defined in

the database. The script collects the contents of the APP_MACHINE, APP_SERVER,

APP_SERVER_MACHINE, APP_MACHINE_FAILOVER, APP_SERVER_DATA_QUEUE_DEF,

APP_SERVER_DOMAIN, APP_SERVER_FUNCTION and APP_SERVER_QUEUE.

Note: This information is typically requested on all performance or scalability related issues and can be

difficult to provide if not collected using the analyzer script.

We also check to see if you have any records in the OBJECT_LOCK table. If records are found we know

you have Scalability enabled. When this condition is met we do another check to see if you are running

the Object Lock Cleanup process. If we don’t see the clean process is being scheduled a warning would

be generated referencing a MOS note about scheduling this process.

Agent Health Checks

The Agent Health Checks are designed to look problems with the use of direct SQL insert, updates and

the use of stored procedures within agents. Correcting the issues identified by these checks will help

improve system performance and reduce JMS messaging between servers when scalability is enabled.

Note: In each section we identify the Agent Gid, the action sequence, the action that is being run and

the action parameters being used.

The first check looks to see if you are using any stored procedures in your agents. If so, any that are

found with the statement type set to Plain SQL Statement would show up in this section.

Notice that here we provide a clear action plan on what needs to be done to correct the configuration of

the agent.

Note: Each of the sections in the Agent Health Check can contain a number of lines of data. By default

we are only showing 5 lines. Press the Show More link to display up to 200 lines of data.

Another common configuration problem found with agents is the use of the refreshCache setting. When

set to true, OTM will update its own internal cache and also send the updated cache information to

other servers when scalability has been enabled.

The Verify refreshCache setting for Direct SQL Inserts checks the setting the refreshCache flag and the

SQL statement being run. If the SQL is an insert statement and the refreshCache is true the following

warning would be generated.

The warning provides a clear action plan to the user.

The Verify refreshCache setting for Direct SQL Updates checks the setting the refreshCache flag and the

SQL statement being run. If the SQL is an update statement and the refreshCache is false the following

warning would be generated.

Note: Review the agents returned by this check in detail to see if other agent actions within the same

agents are setup to trigger the cache refresh, if not the refreshCache should be changed to true.

The Verify the parseSQLForRefresh setting for Direct SQL Updates checks to see if you are doing a Direct

SQL Update. If so it checks to see if the parseSQLForRefresh is set to false, if so the following warning

would be generated.

Here the solution becomes a bit more complicated for the user. First the need to look at the SQL update

statement being run to determine if this is something we could easily parse to determine what cache

needs to be refreshed.

Take the SQL returned in the line of this section:

-refreshCache true -sqlStatement {UPDATE SHIPMENT SET PORT_DISCHARGE_ETA = TO_DATE ($ETA,

'MM/DD/YYYY HH:MI:SS AM') WHERE SHIPMENT_GID=$GID} -selectSQLForRefresh null -

parseSQLForRefresh false -role ADMIN -statementType {Plain SQL Statement} -user EP.ADMIN

Here we can easily see we are updating the SHIPMENT table. So setting the parseSQLForRefresh to true

would allow OTM to do an intelligent cache refresh of the shipment cache and prevent a full refresh of

the entire shipment object to be done. Keeping this set to false would trigger updates to the cache of

the shipment, shipment stop, shipment remarks, shipment refnums and other cache related object of

the shipment.

A few lines down you see the following being returned.

-refreshCache true -sqlStatement {UPDATE invoice inv SET consolidation_type = 'CHILD',

parent_invoice_gid = (SELECT invoice_refnum_value FROM invoice_refnum ir, shipment_bill sb WHERE

ir.invoice_gid = sb.bill_gid AND ir.invoice_refnum_qual_gid = 'TOLL.CONSOLIDATED_BILL_NUMBER' AND

sb.shipment_gid = $gid AND ir.invoice_gid = inv.invoice_gid) WHERE invoice_gid IN(SELECT invoice_gid

FROM invoice_refnum ir, shipment_bill sb WHERE ir.invoice_gid = sb.bill_gid AND

ir.invoice_refnum_qual_gid = 'TOLL.CONSOLIDATED_BILL_NUMBER' AND sb.shipment_gid = $gid)} -

selectSQLForRefresh null -parseSQLForRefresh false -role ADMIN -statementType {Plain SQL Statement}

-user GURUK.ADMIN

Here we see an update to the INVOICE table but within the update statement we do several sub-selects

making this not a good candidate for setting the parseSQLForRefresh to true. For these cases clients

would need to set the parseSQLForRefresh to false and provide a SQL statement in the

selectSQLForRefresh section of the agent action. Adding a query to the field takes precedence over

Parse for SQL Refresh and default cache refresh. This SQL will be the one to be parsed to determine

what needs to be refreshed.

System Health Checks

System Health Checks look for common issues that if addressed can help to improve system

performance.

The first check looks to see what external systems have been used in the last 30 days and either don’t

have an outbound XML profile assigned to them, or are using one that always sends the maximum

amount of integration, GLOG_MAX or V55MAX.

As mentioned in the warning section of this note you should review what integration is being sent to this

external system and determine how much of the information being sent is actually being used.

The next system checks determines if you are running the gather table stats job on a regular basis, if you

are using the Oracle provided job or the one provided specifically by OTM and if the sampling size is

correct.

The first portion of the check looks to see if you are running the job on a regular basis, we use 7 days as

our guideline. Any OTM table that has not been analyzed in that time will show up in this section of the

report. Next we provide a warning if we see you are not using the OTM gather table stats job and the

final section showing any tables where the sample size to be analyzed is not the same as the number of

rows in the table.

Note: Since this section can contain a number of lines of data we are only showing 5 on the screen.

Press the Show More link to display up to 200 lines of data.

The next check looks for any invalid objects that are owned by GLOGOWNER. The warning provides the

person reviewing the report the expected action to be taken.

We also look to see if any indexes were created within the last 7 days. This information may be

important if the analyzer is being run to investigate performance issues.

Another common issue we check for is the policy type being used for GLOGOWNER and REPORTOWNER.

If these are not set to SHARED_CONTEXT_SENSITIVE you could experience poor performance and

problems with VPD. If problems are found a warning similar to other sections would be generated with a

clear solution to the user for resolving the issue.

Note: Typically we see these problems where the instance was created by cloning the database.

System Transaction Checks

The System Transaction Check provides some basic information about the types of transmissions that

are being processes, what type of objects are coming in, how many of them, where they inbound or

outbound and average XML Blob length. The next section shows some basic information about the

processing time of the transactions over the same 15 day period the first system check used.

The OTM Functional Table Stats section it typically used to when investigating performance issues with

OTM. This helps us look for possible changes in key tables that typically contain static data. We look for

record counts min and max update dates and look for records that were created or updated recently in

these tables.

Process Control Request/History Check

The final section of the report is the Process Control Request/History Check. This section is another one

that we typically look at when performance issues are reported, but it does generate warnings in some

sections that should be reviewed if displayed. The first check looks for an old issue that would be

reported by clients as a performance issue. In older 6.x versions you were able to delete a user even

though they may have a scheduled process in the process control request able. This is no longer possible

in 6.1.7, 6.2.6 or any of the 6.3.x releases. If this condition is found a warning message would be

generated telling the user how to correct the issue.

The next check looks to see what recurring processes are scheduled to be triggered over the next 15

days. This may be used to investigate performance issues.

Another possible cause for performance issues are long running scheduled processes that do not have

the create new process flag set to true. If the queries being used returns large amounts of data you may

want to consider setting the create new process flag to true in order to create new processes for each

object returned by the query.

The last section shows what recurring processes ran over the last 15 days. This information is typically

used to troubleshoot performance issues.

qdlogs.pl support for the OTM Analyzer

If anyone of the call is familiar with qdlogs.pl script we use to collect some of the data needed to

investigate OTM issues you will be happy to know the latest version of the qdlogs script has an option

that allows you to run the OTM Analyzer and collect the html file as part of the qdlogs collection.

Run the following command showing the location and name of the script and the database SID, if no SID

is provided it will be taken from the glog.properties file from the glog.database.sid.

perl qdlogs.pl -prop /opt/otm62/glog/config/glog.properties -analyzer /opt/otm633/otm_analyzer.sql

Note: You will be prompted for the GLOGOWNER user password to connect the database before the

script is executed.

Once the script has completed a html file will be created in the same directory where the script was run.

otm_analyzer_<server>_<SID>_<YYYY>-<MMM>-<DD>_<HH>-<MM>.html

For the latest version of the qdlogs.pl script please review the following note showing the features and

typically collections we may request.

How to Use the qdlogs.pl Script to Collect Data for Service Requests for Oracle Transportation

Management (Doc ID 1519887.1)