Optimizing the Enterprise Manager 12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential

Optimizing the Enterprise Manager 12c EnvironmentEnterprise Manager 12c, Release 4

Kellyn Pot’Vin-GormanConsulting Member of Enterprise Manager Technical TeamStrategic Customer ProgramMay, 2015

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Oracle Confidential 4

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 5

Program Agenda

1

2

3

4

5

Oracle Cloud Control User Interface, (Console)

Oracle Management Service, (OMS)

Oracle Management Repository, (OMR)

Metric Collections and Jobs

Weblogic and Agent Tuning

Summary6


The Challenge•Slow Login Page Response

•Slow EM Page Response

•No Data Results to Screen

•Slow Refresh of Data Upon Request

•Job Delays

•Upload or Collection Delays/Not Uploading


Architecture


Program Agenda

1

2

3

4

5

Oracle Cloud Control User Interface, (Console)

6


Health OverviewSetup Manage Cloud Control Health Overview


Console Performance Page

Setup Manage Cloud Control Health Overview

OMS and Repository Monitoring Page Performance


Page Performance OMS and Repository Monitoring Page Performance


Page Correlation vs. Network/OMR/OMS


Which Pages Are Highest Impacting

OMS Repository Monitoring Page Performance


Java or Agent?


SQL and Plan

SQL from Page

Plans

Compare to awrsqrpt.sql

(aka sql_id specific awr)


Job Run

Not to be confused with user sessions


Load Balancers


Why a Load Balancer?

Load Balancer setup and configuration

– Users require session affinity to a particular OMS

This allows for new connections to perform Round-Robin vs. Most Requested which is less efficient.

Allows for simpler scaling for environments to support future growth.


LOAD

ER STATISTICS


Distributing Work Load to Multiple Oracle Mgmt Servers

Setup Manage Cloud Control Repository


Using the Access Log

$OMS_HOME/EMGC_OMS1/access.log

•Tracks each login, along with app/module logged in from:

Use NSLookup to locate workstation name.


Using EM CLI to Track UI Page


Controlling Login and Login Time

•Automatically controls logout of inactive sessions

•Single configuration setting for ALL OMS in environment-

•default is 45 minutes, but can be changed with EMCTL command:

$emctl set property -name oracle.sysman.eml.maxInactiveTime -value 30


Have You Succeeded? Bottom of Page Performance Page:

OMS and Repository Monitoring Page Performance


Program Agenda with Highlight

1

2

3

4

5

Oracle Management Service, (OMS)

6


What We Think Of When We Think of EM12c

FAILED


Enterprise Level- Enterprise Manager Environment

Maximum Availability Architecture


EM12c Project Consideration

•Although it requires licensing, consider RAC-One Node for any production repository to allow for easy scalability in future.

•8Gb for a small OMR, 12Gb+ for medium and 16Gb+ for large.

•Consider ASM to assist with tablespace maintenance and enhanced management. Remember, everything is in the MGMT_TABLESPACE.

•Consider installing the OMS and OMR on separate hardware. Each requires different resources, but can compete with each other if constrained.


Initialization Parameters for the Repository Database

Note the Compliance Checks

Setup Manage Cloud Control Repository


Why ODL Format Log Files Are ImportantOracle Diagnostic Log Files provide a lot of data in an aggregated format that allows you to locate what you need if you know what you are looking for!

Located in following directories for EM12c:$OMS_HOME/EMGC_OMS1$OMS_HOME/EMCG_ADMINSERVER$OMS_HOME/emoms$OMS_HOME/nodemanager$OMS_HOME/opmn_logs, etc., etc.*Agent has similar directory structures


EMGC AdminServer Diagnostics Log File


Format Translations

TimestampSource and Type

Weblogic Server Mgmt Product

Component ID

Error Message

Execution Context ID

Transaction ID


Google is our Friend

Search using the error message first, then add in filters with the context or type information to assist.


Tuning Java Heap Size

JAVA heap size for the OMS is a crucial setting. This value is used to accommodate memory for each user logged in.

$OMS_HOME/user_projects/domains/GCDomain/bin/startEMServer.sh

– Use OMS_HEAP_MAX, OMS_HEAP_MIN, OMS_PERMGEN_MAX, OMS_PERMGEN_MIN parameters to change the heap settings

$emctl set property -name OMS_HEAP_MAX -value 2560

– Stay between JAVA Heap size between 2Gb and 4Gb

Default values: OMS_HEAP_MAX: 1740M OMS_HEAP_MIN: 56M OMS_PERMGEN_MAX: 768M OMS_PERMGEN_MIN: 128M


Using EM Diagnostics

Repvfy(Repository Verify)

•Run level 9 –details

•Review Jmap output file.


EMGC_AdminServer.out

Includes Java Heap Settings in first five lines.

$OMS_HOME/EMGC_ADMINSERVER/EMGC_AdminServer.out

JAVA Memory arguments: -Xms256m -Xmx512m -XX:MaxPermSize=512m



1

3

2

4

5

Oracle Management Repository, (OMR)

6


Repository Support


The OMR SchemaSetup Manage Cloud Control Repository


Repository Space Usage in Table View


Partition Retention


Retention Killing Your Space?

Change at Table/Partition Level

> execute gc_interval_partition_mgr.set_retention('SYSMAN', <table name>, <number of partitions to retain>);

Modify retention or view impact of changing.

> repvfy send set_purge_retention -name {ALERTS|ERROR} -id <num days>


Management Server Incidents


OMR High Redo

Very common, consider at least 500M-1G Redo logs with at least 5 groups for high demand EM environments and consider retention data

>repvfy send set_purge_retention -name CONFIGURATION -id <num days>


The “Weight” of Different Tasks


Most Important Graphs for OMR

Setup Manage Cloud Control Repository Repository Tab


Upload Performance

•Uploads are collections of data from agents to the OMS.•If backlogs occur, experiences of-• Delayed notifications• Unresponsive agents• Inaccurate information to console• Delayed response to incidents from

jobs


SQL Baselines

For the SYSMAN schema, no SQL plan baselines should get captures, and no baselines should be used. The use of SQL plan baselines can severely impact the performance of the OMS and the repository (in a very negative way).SQL> ALTER SYSTEM SET optimizer_capture_sql_plan_baselines=false SCOPE=both;

SQL> ALTER SYSTEM SET optimizer_use_sql_plan_baselines=false SCOPE=both;


Repository Performance•Average Collection Duration in Seconds

•Number of Collections in Backlog**

•Throughput per Second

•Metric Data Rollup Performance, (metric extensions included)


Task Worker ThreadsThese workers process tasks and messages from the internal subsystems. These internal housekeeping jobs are executed by 2 main DBMS_SCHEDULER jobs in the repository database:

One to execute small jobs (SHORT worker pool), and one to execute larger, more time consuming jobs (LONG worker pool).

The number of 'workers' for each of these pools is configurable, and can be changed. To set the minimum of 2 workers for both the short and long running pool, use this command:

>repvfy send set_workers



1

5

2

3

4 Metric Collections and Jobs

6


All Target Metrics Are Not Created Equal- Uploads by Day

Exadata Storage Server Target: 27K rows, 9.5M

Cluster Target: 23K rows, 29M

Infiniband Switch: 27K rows, 11M

Database Instance: 122K rows, 7M

Oracle Home: 8K rows, 3M


How Much Metric Data Am I Receiving?

Data can be gathered by running an EMDiag Report:

>repvfy dump metric_stats

Or….


Top Data Loading MetricsSetup Manage Cloud Control Repository Metrics Tab


Setup Manage Cloud Control Repository Metrics Tab


Demands and Consequences


Cluster Detail Bubble Map


Metric Collection ErrorsSetup Manage Cloud Control Repository Metrics Tab


Collection Performance ReconfigureUpdate from Short Running to Long Running Graphed


Metric Raw Data

Metric data represents all the metric collections that are enabled against a target (3 main settings).

Raw Data, 7 days by default. Largest data, take care in changing retention.

Once per day, raw data is aggregated by day, month and 366 days. To change this retention:

>repvfy send set_data_retention -name {RAW|HOUR|DAY} -id <num days>


Advanced Troubleshooting•Highlight a metric alert type•Hover cursor to view options to:

•Problem Analysis•Metric Details•Target Home

OMS and Repository Monitoring Metrics Collections


Problem Analysis


Files Processed Vs. Backoff

Outage


Collection Performance- Improvement?Setup Manage Cloud Control Repository


Repository Job Activity

>repvfy send set_job_retention -name <job module> -id <num days>


Optimizing OMS Jobs

•Backload of Jobs, consider updating the parameter job_queue_processes up to 10.

•Edit the schedule for the daily jobs to run in off-peak time or more often to lessen impact.


Job Pool Tuning

In any environment relying on EM to execute jobs and deployment procedures, the size of the job connection pool from the OMS to the repository will have to get tuned.

To change the site of the job pool connection pool, run this command on each OMS in the environment:

>emctl set property -name oracle.sysman.core.conn.maxConnForJobWorkers-value <recommended> -module emoms



1

5

2

3

4

Weblogic and Agent Tuning

6


Weblogic Server- Stuck Threads

Stuck Thread Timeout (BEA-337 incidents in the WebLogic log/trc files)

From OMS in Admin Console:

Stuck Thread Timeout = 1800

Stuck Thread Timer Interval = 120rom OMS in Admin Console:

Admin Console → Environment → Servers → EMGC_OMSx→ Tuning


Weblogic: LDAP Configuration Flag

For LDAP and SSO setups, always make sure the local LDAP can be used.

Without this set, users are unable to login, (even as SYSMAN) remotely. The console will report remote authorization and authentication as down or unavailable.To correct, set in the Admin Console:

LDAP Provider Control Flag = Sufficient

Admin Console → Security Realm -> myrealm -> Providers → LDAP Provider


Agents

•Separate Management Area from all OMS/OMR Performance Screens.

•How many targets are monitored by the agent.

•Incidents involving the agent or dependent targets.

•Last successful upload to the OMS.


Agents

Setup Manage Cloud Control Agents


Agent Down Detection

To eliminate staleness of agents after downtime and allow OMS's to catch up, force the agents to send heartbeats with a more effective grace period by defining in the system. This grace period serves as a time period right after an outage during which the OMS’s will NOT report any Agent as 'down'.

SQL> exec gc_diag2_ext.SetPingGracePeriod;

To set the grace period to a specific value, use this command:

SQL> exec gc_diag2_ext.SetPingGracePeriod(90);

The value for the grace period should be between 60 and 300 seconds. Higher values should only be used if specifically instructed by Oracle.

Setup Manage Cloud Control Agents


View by Name Default


Agent HeartbeatIncoming heartbeats from agents are “batched” to allow for bulk updates in the repository. For optimal performance, the size of such a bulk operation performed by a single thread in the OMS should never exceed 1000 Agents.

The default number of threads configured per OMS is two, which should be sufficient to handle up to 2000 Agents per OMS. If the number of Agents is above this, the number of threads to do the Agent heartbeat processing should be increased:

>emctl set property -name oracle.sysman.core.omsAgentComm.ping.heartbeatPingRecorderThreads –value "3"


Auto Tuning Set in Agents

To make sure the auto tuning is enabled check the value of the 'enableAutoTuning' parameter:

>emctl getproperty agent -name enableAutoTuning

If the value is not set to true, change it, and restart the Agent.


Agents with Variety of Target Types

If the Agent is monitoring a various number of different target types, the overall timeout value can be changed, to avoid having to specify the timeout for each individual target type one by one:

>emctl setproperty agent -allow_new -name dynamicPropsComputeTimeout -value 180


Agent Tuning on Incoming to OMS

Similarly, if there are a lot of targets (+50) of a given type discovered on the Agent, and those targets are administered in bulk (by applying templates, submitting jobs, or performing repetitive tasks), the number of incoming connections from the OMS to the Agent can exceed the default of 25. For those cases, the default value can be increased. Values above 50 should be extreme cases, and need to be validated and verified by Oracle.

>emctl setproperty agent -allow_new -name MaxInComingConnections –value 40


Agent Patching TipsPre-stage patches

To apply a set of patches automatically to an Agent while deploying it via the console, stage the patches in the

>$OMS_HOME/install/oneoffs/<version>/Generic/ (generic platform independent patches)

and >$OMS_HOME/install/oneoffs/<version>/<platform>/ (platform specific

patches)Every Agent path to apply will have to get copied on each of the OMS’s, to make sure that regardless which OMS does the Agent install, the pre-staged patches will get applied.

Download Agent Software and OpatchThe Agent software for the managed platforms can be downloaded using the Self Update pages in the console.Before installing additional agents for other platforms, ensure that the OPatch download job has completed. This job runs on a 24hr cycle, and will get the latest OPatch from My Oracle Support.



1

5

2

3

4

Summary6


Summary•Best Performance requires different skills for optimizing Enterprise Manager Cloud Control than for a database or engineered system.

•There are impressive enhancements and bug fixes in the PSU patches and yes, they exist for EM12c- the OMS, OMR and Agent. Address this FIRST!

•Don’t get overwhelmed- use the tools provided in the Manage Cloud Control and rep/agt/oms vfy to inspect the environment on a quarterly basis.

•Don’t just settle with the default values for the environment- many times these are set by development and you most often use your EM12c differently than they imagined or plugins, etc. can change usage.


Staying on Top of PSU Patching

96% of issues experienced are addressed by applying OMS and Agent patches. Please remember to provide tender loving care to your Enterprise Manager…


Tip Working With MOS and Service Requests

omsvfy “Diagnostic dump” contains a file called “env”.

File contains all lib files, all environment information, ports, etc. that will be required for an service request. Upload this one file and save a LOT of time..


Want to Learn More?

DBAKevlar Blog Posts: http://dbakevlar.com/2014/10/em12c-release-4-health-overview/http://dbakevlar.com/2014/11/em12c-rel-4-oms-and-omr-health-part-ii/http://dbakevlar.com/2014/12/oms-and-omr-performance-part-iii-metrics-page/http://dbakevlar.com/2014/11/agents-management-via-em12c-release-4-console/Change Retention: http://docs.oracle.com/cd/E24628_01/doc.121/e24473/repository.htm#EMADM12785

http://dbakevlar.com/2014/10/em12c-release-4-health-overview/

http://dbakevlar.com/2014/11/em12c-rel-4-oms-and-omr-health-part-ii/

http://dbakevlar.com/2014/12/oms-and-omr-performance-part-iii-metrics-page/

http://dbakevlar.com/2014/12/oms-and-omr-performance-part-iii-metrics-page/

http://dbakevlar.com/2014/11/agents-management-via-em12c-release-4-console/

http://dbakevlar.com/2014/11/agents-management-via-em12c-release-4-console/

http://docs.oracle.com/cd/E24628_01/doc.121/e24473/repository.htm

http://docs.oracle.com/cd/E24628_01/doc.121/e24473/repository.htm


Connect with me via Social Media:

Optimizing the Enterprise Manager 12c

Technology

Transcript of Optimizing the Enterprise Manager 12c