Pandora FMS: Oracle Enterprise Plugin

Pandora FMSAdministrator Manual

Oracle Monitoring on Unix environments

Administrator Manual Oracle Monitoring on Unix environments

© Artica Soluciones Tecnológicas 20052012

Index1Changelog...........................................................................................................................................32Introduction........................................................................................................................................43Compatibility matrix..........................................................................................................................54Compulsory documentation to hand in by the area that requires the monitoring .............................65Plugin modules...................................................................................................................................7

5.1.Multi instance Monitoring.........................................................................................................85.2.Monitoring Automatic Pre-conditions ......................................................................................85.3.Plugin parametrization...............................................................................................................9

5.3.1.Location of the Oratab file.................................................................................................95.3.2.Information and Error Log ................................................................................................95.3.3.“Skip” Checks by default.................................................................................................105.3.4.Configuration block of one instance................................................................................10

5.3.4.1.Configuring the listener and services.......................................................................105.3.4.2.Monitoring Preconditions ........................................................................................11

5.3.5.Restarting of the listener..................................................................................................115.3.6.Monitoring Specifics SID................................................................................................11

5.3.6.1.Monitoring an Specific SID(sid)..............................................................................125.3.6.2.SID Specific Monitoring(only_sid)..........................................................................125.3.6.3.Monitoring Specific SID (skip_sid).........................................................................12

1 CHANGELOG

Date Author Change Version

29/09/11 Juanma First Version v1r1

13/12/11 Juanma New features v1v4

22/12/11 Juanma New features v1r5

13/03/12 Juanma New features V1r7

18/06/12 Juanma New modules V2r1

25/07/12 Juanma New features v2r2

17/09/13 Mario New features v2r3

2 INTRODUCTION

The main aim of this document is to describe the Oracle systems monitoring based on Unix. Some

“base” modules have been chosen according with our experience in system monitoring and the

necesities of some of our clients. Besides, all the specifications collected in different real production

environments have been added, taking real specifications of database administrators.

For extracting information we use:

• An external configuration file where all the plugin parameterization is defined. This

configuration file could do calls (includes) to other files.

• One file of environment variables parameterized by instance.This file is located at

$ORACLE_HOME/dbs/orauser_$SID directory.

• We use the software that is already installed in the system (SQLplus, lsnrctl,Oracle alert

files, etc), for the monitoring done by the plugin without having to install libraries or third

parts utilities. Optionally, the ORATAB file could be used if the instances to monitor are not

configured directly in the plugin configuration file.

• An already existing log parser is used (the one from Pandora) to process Oracle alert

logs . The model to search could be parameterized through one token in the plugin

configuration file. The detection of the alert log files will be automatic for each instance.

• Autodetection of SID, instances, and system tablespaces are done, but the administrator

could be deleted, and/or force specific SID , configuring different tokens in the plugin

configuration file .

• Some basic checks “by default” are done, but they could be deleted or customized.

• An “open” interface is available to specify free SQL queries, allowing to modelate all

kind of SQL queries that are done with other tools or in a manual way for the

administrators. We also provide some queries usually used by administrators in all the

world, and that offer important imformation about the performance.

• The system is integrated with the Unix agent and it could distribute file colections, so it is

posible to distribute the plugin by one hand and the file collections in an individual way by

agentand/or by policy.

We need to mention that as with the rest of monitoring with Pandora FMS, the Oracle monitoring

plugin could be used to collect information type “text string”(to process it as events) or type

numeric (to do performance management).

3 COMPATIBILITY MATRIX

The compatibility matrix for the plugin is shown here:

Systems where it has been tested• Solaris 10 Sparc, Oracle 11g• Linux RHEL 5, Oracle 10.x, Oracle 11.x

Systems where it should work

• Solaris 10 and higher, Oracle 11.x and higher

• Linux RHEL 5 and higher, Oracle 10.x, Oracle 11.x and higher

4 COMPULSORY DOCUMENTATION TO HAND IN BY THE AREA THAT REQUIRES THE MONITORING

In order that the Oracle monitoring could be done correctly, it is necessary that the technical Area

sends same specific information that will be included in the configuration files. This information is

the following:

• User and password with access to the Oracle engine via SQLplus. This user should have

permission to read in all the tables of the system (see following point) and be in the Oracle

system group.

• Name of the DDBB instance (Oracle SID).

• In case of using the ORATAB file: Verify that there is a file /var/opt/oracle/oratab(or in

other path as /etc/oratab) that is correct and also with the paths and SID corrects.

• If you want to monitor other component that does not come defined “by default” it will be

necessary that you gives the SQL code to do this monitoring, and also an exit data example,

specifying if it is numeric, kind string, etc).

• If the DBA environment variables (like PATH to the SQLplus and lsnrctl binaries,

TNS_ADMIN, ORACLE_HOME, ORACLE_BASE, etc.) are not exported as environment

variables, the plugin will need access to the environment variables file:

$ORACLE_HOME/dbs/orauser_$ORACLE_SID of each instance and then you could show it

specifically in the plugin configuration file.

5 PLUGIN MODULES

The plugin does“by default” the following things:

• Scanning of the tablespace to get free space.

• Scanning of the tablespace to get status.

• Scanning of Oracle filesystems working on the system (in base of one prefix and on the

name of the instance (SID).

• Verification of the connection to the DB (through one query (Select) and getting one date).

• Verification of the services with a listener (only at instance level).

• Verification of Pmon process of each SID .

• Verification of the general status of the listener (for all instances).

• Error parsing looking for a regular expression in the alert log. The alert log depends on each

instance and its exact location is auto detected.The plugin is necessary to parse Pandora logs

in order to process these logs.

Besides it also monitors the following performance parameters, such as is recommended by many

administrators who are experts in Oracle for different kinds of environments:

• Dictionary Cache Hit Ratio

• Library Cache Hit Ratio

• DB Block Buffer Cache Hit Ratio

• Latch Hit Ratio

• Disk Sort Ratio

• Rollback Segment Waits

• Dispatcher Workload

• Active sessions

• Concurrent parallel sessions

• Redo log write time

• Redo log buffer space wait

• Enqueue waits

• Free buffer waits

NOTE: Its important to say that if the database to monitor does not have configured some of the

statistics before mentioned, then the corrresponding modules will figure as not initialized in the

Pandora FMS console.

None of the performance parameters has associated by default any kind of event, and it is posible to

assign thresolds to assign them alerts.

All these modules come parameterized in the “oracle.conf” file that comes in the plugin package.

These modules are SQL, that could be checked in the file and that could be deleted, modified or

extended by an Oracle administrator.

You can find more information about these queries and other, recommended by several Oracle

administrators communities in the following link:

http://www.hoopoes.com/cs/oracle_tune.shtml

5.1. Multi instance Monitoring

The plugin, besides doing a monitoring in a general level of all the instances, allows to monitor

several instances in a more specific way.

This way the plugin allows a more flexible configuration:

• Monitoring at a general level: By one part, the monitoring by default will be done, as is

explained in the previous point.

• Monitoring at an instance level: You could configure the monitoring by specific instance.

NOTE : It is important to mention that the monitoring at an instance level has priority over the

general monitoring, so it is considered as a more specific one. This will avoid that duplicated modules

will be generated in case that the same instance will be configured for general checks and in a block of

instance configuration(see in the following point) of the configuration file.

In order that the modules of the different instances dont delete themselves, the notation that the

modules use will be

ORA_{SID}_{Comprobación}

5.2. Monitoring Automatic Preconditions

To avoid that duplicated modules would be generated, some conditions have been fixed that should

be fulfilled before checking. If these conditions are not satified, then, the monitoring will be

aborted. These conditions are at two levels:

http://www.hoopoes.com/cs/oracle_tune.shtml

• At a general level: the access to the configuration file will be verified, and also that the SQL

plus binary would be available, the listener status (if the listener verification is not available)

and that the pmon process is running (if it is not disabled)

• At an instance level: It will be check if there is an script that determines the monitoring of the

corresponding instance, the pmon process is verified for that instance (if it is not disabled),

then the listener status is verified and also their services (if the listener verification is not

disabled).

5.3. Plugin parametrization

The plugin is used through the configuration in an external file of the configuration

NOTE: It is extremely important to consider that the configuration files thought for the UNIX plugin should

be edited and stored with carriage returns kind “UNIX” and that if “WINDOWS” carriage returns are used,

then the plugin will not work correctly.

There are some specific checks that have their own configuration “tokens” that are described next:

5.3.1. Location of the Oratab file

In case of using it. Remember that the oratab file stores the instance information and the home directory of

that instance. By default, the script understand that the oratab file is located at:

/var/opt/oracle/oratab

If the oratab file would not in that location, it is posible to specify an alternative in the

configuration file.

oratab /etc/oratab

This will do that it looks for the oratab file in an alternative location. In this example the /etc.

directory.

5.3.2. Information and Error Log

All the actions done by the plugin will be registered in a log file which name will be parameterized in the

following way:

/tmp/pandora_ora_{SID}

5.3.3. “Skip” Checks by default

It is posible to “skip” the checks by default using the correct configuration token:

skip_listener_status skip_tablespace_free skip_tablespace_status skip_oracle_fs skip_connect_check skip_alert_log skip_pmon

These tokens will be applied at a general level or at an instance level. This will allow to do the

monitoring flexible. For example, supposing that in oratab we have configured the “hpsacert”

instance and in an instance configuration block we have it configured also (see next). If we want to

do an specific parse on the alert log of this instance and not on the other ones, then we should leave

out the skip_alert_log token (see next) in the instance configuration block . As we mentioned before, the

monitoring at an instance level has priority over the general one, so an specific parse will be executed for

this instance.

5.3.4. Configuration block of one instance

To do the specific configuration of one instance, you should use the following tokens:

instance_begin instance_name xxxxxx <tokens de configuración>instance_end

On this block, it is posible to use the tokens that have been explained before, and this way to chose

if the corresponding check would be done or not.

Besides, the following tokens could be used only in this block:

5.3.4.1. Configuring the listener and services

In order that the plugin monitors the service status through the listener, you should configure both values, the

listener name and the name of each of the services . For this we will use the following configuration

syntax:

listener_name LISTENER_RHGE0208

This will define the name of the listener, in this example “LISTENER_RHGE0208”. It should be the

exact name of the listener that we want to monitor.

By other hand, it is necessary to specify the service or the services that we want to check that are listening in

this listener.This is done in the following configuration tokens:

service OIM service OIMXA service OIMXDB service OIM_XPT

NOTE : In case there were several listeners in a machine, we should configure these tokens in each of the

configuration blocks of the instance.

5.3.4.2. Monitoring Preconditions

It is posible to determine the monitoring of one instance if one scrip returns 1. For this, it is posible to use the

token conditional_monitoring in the instance block.

For example:

conditional_monitoring /var/plugin\ oracle/conditional\ script

NOTE: You should write the complete path of the script. In case that the path or the name of the script have

spaces, they should be masked.

5.3.5. Restarting of the listenerIn case that the listener checking (would be it like global check or an instance check) does not answer, it is

posible to enable one scrip for it could be executed and try to restart the listener. Each time that this script

would be executed, the number of the listener restarts should be stored in the file o /tmp/

{sid}_listener_restart. This counter will be reset every 24 hours. The format of this file is the following:

RefDate: yyyy-mm-dd hh:mi:ssReinicios totales: xx

RefDate will have the date in which the listener restart counter was reseted.

5.3.6. Monitoring Specifics SIDIt is posible to limit the number of instances to which we are going to do the checks in a general level.For

doing this we have the following tokens: sid, only_sid y skip_sid.

5.3.6.1. Monitoring an Specific SID(sid)

In the common parameters for all the instances, it is posible to configure this parmeter:

Example:

sid <nombre instancia>

With this, all checks at a general level will be done only on this instance, without consider the rest

of the configured instances.

5.3.6.2. SID Specific Monitoring(only_sid)

This token is useful to limit the number of instances to monitor. Through the only_sid token the system is

forced to monitor that SID (or several if we introduce several lines).

Example:

only_sid pandora01

With this call, of all SIDS that the DDBB contains, it will only monitor the “pandora01” instance.

5.3.6.3. Monitoring Specific SID (skip_sid)

If with the token only_sid xxxx we limit the number of objective instances, the token skip_sid is useful just

for the contrary. With it, the instance (or instances) selected will be not included in the global checks.

Example:

skip_sid pandora01

So, then we will not include the instance “pandora01” in the global checks.

5.3.6.4. Assignment of these tokens

Due to that these tokens are not compatible between them, the way to assign them will be the following and

one of them will prevail over the other:

1. sid

2. only_sid

3. skip_sid

5.3.7. Monitoring the Disk volumes of one Instance The system will try to “auto detect” the volumes in base of one “prefix” and to the name of each monitored

instance. For this, we use the token volume_preffix to define the prefix of all the Oracle volumes.

For example:

volume_preffix /aplicaciones/oracle

If one of the SID is “pandora01”, then automatically will detect all the disk volumes contained in one path

that has “oracle/applications” and that has “pandora01” in its name, as for example:

/aplicaciones/oracle/arc_pandora01

/aplicaciones/oracle/rdbms_pandora01

5.3.8. Not using the ORATAB file In some of the Oracle installations (for example the RAC ones) there is not available one ORATAB file that

reflects the different instances that are going to be monitored, so this file will not be used. This way you

should show specifically through tokens only_sid in the configuration file the Sids of the instances to

monitor and the variable $ORACLE_HOME.

For example:

skip_oratabonly_sid pandora01only_sid AST1oracle_home /oracle/product/11.0.1

5.3.9. Monitoring “Separated” disk volumes The system tries to “auto detect” the Oracle volumes, according ot the previous point. If it is not able to

detect it, or the administrator want to include “separated” volumes, then it is posible to monitor using the

volume token.

For example:

volume /var

volume /oracle

5.3.10. Access CredentialsThey are necessary to use the plugin

user syspassword pandoraA01 AS SYSDBA

Note: The use of the connection as SYSDBA is optional. The same line could be like this (if the user has

permissions for the tables and special views that are required).

user oracle password oracle

Please, consider that here the “full”mode is not used, and that a kind of numerical data is used.

5.3.11. Log parserBy default, we use the log parse plugin that is located in the path /etc/pandora/plugins/grep_log but it is

posible to modify the path through the logparser token. For example:

logparser /var/opt/PandoraFMS/etc/pandora/plugins/grep_log

5.3.12. Processing the Alert logs To process the Oracle alert logs, you should omit the skip_alert_log token and configure the log_pattern

token as a regular expression. For example, to search the message ORA- in the alert log:

#skip_alert_loglog_pattern ORA-*

5.3.13. Monitoring vía SQLOne of the most powerful features of the plugin is the posibility of specify its own SQL order to get the

value. Let's see some examples:

custom_query_begin custom_query_name NUM_MAX_FICHEROS custom_query_sid xxxxcustom_query_type generic_data custom_query_sql_begin select round ((ficheros_act/ficheros_max)*100) ratio from (select count(*) ficheros_act from dba_data_files), (select value ficheros_max from v$parameter where name='db_files'); custom_query_sql_end custom_query_end

Custom_query_sid (Optional):Defines one SID for this SQL query, and it will only execute it for this

instance. If it is not specified it will be executed for all instances.

Custom_query_type: It should be of a valid type in Pandora, for example generic_data for numeric data or

async_string for text strings .

Custom_query_mode: It will be “full” when we want to get all the exit (for example one list). When we use

the “full” mode we always should use a kind of data kind string (async_string).

Custom_query_description It is optional and is useful to send a description with the module.

Custom_query_condition <script> If this script returns 1, the query of this block will be executed. On the

contrary, it will not. You should write the complete path and mask the spaces.

Lets see an example that uses an SQL query to get discrete values:

custom_query_begin custom_query_name SQL_SESS_CONCURRENTES custom_query_type generic_datacustom_query_condition /var/oracle/conditional\ scriptcustom_query_description Devuelve el porcentaje de sesiones concurrentes.custom_query_sql_begin select round ( (sesiones*100)/value ) from (select count(*) sesiones from v$session), v$parameter ; custom_query_sql_end custom_query_end

Please consider that here we do not use the “full” mode, and that we use a kind of numeric data

instead.

In the plugin package is distributed one script (pmon_ASM_test) that allows to check if the process

pmon_+ASM is running and that could be used with the token custom_query_condition. If so it will give

back 1. On the contrary, it will return 0.

5.3.14. Creating modules for logs by handSo as the error log module does not send information until there is an error, the module should be

created manually. The name of the module always depends on the DDBB instance. The DDBB

instance names will be shown in the agent monitoring in serveral places, for example:

In this case, the DDBB instance is called “hpsacert”. For it, we will create manually a new

module, like this:

And after we should create an alert. Before assigning the alert templatea we should be sure that we

have an alert template that sends an event when we get something in the module. This is done with

a template similar to this:

The fire conditions are special, so we will be interested in get a short timethreshold, just in case that

more events come, that will be always different and that could be sent. However, we will put it a

minimum time to avoid an storm of repeated events.

And finally, we will asign the alert module log to the error template in logs:

6 REQUISITES

The requisites for this monitoring could work properly are these:

• To install the Pandora FMS agent.

• To have one Oracle installed in the machine where it is going to be monitored, with the

basic tools (SQLplus, lsnrctl).

• To specify the name of the Oracle listener that you want to monitor, and also the name of

each service that you want to monitor in the listener for each of the instances reflected in

the configuration file. Both things are necessary to monitor the listener at an instance level.

• It is necessary that the user with which the Pandora FMS agent is executed, that is the user

that will execute the plugin, has access to the following resources of Oracle:

◦ SQLplus

◦ lsnrctl (correctly configured).

◦ All DBA environment variables exported to the user that executes the plugin. This

user also needs to belong to the Oracle system group.

◦ Alert log files (dinamically got for each instance).

◦ The PATH to the SQLplus and lsnrctl binaries should be available for the pluguin:

▪ If a monitoring is done on a single instance it could be exported as environment

variable:

▪ The plugin will search this variable in the environment variable files

$ORACLE_HOME/dbs/orauser_$ORACLE_SID and in case that you find them it

will load them.

◦ The variables $ORACLE_HOME y $ORACLE_SID are available for the plugin. In case

of using the ORATAB file, they will be extracted from this file. In case that you dont

use this file, you could show it in the plugin configuration file or they could be

charged as environment variables in case of monoinstance monitoring.

• To do the monitoring, the plugin will call to SQLplus to do different SQL queries getting the

information.

• The plugin should be able to write temporal files in the path /tmp

• It should be necessary to give the list of disk volumes to monitor and in case of doing the

listeners monitoring, the listener and services list associated.

• The user that executes the Pandora FMS agent should belong to the database explotation

group to could get access to the the tnsnames.ora file

6.1. Requisites to could have access to the DDBB

• The plugin needs an user and a password to connect with the DDBB. This user should have

enough privileges to check some of the system tables, in order to get information.

• The user that is provided could be “normal” or use a connection like SYSDBA, in any case

this is specified in the “password”parameter of the plugin configuration file. And belong to

the Oracle system group.

• Preparation of the database. It is necessary to have an user with access priviledges for some

of the tables. In this example is detailed how to give access to same tables that the plugin

use by default, for the user “pandora” with password "pandora". The plugin will always do

the connections to the local machine (localhost).

CREATE USER pandora IDENTIFIED BY pandora; GRANT CREATE SESSION TO pandora; GRANT SELECT any dictionary TO pandora; GRANT SELECT ON V_$SYSSTAT TO pandora; GRANT SELECT ON V_$STATNAME TO pandora; GRANT SELECT ON gv$sysstat TO pandora; GRANT SELECT ON v$sesstat TO pandora; GRANT SELECT ON V_$INSTANCE TO pandora; GRANT SELECT ON V_$LOG TO pandora; GRANT SELECT ON SYS.DBA_DATA_FILES TO pandora; GRANT SELECT ON SYS.DBA_FREE_SPACE TO pandora; GRANT SELECT ON V_$parameter TO pandora; GRANT SELECT ON dba_tablespaces TO pandora; GRANT SELECT ON dba_data_files TO pandora; GRANT SELECT ON dba_free_space TO pandora; . . (others GRANTs necessary, for all tables used in the plugin configuration file) . .

-- -- if somebody still uses Oracle 8.1.7...

GRANT SELECT ON sys.dba_tablespaces TO pandora; GRANT SELECT ON dba_temp_files TO pandora; GRANT SELECT ON sys.v_$Temp_extent_pool TO pandora; GRANT SELECT ON sys.v_$TEMP_SPACE_HEADER TO pandora; GRANT SELECT ON sys.v_$session TO pandora;

7 INSTALLATION

Copy the plugin to the agent plugin directory, or distribute it with file collections. Do the same with

the conf.file . The call from the agent will be similar to this, but using the paths where the plugin

and the conf are installed.

module_plugin perl /var/opt/PandoraFMS/etc/pandora/plugins/oracle.pl /var/opt/PandoraFMS/etc/pandora/collections/fc_23/oracle.conf

8 USING THE PLUGIN IN POLICIES

The use of this script with policies is very easy. It is based on creating a policy that contains the

plugin (pandora_oracle.pl) and a configuration file for each agent. This way, the call to the policy

plugin would use the $HOSTNAME variable to replace it in time of execution by the hostname

(complete) of the machine. This is an example:

If the machine is called ilp0x068.tsm.inet, we should upload a file named ilp0x068.tsm.inet

oracle.conf as we can see in the following screenshot.

The content of the .conf is specific for each system, but all the .conf are copies to all the systems

associated to the policy. This way, although with permissions only the user that executes the

pandora agent could read these .conf (where the access password to the oracle are), it is

recommended to use user of only reading and that these users will have only local access.

Pandora FMS: Oracle Enterprise Plugin

Technology

Transcript of Pandora FMS: Oracle Enterprise Plugin