1 Copyright © 2013, Oracle and/or its affiliates. All ... · Exadata Database Machine Monitoring...

Comprehensive Oracle Exadata Database Machine Monitoring Made Easy James Viscusi – Consulting Member of Technical Staff, MAA Team

Larry Justice – Master Principal Sales Consultant, ESG


Safe Harbor

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


What questions will we address today?

 Who are we?

 Who should be here? –  The Exadata administrator and their management

 What is the value of your hour? –  We’ll review the tool box available for Exadata Manageability –  We’ll give you some easy ways to get started –  We’ll show you how the tools help with common use cases –  We’ll mention what others are doing in house and in the field


Managing An Exadata Database Machine

  Traditional systems and Exadata both run the same Oracle Database software

–  Everything you know and do today to run the Oracle Database is the same on Exadata – It’s Just Faster!

  The DBA team will naturally take on a larger role in managing a purpose built database system like the Exadata Database Machine

–  Many of the administrative activities are database specific and best handled by the DBA team

–  You need to decide who is going to take on the Administrative leadership of Exadata systems

Trend and Direction: the Database Machine Administrator


Tools for the Exadata Administrator…

  Inventory the tool box –  EM –  ASR –  Exachk

 Deploy the tools –  Exakit

 Keep them sharp / know when to use them –  Audit the install


What is Enterprise Manager? Repository:   Heart   Data storage, rollup and purging

Management Server (OMS):   Brains   Handles management data from Agents   Handles administration requests from UI   Sends out notifications

Agents:   Worker Bees   Gather management information   Perform administration task

Repository Database

Management Server

EM Users: EMCLI Console / Reports

Agent

Agent

Agent

Firewall

Notifications

Connectors

Internet


OMS, Agent and OEM Exadata Plug-in interrelationship 1.  OEM Agent deployed on the

Compute Node 2.  Exadata Plug-in deployed

with the Agent 3.  Agent communicates with

Storage Server and Infiniband Switch targets using ssh.

4.  Exadata Plug-In also monitors the other DBM components

5.  Agent collects data and communicates with the EM Management Server

ORACLE DATABASE MACHINE COMPUTE NODE #1

DATABASE SERVER

Oracle Enterprise Manager 12c Agent

Exadata Plug-‐in

Exadata Storage Server

Exadata Infiniband Switches and Network

Other DBM Devices PDU

ILOM CISCO S/W

KVM

1

EM Repository

5

4 3

2


Plug-in Framework

Update entities: •  Download from My Oracle Support via the

Self Update feature in the UI •  Or manually upload into the repository via

EMCLI

Deploy entities: •  OMS’s side (for everyone) •  Agent side (selective)

Plug-ins for the Engineered systems: •  Exadata •  ExaCheck (Health-Check) •  Exalogic •  More…


Exadata ‘ExaKits’ for OMS and Agent

•  For the infrastructure (OMS/OMR), download patch 13546881 •  Installs and configures a new Enterprise Manager environment

•  Includes repository database (11gR2 + PSU) and OMS

•  For Agent Deployments •  Use the Agent push method from the UI and deploy the plug-ins •  Or use the Exadata Agent Kit (download patch 13551607)

Note: The Exadata Kits get updated after each PSU release of EM, the plug-ins and Exadata HW refreshes


EMDIAG - Exadata Diagnostics •  Diagnostics tools for Enterprise Manager

  Tool for each tier: AGTVFY, OMSVFY, REPVFY

•  Exadata specific diagnostics  On the repository side:

$ repvfy verify exadata [-level {0-9}] [-details] $ repvfy dump exadata –name "<db machine name>" $ repvfy show exadata_master_agent –name "<db machine name>"

 On the Agent side: $ agtvfy verify exadata [-level {0-9}] [-details] $ agtvfy version exadata

•  CON6287 - Managing the Manager: Tips and Tricks •  Operational Considerations and Troubleshooting Oracle

Enterprise Manager 12c


Direct action - Oracle ASR

Comprehensive H/W Fault Coverage

•  CPU •  Disk controllers •  Disks •  Flash Cards •  Flash modules •  Memory •  System Board •  Power supplies •  Fans •  IB switches!

Customer Data Center Oracle Support Services

! Fault

Product’s auto-diagnosis facility sends SNMP trap to ASR

Manager

Service Request created

FRU dispatched by Support Engineer

FRU replaced by Field Engineer

ASR Manager

SR creation email notification to customer

Fault telemetry securely transmitted to Oracle

SR to routed Support Engineer

Customer

Oracle Field Engineer

ASR Service


Set up ASR and watch it run! •  Documentation and fault list available from www.oracle.com/asr

•  Exadata specific documentation – Exadata Quick install Guide

•  ASR errors and notes – by ILOM, for every supported ILOM (more than Exadata)

•  Latest version now auto update (from v.4.3 on)

•  One Command Integration (from 11.2.3.1 forward) •  Now works with latest version of OneCommand •  Set up the SNMP trap forwarding •  Produces the script to register with ASR Manager

•  Auditing your install •  MOS 1450112.1 - download the latest version of asrexacheck

•  Verifies connectivity to Oracle back end •  Now Integrated with Exacheck


Exadata Healthchecks (Exachk)

Other Best Practices checks

RAC Exadata MAA

Configuration Best Practices

Operating System Clusterware ASM Infiniband

Hardware

Database Server Infiniband Exadata Storage Servers

Software Checks

Firmware Operating System Clusterware ASM Database

•  Based on exachk (1070954.1) utility

•  Pre-Installed in Exadata deployments

•  Audits configuration settings of the following categories

•  Software •  Hardware •  Configuration Best

Practices •  Oracle Exadata Best

Practices (757552.1)


When should you execute exachk ?

3

Metrics X2-2 V2 X2-8

Compute Node Level

94 118 56

DB level 52 58 44

Cluster wide Level

11 19 1

Storage Server Level*

25 44 19

IB level* 11 11 11

Total 193 250 131

* Checks are skipped in silent mode due to access constraints

•  After initial Oracle Database Machine deployment

•  Before planned maintenance activities

•  After planned maintenance activities

•  At least every three months as part of regular operational procedures and at times of least load


Exadata Management Integrated View of Hardware and Software

  Hardware view •  Schematic of cells, compute nodes

and switches •  Hardware components alerts

  Software/system view •  Performance, availability, usage by

databases, services, clusters •  Software alerts db, cluster, ASM •  Topology view of DB systems/

clusters   Configuration view

•  Version summary of all components


Discovering Exadata System

Install Agent on Compute

Node

Launch Auto

Discovery

Assign Monitoring

Agents Component Credentials

Monitoring Properties Review

•  Discovery Wizard (guided discovery) •  Plug-in has all the components bundled (Cell, KVM, PDU, Cisco Switch)

Exadata Cell and the Infiniband components discovered out-of-the box •  Option to discover new DB Machine or rediscover (Add DB Machine Members)


Post configuration steps  Even if things go well, double check!

–  Audit Script - 1473912.1: Prerequisite Script for Exadata Discovery with Enterprise Manager 12c

–  Verify SNMP trap forwarding for Cisco, KVM and PDU –  Exadata Management Getting Started Guide

 Patching EM for Engineered Systems? –  Check MOS 1323298.1

 Need an executive view? – Build a dashboard! –  MOS 1458346.1 Build a monitoring dashboard with EM 12c


Exadata Dashboard

  Script provided for creating dashboard to monitor performance and usage metrics

–  Database Machine System –  Database Machine components

How do you create the dashboard ?

1.  Execute job “Dashboard job” in Enterprise Manager 12c PS1

2.  Go to the following directory:

/opt/Oracle.SupportTools/emkit/exadata/dashboard/ perl DashbaordOperations.pl


Engineered Systems Health Check Plugin Execution Model

Execute Exachk (2.1.3 and above)

OEM 12c Agent

Exadata Health Checks Plugin

Metric Evaluation

Execution Output XML XML XML

OMS

Console

•  Exadata Healthchecks plug-in consumes the exachk output •  Evaluates the output against pre-defined health check templates •  Generates relevant alerts


HealthCheck plugin – Automatic Alerts Exadata health-check plug-in is designed to raise alerts for the following checks automatically :

•  Database Server •  Disk Cache Policy. •  Disk controllers using write back cache. •  RAID Controller Battery Condition •  RAID Controller Battery Temperature (database

server). •  Virtual Drive Configuration. •  Physical Drive Configuration.

•  Exachk not running. •  Results and Exception file(s) missing.

Add more alerts via the health-check monitoring templates.


Incident Management

 Monitor and resolve service disruptions quickly and efficiently

  Instead of managing numerous discrete events, manage fewer, meaningful Incidents:

–  By business priority –  Across their lifecycle

 Centralized incident console for incident management


Exadata Management

Database

Storage Server

Infiniband Network

PDU, ILOM, CISCO SWITCH


Exadata Management Integrated View of Hardware and Software

  Hardware view •  Schematic of cells, compute nodes

and switches •  Hardware components alerts

  Software/system view •  Performance, availability, usage by

databases, services, clusters •  Software alerts db, cluster, ASM •  Topology view of DB systems/clusters

  Configuration view •  Version summary of all components


  Storage Cell monitoring and administration support

  Cell Home page and performance pages

  Actions supported: Start/stop Cell, verify connectivity, setup SSH

  Automatic discovery of Cells   Management by Cell Group

  All cells used by a database automatically placed in a group

  Cell Group level administration operations (batch job monitoring)

Storage Cell Monitoring


Storage Cell Performance Page   Can view information at the

Cell Group level or individual Cell level

  View IO Statistics   Ability to view Hard Disks

or Flash Disks   Flash Information

–  Usage

–  Throughput

–  IOPS

  Workload Distribution   Ability to drill down to into

individual metrics/IORM setting pages


Monitoring Performance SLAs or – Metrics and what to do with them…

Thresholds are set -They are starting points only! -Your mileage _will_ vary

Tune systems and establish local baselines


Some Improvements in the plugin Showing the hardware thresholds


  Linked to cell performance data page

  Shows active cells   Set Disk I/O objective   Assign resources to each

database running in your Exadata

IORM Management

Managing Consolidation on Exadata with Enterprise Manager


Target Comparisons

 Really useful in comparing hosts to one another. Especially hosts in the same RAC.

 Very useful for other targets as well


Storage Cell Management

•  Perform Cell Administration tasks –  Execute Cellcli commands on a set of cells or all cells

•  Setup IORM for database targets –  Monitor workloads from multiple databases


Infiniband Network Management

•  Infiniband network and switches as CC targets

-  Automatic discovery •  Network home page and performance page

-  Real time and historical •  Topology view of Network with switch and

port level details

•  Full monitoring -  Alerts (switch generated and OEM

generated) -  Performance metrics -  Configuration metrics – detect and

notify configuration changes/best practice violations


Infiniband Network Management

•  Perform Infiniband Administration tasks -  Enable Port -  Disable Port -  Clear Performance counters -  Clear Error Counters


Monitoring: ILOM, Cisco Switch

Common metrics monitored

•  Power supply failure •  Fan failure •  Temperature out of range

Specific metrics monitored •  Cisco Switch

-  Configuration change tracking and reporting -  Unauthorized SNMP access

•  Keyboard, Video, Mouse (KVM) – for older systems -  Server connected, added/removed, powered on/off


Don’t take our word for it….

 Epsilon –  Real-World Operating Excellence with Oracle Enterprise Manager 12c

[CON11255]

 Bank of America –  Upgrade with Oracle Database 11g’s SQL Plan Management: Best

Practices from Bank of America [CON6938]


See it live: Come see us at the demo booth in Moscone North!

Oracle Website: •  Oracle: Oracle Exadata Database Machine

(www.oracle.com/us/products/database/exadata/overview/index.html)

•  Oracle: Oracle Exadata Database Machine resources (www.oracle.com/us/products/database/exadata/resources/index.html)

My Oracle Support: •  0757552.1: Oracle Exadata Best Practices •  888828.1: Database Machine and Exadata Storage Server 11gR2 Supported Versions •  1070954.1: Database Machine exachk or HealthCheck •  1110675.1: Database Machine Monitoring Best Practices •  1329170.1: Master Reference Note for exachk

More to discover


Oracle Website: •  Certification matrix: Oracle Enterprise Manager

(www.oracle.com/us/products/enterprise-manager)

•  High-Availability: Maximum Availability Architecture (MAA) (www.oracle.com/technetwork/database/features/availability)

•  MAA Manageability Blog: Oracle Enterprise Manager (blogs.oracle.com/EMMAA/)

My Oracle Support: •  412431.1: Grid Control Certification Checker •  421053.1: EMDIAG Master Index •  822485.1: Oracle Recommended Patches - Oracle Enterprise Manager

Demos and screen watches: •  Youtube – Enterprise Manager

(www.youtube.com/playlist?list=PLF45DBCDF595F1D1D&feature=plcp)

•  Youtube - Monitor an Exadata environment (www.youtube.com/watch?v=1tYiuj5vPTw)

Where to find more on EM


Questions

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template 40 Oracle Highly Confidential

Thank You!


Appendix


Appendix A Online References

  Websites –  Cloud Control (oracle.com):

http://www.oracle.com/us/products/enterprise-manager/index.html

  Documentation –  Oracle Learning Library:

http://www.oracle.com/goto/oll http://www.youtube.com/user/OracleLearning

–  12cR3 documentation set: http://docs.oracle.com/cd/E24628_01/index.htm

  Blogs –  Oracle Enterprise Manager:

https://blogs.oracle.com/oem

–  Enterprise Manager Best Practices: https://blogs.oracle.com/EMMAA


Appendix A Enterprise Manager Resources   Oracle.com: http://www.oracle.com/enterprisemanager

–  Enterprise Manager Customer References

  OTN: http://www.oracle.com/technetwork/oem –  Demos on Demand –  Enterprise Manager Forums

  Enterprise Manager Training from Oracle University –  Oracle Enterprise Manager 12c Learning Library –  Oracle Enterprise Manager Cloud Control 12c: Install and Upgrade –  Using Oracle Enterprise Manager Cloud Control 12c Ed 1 –  Oracle Enterprise Manager 12c: Overview Bundle Self-Study –  Oracle Enterprise Manager 12c: Management Bundle Self-Study

  Partners –  List of Enterprise Manager Specialized Partners –  Oracle Enterprise Manager OPN Knowledge Zone –  Oracle Enterprise Manager OPN Specialization –  Oracle Enterprise Manager Application Quality Management OPN Specialization –  IOUG Oracle Enterprise Manager Special Interest Group

  Social Media –  Twitter, Facebook, YouTube, Linkedin, Blog


Appendix B Oracle OpenWorld 2013 - Related Sessions

Session Session Title Day Time Location

CON6287 Managing the Manager: Tips and Tricks for Maintaining Oracle Enterprise Manager 12c Monday 10:45 AM Moscone South - 102

CON9590 Oracle Enterprise Manager 12c: Architecture, Concepts, and Best Practices Monday 1:45 PM Moscone South - 104

CON9580 Best Practices for Maximizing Oracle Enterprise Manager 12c Availability Tuesday 3:45 PM Moscone South - 308

CON11255 Real-World Operating Excellence with Oracle Enterprise Manager 12c Wednesday 10:15 AM Westin - Metropolitan I


Appendix B Enterprise Manager Sessions - Monday


CON3267 Managing Oracle WebLogic Server and Oracle Coherence with Oracle Enterprise Manager 12c Monday 10:45 AM Moscone South - 130

CON6287 Managing the Manager: Tips and Tricks for Maintaining Oracle Enterprise Manager 12c Monday 10:45 AM Moscone South - 102

CON9572 Make the Most of Your Oracle Exalytics and BI Investments with Oracle Enterprise Manager 12c Monday 10:45 AM Moscone South – 131

CON9576 Six Essential Features for Effective Cloud-Based Testing Monday 10:45 AM Moscone South – 103

CON2131 DBA Best Practices for Performance Tuning in a Pluggable World Monday 12:15 PM Moscone South – 308

CON9582 Oracle Exadata Management Deep Dive with Oracle Enterprise Manager 12c Monday 12:15 PM Westin - Metropolitan I

CON9573 Managing the Oracle Identity Management Platform with Oracle Enterprise Manager Monday 1:45 PM Moscone South - 130

CON9590 Oracle Enterprise Manager 12c: Architecture, Concepts, and Best Practices Monday 1:45 PM Moscone South - 104

CON9578 Automatic Workload Repository Soup to Nuts: Fundamentals of Database Performance Analysis Monday 3:15 PM Moscone South - 104

CON9591 Oracle Enterprise Manager 12c Deployment and Tuning Best Practices Monday 4:45 PM Moscone South - 308


Appendix B Enterprise Manager Sessions - Tuesday


CON8788 Maximizing Database Performance with Database Replay Tuesday 10:30 AM Moscone South - 308

CON9585 Zero to Cloud: Real Customers, Real-World Success Stories Tuesday 10:30 AM Moscone South - 305

GEN9592 General Session: Oracle Enterprise Manager 12c - Product Update and Customer Perspectives Tuesday 12:00 PM Moscone South - 103 CON9564 Optimize Oracle WebLogic Performance/Manageability: Oracle Enterprise Manager Cloud Control 12c Tuesday 3:45 PM Moscone South - 130

CON9580 Best Practices for Maximizing Oracle Enterprise Manager 12c Availability Tuesday 3:45 PM Moscone South - 308

CON9583 Harness the Power of Oracle Database 12c with Oracle Enterprise Manager Database as a Service Tuesday 3:45 PM Moscone South - 305

CON3991 Deploying Oracle Enterprise Manager 12c in a Complex Network Without Compromising Security Tuesday 5:00 PM Moscone South - 300

CON9586 DBA’s New Favorite: Lifecycle Management with Oracle Enterprise Manager 12c Tuesday 5:00 PM Moscone South - 306

CON9574 Proactively Monitor Oracle WebLogic User Activity and Solve Issues Before End Users Complain Tuesday 5:15 PM Moscone South - 130


Appendix B Enterprise Manager Sessions - Wednesday


CON11255 Real-World Operating Excellence with Oracle Enterprise Manager 12c Wednesday 10:15 AM Westin - Metropolitan I

CON9480 Simplify and Enhance Siebel Management Technologies with Oracle Enterprise Manager Wednesday 10:15 AM Moscone West – 3001

CON9566 Get Up and Running with Oracle Enterprise Manager for SOA and Oracle Fusion Middleware Wednesday 10:15 AM Moscone South – 130

CON9584 Deploying Applications in a Private Cloud with Oracle Enterprise Manager Wednesday 10:15 AM Moscone South – 308

GEN8792 General Session: Database Management Innovations - Oracle Database 12c Manageability Highlights Wednesday 10:15 AM Moscone South – 103 CON4101 Take Your Oracle WebLogic Applications to the Next Level with Oracle Enterprise Manager 12c Wednesday 11:45 AM Moscone South – 130

CON9581 Optimize Oracle Enterprise Manager Monitoring for Your Global Data Center Wednesday 11:45 AM Moscone South – 308

CON9579 Step-by-Step Cookbook for Identifying and Tuning SQL Problems Wednesday 1:15 PM Moscone South – 103

CON9589 Building a SPARC Cloud with Oracle Enterprise Manager Ops Center 12c Wednesday 1:15 PM Westin - Metropolitan I

CON4666 Oracle Enterprise Manager 12c Database Lifecycle Management Automatic Provisioning and Patching Wednesday 3:30 PM Moscone South – 131

CON8768 DBA Best Practices for Protecting Data Privacy with Oracle’s Data Masking Wednesday 3:30 PM Moscone West – 2024

CON9577 Active Session History Deep Dive: Advanced Performance Analysis Tips Wednesday 3:30 PM Moscone South – 104

CON3255 Being Sure: Confident Consolidations with Oracle Real Application Testing 12c Wednesday 5:00 PM Moscone South - 306

CON6217 Using Oracle Enterprise Manager 12c to Manage Engineered Systems and Oracle Fusion Middleware Wednesday 5:00 PM Moscone South - 130

CON9588 Get Proactive: Best Practices for Upgrading to Oracle Enterprise Manager 12c Wednesday 5:00 PM Moscone South - 308


Appendix B Enterprise Manager Sessions - Thursday


CON9605 Databases “On the Fly”: Unravel the Cloud Potential in Oracle Enterprise Manager 12c Thursday 11:00 AM Moscone South - 308

CON3103 Real Oracle Real Application Testing: What to Expect and Prepare For Thursday 12:30 PM Moscone South - 104

CON6129 Migrating Oracle Enterprise Manager to a New Data Center with Near-Zero Downtime Thursday 12:30 PM Moscone South - 308

CON9575 Keyword-Driven Testing with Oracle Application Testing Suite: Application Testing Reinvented Thursday 12:30 PM Moscone West - 3018

CON5677 Bank of America and Oracle Enterprise Manager 12c: Delivering Productivity and Efficiency Thursday 2:00 PM Moscone South - 308

CON9571 JVM Diagnostics: Java Profiling in Production Environments Thursday 2:00 PM Moscone South - 130

CON9587 Manage Beyond Limits: Oracle Enterprise Manager CLI and Other Extensibility Features Thursday 3:30 PM Moscone South - 308


Appendix B Enterprise Manager DEMOgrounds

Session Title Location Application and Infrastructure Testing Moscone West - W-098 Automatic Application and SQL Tuning Moscone South, Left - SL-001 Automatic Fault Diagnostics Moscone South, Left - SL-005 Automatic Performance Diagnostics Moscone South, Left - SL-006 Complete Data Center Monitoring Moscone South, Left - SL-018 Complete Database Lifecycle Management Moscone South, Left - SL-014 Data Masking and Data Subsetting Moscone South, Left - SL-015 Delivering and Managing Database as a Service Moscone South, Left - SL-017 End-User Diagnostics Moscone South, Right - SR-222 End-to-End Management of Oracle E-Business Suite Moscone West - W-014 Identity Management Monitoring with Oracle Enterprise Manager Moscone South, Right - SR-250 Infrastructure as a Service Moscone South, Center - SC-175 Oracle Applications Management Moscone West - W-062 Oracle Enterprise Manager Cloud Control 12c Overview Moscone South, Left - SL-013 Oracle Real Application Testing Moscone South, Left - SL-002 Oracle WebLogic Server, Oracle Coherence, and Middleware Cloud Management Moscone South, Right - SR-206 SOA Management Moscone South, Right - SR-224 Zero to Cloud: Infrastructure to Testing as a Service Moscone South, Left - SL-016


Appendix B Enterprise Manager Hands-on Labs - Marriott Marquis - Salon 14/15

Session Hands-on Lab Title Day Time

HOL10158 Database as a Service with Oracle Enterprise Manager and Oracle Database 12c Monday 10:45 AM

HOL10209 Oracle Database 12c Lifecycle Management with Oracle Enterprise Manager Monday 12:15 PM

HOL10160 Maximizing Oracle Database 12c Performance with Oracle Enterprise Manager Monday 01:45 PM

HOL10202 Oracle WebLogic and Oracle Fusion Middleware Diagnostics with Oracle Enterprise Manager 12c Monday 03:15 PM

HOL10303 Middleware as a Service with Oracle Enterprise Manager 12c Monday 04:45 PM

HOL11733 Oracle WebLogic Plus Oracle Fusion Middleware Diagnostics with Oracle Enterprise Manager 12c Tuesday 03:45 PM

HOL11305 Oracle Database 12c Lifecycle Management with Oracle Enterprise Manager Wednesday 10:15 AM

HOL11304 Maximizing Oracle Database 12c Performance with Oracle Enterprise Manager Wednesday 11:45 AM

HOL11284 Database as a Service with Oracle Enterprise Manager and Oracle Database 12c Wednesday 01:15 PM

HOL11734 Middleware as a Service using Oracle Enterprise Manager 12c Wednesday 03:30 PM

1 Copyright © 2013, Oracle and/or its affiliates. All ... · Exadata Database Machine Monitoring...

Documents

Transcript of 1 Copyright © 2013, Oracle and/or its affiliates. All ... · Exadata Database Machine Monitoring...