© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1
• Ravi Ammamuthu
• Sarayu Damodaran
• Virendra Galotra Sep 25, 2013
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
Integrated Global Monitoring
Database Provisioning
Database Patching
Q & A
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
IT manages hundreds of databases if not thousands
Global Data Centers
Heterogeneous Systems
Complexity
Lack of Agility
Flat/Shrinking IT budget
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
Integrated Global
Monitoring
Database Provisioning
Database Patching
Database Upgrade
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
Prod
293 *
Standalone
86
RAC
207
Standby
96
Non-Prod
1011 *
Standalone
486
RAC
525
Standby
40 * Total excluding Standby DBs
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
Cloud Control & Targets
Targets Only
Legend
Total targets - ~22000
Targets by category are: • Prod - 488 hosts, 293 DBs • NonProd – 1216 hosts,
1011 DBs • DG – 69 hosts, 96 DBs
Targets on UCS, Sun-OS
DB Repository Size
• 600 GB for PROD1
• 1500 GB for PROD2
112 Metrics
32 Reports & 20 Jobs
Global OEM users
STG &
DEV
PROD2 NonProd
Targets
PROD1 Prod
Targets
DR
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
DB Repos
Passive
OMS
DB Repos
Active
OMS
Agents
Global Site Selector (GSS)
ACE ACE
RTP RCDN
Redo Shipping
S/W Library Replication
Snapmirror-Async
DR Primary
Richardson RTP Amsterdam
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
DB Repos
Active
OMS
Active
OMS
Passive
OMS
DB Repos
DB Repos
Agents
Global Site Selector (GSS)
ACE ACE ACE
RTP Allen RCDN
Redo Shipping Redo Shipping
S/W Library
Snapmirror- Async
MVDC DR
S/W Library
Metrocluster
Richardson RTP Amsterdam
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
• Self Healing Through Corrective Actions
Feature
Restart Database, Listener
• Tablespace Space
Addition
• File System Cleanup
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
• Cluster Database Performance
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
• Historical Information
• Ability to drill down and narrow down
• Graphical Representation
• Comparison of configurations across
environments
• Agile report generation
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
• Segregation of Prod and Non-Prod Targets
• Apply Standard Templates Periodically
• Adopt Metric Extensions
• Avoid Storing Job Outputs in Repository
• Always be on Latest Patches
• Eliminate Customization as much as Possible
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
• OEM 10.2.0.5
• Centralized Monitoring of Prod and Non-Prod DBs
• Data Masking
• User Defined Metrics (UDM)
• OEM Infrastructure Migration to new Data Center
• 4 OEM Cloud Control LC Envs
• Enterprise wide Patching of DBs using OEM
• OEM 12.1.0.1 Upgrade
• Monitoring ASM DBs
• Self Healing
• Automatic feed of OEM data to Portal
• Provision Grid Infrastructure
• Provision RAC & Standalone DBs
• OEM 12.1.0.2 Upgrade
• Metric Extensions
2012 2009-2011 2013
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
Situation
• IT Vision : Provision Infrastructure Stack in 5 Days
Complication
• Complexity : Virtualization, Cloud, Inter-connected Services
Implication
• Scalability : Enhancement to Custom Tool for each new release
Position
• Identify Tool to scale horizontally & vertically to meet demands - OEM
Action
• Use Out-of-Box Solution with little or no Customizations
Benefits
• Faster Provisioning
• Optimization
• Quality
• Adhere Standards
• Less OPEX
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
Software Deployment
Validate Server, Network, Storage, Network, Storage
Set up Oracle Home from existing Environment
Standalone Database
Validate Server, Network, Storage
Install Oracle Software
Create Standalone Database
Setup/Validate Cisco Standards
RAC Database
Validate Server, Network, Storage
Install Grid Infrastructure
Configure Grid Infrastructure
Install Oracle Software
Create RAC Database
Setup/Validate Cisco Standards
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
• Manual Installation Process scripts, silent mode pre-requisite checks
• Lack of Standardization
• Long and Error Prone Manual Process
• Increased Resource Requirement
• Maintain Up-to-date Releases
Solution:
Enterprise
Manager (OEM) to
Provision
Databases
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
Clone Oracle Home
Clone, Configure
GI
Create Database
Create RAC
Database
Clone Oracle Home
DB
Type
?
Deployment Procedures
A. Cisco Provision Oracle
Database 1. Deploy Database Software
2. Create a New Database
Non-RAC
RAC
A.1 A.2
B.1, B.3 B.2 B.4
B. Cisco Provision Oracle
RAC Database 1. Deploy Grid Infrastructure
2. Deploy Database Software
3. Configure Grid
Infrastructure
4. Create a New Database
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
• Launch Cisco Procedure and provide inputs
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
• Real Time monitoring of step status • Drill down to log levels directly
Deployment procedures
extended to include CISCO
Customizations
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
Prod
293 *
Standalone 86
RAC
207
Standby
96
Non-Prod
1011 *
Standalone
486
RAC
525
Standby
40 * Total excluding Standby DBs
* 60% software only deployments
OEM Provision
222
Standalone DP
• DB 43
• w/o DB 65 *
RAC DP
• DB 48
• w/o DB 69 *
From
Feb’13
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
0
2
4
6
8
10
12
14
16
18
Manual OEM
8
2
3
2
6
0.5
H
o
u
r
s
RAC DB
Create RAC DB
Clone Oracle Home
Install & Configure GI
0
1
2
3
4
5
6
7
Manual OEM
3
2
4
0.5
H
o
u
r
s
Standalone DB
Create non-RAC DB
Clone Oracle Home
65%
Savings
74%
Savings
* Grid and Oracle Home includes patch storage
Multiple Environments are also provisioned at the same time
Environments Provisioned using EM12c = 222 (from Feb 2013)
Avg. No. of Concurrent Envs. Provisioned = 6
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
Value Metrics
Faster Provisioning of DBs
• 4x reduction for RAC cluster
Increased Agility • 75% SLA Improvement
Reduced OPEX • Scaled down resources from 4
to 1 for 222 Envs
Increased Scalability • Multiple clusters in a single go
• No Extra Efforts for Scale-Ins
Increased Quality • Standards Maintained
• Incident Prevention
• Minimal Errors
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
• Apply patch for DBCA in Cisco’s Gold Code Tree
DB Templates not supporting multiple file locations
• Fix manually; Proceed with ignore option
Sudo privileges prevent running Auto-Fix automatically
• Use reference target from same DC as source
Longer time to copy software across WAN
• Plug-in scripts to create database listeners
Database listeners are not registered on RAC DB
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
Standardization using Gold Image
Leverage hooks for any customizations
Use Referenced Components for Remote DBs; saves bandwidth and time
Define Templates; RAC, Non-RAC, ASM, dNFS
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
Enhance Standard Deployments
• Provisioning Profile with Locked inputs to
Deployment Procedures
• Segregation of Designer and Operator roles
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
• Quarterly Patching
• PSU, CPUs, One-off Patches
• 3200 DB Instances
Audit Requirement
• Extended Outages
• Resource Intensive
• Not Scalable Challenges
• Leveraged OEM to Patch DBs Solutions
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30
STEPS Manual Mode EM Mode
Login Login to individual
Boxes
Centralized login
to EM
Schedule Patch N/A
Automated using
Enterprise
Manager
Download Patch
Run Pre-Patch
Shutdown DB
Apply Patch
Startup DB
Post Patch
Health Checks
Manual process by every single
DBA on every Database
Multiple patches can be
scheduled by one DBA or by
multiple DBAs
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
• Patch RAC & Standalone DBs
• Customized DP for Cisco’s processes using hook-ups Pre and Post requisite checks
Application start/stop scripts
• Merge & Apply many patches in a plan
• Rolling Patching
• Rules to notify failures/success
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32
Dire
ct
Indire
ct
1000 DBs
Savings 3 DBAs.
Reduced
Incidents
Reduced
Human Error
Highly
Scalable
Disclaimer: Cisco specific numbers
Downtime (Hrs)
4000
Before OEM
After OEM
$65K
Before OEM
After OEM
Effort (Hrs)
660
4000 Before OEM
After OEM 2000
$385K
Cost
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33
• Cause : Duplicate Cluster Target; unable to override named credential
• Disabled Auto-Discovery of Targets
Jobs Failed to Submit
• Cause : Opatch Bug 17417108
• STF : Group into multiple plans Multiple
Patch Plans
• Working with Oracle Performance Issue in UI
• Cause : Corrupted software library
• Fix : Rebuilt software library
Opatch download
process fails
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34
• Run Patch Analysis before maintenance window
• Use MOS Online option for Patching, helps in direct patch download and conflict resolution
• Use Rolling Patch option to minimize downtime for RAC DBs
• Include multiple patches in a single plan to go in one downtime
• Manage Oracle resources using Central Inventory
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
Ease of patching for 12 Node Clusters
2 DBAs to patch 42 DBs in 6 Hrs.
* Increased Productivity
* Met Auditing timelines
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36
Reduce Risk and Downtime further • Out of Place Patching of Standalone and RAC
Reduce Documentation:
• Better use of Patch Template to reduce
documentation and communication to DBAs
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
Q1FY14
OEM MAA (Level-4) using MetroCluster
Agent Restart
Monitor TimesTen
Auto Space Addition
Q2FY14
Cron Less Monitoring
Auto Agent Install
FS Clean Up
DB Upgrade
DG Build
Monitor Golden Gate
Q3FY14
EM Target Self Management
DB Mender
Monitor FMW
Upgrade OMS
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39
Manager
• Ravi Ammamuthu
Architect
• Sarayu Damodaran
• Kunal Chakrabarty
Technical Team
• S Venky Sadagopan
• Pascal Laporte
• Manickaraja Kumarappan
• Surekha Jampala
• Venkat Kiran Gunturi
• Manikandan Ganesan
• Sairam Kantheti
PM
• Meena Chanda
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40
Thank You