Does Change Management Include Patches?

Joel Howard, RingMaster Software

Northern California OAUG

San Ramon 2004

Agenda

1. Change Management

2. Answer the Question

3. Realities of 11i

4. APM - a Change Mgmt Solution

What does Change Management cover?

• Upgrades – Software

• Upgrades – Hardware

• Document Management

• Configuration Management

• Software Development

• Maintenance

Change Management

Why is Change Mgmt important?

Disparate components share a relationship to one another…

SHARED DEPENDENCIES!!

Changes…Dependencies

• Maintenance Pack >> Apache Pre-reqs

• Apache upgrade >> DB Pre-requisites

• One off patch >> Pre- and Post-requisites

• Apps Upgrade >> DB Upgrade

Does CM include patches? YES !!

• The biggest, most complex release ever

– Release 10.7 40 products

– Release 11.5.5 161 products– Release 11.5.7 179 products– Release 11.5.8 191 products – Release 11.5.9 197 products

ALL OR NOTHING!

Release 11i Dependencies

– Release 11.5.5 > 20,000 bug fixes– Release 11.5.7 > 40,000 bug fixes– Release 11.5.8 > 64,000 bug fixes– Release 11.5.9 > 100,000 bug fixes

Release 11i

What are the two most difficult issues in maintaining software?

1. Patching

2. Not Patching

Not Unique to Oracle

The Impact of Patching

• Problems Resolved• New, more stable technology• Added Functionality• Improved Security• Faster Performance

Who is impacted? EVERYONE.

The Impact of Patching

• Large amounts of new code introduced• Exact Impact unknown…undocumented• 2000-5000 distinct bug fixes in an average family

pack• 2-50 different modules impacted• User unaware of changed processes/attributes• Disabled forms/reports• Extensive testing• Disruption of production system

The impact of NOT patching

Why not patch? Stability… Perceived

From Metalink FAQ:“At a minimum, apply maintenance packs to stay within two

maintenance releases” of the most current release.

Timely support:Call Oracle support for a problem and you’re not patched

current?

PerformanceLatest performance updates as part of a patch.

The impact of NOT patching

Security

Latest security updates: issued in the form of patches

New Functionality

Improvements in functionality and stability

Politics

User complaints, frustration over a lack of fixes, requests for added functionality

Why avoid it?

• Multiple manual steps per patch, per environment

• No detailed impact analysis• No structure to enforce pre-requisites• Inability to flag affected customizations• No mechanism to manage testing,

approvals or workflow

Manual Steps

Download Patch

Transfer Patch

UnZip Patch

Review “Readme”

Issue search

Review Drivers

Parse Actions

Compare Envir’s

Apply to test

Transfer PatchServer LoginUnZip PatchPrepare Envir.

Review Log File

Review Invalid DB Objects

File SystemImpact

Database Impact

CustomizationImpact

Create Summary

Update Spread Sheets

Check Applied Patches

Run Patchsets.sh

Each Tier

Run AdpatchEach Driver

For Each LanguageRun Adadmin

Run Adctrl

Run Datainstall

Find Pre-reqs

Maint Pack Download

Change Testing

Maint Pack Research

Maint Pack Application

Change Reporting

Strategy

1. AUTOMATE the maintenance of your applications

2. Patch Proactively: Stay current on your watch… not Oracle’s

Software issues are typically encountered when you least have time for them– Month end (Mgmt reports due, Payroll filing)– Quarter end (SEC Reporting)– FDA or other certification requirements– Year end (1099s, W-2s, Sarbanes Oxley)

Not patched current = major project when you can least afford it

APMA Change Management Solution

• Identify the change• Control the change• Communicate the change

APMReal-time analysis by APM file system impact (APPL_TOP) database impact

(table,view,sequence,index) list of affected customizations required prerequisite patches is the prereq obsolete? determine “patch impact ratio” summary of bug numbers in patch links to Oracle bug database list environments where patch/bug

already applied Patch Prediction

• Change requested

• Research performed on Metalink

• Patch obtained

• Patch impact analyzed

• Dependency information obtained

• Knowledge disseminated

• Implementation planned

• Plan implemented

• Testing performed

• Patch Approved

• Patch Migrated

Concurrent Documentation

Change Management

APM• Automatic scheduled

download of patch from MetaLink

• Notification of obsolescence• Enforcement of pre-

requisites• Establish pre-reqs• Automate manual tasks

• Change requested

Research performed on Metalink

• Patch obtained

Patch impact analyzed

• Dependency information obtained

• Knowledge disseminated

• Implementation planned

• Plan implemented

• Testing performed

• Patch Approved

• Patch Migrated

Concurrent Documentation

Change Management

Automated processes within APM

• patch download• transfer patch to multiple

servers• schedule single patch or patch

group• merge patches in background• ADADMIN, ADCTRL,

sqlplus/shell scripts• compile / list invalid objects• start/stop Oracle processes• email/pager notification

• Change requested

• Research performed on Metalink

• Patch obtained

• Patch impact analyzed

• Dependency information obtained

• Knowledge disseminated

• Implementation planned

• Plan implemented

• Testing performed

• Patch Approved

• Patch Migrated

• Concurrent Documentation

Change Management

Workflow / Approval Process

PROD

TestPatch TEST1

DEV2

DEV1

TRAIN

DBADBA

No

STOP

YesOk?

Ok?

DBAYes

DBA

DBAYes

Ok?

Ok?Yes

• Change requested

Research performed on Metalink

Patch obtained

Patch impact analyzed

Dependency information obtained

• Knowledge disseminated

• Implementation planned

Plan implemented

• Testing performed

• Patch Approved

Patch Migrated

Concurrent Documentation

APM• Automatic Alerts: emails,

pager, etc.• Central Repository for all

environments for all patching information

• Dozens of useful reports Environment comparisons, patch detail, patch impact

analysis, patches applied by date,

by environment, by application, by person, etc.

find patchset levels, find where patch was

applied, compare your patches

against those released from Oracle

Change Management

Application DBA

Functional Analyst

Patching Information

Information Flows

Environment Comparison

The Impact of APM

Functional Group• Reduced, focused user

testing• Verify testing took place• Faster support from Oracle

Technical Group• Reduced workload through

automation• Reduced risk through

impact analysis• Complete, accessible audit

trail• Faster problem resolution

Organizational Effect• Enforced procedures

• Decreased workload

• Communication, accountability between business groups

• Improved problem resolution

• Reduced costs

Oracle and RingMaster

11.5.9, Oracle Application Manager (OAM)• Improved Reporting• No automation, scheduling, workflow• No pre-requisite checking• No mechanism to manage testing or approvals• No detailed impact analysis• No environment comparison reports

11.5.10, Oracle Application Manager (OAM)• Improved Impact Analysis on file system only• Pre-requisite checking• No automation, scheduling, workflow• No mechanism to manage testing or approvals• No impact analysis for DB changes• No environment comparison reports

• 1995: Founded as a consulting organization with expertise in Oracle Applications

• 1998: Began developing and selling software specifically for Oracle Applications Users

• Offices in Boston, Dallas, Los Angeles, Atlanta, Chicago, St. Paul, and Vermont

• More than 100 customers in the US, Canada, and Europe

RingMaster Software Corporation

Representative Customers