Oracle Fleet Management:Patch and Upgrade Your

DatabasesGary HendersonVaithianathan Soundararajan

Contents

Nationwide Database Teams

Insurance and Financial Services Company

Fortune 100 Company

Founded in 1925

43 Billion in Revenues*

221 Billion in Assets*

33,135 Employees*

#53 on Fortune’s "Best Companies to Work For"

#27 in Computer World’s "Best Places to Work in IT."

*Fortune 500

Nationwide

AGENDA• Why Patch?

• Our Configuration

• Patching Alternatives

• Our Constraints

• Our Results

• How We Did It

• Our Timeline of Progress

• Summary

• Questions

SECURITY..SECURITY..SECURITYCenter for Internet Security (CIS)

1 Oracle Database Installation and Patching Requirements One of the best ways to ensure secure Oracle security is to implement Critical Patch Updates (CPUs) as they come out, along with any applicable OS patches that will not interfere with system operations.

1.1 Ensure the Appropriate Version/Patches for Oracle Software Is Installed Profile Applicability:

• Level 1 - ...The Oracle installation version and patches should be the most recent that are compatible with the organization's operational needs.

Rationale: Using the most recent Oracle database software, along with all applicable patches can help limit the possibilities for vulnerabilities in the software, the installation version and/or patches applied during setup should be established according to the needs of the organization. Ensure you are using a release that is covered by a

level of support that includes the generation of Critical Patch Updates.

• https://workbench.cisecurity.org/files/2121/download/2346

Oracle’s Annual Release Roadmap Doc ID 742060.1

2014 Clustering Technologies vs DB Versions

RAC - MultiNode17%

RAC 1 Node

17%

Veritas32%

Non-Clustered34%

9.2.0.71%

10.2.0.512%

11.1.0.719%

11.2.0.230%

11.2.0.336%

11.2.0.41%

12.1.0.21%

2016 Clustering Technologies vs DB Versions

RAC - MultiNode16%

RAC 1 Node

60%

Veritas10%

Non-Clustered14%

10.2.0.51%

11.1.0.76%

11.2.0.21%

11.2.0.32%

11.2.0.439%

12.1.0.251%

2018 Clustering Technologies vs DB Versions

RAC - MultiNode12%

RAC 1 Node79%

Non-Clustered9% 10.2.0.5

0% 11.2.0.49%

12.1.0.290%

18.3.0.01%

Opa

tch • In place

• Challenging error recovery

• Longer Outage window

• All databases in the Home have to be patched together

• Rollback challenging

• Requires less storage than out of place options

• *no extra licensing

Mul

ti-te

nanc

y • Out of Place• Unplug from

old and plug into updated/patched

• May require double the memory resources

• *Multi-Tenant license required

Rapi

d Ho

me

Prov

ision

ing • Out of Place

• Gold Image Homes/ Standardization

• Minimal Outage window

• *LifeCycleLicense Required?

OEM

Fle

et M

aint

enan

ce • Out of Place• Version 1:

Switch• Version 2: Db

software maintenance

• Gold Image Homes / Standardization

• Minimal Outage window

• Utilizes OEM’s deployment job system

• Scalable• *LifeCycle

License Required

Clou

d DB

aaS • Cloud

Provider applies Patches

Patching Alternatives

Quarterly Patching WindowsSwitch/Update

JANUARY FEBRUARY MARCH m t w t f s s 1 2 3 4 5 6 7

8 9 10 11 12 13 14

15 16 17 18 19 20 21

22 23 24 25 26 27 28

29 30 31

m t w t f s s 1 2 3 4

5 6 7 8 9 10 11

12 13 14 15 16 17 18

19 20 21 22 23 24 25

26 27 28

m t w t f s s 1 2 3 4

5 6 7 8 9 10 11

12 13 14 15 16 17 18

19 20 21 22 23 24 25

26 27 28 29 30 31

Each Quarter patch to previous quarter’s PSU

Development patched during Wednesday

window

Using a team of three:DBA – Db patcher

DBA – Clusterware patcher/TesterOEM - Administrator

Production patched during IRW window

PATCHING RESULTS

Windows Success% #DbsTotExecTime

(Hrs)WindowTime

(Hrs)

2016:Devl 31 100 693 98.15 38.35

2017:Devl 44 99 1080 160.67 52.77

2018:Devl 42 100 1575 183.56 48.14

2016:Production 12 98 340 44.36 18.34

2017:Production 12 100 706 79.38 20.85

2018:Production 10 99 1000 105.35 18.2411

OEM Fleet Maintenance (EMCLI)

Version 1: Switch• SWITCH_DATABASE

• SWITCH_GI

Version 2: Db_Software_Maintenance (Update)• createSoftwareImage

• subscribeTarget

• checkApplicability

• performOperation• DEPLOY_GI_SOFTWARE

• UPDATE_GI

• DEPLOY_DB_SOFTWARE

• UPDATE_DB

• DEPLOY_RAC_SOFTWARE

• UPDATE_RACDB

• DEPLOY_CDB

• ATTACH_CDB

• ROLLBACK_DB …RACDB ..GI

• CLEANUP_SOFTWARE

Fleet Reference: https://bit.ly/2S4FZsz

Software Standardization Advisor

DB Software MaintenanceCreate Gold

Image/VersioncreateSoftwareImage

Prior Quarter

Associate Target to Corresponding Image

subscribeTargetOne time (two weeks prior)

Deploy new Oracle HomeDEPLOY_RAC_SOFTWARE

Two Weeks prior

Patch Database moving it from the old to the new patched home

UPDATE_RACDB

When the old home is no longer used ..

Remove it.CLEANUP_SOFTWARE

Gold Image Version – Lineage/Swim Lanes12.1 DBOne-Off RAC

Ver 1 180417

12.1 DBOne-Off Stand-Alone

Ver 1 180417

12.1 DB Standard RAC

Ver 1 180417

11.2 DB Standard RAC

Ver 1 180417

12.1 Grid Standard RAC

Ver 1 180417

12.1 DB Standard Stand-Alone

Ver 1 180417

11.2 DB StandardStand-Alone

Ver 1 180417

12.1 Restart Standard Stand-Alone

Ver 1 180417

Ver 2 180717

Ver 2 180717

Ver 2 180717

Ver 2 180717

Ver 2 180717

Ver 2 180717

Ver 2 180717

Ver 2 180717

Ver 3 1810??

Ver 3 1810??

Ver 3 1810??

Ver 3 1810??

Ver 3 1810??

Ver 3 1810??

Ver 3 1810??

Ver 3 1810??

Clus_server1

Clus_server2

Our Automation Leveraging Db Software Maintenance

PROGRESS – CRAWL, WALK, RUN … FLY

2014 Gold Image Provision of Database Homes

2015 RAC Clusterware Upgrade from 11.2 to 12.1 (60)Some Database upgrades 11.2.0.2/3 to 11.2.0.4

2016 Q1 & Q2 Mandate ALL DB homes OEM provisionedMandate ALL RAC Clusters built with OEMContinue upgrades 11.2.0.4 & 12.1.0.2

Q3 & Q4 Patch 1033 databases using Switch Database

2017 1786 databases patched using Switch Database

2018 2575 databases patched (so far)All databases patched in 2nd quarter3rd Quarter moved from Switch to DB_Software_Maintenance 18

• SUMMARY:

• RECOGNITION:

• Missing Nationwide team member John Norman

• Oracle OEM Product Team: Harini Srinivasan

Bharat Paliwal

Martin Pena

Saurabh B Jain

• Oracle OEM Dev Team: Harmeet Kaur

Paras Narang

Karthika Thirumalasamy

• Oracle Senior Enterprise Account Executive Stephan Saade

• Oracle Key Account Director for Nationwide Joe Johnston

• OTHER SESSIONS:

• Hands On Lab (HOL6350) Next-Generation Database Patching - Wedneday 11:15

• OEM kiosks at the Demo Grounds (MGMT-WU2)

• Oracle Fleet Reference Manual: https://bit.ly/2S4FZsz

• QUESTIONS ??? 19