Deep Dive: Snap Clone and Data Refresh Solutions using Oracle … · Powerful Storage Support...

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |

Deep Dive: Snap Clone and Data Refresh Solutions using Oracle Enterprise ManagerCON6982

Martin PeñaSenior Director, Product Management

Subhadeep SenguptaConsulting Product Manager

Janardhana Korapala, Database Admin Consultant, Dell Inc

Presented with


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Agenda

1

2

3

4

Snap Clone Overview

Features and Use Cases

Customer Use Case: Dell - Snap Clone with EM13

Q & A


Hurdles to Rapid Development and Deployment

• Increasing the speed of application development

– Fast moving application/database requirements require systematic approach to dev/test

– System/application patch/upgrades can introduce risk to dev/test

– Custom application/development require deployment in less time

• Efficient and up-to-date copies of mission-critical databases for agile application development

– Multiple clones of production databases/systems often required to support dev/test activities

– Dev/test and cloning of application and databases must be non-disruptive to production database

Innovation is a key requirement for business growth

62% manage more than 5 DB copies, 34% manage more than 20

IOUG Database Storage Survey, 2014


Lengthy and complex database provisioning

Challenges to Development Agility Today

Access DBConfigureCluster & DB

AllocateStorage

Setup OS& Network

Request HW & StorageApprovalRequest DB

QA Engineer Manager Storage AdminDBA Network Admin DBA QA Engineer


Challenges with Cloning Today

• Lack of automation– Scripts are not a viable solution for an agile development process

• Traditional cloning techniques are slow– Existing storage vendor solutions are DBA unfriendly

• Data explosion often outpaces storage capacity– Inhibits ITs ability to provide clones for development and testing

• Archaic processes that require multiple users to share a single clone, or only supports fixed refresh cycles– Different priorities between DBAs and Storage administrators

Impeding your efforts to innovate


Provision Thin ClonesCreate Test Master

• Space Efficient

– Over 90% storage savings

• Rapid Cloning

– ~ 2 minutes to provision a 1 TB database

• Storage Agnostic

– Supports NAS and SAN storage

• Data Refresh

– Refresh from production

• Integrated DB Lifecycle Mgmt.– Masking, subsetting, lineage and association

• Complete On Premise & Cloud Solutions

Enterprise Manager Snap Clone

Test Master

Production DB

Functional CopiesUser can create additional copies for functional testing without consuming spaceMasking &

Sub-setting


Development Agility

Storage savings, rapid provisioning, Self Service

Powerful Storage Support

Supports ALL storage vendors (NAS & SAN)

Data Refresh

Empower the user to make adhoc clones and restores

Database Cloning across Clouds

For Compute Services and Database Cloud Services

Enterprise Manager Snap Clone


Snap Clone SolutionDevelopment Agility


Base Image1TB Data Files

Clone A1TB Data Files

Clone B1TB Data Files

Traditional ClonesFull copy of the original database

3TB for original + clones

Data Reduction via Thin ClonesThin Clones

Only changed blocks in new clone space

1.015TB for original + clones

Base Image1TB Data Files

Clone A5GB Data Files

Clone B10GB Data Files

Pointers to existing blocks

Pointers to existing blocks

New data

New data


Oracle Database as a ServiceSnap Clone Leverages DBaaS Framework and DB Management

IntegrationAPI integration with 3rd party tools ,

change requests, dev workflows

EfficiencyConsolidated, shared platform

maintained with automation

ElasticityAgile scale up, scale down, relocation

of services

AccessibilityAutomated request channel via

self-service

AccountabilityMeter usage, quota, compliance and

governance

Dedicated Database Pluggable Database Database Schema

Database

Data

Infrastructure

Full CloneEmpty Snap clones

OROR

Physical, Virtual, or Engineered Systems(Hybrid Cloud)

On Premise & On Oracle Cloud


Snap Clone SolutionPowerful Storage Support


Data Cloning Technology OptionsLeverage your Existing Investments

Hardware Solutions Software SolutionsDatabase Native

Storage Agnostic

Vendor Specific Vendor Agnostic

ACFS

RMAN Restore

RMAN Duplicate

Data Pump


Snap Clone on Exadata

• Initiated by using Database snapshot integration features

– SNAPSHOT COPY (for pluggable databases)

– CLONEDB (for non-container databases)

Leveraging Sparse Clones

• Fast, space-efficient snapshot copies of database

• Available for Self-Service consumption.

• Integration with pluggable database enables creation of Snap Clone with a “Single Click”

• All Exadata features work on snapshots (smart scans, smart flash cache, resource management …)

10 x Snap Clones created concurrently from 60TB database in 15 minutes.


ASM SPARSE DG

Hierarchy of Snap Clones on ExadataDEV/TEST

P

T1-R/O

DB1 DB2 DB3ASM SPARSE DG

T2-R/O

DB4 DB5

T1 T2

T3-R/O

DB6

DB7

T3

EXADATA - ASM CLUSTER


Snap Clone with Oracle ZSF Storage Appliance

• Oracle ZS Storage Appliance

– Optimized Oracle Database efficiency with HCC

– Optimized RMAN backup and recovery

– Optimized data reuse in development and test

– DBAs manage snapshots and clones directly

Engineered for optimizing database storage


Snap Clone using Solaris File System (ZFS)

• Uses a single stock Solaris 11.1+ image – physical or virtual [and NOT the Sun ZS3 Appliance ]

• Supports any kind storage – NAS or SAN

• For SAN, mount luns as raw disk and format with ZFS filesystem

• Does NOT require the snapshot/clone licenses from the storage vendor, these features are available for free

• Additional features include compression, deduplication, IO caching, etc

• HA has to be handled externally either via Solaris Clusters, or by using HA features of the underlying hypervisor


Snap Clone SolutionAgile Data Refresh


RMAN Backup

Data Guard

Data Cloning begins from…

TEST MASTER

P SChange Config.

(SI/RAC)

Mask,Change Config.(SI/RAC)

SNAP CLONES

PRODUCTION DBS

Test Master Snapshot

Data

Archive

RMAN incremental data backup

RMAN backup of archive logs

SNAP SHOTS


Deployment Scenarios

Continuous or Discrete Replication Standby / Test Master Database

Production Database

Snap Clones using Standby

Private backups (snapshots) for SSA user

Continuous Discrete

Technology Data Guard, Golden Gate RMAN, Snap Mirror, import/export …

Data Refresh Automatic and instantaneous Manual and at scheduled intervals

Masking and Subsetting

Not possible At source (in production), or in place at test master

Replication Types:


Test Master t0 t1 t2 …. tN

Scheduled or Manual Storage Snapshots or RMAN Backups of the test master database, called Profiles

Private backups

Clones

• Clones can be created from any profile• Each user gets a personal read-write database clone

Test master is regularly

refreshed with current data from

production Data Refresh

Agile ‘Data Refresh’Time Travel – Refresh, Rewind across Snapshots


Data Cloning & Refresh flow

• Non Self Service (Admin) flows

• Scheduled Snapshot/RMAN backup creation

• Clone from a library– snapshots / backups / Image copies

• Clone from a snapshot to a point-in-time or SCN

• 1-click Refresh of clones from source

• REST APIs & Emcli verbs

It’s Not Cloudy any more…


Snap Clone SolutionComing soon … on Oracle Cloud


Data Cloning & Refresh for DevOps

Production Masking

Test Master

60,000 refreshes per year supporting DevOps for

2000+ DevelopersSnap

Clo

ne

s

49 clones/month

60X faster with 180TB savings over EMC


Summary


Oracle Snap Clone

• Scalable

– Supports 1 to 1,000s of clones

• Protects your existing Storage Investments

– Choice between hardware and software solution

– Use of trusted technologies like data guard for test master refresh

• Integrated DB Management with Enterprise Manager

– Oracle’s flagship management product for all your database needs

– In sync with DB releases (e.g. support for PDBs on day one)

– Secure and role based access control; used by Fortune 1,000 customers

– Protection from unnecessary point tools; reduce TCO


Customer Overview

Customer Thumbnail

• One of the largest European financial services groups, with group net income of €23 billion 2012.

• 54,000 employees, 76 countries, and 32 million clients worldwide

Key Pains & Challenges

• 70,000+ full DB refreshes/year

• 10-20 cloning issues/day

• Majority refreshes scheduled; 10% on demand

Solution

• Enterprise Manager Snap Clone on NetApp

• Refresh takes couple of minutes

• Cloning from Standby databases; no impact on production

• Using REST APIs; zero manual intervention


Customer Overview

Customer Thumbnail

• Largest mobile phone company in Egypt in terms of active subscribers

• 36 million customer, 22k employees


• Average 5 days to provision UAT Database(s) for Dev Team

• Sprawling Computing Resources

• Limited Space in Data center

Solution

• Storage Reduction on Exadata, ZFS Storage Appliance

• DB provisioning time reduced from 4 Days to <1 hour for 1TB DB

• Integrated Data Masking

• Resource Pooling, Rapid Elasticity, Metering and Chargeback


Customer Overview

Customer Thumbnail

• International container transportation, logistics and terminal companies

• 320 offices in 70 countries

• Fleet of more than 300 ships


• Long provisioning cycle

• Quality concerns due to limited data set

• Not possible to create training environments on demand

Solution

• Delivered comprehensive service catalog• Enable self services for application owners to request new

or modify environments.

• Implement DBaaS Snap Clone.• 90% Storage Savings demonstrated on both EMC and ZFS

storage.

• Data Refresh with Data Guard /Standby DB.


Customer Overview

Customer Thumbnail

• Founded 1856

• 47,180 employees from over 50 countries

• Recognized as one of the world's most important banks; one of Fortune Magazine's most admired companies.


• Dozens of concurrent development projects

• Synchronization of application and DB deployments

• Cloning frequency

• API for automation and integration

Solution

• Full Clones using RM Backup Provisioning for Performance, Stress, UAT testing

• Snap Clones using ZFS SA and Oracle Database Appliances

• Exadata Sparse Disk Snapshots provide Exadata performance on thin provisioned clones

Global Marketing

Snap Clone Solutions Using OEM13c

Janardhana Korapala, Dell

Global Marketing

Agenda

• Overview Oracle Enterprise Manager at Dell

• Enterprise Manager in action at Dell

• New Enterprise Manager Architecture

Oracle Enterprise Manager at Dell

• Business Drivers

• How Snap Clone Works?

• Snap Clone Pilot Architecture

• Snap Clones Interface• Snap Clones (Self-Service)• Proof of Concept Summary & Next Steps……

Snap Clone at Dell

Q & A

Global Marketing

Oracle Enterprise Manager at Dell

Global Marketing

Overview Oracle Enterprise Manager at Dell

7.4KURL Monitoring

29KDatabase Targets

57KApplication Targets

(Middleware/EBS/Siebel/PS)

App/DB Servers Monitored in Production Enterprise Manager

App/DB Servers Monitored in Non-Production Enterprise Manager

2.6KAgents

350Reports

5DownstreamApplication

1.4KUDM

DB/Host

15Jobs /

Automation

1.5KUser

3KGrouping

Global Marketing

Monitored Targets Dell In-house

Applications

Oracle Enterprise

Manager

Baseline Metric

Admin Group

Incident Rule

User Access

Reports

UDM

Notify Method

Targets Discovery

Agent Installation

Compliance Policies

Groups

Targets Properties

OMR

(Database)

NT Authentication

PAC Group

Email Notification

SMTP Server

Incident (Remedy)Ticketing

RMAN BackupDeployed in 2007

DMMR

Database Metadata Repository

DBCM

Database Compliance Mgmt

Deployed in 2010

Deployed in 2010

KPI

Key Performance Indicator

Deployed in 2013

DMP

Database Maintenance Planner

Deployed in 2012

Deployed in 2007

Oracle DB Monitoring

10g, 11g, 12c

Pilot Test 2011

Deployed in 2012

Oracle SOA Monitoring

10g, 11g

Pilot Test 2013

Deployed in 2015

Pilot Test 2013

Deployed in 2016

Oracle EBS Monitoring

11g, 12c

Oracle SIEBEL Monitoring

10g, 11g, 12c, 13c

Pilot Test & Deployed in 2015

URL Monitoring

N/A

Enterprise Manager in action at Dell

Incident (ServiceNow)

JsonIncident

Convert to ServiceNowDec/2016

Automation

EMCLI

Jobs

Automation Service

Improvement

Since 2010-2016

Global Marketing

Support Disaster Recovery

Zero downtime for OEM rolling

patching

New Enterprise Manager ArchitectureFrom 12c Level 3…

DataGuard

Oracle

Management Service

(13c)

Oracle Management

Repository (13c)

Oracle Database 12c

Oracle

Management Agent

13c

Archive: 512GB, FRA: 512GB, Data: 3TB Archive: 512GB, FRA: 512GB, Data: 3TB

Production Application Server Cluster

(U.S., Austin, PC1) Dell 13g M630'sDR Application Server Cluster

(U.S., Austin, PS3) Dell 13g M630's

EM12P3.PrdRAM 128GB

CPU 36 Core/u01 100GB

EM12P1.PrdRAM 128GB


EM12P2.PrdRAM 128GB


EM12P4.PrdRAM 128GB


Primary Cluster DB

(U.S., Austin, PC1) Dell 13g M630's

DR Cluster DB

(U.S., Austin, PS3) Dell 13g M630's

EM12P1.DRRAM 128GB


EM12P2.DRRAM 128GB


EM12P3.DRRAM 128GB


EM12P4.DRRAM 128GB


400GB

Additional OMSRAM 128GB




Primary ServerRAM 128GB




Primary ServerRAM 128GB






Agent Agent Agent Agent Agent Agent

400GBRsync







…to 13c Level 4

Shared Storage

Production Application Server Cluster

(U.S., Austin, PC1)

EM12P3EM12P1 EM12P2 EM12P4

em12p.prd.amer.dell.com (U.S., Austin, PC1)

Additional OMSAdditional OMSPrimary Server Additional OMS

Agent Agent Agent

Global Marketing

Snap Clone at Dell

Global Marketing

Business Drivers

“Data” issues for testing

Data refresh is manual

Existing solution (Optim) unsupportable, expensive

Too many database copies

Data masking via custom scripts – cumbersome

Virtual Database cloning solution

Testing possible with Real production data

Global Marketing

How Snap Clone Works?

Standby / Test

Master Database

Continuous or Discrete

Replication

Source Database

Test master is regularly refreshed

with current data from production

User A

User B

Each user gets a personal read-write

database clone

Scheduled or Manual Storage Snapshots or RMAN Backups of the test master database, called Profiles

T0

Source A

Source B

Source C

Clones can be created

from a library of profiles

T1

Source A

Source B

…

…

… TN

Source A

Source B

Global Marketing

SnapClone Pilot Architecture

Oracle Enterprise Manager

13c

Source Database Source Database Source Database

OEM Agent

OEM Agent

OEM Agent

San Storage Pool

50 TB

50 TB

50 TB

Snap Clone Appliance

DB PoolTest Master

Scheduled or Manual Storage Snapshots or

RMAN Backups

Global Marketing

Snap Clones Interface

• One Test Master

• 4 Single Instance Snap Clones running from storage snapshots of

Test Master DB mounted via NFS

Global Marketing

Snap Clones (Self-Service)One Test Master

• All Datafiles are thin clones of parent files from Test Master

• Five (5) Snap Clones and many more can be created as system resources allow (CPU & Memory)

Snap Clone Analyzer: >95% Storage Savings using Snap Clone vs. Full Clones

Global Marketing

Proof of Concept SummaryDatabase Cloud Capabilities

• Self Service Enablement: Service Portal, 3rd Party Automation, Self Service Performance Monitoring, Service Instance Snap Shots/Roll-back, Resource Collaboration

• User Governance: Role Based Service Catalog and Quota Control, Show-back Value

Private Cloud DBaaS

• PaaS: Platform Virtualization, Automated Resource Allocation, Resource Governance, Resource Optimization, Cloud Resource Monitoring, Service Refresh/Control, Service Consumption Monitoring

Platform as a Service (PaaS)

• RESTful Web Service API: All PaaS Functionality Exposed via Web Service API for Dell Enterprise Automation, Custom Service Creation

PaaS Enterprise Integration

Global Marketing

Next StepsEnterprise Automation – “Push Button” Application Deployment

“Push Button” Application Deployment – Full Function Application Deployment, Dell Internal SaaS

MWaaS – Continued Focus on Platform Virtualization

Orchestration Integration Templates

Application Blueprints – SDDC Support

PaaS IT Service Management


Additional Resources

Related VideosYoutube.com/OracleEnterpriseMgr

Join the Conversation Twitter.com/oracle_em

Oracle.com/newsletters

Facebook.com/oracleenterprisemanagerBlogs.Oracle.com/OEM

Enterprise Manager Page on O.com

Database as a Service Page on OTN

Cloud Administration Guide (Documentation)

MOS Note 1549855.1 : Enterprise Manager Cloud Control Recommended Plug-Ins and Patches for Database as a Service (DBaaS)

http://www.oracle.com/us/products/enterprise-manager/index.html

http://www.oracle.com/technetwork/oem/dbaas-primary-2394629.html

http://docs.oracle.com/cd/E63000_01/EMCLO/cloud_overview_new.htm

Deep Dive: Snap Clone and Data Refresh Solutions using Oracle … · Powerful Storage Support...

Documents

Transcript of Deep Dive: Snap Clone and Data Refresh Solutions using Oracle … · Powerful Storage Support...