Data as a Service

Accelerating Application Projects by 50% with Data as a Service

kylehailey.com [email protected] @virtdata

http://kylehailey.com

mailto:[email protected]

1990 Oracle– 90 support

– 92 Ported v6

– 93 France

– 95 Benchmarking

– 98 ST Real World Performance

2000 Dot.Com

2001 Quest

2002 Oracle OEM 10g

Success!

First successful OEM design

Who is Kyle Hailey

1990 Oracle– 90 support

– 92 Ported v6

– 93 France

– 95 Benchmarking

– 98 ST Real World Performance

2000 Dot.Com

2001 Quest

2002 Oracle OEM 10g

2005 Embarcadero– DB Optimizer

Who is Kyle Hailey

Who is Kyle Hailey

• 1990 Oracle 90 support 92 Ported v6 93 France 95 Benchmarking 98 ST Real World Performance

• 2000 Dot.Com• 2001 Quest • 2002 Oracle OEM 10g• 2005 Embarcadero

DB Optimizer

• 2010 Delphix

When not being a Geek- Have a little 6 year old boy

& new babywho take up all my time

• Data Constraint• Solution• Use Cases

In this presentation :

The Phoenix Project

What is the constraint

in IT ?

IT is the factory floor of this century

7

AutomationJenkins Team City Travis

Data

Virtualizatio

n

Configurati

on Chef Puppet Ansible

Compute

Virtualizatio

n Vmware OpenStack Docker

?

Put your energy into the constraint

Top 5 constraints in IT

1. Dev environments setup2. QA setup3. Code Architecture4. Development5. Product management

- Gene Kim Surveyed • 14000 companies• 100s of CIOs

Flow of Features

9

1

DevelopmentEnvironments

2

QA & Testing Environments

Product ManagementFeatures

2 2

Code Architecture

3Code Speed

4

5

Data

Data is the constraint

60% Projects Over Schedule

85% delayed waiting for data

Data is the Constraint

CIO Magazine Survey:

only getting worseGartner: Data Doomsday, by 2017 1/3rd IT in crisis

Copy Databases:

Application Development Problems

12

• Not enough resources• Contention on shared environments • Lack of enough environments• Late stage bug discovery

• Faulty Data leading to bugs• Subsets • Synthetic data• Old data

• Slow environment builds• Delays• Developers waiting• QA slow and expensive

Physical Data: shared bottlenecks

Frustration Waiting

Physical Data: bugs because of old data

Old Unrepresentative Data

Physical Data: subsets hard & lead to bugs

False NegativesFalse PositivesBugs in Production

16

Physical Data: Production Wall

Physical Data: Developers wait

Physical Data: limited environments

Physical Data : late stage bugs

010203040506070

1 2 3 4 5 6 7Delay in Fixing the bug

Cost ToCorrect

Dev QA UAT

# of bugsfound

Software Engineering Economics – Barry Boehm (1981)

Virtual Data : Expensive Refresh

20

20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST

8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs 8 Hrs

Copies

21

• Oracle: 8-12 copies

• Fortune 2000 : 1000s of DBs

• Staggering Storage amounts

• Hardware– storage, systems, network, – rack space, power cooling

• People – 1000s hours per year just for DBAs – DBAs– SYS Admin– Storage Admin– Backup Admin – Network Admin

• $10s Millions for data center modernizations

Copies require People & Time

Typical Architecture

Production

Instance

File system

Database


Production

Instance

Backup

File system

Database

File system

Database


Production

Instance

Reporting Backup

File system

Database

Instance

File system

Database

File system

Database


Production

Instance

File system

Database

Instance

File system

Database

File system

Database

File system

Database

InstanceInstance

Instance

File system

Database

File system

Database

Dev, QA, UAT Reporting Backup

Triple Tax


Production

Instance

File system

Database

Instance

File system

Database

File system

Database

File system

Database

InstanceInstance

Instance

File system

Database

File system

Database

companies unaware

companies unaware

Developer or AnalystBoss, Storage Admin, DBA

Metrics

– Time – Old Data – Storage

Other – Analysts – Audits – Data Center Modernization

companies unaware

"we say no, no, no until we can't say no anymore" response when IT asked for copies of prod DB

• Data Constraint• Solution• Use Cases

In this presentation :

Development UATQA

99% of blocks are identical

Solution

Development QA UAT

Thin Clone

• EMC Symmetrix• Netapp & EMC VNX• Solaris ZFS

Technology Core : file system snapshots

Also check out new SSD storage such as: Pure Storage, EMC XtremIO

Fuel not equal car

Challenges

1. Technical2. Bureaucracy

1. Bureaucracy

Developer Asks for DB Get Access

Manager approves

DBA Request system

Setup DB

System Admin

Requeststorage

Setupmachine

Storage Admin

Allocate storage (take snapshot)

Why are hand offs so expensive?

1hour1 day

9 days

1. Bureaucracy

2. Technical Challenge

Database Luns

Production FilerTarget A

Target B

Target C

snapshotclones

InstanceInstance

InstanceInstance

InstanceInstance

InstanceInstance

Instance

Source

Database LUNs

snapshot

clonesProduction Filer

Development Filer

2. Technical Challenge

Instance

Target A

Target B

Target C

InstanceInstance

InstanceInstance

InstanceInstance

Instance

Data Flow Optimization

42

43© 2015 Delphix. All Rights Reserved. Private & Confidential.

Install Delphix on Intel hardware

• .

• .

• .

• .

• .

• Data

• .

• Binaries

• Application Stacks

• EBS

• SAP

• Flat files


Allocate Any Storage to Delphix

Allocate Storage

Any typePure Storage + Delphix

Better Performance for

1/10 the cost


One time backup of source database

Data is

compressed

typically 1/3

size

Production

3 TB 1 TB


Incremental forever change collection

Two week time flow

Production


Clones: Fast, Free, Full

Production

Two week time flow

NFS

Three Physical CopiesThree Virtual Copies

Data Virtualization Appliance

Before Virtual Data

Production Dev, QA, UAT

Instance

Reporting Backup

File system

Database

Instance

File system

Database

File system

Database

File system

Database

InstanceInstance

Instance

File system

Database

File system

Database

“triple data

tax”

With Virtual DataProduction

Instance

Dev & QA

Instance

Reporting

Instance

Backup

Instance Instance InstanceInstanceInstance

Instance

File system

Database

Instance

Instance

• Problem in the Industry• Solution• Use Cases

1. Development & QA2. Production Support3. Business

Use Cases

Development: Virtual Data

Development

Virtual Data: Easy

Source

Clone 1

Clone 2

Clone 3

Virtual Data: Parallelize

gif by Steve Karam

Virtual Data: Full size

Virtual Data: Self Service

Environments: almost unlimited

QA : Virtual Data• Fast • Parallel• A/B testing

Physical Data : find bugs fast

Dev QA UAT

# of bugsfound

010203040506070

1 2 3 4 5 6 7Delay in Fixing the bug

Cost ToCorrect

Dev

QA

Instance

Prod

DVA

• Fast

• Full Size

• Run Parallel QA

• Lots of environments for projects like ERP

Upgrades

Virtual Data : Parallel

Production Time Flow

Virtual Data: Rewind

DVAInstance

QA

Prod


Virtual Data : Fast Refresh

63

20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST

• Fast

• Full

• Fresh

• Efficient

8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs 8 Hrs

20 MIN

TEST

Virtual Data: A/B

DVAInstance

Instance

Instance

Index 1

Index 2


Virtual Data: Version Control

4/30/2015 65

Dev

QA

2.1

Dev

QA

2.2

2.1 2.2

Instance

Prod

DVA Production Time Flow

Physical Data: Copies increase the surface area of risk !

Production

Virtual Data: reduce surface area further protected by masking

ProductionNFS

1. Development and QA2. Production Support3. Business

Use Cases

• Recovery• Forensics• Migration

Production Support

9TB database 1TB change day : 30 days

0

10

20

30

40

50

60

70w

eek

1

wee

k 2

wee

k 3

wee

k 4

original

Oracle

Delphix

StorageRequired(TB)

Days

RPO & RTO

71

• RPO

– Any time in last 30 days

– Down to the second

• RTO

– Minutes

– Push button

0

5

10

15

wee

k 1

wee

k 2

wee

k 3

wee

k 4

original

Delphix

Virtual Data: Recovery

Instance

Instance

Recover VDB

Drop

Source

DVA Production Time Flow

Virtual Data: Forensics

Instance

Development

DVA

Source


Virtual Data: Development recovery

Instance

Development

DVA

Source

Development

Prod & VDB Time Flow

Virtual Data: Migration

Cloud Migration and Replication

76

1. Development and QA2. Production Support3. Business Continuity

Use Cases

Business Intelligence

• Audits• ETL• Temporal• Federated data• Consolidated data


Virtual Data: Audit

4/30/2015 79

Instance

Prod

DVA

Live Archive

Live Archive data for years• Archive EBS R11 before upgrade to R12• Sarbanes-Oxley• Dodd-Frank• Financial Stress tests

Business Intelligence: ETL and Refresh Windows

1pm 10pm 8amnoon

Business Intelligence: batch taking too long

1pm 10pm 8amnoon

2011

2012

2013

2014

2015

2012

2013

2014

2015

1pm 10pm 8amnoon

10pm 8am noon 9pm

6am 8am 10pm

Business Intelligence: ETL and DW Refreshes

Instance

Prod

Instance

DW & BI

• Collect only Changes• Refresh in minutes

Instance

Prod

BI and DW

ETL24x7

DVA

Virtual Data: Fast Refreshes

Time Flow

Modernization: Federated

Instance

Instance

Source1

Source2

Production Time Flow 1

Production Time Flow 2

Physical Data: Federated

“I looked like a hero”Tony Young, CIO Informatica

Virtual Data: Federated

Virtual Data: Temporal Data

Virtual Data: Confidence testing

1. Development & QA– Dev throughput increase by 2x

2. Production Support– 30 days in size of source

3. Business Continuity– 24x7 ETL & federated cloning

Use Case Summary

© 2015 Delphix. All Rights Reserved. Private & Confidential. P91.© 2015 Delphix. All Rights Reserved. Private & Confidential. P91.

Shift Left

ROI

Time

Reduced

OpEx, CapEx

B

• Insurance product “about 50 days ... to about 23 days”

– Presbyterian Health

• “Can't imagine working without it”

– State of California

• Projects “12 months to 6 months.”

– New York Life

• Projects “12 months to 6 months.”– New York Life

• Insurance product “about 50 days ... to about 23 days”– Presbyterian Health

• “Can't imagine working without it”– State of California

Virtual Data Quotes

• Problem: Data constraint • Solution: Data Virtualization

Summary

Innovation• Transformative• Automation• Self Service

Thank you!

• Kyle Hailey - Technical Evangelist (Oracle Ace, Oaktable)

– [email protected]

– kylehailey.com

– slideshare.net/khailey

– @virtdata

One other thing: Performance

• Performance

95

Oracle 12c

80GB buffer cache ?

5000

Tnxs

/ m

inLa

ten

cy

300 ms

1 5 10 20 30 60 100 200

with

1 5 10 20 30 60 100 200Users

200GBCache

5000

Tnxs

/ m

inLa

ten

cy

300 ms

1 5 10 20 30 60 100 200Users

with

1 5 10 20 30 60 100 200

$1,000,000 1TB cache on SAN

$6,000200GB shared cache on Delphix

Five 200GB database copies are cached with :

Goal : virtualize, govern, deliver

103

• Masking: Masking• Security: Chain of custody• Self Service: Logins• Developer: Versioning , branching• Audit: Live Archive

Snap Shots

Thin Cloning

Copy Data Management

Data as a Service31 2

2

32

EMC, Netapp, ZFS

Oracle Snap Clone, Clone DB

ActifioDatical,Oracle DBaaS

Delphix

• EMC Symmetrix– 16 snapshots – Write performance impact– No snapshots of snapshots

• Netapp & EMC VNX– 255 snapshots

• ZFS– Compression– Unlimited snapshots– Snapshots of Snapshots

• DxFS– Compression– Unlimited snapshots– Snapshots of Snapshots– Shared cache in memory

Technology Core : file system snapshots

Also check out new SSD storage such as: Pure Storage, EMC XtremIO

ActifioProduction

InstanceInstanceInstance

Actifio

InstanceInstance Instance

TargetActifio

Instance

Target

Oracle Snap Clone

ZFSSAor

NetApp

Instance

TargetEM 12c

Instance

Target

Production


Oracle Snap CloneProduction


Data Guard


ZFSSAor

NetApp

Instance

TargetEM 12c

Instance

Target

Oracle Snap CloneProduction


Solaris

ZFS

Instance

TargetData Guard

Instance

Instance

Target

Any storage

EM 12c

Incremental forever collect changesProduction


Time Flow

ChangesInstance

NFS

Target

Instance

Target

Data virtualization

• Fast becoming the new norm

• Used by Over 100 of Fortune 500

• Enables DevOps

111

AutomationJenkins Team City Travis

Data

Virtualizatio

n Delphix Open ZFS Flocker

Configurati

on

Managemen

tChef Puppet Ansible

Compute

Virtualizatio

n VMware Vagrant Docker AWS OpenStack

112

Jenkins, Team City, Travis

Open Stack, Vagrant, Docker

Chef, Puppet, Ansible

Delphix

DevOps : Automation + Culture

Snapshot 2 - apply


a b c d e f g h ib' c'

Snapshot 1 – drop


b' c'a d e f g h i

Creating a VDB


b' c'a d e f g h i

My vDB(filesystem)

Your vDB(filesystem)

b' c'a d e f g h i

Modify a vDB


b' c'a d e f g h i

My vDB(filesystem)

Your vDB(filesystem)

i’b' c'a d e f g h ib' c'a d e f g h i

What is DevOps ?

119

• Not Tools (required)• Not a Process (not standardized yet)• Not Culture (critical)

DevOps is a Goal

DevOps Goal :

120

Fast flow of features from development

to IT operations to the customers

- Gene Kim

The Problem: Flow of Features

121

Features Customer

The Goal : eliminate the constraint

Improvementnot made at the constraintis an illusion

Theory of Constraints

Factory floor

Factory floorconstraint


Tuning here

Stock piling


Tuning here

Starvation


Goal: • find constraint • optimize it

A database refresh in 15 minutes?

That is mind blowing!

Delphix nailed it for us. - Matt Lawrence , Sr Director Wind River (Intel)

Took 3 weeks to build a dev env

now with Delphix takes less than a day

the db part is less than 15 minutes- Marty Boos , Stubhub (Ebay)

Delphix goes beyond storage

Delphix so much more than

We thought it was-Michael Brow State of Colorado

Worth investing on this product

the technology is strong and

value prop is high- Deloitte

I'm convinced about Delphix's

technology Delphix can really

increase the quality of Dev / QA - Oaktable Member

Delphix allows us to move fast and setup database copies in seconds

Delphix is powerful and allowed us to scale from 2 projects to 11

We need Delphix to scale our agile environment

– Tim Campos, CIO, Facebook

Data as a Service

Business

Transcript of Data as a Service