Post on 05-Jan-2016
Copy Data Management for Analytics
Catalogic ECXManage, Orchestrate and Analyze
Copy Data for Analytics
Agenda
2
Copy Data Analytics Challenges
Copy Data Management Value Proposition
Catalogic Software’s Solution Overview
Product Demonstration – http://data.catalogicsoftware.com/ibm
IBMinfo@catalogicsoftware.com
3
• Four use cases• Each requiring its own solution• All with common, overlapping
storage services• Separate and complex to
deploy• Costly to implement & manage• Results in data silos• Creates software, hardware and
data sprawl
Multiple Solutions – Multiple ‘Common’ Services
BACKUP/DR
Dev/Test - DevOps
TEST/DEV, DEVOPS
ANALYTICS
DATA SERVICES
COPY OPTIMIZE
MOVE STORE
DATA SERVICES
COPY OPTIMIZE
MOVE STORE
DATA SERVICES
COPY OPTIMIZE
MOVE STORE
DATA SERVICES
COPY OPTIMIZE
MOVE STORE
4
Big Data workload(Analytics)
Resilient workload(Mirror)
Non productionworkload(Test/Dev or DevOps)
Compliance workload(Archive)
Resilient workload(Disk Backup)
Recovery workload(Tape Backup)
PrimaryData
Today’s IT Challenge: Drowning in a Deluge of Copy DataResult: Increased cost and complexity and no additional business value
Source: IDC, The Copy Data Problem: An Order of Magnitude Analysis, doc #239875
COPY DATA GROWTH
Stor
age
Gro
wth
Time
Primary Data
~35%YoY
23
6
1
1
?
Copy Data will be a $51B problem by 2018• Consumes as much as 60% of disk capacity• Drives 65% of Storage Software and 85% of the
Storage Hardware spending• Almost all copies sit idle
Copy Data Mgmt Gap
Geometric CopyData Growth
Appropriate Data Growth
Linear Data Growth
50+Copies
231
1
Analytics Process – Before
• Challenge with this process
– ETL data is ‘stale’
– Difficult, slow, long process – each group needs
separate ETL process for data – lots of scripting
– Resultant analysis is not useful, stale, not real-
time
– No competitive advantage
5
Data set made available via data recovery process
(never refreshed)
Marketing team performs analytics on
some ETL data set
Sales team performs analytics on some ETL
data set
M
S
Business Decision Made
F
Finance team performs analytics on some ETL
data set
LEV
ERA
GE
Catalogic’s Copy Data Management Platform
6
Your Infrastructure
VM VM VMVM VM VM
Virtual snapshot and replication capabilities
Catalogic ECX
Copy DataManagement
Platform
Catalog, Manage, Orchestrate and Analyze
Copy Data across your Enterprise and Cloud
Downloadable VM
Agentless
Use Cases
• Automated DR, Enhanced Test/Dev, DevOps Automation, Analytics and Enhanced Recovery
• Orchestration and Automation of application consistent Snapshots and VMware VADP Copy Images
Copy Data Orchestration and Automation
• Full actionable catalog for Snapshot / File analytics /Protection Compliance
• Search, Report and Analyze
Copy Data Catalog and Analytics
IBM(GA ECX 2.2): Storwize, SVC, v9000, VersaStack,
Flash Copy Manager & SoftLayer
AND/OR • Efficient management of Copy Data through full lifecycle (Copy Creation, Copy Use, Compliance and EOL), across heterogeneous storage infrastructure
• Eliminate the need for scripting based upon unique APIs per storage platform
Copy Data Lifecycle Management
• Self-service provisioning as well as Self-Service automated DR, Test/Dev, DevOps and analytics
• App teams gain speed and IT keeps control and gains SLA management
Self-Service for App teams or Admins
7
How it Works
VM VM VM
VM VM VM
VM VM VM
VM VM VM
The Copy Data Reality
8
Current Environment• Highly efficient and resilient• Replicated• Primary and secondary
Snapshots
Separate Environments for:• Test/Dev• Analytics or Data warehouse• Operational Recovery• DR• Archive
…Over budget and …“data starved”
PRIMARY SITE
Storage Array
VMware
VM VM VM
VM VM VM
SECONDARY SITE
Storage Array
VMware
VM VM VM
VM VM VM
VM VM VM
VM VM VM
VM VM VM
VM VM VM
Array Replication
Copy Data Management
9
PRIMARY SITE
Storage APIs
SECONDARY SITE
Storage APIs
VMware API
VM VM VM
VM VM VM
VMware API
VM VM VM
VM VM VM
Array Replication
CatalogicECX
Copy Data Management“Catalog” provides:
• Catalog all snaps, replicas, and VMware objects
• Detailed visibility and context of your current environment
• Copy Lineage and location
• Compliance reporting
Copy Data Management
10
PRIMARY SITE
Storage Array
SECONDARY SITE
Storage Array
VM VM VM
VM1 VM2 VM3
VM VM VM
VM VM VM
Copy Data Managementactionable catalog provides:
• Catalog all snaps, replicas and VMware objects
• Detailed visibility and context of your current environment
• Copy Lineage and location
• VMware / Storage correlation
• Compliance / Audit reporting
VMware VMware
Array Replication
Production Data
Full Copy
Snapshot
CatalogicECX
Copy Data Management
11
PRIMARY SITE
Storage Array
SECONDARY SITE
Storage Array
VM VM VM
VM1 VM2 VM3
VM VM VM
VM VM VM
Automation & orchestration enables powerful use cases:• Enhanced Recovery, Automated
Daily DR, DevOps, Analytics
• Workflows run scheduled, ad hoc or via REST API
• Refresh Data as often as you snapshot
VMware VMware
Array Replication
Production Data
Full Copy
Snapshot
CatalogicECX
Copy Data Lifecycle Management• Simple process to consistently
configure and deploy snapshots/Replication
• Replaces complex custom scripting
• Full visibility, analytics and SLA reporting
VMware VMware API
Copy Data Management
12
PRIMARY SITE
Storage Array
SECONDARY SITE
Storage Array
VM VM VM
VM1 VM2 VM2
Automated DR / DevOps / Analytics:
• Workflows leverage actionable catalog
• Bring up applications in a fenced network
• Bring up applications in order leveraging existing application consistent snapshots
• Test DR every day
• Enable DevOps agility; faster time to market
• Fresh data for analytics; greater accuracy
VM VM VM
VM VM VM VM1’ VM2’ VM3’
Fenced VM Environment
VM VM VM
VM VM VM
VM1’ VM2’ VM3’
FencedVM Environment
Array Replication
Production Data
Full Copy
Snapshot
CatalogicECX
Copy Data Management
13
PRIMARY SITE
Storage Array
SECONDARY SITE
Storage Array
VM VM VM
VM VM VM
Environments can be:
• Cleaned up
• Promoted to production
• Moved to permanent storagefor longer use
VM VM VM
VM VM VMVM1 VM2 VM3 VM1’ VM2’ VM3’
Fenced VM Environment
VM VM VM
VM VM VM
• Cleaned up
VMware VMware
Array Replication
Production Data
Full Copy
Snapshot
CatalogicECX
Copy Data Management
14
PRIMARY SITE
Storage Array
SECONDARY SITE
Storage Array
VM VM VM
VM VM VM
Environments can be:
• Cleaned up
• Promoted to production
• Moved to permanent storagefor longer use
VM VM VM
VM VM VMVM1 VM2 VM3 VM1’ VM2’ VM3’
Fenced VM Environment
VM VM VM
VM VM VM
• Promoted to production
VMware VMware
Array Replication
Production Data
Full Copy
Snapshot
CatalogicECX
VMware VMware
Copy Data Management
15
PRIMARY SITE
Storage array
SECONDARY SITE
Storage Array
VM VM VM
VM VM VM
Environments can be:
• Cleaned up
• Promoted to production
• Moved to permanent storagefor longer use
VM VM VM
VM VM VMVM1 VM2 VM3 VM1’ VM2’ VM3’
Fenced VM Environment
VM VM VM VM
VM VM VM2’VM1’ VM3’
• Moved to permanent storagefor longer use
Array Replication
Production Data
Full Copy
Snapshot
CatalogicECX
16
The Power of Storage Workflow Templates
Current situation
Measured on:Up-time
Containing costAgility for the Business
Measured on:Speed of software
deliveryQuality of code
not up-time
STORAGE TEAM
• Understand the environment and has the storage skills to set the environment up and provide the required up-time
• Setting up environments is a manual and error prone task that takes time
• Doesn’t have the tools to provide self service to App teams
APP TEAM
• Would like more control over how storage for an application is designed.
• Would like to self-serve storage – eliminate the delay created by Storage team
• At best, the App Team is frustrated with the process
• At worst, the App team goes around the processes and does their own exports or dumps to take control and speed up their work
INTERNAL COMMUNICATION GAP
CIO
Business Unit Leader
17
The Power of Storage Workflow Templates
Templates provide “Common Ground”
CIO
Business Unit Leader
• Understand the environment and has the storage skills to set the environment up and provide the required up-time
• Setting up environments is a manual and error prone task that takes time
• Doesn’t have the tools to provide self service to App teams
• Would like more control over how storage for an application is designed.
• Would like to self-serve storage – eliminate the delay created by Storage team
• At best, the App Team is frustrated with the process
• At worst, the App team goes around the processes and does their own exports or dumps to take control and speed up their work
INTERNAL COMMUNICATION GAP
STORAGE TEAM
APP TEAM
Self-services Provisioning, Automated DR, Test/Dev Refresh, DevOps Automation and Analytics
Storage team: Keeps control and gain SLA management
App Team: Gets the speed
they need through
Self-Service
Measured on:Up-time
Containing costAgility for the Business
Measured on:Speed of software
deliveryQuality of code
not up-time
Big Data / Analytics – After
18
Sales team performs analytics on full, fresh
data set
Finance tam performs analytics on full, fresh
data set
S
F
Marketing team performs analytics on full, fresh data set (any location)
M
Data set made available via automation & orchestration with
application consistent snaps / mirrors
BetterBusiness Decision Made
• Volume, Velocity, Veracity– Data sets growing exponentially– Data sets changing quickly – Data sets take on more and different profiles
• All of this requires data is updated as frequently as (real-time) possible (as often as you take a snap) to yield better results
Data Analytics Automate and Orchestrate Data Copies for Business Agility
19
Production Environment
If you’ve got it, Mount it!Leverage your Copy Data!
• Big Data / Analytics– Same ‘copy’ / ‘use’ data workflows as data
recovery
– Full ECX automation / orchestration
– No manual effort / scripting
– Exception reporting
– Real-time data analytics
– Analyze latest production data
– More accurate analytic results
– Analyze in any location, including the
cloud
– Ensure analytic teams have direct access
to the right data, instantly regardless of
location
– Op/Ex and Cap/Ex savings
Leverage Copy Data
Save CapEx and OpEx – Improve ROA• General operational savings
– Eliminate complex and hard to maintain scripts
– Self Service storage provisioning, DR, Test/Dev,
DevOps and Analytics
– Decrease unplanned downtime through compliance reporting and availability audits
– Replaces multiple tools
• Automatic Daily DR– Leverage existing replicated snapshots – better ROA
– No need for dedicated storage
– Eliminate time consuming and rarely successful DR weekends
• Enhanced Test/Dev or DevOps– Leverage existing replicated snapshots – better ROA
– No need for dedicated storage
– Reduce manual tasks for data population
– Dramatically increase developer productivity
– Increase time to market
• Near Real-time data access for analytics– Eliminate data extraction process via scripting
– Increase analytic benefits through Near-Real-time data access
20
Catalogic Software & IBM Partnership with ECX
21
Catalogic ECX
Manage, Orchestrate and Analyze Copy Data across your Enterprise
and Cloud
Downloadable VM
Agentless
Flash Copy Manager• New & exciting uses cases for Flash Copy
Manager volumes• Automated DR, Enhanced Test/Dev,
DevOps Automation and Analytics
Flash System v9000• Differentiate against other Flash Arrays
with Copy Data use cases– Automated DR, Test/Dev, Analytics and
hybrid cloud
• Leverage application consistent snapshots for dramatically more powerful Test/Dev, DevOps automation and analytics use cases
• Leverage all of v9000’s advanced Copy Services (Snapshots, replication, compression and Easy Tier)
Storwize and SVC• Copy Data use cases for all Storwize and
SVC environments• Automated DR, Enhanced Test/Dev,
DevOps Automation, Analytics and hybrid cloud
• Additionally provides entry point into any VMware environment
• Leverage all of Storwize’s advanced Copy Services (Snapshots, replication, compression and Easy Tier)
VersaStack• Copy Data use cases for VersaStack
environments• Automated DR, Enhanced Test/Dev,
DevOps Automation, Analytics and hybrid cloud
• Additionally provides entry point into any VMware environment
• Leverage all of Storwize’s advanced Copy Services (Snapshots, replication, compression and Easy Tier)
Product Demonstration
Catalogic Software
http://catalogicsoftware.com/ibm/landingpage_demo
23
Software-Defined Copy Data Management
Catalogic ECX
Reduced cost and complexityand increased business agility
Result
Available for the IBM Storwize family, SVC, V9000 and Flash Copy Manager
in Catalogic ECX 2.2
For more information: Data.CatalogicSoftware.com/IBM or email us at IBMinfo@catalogicsoftware.com
24
Thank You