Data-driven DevOps with IBM Virtual Data Pipeline and IBM ...

© 2019 by The Enterprise Strategy Group, Inc. All Rights Reserved.

Reducing the Cost and Complexity of Functional Test Development for Oracle RAC at Scale By Brian Garrett July 2019 .

This ESG Technical Validation was commissioned by IBM and is distributed under license from ESG

Enterprise Strategy Group | Getting to the bigger truth.™

Technical Validation

Data-driven DevOps with IBM Virtual Data Pipeline and IBM Cloud Object Storage

Technical Validation: Data-driven DevOps with IBM VDP and IBM COS 2


Contents

Introduction ............................................................................................................................................................................ 3

Background ......................................................................................................................................................................... 3

Solution ............................................................................................................................................................................... 3

IBM Virtual Data Pipeline (VDP) ...................................................................................................................................... 4

IBM Cloud Object Storage (COS) ..................................................................................................................................... 4

ESG Technical Validation ......................................................................................................................................................... 5

Getting Started .................................................................................................................................................................... 5

Policy-based Provisioning ................................................................................................................................................... 5

Timeline-based Snapshot Image Management .................................................................................................................. 7

Self-service Provisioning and Data Masking ....................................................................................................................... 7

Time Savings Analysis .......................................................................................................................................................... 9

Economics ......................................................................................................................................................................... 10

Productivity ................................................................................................................................................................... 10

Storage .......................................................................................................................................................................... 10

Three-year TCO Analysis ............................................................................................................................................... 11

Other Considerations .................................................................................................................................................... 12

The Bigger Truth .................................................................................................................................................................... 13

ESG Technical Validations

The goal of ESG Technical Validations is to educate IT professionals about information technology solutions for companies of all types and sizes. ESG Technical Validations are not meant to replace the evaluation process that should be conducted before making purchasing decisions, but rather to provide insight into these emerging technologies. Our objectives are to explore some of the more valuable features and functions of IT solutions, show how they can be used to solve real customer problems, and identify any areas needing improvement. The ESG Validation Team’s expert third-party perspective is based on our own hands-on testing as well as on interviews with customers who use these products in production environments.



Introduction

This ESG Technical Validation is an evaluation of IBM Virtual Data Pipeline (VDP) with IBM Cloud Object Storage (COS) with a focus on the financial, performance, and time-savings value of the combined solution for cost-effective test and development database environments. The validation test copies a 50TB production Oracle database to five remote DevOps copies, going from block storage to object storage while maintaining high performance.

Background

The digital economy is driving massive volumes of information into databases. Database administrators (DBAs) are struggling to balance operational needs such as performance and availability with the needs of their development teams who want more database copies and faster refreshes of the current copies. These challenges also inhibit innovation and

make it difficult for the DBA to consider and implement new offerings such as self-service database provisioning, cloud integration, and instant database recovery solutions.

Data has immense value beyond just data protection, yet access by non-IT users is limited. ESG research indicates that 42% of secondary storage is used for non-data protection purposes, such as DevOps, data science, and testing.1 IT needs to provide self-service data delivery for developers, data scientists, and application owners with a goal of faster development cycles and less risk at the lowest possible cost.

Solution

Advances in cloud object storage have changed its perception from that of a low-cost, lower-performing archiving technology to one suitable for DevOps database management, data science, and test environments that require higher performance. This is a shift from the belief that you can’t use COS for DevOps, due to issues translating between block and COS storage and the idea that COS doesn’t provide the required performance. In fact, ESG’s validation testing dispels this myth. Figure 1 is the topology of the testing environment, where a 50TB block storage production Oracle database was replicated across a WAN to a remote COS environment.

Figure 1. The Solution: Data-driven DevOps with IBM VDP and IBM COS

Source: Enterprise Strategy Group

1 Source: ESG Master Survey Results, Copy Data Management Trends, March 2018.

of secondary storage is used for non-data protection purposes, such as DevOps, data science, and testing.

https://research.esg-global.com/reportaction/CopyDataManagementTrends/Toc



IBM Virtual Data Pipeline (VDP)

IBM VDP is fueling the data-driven enterprise with its software platform, which enables enterprises to consume data as a service. IBM VDP technology delivers application-centric, SLA-driven automation that decouples the management of the data pipeline for DevOps. The result is the ability to create on-demand virtual copies of production data for application development, testing, and analytics. VDP manages the data through a single platform from one “Golden Copy” of the production data. DevOps access a single, masked, any-point-in-time copy of the data through a self-service interface.

IBM VDP has a strong value proposition, which includes:

• Speed: The typical process associated with making a copy of production data includes gaining approval, provisioning compute and storage, creating clones, and masking data. For many enterprises, this is a slow process that can take weeks compared to the hours it takes using IBM VDP.

• Quality: Product quality suffers when development and QA engineers are not able to test with the right data at

the appropriate time in the development cycle. Not having the right data can lead to gaps in test coverage, potentially missing functional and system requirements. VDP ensures production-like test data is available to testers throughout the development lifecycle.

• Control: Once copies are created, many organizations struggle to provide role-based access controls (RBAC) and

an audit trail of who has access to data sets. A lack of control increases security and compliance risks. IBM VDP provides RBAC, audit trails, and data masking (with IBM InfoSphere Optim Data Privacy) to hide PII (personally identifiable information) from developers.

• Cost: When organizations decide to create copies of production data sets, they unfortunately end up creating many physical copies, which leads to high storage costs and a burden on the operations team. IBM VDP creates lower cost, fully functional virtual clones at a fraction of the cost.

IBM Cloud Object Storage (COS)

IBM Cloud Object Storage makes it possible to store practically limitless amounts of data, simply and cost-effectively. It is commonly used for data archiving and backup, for web and mobile applications, and as scalable, persistent storage for analytics. Flexible storage class tiers with a policy-based archive let you effectively manage costs while meeting data access needs. The integrated IBM Aspera high-speed data transfer option makes it easy to transfer data to and from Cloud Object Storage, and query-in-place functionality allows you to run analytics directly on your data. Some key features include:

• Flexible Storage Class Tiers – It provides five storage class tiers: Standard for data with active workloads, Vault for less active workloads, Cold Vault for cold workloads, Flex for dynamic workloads, and Archive for long-term retention.

• Low-cost Archiving – Policy-based archive is a low-cost option for data that is rarely accessed and requires long-term retention. Customers can manage their data lifecycle by applying an archive policy for objects written to their object storage buckets in any of the storage class tiers. After the specified duration, objects are archived.

• Resiliency – IBM offers worldwide locations for data storage needs with three resiliency options: Cross Region, where data is stored across three regions within a geography; Regional, where data is stored in multiple data centers within a single geographic region; and Single Data Center, where data is stored across multiple devices in a single data center.

• Fast Data Transfer – Aspera high-speed data transfer capabilities are integrated with IBM Cloud Object Storage, offering a fully integrated user interface to quickly transfer data to and from IBM Cloud Object Storage.



• Immutable Object Storage – This allows organizations to protect data from deletion or modification, preserve records and maintain data integrity in a WORM (Write-Once-Read-Many), non-erasable, and non-rewritable manner until the end of retention periods and the removal of any legal holds.

ESG Technical Validation

ESG performed evaluation and testing of IBM VDP with IBM COS. This involved backing up a production Oracle database to a local copy in one data center and then replicating across a WAN to IBM COS in IBM cloud where five DevOps copies were produced. This process was then compared to a traditional DevOps provisioning model. During the testing, we looked at performance, time savings, and core VDP and COS features including policy-based provisioning, timeline-based snapshot image management, self service provisioning, and data masking.

Getting Started

To get started, we developed the test bed, as seen in Figure 2. This started with a production Oracle 12c database located in IBM Cloud in Dallas, Texas. IBM VDP uses Oracle RMAN APIs to create a local copy on block storage, which is also set for daily incremental updates. Compression technology in VDP reduced the database to nearly a third of the size. The backup/Golden Copy was remotely replicated using IBM VDP over a 10gbps wide area network to IBM COS storage in the DevOps environment. From here, an end-user or a single administrator could provision DevOps copies. In this scenario, five separate DevOps copies were produced.

Figure 2. The ESG Validation Test Bed


Policy-based Provisioning

In most organizations, the typical process associated with making a copy of production data for developers includes gaining approval, provisioning compute and storage, creating clones, and masking data. For many enterprises, this is a slow process that can take weeks or even months. Environments such as SAP or Oracle ERP often require ten or more databases for backup and development.



In the test environment, IBM VDP was implemented at the data center to test the performance and ease with which IT can produce copies of the 50TB Oracle database for use by DevOps. Included in the test was a query of 600 million lines, which returned the results in 8 minutes running on IBM Cloud Object Storage (COS). This was even faster than running on the production environment, where a similar query took over 13 minutes on block storage.

The first step in the demonstration was to view the production and backup database, which was replicated using RMAN for Oracle with IBM VDP to create the “Golden Copy.” This is a fully replicated copy with real-time updates from production. Next, we were introduced to the VDP interface, which has the ability to create on-demand virtual copies for application development, test, and analytics. VDP manages the data through a single platform from the one Golden Copy of the production data.

In the demonstration, one IT administrator was able to create five clones of the database in a matter of minutes. These instant virtual copies were non-disruptive clones of read/write copies and consumed virtually no additional storage. The interface allowed for setting policies and privileges that can be used by DevOps in a self-service model to create copies on demand. This can dramatically reduce the time required for the DevOps team to access data.

Figure 3 is the provisioning interface, which shows a topology of the deployed systems and policy management. In this diagram, you can see production, snapshot, and OnVault databases, along with the Snapshot policies. In this example, incremental changes are set to be made daily between 19:00 and 7:00.

Figure 3. Policy-based Copy Data Management


Once ready, the DevOps team access a single, masked, any-point-in-time copy of the production data through the self-service interface. The instant point-in-time function allows them to also access previous points in time near-instantly, even for this 50TB-sized database. Data consistency is used to refresh the copies with any recent updates. Product quality often suffers when development and QA engineers are not able to test with the right data early in the development cycle. Not



having the right data can lead to gaps in test coverage, potentially causing teams to miss requirements. Finding critical bugs late in the development cycle can also lead to software quality issues and delays in the release cycle.

The benefits to DevOps include a rapidly deployable self-service DevOps environment that requires no storage for additional copies of up-to-date production data and that is ready for immediate use with sensitive data masking in compliance with organizational privacy requirements.

Timeline-based Snapshot Image Management

With all backup solutions, it’s important to have a clear current and historical timeline. Figure 4 shows how IBM VDP graphically displays the current environment, providing users with the ability to scroll back in time to see each replication, update, and deployment, including snapshots, dedupe, remote dedupe, remote snapshot, and OnVault clones. The snapshot from our testing is displayed in gold with our hourly DevOps snapshots shown in blue.

Figure 4. Timeline-driven Snapshot Management


Self-service Provisioning and Data Masking

Among the strongest features of IBM VDP are the self-service capabilities, driven by an intuitive, role-based interface, as seen in Figure 5. Remote team members, once authorized for self-service, can securely log in and access multiple virtual databases instantly without any manual processes. Once copies are created, it can be difficult for organizations to provide role-based access controls (RBAC) limiting who has access to copies of production data sets or to accurately track audit



trails. A lack of control increases the risks associated with a security breach or non-compliance. The DevOps team can access a single, masked, any-point-in-time copy of the production data through the VDP self-service interface.

Figure 5. Role-based Access Control


Performance

The performance of the DevOps databases was measured with the production version as the benchmark. This was a hard look at how well the DevOps database performed while running on lower cost Cloud Object Storage versus the production database running on standard 10 IOPS per GB block storage with the same server hardware. First, we ran a simple line

count query on the production system and the DevOps system. On the production system, this simple query completed in 13:17 (minutes:seconds), and on the DevOps system, it took 5:02, which was 62% faster. The we ran a 600M row complex query on both systems . On the production system, the complex query took 15:49, and on the DevOps system, the same query completed in 8:00, which was 49% faster.

The DevOps environment running on object storage was faster than the production environment running on block storage for every query that was tested. We attribute these results to the relatively low performance of the general-purpose block storage in the production environment (IBM block storage as a service rated at 10 IOPS/GB). ESG expects that the performance gap would narrow if the production Oracle RAC environment was running on an enterprise-class all-flash array. Regardless of the performance of the primary database infrastructure, ESG is confident that the performance of database clones that are running on IBM COS with IBM VDP is more than adequate for functional test development.

600M row complex query completed in

8 minutes on IBM Cloud Object Storage



Time Savings Analysis

Over the past years, IT and DevOps have become strategic profit centers for many organizations, as technology has become a corporate competitive edge. IT no longer has weeks available with which to provision servers and storage for DevOps, which need to run as agile environments to enable them to perform break-fix functions and develop application enhancements and long-term strategic software developments. This is coupled with the need to always be working with the latest data. In Figure 6, we compared a traditional provisioning time for producing five 50TB Oracle database copies for use by DevOps to using IBM VDP on IBM COS. In the traditional model, IT needs to provision 250 TB of storage to account for the five database copies. This involves up to four administrators working on server deployment, storage management, database administration, DevOps requirements, and more. This is compared to using one assigned DevOps user using the IBM VDP self-service dashboard and its ability to provision all five copies in under 90 minutes and only utilizing 17 TB of storage. This represents a massive reduction in operational burden, time savings, and storage savings. In addition, all of the copies can be set to receive updates to refresh the database, which is critical for any development work. For security, all PII (personally identifiable information) is masked. This agile environment allows the organization to save provisioning time, increase the quality of code development, and apply fewer IT resources.

Figure 6. Time and Complexity Reduction Analysis


Why This Matters The near instant creation of database copies for use by development and operations is a critical step towards a data-driven organization, where time to market, agility, and performance can be directly tied to the organization’s bottom line.

IBM Virtual Data Pipelines (VDP) combined with IBM Cloud Object Storage (COS) is a powerful combination of high performance at a low-cost point. Our testing shows that IBM COS provides the performance needed for managing DevOps copies of databases and delivers the speed and agility required for agile development environments, where time to market and quality of code are critical factors. VDP self-service features combined with access controls move deployments into the hands of the IT organization with the ability to spin up database clones quickly and easily. C-level executives and IT security personnel can also take comfort in the integration with IBM InfoSphere Optim Data Privacy, which enforces corporate-driven privacy policy management and eliminates access to sensitive information in backup copies.



Economics

To evaluate the economic value, the total cost of ownership was modelled with the 50TB production database footprint, a full copy backup, and five cloned DevOps copies. For many organizations, faced with the need to reduce costs while increasing time to market and overall operational efficiencies, this represents one of dozens, hundreds, or even thousands of databases. The scenario included the use of IBM Cloud Object Storage (COS) running in the data center locations. The DevOps environments were a combination of Oracle RAC and ASM Databases. The compelling economic advantages were validated in this extensive model.

The model was shown to increase productivity and reduce the cost of data-driven DevOps due to a variety of factors that can be classified into two major categories: Productivity and Storage.

Productivity

As infrastructure has become more commoditized, applications have become more strategic. IBM VDP decouples applications from infrastructure, improving business resiliency, agility, and cloud mobility.

IBM VDP eliminates redundant data copies, consolidates overlapping storage services, and performs the basic functions of copy, store, move, and optimize. Using a common data management platform means less software to support, fewer licenses to buy, fewer integration headaches, and above all, lower cost and less operational complexity. With VDP, dozens of virtual database clones can be provisioned in a matter of minutes, compared to 14 days in a typical process. The result is less data moving across networks, less data to store, greater efficiency in long-term retention, and substantial reductions in TCO (total cost of ownership).

IBM VDP is making it easier for organizations to build agile environments by, for instance, increasing organizational efficiencies such as the cross-functional roles of DevOps. Developers (dev, QA, analytics, data scientists, AI) and operations (UAT, pre-production, release management, application management, and monitoring teams) make up the two key constituencies of DevOps. Both have separate but overlapping core objectives. Development is focused on TTM (time to market) where operations is focused on data protection, backup, recovery time objectives (RTO), and operational efficiencies. VDP helps to create more synergy between the organizations that utilize the same data resources.

Storage

The rapid creation of DevOps environments is crucial for development and the delivery of updates and bug fixes to satisfy business objectives and customer requirements. By using IBM VDP with IBM COS, organizations can rapidly create DevOps environments with up-to-date virtual copies of production data while dramatically

lowering the cost of storage.

IT administrators can access a single, masked, any-point-in-time copy of the data through a self-service interface, and create virtual database clones on additional virtual servers, including VMware servers provisioned with compute and storage resources. The cloned virtual servers are storage-efficient and do not increase storage capacity except for any

From 14 days to less than two hours

Elapsed time to create and provision 50TB clones for five

functional test developers



additional writes. This decreases time to market for software releases and allows application teams to develop code with more agility. We saw a 95% storage capacity savings due to capacity-efficient IBM VDP snapshots instead of full copies for functional TestDev.

On the operations side, many organizations suffer with slow restore times for databases over 1 Terabyte in size, which could take more than 24 hours to restore and involve many different staffers. That’s in addition to the time and effort it takes to refresh non-production test and development environments with updates, which could take

multiple weeks and involve a long approval and provisioning process.

Because VDP keeps Oracle databases in their native application format, and then simply presents (instantly mounted) virtual databases (synthetic full point-in-time copies) back to hosts for restorations, the database and ops teams can now recover their databases in minutes.

Three-year TCO Analysis

The TCO looked at the process of working with the Golden Copy (backup of production) of the database and the hard costs associated with producing copies of the database using IBM VDP. This was compared to the more traditional model of running backups or replicating full copies of the database for each DevOps copy that was needed. In the analysis shown in Figure 7, we assumed a fully burdened storage cost of $.58 per GB per month for the traditional model using block storage and DevOps copies on COS at a cost of $.023 per GB. With IBM VDP, the DevOps copies are virtual clones of the database with a storage cost equivalent to 5% of the full backup copy. Assuming an organization has five DevOps database copies running for a 12-month period, the IBM TCO over three years is $1,676,529 compared to $7,946,160 in a traditional block storage model. This represents a storage cost savings of $6,269,261, a 79% cost reduction. For an organization with hundreds of databases, this represents millions in annual savings and a simplified model.

Figure 7. Total Cost of Ownership Analysis


95% Storage capacity savings due to capacity-efficient snapshots instead of full copies for functional TestDev



Other Considerations

A variety of factors that increase the value were not modelled but should be considered.

Time to Market: IBM’s capabilities mesh well into any agile development strategy by allowing faster mounting of DevOps databases, which are kept in sync with production. This may decrease time to market of new releases and enhancements with higher quality tested code.

Data Privacy Compliance: Before a DevOps team has access to virtual clones, sensitive information is masked in compliance with corporate governance policies and privacy regulations.

Customer Satisfaction: The ability to quickly address bugs and customer enhancements is greatly increased and leads to higher customer satisfaction. DevOps can reproduce and address issues faster.

Increased Revenue and Profitability: The decrease in storage capacity and CPU cores, reduction in complexities, and faster time to market should result in lower costs, and increased revenue and profitability.

Quality Control: Virtual clones are linked back to the Golden Copy, which is updated with incremental changes from the production environment. This allows IT to control the quality of the DevOps copies and eliminates a working copy from becoming obsolete. Quality increases since developers are working from a full, current data set.

Virtual Cloning on Demand: New clones can be created in minutes. This can be under the control of an IT administrator or designed as self-service.

Employee Productivity: DevOps teams get quicker copies of current production data for test and development.

The level of savings that your organization will achieve depends on a variety of factors, including the size of your environment, number of databases, FTEs, and the number of DevOps copies you maintain. That said, if your organization is running an agile development or plans to and is looking to reduce data center costs and complexities, then ESG is confident that the economic benefits will hold true.

Why This Matters ESG validated that the three-year TCO for creating five DevOps copies of a 50TB Oracle Database in IBM VDP with IBM COS is 79% lower than a traditional full copy model. This significant savings is amplified if you are an organization with many databases used by development and operations teams.

We looked further into the analysis to identify some of the soft benefits, which include self-service provisioning, compliance-driven data masking, role-based access controls, and real-time updates from the database to DevOps copies. Combined, these benefits help an organization to be more agile, deliver solutions to market faster, and increase the quality of code developed since the DevOps copies are always current.



The Bigger Truth

DevOps teams are faced with the increased challenge of working in a fast-paced agile environment where secure, easy access to current production data is critical for time to market and the delivery of quality code. Doing it right can result in a competitive advantage for organizations, higher customer satisfaction, and even increased productivity and satisfaction for employees. IBM VDP with IBM COS is a strong combination that reduces cost and complexity while also delivering on the promise to reduce the time to produce quality applications. The combined benefits make IBM VDP with IBM COS a clear choice for organizations looking to increase productivity and decrease IT costs.

ESG validation testing has confirmed that:

• IBM Virtual Data Pipeline with IBM Cloud Object Storage dramatically reduces the complexity of creating five Oracle 12c clones for functional test developers.

• Automated policy-based virtual cloning of five DevOps copies that took less than 2 hours (versus 14 days for traditional cloning methods) resulted in a three-year TCO reduction of 79%.

• Performance of simple and complex queries on Oracle 12c database clones that were transparently deployed on object storage in the IBM Cloud was more than adequate for functional test development.

If your organization is looking to streamline IT infrastructure and optimize DevOps workflows, then ESG believes that you should seriously consider the economic advantages of fueling the data-driven enterprise with IBM Virtual Data Pipeline and IBM Cloud Object Storage.

All trademark names are property of their respective companies. Information contained in this publication has been obtained by sources The Enterprise Strategy Group (ESG) considers to be reliable but is not warranted by ESG. This publication may contain opinions of ESG, which are subject to change from time to time. This publication is copyrighted by The Enterprise Strategy Group, Inc. Any reproduction or redistribution of this publication, in whole or in part, whether in hard-copy format, electronically, or otherwise to persons not authorized to receive it, without the express consent of The Enterprise Strategy Group, Inc., is in violation of U.S. copyright law and will be subject to an action for civil damages and, if applicable, criminal prosecution. Should you have any questions, please contact ESG Client Relations at 508.482.0188.

www.esg-global.com [email protected] P. 508.482.0188

Enterprise Strategy Group is an IT analyst, research, validation, and strategy firm that provides market intelligence and actionable insight to the global IT community.


Data-driven DevOps with IBM Virtual Data Pipeline and IBM ...

Documents

Transcript of Data-driven DevOps with IBM Virtual Data Pipeline and IBM ...