H7052 - EMC Backup and Recovery for SAP Enabled by EMC ... · PDF file(ABAP) to load data into...

37
EMC Backup and Recovery for SAP Enabled by EMC NetWorker Module for SAP with EMC Avamar A Detailed Review EMC Information Infrastructure Solutions Abstract This white paper describes how the backup and restore operation of SAP systems benefit from client-based deduplication technology using EMC ® Avamar ® . The white paper also addresses how customers can use the EMC NetWorker ® PowerSnap module to reduce the performance impact on the SAP production server during backup. July 2010

Transcript of H7052 - EMC Backup and Recovery for SAP Enabled by EMC ... · PDF file(ABAP) to load data into...

EMC Backup and Recovery for SAP Enabled by EMC NetWorker Module for SAP

with EMC Avamar A Detailed Review

EMC Information Infrastructure Solutions

Abstract

This white paper describes how the backup and restore operation of SAP systems benefit from client-based deduplication technology using EMC® Avamar®. The white paper also addresses how customers can use the EMC NetWorker® PowerSnap™ module to reduce the performance impact on the SAP production server during backup.

July 2010

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 2

Copyright © 2010 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com

All other trademarks used herein are the property of their respective owners.

Part number: H7052

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 3

Table of Contents

Executive summary ........................................................................................................................... 5 Business case ............................................................................................................................... 5 Product solution............................................................................................................................. 5 Key results ..................................................................................................................................... 5

Introduction........................................................................................................................................ 6 Overview ....................................................................................................................................... 6 Purpose ......................................................................................................................................... 6 Scope ............................................................................................................................................ 6 Audience ....................................................................................................................................... 6 Terminology ................................................................................................................................... 6

Key components ............................................................................................................................... 8 Introduction .................................................................................................................................... 8 EMC Symmetrix VMAX ................................................................................................................. 8 EMC Avamar ................................................................................................................................. 8 EMC NetWorker PowerSnap Module for Symmetrix .................................................................... 8 EMC NetWorker Module for SAP with Oracle ............................................................................... 8 VMware vSphere ........................................................................................................................... 9 IDES SAP ERP ............................................................................................................................. 9 Data Protection Advisor ................................................................................................................ 9

Physical architecture ....................................................................................................................... 10 Overview ..................................................................................................................................... 10 Physical environment .................................................................................................................. 10 Hardware resources .................................................................................................................... 11 Software resources ..................................................................................................................... 11

Configuration and design ................................................................................................................ 12 Introduction .................................................................................................................................. 12 Key considerations ...................................................................................................................... 12 LUN design .................................................................................................................................. 13 SAP application server configuration .......................................................................................... 14 PowerSnap specific configuration ............................................................................................... 15 Data load configuration ............................................................................................................... 16

Testing and validation ..................................................................................................................... 17 Introduction .................................................................................................................................. 17 Scenarios overview ..................................................................................................................... 17 Test group 1: Baseline with BCT off............................................................................................ 18 Test group 2: Avamar on with BCT off ........................................................................................ 18 Test group 3: Baseline with BCT on............................................................................................ 20 Test group 4: Avamar on with BCT on ........................................................................................ 21 Results validation ........................................................................................................................ 22

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 4

Avamar compared to baseline .................................................................................................... 22 Avamar compared to BCT ........................................................................................................... 24

Conclusion....................................................................................................................................... 28 Introduction .................................................................................................................................. 28 Client-based deduplication by Avamar........................................................................................ 28 Avamar and Oracle BCT ............................................................................................................. 28 Automation of snapshot-based backup using PowerSnap ......................................................... 28 Centralized management with NetWorker integration ................................................................ 28

References ...................................................................................................................................... 29 White papers ............................................................................................................................... 29 Product documentation ............................................................................................................... 29 Other documentation ................................................................................................................... 29

Additional information ...................................................................................................................... 30 Introduction .................................................................................................................................. 30 Baseline - BR*Tools configuration file for RMAN ........................................................................ 30 Baseline - BR*Tools configuration file for PowerSnap ................................................................ 31 Baseline - NMSAP configuration file for RMAN .......................................................................... 32 Baseline - NMSAP configuration file for PowerSnap .................................................................. 33 Avamar - BR*Tools configuration file for RMAN ......................................................................... 33 Avamar - BR*Tools configuration file for PowerSnap ................................................................. 35 Avamar - NMSAP configuration file for RMAN ............................................................................ 36 Avamar - NMSAP configuration file for PowerSnap .................................................................... 37

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 5

Executive summary

Business case SAP data growth can be as much as 60 percent per year. In many cases there are

multiple SAP solutions with production and non-production data growing continuously. This situation is driving SAP customers to revisit their backup and recovery strategies as traditional backup systems are being pushed to capacity.

The EMC Backup and Recovery for SAP – Enabled by NetWorker Module for SAP with Avamar solution is designed to enable SAP customers to achieve new levels of data reduction and provide fast, secure backup for their production and non-production SAP environments. In the process, SAP backup time and growth of secondary storage are reduced, along with decreased network utilization. In case of data corruption, users can recover the system in minimal time by using full backup.

Product solution

This solution demonstrates the benefits of client-based deduplication in an SAP/Oracle environment on the EMC® Symmetrix® platform as well as the benefits of integrating EMC PowerSnap™ and the EMC NetWorker® Module for SAP with Oracle (NMSAP) using the EMC Avamar® plug-in and NetWorker.

The solution builds a replica of a common customer environment and validates the configuration for both performance and functionality of client-based deduplication enabled by Avamar.

NetWorker is used as the backup management tool to perform the multi-stream backup to Avamar. Through interaction with the PowerSnap module, NetWorker controls the process of snapshot-based backup and eliminates the performance impact of the backup job on the production server being backed up.

EMC Data Protection Advisor (DPA) collects relevant backup information. In this solution, we will show how you can use this analysis tool for data collection, data recording, and backup metrics management.

Key results This white paper demonstrates the following benefits for the solution:

• Avamar reduces the network utilization of the backup jobs by 57 percent while achieving a deduplication ratio of 10:1.

• NetWorker PowerSnap enables the client-based deduplication without additional CPU utilization on the production host.

• Multi-streaming with NMSAP enabled by NetWorker allows the backup job to finish over two and a half times faster than if performed with a single stream.

• Restoration from Avamar taking advantage of multi-streaming with NMSAP enabled by NetWorker is six times faster than Recovery Manager (RMAN) with BCT.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 6

Introduction

Overview This white paper clarifies the benefits of client-based deduplication and mirror-based

backups for SAP landscapes. It validates the feasibility of implementing a deduplication approach in a typical SAP environment.

The reference architecture presented in this white paper will help you prepare a backup and recovery plan and topology for SAP systems.

Purpose The purpose of the white paper is to:

• Provide an understanding of client-based deduplication for backup of SAP systems

• Supply an overview of the snapshot-based backup method and explain the reason for using this backup method

• Demonstrate the integration capability of NetWorker Module for SAP with Oracle (NMSAP), PowerSnap, and Avamar with central control by NetWorker

Scope This white paper documents the software and hardware resources involved in this

solution as well as the corresponding configuration for building the test environment. The white paper also addresses the solution design, the test scenario pattern, and the conclusion based on test results and analysis.

It is assumed the reader has knowledge and experience using BR*Tools for backup and recovery of SAP with Oracle databases. Detailed explanations of these concepts are not in the scope of this paper.

Audience This white paper is intended for customers, partners, and EMC employees, including

IT planners, SAP architects, Basis administrators, and any other people involved in evaluating, acquiring, managing, operating, or designing an SAP landscape infrastructure.

Terminology This section defines terms used in this document.

Term Definition

DEV SAP development system

QAS SAP quality assurance system, also referred to as test system

PRD SAP production system

RMAN Recovery Manager, an Oracle built-in tool for backup and restore, which performs backup at the database level

BCT Block Change Tracking, an Oracle feature that reduces backup times on incremental backups by tracking changed blocks since

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 7

the last backup on a dedicated file

BR*Tools Toolset that the SAP for Oracle database administration (DBA) tasks provide

Proxy Host Also known as Data Mover, the secondary host, from which PowerSnap implements snapshot-based backup

BACKINT The external backup library called by BR*Tools to perform SAP system backup at the file system level. It is SAP certified.

Baseline backup

Backup to SATA volumes without using the deduplication technology

LSMW Legacy System Migration Workbench (LSMW), an SAP specific tool written in Advance Business Application Programming (ABAP) to load data into the system

PIT recovery Point-in-time recovery

Solution Enabler

EMC shared libraries for use by Solutions Enabler applications, and the Symmetrix Command Line Interface (SYMCLI) for use by storage administrators and systems engineers.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 8

Key components

Introduction This section briefly describes the key solution components. For details on all of the

components that make up the architecture, see the “Physical architecture” section in this paper.

EMC Symmetrix VMAX

EMC Symmetrix VMAX™ provides high-end storage for the virtual data center. Symmetrix VMAX scales up to 2 PB of usable protected capacity and consolidates more workloads with a much smaller footprint than alternative arrays.

The innovative EMC Symmetrix Virtual Matrix Architecture™ of VMAX seamlessly scales performance, capacity, and connectivity on demand to meet all application requirements. Symmetrix VMAX can be deployed with Flash drives, and Fibre Channel and Serial Advanced Technology Attachment (SATA) drives, with tiering fully automated with FAST. It supports virtualized and physical servers, including open systems, mainframe, and IBM i hosts.

EMC Avamar EMC Avamar backup and recovery solutions utilize patented global data

deduplication technology to identify redundant data at the source, minimizing backup data before it is sent over the LAN/WAN. With Avamar, you can achieve new levels of data reduction and enable fast, secure backup for your VMware environments, remote offices, LAN/NAS servers, and desktop/laptop systems. In the process, you will reduce backup time, growth of secondary storage, and network utilization.

Avamar deduplicated backups function like full backups and can be recovered in just one step. You do not need to restore full backups and subsequent incremental backups. In addition, Avamar verifies backup data recoverability, so there are no surprises, and encrypts data for secure electronic backups.

EMC NetWorker PowerSnap Module for Symmetrix

The PowerSnap Module enhances NetWorker by allowing snapshot-based data protection and availability during backups for the Symmetrix storage system. The backup is virtually instantaneous, regardless of the volume size. The PowerSnap Module software enables frequent backups with little impact on the network or application server, and provides backups of large databases or data stored in file systems.

EMC NetWorker Module for SAP with Oracle

EMC NetWorker Module for SAP with Oracle (NMSAP) ensures that the mission-critical data is available by delivering fast, online backup and recovery for SAP. With NMSAP, SAP administrators can centrally manage backup operations and schedules.

Featuring data deduplication support through its embedded Avamar client, NMSAP reduces the impact of data protection operations on production servers and networks, and significantly reduces the amount of backup storage required.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 9

VMware vSphere

VMware vSphere is the foundation for a dynamic, self-optimizing IT infrastructure. It is a robust, production-proven virtualization layer that abstracts processor, memory, storage, and networking resources into multiple virtual machines. VMware vSphere allows enterprises to dramatically reduce hardware and operating costs by sharing resources across a virtual environment.

IDES SAP ERP SAP ERP IDES is the "Internet Demonstration and Evaluation System" in the R/3

System, and represents a model company. It consists of an international group with subsidiaries in several countries. IDES contains application data for various business scenarios you can run in the SAP system. The business processes in the IDES system are designed to reflect real-life business requirements, and have access to many realistic characteristics. IDES uses easy-to-follow business scenarios to show you the comprehensive functions of the R/3 System. The focal point of IDES, however, is not the functionality itself, but the business processes and their integration.

Data Protection Advisor

Data Protection Advisor (DPA) is an advanced backup reporting and analysis application that gives its users complete visibility into everything happening within their backup environment. DPA gathers data from backup servers, backup clients, databases, network switches, SAN switches, and tape libraries. DPA correlates information from each of these areas and provides advanced reporting, troubleshooting, performance management, and capacity planning.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 10

Physical architecture

Overview The following section identifies and briefly describes the technology and components

used in the environment.

Physical environment

The following diagram illustrates the overall physical architecture of the environment:

The SAP DEV and QAS systems, NetWorker server, and DPA server are all built on the virtual machine managed by the VMware ESX server. NetWorker controls the backup and restoration processes, and the DPA server captures the data results.

The SAP PRD system and the proxy host reside on the physical machines, which are connected with the ESX server through a LAN for communication purposes

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 11

between NMSAP, PowerSnap and the NetWorker server. EMC Symmetrix VMAX is attached to all application servers through the SAN. In addition, the NetWorker defines Avamar as the deduplication node in device configuration.

Hardware resources

The hardware used to validate the solution is listed in the following table.

Equipment Quantity Configuration

EMC VMAX SE 1 52 x 300 GB 15k FC disks 52 x 1 TB 7.2k SATA disks

VMware ESX server 1 Dell PowerEdge R900 16 Core

SAP host 4 • DEV: VMware Virtual Machine 2 CPUs, 8 GB RAM

• QAS: VMware Virtual Machine 2 CPUs, 8 GB RAM

• PRD: Dell PowerEdge 6850 16 Core

• Proxy host: Dell PowerEdge R710 16 Core

EMC NetWorker server 1 VMware Virtual Machine 2 CPUs, 8 GB RAM

EMC DPA server 1 VMware Virtual Machine 2 CPUs, 8 GB RAM

SAN switch 1 Cisco Catalyst 9509

EMC Avamar Appliance 1 EMC Avamar Data Store Grid (six nodes) Gen2

Software resources

The software used to validate the solution is listed in the following table.

Software Version

Linux Red Hat Enterprise 5.3 AS5 U2 64 bit

Oracle Database 10.2.0.4

SAP Application IDES ERP 6.0 EhP4

EMC NetWorker 7.6

EMC NetWorker Module for SAP with Oracle 4.0

EMC NetWorker PowerSnap Module for Symmetrix DMX™/VMAX

2.5

EMC Solutions Enabler 7.0.1.0

VMware vSphere Server 4.0 Update 1

EMC Avamar software 4.1

EMC Data Protection Advisor 5.5

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 12

Configuration and design

Introduction This section provides the configuration information for this solution, including:

• Disk allocation pattern

• Relevant SAP system settings

• Corresponding NetWorker components

• Key considerations or prerequisites for the proposed test scenarios

The SAP systems presented in this solution follow the principle of the standard three-system landscape known as DEV, QAS, and PRD.

Key considerations

EMC considered the following when performing the test scenarios in this solution:

• An Advanced File Type Device is created on SATA disks on VMAX as the destination for baseline backup.

• RMAN is configured as the default backup tool in BR*Tools for backing up database files to backup destination. The database control file, the BR*Tools profiles, and logs are still backed up through BACKINT.

• BACKINT is specified as the default method for backing up all files with PowerSnap, including database data files, the control file, the BR*Tools profiles, and logs.

• You do not need to install a separate Avamar client. Avamar works as a deduplication node in the NetWorker server and NetWorker manages all backup streams to Avamar.

• By default, SAP performs a cumulative backup when running an incremental backup job.

• Backup for archive logs is not configured for the development and testing systems. The production system is separately configured to run every two hours without connecting to Avamar or PowerSnap.

A baseline backup is established to compare the results with that of the client-based deduplication backup. To facilitate database size changes, EMC used Legacy System Migration Workbench (LSMW) for data population to simulate a realtime data increase in the customer environment. In this solution, the comparison between baseline and deduplication backup is only conducted on the production system. EMC also used the development and testing systems in this solution to validate the feasibility of client-based deduplication on virtualized platforms.

When doing backup in a traditional environment that demands optimized performance, you should perform an incremental backup with BCT to reduce the backup duration and network traffic. Therefore it is useful in this solution to measure the performance difference between BCT and Avamar.

Taking the performance impact on the backup client into consideration, EMC developed a series of approaches for backing up a client through a mirrored volume. PowerSnap works together with NetWorker to interact with Solutions Enabler to accomplish this task. EMC recommends that you configure BACKINT as the default backup method for all files when integrating PowerSnap with NMSAP and NetWorker.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 13

LUN design The following table describes the storage design pattern for the SAP DEV and QAS

systems built on virtual machines.

RAID type

Disk type

LUN size Purpose Filesystem

10 VMFS 64 GB Linux root and swap

/

10 RDM 32 GB Original redo log /oracle/<SID>/origlogA /oracle/<SID>/origlogB

10 RDM 32 GB Mirrored redo log /oracle/<SID>/mirrlogA /oracle/<SID>/mirrlogA

5 RDM 64 GB Oracle archive log /oracle/<SID>/oraarch

5 RDM 64 GB x 4

Database data files /oracle/<SID>/sapdata1

5 RDM 64 GB x 4

Database data files /oracle/<SID>/sapdata2

5 RDM 64 GB x 4

Database data files /oracle/<SID>/sapdata3

5 RDM 64 GB x 4

Database data files /oracle/<SID>/sapdata4

5 RDM 32 GB SAP BIN /sapmnt/<SID> /usr/sap/<SID> /usr/sap/trans

5 RDM 32 GB Oracle BIN /oracle/client /oracle/stage /oracle/<SID>/102_64

The following table describes the storage design pattern for the SAP PRD system built on physical machines.

RAID type LUN size Purpose Filesystem

10 64 GB Linux root and swap /

10 32 GB Original redo log /oracle/<SID>/origlogA /oracle/<SID>/origlogB

10 32 GB Mirrored redo log /oracle/<SID>/mirrlogA /oracle/<SID>/mirrlogA

5 64 GB Oracle archive log /oracle/<SID>/oraarch

5 64 GB x 4 Database data files /oracle/<SID>/sapdata1

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 14

5 64 GB x 4 Database data files /oracle/<SID>/sapdata2

5 64 GB x 4 Database data files /oracle/<SID>/sapdata3

5 64 GB x 4 Database data files /oracle/<SID>/sapdata4

5 32 GB SAP BIN /sapmnt/<SID> /usr/sap/<SID> /usr/sap/trans

5 32 GB Oracle BIN /oracle/client /oracle/stage /oracle/<SID>/102_64

SAP application server configuration

The solution is validated with the following configuration specified to SAP systems, where PowerSnap is disabled. Configuration of PRD and the proxy host specific to PowerSnap is outlined separately.

DEV Profile characteristic Value

Database size 271 GB

Installed agent NMSAP

BR*Tools - Backup device type rman_util

BR*Tools - Backup type online_cons

BR*Tools - Backup mode full

QAS Profile characteristic Value

Database size 270 GB

Installed agent NMSAP

BR*Tools - Backup device type rman_util

BR*Tools - Backup type online_cons

BR*Tools - Backup mode full

PRD

Profile characteristic Value

Database size 278 GB

Installed agent NMSAP, PowerSnap (valid in PowerSnap enabled backup)

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 15

BR*Tools - Backup device type rman_util

BR*Tools - Backup type online_cons

BR*Tools - Backup mode full

Note The meanings of BR*Tools parameters are explained below. • rman_util

RMAN backup in combination with a backup library and the backup tool of another manufacturer. BACKINT provides an interface to the external backup tool and is also used to back up profiles, log files, and the control file.

• online_cons The backup is performed when the database is open. Both the database files and the offline redo log files generated during the backup are copied to the same volume. As a result, you have a logically consistent dataset available.

• Full Full database backup at level 0. This backup can be part of an incremental backup strategy.

For more information about the explanation of a BR*Tools configuration, refer to SAP online help.

Refer to the “Additional information” section for more information about the configuration for the PRD system.

PowerSnap specific configuration

The following tables list the configuration characteristics of PRD and the proxy host using PowerSnap.

PRD Profile characteristic Value

Database size 278 GB

Installed agent NMSAP, PowerSnap

BR*Tools - Backup device type util_file_online

BR*Tools - Backup type online_cons

BR*Tools - Backup mode all_data

Note The new value of BR*Tools parameter is explained below. • util_file_online

Use external backup programs for file-by-file backup. Use this parameter value for an online backup, if it is supported by the external backup program. This dynamically sets and ends the backup status of the tablespaces to be saved and thus greatly reduces the volume of offline redo log files during the backup.

• all_data Back up the files of all tablespaces except pure index tablespaces or empty tablespaces.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 16

Proxy host

Profile characteristic Value

Installed agent PowerSnap

Synchronized volumes Database data files

Snapshot type BCV

PowerSnap backup type Immediate live backup (serverless snapshot backup)

Note • This solution uses BCV emulation because PowerSnap requires this mode. You

can define BCV as the value of the snapshot type for the proxy host configuration. In addition, you can use BCVs for local replication and mount them to the proxy host in this solution.

• The database control file, the BR*Tools profile, and log files are always backed up directly from the production host.

Immediate Live Backup refers to a process where BCVs are split and immediately backed up by the PowerSnap module to conventional storage. The BCVs are automatically synchronized with the source volumes.

For more information about the backup/restore definition enabled by NMSAP with PowerSnap, refer to the EMC NetWorker Module for SAP with Oracle 4.0 Administration Guide.

Refer to the “Additional information” section for more information about the configuration for the PRD system.

Data load configuration

Data load simulates the data volume increase and changes generated by day-to-day business operations in the SAP systems. In order to get an acceptable and relatively stable result, we performed three data loads, as shown in each scenario.

The load pattern is outlined in the table below.

Profile characteristic Value

SAP module where the data is generated SD

Object to be generated Sales order

How many records generated each time 100,000

Data volume increase in database by each load

Approximately 2.7 GB (10% data increase of total database)

Note Sales orders were generated randomly to avoid generating records that would artificially tend to deduplicate well. This random process retrieves data from over 20,000 possible customers and 10,000 possible items in the master data and combines them to generate random orders.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 17

Testing and validation

Introduction This section provides detailed information on the test scenarios’ result validation.

The test scenarios were grouped and carried out based on different combinations of key components’ functionality. The key components refer to client-based deduplication enabled by Avamar and Oracle incremental backup optimized by BCT.

The following statements apply to all tested scenarios:

• The backup stream is directed to the Advance File Type Device (baseline backup) when Avamar is disabled.

• The backup stream is always directed to the Avamar deduplication node through NetWorker when Avamar is enabled.

• A full backup is conducted with BCT disabled.

• An incremental backup is conducted with BCT enabled.

• All resulting data is captured by DPA.

Unless otherwise specified, PowerSnap is disabled for the PRD system during backup.

All SAP systems are copied using storage cloning for rapid refresh and consistency in the context of all test scenarios.

Scenarios overview

The test performed on the target system covers both backup and restore operations. Backup is divided into two stages: Stage 1 means an initial backup, and stage 2 refers to the subsequent full/incremental backup upon completion of each data load. Data load applies to the production system in particular.

EMC processed four groups of tests with different configurations of Avamar and BCT in order to meet the following objectives:

• Measure the achieved deduplication ratio as well as any advantages brought by Avamar

• Measure and compare the advantages and disadvantages of using Oracle BCT with Avamar for backups

• Observe how PowerSnap improves the backup process for the production system

• Compare the efficiency between multi-stream backups and single-stream backups

• Validate Avamar support for virtualized SAP systems

The following table summarizes the different scenarios that were tested in this solution.

Factor Backup stage 1 (Initial)

Backup stage 2 (3 data loads) Restore

DEV QAS PRD DEV QAS PRD DEV QAS PRD

Avamar off BCT off F F F N/A N/A N/A C C C

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 18

Avamar on BCT off F F F* N/A N/A F/F/F* N/A N/A N/A

Avamar off BCT on N/A N/A F N/A N/A I/I/I N/A N/A N/A

Avamar on BCT on F F F N/A N/A I/I/I N/A N/A C P

Note F=Full Backup, I=Incremental Backup, C=Complete Restore, P=PIT Restore. *Backup will be performed twice for PRD: one is conducted with RMAN; and the other is conducted with PowerSnap.

Test group 1: Baseline with BCT off

Procedure Step Action

1 Performed one-time backup for DEV, QAS, and PRD.

2 Wiped out the database of the above system and restored from the existing backup.

Result

SID Scenario CPU utilization (%)

Network throughput (MB/s)

Data transfer (GB

Backup source size (GB)

Duration (min)

PRD Backup 4.37 78.55 218.76 215 47.53

SID Scenario Restore target size (GB) Duration (min)

PRD Restore 215 79.1

Note The above results were captured directly from DPA.

The baseline measurement recorded is used for comparison with the following test.

Test group 2: Avamar on with BCT off

Procedure

Step Action

1 Performed backup for DEV and QAS to Avamar.

2 Performed backup for PRD, which contains two phases: a. Initial backup (full backup) b. Subsequent backup (full backup), each with data load with the

predefined pattern The second phase was repeated three times to achieve a stable and

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 19

reasonable result. Switched the backup stream from multiple to single to see how it affected the backup speed.

3 a. Switched backup method to BACKINT and enabled PowerSnap.

b. Executed the same procedure as step 2. c. Wiped out the database and restore from the latest backup.

Result The following tables show the key metrics regarding backup process for PRD.

• Without PowerSnap

Scenario CPU utilization (%)

Network throughput (MB/s)

Data transferred (GB)

Backup source size (GB)

Duration (min)

Initial 25.13 33.62 77.46 215 39.32

1st data load 26.78 12.38 24.25 218 33.43

2nd data load

27.82 11.06 20.88 222 32.22

3rd data load 27.45 11.73 22.75 225 33.10

Note The above results were captured directly from DPA.

Deduplication occurs when the SAP system has been initially backed up and has achieved a 64 percent deduplication rate (roughly, 3:1 ratio). When the data volume increases, the average deduplication ratio can rise up to 10:1. Compared to the baseline backup, the network utilization drops by 57 percent, and the CPU utilization increases by 21 percent.

• Without PowerSnap – Single-stream mode

CPU utilization (%)

Network throughput (MB/s)

Data transferred (GB)

Backup source size (GB)

Duration (min)

10.33 5.21 26.21 228 85.87

Note The above results were captured directly from DPA.

Working in single-stream mode does not fully utilize the network bandwidth and CPU recourses, resulting in longer backup duration over the multi-stream mode. The multi-stream backup finishes in less than half the time of the single-stream job.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 20

• With PowerSnap

Scenario CPU utilization (%) on PRD

CPU utilization (%) on proxy host

Network throughput (MB/s)

Data transfer (GB)

Backup source size (GB)

Duration (min)

Initial 0.28 18.91 28.56 75.05 321 44.85

1st data load

0.19 23.55 1.15 2.03 321 30.15

2nd data load

0.18 23.59 1.11 1.96 321 30.20

3rd data load

0.25 24.63 1.23 2.17 321 30.13

SID Scenario Restore target size (GB) Duration (min)

PRD Restore 321 49.57

Note The above results were captured directly from DPA.

The deduplication ratio of initial backup in PowerSnap mode is around 4:1. It increases sharply to 100:1 on average, with each data load.

In the context of the PowerSnap backup, the proxy host takes over the workload of both the CPU and the network from the production host. The CPU utilization on the PRD host stays under 1 percent throughout the backup job (as expected).

Backup using an external library (BACKINT) takes full advantage of client-based deduplication by transferring data through file-system-level copy. The amount of transferred data is almost identical to the size data volume increases in the database along with data load. The backup source size remains the same as there is sufficient free space in the database to accommodate the appended data.

The restoration with BACKINT in PowerSnap mode is 1.6 times faster than the baseline.

Test group 3: Baseline with BCT on

Procedure

Step Action

1 Performed initial backup for PRD with BCT turned on.

2 Performed data load with the same pattern as step 2 in the previous group. The subsequent backup with each data load was run in the incremental mode.

3 Repeated the above step three times.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 21

Result

Scenario CPU utilization (%)

Network throughput (MB/s)

Data transfer (GB)

Backup target size (GB)

Duration (min)

Initial 3.32 60.54 217.38 215 61.28

1st data load 0.41 36.32 4.75 4.16 2.23

2nd data load 1.19 43.05 7.95 7.4 3.15

3rd data load 1.97 44.27 10.82 10 4.17

Note The above results were captured directly from DPA.

Data transferred is slightly greater than the backup source size, because data transmission over the network has overhead on the carrier in the TCP/IP layer.

Incremental backup enhanced by BCT shows outstanding performance in terms of the backup length. BCT reduces the backup duration because it only backs up the changed data since the last backup.

Test group 4: Avamar on with BCT on

Procedure

Step Action

1 Performed backup for DEV and QAS to Avamar with BCT turned on.

2 Performed initial backup for PRD to Avamar with BCT turned on.

3 Performed data load with the same pattern as step 2 in the previous group. The subsequent backup with each data load was run in the incremental mode.

4 Repeated the above step three times.

5 Wiped out the PRD database and restored it from the most recent full backup plus incremental backup.

6 Performed a PIT restoration based on the previous step to revert the SAP system to an earlier status.

Result

Scenario CPU utilization (%)

Network throughput (MB/s)

Data transfer (GB)

Backup target size (GB)

Duration (min)

Initial 22.56 29.58 75.48 215 43.55

1st data load 11.95 10.16 1.68 4.19 2.82

2nd data load 16.08 16.78 3.17 7.41 3.22

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 22

3rd data load 16.43 19.78 4.75 10 4.1

Note The above results were captured directly from DPA.

Incorporating Avamar with BCT results in a 3:1 deduplication ratio for the initial backup and 2:1 on average for the subsequent incremental backup.

The backup source size is similar to the one as documented in group 3. The only difference is that Avamar deduplicates the data before transferring it, which reflects the actual increased size of the database. This test shows that using NMSAP with the Avamar plug-in in conjunction with Oracle BCT enables faster incremental backups (15 times faster than the Avamar-enhanced full backup) with a reduced network utilization and storage footprint.

SID Scenario Restore target size (GB) Duration (min)

PRD Complete Restore 223 313.95

PRD PIT Restore 215 300.27

Note The above results were captured directly from DPA.

The table shows that the restoration with BCT from Avamar is far slower than the baseline. Therefore, this test scenario is not practical because it is too time-consuming to recover the SAP production system.

Results validation

The following sections consolidate the test results and provide an in-depth analysis across test scenarios.

Avamar compared to baseline

The SAP system backup can benefit from Avamar in terms of reducing the backup storage footprint as well as improving performance.

The following section compares the configuration of baseline backup with that of an Avamar backup (BACKINT with Avamar).

Backup storage footprint The delta size between two full backups is relatively small compared to the database size. For backup cycles, the conventional backup storage consumption increases linearly. When Avamar is introduced, the backup storage consumption increases with the same proportion as that of the delta size between two full backups for SAP databases which decreases the backup storage footprint significantly.

Thus, for very large (up to TB level) SAP databases, using Avamar as the backup destination facilitates the backup storage management with high deduplication.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 23

Performance As the graph below shows, the Avamar-enabled full backup completes in 20 percent less time while using 60 percent less bandwidth. The side effect is that Avamar consumes more CPU resources at runtime because Avamar always runs a comparison of the differences between the data yet to be backed up and the data that has already been backed up.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 24

Avamar compared to BCT

Starting from Oracle 10g, Block Change Tracking (BCT) is provided to speed up database incremental backup. When BCT is active, Oracle uses the Block Change Tracking File (BCTF) book to list the data blocks that have been changed since the last RMAN level 0 backup. RMAN then uses the information from the BCTF in incremental backups to read and back up only those blocks that have been changed. For detailed information, refer to SAP Note 964619.

The following comparison is between the configuration of BACKINT with Avamar, BCT without Avamar, and BCT with Avamar.

Backup Storage Footprint The total data transferred to Avamar with initial and all three subsequent backups is around 81 GB, 66 percent less than the data conducted by incremental backup jobs using BCT only. When BCT works together with Avamar, it requires almost the same space as BACKINT with Avamar.

Performance For initial backup (level 0) to the same SAP system, BCT without Avamar leads to higher network traffic and a longer backup duration over the pure Avamar-based backup, but requires less CPU resources. Incorporating BCT and Avamar brings similar results as BACKINT with Avamar.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 25

• Initial Full Backup

Note The CPU utilization in the scenario of BACKINT backup with Avamar refers to the one that happens on the proxy host.

• Subsequent Incremental Backup

In subsequent incremental backup, BACKINT with Avamar saves much more network bandwidth, but BCT requires much less time to complete the backup. BCT with Avamar offers better performance than BCT only in terms of both average network bandwidth and CPU utilization.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 26

Note The CPU utilization in the scenario of BACKINT backup with Avamar refers to the one that happens on the proxy host. The charts presented above show the average measurement results of three data loads.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 27

As the following chart illustrates, Avamar with BACKINT restores the same SAP system 6.3 times faster than BCT from Avamar.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 28

Conclusion

Introduction This section provides a comprehensive summary to the product benefits and

limitations through comparison of the functions of the key components.

Client-based deduplication by Avamar

Avamar reduces the backup storage footprint significantly by eliminating duplicated data from the backup set. Avamar only transfers new or changed data, based on preselected settings, since the last backup.

Conventional backup through Ethernet requires high network bandwidth to ensure a short backup window. It is ideal to have a dedicated subnet for the backup environment. When sharing network with other applications, both the backup job and the application efficiency will be affected. By using Avamar, administrators can minimize the utilized network bandwidth attributable to backup jobs and share the network with multiple applications instead. However, backup with Avamar causes higher CPU utilization on the client.

Avamar and Oracle BCT

Avamar offers optimized backup performance regarding network utilization and minimizes the backup storage space required for backup contents. However, Oracle BCT offers the ability to complete incremental backups in the least time. When BCT and Avamar are combined, customers can achieve not only data deduplication and average-level network bandwidth, but also a short backup window for incremental backup. However, the restoration time is not acceptable.

Automation of snapshot-based backup using PowerSnap

PowerSnap allows automation of the snapshot/mirror-based backup though NetWorker. In this solution, PowerSnap with the mirror-split mode performs backup to Avamar to eliminate the impact of CPU utilization on the production host when Avamar is introduced. Backing up SAP systems from the proxy client minimizes the performance impact on the SAP production server in terms of CPU, network, and disk I/O.

Centralized management with NetWorker integration

As a central-management backup solution, NetWorker offers seamless integration with SAP, Avamar, and PowerSnap. NMSAP in conjunction with NetWorker allows users to perform a backup to Avamar in the multi-stream mode, which is 2.6 times faster than in a single-stream mode.

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 29

References

White papers For additional information, see the white papers listed below.

• Backing Up Large SAP Oracle Databases Using EMC Avamar and Oracle RMAN - A Detailed Review

• Backup of SAP Oracle Databases in EMC Symmetrix Environments with EMC NetWorker PowerSnap - Applied Technology

Product documentation

For additional information, see the product documents listed below.

• EMC NetWorker Release 7.6 Installation Guide

• EMC NetWorker Release 7.6 Administration Guide

• EMC NetWorker Module for SAP with Oracle Release 4.0 Installation Guide

• EMC NetWorker Module for SAP with Oracle Release 4.0 Administration Guide

• EMC NetWorker PowerSnap Module for Symmetrix DMX Version 2.5 Installation Guide

• EMC NetWorker PowerSnap Module for Symmetrix DMX Version 2.5 Administration Guide

• EMC Avamar 4.1 Server Software Installation Manual

• EMC Avamar Backup Clients 4.1 Installation Guide and Reference Manual

• EMC Avamar 4.1 System Administration Manual

• Master Guide for SAP ERP 6.0

Other documentation

For additional information, see the document listed below.

• SAP Note 964619 - RMAN: Incremental backups with Block Change Tracking

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 30

Additional information

Introduction This section describes the sample scripts used in this white paper.

Baseline - BR*Tools configuration file for RMAN

initPRDrman.sap ************************************************************************************************

backup_mode = full

restore_mode = full

backup_type = online_cons

backup_dev_type = rman_util

backup_root_dir = /oracle/PRD/sapbackup

stage_root_dir = /oracle/PRD/sapbackup

compress = no

archive_function = save_delete

archive_copy_dir = /oracle/PRD/sapbackup

archive_stage_dir = /oracle/PRD/sapbackup

saveset_members = 1

rman_channels = 4

rman_parms = "ENV=(NSR_DEBUG_LEVEL=9, NSR_SERVER=networker, NSR_CLIENT=sapprd, NSR_DATA_VOLUME_POOL=UC334PRDF, NSR_NWPATH=/usr/sbin/)"

util_par_file = /oracle/PRD/102_64/dbs/initPRDrman.utl

************************************************************************************************

initPRDrman.utl ************************************************************************************************

NSR_NWPATH = /usr/sbin/

parallelism = 4

ss_group_by_fs = TRUE

expiration = 3 Months

pool = UC334PRDF

server = networker

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 31

client = sapprd

ssNameFormat = new

NSR_SAVESET_RETENTION = 3 Months

NSR_SAVESET_BROWSE = 3 Months

************************************************************************************************

Baseline - BR*Tools configuration file for PowerSnap

initPRD.sap ************************************************************************************************

backup_mode = full

restore_mode = full

backup_type = online_cons

backup_dev_type = util_file_online

backup_root_dir = /oracle/PRD/sapbackup

stage_root_dir = /oracle/PRD/sapbackup

compress = no

archive_function = save_delete

archive_copy_dir = /oracle/PRD/sapbackup

archive_stage_dir = /oracle/PRD/sapbackup

util_par_file = /oracle/PRD/102_64/dbs/initPRD.utl

************************************************************************************************

initPRD.utl ************************************************************************************************

NSR_NWPATH = /usr/sbin/

parallelism = 4

ss_group_by_fs = TRUE

expiration = 3 Months

pool = UC334PRDF

server = networker

client = sapprd

ssNameFormat = new

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 32

NSR_SAVESET_RETENTION = 3 Months

NSR_SAVESET_BROWSE = 3 Months

ps_backup_mode = TRUE

ps_archive_mode = FALSE

ps_restore_mode = TRUE

ps_inquire_mode = TRUE

ps_exclude_backup_bi_run_nums = 2

ps_exclude_archive_bi_run_nums = 1;2

ps_exclude_backup_paths = /oracle/PRD/102_64/dbs/*;/oracle/PRD/sapreorg/*;/oracle/PRD/sapbackup/*

ps_opaque_pfilename = /oracle/PRD/102_64/dbs/nsrsapps.cfg

************************************************************************************************

nsrsapps.cfg ************************************************************************************************

NSR_DATA_MOVER = sapprx

SYMM_SNAP_REMOTE = FALSE

SYMM_SNAP_TECH = BCV

NSR_PS_DEBUG_LEVEL = 9

SYMM_ON_DELETE = RETAIN_RESOURCE

NSR_SNAP_TYPE = symm-dmx

NSR_PS_SAVE_PARALLELISM = 4

************************************************************************************************

Baseline - NMSAP configuration file for RMAN

nsrsapsvrman.cfg ************************************************************************************************

BR_EXEC = brbackup -p initPRDrman.sap

ORACLE_HOME = /oracle/PRD/102_64

ORACLE_SID = PRD

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 33

NLS_LANG = AMERICAN_AMERICA.US7ASCII

SAPDATA_HOME = /oracle/PRD

SAP_BIN = /sapmnt/PRD/exe

LD_LIBRARY_PATH = /oracle/PRD/102_64/lib:/usr/lib/nsr:/usr/lib

OS_USR_PASSWD=xxxxxxxxxxxxxxxx

************************************************************************************************

Baseline - NMSAP configuration file for PowerSnap

nsrsapsv.cfg ************************************************************************************************

BR_EXEC = brbackup -p initPRD.sap -d util_file_online -r /oracle/PRD/102_64/dbs/initPRD.utl -t online_cons -m all_data -k no -l E

ORACLE_HOME = /oracle/PRD/102_64

ORACLE_SID = PRD

NLS_LANG = AMERICAN_AMERICA.US7ASCII

SAPDATA_HOME = /oracle/PRD

SAP_BIN = /sapmnt/PRD/exe

LD_LIBRARY_PATH = /oracle/PRD/102_64/lib:/usr/lib/nsr:/usr/lib

OS_USR_PASSWD=xxxxxxxxxxxxxxxx

************************************************************************************************

Avamar - BR*Tools configuration file for RMAN

initPRDrmanavt.sap ************************************************************************************************

backup_mode = full

restore_mode = full

backup_type = online_cons

backup_dev_type = rman_util

backup_root_dir = /oracle/PRD/sapbackup

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 34

stage_root_dir = /oracle/PRD/sapbackup

compress = no

archive_function = save_delete

archive_copy_dir = /oracle/PRD/sapbackup

archive_stage_dir = /oracle/PRD/sapbackup

saveset_members = 1

rman_channels = 4

rman_send=(

"'NSR_ENV=(NSR_DEDUP_BACKUP=TRUE,

NSR_DEDUP_NODE=avamar01-u)'",

"channel sbt_1 'NSR_ENV=(NSR_DEDUP_CACHE_TAG=prdavt_1)'",

"channel sbt_2 'NSR_ENV=(NSR_DEDUP_CACHE_TAG=prdavt_2)'",

"channel sbt_3 'NSR_ENV=(NSR_DEDUP_CACHE_TAG=prdavt_3)'",

"channel sbt_4 'NSR_ENV=(NSR_DEDUP_CACHE_TAG=prdavt_4)'"

)

rman_parms = "ENV=(NSR_DEBUG_LEVEL=9, NSR_SERVER=networker, NSR_CLIENT=sapprd, NSR_DATA_VOLUME_POOL=UC334PRDFAVT, NSR_NWPATH=/usr/sbin/)"

util_par_file = /oracle/PRD/102_64/dbs/initPRDrmanavt.utl

************************************************************************************************

initPRDrmanavt.utl ************************************************************************************************

NSR_NWPATH = /usr/sbin/

parallelism = 4

ss_group_by_fs = TRUE

expiration = 3 Months

pool = UC334PRDFAVT

server = networker

client = sapprd

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 35

ssNameFormat = new

NSR_SAVESET_RETENTION = 3 Months

NSR_SAVESET_BROWSE = 3 Months

************************************************************************************************

Avamar - BR*Tools configuration file for PowerSnap

initPRD.sap ************************************************************************************************

backup_mode = full

restore_mode = full

backup_type = online_cons

backup_dev_type = util_file_online

backup_root_dir = /oracle/PRD/sapbackup

stage_root_dir = /oracle/PRD/sapbackup

compress = no

archive_function = save_delete

archive_copy_dir = /oracle/PRD/sapbackup

archive_stage_dir = /oracle/PRD/sapbackup

util_par_file = /oracle/PRD/102_64/dbs/initPRD.utl

************************************************************************************************

initPRDavt.utl ************************************************************************************************

NSR_NWPATH = /usr/sbin/

parallelism = 4

ss_group_by_fs = TRUE

expiration = 3 Months

pool = UC334PRDFAVT

server = networker

client = sapprd

ssNameFormat = new

NSR_SAVESET_RETENTION = 3 Months

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 36

NSR_SAVESET_BROWSE = 3 Months

ps_backup_mode = TRUE

ps_archive_mode = FALSE

ps_restore_mode = TRUE

ps_inquire_mode = TRUE

ps_exclude_backup_bi_run_nums = 2

ps_exclude_archive_bi_run_nums = 1;2

ps_exclude_backup_paths = /oracle/PRD/102_64/dbs/*;/oracle/PRD/sapreorg/*;/oracle/PRD/sapbackup/*

ps_opaque_pfilename = /oracle/PRD/102_64/dbs/nsrsapps.cfg

************************************************************************************************

nsrsapps.cfg ************************************************************************************************

NSR_DATA_MOVER = sapprx

SYMM_SNAP_REMOTE = FALSE

SYMM_SNAP_TECH = BCV

NSR_PS_DEBUG_LEVEL = 9

SYMM_ON_DELETE = RETAIN_RESOURCE

NSR_SNAP_TYPE = symm-dmx

NSR_PS_SAVE_PARALLELISM = 4

Avamar - NMSAP configuration file for RMAN

nsrsapsvrmanavt.cfg ************************************************************************************************

BR_EXEC = brbackup -p initPRDrmanavt.sap

ORACLE_HOME = /oracle/PRD/102_64

ORACLE_SID = PRD

NLS_LANG = AMERICAN_AMERICA.US7ASCII

SAPDATA_HOME = /oracle/PRD

EMC Backup and Recovery for SAP

Enabled by EMC NetWorker Module for SAP with EMC Avamar—A Detailed Review 37

SAP_BIN = /sapmnt/PRD/exe

LD_LIBRARY_PATH = /oracle/PRD/102_64/lib:/usr/lib/nsr:/usr/lib

OS_USR_PASSWD=xxxxxxxxxxxxxxxx

************************************************************************************************

Avamar - NMSAP configuration file for PowerSnap

nsrsapsvavt.cfg ************************************************************************************************

BR_EXEC = brbackup -p initPRD.sap -d util_file_online -r /oracle/PRD/102_64/dbs/initPRDavt.utl -t online_cons -m all_data -k no -l E

ORACLE_HOME = /oracle/PRD/102_64

ORACLE_SID = PRD

NLS_LANG = AMERICAN_AMERICA.US7ASCII

SAPDATA_HOME = /oracle/PRD

SAP_BIN = /sapmnt/PRD/exe

LD_LIBRARY_PATH = /oracle/PRD/102_64/lib:/usr/lib/nsr:/usr/lib

OS_USR_PASSWD=xxxxxxxxxxxxxxxx

************************************************************************************************