Managing SRDF using Solutions Enabler SYMCLI - 1.doc

EMC® Customer Symmetrix® Procedure Generator

T o p i c

Managing SRDF using Solutions Enabler SYMCLI

S e l e c t i o n sSelect a task: Managing SRDF/ASelect task(s): Starting a session

Procedure Generated: 06/09/2013 12:47:01

IMPORTANT: To be sure you have the latest procedure generator application and content, always choose Yes when asked if you want to update the application or repository.

SYMCLI and SYMAPI normally capture significant events and command actions to a daily log file.

UNIX: /var/symapi/log/symapi-yyymmdd.log

Windows: C:\Program Files\EMC\Symapi\log\symapi-yyyymmdd.log

REPORT PROBLEMS

If you find any errors in this procedure or have comments regarding this application, send email to Procedure [email protected]

Copyright© 2010 – 2023 EMC Corporation. All rights reserved.

Publication Date: September, 2013

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date regulatory document for your product line, go to the Technical Documentation and Advisories section on EMC Powerlink.

For the most up-to-date listing of EMC trademarks, see the list of EMC Corporation Trademarks on EMC.com.

All other trademarks used herein are the property of their respective owners.

EMC CONFIDENTIAL version: 3.2.0

1 of 12

http://www.emc.com/legal/emc-corporation-trademarks.htm

mailto:Procedure%[email protected]?subject=Avamar%20Procedure%20Generator


Contents

Starting an SRDF/A session.....................................................................................3


2 of 12


srdfaStart_sym001

Starting an SRDF/A sessionThis procedure describes the steps necessary to implement and start a single source SRDF/Asynchronous mode (SRDF/A) session using static (non-dynamic) Symmetrix devices. The instructions use the Solutions Enabler SYMCLI command line interface to manage devices by a device group. The user should be familiar with the concepts of SRDF/A, and have some experience using Solutions Enabler SYMCLI commands before attempting this procedure.

This procedure is based on content from the following EMC manuals: EMC Solutions Enabler Symmetrix SRDF Family CLI Product Guide SRDF/A and SRDF/A Multi-Session Consistency on UNIX and Windows EMC

Engineering Solutions Guide EMC Solutions Enabler Symmetrix Array Management CLI Product Guide. EMC Solutions Enabler TimeFinder Family CLI Product Guide

These documents can be obtained from the EMC Powerlink website at:http://Powerlink.EMC.comSRDF Brief Summary and Overview of This Procedure

In SRDF/A, the Symmetrix array provides a dependent-write consistent point-in-time image on the target (R2) device, which is a short period of time behind the source (R1) device. Managed in sessions, SRDF/A transfers data in predefined timed cycles or delta sets to ensure that data at the remote (R2) site is dependent-write consistent. This mode requires an SRDF/A license.

The steps outlined in this procedure must be followed in order and are explained in three separate phases as described below:

Three Phase Implementation

Phase I ─ Initial steps for setting up a single source SRDF/A replication session

Identify potential source and target devices on the Symmetrix arrays Identify or create a device group to manage the SRDF/A session Verify device group details

Phase II ─ Transitional steps to perform before session activation

Change the SRDF mode to adaptive copy disk mode Perform a full establish on the SRDF pairs Query device group status

Phase III ─ Activate the SRDF/A Configuration Verify the number of invalid tracks Change the SRDF mode to asynchronous Enable and verify consistency protection for SRDF/A mode

1. Identify source (R1) and target (R2) devices on the Symmetrix arraysIn this step all components in the replication environment are identified, and application details including the number and size of devices are determined.

Using the example provided, determine the Symmetrix device types for the candidate R1 devices and identify candidate R2 devices on the target Symmetrix.


3 of 12

http://Powerlink.EMC.com/


Devices contain the application data and are used as sources and targets for the SRDF/A functionality. In circumstances where an application is already on the Symmetrix array but no device groups have been created, Symmetrix devices need to be identified by using your host-specific LVM commands instead of SYMCLI commands.

There are several SYMCLI commands that can be used to identify SRDF/A devices, some of these include: symrdf list, symdev list and sympd list. For extensive information on how obtain device information, refer to the EMC Solutions Enabler Symmetrix Array Management CLI Product Guide. A few examples are provided here in brief. It is the user’s responsibility to properly identify the devices to be used in the SRDF/A session for this procedure.

To identify candidate source (R1) and target (R2) devices for SRDF/A, you can use the symrdf list command with the –rdfa option as shown below:

symrdf list –rdfa

Symmetrix ID: 000000006163

Local Device View----------------------------------------------------------------------- STATUS MODES RDF S T A T E SSym RDF --------- ----- R1 Inv R2 Inv --------------------Dev RDev Typ:G SA RA LNK MDA Tracks Tracks Dev RDev Pair---- ---- ------ --------- ----- ------- ------- --- ---- -----------00F2 00E6 R1:3 RW RW RW S.. 0 0 RW WD Synchronized00F3 00E7 R1:3 RW RW RW S.. 0 0 RW WD Synchronized00F4 00E8 R1:3 RW RW RW S.. 0 0 RW WD Synchronized00F5 00E9 R1:3 RW RW RW S.. 0 0 RW WD Synchronized00F6 00EA R1:3 RW RW RW S.. 0 0 RW WD Synchronized00F7 00EB R1:3 RW RW RW S.. 0 0 RW WD Synchronized00F8 00EC R1:3 RW RW RW S.. 0 0 RW WD Synchronized00F9 00ED R1:3 RW RW RW S.. 0 0 RW WD Synchronized00FA 00EE R1:3 RW RW RW S.. 0 0 RW WD Synchronized00FB 00EF R1:3 RW RW RW S.. 0 0 RW WD Synchronized00FC 0104 R1:3 RW RW RW S.. 0 0 RW WD Synchronized00FD 0105 R1:3 RW RW RW S.. 0 0 RW WD Synchronized00FE 0106 R1:3 RW RW RW S.. 0 0 RW WD Synchronized00FF 0107 R1:3 RW RW RW S.. 0 0 RW WD Synchronized0100 0108 R1:3 RW RW RW S.. 0 0 RW WD Synchronized0101 0109 R1:3 RW RW RW S.. 0 0 RW WD SynchronizedTotal -------- -------- Track(s) 0 0 MB(s) 0.0 0.0

Legend for MODES:

M(ode of Operation):A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy D(omino) : X = Enabled, . = DisabledA(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off

The above output shows that RDF (RA) group 3 has been configured to provide SRDF/A operations. Devices in this type of RDF group must be either R1 or R2 devices. You can not mix device types in the same group.

Before creating a device group and adding devices to it, examine the devices on the local Symmetrix array to determine which ones are source devices (SymDev), which are remote target devices (RDev), and whether a device is an R1 or an R2 type device. The symrdf list command displays this information, as well as other relevant data, such as RDF group (G),


4 of 12


replication method (column M), pair state, invalid tracks, and the state of each device and the RDF links that connect them. The ellipsis (…) represents truncated output.

symrdf list


Local Device View------------------------------------------------------------------------- STATUS MODES RDF S T A T E SSym RDF --------- ----- R1 Inv R2 Inv ----------------------Dev RDev Typ:G SA RA LNK MDA Tracks Tracks Dev RDev Pair---- ---- ------ --------- ----- ------- ------- --- ---- -------------…………………………………………………………………………………………………………………………………………………………………………………………………0045 0045 R2:2 RW WD NR S.. 0 49500 WD RW Suspended0046 0046 R2:2 ?? WD NR S.. 0 33000 WD RW Suspended0047 0047 R2:2 ?? WD NR S.. 0 0 WD RW Suspended009C 0054 R1:2 RW RW RW S.. 0 0 RW NR Synchronized009D 0055 R1:2 RW RW RW S.. 0 0 RW NR Synchronized009E 0056 R1:2 RW RW RW S.. 0 0 RW NR Synchronized009F 0057 R1:2 RW RW RW S.. 0 0 RW NR Synchronized00A0 0058 R1:2 RW RW RW A.W 0 0 RW NR Synchronized00A1 0059 R1:2 RW RW RW A.W 0 0 RW NR Synchronized00A2 005A R1:2 RW RW RW A.W 0 0 RW NR Synchronized00A3 005B R1:2 RW RW RW A.W 0 0 RW NR Synchronized...The symdev list command with the –r1 option displays all R1 devices. Those R1 devices that are not already part of a device group are displayed as N/Grp’d, which means they are available to be added to a new RDF1 device group.

symdev list –r1


Device Name Directors Device---------------------------- ------------ -------------------------------

CapSym Physical SA :P DA :IT Config Attribute Sts (MB)---------------------------- ------------ -----------------------------

009C /dev/rdsk/emcpower84c 16B:1 01A:C0 RDF1 N/Grp'd RW 516009D /dev/rdsk/emcpower85c 16B:1 02B:D3 RDF1 N/Grp'd RW 516009E /dev/rdsk/emcpower90c 16B:1 02A:C0 RDF1 N/Grp'd RW 516009F /dev/rdsk/emcpower91c 16B:1 01B:D3 RDF1 N/Grp'd RW 51600A0 /dev/rdsk/emcpower92c 16B:1 01B:C0 RDF1 N/Grp'd RW 51600A1 /dev/rdsk/emcpower93c 16B:1 02A:D3 RDF1 Grp'd RW 51600A2 /dev/rdsk/emcpower94c 16B:1 02B:C0 RDF1 N/Grp'd RW 516...

Using the above SYMCLI examples, identify and create a list of the SRDF devices that will be added to your device group for the SRDF/A session.

2. Identify or create a device group to manage the SRDF/A session and add the devices

The symdg list command with the Symmetrix ID number can be used to identify existing device groups on a particular Symmetrix array.

symdg list –sid 3264

After identifying SRDF-capable devices and selecting the required number needed, devices


5 of 12


are added into a Symmetrix device group. The SRDF/A session created using this procedure will be managed by a single device group, rather than individual devices.

For example, to create a device group named pocsun100data using devices 00e1 through 00e4 on Symmetrix 000187461501, enter the following commands:

symdg create pocsun100data

symld –g pocsun100data addall dev –range 00e1:00e4 –sid 502

Note: For additional information on how to display and manage device groups using the symdg command, refer to the EMC Solutions Enabler Symmetrix Array Management CLI Product Guide.Using the previous example, identify or create a device group to manage your SRDF/A session and add the devices you selected in the previous step.

3. Verify device group detailsAfter creating the device group, confirm and verify the device group details by using the symdg list and show commands.

symdg list

D E V I C E G R O U P S

Number of Name Type Valid Symmetrix ID Devs GKs BCVs VDEVs pocsun100data RDF1 Yes 000187461502 4 0 0 0

You can display detailed information about a device group using the symdg show command.

symdg show pocsun100data

Group Name: pocsun100data

Group Type : RDF1 Device Group in GNS : Yes Valid : Yes Symmetrix ID : 000187461502 Group Creation Time : Sat Mar 18 21:57:12 2006 Vendor ID : EMC Corp Application ID : SMC

Number of STD Devices in Group : 4 Number of Associated GK's : 0 Number of Locally-associated BCV's : 0 Number of Locally-associated VDEV's : 0 Number of Remotely-associated VDEV's(STD RDF): 0 Number of Remotely-associated BCV's (STD RDF): 0 Number of Remotely-associated BCV's (BCV RDF): 0 Number of Remotely-assoc'd RBCV's (RBCV RDF) : 0

Standard (STD) Devices (4): { --------------------------------------------------------------- Sym Cap LdevName PdevName Dev Att. Sts (MB) ---------------------------------------------------------------- DEV001 N/A 00E1 RW 4315 DEV002 N/A 00E2 RW 4315 DEV003 N/A 00E3 RW 4315


6 of 12


DEV004 N/A 00E4 RW 4315}

The previous example identifies devices that are used by the application. Always make sure this information matches the information provided by your application administrator.

4. Change the SRDF mode to adaptive copy disk modeNow that you have identified your SRDF pairs and added them to a device group, the SRDF mode should be changed to adaptive copy disk mode (acp_disk). This transitional step allows you to copy the local data across the SRDF link to the remote site as efficiently as possible. If you allowed the SRDF establish to start in synchronous mode, you could create serious I/O latency issues for the host application.

In adaptive copy disk mode, the host does not have any I/O performance impact. This is the added benefit of using acp_disk mode. By specifying acp_disk rather than adaptive copy write pending (acp_wp), you do not waste valuable cache resources on the local Symmetrix array during the SRDF establish. This further reduces the risk of performance impact if there is production occurring on the local Symmetrix array.

To set the SRDF mode to acp_disk, use the symrdf set mode command. For example, set the mode for the pocsun100data device group as follows:

symrdf -g pocsun100data set mode acp_disk –nop

An RDF Set 'ACp Disk Mode ON' operation execution is inprogress for device group 'pocsun100data'. Please wait...The RDF Set 'ACp Disk Mode ON' operation successfully executedfor device group 'pocsun100data'.

After setting the SRDF mode, run a symrdf query command against the device group to verify that it is in adaptive copy disk mode.

symrdf -g pocsun100data query

Device Group (DG) Name : pocsun100dataDG's Type : RDF1DG's Symmetrix ID : 000187461502

Source (R1) View Target (R2) View MODES-------------------------------- ------------------------ ----- ------ ST LI STStandard A N ALogical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF PairDevice Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE-------------------------------- -- ------------------------ ----- ------DEV001 00E1 RW 0 138090 NR 00E1 WD 0 0 C.D SuspendedDEV002 00E2 RW 0 138090 NR 00E2 WD 0 0 C.D SuspendedDEV003 00E3 RW 0 138090 NR 00E3 WD 0 0 C.D SuspendedDEV004 00E4 RW 0 138090 NR 00E4 WD 0 0 C.D SuspendedTotal -------- -------- -------- -------- Track(s) 0 552360 0 0 MB(s) 0.0 17261.3 0.0 0.0

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy D(omino) : X = Enabled, . = Disabled


7 of 12


A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off

This example shows the RDF pair state is still suspended. The operation has changed only the mode from synchronous to acp_disk (C.D). When you set the SRDF mode to adaptive copy disk, the Symmetrix array acknowledges all writes to source (R1) devices as if they were local devices. New data accumulates on the source (R1) device and is marked by the source (R1) side as invalid tracks until it is subsequently transferred to the target (R2) device. The remote director transfers each write to the target (R2) device whenever link paths become available. In addition, tracks are read from the physical disk for transfer to the R2.

5. Perform a full establish on the SRDF pairsOnce the device groups are in adaptive copy disk mode, start the initial full SRDF establish for each device group. The establish operation can take a long time. It is best to launch this during nonpeak hours to minimize traffic over the RDF link.

symrdf -g pocsun100data establish -full –nop

An RDF 'Full Establish' operation execution isin progress for device group 'pocsun100data'. Please wait... Suspend RDF link(s).......................................Done. Merge device track tables between source and target.......Started. Devices: 00E1-00E4 ...................................... Merged. Merge device track tables between source and target.......Done. Resume RDF link(s)........................................Started. Resume RDF link(s)........................................Done.The RDF 'Full Establish' operation successfully initiated fordevice group 'pocsun100data'.

The establish operation begins by merging the track tables between the R1 and R2 devices. It then enables the RDF link (makes it enabled for read and write requests) and begins the copy of data from the source device to the device which was designated as invalid. In this case the data flow is from the R1 device to the R2 device. Again, this is because the example specified the invalidate R2 option when it created the pairing. Had it specified invalidate R1, it would be overwriting the R1 data from the R2.

6. Query device group statusPeriodically, query the progress of the initial SRDF establish. Check to make sure the amount of invalid tracks owed from the R1 to the R2 is decreasing. This value is visible in the R2 Inv Tracks column.

symrdf -g pocsun100data query


Source (R1) View Target (R2) View MODES-------------------------------- ------------------------ ----- ----------- ST LI STStandard A N ALogical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF PairDevice Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE-------------------------------- -- ------------------------ ----- ----------


8 of 12


DEV001 00E1 RW 0 113166 RW 00E1 WD 0 0 C.D SyncInProgDEV002 00E2 RW 0 110506 RW 00E2 WD 0 0 C.D SyncInProgDEV003 00E3 RW 0 114995 RW 00E3 WD 0 0 C.D SyncInProgDEV004 00E4 RW 0 114888 RW 00E4 WD 0 0 C.D SyncInProgTotal -------- -------- -------- -------- Track(s) 0 453555 0 0 MB(s) 0.0 14173.6 0.0 0.0

Synchronization rate : 33.0 MB/SEstimated time to completion : 00:07:10

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy D(omino) : X = Enabled, . = Disabled A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off

Periodically examine the output from the symrdf query command and verify the following:

The R1 device is now read and write enabled.

The R2 invalid tracks have been calculated and are now being copied. Verify that this value is greater than zero and that the number of tracks are decreasing as the establish continues.

The link is now ready (read and write enabled).The above conditions mean that the SRDF pairs are synchronizing (SyncInProg) as shown in the above example. Once the invalid R2 tracks go to zero, the RDF pair state changes to Synchronized.

7. Document the SRDF pairing information in a configuration worksheet

The SRDF pairing information created for this procedure should be documented in a configuration worksheet. The recorded information may be necessary in the event of a failure and should be made available to the system administrator to keep on record.

8. Verify that the number of invalid tracks is below 30,000IMPORTANT

It could be a dangerous situation if SRDF/A mode is activated with a large number of invalid tracks. It is a best practice to switch into SRDF/A mode once the total amount of invalid tracks has fallen below 30,000. Activating SRDF/A with a large number of invalid tracks owed to the R2 side could overload the DA, flood the RDF link, and flood the R1 cache. All of these scenarios could cause the SRDF/A session to drop again. At a minimum, it will have a negative effect on cycle time and could cause other active SRDF/A sessions in the Symmetrix array to experience elongated cycle times.

Before switching into SRDF/A mode, you should remain in adaptive copy disk mode until the invalid track count falls below 30,000, or one cycle set of data. This way you greatly minimize your risk of flooding the cache or your link.

Using the example shown below, enter the symrdf query command for your device group to determine the invalid track count.

symrdf -g pocsun100data query -i 5

Device Group (DG) Name : pocsun100data


9 of 12


DG's Type : RDF1DG's Symmetrix ID : 000187461502

Source (R1) View Target (R2) View MODES-------------------------------- ------------------ ----- ----------- ST LI STStandard A N ALogical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF PairDevice Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE-------------------------------- -- ------------------- ---- -----------

DEV001 00E1 RW 0 25891 RW 00E1 WD 0 0 C.D SyncInProgDEV002 00E2 RW 0 0 RW 00E2 WD 0 0 C.D SynchronizedDEV003 00E3 RW 0 0 RW 00E3 WD 0 0 C.D SynchronizedDEV004 00E4 RW 0 0 RW 00E4 WD 0 0 C.D Synchronized

Total -------- -------- -------- -------- Track(s) 0 25891 0 0 MB(s) 0.0 809.1 0.0 0.0

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy D(omino) : X = Enabled, . = Disabled A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off

The R2 invalid track columns in this example show that the amount of invalid tracks has fallen below 30,000. Device group pocsun100data has 25,891 invalid tracks. This means that it takes just one SRDF/A cycle switch to make the RDF pairs consistent for both device groups.

9. Change the SRDF mode to asynchronousTo change to asynchronous mode, use the symrdf set mode async command.

symrdf -g pocsun100data set mode async –nop

An RDF Set 'Asynchronous Mode' operation execution is in progress for device group 'pocsun100data'. Please wait... The RDF Set 'Asynchronous Mode' operation successfully executed for device group 'pocsun100data'.

10.Enable and verify consistency protection for SRDF/A modeEnable SRDF/A consistency protection by disabling tolerance mode. Tolerance mode allows a device that is not ready on the link to be added to an active SRDF/A group without dropping the session. When tolerance mode is on, if a device that is already part of the SRDF/A group becomes NR on the link, the SRDF/A group will not drop. This breaks the guarantee of R2 consistency until tolerance mode is turned off again.

To enable SRDF/A consistency protection, use the symrdf enable command.

symrdf -g pocsun100data enable –nop

An RDF 'Enable' operation execution is in progress for device group 'pocsun100data'. Please wait... The RDF 'Enable' operation successfully executed for device group 'pocsun100data'.

Consistency mode takes one full RDFA cycle to activate.


10 of 12


To verify that consistency protection is enabled and the RDF status is consistent, query the status by issuing the symrdf query command with –rdfa option (which provides specific details about the SRDF/A session for the device group).

symrdf -g pocsun100data query –rdfa


RDFA Session Number : 34RDFA Cycle Number : 4RDFA Session Status : ActiveRDFA Minimum Cycle Time : 00:00:30RDFA Avg Cycle Time : 00:00:29Duration of Last cycle : 00:00:29RDFA Session Priority : 33Tracks not Committed to the R2 Side: 0Time that R2 is behind R1 : 00:00:47RDFA R1 Side Percent Cache In Use : 0RDFA R2 Side Percent Cache In Use : 0

Source (R1) View Target (R2) View MODES-------------------------------- ------------------ ----- ----------- ST LI STStandard A N ALogical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF PairDevice Dev E Tracks Tracks S Dev E Tracks Tracks MDAC STATE-------------------------------- -- ------------------- ----- -----------

DEV001 00E1 RW 0 0 RW 00E1 WD 0 0 A..X ConsistentDEV002 00E2 RW 0 0 RW 00E2 WD 0 0 A..X ConsistentDEV003 00E3 RW 0 0 RW 00E3 WD 0 0 A..X ConsistentDEV004 00E4 RW 0 0 RW 00E4 WD 0 0 A..X Consistent

Total -------- -------- -------- -------- Track(s) 0 0 0 0 MB(s) 0.0 0.0 0.0 0.0

Legend for MODES: M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy D(omino) : X = Enabled, . = Disabled A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off C(onsistency State): X = Enabled, . = Disabled, - = N/A

This example represents a device group running in normal SRDF/A operations. In the example, the Symmetrix has assigned an SRDF/A session number of 34 with a session priority of 33. The SRDF/A session status for device group pocsun100data is active. If you performed the same query on a device group that was in an RDF mode other than asynchronous, this value would say inactive because there would be no running SRDF/A session associated with it.

Other values to examine in your output are the cycle time statistics. Here you can get the configurable parameter minimum cycle time, average cycle time, and the actual duration time of the last cycle. The example device group shown is using a minimum cycle time of 30 seconds. This is the default value for SRDF/A. Recalling that the R2 devices are two cycle sets behind the R1, you would assume that the R2 devices should be approximately behind the R1 device by 60 seconds. The output can verify this in the “Time that R2 is behind R1” value.


11 of 12


These values are snapshots and should be considered approximate numbers.

If you look at the current SRDF modes, you can see that the example session is operating in asynchronous mode (A) and that RDFA consistency protection is enabled (X) by the output of “A..X”. You can also see that the RDF pair state is consistent. The consistent state signifies the normal state of operation for device pairs operating in asynchronous mode.

Table of Contents


12 of 12

Managing SRDF using Solutions Enabler SYMCLI - 1.doc

Documents

Transcript of Managing SRDF using Solutions Enabler SYMCLI - 1.doc