SDP-OC-NetApp Filer Installation - V2_2

106
Service Design Package Operation Checklist NetApp Filer Installation & Configuration Project Name Project ID Version 2.2 Author(s) Patrick King Neal Carlino - DELL Document Date October 22nd, 2010 Document Control Number Document File Name SDP-OC-NetApp Filer Installation – V2.2.doc Operation Checklist Template Version 3.0, December 8, 2008 Purpose To define the steps and sequence for a procedure identified in an operational process, and to verify that all steps have been completed. Content Summary

description

SDP-OC-NetApp Filer Installation - V2_2

Transcript of SDP-OC-NetApp Filer Installation - V2_2

5 Operation Checklist Template

Service Design PackageOperation ChecklistNetApp Filer Installation & ConfigurationProject Name

Project ID

Version2.2

Author(s)Patrick KingNeal Carlino - DELL

Document DateOctober 22nd, 2010

Document Control Number

Document File NameSDP-OC-NetApp Filer Installation V2.2.doc

Operation Checklist Template Version 3.0, December 8, 2008PurposeTo define the steps and sequence for a procedure identified in an operational process, and to verify that all steps have been completed. Content Summary

The Operation Checklist template includes usage instructions for creating and publishing checklists, preliminary configuration information and prerequisite requirements, procedural steps, verification activities, and applicable appendices.Contents

41Purpose

42Operation Procedure Identification

43Conventions Used in this Document

54Before You Begin

54.1Enter Configuration Information

54.2If Applicable, Print the Document

65Sign In

76Preliminary Configuration Information

86.1Decide Local or Remote Setup

96.2Naming Standards

106.3Pre-requisites

127Initial Filer Setup

127.1Out of the Box

127.2NetApp Onsite Initial Configuration

138Configuration Tables

138.1Netapp Configuration Worksheet

138.2Virtual Interface Configuration Table

149Data OnTap Configuration

149.1Console Access Methods

149.1.1Local Console access via serial connection

149.1.2Remote Access via Telnet then SSH session

159.2Running setup

269.3Running cifs setup

319.4Check Licenses

329.5Set Timeserver

329.6If Applicable - Anti-virus Configuration

359.7Active/Active Cluster Configuration (clustered filers only)

369.8Hardware Assisted Takeover Configuration (clustered filers only)

379.9Disable WINS on the ACP Ethernet Port

389.10Secure Filer

419.11Security Banner

4310Monitoring Configuration

4310.1AutoSupport Configuration

4410.2Register Filer on NetApp NOW Website

4810.3Apply DataFabric Manager Licenses

4810.4Add Filer to DataFabric Manager / Operations Manager

5010.5Set DataFabric Manager Server Link

5111Storage Configuration

5111.1Assigning Disks

5111.1.1Non-Clustered 2000 Series Disk Layout (No External Shelves)

5111.1.2Clustered 2000 Series Disk Layout (No External Shelves)

5111.2RAID DP

5211.3Spare Disks Standards

5312If Applicable - Data OnTap and Firmware Updates

5513Verify Filer Functionality

5513.1Verify Basic Functionality

6113.2Verify Cluster Configuration

6413.3Verification Sign-off

6514Document History

6515References

67Appendix A Network Connections

73FAS 6000 Series Network Connectivity Dual Switch With EtherChannel (10 Gbit)

74Appendix B NetApp Documentation

75Appendix C Timezones

1 Purpose

This checklist is use to perform Johnson & Johnson standard initial configuration, Data OnTap setup and Active-Active Cluster configuration of NetApp Filers. Procedures to verify functionality of the above are listed. It also describes the approved methodologies to update all firmware, update Data OnTap, create snapshots, and enable quotas and deduplication. It also has an index of links to NetApp documentation to provide background information to the reader. The audience for this document is technologists tasked with building NetApp filers for production use in Johnson & Johnson data centers.2 Operation Procedure Identification

This checklist is used for the following procedures:

Procedure IDProcedure NameDescription

3 Conventions Used in this Document

StyleUsageMS Style Name

BoldUsed for CD-ROM names, command names, options and keywords, including shell script names, system call function names, and facility names. Bold also indicates menu selections and field choices within graphical user interfaces.command-script-gui-field

ItalicUsed to show the system response to an action taken by the user.System Response

underscoreUsed for URLs.hyperlink

courierUsed to show output from commands.computer output

Arial boldUsed to show commands typed by the user.user input

Arial bold italicUsed to show variables within commands or information typed by the user.user input variable

Z:Indicates the CD-ROM drive.

4 Before You Begin

Before you execute or print this checklist, MS Word may prompt you for configuration information that will be recorded in the appropriate places within the document. This section explains how you are to enter information in the required fields.

4.1 Enter Configuration Information

Follow these steps to enter the configuration information for your checklist, if needed (if no configuration is required, delete this section).

1. Open the checklist document. Then press CTRL+A to select the entire document.

2. Press F9.If the Update Table of Contents box is displayed, click on Update entire table. Otherwise, go to the next step. 3. You may be prompted to enter a specific value for a series of standard fields that may be used throughout the document. Each field should initially contain a placeholder value, such as REPLACE THIS TEXT, or a description of the field. When prompted, simply replace the placeholder text with the appropriate value for the field.

4. View the Header and Footer.5. Click the page header, and press CTRL+A to select the entire page header.

6. Press F9 to update all fields in the header.

7. Verify that the correct information is now displayed in the page header. If a field is not updated, right click on the individual field and select Update Field.

8. Close the Header and Footer.4.2 If Applicable, Print the Document

If you will execute this checklist on a printed copy, enter the appropriate information in the fields and print the document for execution and submission with wet signatures.

5 Sign In

Important! Before you begin executing any part of this checklist, you must do the following:

1. If Electronic Execution: Type your information into the table below before executing any checklist steps. Type N/A in the signature column.

2. If Printed Execution: Print your full name on the chart below before executing any checklist steps. Print your name legibly; provide your signature, initials, and the date.

3. Record your initials in the designated space after each procedure you perform.

Full Name (Type or Print)Signature (Type N/A for electronic execution)InitialsDate

6 Preliminary Configuration Information

The following configuration information is required prior to the execution of this checklist. Details for each of these items are found in the body or appendices of this document; this list is meant to serve as a preliminary reminder. Some of these items may require significant lead time, so early planning will help assure a smooth implementation.Preliminary Configuration Requirements

Filer ModelFAS xxxx (i.e. FAS 2000, 3000 or 6000 series) (

Filer Name(s)

1 for single filer, 2 for a clusterConfirm number of last installed filer at location, use standard naming convention described on next page (

Filer Serial #(s)Register filer(s) on the NOW support website (

Filer Registration InformationAll details needed to complete the section Register filer on NetApp NOW Website including address, location and contacts. (

Rack/Floor SpaceEnsure filer has been correctly racked, powered and cabled and no warning indications appear on hardware (

Network ConnectivityEnsure all necessary network drops are run to proper switches. Refer to network administrators to understand port settings, LACP (

Production IP AddressesRequest static IP addresses from Networks team for all public VIFs (

Management IP AddressesRequest static IP addresses from Networks team for all management ports (BMC or RLM) (

DNS ServersProper regional DNS server IP addresses (

DNS Request Email Sent email to [email protected] for all VIFs and management interfaces (BMC or RLM) (

WINSIf applicable - Proper WINS server addresses for the region (

Computer AccountWork with local administration group to create the Active Directory machine account for the filer as per naming convention found in the following section (

Active Directory DomainName of the AD domain where the Filer will reside (

Storage AdministrationDetermine the proper global storage administration AD group to be assigned filer administration rights during the setup process (

Anti-virus ServersIf applicable, coordinate with local team responsible for anti-virus servers to ensure configuration and links to the new filer from multiple anti-virus servers are complete (

TimeserverName of regional timeserver

DFM Account Ensure you have the proper domain service account to allow access for DFM administration (

SNMP StringEnsure you have the proper SNMP string to enter to allow traps to be forwarded (

AV AccountIf applicable, consult with local anti-virus and account administrations teams to determine that the proper domain service account to be used by anti-virus servers is activated (

AutosupportKnow SMTP mail server and Autosupport mail recipients (

Check NetApp NOW WebsiteEnsure all needed licenses are associated with filer serial number (

IQ/OQ Filer ScreenshotFile _filer_iqoq.jpg will be collected for IQ/OQ documentation for each filer or vfiler (

IQ/OQ Cluster ScreenshotFor clustered filers only file _cluster_iqoq.jpg will be collected for IQ/OQ documentation for each cluster (

6.1 Decide Local or Remote Setup

If filer to be configured is local, you have the option to run setup at the console using the following method:

NetApp Console Serial Cable with proper pinout to connect laptop to serial port on filer for console access

Laptop (or PC) with serial port or USB to serial converter to facilitate above connection Use an approved client application to establish a serial connection with the filer If filer to be configured is at a remote location use the following to run setup over the wire: IP address of management port (BMC or RLM) configured onsite by NetApp (assumption is that remote servers have had initial configuration performed by NetApp) Use an SSH client application to establish a secure session to the management port6.2 Naming Standards Filer Name - according to the standard device naming convention below:

ITSxxyyNASzz (Non-clustered)Or

ITSxxyyNASzz01 (Clustered node 1)

ITSxxyyNASzz02 (Clustered node 2)

Where ITS and FS are static;xx=2-digit country code; yy=site codeand zz=01, 02, etc;and n1 or n2 signify node 1 or node 2 of a filer cluster (nothing for a single non-clustered filer)

For example: ITSUSRANAS01n1 for node 1 of the first filer cluster in RaritanITSUSRANAS01n2 for node 2 of the first filer cluster in Raritan

ITSUSRONAS01 for a non-clustered filer in Rogers Arkansas VIF (Virtual Interface) Names - according to the standard device naming convention:

VIF#

Where # is the number of the VIF in the order they were createdFor example: VIF1, VIF2, VIF3 Management Port Name - according to the standard device naming convention:

filernamer

Where filername is as per standard above and r stands for remoteFor example: ITSUSRANAS01N1r for the management port of the first node of the first cluster in Raritan. 6.3 Pre-requisites

The following section lists the hardware and/or software components you must have on hand in order to complete the operation, and any preliminary criteria that must be met outside the scope of this Checklist.

Before you begin, assemble the following resources: All approved versions for software and firmware components are gathered from the Storage Desired State spreadsheet found on the TED Storage Platform Engineering SharePoint link below:

http://ted.jnj.com/Teams/ServerStorage/StoragePlatforms/Shared%20Documents/Storage_Desire_State.xls Proper Data OnTap and Filer firmware versions are downloaded from the J&J Depot as per link below\\na.jnj.com\jnjdfsroot\Windows_Server_Technology\NCS_Standard_Technologies\NAS_TechnologiesBefore you begin, ensure the following criteria are met:StepTaskCheck

1.Filer and disk enclosures are racked and cabled properly for production use (

2.Send memo to [email protected] for public production VIFs or second level VIFs (

3.Send memo to [email protected] for management IP addresses (BMC or RLM) (

4.Redundant network links are available for production data access (VIF pairs) (

5.Management network link is available (two are needed for a cluster) (

6.If applicable,virus scanning servers (either physical or virtual machines) are up and running and if possible connected to the same production subnet as the filer. If a common subnet is unavailable, minimize the number of hops between filer and anti-virus server. (

7.The Netapp Configuration Worksheet has been completed and submitted back to Netapp to facilitate the installation (

8.Access has been granted on the SMTP gateway to allow the routing of Autosupport emails from the Filers

Section Completed By (initials): __________ Date: ___/___/___

7 Initial Filer Setup 7.1 Out of the BoxBefore shipment, the following tasks are performed by NetApp:

The storage system was configured with an aggregate and FlexVol root Bootloader files and firmware updates, including primary and secondary BIOS images, are copied to the boot device that shipped with the system. Licenses for protocols and features (such as CIFS, NFS, SnapVault, and controller failover) purchased by J&J should be installed on your system.

Upon delivery NetApp will connect all cables for the controllers and shelves

NetApp will configure basic remote management network connectivity during initial configuration

7.2 NetApp Onsite Initial Configuration

NetApp will perform the initial setup, cabling and basic network configuration for each filer deployed at J&J based on information contained in the Netapp Configuration Worksheet that is completed by J&J per Filer. Drives will be assigned to provide space for the root volumes and the management module (BMC or RLM depending on model) will be configured. NetApp will run through the basic setup script (Section 9.2 Running Setup) and use information provided by J&J to configure network interfaces, filer name, Autosupport settings, and other basic functions. The filer will be configured to a point where J&J personnel can complete the setup procedures remotely over the network.

The J&J administrator will then take ownership of the filer configuration and will repeat the setup script (Section 9.2) to confirm NetApp initial configuration settings. The setup script shows previous settings and allows the administrator to accept those settings or edit them as needed. After completion of the setup script, the J&J administrator will continue with subsequent sections to configure the high level filer functions.

8 Configuration Tables 8.1 Netapp Configuration WorksheetThe Netapp configuration worksheet is a list of the entries and values you will use during the setup process and is prerequisite to the physical installation performed by Netapp. This worksheet must be sent to the Netapp engineer prior to the physical installation. Please have a copy of this worksheet on-hand so it can be referred to as needed for the following steps.8.2 Virtual Interface Configuration TableNote: Refer to Appendix A and consult with Network administrators to determine correct virtual interface configuration settings including port speed, duplexing, load balancing, and VIF type.Note: Change this table as needed to reflect the Filer configuration

Virtual Interface Configuration Table (VIFs)

Network configuration information

Note: Change this table as neededBasic Interfaces to be teamed to make up the VIF belowe0a

e0b

e0c

e0d

e1a

e1b

e1c

e1d

Virtual Interface (VIF) Name See Naming Standard in Preliminary Configuration Sect.Virtual interface: [Y/N]

YesYesYesYesYesYesYesYesVIF IP addresses: First 3 #s

______._____._____

VIF Netmask:

______._____._____.______

VIF Port Speed: 100tx or 1000tx (do not use auto)

Flow control (set to None)None NoneNoneNoneNoneNoneNoneNoneEnable jumbo frames? (Set to No)No

NoNoNoNo

NoNoNo

Load Balancing (Set to IP)

IP

IPIPIPIPIPIPIPVIF Type (single, multi or LACP)

The following three items are for clustered servers only

Should interface take over a partner IP address during failover? Y/N If yes note cluster partner interface or VIF nameYes

YesYesYesYesYesYesYesCluster partner interface IP address

______._____._____._____

9 Data OnTap Configuration

After the initial filer setup described in the preceding section, Data OnTap may be configured by running the setup program either from the console via a serial connection, or remotely via a telnet or SSH session. Since telnet access is prohibited by J&J security standards, SSH sessions will be used to complete the filer build once network connectivity has been established. It is likely that many Remote Site filers will be configured over the wire since a site visit will not be practical. The two methods of console access are described below.

9.1 Console Access Methods

Access to the filer console to perform the configuration may be accomplished using one of the following methods.9.1.1 Local Console access via serial connection

Console access is possible via the serial port on the back of the filer Serial port settings are 9600, none, 8, 1

Serial cable (shipped with filer, Part # 112-00054) Serial cable may be connected to laptop USB with third party converter Once session is established commands and responses are viewable via the console9.1.2 Remote Access via Telnet then SSH session Telnet access is prohibited by J&J security standards, however SSH must be enabled on the filer during setup. Steps to enable SSH and disable telnet are summarized below and listed in detail in the subsequent checklist. The initial configuration performed by NetApp personnel upon delivery will include basic network connectivity. Also included is configuration of the management device (BMC or RLM) which also allows access to Data OnTap configuration via the console. The bullet list below is a summary of the detailed steps described in the following sections: Obtain the IP address of the management port (BMC or RLM) Establish a telnet session to the filer Perform the setup as described in the following section

Upon completion of setup, use the commands listed to enable SSH

Use an SSH client application to establish a secure SSH session When SSH session is established, disable telnet on the filer as described9.2 Running setupThe following section describes the setup queries and responses as per information gathered in the Netapp Configuration Worksheet referenced in the preceding section. Note: At any time during the setup process press CTRL-C to exit the setup program without saving changesNote: As part of their initial configuration, NetApp personnel will run the setup script on all new filers. Basic parameters such as filer name, network interfaces, and remote management capability will be configured to allow J&J administrators to complete the high-level configuration. The setup script described in this section (Running Setup) will be run by both NetApp and J&J administrators. Subsequent sections will be completed by J&J.

Note: The setup script below is adaptive and certain prompts may or may not appear depending on user responses and whether the filer was previously configured or is being built for the first time. Steps that may or may not appear are shaded in gray like this cell. If the shaded prompt does not appear, proceed to the next step in the process.

StepFiler SetupCheck

1.Establish Either a Serial OR Telnet SessionEstablish a serial console session to the filer by executing the following commands:Connect the serial cable to both serial ports (RJ-45 connector with wrench icon on the filer, DB-9 connector on the laptop/PC). Establish a terminal session using settings serial 9600, none, 8, 1. (OR

Establish a telnet console session to the filer by executing the following commands: Telnet to BMC (2000 series) or RLM (3000 & 6000 series) IP address ( Login to BMC/RLM ( At BMC/RLM prompt type priv set advanced ( At BMC/RLM prompt type system console which brings you to the Data OnTap console ( Login to the Data OnTap console

(Note: When first powered on, filers will search for a DHCP server. Since static IP addresses are standard for J&J servers, press Ctrl-C to end the DHCP search if it is still ongoing. (

2.Note: The very first time a filer boots up, it will start the setup script automatically. Since most filers will have been put through the initial configuration by NetApp personnel, it is expected that setup will have to be started manually as described below. Pressing Enter can accept the settings already configured by Netapp.At Data OnTap console type start the setup process by typing the command:setupYou will be prompted with a warning >Do you want to continue? Type:[yes]

(

3.>Please enter the new hostname. [hostname]

Enter the hostname for the filer according to the J&J standard described in previous section. FQDN is not required at this point. (

4.>Do you want to enable IPv6? [n] (IPv4 is the J&J standard) (

5.>Do you want to configure virtual network interfaces? [y]

Note: The responses to the following prompts depend on the model of filer you are configuring and the number of interfaces. Consult the VIF Configuration Table in previous section for details on how interfaces are paired and configured for each model. (

5.Number of virtual interfaces to configureType [ l ] if only a single VIF will be configuredor

Type [ 3 ] if a second-level VIF will be configured (

6.Name of virtual interface

Note: See Preliminary Configuration Section for Naming Conventions (

7.Is vif a single [s], multi [m] or a lacp [l] virtual interface?

Type [ l ] for LACPOrType [ s ] for Single

LACP assumes the network switches will be configured appropriately. Consult the Basic Network Configuration Table and VIF Configuration Table in previous section for details on how interfaces are paired and configured for each model. If LACP is not supported at the site, choose Single mode. (

8.Is vif to support IP based (i), MAC based (m), Round-robin (r) based or Port based (p)load balancing?Type [ i ] for IP (

9.Number of links for (virtual interface)Type [ 2 ] (

10.Name of link #1 (i.e. e0a)

(

11.Name of link #2 (i.e. e0b)If creating a second-level VIF, repeat the last 6 steps for the remaining two VIFs. Note that the 3rd VIF should use the first 2 VIFs as link #1 & link #2. (

12.>Please enter the IP address for Network Interface Refer to configuration tables and enter the IP address for the interface [nnn.nnn.nnn.nnn] (

13.>Please enter the netmask for Network Interface Refer to configuration tables and enter the netmask for the network [nnn.nnn.nnn.nnn] (

14.Note: This prompt may appear on legacy network equipment. Consult the local network team for the proper response.

>Please enter the subnet prefix length for Network Interface e0a

For example [nn] set as per local network standards (

15.>Should interface e0a take over a partner IP address during failover ?

You can type either y or n at this prompt, refer to configuration tables.

Type [y] or [n] as per below

Then you are...

[y] For clustered systems (controller failover license installed on clustered FAS 2040, 3140, 3170, 6080)Prompted to enter the IP address or interface name to be taken over by e0a. Since the partner is a VIF, you must use the interface name. Note: Interface pairs should be laid out in the VIF Configuration Table as per Appendix A Network Connections

[n] For non-clustered FAS 2040Directed to the next prompt.

(

16.>Please enter media type for vif (100tx-fd, tp-fd, 100tx, tp, auto (10/100/1000)) [enter appropriate local setting]

(

17.>Please enter flow control for e0a {none, receive, send, full} [Enter none]

(

18.>Do you want e0a to support jumbo frames? [n]

(

19.If you are prompted to enter IP addresses for additional network interfaces that are not used, just type Enter. (

20.>Should interface e0M take over a partner IP address during failover? [n]: (

21.>Would you like to continue setup through the Web interface? [n] (

22.>Please enter the name or IP address of the IPv4 default gateway.

Enter the primary gateway that is used to route outbound network traffic. (

23.Enter an Administration host if NFS is to be used, or skip this step by hitting enter

>Please enter the name or IP address for administrative host. The administration host is given root access to the storage system's /etc files for system administration. To allow /etc root access to all NFS clients enter RETURN below. Enter the name of the administrative host

(

24.>Please enter timezone

GMT is the default setting. Select and enter a valid value for your time zone from Appendix C Timezones at the end of this document.

(

25.>Where is the filer located?

Enter the J&J location and complete details including rack number if applicable (

26.>What language will be used for multiprotocol files? [en_US] (

27.If HTTP is licensed, the following prompt will appear

>Enter the root directory for HTTP files [ ]

Enter the root directory for HTTP or accept default (

28.>Do you want to run DNS resolver? [y]You are prompted for the fully qualified DNS domain name and associated nameserver IP address. Enter at least two nameserver IP addresses.

[nnn.nnn.nnn.nnn] (

29.Please enter DNS domain name Enter the proper DNS domain - for example [na.jnj.com] (

30.You may enter up to 3 nameservers

Please enter the IP address for first nameserver [nnn.nnn.nnn.nnn] (

31.Do you want another nameserver? [y] (

32.Please enter the IP address for second nameserver [nnn.nnn.nnn.nnn] (

33.Do you want another nameserver? [n]Note: You may add another nameserver if desired. A minimum of two should be defined. (

34.>Do you want to run NIS client?

[n]

(

35.You may be prompted with the option to disable AutoSupport which should remain enabled.

Press Return key to continue (

36.Note: This step and sub-steps will pop up for FAS 2000 series only. For FAS 3000 and 6000 series skip to the next step to configure the RLM instead.

>Would you like to configure the BMC LAN interface ?

Enter [y] at the prompt and configure BMC as per Filer Configuration TableThe following is an example for using BMC setup with a static IP address:

Would you like to enable DHCP on the BMC LAN interface:

[n] (Please enter the IP address for the BMC: nnn.nnn.nnn.nnn

Enter IP address as per Network Configuration Table (Please enter the netmask for the BMC: nnn.nnn.nnn.nnn

Enter netmask as per Network Configuration Table (Please enter the IP address for the BMC Gateway:

nnn.nnn.nnn.nnn (Please enter the gratuitous ARP interval for the BMC:

(accept default by hitting return) (The mail host is required by your system to enable BMC to send AutoSupport message when filer is down

Please enter the name or IP address of the mail host [mail host ]

Enter the SMTP mail host use for Operational support messages (

(

37.>Would you like to configure the RLM LAN interface ?

Note: This step does not apply and will not pop up for FAS 2000 series.

For FAS 3000 and 6000 series type [y] at the prompt and enter the RLM configuration values as per the Filer Configuration Table.

Enter [y] at the prompt and configure RLM as per Filer Configuration TableThe following is an example of RLM setup with a static IP address: Would you like to enable DHCP on the RLM LAN interface:

[n] (Please enter the IP address for the RLM: nnn.nnn.nnn.nnn

Enter IP address as per Network Configuration Table (Please enter the netmask for the RLM: nnn.nnn.nnn.nnn

Enter netmask as per Network Configuration Table (Please enter the IP address for the RLM Gateway:

nnn.nnn.nnn.nnn (Please enter the gratuitous ARP interval for the RLM:

(accept default by hitting return) (The mail host is required by your system to enable RLM to send AutoSupport message when filer is down

Please enter the name or IP address of the mail host [mail host ]

Enter the SMTP mail host use for Operational support messages (

(

38.>Do you want to configure the Shelf Alternate Control Path Management interface for SAS shelves ?

Some small 2000 series filer configurations do not have external disk shelves and do not require ACP configuration. For those exceptions choose [n], for all other filersSelect [y] (

39.Note: ACP will be set up by NetApp personnel during the initial configuration.

You will see the following prompt: >Set Alternate Control Path interfaceConfirm that the Alternate Control Path interface port is set as per Appendix A Network Configuration

Note: On 2000 series filers there is a dedicated ACP interface port (e0P). For 3000 and 6000 series filers an ACP interface is assigned (i.e. port e0A). Consult Appendix A - Network Configuration for ACP interface standards. (

40.You will see the following prompt: >Set Alternate Control Path domainConfirm that the Alternate Control Path domain is set by accepting NetApp default settings (

41.Please enter the netmask for Network Interface :

nnn.nnn.nnn.nnn

Use the default netmaskNote : Look for the message Ethernet : Link Up to confirm ACP functionality (

42.At the prompt, enter ifconfig a (

43.For each network interface listed in the previous step, enter the following at the command prompt:wrfile a /etc/rc ifconfig interface flowcontrol none

Example: wrfile a /etc/rc ifconfig e0a flowcontrol noneNote that interface = a Filer network interface (

44.To disable the disk auto assign option, type the following command at the filer prompt:

options disk.auto_assign off (

Section Completed By (initials): __________ Date: ___/___/___

9.3 Running cifs setupNote: This section should be skipped for NFS-only filersUpon completion of the setup configuration in the previous section, cifs setup needs to be run on all GFS and other CIFS filers. Note: All filers that have a cifs license loaded should start cifs setup automatically upon completion of setup. All GFS filers may come pre-loaded with a cifs license upon delivery from NetApp.During this section each step displays the cifs setup command prompt. Provide an appropriate response from the information gathered previously in the Preliminary Configuration Requirements table. At any time during cifs setup press CTRL-C to exit without committing changes.StepCIFS SetupCheck

1.To start the CIFS configuration script type the following at the console prompt:

cifs setup (

2.> Do you want to make the system visible via WINS? [y]Note: GFS Filers will use WINS. If WINS is not desired, skip this step.

Follow the prompts to enter the IP addresses of two WINS servers as per the Filer Configuration TableEnter the IP address of the first WINS server nnn.nnn.nnn.nnn (Enter the IP address of the second WINS server nnn.nnn.nnn.nnn (When prompted to enter the IP address for an additional WINS server, hit Return (

(

3.>(1) Multiprotocol filer(2) NTFS-only filer For all GFS and CIFS-only filers choose [2] NTFS-only filer

For filers with a combination of CIFS, NFS or HTTP licenses, choose [1] Multiprotocol filer

(If NTFS-only, the following steps 4 and 5 below will not appear onscreen)

(

4.>Should CIFS create default /etc/passwd and /etc/group files?

Enter [y] here if you have a multiprotocol environment as per previous step. Otherwise, enter [n]

(

5.>Would you like to enable NIS group caching? [n]

(

6.>Enter the password for the root user []Enter the administrative password as per Operations standards (

7.>When the default name of the CIFS server is listed, you see the following prompt:

>Would you like to change this name? [n]Accept the default name

(

8.>Select the style of user authentication appropriate to your environment: (1) Active Directory domain authentication (Active Directory domains only)(2) Windows NT 4 domain authentication (Windows NT or Active Directory domains)(3) Windows Workgroup authentication using the filer's local user accounts(4) /etc/passwd and/or NIS/LDAP authentication

For all GFS and CIFS-only filers choose [1] Active Directory domain authentication

For filers with NFS licenses OR multiprotocol filers, choose [4] /etc/passwd and/or NIS/LDAP authentication

(

9.>What is the name of the Active Directory domain?

Enter the fully qualified domain name of the regional Active Directory domain that the filer will reside in

(

10.>Would you like to configure time services? [y]>Enter the time server host(s) and/or addresses?

Note: This step will appear if Active Directory is not providing time services. If prompted, enter the appropriate NTP server for your region (for example ntp.jnj.com)

(

11.>Enter the name of the Windows user. Enter the password for the domain. Enter the name and password of the Windows account with sufficient privileges to add computers to the Active Directory domain using the format :

domain_name\admin_account ( Enter the password for the above account

(

12.If the subnet you are building the filer on is not accounted for in Active Directory, you are prompted to enter an Active Directory site. Select Option (2) Choose a site name from the list of available sites

to choose the appropriate site. (

13.If you saw the previous prompt you are then prompted with the following:

>Do you want your filer to be a member of this site?

Enter [y]Note: If the prompts in steps 12 and 13 appear, contact the AD team with subnet details for a site to be added to Active Directory (

14.Since a machine account has already been requested and created by the regional domain administration, the following message will appear:

>An account that matches the name already exists in Active Directory. You may continue by using this account or changing the name of the CIFS server

Do you want to re-use this machine account? [y]

Answer yes to use the existing account. After a few seconds you should see the following message in the stream of messages generated onscreen:>Welcome to the Active Directory domain (

15.>Do you want to create the (name of filer) administrator account? [y]

(

16.>Enter the new password for (storage system name). Retype the password.

Enter the administrative password as per Operations standards (

17.>Would you like to specify a user or group that can administer CIFS? [y]Enter the domain and name of the proper Active Directory global storage administration group that will administer the filer. For example: NA\Storage AdminsNote that this group will effectively receive root access.

(

18.After you complete this step, CIFS is configured and the name registrations are complete. You see the following message:

>CIFS local server is running.

(

19.If applicable, enable SMB 2.0 protocol on the filer by typing the following command

options cifs.smb2.enable onNote: Perform this step on all GFS Filers (

20.Remove the Active Directory group Domain Admins from the Local Administrators group on the filer by typing the following command at the console:

useradmin domainuser delete domain\domain admins -g Administrators

where domain is the regional Active Directory domain (i.e. NA)

(

Section Completed By (initials): __________ Date: ___/___/___

9.4 Check LicensesStepTask Check LicensesCheck

1.To check licenses type the following command at the filer prompt

license

Note: If you see unexpected results when checking licenses, go to the NetApp NOW website and search by filer serial number as described in Section 10.4 (

2.If necessary to add a license, type the following command at the filer prompt

license add xxxxxxxwhere xxxxxxx is the license key provided by NetApp (

Section Completed By (initials): __________ Date: ___/___/___

9.5 Set Timeserver

StepSet TimeserverCheck

1.To enable the timeserver option type the following command at the filer prompt:

options timed.enable on (

2.To set the timeserver type the following command at the filer prompt:

options timed.servers nnn.nnn.nnn.nnn

(insert the IP address of the regional timeserver) (

Section Completed By (initials): __________ Date: ___/___/___

9.6 If Applicable - Anti-virus Configuration

Note: Anti-virus servers should be on the same subnet as the filers if possible in order to maximize performance.Note: Anti-virus scanning does not need to be enabled on Filers acting solely as SnapVault secondaries and/or NFS only Filers.

Data ONTAP allows virus-scanning servers running a compliant antivirus application to scan files before a CIFS client is allowed to open it. When you enable the virus-scanning process through Data ONTAP on the storage system, the virus-scanning application tells the storage system to send file scanning requests. The virus-scanning application watches for requests from the storage system. Whenever the types of files you specify are opened or changed on the storage system, Data ONTAP sends the scanning server a request to scan the file. File types scanned are listed below:

??_DL?IM?OFTSMM

ARJDOCINIOLESWF

ASPDOTJS?OV?SYS

BATDRVLZHPIFVBS

BINEMLMD?POTVS?

CABEXEMPPPP?VXD

CDRGMSMPTRARWBK

CL?GZ?MSGRTFWPD

COMHLPMSOSCRXL?

CSCHT?OCXSHSXML

StepAnti-virus Scanning ConfigurationCheck

1.The filer build administrator should work with the local Anti-Virus team to ensure that McAfee anti-virus server is configured and enabled to receive requests and scan files. (

2.Ensure that the anti-virus servers are using a domain service account that is equivalent to Backup Operators level privileges on the storage systems on which they will conduct virus scanning.

useradmin domainuser add login_name g Backup OperatorsExample: useradmin domainuser add NA\user1 -g "Backup Operators"

(

3.On the filer console enter the following command:

vscan on (

4.After installation and configuration of the virus scan software on the server is complete, confirm the success of the installation by listing the IP addresses of the PCs now configured as virus-scanning clients. At the storage system console, enter the following command:

vscan scanners

The system displays a table listing the IP addresses of the active virus-scanning clients for this storage system. Leave the virus-scanning client on and connected to the storage system or storage systems on which it is carrying out its virus scan operations.

1. >vscan scanners

2. Virus scanners(IP and Name) P/S ...

3. ----------------------------------------

4. 132.132.59.12 \\XLAB-WTS1 Primary5. 132.132.59.13 \\XLAB-WTS2 PrimaryIf scanning servers dont appear in list above as expected, contact the anti-virus administrative team to confirm link between AV server and filer is established

(

5.The default setting is that file access is denied if a successful scan cannot be performed, that is, the option is set to On. Reverse this setting to allow access, even if no successful scan can be performed, by setting the option to Off.

Enter the following command

vscan options mandatory_scan off

(

Section Completed By (initials): __________ Date: ___/___/___

9.7 Active/Active Cluster Configuration (clustered filers only)StepCluster ConfigurationCheck

1.Reboot Both Nodes - Ensure that both nodes have been rebooted after completing the setup and cifs setup configurations in the previous two sections. To reboot the nodes type the following command at the consoles of both nodesreboot (

2.Enable cluster failover - by typing the following command at the console cf enable

Note: This step is required on only one node to enable clustering on both nodes (

3.Verify Failover Enabled - Verify that controller failover is enabled by entering the following command at the console of both nodes:

cf status at filer 1 which should generate a response such as >Cluster enabled, filer2 is up.

(

Note: Because some Data ONTAP options need to be the same on both nodes of an active/active cluster, you need to check the following options commands on each node and change them as necessary. The parameters listed in the following Steps 4 through 6 must be the same on both cluster nodes so that takeover is smooth and data serving is transferred between the nodes correctly.

4.Setting up options - NDMP

At both nodes type the command:

options ndmpd.enable on (

5.Setting up options Route Enabled

At both nodes type the command:

routed on (

6.Setting up options - Timezone

At both nodes type the command:

timezone (

Section Completed By (initials): __________ Date: ___/___/___

9.8 Hardware Assisted Takeover Configuration (clustered filers only)Note: Skip this section for 2000 series filers as they do not have this capabilityRequirements for hardware-assisted takeover

The hardware-assisted takeover feature is available only on systems that support Remote LAN Modules (RLMs) which at the time of this writing are the FAS 3000 and 6000 series models. The hardware-assisted takeover option (cf.hw_assist) is enabled by default but is not available on FAS 2000 models.StepHardware Assisted Takeover ConfigurationCheck

1.Enabling the hardware-assisted takeover option (Models 3000 and 6000 series)Hardware-assisted takeover is enabled by default on systems that use an RLM. Enter the following command on both nodes to enable the cf.hw_assist option:options cf.hw_assist.enable on

(

2.Setting the partner address for hardware-assisted takeover

The cf.hw_assist.address option enables you to change the partner address used by the hardware-assisted takeover process on the remote management card. The default is the IP address on the e0a port of the partner.

Enter the following command on both nodes to set the IP address to which the hardware failure notification is sent:options cf.hw_assist.partner.address Enter the IP address of the partner node

(

Section Completed By (initials): __________ Date: ___/___/___

9.9 Disable WINS on the ACP Ethernet Port

StepDisable WINS on the ACP Ethernet PortCheck

1.Disable WINS on the Ethernet port assigned to the ACP bus

For Clustered Filers:

ifconfig exx wins partner exx

For Non-clustered Filers:

ifconfig exx winsWhere xx = the Ethernet port dedicated to the ACP bus.

Note: Typically this would be port e0a on 3100 & 6000 series Filers and always port e0P on 2000 series Filers

(

2.To add the command above to the /etc/rc file, enter the following at the command prompt:

Where xx = the Ethernet port dedicated to the ACP bus.For clustered Filers:

wrfile a /etc/rc ifconfig exx wins partner exx

For Non-clustered Filers:

wrfile a /etc/rc ifconfig exx wins

(

Section Completed By (initials): __________ Date: ___/___/___

9.10 Secure Filer

Telnet access is restricted by J&J IAPP standards. Console access to filers must be accomplished using a secure SSH client as per the following steps: Enable SSH on the filer

Establish an SSH session When SSH session is established, disable telnet on the filerStepSecure FilerCheck

1.To enable SSH Type the following command which enables SSH service for SSH 2.0 clients

secureadmin setup ssh

(

2.Take the default settings by pressing enter to all 3 questions below:

Please enter the size of host key for ssh1.x protocol [768] :

Please enter the size of server key for ssh1.x protocol [512] :

Please enter the size of host keys for ssh2.0 protocol [768] :

(

3.When prompted with Is this correct? Select Yes(

4.To enable SSH Type the following command which enables SSH service for SSH 2.0 clients

secureadmin enable ssh2

(

5To disable SSH1 Type the following command:

secureadmin disable ssh1

(

6.Start SSH Session use a secure SSH client to establish a session with the filer. Note: Simultaneous Telnet and SSH sessions conflict and cannot be established. Halt the Telnet session before connecting via SSH. Since Data OnTap is configured at this point, you can establish an SSH session directly to the filer (not via the BMC or RLM).

Use a secure SSH client to establish a session

If you are prompted to do so, reply Yes to accept certificates

Log in to the filer console (

7.Note: Perform the following step only after you have established an SSH session

Disable Telnet according to security standards disable telnet access to filers by typing the following command:

options telnet.enable off (

8.Setting up options - Disable Dynamic DNS

type the command:options dns.update.enable off (

9.Enable CIFS auditing:

options cifs.audit.enable on

(

10.Setup SSL:

secureadmin setup ssl (

11.Enter the 2-digit country code (

12.Enter the State or Province Name (full name) (

13.Enter the Locality Name (city, town, etc.) (

14.Enter the Organization Name (company) [Your Company]: Johnson &Johnson (

15.Enter the Organization Unit Name (division): ITSS (

16.For Common Name, enter the host name of the Filer and hit Enter. (

17.For Administrator Email, enter the SMTP mailbox used to monitor the Filers. (

18.For Days until expires, take the default and hit Enter. (

19.For Key length (bits), type 2048 hit Enter.At this point, a new SSL certificate has been created. (

20.At the prompt, type the following command to force FilerView to use HTTPS:

options httpd.admin.enable off (

21.At the prompt, type the following command to force password rules for administrators:

options security.passwd.rules.everyone on (

22.At the prompt, type the following command to limit local password retries:

options security.passwd.lockout.numtries 5 (

23.At the prompt, type the following command to disabke RSH access:

options rsh.enable off

(

24.At the prompt, type the following command to set the audit log size to 200MB:

options cifs.audit.logsize 209715200 (

25.At the prompt, type the following command to monitor management events:

options cifs.audit.account_mgmt_events.enable on (

Section Completed By (initials): __________ Date: ___/___/___

9.11 Security Banner

StepSecurity BannerCheck

1.Copy the following banner text into memory on your workstation (i.e. Right click > Copy on a Windows workstation)WARNING NOTICE: You are about to enter a Private Network that is intended for the authorized use of a Private Company and its affiliate companies (the Company) for business purposes only. The actual or attempted unauthorized access, use or modification of this network is prohibited by the Company. Unauthorized users and/or unauthorized use are subject to Company disciplinary proceedings and/or criminal and civil penalties in accordance with applicable domestic and foreign laws. The use of this system may be monitored and recorded for administrative and security reasons. If such monitoring and/or recording reveals possible evidence of criminal activity, the Company may provide the monitored evidence of such activity to law enforcement officials.

(

2.At the filer console type the following command:

wrfile /etc/motd

This will open the motd (message of the day) file in edit mode (

3.Right click in the filer prompt window to paste the text above to the motd file:

Press the Enter key (

4.Press CNTRL-C to exit and save (

5.Type the following command at the filer prompt to verify the file was written:

rdfile /etc/motd

(

Section Completed By (initials): __________ Date: ___/___/___

10 Monitoring Configuration

This section describes how to configure Autosupport and add the newly built filer to the NetApp Operations Manager console to enable ongoing monitoring and health checks. 10.1 AutoSupport Configuration

StepAutoSupport ConfigurationCheck

1.The https:// AutoSupport address should be configured at the factory

Use the following command to set the transport to http options autosupport.support.transport https (

2.Ensure that AutoSupport has been enabled by typing the following command

options autosupport.enable on (

3.Set AutoSupport to send alerts to the appropriate J&J support groups by options autosupport.mailhost

For smtp server enter the appropriate regional Operations smtp server address to receive alerts from the filer (

4.Ensure that AutoSupport email to J&J Operations staff is configured by typing the following command:

options autosupport.to where is the email account monitored by regional Operations staff. (

Section Completed By (initials): __________ Date: ___/___/___

10.2 Register Filer on NetApp NOW Website

Go to the NetApp NOW website and register the filers by serial number. Be sure to list all other pertinent details to streamline future support and troubleshooting efforts

StepRegister FilerCheck

1.Open your web browser and go to the following url:

http://www.netapp.com (

2.At the top of the screen choose Login Support

Use your NOW credentials to login in the pane on the right

Note: All J&J administrators should receive NOW accounts with Premium access level (

3.You are brought to the Service and Support page.

On the right side MyNOWLinks pane choose My Products.

On the right side Related Links pane choose Register My Products (

4.Enter the Serial Number of the filer(s) to be registered. The detailed information screen on the following page will pop up.

Enter all pertinent details to ensure proper future support of the filer by regional NetApp personnel. Technicians will report to and needed parts will be shipped to the location according to the data entered in the fields below

5.Product Registration Product Order Details

Serial Number

Additional Serial Numbers

(Examples: cluster partner serial numbers, other serial numbers from your order, client-based software, etc.)

End User Organization*

[]

Website URL of Organization *

[]

Date Purchased

[01\/]- [Jan\/]- [1996\/]

Sales Order Number[]

Purchased From*

[- Select One -\/]

Company Name *

[] (Company that sold the product(s) to you if it was not NetApp)

Physical Address of Product

Is this product located at a Third Party/hosted location?*

()Yes ()No

Name of Third Party(if applicable) []

Address*

[](Where the product is installed)

[]

[]

City*

[]

State/Province

ZIP*

[]

Country*

Primary Contact to Receive Support

Who should receive AutoSupport notifications and renew support contracts?

First Name*

[]

Office Phone*

[]

Last Name*

[]

Email Address*

[]

NOW Login/Username

[] (For record validation purposes only.)

Parts Receiving Address

Parts should be shipped to:*() Same as the product's physical address above () Different from the product's physical address above(Complete the address form below.) Organization

[]

Address

[](40 character limit per address line)

[]

[]

City

[]

State/Province

[

ZIP

[]

Country

Parts Receiving Hours:*

() [Monday\/]through [Friday\/] , [8\/]am - [5\/]pm()24 x 7()Other:[]

Parts Receiving Contact

Parts should be shipped to the attention of:*() Same as the primary contact above () Different from Primary contact above (Fill in form below with contact information.)First Name

[]

Office Phone

[]

Last Name

[]

Email Address

[]

NOW Login/Username

[] (For record validation purposes only)

Service Report-To Address

The service engineer should arrive:*() Same as the product's physical address above() Same as the Parts Receiving address above() Different from either address above.(Complete the address form below.)Organization

[]

Address

[](40 character limit per address line)

[]

[]

City

[]

State/Province

ZIP

[]

Country

Service Report-To Hours(for engineer arrival):*

() [Monday\/]through [Friday\/] , [8\/]am - [5\/]pm()24 x 7()Other: []

Service Report-To Contact

Upon arriving the service engineer will ask for:*() Same as the Primary contact above () Same as the Parts receiving contact above () Different from either contact above. (Fill in form below with contact information.)First Name

[]

Office Phone

[]

Last Name

[]

Email Address

[]

NOW Login/Username

[] (For record validation purposes only.)

(

6.Go to the NOW web page described below to obtain the Operations Manager, Protection Manager and Provisioning Manager license keys for your filer.

On the NOW site go to Service & Support > My Products > My NetApp Products > My Licenses for Serial Numbers and click Accept at the bottom of the license agreement. Enter the serial number of the filer and click GoNote the license keys here for the following features:

Operations Manager for all J&J filers

In addition Protection Manager and Provisioning Manager are required for all GFS filers

DFM_OPSMGR_AD _________________________________________________DFM_PROTMGR_AD _________________________________________________DFM_PROVMGR_AD _________________________________________________ These keys will be applied in the next section. (

Section Completed By (initials): __________ Date: ___/___/___

10.3 Apply DataFabric Manager Licenses

StepApply DataFabric Manager LicensesCheck

1.Get license keys from the NOW website as described in previous section

(

2.From your workstation Login to the DFM server by opening your browser to

https://dfm_server:8443where dfm_server is the FQDN or IP address of the regional DataFabric Manager server (

3.Under Control Center > Setup > Options click on Licensed Features (

4.Enter the license keys for each of the required features and click UpdateAdd Operations Manager for all J&J filersAdd Protection Manager and Provisioning Manager for all GFS filers

(

5.Check the Licensed Features list on the left side of the pane to ensure that the license key was accepted (

Section Completed By (initials): __________ Date: ___/___/___

10.4 Add Filer to DataFabric Manager / Operations Manager

StepAdd Filer to DFMCheck

1.Add the DFM service account to the local Administrators group on the Filer by issuing the following command

useradmin domainuser add domain\login_name g AdministratorsExample: useradmin domainuser add NA\SA_DFM-acct -g "Administrators"

(

2.Set Filer SNMP stringType the following command at the console to set the SNMP community

snmp community add ro

where string is the proper regional SNMP string (

3.It is recommended to remove the public SNMP community by typing the following command

snmp community delete ro public (

4.Set SNMP Traphost

snmp traphost add

where DFM server is the fully qualified domain name of the regional DFM/Operations Manager server or cluster (

5.From your workstation Login to the DFM server by opening your browser to

https://dfm_ip_address:8443where dfm_ip_address is the IP of the regional Data Fabric Manager server (

6.Select Login in the upper right corner and enter the name and password for the DFM administrative account (

7.Click Global in top left pane (

8.Click Member Details tab in bottom left pane (

9.Check the List of Appliances in the in the center pane to see if the filer was auto-discovered. This will occur if filer resides on the same subnet as the DFM server. If not, force discovery as described in the next step. (

10.In the New Storage System at the bottom of the list, type the fully qualified domain name of the filer and press AddCheck that the filer is added to the list of appliances. (This may take several minutes.)Note : If required, add the filer to the appropriate group using the Add To Group button in the lower left hand corner (

11.Click on filer in the Member Details paneUnder Storage Controller Tools in lowest left hand pane (scroll down) choose Edit Settings (

12.Enter proper Login and Password field information for the DFM service account (used in Step 1 of this section) using format domain\account. For example na\sa_dfm. Click Update in lower right hand corner (

13.After receiving confirmation that the credentials in step above have been accepted:Fill in the Remote Platform Management IP Address (BMC for 2000 series, RLM for 3000 and 6000 series). (

14.Change the Login Protocol setting to be SSH. (

15.Change the Administration Transport setting to be HTTPS. (

16.Click Update (

Section Completed By (initials): __________ Date: ___/___/___

10.5 Set DataFabric Manager Server LinkStepSet DataFabric Manager Server LinkCheck

1.Establish a FilerView session and log in to the filer by entering the following url in your workstation web browser:

https:///na_admin

where is either the fully qualified name or the short name of the storage system or the IP address of the storage system (

2.Click on Filer View > Operations Manager >Access on the left-hand side

(

3.Check the Enable box in the Operations Manager Links section

(

4.Enter the fully qualified domain name of the DFM Server in the DataFabric Manager server field

(

5.In the DataFabric Manager Protocol field, select HTTPS

(

6.In the DataFabric Manager Port field, select 8443

(

7.Click Apply (

Section Completed By (initials): __________ Date: ___/___/___

11 Storage Configuration11.1 Assigning Disks

Note: NetApp personnel will assign disks during the initial configuration process before handoff to J&J.In order to balance workload and optimize performance of the filer cluster, usually stacks/shelves/disks are divided equally between the controllers when possible. When multiple stacks are present, it is recommended to assign each stack to a specific controller. When a single stack is present, it is recommended to assign each shelf to a specific controller. The exception to this is the scenario where only a single / no external shelf is present. In which case the assignment must be per disk. 11.1.1 Non-Clustered 2000 Series Disk Layout (No External Shelves)One exception to the standard explained above is the smallest model FAS 2000 series non-clustered with only the 12 internal SAS drives. This configuration will be used in the smallest offices and will be configured as a non-clustered filer. The 12 internal disks will be assigned in the following fashion:

DataRAID-DPDataRAID-DPDataRAID-DPSpareDataRAID-DPDataRAID-DPDataRAID-DPDataRAID-DPDataRAID-DPDataRAID-DPDataRAID-DPDataRAID-DP

11.1.2 Clustered 2000 Series Disk Layout (No External Shelves)Another exception to the standard explained above is the smallest model FAS 2000 series with only the 12 internal SAS drives. This configuration will be used in the smallest offices and will be configured as active/passive cluster due to the limited number of disks. The 12 internal disks will be split in the following fashion: Note that the passive 2040 controller will use RAID-4 to optimize storage.Active Node 1RAID-DPActive Node 1RAID-DPActive Node 1RAID-DPActive Node 1RAID-DPPassive Node 2

RAID-4Passive Node 2

RAID-4Active Node 1SparePassive Node 2

SpareActive Node 1RAID-DPActive Node 1RAID-DPActive Node 1RAID-DPActive Node 1RAID-DP

11.2 RAID DPThe Johnson & Johnson standard for NetApp filer RAID configuration is RAID DP (with one exception above), which stands for double parity. With RAID-DP, you can use larger RAID groups because they offer more protection. A RAID-DP group is more reliable than a RAID4 group that is half its size, even though a RAID-DP group has twice as many disks. Thus, the RAID-DP group provides better reliability. With RAID-DP, the RAID groups can withstand the failure of two drives without interruption to service. Data ONTAP classifies disks as one of four types for RAID: data, hot spare, parity, or dParity. The

RAID disk type is determined by how RAID is using a disk. A data disk holds data stored on behalf of clients within RAID groups (and any data generated about the state of the storage system as a result of a malfunction). A hot spare disk does not hold usable data, but is available to be added to a RAID group in an aggregate. Any functioning disk that is not assigned to an aggregate but is assigned to a system functions as a hot spare disk.

A parity disk stores data reconstruction information within RAID groups.

A dParity disk stores double-parity information within RAID groups, if RAID-DP is enabled.Within each aggregate, RAID groups are named rg0, rg1, rg2, and so on in order of their creation. You cannot specify the names of RAID groups. Disk typeMinimum group sizeMaximum group sizeDefault group size

ATA or SATA31614

FC or SAS32816

Note: Use the default group size for the type of disk where possible. 11.3 Spare Disks StandardsSpare disks are used to replace failed disks to prevent loss of data availability. The tables below list the approved disk sparing standards. Configure the filer to match the number of recommended spares.Number of Disks Recommended Spares

28 2

84 2

112 3

168 3

336 4

504 6

1,008 12

12 If Applicable - Data OnTap and Firmware Updates

NetApp should pre-load the proper firmware versions prior to shipping the filer. To check the firmware levels use the version command at the console. If versions are incorrect use the instructions in this section which describe the procedures to update system and disk firmware for all filer models.

Approved versions for software and firmware components may be found in the Storage Desired State spreadsheet found on the TED Storage Platform Engineering SharePoint link below:

http://ted.jnj.com/Teams/ServerStorage/StoragePlatforms/Shared%20Documents/Storage_Desire_State.xls Proper filer Data OnTap versions may be downloaded from the J&J Depot at the link below:

\\na.jnj.com\jnjdfsroot\Windows_Server_Technology\NCS_Standard_Technologies\NAS_Technologies\Data_ONTAPNote: This is procedure will disrupt user connectivity and is meant for new filer builds only

StepData OnTap and Firmware UpdatesCheck

1.To determine the current version of Data OnTap, type the following command at the filer:

version

Note: If the filer is at the correct version as per the Storage Desired State spreadsheet (see link above), skip this section and proceed to the next section (

2.Copy the Data OnTap .exe image to the /etc/software directory on the filer(s) to be updated. From a workstation or server map a drive to the filers C$ share and drill down to the etc\software directory. Copy the appropriate Data OnTap .exe file to the etc\software directory

(i.e. 732_setup_q.exe) (

3.To disable cluster failover type the following command at the console:

cf disableSkip this step if not performing this process on a cluster (

4.To update the version of Data OnTap use the following console command:

software update file_name -rWhere file_name is the name of the .exe image you copied to the /etc/software directory (

5.You will see a number of messages and progress dots. After a few minutes look for the message Operator requested download completed (

6.Enter the following command at the console

halt (

7.After the system shuts down the LOADER> prompt appears. Depending on whether the filer firmware is up to the revision that was downloaded as part of the .exe file, you may or may not see the following message:

Warning: The CompactFlash contains newer firmware.

If you see this message perform Step 8 If you do not see this message skip Step 8

(

8.At the LOADER> prompt type the following command

update_flash (

9.When complete type the following command to reboot the system

bye (

10.Repeat the above process (Steps 1 through 8) for the second node if filer is clustered (most models are clustered) (

11.For clustered filers only:

When both filers are completely updated and rebooted type the following command at one of the filer consoles to re-enable clustering

cf enable (

Section Completed By (initials): __________ Date: ___/___/___

13 Verify Filer Functionality

Follow these steps to verify the operational functionality of the filer.

13.1 Verify Basic Functionality

StepVerify Basic FunctionalityCheck

1.Verify Hostname Resolution - Ping the filer hostname from a workstation on another subnet to check name resolution (

2.Verify licenses: enter the following command at both nodes

license

Confirm that all needed licenses are loaded

(

3.Verify BMC If configuring a 2000 series filer, use this procedure to verify that the Baseboard Management Controller is set up correctly and connected to the network.

The BMC network interface is not used for serving data, so it does not show up in the output for the ifconfig command.

Sub-Step

Verify Baseboard Management Controller

Check

A.

To verify that AutoSupport is enabled and AutoSupport options are valid, enter the following command:

options autosupport The AutoSupport options should be set as follows:

autosupport.enable on

autosupport.support.enable on

autosupport.mailhost name or IP address of mailhostautosupport.content complete

Note: The BMC does not rely on the storage system's autosupport.support.transport option to send notifications. The BMC uses the Simple Mail Transport Protocol (SMTP).

(B..

Enter the following command to verify that the BMC's network configuration is correct or to display the MAC address of the BMC:

bmc status

(

(

4.Verify RLM If configuring a 3000 or 6000 series filer, use this procedure to verify that the remote LAN module (RLM) is set up correctly and connected to the network.

The RLM network interface is not used for serving data, so it does not show up in the output for the ifconfig command.

Sub-Step

Verify RLM

Check

A.

To verify that AutoSupport is enabled and AutoSupport options are valid, enter the following command:

options autosupport The AutoSupport options should be set as follows:

autosupport.enable on

autosupport.support.enable on

autosupport.mailhost name or IP address of mailhostautosupport.support.to name or email address of alert recipients

autosupport.content complete

(B..

Enter the following command to verify the configuration of the RLM interface:

rlm status

Note: It might take a few minutes for the new network settings for the RLM to take effect.

(

(

5.Verify ACP - If DS4243 disk shelves are connected to your system, use this procedure to verify that Shelf Alternate Control Path Management is set up correctly and connected to a local LAN.

To verify that ACP is enabled, enter the following command:

storage show acp

You should see command output similar to the following:

Alternate Control Path: enabled

Ethernet Interface: e0b

ACP Status: Active

ACP IP address: 198.15.1.212

ACP domain: 198.15.1.0

ACP netmask: 255.255.255.0

ACP Connectivity Status: Full Connectivity

Shelf Module Reset Cnt IP address FW Version Status

---------------------------------------------------------------------

7a.001.A 002 198.15.1.145 01.05 active

7a.001.B 003 198.15.1.146 01.05 active

7c.002.A 000 198.15.1.206 01.05 active

7c.002.B 001 198.15.1.204 01.05 active

(

6.If clustered, verify cluster Takeover/Giveback - From one node enter the following command:cf takeoverCluster will fail over to opposite node

(

7.To determine the failover status, type the following command at the filer prompt:

cf status

After takeover completes, test basic functionality; ensure all VIFs have failed over successfully and filer is viable

An example of the output from the cf status command is shown below:

itsusragfsn02a(takeover)> cf status

itsusragfsn02a has taken over itsusragfsn02b.

itsusragfsn02b is ready for givebackTakeover due to negotiated failover, reason: operator initiated cf takeover

(

8.Enter the following command to reverse the takeover:

cf giveback

(

9.Determine the state of the giveback by typing the following command:

cf status

An example of the output from the cf status command is shown below:

itsusragfsn02a> cf status

Cluster enabled, itsusragfsn02b is up

10.Establish a FilerView session with the filer by entering the following url in your workstation web browser:

https:///na_adminwhere is either the fully qualified name or the short name of the storage system or the IP address of the storage system (

11.Capture a screen shot of FilerView for the filer showing an all green status for IQ/OQ evidence as per example below. Save the file with the following naming format: _filer_iqoq.jpg

(

Section Completed By (initials): __________ Date: ___/___/___

13.2 Verify Cluster ConfigurationNote: If the filer is non-clustered, skip this section and proceed with the next sectionNote: This section assumes both nodes in the cluster have been built up to this section.

When you configure active/active systems, the following configuration information needs to be the same on both systems:

Options Parameters

Network interfaces

Configuration files

Licenses

The process described below will check critical configuration settings on both nodes of a cluster pair and identify discrepancies. All discovered issues must be resolved and the test re-run until both nodes are identical. StepRun Cluster Configuration Checker UtilityCheck

1.Open your workstations browser to https://:8443Where dfm server is the FQDN or IP address or the regional DataFabric Manager server.

Login with an account with administrative credentials. (

2.Under the Control Center tab click Globaland click on one node of the cluster to be checked. (

3.In the bottom left hand Storage Controller Tools pane select View Cluster Console

(

4.Next to one of the Filers, click on the arrow next to the Select A Tool to Run drop down box

(

5.Select Check Cluster Configuration (

6..Repair any inconsistencies discovered by the utility using commands described in previous sections

Note: Errors that can be detected by the Check Cluster Configuration utility: Services licensed not identical on partner (some services may be unavailable on takeover)

Options settings not identical on partner (some options may end up changed on takeover)

Network interfaces configured wrong (clients will disconnect during takeover)

FCP cfmode settings not identical on storage controllers with FCP licensed

Checks /etc/rc on each storage controller to see that all interfaces have a failover set

(

7.Capture a screen shot of the Cluster Configuration Checker Utility showing all green status for IQ/OQ evidence as per example below: Save the file with the following naming format: _cluster_iqoq.jpg

(

Section Completed By (initials): __________ Date: ___/___/___

13.3 Verification Sign-off

When all verification steps in this section have been completed satisfactorily, Verification Results (Check Below)

Passed:Failed:

Comments: (if none, check N/A) N/A FORMCHECKBOX ____________________________________________________________________________________

____________________________________________________________________________________

____________________________________________________________________________________

Completed by (Signature): _______ _________________________ ___ Date: ___/___/___

14 Document History

This section reflects the document history of this template.

Revision No.ContactAction and Status at Action DateAction Date

DRAFT 1-38Patrick King,

Neal CarlinoInitial draft sent to TED, Networks, ST for reviewJan 10 2010

1.0Patrick King,

Neal CarlinoPublishedFebruary 24 2010

1.1Patrick KingRemoved uneccessary sections per Operational feedbackMay 5 2010

1.2Patrick KingAdded steps per Security AssesmentAugust 25 2010

1.3Patrick KingAdded steps per Security AssesmentSeptember 7 2010

1.4Patrick KingAdded steps per Security AssesmentSeptember 8 2010

1.5Patrick KingChanges made per DFM 4.xSeptember 17 2010

1.6Patrick KingAdditional CIFS auditing changesSeptember 29 2010

1.7Patrick KingChanges per Netapp InstallationSeptember 30 2010

1.8Patrick KingAdded SMTP access prerequisiteOctober 6 2010

1.9Patrick KingUpdated the VIF naming standardsOctober 14 2010

2.0Patrick KingUpdated with Operations Feedback on ACP/WINS configurationOctober 19 2010

2.1Patrick KingUpdated with Operations Feedback on ACP Failover settingsOctober 21 2010

2.2Patrick KingUpdated with 2048 bit SSL certficateOctober 22 2010

15 References

Document NameDocumentNumberFile Name/Location

SDP-TD-GFS.docxV1.0TED SharePoint

JnJ BIS Gen2 NetApp POC Test Plan Checklist

V1.0TED SharePoint

SDP GFS Requirements Specification v0.57.docx

V0,57TED SharePoint

GFS Vision/Scope Document V0.3 TED SharePoint

Appendix A Network Connections

The diagrams below illustrate how filers will be connected to network switches.Standards for network connectivity for each filer model are described for each model below. The means of connectivity at a particular site will depend on the network infrastructure and switch configuration at that site. The diagrams below depict the ideal means of connections if all needed network switch infrastructure is available.

Note: Some filers may be shipped with single-port 10Gbit Ethernet cards. Some sites may not have 10Gbit Ethernet capability and these ports are reserved for future use.

FAS 2000 Series Network Connectivity Example One Switch (1 Gbit)

ControllerInterfaceVIFModeA/A or A/PLevel

Controller Ae0aVIF1SingleActive/PassiveFirst

Controller Ae0cVIF1SingleActive/PassiveFirst

Controller Ae0PACP

Controller Be0aVIF1SingleActive/PassiveFirst

Controller Be0cVIF1SingleActive/PassiveFirst

Controller Be0PACP

FAS 2000 Series Network Connectivity Two Switches Without EtherChannel (1Gbit)

ControllerInterfaceVIFModeA/A or A/PLevel

Controller Ae0aVIF1SingleActive/PassiveFirst

Controller Ae0cVIF1SingleActive/PassiveFirst

Controller Ae0bVIF2SingleActive/PassiveFirst

Controller Ae0dVIF2SingleActive/PassiveFirst

Controller AVIF1&VIF2VIF3SingleActive/PassiveSecond

Controller Ae0PACP

Controller Be0aVIF1SingleActive/PassiveFirst

Controller Be0cVIF1SingleActive/PassiveFirst

Controller Be0bVIF2SingleActive/PassiveFirst

Controller Be0dVIF2SingleActive/PassiveFirst

Controller BVIF1&VIF2VIF3SingleActive/PassiveSecond

Controller Be0PACP

FAS 2000 Series Network Connectivity Two Switches With EtherChannel (1Gbit)

ControllerInterfaceVIFModeA/A or A/PLevel

Controller Ae0aVIF1LACPActive/ActiveFirst

Controller Ae0cVIF1LACPActive/ActiveFirst

Controller Ae0bVIF2LACPActive/ActiveFirst

Controller Ae0dVIF2LACPActive/ActiveFirst

Controller AVIF1&VIF2VIF3SingleActive/PassiveSecond

Controller Ae0PACP

Controller Be0aVIF1LACPActive/ActiveFirst

Controller Be0cVIF1LACPActive/ActiveFirst

Controller Be0bVIF2LACPActive/ActiveFirst

Controller Be0dVIF2LACPActive/ActiveFirst

Controller BVIF1&VIF2VIF3SingleActive/PassiveSecond

Controller Be0PACP

FAS 3000 Series Network Connectivity Two Switches Without EtherChannel (1Gbit)

ControllerInterfaceVIFModeA/A or A/PLevel

Controller Ae0aVIF1SingleActive/PassiveFirst

Controller Ae1aVIF1SingleActive/PassiveFirst

Controller Ae0bVIF2SingleActive/PassiveFirst

Controller Ae1bVIF2SingleActive/PassiveFirst

Controller AVIF1&VIF2VIF3SingleActive/PassiveSecond

Controller Ae1cACP

Controller Be0aVIF1SingleActive/PassiveFirst

Controller Be1aVIF1SingleActive/PassiveFirst

Controller Be0bVIF2SingleActive/PassiveFirst

Controller Be1bVIF2SingleActive/PassiveFirst

Controller BVIF1&VIF2VIF3SingleActive/PassiveSecond

Controller Be1cACP

FAS 3000 Series Network Connectivity Two Switches With EtherChannel (1Gbit)

.ControllerInterfaceVIFModeA/A or A/PLevel

Controller Ae0aVIF1LACPActive/ActiveFirst

Controller Ae1aVIF1LACPActive/ActiveFirst

Controller Ae0bVIF2LACPActive/ActiveFirst

Controller Ae1bVIF2LACPActive/ActiveFirst

Controller AVIF1&VIF2VIF3SingleActive/PassiveSecond

Controller Ae1cACP

Controller Be0aVIF1LACPActive/ActiveFirst

Controller Be1aVIF1LACPActive/ActiveFirst

Controller Be0bVIF2LACPActive/ActiveFirst

Controller Be1bVIF2LACPActive/ActiveFirst

Controller BVIF1&VIF2VIF3SingleActive/PassiveSecond

Controller Be1cACP

FAS 3000 Series Network Connectivity Two Switches With EtherChannel (10 Gbit)

ControllerInterfaceVIFModeA/A or A/PLevel

Controller AE1aVIF1LACPActive/ActiveFirst

Controller AE2aVIF1LACPActive/ActiveFirst

Controller AE1bVIF2LACPActive/ActiveFirst

Controller AE2bVIF2LACPActive/ActiveFirst

Controller AVIF1&VIF2VIF3SingleActive/PassiveSecond

Controller AE0aACP

Controller BE1aVIF1LACPActive/ActiveFirst

Controller BE2aVIF1LACPActive/ActiveFirst

Controller BE1bVIF2LACPActive/ActiveFirst

Controller BE2bVIF2LACPActive/ActiveFirst

Controller BVIF1&VIF2VIF3SingleActive/PassiveSecond

Controller BE0aACP

FAS 6000 Series Network Connectivity Dual Switch With EtherChannel (10 Gbit)

ControllerInterfaceVIFModeA/A or A/PLevel

Controller AE1aVIF1LACPActive/ActiveFirst

Controller AE2aVIF1LACPActive/ActiveFirst

Controller AE1bVIF2LACPActive/ActiveFirst

Controller AE2bVIF2LACPActive/ActiveFirst

Controller AVIF1&VIF2VIF3SingleActive/PassiveSecond

Controller AE0aACP

Controller BE1aVIF1LACPActive/ActiveFirst

Controller BE2aVIF1LACPActive/ActiveFirst

Controller BE1bVIF2LACPActive/ActiveFirst

Controller BE2bVIF2LACPActive/ActiveFirst

Controller BVIF1&VIF2VIF3SingleActive/PassiveSecond

Controller BE0aACP

Appendix B NetApp Documentation NetApp Documentation is available on their NOW site, which requires an account and password to access. Since J&J is arranging for Support Premium service, full access accounts are available to all administrators.Data OnTap System Administration Guide Filer access & admin, Autosupport, BMC, RLM, performance

http://now.netapp.com/NOW/knowledge/docs/ontap/rel732_vs/pdfs/ontap/sysadmin.pdfStorage Mangement Guide RAID groups, parity, spare discs, aggregates, flexvols, qtrees, quotashttp://now.netapp.com/NOW/knowledge/docs/ontap/rel732_vs/pdfs/ontap/smg.pdfActive-Active Configuration Guide cluster setup background informationhttp://now.netapp.com/NOW/knowledge/docs/ontap/rel732_vs/pdfs/ontap/aaconfig.pdfFirmware Updates details for system, disk, disk shelf, BMC and RLM firmware updateshttp://now.netapp.com/NOW/knowledge/docs/ontap/rel732/html/ontap/upgrade/GUID-B376C9C6-44F0-4D36-A6C1-0383CF4338C3.htmlUpgrade Advisor describes how to use the online NOW tool to plan for Data OnTap upgrade

http://now.netapp.com/NOW/knowledge/docs/ontap/rel732/html/ontap/upgrade/GUID-9194A9CF-F7F8-48B5-82B0-D494D3D64105.htmlAppendix C Timezones

Tables in these sections list valid time zones grouped by geographical region, in alphabetical order.

Africa

Africa/Abidjan Africa/Djibouti Africa/Maputo

Africa/Accra Africa/Douala Africa/Maseru

Africa/Addis_Ababa Africa/Freetown Africa/Mbabane

Africa/Algiers Africa/Gaborone Africa/Mogadishu

Africa/Asmera Africa/Harare Africa/Monrovia

Africa/Bamako Africa/Johannesburg Africa/Nairobi

Africa/Bangui Africa/Kampala Africa/Ndjamena

Africa/Banjul Africa/Khartoum Africa/Niamey

Africa/Bissau Africa/Kigali Africa/Nouakchott

Africa/Blantyre Africa/Kinshasa Africa/Ouagadougou

Africa/Brazzaville Africa/Lagos Africa/Porto-Novo

Africa/Bujumbura Africa/Libreville Africa/Sao_Tome

Africa/Cairo Africa/Lome Africa/Timbuktu

Africa/Casablanca Africa/Luanda Africa/Tripoli

Africa/Conakry Africa/Lumumbashi Africa/Tunis

Africa/Dakar Africa/Lusaka Africa/Windhoek

Africa/Dar_es_Salaam Africa/Malabo

America

America/Adak America/Grenada America/Noronha

America/Anchorage America/Guadeloupe America/Panama

America/Anguilla America/Guatemala America/Pangnirtung

America/Antigua America/Guayaquil America/Paramaribo

America/Aruba America/Guyana America/Phoenix

America/Asuncion America/Halifax America/Port_of_Spain

America/Atka America/Havana America/Port-au-Prince

America/Barbados America/Indiana America/Porto_Acre

America/Belize America/Indianapolis America/Puerto_Rico

America/Bogota America/Inuvik America/Rainy_River

America/Boise America/Iqaluit America/Rankin_Inlet

America/Buenos_Aires America/Jamaica America/Regina

America/Caracas America/Jujuy America/Rosario

America/Catamarca America/Juneau America/Santiago

America/Cayenne America/Knox_IN America/Santo_Domingo

America/Cayman America/La_Paz America/Sao_Paulo

America/Chicago America/Lima America/Scoresbysund

America/Cordoba America/Los_Angeles America/Shiprock

America/Costa_Rica America/Louisville America/St_Johns

America/Cuiaba America/Maceio America/St_Kitts

America/Curacao America/Managua America/St_Lucia

America/Dawson America/Manaus America/St_Thomas

America/Dawson_Creek America/Martinique America/St_Vincent

America/Denver America/Mazatlan America/Swift_Current

America/Detroit America/Mendoza America/Tegucigalpa

America/Dominica America/Menominee America/Thule

America/Edmonton America/Mexico_City America/Thunder_Bay

America/El_Salvador America/Miquelon America/Tijuana

America/Ensenada America/Montevideo America/Tortola

America/Fort_Wayne America/Montreal America/Vancouver

America/Fortaleza America/Montserrat America/Virgin

America/Glace_Bay America/Nassau America/Whitehorse

America/Godthab America/New_York America/Winnipeg

America/Goose_Bay America/Nipigon America/Yakutat

America/Grand_Turk America/Nome America/Yellowknife

Asia

Asia/Aden Asia/Irkutsk Asia/Qatar

Asia/Alma-Ata Asia/Ishigaki Asia/Rangoon

Asia/Amman Asia/Istanbul Asia/Riyadh

Asia/Anadyr Asia/Jakarta Asia/Saigon

Asia/Aqtau Asia/Jayapura Asia/Seoul

Asia/Aqtobe Asia/Jerusalem Asia/Shanghai

Asia/Ashkhabad Asia/Kabul Asia/Singapore

Asia/Baghdad Asia/Kamchatka Asia/Taipei

Asia/Bahrain Asia/Karachi Asia/Tashkent

Asia/Baku Asia/Kashgar Asia/Tbilisi

Asia/Bangkok Asia/Katmandu Asia/Tehran

Asia/Beirut Asia/Krasnoyarsk Asia/Tel_Aviv

Asia/Bishkek Asia/Kuala_Lumpur Asia/Thimbu

Asia/Brunei Asia/Kuching Asia/Tokyo

Asia/Calcutta Asia/Kuwait Asia/Ujung_Pandang

Asia/Chungking Asia/Macao Asia/Ulan_Bator

Asia/Colombo Asia/Magadan Asia/Urumqi

Asia/Dacca Asia/Manila Asia/Vientiane

Asia/Damascus Asia/Muscat Asia/Vladivostok

Asia/Dubai Asia/Nicosia Asia/Yakutsk

Asia/Dushanbe Asia/Novosibirsk Asia/Yekaterinburg

Asia/Gaza Asia/Omsk Asia/Yerevan

Asia/Harbin Asia/Phnom_Penh

Asia/Hong_Kong Asia/Pyongyang

Atlantic

Atlantic/Azores Atlantic/Faeroe Atlantic/South_Georgia

Atlantic/Bermuda Atlantic/Jan_Mayen Atlantic/St_Helena

Atlantic/Canary Atlantic/Madeira Atlantic/Stanley

Atlantic/Cape_Verde Atlantic/Reykjavik

Australia

Australia/ACT Australia/LHI Australia/Queensland

Australia/Adelaide Australia/Lindeman Australia/South

Australia/Brisbane Australia/Lord Howe Australia/Sydney

Australia/Broken_Hill Australia/Melbourne Australia/Tasmania

Australia/Canberra Australia/NSW Australia/Victoria

Australia/Darwin Australia/North Australia/West

Australia/Hobart Australia/Perth Australia/Yancowinna

Brazil

Brazil/Acre Brazil/East

Brazil/DeNoronha Brazil/West

Canada

Canada/Atlantic Canada/Eastern Canada/Pacific

Canada/Central Canada/Mountain Canada/Saskatchewan

Canada/East- Saskatchewan Canada/Newfoundland Canada/Yukon

Chile

Chile/Continental Chile/EasterIsland

Europe

Europe/Amsterdam Europe/Kiev Europe/San_Marino

Europe/Andorra Europe/Kuybyshev Europe/Sarajevo

Europe/Athens Europe/Lisbon Europe/Simferopol

Europe/Belfast Europe/Ljubljana Europe/Skopje

Europe/Belgrade Europe/London (BST) Europe/Sofia

Europe/Berlin Europe/Luxembourg Europe/Stockholm

Europe