Network Design and Configuration Guide -...

FlexFrame®

Version 5.3A/1.0A

Network Design and Configuration Guide

Edition September 2013 Document Version 1.0

Fujitsu Limited

© Copyright Fujitsu Technology Solutions 2013

FlexFrame® and PRIMERGY™ are trademarks or registered trademarks of Fujitsu Limited in

Japan and other countries.

SAP® and NetWeaver™ are trademarks or registered trademarks of SAP AG in Germany

and in several other countries

Linux® is a registered trademark of Linus Torvalds

SUSE® Linux is a registered trademark of Novell, Inc., in the United States and other coun-

tries

Oracle™ and Java™ are trademarks of ORACLE Corporation and/or its affiliates

Intel® and PXE® are registered trademarks of Intel Corporation in the United States and oth-

er countries

MaxDB® is a registered trademark of MySQL AB, Sweden

MySQL® is a registered trademark of MySQL AB, Sweden

NetApp® and the Network Appliance® logo are registered trademarks and Network Appli-

ance™ and Data ONTAP™ are trademarks of NetApp, Inc. in the U.S. and other countries.

EMC®, CLARiiON®, Symmetrix® and Celerra™ are trademarks or registered trademarks of

EMC Corporation in the United States and other countries

VMware®, ESX®, ESXi, VMware vCenter, VMware vSphere are registered trademarks or

trademarks of VMware, Inc. in the United States and/or other jurisdictions.

Ethernet® is a registered trademark of XEROX, Inc., Digital Equipment Corporation and Intel

Corporation

Windows® and Word® are registered trademarks of Microsoft Corporation

All other hardware and software names used are trademarks of their respective companies.

All rights, including rights of translation, reproduction by printing, copying or similar methods,

in part or in whole, are reserved.

Offenders will be liable for damages.

All rights, including rights created by patent grant or registration of a utility model or design,

are reserved.

Delivery subject to availability. Right of technical modification reserved.

Network Design and Configuration Guide

Contents

General product information and naming ...................................................................... 1

1 Introduction ..................................................................................................... 2 1.1 Purpose of this Document ................................................................................. 2 1.2 Notational Conventions ..................................................................................... 2 1.3 Document History .............................................................................................. 2 1.4 Related Documents ........................................................................................... 3

2 Concept and Design ........................................................................................ 5 2.1 Switch Groups ................................................................................................... 5 2.2 Network Speed .................................................................................................. 6 2.3 Network Connection .......................................................................................... 7 2.4 Linux Bonding .................................................................................................... 8 2.5 ESX NIC Teaming ............................................................................................. 9 2.6 NetApps Interface Grouping ............................................................................ 10 2.7 BX Link State Propagation .............................................................................. 10 2.8 Virtual LAN ...................................................................................................... 11 2.9 Uplink Connection ........................................................................................... 13 2.10 Client LAN Connection .................................................................................... 14

3 FlexFrame Network Versions ....................................................................... 17 3.1 Small Version .................................................................................................. 17 3.2 Directly Connected Versions ........................................................................... 17 3.3 Enterprise Version ........................................................................................... 21

4 Configuration Commands ............................................................................ 23 4.1 Linux Bonding Interface ................................................................................... 23 4.2 Linux VLAN Interface ...................................................................................... 24 4.3 NetApp Filer Configuration .............................................................................. 24 4.4 Network Switch Configuration ......................................................................... 25

5 Abbreviations ................................................................................................ 31

6 Glossary ......................................................................................................... 33

7 Index ............................................................................................................... 36

Network Design and Configuration Guide 1

General product information and naming

This document describes the network concepts of the solution FlexFrame®.

This solution is offered in two characteristics:

the FlexFrame® for SAP

® V5.3A solution

This is the successor of the well-known solution FlexFrame® for SAP

® which is

in the market for years with its latest version V5.2A.

the FlexFrame® Orchestrator V1.0A solution

This is a new approach to offer enhanced functionality and features (e.g. support

of SAP HANA, support of KVM as virtual machine) step by step and become

more and more independent from certain hardware and software components.

These additional new features compared to the traditional FlexFrame for SAP

features are available by special licenses only.

This documentation is for both characteristics:

all descriptions regarding FlexFrame Orchestrator have a corresponding notifica-

tion within their chapters

if there is no notification regarding FlexFrame Orchestrator the described func-

tionality is valid for both characteristics

The name FlexFrame® is a generic term for both "FlexFrame

® for SAP

®" (FF4S) and

"FlexFrame® Orchestrator" (FFO).

The trademark myAMC® is not being used any more. The brand myAMC

® will be re-

placed by the brand FlexFrame®, e.g. instead of myAMC Accounting we now speak

about FlexFrame Accounting.

The abbreviation FA for FlexFrame Autonomy now denotes the FlexFrame Agents

(Application Agents, Control Agents and the Frame Agent). The FA Suite now denotes

the FlexFrame Agents Suite.

Introduction Purpose of this Document

2 Network Design and Configuration Guide

1 Introduction

1.1 Purpose of this Document

This document describes the FlexFrame networking concept and design and possible

networking topologies.

The reader should be familiar with IP networking and needs basic knowledge about

Ethernet-based networking with switches and virtual LANs. Knowledge of network switch

configuration is recommended. The knowledge of the FlexFrame Installation Guide and

the FlexFrame Administration and Operation Guide would be an advantage.

1.2 Notational Conventions

The following conventions are used in this manual:

Additional information that should be observed.

Warning that must be observed.

fixed font Names of paths, files, commands, and system output.

<fixed font> Names of variables.

fixed font User input in command examples

(if applicable using <> with variables).

1.3 Document History

Document Version Changes Date

1.0 First Edition 2013-09-18

Related Documents Introduction


1.4 Related Documents

FlexFrame® – Administration and Operation

FlexFrame® – HW Characteristics Quickguides

FlexFrame® – Installation and Configuration of LVM 1.0 Standard Edition

FlexFrame® – Installation Guide for SAP Solutions

FlexFrame® – Installation of a FlexFrame Environment

FlexFrame® – Management Tool

FlexFrame® – FlexFrame Agents Installation and Administration

FlexFrame® – FlexFrame Messenger Installation and Administration

FlexFrame® – FlexFrame LogAgent Installation and Administration

FlexFrame® – Network Design and Configuration Guide

FlexFrame® – Security Guide

FlexFrame® – Technical White Paper

FlexFrame® – Upgrading FlexFrame 5.1A or 5.2A to 5.3A/1.0A

ServerView Documentation

SUSE Linux Enterprise Server Documentation


2 Concept and Design

The design goals for the FlexFrame network concept:

avoid single points of failure (SPOF)

avoid unnecessary complexity

are achieved by the following means:

use redundant network

use a lightweight design

use standard interfaces and configurations

keep interface configuration as identical as possible on all server systems

keep design flexible enough to meet the requirements from small to medium up

to enterprise sized installations

reduce count of needed interfaces per system

and results in the concept described above.

2.1 Switch Groups

Switch 2

Switch 1

SwitchGroup

LAN A

LAN B

Application Node

PORT a

Application Node

Blade Rack

PORT c PORT b

Control Node 2

PORT a

PORT c PORT b

LAN A

LAN B

Application Node

LAN A

LAN B

Control Node 1

LAN A

LAN B

NAS

LAN A

LAN B

The very simplified drawing above may visualize the general concept (only data connec-

tions are considered). The core of the design is the switch group which provides redun-

dant usable ports. If one member of the switch group fails the ports of the other members

Concept and Design


are still working. Every end system and the uplink are using this redundancy as described

later. In this way a failure of a single member of the switch group is tolerated and allows

continuous operation.

FlexFrame distinguishes the following switch group types:

CAT3750-STACK

Two up to nine switches of the Cisco Catalyst 3750G, 3750E or 3750X switch family

are building a Cisco Catalyst switch stack. The switches are connected via Cisco

StackWise cabling and behave like a single switch. For more than 4 switches with

10GbE ports the StackWise cabling may be a bottleneck.

NEXUS5000-VPC

Exactly two switches (nexus50xx) of the Cisco Nexus 5000 Switch family are building

a Cisco Nexus vPC domain. For the vPC peer-link as a necessary special channel

between the switches FlexFrame configures two ports on each switch. For the vPC

peer-keepalive link as a necessary alternative Layer 3 connectivity between the

switches FlexFrame uses the mgmt0 interface IP addresses as recommended from

Cisco.

NEXUS5500-VPC

Exactly two switches (nexus55xx) of the Cisco Nexus 5000 Switch family are building

a Cisco Nexus vPC domain. For the vPC peer-link as a necessary special channel

between the switches FlexFrame configures two ports on each switch. For the vPC

peer-keepalive link as a necessary alternative Layer 3 connectivity between the

switches FlexFrame uses the mgmt0 interface IP addresses as recommended from

Cisco.

For further details about supported switches see the FlexFrame Support Matrix.

For further details about the switches see the Cisco manuals.

For further details about the purpose of the different switch group types see below.

2.2 Network Speed

FlexFrame supports network connections for data communication with the following net-

work speeds:

1Gbit/sec (1GbE)

10Gbit/sec (10GbE)

Pay attention to the fact that 10GbE connected servers may also request a 10GbE con-

nected NAS system and 10GbE uplinks between.

For further details about supported end systems see the FlexFrame Support Matrix.

Concept and Design


2.3 Network Connection

Network connection is established by connecting ports of end systems to ports of network

devices which itself are connected to other network devices. The characteristics of con-

nected ports must be compatible and a proper connector must be used.

Cat5 or Cat5e cable are used to connect

RJ45 Service LAN ports of RX servers

RJ45 management blade ports of BX cabinets

to RJ45 10/100/1000 ethernet ports of switches of the Cisco Catalyst 3750 switch family.

Cat5e cable are used to connect

RJ45 1GbE LAN ports of RX servers

RJ45 1GbE LAN ports of NAS systems

RJ45 1GbE switch blade ports of BX cabinets

to RJ45 10/100/1000 ethernet ports of switches of the Cisco Catalyst 3750 switch family.

LC-LC fibre optic cable are used to connect

10GbE LC ports of RX servers

10GbE SFP+ modules plugged into RX servers

10GbE SFP+ modules plugged into switches of the Cisco Nexus 5000 switch family

10GbE SFP+ modules plugged into the Cisco Catalyst 3750x 10G module

to 10 GbE SFP+ modules plugged into switches of the Cisco Nexus 5000 switch family

or to 10 GbE SFP+ modules plugged into the Cisco Catalyst 3750x 10G module

or to connect

1GbE SFP modules plugged into switches of the Cisco Catalyst 3750 switch family

1GbE SFP modules plugged into dual speed ports of switches of the Cisco Nexus

5000 switch family

to 1GbE SFP modules plugged into switches of the Cisco Catalyst 3750 switch family.

LC-SC fibre optic cable are used to connect

10GbE LC ports of RX servers

10GbE SFP+ modules plugged into RX servers

10GbE SFP+ modules plugged into switches of the Cisco Nexus 5000 switch family

to 10 GbE ports of switches of the Cisco Catalyst 3750E switch family.

10GBASE-CU SFP+ cable (Twinax) are used to connect

10GbE ports of switches of the Cisco Nexus 5000 switch family used for peer link

to switches of the Cisco Nexus 5000 switch family.

Concept and Design


2.4 Linux Bonding

Linux provides a mechanism called bonding which can aggregate several physical inter-

faces to a virtual interface (bond). A bond interface can be used like a usual interface and

is operational as long as at least one physical interface assigned to the bond is opera-

tional. Several bonding configurations are available.

Switch 2

Switch 1

active eth0

eth1

rack server or

server blade

bond0

active

When running Linux on a physical server (rack server or server blade) used as Applica-

tion Node FlexFrame aggregates 2 physical interfaces using adaptive load balancing

(mode 6) and failure detection relies solely on the link state provided by the adapter (MII

monitoring). This is an active/active method and has no requirements on the link partners

which can be switch blades or switch group switches. The requirement on the adapter to

be able to change the MAC address is fulfilled from all supported servers.

On the Control Nodes bonding mode 1 (active-backup) is used. No requirements have to

be fulfilled neither from the link partners nor from the adapter. Failure detection relies

solely on the link state provided by the adapter (MII monitoring).

Concept and Design


2.5 ESX NIC Teaming

ESX provides a mechanism called NIC teaming which is used if multiple physical Ether-

net adapters are assigned to a single virtual switch (vSwitch). A vSwitch is operational as

long as at least one physical Ethernet adapter assigned to the vSwitch is operational.

Several NIC teaming policies are available and can be set on the port group level.

Switch 2

Switch 1

active vmnic0

vmnic1

rack server or

server blade

vSwitch active

FlexFrame assigns 2 physical Ethernet adapters in ESX default configuration to the

vSwitch and puts all these adapters in the active list for the FlexFrame generated port

groups. The adapters are used with load balancing based on the originating virtual Port

ID and failure detection relies solely on the link state provided by the adapter

This method has no requirements on the link partners which can be switch blades or

switch group switches.

Concept and Design


2.6 NetApps Interface Grouping

NetApps Data ONTAP provides a mechanism called interface grouping which can aggre-

gate several physical interfaces to a virtual interface. An interface group can be used like

a usual interface and is operational as long as at least one physical interface assigned to

the group is operational. Several modes are available.

Switch 2

Switch 1

SwitchGroup

NetApp Filer

ifgrp

e0a

e0b

active

active

FlexFrame aggregates at least 2 physical interfaces into a dynamic multimode interface

group which is compliant with IEEE 802.3ad. All physical interfaces are active. The corre-

sponding ports on the switch group switches must build a channel also complaint with

IEEE 802.3ad (LACP).

2.7 BX Link State Propagation

BX switch blades provide a mechanism which allows associating several downstream

ports with an upstream port or channel for link state propagation. If the link of the up-

stream port or channel goes down the link of all associated downstream ports are also

forced to go down. In this way a bonding driver on a server blade will get aware of the

upstream problem and switches the traffic of concerned interfaces.

Concept and Design


active

eth0

eth1

server blade

bond0 active

Switch 2

Switch 1

SwitchGroup BX chassis

Switchblade 1

Switchblade 2

FlexFrame associates all downstream ports of a switch blade and a single upstream

channel for link state propagation. The upstream channel is compliant with IEEE 802.3ad.

The corresponding ports on the switch group switches must also build a channel com-

plaint with IEEE 802.3ad (LACP). The channel consists of at least two ports with crosso-

ver cabling. More ports can be used for higher bandwidth.

2.8 Virtual LAN

Virtual LAN (VLAN) is a mechanism which allows having multiple separated networks on

the same physical network infrastructure. A widely used standard is tagged VLAN accord-

ing IEEE 802.1q. The protocol specifies an extension of the Ethernet header by a VLAN

identifier (tag). A packet belongs to the VLAN identified by the tag. Communication be-

tween VLANs is only possible by routing as is common for usual networks. For compati-

bility reason packets without a tag are also allowed. A switchport can be configured to

assign received packets without tag to a specific VLAN and remove the tag when a pack-

et of this VLAN should be send. The VLAN is then called native or untagged. End sys-

tems which access only one VLAN or cannot handle VLAN tags are connected this way.

End systems used within FlexFrame like NAS systems or servers running Linux are able

to handle VLAN traffic according IEEE 802.1q.

FlexFrame can handle different pools with independent networks. The traffic between

communication partner types (client – server, server – server and server – storage) is al-

so separated. Therefore VLANs are used as helpful mechanism to reduce the necessary

physical infrastructure and increase the flexibility with respect to the pooling concept.

According IEEE 802.1q the VLAN tag has a value range from 1 to 4094 but there are

switch specific restrictions.

Concept and Design


Cisco Catalyst 3750 Switch Family

1005 VLANs are supported. Normal usable VLANs are in the range from 1 to 1001. VLAN

numbers 1002 through 1005 are reserved for Token Ring and FDDI VLANs. VLAN num-

bers 1006 through 4094 are extended-range VLANs and usable if not already used inter-

nally for routed ports.

Cisco Nexus 5000 Switch Family

507 VLANs are supported. Normal usable VLANs are in the range from 1 to 1005. VLAN numbers 1006 through 4094 are extended-range VLANs and usable with the following restrictions. The state is always active and VLANs 3968 to 4047 and 4094 are reserved for internal use.

Physical Network Layout & Virtual LANs per Pool

V1 V2 V4

eth 2 eth 1eth 0

IPMI

Onboard

LAN Port 3

Onboard

LAN Port 1

V1 V2 V4

Bond 0

eth 2 eth 1eth 0

IPMI

Control Node A (RX300S6) Control Node B (RX300S6)

Onboard

LAN Port 2Onboard

LAN Port 3

Onboard

LAN Port 1

Onboard

LAN Port 2

Cisco Switch BCisco Switch A Switchgroup

Control Nodes and Linux Application Nodes (RX300S6), both Storage Types

V3

V1

cge0 cge1

Data

Mover

trunk

EMC Celerra

Logical View: VLANs

VLANsV4: Client

V3: Control V2: Server

V1: Storage

Control

Station

V2 V4

eth 2 eth 1eth 0

IPMI

Application Node 1 (RX300S6)

Application Node n

Onboard

LAN Port 3

Onboard

LAN Port 1Onboard

LAN Port 2

Bond 0V1 V3

Bond: vif

eXa eXb

NetApp

Filer

Physical View:

Redundant Connections

Bond 0

The picture shows the physical and logical view of networking in FlexFrame with respect

to a single pool. That is:

An Application Node is assigned to a pool and connected to the pools client, server and

storage VLAN. When running on ESX server for every connection an interface is created

on the vSwitch, when running native the interfaces for client and server LAN are created

Concept and Design


as VLAN interface on top of the bond. The storage LAN is also used for PXE boot and

assigned native because no tags can be handled in this early phase. Therefore the stor-

age LAN is accessible via the bond directly.

An ESX server is connected to all VLANs necessary for the Application Nodes and to the

control LAN which is assigned native.

IPMI interfaces of rack servers and management blades of BX cabinets are connected

native to the control LAN.

A Control Node is connected to all VLANs, the control LAN is native.

A NAS system is connected to all VLANs, all VLANs are tagged.

2.9 Uplink Connection

Switch 2

Switch 1

SwitchGroup

FlexFrame can assign uplinks to a switch group. An uplink consists of ports from different

switch group members building a channel and carries the traffic of all used VLANs. An

uplink is used to connect a switch group to another switch group or to the customer cor-

porate LAN. The supported topologies stick to the following rules:

A CAT3750-STACK has maximal one uplink and is connected to another

CAT3750-STACK or to a NEXUS5x00-VPC or to the customer corporate LAN.

A NEXUS5x00-VPC may have more than one uplink but maximal one of them is

connected to another NEXUS5x00-VPC or to the customer corporate LAN. Each

other uplink is connected to a CAT3750-STACK.

Concept and Design


Following the above rules ensures a topology without loop. Nevertheless, the Cisco per

default enabled rapid per VLAN spanning tree (rapid-PVST) algorithm is used to avoid

loops which may appear e.g. in early phase when booting a switch or by misconfiguration

especially because FlexFrame does not check whether the rules are followed.

Examples for supported topologies are given in chapter 3.

When connected to customer corporate LAN the connected customer ports must also

build a channel. The channel must be compliant with IEEE 802.3ad (LACP) and all used

VLANs must be allowed tagged.

2.10 Client LAN Connection

For accessing a FlexFrame infrastructure solution connection to the client LAN must be

established. In general this means the client LAN must be available in the context of a

router and appropriate routing must be configured. This is a manual task and the custom-

er decides which network should have access to a special client LAN.

To bring a client LAN in the context of a router different ways are available:

a) If uplinks of switch groups are connected to the customer corporate LAN e.g. as

described in chapter 3.3 the client LANs are already available and no further ar-

rangements are necessary.

b) Special uplinks for client LAN connections are configured (e.g. see

ff_swgroup_adm.pl) and connected to the customer corporate LAN like normal

uplinks. The VLANs on these uplinks are restricted to client LAN VLANs.

c) Special ports for client LAN connections are configured. These CLAN ports are

thought being connected to a router and therefore configured as spanning tree

edge ports. The ports can be distributed over several switch groups and it is

possible to have particular ports for every client LAN. Only Ports of switches of a

CAT3750-STACK are allowed to be used as CLAN ports. See below an exam-

ple how CLAN ports configured for a single client LAN can be used.

Concept and Design


Switch 2

Switch 1

SwitchGroup 1

Switch 2

Switch 1

SwitchGroup 2

uplink channel

Virtual Gateway Adress IP3

Layer 3 port IP1

Layer 3 port IP2

outside FlexFrame

The ports configured for client LAN connection are connected to Layer 3 ports of different

Layer 3 switches. An IP address is assigned to every Layer 3 port and the Layer 3

switches are configured to build a logical router e.g. via HSRP. The logical router sup-

ports a virtual IP address (IP3) which is used as standard gateway for the special client

LAN.


3 FlexFrame Network Versions

The FlexFrame network design is flexible enough to meet requirements from small to en-

terprise size networks. In the following some examples are given.

3.1 Small Version

This version is based on one switch group as there is no need for

splitting FlexFrame into different data centers

providing more switch ports as one switch group can offer.

Switch 2

Switch 1

CAT3750-STACK

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

client LAN

All devices are connected to a single CAT3750-STACK. The connection of the devices is

shown simplified only with regard to the redundant connections for data communication.

For client LAN connection two ports are configured on different members of the switch

group. No uplink needs to be configured.

3.2 Directly Connected Versions

Topologies according to the rules mentioned in 2.9 with more than one switch group and

without any uplink connection to the customer corporate LAN are called directly connect-

ed versions.

The following version may meet the requirements if

FlexFrame has to be split into two data centers or

one switch group might have not enough ports for all devices

FlexFrame Network Versions


Example1: Two CAT3750-STACKs

Switch 2

Switch 1

CAT3750- STACK

Switch 2

Switch 1

CAT3750- STACK

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

Uplink Channel

client LAN

All devices are connected to one of two CAT3750-STACKs. The connection of the devic-

es is shown simplified only with regard to the redundant connections for data communica-

tion. For client LAN connection one port is configured on every switch group.

The switch groups are connected via uplink channel a) directly without any other network Layer2 device between per copper or for

greater distance per fiber optic cable.

b) via network devices allowing a Layer2 connection e.g. DWDM for even greater

distance. In this case the network devices between must be transparent to the

switch groups, especially regarding link state. That means, if a port is shutdown

on one switch group the corresponding port on the other switch group must see

a link down event. Also if the connection between is broken both ports must see

a link down event.



Example 2: single NEXUS5000-VPC with one CAT3750-STACK

Switch 2

Switch 1

NEXUS5000-VPC

Switch 2

Switch 1

CAT3750- STACK

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

Uplink Channel

client LAN

The CAT3750-STACK is directly connected to the NEXUS5000-VPC without any other

network Layer2 device between. The service LAN ports of rack servers and the control

nodes and the management ports of the nexus switches are connected to the CAT3750-

STACK. The end systems connected to NEXUS5000-VPC are 10GbE-connected. For

client LAN connection two ports are configured on different members of the CAT3750-

STACK.



Example 3: two NEXUS5000-VPCs each with connection to one CAT3750-STACK

Switch 2

Switch 1

CAT3750-STACK

Switch 2

Switch 1

CAT3750-STACK

Switch 2

Switch 1

NEXUS5000-VPC

Switch 2

Switch 1

NEXUS5000-VPC LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

uplink channel

client LAN

The service LAN ports of rack servers and the control nodes are connected to a

CAT3750-STACK which is connected to the NEXUS5000-VPC. The end systems con-

nected to NEXUS5000-VPC are 10GbE-connected.

The NEXUS5000-VPC switch groups are connected via uplink channel

directly without any other network Layer2 device between per fiber optic cable.

via network devices allowing a Layer2 connection e.g. DWDM for greater dis-

tance. In this case the network devices between must be transparent to the

switch groups, especially regarding link state. That means, if a port is shutdown

on one NEXUS5000-VPC switch group the corresponding port on the other

NEXUS5000-VPC switch group must see a link down event. Also if the connec-

tion between is broken both ports must see a link down event.



3.3 Enterprise Version

The possibilities of directly connected versions are restricted. Greater configurations need

uplinks to customer corporate LAN e.g. the customer core switches.

Customer own Core

Switch

Customer own Core

Switch

Customer Corporate LAN

Node

Node

Node

Node

Node

Node

Node

CAT3750-STACK

Uplink Channel

NEXUS5000-VPC

NEXUS5000-VPC

Node

Uplink Channel Uplink Channel

CAT3750-STACK

CAT3750-STACK

Uplink Channel

Uplink Channel

Each NEXUS5000-VPC switch group is connected with its uplink ports to core switch

ports. CAT3750-STACK switch groups may be connected to core switches or to

NEXUS5000-VPC switch groups. The switch groups may be connected to different core

switches. Like the Uplink the corresponding ports on the core switch(es) must:

build a channel

carry all VLANs used within FlexFrame

transfer traffic without blocking

The core switches are not part of FlexFrame, neither of automated configuration nor sup-

port. Configuration changes at core switches have to be done manually. There are no

messages or instructions from FlexFrame to support this.


4 Configuration Commands

As described in Chapter 2 FlexFrame uses high availability configurations concerning

network on many devices. This chapter describes how this is done in detail.

4.1 Linux Bonding Interface

On the control nodes the bonding configuration is performed via configuration files

/etc/sysconfig/network/ifcfg-bond0:

STARTMODE=onboot

BOOTPROTO=static

IPADDR=192.168.20.1

NETMASK=255.255.255.0

NETWORK=192.168.20.0

BROADCAST=192.168.20.255

BONDING_MASTER=yes

BONDING_SLAVE_0=eth0

BONDING_SLAVE_1=eth1

BONDING_MODULE_OPTS="miimon=100 mode=6"

/etc/sysconfig/network/ifcfg-eth0

/etc/sysconfig/network/ifcfg-eth1 (both are the same):

STARTMODE=auto

On application nodes running native the bonding configuration is performed during boot

via commands. The network parameters are obtained through DHCP and include the IP

address for the storage LAN.

Configuration Commands


4.2 Linux VLAN Interface

VLAN configuration is performed via configuration files

/etc/sysconfig/network/ifcfg-vlan<VLAN ID>

e.g. for VLAN 10: /etc/sysconfig/network/ifcfg-vlan10:

STARTMODE='onboot'

BOOTPROTO='static'

IPADDR=192.168.200.11

NETMASK=255.255.255.0

NETWORK=192.168.200.0

BROADCAST=192.168.200.255

ETHERDEVICE='bond0'

WIRELESS='no'

4.3 NetApp Filer Configuration

The configuration has to be done manually e.g. via serial cable (see the corresponding

HW Characteristics Quickguide you find on the Service CD in doc/hwinfo).


4.4 Network Switch Configuration

The network switch configuration is performed via commands. The following tables shows

Cisco IOS and Cisco NX-OS configuration commands for different purposes.

Cisco Catalyst 3750 Family Configuration (IOS commands)

Basic configuration service password encryption

enable password secret

clock timezone CET 2

clock summer-time cet recurring last Sun Mar 2:00 last

Sun Oct 3:00

vtp mode transparent

udld aggressive

no ip http server

snmp-server community public RO

spanning-tree mode rapid-pvst

spanning-tree loopguard default

no spanning-tree optimize bpdu transmission

spanning-tree extend system-id

VLAN configuration for a system

with a pool pool1

vlan 10

name client-pool1

vlan 11

name storage-pool1

vlan 12

name server-pool1

vlan 13

name control

port configuration for a data port

of a rack server used as applica-

tion node

interface GigabitEthernet1/0/1

description Node rx300s6 dataNic 1

switchport trunk encapsulation dot1q

switchport trunk native vlan 11

switchport trunk allowed vlan 10-12

switchport mode trunk

no ip address

speed auto

no mdix auto

udld port

storm-control broadcast level 20

storm-control multicast level 10

no shutdown




spanning-tree portfast trunk


of a rack server used as esx-

server


description Node rx300s6



switchport trunk allowed vlan 10-12,20-22


no ip address

speed auto

no mdix auto

udld port



no shutdown


port configuration for a IPMI port

of a rack


description Node rx300s6 mgmtNic



switchport trunk allowed vlan 13


no ip address

speed auto

no mdix auto

udld port



no shutdown


channel configuration e.g. for an

uplink

interface PortChannel 1

description ISL Uplink




no shutdown






no ip address




speed auto

mdix auto

udld port



cdp enable

channel-group 1 mode active

no shutdown






no ip address

speed auto

mdix auto

udld port



cdp enable


no shutdown

Save configuration copy running-config startup-config

For details see the “Catalyst 3750 Switch Software Configuration Guide” at

http://www.cisco.com.

Cisco Nexus 5000 Family Configuration (NX-OS commands)

Basic configuration no feature telnet

no telnet server enable

cfs eth distribute

feature udld

feature lacp

feature vpc

vpc domain 2

peer-keepalive destination 172.11.6.19

vrf context management

ssh key rsa 2048

udld aggressive

spanning-tree mode rapid-pvst

http://www.cisco.com/




spanning-tree loopguard default

VLAN configuration for a system

with a pool pool1

vlan 10

name client-pool1

vlan 11

name storage-pool1

vlan 12

name server-pool1

vlan 13

name control


of a rack server used as applica-

tion node

interface Ethernet1/1

description Node rx300s6 dataNic 1


switchport trunk allowed vlan 10-12


spanning-tree port type edge trunk

udld aggressive



no shutdown


of a rack server used as esx-

server


description Node rx300s6




spanning-tree port type edge trunk

udld aggressive



no shutdown

vpc configuration e.g. for an up-

link (same vpc number on both

nexus switches of the switch

group, per convention equal to

port-channel number)

interface port-channel2


switchport trunk allowed vlan 1,10-13,20-22


vpc 2



no shutdown



switchport trunk allowed vlan 1,10-13,20-22


udld aggressive







no shutdown

Save configuration copy running-config startup-config

For details see the “Cisco Nexus 5000 Series NX-OS Layer 2 Switching Configuration

Guide” at http://www.cisco.com.

http://www.cisco.com/


5 Abbreviations

DART Data Access in Real Time

DHCP Dynamic Host Configuration Protocol

FF FlexFrame

FF4S FlexFrame for SAP

FFO FlexFrame Orchestrator

IP Internet Protocol

LAN Local Area Network

MAC Media Access Control

MII Media Independent Interface

NAS Network Attached Storage

NIC Network Interface Card

ONTAP Open Network Technology for Appliance Products

PXE Preboot Execution Environment

SPOF Single Point Of Failure

TFTP Trivial File Transfer Protocol

UDP User Datagram Protocol

VLAN Virtual Local Area Network

VPC Virtual Port Channel


6 Glossary

Application Node

A host for applications (e.g. SAP instances db, ci, agate, wgate, app etc.). This

definition includes Application Servers as well as Database Servers.

Blade

A special form factor for computer nodes.

Celerra

NAS system of EMC.

Client LAN

Virtual network segment within FlexFrame, used for client-server traffic.

Computing Node

From the SAP ACI perspective: A host that is used for applications.

Control LAN

Virtual network segment within FlexFrame, used for system management traffic.

Control Node

A physical computer system, controlling and monitoring the entire FlexFrame land-

scape and running shared services in the rack (dhcp, tftp, ldap etc.).

Control Station

A Control Node in an SAP ACI environment.

Dynamic Host Configuration Protocol

DHCP is a protocol for assigning dynamic IP addresses to devices on a network.

Dynamic Host Configuration Protocol server

A DHCP server provides configuration parameters specific to the DHCP client host,

required by the host to participate on the Internet.

EMC NAS

Network attached storage for file systems of EMC.

Ethernet

A Local Area Network which supports data transfer rates of 10 megabits per second.

Filer

Network attached storage for file systems of NetApp.

FlexFrame®

The name FlexFrame® is a generic term for both „FlexFrame

® for SAP

®“ and „Flex-

Frame® Orchestrator“.

FlexFrame® Orchestrator

This is the advancement of the Fujitsu solution FlexFrame for SAP and means a new

Glossary


approach to offer enhanced functionality and features step by step and become more

and more independent from certain hardware and software components.

FlexFrame® for SAP

®

FlexFrame® for SAP

® is a Fujitsu solution and means a radically new architecture for

SAP environments. It exploits the latest business-critical computing technology to de-

liver major cost savings for SAP customers. FlexFrame for SAP is a joint project in

which the main partners are SAP, Network Appliance, Intel and Fujitsu.

FlexFrame internal LAN Switch

Network switches which are integral part of the FlexFrame for SAP hardware configu-

ration and which are automatically configured by the FlexFrame for SAP software.

Gigabit Ethernet

A Local Area Network which supports data transfer rates of 1 gigabit (1,000 mega-

bits) per second.

Host name

The name of a node (assigned to an interface) that is resolved to a unique IP ad-

dress. One node can have multiple host names (cf. node name).

In SAP environments host names are currently limited to 13 alphanumeric characters

including the hyphen (“ - “). The first character must be a letter. In the SAP environ-

ment host names are case-sensitive.

Internet Protocol Address

A unique number used by computers to refer to each other when sending information

through networks using the Internet Protocol.

Local Area Network

A computer network that spans a relatively small area. Most LANs are confined to a

single building or group of buildings. However, one LAN can be connected to other

LANs over any distance via telephone lines and radio waves. A system of LANs con-

nected in this way is called a Wide Area Network (WAN).

Local host name

The name of the node (physical computer); it can be displayed and set using the

command /bin/hostname.

Media Access Control address

An identifier for network devices, usually unique. The MAC address is stored physi-

cally on the device.

NAS system

Network Attached Storage of any vendor (in our context: EMC NAS or NetApp Filer).

Network Attached Storage

A data storage device that is connected via a network to one or multiple computers.

Network Interface Card

A hardware device that allows computer communication via networks.

Glossary


Node

A physical computer system controlled by an OS.

Node name

The name of a physical node as returned by the command uname -n. Each node

name within a FlexFrame environment must be unique.

Open Network Technology for Appliance Products

The operating system of Network Appliance Filers.

Physical host

Name of a physical computer system (node).

Preboot Execution Environment

An environment that allows a computer to boot from a network resource without hav-

ing a local operating system installed.

Server

A physical host (hardware), same as node.

Service

A software program providing functions to clients.

Storage LAN

A virtual LAN segment within a FlexFrame environment, carrying the traffic to NAS

systems.

Trivial File Transfer Protocol

A simple form of the File Transfer Protocol (FTP). TFTP uses the User Datagram

Protocol (UDP) and provides no security features. It is often used by servers to boot

diskless workstations, X-terminals, and routers.

TFTP server

A simple FTP implementation.

Virtual host

The name of the virtual host on which an application runs; it is assigned to a physical

node when an application is started.

Virtual Local Area Network

A VLAN is a logically segmented network mapped over physical hardware according

to the IEEE 802.1q standard.


7 Index

C

concept and design 5

F

FlexFrame network versions 17

enterprise version 21

medium version 17

small version 17

L

link aggregation 5, 8, 9, 10, 11, 14, 15

Linux

bonding interface 23

VLAN interface 24

N

NetApp Filer configuration 24

network speed 6

network switch configuration 25

node configuration 23

R

related documents 3

V

virtual LAN 12

Network Design and Configuration Guide -...

Documents

Transcript of Network Design and Configuration Guide -...