Practical Tips for Novell Cluster Services

Novell Corporate Presentation Template 2009 - blue

Practical Tips for
Novell Cluster Services

Mark RobinsonCTO Linux, [email protected]

Tim HeywoodCTO, [email protected]

Agenda

Introduction

Cluster Services in OES2

Our Environment

AutoYaST

Cluster Build Methodology

Creating Resources

Cluster Management

Troubleshooting

Introduction

Introduction

Mark RobinsonLinux Geek

Working with SUSE since 1998

Working with OES since OES1 Beta 5

CLP, CLE, NCE ES, CNI, etc.

Ex-SysOp

Tim HeywoodWorking with Novell since ????

Working with OES since OES1 Beta 5

CNE, MCNE, CNI(ish)

Novell Knowledge Partner (SysOp)

Introduction

NDS8 Network Design and Support Ltd.Platinum Consulting Partner

Based in Edinburgh, work worldwide

Specialities:Linux

Workgroup

SRM



New features are Linux only

New from OES2 FCS on:Resource monitoring

XEN virtualization support

x86_64 platform supportIncluding mixed 32/64 bit node support

Dynamic Storage Technology

What's new in SP1/2?

Major rewrite of cluster code for SP2Removed NetWare translation layer

Much faster

Much lower system load

Typical load average of 0.2!

New/improved clustering for:iFolder 3

AFP

NCP virtual server for POSIX filesystem resources

Types of Clusters

Traditional clusterServers (nodes)

ResourcesNSS

GroupWise

iPrint

XEN clusterDom0 hosts (nodes)

XEN guests (DomU) resources

Each resource is a server in its own right

Live migration with para-virtualised DomU

XEN Cluster Architecture

AutoYaSTOCFS2 LUNDomU Files

Cluster Node
Xen Dom0

Cluster Node
Xen Dom0

Cluster Node
Xen Dom0

Resource
DomU
Linux
iPrint

Resource
DomU
Linux
iPrint

Resource
DomU
Linux
iFolder

Resource
DomU
Linux
GroupWise

Resource
DomU
NetWare
pCounter

Live Migrate

Live Migrate

Our Environment

Our Environment

VMware Workstation based

VMware shared disk as an alternative to iSCSI

Virtual MachinesResource Server

Node 1 (built, in the cluster)

Node 2 (to be joined to cluster)

Node 3 (to be built)

SUSE Linux Enterprise Server 10 SP3/OES2 SP2 combined iso

Explain about building combined iso

VMware Setup

Create disks standalone

Add config to node vmx files

disk.locking = "false"diskLib.dataCacheMaxSize = "0"

scsi1.present = "TRUE"scsi1.sharedBus = "none"scsi1.virtualDev = "lsilogic"scsi1.pciSlotNumber = "35"

scsi1:0.present = "TRUE"scsi1:0.fileName = "cluster-lun0.vmdk"scsi1:0.mode = "independent-persistent"scsi1:0.redo = ""

scsi1:1.present = "TRUE"scsi1:1.fileName = "cluster-lun1.vmdk"scsi1:1.mode = "independent-persistent"scsi1:1.redo = ""

vmware-vdiskmanager -c -s 100mb -a lsilogic -t 2 cluster-lun0.vmdkvmware-vdiskmanager -c -s 1gb -a lsilogic -t 2 cluster-lun1.vmdk

Discuss flags for vmware-vdiskmanager especially -t

Disk types:0 : single growable virtual disk1 : growable virtual disk split in 2GB files2 : preallocated virtual disk3 : preallocated virtual disk split in 2GB files4 : preallocated ESX-type virtual disk5 : compressed disk optimized for streaming

Discuss path to SAN virtual disks

Mention different VMware versions

Our Environment

Resource Server
(Tree master, iManager,Installation services,AutoYaST, SMT)

Storage

VMware Shared Disks

OES2 SP2 Nodes

Our Environment

AutoYaST Third Node

Resource Server
(Tree master, iManager,Installation services,AutoYaST, SMT)

Storage

VMware Shared Disks

OES2 SP2 Nodes

Tell audience that the autoyast build may take a while we'll concentrate on existing two nodes for most of the demo.

AutoYaST

Why AutoYaST?

Repeatable (exactly)No human element

XML forms part of Documentation

Drink coffee (or suitable non-caffeinated beverage) while server builds itself!

Multiple simultaneous buildsStagger by at least 15 minutes

Easy to expand cluster with new nodes

Helps with DR

Why AutoYaST?

What will AutoYaST do?Disk partitioning

Software patterns

Network configuration (including VLAN, bonding etc)

OES serviceseDirectory new or existing tree

NSS

NCS

Security lockdown

Scripts/Complete config files to do the rest

At the end of the install we will migrate an NSS resource to the new node with no additional configuration!

AutoYaST New Cluster

cn=admin.o=novell novell cn=cluster,ou=resources,o=novell 10.0.0.100 New Cluster node IP,LDAP server IP 636 nodename Later sdx sdy

Offer copy of autoyast profiles used in demos.

AutoYaST Existing Cluster

cn=admin.o=novell novell cn=cluster,ou=resources,o=novell Existing Cluster node IP,LDAP server IP 636 nodename Later

Demo
AutoYaST



Start with a Resource ServeriManager

Network Installation Server HTTP or NFS

AutoYaST repository (can be password protected on HTTP)

SMT for patching

Magic PiXiEs server


Create a template nodeNodeZ

Use it toCreate AutoYaST template using Clone this system...

Test the shared disk

Create the cluster

Create the SBD

Creating resources

This build will NOT be part of final production cluster


Copy this XML for additional nodesModifyserver name

IP address(es)

Use diffuse to compare XML files

Build the other nodes Use the XML created above

If not 100% right, whack it, modify XML and start build again

Remember these are now commodity items

Whack NodeZ and rebuild to complete the system


Implement NIC bondingNIC driver independent

7 different methods some require switch support

Link state vs arp monitoring blades often cannot lose local link!

Configurable with AutoYaST

Implement Multipath (MPIO)Very simple to configure mainly autodetect

Wide range of SAN support

Friendly LUN naming

Configuration file can be used (put in place with autoYaST)

Creating Resources

File Sharing Resources

An NSS poolUse iManager

Will end up as Primary for DST pair

Another NSS poolUse NSSMU (just because we can)

Will end up as Shadow for DST pair

Combine them into one resourceDelete resource for shadow

Modify load script for primary

Talk about creating the two pools and why.

DEMO: Create pool1/vol1Create pool1_shd/vol1_shd


POSIX filesystem based resource with NCPEasier than Samba to access files

Can be used for iPrint, DHCP etc

Use evmsgui to create and format the volume

Create the resource in iManager

Script to create NCP virtual server




Add resource monitoring

Add NFS accessLUM enablement of target users

NSS/POSIX rights

exportfs in load script rather then /etc/exports on nodes

Use fsid=x for NSS



NFS access

FPC1

FPC2

FPC3

FPC4

FPC5

SHARED1Virtual Server

SHARED1Volume

eDirectory

fpc.server.novell

NFSaccess

MisUID: 1010

OracleUID: 60003

Iface
UID: 1012

Mis-dweeb
UID: 1004

LUM

NSS Rights

MisUID: 1010

OracleUID:
60003

Iface
UID: 1012

Dweeb

MisUID: 1010

OracleUID:
60003

Iface
UID: 1012

Gromit

MisUID: 1010

OracleUID:
60003

Iface
UID: 1012

Wallace

NFS



iPrint

Create iPrint on NSS

Run iprint_nss_relocate on each node with volume in place

NB: only one iPrint resource may run on a node

Need to accept certificates in iManager for each node



iFolder

Create iFolder on POSIX/mnt/cluster/ifolder

Run /opt/novell/ifolder3/bin/ifolder_cluster_setup on each nodeCopy /etc/sysconfig/novell/ifldr3_2_sp2 to nodes first

NB: Only one iFolder resource may run on a node



DNS

DNS must be on NSS as NCP server required for eDirectory integration

Check NCP:NCPServer objects

LUM user required for NSS rights



DHCP

Create DHCP on NSS

Leases file on NSS volume

Log file on NSS volumeSyslog-ng configuration

Logrotate configuration

Default AppArmor configuration will not allow logging to here!



GroupWise

Create PO on NSS

Set namespace in load script/opt=ns=long

Disable atime/diratime on volumeOpen nsscon

Run /noatime=volname



OCFS2 Shared Storage

Shared disk! Multi-mount, read/write with distributed lock management

/etc/ocfs2/cluster.conf automagically created by NCS

Fstab mounting uses /etc/init.d/ocfs2 service



Cluster Management

Cluster Management

iManagerThe ONLY way to create/delete/edit resources

View event log

Cluster commandSame as NetWare

No cvb rebuild or device scan. These are not required on Linux as EVMS does it for you

Console One



The cluster Command

The usual suspectscluster online/offline/migrate

cluster join/leave

cluster status/resources/view/info

More interestingcluster stats display check heartbeat/SBD ticks

cluster pools check NSS pools and location

cluster set modify heartbeat etc

cluster exec potentially very dangerous

Lots of BCC commands



Troubleshooting

Useful Linux Tools

ip command manage TCP/IP on Linuxip addess show/add/del

ip route show

ethtool NIC settings

cat /proc/net/bonding/bondX

netstat network communication statusCheck which ports services are listening on

Check IP based connections to node

nmap network/port scanner

multipath -ll

Script logs are rolling logs now show all operations on this node for a particular resource

LUN Identification

Which LUN is which?

lsscsi shows LUN ID numbers

ls -l /dev/disk/by-idscsi-360a980004334616f6b5a55572d625550

Need to find ID on SAN. (Netapp uses ASCII!)

Multipathing will show the ID as the multipath nameUse friendly naming


Useful Tools

sbdutil create/check/modify the SBDsbdutil -f to find the SBD

sbdutil -v to view the current state of the SBD

/opt/novell/ncs/bin/ncs-configd.py-init option to pull down load scripts, fix node names etc

cifsPool.py to fix CIFS attributes (TID #7005192)

OES2 NCS Master Reference TID, FAQ and Troubleshooting TID #7001433

NSA Novell Support AdvisorMany patterns for NCS


File Locations

Cluster configuration file/etc/opt/novell/ncs/clstrlib.conf

Load/Unload scripts/var/run/ncs (run from here)

/var/opt/novell/ncs

Load script output logs/var/opt/novell/log/ncs

System Log/var/log/messages


File Locations

Admin filesystem virtual filesystem for NCS management/admin/Novell/Cluster

Proc filesystem virtual filesystem for Linux/NCS management/proc/ncs

Cluster event logiManager

/admin/Novell/Cluster/EventLog.xml


/proc/ncs Magic

Enable serious debugging!echo -n "TRACE ON" > /proc/ncs/vll

echo -n "TRACE SBD ON" > /proc/ncs/vll

echo -n "TRACE GIPC ON" > /proc/ncs/vll

echo -n "TRACE MCAST ON" > /proc/ncs/vll

echo -n "TRACE CVB ON" > /proc/ncs/cluster

Can be made permanent by editing /opt/novell/ncs/bin/ldncs

Find the SBDcat /proc/ncs/sbdlib


AdminFS Magic

Two type of file in /admin/Novell/Cluster*.xml contain cluster/state information

*.cmd write then read files for issuing cluster commands


Known issues

EVMS issue with no NSSUnpatched SP2 nodes without NSS cannot load SBD kernel moduled

iFolder shutdown scriptDoesn't shut down components if names have been changed during configuration

IP address problemUnpatched nodes can allow duplicate IP addresses on network

Resources in NDS sync stateCheck replica rings/referrals

Check case of cluster DN in clstrlib.conf


www.nds8.co.uk

Unpublished Work of Novell, Inc. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.

General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents
of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for Novell products remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.

Click to enter the title (44pt)
Second line or subtitle (22pt)

Presenter Name (16pt)Presenter Title (14pt)

Company/email (14pt)

Click to Edit Section Break Text (32pt)
Right Justified

piece in master that I can't get rid of

Click to edit the title text format

Novell, Inc. All rights reserved.

Click to edit the title text format

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline Level

Practical Tips for Novell Cluster Services

Technology

Transcript of Practical Tips for Novell Cluster Services