IBM Spectrum Scalevladimir/export/Container... · 2020. 8. 31. · This edition applies to IBM...

IBM Spectrum Scale

Container Storage Interface Driver GuideVersion 2.0.0

IBM

SC28-3113-02

Note

Before using this information and the product it supports, read the information in “Notices” on page49.

This edition applies to version 5 release 0 modification 5 of the following products, and to all subsequent releases andmodifications until otherwise indicated in new editions:

• IBM Spectrum Scale Data Management Edition ordered through Passport Advantage® (product number 5737-F34)• IBM Spectrum Scale Data Access Edition ordered through Passport Advantage (product number 5737-I39)• IBM Spectrum Scale Erasure Code Edition ordered through Passport Advantage (product number 5737-J34)• IBM Spectrum Scale Data Management Edition ordered through AAS (product numbers 5641-DM1, DM3, DM5)• IBM Spectrum Scale Data Access Edition ordered through AAS (product numbers 5641-DA1, DA3, DA5)• IBM Spectrum Scale Data Management Edition for IBM® ESS (product number 5765-DME)• IBM Spectrum Scale Data Access Edition for IBM ESS (product number 5765-DAE)

Significant changes or additions to the text and illustrations are indicated by a vertical line (|) to the left of the change.

IBM welcomes your comments; see the topic “How to send your comments” on page xxii. When you send information toIBM, you grant IBM a nonexclusive right to use or distribute the information in any way it believes appropriate withoutincurring any obligation to you.© Copyright International Business Machines Corporation 2015, 2020.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract withIBM Corp.

Contents

Tables................................................................................................................... vAbout this information......................................................................................... vii

Prerequisite and related information........................................................................................................xxiConventions used in this information....................................................................................................... xxiHow to send your comments....................................................................................................................xxii

Chapter 1. Summary of changes.............................................................................1

Chapter 2. Introduction to IBM Spectrum Scale Container Storage Interfacedriver ............................................................................................................... 3

Chapter 3. Planning for IBM Spectrum Scale Container Storage Interface driver..... 5Hardware and Software requirements........................................................................................................ 5Deployment considerations.........................................................................................................................5

Chapter 4. Installation of IBM Spectrum Scale Container Storage Interfacedriver................................................................................................................ 7Performing pre-installation tasks................................................................................................................7Installing IBM Spectrum Scale Container Storage Interface driver using Operator Lifecycle

Manager...................................................................................................................................................9Installing IBM Spectrum Scale Container Storage Interface driver using CLIs.......................................10Cleaning up IBM Spectrum Scale Container Storage Interface driver and Operators by using OLM......11Cleaning up IBM Spectrum Scale Container Storage Interface driver and Operator using CLIs............ 12

Chapter 5. Upgrading IBM Spectrum Scale Container Storage Interface driver...... 13From version 1.1.0 to version 2.0.0.......................................................................................................... 13

Chapter 6. Migrating from IBM Storage Enabler for Containers to IBM SpectrumScale Container Storage Interface driver.......................................................... 15High-level steps ........................................................................................................................................ 15Detailed steps ........................................................................................................................................... 16

Chapter 7. Configuring IBM Spectrum Scale Container Storage Interface driver.... 19IBM Spectrum Scale Container Storage Interface driver configurations.................................................19

Secrets..................................................................................................................................................19Certificates........................................................................................................................................... 19Operator................................................................................................................................................19Storage class........................................................................................................................................ 26

Changing the configuration after deployment.......................................................................................... 29

Chapter 8. Using IBM Spectrum Scale Container Storage Interface driver............. 31Dynamic provisioning.................................................................................................................................31Static provisioning..................................................................................................................................... 31

Generating a PV manifest.....................................................................................................................31Creating a persistent volume (PV)....................................................................................................... 32Creating a PersistentVolumeClaim (PVC)............................................................................................ 33

Creating pods............................................................................................................................................. 33Considerations for mounting read-write many (RWX) volumes......................................................... 34

iii

Chapter 9. Managing IBM Spectrum Scale when used with IBM Spectrum ScaleContainer Storage Interface driver................................................................... 35Adding a new node to the Kubernetes or Red Hat OpenShift cluster...................................................... 35Unmounting IBM Spectrum Scale file system.......................................................................................... 35Shutting down IBM Spectrum Scale..........................................................................................................35IBM Spectrum Scale monitoring considerations...................................................................................... 36Upgrading IBM Spectrum Scale on IBM Spectrum Scale Container Storage Interface driver nodes.....36

On the worker nodes............................................................................................................................ 36On the nodes running provisioner and attacher pods.........................................................................37

Chapter 10. Limitations....................................................................................... 41

Chapter 11. Troubleshooting................................................................................43Debug data collection................................................................................................................................ 43Debugging initialization issues.................................................................................................................. 44Debugging PVC creation issues................................................................................................................. 44Debugging pod mounting issues............................................................................................................... 45

Accessibility features for IBM Spectrum Scale......................................................47Accessibility features.................................................................................................................................47Keyboard navigation.................................................................................................................................. 47IBM and accessibility.................................................................................................................................47

Notices................................................................................................................49Trademarks................................................................................................................................................ 50Terms and conditions for product documentation................................................................................... 50IBM Online Privacy Statement.................................................................................................................. 51

Glossary..............................................................................................................53

Index.................................................................................................................. 61

iv

Tables

1. IBM Spectrum Scale library information units........................................................................................... viii

2. Conventions................................................................................................................................................xxii

3. PVC parameter details................................................................................................................................ 16

4. CSIScaleOperator configuration parameter description........................................................................... 21

5. Parameter description.................................................................................................................................24

v

About this information

This edition applies to IBM Spectrum Scale version 5.0.5 for AIX®, Linux®, and Windows.

IBM Spectrum Scale is a file management infrastructure, based on IBM General Parallel File System(GPFS) technology, which provides unmatched performance and reliability with scalable access to criticalfile data.

To find out which version of IBM Spectrum Scale is running on a particular AIX node, enter:

lslpp -l gpfs\*

To find out which version of IBM Spectrum Scale is running on a particular Linux node, enter:

rpm -qa | grep gpfs (for SLES and Red Hat Enterprise Linux)

dpkg -l | grep gpfs (for Ubuntu Linux)

To find out which version of IBM Spectrum Scale is running on a particular Windows node, open Programsand Features in the control panel. The IBM Spectrum Scale installed program name includes the versionnumber.

Which IBM Spectrum Scale information unit provides the information you need?

The IBM Spectrum Scale library consists of the information units listed in Table 1 on page viii.

To use these information units effectively, you must be familiar with IBM Spectrum Scale and the AIX,Linux, or Windows operating system, or all of them, depending on which operating systems are in use atyour installation. Where necessary, these information units provide some background information relatingto AIX, Linux, or Windows. However, more commonly they refer to the appropriate operating systemdocumentation.

Note: Throughout this documentation, the term "Linux" refers to all supported distributions of Linux,unless otherwise specified.

© Copyright IBM Corp. 2015, 2020 vii

Table 1. IBM Spectrum Scale library information units

Information unit Type of information Intended users

IBM Spectrum Scale:Concepts, Planning, andInstallation Guide

This guide provides the followinginformation:

Product overview

• Overview of IBM Spectrum Scale• GPFS architecture• Protocols support overview:

Integration of protocol accessmethods with GPFS

• Active File Management• AFM-based Asynchronous

Disaster Recovery (AFM DR)• Data protection and disaster

recovery in IBM Spectrum Scale• Introduction to IBM Spectrum

Scale GUI• IBM Spectrum Scale management

API• Introduction to Cloud services• Introduction to file audit logging• Introduction to watch folder API• Introduction to clustered watch

folder• IBM Spectrum Scale in an

OpenStack cloud deployment• IBM Spectrum Scale product

editions• IBM Spectrum Scale license

designation• Capacity based licensing• IBM Spectrum Storage™ Suite• Understanding call home

Planning

• Planning for GPFS• Planning for protocols• Planning for Cloud services• Planning for AFM• Planning for AFM DR• Firewall recommendations• Considerations for GPFS

applications• Security-Enhanced Linux support• Space requirements for call home

data upload

System administrators, analysts,installers, planners, andprogrammers of IBM SpectrumScale clusters who are veryexperienced with the operatingsystems on which each IBMSpectrum Scale cluster is based

viii IBM Spectrum Scale: Container Storage Interface Driver Guide Version 2.0.0

Table 1. IBM Spectrum Scale library information units (continued)



Installing

• Steps for establishing and startingyour IBM Spectrum Scale cluster

• Installing IBM Spectrum Scale onLinux nodes and deployingprotocols

• Installing IBM Spectrum Scale onAIX nodes

• Installing IBM Spectrum Scale onWindows nodes

• Installing Cloud services on IBMSpectrum Scale nodes

• Installing and configuring IBMSpectrum Scale management API

• Installation of Active FileManagement (AFM)

• Installing and upgrading AFM-based Disaster Recovery

• Installing call home• Installing file audit logging• Installing watch folder API• Installing clustered watch folder• Steps to permanently uninstall

GPFS

Upgrading

• IBM Spectrum Scale supportedupgrade paths

• Upgrading to IBM Spectrum Scale5.0.x from IBM Spectrum Scale4.2.y or later

• Upgrading to IBM Spectrum Scale4.2.y from IBM Spectrum Scale4.1.x

• Online upgrade support forprotocols and performancemonitoring


About this information ix




• Upgrading IBM Spectrum® Scalenon-protocol Linux nodes

• Upgrading IBM Spectrum Scaleprotocol nodes

• Upgrading AFM and AFM DR• Upgrading object packages• Upgrading SMB packages• Upgrading NFS packages• Upgrading call home• Manually upgrading the

performance monitoring tool• Manually upgrading pmswift• Manually upgrading the IBM

Spectrum Scale management GUI• Upgrading Cloud services• Upgrading to IBM Cloud Object

Storage software level 3.7.2 andabove

• Upgrade paths and commands forfile audit logging, watch folderAPI, and clustered watch folder

• Upgrading with clustered watchfolder enabled

• Upgrading IBM Spectrum Scalecomponents with the installationtoolkit

• Changing IBM Spectrum Scaleproduct edition

• Completing the upgrade to a newlevel of IBM Spectrum Scale

• Reverting to the previous level ofIBM Spectrum Scale



• Coexistence considerations• Compatibility considerations• Considerations for IBM Spectrum

Protect for Space Management• Applying maintenance to your

GPFS system• Guidance for upgrading the

operating system on IBMSpectrum Scale nodes

• Servicing IBM Spectrum Scaleprotocol nodes

• Offline upgrade with completecluster shutdown

x IBM Spectrum Scale: Container Storage Interface Driver Guide Version 2.0.0



IBM Spectrum Scale:Administration Guide


Configuring

• Configuring the GPFS cluster• Configuring the CES and protocolconfiguration

• Configuring and tuning yoursystem for GPFS

• Parameters for performancetuning and optimization

• Ensuring high availability of theGUI service

• Configuring and tuning yoursystem for Cloud services

• Configuring IBM Power Systemsfor IBM Spectrum Scale

• Configuring the message queue• Configuring file audit logging• Configuring clustered watch

folder• Configuring Active File

Management• Configuring AFM-based DR• Tuning for Kernel NFS backend on

AFM and AFM DR• Configuring call home

Administering

• Performing GPFS administrationtasks

• Verifying network operation withthe mmnetverify command

• Managing file systems• File system format changes

between versions of IBMSpectrum Scale

• Managing disks• Managing protocol services• Managing protocol user

authentication• Managing protocol data exports• Managing object storage• Managing GPFS quotas• Managing GUI users• Managing GPFS access control

lists

System administrators orprogrammers of IBM SpectrumScale systems

About this information xi



IBM Spectrum Scale:Administration Guide

• Native NFS and GPFS• Considerations for GPFS

applications• Accessing a remote GPFS file

system• Information lifecycle

management for IBM SpectrumScale

• Creating and maintainingsnapshots of file systems

• Creating and managing file clones• Scale Out Backup and Restore

(SOBAR)• Data Mirroring and Replication• Implementing a clustered NFS

environment on Linux• Implementing Cluster Export

Services• Identity management on

Windows / RFC 2307 Attributes• Protocols cluster disaster

recovery• File Placement Optimizer• Encryption• Managing certificates to secure

communications between GUIweb server and web browsers

• Securing protocol data• Cloud services: Transparent cloud

tiering and Cloud data sharing• Managing file audit logging• Performing a watch with watch

folder API• RDMA tuning• Configuring Mellanox Memory

Translation Table (MTT) for GPFSRDMA VERBS Operation

• Administering AFM• Administering AFM DR• Highly-available write cache

(HAWC)• Local read-only cache• Miscellaneous advanced

administration• GUI limitations

System administrators orprogrammers of IBM SpectrumScale systems

xii IBM Spectrum Scale: Container Storage Interface Driver Guide Version 2.0.0



IBM Spectrum Scale:Problem DeterminationGuide


Monitoring

• Performance monitoring• Monitoring system health through

the IBM Spectrum Scale GUI• Monitoring system health by using

the mmhealth command• Monitoring events through

callbacks• Monitoring capacity through GUI• Monitoring AFM and AFM DR• GPFS SNMP support• Monitoring the IBM Spectrum

Scale system by using call home• Monitoring remote cluster through

GUI• Monitoring the message queue• Monitoring file audit logging• Monitoring clustered watch

Troubleshooting

• Best practices for troubleshooting• Understanding the system

limitations• Collecting details of the issues• Managing deadlocks• Installation and configuration

issues• Upgrade issues• Network issues• File system issues• Disk issues• Security issues• Protocol issues• Disaster recovery issues• Performance issues

System administrators of GPFSsystems who are experienced withthe subsystems used to managedisks and who are familiar with theconcepts presented in the IBMSpectrum Scale: Concepts, Planning,and Installation Guide

About this information xiii



IBM Spectrum Scale:Problem DeterminationGuide

• GUI issues• AFM issues• AFM DR issues• Transparent cloud tiering issues• File audit logging issues• Troubleshooting watch folder API• Troubleshooting mmwatch• Message queue issues• Maintenance procedures• Recovery procedures• Support for troubleshooting• References

xiv IBM Spectrum Scale: Container Storage Interface Driver Guide Version 2.0.0



IBM Spectrum Scale:Command and ProgrammingReference


Command reference

• gpfs.snap command• mmaddcallback command• mmadddisk command• mmaddnode command• mmadquery command• mmafmconfig command• mmafmctl command• mmafmlocal command• mmapplypolicy command• mmaudit command• mmauth command• mmbackup command• mmbackupconfig command• mmblock command• mmbuildgpl command• mmcachectl command• mmcallhome command• mmces command• mmcesdr command• mmchattr command• mmchcluster command• mmchconfig command• mmchdisk command• mmcheckquota command• mmchfileset command• mmchfs command• mmchlicense command• mmchmgr command• mmchnode command• mmchnodeclass command• mmchnsd command• mmchpolicy command• mmchpool command• mmchqos command• mmclidecode command

• System administrators of IBMSpectrum Scale systems

• Application programmers who areexperienced with IBM SpectrumScale systems and familiar withthe terminology and concepts inthe XDSM standard

About this information xv




• mmclone command• mmcloudgateway command• mmcrcluster command• mmcrfileset command• mmcrfs command• mmcrnodeclass command• mmcrnsd command• mmcrsnapshot command• mmdefedquota command• mmdefquotaoff command• mmdefquotaon command• mmdefragfs command• mmdelacl command• mmdelcallback command• mmdeldisk command• mmdelfileset command• mmdelfs command• mmdelnode command• mmdelnodeclass command• mmdelnsd command• mmdelsnapshot command• mmdf command• mmdiag command• mmdsh command• mmeditacl command• mmedquota command• mmexportfs command• mmfsck command• mmfsctl command• mmgetacl command• mmgetstate command• mmhadoopctl command• mmhdfs command• mmhealth command• mmimgbackup command• mmimgrestore command• mmimportfs command• mmkeyserv command



xvi IBM Spectrum Scale: Container Storage Interface Driver Guide Version 2.0.0




• mmlinkfileset command• mmlsattr command• mmlscallback command• mmlscluster command• mmlsconfig command• mmlsdisk command• mmlsfileset command• mmlsfs command• mmlslicense command• mmlsmgr command• mmlsmount command• mmlsnodeclass command• mmlsnsd command• mmlspolicy command• mmlspool command• mmlsqos command• mmlsquota command• mmlssnapshot command• mmmigratefs command• mmmount command• mmmsgqueue command• mmnetverify command• mmnfs command• mmnsddiscover command• mmobj command• mmperfmon command• mmpmon command• mmprotocoltrace command• mmpsnap command• mmputacl command• mmquotaoff command• mmquotaon command• mmreclaimspace command• mmremotecluster command• mmremotefs command• mmrepquota command• mmrestoreconfig command• mmrestorefs command• mmrestripefile command



About this information xvii




• mmrestripefs command• mmrpldisk command• mmsdrrestore command• mmsetquota command• mmshutdown command• mmsmb command• mmsnapdir command• mmstartup command• mmtracectl command• mmumount command• mmunlinkfileset command• mmuserauth command• mmwatch command• mmwinservctl command• spectrumscale command

Programming reference

• IBM Spectrum Scale DataManagement API for GPFSinformation

• GPFS programming interfaces• GPFS user exits• IBM Spectrum Scale management

API commands• Watch folder API



xviii IBM Spectrum Scale: Container Storage Interface Driver Guide Version 2.0.0



IBM Spectrum Scale: BigData and Analytics Guide


Summary of changes

Hadoop Scale Storage Architecture

• Elastic Storage Server (ESS)• Erasure Code Edition• Share Storage (SAN-based

storage)• File Placement Optimizer (FPO)• Deployment model• Additional supported features

about storage

IBM Spectrum Scale support forHadoop

• HDFS transparency• Supported IBM Spectrum Scale

storage modes• Hadoop cluster planning• CES HDFS• Installation and configuration of

HDFS transparency• Application interaction with HDFS

transparency• Upgrading the HDFS

Transparency cluster• Rolling upgrade for HDFS

Transparency• Security• Advanced features• Hadoop distribution support• Limitations and differences from

native HDFS• Problem determination

IBM Spectrum Scale Hadoopperformance tuning guide

• Overview• Performance overview• Hadoop Performance Planning

over IBM Spectrum Scale• Performance guide



About this information xix



IBM Spectrum Scale: BigData and Analytics Guide

Hortonworks Data Platform 3.X

• Planning• Installation• Upgrading and uninstallation• Configuration• Administration• Limitations• Problem determination

Open Source Apache Hadoop

• Open Source Apache Hadoopwithout CES HDFS

• Open Source Apache Hadoop withCES HDFS

BigInsights® 4.2.5 and HortonworksData Platform 2.6

• Planning• Installation• Upgrading software stack• Configuration• Administration• Troubleshooting• Limitations• FAQ



xx IBM Spectrum Scale: Container Storage Interface Driver Guide Version 2.0.0



IBM Spectrum Scale ErasureCode Edition Guide

IBM Spectrum Scale Erasure CodeEdition

• Summary of changes• Introduction to IBM Spectrum

Scale Erasure Code Edition• Planning for IBM Spectrum Scale

Erasure Code Edition• Installing IBM Spectrum Scale

Erasure Code Edition• Uninstalling IBM Spectrum Scale

Erasure Code Edition• Incorporating IBM Spectrum

Scale Erasure Code Edition in anElastic Storage Server (ESS)cluster

• Creating an IBM Spectrum ScaleErasure Code Edition storageenvironment

• Upgrading IBM Spectrum ScaleErasure Code Edition

• Administering IBM SpectrumScale Erasure Code Edition

• Troubleshooting• IBM Spectrum Scale RAID

Administration



Prerequisite and related informationFor updates to this information, see IBM Spectrum Scale in IBM Knowledge Center (www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.html).

For the latest support information, see the IBM Spectrum Scale FAQ in IBM Knowledge Center(www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html).

Conventions used in this informationTable 2 on page xxii describes the typographic conventions used in this information. UNIX file nameconventions are used throughout this information.

Note: Users of IBM Spectrum Scale for Windows must be aware that on Windows, UNIX-style filenames need to be converted appropriately. For example, the GPFS cluster configuration data is stored inthe /var/mmfs/gen/mmsdrfs file. On Windows, the UNIX namespace starts under the %SystemDrive%\cygwin64 directory, so the GPFS cluster configuration data is stored in the C:\cygwin64\var\mmfs\gen\mmsdrfs file.

About this information xxi

http://www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.htmlhttp://www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.htmlhttp://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.htmlhttp://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html

Table 2. Conventions

Convention Usage

bold Bold words or characters represent system elements that you must use literally,such as commands, flags, values, and selected menu options.

Depending on the context, bold typeface sometimes represents path names,directories, or file names.

boldunderlined

bold underlined keywords are defaults. These take effect if you do not specify adifferent keyword.

constant width Examples and information that the system displays appear in constant-widthtypeface.

Depending on the context, constant-width typeface sometimes represents pathnames, directories, or file names.

italic Italic words or characters represent variable values that you must supply.

Italics are also used for information unit titles, for the first use of a glossary term,and for general emphasis in text.

Angle brackets (less-than and greater-than) enclose the name of a key on thekeyboard. For example, refers to the key on your terminal or workstationthat is labeled with the word Enter.

\ In command examples, a backslash indicates that the command or coding examplecontinues on the next line. For example:

mkcondition -r IBM.FileSystem -e "PercentTotUsed > 90" \-E "PercentTotUsed < 85" -m p "FileSystem space used"

{item} Braces enclose a list from which you must choose an item in format and syntaxdescriptions.

[item] Brackets enclose optional items in format and syntax descriptions.

The notation indicates a control character sequence. For example, means that you hold down the control key while pressing .

item... Ellipses indicate that you can repeat the preceding item one or more times.

| In synopsis statements, vertical lines separate a list of choices. In other words, avertical line means Or.

In the left margin of the document, vertical lines indicate technical changes to theinformation.

Note: CLI options that accept a list of option values delimit with a comma and no space between values.As an example, to display the state on three nodes use mmgetstate -N NodeA,NodeB,NodeC.Exceptions to this syntax are listed specifically within the command.

How to send your commentsYour feedback is important in helping us to produce accurate, high-quality information. If you have anycomments about this information or any other IBM Spectrum Scale documentation, send your commentsto the following e-mail address:

[email protected]

Include the publication title and order number, and, if applicable, the specific location of the informationabout which you have comments (for example, a page number or a table number).

xxii IBM Spectrum Scale: Container Storage Interface Driver Guide Version 2.0.0

To contact the IBM Spectrum Scale development organization, send your comments to the following e-mail address:

[email protected]

About this information xxiii

xxiv IBM Spectrum Scale: Container Storage Interface Driver Guide Version 2.0.0

Chapter 1. Summary of changesSummary of changes for IBM Spectrum Scale Container Storage Interface driver.

The following enhancements have been made in this release:

• Support for Red Hat® OpenShift® 4.4 and Kubernetes 1.18• Support for imagePullSecrets• Upgraded the operator-sdk version from 0.11.0 to 0.17.0• Upgraded the ansible-operator version from v0.14.0 to v0.17.0• Made the clusterId parameter optional in storageClass (default: Primary Cluster ID)• Made the primaryFset parameter optional for creating the primary fileset (default: spectrum-scale-csi-

volume-store)• Improved custom resource parameter validation and error messages• Improvements in the debug data collection tool• Removed the remoteFs parameter from the csiscaleoperators.csi.ibm.com_cr.yaml file.

© Copyright IBM Corp. 2015, 2020 1

2 IBM Spectrum Scale: Container Storage Interface Driver Guide Version 2.0.0

Chapter 2. Introduction to IBM Spectrum ScaleContainer Storage Interface driver

This section provides a brief introduction to IBM Spectrum Scale Container Storage Interface driver.

IBM Spectrum Scale is a clustered file system that provides concurrent access to a single file system orset of file systems from multiple nodes. The nodes can be SAN attached, network attached, a mixture ofSAN attached, and network attached, or in a shared nothing cluster configuration. This enables highperformance access to this common set of data to support a scale-out solution or to provide a highavailability platform. For more information on IBM Spectrum Scale features, see the Product overviewsection in the IBM Spectrum Scale: Concepts, Planning, and Installation Guide.

Container Storage Interface (CSI) is a standard for exposing arbitrary block and file storage systems tocontainerized workloads on Container Orchestration Systems (COs) like Kubernetes. The IBM SpectrumScale Container Storage Interface driver specification is defined at https://github.com/container-storage-interface/spec/blob/master/spec.md.

IBM Spectrum Scale Container Storage Interface driver allows IBM Spectrum Scale to be used as apersistent storage for stateful application running in Kubernetes clusters. Through the IBM SpectrumScale Container Storage Interface driver, Kubernetes persistent volumes (PVs) can be provisioned fromIBM Spectrum Scale. Thus, the containers can be used with stateful microservices like MongoDB,PostgreSQL, and so on.

Figure 1. IBM Spectrum Scale Container Storage Interface Driver

IBM implements the CSI specification of storage plug-in in the following manner:

The external-provisioner and external-attacher Sidecar Containers are StatefulSets, which should bedeployed on separate infrastructure hosts for resiliency. The external-provisioner watches for create ordelete API calls while the external-attacher watches for mount or unmount API calls. The node-driver-registrar is a kubelet service, which runs alongside the node plug-in at the time of initialization. The IBM


https://github.com/container-storage-interface/spec/blob/master/spec.mdhttps://github.com/container-storage-interface/spec/blob/master/spec.md

Spectrum Scale Container Storage Interface driver provides the interconnect for persistent volume mountfrom the container worker node to the underlying storage system and is deployed as a DaemonSet. TheIBM Spectrum Scale Container Storage Interface driver also makes the REST API calls on the IBMSpectrum Scale storage system to perform any provisioning requests on the storage. For moreinformation on StatefulSets and Daemonsets, see StatefulSets.

Features covered

The following features are available with IBM Spectrum Scale Container Storage Interface driver:

• Static provisioning: Ability to use existing directories and filesets as persistent volumes• Lightweight dynamic provisioning: Ability to create directory-based volumes dynamically• Fileset-based dynamic provisioning: Ability to create fileset-based volumes dynamically• Multiple file systems support: Volumes can be created across multiple file systems• Remote mount support: Volumes can be created on a remotely mounted file system• Operator support for easy deployment, upgrade, and cleanup• Supported volume access mode: RWX (Read-write many)


https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

Chapter 3. Planning for IBM Spectrum ScaleContainer Storage Interface driver

This topic describes the planning information for using IBM Spectrum Scale Container Storage Interfacedriver.

Note: If you are using a Red Hat OpenShift cluster, ensure to replace "kubectl" with "oc" in all commands.

Hardware and Software requirementsThe following hardware and software requirements must be met for using IBM Spectrum Scale ContainerStorage Interface driver at your site:

• RHEL version 7.5, 7.6, and 7.7 (x86_64 architecture)• IBM Spectrum Scale version 5.0.4.1 or later• Kubernetes version 1.16, 1.17, and 1.18• Red Hat OpenShift 4.3 and 4.4 (x86_64 architecture) through RHEL worker nodes

Deployment considerationsEnsure that the following steps are completed before deploying IBM Spectrum Scale Container StorageInterface driver in your cluster.

• Worker nodes selection: By default, Kubernetes/Red Hat OpenShift schedules the IBM Spectrum ScaleContainer Storage Interface driver pods on all worker nodes. It is essential to have IBM Spectrum Scaleclient installed on all these nodes. If you wish to schedule the IBM Spectrum Scale Container StorageInterface driver pods only on selected worker nodes, you must label the selected nodes and use thislabel in node selector. For more information, see “Using the node selector” on page 25.

• Node selection for StatefulSets: CSI external attacher and CSI external provisioner are sidecarcontainers that run as two separate StatefulSets. These pods can be scheduled on any of the workernodes by Kubernetes. As a best practice, it is recommended to run these on two separate stable nodes.The StatefulSets by design of Kubernetes do not automatically fail over to another node, hence it isrecommended to schedule them to run on reliable nodes. On Red Hat OpenShift, if the infrastructurenodes are worker nodes, it is recommended to schedule the sidecar containers to run on infrastructurenodes. Scheduling them to run on specific nodes can be achieved by using nodes labels andnodeSelectors. For more information, see “Using the node selector” on page 25. IBM Spectrum ScaleContainer Storage Interface driver pod must also be scheduled on the nodes that runStatefulSets.

• Remote cluster setup: If you plan to use remotely mounted file system for PVC provisioning, ensure thefollowing:

– IBM Spectrum Scale GUI is initialized and running on both clusters (owning cluster and accessingcluster)

– Remote cluster details are added to the Operator configuration. For more information, see “Remotecluster support” on page 21.

• SELinux considerations: Different Kubernetes distributions handle the SELinux enforcing modedifferently. There might be differences in terms of SELinux context being set on files, relabeling ofvolumes and the process context of containers. As a prerequisite, appropriate SELinux rules must be setup to allow IBM Spectrum Scale Container Storage Interface driver containers to access the requiredresources on host. For example “container_t” context needs to have access to csi.sock and the IBMSpectrum Scale file system, or the files needing access from containers need to have the“container_file_t” context set. Refer to audit logs for any SELinux failures and set up appropriaterules as required.


• Node names: At times, it is possible that IBM Spectrum Scale cluster and Kubernetes/Red HatOpenShift cluster are configured with different node names for the same host. Use the mmlsclusterand kubectl get nodes commands and check the node names of IBM Spectrum Scale cluster andKubernetes cluster. If the names are different, then configure node mapping in the Operatorconfiguration. For more information, see “Kubernetes to IBM Spectrum Scale node mapping” on page26.

Note: Node names should match with the output of the mmlsmount -L command, or thecurl command curl --insecure -u 'username:password' -X GET https://guiHost:443/scalemgmt/v2/filesystems/?fields=mount.

• Internet connectivity: If your worker nodes have internet connectivity and access to the quay.ioregistry, the IBM Spectrum Scale Container Storage Interface driver downloads and uploads therequired images automatically during deployment. Otherwise, you need to manually download thefollowing images:

– quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver:v2.0.0– quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator:v2.0.0– quay.io/k8scsi/csi-node-driver-registrar:v1.2.0– quay.io/k8scsi/csi-attacher:v2.1.1– quay.io/k8scsi/csi-provisioner:v1.5.0

Note: Upload these images on all worker nodes using docker/podman command. In case you use aprivate registry for your Red Hat OpenShift/Kubernetes environment, upload the images to your privateregistry. Use the new image URL and registry credentials in the imagePullSecrets section of thecsiscaleoperators.csi.ibm.com_cr.yaml file. For more information, see the following topics:

– To create secrets using private registry credentials, see Pull an Image from a Private Registry– For information on the imagePullSecrets section, see thecsiscaleoperators.csi.ibm.com_cr.yaml file on the “Operator” on page 19 page.

• IBM Spectrum Scale services: As a best practice, it is recommended that GUI nodes, protocol nodes,and NSD nodes are not part of the Kubernetes cluster.


https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

Chapter 4. Installation of IBM Spectrum ScaleContainer Storage Interface driver

This section explains how to install or clean up the IBM Spectrum Scale Container Storage Interfacedriver. CSI Operators are used for performing these activities.

Performing pre-installation tasksComplete the following tasks before you start installing the IBM Spectrum Scale Container StorageInterface driver:

• Install IBM Spectrum Scale along with the IBM Spectrum Scale management API (GUI).

– For the supported versions, see IBM Spectrum Scale FAQ.– For information on installing IBM Spectrum Scale, see the Installing IBM Spectrum Scale on Linux

nodes with the installation toolkit topic in IBM Spectrum Scale: Concepts, Planning, and InstallationGuide.

• Install and set up either Kubernetes or Red Hat OpenShift.

– For supported versions, see IBM Spectrum Scale FAQ.– For information on installing Kubernetes, see the Install and Set Up kubectl topic in the Kubernetes

documentation.– For information on installing Red Hat OpenShift, see the Installing and configuring OpenShift

Container Platform clusters topic in the Red Hat OpenShift documentation.• Install the IBM Spectrum Scale client on the required Kubernetes worker nodes, and these nodes are

added to the IBM Spectrum Scale cluster.• Mount the primary file system that is used for IBM Spectrum Scale configuration on all Kubernetes

worker nodes and on the IBM Spectrum Scale GUI nodes.• Create an IBM Spectrum Scale user group "CsiAdmin" if it does not already exist. Use the /usr/lpp/mmfs/gui/cli/mkusergrp CsiAdmin --role csiadmin command to create the CsiAdmin user.Create an IBM Spectrum Scale user in the "CsiAdmin" group. This user must be used on IBM SpectrumScale Container Storage Interface driver configuration. Issue this command on the GUI node to createthe user:

/usr/lpp/mmfs/gui/cli/mkuser -p -g CsiAdmin

• Ensure that all the worker nodes where IBM Spectrum Scale Container Storage Interface driver shouldbe scheduled are running Red Hat Enterprise Linux (RHEL) x86_64. For supported versions, see https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html.

• Initialize the IBM Spectrum Scale GUI either by logging into the GUI console once or by running thiscommand on the GUI node:

/usr/lpp/mmfs/gui/cli/initgui

• Issue the following command from the Kubernetes node to ensure that the GUI server is running andcan communicate with the Kubernetes nodes:

curl --insecure -u 'username:password' -X GET https://guihost:443/scalemgmt/v2/cluster

The command gives an output similar to the following:

{ "cluster" : { "clusterSummary" : { "clusterId" : 17258972170939727157,


https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.htmlhttps://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.htmlhttps://kubernetes.io/docs/tasks/tools/install-kubectl/https://access.redhat.com/documentation/en-us/openshift_container_platform/4.4/html-single/installing/indexhttps://access.redhat.com/documentation/en-us/openshift_container_platform/4.4/html-single/installing/indexhttps://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.htmlhttps://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html

"clusterName" : "node10.node10", "primaryServer" : "node10", "rcpPath" : "/usr/bin/scp", "rcpSudoWrapper" : false, "repositoryType" : "CCR", "rshPath" : "/usr/bin/ssh", "rshSudoWrapper" : false, "uidDomain" : "node10.node10" }, "capacityLicensing" : { "liableCapacity" : 96636764160, "liableNsdCount" : 2, "liableNsds" : [ { "nsdName" : "nsd1", "liableCapacity" : 53687091200 }, { "nsdName" : "nsd2", "liableCapacity" : 42949672960 } ] } }, "status" : { "code" : 200, "message" : "The request finished successfully." }}

• Ensure that perfileset quota on the file systems to be used by IBM Spectrum Scale Container StorageInterface driver is set to "No"

# mmlsfs fs1 --perfileset-quotaflag value description------------------- ------------------------ ----------------------------------- --perfileset-quota No Per-fileset quota enforcement

• Enable quota for all the file systems (if not already done) being used for creating persistent volumesthat are required for fileset-based dynamic provisioning.

mmchfs gpfs0 -Q yes

Verify that quota is enabled.

mmlsfs gpfs0 -Q

This command gives an output similar to the following:

flag value description------------------- ------------------------ ----------------------------------- -Q user;group;fileset Quotas accounting enabled user;group;fileset Quotas enforced none Default quotas enabled

• Enable quota for root user by issuing this command:

mmchconfig enforceFilesetQuotaOnRoot=yes -i

• For Red Hat OpenShift, ensure that the controlSetxattrImmutableSELinux parameter is set to"yes" by issuing the following command:

mmchconfig controlSetxattrImmutableSELinux=yes -i

• To display the correct volume size in the container, enable filesetdf of the file system by using thefollowing command:

mmchfs --filesetdf

• Mount the file system that is used for IBM Spectrum Scale Container Storage Interface driver on thesame mount point on worker nodes.


• Issue the following command to label the Kubernetes worker nodes where IBM Spectrum Scale client isinstalled and where IBM Spectrum Scale Container Storage Interface driver should run:

kubectl label node node1 scale=true --overwrite=true

For more information, see “Using the node selector” on page 25.

Installing IBM Spectrum Scale Container Storage Interface driver usingOperator Lifecycle Manager

This topic describes the procedure for installing IBM Spectrum Scale Container Storage Interface driverusing Operator Lifecycle Manager (OLM).

OLM runs by default on Red Hat OpenShift Container Platform 4.2 and later releases. For moreinformation, see Operator Lifecycle Manager workflow and architecture. OLM is not available by defaulton Kubernetes, and it is recommended to use CLI-based installation of the IBM Spectrum Scale ContainerStorage Interface driver on Kubernetes. For more information, see “Installing IBM Spectrum ScaleContainer Storage Interface driver using CLIs” on page 10. However, if OLM is already installed on theKubernetes cluster, then use the method that is described here.

Note: If you refer to the installation instructions on the Operator Hub (https://operatorhub.io/operator/ibm-spectrum-scale-csi-operator), make note of the following:

• Ignore step #2 (kubectl create -f https://operatorhub.io/install/ibm-spectrum-scale-csi-operator.yaml), and do the following steps:

1. Download the yaml file.

curl -O https://operatorhub.io/install/ibm-spectrum-scale-csi-operator.yaml

2. Remove "my-" from any names.

sed -i 's/my-//g' ./ibm-spectrum-scale-csi-operator.yaml

3. Apply the yaml file.

kubectl apply -f ibm-spectrum-scale-csi-operator.yaml

• Ignore step #3 (kubectl get csv -n my-ibm-spectrum-scale-csi-operator), and issue thefollowing command:

kubectl get csv -n ibm-spectrum-scale-csi-operator

Before performing IBM Spectrum Scale Container Storage Interface driver installation, ensure that theprerequisites are met. For more information, see “Performing pre-installation tasks” on page 7.

Create the Operator from the Red Hat OpenShift console by performing the following steps:

a. Log in to the Red Hat OpenShift Container Platform.b. From the left panel, click Operators > OperatorHub. The OperatorHub page appears.c. From the Project drop-down list, select the project or create a new project by clicking Create

Project.d. Under All Items, select Storage from the list.e. In the Filter by keyword box, type "IBM Spectrum Scale CSI".f. Click IBM Spectrum Scale CSI Plugin Operator. The IBM Spectrum Scale CSI Plugin Operator page

appears.g. Click Install. The Create Operator Subscription page appears.

Chapter 4. Installation of IBM Spectrum Scale Container Storage Interface driver 9

https://docs.openshift.com/container-platform/4.4/operators/understanding_olm/olm-understanding-olm.htmlhttps://operatorhub.io/operator/ibm-spectrum-scale-csi-operatorhttps://operatorhub.io/operator/ibm-spectrum-scale-csi-operator

h. On this page, select a namespace as that of operator from the available options, select the approvalstrategy (automatic or manual), and click Subscribe. The Installed Operators page appears, whereIBM Spectrum Scale CSI Plugin Operator is listed as successfully installed.

i. On the Installed Operators page, click IBM Spectrum Scale CSI Plugin Operator, and go to the IBMCSI Spectrum Scale Driver tab.

j. Click Create CSIScale Operator. The Create CSIScale Operator page appears.k. On this page, an editor appears, where you can update the manifest file according to your

environment. For more information, see “Operator” on page 19, “Secrets” on page 19, and“Certificates” on page 19.

Installing IBM Spectrum Scale Container Storage Interface driver using CLIsThis topic describes the procedures for installing IBM Spectrum Scale Container Storage Interface driverby using CLIs.

Before performing IBM Spectrum Scale Container Storage Interface driver installation, ensure that theprerequisites are met. For more information, see “Performing pre-installation tasks” on page 7.

Note: This procedure is applicable for both Kubernetes and Red Hat OpenShift. For Red Hat OpenShift,replace "kubectl" with "oc" in all the commands.

Installing IBM Spectrum Scale Container Storage Interface driver using Operator involves the followingphases:

1. Deploy the Operator on your cluster2. Use the Operator for deploying IBM Spectrum Scale Container Storage Interface driver

Phase 1: Deploying the Operator

To deploy Operator on your cluster, do the following steps:

1. Create a namespace.

kubectl create namespace ibm-spectrum-scale-csi-driver

Note: For Red Hat OpenShift, use this command:

oc new-project ibm-spectrum-scale-csi-driver

2. kubectl create -f https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.0.0/generated/\installer/ibm-spectrum-scale-csi-operator.yaml

3. Verify that the Operator is deployed, and the Operator pod is in running state.

kubectl get pod,deployment -n ibm-spectrum-scale-csi-driver

NAME READY STATUS RESTARTS AGEpod/ibm-spectrum-scale-csi-operator-6ff9cf6979-v5g4c 1/1 Running 0 6d3h

NAME READY UP-TO-DATE AVAILABLE AGEdeployment.extensions/ibm-spectrum-scale-csi-operator 1/1 1 1 6d3h

Phase 2: Deploying IBM Spectrum Scale Container Storage Interface driver

Now that the Operator is up and running, you must access the Operator's API and request a deployment.This is done through the use of the CSIScaleOperator custom resource.

Do the following steps:

1. Create a secret with IBM Spectrum Scale GUI server’s credentials in the ibm-spectrum-scale-csi-driver namespace. For more information, see “Secrets” on page 19.

Note: If you are using a remote cluster setup, then create a secret object for each cluster’s GUI server.


2. Download the sample custom resource file on your cluster:

curl -O https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.0.0/operator/deploy/crds/\csiscaleoperators.csi.ibm.com_cr.yaml

3. Modify the parameters in the file to suit your environment. For more information, see “Operator” onpage 19.

4. Apply the custom resource file to deploy IBM Spectrum Scale Container Storage Interface driver.

kubectl apply -f csiscaleoperators.csi.ibm.com_cr.yaml

5. Verify that IBM Spectrum Scale Container Storage Interface driver is installed, Operator and driverresources are ready, and pods are in running state:

# kubectl get pod,daemonset,statefulset -n ibm-spectrum-scale-csi-driverNAME READY STATUS RESTARTS AGEpod/ibm-spectrum-scale-csi-attacher-0 1/1 Running 0 4d2hpod/ibm-spectrum-scale-csi-dxslh 2/2 Running 0 4d2hpod/ibm-spectrum-scale-csi-provisioner-0 1/1 Running 0. 4d2hpod/ibm-spectrum-scale-csi-operator-6ff9cf6979-v5g4c 1/1 Running 0 1h

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGEdaemonset.apps/ibm-spectrum-scale-csi 1 1 1 1 1 4d2h

NAME READY AGEstatefulset.apps/ibm-spectrum-scale-csi-attacher 1/1 4d2hstatefulset.apps/ibm-spectrum-scale-csi-provisioner 1/1 4d2h

For more information, see https://github.com/IBM/ibm-spectrum-scale-csi.

Cleaning up IBM Spectrum Scale Container Storage Interface driver andOperators by using OLM

This topic describes the procedure for uninstalling IBM Spectrum Scale Container Storage Interface driverand Operator by using OLM.

Cleanup procedure includes two tasks: deleting IBM Spectrum Scale Container Storage Interface driverand deleting the Operator.

Cleaning up IBM Spectrum Scale Container Storage Interface driver


1. Log in to the Red Hat OpenShift Container Platform.2. Click Operators > Installed Operators. The Installed Operators page appears, where IBM Spectrum

Scale CSI Plugin Operator is listed.3. From the Project : default drop-down list, select the project in which the Operator is installed.4. On the Installed Operators page, click IBM Spectrum Scale CSI Plugin. The Operator Details page

appears.5. Go to the IBM CSI Spectrum Scale Driver tab.6. Click ibm-spectrum-scale-csi and select CSIScale Operator Delete from the Actions drop-down list.7. Click Delete to confirm deletion.

Cleaning up the Operator


1. Log in to the Red Hat OpenShift Container Platform.2. Click Operators > Installed Operators. The Installed Operators page appears, where IBM Spectrum

Scale CSI Plugin Operator is listed.

Chapter 4. Installation of IBM Spectrum Scale Container Storage Interface driver 11

https://github.com/IBM/ibm-spectrum-scale-csi

3. From the Project : default drop-down list, select the project in which the Operator is installed.4. Click IBM Spectrum Scale CSI Plugin Operator and select Uninstall Operator from the Actions drop-

down list.5. Click Remove for confirmation.

Cleaning up IBM Spectrum Scale Container Storage Interface driver andOperator using CLIs

This topic describes the procedure to clean up or uninstall the IBM Spectrum Scale Container StorageInterface driver and the Operator.

To manage IBM Spectrum Scale Container Storage Interface driver, it is essential that the Operator isalways running on your cluster. If the Operator is deleted for some reason, ensure to re-deploy it by usingthe kubectl apply -f ibm-spectrum-scale-csi-operator.yaml command before proceedingwith the following steps:

1. To stop and uninstall IBM Spectrum Scale Container Storage Interface driver, issue the followingcommand:

kubectl delete -f csiscaleoperators.csi.ibm.com_cr.yaml

2. To uninstall Operator and clean up all resources, issue the following commands:

kubectl delete -f ibm-spectrum-scale-csi-operator.yamlkubectl delete namespace ibm-spectrum-scale-csi-driver

Note: Delete the secrets for GUI credentials and configmap for CA certificates (if any) under the ibm-spectrum-scale-csi-driver namespace.

3. To completely remove IBM Spectrum Scale Container Storage Interface driver and the Operatorimages, do the following steps:

a. Find the images by issuing the following command:

# docker images | grep spectrum-scale-csiquay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver v2.0.0 8b6cb1a1743e 4 hours ago 122 MBquay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator v2.0.0 ced0267b0a45 4 hours ago 505 MB

b. Remove the images by issuing the following command:

# docker rmi 8b6cb1a1743e

Note: If you use a different container engine than Docker, replace the Docker commands with thecommands of the container engine that you use.

4. To delete PVC data, unlink and delete the primary fileset defined in thecsiscaleoperators.csi.ibm.com_cr.yaml file from your IBM Spectrum Scale cluster, byissuing the following commands:

/usr/lpp/mmfs/bin/mmunlinkfileset gpfs0 spectrum-scale-csi-volume-store/usr/lpp/mmfs/bin/mmdelfileset gpfs0 spectrum-scale-csi-volume-store

Note: This will completely delete the PVC data, and any PVCs that were created before will no longerbe useful even if the IBM Spectrum Scale Container Storage Interface driver is reinstalled.


Chapter 5. Upgrading IBM Spectrum Scale ContainerStorage Interface driver

This topic describes the procedure for upgrading IBM Spectrum Scale Container Storage Interface driver.

From version 1.1.0 to version 2.0.0Perform the following steps to upgrade IBM Spectrum Scale Container Storage Interface driver fromversion 1.1.0 to version 2.0.0:

1. Download the operator manifest file on your cluster by issuing the following command:

curl -O https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.0.0/generated\/installer/ibm-spectrum-scale-csi-operator.yaml

2. Apply the manifest file by issuing the following command:

kubectl apply -f ibm-spectrum-scale-csi-operator.yaml

This upgrades the Operator and the IBM Spectrum Scale Container Storage Interface driver. TheOperator and the pods are restarted with the upgraded image.

3. Verify that the pods are back in the running state by issuing the following command:

# kubectl get pod -n ibm-spectrum-scale-csi-driver

The system displays an output similar to this:

NAME READY STATUS RESTARTS AGEibm-spectrum-scale-csi-662zv 2/2 Running 0 29mibm-spectrum-scale-csi-87rw9 2/2 Running 0 29mibm-spectrum-scale-csi-attacher-0 1/1 Running 0 29mibm-spectrum-scale-csi-operator-ddfcc6cb5-z9g29 1/1 Running 0 31mibm-spectrum-scale-csi-provisioner-0 1/1 Running 0 29mibm-spectrum-scale-csi-wkg42 2/2 Running 0 29m

4. Verify that the Operator and the pods are using the upgraded images by issuing the followingcommand:

# kubectl describe pod ibm-spectrum-scale-csi-operator-ddfcc6cb5-z9g29 -n ibm-spectrum-scale-csi-driver \| grep "Image:" | grep ibm-spectrum-scale

Image: quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator:v2.0.0

# kubectl describe pod ibm-spectrum-scale-csi-662zv -n ibm-spectrum-scale-csi-driver |\grep "Image:" | grep ibm-spectrum-scale

Image: quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver:v2.0.0


Chapter 6. Migrating from IBM Storage Enabler forContainers to IBM Spectrum Scale Container StorageInterface driver

This topic describes the procedure for migrating from IBM Storage Enabler for Containers to IBMSpectrum Scale Container Storage Interface driver.

There can be three scenarios in IBM Storage Enabler for Containers, as follows:

• Kubernetes is used: In this case, Kubernetes version 1.12 or earlier used in IBM Storage Enabler forContainers should be upgraded to 1.16 or later.

• IBM Cloud Private (ICP) is used. In this case, ICP should be upgraded to Red Hat OpenShift 4.x by:

– Uninstalling ICP– Freshly installing Red Hat OpenShift 4.x

• Red Hat OpenShift is used. In this case, Red Hat OpenShift version 3.x should be upgraded to version4.x by:

– Uninstalling Red Hat OpenShift 3.x– Freshly installing Red Hat OpenShift 4.x

Note: During the Red Hat OpenShift upgrade, IBM Spectrum Scale client nodes are removed and areadded back after the upgrade.

After the upgrade

• Dynamic provisioning, especially deletion, will not work on migrated volumes.• All dynamically provisioned volumes from IBM Storage Enabler for Containers will be used as static

provisioned volumes in IBM Spectrum Scale Container Storage Interface driver.

High-level stepsThe following procedure provides a high-level overview on migration of IBM Storage Enabler forContainers to IBM Spectrum Scale Container Storage Interface driver.

This is a manual process, which involves downtime till migration completes. For detailed steps, see“Detailed steps ” on page 16.

1. Find all the PersistentVolumeClaim (PVCs) created in IBM Storage Enabler for Containers.2. Create pv.yaml and pvc.yaml for each PV and PVC.3. Change the reclaim policy of all PVs to retain.4. Stop the workloads that are there in the IBM Storage Enabler for Containers.5. Delete all PVs and PVCs created in IBM Storage Enabler for Containers except ibm-ubiquity-db or

equivalent.6. Uninstall IBM Storage Enabler for Containers.7. Upgrade Kubernetes/Red Hat OpenShift/IBM Cloud® Private. For more information, see the

respective documentation.8. Deploy the IBM Spectrum Scale Container Storage Interface driver.9. Create PVs and PVCs using the yaml files (pv.yaml and pvc.yaml).

10. Start the workload using the newly defined PVCs.


Detailed stepsThis topic describes the detailed steps for migrating from IBM Storage Enabler for Containers to IBMSpectrum Scale Container Storage Interface driver.

1. Get all StorageClasses that are created in IBM Storage Enabler for Containers by issuing the followingcommand:

kubectl get sc | grep ubiquity/flex

2. Get all namespaces in IBM Storage Enabler for Containers by issuing the following command:

kubectl get ns

3. For each namespace, get all PVCs for the Storage Classes by issuing the following command:

kubectl get pvc -n | grep

4. For each PVC:

a. Collect the following details:

Table 3. PVC parameter details

ParameterName

Description Example syntax

pvname Name of the PV towhich the PVC isbounded

kubectl get pvc -n -ojsonpath={.spec.volumeName}

pvmountpoint

File systemmount point fromPV definition

kubectl get pv -ojsonpath={.spec.flexVolume.options.mountpoint}

pvfsname File systemmount point fromPV definition

kubectl get pv -ojsonpath={.spec.flexVolume.options.filesystem}

pvcapacity Size of the PVfrom PV definition

kubectl get pv -ojsonpath={.spec.capacity.storage}

pvccapacity Size of the PVCfrom the PVCdefinition

kubectl get pvc -n -ojsonpath={.spec.resources.requests.storage}

pvcaccessmode

Get the PVCaccess modefrom the PVCdefinition

kubectl get pvc -n -ojsonpath={.spec.accessModes[0]}

pvaccessmode

Get the PVCaccess modefrom the PVdefinition

kubectl get pv -n -ojsonpath={.spec.accessModes[0]}

b. Generate pvc.yaml files as shown in the following example. Adjust the parameters according toyour environment.

apiVersion: v1kind: PersistentVolumeClaimmetadata: name: namespace: spec: accessModes: -


resources: requests: storage: volumeName:

Note: Update any other parameters such as storageClass based on your requirement in thepvc.yaml files. Ensure that appropriate modification is made to the dependent pv.yaml files.

c. Generate pv.yaml files using the generate_pv_yaml.sh tool. Adjust the parameters accordingto your environment.

• If primary file system in IBM Storage Enabler for Containers is same as the volume file system($pvfsname), use this command:

./generate_pv_yaml.sh --filesystem $pvfsname --linkpath $pvmountpoint --size $pvcapacity --pvname $pvname --accessmode $pvaccessmode

• If primary file system in IBM Storage Enabler for Containers is different from $pvfsname:

1) Create a softlink in the primary file system area pointing to $pvmountpoint by issuing thefollowing command:

ln -s $pvmountpoint

2) Generate the yaml file by issuing the following command:

./generate_pv_yaml.sh --filesystem $pvfsname --linkpath --size $pvcapacity --pvname $pvname --accessmode $pvaccessmode

Note: Update or add any other parameter based on your requirement in the pv.yaml file.d. Change the reclaim policy of PV to retain.

kubectl patch pv -p '{‘spec"{"persistentVolumeReclaimPolicy":"Retain"}}’

5. Stop all the I/Os for the PVCs in IBM Storage Enabler for Containers.6. Delete all PVCs except ibm-ubiquity-db.7. Delete all PVs except ibm-ubiquity-db.8. Uninstall IBM Storage Enabler for Containers.9. Upgrade Kubernetes/ICP/Openshift.

10. Create all PVCs using the pvc.yaml file created in step 4 b.11. Create all PVs using the pv.yaml created in step 4 c.12. Start I/O using the newly created PVCs.

Chapter 6. Migrating from IBM Storage Enabler for Containers to IBM Spectrum Scale Container StorageInterface driver 17

Chapter 7. Configuring IBM Spectrum ScaleContainer Storage Interface driver

This topic describes the options available for configuring IBM Spectrum Scale Container Storage Interfacedriver at your site.

IBM Spectrum Scale Container Storage Interface driver configurationsDuring IBM Spectrum Scale Container Storage Interface driver plug-in deployment, parameters requiredfor communication with IBM Spectrum Scale must be configured in Kubernetes’ Secrets.

SecretsSecret is needed to store credentials to connect to IBM Spectrum Scale REST API server. Secrets aredefined in data field with base64 encoded values in json file. The GUI user must have csiadmin role.

Perform the following steps:

1. Create secrets.yaml by issuing the following command:

apiVersion: v1kind: Secretmetadata: name: [secret_name] labels: product: ibm-spectrum-scale-csidata: username: [base64_username] password: [base64_password]

Note: Credentials must be base64 encoded. For example, echo -n 'my-password' | base64.2. Apply the secrets, as follows:

kubectl apply -f secrets.yaml -n ibm-spectrum-scale-csi-driver

CertificatesIn case of secure SSL mode, a CA certificate must be specified. This is used in SSL communication withthe IBM Spectrum Scale GUI server. The certificate should be created as a ConfigMap. There should be asmany ConfigMaps as the number of clusters with secure SSL enabled.

For example,

kubectl create configmap cert1 --from-file=cert1=/path/to/mycertificate.pem -n ibm-spectrum-scale-csi-driver

The value from “file=” should be the one that should be used as “cacert” value in CSIScaleOperator

Note: Configmap name and --from-file value should match and this value should be used as“cacert” value in Operator.

OperatorThis topic describes the configuration parameters to be defined for creating a CSIScaleOperator customresource that is used to configure IBM Spectrum Scale Container Storage Interface driver.

A sample configuration yaml file is available here: https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.0.0/operator/deploy/crds/csiscaleoperators.csi.ibm.com_cr.yaml.


https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.0.0/operator/deploy/crds/csiscaleoperators.csi.ibm.com_cr.yamlhttps://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.0.0/operator/deploy/crds/csiscaleoperators.csi.ibm.com_cr.yaml

Figure 2. Operator configuration

• Primary cluster: IBM Spectrum Scale cluster where some or all of the client nodes are also workernodes of Red Hat OpenShift/Kubernetes cluster. The aim of running IBM Spectrum Scale on workernode is to provide persistent storage from IBM Spectrum Scale to the application running onKubernetes/Red Hat OpenShift.

• Primary file system: One of the existing IBM Spectrum Scale file systems from the primary cluster has tobe designated as the primary file system. One fileset from this file system is used by the IBM SpectrumScale Container Storage Interface driver internally to store the volume references. This fileset isreferred to as primary fileset. For proper functioning of IBM Spectrum Scale Container Storage Interfacedriver, the primary file system must be mounted on all worker nodes all the time.

The CSIScaleOperator custom resource for a sample deployment looks like the following. There are twofile systems gpfs0 and gpfs1. For this deployment, we chose gpfs0 as PrimaryFs.

csiscaleoperators.csi.ibm.com_cr.yam file

---apiVersion: csi.ibm.com/v1kind: "CSIScaleOperator"metadata: name: "ibm-spectrum-scale-csi" namespace: "ibm-spectrum-scale-csi-driver" labels: app.kubernetes.io/name: ibm-spectrum-scale-csi-operator app.kubernetes.io/instance: ibm-spectrum-scale-csi-operator app.kubernetes.io/managed-by: ibm-spectrum-scale-csi-operator release: ibm-spectrum-scale-csi-operatorstatus: {}spec: scaleHostpath: "/ibm/gpfs0" clusters:


- id: "" secrets: "guisecretNode1" secureSslMode: false primary: primaryFs: "gpfs0" restApi: - guiHost: "" attacherNodeSelector: - key: "scale" value: "true" provisionerNodeSelector: - key: "scale" value: "true" pluginNodeSelector: - key: "scale" value: "true"---

Table 4. CSIScaleOperator configuration parameter description

Parameter Usage Description

id Mandatory Cluster ID of the primary IBM Spectrum Scale cluster. For moreinformation, see mmlscluster command in the IBM Spectrum Scale:Concepts, Planning, and Installation Guide.

primaryFs Mandatory Primary file system name.

primaryFset Optional Primary fileset name. This will be created if the fileset does not exist.Default value: spectrum-scale-csi-volume-store

inodeLimit Optional Inode limit for the primary fileset. If not specified, fileset is createdwith 1 M inodes, which is the IBM Spectrum Scale default.

cacert Mandatory ifsecureSslModeis true.

Name of the pre-created CA certificate configmap that is used toconnect to the GUI server (running on the "guiHost"). For moreinformation, see “Certificates” on page 19.

secrets Mandatory Name of the pre-created Secret containing username and passwordthat are used to connect to the GUI server for the cluster specificedagainst the id parameter. For more information, see “Secrets” onpage 19.

guiHost Mandatory FQDN or IP address of the GUI node of IBM Spectrum Scale clusterthat is specified against the id parameter.

scaleHostpath

Mandatory Mount path of the primary file system (primaryFs).

imagePullSecrets

Optional An array of imagePullSecrets to be used for pulling images from aprivate registry. This is a pass-through option that distributes theimagePullSecrets array to the containers generated by the Operator.For more information on creating imagePullSecrets, see https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/.

For deployment involving two or more IBM Spectrum Scale clusters, see “Remote cluster support” onpage 21.

Remote cluster supportIBM Spectrum Scale provides a feature to mount IBM Spectrum Scale file systems from one IBMSpectrum Scale cluster (owning cluster) to another IBM Spectrum Scale cluster (accessing cluster). Youcan configure an IBM Spectrum Scale Container Storage Interface driver to work with a remotely mountedIBM Spectrum Scale.

The cluster that owns the file system is responsible for administering the file system and granting accessto other clusters on a per-cluster basis. After access to a file system has been granted to nodes in another

Chapter 7. Configuring IBM Spectrum Scale Container Storage Interface driver 21

https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

IBM Spectrum Scale cluster, the nodes can mount the file system and perform data operations as if thefile system are locally owned.

For more information on the remote mount setup, see Accessing a remote GPFS file system in the IBMSpectrum Scale: Command and Programming Reference.

Note: Remote mount setup must be done before configuring IBM Spectrum Scale Container StorageInterface driver.

Figure 3. Deployment of two IBM Spectrum Scale clusters with remote-mounted file systems

Primary cluster is IBM Spectrum Scale cluster where Red Hat OpenShift/Kubernetes worker nodescoexist with IBM Spectrum Scale client nodes. In this example deployment, cluster A is designated as theprimary cluster.

The cluster O is another IBM Spectrum Scale cluster having two file systems gpfs and fs1. The filesystem gpfs is mounted on Cluster A as file system mygpfs while file system fs1 is not exposed toCluster A.

For each IBM Spectrum Scale cluster, cluster entry must be added under the clusters section of thecustom resource.

- id: "” restApi: - guiHost: "< FQDN or IP GUI node of IBM Spectrum Scale cluster>" secrets: "" secureSslMode: false

One IBM Spectrum Scale cluster has to be the primary cluster for IBM Spectrum Scale Container StorageInterface driver deployment. Primary cluster is marked by adding the primary section in the respectivecluster entry. In the example described in the figure, deployment Cluster A is the primary cluster. Entryfor primary cluster looks as follows:

- id: "” primary: primaryFs: restApi: - guiHost: "< FQDN or IP of Primary Spectrum Scale cluster’s GUI node>" secrets: "” secureSslMode: false

In the example deployment, there are two IBM Spectrum Scale clusters, hence two entries of clusters areadded, one for the primary cluster (Cluster A) and another one for cluster O (Owning cluster).


The custom resource configuration slightly changes based on whether the primary file system is locallyowned (gpfs0 in the example deployment) or remotely mounted ( mygpfs in the example deployment).The changes will be in the primary section of the primary cluster entry.

The custom resource for the example deployment when primaryFS is a locally owned file system (gpfs0)looks as follows:

---apiVersion: csi.ibm.com/v1kind: "CSIScaleOperator"metadata: name: "ibm-spectrum-scale-csi" namespace: "ibm-spectrum-scale-csi-driver" labels: app.kubernetes.io/name: ibm-spectrum-scale-csi-operator app.kubernetes.io/instance: ibm-spectrum-scale-csi-operator app.kubernetes.io/managed-by: ibm-spectrum-scale-csi-operator release: ibm-spectrum-scale-csi-operatorstatus: {}spec: scaleHostpath: "/ibm/gpfs0" clusters: - id: "" secrets: "guisecretNode1" secureSslMode: false primary: primaryFs: "gpfs0" restApi: - guiHost: "" - id: "" secrets: "guisecretNodeA" secureSslMode: false restApi: - guiHost: "" attacherNodeSelector: - key: "scale" value: "true" provisionerNodeSelector: - key: "scale" value: "true" pluginNodeSelector: - key: "scale" value: "true"---

The custom resource for example deployment when primaryFs is a remotely mounted file system(mygpfs) looks like this.

---apiVersion: csi.ibm.com/v1kind: "CSIScaleOperator"metadata: name: "ibm-spectrum-scale-csi" namespace: "ibm-spectrum-scale-csi-driver" labels: app.kubernetes.io/name: ibm-spectrum-scale-csi-operator app.kubernetes.io/instance: ibm-spectrum-scale-csi-operator app.kubernetes.io/managed-by: ibm-spectrum-scale-csi-operator release: ibm-spectrum-scale-csi-operatorstatus: {}spec: scaleHostpath: "/ibm/mygpfs" clusters: - id: "" secrets: "guisecretNode1" secureSslMode: false primary: primaryFs: "mygpfs" remoteCluster: "" restApi: - guiHost: "" - id: "" secrets: "guisecretNodeA" secureSslMode: false restApi: - guiHost: "" attacherNodeSelector: - key: "scale"


value: "true" provisionerNodeSelector: - key: "scale" value: "true" pluginNodeSelector: - key: "scale" value: "true"---

Table 5. Parameter description

Parameter name Status Parameter Description

id Mandatory Cluster ID of IBM Spectrum Scalecluster. For more information,see mmlscluster in the IBMSpectrum Scale: Command andProgramming Reference.

primaryFs Mandatory if cluster is primary Name of the primary file systemon the primary cluster.

primaryFset Optional Primary fileset name. Fileset ofthe specified name will becreated if it does not exist.Default: spectrum-scale-csi-volume-store

remoteCluster Mandatory if the primary filesystem (PrimaryFS) is a remotelymounted file system

Cluster ID of the remote cluster,who is the owner of the filesystem that is specified againstprimaryFs.

inodeLimit Optional Inode limit for the primary fileset.If not specified, fileset is createdwith 1 M inodes, which is the IBMSpectrum Scale default value.

cacert Mandatory if secureSslMode istrue.

Name of the pre-created CAcertificate configmap that is usedto connect to the GUI server thatis running on the guiHost. Formore information, see“Certificates” on page 19.

secrets Mandatory Name of the pre-created Secretcontaining username andpassword to connect to the GUIrunning on the guiHost for clusterspecified against the idparameter. For more information,see “Secrets” on page 19.

guiHost Mandatory FQDN or IP address of the GUInode of IBM Spectrum Scalecluster specified against the idparameter.

scaleHostpath Mandatory Mount path of the primary the filesystem (primaryFs) on primarycluster.


Table 5. Parameter description (continued)

Parameter name Status Parameter Description

imagePullSecrets Optional An array of imagePullSecrets tobe used for pulling images from aprivate registry. This is a pass-through option that distributesthe imagePullSecrets array to thecontainers generated by theOperator. For more informationon creating imagePullSecrets,see https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/.

Note:

• Owning cluster might have more than one file system and not all file systems need to be remotelymounted on the accessing cluster.

• There can be more than one owning cluster exposing their file systems to the accessing cluster.• Accessing cluster/primary cluster can be compute-only cluster without any of its own file system.• Secrets contain the credentials to connect to the GUI for a specified cluster. For each cluster in the

custom resource, there should be a pre-created secret before Operator deployment. For moreinformation, see “Secrets” on page 19. Same secret cannot be used for multiple clusters even if thecredential are same.

• Custom resource also contains other parameters that are optional, so those parameter should be addedas per your requirement.

Using the node selector

By default, the IBM Spectrum Scale Container Storage Interface driver gets deployed on all worker nodes.Node selector controls on which Kubernetes worker nodes IBM Spectrum Scale Container StorageInterface driver should be running. It helps in cases where new worker nodes are added to Kubernetescluster but do not have IBM Spectrum Scale installed. It helps in ensuring that StatefulSets are running onthe desired nodes.

To configure node selector, perform the following steps:

1. Label the Kubernetes worker nodes where StatefulSets should run, as shown in the following example:

• kubectl label node node1 infranode=1 --overwrite=true

• kubectl label node node2 infranode=2 --overwrite=true

Note:

• Use specific labels like the one for attacher and provisioner StatefulSet, only if there is a requirementof running these StatefulSets of very specific nodes. Otherwise, use single label like scale=true forrunning StatefulSets and IBM Spectrum Scale Container Storage Interface driver DaemonSet.

• Nodes marked for running StatefulSet must be subset of the nodes marked with the scale=truelabel.

2. Label the Kubernetes worker nodes where IBM Spectrum Scale Container Storage Interface drivershould run, as follows:

kubectl label node node1 scale=true --overwrite=true


https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

3. Configure the following parameters in the Operator custom resource (ibm-spectrum-scale-csi-operator-cr.yaml) under the "spec" section: IBM Spectrum Scale client must be installed andrunning on the nodes that have the scale=true label.

attacherNodeSelector: - key: "scale" value: "true"# - key: "infranode" # Only if there is requirement of running Attacher # value: "2" # on specific Node

provisionerNodeSelector: - key: "scale" value: "true"# - key: "infranode". # Only if there is requirement of running Provisioner# value: "1" # on specific Node

pluginNodeSelector: - key: "scale" value: "true"

Note: If you choose to run IBM Spectrum Scale Container Storage Interface d

IBM Spectrum Scalevladimir/export/Container... · 2020. 8. 31. · This edition applies to IBM...

Documents

Transcript of IBM Spectrum Scalevladimir/export/Container... · 2020. 8. 31. · This edition applies to IBM...